Cheap fast disc

fivemack · 2009-03-23, 00:03

I'm trying to write some code which needs to do lots of passes which read a ~60G file and write another one (what do you mean, not everybody dreams of a size 2^33 mod-{2^61-1} NTT ? ) It's streaming rather than seeking (read 4GB, write 1GB in each of four places, repeat ...), so basically I want fast transfer rates. I've also run out of space on my current discs.

I'm tempted to get two of the smallest-capacity models of a modern range of hard drives (eg something like http://www.scan.co.uk/Products/320GB...6MB-Cache-0-ms) and use them under Linux software RAID0. Is this sensible? IIRC modern drives read from one platter at a time, so I should expect this one-platter model to have the same 90MB/s read-and-write rates as the terabyte model measured at http://www.tomshardware.com/reviews/...te,2017-6.html, and two of them together ought to be able to read or write ten gigabytes a minute. It's an X58 motherboard so I assume it can handle two independent fast SATA channels - the SATA controllers are at the end of a PCIe x4 link which is 2GB/sec so not a bottleneck.

SSD would be four times the price for a 120G drive, which is not quite enough capacity. 150G VelociRaptor is twice the price, and sort of just about big enough, though might be a bit faster.

S485122 · 2009-03-23, 00:20

Quote:

Originally Posted by fivemack

It's streaming rather than seeking (read 4GB, write 1GB in each of four places, repeat ...), so basically I want fast transfer rates. I've also run out of space on my current discs.

Best thing to do would be to spread the data over 5 disks. No RAID : read on disk 1 write on 2, 3, 4 and 5.... One busy file per disk

Quote:

Originally Posted by fivemack

SSD would be four times the price for a 120G drive, which is not quite enough capacity. 150G VelociRaptor is twice the price, and sort of just about big enough, though might be a bit faster.

Be sure to read the AnandTech article about SSD's before going that way : The SSD Anthology: Understanding SSDs and New Drives from OCZ. There are a limit on the number of writes on SSD disks and other limitations but it may be interesting for you depending on what the disk usage will be exactly.

Jacob

akruppa · 2009-03-23, 00:42

The latest generation of hard disk drives fits 500GB on each platter. So I'd try for a pair of 500GB one-platter drives with 7200rpm, the higher data density should give better linear read performance than a 320GB drive at the same rpm. I have a Samsung HD502HI now which is 500GB one-platter with only 5400rpm (very quiet drive though!), but afaik Samsung and Seagate offer 500GB one-platter at 7200rpm as well.

Alex

fivemack · 2009-03-23, 13:17

Quote:

Originally Posted by S485122

Best thing to do would be to spread the data over 5 disks. No RAID : read on disk 1 write on 2, 3, 4 and 5.... One busy file per disk

I don't think that can be right; it seems very constrained by the read rate from a single disc. Four discs, two partitions per disc, one set of partitions bonded as RAID0 for reading and the others left alone for writing, ought to get full performance. But I don't want the cost and the electricity use of four discs; I'm not really prepared to spend 30% more to get 10% more speed by using 500G/platter rather than 333G/platter models.

lfm · 2009-03-24, 07:41

Quote:

Originally Posted by fivemack

I don't think that can be right; it seems very constrained by the read rate from a single disc. Four discs, two partitions per disc, one set of partitions bonded as RAID0 for reading and the others left alone for writing, ought to get full performance. But I don't want the cost and the electricity use of four discs; I'm not really prepared to spend 30% more to get 10% more speed by using 500G/platter rather than 333G/platter models.

If you're only willing to spring for two drives then you can't get more bandwidth without buying more drives. You could just read from one drive and write to another and you probably won't improve much with other partitioning schemes.

Jeff Gilchrist · 2009-03-24, 12:35

Quote:

Originally Posted by lfm

If you're only willing to spring for two drives then you can't get more bandwidth without buying more drives. You could just read from one drive and write to another and you probably won't improve much with other partitioning schemes.

If he is reading and then writing but not doing both at the same time, it should be faster to have those two drives in RAID-0. You basically double your read and write speeds because it reads/writes half the file to one disc and half to the other simultaneously. So if he is reading in a 4GB file, then doing something and writing four 1 GB files using RAID-0 would half the read time and half the write time compared to just using 1 drive for reading and 1 drive for writing.

rudi_m · 2009-03-29, 14:32

If you often read/write such big files I think the most important is the disk cache.
Nowadays you will get 64GB RAM for about 1200-1600 EUR (plus a motherboard which can handle that).
You will get 50-100 times more throughput if your threads are not reading/writing all at the same time and if you can produce that data as fast at all.

fivemack · 2009-03-29, 20:52

Yes, if I'm willing to spend £400 on a Tyan S2937 board, £500 on two CPUs for it and a further £1600 on sixteen 4GB memory modules, I could put 64GB on a single machine, and it would be extremely fast.

But spending a thirtieth of that on disc drives is getting me something which is, I am reasonably sure, adequate for my purposes.

victor · 2009-04-06, 11:32

Besides the hardware aspect of the question, take a look at ext4. I've been using it for 2 weeks and it looks pretty good. Interresting benchmark here : http://www.phoronix.com/scan.php?pag...nchmarks&num=1

xkey · 2009-04-08, 14:09

Quote:

Originally Posted by fivemack

Yes, if I'm willing to spend £400 on a Tyan S2937 board, £500 on two CPUs for it and a further £1600 on sixteen 4GB memory modules, I could put 64GB on a single machine, and it would be extremely fast.

But spending a thirtieth of that on disc drives is getting me something which is, I am reasonably sure, adequate for my purposes.

How about $400 US for this ramdisk - http://www.acard.com.tw/english/fb01...&type1_idno=13 and then the expense of 8x 8gb ddr2 dimms?

I can provide feedback on its performance circa late May - I am ordering one at the end of this month [those dreaded paycheck vs bills vs child's needs tradeoffs]. It's sustained data transfer rate "should be" approximately 4x that of 10,000 rpm SATA drives.

C++ya,
xkey [who apparently had to re-register after umpteen years of silently reading]

Freightyard · 2009-04-09, 04:33

First question is whether the reads/writes will be concurrent or interleaved. If they are concurrent, you'll want separate drives so you can do both at the same time.

Are you going to have >4GB RAM (leaving room for the OS)? Maybe you should read 2GB at a time? Or, possible write out 2 GB while also reading in the next 2 GB?

Check to make sure there are Intel RAID drivers available for *nix as well. I think there are some serious issues regarding Intel MatrixRAID and *nix.

2009-03-23, 00:03	#1
fivemack (loop (#_fork)) Feb 2006 Cambridge, England 2·7·461 Posts	Cheap fast disc I'm trying to write some code which needs to do lots of passes which read a ~60G file and write another one (what do you mean, not everybody dreams of a size 2^33 mod-{2^61-1} NTT ? ) It's streaming rather than seeking (read 4GB, write 1GB in each of four places, repeat ...), so basically I want fast transfer rates. I've also run out of space on my current discs. I'm tempted to get two of the smallest-capacity models of a modern range of hard drives (eg something like http://www.scan.co.uk/Products/320GB...6MB-Cache-0-ms) and use them under Linux software RAID0. Is this sensible? IIRC modern drives read from one platter at a time, so I should expect this one-platter model to have the same 90MB/s read-and-write rates as the terabyte model measured at http://www.tomshardware.com/reviews/...te,2017-6.html, and two of them together ought to be able to read or write ten gigabytes a minute. It's an X58 motherboard so I assume it can handle two independent fast SATA channels - the SATA controllers are at the end of a PCIe x4 link which is 2GB/sec so not a bottleneck. SSD would be four times the price for a 120G drive, which is not quite enough capacity. 150G VelociRaptor is twice the price, and sort of just about big enough, though might be a bit faster.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Do normal adults give themselves an allowance? (...to fast or not to fast - there is no question!)	jasong	jasong	35	2016-12-11 00:57
Phonograph Cylinder > Compact Disc	roger	Science & Technology	23	2008-12-26 07:11
Worth thrice their weight in disc space	fivemack	Hardware	0	2007-05-01 08:48
Recovering from out-of-disc in ggnfs	fivemack	Factoring	3	2007-02-14 14:51
cheap procs	moo	Hardware	8	2006-07-01 18:55

2009-03-23, 00:42	#3
akruppa "Nancy" Aug 2002 Alexandria 9A3₁₆ Posts	The latest generation of hard disk drives fits 500GB on each platter. So I'd try for a pair of 500GB one-platter drives with 7200rpm, the higher data density should give better linear read performance than a 320GB drive at the same rpm. I have a Samsung HD502HI now which is 500GB one-platter with only 5400rpm (very quiet drive though!), but afaik Samsung and Seagate offer 500GB one-platter at 7200rpm as well. Alex

2009-03-29, 14:32	#7
rudi_m Jul 2005 2×7×13 Posts	If you often read/write such big files I think the most important is the disk cache. Nowadays you will get 64GB RAM for about 1200-1600 EUR (plus a motherboard which can handle that). You will get 50-100 times more throughput if your threads are not reading/writing all at the same time and if you can produce that data as fast at all.

2009-03-29, 20:52	#8
fivemack (loop (#_fork)) Feb 2006 Cambridge, England 2×7×461 Posts	Yes, if I'm willing to spend £400 on a Tyan S2937 board, £500 on two CPUs for it and a further £1600 on sixteen 4GB memory modules, I could put 64GB on a single machine, and it would be extremely fast. But spending a thirtieth of that on disc drives is getting me something which is, I am reasonably sure, adequate for my purposes.

2009-04-06, 11:32	#9
victor Oct 2005 Fribourg, CH 11111100₂ Posts	Besides the hardware aspect of the question, take a look at ext4. I've been using it for 2 weeks and it looks pretty good. Interresting benchmark here : http://www.phoronix.com/scan.php?pag...nchmarks&num=1

2009-04-09, 04:33	#11
Freightyard Nov 2008 San Luis Obispo CA 10000000₂ Posts	First question is whether the reads/writes will be concurrent or interleaved. If they are concurrent, you'll want separate drives so you can do both at the same time. Are you going to have >4GB RAM (leaving room for the OS)? Maybe you should read 2GB at a time? Or, possible write out 2 GB while also reading in the next 2 GB? Check to make sure there are Intel RAID drivers available for nix as well. I think there are some serious issues regarding Intel MatrixRAID and nix.