Oh, there's now a NFS@Home section, and Greg is now a moderator :smile:

[QUOTE=debrouxl;311111]Oh, there's now a NFS@Home section, and Greg is now a moderator :smile:[/QUOTE]
As is someone else... :smile:

Oh ? I was blue when I posted my previous post. In retrospect, I can now see how my post might have been interpreted as an indirect way to ask for becoming green... oh well, I'm not going to complain :grin:

Thanks, I'm honored :wink:

[QUOTE=frmky;310658]

If you wanted to get your hands dirty with OpenMPI, using two reasonably comparable computers connected to the same GigE switch would give you a noticeable speedup. Running two jobs independently, one on each computer, would be more efficient but perhaps not as much fun. :smile:[/QUOTE]

Is it worth to install Infiniband cards over GigE ones when your disk is SATA?
Those core 2 you use for post-processing are connected via InfiniBand 10, 40 or 56 GB/s? What's the MPI ping latency? What's their hard disk interface connection? How much information is red on LA stage to justify having an InfiniBand card?

My questions follows my prices concern on the InfiniBand cards and on SSD drives.

Linear algebra (rather than the matrix-building phase) basically doesn't use the disc - if the matrix doesn't fit in the ensemble RAM you're already doomed. So the disc interface is pretty immaterial.

I've looked a bit at infiniband but the end-of-line SDR cards and switches have now pretty much evaporated; you can get DDR cards from retired compute clusters cheap ($50) on ebay (versus $475 for new DDR and $545 for new QDR), but the cheap switches on ebay are ones that plug into blade servers rather than self-contained boxes, and the cheapest new switch (colfaxdirect.com is the place to go) is $1750 for eight QDR ports.

fivemack,

How much is transferred during a MPI task between nodes with InfiniBand interface to justify the latter?
My point is, I want to buy four equal machines and connect them with a switch so I can share a post-processing task. Will I gain with a InfiniBand connection or GigE is enough for my needs? Also InfiniBand cards are very expensive, I saw in the internet ones costing $8000?!?!

Another doubt I have is why have InfiniBand cards with rates of 56GB/s when an SSD OCZ 800GB drive has a writing rate of 375 MB/s and reading rate of 550 MB/s and costs ~$3000? I can't take advantage of the InfiniBand card, am I thinking right?

Carlos

During an MPI task on a 2x2 grid of machines with a matrix of size 32 million, about 128 megabytes (half the vector of 32 million 64-bit words being handled in the iteration) is passed from each machine to one of its neighbours every iteration. Each iteration deals with about-64 dimensions and there are about-32-million dimensions to deal with, so 500k iterations.

So the total data transfer is about 512 megabytes * 500,000 iterations = 256 terabytes. On a switched gigabit ethernet with a non-blocking switch, the four inter-machine transfers happen in parallel, but you're still waiting for 64 terabytes at 125 megabytes a second, which takes about six days.

On QDR Infiniband, the transfers are happening at five gigabytes a second and the network wait time is down to four hours.

I suppose you're mentioning disc speeds because you generally think of networking in terms of fileservers: but there's nothing going to disc here - the data goes straight from the memory on one machine to the memory on its neighbour.

Small correction to Tom's numbers: every Lanczos iteration involves a matrix multiply and a transpose matrix multiply, so there's actually twice as much data transfer as described above.

I have thought quite seriously a couple of times about building the sort of thing pinhodecarlos is talking about. But I'd be spending €7000 or so plus €1000 a year in electricity on a machine whose only real purpose is linear algebra.

And I'm not convinced there's enough sieving done for it to be worth having such a machine; I could have easily kept up with RSALS on my current 48-core Opteron (runs a 33M matrix in 35 days)

The special cluster could probably handle the 90M matrix for 2^1061-1 (though it would take a fair fraction of a year); it's not big enough to handle the 192M matrix for RSA768 in reasonable time.

But it appears that there are people able to run these large matrices on academic clusters whose running cost comes out of things with names like 'National Science Foundation XSEDE grant number TG-DMS100027' rather than out of anyone's individual wallet.

I was going to suggest 10 GigE as an alternative, but a little research tells me that prices haven't come down as I would have expected, and its cost/performance just isn't competitive with Infiniband now.

The cluster here uses 4x SDR, so it supports only 10 Gb/s in each direction. Ping times are ~ 1-2 microseconds. The compute nodes are diskless. They access data using NFS (the other one) over gigabit ethernet. Other than the initial load of the matrix and the hourly writing of a checkpoint, the disk isn't used in the calculation.

Investing in a personal cluster with Infiniband would make this a very expensive hobby! :smile:

Yeah, 10 GbE is disappointingly expensive...

[quote]Investing in a personal cluster with Infiniband would make this a very expensive hobby! :smile:[/quote]
Yeah, I'd say 20K +/- 5K $ / € for 8 server-class computers + 8 QDR Infiniband cards + a 8-port QDR switch. For a 4x4 grid, with Infiniband FDR equipment, the price tag will be 40-50K...