Purpose Built rig?

[fivemack]

Originally Posted by GP2

Probably the Sandy Bridge box with six cores is not the most cost effective, however. A high-end Xeon box with a lot more cores might make more sense despite the much higher upfront cost. It would be interesting to do the analysis for that too.

My £300 200W second-hand Sandy Bridge boxes are dual eight-core 2GHz Xeons; when I used them for mprime, I ran four workers at four threads each, which gets me (so far as I can parse out from mersenne.org results for 'birch3' of user 'fivemack') four 73M numbers completed per box about every twelve days.

[chalsall]

Originally Posted by Madpoo

I asked how they're measuring power to our circuits... is it a clamp-on ammeter, or something else? No idea. I can only rely on what the APC strips tell me.

Not meaning to make too fine of a point, but do you have SNMP access to meaningful data wrt power consumption?

[westicles]

Probably worth waiting a couple weeks to see if Skylake-X supports AVX-512.

[ewmayer]

Originally Posted by westicles

Probably worth waiting a couple weeks to see if Skylake-X supports AVX-512.

If by 'X' you mean Xeon, we already know that those do indeed support AVX-512. It's only a question of when they become available.

[westicles]

Originally Posted by ewmayer

If by 'X' you mean Xeon, we already know that those do indeed support AVX-512. It's only a question of when they become available.

Not the server chip, but the high-end desktop. This one says it will support AVX-512 but others say it won't.

https://hothardware.com/news/monster...9-7920x-leaked

[ewmayer]

Originally Posted by westicles

Not the server chip, but the high-end desktop. This one says it will support AVX-512 but others say it won't.

https://hothardware.com/news/monster...9-7920x-leaked

Ah, gotcha - I believe the A to your Q is at very end of the linked piece:

Other reportedly confirmed specs for these new parts include supports for dual-channel DDR4-2666 on Kaby Lake-X (quad-channel for Skylake-X), 112W TDP for Kaby Lake-X, 160W TDP for Skylake-X and AVX-512 support for all Core i9 SKUs.

So my surmise is AVX-512 support in all the i9s[Skylake-X], but not in the new i7s[Kaby Lake-X].

[GP2]

Originally Posted by airsquirrels

In reality you could put 3-4 GPUs in one 100W host and drop your incremental $/exponent down to $1.14 or so.

I'm not well-informed about hardware since I use the cloud, but I was wondering if this configuration is in fact realistic? Are you yourself, or anyone else, actually doing this?

Can you really fit four Titan Blacks — all of them generating an amount of heat proportionate to the 250 W that each one consumes — into a single inexpensive host box?

From images, these cards are quite thick and solid, and it's not even clear that you could fit one next to another in adjacent slots. Not to mention that doing so would completely block the airflow of the cooling fan.

Presumably you'd be using some beefed-up liquid cooling system in a very large case, but wouldn't that entail hardware costs above what you initially assumed, and probably additional electricity costs to circulate the liquid?

Can the average motherboard even accept four GPU cards, or would you need to buy a more expensive special-purpose one? Or would this require some kind of elaborate data-center rack mount system, again with additional hardware expense to be taken into account?

[airsquirrels]

Originally Posted by GP2

I'm not well-informed about hardware since I use the cloud, but I was wondering if this configuration is in fact realistic? Are you yourself, or anyone else, actually doing this?

Can you really fit four Titan Blacks — all of them generating an amount of heat proportionate to the 250 W that each one consumes — into a single inexpensive host box?

From images, these cards are quite thick and solid, and it's not even clear that you could fit one next to another in adjacent slots. Not to mention that doing so would completely block the airflow of the cooling fan.

Presumably you'd be using some beefed-up liquid cooling system in a very large case, but wouldn't that entail hardware costs above what you initially assumed, and probably additional electricity costs to circulate the liquid?

Can the average motherboard even accept four GPU cards, or would you need to buy a more expensive special-purpose one? Or would this require some kind of elaborate data-center rack mount system, again with additional hardware expense to be taken into account?

I do indeed have multiple systems with density between 3 and 10 GPUs per host. Of course the 10 GPU system has significant 11000RPM fans blowing through the horizontal cooling v fins allowing the GPUs to be side by side without needing as much airflow. The 8GPU systems are liquid cooled via an external loop, so that is not a good comparison.

A better example is the system I built over the weekend. $250 Kaby Lake 51W CPU host with 1x PCIe risers and 7 GPUs spread out with ~ 2 slot spacing separate from the motherboard. The cards (RX580s) are around 65* at load with no additional fans in a 72* room. This of course does not work as well for GPU loads that need lots of PCIe bandwidth, however neither LL or TF requires this.

[airsquirrels]

Here is what the 8 Titan Blacks looked liked in the SuperMicro (expensive) GPU host:

[kladner]

Originally Posted by airsquirrels

Here is what the 8 Titan Blacks looked liked in the SuperMicro (expensive) GPU host:

Gulp.
What is the combined approximate LL throughput? What is the price range of the host board?

[ewmayer]

Originally Posted by kladner

Gulp.
What is the combined approximate LL throughput? What is the price range of the host board?

You probably don't wanna know. David's budget-priced new KabyLake + multi-GPUs system, OTOH, might within the reach of those of modest means, especially if populated with used GPUs of the kind David is also selling in a neighboring thread - though the stuff he sells tends to draw significantly more Watts/FLOP than the newer gear he replaces it with, so you def. need to factor in electricity cost when evaluating such a 'budget gear' option.