Bargain Hardware Thread

[LaurV]

Originally Posted by Batalov

You can play the gift of the magi game with Brain! You could send him the system and he will send you the GPU. ;-)

Or you both send me the system and the GPU and I make them run together

[0PolarBearsHere]

Originally Posted by LaurV

Or you both send me the system and the GPU and I make them run together

I might be happy to take a fully populated one of these off someone's hands (for free).
http://www.rave.com/product/rr-2411-...supercomputer/

Not really bargain hardware though :P

[kladner]

Originally Posted by 0PolarBearsHere

I might be happy to take a fully populated one of these off someone's hands (for free).
http://www.rave.com/product/rr-2411-...supercomputer/

Not really bargain hardware though :P

*The ambient operating temperature must not exceed 25°C/77°F when six K80 GPUs are populated.



Ya reckon it would be a bit loud?

[lavalamp]

300 W TDP per card, add in another 200 W for the rest of the system too, probably looking at a maximum of 2 kW power use in a 2U rack. Still 17.5 DP TFLOPs, can't argue with that.

Just the power bill alone for a 48U rack full of these would be eye watering.

[kladner]

Originally Posted by lavalamp

300 W TDP per card, add in another 200 W for the rest of the system too, probably looking at a maximum of 2 kW power use in a 2U rack. Still 17.5 DP TFLOPs, can't argue with that.

Just the power bill alone for a 48U rack full of these would be eye watering.

Not to mention the A/C cost. Best be in a cold place with cheap power. Without monstrous cooling those cards would throttle.

[Madpoo]

Originally Posted by lavalamp

300 W TDP per card, add in another 200 W for the rest of the system too, probably looking at a maximum of 2 kW power use in a 2U rack. Still 17.5 DP TFLOPs, can't argue with that.

Just the power bill alone for a 48U rack full of these would be eye watering.

Assuming you could populate a 48U rack full of these, and going with 2KW per server, that's 48 KW in a single cabinet. That's a lot. You'd be talking a full load of 4 3-phase power strips (60A x 208V) per cabinet, and even then you'd really be going max capacity...probably need to shoehorn in a 5th strip. Datacenters will generally call a circuit saturated when you're at the 0.8 * W level.

Many datacenters I've dealt with won't even allow a power density like that in the first place just for cooling reasons alone. They typically list the BTUs per square foot they'll handle and 48KW in the footprint of a single cabinet is pushing it for all but the most efficient locations. Those would be the datacenters with hot and cold aisles separated by plastic walls and doors to make sure the cooling is directed exactly where it needs to be, and they'd actually make use of the top-mounted fans on the cabinet.

The location where Primenet itself is hosted is kind of like that... I don't know what their power density is exactly, but since they rent cabinet space per rack-unit, they want to optimize how many can fit in a single cabinet. The cages are lined to keep the hot/cold aisles separate, blank panels on all empty U, etc.

EDIT: After looking at some actual estimates from Raritan on their 3-phase PDUs, looks like a 60A 3-phase strip could handle 17.3 KVA, not the 12.5 KVA I foolishly assumed, because we're talking 3-phase here. So 4 of those actually would get you going, provided you had the cooling capacity to handle a beast like that.

[retina]

Originally Posted by Madpoo

You'd be talking a full load of 4 3-phase power strips (60A x 208V) ...

So many assumptions. More than a few^[1] places in the world use 415V 3Φ.

^[1] By "more than a few", I mean most.

[chalsall]

Originally Posted by retina

So many assumptions. More than a few^[1] places in the world use 415V 3Φ.

^[1] By "more than a few", I mean most.

Care to list them for distribution rather than transmission?

[LaurV]

Originally Posted by Madpoo

Assuming you could populate a 48U rack full of these...

Very good and informative post.

Well, I would be happy with only one, that one with 6 cards.
Anyhow, to stay in the "power" side of the discussion, they ARE passive cooled, but the case has 12 fans in the front, I think you can install thick fans on two rows, which is like 24 fans, high speed and high noise! And not low power, they consume about 10-15W each... So, beside of cards/cpus, you add the fans, divide by 0.8 (the efficiency of PSU) then multiply by two (redundant power supply, eventually hot-swap)...

I am still wondering how they solve all those problems. I would really like to work in a company that design or produce those toys...

[LaurV]

Originally Posted by chalsall

Care to list them for distribution rather than transmission?

What's the difference? You take an insulated knife, cut the transmission line and connect the hot side to your computer directly, for distribution. And don't tell anybody..

Disclaimer: don't do that at home!

[Mark Rose]

Originally Posted by LaurV

So, beside of cards/cpus, you add the fans, divide by 0.8 (the efficiency of PSU) then multiply by two (redundant power supply, eventually hot-swap)...

If I were planning to run a 2 kW system, I'd spec a more efficient power supply than that, like this one that's 90%+ efficient, and save 277 watts over an 80% efficient unit.