Running mfaktc on multiple GPUs tripped circuit breaker?

[ixfd64]

I recently got access to some spare GPU servers. They typically have four or eight GPUs, either GeForce GTX 1080's or Quadro P5000's. I was able to run mfaktc on a four-GPU machine without issues. But when I launched mfaktc on an eight-GPU server connected to the same power strip, the circuit breaker promptly tripped.

All of the servers have a single 2000 watt PSU installed. The power strip in question is connected to four GPU servers and a few quad-core machines. Each outlet is rated to handle 10 amps.

So my questions are:

1. Anyone know how much current a GeForce GTX 1080 or Quadro P5000 draws?
2. According to the specs on the manufacturer's website, the eight-GPU servers use 2+2 redundant power supplies. Would adding a second PSU solve the problem?

[PhilF]

Each outlet may be able to handle 10 amps, but the entire strip is rated for only 15 or 20 amps, as is the outlet it is connected to. Your strip is quite overloaded. You will need to separate your power hungry servers onto a couple of different circuits.

[chalsall]

Originally Posted by PhilF

Each outlet may be able to handle 10 amps, but the entire strip is rated for only 15 or 20 amps, as is the outlet it is connected to. Your strip is quite overloaded. You will need to separate your power hungry servers onto a couple of different circuits.

Further to this...

When you get to this level of power consumption, it is well advised to think all the way back to the panel, and then to the provider's demarcation point.

If you have access to more than one phase, it is advised to try to balance your loads across them.

[ixfd64]

I took a closer look and found out the breaker is rated at 15 amps. It's definitely not enough for all 12 P5000's, let alone the other systems it's connected to. Talk about a first world problem.

[nomead]

Originally Posted by ixfd64

All of the servers have a single 2000 watt PSU installed. The power strip in question is connected to four GPU servers and a few quad-core machines. Each outlet is rated to handle 10 amps.

So my questions are:

1. Anyone know how much current a GeForce GTX 1080 or Quadro P5000 draws?
2. According to the specs on the manufacture's website, the eight-GPU servers use 2+2 redundant power supplies. Would adding a second PSU solve the problem?

1. They're both rated at 180W TDP and that's what they probably take, too, if you don't adjust any clocks or power limits manually. So if it's a 120 VAC power strip, that's 12 Amps already? And then you need to realise, that the power supplies aren't 100% efficient, and neither is the VRM on those cards. Plus the CPU and any peripherals on the machine. So if there's a 10A circuit breaker, no wonder it tripped.
2. Maybe, just maybe, if the power supplies balance the load across themselves, but then you still need to think about what's feeding that power strip and what's the limit there.

Disclaimer, I really don't know that much about the electrics in USA. I'm used to 230VAC and that is a lot more forgiving in these circumstances.

[kriesel]

Originally Posted by ixfd64

I recently got access to some spare GPU servers. These servers typically have four or eight GPUs, either GeForce GTX 1080's or Quadro P5000's. I was able to run mfaktc on a four-GPU machine without issues. But when I launched mfaktc on an eight-GPU server connected to the same power strip, the circuit breaker tripped seconds later.

All of the servers have a single 2000 watt PSU installed. The power strip in question is connected to four GPU servers and a few quad-core machines. Each outlet is rated to handle 10 amps.

So my questions are:

1. Anyone know how much current a GeForce GTX 1080 or Quadro P5000 draws?
2. According to the specs on the manufacture's website, the eight-GPU servers use 2+2 redundant power supplies. Would adding a second PSU solve the problem?

Sounds like you have access to use systems in a server rack, that someone other than you owns. It's not clear what voltage your input is, or the rating of the upstream breaker. But I'll speak from my experience running my own equipment in a residence. Some of the same considerations will apply to server racks, or UPS sizing if considering that.

A GTX1080 or P5000 is nominally 180 watts TDP, but that's DC power at the card, and varies with clock rate and work type. Currently my GTX1080 is pulling 150 watts running CUDAPm1 and GPU-Z indicates it's at 62.5% of TDP, so that would be 240 watts at 100%. (The system it's in is a Lenovo D30 with nominal 1120Watt power supply IIRC; attempting to run mfaktc on it draws full rated power or more on the gpu and crashes the system. Power supplies apparently decline in output capability with age. I've seen that on many systems in the past 2 years.) Multiply by number of gpus. Pad that number to include the supporting system's requirements. Divide by the power supply's efficiency as a fraction. Leave a safety margin to avoid breaker tripping, account for line voltage drop in the building wiring and power cords, etc.

Remember that it does you no good if each outlet on the power strip is rated for 10 Amps, but the sum of however many outlets it has is still limited to the standard US 120VAC nominal wall socket 's rating of 15 amps.
It's common to see up to 125VAC in an unloaded wall socket in my part of the US. Loading a home circuit nearly to capacity can pull that wall socket voltage down by volts. (I remember as a youngster seeing old equipment or light bulbs labeled 110 and even 108.)
I've gone to some trouble to scatter my systems so that they are drawing from different circuits, therefore different breakers.

Simplified example (don't hold me to most of these numbers, some are from memory or guesses)
GTX1080 180 W
Q2000 50W
2 HD 12W
2 135W CPU packages 270W
ram & misc 120W
subtotal 632W
/0.8 power efficiency = 790W; at 125V that would be 6.32 amps at 100% power factor
(amps exactly in phase with volts). Amps get worse as power factor declines, or as line resistance drops the at-the-plug voltage.
Actual power at the power strip is a bit more than the calculation because there's a bit of loss in the cord. Plug in a KillAWatt or similar and the system into that to see actual usage, & voltage, amperage, power factor. Some libraries have them for loan, or the usual online retailers are your friend for a small fee. They come in US and other nationality compatible plug and socket configurations. The utility bill will be higher than that reflects because voltage drop in the wiring to the meter is purchased power too.

UPSes rated up to ~900-1000 watts & 1500VA are not too awfully expensive, but above that it gets expensive quickly.

Re multiple power supplies in a single system, these are more commonly connected to different sources for failover. If one circuit trips or one phase drops from the utility supply, another may carry the system in that case.
Putting redundant supplies on one system on the same source circuit would increase base load on that circuit; the supply draws more than zero when idle.

https://www.nvidia.com/en-us/geforce...orce-gtx-1080/

https://www.techpowerup.com/gpu-spec...ro-p5000.c2864

For the quantity of gear you mention access to, a careful power diagram and measurements are advisable.


Are the PSU's 240V units? a 2000 W output 120V input PSU even at excellent 97% efficiency and unity power factor would be 2000/120/0.97 =~17.2 amps so had better be a 20-amp receptacle and an input cord designed for it.

[kriesel]

What's a "spare gpu server"? :)

[chalsall]

Originally Posted by ixfd64

I took a closer look and found out the breaker is rated at 15 amps. It's definitely not enough for all 12 P5000's, let alone the other systems it's connected to. Talk about a first world problem.

Hey, not just "first world"; we catch-ups (AKA "the developing nations" et al) are in the same boat. I prefer to refer to such instances as "contemporary wanderers"...

What isn't taught in grade school is simple electrical deployment. Happy to be corrected on this, but I have yet to speak with a University graduate who could wire a new outlet to a new circuit breaker inside the distribution panel.

It's just math. You want to feed a stack something like 40 A of power (by way of electricity).

No problem.

1. What voltages are you talking about?

1.1. 110V? 220V?

1.2. At what frequencies will your kit be happy? (60Hz isn't universal...)

2. Do you already have the "Layer 0" conduit runs from your electrical panel to your stack?

3. Do you already have planned the need to get rid of all that compute by-product (read: heat).

It's just math.

[kriesel]

A lot of gear is either manually switchable 120/240 (usually a little slide switch at the UL or whatever certification compliant connector and protection on chassis, or tolerates a range 100-240V. Frequency typ 50-60 or 47 - 63 Hz.

Absolutely, check all this stuff, before applying electricity, as chalsall implies.

[petrw1]

I run 4 Quad CPUs; 1 with a GTX-980 GPU split across 2 power bars but both in the same wall-outlet.

Then I have a Dual Core LapTop in another outlet but all these will go the same breaker.

I think I must be close to capacity but no breakers tripped yet in 5 years.

[chalsall]

Originally Posted by petrw1

I think I must be close to capacity but no breakers tripped yet in 5 years.

If you want to know exactly how much capacity you have, you can open up your panel and read.

An alternative is to pay an electrician to do it for you.

They have the kit. You don't.

You want a question answered. They can answer it.

Not quite a "Prisoners dilemma, but reasonably close...