20 Graphics cards per pc

[mackerel]

The use case for these is that most cards will be mounted off board, connected using PCIe riser cables.

There are some driver/OS limits on number of GPUs supported so that would be the next problem.

Originally Posted by mackerel

The use case for these is that most cards will be mounted off board, connected using PCIe riser cables.

There are some driver/OS limits on number of GPUs supported so that would be the next problem.

On Debian with amdgpu-pro there are no limits, but you would have to install also the nvidia-driver to mix GPUs on the same main board.

Originally Posted by SELROC

On Debian with amdgpu-pro there are no limits, but you would have to install also the nvidia-driver to mix GPUs on the same main board.

I have to retract on this one. Mixing drivers will certainly cause more problems. I have not attempted that myself yet as I don't own a CUDA GPU.


However, the amdgpu driver is still having problems when I want to use more that 4 GPUs together. It shows a persisting error in this case.


I have been able to avoid the error by using only 4 GPUs at the same time.


Is this a driver limit? I don't know.

[niconiconi]

Originally Posted by SELROC

I have to retract on this one. Mixing drivers will certainly cause more problems.

The problem of mixing drivers can be overcame by using virtualization technology.

IOMMU, also known as VT-d on Intel processor, is a hardware mechanism to map and isolate PCI-e in its own address space. This can improve security by stopping malicious hardware from access the main memory directly, but it is mainly used for accessing hardware directly from a virtual machine.

The setup is simple - you enable IOMMU on Linux host, and run two Linux KVM guests with PCI passthrough, on separate virtual machines, one for Nvidia, another one for AMD. Then the drivers and GPUs can work independently without interference. The overhead is negligible, because most work is done on GPUs, even computation on CPU is nearly as-fast-as native, since virtualization itself is accelerated by VT-x.

Here is a guide, https://bluehatrecord.wordpress.com/2015/12/05/performing-iommu-based-gpu-pci-passthrough-with-fedora-22/

The only problem is that, VT-d requires CPU support. Most Core i5 and Core i7 CPU does support it, but this feature is removed on CPUs ending with "K". Probably it is an intentional plan by Intel, to prevent cost-effective enthusiasts CPUs to be used in production. But no problem if you use Xeon.

Originally Posted by mackerel

The use case for these is that most cards will be mounted off board, connected using PCIe riser cables.

There are some driver/OS limits on number of GPUs supported so that would be the next problem.

And to mount 20 GPUs you need 3 power supplies on the mainboard. Problem is switching PSUs are not made to run in parallel...


we tried and got the beginnings of a fire outbreak.

[niconiconi]

Originally Posted by SELROC

Problem is switching PSUs are not made to run in parallel...

Connecting +12V/Vcc together is surely problematic for most power supplies in general, but it shouldn't be a problem if you power each row of GPUs independently (don't interconnect +12V in between different supplies, e.g. each row of six GPUs to be powered by +12V from a single 1200W power supply). For those PCI-e riser card extender with Molex power connectors and USB 3.0 cables to carry PCI-e signal, the power is effectively isolated, and only the GND on PCI-e signal circuitry is interconnected. See https://nerdralph.blogspot.com/2017/...ning-rigs.html All don't forget to bridge a few control wires (e.g. Power-Good) from three power supplies together with two dual PSU adapters, to ensure they starts properly at the same time.

Originally Posted by niconiconi

Connecting +12V/Vcc together is surely problematic for most power supplies in general, but it shouldn't be a problem if you power each row of GPUs independently (don't interconnect +12V in between different supplies, e.g. each row of six GPUs to be powered by +12V from a single 1200W power supply). For those PCI-e riser card extender with Molex power connectors and USB 3.0 cables to carry PCI-e signal, the power is effectively isolated, and only the GND on PCI-e signal circuitry is interconnected. See https://nerdralph.blogspot.com/2017/...ning-rigs.html All don't forget to bridge a few control wires (e.g. Power-Good) from three power supplies together with two dual PSU adapters, to ensure they starts properly at the same time.

The mainboard has 3 PSU connectors and it is supposed to power 18 GPU with 3 1200W PSU. The PSU are not IBM Dell or HP but absolutely CoolerMaster with 94% efficiency.


A total of 3600W will flow in the mainboard.


Personal advice: to do cryptomining you only run 1 program which exploits all GPUs, but with Mersenne you need to run 1 program for each GPU. It is difficult to manage and you need to develop scripts to start/stop the programs.

[niconiconi]

Originally Posted by SELROC

The mainboard has 3 PSU connectors.

I didn't realize the mainboard has all the PSU connectors. Instead of being powered externally, all the power flows into the board?! Now I'm suspicious about the stability of this setup, 3300 W is even beyond the rating of a standard 230 VAC outlet/connector. I think it's better to rather stay around six cards, which is a time-proven setup used by many in for GPU computing...

Originally Posted by SELROC

Personal advice: to do cryptomining you only run 1 program which exploits all GPUs, but with Mersenne you need to run 1 program for each GPU. It is difficult to manage and you need to develop scripts to start/stop the programs.

Good point. It shouldn't be too difficult though. If anyone wants to mix AMD and Nvidia GPU, use KVM virtual machines w/ PCI passthough, and control them from host via virtual network interface with a script should work.

Nevertheless, just my two cents... I've only played with some BOINC projects, weak SHA bruteforcing analysis, GPU video encoding, and only became interested in factoring some numbers recently - all of these are just nearly fire-and-forgot tasks without needs for scheduling. I have absolutely no authority on GIMPS, nor large GPU clusters.

Originally Posted by niconiconi

I didn't realize the mainboard has all the PSU connectors. Instead of being powered externally, all the power flows into the board?! Now I'm suspicious about the stability of this setup, 3300 W is even beyond the rating of a standard 230 VAC outlet/connector. I think it's better to rather stay around six cards, which is a time-proven setup used by many in for GPU computing...



Good point. It shouldn't be too difficult though. If anyone wants to mix AMD and Nvidia GPU, use KVM virtual machines w/ PCI passthough, and control them from host via virtual network interface with a script should work.

Nevertheless, just my two cents... I've only played with some BOINC projects, weak SHA bruteforcing analysis, GPU video encoding, and only became interested in factoring some numbers recently - all of these are just nearly fire-and-forgot tasks without needs for scheduling. I have absolutely no authority on GIMPS, nor large GPU clusters.

You are right to be suspicious, that system *has* stability problems.


Cryptomining is done with a proprietary system, I think they use a proprietary driver (not AMD) and that driver supports 18 gpus. The AMD driver shows errors with more than 4 gus.


After all, that makes sense, gaming mainboards come with maximum 4 Gen3 slots.

News from AMD: https://www.phoronix.com/scan.php?pa...m-Polaris-Soon

[mackerel]

Originally Posted by SELROC

The mainboard has 3 PSU connectors and it is supposed to power 18 GPU with 3 1200W PSU. The PSU are not IBM Dell or HP but absolutely CoolerMaster with 94% efficiency.

A total of 3600W will flow in the mainboard.

It doesn't work that way. The bulk of the power will be provided directly from the PSU to the GPUs via PCIe 6 or 8 pin power connectors. A small amount of power may be provided via the PCIe slot, but that looks like it can be taken care of either by the 4 pin molex connectors on the mobo, or risers boards often have their own separate power input.

The ATX 24-pin PSU connectors, likely one is the main one that actually powers the rest of the mobo, and the other two are provided to send a switch signal to the other two PSUs. CPU is powered by the EPS connector. In modern systems not a lot of power goes through the ATX power connectors.