Best setup for around $3000 recommendations
 

Good evening gentlemen,

I got my year-end bonus, and I'm thinking of blowing part of it on a full-time GIMPS setup. I've been running PRP-WR on my gaming PC but I want to build a machine/machines that will be min-maxed for GIMPS. I've been focusing on large core count CPUs with fast RAM but I can't decide if it would be better to go all out on one machine or get two more modest machines. I'm looking to spend around $3k. Does anyone have any setups that worked well for them or any sage advice?

Thank you

[QUOTE=MarkVanCoutren;622308]Does anyone have any setups that worked well for them or any sage advice?[/QUOTE]

Rent from the clouds.

Seriously.

Unless you're heating your place with the compute, it's far cheaper.

[QUOTE=chalsall;622309]Rent from the clouds.
...
Unless you're heating your place with the compute, it's far cheaper.[/QUOTE]Depends upon the situation. In my case it is far cheaper to run my own gear.

[QUOTE=retina;622313]Depends upon the situation. In my case it is far cheaper to run my own gear.[/QUOTE]

Of course.

In your case, you support a secret lair.

Correct?

[QUOTE=MarkVanCoutren;622308]Good evening gentlemen,

I got my year-end bonus, and I'm thinking of blowing part of it on a full-time GIMPS setup. I've been running PRP-WR on my gaming PC but I want to build a machine/machines that will be min-maxed for GIMPS. I've been focusing on large core count CPUs with fast RAM but I can't decide if it would be better to go all out on one machine or get two more modest machines. I'm looking to spend around $3k. Does anyone have any setups that worked well for them or any sage advice?

Thank you[/QUOTE]

My observation is that you can most efficiently use 2 cores per 1 channel of memory as a rule of thumb. Beyond that are diminishing returns. For desktop class hardware with 2 memory channels, you'd be looking at a 4 core CPU. The one exception to that are the 3D chips from AMD. The 5800X3D can fit wavefront exponents entirely in its L3 cache.

I wouldn't run out and buy a 5800X3D though: the new 7800X3D will be out on February 14th. In addition to having 96 MB of L3 it also has AVX512 that will give a significant performance boost. The memory sweet spot for the 7000 series is DDR5-6000, using only one DIMM per channel (two DIMMs per channel often requires lowering speeds to 4800 or lower for stability).

You may also hear about the 7900X3D and 7950X3D, but only one of their chiplets has the additional L3. In theory you could run one worker on the chiplet with the extra cache and another worker on the other, which would end up using main memory bandwidth. This is entirely untested, of course, because anyone who has the chips is under NDA.

Personally I'd wait and build a pair of 7800X3D systems. All the 7000 series chips have a basic integrated GPU, so you don't need to buy one. The most cost effective GPUs for your task are used Radeon VII off eBay, running gpuOwl (which I have no experience running personally).

I would probably recommend going a low-power route. Most of today's cpus are clocked to really inefficient speeds. For example the 7900X(170W) clocks at base/boost 4.7/5.6 Ghz and the 7900(65W) clocks at 3.7/5.4 Ghz. The 7900 is far more efficient.
Another benefit of underclocking cores is that you can use more before saturating memory bandwidth. Given most cpus have more cores than memory bandwidth can handle, underclocking is a good way forward. Assuming they are memory bandwidth bound, the 7900 and 7900X may produce pretty similar performance for some workloads(such as WR-PRP tests) at very different power budgets.

Does anyone have any recent benchmark data that confirms that about 2 cores per DDR5 channel is what we can get away with for Zen 4?
It would also be interesting to know how much benefit having two CCDs(and double the L3 cache) helps performance per watt.

I would opt for a second-hand EPYC 7H12 chip. Water cool it. Give it 8 DIMMS. Okay, maybe not ideal for GIMPS.

PRP for world-record mersennes? Even the authors of prime95 and mlucas run gpuowl for performance and efficiency, on Radeon VII GPUs, several to a system. Consider an open frame like the coin miners use, but a more capable CPU and motherboard than they use; a KW+ high efficiency power supply (overspec it and operate it near its peak conversion efficiency, that is if you expect to actually use 1KW, get a 1600W)
Rough budget from memory, with adjustments for current guessed pricing:
$200 PSU 1600W output rated, gold or higher efficiency rating;
100 good frame that provides support both ends for the gpus; 8-GPU size is nice and roomy for 6 or less;
4 x $500 for Radeon VIIs, ideally all from the same brand name, with Hynix VRAM not Samsung;
6 x $10 for PCIe extenders (keep a spare or 2 at all times, they do fail)
total so far ~$2360; spend the other 640 on the following, and if there's any money and power budget left, one more GPU & extender:
$10 plugin surge suppressor;
at least 64MB of ram, to enable v30.8 efficient P-1 stage 2 or simultaneous P-1 on all GPUs;
motherboard with 6 PCIe slots
cpu; AVX512 is nice in that it lets one explore the full range of prime95/mprime, but AVX2 allows up to 920M exponent
OS Ubuntu, so you can use rocm driver for high performance and save money, or go Windows but that will cost a license fee.
Optionally, a roll of hardware cloth or other inexpensive EMI shielding material, from which with tin snips to craft an EMI shield box in halves. (Otherwise cell phone battery life and TV on air reception suffer.)
Assemble as if your plan is to install 6 GPUs. Spacing for air flow but occasionally mounting a replacement or spare can be handy.
Don't put much else on the same circuit breaker; a microwave oven or vacuum cleaner added & running will trip the breaker downing the system abruptly.

For CPU performance, regarding the performance of gpuowl, it is not critical, as it matters briefly, during validation of past work during resumption of runs in progress, during GCD in P-1, and during building proof files at the end of PRP runs. The other 99% of the time it's the GPU's performance that matters. I've used such a rig with only 16GB DDR3 ram, i7-4790 or G1840. CPU speed and installed ram will matter more if you plan to use Gpuowl to do P-1 stage 1 and transfer the P-1 to prime95 on the CPU to do the stage 2, as is offered in a recent update to gpuowl.
Run the GPUs at reduced power from nominal, for higher economic efficiency, and to keep from overheating the 15-amp wall socket & surge suppressor. I've seen the plastic discolor or melt on a surge suppressor or inline power monitor, and suppressor and monitor electronic failures, while the outlet still looked ok. (It was presumably from lightning hits conducted through the buried power cabling, although I can't rule out chronic running hot as a contributing factor. Bolts also have taken out unprotected appliances, including a washing machine, the occasional GPU PCB, and motherboard.) Do not run mfakto on the IGP. (I lost two motherboards and 2 GPUs trying that; the combined load of CPU, IGP, and PCIe draw produced sparks and flame on one location on the motherboard.)

Also keep in mind that household circuits in North America are limited to 80% or rated capacity for continuous duty, so a 120 volt 15 amp circuit is limited to 1440 watts, and some circuit breakers will trip even below this point after many hours or even days.

My repurposed mining rig is running 3 3070s on one power supply and 5 3070s on another. I cap them at 200 watts each, and unrestricted they will pull 250 running mfaktc, close to maxing out the one circuit.

So you may need a dedicated circuit (or two!). It's something to think about before building a monster machine.

I would only buy a platinum rated power supply if drawing that much as the additional cost will pay for itself over a year or two in electricity savings.

As an FYI for those reading: not all GPUs are good to use with all risers. My 3070s are fine with USB risers because they don't draw much from the PCIe slot. I believe the same with the Radeon VIIs. My 1070s draw too much from the PCIe slot to use with USB risers. Risers should be about $2 each after the mining crash.

If you're running a high amp machine, use a cord rated above it. Lets say I don't throttle my five 1070s on the one power supply, they would pull 1250 watts, plus maybe 50 for the motherboard/CPU. Call it 1300 watts. It's a platinum power supply running at 90% efficiency, so that's 1444 watts or 12 amps. That exceeds what a 16 AWG cord can handle continuously (10 amps), if the cord is even built to handle that much (some will get too hot at the plug at either end). I recently purchased a 14 AWG cord rated for 15 amps and it's no longer a concern. The plug at the end still heats up 10°C, drawing just 1150 watts though, measured by infrared thermometer!

[QUOTE=kriesel]PRP for world-record mersennes?....[/QUOTE]

TL;DR, but what I glanced at is overkill, IMO. He said $3,000. Something really good can be built for half of that. I always started with the CPU and build around it. In his case, a 16 core, 32 thread. Intel makes one, which probably means AMD does as well. Users' choice.

[QUOTE=Mark Rose]I recently purchased a 14 AWG cord rated for 15 amps and it's no longer a concern. The plug at the end still heats up 10°C, drawing just 1150 watts though, measured by infrared thermometer!
[/QUOTE]

I always found a warm or hot plug in a wall outlet concerning. In this situation, I would visit Lowes and get a length of 12 AWG and add a higher capacity plug and receptacle. I am sure you have the proper tools to put it all together. Just a thought.

If PRP is your goal, then by far the best solution IMO is used Radeon VIIs.

My highest throughput machine is an old Intel CPU that I bought to develop AVX-512 code. The mobo supports 4 GPUs. With a 1000W Platinum power supply, I have four Radeon VIIs with overclocked GPU memory and underclocked GPU compute. Each card draws ~150W at the wall. For 112M exponents, the four GPUs will finish 3.7 PRP tests each day.

I recently picked up two Radeon VIIs for about $325 each including taxes and shipping.

Energy costs are very significant. Platinum power supplies are a must. Running underclocked is a must. If you ever get the urge to run at normal clocks, compute the extra energy costs and instead put that to buying more underclocked GPUs. The payback period for me was just under 2 years.

[QUOTE=storm5510;622365]I always found a warm or hot plug in a wall outlet concerning. In this situation, I would visit Lowes and get a length of 12 AWG and add a higher capacity plug and receptacle. I am sure you have the proper tools to put it all together. Just a thought.[/QUOTE]

Indeed. I once had bad wiring on an electrical socket: some of the insulation was under the screw on the side of the plug, which didn't like having ~900 watts pulled through it for GIMPS. It partially melted the plug on my laptop charger and was damaging the insulation on the house wiring. I measured it at 72°C (162°F).

Before getting that 15 amp cord with 14 AWG wiring, I had a 16 AWG cord. The cord wasn't getting too warm, but the plug had reached 56° (133°F). The new cord's plug reaches about 30° (86°F) in a 20° (68°F) basement. I'm not worried about it as it's staying there steadily.

[QUOTE=Mark Rose;622379]Indeed. I once had bad wiring on an electrical socket: some of the insulation was under the screw on the side of the plug, which didn't like having ~900 watts pulled through it for GIMPS. It partially melted the plug on my laptop charger and was damaging the insulation on the house wiring. I measured it at 72°C (162°F).

Before getting that 15 amp cord with 14 AWG wiring, I had a 16 AWG cord. The cord wasn't getting too warm, but the plug had reached 56° (133°F). The new cord's plug reaches about 30° (86°F) in a 20° (68°F) basement. I'm not worried about it as it's staying there steadily.[/QUOTE]

I imagine you could smell it cooking.

I have several things here that get warm right where the wire goes into the back of the plug. The worst is a Bissell vacuum cleaner. It almost gets too warm to pull out of a wall receptacle by hand if I use it more than a few minutes. I would stay the wire inside the jacket is 16 AWG by looking at it. Too small for that application.

Some of the cords that ship with equipment or permanently attached to equipment are not adequate for their continuous operation. That's malpractice IMO.

Re the budget for the hardware suggestions: the OP set the budget.[QUOTE=MarkVanCoutren;622308]I'm looking to spend around $3k.[/QUOTE]

[QUOTE=storm5510;622365]Something really good can be built for half of that.[/QUOTE]That is true. And building two such half-price systems might give better throughput than a single full-price system. Plus there is some redundancy in case one is out of action.

Another possibility for the OP's stated budget is a used system with dual lotsa-core Xeons and half a TB of ECC ram, to do some serious P-1 factoring alongside substantial PRP. (Then scrape up some more funds later and put a couple radeon VIIs in it, for more-efficient PRP.)
As someone who started out on the lotsa cheap systems route, fewer more capable boxes has some appeal for controlling sysadmin overhead.

It's been a while since I built my i5 6600 systems, but I found them by far the cheapest way to get performance at the time (8.5 years ago, damn!):

I bought 4 x i5-6600 at $148, 4 x 32 GB at $120 ($480 for 8 DIMMs), 4 x motherboard at $40 each, and a single 650 watt power supply for $95.

Cost me $2020 after tax. I also bought three ATX splitter cables for US$63. When operating off the one power supply, turbo disabled, each system consumed about 67.5 watts at the wall with a 0.1 undervolt.

I did eventually buy GTX 1070s, more power supplies, and cases. Those machines are all doing P-1 currently.

I bought 8 channels of bandwidth for 16 cores. The challenge has always been memory bandwidth.

I should probably sell those systems while they still have some value. They're not very power efficient these days. It's nice to have a cluster to tinker with though.

[QUOTE=kriesel]Another possibility for the OP's stated budget is a used system with dual lotsa-core Xeons... [/QUOTE]

I would [U]not[/U] recommend a dual CPU setup until software openly supports it, i.e. [I]Prime95[/I] and others.

[QUOTE=Mark Rose]...I should probably sell those systems while they still have some value. They're not very power efficient these days. It's nice to have a cluster to tinker with though.[/QUOTE]

Everything I have is getting long-in-the-tooth. Two HP workstations from the early 2010's and the custom i7 I built in 2018. It is hard to believe nearly five years has passed on the newest one. Tinker is right, and I do it frequently.

[QUOTE=storm5510;622436]I would [U]not[/U] recommend a dual CPU setup until software openly supports it, i.e. [I]Prime95[/I] and others.
[/QUOTE]I've been running prime95 and mlucas on dual-xeon systems for years. It's easy. Single instance by default.
For a little more performance, one can run separate prime95 instances launched by Windows batch files specifying which NUMA node & cpu affinities each runs on. But that is not required, just a choice, optional. Probably the same applies on Linux.
Dual-xeon used former servers with 128GiB ECC ram can be found for ~$600. I have one of those churning through P-1 redo for exponents that got only stage1 or low stage 2 the first time.

What makes Radeon VII so good? is that the HBM2 memory?
Would not modern professional cards be better at the task? Ignoring cost of card.

It's the memory and good FP64 ratio. HBM and good DP ratio is now mostly only in datacentre cards, but if a modern pro card has HBM it's probably better than a Radeon VII. If they come with GDDR6/X then it's likely that they're pro versions of consumers cards which are good at rendering but not necessarily PRP, the main difference is validation and often there's twice as much memory capacity for models. Nvidia's consumer FP64 ratio is awful, and it looks like AMD's RDNA3 FP64 ratio is half as good as RDNA2 so that might be one to avoid, the 6950XT RDNA2 is comparable to the Radeon VII for PRP thanks to the extra cache making bandwidth less relevant.

[QUOTE=M344587487;631070]It's the memory and good FP64 ratio. HBM and good DP ratio is now mostly only in datacentre cards, but if a modern pro card has HBM it's probably better than a Radeon VII. If they come with GDDR6/X then it's likely that they're pro versions of consumers cards which are good at rendering but not necessarily PRP, the main difference is validation and often there's twice as much memory capacity for models. Nvidia's consumer FP64 ratio is awful, and it looks like AMD's RDNA3 FP64 ratio is half as good as RDNA2 so that might be one to avoid, the 6950XT RDNA2 is comparable to the Radeon VII for PRP thanks to the extra cache making bandwidth less relevant.[/QUOTE]

Thanks for the explanation!
How about professional accelerators like AMD Instinct? Link: [url]https://www.amd.com/en/products/server-accelerators/instinct-mi250x[/url]

[QUOTE=Jurzal;631094]
How about professional accelerators like AMD Instinct? Link: [url]https://www.amd.com/en/products/server-accelerators/instinct-mi250x[/url][/QUOTE]

Instinct MI250: those are extremely expensive, and not available for purchase (by an individual that is) anywhere.

I guess if you want to buy some multiple of 1000s of Instinct GPUs, you contact AMD sales directly and they tell you the price for the lot.

OTOH old-gen MI50 and MI60 may be found for reasonable prices. They are good GPUs, comparable to R7, with double the RAM (32GB), with ECC all-over (but ECC brings no benefit for PRP). But they have no cooling, so you'd need to improvise some cooling solution up.

[QUOTE=preda;631103]Instinct MI250: those are extremely expensive, and not available for purchase (by an individual that is) anywhere.

I guess if you want to buy some multiple of 1000s of Instinct GPUs, you contact AMD sales directly and they tell you the price for the lot.

OTOH old-gen MI50 and MI60 may be found for reasonable prices. They are good GPUs, comparable to R7, with double the RAM (32GB), with ECC all-over (but ECC brings no benefit for PRP). But they have no cooling, so you'd need to improvise some cooling solution up.[/QUOTE]

What about Radeon VII Pro vs Instinct MI50? With some home depot cooling :D
In screenshots, current retail place in Latvia e-comms.

R7 Pro has 6.5 TFLOPs FP64
MI50 has 6.7 TFLOPs FP64
R7 (nonpro) has 3.4 TFLOPs FP64

3D printed fan tunnel attachment

[QUOTE=Jurzal;631125]What about Radeon VII Pro vs Instinct MI50? With some home depot cooling :D
In screenshots, current retail place in Latvia e-comms.

R7 Pro has 6.5 TFLOPs FP64
MI50 has 6.7 TFLOPs FP64
R7 (nonpro) has 3.4 TFLOPs FP64[/QUOTE]
Per Preda a while back, even the Radeon VII at 3.46 TFLOPs FP64 & 1TBps HBM2 ram bandwidth is thought to be memory bandwidth limited in Gpuowl.
A reason to go to MI60 is greater total ram, for P-1 stage 2 on large exponents with Gpuowl.
MI60 requires Ubuntu or SLES (no Windows support); and either a server case that pushes air through the passively cooled GPU lengthwise, or a blower & adapter. Plus not all motherboards will work with an MI60:
“*PLEASE NOTE* This unit does NOT work with windows. It only works with Linux. This unit does not work with any Radeon drivers and only works with ROCm. For multiple cards to work in one system, this card typically needs a server motherboard. In our testing with a server motherboard, we are able to connect up to 4 of these MI60 cards at once, but not more than 4. We were able to connect additional cards that were not MI60 in addition to the 4 units. For example, we were able to have 4 units of MI60 and 4 units of RTX 3060 working simultaneously with a server motherboard. We have also had good luck with the Gigabyte Z390 UD motherboard working with 4 units. In order to get to 80MH on this card you must use teamredminer in C state.” [url]https://www.ebay.com/itm/125006475381[/url] & MI60 apparently requires PCIe v3 or 4. Radeon VII can run on PCIE V2 or 1.1 extenders.
[url]https://www.amd.com/en/support/server-accelerators/amd-instinct/amd-instinct-mi-series/instinct-mi60[/url]

[QUOTE=MarkVanCoutren;622308]Good evening gentlemen,

I got my year-end bonus, and I'm thinking of blowing part of it on a full-time GIMPS setup. I've been running PRP-WR on my gaming PC but I want to build a machine/machines that will be min-maxed for GIMPS. I've been focusing on large core count CPUs with fast RAM but I can't decide if it would be better to go all out on one machine or get two more modest machines. I'm looking to spend around $3k. Does anyone have any setups that worked well for them or any sage advice?

Thank you[/QUOTE]

For 3 grand you easily set up a 6 GPU rig and have change to run it. With 6 Radeon VII each drawing no more than 250w you'd need a couple of beefy PSUs. The rest of the system is chicken feed. In fact the GPU cards can be adjusted to draw 200w each -- so a couple of 1200w PSUs is recommended so that you don't fry things. Maybe two simple motherboards too with adequate PCI-E slots. The downside is that each GPU has 3 cooling fans. So the noise will be a problem -- you need a separate room for the rig, and adequate room cooling during the summer.