Re. my impulse purchase of 3 deal-priced R7s yesterday, just got "Order Canceled" e-mail from Amazon:
[quote]Order Details
Order #114-4034429-1352253
Placed on Saturday, May 23, 2020

3 x XFX AMD Radeon VII 16GB HBM2, 1750 MHz Boost, 1801 MHz Peak, 3xDP 1xHDMI Pci-E 3.0

Sold by GProME Direct Storefront

Cancel Reason: Technical Problem [/quote]
I'm guessing "Technical Problem" translates to "we just realized we could've sold them for more". Apologies to those who PMed me about purchasing one.

Radeon VII $549 & free shipping at Best Buy came up today;

[URL]https://www.bestbuy.com/site/xfx-amd-radeon-vii-16gb-hbm2-pci-express-3-0-graphics-card/6330849.p?skuId=6330849[/URL]

(Briefly; showed sold out minutes after I saw it.)

Another rough cut:
Following assumes Pro sells near list at least early, while Radeon VIIs are routinely discounted when available at all. Sales tax and shipping neglected, as they will vary. Time value of money ignored in these low interest rate times. Utility rate implied is $/watt-year; power level is rated consumption, which is likely very pessimistic. Heating benefit/ cooling cost omitted.

Radeon Pro VII
$1849
3x$250 =750
2599
/6.5/3 = $400/3 = $133/DPTflops-year

Radeon VII
$549
3x$300=900
$1449
/3.5/3 = $457/3 = $138/DPTflops-year

A Radeon Pro VII might make a good candidate for running PRP or LL/jacobi on a suspected new prime discovery in record confirmation time.
Or speeding the work of our dedicated software developers during alpha test.

As to where the 6.5 vs 3.5 comes from, I'm guessing they doubled the DP/SP ratio in the Pro design, and moved gpu clock rates down a bit to control wattage.

[QUOTE=kriesel;546445]Radeon VII
$549
3x$300=900
$1449
/3.5/3 = $457/3 = $138/DPTflops-year[/QUOTE]

Should that be $152/DPTflops-year ?

[QUOTE=ewmayer;546453]Should that be $152/DPTflops-year ?[/QUOTE]
1449/3.5=414; /3 = $138/DPTFlops-year.
Thereby demonstrating the waterbed corollary to Murphy's law; push back on errors here and there, and they'll pop up elsewhere..

[QUOTE=preda;546071]I think something like this is what you want:

[url]https://www.amazon.com/6-Pack-Graphics-Extension-Powered-Adapter/dp/B075FR7Q5Q/ref=sr_1_2?dchild=1&keywords=pci-e+mining+rig&qid=1590035671&sr=8-2[/url]

(I'm not endorsing this particular instance, you should shop around, this is just an example)

You get 6x for $30, and the USB cable is easy to route. The downside is, you need to power this adapter (more power cords). Also be sure to set the PCIe to Gen1 on the motherboard, as these extendenders don't handle Gen3 usually.[/QUOTE]

GPU #3 finally installed in its floating-above-the-CPU location, after ~6 hrs of wrestling with custom-Alu.-bracket-making, spread over 3 days (we had major heat wave over those same days, set A/C to a warmish 77F to keep the power bills reasonable, which is uncomfortably warm for hacksawing and other physical labor). By way of seeing what the minimal BIOS/OS setup needed is, I booted without changing the BIOS settings as recommended by Mihai above ... ROCm recognized the card no problem, this is after running the 2-GPU setup script and firing up the usual 2 gpuOwl jobs on each of GPU 1 and 2 (devices 0 and 1 using the ROCm indexing convention):
[code]GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU%
0 76.0c 166.0W 1373Mhz 1201Mhz 73.73% manual 250.0W 4% 100%
1 77.0c 223.0W 1547Mhz 1201Mhz 64.71% manual 250.0W 4% 100%
2 37.0c 19.0W 1373Mhz 351Mhz 55.69% manual 250.0W 0% 0%[/code]
In my previous brief fling with the 3rd card hanging horizontally via a pair of PICx16 adapters (right-angle followed by 3/4"-straight one) off the far left end of the mobo, I had set up 2 more run directories for the jobs on that card, inited the worktodo files and verified that everything worked - those are the 18 May checkpoint data in the logfile excerpts below. With the new physical setup I again set sclk=3 and fan=120 on that card (device 2 above) and restarted both jobs ... they fired up OK, the wall wattage spiked from ~580W to 750W, more than I intend to draw long-term (i.e. will set device 1 sclk to 3 as well, which will cut 50-60W) but with this new 850W PSU I figured for a few hours it should be OK. I kept a close watch on the just-loaded card's temperature, it looked fine, this card gets its intake air from the direction of the CPU below, which is [a] not generating significant amounts of heat, and [b] blowing its own output warm air out in a sideways direction, not in the direction of any of the GPU intake fan arrays.

But both jobs on the 3rd card quit shortly thereafter, here is the logfile for one of them, showing the earlier short-length (2M iter) run and today's attempt, which hits repeatable errors at the next checkpoint:
[code]2020-05-18 14:58:35 Note: not found 'config.txt'
2020-05-18 14:58:35 config: -d 2
2020-05-18 14:58:35 device 2, unique id '412688e172fd62d9'
2020-05-18 14:58:35 412688e172fd62d9 105843821 FFT: 5.50M 1K:11:256 (18.35 bpw)
2020-05-18 14:58:35 412688e172fd62d9 Expected maximum carry32: 59EF0000
2020-05-18 14:58:36 412688e172fd62d9 OpenCL args "-DEXP=105843821u -DWIDTH=1024u -DSMALL_HEIGHT=256u -DMIDDLE=11u -DWEIGHT_STEP=0x1.90ebe27014974p+0 -DIWEIGHT_STEP=0x1.46ed499a2046ap-1 -DWEIGHT_BIGSTEP=0x1.ae89f995ad3adp+0 -DIWEIGHT_BIGSTEP=0x1.306fe0a31b715p-1 -DPM1=0 -DAMDGPU=1 -DMM_CHAIN=1u -DMM2_CHAIN=1u -cl-fast-relaxed-math -cl-std=CL2.0 "
2020-05-18 14:58:40 412688e172fd62d9 OpenCL compilation in 3.48 s
2020-05-18 14:58:41 412688e172fd62d9 105843821 OK 0 loaded: blockSize 400, 0000000000000003
2020-05-18 14:58:42 412688e172fd62d9 105843821 OK 800 0.00%; 765 us/it; ETA 0d 22:30; d03832a517c973f8 (check 0.47s)
2020-05-18 15:03:18 412688e172fd62d9 105843821 OK 200000 0.19%; 1384 us/it; ETA 1d 16:37; 4cc8130dfd025b4d (check 0.73s)
2020-05-18 15:08:04 412688e172fd62d9 105843821 OK 400000 0.38%; 1425 us/it; ETA 1d 17:44; 86796fbdbfc139e1 (check 0.78s)
2020-05-18 15:12:50 412688e172fd62d9 105843821 OK 600000 0.57%; 1424 us/it; ETA 1d 17:38; c57cf276f8c6fa21 (check 0.77s)
2020-05-18 15:17:39 412688e172fd62d9 105843821 OK 800000 0.76%; 1445 us/it; ETA 1d 18:09; e50913ddfa7399ab (check 0.78s)
2020-05-18 15:22:29 412688e172fd62d9 105843821 OK 1000000 0.94%; 1442 us/it; ETA 1d 18:00; bc2a65f4bda7858a (check 0.81s)
2020-05-18 15:27:20 412688e172fd62d9 105843821 OK 1200000 1.13%; 1455 us/it; ETA 1d 18:17; 0c21f064e2d47580 (check 0.78s)
2020-05-18 15:32:10 412688e172fd62d9 105843821 OK 1400000 1.32%; 1446 us/it; ETA 1d 17:57; 04e192a123851613 (check 0.78s)
2020-05-18 15:37:00 412688e172fd62d9 105843821 OK 1600000 1.51%; 1443 us/it; ETA 1d 17:48; 230bcfa6ea972d80 (check 0.78s)
2020-05-18 15:41:51 412688e172fd62d9 105843821 OK 1800000 1.70%; 1451 us/it; ETA 1d 17:57; 2e0da3b04cab4dd9 (check 0.79s)
2020-05-18 15:46:42 412688e172fd62d9 105843821 OK 2000000 1.89%; 1452 us/it; ETA 1d 17:53; 949042a867be519b (check 0.80s)
2020-05-28 18:03:07 Note: not found 'config.txt'
2020-05-28 18:03:07 config: -d 2
2020-05-28 18:03:07 device 2, unique id '412688e172fd62d9'
2020-05-28 18:03:07 412688e172fd62d9 105843821 FFT: 5.50M 1K:11:256 (18.35 bpw)
2020-05-28 18:03:07 412688e172fd62d9 Expected maximum carry32: 59EF0000
2020-05-28 18:03:07 412688e172fd62d9 OpenCL args "-DEXP=105843821u -DWIDTH=1024u -DSMALL_HEIGHT=256u -DMIDDLE=11u -DWEIGHT_STEP=0x1.90ebe27014974p+0 -DIWEIGHT_STEP=0x1.46ed499a2046ap-1 -DWEIGHT_BIGSTEP=0x1.ae89f995ad3adp+0 -DIWEIGHT_BIGSTEP=0x1.306fe0a31b715p-1 -DPM1=0 -DAMDGPU=1 -DMM_CHAIN=1u -DMM2_CHAIN=1u -cl-fast-relaxed-math -cl-std=CL2.0 "
2020-05-28 18:03:11 412688e172fd62d9 OpenCL compilation in 3.44 s
2020-05-28 18:03:12 412688e172fd62d9 105843821 OK 2000000 loaded: blockSize 400, 949042a867be519b
2020-05-28 18:03:13 412688e172fd62d9 105843821 OK 2000800 1.89%; 731 us/it; ETA 0d 21:05; 5e3d8b9f5d802065 (check 0.51s)
2020-05-28 18:07:21 412688e172fd62d9 105843821 EE 2200000 2.08%; 1241 us/it; ETA 1d 11:44; d248b1bfc559baf4 (check 0.71s)
2020-05-28 18:07:22 412688e172fd62d9 105843821 OK 2000800 loaded: blockSize 400, 5e3d8b9f5d802065
2020-05-28 18:08:57 412688e172fd62d9 105843821 EE 2100000 1.98%; 956 us/it; ETA 1d 03:34; e8dbe286cbaaf37e (check 0.47s) 1 errors
2020-05-28 18:08:58 412688e172fd62d9 105843821 EE 2000800 loaded: blockSize 400, 9285fb72d902fcb3 (expected 5e3d8b9f5d802065)
2020-05-28 18:08:58 412688e172fd62d9 Exiting because "error on load"
2020-05-28 18:08:58 412688e172fd62d9 Bye[/code]
The 2nd run differs only hitting the "error on load" immediately on the first 'EE' retry-from-last-savefile:
[code]2020-05-18 14:58:47 Note: not found 'config.txt'
2020-05-18 14:58:47 config: -d 2
2020-05-18 14:58:47 device 2, unique id '412688e172fd62d9'
2020-05-18 14:58:47 412688e172fd62d9 105892693 FFT: 5.50M 1K:11:256 (18.36 bpw)
2020-05-18 14:58:47 412688e172fd62d9 Expected maximum carry32: 5A770000
2020-05-18 14:58:48 412688e172fd62d9 OpenCL args "-DEXP=105892693u -DWIDTH=1024u -DSMALL_HEIGHT=256u -DMIDDLE=11u -DWEIGHT_STEP=0x1.8e92c89f8db6dp+0 -DIWEIGHT_STEP=0x1.48da5609e739dp-1 -DWEIGHT_BIGSTEP=0x1.ae89f995ad3adp+0 -DIWEIGHT_BIGSTEP=0x1.306fe0a31b715p-1 -DPM1=0 -DAMDGPU=1 -DMM_CHAIN=1u -DMM2_CHAIN=1u -cl-fast-relaxed-math -cl-std=CL2.0 "
2020-05-18 14:58:51 412688e172fd62d9 OpenCL compilation in 3.48 s
2020-05-18 14:58:53 412688e172fd62d9 105892693 OK 0 loaded: blockSize 400, 0000000000000003
2020-05-18 14:58:54 412688e172fd62d9 105892693 OK 800 0.00%; 1429 us/it; ETA 1d 18:01; 4dc047ae8762b0a1 (check 0.77s)
2020-05-18 15:03:40 412688e172fd62d9 105892693 OK 200000 0.19%; 1427 us/it; ETA 1d 17:54; a73a8405bcd41ed8 (check 0.74s)
2020-05-18 15:08:25 412688e172fd62d9 105892693 OK 400000 0.38%; 1425 us/it; ETA 1d 17:45; 3e3786c8b16ef8c2 (check 0.75s)
2020-05-18 15:13:11 412688e172fd62d9 105892693 OK 600000 0.57%; 1425 us/it; ETA 1d 17:41; 6f318d5133311e1f (check 0.78s)
2020-05-18 15:18:01 412688e172fd62d9 105892693 OK 800000 0.76%; 1445 us/it; ETA 1d 18:11; 14321f1352fd5127 (check 0.79s)
2020-05-18 15:22:50 412688e172fd62d9 105892693 OK 1000000 0.94%; 1443 us/it; ETA 1d 18:02; 6498dc7abb4f4771 (check 0.79s)
2020-05-18 15:27:42 412688e172fd62d9 105892693 OK 1200000 1.13%; 1454 us/it; ETA 1d 18:17; cdb95d34f52909fa (check 0.78s)
2020-05-18 15:32:32 412688e172fd62d9 105892693 OK 1400000 1.32%; 1446 us/it; ETA 1d 17:59; c815b7aa86c21579 (check 0.83s)
2020-05-18 15:37:21 412688e172fd62d9 105892693 OK 1600000 1.51%; 1444 us/it; ETA 1d 17:50; 93bb7b08e1332a90 (check 0.79s)
2020-05-18 15:42:12 412688e172fd62d9 105892693 OK 1800000 1.70%; 1451 us/it; ETA 1d 17:58; e61a45d586269b6c (check 0.77s)
2020-05-18 15:47:04 412688e172fd62d9 105892693 OK 2000000 1.89%; 1452 us/it; ETA 1d 17:54; c618bf5a67675ee2 (check 0.80s)
2020-05-28 18:03:33 Note: not found 'config.txt'
2020-05-28 18:03:33 config: -d 2
2020-05-28 18:03:33 device 2, unique id '412688e172fd62d9'
2020-05-28 18:03:33 412688e172fd62d9 105892693 FFT: 5.50M 1K:11:256 (18.36 bpw)
2020-05-28 18:03:33 412688e172fd62d9 Expected maximum carry32: 5A770000
2020-05-28 18:03:34 412688e172fd62d9 OpenCL args "-DEXP=105892693u -DWIDTH=1024u -DSMALL_HEIGHT=256u -DMIDDLE=11u -DWEIGHT_STEP=0x1.8e92c89f8db6dp+0 -DIWEIGHT_STEP=0x1.48da5609e739dp-1 -DWEIGHT_BIGSTEP=0x1.ae89f995ad3adp+0 -DIWEIGHT_BIGSTEP=0x1.306fe0a31b715p-1 -DPM1=0 -DAMDGPU=1 -DMM_CHAIN=1u -DMM2_CHAIN=1u -cl-fast-relaxed-math -cl-std=CL2.0 "
2020-05-28 18:03:38 412688e172fd62d9 OpenCL compilation in 3.45 s
2020-05-28 18:03:39 412688e172fd62d9 105892693 OK 2000000 loaded: blockSize 400, c618bf5a67675ee2
2020-05-28 18:03:41 412688e172fd62d9 105892693 OK 2000800 1.89%; 1358 us/it; ETA 1d 15:11; fa1e66f696eee754 (check 0.77s)
2020-05-28 18:08:10 412688e172fd62d9 105892693 EE 2200000 2.08%; 1348 us/it; ETA 1d 14:50; 0000000000000000 (check 0.70s)
2020-05-28 18:08:11 412688e172fd62d9 105892693 EE 2000800 loaded: blockSize 400, 8b867c162df27981 (expected fa1e66f696eee754)
2020-05-28 18:08:11 412688e172fd62d9 Exiting because "error on load"
2020-05-28 18:08:11 412688e172fd62d9 Bye[/code]
Is this the failure signature one would expect from a this-card-needs-to-be-operated-using-PCI-Gen1 condition, or does it indicate something else?

So close... (I hope :).

[QUOTE=ewmayer;546723]Is this the failure signature one would expect from a this-card-needs-to-be-operated-using-PCI-Gen1 condition, or does it indicate something else?
[/QUOTE]

I'm not sure about the reason, but to me it looks like a problem with either the memory or the processor on the GPU. I would dial back the memory overclork and/or increase the voltage and see the impact. I do not identify this error pattern as being caused by PCIe; but by all means you should start by setting the PCIe to Gen1 just to rule that one out.

[QUOTE=preda;546726]I'm not sure about the reason, but to me it looks like a problem with either the memory or the processor on the GPU. I would dial back the memory overclork and/or increase the voltage and see the impact. I do not identify this error pattern as being caused by PCIe; but by all means you should start by setting the PCIe to Gen1 just to rule that one out.[/QUOTE]

In my setup, in order to fiddle with the BIOS I'd need to shut the system down, physically transfer the keyboard/mouse/monitor cabling from the Haswell system to the new build, then reboot - so I'd rather leave that as a last resort. :)

So tried backing off on the mem-OC on card3 from 1200 to 1150, that works - outstanding. Same thing I had to do on the single R7 in my Haswell system ... I keep forgetting how persnickety these cards are. Also dialed back the sclk setting on card1 (at end of board, best airflow) to sclk=3, matching the others, in order to get the power draw to something more reasonable for this modest 850W PSU ... with all 3 cards at sclk=3 drawing ~720W at wall, will average around 3.5 PRPs of expos ~105M per day.

This is great - all 3 cards running, mix of PCIe x16 and x1-plus-powered-riser-adapter, no BIOS fiddling whatsover needed.

[QUOTE=ewmayer;546757]will average around 3.5 PRPs of expos ~105M per day.
[/QUOTE]

You are nearly in the top 50 "[URL="https://www.mersenne.org/report_top_500_ll/"]First Primality Tests Top Producers[/URL]" and it won't be long until you overtake me with my single R7. I max out my bookings, one set for each running instance. I have about two weeks work before I need more. Let's see if you can overtake me before I submit my next batches! :smile:

[QUOTE=paulunderwood;546764]You are nearly in the top 50 "[URL="https://www.mersenne.org/report_top_500_ll/"]First Primality Tests Top Producers[/URL]" and it won't be long until you overtake me with my single R7. I max out my bookings, one set for each running instance. I have about two weeks work before I need more. Let's see if you can overtake me before I submit my next batches! :smile:[/QUOTE]

Using my previous estimate of ~15,000 GHz-days/week with all 4 GPUs blasting, I won't quite catch you in 2 weeks, but it'll be close. That, though, assumes your 52-week total remains the same - if, as you say, you're waiting 2 weeks to batch-submit your recent work, your 52-week total may be drop in the interim due to older results rolling off the end.

One more and hopefully-final round of small hardware tweaks needed - need a small height adjustment to front of card 3 in its custom bracketing frame, will re-use 3/4" straight PCIe adapater-spacer to move riser card further away from side of card 3 in order to improve hot-air-exit flow from vents facing said riser card, use zip ties to tidy up cabling, then will post pics. The 2 glass-topped identical monitor stands I ordered are supposed to arrive today, so in a couple days more, once I decide how to affix their foot bottoms to each other, I can also show the beast in its lair. :)

It was a lot of work, but the result is very satisfying - a massive amount of compute power in a very compact bundle which can fit on one's desktop, and paying careful attention to geometry allows all 3 GPUs to have excellent airflow, no external fans needed.

The Gimp
 

Meet The Gimp (a dual GIMPS and [i]Pulp Fiction[/i] reference), a compact 18" x 12" x 8" li'l beast of a desktop supercruncher. Running at these Radeon VII settings, it will average ~3.5 105M-sized PRP tests per day:

GPU #1 (middle of the 3): sclk = 3, fan = 150 - worst airflow of the trio, thus the higher fan setting
GPU #2 (leftmost of the 3): sclk = 4, fan = 120 - best airflow of the trio, thus runnable at higher clock
GPU #3 (rightmost of the 3): sclk = 3, fan = 120 - adding PCIe 16x riser card (arrowed) really helps airflow, by moving riser card farther away from long back of hot-air-out vents running along that side of the card.

Those settings roughly equalize the 3 GPU temps at ~80C on a typical recent day in my place, interior temp ~74F, no external fans (the latter are useful for both man and beast on warm days). I've decided to be aggressive and push the 850w PSU to nearly 800W by way of running GPU #2 at the higher clock ... sclk = 3 saves 60W, but it seems a shame to waste the excellent airflow this one enjoys. If I fry my PSU, I'll consider it 'lesson learned' and buy another - trivial to replace in this test-frame mount.

Here the final bill of materials - the crucial item, the "sine qua non" one for this build [url=https://mersenneforum.org/showpost.php?p=545197&postcount=182]as it proved[/url] is listed first. :) In retrospect it's actually a good thing that I ended up returning the first mobo+cpu bundle, the Ryzen5 octocore one with the water cooler - given my wattage constraints on this build (the PSU was bought several months ago, initially as a backup for the one in my Haswell ATX-case system) running GIMPS work on the CPU would have really pushed the wattage into the red, and the water cooler would've prevented me from mounting GPU #3 as I did in the final version, above the CPU. So hopefully some other eBayer is putting the Ryzen to better use.

All amounts rounded to nearest $:

o $8 - [url=https://www.amazon.com/gp/product/B00SW9JI9A/ref=ppx_yo_dt_b_asin_title_o04_s01?ie=UTF8&psc=1]HDMI to VGA Adapter[/url]
o $27 - [url=https://www.amazon.com/gp/product/B07VPTN8FZ/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1]Wattmeter[/url]
o $14 - [url=https://www.amazon.com/gp/product/B00EQT0YK2/]300 Mbps USB WiFi Stick[/url]
o $128 - [url=https://www.amazon.com/gp/product/B07RCKG95L/]850W 80+ Gold Certified Fully Modular Power Supply[/url]
o $1440 - 3 of [url=https://www.ebay.com/itm/XFX-AMD-Radeon-VII-16GB-HBM2/133387856426]XFX AMD Radeon VII 16GB HBM2[/url]
o $9 - [url=https://www.amazon.com/gp/product/B01FD7R0ZM/]SATA Data Cable and SATA Power Splitter Cable[/url] (I only used the one straight-plug SATA cable)
o $38 - [url=https://www.amazon.com/gp/product/B0719D9YL7/]240GB 2.5” SATA III Internal Solid State Drive (SSD)[/url]
o $50 - [url=https://www.amazon.com/gp/product/B075469KH1/]ATX Motherboard Test Bench Open Air Frame Computer Case Aluminum Bracket DIY Bare Frame[/url]
o $65 - [url=https://www.amazon.com/gp/product/B07LCP1JWS/]16GB (2 x 8GB) DDR4-RAM-2400MHz (PC4 19200) 288-pin CL17 1.2V Non ECC Unbuffered RAM[/url]
o $120 - [url=https://www.ebay.com/itm/MSI-Z270-SLI-Plus-Socket-1151-Motherboard-with-a-Celeron-G3900-2-8ghz-CPU/223991583279]MSI Z270 SLI Plus Socket 1151 Motherboard with a Celeron G3900 2.8ghz CPU[/url]
o $12 - [url=https://www.amazon.com/dp/B079SS63WV]8 Pin PCI-E PCI Express Graphics Video Card GPU VGA 8 Pin Female to Dual GPU 8Pin (6+2) Pin Male Braided Sleeved GPU 8pin Splitter Power Extension Cable (9-inch) 2 Pack[/url]
o $9 - [url=https://www.amazon.com/GODSHARK-PCI-Express-Extension-Protector-Adapter/dp/B07RWRK2L6/]PCI-E 164 pin Slot PCI-Express 16X Riser Card, Male to Female Extension Protector Upward Riser Adapter Card[/url]
o $5 - [url=https://www.amazon.com/Dr-meter-6-Pack-Powered-Adapter-Extension/dp/B078WLF5F3]16x to 1x Powered Riser Adapter Card [/url]
Total: $1,922, plus 8.5% CA sales tax comes to $2085, a smidge above my target of $2000.