Radeon VII (2nd gen consumer Vega GPU)

[M344587487]

4608K mem OD 19% perf_level test:
Used --setfan 200 as I was paranoid that the fans wouldn't ramp properly with memory temps. perf_level 8 omitted as it was hitting the power cap. The power cap can be changed with "--setpoweroverdrive WATTS" but there's no point until I figure out undervolting IMO.

perf_level wall_power rocm-smi_power temp sclk mclk ms_per_it_4608K joules_per_it
7          319        244            97   1774 1192 0.86            0.20984
6          306        235            95   1750 1192 0.87            0.20445
5          280        211            85   1684 1192 0.88            0.18568
4          236        173            73   1547 1192 0.92            0.15916
3          198        140            61   1373 1192 0.98            0.1372

Originally Posted by diep

Much appreciated - yes i like to know the size of the LDS and how many clocks it is to store and retrieve from it - as i assume the 16KB L1 it shows is the instruction cache. Knowing timing is crucial because losing suddenly more than 1 clock throughput to using the LDS makes it worthless as that throws away factor 2 performance of the GPU potentially. And also whether it can store/retrieve simultaneously for all 64 units in each SIMD the data in that 1 clock. Because that would result again in slow latencies to the LDS.

I don't know where to read that info from or how to determine it with a minimal benchmark, if anyone knows how please guide me.

Originally Posted by xx005fs

Impressive, it is faster than a Titan V with HBM overclocked to as high as it goes, by like around 10% even at stock. But it is not the improvement I was hoping for as the thread discussing about memory use for GPUOWL should means that RVII is no where near memory limited with its DP capability and should be a great deal faster than Vega 64 (0.8TFLOP vs 3.4TFLOP). So maybe it is still memory limited and that overclocking HBM would greatly help?

1200MHz is the default limit for mclk overclocks, not sure if that barrier can be broken or by how much. The above numbers should improve in efficiency with an undervolt and it would also allow for a little more thermal headroom for a faster clock albeit sacrificing efficiency. The throughput is not going to improve massively from here with the stock cooler, some people have pushed the clocks to ~2200MHz with water cooling but I doubt gpuowl would be stable. There's also potential to mod the card to improve temps, the washer mod to improve die contact with the heatsink has been reported to give decent thermal gains although the amount varies. I don't know if I have the stones to potentially crack the die on a day old £650 card but we'll see.

A few other notes:

• There's an option to change PCIe level from 308 MHz to 80 MHz with rocm-smi --setpclk 0 but it doesn't stick, as soon as the device gets under any sort of load it goes to and stays at level 1. Maybe level 1's speed can be reduced to save a little juice (if it saves any juice) as PCIe speed shouldn't matter, looks like that's a possibility in 5.2: https://www.phoronix.com/scan.php?pa...MU-Export-More
• Kernel 5.0.0 has a target temp of 95 but not a max temp which should be 105 (fixed in 5.1), be careful when overclocking
• openowl needs "-device 0" to be passed otherwise 'Exiting because "No OpenCL device"'. Probably a change I missed, I could swear it used to work with no args
• I think I've found the answer to undervolting and proper clock setting, it works a little differently on Vega 20 with a voltage curve of sorts. Testing that is for another day: https://www.reddit.com/r/linux_gamin..._undervolting/

[xx005fs]

Originally Posted by M344587487

4608K mem OD 19% perf_level test:
Used --setfan 200 as I was paranoid that the fans wouldn't ramp properly with memory temps. perf_level 8 omitted as it was hitting the power cap. The power cap can be changed with "--setpoweroverdrive WATTS" but there's no point until I figure out undervolting IMO.

perf_level wall_power rocm-smi_power temp sclk mclk ms_per_it_4608K joules_per_it
7          319        244            97   1774 1192 0.86            0.20984
6          306        235            95   1750 1192 0.87            0.20445
5          280        211            85   1684 1192 0.88            0.18568
4          236        173            73   1547 1192 0.92            0.15916
3          198        140            61   1373 1192 0.98            0.1372

I don't know where to read that info from or how to determine it with a minimal benchmark, if anyone knows how please guide me.


1200MHz is the default limit for mclk overclocks, not sure if that barrier can be broken or by how much. The above numbers should improve in efficiency with an undervolt and it would also allow for a little more thermal headroom for a faster clock albeit sacrificing efficiency. The throughput is not going to improve massively from here with the stock cooler, some people have pushed the clocks to ~2200MHz with water cooling but I doubt gpuowl would be stable. There's also potential to mod the card to improve temps, the washer mod to improve die contact with the heatsink has been reported to give decent thermal gains although the amount varies. I don't know if I have the stones to potentially crack the die on a day old £650 card but we'll see.

A few other notes:

• There's an option to change PCIe level from 308 MHz to 80 MHz with rocm-smi --setpclk 0 but it doesn't stick, as soon as the device gets under any sort of load it goes to and stays at level 1. Maybe level 1's speed can be reduced to save a little juice (if it saves any juice) as PCIe speed shouldn't matter, looks like that's a possibility in 5.2: https://www.phoronix.com/scan.php?pa...MU-Export-More
• Kernel 5.0.0 has a target temp of 95 but not a max temp which should be 105 (fixed in 5.1), be careful when overclocking
• openowl needs "-device 0" to be passed otherwise 'Exiting because "No OpenCL device"'. Probably a change I missed, I could swear it used to work with no args
• I think I've found the answer to undervolting and proper clock setting, it works a little differently on Vega 20 with a voltage curve of sorts. Testing that is for another day: https://www.reddit.com/r/linux_gamin..._undervolting/

I thought I have seen a post on Reddit that someone realized the vapor chamber coldplate was really convex, so after sanding it flat and polished the temperature greatly improved. I reckon that would be less risky than the washer method as you aren't really adding any mounting pressure on the die? https://www.reddit.com/r/Amd/comment..._stock_cooler/

Speaking about the memory frequency, I am pretty sure that if you managed to figure out how to edit the power play table, it can go above 1200MHz and be set to whatever you want, and also you will have more flexibility controlling the frequency and voltage rather than using the curve Wattman provides.

[chalsall]

Originally Posted by xx005fs

I am pretty sure that if you managed to figure out how to edit the power play table, it can go above 1200MHz and be set to whatever you want, and also you will have more flexibility controlling the frequency and voltage rather than using the curve Wattman provides.

How will this help you?

What are you trying to achieve?

[diep]

M344587487 - the question to benchmark it you can ignore it, if what i read in some documentation happens to be true - namely that you cannot manual read/write into the LDS.

It's only relevant question when you can read/write to/from it.

Much of the documentation of AMD seems to get written by the latest Indian helpdesk girl they have hired - who still has to learn how a GPU looks like - and seemingly they do not have the actual hardware at the spot where they write the documentation to verify what they write is true. Which is why information from forums is critical.

I lost months to some of the Nvidia architectures as wasted time just because i didn't realize how bad Fermat architecture is in prefetching GDDR5 versus Kepler and newer generations.
arguably it is also possible it wasn't the prefetching yet profitting more from running multiple warps at the same time could be the reason for this performance penalty as well at Fermat versus Kepler - yet the solution to sieving at Kepler generation and 900 and 1000 series Nvidia is much simpler than for Fermat architecture in that case and i lost months to figuring that one out.

So having the correct throughput latencies of features you can potentially use to speedup is very important.

However if they can't get used by the programmer then they're useless to benchmark.

In general spoken AMD and Nvidia suck bigtime in giving correct documentation for programmers - like throughput latencies and how the actual execution works of threads/warps or whatever you want to call it at the SIMDs. In fact at latest Nvidia architecture they're not giving away how many execution units it has for example for integer multiplication.

How can those manufacturers expect lots of good programs for those gpu's giving away so little information?

[diep]

Originally Posted by xx005fs

Impressive, it is faster than a Titan V with HBM overclocked to as high as it goes, by like around 10% even at stock. But it is not the improvement I was hoping for as the thread discussing about memory use for GPUOWL should means that RVII is no where near memory limited with its DP capability and should be a great deal faster than Vega 64 (0.8TFLOP vs 3.4TFLOP). So maybe it is still memory limited and that overclocking HBM would greatly help?

Out of curiosity - how many fp64 instructions (so not the factory times 2 number) get roughly executed in a single iteration on this test?

3.4 Tflops means it can execute 1.7T FP64 instructions a clock.

Parallellizing the entire transform over what is it like 80 compute units makes it really problematic to get a high percentage there out of that 1.7T.

Parallellizing supercomputers and processes at CPU's, even with complex game tree search algorithms (where the parallel algorithm just to get the task paralelllized as nonstop searches start and stop and get started based upon the result of this search) was 40 pages of a4 - despite that i usually got nearby 100% scaling and over 50% speedup at hundreds of cpu's. Compare to deep blue that was more like at under 1% there though they claimed 3% in later publication.

That > 90% is total wishful thinking at GPU's with 80 compute units.

What you can do is run coupleo of dozens of those 89M tests at the same time and then log the average iteration times after couple of minutes. And then look at the average iteration time which you will need to calculate from all the iteration times after couple of minutes. Then draw conclusions based upon that.

Start with running 2, then 4 etc.

[xx005fs]

Originally Posted by chalsall

How will this help you?

What are you trying to achieve?

Undervolting and overclocking, not for GIMPS tho. It's a common practice by miners to drop voltages below what AMD allows in Wattman (by changing voltage for other P-states or such) or Overclockers that just want to max out the hardware by changing the max power limit and voltage (obv clock speed too).

[M344587487]

Got undervolting working with that link. Turns out every R7 GPU is tested and ships with a different stock voltage, probably indicating how well you've done in the silicon lottery. Because of this undervolting works but probably won't yield as much of a benefit as undervolting Vega 56/64 does (which I believe had a flat insanely high stock voltage of 1.2V). My stock voltage is 1081mV, other people have reported stock voltages of 1127mV, 1133mV and 1040mV.

The way the voltage is set is on a curve. Here's my stock curve:

OD_VDDC_CURVE:
0:        808Mhz        690mV
1:       1304Mhz        797mV
2:       1801Mhz       1081mV

For simplicity (because there was weirdness with a phantom OD % erroneously being applied when trying to set a lower max frequency) I'm not touching the perf_level sclk's and am keeping the max frequency at 1801. Other than setting the mclk to 1200 (which reads as 1201 in rocm-smi and you push with "m 1 1200"), the only thing we change is the undervolt for perf_level 8. To undervolt in this way you simply push "vc 2 1801 $voltage_mV" to pp_od_clk_voltage, push a dummy clock change so that the undervolt applies ("s 1 1801"), push "c" to apply, then pick the desired perf_level with rocm-smi --setsclk #. A perf_level<8 should have some voltage less than or equal to that pushed. Because the card seems to have trouble sticking to the target temp of 95 when memory is overclocked these tests set the fan speed manually so that the temp is around 95.

4608K mclk=1200 perf_level=5 undervolt test:

undervolt_setting perf_level wall_power rocm-smi_power --setfan ms_per_it_4608K joules_per_it
vc 2 1801 1081    5          282        213            160      0.88            0.18744
vc 2 1801 1030    5          270        200            145      0.88            0.176
vc 2 1801 1020    5          270        200            135      0.88            0.176
vc 2 1801 1015    5          268        197            125      0.88            0.17336
vc 2 1801 1010 errors within minutes. Previous result had no errors within half an hour but it
               needs an overnight stability test and then get backed off to 1020 if it passes

4608K mclk=1200 perf_level=4 undervolt test:

undervolt_setting perf_level wall_power rocm-smi_power --setfan ms_per_it_4608K joules_per_it
vc 2 1801 1081    4          243        175            110      0.92            0.161
vc 2 1801 1030    4          237        171            105      0.92            0.15732
vc 2 1801 1020    4          236        169            100      0.92            0.15548
vc 2 1801 1010    4          234        166             97      0.92            0.15272
vc 2 1801 1000    4          233        166             95      0.92            0.15272
vc 2 1801  990 errors within minutes. Previous result had no errors within half an hour but it
               needs an overnight stability test and then get backed off to 1010 if it passes

Originally Posted by xx005fs

I thought I have seen a post on Reddit that someone realized the vapor chamber coldplate was really convex, so after sanding it flat and polished the temperature greatly improved. I reckon that would be less risky than the washer method as you aren't really adding any mounting pressure on the die? https://www.reddit.com/r/Amd/comment..._stock_cooler/

That's not a bad option, should probably buy the stuff to do it even if I end up using it for something else.

Originally Posted by xx005fs

Speaking about the memory frequency, I am pretty sure that if you managed to figure out how to edit the power play table, it can go above 1200MHz and be set to whatever you want, and also you will have more flexibility controlling the frequency and voltage rather than using the curve Wattman provides.

I've done manual editing of the powerplay table on a Vega 56 and it's a pain at least for core clocks/voltages, you need to work within certain constraints like strictly descending clocks/voltages and vega 56 had some quirks which may or may not be present for the R7. This ( https://github.com/sibradzic/upp ) looks like a good way to do it but haven't tested. Looks like it is needed to go beyond 1200 MHz, pushing "m 1 1201" to pp_od_clk_voltage complains and doesn't stick.

Out of curiosity I just tested running two instances of gpuowl at the same time and surprisingly it worked. More surprisingly it increased throughput and was more efficient to boot. Power use increased compared to a single instance at the same settings but more than made up for it in throughput. I only have time for one data point now, the rest will have to wait until tomorrow:

undervolt_setting workers perf_level wall_power rocm-smi_power --setfan  ms_per_it_for_each_worker effective_ms_per_it_4608K joules_per_it
vc 2 1801 1030    2       4          249        180            115       1.69                      0.845                     0.1521

Is this possible with Vega 10 and other cards too and I've just been oblivious?

[xx005fs]

I realized that the Vega series cards are very temperature sensitive, especially the HBM memories. Previously, I used the stock air cooler on my Vega 56 and I managed 1600MHz on 1050mV (actual load voltage is more like 1000mV), and HBM was way down to 1060MHz (for gaming that was fine but for GPUOWL i had to drop to 1020MHz). On water cooling however I managed near 1650MHz on the same voltage for core and 1150MHz on HBM (with GPUOWL dropping to 1080-1100MHz for stability). I don't know how R7 behaves when temperature decreases but it's certainly interesting (so far i have been seeing good results online with custom watercooling).
As a side note, I think the person in reddit used 2000grit polishing gel to polish the surface at the end, and I felt that the step is unnecessary as lapping something is usually just to even out the surface. As long as it's not too rough (say 200grit with very noticable grooves) it should be fine and the thermal grease should just fill it in.

[kriesel]

Originally Posted by M344587487

Is this possible with Vega 10 and other cards too and I've just been oblivious?

A single datapoint here: There's a slight 2.6% advantage in gpuowl 5.0-9c13870 on RX480 in running two instances on Windows.

[M344587487]

The results are in. Two instances is the limit for simultaneous execution, a third compiles the kernel but doesn't seem to execute. Undervolted by finding the minimum voltage that produced no errors in half an hour, then backed off the voltage by 10mV for safety. If that turns out to be unstable when tested properly all it means is that a few watts get added to the below figures.

Best results:

target               workers mclk sclk 4608K_combined_throughput_ms_it 5M_combined_throughput_ms_it rocm-smi_power_after_undervolt_YMMV
efficient_throughput 2       1201 1547 0.845                           0.95                         176
quick_single_test    1       1201 1802 0.86                            0.95                         232

That's ~443 4608K exponents per year at ~3.47kWh per exponent, or ~356 5M tests per year at ~4.32kWh per exponent using the rocm-smi figures and tuned to my card.

Here's an approximation of the script I'm using to init the card:

#!/bin/bash

if [ "$EUID" -ne 0 ]; then echo "Radeon VII init script needs to be executed as root" && exit; fi

#Allow manual control
echo "manual" >/sys/class/drm/card0/device/power_dpm_force_performance_level
#Undervolt by setting max voltage
#               V Set this to 50mV less than the max stock voltage of your card (which varies from card to card), then optionally tune it down
echo "vc 2 1801 1010" >/sys/class/drm/card0/device/pp_od_clk_voltage
#Overclock mclk to 1200
echo "m 1 1200" >/sys/class/drm/card0/device/pp_od_clk_voltage
#Push a dummy sclk change for the undervolt to stick
echo "s 1 1801" >/sys/class/drm/card0/device/pp_od_clk_voltage
#Push everything to the card
echo "c" >/sys/class/drm/card0/device/pp_od_clk_voltage
#Put card into desired performance level
/opt/rocm/bin/rocm-smi --setsclk 4 --setfan 110

"/sys/class/drm/card0/device/" or similar is a symlink which can change between boots if you have multiple GPUs. It's a good idea to use the static path of your card directly especially if non-R7 GPUs are present, which you can find with "readlink -f /sys/class/drm/card0/device".

As a quick and dirty guide to getting the same setup as this:

• Install Ubuntu 19.04 Disco which comes with kernel 5.0.x
• Edit /etc/default/grub to add amdgpu.ppfeaturemask=0xffffffff to GRUB_CMDLINE_LINUX_DEFAULT
• sudo update-grub
• sudo apt install libnuma-dev
• wget -qO - http://repo.radeon.com/rocm/apt/debian/rocm.gpg.key | sudo apt-key add -
• echo 'deb [arch=amd64] http://repo.radeon.com/rocm/apt/debian/ xenial main' | sudo tee /etc/apt/sources.list.d/rocm.list
• sudo apt update && sudo apt install rocm-dev
• echo 'SUBSYSTEM=="kfd", KERNEL=="kfd", TAG+="uaccess", GROUP="video"' | sudo tee /etc/udev/rules.d/70-kfd.rules
• reboot
• git clone https://github.com/preda/gpuowl && cd gpuowl && make
• Take the above init script and tweak it to suit your card
• Setup is done. Now at every boot you run the init script as root and run two gpuowl instances

Originally Posted by M344587487

The results are in. Two instances is the limit for simultaneous execution, a third compiles the kernel but doesn't seem to execute. Undervolted by finding the minimum voltage that produced no errors in half an hour, then backed off the voltage by 10mV for safety. If that turns out to be unstable when tested properly all it means is that a few watts get added to the below figures.

Best results:

target               workers mclk sclk 4608K_combined_throughput_ms_it 5M_combined_throughput_ms_it rocm-smi_power_after_undervolt_YMMV
efficient_throughput 2       1201 1547 0.845                           0.95                         176
quick_single_test    1       1201 1802 0.86                            0.95                         232

That's ~443 4608K exponents per year at ~3.47kWh per exponent, or ~356 5M tests per year at ~4.32kWh per exponent using the rocm-smi figures and tuned to my card.

Here's an approximation of the script I'm using to init the card:

#!/bin/bash

if [ "$EUID" -ne 0 ]; then echo "Radeon VII init script needs to be executed as root" && exit; fi

#Allow manual control
echo "manual" >/sys/class/drm/card0/device/power_dpm_force_performance_level
#Undervolt by setting max voltage
#               V Set this to 50mV less than the max stock voltage of your card (which varies from card to card), then optionally tune it down
echo "vc 2 1801 1010" >/sys/class/drm/card0/device/pp_od_clk_voltage
#Overclock mclk to 1200
echo "m 1 1200" >/sys/class/drm/card0/device/pp_od_clk_voltage
#Push a dummy sclk change for the undervolt to stick
echo "s 1 1801" >/sys/class/drm/card0/device/pp_od_clk_voltage
#Push everything to the card
echo "c" >/sys/class/drm/card0/device/pp_od_clk_voltage
#Put card into desired performance level
/opt/rocm/bin/rocm-smi --setsclk 4 --setfan 110

"/sys/class/drm/card0/device/" or similar is a symlink which can change between boots if you have multiple GPUs. It's a good idea to use the static path of your card directly especially if non-R7 GPUs are present, which you can find with "readlink -f /sys/class/drm/card0/device".

As a quick and dirty guide to getting the same setup as this:

• Install Ubuntu 19.04 Disco which comes with kernel 5.0.x
• Edit /etc/default/grub to add amdgpu.ppfeaturemask=0xffffffff to GRUB_CMDLINE_LINUX_DEFAULT
• sudo update-grub
• sudo apt install libnuma-dev
• wget -qO - http://repo.radeon.com/rocm/apt/debian/rocm.gpg.key | sudo apt-key add -
• echo 'deb [arch=amd64] http://repo.radeon.com/rocm/apt/debian/ xenial main' | sudo tee /etc/apt/sources.list.d/rocm.list
• sudo apt update && sudo apt install rocm-dev
• echo 'SUBSYSTEM=="kfd", KERNEL=="kfd", TAG+="uaccess", GROUP="video"' | sudo tee /etc/udev/rules.d/70-kfd.rules
• reboot
• git clone https://github.com/preda/gpuowl && cd gpuowl && make
• Take the above init script and tweak it to suit your card
• Setup is done. Now at every boot you run the init script as root and run two gpuowl instances

Are you sure that tuning gpu manually is better than leaving it on automatic ?