mersenneforum.org gpuOwL-specific reference material
 Register FAQ Search Today's Posts Mark Forums Read

 2020-06-01, 16:45 #23 kriesel     "TF79LL86GIMPS96gpu17" Mar 2017 US midwest 3·11·149 Posts Setting up in Linux There's a separate post in the Google Colab thread for setting that up. Ewmayer has recently created a comprehensive guide, at https://mersenneforum.org/showthread.php?t=25601 Earlier: Prime95 suggested for setting up on a local Linux system, https://mersenneforum.org/showpost.p...5&postcount=76 and "also do a sudo apt install libncurses5" https://www.mersenneforum.org/showpo...postcount=2242 Ernst's needed packages list https://mersenneforum.org/showpost.p...postcount=2243 Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=23391 Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1 Last fiddled with by kriesel on 2020-07-16 at 19:21
2020-07-30, 15:17   #24
kriesel

"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

3×11×149 Posts
Gpuowl gpu ram use scaling with exponent

GPU ram occupancy by Gpuowl v6.11-340 was observed using GPU-Z v2.33 on Windows 10 during performance of PRP iterations, for several scattered exponents. Idle gpu ram occupancy was entered for exponent 0. V6.11-364 was spot tested and observed essentially identical for the same exponent. At 100Mdigit, LL gpu ram usage was observed to be about 80% that of PRP. Note, no PRP proof computations have yet been observed for gpu ram usage. System ram usage for the process was observed to be less than gpu ram usage.
The regression fit was somewhat better plotted against FFT length than against exponent, not surprisingly.

Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Files
 gpuowl gpu ram use.pdf (11.1 KB, 120 views)

Last fiddled with by kriesel on 2020-08-10 at 02:08

2020-08-10, 00:30   #25
kriesel

"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

3·11·149 Posts
Using gpuowl's primenet.py on Windows

The script requires Python 3. Programming environments are not standard parts of a default Windows install, so Python, perl, C etc may be absent. Python may have been already installed as part of a MS Visual Studio install, or separately.

On Windows 7:
Start, Control Panel, Programs, Uninstall a Program, scroll down the list of installed programs to see if a version of Python is installed.
Or on Windows 10:
Settings, Apps, scroll down the list of installed programs to see if a version of Python is installed.
If no Python 3.x is installed, follow the installation directions at https://docs.python.org/3/using/windows.html
Test that it can be easily invoked from a command prompt.
At a command prompt: python
If python is included in the path environment variable, it will respond with something like
Code:
Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)] on win32
>>>
(Ctl-Z Enter to exit interactive python.)
If it does not recognize python, it might just be missing from the path environment string.
On Windows 7:
Fix the path in a permanent way so that Python can be run in any future command prompt:
Start, Control Panel, System, Advanced System Settings, Environment Variables...
In the System variables portion, if it's not present, append something like the following to add the location of the Python folder on your system to the path
;C:\Program Files\Python36
(Scroll down to Variable "Path", click the line, click "Edit...", scroll to and click on the right end of the existing path, paste in the necessary additional text, click OK.
(This was necessary in my case because Python was not present in the path, and the cmd line setx /M did not work because the path was already beyond its 255-char limit, thanks to a lot of NVIDIA related content, and ordinary set can handle the long path length but is temporary, as is the command prompt box in which it is set.)

Make a hi.py file to test the Python install, containing
Quote:
 print ('Hello!')
and try it:
Code:
>python hi.py
Hello!

>
If it worked, continue. If not, check that path modification.

If Gpuowl's primenet.py and upload.py are not present, copy them in. For test purposes I put mine in a gpuowl working directory.
If you have previously reported results in the results.txt file, now is a good time to move or remove them. (Otherwise they will generate "already reported" errors.)

Try running primenet.py with no parameters:
Quote:
 >python primenet.py
If the PYTHONPATH environment variable is not set properly or at all, it will produce something like
Code:
Traceback (most recent call last):
File "primenet.py", line 9, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
Create new system environment variable PYTHONPATH, per a hint from Mihai, since I found it did not exist.
On Windows 7,
Start, Control Panel, System and Security, System, Advanced system settings, Environment Variables..., scroll the system variables list. If there's no PYTHONPATH there, click New..., enter the Variable name PYTHONPATH, and enter a string corresponding to the location of requests.py as the Variable value.

By experimentation, the value needed seems to be in my case,
Quote:
 C:\Program Files\Python36\Lib\site-packages\pip\_vendor
There should already be a proof subfolder in the working directory. Otherwise it will generate an error from the script.
When the script runs it will create subfolders __pycache__ and uploaded.
It requires at least a username. Following is the beginning of a successful run.
Code:
>python primenet.py -u primenet-uid

Work type: 150
User: primenet-uid
Watched dirs:  ./
Primenet password: primenet-pwd
It will wait and loop until terminated.

For reference, from gpuowl's readme.md, primenet.py's case-sensitive options are
Code:
## Primenet.py Arguments
-h, --help            show this help message and exit\
-t TIMEOUT            Seconds to sleep between updates\
--dirs DIR $DIR ...$  GpuOwl directories to scan\
-w \{PRP,PM1,LL_DC,PRP_DC,PRP_WORLD_RECORD,PRP_100M\}   GIMPS work type
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2020-08-10 at 02:30

2020-08-10, 13:28   #26
kriesel

"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

3×11×149 Posts

Gpuowl now supports generation of PRP proof files. These require uploading to the server for verification. Preda has provided primenet.py and upload.py for performing the upload. Installing Python on every system to use them may be unappealing or impractical for some. Here are some alternatives. Note, upload the JSON results records first, then the corresponding proof files later.
1. Python interpreter, primenet.py, upload.py on each system. (Might not be allowed by corporate policy, disk space, available user time, etc.) This is I think the preferred method supported and created by Preda, author of Gpuowl.
2. Python interpreter, primenet.py, upload.py on one system. Manually or by small script copy proof files over from other systems, to a file share on the Python-equipped system in which primenet.py operates.
3. Upload via a prime95 or mprime install or multiple installs. As George described it (mostly) at https://www.mersenneforum.org/showpo...&postcount=220
Code:
1) Manually submit your results.json.txt first, before giving prime95 or mprime a chance at the proof file.
2) move the proofs to a folder where prime95 v30.2 (or newer, or mprime) is running.
3) wait for the proofs to get uploaded (I archive my proofs using advanced resource limits dialog box).
Note that applied to prime95; for gpuowl submit results.txt contents in step 1. Note that if there's an MD5 mismatch on the proof file, it may just loop on retries ~65 minutes apart until you manually intervene, logged in results.txt, in prime95 V30.3b3, resembling the following.
Code:
[Fri Sep  4 07:51:19 2020]
Unexpected error during 58000013-8.proof upload: {"error_status":401,"error_message":"Unauthorized","error_description":""}
A nonnull error description would be helpful there; something like "MD5 mismatch detected", or whatever the case may be. I think this is a server-side consideration.
4. For V1 proofs produced by gpuowl V6.11-380 to v7.1-x, but not V2 proofs produced by gpuowl V7.2 or later: Use the standalone uploader.exe program Prime95 previously provided for 64-bit Windows at https://www.mersenneforum.org/showpo...&postcount=154. (No longer available on dropbox from that link. See attachment on this post below. Uploader reposted with George's permission.) Usage is
Code:
uploader user_id proof_filename[ chunk_size[  upload_rate_limit]]
with chunk_size expressed in MB and upload_rate_limit expressed in Mbps apparently.
A file share and tiny batch script can expedite this.
Code:
for %%a in ( *.proof ) do uploader your-primenet-uid %%a
Or for a bit of logging of the uploads,
Code:
for %%a in (*.proof) do powershell ".\uploader.exe your-primenet-uid %%a | tee uplog.txt -Append"
To include time stamps in the logging, and console output:
Code:
prompt $d$t $p$g
for %%a in (*.proof) do powershell "echo start upload %%a %date% %time% >> uplog.txt;.\uploader.exe kriesel %%a | tee uplog.txt -Append; echo end upload %%a %date% %time% >>uplog.txt"
Code:
Thu 08/27/2020 10:59:53.11 E:\PRP-PROOF-CACHE>for %a in (*.proof) do uploader kriesel %a

Thu 08/27/2020 10:59:53.12 E:\PRP-PROOF-CACHE>uploader kriesel 160456789-8.proof
MD5 of 160456789-8.proof is 5f8f1c693b35765a465072735a042649
Proof file exponent is 160456789
Filesize of 160456789-8.proof is 180513949
No entries in need list: {"error_status":401,"error_message":"Unauthorized","error_description":""}

Thu 08/27/2020 10:59:55.76 E:\PRP-PROOF-CACHE>up

Thu 08/27/2020 11:01:23.41 E:\PRP-PROOF-CACHE>for %a in (*.proof) do uploader kriesel %a

Thu 08/27/2020 11:01:23.42 E:\PRP-PROOF-CACHE>uploader kriesel 160456789-8.proof
MD5 of 160456789-8.proof is 5f8f1c693b35765a465072735a042649
Proof file exponent is 160456789
Filesize of 160456789-8.proof is 180513949
Success!

Thu 08/27/2020 11:01:46.65 E:\PRP-PROOF-CACHE>
Occasionally an upload will report an error, but a retry will report it has already been uploaded.
Code:
E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 39000037-8.proof | tee uplog.txt -Append"
MD5 of 39000037-8.proof is ed9f4f30202027b27a6bbed5caa84691
Proof file exponent is 39000037
Filesize of 39000037-8.proof is 43875102
CURL library error: Operation timed out after 180012 milliseconds with 0 bytes received

E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
Proof file exponent is 41000051
Filesize of 41000051-8.proof is 46125120
CURL library error: Operation timed out after 180011 milliseconds with 0 bytes received
E:\PRP-PROOF-CACHE>up

E:\PRP-PROOF-CACHE>for %a in (*.proof) do powershell ".\uploader.exe kriesel %a | tee uplog.txt -Append"

E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 39000037-8.proof | tee uplog.txt -Append"
MD5 of 39000037-8.proof is ed9f4f30202027b27a6bbed5caa84691
Proof file exponent is 39000037
Filesize of 39000037-8.proof is 43875102

E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
Proof file exponent is 41000051
Filesize of 41000051-8.proof is 46125120
CURL library error: Operation timed out after 180005 milliseconds with 0 bytes received
E:\PRP-PROOF-CACHE>up

E:\PRP-PROOF-CACHE>for %a in (*.proof) do powershell ".\uploader.exe kriesel %a | tee uplog.txt -Append"

E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
Proof file exponent is 41000051
Filesize of 41000051-8.proof is 46125120
E:\PRP-PROOF-CACHE>
In my experience these do go on to get certs run, successfully, so seem not to be an issue. Note that if there's an MD5 mismatch, it may look sort of like this:
Code:
MD5 of 58000013-8.proof is fed107d6833f4c8e1f67a450e15272bc
Proof file exponent is 58000013
Filesize of 58000013-8.proof is 65250075
Server response missing URLToUse: {"error_status":401,"error_message":"Unauthorized","error_description":""}
A nonnull error description would be helpful there; something like "MD5 mismatch detected", or whatever the case may be. I think this was a server-side security consideration. It has been changed to give a list of possible reasons, which the end user can check. At some point the standalone uploader may become no longer compatible with gpuowl or future proof file versions in general. That may be V7.2 or proof version 2. So far, I've seen these combinations (and more):
Code:
gpuowl V# ok? examples
V6.11-364 yes 66000013
V7.0-35   yes 150714349, 149929849
V7.0-40   yes 123456841
V7.1-1    ? stay tuned
V7.1-11   yes 153021377
V7.2-13   no  120132049 "Error getting version number from proof header"
5. Roll your own uploader script using curl or perl or whatever. For example, https://www.mersenneforum.org/showpo...&postcount=271
6. Translation of the python scripts to a preferred language, such as perl. This is unappealing, as it would create a recurring maintenance workload whenever Preda revises his python scripts. Also knowing both the preferred language and knowing or learning python.
7. Compilation of the gpuowl primenet.py, upload.py into a standalone executable. (I'm exploring this. If it works out, the primenet executable could be included with future Windows gpuowl build posts.) This would approximate #1 above, without the need to install a Python interpreter on every system running gpuowl.
Possibly some client management applications will add proof uploading capability at some point, also. (Mlucas primenet.py seems likely)
The total number of different uploaders ought be limited to a manageable set for the server maintainers to support on that end.

Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Files

Last fiddled with by kriesel on 2020-12-07 at 21:04 Reason: added uploader gpuowl version data; results.txt vs results.json.txt

2020-12-06, 15:33   #27
kriesel

"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

3·11·149 Posts
User base OS mix indications

There was a comment in the http://mersenneforum.org/showthread.php?t=22204 thread several months ago that 99% of gpuowl users ran it on Linux. This led me to think about what the real mix of gpuowl usage versus OS was, and where to find indications. I came up with several, which vary in mix indication, but seem to indicate about equal usage of Windows and Linux. These were examined in July, and I'm finally getting around to posting it now.
This post is not about bashing any participant, OS, etc. but about examining what usage distribution is by all practical means. All the following are limited by small or moderate statistical sample size. Within that limitation, they are consistent with gpuowl usage rates roughly comparable between Linux and Windows.

1) I asked the commenter for the basis of the statement. There was no response. I now believe it was hyperbole.

2) I started a poll. The results indicated 48% Linux, 57% Windows. https://www.mersenneforum.org/showpo...34&postcount=5

3) I reviewed the gpuowl thread for what users who posted there indicated about their own use. That showed 44% Windows-only, 37% Linux only, 10% both, 0% other, 10% undetermined (rounded to the nearest percent), or 54% Windows, 47% Linux after factoring in users of both (including Google colab users like myself as Linux usage).

4) I reviewed all the posts of Windows or Linux builds, as of mid June, and tabulated then how many views (downloads) were indicated as having occurred for each. There were many Windows builds posted, but only one for Linux.
Average number of downloads per build post for Windows was ~31; for Linux, 16, a ratio of nearly 2:1. (Maximum download count per build was 121 then, in post 1877). The build view count if accurate is a lower bound on users, since others may build their own, and because kracker and I and perhaps others post builds, not download our own builds. There does seem to be a decline over time in the number of views (downloads) per Windows build posted. This could be reflective of a shift toward Linux or off Windows, or simply that downloads are cumulative.
The scatter is quite substantial. One theory for it is that a moderator has modified some of the counts. Another is the long gap in builds prior to that post 1877 and another that had 90 views led to an unusually high number of updates using the build. Another is a bot or spider regularly retrieving it. Another is that some builds offer new features of significant value that will motivate a large number of downloads. In considering the nearly 2:1 ratio, note that Linux users are much more likely to do their own builds, for various reasons, including the large number of Linux variants, and the prevalence of zero cost build environments in Linux.

5) I asked James Heinrich (administrator of mersenne.ca and the GIMPS software download mirror) if he analyzed server logs for the download.mersenne.ca software mirror site. He indicated no, but sent me (anonymized) summary log data for May 2020 and oked sharing analysis of it.
For gpuowl, this indicated downloads for Windows, but not Linux. The counts likely are contaminated somewhat by spiders and bots activity. This analysis seeems to indicate substantial Windows usage on all GIMPS gpu applications, as well as on prime95.

6) James Heinrich also provided anonymized summary log data for early June. From June 1 to June 15, 2020, there were 8 downloads of gpuowl for Linux, and 9 of gpuowl for Windows. Again, rough comparability and small sample size.
The overall download rate of all apps was higher for Windows (40), than for Linux (22)

7) I asked Mihai Preda (gpuowl author) about any available statistics from the github repository for gpuowl. None were available. They would not reflect OS usage very well, since every time msys2 is used to create a Windows build, it is probably mistaken for Linux. And that one access results in dozens of downloads from builds posted to the forum.

8) Somewhat arbitrarily averaging as follows, the various measures above, combining 5 and 6 because they are the same measure for two different time periods, gives an aggregate of all samples available:
Code:
Average of 2, 3, 4, (5+6)/2
Linux Windows
2       48    57
3       47    54
4       34    66
5 /2     0    50
6 /2    18    32
average 37    65
None of the above indicates how many gpus are running gpuowl per user or total or versus OS, or any difference in prevalence of fast versus slow gpus. I build and post once and deploy the build to many gpus on Windows. Other users are running multiple-gpu systems on Linux (including George and Ernst and Mihai).

Little of the above indicates the impact of cloud computing. Paid cloud computing price structures heavily favor Linux. Free Google Colaboratory usage is Linux-only, including my own regular attempts to do P-1 or TF there. In fact that is why I list myself as using both Windows and Linux in the tabulation of users in the gpuowl thread.

From the available sparse data we can conclude that the Linux and Windows contingents were roughly comparable. All the available measures indicated higher Windows adoption than Linux adoption. Mihai Preda made a wise choice to support both from the very first week of the original gpuowl announcement thread.

There are reports of speed advantages using Linux with the ROCm driver, and a few features require that driver. It's likely to motivate some migration from Windows to Linux/ROCm over time. Unfortunately while WSL2 is bringing some gpu support, it's CUDA and DirectML, not OpenCL which gpuowl depends on. Maybe later.

Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Files
 gpuowl build post views.pdf (13.5 KB, 43 views) user mix per forum thread posts.pdf (41.7 KB, 37 views) linux windows prevalence.txt (496 Bytes, 41 views) downloads from mirror by OS summary.pdf (9.3 KB, 43 views) downloads from mirror by OS detail.pdf (20.4 KB, 33 views)

 Similar Threads Thread Thread Starter Forum Replies Last Post kriesel kriesel 62 2020-12-12 08:57 kriesel kriesel 31 2020-07-09 14:04 kriesel kriesel 9 2020-05-28 23:32 kriesel kriesel 8 2020-04-17 03:50 kriesel kriesel 12 2019-08-12 15:51

All times are UTC. The time now is 06:36.

Thu Mar 4 06:36:22 UTC 2021 up 91 days, 2:47, 1 user, load averages: 2.57, 2.62, 2.51