mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Hardware > GPU Computing > GpuOwl

Reply
 
Thread Tools
Old 2020-07-18, 18:59   #1
preda
 
preda's Avatar
 
"Mihai Preda"
Apr 2015

7·173 Posts
Default GpuOwl PRP-Proof changes

A mention of recent changes related to PRP-Proof in GpuOwl as of 2020-07-19:

Proof generation is now enabled by default, no need to do anything for that.

1. Power
The proof power can be controlled with -proof <power> argument, in the range 6-9. By default (i.e. when not using -proof) the power is 8. If low on disk space use 7 or 6, otherwise the default 8 should be good. The minimum disk use (at power 6) is about 0.8 GB for a 100M exponent.

On startup the program validates the list of temporary proof residues that should be there already, and tries to find a proof power that matches the existing residues. If the needed residues aren't there, proof generation is disabled for the current exponent (but will probably work correctly for the next exponent).

It's not possible to change the proof power on an ongoing exponent -- but the program should detect and ignore such change attempts (but they will be applied on the next exponent started from the start).

2. Storage
By default the temporary residues (many files) are stored in:
working-directory-of-gpuowl/<exponent>/proof/

Because these temporary residues may take up a lot of space, there were requests to allow storing them to a different location (such as a NFS mount) that may have more space. This is possible with the new -tmpDir argument, in which case the tmp residues are store into:
<tmpDir>/<exponent>/proof/

The temporary storage is not cleaned yet -- but this should be done soon.

3. Results
When a PRP test that has proof enabled is finished, the proof will be generated automatically. The proof generation is fast, takes on the order of minutes. The proof file location should be displayed on logs, but it goes to:
gpuowl-working-dir/proof/ if not using -pool <dir>, or to:
pool-dir/proof/ if using -pool <dir>

The proof file name is formed from exponent and power like this: 97309379-8.proof

If for some reason the proof for the same exponent&power is re-generated, the existing proof is overwritten.

The results JSON for a PRP test that generated a proof will have a new section "proof" that looks like this:
"proof":{"version":"1", "power":"8", "hashsize":"64", "MD5":"5818c5af84ab468717dae240ffffaaaa"}
which includes the MD5 of the proof file.

There will be a mechanism for uploading the proof files themselves soon, but for now they should just be stored. (i.e. don't delete the .proof files yet)

Last fiddled with by preda on 2020-07-18 at 19:13
preda is offline   Reply With Quote
Old 2020-07-18, 19:07   #2
preda
 
preda's Avatar
 
"Mihai Preda"
Apr 2015

121110 Posts
Default

There may be bugs. It's still in flux, there are still changes coming in (such as the cleaning of the temporary residues), so a general-availability build is not needed yet -- the intention was to let everybody know of the upcoming changes early.
preda is offline   Reply With Quote
Old 2020-07-19, 00:52   #3
preda
 
preda's Avatar
 
"Mihai Preda"
Apr 2015

7×173 Posts
Default Proof temporary storage cleanup

Now the proof temporary storage should be cleaned up properly after proof generation.

This means:
- the directory <base>/<exponent>/proof/ will be emptied and removed
- the directory <base>/<exponent> will be removed only if empty

where <base> is the temporary-storage directory if specified (-tmpDir), otherwise the GpuOwl's working directory (-dir).

An undocumented option exists, "-keep proof", that disables the above cleanup. This option is not considered generally useful (thus not documented), it's intended for debugging proof generation, and will probably be dropped in the future.
preda is offline   Reply With Quote
Old 2020-07-20, 01:09   #4
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

22×1,109 Posts
Default

https://www.mersenne.org/report_expo...7000009&full=1
PRP -proof 8. Proof file 150.5 MB cached. Temporary files total 4.08 GB.
kriesel is offline   Reply With Quote
Old 2020-07-24, 09:43   #5
preda
 
preda's Avatar
 
"Mihai Preda"
Apr 2015

7·173 Posts
Default Proof upload

It is now possible to upload proofs, here's how:

1. Manual upload of a single proof
Use the new script tools/upload.py , invoke it like this:
upload.py <user-ID> <proof-file>
where <user-ID> is your mersenne.org ID.

2. Integrated into tools/primenet.py
The old primenet.py script will now watch the "proof/" subfolder in the watched dirs. If .proof files are found there, the script attempts to upload them. If uploaded successfully, the proofs are moved to a subfolder "uploaded/". (this "uploaded" subfolder is created if not existing)

The primenet.py script works best in conjunction with the -pool <dir> option to gpuowl, which has the effect that the script can be run with a single folder to watch (the pool directory) where all the results and proofs from all the gpuowl instances accumulate.

Note: the PRP JSON textual result now contains a "proof" section mentioning the generated proof. The old JSONs didn't have this part. It is possible that proof files exist for JSONs without a "proof" section -- they should still be accepted by primenet, but it is possible that the upload fails for this case ATM (this situation is investigated). It may be useful to report upload failures in any case, just to double check the "why" this being rather new.

Kudos to George, Aaron & James who did the heavy lifting server-side with such speed.

Last fiddled with by preda on 2020-07-24 at 09:51
preda is offline   Reply With Quote
Old 2020-07-24, 10:08   #6
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

3·5·227 Posts
Default

Two problems:

1. I can't compile the latest gpuOwl:

Code:
 make gpuowl
echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-361-g2274207-dirty"
g++-8 -MT Pm1Plan.o -MMD -MP -MF .d/Pm1Plan.Td -Wall -O2 -std=c++17   -c -o Pm1Plan.o Pm1Plan.cpp
g++-8 -MT GmpUtil.o -MMD -MP -MF .d/GmpUtil.Td -Wall -O2 -std=c++17   -c -o GmpUtil.o GmpUtil.cpp
g++-8 -MT Worktodo.o -MMD -MP -MF .d/Worktodo.Td -Wall -O2 -std=c++17   -c -o Worktodo.o Worktodo.cpp
g++-8 -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17   -c -o common.o common.cpp
g++-8 -MT main.o -MMD -MP -MF .d/main.Td -Wall -O2 -std=c++17   -c -o main.o main.cpp
g++-8 -MT Gpu.o -MMD -MP -MF .d/Gpu.Td -Wall -O2 -std=c++17   -c -o Gpu.o Gpu.cpp
In file included from Gpu.cpp:4:
ProofSet.h:161:8: error: ‘unordered_map’ in namespace ‘std’ does not name a template type
   std::unordered_map<u32, Words> pending;
        ^~~~~~~~~~~~~
ProofSet.h:161:3: note: ‘std::unordered_map’ is defined in header ‘<unordered_map>’; did you forget to ‘#include <unordered_map>’?
ProofSet.h:16:1:
+#include <unordered_map>
 
ProofSet.h:161:3:
   std::unordered_map<u32, Words> pending;
   ^~~
ProofSet.h: In member function ‘void ProofCache::flush()’:
ProofSet.h:206:20: error: ‘pending’ was not declared in this scope
     for (auto it = pending.cbegin(), end = pending.cend(); it != end && write(it->first, it->second); it = pending.erase(it));
                    ^~~~~~~
ProofSet.h:207:10: error: ‘pending’ was not declared in this scope
     if (!pending.empty()) {
          ^~~~~~~
ProofSet.h: In member function ‘void ProofCache::save(u32, const Words&)’:
ProofSet.h:218:9: error: ‘pending’ was not declared in this scope
     if (pending.empty() && write(k, words)) { return; }
         ^~~~~~~
ProofSet.h:219:5: error: ‘pending’ was not declared in this scope
     pending[k] = words;
     ^~~~~~~
ProofSet.h: In member function ‘Words ProofCache::load(u32, u32) const’:
ProofSet.h:224:15: error: ‘pending’ was not declared in this scope
     auto it = pending.find(k);
               ^~~~~~~
ProofSet.h: In member function ‘void ProofCache::clear()’:
ProofSet.h:228:18: error: ‘pending’ was not declared in this scope
   void clear() { pending.clear(); }
                  ^~~~~~~
make: *** [Makefile:30: Gpu.o] Error 1
2. I Can't upload:

Code:
 ./upload.py "PaulUnderwood" ../../../../gpuowl/104982617/104982617-9.proof 
Uploading M104982617 from "../../../../gpuowl/104982617/104982617-9.proof"
http://mersenne.org/proof_upload/?UserID=PaulUnderwood&Exponent=104982617&FileSize=131228338&FileMD5=98b1ab8540e226083ada38ef1abb7ad5
{'error_status': 401, 'error_message': 'Unauthorized', 'error_description': ''}
paulunderwood is offline   Reply With Quote
Old 2020-07-24, 10:22   #7
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

307 Posts
Default

On my end, the uploading works fine.
kruoli is offline   Reply With Quote
Old 2020-07-24, 11:58   #8
preda
 
preda's Avatar
 
"Mihai Preda"
Apr 2015

7×173 Posts
Default

Quote:
Originally Posted by paulunderwood View Post
Two problems:

1. I can't compile the latest gpuOwl:
2. I Can't upload:
Please re-try the compilation, I attempted a fix.

The upload problem you see may be related to what I mentioned, the JSON result without "proof" -- under investigation.

Last fiddled with by preda on 2020-07-24 at 12:30
preda is offline   Reply With Quote
Old 2020-07-24, 12:05   #9
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

D4D16 Posts
Default

Quote:
Originally Posted by preda View Post
Please re-try the compilation, I attepted a fix.

The upload problem you see may be related to what I mentioned, the JSON result without "proof" -- under investigation.
Compiled well.

Edit: Uploading requires the new format

Last fiddled with by paulunderwood on 2020-07-24 at 14:03
paulunderwood is offline   Reply With Quote
Old 2020-07-25, 00:35   #10
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

3·5·227 Posts
Default

Quote:
Originally Posted by paulunderwood View Post
Edit: Uploading requires the new format
Uploading seems to work now for the old proof result format.
paulunderwood is offline   Reply With Quote
Old 2020-07-26, 11:52   #11
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

65158 Posts
Default

Uploading ability seems to be intermittent. 401 error. I will try later.
paulunderwood is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Gpuowl Windows builds kriesel GpuOwl 8 2020-09-08 03:45
gpuOWL for Wagstaff GP2 GpuOwl 22 2020-06-13 16:57
Gpuowl / Linux question Prime95 GpuOwl 13 2020-01-03 22:44
gpuowl tuning M344587487 GpuOwl 14 2018-12-29 08:11

All times are UTC. The time now is 11:12.

Sun Sep 27 11:12:10 UTC 2020 up 17 days, 8:23, 0 users, load averages: 1.28, 1.43, 1.43

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.