View Single Post
Old 2020-08-10, 13:28   #26
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

4,931 Posts
Default Methods of uploading proof files

Gpuowl now supports generation of PRP proof files. These require uploading to the server for verification. Preda has provided primenet.py and upload.py for performing the upload. Installing Python on every system to use them may be unappealing or impractical for some. Here are some alternatives. Note, upload the JSON results records first, then the corresponding proof files later.
  1. Python interpreter, primenet.py, upload.py on each system. (Might not be allowed by corporate policy, disk space, available user time, etc.) This is I think the preferred method supported and created by Preda, author of Gpuowl.
  2. Python interpreter, primenet.py, upload.py on one system. Manually or by small script copy proof files over from other systems, to a file share on the Python-equipped system in which primenet.py operates.
  3. Upload via a prime95 or mprime install or multiple installs. As George described it (mostly) at https://www.mersenneforum.org/showpo...&postcount=220
    Code:
    1) Manually submit your results.json.txt first, before giving prime95 or mprime a chance at the proof file.
    2) move the proofs to a folder where prime95 v30.2 (or newer, or mprime) is running.
    3) wait for the proofs to get uploaded (I archive my proofs using advanced resource limits dialog box).
    Note that applied to prime95; for gpuowl submit results.txt contents in step 1. Note that if there's an MD5 mismatch on the proof file, it may just loop on retries ~65 minutes apart until you manually intervene, logged in results.txt, in prime95 V30.3b3, resembling the following.
    Code:
    [Fri Sep  4 07:51:19 2020]
    Unexpected error during 58000013-8.proof upload: {"error_status":401,"error_message":"Unauthorized","error_description":""}
    A nonnull error description would be helpful there; something like "MD5 mismatch detected", or whatever the case may be. I think this is a server-side consideration.
  4. For V1 proofs produced by gpuowl V6.11-380 to v7.1-x, but not V2 proofs produced by gpuowl V7.2 or later: Use the standalone uploader.exe program Prime95 previously provided for 64-bit Windows at https://www.mersenneforum.org/showpo...&postcount=154. (No longer available on dropbox from that link. See attachment on this post below. Uploader reposted with George's permission.) Usage is
    Code:
    uploader user_id proof_filename[ chunk_size[  upload_rate_limit]]
    with chunk_size expressed in MB and upload_rate_limit expressed in Mbps apparently.
    A file share and tiny batch script can expedite this.
    Code:
    for %%a in ( *.proof ) do uploader your-primenet-uid %%a
    Or for a bit of logging of the uploads,
    Code:
    for %%a in (*.proof) do powershell ".\uploader.exe your-primenet-uid %%a | tee uplog.txt -Append"
    To include time stamps in the logging, and console output:
    Code:
    prompt $d $t $p$g 
    for %%a in (*.proof) do powershell "echo start upload %%a %date% %time% >> uplog.txt;.\uploader.exe kriesel %%a | tee uplog.txt -Append; echo end upload %%a %date% %time% >>uplog.txt"
    Retry in case of upload failure is manual; remove or rename successfully uploaded proof files or exclude them with the filespec. Delete the local proof copies after successful upload, if at all. (I'm currently using uploader.exe on a file share for occasional manual upload of gpuowl-produced proof files, and deleting local proof files after successful upload. Earlier I was waiting to delete until after a successful verification was manually confirmed.) Note, while prime95 and mprime's upload capability is being maintained, the standalone uploader may not be, and error handling may be somewhat less effective. If I recall correctly, this was created as a stopgap, before prime95 included proof file upload capability. Mprime/prime95 upload is preferred over the standalone uploader. The standalone uploader may require more than one user initiated try, as in the following.
    Code:
    Thu 08/27/2020 10:59:53.11 E:\PRP-PROOF-CACHE>for %a in (*.proof) do uploader kriesel %a
    
    Thu 08/27/2020 10:59:53.12 E:\PRP-PROOF-CACHE>uploader kriesel 160456789-8.proof
    MD5 of 160456789-8.proof is 5f8f1c693b35765a465072735a042649
    Proof file exponent is 160456789
    Filesize of 160456789-8.proof is 180513949
    No entries in need list: {"error_status":401,"error_message":"Unauthorized","error_description":""}
    
    Thu 08/27/2020 10:59:55.76 E:\PRP-PROOF-CACHE>up
    
    Thu 08/27/2020 11:01:23.41 E:\PRP-PROOF-CACHE>for %a in (*.proof) do uploader kriesel %a
    
    Thu 08/27/2020 11:01:23.42 E:\PRP-PROOF-CACHE>uploader kriesel 160456789-8.proof
    MD5 of 160456789-8.proof is 5f8f1c693b35765a465072735a042649
    Proof file exponent is 160456789
    Filesize of 160456789-8.proof is 180513949
    Success!
    
    Thu 08/27/2020 11:01:46.65 E:\PRP-PROOF-CACHE>
    Occasionally an upload will report an error, but a retry will report it has already been uploaded.
    Code:
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 39000037-8.proof | tee uplog.txt -Append"
    MD5 of 39000037-8.proof is ed9f4f30202027b27a6bbed5caa84691
    Proof file exponent is 39000037
    Filesize of 39000037-8.proof is 43875102
    CURL library error: Operation timed out after 180012 milliseconds with 0 bytes received
    
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
    MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
    Proof file exponent is 41000051
    Filesize of 41000051-8.proof is 46125120
    CURL library error: Operation timed out after 180011 milliseconds with 0 bytes received
    E:\PRP-PROOF-CACHE>up
    
    E:\PRP-PROOF-CACHE>for %a in (*.proof) do powershell ".\uploader.exe kriesel %a | tee uplog.txt -Append"
    
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 39000037-8.proof | tee uplog.txt -Append"
    MD5 of 39000037-8.proof is ed9f4f30202027b27a6bbed5caa84691
    Proof file exponent is 39000037
    Filesize of 39000037-8.proof is 43875102
    Server response missing URLToUse: {"error_status":409,"error_message":"Conflict","error_description":"Proof already uploaded"}
    
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
    MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
    Proof file exponent is 41000051
    Filesize of 41000051-8.proof is 46125120
    CURL library error: Operation timed out after 180005 milliseconds with 0 bytes received
    E:\PRP-PROOF-CACHE>up
    
    E:\PRP-PROOF-CACHE>for %a in (*.proof) do powershell ".\uploader.exe kriesel %a | tee uplog.txt -Append"
    
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
    MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
    Proof file exponent is 41000051
    Filesize of 41000051-8.proof is 46125120
    Server response missing URLToUse: {"error_status":409,"error_message":"Conflict","error_description":"Proof already uploaded"}
    E:\PRP-PROOF-CACHE>
    In my experience these do go on to get certs run, successfully, so seem not to be an issue. Note that if there's an MD5 mismatch, it may look sort of like this:
    Code:
    MD5 of 58000013-8.proof is fed107d6833f4c8e1f67a450e15272bc
    Proof file exponent is 58000013
    Filesize of 58000013-8.proof is 65250075
    Server response missing URLToUse: {"error_status":401,"error_message":"Unauthorized","error_description":""}
    A nonnull error description would be helpful there; something like "MD5 mismatch detected", or whatever the case may be. I think this was a server-side security consideration. It has been changed to give a list of possible reasons, which the end user can check. At some point the standalone uploader may become no longer compatible with gpuowl or future proof file versions in general. That may be V7.2 or proof version 2. So far, I've seen these combinations (and more):
    Code:
    gpuowl V# ok? examples
    V6.11-364 yes 66000013
    V7.0-35   yes 150714349, 149929849
    V7.0-40   yes 123456841
    V7.1-1    ? stay tuned
    V7.1-11   yes 153021377
    V7.2-13   no  120132049 "Error getting version number from proof header"
  5. Roll your own uploader script using curl or perl or whatever. For example, https://www.mersenneforum.org/showpo...&postcount=271
  6. Translation of the python scripts to a preferred language, such as perl. This is unappealing, as it would create a recurring maintenance workload whenever Preda revises his python scripts. Also knowing both the preferred language and knowing or learning python.
  7. Compilation of the gpuowl primenet.py, upload.py into a standalone executable. (I'm exploring this. If it works out, the primenet executable could be included with future Windows gpuowl build posts.) This would approximate #1 above, without the need to install a Python interpreter on every system running gpuowl.
Possibly some client management applications will add proof uploading capability at some point, also. (Mlucas primenet.py seems likely)
The total number of different uploaders ought be limited to a manageable set for the server maintainers to support on that end.


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=23391
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Files
File Type: zip uploader.zip (1.93 MB, 134 views)

Last fiddled with by kriesel on 2020-12-07 at 21:04 Reason: added uploader gpuowl version data; results.txt vs results.json.txt
kriesel is online now