The Next Big Development for GIMPS

[Fan Ming]

In my view, the default power should be 10 or even bigger. 10(or dozens of) GB free space is not too much for current PCs or most servers. But for most PCs/servers, it would cost a lot on hardware to let their compute capability 1.5x (or more) faster (without overclock).
Moreover, the disk space is much cheaper than compute capability, and It's really worth to cost more disk space to save compute times, even dozens of GB.
When the disk is indeed not too much, I think it would not be hard for software to detect the disk space and decrease the power compared to default settings.

[srow7]

what happens when there is not enough disk space for any power ?
does software check for enough disk space before starting calc ?
if not enough, does it just move to next assignment ?

[preda]

Originally Posted by Prime95

Prime95 can now generate proofs. Can't upload them or verify them, but it is a start.

Some interesting data and a decision to be made.

For a 100Mbit number, the optimal proof power is 10. Requiring proof generator and verifier do 182K squarings. Temporary disk space is 12.8GB. Proof file size is 138MB.

The power=9 proof is almost as good at 239K squarings, disk space 6.4GB, proof file size is 125MB. The difference of 57K squarings is only 0.06% the cost of a full PRP test.

What about power=8? 413K squarings, disk space 3.2GB, proof file size is 113MB. The 231K extra squarings is 0.23% of a PRP test.


So the question is what should be prime95's default setting? The optimize-to-the-max in me says default should be power=9. The minimal-impact-on-average-user in me says go with power=8 default.

I think the biggest impact on the average user is the disk space consumed. Power=8 will result in less proofs lost due to disk full errors. The cost is just 0.17% of a PRP test.

I think the preferences dialog box needs a "Max disk space each worker can use" setting. From that I can derive the best power setting, but most users will never change this preference so the question just morphs into what the default for the max GB of disk should be,

GpuOwl also can generate and verify proofs now!
And, most probably, the proofs are cross-compatible between the two software! And, the two programs (Prime95, GpuOwl) have distinct implementations written independently by different people, so it's highly unlikely to suffer from the same programmer error in the same way. They also run on different hardware, so it's highly unlikely to trigger the same hardware error in the same way. That is to say, such a cross-verification is quite strong. (I think, even the SHA3 library used by the two is different :)

For GpuOwl, the default proof power will be 8. This would result in temporary disk requirements of 3GB per 100M exponent. Users with plenty of disk would have an option to use power 9, doubling the temporary disk requirements.

[kriesel]

Originally Posted by preda

GpuOwl also can generate and verify proofs now!
And, most probably, the proofs are cross-compatible between the two software! And, the two programs (Prime95, GpuOwl) have distinct implementations written independently by different people, so it's highly unlikely to suffer from the same programmer error in the same way. They also run on different hardware, so it's highly unlikely to trigger the same hardware error in the same way. That is to say, such a cross-verification is quite strong. (I think, even the SHA3 library used by the two is different :)

For GpuOwl, the default proof power will be 8. This would result in temporary disk requirements of 3GB per 100M exponent. Users with plenty of disk would have an option to use power 9, doubling the temporary disk requirements.

Sounds like great progress on both the gpuowl and prime95 fronts.
The payoff to the project is so large for avoiding DCs of future PRP tests, that I'm in favor of whatever promotes rapid and widespread adoption of PRP-with-proof.
For 100M exponent, to recap in aligned tabular form:

A) power= 8,  3.2GB temporary disk space needed, proof file size 113MB, 413K squarings = 0.41% of a full DC, gpuowl default
B) power= 9,  6.4GB temporary disk space needed, proof file size 125MB, 239K squarings = 0.24% of a full DC
C) power=10, 12.8GB temporary disk space needed, proof file size 138MB, 182K squarings = 0.18% of a full DC.

And note the numbers rise about linearly with exponent, so 100MDigit is ~42.5GB each at power 10. And these files will add load to backup provisions.

Two instances / gpu x 5 gpus + prime95 1 worker means 11x, potentially 35GB to 141GB additional temporary space. How long do those temporaries hang around? On average, half present, ramping up from none to all during the course of a PRP-with-proof run? Do they linger afterward or does the program clear them out once the proof file is written? Do the proof files accumulate as multiple exponents are run?
141GB is a big ask on a system running with 270GB HD and compression already enabled, multiple applications installed per gpu, plus prime95 running on the 4 cpu cores. AFTER clearing a bunch of cudalucas save files off it this morning, it's now back up to 26GB free space, from 9GB.
Out of the 10 systems I checked just now, 2 of them would be marginal or worse.

What's a reasonable assumption for the percentage of systems where increased disk space requirements will deter or delay proof adoption; 1-5%?
Considering how many no-proof PRP tests may occur while people are getting around to either upgrading our systems with bigger storage, or doing severe housecleaning on what we've got, power 8's slight increase in verification effort seems like a bargain compared to the 100% PRP effort of a full blown PRP DC made necessary by a lack of a proof.

[ewmayer]

George, Mihai, is there a post detailing the 'official' bytewise format of the proof files, and the precise points of the PRP test they represent? It seems future GIMPS-software developers are going to be required to hew to this protocol, we should have a minimal set of links for them to peruse in order to be able to implement the scheme. ISTR coming across a detailed format-post earlier in the thread, but you know that sort of "it's right there in front of you, on page 77 of this thread over here"-ness is a pet peeve of mine when it comes to crucial how-to guides.

[Prime95]

Originally Posted by kriesel

How long do those temporaries hang around? On average, half present, ramping up from none to all during the course of a PRP-with-proof run?

I think Prime95 will default to allocating the disk space upfront so that the test does not get 89% complete and run into a disk full error. Better to get a disk full error upfront and drop the proof power down.

Do they linger afterward or does the program clear them out once the proof file is written? Do the proof files accumulate as multiple exponents are run?

Prime95 deletes the temporary disk space once the proof is generated. The proof file must hang around until it can be uploaded.

power 8's slight increase in verification effort seems like a bargain compared to the 100% PRP effort of a full blown PRP DC made necessary by a lack of a proof.

I'm convinced. Prime95 will default to 5GB max disk space per worker. That equates to proof power 8 until exponents hit 150M. Maybe I should make that 6GB default.

[Prime95]

Originally Posted by ewmayer

George, Mihai, is there a post detailing the 'official' bytewise format of the proof files, and the precise points of the PRP test they represent? It seems future GIMPS-software developers are going to be required to hew to this protocol, we should have a minimal set of links for them to peruse in order to be able to implement the scheme. ISTR coming across a detailed format-post earlier in the thread, but you know that sort of "it's right there in front of you, on page 77 of this thread over here"-ness is a pet peeve of mine when it comes to crucial how-to guides.

The last details were ironed out by email. Mihai started a wiki at gpuowl that we should update.

[Runtime Error]

Originally Posted by Prime95

So the question is what should be prime95's default setting? The optimize-to-the-max in me says default should be power=9. The minimal-impact-on-average-user in me says go with power=8 default.

I think the biggest impact on the average user is the disk space consumed. Power=8 will result in less proofs lost due to disk full errors. The cost is just 0.17% of a PRP test.

I'd vote set the default to consume the most disk space, but allow users to optionally decrease it. A while back, I accidentally didn't increase the default RAM allowance on a machine, and unfortunately it did a few P-1's with B1=B2 before I realized to change it. (Edit: maybe I don't fully understand the trade-off between disk full errors and verification time, it might make more sense to go with less disk space.)

Also, Ben Delo is doing over 53% percent of first time tests (by ghz/days), which kind of makes him the "average" user.

[kriesel]

Originally Posted by Runtime Error

Ben Delo is doing over 53% percent of first time tests (by ghz/days), which kind of makes him the "average" user.

Someone familiar with AWS or whatever he's running drive size please chime in.
Google drive space for Colaboratory accounts is 15GB total per related email account, to hold programs, worktodo, results, data, temporaries, logs, save files, proofs, trash, etc, for mprime and gpuowl instances.

[preda]

Originally Posted by ewmayer

George, Mihai, is there a post detailing the 'official' bytewise format of the proof files, and the precise points of the PRP test they represent? It seems future GIMPS-software developers are going to be required to hew to this protocol, we should have a minimal set of links for them to peruse in order to be able to implement the scheme. ISTR coming across a detailed format-post earlier in the thread, but you know that sort of "it's right there in front of you, on page 77 of this thread over here"-ness is a pet peeve of mine when it comes to crucial how-to guides.

An attempt at documenting the file format is here:
https://github.com/preda/gpuowl/wiki...roof-File-Spec

[preda]

Originally Posted by preda

An attempt at documenting the file format is here:
https://github.com/preda/gpuowl/wiki...roof-File-Spec

Also an example proof file is provided at the end of that page, which can be used for testing the proof processing with a new software.

https://github.com/preda/gpuowl/wiki...216091-9.proof

And a full-sized proof can be found here: https://drive.google.com/file/d/1vqp...ftY7nJG0l72p5J