Getting reliable LL from unreliable hardware

[preda]

A bit more about the error-checking implementation:

- data is validated before saving to save-files, and again upon loading from savefile. This way a "good" rollback point is always present in RAM.

- because the validation is done GPU-side, the residue+check data is always round-tripped to the CPU and back to the GPU before validation. This ensures that no errors on reading GPU->CPU can pass undetected. (i.e. what is effectively validated is the CPU-side data, not just the GPU-side which would be weaker).

- the time cost of a validation run is 1000 iterations (on the order of 2 seconds).

[preda]

Originally Posted by ewmayer

My code has been computing/outputting the well-known Selfridge-Hurwitz residues (mod 2^36,2^35-1,2^36-1) to the exponent-associated results file (pXXXXX.stat, stat short for 'status') for many years now. Those are standard in reporting results of Pepin tests of Fermat numbers.

Since 2^36 is obviously just the low 9 hexits of the GIMPS-standard Res64, that gives 135 bits of 'residueness' (64+35+36).

OK, sounds better than plain-128, and has a tradition. I'll think about adding that as well. Did you decide on a particular textual representation for those 135bits?

I'm curious though, where does 36 (&35) come from? (why not 32&31, or 64&63).

[kriesel]

Originally Posted by Mark Rose

That may take a while. Many people are still using version 27 because "28 makes my overclocked computer unstable". Many are still using 28, because "29 doesn't load the cores up the way I want for my stress test".

And there are people who just haven't had a need to upgrade. So there will be new LL tests to DC for many years to come. Prime95 can always handle those, I suppose.

For that matter, George has not yet announced prime95 V29.2 as ready for general use.
Nor that it implements PRP3. (I think it doesn't, because of that.)
https://www.mersenne.org/download/ still says 28.10 at the top.
The individual download links say 28.10, down to 26.6, depending on platform.

http://mersenneforum.org/showthread.php?t=22377&page=6 shows an entry by George 8/27 indicating he's making code changes relating to the user interface. He listens to users. It's great.

[preda]

Originally Posted by Madpoo

{
	"result": {
		"exponent": "74207281",
		"worktype": "LL",
		"residue64": "0",
		"username": "madpoo",
		"cpuname": "bigboy"
		"program": "Prime95 v29.2 x64"
		"assignmentid": "12345"
	}
}

I'm starting to look into generating this. One important field missing is the outcome prime/composite (for LL or, probable/composite for PRP). [this information is not, strictly speaking, contained in the res64].

Should this be a boolean "isprime" : true|false, using true for a PRP?

Other small points: I suppose the key names are case-sensitive. Do you propose all-lowercase, merged-words (like "username")?

For the PRP-3, should I use "worktype" : "P3" (vs. "LL").

For the residue64: is it OK if I add 2 check-digits at the end (thus 18 hex-digits instead of 16)? These check-digits can be ignored if verification is not done.

Another point, right now in text I prefix the residue with "P3-" to differentiate from LL. Should this prefix be kept in the residue64?
e.g.
"residue64" : "001122...ff" vs.
"residue64" : "P3-001122...ffcc" (note: prefix, and check digits).

[kriesel]

Originally Posted by preda

I'm starting to look into generating this. One important field missing is the outcome prime/composite (for LL or, probable/composite for PRP). [this information is not, strictly speaking, contained in the res64].

Should this be a boolean "isprime" : true|false, using true for a PRP?

Other small points: I suppose the key names are case-sensitive. Do you propose all-lowercase, merged-words (like "username")?

For the PRP-3, should I use "worktype" : "P3" (vs. "LL").

For the residue64: is it OK if I add 2 check-digits at the end (thus 18 hex-digits instead of 16)? These check-digits can be ignored if verification is not done.

Another point, right now in text I prefix the residue with "P3-" to differentiate from LL. Should this prefix be kept in the residue64?
e.g.
"residue64" : "001122...ff" vs.
"residue64" : "P3-001122...ffcc" (note: prefix, and check digits).

Or P3-00...ff-cc, more reliable for human readers, or other possibilities previously mentioned.

CUDALucas uses C for composite or P for prime, very concise.

[kriesel]

This was in response to one of Madpoo's earlier posts: Well organized!

(Delayed while I collected an ever increasing variety of sample outputs from various programs)

The result items might read and parse easier if program and version led the parade. Which they do not now, in all cases. On the other hand, skimming for exponent down the left margin can be quick.

Goals seem to me to include:
Clear/unambiguous, concise, sufficient, human-readable (for laymen too), support traceability to hardware.
Backward compatibility is a plus. Retaining error type counts for reliability of the data seems like a plus too; higher counts could merit double-check priority of specific results. (Could that expand beyond prime95?)

What's been done till now for result records generally fits on a single line per result (date/time sometimes is a preceding line).
Some users have expressed interest in program output not extending all the way across their screen or more.

The status quo seems to me like a single-line version of JSON with a lot of punctuation stripped out and perhaps exclusion of embedded spaces.
(Compare to JSON and YAML examples at https://en.wikipedia.org/wiki/JSON)

P-1 bounds expands to at least B1, B2, and e labeled values.

Some assembled sample record styles from the status quo, for the convenience of those who run different software follow. (All have bogus characters injected, some in multiple fields, so they are not valid, just illustrative of layout)

Avoiding unnecessary differences in style would be a good thing.

CPUS
----

Prime95 TF result
subtype no factor found:
[Mon Aug 21 23:25:59 2017]
UID: Kriesel/condorette, M152766327 no factor from 2^70 to 2^71, Wf4: BA18587D, AID: 0208C74751642C84B34FAA5D106C2D52
subtype factor found:
(no recent samples here)

Prime95 P-1 result
subtype no factor found:
[Thu Jan 27 00:57:28 2011]
UID: Kriesel/HP, M42178167 completed P-1, B1=535000, B2=9496250, Wd4: DEB7579A, AID: E50C33B37B3778BA25C349256FAD27F7
subtype factor found:
(none handy to share)

Prime95 ECM result
subtype no factor:
[Tue Jul 11 04:11:21 2017]
UID: Kriesel/x, M10351953 completed 50 ECM curves, B1=50000, B2=5000000, We1: 27C5B75A, AID: 07C43F83NY322573F26245A03AB5EE32
subtype ECM factor found:
(sorry, no current sample here)

Prime95 LL result
subtype not prime
[Mon Jan 03 02:56:20 2011]
UID: Kriesel/HP, M234745067 is not prime. Res64: 46ECF1E803DDA820. Wd4: E986021D,15615940,00000000, AID: 915C453965A8DF2B1ADE0FBB36CD0787
subtype prime, no AID
[Tue Aug 22 00:37:26 2017]
UID: Kriesel/condorette, M6972593 is prime! We4: A4277B69,00000000

Prime95 PRP3 result
(tbd)

Mlucas?

recast into single-line JSON, a Prime95 format result does not fit on one line even at 1600x900 resolution in the smallest Windows FixedSys font:
UID: "Kriesel/system-name", exponent: "M234745067", result: "is not prime", Res64: "46ECF1E803DDA820" Wd4: "E986021D,15615940,00000000", AID: "915C453965A8DF2B1ADE0FBB36CD0787", MD5: "0123456789ABCDEF0123456789ABCDEF"

UID: "Kriesel/system-name", expo
nent: "M234745067", result: "is
not prime", Res64: "46ECF1E803DD
A820" Wd4: "E986021D,15615940,00
000000", AID: "915C453965A8DF2B1
ADE0FBB36CD0787",
MD5: "0123456789ABCDEF0123456789ABCDEF"


GPUS
----
Mfaktc TF
subtype no factor result:
[Mon Aug 21 06:17:13 2017]
UID: kriesel/Lenovo-GTX480-1, no factor for M156362247 from 2^71 to 2^72 [mfaktc 0.20 barrett76_mul32_gs]
subtype factor found result barret76:
[Sat Aug 19 02:30:50 2017]
UID: kriesel/Lenovo-GTX480-1, M139528567 has a factor: 8147929265992407949879 [TF:72:73:mfaktc 0.20 barrett76_mul32_gs]
subtype factor found result 75bit_mul32:
[Mon Aug 28 19:25:25 2017]
UID: kriesel/KEN-Lenovo-GTX480-1, M290801377 has a factor: 1436165993277492383 [TF:48:69:mfaktc 0.20 75bit_mul32]
subtype factor count result:
[Sun Aug 27 02:08:39 2017]
UID: kriesel/KEN-Lenovo-GTX480-1, found 1 factor for M143985481 from 2^75 to 2^76 [mfaktc 0.20 barrett76_mul32_gs]
[Mon Aug 28 19:33:23 2017]
UID: kriesel/KEN-Lenovo-GTX480-1, found 4 factors for M290801377 from 2^48 to 2^69 [mfaktc 0.20 75bit_mul32]

CUDAPm1 P-1
subtype no factor found result:
M82468099 found no factor (P-1, B1=725000, B2=16131250, e=2, n=4704K, aid=CDBC76ACA9707874630FC9D4B98096CB CUDAPm1 v0.20)
subtype factor found result:
M50991781 has a factor: 4392938042656798431087689 (P-1, B1=430000, B2=5000000, e=2, n=2688K CUDAPm1 v0.20)

CUDALucas LL test result
subtype composite, AID in worktodo entry
M( 53161253 )C, 0x06c9ff03bd581bba, offset = 4119, n = 2520K, CUDALucas v2.05.1, AID: 03B180C2BA881B29CF94BF392CA7AF3B
subtype, prime, no AID in worktodo entry
M( 6972593 )P, offset = 3488212, n = 360K, CUDALucas v2.05.1

CUDALucas PRP3 test result (CUDAP3?)
(tbd)

(add gpuowl, cllucas, etc. to the collection)


Though it is not something to do lightly, there is certainly precedent for change in formats. This is what some Prime95 result formats looked like in mid 1996, and 2000, run from an assigned range requested and received by email, and downloaded database initially, and a local prime server by 2000:
572023 no factor to 2^52, VC: CB23FBEE
572023 is not prime. Res64: 374329CF,917F8DC7. VC: 26ADB834
573031 no factor from 2^0 to 2^48, VCfix: 3C723FD4
141023 is not prime. Res64: 39AF8553,A6E37986. VC: B689C6B2
745167 has a factor: 648906361124241
In 2000:
UID: S00005, M12005479 completed P-1, B1=180000, B2=180000, WW5: 36B0451D
[Mon Jun 05 18:02:35 2000]
UID: S00005, M12005489 has a factor: 12167593809507929
[Tue May 30 23:25:13 2000]
P-1 found a factor in stage #1, B1=180000, B2=180000.
UID: S00005, M120534921 has a factor: 204515199879938766471
[Wed Sep 13 03:59:18 2000]
ECM found a factor in curve #1, stage #1
Sigma=7970827368464435, B1=250000, B2=25000000.
M25909 has a factor: 718040331963789452016551822407

[ewmayer]

Originally Posted by preda

OK, sounds better than plain-128, and has a tradition. I'll think about adding that as well. Did you decide on a particular textual representation for those 135bits?

I'm curious though, where does 36 (&35) come from? (why not 32&31, or 64&63).

My Mlucas homepage has several examples, here is one - note the SH residues are printed in base-10, again because tradition:

100 iterations of M3888517 with FFT length 196608 = 192 K
Res64: 579D593FCE0707B2. AvgMaxErr = 0.260239955. MaxErr = 0.343750000. Program: E14.1
Res mod 2^36     =          67881076658
Res mod 2^35 - 1 =          21674900403
Res mod 2^36 - 1 =          42893438228

No idea why 35/36-bit - you;d have to ask Messrs J. Selfridge and A. Hurwitz, but those worthy gentleman are alas no longer with us. (I did have the pleasure of having met John S. at one of the West Coast Number Theory confabs at the Asilomar conf. center in the late 90s, and having drinks with him and some of the other attendees.)

[CRGreathouse]

Maybe because those give a good spread of small prime factors?

I seem to remember something in Knuth along these lines.

[ewmayer]

Originally Posted by CRGreathouse

Maybe because those give a good spread of small prime factors?

I seem to remember something in Knuth along these lines.

That sounds like a reasonable criterion, in addition to the obvious relative-primality of the two moduli. Not sure if the bit-ness had anything to do with the hardware available at the time.

[fivemack]

Originally Posted by ewmayer

That sounds like a reasonable criterion, in addition to the obvious relative-primality of the two moduli. Not sure if the bit-ness had anything to do with the hardware available at the time.

Yes, the original tests (eg M4423) were conducted on IBM hardware with 36-bit words (why 36-bit? because they wanted to support ten-digit decimal arithmetic as well as hex, so you need a multiple of 4 greater than 33.2ish)

[preda]

OK, how does this JSON sound:

{ "exponent": 25000000, "worktype": "P3", "status": "C", "res64": "fffffffffffffffd-dd", "user": "foo", "cpu": "bar", "program": "gpuowl v1.0", "aid": "FDEF0000" }