mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Software

Reply
 
Thread Tools
Old 2018-06-08, 20:05   #34
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

667910 Posts
Default

Quote:
Originally Posted by preda View Post
I was thinking the header, which could be the first line if single-line, or the first few lines otherwise, could be a textual format. So the exponent would be a string of ASCII digits, part of the header.

After the header would come the "bits" which would be binary to save space and simplify save/load.

This would allow easy human inspection of the savefile.
High human readability sounds great.

Bits: binary saves space, ASCII hexadecimal makes it readable and displayable in normal editors. (I'd rather not binhex as a density compromise; the readability is poor.) Since these would be used much less often than n-interim-savefiles-for-every-exponent, transmission size and time is less of a concern, and the entire file or small set of them can be put in a compressed archive.

Getting close. This format needs a name. "Mersenne neutral file format"?

Does something like this exist elsewhere in the number theory community? (Could we avoid some reinventing the wheel, or copy some good design features?)

Do you see software supporting both import and export?

Export would be handy for debugging on occasion, either in code development or in use.
For example, if a variant were available for P-1, in CUDAPm1, I'd be using it right now to try to understand what's going on with some bad runs on large exponents. Some problems due to limited memory are transferable from a gpu to another gpu model with much more memory, contained in the save file from the first gpu. The files are unreadable binary. Run parameters chosen on the first gpu are unnecessarily restrictive on the second. And some runs fail in the same way on the larger memory gpu as the first.

Import maybe should require some security key or authentication.
kriesel is offline   Reply With Quote
Old 2018-06-08, 20:47   #35
preda
 
preda's Avatar
 
"Mihai Preda"
Apr 2015

2·691 Posts
Default

Quote:
Originally Posted by kriesel View Post
Do you see software supporting both import and export?
The format would be worth only if there's buy-in from more than 1 developer.

If I consider it a good format, I would just use it for gpuOwl completely, as its only format. (thus, no separate import/export would be needed).

One piece of info that I put now in owl's files, that's not included in the portable format, is "error count", which was requested by George. Maybe the format would have a set of optional "comments" that are not essential, but are preserved by the software that does not understand them.

Last fiddled with by preda on 2018-06-08 at 20:47
preda is offline   Reply With Quote
Old 2018-06-08, 21:54   #36
ewmayer
2ω=0
 
ewmayer's Avatar
 
Sep 2002
República de California

22×5×587 Posts
Default

Quote:
Originally Posted by kriesel View Post
"Serge Batalov also verified it using Ernst Mayer's MLucas software on two Intel Xeon 18-core Amazon EC2 servers in 3.5 days." https://www.mersenne.org/primes/?press=M74207281
Not first and second half of iterations, but duplicate full runs?

I'd expect the n-parallel approach to work well with prime95/mprime since it is very common.
Yes, Serge's (and more recently, Anders') dual verify runs at different FFT lengths were both end-to-end.

Here is the current Mlucas file format:

o 1 byte for test type, numerically, i.e. 256 possible values, mapped to an internal table;
o 1 byte for modulus type, currently only Mersennes and Fermats supported;
o 8 bytes for iteration count of the residue stored in the file;
o ceiling(p/8) bytes for the residue R - i.e. maximally byte-compact, endian-and-FFT-length-of-run-independent;
o 8 bytes for Res64 = R (mod 2^64), which should match the leading 8 full-residue bytes in the above bytewise form);
o 5 bytes for R (mod 2^35-1);
o 5 bytes for R (mod 2^36-1) [these last 2 a.k.a. the Selfridge-Hurwitz residues, based on those guy's Fermat-number work, using a 36-bit-hardware-integer machine; SH also used R (mod 2^36), but that is just the low 36 bits of GIMPS' Res64];

After reading R, I directly compute the two SH residues and compare to the above file-stored checksums; this gives me an md5/sha1-style integrity check of the whole residue R, which the Res64 does not.

For v18, I am adding several new fields:
o 3 bytes for FFT-length-in-Kdoubles which the code was using at time of savefile write. This is so that if the code switches to a larger-than-default FFT length mid-run based on ROE behavior for the exponent in question, it will immediately resume using the larger FFT length on restart-from-interrupt, rather than resuming using the smaller default FFT length as the current release does.
o 8 bytes for circular-shift to apply to the (unshifted) residue read from the file. I include the shift-count-at-iteration-of-savefile-write because [a] the code will choose a random shift count at run-start time (i.e. since this is not specified by the Primenet server, it cannot be read from the worktodo file), and [b] it saves the need for taking an initial-shift value s from the savefile and computing s * 2^iter (mod p). I remove the shift from R prior to the savefile write, so in fact there's really no need to store s to the file (i.e. I could resume-from-interrupt using an entirely different random shift value, applied to R after reading it from the savefile), but for aesthetic reasons I like the idea of doing the whole run based on a single initial value of s, rather than as-many-values-of-s-as-there-were-run-interrupts.

Last fiddled with by ewmayer on 2019-12-06 at 02:26
ewmayer is offline   Reply With Quote
Old 2018-06-08, 23:11   #37
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

667910 Posts
Default

Quote:
Originally Posted by ewmayer View Post
Yes, Serge's (and more recently, Anders') dual verify runs at different FFT lengths were both end-to-end.

Here is the current Mlucas file format:

o 1 byte for test type, numerically, i.e. 256 possible values, mapped to an internal table;
o 1 byte for modulus type, currently only Mersennes and Fermats supported;
o 8 bytes for iteration count of the residue stored in the file;
o ceiling(p/8) bytes for the residue R - i.e. maximally byte-compact, endian-and-FFT-length-of-run-independent;
o 8 bytes for Res64 = R (mod 2^64), which should match the leading 8 full-residue bytes in the above bytewise form);
o 5 bytes for R (mod 2^35-1);
o 5 bytes for R (mod 2^36-1) [these last 2 a.k.a. the Selfridge-Hurwitz residues, based on those guy's Fermat-number work, using a 36-bit-hardware-integer machine; SH also used R (mod 2^36), but that is just the low 36 bits of GIMPS' Res64];

After reading R, I directly compute the two SH residues and compare to the above file-stored checksums; this gives me an md5/sha1-style integrity check of the whole residue R, which the Res64 does not.

For v18, I am adding several new fields:
o 4 bytes for FFT-length-in-K which the code was using at time of savefile write. This is so that if the code switches to a larger-than-default FFT length mid-run based on ROE behavior for the exponent in question, it will immediately resume using the larger FFT length on restart-from-interrupt, rather than resuming using the smaller default FFT length as the current release does.
o 8 bytes for circular-shift to apply to the (unshifted) residue read from the file. I include the shift-count-at-iteration-of-savefile-write because [a] the code will choose a random shift count at run-start time (i.e. since this is not specified by the Primenet server, it cannot be read from the worktodo file), and [b] it saves the need for taking an initial-shift value s from the savefile and computing s * 2^iter (mod p). I remove the shift from R prior to the savefile write, so in fact there's really no need to store s to the file (i.e. I could resume-from-interrupt using an entirely different random shift value, applied to R after reading it from the savefile), but for aesthetic reasons I like the idea of doing the whole run based on a single initial value of s, rather than as-many-values-of-s-as-there-were-run-interrupts.

That's such a clear and well organized format description, I think I'd like to quote you in my reference material threads.
kriesel is offline   Reply With Quote
Old 2018-06-09, 00:41   #38
ewmayer
2ω=0
 
ewmayer's Avatar
 
Sep 2002
República de California

22×5×587 Posts
Default

Quote:
Originally Posted by kriesel View Post
That's such a clear and well organized format description, I think I'd like to quote you in my reference material threads.
Be my guest - note I make no claim to my format being as complete as is desirable, for example it lacks any kind of errors-during-the-run encoding - but I think most of the key features (byte-compact, endian-and-FFT-length-independent, some kind of whole-residue checksum included) should be requirements for any common savefile format which may end up getting agreed on.
ewmayer is offline   Reply With Quote
Old 2018-06-09, 13:59   #39
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

6,679 Posts
Default

Quote:
Originally Posted by ewmayer View Post
o 8 bytes for circular-shift to apply to the (unshifted) residue read from the file. I include the shift-count-at-iteration-of-savefile-write because [a] the code will choose a random shift count at run-start time (i.e. since this is not specified by the Primenet server, it cannot be read from the worktodo file), and [b] it saves the need for taking an initial-shift value s from the savefile and computing s * 2^iter (mod p). I remove the shift from R prior to the savefile write, so in fact there's really no need to store s to the file (i.e. I could resume-from-interrupt using an entirely different random shift value, applied to R after reading it from the savefile), but for aesthetic reasons I like the idea of doing the whole run based on a single initial value of s, rather than as-many-values-of-s-as-there-were-run-interrupts.
It seems to me that recording the circular shift value provides much greater reproducibility from run to run. That could be very useful if debugging some shift-dependent case.
kriesel is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
How to know if you found a mersenne prime shaytan1986 Information & Answers 1 2015-07-26 03:38
Mersenne Prime Found? No. houding PrimeNet 4 2014-09-21 15:32
How to know if you found a mersenne prime. Sutton Shin PrimeNet 7 2012-10-02 05:57
new mersenne prime found unregistered Data 25 2005-01-07 19:26
Strategies when a Mersenne prime is found GP2 Lounge 30 2003-12-02 19:45

All times are UTC. The time now is 21:05.


Fri Aug 19 21:05:45 UTC 2022 up 1 day, 18:34, 0 users, load averages: 3.04, 2.27, 1.82

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

≠ ± ∓ ÷ × · − √ ‰ ⊗ ⊕ ⊖ ⊘ ⊙ ≤ ≥ ≦ ≧ ≨ ≩ ≺ ≻ ≼ ≽ ⊏ ⊐ ⊑ ⊒ ² ³ °
∠ ∟ ° ≅ ~ ‖ ⟂ ⫛
≡ ≜ ≈ ∝ ∞ ≪ ≫ ⌊⌋ ⌈⌉ ∘ ∏ ∐ ∑ ∧ ∨ ∩ ∪ ⨀ ⊕ ⊗ 𝖕 𝖖 𝖗 ⊲ ⊳
∅ ∖ ∁ ↦ ↣ ∩ ∪ ⊆ ⊂ ⊄ ⊊ ⊇ ⊃ ⊅ ⊋ ⊖ ∈ ∉ ∋ ∌ ℕ ℤ ℚ ℝ ℂ ℵ ℶ ℷ ℸ 𝓟
¬ ∨ ∧ ⊕ → ← ⇒ ⇐ ⇔ ∀ ∃ ∄ ∴ ∵ ⊤ ⊥ ⊢ ⊨ ⫤ ⊣ … ⋯ ⋮ ⋰ ⋱
∫ ∬ ∭ ∮ ∯ ∰ ∇ ∆ δ ∂ ℱ ℒ ℓ
𝛢𝛼 𝛣𝛽 𝛤𝛾 𝛥𝛿 𝛦𝜀𝜖 𝛧𝜁 𝛨𝜂 𝛩𝜃𝜗 𝛪𝜄 𝛫𝜅 𝛬𝜆 𝛭𝜇 𝛮𝜈 𝛯𝜉 𝛰𝜊 𝛱𝜋 𝛲𝜌 𝛴𝜎𝜍 𝛵𝜏 𝛶𝜐 𝛷𝜙𝜑 𝛸𝜒 𝛹𝜓 𝛺𝜔