2020 Prime95 observations, issues, and suggestions

[rainchill]

Hey Folks - just a little list of things I have been thinking about to start the new year with.

1. Can the Prime95 version and build be displayed in main window title bar by default? This will help determine what version is in use on screenshots on various web forums and sites that use prime95 for testing.
2. Add ETA to worker title window. So it would look like: Worker #X - X Cores - X.XX% of PRP XXXXXXXX ms/iter: x.xx, ETA: X days
3. Add separate options for how often to output status to worker window and worker title.
4. Option to output to worker window and title bar based on percentage, not iterations completed.
5. Option to hide Jacobi/Gerbicz error checking output if no errors are noted.
6. Are benchmarks every night really necessary?
7. Since benchmarks are collected so frequently, do warnings need to be added and/or emailed to users if it is noted their system is running significantly slower ms/iter than other similar systems?
8. I noticed when I upgraded to a i9-9900k that Prime95 can now run a test on multiple cores. Assuming this can be more efficient than 1x1, can GUI options be added to configure it? So instead of the current v29.8b6 Test > Work windows > "Number of worker windows to run" field it would be rejigged to something like number of tests to run and cores per test. This will be beneficial especially as core counts will be increasing in the foreseeable future.
9. Is Prime95 NUMA aware so it will keep tests running on a specific socket only in the local RAM for that node?
10. Add a function for Prime95 to monitor CPU temp and options to back off work load if it gets too high. This could be useful if folks notice instability at or above certain temps. I have sometimes noted this to be a problem in the summer especially when I am on vacation with the AC not running as much.
11. Is there a plan for tests past double checking? Only pointing out that the DB needs to be prepared for it. In case at some point in the future someone wants to do it or issues are found calling into question the validity of past tests.
12. Is there a plan for what to do if it is discovered that a bug invalidates all tests beginning at a certain date or software release and how this would be accounted for in the DB and to re-check those exponents with verified fixed clients?
13. Primenet stop automatically assigning first time tests to clients running older builds.
14. Have separate shell and execution core. When new code is ready it can automatically download the new execution code similar to how folding@home works. And/or possibly join boinc.
15. Could estimated future dates be added to the milestones page? For instance, when it is expected all exponents below 50 million digits, 100 million, 1 billion etc…? This may seem whimsical, but it would be cool to see how long it is estimated to reach future goals. I currently guestimate it will take more than 50 years at current rates to test all exponents below 100 million digits.
16. At what point do we say "thanks, but no thanks." to testing on old/slower hardware? Someone may think it is cool to fire up Linux and Prime95 on a 1000 old 386 systems they found in storage - but really, that can't practically add to the project. I propose not assigning work to new users/systems that cannot be completed in 6-12 months for LL/PRP test and 18 months for 100M digit tests.
17. Add a timebomb to prevent Prime95 from accepting work assignments from the server if the client build is more than 3 years from the date it was compiled. This will prevent old builds from running forever, encourage folks to upgrade, and help prevent old potentially buggy or unoptimized code from staying out there too long. Older build would only work for testing and benchmarks.
18. Sometime around 2015/2016 the ETA calculation for completion of testing became unreliable. This has been mentioned in various other discussions.
19. Add GPU testing to the client.
20. What considerations have been made to continue the project past George's and everyone else's lifetimes?

[ixfd64]

6. Are benchmarks every night really necessary?

This can be disabled by setting AutoBench=0 in your prime.txt file.

9. Is Prime95 NUMA aware so it will keep tests running on a specific socket only in the local RAM for that node?

No as of September 2017: https://mersenneforum.org/showpost.p...06&postcount=2

10. Add a function for Prime95 to monitor CPU temp and options to back off work load if it gets too high.

I like this idea.

14. Have separate shell and execution core. When new code is ready it can automatically download the new execution code similar to how folding@home works.

This isn't a bad idea, but the potential security issues will need to be resolved first. However, it wouldn't hurt to notify users when new versions are available.

Now for my own suggestions:

1. Finer control of individual workers: https://mersenneforum.org/showthread.php?t=14671
2. Remove the broken help feature: https://mersenneforum.org/showthread.php?t=24670
3. Ability to dump the status via the command line: https://mersenneforum.org/showthread.php?t=19818

[kriesel]

https://www.mersenneforum.org/showthread.php?t=25066

[kriesel]

Originally Posted by rainchill

Hey Folks - just a little list of things I have been thinking about to start the new year with.

1. Can the Prime95 version and build be displayed in main window title bar by default? This will help determine what version is in use on screenshots on various web forums and sites that use prime95 for testing.

And it enables showing version and status subwindow in the same screen shot easily.

2. Add ETA to worker title window. So it would look like: Worker #X - X Cores - X.XX% of PRP XXXXXXXX ms/iter: x.xx, ETA: X days

and PRP residue type.

3. Add separate options for how often to output status to worker window and worker title.
4. Option to output to worker window and title bar based on percentage, not iterations completed.

Meh. Diminishing returns, increasing complexity, and diversion of George's time and talent from other things that may be more beneficial.

5. Option to hide Jacobi/Gerbicz error checking output if no errors are noted.

No. These show the checking is occurring.

6. Are benchmarks every night really necessary?

My impression is they taper off after an instance has been running a while. Benchmarking for the purpose of the program choosing the right fft code for a given fft length starts from zero for that installation, and over time characterizes it.

7. Since benchmarks are collected so frequently, do warnings need to be added and/or emailed to users if it is noted their system is running significantly slower ms/iter than other similar systems?

No.

8. I noticed when I upgraded to a i9-9900k that Prime95 can now run a test on multiple cores. Assuming this can be more efficient than 1x1, can GUI options be added to configure it? So instead of the current v29.8b6 Test > Work windows > "Number of worker windows to run" field it would be rejigged to something like number of tests to run and cores per test. This will be beneficial especially as core counts will be increasing in the foreseeable future.

The assumption is valid. I don't see what you think is needed that is not already available. See also https://www.mersenneforum.org/showpo...18&postcount=4 and the post following it.

9. Is Prime95 NUMA aware so it will keep tests running on a specific socket only in the local RAM for that node?

It allows the user to do what they want, and try straddling multiple multicore cpu/cache packages with a worker. This makes available both maximum aggregate throughput configurations, and minimum-latency configurations.

10. Add a function for Prime95 to monitor CPU temp and options to back off work load if it gets too high. This could be useful if folks notice instability at or above certain temps. I have sometimes noted this to be a problem in the summer especially when I am on vacation with the AC not running as much.

This would be useful on my ancient i3-370M which is thermal-dissipation-challenged and experiences variable cpu loads from other processes.

11. Is there a plan for tests past double checking? Only pointing out that the DB needs to be prepared for it. In case at some point in the future someone wants to do it or issues are found calling into question the validity of past tests.
12. Is there a plan for what to do if it is discovered that a bug invalidates all tests beginning at a certain date or software release and how this would be accounted for in the DB and to re-check those exponents with verified fixed clients?

Triple checks or higher when DC does not match first test are manually managed and expected to become more rare as PRP operation increases and LL operation diminishes. Such bugs as the v17 shift bug were also handled manually. It would be very difficult to automate PrimeNet finding and reissuing assignments for unknown bugs. Note the DB already handles and contains exponents with many tests reported. See for example https://www.mersenne.org/report_expo...0000300&full=1

13. Primenet stop automatically assigning first time tests to clients running older builds.

I suspect this is already the case in a sense. Try running up an instance before V24.12 which can't do above 79.3M and request a first-test. Sufficiently older versions are not able to talk PrimeNet 5.0 API. And see also Error invalid work type in the PrimeNet API.

14. Have separate shell and execution core. When new code is ready it can automatically download the new execution code similar to how folding@home works. And/or possibly join boinc.

No. No autoupdates. That will break approval for some users for use on company-owned machines, for example. And user acceptance.

15. Could estimated future dates be added to the milestones page? For instance, when it is expected all exponents below 50 million digits, 100 million, 1 billion etc…? This may seem whimsical, but it would be cool to see how long it is estimated to reach future goals. I currently guesstimate it will take more than 50 years at current rates to test all exponents below 100 million digits.

The milestones pages are what DID happen. Estimates are what MIGHT happen, given some only weakly defensible assumptions. The milestones page used to contain predictions for the very next milestone, but they were shown to be unreliable. Network behavior is unpredictable, and so is human behavior, and milestones are the product of both. But see https://www.mersenneforum.org/showpo...5&postcount=11 for some wild extrapolations.

16. At what point do we say "thanks, but no thanks." to testing on old/slower hardware? Someone may think it is cool to fire up Linux and Prime95 on a 1000 old 386 systems they found in storage - but really, that can't practically add to the project. I propose not assigning work to new users/systems that cannot be completed in 6-12 months for LL/PRP test and 18 months for 100M digit tests.

I believe something similar is already in place. Prime95 warns about run times exceeding a year, for example. But if someone really wants to run a 2-year-long PRP with GEC, why not? And how would you hope to implement such limits for manual assignments, where the hardware and even the application are unknown to the PrimeNet server? It can't know whether a manual assignment is going on all cores of a multi-Xeon in mprime, one core in mlucas, a Radeon VII in gpuowl, or a Quadro 2000 in an old version of CUDALucas, or a TF assignment on an RTX2080Ti with 3500 GhzD/day throughput, an IGP model with 20 or 5 GhzD/day throughput, or an NVS295 with 1.74 GhzD/day throughput. I have a well maintained i3-370M that is 10 years old, and older gear than that. I have a 12 year old core 2 system that hosts a gpu or two. All manual assignments per user account are lumped into one pile on anonymous hardware of unknown type and application of unkown type and version. Manual results MAY contain hardware and application information.

17. Add a timebomb to prevent Prime95 from accepting work assignments from the server if the client build is more than 3 years from the date it was compiled. This will prevent old builds from running forever, encourage folks to upgrade, and help prevent old potentially buggy or unoptimized code from staying out there too long. Older build would only work for testing and benchmarks.

There are probably some people (curtisc?) who set up a system and leave it run as is for its lifetime. This is probably why there is still so much LL first test going on, more than a year after PRP/GEC was rolled out. What you propose is likely to idle a lot of machines run this way. Three years would be too short in many cases. Five might be better.

18. Sometime around 2015/2016 the ETA calculation for completion of testing became unreliable. This has been mentioned in various other discussions.

Status output is based on average progress, ETA in the worker window is based on recent ms/iter timings. This is a feature, not a bug.

19. Add GPU testing to the client.

No. There is only one George. There are several gpu application authors and maintainers/helpers. The variable usages, ini file directives, etc among the various applications are in conflict. Input and output formats differ. Etc. Even merging 2 gpu applications is a major undertaking. I've looked at the code. It would also complicate build and install. The PrimeNet API would require updating documentation, and extension. See https://www.mersenneforum.org/showthread.php?t=23925

20. What considerations have been made to continue the project past George's and everyone else's lifetimes?

I've raised the question of continuity planning before, but it got no response. I suppose the forum and source code repositories help carry the knowledge forward. The board could select a new member to replace anyone no longer available to serve. Finding people with the talent of George, Mihai, Ernst, et al willing to commit large amounts of their time to further software development is not easy. They are rare. There are about 100,000 people who signed up to do GIMPS computations, ~35,000 that completed at least one, 6000+ who have been active recently, maybe 100 or more that have compiled anything, and around 18 by my count that have done significant programming useful for the GIMPS effort, among all the relevant cpu and gpu applications. See also https://www.mersenneforum.org/showpo...46&postcount=6

[Uncwilly]

Originally Posted by kriesel

My impression is they taper off after an instance has been running a while.

I believe that to be the case too.

Originally Posted by kriesel

The assumption is valid. I don't see what you think is needed that is not already available.

Yeah, I don't quite understand what is lacking in that regard.

Originally Posted by kriesel

This would be useful on my ancient i3-370M which is thermal-issipation-challenged and experiences variable cpu loads from other processes.

No way no how UNLESS the user and PrimeNet are notified that this is happening. Chris will tell you that he uses Prime95 to check machines for health and stability. If Prime95 started to silently lower its demands, that would be bad. Also, if one is using Prime95 to run a burn-in on a machine, this would defeat the purpose.

Originally Posted by kriesel

Triple checks or higher when DC does not match first test are manually managed and expected to become more rare as PRP operation increases and LL operation diminishes. Such bugs as the v17 shift bug were also handled manually. It would be very difficult to automate PrimeNet finding and reissuing assignments for unknown bugs. Note the DB already handles and contains exponents with many tests reported. See for example https://www.mersenne.org/report_expo...0000300&full=1

Also, people are using MLucas and gpuowl, etc. to check exponents. There are some projects to triple check exponents in representative batches with non-Prime95 programs.

Originally Posted by kriesel

I suspect this is already the case in a sense. Try running up an instance before V24.12 which can't do above 79.3M and request a first-test.

Further as software gets older, the level of productivity, relative to the norm will decline. The assignment rules will push the exponents assigned for new request for old machine further in front of the trailing edge.

Originally Posted by kriesel

No. No autoupdates. That will break approval for some users for use on company-owned machines, for example. And user acceptance.

I totally agree with kriesel. In situations like chalsall's, he manages the software. He needs to be free to decide when to change it. Also, some older machines don't benefit from newer versions of the program. Back in the day, on some older machines, the newer version could be slower.

Originally Posted by kriesel

The milestones page used to contain predictions for the very next milestone, but they were shown to be unreliable. Network behavior is unpredictable, and so is human behavior, and milestones are the product of both.

When there were automatic predictions, that had a tendency to encourage poaching, so that we made those projected milestones. This is why they were removed (in part).

Originally Posted by kriesel

I believe something similar is already in place.

Again, the assignment rules that are in place help prevent this problem.

Originally Posted by kriesel

There are probably some people (curtisc?) who set up a system and leave it run as is for its lifetime. This is probably why there is still so much LL first test going on, more than a year after PRP/GEC was rolled out. What you propose is likely to idle a lot of machines run this way. Three years would be too short in many cases. Five might be better.

This is the same as an auto update. NO. Like curtisc, there are others that are fire and forget. If I deploy Prime95 on granny's computer, I can't expect her to keep the code up to date, nor waste my precious moments with her worrying about version numbers.

Originally Posted by kriesel

No. There is only one George. There are several gpu application authors and maintainers/helpers. The variable usages, ini file directives, etc among the various applications are in conflict. Input and output formats differ. Etc.

There have been discussions and baby steps made toward standardized outputs for some apps. I suspect a small program could be written to convert some of the intermediate files for TF. It is better to let them grow on their own. This also lets them be used as truly independent checks.

Originally Posted by kriesel

I've raised the question of continuity planning before, but it got no response. I suppose the forum and source code repositories help carry the knowledge forward. The board could select a new member to replace anyone no longer available to serve. Finding people with the talent of George, Mihai, Ernst, et al willing to commit large amounts of their time to further software development is not easy. They are rare.

George does not plan on continuing with the project after major wetware failure. New professional mathematicians and programmers shall arise. Looking for Mersenne Primes did not die out with the death of Lucas or Robinson. Scott has left the project, Aaron took over (he returned after a 'vacation'). James H has brought a lot of good to the project. There are enough trusted people in place to continue things. Also, Mersenne Research, Inc. (qv) was established. The board will work to continue the project.

[kriesel]

Thanks to rainchill for initiating the discussion and taking the time to write up lots of ideas. I hope you don't feel we're being harsh or dismissive.

Thanks to uncwilly for additional input, and ixfd64 for pointers to past threads and ongoing ideas.

It's from questions and discussions like this, that the Jacobi check for LL testing, and later the GEC for PRP testing, were introduced.

Re temperature-based adjusted throttling:

Originally Posted by Uncwilly

No way no how UNLESS the user and PrimeNet are notified that this is happening. Chris will tell you that he uses Prime95 to check machines for health and stability. If Prime95 started to silently lower its demands, that would be bad. Also, if one is using Prime95 to run a burn-in on a machine, this would defeat the purpose.

The key word in the original idea, for me at least, was "optional". No way that it would be the default. That seems like something the user ought have to deliberately enable. Like fixed percentage throttling is now. This is throttling with thermal feedback. My i3, as one example, is throttled to about 20% duty cycle for the rare worst case combination of high background load and high ambient, and could be contributing a lot more as a percentage of its current throughput, if it had thermal feedback sufficient to avoid both Windows shutdowns and BIOS shutdowns from thermal limits, while dynamically adjusting throttle. And it is still energy efficient enough to be worthwhile running it, on DC.

Re autoupdate vs. planned obsolescence of a version, what I would like to see is simply display of the current running version, right next to display of the current latest released version/build, for the same platform (Win32, Win64, linux64, MacOS). Perhaps in the app title bar. NOT a popup that requires someone to click on it to make it go away. I'd find that annoying on my small fleet. Take pity on curtisc or perhaps his student, dealing with 800 systems. It could check https://www.mersenne.org/download/ or some purpose-built location for current version/build, and display it. Daily would be plenty frequent. Or it could be a PrimeNet API extension.

George does not plan on continuing with the project after major wetware failure.

As founder of GIMPS, George should feel free and welcome to participate however he likes for as long as he likes. Even if that is lurking on the forum from a rocking chair and blanket some day in the far future. And I can't think of anyone who's earned the right to retire from the effort whenever so inclined, more than George already has. He's already contributed on the cpu side, and organizational side, very considerably. And as I recall also some contribution to mfaktx, and now with some nice speedups to gpuowl as well, very recently. https://www.goodreads.com/quotes/526...domain-he-wept
New additional talent is always welcome. And ideas that open up new areas for possible improvements.

[kriesel]

Originally Posted by ixfd64

3. Ability to dump the status via the command line: https://mersenneforum.org/showthread.php?t=19818

I think creating something approximating that is feasible, as a separate command line executable.
Read worktodo.txt, find how many workers there are, what the current assignments are, check the vintage of the corresponding save files. Delving deeper would probably involve deriving some code from the prime95 source to read the save files to determine last saved progress state. A variant of it could be a sort of watchdog monitor.

[chalsall]

Worthwhile discussion. I'm sure George will implement (over time) what makes sense, and is sane, within his HPU (Human Processing Unit) cycle constraints...

One minor thing which has always bugged me... To exit the mprime command-line configuration system, you need to enter "5". I have no idea why.

Any chance "Q" could also work? Thanks.

[Uncwilly]

Originally Posted by kriesel

Re temperature-based adjusted throttling:The key word in the original idea, for me at least, was "optional". No way that it would be the default. That seems like something the user ought have to deliberately enable.

Preferably that would be an undoc.txt item.

As founder of GIMPS, George should feel free and welcome to participate however he likes for as long as he likes.

My joke was that he didn't plan on continuing being an active role in GIMPS after he has shuffled off this mortal coil.

[kriesel]

Originally Posted by Uncwilly

My joke was that he didn't plan on continuing being an active role in GIMPS after he has shuffled off this mortal coil.

Well, there's "major", and then there's "complete" failure of systems.

[Uncwilly]

Originally Posted by rainchill

Hey Folks - just a little list of things I have been thinking about to start the new year with.
.....
20. What considerations have been made to continue the project past George's and everyone else's lifetimes?

I was just thinking:
With the work that Raphael Robinson, he published his data. Those before only published if it was or was not a prime. With a number of the later searchers, they published residues and known factors. All of those were eyeball friendly. This was good when George started GIMPS, we could DC previous work, because the previous data was available.

My suggestion is:
Work with Archive.org to set up a repository of Mersenne data from the various projects. Each of the main locations that maintain Mersenne data: Mersenne.org, Mersenne.ca, Will Edgington, Double Mersenne, etc. provide an annual snap shot of their database (some parts might be redacted) and a readme on how the database is structured. That way if something happens, the data are saved and available in the future. Obviously there will be some overlap in the data. It might be useful to have a common data format that is used that only contains the most important data (leave out bad LL tests, only record the cumulative ECM data, etc.) If the data is in a standard format, there could be a single portal to search the combined data.


This is just me thinking about preserving the most important data long term.