Getting reliable LL from unreliable hardware

[GP2]

Originally Posted by chalsall

It should be. The GNU Lesser General Public License allows inclusion of the code into proprietary software.

Then GMP should be OK too, since it's dual-licensed under GNU LGPL v3 and GNU GPL v2, so you can elect to use the former. But MPIR is probably easier to use with Windows.

[preda]

[experimental]
I just added the Jacobi check to gpuOwL ( https://github.com/preda/gpuowl ).

It checks it on each "persist" checkpoint save, which by default is every 10M iterations. (thus the performance impact of about 0.2%). It uses GMP; compilation with this check is enabled by defining JACOBI like this:

export JACOBI=1
make

or make JACOBI=1

[kriesel]

Originally Posted by Prime95

55 seconds every ten or hundred thousand iterations for a 50% chance of discovering an error seems like pretty cheap insurance to me.

Is there free Jacobi code available ( GMP has poisonous GNU license restrictions )? P.S. I'd google for free code but my Internet availability is severely restricted at the moment. P.P.S. I think using libgmp would avoid license issues assuming one can freely ship libgmp for all platforms that prime95 supports. But then again IANAL.

Easy, George. I can only imagine what those irate red faces translate into verbally. Don't blow a gasket or a cranial artery!

If the Jacobi occasionally applied could catch 40% of the few percent of bad LL test residues early at minimal percentage cost and stop them in their tracks, triggering redoing the calculation from the last believed-good save file, that would be exciting news, applicable not only to prime95 but also CUDALucas and perhaps a whole fleet of codes, raising productive output a couple more percent on the same hardware and electrical budget, independently of other potential optimizations.

I don't recall seeing sum of inputs checks in the CUDALucas or CUDAPm1 source.

CUDAPm1 hasn't the illegal-residue check yet. I have seen the repeating-zero-residue problem show up in stage two of a CUDAPm1 run just today (2.5 day run on ~250M, ouch). I don't know enough about the underlying math to know whether Jacobi or sum of inputs are applicable to P-1.

What checks are built into the prime95 P-1 code? (Haven't even tried to read that.)

[science_man_88]

Originally Posted by kriesel

Easy, George. I can only imagine what those irate red faces translate into verbally.

in some browser you can hover over the image for the tool tip ...

[GP2]

Originally Posted by preda

[experimental]
I just added the Jacobi check to gpuOwL ( https://github.com/preda/gpuowl ).

It checks it on each "persist" checkpoint save, which by default is every 10M iterations. (thus the performance impact of about 0.2%).

In addition to any interim checking, the Jacobi check should be done at the very end, on the final residue, before it is truncated to 64 bits and sent to GIMPS.

The interim checks benefit only the user in question by saving some wasted computing cycles; the check at the end on the other hand gives GIMPS a 50% chance of identifying a bad result with certainty.

If this final check fails, and no recovery from an interim savefile is possible or desired, then the 64-bit residue is garbage and rather than reporting a meaningless wrong value it should be set to some special marker value like 00000000DEADBEEF.

[preda]

Yes it makes sense to [also] do the check at the end before submitting the final residue.

It's not clear to me what is the right behavior of the software when the check fails -- should it roll back to the most recent "good" point and re-attempt from there, or should it report the hardware as hopelessly broken and give up.

The situation is that, when the jacobi detects the first "sure" error, there is a high probability that there were also undetected errors before. Thus rolling back to a recent point only fixes the visible error, while preserving the hidden errors -- not good.

[preda]

BTW, here is a paper discussing an efficient algorithm for computing the Jacobi symbol, in case anybody's interested [in attempting to improve GMP's implem for huge numbers]:

https://maths-people.anu.edu.au/~bre..._ACCMCC_10.pdf

[GP2]

Originally Posted by preda

It's not clear to me what is the right behavior of the software when the check fails -- should it roll back to the most recent "good" point and re-attempt from there, or should it report the hardware as hopelessly broken and give up.

The situation is that, when the jacobi detects the first "sure" error, there is a high probability that there were also undetected errors before. Thus rolling back to a recent point only fixes the visible error, while preserving the hidden errors -- not good.

I disagree. It is not common for hardware to be hopelessly broken. There are many cases where some users are turning in, say, 10% or 20% or 40% bad results, which is certainly an elevated error rate, but they are still returning a large number of good results. So a significant number of the bad results might well be due to a single error, because it would be strange if there were only zero-errors and many-errors, without any one-error cases in between.

"Hidden errors" are not a huge problem... they have always existed, and until now we have only been able to discover them when a double check is done, often years later. This Jacobi check offers a shortcut in some cases, it isn't meant to be a guaranteed filter. The worst-case scenario is just the current status quo.

In the end, it's up to the user to decide if they want to run LL testing software on their crappy computer. If the software gives up, that doesn't necessarily mean the user will give up. They might have set it up so that the software just keeps running and fetching new exponents.

[preda]

Yes I agree that the point is not to fight the user.

But assuming the user is well intended and wants to produce high quality results, upon Jacobi-error should we revert to "the most recent point with no visible error" or revert to start.

[error]

Originally Posted by preda

BTW, here is a paper discussing an efficient algorithm for computing the Jacobi symbol, in case anybody's interested [in attempting to improve GMP's implem for huge numbers]:

https://maths-people.anu.edu.au/~bre..._ACCMCC_10.pdf

Nice presentation!

But I did not understand why it is a problem if r < 0

Since (a*b|n) = (a|n)(b|n) and (-1|n) = (-1)^((n-1)/2), isn't (-r|n) = (r|n)(-1)^((n-1)/2) so that the negative value can be dealt with?

[error]

Originally Posted by preda

Yes I agree that the point is not to fight the user.

But assuming the user is well intended and wants to produce high quality results, upon Jacobi-error should we revert to "the most recent point with no visible error" or revert to start.

This would probably depend on the frequency of the errors caught.

If single errors are observed only during a few complete tests and there are many (at least apparently) good tests in between, it is probably safe to start from the last (apparently) good iteration.

If most tests are interrupted with errors, one should probably start from scratch, and then proceed as long as the permanent checkpoints agree. The last such can then be marked as a good continuation point. The test would be carried on until another error is spotted (or till the end). If the run will be completed, it might be a good idea to make another run from the last good continuation point (perhaps depending on the number of errors seen during the run). (That would result in a self-verified doublecheck.)