Error rate plot

[ewmayer]

Is the error rate really dropping as the exponents get larger, or is that an artifact of the sampling methodology?

[Madpoo]

Originally Posted by ewmayer

Is the error rate really dropping as the exponents get larger, or is that an artifact of the sampling methodology?

My own (probably unscientific) prediction is that error rates will increase with the larger exponents. For the simple reason that the longer a test takes, the more likely a memory or CPU error will creep in.

On the other hand, these larger tests are done by (generally) more reliable systems which may balance things out.

FYI, hopefully we'll see a curious "bump" in proven bad results as a direct result of our effort to find these bad CPUs and do double-checking ahead of the curve.

When we find those bad systems, they're still technically unverified until a triple-check is done, so it may still be a while until those get done.

[Dubslow]

Originally Posted by ewmayer

Is the error rate really dropping as the exponents get larger, or is that an artifact of the sampling methodology?

I believe that's the sampling methodology -- those tests simply have been largely not double checked, so anything with zero error code is assumed to be good, even though that's not true.

[Madpoo]

Originally Posted by Dubslow

I believe that's the sampling methodology -- those tests simply have been largely not double checked, so anything with zero error code is assumed to be good, even though that's not true.

In my analysis of known bad results, if it had a zero error code there's something like less than 1% bad. If it's a non zero error, it's more like a 50% chance of being bad.

I should clarify that by "non zero" I mean any error code that would mark it as "suspect". There are non-zero errors that are normal... like repeatable rounding errors during the run. I should say "suspect" and "non-suspect" instead of zero/non-zero.

Well, I forget the actual rates, but it was something like that.

[Madpoo]

Originally Posted by Madpoo

In my analysis of known bad results, if it had a zero error code there's something like less than 1% bad. If it's a non zero error, it's more like a 50% chance of being bad.

I should clarify that by "non zero" I mean any error code that would mark it as "suspect". There are non-zero errors that are normal... like repeatable rounding errors during the run. I should say "suspect" and "non-suspect" instead of zero/non-zero.

Well, I forget the actual rates, but it was something like that.

Just checked again...

If the result is marked "clean" (not suspect), counting up all of the known good/bad results, the odds of it being bad are just a hair under 2%.

(I don't consider the still-unknowns when calculating, i.e. I just use (bad)/(bad+good)).

Here are the raw stats on those "clean" results:

Unknown	Bad	Good
648674	29698	1501012

For the dirty/suspect results, 54% of the suspect results end up being bad.

Raw stats:

Unknown	Bad	Good
5904	31827	27562

EDIT: To be clear, there are certain suspect error codes that always end up with a bad result. For example, if you see an error code of "00000304" then beware, your machine hosed up. :) But on average, results marked suspect are bad more often than good.

[Madpoo]

Another couple data points...

If the error code actually is just "00000000" the odds of it being bad are currently 1.84%

The *total* known error rate no matter what the result code = 3.87%

[patrik]

Originally Posted by diep

Great!

Do i correctly interpret the red as that from all verified tests between about 2% to 4% in the 40M range gives an error?

So

(P( has error AND was verified test ) / P( all verified tests) ) = [2% ; 4%]

Did i read this correct?

If so - excuse my ignorance, how can the green be above the red in the estimate?

No, the red curve does not contain information about the error code of the test. Instead it is the number of known bad LL results in an interval divided by the number of all¹ LL results (good + bad) in the same interval, i.e.

( # known bad LL results ) / ( # known bad LL results + # known good LL results )

A result is known to be bad when two other LL results for the same exponent don't match the result but still match each other.

I use the publicly available information at mersenne.org to collect information (Reports->Detailed Reports->LL Results).

The green curve is above the red one since I count a number of unverified tests as bad: When there are still unverified exponents in the interval (i.e. there are exponents left, for which there are still not two matching tests), if there are n non-matching tests for an exponent, I count n-1 of them as bad. The red curve does not use the unverified tests at all.

¹Not including unverified tests, though, if any.

[patrik]

Originally Posted by ewmayer

Is the error rate really dropping as the exponents get larger, or is that an artifact of the sampling methodology?

I think that the green curve does not accurately describe the error rate. I don't understand, though, why it looks as it does. The effect of the decreasing green curve was present in my second post in this thread, but in the latest plot the effect has disappeared in that range.

Still, a fairly constant error rate at around 4% would mean that the probability per iteration of an error is dropping at the same time as the amount of computations (per iteration) is increasing (due to the larger FFT:s).

[henryzz]

It would be interesting to see the graph with fft boundaries added to see if they match the variations in the graph at all.

[chalsall]

Originally Posted by henryzz

It would be interesting to see the graph with fft boundaries added to see if they match the variations in the graph at all.

That would indeed be interesting.

Are you willing to work it?

[henryzz]

Originally Posted by chalsall

That would indeed be interesting.

Are you willing to work it?

If I had the data I could persuade R to produce a graph displaying this information.