Getting reliable LL from unreliable hardware

[henryzz]

Originally Posted by retina

Torture tests should only be on known verified results. So torture tests need to be redundant, and therefore useless in terms of progressing the "goal" forward.

If a mismatch occurs then the torture testing pc can run a triple check with a different fft length.

[CRGreathouse]

Originally Posted by henryzz

I suppose torture tests could actually be doing useful doublechecks.

Originally Posted by retina

Torture tests should only be on known verified results. So torture tests need to be redundant, and therefore useless in terms of progressing the "goal" forward.

Torture tests could be non-redundant, but then ~5% of them would return "indeterminate" because the residues wouldn't match and it wouldn't be clear which was wrong.

Edit: henryzz beat me to it.

[Madpoo]

Originally Posted by Mark Rose

If 5% are bad, and we do a triple check 5% of the time, that's only 5/205 or 2.4% work saved, no?

LOL... look, I didn't say I worked out all the math exactly.

Actually I had one of those "late at night" moments where I thought back to this post and realized just what you mentioned, that the increased throughput isn't really 5% but approximately half of that (or less, since as you note the double-checks are also sometimes wrong as well leading to the necessity of quadruple+ checks just to finally get two that match).

Anyway, the point was, in theory it's good but in practice there needs to be that balance of making sure results are not only accurate, but also legitimate.

[Madpoo]

Originally Posted by ewmayer

It' also easy in theory to envision how one might do this in practice - now, the first-time-tester's interim checkpoint files (say every 10Miter) get deposited on the Primenet server, and the DCer's Res64s at those same iterations diffed against those, etc. But that would require a massive increase in both storage capacity at the server end and comms-bandwidth between the users' machines and the server. The low-bandwidth alternative would be to let the interim CP files remain on the users' machines and then have the server direct the restart-both-runs-from-last-matching-CP mechanism, but I shudder to think of the effort needed to support that functionality, not to mention the myriad real-world reasons why it would be inherently fragile.

If we had a system where pairs of users were working on the same exponent, they could send periodic 64-bit residues like the final one, but in increments of every 5% of progress or something like that. It would also save that interim file in case a roll back is needed to one of those points.

If at any point they mismatched (during their daily checkins maybe the server informs them of a mismatch at iteration XXX) they roll back to that point and try again.

The trouble is, pairs of users on the same exponent are unlikely to progress at the same rate, so you could have one system racing ahead and finishing entirely by the time the other one even hits 10%. Or maybe the other system stops reporting entirely.

You'd have to also build in some kind of system that tries to evenly match users and systems to mitigate that, but even the most regular contributor sometimes takes a system down for maintenance or has a failure of some kind. I noticed that many of Curtis' machines have stopped reporting in (I noticed because he had a bunch reserved in the category 0 first-time-check ranges).

In terms of making sure the horsepower is evenly matched, it makes the most sense to have the same user doing both checks, but again we get into that question of how to keep out the bogus results.

I haven't really thought about it much, but it seems like maybe there would be a way for systems to report in a partial residue at some lower iteration count. Then if we had reason to suspect something was odd, we could run up to that point and compare that residue just to see if they even bothered going that far. Someone trying to cheat ... would they really run it 10% of the way or would they just make stuff up entirely? Who knows. People are weird, but we could probably count on most of them being lazy enough to not only cheat, but to do it in the laziest way possible.

[GP2]

If cloud computing eats the world, and everyone eventually starts using it instead of home server farms, then we'll be in an interesting situation, because those cloud servers have an observed error rate of zero with mprime (certainly bounded below 0.05% in any case, based on several thousand error-free double-checked residues on AWS so far). Those cloud servers have ECC memory, strictly controlled ambient temperatures in the data centers, and they are routinely taken out of service whenever automated monitoring detects that they have started acting up.

In that hypothetical future it simply won't make sense to double-check exponents that were first-time checked using the cloud, because the payoff will be so low. Except, to prevent any sort of fraud or shenanigans, the actual LL testing would probably need to be centralized so that it would be certifiably known that the LL tests were performed on an approved server type in an approved cloud. So people would pay $ X or € to mersenne.org and purchase a batch of, say, 10 LL tests at a time. Would that still be as much fun, or too passive and boring? Hard to say.

[S485122]

[QUOTE=GP2;464217... So people would pay ... and purchase ...
Would that still be as much fun, or too passive and boring? Hard to say.[/QUOTE]"Hard to say ?"
It would stop me immediately from wasting energy for a a project whose only but huge merit is fun and cooperation : both of them would be replaced by sending money to some big corporation.
Would I be alone ?

Jacob

P.S. : Another thing : what data did you use to produce that error rate of 0,05 % for cloud computing ?

P.P.S. : I suppose that by paying some more money one would get a positive result : one of the exponents that yield a prime ? Oh ! What fun ! What a pleasure !

[GP2]

Originally Posted by S485122

P.S. : Another thing : what data did you use to produce that error rate of 0,05 % for cloud computing ?

0.05% would be 1 in 2000.

I've personally run about 3000 double-checks, with zero errors so far, and all mismatches were verified in my favor except the most recent ten or so which are still pending.

Also, a few years ago, a now-dormant account named "Amazon EC2" ran about 27000 LL tests, of which the vast majority remain unverified but about 2900 so far have been verified by double-checking. The account name strongly suggests all those LL tests were run on the cloud, and the sheer quantity suggest some entity with large financial resources, whose name might even start with Ama and rhyme with zon.

So that 0.05% is actually just a conservative upper bound. The actual error rate would be considerably lower still.

[ewmayer]

Originally Posted by GP2

Also, a few years ago, a now-dormant account named "Amazon EC2" ran about 27000 LL tests, of which the vast majority remain unverified but about 2900 so far have been verified by double-checking. The account name strongly suggests all those LL tests were run on the cloud, and the sheer quantity suggest some entity with large financial resources, whose name might even start with Ama and rhyme with zon.

This lady? Just curious, what pointed you in her direction?

[Madpoo]

Originally Posted by GP2

0.05% would be 1 in 2000.

I've personally run about 3000 double-checks, with zero errors so far, and all mismatches were verified in my favor except the most recent ten or so which are still pending.

Also, a few years ago, a now-dormant account named "Amazon EC2" ran about 27000 LL tests, of which the vast majority remain unverified but about 2900 so far have been verified by double-checking. The account name strongly suggests all those LL tests were run on the cloud, and the sheer quantity suggest some entity with large financial resources, whose name might even start with Ama and rhyme with zon.

So that 0.05% is actually just a conservative upper bound. The actual error rate would be considerably lower still.

A few notes...

I've had the pleasure of double-checking some of those "Amazon EC2" results and they've all matched (part of my thing to make sure each computer has at least one DC'd result in its history).

So, based on its results so far it is safe to say that ECC, no overclocking, and climate controlled environments make a big difference in reliability. I see that on my own systems too... no bad results (due to hardware) over thousands and thousands of double/triple+ checks.

Another note is that most people participating in GIMPS are using their desktops, not servers. While cloud computing may be on a tear lately (will it be sustained though, or just a fad or more of a niche long-term?) I don't think people will be swapping out their home machines (or work desktops, school computers, etc) for any typical kind of cloud solution.

I could be wrong... I've been wrong before when making bold predictions (I predicted Facebook stock would hover in the low $20s). I just have a hard time thinking society is ready to re-adopt the "dumb terminal" paradigm where simple equipment was connecting to some huge mainframe to do the heavy lifting.

I think the cloud server market is kind of weak and lazy personally, but then I'm biased about that. I just can't figure out why someone would pay more money to use someone else's computers when you can do it for half the cost (or get twice+ as much power for the same price) if you roll your own. Whatever... some people don't change their own oil either, or mow their own lawns.

[Mark Rose]

Originally Posted by Madpoo

I think the cloud server market is kind of weak and lazy personally, but then I'm biased about that. I just can't figure out why someone would pay more money to use someone else's computers when you can do it for half the cost (or get twice+ as much power for the same price) if you roll your own. Whatever... some people don't change their own oil either, or mow their own lawns.

1. Capital efficiency. The less money I have to raise to pay up-front for hardware, the less dilution I suffer.

2. Unknown future hardware requirements. The cloud essentially lets you pick very short lease terms.

3. Bursty hardware requirements.

4. Outsourcing non-expertise. I don't want to run multiple data centers, a CDN, etc.

[kriesel]

Originally Posted by Madpoo

If we had a system where pairs of users were working on the same exponent, they could send periodic 64-bit residues like the final one, but in increments of every 5% of progress or something like that. It would also save that interim file in case a roll back is needed to one of those points.

If at any point they mismatched (during their daily checkins maybe the server informs them of a mismatch at iteration XXX) they roll back to that point and try again.

The trouble is, pairs of users on the same exponent are unlikely to progress at the same rate, so you could have one system racing ahead and finishing entirely by the time the other one even hits 10%. Or maybe the other system stops reporting entirely.

You'd have to also build in some kind of system that tries to evenly match users and systems to mitigate that, but even the most regular contributor sometimes takes a system down for maintenance or has a failure of some kind. I noticed that many of Curtis' machines have stopped reporting in (I noticed because he had a bunch reserved in the category 0 first-time-check ranges).

In terms of making sure the horsepower is evenly matched, it makes the most sense to have the same user doing both checks, but again we get into that question of how to keep out the bogus results.

I haven't really thought about it much, but it seems like maybe there would be a way for systems to report in a partial residue at some lower iteration count. Then if we had reason to suspect something was odd, we could run up to that point and compare that residue just to see if they even bothered going that far. Someone trying to cheat ... would they really run it 10% of the way or would they just make stuff up entirely? Who knows. People are weird, but we could probably count on most of them being lazy enough to not only cheat, but to do it in the laziest way possible.

Interesting thread. The QA team for prime95 V18 to 20 faced similar issues, with long run times, mismatched equipment speeds, on smaller exponents but slower hardware years ago.

QA team members were volunteers. We were paired with another running the same exponents at the same time, and emailing 64-bit interim residues back and forth and generating and saving savefiles for possible restart. The emails cced prime95 and one of the volunteers who had additionally volunteered to put up an ftp server to store interim save files as an early form of cloud backup. To exercise lots of code paths, on lots of hardware types, quickly, some thoughtfully chosen 400-iteration candidate exponents were selected, and known-double-checked-by-different-software-on-different-hardware-architecture interim residues produced.

So substitute the primenet server for email, expand its detailed exponent report to optionally include lsb-masked interim residues every 10M or 100M, like residues are shown for not-yet-doublechecked LL tests, and manual submission to accommodate a new interim 64-bit residue result type, and it could indicate any of the following when an interim result is submitted:
-exponent not assigned to you
-first occurrence of that iteration count residue for that exponent, validity tbd
-not first occurrence of that iteration count residue for that exponent, matches another from another user id
-at least one other report received for that iteration count residue for that exponent, but no match
-at least one other report received for that iteration count residue for that exponent, and yours matches a previous report
-multiple other reports from other user ids for that iteration count residue for that exponent. Your reported residue does not match other matching reports

It could also involve a new type of assignment, called something like coordinated co-test.

I don't think there's much chance of cheating to fake double checking if going to the trouble of computing a long test. There also seems no simple way of preventing it.