Loading of manual results into the DB

[AMDave]

@ kar_bon

Indeed.
I see it.
I also see the point in the processing sequence that allowed that to happen.
ho hum.
I have to first debug the loading parsing that should have loaded gary's prime as a prime.
Then I have to fix the primes decanting procedure.
But, before I do that I will look for any other occurrences so all are fixed.

[AMDave]

Total of 3 occurrences found

+----------+-----+--------+------------+-----+--------+
| username | k   | n      | username   | k   | n      |
+----------+-----+--------+------------+-----+--------+
| Unknown  | 301 | 566979 | gd_barnes  | 301 | 566979 | 
| Unknown  | 371 | 571042 | gd_barnes  | 371 | 571042 | 
| Unknown  | 331 | 575199 | Flatlander | 331 | 575199 | 
+----------+-----+--------+------------+-----+--------+

all 3 in the same MN file and all 3 processed from the same server port knpairs file.

results_20081103_0657_IB_nplb_5000.txt:301*2^566979-1 prime! Time : sec.
results_20100415_2035_MN_nplb_0003.txt:301*2^566979-1 is prime! Time : 0.0 sec.

results_20081106_0657_IB_nplb_5000.txt:331*2^575199-1 prime! Time : sec.
results_20100415_2035_MN_nplb_0003.txt:331*2^575199-1 is prime! Time : 0.0 sec.

results_20081104_0657_IB_nplb_5000.txt:371*2^571042-1 prime! Time : sec.
results_20100415_2035_MN_nplb_0003.txt:371*2^571042-1 is prime! Time : 0.0 sec.

Perspective: 0.0546 % error rate due to combination of (a) unnacounted for source data issues and (b) processing logic
Observation: If the overlap had not occurred this bug would not have been found. How weird is that.

patch development and repair in progress etf 24 hrs

[mdettweiler]

Originally Posted by gd_barnes

Why are we making this so extremely difficult? I have now stepped in and recommended that we dispense with this entire reloading of every result that we have into the DB.

The intent was that this be a manual results import not an all results import, which would take months.

If some stuff got overwritten, that is very bad news and shows that we did not properly analyze the situation ahead of time. If that is what happened, is there a way that we can restore them?

There should be very few "unknown" results. Perhaps no more than 50-100 on the entire project. For manual results, we should know who did all of them and it's just a matter of looking it up in the 1st post of each drive. The only ones that should end up entered in the DB as "unknown" are where the server might have somehow missed 1-2 of them and either Max, Karsten, or I ran the pairs manually and put them in our file that we keep so that they matched up with original sieve file.

Max, all results for the entire 5th thru 10th drives and mini-drive are now on Jeepford. Please analyze which ones of them were done manually, associate who did them and when they were done, and load only those into the DB. For each drive, the manual loading should go fairly quickly. Since this project started the 5th drive, 95-98% of all results have been done by the servers. The lion's share of the manual stats import is coming from the 1st thru 3rd drives but even those were largely done by servers.

Going forward, the only drives that should take a little while to load into the DB are the fully manual ones; that is the individual-k and mini drive. Everything else after the 3rd drive should go very fast.

In the future, before loading any manual pairs into the DB, I want to review what is being loaded.


Gary

Let me clarify: nothing has been overwritten. All that happened was that some rather large chunks of work that had never been loaded the first time around were imported just now under "Unknown", therefore uncovering for us a gaping hole in our DB--an almost 40K LLRnet range from the early 3rd Drive was missing entirely. Since Karsten has all the results on file, all we have to do is "upsert" Karsten's files into the DB (as Dave termed it) so that the "real" results replace the "Unknown"s where applicable. Easy peasy.

Rest assured, though, yes, I will not try to import the LLRnet ranges along with the manual ones for the later drives; it was only the 1st, 2nd, and 3rd I was worried about since a lot about our servers was in a state of flux at that time and there was a very high probability of errors having been made that had gone unnoticed in the years since, and indeed that's what we turned up with this. But by the time of anything later, the server process had been cleaned up to essential clockwork, so there shouldn't be anything significant missing from there on out and therefore no need to further "re-import" any LLRnet ranges.

BTW, regarding the prime issue which we uncovered: having the "Unknown" results imported like this actually revealed a bug in Dave's duplicate-screening process which would have otherwise gone unnoticed. Now a fix is on the way.

Meanwhile, per Dave's suggestion I'm splitting off all posts related to this to a separate thread since they don't really belong in the News thread.

[AMDave]

To contradict myself "patch development and repair in progress etf 24 hrs "
The patch fix is the easy bit, but the repair (data fix) requires due care.
I anticipate this patch and repair will be completed this coming weekend.

[mdettweiler]

Originally Posted by AMDave

To contradict myself "patch development and repair in progress etf 24 hrs "
The patch fix is the easy bit, but the repair (data fix) requires due care.
I anticipate this patch and repair will be completed this coming weekend.

Not to bug you, but I'd just like to verify: is this complete now?

Meanwhile, to all: The 12th Drive manual results have been imported (except for the k=2400-2600 range, which is already half in the DB, but will need some tricky import procedures to finish off due to the weirdness with that range--that will be later). Another one down!

More than half of this drive was done entirely manually by Bruce, so this represents a big boost in stats for him. Bruce, I know you haven't shown up on the forum for a while now, but if you see this, just wanted to let you know that we finally got your results imported!

[Brucifer]

Hey gents, thank you for the kind thoughts very much. :-)

I'm sorry that I haven't been around much as there has been some other stuff going on in my world that has shoved a lot of the distributed computing stuff more to the background than I would like but that's the way the ball bounces some times.

This whole effort is a lot of work on a lot of fronts by a lot of people. The stats thing though I have to say that Max and AMDave have really been putting in a TON of midnight oil. So congrats to you all. :-)

[mdettweiler]

Originally Posted by Brucifer

Hey gents, thank you for the kind thoughts very much. :-)

I'm sorry that I haven't been around much as there has been some other stuff going on in my world that has shoved a lot of the distributed computing stuff more to the background than I would like but that's the way the ball bounces some times.

This whole effort is a lot of work on a lot of fronts by a lot of people. The stats thing though I have to say that Max and AMDave have really been putting in a TON of midnight oil. So congrats to you all. :-)

Howdy--nice to have you back! If you're by chance ready to lend a hand again, our current priority is Drive #5, which we hope to finish by year's end--we have an LLRnet and PRPnet server available for it. Also, we've resumed the 12th Drive for n=250K-350K on LLRnet port 7000, if you'd prefer that.

(Of course, if you're still largely busy with other things, no problem--I definitely understand, being rather busy with non-prime-related things myself lately. )

[mdettweiler]

I forgot to mention this, but all manual results so far on Drive #5 and #6 were imported into the DB a few days ago. Drive #7 should follow at tonight's daily stats rollover.

Edit: Drive #8 is now all set to be imported tonight as well.

[gd_barnes]

Very nice work Max and Dave!

[AMDave]

Max knocked them all over for you.
The database is doing it's job.
Mostly for me, it is 'look ma, no hands!'

[kar_bon]

What about the user 'Unknown'?

There're still 18 primes (300<k<400) found by this user and ~400,000 pairs in the database!