mersenneforum.org  

Go Back   mersenneforum.org > Prime Search Projects > No Prime Left Behind

Reply
 
Thread Tools
Old 2010-04-17, 09:34   #12
AMDave
 
AMDave's Avatar
 
Jan 2006
deep in a while-loop

2·7·47 Posts
Default

@ kar_bon

Indeed.
I see it.
I also see the point in the processing sequence that allowed that to happen.
ho hum.
I have to first debug the loading parsing that should have loaded gary's prime as a prime.
Then I have to fix the primes decanting procedure.
But, before I do that I will look for any other occurrences so all are fixed.

Last fiddled with by AMDave on 2010-04-17 at 09:44
AMDave is offline   Reply With Quote
Old 2010-04-17, 09:43   #13
AMDave
 
AMDave's Avatar
 
Jan 2006
deep in a while-loop

2·7·47 Posts
Default

Total of 3 occurrences found
Code:
+----------+-----+--------+------------+-----+--------+
| username | k   | n      | username   | k   | n      |
+----------+-----+--------+------------+-----+--------+
| Unknown  | 301 | 566979 | gd_barnes  | 301 | 566979 | 
| Unknown  | 371 | 571042 | gd_barnes  | 371 | 571042 | 
| Unknown  | 331 | 575199 | Flatlander | 331 | 575199 | 
+----------+-----+--------+------------+-----+--------+
all 3 in the same MN file and all 3 processed from the same server port knpairs file.

results_20081103_0657_IB_nplb_5000.txt:301*2^566979-1 prime! Time : sec.
results_20100415_2035_MN_nplb_0003.txt:301*2^566979-1 is prime! Time : 0.0 sec.

results_20081106_0657_IB_nplb_5000.txt:331*2^575199-1 prime! Time : sec.
results_20100415_2035_MN_nplb_0003.txt:331*2^575199-1 is prime! Time : 0.0 sec.

results_20081104_0657_IB_nplb_5000.txt:371*2^571042-1 prime! Time : sec.
results_20100415_2035_MN_nplb_0003.txt:371*2^571042-1 is prime! Time : 0.0 sec.

Perspective: 0.0546 % error rate due to combination of (a) unnacounted for source data issues and (b) processing logic
Observation: If the overlap had not occurred this bug would not have been found. How weird is that.

patch development and repair in progress etf 24 hrs

Last fiddled with by AMDave on 2010-04-17 at 10:18
AMDave is offline   Reply With Quote
Old 2010-04-17, 10:49   #14
mdettweiler
A Sunny Moo
 
mdettweiler's Avatar
 
Aug 2007
USA (GMT-5)

624910 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
Why are we making this so extremely difficult? I have now stepped in and recommended that we dispense with this entire reloading of every result that we have into the DB.

The intent was that this be a manual results import not an all results import, which would take months.

If some stuff got overwritten, that is very bad news and shows that we did not properly analyze the situation ahead of time. If that is what happened, is there a way that we can restore them?

There should be very few "unknown" results. Perhaps no more than 50-100 on the entire project. For manual results, we should know who did all of them and it's just a matter of looking it up in the 1st post of each drive. The only ones that should end up entered in the DB as "unknown" are where the server might have somehow missed 1-2 of them and either Max, Karsten, or I ran the pairs manually and put them in our file that we keep so that they matched up with original sieve file.

Max, all results for the entire 5th thru 10th drives and mini-drive are now on Jeepford. Please analyze which ones of them were done manually, associate who did them and when they were done, and load only those into the DB. For each drive, the manual loading should go fairly quickly. Since this project started the 5th drive, 95-98% of all results have been done by the servers. The lion's share of the manual stats import is coming from the 1st thru 3rd drives but even those were largely done by servers.

Going forward, the only drives that should take a little while to load into the DB are the fully manual ones; that is the individual-k and mini drive. Everything else after the 3rd drive should go very fast.

In the future, before loading any manual pairs into the DB, I want to review what is being loaded.


Gary
Let me clarify: nothing has been overwritten. All that happened was that some rather large chunks of work that had never been loaded the first time around were imported just now under "Unknown", therefore uncovering for us a gaping hole in our DB--an almost 40K LLRnet range from the early 3rd Drive was missing entirely. Since Karsten has all the results on file, all we have to do is "upsert" Karsten's files into the DB (as Dave termed it) so that the "real" results replace the "Unknown"s where applicable. Easy peasy.

Rest assured, though, yes, I will not try to import the LLRnet ranges along with the manual ones for the later drives; it was only the 1st, 2nd, and 3rd I was worried about since a lot about our servers was in a state of flux at that time and there was a very high probability of errors having been made that had gone unnoticed in the years since, and indeed that's what we turned up with this. But by the time of anything later, the server process had been cleaned up to essential clockwork, so there shouldn't be anything significant missing from there on out and therefore no need to further "re-import" any LLRnet ranges.

BTW, regarding the prime issue which we uncovered: having the "Unknown" results imported like this actually revealed a bug in Dave's duplicate-screening process which would have otherwise gone unnoticed. Now a fix is on the way.

Meanwhile, per Dave's suggestion I'm splitting off all posts related to this to a separate thread since they don't really belong in the News thread.
mdettweiler is offline   Reply With Quote
Old 2010-04-19, 13:18   #15
AMDave
 
AMDave's Avatar
 
Jan 2006
deep in a while-loop

2·7·47 Posts
Default

To contradict myself "patch development and repair in progress etf 24 hrs "
The patch fix is the easy bit, but the repair (data fix) requires due care.
I anticipate this patch and repair will be completed this coming weekend.
AMDave is offline   Reply With Quote
Old 2010-05-03, 16:11   #16
mdettweiler
A Sunny Moo
 
mdettweiler's Avatar
 
Aug 2007
USA (GMT-5)

3×2,083 Posts
Default

Quote:
Originally Posted by AMDave View Post
To contradict myself "patch development and repair in progress etf 24 hrs "
The patch fix is the easy bit, but the repair (data fix) requires due care.
I anticipate this patch and repair will be completed this coming weekend.
Not to bug you, but I'd just like to verify: is this complete now?

Meanwhile, to all: The 12th Drive manual results have been imported (except for the k=2400-2600 range, which is already half in the DB, but will need some tricky import procedures to finish off due to the weirdness with that range--that will be later). Another one down!

More than half of this drive was done entirely manually by Bruce, so this represents a big boost in stats for him. Bruce, I know you haven't shown up on the forum for a while now, but if you see this, just wanted to let you know that we finally got your results imported!
mdettweiler is offline   Reply With Quote
Old 2010-05-04, 20:43   #17
Brucifer
 
Brucifer's Avatar
 
Dec 2005

1001110012 Posts
Default

Hey gents, thank you for the kind thoughts very much. :-)

I'm sorry that I haven't been around much as there has been some other stuff going on in my world that has shoved a lot of the distributed computing stuff more to the background than I would like but that's the way the ball bounces some times.

This whole effort is a lot of work on a lot of fronts by a lot of people. The stats thing though I have to say that Max and AMDave have really been putting in a TON of midnight oil. So congrats to you all. :-)

Last fiddled with by Brucifer on 2010-05-04 at 20:44
Brucifer is offline   Reply With Quote
Old 2010-05-04, 21:06   #18
mdettweiler
A Sunny Moo
 
mdettweiler's Avatar
 
Aug 2007
USA (GMT-5)

3·2,083 Posts
Default

Quote:
Originally Posted by Brucifer View Post
Hey gents, thank you for the kind thoughts very much. :-)

I'm sorry that I haven't been around much as there has been some other stuff going on in my world that has shoved a lot of the distributed computing stuff more to the background than I would like but that's the way the ball bounces some times.

This whole effort is a lot of work on a lot of fronts by a lot of people. The stats thing though I have to say that Max and AMDave have really been putting in a TON of midnight oil. So congrats to you all. :-)
Howdy--nice to have you back! If you're by chance ready to lend a hand again, our current priority is Drive #5, which we hope to finish by year's end--we have an LLRnet and PRPnet server available for it. Also, we've resumed the 12th Drive for n=250K-350K on LLRnet port 7000, if you'd prefer that.

(Of course, if you're still largely busy with other things, no problem--I definitely understand, being rather busy with non-prime-related things myself lately. )
mdettweiler is offline   Reply With Quote
Old 2010-05-10, 16:30   #19
mdettweiler
A Sunny Moo
 
mdettweiler's Avatar
 
Aug 2007
USA (GMT-5)

141518 Posts
Default

I forgot to mention this, but all manual results so far on Drive #5 and #6 were imported into the DB a few days ago. Drive #7 should follow at tonight's daily stats rollover.

Edit: Drive #8 is now all set to be imported tonight as well.

Last fiddled with by mdettweiler on 2010-05-10 at 16:44
mdettweiler is offline   Reply With Quote
Old 2010-05-10, 21:09   #20
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

236468 Posts
Default

Very nice work Max and Dave!
gd_barnes is offline   Reply With Quote
Old 2010-05-17, 12:50   #21
AMDave
 
AMDave's Avatar
 
Jan 2006
deep in a while-loop

2×7×47 Posts
Default

Max knocked them all over for you.
The database is doing it's job.
Mostly for me, it is 'look ma, no hands!'
AMDave is offline   Reply With Quote
Old 2010-05-17, 13:01   #22
kar_bon
 
kar_bon's Avatar
 
Mar 2006
Germany

7×13×31 Posts
Default

What about the user 'Unknown'?

There're still 18 primes (300<k<400) found by this user and ~400,000 pairs in the database!
kar_bon is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
GPU id/name for manual results preda GPU Computing 15 2017-08-16 17:34
mersenne.ca and manual results Gordon PrimeNet 3 2015-08-31 03:08
manual results ramgeis PrimeNet 8 2013-05-30 06:33
manual results reporting bug for P-1 factoring dgould Information & Answers 4 2009-12-11 03:26
Manual Checkin of P-1 Results Unregistered PrimeNet 1 2004-05-18 03:15

All times are UTC. The time now is 06:56.

Wed Jul 8 06:56:40 UTC 2020 up 105 days, 4:29, 0 users, load averages: 1.50, 1.31, 1.27

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.