Loading of manual results into the DB

[mdettweiler]

Admin edit: split this conversation off into a separate thread due to its having veered off somewhat significantly from the general theme of the "News" thread

Drive #3 has now been imported into the DB. More should follow soon!

[kar_bon]

Originally Posted by mdettweiler

Drive #3 has now been imported into the DB.

So you have to redo this!

All LLRnet-results (not sure yet, but it seems so) were added under the contributor "Unknown"!
See the Top Participants.

Rank Participant Team Pairs Returned Pair Score
11 Unknown - 399398 584,472.485

The first range from Drive #3 was at n=340k and the first prime found there

user=kar_bon
[03/01/08 15:52:33]
399*2^341641-1 is prime!  Time : 150.0 sec.

but the primelist for filter "k=399" here says

Participant Date Time k Base n Sign One Seconds Server Port Pair Score Prime Score Digits 
Bruce 2009-11-29 15:09:08 2399 2 224852 - 1 1475 GB 7000 0.315990 0.177628 67691 
Bruce 2009-11-29 14:08:13 2399 2 52196 - 1 133 GB 7000 0.017028 0.002222 15716 
PCZ 2009-01-13 12:52:53 1399 2 142227 - 1 181 IB 9000 0.126428 0.044954 42818 
gd_barnes 2008-05-24 06:22:05 399 2 496100 - 1 589 IB 5000 1.538220 1.907777 149344 
Unknown 2008-04-10 13:03:00 399 2 341641 - 1 50 MN 0003 0.729491 0.623060 102847 
Beyond 2008-03-20 07:25:00 399 2 290555 - 1 836 MN 0003 0.527639 0.383270 87469

Sorry guys! Seems you overwrote all LLRnet-results with 'Unknown'!

'Unknown' owns 18 primes now!

PS: As I can say, Drive #3 got 386082 pairs done by LLRnet-server I5000/I8000.
The count (399398) from above is more than this! Question: Which pairs were false inserted, too?

[mdettweiler]

Originally Posted by kar_bon

So you have to redo this!

All LLRnet-results (not sure yet, but it seems so) were added under the contributor "Unknown"!
See the Top Participants.

Rank Participant Team Pairs Returned Pair Score
11 Unknown - 399398 584,472.485

The first range from Drive #3 was at n=340k and the first prime found there

user=kar_bon
[03/01/08 15:52:33]
399*2^341641-1 is prime!  Time : 150.0 sec.

but the primelist for filter "k=399" here says

Participant Date Time k Base n Sign One Seconds Server Port Pair Score Prime Score Digits 
Bruce 2009-11-29 15:09:08 2399 2 224852 - 1 1475 GB 7000 0.315990 0.177628 67691 
Bruce 2009-11-29 14:08:13 2399 2 52196 - 1 133 GB 7000 0.017028 0.002222 15716 
PCZ 2009-01-13 12:52:53 1399 2 142227 - 1 181 IB 9000 0.126428 0.044954 42818 
gd_barnes 2008-05-24 06:22:05 399 2 496100 - 1 589 IB 5000 1.538220 1.907777 149344 
Unknown 2008-04-10 13:03:00 399 2 341641 - 1 50 MN 0003 0.729491 0.623060 102847 
Beyond 2008-03-20 07:25:00 399 2 290555 - 1 836 MN 0003 0.527639 0.383270 87469

Sorry guys! Seems you overwrote all LLRnet-results with 'Unknown'!

'Unknown' owns 18 primes now!

Some of the early LLRnet results from the C*** servers never got imported into the DB; unless you still have the original files sitting around, the data on who specifically did what in those ranges has long since been lost. Those have therefore been imported as "Unknown".

However, if you do have the original results for those, please send them to me--I'll then send them to Dave and ask him to do a special re-import on them to overwrite the "Unknown"'s.

[kar_bon]

Originally Posted by mdettweiler

Some of the early LLRnet results from the C*** servers never got imported into the DB; unless you still have the original files sitting around, the data on who specifically did what in those ranges has long since been lost. Those have therefore been imported as "Unknown".

However, if you do have the original results for those, please send them to me--I'll then send them to Dave and ask him to do a special re-import on them to overwrite the "Unknown"'s.

As mentioned those results were from IB5000 and IB 8000 servers!

Tell me which results you missing and i see what i got.

[mdettweiler]

Originally Posted by kar_bon

As mentioned those results were from IB5000 and IB 8000 servers!

Tell me which results you missing and i see what i got.

Okay, I've uploaded a dump of all results with user "Unknown" to:
http://www.noprimeleftbehind.net/unknown_results.txt Edit by Max: link taken offline (no longer needed)

A few of the "Unknown" results are just occasional pairs that you, myself, or Gary filled in during processing, but there definitely appear to be some whole ranges in there. They are:

1st Drive, n=510K-510K.5K
Strange...that range was done manually by me. It's possible that this range was accidentally mislabeled in manual-import preprocessing. This can be easily remedied with a quick SQL command.

1st Drive, n=543038-543749
This range was originally done by the C443 server. It looks like we missed one file when we manually downloaded the results from Carlos's web site and sent them to Dave for importing.

3rd Drive, n=340K-383274
Originally done by IB5000. This may have been before we moved the DB over to David's server; if it was still on Adam's machine, then we would have been doing these manually. It seems these files here were somehow neglected.

Karsten, if you have the original LLRnet-format results for the latter two, send them my way--I'll talk to Dave about figuring out some way to have the DB do a special import on them where it will use them to replace "Unknown" results where appropriate. As for the first one (the 1st Drive range done by me), since that was a manual range done entirely by one person it's simply a matter of mislabeling and can be easily corrected.

BTW, I didn't see any "Unknown" results from IB8000 ranges; could you possibly point those out to me?

[kar_bon]

Originally Posted by mdettweiler

Karsten, if you have the original LLRnet-format results for the latter two, send them my way

Mailed!

Originally Posted by mdettweiler

BTW, I didn't see any "Unknown" results from IB8000 ranges; could you possibly point those out to me?

IB8000 seems ok, as I can say from here.

But please check Drive #3 IB5000 again:
For example this primes:
Unknown 2008-11-10 10:24:00 371 2 571042

but it was found by Gary!

From resultfile "results_20081104_0657_IB_nplb_5000.txt":

user=gd_barnes
[11/04/08 05:35:56]
371*2^571042-1 is prime!  Time :  sec.

There're several ranges done by IB5000 on Drive #3, see here: I've documented all ranges with server or contributor there.

[mdettweiler]

Originally Posted by kar_bon

Mailed!

Thanks, got 'em.

IB8000 seems ok, as I can say from here.

But please check Drive #3 IB5000 again:
For example this primes:
Unknown 2008-11-10 10:24:00 371 2 571042

but it was found by Gary!

From resultfile "results_20081104_0657_IB_nplb_5000.txt":

user=gd_barnes
[11/04/08 05:35:56]
371*2^571042-1 is prime!  Time :  sec.

There're several ranges done by IB5000 on Drive #3, see here: I've documented all ranges with server or contributor there.

Now that is strange. No results for that n appear in the list of "Unknown" results I posted earlier, yet I generated that list with the SQL command "select * from results where username = 'Unknown'", which should produce each and every "Unknown" result in the DB. Something isn't adding up here.

To investigate this further I tried selecting just this one k/n pair from the DB:

mysql> select * from results where n = 571042 and k = 371;
+-----------+------------+----------+-----+------+--------+------+-----+-------+----------+---------+--------+------+----------------+-------------+
| username  | date       | time     | k   | base | n      | sign | one | prime | residual | seconds | server | port | pair_score     | prime_score |
+-----------+------------+----------+-----+------+--------+------+-----+-------+----------+---------+--------+------+----------------+-------------+
| gd_barnes | 2008-11-04 | 05:35:56 | 371 |    2 | 571042 | -    |   1 |     0 |          |       0 | IB     | 5000 | 2.038056036025 |        NULL |
+-----------+------------+----------+-----+------+--------+------+-----+-------+----------+---------+--------+------+----------------+-------------+
1 row in set (0.13 sec)

So it is in there under Gary's name, not Unknown. One thing I did find strange is that the "prime" column seemed to be set to 0 despite the number being prime but it's possible I'm just misinterpreting Dave's method of filing primes in the DB.

Yet the website lists it under "Unknown".

Dave, any idea of the cause of this apparent disconnect?

[gd_barnes]

Why are we making this so extremely difficult? I have now stepped in and recommended that we dispense with this entire reloading of every result that we have into the DB.

The intent was that this be a manual results import not an all results import, which would take months.

If some stuff got overwritten, that is very bad news and shows that we did not properly analyze the situation ahead of time. If that is what happened, is there a way that we can restore them?

There should be very few "unknown" results. Perhaps no more than 50-100 on the entire project. For manual results, we should know who did all of them and it's just a matter of looking it up in the 1st post of each drive. The only ones that should end up entered in the DB as "unknown" are where the server might have somehow missed 1-2 of them and either Max, Karsten, or I ran the pairs manually and put them in our file that we keep so that they matched up with original sieve file.

Max, all results for the entire 5th thru 10th drives and mini-drive are now on Jeepford. Please analyze which ones of them were done manually, associate who did them and when they were done, and load only those into the DB. For each drive, the manual loading should go fairly quickly. Since this project started the 5th drive, 95-98% of all results have been done by the servers. The lion's share of the manual stats import is coming from the 1st thru 3rd drives but even those were largely done by servers.

Going forward, the only drives that should take a little while to load into the DB are the fully manual ones; that is the individual-k and mini drive. Everything else after the 3rd drive should go very fast.

In the future, before loading any manual pairs into the DB, I want to review what is being loaded.


Gary

[AMDave]

grep search shows the following result

results_20081104_0657_IB_nplb_5000.txt:
user=gd_barnes^M
[11/04/08 05:35:56]^M
371*2^571042-1 prime! Time : sec.^M

results_20100415_2035_MN_nplb_0003.txt:
user=Unknown
[2008-11-10 10:24:00]
371*2^571042-1 is prime! Time : 0.0 sec.

Re the first file, that whole file is full of CTRL characters.
I will check if that has had an impact on the loading.
The prime detection is looking for "prime!" so the first instance should have been loaded and registered as a prime.
It could be the missing search-time. I will test the parsing.

Re the second file, this pair was not loaded because the pair already exists in the results table.
Why the pair was subsequently submitted again in manual search I do not know.

[kar_bon]

Originally Posted by mdettweiler

So it is in there under Gary's name, not Unknown. One thing I did find strange is that the "prime" column seemed to be set to 0 despite the number being prime but it's possible I'm just misinterpreting Dave's method of filing primes in the DB.

Yet the website lists it under "Unknown".

As i can say from an export of Adam's DB from 2008-09-15 17:09:22 there are tables for pairs and an extra table for primes!

So the pair is correctly in that table you requested but the prime as shown here is from the prime-table and there is contributor = 'Unknown'!

[AMDave]

suggest we splice these posts out of the News thread