mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Msieve

Reply
 
Thread Tools
Old 2011-10-27, 15:43   #1
fivemack
(loop (#_fork))
 
fivemack's Avatar
 
Feb 2006
Cambridge, England

13×491 Posts
Default Odd bug

OK, the answer is probably 'don't do that', but I inadvertently put two copies of all the relations for a 30-bit LP job into the msieve.dat file one after the other.

All seemed to go reasonably for a little while, but something went disastrously wrong when reading the small ideals:

Code:
found 98209592 hash collisions in 196628532 relations
commencing duplicate removal, pass 2
found 111526275 duplicates and 85102257 unique relations
memory use: 2387.0 MB
reading ideals above 65994752
commencing singleton removal, initial pass
memory use: 1506.0 MB
reading all ideals from disk
memory use: 1383.8 MB
commencing in-memory singleton removal
begin with 85102257 relations and 75757943 unique ideals
reduce to 44582950 relations and 29695136 ideals in 17 passes
max relations containing the same ideal: 19
reading ideals above 720000
commencing singleton removal, initial pass
memory use: 1506.0 MB
reading all ideals from disk
memory use: 4691.7 MB
keeping 78900649 ideals with weight <= 200, target excess is 1143275
commencing in-memory singleton removal
begin with 129598530 relations and 78900649 unique ideals
reduce to 98566756 relations and 41906339 ideals in 12 passes
max relations containing the same ideal: 182
removing 4232495 relations and 2412831 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 94334261 relations and 41906339 unique ideals
reduce to 93842704 relations and 38704399 ideals in 4 passes
max relations containing the same ideal: 174
removing 4000128 relations and 2500812 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 89842576 relations and 38704399 unique ideals
reduce to 89806370 relations and 36165826 ideals in 3 passes
max relations containing the same ideal: 172
removing 406036 relations and 235616 ideals in 203017 cliques
commencing in-memory singleton removal
begin with 89400334 relations and 36165826 unique ideals
reduce to 89393134 relations and 35922943 ideals in 3 passes
max relations containing the same ideal: 172
removing 103276 relations and 51666 ideals in 51638 cliques
commencing in-memory singleton removal
begin with 89289858 relations and 35922943 unique ideals
reduce to 89286786 relations and 35868196 ideals in 2 passes
max relations containing the same ideal: 172
removing 48472 relations and 24246 ideals in 24236 cliques
commencing in-memory singleton removal
begin with 89238314 relations and 35868196 unique ideals
reduce to 89236868 relations and 35842502 ideals in 2 passes
max relations containing the same ideal: 172
removing 23432 relations and 11717 ideals in 11716 cliques
commencing in-memory singleton removal
begin with 89213436 relations and 35842502 unique ideals
reduce to 89212760 relations and 35830109 ideals in 2 passes
max relations containing the same ideal: 172
relations with 0 large ideals: 53569
relations with 1 large ideals: 716279
relations with 2 large ideals: 4235090
relations with 3 large ideals: 13643674
relations with 4 large ideals: 25150395
relations with 5 large ideals: 26037561
relations with 6 large ideals: 14401854
relations with 7+ large ideals: 4974338
commencing 2-way merge
reduce to 89207260 relation sets and 35824609 unique ideals
commencing full merge
memory use: 3119.0 MB
found 60245546 cycles, need 8079475
weight of 8079475 cycles is about 130110643 (16.10/cycle)
distribution of cycle lengths:
1 relations: 5326706
2 relations: 2752769
3 relations: 0
4 relations: 0
5 relations: 0
6 relations: 0
7 relations: 0
8 relations: 0
9 relations: 0
10+ relations: 0

Last fiddled with by fivemack on 2011-10-27 at 15:44
fivemack is offline   Reply With Quote
Old 2011-10-27, 16:27   #2
jasonp
Tribal Bullet
 
jasonp's Avatar
 
Oct 2004

DD016 Posts
Default

It looks like this job was so oversieved that you ran out of cliques to prune, so that there was a massive amount of excess going into the merge phase. For reasons I still have trouble understanding, making the filtering work in the face of that is almost impossible.
jasonp is offline   Reply With Quote
Old 2011-10-27, 18:52   #3
Batalov
 
Batalov's Avatar
 
"Serge"
Mar 2008
Phi(4,2^7658614+1)/2

24AD16 Posts
Default

Capt Obvious adds that doubling rel set didn't cause this behaviour. The single set would have given the same result (and then on to Jason's message, which sums up to that this is an odd old bug).
Batalov is offline   Reply With Quote
Old 2011-10-27, 22:56   #4
fivemack
(loop (#_fork))
 
fivemack's Avatar
 
Feb 2006
Cambridge, England

13×491 Posts
Default

The single set works fine - it's a bit over sieved, 95M for something where 83M is adequate, but it's produced a perfectly fine matrix which is running happily overnight. That's why I made the bug report - it looks as if there's something very odd going on in the interaction between duplicate removal and picking up the medium-sized ideals.
fivemack is offline   Reply With Quote
Old 2011-10-28, 00:05   #5
Batalov
 
Batalov's Avatar
 
"Serge"
Mar 2008
Phi(4,2^7658614+1)/2

24AD16 Posts
Default

Oh. Right! This is where it gets screwy - much earlier than the end:
Code:
commencing in-memory singleton removal
begin with 129598530 relations and 78900649 unique ideals
(when the program already knew from above "remdups"-msieve-style that there are only 85102257 unique relations)
Batalov is offline   Reply With Quote
Old 2011-10-28, 23:19   #6
jasonp
Tribal Bullet
 
jasonp's Avatar
 
Oct 2004

24·13·17 Posts
Default

Yeah, missed that the first time I read it. Ouch, no clue what it could be, especially since getting rid of the duplicates makes it work without issue.
jasonp is offline   Reply With Quote
Reply

Thread Tools


All times are UTC. The time now is 12:13.

Fri Apr 16 12:13:06 UTC 2021 up 8 days, 6:53, 0 users, load averages: 2.49, 2.08, 2.13

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.