20111027, 15:43  #1 
(loop (#_fork))
Feb 2006
Cambridge, England
13×491 Posts 
Odd bug
OK, the answer is probably 'don't do that', but I inadvertently put two copies of all the relations for a 30bit LP job into the msieve.dat file one after the other.
All seemed to go reasonably for a little while, but something went disastrously wrong when reading the small ideals: Code:
found 98209592 hash collisions in 196628532 relations commencing duplicate removal, pass 2 found 111526275 duplicates and 85102257 unique relations memory use: 2387.0 MB reading ideals above 65994752 commencing singleton removal, initial pass memory use: 1506.0 MB reading all ideals from disk memory use: 1383.8 MB commencing inmemory singleton removal begin with 85102257 relations and 75757943 unique ideals reduce to 44582950 relations and 29695136 ideals in 17 passes max relations containing the same ideal: 19 reading ideals above 720000 commencing singleton removal, initial pass memory use: 1506.0 MB reading all ideals from disk memory use: 4691.7 MB keeping 78900649 ideals with weight <= 200, target excess is 1143275 commencing inmemory singleton removal begin with 129598530 relations and 78900649 unique ideals reduce to 98566756 relations and 41906339 ideals in 12 passes max relations containing the same ideal: 182 removing 4232495 relations and 2412831 ideals in 2000000 cliques commencing inmemory singleton removal begin with 94334261 relations and 41906339 unique ideals reduce to 93842704 relations and 38704399 ideals in 4 passes max relations containing the same ideal: 174 removing 4000128 relations and 2500812 ideals in 2000000 cliques commencing inmemory singleton removal begin with 89842576 relations and 38704399 unique ideals reduce to 89806370 relations and 36165826 ideals in 3 passes max relations containing the same ideal: 172 removing 406036 relations and 235616 ideals in 203017 cliques commencing inmemory singleton removal begin with 89400334 relations and 36165826 unique ideals reduce to 89393134 relations and 35922943 ideals in 3 passes max relations containing the same ideal: 172 removing 103276 relations and 51666 ideals in 51638 cliques commencing inmemory singleton removal begin with 89289858 relations and 35922943 unique ideals reduce to 89286786 relations and 35868196 ideals in 2 passes max relations containing the same ideal: 172 removing 48472 relations and 24246 ideals in 24236 cliques commencing inmemory singleton removal begin with 89238314 relations and 35868196 unique ideals reduce to 89236868 relations and 35842502 ideals in 2 passes max relations containing the same ideal: 172 removing 23432 relations and 11717 ideals in 11716 cliques commencing inmemory singleton removal begin with 89213436 relations and 35842502 unique ideals reduce to 89212760 relations and 35830109 ideals in 2 passes max relations containing the same ideal: 172 relations with 0 large ideals: 53569 relations with 1 large ideals: 716279 relations with 2 large ideals: 4235090 relations with 3 large ideals: 13643674 relations with 4 large ideals: 25150395 relations with 5 large ideals: 26037561 relations with 6 large ideals: 14401854 relations with 7+ large ideals: 4974338 commencing 2way merge reduce to 89207260 relation sets and 35824609 unique ideals commencing full merge memory use: 3119.0 MB found 60245546 cycles, need 8079475 weight of 8079475 cycles is about 130110643 (16.10/cycle) distribution of cycle lengths: 1 relations: 5326706 2 relations: 2752769 3 relations: 0 4 relations: 0 5 relations: 0 6 relations: 0 7 relations: 0 8 relations: 0 9 relations: 0 10+ relations: 0 Last fiddled with by fivemack on 20111027 at 15:44 
20111027, 16:27  #2 
Tribal Bullet
Oct 2004
DD0_{16} Posts 
It looks like this job was so oversieved that you ran out of cliques to prune, so that there was a massive amount of excess going into the merge phase. For reasons I still have trouble understanding, making the filtering work in the face of that is almost impossible.

20111027, 18:52  #3 
"Serge"
Mar 2008
Phi(4,2^7658614+1)/2
24AD_{16} Posts 
Capt Obvious adds that doubling rel set didn't cause this behaviour. The single set would have given the same result (and then on to Jason's message, which sums up to that this is an

20111027, 22:56  #4 
(loop (#_fork))
Feb 2006
Cambridge, England
13×491 Posts 
The single set works fine  it's a bit over sieved, 95M for something where 83M is adequate, but it's produced a perfectly fine matrix which is running happily overnight. That's why I made the bug report  it looks as if there's something very odd going on in the interaction between duplicate removal and picking up the mediumsized ideals.

20111028, 00:05  #5 
"Serge"
Mar 2008
Phi(4,2^7658614+1)/2
24AD_{16} Posts 
Oh. Right! This is where it gets screwy  much earlier than the end:
Code:
commencing inmemory singleton removal begin with 129598530 relations and 78900649 unique ideals 
20111028, 23:19  #6 
Tribal Bullet
Oct 2004
2^{4}·13·17 Posts 
Yeah, missed that the first time I read it. Ouch, no clue what it could be, especially since getting rid of the duplicates makes it work without issue.
