Odd bug

fivemack · 2011-10-27, 15:43

OK, the answer is probably 'don't do that', but I inadvertently put two copies of all the relations for a 30-bit LP job into the msieve.dat file one after the other.

All seemed to go reasonably for a little while, but something went disastrously wrong when reading the small ideals:

Code:

found 98209592 hash collisions in 196628532 relations
commencing duplicate removal, pass 2
found 111526275 duplicates and 85102257 unique relations
memory use: 2387.0 MB
reading ideals above 65994752
commencing singleton removal, initial pass
memory use: 1506.0 MB
reading all ideals from disk
memory use: 1383.8 MB
commencing in-memory singleton removal
begin with 85102257 relations and 75757943 unique ideals
reduce to 44582950 relations and 29695136 ideals in 17 passes
max relations containing the same ideal: 19
reading ideals above 720000
commencing singleton removal, initial pass
memory use: 1506.0 MB
reading all ideals from disk
memory use: 4691.7 MB
keeping 78900649 ideals with weight <= 200, target excess is 1143275
commencing in-memory singleton removal
begin with 129598530 relations and 78900649 unique ideals
reduce to 98566756 relations and 41906339 ideals in 12 passes
max relations containing the same ideal: 182
removing 4232495 relations and 2412831 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 94334261 relations and 41906339 unique ideals
reduce to 93842704 relations and 38704399 ideals in 4 passes
max relations containing the same ideal: 174
removing 4000128 relations and 2500812 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 89842576 relations and 38704399 unique ideals
reduce to 89806370 relations and 36165826 ideals in 3 passes
max relations containing the same ideal: 172
removing 406036 relations and 235616 ideals in 203017 cliques
commencing in-memory singleton removal
begin with 89400334 relations and 36165826 unique ideals
reduce to 89393134 relations and 35922943 ideals in 3 passes
max relations containing the same ideal: 172
removing 103276 relations and 51666 ideals in 51638 cliques
commencing in-memory singleton removal
begin with 89289858 relations and 35922943 unique ideals
reduce to 89286786 relations and 35868196 ideals in 2 passes
max relations containing the same ideal: 172
removing 48472 relations and 24246 ideals in 24236 cliques
commencing in-memory singleton removal
begin with 89238314 relations and 35868196 unique ideals
reduce to 89236868 relations and 35842502 ideals in 2 passes
max relations containing the same ideal: 172
removing 23432 relations and 11717 ideals in 11716 cliques
commencing in-memory singleton removal
begin with 89213436 relations and 35842502 unique ideals
reduce to 89212760 relations and 35830109 ideals in 2 passes
max relations containing the same ideal: 172
relations with 0 large ideals: 53569
relations with 1 large ideals: 716279
relations with 2 large ideals: 4235090
relations with 3 large ideals: 13643674
relations with 4 large ideals: 25150395
relations with 5 large ideals: 26037561
relations with 6 large ideals: 14401854
relations with 7+ large ideals: 4974338
commencing 2-way merge
reduce to 89207260 relation sets and 35824609 unique ideals
commencing full merge
memory use: 3119.0 MB
found 60245546 cycles, need 8079475
weight of 8079475 cycles is about 130110643 (16.10/cycle)
distribution of cycle lengths:
1 relations: 5326706
2 relations: 2752769
3 relations: 0
4 relations: 0
5 relations: 0
6 relations: 0
7 relations: 0
8 relations: 0
9 relations: 0
10+ relations: 0

jasonp · 2011-10-27, 16:27

It looks like this job was so oversieved that you ran out of cliques to prune, so that there was a massive amount of excess going into the merge phase. For reasons I still have trouble understanding, making the filtering work in the face of that is almost impossible.

Batalov · 2011-10-27, 18:52

Capt Obvious adds that doubling rel set didn't cause this behaviour. The single set would have given the same result (and then on to Jason's message, which sums up to that this is an ~~odd~~ old bug).

fivemack · 2011-10-27, 22:56

The single set works fine - it's a bit over sieved, 95M for something where 83M is adequate, but it's produced a perfectly fine matrix which is running happily overnight. That's why I made the bug report - it looks as if there's something very odd going on in the interaction between duplicate removal and picking up the medium-sized ideals.

Batalov · 2011-10-28, 00:05

Oh. Right! This is where it gets screwy - much earlier than the end:

Code:

commencing in-memory singleton removal
begin with 129598530 relations and 78900649 unique ideals

(when the program already knew from above "remdups"-msieve-style that there are only 85102257 unique relations)

jasonp · 2011-10-28, 23:19

Yeah, missed that the first time I read it. Ouch, no clue what it could be, especially since getting rid of the duplicates makes it work without issue.

2011-10-27, 16:27	#2
jasonp Tribal Bullet Oct 2004 3,541 Posts	It looks like this job was so oversieved that you ran out of cliques to prune, so that there was a massive amount of excess going into the merge phase. For reasons I still have trouble understanding, making the filtering work in the face of that is almost impossible.

2011-10-27, 18:52	#3
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 9477₁₀ Posts	Capt Obvious adds that doubling rel set didn't cause this behaviour. The single set would have given the same result (and then on to Jason's message, which sums up to that this is an ~~odd~~ old bug).

2011-10-27, 22:56	#4
fivemack (loop (#_fork)) Feb 2006 Cambridge, England 7²×131 Posts	The single set works fine - it's a bit over sieved, 95M for something where 83M is adequate, but it's produced a perfectly fine matrix which is running happily overnight. That's why I made the bug report - it looks as if there's something very odd going on in the interaction between duplicate removal and picking up the medium-sized ideals.

2011-10-28, 00:05	#5
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 3⁶·13 Posts	Oh. Right! This is where it gets screwy - much earlier than the end: Code: commencing in-memory singleton removal begin with 129598530 relations and 78900649 unique ideals (when the program already knew from above "remdups"-msieve-style that there are only 85102257 unique relations)

2011-10-28, 23:19	#6
jasonp Tribal Bullet Oct 2004 3541₁₀ Posts	Yeah, missed that the first time I read it. Ouch, no clue what it could be, especially since getting rid of the duplicates makes it work without issue.