I don't understand these relation counts
 

[code]
Wed Oct 3 10:35:56 2012 found 28112108 hash collisions in 218362761 relations
Wed Oct 3 11:36:36 2012 found 22314936 duplicates and 197266829 unique relations
Wed Oct 3 15:33:43 2012 begin with 197266829 relations and 280985260 unique ideals
------
Sun Oct 28 19:00:41 2012 found 47095370 hash collisions in 288197220 relations
Sun Oct 28 19:41:43 2012 found 39606683 duplicates and 249809541 unique relations
Sun Oct 28 21:29:32 2012 begin with 249809541 relations and 264041178 unique ideals
[/code]

The run today is using a strict superset of the relations from 3 October. I've added seventy million raw relations, and sixteen million unique ideals seem to have disappeared.

Have a lot of the ideals now got weights above 200 that hadn't before? Has the target excess shot up by 16M?

That's what I bet is happening. Either that or the large prime bound is much higher in the second case.

Yes, it was using a very different large-ideal bound the second time through, and I hadn't noticed because I read the log files with grep rather than from the top to the bottom. In the first case

[code]
Wed Oct 3 10:37:32 2012 commencing duplicate removal, pass 2
Wed Oct 3 11:36:36 2012 found 22314936 duplicates and 197266829 unique relations
Wed Oct 3 11:36:36 2012 memory use: 852.8 MB
Wed Oct 3 11:36:37 2012 reading ideals above 720000
Wed Oct 3 11:36:37 2012 commencing singleton removal, initial pass
Wed Oct 3 15:11:04 2012 memory use: 6024.0 MB
Wed Oct 3 15:11:06 2012 reading all ideals from disk
Wed Oct 3 15:19:46 2012 memory use: 8543.4 MB
Wed Oct 3 15:24:41 2012 keeping 280985260 ideals with weight <= 200, target excess is 982466
Wed Oct 3 15:30:00 2012 commencing in-memory singleton removal
Wed Oct 3 15:33:43 2012 begin with 197266829 relations and 280985260 unique ideals
Wed Oct 3 15:41:18 2012 reduce to 36019 relations and 0 ideals in 8 passes
Wed Oct 3 15:41:18 2012 max relations containing the same ideal: 0
[/code]

and in the second

[code]
Sun Oct 28 19:01:29 2012 commencing duplicate removal, pass 2
Sun Oct 28 19:41:43 2012 found 39606683 duplicates and 249809541 unique relations
Sun Oct 28 19:41:43 2012 memory use: 1321.5 MB
Sun Oct 28 19:41:43 2012 reading ideals above 428670976
Sun Oct 28 19:41:43 2012 commencing singleton removal, initial pass
Sun Oct 28 21:25:25 2012 memory use: 6024.0 MB
Sun Oct 28 21:25:26 2012 reading all ideals from disk
Sun Oct 28 21:27:15 2012 memory use: 4233.5 MB
Sun Oct 28 21:28:25 2012 commencing in-memory singleton removal
Sun Oct 28 21:29:32 2012 begin with 249809541 relations and 264041178 unique ideals
Sun Oct 28 21:41:33 2012 reduce to 60249620 relations and 38031770 ideals in 34 passes
Sun Oct 28 21:41:33 2012 max relations containing the same ideal: 20
Sun Oct 28 21:41:51 2012 reading ideals above 720000
Sun Oct 28 21:41:51 2012 commencing singleton removal, initial pass
Sun Oct 28 22:43:21 2012 memory use: 1506.0 MB
Sun Oct 28 22:43:22 2012 reading all ideals from disk
Sun Oct 28 22:44:34 2012 memory use: 2589.1 MB
Sun Oct 28 22:45:01 2012 keeping 81610960 ideals with weight <= 200, target excess is 308877
Sun Oct 28 22:45:27 2012 commencing in-memory singleton removal
Sun Oct 28 22:45:53 2012 begin with 60249620 relations and 81610960 unique ideals
Sun Oct 28 22:48:26 2012 reduce to 11516 relations and 0 ideals in 13 passes
Sun Oct 28 22:48:26 2012 max relations containing the same ideal: 0
[/code]

I've clearly misunderstood what 'reading all ideals from disk' meant - I thought it meant 'all ideals' as opposed to 'ideals above large_ideal_bound'.

It does mean 'all ideals selected to participate in the filtering', not 'all ideals'.