v1.40 patch for massive NFS oversieving

jasonp · 2009-03-30, 03:58

If you can build msieve v1.40 and have a NFS factorization lying around that is very heavily oversieved, could you try replacing the common/filter directory with the attached, then rerunning the filtering for your job? The patch will detect cases where you have a large number of excess relations, then forces the removal of the required number of relations and reruns the singleton removal. Hopefully this can make msieve's NFS postprocessing work in cases where users would ordinarily have to delete a bunch of relations manually.

PS: If you still have the .s file from the last time you attempted the filtering, I can also walk you through the changes needed in gnfs/filter/filter.c so that you don't have to spend all the time needed to rerun duplicate removal or coarse singleton removal.

frmky · 2009-03-30, 18:32

It didn't seem to make any difference. Here's the log with the patch.

Code:

Mon Mar 30 11:18:57 2009  Msieve v. 1.40
Mon Mar 30 11:18:57 2009  random seeds: 756666bf 744ca5f5
Mon Mar 30 11:18:57 2009  factoring 2298593707803896646567685037631740336061618006569611635364710850989523321245130212251342764710553157 (100 digits)
Mon Mar 30 11:18:57 2009  no P-1/P+1/ECM available, skipping
Mon Mar 30 11:18:57 2009  commencing number field sieve (100-digit input)
Mon Mar 30 11:18:57 2009  R0: -14587312894086558696
Mon Mar 30 11:18:57 2009  R1:  3231702353
Mon Mar 30 11:18:57 2009  A0:  28756532880986067220512685
Mon Mar 30 11:18:57 2009  A1:  4408745270505337006053
Mon Mar 30 11:18:57 2009  A2: -212966509482185501
Mon Mar 30 11:18:57 2009  A3: -10144220854233
Mon Mar 30 11:18:57 2009  A4:  234264076
Mon Mar 30 11:18:57 2009  A5:  3480
Mon Mar 30 11:18:57 2009  skew 1.00, size 1.487336e-09, alpha -6.096969, combined = 1.226915e-10
Mon Mar 30 11:18:57 2009  
Mon Mar 30 11:18:57 2009  commencing relation filtering
Mon Mar 30 11:18:57 2009  commencing duplicate removal, pass 1
Mon Mar 30 11:18:59 2009  error -5 reading relation 181658
Mon Mar 30 11:19:10 2009  error -5 reading relation 1502639
Mon Mar 30 11:19:25 2009  error -5 reading relation 3251271
Mon Mar 30 11:19:26 2009  error -5 reading relation 3404685
Mon Mar 30 11:19:30 2009  error -5 reading relation 3885137
Mon Mar 30 11:19:33 2009  error -5 reading relation 4314533
Mon Mar 30 11:19:46 2009  error -5 reading relation 5897499
Mon Mar 30 11:19:50 2009  error -5 reading relation 6403977
Mon Mar 30 11:19:52 2009  found 484156 hash collisions in 6644259 relations
Mon Mar 30 11:20:08 2009  commencing duplicate removal, pass 2
Mon Mar 30 11:20:12 2009  found 433812 duplicates and 6210447 unique relations
Mon Mar 30 11:20:12 2009  memory use: 41.3 MB
Mon Mar 30 11:20:12 2009  reading rational ideals above 1638400
Mon Mar 30 11:20:12 2009  reading algebraic ideals above 1638400
Mon Mar 30 11:20:12 2009  commencing singleton removal, pass 1
Mon Mar 30 11:21:04 2009  relations with 0 large ideals: 74170
Mon Mar 30 11:21:04 2009  relations with 1 large ideals: 594893
Mon Mar 30 11:21:04 2009  relations with 2 large ideals: 1853209
Mon Mar 30 11:21:04 2009  relations with 3 large ideals: 2465954
Mon Mar 30 11:21:04 2009  relations with 4 large ideals: 1177509
Mon Mar 30 11:21:04 2009  relations with 5 large ideals: 12930
Mon Mar 30 11:21:04 2009  relations with 6 large ideals: 31782
Mon Mar 30 11:21:04 2009  relations with 7+ large ideals: 0
Mon Mar 30 11:21:04 2009  6210447 relations and about 5475672 large ideals
Mon Mar 30 11:21:04 2009  commencing singleton removal, pass 2
Mon Mar 30 11:21:56 2009  found 2053331 singletons
Mon Mar 30 11:21:56 2009  current dataset: 4157116 relations and about 3084389 large ideals
Mon Mar 30 11:21:56 2009  commencing singleton removal, pass 3
Mon Mar 30 11:22:32 2009  found 487400 singletons
Mon Mar 30 11:22:32 2009  current dataset: 3669716 relations and about 2574290 large ideals
Mon Mar 30 11:22:32 2009  commencing singleton removal, final pass
Mon Mar 30 11:23:06 2009  memory use: 66.6 MB
Mon Mar 30 11:23:06 2009  commencing in-memory singleton removal
Mon Mar 30 11:23:07 2009  begin with 3669716 relations and 2655411 unique ideals
Mon Mar 30 11:23:12 2009  reduce to 3372430 relations and 2352948 ideals in 11 passes
Mon Mar 30 11:23:12 2009  max relations containing the same ideal: 26
Mon Mar 30 11:23:13 2009  reading rational ideals above 720000
Mon Mar 30 11:23:13 2009  reading algebraic ideals above 720000
Mon Mar 30 11:23:13 2009  commencing singleton removal, final pass
Mon Mar 30 11:23:48 2009  keeping 2411341 ideals with weight <= 20, new excess is 189584
Mon Mar 30 11:23:49 2009  memory use: 89.9 MB
Mon Mar 30 11:23:49 2009  commencing in-memory singleton removal
Mon Mar 30 11:23:50 2009  begin with 3372438 relations and 2411341 unique ideals
Mon Mar 30 11:23:51 2009  reduce to 3372428 relations and 2411294 ideals in 2 passes
Mon Mar 30 11:23:51 2009  max relations containing the same ideal: 20
Mon Mar 30 11:23:53 2009  removing 1201443 relations and 830835 ideals in 370608 cliques
Mon Mar 30 11:23:54 2009  commencing in-memory singleton removal
Mon Mar 30 11:23:54 2009  begin with 2170985 relations and 2411294 unique ideals
Mon Mar 30 11:23:57 2009  reduce to 1959457 relations and 1339466 ideals in 8 passes
Mon Mar 30 11:23:57 2009  max relations containing the same ideal: 19
Mon Mar 30 11:23:58 2009  removing 907500 relations and 536892 ideals in 370608 cliques
Mon Mar 30 11:23:59 2009  commencing in-memory singleton removal
Mon Mar 30 11:23:59 2009  begin with 1051957 relations and 1339466 unique ideals
Mon Mar 30 11:24:00 2009  reduce to 871986 relations and 592046 ideals in 11 passes
Mon Mar 30 11:24:00 2009  max relations containing the same ideal: 14
Mon Mar 30 11:24:01 2009  removing 214270 relations and 154248 ideals in 60022 cliques
Mon Mar 30 11:24:01 2009  commencing in-memory singleton removal
Mon Mar 30 11:24:01 2009  begin with 657716 relations and 592046 unique ideals
Mon Mar 30 11:24:01 2009  reduce to 619575 relations and 396689 ideals in 10 passes
Mon Mar 30 11:24:01 2009  max relations containing the same ideal: 14
Mon Mar 30 11:24:02 2009  relations with 0 large ideals: 47381
Mon Mar 30 11:24:02 2009  relations with 1 large ideals: 182399
Mon Mar 30 11:24:02 2009  relations with 2 large ideals: 234444
Mon Mar 30 11:24:02 2009  relations with 3 large ideals: 126418
Mon Mar 30 11:24:02 2009  relations with 4 large ideals: 27118
Mon Mar 30 11:24:02 2009  relations with 5 large ideals: 1496
Mon Mar 30 11:24:02 2009  relations with 6 large ideals: 319
Mon Mar 30 11:24:02 2009  relations with 7+ large ideals: 0
Mon Mar 30 11:24:02 2009  commencing 2-way merge
Mon Mar 30 11:24:02 2009  reduce to 419565 relation sets and 196679 unique ideals
Mon Mar 30 11:24:02 2009  commencing full merge
Mon Mar 30 11:24:04 2009  memory use: 17.9 MB
Mon Mar 30 11:24:04 2009  found 221693 cycles, need 191284
Mon Mar 30 11:24:05 2009  weight of 191284 cycles is about 9990697 (52.23/cycle)
Mon Mar 30 11:24:05 2009  distribution of cycle lengths:
Mon Mar 30 11:24:05 2009  1 relations: 47383
Mon Mar 30 11:24:05 2009  2 relations: 25592
Mon Mar 30 11:24:05 2009  3 relations: 20032
Mon Mar 30 11:24:05 2009  4 relations: 16725
Mon Mar 30 11:24:05 2009  5 relations: 14606
Mon Mar 30 11:24:05 2009  6 relations: 12758
Mon Mar 30 11:24:05 2009  7 relations: 11205
Mon Mar 30 11:24:05 2009  8 relations: 10009
Mon Mar 30 11:24:05 2009  9 relations: 8615
Mon Mar 30 11:24:05 2009  10+ relations: 24359
Mon Mar 30 11:24:05 2009  heaviest cycle: 15 relations
Mon Mar 30 11:24:05 2009  matrix can improve, retrying
Mon Mar 30 11:24:05 2009  reading rational ideals above 720000
Mon Mar 30 11:24:05 2009  reading algebraic ideals above 720000
Mon Mar 30 11:24:05 2009  commencing singleton removal, final pass
Mon Mar 30 11:24:40 2009  keeping 2419476 ideals with weight <= 25, new excess is 181449
Mon Mar 30 11:24:41 2009  memory use: 89.9 MB
Mon Mar 30 11:24:41 2009  commencing in-memory singleton removal
Mon Mar 30 11:24:42 2009  begin with 3372438 relations and 2419476 unique ideals
Mon Mar 30 11:24:43 2009  reduce to 3372428 relations and 2419429 ideals in 2 passes
Mon Mar 30 11:24:43 2009  max relations containing the same ideal: 25
Mon Mar 30 11:24:46 2009  removing 1202745 relations and 831486 ideals in 371259 cliques
Mon Mar 30 11:24:46 2009  commencing in-memory singleton removal
Mon Mar 30 11:24:47 2009  begin with 2169683 relations and 2419429 unique ideals
Mon Mar 30 11:24:49 2009  reduce to 1957751 relations and 1346429 ideals in 8 passes
Mon Mar 30 11:24:49 2009  max relations containing the same ideal: 24
Mon Mar 30 11:24:51 2009  removing 908686 relations and 537427 ideals in 371259 cliques
Mon Mar 30 11:24:51 2009  commencing in-memory singleton removal
Mon Mar 30 11:24:51 2009  begin with 1049065 relations and 1346429 unique ideals
Mon Mar 30 11:24:53 2009  reduce to 868959 relations and 598274 ideals in 11 passes
Mon Mar 30 11:24:53 2009  max relations containing the same ideal: 16
Mon Mar 30 11:24:53 2009  removing 214855 relations and 154651 ideals in 60204 cliques
Mon Mar 30 11:24:53 2009  commencing in-memory singleton removal
Mon Mar 30 11:24:53 2009  begin with 654104 relations and 598274 unique ideals
Mon Mar 30 11:24:54 2009  reduce to 615401 relations and 401901 ideals in 10 passes
Mon Mar 30 11:24:54 2009  max relations containing the same ideal: 16
Mon Mar 30 11:24:54 2009  relations with 0 large ideals: 42520
Mon Mar 30 11:24:54 2009  relations with 1 large ideals: 170336
Mon Mar 30 11:24:54 2009  relations with 2 large ideals: 231852
Mon Mar 30 11:24:54 2009  relations with 3 large ideals: 135573
Mon Mar 30 11:24:54 2009  relations with 4 large ideals: 32606
Mon Mar 30 11:24:54 2009  relations with 5 large ideals: 2185
Mon Mar 30 11:24:54 2009  relations with 6 large ideals: 329
Mon Mar 30 11:24:54 2009  relations with 7+ large ideals: 0
Mon Mar 30 11:24:54 2009  commencing 2-way merge
Mon Mar 30 11:24:55 2009  reduce to 416090 relation sets and 202590 unique ideals
Mon Mar 30 11:24:55 2009  commencing full merge
Mon Mar 30 11:24:57 2009  memory use: 18.0 MB
Mon Mar 30 11:24:57 2009  found 215728 cycles, need 186149
Mon Mar 30 11:24:57 2009  weight of 186149 cycles is about 10149212 (54.52/cycle)
Mon Mar 30 11:24:57 2009  distribution of cycle lengths:
Mon Mar 30 11:24:57 2009  1 relations: 42543
Mon Mar 30 11:24:57 2009  2 relations: 23129
Mon Mar 30 11:24:57 2009  3 relations: 18917
Mon Mar 30 11:24:57 2009  4 relations: 16541
Mon Mar 30 11:24:57 2009  5 relations: 14791
Mon Mar 30 11:24:57 2009  6 relations: 13238
Mon Mar 30 11:24:57 2009  7 relations: 11706
Mon Mar 30 11:24:57 2009  8 relations: 10480
Mon Mar 30 11:24:57 2009  9 relations: 9151
Mon Mar 30 11:24:57 2009  10+ relations: 25653
Mon Mar 30 11:24:57 2009  heaviest cycle: 15 relations
Mon Mar 30 11:24:57 2009  matrix can improve, retrying
Mon Mar 30 11:24:58 2009  reading rational ideals above 720000
Mon Mar 30 11:24:58 2009  reading algebraic ideals above 720000
Mon Mar 30 11:24:58 2009  commencing singleton removal, final pass
Mon Mar 30 11:25:32 2009  keeping 2422294 ideals with weight <= 30, new excess is 178631
Mon Mar 30 11:25:34 2009  memory use: 89.9 MB
Mon Mar 30 11:25:34 2009  commencing in-memory singleton removal
Mon Mar 30 11:25:34 2009  begin with 3372438 relations and 2422294 unique ideals
Mon Mar 30 11:25:36 2009  reduce to 3372428 relations and 2422247 ideals in 2 passes
Mon Mar 30 11:25:36 2009  max relations containing the same ideal: 30
Mon Mar 30 11:25:38 2009  removing 1203197 relations and 831712 ideals in 371485 cliques
Mon Mar 30 11:25:39 2009  commencing in-memory singleton removal
Mon Mar 30 11:25:39 2009  begin with 2169231 relations and 2422247 unique ideals
Mon Mar 30 11:25:42 2009  reduce to 1957149 relations and 1348826 ideals in 8 passes
Mon Mar 30 11:25:42 2009  max relations containing the same ideal: 26
Mon Mar 30 11:25:43 2009  removing 909079 relations and 537594 ideals in 371485 cliques
Mon Mar 30 11:25:44 2009  commencing in-memory singleton removal
Mon Mar 30 11:25:44 2009  begin with 1048070 relations and 1348826 unique ideals
Mon Mar 30 11:25:45 2009  reduce to 867858 relations and 600388 ideals in 11 passes
Mon Mar 30 11:25:45 2009  max relations containing the same ideal: 19
Mon Mar 30 11:25:46 2009  removing 214969 relations and 154711 ideals in 60258 cliques
Mon Mar 30 11:25:46 2009  commencing in-memory singleton removal
Mon Mar 30 11:25:46 2009  begin with 652889 relations and 600388 unique ideals
Mon Mar 30 11:25:47 2009  reduce to 614038 relations and 403787 ideals in 10 passes
Mon Mar 30 11:25:47 2009  max relations containing the same ideal: 16
Mon Mar 30 11:25:47 2009  relations with 0 large ideals: 40668
Mon Mar 30 11:25:47 2009  relations with 1 large ideals: 166034
Mon Mar 30 11:25:47 2009  relations with 2 large ideals: 230429
Mon Mar 30 11:25:47 2009  relations with 3 large ideals: 138982
Mon Mar 30 11:25:47 2009  relations with 4 large ideals: 34939
Mon Mar 30 11:25:47 2009  relations with 5 large ideals: 2637
Mon Mar 30 11:25:47 2009  relations with 6 large ideals: 349
Mon Mar 30 11:25:47 2009  relations with 7+ large ideals: 0
Mon Mar 30 11:25:47 2009  commencing 2-way merge
Mon Mar 30 11:25:47 2009  reduce to 414955 relation sets and 204704 unique ideals
Mon Mar 30 11:25:47 2009  commencing full merge
Mon Mar 30 11:25:50 2009  memory use: 18.1 MB
Mon Mar 30 11:25:50 2009  found 213403 cycles, need 184331
Mon Mar 30 11:25:50 2009  weight of 184331 cycles is about 10275864 (55.75/cycle)
Mon Mar 30 11:25:50 2009  distribution of cycle lengths:
Mon Mar 30 11:25:50 2009  1 relations: 40716
Mon Mar 30 11:25:50 2009  2 relations: 22367
Mon Mar 30 11:25:50 2009  3 relations: 18498
Mon Mar 30 11:25:50 2009  4 relations: 16312
Mon Mar 30 11:25:50 2009  5 relations: 14702
Mon Mar 30 11:25:50 2009  6 relations: 13183
Mon Mar 30 11:25:50 2009  7 relations: 11823
Mon Mar 30 11:25:50 2009  8 relations: 10651
Mon Mar 30 11:25:50 2009  9 relations: 9268
Mon Mar 30 11:25:50 2009  10+ relations: 26811
Mon Mar 30 11:25:50 2009  heaviest cycle: 15 relations
Mon Mar 30 11:25:50 2009  matrix can improve, retrying
Mon Mar 30 11:25:50 2009  reading rational ideals above 720000
Mon Mar 30 11:25:50 2009  reading algebraic ideals above 720000
Mon Mar 30 11:25:50 2009  commencing singleton removal, final pass
Mon Mar 30 11:26:25 2009  keeping 2425114 ideals with weight <= 35, new excess is 175811
Mon Mar 30 11:26:27 2009  memory use: 89.9 MB
Mon Mar 30 11:26:27 2009  commencing in-memory singleton removal
Mon Mar 30 11:26:27 2009  begin with 3372438 relations and 2425114 unique ideals
Mon Mar 30 11:26:28 2009  reduce to 3372428 relations and 2425067 ideals in 2 passes
Mon Mar 30 11:26:28 2009  max relations containing the same ideal: 35
Mon Mar 30 11:26:31 2009  removing 1203647 relations and 831937 ideals in 371710 cliques
Mon Mar 30 11:26:32 2009  commencing in-memory singleton removal
Mon Mar 30 11:26:32 2009  begin with 2168781 relations and 2425067 unique ideals
Mon Mar 30 11:26:35 2009  reduce to 1956513 relations and 1351202 ideals in 8 passes
Mon Mar 30 11:26:35 2009  max relations containing the same ideal: 30
Mon Mar 30 11:26:36 2009  removing 909476 relations and 537766 ideals in 371710 cliques
Mon Mar 30 11:26:37 2009  commencing in-memory singleton removal
Mon Mar 30 11:26:37 2009  begin with 1047037 relations and 1351202 unique ideals
Mon Mar 30 11:26:38 2009  reduce to 866760 relations and 602515 ideals in 11 passes
Mon Mar 30 11:26:38 2009  max relations containing the same ideal: 21
Mon Mar 30 11:26:39 2009  removing 215092 relations and 154788 ideals in 60304 cliques
Mon Mar 30 11:26:39 2009  commencing in-memory singleton removal
Mon Mar 30 11:26:39 2009  begin with 651668 relations and 602515 unique ideals
Mon Mar 30 11:26:39 2009  reduce to 612655 relations and 405658 ideals in 10 passes
Mon Mar 30 11:26:39 2009  max relations containing the same ideal: 17
Mon Mar 30 11:26:40 2009  relations with 0 large ideals: 38885
Mon Mar 30 11:26:40 2009  relations with 1 large ideals: 161417
Mon Mar 30 11:26:40 2009  relations with 2 large ideals: 228574
Mon Mar 30 11:26:40 2009  relations with 3 large ideals: 142278
Mon Mar 30 11:26:40 2009  relations with 4 large ideals: 37731
Mon Mar 30 11:26:40 2009  relations with 5 large ideals: 3371
Mon Mar 30 11:26:40 2009  relations with 6 large ideals: 399
Mon Mar 30 11:26:40 2009  relations with 7+ large ideals: 0
Mon Mar 30 11:26:40 2009  commencing 2-way merge
Mon Mar 30 11:26:40 2009  reduce to 413926 relation sets and 206929 unique ideals
Mon Mar 30 11:26:40 2009  commencing full merge
Mon Mar 30 11:26:43 2009  memory use: 18.2 MB
Mon Mar 30 11:26:43 2009  found 211037 cycles, need 182511
Mon Mar 30 11:26:43 2009  weight of 182511 cycles is about 10434546 (57.17/cycle)
Mon Mar 30 11:26:43 2009  distribution of cycle lengths:
Mon Mar 30 11:26:43 2009  1 relations: 38971
Mon Mar 30 11:26:43 2009  2 relations: 21545
Mon Mar 30 11:26:43 2009  3 relations: 18042
Mon Mar 30 11:26:43 2009  4 relations: 16147
Mon Mar 30 11:26:43 2009  5 relations: 14454
Mon Mar 30 11:26:43 2009  6 relations: 13166
Mon Mar 30 11:26:43 2009  7 relations: 11798
Mon Mar 30 11:26:43 2009  8 relations: 10656
Mon Mar 30 11:26:43 2009  9 relations: 9317
Mon Mar 30 11:26:43 2009  10+ relations: 28415
Mon Mar 30 11:26:43 2009  heaviest cycle: 16 relations
Mon Mar 30 11:26:43 2009  matrix can improve, retrying
Mon Mar 30 11:26:43 2009  reading rational ideals above 720000
Mon Mar 30 11:26:43 2009  reading algebraic ideals above 720000
Mon Mar 30 11:26:43 2009  commencing singleton removal, final pass
Mon Mar 30 11:27:18 2009  keeping 2429226 ideals with weight <= 40, new excess is 171699
Mon Mar 30 11:27:20 2009  memory use: 89.9 MB
Mon Mar 30 11:27:20 2009  commencing in-memory singleton removal
Mon Mar 30 11:27:20 2009  begin with 3372438 relations and 2429226 unique ideals
Mon Mar 30 11:27:21 2009  reduce to 3372428 relations and 2429179 ideals in 2 passes
Mon Mar 30 11:27:21 2009  max relations containing the same ideal: 40
Mon Mar 30 11:27:24 2009  removing 1204305 relations and 832266 ideals in 372039 cliques
Mon Mar 30 11:27:25 2009  commencing in-memory singleton removal
Mon Mar 30 11:27:25 2009  begin with 2168123 relations and 2429179 unique ideals
Mon Mar 30 11:27:28 2009  reduce to 1955604 relations and 1354687 ideals in 8 passes
Mon Mar 30 11:27:28 2009  max relations containing the same ideal: 34
Mon Mar 30 11:27:29 2009  removing 910069 relations and 538030 ideals in 372039 cliques
Mon Mar 30 11:27:30 2009  commencing in-memory singleton removal
Mon Mar 30 11:27:30 2009  begin with 1045535 relations and 1354687 unique ideals
Mon Mar 30 11:27:31 2009  reduce to 865220 relations and 605675 ideals in 11 passes
Mon Mar 30 11:27:31 2009  max relations containing the same ideal: 21
Mon Mar 30 11:27:32 2009  removing 215362 relations and 154988 ideals in 60374 cliques
Mon Mar 30 11:27:32 2009  commencing in-memory singleton removal
Mon Mar 30 11:27:32 2009  begin with 649858 relations and 605675 unique ideals
Mon Mar 30 11:27:32 2009  reduce to 610669 relations and 408415 ideals in 10 passes
Mon Mar 30 11:27:32 2009  max relations containing the same ideal: 19
Mon Mar 30 11:27:33 2009  relations with 0 large ideals: 36202
Mon Mar 30 11:27:33 2009  relations with 1 large ideals: 154099
Mon Mar 30 11:27:33 2009  relations with 2 large ideals: 225355
Mon Mar 30 11:27:33 2009  relations with 3 large ideals: 147119
Mon Mar 30 11:27:33 2009  relations with 4 large ideals: 42606
Mon Mar 30 11:27:33 2009  relations with 5 large ideals: 4768
Mon Mar 30 11:27:33 2009  relations with 6 large ideals: 520
Mon Mar 30 11:27:33 2009  relations with 7+ large ideals: 0
Mon Mar 30 11:27:33 2009  commencing 2-way merge
Mon Mar 30 11:27:33 2009  reduce to 412486 relation sets and 210232 unique ideals
Mon Mar 30 11:27:33 2009  commencing full merge
Mon Mar 30 11:27:36 2009  memory use: 18.3 MB
Mon Mar 30 11:27:36 2009  found 207519 cycles, need 179899
Mon Mar 30 11:27:36 2009  weight of 179899 cycles is about 10713605 (59.55/cycle)
Mon Mar 30 11:27:36 2009  distribution of cycle lengths:
Mon Mar 30 11:27:36 2009  1 relations: 36362
Mon Mar 30 11:27:36 2009  2 relations: 20432
Mon Mar 30 11:27:36 2009  3 relations: 17297
Mon Mar 30 11:27:36 2009  4 relations: 15632
Mon Mar 30 11:27:36 2009  5 relations: 14124
Mon Mar 30 11:27:36 2009  6 relations: 13049
Mon Mar 30 11:27:36 2009  7 relations: 11746
Mon Mar 30 11:27:36 2009  8 relations: 10644
Mon Mar 30 11:27:36 2009  9 relations: 9562
Mon Mar 30 11:27:36 2009  10+ relations: 31051
Mon Mar 30 11:27:36 2009  heaviest cycle: 16 relations
Mon Mar 30 11:27:36 2009  matrix not dense enough, retrying

jasonp · 2009-03-30, 18:42

The patch activates only when you have so much excess that you run out of 2-way cliques and still have more than 100k of excess above the target (i.e. the number after 'new excess is ...')

Your example is actually a medium amount of excess; there isn't enough excess to trigger the patch but it's still too much for the code to deal with. Does the filtering converge if you make the filtering bound 100000? What about changing the '100000' at the bottom of common/filter/clique.c to a smaller number like 10000? I think the latter will cause the filtering to converge but the matrix will probably end up larger than it has to be.

frmky · 2009-03-30, 19:38

Quote:

Originally Posted by jasonp

Does the filtering converge if you make the filtering bound 100000?

Yes, starting with -nc1 100000,0 worked.

bdodson · 2009-04-07, 11:31

Quote:

Originally Posted by jasonp

... forces the removal of the required number of relations and reruns the singleton removal. Hopefully this can make msieve's NFS postprocessing work in cases where users would ordinarily have to delete a bunch of relations manually.

PS: If you still have the .s file from the last time you attempted the filtering, I can also walk you through the changes needed in gnfs/filter/filter.c so that you don't have to spend all the time needed to rerun duplicate removal or coarse singleton removal.

I'm having trouble locating the thread on singleton removal and/or
pruning data; which was posed as a possible stand-alone program
(like duplicate removal), but Jason reports a preference for having
it included within msieve. So here's a test case. We had some
confusion of the intended sieving region; and then I got busy elsewhere
and over-ran a trimmed region by an extra 10M (better than the 20M
for 2-877!).

So anyway, we (B+D) were aiming for 200M unique, and I hit
227M instead. This data is supposed to go over DSL, and will
likely take 12hours, and then be trouble-somely large on arrival.
Can I run a current msieve (or some prospective utility program
intended to support msieve's filtering) to trim some 20M relns
with a better prospective matrix/filtering than just randomly leaving
off the last 20M? I'm looking for an output file with a selection of
the 227M, in the same format (txt), to be split, gzip'd and shipped
off for Batalov to fiddle with. -Bruce

10metreh · 2009-04-07, 11:34

Quote:

Originally Posted by bdodson

2-877

Do you mean 2-827 or 2-857? I thought Tom had reserved 2-877 for the forum.

jasonp · 2009-04-07, 13:25

Quote:

Originally Posted by bdodson

I'm having trouble locating the thread on singleton removal and/or
pruning data; which was posed as a possible stand-alone program
(like duplicate removal), but Jason reports a preference for having
it included within msieve. So here's a test case. We had some
confusion of the intended sieving region; and then I got busy elsewhere
and over-ran a trimmed region by an extra 10M (better than the 20M
for 2-877!).

So anyway, we (B+D) were aiming for 200M unique, and I hit
227M instead. This data is supposed to go over DSL, and will
likely take 12hours, and then be trouble-somely large on arrival.
Can I run a current msieve (or some prospective utility program
intended to support msieve's filtering) to trim some 20M relns
with a better prospective matrix/filtering than just randomly leaving
off the last 20M? I'm looking for an output file with a selection of
the 227M, in the same format (txt), to be split, gzip'd and shipped
off for Batalov to fiddle with. -Bruce

I think you mean the thread for 2,908+

Msieve v1.41 will remove 2-way cliques, and if there is still too much excess will then delete the largest relations and remove resulting singletons until the target excess (~116% of the ignored ideals) remains. Note that if this happens early you will get a matrix and it will be unusually dense, because it would have been a better idea to increase the maximum ideal weight; the code does not detect this and cannot recover.

bdodson · 2009-04-07, 13:25

Quote:

Originally Posted by 10metreh

Do you mean 2-827 or 2-857? I thought Tom had reserved 2-877 for the forum.

Yes, sorry; 2-827 is the one with an extra 20M range of qs. I've
been submitting ecm curves for 2-877, and got them mixed up. Not
2-857 which was finished, C/D. -bd

bdodson · 2009-04-07, 13:41

Quote:

Originally Posted by jasonp

I think you mean the thread for 2,908+

Msieve v1.41 will remove 2-way cliques, and if there is still too much excess will then delete the largest relations and remove resulting singletons until the target excess (~116% of the ignored ideals) remains. Note that if this happens early you will get a matrix and it will be unusually dense, because it would have been a better idea to increase the maximum ideal weight; the code does not detect this and cannot recover.

Sounds good. For the data transfer to Batalov, is there some way of
getting at a trimmed msieve.dat reflecting some of the early trimming?
Otherwise I suppose I'll just send the entire 227M. -Bruce

Jeff Gilchrist · 2009-04-07, 14:48

Quote:

Originally Posted by bdodson

Sounds good. For the data transfer to Batalov, is there some way of
getting at a trimmed msieve.dat reflecting some of the early trimming?
Otherwise I suppose I'll just send the entire 227M. -Bruce

The .dat file should get some good compression no? That should cut down on the transmission time and also act as a way to detect any corruption in the transfer as well.

jasonp · 2009-04-07, 15:40

Quote:

Originally Posted by bdodson

Sounds good. For the data transfer to Batalov, is there some way of
getting at a trimmed msieve.dat reflecting some of the early trimming?
Otherwise I suppose I'll just send the entire 227M. -Bruce

The .s file contains the line numbers of relations that survive duplicate and singleton removal (clique processing only happens on in-memory relations and the results are not saved). Line numbers start from zero and increment whenever MIN(one_line, 300_chars) are read in. I don't have the means to test it now, but it would be straightforward to write a program that pulls out the relations that survived and dumps them to a smaller dat file.

2009-03-30, 18:42	#3
jasonp Tribal Bullet Oct 2004 3541₁₀ Posts	The patch activates only when you have so much excess that you run out of 2-way cliques and still have more than 100k of excess above the target (i.e. the number after 'new excess is ...') Your example is actually a medium amount of excess; there isn't enough excess to trigger the patch but it's still too much for the code to deal with. Does the filtering converge if you make the filtering bound 100000? What about changing the '100000' at the bottom of common/filter/clique.c to a smaller number like 10000? I think the latter will cause the filtering to converge but the matrix will probably end up larger than it has to be. Last fiddled with by jasonp on 2009-03-30 at 18:54

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Would a massive cache make much difference?	tServo	Hardware	14	2018-04-01 22:31
"lanczos error: only trivial dependencies found" with massive oversieving	eigma	Msieve	21	2015-05-28 03:27
Oversieving	chris2be8	Msieve	7	2010-03-13 21:51
Oversieving in msieve	fivemack	Msieve	1	2007-09-28 18:26
Massive P727 found for (10^49081-1)/9-1	AntonVrba	Factoring	5	2005-12-26 17:02