[QUOTE=frmky;121354]Here's the rest of the log for 10,239-. The filtering portion of the log is in an earlier post. The restart code was tested on this one. About 60% of the way through the BL, the computer went down. It restarted successfully. The sqrt again used less memory than build_matrix_core, so it currently seems that if there's enough memory to build the matrix, there's enough for sqrt.
[/QUOTE]
Awesome. The memory use of the square root will be a problem for big GNFS, since every relation always has polynomial coefficients that are log2(leading algebraic coefficient) larger than the raw size of the (a,b) pair. Multiplying millions of these relations together will double the memory use of the square root when we get to 6,383+.

I regret that I've been procrastinating adding the square root improvements, and have instead been untangling the polynomial selection code in GGNFS.

Here's the log of the filtering for 2,787-. I used msieve 1.30 since it's running on a single core anyway. The linear algebra is again expected to take about 18 days.

Greg

[CODE]Sun Dec 23 10:54:34 2007 Msieve v. 1.30
Sun Dec 23 10:54:34 2007 random seeds: 2a590e16 da1c8ed8
Sun Dec 23 10:54:34 2007 factoring 847485796604747326165327482849934275105622414539817731615514207366361791907216255429043094060871322325590021965236616107269568655093084140543593046072375681421163547606143705166808288886982781571997632241 (204 digits)
Sun Dec 23 10:54:35 2007 commencing number field sieve (204-digit input)
Sun Dec 23 10:54:35 2007 R0: -2722258935367507707706996859454145691648
Sun Dec 23 10:54:35 2007 R1: 1
Sun Dec 23 10:54:35 2007 A0: -1
Sun Dec 23 10:54:35 2007 A1: 0
Sun Dec 23 10:54:35 2007 A2: 0
Sun Dec 23 10:54:35 2007 A3: 0
Sun Dec 23 10:54:35 2007 A4: 0
Sun Dec 23 10:54:35 2007 A5: 0
Sun Dec 23 10:54:35 2007 A6: 2
Sun Dec 23 10:54:35 2007 size score = 2.490373e-11, Murphy alpha = 2.412318, combined = 1.250069e-11
Sun Dec 23 11:02:28 2007 restarting with 99010555 relations
Sun Dec 23 11:02:28 2007
Sun Dec 23 11:02:28 2007 commencing relation filtering
Sun Dec 23 11:02:28 2007 commencing duplicate removal, pass 1
... relation errors removed ...
Sun Dec 23 11:18:56 2007 found 17502445 hash collisions in 99010333 relations
Sun Dec 23 11:18:56 2007 commencing duplicate removal, pass 2
Sun Dec 23 11:27:22 2007 found 1865301 duplicates and 97145032 unique relations
Sun Dec 23 11:27:22 2007 memory use: 504.8 MB
Sun Dec 23 11:27:56 2007 ignoring smallest 3468391 rational and 3467431 algebraic ideals
Sun Dec 23 11:27:57 2007 filtering ideals above 58322523
Sun Dec 23 11:27:57 2007 need 11790897 more relations than ideals
Sun Dec 23 11:27:57 2007 commencing singleton removal, pass 1
Sun Dec 23 11:45:02 2007 relations with 0 large ideals: 959491
Sun Dec 23 11:45:02 2007 relations with 1 large ideals: 6688045
Sun Dec 23 11:45:02 2007 relations with 2 large ideals: 19923649
Sun Dec 23 11:45:02 2007 relations with 3 large ideals: 31694340
Sun Dec 23 11:45:02 2007 relations with 4 large ideals: 27262978
Sun Dec 23 11:45:02 2007 relations with 5 large ideals: 10616529
Sun Dec 23 11:45:02 2007 relations with 6 large ideals: 0
Sun Dec 23 11:45:02 2007 relations with 7+ large ideals: 0
Sun Dec 23 11:45:02 2007 97145032 relations and about 62623743 large ideals
Sun Dec 23 11:45:02 2007 commencing singleton removal, pass 2
Sun Dec 23 12:02:54 2007 found 14131925 singletons
Sun Dec 23 12:02:54 2007 current dataset: 83013107 relations and about 47640969 large ideals
Sun Dec 23 12:02:54 2007 commencing singleton removal, pass 3
Sun Dec 23 12:18:11 2007 relations with 0 large ideals: 959491
Sun Dec 23 12:18:11 2007 relations with 1 large ideals: 6346791
Sun Dec 23 12:18:11 2007 relations with 2 large ideals: 17957213
Sun Dec 23 12:18:11 2007 relations with 3 large ideals: 27174055
Sun Dec 23 12:18:11 2007 relations with 4 large ideals: 22278817
Sun Dec 23 12:18:11 2007 relations with 5 large ideals: 8296740
Sun Dec 23 12:18:11 2007 relations with 6 large ideals: 0
Sun Dec 23 12:18:11 2007 relations with 7+ large ideals: 0
Sun Dec 23 12:18:11 2007 83013107 relations and about 59845136 large ideals
Sun Dec 23 12:18:11 2007 commencing singleton removal, pass 4
Sun Dec 23 12:34:17 2007 found 11728093 singletons
Sun Dec 23 12:34:17 2007 current dataset: 71285014 relations and about 47435064 large ideals
Sun Dec 23 12:34:17 2007 commencing singleton removal, pass 5
Sun Dec 23 12:48:36 2007 found 2278398 singletons
Sun Dec 23 12:48:36 2007 current dataset: 69006616 relations and about 45126260 large ideals
Sun Dec 23 12:48:36 2007 commencing singleton removal, pass 6
Sun Dec 23 13:02:52 2007 found 397534 singletons
Sun Dec 23 13:02:52 2007 current dataset: 68609082 relations and about 44727711 large ideals
Sun Dec 23 13:02:52 2007 commencing singleton removal, final pass
Sun Dec 23 13:18:18 2007 memory use: 2581.2 MB
Sun Dec 23 13:18:18 2007 commencing in-memory singleton removal
Sun Dec 23 13:18:27 2007 begin with 68609082 relations and 51982156 unique ideals
Sun Dec 23 13:20:29 2007 reduce to 62186002 relations and 45427340 ideals in 13 passes
Sun Dec 23 13:20:29 2007 max relations containing the same ideal: 38
Sun Dec 23 13:21:11 2007 removing 2992676 relations and 2592676 ideals in 400000 cliques
Sun Dec 23 13:21:13 2007 commencing in-memory singleton removal
Sun Dec 23 13:21:21 2007 begin with 59193326 relations and 45427340 unique ideals
Sun Dec 23 13:22:20 2007 reduce to 59117873 relations and 42758613 ideals in 7 passes
Sun Dec 23 13:22:20 2007 max relations containing the same ideal: 37
Sun Dec 23 13:22:59 2007 removing 2240776 relations and 1840776 ideals in 400000 cliques
Sun Dec 23 13:23:01 2007 commencing in-memory singleton removal
Sun Dec 23 13:23:09 2007 begin with 56877097 relations and 42758613 unique ideals
Sun Dec 23 13:24:04 2007 reduce to 56829418 relations and 40869823 ideals in 7 passes
Sun Dec 23 13:24:04 2007 max relations containing the same ideal: 35
Sun Dec 23 13:24:42 2007 removing 1999894 relations and 1599894 ideals in 400000 cliques
Sun Dec 23 13:24:44 2007 commencing in-memory singleton removal
Sun Dec 23 13:24:50 2007 begin with 54829524 relations and 40869823 unique ideals
Sun Dec 23 13:25:36 2007 reduce to 54790027 relations and 39230170 ideals in 6 passes
Sun Dec 23 13:25:36 2007 max relations containing the same ideal: 35
Sun Dec 23 13:26:11 2007 removing 1854461 relations and 1454461 ideals in 400000 cliques
Sun Dec 23 13:26:13 2007 commencing in-memory singleton removal
Sun Dec 23 13:26:19 2007 begin with 52935566 relations and 39230170 unique ideals
Sun Dec 23 13:27:10 2007 reduce to 52898387 relations and 37738293 ideals in 7 passes
Sun Dec 23 13:27:10 2007 max relations containing the same ideal: 33
Sun Dec 23 13:27:44 2007 removing 1757642 relations and 1357642 ideals in 400000 cliques
Sun Dec 23 13:27:45 2007 commencing in-memory singleton removal
Sun Dec 23 13:27:52 2007 begin with 51140745 relations and 37738293 unique ideals
Sun Dec 23 13:28:33 2007 reduce to 51106308 relations and 36345994 ideals in 6 passes
Sun Dec 23 13:28:33 2007 max relations containing the same ideal: 32
Sun Dec 23 13:29:05 2007 removing 1689868 relations and 1289868 ideals in 400000 cliques
Sun Dec 23 13:29:07 2007 commencing in-memory singleton removal
Sun Dec 23 13:29:13 2007 begin with 49416440 relations and 36345994 unique ideals
Sun Dec 23 13:29:52 2007 reduce to 49383084 relations and 35022553 ideals in 6 passes
Sun Dec 23 13:29:52 2007 max relations containing the same ideal: 31
Sun Dec 23 13:30:23 2007 removing 1631194 relations and 1231194 ideals in 400000 cliques
Sun Dec 23 13:30:25 2007 commencing in-memory singleton removal
Sun Dec 23 13:30:31 2007 begin with 47751890 relations and 35022553 unique ideals
Sun Dec 23 13:31:09 2007 reduce to 47720130 relations and 33759369 ideals in 6 passes
Sun Dec 23 13:31:09 2007 max relations containing the same ideal: 31
Sun Dec 23 13:31:39 2007 removing 1584699 relations and 1184699 ideals in 400000 cliques
Sun Dec 23 13:31:40 2007 commencing in-memory singleton removal
Sun Dec 23 13:31:46 2007 begin with 46135431 relations and 33759369 unique ideals
Sun Dec 23 13:32:16 2007 reduce to 46104307 relations and 32543338 ideals in 5 passes
Sun Dec 23 13:32:16 2007 max relations containing the same ideal: 30
Sun Dec 23 13:32:44 2007 removing 1550127 relations and 1150127 ideals in 400000 cliques
Sun Dec 23 13:32:46 2007 commencing in-memory singleton removal
Sun Dec 23 13:32:51 2007 begin with 44554180 relations and 32543338 unique ideals
Sun Dec 23 13:33:20 2007 reduce to 44522792 relations and 31361593 ideals in 5 passes
Sun Dec 23 13:33:20 2007 max relations containing the same ideal: 29
Sun Dec 23 13:33:47 2007 removing 1519801 relations and 1119801 ideals in 400000 cliques
Sun Dec 23 13:33:49 2007 commencing in-memory singleton removal
Sun Dec 23 13:33:54 2007 begin with 43002991 relations and 31361593 unique ideals
Sun Dec 23 13:34:27 2007 reduce to 42970886 relations and 30209420 ideals in 6 passes
Sun Dec 23 13:34:27 2007 max relations containing the same ideal: 28
Sun Dec 23 13:34:53 2007 removing 1484808 relations and 1084808 ideals in 400000 cliques
Sun Dec 23 13:34:54 2007 commencing in-memory singleton removal
Sun Dec 23 13:34:59 2007 begin with 41486078 relations and 30209420 unique ideals
Sun Dec 23 13:35:30 2007 reduce to 41452873 relations and 29091139 ideals in 6 passes
Sun Dec 23 13:35:30 2007 max relations containing the same ideal: 28
Sun Dec 23 13:35:56 2007 removing 1463987 relations and 1063987 ideals in 400000 cliques
Sun Dec 23 13:35:57 2007 commencing in-memory singleton removal
Sun Dec 23 13:36:01 2007 begin with 39988886 relations and 29091139 unique ideals
Sun Dec 23 13:36:31 2007 reduce to 39957715 relations and 27995708 ideals in 6 passes
Sun Dec 23 13:36:31 2007 max relations containing the same ideal: 28
Sun Dec 23 13:36:55 2007 removing 707581 relations and 536472 ideals in 171109 cliques
Sun Dec 23 13:36:57 2007 commencing in-memory singleton removal
Sun Dec 23 13:37:01 2007 begin with 39250134 relations and 27995708 unique ideals
Sun Dec 23 13:37:20 2007 reduce to 39243628 relations and 27452718 ideals in 4 passes
Sun Dec 23 13:37:20 2007 max relations containing the same ideal: 28
Sun Dec 23 13:37:26 2007 dataset has 70.0% excess relations
Sun Dec 23 13:37:57 2007 ignoring smallest 3141370 rational and 3140029 algebraic ideals
Sun Dec 23 13:37:57 2007 filtering ideals above 52490270
Sun Dec 23 13:37:57 2007 need 10611819 more relations than ideals
Sun Dec 23 13:37:57 2007 commencing singleton removal, final pass
Sun Dec 23 13:48:31 2007 memory use: 1307.2 MB
Sun Dec 23 13:48:31 2007 commencing in-memory singleton removal
Sun Dec 23 13:48:36 2007 begin with 39243628 relations and 28106740 unique ideals
Sun Dec 23 13:48:59 2007 reduce to 39239234 relations and 28102346 ideals in 5 passes
Sun Dec 23 13:48:59 2007 max relations containing the same ideal: 28
Sun Dec 23 13:49:22 2007 removing 924716 relations and 662182 ideals in 262534 cliques
Sun Dec 23 13:49:23 2007 commencing in-memory singleton removal
Sun Dec 23 13:49:27 2007 begin with 38314518 relations and 28102346 unique ideals
Sun Dec 23 13:49:50 2007 reduce to 38298290 relations and 27423809 ideals in 5 passes
Sun Dec 23 13:49:50 2007 max relations containing the same ideal: 28
Sun Dec 23 13:50:12 2007 removing 945449 relations and 682915 ideals in 262534 cliques
Sun Dec 23 13:50:13 2007 commencing in-memory singleton removal
Sun Dec 23 13:50:17 2007 begin with 37352841 relations and 27423809 unique ideals
Sun Dec 23 13:50:40 2007 reduce to 37337373 relations and 26725329 ideals in 5 passes
Sun Dec 23 13:50:40 2007 max relations containing the same ideal: 27
Sun Dec 23 13:51:01 2007 removing 1648 relations and 1424 ideals in 224 cliques
Sun Dec 23 13:51:02 2007 commencing in-memory singleton removal
Sun Dec 23 13:51:06 2007 begin with 37335725 relations and 26725329 unique ideals
Sun Dec 23 13:51:10 2007 reduce to 37335725 relations and 26723905 ideals in 1 passes
Sun Dec 23 13:51:10 2007 max relations containing the same ideal: 27
Sun Dec 23 13:51:16 2007 dataset has 53.0% excess relations
Sun Dec 23 13:51:43 2007 ignoring smallest 2812301 rational and 2810613 algebraic ideals
Sun Dec 23 13:51:43 2007 filtering ideals above 46658018
Sun Dec 23 13:51:43 2007 need 8489456 more relations than ideals
Sun Dec 23 13:51:43 2007 commencing singleton removal, final pass
Sun Dec 23 14:02:08 2007 memory use: 1307.2 MB
Sun Dec 23 14:02:08 2007 commencing in-memory singleton removal
Sun Dec 23 14:02:12 2007 begin with 37335725 relations and 27382356 unique ideals
Sun Dec 23 14:02:31 2007 reduce to 37335128 relations and 27381759 ideals in 4 passes
Sun Dec 23 14:02:31 2007 max relations containing the same ideal: 30
Sun Dec 23 14:02:53 2007 removing 1360622 relations and 960622 ideals in 400000 cliques
Sun Dec 23 14:02:55 2007 commencing in-memory singleton removal
Sun Dec 23 14:02:59 2007 begin with 35974506 relations and 27381759 unique ideals
Sun Dec 23 14:03:26 2007 reduce to 35940982 relations and 26387308 ideals in 6 passes
Sun Dec 23 14:03:26 2007 max relations containing the same ideal: 30
Sun Dec 23 14:03:47 2007 removing 1404760 relations and 1004760 ideals in 400000 cliques
Sun Dec 23 14:03:49 2007 commencing in-memory singleton removal
Sun Dec 23 14:03:53 2007 begin with 34536222 relations and 26387308 unique ideals
Sun Dec 23 14:04:18 2007 reduce to 34501608 relations and 25347588 ideals in 6 passes
Sun Dec 23 14:04:18 2007 max relations containing the same ideal: 29
Sun Dec 23 14:04:39 2007 removing 1390598 relations and 990598 ideals in 400000 cliques
Sun Dec 23 14:04:40 2007 commencing in-memory singleton removal
Sun Dec 23 14:04:44 2007 begin with 33111010 relations and 25347588 unique ideals
Sun Dec 23 14:05:04 2007 reduce to 33074348 relations and 24319921 ideals in 5 passes
Sun Dec 23 14:05:04 2007 max relations containing the same ideal: 28
Sun Dec 23 14:05:24 2007 removing 976345 relations and 711375 ideals in 264970 cliques
Sun Dec 23 14:05:25 2007 commencing in-memory singleton removal
Sun Dec 23 14:05:29 2007 begin with 32098003 relations and 24319921 unique ideals
Sun Dec 23 14:05:52 2007 reduce to 32079124 relations and 23589517 ideals in 6 passes
Sun Dec 23 14:05:52 2007 max relations containing the same ideal: 28
Sun Dec 23 14:06:11 2007 removing 1113 relations and 963 ideals in 150 cliques
Sun Dec 23 14:06:12 2007 commencing in-memory singleton removal
Sun Dec 23 14:06:15 2007 begin with 32078011 relations and 23589517 unique ideals
Sun Dec 23 14:06:19 2007 reduce to 32078011 relations and 23588554 ideals in 1 passes
Sun Dec 23 14:06:19 2007 max relations containing the same ideal: 28
Sun Dec 23 14:06:24 2007 dataset has 22.4% excess relations
Sun Dec 23 14:06:48 2007 ignoring smallest 2480865 rational and 2479733 algebraic ideals
Sun Dec 23 14:06:48 2007 filtering ideals above 40825766
Sun Dec 23 14:06:48 2007 need 7490687 more relations than ideals
Sun Dec 23 14:06:48 2007 commencing singleton removal, final pass
Sun Dec 23 14:16:30 2007 memory use: 1307.2 MB
Sun Dec 23 14:16:30 2007 commencing in-memory singleton removal
Sun Dec 23 14:16:34 2007 begin with 32078011 relations and 24250791 unique ideals
Sun Dec 23 14:16:54 2007 reduce to 32076955 relations and 24249735 ideals in 5 passes
Sun Dec 23 14:16:54 2007 max relations containing the same ideal: 29
Sun Dec 23 14:17:13 2007 removing 633383 relations and 465117 ideals in 168266 cliques
Sun Dec 23 14:17:14 2007 commencing in-memory singleton removal
Sun Dec 23 14:17:18 2007 begin with 31443572 relations and 24249735 unique ideals
Sun Dec 23 14:17:37 2007 reduce to 31435654 relations and 23776660 ideals in 5 passes
Sun Dec 23 14:17:37 2007 max relations containing the same ideal: 29
Sun Dec 23 14:17:56 2007 removing 589740 relations and 421474 ideals in 168266 cliques
Sun Dec 23 14:17:57 2007 commencing in-memory singleton removal
Sun Dec 23 14:18:00 2007 begin with 30845914 relations and 23776660 unique ideals
Sun Dec 23 14:18:20 2007 reduce to 30837459 relations and 23346682 ideals in 5 passes
Sun Dec 23 14:18:20 2007 max relations containing the same ideal: 28
Sun Dec 23 14:18:25 2007 dataset has 8.0% excess relations
Sun Dec 23 14:18:29 2007 relations with 0 large ideals: 381359
Sun Dec 23 14:18:29 2007 relations with 1 large ideals: 2570405
Sun Dec 23 14:18:29 2007 relations with 2 large ideals: 6940262
Sun Dec 23 14:18:29 2007 relations with 3 large ideals: 9791811
Sun Dec 23 14:18:29 2007 relations with 4 large ideals: 7567508
Sun Dec 23 14:18:29 2007 relations with 5 large ideals: 3113843
Sun Dec 23 14:18:29 2007 relations with 6 large ideals: 441587
Sun Dec 23 14:18:29 2007 relations with 7+ large ideals: 30684
Sun Dec 23 14:18:29 2007 commencing 2-way merge
Sun Dec 23 14:18:55 2007 reduce to 22116284 relation sets and 14625507 unique ideals
Sun Dec 23 14:18:55 2007 commencing full merge
Sun Dec 23 14:24:05 2007 found 11224044 cycles, need 8695707
Sun Dec 23 14:24:11 2007 weight of 8695707 cycles is about 565305823 (65.01/cycle)
Sun Dec 23 14:24:12 2007 distribution of cycle lengths:
Sun Dec 23 14:24:12 2007 1 relations: 898823
Sun Dec 23 14:24:12 2007 2 relations: 1044695
Sun Dec 23 14:24:12 2007 3 relations: 1158611
Sun Dec 23 14:24:12 2007 4 relations: 1154459
Sun Dec 23 14:24:12 2007 5 relations: 1126441
Sun Dec 23 14:24:12 2007 6 relations: 1041288
Sun Dec 23 14:24:12 2007 7 relations: 930189
Sun Dec 23 14:24:12 2007 8 relations: 785344
Sun Dec 23 14:24:12 2007 9 relations: 540686
Sun Dec 23 14:24:12 2007 10+ relations: 15171
Sun Dec 23 14:24:12 2007 heaviest cycle: 10 relations
Sun Dec 23 14:24:13 2007 commencing cycle optimization
Sun Dec 23 14:24:32 2007 start with 40773774 relations
Sun Dec 23 14:27:52 2007 pruned 772436 relations
Sun Dec 23 14:27:53 2007 distribution of cycle lengths:
Sun Dec 23 14:27:53 2007 1 relations: 898823
Sun Dec 23 14:27:53 2007 2 relations: 1065236
Sun Dec 23 14:27:53 2007 3 relations: 1200745
Sun Dec 23 14:27:53 2007 4 relations: 1188568
Sun Dec 23 14:27:53 2007 5 relations: 1168920
Sun Dec 23 14:27:53 2007 6 relations: 1066704
Sun Dec 23 14:27:53 2007 7 relations: 934538
Sun Dec 23 14:27:53 2007 8 relations: 731780
Sun Dec 23 14:27:53 2007 9 relations: 429224
Sun Dec 23 14:27:53 2007 10+ relations: 11169
Sun Dec 23 14:27:53 2007 heaviest cycle: 10 relations
Sun Dec 23 14:28:10 2007
Sun Dec 23 14:28:15 2007 commencing linear algebra
Sun Dec 23 14:28:17 2007 read 8695707 cycles
Sun Dec 23 14:29:17 2007 cycles contain 23289527 unique relations
Sun Dec 23 14:37:40 2007 read 23289527 relations
Sun Dec 23 14:42:41 2007 using 32 quadratic characters above 1073740544
Sun Dec 23 16:55:39 2007 read 8695707 cycles
Sun Dec 23 16:57:38 2007 matrix is 8690575 x 8695707 with weight 763789273 (avg 87.84/col)
Sun Dec 23 17:04:07 2007 filtering completed in 3 passes
Sun Dec 23 17:04:10 2007 matrix is 8602155 x 8602355 with weight 758171604 (avg 88.14/col)
Sun Dec 23 17:22:41 2007 read 8602355 cycles
Sun Dec 23 17:24:08 2007 matrix is 8602155 x 8602355 with weight 758171604 (avg 88.14/col)
Sun Dec 23 17:24:08 2007 saving the first 48 matrix rows for later
Sun Dec 23 17:24:14 2007 matrix is 8602107 x 8602355 with weight 574472114 (avg 66.78/col)
Sun Dec 23 17:24:14 2007 matrix includes 64 packed rows
Sun Dec 23 17:24:14 2007 using block size 43690 for processor cache size 1024 kB
Sun Dec 23 17:25:09 2007 commencing Lanczos iteration
[/CODE]

One tiny question
 

Would it be possible to list memory usage for the full-merge, linalg and square-root steps? Full merge is the one at which I've had a machine actually run out of memory (before the improvements in 1.30), but it would be interesting to know how memory-intensive those other stages are.

518-bit completion, after fairly significant over-sieving, is demonstrably possible in 4G

[QUOTE=fivemack;121409]Would it be possible to list memory usage for the full-merge, linalg and square-root steps? Full merge is the one at which I've had a machine actually run out of memory (before the improvements in 1.30), but it would be interesting to know how memory-intensive those other stages are.
[/QUOTE]
It's on my todo list

Hi Jason,

I'm trying to filter the relations for 2,841- a bit before I send them to Tom, to reduce file size.
What is the format of the .s file? Is it simply the line numbers of relations that survived dupe and singleton removal, stored as 32 bit words? Does the first line have number 0 or number 1?

Thanks,
Alex

[QUOTE=akruppa;121957]
What is the format of the .s file? Is it simply the line numbers of relations that survived dupe and singleton removal, stored as a 32 bit words? Does the first line have number 0 or number 1?
[/QUOTE]
Yes, that's the format. Line numbers start from 0, but lines that do not begin with a number or with '-' are not counted.

Note that while the code can in principle be modified to start directly from a .s file, the filtering has gotten increasingly convoluted so that it currently is somewhat tricky to do so.

[QUOTE=akruppa;121957]Hi Jason,

I'm trying to filter the relations for 2,841- a bit before I send them to Tom, to reduce file size.
What is the format of the .s file? Is it simply the line numbers of relations that survived dupe and singleton removal, stored as a 32 bit words? Does the first line have number 0 or number 1?

Thanks,
Alex[/QUOTE]

For my lattice sieve, I reduce my data a bit before I even submit it to
the filter code.

How?

cat filelist | sort > file1
uniq file1 file2

works quite quickly, even for large files.

[QUOTE=R.D. Silverman;121967]For my lattice sieve, I reduce my data a bit before I even submit it to
the filter code.

How?

cat filelist | sort > file1
uniq file1 file2

works quite quickly, even for large files.[/QUOTE]
Is that quicker than

cat filelist | sort -u > file2

?
Chris

[QUOTE=Chris Card;121991]Is that quicker than

cat filelist | sort -u > file2

?
Chris[/QUOTE]I doubt it, but then you wouldn't have a file containing all the data, duplicates and all. :wink:

[QUOTE=xilman;121993]I doubt it, but then you wouldn't have a file containing all the data, duplicates and all. :wink:[/QUOTE]

also, isn't

sort filelist > file1

better than

cat filelist | sort > file1

?
I seem to remember Randal Schwartz always pointing that one out ...

Chris

Unfortunately, the ggnfs data is not in a canonical form. You can have multiple representations of the same relation, particularly when doing lattice sieving because the "special-q" appears at the end of the factors.