[QUOTE=jbristow;117337]I got a "msieve has stopped working" message from Windows while msieve 1.29 was in the square root phase (near the end of multiplying).[/QUOTE]Same thing happened to me with an earlier version (although i always switch off windows error reporting, so i didn't get that error).

Jason already has the log

smh: yes, in hindsight this is very likely what was going wrong with your run

The memory use of the square root is something that's unavoidable given the choices I made for implementing it. There are three algorithms that can be used: you can use brute force, Couveignes' algorithm, and Montgomery/Nguyen's algorithm.

The last one is by far the best, since it's fast and uses little memory, but unfortunately it's incredibly complicated. It needs knowledge of a lot of computational algebraic number theory, and a lot of code. I spent weeks trying to understand it, and even with Cohen's book (which admittedly isn't great in the clarity department) and a working implementation inside GGNFS, I just couldn't wrap my brain around it.

Couveignes' algorithm is like brute force with a little extra information that you know because of the NFS sieving. It can only handle odd polynomial degree (so no giant SNFS jobs allowed) and it avoids having to deal with huge numbers by instead having to deal with huge amounts of small numbers.

Until a few years ago the brute force algorithm was completely unrealistic for NFS jobs of interesting size; all of the main NFS papers were written in the early-to-mid-90s and only mention it in passing. Then DJ Bernstein said it was the way to go because computers are much better now and nobody should be scared of large-integer arithmetic.

And I'm convinced he was right. I estimated the memory use needed by the brute force algorithm to perform the square root for a 512-bit GNFS factorization and it came out to under 2GB, which I think is reasonable to ask that people have in order to solve cutting-edge math problems. At any rate, I estimated it would need less memory than the NFS linear algebra. You can implement the brute force square root in under 1000 lines of code (plus an arbitrary-precision math library that uses Fourier-transform based arithmetic, but I've been coding those for a decade now) and this is 50x less code than Montgomery/Nguyen, so I decided to take a chance and try it.

It's worked great. Lately the linear algebra has been getting a lot of memory optimizations, so the memory footprint of the square root is probably the limiting factor now, especially for GNFS. Your msieve run has to play with all the other programs running on your computer, and the OS may not have all the needed memory available even if you have enough RAM. Plus you're limited to the maximum size of multiply the FFTs can manage, and everyone is approaching that limit distressingly quickly.

I think that as computers get more and more memory that developers will make these kinds of tradeoffs more and more often. Maybe this is the big advantage that 64-bit systems have over 32-bit systems.

[QUOTE=jasonp;117235]Now available. Highlights include:

- Major changes to the Lanczos routines. The new version incorporates linear algebra checkpoint and restart; checkpoints happen for large matrices only, once every 2-3 hours and whenever the library is interrupted. The demo binary can restart from a previous checkpoint with the -ncr instead of the -nc2 option. The updated source is also slightly faster

[/QUOTE]

On Linux, I can compile and run both 32-bit (with large file support) and 64-bit versions. Can I do the filtering and begin the linear algebra with a 32-bit binary, and restart and finish the linear algebra with a 64-bit binary? I tried it, and it seemed to restart ok, but I didn't let it run anywhere near completion. The computer I'm running this on doesn't quite have enough memory to do the filtering with the 64-bit binary, but the linear algebra is a bit faster with the 64-bit binary.

Greg

[QUOTE=frmky;118018]On Linux, I can compile and run both 32-bit (with large file support) and 64-bit versions. Can I do the filtering and begin the linear algebra with a 32-bit binary, and restart and finish the linear algebra with a 64-bit binary? I tried it, and it seemed to restart ok, but I didn't let it run anywhere near completion. The computer I'm running this on doesn't quite have enough memory to do the filtering with the 64-bit binary, but the linear algebra is a bit faster with the 64-bit binary.
[/QUOTE]
The result from the filtering is a file full of 32-bit words, and both 32-bit and 64-bit library versions should generate the same file. It should work out fine.

If you feel adventurous, I've overhauled the filtering to move away from pointer-heavy data structures. If you replace the gnfs/filter subdirectory with the source files in the attached zip and recompile, filtering on 64-bit systems should become faster and enormously more memory efficient (~60% less memory in my tests).

The modified source will appear in the next release, so treat this as pre-release code.

[QUOTE=jasonp;118020]
If you feel adventurous, I've overhauled the filtering to move away from pointer-heavy data structures. If you replace the gnfs/filter subdirectory with the source files in the attached zip and recompile, filtering on 64-bit systems should become faster and enormously more memory efficient (~60% less memory in my tests).
[/QUOTE]

Thanks! It worked perfectly. For the filtering of 2,779+, the 64-bit memory usage dropped from about 3.9 GB to about 1.9 GB. It's now starting to work on an approx. 7.5M matrix.

Greg

[QUOTE=frmky;118033]Thanks! It worked perfectly. For the filtering of 2,779+, the 64-bit memory usage dropped from about 3.9 GB to about 1.9 GB. It's now starting to work on an approx. 7.5M matrix.
[/QUOTE]
This logfile will be fun reading. Did the filtering take less time?

I suppose if the matrix will fit in memory then the square root will too, at least I really hope so.

[QUOTE=jasonp;118049]This logfile will be fun reading. Did the filtering take less time?[/QUOTE]

I didn't save the first log file so I'm not sure, but in the steps leading up to the full merge, they might have been slightly quicker but not much. Full merge is where the memory use grew dramatically, so my first run never got past this point.

[QUOTE=jasonp;118049]
I suppose if the matrix will fit in memory then the square root will too, at least I really hope so.[/QUOTE]

If it doesn't, I can move everything to another computer with more memory. I'm running this on a single-core 2.4GHz Opteron, so the linear algebra will take 14 days to finish. We'll find out then. For your entertainment, here's the log:

[CODE]Wed Nov 7 21:05:35 2007 Msieve v. 1.29
Wed Nov 7 21:05:35 2007 random seeds: a740f2b7 0cf1e780
Wed Nov 7 21:05:35 2007 factoring 21499173951598023655871526129741238864252274176505248438905816972331478841874026717266127459812853910615830233333737201439633130982196868280103768992238493630431515155684471825756247809234523310884830836516644313 (212 digits)
Wed Nov 7 21:05:35 2007 commencing number field sieve (212-digit input)
Wed Nov 7 21:05:36 2007 R0: -1361129467683753853853498429727072845824
Wed Nov 7 21:05:36 2007 R1: 1
Wed Nov 7 21:05:36 2007 A0: 2
Wed Nov 7 21:05:36 2007 A1: 0
Wed Nov 7 21:05:36 2007 A2: 0
Wed Nov 7 21:05:36 2007 A3: 0
Wed Nov 7 21:05:36 2007 A4: 0
Wed Nov 7 21:05:36 2007 A5: 0
Wed Nov 7 21:05:36 2007 A6: 1
Wed Nov 7 21:05:36 2007 size score = 2.834873e-11, Murphy alpha = 1.842138, combined = 1.674763e-11
Wed Nov 7 21:10:21 2007 restarting with 92301399 relations
Wed Nov 7 21:10:21 2007
Wed Nov 7 21:10:21 2007 commencing relation filtering
Wed Nov 7 21:10:21 2007 commencing duplicate removal, pass 1
Wed Nov 7 21:11:25 2007 error -1 reading relation 7165101
Wed Nov 7 21:11:28 2007 error -4 reading relation 7430114
Wed Nov 7 21:22:34 2007 error -14 reading relation 84653095
Wed Nov 7 21:23:05 2007 error -14 reading relation 88388346
Wed Nov 7 21:23:39 2007 found 15480559 hash collisions in 92301354 relations
Wed Nov 7 21:23:39 2007 commencing duplicate removal, pass 2
Wed Nov 7 21:31:59 2007 found 1804416 duplicates and 90496938 unique relations
Wed Nov 7 21:31:59 2007 memory use: 504.8 MB
Wed Nov 7 21:32:31 2007 ignoring smallest 3236951 rational and 3236317 algebraic ideals
Wed Nov 7 21:32:31 2007 filtering ideals above 54192062
Wed Nov 7 21:32:31 2007 need 11004555 more relations than ideals
Wed Nov 7 21:32:31 2007 commencing singleton removal, pass 1
Wed Nov 7 21:46:32 2007 relations with 0 large ideals: 1440982
Wed Nov 7 21:46:32 2007 relations with 1 large ideals: 8084380
Wed Nov 7 21:46:32 2007 relations with 2 large ideals: 23156082
Wed Nov 7 21:46:32 2007 relations with 3 large ideals: 32649691
Wed Nov 7 21:46:32 2007 relations with 4 large ideals: 21306953
Wed Nov 7 21:46:32 2007 relations with 5 large ideals: 3858850
Wed Nov 7 21:46:32 2007 relations with 6 large ideals: 0
Wed Nov 7 21:46:32 2007 relations with 7+ large ideals: 0
Wed Nov 7 21:46:32 2007 90496938 relations and about 61560280 large ideals
Wed Nov 7 21:46:32 2007 commencing singleton removal, pass 2
Wed Nov 7 22:01:12 2007 found 15146451 singletons
Wed Nov 7 22:01:12 2007 current dataset: 75350487 relations and about 45339080 large ideals
Wed Nov 7 22:01:13 2007 commencing singleton removal, pass 3
Wed Nov 7 22:13:44 2007 relations with 0 large ideals: 1440982
Wed Nov 7 22:13:44 2007 relations with 1 large ideals: 7565914
Wed Nov 7 22:13:44 2007 relations with 2 large ideals: 20283422
Wed Nov 7 22:13:44 2007 relations with 3 large ideals: 26784097
Wed Nov 7 22:13:44 2007 relations with 4 large ideals: 16426523
Wed Nov 7 22:13:44 2007 relations with 5 large ideals: 2849549
Wed Nov 7 22:13:44 2007 relations with 6 large ideals: 0
Wed Nov 7 22:13:44 2007 relations with 7+ large ideals: 0
Wed Nov 7 22:13:44 2007 75350487 relations and about 57090163 large ideals
Wed Nov 7 22:13:44 2007 commencing singleton removal, pass 4
Wed Nov 7 22:26:52 2007 found 12496973 singletons
Wed Nov 7 22:26:52 2007 current dataset: 62853514 relations and about 43716489 large ideals
Wed Nov 7 22:26:52 2007 commencing singleton removal, pass 5
Wed Nov 7 22:39:03 2007 found 2820361 singletons
Wed Nov 7 22:39:03 2007 current dataset: 60033153 relations and about 40845657 large ideals
Wed Nov 7 22:39:03 2007 commencing singleton removal, pass 6
Wed Nov 7 22:51:01 2007 found 588967 singletons
Wed Nov 7 22:51:01 2007 current dataset: 59444186 relations and about 40254338 large ideals
Wed Nov 7 22:51:02 2007 commencing singleton removal, pass 7
Wed Nov 7 23:03:07 2007 found 115899 singletons
Wed Nov 7 23:03:07 2007 current dataset: 59328287 relations and about 40138336 large ideals
Wed Nov 7 23:03:07 2007 commencing singleton removal, final pass
Wed Nov 7 23:15:29 2007 memory use: 2116.1 MB
Wed Nov 7 23:15:29 2007 commencing in-memory singleton removal
Wed Nov 7 23:15:37 2007 begin with 59328287 relations and 46506881 unique ideals
Wed Nov 7 23:17:13 2007 reduce to 52320955 relations and 39332648 ideals in 13 passes
Wed Nov 7 23:17:13 2007 max relations containing the same ideal: 34
Wed Nov 7 23:17:47 2007 removing 3309517 relations and 2909517 ideals in 400000 cliques
Wed Nov 7 23:17:50 2007 commencing in-memory singleton removal
Wed Nov 7 23:17:55 2007 begin with 49011438 relations and 39332648 unique ideals
Wed Nov 7 23:18:53 2007 reduce to 48890075 relations and 36300633 ideals in 9 passes
Wed Nov 7 23:18:53 2007 max relations containing the same ideal: 33
Wed Nov 7 23:19:25 2007 removing 2452808 relations and 2052808 ideals in 400000 cliques
Wed Nov 7 23:19:27 2007 commencing in-memory singleton removal
Wed Nov 7 23:19:32 2007 begin with 46437267 relations and 36300633 unique ideals
Wed Nov 7 23:20:20 2007 reduce to 46361623 relations and 34171573 ideals in 8 passes
Wed Nov 7 23:20:20 2007 max relations containing the same ideal: 33
Wed Nov 7 23:20:50 2007 removing 2178875 relations and 1778875 ideals in 400000 cliques
Wed Nov 7 23:20:51 2007 commencing in-memory singleton removal
Wed Nov 7 23:20:56 2007 begin with 44182748 relations and 34171573 unique ideals
Wed Nov 7 23:21:36 2007 reduce to 44118986 relations and 32328422 ideals in 7 passes
Wed Nov 7 23:21:36 2007 max relations containing the same ideal: 32
Wed Nov 7 23:22:03 2007 removing 2024077 relations and 1624077 ideals in 400000 cliques
Wed Nov 7 23:22:05 2007 commencing in-memory singleton removal
Wed Nov 7 23:22:10 2007 begin with 42094909 relations and 32328422 unique ideals
Wed Nov 7 23:22:41 2007 reduce to 42035244 relations and 30644206 ideals in 6 passes
Wed Nov 7 23:22:41 2007 max relations containing the same ideal: 32
Wed Nov 7 23:23:07 2007 removing 1862111 relations and 1475629 ideals in 386482 cliques
Wed Nov 7 23:23:09 2007 commencing in-memory singleton removal
Wed Nov 7 23:23:13 2007 begin with 40173133 relations and 30644206 unique ideals
Wed Nov 7 23:23:43 2007 reduce to 40119800 relations and 29114838 ideals in 6 passes
Wed Nov 7 23:23:43 2007 max relations containing the same ideal: 31
Wed Nov 7 23:24:07 2007 removing 4205 relations and 3799 ideals in 406 cliques
Wed Nov 7 23:24:08 2007 commencing in-memory singleton removal
Wed Nov 7 23:24:13 2007 begin with 40115595 relations and 29114838 unique ideals
Wed Nov 7 23:24:18 2007 reduce to 40115595 relations and 29111039 ideals in 1 passes
Wed Nov 7 23:24:18 2007 max relations containing the same ideal: 31
Wed Nov 7 23:24:24 2007 dataset has 70.0% excess relations
Wed Nov 7 23:24:52 2007 ignoring smallest 2931651 rational and 2931353 algebraic ideals
Wed Nov 7 23:24:52 2007 filtering ideals above 48772855
Wed Nov 7 23:24:52 2007 need 9904100 more relations than ideals
Wed Nov 7 23:24:52 2007 commencing singleton removal, final pass
Wed Nov 7 23:35:28 2007 memory use: 1307.2 MB
Wed Nov 7 23:35:28 2007 commencing in-memory singleton removal
Wed Nov 7 23:35:33 2007 begin with 40115595 relations and 29721058 unique ideals
Wed Nov 7 23:36:02 2007 reduce to 40111934 relations and 29717397 ideals in 6 passes
Wed Nov 7 23:36:02 2007 max relations containing the same ideal: 33
Wed Nov 7 23:36:26 2007 removing 1207640 relations and 962422 ideals in 245218 cliques
Wed Nov 7 23:36:27 2007 commencing in-memory singleton removal
Wed Nov 7 23:36:31 2007 begin with 38904294 relations and 29717397 unique ideals
Wed Nov 7 23:37:00 2007 reduce to 38880627 relations and 28731195 ideals in 6 passes
Wed Nov 7 23:37:00 2007 max relations containing the same ideal: 33
Wed Nov 7 23:37:23 2007 removing 1133753 relations and 888535 ideals in 245218 cliques
Wed Nov 7 23:37:24 2007 commencing in-memory singleton removal
Wed Nov 7 23:37:28 2007 begin with 37746874 relations and 28731195 unique ideals
Wed Nov 7 23:37:55 2007 reduce to 37723983 relations and 27819630 ideals in 6 passes
Wed Nov 7 23:37:55 2007 max relations containing the same ideal: 32
Wed Nov 7 23:38:18 2007 removing 2442 relations and 2190 ideals in 252 cliques
Wed Nov 7 23:38:18 2007 commencing in-memory singleton removal
Wed Nov 7 23:38:23 2007 begin with 37721541 relations and 27819630 unique ideals
Wed Nov 7 23:38:27 2007 reduce to 37721541 relations and 27817440 ideals in 1 passes
Wed Nov 7 23:38:27 2007 max relations containing the same ideal: 32
Wed Nov 7 23:38:33 2007 dataset has 53.0% excess relations
Wed Nov 7 23:38:58 2007 ignoring smallest 2624651 rational and 2623879 algebraic ideals
Wed Nov 7 23:38:58 2007 filtering ideals above 43353649
Wed Nov 7 23:38:58 2007 need 7923280 more relations than ideals
Wed Nov 7 23:38:58 2007 commencing singleton removal, final pass
Wed Nov 7 23:49:32 2007 memory use: 1307.2 MB
Wed Nov 7 23:49:33 2007 commencing in-memory singleton removal
Wed Nov 7 23:49:37 2007 begin with 37721541 relations and 28431904 unique ideals
Wed Nov 7 23:49:56 2007 reduce to 37721267 relations and 28431630 ideals in 4 passes
Wed Nov 7 23:49:56 2007 max relations containing the same ideal: 32
Wed Nov 7 23:50:20 2007 removing 1717234 relations and 1317234 ideals in 400000 cliques
Wed Nov 7 23:50:21 2007 commencing in-memory singleton removal
Wed Nov 7 23:50:25 2007 begin with 36004033 relations and 28431630 unique ideals
Wed Nov 7 23:50:57 2007 reduce to 35951920 relations and 27061826 ideals in 7 passes
Wed Nov 7 23:50:57 2007 max relations containing the same ideal: 31
Wed Nov 7 23:51:20 2007 removing 1735027 relations and 1335027 ideals in 400000 cliques
Wed Nov 7 23:51:21 2007 commencing in-memory singleton removal
Wed Nov 7 23:51:25 2007 begin with 34216893 relations and 27061826 unique ideals
Wed Nov 7 23:51:55 2007 reduce to 34162060 relations and 25671488 ideals in 7 passes
Wed Nov 7 23:51:55 2007 max relations containing the same ideal: 30
Wed Nov 7 23:52:16 2007 removing 1698236 relations and 1298236 ideals in 400000 cliques
Wed Nov 7 23:52:18 2007 commencing in-memory singleton removal
Wed Nov 7 23:52:21 2007 begin with 32463824 relations and 25671488 unique ideals
Wed Nov 7 23:52:49 2007 reduce to 32405137 relations and 24313985 ideals in 7 passes
Wed Nov 7 23:52:49 2007 max relations containing the same ideal: 29
Wed Nov 7 23:53:09 2007 removing 816534 relations and 648663 ideals in 167871 cliques
Wed Nov 7 23:53:10 2007 commencing in-memory singleton removal
Wed Nov 7 23:53:14 2007 begin with 31588603 relations and 24313985 unique ideals
Wed Nov 7 23:53:37 2007 reduce to 31576572 relations and 23653241 ideals in 6 passes
Wed Nov 7 23:53:37 2007 max relations containing the same ideal: 29
Wed Nov 7 23:53:42 2007 dataset has 22.4% excess relations
Wed Nov 7 23:54:04 2007 ignoring smallest 2315237 rational and 2314817 algebraic ideals
Wed Nov 7 23:54:04 2007 filtering ideals above 37934443
Wed Nov 7 23:54:04 2007 need 6991129 more relations than ideals
Wed Nov 7 23:54:04 2007 commencing singleton removal, final pass
Thu Nov 8 00:03:21 2007 memory use: 1069.9 MB
Thu Nov 8 00:03:21 2007 commencing in-memory singleton removal
Thu Nov 8 00:03:24 2007 begin with 31576572 relations and 24271691 unique ideals
Thu Nov 8 00:03:44 2007 reduce to 31576012 relations and 24271131 ideals in 5 passes
Thu Nov 8 00:03:44 2007 max relations containing the same ideal: 29
Thu Nov 8 00:04:04 2007 removing 691553 relations and 534677 ideals in 156876 cliques
Thu Nov 8 00:04:05 2007 commencing in-memory singleton removal
Thu Nov 8 00:04:08 2007 begin with 30884459 relations and 24271131 unique ideals
Thu Nov 8 00:04:28 2007 reduce to 30873985 relations and 23725935 ideals in 5 passes
Thu Nov 8 00:04:28 2007 max relations containing the same ideal: 29
Thu Nov 8 00:04:47 2007 removing 671735 relations and 514859 ideals in 156876 cliques
Thu Nov 8 00:04:48 2007 commencing in-memory singleton removal
Thu Nov 8 00:04:51 2007 begin with 30202250 relations and 23725935 unique ideals
Thu Nov 8 00:05:14 2007 reduce to 30191796 relations and 23200587 ideals in 6 passes
Thu Nov 8 00:05:14 2007 max relations containing the same ideal: 29
Thu Nov 8 00:05:19 2007 dataset has 8.0% excess relations
Thu Nov 8 00:05:23 2007 relations with 0 large ideals: 725653
Thu Nov 8 00:05:23 2007 relations with 1 large ideals: 3184961
Thu Nov 8 00:05:23 2007 relations with 2 large ideals: 7785145
Thu Nov 8 00:05:23 2007 relations with 3 large ideals: 9619145
Thu Nov 8 00:05:23 2007 relations with 4 large ideals: 6321596
Thu Nov 8 00:05:23 2007 relations with 5 large ideals: 2130989
Thu Nov 8 00:05:23 2007 relations with 6 large ideals: 338678
Thu Nov 8 00:05:23 2007 relations with 7+ large ideals: 85629
Thu Nov 8 00:05:23 2007 commencing 2-way merge
Thu Nov 8 00:05:51 2007 reduce to 20454949 relation sets and 13463740 unique ideals
Thu Nov 8 00:05:51 2007 commencing full merge
Thu Nov 8 00:10:21 2007 found 9985606 cycles, need 7633940
Thu Nov 8 00:10:26 2007 weight of 7633940 cycles is about 496411432 (65.03/cycle)
Thu Nov 8 00:10:27 2007 distribution of cycle lengths:
Thu Nov 8 00:10:27 2007 1 relations: 1127903
Thu Nov 8 00:10:27 2007 2 relations: 851202
Thu Nov 8 00:10:27 2007 3 relations: 892981
Thu Nov 8 00:10:27 2007 4 relations: 881405
Thu Nov 8 00:10:27 2007 5 relations: 853667
Thu Nov 8 00:10:27 2007 6 relations: 797427
Thu Nov 8 00:10:27 2007 7 relations: 731527
Thu Nov 8 00:10:27 2007 8 relations: 646893
Thu Nov 8 00:10:27 2007 9 relations: 537108
Thu Nov 8 00:10:27 2007 10+ relations: 313827
Thu Nov 8 00:10:27 2007 heaviest cycle: 13 relations
Thu Nov 8 00:10:27 2007 commencing cycle optimization
Thu Nov 8 00:11:24 2007 start with 36366518 relations
Thu Nov 8 00:14:37 2007 pruned 681229 relations
Thu Nov 8 00:14:38 2007 distribution of cycle lengths:
Thu Nov 8 00:14:38 2007 1 relations: 1127903
Thu Nov 8 00:14:38 2007 2 relations: 867972
Thu Nov 8 00:14:38 2007 3 relations: 922881
Thu Nov 8 00:14:38 2007 4 relations: 904546
Thu Nov 8 00:14:38 2007 5 relations: 883486
Thu Nov 8 00:14:38 2007 6 relations: 819090
Thu Nov 8 00:14:38 2007 7 relations: 743998
Thu Nov 8 00:14:38 2007 8 relations: 634524
Thu Nov 8 00:14:38 2007 9 relations: 484816
Thu Nov 8 00:14:38 2007 10+ relations: 244724
Thu Nov 8 00:14:38 2007 heaviest cycle: 13 relations
Thu Nov 8 00:14:53 2007
Thu Nov 8 00:14:57 2007 commencing linear algebra
Thu Nov 8 00:14:59 2007 read 7633940 cycles
Thu Nov 8 00:15:49 2007 cycles contain 21694261 unique relations
Thu Nov 8 00:23:48 2007 read 21694261 relations
Thu Nov 8 00:27:25 2007 using 32 quadratic characters above 1073740698
Thu Nov 8 00:54:21 2007 read 7633940 cycles
Thu Nov 8 01:00:54 2007 filtering completed in 3 passes
Thu Nov 8 01:00:56 2007 matrix is 7490253 x 7490451 with weight 666958089 (avg 89.04/col)
Thu Nov 8 01:05:43 2007 read 7490451 cycles
Thu Nov 8 01:06:57 2007 matrix is 7490253 x 7490451 with weight 666958089 (avg 89.04/col)
Thu Nov 8 01:06:57 2007 saving the first 48 matrix rows for later
Thu Nov 8 01:07:02 2007 matrix is 7490205 x 7490451 with weight 503129773 (avg 67.17/col)
Thu Nov 8 01:07:02 2007 matrix includes 64 packed rows
Thu Nov 8 01:07:02 2007 using block size 43690 for processor cache size 1024 kB
Thu Nov 8 01:07:48 2007 commencing Lanczos iteration
[/CODE]

Greg

Is there any way to feed msieve multiple batches of partial relations for postprocessing? I'm doing a factorization where I expect I'll need ~40M partial relations, but this causes the file size to be >4GB, so my Win32 msieve binary can't open the file. I'm trying to compile a Win64 version (which I imagine would fix this issue?), but I was curious if there was another way around this.

[QUOTE]Is there any way to feed msieve multiple batches of partial relations for postprocessing? I'm doing a factorization where I expect I'll need ~40M partial relations, but this causes the file size to be >4GB, so my Win32 msieve binary can't open the file. I'm trying to compile a Win64 version (which I imagine would fix this issue?), but I was curious if there was another way around this. [/QUOTE]

Do you mean you have multiple files for a single number, and you want msieve to use all of the relations included within these multiple files?
If so, (at least on windows), go to the command promt, then to the directory with the files in it, and type

[CODE]copy /B msieve1.dat+msieve2.dat msieve.dat[/CODE]

for however many dat files you have. This will combine the files into one superfile (but not delete the original ones) and you can tell msieve to read from the superfile.

roger

[QUOTE=jbristow;119993]but this causes the file size to be >4GB, so my Win32 msieve binary can't open the file.[/QUOTE]

Roger,

You missed this point. His file system is unable to handle the concatenated file.

[QUOTE=Wacky;120031]His file system is unable to handle the concatenated file.[/QUOTE]
I know that FAT32 has a 2GB file size limit, but this is the first time I've heard about a 4GB file size limit. NTFS isn't supposed to have a limit on file size, is this a limitation of the C runtime library msvcrt.dll on windows? Looking on google doesn't turn up anything obvious.

QS and NFS postprocessing is very careful to only access files sequentially, performing several passes if needed.