Running Msieve - Page 14

frmky · 2007-12-07, 00:44

In 32-bit Linux, '-D_FILE_OFFSET_BITS=64' must be added to the compile flags to handle files over 2 GB. If you're using gcc in either Cygwin or MinGW, you may need to do the same thing.

Greg

Quote:

Originally Posted by jbristow

Is there any way to feed msieve multiple batches of partial relations for postprocessing? I'm doing a factorization where I expect I'll need ~40M partial relations, but this causes the file size to be >4GB, so my Win32 msieve binary can't open the file. I'm trying to compile a Win64 version (which I imagine would fix this issue?), but I was curious if there was another way around this.

jbristow · 2007-12-07, 02:47

Thanks for all of the replies. I'm running 64-bit Windows Vista on a NTFS filesystem, but using the precompiled Win32 msieve binary. I can create files over 4 GB, copy them between folders, etc. When I create a 3.7 GB subset of partial relations, msieve runs fine. However, if I try to use a 4.4 GB set of relations, I get the error:

error: cannot open savefile 'msieve.dat'

right after the line that includes the size score, murphy alpha, etc.

I've tried adding a set of duplicate relations to the successful 3.7 GB file (just to make sure the error isn't due to a corrupt file) that results in a file over 4 GB and I get the same error.

jasonp · 2007-12-07, 05:16

Quote:

Originally Posted by jbristow

Thanks for all of the replies. I'm running 64-bit Windows Vista on a NTFS filesystem, but using the precompiled Win32 msieve binary. I can create files over 4 GB, copy them between folders, etc. When I create a 3.7 GB subset of partial relations, msieve runs fine. However, if I try to use a 4.4 GB set of relations, I get the error:

error: cannot open savefile 'msieve.dat'

right after the line that includes the size score, murphy alpha, etc.

I've tried adding a set of duplicate relations to the successful 3.7 GB file (just to make sure the error isn't due to a corrupt file) that results in a file over 4 GB and I get the same error.

I can reproduce the problem locally. A cygwin-compiled binary reads the file without trouble, and I modified the compile arguments to do the right thing on unix, but the MinGW side of things is problematic.

A little digging turned up this post on one of the MSDN forums that seems to point the blame at Microsoft's C runtime library not doing the right thing when a large file is opened for writing. The solution appears to require switching to Microsoft's API functions, which don't have buffered file IO like their unix equivalents do. This means fixing the problem requires producing windows and non-windows versions of the file-reading code, which would look very different from each other. Grr.

dleclair · 2007-12-07, 16:07

Hi Jason,

Before looking at the Windows API functions for file I/O, try adding "b" to the mode specified with your fopen called as in fopen(filename,"rb").

While Paul Leyland was doing the filtering for the M811 project by NFSNET there were some very odd things happening and I eventually tracked it down to a problem in the Microsoft run time library. The problem manifested itself in a different way but it was related to reading files larger than 4G in size. Simply adding "b" to the mode on fopen() was enough to work around it.

The "b" ought to be ignored when compiled on non-Windows platforms. To be safe you could use a #defined value based on the architecture you're compiling for.

-Don

ATH · 2007-12-07, 23:45

Quote:

Originally Posted by jasonp

I know that FAT32 has a 2GB file size limit, but this is the first time I've heard about a 4GB file size limit. NTFS isn't supposed to have a limit on file size, is this a limitation of the C runtime library msvcrt.dll on windows? Looking on google doesn't turn up anything obvious.

QS and NFS postprocessing is very careful to only access files sequentially, performing several passes if needed.

In Windows XP I have a 4 GB limit on my FAT32 partitions, and no limit on my NTFS partitions.

(A few times I spent time trying to find bugs in my computer, only to realize I tried to place a 4+ Gb file on a FAT32 partition

)

jasonp · 2007-12-08, 16:40

Quote:

Originally Posted by dleclair

Before looking at the Windows API functions for file I/O, try adding "b" to the mode specified with your fopen called as in fopen(filename,"rb").

This partially fixes the problem. The files can be opened and read, but attempting to append bytes using stdio functions like fprintf and fwrite causes a segfault. Looks like I need to write a savefile abstraction layer specifically for windows.

Thanks for the tip!

frmky · 2007-12-09, 20:16

Jason,

I have a case, the Smarandache number sm94 actually, in which the binary from the website crashes in sqrt near the end of "multiplying xxx relations" but a Cygwin compiled binary works properly. The log file is below. Since the Cygwin binary worked, I'm not sure how to debug this locally. What do you use to compile the official binary? Otherwise, I can provide everything for you to download as a 2 GB tar file.

Greg

Code:

Tue Dec 04 13:07:40 2007  
Tue Dec 04 13:07:40 2007  
Tue Dec 04 13:07:40 2007  Msieve v. 1.30
Tue Dec 04 13:07:40 2007  random seeds: dbba6140 82482878
Tue Dec 04 13:07:40 2007  factoring 4881269226159859145363348158028800528357637730704107517078436952997684213093804119278586325654853291448226734543293119800229451979882856232013130645868074794819 (160 digits)
Tue Dec 04 13:07:41 2007  commencing number field sieve (160-digit input)
Tue Dec 04 13:07:41 2007  R0: -10000000000000000000000000000000000
Tue Dec 04 13:07:41 2007  R1:  1
Tue Dec 04 13:07:41 2007  A0: -4703
Tue Dec 04 13:07:41 2007  A1:  0
Tue Dec 04 13:07:41 2007  A2:  0
Tue Dec 04 13:07:41 2007  A3:  0
Tue Dec 04 13:07:41 2007  A4:  0
Tue Dec 04 13:07:41 2007  A5:  604999994990
Tue Dec 04 13:07:41 2007  size score = 3.430393e-014, Murphy alpha = 0.131526, combined = 3.283246e-014
Tue Dec 04 13:09:07 2007  restarting with 26548169 relations
Tue Dec 04 13:09:07 2007  
Tue Dec 04 13:09:07 2007  commencing relation filtering
Tue Dec 04 13:09:07 2007  commencing duplicate removal, pass 1
Tue Dec 04 13:09:44 2007  error -10 reading relation 2539234
Tue Dec 04 13:09:44 2007  error -5 reading relation 2577186
Tue Dec 04 13:09:45 2007  error -14 reading relation 2614999
Tue Dec 04 13:09:45 2007  error -14 reading relation 2617896
Tue Dec 04 13:09:45 2007  error -10 reading relation 2618106
Tue Dec 04 13:09:45 2007  error -8 reading relation 2618350
Tue Dec 04 13:09:49 2007  error -10 reading relation 2954760
Tue Dec 04 13:09:49 2007  error -8 reading relation 2956434
Tue Dec 04 13:09:49 2007  error -4 reading relation 2956785
Tue Dec 04 13:09:49 2007  error -14 reading relation 2962378
Tue Dec 04 13:09:50 2007  error -14 reading relation 2975306
Tue Dec 04 13:09:50 2007  error -8 reading relation 2988104
Tue Dec 04 13:09:50 2007  error -14 reading relation 3015682
Tue Dec 04 13:09:50 2007  error -10 reading relation 3031304
Tue Dec 04 13:09:53 2007  error -14 reading relation 3245310
Tue Dec 04 13:09:54 2007  error -14 reading relation 3295777
Tue Dec 04 13:09:54 2007  error -8 reading relation 3307064
Tue Dec 04 13:10:04 2007  error -10 reading relation 4015672
Tue Dec 04 13:10:04 2007  error -10 reading relation 4069246
Tue Dec 04 13:10:08 2007  error -5 reading relation 4347769
Tue Dec 04 13:10:12 2007  error -10 reading relation 4618563
Tue Dec 04 13:10:22 2007  error -10 reading relation 5364877
Tue Dec 04 13:10:23 2007  error -10 reading relation 5416467
Tue Dec 04 13:10:26 2007  error -5 reading relation 5639577
Tue Dec 04 13:10:26 2007  error -10 reading relation 5639578
Tue Dec 04 13:10:26 2007  error -14 reading relation 5691268
Tue Dec 04 13:10:27 2007  error -8 reading relation 5707916
Tue Dec 04 13:10:27 2007  error -14 reading relation 5742171
Tue Dec 04 13:10:27 2007  error -10 reading relation 5762967
Tue Dec 04 13:10:27 2007  error -14 reading relation 5777682
Tue Dec 04 13:10:28 2007  error -10 reading relation 5789833
Tue Dec 04 13:10:28 2007  error -10 reading relation 5790910
Tue Dec 04 13:10:28 2007  error -10 reading relation 5791815
Tue Dec 04 13:10:28 2007  error -4 reading relation 5823491
Tue Dec 04 13:10:28 2007  error -14 reading relation 5857090
Tue Dec 04 13:10:29 2007  error -14 reading relation 5866635
Tue Dec 04 13:10:29 2007  error -14 reading relation 5871211
Tue Dec 04 13:10:29 2007  error -10 reading relation 5873221
Tue Dec 04 13:10:29 2007  error -14 reading relation 5874190
Tue Dec 04 13:10:29 2007  error -14 reading relation 5905772
Tue Dec 04 13:10:29 2007  error -14 reading relation 5930021
Tue Dec 04 13:10:30 2007  error -14 reading relation 5932480
Tue Dec 04 13:10:30 2007  error -4 reading relation 5934774
Tue Dec 04 13:10:30 2007  error -8 reading relation 5937165
Tue Dec 04 13:10:30 2007  error -4 reading relation 5941390
Tue Dec 04 13:10:30 2007  error -8 reading relation 5977883
Tue Dec 04 13:10:30 2007  error -8 reading relation 5991286
Tue Dec 04 13:10:30 2007  error -14 reading relation 5998515
Tue Dec 04 13:10:31 2007  error -14 reading relation 6003953
Tue Dec 04 13:10:31 2007  error -8 reading relation 6042710
Tue Dec 04 13:10:31 2007  error -8 reading relation 6053487
Tue Dec 04 13:10:31 2007  error -14 reading relation 6064196
Tue Dec 04 13:10:31 2007  error -10 reading relation 6070655
Tue Dec 04 13:10:31 2007  error -14 reading relation 6078638
Tue Dec 04 13:10:32 2007  error -14 reading relation 6080066
Tue Dec 04 13:10:32 2007  error -8 reading relation 6084296
Tue Dec 04 13:10:32 2007  error -4 reading relation 6092742
Tue Dec 04 13:10:32 2007  error -14 reading relation 6103652
Tue Dec 04 13:10:32 2007  error -4 reading relation 6112828
Tue Dec 04 13:10:32 2007  error -10 reading relation 6125742
Tue Dec 04 13:10:32 2007  error -8 reading relation 6135722
Tue Dec 04 13:10:32 2007  error -10 reading relation 6146821
Tue Dec 04 13:10:33 2007  error -14 reading relation 6156376
Tue Dec 04 13:10:33 2007  error -14 reading relation 6169873
Tue Dec 04 13:10:33 2007  error -14 reading relation 6183481
Tue Dec 04 13:10:33 2007  error -8 reading relation 6225142
Tue Dec 04 13:10:34 2007  error -10 reading relation 6276738
Tue Dec 04 13:10:35 2007  error -4 reading relation 6304157
Tue Dec 04 13:10:35 2007  error -10 reading relation 6326617
Tue Dec 04 13:10:35 2007  error -10 reading relation 6354026
Tue Dec 04 13:10:35 2007  error -4 reading relation 6354854
Tue Dec 04 13:10:35 2007  error -14 reading relation 6355304
Tue Dec 04 13:10:36 2007  error -14 reading relation 6376561
Tue Dec 04 13:10:36 2007  error -8 reading relation 6378968
Tue Dec 04 13:10:47 2007  error -10 reading relation 7278118
Tue Dec 04 13:10:48 2007  error -10 reading relation 7334584
Tue Dec 04 13:10:52 2007  error -10 reading relation 7655016
Tue Dec 04 13:10:53 2007  error -10 reading relation 7674321
Tue Dec 04 13:10:55 2007  error -8 reading relation 7837881
Tue Dec 04 13:10:55 2007  error -14 reading relation 7859138
Tue Dec 04 13:10:55 2007  error -14 reading relation 7869217
Tue Dec 04 13:10:55 2007  error -14 reading relation 7886570
Tue Dec 04 13:10:55 2007  error -14 reading relation 7891449
Tue Dec 04 13:10:57 2007  error -10 reading relation 7992010
Tue Dec 04 13:10:57 2007  error -5 reading relation 7992011
Tue Dec 04 13:10:57 2007  error -10 reading relation 8032342
Tue Dec 04 13:11:00 2007  error -5 reading relation 8235557
Tue Dec 04 13:11:02 2007  error -10 reading relation 8362862
Tue Dec 04 13:11:02 2007  error -5 reading relation 8365669
Tue Dec 04 13:11:02 2007  error -10 reading relation 8370094
Tue Dec 04 13:11:02 2007  error -10 reading relation 8375893
Tue Dec 04 13:11:02 2007  error -10 reading relation 8412589
Tue Dec 04 13:11:03 2007  error -10 reading relation 8470909
Tue Dec 04 13:11:05 2007  error -8 reading relation 8626055
Tue Dec 04 13:11:06 2007  error -14 reading relation 8669796
Tue Dec 04 13:11:06 2007  error -14 reading relation 8670002
Tue Dec 04 13:11:06 2007  error -5 reading relation 8676506
Tue Dec 04 13:11:06 2007  error -5 reading relation 8691497
Tue Dec 04 13:11:09 2007  error -14 reading relation 8908974
Tue Dec 04 13:11:09 2007  error -10 reading relation 8909405
Tue Dec 04 13:11:09 2007  error -4 reading relation 8915497
Tue Dec 04 13:11:09 2007  error -4 reading relation 8922181
Tue Dec 04 13:11:10 2007  error -5 reading relation 9011279
Tue Dec 04 13:11:10 2007  error -14 reading relation 9015500
Tue Dec 04 13:11:11 2007  error -1 reading relation 9086755
Tue Dec 04 13:11:11 2007  error -1 reading relation 9088676
Tue Dec 04 13:11:11 2007  error -10 reading relation 9090584
Tue Dec 04 13:11:11 2007  error -5 reading relation 9090585
Tue Dec 04 13:11:11 2007  error -10 reading relation 9092037
Tue Dec 04 13:11:12 2007  error -14 reading relation 9096909
Tue Dec 04 13:11:12 2007  error -14 reading relation 9135155
Tue Dec 04 13:11:14 2007  error -10 reading relation 9297646
Tue Dec 04 13:11:14 2007  error -5 reading relation 9297648
Tue Dec 04 13:11:17 2007  error -14 reading relation 9519549
Tue Dec 04 13:11:17 2007  error -10 reading relation 9521368
Tue Dec 04 13:11:21 2007  error -5 reading relation 9782145
Tue Dec 04 13:11:22 2007  error -10 reading relation 9913211
Tue Dec 04 13:11:22 2007  error -10 reading relation 9919713
Tue Dec 04 13:11:24 2007  error -10 reading relation 10055293
Tue Dec 04 13:11:25 2007  error -10 reading relation 10077286
Tue Dec 04 13:11:25 2007  error -10 reading relation 10102200
Tue Dec 04 13:11:26 2007  error -5 reading relation 10185695
Tue Dec 04 13:11:26 2007  error -5 reading relation 10193956
Tue Dec 04 13:11:26 2007  error -10 reading relation 10193959
Tue Dec 04 13:11:30 2007  error -5 reading relation 10456338
Tue Dec 04 13:11:30 2007  error -10 reading relation 10506088
Tue Dec 04 13:11:30 2007  error -5 reading relation 10523637
Tue Dec 04 13:11:30 2007  error -5 reading relation 10523639
Tue Dec 04 13:11:31 2007  error -5 reading relation 10542475
Tue Dec 04 13:11:34 2007  error -8 reading relation 10764915
Tue Dec 04 13:11:35 2007  error -10 reading relation 10850306
Tue Dec 04 13:11:36 2007  error -5 reading relation 10936120
Tue Dec 04 13:11:38 2007  error -10 reading relation 11141850
Tue Dec 04 13:11:40 2007  error -5 reading relation 11269458
Tue Dec 04 13:11:41 2007  error -5 reading relation 11304224
Tue Dec 04 13:11:42 2007  error -14 reading relation 11386501
Tue Dec 04 13:11:42 2007  error -14 reading relation 11405973
Tue Dec 04 13:11:42 2007  error -10 reading relation 11422309
Tue Dec 04 13:11:42 2007  error -8 reading relation 11441806
Tue Dec 04 13:11:43 2007  error -8 reading relation 11464991
Tue Dec 04 13:11:43 2007  error -8 reading relation 11483920
Tue Dec 04 13:11:45 2007  error -5 reading relation 11660410
Tue Dec 04 13:11:50 2007  error -10 reading relation 12003827
Tue Dec 04 13:11:54 2007  error -5 reading relation 12342825
Tue Dec 04 13:11:57 2007  error -5 reading relation 12597125
Tue Dec 04 13:12:01 2007  error -5 reading relation 12843945
Tue Dec 04 13:12:01 2007  error -5 reading relation 12843948
Tue Dec 04 13:12:03 2007  error -10 reading relation 13001085
Tue Dec 04 13:12:04 2007  error -10 reading relation 13133904
Tue Dec 04 13:12:04 2007  error -10 reading relation 13133907
Tue Dec 04 13:12:08 2007  error -10 reading relation 13379450
Tue Dec 04 13:12:08 2007  error -10 reading relation 13379454
Tue Dec 04 13:12:12 2007  error -5 reading relation 13685013
Tue Dec 04 13:12:12 2007  error -10 reading relation 13685015
Tue Dec 04 13:12:12 2007  error -10 reading relation 13740655
Tue Dec 04 13:12:12 2007  error -5 reading relation 13750759
Tue Dec 04 13:12:13 2007  error -10 reading relation 13779848
Tue Dec 04 13:12:16 2007  error -10 reading relation 13993393
Tue Dec 04 13:12:18 2007  error -10 reading relation 14147629
Tue Dec 04 13:12:19 2007  error -5 reading relation 14252038
Tue Dec 04 13:12:22 2007  error -10 reading relation 14488376
Tue Dec 04 13:12:23 2007  error -5 reading relation 14557023
Tue Dec 04 13:12:27 2007  error -10 reading relation 14857807
Tue Dec 04 13:12:29 2007  error -10 reading relation 14996690
Tue Dec 04 13:12:29 2007  error -5 reading relation 15041958
Tue Dec 04 13:12:30 2007  error -10 reading relation 15101064
Tue Dec 04 13:12:32 2007  error -5 reading relation 15216416
Tue Dec 04 13:12:33 2007  error -5 reading relation 15319216
Tue Dec 04 13:12:34 2007  error -5 reading relation 15371970
Tue Dec 04 13:12:36 2007  error -10 reading relation 15516065
Tue Dec 04 13:12:37 2007  error -10 reading relation 15593600
Tue Dec 04 13:12:43 2007  error -10 reading relation 16066864
Tue Dec 04 13:12:43 2007  error -5 reading relation 16075848
Tue Dec 04 13:12:45 2007  error -10 reading relation 16245702
Tue Dec 04 13:12:45 2007  error -5 reading relation 16246447
Tue Dec 04 13:12:46 2007  error -10 reading relation 16325059
Tue Dec 04 13:12:47 2007  error -5 reading relation 16404016
Tue Dec 04 13:12:48 2007  error -5 reading relation 16490631
Tue Dec 04 13:12:56 2007  error -10 reading relation 17051040
Tue Dec 04 13:12:57 2007  error -5 reading relation 17153360
Tue Dec 04 13:13:00 2007  error -5 reading relation 17389244
Tue Dec 04 13:13:01 2007  error -5 reading relation 17486671
Tue Dec 04 13:13:02 2007  error -10 reading relation 17532918
Tue Dec 04 13:13:02 2007  error -10 reading relation 17546351
Tue Dec 04 13:13:03 2007  error -5 reading relation 17579416
Tue Dec 04 13:13:05 2007  error -5 reading relation 17771751
Tue Dec 04 13:13:08 2007  error -5 reading relation 17964094
Tue Dec 04 13:13:08 2007  error -5 reading relation 18021212
Tue Dec 04 13:13:09 2007  error -10 reading relation 18047575
Tue Dec 04 13:13:09 2007  error -5 reading relation 18093020
Tue Dec 04 13:13:11 2007  error -5 reading relation 18197535
Tue Dec 04 13:13:11 2007  error -5 reading relation 18197539
Tue Dec 04 13:13:11 2007  error -10 reading relation 18201387
Tue Dec 04 13:13:11 2007  error -5 reading relation 18256273
Tue Dec 04 13:13:15 2007  error -10 reading relation 18512018
Tue Dec 04 13:13:16 2007  error -10 reading relation 18621036
Tue Dec 04 13:13:18 2007  error -5 reading relation 18787507
Tue Dec 04 13:13:20 2007  error -10 reading relation 18923838
Tue Dec 04 13:13:23 2007  error -5 reading relation 19113716
Tue Dec 04 13:13:24 2007  error -10 reading relation 19207896
Tue Dec 04 13:13:29 2007  error -14 reading relation 19418663
Tue Dec 04 13:13:30 2007  error -8 reading relation 19435546
Tue Dec 04 13:13:31 2007  error -14 reading relation 19453889
Tue Dec 04 13:13:31 2007  error -14 reading relation 19453992
Tue Dec 04 13:13:36 2007  error -10 reading relation 19827945
Tue Dec 04 13:13:40 2007  error -10 reading relation 20151482
Tue Dec 04 13:13:45 2007  error -10 reading relation 20552740
Tue Dec 04 13:13:48 2007  error -10 reading relation 20753907
Tue Dec 04 13:13:50 2007  error -10 reading relation 20924464
Tue Dec 04 13:13:50 2007  error -10 reading relation 20924466
Tue Dec 04 13:13:52 2007  error -10 reading relation 21055608
Tue Dec 04 13:13:52 2007  error -5 reading relation 21072465
Tue Dec 04 13:13:53 2007  error -5 reading relation 21153207
Tue Dec 04 13:13:54 2007  error -10 reading relation 21216470
Tue Dec 04 13:14:02 2007  error -5 reading relation 21835413
Tue Dec 04 13:14:02 2007  error -10 reading relation 21871369
Tue Dec 04 13:14:11 2007  error -10 reading relation 22507476
Tue Dec 04 13:14:11 2007  error -5 reading relation 22514973
Tue Dec 04 13:14:11 2007  error -5 reading relation 22568538
Tue Dec 04 13:14:13 2007  error -10 reading relation 22670288
Tue Dec 04 13:14:13 2007  error -5 reading relation 22696838
Tue Dec 04 13:14:16 2007  error -10 reading relation 22881214
Tue Dec 04 13:14:17 2007  error -5 reading relation 22960097
Tue Dec 04 13:14:22 2007  error -5 reading relation 23337293
Tue Dec 04 13:14:22 2007  error -10 reading relation 23341841
Tue Dec 04 13:14:22 2007  error -10 reading relation 23362747
Tue Dec 04 13:14:24 2007  error -5 reading relation 23511597
Tue Dec 04 13:14:24 2007  error -10 reading relation 23544015
Tue Dec 04 13:14:24 2007  error -10 reading relation 23544017
Tue Dec 04 13:14:26 2007  error -10 reading relation 23650203
Tue Dec 04 13:14:28 2007  error -10 reading relation 23833999
Tue Dec 04 13:14:28 2007  error -10 reading relation 23834001
Tue Dec 04 13:14:30 2007  error -10 reading relation 23969182
Tue Dec 04 13:14:31 2007  error -5 reading relation 24044865
Tue Dec 04 13:14:31 2007  error -10 reading relation 24068968
Tue Dec 04 13:14:32 2007  error -10 reading relation 24126474
Tue Dec 04 13:14:36 2007  error -5 reading relation 24478444
Tue Dec 04 13:14:37 2007  error -10 reading relation 24501442
Tue Dec 04 13:14:37 2007  error -5 reading relation 24501447
Tue Dec 04 13:14:39 2007  error -10 reading relation 24660207
Tue Dec 04 13:14:42 2007  error -10 reading relation 24942708
Tue Dec 04 13:14:44 2007  error -5 reading relation 25036910
Tue Dec 04 13:14:45 2007  error -5 reading relation 25105373
Tue Dec 04 13:14:47 2007  error -5 reading relation 25312534
Tue Dec 04 13:14:48 2007  error -10 reading relation 25351194
Tue Dec 04 13:14:49 2007  error -10 reading relation 25461209
Tue Dec 04 13:14:50 2007  error -5 reading relation 25542870
Tue Dec 04 13:14:50 2007  error -10 reading relation 25542879
Tue Dec 04 13:14:51 2007  error -10 reading relation 25610227
Tue Dec 04 13:14:52 2007  error -10 reading relation 25643670
Tue Dec 04 13:14:57 2007  error -10 reading relation 26084956
Tue Dec 04 13:14:59 2007  error -5 reading relation 26214707
Tue Dec 04 13:15:00 2007  error -10 reading relation 26252583
Tue Dec 04 13:15:00 2007  error -10 reading relation 26260375
Tue Dec 04 13:15:01 2007  error -5 reading relation 26355971
Tue Dec 04 13:15:01 2007  error -10 reading relation 26367552
Tue Dec 04 13:15:02 2007  error -10 reading relation 26466163
Tue Dec 04 13:15:03 2007  error -10 reading relation 26507046
Tue Dec 04 13:15:05 2007  found 5247486 hash collisions in 26547911 relations
Tue Dec 04 13:15:05 2007  commencing duplicate removal, pass 2
Tue Dec 04 13:17:29 2007  found 4354642 duplicates and 22193269 unique relations
Tue Dec 04 13:17:29 2007  memory use: 153.2 MB
Tue Dec 04 13:17:38 2007  ignoring smallest 908101 rational and 908887 algebraic ideals
Tue Dec 04 13:17:38 2007  filtering ideals above 13966909
Tue Dec 04 13:17:38 2007  need 3088879 more relations than ideals
Tue Dec 04 13:17:38 2007  commencing singleton removal, pass 1
Tue Dec 04 13:22:58 2007  relations with 0 large ideals: 219274
Tue Dec 04 13:22:58 2007  relations with 1 large ideals: 1343800
Tue Dec 04 13:22:58 2007  relations with 2 large ideals: 4357252
Tue Dec 04 13:22:58 2007  relations with 3 large ideals: 7368268
Tue Dec 04 13:22:58 2007  relations with 4 large ideals: 6507316
Tue Dec 04 13:22:58 2007  relations with 5 large ideals: 2397359
Tue Dec 04 13:22:58 2007  relations with 6 large ideals: 0
Tue Dec 04 13:22:58 2007  relations with 7+ large ideals: 0
Tue Dec 04 13:22:58 2007  22193269 relations and about 19993450 large ideals
Tue Dec 04 13:22:58 2007  commencing singleton removal, pass 2
Tue Dec 04 13:28:19 2007  found 6643525 singletons
Tue Dec 04 13:28:19 2007  current dataset: 15549744 relations and about 12409396 large ideals
Tue Dec 04 13:28:19 2007  commencing singleton removal, pass 3
Tue Dec 04 13:32:28 2007  found 1670389 singletons
Tue Dec 04 13:32:28 2007  current dataset: 13879355 relations and about 10663538 large ideals
Tue Dec 04 13:32:28 2007  commencing singleton removal, pass 4
Tue Dec 04 13:36:15 2007  found 460334 singletons
Tue Dec 04 13:36:15 2007  current dataset: 13419021 relations and about 10196784 large ideals
Tue Dec 04 13:36:15 2007  commencing singleton removal, final pass
Tue Dec 04 13:40:15 2007  memory use: 544.9 MB
Tue Dec 04 13:40:15 2007  commencing in-memory singleton removal
Tue Dec 04 13:40:18 2007  begin with 13419021 relations and 11408723 unique ideals
Tue Dec 04 13:40:53 2007  reduce to 10908638 relations and 8814846 ideals in 16 passes
Tue Dec 04 13:40:53 2007  max relations containing the same ideal: 35
Tue Dec 04 13:40:54 2007  dataset has 15.2% excess relations
Tue Dec 04 13:41:02 2007  ignoring smallest 822974 rational and 823921 algebraic ideals
Tue Dec 04 13:41:02 2007  filtering ideals above 12570218
Tue Dec 04 13:41:02 2007  need 1962347 more relations than ideals
Tue Dec 04 13:41:02 2007  commencing singleton removal, final pass
Tue Dec 04 13:45:02 2007  memory use: 544.9 MB
Tue Dec 04 13:45:02 2007  commencing in-memory singleton removal
Tue Dec 04 13:45:05 2007  begin with 13419021 relations and 11578777 unique ideals
Tue Dec 04 13:45:41 2007  reduce to 10906550 relations and 8982716 ideals in 16 passes
Tue Dec 04 13:45:41 2007  max relations containing the same ideal: 136
Tue Dec 04 13:45:43 2007  dataset has 5.9% excess relations
Tue Dec 04 13:45:44 2007  relations with 0 large ideals: 165233
Tue Dec 04 13:45:44 2007  relations with 1 large ideals: 819055
Tue Dec 04 13:45:44 2007  relations with 2 large ideals: 2349088
Tue Dec 04 13:45:44 2007  relations with 3 large ideals: 3580706
Tue Dec 04 13:45:44 2007  relations with 4 large ideals: 2928071
Tue Dec 04 13:45:44 2007  relations with 5 large ideals: 1064397
Tue Dec 04 13:45:44 2007  relations with 6 large ideals: 0
Tue Dec 04 13:45:44 2007  relations with 7+ large ideals: 0
Tue Dec 04 13:45:44 2007  commencing 2-way merge
Tue Dec 04 13:46:01 2007  reduce to 6538867 relation sets and 4615431 unique ideals
Tue Dec 04 13:46:01 2007  ignored 398 oversize relation sets
Tue Dec 04 13:46:01 2007  commencing full merge
Tue Dec 04 13:48:02 2007  found 3200440 cycles, need 2957631
Tue Dec 04 13:48:03 2007  weight of 2957631 cycles is about 192515367 (65.09/cycle)
Tue Dec 04 13:48:03 2007  distribution of cycle lengths:
Tue Dec 04 13:48:03 2007  1 relations: 493463
Tue Dec 04 13:48:03 2007  2 relations: 395406
Tue Dec 04 13:48:03 2007  3 relations: 377972
Tue Dec 04 13:48:03 2007  4 relations: 328986
Tue Dec 04 13:48:03 2007  5 relations: 286649
Tue Dec 04 13:48:03 2007  6 relations: 233674
Tue Dec 04 13:48:03 2007  7 relations: 193559
Tue Dec 04 13:48:03 2007  8 relations: 161830
Tue Dec 04 13:48:03 2007  9 relations: 132275
Tue Dec 04 13:48:03 2007  10+ relations: 353817
Tue Dec 04 13:48:03 2007  heaviest cycle: 15 relations
Tue Dec 04 13:48:04 2007  commencing cycle optimization
Tue Dec 04 13:48:12 2007  start with 14476317 relations
Tue Dec 04 13:48:55 2007  pruned 233348 relations
Tue Dec 04 13:48:55 2007  distribution of cycle lengths:
Tue Dec 04 13:48:55 2007  1 relations: 493463
Tue Dec 04 13:48:55 2007  2 relations: 401548
Tue Dec 04 13:48:55 2007  3 relations: 387785
Tue Dec 04 13:48:55 2007  4 relations: 333416
Tue Dec 04 13:48:55 2007  5 relations: 290778
Tue Dec 04 13:48:55 2007  6 relations: 234899
Tue Dec 04 13:48:55 2007  7 relations: 194525
Tue Dec 04 13:48:55 2007  8 relations: 160874
Tue Dec 04 13:48:55 2007  9 relations: 130758
Tue Dec 04 13:48:55 2007  10+ relations: 329585
Tue Dec 04 13:48:55 2007  heaviest cycle: 15 relations
Tue Dec 04 13:49:14 2007  
Tue Dec 04 13:49:14 2007  commencing linear algebra
Tue Dec 04 13:49:25 2007  read 2957631 cycles
Tue Dec 04 13:49:41 2007  cycles contain 8821038 unique relations
Tue Dec 04 13:56:33 2007  read 8821038 relations
Tue Dec 04 14:00:41 2007  using 32 quadratic characters above 268434998
Tue Dec 04 14:13:24 2007  read 2957631 cycles
Tue Dec 04 14:15:16 2007  matrix is 2954299 x 2957631 with weight 267444145 (avg 90.43/col)
Tue Dec 04 14:19:27 2007  filtering completed in 3 passes
Tue Dec 04 14:19:28 2007  matrix is 2907182 x 2907380 with weight 263467354 (avg 90.62/col)
Tue Dec 04 14:20:44 2007  read 2907380 cycles
Tue Dec 04 14:32:40 2007  matrix is 2907182 x 2907380 with weight 263467354 (avg 90.62/col)
Tue Dec 04 14:32:42 2007  saving the first 48 matrix rows for later
Tue Dec 04 14:32:46 2007  matrix is 2907134 x 2907380 with weight 199681658 (avg 68.68/col)
Tue Dec 04 14:32:46 2007  matrix includes 64 packed rows
Tue Dec 04 14:32:46 2007  using block size 43690 for processor cache size 1024 kB
Tue Dec 04 14:33:17 2007  commencing Lanczos iteration
Sat Dec 08 15:03:08 2007  lanczos halted after 45976 iterations (dim = 2907130)
Sat Dec 08 15:04:00 2007  recovered 48 nontrivial dependencies
Sat Dec 08 15:04:34 2007  
Sat Dec 08 15:04:34 2007  commencing square root phase
Sat Dec 08 15:04:34 2007  reading relations for dependency 1
Sat Dec 08 15:05:56 2007  read 1453274 cycles
Sat Dec 08 15:06:03 2007  cycles contain 5240367 unique relations
Sat Dec 08 15:17:04 2007  read 5240367 relations
Sat Dec 08 15:18:07 2007  multiplying 7036248 relations

Sat Dec  8 18:10:13 2007  Msieve v. 1.30
Sat Dec  8 18:10:13 2007  random seeds: 27cce325 7f0339cc
Sat Dec  8 18:10:13 2007  factoring 4881269226159859145363348158028800528357637730704107517078436952997684213093804119278586325654853291448226734543293119800229451979882856232013130645868074794819 (160 digits)
Sat Dec  8 18:10:14 2007  commencing number field sieve (160-digit input)
Sat Dec  8 18:10:14 2007  R0: -10000000000000000000000000000000000
Sat Dec  8 18:10:14 2007  R1:  1
Sat Dec  8 18:10:14 2007  A0: -4703
Sat Dec  8 18:10:14 2007  A1:  0
Sat Dec  8 18:10:14 2007  A2:  0
Sat Dec  8 18:10:14 2007  A3:  0
Sat Dec  8 18:10:14 2007  A4:  0
Sat Dec  8 18:10:14 2007  A5:  604999994990
Sat Dec  8 18:10:14 2007  size score = 3.430393e-14, Murphy alpha = 0.146027, combined = 3.267415e-14
Sat Dec  8 18:10:14 2007  
Sat Dec  8 18:10:14 2007  commencing square root phase
Sat Dec  8 18:10:14 2007  reading relations for dependency 1
Sat Dec  8 18:10:20 2007  read 1453274 cycles
Sat Dec  8 18:10:26 2007  cycles contain 5240367 unique relations
Sat Dec  8 18:12:30 2007  read 5240367 relations
Sat Dec  8 18:13:34 2007  multiplying 7036248 relations
Sat Dec  8 19:26:23 2007  multiply complete, coefficients have about 423.52 million bits
Sat Dec  8 19:26:47 2007  initial square root is modulo 39744721
Sat Dec  8 22:11:16 2007  reading relations for dependency 2
Sat Dec  8 22:11:29 2007  read 1453672 cycles
Sat Dec  8 22:11:38 2007  cycles contain 5239069 unique relations
Sat Dec  8 22:13:56 2007  read 5239069 relations
Sat Dec  8 22:14:57 2007  multiplying 7033374 relations
Sat Dec  8 23:32:29 2007  multiply complete, coefficients have about 423.35 million bits
Sat Dec  8 23:32:58 2007  initial square root is modulo 39464081
Sun Dec  9 02:17:16 2007  prp79 factor: 1825097233762709447432521941926649289213154260264910537140594516431173070300371
Sun Dec  9 02:17:16 2007  prp82 factor: 2674525573684858697560701870658348933916102325593721165422426453989766526938215889
Sun Dec  9 02:17:16 2007  elapsed time 08:07:03

jasonp · 2007-12-09, 20:51

Quote:

Originally Posted by frmky

I have a case, the Smarandache number sm94 actually, in which the binary from the website crashes in sqrt near the end of "multiplying xxx relations" but a Cygwin compiled binary works properly. The log file is below. Since the Cygwin binary worked, I'm not sure how to debug this locally. What do you use to compile the official binary? Otherwise, I can provide everything for you to download as a 2 GB tar file.

My guess is that win32 fails to allocate a huge number of small blocks of memory, but cygwin uses an advanced package like dlmalloc (IIRC this is also used in GGNFS) which succeeds. Does the sqrt need close to 2GB of memory before failing? At the least there should be complaints on malloc failure throughout the library.

Please PM when the download is ready. The official binary is compiled in MinGW using gcc 3.4.5

frmky · 2007-12-09, 20:59

Quote:

Originally Posted by jasonp

My guess is that win32 fails to allocate a huge number of small blocks of memory, but cygwin uses an advanced package like dlmalloc (IIRC this is also used in GGNFS) which succeeds. Does the sqrt need close to 2GB of memory before failing? At the least there should be complaints on malloc failure throughout the library.

The memory use oscillates from as large as 700MB to as little as 150MB or so during the run, but at the point of the crash it's using about 350MB. The computer this was running on only has 1.25GB of memory.

The tar file contains everything ready to run the sqrt step.

Greg

jasonp · 2007-12-12, 05:31

Quote:

Originally Posted by frmky

The memory use oscillates from as large as 700MB to as little as 150MB or so during the run, but at the point of the crash it's using about 350MB. The computer this was running on only has 1.25GB of memory.

I downloaded your tar file. As I suspected, the win32 binary runs out of VM just before completing the product of all the relations (this is when the biggest FFTs happen). It looks like my top priority is going to be getting the square root to take up less memory than the linear algebra; it's already getting in the way of finishing factorizations that would otherwise fit into one machine, and the problems will only get worse as people run huge GNFS jobs.

jbristow · 2007-12-21, 08:39

I've been getting errors trying to use 4-threaded Lanczos using Msieve 1.32 on my quad core. I just compiled a 64-bit version using Visual Studio 2008 tonight, so that may be relevant. The rest of the filtering has been working fine. I'm going to try 2 threads overnight to see if that works. Thanks.

Code:

Thu Dec 20 22:32:18 2007  commencing linear algebra
Thu Dec 20 22:32:19 2007  read 3940821 cycles
Thu Dec 20 22:32:33 2007  cycles contain 10791573 unique relations
Thu Dec 20 22:34:17 2007  read 10791573 relations
Thu Dec 20 22:34:33 2007  using 32 quadratic characters above 268435130
Thu Dec 20 22:39:46 2007  read 3940821 cycles
Thu Dec 20 22:39:53 2007  matrix is 3940621 x 3940821 with weight 357751078 (avg 90.78/col)
Thu Dec 20 22:40:06 2007  filtering completed in 1 passes
Thu Dec 20 22:40:07 2007  matrix is 3940621 x 3940821 with weight 357751078 (avg 90.78/col)
Thu Dec 20 22:40:31 2007  read 3940821 cycles
Thu Dec 20 22:40:35 2007  matrix is 3940621 x 3940821 with weight 357751078 (avg 90.78/col)
Thu Dec 20 22:40:35 2007  saving the first 48 matrix rows for later
Thu Dec 20 22:40:38 2007  matrix is 3940573 x 3940821 with weight 273207890 (avg 69.33/col)
Thu Dec 20 22:40:38 2007  matrix includes 64 packed rows
Thu Dec 20 22:40:38 2007  using block size 65536 for processor cache size 4096 kB
Thu Dec 20 22:40:55 2007  commencing Lanczos iteration (4 threads)
Thu Dec 20 23:16:06 2007  lanczos error: submatrix is not invertible
Thu Dec 20 23:16:06 2007  lanczos error (dim = 81606): not all columns used
Thu Dec 20 23:16:06 2007  lanczos halted after 1292 iterations (dim = 81606)
Thu Dec 20 23:16:06 2007  linear algebra failed; retrying...
Thu Dec 20 23:16:06 2007  commencing Lanczos iteration (4 threads)
Thu Dec 20 23:23:07 2007  lanczos error: submatrix is not invertible
Thu Dec 20 23:23:09 2007  lanczos error (dim = 16961): not all columns used
Thu Dec 20 23:23:09 2007  lanczos halted after 270 iterations (dim = 16961)
Thu Dec 20 23:23:09 2007  linear algebra failed; retrying...
Thu Dec 20 23:23:09 2007  commencing Lanczos iteration (4 threads)
Thu Dec 20 23:23:23 2007  lanczos halted after 7 iterations (dim = 439)
Thu Dec 20 23:23:23 2007  elapsed time 00:51:06

Code:

Thu Dec 20 23:55:01 2007  commencing linear algebra
Thu Dec 20 23:55:03 2007  read 3985391 cycles
Thu Dec 20 23:55:16 2007  cycles contain 10849197 unique relations
Thu Dec 20 23:56:54 2007  read 10849197 relations
Thu Dec 20 23:57:11 2007  using 32 quadratic characters above 268435130
Fri Dec 21 00:02:29 2007  read 3985391 cycles
Fri Dec 21 00:02:41 2007  matrix is 3982047 x 3985391 with weight 361094950 (avg 90.60/col)
Fri Dec 21 00:03:48 2007  filtering completed in 3 passes
Fri Dec 21 00:03:50 2007  matrix is 3940621 x 3940821 with weight 357751078 (avg 90.78/col)
Fri Dec 21 00:04:16 2007  read 3940821 cycles
Fri Dec 21 00:04:34 2007  matrix is 3940621 x 3940821 with weight 357751078 (avg 90.78/col)
Fri Dec 21 00:04:34 2007  saving the first 48 matrix rows for later
Fri Dec 21 00:04:37 2007  matrix is 3940573 x 3940821 with weight 273207890 (avg 69.33/col)
Fri Dec 21 00:04:37 2007  matrix includes 64 packed rows
Fri Dec 21 00:04:37 2007  using block size 65536 for processor cache size 4096 kB
Fri Dec 21 00:04:54 2007  commencing Lanczos iteration (4 threads)
Fri Dec 21 00:23:28 2007  lanczos error: submatrix is not invertible
Fri Dec 21 00:23:28 2007  lanczos error (dim = 44431): not all columns used
Fri Dec 21 00:23:28 2007  lanczos halted after 704 iterations (dim = 44431)
Fri Dec 21 00:23:28 2007  linear algebra failed; retrying...
Fri Dec 21 00:23:28 2007  commencing Lanczos iteration (4 threads)
Fri Dec 21 00:23:46 2007  lanczos halted after 9 iterations (dim = 570)

2007-12-07, 16:07	#147
dleclair Mar 2003 1001101₂ Posts	Hi Jason, Before looking at the Windows API functions for file I/O, try adding "b" to the mode specified with your fopen called as in fopen(filename,"rb"). While Paul Leyland was doing the filtering for the M811 project by NFSNET there were some very odd things happening and I eventually tracked it down to a problem in the Microsoft run time library. The problem manifested itself in a different way but it was related to reading files larger than 4G in size. Simply adding "b" to the mode on fopen() was enough to work around it. The "b" ought to be ignored when compiled on non-Windows platforms. To be safe you could use a #defined value based on the architecture you're compiling for. -Don Last fiddled with by dleclair on 2007-12-07 at 16:08

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
error when running msieve 1.53 with cuda	aein	Msieve	9	2019-02-25 14:09
Help need to running Msieve	appleseed	Msieve	12	2016-04-10 02:31
Problem in running msieve with CUDA	mohamed	Msieve	20	2013-08-01 08:27
CUDA_ERROR_LAUNCH_OUT_OF_RESOURCES when running msieve 1.5.0 with CUDA	ryanp	Msieve	3	2012-06-12 03:27
Trouble Running Msieve	Sab	Msieve	4	2009-07-07 06:19

2007-12-07, 02:47	#145
jbristow Aug 2007 135₈ Posts	Thanks for all of the replies. I'm running 64-bit Windows Vista on a NTFS filesystem, but using the precompiled Win32 msieve binary. I can create files over 4 GB, copy them between folders, etc. When I create a 3.7 GB subset of partial relations, msieve runs fine. However, if I try to use a 4.4 GB set of relations, I get the error: error: cannot open savefile 'msieve.dat' right after the line that includes the size score, murphy alpha, etc. I've tried adding a set of duplicate relations to the successful 3.7 GB file (just to make sure the error isn't due to a corrupt file) that results in a file over 4 GB and I get the same error.