MSieve - problem with 106-digit effort
 

Jason,
I'm trying to complete a 106-digit factorization and have been having trouble. It seems to be related to the size of the msieve.dat file.
Platform: Win32, 2GHz Athlon, 1024MB RAM.
Here's the relevant part of the log file:
[CODE]Sun Aug 27 11:21:56 2006 restarting with 21328 full and 1330462 partial relations
Mon Aug 28 13:44:02 2006 158704 relations (37546 full + 121158 combined from 2338819 partial), need 158588
Mon Aug 28 13:44:04 2006 begin with 1330462 relations
Mon Aug 28 13:44:07 2006 reduce to 354578 relations in 6 passes
Mon Aug 28 13:44:07 2006 attempting to read 21328 full and 354578 partial relations
Mon Aug 28 13:44:14 2006 recovered 21328 full and 354578 partial relations
Mon Aug 28 13:44:14 2006 recovered 363583 polynomials
Mon Aug 28 13:44:15 2006 attempting to build 20686 cycles
Mon Aug 28 13:44:16 2006 found 20686 cycles in 5 passes
Mon Aug 28 13:44:17 2006 distribution of cycle lengths:
Mon Aug 28 13:44:17 2006 length 2 : 8819
Mon Aug 28 13:44:17 2006 length 3 : 5410
Mon Aug 28 13:44:17 2006 length 4 : 2961
Mon Aug 28 13:44:17 2006 length 5 : 1666
Mon Aug 28 13:44:17 2006 length 6 : 922
Mon Aug 28 13:44:17 2006 length 7 : 450
Mon Aug 28 13:44:17 2006 length 8 : 234
Mon Aug 28 13:44:17 2006 length 9+: 224
Mon Aug 28 13:44:17 2006 largest cycle: 16 relations
Mon Aug 28 13:44:17 2006 158492 x 42014 system, weight 2006412 (avg 47.76/col)
Mon Aug 28 13:44:17 2006 reduce to 0 x 0 in 2 passes
Mon Aug 28 13:44:17 2006 matrix is corrupt; skipping linear algebra
Mon Aug 28 13:44:18 2006 elapsed time 26:22:26
[/CODE]
I think the problem is the size of the dat file. It's around 330MB and for some reason MSieve can't see all of it. After this run bombed out, I tried rebooting and restarting the run. This resulted in this happy news:[CODE]Mon Aug 28 13:52:51 2006 Msieve v. 1.10
Mon Aug 28 13:52:51 2006 random seeds: bb16ba04 9c9927d6
Mon Aug 28 13:52:51 2006 factoring 6531298320380971696488045124260547244186024707542979107601519157125452153281839347430584540873072615628919 (106 digits)
Mon Aug 28 13:52:52 2006 using multiplier of 1
Mon Aug 28 13:52:52 2006 sieve interval: 11 blocks of size 65536
Mon Aug 28 13:52:52 2006 processing polynomials in batches of 5
Mon Aug 28 13:52:52 2006 using a sieve bound of 4523059 (158492 primes)
Mon Aug 28 13:52:52 2006 using large prime bound of 678458850 (29 bits)
Mon Aug 28 13:52:52 2006 using double large prime bound of 7883921834550150 (45-53 bits)
Mon Aug 28 13:52:52 2006 using trial factoring cutoff of 60 bits
Mon Aug 28 13:52:52 2006 polynomial 'A' values have 14 factors
Mon Aug 28 13:52:55 2006 restarting with 21328 full and 1330462 partial relations[/CODE]
As you can see, it's not reporting all the relations that resulted from the last run. In fact, several restarts resulted in the same result: only a portion of the total relations available are being reported....

Later,
Frank

[QUOTE=schickel;85779]Jason,
I'm trying to complete a 106-digit factorization and have been having trouble. It seems to be related to the size of the msieve.dat file.
[/QUOTE]
The file should be okay; more likely it's a minor bug in the postprocessing since that got a fairly major overhaul in v1.08

I think I know what's wrong and will PM you with a test binary later tonight. The trouble with the last few versions has made it clear that I need stable and unstable branches available again. I should repost 1.07; if somebody wants to make that version available, it can probably finish your factorization immediately.

jasonp

Jason,
[QUOTE=jasonp] I should repost 1.07; if somebody wants to make that version available, it can probably finish your factorization immediately.

jasonp[/QUOTE]
Actually, I was running 1.07 and downloaded 1.10 in case it could finish:[CODE]Mon Aug 28 13:51:06 2006 Msieve v. 1.07
Mon Aug 28 13:51:06 2006 random seeds: 7dc36fae 31bbd887
Mon Aug 28 13:51:06 2006 factoring 6531298320380971696488045124260547244186024707542979107601519157125452153281839347430584540873072615628919 (106 digits)
Mon Aug 28 13:51:07 2006 using multiplier of 1
Mon Aug 28 13:51:07 2006 sieve interval: 11 blocks of size 65536
Mon Aug 28 13:51:07 2006 processing polynomials in batches of 5
Mon Aug 28 13:51:07 2006 using a sieve bound of 4523059 (158492 primes)
Mon Aug 28 13:51:07 2006 using large prime bound of 678458850 (29 bits)
Mon Aug 28 13:51:07 2006 using double large prime bound of 7883921834550150 (45-53 bits)
Mon Aug 28 13:51:07 2006 using trial factoring cutoff of 60 bits
Mon Aug 28 13:51:07 2006 polynomial 'A' values have 14 factors
Mon Aug 28 13:51:10 2006 restarting with 21328 full and 1330462 partial relations[/CODE]It looks like it's still not seeing all the relations either.....:confused:

Later,
Frank

[QUOTE=schickel;85786]
Actually, I was running 1.07 and downloaded 1.10 in case it could finish

It looks like it's still not seeing all the relations either.....:confused:
[/QUOTE]
That's a bit strange.

If 1.07 also messed up, then it's probably something to do with the data, like you thought. 330MB is not that big for a savefile; my NFS tests use a savefile more than twice that size. Unfortunately, it's far too large for anything other than remote forensics.

The explanation that fits the evidence is that something is wrong with the first character of relations generated during the restarted run. If the relations were seen but were corrupted somehow, you'd have lots of complaints in the savefile. Does msieve.dat contain relations generated from both unix and windows systems?

If you don't have unix utilities on your winXP system, could you download

[url]ftp://ftp.delorie.com/pub/djgpp/current/v2gnu/txt20b.zip[/url]

and then
- unzip bin/wc.exe, put it somewhere in your path, and run 'wc msieve.dat'
- unzip bin/tail.exe, put it somewhere in your path, and run 'tail msieve.dat'; this should give you the last ten lines of the savefile.

jasonp

Jason,
I'll give it a try. But I'm running Win98, not XP; could that be the problem? Also, this was the second line of a two-line worktodo.ini. Will the dat file have been erased when it started the second run?

Later,
Frank

PS. Ok, had to get the info off the other PC, but here it is:
[CODE]C:\msieve>wc msieve.dat
4327893 82003236 333592502 msieve.dat

C:\msieve>





R 49a0 1fdb 0 3 3 3 4 17 5f 67 257 549 659 1c59e 2019d 2343e L 58e5a7 191af2f
R 42c49 1fdd 0 1 3 4 18c 315 427 1f9c 5357 5da7 82ae 8fc4 L 11a82a5 876adcb
R -5c84a 1fe3 0 1 2 2 2 3 5 34 3c 898 1664 38f6 40d4 142ac 1a32e L a79461 297a793
R 9e4db 1fe3 2 4 5 7 9 1f ac 143 9d8 a85 6c82 21202 213db L a1c399 5b7db63
R 8ed6d 1fe4 1 3 4 5 d 17 32 7a 32f 399 cf8 2790 9220 18914 1f1b7 L 1 3c704bf
R 2f4bc 1fed 0 3 5 5 8 be f4 3c7 948 103c 568e 107d9 22918 L fd006d 171a8a1
R -73547 1ff2 1 2f 9fd c3e 13bc 37f5 3ffc b8c8 1c376 L b33c59 2016c3a7
R 81c3d 1ff3 1 2 7 b 2e 52 7a 20dc 83f7 986d 1065b 1843a 260d1 L 1 1e8160bf
R -51896 1ff6 0 4 b f 130 1b7 40c 1a36 46f5 5464 6a11 bf70 ec60 L 1 149e073
R -940a4 1ff8 1 4 5 c 2c 82 1bd 28ad 34ba 5b22 128d1 1febe L 565e71 d98f1ed[/CODE]
Thanks for taking the time to look at this!

Looks like the problem was a computer crash corruptinhg the msieve.dat file. Kudos to Jason for the effort to try and help recover the work done on this factorization, but looks like no go; just have to rerun it.....