Msieve v1.46 feedback

[jasonp]

Could you post the bottom 20 lines or so from msieve.log? Serge noticed a few small problems just after v1.46 was released

[em99010pepe]

Originally Posted by jasonp

Could you post the bottom 20 lines or so from msieve.log? Serge noticed a few small problems just after v1.46 was released

It only has the last ETA for LA from last time I resumed the client.
Meanwhile I moved to 1.44 version because it was the second time I had this issue. When I get home I'll try to upload here the log file, right now I am connecting remotely to the machine and the connection is too slow, can't do much thing.

[em99010pepe]

Nothing on the log file when I got twice this issue.

[Jeff Gilchrist]

Originally Posted by em99010pepe

Nothing on the log file when I got twice this issue.

Can you at least recover and resume where it left off when the shutdown happens? I haven't run a LA yet with the new binaries for more than 20 hours so not sure if it happens on my machine or not.

[em99010pepe]

Originally Posted by Jeff Gilchrist

Can you at least recover and resume where it left off when the shutdown happens? I haven't run a LA yet with the new binaries for more than 20 hours so not sure if it happens on my machine or not.

Yes I can but msieve doesn't write nothing on the log file about the issue.

[Jeff Gilchrist]

Looks like I've found a 1.46 issue (Linux 64bit gcc). When doing -nc1 on a 13GB .dat file it exits very quickly complaining there aren't enough relations.

Log:

Thu Aug 12 12:14:30 2010
Thu Aug 12 12:14:30 2010
Thu Aug 12 12:14:30 2010  Msieve v. 1.46
Thu Aug 12 12:14:30 2010  random seeds: 44e3e62f c129137c
Thu Aug 12 12:14:30 2010  factoring 335667454643536401997713550684360388383673964653341689017234725356575148227657763537670896061660038005916979269185735713736837775567776947886978999115079022247460420423249240186294119945185233833921432222687627049 (213 digits)
Thu Aug 12 12:14:33 2010  no P-1/P+1/ECM available, skipping
Thu Aug 12 12:14:33 2010  commencing number field sieve (213-digit input)
Thu Aug 12 12:14:33 2010  R0: -765553441981431173142397222131008751601
Thu Aug 12 12:14:33 2010  R1:  1
Thu Aug 12 12:14:33 2010  A0: -599
Thu Aug 12 12:14:33 2010  A1:  0
Thu Aug 12 12:14:33 2010  A2:  0
Thu Aug 12 12:14:33 2010  A3:  0
Thu Aug 12 12:14:33 2010  A4:  0
Thu Aug 12 12:14:33 2010  A5:  0
Thu Aug 12 12:14:33 2010  A6:  1
Thu Aug 12 12:14:33 2010  skew 2.90, size 1.619e-11, alpha 0.515, combined = 9.089e-13 rroots = 2
Thu Aug 12 12:14:33 2010
Thu Aug 12 12:14:33 2010  commencing relation filtering
Thu Aug 12 12:14:33 2010  estimated available RAM is 32159.2 MB
Thu Aug 12 12:14:33 2010  commencing duplicate removal, pass 1
Thu Aug 12 12:14:33 2010  error -15 reading relation 14690
Thu Aug 12 12:14:39 2010  error -9 reading relation 290519
Thu Aug 12 12:14:43 2010  found 5716 hash collisions in 514614 relations
Thu Aug 12 12:14:48 2010  added 323599 free relations
Thu Aug 12 12:14:48 2010  commencing duplicate removal, pass 2
Thu Aug 12 12:14:54 2010  found 1888 duplicates and 836325 unique relations
Thu Aug 12 12:14:54 2010  memory use: 4.2 MB
Thu Aug 12 12:14:54 2010  reading ideals above 30000
Thu Aug 12 12:14:54 2010  commencing singleton removal, initial pass
Thu Aug 12 12:15:17 2010  memory use: 94.1 MB
Thu Aug 12 12:15:18 2010  reading all ideals from disk
Thu Aug 12 12:15:18 2010  memory use: 35.8 MB
Thu Aug 12 12:15:18 2010  commencing in-memory singleton removal
Thu Aug 12 12:15:18 2010  begin with 836325 relations and 4660339 unique ideals
Thu Aug 12 12:15:18 2010  reduce to 269 relations and 0 ideals in 3 passes
Thu Aug 12 12:15:18 2010  max relations containing the same ideal: 0
Thu Aug 12 12:15:18 2010  filtering wants 1000000 more relations
Thu Aug 12 12:15:18 2010  elapsed time 00:00:48

Going back to 1.45 it seems to be processing happily along:

Thu Aug 12 12:23:32 2010  
Thu Aug 12 12:23:32 2010  
Thu Aug 12 12:23:32 2010  Msieve v. 1.45
Thu Aug 12 12:23:32 2010  random seeds: edccfec1 c9935416
Thu Aug 12 12:23:32 2010  factoring 335667454643536401997713550684360388383673964653341689017234725356575148227657763537670896061660038005916979269185735713736837775567776947886978999115079022247460420423249240186294119945185233833921432222687627049 (213 digits)
Thu Aug 12 12:23:35 2010  no P-1/P+1/ECM available, skipping
Thu Aug 12 12:23:35 2010  commencing number field sieve (213-digit input)
Thu Aug 12 12:23:35 2010  R0: -765553441981431173142397222131008751601
Thu Aug 12 12:23:35 2010  R1:  1
Thu Aug 12 12:23:36 2010  A0: -599
Thu Aug 12 12:23:36 2010  A1:  0
Thu Aug 12 12:23:36 2010  A2:  0
Thu Aug 12 12:23:36 2010  A3:  0
Thu Aug 12 12:23:36 2010  A4:  0
Thu Aug 12 12:23:36 2010  A5:  0
Thu Aug 12 12:23:36 2010  A6:  1
Thu Aug 12 12:23:36 2010  skew 2.90, size 1.619349e-11, alpha 0.514692, combined = 9.088922e-13
Thu Aug 12 12:23:36 2010  
Thu Aug 12 12:23:36 2010  commencing relation filtering
Thu Aug 12 12:23:36 2010  estimated available RAM is 32159.2 MB
Thu Aug 12 12:23:36 2010  commencing duplicate removal, pass 1
Thu Aug 12 12:24:09 2010  error -9 reading relation 3678290
Thu Aug 12 12:24:12 2010  error -9 reading relation 4014507
Thu Aug 12 12:24:28 2010  error -15 reading relation 5803238
Thu Aug 12 12:24:29 2010  error -5 reading relation 5921577
Thu Aug 12 12:24:33 2010  error -9 reading relation 6445007
Thu Aug 12 12:24:41 2010  error -9 reading relation 7265536
Thu Aug 12 12:24:42 2010  error -9 reading relation 7447161
Thu Aug 12 12:24:42 2010  error -9 reading relation 7466817
Thu Aug 12 12:24:47 2010  error -9 reading relation 8011132
Thu Aug 12 12:24:56 2010  error -15 reading relation 8960855
Thu Aug 12 12:25:03 2010  error -15 reading relation 9748335
Thu Aug 12 12:25:50 2010  error -9 reading relation 15027256
Thu Aug 12 12:25:52 2010  error -5 reading relation 15265787
Thu Aug 12 12:26:10 2010  error -9 reading relation 17312295
Thu Aug 12 12:26:37 2010  error -15 reading relation 20298573
Thu Aug 12 12:26:40 2010  error -5 reading relation 20597954
Thu Aug 12 12:26:44 2010  error -5 reading relation 21081025
Thu Aug 12 12:26:45 2010  error -15 reading relation 21167922
Thu Aug 12 12:27:01 2010  error -15 reading relation 23021910
Thu Aug 12 12:27:37 2010  error -15 reading relation 26989824
Thu Aug 12 12:28:15 2010  error -15 reading relation 31146499
Thu Aug 12 12:28:16 2010  error -15 reading relation 31325475
Thu Aug 12 12:28:21 2010  error -15 reading relation 31875893
Thu Aug 12 12:28:48 2010  error -15 reading relation 34878454
Thu Aug 12 12:28:55 2010  error -15 reading relation 35626200
Thu Aug 12 12:28:55 2010  error -9 reading relation 35626662
...
Thu Aug 12 12:41:00 2010  found 21979631 hash collisions in 112609571 relations
Thu Aug 12 12:41:30 2010  added 1218611 free relations
Thu Aug 12 12:41:30 2010  commencing duplicate removal, pass 2

Any idea what might be wrong or what I can do to help you track this one down?

And can I still use 1.46 to do -nc2 an -nc3 without any issues from a 1.45 -nc1?

Jeff.

[em99010pepe]

At least your msieve wrote something on the log file.
Meanwhile I started a new LA with 1.46 windows 64-bit version, let's see what happens. So far no problems.

Edit:
A question for the experts, why everytime a restart LA a new windows explorer.exe service is turned on. This only happens with msieve, with ecmclient doesn't happens, and both use a DOS windows. Thank you.

[jasonp]

Jeff, can you recompile with NO_ZLIB=1 and see if that works?

Carlos: as far as I know msieve.exe is an ordinary windows console application; I have no idea why an explorer window would be popping up, unless maybe the LA chews up a lot of memory and your system starts to misbehave because of that.

Serge: even on my local linux system that has zlib, when trying to compile the latest SVN I get an undefined symbol error for 'gzdirect'; is this a function that only appears in a modern version of zlib?

[Jeff Gilchrist]

Originally Posted by jasonp

Jeff, can you recompile with NO_ZLIB=1 and see if that works?

Seems to be working so far, still processing and reporting relation errors so looks like that is it. Made it to pass 2:

Thu Aug 12 14:33:18 2010  skipped 3 relations with b > 2^32
Thu Aug 12 14:33:18 2010  found 22057330 hash collisions in 113828182 relations
Thu Aug 12 14:33:41 2010  commencing duplicate removal, pass 2
Thu Aug 12 14:35:10 2010  found 21598096 duplicates and 92230086 unique relations
Thu Aug 12 14:35:10 2010  memory use: 660.8 MB

Jeff.

[Batalov]

Originally Posted by jasonp

Jeff, can you recompile with NO_ZLIB=1 and see if that works?

Carlos: as far as I know msieve.exe is an ordinary windows console application; I have no idea why an explorer window would be popping up, unless maybe the LA chews up a lot of memory and your system starts to misbehave because of that.

Serge: even on my local linux system that has zlib, when trying to compile the latest SVN I get an undefined symbol error for 'gzdirect'; is this a function that only appears in a modern version of zlib?

I heard about gzdirect from Greg, too. This puzzles me.
Note that gzdirect() is available for >5 years in zlib. For a check, I have retrieved some _very_ old versions of zlib on the internet in wayback-machine archives. gzdirect is absent from v.1.1.* (which are dated circa 1998!), appeared after v.1.2.1 (dated 2002) and is present in 1.2.3 (dated 2005). But even 1.2.3 is five years old. What kind of a computer are you guys running there? :-)
Could you have a look in /usr/include/zlib.h for the version and timestamp?

I will write a spoof of this function; in a nutshell, it checks if the file has gzip, zip or .Z magic header and returns 0. If they are absent, file is treated as direct (i.e. plain), and this function returns 1. Because it is only two bytes to read, I can get rid of this function. Heck, even my SGI has zlib v.1.2.3 -- and it is an 11 year old computer. (Of course, I did install it, when I've built libpng. Originally, it did have v.1.1.3 a.f.a.i.r.)

In any case, update to 1.2.5 is recommended - there are security vulnerabilities (e.g. in a malicious PNG file).
_________

@Jeff: It seems that reading the file is abandoned after 2 or 4Gb. Did you compile msieve with -D_LARGEFILE64_SOURCE ? It may be needed for zlib on some but not all 64-bit systems, and not on 32-bit systems of course. (You probably already have -D_FILE_OFFSET_BITS=64 set up.)

/sigh/ We may need a full-blown ./configure some time soon.

[Jeff Gilchrist]

Originally Posted by Batalov

@Jeff: It seems that reading the file is abandoned after 2 or 4Gb. Did you compile msieve with -D_LARGEFILE64_SOURCE ? It may be needed for zlib on some but not all 64-bit systems, and not on 32-bit systems of course. (You probably already have -D_FILE_OFFSET_BITS=64 set up.)

I just used the standard Makefile so -D_FILE_OFFSET_BITS=64 is set by default but there is no -D_LARGEFILE64_SOURCE in the Makefile.

I re-compiled with ZLIB re-enabled and added -D_LARGEFILE64_SOURCE but it still has the same problem, dies very quickly after reading a small number of relations so that doesn't seem to be the problem.

So the v1.46 (NO_ZLIB=1) completely -nc1 successfully. Nice to see that v1.46 (elapsed time 01:11:33) has a speed improvement from v1.45 (elapsed time 01:29:11).

Jeff.