Python Driver for GGNFS and MSIEVE

[Brian Gladman]

Unfortunately there is no very easy fix for this because adding quotes in all cases will fail when there are no spaces. The quotes have to be added but only when the file name contains a space. I'll look into this.

Brian

[Brian Gladman]

I have no idea whether this will work (I'm doubtful) but in run_msieve, in place of:

rel = os.path.relpath(cwd)
dp = os.path.join(rel, DATNAME)

you could try:

rel = os.path.relpath(cwd)
if rel.find(' ') != -1:
  rel = '"' + rel + '"'
dp = os.path.join(rel, DATNAME)

Brian

[EdH]

Any words of wisdom would be appreciated:

Total yield: 616370
0/0 mpqs failures, 59001/28820 vain mpqs
milliseconds total: Sieve 2835180 Sched 0 medsched 506619
TD 2306222 (Init 162583, MPQS 262133) Sieve-Change 1803637
TD side 0: init/small/medium/large/search: 23731 244635 34167 61537 840267
sieve: init/small/medium/large/search: 66764 822141 87382 399003 193663
TD side 1: init/small/medium/large/search: 9125 152179 33135 55861 688322
sieve: init/small/medium/large/search: 46606 531264 82736 404288 201333
appending spairs.out.T0 to spairs.out
appending spairs.out to test.dat
Found 11171072 relations, 100.5% of the estimated minimum (11115000).
-> msieve -s ggnfs_5835729598046643344594576944853108561052208334385324070741681
153732787325381841796727815086982722157617747\test.dat -l ggnfs_5835729598046643
34459457694485310856105220833438532407074168115373278732538184179672781508698272
2157617747\test.log -i ggnfs_583572959804664334459457694485310856105220833438532
4070741681153732787325381841796727815086982722157617747\test.ini -nf ggnfs_58357
29598046643344594576944853108561052208334385324070741681153732787325381841796727
815086982722157617747\test.fb -t 1 -nc1
read 10M relations
error: too many merge attempts
Return value 0. Terminating...
siever terminated
WARNING: gnfs failed to find a factor. This really shouldn't happen.
I'll just run ecm till the end of time or a factor turns up...
Let's hope you don't run out of disk space before either of those.
c106: running 0 auto-increasing ecm curves...

2.4GHz Intel Pentium 4, 512MB, WinXP, Python 3.1, script 0.62

Since it's late, I have started another run to (hopefully) add to the previous relations. I should know more in the morning...

[10metreh]

Have you got the ggnfs.log file? It seems there was no -v in the msieve command line, so msieve didn't print much out.

This error can be caused by (among other things) having way too many relations. What were the large prime bounds? (lpbr and lpba in the .job file) If 26, then you have more than twice the necessary number of rels. If 27, then 7M relations should be better.

[Brian Gladman]

It is not a Python problem. Given 10metreh's comment, should I add a -v to any of the msieve command lines in the Python code?

Brian

[smh]

Originally Posted by Brian Gladman

I have no idea whether this will work (I'm doubtful) but in run_msieve, in place of:

rel = os.path.relpath(cwd)
dp = os.path.join(rel, DATNAME)

you could try:

rel = os.path.relpath(cwd)
if rel.find(' ') != -1:
  rel = '"' + rel + '"'
dp = os.path.join(rel, DATNAME)

Brian

That makes it even worse. Now it also doesn't do the sieving.

[10metreh]

Originally Posted by Brian Gladman

It is not a Python problem. Given 10metreh's comment, should I add a -v to any of the msieve command lines in the Python code?

Brian

I should have been clearer. I was referring to "too many merge attempts", and that is from msieve. Running msieve with -v makes it print all its output to the screen as well as the logfile. This makes it easier to observe what's going on and what exactly is msieve output, so I would add it. In EdH's case, it might be possible to determine exactly why msieve failed if we had the log (and EdH probably still does). "WARNING: gnfs failed to find a factor. This really shouldn't happen." and onwards is aliqueit. I'm not sure about the "siever terminated", though.

[WraithX]

Originally Posted by smh

That makes it even worse. Now it also doesn't do the sieving.

@smh, Could you show us what the path to your msieve.exe is? And also the path of your ggnfs exe's? I'd like to reproduce the error locally to see what the error message says and try to fix it here.

@Brian, if we add verboseness to the script, can we make it optional? Maybe on by default (true), but with the option to be false so we get the same output as now?

[Brian Gladman]

Originally Posted by 10metreh

I should have been clearer. I was referring to "too many merge attempts", and that is from msieve. Running msieve with -v makes it print all its output to the screen as well as the logfile. This makes it easier to observe what's going on and what exactly is msieve output, so I would add it. In EdH's case, it might be possible to determine exactly why msieve failed if we had the log (and EdH probably still does). "WARNING: gnfs failed to find a factor. This really shouldn't happen." and onwards is aliqueit. I'm not sure about the "siever terminated", though.

That's just a wrong description of the Python script finishing with some error from a called program. It should really say 'factmsieve terminating'.

Brian

[EdH]

Originally Posted by 10metreh

Have you got the ggnfs.log file? It seems there was no -v in the msieve command line, so msieve didn't print much out.

This error can be caused by (among other things) having way too many relations. What were the large prime bounds? (lpbr and lpba in the .job file) If 26, then you have more than twice the necessary number of rels. If 27, then 7M relations should be better.

Then I guess last night's run made it even worse.

Found 12437340 relations, 111.9% of the estimated minimum (11115000).

It ended the same, as would be expected. BTW, this is the trouble I experienced on my trip, which I referenced in post #363. The only differences were: I was running 0.51 and it was a c125.

On to further info: (The ggnfs.log file only had a series of "LatSieveTime: 8078" type entries.

Test.job.T0:

n: 5835729598046643344594576944853108561052208334385324070741681153732787325381841796727815086982722157617747
c5: 3060
c4: -282923474
c3: 3252893327075
c2: -61156523169760400
c1: -2113433513605023016052
c0: -5560560654456680704172400
Y1: 95330612729
Y0: -285810003640834066631
skew: 25011.05  
rlim: 2500000
alim: 2500000
lpbr: 26
lpba: 26
mfbr: 49
mfba: 49
rlambda: 2.6
alambda: 2.6
q0: 3650000
qintsize: 150000
#q1:3800000

msieve.log

Sat Apr 03 07:33:43 2010  
Sat Apr 03 07:33:43 2010  
Sat Apr 03 07:33:43 2010  Msieve v. 1.44
Sat Apr 03 07:33:43 2010  random seeds: 641ef5dc 51fb871a
Sat Apr 03 07:33:43 2010  factoring 5835729598046643344594576944853108561052208334385324070741681153732787325381841796727815086982722157617747 (106 digits)
Sat Apr 03 07:33:45 2010  searching for 15-digit factors
Sat Apr 03 07:33:46 2010  commencing number field sieve (106-digit input)
Sat Apr 03 07:33:46 2010  commencing number field sieve polynomial selection
Sat Apr 03 07:33:46 2010  time limit set to 0.62 hours
Sat Apr 03 07:33:46 2010  searching leading coefficients from 1 to 6474
Sat Apr 03 08:19:13 2010  polynomial selection complete
Sat Apr 03 08:19:13 2010  R0: -285810003640834066631
Sat Apr 03 08:19:13 2010  R1:  95330612729
Sat Apr 03 08:19:13 2010  A0: -5560560654456680704172400
Sat Apr 03 08:19:13 2010  A1: -2113433513605023016052
Sat Apr 03 08:19:13 2010  A2: -61156523169760400
Sat Apr 03 08:19:13 2010  A3:  3252893327075
Sat Apr 03 08:19:13 2010  A4: -282923474
Sat Apr 03 08:19:13 2010  A5:  3060
Sat Apr 03 08:19:13 2010  skew 25011.05, size 4.090357e-010, alpha -5.294781, combined = 1.611910e-009
Sat Apr 03 08:19:13 2010  elapsed time 00:45:30
Sun Apr 04 23:51:11 2010  
Sun Apr 04 23:51:11 2010  
Sun Apr 04 23:51:11 2010  Msieve v. 1.44
Sun Apr 04 23:51:11 2010  random seeds: 7bc0a738 d4f0ebee
Sun Apr 04 23:51:11 2010  factoring 5835729598046643344594576944853108561052208334385324070741681153732787325381841796727815086982722157617747 (106 digits)
Sun Apr 04 23:51:13 2010  searching for 15-digit factors
Sun Apr 04 23:51:14 2010  commencing number field sieve (106-digit input)
Sun Apr 04 23:51:14 2010  R0: -285810003640834066631
Sun Apr 04 23:51:14 2010  R1:  95330612729
Sun Apr 04 23:51:14 2010  A0: -5560560654456680704172400
Sun Apr 04 23:51:14 2010  A1: -2113433513605023016052
Sun Apr 04 23:51:14 2010  A2: -61156523169760400
Sun Apr 04 23:51:14 2010  A3:  3252893327075
Sun Apr 04 23:51:14 2010  A4: -282923474
Sun Apr 04 23:51:14 2010  A5:  3060
Sun Apr 04 23:51:14 2010  skew 25011.05, size 4.090357e-010, alpha -5.294781, combined = 1.611910e-009
Sun Apr 04 23:51:14 2010  elapsed time 00:00:03

Is my next course to manually run msieve, possibly with the -nc option, or physically edit the test.dat file to cut down the number of relations or perhaps another route?

If this relations issue is the trouble, then any future runs of the py script will also fail, since it will still want >1e7 relations.

Thank you for all the help.

Take Care,
Ed (copying the directory prior to further play...)

[10metreh]

Originally Posted by EdH

On to further info: (The ggnfs.log file only had a series of "LatSieveTime: 8078" type entries.

Sorry, I meant test.log. That should have the msieve log from the filtering run.

Test.job.T0:

I reckon something is wrong with the script's minrels-estimating procedure.

msieve.log

That is just the log from the polynomial selection followed by the attempted polynomial selection when you resumed the run via aliqueit (which went straight through because there was already a poly).

Is my next course to manually run msieve, possibly with the -nc option, or physically edit the test.dat file to cut down the number of relations or perhaps another route?

Run (manually, replacing <n> with the real c106):

msieve -s ggnfs_<n>\test.dat -l ggnfs_<n>\test.log -i ggnfs_<n>\test.ini -nf ggnfs_<n>\test.fb -v -nc1 0,5000000

This will run filtering with only the first 5M relations, and will ouptut the log to the screen (minus the timestamps).
If this fails, then gradually lower the 5000000 by 100000 at a time until it works.
Then run the same command line, replacing -nc1 with -nc2 and removing the 0,5000000. This will run the linear algebra. Once that is done, replace -nc2 with -nc3 for the square root, producing the factors. Restart aliqueit with

aliqueit <the_sequence> -f <one_of_the_factors>

to input the factors.