Python Driver for GGNFS and MSIEVE

[xilman]

Originally Posted by 10metreh

I know that schickel uses an ECMNET server to run ECM on his sequences, which are all quite large. It is fine for running aliquot sequences. However, the aliquot sequence project is currently structured completely differently: people reserve whole sequences, not just composites for NFS, with a bit of ECM added in, like the Homogeneous Cunninghams project. I'm not saying that ECMNET wouldn't work for aliquot sequences; just that it would eliminate the element of fun that constitutes a downdriver run.

So, just reserve a few dozen whole sequences, or a few thousand, depending on your resources.

At base, I'm suggesting that you use parallel, not serial, processing.


Paul

[xilman]

I've been running the Python script on a 32-bit XP installation --- the "machine" is actually a VirtualBox VM.

Windoze Update re-booted my mchine when the LA was about 5% through. After it came up again, the LA failed to restart. Here is the log file:

Mon Feb 01 15:37:02 2010 -> Running setup ...
Mon Feb 01 15:37:02 2010 -> Estimated minimum relations needed: 1.14062e+07
Mon Feb 01 15:37:02 2010 -> cleaning up before a restart
Mon Feb 01 15:37:02 2010 -> Running lattice siever ...
Mon Feb 01 15:37:02 2010 -> entering sieving loop
Mon Feb 01 15:37:02 2010 -> Running matrix solving step ...
Mon Feb 01 15:37:02 2010 -> msieve -s foo.dat -l foo.log -i foo.ini -v -nf foo.fb -t 1 -nc2
Mon Feb 01 15:37:02 2010  
Mon Feb 01 15:37:02 2010  
Mon Feb 01 15:37:02 2010  Msieve v. 1.43
Mon Feb 01 15:37:02 2010  random seeds: bbf8a680 625a66ff
Mon Feb 01 15:37:02 2010  factoring 667042581886253540421282123249269716618711643280851695815513316831606593912859018655030322042314014706215437995357683550559598451619213049278694422649 (150 digits)
Mon Feb 01 15:37:04 2010  searching for 15-digit factors
Mon Feb 01 15:37:06 2010  commencing number field sieve (150-digit input)
Mon Feb 01 15:37:06 2010  R0: -6366805760909027985741435139224001
Mon Feb 01 15:37:06 2010  R1:  1
Mon Feb 01 15:37:06 2010  A0:  1
Mon Feb 01 15:37:06 2010  A1:  0
Mon Feb 01 15:37:06 2010  A2:  0
Mon Feb 01 15:37:06 2010  A3:  0
Mon Feb 01 15:37:06 2010  A4:  0
Mon Feb 01 15:37:06 2010  A5:  200
Mon Feb 01 15:37:06 2010  skew 1.00, size 8.558778e-012, alpha 0.373617, combined = 2.874687e-010
Mon Feb 01 15:37:06 2010  
Mon Feb 01 15:37:06 2010  commencing linear algebra
Mon Feb 01 15:37:07 2010  read 861427 cycles
Mon Feb 01 15:37:10 2010  cycles contain 2851091 unique relations
Mon Feb 01 15:37:10 2010  read 0 relations
Mon Feb 01 15:37:10 2010  error: cannot locate relation 11556978

The foo.dat file had been truncated to a single line and the script did not use previously generated files to try to recover gracefully.

Paul

[Brian Gladman]

Hi Paul

From the log it seems that the termination was such that the resume file (foo.resume) was not written.

If a *.resume file is not found the script is supposed to delete all the intermediate files and start again. But it seems that some files might have been left around and caused a bad restart. Do you know what files were around after the failure?

Brian

[10metreh]

Originally Posted by xilman

<snip>

I found this same bug in factMsieve.pl a while ago, and it still hasn't been found, let alone fixed. It seems it's been carried over to the new script.

[xilman]

Originally Posted by Brian Gladman

Hi Paul

From the log it seems that the termination was such that the resume file (foo.resume) was not written.

If a *.resume file is not found the script is supposed to delete all the intermediate files and start again. But it seems that some files might have been left around and caused a bad restart. Do you know what files were around after the failure?

Brian

Sorry, I don't remember and the scene-of-crime has been so disturbed by a subsequent (and successful) restart that it's now impossible to say.


Paul

[tmorrow]

A question about factmsieve.py version 0.44:

# grep fiile factmsieve.py
    print('rlim: {0:d}'.format(lats_p['rlim']), fiile = out_f)
    print('alim: {0:d}'.format(lats_p['alim']), fiile = out_f)
    print('lbpr: {0:s}'.format(LBPR), fiile = out_f)
    print('lbpa: {0:s}'.format(LBPA), fiile = out_f)

Should the "fiile" references be "file" instead?

[Brian Gladman]

Originally Posted by tmorrow

A question about factmsieve.py version 0.44:

# grep fiile factmsieve.py
    print('rlim: {0:d}'.format(lats_p['rlim']), fiile = out_f)
    print('alim: {0:d}'.format(lats_p['alim']), fiile = out_f)
    print('lbpr: {0:s}'.format(LBPR), fiile = out_f)
    print('lbpa: {0:s}'.format(LBPA), fiile = out_f)

Should the "fiile" references be "file" instead?

Yes - thanks for finding this. Corrected version attached.

Brian

[smh]

I haven't checked this latest version, but 0.44 does way to much sieving for small composites. Can you lower the minimum amount needed?

[Brian Gladman]

I am wondering if we now have enough information to set minrels sensibly?

From what has been said, it seems that we need something like minrels verses decimal digits for each lpbr/a value.

Is anyone willing to suggest some values?

Brian

[tmorrow]

Originally Posted by Brian Gladman

From what has been said, it seems that we need something like minrels verses decimal digits for each lpbr/a value.

Is anyone willing to suggest some values?

I think minrels versus decimal digits for each lpbr/a value is a good approach.

I'm trying to determine what values are selected by the program currently over the range of decimal digits for GNFS - lpbr/a and minrel. I need some more time for this as I haven't had a lot of spare time.

There are a few different functions that load the parameters, so far I've looked at load_default_parameters. This function first tries to load the parameter from def-par.txt satisying abs(digits-file_digits) at a minimum. These values are then possibly overwritten magic formulas in facmsieve if fact_p['digs']=digits/0.72, which occurs for c116+ numbers.

Based on what I've seen in the script, decimal digits determines lpbr/a (from def-par.txt lookup) and minrel (from formula in script). The vanilla/default def-par.txt has so many holes (mine only has 15 gnfs rows for C70-C126) that those choices for lpbr/a are not very good and result in poor minrel estimates that match the chosen lpbr/a.

There's more than enough empirical data around to collate a practical set of parameters that should work in most circumstances and on most systems. Perhaps we can start with seeing if we can get a consensus on the lpbr/a dimension vs decimal digits. We can start with the values currently selected by the program (and are suboptimal in places) and going from there.

[tmorrow]

Bug in miller_rabin.

Traceback (most recent call last):
File "../factmsievetim.py", line 1963, in <module>
if probable_prime_p(fact_p['n'], 10):
File "../factmsievetim.py", line 383, in probable_prime_p
return 1 if miller_rabin(n, r) else 0
NameError: global name 'miller_rabin' is not defined
siever terminated

Should the miller_rabin call be miller_rabin_1?