Python Driver for GGNFS and MSIEVE

[mdettweiler]

Originally Posted by dekcarki

from http://gladman.plushost.co.uk/oldsit...factoring.php:

Ah, whoops, I didn't see that. It turns out Cygwin doesn't have anything newer than 2.5.2 in its package manager, so I downloaded a separate "portable" version of 2.6 so that it wouldn't cause problems with the existing 2.5 setup. That's worked quite well now.

One other admittedly minor problem I've encountered is that when I set DOCLASSICAL to True, as I usually have it in factMsieve.pl, the script produces an error when it tries to run the line sieve. Here's the console output I got when the script proceeded out of poly selection and into line sieving:

polynomial selection complete
R0: -10611311574274716217360
R1:  860316133067
A0:  20824087731076860280730091
A1:  490691822134716217276644
A2: -3654358774808750777
A3: -229228659485384
A4:  285805258
A5:  3120
skew 100317.19, size 7.518602e-011, alpha -6.321125, combined = 6.557688e-010
elapsed time 02:22:36
-> Selected default factorization parameters for 112 digit level.
-> Selected lattice siever: gnfs-lasieve4I13e
-> Creating param file to detect parameter changes...
Traceback (most recent call last):
  File "./factmsieve.py", line 1926, in <module>
    run_classical_sieve(clas_p['cl_a'], 1, clas_p['cl_b'])
  File "./factmsieve.py", line 1507, in run_classical_sieve
    print('SLINE {0:s}'.format(a), file = out_f)
ValueError: Unknown conversion type s

It's worked fine, though, with DOCLASSICAL set to False. As I said, a minor problem since the line sieving isn't of much particular value, but it might be worth looking into nonetheless.

Also, one quick question: does factmsieve.py support minrels.txt like factMsieve.pl does? I tried searching for references to "minrels.txt" in the source but couldn't find any.

[Brian Gladman]

Originally Posted by mdettweiler

Ah, whoops, I didn't see that. It turns out Cygwin doesn't have anything newer than 2.5.2 in its package manager, so I downloaded a separate "portable" version of 2.6 so that it wouldn't cause problems with the existing 2.5 setup. That's worked quite well now.

One other admittedly minor problem I've encountered is that when I set DOCLASSICAL to True, as I usually have it in factMsieve.pl, the script produces an error when it tries to run the line sieve. Here's the console output I got when the script proceeded out of poly selection and into line sieving:

polynomial selection complete
R0: -10611311574274716217360
R1:  860316133067
A0:  20824087731076860280730091
A1:  490691822134716217276644
A2: -3654358774808750777
A3: -229228659485384
A4:  285805258
A5:  3120
skew 100317.19, size 7.518602e-011, alpha -6.321125, combined = 6.557688e-010
elapsed time 02:22:36
-> Selected default factorization parameters for 112 digit level.
-> Selected lattice siever: gnfs-lasieve4I13e
-> Creating param file to detect parameter changes...
Traceback (most recent call last):
  File "./factmsieve.py", line 1926, in <module>
    run_classical_sieve(clas_p['cl_a'], 1, clas_p['cl_b'])
  File "./factmsieve.py", line 1507, in run_classical_sieve
    print('SLINE {0:s}'.format(a), file = out_f)
ValueError: Unknown conversion type s

It's worked fine, though, with DOCLASSICAL set to False. As I said, a minor problem since the line sieving isn't of much particular value, but it might be worth looking into nonetheless.

Also, one quick question: does factmsieve.py support minrels.txt like factMsieve.pl does? I tried searching for references to "minrels.txt" in the source but couldn't find any.

Thanks for the feedback.

The issue you found in the classical siever should be solved if you replace {0:s} with {0:d} in the following two lines around line 1510:

print('SLINE {0:s}'.format(a), file = out_f)
print('-> Line file scanned: resuming classical sieve from b = {0:s}.'
.format(max_b))

But the original Perl says that classical sieving is broken but doesn't say what is wrong. If anyone knows what needs to be done I will be happy to have a look at it.

Search for 'MINRELS.txt' :-)

There are, I am afraid, a few glitches like this caused by various Perl features that I still need to get rid of.

Brian

[dekcarki]

'SAVEPAIRS = True'

    with open(fr, 'rb') as in_file:
      out_file = gzip.open(to, 'ab')   # 'a' for appending
      out_file.writelines(in_file)
      out_file.close()

works! :)

maybe adding around line 1650

write_string_to_log('Found {0:d} relations, need at least {1:d} to proceed.'
                                .format(curr_rels, lats_p['minrels']))

next: USE_MSIEVE_POLY = True (msieve with CUDA)

1. only works if the ptx-files are in the same directory where you start the script from.

2. in *.poly is only the converted poly without 'a/rlim' etc.

[Jeff Gilchrist]

Originally Posted by dekcarki

next: USE_MSIEVE_POLY = True (msieve with CUDA)

2. in *.poly is only the converted poly without 'a/rlim' etc.

I'm not really sure what you mean for 2. Is this happening only when using the msieve with CUDA or any msieve version? Can you provide an example of what the .poly contains and what you think it should contain?

Jeff.

[dekcarki]

Originally Posted by Jeff Gilchrist

I'm not really sure what you mean for 2. Is this happening only when using the msieve with CUDA or any msieve version? Can you provide an example of what the .poly contains and what you think it should contain?

USE_KLEINJUNG_FRANKE_PS = True
USE_MSIEVE_POLY = False

name: test2
n:  70601405003294056911641692576788075231594387150301442003185809408166914956302721
m:  2642458626871365679
deg: 4
c4: 1448040
c3: 2318105623
c2: -1558036700474027
c1: -6315360413557101
c0: 753341220483492715470
skew: 1379.250
type: gnfs
# adj. I(F,S) = 45.622
# E(F1,F2) = 3.929089e-03
# GGNFS version 0.77.1-20060722-k8 polyselect.
# Options were: 
# lcd=1, enumLCD=24, maxS1=52.00000000, seed=1263495658.
# maxskew=1500.0
# These parameters should be manually set:
rlim: 350000
alim: 500000
lpbr: 24
lpba: 24
mfbr: 37
mfba: 37
rlambda: 1.7
alambda: 1.7
qintsize: 10000
type: gnfs

USE_KLEINJUNG_FRANKE_PS = False
USE_MSIEVE_POLY = True (msieve-trunk, with and without CUDA)

n: 70601405003294056911641692576788075231594387150301442003185809408166914956302721
Y0: -14072536434508903170
Y1: 9335769749
c0: -189065550305088831923479
c1: -9433303452450188290
c2: 20539529174531
c3: 320578470
c4: 1800
skew: 150135.59 
type: gnfs

btw: with polyselect:

Traceback (most recent call last):
  File "../factmsieve.py", line 1900, in <module>
    pols_p['goodscore'] = run_poly_select(fact_p, pols_p, lats_p)
  File "../factmsieve.py", line 848, in run_poly_select
    return goodscore
NameError: global name 'goodscore' is not defined

pol5 not tested.

[mdettweiler]

I've been running factmsieve.py on a C114 GNFS since yesterday; it was interrupted early this morning, and I resumed it a few hours later. However, to my shock, it apparently restarted the job from the beginning (sans poly selection, of course, which it thankfully didn't re-do)--and wiped out the .dat file of previous relations to boot! I've attached the log of what happened; hopefully it will prove helpful. Note that according to the console output, the total # of relations present in test.dat after the latest q=1750000-1850000 run was 827815, as opposed to the 4989207 present prior to the restart.

Note that this doesn't seem to happen after a "nice" interrupt with Ctrl-C; in this case, the system was shut down, and in my experience Windows doesn't tend to give programs much warning before doing that.

[Brian Gladman]

Originally Posted by mdettweiler

I've been running factmsieve.py on a C114 GNFS since yesterday; it was interrupted early this morning, and I resumed it a few hours later. However, to my shock, it apparently restarted the job from the beginning (sans poly selection, of course, which it thankfully didn't re-do)--and wiped out the .dat file of previous relations to boot! I've attached the log of what happened; hopefully it will prove helpful. Note that according to the console output, the total # of relations present in test.dat after the latest q=1750000-1850000 run was 827815, as opposed to the 4989207 present prior to the restart.

Note that this doesn't seem to happen after a "nice" interrupt with Ctrl-C; in this case, the system was shut down, and in my experience Windows doesn't tend to give programs much warning before doing that.

I am sorry that you lost data on this run. It is designed to do an orderly shutdown if terminated by a Ctrl-C but it has no defence against a forced termination on shutdown. I'll look at what can be done but I am not even sure if there is a solution.

Brian

[mdettweiler]

Originally Posted by Brian Gladman

I am sorry that you lost data on this run. It is designed to do an orderly shutdown if terminated by a Ctrl-C but it has no defence against a forced termination on shutdown. I'll look at what can be done but I am not even sure if there is a solution.

Brian

Yeah, I figured there wasn't much it could do against a forced termination. But, as I recall, the factMsieve.pl script would at least just pick up from the beginning of the current sieve run if terminated forcefully; factmsieve.py, however, apparently starts afresh and deletes the entire .dat file.

[Brian Gladman]

Originally Posted by mdettweiler

Yeah, I figured there wasn't much it could do against a forced termination. But, as I recall, the factMsieve.pl script would at least just pick up from the beginning of the current sieve run if terminated forcefully; factmsieve.py, however, apparently starts afresh and deletes the entire .dat file.

In fact I think factmsieve will now restart the last cycle - earlier today I added a modification to record the start point of each sieve cycle as a 'fall back' resume position.

I find I can also add an exit handler that might trap forced exits. I'll see if this works.

Brian

[10metreh]

Originally Posted by mdettweiler

Yeah, I figured there wasn't much it could do against a forced termination. But, as I recall, the factMsieve.pl script would at least just pick up from the beginning of the current sieve run if terminated forcefully; factmsieve.py, however, apparently starts afresh and deletes the entire .dat file.

If factMsieve.pl is forcefully terminated, then you can get it to resume from the exact point it left off: rename "spairs.out" to "spairs.add" and change the q0 value in the .job file to the number in ".last_spq0".

[mdettweiler]

Originally Posted by Brian Gladman

In fact I think factmsieve will now restart the last cycle - earlier today I added a modification to record the start point of each sieve cycle as a 'fall back' resume position.

I find I can also add an exit handler that might trap forced exits. I'll see if this works.

Brian

Okay, cool. I've downloaded the updated version; I'll give it a whirl and have my computer do a similar automatic shutdown tonight in what will probably be the middle of the job. That should give a pretty conclusive answer as to whether the modifications work. And this time, I'll back up the sieve folder before resuming, so if there's a problem again, I can do as 10metreh suggests and force it to resume properly manually.

BTW, one other unrelated thing I noticed with the new version. Since this is the first time I've tried a version with the classical sieve routine fixed, it tried to do it after noticing that there was no record of having done it before. But, it seems that it had a problem running msieve:

-> ________________________________________________________________
-> | Running factmsieve.py, a Python driver for MSIEVE with GGNFS |
-> | sieving support. It is Copyright, 2010, Brian Gladman and is |
-> | a conversion of factmsieve.pl that is Copyright, 2004, Chris |
-> | Monico.          This is version 0.2, dated 14 January 2010. |
-> |______________________________________________________________|
-> This is client 1 of 1
-> Using 1 threads
-> Working with NAME = test
-> Selected default factorization parameters for 112 digit level.
-> Selected lattice siever: gnfs-lasieve4I13e
-> No parameter change detected. Resuming.
-> Line file scanned: resuming classical sieve from b = 1.
-> msieve -s test.dat -l test.log -i test.ini -v -nf test.fb -t 1 -ns 1,300
cannot open input file 'test.ini'
-> Q0 = 2050000, QSTEP = 100000.
-> making sieve job for q from 2065523 to 2150000 as file test.job.T0.
-> Lattice sieving algebraic q-values from q = 2050000 to 2150000.
-> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
FBsize 151905+0 (deg 5), 250149+0 (deg 1)
total yield: 27608, q=2068499 (0.00950 sec/rel)

The strange thing is, test.ini was verifiably present and had its number in there exactly as it should. So I'm not entirely sure what went wrong. In this case it's no big deal since it went on to resume the GNFS without a problem, but nonetheless it's worth looking into.