Python Driver for GGNFS and MSIEVE

[tgrdy]

factmsieve.py v69 ERROR:

-> ________________________________________________________________
-> | Running factmsieve.py, a Python driver for MSIEVE with GGNFS |
-> | sieving support. It is Copyright, 2010, Brian Gladman and is |
-> | a conversion of factmsieve.pl that is Copyright, 2004, Chris |
-> | Monico. Version 0.69 (Python 2.6 or later) 24th April 2010. |
-> |______________________________________________________________|

Traceback (most recent call last):
File "D:\rsatool\ggnfs\factmsieve.py", line 1953,
in <module>
sumname = os.path.join(dname, tmp + '-' + fname + '.txt')
NameError: name 'tmp' is not defined

Originally Posted by Brian Gladman

Thats good as all I need to do is exit if no *.fb exists. It would be nice to write a summary file but this might be a bit tricky as I would have to skip a lot of normal processing to get to the end of the script.

Brian

Edit: here is an attachment that attempts to remove this problem.

[Brian Gladman]

Sorry about the bug - hopefully fixed in the attached version.

Brian

[firejuggler]

sorry, doesn't seem to fix it

-> ________________________________________________________________
-> | Running factmsieve.py, a Python driver for MSIEVE with GGNFS |
-> | sieving support. It is Copyright, 2010, Brian Gladman and is |
-> | a conversion of factmsieve.pl that is Copyright, 2004, Chris |
-> | Monico.    Version 0.70 (Python 2.6 or later) 1st June 2010. |
-> |______________________________________________________________|
Traceback (most recent call last):
  File "H:\ggnfs\ex\factmsieve.py", line 1961, in <module>
    sumname = os.path.join(dname, tmp + '-' + fname + '.txt')
NameError: name 'tmp' is not defined

switched back to 0.68 who seem to work flawlessly

[Brian Gladman]

Another try.

Brian

[firejuggler]

-> ________________________________________________________________
-> | Running factmsieve.py, a Python driver for MSIEVE with GGNFS |
-> | sieving support. It is Copyright, 2010, Brian Gladman and is |
-> | a conversion of factmsieve.pl that is Copyright, 2004, Chris |
-> | Monico.    Version 0.71 (Python 2.6 or later) 1st June 2010. |
-> |______________________________________________________________|
-> This is client 1 of 1
-> Using 2 threads
-> Working with NAME = ex
-> Selected default factorization parameters for 100 digit level.
-> Selected lattice siever: gnfs-lasieve4I12e
-> Creating param file to detect parameter changes...
-> Running setup ...
-> Estimated minimum relations needed: 4.004e+06
-> resuming a block for q from 1000000 to 1100000
-> Running lattice siever ...
-> entering sieving loop
-> making sieve job for q = 1000000 in 1000000 .. 1050000 as file ex.job.T0
-> making sieve job for q = 1050000 in 1050000 .. 1100000 as file ex.job.T1
-> Lattice sieving algebraic q from 1000000 to 1100000.
-> gnfs-lasieve4I12e -k -o spairs.out.T0 -v -n0 -a ex.job.T0
-> gnfs-lasieve4I12e -k -o spairs.out.T1 -v -n1 -a ex.job.T1
FBsize 78831+0 (deg 4), 135071+0 (deg 1)
FBsize 78831+0 (deg 4), 135071+0 (deg 1)
siever terminated
appending spairs.out.T0 to spairs.out
appending spairs.out.T1 to spairs.out
....

seem to work, thanks

[firejuggler]

and... another problem ....

compressing spairs.out to spairs.save.gz
Found 2051616 relations, 117.4% of the estimated minimum (1748000).
-> msieve145 -s ex\ex.dat -l ex\ex.log -i ex\ex.ini -nf ex\ex.fb -t 2 -nc1
-> Running matrix solving step ...
-> msieve145 -s ex\ex.dat -l ex\ex.log -i ex\ex.ini -nf ex\ex.fb -t 2 -nc2
linear algebra completed 119365 of 119593 dimensions (99.8%, ETA 0h 0m)
Traceback (most recent call last):
  File "H:\ggnfs\ex\factmsieve.py", line 2072, in <module>
    if not os.path.exists(sumname):
NameError: name 'sumname' is not defined

[Brian Gladman]

Yet one more try at getting this right.

Brian

[firejuggler]

all right...i managed to finish a factorisation. so far, no error

[tgrdy]

I have run the GGNFS factmsieve.py.72 to factor a C153 number.

How weird was that? It was making duplicated sieving jobs by automatic,
after I changed the NUM_CPUS from 1 to 6.
# Set the number of CPU's and threads
NUM_CPUS = 1
to
NUM_CPUS = 6

I have not notice the cruel fact , until after about two weeks.
Now, the 66.2% relations (35688330 relations) was just about
a one to six duplicated copy.

Now I have modified the C153.job.Tn (n=0,1,2,3,4,5) by hand.
to:
17500000 .. 17600000;
17600000 .. 17700000;
17700000 .. 17800000;
...
18000000 .. 18100000

Should I use remdups (a relation-file filter) to ruduce the duplicated relations?
Thanks.

duplicated sieving jobs like that:

Found 35688330 relations, 66.2% of the estimated minimum (53914715).
-> making sieve job for q = 17500000 in 17500000 .. 17585841 as file c153.job.T0
-> making sieve job for q = 17500000 in 17500000 .. 17600000 as file c153.job.T1
-> making sieve job for q = 17500000 in 17500000 .. 17571712 as file c153.job.T2
-> making sieve job for q = 17500000 in 17500000 .. 17571713 as file c153.job.T3
-> making sieve job for q = 17500000 in 17500000 .. 17585849 as file c153.job.T4
-> making sieve job for q = 17500000 in 17500000 .. 17585859 as file c153.job.T5

[Brian Gladman]

This could happen if you attempted to change the number of cpu's without deleting the output of an earlier run with fewer cpu's set. I do attempt to cope with this but it is pretty primitive right now and I don't recommend relying on it.

Brian

[VolMike]

When filtering stage of factmsieve.py msieve returns error: &quot;error: number of relsets too large&quot;

Part of console log:

Found 4230021 relations, 106.3% of the estimated minimum (3981071).
-> msieve -s ..\..\ggnfs\job\snfs_x.dat -l ..\..\ggnfs\job\snfs_x.log -i ..\..\ggnfs\job\snfs_x.ini -nf ..\..\ggnfs\job\snfs_x.fb -t 8 -nc1
error: number of relsets too large
Return value 0. Terminating...
siever terminated

Part of snfs_x log:

Wed Jun 30 07:46:46 2010 Found 4230021 relations, 106.3% of the estimated minimum (3981071).
Wed Jun 30 07:46:46 2010  
Wed Jun 30 07:46:46 2010  
Wed Jun 30 07:46:46 2010  Msieve v. 1.44
Wed Jun 30 07:46:46 2010  random seeds: de8b6ad0 fcbf1d0d
Wed Jun 30 07:46:46 2010  factoring 36529217761436240792289344397796605728167575469653294782671061252118093829162143567715263190731656613443339986228258883868277 (125 digits)
Wed Jun 30 07:46:47 2010  searching for 15-digit factors
Wed Jun 30 07:46:47 2010  commencing number field sieve (125-digit input)
Wed Jun 30 07:46:47 2010  R0: -1237940039285380274899124224
Wed Jun 30 07:46:47 2010  R1:  1
Wed Jun 30 07:46:47 2010  A0:  1
Wed Jun 30 07:46:47 2010  A1:  0
Wed Jun 30 07:46:47 2010  A2:  0
Wed Jun 30 07:46:47 2010  A3:  0
Wed Jun 30 07:46:47 2010  A4:  0
Wed Jun 30 07:46:47 2010  A5:  3616
Wed Jun 30 07:46:47 2010  skew 0.19, size 1.022282e-009, alpha -0.258613, combined = 4.583911e-009
Wed Jun 30 07:46:47 2010  
Wed Jun 30 07:46:47 2010  commencing relation filtering
Wed Jun 30 07:46:47 2010  estimated available RAM is 3063.0 MB
Wed Jun 30 07:46:47 2010  commencing duplicate removal, pass 1
Wed Jun 30 07:47:10 2010  found 768020 hash collisions in 4230020 relations
Wed Jun 30 07:47:14 2010  added 230 free relations
Wed Jun 30 07:47:14 2010  commencing duplicate removal, pass 2
Wed Jun 30 07:47:16 2010  found 785892 duplicates and 3444358 unique relations
Wed Jun 30 07:47:16 2010  memory use: 20.6 MB
Wed Jun 30 07:47:16 2010  reading ideals above 100000
Wed Jun 30 07:47:16 2010  commencing singleton removal, initial pass
Wed Jun 30 07:47:39 2010  memory use: 86.1 MB
Wed Jun 30 07:47:39 2010  reading all ideals from disk
Wed Jun 30 07:47:39 2010  memory use: 101.5 MB
Wed Jun 30 07:47:39 2010  keeping 2748721 ideals with weight <= 200, target excess is 22302
Wed Jun 30 07:47:39 2010  commencing in-memory singleton removal
Wed Jun 30 07:47:39 2010  begin with 3444358 relations and 2748721 unique ideals
Wed Jun 30 07:47:40 2010  reduce to 2157173 relations and 1205316 ideals in 8 passes
Wed Jun 30 07:47:40 2010  max relations containing the same ideal: 153
Wed Jun 30 07:47:40 2010  removing 785186 relations and 439248 ideals in 345938 cliques
Wed Jun 30 07:47:41 2010  commencing in-memory singleton removal
Wed Jun 30 07:47:41 2010  begin with 1371987 relations and 1205316 unique ideals
Wed Jun 30 07:47:41 2010  reduce to 1346483 relations and 738285 ideals in 4 passes
Wed Jun 30 07:47:41 2010  max relations containing the same ideal: 102
Wed Jun 30 07:47:41 2010  removing 185216 relations and 96331 ideals in 88885 cliques
Wed Jun 30 07:47:41 2010  commencing in-memory singleton removal
Wed Jun 30 07:47:41 2010  begin with 1161267 relations and 738285 unique ideals
Wed Jun 30 07:47:41 2010  reduce to 1158627 relations and 639256 ideals in 3 passes
Wed Jun 30 07:47:41 2010  max relations containing the same ideal: 93
Wed Jun 30 07:47:41 2010  removing 64335 relations and 32668 ideals in 31667 cliques
Wed Jun 30 07:47:41 2010  commencing in-memory singleton removal
Wed Jun 30 07:47:41 2010  begin with 1094292 relations and 639256 unique ideals
Wed Jun 30 07:47:41 2010  reduce to 1093829 relations and 606120 ideals in 3 passes
Wed Jun 30 07:47:41 2010  max relations containing the same ideal: 88
Wed Jun 30 07:47:41 2010  removing 26437 relations and 13309 ideals in 13128 cliques
Wed Jun 30 07:47:41 2010  commencing in-memory singleton removal
Wed Jun 30 07:47:41 2010  begin with 1067392 relations and 606120 unique ideals
Wed Jun 30 07:47:42 2010  reduce to 1067303 relations and 592722 ideals in 3 passes
Wed Jun 30 07:47:42 2010  max relations containing the same ideal: 87
Wed Jun 30 07:47:42 2010  relations with 0 large ideals: 513
Wed Jun 30 07:47:42 2010  relations with 1 large ideals: 972
Wed Jun 30 07:47:42 2010  relations with 2 large ideals: 11583
Wed Jun 30 07:47:42 2010  relations with 3 large ideals: 61150
Wed Jun 30 07:47:42 2010  relations with 4 large ideals: 173280
Wed Jun 30 07:47:42 2010  relations with 5 large ideals: 286117
Wed Jun 30 07:47:42 2010  relations with 6 large ideals: 292761
Wed Jun 30 07:47:42 2010  relations with 7+ large ideals: 240927
Wed Jun 30 07:47:42 2010  commencing 2-way merge
Wed Jun 30 07:47:42 2010  reduce to 1061582 relation sets and 587001 unique ideals
Wed Jun 30 07:47:42 2010  commencing full merge

Is it possible to proceed factorisation anyway without restarting the sieving stage?