Python Driver for GGNFS and MSIEVE - Page 49

2011-01-08, 21:22

Hi Jeff,

I did a few factorization with your script and I found it very useful. However, I observed that sieving with multi client + multi threads configuration gives about ~45-50% of duplicate relations. In my case this case two issues

1. Script finish sieving for all clients when got minimal number of relations. but because of huge number of duplicates msieve will failed at nc2 test with well know error
"matrix needs more columns than rows; try adding 2-3% more relations"

And here we have 2nd issue. After "successful" sieving script doesn't generate .resume and you can't start automatically sieving from last position.
I coded simple perl script which generate resume file based on log file.

So it might be possible to add two simple feature
1. duplicates cleanup .. for example based on remdups and excute it just after gzip the dat file. For example if you reached 90% of required relations.

Code:

if DUPCHECK:
    exec_remdups

2. add lock file for sieving clients if executed with multiclient configuration but still keep .resume file in case of enough relation number. Dangerous is clean up part if executed from "master" node without .resume :x

What do you think about that?

Cheers

Jeff Gilchrist · 2011-01-11, 15:09

Sorry Carpi, I didn't mean for you to think that I wrote the script, it was actually Brian Gladman who wrote it. We will have to wait and see what he thinks.

If you are getting something like 50% duplicate relations are the ranges not being manged properly and the different threads/clients are sieving the same range?

Brian Gladman · 2011-01-13, 11:46

I would be happy to add things to the script when I have a good idea of what is needed.

I am not familiar with remdups so you would need to explain what this is, how it would be used (i.e. what parameter values it would be called with) and how it would be available on both Linux and Windows.

I don't understand what you want 'locked' so you would need to explain this.

Right now I delete the resume file and use its non existence as a signal that sieving does not need to be resumed. The easiest way of solving your problem with this would be to rename the resume file rather than deleting it. You could then simply restore its original name if you wanted to resume sieving. This is a very easy change to make.

Brian

firejuggler · 2011-01-20, 16:25

Hi.
I just bought and installed a GTX 460, wich is supposed to be cuda-enabled.
When I put 'True' for use_cuda, in factmsieve.py

Quote:

# Set binary directory paths
GGNFS_PATH = '../'
MSIEVE_PATH = '../'

[snip]

USE_CUDA = True
GPU_NUM = 0
MSIEVE_POLY_TIME_LIMIT = 0

i get the following error

Quote:

-> -> ________________________________________________________________
-> | Running factmsieve.py, a Python driver for MSIEVE with GGNFS |
-> | sieving support. It is Copyright, 2010, Brian Gladman and is |
-> | a conversion of factmsieve.pl that is Copyright, 2004, Chris |
-> | Monico. Version 0.76 (Python 2.6 or later) 10th Nov 2010. |
-> |______________________________________________________________|
-> This is client 1 of 1
-> Running on 2 Cores with 1 hyper-thread per Core
-> Working with NAME = test
-> Error: Polynomial file test.poly does not exist!
-> Found n = 1009465036999274771364171378707819237562672261014762539098800356744
11340204945135025119946813013045736369517249633929818919926914252843663.
-> Running polynomial selection ...
-> msieve148 -s ggnfs_1009465036999274771364171378707819237562672261014762539098
80035674411340204945135025119946813013045736369517249633929818919926914252843663
\test.dat -l ggnfs_1009465036999274771364171378707819237562672261014762539098800
35674411340204945135025119946813013045736369517249633929818919926914252843663\te
st.log -i ggnfs_1009465036999274771364171378707819237562672261014762539098800356
74411340204945135025119946813013045736369517249633929818919926914252843663\test.
ini -nf ggnfs_100946503699927477136417137870781923756267226101476253909880035674
411340204945135025119946813013045736369517249633929818919926914252843663\test.fb
-g 0 -v -np

Msieve v. 1.48

usage: msieve148.exe [options] [one_number]

numbers starting with '0' are treated as octal,
numbers starting with '0x' are treated as hexadecimal

options:

[snip]

Msieve Error: return value -1. Is CUDA enabled? Terminating...
WARNING: gnfs failed to find a factor. This really shouldn't happen.
I'll just run ecm till the end of time or a factor turns up...
Let's hope you don't run out of disk space before either of those.
c138: running 0 auto-increasing ecm curves...

I also have the SDK installed.
What should I do to be able to use my newly acquired card?
Would I put 'False' instead, it work with no problems.

Karl M Johnson · 2011-01-20, 16:51

I think the issue is with the drivers.
If you're on < 26x.xx, try latest WHQL 26x.xx.
If you're on > 26x.xx, and it doesnt work, try "golden" 258.96.
Did I guess it right?

firejuggler · 2011-01-20, 18:08

I forgot to tell , I have Windows XP. and mfaktc work and detect my GTX 460.

firejuggler · 2011-01-20, 20:34

ok, since

Quote:

>msieve -s ggnfs_1009\test.dat -l ggnfs_1009\test.log -i ggnfs_1009\test.ini -nf ggnfs_1009\test.fb -d 2000 -v -np

initialise the poly selection,, it is the -g switch that make it fail. is there a specific version of msieve that support the switch?

edit : nevermind, no wonder it didn't work. I needed the msieve gpu version. It does work now.

Quote:

searching for 15-digit factors
commencing number field sieve (138-digit input)
commencing number field sieve polynomial selection
time limit set to 45.18 hours
searching leading coefficients from 1 to 236093
using GPU 0 (GeForce GTX 460)
deadline: 200 seconds per coefficient
coeff 60-2400 27573171 30330488 30330489 33363537
------- 27573171-30330488 30330489-33363537
poly 16 p 27590657 q 30331703 coeff 836871613698871
poly 34 p 27589897 q 30458357 coeff 840342932419229

jasonp · 2011-01-20, 21:07

Consider updating your msieve version as well; yours looks to be a few months old and the real v1.48 release has much better poly selection.

firejuggler · 2011-01-20, 21:16

svn 473 seem to not support the -g switch

Quote:

>msieve148svn473 -s ggnfs_1009\test.dat -l ggnfs_1009\test.log -i ggnfs_1009\test.ini -nf ggnfs_1009\test.fb -g 0 -d 2000 -v -np

Quote:

Msieve v. 1.48

usage: msieve148svn473 [options] [one_number]

numbers starting with '0' are treated as octal,
numbers starting with '0x' are treated as hexadecimal

options:
-s <name> save intermediate results to <name>
instead of the default msieve.dat
-l <name> append log information to <name>
instead of the default msieve.log
-i <name> read one or more integers to factor from
<name> (default worktodo.ini) instead of
from the command line
-m manual mode: enter numbers via standard input
-mb <num> hint for number of megabytes of memory for
postprocessing (set automatically if unspec-
ified or zero)
-q quiet: do not generate any log information,
only print any factors found
-d <min> deadline: if still sieving after <min>
minutes, shut down gracefully (default off)
-r <num> stop sieving after finding <num> relations
-p run at idle priority
-v verbose: write log information to screen
as well as to logfile
-t <num> use at most <num> threads

elliptic curve options:
-e perform 'deep' ECM, seek factors > 15 digits

quadratic sieve options:
-c client: only perform sieving

number field sieve options:
-n use the number field sieve (80+ digits only;
performs all NFS tasks in order)
-nf <name> read from / write to NFS factor base file
<name> instead of the default msieve.fb
-np [X,Y] perform only NFS polynomial selection; if
specified, only cover leading coefficients
in the range from X to Y inclusive
-np1 [X,Y] perform stage 1 of NFS polynomial selection; if
specified, only cover leading coefficients
in the range from X to Y inclusive
-np2 perform stage 2 of NFS polynomial selection
-ns [X,Y] perform only NFS sieving; if specified,
handle sieve lines X to Y inclusive
-nc perform only NFS combining (all phases)
-nc1 [X,Y] perform only NFS filtering. Filtering will
track ideals >= X (determined automatically
if 0 or unspecified) and will only use the
first Y relations (or all relations, if 0
or unspecified)
-nc2 perform only NFS linear algebra
-ncr perform only NFS linear algebra, restarting
from a previous checkpoint
-nc3 [X,Y] perform only NFS square root (compute
dependency numbers X through Y, 1<=X<=Y<=64)

jrk · 2011-01-20, 22:09

Quote:

Originally Posted by firejuggler

svn 473 seem to not support the -g switch

Try building with CUDA=1

firejuggler · 2011-01-20, 22:15

i'm rather new at binary compiling, so i have no idea of how to do it. Will try to find a method.

2011-01-08, 21:22	#529
Carpi F14₁₆ Posts	Hi Jeff, I did a few factorization with your script and I found it very useful. However, I observed that sieving with multi client + multi threads configuration gives about ~45-50% of duplicate relations. In my case this case two issues 1. Script finish sieving for all clients when got minimal number of relations. but because of huge number of duplicates msieve will failed at nc2 test with well know error "matrix needs more columns than rows; try adding 2-3% more relations" And here we have 2nd issue. After "successful" sieving script doesn't generate .resume and you can't start automatically sieving from last position. I coded simple perl script which generate resume file based on log file. So it might be possible to add two simple feature 1. duplicates cleanup .. for example based on remdups and excute it just after gzip the dat file. For example if you reached 90% of required relations. Code: if DUPCHECK: exec_remdups 2. add lock file for sieving clients if executed with multiclient configuration but still keep .resume file in case of enough relation number. Dangerous is clean up part if executed from "master" node without .resume :x What do you think about that? Cheers

2011-01-13, 11:46	#531
Brian Gladman May 2008 Worcester, United Kingdom 2²·7·19 Posts	I would be happy to add things to the script when I have a good idea of what is needed. I am not familiar with remdups so you would need to explain what this is, how it would be used (i.e. what parameter values it would be called with) and how it would be available on both Linux and Windows. I don't understand what you want 'locked' so you would need to explain this. Right now I delete the resume file and use its non existence as a signal that sieving does not need to be resumed. The easiest way of solving your problem with this would be to rename the resume file rather than deleting it. You could then simply restore its original name if you wanted to resume sieving. This is a very easy change to make. Brian Last fiddled with by Brian Gladman on 2011-01-13 at 11:47 Reason: remove extraneous text

2011-01-20, 16:51	#533
Karl M Johnson Mar 2010 3×137 Posts	I think the issue is with the drivers. If you're on < 26x.xx, try latest WHQL 26x.xx. If you're on > 26x.xx, and it doesnt work, try "golden" 258.96. Did I guess it right? Last fiddled with by Karl M Johnson on 2011-01-20 at 16:52 Reason: Yes.

2011-01-20, 18:08	#534
firejuggler Apr 2010 Over the rainbow 19·137 Posts	I forgot to tell , I have Windows XP. and mfaktc work and detect my GTX 460. Last fiddled with by firejuggler on 2011-01-20 at 18:08

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Msieve & ggnfs on MacOS	xilman	Msieve	8	2017-05-20 00:12
Factorizing with MSIEVE, GGNFS & Factmsieve.py	Romuald	Msieve	24	2015-11-09 20:16
Infinite loop for ggnfs or msieve	Greebley	Aliquot Sequences	4	2013-02-06 19:28
Error running GGNFS+msieve+factmsieve.py	D. B. Staple	Factoring	6	2011-06-12 22:23
A new driver? (or type of driver?)	10metreh	Aliquot Sequences	3	2010-02-15 15:57

2011-01-11, 15:09	#530
Jeff Gilchrist Jun 2003 Ottawa, Canada 3×17×23 Posts	Sorry Carpi, I didn't mean for you to think that I wrote the script, it was actually Brian Gladman who wrote it. We will have to wait and see what he thinks. If you are getting something like 50% duplicate relations are the ranges not being manged properly and the different threads/clients are sieving the same range?

2011-01-20, 21:07	#536
jasonp Tribal Bullet Oct 2004 3·1,181 Posts	Consider updating your msieve version as well; yours looks to be a few months old and the real v1.48 release has much better poly selection.

2011-01-20, 22:15	#539
firejuggler Apr 2010 Over the rainbow A2B₁₆ Posts	i'm rather new at binary compiling, so i have no idea of how to do it. Will try to find a method.