Python Driver for GGNFS and MSIEVE

[Brian Gladman]

Originally Posted by mdettweiler

Okay, cool. I've downloaded the updated version; I'll give it a whirl and have my computer do a similar automatic shutdown tonight in what will probably be the middle of the job. That should give a pretty conclusive answer as to whether the modifications work. And this time, I'll back up the sieve folder before resuming, so if there's a problem again, I can do as 10metreh suggests and force it to resume properly manually.

BTW, one other unrelated thing I noticed with the new version. Since this is the first time I've tried a version with the classical sieve routine fixed, it tried to do it after noticing that there was no record of having done it before. But, it seems that it had a problem running msieve:

-> ________________________________________________________________
-> | Running factmsieve.py, a Python driver for MSIEVE with GGNFS |
-> | sieving support. It is Copyright, 2010, Brian Gladman and is |
-> | a conversion of factmsieve.pl that is Copyright, 2004, Chris |
-> | Monico.          This is version 0.2, dated 14 January 2010. |
-> |______________________________________________________________|
-> This is client 1 of 1
-> Using 1 threads
-> Working with NAME = test
-> Selected default factorization parameters for 112 digit level.
-> Selected lattice siever: gnfs-lasieve4I13e
-> No parameter change detected. Resuming.
-> Line file scanned: resuming classical sieve from b = 1.
-> msieve -s test.dat -l test.log -i test.ini -v -nf test.fb -t 1 -ns 1,300
cannot open input file 'test.ini'
-> Q0 = 2050000, QSTEP = 100000.
-> making sieve job for q from 2065523 to 2150000 as file test.job.T0.
-> Lattice sieving algebraic q-values from q = 2050000 to 2150000.
-> gnfs-lasieve4I13e -k -o spairs.out.T0 -v -n0 -a test.job.T0
FBsize 151905+0 (deg 5), 250149+0 (deg 1)
total yield: 27608, q=2068499 (0.00950 sec/rel)

The strange thing is, test.ini was verifiably present and had its number in there exactly as it should. So I'm not entirely sure what went wrong. In this case it's no big deal since it went on to resume the GNFS without a problem, but nonetheless it's worth looking into.

I made a change that I now think messes up running msieve. I am pretty certain I will need to reverse it.

The default directory is traditionally set to the directory containing all the files associated with the number being tested.

But the CUDA version of msieve then looks for its *.ptx support files in this default directory, which is not where they will normally be located. In an effort to solve problem, I changed the default directory when msieve is being run to the msiieve binary path. But this messes up files listed on the msieve command line since these are expeccted to be in the default directory.

I'll update the program to put the normal default directory behaviour back and look for another solution to the *.ptx issue.

Brian

[dekcarki]

version 0.3, dated 15 January 2010.

Traceback (most recent call last):
  File "../factmsieve.py", line 1952, in <module>
    run_siever(CLIENT_ID, NUM_THREADS, fact_p, lats_p)
  File "../factmsieve.py", line 1594, in run_siever
    procs.append(run_exe(lats_p['siever'], args, wait = False))
  File "../factmsieve.py", line 263, in run_exe
    al['preexec_fn'] = NICE_BIN_PATH
NameError: global name 'NICE_BIN_PATH' is not defined

line 72: NICE_PATH = ''

line 263: al['preexec_fn'] = NICE_BIN_PATH

[Brian Gladman]

Originally Posted by dekcarki

version 0.3, dated 15 January 2010.

Traceback (most recent call last):
  File "../factmsieve.py", line 1952, in <module>
    run_siever(CLIENT_ID, NUM_THREADS, fact_p, lats_p)
  File "../factmsieve.py", line 1594, in run_siever
    procs.append(run_exe(lats_p['siever'], args, wait = False))
  File "../factmsieve.py", line 263, in run_exe
    al['preexec_fn'] = NICE_BIN_PATH
NameError: global name 'NICE_BIN_PATH' is not defined

line 72: NICE_PATH = ''

line 263: al['preexec_fn'] = NICE_BIN_PATH

Thanks!

Brian

[dekcarki]

please change line 217-220 to

with open(fr, 'rb') as in_file:
  out_file = gzip.open(to, 'ab')  # a for appending
  out_file.writelines(in_file)
  out_file.close()

otherwise you overwrite the compressed file and only save the last sieved range

[jasonp]

Originally Posted by Brian Gladman

I'll update the program to put the normal default directory behaviour back and look for another solution to the *.ptx issue.

Perhaps the right thing to do is compile the PTX code into the msieve binary; gcc can't do that on windows but the MS tools can...

[Brian Gladman]

Originally Posted by jasonp

Perhaps the right thing to do is compile the PTX code into the msieve binary; gcc can't do that on windows but the MS tools can...

Thanks Jason - is it obvious how to do this (I am not on my development machine right now)

Brian

[jasonp]

Originally Posted by Brian Gladman

Thanks Jason - is it obvious how to do this (I am not on my development machine right now)

You just have to pass the right flag ('-cubin'?) to nvcc, and it will turn the ptx file into a giant string and use the MS tools to compile into a COFF .obj file. Unfortunately, the program then has to get a pointer to the string and pass it to the driver call, rather than pointing to a file. I guess it isn't so easy :)

[mdettweiler]

Okay, I took another whack at restarting my C114 job after a forced shutdown and it still started from the very beginning. But, I got some good information possibly indicating the root of the problem, and this time I had a backup so I didn't lose anything.

I noticed that after I restarted the script, and it went back to q=1750000 (the beginning of the job), the contents of the test.RESUME file were as follows:

Q0: 1750000
QSTEP: 100000
QLAST0: 2766199

The value shown in QLAST0 is the number it's supposed to be restarting from--exactly where it was when the forced shutdown happened. But the strange thing is, even though it apparently had access to this information, it didn't utilize it, and went back to 1750000 anyway.

I was able to rather easily fix this up by changing Q0 in the test.RESUME file to the value shown in QLAST0 above, and concatenating the spairs.out.T0 from my backup to the spairs.add file in the main directory.

[Brian Gladman]

Originally Posted by mdettweiler

Okay, I took another whack at restarting my C114 job after a forced shutdown and it still started from the very beginning. But, I got some good information possibly indicating the root of the problem, and this time I had a backup so I didn't lose anything.

I noticed that after I restarted the script, and it went back to q=1750000 (the beginning of the job), the contents of the test.RESUME file were as follows:

The value shown in QLAST0 is the number it's supposed to be restarting from--exactly where it was when the forced shutdown happened. But the strange thing is, even though it apparently had access to this information, it didn't utilize it, and went back to 1750000 anyway.

I was able to rather easily fix this up by changing Q0 in the test.RESUME file to the value shown in QLAST0 above, and concatenating the spairs.out.T0 from my backup to the spairs.add file in the main directory.

Thanks - that is very useful.

The problem is that, for some reason, the QLAST0 value doesn't satify Q0 <= QLAST0 <= Q0 + QSTEP. So it looks like Q0 is not being updated properly when each range is completed.

Thanks ffor the detective work!

Brian

[Brian Gladman]

Originally Posted by jasonp

You just have to pass the right flag ('-cubin'?) to nvcc, and it will turn the ptx file into a giant string and use the MS tools to compile into a COFF .obj file. Unfortunately, the program then has to get a pointer to the string and pass it to the driver call, rather than pointing to a file. I guess it isn't so easy :)

My bit is easier than your bit it would seem!

Brian

[Brian Gladman]

Originally Posted by mdettweiler

Okay, I took another whack at restarting my C114 job after a forced shutdown and it still started from the very beginning. But, I got some good information possibly indicating the root of the problem, and this time I had a backup so I didn't lose anything.

I noticed that after I restarted the script, and it went back to q=1750000 (the beginning of the job), the contents of the test.RESUME file were as follows:

The value shown in QLAST0 is the number it's supposed to be restarting from--exactly where it was when the forced shutdown happened. But the strange thing is, even though it apparently had access to this information, it didn't utilize it, and went back to 1750000 anyway.

I was able to rather easily fix this up by changing Q0 in the test.RESUME file to the value shown in QLAST0 above, and concatenating the spairs.out.T0 from my backup to the spairs.add file in the main directory.

Are there any clues in the log file on why Q0 is not being updated after each sieving round?

Brian