Python Driver for GGNFS and MSIEVE

[yoyoyogi]

Yes, example.log was created. I've attached that file along with a txt file that is a transcript of the command prompt window executing the driver (in case that helps).

-Yogesh

[Brian Gladman]

It appears that running the script a second time may have deleted key data from the first run that might have told us what was wrong.

After such a big investement of time it is always advisable to back up the data you have got before attempting any recovery.

There are people here who can give you better advice than I can offer but I am rather doubtful that there is now sufficient data around to support any recovery action.

Brian

[yoyoyogi]

That's unfortunate because I'm using Amazon EC2 for this task (pay by hour), but I already created a new working folder and began the process over again yesterday. Hopefully this new attempt will have better results, otherwise I'll be back in 10 days asking for help (backing up data first, of course). Thank you again for all your help.

-Yogesh

[tmorrow]

Reporting some C130+ factorisation metrics using recent versions of factmsieve.py, currently v0.51.

---------  ------------  ---------------  ------------  -------------  -----------  ------------
Composite  RawRelations  UniqueRelations  MinRelations  RawRel/MinRel  BuildMemory  MatrixMemory
---------  ------------  ---------------  ------------  -------------  -----------  ------------
C131       20428935      18229951         15671910      1.30              632.2 MB      456.5 MB
C132       20686219      18512754         16577238      1.25              631.6 MB      471.3 MB
C132       21279166      18674765         16577238      1.28              681.1 MB      501.3 MB
C132       21443233      18704667         16577238      1.29              630.3 MB      469.5 MB
C134       21421289      18846476         18547811      1.15              717.5 MB      528.5 MB
C134       22332152      19221500         18547811      1.20              739.4 MB      554.2 MB
C135       21979269      19133024         19619273      1.12              713.9 MB      538.9 MB
C135       22145307      19135965         19619273      1.13              760.7 MB      567.5 MB
C135       22807844      19305359         19619273      1.16              806.8 MB      602.8 MB
---------  ------------  ---------------  ------------  -------------  -----------  ------------

Looks like minrel is around 10% too low in this range based on the numbers I've been factoring (cyclotomic numbers).

I notice v0.55 is now out. Are there any release notes indicating what's different/new between versions, it's getting hard to keep up.

[Brian Gladman]

Thanks for the additional values - are all of these for lpbr/a = 27?

Now that the program is mostly working, I hope that releases will be less frequent and will be mostly enhancements rather than bug fixes. I will then explain what changes between versions. The changes between 0.51 and 0.55 are (with, I think, one exception) relatively minor bug fixes that have not impacted on core functionality but have improved what happens when things go wrong.

I did however make a change to make it easier to use the GPU version of msieve. The msieve -g switch is now accepted and msieve runs in its own install directory, which makes it unnecessary to move the *.ptx files when running this msieve version.

I have now had a lot of feedback to suggest that the core functionality of the program is pretty robust. The areas where I have been asked to consider improvements are: (a) better minrels values (which is onngoing), (b) automated collection of a few more relations when the -nc2 step fails, and (c) automated recovery when a siever fails for some reason.

I am happy to collect others provided this does not create expectations that they will be acted on :-)

Brian

[tmorrow]

No, I'm using all factmsieve.py defaults (except setting paths and NUM_CPU's) and it appears the program is using lpbr/a=28 in all cases. I'm guessing this from the g???-c???.txt files that all have a line like:

Prototype def-par.txt line would be: gnfs,134,5,67,2000,5e-06,0.28,250,20,50000,3600,9300000,9300000,28,28,55,55,2.5,2.5,100000

I'm going to try the GPU version of msieve. I had to update my nvidia driver since it was complaining about nvcuda.dll not being found indicating my driver version wasn't CUDA enabled.

I'm still grinding out a c136 factorisation, after that I'll try the factmsieve.py with CUDA enabled msieve.

[Jeff Gilchrist]

Originally Posted by Brian Gladman

I did however make a change to make it easier to use the GPU version of msieve. The msieve -g switch is now accepted and msieve runs in its own install directory, which makes it unnecessary to move the *.ptx files when running this msieve version.

CUDA is enabled by default in the latest version which gives a "ret error" if you are using the non-GPU version of msieve. I have updated the GNFS tutorial accordingly to make sure people now set the CUDA value to True or False depending on their setup.

Jeff.

[Brian Gladman]

Thanks Jeff - I'll see if I can exit more gracefully if this is set up wrongly.

Brian

[Brian Gladman]

Originally Posted by tmorrow

No, I'm using all factmsieve.py defaults (except setting paths and NUM_CPU's) and it appears the program is using lpbr/a=28 in all cases.

I've added a minrels estimate for lpbr/a = 28 based on your table to v0.56 of the Python driver attached.

Brian

[tmorrow]

I ran into a problem after the sieving with v0.56. Here's the error:

Found 9167462 relations, 100.3% of the estimated minimum (9143750).
Traceback (most recent call last):
  File "../factmsieve.py", line 2003, in <module>
    run_siever(CLIENT_ID, NUM_THREADS, fact_p, lats_p)
  File "../factmsieve.py", line 1715, in run_siever
    if run_msieve(MSIEVE, '-t {0:d} -nc1'.format(NUM_CPUS)):
TypeError: run_msieve() takes exactly 1 argument (2 given)

Line 1715 is the run_msieve call and is now in error, I'm assuming the MSIEVE parameter should not be there.

[Brian Gladman]

Thanks - I thought I had removed all of these but it seems I might have inadvertantly put one back.

Brian