Python Driver for GGNFS and MSIEVE

[Brian Gladman]

Originally Posted by Mini-Geek

Huh, I hadn't thought of it that way...I just wanted to give it a little wiggle room so it'd try to post-proc before it was quite at 100%. But still, I guess you're right about what I was implying (though I didn't realize it at the time).

Or, to look at the issue from another point of view, maybe the script could be modified to see this information:
"Each section of work gets about x% of the estimate." and
"I'm now much closer to the estimate than x%."
And, instead of taking this action:
"So I'll sieve another full section of work and overshoot 100% by a lot."
Take this one:
"So I'll sieve what I can expect will put me a little over 100% and then see if I'm there."
e.g. if it goes from 84.5% to 99.5%, (15%) then it decides to only sieve 1/20th of the normal length, finishes at about 100.25%, (99.5%+15/20=100.25%) and continues to post-proc (or, in the event that it hasn't gone over 100%, the process repeats).

Reducing the step size when we have almost enough relations is a frequent request and one I would like to implement when we have enough information to do this.

However, if this is to work we have to have really good minrels estimates to start with and I am not sure we are quite there yet. I have been using the results people have been providing here to refine the estimates and I am certainly getting less 'complaints' about serious under or over sieving.

Your results are useful as I have not had many results for lpbr/a = 25 and it looks like I can cut the estimate here by about 5%

The numbers I've ran since my modification were in the 92-95 digit range. They all have this line in their g9x-test.txt file:

Large prime bits: 25/25

I'm assuming that means the lpbr/a values are 25. (there are no other references to it in the files left behind after the cleanup)
I'd have to run more tests to see just how few relations it can have at different sizes for it to still finish, but on a c92 I got to 95.7% and it still finished. ("Found 1677546 relations, 95.7% of the estimated minimum (1753486).")
If you want me to run some trials to find the minimum relations for some size(s), let me know what size(s) and any other info I need.

All information that helps to improve the minrels estimate will be gratefully received! Currently the estimates are categorised by their lpbr/a values so it is important to report both relations and lpbr/a values.

If people are content that the estimates I am using are working, I can then reduce the step size near to completion. This was one of the first requests I had but it soon became clear that we needed better minrels values before we can seriously contemplate this.

But rest assured your idea has not fallen on deaf ears

Brian

[Mini-Geek]

For a c92 (the one that previously finished at 95.7%, rerunning the sieving), using lpbr/a 25/25, I found that it needs a minimum between 1715000 (97.8%) and 1720000 (98.1%) relations to finish (it could not finish with 97.8%, but could with 98.1%). The current estimate is 1753486.
I tested this by running factmsieve.py on the original poly with FRACTION_OF_ESTIMATED set to start trying to post-proc at 50% and the cleanup disabled. Then after it finished with that, I manually removed the last relations in multiples of 10000 and then 5000 before I narrowed it to that range. Finally, I had msieve run the full solve with 1720000 relations, all the way until factors appeared, to make sure no problems cropped up with the LA or anything.

I'm very surprised to see such a different result from my first run of this number...from what I understand, that's not how the GNFS should behave. (the original relations files were cleaned up immediately, so I can't try it out on those)

I've attached the g92-test.txt and test.log files. msieve shows 1 less than the number of lines that were in the file. I'm assuming this is because the first line has the n in it instead of a relation.

[sleigher]

Thanks Jason.

I have been looking at boh the factmsieve.pl (perl) and the factmsieve.py (python) scripts. Seems they both say they can run on multi hosts but the python script dies. I have tried searching for a solution but cannot seem to locate anything.

Does it require any additional modules? I am trying to perl script now from:

http://ggnfs.svn.sourceforge.net/vie.../factMsieve.pl

[sleigher]

okay, using the perl script seems to be working. I have 3 hosts with 7 cores a piece all running from the same NFS mounted directory.

Jobs are numbered properly and all is well.

One of the hosts has CPU's that are a little faster than the other 2. Think that will matter?

[schickel]

Originally Posted by sleigher

okay, using the perl script seems to be working.

Here's a couple of notes from my experience running multiple machines:

I have 3 hosts with 7 cores a piece all running from the same NFS mounted directory.

One thing to be aware of is that as ranges finish, when appending the spairs.out.* files, the remote machines will read/write the entire block of "their" relations over the network. This means that your network traffic will take a huge spike every couple of hours (or so, depending on your CPU speeds).

I run my distributed jobs isolated on each machine and move the spairs.add.* files to the central machine once or twice a day. The central machine then finds everything and collates it as it goes.

One of the hosts has CPU's that are a little faster than the other 2. Think that will matter?

No problem there, that just means that that machine will do a little more of the work than the others....

[Brian Gladman]

Do you have any information from the log files on what happens when the Python program exists?

Brian

[chris2be8]

Originally Posted by schickel

One thing to be aware of is that as ranges finish, when appending the spairs.out.* files, the remote machines will read/write the entire block of "their" relations over the network. This means that your network traffic will take a huge spike every couple of hours (or so, depending on your CPU speeds).

That's been on my list of "things to fix once I have home network" for factMsieve.pl for over a year. It should be possible to have the slaves just rename their spairs.out.* files to something like spairs.add.Hn.Tn.$time and the master copy read any such. But I havn't got a home network to test it on yet.

Chris K

[sleigher]

Originally Posted by Brian Gladman

Do you have any information from the log files on what happens when the Python program exists?

Brian

I do have the info. I tried different way bu the output from the script seemed to say that this was the right way. I guess this is the console out.

[root@localhost fact]# /usr/local/bin/python2.6 ./factmsieve.py myjob 1 3
-> ________________________________________________________________
-> | Running factmsieve.py, a Python driver for MSIEVE with GGNFS |
-> | sieving support. It is Copyright, 2010, Brian Gladman and is |
-> | a conversion of factmsieve.pl that is Copyright, 2004, Chris |
-> | Monico.          This is version 0.58, dated 7th March 2010. |
-> |______________________________________________________________|
Traceback (most recent call last):
  File "./factmsieve.py", line 1929, in <module>
    PNUM += CLIENT_ID
TypeError: unsupported operand type(s) for +=: 'int' and 'str'

[Brian Gladman]

Here is an updated version that will (I hope) remove some of these errors. Thanks to all for the bug reports.

Brian

[sleigher]

Tested that and it worked. Thanks.

I cannot switch to the python script now after running the perl script for this long though can I? Are they interchangeable to the point that I could sop my perl job and start where I left off with a python job?

[miklin]

Now All has earned.
I the truth would add in the file beginning #!/usr/bin/python
or in a case as at Sleigher #!/usr/local/bin/python
That the call worked for it from the house catalogue.
I think that so it will be more convenient.