Python Driver for GGNFS and MSIEVE

[jasonp]

Originally Posted by 10metreh

Does the figure of 3.97003e+10 change sometimes?

Doubtful, though it probably is increasing. The perl script sets a range of leading coefficients to search that is a few orders of magnitude smaller than is printed out in the 5 digits above.

The SVN version of msieve has parameters for up to C190, though that does not make the job any easier. Big jobs like this are still probably better done with the Kleinjung tools, and the parameters definitely have to be selected by hand.

[miklin]

Originally Posted by Brian Gladman

On the processor information output I have found that I can get most of what is needed direct in Python without the need for an auxilliary program. The only bit that is a bit tricky is the speed output, which needed a bit of work.

In the attached version of the script I have got the speed part working on Windows and I have tried to give a *nix solution but it may fail as I cannot test or debug it. This does not now need the proc_info binary.

Brian

 Found 1197203 relations, 101.2% of the estimated minimum (1183506).
-> ./msieve -s ../tests/test/test.dat -l ../tests/test/test.log -i ../tests/test/test.ini -nf ../tests/test/test.fb -t 2 -nc1
-> Running matrix solving step ...
-> ./msieve -s ../tests/test/test.dat -l ../tests/test/test.log -i ../tests/test/test.ini -nf ../tests/test/test.fb -t 2 -nc2
-> Running square root step ...
-> ./msieve -s ../tests/test/test.dat -l ../tests/test/test.log -i ../tests/test/test.ini -nf ../tests/test/test.fb -t 2 -nc3
-> Computing 1.2709e+09 scale for this machine...
-> procrels -speedtest> PIPE
Scaled time: 0.75 units (timescale= 1.887).
processors: 2, speed: 2.0GHz
-> Factorization summary written to g85-test.txt
siever terminated

[Brian Gladman]

Thanks Miklin - that is most helpful. I made an error in sending the processors/speed output to the screen instead of the summary file. But the hard bit appears to work. Thanks for the summary file - on your system the processor description line is empty - I don't know why.

Maybe platform.processor() this doesn't work in Python on Linux.

Thanks again.

Brian

[miklin]

Originally Posted by Brian Gladman

Thanks Miklin - that is most helpful. I made an error in sending the processors/speed output to the screen instead of the summary file. But the hard bit appears to work. Thanks for the summary file - on your system the processor description line is empty - I don't know why.

Maybe platform.processor() this doesn't work in Python on Linux.

Thanks again.

Brian

http://www.bramz.net/projects-code/pycpuid/

pycpuid: CPUID for Python

pycpuid is a very simple Python extension. It reads the information available from the CPUID assembly instruction, and makes it available to any Python program. I needed it to decide on some codepath based on whether the box supported SSE2. In particular, I needed to know if I could import an extension module that was optimized with SSE2
import pycpuid
if pycpuid.HAS_SSE2:
import foobar_sse2 as foobar
else:
import foobar
I didn’t found anything alike, so I coded it myself. And here it is for the rest of you …
It is not the goal of pycpuid to provide a full report of all CPUID information available. It’s merely a way to get raw access to the machine instruction from within Python. Some functions are provided for translation to something human readable, but this is far from complete. Full details on how to interpret the raw data can be found in the application notes of Intel and AMD.
using it

There’s not much to it, really. pycpuid is just a bunch of module constants. Just import the module and access the constants. the HAS_FOOBARs are Boolean flags to indicate whether the feature is available. The function features() returns a list of all the available features as strings. There are some other functions like vendor() and brand_string() you can use to identify the CPU.
import pycpuid
print "has SSE2:", pycpuid.HAS_SSE2
print "all availabe features:", pycpuid.features()
print "brand string:", pycpuid.brand_string()
subversion repository

• http://svn.bramz.net/pycpuid/

download source

• pycpuid-0.2.0.zip (ChangeLog)
• pycpuid-0-1-1.zip

building and installing

There’s two simple steps to it:

1. Unpack the source code into a temporary directory, or grab it from the repository
2. Enter setup.py install on the command line. If you’re doing this on a Windows box and things freak out because you don’t have a C++ compiler installed or properly configured, you might be interested in my five little steps how to use the Visual C++ 2005 Express compiler (which is availabe for free!)

license

pycpuid is licensed under the GNU Lesser General Public License (LGPL) so that you can import pycpuid in your Python code without it imposing any license restrictions on your code.

[Brian Gladman]

Hi Miklin,

Thank you for the link to CPUID for Python. It's certainly comprehansive but goes a long way beyond what we really need for the Python script. In particular I am trying to avoid users having to install additional programs.

With your useful feedback, I think I now have a solution since your output was fine - the empty line is normal on many Linux systems so I can just leave it out when it is empty.

Brian

[miklin]

Originally Posted by Brian Gladman

Hi Miklin,

Thank you for the link to CPUID for Python. It's certainly comprehansive but goes a long way beyond what we really need for the Python script. In particular I am trying to avoid users having to install additional programs.

With your useful feedback, I think I now have a solution since your output was fine - the empty line is normal on many Linux systems so I can just leave it out when it is empty.

Brian

I here only think that it makes sense to GGNFS and MSIEVE to add nevertheless CPUID as is made in http://www.mersenne.org/freesoft/
It will make sense for more delicate work with the processor.
Instead of to compile assemblages under each processor with its features.

P.S And here still as a wish to managers GGNFS. While already to result a branch trank in an order. To clean all superfluous having left only lasieve4, pol5, experimental. All the rest simply to combine in archive.

[EdH]

Hi Brian,

Just a note to let you know what one of the linux machines logged for CPU info with 0.66:

...
total time: 3.93 hours.
athlon
Linux-2.6.30.10-105.2.23.fc11.i586-i686-athlon-with-fedora-11-Leonidas

I haven't swapped the script on the WinXP machine yet, since the last I looked it was still within a sieving run and although there shouldn't be an issue between versions, I'd rather wait until I'm sure I won't have to restart, before swapping.

Thanks again.

Take Care,
Ed

[Brian Gladman]

Originally Posted by EdH

Hi Brian,

Just a note to let you know what one of the linux machines logged for CPU info with 0.66:

...
total time: 3.93 hours.
athlon
Linux-2.6.30.10-105.2.23.fc11.i586-i686-athlon-with-fedora-11-Leonidas

I haven't swapped the script on the WinXP machine yet, since the last I looked it was still within a sieving run and although there shouldn't be an issue between versions, I'd rather wait until I'm sure I won't have to restart, before swapping.

Thanks Ed,

That is most helpful as it shows the processor information is working on Linux. The processor and speed are not there because I sent them to the screen rather than the summary file but Miklin's output suggests that this also works on *nix.

The attached version is identical to v66 except that this error has been corrected.

Thanks again for your input.

Brian

[miklin]

Originally Posted by Brian Gladman

The attached version is identical to v66 except that this error has been corrected.

Thanks again for your input.

Brian

Number: test
N = 1877138824359859508015524119652506869600959721781289179190693027302028679377371001561 (85 digits)
Divisors found:
r1=1263789702211268559063981919736415575710439 (pp43)
r2=1485325304578290487744454354798448608807999 (pp43)
Version: Msieve v. 1.45
Total time: 0.30 hours.
Factorization parameters were as follows:
n: 1877138824359859508015524119652506869600959721781289179190693027302028679377371001561
Y0: -190015192251850828224
Y1: 40798639309
c0: -390009027932885683113911
c1: 16339927832902891098
c2: 13428239826023
c3: -299606994
c4: 1440
skew: 197078.47 
type: gnfs
Factor base limits: 550000/550000
Large primes per side: 3
Large prime bits: 24/24
Sieved algebraic special-q in [0, 0)
Total raw relations: 0
Relations: 51288 relations
Pruned matrix : 40114 x 40339
Polynomial selection time: 0.00 hours.
Total sieving time: 0.28 hours.
Total relation processing time: 0.01 hours.
Matrix solve time: 0.01 hours.
time per square root: 0.00 hours.
Prototype def-par.txt line would be: gnfs,84,4,56,1500,0.001,0.3,200,15,10000,500,550000,550000,24,24,40,40,1.9,1.9,10000
total time: 0.30 hours.

Linux-2.6.26-2-amd64-x86_64-with-debian-5.0.4
processors: 2, speed: 2.00GHz

[EdH]

Hi Brian,

Here's my latest linux (the second linux machine is on vacation):

. . .
total time: 2.34 hours.
athlon
Linux-2.6.30.10-105.2.23.fc11.i586-i686-athlon-with-fedora-11-Leonidas
processors: 1, speed: 1.79GHz

But, my WinXp didn't like something. The factoring went OK, but the script ended with the following:

. . .
Scaled time: 9.70 units (timescale= 0.563). 
Traceback (most recent call last): 
  File "C:\MathWork\ggnfs\factmsieve.py", line 2073, in <module> 
    output_summary(sumname, timescale, fact_p, pols_p, poly_p, lats_p) 
  File "C:\MathWork\ggnfs\factmsieve.py", line 1892, in output_summary 
    .format(multiprocessing.cpu_count(), proc_speed()), file = out_f) 
  File "C:\MathWork\ggnfs\factmsieve.py", line 269, in proc_speed 
    import _winreg 
ImportError: No module named _winreg

The g###-test.txt says:

. . .
total time: 17.23 hours. 
x86 Family 15 Model 2 Stepping 7, GenuineIntel 
Windows-XP-5.1.2600-SP3

I'm using Python 3.1 on the WinXP machine.

Thanks for all the work.

Take Care,
Ed

[Brian Gladman]

Hi Ed,

That's because my code is not intended for Python 3.1 yet - in 3.1 the '_winreg' module has been renamed to 'winreg' so it might work if you take the underscore out.

Brian