mersenneforum.org - Python Driver for GGNFS and MSIEVE

mersenneforum.org (https://www.mersenneforum.org/index.php)

- Msieve (https://www.mersenneforum.org/forumdisplay.php?f=83)

- - Python Driver for GGNFS and MSIEVE (https://www.mersenneforum.org/showthread.php?t=12981)

Hello EdH,

I've made a few changes to the file Brian provided and am including it below. I don't have a linux machine to test out my changes, but hopefully this goes better than the last thing I tried to fix. :wink: Brian may decide on a different way to fix some of this, but until then, try this and let us know how it goes.

Debian / Ubuntu

mik@homes102:~$ sudo apt-get update

mik@homes102:~$ sudo apt-get install x86info

[code]mik@homes102:~$ x86info
x86info v1.21. Dave Jones 2001-2007
Feedback to <davej@redhat.com>.

Found 2 CPUs
--------------------------------------------------------------------------
CPU #1
/dev/cpu/0/cpuid: No such file or directory
Family: 15 Model: 75 Stepping: 2
CPU Model : Athlon 64 X2 Dual-Core (BH-F2)
Processor name string: AMD Athlon(tm) 64 X2 Dual Core Processor 3600+

Feature flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflsh mmx fxsr sse sse2 ht sse3 cmpxchg16b
Extended feature flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 nx mmxext mmx fxsr ffxsr rdtscp lm 3dnowext 3dnow lahf/sahf CmpLegacy svm ExtApicSpace LockMovCr0

SVM: revision 1, 64 ASIDs
Address Size: 48 bits virtual, 40 bits physical
The physical package has 2 of 2 possible cores implemented.
--------------------------------------------------------------------------
CPU #2
Family: 15 Model: 75 Stepping: 2
CPU Model : Athlon 64 X2 Dual-Core (BH-F2)
Processor name string: AMD Athlon(tm) 64 X2 Dual Core Processor 3600+

Feature flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflsh mmx fxsr sse sse2 ht sse3 cmpxchg16b
Extended feature flags:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 nx mmxext mmx fxsr ffxsr rdtscp lm 3dnowext 3dnow lahf/sahf CmpLegacy svm ExtApicSpace LockMovCr0

SVM: revision 1, 64 ASIDs
Address Size: 48 bits virtual, 40 bits physical
The physical package has 2 of 2 possible cores implemented.
--------------------------------------------------------------------------
WARNING: Detected SMP, but unable to access cpuid driver.
Used Uniprocessor CPU routines. Results inaccurate.
[/code][code]
mik@homes102:~$ cpuid
eax in eax ebx ecx edx
00000000 00000001 68747541 444d4163 69746e65
00000001 00040fb2 00020800 00002001 178bfbff
80000000 80000018 68747541 444d4163 69746e65
80000001 00040fb2 000008cb 0000001f ebd3fbff
80000002 20444d41 6c687441 74286e6f 3620296d
80000003 32582034 61754420 6f43206c 50206572
80000004 65636f72 726f7373 30363320 00002b30
80000005 ff08ff08 ff20ff20 40020140 40020140
80000006 00000000 42004200 01008140 00000000
80000007 00000000 00000000 00000000 0000003f
80000008 00003028 00000000 00000001 00000000
80000009 00000000 00000000 00000000 00000000
8000000a 00000001 00000040 00000000 00000000
8000000b 00000000 00000000 00000000 00000000
8000000c 00000000 00000000 00000000 00000000
8000000d 00000000 00000000 00000000 00000000
8000000e 00000000 00000000 00000000 00000000
8000000f 00000000 00000000 00000000 00000000
80000010 00000000 00000000 00000000 00000000
80000011 00000000 00000000 00000000 00000000
80000012 00000000 00000000 00000000 00000000
80000013 00000000 00000000 00000000 00000000
80000014 00000000 00000000 00000000 00000000
80000015 00000000 00000000 00000000 00000000
80000016 00000000 00000000 00000000 00000000
80000017 00000000 00000000 00000000 00000000
80000018 00000000 00000000 00000000 00000000

Vendor ID: "AuthenticAMD"; CPUID level 1

AMD-specific functions
Version 00040fb2:
Family: 15 Model: 11 []

Standard feature flags 178bfbff:
Floating Point Unit
Virtual Mode Extensions
Debugging Extensions
Page Size Extensions
Time Stamp Counter (with RDTSC and CR4 disable bit)
Model Specific Registers with RDMSR & WRMSR
PAE - Page Address Extensions
Machine Check Exception
COMPXCHG8B Instruction
APIC
SYSCALL/SYSRET or SYSENTER/SYSEXIT instructions
MTRR - Memory Type Range Registers
Global paging extension
Machine Check Architecture
Conditional Move Instruction
PAT - Page Attribute Table
PSE-36 - Page Size Extensions
19 - reserved
MMX instructions
FXSAVE/FXRSTOR
25 - reserved
26 - reserved
28 - reserved
Generation: 15 Model: 11
Extended feature flags ebd3fbff:
Floating Point Unit
Virtual Mode Extensions
Debugging Extensions
Page Size Extensions
Time Stamp Counter (with RDTSC and CR4 disable bit)
Model Specific Registers with RDMSR & WRMSR
PAE - Page Address Extensions
Machine Check Exception
COMPXCHG8B Instruction
APIC
SYSCALL/SYSRET or SYSENTER/SYSEXIT instructions
MTRR - Memory Type Range Registers
Global paging extension
Machine Check Architecture
Conditional Move Instruction
PAT - Page Attribute Table
PSE-36 - Page Size Extensions
20 - reserved
AMD MMX Instruction Extensions
MMX instructions
FXSAVE/FXRSTOR
25 - reserved
27 - reserved
29 - reserved
3DNow! Instruction Extensions
3DNow instructions

Processor name string: AMD Athlon(tm) 64 X2 Dual Core Processor 3600+
L1 Cache Information:
2/4-MB Pages:
Data TLB: associativity 255-way #entries 8
Instruction TLB: associativity 255-way #entries 8
4-KB Pages:
Data TLB: associativity 255-way #entries 32
Instruction TLB: associativity 255-way #entries 32
L1 Data cache:
size 64 KB associativity 2-way lines per tag 1 line size 64
L1 Instruction cache:
size 64 KB associativity 2-way lines per tag 1 line size 64

L2 Cache Information:
2/4-MB Pages:
Data TLB: associativity L2 off #entries 0
Instruction TLB: associativity L2 off #entries 0
4-KB Pages:
Data TLB: associativity 2-way #entries 0
Instruction TLB: associativity 2-way #entries 0
size 1 KB associativity L2 off lines per tag 129 line size 64

Advanced Power Management Feature Flags
Has temperature sensing diode
Supports Frequency ID control
Supports Voltage ID control
Maximum linear address: 48; maximum phys address 40
[/code][code]mik@homes102:~$ /bin/dmesg | grep CPU
[ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[ 0.000000] PERCPU: Allocating 37168 bytes of per cpu data
[ 0.000000] NR_CPUS: 32, nr_cpu_ids: 2
[ 0.000000] Initializing CPU#0
[ 0.087966] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[ 0.087969] CPU: L2 Cache: 256K (64 bytes/line)
[ 0.087972] CPU 0/0 -> Node 0
[ 0.087983] CPU: Physical Processor ID: 0
[ 0.087985] CPU: Processor Core ID: 0
[ 0.142576] CPU0: AMD Athlon(tm) 64 X2 Dual Core Processor 3600+ stepping 02
[ 0.156009] Initializing CPU#1
[ 0.156009] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[ 0.156009] CPU: L2 Cache: 256K (64 bytes/line)
[ 0.156009] CPU 1/1 -> Node 0
[ 0.156009] CPU: Physical Processor ID: 0
[ 0.156009] CPU: Processor Core ID: 1
[ 0.235893] CPU1: AMD Athlon(tm) 64 X2 Dual Core Processor 3600+ stepping 02
[ 0.235905] Brought up 2 CPUs
[ 0.236014] CPU0 attaching sched-domain:
[ 0.236014] CPU1 attaching sched-domain:
[ 0.387560] Switched to high resolution mode on CPU 0
[ 0.390937] Switched to high resolution mode on CPU 1
[/code] [code]
mik@homes102:~$ /bin/dmesg | grep Memory
[ 0.004000] Memory: 4057444k/5505024k available (2228k kernel code, 136408k reserved, 1084k data, 392k init)
[ 38.314072] VBoxDrv: dbg - g_abExecMemory=ffffffffa0305360
mik@homes102:~$
[/code]

[quote=WraithX;211108]Hello EdH,

I've made a few changes to the file Brian provided and am including it below. I don't have a linux machine to test out my changes, but hopefully this goes better than the last thing I tried to fix. :wink: Brian may decide on a different way to fix some of this, but until then, try this and let us know how it goes.[/quote]

[code]mik@homes102:~/Desktop/proc_info$ g++ proc_info.c -o proc_info
mik@homes102:~/Desktop/proc_info$ ./proc_info
Processor Information:
name : AMD Athlon(tm) 64 X2 Dual Core Processor 3600+
identity : AuthenticAMD Family 15 Model 75 Stepping 2
processors : 2
speed : 1 GHz
mik@homes102:~/Desktop/proc_info$
[/code]
[code]mik@homes102:~/Desktop/proc_info$ /bin/dmesg | grep CPU
[ 0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[ 0.000000] PERCPU: Allocating 37168 bytes of per cpu data
[ 0.000000] NR_CPUS: 32, nr_cpu_ids: 2
[ 0.000000] Initializing CPU#0
[ 0.087966] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[ 0.087969] CPU: L2 Cache: 256K (64 bytes/line)
[ 0.087972] CPU 0/0 -> Node 0
[ 0.087983] CPU: Physical Processor ID: 0
[ 0.087985] CPU: Processor Core ID: 0
[ 0.142576] CPU0: AMD Athlon(tm) 64 X2 Dual Core Processor 3600+ stepping 02
[ 0.156009] Initializing CPU#1
[ 0.156009] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
[ 0.156009] CPU: L2 Cache: 256K (64 bytes/line)
[ 0.156009] CPU 1/1 -> Node 0
[ 0.156009] CPU: Physical Processor ID: 0
[ 0.156009] CPU: Processor Core ID: 1
[ 0.235893] CPU1: AMD Athlon(tm) 64 X2 Dual Core Processor 3600+ stepping 02
[ 0.235905] Brought up 2 CPUs
[ 0.236014] CPU0 attaching sched-domain:
[ 0.236014] CPU1 attaching sched-domain:
[ 0.387560] Switched to high resolution mode on CPU 0
[ 0.390937] Switched to high resolution mode on CPU 1
mik@homes102:~/Desktop/proc_info$
[/code]

Perfect! Brian,

Thank you again to everyone.
[code]
[ID@CompName]$ gcc proc_info.c -o proc_info
[ID@CompName]$ ./proc_info
Processor Information:
name : AMD Athlon(tm) Processor
identity : AuthenticAMD Family 6 Model 8 Stepping 1
processors : 0
speed : 1.79508 GHz
[/code]Is it odd that I have 0 processors, or is it just because with only one, they didn't incorporate the ability to provide the number yet?

[quote=EdH;211169]Perfect! Brian,

Thank you again to everyone.
[code]
[ID@CompName]$ gcc proc_info.c -o proc_info
[ID@CompName]$ ./proc_info
Processor Information:
name : AMD Athlon(tm) Processor
identity : AuthenticAMD Family 6 Model 8 Stepping 1
processors : 0
speed : 1.79508 GHz
[/code]Is it odd that I have 0 processors, or is it just because with only one, they didn't incorporate the ability to provide the number yet?[/quote]

It looks like the *nix version needs some debugging since it is not reporting your number of processors or miklin's speed correctly. Unfortunately I don't develop on Linux so I cannot do this.

I did notice that some of the local and output variables need initilaising since they won't be set if the /proc/cpuinfo file doesn't exist or doesn't have all the expected lines in the expected format.

Brian

PS: I have just looked up the specification of proc/cpuinfo and it returns 0 for single core systems - so I can correct that easily.

[quote=Brian Gladman;211177]It looks like the *nix version needs some debugging since it is not reporting your number of processors or miklin's speed correctly. Unfortunately I don't develop on Linux so I cannot do this.

I did notice that some of the local and output variables need initilaising since they won't be set if the /proc/cpuinfo file doesn't exist or doesn't have all the expected lines in the expected format.

Brian

PS: I have just looked up the specification of proc/cpuinfo and it returns 0 for single core systems - so I can correct that easily.[/quote]

It is pleasant to me as it it is realised here in this project [URL]http://www.mersenne.org/freesoft/[/URL]

[SIZE=2][SIZE=2][SIZE=2][SIZE=4][B]Software Source Code[/B] [/SIZE][/SIZE][/SIZE][/SIZE][SIZE=2][SIZE=2][SIZE=2][SIZE=2][B]If you use GIMPS source code to find Mersenne primes, you must agree to adhere to the [URL="http://www.mersenne.org/legal/#EULA"]GIMPS free software license agreement[/URL][/B]. Other than that restriction, you may use this code as you see fit.[/SIZE][/SIZE][/SIZE][/SIZE]
[SIZE=2][SIZE=2][SIZE=2][SIZE=2]The source code for the program is highly optimized Intel assembly language. There are many more-readable FFT algorithms available on the web and in textbooks. The program is also completely non-portable. If you are curious anyway, you can [URL="http://mersenneforum.org/gimps/source2511.zip"]download all the source code(4.1MB)[/URL]. This file includes all the version 25.11 source code for Windows 95/98/Me/NT/2000/XP/Vista, Linux, FreeBSD, and Mac OS X. Last updated: July 13, 2009.[/SIZE][/SIZE][/SIZE][/SIZE]
[SIZE=2][SIZE=2][SIZE=2][SIZE=2]The GIMPS program is very loosely based on C code written by Richard Crandall. Luke Welsh has started a web page that points to Richard Crandall's program and [URL="http://www.mersenne.org/various/freeware.htm"]other available source code[/URL] you can use to help search for Mersenne primes. This is a great place for UNIX users to go.[/SIZE][/SIZE][/SIZE][/SIZE]

pol51m0b.exe crash

Not sure if this is a problem you can remedy or not but...

Silly me, being the crank that I am was trying to run Polynomial selection on a c190. MSIEVE Complains about anything bigger than c180 so I set USE_KLEINJUNG_FRANKE_PS = False to true to use the poly select tool.

After about a day with no polynomials found it crashed. The problem related to the py script is that it forks each command to the console and log file and then greps the log file "success".

1) Is it possible to not print out every command to the console because I am pretty sure that slows down the polynomial selection.

2) The Log file grew to almost 10GB full of nothing but:

[CODE]
Tue Apr 06 15:58:11 2010 -> Searching leading coefficients from 3.97003e+10 to 3.97003e+10
Tue Apr 06 15:58:11 2010 -> pol51m0b -b rsa190.polselstaging1.624 -v -v -p 7 -n 7.12e+23 -a 3.97003e+07 -A 3.97003e+07>> rsa190.polselstaging1.624.log
Tue Apr 06 15:58:11 2010 -> Searching leading coefficients from 3.97003e+10 to 3.97003e+10
Tue Apr 06 15:58:11 2010 -> pol51m0b -b rsa190.polselstaging1.624 -v -v -p 7 -n 7.12e+23 -a 3.97003e+07 -A 3.97003e+07>> rsa190.polselstaging1.624.log
Tue Apr 06 15:58:11 2010 -> Searching leading coefficients from 3.97003e+10 to 3.97003e+10
[/CODE]

I am sure there must be a better implementation. I am not sure why it crashed, but perhaps it had problems "grepping" that big of a log file.

If it must be saved to the log file, perhaps the log file can be truncated in some way.

I'll leave with this question... even through I most likely do not have the capabilities to sieve such a large number, what are the recommended methods selecting a polynomial for such number numbers.

The parameters are horrible: they are taken from the file def-nm-params.txt, and they only go up to 155 digits. As a result, pol51 is trying to find polys with a normal score for a c155, but for a c190 the polys are much worse. So it won't find any polys and will thus go on searching further and further. However, [code]Searching leading coefficients from 3.97003e+10 to 3.97003e+10[/code] shows that it's hardly searching at all: it is just searching for polys with the leading coefficient 39700300000, and that probably isn't a very good one. Does the figure of 3.97003e+10 change sometimes?

For (1) set VERBOSE = False (early in the script).

Brian

On the processor information output I have found that I can get most of what is needed direct in Python without the need for an auxilliary program. The only bit that is a bit tricky is the speed output, which needed a bit of work.

In the attached version of the script I have got the speed part working on Windows and I have tried to give a *nix solution but it may fail as I cannot test or debug it. This does not now need the proc_info binary.

Brian