msieve MPI error

david314 · 2013-04-13, 07:12

Hello ,

I am trying for first time the MPI support in msieve. For my first run, I am using only 1x1 grid.

I get this error. What does it mean? Thank you

Code:

Msieve v. 1.52 (SVN 880)
Fri Apr 12 23:59:41 2013
random seeds: 70f21d5e ef3ba596
MPI process 0 of 1
factoring 37120944689800609412002515056317303932773604020034145686079164382030167043089316373333849352676912740718719896128949056002154056277661151343 (140 digits)
no P-1/P+1/ECM available, skipping
commencing number field sieve (140-digit input)
R0: -32534952778982469485491103546896674762721
R1: 1
A0: -2377
A1: 0
A2: 0
A3: 0
A4: 0
A5: 1
skew 4.73, size 5.749e-14, alpha -0.564, combined = 1.380e-11 rroots = 1

commencing linear algebra
initialized process (0,0) of 1 x 1 grid
read 406172 cycles
cycles contain 2208513 unique relations
read 2208513 relations
using 20 quadratic characters above 536847050
building initial matrix
error: overflow merging ideals

jasonp · 2013-04-13, 12:46

It means that a column in the matrix has too many nonzeros in it for the fixed-size buffers the matrix building code uses (1000 in this case)

Note that a matrix of size 2.2M is quite large for a C140, especially for SNFS, so it's likely that you only barely have enough relations to create a matrix. If you sieve a little bit more, the problematic matrix columns will be thrown away and the matrix will build successfully.

PS: I see that the original number you were factoring was 203 digits, and that the matrix is actually 400k in size, which is actually remarkably small. The underlying cause is the same (barely enough sieving), but it's unusual to have seen it work. Could you post the filtering log? (as an attachment if it's more than 10k in size)

david314 · 2013-04-13, 16:11

I tried the -nc1 step again on this machine. Even stranger results.

Code:

Msieve v. 1.52 (SVN 880)
Sat Apr 13 05:36:59 2013
random seeds: 982d5ff4 0a09ba06
MPI process 0 of 1
factoring 37120944689800609412002515056317303932773604020034145686079164382030167043089316373333849352676912740718719896128949056002154056277661151343 (140 digits)
no P-1/P+1/ECM available, skipping
commencing number field sieve (140-digit input)
R0: -32534952778982469485491103546896674762721
R1: 1
A0: -2377
A1: 0
A2: 0
A3: 0
A4: 0
A5: 1
skew 4.73, size 5.749e-14, alpha -0.564, combined = 1.380e-11 rroots = 1

commencing relation filtering
estimated available RAM is 32177.6 MB
commencing duplicate removal, pass 1
read 10M relations
read 20M relations
read 30M relations
read 40M relations
read 50M relations
read 60M relations
found 7010116 hash collisions in 61617384 relations
commencing duplicate removal, pass 2
found 4779477 duplicates and 56837907 unique relations
memory use: 213.2 MB
reading ideals above 720000
commencing singleton removal, initial pass
memory use: 1378.0 MB
reading all ideals from disk
memory use: 1948.4 MB
keeping 47348982 ideals with weight <= 200, target excess is 304146
commencing in-memory singleton removal
begin with 56837907 relations and 47348982 unique ideals
reduce to 39855511 relations and 28825503 ideals in 11 passes
max relations containing the same ideal: 163
removing 6751492 relations and 4751492 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 33104019 relations and 28825503 unique ideals
reduce to 32204736 relations and 23117721 ideals in 8 passes
max relations containing the same ideal: 144
removing 5571076 relations and 3571076 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 26633660 relations and 23117721 unique ideals
reduce to 26023682 relations and 18899805 ideals in 7 passes
max relations containing the same ideal: 129
removing 5241972 relations and 3241972 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 20781710 relations and 18899805 unique ideals
reduce to 20218656 relations and 15056591 ideals in 8 passes
max relations containing the same ideal: 108
removing 5059845 relations and 3059845 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 15158811 relations and 15056591 unique ideals
reduce to 14566468 relations and 11356184 ideals in 7 passes
max relations containing the same ideal: 83
removing 4916867 relations and 2916867 ideals in 2000000 cliques
commencing in-memory singleton removal
begin with 9649601 relations and 11356184 unique ideals
reduce to 8959286 relations and 7674883 ideals in 8 passes
max relations containing the same ideal: 59
removing 2467642 relations and 1536049 ideals in 931593 cliques
commencing in-memory singleton removal
begin with 6491644 relations and 7674883 unique ideals
reduce to 6047516 relations and 5651320 ideals in 8 passes
max relations containing the same ideal: 47
removing 214732 relations and 171346 ideals in 43386 cliques
commencing in-memory singleton removal
begin with 5832784 relations and 5651320 unique ideals
reduce to 5825483 relations and 5472613 ideals in 6 passes
max relations containing the same ideal: 47
relations with 0 large ideals: 6543
relations with 1 large ideals: 7546
relations with 2 large ideals: 93140
relations with 3 large ideals: 457709
relations with 4 large ideals: 1142888
relations with 5 large ideals: 1624338
relations with 6 large ideals: 1449097
relations with 7+ large ideals: 1044222
commencing 2-way merge
reduce to 3973954 relation sets and 3621084 unique ideals
commencing full merge
Illegal instruction

This step fails as well. (When I had tried -nc2 1,1 before, the .dat.cyc was from another machine. I thought, perhaps if I recreated it on an MPI node in the cluster...)

Batalov · 2013-04-13, 16:27

Your binary may be compiled for a different processor, if you are getting "Illegal instruction" exit.

Did you build it yourself? Did you modify the Makefile to match the CPU?
(in the line OPT_FLAGS = -O3 -fomit-frame-pointer -march=core2 ...)

david314 · 2013-04-13, 21:30

Thanks Batalov, that helps for -nc1 ! Recompiled with -march=opteron and fixed it.

To be clear, for MPI, does one use different M,N in each command line, e.g. "-nc2 1,4" "-nc2 2,4" etc. for a 4x4 grid? Or should you pass "-nc2 4,4" to each?? Thanks again!

jasonp · 2013-04-14, 00:07

Usually your MPI has an 'mpirun' script that you pass a single command line to, with one choice of 'M,N'. That gets used by everybody (if I understand what you're asking)

The best M,N depends on the machine(s), the interconnect(s) between machines, and the size of the problem. Smaller problems will only speed up if you break them up into a few MPI processes. You can also use threads and MPI together; the split between how many threads and how much MPI depends on how good your machine is and how bad your MPI library is.

All this is a roundabout way of saying you will need to do some trial and error.

2013-04-13, 12:46	#2
jasonp Tribal Bullet Oct 2004 3²·5·79 Posts	It means that a column in the matrix has too many nonzeros in it for the fixed-size buffers the matrix building code uses (1000 in this case) Note that a matrix of size 2.2M is quite large for a C140, especially for SNFS, so it's likely that you only barely have enough relations to create a matrix. If you sieve a little bit more, the problematic matrix columns will be thrown away and the matrix will build successfully. PS: I see that the original number you were factoring was 203 digits, and that the matrix is actually 400k in size, which is actually remarkably small. The underlying cause is the same (barely enough sieving), but it's unusual to have seen it work. Could you post the filtering log? (as an attachment if it's more than 10k in size) Last fiddled with by jasonp on 2013-04-13 at 13:01

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Error while running Msieve 1.53 with factmsieve.py	FelicityGranger	Msieve	3	2022-08-21 20:44
error when running msieve 1.53 with cuda	aein	Msieve	9	2019-02-25 14:09
Error running GGNFS+msieve+factmsieve.py	D. B. Staple	Factoring	6	2011-06-12 22:23
msieve C157 sqrt error: relation xxxx corrupt	tgrdy	Msieve	6	2010-08-20 21:51
Error compiling msieve	Wishper	Msieve	2	2009-12-09 01:31

2013-04-13, 16:27	#4
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 89·113 Posts	Your binary may be compiled for a different processor, if you are getting "Illegal instruction" exit. Did you build it yourself? Did you modify the Makefile to match the CPU? (in the line OPT_FLAGS = -O3 -fomit-frame-pointer -march=core2 ...)

2013-04-13, 21:30	#5
david314 Jul 2012 13 Posts	Thanks Batalov, that helps for -nc1 ! Recompiled with -march=opteron and fixed it. To be clear, for MPI, does one use different M,N in each command line, e.g. "-nc2 1,4" "-nc2 2,4" etc. for a 4x4 grid? Or should you pass "-nc2 4,4" to each?? Thanks again!

2013-04-14, 00:07	#6
jasonp Tribal Bullet Oct 2004 3²·5·79 Posts	Usually your MPI has an 'mpirun' script that you pass a single command line to, with one choice of 'M,N'. That gets used by everybody (if I understand what you're asking) The best M,N depends on the machine(s), the interconnect(s) between machines, and the size of the problem. Smaller problems will only speed up if you break them up into a few MPI processes. You can also use threads and MPI together; the split between how many threads and how much MPI depends on how good your machine is and how bad your MPI library is. All this is a roundabout way of saying you will need to do some trial and error.