Running Msieve

[miklin]

Originally Posted by jasonp

Excellent. Don't worry, your english is far better than my Russian :)

In the v1.33 source, could you replace gnfs/sqrt/sqrt_a.c with the version from v1.32, then recompile and retry the square root, looking for errors? If you do not get errors this will help isolate what is happening.

Here not that has copied.

Wed Feb 06 21:49:49 2008  Msieve v. 1.33
Wed Feb 06 21:49:49 2008  random seeds: ce1869a0 8c0362b8
Wed Feb 06 21:49:49 2008  factoring 8266309047181022405353777302426117907484231464195030961205492983304640804057155643239818038225508173250310696877219392691052751978838088765473026049013959 (154 digits)
Wed Feb 06 21:49:51 2008  no P-1/P+1/ECM available, skipping
Wed Feb 06 21:49:51 2008  commencing number field sieve (154-digit input)
Wed Feb 06 21:49:51 2008  R0: -259180767284417684872274319851
Wed Feb 06 21:49:51 2008  R1:  130576890561073429
Wed Feb 06 21:49:51 2008  A0:  321734774100366992493104066152882456
Wed Feb 06 21:49:51 2008  A1:  2324280139698946250758568017311
Wed Feb 06 21:49:51 2008  A2: -12665674762234440961836116
Wed Feb 06 21:49:51 2008  A3: -7047205438924859234
Wed Feb 06 21:49:51 2008  A4:  40087789172148
Wed Feb 06 21:49:51 2008  A5:  7068000
Wed Feb 06 21:49:51 2008  size score = 6.079972e-016, Murphy alpha = -5.836125, combined = 4.253703e-015
Wed Feb 06 21:49:51 2008  
Wed Feb 06 21:49:51 2008  commencing square root phase
Wed Feb 06 21:49:51 2008  reading relations for dependency 1
Wed Feb 06 21:50:03 2008  read 1889906 cycles
Wed Feb 06 21:50:09 2008  cycles contain 5628669 unique relations
Wed Feb 06 22:00:19 2008  read 5628669 relations
Wed Feb 06 22:01:32 2008  multiplying 8501850 relations
Wed Feb 06 22:57:30 2008  multiply complete, coefficients have about 502.23 million bits
Wed Feb 06 22:57:39 2008  error: relation product is incorrect; if this is a debug binary, make sure that NDEBUG is not defined
Wed Feb 06 22:57:39 2008  algebraic square root failed
Wed Feb 06 22:57:39 2008  reading relations for dependency 2
Wed Feb 06 22:57:41 2008  read 1890135 cycles
Wed Feb 06 22:57:48 2008  cycles contain 5629806 unique relations

[jasonp]

Originally Posted by miklin

Here such result has received. Speak what to try, I shall make.

Okay, now start with v1.33 and on line 561 of common/fastmult.c change the function get_log2_fhtsize() to

static int32 get_log2_fhtsize(int32 nwords, int32 *log2_base) {

	int32 i;
	int32 runlength;
	int32 bits;

	/* begin by assuming base 2^16 for the FHT multiply */

	bits = 16;
	runlength = 1;
	while ((1 << runlength) < 2 * nwords)
		runlength++;

	*log2_base = bits;
	return runlength;
}

then recompile and rerun the square root. Any difference?

[miklin]

Originally Posted by jasonp

Okay, now start with v1.33 and on line 561 of common/fastmult.c change the function get_log2_fhtsize() to
then recompile and rerun the square root. Any difference?

Has corrected, we wait result.
Can on mail we shall pass???
And which at you there now hour???

miklin@diakom.ru
Sergey Miklin

[miklin]

Originally Posted by jasonp

Okay, now start with v1.33 and on line 561 of common/fastmult.c change the function get_log2_fhtsize() to
then recompile and rerun the square root. Any difference?

This change too has not helped.

Thu Feb 07 00:30:06 2008  commencing square root phase
Thu Feb 07 00:30:06 2008  reading relations for dependency 1
Thu Feb 07 00:30:08 2008  read 1889906 cycles
Thu Feb 07 00:30:14 2008  cycles contain 5628669 unique relations
Thu Feb 07 00:40:29 2008  read 5628669 relations
Thu Feb 07 00:41:42 2008  multiplying 8501850 relations
Thu Feb 07 01:38:50 2008  multiply complete, coefficients have about 502.23 million bits
Thu Feb 07 01:38:59 2008  error: relation product is incorrect; if this is a debug binary, make sure that NDEBUG is not defined
Thu Feb 07 01:38:59 2008  algebraic square root failed
Thu Feb 07 01:38:59 2008  reading relations for dependency 2
Thu Feb 07 01:39:01 2008  read 1890135 cycles
Thu Feb 07 01:39:08 2008  cycles contain 5629806 unique relations
Thu Feb 07 01:49:19 2008  read 5629806 relations
Thu Feb 07 01:50:32 2008  multiplying 8503188 relations

[xilman]

I decided to investigate msieve in earnest, having heard so many good things about its NFS post-processing. Previously I'd used it only for MPQS on small integers.

I'd like advice on two matters, if possible.

First, how in practice do I use it for the post processing? I've a ggnfs siever busy creating relations into a file spairs.out. I also fired up msieve to get the polynomial file and produce some relations from its line siever. These appear in msieve.dat. (Yes, I've read the doc where it's said that the ggnfs polynomial finder may be better, but this is a toy factorization of a C118 as a learning exercise.) When the ggnfs siever crashes after running a batch (the subject of the second question), I've been appending spairs.out to msieve.dat. Is this the right thing to do, and what must I do when there are enough relations in it?

The second matter concerns the ggnfs siever. It fires up fine and runs a batch of special-q in a range of 100K. However, each job crashes as follows:

Attempt 50 / 50 for: 1860275116805366835360883350634230901440647651446421083579260567272044334669573697107233781966234371079653406546373061
Monic polynomial: T=-433537265046439130785134915151590456362713515720057710368441977804976253184000000000000-10783833185000965790015287955584003256887059019893082649706624000000000X-29100662953302191018121148275402779971380760416000000X^2 + 290293756100973932442482108633256000X^3-861398521692980001X^4 + 1X^5
Obtained integral basis:
W = 
121915830631347063321569789508142702722517248000000000  0  0  60957915315673531660784894754071351361258624000000000  91436872973510297491177342131107027041887936000000000  
0  40638610210449021107189929836047567574172416000000000  2779227567407944813460117943759304551590622528000000  37368091772074993803490031804373580149167570880000000  16743628413329954526125570650157436240812446704000000  
0  0  562903328421084622898571809472000000  6416051303723462998429547256696000  530183519029528744482244985293836000  
0  0  0  1299503745767304000  221521266446439999  
0  0  0  0  1  
denominator = 121915830631347063321569789508142702722517248000000000
Checking file rels.bin.0 ...
Largest prel file size is 0 versus max allowed of 128000000.
Warning: Could not stat processed file rels.bin.0. Is this the first run?.
     New file is 53.39263MB.
     New file appears to have 494487 relations.
Building (a,b) hash table...0..makeABList() Failed to open rels.bin.0 for read!

makeABLookup() : Sorting abList...Done.
Before processing new relations, there are 0 total.
Return value 11. Terminating...ns from spairs.out... (at 11459.88 rels/sec)

I've tried "touch rels.bin.0" to create the file but the crash still happens.

I've Googled the error message and found 3 or 4 complaints from others about the crash but no answer was forthcoming.

Thanks for any assistance.

Paul

[rogue]

Originally Posted by xilman

I decided to investigate msieve in earnest, having heard so many good things about its NFS post-processing. Previously I'd used it only for MPQS on small integers.

Greg Childers created the script factMsieve.pl to be used in place of factLat.pl. It uses the ggnfs suite for polynomial selection and sieving and msieve for the rest. I've used it a few times and it has worked very well. This post from Greg explains how to use factMsieve.pl to complete a factorization started by factLat.pl.

[xilman]

Originally Posted by rogue

Greg Childers created the script factMsieve.pl to be used in place of factLat.pl. It uses the ggnfs suite for polynomial selection and sieving and msieve for the rest. I've used it a few times and it has worked very well. This post from Greg explains how to use factMsieve.pl to complete a factorization started by factLat.pl.

Thanks Mark. I'll give it a try when the current sieving job ends.

Paul

[xilman]

Originally Posted by xilman

Thanks Mark. I'll give it a try when the current sieving job ends.

Paul

Hmmm.

pcl@imhotep ~/msieve-1.33/tmp $ factMsieve.pl
This Perl has neither ithreads nor 5005threads at /usr/lib64/perl5/5.8.8/Thread.pm line 335
BEGIN failed--compilation aborted at /usr/lib64/perl5/5.8.8/Thread.pm line 335.
Compilation failed in require at /home/pcl/bin/factMsieve.pl line 65.
BEGIN failed--compilation aborted at /home/pcl/bin/factMsieve.pl line 65.
pcl@imhotep ~/msieve-1.33/tmp $

Oh well, time to start hacking Perl and/or factMsieve.pl I guess.

Paul

[xilman]

Originally Posted by xilman

Oh well, time to start hacking Perl and/or factMsieve.pl I guess.

Paul

This change

=for comment
if ($NUM_THREADS != 1) {
  use Config; 
  if (!($Config{usethreads})) { 
    printf "This version of Perl doesn't support multiple threads.  Only using 1 thread.";
    $NUM_THREADS=1; 
  }
  else {
    use Thread qw(async);
  }
}
=cut

may be brutal, but it seems to have let the siever start. It remains to be seen what else will die in the future.

Paul

[frmky]

Originally Posted by xilman

(cut) may be brutal, but it seems to have let the siever start. It remains to be seen what else will die in the future.

Paul

That'll work. Or just set NUM_THREADS to 1 before that section.

As for the crashing, the GGNFS lattice siever likes to do that. I've found it very mysterious. Compiling with -fomit-frame-pointer seems to be a recipe to create a broken binary every time. Even without that tho, I've had a few binaries that crash frequently, and others that never seem to crash. It seems to depend on the compiler you use, and perhaps even on the day of the week and whether it's raining or not.

Greg

[bsquared]

Originally Posted by xilman

The second matter concerns the ggnfs siever. It fires up fine and runs a batch of special-q in a range of 100K. However, each job crashes as follows:

The crash you describe is happening in the procrels program, not the siever. Perhaps something is corrupted with the spairs.out file, or it is too big? I've had procrels crash when spairs.out is big, on the order of 1GB.

Originally Posted by xilman

... how in practice do I use it for the post processing?

My scheme to use ggnfs/msieve is to run a simple modified version of factLat.pl, where the modification is to stick an exit 0 right before it starts the matrix process. Then I do a

procrels -prel rels.bin -fb input.fb -dump

to dump the relations in a format msieve can handle. then:

 
make_msieve.pl input.poly spairs.dump.*

to build up msieve.dat the factorbase (with poly), and worktodo.ini

and finally

 
msieve -v -nc -i worktodo.ini

I got make_msieve.pl from this forum somewhere. I think it came from fivemack. I've attached it as well, if you are interested in this approach.

These 4 commands can go in a simple shell script for fire-n-forget NFS.

Sometimes factLat.pl doesn't work so well with SNFS jobs, in which case I run the lattice siever and procrels manually until there are enough relations to do the rest of the steps and start msieve.