Running GGNFS

[yoyo]

Hello,
does msieve and gnfs have automatic checkpoints, so they can be restarted if the computer crashed or is shut downed?
yoyo

[jasonp]

Msieve will checkpoint any linear algebra where the matrix is large enough but you have to tell it to restart from the checkpoint. Also, the quadratic sieve is fully automatic and will restart properly. Sieving using the NFS line siever also will checkpoint if there is an orderly shutdown, and restarting is automatic in that case too.

[bsquared]

Originally Posted by Jeff Gilchrist

And let me know if I'm missing something or if you find any mistakes. It is not supposed to be a extensive guide explaining all the different options or best methods to do everything but just to help someone get started.

Thanks,
Jeff.

Thanks for putting this together, I wish something like this existed when I was first learning to use the tools.

A couple comments:
1.) Some precompiled binaries exist for linux. The ones I'm thinking of are from frmky, and contain the amd64 assembly enhancements, so are not generally applicable but still worthy of note. I can dig up links, if you want.
2.) Maybe add at least a pointer to http://www.mersennewiki.org/index.ph...mial_Selection when you mention SNFS. You could also point to the phi tools that build SNFS polynomials for cyclotomic numbers that akruppa has posted to this forum. I can dig up links to that also if you want to add it. You probably don't want to touch SNFS polynomial generation in this guide, but links to resources that do cover it might be nice for the interested and more advance reader.

great job!
- ben.

[Jeff Gilchrist]

Originally Posted by jasonp

Sieving using the NFS line siever also will checkpoint if there is an orderly shutdown, and restarting is automatic in that case too.

Are you sure about that? I thought automatic restart was only if you were using one of the factLat.pl or factMsieve.pl scripts? Or do you mean if you pass the siever a polynomial file on the command line that doesn't have a starting Q value specified it will just start from the .last.sp0 or whatever file and continue on?

[Jeff Gilchrist]

Originally Posted by bsquared

Thanks for putting this together, I wish something like this existed when I was first learning to use the tools.

A couple comments:
1.) Some precompiled binaries exist for linux. The ones I'm thinking of are from frmky, and contain the amd64 assembly enhancements, so are not generally applicable but still worthy of note. I can dig up links, if you want.
2.) Maybe add at least a pointer to http://www.mersennewiki.org/index.ph...mial_Selection when you mention SNFS. You could also point to the phi tools that build SNFS polynomials for cyclotomic numbers that akruppa has posted to this forum. I can dig up links to that also if you want to add it. You probably don't want to touch SNFS polynomial generation in this guide, but links to resources that do cover it might be nice for the interested and more advance reader.

I wished the same thing so I figured I should make it easier for the next group of people starting. Thanks for the comments, if you could track down the URL for the linux binaries and the SNFS details I will add them.

I also forgot to add a link to the GGNFS message board if people want more info.

[jasonp]

Originally Posted by Jeff Gilchrist

Are you sure about that? I thought automatic restart was only if you were using one of the factLat.pl or factMsieve.pl scripts? Or do you mean if you pass the siever a polynomial file on the command line that doesn't have a starting Q value specified it will just start from the .last.sp0 or whatever file and continue on?

The NFS line sieve in msieve has its own restart mechanism; as part of an orderly shutdown it writes a *.line file with the first unprocessed sieve line. Restarting the sieving on the same input will read the *.line file if you have not explicitly provided a range of lines to sieve from.

This is completely independent of the perl scripts, and of what the GGNFS siever does (its restart mechanism is similar).

PS: Kudos for a very nice tutorial, it is sorely needed with the proliferation of versions of tools that we deal with now.

[smh]

Good to see a beginners guide on GNFS. A few comments:

On a windows environment i find it easier to use a pearl client like activeperl instead of cygwin. Especially if you want to run multiple instances of the siever over the network. Although i think you also need a windows version of a few *nix commands like 'cat', etc.

You might want to describe how to run multiple sievers with the perl scripts.

For smaller composites, i find it easier to just let the script handle poly selection with pol51. Msieve might give you a better poly, but most of the time i'm not around to do the conversion and start sieving. Maybe factmsieve should have an option to do poly selection using msieve?

What about special numbers? It might seem obvious to experienced users, but an example would probably be nice for a beginner.

[Jeff Gilchrist]

Originally Posted by smh

On a windows environment i find it easier to use a pearl client like activeperl instead of cygwin. Especially if you want to run multiple instances of the siever over the network. Although i think you also need a windows version of a few *nix commands like 'cat', etc.

It is for that reason I am suggesting cygwin since you get perl and all the UNIX commands you need in one shot. Trying to explain to people where to find those utils and make sure you have the right support DLLs for them is a lot more complicated, especially if you want to run the tools at low priority and have everything run smoothly with the perl scripts.

Originally Posted by smh

You might want to describe how to run multiple sievers with the perl scripts.

I thought I did in the configuration section explaining how to set the number of CPUs to use for sieving. Or are you talking about something else?

Originally Posted by smh

For smaller composites, i find it easier to just let the script handle poly selection with pol51. Msieve might give you a better poly, but most of the time i'm not around to do the conversion and start sieving. Maybe factmsieve should have an option to do poly selection using msieve?

It might be easier, I have seen pol51 give some strange errors and just refuse to work on some numbers so I figured it was safer for beginners just to use msieve. Jason was changing the polynomial code for msieve 1.40 after which I was going to try and tackle changing factMsieve.pl to use msieve for polynomial selection instead of pol51 then it will be automatic and fast.

Originally Posted by smh

What about special numbers? It might seem obvious to experienced users, but an example would probably be nice for a beginner.

Maybe in the future. I'm still trying to fully understand the ins and outs myself so I'm not confident enough that I could do it properly.

Thanks for the comments,
Jeff.

[smh]

Originally Posted by smh

You might want to describe how to run multiple sievers with the perl scripts.

Setting num_cpu in the script only determines the number of threads msieve uses to solve the matrix.

[tmorrow]

Nice work Jeff, very useful for us beginners. I've been using the GGNFS - MSIEVE combo for many smallish factorisations. Up to this point I've just created a example.n file containing the number to be factored and let factMsieve.pl do the rest. Your instructions suggest generating a polynomial in step 3 using msieve to create example.poly file (after reformat). Why is this approach better? Are better polynomials produced by msieve?

For those lazy (like me :-) the conversion from msieve.fb to example.poly can be automated with sed/awk or with perl scripts, e.g. in your webpage example:

echo "type: gnfs" | sed '
s/^N/n:/
s/^SKEW/skew:/
s/^R\(.\)/Y\1:/
s/^A\(.\)/c\1:/
' msieve.fb - | awk '
{
  if ($1=="skew:")
    skew=$0
  else if ($1=="type:")
    type=$0
  else
    print
}
END {
  print skew
  print type
}
' > example.poly

[jasonp]

Msieve will take longer than pol5 to produce a polynomial, but at least for small to moderate size inputs the polynomial produced by msieve tends to make the sieving complete in 10-20% less time.

When the input is larger (~140 digits and up) I don't trust either the polynomial selection parameters or the optimization in msieve to find a better polynomial all the time; it has done so, but I see a lot of work needed for larger jobs.