GGNFS under SuSe cluster

[VolMike]

I am going to use cluster for factorization problems. It is under SuSe Linux Enterprise Server 10, with mvapich2-0.9.8 ,
gcc 4.1.0, Intel FORTRAN compiler 9.1, Intel C/C++ compiler 9.1,
and some extra software : Blas2, NAG Parallel Library
installed.
What steps should I do to implement GGNFS on this cluster (and to make the most productivity)?

[VolMike]

Well,did somebody use GGNFS on linux cluster?

[jasonp]

Originally Posted by VolMike

Well,did somebody use GGNFS on linux cluster?

I'm sure several people do, but I think nobody has responded because we're not sure where to start with answering your question. Why does it matter that you have all of these nice things installed on your cluster if GGNFS does not use them, and is only known to work with gcc? Msieve may be able to work with the Intel tools, but nobody has tried that I know of. That you're asking a question this basic means that you haven't done any homework before asking for advice.

Grab the latest SVN revision of GGNFS from sourceforge, build with gcc, try running a few small jobs yourself. Factoring RSA100 is a good start. The perl script can run multiple sieve jobs in parallel, try using that after you know that everything works correctly on a single machine

[VolMike]

Originally Posted by jasonp

I'm sure several people do, but I think nobody has responded because we're not sure where to start with answering your question. Why does it matter that you have all of these nice things installed on your cluster if GGNFS does not use them, and is only known to work with gcc? Msieve may be able to work with the Intel tools, but nobody has tried that I know of. That you're asking a question this basic means that you haven't done any homework before asking for advice.

Grab the latest SVN revision of GGNFS from sourceforge, build with gcc, try running a few small jobs yourself. Factoring RSA100 is a good start. The perl script can run multiple sieve jobs in parallel, try using that after you know that everything works correctly on a single machine

Well, I used both Msieve and GGNFS (with perl script) on single PC under Windows platform and factorize some numbers (f.e. some results are shown at http://mersenneforum.org/showthread.php?t=5722&page=5).
But I really don't know how (if it's needed) to improve the sources to be worked under cluster. That's why I'am interesting if sources should be changed to get more productivity on cluster, or I can just simply compile original sources and result program automaticly starts work with all 32 processors.

[jasonp]

Originally Posted by VolMike

But I really don't know how (if it's needed) to improve the sources to be worked under cluster. That's why I'am interesting if sources should be changed to get more productivity on cluster, or I can just simply compile original sources and result program automaticly starts work with all 32 processors.

You can start the perl script on each cluster node with additional arguments, in order to perform distributed sieving. You will need to combine the generated output files manually before running msieve.

Sieving and polynomial selection are both essentially completely parallel, so you don't need a cluster with fast interconnect to get high efficiency out of many machines. A cluster can be useful for the linear algebra phase of the postprocessing, but the code changes needed to implement cluster-aware linear algebra would be quite complex. Even if they were made, it only makes sense to use your cluster on very large factorizations, which would probably tie up the cluster for longer than you are allowed.

[Wacky]

Originally Posted by jasonp

You can start the perl script on each cluster node with additional arguments, in order to perform distributed sieving. You will need to combine the generated output files manually before running msieve.

Sieving and polynomial selection are both essentially completely parallel, so you don't need a cluster with fast interconnect to get high efficiency out of many machines. A cluster can be useful for the linear algebra phase of the postprocessing, but the code changes needed to implement cluster-aware linear algebra would be quite complex. Even if they were made, it only makes sense to use your cluster on very large factorizations, which would probably tie up the cluster for longer than you are allowed.

I will point out that, as Jason notes, the sieving is a highly parallel, but decoupled, operation.

On the other hand, the linear algebra is highly coupled. The ability of the many processors to synchronize and exchange information becomes the limiting factor.

[Jay]

There has been a factMsieve.pl Perl script posted which allows the (highly) automated usage of MSieve alongside the Lattice sievers. It replaces the factLat.pl script.

You may want to check the posts on the GGNFS Yahoo group regarding this.

http://tech.groups.yahoo.com/group/ggnfs/message/2195

Essentially it brings together the best of GGNFS and MSieve, resulting in a nice marriage.

Note that it is still a bit rough around the edges, but it does work. So far I've used it to crack two composites.

[bsquared]

Originally Posted by Jay

Essentially it brings together the best of GGNFS and MSieve, resulting in a nice marriage.

I modified factLat.pl and wrote a script to do that quite some time ago, because I could never get GGNFS's sqrt to work on a minGW build. If there is interest, I could post it here, although I'm not sure my factLat hacks will work with everyone's systems.

- ben.