Feedback for new MPQS utility sought

[Greenbank]

Personally I'd go for the sockets approach. Example code is easy enough to find.

Just dedicate a seperate machine to be the central server (it doesn't have to be that powerful) and watch the relations come in.

Another way, but based along the same lines, would be to use libcurl to send the results over HTTP. You can then implement the server side as a cgi-bin and either stick the relations in a database or (using file locking but simplified as it will be on one machine) dump them into a flat file. flock() is pretty easy to call from Perl (if you were to use that for the cgi-bin).

If you go the libcurl route make sure you might need to add a bit of code to batch up the relation uploads, or at least disable persistent connections. You don't want the central webserver to have all its connections eaten up by the first n clients connecting (depends on how you've configured the webserver).

I'm looking at putting network comms (using libcurl) in proth_sieve so that Proth/Riesel sieving can be controlled by a central server. Clients would be able to reserve ranges via a website and then configure any number of clients to get small chunks of this range. If no range is assigned the server can give the clients small unassigned ranges. People with no direct (or intermittent) network access can just do their sieve range reservation/factor submission in the normal manual way.

Alex Kruppa wrote: Problems I can see: when starting factorisations on different numbers in the same directory. How would processes know that the msieve.dat file is currently in use and not a leftover from an old factorisation? Also, if processes are designated to only sieving (i.e. post-processing inhibited) it should be more efficient to not let them count cycles, but only dump relations into the msieve.dat file. However, if all processes that can post-process die, the only-sievers will keep sieving until the end of time (or the hard disk fills up, whichever comes first).

With each communication specify a unique string for the client (i.e. hostname or IP address) and the number being worked on. Bonus here is that if you keep track of the last time a client sent its update you can see if any nodes have stopped working. A local file cache (again based on hostname) would be a good idea until the relations have been uploaded.

[jasonp]

Originally Posted by akruppa

One thing the file locking idea has going for it is that processes can be added/removed dynamically without specifying a master process or anything. It's simple and convenient for the user if nothing else.

You're right, I should do that.

Also, if processes are designated to only sieving (i.e. post-processing inhibited) it should be more efficient to not let them count cycles, but only dump relations into the msieve.dat file.

Note that the current beta can turn cycle counting off.

jasonp

[jasonp]

The second beta is available:

www.boo.net/~jasonp/msieve102b2.tar.gz
www.boo.net/~jasonp/msieve102b2.exe

This includes more tuning for 90-110 digit factorizations, and fixes for a few long-standing problems people have repeatedly pointed out. See the changelog for details.

The speedup can be pretty substantial:

             v1.01          v1.02
C95          7 hrs          4.5 hrs
C100         17 hrs         14.5 hrs
C105         90 hrs         54 hrs

I'll probably make the full release when my test C90, the C105 and RSA110 on 2 CPUs all finish.

jasonp

[jasonp]

Now available at the usual place.

Final runtimes are as follows:

          1.01     1.02
C95        7 h    4.5 h
C100      17 h   14.5 h
C105      90 h     54 h
C110     ~384 h   114 h

Boy, what a difference better parameters can make!

I didn't get to the file locking stuff we discussed. Msieve writes a polynomial A value and then all the relations that use it, so the writing to the savefile is stateful. Even with proper file locking, one client could write an A value and another client can then write relations corresponding to another A value. Also, the tracking of cycles at present assumes that the pool of relations will not change unless the binary tracking the cycles actually adds relations. Modifying these assumptions isn't too difficult, but this release has already been delayed too long.

Happy factoring,

jasonp

[jasonp]

This release fixes a few small bugs and makes a few tiny cleanups. It is safe to use the new binary on a factorization already in progress.

Unless somebody notices a major bug, this is going to be the last msieve release for a long time. I don't understand nearly everything about NFS (in particular the square root is a complete mystery), but the parts I do understand have hit critical mass so that I'm not scared of it anymore.

Someday, maybe a year from now, msieve will have a GNFS module alongside the current MPQS module. The API should be essentially the same: a self-contained, fire-and-forget black box library for integer factorization. Don't be surprised if the first release is very crude, since even the component parts of an NFS implementation are little universes unto themselves.

The result will not be a substitute for GGNFS; by now man-years of work have gone into that package. As before, this will just be to teach myself how things work. The obsession with polishing it can come later :)

Happy factoring,
jasonp

[roger]

I've skimmed through this discussion, and (finally) have msieve.
I'm working on some C100+'s, and will be doing enough that I want something quick. I am also using Alpertron's ECM applet to find small factors quickly, to trim the size of the composite.
From before in this thread, was it said that GNFS or GGNFS is quicker than msieve? (I have a Pentium III 550 Mhz that I should recycle)
Also, as I'm no good with programming, is GNFS/GGNFS easy to use, as is msieve?
BTW : the composites have no form to them.

Thanks,

Roger

[jasonp]

Originally Posted by roger

I've skimmed through this discussion, and (finally) have msieve.
I'm working on some C100+'s, and will be doing enough that I want something quick. I am also using Alpertron's ECM applet to find small factors quickly, to trim the size of the composite.
From before in this thread, was it said that GNFS or GGNFS is quicker than msieve? (I have a Pentium III 550 Mhz that I should recycle)
Also, as I'm no good with programming, is GNFS/GGNFS easy to use, as is msieve?

The crossover point where GGNFS becomes faster used to be 94-97 digits on a Pentium4, but msieve has gotten considerably faster since those tests were made. On the other hand, the lattice siever in GGNFS is tuned for P3 machines by default. My guess is that the crossover point is around 100 digits, possibly a little bit higher. By 105 digits GGNFS will be faster.

GGNFS is a lot harder to set up. You will need windows binaries from the GGNFS sourceforge page, plus an implementation of perl (Active Perl is very nice for windows). I think you will also need a few files from the source code (you won't have to compile anything, but the binaries by themselves are not enough). Once that's all in place, running GGNFS is as easy as running msieve.

Note that if you want to finish this work fast and cheaply, your best bet is to go to your local small computer shop and spend about $200 to get an old motherboard, an athlon XP and some memory. Either utility will crunch through factorizations about 4-6 times faster than you're seeing now. You get a much nicer computer as a bonus.

jasonp

[roger]

Thanks, JasonP

In regard to the C105, I've given up with ECM as I've done over 2000 curves, with no success. I tried msieve, and it said it would take a couple days, maybe more than a week, so I switched to the NFS option. Maybe 5-8 days?
I'll see what I can accomplish with the GGNFS and perl scripts. Hopefully it will work without complications.
Good program!

Roger

[roger]

The C105 I was working on finished, but with errors in the relations. When the program tried to reduce them, or use them to factor, or something (will go back and look), it showed the relations (1->~5400) that had a problem, and that a step of the program failed.
I restarted the factorization from the last save point, and it did the same thing.
Is there some way I can regain those 5400 relations without having to restart the whole (1.86M relations) thing over again?

Thanks for all your assistance already!

Roger

[jasonp]

Originally Posted by roger

The C105 I was working on finished, but with errors in the relations. When the program tried to reduce them, or use them to factor, or something (will go back and look), it showed the relations (1->~5400) that had a problem, and that a step of the program failed.
I restarted the factorization from the last save point, and it did the same thing.
Is there some way I can regain those 5400 relations without having to restart the whole (1.86M relations) thing over again?

Thanks for all your assistance already!

Roger

5400 more relations is very little additional work; just run with '-r [number_of_relations_you_need_plus_10000]' or something similar and msieve will collect exactly 10000 more relations and proceed to the postprocessing. Alternately, start a new msieve run, let it go for 10000 relations or thereabouts, then stop it and append msieve.dat to your old msieve.dat, then restart from the old savefile. If that doesn't work, consider sending me the logfile from your restarted run as a private message. Be careful that your old savefile doesn't get overwritten; that's a big time investment.

Good luck,
jasonp

[roger]

I tried the -r command, but unfortunately the program still gives the same result. The output is that there are errors reading the first 5400+ relations. Perhaps Msieve found 5400 relations on the end rather than the corrupted ones at the beginning?
Is there a way around this, such as restarting the composite with a different savefile, then combining them later so that they can Finally give a result?
Thanks for everything,

Roger