Saving interim residues of huge P-1?

[Andi47]

Is it possible to save interim residues of (huge) p-1 factorisation attempts with GMP-ECM - say every hour or so?

I am doing a p-1 with large B1 and B2 bounds now, and have the problem that windows has installed an update and now comes up with a massage like "an update has been installed and needs a reboot to complete installation. It will reboot in 5 minutes". When I leave the PC allone and fail to click "reboot later" I will definitly lose nearly 60 CPU hours of P-1 work....

[akruppa]

Sorry, GMP-ECM has no feature to save residues periodically yet. In 6.1, we catch signals and try to exit gracefully upon SIGTERM and SIGINT, including writing the residue of stage 1 done so far. It works nicely on unix-like OSs, but I don't know if it works at all under Windows/MinGW. Can you post the lines from config.h in your GMP-ECM 6.1 directory that contain the string "HAVE_SIGNAL"?

Alex

[fetofs]

Originally Posted by akruppa

Sorry, GMP-ECM has no feature to save residues periodically yet. In 6.1, we catch signals and try to exit gracefully upon SIGTERM and SIGINT, including writing the residue of stage 1 done so far. It works nicely on unix-like OSs, but I don't know if it works at all under Windows/MinGW. Can you post the lines from config.h in your GMP-ECM 6.1 directory that contain the string "HAVE_SIGNAL"?

Alex

On my MSys GMP-ECM directory, it's:

/* Define to 1 if you have the `signal' function. */
#define HAVE_SIGNAL 1

[Andi47]

Originally Posted by akruppa

Sorry, GMP-ECM has no feature to save residues periodically yet. In 6.1, we catch signals and try to exit gracefully upon SIGTERM and SIGINT, including writing the residue of stage 1 done so far. It works nicely on unix-like OSs, but I don't know if it works at all under Windows/MinGW. Can you post the lines from config.h in your GMP-ECM 6.1 directory that contain the string "HAVE_SIGNAL"?

Alex

Here they are, there are two of them:

/* Define to 1 if you have the `signal' function. */
#define HAVE_SIGNAL 1

/* Define to 1 if you have the <signal.h> header file. */
#define HAVE_SIGNAL_H 1

As for my huge p-1 attempt: I had to restart from scratch; now I increase B1 stepwise using -save and -resume options. This is a bit more time consuming, but I won't lose *everything* again if my computer crashes or desides to reboot for windows updates.

One of these steps crashed halfway through stage 2, I guess it was out of memory. The save file was empty. Why doesn't it write the residue to the save file just after finishing stage 1?

One more strange observation:
The README file suggests to resume using a line like this:

$ ./ecm -resume toto 1e7

If I do this (adding the statement >outputfile.out), the output file looks like this

GMP-ECM 6.1 [powered by GMP 4.2.1] [ECM]
Resuming P-1 residue saved by Andreas@ANDREAS with GMP-ECM 6.1 on Thu Jun 15 12:00:42 2006 
Input number is 2^1061-1 (320 digits)
Using B1=1000000000-3000000000, B2=3181298620516, polynomial x^120
Step 1 took 8718250ms

[ECM]? So what did it do actually? ECM or P-1?

If I use this imput line

$ ./ecm -resume toto -pm1 1e7 >outputfile.out

the outputfile starts like this:

GMP-ECM 6.1 [powered by GMP 4.2.1] [P-1]

so NOW it looks like p-1...

[ColdFury]

It works nicely on unix-like OSs, but I don't know if it works at all under Windows/MinGW.

Windows supports some signals, including SIGTERM I believe.

[akruppa]

Originally Posted by Andi47

Here they are, there are two of them:

/* Define to 1 if you have the `signal' function. */
#define HAVE_SIGNAL 1

/* Define to 1 if you have the <signal.h> header file. */
#define HAVE_SIGNAL_H 1

Oh good, so GMP-ECM will register the signal handler. Can you try interrupting GMP-ECM in stage 1 with the kill command? It should say "Interrupted at prime xxx" and exit.

Originally Posted by Andi47

As for my huge p-1 attempt: I had to restart from scratch; now I increase B1 stepwise using -save and -resume options. This is a bit more time consuming, but I won't lose *everything* again if my computer crashes or desides to reboot for windows updates.

That's what I do when I do a very long P-1 run. Not the most elegant solution, but works.

Originally Posted by Andi47

One of these steps crashed halfway through stage 2, I guess it was out of memory. The save file was empty. Why doesn't it write the residue to the save file just after finishing stage 1?

We don't know whether we are to write a save file line after stage 1. Stage 2 might find a factor and complete the factorisation. Or it might find a factor and leave a composite cofactor, but in this case the new cofactor should go in the save file. Another reason is that at the moment, all the save/resume logic is in the main module, not in the ecm/p-1/p+1 modules. At least the code for resuming needs to be, as main() makes the choice which factoring method to call. I've thought about moving the save code into the respective method's module. But I think it might be better to write checkpoints in addition to the regular save files, where the checkpoint is written periodically during and at the end of stage 1.

Originally Posted by Andi47

[ECM]? So what did it do actually? ECM or P-1?

The info stored in the save files overrides the method given on the command line (default ecm, if -pp1 and -pm1 are not given). The "Resuming P-1 residue" is what matters here. The "[ECM]" text is a bit confusing in this case, maybe we should just remove that.

Alex

[Andi47]

Originally Posted by akruppa

Oh good, so GMP-ECM will register the signal handler. Can you try interrupting GMP-ECM in stage 1 with the kill command? It should say "Interrupted at prime xxx" and exit.

Where do I enter the command, and what's the exact line?

We don't know whether we are to write a save file line after stage 1. Stage 2 might find a factor and complete the factorisation. Or it might find a factor and leave a composite cofactor, but in this case the new cofactor should go in the save file. Another reason is that at the moment, all the save/resume logic is in the main module, not in the ecm/p-1/p+1 modules. At least the code for resuming needs to be, as main() makes the choice which factoring method to call. I've thought about moving the save code into the respective method's module. But I think it might be better to write checkpoints in addition to the regular save files, where the checkpoint is written periodically during and at the end of stage 1.

As I have seen, it would be nice to have a save file just at the end of Stage 1 - in case the program crashes at stage two. Otherwise the time of stage 1 AND two are lost. IF a factor is found, the save file can still be modified, overwritten or deleted.

See it like this: Writing the save file consumes a split second of CPU-time. A crash halfway through stage 2 (out of memory, or something?) whithout having a savefile may loose a couple of hours of cpu-time.

As for the checkpoints: I agree - no more 'ing because of windoze updates forcing a reboot after 60 hours of unsaved p-1 halfway through stage 1...

The info stored in the save files overrides the method given on the command line (default ecm, if -pp1 and -pm1 are not given). The "Resuming P-1 residue" is what matters here. The "[ECM]" text is a bit confusing in this case, maybe we should just remove that.

Or move it to the ecm, p-1 and p+1 -modules: The main module could write a line like

GMP-ECM 6.1 [powered by GMP 4.2.1]

and the submodule could add the [ECM] or [P-1] or [P+1].

(edit: typos)

[smh]

Regarding Windows Update, why don't you download the updates automatically but choose to install them when it suits you?

I think if you open an application like wordpad, type a few characters but NOT save the document, windows can't reboot. It will stop at the save document dialog box.

It might have closed gmp-ecm before that though.

[Andi47]

GMP-ECM 6.1 [powered by GMP 4.2.1] [P-1]
Resuming P-1 residue saved by Andreas@ANDREAS with GMP-ECM 6.1 on Thu Jun 15 12:00:42 2006 
Input number is 2^1061-1 (320 digits)
Using B1=1000000000-5000000000, B2=705032704-6357399348316, polynomial x^120
Step 1 took 17439172ms
Step 2 took 56984ms

Savefile is empty...

[akruppa]

Originally Posted by Andi47

Where do I enter the command, and what's the exact line?

At the MinGW bash command prompt. Try

$ ecm 1e9 1 <somecomposite.txt &
[1] pid
<GMP-ECM output>
$ kill pid

The number after [1] is the process id of the GMP-ECM process. You need to give it to the kill command to specify which process to kill. Note: if there's no $ prompt after the GMP-ECM output, just hit enter once.

Also, can you please run GMP-ECM as above, but instead using the kill command terminate the process in the Windows task manager? Does it print the "Interrupted at prime ..." message then?

Originally Posted by Andi47

Savefile is empty...

What exact command line did you use for this run? Did you specify both -resume and -save?

Alex

[Andi47]

Originally Posted by akruppa

At the MinGW bash command prompt. Try

$ ecm 1e9 1 <somecomposite.txt &
[1] pid
<GMP-ECM output>
$ kill pid

The number after [1] is the process id of the GMP-ECM process. You need to give it to the kill command to specify which process to kill. Note: if there's no $ prompt after the GMP-ECM output, just hit enter once.

The process dies without any "interrupted at prime..." message.

Also, can you please run GMP-ECM as above, but instead using the kill command terminate the process in the Windows task manager? Does it print the "Interrupted at prime ..." message then?

I did this after trying the kill command and get this:

$ecm 1e9 <1701-c205 &
[2] 1080
[1]   Terminated           ecm 1e9 <1701-c205

What exact command line did you use for this run? Did you specify both -resume and -save?

I typed

ecm -save m1061_5e9.sav -resume m1601_1e9.sav -pm1 -mpzmod -maxmem 1600 -treefile /var/tmp/ecmtree 5e9 >m1601_5e9.out

My savefile which I resumed from (m1601_1e9.sav) reads

METHOD=P-1; B1=1000000000; N=2^1061-1; X=0x18754a43f3406f17854426c70fcb5066cd33b9a9b8317e965f7f9dabe3d8ca3527b32ab0f5633fc4c257db1e969dbf8a4d91a774ad7e728f94af6ea8c110a5b086baa319ba7cb044ca8e20fdfae80db0e7781286bac6143cec25f2c687cb6f0aafe32df371e50fa05ee6641b522f4877da01e14bce95c69741b36b0fd8eff307fe9fa700aa; CHECKSUM=653811958; PROGRAM=GMP-ECM 6.1; X0=0xd3acfc5e; WHO=Andreas@ANDREAS; TIME=Thu Jun 15 12:00:42 2006;

I did analogous lines with lower bounds before, but without the -maxmem and -treefile statements, and it worked with no problems.

Edit: Corrected my input line