mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > CADO-NFS

Reply
 
Thread Tools
Old 2021-07-25, 13:39   #1
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

178510 Posts
Default Re-generating the dups files

I inadvertently deleted the folder with the dups files, and I can't figure out how to generate them again (or get CADO to do it automatically). This causes an error after CADO has collected enough relations:

OSError: output file /tmp/cado.v9x5al17/c212_snfs215.dup1//1/dup1.0.0000.gz does not exist

How can I re-create this file or set of files?
wombatman is offline   Reply With Quote
Old 2021-07-25, 14:31   #2
charybdis
 
charybdis's Avatar
 
Apr 2020

2·3·7·13 Posts
Default

Re-create the directory /tmp/cado.v9x5al17/c212_snfs215.dup1, and make subdirectories named 0 and 1 inside it.
Re-run all of the dup1 and dup2 commands in the c212_snfs215.cmd file in order. If this runs smoothly then the 0 and 1 directories should contain some large files with names like dup1.0.0000.gz.
Now restart CADO.
charybdis is offline   Reply With Quote
Old 2021-07-25, 16:34   #3
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

3×5×7×17 Posts
Default

The first step:

Code:
/home/wombat/cado-nfs/build/Ben-PC/filter/dup1 -prefix dup1.2 -out /tmp/cado.v9x5al17/c212_snfs215.dup1/ -n 1 -filelist /tmp/cado.v9x5al17/c212_snfs215.dup1.filelist.3 > /tmp/cado.v9x5al17/c212_snfs215.dup1.stdout.3 2> /tmp/cado.v9x5al17/c212_snfs215.dup1.stderr.3
works fine.

This command:

Code:
/home/wombat/cado-nfs/build/Ben-PC/filter/dup2 -poly /tmp/cado.v9x5al17/c212_snfs215.poly -nrels 29153560 -renumber /tmp/cado.v9x5al17/c212_snfs215.renumber.gz /tmp/cado.v9x5al17/c212_snfs215.dup1//0/dup1.0.0000.gz /tmp/cado.v9x5al17/c212_snfs215.dup1//0/dup1.1.0000.gz /tmp/cado.v9x5al17/c212_snfs215.dup1//0/dup1.2.0000.gz > /tmp/cado.v9x5al17/c212_snfs215.dup2.slice0.stdout.3 2> /tmp/cado.v9x5al17/c212_snfs215.dup2.slice0.stderr.3
is aborting.

Last fiddled with by wombatman on 2021-07-25 at 16:34
wombatman is offline   Reply With Quote
Old 2021-07-25, 16:46   #4
charybdis
 
charybdis's Avatar
 
Apr 2020

2×3×7×13 Posts
Default

Quote:
Originally Posted by wombatman View Post
The first step:

Code:
/home/wombat/cado-nfs/build/Ben-PC/filter/dup1 -prefix dup1.2 -out /tmp/cado.v9x5al17/c212_snfs215.dup1/ -n 1 -filelist /tmp/cado.v9x5al17/c212_snfs215.dup1.filelist.3 > /tmp/cado.v9x5al17/c212_snfs215.dup1.stdout.3 2> /tmp/cado.v9x5al17/c212_snfs215.dup1.stderr.3
works fine.
This cannot have been the first dup1 command: the .3 endings on the files mean that it was the third. Try emptying the dup1 folder and trying again from the start; make sure you look back far enough in the .cmd file to find all the dup1 and dup2 commands.

Last fiddled with by charybdis on 2021-07-25 at 16:47
charybdis is offline   Reply With Quote
Old 2021-07-25, 17:09   #5
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

3×5×7×17 Posts
Default

You're right. It's not, but even running the first instances, it does the same thing. I notice that in the command line is a parameter "-nrels 24996721". Does this need to be updated to reference a split of the current number of relations (~120M)? Didn't work.

Edit: I do notice that there's no dup1.filelist as expected from the first command either and there's nothing in the stdout.1 output file either.

Last fiddled with by wombatman on 2021-07-25 at 17:50
wombatman is offline   Reply With Quote
Old 2021-07-25, 18:09   #6
charybdis
 
charybdis's Avatar
 
Apr 2020

2×3×7×13 Posts
Default

So you don't have a file c212_snfs215.dup1.filelist.1? That's odd, it obviously existed when the command ran originally, and it doesn't usually get deleted during the run.

If you definitely have lost the filelist and need to recreate it, use something like
Code:
grep -o "in .*gz" c212_snfs215.log | cut -c 5- >> c212_snfs215.dup1.filelist
and then cut it into chunks: everything up to the first appearance of "Reached target of xxxxxxxx relations" in the logfile is filelist.1, everything from there to the next "Reached target" (NOT including the contents of filelist.1!) is filelist.2, and so on.

The -nrels flag in the dup2 command line refers to the number of relations in one slice, i.e. roughly half the number of relations you had at the time the command was run.
charybdis is offline   Reply With Quote
Old 2021-07-25, 19:35   #7
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

110111110012 Posts
Default

Thanks to your help, I'm getting closer. After rebuilding the filelist, I was able to run the dup2 commands. When I try and start the primary python script, it fails again because it's looking for a file that doesn't exist (dup1.1.0000.gz). I checked both the dup1.filelist.1 file and the purge.filelist.1 files. The purge filelist had files that no longer exist in them, so I pared it down to only the two files (plus the freerels.gz) I had created: dup1.0.0000.gz in folders 0 and 1.

Is there another filelist that the main script is pulling from? I've search the /tmp/cado* directory and don't see anything else.

The grep command is also not collecting all of the completed workunits, missing about 20M q worth. I've confirmed via spot check that the missing workunits appear in the log file with the same format as the collected workunits.
wombatman is offline   Reply With Quote
Old 2021-07-25, 19:50   #8
charybdis
 
charybdis's Avatar
 
Apr 2020

2×3×7×13 Posts
Default

Quote:
Originally Posted by wombatman View Post
Thanks to your help, I'm getting closer. After rebuilding the filelist, I was able to run the dup2 commands. When I try and start the primary python script, it fails again because it's looking for a file that doesn't exist (dup1.1.0000.gz). I checked both the dup1.filelist.1 file and the purge.filelist.1 files. The purge filelist had files that no longer exist in them, so I pared it down to only the two files (plus the freerels.gz) I had created: dup1.0.0000.gz in folders 0 and 1.
dup1.1.0000.gz should be created by the second dup1 run - confusingly, CADO isn't consistent on whether to start its indexing at 0 or 1. Assuming you did run dup1 a second time, what are the contents of dup1.stdout.2 and dup1.stderr.2?

Quote:
The grep command is also not collecting all of the completed workunits, missing about 20M q worth. I've confirmed via spot check that the missing workunits appear in the log file with the same format as the collected workunits.
There's a chance I've been an idiot and somehow got the command wrong, so if anyone here spots a mistake, please don't hesitate to correct me. Otherwise you could attach the file and I'll see if I can get grep to work.
charybdis is offline   Reply With Quote
Old 2021-07-25, 20:03   #9
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

3·5·7·17 Posts
Default

There's a command line the .cmd file for the 2nd dup1 run, but I would need to generate a second filelist to recreate it. At this point, I would ideally just generate the dup1.0 files and go from there.

For the grep command, here's an example line that's not being picked up by it:
Code:
PID32137 2021-07-25 14:04:33,789 Info:Lattice Sieving: Found 24569 relations in '/tmp/cado.v9x5al17/c212_snfs215.upload/c212_snfs215.56430000-56440000.xaxayqid.gz', total is now 121252276/130000000

Last fiddled with by wombatman on 2021-07-25 at 20:03
wombatman is offline   Reply With Quote
Old 2021-07-25, 20:17   #10
charybdis
 
charybdis's Avatar
 
Apr 2020

54610 Posts
Default

Quote:
Originally Posted by wombatman View Post
There's a command line the .cmd file for the 2nd dup1 run, but I would need to generate a second filelist to recreate it. At this point, I would ideally just generate the dup1.0 files and go from there.
That's what I was trying to explain how to do here:

Quote:
Originally Posted by charybdis View Post
and then cut it into chunks: everything up to the first appearance of "Reached target of xxxxxxxx relations" in the logfile is filelist.1, everything from there to the next "Reached target" (NOT including the contents of filelist.1!) is filelist.2, and so on.
Of course you'll need grep to work properly in order to do this. I inserted your line into one of my old logfiles and grep found it successfully, so I'm mystified.
charybdis is offline   Reply With Quote
Old 2021-07-25, 20:21   #11
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

3·5·7·17 Posts
Default

Here's the output of the grep command as you gave it (without sending it to cut).
Attached Files
File Type: txt example_output.txt (403.8 KB, 44 views)
wombatman is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Posting log files or other text files Xyzzy Forum Feedback 3 2018-12-30 19:37
Generating Low-Weight Ks henryzz Riesel Prime Search 9 2012-04-09 21:36
Generating Random Primes davar55 Math 14 2011-02-20 16:06
Generating 2005 Wacky Puzzles 46 2005-06-05 22:19
Prime-generating polynomials Orgasmic Troll Math 28 2004-12-02 16:37

All times are UTC. The time now is 10:09.


Fri Dec 3 10:09:42 UTC 2021 up 133 days, 4:38, 0 users, load averages: 1.06, 1.15, 1.20

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.