mersenneforum.org Re-generating the dups files
 Register FAQ Search Today's Posts Mark Forums Read

 2021-07-25, 13:39 #1 wombatman I moo ablest echo power!     May 2013 23×223 Posts Re-generating the dups files I inadvertently deleted the folder with the dups files, and I can't figure out how to generate them again (or get CADO to do it automatically). This causes an error after CADO has collected enough relations: OSError: output file /tmp/cado.v9x5al17/c212_snfs215.dup1//1/dup1.0.0000.gz does not exist How can I re-create this file or set of files?
 2021-07-25, 14:31 #2 charybdis     Apr 2020 49810 Posts Re-create the directory /tmp/cado.v9x5al17/c212_snfs215.dup1, and make subdirectories named 0 and 1 inside it. Re-run all of the dup1 and dup2 commands in the c212_snfs215.cmd file in order. If this runs smoothly then the 0 and 1 directories should contain some large files with names like dup1.0.0000.gz. Now restart CADO.
 2021-07-25, 16:34 #3 wombatman I moo ablest echo power!     May 2013 23×223 Posts The first step: Code: /home/wombat/cado-nfs/build/Ben-PC/filter/dup1 -prefix dup1.2 -out /tmp/cado.v9x5al17/c212_snfs215.dup1/ -n 1 -filelist /tmp/cado.v9x5al17/c212_snfs215.dup1.filelist.3 > /tmp/cado.v9x5al17/c212_snfs215.dup1.stdout.3 2> /tmp/cado.v9x5al17/c212_snfs215.dup1.stderr.3 works fine. This command: Code: /home/wombat/cado-nfs/build/Ben-PC/filter/dup2 -poly /tmp/cado.v9x5al17/c212_snfs215.poly -nrels 29153560 -renumber /tmp/cado.v9x5al17/c212_snfs215.renumber.gz /tmp/cado.v9x5al17/c212_snfs215.dup1//0/dup1.0.0000.gz /tmp/cado.v9x5al17/c212_snfs215.dup1//0/dup1.1.0000.gz /tmp/cado.v9x5al17/c212_snfs215.dup1//0/dup1.2.0000.gz > /tmp/cado.v9x5al17/c212_snfs215.dup2.slice0.stdout.3 2> /tmp/cado.v9x5al17/c212_snfs215.dup2.slice0.stderr.3 is aborting. Last fiddled with by wombatman on 2021-07-25 at 16:34
2021-07-25, 16:46   #4
charybdis

Apr 2020

2·3·83 Posts

Quote:
 Originally Posted by wombatman The first step: Code: /home/wombat/cado-nfs/build/Ben-PC/filter/dup1 -prefix dup1.2 -out /tmp/cado.v9x5al17/c212_snfs215.dup1/ -n 1 -filelist /tmp/cado.v9x5al17/c212_snfs215.dup1.filelist.3 > /tmp/cado.v9x5al17/c212_snfs215.dup1.stdout.3 2> /tmp/cado.v9x5al17/c212_snfs215.dup1.stderr.3 works fine.
This cannot have been the first dup1 command: the .3 endings on the files mean that it was the third. Try emptying the dup1 folder and trying again from the start; make sure you look back far enough in the .cmd file to find all the dup1 and dup2 commands.

Last fiddled with by charybdis on 2021-07-25 at 16:47

 2021-07-25, 17:09 #5 wombatman I moo ablest echo power!     May 2013 33708 Posts You're right. It's not, but even running the first instances, it does the same thing. I notice that in the command line is a parameter "-nrels 24996721". Does this need to be updated to reference a split of the current number of relations (~120M)? Didn't work. Edit: I do notice that there's no dup1.filelist as expected from the first command either and there's nothing in the stdout.1 output file either. Last fiddled with by wombatman on 2021-07-25 at 17:50
 2021-07-25, 18:09 #6 charybdis     Apr 2020 7628 Posts So you don't have a file c212_snfs215.dup1.filelist.1? That's odd, it obviously existed when the command ran originally, and it doesn't usually get deleted during the run. If you definitely have lost the filelist and need to recreate it, use something like Code: grep -o "in .*gz" c212_snfs215.log | cut -c 5- >> c212_snfs215.dup1.filelist and then cut it into chunks: everything up to the first appearance of "Reached target of xxxxxxxx relations" in the logfile is filelist.1, everything from there to the next "Reached target" (NOT including the contents of filelist.1!) is filelist.2, and so on. The -nrels flag in the dup2 command line refers to the number of relations in one slice, i.e. roughly half the number of relations you had at the time the command was run.
 2021-07-25, 19:35 #7 wombatman I moo ablest echo power!     May 2013 23·223 Posts Thanks to your help, I'm getting closer. After rebuilding the filelist, I was able to run the dup2 commands. When I try and start the primary python script, it fails again because it's looking for a file that doesn't exist (dup1.1.0000.gz). I checked both the dup1.filelist.1 file and the purge.filelist.1 files. The purge filelist had files that no longer exist in them, so I pared it down to only the two files (plus the freerels.gz) I had created: dup1.0.0000.gz in folders 0 and 1. Is there another filelist that the main script is pulling from? I've search the /tmp/cado* directory and don't see anything else. The grep command is also not collecting all of the completed workunits, missing about 20M q worth. I've confirmed via spot check that the missing workunits appear in the log file with the same format as the collected workunits.
2021-07-25, 19:50   #8
charybdis

Apr 2020

1111100102 Posts

Quote:
 Originally Posted by wombatman Thanks to your help, I'm getting closer. After rebuilding the filelist, I was able to run the dup2 commands. When I try and start the primary python script, it fails again because it's looking for a file that doesn't exist (dup1.1.0000.gz). I checked both the dup1.filelist.1 file and the purge.filelist.1 files. The purge filelist had files that no longer exist in them, so I pared it down to only the two files (plus the freerels.gz) I had created: dup1.0.0000.gz in folders 0 and 1.
dup1.1.0000.gz should be created by the second dup1 run - confusingly, CADO isn't consistent on whether to start its indexing at 0 or 1. Assuming you did run dup1 a second time, what are the contents of dup1.stdout.2 and dup1.stderr.2?

Quote:
 The grep command is also not collecting all of the completed workunits, missing about 20M q worth. I've confirmed via spot check that the missing workunits appear in the log file with the same format as the collected workunits.
There's a chance I've been an idiot and somehow got the command wrong, so if anyone here spots a mistake, please don't hesitate to correct me. Otherwise you could attach the file and I'll see if I can get grep to work.

 2021-07-25, 20:03 #9 wombatman I moo ablest echo power!     May 2013 178410 Posts There's a command line the .cmd file for the 2nd dup1 run, but I would need to generate a second filelist to recreate it. At this point, I would ideally just generate the dup1.0 files and go from there. For the grep command, here's an example line that's not being picked up by it: Code: PID32137 2021-07-25 14:04:33,789 Info:Lattice Sieving: Found 24569 relations in '/tmp/cado.v9x5al17/c212_snfs215.upload/c212_snfs215.56430000-56440000.xaxayqid.gz', total is now 121252276/130000000 Last fiddled with by wombatman on 2021-07-25 at 20:03
2021-07-25, 20:17   #10
charybdis

Apr 2020

2·3·83 Posts

Quote:
 Originally Posted by wombatman There's a command line the .cmd file for the 2nd dup1 run, but I would need to generate a second filelist to recreate it. At this point, I would ideally just generate the dup1.0 files and go from there.
That's what I was trying to explain how to do here:

Quote:
 Originally Posted by charybdis and then cut it into chunks: everything up to the first appearance of "Reached target of xxxxxxxx relations" in the logfile is filelist.1, everything from there to the next "Reached target" (NOT including the contents of filelist.1!) is filelist.2, and so on.
Of course you'll need grep to work properly in order to do this. I inserted your line into one of my old logfiles and grep found it successfully, so I'm mystified.

2021-07-25, 20:21   #11
wombatman
I moo ablest echo power!

May 2013

33708 Posts

Here's the output of the grep command as you gave it (without sending it to cut).
Attached Files
 example_output.txt (403.8 KB, 34 views)

 Similar Threads Thread Thread Starter Forum Replies Last Post Xyzzy Forum Feedback 3 2018-12-30 19:37 henryzz Riesel Prime Search 9 2012-04-09 21:36 davar55 Math 14 2011-02-20 16:06 Wacky Puzzles 46 2005-06-05 22:19 Orgasmic Troll Math 28 2004-12-02 16:37

All times are UTC. The time now is 17:33.

Tue Oct 19 17:33:11 UTC 2021 up 88 days, 12:02, 0 users, load averages: 1.00, 1.31, 1.27