mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Aliquot Sequences

Reply
 
Thread Tools
Old 2021-08-16, 02:10   #474
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3×372 Posts
Default

I am going to be delayed somewhat. If anyone would like, they should go ahead and take it.
EdH is offline   Reply With Quote
Old 2021-08-16, 05:37   #475
bur
 
bur's Avatar
 
Aug 2020
79*6581e-4;3*2539e-3

419 Posts
Default

I could do it on wednesday if it hasn't been taken until then.
bur is offline   Reply With Quote
Old 2021-08-16, 09:03   #476
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

2·383 Posts
Default

I'll start it today.
kruoli is offline   Reply With Quote
Old 2021-08-16, 12:59   #477
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3×372 Posts
Default

Quote:
Originally Posted by kruoli View Post
I'll start it today.
Excellent! Thanks!
EdH is offline   Reply With Quote
Old 2021-08-17, 21:39   #478
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

76610 Posts
Default

The C147 split as P49 and P99. The result is uploaded. I used a GIT version of CADO which ran into some problems.

I'll keep crunching the next lines until the next "blocker" comes.

@VBCurtis: I used your optimised param files, do you want the log file? Because of the crashes, I needed to restart two or three times, but maybe it is still helpful?
kruoli is offline   Reply With Quote
Old 2021-08-17, 21:43   #479
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

117158 Posts
Default

Which problems came up with CADO?

I have a few iterations at that size, but on my own hardware- other CPUs are useful for timing data, esp since all mine are 2014 or older.

If you have the timing summary for poly select time (size and root or a single number), sieving time, LA time, I'd like that.
Additionally useful info are matrix size (dimensions and/or total weight) and Q-range sieved.
VBCurtis is online now   Reply With Quote
Old 2021-08-18, 10:24   #480
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

2·383 Posts
Default

The first occured when it tried duplication removal for the first time:
Code:
PID5049 2021-08-17 05:59:23,222 Warning:Command: Process with PID 29069 finished with return code -6
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass: Program run on server failed with exit code -6
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass: Command line was: /home/oliver/cado-nfs/build/debian/filter/dup1 -prefix dup1.0 -out /tmp/cado.rund25tz/c145.dup1/ -n 1 -filelist /tmp/cado.rund25tz/c145.dup1.filelist.1 > /tmp/cado.rund25tz/c145.dup1.stdout.1 2> /tmp/cado.rund25tz/c145.dup1.stderr.1
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass: Stderr output (last 10 lines only) follow (stored in file /tmp/cado.rund25tz/c145.dup1.stderr.1):
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass: 	antebuffer set to /home/oliver/cado-nfs/build/debian/utils/antebuffer
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass: 	# Opening output file for slice 0 : /tmp/cado.rund25tz/c145.dup1//0/dup1.0.0000.gz
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass: 	# Opening output file for slice 1 : /tmp/cado.rund25tz/c145.dup1//1/dup1.0.0000.gz
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass: 	code BUG() : condition rc == 0 failed in split_iter_open_next_file at /home/oliver/cado-nfs/filter/dup1.c:111 -- Abort
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass:
After that, I pulled the GIT version freshly and recompiled it. I reran from the checkpoint, and got:
Code:
PID18021 2021-08-17 19:44:53,806 Debug:Command: Process with PID 18997 stderr: b'CLIENT ERROR: Download failed, URL error: HTTP Error 404: No work available\nCLIENT ERROR: Waiting 10.0 seconds before retrying (I have been waiting for 30.0 seconds)\nCLIENT ERROR: spin=6 is_wu=True blog=0\nCLIENT ERROR: Downloading https://localhost:39989/cgi-bin/getwu?clientid=localhost+2 to /tmp/cado.rund25tz/client/download/WU.localhost+21016031422 (cafile = /tmp/cado.rund25tz/client/download/server.def27840c5a9423b8c771bde86da71d02187a609.pem)\nCLIENT ERROR: Opened URL https://localhost:39989/cgi-bin/getwu?clientid=localhost+2 after 40.0 seconds wait\nCLIENT ERROR: /tmp/cado.rund25tz/client/download/c145.poly already exists, not downloading\nCLIENT ERROR: /tmp/cado.rund25tz/client/download/c145.roots1.gz already exists, not downloading\nCLIENT ERROR: Existing file /tmp/cado.rund25tz/client/download/las has wrong checksum ea390b288e3d5b3a06ce07f22007ecd4765e8b8d, workunit specified 433beb57bb5e560e226590db4c50fdb096f52446. Deleting file.\nCLIENT ERROR: Traceback (most recent call last):\nCLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 2491, in <module>\nCLIENT ERROR:     client_ok = client.process()\nCLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 2194, in process\nCLIENT ERROR:     self.workunit = downloader.get_wu_full()\nCLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 2029, in get_wu_full\nCLIENT ERROR:     self.get_files(workunit)\nCLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 1923, in get_files\nCLIENT ERROR:     mandatory_server=server)\nCLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 1866, in get_missing_file\nCLIENT ERROR:     os.remove(filename)\nCLIENT ERROR: FileNotFoundError: [Errno 2] No such file or directory: \'/tmp/cado.rund25tz/client/download/las\'\n'
PID18021 2021-08-17 19:44:53,806 Critical:Client Launcher: DEAD: client id localhost+2 (Host localhost, PID 5073)
PID18021 2021-08-17 19:44:53,806 Critical:Client Launcher: Stderr: CLIENT ERROR: Download failed, URL error: HTTP Error 404: No work available
CLIENT ERROR: Waiting 10.0 seconds before retrying (I have been waiting for 30.0 seconds)
CLIENT ERROR: spin=6 is_wu=True blog=0
CLIENT ERROR: Downloading https://localhost:39989/cgi-bin/getwu?clientid=localhost+2 to /tmp/cado.rund25tz/client/download/WU.localhost+21016031422 (cafile = /tmp/cado.rund25tz/client/download/server.def27840c5a9423b8c771bde86da71d02187a609.pem)
CLIENT ERROR: Opened URL https://localhost:39989/cgi-bin/getwu?clientid=localhost+2 after 40.0 seconds wait
CLIENT ERROR: /tmp/cado.rund25tz/client/download/c145.poly already exists, not downloading
CLIENT ERROR: /tmp/cado.rund25tz/client/download/c145.roots1.gz already exists, not downloading
CLIENT ERROR: Existing file /tmp/cado.rund25tz/client/download/las has wrong checksum ea390b288e3d5b3a06ce07f22007ecd4765e8b8d, workunit specified 433beb57bb5e560e226590db4c50fdb096f52446. Deleting file.
CLIENT ERROR: Traceback (most recent call last):
CLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 2491, in <module>
CLIENT ERROR:     client_ok = client.process()
CLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 2194, in process
CLIENT ERROR:     self.workunit = downloader.get_wu_full()
CLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 2029, in get_wu_full
CLIENT ERROR:     self.get_files(workunit)
CLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 1923, in get_files
CLIENT ERROR:     mandatory_server=server)
CLIENT ERROR:   File "/home/oliver/cado-nfs/build/debian/cado-nfs-client.py", line 1866, in get_missing_file
CLIENT ERROR:     os.remove(filename)
CLIENT ERROR: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/cado.rund25tz/client/download/las'
I simply used the newly generated checkpoint to rerun it again then and after that it went through. Because of that, I am not sure how "correct" the timing values etc. will be. I list them here nonetheless.

Code:
Polynomial Selection (size optimized): Total time: 24748.3
Polynomial Selection (root optimized): Total time: 2006.02
Polynomial Selection (root optimized): Rootsieve time: 2005.77
Polynomial Selection (root optimized): Finished, best polynomial has Murphy_E = 9.521e-07
Lattice Sieving: Total time: 653913s
Filtering - Merging: Merged matrix has 2204205 rows and total weight 287708721 (130.5 entries per row on average)
Linear Algebra: Total cpu/real time for bwc: 63865.1/5503.86
Linear Algebra: Aggregate statistics:
Linear Algebra: Krylov: CPU time 39776.79, WCT time 3383.25 (70000 iterations)
Linear Algebra: Lingen CPU time 1078.29, WCT time 34.93
Linear Algebra: Mksol: CPU time 21109.95,  WCT time 1923.11 (34000 iterations)
Regarding the Q-range, I found nothing in particular in the log file that stated a range. I guess you can derive that from the names of the work units? They started at 1000000 and finished with 13240000, if I saw that correctly.

There are some stderr files I could post, but for now I do not since the error resolved by restarting multiple times. Maybe I am going to use a release version again next time.
kruoli is offline   Reply With Quote
Old 2021-08-18, 12:13   #481
charybdis
 
charybdis's Avatar
 
Apr 2020

22216 Posts
Default

Quote:
Originally Posted by kruoli View Post
The first occured when it tried duplication removal for the first time:
Code:
...
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass: 	code BUG() : condition rc == 0 failed in split_iter_open_next_file at /home/oliver/cado-nfs/filter/dup1.c:111 -- Abort
PID5049 2021-08-17 05:59:23,227 Error:Filtering - Duplicate Removal, splitting pass:
After that, I pulled the GIT version freshly and recompiled it. I reran from the checkpoint, and got:
Code:
...CLIENT ERROR: Existing file /tmp/cado.rund25tz/client/download/las has wrong checksum ea390b288e3d5b3a06ce07f22007ecd4765e8b8d, workunit specified 433beb57bb5e560e226590db4c50fdb096f52446. Deleting file....
I simply used the newly generated checkpoint to rerun it again then and after that it went through. Because of that, I am not sure how "correct" the timing values etc. will be. I list them here nonetheless.
This is the same bug that was briefly discussed here. It turned out to cause a crash whenever rels_wanted was set above 2^26=67108864, and somehow went unnoticed for over 3 months. It has now been fixed, as you discovered. Unsurprisingly, it seems CADO throws a checksum error if you change version mid-job.

It would be useful if there was some record of which git releases are "semi-stable" in that they don't have any major bugs. Commit 8ab2eea7 from October last year would be a good bet: it fixed a fairly uncommon but very long-standing bug in the square root (see this thread). The latest release, commit c5b20eac, is hopefully semi-stable too. Both of these versions have a (diagnosed but as yet unfixed) bug that leads to relations with composite factors being produced very occasionally when using adjust_strategy=2, but you can get around this by either not using strategy 2 or by using msieve for postprocessing if you do encounter the bug.
charybdis is offline   Reply With Quote
Old 2021-08-18, 14:16   #482
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

2FE16 Posts
Default

Next on is a C199. I just finished t50 and t55 is underway, ETA three to four days.
kruoli is offline   Reply With Quote
Old 2021-08-18, 14:35   #483
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

117158 Posts
Default

Thanks for the timing info. I've run a few 143-145 jobs, but not a 147 (the largest that would use the C145 params). We've learned that a ratio of Q-max to Q-min above 10 results in a lot of duplicate relations. Your Q-max of 13.4M means that starting Q of 1.0M is too small. Perhaps you had an unlucky/bad poly, but your data calls for starting Q to be raised to at least 1.4M.

Your matrix solved in about 10% of the time you took to sieve. That suggests a bit less sieving would result in a net-speed increase, I think... did your job filter more than once? Is there a required-excess setting in the params file? If yes to both, reduce the required-excess setting (say, by 0.02). If it only filtered once, then reduce the target relations by 1M.

Some of these edits reduce the average time of a job of this size, while others reduce the best-case time. For a long time I judged my settings by the shortest job time on record for a size, but I've shifted to trying to reduce the time taken for "unlucky" jobs (say, by increasing starting Q).
VBCurtis is online now   Reply With Quote
Old 2021-08-18, 14:51   #484
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

2·383 Posts
Default

Yes, I had a lot of duplicates (Of 106068326 newly added relations 67468038 were unique (ratio 0.636081) was the first output). Yes, it filtered more than once. Yes, tasks.filter.required_excess = 0.10. I'll do as you suggested, thanks!

I am sorry for the second error I posted above. I should have gotten that by reading it thoroughly…
kruoli is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Primes in n-fibonacci sequence and n-step fibonacci sequence sweety439 And now for something completely different 17 2017-06-13 03:49
Team sieve #41: C165 from 3366:i2098 RichD Aliquot Sequences 36 2013-11-29 07:03
80M to 64 bits ... but not really reserved petrw1 Lone Mersenne Hunters 82 2010-01-11 01:57
What's the next in the sequence? roger Puzzles 16 2006-10-18 19:52
Sequence Citrix Puzzles 5 2005-09-14 23:33

All times are UTC. The time now is 06:20.


Wed Dec 1 06:20:53 UTC 2021 up 131 days, 49 mins, 1 user, load averages: 0.90, 0.98, 1.11

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.