mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > XYYXF Project

Reply
 
Thread Tools
Old 2020-10-01, 21:15   #23
pxp
 
pxp's Avatar
 
Sep 2010
Weston, Ontario

101101102 Posts
Default

Quote:
Originally Posted by pxp View Post
I am seeing much earlier ETC dates on restarted interruptions.
Unfortunately, this fix is no longer working well on my now much larger input files. The interruptions (after about a day of sieving) run into an error that has one or more workers not responding within a ten-minute default and xyyxsieve aborts the writing of the final output file. The existing output file is smaller than the input file but I have been afraid to use it because I fear that it has been contaminated by that final aborted write attempt.

One of the problems here was that I was not letting it run long enough. The existing output file might have had "// Sieved to 5" which confused me because the run was at p=600000+. A couple of times I even had a "// Sieved to 0" which strikes me as nonsensical. Clearly this is all about when exactly the program chooses to write an output and how long it actually takes to write that output. And I understand now that it is the writing of that output for very large files that is driving the ETC dates unrealistically into the distant future.

An unfinished example is a run on interval #18's almost 28 million term input file (406 MB) started on September 26:

Code:
./xyyxsieve -i18.ini -o18.txt -W4 -p3 -P1e7
which, as I write, is at:

Code:
p=4798627, 2.361 p/sec, 24100965 factors found at 0.000 f/sec, 48.0% done. ETC 2020-10-07 07:04
The current output file is sieved to 1481 and is down to 57 MB. When I get to the end I think that I will still run into this ten-minute unresponsive-worker abort of the final output update but the existing output file at that time should be sieved to a large enough number to be usable. Should I worry that such a file may have been corrupted by that final unsuccessful write attempt? What if the aborted file-write caused the file that is there at the finish to be missing a whole bunch of terms? That would directly impact the possibility of my not finding all PRPs in that interval.
pxp is offline   Reply With Quote
Old 2020-10-01, 22:46   #24
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

604510 Posts
Default

The program will update the output file once every 60 minutes. The "Sieved to" value in that file is the largest prime that the program can conclusively state has been successfully sieved to with no gaps.

My recommendation is to sieve with a single thread to 1e4 or 1e5 to knock out most of the candidates. Then start again on the output file with multiple threads. You should also consider using -w1e4 to reduce the number of primes per check. This also means that ^C will terminate the program faster. It might still take many minutes due to the number of terms, but it will terminate.

The ETC dates are far into the future because the time to sieve the first chunk takes a really long time and the program extrapolates based upon that. The ETC will come closer to the present as the number of terms is reduced. That is why I suggest sieving to 1e4 or 1e5 with a single worker before running again with multiple threads.
rogue is offline   Reply With Quote
Old 2020-10-03, 19:04   #25
pxp
 
pxp's Avatar
 
Sep 2010
Weston, Ontario

101101102 Posts
Default

Quote:
Originally Posted by rogue View Post
You should also consider using -w1e4 to reduce the number of primes per check. This also means that ^C will terminate the program faster. It might still take many minutes due to the number of terms, but it will terminate.
According to the xyyxsieve help file the default for -w is 1e4, so that is what I have (unknowingly) been using. Trying 1e3 is "out of range". What is the point of larger -w arguments? I will guess faster processing but perhaps at the expense of RAM. That is not for me an issue as I have 32 GB on all of my machines and I have not found either xyyxsieve or pfgw take any advantage of it.
pxp is offline   Reply With Quote
Old 2020-10-03, 21:19   #26
pxp
 
pxp's Avatar
 
Sep 2010
Weston, Ontario

101101102 Posts
Default

I tried:

Code:
mm4:rogue1 pxp$ ./xyyxsieve -i19.ini -o19.txt -w1e4 -p3 -P1e4
xyyxsieve v1.5, a program to find factors numbers of the form x^y+y^x
Sieve started: 3 < p < 1e4 with 17782120 terms (27440 <= x <= 425055, 2 <= y <= 28701) (expecting 15661063 factors)
  p=0, 0.000 p/sec, 3187205 factors found at 176.5 f/sec, 0.0% done.            
1 workers didn't stop after 10 minutes
So the single thread and -P1e4 made no difference.
pxp is offline   Reply With Quote
Old 2020-10-03, 21:43   #27
pxp
 
pxp's Avatar
 
Sep 2010
Weston, Ontario

2·7·13 Posts
Default

What I am going to do is multi-thread these runs to -P1e9 and after several days just rescue (move elsewhere) whatever output file is there and then kill the run. Also, in the future, I think I will do some small sieving in Mathematica prior to creating the ABC files. I had been reluctant to do this initially because I figured xyyxsieve would be just so much faster.
pxp is offline   Reply With Quote
Old 2020-10-03, 22:01   #28
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

3×5×13×31 Posts
Default

With so many candidates, it will take a long time to get to 1e4. Yes -w must be at least 1e4.

The version of xyyxsieve that I gave to you won't benefit when adding more candidates. You might as well divide the input to smaller files and sieve them.
rogue is offline   Reply With Quote
Old 2020-10-04, 23:10   #29
pxp
 
pxp's Avatar
 
Sep 2010
Weston, Ontario

2·7·13 Posts
Default

Quote:
Originally Posted by rogue View Post
The program will update the output file once every 60 minutes.
I am looking at this now. One of my runs was updated at 2:48 pm today: sieved to 11 (149.2 MB); then at 3:49 pm: sieved to 13 (142.2 MB); then at 4:50 pm: sieved to 13 (140.3 MB); then at 5:51 pm: sieved to 17 (136.3 MB); then at 6:52 pm: sieved to 17 (132.7 MB).

At this point xyyxsieve has:

Code:
Sieve started: 3 < p < 1e9 with 22133879 terms (30455 <= x <= 476849, 2 <= y <= 31873) (expecting 20960485 factors)
p=243011, 0.098 p/sec, 13031312 factors found at 5.358 f/sec, 0.0% done. ETC 2032-08-29 10:30
Note that p is past 243000. It has already found 13 million of an expected 21 million factors. Checking the 6:52 pm file, has it just over 9 million terms in it which is exactly in line with the 13 million factors found. Most troubling are the sieved-to numbers: 11, 13, 13, 17, 17. All the while p has been at 200000+. My guess is that the sieved-to numbers are incorrect. Should they not be where p is at?

Last fiddled with by pxp on 2020-10-04 at 23:15 Reason: factual correction
pxp is offline   Reply With Quote
Old 2020-10-05, 02:11   #30
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

3×5×13×31 Posts
Default

Quote:
Originally Posted by pxp View Post
Most troubling are the sieved-to numbers: 11, 13, 13, 17, 17. All the while p has been at 200000+. My guess is that the sieved-to numbers are incorrect. Should they not be where p is at?
I don't understand.
rogue is offline   Reply With Quote
Old 2020-10-05, 16:22   #31
pxp
 
pxp's Avatar
 
Sep 2010
Weston, Ontario

2·7·13 Posts
Default

When p=3 and one has divided all initial candidates by 3, one has sieved to 3. When p=5 and one has divided all remaining candidates by 5, one has sieved to 5. When p=7 and one has divided all remaining candidates by 7, one has sieved to 7. Is that not correct or the way the program works?
pxp is offline   Reply With Quote
Old 2020-10-05, 18:34   #32
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

10111100111012 Posts
Default

Quote:
Originally Posted by pxp View Post
When p=3 and one has divided all initial candidates by 3, one has sieved to 3. When p=5 and one has divided all remaining candidates by 5, one has sieved to 5. When p=7 and one has divided all remaining candidates by 7, one has sieved to 7. Is that not correct or the way the program works?
That is correct. I did not understand this:

Quote:
Most troubling are the sieved-to numbers: 11, 13, 13, 17, 17. All the while p has been at 200000+. My guess is that the sieved-to numbers are incorrect. Should they not be where p is at?
Are you suggesting that many p were skipped (in other words the value for "sieved to" is wrong)? Are you suggesting that some factors were missed for p that were tested? Is there some other issue? I just don't understand your comments.
rogue is offline   Reply With Quote
Old 2020-10-06, 20:24   #33
pxp
 
pxp's Avatar
 
Sep 2010
Weston, Ontario

2×7×13 Posts
Default

As per my examples, when p = n has been reached, I expect the output file to be sieved to n (and you have said that my expectation is correct). Instead, the program is saying p ~ 200000 but the output file is not sieved to ~200000 but, rather, to 17. An hour later, p (supposedly, if the program value is to be believed) has advanced by thousands but the new output sieved-to number is still 17. There is no correspondence, so yes, either the sieved-to-number is wrong or the "p=" number in the console is wrong. I can't fathom either possibility which is why I had rather expected you to say that my understanding of the correspondence was incorrect.
pxp is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Reservations ET_ Operazione Doppi Mersennes 494 2020-11-30 20:12
Reservations for b^(k*2^n); b = 3, k<750 lukerichards And now for something completely different 13 2019-05-11 23:27
Reservations kar_bon Riesel Prime Data Collecting (k*2^n-1) 129 2016-09-05 09:23
Reservations? R.D. Silverman NFS@Home 15 2015-11-29 23:18
4-5M Reservations paulunderwood 3*2^n-1 Search 15 2008-06-08 03:29

All times are UTC. The time now is 12:18.

Sat Dec 5 12:18:48 UTC 2020 up 2 days, 8:30, 0 users, load averages: 1.69, 1.63, 1.73

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.