Reservations for x^y+y^x - Page 3

pxp · 2020-10-01, 21:15

Quote:

Originally Posted by pxp

I am seeing much earlier ETC dates on restarted interruptions.

Unfortunately, this fix is no longer working well on my now much larger input files. The interruptions (after about a day of sieving) run into an error that has one or more workers not responding within a ten-minute default and xyyxsieve aborts the writing of the final output file. The existing output file is smaller than the input file but I have been afraid to use it because I fear that it has been contaminated by that final aborted write attempt.

One of the problems here was that I was not letting it run long enough. The existing output file might have had "// Sieved to 5" which confused me because the run was at p=600000+. A couple of times I even had a "// Sieved to 0" which strikes me as nonsensical. Clearly this is all about when exactly the program chooses to write an output and how long it actually takes to write that output. And I understand now that it is the writing of that output for very large files that is driving the ETC dates unrealistically into the distant future.

An unfinished example is a run on interval #18's almost 28 million term input file (406 MB) started on September 26:

Code:

./xyyxsieve -i18.ini -o18.txt -W4 -p3 -P1e7

which, as I write, is at:

Code:

p=4798627, 2.361 p/sec, 24100965 factors found at 0.000 f/sec, 48.0% done. ETC 2020-10-07 07:04

The current output file is sieved to 1481 and is down to 57 MB. When I get to the end I think that I will still run into this ten-minute unresponsive-worker abort of the final output update but the existing output file at that time should be sieved to a large enough number to be usable. Should I worry that such a file may have been corrupted by that final unsuccessful write attempt? What if the aborted file-write caused the file that is there at the finish to be missing a whole bunch of terms? That would directly impact the possibility of my not finding all PRPs in that interval.

rogue · 2020-10-01, 22:46

The program will update the output file once every 60 minutes. The "Sieved to" value in that file is the largest prime that the program can conclusively state has been successfully sieved to with no gaps.

My recommendation is to sieve with a single thread to 1e4 or 1e5 to knock out most of the candidates. Then start again on the output file with multiple threads. You should also consider using -w1e4 to reduce the number of primes per check. This also means that ^C will terminate the program faster. It might still take many minutes due to the number of terms, but it will terminate.

The ETC dates are far into the future because the time to sieve the first chunk takes a really long time and the program extrapolates based upon that. The ETC will come closer to the present as the number of terms is reduced. That is why I suggest sieving to 1e4 or 1e5 with a single worker before running again with multiple threads.

pxp · 2020-10-03, 19:04

Quote:

Originally Posted by rogue

You should also consider using -w1e4 to reduce the number of primes per check. This also means that ^C will terminate the program faster. It might still take many minutes due to the number of terms, but it will terminate.

According to the xyyxsieve help file the default for -w is 1e4, so that is what I have (unknowingly) been using. Trying 1e3 is "out of range". What is the point of larger -w arguments? I will guess faster processing but perhaps at the expense of RAM. That is not for me an issue as I have 32 GB on all of my machines and I have not found either xyyxsieve or pfgw take any advantage of it.

pxp · 2020-10-03, 21:19

I tried:

Code:

mm4:rogue1 pxp$ ./xyyxsieve -i19.ini -o19.txt -w1e4 -p3 -P1e4
xyyxsieve v1.5, a program to find factors numbers of the form x^y+y^x
Sieve started: 3 < p < 1e4 with 17782120 terms (27440 <= x <= 425055, 2 <= y <= 28701) (expecting 15661063 factors)
  p=0, 0.000 p/sec, 3187205 factors found at 176.5 f/sec, 0.0% done.            
1 workers didn't stop after 10 minutes

So the single thread and -P1e4 made no difference.

pxp · 2020-10-03, 21:43

What I am going to do is multi-thread these runs to -P1e9 and after several days just rescue (move elsewhere) whatever output file is there and then kill the run. Also, in the future, I think I will do some small sieving in Mathematica prior to creating the ABC files. I had been reluctant to do this initially because I figured xyyxsieve would be just so much faster.

rogue · 2020-10-03, 22:01

With so many candidates, it will take a long time to get to 1e4. Yes -w must be at least 1e4.

The version of xyyxsieve that I gave to you won't benefit when adding more candidates. You might as well divide the input to smaller files and sieve them.

pxp · 2020-10-04, 23:10

Quote:

Originally Posted by rogue

The program will update the output file once every 60 minutes.

I am looking at this now. One of my runs was updated at 2:48 pm today: sieved to 11 (149.2 MB); then at 3:49 pm: sieved to 13 (142.2 MB); then at 4:50 pm: sieved to 13 (140.3 MB); then at 5:51 pm: sieved to 17 (136.3 MB); then at 6:52 pm: sieved to 17 (132.7 MB).

At this point xyyxsieve has:

Code:

Sieve started: 3 < p < 1e9 with 22133879 terms (30455 <= x <= 476849, 2 <= y <= 31873) (expecting 20960485 factors)
p=243011, 0.098 p/sec, 13031312 factors found at 5.358 f/sec, 0.0% done. ETC 2032-08-29 10:30

Note that p is past 243000. It has already found 13 million of an expected 21 million factors. Checking the 6:52 pm file, has it just over 9 million terms in it which is exactly in line with the 13 million factors found. Most troubling are the sieved-to numbers: 11, 13, 13, 17, 17. All the while p has been at 200000+. My guess is that the sieved-to numbers are incorrect. Should they not be where p is at?

rogue · 2020-10-05, 02:11

Quote:

Originally Posted by pxp

Most troubling are the sieved-to numbers: 11, 13, 13, 17, 17. All the while p has been at 200000+. My guess is that the sieved-to numbers are incorrect. Should they not be where p is at?

I don't understand.

pxp · 2020-10-05, 16:22

When p=3 and one has divided all initial candidates by 3, one has sieved to 3. When p=5 and one has divided all remaining candidates by 5, one has sieved to 5. When p=7 and one has divided all remaining candidates by 7, one has sieved to 7. Is that not correct or the way the program works?

rogue · 2020-10-05, 18:34

Quote:

Originally Posted by pxp

When p=3 and one has divided all initial candidates by 3, one has sieved to 3. When p=5 and one has divided all remaining candidates by 5, one has sieved to 5. When p=7 and one has divided all remaining candidates by 7, one has sieved to 7. Is that not correct or the way the program works?

That is correct. I did not understand this:

Quote:

Most troubling are the sieved-to numbers: 11, 13, 13, 17, 17. All the while p has been at 200000+. My guess is that the sieved-to numbers are incorrect. Should they not be where p is at?

Are you suggesting that many p were skipped (in other words the value for "sieved to" is wrong)? Are you suggesting that some factors were missed for p that were tested? Is there some other issue? I just don't understand your comments.

pxp · 2020-10-06, 20:24

As per my examples, when p = n has been reached, I expect the output file to be sieved to n (and you have said that my expectation is correct). Instead, the program is saying p ~ 200000 but the output file is not sieved to ~200000 but, rather, to 17. An hour later, p (supposedly, if the program value is to be believed) has advanced by thousands but the new output sieved-to number is still 17. There is no correspondence, so yes, either the sieved-to-number is wrong or the "p=" number in the console is wrong. I can't fathom either possibility which is why I had rather expected you to say that my understanding of the correspondence was incorrect.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Reservations	ET_	Operazione Doppi Mersennes	495	2020-12-19 19:41
Reservations for b^(k*2^n); b = 3, k<750	lukerichards	And now for something completely different	13	2019-05-11 23:27
Reservations	kar_bon	Riesel Prime Data Collecting (k*2^n-1)	129	2016-09-05 09:23
Reservations?	R.D. Silverman	NFS@Home	15	2015-11-29 23:18
4-5M Reservations	paulunderwood	3*2^n-1 Search	15	2008-06-08 03:29

2020-10-01, 22:46	#24
rogue "Mark" Apr 2003 Between here and the 5×1,249 Posts	The program will update the output file once every 60 minutes. The "Sieved to" value in that file is the largest prime that the program can conclusively state has been successfully sieved to with no gaps. My recommendation is to sieve with a single thread to 1e4 or 1e5 to knock out most of the candidates. Then start again on the output file with multiple threads. You should also consider using -w1e4 to reduce the number of primes per check. This also means that ^C will terminate the program faster. It might still take many minutes due to the number of terms, but it will terminate. The ETC dates are far into the future because the time to sieve the first chunk takes a really long time and the program extrapolates based upon that. The ETC will come closer to the present as the number of terms is reduced. That is why I suggest sieving to 1e4 or 1e5 with a single worker before running again with multiple threads.

2020-10-03, 21:43	#27
pxp Sep 2010 Weston, Ontario 2⁶·3 Posts	What I am going to do is multi-thread these runs to -P1e9 and after several days just rescue (move elsewhere) whatever output file is there and then kill the run. Also, in the future, I think I will do some small sieving in Mathematica prior to creating the ABC files. I had been reluctant to do this initially because I figured xyyxsieve would be just so much faster.

2020-10-03, 22:01	#28
rogue "Mark" Apr 2003 Between here and the 6245₁₀ Posts	With so many candidates, it will take a long time to get to 1e4. Yes -w must be at least 1e4. The version of xyyxsieve that I gave to you won't benefit when adding more candidates. You might as well divide the input to smaller files and sieve them.

2020-10-05, 16:22	#31
pxp Sep 2010 Weston, Ontario 2⁶×3 Posts	When p=3 and one has divided all initial candidates by 3, one has sieved to 3. When p=5 and one has divided all remaining candidates by 5, one has sieved to 5. When p=7 and one has divided all remaining candidates by 7, one has sieved to 7. Is that not correct or the way the program works?

2020-10-06, 20:24	#33
pxp Sep 2010 Weston, Ontario 2⁶·3 Posts	As per my examples, when p = n has been reached, I expect the output file to be sieved to n (and you have said that my expectation is correct). Instead, the program is saying p ~ 200000 but the output file is not sieved to ~200000 but, rather, to 17. An hour later, p (supposedly, if the program value is to be believed) has advanced by thousands but the new output sieved-to number is still 17. There is no correspondence, so yes, either the sieved-to-number is wrong or the "p=" number in the console is wrong. I can't fathom either possibility which is why I had rather expected you to say that my understanding of the correspondence was incorrect.