PFGW 3.6.x problems on small primes - Page 2

rogue · 2014-07-12, 17:58

Quote:

Originally Posted by gd_barnes

To the best of my knowledge in my doublecheck, all of the problems were for n<=5 (maybe n<=6) but I doublechecked all n<=1000. But I can't say for sure that all problems were for n<=6 because I didn't do an automated line-by-line comparison. I just visually compared the two.

The comparison is a bit tricky because sometimes it would miss a prime for n=4 and then later find a prime for the same k for, say, n=500. Other times it would not find a prime at all for that k. Other times for another k it would have a "prime" for n=4 when in fact the "prime" was composite.

It's a difficult task to sort out all of the scenarios, which is why I don't see any alternative but to start the bases completely over.

Edit: Perhaps I can send you both the good and bad files and you can come up with an automated comparison so that we're more confident in what we're dealing with.

If you e-mail me the good and bad files I can compare to see which n are impacted. I would need to know which version of pfgw those were run with. That will help me narrow get the upper limit of the size of the numbers that are impacted. I would then run with 3.7.7 and compare to the pre-3.6.0 output you have. If they don't match then more investigation is needed.

Do we know which bases are impacted or do you not trust any of them?

gd_barnes · 2014-07-12, 18:15

Quote:

Originally Posted by rogue

Do we know which bases are impacted or do you not trust any of them?

No I don't know which bases are impacted outside of S66/S79. R51/S51/S66/R79/S79 and some ranges of R3/S3/R7/S7/R15/S15 likely have the greatest chance of being impacted. I suppose the best thing to do would be to look at bases that were started around the time that PFGW 3.6.x was released.

I will send you "good" and "bad" files for S66 & S79. There is a tricky issue on comparing them: My doublecheck is for n<=1000. The original (bad) files are for n<=2500. Here is how I think a comparison would need to be done:

good file scenario / bad file scenario
1. has prime / has same prime : OK
2. has prime / has different prime : bad
3. has prime / has no prime : bad
4. has no prime / has prime n>1000 : OK
5. has no prime / has prime n<=1000: bad
6. has no prime / has no prime : OK

So in evaluating "true" differences, you're looking for 2 situations:
1. The good file has a prime but the bad file has a different prime or no prime at all.
2. The good file has no prime but the bad file has a prime for n<=1000.

The files for S66 are huge since the conjecture is k>20M. I'll attempt to send the whole files in gmail but if it won't take due to size, I'll cut the S66 files down to an appropriate size. There are so many differences that even a relatively small sample of primes should be enough for you to get an idea of the problems.

The "bad" files that I will send you were run with PFGW 3.6.0. The "good" (doublecheck) files that I will send you were run with PFGW 3.3.6.

KEP · 2014-07-12, 18:54

I have to correct myself. The range tested by PFGW 3.6.3 is R3 11.0-12.0G and by tomorrow I'll have 11.00G to 11.01G send to you. The range will contain the primes for n<=25K.

rogue · 2014-07-12, 19:04

Quote:

Originally Posted by gd_barnes

No I don't know which bases are impacted outside of S66/S79. R51/S51/S66/R79/S79 and some ranges of R3/S3/R7/S7/R15/S15 likely have the greatest chance of being impacted. I suppose the best thing to do would be to look at bases that were started around the time that PFGW 3.6.x was released.

I will send you "good" and "bad" files for S66 & S79. There is a tricky issue on comparing them: My doublecheck is for n<=1000. The original (bad) files are for n<=2500. Here is how I think a comparison would need to be done:

good file scenario / bad file scenario
1. has prime / has same prime : OK
2. has prime / has different prime : bad
3. has prime / has no prime : bad
4. has no prime / has prime n>1000 : OK
5. has no prime / has prime n<=1000: bad
6. has no prime / has no prime : OK

So in evaluating "true" differences, you're looking for 2 situations:
1. The good file has a prime but the bad file has a different prime or no prime at all.
2. The good file has no prime but the bad file has a prime for n<=1000.

The files for S66 are huge since the conjecture is k>20M. I'll attempt to send the whole files in gmail but if it won't take due to size, I'll cut the S66 files down to an appropriate size. There are so many differences that even a relatively small sample of primes should be enough for you to get an idea of the problems.

The "bad" files that I will send you were run with PFGW 3.6.0. The "good" (doublecheck) files that I will send you were run with PFGW 3.3.6.

OK. BTW, use 7-zip to compress. It will create smaller files than a normal zip.

Nevermind. I got them.

rogue · 2014-07-12, 21:20

I have done a comparison of the results for these two bases and sent that information to Gary. He might want me to run the results for more bases, but I think in the end he will concur with what I've presented to him.

The problem only appears for numbers up to about 40 bits in size. pfgw 3.6.0 reports numbers as prime that are not prime, but it also misses primes. For these two bases the largest n with a bad or missing prime was 5.

I did not run those bases with pfgw 3.7.7. That needs to be done to verify correctness of the current version of pfgw.

This is what I would recommend (once we know that pfgw is working correctly):

1) Determine which bases are suspicious, i.e. those started after pfgw 3.6.0 was released.
2) Run those bases to n=100 (no reason at this time to go further).
3) Extract the primes where n <= 100 from the original submitted results.
4) If they are a match, done.
5) If there is a discrepancy, then use pl_remain.txt to do the following:
a) Remove k that have a prime for n > 100 from known results.
b) Remove known k that do not have a prime (as they have already been tested).
c) Run the remaining k to n=25000.
d) Run the remaining k to the conjecture's search limit.
e) Any remaining k get added back as needed to prove the conjecture.

I could write a program to make this easier. I would need the following:

1) A good version of pl_prime.txt (from step 2 above)
2) A good version of pl_remain.txt (from step 2 above)
3) A list of known primes, regardless of pfgw version. The program would ignore primes for n <= 100.
4) The current known list of k needed to prove the conjecture (although I could technically scrape that from the web pages).

I should be able to output a good list of primes (in k sequence) and a list of k needing further testing.

gd_barnes · 2014-07-12, 22:13

I concur with everything you said. That would be outstanding if we didn't have to start from scratch on the problem bases. When all is said and done, what I definitely need is one nice big file of "good" primes sorted by k, at least for primes n<2500. It looks like you can do that so that covers everything as far as I'm concerned. Considering the huge # of k's on these large-conjectured bases, I don't need more bases run to be convinced that the problem is only for very small primes.

Here is what I would suggest to start with: Write your program to do what you suggested and use it on the S66 and S79 files that I already sent you. I will also send you the rest of the primes for n=2.5K to 25K (S66) or 50K (S79) as well as the needed k's remaining files. The filenames with "good" in them are definitely good. The only difference vs. what you are suggesting that we do is that the "good" file has been searched to n=1000 vs. n=100. That shouldn't matter since the problem is for primes <= 40 bits.

rogue · 2014-07-12, 23:33

Quote:

Originally Posted by gd_barnes

I concur with everything you said. That would be outstanding if we didn't have to start from scratch on the problem bases. When all is said and done, what I definitely need is one nice big file of "good" primes sorted by k, at least for primes n<2500. It looks like you can do that so that covers everything as far as I'm concerned. Considering the huge # of k's on these large-conjectured bases, I don't need more bases run to be convinced that the problem is only for very small primes.

Here is what I would suggest to start with: Write your program to do what you suggested and use it on the S66 and S79 files that I already sent you. I will also send you the rest of the primes for n=2.5K to 25K (S66) or 50K (S79) as well as the needed k's remaining files. The filenames with "good" in them are definitely good. The only difference vs. what you are suggesting that we do is that the "good" file has been searched to n=1000 vs. n=100. That shouldn't matter since the problem is for primes <= 40 bits.

Correct about 100 vs 1000. I was referring to any other bases that have problems. This way the retesting of those ranges will take far less time.

rob147147 · 2014-07-13, 11:51

Slightly off this discussion, but I have a small suggestion.

Would it be possible to get the new bases script to print the version of PFGW used in all of the output files it produces? Then if we have a similar problem to this in the future it will be a much simpler case of just checking the output files to see which version they were produced with (assuming we can get people to use the latest version of the script!).

2014-07-12, 21:20	#16
rogue "Mark" Apr 2003 Between here and the 18D0₁₆ Posts	I have done a comparison of the results for these two bases and sent that information to Gary. He might want me to run the results for more bases, but I think in the end he will concur with what I've presented to him. The problem only appears for numbers up to about 40 bits in size. pfgw 3.6.0 reports numbers as prime that are not prime, but it also misses primes. For these two bases the largest n with a bad or missing prime was 5. I did not run those bases with pfgw 3.7.7. That needs to be done to verify correctness of the current version of pfgw. This is what I would recommend (once we know that pfgw is working correctly): 1) Determine which bases are suspicious, i.e. those started after pfgw 3.6.0 was released. 2) Run those bases to n=100 (no reason at this time to go further). 3) Extract the primes where n <= 100 from the original submitted results. 4) If they are a match, done. 5) If there is a discrepancy, then use pl_remain.txt to do the following: a) Remove k that have a prime for n > 100 from known results. b) Remove known k that do not have a prime (as they have already been tested). c) Run the remaining k to n=25000. d) Run the remaining k to the conjecture's search limit. e) Any remaining k get added back as needed to prove the conjecture. I could write a program to make this easier. I would need the following: 1) A good version of pl_prime.txt (from step 2 above) 2) A good version of pl_remain.txt (from step 2 above) 3) A list of known primes, regardless of pfgw version. The program would ignore primes for n <= 100. 4) The current known list of k needed to prove the conjecture (although I could technically scrape that from the web pages). I should be able to output a good list of primes (in k sequence) and a list of k needing further testing. Last fiddled with by rogue on 2014-07-12 at 21:33

2014-07-12, 22:13	#17
gd_barnes May 2007 Kansas; USA 24243₈ Posts	I concur with everything you said. That would be outstanding if we didn't have to start from scratch on the problem bases. When all is said and done, what I definitely need is one nice big file of "good" primes sorted by k, at least for primes n<2500. It looks like you can do that so that covers everything as far as I'm concerned. Considering the huge # of k's on these large-conjectured bases, I don't need more bases run to be convinced that the problem is only for very small primes. Here is what I would suggest to start with: Write your program to do what you suggested and use it on the S66 and S79 files that I already sent you. I will also send you the rest of the primes for n=2.5K to 25K (S66) or 50K (S79) as well as the needed k's remaining files. The filenames with "good" in them are definitely good. The only difference vs. what you are suggesting that we do is that the "good" file has been searched to n=1000 vs. n=100. That shouldn't matter since the problem is for primes <= 40 bits. Last fiddled with by gd_barnes on 2014-07-12 at 22:29

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Sieving with powers of small primes in the Small Prime variation of the Quadratic Sieve	mickfrancis	Factoring	2	2016-05-06 08:13
Small primes	kar_bon	Riesel Prime Data Collecting (k*2^n-1)	3	2013-05-11 04:56
PFGW can't find a small factor.	Arkadiusz	Software	7	2013-02-18 12:43
Small Primes	Housemouse	Math	2	2008-06-04 05:23
Problems with Large FFT but not Small FFT's?	RichTJ99	Hardware	2	2006-02-08 23:38

2014-07-12, 18:54	#14
KEP Quasi Admin Thing May 2005 2×3×7×23 Posts	I have to correct myself. The range tested by PFGW 3.6.3 is R3 11.0-12.0G and by tomorrow I'll have 11.00G to 11.01G send to you. The range will contain the primes for n<=25K.

2014-07-13, 11:51	#19
rob147147 Apr 2013 Durham, UK 63₁₀ Posts	Slightly off this discussion, but I have a small suggestion. Would it be possible to get the new bases script to print the version of PFGW used in all of the output files it produces? Then if we have a similar problem to this in the future it will be a much simpler case of just checking the output files to see which version they were produced with (assuming we can get people to use the latest version of the script!).