PFGW 3.6.x problems on small primes

gd_barnes · 2014-07-11, 08:59

Quote:

Originally Posted by Lennart

taking s66 to 50k

Lennart

Lennart,

Can you hold off on doing this? We have found many problems in the k's remaining on S66 and S79. I am trying to think of the best way to tackle the problem. Sorry.

Lennart · 2014-07-11, 09:42

Quote:

Originally Posted by gd_barnes

Lennart,

Can you hold off on doing this? We have found many problems in the k's remaining on S66 and S79. I am trying to think of the best way to tackle the problem. Sorry.

I am only sieving at this time.. I will not start LLR.

Are you redoing them from scratch ?

Lennart

gd_barnes · 2014-07-11, 10:52

Quote:

Originally Posted by Lennart

I am only sieving at this time.. I will not start LLR.

Are you redoing them from scratch ?

Lennart

I have not done anything with them yet but that is likely what we'll have to do. A bad version of PFGW, version 3.6.0, was used to script the bases. Many incorrect and missing small primes were found, around n=2, 3, 4, and 5; 100s of errors. It is a huge loss of work but I think it would be too risky to do anything else at this point.

This was pointed out to me by Phillip (MagentaSunrise) after he searched S79 for n=25K-50K and subsequently doublechecked all of the k's remaining to n=100. I think we'll also need to eventually look at many other large-conjectured bases, especially bases started since PFGW 3.6.0 was released. It's part of the reason that I doublechecked Riesel base 3 for k<1G to n=100K.

rogue · 2014-07-11, 14:52

Quote:

Originally Posted by gd_barnes

I have not done anything with them yet but that is likely what we'll have to do. A bad version of PFGW, version 3.6.0, was used to script the bases. Many incorrect and missing small primes were found, around n=2, 3, 4, and 5; 100s of errors. It is a huge loss of work but I think it would be too risky to do anything else at this point.

This was pointed out to me by Phillip (MagentaSunrise) after he searched S79 for n=25K-50K and subsequently doublechecked all of the k's remaining to n=100. I think we'll also need to eventually look at many other large-conjectured bases, especially bases started since PFGW 3.6.0 was released. It's part of the reason that I doublechecked Riesel base 3 for k<1G to n=100K.

Do you know the scope of the problem? In other words do you know how many bases were impacted?

gd_barnes · 2014-07-11, 19:16

Quote:

Originally Posted by rogue

Do you know the scope of the problem? In other words do you know how many bases were impacted?

No. I haven't looked at other bases yet but I did catch some serious errors in a base 7 submission from Ian at one point and the problem was missing (or sometimes additional) teeny primes. There were clearly too many k's remaining in one of his searches. What I was thinking of doing to start with is looking at bases with more than ~500 k's remaining and doublechecking them to n=1000. I don't know which versions of PFGW were bad but I suspect that it may be limited to 3.6.0. I have "baseline" versions of PFGW (3.3.6 & 3.4.1) that I'm very confident are correct across all prime sizes, which I use for doublechecking.

I suppose the best thing to do would be to get a copy of every version of PFGW since 3.3.6 and run each of them on the script for a few 1000 k's on S66/S79 to n=1000 to see if there are any other erroneous versions so that people get rid of or update their bad versions. (I know that x.x.0 versions are a little more susceptible to problems since x.x.1-x.x.9 versions usually correct any bugs in extensive updates to the program.) I suspect that it may be related to updates in the GWNUM libraries. I know that George does extensive testing of those when doing LLR updates but testing on very small primes may not always be done.

This is such a time-consuming effort to either do or coordinate that I haven't wanted to take the time to do either. The error was first pointed out to me about 2-3 months ago.

rogue · 2014-07-11, 21:28

Quote:

Originally Posted by gd_barnes

No. I haven't looked at other bases yet but I did catch some serious errors in a base 7 submission from Ian at one point and the problem was missing (or sometimes additional) teeny primes. There were clearly too many k's remaining in one of his searches. What I was thinking of doing to start with is looking at bases with more than ~500 k's remaining and doublechecking them to n=1000. I don't know which versions of PFGW were bad but I suspect that it may be limited to 3.6.0. I have "baseline" versions of PFGW (3.3.6 & 3.4.1) that I'm very confident are correct across all prime sizes, which I use for doublechecking.

I suppose the best thing to do would be to get a copy of every version of PFGW since 3.3.6 and run each of them on the script for a few 1000 k's on S66/S79 to n=1000 to see if there are any other erroneous versions so that people get rid of or update their bad versions. (I know that x.x.0 versions are a little more susceptible to problems since x.x.1-x.x.9 versions usually correct any bugs in extensive updates to the program.) I suspect that it may be related to updates in the GWNUM libraries. I know that George does extensive testing of those when doing LLR updates but testing on very small primes may not always be done.

This is such a time-consuming effort to either do or coordinate that I haven't wanted to take the time to do either. The error was first pointed out to me about 2-3 months ago.

In looking at the release notes, I would guess that versions up to 3.6.3 had the issue and that it was fixed in 3.6.4.

Did this result in erroneously reported primes, missing primes, or both? From what you've written it appears that this only affects numbers with small n. Is that correct? If so do you know the upper bound of n that was affected?

gd_barnes · 2014-07-12, 00:11

Quote:

Originally Posted by rogue

In looking at the release notes, I would guess that versions up to 3.6.3 had the issue and that it was fixed in 3.6.4.

Did this result in erroneously reported primes, missing primes, or both? From what you've written it appears that this only affects numbers with small n. Is that correct? If so do you know the upper bound of n that was affected?

It did both erroneous primes and missing primes, which is why it would be very risky not to start from scratch on these bases. Initially I thought it was just missing primes, which would mean it would be easy enough to remove them from k's remaining on the pages, but unfortunately the problem wasn't that simple. From what I can tell it only affected primes for n<10 but I can't be sure enough of it to say for sure. Most of the bad or missing primes were n=2,3,4,or5.

Probably anytime there is a change in GWNUM libraries and before releasing to the public, there should be a test run of several 100 or 1000 small primes (n<100 on many varying bases) to make sure nothing was affected at the small end of things.

I re-ran a good chunk of those two bases up to n=1000, although it's such a mess I haven't made an attempt to change the CRUS pages. I think I can provide both the "bad" and "good" files of primes for you if you'd like.

rogue · 2014-07-12, 00:23

Quote:

Originally Posted by gd_barnes

It did both erroneous primes and missing primes, which is why it would be very risky not to start from scratch on these bases. Initially I thought it was just missing primes, which would mean it would be easy enough to remove them from k's remaining on the pages, but unfortunately the problem wasn't that simple. From what I can tell it only affected primes for n<10 but I can't be sure enough of it to say for sure. Most of the bad or missing primes were n=2,3,4,or5.

Probably anytime there is a change in GWNUM libraries and before releasing to the public, there should be a test run of several 100 or 1000 small primes (n<100 on many varying bases) to make sure nothing was affected at the small end of things.

I re-ran a good chunk of those two bases up to n=1000, although it's such a mess I haven't made an attempt to change the CRUS pages. I think I can provide both the "bad" and "good" files of primes for you if you'd like.

I would like to know what the upper bound for n is. You stated that most are for n <= 5, yet you tested to n=1000. Maybe you can narrow down the search range for any double-checking effort.

gd_barnes · 2014-07-12, 00:30

Quote:

Originally Posted by rogue

I would like to know what the upper bound for n is. You stated that most are for n <= 5, yet you tested to n=1000. Maybe you can narrow down the search range for any double-checking effort.

To the best of my knowledge in my doublecheck, all of the problems were for n<=5 (maybe n<=6) but I doublechecked all n<=1000. But I can't say for sure that all problems were for n<=6 because I didn't do an automated line-by-line comparison. I just visually compared the two.

The comparison is a bit tricky because sometimes it would miss a prime for n=4 and then later find a prime for the same k for, say, n=500. Other times it would not find a prime at all for that k. Other times for another k it would have a "prime" for n=4 when in fact the "prime" was composite.

It's a difficult task to sort out all of the scenarios, which is why I don't see any alternative but to start the bases completely over.

Edit: Perhaps I can send you both the good and bad files and you can come up with an automated comparison so that we're more confident in what we're dealing with.

KEP · 2014-07-12, 17:21

Quote:

Originally Posted by gd_barnes

It's a difficult task to sort out all of the scenarios, which is why I don't see any alternative but to start the bases completely over.

I'm in support of this idea, since too much work has to be done in order to sort out wich primes is bad primes, wich primes were missed and wich k's that were legitimate as remaining.

However, before starting from scratch, I feel like we have to do following:

1. choose a base <1030
2. choose a conjecture <1000
3. find as many as possible versions of PFGW (both 32- and 64-bit)
4. test the choosen conjecture in each available version of PFGW 32 bit*
5. test the choosen conjecture in each available version of PFGW 64 bit*

* using the starting new bases script, going from n=1 to n=25000.

6. compare the different resultfiles to each other, to see wich files is simmilar and wich is different, thereby we can see wich versions of PFGW that is at least working similarly
7. create a k-list with all k's <conjectured k
8. sieve the range of k's from n=1 to n=25000, using srsieve and sr2sieve
9. LLR test the remaining candidates, using LLR 3.8.9 and StopOnPrimedK=1
10. Compare the primelist made by LLR with the primelist made by PFGW
11. compare/verify the mob list made by PFGW (manually)
12. compare/verify the trivial factors list made by PFGW (manually)
13. by finish of compare, make an official zip package, with a verified safe version of PFGW wich has to be used when starting new bases
14. start a drive to start all bases one-by-one from n=1 to n=25K (maybe the good folks at Primegrid can be of help here, since it could be a great PSA project)**

** This would mean, that everything is correct and doublechecked, such that future searchers should not be concerned about mallock material. Also it will make sure that no search ranges for the relative low bases with a very big conjecture is left unsearched for months or years at a time.

Maybe if a drive is setup, we should have 2 users search the same range of k's. Even though it is no easy task to compare millions of lines, I think I'll be able to offer a helping hand if you need it.

On a sad note, I just noticed that my R3 13G-14G range was searched using PFGW 32-bit version 3.6.3 ... so I guess that will make the range suspiscious...

Take care

Kenneth

gd_barnes · 2014-07-12, 17:53

Sounds great if someone has unlimited personal time to coordinate such an effort. I think this is something that we'll just have to do piece-meal as we have the time and inclination to do so. At some point, I think I'll need to add a doublecheck column to the CRUS pages so that we know what has been done.

As for R3 k=13G-14G, can you provide me with a small list of primes, perhaps k=13G-13.01G to n=10K or 25K using PFGW 3.6.3? I'll then run PFGW 3.3.6 or 3.4.1 against it and we'll compare the two.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Sieving with powers of small primes in the Small Prime variation of the Quadratic Sieve	mickfrancis	Factoring	2	2016-05-06 08:13
Small primes	kar_bon	Riesel Prime Data Collecting (k*2^n-1)	3	2013-05-11 04:56
PFGW can't find a small factor.	Arkadiusz	Software	7	2013-02-18 12:43
Small Primes	Housemouse	Math	2	2008-06-04 05:23
Problems with Large FFT but not Small FFT's?	RichTJ99	Hardware	2	2006-02-08 23:38

2014-07-12, 17:53	#11
gd_barnes May 2007 Kansas; USA 101×103 Posts	Sounds great if someone has unlimited personal time to coordinate such an effort. I think this is something that we'll just have to do piece-meal as we have the time and inclination to do so. At some point, I think I'll need to add a doublecheck column to the CRUS pages so that we know what has been done. As for R3 k=13G-14G, can you provide me with a small list of primes, perhaps k=13G-13.01G to n=10K or 25K using PFGW 3.6.3? I'll then run PFGW 3.3.6 or 3.4.1 against it and we'll compare the two.