mersenneforum.org  

Go Back   mersenneforum.org > Prime Search Projects > Conjectures 'R Us

Reply
 
Thread Tools
Old 2014-07-11, 08:59   #1
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

3×5×701 Posts
Default PFGW 3.6.x problems on small primes

Quote:
Originally Posted by Lennart View Post
taking s66 to 50k

Lennart
Lennart,

Can you hold off on doing this? We have found many problems in the k's remaining on S66 and S79. I am trying to think of the best way to tackle the problem. Sorry.
gd_barnes is offline   Reply With Quote
Old 2014-07-11, 09:42   #2
Lennart
 
Lennart's Avatar
 
"Lennart"
Jun 2007

112010 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
Lennart,

Can you hold off on doing this? We have found many problems in the k's remaining on S66 and S79. I am trying to think of the best way to tackle the problem. Sorry.

I am only sieving at this time.. I will not start LLR.

Are you redoing them from scratch ?

Lennart
Lennart is offline   Reply With Quote
Old 2014-07-11, 10:52   #3
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

3×5×701 Posts
Default

Quote:
Originally Posted by Lennart View Post
I am only sieving at this time.. I will not start LLR.

Are you redoing them from scratch ?

Lennart
I have not done anything with them yet but that is likely what we'll have to do. A bad version of PFGW, version 3.6.0, was used to script the bases. Many incorrect and missing small primes were found, around n=2, 3, 4, and 5; 100s of errors. It is a huge loss of work but I think it would be too risky to do anything else at this point.

This was pointed out to me by Phillip (MagentaSunrise) after he searched S79 for n=25K-50K and subsequently doublechecked all of the k's remaining to n=100. I think we'll also need to eventually look at many other large-conjectured bases, especially bases started since PFGW 3.6.0 was released. It's part of the reason that I doublechecked Riesel base 3 for k<1G to n=100K.

Last fiddled with by gd_barnes on 2014-07-11 at 10:54
gd_barnes is offline   Reply With Quote
Old 2014-07-11, 14:52   #4
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

6,427 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
I have not done anything with them yet but that is likely what we'll have to do. A bad version of PFGW, version 3.6.0, was used to script the bases. Many incorrect and missing small primes were found, around n=2, 3, 4, and 5; 100s of errors. It is a huge loss of work but I think it would be too risky to do anything else at this point.

This was pointed out to me by Phillip (MagentaSunrise) after he searched S79 for n=25K-50K and subsequently doublechecked all of the k's remaining to n=100. I think we'll also need to eventually look at many other large-conjectured bases, especially bases started since PFGW 3.6.0 was released. It's part of the reason that I doublechecked Riesel base 3 for k<1G to n=100K.
Do you know the scope of the problem? In other words do you know how many bases were impacted?
rogue is online now   Reply With Quote
Old 2014-07-11, 19:16   #5
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

3·5·701 Posts
Default

Quote:
Originally Posted by rogue View Post
Do you know the scope of the problem? In other words do you know how many bases were impacted?
No. I haven't looked at other bases yet but I did catch some serious errors in a base 7 submission from Ian at one point and the problem was missing (or sometimes additional) teeny primes. There were clearly too many k's remaining in one of his searches. What I was thinking of doing to start with is looking at bases with more than ~500 k's remaining and doublechecking them to n=1000. I don't know which versions of PFGW were bad but I suspect that it may be limited to 3.6.0. I have "baseline" versions of PFGW (3.3.6 & 3.4.1) that I'm very confident are correct across all prime sizes, which I use for doublechecking.

I suppose the best thing to do would be to get a copy of every version of PFGW since 3.3.6 and run each of them on the script for a few 1000 k's on S66/S79 to n=1000 to see if there are any other erroneous versions so that people get rid of or update their bad versions. (I know that x.x.0 versions are a little more susceptible to problems since x.x.1-x.x.9 versions usually correct any bugs in extensive updates to the program.) I suspect that it may be related to updates in the GWNUM libraries. I know that George does extensive testing of those when doing LLR updates but testing on very small primes may not always be done.

This is such a time-consuming effort to either do or coordinate that I haven't wanted to take the time to do either. The error was first pointed out to me about 2-3 months ago.

Last fiddled with by gd_barnes on 2014-07-11 at 19:21
gd_barnes is offline   Reply With Quote
Old 2014-07-11, 21:28   #6
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

191B16 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
No. I haven't looked at other bases yet but I did catch some serious errors in a base 7 submission from Ian at one point and the problem was missing (or sometimes additional) teeny primes. There were clearly too many k's remaining in one of his searches. What I was thinking of doing to start with is looking at bases with more than ~500 k's remaining and doublechecking them to n=1000. I don't know which versions of PFGW were bad but I suspect that it may be limited to 3.6.0. I have "baseline" versions of PFGW (3.3.6 & 3.4.1) that I'm very confident are correct across all prime sizes, which I use for doublechecking.

I suppose the best thing to do would be to get a copy of every version of PFGW since 3.3.6 and run each of them on the script for a few 1000 k's on S66/S79 to n=1000 to see if there are any other erroneous versions so that people get rid of or update their bad versions. (I know that x.x.0 versions are a little more susceptible to problems since x.x.1-x.x.9 versions usually correct any bugs in extensive updates to the program.) I suspect that it may be related to updates in the GWNUM libraries. I know that George does extensive testing of those when doing LLR updates but testing on very small primes may not always be done.

This is such a time-consuming effort to either do or coordinate that I haven't wanted to take the time to do either. The error was first pointed out to me about 2-3 months ago.
In looking at the release notes, I would guess that versions up to 3.6.3 had the issue and that it was fixed in 3.6.4.

Did this result in erroneously reported primes, missing primes, or both? From what you've written it appears that this only affects numbers with small n. Is that correct? If so do you know the upper bound of n that was affected?
rogue is online now   Reply With Quote
Old 2014-07-12, 00:11   #7
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

3×5×701 Posts
Default

Quote:
Originally Posted by rogue View Post
In looking at the release notes, I would guess that versions up to 3.6.3 had the issue and that it was fixed in 3.6.4.

Did this result in erroneously reported primes, missing primes, or both? From what you've written it appears that this only affects numbers with small n. Is that correct? If so do you know the upper bound of n that was affected?
It did both erroneous primes and missing primes, which is why it would be very risky not to start from scratch on these bases. Initially I thought it was just missing primes, which would mean it would be easy enough to remove them from k's remaining on the pages, but unfortunately the problem wasn't that simple. From what I can tell it only affected primes for n<10 but I can't be sure enough of it to say for sure. Most of the bad or missing primes were n=2,3,4,or5.

Probably anytime there is a change in GWNUM libraries and before releasing to the public, there should be a test run of several 100 or 1000 small primes (n<100 on many varying bases) to make sure nothing was affected at the small end of things.

I re-ran a good chunk of those two bases up to n=1000, although it's such a mess I haven't made an attempt to change the CRUS pages. I think I can provide both the "bad" and "good" files of primes for you if you'd like.

Last fiddled with by gd_barnes on 2014-07-12 at 00:12
gd_barnes is offline   Reply With Quote
Old 2014-07-12, 00:23   #8
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

11001000110112 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
It did both erroneous primes and missing primes, which is why it would be very risky not to start from scratch on these bases. Initially I thought it was just missing primes, which would mean it would be easy enough to remove them from k's remaining on the pages, but unfortunately the problem wasn't that simple. From what I can tell it only affected primes for n<10 but I can't be sure enough of it to say for sure. Most of the bad or missing primes were n=2,3,4,or5.

Probably anytime there is a change in GWNUM libraries and before releasing to the public, there should be a test run of several 100 or 1000 small primes (n<100 on many varying bases) to make sure nothing was affected at the small end of things.

I re-ran a good chunk of those two bases up to n=1000, although it's such a mess I haven't made an attempt to change the CRUS pages. I think I can provide both the "bad" and "good" files of primes for you if you'd like.
I would like to know what the upper bound for n is. You stated that most are for n <= 5, yet you tested to n=1000. Maybe you can narrow down the search range for any double-checking effort.
rogue is online now   Reply With Quote
Old 2014-07-12, 00:30   #9
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

3×5×701 Posts
Default

Quote:
Originally Posted by rogue View Post
I would like to know what the upper bound for n is. You stated that most are for n <= 5, yet you tested to n=1000. Maybe you can narrow down the search range for any double-checking effort.
To the best of my knowledge in my doublecheck, all of the problems were for n<=5 (maybe n<=6) but I doublechecked all n<=1000. But I can't say for sure that all problems were for n<=6 because I didn't do an automated line-by-line comparison. I just visually compared the two.

The comparison is a bit tricky because sometimes it would miss a prime for n=4 and then later find a prime for the same k for, say, n=500. Other times it would not find a prime at all for that k. Other times for another k it would have a "prime" for n=4 when in fact the "prime" was composite.

It's a difficult task to sort out all of the scenarios, which is why I don't see any alternative but to start the bases completely over.

Edit: Perhaps I can send you both the good and bad files and you can come up with an automated comparison so that we're more confident in what we're dealing with.

Last fiddled with by gd_barnes on 2014-07-12 at 00:33
gd_barnes is offline   Reply With Quote
Old 2014-07-12, 17:21   #10
KEP
Quasi Admin Thing
 
KEP's Avatar
 
May 2005

967 Posts
Default

Quote:
Originally Posted by gd_barnes View Post
It's a difficult task to sort out all of the scenarios, which is why I don't see any alternative but to start the bases completely over.
I'm in support of this idea, since too much work has to be done in order to sort out wich primes is bad primes, wich primes were missed and wich k's that were legitimate as remaining.

However, before starting from scratch, I feel like we have to do following:

1. choose a base <1030
2. choose a conjecture <1000
3. find as many as possible versions of PFGW (both 32- and 64-bit)
4. test the choosen conjecture in each available version of PFGW 32 bit*
5. test the choosen conjecture in each available version of PFGW 64 bit*

* using the starting new bases script, going from n=1 to n=25000.

6. compare the different resultfiles to each other, to see wich files is simmilar and wich is different, thereby we can see wich versions of PFGW that is at least working similarly
7. create a k-list with all k's <conjectured k
8. sieve the range of k's from n=1 to n=25000, using srsieve and sr2sieve
9. LLR test the remaining candidates, using LLR 3.8.9 and StopOnPrimedK=1
10. Compare the primelist made by LLR with the primelist made by PFGW
11. compare/verify the mob list made by PFGW (manually)
12. compare/verify the trivial factors list made by PFGW (manually)
13. by finish of compare, make an official zip package, with a verified safe version of PFGW wich has to be used when starting new bases
14. start a drive to start all bases one-by-one from n=1 to n=25K (maybe the good folks at Primegrid can be of help here, since it could be a great PSA project)**

** This would mean, that everything is correct and doublechecked, such that future searchers should not be concerned about mallock material. Also it will make sure that no search ranges for the relative low bases with a very big conjecture is left unsearched for months or years at a time.

Maybe if a drive is setup, we should have 2 users search the same range of k's. Even though it is no easy task to compare millions of lines, I think I'll be able to offer a helping hand if you need it.

On a sad note, I just noticed that my R3 13G-14G range was searched using PFGW 32-bit version 3.6.3 ... so I guess that will make the range suspiscious...

Take care

Kenneth
KEP is offline   Reply With Quote
Old 2014-07-12, 17:53   #11
gd_barnes
 
gd_barnes's Avatar
 
May 2007
Kansas; USA

244238 Posts
Default

Sounds great if someone has unlimited personal time to coordinate such an effort. I think this is something that we'll just have to do piece-meal as we have the time and inclination to do so. At some point, I think I'll need to add a doublecheck column to the CRUS pages so that we know what has been done.

As for R3 k=13G-14G, can you provide me with a small list of primes, perhaps k=13G-13.01G to n=10K or 25K using PFGW 3.6.3? I'll then run PFGW 3.3.6 or 3.4.1 against it and we'll compare the two.
gd_barnes is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Sieving with powers of small primes in the Small Prime variation of the Quadratic Sieve mickfrancis Factoring 2 2016-05-06 08:13
Small primes kar_bon Riesel Prime Data Collecting (k*2^n-1) 3 2013-05-11 04:56
PFGW can't find a small factor. Arkadiusz Software 7 2013-02-18 12:43
Small Primes Housemouse Math 2 2008-06-04 05:23
Problems with Large FFT but not Small FFT's? RichTJ99 Hardware 2 2006-02-08 23:38

All times are UTC. The time now is 17:46.


Sat Oct 16 17:46:30 UTC 2021 up 85 days, 12:15, 1 user, load averages: 1.50, 1.23, 1.21

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.