Starting new bases - Page 9

gd_barnes · 2010-01-12, 08:11

I've now added a link to the latest version of the starting new bases script to the 1st post in this thread. Instead of continuing to post them as attachments in this thread, it will be better to have them quickly available in the 1st post. I'll remove the requirement of PFGW 3.2.3 and earlier after the Linux version of PFGW 3.2.7 is available and I have updated it to include the necessary changes.

gd_barnes · 2010-01-20, 13:49

I have now modified the starting bases script to work with PFGW versions 3.2.7 and later. The 1st post in this thread has been changed to include both versions. 3 major enhancements have been made to the 3.2.7 version:

1. It will automatically primality test unproven PRPs. You never again have to prove the pfgw.log file.

2. It will write out composite PRPs to file pl_compPRP.txt. This will sometimes be an interesting list for some of the small bases with huge conjectures.

3. It will no longer write empty files. I personally found irritating the frequently empty GFN, k's remaining, MOB, and trivial factors files that needed to be deleted. No more. If the file doesn't exist at the end of your run, it means there were no k's that were applicable for it.

Note that these changes were not made and the first 2 not even possible with the 3.2.3 version.

I only had time to parallel test 3-4 bases although I did attempt to hit all areas of code that were changed. I did find a few old composite PRPs that I was able to get it to write out for base 3. If anyone can run some more parallel tests on it, I'd appreciate it.

Have fun!

Gary

Mini-Geek · 2010-01-20, 14:34

Quote:

Originally Posted by gd_barnes

I only had time to parallel test 3-4 bases although I did attempt to hit all areas of code that were changed. I did find a few old composite PRPs that I was able to get it to write out for base 3. If anyone can run some more parallel tests on it, I'd appreciate it.

Did you check these "composite PRPs" with -a1 (or higher) to check if it was really the prime proof that was off, and not the PRP?

Mini-Geek · 2010-01-20, 22:25

In my tests on Riesel base 3, I noticed that using PFGW 3.3.1 and the new script made for it makes the memory usage keep growing roughly linearly throughout the run. It's currently at about 400 MB memory usage at about 4/5th through it, so I'd estimate it at 200 MB per million k, (remember that odd k's are eliminated for this work due to trivial factors, so a 4M k range is 2M k's) or about 210 bytes per k (for Riesel base 3 k=805M-810M, n<=1000). I'm not sure what's causing this memory usage. This doesn't occur with the older script and PFGW version, so it must've been introduced somehow in one of those two. Is is storing some information from every k in memory or something?

rogue · 2010-01-20, 23:43

Quote:

Originally Posted by Mini-Geek

In my tests on Riesel base 3, I noticed that using PFGW 3.3.1 and the new script made for it makes the memory usage keep growing roughly linearly throughout the run. It's currently at about 400 MB memory usage at about 4/5th through it, so I'd estimate it at 200 MB per million k, (remember that odd k's are eliminated for this work due to trivial factors, so a 4M k range is 2M k's) or about 210 bytes per k (for Riesel base 3 k=805M-810M, n<=1000). I'm not sure what's causing this memory usage. This doesn't occur with the older script and PFGW version, so it must've been introduced somehow in one of those two. Is is storing some information from every k in memory or something?

I have an idea as to what could be causing the memory leak. I'll investigate, but I'm not too concerned, since it is likely related to the size of the input. Few users will run millions of k through PFGW at one time.

Mini-Geek · 2010-01-21, 01:54

The output of the old and new setups match, (Riesel base 3 k=805M-810M) ignoring the differences caused by the compPRP removals. These primes all silently fail and return composite with the newest setup: (this is, coincidentally(?), the entire compPRP file)

Code:

806928292*3^55-1
808640228*3^54-1
808846184*3^48-1
808953748*3^490-1
809396984*3^49-1
809629928*3^49-1
809696008*3^734-1

The only apparent problems I see are minor issues with PFGW that I've already reported (a memory leak, posted earlier, and the silently failing primes posted here). The script seems to be working great.

gd_barnes · 2010-01-21, 04:16

Quote:

Originally Posted by rogue

I have an idea as to what could be causing the memory leak. I'll investigate, but I'm not too concerned, since it is likely related to the size of the input. Few users will run millions of k through PFGW at one time.

It's more common than you might think. Mainly our Riesel and Sierp base 3 efforts but also base 7 and 15 in the future is a big reason that the script comes in handy.

Therefore I'm concerned about it. I recently got a PM from Lennart that he is interested in doing some testing on Sierp base 3. With his resources, he's liable to really test this kind of issue.

Gary

rogue · 2010-01-21, 12:31

Quote:

Originally Posted by gd_barnes

It's more common than you might think. Mainly our Riesel and Sierp base 3 efforts but also base 7 and 15 in the future is a big reason that the script comes in handy.

Therefore I'm concerned about it. I recently got a PM from Lennart that he is interested in doing some testing on Sierp base 3. With his resources, he's liable to really test this kind of issue.

On this project, yes, but not on other projects.

gd_barnes · 2010-02-10, 23:07

Quote:

Originally Posted by rogue

On this project, yes, but not on other projects.

Didn't you introduce this problem in the last couple of versions of PFGW as a result of correcting other problems? If so, it shouldn't matter how many projects are impacted. It needs to be fixed.

I lost 4 days processing on a full quad while out of town for 7 days as a result of the memory leak that never existed on previous versions of PFGW. To say I was miffed is an understatement. It makes me want to stick with 3.2.3 and prior. I can code around it by running batch files with smaller ranges but that shouldn't be necessary.

I suspect this may be why Lennart hasn't worked or has worked very little on his k=810M-820M range for Riesel base 3 that he reserved.

If you have an idea of what is causing it, please look into it. Thank you.

gd_barnes · 2010-02-10, 23:33

Introducing: New starting bases script 4.3. See the 1st post of this thread.

This is a modified starting bases script, the changes of which should come in especially handy if you're running 1000's or millions of k's on smaller bases. The main crux of the changes was to do a better job of proving PRPs and reducing "composite" PRPs that are really primes.

I added versioning. See the first part of the script. This is version 4.3. The last was 4.2.

The crux of the changes:

Previously when a PRIMEM/PRIMEP test came back as PRP (i.e. not proven but not composite either), the script assumed it was composite and immediately wrote it to the compPRP file because it could not prove it. This likely had little impact on most of you and I did not experience it at all in testing previously when doing full factoring, i.e. with the -f100 (or -f) switch. It happens most frequently when attempting to save time with large #'s of k's by having factoring set to < 100%, i.e. -f50 or -f30 instead of -f100 (or -f). This was causing quite a bit of problem on base 3 when I was using -f30 factoring for speed-up purposes. Now 2 things will potentially happen if the above tests comes back as either PRP or (possibly erroneous) composite after it came back originally as PRP:

1. It will immediately do a full factorization up to as high as P=100M. A large majority of the time, this will "trivially" prove an unproven PRP if it has < 16 digits.

2. If the unproven PRP (or erroneous) composite is not trivially proven prime by the "full" factoring in #1, it will do a combined test with PRIMEC (the equivalent of the -tc switch). Sometimes this additional check will prove it or at least "switch" it from an "errneous composite" to PRP.

Now, one of 3 things will happen:

1. If at any step, it comes back as a proven prime, like before, it will write it to the pl_prime.txt file and go on to the next k.

2. If after all of the above, it shows as composite, like before, it will write it to the compPRP.txt file and continue with higher n-values on the k.

3. If after all of the above, it still comes back as an unproven PRP, it will "assume" that the PRP is prime and will write it to BOTH a NEW pl_prp.txt file and the pl_prime.txt file and goes on to the next k.

I put quite a few comments in the code about all of this. Although this extensively reduces unproven PRPs that were thought by the script to previously be composites, it only helps "erroneous" composites a little bit.

For most of you, the main problem that will still exist is erroneous composites that have an exponent > 25. That is a LLR/PFGW/GWNUM problem that the script cannot code around.

For those of you concerned about speed, don't worry. In the sceme of things, the above code is likely executed < 1 in 10,000 tests (at the most), especially if you're using full factoring with -f or -f100. This is one of those cases where we spend 95% of our time coding around 5% of issues (well maybe < .001% of issues here). Even on bases 3/7/15 with factoring set to -f30, it likely gets hit < 1 in 1,000 test. The slowdown even on those bases should be < 1%.

Testing:

A k=20M range has now been tested with the new script on base 3 with factoring set to -f30 (30%). All facets of changed code were touched. Interestingly in order to "make" the script write to the new pl_prp.txt file, I had to reduce the factoring down to only factoring to < 10000. In other words, I previously had a lot of unproven PRP's that all became proven once I changed factoring up to 100M. It didn't write any to the pl_prp.txt file once the code was changed, which is what I had hoped for.

I also parallel tested 5 other bases with no differences.

Since this is not a large amount of parallel testing, I'd appreciate it if others could do some additional parallel testing and report back any problems. Like before, this version only works for PFGW versions 3.2.7 and later.

Gary

rogue · 2010-02-11, 01:13

Quote:

Originally Posted by gd_barnes

Didn't you introduce this problem in the last couple of versions of PFGW as a result of correcting other problems? If so, it shouldn't matter how many projects are impacted. It needs to be fixed.

I lost 4 days processing on a full quad while out of town for 7 days as a result of the memory leak that never existed on previous versions of PFGW. To say I was miffed is an understatement. It makes me want to stick with 3.2.3 and prior. I can code around it by running batch files with smaller ranges but that shouldn't be necessary.

I suspect this may be why Lennart hasn't worked or has worked very little on his k=810M-820M range for Riesel base 3 that he reserved.

If you have an idea of what is causing it, please look into it. Thank you.

I'm pretty certain that I know the change that caused the memory leak. It was tied to the changes PFGW output to the GUI. I fixed that issue in the current release, but it is possible that other memory leaks exist. I have not had the time to investigate further. Which version did you experience the memory leak with?

When I have a chance, I'll run PFGW for a few hours (or days) and see if it is still leaking and if so, try to discover where the leak is. I'll most likely use base 3 because it will go through many more tests in that time frame than it would for most other bases. Can you tell me the upper bound of n that was being used when you ran into the problem? I don't expect that to be a factor, but you never know.

BTW, if I need to make any changes, I will probably modify PFGW to automatically apply -a1 to primality tests until that problem is resolved. If you have followed the LLR 3.8.0 threads it appears that it might also be affected by the same problem in gwnum.

2010-01-20, 13:49	#90
gd_barnes May 2007 Kansas; USA 101·103 Posts	I have now modified the starting bases script to work with PFGW versions 3.2.7 and later. The 1st post in this thread has been changed to include both versions. 3 major enhancements have been made to the 3.2.7 version: 1. It will automatically primality test unproven PRPs. You never again have to prove the pfgw.log file. 2. It will write out composite PRPs to file pl_compPRP.txt. This will sometimes be an interesting list for some of the small bases with huge conjectures. 3. It will no longer write empty files. I personally found irritating the frequently empty GFN, k's remaining, MOB, and trivial factors files that needed to be deleted. No more. If the file doesn't exist at the end of your run, it means there were no k's that were applicable for it. Note that these changes were not made and the first 2 not even possible with the 3.2.3 version. I only had time to parallel test 3-4 bases although I did attempt to hit all areas of code that were changed. I did find a few old composite PRPs that I was able to get it to write out for base 3. If anyone can run some more parallel tests on it, I'd appreciate it. Have fun! Gary Last fiddled with by gd_barnes on 2010-01-20 at 13:53

2010-01-20, 22:25	#92
Mini-Geek Account Deleted "Tim Sorbera" Aug 2006 San Antonio, TX USA 17·251 Posts	In my tests on Riesel base 3, I noticed that using PFGW 3.3.1 and the new script made for it makes the memory usage keep growing roughly linearly throughout the run. It's currently at about 400 MB memory usage at about 4/5th through it, so I'd estimate it at 200 MB per million k, (remember that odd k's are eliminated for this work due to trivial factors, so a 4M k range is 2M k's) or about 210 bytes per k (for Riesel base 3 k=805M-810M, n<=1000). I'm not sure what's causing this memory usage. This doesn't occur with the older script and PFGW version, so it must've been introduced somehow in one of those two. Is is storing some information from every k in memory or something? Last fiddled with by Mini-Geek on 2010-01-20 at 22:31

2010-01-21, 01:54	#94
Mini-Geek Account Deleted "Tim Sorbera" Aug 2006 San Antonio, TX USA 17×251 Posts	The output of the old and new setups match, (Riesel base 3 k=805M-810M) ignoring the differences caused by the compPRP removals. These primes all silently fail and return composite with the newest setup: (this is, coincidentally(?), the entire compPRP file) Code: 8069282923^55-1 8086402283^54-1 8088461843^48-1 8089537483^490-1 8093969843^49-1 8096299283^49-1 8096960083^734-1 The only apparent problems I see are minor issues with PFGW that I've already reported (a memory leak, posted earlier, and the silently failing primes posted here). The script seems to be working great. Last fiddled with by Mini-Geek on 2010-01-21 at 01:55*

2010-02-10, 23:33	#98
gd_barnes May 2007 Kansas; USA 101×103 Posts	Introducing: New starting bases script 4.3. See the 1st post of this thread. This is a modified starting bases script, the changes of which should come in especially handy if you're running 1000's or millions of k's on smaller bases. The main crux of the changes was to do a better job of proving PRPs and reducing "composite" PRPs that are really primes. I added versioning. See the first part of the script. This is version 4.3. The last was 4.2. The crux of the changes: Previously when a PRIMEM/PRIMEP test came back as PRP (i.e. not proven but not composite either), the script assumed it was composite and immediately wrote it to the compPRP file because it could not prove it. This likely had little impact on most of you and I did not experience it at all in testing previously when doing full factoring, i.e. with the -f100 (or -f) switch. It happens most frequently when attempting to save time with large #'s of k's by having factoring set to < 100%, i.e. -f50 or -f30 instead of -f100 (or -f). This was causing quite a bit of problem on base 3 when I was using -f30 factoring for speed-up purposes. Now 2 things will potentially happen if the above tests comes back as either PRP or (possibly erroneous) composite after it came back originally as PRP: 1. It will immediately do a full factorization up to as high as P=100M. A large majority of the time, this will "trivially" prove an unproven PRP if it has < 16 digits. 2. If the unproven PRP (or erroneous) composite is not trivially proven prime by the "full" factoring in #1, it will do a combined test with PRIMEC (the equivalent of the -tc switch). Sometimes this additional check will prove it or at least "switch" it from an "errneous composite" to PRP. Now, one of 3 things will happen: 1. If at any step, it comes back as a proven prime, like before, it will write it to the pl_prime.txt file and go on to the next k. 2. If after all of the above, it shows as composite, like before, it will write it to the compPRP.txt file and continue with higher n-values on the k. 3. If after all of the above, it still comes back as an unproven PRP, it will "assume" that the PRP is prime and will write it to BOTH a NEW pl_prp.txt file and the pl_prime.txt file and goes on to the next k. I put quite a few comments in the code about all of this. Although this extensively reduces unproven PRPs that were thought by the script to previously be composites, it only helps "erroneous" composites a little bit. For most of you, the main problem that will still exist is erroneous composites that have an exponent > 25. That is a LLR/PFGW/GWNUM problem that the script cannot code around. For those of you concerned about speed, don't worry. In the sceme of things, the above code is likely executed < 1 in 10,000 tests (at the most), especially if you're using full factoring with -f or -f100. This is one of those cases where we spend 95% of our time coding around 5% of issues (well maybe < .001% of issues here). Even on bases 3/7/15 with factoring set to -f30, it likely gets hit < 1 in 1,000 test. The slowdown even on those bases should be < 1%. Testing: A k=20M range has now been tested with the new script on base 3 with factoring set to -f30 (30%). All facets of changed code were touched. Interestingly in order to "make" the script write to the new pl_prp.txt file, I had to reduce the factoring down to only factoring to < 10000. In other words, I previously had a lot of unproven PRP's that all became proven once I changed factoring up to 100M. It didn't write any to the pl_prp.txt file once the code was changed, which is what I had hoped for. I also parallel tested 5 other bases with no differences. Since this is not a large amount of parallel testing, I'd appreciate it if others could do some additional parallel testing and report back any problems. Like before, this version only works for PFGW versions 3.2.7 and later. Gary Last fiddled with by gd_barnes on 2010-02-11 at 05:39

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Starting mprime at boot	daxmick	PrimeNet	18	2019-03-17 01:12
Issues With Starting CUDALucas	Smokingenius	GPU Computing	8	2015-11-13 17:46
mfaktc not starting in Mac OSX	bayanne	GPU Computing	0	2014-05-10 14:38
Disk starting to go	Chuck	Hardware	8	2013-05-20 06:40
mprime starting	spaz	Software	9	2009-05-03 06:41

2010-01-12, 08:11	#89
gd_barnes May 2007 Kansas; USA 101×103 Posts	I've now added a link to the latest version of the starting new bases script to the 1st post in this thread. Instead of continuing to post them as attachments in this thread, it will be better to have them quickly available in the 1st post. I'll remove the requirement of PFGW 3.2.3 and earlier after the Linux version of PFGW 3.2.7 is available and I have updated it to include the necessary changes.