Doublecheck drive #1: k=3-1001 n=100K-260K

[mdettweiler]

Originally Posted by gd_barnes

Srfile can write out by k using the -g switch. This is ideal for what we want. You then just do the copy command to copy 3 files at a time into a 3k file for LLRing. There's two extra instances of the header in the file but LLR doesn't care.

Yes, I already knew about the -g switch--I used that option to split the file up into individual files for each k, and then I've been using the -G option to combine three files at a time into one.

Obviously you're sorting by k in some manner. However you got it down to 3k files is the way to get it down to 1k files and then combine them with a copy command into one file.

Excel can also be used for sorting small files like this but would take quite a bit longer with 167 or so total 3k files.

This is just for future reference. No need to mess with it now since people will post files of primes at the end of a range. It's how I divided k=300-1001 for n=50K-100K up into 50k chunks for splitting amongst 7 machines.

Hmm. Yeah, I see your point--but, I think that for a team drive such as this, it's better to have the files sorted by n, especially since we're just going to be collecting the primes files. That way, the candidates are distributed throughout the file in a smooth progression of lowest to highest n (rather than going up to 260K, then jumping back down to 100K for the next k, which might confuse less-experienced users), not to mention that we don't have the file loaded with extra messy NewPGen headers.

[mdettweiler]

Those of you who liked the team-sieve thing we did for this doublecheck effort might want to check out the new Sierpinski Base 6 Team Sieve at Conjectures 'R Us. It's just like the one we did for this doublecheck effort, except that of course it's doing different numbers.

[mdettweiler]

Taking 401-405.

[gd_barnes]

If people wouldn't mind posting the approximate CPU time that it takes to complete a 3k-range and the CPU being used, that would help us in case we need to increase or decrease future file sizes. I kind of cringed when I saw some 25000 k/n pair files but I have to remember that the n-range is much lower and that is certainly the exception. But the average CPU time may be more like a drive 3 n=2K file at n=420K, the last one before we dropped the file size to n=1K.

There may be some 3k-ranges that have 2 very heavy-weight k's and we may want to post only a 2k-range for it or vice-versa for 2 very light-weight k's. The variability becomes greater as the k-values increase so shouldn't be as much of a problem for k<400.


Gary

[Mini-Geek]

Originally Posted by gd_barnes

If people wouldn't mind posting the approximate CPU time that it takes to complete a 3k-range and the CPU being used, that would help us in case we need to increase or decrease future file sizes. I kind of cringed when I saw some 25000 k/n pair files but I have to remember that the n-range is much lower and that is certainly the exception. But the average CPU time may be more like a drive 3 n=2K file at n=420K, the last one before we dropped the file size to n=1K.

There may be some 3k-ranges that have 2 very heavy-weight k's and we may want to post only a 2k-range for it or vice-versa for 2 very light-weight k's. The variability becomes greater as the k-values increase so shouldn't be as much of a problem for k<400.


Gary

Okay, I will. In the mean time, some statistics from what I've got so far:
I've reserved two ranges, one with ~12K candidates, and one with ~25K. At the current rate for the smaller file, it would be 3.26 days, but it will be slowing as n increases (currently at n=~135700 and ~30 seconds per candidate at 7K used FFT).
For the larger file, it's currently coming out to 4.82 days, but of course will slow as well (currently n=~124400, switching between ~20 and ~27 seconds with switching between 6K and 7K FFTs).
I think it's still too early to tell if the files are too large or not.

[kar_bon]

reserving 21-25

[gd_barnes]

Originally Posted by Mini-Geek

Okay, I will. In the mean time, some statistics from what I've got so far:
I've reserved two ranges, one with ~12K candidates, and one with ~25K. At the current rate for the smaller file, it would be 3.26 days, but it will be slowing as n increases (currently at n=~135700 and ~30 seconds per candidate at 7K used FFT).
For the larger file, it's currently coming out to 4.82 days, but of course will slow as well (currently n=~124400, switching between ~20 and ~27 seconds with switching between 6K and 7K FFTs).
I think it's still too early to tell if the files are too large or not.

OK, then I expect you to figure exactly what it will take assuming that the LLR time increases by the square of the n-range. lol

It's actually more complicated then that and related to FFTlen jumps but it should give a close estimate because over the long run LLR time DOES increase with the square of the n-range. I'll have a try at some calculations.


Gary

[gd_barnes]

Originally Posted by Mini-Geek

Okay, I will. In the mean time, some statistics from what I've got so far:
I've reserved two ranges, one with ~12K candidates, and one with ~25K. At the current rate for the smaller file, it would be 3.26 days, but it will be slowing as n increases (currently at n=~135700 and ~30 seconds per candidate at 7K used FFT).
For the larger file, it's currently coming out to 4.82 days, but of course will slow as well (currently n=~124400, switching between ~20 and ~27 seconds with switching between 6K and 7K FFTs).
I think it's still too early to tell if the files are too large or not.

OK, for range one, I can extrapolate that since you said it would take 3.26 days to do the whole thing that you did an exact multiplication calculation based on the number of candidates processed vs. remaining. For ease, I'll use the n-range total.

Therefore:

You've tested 35700 / 160000 of your n-range or 22.31%.

22.31% of your total estimate of 3.26 days means you've spent 0.727388 of a day so far.

So if you've spent 0.727388 of a day so far, using algebra and incremental analysis, we can calculate that it took you .001452 of a day (I'll call that 'T' for time) to do n=100000-100100, T=.001455 to do n=100100-100200, etc. with the length of time increasing by the square of n such that it would take you T=.002669 to do n=135600-135700 with the total of T for all 100n ranges up to n=135700 being the 0.727388 days that you presumably spent for the range.

Further incrementing and summing up to n=260K shows that it should take you 8.027126 days to do n=100K-260K for a file with 12000 candidates with the final range of n=259900-260000 taking .009814 of a day, assuming that you used the straight multiplication to give your original estiamte.

Based on this, the 25000 candidate range should take 25000/12000 * 8.027126 = 16.72318 days.

And further, the average file size currently posted is 19563 candidates. Based on that, the average file size should take 19563 / 12000 * 8.027126 = 13.08639 days.

So, it looks like we're in the ballpark but a little large. I suspect that you have a high-speed machine so this average is somewhat larger than I would like but not too bad. If Anon wants to mess with it, if people with slower resources want to chip in and not spend up to 4 weeks on a file, then we could consider splitting up a file or two.

And finally, to prove that the incremental analysis is correct using the beginning and ending increments, the final range should take (260/100)^2 or 6.76 times as long as the first range. If you take the ratio of the time calculated for the beginning and ending ranges, you have .009814 / 001452 = 6.76. So there you have it!

Math is fun!

You'll have to let me know how close this is to the actual amount of time taken for the files.

Edit: I did this relatively quickly in an Excel spreadsheet but I'm pretty confident of its accuracy. But I have to mention that it's still only a relatively rough estimate because the true LLR times jump in fits and spurts. The spreadsheet is a little rough and hard to read but some people may find the algebra and formulas useful. If anyone wants me to post the spreadsheet, I will.


Gary

[Mini-Geek]

Your extrapolation of my CPU time is practically precisely what I recorded. Task Manager told me it was ~17.5 CPU hours per instance when I did the things, and I could have made my calc's easier with using your reverse-engineering. I calculated ([total pairs]/(([total pairs]-[pairs remaining])/[CPU hours]))/24=. For reference, yours would be 1/([completed n]/160000)*[CPU hours]/24=.
Mine would probably be a little more precise due to varying remaining pairs per n as n increases, but yours is far easier.

Originally Posted by gd_barnes

Further incrementing and summing up to n=260K shows that it should take you 8.027126 days to do n=100K-260K for a file with 12000 candidates with the final range of n=259900-260000 taking .009814 of a day, assuming that you used the straight multiplication to give your original estiamte.

Based on this, the 25000 candidate range should take 25000/12000 * 8.027126 = 16.72318 days.

And further, the average file size currently posted is 19563 candidates. Based on that, the average file size should take 19563 / 12000 * 8.027126 = 13.08639 days.

A note on the exact sizes of the ranges (as I put it in off the top of my head, before I did all the calculations which used the exact sizes): They're 12969 and 22674. You may want to plug that in to your spreadsheet and see if it changes a significant amount.

Originally Posted by gd_barnes

So, it looks like we're in the ballpark but a little large. I suspect that you have a high-speed machine so this average is somewhat larger than I would like but not too bad. If Anon wants to mess with it, if people with slower resources want to chip in and not spend up to 4 weeks on a file, then we could consider splitting up a file or two.

It's on a dual-core 2.5 GHz Athlon (http://www.newegg.com/product/produc...82E16819103778 if you want to know exact). You may or may not consider that high-speed. I think the ranges are too large currently.

Originally Posted by gd_barnes

You'll have to let me know how close this is to the actual amount of time taken for the files.

Will do. I'll be sure to get the exact, or closest estimate, time each range finishes, and I can look back to when I reserved for very close estimates of when I started them. I might be able to get CPU time, too, but I'm not sure my computer won't be rebooted or I won't restart an LLR instance before then.

[em99010pepe]

What about CPU timings?

With a quad-core overclocked to 2.968 GHz:

27*2^100000-1 is not prime.  LLR Res64: B033EEE9274FF1F1  Time : 7.786 sec.
29*2^100000-1 is not prime.  LLR Res64: 7DF93B2FC5EF6077  Time : 7.785 sec.
27*2^100012-1 is not prime.  LLR Res64: 69F091E95935EE0E  Time : 7.797 sec.
27*2^100013-1 is not prime.  LLR Res64: CDAAE5C5F00C6FF5  Time : 7.794 sec.
31*2^100019-1 is not prime.  LLR Res64: 418D2ADE4901D642  Time : 7.807 sec.
31*2^100039-1 is not prime.  LLR Res64: FB1F60CAE7646CB3  Time : 7.779 sec.
31*2^100061-1 is not prime.  LLR Res64: 780E5645B0395BA2  Time : 7.779 sec.
31*2^100129-1 is not prime.  LLR Res64: CD3F553F0B3B61B4  Time : 7.808 sec.
27*2^100133-1 is not prime.  LLR Res64: 0E506AE171784315  Time : 7.847 sec.
27*2^259952-1 is not prime.  LLR Res64: A1E5B8F7FD1C606A  Time : 52.654 sec.
27*2^259973-1 is not prime.  LLR Res64: 5036B0FD0DFDBE74  Time : 52.588 sec.
31*2^259975-1 is not prime.  LLR Res64: D6A76B5B89DA91F8  Time : 52.493 sec.
27*2^260000-1 is not prime.  LLR Res64: B5D601E69D7F123F  Time : 52.633 sec.

[Mini-Geek]

Originally Posted by em99010pepe

What about CPU timings?

With a quad-core overclocked to 2.968 GHz:

27*2^100000-1 is not prime.  LLR Res64: B033EEE9274FF1F1  Time : 7.786 sec.
29*2^100000-1 is not prime.  LLR Res64: 7DF93B2FC5EF6077  Time : 7.785 sec.
27*2^100012-1 is not prime.  LLR Res64: 69F091E95935EE0E  Time : 7.797 sec.
27*2^100013-1 is not prime.  LLR Res64: CDAAE5C5F00C6FF5  Time : 7.794 sec.
31*2^100019-1 is not prime.  LLR Res64: 418D2ADE4901D642  Time : 7.807 sec.
31*2^100039-1 is not prime.  LLR Res64: FB1F60CAE7646CB3  Time : 7.779 sec.
31*2^100061-1 is not prime.  LLR Res64: 780E5645B0395BA2  Time : 7.779 sec.
31*2^100129-1 is not prime.  LLR Res64: CD3F553F0B3B61B4  Time : 7.808 sec.
27*2^100133-1 is not prime.  LLR Res64: 0E506AE171784315  Time : 7.847 sec.
27*2^259952-1 is not prime.  LLR Res64: A1E5B8F7FD1C606A  Time : 52.654 sec.
27*2^259973-1 is not prime.  LLR Res64: 5036B0FD0DFDBE74  Time : 52.588 sec.
31*2^259975-1 is not prime.  LLR Res64: D6A76B5B89DA91F8  Time : 52.493 sec.
27*2^260000-1 is not prime.  LLR Res64: B5D601E69D7F123F  Time : 52.633 sec.

27*2^100000-1 is not prime.  LLR Res64: B033EEE9274FF1F1  Time : 16.464 sec.
29*2^100000-1 is not prime.  LLR Res64: 7DF93B2FC5EF6077  Time : 16.583 sec.
27*2^100012-1 is not prime.  LLR Res64: 69F091E95935EE0E  Time : 16.588 sec.
27*2^100013-1 is not prime.  LLR Res64: CDAAE5C5F00C6FF5  Time : 16.586 sec.
31*2^100019-1 is not prime.  LLR Res64: 418D2ADE4901D642  Time : 16.610 sec.
31*2^100039-1 is not prime.  LLR Res64: FB1F60CAE7646CB3  Time : 16.599 sec.
31*2^100061-1 is not prime.  LLR Res64: 780E5645B0395BA2  Time : 16.591 sec.
31*2^100129-1 is not prime.  LLR Res64: CD3F553F0B3B61B4  Time : 16.618 sec.
27*2^100133-1 is not prime.  LLR Res64: 0E506AE171784315  Time : 16.612 sec.
27*2^259952-1 is not prime.  LLR Res64: A1E5B8F7FD1C606A  Time : 107.655 sec.
27*2^259973-1 is not prime.  LLR Res64: 5036B0FD0DFDBE74  Time : 107.636 sec.
31*2^259975-1 is not prime.  LLR Res64: D6A76B5B89DA91F8  Time : 107.720 sec.
27*2^260000-1 is not prime.  LLR Res64: B5D601E69D7F123F  Time : 107.740 sec.

Would these CPU timings help for a more accurate estimate, or something?