Can more P-1 help the DC effort

[Prime95]

Originally Posted by chalsall

I was just going to start by bringing in stuff in 102M through 110M which are "hanging". Everything in 101M is already assigned to my swarm (by way of GPU72; Primenet doesn't know about this except for the results it sees).

We're trying to pull in only exponents that have
a) Been tested once and not DC'ed.
b) Have P-1 where B1 = B2 (or no P-1 if there are any of those)
c) Are not currently assigned.

[chalsall]

Originally Posted by Prime95

We're trying to pull in only exponents that have
a) Been tested once and not DC'ed.
b) Have P-1 where B1 = B2 (or no P-1 if there are any of those)
c) Are not currently assigned.

George... Please PM me an SQL Query. A [C|T]SV vector, et al. Seriously.

Please just tell me what we should facilitate, and it will happen.

I'm currently (privately) working at doing some P-1'ing on the DCs up in the 101M etc ranges, which have not had "appropriate" P-1 work done. This result in the "P-1" candidates being seen in the Work Available report.

It took some effort to figure out how to identify and then work those DC candidates.

Please forgive my OCD. It comes naturally... 9^)

[Prime95]

I sent two SQL queries to you and James.

I am concerned by what you mean by "appropriate" when you say you are redoing some P-1 on 101M exponents. If no P-1 was done, clearly P-1 before a DC will be beneficial. If some P-1 was done (B1 = B2) then whether more P-1 is profitable depends on the amount of memory one can throw at the problem.

Ken has shown that somewhere around 5GB it becomes profitable to do more P-1 on exponents around 62M and B1=B2=~700K. More memory = more profitable (why I want to limit participation to 32+GB machines). However, Ken has not shown how much memory is required for P-1 to be profitable at 80M/B1=B2=900K or 100M/B1=B2=1M, etc.

IMO, we're getting a little ahead of ourselves talking about the 100M+ exponents -- a little more research is necessary.

[congsz]

Originally Posted by Prime95

We're trying to pull in only exponents that have
a) Been tested once and not DC'ed.
b) Have P-1 where B1 = B2 (or no P-1 if there are any of those)
c) Are not currently assigned.

There are some (although rare) exponents that satisfy all of the conditions above, but the B1=B2 is high enough, such as M66677777 with B1=B2=2M. Should these exponents be excluded?
These exponents can be searched by https://www.mersenne.ca/pm1_worst.php with exponent range: 60000000-100000000, prob: 3.2%-10%, stage 1 only.

And there are some (also rare) exponents with B1=/=B2 but the bounds are very low, such as M94016093, B1=2e5, B2=1.5e6. I think these exponents are needed to be redone P-1.
These exponents can be searched by https://www.mersenne.ca/pm1_worst.php with exponent range: 60000000-100000000, prob: 0%-5%, stage 2 only.

P.S.:
1. ~6000 exponents are indeed a piece of cake. It may be done even within 2 or 3 months if collaborative efforts are made.
2. I'm very interested in P-1. but sadly, I'm working on optimizing P-1 bounds for 8.9M now, and my 16GB machine seems too low if higher-RAM machines are present.

[kriesel]

Test system i5-1035G1 (AVX512 capable), prime95 v30.8b15, 6 GiB memory use allowed
B1=B2=705k 1.5815% https://www.mersenne.ca/prob.php?exp...5000&b2=705000

299k, 705k 1.5311% https://www.mersenne.ca/prob.php?exp...9000&b2=705000

B1=299000, B2=19575000 chosen, 3.52% per prime95, 3.5420% per
https://www.mersenne.ca/prob.php?exp...00&b2=19575000

P = 72410621 prp est 6d 10h 14 min = 555240 seconds

[Sep 2 00:41:43] Worker starting
[Sep 2 00:41:43] Setting affinity to run worker on CPU core #1
[Sep 2 00:41:43] Optimal P-1 factoring of M72410621 using up to 6144MB of memory.
[Sep 2 00:41:43] Assuming no factors below 2^75 and 1 primality test saved if a factor is found.
[Sep 2 00:41:43] Optimal bounds are B1=299000, B2=19575000
[Sep 2 00:41:43] Chance of finding a factor is an estimated 3.52%
[Sep 2 00:41:43] 
[Sep 2 00:41:44] Using AVX-512 FFT length 3840K, Pass1=2K, Pass2=1920, clm=1, 4 threads
[Sep 2 00:41:44] Setting affinity to run helper thread 1 on CPU core #2
[Sep 2 00:41:44] Setting affinity to run helper thread 2 on CPU core #3
[Sep 2 00:41:44] Setting affinity to run helper thread 3 on CPU core #4
[Sep 2 00:50:04] M72410621 stage 1 is 11.59% complete. Time: 500.271 sec.
[Sep 2 00:58:24] M72410621 stage 1 is 23.19% complete. Time: 500.119 sec.
[Sep 2 01:07:46] M72410621 stage 1 is 34.79% complete. Time: 561.465 sec.
[Sep 2 01:16:02] M72410621 stage 1 is 46.40% complete. Time: 496.426 sec.
[Sep 2 01:24:20] M72410621 stage 1 is 58.00% complete. Time: 498.051 sec.
[Sep 2 01:32:33] M72410621 stage 1 is 69.60% complete. Time: 493.166 sec.
[Sep 2 01:41:11] M72410621 stage 1 is 81.21% complete. Time: 517.308 sec.
[Sep 2 01:49:25] M72410621 stage 1 is 92.81% complete. Time: 494.342 sec.
[Sep 2 01:54:33] M72410621 stage 1 complete. 861970 transforms. Total time: 4369.228 sec.
[Sep 2 01:55:22] Inversion of stage 1 result complete. 5 transforms, 1 modular inverse. Time: 48.780 sec.
[Sep 2 01:55:22] Available memory is 6144MB.
[Sep 2 01:55:23] Switching to AVX-512 FFT length 4200K, Pass1=1920, Pass2=2240, clm=1, 4 threads
[Sep 2 01:55:23] Estimated stage 2 vs. stage 1 runtime ratio: 1.106
[Sep 2 01:55:23] Setting affinity to run helper thread 1 on CPU core #2
[Sep 2 01:55:23] Setting affinity to run helper thread 2 on CPU core #3
[Sep 2 01:55:23] Setting affinity to run helper thread 3 on CPU core #4
[Sep 2 01:55:24] Using 6136MB of memory.  D: 390, 48x141 polynomial multiplication.
[Sep 2 01:55:25] Setting affinity to run polymult helper thread on CPU core #2
[Sep 2 01:55:25] Setting affinity to run polymult helper thread on CPU core #3
[Sep 2 01:55:25] Setting affinity to run polymult helper thread on CPU core #4
[Sep 2 01:55:47] Stage 2 init complete. 1153 transforms. Time: 24.597 sec.
[Sep 2 02:02:05] M72410621 stage 2 at B2=4413630 [9.61%].  Time: 378.829 sec.
[Sep 2 02:08:16] M72410621 stage 2 at B2=6045780 [19.33%].  Time: 370.784 sec.
[Sep 2 02:14:20] M72410621 stage 2 at B2=7677930 [29.04%].  Time: 363.785 sec.
[Sep 2 02:20:55] M72410621 stage 2 at B2=9310080 [38.76%].  Time: 395.424 sec.
[Sep 2 02:27:09] M72410621 stage 2 at B2=10942230 [48.48%].  Time: 373.766 sec.
[Sep 2 02:33:23] M72410621 stage 2 at B2=12574380 [58.20%].  Time: 373.673 sec.
[Sep 2 02:39:34] M72410621 stage 2 at B2=14206530 [67.92%].  Time: 370.820 sec.
[Sep 2 02:45:53] M72410621 stage 2 at B2=15838680 [77.63%].  Time: 379.020 sec.
[Sep 2 02:52:42] M72410621 stage 2 at B2=17470830 [87.35%].  Time: 407.551 sec.
[Sep 2 02:59:51] M72410621 stage 2 at B2=19102980 [97.07%].  Time: 429.777 sec.
[Sep 2 03:02:33] M72410621 stage 2 complete. 344903 transforms. Total time: 4005.553 sec.
[Sep 2 03:02:33] Starting stage 2 GCD - please be patient.
[Sep 2 03:03:09] Stage 2 GCD complete. Time: 35.884 sec.
[Sep 2 03:03:09] M72410621 completed P-1, B1=299000, B2=19594380, Wi4: 50797C09

Sep 2 00:41:43 to Sep 2 03:03:09 = 2:21:26 = 8486 seconds for the P-1 retry

Factor odds 3.542% - 1.5311% = .020109 (1 in ~49.73 exponents)

Average over many exponents expected is saving 555240 * .020109 - 8486 = 2679 seconds = 0.74 hours; 0.48% of estimated PRP time, 31.6% of P-1 retry time.

Utility at least down to 6 GiB allocation confirmed. (0.0829 GiB/M of exponent)

[kriesel]

Originally Posted by Prime95

We're trying to pull in only exponents that have
a) Been tested once and not DC'ed.
b) Have P-1 where B1 = B2 (or no P-1 if there are any of those)
c) Are not currently assigned.

b) have had P-1 with bounds so grossly inadequate that a redo to suitable bounds with efficient code provides a probable net savings on average. No P-1, B1=B2 (no stage 2) with low, normal or somewhat elevated B1 values, or too-low B2 or B1 may qualify.

I think that is in the vicinity of ~1 to 1.5% new odds of factor or above. (Speculation only at this point.)

[axn]

Originally Posted by Prime95

The actual bounds selected are not optimal as prime95 should be tweaked to deduct the original P-1's chance of finding a factor.

Prior P-1 wouldn't change the optimality of the selected bounds. It only changes the profitability of P-1 -- i.e. whether more P-1 is warranted or directly proceed to DC.

We're trying to maximize LL Cost*tests saved*success prob - P-1 cost. With prior P-1, the formula changes to LL Cost*tests saved*(success prob-prior P-1 prob) - P-1 cost. This reduces our profitability, but equally for all possible P-1 plans we consider. Hence the optimal plan won't change, but merely the profitability.

[kriesel]

(axn probably gets this, but some other readers might not at least immediately.) When a fat stage 1 is run, not all its probability is properly subtracted from the odds of factoring in a P-1 retry with better bounds.
Subtract only the factor odds of that B1,B2 rectangle that is duplicated coverage (intersection of coverage sets in the B1, B2 graph). See toward end of mersenne.ca status pages of retried P-1 exponents, for example https://www.mersenne.ca/exponent/65827763

For example, B1=B2=2M covers some different possible factors that B1=700K, B2=13M doesn't; B1 ranging from 700k-2M & B2 up to 2M.

The portion of the fat single stage's run odds to subtract is for B1=700K, B2=2M because that is the portion of the retry that is duplicative, overlapping on the two runs. Some of my previous posts show the odds determinations and subtractions for specific exponents and bounds.

It matters for any DC type, not only LL; PRP with or without proof generation for example.

For locating candidate exponents for retry, anything with B2 < ~5-10M may be a suitable candidate.
Madpoo has done frequent 100K or 1M bounds tries, so screening his P-1 in a range of interest may be useful.


George: I'm not seeing prime95 do and log a stage 1 GCD. Has it been determined it's more efficient to only do one at the end of stage 2?

[kriesel]

Test system i5-1035G1, prime95 v30.8b15, 6 GiB memory use allowed
First P-1: B1=B2=2M 2.3662% https://www.mersenne.ca/prob.php?exp...000&b2=2000000

Retry bounds B1=285000, B2=19112000 chosen, 3.43% per prime95, 3.4598% per
https://www.mersenne.ca/prob.php?exp...00&b2=19112000

Overlap region: 285k, 2M 2.0927% https://www.mersenne.ca/prob.php?exp...000&b2=2000000

P = 66677777 prp est 5d 14hr 41 min = 484860 seconds

[Sep 2 07:58:24] Worker starting
[Sep 2 07:58:24] Setting affinity to run worker on CPU core #1
[Sep 2 07:58:24] Optimal P-1 factoring of M66677777 using up to 6144MB of memory.
[Sep 2 07:58:24] Assuming no factors below 2^75 and 1 primality test saved if a factor is found.
[Sep 2 07:58:24] Optimal bounds are B1=285000, B2=19112000
[Sep 2 07:58:24] Chance of finding a factor is an estimated 3.43%
[Sep 2 07:58:24] 
[Sep 2 07:58:25] Setting affinity to run helper thread 1 on CPU core #2
[Sep 2 07:58:25] Setting affinity to run helper thread 2 on CPU core #3
[Sep 2 07:58:25] Setting affinity to run helper thread 3 on CPU core #4
[Sep 2 07:58:25] Using AVX-512 FFT length 3528K, Pass1=1344, Pass2=2688, clm=1, 4 threads
[Sep 2 08:05:21] M66677777 stage 1 is 12.14% complete. Time: 415.956 sec.
[Sep 2 08:12:19] M66677777 stage 1 is 24.29% complete. Time: 418.289 sec.
[Sep 2 08:19:21] M66677777 stage 1 is 36.45% complete. Time: 421.706 sec.
[Sep 2 08:26:32] M66677777 stage 1 is 48.60% complete. Time: 430.796 sec.
[Sep 2 08:33:50] M66677777 stage 1 is 60.76% complete. Time: 437.935 sec.
[Sep 2 08:41:22] M66677777 stage 1 is 72.91% complete. Time: 452.346 sec.
[Sep 2 08:49:31] M66677777 stage 1 is 85.07% complete. Time: 488.785 sec.
[Sep 2 08:58:43] M66677777 stage 1 is 97.22% complete. Time: 552.453 sec.
[Sep 2 09:00:46] M66677777 stage 1 complete. 822880 transforms. Total time: 3741.832 sec.
[Sep 2 09:01:44] Inversion of stage 1 result complete. 5 transforms, 1 modular inverse. Time: 57.759 sec.
[Sep 2 09:01:45] Available memory is 6144MB.
[Sep 2 09:01:47] Switching to AVX-512 FFT length 3840K, Pass1=2K, Pass2=1920, clm=1, 4 threads
[Sep 2 09:01:47] Estimated stage 2 vs. stage 1 runtime ratio: 1.042
[Sep 2 09:01:47] Setting affinity to run helper thread 2 on CPU core #3
[Sep 2 09:01:47] Setting affinity to run helper thread 1 on CPU core #2
[Sep 2 09:01:47] Setting affinity to run helper thread 3 on CPU core #4
[Sep 2 09:01:47] Using 6124MB of memory.  D: 390, 48x158 polynomial multiplication.
[Sep 2 09:01:49] Setting affinity to run polymult helper thread on CPU core #4
[Sep 2 09:01:49] Setting affinity to run polymult helper thread on CPU core #2
[Sep 2 09:01:49] Setting affinity to run polymult helper thread on CPU core #3
[Sep 2 09:02:13] Stage 2 init complete. 1157 transforms. Time: 28.437 sec.
[Sep 2 09:10:24] M66677777 stage 2 at B2=4352010 [9.88%].  Time: 491.056 sec.
[Sep 2 09:18:01] M66677777 stage 2 at B2=5996250 [19.91%].  Time: 456.057 sec.
[Sep 2 09:24:38] M66677777 stage 2 at B2=7640490 [29.94%].  Time: 396.983 sec.
[Sep 2 09:31:12] M66677777 stage 2 at B2=9284730 [39.97%].  Time: 394.507 sec.
[Sep 2 09:37:48] M66677777 stage 2 at B2=10928970 [49.99%].  Time: 395.306 sec.
[Sep 2 09:44:07] M66677777 stage 2 at B2=12573210 [60.02%].  Time: 379.755 sec.
[Sep 2 09:49:49] M66677777 stage 2 at B2=14217450 [70.05%].  Time: 341.796 sec.
[Sep 2 09:55:31] M66677777 stage 2 at B2=15861690 [80.08%].  Time: 341.968 sec.
[Sep 2 10:01:28] M66677777 stage 2 at B2=17505930 [90.11%].  Time: 356.654 sec.
[Sep 2 10:07:27] M66677777 stage 2 complete. 336671 transforms. Total time: 3913.653 sec.
[Sep 2 10:07:27] Starting stage 2 GCD - please be patient.
[Sep 2 10:08:00] Stage 2 GCD complete. Time: 32.851 sec.
[Sep 2 10:08:00] M66677777 completed P-1, B1=285000, B2=19125990, Wi4: BDDD22FD

07:58:24 to 10:08:00 = 2:09:36 = 7776 seconds for the P-1 retry

Factor odds difference 0.034598 - 0.020927 = .013671 (1 in ~73.15 exponents)

Average over many exponents expected is saving 484860 * .013671 - 7776 = -1147 seconds = -0.319 hours (net loss of computing time, so better to skip the retry in this case; maybe more ram would have helped enough to make it marginally worthwhile)

[kriesel]

Originally Posted by Prime95

My test setup is working well.

I have two workers each using two cores. I have MaxHighMemWorkers=1 (the v30.8 default).
Initially, both workers run P-1 stage 1. The first to complete starts stage 2. The second to complete starts stage 1 on the next exponent in the queue. After that the workers basically alternate stage 2. The 64GB of memory is almost always in use.

Ok, but given that prime95 defaults to 4 cores per worker, presumably because that is often more efficient, is 2 workers x 2 cores each, or 1 worker x 4 cores each, more productive (higher aggregate throughput rate) for pure v30.8 P-1 with plentiful ram? For mixed P-1 followed by PRP for the same exponent(s)?
What's the stage 2 time / stage 1 time ratio for your runs at ~64 GiB allowed? I get 0.91 at 128 GiB but > 1.01 at 8 GiB or below, so for pure P-1 & dual workers, one will stall a bit at limited ram allocations.

[chalsall]

Originally Posted by Prime95

I sent two SQL queries to you and James..

Received with thanks. And copied James' reply.

We definitely want to go API with this. Let's continue talking off-channel. But, not today. BUSY!!!

Originally Posted by Prime95

I am concerned by what you mean by "appropriate" when you say you are redoing some P-1 on 101M exponents. If no P-1 was done, clearly P-1 before a DC will be beneficial.

To be explicit about what I meant by "Appropriately P-1'ed"...

This is actually a function of one of Primenet's reports. The Work Distribution Map, to be specific.

Scroll down to the 100M range on the page (three "Page Down" and three "Up arrow" keys in my environment). See all the numbers in the "Available P-1" column in 101M through to 109M (inclusive)? I've been personally bringining my compute to bear to lower these.

My kit. My time. Interestingly, no longer my electrons... 9^)

Originally Posted by Prime95

IMO, we're getting a little ahead of ourselves talking about the 100M+ exponents -- a little more research is necessary.

Conversely... It never hurts to run some experiments while figuring out what makes sense...