Thinking out loud about getting under 20M unfactored exponents

[Prime95]

Originally Posted by Prime95

I'm curious if the B2=136*B1 is an indication of a bug.

I stepped through the code with your exponent, B1, memory and it is operating correctly. My 40x to 60x values were using a "puny" B1 of 500000. Apparently as B1 gets larger, it makes sense for the B2/B1 ratio to go up as well.

Looking at mersenne.ca https://www.mersenne.ca/prob.php?exp...=69&prob=12.75 you might try a B1 in the 15M to 20M area with B2 auto-computed for your 10 hour runs and see which gives a higher chance of finding a factor.

[LaurV]

Originally Posted by axn

I'm running P-1...

Very good post, kinda my issues too, but you said it better than I could say it!

If we are to trust RDS' papers which he always push in front as much as he can (albeit they refer to ECM mostly) the best choice (i.e. most efficient, wallclock per probability of finding a factor) is when the program spends about the same amount of time in stage 1 as it does in stage 2. Regardless of how fast one stage is done, comparable with the other stage. If stage 2 becomes more efficient in the newer version, then it seems common sense that B2 will grow related to B1. But 150 times seems a bit too much...

Just saying...

[nordi]

I got an out of memory error during my test run. Amongst many other things, the Linux kernel said

Killed process 22440 (mprime) total-vm:112100576kB, anon-rss:109294088kB, file-rss:0kB, shmem-rss:4kB

The memory limit in local.txt is set to "100000" which should be 100,000,000,000 bytes or 104,857,600,000 bytes if MiB are used instead of MB. But apparently, mprime was using 111,917,146,112 bytes, which is either 11.9 or 7 GB more than it should.

That was probably triggered by stage 2 of ECM needing more RAM than before. I was running 32 threads of ECM of all sizes, from M1277 to M9,100,919. Also, I had some programs still running so mprime really had to observe the memory limit. So not necessarily a new issue, just one that surfaces now.

[nordi]

Originally Posted by nordi

Killed process 22440 (mprime) total-vm:112100576kB, anon-rss:109294088kB, file-rss:0kB, shmem-rss:4kB

I freed up some memory and started a second test run, which aborted after a few minutes with the kernel saying

[1310480.143387] Killed process 28361 (mprime) total-vm:127209968kB, anon-rss:123919728kB, file-rss:0kB, shmem-rss:0kB

I'll set "Memory=50000" instead of 100000 and keep testing.

[nordi]

Originally Posted by nordi

I'll set "Memory=50000" instead of 100000 and keep testing.

Even with that setting, mprime consumes up to ~100GB instead of 50GB, i.e. twice as much as it should.

I'm monitoring memory usage with

while true; do grep "RssAnon" /proc/$(pidof mprime)/status; sleep 10; done

and the highest I got so far was "RssAnon: 105544236 kB".


Edit:
And a bit later, mprime segfaulted. Kernel log says


[1315248.083098] show_signal_msg: 38 callbacks suppressed
[1315248.083102] mprime[29045]: segfault at 7f7878287f26 ip 00000000004166e5 sp 00007f777eff2d60 error 6 in mprime[400000+2190000]

[masser]

Originally Posted by Prime95

1) rerun successful P-1 and ECM attempts to make sure the new code does not miss any factors.

I feel confident that the new code is not missing any factors. I've attached results from two machines; the new code found all of the known factors I sought. For P-1, I first used the minimal B1,B2 and so a lot of factors were missed, as B2 was ignored. On the second pass, I used B1=B2/10 and that worked to find the remaining factors. On the Haswell i5, I used 7 GB of ram. On the Skylake i7, I used 3 GB or ram. For ECM, I had to make several passes over the remaining candidates to find all of the factors. With each pass I increased the available memory and the code responded with higher B2 values, as expected.

Originally Posted by Prime95

2) make sure save and restore work

I feel less confident about this point. See the attached file, bad_read_result.txt. I ran a short P-1 attempt on M21150827 and then something went wrong when I later tried a longer P-1 attempt. I checked to see if the error was reproducible; running the two P-1 attempts back-to-back in a clean directory with the new executable worked fine the second time. Maybe it was a fluke on my local system or maybe others will report a similar occurrence.

I will re-run some of the longer attempts above with version 30.3, collect some timing comparisons and report back later.

[James Heinrich]

Running some P-1 in the 17M range. v30.3 was using 7-8GB RAM (out of 40GB available) and 960 relative primes.
v30.4:

Optimal P-1 factoring of M17847311 using up to 40960MB of memory.
Assuming no factors below 2^65 and 4 primality tests saved if a factor is found.
Optimal bounds are B1=404000, B2=27115000
Chance of finding a factor is an estimated 8.4%
...
Starting stage 1 GCD - please be patient.
Stage 1 GCD complete. Time: 4.691 sec.
D: 2310, relative primes: 4918, stage 2 primes: 1655650, pair%=95.05
Using 38019MB of memory.
Stage 2 init complete. 52544 transforms. Time: 57.675 sec.

Stage2 memory initialization seems faster than I'm used to for previous versions, which is a very good thing.

Comparative result output for adjacent assignments:

v30.3: M17843899 completed P-1, B1=370000, B2=12533000, E=12
v30.4: M17847311 completed P-1, B1=404000, B2=27115000

Somewhat higher B1, vastly higher B2, no Brent-Suyama?

[axn]

Originally Posted by James Heinrich

Stage2 memory initialization seems faster than I'm used to for previous versions

It creates more temporaries yet faster, because it is not doing B-S.

Originally Posted by James Heinrich

Comparative result output for adjacent assignments:

How do the runtimes and probabilities compare?

Originally Posted by James Heinrich

Somewhat higher B1, vastly higher B2, no Brent-Suyama?

Higher B2 (owing to faster stage 2) & no B-S are the key features.

EDIT:

Assuming no factors below 2^65 and 4 primality tests saved if a factor is found.

This is not good. For some reason it is thinking the exponent has been factored to 2^65 when it has been factored to 2^70. This means the bounds it has calculated won't be optimal.

[Prime95]

Originally Posted by axn

This is not good. For some reason it is thinking the exponent has been factored to 2^65 when it has been factored to 2^70. This means the bounds it has calculated won't be optimal.

The 2^65 value comes from worktodo.txt.

Pfactor= in worktodo.txt will NOT calculate optimal bounds as far as this project is concerned. Pfactor is optimizing bounds to maximize the number of LL/PRP tests saved per unit of P-1 time invested. Pminus1= lines in worktodo.txt optimizes the B2 bound to maximize the chance of finding a factor per unit of P-1 time invested (user is responsible for picking the B1 bound). I know -- it is all very confusing.

[axn]

Originally Posted by Prime95

The 2^65 value comes from worktodo.txt.

Ok, so James gave it wrong values? Regardless, this affects the optimality of bounds (by affecting the probability calculations).

[James Heinrich]

Originally Posted by axn

Ok, so James gave it wrong values? Regardless, this affects the optimality of bounds (by affecting the probability calculations).

Yes, I gave it "wrong" values on purpose, specifically to affect the bounds. Using Pfactor worktodo lines, specifying the "PrimeNet-default" TF level and a large number of tests-saved (anywhere from 2-10 depending on exponent size) forces Prime95 to choose extra-large bounds that I deem suitable for re-doing of P-1 work. This would of course be inappropriate at 100M/wavefront P-1 work, but I think entirely appropriate at <20M.

BTW, at George's request, my Worst P-1 page now includes your choice of Pfactor or Pminus1 worktodo formats.
Note that I make no claim that Prime95 will select the same bounds from both variants, just that either are a reasonably-suitable starting point for P-1 redo work.