Thinking out loud about getting under 20M unfactored exponents

[axn]

First impressions. All testing done using Linux version. ECM tests were done in Colab VMs running both FMA3 & AVX-512. P-1 was done with my Ryzen 5 3600. More details on these tests can be provided upon request.

ECM
Good
1. Finds new factors (haven't checked for existing factors)
2. Stage 1 & Stage 2 are faster (despite stage 2 being slightly bigger)
3. Uses more memory.


P-1
Good
1. Finds existing factors
2. Stage 2 is faster (about 50% faster)

Bad
1. B2 chosen is much too big (Pminus1). Needs a way to control B2 selection / honor specified B2
2. Potential % display bug.
3. Saw some weird output (sort of like infinte loop) when multiple workers entered Stage 2 together
4. Potential issue with MaxHighMemWorkers setting - more than the specified number of workers proceeded to Stage 2 (CPU: Ryzen 5 3600, 6 workers)

[Prime95]

Originally Posted by axn

1. B2 chosen is much too big (Pminus1). Needs a way to control B2 selection / honor specified B2
2. Potential % display bug.
3. Saw some weird output (sort of like infinte loop) when multiple workers entered Stage 2 together
4. Potential issue with MaxHighMemWorkers setting - more than the specified number of workers proceeded to Stage 2 (CPU: Ryzen 5 3600, 6 workers)

1. Why do you say B2 is too big? The algorithm tries to find the B2 such that if you were to invest more CPU time you'd get the same increase in chance of finding a factor if you invested that CPU time increasing B1 or B2. I'm not saying you're not right, there could be a code bug or inaccurate estimation of P-1 B1 or B2 costs.
The new B2 selection can be circumvented by leaving off the sieve depth in worktodo.txt or by Pminus1BestB2=0 in prime.txt

2. Can you send a screenshot or a cut/paste of the screen output?

3. Same request as 2.

4. I'll try to reproduce. I did not test that.

[lycorn]

Originally Posted by masser

Could someone teach me how to find the sigma that was used to find an ECM factor from the mersenne.org website?

I'm trying to test that the new ECM in version 30.4 recovers known factors without running hundreds of curves.

Have a look at the results.txt file. The sigma used is displayed along with the relevant parameters of the successful run. The following is a copy/paste from my results file upon finding a factor:

(...)

[Sat Dec 19 18:01:38 2020]
UID: lycorn/supernova, M544837 completed 427 ECM curves, B1=250000, B2=25000000, Wg4: 0FA45DC6, AID: 091211D7E0C858E32CCEB53CD41ACA9E
[Sat Dec 19 19:23:21 2020]
ECM found a factor in curve #63, stage #2
Sigma=7091917254266823, B1=250000, B2=25000000.
UID: lycorn/supernova, M544721 has a factor: 479701949456122248252251360609 (ECM curve 63, B1=250000, B2=25000000), AID: 56052C6C12AC548F71FBD1F0B34779CF

(...)

[Happy5214]

Originally Posted by masser

Could someone teach me how to find the sigma that was used to find an ECM factor from the mersenne.org website?

Originally Posted by lycorn

Have a look at the results.txt file. The sigma used is displayed along with the relevant parameters of the successful run. The following is a copy/paste from my results file upon finding a factor:

That's not what was asked. Most ECM results on mersenne.org don't display the sigma, though I'm not sure if the server has a record of it. A few ECM results (like Ryan Propper's) list the sigma after the B2. For example, on 1399, you'll see the line "Factor: 9729831901051958663829453004687723271026191923786080297556081 / (ECM curve 1, B1=850000000, B2=15892628251516, Sigma=16318523421442679557)".

[masser]

Originally Posted by Happy5214

Most ECM results on mersenne.org don't display the sigma, though I'm not sure if the server has a record of it.

Thanks for clarifying my question: does anyone know if the server keeps a record of the sigma value?

[axn]

Originally Posted by Prime95

1. Why do you say B2 is too big? The algorithm tries to find the B2 such that if you were to invest more CPU time you'd get the same increase in chance of finding a factor if you invested that CPU time increasing B1 or B2. I'm not saying you're not right, there could be a code bug or inaccurate estimation of P-1 B1 or B2 costs.

Too long from my perspective. I'm running P-1 with (30m, 600m) which completes in about 12 hours (7+5) for a probability of 11.1%. With the new code, it is running (30m,4080m) in about 28 hours (7+21) for a probability of 14.5%. If it would just run the given B2, it would complete it in about 10 hours (7+3) which would be a much better deal for me.
However, I think I understand what you're saying. If I'm to spend 10 hours on a P-1, my best chance would not be a (30m, 600m) (7+3) split, but rather smaller B1, bigger B2. So, I guess the problem was that I didn't understand how the logic was selecting B2. My guess is, something like (10m, whatever program selects) might be a better use of my compute time. Anyway, something to play with.
I would still want to reduce the B2, because of a different problem -- I want to avoid the number of stage 2 running in parallel because it can cause up to 15% slow down when too many stage 2 are running. However this is probably too much for the s/w to take into account.
Incidentally, it appears that the formula for probability has been tweaked. It is reporting p(30m, 4080m, 69 bits) as 14.5% whereas mersenne.ca is giving it as 14.0%.

Originally Posted by Prime95

The new B2 selection can be circumvented by leaving off the sieve depth in worktodo.txt or by Pminus1BestB2=0 in prime.txt

Thanks. Leaving off the tf depth is working fine! That helps.

Originally Posted by Prime95

2. Can you send a screenshot or a cut/paste of the screen output?

Sorry, don't have it. But easy enough to describe. With the (30m, 4080m) run, I have allocated enough memory per worker for about 6300 temps. That means about 5.3 passes are needed. EDIT:- I might have misunderstood the relation between the temps and passes. Anyway, based on the number of stage 2 primes it reported, I estimated about 5 and bit passes.
In the first pass, the stage 2 % goes from 0 to about 19%. However, when the second pass starts, instead of the % going to 20% and on, it restarts at 0% and keeps going till another 19%, and again restarts on next pass and so on. Looks like the % variable is getting reset for every pass. ECM is fine, btw.

Originally Posted by Prime95

3. Same request as 2.

Don't have this one either. But it was sort of an infinite loop "x memory being used" shown by the final worker entering stage 2. It happened when I didn't have per-worker memory configured, and all the workers entered stage 2 in quick succession. The first one took everything, then when the second one came, there was some adjustment, and then the third came in and more adjustment, and so on until, bam, all hell broke loose. Infinite loop, and it wouldn't even respond to ^C, and had to kill -9.
Anyway, just thought I'd let you know. But I have no interest in trying to reproduce this one.

Originally Posted by Prime95

4. I'll try to reproduce. I did not test that.

Sure, thanks. Could it be a Ryzen thing? Reason I'm asking is, when I gave that setting as 2, it enforced it as 4 (i.e. allowed 4/6 stage 2, but stopped two of them), so wondering if it enforced two per L3 block or some such weirdness.
Anyway, for now, I am looking at enforcing memory at per-worker level. However if this could be sorted out, I could be allocating more per stage 2 safely.


One other (potential) bug. On AVX-512, doing ECM on wagstaff, it didn't display the x bits-per-word below FFT limit (more than 0.5 allows extra optimizations) text. Only this particular combination. If have done both mersenne & wagstaff ECM on both FMA3 & AVX-512 and all the other combinations show this.

[Prime95]

Originally Posted by axn

Too long from my perspective. I'm running P-1 with (30m, 600m) which completes in about 12 hours (7+5) for a probability of 11.1%. With the new code, it is running (30m,4080m) in about 28 hours (7+21) for a probability of 14.5%.
However, I think I understand what you're saying. If I'm to spend 10 hours on a P-1, my best chance would not be a (30m, 600m) (7+3) split, but rather smaller B1, bigger B2.

Can you tell me what the exponent, TF level, and memory settings were? On the samples I've run I'm getting a B2 or 40x to 60x B1 (8GB memory). I'm curious if the B2=136*B1 is an indication of a bug.

Yes, you understand correctly. The program is saying (if the code is bug-free) that for a 10 hour run you'd be better off with a smaller B1 and larger B2.

I've got a fix in place for the stage 2 % complete. Thanks.

[firejuggler]

I may have misunderstood something.
Back in the day, for PM1, a reasonnable multiplier between B1 and B2 was 30.
More recently, with the advent of reliable PRP, that mult was lowered to 20.
And now that 'we' drop BS extension, we go back to a mult of 60, with a slightly lowered B1?

[masser]

Originally Posted by firejuggler

And now that 'we' drop BS extension, we go back to a mult of 60, with a slightly lowered B1?

I don't believe that is the case here. Now, we drop the BS extension and use an optimal multiplier, with optimal determined on a case-by-case basis given the available system memory and perhaps other hardware considerations.

[Prime95]

Originally Posted by firejuggler

I may have misunderstood something.
Back in the day, for PM1, a reasonnable multiplier between B1 and B2 was 30.
More recently, with the advent of reliable PRP, that mult was lowered to 20.
And now that 'we' drop BS extension, we go back to a mult of 60, with a slightly lowered B1?

Version 30.4 is more efficient at stage 2 than earlier versions. Thus, it makes sense that the the optimal B2/B1 ratio will go up.

I've not looked at wavefront PRP/LL tests where far fewer temporaries can be allocated (because the numbers are so much larger). Stage 2 efficiency will not increase nearly as much thus the best B2/B1 ratio will not go up as much.

Also, the 20 and 30 ratios you quote were guidelines. This is the first time we've gone to the effort of accurately predicting the best B2 value.

[axn]

Originally Posted by Prime95

Can you tell me what the exponent, TF level, and memory settings were? On the samples I've run I'm getting a B2 or 40x to 60x B1 (8GB memory). I'm curious if the B2=136*B1 is an indication of a bug.

These are the ones I tested:
M3699277
M3801709
M3802999
M3804763
M3804937
M3805183

TF depth was 69 bits. Memory allocated was 9.5 GB per worker. The prime-pairing % ranged about 90-93% between different blocks.

IF, for some reason, you want to test one of these exponent. let me know; I have save files from near the end of stage 1.

Incidentally, in the first run, I had allocated 24GB RAM, but without the per-worker restriction. The first worker entering stage 2 took all 24 GB RAM and selected about 200x multiplier!!

Originally Posted by Prime95

I've got a fix in place for the stage 2 % complete. Thanks.

Thanks. Looking forward to the next build. If there are no glaring issues, I will start using that for my "production" P-1. I am already using the current one for ECM.