Oops, I'm at 2800@43e6 myself. Oh well. Maybe yoyo should only do 17K@110e6 before switching to 42K@260e6.

A full t60 is obviously required, but it will take some analysis to find out how much of the t65 we should do.

While yoyo is handling the t55 and t60, I'll run some curves at 850e6 to get a CPU time analysis started.

VBCurtis/fivemack, what do we estimate the GNFS time will be? 2^(8/5)*GNFS187 ~= a bit over 300K thread hours?

[QUOTE=Dubslow;427867]
VBCurtis/fivemack, what do we estimate the GNFS time will be? 2^(8/5)*GNFS187 ~= a bit over 300K thread hours?[/QUOTE]

That depends on parameter choice. Will it be 15e at NFS@home? GNFS-195 is stretching 15e, so the time estimate should be padded a bit compared to the usual 16e choice for a number this size, say by 8-10%. 15e/33 should work okay, just a bit slower than 16/33.

I'll see if I can improve on this rough 325k thread-hr estimate by finding results from similar-sized factorizations on the forum.

After a fiasco in which I didn't realize how much memory stage 2 would take, causing my computer to lock up due to lack of ram (the post mortem swap-cleaning took ten minutes!), it seems that each curve takes around 1.77 thread hours at 850e6. Given that a t65 requires 69.4K curves, we're looking at roughly 123.3K thread hours for a full t65, and I estimate there's roughly an 8% chance of a factor existing in the 61-65 digit range, far less than the 30+% chance to make the whole t65 worth it by fivemack's rule.

Abusing the rule same as before would suggest ~30K thread hours at the t65 level (~17.5K@850e6), but I'm still not convinced how useful this "technique" is.

Repeat your analysis for a full t60 to first demonstrate that's worth it. If it's barely worth it to complete a t60, we can run just a couple thousand curves at 850M and call it good. If it's easily worth it to complete t60, then we know it's best to go on.

I'm not sure more than a full t60 is justified.

[QUOTE=VBCurtis;427901]Repeat your analysis for a full t60 to first demonstrate that's worth it. If it's barely worth it to complete a t60, we can run just a couple thousand curves at 850M and call it good. If it's easily worth it to complete t60, then we know it's best to go on.

I'm not sure more than a full t60 is justified.[/QUOTE]

Actually I was just doing that. I'm working up a somewhat more generalized formula that thinks along similar lines, and the numbers I got suggested that even 1000 curves at 850e6 weren't worth it... so I started some test curves at 260e6 around half an hour ago.

[code]In [2]: for n in range(5, 75, 5):
print("{}->{} simple odds: {} better odds: {}".format(n, n+5, 1/(n/5), 1/(n+1)+1/(n+2)+1/(n+3)+1/(n+4)+1/(n+5)))
...:
5->10 simple odds: 1.0 better odds: 0.6456349206349207
10->15 simple odds: 0.5 better odds: 0.38926073926073923
15->20 simple odds: 0.3333333333333333 better odds: 0.27951066391468865
20->25 simple odds: 0.25 better odds: 0.21821852060982494
25->30 simple odds: 0.2 better odds: 0.1790289531668842
30->35 simple odds: 0.16666666666666666 better odds: 0.15179428809647028
35->40 simple odds: 0.14285714285714285 better odds: 0.13176161991951466
40->45 simple odds: 0.125 better odds: 0.11640507661494617
45->50 simple odds: 0.1111111111111111 better odds: 0.10425722277810291
50->55 simple odds: 0.1 better odds: 0.09440687359666272
55->60 simple odds: 0.09090909090909091 better odds: 0.08625820102565003
60->65 simple odds: 0.08333333333333333 better odds: 0.0794051061386466
65->70 simple odds: 0.07692307692307693 better odds: 0.07356123854768738
70->75 simple odds: 0.07142857142857142 better odds: 0.06851887291497556

In [5]: scale=1/69408

In [10]: from math import expm1

In [11]: cdf=lambda x: -expm1(-x*scale)

In [12]: odds=lambda x: 0.08*cdf(x)

In [18]: for i in range(1000, 70000, 1000):
print(i, i*1.77, odds(i), i*1.77/odds(i))
....:
1000 1770.0 0.0011443415070364752 1546741.0638488552
2000 3540.0 0.0022723140455138693 1557883.2542926308
3000 5310.0 0.0033841517615590512 1569078.5680231215
4000 7080.0 0.004480085452009743 1580326.9995271962
5000 8850.0 0.0055603426123236556 1591628.5410876153
6000 10620.0 0.006625147483802312 1602983.1827841753
7000 12390.0 0.007674721100139371 1614390.9124951784
8000 14160.0 0.008709281333303119 1625851.7158992288
9000 15930.0 0.009729042938762636 1637365.5764773528
10000 17700.0 0.010734217600067033 1648932.4755154457
11000 19470.0 0.011725013972787031 1660552.3921070423
12000 21240.0 0.012701637727827973 1672225.3031564078
13000 23010.0 0.013664291594123273 1683951.183381956
14000 24780.0 0.014613175400717188 1695730.005319983
15000 26550.0 0.015548486118245598 1707561.7393287257
16000 28320.0 0.016470417899823463 1719446.353592737
17000 30090.0 0.01737916212134738 1731383.8141275805
18000 31860.0 0.018274907421221682 1743374.0847848384
19000 33630.0 0.019157839739516246 1755417.1272574384
20000 35400.0 0.020028142356564204 1767512.9010852913
21000 37170.0 0.020885995931007532 1779661.3636612413
22000 38940.0 0.021731578537298422 1791862.4702373256
23000 40710.0 0.022565065702664228 1804116.1739313449
24000 42480.0 0.02338663044354366 1816422.4257337356
25000 44250.0 0.02419644330150176 1828781.1745147523
26000 46020.0 0.024994672378631195 1841192.3670319472
27000 47790.0 0.025781483372447102 1853655.9479379528
28000 49560.0 0.0265570396102828 1866171.8597885633
29000 51330.0 0.027321502083193547 1878740.0430511087
30000 53100.0 0.028075029479375246 1891360.4361131247
31000 54870.0 0.028817778217105197 1904032.9752913131
32000 56640.0 0.02954990247721161 1916757.594840789
33000 58410.0 0.03027155423507867 1929534.226964617
34000 60180.0 0.030982883292193817 1942362.8018236263
35000 61950.0 0.03168403730724374 1955243.247546509
36000 63720.0 0.03237516182676557 1968175.4902401958
37000 65490.0 0.03305640031535967 1981159.4540005026
38000 67260.0 0.03372789418547015 1994195.0609230553
39000 69030.0 0.034389782826739525 2007282.2311144758
40000 70800.0 0.0350422036349434 2020420.8827038386
41000 72570.0 0.03568529204051125 2033610.9318543866
42000 74340.0 0.036319181536639274 2046852.2927755082
43000 76110.0 0.03694400370700114 2060144.877734966
44000 77880.0 0.03755988825306223 2073488.597071385
45000 79650.0 0.03816696302100332 2086883.35920698
46000 81420.0 0.038765354028259036 2100329.0706605366
47000 83190.0 0.039355185489676765 2113825.6360606286
48000 84960.0 0.039936579843301276 2127372.958159077
49000 86730.0 0.04050965777579068 2140970.9378446406
50000 88500.0 0.04107453824746865 2154619.4741569394
51000 90270.0 0.041631338517018425 2168318.4643006045
52000 92040.0 0.04218017416582352 2182067.803659649
53000 93810.0 0.042721159121960256 2195867.3858120623
54000 95580.0 0.04325440568384712 2209717.1025446155
55000 97350.0 0.04378002454355584 2223616.8438678808
56000 99120.0 0.04429812480978898 2237566.4980314584
57000 100890.0 0.04480881403052891 2251565.9515394038
58000 102660.0 0.04531219821536273 2265615.0891658566
59000 104430.0 0.04580838185748791 2279713.793970867
60000 106200.0 0.04629746795540313 2293861.9473164077
61000 107970.0 0.04677955803428887 2308059.428882574
62000 109740.0 0.04725475216708211 2322306.116683973
63000 111510.0 0.04772314899524966 2336601.8870862788
64000 113280.0 0.048184845749264266 2350946.614822974
65000 115050.0 0.04863993826878782 2365340.173012255
66000 116820.0 0.04908852102256597 2379782.4331741
67000 118590.0 0.04953068712803802 2394273.265247502
68000 120360.0 0.04996652837066635 2408812.537607861
69000 122130.0 0.05039613522298947 2423400.1170845204[/code]

Edit: At 260e6 I got around 2040 seconds per curve, resulting in the following analysis:

[code]In [27]: scale=1/42017

In [28]: cdf=lambda x: -expm1(-x*scale)

In [29]: odds=lambda x: 0.08625820102565003*cdf(x)

In [30]: workpercurve=2040/3600

In [32]: for i in range(1000, 45000, 1000):
print(i, i*workpercurve, odds(i), i*workpercurve/odds(i))
....:
1000 566.6666666666666 0.0020286985793124416 279325.2149162144
2000 1133.3333333333333 0.004009684386079575 282649.0127921109
3000 1700.0 0.005944079572545787 285998.8631127809
4000 2266.6666666666665 0.007832979899161562 289374.7585014599
5000 2833.3333333333335 0.009677455355289617 292776.6886348545
6000 3400.0 0.01147855076531271 296204.6402473156
7000 3966.6666666666665 0.013237286380486461 299658.5971361975
8000 4533.333333333333 0.014954658456872423 303138.5401684006
9000 5100.0 0.01663163981967883 306644.4472880898
10000 5666.666666666667 0.018269180414328623 310176.2935255852
11000 6233.333333333333 0.01986820784456702 313734.05100741604
12000 6800.0 0.021429627897913313 317317.68896753184
13000 7366.666666666666 0.02295432505875468 320927.17375965934
14000 7933.333333333333 0.024443163009372527 324562.4688707989
15000 8500.0 0.025896985119185326 328223.53493584564
16000 9066.666666666666 0.027316614922484873 331910.32975332916
17000 9633.333333333334 0.028702856584936844 335622.80830225354
18000 10200.0 0.030056495359109654 339360.9227600296
19000 10766.666666666666 0.031378298029289875 343124.62252148247
20000 11333.333333333334 0.032669013345835995 346913.85421892104
21000 11900.0 0.033929372449316694 350728.5617432531
22000 12466.666666666666 0.03516008928467388 354568.6862661304
23000 13033.333333333332 0.03636186100564498 358434.1662631068
24000 13600.0 0.037535368369673756 362324.93753779045
25000 14166.666666666666 0.03868127612353318 366240.93324697355
26000 14733.333333333332 0.0398002333798789 370182.083926719
27000 15300.0 0.04089287398494662 374148.31751938484
28000 15866.666666666666 0.04195981687760157 378139.55940156634
29000 16433.333333333332 0.04300166643994358 382155.7324129342
30000 17000.0 0.044019012839666284 386196.7568859475
31000 17566.666666666668 0.045012432364364405 390262.55067641946
32000 18133.333333333332 0.04598248774797851 394353.02919491375
33000 18700.0 0.04692972848956215 398468.10543894686
34000 19266.666666666668 0.04785469116455192 402607.69002597465
35000 19833.333333333332 0.04875789972871677 406771.691227138
36000 20400.0 0.04963986581495888 410960.01500174275
37000 20966.666666666668 0.0505010890231339 415172.56503245124
38000 21533.333333333332 0.051342057203055125 419409.2427611565
39000 22100.0 0.05216324673084163 423669.94742551807
40000 22666.666666666668 0.052965122778767 427954.5760961292
41000 23233.333333333332 0.05374813957876154 432263.023714293
42000 23800.0 0.05451274067971714 436595.18313037965
43000 24366.666666666664 0.05525935919874078 440950.94514273555
44000 24933.333333333332 0.05598841806649855 445330.1985371246[/code]

If you ignore the fact that I confused the labels "scale" and "rate", this suggests we should be doing only 8K-16K curves at the t60 level. Of course, this is all assuming that my thread hours and fivemack's GNFS thread hours are comparable. There's obviously a fair bit of wiggle room that depends on precisely how accurate our GNFS work estimate is.

Edit2: Of course, this doesn't include the odds that the prior ECM, presumably only run to one t-level, still left a factor of less than the considered size. That would change the crossover here a bit I think. (Unless that's counter-balanced by the odds that said same prior ECM finds a factor larger than its "meant" to?)

Edit3: These were all run with -maxmem 1500, which should only affect the 850e6 estimate I think.

As it turns out, including the odds of finding a factor the previous level missed does substantially affect the ECM estimate. The third argument is how much of the missed factor odds (exp(-1)*odds of factor at previous level) to add to the odds of a factor existing at this level. It is an arbitrary way of accounting for the fact that the previous level might've found factors at this level.

[code]In [1]: %run nfsecm.py 60 325000 0
With roughly 8.625820% odds of a factor existing, you should do 16312 curves at 60 digits before switching to NFS

In [2]: %run nfsecm.py 60 325000 0.5
With roughly 10.362337% odds of a factor existing, you should do 31373 curves at 60 digits before switching to NFS

In [3]: %run nfsecm.py 60 325000 1
With roughly 12.098855% odds of a factor existing, you should do 46382 curves at 60 digits before switching to NFS

In [4]: %run nfsecm.py 65 325000 0
Not even one curve should be done, forget about ECM at 65 digits.

In [5]: %run nfsecm.py 65 325000 0.5
Not even one curve should be done, forget about ECM at 65 digits.

In [6]: %run nfsecm.py 65 325000 1
Not even one curve should be done, forget about ECM at 65 digits.[/code]

[url]https://gist.github.com/dubslow/916cb29e30277380c9f2[/url]

Edit: Here's how the numbers react to variance in the estimated NFS effort:

[code]In [10]: %run nfsecm.py 60 325000 0
With roughly 8.625820% odds of a factor existing, you should do 16312 curves at 60 digits before switching to NFS

In [11]: %run nfsecm.py 60 330000 0
With roughly 8.625820% odds of a factor existing, you should do 17463 curves at 60 digits before switching to NFS

In [12]: %run nfsecm.py 60 335000 0
With roughly 8.625820% odds of a factor existing, you should do 18614 curves at 60 digits before switching to NFS

In [13]: %run nfsecm.py 60 340000 0
With roughly 8.625820% odds of a factor existing, you should do 19765 curves at 60 digits before switching to NFS

In [14]: %run nfsecm.py 60 345000 0
With roughly 8.625820% odds of a factor existing, you should do 20916 curves at 60 digits before switching to NFS

In [15]: %run nfsecm.py 60 350000 0
With roughly 8.625820% odds of a factor existing, you should do 22067 curves at 60 digits before switching to NFS

In [16]: %run nfsecm.py 60 320000 0
With roughly 8.625820% odds of a factor existing, you should do 15161 curves at 60 digits before switching to NFS

In [17]: %run nfsecm.py 60 315000 0
With roughly 8.625820% odds of a factor existing, you should do 14010 curves at 60 digits before switching to NFS

In [18]: %run nfsecm.py 60 310000 0
With roughly 8.625820% odds of a factor existing, you should do 12860 curves at 60 digits before switching to NFS

In [19]: %run nfsecm.py 60 305000 0
With roughly 8.625820% odds of a factor existing, you should do 11709 curves at 60 digits before switching to NFS

In [20]: %run nfsecm.py 60 300000 0
With roughly 8.625820% odds of a factor existing, you should do 10558 curves at 60 digits before switching to NFS
[/code]

The good news is that the curve around the total NFS effort is relatively flat, getting within ~5K curves is pretty damn close to optimal. The missed factor odds are rather more volatile, and I guess I'd like more feedback about that.

I view the case for 1k-3k curves at 850M after a t60 as a sort of "cleanup" of those missed factors under 60 digits that a t60 might've missed. But, I usually stop before a complete t60 is done and substitute the 850M curves for the last few thousand 260M curves, on the premise that the increased chance of larger factors makes up for the slight inefficiency in using 850M curves to complete a t60.

Great analysis! Here's how I look at those missed factors vs "extra" factors for a t60:
a t55 has a 1/e chance to miss a 55 digit factor. A t60 is about 6t55, so that same missed factor has roughly 1/e^5 chance to still be missed; almost zero. So, a t60 is going to find 1/e of the 55-digit factors (the ones missed by the t55).

A t55 is 2t53, so a 53-digit factor is missed 1/e^2 of the time. the t60 will find those for "sure".

However, the t60 will miss 1/e of the 60-digit factors, so the extra 55 digit factors found are balanced by the missed 60-digit factors. Likewise for 53 vs 58 digit factors; a t60 misses 1/e^2 of the 58 digit factors, but finds the (roughly) same number of 53 digit factors that were missed by the t55.

So, your edit 2 had the right idea- the extra factors picked up at a smaller level come very close to canceling the missed factors in the 5-digit range you're searching. This disregards the "bonus" higher factors, but the t55 would have picked up some bonus factors in your 55-to-60 digit range too- the cancellation argument works the same as before.

So, the method we usually use of pretending all 55 to 60 digit factors will be found by a t60 is pretty accurate for estimating the actual number of factors that will be found- it's just that a fair number of the factors will be 53-55 digits rather than 58-60 digits.

Edit- on projects this big, I think the parallelization of ECM itself has value- that is, solving the matrix requires high-value hardware, while ECM does not. I think that makes it worth doing extra ECM work on the cheap hardware on the chance it saves having to do the matrix on said high-value hardware. Say, 10-20% more curves than strict efficiency arguments call for.

[QUOTE=VBCurtis;427951]A t60 is about 6t55,...
A t55 is 2t53, ...[/QUOTE]

This is one thing I still don't have a good feel for. How do you calculate this?

As for the rest, I'm not entirely sure I made myself clear -- the code accounts for the odds that the t60 misses a 56-60 digit factor. It makes substantial use of the CDF = 1-exp(-curves/median). That is to say, in your sentence here:
[quote=VBCurtis;427951]However, the t60 will miss 1/e of the 60-digit factors, so the extra 55 digit factors found are balanced by the missed 60-digit factors.[/quote]
My code already accounts for the missed 60 digit factors, but not for the missed 55 digit factors (when the third argument is 0), so there is not any such balance.

I guess what I'm asking is, when you've run a t55 but nothing at t60, how much of a t60 is "already" accounted for by a t55? By your first quoted sentence above, 1t55= 1/6 t60...?

Here's the relevant code:

[PASTEBIN]CDdeJsqa[/PASTEBIN]
And...
[PASTEBIN]5pR4AUHc[/PASTEBIN]

[QUOTE=Dubslow;427971]This is one thing I still don't have a good feel for. How do you calculate this?


I guess what I'm asking is, when you've run a t55 but nothing at t60, how much of a t60 is "already" accounted for by a t55? By your first quoted sentence above, 1t55= 1/6 t60...?
[/QUOTE]

I ask GMP-ECM, by comparing the curve counts at a specific B1 value required for a t55 and a t60.
At B1 = 260M, a t55 is 8656 curves while a t60 is 47888 curves. So, when one completes the 47888 curves for a t60, one has done 47888/8656 = 5.5ish t55.

At B1 = 110M, a t55 is 20479 curves, while a t60 is 128305 curves. So, when one completes 20479 curves for a t55, one has completed 20479/128305 of a t60, or just under 1/6th.

To estimate t53 and such, I use a geometric interpolation: if ECM effort doubles every two digits, 5 digits higher would be 2.5 doublings. 2^2.5 is around 5.7, right in between the two ratios mentioned above. So, a rough guide for estimating t53 is half a t55, t57 is 2*t55, t59 is 2*t57 is 4*t55, and t60 is sqrt2*t59 is 4sqrt2 * t55 is 5.7*t55.

Again, the 5.7 is a rounded version of the first calculation I did.

The ratio changes for various B1 values, but not by much. For instance, I use B1 = 150M to run my t55s; a t55 is 14356 curves, whlie a t60 is 86058. On my machine, 14356 curves at 150M run faster than 20479 curves at 110M, while also completing a teensy bit more of a t60 than the set of curves at 110M (i.e. there's a bit more chance of a "bonus" factor above t55).

So bottom line...

t55, followed by t60, followed by ?t65. How many curves?

For comparisons sake, see [url=http://www.mersenneforum.org/showpost.php?p=404530&postcount=1]this post[/url] for a list of GNFS composites with ECM preprocessing requirements

Ultimately, the required level of ECM will be whatever the NFS@Home gatekeeper says it is.:smile:

[QUOTE=swellman;428044]So bottom line...

t55, followed by t60, followed by ?t65. How many curves?

For comparisons sake, see [url=http://www.mersenneforum.org/showpost.php?p=404530&postcount=1]this post[/url] for a list of GNFS composites with ECM preprocessing requirements

Ultimately, the required level of ECM will be whatever the NFS@Home gatekeeper says it is.:smile:[/QUOTE]

By my analysis, we should do no ECM at t65, and in fact the utility of a full t60 is in question, though I guess we'll likely wind up doing a full t60, or perhaps change the last few K of the t60 into higher level curves. I would argue that at the higher end at least, that XYYXF table is skewed pretty heavily into the too-much-ECM territory.

Yes, the gatekeepers are who we ultimately need please.