Some CADO-NFS Work At Around 175-180 Decimal Digits

[charybdis]

Originally Posted by EdH

If you would like the 177 for experimentation and better comparison, it would be OK with me to let you have it. I can grab a 178 next. I can easily unreserve it on the HCN page. You can go ahead and start it and let me know later so I can stop my farming and unreserve it.

Thank you! I wouldn't want you to waste work but if you're happy to give it up then I'll start 5+2_415 with the same parameters that you were using.

[EdH]

Originally Posted by charybdis

Thank you! I wouldn't want you to waste work but if you're happy to give it up then I'll start 5+2_415 with the same parameters that you were using.

No problem at all! I've unreserved it. I will play elsewhere in a bit.

[charybdis]

Originally Posted by EdH

No problem at all! I've unreserved it. I will play elsewhere in a bit.

Thanks again!

Originally Posted by VBCurtis

If you had sieved your 32/32 job until you had 30% more unique rels than you had on the 31/32 job, I suspect the matrix would come out only barely bigger.

Let's have a look:

Mon Apr 20 13:10:09 2020  commencing relation filtering
Mon Apr 20 13:10:09 2020  setting target matrix density to 120.0
...
Mon Apr 20 13:48:51 2020  found 115949063 hash collisions in 374461390 relations
Mon Apr 20 13:49:13 2020  commencing duplicate removal, pass 2
Mon Apr 20 13:56:33 2020  found 158222008 duplicates and 216239382 unique relations
...
Mon Apr 20 15:10:51 2020  matrix is 12730379 x 12730604 (5886.1 MB) with weight 1588223894 (124.76/col)
Mon Apr 20 15:10:51 2020  sparse part has weight 1415690179 (111.20/col)
Mon Apr 20 15:10:51 2020  using block size 8192 and superblock size 884736 for processor cache size 9216 kB
Mon Apr 20 15:11:28 2020  commencing Lanczos iteration (6 threads)
Mon Apr 20 15:11:28 2020  memory use: 5541.4 MB
Mon Apr 20 15:11:59 2020  linear algebra at 0.0%, ETA 70h20m

The 31/32 job had 156M unique so this is 38% more unique relations than that one had. The matrix has almost identical dimensions to the 31/32 job, but I ran this one with a higher TD, so I'd need to sieve even more to get a truly comparable matrix.

[VBCurtis]

Originally Posted by charybdis

The 31/32 job had 156M unique so this is 38% more unique relations than that one had. The matrix has almost identical dimensions to the 31/32 job, but I ran this one with a higher TD, so I'd need to sieve even more to get a truly comparable matrix.

I appreciate this so much! Generating hard data, especially when it disproves my assumptions, just helps a great deal.

While I don't think going down to 31/31 for the c175 file makes sense, your data tells me I should only very slowly add LP sizes as we increase the size of the input. Perhaps still 31/32 on I=15 for c180, then 32/32 I = 15 for C185 and C190. Fivemack has tested 32 vs 33 (without testing 32/33 hybrid) at C193 and found 33 marginally faster for ggnfs, but we were ignoring matrix size; so I think I'd go with 32/33 for c195 and test I=15 against A=30.

Somewhere in the 180-190 range, 3 large primes on one side ought to become faster. I think we should test that starting soon; whomever does a C178-180 next, please post your poly so I can test-sieve a variety of 3LP scenarios.

[EdH]

Originally Posted by VBCurtis

. . .
Somewhere in the 180-190 range, 3 large primes on one side ought to become faster. I think we should test that starting soon; whomever does a C178-180 next, please post your poly so I can test-sieve a variety of 3LP scenarios.

Here's the poly CADO-NFS came up with for 12-7,945 (178 dd):

n: 1258632688840167527990479924759660967727113832014282715541202945214604444063933854224497640052542892497254083608311352572748398102592769128719761767229021194362137011186342088871
skew: 21118322.345
c0: 2400190655725017956973353574287095281260800
c1: 211014402684521569088239119813217640
c2: -39939568153496532334089408958
c3: -1552818850110399495593
c4: 58933721795178
c5: 294840
Y0: -21186307570697279371433463611848173
Y1: 8845196529223700328463
# MurphyE (Bf=4.295e+09,Bg=2.147e+09,area=2.684e+14) = 1.310e-07
# f(x) = 294840*x^5+58933721795178*x^4-1552818850110399495593*x^3-39939568153496532334089408958*x^2+211014402684521569088239119813217640*x+2400190655725017956973353574287095281260800
# g(x) = 8845196529223700328463*x-21186307570697279371433463611848173

========================
On a totally separate note, I appear to have discovered my polyselect stalls. It would seem that one of my clients is failing to return a WU and it is not replaced until the timeout is reached, at what point the new assignment of the WU has to start from scratch. Unfortunately, I was using a 12 hour timeout because some of my ancient machines are yet too young to stay up through the night and I was getting an overrun of maxtimedout for multi-day jobs. I must think out my strategy to cope with this. . .

[VBCurtis]

I have a little bit of test-sieving done on the c178 poly Ed provided a few posts ago.

I am testing a variety of mfb1 settings, leaving mfb0 untouched (at 31/lambda 1.88).
The tests are run 12-threaded, Q starting at 30M (somewhere around the middle third of relation gathering, seemed representative enough). I let CADO run an hour or so, record the number of relations found, Q-range done, and ETA. Workunits are in blocks of 1000; I thought that was small enough to reduce granularity, but it seems maybe not. When I change settings, I restart CADO and let it run 30-40 min to flush out the workunits with the old settings and shuffle up the end times of workers enough to reduce the effect of granular workunits.

Original: mfb1 60, lambda1 1.865. 112500 relations/hr, ETA 81 days 20 hr (for default 237M relations).
Run 2: mfb1 60, no lambda set. 134200 relations/hr, ETA 78 days 15 hr
Run 3: mfb1 64, no lambda. 167000 relations/hr, ETA 70 days 14 hr

Next I'll test a variety of mfb1 3-LP settings, from 88 to 94.

The catch with the apparent clear boost in speed is that using really tight lambda setting reduces relations required quite a lot. A typical 31/32LP job of this size would require 300-325M relations, but our runs require 260-270. I'm confused by the disparity between relations/hr and ETA; if I go by ETA I'm getting ~15% faster for mfb1=64, but I expect to need 15-18% more relations so it's a wash or small loss (though more of the relations would have larger LPs, so one might expect a larger matrix to turn up). If I go by relations/hr, 64 is a clear win. I think the ETA is more accurate, since it accounts for actual workunit length while I may have simply picked lucky endpoints where a bunch of WUs had just ended.

When sieving with ggnfs, using 3LP produces a dramatic dropoff in sec/rel as Q rises; so once I find the best MFB1 setting for 3LP here, I'll repeat the comparison at Q=80M for mfb60/64/whatever is best with 3LP. If that indicates a big change in relative timing, I'll do it again at Q=5M.

[swellman]

Originally Posted by EdH

Here's the poly CADO-NFS came up with for 12-7,945 (178 dd):

n: 1258632688840167527990479924759660967727113832014282715541202945214604444063933854224497640052542892497254083608311352572748398102592769128719761767229021194362137011186342088871
skew: 21118322.345
c0: 2400190655725017956973353574287095281260800
c1: 211014402684521569088239119813217640
c2: -39939568153496532334089408958
c3: -1552818850110399495593
c4: 58933721795178
c5: 294840
Y0: -21186307570697279371433463611848173
Y1: 8845196529223700328463
# MurphyE (Bf=4.295e+09,Bg=2.147e+09,area=2.684e+14) = 1.310e-07
# f(x) = 294840*x^5+58933721795178*x^4-1552818850110399495593*x^3-39939568153496532334089408958*x^2+211014402684521569088239119813217640*x+2400190655725017956973353574287095281260800
# g(x) = 8845196529223700328463*x-21186307570697279371433463611848173

========================

Ed - cow noise scores this poly as 1.5825e-13 with an optimal skew of 28173150.129. This is a new record for a C178. Record shattering in fact: the old record score for a C178 was 1.299e-13. Nicely done!

[VBCurtis]

Whoa, that's 20% better than the previous record!? 15% is a digit, so this is near the C177 record? I suppose I should look that up

(moments pass)

Yep! C177 record is 1.545e-13. So, the CADO poly select and a nice turn of luck took over a digit off the difficulty of this composite. Neat!

I imagine one of Carybdis' C177s broke that record, too.

[VBCurtis]

Originally Posted by VBCurtis

Run 1: mfb1 60, lambda1 1.865. 112500 relations/hr, ETA 81 days 20 hr (for default 237M relations). Yield 3.39
Run 2: mfb1 60, no lambda set. 134200 relations/hr, ETA 78 days 15 hr Yield 3.53
Run 3: mfb1 64, no lambda. 167000 relations/hr, ETA 70 days 14 hr Yield 4.46

I didn't explain my procedure very clearly: After changing settings in the snapshot file, I let CADO run for 30-40 min without timing anything. Then I set a timer for 45 min or 1 hour, and record the # of relations and Q-range processed; I also noted the ETA since that seems more stable as a forecast of sieve speed.
For 2LP runs, I used ncurves1 = 35. For 3LP, I'm using 12; I should test higher also. The ETA gain may be due in part to this drop in ncurves.

Run 4: mfb1 88, no lambda. 171000 rels/hr, ETA 62 days 5 hr. Yield 5.12
Run 5: mfb1 90, no lambda. 174500 rels/hr, ETA 59 days 15 hr. Yield 5.21

I added yield to the first 3 runs in the quote above.
I'll try mfb1 = 92 tomorrow. Yield is now so high that A=28 is even more likely to be faster than I=15; I'll test that too once I decide which mfb is likely to minimize ETA.

Mini-conclusion: Sieve speed up 25% over the params Ed is running, yield up 50%. However, unknown how many additional relations are needed without the tight lambda and with 3LP.

[charybdis]

Originally Posted by VBCurtis

Whoa, that's 20% better than the previous record!? 15% is a digit, so this is near the C177 record? I suppose I should look that up

(moments pass)

Yep! C177 record is 1.545e-13. So, the CADO poly select and a nice turn of luck took over a digit off the difficulty of this composite. Neat!

I imagine one of Carybdis' C177s broke that record, too.

Nope, seems Ed just got extremely lucky:

5-4_1085L  177 (1535...) 1.455e-13
12-11_759L 177 (5646...) 1.315e-13
5+2_415    177 (5721...) 1.440e-13

In fact, taking into account the difference in poly scores between my first two runs, that may account for the difference in speed I observed between 31/32 and 32/32...

It looks like I=15 31/32 is going to end up being a bit faster than A=28 31/32; I'll report more on this later today.

[EdH]

Originally Posted by charybdis

. . ., seems Ed just got extremely lucky:
. . .

What he said! However, the question forms:

Could there have been any possible advantage to my having 30+ totally separate machines searching (maybe close to 200 threads), vs. a few machines with many threads?

@VBCurtis: Are you taking into account duplication ratios? Is that something to even consider?

I will have to see what other poly's look like, if I can find/figure out cow noise. . .