Parameter explorations for CADO 165-170 digits
 

This thread explores parameter options for 163-172 digit numbers.

Things to determine:
What siever? I=14 vs A=28 vs I=15
2LP or 3LP on side 1?
For 2LP side(s), what lambda?
What lims?

I've attached a first guess at c170 params; As we find faster settings I'll update the file(s) on this first post so best-practices are easy to locate.

First guess: A=28, 3LP, lambda0=1.88 (corresponding to mfb0 = 58.3). Lims possibly too high, at 80M/115M. We should try one much smaller also, like 60/85M.

I've been sieving the c168 from 5+3_1215L with these parameters, as suggested in the 175-180 thread:

[code]tasks.A = 28
tasks.qmin = 10000000
tasks.lim0 = 80000000
tasks.lim1 = 115000000
tasks.lpb0 = 31
tasks.lpb1 = 31
tasks.sieve.mfb0 = 58
tasks.sieve.mfb1 = 90
tasks.sieve.ncurves0 = 20
tasks.sieve.ncurves1 = 13[/code]

19.1M CPU-seconds of sieving produced the following:

[code]Wed May 13 12:46:32 2020 Msieve v. 1.54 (SVN 1030M)
Wed May 13 12:46:32 2020 random seeds: 9ffee2f6 41951ee0
Wed May 13 12:46:32 2020 factoring 245609880520494362682705606347409092539758689544648910766910287096779616823469115034356435763879242747931684930827717014249287135786561075491831877899088616371815708911 (168 digits)
Wed May 13 12:46:33 2020 searching for 15-digit factors
Wed May 13 12:46:33 2020 commencing number field sieve (168-digit input)
Wed May 13 12:46:33 2020 R0: -223470566836941710449157486820982
Wed May 13 12:46:33 2020 R1: 79664348591951467630451
Wed May 13 12:46:33 2020 A0: -7589872577078303699524772499775295445
Wed May 13 12:46:33 2020 A1: -196971342286318522372786728569142
Wed May 13 12:46:33 2020 A2: 55213576168005775518349496
Wed May 13 12:46:33 2020 A3: -122057413788280381324
Wed May 13 12:46:33 2020 A4: -6005969398245
Wed May 13 12:46:33 2020 A5: 1324260
Wed May 13 12:46:33 2020 skew 1.00, size 2.278e-16, alpha -6.095, combined = 5.596e-15 rroots = 3
Wed May 13 12:46:33 2020
Wed May 13 12:46:33 2020 commencing relation filtering
Wed May 13 12:46:33 2020 setting target matrix density to 100.0
Wed May 13 12:46:33 2020 estimated available RAM is 15845.9 MB
Wed May 13 12:46:33 2020 commencing duplicate removal, pass 1
Wed May 13 13:04:10 2020 found 40021465 hash collisions in 189567218 relations
Wed May 13 13:04:32 2020 commencing duplicate removal, pass 2
Wed May 13 13:07:48 2020 found 43897302 duplicates and 145669916 unique relations
Wed May 13 13:07:48 2020 memory use: 1321.5 MB
Wed May 13 13:07:49 2020 reading ideals above 89194496
Wed May 13 13:07:49 2020 commencing singleton removal, initial pass
Wed May 13 13:18:08 2020 memory use: 3012.0 MB
Wed May 13 13:18:08 2020 reading all ideals from disk
Wed May 13 13:18:24 2020 memory use: 2748.9 MB
Wed May 13 13:18:28 2020 commencing in-memory singleton removal
Wed May 13 13:18:32 2020 begin with 145669916 relations and 140458395 unique ideals
Wed May 13 13:19:16 2020 reduce to 72650875 relations and 58132947 ideals in 17 passes
Wed May 13 13:19:16 2020 max relations containing the same ideal: 28
Wed May 13 13:19:21 2020 reading ideals above 720000
Wed May 13 13:19:22 2020 commencing singleton removal, initial pass
Wed May 13 13:26:59 2020 memory use: 1506.0 MB
Wed May 13 13:27:00 2020 reading all ideals from disk
Wed May 13 13:27:25 2020 memory use: 2815.9 MB
Wed May 13 13:27:30 2020 keeping 68118732 ideals with weight <= 200, target excess is 360258
Wed May 13 13:27:36 2020 commencing in-memory singleton removal
Wed May 13 13:27:40 2020 begin with 72650877 relations and 68118732 unique ideals
Wed May 13 13:28:22 2020 reduce to 72634839 relations and 68102690 ideals in 10 passes
Wed May 13 13:28:22 2020 max relations containing the same ideal: 200
Wed May 13 13:28:48 2020 removing 7760595 relations and 6760595 ideals in 1000000 cliques
Wed May 13 13:28:50 2020 commencing in-memory singleton removal
Wed May 13 13:28:54 2020 begin with 64874244 relations and 68102690 unique ideals
Wed May 13 13:29:39 2020 reduce to 64141232 relations and 60595322 ideals in 12 passes
Wed May 13 13:29:39 2020 max relations containing the same ideal: 191
Wed May 13 13:30:01 2020 removing 5952038 relations and 4952038 ideals in 1000000 cliques
Wed May 13 13:30:03 2020 commencing in-memory singleton removal
Wed May 13 13:30:06 2020 begin with 58189194 relations and 60595322 unique ideals
Wed May 13 13:30:40 2020 reduce to 57683158 relations and 55128834 ideals in 10 passes
Wed May 13 13:30:40 2020 max relations containing the same ideal: 181
Wed May 13 13:31:01 2020 removing 5427833 relations and 4427833 ideals in 1000000 cliques
Wed May 13 13:31:02 2020 commencing in-memory singleton removal
Wed May 13 13:31:05 2020 begin with 52255325 relations and 55128834 unique ideals
Wed May 13 13:31:32 2020 reduce to 51779832 relations and 50217186 ideals in 9 passes
Wed May 13 13:31:32 2020 max relations containing the same ideal: 169
Wed May 13 13:31:51 2020 removing 5205947 relations and 4205947 ideals in 1000000 cliques
Wed May 13 13:31:52 2020 commencing in-memory singleton removal
Wed May 13 13:31:55 2020 begin with 46573885 relations and 50217186 unique ideals
Wed May 13 13:32:19 2020 reduce to 46074717 relations and 45502789 ideals in 9 passes
Wed May 13 13:32:19 2020 max relations containing the same ideal: 158
Wed May 13 13:32:35 2020 removing 1163580 relations and 1009552 ideals in 154028 cliques
Wed May 13 13:32:36 2020 commencing in-memory singleton removal
Wed May 13 13:32:38 2020 begin with 44911137 relations and 45502789 unique ideals
Wed May 13 13:32:54 2020 reduce to 44886292 relations and 44468291 ideals in 6 passes
Wed May 13 13:32:54 2020 max relations containing the same ideal: 157
Wed May 13 13:33:16 2020 relations with 0 large ideals: 1286
Wed May 13 13:33:16 2020 relations with 1 large ideals: 1573
Wed May 13 13:33:16 2020 relations with 2 large ideals: 30197
Wed May 13 13:33:16 2020 relations with 3 large ideals: 293969
Wed May 13 13:33:16 2020 relations with 4 large ideals: 1561336
Wed May 13 13:33:16 2020 relations with 5 large ideals: 4986951
Wed May 13 13:33:16 2020 relations with 6 large ideals: 10038808
Wed May 13 13:33:16 2020 relations with 7+ large ideals: 27972172
Wed May 13 13:33:16 2020 commencing 2-way merge
Wed May 13 13:33:38 2020 reduce to 27425585 relation sets and 27007584 unique ideals
Wed May 13 13:33:38 2020 commencing full merge
Wed May 13 13:39:32 2020 memory use: 2958.8 MB
Wed May 13 13:39:34 2020 found 12476332 cycles, need 12451784
Wed May 13 13:39:37 2020 weight of 12451784 cycles is about 1245332899 (100.01/cycle)
Wed May 13 13:39:37 2020 distribution of cycle lengths:
Wed May 13 13:39:37 2020 1 relations: 1167437
Wed May 13 13:39:37 2020 2 relations: 1129539
Wed May 13 13:39:37 2020 3 relations: 1141919
Wed May 13 13:39:37 2020 4 relations: 1059954
Wed May 13 13:39:37 2020 5 relations: 1004089
Wed May 13 13:39:37 2020 6 relations: 931457
Wed May 13 13:39:37 2020 7 relations: 840380
Wed May 13 13:39:37 2020 8 relations: 749799
Wed May 13 13:39:37 2020 9 relations: 687709
Wed May 13 13:39:37 2020 10+ relations: 3739501
Wed May 13 13:39:37 2020 heaviest cycle: 28 relations
Wed May 13 13:39:39 2020 commencing cycle optimization
Wed May 13 13:39:54 2020 start with 92367168 relations
Wed May 13 13:41:53 2020 pruned 2737568 relations
Wed May 13 13:41:54 2020 memory use: 2787.8 MB
Wed May 13 13:41:54 2020 distribution of cycle lengths:
Wed May 13 13:41:54 2020 1 relations: 1167437
Wed May 13 13:41:54 2020 2 relations: 1158374
Wed May 13 13:41:54 2020 3 relations: 1184944
Wed May 13 13:41:54 2020 4 relations: 1091585
Wed May 13 13:41:54 2020 5 relations: 1035797
Wed May 13 13:41:54 2020 6 relations: 952023
Wed May 13 13:41:54 2020 7 relations: 858487
Wed May 13 13:41:54 2020 8 relations: 761081
Wed May 13 13:41:54 2020 9 relations: 694773
Wed May 13 13:41:54 2020 10+ relations: 3547283
Wed May 13 13:41:54 2020 heaviest cycle: 28 relations
Wed May 13 13:42:13 2020 RelProcTime: 3340
Wed May 13 13:42:17 2020
Wed May 13 13:42:17 2020 commencing linear algebra
Wed May 13 13:42:18 2020 read 12451784 cycles
Wed May 13 13:42:37 2020 cycles contain 44543120 unique relations
Wed May 13 13:46:53 2020 read 44543120 relations
Wed May 13 13:47:48 2020 using 20 quadratic characters above 4294917295
Wed May 13 13:50:38 2020 building initial matrix
Wed May 13 13:57:23 2020 memory use: 5986.7 MB
Wed May 13 13:58:04 2020 read 12451784 cycles
Wed May 13 13:58:06 2020 matrix is 12451607 x 12451784 (5093.3 MB) with weight 1553697525 (124.78/col)
Wed May 13 13:58:06 2020 sparse part has weight 1185747861 (95.23/col)
Wed May 13 14:00:02 2020 filtering completed in 2 passes
Wed May 13 14:00:04 2020 matrix is 12450174 x 12450351 (5093.2 MB) with weight 1553641047 (124.79/col)
Wed May 13 14:00:04 2020 sparse part has weight 1185735705 (95.24/col)
Wed May 13 14:01:07 2020 matrix starts at (0, 0)
Wed May 13 14:01:09 2020 matrix is 12450174 x 12450351 (5093.2 MB) with weight 1553641047 (124.79/col)
Wed May 13 14:01:09 2020 sparse part has weight 1185735705 (95.24/col)
Wed May 13 14:01:09 2020 saving the first 48 matrix rows for later
Wed May 13 14:01:10 2020 matrix includes 64 packed rows
Wed May 13 14:01:11 2020 matrix is 12450126 x 12450351 (4936.6 MB) with weight 1301689606 (104.55/col)
Wed May 13 14:01:11 2020 sparse part has weight 1169600559 (93.94/col)
Wed May 13 14:01:11 2020 using block size 8192 and superblock size 884736 for processor cache size 9216 kB
Wed May 13 14:01:42 2020 commencing Lanczos iteration (6 threads)
Wed May 13 14:01:43 2020 memory use: 4708.1 MB
Wed May 13 14:02:09 2020 linear algebra at 0.0%, ETA 56h57m[/code]

This strikes me as being a bit slower than we want; I was getting times of 55-60M CPU-seconds at c178. It also doesn't compare too well with a c167 that I ran in March with the parameters
[code]tasks.I = 14
tasks.qmin = 7000000
tasks.lim0 = 38000000
tasks.lim1 = 60000000
tasks.lpb0 = 31
tasks.lpb1 = 32
tasks.sieve.mfb0 = 62
tasks.sieve.mfb1 = 64
tasks.sieve.ncurves0 = 18
tasks.sieve.ncurves1 = 25[/code]
which produced a 9M matrix after 16M CPU-seconds of sieving (not entirely comparable as I didn't run it on exactly the same machines, but if anything it would be even faster on the ones I'm using at the moment).

The obvious difference between these two jobs is 2LP vs 3LP, though the lims could also be having an effect - we need more runs to see what's really going on. Does anyone have some more numbers of this size hanging around?

Kamada’s site has some suitable GNFS jobs available

[url]https://stdkmd.net/nrr/wanted.htm#suitableforgnfs[/url]

I would say that both observations are correct- lims are too big, and 3LP may not be suitable. It's also likely that I=14 is better than A=28; it may be that these even A-values are only faster in unusual cases (say, a low-yielding poly that would normally be a size to use I=14).

That said, 20% slower for 1 digit larger isn't a huge miss; clearly not faster, but not slower-enough to rule out 3LP as still possibly a good idea. The larger-than-your-c167 matrix tells us you didn't oversieve, and we know 3LP matrices turn out bigger.

If we go 2LP on both sides, I suggest also using lambda0 & lambda1 of 1.85 or so to reduce the number of relations needed.

So, possible changes:
I=14
lim's of, say, 50M and 70M
2LP

My notes include a C167 sent to nfs@home 14e queue, in which I used lim's of 67M on both sides. ggnfs likes power-of-2 lim's, but that indicates 50/70 are more likely correct than 80/110.

[QUOTE=charybdis;545232]Does anyone have some more numbers of this size hanging around?[/QUOTE]

Here are the smallest 3 numbers in the Brent tables:
[code]
(95^128+1)/267800772103727976108674546747470814168968227593565342652934454279571759350601155586
# 170 digits:
52578676770574634512595466847841062400406219898703178016790505924363888689365475143019601569100893681427169623580739917336603746042987149267328695780988068362551904560641
[/code]

[code]
(24^179+1)/43663605833505119109437514756662567033795467630332680195392449662667612167525
# 171 digits:
261633088864353542789446012422373060420258849456315466227530404025437348932344174393051880283883795455352667032238196892137483412153757222899501668268383799664562709253229
[/code]

[code]
(86^131-1)/127296350506000474708670512286771610891902627605122671646543238329335622840049145135
# 171 digits:
206291417462093576497275239580773052795596055350565606087502662805051224044201602337850673455835775692863740037805407606265476824913591751307682500480113600344276700606001
[/code]

If you don't want them I can do them with NFS@Home.

Chris

[QUOTE=chris2be8;545254]Here are the smallest 3 numbers in the Brent tables:
<snip>

If you don't want them I can do them with NFS@Home.

Chris[/QUOTE]

Thank you! I'd be happy to do at least some of these, though I'll be doing the c170 from Aliquot 3366 first if it survives ECM. I take it all of these numbers have had enough ECM?

As far as I know everything in the Brent tables has been ECMed to at least t60.

Chris

The c170 from Aliquot 3366 has built a matrix:

[code]Fri May 15 13:30:04 2020 Msieve v. 1.54 (SVN 1030M)
Fri May 15 13:30:04 2020 random seeds: 85bae23a bde7ff6e
Fri May 15 13:30:04 2020 factoring 86631352523777317443318741611426541309296773170541288874048777015161155735273728068286738505405324469613265780485465094938752555194870805288145702027767872412047031948633 (170 digits)
Fri May 15 13:30:05 2020 searching for 15-digit factors
Fri May 15 13:30:05 2020 commencing number field sieve (170-digit input)
Fri May 15 13:30:05 2020 R0: -997917629737604869042704488941047
Fri May 15 13:30:05 2020 R1: 2231945554206892494565181
Fri May 15 13:30:05 2020 A0: 8911247040789664012374838655454593624736
Fri May 15 13:30:05 2020 A1: 58966576361359993167845008084069822
Fri May 15 13:30:05 2020 A2: -14226780425800550539646724351
Fri May 15 13:30:05 2020 A3: -1607926381901953698037
Fri May 15 13:30:05 2020 A4: 79303929919170
Fri May 15 13:30:05 2020 A5: 2457000
Fri May 15 13:30:05 2020 skew 1.00, size 1.375e-16, alpha -8.660, combined = 2.335e-15 rroots = 5
Fri May 15 13:30:05 2020
Fri May 15 13:30:05 2020 commencing relation filtering
Fri May 15 13:30:05 2020 setting target matrix density to 100.0
Fri May 15 13:30:05 2020 estimated available RAM is 15845.9 MB
Fri May 15 13:30:05 2020 commencing duplicate removal, pass 1
...
Fri May 15 13:50:46 2020 found 59360613 hash collisions in 213148587 relations
Fri May 15 13:51:08 2020 commencing duplicate removal, pass 2
Fri May 15 13:55:12 2020 found 75754247 duplicates and 137394340 unique relations
...
Fri May 15 14:40:19 2020 matrix is 10495776 x 10496000 (4191.2 MB) with weight 1118174547 (106.53/col)
Fri May 15 14:40:19 2020 sparse part has weight 993750449 (94.68/col)
Fri May 15 14:40:19 2020 using block size 8192 and superblock size 884736 for processor cache size 9216 kB
Fri May 15 14:40:46 2020 commencing Lanczos iteration (6 threads)
Fri May 15 14:40:46 2020 memory use: 3956.9 MB
Fri May 15 14:41:08 2020 linear algebra at 0.0%, ETA 40h11m[/code]

This took 23.3M CPU-seconds of sieving with parameters
[code]tasks.I = 14
tasks.qmin = 7000000
tasks.lim0 = 50000000
tasks.lim1 = 70000000
tasks.lpb0 = 31
tasks.lpb1 = 32
tasks.sieve.mfb0 = 58
tasks.sieve.mfb1 = 60
tasks.sieve.lambda0 = 1.85
tasks.sieve.lambda1 = 1.85
tasks.sieve.ncurves0 = 20
tasks.sieve.ncurves1 = 25[/code]

I've seen somewhere that sieve time should double every 5.5 digits, in which case this is still a bit slower than we'd like based on the timings at c177/8.

I'll do 95^128+1 next; Chris, do I need to reserve this somewhere? As for parameters, should I maybe try A=28 instead of I=14 and keep everything else the same?

Well, at least the matrix got smaller! 213M rels at 31/32 vs 189M at 31/31/3LP; we could tighten lambda a bit more (1.84 and 1.83, respectively) and expect rels_wanted to drop another ~5M or so. I don't see that getting us the 5-10% speed improvement we "expect", though.

Trying A=28 all else equal should help us decide which siever to stick with for further testing; good plan. It would be nice to compare Q-final for these two runs once you do A=28.

My prior ggnfs runs at 166-170 digits had matrices 9-10M in size, with 275M relations (EDIT: 32/32LP) and no restriction on lambda (mfb=63). I used 170 as my cutoff between 14e and 15e, but I expect CADO to be faster on I=14 than I=15 here.

This thread should do for a reservation. In practice no one but me is working on the Brent tables.

Chris

[code]Sun May 17 19:51:22 2020 Msieve v. 1.54 (SVN 1030M)
Sun May 17 19:51:22 2020 random seeds: 74f448f4 3f9ca822
Sun May 17 19:51:22 2020 factoring 52578676770574634512595466847841062400406219898703178016790505924363888689365475143019601569100893681427169623580739917336603746042987149267328695780988068362551904560641 (170 digits)
Sun May 17 19:51:22 2020 searching for 15-digit factors
Sun May 17 19:51:23 2020 commencing number field sieve (170-digit input)
Sun May 17 19:51:23 2020 R0: -647379397577956473141846534803906
Sun May 17 19:51:23 2020 R1: 34336932387135505576189
Sun May 17 19:51:23 2020 A0: 67111045429448622534099281184997406601
Sun May 17 19:51:23 2020 A1: 247012654371586877957224260881448
Sun May 17 19:51:23 2020 A2: -65521455613079545959441611
Sun May 17 19:51:23 2020 A3: -56319283841035187455
Sun May 17 19:51:23 2020 A4: 3534129102237
Sun May 17 19:51:23 2020 A5: 462210
Sun May 17 19:51:23 2020 skew 1.00, size 1.512e-16, alpha -4.882, combined = 3.112e-15 rroots = 5
Sun May 17 19:51:23 2020
Sun May 17 19:51:23 2020 commencing relation filtering
Sun May 17 19:51:23 2020 setting target matrix density to 100.0
Sun May 17 19:51:23 2020 estimated available RAM is 15845.4 MB
Sun May 17 19:51:23 2020 commencing duplicate removal, pass 1
Sun May 17 20:10:03 2020 found 49623560 hash collisions in 200069821 relations
Sun May 17 20:10:25 2020 commencing duplicate removal, pass 2
Sun May 17 20:14:03 2020 found 61168048 duplicates and 138901773 unique relations
...
Sun May 17 20:55:50 2020 matrix is 9099996 x 9100221 (3602.2 MB) with weight 949734284 (104.36/col)
Sun May 17 20:55:50 2020 sparse part has weight 853284443 (93.77/col)
Sun May 17 20:55:50 2020 using block size 8192 and superblock size 884736 for processor cache size 9216 kB
Sun May 17 20:56:12 2020 commencing Lanczos iteration (6 threads)
Sun May 17 20:56:12 2020 memory use: 3428.1 MB
Sun May 17 20:56:30 2020 linear algebra at 0.0%, ETA 28h16m[/code]

23.7M CPU-seconds of sieving with A=28; the polynomial is supposedly slightly better than the one from the previous job (3.76e-13 vs 3.53e-13) which would suggest I=14 is a little faster. The matrix has come out smaller though, despite a similar number of unique relations (and, in fact, a smaller excess of relations over ideals). Not really clear which will end up being fastest when everything else is optimised.

Final Q-values were 157.7M for the I=14 job and 104.3M for this one; both started at 7M.

For what it's worth, here are the 95^128+1 factors:

[code]p61 factor: 6257613858583898709261755903084673553330399368418555296798721
p109 factor: 8402352391630827885506737623084809472170424349385439829494702478551900066063207448078203277908726932399787521[/code]

I've submitted these to cownoise and factordb - not sure if there's anywhere else they should be sent.

cownoise and factordb are the only places I would submit it to.

Thanks for the factors.

Chris

One more data point: the c169 from 9+8_638M.

Parameters were
[code]tasks.I = 14
tasks.qmin = 7000000
tasks.lim0 = 50000000
tasks.lim1 = 70000000
tasks.lpb0 = 31
tasks.lpb1 = 31
tasks.sieve.mfb0 = 58
tasks.sieve.mfb1 = 58
tasks.sieve.ncurves0 = 20
tasks.sieve.ncurves1 = 25[/code]

Sieving took 21.6M CPU-seconds for 170M relations; max Q was ~138.6M. 165M relations did not built a matrix at TD=100.

[code]Sun Jun 28 15:03:04 2020 Msieve v. 1.54 (SVN 1030M)
Sun Jun 28 15:03:04 2020 random seeds: 020987c2 2e14909e
Sun Jun 28 15:03:04 2020 factoring 6323148101178192997454526289605277170954218433278054116052056166051527590848248128865718264727023155963463084305589386151325042874693217587179972005266685135356161079181 (169 digits)
Sun Jun 28 15:03:05 2020 searching for 15-digit factors
Sun Jun 28 15:03:05 2020 commencing number field sieve (169-digit input)
Sun Jun 28 15:03:05 2020 R0: -424000671411315695105034591696768
Sun Jun 28 15:03:05 2020 R1: 72112096043794064331337
Sun Jun 28 15:03:05 2020 A0: 24834769717672015656695526227332911290885
Sun Jun 28 15:03:05 2020 A1: 20316186393429687042949998116877192
Sun Jun 28 15:03:05 2020 A2: -1818007152430589614430877325
Sun Jun 28 15:03:05 2020 A3: -200551169527456827261
Sun Jun 28 15:03:05 2020 A4: 8997719941109
Sun Jun 28 15:03:05 2020 A5: 459900
Sun Jun 28 15:03:05 2020 skew 1.00, size 1.828e-16, alpha -6.733, combined = 1.955e-15 rroots = 5
Sun Jun 28 15:03:05 2020
Sun Jun 28 15:03:05 2020 commencing relation filtering
Sun Jun 28 15:03:05 2020 setting max relations to 170000000
Sun Jun 28 15:03:05 2020 setting target matrix density to 100.0
Sun Jun 28 15:03:05 2020 estimated available RAM is 15845.9 MB
Sun Jun 28 15:03:05 2020 commencing duplicate removal, pass 1
...
Sun Jun 28 15:19:16 2020 found 49673476 hash collisions in 169989969 relations
Sun Jun 28 15:19:16 2020 commencing duplicate removal, pass 2
Sun Jun 28 15:22:21 2020 found 65292334 duplicates and 104697635 unique relations
Sun Jun 28 15:22:21 2020 memory use: 1321.5 MB
Sun Jun 28 15:22:21 2020 reading ideals above 136511488
Sun Jun 28 15:22:21 2020 commencing singleton removal, initial pass
Sun Jun 28 15:29:47 2020 memory use: 2756.0 MB
Sun Jun 28 15:29:48 2020 reading all ideals from disk
Sun Jun 28 15:29:52 2020 memory use: 1584.8 MB
Sun Jun 28 15:29:54 2020 commencing in-memory singleton removal
Sun Jun 28 15:29:56 2020 begin with 104697635 relations and 98408402 unique ideals
Sun Jun 28 15:30:11 2020 reduce to 41074643 relations and 24856975 ideals in 15 passes
Sun Jun 28 15:30:12 2020 max relations containing the same ideal: 21
Sun Jun 28 15:30:14 2020 reading ideals above 720000
Sun Jun 28 15:30:14 2020 commencing singleton removal, initial pass
Sun Jun 28 15:35:16 2020 memory use: 753.0 MB
Sun Jun 28 15:35:17 2020 reading all ideals from disk
Sun Jun 28 15:35:18 2020 memory use: 1560.4 MB
Sun Jun 28 15:35:20 2020 keeping 39923650 ideals with weight <= 200, target excess is 224283
Sun Jun 28 15:35:23 2020 commencing in-memory singleton removal
Sun Jun 28 15:35:25 2020 begin with 41074647 relations and 39923650 unique ideals
Sun Jun 28 15:36:03 2020 reduce to 39865811 relations and 38709614 ideals in 16 passes
Sun Jun 28 15:36:03 2020 max relations containing the same ideal: 200
Sun Jun 28 15:36:16 2020 removing 3404962 relations and 3004962 ideals in 400000 cliques
Sun Jun 28 15:36:17 2020 commencing in-memory singleton removal
Sun Jun 28 15:36:19 2020 begin with 36460849 relations and 38709614 unique ideals
Sun Jun 28 15:36:36 2020 reduce to 36261732 relations and 35502896 ideals in 8 passes
Sun Jun 28 15:36:36 2020 max relations containing the same ideal: 192
Sun Jun 28 15:36:49 2020 removing 2549850 relations and 2149850 ideals in 400000 cliques
Sun Jun 28 15:36:50 2020 commencing in-memory singleton removal
Sun Jun 28 15:36:52 2020 begin with 33711882 relations and 35502896 unique ideals
Sun Jun 28 15:37:09 2020 reduce to 33582756 relations and 33222484 ideals in 9 passes
Sun Jun 28 15:37:09 2020 max relations containing the same ideal: 184
Sun Jun 28 15:37:21 2020 removing 725709 relations and 625606 ideals in 100103 cliques
Sun Jun 28 15:37:21 2020 commencing in-memory singleton removal
Sun Jun 28 15:37:23 2020 begin with 32857047 relations and 33222484 unique ideals
Sun Jun 28 15:37:36 2020 reduce to 32846772 relations and 32586570 ideals in 7 passes
Sun Jun 28 15:37:36 2020 max relations containing the same ideal: 181
Sun Jun 28 15:37:42 2020 relations with 0 large ideals: 757
Sun Jun 28 15:37:42 2020 relations with 1 large ideals: 797
Sun Jun 28 15:37:42 2020 relations with 2 large ideals: 13803
Sun Jun 28 15:37:42 2020 relations with 3 large ideals: 143045
Sun Jun 28 15:37:42 2020 relations with 4 large ideals: 826045
Sun Jun 28 15:37:42 2020 relations with 5 large ideals: 2883247
Sun Jun 28 15:37:42 2020 relations with 6 large ideals: 6400583
Sun Jun 28 15:37:42 2020 relations with 7+ large ideals: 22578495
Sun Jun 28 15:37:42 2020 commencing 2-way merge
Sun Jun 28 15:37:58 2020 reduce to 20116382 relation sets and 19856180 unique ideals
Sun Jun 28 15:37:58 2020 commencing full merge
Sun Jun 28 15:42:35 2020 memory use: 2417.9 MB
Sun Jun 28 15:42:36 2020 found 9692453 cycles, need 9672380
Sun Jun 28 15:42:38 2020 weight of 9672380 cycles is about 967669235 (100.04/cycle)
Sun Jun 28 15:42:38 2020 distribution of cycle lengths:
Sun Jun 28 15:42:38 2020 1 relations: 733278
Sun Jun 28 15:42:38 2020 2 relations: 915189
Sun Jun 28 15:42:38 2020 3 relations: 994301
Sun Jun 28 15:42:38 2020 4 relations: 930915
Sun Jun 28 15:42:38 2020 5 relations: 871603
Sun Jun 28 15:42:38 2020 6 relations: 780528
Sun Jun 28 15:42:38 2020 7 relations: 688823
Sun Jun 28 15:42:38 2020 8 relations: 597770
Sun Jun 28 15:42:38 2020 9 relations: 523494
Sun Jun 28 15:42:38 2020 10+ relations: 2636479
Sun Jun 28 15:42:38 2020 heaviest cycle: 28 relations
Sun Jun 28 15:42:39 2020 commencing cycle optimization
Sun Jun 28 15:42:52 2020 start with 69254191 relations
Sun Jun 28 15:44:19 2020 pruned 2098869 relations
Sun Jun 28 15:44:19 2020 memory use: 2077.5 MB
Sun Jun 28 15:44:20 2020 distribution of cycle lengths:
Sun Jun 28 15:44:20 2020 1 relations: 733278
Sun Jun 28 15:44:20 2020 2 relations: 937882
Sun Jun 28 15:44:20 2020 3 relations: 1033609
Sun Jun 28 15:44:20 2020 4 relations: 959962
Sun Jun 28 15:44:20 2020 5 relations: 899206
Sun Jun 28 15:44:20 2020 6 relations: 796546
Sun Jun 28 15:44:20 2020 7 relations: 700849
Sun Jun 28 15:44:20 2020 8 relations: 603393
Sun Jun 28 15:44:20 2020 9 relations: 524575
Sun Jun 28 15:44:20 2020 10+ relations: 2483080
Sun Jun 28 15:44:20 2020 heaviest cycle: 28 relations
Sun Jun 28 15:44:34 2020 RelProcTime: 2489
Sun Jun 28 15:44:37 2020
Sun Jun 28 15:44:37 2020 commencing linear algebra
Sun Jun 28 15:44:37 2020 read 9672380 cycles
Sun Jun 28 15:44:52 2020 cycles contain 32650243 unique relations
Sun Jun 28 15:48:25 2020 read 32650243 relations
Sun Jun 28 15:49:03 2020 using 20 quadratic characters above 4294917295
Sun Jun 28 15:51:08 2020 building initial matrix
Sun Jun 28 15:56:02 2020 memory use: 4429.8 MB
Sun Jun 28 15:56:44 2020 read 9672380 cycles
Sun Jun 28 15:56:45 2020 matrix is 9672203 x 9672380 (3951.6 MB) with weight 1209714091 (125.07/col)
Sun Jun 28 15:56:45 2020 sparse part has weight 919819428 (95.10/col)
Sun Jun 28 15:57:58 2020 filtering completed in 2 passes
Sun Jun 28 15:57:59 2020 matrix is 9671585 x 9671762 (3951.6 MB) with weight 1209689275 (125.07/col)
Sun Jun 28 15:57:59 2020 sparse part has weight 919814498 (95.10/col)
Sun Jun 28 15:58:46 2020 matrix starts at (0, 0)
Sun Jun 28 15:58:48 2020 matrix is 9671585 x 9671762 (3951.6 MB) with weight 1209689275 (125.07/col)
Sun Jun 28 15:58:48 2020 sparse part has weight 919814498 (95.10/col)
Sun Jun 28 15:58:48 2020 saving the first 48 matrix rows for later
Sun Jun 28 15:58:49 2020 matrix includes 64 packed rows
Sun Jun 28 15:58:50 2020 matrix is 9671537 x 9671762 (3832.6 MB) with weight 1013022612 (104.74/col)
Sun Jun 28 15:58:50 2020 sparse part has weight 907971208 (93.88/col)
Sun Jun 28 15:58:50 2020 using block size 8192 and superblock size 884736 for processor cache size 9216 kB
Sun Jun 28 15:59:14 2020 commencing Lanczos iteration (6 threads)
Sun Jun 28 15:59:14 2020 memory use: 3620.3 MB
Sun Jun 28 15:59:34 2020 linear algebra at 0.0%, ETA 33h29m[/code]

31/31 is probably a little slower for sieving than 31/32, though not by a large enough margin to rule out further testing - might be worth playing around with the mfb bounds, for example?

I've decided to do 86^131-1 via NFS@Home. I've found the following poly (9 hours msieve-gpu searching):
[code]
n: 206291417462093576497275239580773052795596055350565606087502662805051224044201602337850673455835775692863740037805407606265476824913591751307682500480113600344276700606001
# norm 1.306001e-16 alpha -8.871987 e 2.977e-13 rroots 3
skew: 162345250.84
c0: -577062733978460861005241440490749355102976512
c1: -12260826936151352820546357994499780916
c2: 178538325918319471247024945044
c3: 958590300841474414947
c4: -4423011255032
c5: 11844
Y0: -1770910298352749710979419014235215
Y1: 613792628360770487
type: gnfs
[/code]

The record e-score for 171 digits is 3.730e-13 so this is not too bad for a first try.

If Max could see if he can spin it up I would be very grateful.

Also is anyone interested in 24^179+1 ? If not I'll put it through NFS@Home in a week or so.

Chris

Here are a couple better polys for 86^131-1:

[code]n: 206291417462093576497275239580773052795596055350565606087502662805051224044201602337850673455835775692863740037805407606265476824913591751307682500480113600344276700606001
skew: 41437836.858
c0: -5389519387337140565345516459794841317758000
c1: -143043284702103460455094028784037830
c2: 2239398228891059427583897507
c3: -889691908139885268641
c4: 3496913201934
c5: 95760
Y0: -1659129166378730128203133792666517
Y1: 1742363546744996502291193
# e 3.392e-13

n: 206291417462093576497275239580773052795596055350565606087502662805051224044201602337850673455835775692863740037805407606265476824913591751307682500480113600344276700606001
skew: 3357702.047
c0: -112975106982991917173660977635799247400
c1: 111002282144221726406217931664406
c2: -64145260501719297021407541
c3: -30825474695526305746
c4: 4452459053820
c5: 543690
Y0: -823750842791353826106062726508889
Y1: 34863622316774006451893
# e 3.334e-13[/code]

[QUOTE=chris2be8;549306]Also is anyone interested in 24^179+1 ? If not I'll put it through NFS@Home in a week or so.[/QUOTE]

I'll do it.

Bur requested a 165 params file, so I attach a draft here.

If you use this file, please report:
1. Did filtering work the first time, or did CADO bounce back and forth between filtering and sieving? If filtering ran multiple times, how many relations were needed for sieving to finish?
2. What was the final-Q sieved?
3. What was the matrix size, in dimensions or "total weight"?

I'm still not sure if 3LP is useful in 165-170 digit range; CADO uses 3 large primes starting at 145 or 150 digits in the default files, so I ought to explore it. One of these days....

Ok, I started the factorization. I think final q and the matrix properties can be looked up in the log file?

Should I have cado do the linear algebra or doesn't it matter?

Doesn't matter whether msieve or cado does the matrix; I use matrix size as a marker for whether target_relations was set too high or too low.

Final Q can be found in the log, though it's not in the summary. It's easiest to find by scrolling up on the terminal window to find the last Q sieved before filtering.

Currently sieving ETA is already saturday, 15th, but it usually increased significantly after a while. I report back once the factorization is finished.

LA is still running, but everything seems to have gone fine:

1) the initial 220,000,000 relations were enough, no going back to sieving after the first filtering
2) last q-range was 66,450,000-66,460,000
3) Merged matrix has 5579102 rows and total weight 836915933 (150.0 entries per row on average)

ETA for krylov is 8 hours from now, mksol will also take a while but I expect a total of less than 8 days. That's fast compared to the ~75 hours a C153 and a C154 took on the same machine.


And a question: How do number of relations, matrix weight/density and time for LA depend on each other?

From some posts it seems to me that having more relations allows for a matrix with higher density and those will result in shorte LA times. Is it like that? So increasing target_density will increase the number of required relations but decrease LA time?

Having more available relations allows filtering to work harder, generating a smaller matrix. Setting target density higher explicitly tells filtering to work harder, but even holding target density fixed you'll get a smaller matrix from more relations.

The catch is that it usually takes more time to gather those extra relations than one saves in LA phase. There's a productive amount of oversieving- when one is right at the cusp of building a matrix, a small number of extra relations has a fairly strong effect on matrix size, but diminishing returns sets in rather quickly.

My previous GGNFS/msieve jobs around C165 have had matrices around 9M in size, so this job was rather strongly oversieved. We should cut rels_wanted to 210M for this file and see what matrix comes out.

If you are interested in seeing the effect of the extra relations, zcat all your 220M relations out to a single file, and run msieve's filtering on the file to see what size matrix comes out. Then, restrict the number of relations msieve uses (via a filtering flag, see the -h option list for msieve) to 215, 210, 205 and let us know what size matrices pop out. I suggest a TD of 100 or so for msieve, but you might enjoy trying 100/110/120 on the full dataset to see how target_density affects matrix size.

tasks.qmin can be changed down to 10M. I target a qmin-to-qmax ratio of 7 to 8; your final-Q of 66.5M suggests the chosen qmin of 17M was a little high. Smaller Q sieve faster but yield more duplicate relations, so changing tasks.qmin to 10M should make jobs run a little bit faster because 10-17M will produce more relations (and faster) than 59-66M. Since this job ran faster than expected (based on your ~155 digit experience) and Q is smaller than I expected, you may have found a lucky poly for this job.

Thanks for your data report!

[QUOTE=VBCurtis;578567]My previous GGNFS/msieve jobs around C165 have had matrices around 9M in size, so this job was rather strongly oversieved. We should cut rels_wanted to 210M for this file and see what matrix comes out.[/quote]

Msieve always gives larger but sparser matrices though. Still possible this was oversieved, but probably not massively so.

My hunch is that raising [C]tasks.sieve.mfb1[/C] from 62 to 64 would be more effective than using 3LP here. That said, I've got a better idea of what to do with the other parameters than the last time I tried 3LP on a number of roughly this size (see post #2 in this thread). May give it a go in the near future.

Thanks for the explanations, slowly I start to get more about what is going on. Interesting, that fundmental things like poly selection or q-range are so little understood that it makes an experimental attempt useful. From the little I could get from math papers, especially poly selection could probably be vastly optimized if we knew more about it.

[QUOTE]If you are interested in seeing the effect of the extra relations, zcat all your 220M relations out to a single file, and run msieve's filtering on the file to see what size matrix comes out. Then, restrict the number of relations msieve uses (via a filtering flag, see the -h option list for msieve) to 215, 210, 205 and let us know what size matrices pop out. I suggest a TD of 100 or so for msieve, but you might enjoy trying 100/110/120 on the full dataset to see how target_density affects matrix size.[/QUOTE]So to just see the effect of fewer relations I'll set TD = 100?

And just to make 100% sure: more relations = faster matrix generation & higher matrix density = smaller matrix size = shorter LA times [SIZE="1"](not the newspaper, harhar)[/SIZE]?

No, there's a flag like filter_maxrels that lets you control the number of relations msieve uses.

TD 100 vs TD 110 shows you the effect of using higher target densities for a given set of relations- the difference is not very big. A higher density matrix is slightly tougher to solve than a lower-density matrix of the same size, but matrix size affects time-to-solve much more.

More relations -> smaller matrix.-> shorter LA time.

Higher TD -> makes sure you have enough relations for a reasonably small matrix.

When filtering with msieve, let's say you have 190M relations. TD = 90 might build a matrix 11M in size, while TD 110 fails and asks for more relations. If you use the full 220M relations, TD 90 might build a 7M matrix, while TD 110 builds a 6.5M matrix. The idea is that choosing TD = 110 makes sure you avoid the situation where a matrix barely builds and is quite large (e.g. 11M for this current factorization).
If you actually try these tests, let the matrix-solving step run for 1% completion, and note the ETA (ETA is pretty stable by the time 1% is done). The TD100 and TD110 ETAs won't be very different, I expect.

What I meant was, if I only want to test various numbers of relations, which value for TD should I chose, 100? Or is there a default value?


Testing different target_densities was that just a proposal for me to see the effects, or are the results helpful to you similar to the various numbers of relations?

Ah, I see- yes, I'd use TD 100 to test various relations counts, because higher TDs may not produce matrices. If TD 100 doesn't produce a matrix, that relations count is too low to be useful to us anyway (I think).

The TD 100 vs 110 is useful to us, but I agree it's more for your personal experience. I haven't done as many such tests as I should, so I am curious just how different the ETAs would be; my own limited testing showed the ETAs are similar enough that I rarely use TD 120 even on jobs I expect would easily build a matrix at that density, since the time spent when filtering fails feels greater than the time saved by using 120 instead of 110.

I'm still running the tests, results so far show that 185M relations was too few. 190M still worked, LA took from around 14 h (190M) to 8:45 h (220M). Since sieving took 6 1/2 days, doing only 190M would have been quite a gain.

Would it be good to decrease rels_wanted to 190M generally or might this have been an exception? But even 2-3 additional filtering steps would still make the total time shorter.

I will post detailed results later this day.

I used this command line [C]./msieve -i c165.n -s rels.dat -nf c165.fb -t 10 -v -nc1 "filter_maxrels=200000000"[/C] afterwards the same with -nc2.

220M rels
[CODE]matrix is 6441919 x 6442144 (2334.4 MB) with weight 618273835 (95.97/col)
sparse part has weight 547526160 (84.99/col)
[...]
linear algebra completed 64760 of 6442144 dimensions (1.0%, ETA 8h43m[/CODE]215M rels
[CODE]matrix is 6671510 x 6671738 (2418.3 MB) with weight 640096824 (95.94/col)
sparse part has weight 567230375 (85.02/col)
[...]
linear algebra completed 66759 of 6671738 dimensions (1.0%, ETA 9h28m[/CODE]210M rels
[CODE]matrix is 6881379 x 6881605 (2496.6 MB) with weight 660645774 (96.00/col)
sparse part has weight 585663806 (85.11/col)
[...]
linear algebra completed 72597 of 6881605 dimensions (1.1%, ETA 10h 3m)
[/CODE]205M rels
[CODE]matrix is 7102309 x 7102534 (2579.8 MB) with weight 682640424 (96.11/col)
sparse part has weight 605254018 (85.22/col)
[...]
near algebra completed 71317 of 7102534 dimensions (1.0%, ETA 10h53m)
[/CODE]200M rels
[CODE]matrix is 7405037 x 7405262 (2691.5 MB) with weight 711821792 (96.12/col)
sparse part has weight 631510830 (85.28/col)
[...]
linear algebra completed 74878 of 7405262 dimensions (1.0%, ETA 11h54m)[/CODE]195M rels
[CODE]matrix is 7779754 x 7779979 (2833.7 MB) with weight 749219145 (96.30/col)
sparse part has weight 665027723 (85.48/col)
[...]
linear algebra completed 77887 of 7779979 dimensions (1.0%, ETA 13h25m)[/CODE]190M rels
[CODE]matrix is 8305557 x 8305782 (3029.3 MB) with weight 800256398 (96.35/col)
sparse part has weight 711056605 (85.61/col)
[...]
linear algebra completed 87238 of 8305782 dimensions (1.1%, ETA 15h43m)[/CODE]185M rels
[CODE]found 338230 cycles, need 3940439

too few cycles, matrix probably cannot build
filtering wants 1000000 more relations[/CODE]
Sieving the 220M relations took 158 hours or about 3.5 hours per 5M relations.


I didn't specify TD, which numbers of relations should I chose for the density comparisons?

[QUOTE=bur;578710]
I didn't specify TD, which numbers of relations should I chose for the density comparisons?[/QUOTE]

It might be interesting to find the best TD for each number of relations. In theory, the difference should be even more pronounced than with a constant TD.

Your comparison of 3.5 hrs per 5M rels shows that any oversieving past 190M is inefficient. This surprises me- I would have expected 195 or 200 to be the "sweet spot", where the sieve time spent would be more than or equal to LA time saved.

But, even 195M vs 190M only saves 2hr 20 min LA time at the cost of ~3.5hr sieve time.

If these were all default density, I wonder whether the 96 or the 85 density is the target number- I don't have msieve logs handy to look it up myself, so I'll edit this message later today to correct myself about which of the densities from your logs is the one msieve controls directly. If default is 96 at this size, I doubt TD is going to do much; try TD 100 or 104 or 110 on the 195M relations set to see if you can save another hour of LA time.

Your data also helps someone like Ed, who uses a large farm of ~20 machines to sieve, but just one to LA. Your data at 190.....220 can help him choose how much to sieve.

I think I'll make the params file 190M target; as you say, filtering more than once isn't a big deal if that turns out to not be enough for a C161 or C162. I'll add a note that recommends 200M if sieving on multiple machines.

What CPU architecture is used for these tests? Older CPUs take relatively longer to do the LA section, so might prove to be 3.5hr faster at 195M because the entire LA portion is taking 50% longer. Sieving is less sensitive to CPU architecture than LA. My personal experience is with sandy bridge, ivy bridge, haswell generations of Xeon, all rather old by now.

[QUOTE=VBCurtis;578717]If these were all default density, I wonder whether the 96 or the 85 density is the target number- I don't have msieve logs handy to look it up myself, so I'll edit this message later today to correct myself about which of the densities from your logs is the one msieve controls directly. If default is 96 at this size, I doubt TD is going to do much; try TD 100 or 104 or 110 on the 195M relations set to see if you can save another hour of LA time.[/quote]

Actually, it's neither. There will be a line from earlier in the filtering that matches TD, looking like this:
[code]weight of 9242904 cycles is about 924731720 (100.05/cycle)[/code]
The default in the latest msieve is density 90, and bur's figures match what I see in my logs for 90.

Re general c165 params: lpb 31/31 is definitely worth testing too, unless you already have data showing 31/32 is faster. I also notice you didn't include adjust_strategy 2, which would probably give a boost of a couple percent. Strategy 2 makes A=28 usable but I don't think it'll be optimal yet at this size. Maybe by c170.

[QUOTE=henryzz;578714]It might be interesting to find the best TD for each number of relations. In theory, the difference should be even more pronounced than with a constant TD.[/QUOTE]So I'll use TD = 90, 100, 110, 120? Which number of rels would be interesting, all I tested so far?

It would make 28 tests, but since filtering is single-threaded I could run 10 simultaneously. Would this work with all processes accessing the same rels.dat?


VBCurtis, it was run on a i10-10900K with 16 GB RAM. Nothing else demanding was running at the same time.


BTW, I am just factoring a C147 with your experimental params.c145, should I keep the relations to do similar tests?

As you go up in relations, you will go up in TD too.

I think if you do this for 190 vs 195 vs 200M rels you'll find out the interesting parts about how extra sieving -> higher TD -> smaller-er matrix -> possible saved time overall. If TD 110 fails to build a matrix, try 105? If you work from the top down, you won't need to try all the TDs for all the rels sizes- the idea is to locate the largest TD that works for each relation set, as that "should" save the most time.

I don't think you can run multiple copies of filtering at the same time unless you're really careful about specifying all the output file names to be different for each copy; else they'll trample each other.

[QUOTE=bur;578757]BTW, I am just factoring a C147 with your experimental params.c145, should I keep the relations to do similar tests?[/QUOTE]

No, the files at C150 and below are pretty well tested and refined. It's C155+ that have few jobs run.

Unfortunately you were right about the simultaneous runs, I hardlinked the rels.dat file to different directories, as I thought it was only read from. But apparently the processes want write access to it, because they all quit at the first singleton removal with an error about writing access. I should build an HDD into that computer...

But if working down from TD=110 on 190,195,200 is sufficient, then it's not too many tests anyway.

[QUOTE=bur;578849]Unfortunately you were right about the simultaneous runs, I hardlinked the rels.dat file to different directories, as I thought it was only read from. But apparently the processes want write access to it, because they all quit at the first singleton removal with an error about writing access. I should build an HDD into that computer...
[/QUOTE]

Yes - msieve needs write access to add free relations.

Here are the results, no major gains, but if using more than 10 cores it will be worth to use 195/105 instead of 190/95. On my 10 core i9 the sieving would take 3:30 h longer with 2:40 h saved at LA, so already at 20 cores it would help a bit.


200M rels / TD=110
[CODE]commencing 2-way merge
reduce to 15092846 relation sets and 14808858 unique ideals
commencing full merge
memory use: 682.9 MB
found 21197 cycles, need 3146938
too few cycles, matrix probably cannot build[/CODE]200M rels / TD=105 (took 11:54 h for TD=90)
[CODE]matrix is 7010462 x 7010687 (2900.8 MB) with weight 774175808 (110.43/col)
sparse part has weight 690320583 (98.47/col)
[...]
linear algebra completed 70075 of 7010687 dimensions (1.0%, ETA 11h36m)[/CODE]195M rels / TD=105 (took 13:25 h for TD=90)
[CODE]matrix is 7353588 x 7353812 (3046.7 MB) with weight 812669648 (110.51/col)
sparse part has weight 725127644 (98.61/col)
[...]
linear algebra completed 73595 of 7353812 dimensions (1.0%, ETA 12h49m][/CODE]190M rels / TD=100
[CODE]commencing 2-way merge
reduce to 17035108 relation sets and 16791673 unique ideals
commencing full merge
memory use: 776.3 MB
found 80952 cycles, need 3526279
too few cycles, matrix probably cannot build[/CODE]190M rels / TD=95 (took 15:43 h for TD=90)
[CODE]matrix is 8132721 x 8132946 (3102.7 MB) with weight 822289451 (101.11/col)
sparse part has weight 732019314 (90.01/col)
[...]
linear algebra completed 81232 of 8132946 dimensions (1.0%, ETA 15h30m)[/CODE]

The c159 is in progress, found 65 % of the relations. One thing I notice compared to the c163 is that the number of relations per q decreases strongly. Already at q ~ 29e6 it's only 35,000 rels per 10,000 range, whereas with the c163 it still was 35,000 rels at q ~ 66e6. The c159 will end up at q around 43e6.

So apparently that was a lucky polynomial with the c163. I am not sure if that changes anything regarding the parameters.

The matrix was build at first try with 160,000,000 relations.

Final q sieved: 45300000

matrix is 5597069 x 5597293 (2017.8 MB) with weight 529305051 (94.56/col)
sparse part has weight 472974971 (84.50/col)

Is this as expected or does it make sense to see if fewer relations would work? TD was again the msieve default of 90, while your params.c160 aims for 145. To be honest, I forgot to adjust the msieve parameters...

5.5M matrix is still a bit on the small side. I bet 155M relations would work fine; if you don't mind spending the time to check, I'd appreciate it.

Note that CADO and msieve measure density in different ways; I subtract 50 from CADO density to get a rough msieve density, so your msieve default 90 isn't far from the density I guessed is best.

Thanks for the stats report!

Charybdis sent me a params trial and stats report on PM this week for a C165. These two reports provide ideas for a bit of tinkering with the settings. If you're going to do another 158-167 sized job, let me know and I'll send you a new file to try out.

If we're going to try to refine these settings, we will need one more data point: The poly score as reported by *msieve* (or by the optimal skew calculator at [url]http://myfactors.mooo.com/[/url]). That's because poly scores can vary by 5% for a given size of input, and we should take that into account when comparing runs.

[QUOTE=VBCurtis;579403]If we're going to try to refine these settings, we will need one more data point: The poly score as reported by *msieve* (or by the optimal skew calculator at [url]http://myfactors.mooo.com/[/url]). That's because poly scores can vary by 5% for a given size of input, and we should take that into account when comparing runs.[/QUOTE]

Another interesting data point is the number of unique relations, as reported by msieve filtering, which is a more consistent indicator of when a matrix will build than the number of raw relations. Some polynomials seem to produce more duplicate relations than others, for reasons that I'm not really sure about.

I missed your reply, I'll run tests on the minimum number of relations after that strange C165 SNFS factorization finishes.

I entered the poly parameters into myfactor, but it came up with a different skew (3.3e6 instead of 2.6e6) than the one I used for the factorization and the scores were on the order of 10^-12 instead of 10^7. I found this surprising, is the score calculated that differently or were the myfactor skews so bad? How can I calculate scores by using msieve?


Regarding unique relations, I'll record those for the c159 tests. I don't have that data for the c164, I could re-run the matrix tests on it, should I just do it for the lowest number of relations that worked?

[QUOTE=bur;580516]
I entered the poly parameters into myfactor, but it came up with a different skew (3.3e6 instead of 2.6e6) than the one I used for the factorization and the scores were on the order of 10^-12 instead of 10^7. I found this surprising, is the score calculated that differently or were the myfactor skews so bad? How can I calculate scores by using msieve?
[/QUOTE]

See this [URL="https://www.mersenneforum.org/showpost.php?p=425446&postcount=372"]post[/URL]. The method works quite well.

[QUOTE=bur;580516]I entered the poly parameters into myfactor, but it came up with a different skew (3.3e6 instead of 2.6e6) than the one I used for the factorization and the scores were on the order of 10^-12 instead of 10^7. I found this surprising, is the score calculated that differently or were the myfactor skews so bad? How can I calculate scores by using msieve?[/QUOTE]

The Murphy-E score reported by CADO is not the same as that reported by msieve and cownoise.com. Technically the CADO one is closer to Murphy's definition of E, but it depends on the I value, qmin and the lpb bounds, so it shouldn't be used for comparing polynomials for two different numbers unless they used the same parameters. Msieve's E-score does not depend on the parameters and so can be used to compare polynomials for different numbers, as long as they have the same degree.

This is the score of the c163 I did earlier, calculated with msieve:
[CODE]skew 8385289.96, size 8.965e-16, alpha -7.745, combined = 1.077e-12 rroots = 5[/CODE]

And this is the score of the c159 calculated with msieve:
[CODE]skew 2613329.11, size 1.949e-15, alpha -7.801, combined = 1.703e-12 rroots = 5[/CODE]

I ran tests for the minimal number of relations for the c159:

160M
[CODE]found 47576500 duplicates and 112589785 unique relations (70.3%)
[...]
matrix is 5597069 x 5597293 (2017.8 MB) with weight 529305051 (94.56/col)
sparse part has weight 472974971 (84.50/col)
[...]
linear algebra completed 56142 of 5597293 dimensions (1.0%, ETA 6h19m)[/CODE]

155M
[CODE]found 44436078 duplicates and 110560392 unique relations (71.3%)
[...]
matrix is 5850133 x 5850358 (2110.1 MB) with weight 553183174 (94.56/col)
sparse part has weight 494635444 (84.55/col)
[...]
linear algebra completed 58683 of 5850358 dimensions (1.0%, ETA 6h59m)[/CODE]

150M
[CODE]found 41393391 duplicates and 108603079 unique relations (72.4%)
[...]
matrix is 6136751 x 6136976 (2215.4 MB) with weight 580409331 (94.58/col)
sparse part has weight 519375147 (84.63/col)
[...]
linear algebra completed 119868 of 6136976 dimensions (2.0%, ETA 7h52m)[/CODE]

145M
[CODE]found 38915690 duplicates and 106080857 unique relations (73.2%)
[...]
matrix is 6621872 x 6622097 (2394.0 MB) with weight 626672267 (94.63/col)
sparse part has weight 561353788 (84.77/col)
[...]
linear algebra completed 66224 of 6622097 dimensions (1.0%, ETA 9h17m)[/CODE]

140M
[CODE]found 36920951 duplicates and 103075743 unique relations (73.6%)
[...]
keeping 27769539 ideals with weight <= 200, target excess is 147096
commencing in-memory singleton removal
begin with 27594853 relations and 27769539 unique ideals
reduce to 27476994 relations and 27651624 ideals in 23 passes
max relations containing the same ideal: 200
filtering wants 1000000 more relations[/CODE]
Anything else that should be tested?

edit: An unexpected P41 factor turned up, but I'll look for some other c160 to factor, so if you have new parameters, please let me know.

[I]I'll start work on a [URL="http://factordb.com/index.php?id=1100000002494123081"]c160[/URL], part of AL86610. Do you have a new set of parameters or should I go with the old one, with rels_wanted reduced to 145M?[/I]

73% unique is unusually high; I would expect most ~C160 jobs to need a few rounds of filtering if we set target relations to 145M. I suppose your sieving machine needs more than 80 minutes to sieve from 145M rels to 150M rels?

Then again, with required_excess set as it is, there is almost-no chance of ending up with a matrix so big that more sieving is desired; I'm still working on figuring out the "right" setting for that one. So, try reducing tasks.filter.required_excess by 0.01 when you change rels_wanted to 145M.

Thanks for the data!

It took 125 hours for the 160M relations. That's 2800 s / 1M rels or roughly 45 minutes. So yes, nearly 4 hours for 5 M rels.


I'm currently looking for a useful c160, if you want one, then suddenly ECM finds factors all the time...


So I'd use the same params but with rels_wanted=145M and required_excess reduced by 0.01?

[QUOTE=bur;581171]So I'd use the same params but with rels_wanted=145M and required_excess reduced by 0.01?[/QUOTE]

Yes, exactly. There's no harm in CADO doing multiple filtering runs, so you could use 144M or 143M too if you're curious about how far it can be pushed. The "required_excess" setting is supposed to avoid the case where a matrix *barely* builds and is far too big- that is, just 1-2% more relations builds a much nicer matrix. In the context of this size of job, "barely" would be a matrix 9-10M in size. Your data on that C159 show that a ~7M matrix is "good enough", in that more sieving takes more time than would be saved by the smaller matrix.

We're looking for a required-excess setting that *always* avoids those too-big matrices. If you reduce it by 0.02 to be more aggressive, set rels_wanted to 140M and let us know how many relations it actually takes to build a matrix (it should be more than 140!)

Ok, I found another C159 from an aliquot sequence. I'm finishing ECM on it and then go with the 0.02 and 140M.

Btw, is there anything else to be done on the c159 or c164, such as target_density? Otherwise I'd clear the space for that next job.

[QUOTE=bur;581219]Ok, I found another C159 from an aliquot sequence. I'm finishing ECM on it and then go with the 0.02 and 140M.

Btw, is there anything else to be done on the c159 or c164, such as target_density? Otherwise I'd clear the space for that next job.[/QUOTE]

I'd hold off on this c159 - given [URL="https://mersenneforum.org/showpost.php?p=581221&postcount=3490"]this[/URL], I strongly suspect it's one that I've already factored. In which case I might as well provide the relevant data...

[code]tasks.I = 14
tasks.qmin = 7000000
tasks.lim0 = 25000000
tasks.lim1 = 45000000
tasks.lpb0 = 31
tasks.lpb1 = 31
tasks.sieve.mfb0 = 58
tasks.sieve.mfb1 = 61
tasks.sieve.lambda0 = 2.07
tasks.sieve.lambda1 = 2.17
tasks.sieve.ncurves0 = 19
tasks.sieve.ncurves1 = 24
tasks.sieve.qrange = 10000
tasks.sieve.adjust_strategy = 2[/code]

4.77M CPU-seconds for sieving from Q=7M-43.2M.
Filtering with TD=90:

[code]factoring 8162413651...0362934861 (159 digits)

skew 1612609.51, size 2.057e-15, alpha -5.963, combined = 1.761e-12 rroots = 5

found 47335431 hash collisions in 168523663 relations

found 57680004 duplicates and 110843659 unique relations

begin with 110843659 relations and 118457131 unique ideals

begin with 33364618 relations and 32436212 unique ideals
reduce to 33230023 relations and 32301540 ideals in 15 passes

matrix is 6559845 x 6560066 (2367.4 MB) with weight 618605783 (94.30/col)[/code]

163M relations (108M unique) were not enough to build a matrix.

That was the c159...

The number of unique relations required for it isn't that much lower than for my c159 with 106,080,857 uniques, but as vbcurtis already mentioned, the ratio of unique/total was much higher: 73.2% vs 65.8%.

I found a c158 from [URL="http://factordb.com/sequences.php?se=1&aq=32796&action=last20"]AL32796[/URL], I'm curious what ratio I'll end up with. The params I use now are slightly different from charybdis':

[code]tasks.lim0 = 30000000
tasks.lim1 = 47000000
tasks.lpb0 = 31
tasks.lpb1 = 31
tasks.sieve.mfb0 = 58
tasks.sieve.mfb1 = 61
tasks.sieve.lambda0 = 1.84
tasks.sieve.ncurves0 = 19
tasks.sieve.ncurves1 = 24
tasks.I = 14
tasks.qmin = 8000000
tasks.sieve.qrange = 10000
tasks.sieve.rels_wanted = 140000000[/code]

140M wasn't sufficient, but after two additional sievings the matrix was build with:


143466178 relations with 105330573 unique (73.4%)


Again the high number of uniques. Maybe the optimized parameters are that good?


That's what I could find in c160.log about the matrix:


[code]Merging: Merged matrix has 5597842 rows and total weight 818539599 (146.2 entries per row on average)[/code]