CADO NFS

[EdH]

Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 17:20:51 2019 [0.143 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 18:10:45 2019 [0.285 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 19:00:35 2019 [0.428 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 19:50:28 2019 [0.570 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 20:40:24 2019 [0.713 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 21:30:19 2019 [0.855 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 22:20:10 2019 [0.998 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 23:10:04 2019 [1.140 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 23:59:56 2019 [1.283 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 00:49:46 2019 [1.425 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 01:39:38 2019 [1.568 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 02:29:32 2019 [1.710 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 03:19:24 2019 [1.853 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 04:09:15 2019 [1.995 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 04:59:07 2019 [2.138 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 05:48:58 2019 [2.280 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 06:38:50 2019 [2.423 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 07:28:45 2019 [2.565 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 08:18:38 2019 [2.708 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 09:08:29 2019 [2.850 s/iter]
Info:Linear Algebra: mksol: N=860 ; ETA (N=21000): Fri Apr 12 12:40:41 2019 [3.456 s/iter]
Info:Quadratic Characters: Starting
Info:Square Root: Starting
Info:Square Root: Creating file of (a,b) values
Info:Square Root: finished
Info:Square Root: Factors: 76011547790026822726326568236942510573210375997181897566667718127 635610967526542663809596695842346695879983625491514026964044776163
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 38210.8
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 38840/38.730/46.683/50.720/0.852
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 38840/37.740/42.075/47.510/1.204
Info:Polynomial Selection (size optimized): Total time: 7618.67
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 2994.82
Info:Polynomial Selection (root optimized): Rootsieve time: 2993.53
Info:Generate Factor Base: Total cpu/real time for makefb: 50.58/11.3882
Info:Generate Free Relations: Total cpu/real time for freerel: 311.62/45.3888
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 27681208
Info:Lattice Sieving: Average J: 7707.97 for 87893 special-q, max bucket fill -bkmult 1.0,1s:1.113920
Info:Lattice Sieving: Total time: 181352s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 73.17/198.683
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 198.3s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 424.32/391.023
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 367.7s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 368.81/174.105
Info:Filtering - Merging: Total cpu/real time for merge: 879.66/789.756
Info:Filtering - Merging: Total cpu/real time for replay: 67.5/62.5504
Info:Linear Algebra: Total cpu/real time for bwc: 32922.6/8766.06
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 5463.93, iteration CPU time 0.12, COMM 0.01, cpu-wait 0.0, comm-wait 0.0 (42000 iterations)
Info:Linear Algebra: Lingen CPU time 360.93, WCT time 104.03
Info:Linear Algebra: Mksol: WCT time 2974.44, iteration CPU time 0.13, COMM 0.01, cpu-wait 0.0, comm-wait 0.0 (21000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 59.92/19.9351
Info:Square Root: Total cpu/real time for sqrt: 3066.22/474.952
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 418847/14039.8
76011547790026822726326568236942510573210375997181897566667718127 635610967526542663809596695842346695879983625491514026964044776163
nfsDone                                       100%  276     0.3KB/s   00:00    
user@computer:~$ date
Thu Apr 11 17:28:48 EDT 2019

[VBCurtis]

All of my params files include a target number of relations, so I would be able to tell when your runs use one of my files because the number of relations found would be just over my target number set.
Also, my files use a substantially larger LP bound than default, so the number of relations is quite a bit larger than default settings.
The only item I'd want that wasn't in your paste of the data dump is the size of the input number.
For my own data-gathering, I record Q-range sieved (which isn't in the summary, but also isn't that important), poly select time (size and root, but total is enough), sieve time, bwc time, wall-clock time. Recently I've been also recording the total weight of the matrix (the number printed just before linear algebra starts), so that I could see how various density selections influence matrix size; again, not too important but if we start collaborating on C150+ settings it would be nice to have.

[EdH]

I encoded the composite size in "<XXX>" right after the N in the first line:

 N<138> = 107858998268122985412779892463164903278148452826296404480200645178455386381787979632919305998206531705192000227253731053070278282139941833

I put it there instead of at the end, so it would be more accessible.

I'll work on adding your list items. Should I trim the rest, or leave everything.

Actually, I'm wondering if you were referring to the correct post. I had two posts earlier. The first was the samples for a C138, C133 and C135.

The second was to point out a discrepancy in the LA mksol ETAs.

Do any of the interested parties watch this thread for this type of info, or should I try to post to the CADO-NFS project?

[VBCurtis]

I see the composite size now, that's an excellent way to encode it.
I don't know what is valuable for other people; the timings in the summary are all that's really needed to evaluate the params for a composite.

[EdH]

I realize these have duplication, but I haven't figured out how to automatically remove duplicate lines, yet.

Anyway, Is all the info you need contained in the following three samples?

C86, C129, C121:

N<86> = 56667759878435851136396148508097449099773621318662989162599675933873393187564668667973
Lattice Sieving: params = {'maxresubmit': 5, 'lim0': 393010, 'workdir': '/tmp/cadofactor', 'maxwuerror': 2, 'name': 'c85', 'rels_wanted': 0, 'wutimeout': 10800, 'lim1': 551399, 'qmin': 146453, 'sqside': 1, 'run': True, 'gzip': True, 'maxtimedout': 100, 'qrange': 1000, 'maxwu': 10, 'maxfailed': 100}
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c85.upload/c85.polyselect2.dq2cf04y.opt_12 has Murphy_E = 2.95e-06
Generate Factor Base: Total cpu/real time for makefb: 0.47/0.135556
Generate Free Relations: Total cpu/real time for freerel: 3.85/0.515574
Lattice Sieving: sieving.run(): Task state: {'wu_submitted': 0, 'wu_timedout': 0, 'start_achievement': -1, 'qnext': 146453, 'start_real_time': 0, 'rels_wanted': 440080, 'wu_received': 0, 'wu_failed': 0, 'rels_found': 0}
Lattice Sieving: Total number of relations: 470449
Lattice Sieving: Total time: 518.71s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 1.06/0.725687
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 3.44/1.06687
Filtering - Singleton removal: Total cpu/real time for purge: 0.6/0.373629
Lattice Sieving: sieving.run(): Task state: {'wu_submitted': 219, 'rels_wanted': 498934, 'start_achievement': 0.004069714597345937, 'rels_found': 470449, 'qnext': 365000, 'wu_failed': 0, 'stats_avg_J': '1002.231756149383 12237', 'wu_timedout': 0, 'stats_total_cpu_time': '912.3099999999998', 'start_real_time': 3763830410.102031, 'wu_received': 153, 'stats_total_time': '518.71', 'stats_max_bucket_fill': '1.0,1s:1.594150'}
Lattice Sieving: sieving.run(): Task state: {'wu_submitted': 285, 'rels_wanted': 522895, 'start_achievement': 0.004069714597345937, 'rels_found': 504252, 'qnext': 431000, 'wu_failed': 0, 'stats_avg_J': '1001.8836890243903 13120', 'wu_timedout': 0, 'stats_total_cpu_time': '972.5799999999998', 'start_real_time': 3763830410.102031, 'wu_received': 164, 'stats_total_time': '551.3100000000002', 'stats_max_bucket_fill': '1.0,1s:1.594150'}
Filtering - Merging: Merged matrix has 41245 rows and total weight 7011876 (170.0 entries per row on average)
Square Root: Factors: 358031419880876938192954600148936974177 158275940969901931610142052723104530012975662949
Generate Factor Base: Total cpu/real time for makefb: 0.47/0.135556
Generate Free Relations: Total cpu/real time for freerel: 3.85/0.515574
Lattice Sieving: Total number of relations: 651829
Lattice Sieving: Total time: 722.52s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 1.46/1.11501
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 5.82/2.44343
Filtering - Singleton removal: Total cpu/real time for purge: 5.91/2.61454
Filtering - Merging: Total cpu/real time for merge: 13.92/9.40176
Filtering - Merging: Total cpu/real time for replay: 1.34/1.11267
Linear Algebra: Total cpu/real time for bwc: 27.41/8.48
Quadratic Characters: Total cpu/real time for characters: 1.43/0.41662
Square Root: Total cpu/real time for sqrt: 31.34/4.62864
Complete Factorization: Total cpu/elapsed time for entire factorization: 3518.26/98.8682
====================================================
N<129> = 148184180288647082302426937158351493009802210451497766956818199852902417872086615646923058242179971571945498904372691387599121333
Lattice Sieving: params = {'lim1': 44217255, 'qmin': 711270, 'maxresubmit': 5, 'maxwu': 10, 'rels_wanted': 0, 'run': True, 'maxwuerror': 2, 'qrange': 10000, 'wutimeout': 10800, 'sqside': 1, 'name': 'c130', 'workdir': '/tmp/cadofactor', 'gzip': True, 'lim0': 13124945, 'maxtimedout': 100, 'maxfailed': 100}
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c130.upload/c130.polyselect2.qxg0sxtu.opt_48 has Murphy_E = 5.93e-07
Generate Factor Base: Total cpu/real time for makefb: 50/10.6005
Generate Free Relations: Total cpu/real time for freerel: 311.55/40.6101
Lattice Sieving: sieving.run(): Task state: {'qnext': 711270, 'rels_found': 0, 'start_achievement': -1, 'wu_submitted': 0, 'wu_received': 0, 'wu_timedout': 0, 'rels_wanted': 22129742, 'start_real_time': 0, 'wu_failed': 0}
Lattice Sieving: Total number of relations: 22630418
Lattice Sieving: Total time: 101935s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 59.73/158.776
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 312.5/289.52
Filtering - Singleton removal: Total cpu/real time for purge: 64.6/62.8826
Lattice Sieving: sieving.run(): Task state: {'stats_avg_J': '7660.828321977037 47991', 'qnext': 2220000, 'wu_submitted': 151, 'stats_total_time': '101935.03999999998', 'start_real_time': 3763831072.010403, 'rels_wanted': 22856722, 'start_achievement': 0.013695957232578672, 'rels_found': 22630418, 'stats_max_bucket_fill': '1.0,1s:1.120020', 'wu_timedout': 0, 'stats_total_cpu_time': '195411.35000000003', 'wu_received': 67, 'wu_failed': 0}
Filtering - Merging: Merged matrix has 1228821 rows and total weight 208900009 (170.0 entries per row on average)
Square Root: Factors: 12674601840486459448841848682136209646192693900874822145430131644978522484248447 11691426851398408680908975753372972164838725377739
Generate Factor Base: Total cpu/real time for makefb: 50/10.6005
Generate Free Relations: Total cpu/real time for freerel: 311.55/40.6101
Lattice Sieving: Total number of relations: 25564242
Lattice Sieving: Total time: 120205s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 67.39/179.278
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 396.05/371.704
Filtering - Singleton removal: Total cpu/real time for purge: 337.35/309.169
Filtering - Merging: Total cpu/real time for merge: 750.5/700.049
Filtering - Merging: Total cpu/real time for replay: 47.22/38.9662
Linear Algebra: Total cpu/real time for bwc: 24120.6/6204.46
Quadratic Characters: Total cpu/real time for characters: 56.22/15.795
Square Root: Total cpu/real time for sqrt: 3060.65/418.769
Complete Factorization: Total cpu/elapsed time for entire factorization: 304850/10683
====================================================
N<121> = 8752900354798992135535713351999727922615660863269877993396052619685931654568649609816781542442396746201949471440217022437
Lattice Sieving: params = {'lim0': 3000000, 'gzip': True, 'maxwuerror': 2, 'maxtimedout': 100, 'run': True, 'qmin': 1000000, 'qrange': 10000, 'rels_wanted': 0, 'maxresubmit': 5, 'wutimeout': 10800, 'workdir': '/tmp/cadofactor', 'sqside': 1, 'maxfailed': 100, 'maxwu': 10, 'lim1': 5500000, 'name': 'c120'}
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c120.upload/c120.polyselect2.v5ylhooc.opt_90 has Murphy_E = 1.32e-06
Generate Factor Base: Total cpu/real time for makefb: 6.22/1.45258
Generate Free Relations: Total cpu/real time for freerel: 155.53/20.4213
Lattice Sieving: sieving.run(): Task state: {'start_real_time': 0, 'start_achievement': -1, 'wu_submitted': 0, 'rels_found': 0, 'wu_timedout': 0, 'wu_failed': 0, 'wu_received': 0, 'rels_wanted': 11474681, 'qnext': 1000000}
Lattice Sieving: Total number of relations: 11499970
Lattice Sieving: Total time: 42836.9s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 29.66/71.9494
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 125.83/53.849
Filtering - Singleton removal: Total cpu/real time for purge: 21.02/9.09563
Lattice Sieving: sieving.run(): Task state: {'start_achievement': 0.003286453017735308, 'wu_submitted': 453, 'stats_max_bucket_fill': '1.0,1s:1.360290', 'wu_received': 370, 'rels_wanted': 11964241, 'stats_total_time': '42836.850000000006', 'start_real_time': 3763877574.831329, 'stats_avg_J': '1913.329901275031 250190', 'qnext': 5530000, 'stats_total_cpu_time': '79603.62000000001', 'rels_found': 11499970, 'wu_timedout': 0, 'wu_failed': 0}
Lattice Sieving: sieving.run(): Task state: {'start_achievement': 0.003286453017735308, 'wu_submitted': 464, 'stats_max_bucket_fill': '1.0,1s:1.360290', 'wu_received': 429, 'rels_wanted': 13184371, 'stats_total_time': '49254.79', 'start_real_time': 3763877574.831329, 'stats_avg_J': '1911.9272845857893 288247', 'qnext': 5640000, 'stats_total_cpu_time': '91392.13000000002', 'rels_found': 12923469, 'wu_timedout': 0, 'wu_failed': 0}
Lattice Sieving: sieving.run(): Task state: {'start_achievement': 0.003286453017735308, 'wu_submitted': 475, 'stats_max_bucket_fill': '1.0,1s:1.360290', 'wu_received': 446, 'rels_wanted': 13482396, 'stats_total_time': '51120.21000000001', 'start_real_time': 3763877574.831329, 'stats_avg_J': '1911.4947376864536 299203', 'qnext': 5750000, 'stats_total_cpu_time': '94879.38000000002', 'rels_found': 13325503, 'wu_timedout': 0, 'wu_failed': 0}
Lattice Sieving: sieving.run(): Task state: {'start_achievement': 0.003286453017735308, 'wu_submitted': 536, 'stats_max_bucket_fill': '1.0,1s:1.360290', 'wu_received': 453, 'rels_wanted': 13611673, 'stats_total_time': '51708.15000000001', 'start_real_time': 3763877574.831329, 'stats_avg_J': '1911.302068579334 303590', 'qnext': 6360000, 'stats_total_cpu_time': '95994.38000000002', 'rels_found': 13484536, 'wu_timedout': 0, 'wu_failed': 0}
Filtering - Merging: Merged matrix has 742731 rows and total weight 74273389 (100.0 entries per row on average)
Square Root: Factors: 63177231098564251400862698002146040730555401588791894828480702706907 138545172091245831321071555847351711191164740172937791
Generate Factor Base: Total cpu/real time for makefb: 6.22/1.45258
Generate Free Relations: Total cpu/real time for freerel: 155.53/20.4213
Lattice Sieving: Total number of relations: 13906880
Lattice Sieving: Total time: 53405.4s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 35.98/78.1032
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 226.18/131.068
Filtering - Singleton removal: Total cpu/real time for purge: 160.12/80.2486
Filtering - Merging: Total cpu/real time for merge: 169.38/142.37
Filtering - Merging: Total cpu/real time for replay: 19.23/14.9333
Linear Algebra: Total cpu/real time for bwc: 5288.67/1377.31
Quadratic Characters: Total cpu/real time for characters: 28.14/7.54837
Square Root: Total cpu/real time for sqrt: 1244.19/172.8
Complete Factorization: Total cpu/elapsed time for entire factorization: 141585/3000.22

[VBCurtis]

Sure, looks like enough to make some observations.

On your C129, sieve time was 120ksec, while bwc was 24ksec; in wall clock time, bwc was 6200 of your 10600 sec job.

On your C121, sieve time 53ksec, bwc 5300 sec; in wall clock time, bwc was 1300 of your 3000 sec job.

So, the C121 was over-sieved enough to make bwc time 1/10th of sieve time, resulting in a substantial savings in wall clock time.

The files I posted target bwc time to be 1/6th to 1/5th of sieving time; I think your C121 shows the way for your setup at bwc time = 1/10th of sieving time. Factoring a C120 in under an hour is pretty snazzy.

[EdH]

Originally Posted by VBCurtis

Sure, looks like enough to make some observations.

On your C129, sieve time was 120ksec, while bwc was 24ksec; in wall clock time, bwc was 6200 of your 10600 sec job.

On your C121, sieve time 53ksec, bwc 5300 sec; in wall clock time, bwc was 1300 of your 3000 sec job.

So, the C121 was over-sieved enough to make bwc time 1/10th of sieve time, resulting in a substantial savings in wall clock time.

The files I posted target bwc time to be 1/6th to 1/5th of sieving time; I think your C121 shows the way for your setup at bwc time = 1/10th of sieving time. Factoring a C120 in under an hour is pretty snazzy.

I also like the C138 in just over 4.5 hours, although I think my CADO/msieve hybrid would do better, which is what's driving my interest in your params and the mpi bwc.

Anyway, here is a file of data for 87 runs ranging from 74 through 138 dd. It does include all previous samples. These are with unmodified params files from the original developmental version from a few days ago:

[VBCurtis]

I'm running a "last" test of new C120 and C130 files; if results fit the time-vs-size curve I expect, I'll have them posted in the morning for you to the params thread. C125 is still a little slower than the curve suggests should be possible, I still have some tweaking to try there to get a bit more speed.

[EdH]

I have installed all your modified params files from the other thread. These are the last four unmodified that ran. Of course, I realize that not all of the params files in my current set are modified, but I'm considering the next sets as Modified:

[EdH]

I accidentally wiped out all my scripts that ran my Aliquot Sequences on my controlling machine - the ones that communicated with the db and ran the ecm and nfs scripts!

The nfs script itself was untouched and I've found an earlier backup for the main controlling script, which is quite fortunate, since I remember having to solve several issues and I have no recollection of how I did solve them. I would have had to research and relearn a bunch. Still, reconstructing all the other scripts may be time consuming, so it might be a while before I get any more data compiled and uploaded.

BTW, in case you're interested, I used a recovery program to try to bring them back, but was only able to get the filenames, which unfortunately overwrote the original "deleted" files with zero-byte replacements.

[EdH]

I have the scripts pretty much rewritten and they are "mostly" working correctly.

Here is a file of Info for 91 composites factored using VBCurtis' params files, where available. (There may be duplicate runs for some of the numbers.)