mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   CADO-NFS (https://www.mersenneforum.org/forumdisplay.php?f=170)
-   -   CADO NFS (https://www.mersenneforum.org/showthread.php?t=11948)

EdH 2018-08-13 04:01

I'm playing with cado-nfs again and I have some data for a c156 run below. My farm is only about half its size from before and many of those are throttling due to heat.

I wouldn't mind having the optimum params for the sizes I don't yet. I'm probably going to be playing with my hybrid setup (cado-nfs/msieve) more so than completing with cado-nfs. But, if you have a particular request, let me know and I'll run a pure cado-nfs session.

This c156 used your c155 enhanced params:
[code]
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 150312
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 151793/47.410/56.749/62.720/1.079
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 151793/46.330/50.825/57.370/1.514
Info:Polynomial Selection (size optimized): Total time: 295917
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 14779.5
Info:Polynomial Selection (root optimized): Rootsieve time: 14778.1
Info:Generate Factor Base: Total cpu/real time for makefb: 37.33/8.10258
Info:Generate Free Relations: Total cpu/real time for freerel: 2466.57/322.135
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 139321222
Info:Lattice Sieving: Average J: 8229.15 for 1556330 special-q, max bucket fill: 0.629541
Info:Lattice Sieving: Total CPU time: 5.94073e+06s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 756.79/462.649
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 462.3s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 2275.86/656.124
Info:Filtering - Singleton removal: Total cpu/real time for purge: 884.42/348.291
Info:Filtering - Merging: Total cpu/real time for merge: 1805.72/1607.63
Info:Filtering - Merging: Total cpu/real time for replay: 200.41/181.724
Info:Linear Algebra: Total cpu/real time for bwc: 822208/0.000218153
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 68695.93
Info:Linear Algebra: Lingen CPU time 1900.97, WCT time 291.16
Info:Linear Algebra: Mksol: WCT time 37246.96
Info:Quadratic Characters: Total cpu/real time for characters: 247.36/69.3931
Info:Square Root: Total cpu/real time for sqrt: 13184.9/1945.28
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 7.09549e+06/172154
Info:root: Cleaning up computation data in /tmp/cado.s0wg3y09
50991922118453733800893403222829283966167525130934549348631327443632984128217423175047 17463527688603612045476003591361612022388344033034895907221918699698189
[/code]

EdH 2018-08-13 12:35

Here's a c123 run with default params:
[code]
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 21528.1
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 21720/36.270/43.664/48.210/0.888
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 21720/35.540/39.371/44.430/1.211
Info:Polynomial Selection (size optimized): Total time: 2285.24
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 1865.24
Info:Polynomial Selection (root optimized): Rootsieve time: 1863.87
Info:Generate Factor Base: Total cpu/real time for makefb: 10.62/2.3481
Info:Generate Free Relations: Total cpu/real time for freerel: 156.16/20.4439
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 13059349
Info:Lattice Sieving: Average J: 3800.88 for 178693 special-q, max bucket fill: 0.772261
Info:Lattice Sieving: Total CPU time: 149258s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 34.46/59.7937
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 59.7s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 153.42/49.1674
Info:Filtering - Singleton removal: Total cpu/real time for purge: 138.1/36.692
Info:Filtering - Merging: Total cpu/real time for merge: 203.37/175.179
Info:Filtering - Merging: Total cpu/real time for replay: 20.59/17.1169
Info:Linear Algebra: Total cpu/real time for bwc: 11504.3/0.000172377
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 899.45
Info:Linear Algebra: Lingen CPU time 175.19, WCT time 27.82
Info:Linear Algebra: Mksol: WCT time 515.74
Info:Quadratic Characters: Total cpu/real time for characters: 23.33/6.71058
Info:Square Root: Total cpu/real time for sqrt: 995.39/135.9
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 166648/4008.26
Info:root: Cleaning up computation data in /tmp/cado.8n8o3zi4
103994263537692083322430805948369677781344426601569708601629807617921045503073013 4877573650317223159406644589665201167150419
[/code]

VBCurtis 2018-08-13 18:08

3 Attachment(s)
Attached are my current params choices for c115, c120, and c130. I do not plan to tweak c115 any further, but I'm still doing a little tinkering with the bigger files.
I've only done 3 jobs at c125 level, so I don't have a new file for that just yet; I'll get to work this week on that size.

EdH 2018-08-14 02:02

Thanks Curtis,


Now I have to go back and research something, though. My latest run, a c151, which is using your enhanced params.c150, has appeared to have gotten stuck:
[code]
Info:Polynomial Selection (size optimized): Marking workunit c150_polyselect1_184500-185000 as ok (99.2% => ETA Mon Aug 13 21:37:53 2018)
[/code]There is no work being assigned, all clients are in waiting and it doesn't look like the host is doing anything anymore, either. I'll have to see if I can retrieve the restart info and attempt that. I have to go look, though - didn't I have this trouble before?


Ed

VBCurtis 2018-08-14 02:36

Ed-
You did, and I've been putting admin = xxxx settings into c125 and up files since you had that trouble, since it seems that very small starting coeffs sometimes get stuck. Try adding "admin = 8400" directly above the "admax = " line in the file.
Since you use a ton of clients, you may choose to reduce adrange to 840 or 420, so that smaller units of work are distributed. It'll keep more clients fed for more of the poly select phase. Just make sure you use a multiple of 420, since the poly select is 2-threaded and we only search multiples of 210.
That reminds me, I forgot to put admin = 1260 into the c130 params file just above.

EdH 2018-08-14 03:29

[QUOTE=VBCurtis;493844]Ed-
You did, and I've been putting admin = xxxx settings into c125 and up files since you had that trouble, since it seems that very small starting coeffs sometimes get stuck. Try adding "admin = 8400" directly above the "admax = " line in the file.
Since you use a ton of clients, you may choose to reduce adrange to 840 or 420, so that smaller units of work are distributed. It'll keep more clients fed for more of the poly select phase. Just make sure you use a multiple of 420, since the poly select is 2-threaded and we only search multiples of 210.
That reminds me, I forgot to put admin = 1260 into the c130 params file just above.[/QUOTE]
Thanks! I was just going to go search that out. admin was 1e3 and adrange was 5e2. I changed them to 8400 and 420, respectively and restarted from scratch.

I think I'm running 54 clients now, but for some reason, I keep losing some. Since they're in bash generated gnome-terminals, I never catch any crash info. They're just closed and gone.

VBCurtis 2018-08-14 04:27

My current GNFS-180 CADO run is using roughly 15 clients (30 cores), and also drops client tasks randomly. The ones that drop are on a LAN with the server, and on a faster machine than the other clients. It doesn't make much sense, but I just restart the ones that drop. I don't have the CADO server set up to auto-issue clients to other machines.

If you recall what text the server writes when a file doesn't come back in time, you could grep for that text in the log and see what ad-value causes the hangup. If it's in the middle of the poly search range, you might choose to increase the time allowed before giving up on an issued task. I think it's tasks.wu.timeout, but I'm not certain.

I believe the hang occurs because CADO only reissues a task twice before giving up, and the wrapper logic isn't smart enough to issue one more workunit above admax nor move to root opt when one range fails to come back.

henryzz 2018-08-14 09:13

[QUOTE=EdH;493847]Thanks! I was just going to go search that out. admin was 1e3 and adrange was 5e2. I changed them to 8400 and 420, respectively and restarted from scratch.

I think I'm running 54 clients now, but for some reason, I keep losing some. Since they're in bash generated gnome-terminals, I never catch any crash info. They're just closed and gone.[/QUOTE]

I believe that there are good ways of stopping them disappearing. The pause command is a possibility. I would imagine others may have better ideas.

EdH 2018-08-14 16:03

[QUOTE=VBCurtis;493849]My current GNFS-180 CADO run is using roughly 15 clients (30 cores), and also drops client tasks randomly. The ones that drop are on a LAN with the server, and on a faster machine than the other clients. It doesn't make much sense, but I just restart the ones that drop. I don't have the CADO server set up to auto-issue clients to other machines.[/QUOTE]
I start all mine separately, too. They are all LAN connected and run with local scripts that retask them when they are done sieving. I have scripts on each machine for all clients and a single one.

[QUOTE=VBCurtis;493849]If you recall what text the server writes when a file doesn't come back in time, you could grep for that text in the log and see what ad-value causes the hangup. If it's in the middle of the poly search range, you might choose to increase the time allowed before giving up on an issued task. I think it's tasks.wu.timeout, but I'm not certain.

I believe the hang occurs because CADO only reissues a task twice before giving up, and the wrapper logic isn't smart enough to issue one more workunit above admax nor move to root opt when one range fails to come back.[/QUOTE]This part I may have to study more...

[QUOTE=henryzz;493861]I believe that there are good ways of stopping them disappearing. The pause command is a possibility. I would imagine others may have better ideas.[/QUOTE]
I suppose I could add ";sleep 43200" to my gnome-terminal command, too, but I'd have to do that for every script on all the machines. Maybe the next time I do a machine-wide rewrite...

EdH 2018-08-15 12:15

Hey Curtis,


Here's a c151 run with your enhancements from before:
[code]
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 167972
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 168252/45.550/54.833/60.700/1.149
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 168252/43.600/48.610/55.130/1.161
Info:Polynomial Selection (size optimized): Total time: 302064
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 8840.08
Info:Polynomial Selection (root optimized): Rootsieve time: 8838.9
Info:Generate Factor Base: Total cpu/real time for makefb: 32.13/6.7074
Info:Generate Free Relations: Total cpu/real time for freerel: 1239.29/161.525
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 76147350
Info:Lattice Sieving: Average J: 7754.82 for 1053607 special-q, max bucket fill: 0.61839
Info:Lattice Sieving: Total CPU time: 3.56919e+06s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 459.26/299.79
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 298.7s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 1633.11/704.758
Info:Filtering - Singleton removal: Total cpu/real time for purge: 1167.91/523.987
Info:Filtering - Merging: Total cpu/real time for merge: 1506.16/1320.63
Info:Filtering - Merging: Total cpu/real time for replay: 137.02/115.897
Info:Linear Algebra: Total cpu/real time for bwc: 401675/0.000190258
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 33366.92
Info:Linear Algebra: Lingen CPU time 1324.14, WCT time 203.31
Info:Linear Algebra: Mksol: WCT time 18229.37
Info:Quadratic Characters: Total cpu/real time for characters: 169.38/47.1608
Info:Square Root: Total cpu/real time for sqrt: 9452.87/1305.95
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 4.29757e+06/107871
Info:root: Cleaning up computation data in /tmp/cado.rma48tdr
1142592283136731570545301662765774155859478622044326713 2907521994696204311372547245994693384494827511814130029530759988300338906185818977360167429826659
[/code]Ed

EdH 2018-08-17 03:52

Here's a c132 run using your c130 file from post 294. I added admin = 1260 and changed adrange to 840:
[code]
Info:Square Root: Total cpu/real time for sqrt: 2378.22/323.447
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 293781
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 297375/38.750/48.467/58.100/1.577
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 297375/37.450/42.329/49.080/0.895
Info:Polynomial Selection (size optimized): Total time: 77673.3
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 1253.51
Info:Polynomial Selection (root optimized): Rootsieve time: 1252.44
Info:Generate Factor Base: Total cpu/real time for makefb: 8.08/1.76903
Info:Generate Free Relations: Total cpu/real time for freerel: 619.78/81.2041
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 36903672
Info:Lattice Sieving: Average J: 3789.6 for 383367 special-q, max bucket fill: 0.739391
Info:Lattice Sieving: Total CPU time: 333999s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 98.16/316.572
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 316.3s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 458.01/214.492
Info:Filtering - Singleton removal: Total cpu/real time for purge: 184.18/63.6672
Info:Filtering - Merging: Total cpu/real time for merge: 242.93/211.991
Info:Filtering - Merging: Total cpu/real time for replay: 35.22/28.8648
Info:Linear Algebra: Total cpu/real time for bwc: 27425.2/0.000190735
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 2275.83
Info:Linear Algebra: Lingen CPU time 321.19, WCT time 50.15
Info:Linear Algebra: Mksol: WCT time 1244.45
Info:Quadratic Characters: Total cpu/real time for characters: 46.15/12.6496
Info:Square Root: Total cpu/real time for sqrt: 2378.22/323.447
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 444422/9602.2
Info:root: Cleaning up computation data in /tmp/cado.0z5tf6z_
1638528746893449983213861346839041825501626828319370583078901 317138241249483753195631898143827752284973893404091800009687302134027717
[/code]

VBCurtis 2018-08-17 06:20

Thanks, Ed! This gives us a baseline for future tests on your farm, as well as gives me more info about how different systems produce quite different balances of matrix time vs sieve time.
I have a couple of quite fast results at c140 and c145; I'll get those two files posted next week. I have little data in 125 and 135 regions yet, and no factorization results fast enough to fit the trendline of other best results from 100 to 150 digits.

wombatman 2018-10-03 03:43

Is there a way to force re-running of factorization with CADO-NFS at a particular step? My HD filled up when I wasn't paying attention and CADO couldn't save files related to the mksol step. Thus, it errored out at the end of that step and won't restart mksol since it "completed", but I'm not sure if I can edit something to cause just the mksol (and subsequent) steps to be run.

VBCurtis 2018-10-03 05:50

1 Attachment(s)
I got away for a while from refining parameters for small tasks, but I did complete a GNFS-180 (13*2^906-1) using CADO for sieving!
I used msieve for poly select. Poly score was 9.891e-14, a record for C180.
CADO params attached; I did a little test-sieving with GGNFS, as well as trying two or three sets of parameters for a day each on CADO. I ended up using lim's of 60M and 100M, 32/33LP, I=15, and 64/95 for MFB. ncurves was set to 17 on the 2LP side, 12 on 3LP side.
Sieving Q=10M to 87M yielded just over 620M relations; alas, the host machine ran out of disk space while filtering, so I copied all the relations to a single file on another machine and set msieve to work. Density 96 allowed a matrix 23.5M in size; I ran out of patience and disk to sieve enough for my preferred density around 110.
Roughly 50 cores were used for 5 weeks for sieving; nearly my entire farm.
In hindsight, 32/33 is one LP too big. Default CADO uses 31/32 and I=14; I should have bumped one of these one step, but not both. I had initial yield around 12, and average yield of 8.0-8.1; even for me, that's a bit high! If I were to try another this size with CADO, I would choose 31/32LP and I=15. My next job will be GNFS-186, for which I plan to bump the lim's to 80M/120M, but leave the other parameters alone.
Yield from CADO was substantially better than GGNFS; alas, my GGNFS test sieve notes were lost during the factorization and I'm too lazy to repeat the tests.

EdH 2018-10-03 18:44

[QUOTE=wombatman;497263]Is there a way to force re-running of factorization with CADO-NFS at a particular step? My HD filled up when I wasn't paying attention and CADO couldn't save files related to the mksol step. Thus, it errored out at the end of that step and won't restart mksol since it "completed", but I'm not sure if I can edit something to cause just the mksol (and subsequent) steps to be run.[/QUOTE]
If it's helpful, I'm currently running a hybrid CADO-NFS/msieve procedure for my factoring. All the LA is done by msieve.

I use the following in a bash script with the composite ($1) in the command line. I force the temporary directory to be /tmp/hybrid so I know where it is:
[code]
#!/bin/bash/

cd Math/cado-nfs
./cado-nfs.py $1 tasks.workdir=/tmp/hybrid tasks.filter.run=false
echo "Finished cado-nfs!"
cd /tmp/hybrid
cat c*.upload/*.gz >comp.dat.gz
cat *.poly >comp.polyT
mv comp.polyT comp.poly
echo "n: $1" >comp.n
echo "N $1" >comp.fb
~/Math/cado-nfs/poly2fb
~/Math/msieve/msieve -i comp.n -s comp.dat.gz -l compmsieve.log -nf comp.fb -t 8 -nc
cat compmsieve.log | grep " factor: "
cat compmsieve.log | grep " factor: " > ~/FactorList
[/code]The last line is in case /tmp/hybrid gets removed. Of note, I have run into requests for more relations on occasion.

At some point, I'll probably add the procedure to my "How I ..." pages, but I haven't because of not solving several parameter issues yet.

wombatman 2018-10-04 02:42

Thanks. If I can figure out anything else, I'll take a shot with that.

wombatman 2018-10-05 02:19

I was able to get the linear algebra to re-run by deleting the bwc folder under the /tmp/ work directory. Just posting this in case someone else runs into the same issue (or I do again...)

EdH 2018-10-05 03:15

[QUOTE=wombatman;497391]I was able to get the linear algebra to re-run by deleting the bwc folder under the /tmp/ work directory. Just posting this in case someone else runs into the same issue (or I do again...)[/QUOTE]
That sounds good. I wondered if there may be a way, but I don't know enough about CADO-NFS to know where to look. I remember deleting some files to rerun some msieve steps in a similar fashion, but I'd have to research what files I did that with, too.

VictordeHolland 2018-11-27 12:07

I'm running a C146 using the default parameters.c145 using CADO-NFS (3.0-dev of a few months old)
So I can compare it with VBCurtis better parameters :)

VictordeHolland 2018-12-08 12:59

2 Attachment(s)
I did do two C116s with different parameters, but the difference was negligible:

Aliquot 2515098 index 518 C116 (default c115 parameters)
[code]57260624163558114907105482591325041150144795085724341980289789123163352120442484657512017354806368372606893536015689[/code][code]Info:Square Root: Factors: 5366819040225876374810043852991007541729343594563691842154849129 10669378589882202242046713973010894938014632978556641
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 21492.9
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 21788/33.700/41.357/46.600/1.112
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 21788/32.870/36.651/41.770/0.956
Info:Polynomial Selection (size optimized): Total time: 2839.91
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 501.88
Info:Polynomial Selection (root optimized): Rootsieve time: 500.25
Info:Generate Factor Base: Total cpu/real time for makefb: 9.41/0.595049
Info:Generate Free Relations: Total cpu/real time for freerel: 108.79/3.46431
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 7476517
Info:Lattice Sieving: Average J: 1911.27 for 175501 special-q, max bucket fill -bkmult 1.0,1s:1.156260
Info:Lattice Sieving: Total time: 39802.7s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 26.93/38.9862
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 37.9s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 206.85/149.014
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 125.0s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 177.66/156.63
Info:Filtering - Merging: Total cpu/real time for merge: 177.67/141.849
Info:Filtering - Merging: Total cpu/real time for replay: 22.74/18.5985
Info:Linear Algebra: Total cpu/real time for bwc: 4464.77/404.3
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 238.7, iteration CPU time 0.01, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (15000 iterations)
Info:Linear Algebra: Lingen CPU time 189.98, WCT time 17.9
Info:Linear Algebra: Mksol: WCT time 127.46, iteration CPU time 0.01, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (7000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 32.85/5.83791
Info:Square Root: Total cpu/real time for sqrt: 550.98/79.8058
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 102295/3488.36[/code]Aliquot 2324328 index 1754 C116 (parameters from [URL]https://mersenneforum.org/showpost.php?p=493816&postcount=294[/URL])
[code]16231073505513910375586118092559047695476874861018044665771298171360337986331251933825629231182569153705365622682051[/code][code]Info:Square Root: Factors: 24031256700679804831899451534455089680339988350903629141920263071775427401 675415094086809031967441080036510893844651
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 12883.5
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 13313/32.880/41.399/50.410/1.390
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 13313/32.880/36.561/42.320/1.095
Info:Polynomial Selection (size optimized): Total time: 2000.54
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 523.23
Info:Polynomial Selection (root optimized): Rootsieve time: 522.04
Info:Generate Factor Base: Total cpu/real time for makefb: 6.27/0.403468
Info:Generate Free Relations: Total cpu/real time for freerel: 216.27/6.85179
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 13327906
Info:Lattice Sieving: Average J: 1888.34 for 188639 special-q, max bucket fill -bkmult 1.0,1s:1.177090
Info:Lattice Sieving: Total time: 40384.9s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 47.52/70.735
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 70.19999999999999s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 289.07/166.718
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 140.10000000000002s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 191.33/155.735
Info:Filtering - Merging: Total cpu/real time for merge: 156.48/120.167
Info:Filtering - Merging: Total cpu/real time for replay: 22.89/18.0962
Info:Linear Algebra: Total cpu/real time for bwc: 4680.55/469.7
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 292.48, iteration CPU time 0.01, COMM 0.0, cpu-wait 0.01, comm-wait 0.0 (14000 iterations)
Info:Linear Algebra: Lingen CPU time 184.26, WCT time 17.55
Info:Linear Algebra: Mksol: WCT time 138.62, iteration CPU time 0.01, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (7000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 35.84/6.89125
Info:Square Root: Total cpu/real time for sqrt: 660.36/92.6011
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 101763/3579.63[/code]

VictordeHolland 2018-12-08 17:05

Aliquot 2380332 index 1881 C120 (default parameters)

[code]199287076077883734257277351040454144441000030759204246737980837594631635996650954243527367115572838436022957492373596441
[/code][code]nfo:Square Root: Factors: 7881718165498387521633582355355082520893963326151214533539440818991 25284724966473365753246256372756737619902873207696951
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 19893.3
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 20038/34.700/42.654/47.580/1.068
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 20038/33.860/37.787/43.120/0.946
Info:Polynomial Selection (size optimized): Total time: 4368.44
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 1693.06
Info:Polynomial Selection (root optimized): Rootsieve time: 1691.2
Info:Generate Factor Base: Total cpu/real time for makefb: 11.41/0.716381
Info:Generate Free Relations: Total cpu/real time for freerel: 215.96/6.88374
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 13210726
Info:Lattice Sieving: Average J: 1886.16 for 231501 special-q, max bucket fill -bkmult 1.0,1s:1.181410
Info:Lattice Sieving: Total time: 57872.1s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 46.94/79.2834
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 78.29999999999998s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 368.44/240.362
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 200.29999999999998s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 282.74/235.527
Info:Filtering - Merging: Total cpu/real time for merge: 280.69/221.189
Info:Filtering - Merging: Total cpu/real time for replay: 30.63/24.3347
Info:Linear Algebra: Total cpu/real time for bwc: 8233.92/769.52
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 469.89, iteration CPU time 0.02, COMM 0.0, cpu-wait 0.01, comm-wait 0.0 (19000 iterations)
Info:Linear Algebra: Lingen CPU time 247.53, WCT time 24.12
Info:Linear Algebra: Mksol: WCT time 257.82, iteration CPU time 0.02, COMM 0.0, cpu-wait 0.01, comm-wait 0.0 (9500 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 48.31/9.27324
Info:Square Root: Total cpu/real time for sqrt: 892.67/125.86
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 147386/5296.77
[/code]Aliquot 2729472 index 1295 C120 parameters from [URL]https://mersenneforum.org/showpost.php?p=493816&postcount=294[/URL]

[code]205139061732834871832777149640498376066462709627022246947242115024979684074456599994309638679576103310646656416412986993[/code][code]Info:Square Root: Factors: 551378319562220129379188795853993269123323508651567304673518464786424349 372047747353776779859748207675312690256707865957
Info:Square Root: Total cpu/real time for sqrt: 883.47/124.153
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 20828.5
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 20997/35.120/43.125/51.980/1.501
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 20997/34.030/37.941/43.730/1.044
Info:Polynomial Selection (size optimized): Total time: 4206.01
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 1018.39
Info:Polynomial Selection (root optimized): Rootsieve time: 1017.02
Info:Generate Factor Base: Total cpu/real time for makefb: 11.48/0.695448
Info:Generate Free Relations: Total cpu/real time for freerel: 425.41/13.5391
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 17128933
Info:Lattice Sieving: Average J: 1906.83 for 200409 special-q, max bucket fill -bkmult 1.0,1s:1.224170
Info:Lattice Sieving: Total time: 53632.9s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 60.59/98.9291
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 98.8s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 301.1/103.734
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 95.39999999999999s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 124.38/42.177
Info:Filtering - Merging: Total cpu/real time for merge: 202.8/152.94
Info:Filtering - Merging: Total cpu/real time for replay: 28.72/22.8961
Info:Linear Algebra: Total cpu/real time for bwc: 8543.11/836.9
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 485.25, iteration CPU time 0.02, COMM 0.0, cpu-wait 0.01, comm-wait 0.0 (19000 iterations)
Info:Linear Algebra: Lingen CPU time 232.35, WCT time 22.34
Info:Linear Algebra: Mksol: WCT time 303.19, iteration CPU time 0.02, COMM 0.0, cpu-wait 0.01, comm-wait 0.0 (10000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 45.39/7.23988
Info:Square Root: Total cpu/real time for sqrt: 883.47/124.153
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 139378/5141.93
[/code]So those do improve CPU and elapsed time.
147386/5296.77
[B]139378/5141.93[/B]

VBCurtis 2018-12-08 17:08

I sent my params files for C120 and lower to CADO in early 2018; I believe they're included in the 3.0 git versions. The "new" params you list from the forum in August are the result of trying to eke out 5% more speed, e.g. from spending less time in poly select.

I think I didn't have a clear case that the new August params were obviously faster, so I didn't send them to the CADO folks. I do believe they're faster, but barely so. In your case, the second input was a factor of 3 smaller, so "should" have been ~5% faster even using the same params; yet, in this test, the second job was ~500 thread-sec faster (0.5%). Hrmmm

Thanks for reporting your test!

Gimarel 2018-12-09 11:06

I get a 30% improvement (faster sieving and smaller matrix) compared to current git for a c100 with this patch:
[CODE]diff --git a/parameters/factor/params.c100 b/parameters/factor/params.c100
index e0f8b151c..e72a78e41 100644
--- a/parameters/factor/params.c100
+++ b/parameters/factor/params.c100
@@ -27,18 +27,23 @@ tasks.polyselect.ropteffort = 0.5
# Sieve
###########################################################################

-tasks.lim0 = 919082
-tasks.lim1 = 1051872
-tasks.lpb0 = 24
-tasks.lpb1 = 25
-tasks.sieve.mfb0 = 49
-tasks.sieve.mfb1 = 50
+tasks.lim0 = 620000
+tasks.lim1 = 950000
+tasks.lpb0 = 26
+tasks.lpb1 = 26
+tasks.sieve.mfb0 = 52
+tasks.sieve.mfb1 = 52
tasks.sieve.ncurves0 = 11
tasks.sieve.ncurves1 = 16
-tasks.I = 11
+tasks.sieve.lambda0 = 1.775
+tasks.sieve.lambda1 = 1.775
+tasks.I = 12
+
+#tasks.sieve.qrange = 10000
+tasks.sieve.qrange = 2000
+tasks.qmin = 300000
+tasks.sieve.rels_wanted = 3200000

-tasks.sieve.qrange = 10000
-tasks.qmin = 1051872

###########################################################################
# Filtering[/CODE]

VictordeHolland 2018-12-10 23:35

[QUOTE=Gimarel;502155]I get a 30% improvement (faster sieving and smaller matrix) compared to current git for a c100 with this patch:
[/QUOTE]
RSA100

[B]default params[/B]
Total cpu/elapsed time for entire factorization:
22154.3/645.17

[B]Your params[/B]
Total cpu/elapsed time for entire factorization:
16324.5/624.355

:bow:

VBCurtis 2019-01-03 07:27

1 Attachment(s)
Attached is a new params file for C105. (delete the .txt suffix for CADO to recognise)
I've discovered that adding ~5% to target relations wanted yields a 30-50% reduction in matrix-solve time. Second, starting at very low Q produces lots of duplicate relations, but still sieves faster than a more normal starting Q value. I've moved Qmin from lim0 / 4 to lim0 / 16.

I appreciate any comparison runs to CADO-git 3.0 or msieve/factmsieve.py.

VictordeHolland 2019-01-04 00:04

[QUOTE=VBCurtis;504740]Attached is a new params file for C105. (delete the .txt suffix for CADO to recognise)
I've discovered that adding ~5% to target relations wanted yields a 30-50% reduction in matrix-solve time. Second, starting at very low Q produces lots of duplicate relations, but still sieves faster than a more normal starting Q value. I've moved Qmin from lim0 / 4 to lim0 / 16.

I appreciate any comparison runs to CADO-git 3.0 or msieve/factmsieve.py.[/QUOTE]


default c105 params

[code]109632612184518587513828778388808335173534513011647049031182822755389306787273092842624843623516386466817[/code] [code]Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 9952.64
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 9418/30.140/36.778/42.590/1.071
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 9418/29.410/32.699/37.260/0.964
Info:Polynomial Selection (size optimized): Total time: 808.92
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 235.3
Info:Polynomial Selection (root optimized): Rootsieve time: 234.53
Info:Generate Factor Base: Total cpu/real time for makefb: 3.32/0.233966
Info:Generate Free Relations: Total cpu/real time for freerel: 107.57/3.43082
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 5314822
Info:Lattice Sieving: Average J: 1901.73 for 48049 special-q, max bucket fill -bkmult 1.0,1s:1.191890
Info:Lattice Sieving: Total time: 9941.66s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 18.54/26.7606
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 26.4s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 85.03/49.4793
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 43.599999999999994s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 61.84/31.2877
Info:Filtering - Merging: Total cpu/real time for merge: 61.04/42.6015
Info:Filtering - Merging: Total cpu/real time for replay: 8.2/6.48986
Info:Linear Algebra: Total cpu/real time for bwc: 743.78/87.26
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 50.75, iteration CPU time 0, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (6000 iterations)
Info:Linear Algebra: Lingen CPU time 59.71, WCT time 5.66
Info:Linear Algebra: Mksol: WCT time 12.55, iteration CPU time 0, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (3000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 11.27/1.93752
Info:Square Root: Total cpu/real time for sqrt: 183.65/27.1619
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 29423.9/1054.2
20134157657651598100858476591180697221596297298877 5445105479386907888285946208202118057060224985751395221
[/code][B]29,424 CPUsec / 1,054 WCT[/B]

new c105 params
[code]107774169621361997338509361816744634033698066592448031046848063179170577441576972428346870619448060739421[/code][code]Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 4473.5
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 4331/30.060/37.058/44.280/1.429
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 4331/29.800/32.734/37.690/1.062
Info:Polynomial Selection (size optimized): Total time: 413.75
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 156.39
Info:Polynomial Selection (root optimized): Rootsieve time: 155.82
Info:Generate Factor Base: Total cpu/real time for makefb: 2.9/0.21063
Info:Generate Free Relations: Total cpu/real time for freerel: 107.34/3.42818
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 5795540
Info:Lattice Sieving: Average J: 1920.26 for 48157 special-q, max bucket fill -bkmult 1.0,1s:1.252780
Info:Lattice Sieving: Total time: 9458.12s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 19.41/27.519
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 27.2s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 88/33.5168
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 29.499999999999996s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 40.61/18.0718
Info:Filtering - Merging: Total cpu/real time for merge: 92.75/64.2703
Info:Filtering - Merging: Total cpu/real time for replay: 11.11/8.90364
Info:Linear Algebra: Total cpu/real time for bwc: 995.79/81.67
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 36.87, iteration CPU time 0, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (7000 iterations)
Info:Linear Algebra: Lingen CPU time 82.87, WCT time 7.8
Info:Linear Algebra: Mksol: WCT time 27.01, iteration CPU time 0.01, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (4000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 17.61/3.01295
Info:Square Root: Total cpu/real time for sqrt: 255.01/37.0997
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 30104.3/951.544
86095283758496432104324268439192937 1251801084989468478426112275293353943851931081438584113030607271192533
[/code][B]30,104 CPUsec / 952 WCT[/B]

The machine (dual Xeon E5-2650) has quite a bit of variance on these small composites, so I might need to do a couple of runs and take the average to make a fair comparison.

VictordeHolland 2019-01-04 01:11

another fast c105
[code]106940938640385179198475163991847262203326129306969135559785399736892912049980963873983444092667843094963[/code]for this one I used your polyselect params, but slightly different sieve params:

[code]tasks.lim0 = 750000
tasks.lim1 = 1200000
tasks.lpb0 = 26
tasks.lpb1 = 26
tasks.sieve.mfb0 = 52
tasks.sieve.mfb1 = 52
tasks.sieve.ncurves0 = 11
tasks.sieve.ncurves1 = 16
tasks.I = 12

tasks.sieve.qrange = 5000
tasks.sieve.qmin = 100000
tasks.sieve.rels_wanted = 5700000


###########################################################################
# Filtering
###########################################################################

tasks.filter.purge.keep = 170
tasks.filter.maxlevel = 20
tasks.filter.target_density = 155.0[/code][code]Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 4473.5
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 4246/29.740/37.012/44.730/1.470
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 4246/29.280/32.778/38.300/1.149
Info:Polynomial Selection (size optimized): Total time: 405.26
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 173.41
Info:Polynomial Selection (root optimized): Rootsieve time: 172.83
Info:Generate Factor Base: Total cpu/real time for makefb: 2.54/0.168372
Info:Generate Free Relations: Total cpu/real time for freerel: 108.61/3.46267
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 6718856
Info:Lattice Sieving: Average J: 1910.06 for 43933 special-q, max bucket fill -bkmult 1.0,1s:1.294890
Info:Lattice Sieving: Total time: 9501.16s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 23.24/32.0983
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 31.900000000000002s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 103.57/57.6821
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 51.3s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 56.14/39.6721
Info:Filtering - Merging: Total cpu/real time for merge: 73.47/50.1402
Info:Filtering - Merging: Total cpu/real time for replay: 8.51/6.87158
Info:Linear Algebra: Total cpu/real time for bwc: 716.98/59.17
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 29.15, iteration CPU time 0, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (6000 iterations)
Info:Linear Algebra: Lingen CPU time 69.91, WCT time 6.98
Info:Linear Algebra: Mksol: WCT time 14.8, iteration CPU time 0, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (3000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 15.29/2.85822
Info:Square Root: Total cpu/real time for sqrt: 232.4/33.9271
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 28071.1/882.506
96419734812774740119994365852866859903514856721672957 1109118779968024502616433643203969184292927435816559
[/code][B]28,071 CPUsec / 883 WCT[/B]
Could be an outlier, so we need moar data!

VBCurtis 2019-01-04 03:51

Thanks; seems I'm only helping wall-clock time (nice, but not all that much help).
I'll do some more research. I've been tracking poly select time, sieve time, bwc time; but I have been using the sum of those three as a proxy for job length, ignoring filtering time and other little steps (e.g. free relations).

Gimarel 2019-01-04 12:12

C105 Parameter
 
1 Attachment(s)
Better sieving parameters for C105. I didn't change the poly selection part.

In my experience [C]tasks.lim1[/C] should be equal to the top end of the (expected) sieving range.
If you want to oversieve a bit to reduce the matrix size, it's better to use the parameter [C]tasks.filter.required_excess[/C] than to specify [C]tasks.sieve.rels_wanted[/C].

In my example the [C]lambda0[/C] and [C]lambda1[/C] paramters are essential because otherwise the siever produces to much useless relations. These are not optimized but should be about right. The parameter [C]tasks.sieve.rels_wanted[/C] is needed in my example, because cado overestimates the neede relations by a factor 2.

The parameters[C]tasks.lim0[/C] and [C]tasks.qmin[/C] are not optimised.

With these parameters I get a sieving speedup of about 10-15% and a smaller matrix.

VictordeHolland 2019-01-04 19:07

[QUOTE=Gimarel;504908]Better sieving parameters for C105. I didn't change the poly selection part.

In my experience [C]tasks.lim1[/C] should be equal to the top end of the (expected) sieving range.
If you want to oversieve a bit to reduce the matrix size, it's better to use the parameter [C]tasks.filter.required_excess[/C] than to specify [C]tasks.sieve.rels_wanted[/C].

In my example the [C]lambda0[/C] and [C]lambda1[/C] paramters are essential because otherwise the siever produces to much useless relations. These are not optimized but should be about right. The parameter [C]tasks.sieve.rels_wanted[/C] is needed in my example, because cado overestimates the needed relations by a factor 2.

The parameters[C]tasks.lim0[/C] and [C]tasks.qmin[/C] are not optimised.

With these parameters I get a sieving speedup of about 10-15% and a smaller matrix.[/QUOTE]
We want more parameter files from you!

c105
[code]107713203868901378890486921109668147250599518916591688453404410233186403423078799985908643904700547429021[/code]
[code]
Info:Square Root: Factors: 464884127923667954781021992089902060842382186081760172952730363 231699035090712841274341777335868170736967
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 4473.5
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 4342/28.940/37.049/44.340/1.438
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 4342/28.940/32.749/37.280/1.096
Info:Polynomial Selection (size optimized): Total time: 366.14
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 195.9
Info:Polynomial Selection (root optimized): Rootsieve time: 195.31
Info:Generate Factor Base: Total cpu/real time for makefb: 2.06/0.151476
Info:Generate Free Relations: Total cpu/real time for freerel: 215.04/6.83992
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 6319648
Info:Lattice Sieving: Average J: 1918.22 for 63809 special-q, max bucket fill -bkmult 1.0,1s:1.301890
Info:Lattice Sieving: Total time: 8788.08s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 21.97/25.6841
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 25.5s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 111.77/40.0544
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 30.6s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 57.94/24.8971
Info:Filtering - Merging: Total cpu/real time for merge: 45.12/29.8022
Info:Filtering - Merging: Total cpu/real time for replay: 7.98/6.32849
Info:Linear Algebra: Total cpu/real time for bwc: 534.81/42.85
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 18.05, iteration CPU time 0, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (5000 iterations)
Info:Linear Algebra: Lingen CPU time 57.86, WCT time 5.44
Info:Linear Algebra: Mksol: WCT time 12.33, iteration CPU time 0, COMM 0.0, cpu-wait 0.0, comm-wait 0.0 (3000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 13.61/2.23025
Info:Square Root: Total cpu/real time for sqrt: 150.89/22.1238
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 25453.4/787.263[/code][B]25453 CPUsec / 787 WCT[/B]

VBCurtis 2019-01-05 21:43

[QUOTE=Gimarel;504908]Better sieving parameters for C105. I didn't change the poly selection part.

In my experience [C]tasks.lim1[/C] should be equal to the top end of the (expected) sieving range.
If you want to oversieve a bit to reduce the matrix size, it's better to use the parameter [C]tasks.filter.required_excess[/C] than to specify [C]tasks.sieve.rels_wanted[/C].

In my example the [C]lambda0[/C] and [C]lambda1[/C] paramters are essential because otherwise the siever produces to much useless relations. These are not optimized but should be about right. The parameter [C]tasks.sieve.rels_wanted[/C] is needed in my example, because cado overestimates the neede relations by a factor 2.

The parameters[C]tasks.lim0[/C] and [C]tasks.qmin[/C] are not optimised.

With these parameters I get a sieving speedup of about 10-15% and a smaller matrix.[/QUOTE]

Nice! I tested on a C103, and got a faster time than my previous best at C102.
I used my own poly-select parameters (posted on my C105 file).
If I understand Lambda correctly, 1.775 * 27 = 48, so you're using mfb0 and mfb1 48 for a 27LP job. Interesting! I confirmed this by setting those to 48 rather than 54, with almost no change in sieve time.

I then changed qmin to 60k and rels_wanted to 6.5M (to correct for the massive duplicate-relations produced at small Q). This job filtered twice, as did Gimarel's parameters; sieve time, CPU time, and WCT were all 8+% better than Gimarel's settings. I run 30-threaded on a Xeon, so quite a lot of relations are found during the first filtering pass; other testers may find different timings using fewer threads.
My results:
Gimarel's params: sieve time 4890, CPU time 17700, WCT 541.
Setting Q=60k: sieve time 4220, CPU time 16460, WCT 502

I did many other tests, such as lambda = 1.75 or 1.8, Q-min 30k, 40k, 80k; none faster than 502 WCT but lots around 520.

It is clear that CPU time has some calculation flaw, as WCT * threads > CPU time. The machine is a dual 10-core, running 10-threaded msieve LA; I use 30 threads for tasks, 20 threads for server tasks.

I'll next try different lim's and LA settings.

Nooks 2019-01-16 23:59

Curtis:

Regarding the c172 from last week: if I run more GNFS (though I don't plan to right this moment) I will commit to not fiddling with parameters during its run as I did here: particularly I changed the target matrix density to 110 from 170 (I was getting a lot of duplicates and not many relations per workunit) and I tried adjusting qmin down (from 19600000 to 9800000) towards the end, though I don't think that had any effect at all---it certainly did not start sieving below what it had already done.

Lattice Sieving: Total time: 1.49137e+07s
Linear Algebra: Total cpu/real time for bwc: 4.80653e+06/653502
Complete Factorization: Total cpu/elapsed time for entire factorization: 2.65919e+07/1.17675e+06
Filtering - Merging: Merged matrix has 15059957 rows and total weight 1656595352 (110.0 entries per row on average)

Like before, the timing information is a bit of a mess.

To extract timing information from the log:

$ egrep -i 'total.+time' 37771_279.log | grep -viw debug | sed -e's/^.\+Info://' | uniq -c | less

A bit of a hack but it lets me see that the sieving time only increases as I stop and start the process. Apologies if this is less than useful. I need to be a little more methodical about how I approach this.

VBCurtis 2019-01-17 01:02

Thanks for the data! Also thanks for the unix protip to extract the info.
Your changes don't distort the data much, if at all; changing q-min after the run begins won't alter sieve behavior, as you discovered. Changing matrix density alters post-processing, but does nothing to the sieve process.

EdH 2019-03-27 19:20

Ubuntu 18.04 and CADO-NFS
 
I've upgraded some of my machines that were running CADO-NFS from Ubuntu 16.04 to Ubuntu 18.04. Now they won't run the previous CADO-NFS and trying to make from scratch also fails. Any thoughts? Is there something simple I'm missing?

[code]
[ 44%] Building C object sieve/strategies/CMakeFiles/benchfm.dir/utils_st/tab_strategy.c.o
Linking CXX executable benchfm
/usr/bin/ld: CMakeFiles/benchfm.dir/utils_st/tab_point.c.o: relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
sieve/strategies/CMakeFiles/benchfm.dir/build.make:287: recipe for target 'sieve/strategies/benchfm' failed
make[2]: *** [sieve/strategies/benchfm] Error 1
CMakeFiles/Makefile2:1454: recipe for target 'sieve/strategies/CMakeFiles/benchfm.dir/all' failed
make[1]: *** [sieve/strategies/CMakeFiles/benchfm.dir/all] Error 2
Makefile:123: recipe for target 'all' failed
make: *** [all] Error 2
Makefile:7: recipe for target 'all' failed
make: *** [all] Error 2
[/code]I have recompiled GMP and GMP-ECM with no issues and YAFU still runs OK, but I haven't tried recompiling YAFU or any of my other packages, yet.

EdH 2019-03-27 20:44

Previous post update
 
Apparently, the upgrade to 18.04 on my machines resulted in Python being removed. Installing both Python and Python3 has seemed to fix the issue.

swellman 2019-03-28 01:52

[QUOTE=EdH;511983]Apparently, the upgrade to 18.04 on my machines resulted in Python being removed. Installing both Python and Python3 has seemed to fix the issue.[/QUOTE]

Had the same experience a few weeks ago with 18.04. CADO would just crash despite the fact I had installed Python3. It all worked after I installed the missing Python package.

EdH - can you add this tip to your excellent install guide for CADO?

EdH 2019-03-28 02:34

[QUOTE=swellman;512013]Had the same experience a few weeks ago with 18.04. CADO would just crash despite the fact I had installed Python3. It all worked after I installed the missing Python package.

EdH - can you add this tip to your excellent install guide for CADO?[/QUOTE]
Thanks for the confirmation. I do see that I only need to reinstall Python, rather than Python3.

I will add a note in a day or so. I didn't want to edit anything while the board was acting up.

VBCurtis 2019-03-31 08:36

I have one machine running 18.04, and a few running 16.04. The one running 18.04 had a CADO install that won't play well with the installs on the other machines; when trying a distributed GNFS job, the clients on 16.04 (even with the newest CADO git) note that /download/las is different than the one on the server, download las from the server, and it crashes upon invocation because 18.04 has a newer GCC so the lib's don't match.
So, until/unless I upgrade my 16.04 machines and rebuild CADO, I can't run a server/client setup for a big job. Annoying, and I wish CADO wouldn't check the client las versus the server las.
Unfortunately, the one running 18.04 is the only machine that all my others can connect to, sigh.

EdH 2019-03-31 13:53

I'm not sure if this helps, but I had a similar issue due to various hardware and have to use --binddir=build/<username>/ on my clients. My current server is 16.04 and some clients are 18.04.

Nooks 2019-04-01 15:34

EdH is right, --bindir is the flag you want.

I don't point it at my build/ directory, but at /pkg/cado/lib/cado-nfs-3.0.0 in an experimental containerized setup I've been working on, where something like this is required since downloading copies of binaries is ill-advised in this setup.

I've also used it when doing ad-hoc clustering across machines with different CPU types.

VBCurtis 2019-04-02 23:25

[QUOTE=EdH;512288]I'm not sure if this helps, but I had a similar issue due to various hardware and have to use --binddir=build/<username>/ on my clients. My current server is 16.04 and some clients are 18.04.[/QUOTE]

I tried this invocation, got "error: no such option: --binddir" :
./cado-nfs-client.py --server=http://{servername:port} --binddir=build/{install directory}
This is on the 18.04 machine, running recent-GIT (say, two weeks ago) CADO.

Nooks 2019-04-03 04:34

The flag is called --bindir. One "d", not two.

The build/ subdirectory is found in the unpacked source tree after compiling, or in the directory you built from if you followed the instructions in the "configure using cmake directly" section in the README file. In there is usually a directory named for the machine where CADO-NFS was compiled; ".mpi" is added if CADO-NFS was configured to use MPI.

The other option for --bindir is the lib/cado-nfs-3.0.0 subdirectory under the installation root.

Both of these locations have these subdirectories:

[CODE]$ cd ~/cado-nfs/build/{hostname}.mpi; ls -d */
CMakeFiles/ filter/ linalg/ numbertheory/ scripts/ sqrt/ utils/
config/ gf2x/ misc/ polyselect/ sieve/ tests/

$ cd /mnt/pkg/cado/lib/cado-nfs-3.0.0/; ls -d */
filter/ misc/ polyselect/ sieve/ utils/
linalg/ numbertheory/ scripts/ sqrt/
[/CODE]

I have not used the build dir as my setting for --bindir but I expect it would work.

(My installation root is a bit odd because /mnt hosts a glusterfs filesystem and I install 3rd-party packages into subdirectories of /pkg to keep things separated. Theoretically I could keep the CADO source code on glusterfs but with my current setup it makes compiling prohibitively slow. These details aren't really important, though.)

EdH 2019-04-03 14:21

Darn! And, I'm always trying to be so precise...

Sorry about that misspelling.:sad:

EdH 2019-04-08 19:51

Minor Issue With Development Version
 
[code]
...
Linking CXX executable test-flint
[100%] Built target test-flint
Scanning dependencies of target check_rels
[100%] Building C object misc/CMakeFiles/check_rels.dir/check_rels.c.o
Linking CXX executable check_rels
[100%] Built target check_rels
user@machine:~/Math/cado-nfs$ ./cado-nfs.py 90377629292003121684002147101760858109247336549001090677693
Traceback (most recent call last):
File "./cado-nfs.py", line 43, in <module>
import cadotask
File "./scripts/cadofactor/cadotask.py", line 2960, in <module>
patterns.Observer):
File "./scripts/cadofactor/cadotask.py", line 3151, in SievingTask
if tuple(sys.version_info)[0] < 3:
NameError: name 'sys' is not defined
[/code]This was reproduced on several machines.

I added:
[code]
import sys
[/code]to ./scripts/cadofactor/cadotask.py and all machines work now.

Is there any anticipated trouble running the latest (developmental) version on the server and a mixture of dev and 2.3.0 on the clients?

Are there better params files or modifications to make if there will be over a dozen multi-core machines running as clients?

What info would be useful to gather in a compiled log? I'm looking at something like "egrep"ping some of the lines from the log files for all my cado-nfs runs.

VBCurtis 2019-04-08 20:14

Ed-
I have improved params files for C90 through C125 completed; I've been putting off posting them until I have concrete a/b comparisons for timings, something I've been lazy about doing.
I did finish C90 and C95 timing: On 1 thread, CADO-git from Feb factoring a C90 was 2236 seconds, while my file was 941 seconds and YAFU was 841 seconds. On 6 threads, YAFU did a c95 in 1510 seconds, CADO stock 1008 seconds, CADO with my params 625 seconds.

For a farm like yours where sieving is manycore but postprocessing is single-machine, it makes sense to add 10% to the target number of relations. This adds 10% to sieving time, but reduces matrix time by 30-60%. On a single machine this is slightly a waste of time (at least on my params files, where I've tried to find the ideal # of relations such that the extra time spent sieving balances the time saved on matrix), but you'll save quite a lot in wall-clock time.

I have draft files for C130 to C155, which I think are better than stock but not yet fully refined. Which files would you like?

EdH 2019-04-09 03:15

[QUOTE=VBCurtis;513146]Ed-
I have improved params files for C90 through C125 completed; I've been putting off posting them until I have concrete a/b comparisons for timings, something I've been lazy about doing.
I did finish C90 and C95 timing: On 1 thread, CADO-git from Feb factoring a C90 was 2236 seconds, while my file was 941 seconds and YAFU was 841 seconds. On 6 threads, YAFU did a c95 in 1510 seconds, CADO stock 1008 seconds, CADO with my params 625 seconds.

For a farm like yours where sieving is manycore but postprocessing is single-machine, it makes sense to add 10% to the target number of relations. This adds 10% to sieving time, but reduces matrix time by 30-60%. On a single machine this is slightly a waste of time (at least on my params files, where I've tried to find the ideal # of relations such that the extra time spent sieving balances the time saved on matrix), but you'll save quite a lot in wall-clock time.

I have draft files for C130 to C155, which I think are better than stock but not yet fully refined. Which files would you like?[/QUOTE]I caught the new thread on params files. I'll get them from there later.

I have adjusted my plans somewhat:

I think I'll work on refining my data mining first. Then I'll work with the current params to get a larger sample. After that, I'll try the modified files, as they are and finally, try the adjustments you suggest for my "farm."

A question arises:

Although the ideal is to compare exact composite factorizations, is there value in comparing different composites of the same size?

I ask this because my current interest is in scripts that take Aliquot sequences up to ~140 dd via ecmpi and CADO-NFS. If I aggregate the data for same sized, but different, composites, would that still be valuable?


A last, and quite different subject:

Since I have a working openmpi cluster and there is a section within the CADO-NFS documentation discussing mpi distributed LA, I am inclined to study this a bit. Is it implemented and working or just on its way to being available?

VBCurtis 2019-04-09 05:01

Your last question is, to me, the most interesting. I believe MPI is working but not documented/supported by the cado-nfs.py script; I expect you'll have to invoke the filtering/bwc steps individually. That said, I haven't peeked inside the cado-nfs.py file itself to see if there are MPI flags afoot.

I collect data by job-size too, and believe that aggregating average job times for a given length is helpful for refining parameters. I use a combination of "these parameters produced the record-quickest job of this size" and "those parameters have a lower average time per job" to decide on fastest param choices; I've also learned after 100+ jobs that some inputs just get lucky polynomials and the params have little to do with the record-low job time. I conclude that it's important to record poly score for each job, as well as first digit (C130 with first digit 6 will clearly be slower than C130 with first digit 1, more than halfway to a C131). If you also gather multiple results for a specific input size, I think that will be very helpful.

If refining parameters for C130-155 job sizes interests you, I'll be happy to share on the other thread my procedures and which specific things I've tried; my results may be more influenced by my hardware than I realize, so your data-gathering may alter some of the choices.

VBCurtis 2019-04-10 06:44

[QUOTE=EdH;513201]Since I have a working openmpi cluster and there is a section within the CADO-NFS documentation discussing mpi distributed LA, I am inclined to study this a bit. Is it implemented and working or just on its way to being available?[/QUOTE]

Have a look at local.sh.example in the CADO folder. Seems MPI is ready-to-enable.

bsquared 2019-04-10 15:14

Is there a way to run with python 2.7? I would like to test on a system on which I am unable to install new software. Alternatively, is it easy to install python3 somewhere in my home directory (where I am able to) and then have cado use that and not the system python?

[edit]
Found a guide for installing python in a home directory, involving compiling from source (ugh).
[url]https://codeghar.wordpress.com/2013/09/26/install-python-3-locally-under-home-directory-in-centos-6-4/[/url]

EdH 2019-04-11 19:18

Hey Curtis,

Thanks for all the info. I'll need to spend a bit of time with mpi after I get all the other things figured out. Thanks for pointing me to the example file

I've compiled a bunch of data, a taste of which is below. Is there enough info in the listings? I can't seem to find any version info in the logs, nor can I figure a way to display params file unique ids. Would you be able to identify modified versus unmodified params files from any particular entry within the param file that was used. Would you even need that? I suppose I could add things like lpb0/1 and mfb0/1 values, etc., but the less needed, the better.

Here's a sample of what I've compiled thus far:
[code]
N<138> = 107858998268122985412779892463164903278148452826296404480200645178455386381787979632919305998206531705192000227253731053070278282139941833
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c140.upload/c140.polyselect2.4wl6t5p_.opt_60 has Murphy_E = 3.56e-07
Generate Factor Base: Total cpu/real time for makefb: 14.5/2.95031
Generate Free Relations: Total cpu/real time for freerel: 616.08/80.0004
Lattice Sieving: Total number of relations: 32432783
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 89.32/212.046
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 429.15/154.403
Filtering - Singleton removal: Total cpu/real time for purge: 210.65/72.3947
Filtering - Merging: Total cpu/real time for merge: 768.14/667.166
Filtering - Merging: Total cpu/real time for replay: 59.26/48.687
Linear Algebra: Total cpu/real time for bwc: 35648.5/9138.31
Quadratic Characters: Total cpu/real time for characters: 70.24/20.2549
Square Root: Factors: 4549852818280784137048891955891900188206561071 23706041178905384340613948799770096216924619543119243174197081516303277264106367581589913223
Square Root: Total cpu/real time for sqrt: 3600.09/502.14
Generate Factor Base: Total cpu/real time for makefb: 14.5/2.95031
Generate Free Relations: Total cpu/real time for freerel: 616.08/80.0004
Lattice Sieving: Total number of relations: 32432783
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 89.32/212.046
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 429.15/154.403
Filtering - Singleton removal: Total cpu/real time for purge: 210.65/72.3947
Filtering - Merging: Total cpu/real time for merge: 768.14/667.166
Filtering - Merging: Total cpu/real time for replay: 59.26/48.687
Linear Algebra: Total cpu/real time for bwc: 35648.5/9138.31
Quadratic Characters: Total cpu/real time for characters: 70.24/20.2549
Square Root: Total cpu/real time for sqrt: 3600.09/502.14
Complete Factorization: Total cpu/elapsed time for entire factorization: 709733/16633.3
====================================================
N<133> = 1727651299324421928413446724594590880782759412370592923996274508812661722242606590744478625899708857621177484486201878763223620024233
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c135.upload/c135.polyselect2.sy3j0pmg.opt_54 has Murphy_E = 1.56e-07
Generate Factor Base: Total cpu/real time for makefb: 18.49/4.08387
Generate Free Relations: Total cpu/real time for freerel: 308.64/40.4427
Lattice Sieving: Total number of relations: 16820659
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 47.48/122.93
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 237.05/225.074
Filtering - Singleton removal: Total cpu/real time for purge: 125.5/52.3022
Filtering - Merging: Total cpu/real time for merge: 701.01/618.263
Filtering - Merging: Total cpu/real time for replay: 49.69/41.2704
Linear Algebra: Total cpu/real time for bwc: 25661.9/6836.89
Quadratic Characters: Total cpu/real time for characters: 49.05/16.6863
Square Root: Factors: 30694206313771024179893176877347248620777395297910299794744836039393412307041777132449849 56285908867085034103571361947375561950143217
Square Root: Total cpu/real time for sqrt: 2458.36/380.537
Generate Factor Base: Total cpu/real time for makefb: 18.49/4.08387
Generate Free Relations: Total cpu/real time for freerel: 308.64/40.4427
Lattice Sieving: Total number of relations: 16820659
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 47.48/122.93
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 237.05/225.074
Filtering - Singleton removal: Total cpu/real time for purge: 125.5/52.3022
Filtering - Merging: Total cpu/real time for merge: 701.01/618.263
Filtering - Merging: Total cpu/real time for replay: 49.69/41.2704
Linear Algebra: Total cpu/real time for bwc: 25661.9/6836.89
Quadratic Characters: Total cpu/real time for characters: 49.05/16.6863
Square Root: Total cpu/real time for sqrt: 2458.36/380.537
Complete Factorization: Total cpu/elapsed time for entire factorization: 551879/12586.8
====================================================
N<135> = 125844213559622587695829140394565697295294400320119878829054659162532607479389416797743340395107681033406552019669161119071940264791049
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c135.upload/c135.polyselect2.xnurtmbv.opt_90 has Murphy_E = 1.26e-07
Generate Factor Base: Total cpu/real time for makefb: 18.98/4.3001
Generate Free Relations: Total cpu/real time for freerel: 308.8/44.8066
Lattice Sieving: Total number of relations: 16814708
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 48.67/123.119
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 231.22/151.544
Filtering - Singleton removal: Total cpu/real time for purge: 103.84/40.3791
Filtering - Merging: Total cpu/real time for merge: 852.76/755.683
Filtering - Merging: Total cpu/real time for replay: 58.03/48.4404
Linear Algebra: Total cpu/real time for bwc: 34137.1/9071.89
Quadratic Characters: Total cpu/real time for characters: 56.94/19.1343
Square Root: Factors: 1112732091037743546494994765551139874188005873287498358284451704438039 113094800242760311335825733173241877904256425883496448249505690591
Square Root: Total cpu/real time for sqrt: 3070.26/472.152
Generate Factor Base: Total cpu/real time for makefb: 18.98/4.3001
Generate Free Relations: Total cpu/real time for freerel: 308.8/44.8066
Lattice Sieving: Total number of relations: 16814708
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 48.67/123.119
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 231.22/151.544
Filtering - Singleton removal: Total cpu/real time for purge: 103.84/40.3791
Filtering - Merging: Total cpu/real time for merge: 852.76/755.683
Filtering - Merging: Total cpu/real time for replay: 58.03/48.4404
Linear Algebra: Total cpu/real time for bwc: 34137.1/9071.89
Quadratic Characters: Total cpu/real time for characters: 56.94/19.1343
Square Root: Total cpu/real time for sqrt: 3070.26/472.152
Complete Factorization: Total cpu/elapsed time for entire factorization: 680225/16017.4
[/code]This is from a fresh install of the dev version from a few days ago.

EdH 2019-04-11 21:54

Some Disconcerting ETAs, but then "All Is Well..."
 
[code]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 17:20:51 2019 [0.143 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 18:10:45 2019 [0.285 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 19:00:35 2019 [0.428 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 19:50:28 2019 [0.570 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 20:40:24 2019 [0.713 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 21:30:19 2019 [0.855 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 22:20:10 2019 [0.998 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 23:10:04 2019 [1.140 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Thu Apr 11 23:59:56 2019 [1.283 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 00:49:46 2019 [1.425 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 01:39:38 2019 [1.568 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 02:29:32 2019 [1.710 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 03:19:24 2019 [1.853 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 04:09:15 2019 [1.995 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 04:59:07 2019 [2.138 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 05:48:58 2019 [2.280 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 06:38:50 2019 [2.423 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 07:28:45 2019 [2.565 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 08:18:38 2019 [2.708 s/iter]
Info:Linear Algebra: mksol: N=1000 ; ETA (N=21000): Fri Apr 12 09:08:29 2019 [2.850 s/iter]
Info:Linear Algebra: mksol: N=860 ; ETA (N=21000): Fri Apr 12 12:40:41 2019 [3.456 s/iter]
Info:Quadratic Characters: Starting
Info:Square Root: Starting
Info:Square Root: Creating file of (a,b) values
Info:Square Root: finished
Info:Square Root: Factors: 76011547790026822726326568236942510573210375997181897566667718127 635610967526542663809596695842346695879983625491514026964044776163
Info:Polynomial Selection (size optimized): Aggregate statistics:
Info:Polynomial Selection (size optimized): potential collisions: 38210.8
Info:Polynomial Selection (size optimized): raw lognorm (nr/min/av/max/std): 38840/38.730/46.683/50.720/0.852
Info:Polynomial Selection (size optimized): optimized lognorm (nr/min/av/max/std): 38840/37.740/42.075/47.510/1.204
Info:Polynomial Selection (size optimized): Total time: 7618.67
Info:Polynomial Selection (root optimized): Aggregate statistics:
Info:Polynomial Selection (root optimized): Total time: 2994.82
Info:Polynomial Selection (root optimized): Rootsieve time: 2993.53
Info:Generate Factor Base: Total cpu/real time for makefb: 50.58/11.3882
Info:Generate Free Relations: Total cpu/real time for freerel: 311.62/45.3888
Info:Lattice Sieving: Aggregate statistics:
Info:Lattice Sieving: Total number of relations: 27681208
Info:Lattice Sieving: Average J: 7707.97 for 87893 special-q, max bucket fill -bkmult 1.0,1s:1.113920
Info:Lattice Sieving: Total time: 181352s
Info:Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 73.17/198.683
Info:Filtering - Duplicate Removal, splitting pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, splitting pass: CPU time for dup1: 198.3s
Info:Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 424.32/391.023
Info:Filtering - Duplicate Removal, removal pass: Aggregate statistics:
Info:Filtering - Duplicate Removal, removal pass: CPU time for dup2: 367.7s
Info:Filtering - Singleton removal: Total cpu/real time for purge: 368.81/174.105
Info:Filtering - Merging: Total cpu/real time for merge: 879.66/789.756
Info:Filtering - Merging: Total cpu/real time for replay: 67.5/62.5504
Info:Linear Algebra: Total cpu/real time for bwc: 32922.6/8766.06
Info:Linear Algebra: Aggregate statistics:
Info:Linear Algebra: Krylov: WCT time 5463.93, iteration CPU time 0.12, COMM 0.01, cpu-wait 0.0, comm-wait 0.0 (42000 iterations)
Info:Linear Algebra: Lingen CPU time 360.93, WCT time 104.03
Info:Linear Algebra: Mksol: WCT time 2974.44, iteration CPU time 0.13, COMM 0.01, cpu-wait 0.0, comm-wait 0.0 (21000 iterations)
Info:Quadratic Characters: Total cpu/real time for characters: 59.92/19.9351
Info:Square Root: Total cpu/real time for sqrt: 3066.22/474.952
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 418847/14039.8
76011547790026822726326568236942510573210375997181897566667718127 635610967526542663809596695842346695879983625491514026964044776163
nfsDone 100% 276 0.3KB/s 00:00
user@computer:~$ date
Thu Apr 11 17:28:48 EDT 2019
[/code]

VBCurtis 2019-04-11 23:34

All of my params files include a target number of relations, so I would be able to tell when your runs use one of my files because the number of relations found would be just over my target number set.
Also, my files use a substantially larger LP bound than default, so the number of relations is quite a bit larger than default settings.
The only item I'd want that wasn't in your paste of the data dump is the size of the input number.
For my own data-gathering, I record Q-range sieved (which isn't in the summary, but also isn't that important), poly select time (size and root, but total is enough), sieve time, bwc time, wall-clock time. Recently I've been also recording the total weight of the matrix (the number printed just before linear algebra starts), so that I could see how various density selections influence matrix size; again, not too important but if we start collaborating on C150+ settings it would be nice to have.

EdH 2019-04-12 03:12

I encoded the composite size in "<XXX>" right after the N in the first line:
[code]
N<[B]138[/B]> = 107858998268122985412779892463164903278148452826296404480200645178455386381787979632919305998206531705192000227253731053070278282139941833

[/code]I put it there instead of at the end, so it would be more accessible.

I'll work on adding your list items. Should I trim the rest, or leave everything.

Actually, I'm wondering if you were referring to the correct post. I had two posts earlier. The first was the samples for a C138, C133 and C135.

The second was to point out a discrepancy in the LA mksol ETAs.

Do any of the interested parties watch this thread for this type of info, or should I try to post to the CADO-NFS project?

VBCurtis 2019-04-12 03:54

I see the composite size now, that's an excellent way to encode it.
I don't know what is valuable for other people; the timings in the summary are all that's really needed to evaluate the params for a composite.

EdH 2019-04-12 19:10

I realize these have duplication, but I haven't figured out how to automatically remove duplicate lines, yet.

Anyway, Is all the info you need contained in the following three samples?

C86, C129, C121:

[code]
N<86> = 56667759878435851136396148508097449099773621318662989162599675933873393187564668667973
Lattice Sieving: params = {'maxresubmit': 5, 'lim0': 393010, 'workdir': '/tmp/cadofactor', 'maxwuerror': 2, 'name': 'c85', 'rels_wanted': 0, 'wutimeout': 10800, 'lim1': 551399, 'qmin': 146453, 'sqside': 1, 'run': True, 'gzip': True, 'maxtimedout': 100, 'qrange': 1000, 'maxwu': 10, 'maxfailed': 100}
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c85.upload/c85.polyselect2.dq2cf04y.opt_12 has Murphy_E = 2.95e-06
Generate Factor Base: Total cpu/real time for makefb: 0.47/0.135556
Generate Free Relations: Total cpu/real time for freerel: 3.85/0.515574
Lattice Sieving: sieving.run(): Task state: {'wu_submitted': 0, 'wu_timedout': 0, 'start_achievement': -1, 'qnext': 146453, 'start_real_time': 0, 'rels_wanted': 440080, 'wu_received': 0, 'wu_failed': 0, 'rels_found': 0}
Lattice Sieving: Total number of relations: 470449
Lattice Sieving: Total time: 518.71s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 1.06/0.725687
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 3.44/1.06687
Filtering - Singleton removal: Total cpu/real time for purge: 0.6/0.373629
Lattice Sieving: sieving.run(): Task state: {'wu_submitted': 219, 'rels_wanted': 498934, 'start_achievement': 0.004069714597345937, 'rels_found': 470449, 'qnext': 365000, 'wu_failed': 0, 'stats_avg_J': '1002.231756149383 12237', 'wu_timedout': 0, 'stats_total_cpu_time': '912.3099999999998', 'start_real_time': 3763830410.102031, 'wu_received': 153, 'stats_total_time': '518.71', 'stats_max_bucket_fill': '1.0,1s:1.594150'}
Lattice Sieving: sieving.run(): Task state: {'wu_submitted': 285, 'rels_wanted': 522895, 'start_achievement': 0.004069714597345937, 'rels_found': 504252, 'qnext': 431000, 'wu_failed': 0, 'stats_avg_J': '1001.8836890243903 13120', 'wu_timedout': 0, 'stats_total_cpu_time': '972.5799999999998', 'start_real_time': 3763830410.102031, 'wu_received': 164, 'stats_total_time': '551.3100000000002', 'stats_max_bucket_fill': '1.0,1s:1.594150'}
Filtering - Merging: Merged matrix has 41245 rows and total weight 7011876 (170.0 entries per row on average)
Square Root: Factors: 358031419880876938192954600148936974177 158275940969901931610142052723104530012975662949
Generate Factor Base: Total cpu/real time for makefb: 0.47/0.135556
Generate Free Relations: Total cpu/real time for freerel: 3.85/0.515574
Lattice Sieving: Total number of relations: 651829
Lattice Sieving: Total time: 722.52s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 1.46/1.11501
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 5.82/2.44343
Filtering - Singleton removal: Total cpu/real time for purge: 5.91/2.61454
Filtering - Merging: Total cpu/real time for merge: 13.92/9.40176
Filtering - Merging: Total cpu/real time for replay: 1.34/1.11267
Linear Algebra: Total cpu/real time for bwc: 27.41/8.48
Quadratic Characters: Total cpu/real time for characters: 1.43/0.41662
Square Root: Total cpu/real time for sqrt: 31.34/4.62864
Complete Factorization: Total cpu/elapsed time for entire factorization: 3518.26/98.8682
====================================================
N<129> = 148184180288647082302426937158351493009802210451497766956818199852902417872086615646923058242179971571945498904372691387599121333
Lattice Sieving: params = {'lim1': 44217255, 'qmin': 711270, 'maxresubmit': 5, 'maxwu': 10, 'rels_wanted': 0, 'run': True, 'maxwuerror': 2, 'qrange': 10000, 'wutimeout': 10800, 'sqside': 1, 'name': 'c130', 'workdir': '/tmp/cadofactor', 'gzip': True, 'lim0': 13124945, 'maxtimedout': 100, 'maxfailed': 100}
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c130.upload/c130.polyselect2.qxg0sxtu.opt_48 has Murphy_E = 5.93e-07
Generate Factor Base: Total cpu/real time for makefb: 50/10.6005
Generate Free Relations: Total cpu/real time for freerel: 311.55/40.6101
Lattice Sieving: sieving.run(): Task state: {'qnext': 711270, 'rels_found': 0, 'start_achievement': -1, 'wu_submitted': 0, 'wu_received': 0, 'wu_timedout': 0, 'rels_wanted': 22129742, 'start_real_time': 0, 'wu_failed': 0}
Lattice Sieving: Total number of relations: 22630418
Lattice Sieving: Total time: 101935s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 59.73/158.776
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 312.5/289.52
Filtering - Singleton removal: Total cpu/real time for purge: 64.6/62.8826
Lattice Sieving: sieving.run(): Task state: {'stats_avg_J': '7660.828321977037 47991', 'qnext': 2220000, 'wu_submitted': 151, 'stats_total_time': '101935.03999999998', 'start_real_time': 3763831072.010403, 'rels_wanted': 22856722, 'start_achievement': 0.013695957232578672, 'rels_found': 22630418, 'stats_max_bucket_fill': '1.0,1s:1.120020', 'wu_timedout': 0, 'stats_total_cpu_time': '195411.35000000003', 'wu_received': 67, 'wu_failed': 0}
Filtering - Merging: Merged matrix has 1228821 rows and total weight 208900009 (170.0 entries per row on average)
Square Root: Factors: 12674601840486459448841848682136209646192693900874822145430131644978522484248447 11691426851398408680908975753372972164838725377739
Generate Factor Base: Total cpu/real time for makefb: 50/10.6005
Generate Free Relations: Total cpu/real time for freerel: 311.55/40.6101
Lattice Sieving: Total number of relations: 25564242
Lattice Sieving: Total time: 120205s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 67.39/179.278
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 396.05/371.704
Filtering - Singleton removal: Total cpu/real time for purge: 337.35/309.169
Filtering - Merging: Total cpu/real time for merge: 750.5/700.049
Filtering - Merging: Total cpu/real time for replay: 47.22/38.9662
Linear Algebra: Total cpu/real time for bwc: 24120.6/6204.46
Quadratic Characters: Total cpu/real time for characters: 56.22/15.795
Square Root: Total cpu/real time for sqrt: 3060.65/418.769
Complete Factorization: Total cpu/elapsed time for entire factorization: 304850/10683
====================================================
N<121> = 8752900354798992135535713351999727922615660863269877993396052619685931654568649609816781542442396746201949471440217022437
Lattice Sieving: params = {'lim0': 3000000, 'gzip': True, 'maxwuerror': 2, 'maxtimedout': 100, 'run': True, 'qmin': 1000000, 'qrange': 10000, 'rels_wanted': 0, 'maxresubmit': 5, 'wutimeout': 10800, 'workdir': '/tmp/cadofactor', 'sqside': 1, 'maxfailed': 100, 'maxwu': 10, 'lim1': 5500000, 'name': 'c120'}
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c120.upload/c120.polyselect2.v5ylhooc.opt_90 has Murphy_E = 1.32e-06
Generate Factor Base: Total cpu/real time for makefb: 6.22/1.45258
Generate Free Relations: Total cpu/real time for freerel: 155.53/20.4213
Lattice Sieving: sieving.run(): Task state: {'start_real_time': 0, 'start_achievement': -1, 'wu_submitted': 0, 'rels_found': 0, 'wu_timedout': 0, 'wu_failed': 0, 'wu_received': 0, 'rels_wanted': 11474681, 'qnext': 1000000}
Lattice Sieving: Total number of relations: 11499970
Lattice Sieving: Total time: 42836.9s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 29.66/71.9494
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 125.83/53.849
Filtering - Singleton removal: Total cpu/real time for purge: 21.02/9.09563
Lattice Sieving: sieving.run(): Task state: {'start_achievement': 0.003286453017735308, 'wu_submitted': 453, 'stats_max_bucket_fill': '1.0,1s:1.360290', 'wu_received': 370, 'rels_wanted': 11964241, 'stats_total_time': '42836.850000000006', 'start_real_time': 3763877574.831329, 'stats_avg_J': '1913.329901275031 250190', 'qnext': 5530000, 'stats_total_cpu_time': '79603.62000000001', 'rels_found': 11499970, 'wu_timedout': 0, 'wu_failed': 0}
Lattice Sieving: sieving.run(): Task state: {'start_achievement': 0.003286453017735308, 'wu_submitted': 464, 'stats_max_bucket_fill': '1.0,1s:1.360290', 'wu_received': 429, 'rels_wanted': 13184371, 'stats_total_time': '49254.79', 'start_real_time': 3763877574.831329, 'stats_avg_J': '1911.9272845857893 288247', 'qnext': 5640000, 'stats_total_cpu_time': '91392.13000000002', 'rels_found': 12923469, 'wu_timedout': 0, 'wu_failed': 0}
Lattice Sieving: sieving.run(): Task state: {'start_achievement': 0.003286453017735308, 'wu_submitted': 475, 'stats_max_bucket_fill': '1.0,1s:1.360290', 'wu_received': 446, 'rels_wanted': 13482396, 'stats_total_time': '51120.21000000001', 'start_real_time': 3763877574.831329, 'stats_avg_J': '1911.4947376864536 299203', 'qnext': 5750000, 'stats_total_cpu_time': '94879.38000000002', 'rels_found': 13325503, 'wu_timedout': 0, 'wu_failed': 0}
Lattice Sieving: sieving.run(): Task state: {'start_achievement': 0.003286453017735308, 'wu_submitted': 536, 'stats_max_bucket_fill': '1.0,1s:1.360290', 'wu_received': 453, 'rels_wanted': 13611673, 'stats_total_time': '51708.15000000001', 'start_real_time': 3763877574.831329, 'stats_avg_J': '1911.302068579334 303590', 'qnext': 6360000, 'stats_total_cpu_time': '95994.38000000002', 'rels_found': 13484536, 'wu_timedout': 0, 'wu_failed': 0}
Filtering - Merging: Merged matrix has 742731 rows and total weight 74273389 (100.0 entries per row on average)
Square Root: Factors: 63177231098564251400862698002146040730555401588791894828480702706907 138545172091245831321071555847351711191164740172937791
Generate Factor Base: Total cpu/real time for makefb: 6.22/1.45258
Generate Free Relations: Total cpu/real time for freerel: 155.53/20.4213
Lattice Sieving: Total number of relations: 13906880
Lattice Sieving: Total time: 53405.4s
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 35.98/78.1032
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 226.18/131.068
Filtering - Singleton removal: Total cpu/real time for purge: 160.12/80.2486
Filtering - Merging: Total cpu/real time for merge: 169.38/142.37
Filtering - Merging: Total cpu/real time for replay: 19.23/14.9333
Linear Algebra: Total cpu/real time for bwc: 5288.67/1377.31
Quadratic Characters: Total cpu/real time for characters: 28.14/7.54837
Square Root: Total cpu/real time for sqrt: 1244.19/172.8
Complete Factorization: Total cpu/elapsed time for entire factorization: 141585/3000.22
[/code]

VBCurtis 2019-04-12 22:20

Sure, looks like enough to make some observations.

On your C129, sieve time was 120ksec, while bwc was 24ksec; in wall clock time, bwc was 6200 of your 10600 sec job.

On your C121, sieve time 53ksec, bwc 5300 sec; in wall clock time, bwc was 1300 of your 3000 sec job.

So, the C121 was over-sieved enough to make bwc time 1/10th of sieve time, resulting in a substantial savings in wall clock time.

The files I posted target bwc time to be 1/6th to 1/5th of sieving time; I think your C121 shows the way for your setup at bwc time = 1/10th of sieving time. Factoring a C120 in under an hour is pretty snazzy.

EdH 2019-04-13 03:18

1 Attachment(s)
[QUOTE=VBCurtis;513560]Sure, looks like enough to make some observations.

On your C129, sieve time was 120ksec, while bwc was 24ksec; in wall clock time, bwc was 6200 of your 10600 sec job.

On your C121, sieve time 53ksec, bwc 5300 sec; in wall clock time, bwc was 1300 of your 3000 sec job.

So, the C121 was over-sieved enough to make bwc time 1/10th of sieve time, resulting in a substantial savings in wall clock time.

The files I posted target bwc time to be 1/6th to 1/5th of sieving time; I think your C121 shows the way for your setup at bwc time = 1/10th of sieving time. Factoring a C120 in under an hour is pretty snazzy.[/QUOTE]I also like the C138 in just over 4.5 hours, although I think my CADO/msieve hybrid would do better, which is what's driving my interest in your params and the mpi bwc.

Anyway, here is a file of data for 87 runs ranging from 74 through 138 dd. It does include all previous samples. These are with unmodified params files from the original developmental version from a few days ago:

VBCurtis 2019-04-13 03:57

I'm running a "last" test of new C120 and C130 files; if results fit the time-vs-size curve I expect, I'll have them posted in the morning for you to the params thread. C125 is still a little slower than the curve suggests should be possible, I still have some tweaking to try there to get a bit more speed.

EdH 2019-04-13 22:58

The Last Four Unmodified Runs
 
1 Attachment(s)
I have installed all your modified params files from the other thread. These are the last four unmodified that ran. Of course, I realize that not all of the params files in my current set are modified, but I'm considering the next sets as Modified:

EdH 2019-04-15 03:41

Of All the STUPID Things I've Done!!
 
I accidentally wiped out all my scripts that ran my Aliquot Sequences on my controlling machine - the ones that communicated with the db and ran the ecm and nfs scripts!

The nfs script itself was untouched and I've found an earlier backup for the main controlling script, which is quite fortunate, since I remember having to solve several issues and I have no recollection of how I did solve them. I would have had to research and relearn a bunch. Still, reconstructing all the other scripts may be time consuming, so it might be a while before I get any more data compiled and uploaded.

[SIZE=1]BTW, in case you're interested, I used a recovery program to try to bring them back, but was only able to get the filenames, which unfortunately overwrote the original "deleted" files with zero-byte replacements.[/SIZE]

EdH 2019-04-18 03:18

Back Factoring...
 
1 Attachment(s)
I have the scripts pretty much rewritten and they are "mostly" working correctly.

Here is a file of Info for 91 composites factored using VBCurtis' params files, where available. (There may be duplicate runs for some of the numbers.)

ET_ 2019-04-18 08:38

[QUOTE=EdH;514016]I have the scripts pretty much rewritten and they are "mostly" working correctly.

Here is a file of Info for 91 composites factored using VBCurtis' params files, where available. (There may be duplicate runs for some of the numbers.)[/QUOTE]

I downloaded the CADO package last week, per your instructions.

What should I do now with this file?

EdH 2019-04-18 14:25

[QUOTE=ET_;514028]I downloaded the CADO package last week, per your instructions.

What should I do now with this file?[/QUOTE]
I was a bit confusing in my post. The scripts I referred to were the ones I'd lost earlier that I use for my Aliquot factoring via ecmpi and CADO-NFS across several machines. The file I posted was data from the CADO runs I did using VBCurtis' modified params files from the "[URL="https://www.mersenneforum.org/showthread.php?t=24274"]improved params...[/URL]" thread. They are mostly for VBCurtis to review, but anyone else can look them over for timing and poly/sieve info that can be used to adjust the params files for local use.

Sorry for the confusion.

VBCurtis 2019-04-18 15:01

[QUOTE=ET_;514028]I downloaded the CADO package last week, per your instructions.

What should I do now with this file?[/QUOTE]

If you used git to download the package, just cd into the cado-nfs folder and "make". It will ask if you allow downloading cmake, accept.
An hour or so later you'll have a working copy of CADO.

To test it, while in the cado-nfs directory invoke:
./cado-nfs.py {input number}

With stock parameters, CADO is a bit slower than YAFU; with my improved params files, CADO is somewhat faster than YAFU. I await tests like Ed's to measure just how much faster across a variety of hardware, but so far I estimate it's 20 to 30% faster than YAFU. For example, I used 3 Haswell 3.3ghz cores to factor a C139 in 38 hours with CADO.

To use my params files, grab them from the "improved parameters" thread, and save to the folder /cado-nfs/parameters/factor. You can rename the original ones for safekeeping, or overwrite them.

If you do so, please post a before & after time on the same composite or on two composites very close in size (say, a factor of 2 apart). It seems CADO performs quite differently by hardware type.

ET_ 2019-04-18 17:07

[QUOTE=VBCurtis;514046]
To use my params files, grab them from the "improved parameters" thread, and save to the folder /cado-nfs/parameters/factor. You can rename the original ones for safekeeping, or overwrite them.

If you do so, please post a before & after time on the same composite or on two composites very close in size (say, a factor of 2 apart). It seems CADO performs quite differently by hardware type.[/QUOTE]

I will do both tests to improve my knowledge of the system.
Just need a hint on the size of the composites (you said C139?) and where to choose them. I will gladly perform a test on numbers you want to run before and after the file substitution.

VBCurtis 2019-04-18 19:17

Any composites you have an interest in will do; some folks have factored the same number twice, or even three times (Yafu, Cado stock, cado my files) to compare speeds.

If a direct comparison with the same input is your style, I suggest finding a composite between 100 and 120 digits; if you have none from your own interests, PM me and I can send an entry from an aliquot sequence. If you'd rather do productive work, find any two inputs of similar size (say, within a factor of two) for an a/b test, where "a" can be YAFU or stock CADO and "b" is improved CADO.

I am currently developing files for C125, C135, C140; C95 to C120 (and C130) are presently posted. CADO rounds inputs to the nearest multiple-of-5-digits for parameter choice; we can write custom params for an individual number, but until 165+ digits that seems like unnecessary complication.

RichD is currently testing my beta C140 file against stock-CADO on a pair of C138s, a size that takes between 1 and 1.5 days on a quad-core desktop; he already has a strong idea of what speed his rig manages with YAFU.

A C120 takes a couple of hours, while C105 takes half an hour or so. As you can imagine, I've done much more testing on C110 and lower because the tests are so fast; I'm mostly using a best-fit curve of time vs difficulty to evaluate whether C125+ files are "as fast" as the files for smaller numbers.

fivemack 2019-04-22 20:42

CADO for the sieving step
 
So, it's nice and easy to download and install (I was a little surprised that it carefully downloaded and installed its own copy of cmake), and not too difficult to give a polynomial.

The automatic script fell over very quickly after trying to run ten separate two-threaded sievers at 13GB memory usage each on my 96GB machine (though spent ten minutes producing a free-relations file before starting to sieve); it left behind a file project.wucmd which contained plausible-looking raw 'las' command lines.

I am a little disconcerted that I can't find evidence of the out-of-memory-failure in /var/log or in any of the log files (I'm deducing the memory usage by running the line from project.cmd); the console output ends

[code]
Info:Lattice Sieving: Adding workunit L2253_sieving_230045000-230050000 to database
[/code]

and some process is clearly still running on that console even though no 'las' processes are running on the system.

Running las with '-t 20' gives a process which uses 21GB and does seem to be running on twenty CPUs at least some of the time, and generating about three relations a second.

VBCurtis 2019-04-23 03:38

My plan for the C207 team sieve is to set tasks.sieve.las.threads = 4, which will cause every client las (siever) process to run 4-threaded. This causes an occasional small error noting some bucket is full, but it allocates a bit more memory for said bucket (I believe the setting is bkmult) and continues with a suggestion that maybe I'm using too many threads for my choice of lim's. It's consistent, then, that your 20-threaded single process would trigger that error more frequently and require more RAM than my 4-threaded process @12-13GB does.
When running the deg 6 poly 4-threaded on a 20-core machine with 30+ other threads busy, I was also finding about 3 relations per second; top showed CPU use near 400%, so your timing confuses me a little.
I've been trying the obvious flags on the command line for cado-nfs-client.py to set the number of threads on the client side, with no luck.

fivemack 2019-04-23 06:40

The log at the end of the 20-threaded job was
[code]
$ taskset -c 10-19,30-39 /scratch/cado-gnfs/cado-nfs-2.3.0/build/oak/sieve/las -I 16 -poly L2253.jon/L2253.poly -q0 231000000 -q1 231001000 -lim0 268000000 -lim1 268000000 -lpb0 33 -lpb1 33 -mfb0 99 -mfb1 99 -lambda0 3.1 -lambda1 3.2 -fb L2253.jon/L2253.roots.gz -out L2253.jon/231M-231M001 -t 20 -sqside 1 -stats-stderr

# Average J=32760 for 68 special-q's, max bucket fill 0.796379
# Discarded 0 special-q's out of 68 pushed
# Total cpu time 28131.83s [norm 7.14+23.2, sieving 8781.8 (7022.8 + 448.1 + 1310.9), factor 19319.7 (1995.1 + 17324.6)]
# Total elapsed time 2051.09s, per special-q 30.163s, per relation 0.238249s
# PeakMemusage (MB) = 25783
# Total 8609 reports [3.27s/r, 126.6r/sq]
[/code]

This is a *lot* slower than gnfs-lasieve4I16e, but looking at that command line I am using 3LP on the rational side which is usually very slow; rerunning with -mfb0 66 before contemplating changing the factor-large-part parameters.

henryzz 2019-04-23 08:58

Why is number of threads per client not a client side setting?

fivemack 2019-04-23 19:21

[CODE]
I=16, 3lpa 2lpr, -t20
# Average J=32760 for 33 special-q's, max bucket fill 0.507688
# Discarded 0 special-q's out of 33 pushed
# Total cpu time 5855.10s [norm 2.46+10.1, sieving 4878.9 (3963.3 + 224.9 + 690.7), factor 963.7 (532.0 + 431.7)
# Total elapsed time 363.64s, per special-q 11.0193s, per relation 0.0980416s
# PeakMemusage (MB) = 25783
# Total 3709 reports [1.58s/r, 112.4r/sq]

I=15, 3lpa 2lpr, -t20

# Average J=16360 for 47 special-q's, max bucket fill 0.409821
# Discarded 0 special-q's out of 47 pushed
# Total cpu time 1473.23s [norm 0.99+4.8, sieving 1010.2 (705.3 + 76.7 + 228.3), factor 457.2 (197.0 + 260.2)
# Total elapsed time 80.44s, per special-q 1.7114s, per relation 0.0352634s
# PeakMemusage (MB) = 8456
# Total 2281 reports [0.646s/r, 48.5r/sq]

[/CODE]

I should do a -t40 run since my comparison is 40 parallel gnfs-lasieve4I15e jobs

VBCurtis 2019-04-23 19:29

And it looks like I should do an I=15 run both 2- and 4- threaded to see if we benefit from I=16 at all! CADO is just fine with very large Q values, so it's not important if yield is worse than half on I=15 vs I=16; that would suggest Q-range of perhaps 20M to 1100M rather than 20M to 450M, and memory use under 4GB per process. It also may turn out that optimal lim choices for I=15 are a bit higher than I=16; we are constrained by memory footprint on I=16 to have lim's about where I tested (Rlim 260M, Alim 550M), but we have no such constraints on I=15.

Did you run makefb before las to generate the factor-base file? I haven't yet had success finding relations using las free-standing, but that would clearly be preferable.

fivemack 2019-04-23 22:31

I generated the factor-base file by running the cado-nfs script and letting it call makefb for me, then transcribed the command lines in L2253.wucmd to get raw las command lines.

[code]
/scratch/cado-gnfs/cado-nfs-2.3.0/build/oak/sieve/makefb -poly L2253.jon/L2253.poly -lim 268000000 -maxbits 16 -out L2253.jon/L2253.roots.gz -t 40
[/code]

seems to be the command line for makefb

L2253.poly is just as for gnfs-lasieve4I15e except that it's just the polynomial and no sieving parameters:

[code]
n: 2688333615331433020642446747149440986283678638176205541641754312932820814295074220965678187428410875031545875881257723735692836520162515677425285432734833508071695321927492427322546769031971029
skew: 368024608.71
c0: 123272612786479316312350884349842837614862419023
c1: -8888039873820606651882838453601725169289
c2: -6398405309760975776966814915379
c3: -74102914865935467793635
c4: -344134329264960
c5: 556920
Y0: -21713911810858617860786743761277388982
Y1: 11185023447043546081
[/code]

fivemack 2019-04-23 22:52

I would be a bit wary about running with I=15 and trusting in really large Q; yield drops off as Q goes up, and you do start to get perceptible problems with duplicates (try running a C135 with gnfs-lasieve4I12e to see both these effects)

VBCurtis 2019-04-23 23:35

I did not notice the L2253 in your initial post; I thought you were testing the C207 from the Cunningham project. Your results with I=15 vs I=16 make more sense now, and while I'll try I=15 for the sake of thoroughness (and because the CADO default C210 file uses I=15) I won't expect greatness.

Thanks for breaking down the makefb startup & how you got set up for testing.

henryzz 2019-04-24 08:07

If I recall correctly Bob has suggested running smaller special q with a larger sieve range in the past. Maybe very small q(0-1m?) could be run with 17e. I believe duplicates won't be such an issue due to the increased sieve range(unless the sieve range is related to q size?) and yield will be incredible at tiny qs.

fivemack 2019-04-24 10:27

I=15 larger-scale run
 
[code]
$ /scratch/cado-gnfs/cado-nfs-2.3.0/build/oak/sieve/las -I 15 \
-poly L2253.jon/L2253.poly \
-q0 232000000 -q1 233000000 -lim0 268000000 -lim1 268000000 \
-lpb0 33 -lpb1 33 -mfb0 66 -mfb1 99 -lambda0 2.2 -lambda1 3.2 \
-fb L2253.jon/L2253.roots.gz -out L2253.jon/232M \
-t 40 -sqside 1 -stats-stderr

# Average J=16320 for 52112 special-q's, max bucket fill 0.360714
# Discarded 0 special-q's out of 52112 pushed
# Total cpu time 1718332.50s [norm 1106.12+5086.3, sieving 1158819.1 (821794.0 + 100381.7 + 236643.5), factor 553321.0 (283784.5 + 269536.5)]
# Total elapsed time 49211.82s, per special-q 0.944347s, per relation 0.0194419s
# PeakMemusage (MB) = 13467
# Total 2531226 reports [0.679s/r, 48.6r/sq]
[/code]

That's slightly lower yield (but only by a couple of percent) than I would expect from gnfs-lasieve4I15e; on the other hand it used only about a third as much memory (13467M vs forty jobs at 911M apiece=36440M). I can't say much about timings because I haven't run on that machine at that range; will do 233-234 to get a realistic comparison. It's 20% slower than forty-copies-of-15e was on average over Q=120-126.

But first, trying -ncurves0={20,25,30} and -ncurves1={20,25,30} on 16e for a fixed Q range (probably about 40k, which should get reasonable speed and reasonable statistical significance)

fivemack 2019-04-25 09:29

1 Attachment(s)
That's interesting; the timing on the ncurves runs moves around like a thermometer rather than like anything to do with mathematics. Doing large multi-threaded benchmarking runs on contemporary hardware with all its turbo boosts and the like appears quite a difficult problem.

I tried to run a 17e test; you have to rebuild everything with -DLOG_BUCKET_REGION=17 and this then doubles the size of a number of arrays; a one-threaded job tries to allocate 39GB and falls over on my busy 64G machine:

[code]
malloc_aligned(0x200000,2097152) called
malloc_aligned(0x9d1800000,2097152) called
code BUG() : condition rc == 0 failed in malloc_aligned
[/code]

On a busyish 96G machine it falls over when trying to allocate a second 39GB array; after allocating more swap and kill -STOPping everything else on the machine I actually get some relations

[code]
# 291 relation(s) for side-1 (232000009,20697965)
# Time for this special-q: 441.4965s [norm 0.1389+0.6362, sieving 392.7559 (323.9631 + 29.1809 + 39.6120), factor 47.9654 (29.1407 + 18.8247)]

compare 15e
# 52 relation(s) for side-1 (232000009,20697965)
# Time for this special-q: 33.3656s [norm 0.0152+0.1131, sieving 22.2214 (15.6162 + 1.8664 + 4.7388), factor 11.0159 (5.5952 + 5.4208)]

and 16e (unfortunately for a different special-Q, but time/relations is the relevant metric)
# 127 relation(s) for side-1 (231960013,7709469)
# Time for this special-q: 144.9776s [norm 0.0846+0.3650, sieving 113.0854 (82.6639 + 9.1164 + 21.3050), factor 31.4426 (18.1851 + 13.2576)]

So for this number, which is a GNFS-193, we're seeing
15e 0.64s/r
16e 1.14s/r
17e 1.52s/r
[/code]

I suppose the 2019 way to look at this would have been to hire an r5.12xlarge for a couple of hours, which actually would only have cost a dollar.

henryzz 2019-04-25 10:16

[QUOTE=fivemack;514643]That's interesting; the timing on the ncurves runs moves around like a thermometer rather than like anything to do with mathematics.[/QUOTE]

What are the axes?

Y-axis is seconds-per-specialQ and X-axis is just the sequence of specialQ; I'm running the same set of specialQ nine times with different ncurves0/ncurves1 parameters, so I was expecting nine fuzzy lines at different heights.

fivemack 2019-04-25 11:26

Sorry henryzz, I clicked 'edit' rather than 'quote' and abused my supermoderatorial powers.

There is something in the data under all this noise: changing ncurves1 substantially changes the time for that factorisation phase whilst changing the yield very little, so I'm going to try ncurves1=5,10,15 next

[code]
30.20.2:# Total cpu time 284849.56s [norm 197.30+649.6, sieving 223552.3 (161447.4 + 17982.5 + 44122.5), factor 60450.3 (38705.0 + 21745.3)]
30.20.2:# Total 236498 reports [1.2s/r, 112.7r/sq]
30.25.2:# Total cpu time 289695.86s [norm 198.30+650.6, sieving 222878.8 (161022.2 + 17985.1 + 43871.5), factor 65968.2 (38722.5 + 27245.7)]
30.25.2:# Total 236786 reports [1.22s/r, 112.8r/sq]
30.30.2:# Total cpu time 288201.77s [norm 195.66+647.7, sieving 216766.5 (155109.5 + 17964.5 + 43692.4), factor 70591.9 (38023.6 + 32568.3)]
30.30.2:# Total 236814 reports [1.22s/r, 112.8r/sq]
[/code]

The sieving time is the one with masses of multithreaded memory access so I can see an argument that it is going to be noisier than the other lines; indeed, the first component of sieving time contains all the wiggles in the noisy graph I posted, the rest are much closer to flat within a block.

fivemack 2019-04-25 11:47

Revised 17e numbers
 
The sieving part of the first 17e special-Q was much slower than the others, so my numbers in post 369 are unrealistic.

More plausible numbers (comparison is from single-threaded jobs at I=15,16,17 range=232000000..232000010) (note that these are with a binary built with 17-bit bucket support so quite possibly less efficient than the 16-bit-bucket default):

[code]
grep -E "(this special-q|relation)" ../cado-nfs-2.3.0-B17/L2253.j/1?e.x

15e.x:# 50 relation(s) for side-1 (232000009,175376172)
15e.x:# Time for this special-q: 17.4141s [norm 0.0080+0.0920, sieving 12.5049 (9.0474 + 0.8440 + 2.6135), factor 4.8092 (1.7133 + 3.0959)]

16e.x:# 113 relation(s) for side-1 (232000009,175376172)
16e.x:# Time for this special-q: 49.5885s [norm 0.0600+0.1760, sieving 34.8262 (21.4750 + 2.9080 + 10.4432), factor 14.5263 (7.1553 + 7.3710)]

17e.x:# 274 relation(s) for side-1 (232000009,175376172)
17e.x:# Time for this special-q: 266.6024s [norm 0.1643+0.6817, sieving 216.8890 (157.5922 + 14.4013 + 44.8955), factor 48.8674 (30.8008 + 18.0666)]

[/code]

15 0.348s/r
16 0.439s/r
17 0.973s/r

So for numbers this small 17e really doesn't make sense, which we knew already.

A more interesting question is what kind of yield 17e might give on SNFS jobs, particularly ugly quartics; I will use Fib(1625), quartic SNFS difficulty 271.4, as a test case.

fivemack 2019-04-26 09:45

OK, I am reasonably confident that for 16e lpa=33 lim=268000000 it is worth using ncurves1=15 rather than the default 25 (basically, each curve factors 25% of the remaining usable composites).

[code]
ncurves1 yield total time t_factor0 t_factor1
5 185576 270537.39 40156.7 6422.3
10 223265 275086.68 39581.3 11120.9
15 234399 277763.44 39406.4 16285
20 236494 277285.09 38674.7 21739.8
25 236782 289479.05 38263.5 27276.3
30 236810 288675.62 37848.9 32619.5
[/code]

ncurves1=15 comes from fitting lines to t_factor0+t_factor1, adding the average non-factor time, and then optimising expected-yield / total-expected-time.

fivemack 2019-04-26 15:53

17e is not really enough to make difficulty-270 quartic SNFS practical.

The yield is a bit higher, but you have to go to 34-bit large primes to get more than one relation per Q, and collecting a billion relations at nine CPU-seconds per relation is not a plausible job.

[code]
I lpr time/rel rels/Q
15 32 10.09365591 0.06975
15 33 5.61544898 0.1225
15 34 3.367857143 0.21
16 32 11.4795212 0.18275
16 33 6.513562066 0.32425
16 34 3.922789209 0.54675
17 32 28.05035398 0.452
17 33 15.07548578 0.844
17 34 8.857269553 1.432
[/code]

(this is with rlim=268M alim=67M lpba=30, because big quartics are very asymmetric towards the rational side)

With -t4, 15e takes 7579MB, 16e takes 25610MB, 17e takes 89361MB

EdH 2019-05-10 03:20

VBCurtis,

I'm having a trouble I think I had before: My system is crashing during polyselect - the host is hanging up. The composite is a C139, so CADO-NFS is using params.c140. Do you have a modified c140 I should try before I do something with the default one?

Ed

VBCurtis 2019-05-10 04:30

1 Attachment(s)
Ed-
RichD and I have been working on params for c140 and c145.
Attached is the fastest c140 file we've yet run; I'm almost ready to post it "officially", but I want to test a few more inputs to make sure I didn't just get a lucky poly on this run.

I think the hang happens on very small c5 values; we worked around it by using a small admin value such as tasks.polyselect.admin = 1680 (used in this file, in fact).

Note the file has an input and name and number of clients for Rich's 4-core i5; you should reset the top section to what you normally use.

Edit: may as well include timing data: On a non-HT i5-quad-core desktop, 16k thread-sec poly select, 195k thread-sec sieve, 29k thread-sec matrix. Wall clock time 112k sec, 31 hrs and change.

EdH 2019-05-11 03:26

Thanks Curtis,

I'll try to run this tomorrow. I did remove the RichD specific lines.

-Ed

EdH 2019-05-15 19:20

Just an update. The original c139 that I mentioned above would not factor with cado-nfs. The host kept crashing.

I tried a different set of composites and all ran fine. I then put the original c139 back in the queue and ECM factored it while I wasn't looking.

I haven't bothered finding and reconstructing the c139, but several other composites have been using the modified params.c140 file without issue.

I will try to get some details up soon.

Thanks!

SethTro 2019-06-12 17:37

I'm running a c150 by myself (e.g. ./cado-nfs.py <BIG_NUMBER> --screenlog DEBUG --filelog DEBUG), I'm not seeing datetime/timestamps in my log files.

In VBCurtis's coordinated Team CADO solve of 2,2330L all the log files have both PID and datetime.

What option(s) do I set to get datetime in my logs?

VBCurtis 2019-06-12 23:44

The only option I set for CADO jobs is --server-threads, if I didn't set it in the params file.
Maybe try ditching the DEBUG flags?

henryzz 2019-09-07 22:01

Every so often I am getting an error for full buckets while sieving. Is there an easy fix for this?

EdH 2019-09-30 00:25

@VBCurtis:

I modified the "improved" params.c140 into a "modified" params.c150 (although I forgot to rename it within the file) based on your suggestions in the "Improved" thread and am running it on another c152 composite. I hadn't noticed this before, but maybe it's normal behavior: after settling with receipt of several WUs from many clients, it gave an ETA of ~7 PM. Later on a check showed an ETA of ~7:30 and then it kept moving later. I then started up a few more clients and even after they had submitted several WUs, the march into the future continues. At present, the original ETA of ~7 PM has transformed into later than 9:30 PM. I can foresee it as an elusive goal.

I assume (I know!) this is because of diminishing relations as the search area progresses, but thought I'd offer this bit of info in case it is helpful in reaching better parameters.

-Ed

RichD 2019-09-30 00:35

[QUOTE=EdH;526939]@VBCurtis:

I modified the "improved" params.c140 into a "modified" params.c150 (although I forgot to rename it within the file) based on your suggestions in the "Improved" thread and am running it on another c152 composite. I hadn't noticed this before, but maybe it's normal behavior: after settling with receipt of several WUs from many clients, it gave an ETA of ~7 PM. Later on a check showed an ETA of ~7:30 and then it kept moving later. I then started up a few more clients and even after they had submitted several WUs, the march into the future continues. At present, the original ETA of ~7 PM has transformed into later than 9:30 PM. I can foresee it as an elusive goal.

I assume (I know!) this is because of diminishing relations as the search area progresses, but thought I'd offer this bit of info in case it is helpful in reaching better parameters.

-Ed[/QUOTE]

I've noticed it too when I did some stand-alone (single box) jobs for VBCurtis. The ETA on sieving perpetually increases, most likely from what you mentioned above.

EdH 2019-09-30 01:08

[QUOTE=RichD;526941]I've noticed it too when I did some stand-alone (single box) jobs for VBCurtis. The ETA on sieving perpetually increases, most likely from what you mentioned above.[/QUOTE]I was somewhat disappointed that adding several more totally unique clients did not seem to offset the increasing loss.

RichD 2019-09-30 01:51

[QUOTE=EdH;526942]I was somewhat disappointed that adding several more totally unique clients did not seem to offset the increasing loss.[/QUOTE]

I can't think of a reason except that the first few WUs being returned may have a temporary reverse in ETA. After that is established you would have a constant flow of WUs with each being fewer relations [strike]as[/strike] than before.

VBCurtis 2019-09-30 02:34

This is a side effect of one of the settings I use that CADO default does not:
I start with very low Q values, where relations are much faster and higher-yield. CADO starts with Q in the general vicinity of lim0, a higher value than my jobs usually end at.

So, CADO's files run over a Q-range where sec/rel is fairly steady, while we run from Q where relations are very fast to a medium-sized Q where relations are still faster than CADO's ranges but measurably slower than the low starting Q values.

So, since ETA is based on the average rate of relations, the ETA keeps slipping throughout the job.

The massive yield and sec/rel improvement at low Q is also why I shift to I=13 and I=14 so much lower than the default files. Turns out I=13 at 120 digits finds so many relations at small Q values that it's overall faster than I=12, even though I=12 is faster over any particular Q-range as measured by sec/rel.

I'm pretty sure that a hybrid of, say, I=14 at low Q and I=13 at higher Q would be even faster for 135-150 digits, but I haven't gotten 'round to asking the CADO mailing list if they would consider implementing such an option.

EdH 2019-09-30 03:14

[QUOTE=VBCurtis;526946]. . .
I'm pretty sure that a hybrid of, say, I=14 at low Q and I=13 at higher Q would be even faster for 135-150 digits, but I haven't gotten 'round to asking the CADO mailing list if they would consider implementing such an option.[/QUOTE]Since I'm using scripts, it seems I should be able to catch a percentage of completion and interrupt CADO-NFS, then swap to I=13 with a snapshot restart. Isn't this what you manually did with 2,2330L?

I'm also considering more CADO-NFS/msieve hybrid work. The catch is if msieve fails to create a matrix with too few relations. At such a failure, can I alter the relations_wanted and restart with a snapshot?

I've been working with my scripts quite extensively in the recent days and might try to add some of the above as well, if time permits.

Thanks for all your help,
-Ed

VBCurtis 2019-09-30 03:46

Yes, That's what I did with 2330L, and you definitely could alter the snapshot file from I=14 to I=13 and restart. A wild guess is to do so after 30% of relations for c135 params, 50% for 140, 75% for 145, and not to bother (leave it all I=14) for c150. I'm actually planning to test I=13 for c135 params; that might be too soon for I=14.

I also agree that if msieve fails to build a matrix, you can add 10% to rels_wanted in the snapshot file and restart CADO. 5% might build a matrix, but above ~140 digits it's better to have a bit of oversieving for a faster matrix phase.


All times are UTC. The time now is 19:56.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.