CADO NFS - Page 31

Nooks · 2019-04-01, 15:34

EdH is right, --bindir is the flag you want.

I don't point it at my build/ directory, but at /pkg/cado/lib/cado-nfs-3.0.0 in an experimental containerized setup I've been working on, where something like this is required since downloading copies of binaries is ill-advised in this setup.

I've also used it when doing ad-hoc clustering across machines with different CPU types.

VBCurtis · 2019-04-02, 23:25

Quote:

Originally Posted by EdH

I'm not sure if this helps, but I had a similar issue due to various hardware and have to use --binddir=build/<username>/ on my clients. My current server is 16.04 and some clients are 18.04.

I tried this invocation, got "error: no such option: --binddir" :
./cado-nfs-client.py --server=http://{servername:port} --binddir=build/{install directory}
This is on the 18.04 machine, running recent-GIT (say, two weeks ago) CADO.

Nooks · 2019-04-03, 04:34

The flag is called --bindir. One "d", not two.

The build/ subdirectory is found in the unpacked source tree after compiling, or in the directory you built from if you followed the instructions in the "configure using cmake directly" section in the README file. In there is usually a directory named for the machine where CADO-NFS was compiled; ".mpi" is added if CADO-NFS was configured to use MPI.

The other option for --bindir is the lib/cado-nfs-3.0.0 subdirectory under the installation root.

Both of these locations have these subdirectories:

Code:

$ cd ~/cado-nfs/build/{hostname}.mpi; ls -d */
CMakeFiles/  filter/  linalg/  numbertheory/  scripts/  sqrt/   utils/
config/      gf2x/    misc/    polyselect/    sieve/    tests/

$ cd /mnt/pkg/cado/lib/cado-nfs-3.0.0/; ls -d */
filter/  misc/          polyselect/  sieve/  utils/
linalg/  numbertheory/  scripts/     sqrt/

I have not used the build dir as my setting for --bindir but I expect it would work.

(My installation root is a bit odd because /mnt hosts a glusterfs filesystem and I install 3rd-party packages into subdirectories of /pkg to keep things separated. Theoretically I could keep the CADO source code on glusterfs but with my current setup it makes compiling prohibitively slow. These details aren't really important, though.)

EdH · 2019-04-03, 14:21

Darn! And, I'm always trying to be so precise...

Sorry about that misspelling.

EdH · 2019-04-08, 19:51

Code:

...
Linking CXX executable test-flint
[100%] Built target test-flint
Scanning dependencies of target check_rels
[100%] Building C object misc/CMakeFiles/check_rels.dir/check_rels.c.o
Linking CXX executable check_rels
[100%] Built target check_rels
user@machine:~/Math/cado-nfs$ ./cado-nfs.py 90377629292003121684002147101760858109247336549001090677693
Traceback (most recent call last):
  File "./cado-nfs.py", line 43, in <module>
    import cadotask
  File "./scripts/cadofactor/cadotask.py", line 2960, in <module>
    patterns.Observer):
  File "./scripts/cadofactor/cadotask.py", line 3151, in SievingTask
    if tuple(sys.version_info)[0] < 3:
NameError: name 'sys' is not defined

This was reproduced on several machines.

I added:

Code:

import sys

to ./scripts/cadofactor/cadotask.py and all machines work now.

Is there any anticipated trouble running the latest (developmental) version on the server and a mixture of dev and 2.3.0 on the clients?

Are there better params files or modifications to make if there will be over a dozen multi-core machines running as clients?

What info would be useful to gather in a compiled log? I'm looking at something like "egrep"ping some of the lines from the log files for all my cado-nfs runs.

VBCurtis · 2019-04-08, 20:14

Ed-
I have improved params files for C90 through C125 completed; I've been putting off posting them until I have concrete a/b comparisons for timings, something I've been lazy about doing.
I did finish C90 and C95 timing: On 1 thread, CADO-git from Feb factoring a C90 was 2236 seconds, while my file was 941 seconds and YAFU was 841 seconds. On 6 threads, YAFU did a c95 in 1510 seconds, CADO stock 1008 seconds, CADO with my params 625 seconds.

For a farm like yours where sieving is manycore but postprocessing is single-machine, it makes sense to add 10% to the target number of relations. This adds 10% to sieving time, but reduces matrix time by 30-60%. On a single machine this is slightly a waste of time (at least on my params files, where I've tried to find the ideal # of relations such that the extra time spent sieving balances the time saved on matrix), but you'll save quite a lot in wall-clock time.

I have draft files for C130 to C155, which I think are better than stock but not yet fully refined. Which files would you like?

EdH · 2019-04-09, 03:15

Quote:

Originally Posted by VBCurtis

Ed-
I have improved params files for C90 through C125 completed; I've been putting off posting them until I have concrete a/b comparisons for timings, something I've been lazy about doing.
I did finish C90 and C95 timing: On 1 thread, CADO-git from Feb factoring a C90 was 2236 seconds, while my file was 941 seconds and YAFU was 841 seconds. On 6 threads, YAFU did a c95 in 1510 seconds, CADO stock 1008 seconds, CADO with my params 625 seconds.

For a farm like yours where sieving is manycore but postprocessing is single-machine, it makes sense to add 10% to the target number of relations. This adds 10% to sieving time, but reduces matrix time by 30-60%. On a single machine this is slightly a waste of time (at least on my params files, where I've tried to find the ideal # of relations such that the extra time spent sieving balances the time saved on matrix), but you'll save quite a lot in wall-clock time.

I have draft files for C130 to C155, which I think are better than stock but not yet fully refined. Which files would you like?

I caught the new thread on params files. I'll get them from there later.

I have adjusted my plans somewhat:

I think I'll work on refining my data mining first. Then I'll work with the current params to get a larger sample. After that, I'll try the modified files, as they are and finally, try the adjustments you suggest for my "farm."

A question arises:

Although the ideal is to compare exact composite factorizations, is there value in comparing different composites of the same size?

I ask this because my current interest is in scripts that take Aliquot sequences up to ~140 dd via ecmpi and CADO-NFS. If I aggregate the data for same sized, but different, composites, would that still be valuable?

A last, and quite different subject:

Since I have a working openmpi cluster and there is a section within the CADO-NFS documentation discussing mpi distributed LA, I am inclined to study this a bit. Is it implemented and working or just on its way to being available?

VBCurtis · 2019-04-09, 05:01

Your last question is, to me, the most interesting. I believe MPI is working but not documented/supported by the cado-nfs.py script; I expect you'll have to invoke the filtering/bwc steps individually. That said, I haven't peeked inside the cado-nfs.py file itself to see if there are MPI flags afoot.

I collect data by job-size too, and believe that aggregating average job times for a given length is helpful for refining parameters. I use a combination of "these parameters produced the record-quickest job of this size" and "those parameters have a lower average time per job" to decide on fastest param choices; I've also learned after 100+ jobs that some inputs just get lucky polynomials and the params have little to do with the record-low job time. I conclude that it's important to record poly score for each job, as well as first digit (C130 with first digit 6 will clearly be slower than C130 with first digit 1, more than halfway to a C131). If you also gather multiple results for a specific input size, I think that will be very helpful.

If refining parameters for C130-155 job sizes interests you, I'll be happy to share on the other thread my procedures and which specific things I've tried; my results may be more influenced by my hardware than I realize, so your data-gathering may alter some of the choices.

VBCurtis · 2019-04-10, 06:44

Quote:

Originally Posted by EdH

Since I have a working openmpi cluster and there is a section within the CADO-NFS documentation discussing mpi distributed LA, I am inclined to study this a bit. Is it implemented and working or just on its way to being available?

Have a look at local.sh.example in the CADO folder. Seems MPI is ready-to-enable.

bsquared · 2019-04-10, 15:14

Is there a way to run with python 2.7? I would like to test on a system on which I am unable to install new software. Alternatively, is it easy to install python3 somewhere in my home directory (where I am able to) and then have cado use that and not the system python?

[edit]
Found a guide for installing python in a home directory, involving compiling from source (ugh).
https://codeghar.wordpress.com/2013/...in-centos-6-4/

EdH · 2019-04-11, 19:18

Hey Curtis,

Thanks for all the info. I'll need to spend a bit of time with mpi after I get all the other things figured out. Thanks for pointing me to the example file

I've compiled a bunch of data, a taste of which is below. Is there enough info in the listings? I can't seem to find any version info in the logs, nor can I figure a way to display params file unique ids. Would you be able to identify modified versus unmodified params files from any particular entry within the param file that was used. Would you even need that? I suppose I could add things like lpb0/1 and mfb0/1 values, etc., but the less needed, the better.

Here's a sample of what I've compiled thus far:

Code:

N<138> = 107858998268122985412779892463164903278148452826296404480200645178455386381787979632919305998206531705192000227253731053070278282139941833
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c140.upload/c140.polyselect2.4wl6t5p_.opt_60 has Murphy_E = 3.56e-07
Generate Factor Base: Total cpu/real time for makefb: 14.5/2.95031
Generate Free Relations: Total cpu/real time for freerel: 616.08/80.0004
Lattice Sieving: Total number of relations: 32432783
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 89.32/212.046
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 429.15/154.403
Filtering - Singleton removal: Total cpu/real time for purge: 210.65/72.3947
Filtering - Merging: Total cpu/real time for merge: 768.14/667.166
Filtering - Merging: Total cpu/real time for replay: 59.26/48.687
Linear Algebra: Total cpu/real time for bwc: 35648.5/9138.31
Quadratic Characters: Total cpu/real time for characters: 70.24/20.2549
Square Root: Factors: 4549852818280784137048891955891900188206561071 23706041178905384340613948799770096216924619543119243174197081516303277264106367581589913223
Square Root: Total cpu/real time for sqrt: 3600.09/502.14
Generate Factor Base: Total cpu/real time for makefb: 14.5/2.95031
Generate Free Relations: Total cpu/real time for freerel: 616.08/80.0004
Lattice Sieving: Total number of relations: 32432783
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 89.32/212.046
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 429.15/154.403
Filtering - Singleton removal: Total cpu/real time for purge: 210.65/72.3947
Filtering - Merging: Total cpu/real time for merge: 768.14/667.166
Filtering - Merging: Total cpu/real time for replay: 59.26/48.687
Linear Algebra: Total cpu/real time for bwc: 35648.5/9138.31
Quadratic Characters: Total cpu/real time for characters: 70.24/20.2549
Square Root: Total cpu/real time for sqrt: 3600.09/502.14
Complete Factorization: Total cpu/elapsed time for entire factorization: 709733/16633.3
====================================================
N<133> = 1727651299324421928413446724594590880782759412370592923996274508812661722242606590744478625899708857621177484486201878763223620024233
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c135.upload/c135.polyselect2.sy3j0pmg.opt_54 has Murphy_E = 1.56e-07
Generate Factor Base: Total cpu/real time for makefb: 18.49/4.08387
Generate Free Relations: Total cpu/real time for freerel: 308.64/40.4427
Lattice Sieving: Total number of relations: 16820659
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 47.48/122.93
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 237.05/225.074
Filtering - Singleton removal: Total cpu/real time for purge: 125.5/52.3022
Filtering - Merging: Total cpu/real time for merge: 701.01/618.263
Filtering - Merging: Total cpu/real time for replay: 49.69/41.2704
Linear Algebra: Total cpu/real time for bwc: 25661.9/6836.89
Quadratic Characters: Total cpu/real time for characters: 49.05/16.6863
Square Root: Factors: 30694206313771024179893176877347248620777395297910299794744836039393412307041777132449849 56285908867085034103571361947375561950143217
Square Root: Total cpu/real time for sqrt: 2458.36/380.537
Generate Factor Base: Total cpu/real time for makefb: 18.49/4.08387
Generate Free Relations: Total cpu/real time for freerel: 308.64/40.4427
Lattice Sieving: Total number of relations: 16820659
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 47.48/122.93
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 237.05/225.074
Filtering - Singleton removal: Total cpu/real time for purge: 125.5/52.3022
Filtering - Merging: Total cpu/real time for merge: 701.01/618.263
Filtering - Merging: Total cpu/real time for replay: 49.69/41.2704
Linear Algebra: Total cpu/real time for bwc: 25661.9/6836.89
Quadratic Characters: Total cpu/real time for characters: 49.05/16.6863
Square Root: Total cpu/real time for sqrt: 2458.36/380.537
Complete Factorization: Total cpu/elapsed time for entire factorization: 551879/12586.8
====================================================
N<135> = 125844213559622587695829140394565697295294400320119878829054659162532607479389416797743340395107681033406552019669161119071940264791049
Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c135.upload/c135.polyselect2.xnurtmbv.opt_90 has Murphy_E = 1.26e-07
Generate Factor Base: Total cpu/real time for makefb: 18.98/4.3001
Generate Free Relations: Total cpu/real time for freerel: 308.8/44.8066
Lattice Sieving: Total number of relations: 16814708
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 48.67/123.119
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 231.22/151.544
Filtering - Singleton removal: Total cpu/real time for purge: 103.84/40.3791
Filtering - Merging: Total cpu/real time for merge: 852.76/755.683
Filtering - Merging: Total cpu/real time for replay: 58.03/48.4404
Linear Algebra: Total cpu/real time for bwc: 34137.1/9071.89
Quadratic Characters: Total cpu/real time for characters: 56.94/19.1343
Square Root: Factors: 1112732091037743546494994765551139874188005873287498358284451704438039 113094800242760311335825733173241877904256425883496448249505690591
Square Root: Total cpu/real time for sqrt: 3070.26/472.152
Generate Factor Base: Total cpu/real time for makefb: 18.98/4.3001
Generate Free Relations: Total cpu/real time for freerel: 308.8/44.8066
Lattice Sieving: Total number of relations: 16814708
Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 48.67/123.119
Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 231.22/151.544
Filtering - Singleton removal: Total cpu/real time for purge: 103.84/40.3791
Filtering - Merging: Total cpu/real time for merge: 852.76/755.683
Filtering - Merging: Total cpu/real time for replay: 58.03/48.4404
Linear Algebra: Total cpu/real time for bwc: 34137.1/9071.89
Quadratic Characters: Total cpu/real time for characters: 56.94/19.1343
Square Root: Total cpu/real time for sqrt: 3070.26/472.152
Complete Factorization: Total cpu/elapsed time for entire factorization: 680225/16017.4

This is from a fresh install of the dev version from a few days ago.

2019-04-03, 04:34	#333
Nooks Jul 2018 10011₂ Posts	The flag is called --bindir. One "d", not two. The build/ subdirectory is found in the unpacked source tree after compiling, or in the directory you built from if you followed the instructions in the "configure using cmake directly" section in the README file. In there is usually a directory named for the machine where CADO-NFS was compiled; ".mpi" is added if CADO-NFS was configured to use MPI. The other option for --bindir is the lib/cado-nfs-3.0.0 subdirectory under the installation root. Both of these locations have these subdirectories: Code: $ cd ~/cado-nfs/build/{hostname}.mpi; ls -d / CMakeFiles/ filter/ linalg/ numbertheory/ scripts/ sqrt/ utils/ config/ gf2x/ misc/ polyselect/ sieve/ tests/ $ cd /mnt/pkg/cado/lib/cado-nfs-3.0.0/; ls -d / filter/ misc/ polyselect/ sieve/ utils/ linalg/ numbertheory/ scripts/ sqrt/ I have not used the build dir as my setting for --bindir but I expect it would work. (My installation root is a bit odd because /mnt hosts a glusterfs filesystem and I install 3rd-party packages into subdirectories of /pkg to keep things separated. Theoretically I could keep the CADO source code on glusterfs but with my current setup it makes compiling prohibitively slow. These details aren't really important, though.)

2019-04-08, 19:51	#335
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2·3²·197 Posts	Minor Issue With Development Version Code: ... Linking CXX executable test-flint [100%] Built target test-flint Scanning dependencies of target check_rels [100%] Building C object misc/CMakeFiles/check_rels.dir/check_rels.c.o Linking CXX executable check_rels [100%] Built target check_rels user@machine:~/Math/cado-nfs$ ./cado-nfs.py 90377629292003121684002147101760858109247336549001090677693 Traceback (most recent call last): File "./cado-nfs.py", line 43, in <module> import cadotask File "./scripts/cadofactor/cadotask.py", line 2960, in <module> patterns.Observer): File "./scripts/cadofactor/cadotask.py", line 3151, in SievingTask if tuple(sys.version_info)[0] < 3: NameError: name 'sys' is not defined This was reproduced on several machines. I added: Code: import sys to ./scripts/cadofactor/cadotask.py and all machines work now. Is there any anticipated trouble running the latest (developmental) version on the server and a mixture of dev and 2.3.0 on the clients? Are there better params files or modifications to make if there will be over a dozen multi-core machines running as clients? What info would be useful to gather in a compiled log? I'm looking at something like "egrep"ping some of the lines from the log files for all my cado-nfs runs.

2019-04-08, 20:14	#336
VBCurtis "Curtis" Feb 2005 Riverside, CA 1215₁₆ Posts	Ed- I have improved params files for C90 through C125 completed; I've been putting off posting them until I have concrete a/b comparisons for timings, something I've been lazy about doing. I did finish C90 and C95 timing: On 1 thread, CADO-git from Feb factoring a C90 was 2236 seconds, while my file was 941 seconds and YAFU was 841 seconds. On 6 threads, YAFU did a c95 in 1510 seconds, CADO stock 1008 seconds, CADO with my params 625 seconds. For a farm like yours where sieving is manycore but postprocessing is single-machine, it makes sense to add 10% to the target number of relations. This adds 10% to sieving time, but reduces matrix time by 30-60%. On a single machine this is slightly a waste of time (at least on my params files, where I've tried to find the ideal # of relations such that the extra time spent sieving balances the time saved on matrix), but you'll save quite a lot in wall-clock time. I have draft files for C130 to C155, which I think are better than stock but not yet fully refined. Which files would you like? Last fiddled with by VBCurtis on 2019-04-08 at 20:24 Reason: added "factoring a C90" for the first timing notes

2019-04-10, 15:14	#340
bsquared "Ben" Feb 2007 3362₁₀ Posts	Is there a way to run with python 2.7? I would like to test on a system on which I am unable to install new software. Alternatively, is it easy to install python3 somewhere in my home directory (where I am able to) and then have cado use that and not the system python? [edit] Found a guide for installing python in a home directory, involving compiling from source (ugh). https://codeghar.wordpress.com/2013/...in-centos-6-4/ Last fiddled with by bsquared on 2019-04-10 at 16:14 Reason: partial answer

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
CADO-NFS on windows	jux	CADO-NFS	22	2019-11-12 12:08
CADO help	henryzz	CADO-NFS	4	2017-11-20 15:14
CADO and WinBlows	akruppa	Programming	22	2015-12-31 08:37
CADO-NFS	skan	Information & Answers	1	2013-10-22 07:00
CADO	R.D. Silverman	Factoring	4	2008-11-06 12:35

2019-04-01, 15:34	#331
Nooks Jul 2018 19 Posts	EdH is right, --bindir is the flag you want. I don't point it at my build/ directory, but at /pkg/cado/lib/cado-nfs-3.0.0 in an experimental containerized setup I've been working on, where something like this is required since downloading copies of binaries is ill-advised in this setup. I've also used it when doing ad-hoc clustering across machines with different CPU types.

2019-04-03, 14:21	#334
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2×3²×197 Posts	Darn! And, I'm always trying to be so precise... Sorry about that misspelling.

2019-04-09, 05:01	#338
VBCurtis "Curtis" Feb 2005 Riverside, CA 3×1,543 Posts	Your last question is, to me, the most interesting. I believe MPI is working but not documented/supported by the cado-nfs.py script; I expect you'll have to invoke the filtering/bwc steps individually. That said, I haven't peeked inside the cado-nfs.py file itself to see if there are MPI flags afoot. I collect data by job-size too, and believe that aggregating average job times for a given length is helpful for refining parameters. I use a combination of "these parameters produced the record-quickest job of this size" and "those parameters have a lower average time per job" to decide on fastest param choices; I've also learned after 100+ jobs that some inputs just get lucky polynomials and the params have little to do with the record-low job time. I conclude that it's important to record poly score for each job, as well as first digit (C130 with first digit 6 will clearly be slower than C130 with first digit 1, more than halfway to a C131). If you also gather multiple results for a specific input size, I think that will be very helpful. If refining parameters for C130-155 job sizes interests you, I'll be happy to share on the other thread my procedures and which specific things I've tried; my results may be more influenced by my hardware than I realize, so your data-gathering may alter some of the choices.