Register FAQ Search Today's Posts Mark Forums Read

 2019-04-01, 15:34 #331 Nooks   Jul 2018 1316 Posts EdH is right, --bindir is the flag you want. I don't point it at my build/ directory, but at /pkg/cado/lib/cado-nfs-3.0.0 in an experimental containerized setup I've been working on, where something like this is required since downloading copies of binaries is ill-advised in this setup. I've also used it when doing ad-hoc clustering across machines with different CPU types.
2019-04-02, 23:25   #332
VBCurtis

"Curtis"
Feb 2005
Riverside, CA

114216 Posts

Quote:
 Originally Posted by EdH I'm not sure if this helps, but I had a similar issue due to various hardware and have to use --binddir=build// on my clients. My current server is 16.04 and some clients are 18.04.
I tried this invocation, got "error: no such option: --binddir" :
This is on the 18.04 machine, running recent-GIT (say, two weeks ago) CADO.

 2019-04-03, 04:34 #333 Nooks   Jul 2018 19 Posts The flag is called --bindir. One "d", not two. The build/ subdirectory is found in the unpacked source tree after compiling, or in the directory you built from if you followed the instructions in the "configure using cmake directly" section in the README file. In there is usually a directory named for the machine where CADO-NFS was compiled; ".mpi" is added if CADO-NFS was configured to use MPI. The other option for --bindir is the lib/cado-nfs-3.0.0 subdirectory under the installation root. Both of these locations have these subdirectories: Code: $cd ~/cado-nfs/build/{hostname}.mpi; ls -d */ CMakeFiles/ filter/ linalg/ numbertheory/ scripts/ sqrt/ utils/ config/ gf2x/ misc/ polyselect/ sieve/ tests/$ cd /mnt/pkg/cado/lib/cado-nfs-3.0.0/; ls -d */ filter/ misc/ polyselect/ sieve/ utils/ linalg/ numbertheory/ scripts/ sqrt/ I have not used the build dir as my setting for --bindir but I expect it would work. (My installation root is a bit odd because /mnt hosts a glusterfs filesystem and I install 3rd-party packages into subdirectories of /pkg to keep things separated. Theoretically I could keep the CADO source code on glusterfs but with my current setup it makes compiling prohibitively slow. These details aren't really important, though.)
 2019-04-03, 14:21 #334 EdH     "Ed Hall" Dec 2009 Adirondack Mtns 22·7·112 Posts Darn! And, I'm always trying to be so precise... Sorry about that misspelling.
 2019-04-08, 19:51 #335 EdH     "Ed Hall" Dec 2009 Adirondack Mtns 22×7×112 Posts Minor Issue With Development Version Code: ... Linking CXX executable test-flint [100%] Built target test-flint Scanning dependencies of target check_rels [100%] Building C object misc/CMakeFiles/check_rels.dir/check_rels.c.o Linking CXX executable check_rels [100%] Built target check_rels user@machine:~/Math/cado-nfs\$ ./cado-nfs.py 90377629292003121684002147101760858109247336549001090677693 Traceback (most recent call last): File "./cado-nfs.py", line 43, in import cadotask File "./scripts/cadofactor/cadotask.py", line 2960, in patterns.Observer): File "./scripts/cadofactor/cadotask.py", line 3151, in SievingTask if tuple(sys.version_info)[0] < 3: NameError: name 'sys' is not defined This was reproduced on several machines. I added: Code: import sys to ./scripts/cadofactor/cadotask.py and all machines work now. Is there any anticipated trouble running the latest (developmental) version on the server and a mixture of dev and 2.3.0 on the clients? Are there better params files or modifications to make if there will be over a dozen multi-core machines running as clients? What info would be useful to gather in a compiled log? I'm looking at something like "egrep"ping some of the lines from the log files for all my cado-nfs runs.
 2019-04-08, 20:14 #336 VBCurtis     "Curtis" Feb 2005 Riverside, CA 2·472 Posts Ed- I have improved params files for C90 through C125 completed; I've been putting off posting them until I have concrete a/b comparisons for timings, something I've been lazy about doing. I did finish C90 and C95 timing: On 1 thread, CADO-git from Feb factoring a C90 was 2236 seconds, while my file was 941 seconds and YAFU was 841 seconds. On 6 threads, YAFU did a c95 in 1510 seconds, CADO stock 1008 seconds, CADO with my params 625 seconds. For a farm like yours where sieving is manycore but postprocessing is single-machine, it makes sense to add 10% to the target number of relations. This adds 10% to sieving time, but reduces matrix time by 30-60%. On a single machine this is slightly a waste of time (at least on my params files, where I've tried to find the ideal # of relations such that the extra time spent sieving balances the time saved on matrix), but you'll save quite a lot in wall-clock time. I have draft files for C130 to C155, which I think are better than stock but not yet fully refined. Which files would you like? Last fiddled with by VBCurtis on 2019-04-08 at 20:24 Reason: added "factoring a C90" for the first timing notes
2019-04-09, 03:15   #337
EdH

"Ed Hall"
Dec 2009

22×7×112 Posts

Quote:
 Originally Posted by VBCurtis Ed- I have improved params files for C90 through C125 completed; I've been putting off posting them until I have concrete a/b comparisons for timings, something I've been lazy about doing. I did finish C90 and C95 timing: On 1 thread, CADO-git from Feb factoring a C90 was 2236 seconds, while my file was 941 seconds and YAFU was 841 seconds. On 6 threads, YAFU did a c95 in 1510 seconds, CADO stock 1008 seconds, CADO with my params 625 seconds. For a farm like yours where sieving is manycore but postprocessing is single-machine, it makes sense to add 10% to the target number of relations. This adds 10% to sieving time, but reduces matrix time by 30-60%. On a single machine this is slightly a waste of time (at least on my params files, where I've tried to find the ideal # of relations such that the extra time spent sieving balances the time saved on matrix), but you'll save quite a lot in wall-clock time. I have draft files for C130 to C155, which I think are better than stock but not yet fully refined. Which files would you like?
I caught the new thread on params files. I'll get them from there later.

I have adjusted my plans somewhat:

I think I'll work on refining my data mining first. Then I'll work with the current params to get a larger sample. After that, I'll try the modified files, as they are and finally, try the adjustments you suggest for my "farm."

A question arises:

Although the ideal is to compare exact composite factorizations, is there value in comparing different composites of the same size?

I ask this because my current interest is in scripts that take Aliquot sequences up to ~140 dd via ecmpi and CADO-NFS. If I aggregate the data for same sized, but different, composites, would that still be valuable?

A last, and quite different subject:

Since I have a working openmpi cluster and there is a section within the CADO-NFS documentation discussing mpi distributed LA, I am inclined to study this a bit. Is it implemented and working or just on its way to being available?

 2019-04-09, 05:01 #338 VBCurtis     "Curtis" Feb 2005 Riverside, CA 2·472 Posts Your last question is, to me, the most interesting. I believe MPI is working but not documented/supported by the cado-nfs.py script; I expect you'll have to invoke the filtering/bwc steps individually. That said, I haven't peeked inside the cado-nfs.py file itself to see if there are MPI flags afoot. I collect data by job-size too, and believe that aggregating average job times for a given length is helpful for refining parameters. I use a combination of "these parameters produced the record-quickest job of this size" and "those parameters have a lower average time per job" to decide on fastest param choices; I've also learned after 100+ jobs that some inputs just get lucky polynomials and the params have little to do with the record-low job time. I conclude that it's important to record poly score for each job, as well as first digit (C130 with first digit 6 will clearly be slower than C130 with first digit 1, more than halfway to a C131). If you also gather multiple results for a specific input size, I think that will be very helpful. If refining parameters for C130-155 job sizes interests you, I'll be happy to share on the other thread my procedures and which specific things I've tried; my results may be more influenced by my hardware than I realize, so your data-gathering may alter some of the choices.
2019-04-10, 06:44   #339
VBCurtis

"Curtis"
Feb 2005
Riverside, CA

2×472 Posts

Quote:
 Originally Posted by EdH Since I have a working openmpi cluster and there is a section within the CADO-NFS documentation discussing mpi distributed LA, I am inclined to study this a bit. Is it implemented and working or just on its way to being available?
Have a look at local.sh.example in the CADO folder. Seems MPI is ready-to-enable.

 2019-04-10, 15:14 #340 bsquared     "Ben" Feb 2007 2×13×127 Posts Is there a way to run with python 2.7? I would like to test on a system on which I am unable to install new software. Alternatively, is it easy to install python3 somewhere in my home directory (where I am able to) and then have cado use that and not the system python?  Found a guide for installing python in a home directory, involving compiling from source (ugh). https://codeghar.wordpress.com/2013/...in-centos-6-4/ Last fiddled with by bsquared on 2019-04-10 at 16:14 Reason: partial answer
 2019-04-11, 19:18 #341 EdH     "Ed Hall" Dec 2009 Adirondack Mtns 22·7·112 Posts Hey Curtis, Thanks for all the info. I'll need to spend a bit of time with mpi after I get all the other things figured out. Thanks for pointing me to the example file I've compiled a bunch of data, a taste of which is below. Is there enough info in the listings? I can't seem to find any version info in the logs, nor can I figure a way to display params file unique ids. Would you be able to identify modified versus unmodified params files from any particular entry within the param file that was used. Would you even need that? I suppose I could add things like lpb0/1 and mfb0/1 values, etc., but the less needed, the better. Here's a sample of what I've compiled thus far: Code: N<138> = 107858998268122985412779892463164903278148452826296404480200645178455386381787979632919305998206531705192000227253731053070278282139941833 Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c140.upload/c140.polyselect2.4wl6t5p_.opt_60 has Murphy_E = 3.56e-07 Generate Factor Base: Total cpu/real time for makefb: 14.5/2.95031 Generate Free Relations: Total cpu/real time for freerel: 616.08/80.0004 Lattice Sieving: Total number of relations: 32432783 Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 89.32/212.046 Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 429.15/154.403 Filtering - Singleton removal: Total cpu/real time for purge: 210.65/72.3947 Filtering - Merging: Total cpu/real time for merge: 768.14/667.166 Filtering - Merging: Total cpu/real time for replay: 59.26/48.687 Linear Algebra: Total cpu/real time for bwc: 35648.5/9138.31 Quadratic Characters: Total cpu/real time for characters: 70.24/20.2549 Square Root: Factors: 4549852818280784137048891955891900188206561071 23706041178905384340613948799770096216924619543119243174197081516303277264106367581589913223 Square Root: Total cpu/real time for sqrt: 3600.09/502.14 Generate Factor Base: Total cpu/real time for makefb: 14.5/2.95031 Generate Free Relations: Total cpu/real time for freerel: 616.08/80.0004 Lattice Sieving: Total number of relations: 32432783 Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 89.32/212.046 Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 429.15/154.403 Filtering - Singleton removal: Total cpu/real time for purge: 210.65/72.3947 Filtering - Merging: Total cpu/real time for merge: 768.14/667.166 Filtering - Merging: Total cpu/real time for replay: 59.26/48.687 Linear Algebra: Total cpu/real time for bwc: 35648.5/9138.31 Quadratic Characters: Total cpu/real time for characters: 70.24/20.2549 Square Root: Total cpu/real time for sqrt: 3600.09/502.14 Complete Factorization: Total cpu/elapsed time for entire factorization: 709733/16633.3 ==================================================== N<133> = 1727651299324421928413446724594590880782759412370592923996274508812661722242606590744478625899708857621177484486201878763223620024233 Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c135.upload/c135.polyselect2.sy3j0pmg.opt_54 has Murphy_E = 1.56e-07 Generate Factor Base: Total cpu/real time for makefb: 18.49/4.08387 Generate Free Relations: Total cpu/real time for freerel: 308.64/40.4427 Lattice Sieving: Total number of relations: 16820659 Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 47.48/122.93 Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 237.05/225.074 Filtering - Singleton removal: Total cpu/real time for purge: 125.5/52.3022 Filtering - Merging: Total cpu/real time for merge: 701.01/618.263 Filtering - Merging: Total cpu/real time for replay: 49.69/41.2704 Linear Algebra: Total cpu/real time for bwc: 25661.9/6836.89 Quadratic Characters: Total cpu/real time for characters: 49.05/16.6863 Square Root: Factors: 30694206313771024179893176877347248620777395297910299794744836039393412307041777132449849 56285908867085034103571361947375561950143217 Square Root: Total cpu/real time for sqrt: 2458.36/380.537 Generate Factor Base: Total cpu/real time for makefb: 18.49/4.08387 Generate Free Relations: Total cpu/real time for freerel: 308.64/40.4427 Lattice Sieving: Total number of relations: 16820659 Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 47.48/122.93 Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 237.05/225.074 Filtering - Singleton removal: Total cpu/real time for purge: 125.5/52.3022 Filtering - Merging: Total cpu/real time for merge: 701.01/618.263 Filtering - Merging: Total cpu/real time for replay: 49.69/41.2704 Linear Algebra: Total cpu/real time for bwc: 25661.9/6836.89 Quadratic Characters: Total cpu/real time for characters: 49.05/16.6863 Square Root: Total cpu/real time for sqrt: 2458.36/380.537 Complete Factorization: Total cpu/elapsed time for entire factorization: 551879/12586.8 ==================================================== N<135> = 125844213559622587695829140394565697295294400320119878829054659162532607479389416797743340395107681033406552019669161119071940264791049 Polynomial Selection (root optimized): Finished, best polynomial from file /tmp/cadofactor/c135.upload/c135.polyselect2.xnurtmbv.opt_90 has Murphy_E = 1.26e-07 Generate Factor Base: Total cpu/real time for makefb: 18.98/4.3001 Generate Free Relations: Total cpu/real time for freerel: 308.8/44.8066 Lattice Sieving: Total number of relations: 16814708 Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 48.67/123.119 Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 231.22/151.544 Filtering - Singleton removal: Total cpu/real time for purge: 103.84/40.3791 Filtering - Merging: Total cpu/real time for merge: 852.76/755.683 Filtering - Merging: Total cpu/real time for replay: 58.03/48.4404 Linear Algebra: Total cpu/real time for bwc: 34137.1/9071.89 Quadratic Characters: Total cpu/real time for characters: 56.94/19.1343 Square Root: Factors: 1112732091037743546494994765551139874188005873287498358284451704438039 113094800242760311335825733173241877904256425883496448249505690591 Square Root: Total cpu/real time for sqrt: 3070.26/472.152 Generate Factor Base: Total cpu/real time for makefb: 18.98/4.3001 Generate Free Relations: Total cpu/real time for freerel: 308.8/44.8066 Lattice Sieving: Total number of relations: 16814708 Filtering - Duplicate Removal, splitting pass: Total cpu/real time for dup1: 48.67/123.119 Filtering - Duplicate Removal, removal pass: Total cpu/real time for dup2: 231.22/151.544 Filtering - Singleton removal: Total cpu/real time for purge: 103.84/40.3791 Filtering - Merging: Total cpu/real time for merge: 852.76/755.683 Filtering - Merging: Total cpu/real time for replay: 58.03/48.4404 Linear Algebra: Total cpu/real time for bwc: 34137.1/9071.89 Quadratic Characters: Total cpu/real time for characters: 56.94/19.1343 Square Root: Total cpu/real time for sqrt: 3070.26/472.152 Complete Factorization: Total cpu/elapsed time for entire factorization: 680225/16017.4 This is from a fresh install of the dev version from a few days ago.

 Similar Threads Thread Thread Starter Forum Replies Last Post jux CADO-NFS 22 2019-11-12 12:08 henryzz CADO-NFS 4 2017-11-20 15:14 akruppa Programming 22 2015-12-31 08:37 skan Information & Answers 1 2013-10-22 07:00 R.D. Silverman Factoring 4 2008-11-06 12:35

All times are UTC. The time now is 02:16.

Sat Oct 31 02:16:39 UTC 2020 up 50 days, 23:27, 2 users, load averages: 1.78, 1.90, 1.91