How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines

[EdH]

(Note: I expect to keep the first post of each of these "How I..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.)

This thread will explain the steps I use to run CADO-NFS on several computers which are already running Ubuntu and have CADO-NFS installed* and tested per:

How I Install CADO-NFS on my Ubuntu Machines

This procedure should work for other linux distributions as well but the only other one I've currently tested is Debain. In my case I needed to install python3 from the repository on my Debian based machines . More details about CADO-NFS and my early experiments can be found in the CADO-NFS thread.

*In this instance "install" is referring to the acquiring and compilation of the CADO-NFS package only. The scripts will have to be called using their respective paths.

I will be referencing the folder Math/cado-nfs as described in the installation routine above. Make any local changes for your computer.

Determine which machine will be the server and which will be clients.

Open a terminal on each, move to the appropriate folder and test each with the suggested run from the README file:

cd Math/cado-nfs
./cado-nfs.py 90377629292003121684002147101760858109247336549001090677693

For me, the final results are:

Info:root: Using default parameter file ./parameters/factor/params.c60
Info:root: No database exists yet
Info:root: Created temporary directory /tmp/cado.3nu5mplc
Info:Database: Opened connection to database /tmp/cado.3nu5mplc/c60.db
Info:root: Set tasks.threads=8 based on detected logical cpus
Info:root: tasks.polyselect.threads = 2
Info:root: tasks.sieve.las.threads = 2
Info:root: slaves.scriptpath is /home/math79/Math/cado-nfs
Info:root: Command line parameters: ./cado-nfs.py 90377629292003121684002147101760858109247336549001090677693
...
Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 55.2/37.7927
Info:root: Cleaning up computation data in /tmp/cado.3nu5mplc
260938498861057 760926063870977 773951836515617 588120598053661

Now, let's choose a slightly larger composite:

1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281

The machine we are going to use as the server could factor this number without too much trouble, but the completion can be sped up by adding clients. In order to add clients, some extra options will need to be added to the command line on the server machine. There will also need to be some extra options for our clients, too, but for now let's work with the server.

For my setup, I am using ssh and all my machines are able to connect with each other already so I don't use the --server option. However, ssh isn't necessary or described here. If interested, ssh has many pages on the web. For now, we'll use the --server option for these steps.

I prefer to let the default files, that are based on digit size, be called, rather than constructing my own, so the composite will be entered on the command line rather than within a parameter file.

A necessary option to add clients is server.whitelist. The whitelist will need to include the IP of any possible client. This can be individual or in a masked manner. By masked manner, I am referring to 192.168.1.0/24 meaning any IP from 192.168.1.0 through 192.168.1.255. With the bare minimum from above provided, you should be able to start the server running with the command:

./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 --server

The following should be returned:

./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 --server
Info:root: Using default parameter file ./parameters/factor/params.c95
Info:root: No database exists yet
Info:root: Created temporary directory /tmp/cado.4thcy1o_
Info:Database: Opened connection to database /tmp/cado.4thcy1o_/c95.db
Info:root: Set tasks.threads=8 based on detected logical cpus
Info:root: tasks.polyselect.threads = 2
Info:root: tasks.sieve.las.threads = 2
Info:root: Command line parameters: ./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 --server
Info:root: If this computation gets interrupted, it can be resumed with ./cado-nfs.py /tmp/cado.4thcy1o_/c95.parameters_snapshot.0
Info:HTTP server: Using non-threaded HTTPS server
Info:HTTP server: Using whitelist: 192.168.1.0/24
Info:Complete Factorization: Factoring 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281
Info:HTTP server: serving at https://math79:42668 (0.0.0.0)
Info:HTTP server: For debugging purposes, the URL above can be accessed if the server.only_registered=False parameter is added
Info:HTTP server: You can start additional cado-nfs-client.py scripts with parameters: --server=https://math79:42668 --certsha1=c2ee63f9850b3e23d0707a760762f39a83f48628
Info:HTTP server: If you want to start additional clients, remember to add their hosts to server.whitelist
...

If the server machine was left alone, it would fully factor the composite all by itself:

Info:HTTP server: Shutting down HTTP server
Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 6014.18/949.3
Info:root: Cleaning up computation data in /tmp/cado.nprm037c
179231227423414197451601378315047105853969879 11022834899950977366949652871606409040980556071039

But, since this is supposed to show how I use multiple machines, I'm going to add two clients after a brief mention of two other options:

Check out the two bold sections of the following portion of the preceding run:

Info:HTTP server: You can start additional cado-nfs-client.py scripts
with parameters: --server=https://math79:42668
--certsha1=c2ee63f9850b3e23d0707a760762f39a83f48628

The first to look at is the port address (42668). In this case, the number was chosen by the server. When you run the clients, you need to know this number. This means you'll have to wait until the server is running and try to catch this information as it scrolls by. Alternatively, you can assign a number of your own choosing, which is my preferred method. To do this add server.port=<port#>. For the next example I'll use 56789.

The second option to look at is the --certsha1=... This is created by the server and can be used to validate clients. However, you need to catch this as it scrolls to use it in your client commands. If only machines from your local net are in use you may wish to decline using the ssl certificate validation. This can be accomplished with the option server.ssl=no. This is what I will use in the following example.

Let's start the server with all the options discussed and see what it shows:

./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 server.port=56789 server.ssl=no --server
Info:root: Using default parameter file ./parameters/factor/params.c95
Info:root: No database exists yet
Info:root: Created temporary directory /tmp/cado.e7kdh_gs
Info:Database: Opened connection to database /tmp/cado.e7kdh_gs/c95.db
Info:root: Set tasks.threads=8 based on detected logical cpus
Info:root: tasks.polyselect.threads = 2
Info:root: tasks.sieve.las.threads = 2
Info:root: Command line parameters: ./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 server.port=56789 server.ssl=no --server
Info:root: If this computation gets interrupted, it can be resumed with ./cado-nfs.py /tmp/cado.e7kdh_gs/c95.parameters_snapshot.0
Info:HTTP server: Using non-threaded HTTP server
Info:HTTP server: Using whitelist: 192.168.1.0/24
Info:Complete Factorization: Factoring 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281
Info:HTTP server: serving at http://math79:56789 (0.0.0.0)
Info:HTTP server: For debugging purposes, the URL above can be accessed if the server.only_registered=False parameter is added
Info:HTTP server: You can start additional cado-nfs-client.py scripts with parameters: --server=http://math79:56789
Info:HTTP server: If you want to start additional clients, remember to add their hosts to server.whitelist
...

Now that the server is running, let's move to the clients. There's a line above that reads:

Info:HTTP server: You can start additional cado-nfs-client.py scripts with parameters: --server=http://math79:56789

It's kind of right, but we'll actually have to add a little to the address to finish it. There is also one more option that is important if you are using clients with different hardware from your server; --bindir. This option tells your client to use its own programs rather than download them from the server. If you used the standard compilation of the programs, they will be in cado-nfs/build/<machine name>. Here is the command line I will use on both of the two clients. This line can be used on all your clients by changing the bindir path appropriately:

./cado-nfs-client.py --server=http://math79.local:56789 --bindir=build/math77/

I've highlighted the .local above, to explain that if the clients are on the same local network as the server, so you can use .local to connect. You can also connect via the server's IP: @192..., which is what you'll need to use if connecting from outside the local network.

Note: The above line starts one client with two threads on its client machine. If you have more threads available start more clients wih the same command line. My three machines all have 8 threads, so I will be starting four clients on each client machine using the above command in four separate terminals. Each client should show something like:

INFO:root:Starting client math77.4b87c3d8
INFO:root:Python version is 2.7.12
INFO:root:Downloading http://math79.local:56789/cgi-bin/getwu?clientid=math77.4b87c3d8 to download/WU.math77.4b87c3d8 (cafile = None)
INFO:root:Result file math77.4b87c3d8.work/c95.polyselect1.40000-50000 does not exist
INFO:root:Running 'build/math77/polyselect/polyselect' -P 10000 -N 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 -degree 4 -t 2 -admin 40000 -admax 50000 -incr 60 -nq 256 > 'math77.4b87c3d8.work/c95.polyselect1.40000-50000'
INFO:root:Attaching file math77.4b87c3d8.work/c95.polyselect1.40000-50000 to upload
INFO:root:Sending result for workunit c95_polyselect1_40000-50000 to http://math79.local:56789/cgi-bin/upload.py
INFO:root:Cleaning up for workunit c95_polyselect1_40000-50000
...

Eventually, they will show something like:

INFO:root:Running 'build/math77/sieve/las' -I 13 -poly 'download/c95.poly' -q0 54000 -q1 56000 -lim0 3660220 -lim1 2758600 -lpb0 23 -lpb1 23 -mfb0 22 -mfb1 45 -ncurves0 2 -ncurves1 13 -fb 'download/c95.roots.gz' -out 'math77.4b87c3d8.work/c95.54000-56000.gz' -t 2 -stats-stderr
INFO:root:Attaching file math77.4b87c3d8.work/c95.54000-56000.gz to upload
INFO:root:Attaching stderr for command 0 to upload
INFO:root:Sending result for workunit c95_sieving_54000-56000 to http://math79.local:56789/cgi-bin/upload.py
INFO:root:Cleaning up for workunit c95_sieving_54000-56000

After enough sieving has been reported, the server will move to the Linear Algebra work and the clients will be told they are no longer needed:

Info:Lattice Sieving: Cancelling remaining workunits
Info:Filtering - Singleton removal: Total cpu/real time for purge: 7.67/2.33534
Info:Filtering - Merging: Starting
Info:Filtering - Merging: Merged matrix has 85097 rows and total weight 14466502 (170.0 entries per row on average)
Info:Filtering - Merging: Total cpu/real time for merge: 22.66/17.3496
Info:Filtering - Merging: Total cpu/real time for replay: 2.91/2.44187
Info:Linear Algebra: Starting

INFO:root:Client finishing: Received 410 from server. Bye.

And, finally, the server completes the rest on its own:

Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 5273.74/470.649
Info:root: Cleaning up computation data in /tmp/cado.o8quwcb4
179231227423414197451601378315047105853969879 11022834899950977366949652871606409040980556071039

Now, let's compare times for factoring between one machine and three of the same architecture. Single machine:

Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 6014.18/949.3

One server and two client machines. Each machine actually ran four clients, each client with two threads:

Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 5273.74/470.649

With three machines, it took roughly half as long to factor the 94 digt composite. At first look, you may wonder why it didn't take one-third as long. The reason is mostly due to the fact that the Linear Algebra steps are only performed on the server. Overhead also plays a small role.

For further details on some of my testing, specifically the factorization of RSA-130 see the above referenced thread on CADO-NFS, starting with post 137.