CADO NFS

[EdH]

Originally Posted by EdH

. . . More later.

The more:

There is an incompatibility barrier that was crossed sometime in March 2020 that causes WUs to fail across the barrier. This means that the server and clients need to be on the same side of March 2020.

Currently, there is an open issue that causes an Error -6 interruption on rare occasions. When this error occurs, CADO-NFS cannot complete filtering and won't move on. However, I have been able to run Msieve on the relations found, and even rerun CADO-NFS to add relations by using "tasks.sieve.rels_wanted" in the snapshot file and using the snapshot to restart the CADO-NFS run. It still failed at filtering in my tests.

[RichD]

So upgrading everything to the above "commit" should be fine, whether server, client or standalone.

[VBCurtis]

Originally Posted by RichD

So upgrading everything to the above "commit" should be fine, whether server, client or standalone.

Except for the sometimes-broken filtering..

[RichD]

Originally Posted by EdH

Currently, ...

Originally Posted by VBCurtis

Except for the sometimes-broken filtering..

Ah, yes.

[EdH]

Just thought I'd post some more about the latest commit I'm using, since I mentioned the error (exit -6) trouble.

I'm still running the same commit across my entire "farm." I have only experienced one or two more of the troublesome error, but the bulk of my factorizations have been for composites <160 dd.

I have experienced a couple other issues, but they may have been noticed with earlier commits, too. The most annoying is if I run the server without the "--server" switch, quite often, one or more of the localhost polyselect processes gets lost. Then the server stops everything (except the constant checking for it) until the timeout occurs. It then goes ahead and reissues the WU and all is then well. But waiting an hour (or even the 15 minutes I now have it set at) for a lost WU to be replaced is (as mentioned) "annoying."

Because of this, I now use the "--server" switch and invoke the clients on the server machine externally.

[bur]

I read the README and don't quite comprehend why it makes sense to have servers and clients running on the local machine. This example

Run a full factorization on the local machine, using 8 threads for the
server (this includes the linear algebra), and 4 jobs of 2 threads
each for the polynomial selection and the sieving

makes it seem to me that the server does the LA and the clients the poly-selection and sieving?

But if it's a single machine why not use all cores for poly and sieving and afterwards for the LA?

Or put simple: if I have no intention on running CADO distributed, do I need to concern myself with server & clients? Otherwise I'd just use this to factor a C130 with a custom parameters file:

./cado-nfs.py --parameters ./parameters/factor/params.c130 C130

[charybdis]

Originally Posted by bur

I read the README and don't quite comprehend why it makes sense to have servers and clients running on the local machine. This example

Run a full factorization on the local machine, using 8 threads for the
server (this includes the linear algebra), and 4 jobs of 2 threads
each for the polynomial selection and the sieving

makes it seem to me that the server does the LA and the clients the poly-selection and sieving?

But if it's a single machine why not use all cores for poly and sieving and afterwards for the LA?

This would be useful if, say, your machine has 16 threads and you only want to use 8 of them for CADO. But then you could use -t 8 and I'm not sure what the difference is.

Or put simple: if I have no intention on running CADO distributed, do I need to concern myself with server & clients? Otherwise I'd just use this to factor a C130 with a custom parameters file:

./cado-nfs.py --parameters ./parameters/factor/params.c130 C130

If you want to use all threads on your machine, then this is fine. I'd give you one extra piece of advice, which is to create a directory to store the data and logs for each factorization you run and add workdir=/my/c130/directory to the command line. Otherwise the relations and logs end up in a temporary directory that gets wiped when the job finishes, and if your machine happens to reboot then you've lost the factors.

[bur]

Thanks, that's helpful advice.

[bur]

I started CADO on a C112 and after this output nothing happens anymore. top shows no activity:

$ ./cado-nfs.py --parameters ./parameters/factor/vbc/params.c110.txt --workdir=./factorization/C112 2493852636725492741019529213832529872212666052422896535749837232438979207048925012754355124942131700804532974163
Info:root: No database exists yet
Info:Database: Opened connection to database factorization/C112/c110.db
Info:root: Set tasks.linalg.bwc.threads=10 based on detected physical cores
Info:root: Set tasks.threads=20 based on detected logical cpus
Info:root: tasks.threads = 20 [via tasks.threads]
Info:root: tasks.polyselect.threads = 2 [via tasks.polyselect.threads]
Info:root: tasks.sieve.las.threads = 2 [via tasks.sieve.las.threads]
Info:root: tasks.linalg.bwc.threads = 10 [via tasks.linalg.bwc.threads]
Info:root: tasks.sqrt.threads = 8 [via tasks.sqrt.threads]
Info:root: Command line parameters: ./cado-nfs.py --parameters ./parameters/factor/vbc/params.c110.txt --workdir=./factorization/C112 2493852636725492741019529213832529872212666052422896535749837232438979207048925012754355124942131700804532974163
Info:root: If this computation gets interrupted, it can be resumed with ./cado-nfs.py ./factorization/C112/c110.parameters_snapshot.0
Info:HTTP server: Using non-threaded HTTPS server
Info:Lattice Sieving: param rels_wanted is 9200000
Warning:Complete Factorization / Discrete logarithm: Parameter tasks.filter.maxlevel = 21 was not used anywhere
Info:Complete Factorization / Discrete logarithm: Factoring 2493852636725492741019529213832529872212666052422896535749837232438979207048925012754355124942131700804532974163
Info:HTTP server: serving at https://florian-Precision-3640-Tower:45909 (0.0.0.0)
Info:HTTP server: For debugging purposes, the URL above can be accessed if the server.only_registered=False parameter is added
Info:HTTP server: You can start additional cado-nfs-client.py scripts with parameters: --server=https://florian-Precision-3640-Tower:45909 --certsha1=4db43b1c5fb238c8f6fe14fa5bdbb123511ffce9
Info:HTTP server: If you want to start additional clients, remember to add their hosts to server.whitelist
Info:Polynomial Selection (size optimized): Starting
Info:Polynomial Selection (size optimized): 0 polynomials in queue from previous run
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_0-840 to database
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_840-1680 to database
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_1680-2520 to database
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_2520-3360 to database
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_3360-4200 to database
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_4200-5040 to database
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_5040-5880 to database
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_5880-6720 to database
Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_6720-7560 to database
 Info:Polynomial Selection (size optimized): Adding workunit c110_polyselect1_7560-8400 to database

And is it possible to make cado more verbose to screen output? -v doesn't seem to be recognized. It'd be nice to have some idea of the progress so far such as yafu/msieve does with the number of relations needed and found.

[VBCurtis]

Looks like no clients ran. I don't have experience with using params files in non-default locations, but you can try adding --tasks.slaves.nrclients=10 to the command line and see if that triggers your setup to fire 10 (two-threaded by default) client processes for poly select and sieving.

Both phases will show progress in much detail on-screen, when they're actually running.

You might try invoking cado with no flags at all, to see if that works- that would help us figure out why the custom params file isn't triggering client startup.

[charybdis]

I think this could be a manifestation of this bug that was introduced by revisions 0cb1e01c and f15ee502 in November. It caused clients to die immediately and silently with certain versions of Python, and I don't think it was ever fully fixed, because I kept getting the bug with Python 3.5.2 even after the supposed fix. Never got round to talking to the developers about it.

Try building an old revision - perhaps 6605b6e4, the last one before the bug was introduced - and see if you still have problems.