CADO NFS

[VBCurtis]

Originally Posted by EdH

Total wall time:

22:21:21

if my clock math is correct. I think cado-nfs came in at:

27:06:32

Your CADO-data notes wall-clock time for the matrix at ~59000 seconds, on the order of 19 hr. That means poly select and sieving ran in just 8 hours!??

GGNFS/msieve took 14 hrs for poly select/sieving/filtering. So, outside of LA, it appears CADO is over 50% faster than GGNFS at this size!
One reason to note "outside of LA" is that it's possible in principle to use CADO for poly and sieve with msieve for LA, as you noted a couple posts up. A more important reason is that for a normal person with just one machine doing a job, LA time won't dominate the wall-clock measurement the way it does with your farm. If it takes me 80hr rather than 140 to sieve a number, I don't mind that LA takes 19 hr rather than 8!!

[EdH]

Originally Posted by VBCurtis

Your CADO-data notes wall-clock time for the matrix at ~59000 seconds, on the order of 19 hr. That means poly select and sieving ran in just 8 hours!??

I've been watching that, specifically on the fact that msieve/ggnfs is maxed with this C155 for the sieve time available for some of my machines. CADO-NFS, OTOH, has a bit more time available for sieving.

Originally Posted by VBCurtis

GGNFS/msieve took 14 hrs for poly select/sieving/filtering. So, outside of LA, it appears CADO is over 50% faster than GGNFS at this size!
One reason to note "outside of LA" is that it's possible in principle to use CADO for poly and sieve with msieve for LA, as you noted a couple posts up.

I plan to experiment in this area. I need to find a way to automate the changeover. I'll have to work with some smaller composites before I can try one of these larger ones for another comparison.

Originally Posted by VBCurtis

A more important reason is that for a normal person with just one machine doing a job, LA time won't dominate the wall-clock measurement the way it does with your farm. If it takes me 80hr rather than 140 to sieve a number, I don't mind that LA takes 19 hr rather than 8!!

Quite true. My farm is bottlenecked at LA. Less machines would swing the run into a different light. Something else to note, is that I'm seeing a lot of idle time on my clients while the final poly selection is made. This would not occur on a sole machine.

You've modified my understanding a bit, in that due to the ratio of multi-machine to single machine processes, there is more difference based on the makeup of the overall system of machines than I realized.

[VBCurtis]

Originally Posted by EdH

I've been watching that, specifically on the fact that msieve/ggnfs is maxed with this C155 for the sieve time available for some of my machines. CADO-NFS, OTOH, has a bit more time available for sieving.

Note that CADO's server setup is designed for clients to appear and disappear at will; you can shut down the client-farm overnight while the machine with the CADO server sieves a bit, and in the morning reconnect the farm/CADO clients without any hiccups.
If C155 took ~8 hrs of the 14 available, you can do ~C160 worth of sieving each day (since poly select only happens the first day), so a C170 would take about 5 daytime-days of sieve-farming (plus a lengthy matrix, of course).
Edit: For that matter, you can shut down the CADO server, too- you'd just need to restart it using the snapshot invocation rather than the usual "start this job" command.

[henryzz]

Originally Posted by VBCurtis

Note that CADO's server setup is designed for clients to appear and disappear at will; you can shut down the client-farm overnight while the machine with the CADO server sieves a bit, and in the morning reconnect the farm/CADO clients without any hiccups.
If C155 took ~8 hrs of the 14 available, you can do ~C160 worth of sieving each day (since poly select only happens the first day), so a C170 would take about 5 daytime-days of sieve-farming (plus a lengthy matrix, of course).
Edit: For that matter, you can shut down the CADO server, too- you'd just need to restart it using the snapshot invocation rather than the usual "start this job" command.

It would be possible to go further than that. There is no reason why someone on the forum couldn't host a server for all of us to connect to. Once it becomes possible to do a very large factorization in a similar or better time than ggnfs we should probably start doing that as a forum.

[EdH]

Originally Posted by VBCurtis

Note that CADO's server setup is designed for clients to appear and disappear at will; you can shut down the client-farm overnight while the machine with the CADO server sieves a bit, and in the morning reconnect the farm/CADO clients without any hiccups.
If C155 took ~8 hrs of the 14 available, you can do ~C160 worth of sieving each day (since poly select only happens the first day), so a C170 would take about 5 daytime-days of sieve-farming (plus a lengthy matrix, of course).
Edit: For that matter, you can shut down the CADO server, too- you'd just need to restart it using the snapshot invocation rather than the usual "start this job" command.

I knew I could remove and restart clients and in my case, I can interrupt them for ggnfs, too, since I only suspend the clients. In the case of ggnfs, the clients will try to complete their range even after the server is running LA. I do end up stopping manually the clients that continue on. I was looking at this in a pure running sense for these experiments. I would consider a break of anything other than minor as a loss of validity. Maybe that's not correct since times are computed, at least for CADO-NFS.

I have seen the snapshot info scroll by and expected to be able to use it, but have not looked to see if it is in the temp folder. The temp folder seems to disappear once the factorization is complete. Is it in a log where I can find it or do I need to capture it before it scrolls out of range in the terminal? (I can look for it myself during my next runs)

Originally Posted by henryzz

It would be possible to go further than that. There is no reason why someone on the forum couldn't host a server for all of us to connect to. Once it becomes possible to do a very large factorization in a similar or better time than ggnfs we should probably start doing that as a forum.

Team Aliquot projects could take on a different aspect. That could be done now, but for the LA portion. If the LA is able to be done by msieve with a savings in time, it would already be feasible.

[VBCurtis]

Originally Posted by EdH

I have seen the snapshot info scroll by and expected to be able to use it, but have not looked to see if it is in the temp folder. The temp folder seems to disappear once the factorization is complete. Is it in a log where I can find it or do I need to capture it before it scrolls out of range in the terminal? (I can look for it myself during my next runs)

Team Aliquot projects could take on a different aspect. That could be done now, but for the LA portion. If the LA is able to be done by msieve with a savings in time, it would already be feasible.

The snapshot file is indeed in the temp folder, which does not disappear if you close CADO server while it's running. The file is named something obvious, like jobname.snapshot.0. When you know the name, invocation from the CADO folder points to that file rather than the local params file. You may also edit the params in the snapshot if you wish to change something during a job (say, changing tasks.qrange or adding more poly select by changing admax). Close CADO-server, edit the snapshot file, start server with "./cado-nfs.py /tmp/foldername/snapshotfilename"

When I was running a bunch of CADO tests on a multi-machine setup, I'd run the server with zero threads for sieving; when one job went to LA (taking up all CPUs), I'd fire up another CADO instance using zero sieve threads for the clients to connect to. My clients were roughly fast enough to finish sieving one problem in about the time the server finished LA on the previous one.

[RichD]

It seems between henryzz and VBCurtis comments there could be a dedicated server for dishing out work and performing post-processing for Aliquot Sequences. Would this extend the reach of yafu@home into the C140s & C150s or are we talking bigger numbers? Who would do the ECM work prior to submitting the number?

[VBCurtis]

I have a 32GB server with a public-facing internet connection that runs CADO jobs already; RichD connected to it for a past job as proof-of-concept for a shared CADO effort.

If there is interest in doing group-testing for CADO in 160-180 digit range, I'm game for selecting GNFS tasks from whatever project and firing up the CADO-server. Ed seems to have sub-160 covered in just a day with his own farm, so group work doesn't need to extend below 160.
I'm running a C149 on a single machine presently to get timing info on a non-farm environment; sieving is ETA'ed at just over 5 days on half of a 12-core Sandy Bridge (HP Z600).

Ideally, we find a pair of similar-sized numbers so I can adjust params after run 1 to try to improve performance in run 2; if we get good results, I can send the params to the CADO group for inclusion in their git release.

Edit: I am more interested in testing to improve params, but I'm game to do so on a bunch of C150s for aliquot work with my setup. I'll have time for the manual job-entry part on smaller jobs once the term ends early June.

[VBCurtis]

Originally Posted by VBCurtis

I'm running a C149 on a single machine presently to get timing info on a non-farm environment; sieving is ETA'ed at just over 5 days on half of a 12-core Sandy Bridge (HP Z600).

This seemed high, so I did a little investigating and learned I didn't set qmin (the starting Q for sieving). CADO chose something near the midpoint of alim and rlim as default, and who knew that sieving 24M-42M would be less efficient than 4M-20M?
I restarted the job.

[RichD]

Originally Posted by VBCurtis

... and who knew that sieving 24M-42M would be less efficient than 4M-20M?.

I assume that was a joke.

[Dubslow]

Originally Posted by RichD

I assume that was a joke.

Well I think it used to be not a joke until he figured out recently that CADO siever is way more capable than GGNFS! :D