Okay, I think you'll probably get faster/shorter times with the parameter files from 3.0 .

Speaking of the parameter file:
How do the
tasks.filter.target_density = 160
and
tasks.filter.purge.keep =160
Work? Do they correspond to Target density and excess in Msieve?

Yes, to target density; though it is measured differently. I haven't gotten too firm an estimate, but density 140 in CADO is around TD 80 in msieve (I think). The relationship should be constant-difference or nearly so, so 180 in CADO would be a rather high density.

Filtering large sparse matrix using CADO
 

have a sparse matrix with 5221950 rows and 4222046 columns in .txt format. I wish to filter this matrix to make it dense. What is the CADO code for doing it.

Both the CADO and msieve tools are designed to solve sparse matrices. What density is your matrix, and how dense do you want it to be? Why not just solve it as it is? If your matrix is of the typical density from an NFS job, you're looking at under a day to solve it as-is.

Secondly, filtering is a procedure applied to NFS relations, not to a matrix. Matrix-building happens after the filtering step.

[QUOTE=VBCurtis;491380]Both the CADO and msieve tools are designed to solve sparse matrices. What density is your matrix, and how dense do you want it to be? Why not just solve it as it is? If your matrix is of the typical density from an NFS job, you're looking at under a day to solve it as-is.

Secondly, filtering is a procedure applied to NFS relations, not to a matrix. Matrix-building happens after the filtering step.[/QUOTE]
I tried to solve the matrix as it is. It requires a huge amount of memory for doing so.

The computation started on July 8. It says krylov steps will be done by July 28. Today is july 10 and already 600 GB of memory has been used up.

We have limited computational resources. So providing this huge memory may be difficult.
Actually, I wanted to run Block Wiedemann to the matrix.
I have run the code below.
"software/cado-nfs/build/magma4/linalg/bwc/bwc.pl :complete thr=90 m=4 n=2 nullspace=right matrix=/user1/res/asu/madhurima_r/software/cado-nfs//sparse-folder22bit/fullmat.sparse.bin"
Can something be done so that I am able to run Block Wiedemann on the matrix, in such a way that filtering is done automatically and so much memory is not needed.

[QUOTE=madhurima;491395]I tried to solve the matrix as it is. It requires a huge amount of memory for doing so.

The computation started on July 8. It says krylov steps will be done by July 28. Today is july 10 and already 600 GB of memory has been used up. [/QUOTE]
Solving a GNFS 5.2M matrix shouldn't take that long and not use that much memory.

For instance a GNFS160 with Msieve uses less than 3GB and takes less than a day to solve on a quadcore.

[code]
Wed Oct 28 16:15:20 2015 matrix is 5796592 x 5796817 (2608.9 MB) with weight 698630622 (120.52/col)
Wed Oct 28 16:15:20 2015 sparse part has weight 625952164 (107.98/col)
Wed Oct 28 16:15:20 2015 using block size 8192 and superblock size 786432 for processor cache size 8192 kB
Wed Oct 28 16:15:53 2015 commencing Lanczos iteration (4 threads)
Wed Oct 28 16:16:15 2015 linear algebra at 0.0%, ETA 22h17m[/code][quote]
Can something be done so that I am able to run Block Wiedemann on the matrix, in such a way that filtering is done automatically and so much memory is not needed.[/quote]CADO-NFS or Msieve should be able to do the filtering of the relations and produce a solvable matrix. I would rerun the filtering step, maybe an error occurred which resulted in a unsolvable matrix.

VBCurtis, we talked over in the HP49 thread about timing information from jobs of 160+ digits.

My factorization of the 181-digit N at [url]http://stdkmd.com/nrr/cont/3/37771.htm#N195_C181[/url] just completed; the factors were a p78 and p103.

[QUOTE]Just need wall-clock time for poly select, sieving, and LA; and the CPU specs for the machine used.
[/QUOTE]

You can see the final section of the CADO-NFS logfile at the above url; if I am reading correctly, WCT for the polyselect, sieving and LA steps were 9.15021e+06s, 4.30315e+07s, and 2.03259e+06s respectively. (If I have those wrong or you'd like to see the full logfile I'm happy to provide it.)

The execution environment was six machines as follows with 64GB RAM each, with just one machine running the non-distributed portion.

[CODE]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 1
Model name: AMD Ryzen 7 1700X Eight-Core Processor
Stepping: 1
CPU MHz: 2200.000
CPU max MHz: 3400.0000
CPU min MHz: 2200.0000
BogoMIPS: 6799.72
Virtualization: AMD-V
L1d cache: 32K
L1i cache: 64K
L2 cache: 512K
L3 cache: 8192K
NUMA node0 CPU(s): 0-15
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 mwaitx hw_pstate vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf ibpb arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic overflow_recov succor smca
[/CODE]

I used the default c180 parameters from git head.

I will say that keeping this factorization running smoothly was a little tricky thanks to several of the machines locking up and needing reboots; this happened enough that eventually the factorization stopped completely due to too many failures.

[QUOTE=Nooks;492122]
You can see the final section of the CADO-NFS logfile at the above url; if I am reading correctly, WCT for the polyselect, sieving and LA steps were 9.15021e+06s, 4.30315e+07s, and 2.03259e+06s respectively.
[/CODE]

I used the default c180 parameters from git head.

I will say that keeping this factorization running smoothly was a little tricky thanks to several of the machines locking up and needing reboots; this happened enough that eventually the factorization stopped completely due to too many failures.[/QUOTE]
Your timing report suggests that CADO only tracked the sieve effort after your last reboot. I don't think a C180 can be sieved in 43 megaCPUseconds! CADO reports CPU time rather than wall clock time for poly select and sieving; for LA, both CPU and wall clock time are reported.
It doesn't make sense for 9M sec poly select, 43M secs sieve, 20M sec LA; so I think sieve time was only the last invocation after a restart.
Thanks for the data, all the same! If you do another 180-class CADO, let me know and I can try to improve the default parameters; I've saved 30-50% CPU time over default in previous 145-155 digit efforts (see above in this thread). My next personal job is a GNFS180 with CADO and my own parameters.

[QUOTE=VBCurtis;492127]Thanks for the data, all the same! If you do another 180-class CADO, let me know and I can try to improve the default parameters; I've saved 30-50% CPU time over default in previous 145-155 digit efforts (see above in this thread). My next personal job is a GNFS180 with CADO and my own parameters.[/QUOTE]

I have a c185 ([url]http://stdkmd.com/nrr/c.cgi?q=37771_258[/url]) 2/3 of the way through polyselect now; I guess we can check back in a few months.

I have to say that I have found it challenging to not have to stop the CADO server from time to time to recover from client failures; the machine running the server last restarted on June 15 --- I've had to issue eleven resets for 6 machines so far; I'm starting to suspect that they've been crashing under memory pressure, but I'm not sure.

I do have a complete CADO logfile and all the data files produced by the 37771_195 factorization if useful estimates of actual CPU time used could be collected.

[QUOTE=Nooks;492122]VBCurtis, we talked over in the HP49 thread about timing information from jobs of 160+ digits.

My factorization of the 181-digit N at [url]http://stdkmd.com/nrr/cont/3/37771.htm#N195_C181[/url] just completed; the factors were a p78 and p103.
[/QUOTE]

[QUOTE=VBCurtis;492127]Your timing report suggests that CADO only tracked the sieve effort after your last reboot. I don't think a C180 can be sieved in 43 megaCPUseconds! CADO reports CPU time rather than wall clock time for poly select and sieving; for LA, both CPU and wall clock time are reported.[/QUOTE]

We are speaking about a factorization of a factor of N=34*10^195-61. What is a pretty nice totally trivial snfs number. What an overshoot doing this with gnfs..... Personally I don't recommend everybody Maths and programming.

Apologies, I could have mentioned that this is a number better suited to SNFS.

I have two more in the pipeline for GFNS treatment before committing to learning how to import Kamada's SNFS polynomials.