mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > CADO-NFS

Reply
 
Thread Tools
Old 2018-07-03, 17:31   #276
VictordeHolland
 
VictordeHolland's Avatar
 
"Victor de Hollander"
Aug 2011
the Netherlands

49616 Posts
Default

Okay, I think you'll probably get faster/shorter times with the parameter files from 3.0 .

Speaking of the parameter file:
How do the
tasks.filter.target_density = 160
and
tasks.filter.purge.keep =160
Work? Do they correspond to Target density and excess in Msieve?
VictordeHolland is offline   Reply With Quote
Old 2018-07-03, 20:30   #277
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

23·32·61 Posts
Default

Yes, to target density; though it is measured differently. I haven't gotten too firm an estimate, but density 140 in CADO is around TD 80 in msieve (I think). The relationship should be constant-difference or nearly so, so 180 in CADO would be a rather high density.
VBCurtis is offline   Reply With Quote
Old 2018-07-09, 19:11   #278
madhurima
 
Jul 2018

2 Posts
Default Filtering large sparse matrix using CADO

have a sparse matrix with 5221950 rows and 4222046 columns in .txt format. I wish to filter this matrix to make it dense. What is the CADO code for doing it.
madhurima is offline   Reply With Quote
Old 2018-07-10, 01:09   #279
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

23·32·61 Posts
Default

Both the CADO and msieve tools are designed to solve sparse matrices. What density is your matrix, and how dense do you want it to be? Why not just solve it as it is? If your matrix is of the typical density from an NFS job, you're looking at under a day to solve it as-is.

Secondly, filtering is a procedure applied to NFS relations, not to a matrix. Matrix-building happens after the filtering step.
VBCurtis is offline   Reply With Quote
Old 2018-07-10, 07:51   #280
madhurima
 
Jul 2018

2 Posts
Default

Quote:
Originally Posted by VBCurtis View Post
Both the CADO and msieve tools are designed to solve sparse matrices. What density is your matrix, and how dense do you want it to be? Why not just solve it as it is? If your matrix is of the typical density from an NFS job, you're looking at under a day to solve it as-is.

Secondly, filtering is a procedure applied to NFS relations, not to a matrix. Matrix-building happens after the filtering step.
I tried to solve the matrix as it is. It requires a huge amount of memory for doing so.

The computation started on July 8. It says krylov steps will be done by July 28. Today is july 10 and already 600 GB of memory has been used up.

We have limited computational resources. So providing this huge memory may be difficult.
Actually, I wanted to run Block Wiedemann to the matrix.
I have run the code below.
"software/cado-nfs/build/magma4/linalg/bwc/bwc.pl :complete thr=90 m=4 n=2 nullspace=right matrix=/user1/res/asu/madhurima_r/software/cado-nfs//sparse-folder22bit/fullmat.sparse.bin"
Can something be done so that I am able to run Block Wiedemann on the matrix, in such a way that filtering is done automatically and so much memory is not needed.
madhurima is offline   Reply With Quote
Old 2018-07-10, 20:07   #281
VictordeHolland
 
VictordeHolland's Avatar
 
"Victor de Hollander"
Aug 2011
the Netherlands

100100101102 Posts
Default

Quote:
Originally Posted by madhurima View Post
I tried to solve the matrix as it is. It requires a huge amount of memory for doing so.

The computation started on July 8. It says krylov steps will be done by July 28. Today is july 10 and already 600 GB of memory has been used up.
Solving a GNFS 5.2M matrix shouldn't take that long and not use that much memory.

For instance a GNFS160 with Msieve uses less than 3GB and takes less than a day to solve on a quadcore.

Code:
Wed Oct 28 16:15:20 2015  matrix is 5796592 x 5796817 (2608.9 MB) with weight 698630622 (120.52/col)
Wed Oct 28 16:15:20 2015  sparse part has weight 625952164 (107.98/col)
Wed Oct 28 16:15:20 2015  using block size 8192 and superblock size 786432 for processor cache size 8192 kB
Wed Oct 28 16:15:53 2015  commencing Lanczos iteration (4 threads)
Wed Oct 28 16:16:15 2015  linear algebra at 0.0%, ETA 22h17m
Quote:
Can something be done so that I am able to run Block Wiedemann on the matrix, in such a way that filtering is done automatically and so much memory is not needed.
CADO-NFS or Msieve should be able to do the filtering of the relations and produce a solvable matrix. I would rerun the filtering step, maybe an error occurred which resulted in a unsolvable matrix.
VictordeHolland is offline   Reply With Quote
Old 2018-07-19, 14:40   #282
Nooks
 
Jul 2018

19 Posts
Default

VBCurtis, we talked over in the HP49 thread about timing information from jobs of 160+ digits.

My factorization of the 181-digit N at http://stdkmd.com/nrr/cont/3/37771.htm#N195_C181 just completed; the factors were a p78 and p103.

Quote:
Just need wall-clock time for poly select, sieving, and LA; and the CPU specs for the machine used.
You can see the final section of the CADO-NFS logfile at the above url; if I am reading correctly, WCT for the polyselect, sieving and LA steps were 9.15021e+06s, 4.30315e+07s, and 2.03259e+06s respectively. (If I have those wrong or you'd like to see the full logfile I'm happy to provide it.)

The execution environment was six machines as follows with 64GB RAM each, with just one machine running the non-distributed portion.

Code:
$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             AuthenticAMD
CPU family:            23
Model:                 1
Model name:            AMD Ryzen 7 1700X Eight-Core Processor
Stepping:              1
CPU MHz:               2200.000
CPU max MHz:           3400.0000
CPU min MHz:           2200.0000
BogoMIPS:              6799.72
Virtualization:        AMD-V
L1d cache:             32K
L1i cache:             64K
L2 cache:              512K
L3 cache:              8192K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 mwaitx hw_pstate vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf ibpb arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic overflow_recov succor smca
I used the default c180 parameters from git head.

I will say that keeping this factorization running smoothly was a little tricky thanks to several of the machines locking up and needing reboots; this happened enough that eventually the factorization stopped completely due to too many failures.
Nooks is offline   Reply With Quote
Old 2018-07-19, 15:39   #283
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

23×32×61 Posts
Default

Quote:
Originally Posted by Nooks View Post
You can see the final section of the CADO-NFS logfile at the above url; if I am reading correctly, WCT for the polyselect, sieving and LA steps were 9.15021e+06s, 4.30315e+07s, and 2.03259e+06s respectively.
[/CODE]

I used the default c180 parameters from git head.

I will say that keeping this factorization running smoothly was a little tricky thanks to several of the machines locking up and needing reboots; this happened enough that eventually the factorization stopped completely due to too many failures.
Your timing report suggests that CADO only tracked the sieve effort after your last reboot. I don't think a C180 can be sieved in 43 megaCPUseconds! CADO reports CPU time rather than wall clock time for poly select and sieving; for LA, both CPU and wall clock time are reported.
It doesn't make sense for 9M sec poly select, 43M secs sieve, 20M sec LA; so I think sieve time was only the last invocation after a restart.
Thanks for the data, all the same! If you do another 180-class CADO, let me know and I can try to improve the default parameters; I've saved 30-50% CPU time over default in previous 145-155 digit efforts (see above in this thread). My next personal job is a GNFS180 with CADO and my own parameters.
VBCurtis is offline   Reply With Quote
Old 2018-07-19, 16:23   #284
Nooks
 
Jul 2018

19 Posts
Default

Quote:
Originally Posted by VBCurtis View Post
Thanks for the data, all the same! If you do another 180-class CADO, let me know and I can try to improve the default parameters; I've saved 30-50% CPU time over default in previous 145-155 digit efforts (see above in this thread). My next personal job is a GNFS180 with CADO and my own parameters.
I have a c185 (http://stdkmd.com/nrr/c.cgi?q=37771_258) 2/3 of the way through polyselect now; I guess we can check back in a few months.

I have to say that I have found it challenging to not have to stop the CADO server from time to time to recover from client failures; the machine running the server last restarted on June 15 --- I've had to issue eleven resets for 6 machines so far; I'm starting to suspect that they've been crashing under memory pressure, but I'm not sure.

I do have a complete CADO logfile and all the data files produced by the 37771_195 factorization if useful estimates of actual CPU time used could be collected.
Nooks is offline   Reply With Quote
Old 2018-07-19, 16:38   #285
R. Gerbicz
 
R. Gerbicz's Avatar
 
"Robert Gerbicz"
Oct 2005
Hungary

2·3·5·47 Posts
Default

Quote:
Originally Posted by Nooks View Post
VBCurtis, we talked over in the HP49 thread about timing information from jobs of 160+ digits.

My factorization of the 181-digit N at http://stdkmd.com/nrr/cont/3/37771.htm#N195_C181 just completed; the factors were a p78 and p103.
Quote:
Originally Posted by VBCurtis View Post
Your timing report suggests that CADO only tracked the sieve effort after your last reboot. I don't think a C180 can be sieved in 43 megaCPUseconds! CADO reports CPU time rather than wall clock time for poly select and sieving; for LA, both CPU and wall clock time are reported.
We are speaking about a factorization of a factor of N=34*10^195-61. What is a pretty nice totally trivial snfs number. What an overshoot doing this with gnfs..... Personally I don't recommend everybody Maths and programming.
R. Gerbicz is offline   Reply With Quote
Old 2018-07-19, 17:32   #286
Nooks
 
Jul 2018

19 Posts
Default

Apologies, I could have mentioned that this is a number better suited to SNFS.

I have two more in the pipeline for GFNS treatment before committing to learning how to import Kamada's SNFS polynomials.
Nooks is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
CADO-NFS on windows jux CADO-NFS 22 2019-11-12 12:08
CADO help henryzz CADO-NFS 4 2017-11-20 15:14
CADO and WinBlows akruppa Programming 22 2015-12-31 08:37
CADO-NFS skan Information & Answers 1 2013-10-22 07:00
CADO R.D. Silverman Factoring 4 2008-11-06 12:35

All times are UTC. The time now is 12:47.

Sat Oct 24 12:47:25 UTC 2020 up 44 days, 9:58, 0 users, load averages: 1.19, 1.42, 1.45

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.