mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > CADO-NFS

Reply
 
Thread Tools
Old 2020-04-01, 17:03   #463
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

344910 Posts
Default

I have run into a problem that I already have here, but work around:

When I run an snfs job, the server doesn't start any local sievers. Here, I separately start some on the server machine, but I can't figure out how to do so on my Colab session.

-------
When you use remdups, do you decompress the *.gz files first? When I tried to cat all the *.gz files and send them through remdups4, I ended up with 1 good relation and a couple hundred million bad relations.
EdH is offline   Reply With Quote
Old 2020-04-01, 17:53   #464
Xyzzy
 
Xyzzy's Avatar
 
"Mike"
Aug 2002

3×19×137 Posts
Default

zcat relations.dat.gz | ./remdups4 10000 > relations.dat
Xyzzy is offline   Reply With Quote
Old 2020-04-01, 18:33   #465
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

65718 Posts
Default

Thanks, Xyzzy!

So, they must be decompressed first. I've been "cat"ing the .gz files into a composite for msieve, which has no trouble with the relations being compressed. I will play with this for a bit and see whether it makes more sense for me to decompress and use remdups or simply let msieve do its "thing."
EdH is offline   Reply With Quote
Old 2020-04-01, 19:23   #466
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3,449 Posts
Default

I got back 31k relations from my first sieve run in my Colab instance and was able to get those added to the msieve parallel run. Hopefully, that will help with the matrix build.

However, I was too late to try to add the relations to the CADO-NFS run, as it had already declared the move into "Filtering - Merging: Starting."

I wonder if the associated *.stderr0 file for the relations might allow them to be added to the server run, or if just existing within the directory might allow them to be added? Something for another experiment on another day. . .
EdH is offline   Reply With Quote
Old 2020-04-02, 13:34   #467
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3,449 Posts
Default

Well, . . . My server finally declared it had enough relations at 274402507 and told all the clients to, "Knock it off!"

Unfortunately, it then proceeded to get memory stuck using 99.8% of 15.6 GiB RAM and 50% of 8.0 GiB Swap during the merge. I'm leaving it sit out of curiosity for the time being.

Disappointingly, my msieve parallel (to the CADO-NFS) process didn't think the relations sufficient to build a matrix:
Code:
**remdups4 was run prior**
Wed Apr  1 20:03:57 2020  found 6580868 hash collisions in 172527688 relations
Wed Apr  1 20:04:05 2020  added 3657741 free relations
Wed Apr  1 20:04:05 2020  commencing duplicate removal, pass 2
Wed Apr  1 20:09:23 2020  found 0 duplicates and 176185429 unique relations
Wed Apr  1 20:09:23 2020  memory use: 506.4 MB
Wed Apr  1 20:09:23 2020  reading ideals above 81068032
Wed Apr  1 20:09:23 2020  commencing singleton removal, initial pass
Wed Apr  1 20:22:05 2020  memory use: 5512.0 MB
Wed Apr  1 20:22:05 2020  reading all ideals from disk
Wed Apr  1 20:22:44 2020  memory use: 3903.9 MB
Wed Apr  1 20:22:52 2020  commencing in-memory singleton removal
Wed Apr  1 20:23:00 2020  begin with 176185429 relations and 182285468 unique ideals
Wed Apr  1 20:24:36 2020  reduce to 77811022 relations and 69781418 ideals in 21 passes
Wed Apr  1 20:24:36 2020  max relations containing the same ideal: 32
Wed Apr  1 20:24:41 2020  reading ideals above 720000
Wed Apr  1 20:24:41 2020  commencing singleton removal, initial pass
Wed Apr  1 20:32:30 2020  memory use: 1506.0 MB
Wed Apr  1 20:32:30 2020  reading all ideals from disk
Wed Apr  1 20:32:57 2020  memory use: 3071.3 MB
Wed Apr  1 20:33:06 2020  keeping 78830602 ideals with weight <= 200, target excess is 406990
Wed Apr  1 20:33:14 2020  commencing in-memory singleton removal
Wed Apr  1 20:33:21 2020  begin with 77811022 relations and 78830602 unique ideals
Wed Apr  1 20:34:02 2020  reduce to 77808983 relations and 78828557 ideals in 6 passes
Wed Apr  1 20:34:02 2020  max relations containing the same ideal: 200
Wed Apr  1 20:34:08 2020  filtering wants 1000000 more relations
I'm currently doing some more sieving in an attempt to convince msieve it should take on the LA for this project.

On a positive note, I was successful in running the 100k-150k area via my Colab instance and retrieved around 150k relations from that experiment.
EdH is offline   Reply With Quote
Old 2020-04-02, 14:29   #468
RichD
 
RichD's Avatar
 
Sep 2008
Kansas

2×7×227 Posts
Default

From my limited experience, you will need about 180M (or greater) unique relations for a 31-bit job. The sweet spot would be around 200M if you want to build a matrix at TD=120-130.
RichD is offline   Reply With Quote
Old 2020-04-02, 16:29   #469
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3,449 Posts
Default

Thanks richD. I became impatient with my server that was stuck, so I restarted it to gather more relations for my msieve run. This was done partly because I can't get any of my other machines to accept clients when I run a CADO-NFS server. I use the same setup as my current server (excepting, of course, IPs), but cannot get clients running.
EdH is offline   Reply With Quote
Old 2020-04-07, 11:42   #470
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

D7916 Posts
Default

Quote:
Originally Posted by RichD View Post
From my limited experience, you will need about 180M (or greater) unique relations for a 31-bit job. The sweet spot would be around 200M if you want to build a matrix at TD=120-130.
Not sure if you're following my current thread on this run (in my blog area), but you were right on with 180M:
Code:
Thu Apr  2 21:15:02 2020  found 0 duplicates and 181713072 unique relations
EdH is offline   Reply With Quote
Old 2020-04-10, 16:31   #471
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

65718 Posts
Default

I'm working on a Colab project with CADO-NFS and have a couple questions that I will eventually discover the answers to, if I don't get them here, but I thought I'd try here first, and possibly save some work:

1. Is the siever (las) a complete stand-alone program which can run on its own without any of the rest of the CADO-NFS package?

2. Is there a method of compiling las without building the entire CADO-NFS package?

3. Is there any advantage to recompiling las for different Xeon CPUs across different Colab sessions?

Thanks for any thoughts. . .
EdH is offline   Reply With Quote
Old 2020-04-10, 16:49   #472
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

10001100001002 Posts
Default

1. Definitely.
2. Yes, but I don't know how. The documentation refers to the possibility of compiling just las for windows, even though the overall package doesn't compile on windows.
3. Probably? There are massive speed differences among architectures in las, and I don't imagine the range of possible architecture-optimizations exist within each compiled binary; seems particularly unlikely since there is no officially-released binary, rather each one is self-compiled by users by design. This is just a guess based on function, rather than from any personal experience I have with the code.
VBCurtis is online now   Reply With Quote
Old 2020-04-16, 03:13   #473
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3,449 Posts
Default

Well, I'm becoming "annoyed!" I would probably have been through with sieving if I hadn't had to restart the server 8 times to get it to assign WUs. I was sure I'm using the install that never gave this trouble in the past. Is it possible it has something to do with the size of the composite/corresponding params?
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
CADO-NFS on windows jux CADO-NFS 22 2019-11-12 12:08
CADO help henryzz CADO-NFS 4 2017-11-20 15:14
CADO and WinBlows akruppa Programming 22 2015-12-31 08:37
CADO-NFS skan Information & Answers 1 2013-10-22 07:00
CADO R.D. Silverman Factoring 4 2008-11-06 12:35

All times are UTC. The time now is 20:59.

Fri Nov 27 20:59:54 UTC 2020 up 78 days, 18:10, 3 users, load averages: 1.39, 1.29, 1.29

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.