Pascal's OPN roadblock files - Page 80

charybdis · 2021-01-11, 17:58

Quote:

Originally Posted by ryanp

Wedged again for about 12 hours so far:

Code:

Sun Jan 10 19:56:49 2021  commencing in-memory singleton removal
Sun Jan 10 19:58:41 2021  begin with 621912609 relations and 624555581 unique ideals
Sun Jan 10 20:33:55 2021  reduce to 616867386 relations and 611731201 ideals in 17 passes
Sun Jan 10 20:33:55 2021  max relations containing the same ideal: 167
Sun Jan 10 20:37:11 2021  relations with 0 large ideals: 289519
Sun Jan 10 20:37:11 2021  relations with 1 large ideals: 190512
Sun Jan 10 20:37:11 2021  relations with 2 large ideals: 1663725
Sun Jan 10 20:37:11 2021  relations with 3 large ideals: 9813804
Sun Jan 10 20:37:11 2021  relations with 4 large ideals: 36668291
Sun Jan 10 20:37:11 2021  relations with 5 large ideals: 89881466
Sun Jan 10 20:37:11 2021  relations with 6 large ideals: 145991448
Sun Jan 10 20:37:11 2021  relations with 7+ large ideals: 332368621
Sun Jan 10 20:37:11 2021  commencing 2-way merge

I'm going to give it a bit more time, then trying adding more relations and repeating.

Seems worrying that it's stuck at 2-way merge rather than full merge. A bug in msieve, perhaps?

Also I'm thinking "well over 100M" may be an understatement...

ryanp · 2021-01-11, 22:13

Quote:

Originally Posted by charybdis

Seems worrying that it's stuck at 2-way merge rather than full merge. A bug in msieve, perhaps?

Also I'm thinking "well over 100M" may be an understatement...

I'm letting it stay wedged until I manage to gather new relations + merge + remdups4 and gzip, which could be a few more hours at least. At that point, I'll again kill the msieve run and try again.. need to better automate all this!

VBCurtis · 2021-01-11, 22:58

Has anyone had success letting CADO filter, and then running msieve on the resulting matrix files? I think CADO produces smaller matrices from a given dataset, and more of the filtering is multi-threaded.

ryanp · 2021-01-11, 23:02

Quote:

Originally Posted by VBCurtis

Has anyone had success letting CADO filter, and then running msieve on the resulting matrix files? I think CADO produces smaller matrices from a given dataset, and more of the filtering is multi-threaded.

I haven't tried it before, but would be curious to give it a shot. At the bottom of README.msieve there's this blurb:

Code:

up from msieve (svn) revision 891, msieve can read a cycle file produced by
CADO-NFS. To use it, you will have to:

- use CADO-NFS for the filtering. In what follows, let 'prefix' be
  the prefix used for all the CADO filenames
- use the CADO 'replay' binary with --for_msieve to produce
  a file <prefix>.cyc
- concatenate all the relation files specified by purge.log in
  the order specified, and name the file <prefix> in the same
  directory as all the other CADO intermediate files. If Msieve was
  compiled with zlib support, the files do not have to be uncompressed
- create a <prefix>.fb file with the polynomials in Msieve format
- create worktodo.ini with a single line containing N
- run Msieve LA with

  -v -nf <prefix>.fb -s <prefix> -nc2 "cado_filter=1"

charybdis · 2021-01-11, 23:07

According to jasonp the CADO filtering format has changed since that code was written. An old version of CADO could work but I don't know whether it had been sufficiently tested on jobs this big back then, and this would certainly have been before the multithreaded merge code was written.

ryanp · 2021-01-11, 23:08

So I successfully cloned the latest CADO repo and built the replay binary, but not quite sure how to invoke it.

Code:

The available parameters are the following:
    -purged    input purged file
    -his       input history file
    -out       basename for output matrices
    -skip      number of heaviest columns that go to the dense matrix (default 32)
    -index     file containing description of rows (relations-sets) of the matrix
    -ideals    file containing correspondence between ideals and matrix columns
    -force-posix-threads (switch) force the use of posix threads, do not rely on platform memory semantics
    -path_antebuffer path to antebuffer program
    -for_msieve (switch) output matrix in msieve format
    -Nmax      stop at Nmax number of rows (default 0)
    -col0      print only columns with index >= col0
    -colmax    print only columns with index < colmax
    -verbose_flags fine grained control on which messages get printed

I've got an input.dat.gz containing all my relations (gzipped) and already have msieve's input.fb and input.ini. Any advice?

chris2be8 · 2021-01-12, 16:36

fivemack wrote a program called remsing to remove singleton relations. It might help.

See https://mersenneforum.org/showthread...ing#post525512 and related posts.

Chris

ryanp · 2021-01-12, 18:33

Quote:

Originally Posted by chris2be8

fivemack wrote a program called remsing to remove singleton relations. It might help.

See https://mersenneforum.org/showthread...ing#post525512 and related posts.

Chris

Thanks - to be clear, is the suggestion that after "remdups4" (I now have ~2.1B uniques) to then run that through remsing before handing the relations to msieve?

VBCurtis · 2021-01-12, 18:55

Yes. The idea, I think (?) is to run the first pass of singleton removal externally, so msieve handles a smaller dataset. This speeds things up when e.g. trying a handful of target densities, and in the case where relations counts are really big like this job it may help msieve handle the dataset.

You can run it multiple times, just like msieve runs multiple singleton-removal passes, but I found a single run helpful for the GNFS-207 team-sieve job.

ryanp · 2021-01-12, 19:01

I've got it running now. The output is currently at:

Code:

$ ~/remsing/remsing input.dat input.rem
1449910000 read  1102385169 bad

Does that ration of "bad" to "read" seem reasonable?

VBCurtis · 2021-01-12, 19:09

No. The numbers should look quite similar to the first pass of msieve, such as these from your log on 9th Jan:

Code:

commencing in-memory singleton removal
begin with 2069585205 relations and 2015728165 unique ideals
reduce to 940662702 relations and 730460437 ideals in 20 passes

Using this log as a very rough guide, you should get output of about half your input for the first pass through remsing.

Edit: I think that happened to me the first time I tried it; an updated remsing was later posted that worked fine for me. The link posted here was to the post #608 that had the updated remsing, though. Hrmmm.....

2021-01-11, 23:07	#874
charybdis Apr 2020 3³·7 Posts	According to jasonp the CADO filtering format has changed since that code was written. An old version of CADO could work but I don't know whether it had been sufficiently tested on jobs this big back then, and this would certainly have been before the multithreaded merge code was written. Last fiddled with by charybdis on 2021-01-11 at 23:07

2021-01-12, 19:01	#879
ryanp Jun 2012 Boulder, CO 100000111₂ Posts	I've got it running now. The output is currently at: Code: $ ~/remsing/remsing input.dat input.rem 1449910000 read 1102385169 bad Does that ration of "bad" to "read" seem reasonable?

2021-01-12, 19:09	#880
VBCurtis "Curtis" Feb 2005 Riverside, CA 2⁶×73 Posts	No. The numbers should look quite similar to the first pass of msieve, such as these from your log on 9th Jan: Code: commencing in-memory singleton removal begin with 2069585205 relations and 2015728165 unique ideals reduce to 940662702 relations and 730460437 ideals in 20 passes Using this log as a very rough guide, you should get output of about half your input for the first pass through remsing. Edit: I think that happened to me the first time I tried it; an updated remsing was later posted that worked fine for me. The link posted here was to the post #608 that had the updated remsing, though. Hrmmm..... Last fiddled with by VBCurtis on 2021-01-12 at 19:10

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Passive Pascal	Xyzzy	GPU Computing	1	2017-05-17 20:22
Tesla P100 — 5.4 DP TeraFLOPS — Pascal	Mark Rose	GPU Computing	52	2016-07-02 12:11
Nvidia Pascal, a third of DP	firejuggler	GPU Computing	12	2016-02-23 06:55
Calculating perfect numbers in Pascal	Elhueno	Homework Help	5	2008-06-12 16:37
Factorization attempt to a c163 - a new Odd Perfect Number roadblock	jchein1	Factoring	30	2005-05-30 14:43

2021-01-11, 22:58	#872
VBCurtis "Curtis" Feb 2005 Riverside, CA 2⁶·73 Posts	Has anyone had success letting CADO filter, and then running msieve on the resulting matrix files? I think CADO produces smaller matrices from a given dataset, and more of the filtering is multi-threaded.

2021-01-12, 16:36	#876
chris2be8 Sep 2009 2·7·11·13 Posts	fivemack wrote a program called remsing to remove singleton relations. It might help. See https://mersenneforum.org/showthread...ing#post525512 and related posts. Chris

2021-01-12, 18:55	#878
VBCurtis "Curtis" Feb 2005 Riverside, CA 2⁶×73 Posts	Yes. The idea, I think (?) is to run the first pass of singleton removal externally, so msieve handles a smaller dataset. This speeds things up when e.g. trying a handful of target densities, and in the case where relations counts are really big like this job it may help msieve handle the dataset. You can run it multiple times, just like msieve runs multiple singleton-removal passes, but I found a single run helpful for the GNFS-207 team-sieve job.