Msieve 1.53 feedback

[Batalov]

Originally Posted by pinhodecarlos

What should I do?

The matrix is too sparse. Restarting from the check file will not help. Most likely you will get the same result over and over.
The problem is not what you highlighted but this one:

Originally Posted by pinhodecarlos

Thu Nov 24 23:35:07 2016  lanczos halted after 175699 iterations (dim = 11109994)
Thu Nov 24 23:35:33 2016  lanczos error: only trivial dependencies found
Thu Nov 24 23:35:36 2016  BLanczosTime: 1819

You should sieve some more (and dispose of the matrix, .cyc and .chk files), and re-filter.

[pinhodecarlos]

Thank you Batalov.

[jasonp]

Originally Posted by fivemack

I hadn't expected msieve.dat.cyc to be being rewritten at the -nc2 phase

The start of the linear algebra is the first place that has all the information about all the ideals that survive the filtering, and so this is the first place that singleton rows and columns can be deleted. The matrix columns are also permuted to approximately homogenize the density of matrix nonzeros. Both effects require rewriting the cycle file so that future linear algebra runs don't have to redo these steps.

[xilman]

In recent months I've had problems building GPU-enabled msieve binaries. The error message is along the lines of

CUDA /usr/include/string.h:652:42: error: 'memcpy' was not declared in this scope

and the solution can be found in http://stackoverflow.com/questions/3...ed-in-this-sco

TL;DR: Add -D_FORCE_INLINES to NVCCFLAGS

[vasyannyasha]

What does it means?
error -10 converting '(52^351*351^52+1)/1259110360242031927'

[jasonp]

Originally Posted by vasyannyasha

What does it means?
error -10 converting '(52^351*351^52+1)/1259110360242031927'

The numbers msieve handles can be ~310 digits in size or less; the above is 717 digits.

[jasonp]

I have a little spare time for the first time in what feels like forever. Does anyone have an NFS dataset that we think suffers from the large dataset bug? chris2be8 very kindly kept one such dataset accessble since mid 2015, but I don't see a crash running it locally like he did.

I also have two enormous datasets from ryanp on a hard drive mailed to me in maybe 2013, for which filtering failed over and over again. Analyzing those would be excruciating.

[Xyzzy]

Originally Posted by jasonp

I have a little spare time for the first time in what feels like forever.

http://www.mersenneforum.org/jasonp/

[Brian Gladman]

Originally Posted by jasonp

Nothing in the poly selection code is different between 1.52 and 1.53. Also, the latest CUDA toolkit does not want the ancient MSVC2008 express version I had been using to keep it happy, so I need the time to install the 2013Visual Studio compiler.

You could move directly to Visual Studio 2015, for which I can supply fully working build files for both the CUDA and non-CUDA versions of Msieve.

[jasonp]

It is sufficient for me to use MSVC2012 and CUDA 6.5; my employer has standardized on these and they work fine for development purposes. On windows I'd actually prefer to keep using the makefile and gcc under MSYS2, that will make sure windows-only changes get unix versions as well.

[Alfred]

Using the precompiled exe svn 1005 (downloaded from sourceforge) I get:

Sat Sep 30 17:23:38 2017  
Sat Sep 30 17:23:38 2017  
Sat Sep 30 17:23:38 2017  Msieve v. 1.53 (SVN 1005)
Sat Sep 30 17:23:38 2017  random seeds: 20666a00 cd88bd15
Sat Sep 30 17:23:38 2017  factoring 20508929330948193698572470599897897516304792943826902972122198941094004036574931871066771664508847800829028047903019470182253564143086946641713384879471166102489510360704972302100259660442469 (191 digits)
Sat Sep 30 17:23:39 2017  searching for 15-digit factors
Sat Sep 30 17:23:39 2017  commencing number field sieve (191-digit input)
Sat Sep 30 17:23:39 2017  R0: -10782326123572392391655551379588828404027234081
Sat Sep 30 17:23:39 2017  R1: 1
Sat Sep 30 17:23:39 2017  A0: -1
Sat Sep 30 17:23:39 2017  A1: 0
Sat Sep 30 17:23:39 2017  A2: 0
Sat Sep 30 17:23:39 2017  A3: 0
Sat Sep 30 17:23:39 2017  A4: 0
Sat Sep 30 17:23:39 2017  A5: 567661
Sat Sep 30 17:23:39 2017  skew 0.07, size 9.252e-017, alpha 1.180, combined = 2.452e-013 rroots = 1
Sat Sep 30 17:23:39 2017  
Sat Sep 30 17:23:39 2017  commencing relation filtering
Sat Sep 30 17:23:39 2017  setting max relations to 75000000
Sat Sep 30 17:23:39 2017  estimated available RAM is 130995.6 MB
Sat Sep 30 17:23:39 2017  commencing duplicate removal, pass 1
Sat Sep 30 17:36:57 2017  found 1280241 hash collisions in 75000000 relations
Sat Sep 30 17:36:57 2017  commencing duplicate removal, pass 2
Sat Sep 30 17:37:49 2017  found 0 duplicates and 75000000 unique relations
Sat Sep 30 17:37:49 2017  memory use: 318.6 MB
Sat Sep 30 17:37:49 2017  reading ideals above 720000
Sat Sep 30 17:37:49 2017  commencing singleton removal, initial pass
Sat Sep 30 17:55:04 2017  memory use: 3012.0 MB
Sat Sep 30 17:55:04 2017  reading all ideals from disk
Sat Sep 30 17:55:06 2017  memory use: 2917.8 MB
Sat Sep 30 17:55:24 2017  keeping 116837855 ideals with weight <= 200, target excess is 397776
Sat Sep 30 17:55:45 2017  commencing in-memory singleton removal
Sat Sep 30 17:56:00 2017  begin with 75000000 relations and 116837855 unique ideals
Sat Sep 30 17:56:28 2017  reduce to 84 relations and 0 ideals in 7 passes
Sat Sep 30 17:56:28 2017  max relations containing the same ideal: 0
Sat Sep 30 17:56:28 2017  filtering wants 1384008 more relations
Sat Sep 30 17:56:28 2017  elapsed time 00:32:50

In order

Sat Sep 30 18:12:53 2017  
Sat Sep 30 18:12:53 2017  
Sat Sep 30 18:12:53 2017  Msieve v. 1.53 (SVN 1005)
Sat Sep 30 18:12:53 2017  random seeds: d2ad1a90 9da147f5
Sat Sep 30 18:12:53 2017  factoring 20508929330948193698572470599897897516304792943826902972122198941094004036574931871066771664508847800829028047903019470182253564143086946641713384879471166102489510360704972302100259660442469 (191 digits)
Sat Sep 30 18:12:54 2017  searching for 15-digit factors
Sat Sep 30 18:12:54 2017  commencing number field sieve (191-digit input)
Sat Sep 30 18:12:54 2017  R0: -10782326123572392391655551379588828404027234081
Sat Sep 30 18:12:54 2017  R1: 1
Sat Sep 30 18:12:54 2017  A0: -1
Sat Sep 30 18:12:54 2017  A1: 0
Sat Sep 30 18:12:54 2017  A2: 0
Sat Sep 30 18:12:54 2017  A3: 0
Sat Sep 30 18:12:54 2017  A4: 0
Sat Sep 30 18:12:54 2017  A5: 567661
Sat Sep 30 18:12:54 2017  skew 0.07, size 9.252e-017, alpha 1.180, combined = 2.452e-013 rroots = 1
Sat Sep 30 18:12:54 2017  
Sat Sep 30 18:12:54 2017  commencing relation filtering
Sat Sep 30 18:12:54 2017  setting max relations to 100000000
Sat Sep 30 18:12:54 2017  estimated available RAM is 130995.6 MB
Sat Sep 30 18:12:54 2017  commencing duplicate removal, pass 1
Sat Sep 30 18:29:35 2017  found 2258433 hash collisions in 100000000 relations
Sat Sep 30 18:29:35 2017  commencing duplicate removal, pass 2
Sat Sep 30 18:30:27 2017  found 0 duplicates and 100000000 unique relations
Sat Sep 30 18:30:27 2017  memory use: 322.6 MB
Sat Sep 30 18:30:27 2017  reading ideals above 720000
Sat Sep 30 18:30:27 2017  commencing singleton removal, initial pass
Sat Sep 30 20:18:45 2017  memory use: 3012.0 MB
Sat Sep 30 20:18:45 2017  reading all ideals from disk
Sat Sep 30 20:19:02 2017  memory use: 3890.5 MB
Sat Sep 30 20:19:17 2017  keeping 134021290 ideals with weight <= 200, target excess is 539735
Sat Sep 30 20:19:35 2017  commencing in-memory singleton removal
Sat Sep 30 20:19:47 2017  begin with 100000000 relations and 134021290 unique ideals
Sat Sep 30 20:20:15 2017  reduce to 52 relations and 0 ideals in 11 passes
Sat Sep 30 20:20:15 2017  max relations containing the same ideal: 0
Sat Sep 30 20:20:15 2017  filtering wants 1878120 more relations
Sat Sep 30 20:20:15 2017  elapsed time 02:07:22

In order

Sat Sep 30 20:34:30 2017  
Sat Sep 30 20:34:30 2017  
Sat Sep 30 20:34:30 2017  Msieve v. 1.53 (SVN 1005)
Sat Sep 30 20:34:30 2017  random seeds: 899a2f68 e212083d
Sat Sep 30 20:34:30 2017  factoring 20508929330948193698572470599897897516304792943826902972122198941094004036574931871066771664508847800829028047903019470182253564143086946641713384879471166102489510360704972302100259660442469 (191 digits)
Sat Sep 30 20:34:31 2017  searching for 15-digit factors
Sat Sep 30 20:34:31 2017  commencing number field sieve (191-digit input)
Sat Sep 30 20:34:31 2017  R0: -10782326123572392391655551379588828404027234081
Sat Sep 30 20:34:31 2017  R1: 1
Sat Sep 30 20:34:31 2017  A0: -1
Sat Sep 30 20:34:31 2017  A1: 0
Sat Sep 30 20:34:31 2017  A2: 0
Sat Sep 30 20:34:31 2017  A3: 0
Sat Sep 30 20:34:31 2017  A4: 0
Sat Sep 30 20:34:31 2017  A5: 567661
Sat Sep 30 20:34:31 2017  skew 0.07, size 9.252e-017, alpha 1.180, combined = 2.452e-013 rroots = 1
Sat Sep 30 20:34:32 2017  
Sat Sep 30 20:34:32 2017  commencing relation filtering
Sat Sep 30 20:34:32 2017  setting max relations to 125000000
Sat Sep 30 20:34:32 2017  estimated available RAM is 130995.6 MB
Sat Sep 30 20:34:32 2017  commencing duplicate removal, pass 1
Sat Sep 30 20:55:37 2017  found 3501087 hash collisions in 125000000 relations
Sat Sep 30 20:55:37 2017  commencing duplicate removal, pass 2
Sat Sep 30 20:56:44 2017  found 0 duplicates and 125000000 unique relations
Sat Sep 30 20:56:44 2017  memory use: 389.2 MB
Sat Sep 30 20:56:44 2017  reading ideals above 720000
Sat Sep 30 20:56:44 2017  commencing singleton removal, initial pass
Sat Sep 30 21:18:38 2017  memory use: 3012.0 MB
Sat Sep 30 21:18:38 2017  reading all ideals from disk
Sat Sep 30 21:18:39 2017  memory use: 4867.1 MB
Sat Sep 30 21:18:42 2017  keeping 52964801 ideals with weight <= 200, target excess is 95229556
Sat Sep 30 21:18:44 2017  commencing in-memory singleton removal
Sat Sep 30 21:18:47 2017  begin with 125000000 relations and 52964801 unique ideals
Sat Sep 30 21:18:51 2017  reduce to 105180243 relations and 0 ideals in 4 passes
Sat Sep 30 21:18:51 2017  max relations containing the same ideal: 0
Sat Sep 30 21:18:51 2017  filtering wants 15858123 more relations
Sat Sep 30 21:18:51 2017  elapsed time 00:44:21

Out of order, targetexcess is exploding.

In my experience, one single relation causes this explosion.

When using msieve 153 svn 967 the factorization was finished (easily).

Two questions:

• Is this behaviour (of svn 1005) already known?
• Is it a bug (of svn 1005) ?