Msieve 1.53 feedback - Page 12

pinhodecarlos · 2016-11-04, 18:49

Looks like it is quicker to run 8 threads although heat dissipation increases. This is on Windows so no MPI.

(4 threads)

Code:

Fri Nov  4 17:02:25 2016  matrix is 13345605 x 13345805 (3608.8 MB) with weight 1042472315 (78.11/col)
Fri Nov  4 17:02:25 2016  sparse part has weight 799213062 (59.88/col)
Fri Nov  4 17:02:25 2016  saving the first 48 matrix rows for later
Fri Nov  4 17:02:28 2016  matrix includes 64 packed rows
Fri Nov  4 17:02:30 2016  matrix is 13345557 x 13345805 (3424.1 MB) with weight 824694687 (61.79/col)
Fri Nov  4 17:02:30 2016  sparse part has weight 764141890 (57.26/col)
Fri Nov  4 17:02:30 2016  using block size 8192 and superblock size 589824 for processor cache size 6144 kB
Fri Nov  4 17:03:27 2016  commencing Lanczos iteration (4 threads)
Fri Nov  4 17:03:27 2016  memory use: 2825.0 MB
Fri Nov  4 17:03:27 2016  restarting at iteration 41751 (dim = 2640018)
Fri Nov  4 17:04:14 2016  linear algebra at 19.8%, ETA 92h18m

(6 threads)

Code:

Fri Nov  4 16:52:37 2016  matrix is 13345605 x 13345805 (3608.8 MB) with weight 1042472315 (78.11/col)
Fri Nov  4 16:52:37 2016  sparse part has weight 799213062 (59.88/col)
Fri Nov  4 16:52:37 2016  saving the first 48 matrix rows for later
Fri Nov  4 16:52:40 2016  matrix includes 64 packed rows
Fri Nov  4 16:52:41 2016  matrix is 13345557 x 13345805 (3424.1 MB) with weight 824694687 (61.79/col)
Fri Nov  4 16:52:41 2016  sparse part has weight 764141890 (57.26/col)
Fri Nov  4 16:52:41 2016  using block size 8192 and superblock size 589824 for processor cache size 6144 kB
Fri Nov  4 16:53:38 2016  commencing Lanczos iteration (6 threads)
Fri Nov  4 16:53:38 2016  memory use: 2825.0 MB
Fri Nov  4 16:53:39 2016  restarting at iteration 41751 (dim = 2640018)
Fri Nov  4 16:54:26 2016  linear algebra at 19.8%, ETA 92h18m

(8 threads)

Code:

Fri Nov  4 16:58:26 2016  matrix is 13345605 x 13345805 (3608.8 MB) with weight 1042472315 (78.11/col)
Fri Nov  4 16:58:26 2016  sparse part has weight 799213062 (59.88/col)
Fri Nov  4 16:58:26 2016  saving the first 48 matrix rows for later
Fri Nov  4 16:58:29 2016  matrix includes 64 packed rows
Fri Nov  4 16:58:31 2016  matrix is 13345557 x 13345805 (3424.1 MB) with weight 824694687 (61.79/col)
Fri Nov  4 16:58:31 2016  sparse part has weight 764141890 (57.26/col)
Fri Nov  4 16:58:31 2016  using block size 8192 and superblock size 589824 for processor cache size 6144 kB
Fri Nov  4 16:59:28 2016  commencing Lanczos iteration (8 threads)
Fri Nov  4 16:59:28 2016  memory use: 2825.0 MB
Fri Nov  4 16:59:28 2016  restarting at iteration 41751 (dim = 2640018)
Fri Nov  4 17:00:11 2016  linear algebra at 19.8%, ETA 84h26m

henryzz · 2016-11-04, 20:01

You would be more efficient running something else on the other cores though.
If you have a recent k series cpu it may be interesting seeing how the speed of msieve changes when the max TDP of the cpu is decreased. It is quite possible that 8 cores would be most efficient but with a reduced TDP especially if it is not cpu clock speed limitted.
Thinking about it I possibly should be reducing my max TDP as any Prime95 stuff will be memory limitted.
Note that the max turbo TDP can be limitted on the fly using the Intel Extreme Tuning Utility at least on skylake. That is how I would recommend reducing the heat output of your cpu in the summer.

pinhodecarlos · 2016-11-04, 20:05

This is an Ivy Bridge laptop so no playing around with it. In my point of view I've already reached the pintch point, 4 threads for msieve.

henryzz · 2016-11-04, 20:08

Quote:

Originally Posted by pinhodecarlos

This is an Ivy Bridge laptop so no playing around with it. In my point of view I've already reached the pintch point, 4 threads for msieve.

I was thinking it must be a powerful cpu to have problems with the heat. A laptop is the other option I suppose.

jyb · 2016-11-15, 18:30

I'm seeing the following behavior during the square root phase:

Code:

Mon Nov 14 00:18:06 2016  commencing square root phase
Mon Nov 14 00:18:06 2016  reading relations for dependency 1
Mon Nov 14 00:18:18 2016  read 6758950 cycles
Mon Nov 14 00:18:35 2016  cycles contain 22000068 unique relations
Mon Nov 14 00:19:05 2016  error: relation 18214742 corrupt

After reporting the error, msieve exits.

When I try with other dependencies, the same thing happens, sometimes with the same relation, sometimes with other relations. Upon closer examination of the relations, it seems to be happening because the value of 'b' in those relations is greater than 2^32.

During filtering, it did report skipping 223 relations with this issue. So how did those relations end up being part of cycles that got used for dependencies? Is this most likely caused by some kind of mismatch between the saved relation numbers and the actual save file?

And now for the really important questions:

1) Is there any way to fix this up without having to rerun the entire post processing suite?
2) Is it possible this could be caused by a problem with the relations file itself, such that the number needs to be resieved?

jasonp · 2016-11-15, 19:20

Is this with the latest source? If yes, then maybe running the square root with an older version will work. If it doesn't work then it's either a data problem or a weird polynomial.

jyb · 2016-11-15, 20:03

Quote:

Originally Posted by jasonp

Is this with the latest source? If yes, then maybe running the square root with an older version will work. If it doesn't work then it's either a data problem or a weird polynomial.

No, this was with r988, but it also happened with the latest. There's nothing weird about the polynomial (at least, I don't think so, though it might help to know how you'd define "weird").

Any thoughts on my final questions? What exactly does a "data problem" mean?

jasonp · 2016-11-19, 20:37

If the filtering thinks that a relation that it previously used is suddenly unusable, that to me means that the collection of files that filtering generated has now changed somehow. That's what I meant as a data problem.

If there was a problem with the sieving then the filtering would be complaining much more; every relation is checked so that you can be sure that the (a,b) of the relation generates polynomial values whose factorizations are really as given in the relation file.

jyb · 2016-11-20, 15:58

Quote:

Originally Posted by jasonp

If the filtering thinks that a relation that it previously used is suddenly unusable, that to me means that the collection of files that filtering generated has now changed somehow. That's what I meant as a data problem.

If there was a problem with the sieving then the filtering would be complaining much more; every relation is checked so that you can be sure that the (a,b) of the relation generates polynomial values whose factorizations are really as given in the relation file.

Well this turned out to be interesting. The problem was indeed a bad relations file, though nothing in it seems to have changed between the end of sieving and the beginning of square root (other than the addition of free relations during filtering, of course). The relations file had about 250 KB of total garbage right in the middle of it. As if someone had taken random noise and shoved it in there. It's not easy to spot 250 KB in a 29 GB file!

I'm still not clear on why this would cause the problem it did. The garbage shouldn't have caused any inconsistencies in the relation numbering: lines are always broken up the same way, even if they no longer make much sense (and often only part of a "line" at a time can be read); most of them end up being omitted entirely, and the rest appear in the filtering as corrupt so they get skipped (and therefore shouldn't be used as part of any cycle).

Nonetheless, when I removed this garbage and reran post-processing from the beginning it succeeded, so I conclude that the garbage was indeed the problem somehow. If you're interested in tracking down exactly what went wrong, the relations file is available for you.

fivemack · 2016-11-21, 23:30

Code:

Mon Nov 21 18:47:45 2016  commencing linear algebra
Mon Nov 21 18:47:46 2016  read 14013186 cycles
Mon Nov 21 18:48:15 2016  cycles contain 47737608 unique relations
Mon Nov 21 18:52:30 2016  read 47737608 relations
Mon Nov 21 18:53:48 2016  using 20 quadratic characters above 4294917295
Mon Nov 21 18:56:49 2016  building initial matrix
Mon Nov 21 19:06:05 2016  memory use: 6180.7 MB
Mon Nov 21 19:06:11 2016  read 14013186 cycles
Mon Nov 21 19:06:13 2016  matrix is 14013030 x 14013186 (7456.2 MB) with weight 2130821074 (152.06/col)
Mon Nov 21 19:06:13 2016  sparse part has weight 1800463109 (128.48/col)
Mon Nov 21 19:09:37 2016  filtering completed in 2 passes
Mon Nov 21 19:09:40 2016  matrix is 14012924 x 14013090 (7456.2 MB) with weight 2130816623 (152.06/col)
Mon Nov 21 19:09:40 2016  sparse part has weight 1800460937 (128.48/col)
Mon Nov 21 19:10:33 2016  matrix starts at (0, 0)
Mon Nov 21 19:10:34 2016  matrix is 14012924 x 14013090 (7456.2 MB) with weight 2130816623 (152.06/col)
Mon Nov 21 19:10:34 2016  sparse part has weight 1800460937 (128.48/col)
Mon Nov 21 19:10:34 2016  saving the first 48 matrix rows for later
Mon Nov 21 19:10:37 2016  matrix includes 64 packed rows
Mon Nov 21 19:10:38 2016  matrix is 14012876 x 14013090 (7163.7 MB) with weight 1862723217 (132.93/col)
Mon Nov 21 19:10:38 2016  sparse part has weight 1737785554 (124.01/col)

Then I noticed that I had started the job with -t7 on a six-core machine, which seemed likely to be inefficient, so I killed it and restarted, and got

Code:

Mon Nov 21 23:18:03 2016  commencing linear algebra
Mon Nov 21 23:18:04 2016  read 14013090 cycles
Mon Nov 21 23:18:33 2016  cycles contain 47737591 unique relations
Mon Nov 21 23:23:19 2016  read 47737591 relations
Mon Nov 21 23:23:30 2016  error: cannot locate relation 10091370

Code:

% ls -l msieve.dat msieve.dat.cyc
-rw-rw-r--  1 pumpkin pumpkin 53137808302 Nov 20 22:41 msieve.dat
-rw-rw-r--  1 pumpkin pumpkin   589022792 Nov 21 19:09 msieve.dat.cyc

I hadn't expected msieve.dat.cyc to be being rewritten at the -nc2 phase

pinhodecarlos · 2016-11-24, 23:38

After arriving from a few days off I've just noticed that on beginning of square root I am getting this:

Code:

Thu Nov 24 22:48:21 2016  BLanczosTime: 257814
Thu Nov 24 22:48:21 2016  
Thu Nov 24 22:48:21 2016  commencing square root phase
Thu Nov 24 22:48:21 2016  handling dependencies 1 to 64
Thu Nov 24 22:48:21 2016  reading relations for dependency 1
Thu Nov 24 22:48:21 2016  error: read_cycles can't open dependency file
Thu Nov 24 23:05:16 2016  
Thu Nov 24 23:05:16 2016  
Thu Nov 24 23:05:16 2016  Msieve v. 1.53 (SVN unknown)
Thu Nov 24 23:05:16 2016  random seeds: 80758208 2a4af5e7
Thu Nov 24 23:05:16 2016  factoring 1844994401868809535615822499230108363103344173517934192738049602765603435725048277274960946637692879605849386632016724036829362668587980375791404019492697122166600105296727972022009735421631 (190 digits)
Thu Nov 24 23:05:17 2016  searching for 15-digit factors
Thu Nov 24 23:05:17 2016  commencing number field sieve (190-digit input)
Thu Nov 24 23:05:17 2016  R0: 12259964326927110866866776217202473468949912977468817408
Thu Nov 24 23:05:17 2016  R1: -3556153025177363557255317383565515512407041673852007
Thu Nov 24 23:05:17 2016  A0: 1
Thu Nov 24 23:05:17 2016  A1: 1
Thu Nov 24 23:05:17 2016  A2: 1
Thu Nov 24 23:05:17 2016  A3: 1
Thu Nov 24 23:05:17 2016  A4: 1
Thu Nov 24 23:05:17 2016  skew 1.00, size 1.886e-22, alpha 1.694, combined = 2.169e-13 rroots = 0
Thu Nov 24 23:05:17 2016  
Thu Nov 24 23:05:17 2016  commencing linear algebra
Thu Nov 24 23:05:48 2016  matrix starts at (0, 0)
Thu Nov 24 23:05:51 2016  matrix is 11110163 x 11110361 (3121.4 MB) with weight 904460385 (81.41/col)
Thu Nov 24 23:05:51 2016  sparse part has weight 696033678 (62.65/col)
Thu Nov 24 23:05:51 2016  saving the first 48 matrix rows for later
Thu Nov 24 23:05:54 2016  matrix includes 64 packed rows
Thu Nov 24 23:05:55 2016  matrix is 11110115 x 11110361 (2976.2 MB) with weight 720147285 (64.82/col)
Thu Nov 24 23:05:55 2016  sparse part has weight 669079465 (60.22/col)
Thu Nov 24 23:05:55 2016  using block size 8192 and superblock size 589824 for processor cache size 6144 kB
Thu Nov 24 23:06:43 2016  commencing Lanczos iteration (8 threads)
Thu Nov 24 23:06:43 2016  memory use: 2464.6 MB
Thu Nov 24 23:06:43 2016  restarting at iteration 174592 (dim = 11040060)
Thu Nov 24 23:07:17 2016  linear algebra at 99.4%, ETA 0h25m
Thu Nov 24 23:07:27 2016  checkpointing every 170000 dimensions
Thu Nov 24 23:35:07 2016  lanczos halted after 175699 iterations (dim = 11109994)
Thu Nov 24 23:35:33 2016  lanczos error: only trivial dependencies found
Thu Nov 24 23:35:36 2016  BLanczosTime: 1819
Thu Nov 24 23:35:36 2016  
Thu Nov 24 23:35:36 2016  commencing square root phase
Thu Nov 24 23:35:36 2016  reading relations for dependency 1
Thu Nov 24 23:35:36 2016  error: read_cycles can't open dependency file

What should I do?

2016-11-04, 20:01	#123
henryzz Just call me Henry "David" Sep 2007 Cambridge (GMT/BST) 2×3³×109 Posts	You would be more efficient running something else on the other cores though. If you have a recent k series cpu it may be interesting seeing how the speed of msieve changes when the max TDP of the cpu is decreased. It is quite possible that 8 cores would be most efficient but with a reduced TDP especially if it is not cpu clock speed limitted. Thinking about it I possibly should be reducing my max TDP as any Prime95 stuff will be memory limitted. Note that the max turbo TDP can be limitted on the fly using the Intel Extreme Tuning Utility at least on skylake. That is how I would recommend reducing the heat output of your cpu in the summer. Last fiddled with by henryzz on 2016-11-04 at 20:05

2016-11-15, 18:30	#126
jyb Aug 2005 Seattle, WA 2·883 Posts	I'm seeing the following behavior during the square root phase: Code: Mon Nov 14 00:18:06 2016 commencing square root phase Mon Nov 14 00:18:06 2016 reading relations for dependency 1 Mon Nov 14 00:18:18 2016 read 6758950 cycles Mon Nov 14 00:18:35 2016 cycles contain 22000068 unique relations Mon Nov 14 00:19:05 2016 error: relation 18214742 corrupt After reporting the error, msieve exits. When I try with other dependencies, the same thing happens, sometimes with the same relation, sometimes with other relations. Upon closer examination of the relations, it seems to be happening because the value of 'b' in those relations is greater than 2^32. During filtering, it did report skipping 223 relations with this issue. So how did those relations end up being part of cycles that got used for dependencies? Is this most likely caused by some kind of mismatch between the saved relation numbers and the actual save file? And now for the really important questions: 1) Is there any way to fix this up without having to rerun the entire post processing suite? 2) Is it possible this could be caused by a problem with the relations file itself, such that the number needs to be resieved?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Msieve 1.50 feedback	firejuggler	Msieve	99	2013-02-17 11:53
Msieve v1.48 feedback	Jeff Gilchrist	Msieve	48	2011-06-10 18:18
Msieve 1.43 feedback	Jeff Gilchrist	Msieve	47	2009-11-24 15:53
Msieve 1.42 feedback	Andi47	Msieve	167	2009-10-18 19:37
Msieve 1.41 Feedback	Batalov	Msieve	130	2009-06-09 16:01

2016-11-04, 20:05	#124
pinhodecarlos "Carlos Pinho" Oct 2011 Milton Keynes, UK 3·17·97 Posts	This is an Ivy Bridge laptop so no playing around with it. In my point of view I've already reached the pintch point, 4 threads for msieve.

2016-11-15, 19:20	#127
jasonp Tribal Bullet Oct 2004 3×1,181 Posts	Is this with the latest source? If yes, then maybe running the square root with an older version will work. If it doesn't work then it's either a data problem or a weird polynomial.

2016-11-19, 20:37	#129
jasonp Tribal Bullet Oct 2004 3×1,181 Posts	If the filtering thinks that a relation that it previously used is suddenly unusable, that to me means that the collection of files that filtering generated has now changed somehow. That's what I meant as a data problem. If there was a problem with the sieving then the filtering would be complaining much more; every relation is checked so that you can be sure that the (a,b) of the relation generates polynomial values whose factorizations are really as given in the relation file.