grblbl ..
[code]
gcc -D_FILE_OFFSET_BITS=64 -O3 -fomit-frame-pointer -march=k8 -DNDEBUG -D_LARGEF
ILE64_SOURCE -Wall -W -DMSIEVE_SVN_VERSION="\"unknown\"" -I. -Iinclude -Ignfs -
Ignfs/poly -Ignfs/poly/stage1 -DNO_ZLIB -c -o common/polyroot.o common/polyroot.
c
In file included from include/polyroot.h:19:0,
from common/polyroot.c:15:
include/dd.h: In function 'dd_set_precision_ieee':
include/dd.h:46:2: attention : implicit declaration of function '_control87' [-W
implicit-function-declaration]
include/dd.h:47:13: erreur: '_PC_53' undeclared (first use in this function)
include/dd.h:47:13: note: each undeclared identifier is reported only once for e
ach function it appears in
include/dd.h:47:21: erreur: '_MCW_PC' undeclared (first use in this function)
include/dd.h: In function 'dd_precision_is_ieee':
include/dd.h:73:19: erreur: '_MCW_PC' undeclared (first use in this function)
include/dd.h:73:31: erreur: '_PC_53' undeclared (first use in this function)
make: *** [common/polyroot.o] Error 1
[/code]

Hmm, what compiler and OS was this on?

What environment/compiler are you using? It looks like you have WIN32 or _WIN64 defined, but your float.h must lack the bits that msieve expects it to have in that case.

Jason, is it expected that Lanczos iterations towards the end are slightly slower than in the beginning? I wonder if there is a simple qualitative explanation.
It is hard to observe on small runs, and even on larger runs this is visible not with ETA but in the growing intervals between .chk file writes; in my case, these intervals grew from 67 minutes (early in the run) to 70 and slowly to 75 minutes (day 18 out of 20). The memory is not being swapped - I checked.

The Lanczos runtime should not vary over time. Although if the iteration genuinely slowed down by 10% over the course of the run then the ETA would be affected. The ETA is recalculated on every iteration using the unix seconds counter. Could yours be drifting? I've also notice in earlier versions that the ETA at the beginning of a run is often about 5% high, compared to the actual elapsed time measured at the end of the run.

Thank you; makes sense; I thought so, but then I thought that I've overlooked something: the matrix is a constant object, the split for threads is done once and is constant, the state vector is essentially random, so the rate of churning should be like you said - constant over time.

The ETA surely drifts at the very beginning, but that's different. This is already after a several day of uninterrupted run, and then ETA is usually very stable. So, I've stopped restarted and the .chk-writing interval is back to 70 minutes.

Now what I think is that this could be something boring like throttling - I've noticed that the Scythe CPU fan is slightly rattling, so I will have to replace it. The temp is fine right now but maybe it comes and goes. I'll try to finish the matrix as is and then run some tests with hardware. We are still on schedule to finish the c258 this year.

Out of curiosity (I'm not proposing this as a task for msieve 1.50 :smile:): would it be difficult to modify msieve so as to optionally speed up re-filtering at a different target density ?
(at the space penalty of keeping intermediate files, I'm sure, but I'm OK with that)


For instance, in the run reproduced below (a 30-bit LPs task on the easy side, I could have made RSALS produce only ~110M relations...), I might have wanted to raise the target density further. However, I didn't, because re-filtering everything would have subtracted an hour to the potential speed increase yielded by a smaller, denser matrix.
If re-filtering had taken half of that time (the latest 23 minutes or so, plus maybe some time to read additional data ?), I probably would have re-filtered :smile:

[code]Tue Dec 13 20:38:39 2011 Msieve v. 1.50 (SVN 688M)
Tue Dec 13 20:38:39 2011 random seeds: dc3e000b aea72940
Tue Dec 13 20:38:39 2011 factoring 9000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001 (226 digits)
Tue Dec 13 20:38:41 2011 searching for 15-digit factors
Tue Dec 13 20:38:41 2011 commencing number field sieve (226-digit input)
Tue Dec 13 20:38:41 2011 R0: -10000000000000000000000000000000000000
Tue Dec 13 20:38:41 2011 R1: 1
Tue Dec 13 20:38:41 2011 A0: 1
Tue Dec 13 20:38:41 2011 A1: 0
Tue Dec 13 20:38:41 2011 A2: 0
Tue Dec 13 20:38:41 2011 A3: 0
Tue Dec 13 20:38:41 2011 A4: 0
Tue Dec 13 20:38:41 2011 A5: 0
Tue Dec 13 20:38:41 2011 A6: 9000
Tue Dec 13 20:38:41 2011 skew 0.22, size 2.067e-11, alpha 0.996, combined = 1.105e-12 rroots = 0
Tue Dec 13 20:38:41 2011
Tue Dec 13 20:38:41 2011 commencing relation filtering with target density 85.00
Tue Dec 13 20:38:41 2011 estimated available RAM is 3072.0 MB
Tue Dec 13 20:38:41 2011 commencing duplicate removal, pass 1
<snip errors reading relations>
Tue Dec 13 20:50:48 2011 skipped 131 relations with b > 2^32
Tue Dec 13 20:50:48 2011 found 22220107 hash collisions in 118474319 relations
Tue Dec 13 20:51:03 2011 added 1 free relations
Tue Dec 13 20:51:03 2011 commencing duplicate removal, pass 2
Tue Dec 13 20:54:19 2011 found 21254598 duplicates and 97219722 unique relations
Tue Dec 13 20:54:19 2011 memory use: 660.8 MB
Tue Dec 13 20:54:20 2011 reading ideals above 720000
Tue Dec 13 20:54:20 2011 commencing singleton removal, initial pass
Tue Dec 13 21:10:58 2011 memory use: 2756.0 MB
Tue Dec 13 21:10:59 2011 removing singletons from LP file
Tue Dec 13 21:10:59 2011 start with 97219722 relations and 90570125 ideals
Tue Dec 13 21:12:56 2011 pass 1: found 26769593 singletons
Tue Dec 13 21:13:51 2011 pass 2: found 6437137 singletons
Tue Dec 13 21:14:43 2011 pass 3: found 1637660 singletons
Tue Dec 13 21:15:34 2011 pass 4: found 415448 singletons
Tue Dec 13 21:17:20 2011 pruned dataset has 61959884 relations and 51582595 large ideals
Tue Dec 13 21:17:21 2011 reading all ideals from disk
Tue Dec 13 21:18:01 2011 memory use: 2336.6 MB
Tue Dec 13 21:18:14 2011 keeping 51375329 ideals with weight <= 200, target excess is 323344
Tue Dec 13 21:18:26 2011 commencing in-memory singleton removal
Tue Dec 13 21:18:37 2011 begin with 61959884 relations and 51375329 unique ideals
Tue Dec 13 21:20:47 2011 reduce to 61818281 relations and 51233646 ideals in 11 passes
Tue Dec 13 21:20:47 2011 max relations containing the same ideal: 200
Tue Dec 13 21:21:37 2011 removing 9702740 relations and 7702741 ideals in 2000000 cliques
Tue Dec 13 21:21:40 2011 commencing in-memory singleton removal
Tue Dec 13 21:21:50 2011 begin with 52115541 relations and 51233646 unique ideals
Tue Dec 13 21:23:17 2011 reduce to 51192976 relations and 42574151 ideals in 9 passes
Tue Dec 13 21:23:17 2011 max relations containing the same ideal: 186
Tue Dec 13 21:23:57 2011 removing 7426975 relations and 5426975 ideals in 2000000 cliques
Tue Dec 13 21:24:01 2011 commencing in-memory singleton removal
Tue Dec 13 21:24:08 2011 begin with 43766001 relations and 42574151 unique ideals
Tue Dec 13 21:25:10 2011 reduce to 43052619 relations and 36407109 ideals in 8 passes
Tue Dec 13 21:25:10 2011 max relations containing the same ideal: 167
Tue Dec 13 21:25:45 2011 removing 6751847 relations and 4751847 ideals in 2000000 cliques
Tue Dec 13 21:25:48 2011 commencing in-memory singleton removal
Tue Dec 13 21:25:54 2011 begin with 36300772 relations and 36407109 unique ideals
Tue Dec 13 21:26:44 2011 reduce to 35602587 relations and 30927673 ideals in 8 passes
Tue Dec 13 21:26:44 2011 max relations containing the same ideal: 144
Tue Dec 13 21:27:13 2011 removing 6427785 relations and 4427785 ideals in 2000000 cliques
Tue Dec 13 21:27:16 2011 commencing in-memory singleton removal
Tue Dec 13 21:27:21 2011 begin with 29174802 relations and 30927673 unique ideals
Tue Dec 13 21:28:00 2011 reduce to 28429780 relations and 25717778 ideals in 8 passes
Tue Dec 13 21:28:00 2011 max relations containing the same ideal: 123
Tue Dec 13 21:28:23 2011 removing 6113219 relations and 4113219 ideals in 2000000 cliques
Tue Dec 13 21:28:26 2011 commencing in-memory singleton removal
Tue Dec 13 21:28:29 2011 begin with 22316561 relations and 25717778 unique ideals
Tue Dec 13 21:29:06 2011 reduce to 21407079 relations and 20637037 ideals in 10 passes
Tue Dec 13 21:29:06 2011 max relations containing the same ideal: 102
Tue Dec 13 21:29:23 2011 removing 1759028 relations and 1364066 ideals in 394962 cliques
Tue Dec 13 21:29:25 2011 commencing in-memory singleton removal
Tue Dec 13 21:29:28 2011 begin with 19648051 relations and 20637037 unique ideals
Tue Dec 13 21:29:51 2011 reduce to 19558422 relations and 19181783 ideals in 7 passes
Tue Dec 13 21:29:51 2011 max relations containing the same ideal: 96
Tue Dec 13 21:30:12 2011 relations with 0 large ideals: 11658
Tue Dec 13 21:30:12 2011 relations with 1 large ideals: 2349
Tue Dec 13 21:30:12 2011 relations with 2 large ideals: 34734
Tue Dec 13 21:30:12 2011 relations with 3 large ideals: 266600
Tue Dec 13 21:30:12 2011 relations with 4 large ideals: 1134118
Tue Dec 13 21:30:12 2011 relations with 5 large ideals: 2922338
Tue Dec 13 21:30:12 2011 relations with 6 large ideals: 4772368
Tue Dec 13 21:30:12 2011 relations with 7+ large ideals: 10414257
Tue Dec 13 21:30:12 2011 commencing 2-way merge
Tue Dec 13 21:30:33 2011 reduce to 13528344 relation sets and 13151705 unique ideals
Tue Dec 13 21:30:33 2011 commencing full merge
Tue Dec 13 21:37:47 2011 memory use: 1708.7 MB
Tue Dec 13 21:37:49 2011 found 6716964 cycles, need 6669905
Tue Dec 13 21:37:52 2011 weight of 6669905 cycles is about 567177283 (85.04/cycle)
Tue Dec 13 21:37:52 2011 distribution of cycle lengths:
Tue Dec 13 21:37:52 2011 1 relations: 504298
Tue Dec 13 21:37:52 2011 2 relations: 636713
Tue Dec 13 21:37:52 2011 3 relations: 698069
Tue Dec 13 21:37:52 2011 4 relations: 683235
Tue Dec 13 21:37:52 2011 5 relations: 658665
Tue Dec 13 21:37:52 2011 6 relations: 602260
Tue Dec 13 21:37:52 2011 7 relations: 537367
Tue Dec 13 21:37:52 2011 8 relations: 465253
Tue Dec 13 21:37:52 2011 9 relations: 396310
Tue Dec 13 21:37:52 2011 10+ relations: 1487735
Tue Dec 13 21:37:52 2011 heaviest cycle: 25 relations
Tue Dec 13 21:37:55 2011 commencing cycle optimization
Tue Dec 13 21:38:13 2011 start with 43760349 relations
Tue Dec 13 21:40:17 2011 pruned 1604684 relations
Tue Dec 13 21:40:18 2011 memory use: 1267.6 MB
Tue Dec 13 21:40:18 2011 distribution of cycle lengths:
Tue Dec 13 21:40:18 2011 1 relations: 504298
Tue Dec 13 21:40:18 2011 2 relations: 651686
Tue Dec 13 21:40:18 2011 3 relations: 726090
Tue Dec 13 21:40:18 2011 4 relations: 708704
Tue Dec 13 21:40:18 2011 5 relations: 686733
Tue Dec 13 21:40:18 2011 6 relations: 623267
Tue Dec 13 21:40:18 2011 7 relations: 554164
Tue Dec 13 21:40:18 2011 8 relations: 473805
Tue Dec 13 21:40:18 2011 9 relations: 398676
Tue Dec 13 21:40:18 2011 10+ relations: 1342482
Tue Dec 13 21:40:18 2011 heaviest cycle: 25 relations
Tue Dec 13 21:40:39 2011 RelProcTime: 3718[/code]

Two notes on that run:
* only 1 free relation was found, because the free relations had already been found by a previous run, which put the 4 GB machine running 64-bit OS and msieve into thrashing. So I killed msieve, and restarted using -mb 3072, which coerced it into parforming singleton removal on the disk.
* the "commencing relation filtering with target density 85.00" trace is the result of the patch I posted in comment #57 of this thread. I'm using it on most of my copies of msieve SVN, hence the "688M" revision number.

Jasonp, did you see Lionel question? I'm also interested in your answer.
Meanwhile, will next version of msieve support AVX if applicable?

I saw Lionel's question but have been very busy lately. To restart the merge phase, you would have to remember the current list of relations for each column of the matrix as well as the list of ideals that column is tracking. It's straightforward to do so, but maybe it would be better to improve the merge phase itself so that you don't have to guess at the best target density in the first place. I'll think about it.

Carlos, I don't see how AVX would be able to improve NFS postprocessing. Nothing in the linear algebra can make effective use of longer vectors. I've tried several times and making the LA use longer vectors just slows it down because you run out of cache too quickly. On the other hand, akruppa's work on Block Lanczos definitely benefits from extremely long vectors, so it's entirely possible I'm missing something.

[QUOTE=jasonp;286012]I saw Lionel's question but have been very busy lately. To restart the merge phase, you would have to remember the current list of relations for each column of the matrix as well as the list of ideals that column is tracking. It's straightforward to do so, but maybe it would be better to improve the merge phase itself so that you don't have to guess at the best target density in the first place. I'll think about it.

Carlos, I don't see how AVX would be able to improve NFS postprocessing. Nothing in the linear algebra can make effective use of longer vectors. I've tried several times and making the LA use longer vectors just slows it down because you run out of cache too quickly. On the other hand, akruppa's work on Block Lanczos definitely benefits from extremely long vectors, so it's entirely possible I'm missing something.[/QUOTE]
What about QS? Would AVX help that?

[QUOTE=jasonp;286012]

Carlos, I don't see how AVX would be able to improve NFS postprocessing. Nothing in the linear algebra can make effective use of longer vectors. I've tried several times and making the LA use longer vectors just slows it down because you run out of cache too quickly. On the other hand, akruppa's work on Block Lanczos definitely benefits from extremely long vectors, so it's entirely possible I'm missing something.[/QUOTE]

But did you try your code on a Sandy Bridge processor? Maybe now you don't run out of cache. Please take a look at [URL]http://software.intel.com/en-us/avx/[/URL][URL="http://software.intel.com/en-us/articles/practical-intel-avx-optimization-on-2nd-generation-intel-core-processors/"]
[/URL]