|
2012-11-10, 18:27
|
#12 | |
|
Basketry That Evening!
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88
3×29×83 Posts |
Quote:
Code:
found 10683242 relations, need at least 10040346, proceeding with filtering ... nfs: commencing msieve filtering 807906643475587166469565990012892661378984895129767261906926840779743774370791677568184427418567348310699868548280261 commencing relation filtering estimated available RAM is 12019.3 MB commencing duplicate removal, pass 1 read 10M relations found 881412 hash collisions in 10683242 relations added 63479 free relations commencing duplicate removal, pass 2 found 567919 duplicates and 10178802 unique relations memory use: 34.6 MB reading ideals above 720000 commencing singleton removal, initial pass memory use: 344.5 MB reading all ideals from disk memory use: 284.5 MB keeping 10850144 ideals with weight <= 200, target excess is 116184 commencing in-memory singleton removal begin with 10178802 relations and 10850144 unique ideals reduce to 4027461 relations and 3611790 ideals in 18 passes max relations containing the same ideal: 93 removing 935103 relations and 794654 ideals in 140449 cliques commencing in-memory singleton removal begin with 3092358 relations and 3611790 unique ideals reduce to 2945657 relations and 2664125 ideals in 10 passes max relations containing the same ideal: 75 removing 692348 relations and 551899 ideals in 140449 cliques commencing in-memory singleton removal begin with 2253309 relations and 2664125 unique ideals reduce to 2132556 relations and 1985706 ideals in 10 passes max relations containing the same ideal: 57 removing 101017 relations and 88941 ideals in 12076 cliques commencing in-memory singleton removal begin with 2031539 relations and 1985706 unique ideals reduce to 2028312 relations and 1893522 ideals in 5 passes max relations containing the same ideal: 54 relations with 0 large ideals: 519 relations with 1 large ideals: 4389 relations with 2 large ideals: 48978 relations with 3 large ideals: 218605 relations with 4 large ideals: 494221 relations with 5 large ideals: 613265 relations with 6 large ideals: 432986 relations with 7+ large ideals: 215349 commencing 2-way merge reduce to 1195098 relation sets and 1060308 unique ideals commencing full merge memory use: 112.2 MB found 584176 cycles, need 566508 weight of 566508 cycles is about 39875471 (70.39/cycle) distribution of cycle lengths: 1 relations: 63474 2 relations: 60380 3 relations: 60776 4 relations: 57860 5 relations: 53241 6 relations: 47117 7 relations: 41223 8 relations: 36201 9 relations: 30831 10+ relations: 115405 heaviest cycle: 19 relations commencing cycle optimization start with 3442887 relations pruned 78833 relations memory use: 114.0 MB distribution of cycle lengths: 1 relations: 63474 2 relations: 61715 3 relations: 62793 4 relations: 59144 5 relations: 54515 6 relations: 47760 7 relations: 41881 8 relations: 36443 9 relations: 30812 10+ relations: 107971 heaviest cycle: 19 relations RelProcTime: 194 nfs: commencing msieve linear algebra commencing linear algebra read 566508 cycles cycles contain 1900156 unique relations read 1900156 relations using 20 quadratic characters above 134217530 building initial matrix memory use: 249.3 MB read 566508 cycles matrix is 640497 x 566508 (172.7 MB) with weight 54656717 (96.48/col) sparse part has weight 38466824 (67.90/col) filtering completed in 2 passes matrix is 579240 x 538417 (163.1 MB) with weight 51591357 (95.82/col) sparse part has weight 36281612 (67.39/col) matrix starts at (0, 0) matrix is 579240 x 538417 (163.1 MB) with weight 51591357 (95.82/col) sparse part has weight 36281612 (67.39/col) matrix needs more columns than rows; try adding 2-3% more relations WARNING: couldn't find factor in factor.log Code:
n: 807906643475587166469565990012892661378984895129767261906926840779743774370791677568184427418567348310699868548280261 skew: 326758.38 c0: 129084363205788348069350664240 c1: -5327426041926132739069596 c2: -74452615988084858292 c3: 215633346996179 c4: 383157650 c5: 240 Y0: -80432057487008881143761 Y1: 469386522941 rlim: 4700000 alim: 4700000 lpbr: 27 lpba: 27 mfbr: 54 mfba: 54 rlambda: 2.5 alambda: 2.5 Also, it just occurred to me: as this is YAFU, this is Msieve SVN 722, so not quite 1.51. |
|
|
|
|
2012-11-10, 19:24
|
#13 |
|
Tribal Bullet
Oct 2004
67258 Posts |
Could you try increasing MAX_RELSET_SIZE in common/filter/filter_priv.h to something larger than the default of 28? Maybe something like 50? It's possible that the merge phase produces big chains of relations when you have barely enough to form a matrix, and this number controls when those big chains are thrown away to save memory.
|
|
|
|
|
2012-11-11, 00:05
|
#14 |
|
Basketry That Evening!
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88
3×29×83 Posts |
Sorry, I got it factored before I saw this, and YAFU automatically deleted the data files. (If you'd like, since the job file is above, I can resieve it and experiment with it as you direct, including the current directions.)
It turns out though, the 2-3% extra was not sufficient to complete the job. Here are the next few filtering runs. Attached are the screen outputs of the filtering runs. (It was too much text for a post.) YAFU's minrels was 10040346; the first filtering attempt, with data above, had "567919 duplicates and 10178802 unique relations". It got to the matrix building stage before getting that warning. All later filtering attempts tried to build a matrix and failed, except for the last one which succeeded. Those later filtering attempts had the following rel counts: found 733628 duplicates and 11665811 unique relations found 1109736 duplicates and 14569742 unique relations found 1542400 duplicates and 17435770 unique relations -- successful |
|
|
|
|
2012-11-11, 00:17
|
#15 |
|
Tribal Bullet
Oct 2004
67258 Posts |
Needing 50% more relations than expected sounds like a serious problem. Could you regenerate the dataset and try some of the fixes above?
The first time I saw behavior like this was a barely-oversieved dataset for RSA704, and I figured that it was some obscure problem caused by pushing the filtering to its limits. But if folks can run into a similar problem even on small-size numbers then we'll need to get to the bottom of it. Last fiddled with by jasonp on 2012-11-11 at 00:17 |
|
|
|
|
2012-11-12, 07:29
|
#16 | |
|
Basketry That Evening!
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88
1C3516 Posts |
Quote:
Code:
commencing relation filtering estimated available RAM is 12019.3 MB commencing duplicate removal, pass 1 read 10M relations found 880443 hash collisions in 10706359 relations commencing duplicate removal, pass 2 found 565217 duplicates and 10141142 unique relations memory use: 34.6 MB reading ideals above 720000 commencing singleton removal, initial pass memory use: 344.5 MB reading all ideals from disk memory use: 283.4 MB keeping 10831051 ideals with weight <= 200, target excess is 116184 commencing in-memory singleton removal begin with 10141142 relations and 10831051 unique ideals reduce to 3981976 relations and 3578191 ideals in 18 passes max relations containing the same ideal: 93 removing 911606 relations and 777100 ideals in 134506 cliques commencing in-memory singleton removal begin with 3070370 relations and 3578191 unique ideals reduce to 2928484 relations and 2653294 ideals in 10 passes max relations containing the same ideal: 75 removing 674102 relations and 539596 ideals in 134506 cliques commencing in-memory singleton removal begin with 2254382 relations and 2653294 unique ideals reduce to 2138968 relations and 1992841 ideals in 11 passes max relations containing the same ideal: 57 removing 96408 relations and 85055 ideals in 11353 cliques commencing in-memory singleton removal begin with 2042560 relations and 1992841 unique ideals reduce to 2039679 relations and 1904882 ideals in 6 passes max relations containing the same ideal: 55 relations with 0 large ideals: 519 relations with 1 large ideals: 4389 relations with 2 large ideals: 49052 relations with 3 large ideals: 219349 relations with 4 large ideals: 497036 relations with 5 large ideals: 617331 relations with 6 large ideals: 435585 relations with 7+ large ideals: 216418 commencing 2-way merge reduce to 1198999 relation sets and 1064202 unique ideals commencing full merge memory use: 112.9 MB found 586815 cycles, need 568402 weight of 568402 cycles is about 40058002 (70.47/cycle) distribution of cycle lengths: 1 relations: 63729 2 relations: 60565 3 relations: 61100 4 relations: 57588 5 relations: 53404 6 relations: 47247 7 relations: 41435 8 relations: 36096 9 relations: 31272 10+ relations: 115966 heaviest cycle: 19 relations commencing cycle optimization start with 3458848 relations pruned 79110 relations memory use: 114.6 MB distribution of cycle lengths: 1 relations: 63729 2 relations: 61946 3 relations: 63103 4 relations: 58944 5 relations: 54442 6 relations: 47896 7 relations: 42279 8 relations: 36259 9 relations: 31189 10+ relations: 108615 heaviest cycle: 19 relations RelProcTime: 204 nfs: commencing msieve linear algebra commencing linear algebra read 568402 cycles cycles contain 1910242 unique relations read 1910242 relations using 20 quadratic characters above 134217530 building initial matrix memory use: 250.5 MB read 568402 cycles matrix is 642864 x 568402 (173.4 MB) with weight 54889111 (96.57/col) sparse part has weight 38641480 (67.98/col) filtering completed in 2 passes matrix is 580879 x 539812 (163.7 MB) with weight 51779712 (95.92/col) sparse part has weight 36427585 (67.48/col) matrix starts at (0, 0) matrix is 580879 x 539812 (163.7 MB) with weight 51779712 (95.92/col) sparse part has weight 36427585 (67.48/col) matrix needs more columns than rows; try adding 2-3% more relations  )
|
|
|
|
|
|
2012-11-12, 12:11
|
#17 |
|
Tribal Bullet
Oct 2004
1101110101012 Posts |
Could you post the .dat and .fb file somewhere? I'll take a look at it.
|
|
|
|
|
2012-11-12, 19:54
|
#18 | |
|
Basketry That Evening!
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88
3×29×83 Posts |
Quote:
(This is another filter run after 4M more SQ were sieved.) Code:
commencing relation filtering estimated available RAM is 12019.3 MB commencing duplicate removal, pass 1 read 10M relations found 1412832 hash collisions in 14011307 relations added 328 free relations commencing duplicate removal, pass 2 found 912301 duplicates and 13099334 unique relations memory use: 49.3 MB reading ideals above 720000 commencing singleton removal, initial pass memory use: 344.5 MB reading all ideals from disk memory use: 367.2 MB keeping 11890692 ideals with weight <= 200, target excess is 116206 commencing in-memory singleton removal begin with 13099334 relations and 11890692 unique ideals reduce to 7796272 relations and 5936084 ideals in 12 passes max relations containing the same ideal: 138 removing 1671936 relations and 1271936 ideals in 400000 cliques commencing in-memory singleton removal begin with 6124336 relations and 5936084 unique ideals reduce to 5937483 relations and 4466682 ideals in 7 passes max relations containing the same ideal: 111 removing 1264918 relations and 864918 ideals in 400000 cliques commencing in-memory singleton removal begin with 4672565 relations and 4466682 unique ideals reduce to 4529389 relations and 3449740 ideals in 7 passes max relations containing the same ideal: 86 removing 1110921 relations and 710921 ideals in 400000 cliques commencing in-memory singleton removal begin with 3418468 relations and 3449740 unique ideals reduce to 3251249 relations and 2557129 ideals in 7 passes max relations containing the same ideal: 69 removing 1027606 relations and 627606 ideals in 400000 cliques commencing in-memory singleton removal begin with 2223643 relations and 2557129 unique ideals reduce to 2022786 relations and 1703657 ideals in 8 passes max relations containing the same ideal: 48 removing 558548 relations and 374218 ideals in 184330 cliques commencing in-memory singleton removal begin with 1464238 relations and 1703657 unique ideals reduce to 1366741 relations and 1223105 ideals in 7 passes max relations containing the same ideal: 37 removing 51485 relations and 42648 ideals in 8837 cliques commencing in-memory singleton removal begin with 1315256 relations and 1223105 unique ideals reduce to 1314274 relations and 1179464 ideals in 5 passes max relations containing the same ideal: 35 relations with 0 large ideals: 522 relations with 1 large ideals: 5634 relations with 2 large ideals: 52649 relations with 3 large ideals: 191959 relations with 4 large ideals: 355475 relations with 5 large ideals: 372515 relations with 6 large ideals: 231664 relations with 7+ large ideals: 103856 commencing 2-way merge reduce to 876843 relation sets and 742033 unique ideals commencing full merge memory use: 86.4 MB found 440637 cycles, need 422233 weight of 422233 cycles is about 29687666 (70.31/cycle) distribution of cycle lengths: 1 relations: 37336 2 relations: 42002 3 relations: 46785 4 relations: 45915 5 relations: 43954 6 relations: 39921 7 relations: 35490 8 relations: 30289 9 relations: 25357 10+ relations: 75184 heaviest cycle: 17 relations commencing cycle optimization start with 2512749 relations pruned 89108 relations memory use: 78.1 MB distribution of cycle lengths: 1 relations: 37336 2 relations: 43082 3 relations: 49011 4 relations: 47781 5 relations: 46215 6 relations: 41386 7 relations: 36522 8 relations: 30717 9 relations: 25120 10+ relations: 65063 heaviest cycle: 17 relations RelProcTime: 247 nfs: commencing msieve linear algebra commencing linear algebra read 422233 cycles cycles contain 1245852 unique relations read 1245852 relations using 20 quadratic characters above 134217158 building initial matrix memory use: 164.7 MB read 422233 cycles matrix is 428326 x 422233 (126.6 MB) with weight 40261482 (95.35/col) sparse part has weight 28126104 (66.61/col) filtering completed in 2 passes matrix is 423024 x 420629 (126.2 MB) with weight 40113862 (95.37/col) sparse part has weight 28026053 (66.63/col) matrix starts at (0, 0) matrix is 423024 x 420629 (126.2 MB) with weight 40113862 (95.37/col) sparse part has weight 28026053 (66.63/col) matrix needs more columns than rows; try adding 2-3% more relations |
|
|
|
|
|
2012-11-13, 19:00
|
#19 |
|
Basketry That Evening!
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88
11100001101012 Posts |
I ran into the third one in three days, this time a C97. I have no idea why it's suddenly happening so much now -- I've been doing aliquots for >6 months just fine, with 2 occurrences about a month ago (as linked in an above post) and now these three.
Code:
commencing relation filtering estimated available RAM is 12019.3 MB commencing duplicate removal, pass 1 found 269102 hash collisions in 3261703 relations commencing duplicate removal, pass 2 found 148182 duplicates and 3113521 unique relations memory use: 12.3 MB reading ideals above 100000 commencing singleton removal, initial pass memory use: 86.1 MB reading all ideals from disk memory use: 95.1 MB keeping 3206660 ideals with weight <= 200, target excess is 19330 commencing in-memory singleton removal begin with 3113521 relations and 3206660 unique ideals reduce to 1495914 relations and 1323918 ideals in 14 passes max relations containing the same ideal: 120 removing 391655 relations and 316869 ideals in 74787 cliques commencing in-memory singleton removal begin with 1104259 relations and 1323918 unique ideals reduce to 1036565 relations and 935128 ideals in 9 passes max relations containing the same ideal: 89 removing 293130 relations and 218343 ideals in 74787 cliques commencing in-memory singleton removal begin with 743435 relations and 935128 unique ideals reduce to 684127 relations and 653062 ideals in 8 passes max relations containing the same ideal: 65 removing 56052 relations and 47410 ideals in 8642 cliques commencing in-memory singleton removal begin with 628075 relations and 653062 unique ideals reduce to 624826 relations and 602357 ideals in 6 passes max relations containing the same ideal: 60 relations with 0 large ideals: 390 relations with 1 large ideals: 152 relations with 2 large ideals: 2974 relations with 3 large ideals: 20670 relations with 4 large ideals: 76932 relations with 5 large ideals: 164984 relations with 6 large ideals: 185557 relations with 7+ large ideals: 173167 commencing 2-way merge reduce to 385258 relation sets and 362789 unique ideals commencing full merge memory use: 40.4 MB found 181425 cycles, need 178989 weight of 178989 cycles is about 12742883 (71.19/cycle) distribution of cycle lengths: 1 relations: 16016 2 relations: 16668 3 relations: 17501 4 relations: 16901 5 relations: 16167 6 relations: 15011 7 relations: 13411 8 relations: 11754 9 relations: 10078 10+ relations: 45482 heaviest cycle: 25 relations commencing cycle optimization start with 1223532 relations pruned 34436 relations memory use: 37.9 MB distribution of cycle lengths: 1 relations: 16016 2 relations: 17037 3 relations: 18068 4 relations: 17380 5 relations: 16676 6 relations: 15400 7 relations: 13697 8 relations: 11875 9 relations: 10150 10+ relations: 42690 heaviest cycle: 24 relations RelProcTime: 55 nfs: commencing msieve linear algebra commencing linear algebra read 178989 cycles cycles contain 608659 unique relations read 608659 relations using 20 quadratic characters above 33554010 building initial matrix memory use: 76.0 MB read 178989 cycles matrix is 180754 x 178989 (53.7 MB) with weight 17274661 (96.51/col) sparse part has weight 12119837 (67.71/col) filtering completed in 2 passes matrix is 179096 x 178511 (53.6 MB) with weight 17229620 (96.52/col) sparse part has weight 12089598 (67.72/col) matrix starts at (0, 0) matrix is 179096 x 178511 (53.6 MB) with weight 17229620 (96.52/col) sparse part has weight 12089598 (67.72/col) matrix needs more columns than rows; try adding 2-3% more relations |
|
|
|
|
2012-11-17, 14:20
|
#20 |
|
Tribal Bullet
Oct 2004
3,541 Posts |
SVN 823 now includes a merge of the fast GPU branch, and a drastically simplified build system. The makefile will now skip generating lots of QS-optimized routines, since they would be optimized for 12-year-old machines that nobody has anymore (and everyone should use Yafu anyway). The default arch to optimize for is now a Core2 (Dubslow: -march=native doesn't work on Mac OS X even with gcc 4.6, though it works everywhere else). You can build now with 'make all' and it should work everywhere.
Let me know if you see anything weird. |
|
|
|
|
2012-11-17, 16:07
|
#21 | |
|
Tribal Bullet
Oct 2004
3,541 Posts |
Quote:
With the latest SVN on win32, the LA starts without a problem. I did add an optimization to the filtering about a month ago (credit to Cyril Bouvier on that, improved filtering is very difficult), so I reverted to the older filtering algorithm, which hasn't changed in years. My filtering log is exactly the same as yours, except that when it says 'commencing cycle optimization' my run starts with 2 relations more than yours. When my LA starts I get Code:
commencing linear algebra read 178989 cycles cycles contain 608754 unique relations read 608754 relations Code:
commencing linear algebra read 178989 cycles cycles contain 608659 unique relations read 608659 relations Code:
matrix is 178809 x 178989 (51.0 MB) with weight 17272200 (96.50/col) sparse part has weight 12117227 (67.70/col) filtering completed in 2 passes matrix is 178675 x 178855 (51.0 MB) with weight 17265607 (96.53/col) sparse part has weight 12114680 (67.73/col) Code:
commencing linear algebra read 178989 cycles cycles contain 608754 unique relations read 608754 relations using 20 quadratic characters above 33554010 building initial matrix memory use: 68.8 MB read 178989 cycles matrix is 178809 x 178989 (52.1 MB) with weight 17609579 (98.38/col) sparse part has weight 12413669 (69.35/col) filtering completed in 2 passes matrix is 178683 x 178863 (52.1 MB) with weight 17602944 (98.42/col) sparse part has weight 12411024 (69.39/col) |
|
|
|
|
|
2012-11-17, 20:24
|
#22 | |
Sep 2009
977 Posts |
The merge of the faststage1 branch lets msieve trunk work on polsel for large tasks much more efficiently, and the simplified build system is another good step
 Quote:
 It lets you the choice between making everyone use -march=native and letting those without a compiler supporting this switch edit their Makefile, or making everyone use -march=core2 and letting those who don't have Core 2 and newer derivatives thereof edit their Makefile... After updating my checkout, I see that I'm carrying two changes locally: * I like having some feedback, right at the beginning of filtering, of the target density (just in case I forgot to raise it): Code:
Index: gnfs/filter/filter.c
===================================================================
--- gnfs/filter/filter.c (révision 823)
+++ gnfs/filter/filter.c (copie de travail)
@@ -184,6 +184,7 @@
if (target_density != 0)
merge->target_density = target_density;
+ logprintf(obj, "target matrix density is %.1f\n", merge->target_density);
filter_make_relsets(obj, filter, merge, extra_needed);
return 0;
}
Last fiddled with by debrouxl on 2012-11-17 at 20:26 |
|
|
|
|
 |
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Msieve 1.53 feedback | xilman | Msieve | 149 | 2018-11-12 06:37 |
| Msieve 1.50 feedback | firejuggler | Msieve | 99 | 2013-02-17 11:53 |
| Msieve 1.43 feedback | Jeff Gilchrist | Msieve | 47 | 2009-11-24 15:53 |
| Msieve 1.42 feedback | Andi47 | Msieve | 167 | 2009-10-18 19:37 |
| Msieve 1.41 Feedback | Batalov | Msieve | 130 | 2009-06-09 16:01 |
