[QUOTE=bdodson;136736] (Greg seems
less worried). -Bruce[/QUOTE]
Not less worried, just hopefully optimistic. :smile:

[QUOTE=bdodson;132633]
[code]
%CPU
10129 bad0 25 0 5326m 5.2g 476 S 591 4.7 65:45.73 msieve
10123 bad0 25 0 4661m 4.5g 476 S 398 4.1 65:02.85 msieve
10105 bad0 18 0 4766m 4.6g 476 S 310 4.2 57:28.75 msieve
[/code]

Uhm, that's 14 threads, four each on the first two matrices,
six on the new one, 10,257+. A monster, by the way, 15.479M^2,
... [/QUOTE]

OK, take 2. Adding 24 hrs of sieving brought the matrix down to
13.00M^2, smallest so far; taking 4.4g of memory.

Filtering wasn't fun; it ran for an hour, wrote the free relns to
msieve.dat then crashed early, some 10M into reading the relns.
Non-duplicates are now 193M; another 23:24hrs for the rest of the
filtering. New msieve binary, 1.36; more density than before. Looks
like 6 weeks - 2 months. I'm voting for more over-sieving (on the
first pass!) on subsequent numbers. We set the sieving region for
7,313+ so the current number in sieving, 7,313- should be good
(real roots; cf. 10,257- versus 10,257+, with region set for 257-).

Perhaps a larger region for the next harder one, 6,392+ c262. -Bruce

Trouble with msieve
 

I just ran into trouble with a c120(msieve-1.36):
[CODE]Sat Aug 9 20:43:43 2008 Msieve v. 1.36
Sat Aug 9 20:43:43 2008 random seeds: db8bc0d2 9dd4e38e
Sat Aug 9 20:43:43 2008 factoring 571767677650874534282700286134907118688339588867273119662614359544426774185905329344208308020066200075418286173232361871 (120 digits)
Sat Aug 9 20:43:44 2008 no P-1/P+1/ECM available, skipping
Sat Aug 9 20:43:44 2008 commencing number field sieve (120-digit input)
Sat Aug 9 20:43:44 2008 R0: -101559385642140526100501
Sat Aug 9 20:43:44 2008 R1: 1528704765731
Sat Aug 9 20:43:44 2008 A0: -10576278874140472243441982304
Sat Aug 9 20:43:44 2008 A1: 949797254285423487604584
Sat Aug 9 20:43:44 2008 A2: 10578947574769476362
Sat Aug 9 20:43:44 2008 A3: -1280256632925143
Sat Aug 9 20:43:44 2008 A4: -4533218788
Sat Aug 9 20:43:44 2008 A5: 52920
Sat Aug 9 20:43:44 2008 size score = 2.446874e-12, Murphy alpha = -5.583043, combined = 1.573403e-11
Sat Aug 9 20:44:52 2008 restarting with 30188288 relations
Sat Aug 9 20:44:52 2008
Sat Aug 9 20:44:52 2008 commencing relation filtering
Sat Aug 9 20:44:52 2008 commencing duplicate removal, pass 1
Sat Aug 9 20:46:20 2008 error -15 reading relation 12470891
Sat Aug 9 20:47:08 2008 error -15 reading relation 18795548
Sat Aug 9 20:47:54 2008 error -15 reading relation 24669845
Sat Aug 9 20:48:33 2008 error -15 reading relation 30185543
Sat Aug 9 20:48:33 2008 found 5747909 hash collisions in 30188284 relations
Sat Aug 9 20:48:33 2008 commencing duplicate removal, pass 2
Sat Aug 9 20:49:42 2008 found 5466401 duplicates and 24721883 unique relations
Sat Aug 9 20:49:42 2008 memory use: 153.2 MB
Sat Aug 9 20:49:50 2008 ignoring smallest 1136932 rational and 1134421 algebraic ideals
Sat Aug 9 20:49:50 2008 filtering rational ideals above 17760256
Sat Aug 9 20:49:50 2008 filtering algebraic ideals above 17760256
Sat Aug 9 20:49:50 2008 need 3407029 more relations than ideals
Sat Aug 9 20:49:50 2008 commencing singleton removal, pass 1
Sat Aug 9 20:52:56 2008 relations with 0 large ideals: 3487252
Sat Aug 9 20:52:56 2008 relations with 1 large ideals: 10955763
Sat Aug 9 20:52:56 2008 relations with 2 large ideals: 8898490
Sat Aug 9 20:52:56 2008 relations with 3 large ideals: 1327025
Sat Aug 9 20:52:56 2008 relations with 4 large ideals: 53185
Sat Aug 9 20:52:56 2008 relations with 5 large ideals: 168
Sat Aug 9 20:52:56 2008 relations with 6 large ideals: 0
Sat Aug 9 20:52:56 2008 relations with 7+ large ideals: 0
Sat Aug 9 20:52:56 2008 24721883 relations and about 10181026 large ideals
Sat Aug 9 20:52:56 2008 commencing singleton removal, pass 2
Sat Aug 9 20:56:03 2008 found 2998373 singletons
Sat Aug 9 20:56:03 2008 current dataset: 21723510 relations and about 7086145 large ideals
Sat Aug 9 20:56:03 2008 commencing singleton removal, pass 3
Sat Aug 9 20:58:49 2008 found 170254 singletons
Sat Aug 9 20:58:49 2008 current dataset: 21553256 relations and about 6915505 large ideals
Sat Aug 9 20:58:49 2008 commencing singleton removal, final pass
Sat Aug 9 21:01:51 2008 memory use: 269.8 MB
Sat Aug 9 21:01:52 2008 commencing in-memory singleton removal
Sat Aug 9 21:01:53 2008 begin with 21553256 relations and 7300467 unique ideals
Sat Aug 9 21:02:01 2008 reduce to 21271774 relations and 7018154 ideals in 5 passes
Sat Aug 9 21:02:01 2008 max relations containing the same ideal: 61
Sat Aug 9 21:02:07 2008 removing 1099701 relations and 699701 ideals in 400000 cliques
Sat Aug 9 21:02:08 2008 commencing in-memory singleton removal
Sat Aug 9 21:02:09 2008 begin with 20172073 relations and 7018154 unique ideals
Sat Aug 9 21:02:15 2008 reduce to 20159501 relations and 6305690 ideals in 4 passes
Sat Aug 9 21:02:15 2008 max relations containing the same ideal: 55
Sat Aug 9 21:02:20 2008 removing 835727 relations and 435727 ideals in 400000 cliques
Sat Aug 9 21:02:21 2008 commencing in-memory singleton removal
Sat Aug 9 21:02:22 2008 begin with 19323774 relations and 6305690 unique ideals
Sat Aug 9 21:02:27 2008 reduce to 19297329 relations and 5842729 ideals in 4 passes
Sat Aug 9 21:02:27 2008 max relations containing the same ideal: 53
Sat Aug 9 21:02:32 2008 removing 811540 relations and 411540 ideals in 400000 cliques
Sat Aug 9 21:02:33 2008 commencing in-memory singleton removal
Sat Aug 9 21:02:34 2008 begin with 18485789 relations and 5842729 unique ideals
Sat Aug 9 21:02:38 2008 reduce to 18479426 relations and 5424668 ideals in 3 passes
Sat Aug 9 21:02:38 2008 max relations containing the same ideal: 51
Sat Aug 9 21:02:42 2008 removing 803157 relations and 403157 ideals in 400000 cliques
Sat Aug 9 21:02:43 2008 commencing in-memory singleton removal
Sat Aug 9 21:02:44 2008 begin with 17676269 relations and 5424668 unique ideals
Sat Aug 9 21:02:47 2008 reduce to 17673937 relations and 5019146 ideals in 3 passes
Sat Aug 9 21:02:47 2008 max relations containing the same ideal: 49
Sat Aug 9 21:02:51 2008 removing 800504 relations and 400504 ideals in 400000 cliques
Sat Aug 9 21:02:52 2008 commencing in-memory singleton removal
Sat Aug 9 21:02:53 2008 begin with 16873433 relations and 5019146 unique ideals
Sat Aug 9 21:02:55 2008 reduce to 16873149 relations and 4618355 ideals in 2 passes
Sat Aug 9 21:02:55 2008 max relations containing the same ideal: 49
Sat Aug 9 21:02:59 2008 removing 800057 relations and 400057 ideals in 400000 cliques
Sat Aug 9 21:02:59 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:00 2008 begin with 16073092 relations and 4618355 unique ideals
Sat Aug 9 21:03:02 2008 reduce to 16073078 relations and 4218284 ideals in 2 passes
Sat Aug 9 21:03:02 2008 max relations containing the same ideal: 48
Sat Aug 9 21:03:05 2008 removing 800002 relations and 400002 ideals in 400000 cliques
Sat Aug 9 21:03:05 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:06 2008 begin with 15273076 relations and 4218284 unique ideals
Sat Aug 9 21:03:08 2008 reduce to 15273075 relations and 3818281 ideals in 2 passes
Sat Aug 9 21:03:08 2008 max relations containing the same ideal: 48
Sat Aug 9 21:03:11 2008 removing 268524 relations and 134263 ideals in 134261 cliques
Sat Aug 9 21:03:11 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:12 2008 begin with 15004551 relations and 3818281 unique ideals
Sat Aug 9 21:03:13 2008 reduce to 15004551 relations and 3684018 ideals in 1 passes
Sat Aug 9 21:03:13 2008 max relations containing the same ideal: 45
Sat Aug 9 21:03:16 2008 too much excess; switching to 3-cliques
Sat Aug 9 21:03:19 2008 removing 1858745 relations and 729482 ideals in 400000 cliques
Sat Aug 9 21:03:20 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:20 2008 begin with 13145806 relations and 3684018 unique ideals
Sat Aug 9 21:03:23 2008 reduce to 13141426 relations and 2950034 ideals in 3 passes
Sat Aug 9 21:03:23 2008 max relations containing the same ideal: 41
Sat Aug 9 21:03:26 2008 removing 1350177 relations and 480270 ideals in 400000 cliques
Sat Aug 9 21:03:26 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:27 2008 begin with 11791249 relations and 2950034 unique ideals
Sat Aug 9 21:03:29 2008 reduce to 11787916 relations and 2466330 ideals in 3 passes
Sat Aug 9 21:03:29 2008 max relations containing the same ideal: 40
Sat Aug 9 21:03:32 2008 removing 1244959 relations and 426482 ideals in 400000 cliques
Sat Aug 9 21:03:32 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:32 2008 begin with 10542957 relations and 2466330 unique ideals
Sat Aug 9 21:03:34 2008 reduce to 10542081 relations and 2038945 ideals in 3 passes
Sat Aug 9 21:03:34 2008 max relations containing the same ideal: 36
Sat Aug 9 21:03:36 2008 removing 1219126 relations and 412816 ideals in 400000 cliques
Sat Aug 9 21:03:36 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:37 2008 begin with 9322955 relations and 2038945 unique ideals
Sat Aug 9 21:03:38 2008 reduce to 9322862 relations and 1626035 ideals in 3 passes
Sat Aug 9 21:03:38 2008 max relations containing the same ideal: 32
Sat Aug 9 21:03:40 2008 removing 719777 relations and 271318 ideals in 266393 cliques
Sat Aug 9 21:03:40 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:40 2008 begin with 8603085 relations and 1626035 unique ideals
Sat Aug 9 21:03:41 2008 reduce to 8603021 relations and 1354653 ideals in 2 passes
Sat Aug 9 21:03:41 2008 max relations containing the same ideal: 29
Sat Aug 9 21:03:42 2008 removing 213613 relations and 72766 ideals in 71054 cliques
Sat Aug 9 21:03:42 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:42 2008 begin with 8389408 relations and 1354653 unique ideals
Sat Aug 9 21:03:43 2008 reduce to 8389395 relations and 1281874 ideals in 2 passes
Sat Aug 9 21:03:43 2008 max relations containing the same ideal: 29
Sat Aug 9 21:03:44 2008 removing 114997 relations and 38807 ideals in 38273 cliques
Sat Aug 9 21:03:44 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:44 2008 begin with 8274398 relations and 1281874 unique ideals
Sat Aug 9 21:03:45 2008 reduce to 8274397 relations and 1243066 ideals in 2 passes
Sat Aug 9 21:03:45 2008 max relations containing the same ideal: 28
Sat Aug 9 21:03:46 2008 removing 63421 relations and 21286 ideals in 21110 cliques
Sat Aug 9 21:03:46 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:46 2008 begin with 8210976 relations and 1243066 unique ideals
Sat Aug 9 21:03:47 2008 reduce to 8210975 relations and 1221779 ideals in 2 passes
Sat Aug 9 21:03:47 2008 max relations containing the same ideal: 28
Sat Aug 9 21:03:47 2008 removing 34756 relations and 11634 ideals in 11573 cliques
Sat Aug 9 21:03:47 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:48 2008 begin with 8176219 relations and 1221779 unique ideals
Sat Aug 9 21:03:48 2008 reduce to 8176219 relations and 1210145 ideals in 1 passes
Sat Aug 9 21:03:48 2008 max relations containing the same ideal: 28
Sat Aug 9 21:03:49 2008 too much excess; switching to 4-cliques
Sat Aug 9 21:03:50 2008 removing 2184038 relations and 600135 ideals in 390154 cliques
Sat Aug 9 21:03:51 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:51 2008 begin with 5992181 relations and 1210145 unique ideals
Sat Aug 9 21:03:51 2008 reduce to 5990541 relations and 608289 ideals in 3 passes
Sat Aug 9 21:03:51 2008 max relations containing the same ideal: 24
Sat Aug 9 21:03:52 2008 removing 852858 relations and 248799 ideals in 202842 cliques
Sat Aug 9 21:03:52 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:52 2008 begin with 5137683 relations and 608289 unique ideals
Sat Aug 9 21:03:53 2008 reduce to 5137591 relations and 359395 ideals in 2 passes
Sat Aug 9 21:03:53 2008 max relations containing the same ideal: 24
Sat Aug 9 21:03:53 2008 removing 344086 relations and 92853 ideals in 84767 cliques
Sat Aug 9 21:03:53 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:53 2008 begin with 4793505 relations and 359395 unique ideals
Sat Aug 9 21:03:53 2008 reduce to 4793499 relations and 266536 ideals in 2 passes
Sat Aug 9 21:03:53 2008 max relations containing the same ideal: 23
Sat Aug 9 21:03:53 2008 removing 131891 relations and 34094 ideals in 32818 cliques
Sat Aug 9 21:03:53 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:53 2008 begin with 4661608 relations and 266536 unique ideals
Sat Aug 9 21:03:54 2008 reduce to 4661608 relations and 232442 ideals in 1 passes
Sat Aug 9 21:03:54 2008 max relations containing the same ideal: 23
Sat Aug 9 21:03:54 2008 removing 49615 relations and 12575 ideals in 12369 cliques
Sat Aug 9 21:03:54 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:54 2008 begin with 4611993 relations and 232442 unique ideals
Sat Aug 9 21:03:54 2008 reduce to 4611993 relations and 219867 ideals in 1 passes
Sat Aug 9 21:03:54 2008 max relations containing the same ideal: 23
Sat Aug 9 21:03:54 2008 too much excess; switching to 5-cliques
Sat Aug 9 21:03:54 2008 removing 657459 relations and 138634 ideals in 107701 cliques
Sat Aug 9 21:03:54 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:54 2008 begin with 3954534 relations and 219867 unique ideals
Sat Aug 9 21:03:54 2008 reduce to 3954533 relations and 81232 ideals in 2 passes
Sat Aug 9 21:03:54 2008 max relations containing the same ideal: 22
Sat Aug 9 21:03:55 2008 removing 117253 relations and 24941 ideals in 23756 cliques
Sat Aug 9 21:03:55 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:55 2008 begin with 3837280 relations and 81232 unique ideals
Sat Aug 9 21:03:55 2008 reduce to 3837280 relations and 56291 ideals in 1 passes
Sat Aug 9 21:03:55 2008 max relations containing the same ideal: 22
Sat Aug 9 21:03:55 2008 too much excess; switching to 6-cliques
Sat Aug 9 21:03:55 2008 removing 223861 relations and 38722 ideals in 34612 cliques
Sat Aug 9 21:03:55 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:55 2008 begin with 3613419 relations and 56291 unique ideals
Sat Aug 9 21:03:55 2008 reduce to 3613419 relations and 17569 ideals in 1 passes
Sat Aug 9 21:03:55 2008 max relations containing the same ideal: 22
Sat Aug 9 21:03:55 2008 too much excess; switching to 7-cliques
Sat Aug 9 21:03:55 2008 removing 85337 relations and 12714 ideals in 12101 cliques
Sat Aug 9 21:03:55 2008 commencing in-memory singleton removal
Sat Aug 9 21:03:55 2008 begin with 3528082 relations and 17569 unique ideals
Sat Aug 9 21:03:55 2008 reduce to 3528082 relations and 4855 ideals in 1 passes
Sat Aug 9 21:03:55 2008 max relations containing the same ideal: 21
Sat Aug 9 21:03:55 2008 too much excess; switching to 8-cliques
Sat Aug 9 21:03:55 2008 too much excess; switching to 9-cliques
Sat Aug 9 21:03:55 2008 too much excess; switching to 10-cliques
Sat Aug 9 21:03:55 2008 too much excess; switching to 11-cliques
Sat Aug 9 21:03:55 2008 too much excess; switching to 12-cliques
Sat Aug 9 21:03:55 2008 too much excess; switching to 13-cliques
Sat Aug 9 21:03:55 2008 too much excess; switching to 14-cliques
Sat Aug 9 21:03:55 2008 too much excess; switching to 15-cliques
Sat Aug 9 21:03:56 2008 too much excess; switching to 16-cliques
Sat Aug 9 21:03:56 2008 too much excess; switching to 17-cliques[/CODE]
and the "too much excess; switching to nn-cliques" is going on endlessy.
And I don't think there is any excess, I think there are too less relations.

Regards,
--nugget

This has been discussed here - [URL]http://mersenneforum.org/showthread.php?t=9816[/URL] ...

Practically, you may want to save a copy of your *.dat file and make a [I]half[/I]-copy of it in the original location by doing something like
[code]mv my.dat my30M.dat
head -15000000 my30M.dat > my.dat
vi *.job
# edit [I]q0: xxxx[/I] to be the value of q1-10
# restart the perl script[/code]
and if fails, play with number 15000000. If it will start more sieveing, increase, if it will do "too much excess", decrease but not too much. You probably need 13-15M raw rels in your case. There's a certain (rather wide) safe range where filtering converges.

Thanks,worked.
In the future,I'll ask msieve how many rels I need:smile:.

--nugget

It's actually a little surprising how difficult it is to automate NFS postprocessing when there is a very large number of excess relations; I didn't expect that you could run out of cliques. I guess there should be a limit there too :)

Bug
 

I probably have just discovered another postprocessing bug:
for small SNFS numbers (c100) the square root fails with:
[CODE]dependency does not form a congruence of squares![/CODE]
Post if the full logfile is needed.

Regards,
nuggetprime

[QUOTE=nuggetprime;139372]I probably have just discovered another postprocessing bug:
for small SNFS numbers (c100) the square root fails with:
[CODE]dependency does not form a congruence of squares![/CODE]
Post if the full logfile is needed.
[/QUOTE]
We can start with PM'ing the logfile, but I doubt it will tell me much. For a small job like that, if the relations can be zipped up to be under 100 MB maybe you can post it on rapidshare.com; otherwise, maybe somebody can volunteer some ftp space?

Do multiple dependencies have this problem?

Yes,the gzipped rels file is just 26 MB. And the problem occurs for all deps.
I'll put the file on rapidshare.

--nugget

[QUOTE=bdodson;137249]OK, take 2. Adding 24 hrs of sieving brought the matrix down to
13.00M^2, smallest so far; taking 4.4g of memory.

Filtering wasn't fun; it ran for an hour, wrote the free relns to
msieve.dat then crashed early, some 10M into reading the relns.
Non-duplicates are now 193M; another 23:24hrs for the rest of the
filtering. New msieve binary, 1.36; more density than before. Looks
like 6 weeks - 2 months. ...

Perhaps a larger region for the next harder one, 6,392+ c262. -Bruce[/QUOTE]

Not sure that this is a representative case for how to deal with a
matrix miss; but the extra sieving brought the size down from
"A monster, ... 15.479M^2" to this 13.0M^2, along with switching
to a newer binary. Not sure that anyone noticed (although Bob's
reporting an extended quietness on my part), but the "6 weeks to
2 months" was something of a dissapointment, compared with the
harder matrix. I am just under the "2 months", with today's
c208 = p61*p148, but if I had continued running on the quadcore
compute server, the matrix would more likely have extended for
another month or two. Looks like I got in three factorizations, plus
the miss on 10,257+ C208, while the server was in pre-production;
but it's far less suitable for threaded computing under user pressure.
Or something; our High Performance Computing group is still in
evaluation, as to why the more expensive/core SGI doesn't seem
to have been worth the premium.

Anyway, ... The second half of this matrix was done in just 10 days
on a stand-alone dual intel/quadcore workstation, with 8Gb. Part of
a life-cycle replacement for older linux workstations, with the extra
quad and 4Gb supplied by math dept funds. Hums along at nearly
4 cpudays/day; while 4 threads on the server has been as bad as
1 cpuday/day (with a 30Gb job using 8 mpi processes, one of those
8 processes using 8 threads, for c. 15 compute cores), and is currently
around 2 cpuday/day (with only what seems like more plausibly configured
heavy jobs). So that's c208 = p61*p148 with p61 =

6621650829759394867011952029659651154933062041670057842400167

The winning workstation has gone on the the second half of the
7,311+ matrix, c. 17 days to go (pending user reboots ...).

On the clusters, 6,392+ c262 finished sieving and was shipped off
to Cal State, Fullerton (and Greg). We made a last minute addition
of 5,389+ at difficulty 272 (a 2/3 ratio would sugest c. c180 gnfs,
perhaps comparable to the current 5, 421- project). Trying to keep
the matrix runtime from growing too fast, we kept 90M factorbase
limits and 31-bit large primes. I'm about to finish 20M-500M rational,
plus 50M-90M algebraic; looks like we may need to bump those limits
on the next hard Childers/Dodson number.

-Bruce

[QUOTE=bdodson;140263](although Bob's
reporting an extended quietness on my part), -Bruce[/QUOTE]


Actually, I meant Franke, Aoki, CWI, et. al., rather than you.