[QUOTE=fivemack;133332]@frmky: would it be possible to run the GNFS benchmark from [url]http://www.mersenneforum.org/showpost.php?p=126056&postcount=1[/url] on various numbers of cores on the Barcelona? I am curious to see how much of an effect memory contention has on the Franke lattice siever, and have no Barcelona-based machine to hand.[/QUOTE]

The processors are 2 GHz Barcelonas, and I used the 64-bit gnfs-lasieve4I14e with ASM optimizations. One instance running alone on a processor:
[CODE]total yield: 1096, q=50001037 (0.29429 sec/rel)[/CODE]

Four instances running simultaneously on the 4 cores of a single processor:
[CODE]total yield: 1096, q=50001037 (0.30738 sec/rel)
total yield: 1096, q=50001037 (0.30733 sec/rel)
total yield: 1096, q=50001037 (0.30535 sec/rel)
total yield: 1096, q=50001037 (0.30728 sec/rel)[/CODE]

Eight instances running simultaneously on two processors:
[CODE]total yield: 1096, q=50001037 (0.30633 sec/rel)
total yield: 1096, q=50001037 (0.30646 sec/rel)
total yield: 1096, q=50001037 (0.30628 sec/rel)
total yield: 1096, q=50001037 (0.30632 sec/rel)
total yield: 1096, q=50001037 (0.29578 sec/rel)
total yield: 1096, q=50001037 (0.29508 sec/rel)
total yield: 1096, q=50001037 (0.29594 sec/rel)
total yield: 1096, q=50001037 (0.29587 sec/rel)[/CODE]

The other 6 processors are busy, so this is as far as I could go for now. Looks like a <5% performance hit at most.

Greg

Here's the same for the gnfs-lasieve4I15e siever...

One instance running alone on a processor:
[CODE]total yield: 2206, q=50001037 (0.43513 sec/rel)[/CODE]

Four instances running simultaneously on the 4 cores of a single processor:
[CODE]total yield: 2206, q=50001037 (0.45859 sec/rel)
total yield: 2206, q=50001037 (0.45864 sec/rel)
total yield: 2206, q=50001037 (0.45853 sec/rel)
total yield: 2206, q=50001037 (0.45864 sec/rel)[/CODE]

Eight instances running simultaneously on two processors:
[CODE]total yield: 2206, q=50001037 (0.46066 sec/rel)
total yield: 2206, q=50001037 (0.46070 sec/rel)
total yield: 2206, q=50001037 (0.46064 sec/rel)
total yield: 2206, q=50001037 (0.46066 sec/rel)
total yield: 2206, q=50001037 (0.44216 sec/rel)
total yield: 2206, q=50001037 (0.44218 sec/rel)
total yield: 2206, q=50001037 (0.44212 sec/rel)
total yield: 2206, q=50001037 (0.44215 sec/rel)[/CODE]

Greg

Thanks very much for those timings; there's really no contention problem at all, so it would make sense to buy a Phenom X4 system for sieving and matrix jobs once I've piled up the necessary $650.

[quote=frmky;133280]Here's the log of the filtering for 12,241- that I did here up to the point that it ran out of memory. At this point, Richard volunteered to complete it.

[code]Mon Mar 17 18:12:06 2008 Msieve v. 1.33
Mon Mar 17 18:12:06 2008 random seeds: cb7c74ac 9de22da8
Mon Mar 17 18:12:06 2008 factoring 10997338991237101669052555789135086631048469132167079175601564990736546875635453637787116099924623852328671942230474398812245482555847896811945602723714843777323127914355152190001139337968698673482733772849274629055901187581713787723561860855660952506650948701 (260 digits)
Mon Mar 17 18:12:09 2008 no P-1/P+1/ECM available, skipping
Mon Mar 17 18:12:09 2008 commencing number field sieve (260-digit input)
Mon Mar 17 18:12:09 2008 R0: -14697715679690864505827555550150426126974976
Mon Mar 17 18:12:09 2008 R1: 1
Mon Mar 17 18:12:09 2008 A0: -1
Mon Mar 17 18:12:09 2008 A1: 0
Mon Mar 17 18:12:09 2008 A2: 0
Mon Mar 17 18:12:09 2008 A3: 0
Mon Mar 17 18:12:09 2008 A4: 0
Mon Mar 17 18:12:09 2008 A5: 0
Mon Mar 17 18:12:09 2008 A6: 12
Mon Mar 17 18:12:09 2008 size score = 1.585606e-12, Murphy alpha = 1.332237, combined = 1.083642e-12
Mon Mar 17 18:24:50 2008 restarting with 153096911 relations
Mon Mar 17 18:24:55 2008 added 45828 free relations
Mon Mar 17 18:24:55 2008
Mon Mar 17 18:24:55 2008 commencing relation filtering
Mon Mar 17 18:24:55 2008 commencing duplicate removal, pass 1
... errors snipped ...
Mon Mar 17 18:52:01 2008 found 30908445 hash collisions in 153141746 relations
Mon Mar 17 18:52:01 2008 commencing duplicate removal, pass 2
Mon Mar 17 19:04:50 2008 found 108 duplicates and 153141638 unique relations
Mon Mar 17 19:04:50 2008 memory use: 973.5 MB
Mon Mar 17 19:05:48 2008 ignoring smallest 5487658 rational and 5485186 algebraic ideals
Mon Mar 17 19:05:48 2008 filtering rational ideals above 94961664
Mon Mar 17 19:05:48 2008 filtering algebraic ideals above 94961664
Mon Mar 17 19:05:48 2008 need 16459266 more relations than ideals
Mon Mar 17 19:05:48 2008 commencing singleton removal, pass 1
Mon Mar 17 19:33:56 2008 relations with 0 large ideals: 1084395
Mon Mar 17 19:33:56 2008 relations with 1 large ideals: 8254914
Mon Mar 17 19:33:56 2008 relations with 2 large ideals: 28009207
Mon Mar 17 19:33:56 2008 relations with 3 large ideals: 50033828
Mon Mar 17 19:33:56 2008 relations with 4 large ideals: 47051982
Mon Mar 17 19:33:56 2008 relations with 5 large ideals: 18707312
Mon Mar 17 19:33:56 2008 relations with 6 large ideals: 0
Mon Mar 17 19:33:56 2008 relations with 7+ large ideals: 0
Mon Mar 17 19:33:56 2008 153141638 relations and about 91579140 large ideals
Mon Mar 17 19:33:56 2008 commencing singleton removal, pass 2
Mon Mar 17 20:02:24 2008 found 18506999 singletons
Mon Mar 17 20:02:24 2008 current dataset: 134634639 relations and about 72156819 large ideals
Mon Mar 17 20:02:24 2008 commencing singleton removal, pass 3
Mon Mar 17 20:27:33 2008 relations with 0 large ideals: 1084395
Mon Mar 17 20:27:33 2008 relations with 1 large ideals: 7915991
Mon Mar 17 20:27:33 2008 relations with 2 large ideals: 25779279
Mon Mar 17 20:27:33 2008 relations with 3 large ideals: 44277071
Mon Mar 17 20:27:33 2008 relations with 4 large ideals: 40135240
Mon Mar 17 20:27:33 2008 relations with 5 large ideals: 15442663
Mon Mar 17 20:27:33 2008 relations with 6 large ideals: 0
Mon Mar 17 20:27:33 2008 relations with 7+ large ideals: 0
Mon Mar 17 20:27:33 2008 134634639 relations and about 103032699 large ideals
Mon Mar 17 20:27:33 2008 commencing singleton removal, pass 4
Mon Mar 17 20:53:49 2008 found 26307678 singletons
Mon Mar 17 20:53:49 2008 current dataset: 108326961 relations and about 74498773 large ideals
Mon Mar 17 20:53:49 2008 commencing singleton removal, pass 5
Mon Mar 17 21:14:46 2008 relations with 0 large ideals: 1084395
Mon Mar 17 21:14:46 2008 relations with 1 large ideals: 7359120
Mon Mar 17 21:14:46 2008 relations with 2 large ideals: 22336110
Mon Mar 17 21:14:46 2008 relations with 3 large ideals: 35882152
Mon Mar 17 21:14:46 2008 relations with 4 large ideals: 30547144
Mon Mar 17 21:14:46 2008 relations with 5 large ideals: 11118040
Mon Mar 17 21:14:46 2008 relations with 6 large ideals: 0
Mon Mar 17 21:14:46 2008 relations with 7+ large ideals: 0
Mon Mar 17 21:14:46 2008 108326961 relations and about 90134063 large ideals
Mon Mar 17 21:14:46 2008 commencing singleton removal, pass 6
Mon Mar 17 21:37:41 2008 found 19828612 singletons
Mon Mar 17 21:37:41 2008 current dataset: 88498349 relations and about 68757462 large ideals
Mon Mar 17 21:37:41 2008 commencing singleton removal, pass 7
Mon Mar 17 22:00:22 2008 found 5798183 singletons
Mon Mar 17 22:00:22 2008 current dataset: 82700166 relations and about 62802258 large ideals
Mon Mar 17 22:00:22 2008 commencing singleton removal, pass 8
Mon Mar 17 22:23:09 2008 found 1654614 singletons
Mon Mar 17 22:23:09 2008 current dataset: 81045552 relations and about 61133708 large ideals
Mon Mar 17 22:23:09 2008 commencing singleton removal, pass 9
Mon Mar 17 22:46:03 2008 found 466614 singletons
Mon Mar 17 22:46:03 2008 current dataset: 80578938 relations and about 60665960 large ideals
Mon Mar 17 22:46:03 2008 commencing singleton removal, final pass
Mon Mar 17 23:16:02 2008 memory use: 1533.9 MB
Mon Mar 17 23:16:02 2008 commencing in-memory singleton removal
Mon Mar 17 23:16:13 2008 begin with 80578938 relations and 68016037 unique ideals
Mon Mar 17 23:19:47 2008 reduce to 69710530 relations and 56908564 ideals in 19 passes
Mon Mar 17 23:19:47 2008 max relations containing the same ideal: 32
Mon Mar 17 23:20:05 2008 filtering rational ideals above 750000
Mon Mar 17 23:20:05 2008 filtering algebraic ideals above 750000
Mon Mar 17 23:20:05 2008 need 120350 more relations than ideals
Mon Mar 17 23:20:05 2008 commencing singleton removal, final pass
Tue Mar 18 00:20:12 2008 keeping 62383957 ideals with weight <= 20, new excess is 5496085
Tue Mar 18 00:22:20 2008 memory use: 1919.4 MB
Tue Mar 18 00:22:20 2008 commencing in-memory singleton removal
Tue Mar 18 00:22:33 2008 begin with 69710530 relations and 62383957 unique ideals
Tue Mar 18 00:25:08 2008 reduce to 69685513 relations and 62358938 ideals in 11 passes
Tue Mar 18 00:25:08 2008 max relations containing the same ideal: 20
Tue Mar 18 00:26:24 2008 removing 4715610 relations and 4315610 ideals in 400000 cliques
Tue Mar 18 00:26:28 2008 commencing in-memory singleton removal
Tue Mar 18 00:26:40 2008 begin with 64969903 relations and 62358938 unique ideals
Tue Mar 18 00:28:52 2008 reduce to 64755292 relations and 57826905 ideals in 10 passes
Tue Mar 18 00:28:52 2008 max relations containing the same ideal: 20
Tue Mar 18 00:29:56 2008 removing 3405744 relations and 3005744 ideals in 400000 cliques
Tue Mar 18 00:29:59 2008 commencing in-memory singleton removal
Tue Mar 18 00:30:10 2008 begin with 61349548 relations and 57826905 unique ideals
Tue Mar 18 00:32:00 2008 reduce to 61225585 relations and 54696419 ideals in 9 passes
Tue Mar 18 00:32:00 2008 max relations containing the same ideal: 20
Tue Mar 18 00:32:59 2008 removing 1318289 relations and 1164582 ideals in 153707 cliques
Tue Mar 18 00:33:01 2008 commencing in-memory singleton removal
Tue Mar 18 00:33:12 2008 begin with 59907296 relations and 54696419 unique ideals
Tue Mar 18 00:34:33 2008 reduce to 59887608 relations and 53512110 ideals in 7 passes
Tue Mar 18 00:34:33 2008 max relations containing the same ideal: 20
Tue Mar 18 00:34:55 2008 relations with 0 large ideals: 159911
Tue Mar 18 00:34:55 2008 relations with 1 large ideals: 1341246
Tue Mar 18 00:34:55 2008 relations with 2 large ideals: 6124530
Tue Mar 18 00:34:55 2008 relations with 3 large ideals: 14660687
Tue Mar 18 00:34:55 2008 relations with 4 large ideals: 19277850
Tue Mar 18 00:34:55 2008 relations with 5 large ideals: 13442993
Tue Mar 18 00:34:55 2008 relations with 6 large ideals: 4155712
Tue Mar 18 00:34:55 2008 relations with 7+ large ideals: 724679
Tue Mar 18 00:34:55 2008 commencing 2-way merge
Tue Mar 18 00:36:46 2008 reduce to 36343498 relation sets and 29968000 unique ideals
Tue Mar 18 00:36:46 2008 commencing full merge[/code]

Greg[/quote]

I just noticed this... only 108 duplicates out of 150 million relations!?!? Did you do some duplicate filtering on the data set prior to running msieve?

[QUOTE=bsquared;133427]I just noticed this... only 108 duplicates out of 150 million relations!?!? Did you do some duplicate filtering on the data set prior to running msieve?[/QUOTE]
Yep. :smile:

[QUOTE=bsquared;133427]I just noticed this... only 108 duplicates out of 150 million relations!?!? Did you do some duplicate filtering on the data set prior to running msieve?[/QUOTE]

Just noticed this myself, suppose I oughtn't to be muttering about
under-sieving on smaller/easier numbers than the 153096911 relations
for 12,241- c260. I still have filter files for three of the four matrices
running here now:

3,536+ had "279613 duplicates and 173806695 unique relations"
(after removing most of the duplicates with Greg's binary, then
adding some stragglers)

10,257- had "19 duplicates and 180016677 unique relations", and

10,257+ had "1 duplicates and 167389323 unique relations"
(which I was characterizing as undersieving, relative to the
above two)

The filter log for 3,547- got lost in a /vtmp refreshing, but the
duplicate remover reported "Found 178670323 unique" by the same
protocall as used on the 257's, so that's likely close to the number
found in -nc1 for 3,547-. Small wonder that the 12,241- matrix
was such a monster, at 18.72M^2. -Bruce

[QUOTE=bdodson;133436]Small wonder that the 12,241- matrix
was such a monster, at 18.72M^2. -Bruce[/QUOTE]

Yeah, it could have used some more sieving, but I ran out of cluster time and this was before Bruce got sieving working in Condor. So I went with what I had. Anyway, my original goal was to push msieve further, and the 18.7M matrix did that as well as the 1247.77 million bit coefficients in the sqrt. I'm still quite astonished that it all went without a hitch. (No offense Jason!)

Greg

[QUOTE=frmky;133438]Anyway, my original goal was to push msieve further, and the 18.7M matrix did that as well as the 1247.77 million bit coefficients in the sqrt. I'm still quite astonished that it all went without a hitch. (No offense Jason!)
[/QUOTE]
None taken, I'm surprised too :)

I predict that nothing will crap out until we're one dependency away from a GNFS world record. That's going to be a long week :)

[QUOTE=fivemack;116546]The yield rate doesn't seem to drop off very fast with q. For a 143-digit GNFS job that I've got lying around, I'm getting ~115000 relations from the special-Qs in a range of size 50,000 around n=18 million, and ~90000 around n=36 million. The factor base limit is 18 million and the large-prime limit 2^29.
[/QUOTE]

The q's you are comparing are already well away from the origin.

Try comparing the yield rate for q near 18M versus q near 10000.

The drop off is solely due to the change in norms as q increases.
Suppose we are applying special q to the linear side. The lattice
reduction yields a matrix with coefficients near sqrt(q). The norms are
divisible by q, but the lattice points themselves (c,d) increase with the
entries in the reduced matrix. The net effect is that the linear side
norms that we need to be smooth decrease approximately as sqrt(q)
as q increases. However, the norms on the algebraic side INCREASE
with q as the d'th power of the reduced matrix coefficients and
thus increase approximately as q^(d/2). This is faster than the
linear side decreases, and produces a yield drop-off as q increases.

[QUOTE=bdodson;133235]OK, the 4th matrix is in,

3,547- ...16.067M^2

Some bad news on timings, the c252 is at 70%, but at 27 of the
estimated 28 days. Adding a factor of (10/7) gives a new estimate
of 40 days, which sounds like May 24. That leaves the c241 257-
somewhere below 40 days, the C208 257+ at something like (1.9/1.5)*40,
with the new c242 at 16.067M^2 likely somewhere out past the
15.48M^2 for the 257+.

PS - all four using 4-threads.[/QUOTE]

An update, 3p536 is at 86.9%, 10m257 at 61.8%, 10p257 at
29.7% and 3m547 at 12.5%. Yet another reboot/crash with
8 hrs of idle cpus and another 7-8hrs or more beyond that since
each matrix wrote its last checkpoint. If c. 40-days computing
walltime is somewhere near the c252 matrix runtime, the remaining
13.1% would be 5 days, 6 hours; so maybe May 27.

Supposing the matrix finishes, am I supposed to be able to get
each of four processors to run two independent dependencies?
I'd appreciate having the exact command line to refer to.

Seems like my chances are still good to finish one factorization before
the new quadcore smp server opens to our public users on June 1; and
I'm hoping to have a second one done before the old server is boxed-up
and shipped back on June 15.
-Bruce

[code]
msieve -v -nc3 1,2 & msieve -v -nc3 3,4 & msieve -v -nc3 5,6 & msieve -v -nc3 7,8 &
[/code]

should run two independent dependencies on each of four jobs.