3^512+1
 

This is quite a large SNFS number with a decidedly awkward polynomial (no real roots, no algebraic roots modulo primes ==7 or ==11 mod 12, two roots modulo primes ==5 mod 12, six roots modulo some primes ==1 mod 12), and so the number of usable special-Q in a particular range is really rather low.

So this one does require the 32768x16384 per-Q sieving range that gnfs-lasieve4I15e offers, and parameter optimisation indicates that it does want 31-bit large primes and very large small primes. Feel free to go smaller (reduce rlim and alim) if you can't face ~400MB per core: 40M gets about 65% the yield of 80M, but the yield drops off very fast below that.

[code]
n: 2652879528384736294387787089866884113161756949676609780113021980279955578028580515829763316598420245173034168388765124717208315443806148182904105317960270313646866242717807445467423472021744641
skew: 1
c6: 9
c0: 1
Y1: -1
Y0: 35917545547686059365808220080151141317043
rlim: 80000000
alim: 80000000
lpbr: 31
lpba: 31
mfbr: 62
mfba: 62
rlambda: 2.6
alambda: 2.6
[/code]

A range of one million Q will take about a week on a core2-2400 and produce about 2.3 million relations; we'll need somewhere between 180M and 200M, so around two core2 years.

Upload in the same way as the 2^1188+1 thread [url]http://www.mersenneforum.org/showthread.php?t=10003[/url]; please call your files something like 3+512.27M-27.3M.bz2 so I don't confuse them with the 2^1188+1 ones. Not sure how much the sieving periods will overlap.

[b]Reservations (between 40M and 125M, please)[/b]
**andi47 40-41 (3593082)
**andi47 41-42 (3507498)
andi47 42-43
bsquared 43-50
**wraithX 50-60 (33605887)
fivemack 67-72
**fivemack 72-80 (24015345 = 6047723+6049878+5985699+5932045)
**fivemack 80-82 (3104511+3040339)
**fivemack 82-90 (23116337 = 5827643+5795180+5791581+5701933)
**bdodson 90-91 (with gaps: 1431138 relations collected)
**bdodson 91-92 (no visible gaps; 2929210 relations)
**bdodson 92-93 (2962600)
**bsquared 93-94 (2878996)
**bsquared 94-96 (5816329)
**bsquared 96-97 (2886020)
**bsquared 97-98 (2873406)
**bsquared 98-101 (8550484)
**bsquared 101-115 (38580970)
**bsquared 115-120 (13328671)
**bsquared 120-123 (7915518)
**bdodson 123-124 (2636395)
**bsquared 124-125 (2615423)

[b]relation counts ('large ideals' are >2500000)[/b]
[code]
02/04/2008 00:06 16212022 relations, 16088425 unique relations and about 38580610 large ideals
11/04/2008 00:15 42926881 relations, 42092537 unique relations and about 63275041 large ideals
29/04/2008 22:15 189611855 relations, 172703615 unique relations and about 96169746 large ideals
weight of 11436921 cycles is about 743426050 (65.00/cycle)
[/code]

reserving 40M - 41M, using rlim = 80M and alim = 40M

Edit: Memory use approx. 251 MB

[QUOTE=fivemack;129612]
[b]Reservations[/b]
andi47 40-41
fivemack 80-82[/QUOTE]

reserving 90-91. There's something like 3.5Gb/core, so I'm using the
defaults. If I understand correctly, 10 ranges

[code]
./gnfs-lasieve4I15e -a 512.poly -f 90000000 -c 10000 -o 3+512.90M-90.01M
./gnfs-lasieve4I15e -a 512.poly -f 90010000 -c 10000 -o 3+512.90.01M-90.02M
./gnfs-lasieve4I15e -a 512.poly -f 90020000 -c 10000 -o 3+512.90.02M-90.03M
[/code]

will do the first 100000, then 20 ranges with -c 45000 will do the other
900000. With 2 cpus left to fiddle with B1 = 850M. -Bruce

[QUOTE=bdodson;129755]... 10 ranges

[code]
./gnfs-lasieve4I15e -a 512.poly -f 90000000 -c 10000 -o 3+512.90M-90.01M
./gnfs-lasieve4I15e -a 512.poly -f 90010000 -c 10000 -o 3+512.90.01M-90.02M
./gnfs-lasieve4I15e -a 512.poly -f 90020000 -c 10000 -o 3+512.90.02M-90.03M
[/code]

do the first 100000, then 20 ranges do the other
900000. With 2 cpus left to fiddle with B1 = 850M. -Bruce[/QUOTE]

Not quite. top reports 28 cpus at 99%-100%, with the last two jobs
at 49% and 51%. Looks like one of the quads has 3 cores reserved
for the "head node". The timings seem to pick up. After a bit more
than an hour, nine of the 1st ten report

[code]
total yield: 14634, q=90015257 (0.26097 sec/rel)
total yield: 14422, q=90024421 (0.26437 sec/rel)
total yield: 14855, q=90035773 (0.26035 sec/rel)
total yield: 14589, q=90046001 (0.26246 sec/rel)
total yield: 14244, q=90055529 (0.26608 sec/rel)
total yield: 14754, q=90064421 (0.25815 sec/rel)
total yield: 14621, q=90074389 (0.25899 sec/rel)
total yield: 14192, q=90085753 (0.27056 sec/rel)
total yield: 14655, q=90095561 (0.25817 sec/rel)
[/code]

But they started out slower; here's one with a late start:

total yield: 4095, q=90101129 (0.31718 sec/rel)

All of the initial timings were noticably above .3, but after ten
minutes ... hmmm, looks like I'll do better droping one of the 10K's,
then pick it up later. -Bruce

[quote=bdodson]reserving 90-91[/quote]

Excellent; I was wondering whether your enormous CPU resources were in a position to be used for lattice sieving, and clearly they are.

Just for my logistic convenience, would you mind concatenating the relation files into slightly larger blocks before uploading? One file for 90-91 would be a good deal easier to handle than thirty.

Don't worry about the fluctuations in the timing: those timings look very close to what I'm seeing here. They do fluctuate a bit, for reasons ranging from the vicissitudes of the distribution of prime numbers and of the skew of the lattices corresponding to the ideals up to memory bandwidth.

[QUOTE=bdodson;129755]reserving 90-91. There's something like 3.5Gb/core, so I'm using the
defaults. If I understand correctly, 10 ranges

[code]
./gnfs-lasieve4I15e -a 512.poly -f 90000000 -c 10000 -o 3+512.90M-90.01M
./gnfs-lasieve4I15e -a 512.poly -f 90010000 -c 10000 -o 3+512.90.01M-90.02M
./gnfs-lasieve4I15e -a 512.poly -f 90020000 -c 10000 -o 3+512.90.02M-90.03M
[/code]

will do the first 100000, then 20 ranges with -c 45000 will do the other
900000. With 2 cpus left to fiddle with B1 = 850M. -Bruce[/QUOTE]

I created an MPI version (making the least possible changes) for sieving 12,241- on the Lonestar cluster. I can package and send you the source if you like. It still records each range to a separate file, but it's easy to cat them together at the end. The only real advantage is that you have one command to run rather than one for each processor. The caveat is that the lattice siever occasionally gets stuck on a special-q. This happened twice in a range of 170M q's in the sieving of 12,241-, and this stalls all of the other processes waiting for the one to complete.

Greg

[QUOTE=fivemack;129785]Excellent; I was wondering whether your enormous CPU resources were in a position to be used for lattice sieving, and clearly they are.

Just for my logistic convenience, would you mind concatenating the relation files into slightly larger blocks before uploading? One file for 90-91 would be a good deal easier to handle than thirty.[/QUOTE]

All but one range finished in 12hrs, that one took an extra 30min; perhaps
there was an effect from another user. My problem with sieving has been
that almost everything here is scheduled though condor, which I haven't yet
figured out how to get to run sieving. The new machine is a replacement
for an older sgi that had Itanium2 chips (and was often also over-busy,
without me).

I'll cat things up (and gzip) as soon as I can track down where the job
I killed left off, and the reset started. I hadn't considered that the
relns are in hex, which makes finding the last q that finished less
transparent.

Another point of puzzlement: is there a factorbase stored somewhere?
Doesn't appear to be one in my filespace (much less 30 separate copies,
which would not have been good). I checked /tmp, but (it's a new
machine and) there's hardly anything there. Condor would be much
simpler (with a whole lot more cores) if it weren't for thinking about the
factorbase. -Bruce

[QUOTE=bdodson;129826]
I'll cat things up (and gzip) as soon as I can track down where the job
I killed left off, and the reset started. I hadn't considered that the
relns are in hex, which makes finding the last q that finished less
transparent.
[/QUOTE]

If you're sieving on the algebraic side, the special-q is always the last hex number on the line:
[code]
for i in *; do j=`tail -n 2 $i | head -n 1 | awk -F, '{print $NF}'`; k=`echo "16i $j pq"|dc`; echo "File $i $k"; done
[/code]

will list them nicely. If you add '| grep -v "9[89]..$"' to the end, it'll do a fairly good job of listing ones that didn't run to the end of their 10000-section.

[quote]
Another point of puzzlement: is there a factorbase stored somewhere?
[/quote]

No, the factorbase is not saved to disc; it's regenerated from scratch each time you start the siever. This takes less than a minute and generally my sieve jobs last between three days and a week.

What are the properties that a job has to have to play nicely with condor?

[QUOTE=frmky;129824]I created an MPI version (making the least possible changes) for sieving 12,241- on the Lonestar cluster[/QUOTE]

The Lonestar cluster seems quite an exciting machine: [url]http://www.tacc.utexas.edu/services/userguides/lonestar/[/url] suggests 10^4 2.66GHz processors. How many were you able to use for the sieving, what small and large primes did you use, and how long did it take?

[QUOTE=fivemack;129832]The Lonestar cluster seems quite an exciting machine: [url]http://www.tacc.utexas.edu/services/userguides/lonestar/[/url] suggests 10^4 2.66GHz processors. How many were you able to use for the sieving, what small and large primes did you use, and how long did it take?[/QUOTE]

The cluster is very busy, but I was able to use up to 400 processors at a time. I used fb limits of 70M on each side and 31-bit large primes. I sieved q from 40M to 200M using gnfs-lasieve4I15e on the rational side. The sieving took a total of a bit over 30,000 CPU hours. This took about 2 weeks real time mainly because I had to figure out how to use the cluster, MPI-ize the siever, and deal with the cluster going down a few times during those two weeks due to file system issues.

Greg

[QUOTE=fivemack;129612] ...
Upload in the same way as the 2^1188+1 thread [url]http://www.mersenneforum.org/showthread.php?t=10003[/url]; ...

[b]Reservations[/b]
andi47 40-41
fivemack 80-82
bdodson 90-91[/QUOTE]

The anonymous server doesn't seem to accept sftp for anonymous@...
Fortunately, they haven't yet gotten to the part of the installation
where they disable ftp (which they do on the compute server this is
upgrading; away from Itanium2, at last!).

While I'm waiting to hear from Greg and/or Richard about current
NFSNET plans, I'll take 91-93. -Bruce

(fairly sure Lehigh won't be amused to have their new server grouped
into Womack/mersenneforum; while we're listed explicitly in Sam's
pages as ... well, some sort of share in nfsnet admin.)

I realise I was a bit callow about attribution for 6+383, and should be more careful here.

Would something like "mersenneforum, Lehigh University + Womack" be acceptable? I could go the whole hog and list all the contributors, though that starts getting to particle-physics levels of silliness for numbers not much bigger than this.

[QUOTE=fivemack;129962]
Would something like "mersenneforum, Lehigh University + Womack" be acceptable? ...[/QUOTE]

Maybe Richard would consider extending the franchise, and let us use

Womack+mersenneforum/NFSNET (?).

For comparison, perhaps I could share Tom's pm on my first range,
for which there appear to be some 20 subranges in which the relns
didn't get recorded --- I hit a filespace quota overnight, and cleared
a storred NFSNET factorbase in the morning, after I found out. So
those 20 ranges had a reasonable first half (or so), missed most of
the second half, then did a last 30min.-1hr. If I understood the pm
correctly, I may be able to see the specific 20 subranges that need
re-running. Way different than the tight control in the NFSNET
server/client; but seems to be workable, with some extra effort on
Tom's part. I should do better this time; I'm running in /tmp.

-Bruce
(credit's not yet an issue, until I see how things look.)

[QUOTE=bdodson;129986]I should do better this time; I'm running in /tmp.[/QUOTE]On a number of systems /tmp is cleared on a reboot; on others /tmp is a ramdisk.

Your system may preserve /tmp over a reboot but I'd check to make sure.


Paul

[QUOTE=fivemack;129962]I realise I was a bit callow about attribution for 6+383, and should be more careful here.[/QUOTE]Better to be callow than callous.

I'll take 93-100

[QUOTE=fivemack;129612]
bdodson 90-91 (with gaps: 1431138 relations collected)
bdodson 91-93[/QUOTE]

[code]
86002582 Mar 28 07:11 3+512.90M-91M.gz
176046496 Mar 28 07:11 3+512.91M-92M.gz
178093292 Mar 28 17:43 3+512.92M-93M.gz
[/code]

Looks like I got less than half before my filespace ran out. The
person doing the installation reports adjusting the default setting
for future initial accounts. I re-ran the three ranges Tom located;
not sure it's worth looking for the other 17. Paul's correct, /tmp
isn't stable; I'm archiving files elsewhere. -Bruce

[QUOTE=fivemack;129832]The Lonestar cluster seems quite an exciting machine: [url]http://www.tacc.utexas.edu/services/userguides/lonestar/[/url] [/QUOTE]

I need to get me one of these. :showoff:

[QUOTE=fivemack;129612]
[b]Reservations[/b]
...
**bdodson 90-91 (with gaps: 1431138 relations collected)
**bdodson 91-92 (no visible gaps; 2929210 relations)
bdodson 92-93
bsquared 93-100[/QUOTE]

I sent in the 3 sub-ranges identified as missing, but I'm
dis-inclined to try tracking down the other ones. Instead,
I'm reserving

bdodson 123-124.

If everything below 120M gets run, I'll replace the
missing 90M subranges with q's another 20M out past
what's running. Wish me luck with getting relns here
within the next day or so. I seem to have lasieve running
on the Opteron cluster under condor (5hrs of 12hrs or so);
except that I won't know whether the output is "tranferred"
back from the local node to the condor master until the
range finishes (or, otherwise, just vanishes).

If this doesn't work, I'll run these q's on the new quadcore cluster
(which permits both use of the condor scheduler and "interactive"
logins, unlike the Opteron cluster and the "old" quadcore cluster).
I also managed to get condor to accept a submisson on the
other ("old") quadcore cluster; so perhaps I'll see whether
those ranges run (and, if so, whether I've managed to correctly
instruct condor what to do with the output). Not clear how
many jobs the scheduler will support, or whether I'll be able
to figure out what to do with a _lot_ more data; but those
would be problems I'd be happy to have. To hit a quarter of
the max that Greg hit, I'd want something like 25-new-quad,
25-Opteron, and 50-old-quad; so as I was saying, wish me
luck on today's 2+5 cpus. -Bruce

As far as data goes, I have a dedicated 60GB partition on the upload machine, which is on a 100Mbit internet connection in Telehouse in London, so if you have some way of pushing straight from the sievers to ftp, I ought to be able to handle any even quite unreasonable amount of data: a whole lpbr=31 factorisation takes IME under 30GB uncompressed, no more than 15G compressed. There may of course be policy at your end making that more difficult, or other bottlenecks.

I'll take the range from 50-60.

edit:
Oops, just realized I don't actually have gnfs-lasieve4I15e. Does anyone have a cygwin or mingw or windows compiled version of this? Or, I have the sources to ggnfs, can someone tell me how to compile it so that it produces the 15e executable?

Also (what I would like more!), does anyone have (or can anyone make) 64-bit binaries of 14e and 15e for Windows? That would be awesome!

[quote=WraithX;130294]I'll take the range from 50-60.

edit:
Oops, just realized I don't actually have gnfs-lasieve4I15e. Does anyone have a cygwin or mingw or windows compiled version of this? Or, I have the sources to ggnfs, can someone tell me how to compile it so that it produces the 15e executable?

Also (what I would like more!), does anyone have (or can anyone make) 64-bit binaries of 14e and 15e for Windows? That would be awesome![/quote]

See post #8 in this thread:
[URL]http://www.mersenneforum.org/showthread.php?t=10003[/URL]

I've compiled it for windows under mingw, but it was not very easy and I don't think I could give concise instructions. I basically did this

[CODE]
do {
err = compile;
if (err) {
hackish attempt to fix error;
} while (err);
[/CODE]

[QUOTE=bsquared;130302]See post #8 in this thread:
[URL]http://www.mersenneforum.org/showthread.php?t=10003[/URL]
[/QUOTE]

That's just what I was looking for. Thanks for that.

So, I'm trying "-f 50000000", and it says "Special q lower bound 50000000 below FB bound 8e+007".

So, my guess is to set either alim or rlim less than 80000000. But which one, or both? And to what lower value should I set it? Should I change any other values? I'm sorry for the amateur questions, but I'd like to get this right.

[QUOTE=WraithX;130312]So, I'm trying "-f 50000000", and it says "Special q lower bound 50000000 below FB bound 8e+007".

So, my guess is to set either alim or rlim less than 80000000. But which one, or both? And to what lower value should I set it? Should I change any other values? I'm sorry for the amateur questions, but I'd like to get this right.[/QUOTE]

Set alim to 50000000. You can keep the other values as they are.

[QUOTE=bdodson;130213] ... Wish me luck with getting relns here
within the next day or so. I seem to have lasieve running
on the Opteron cluster under condor (5hrs of 12hrs or so);
except that I won't know whether the output is "tranferred"
back from the local node to the condor master until the
range finishes (or, otherwise, just vanishes).

... so as I was saying, wish me
luck on today's 2+5 cpus. -Bruce[/QUOTE]

Tom's doing a great job. Here's

[code]
-rw-r--r-- 1 bad0 faculty 14593677 Mar 30 08:19 3+512.123.00M-123.05M
-rw-r--r-- 1 bad0 faculty 15331669 Mar 30 09:13 3+512.123.05M-123.10M
-rw-r--r-- 1 bad0 faculty 15023040 Mar 30 08:45 3+512.123.10M-123.15M
-rw-r--r-- 1 bad0 faculty 14775508 Mar 30 08:23 3+512.123.15M-123.20M
-rw-r--r-- 1 bad0 faculty 14795865 Mar 30 08:43 3+512.123.20M-123.25M
...
wc -l 3+512.123*

130277 3+512.123.00M-123.05M
136873 3+512.123.05M-123.10M
134111 3+512.123.10M-123.15M
131864 3+512.123.15M-123.20M
132083 3+512.123.20M-123.25M
665208 total [/code]

which appears to be 665,208 relns from the old quadcores (40*2*4 = 320
cores). The Opterons ran too (2*59 = 118 cores, when they're all up, which
is rare); 258K from bsquared's range (my fault). I'll set the rest of 123M,
then have a look at Greg's large nfsnet(?) number.

On the data, I have lots of space on the Opteron cluster (which also
hosts the scheduler for the "old" quads). Good thing, there's no ftp
(only sftp); automated ftp seems unlikely. -Bruce

[quote=bdodson;130384]
258K from bsquared's range (my fault). [/quote]

Where exactly did you overlap? It seems you're well outside my range (93 to 100M), unless I'm misunderstanding something.

Not a problem if so, and if I'm not already done with the overlapped part then I'll just not do that particular part. I'm done up through 96M as of now.

[QUOTE=bsquared;130386]Where exactly did you overlap? It seems you're well outside my range (93 to 100M), unless I'm misunderstanding something.
[/QUOTE]

I was fairly sure the jobs wouldn't run (this is a condor thing; ask Richard
or Greg!). The range was 93.0-93.1. I reserved 123M after seeing your
reservation of 93M-100M (nice!). My first jobs were set on the Opterons
(without checking or reserving); when I did check, I reserved 123M and set
the next quadcore jobs there. I wouldn't have mentioned running them at all,
except that I didn't want to leave the impression that only the quadcores
were running. So the initial experiment was 2+5 jobs; I'm waiting for
5+10 (amd+xeon_quad) to finish 123M; then will try ... uhm, c.100+300
available, so maybe 20+60 jobs, 1M+3M (but on 3p536)? That sounds like
the next sequence term would be 40+120, 2M+6M (these are 12-13 hour
runs), along with 20 on the new quads. Hmm, 60+180+20 might be pushing
things; maybe 50+150+20, which would double the 25% of Greg's max I
was considering before the first jobs ran. -Bruce

[quote=bdodson;130419]I was fairly sure the jobs wouldn't run (this is a condor thing; ask Richard
or Greg!). The range was 93.0-93.1. I reserved 123M after seeing your
reservation of 93M-100M (nice!). My first jobs were set on the Opterons
(without checking or reserving); when I did check, I reserved 123M and set
the next quadcore jobs there. I wouldn't have mentioned running them at all,
except that I didn't want to leave the impression that only the quadcores
were running. So the initial experiment was 2+5 jobs; I'm waiting for
5+10 (amd+xeon_quad) to finish 123M; then will try ... uhm, c.100+300
available, so maybe 20+60 jobs, 1M+3M (but on 3p536)? That sounds like
the next sequence term would be 40+120, 2M+6M (these are 12-13 hour
runs), along with 20 on the new quads. Hmm, 60+180+20 might be pushing
things; maybe 50+150+20, which would double the 25% of Greg's max I
was considering before the first jobs ran. -Bruce[/quote]

Well, I guess we'll have a small amount of overlap, which I don't think is any big deal. I'll submit my files as normal and we'll let msieve sort 'em out :)

I know you are very productive with it, but I don't know anything about the condor cluster you mention. Is there a website I can go to to read more about it? I'm running my ranges on a small cluster of dual dual core xeon 5160's and dual quad core xeon X5365's, available (nights and weekends, mostly) due to the generosity of my employers. Running part time I can do about 1M special-q a day on this number.

I'm working up the nerve to ask them about longer term usage of a single node... I'm thinking that a 16Gb, dual quad core X5365 would work nicely for post-processing. But I'm doubtful I'll be able to monopolize one box for so long :(

- ben.

Explanation of very long reservation
 

I realize that I've made an enormous reservation, about 20 CPU-weeks of work; the reason is that I'm going off for a two-week vacation in a couple of weeks, and will be leaving ten CPUs running. I have only started 80-82, and that only yesterday (sieving 2^1188+1 is still quite a lot of work, not to mention the linear algebra).

We're getting nearly three relations/Q on average - I must have done trial sieving over a particularly sparse patch - so probably eighty million Q will suffice and it would be sensible not to reserve beyond 125M. I hadn't expected to be able to get people with clusters involved with the sieving, if I had I would have picked a significantly bigger number.

[QUOTE=bsquared;130425]
I'm working up the nerve to ask them about longer term usage of a single node... I'm thinking that a 16Gb, dual quad core X5365 would work nicely for post-processing. But I'm doubtful I'll be able to monopolize one box for so long :([/QUOTE]

A 16GB dual quad-core is overkill for post-processing on numbers this 'small'; my hope is to over-sieve sufficiently that I can get a matrix which fits on a 4GB quad-core (we nearly managed this for 2,841- which was a noticeably bigger number). The matrix-production step needed an 8G machine for 2,841- or the 165-digit GNFS job of earlier this year, I have one of those available.

The 16GB machine would be very interesting for the matrix-building step if we decide to try to beat Aoki's larger GNFS or smaller SNFS record, both of which are I think just about plausible targets for a lattice-sieving effort with substantial distributed resources (20 CPU-years or so), at least if msieve consistently works as much better than Aoki's code as I saw on 6,383+; the matrix-running step for those would be several months on a quad-core, so really has to run on a personal machine, but the matrix-build takes a bit more memory for no more than a week.

[QUOTE=bsquared;130425]Well, I guess we'll have a small amount of overlap, which I don't think is any big deal. I'll submit my files as normal and we'll let msieve sort 'em out :)

I know you are very productive with it, but I don't know anything about the condor cluster you mention. Is there a website I can go to to read more about it?
- ben.[/QUOTE]

I didn't reserve that range; you did; I only ran those q's as an experiment,
and don't plan on submitting the output (if it's still around). Also, duplicates
make the filtering harder, and should usually be avoided.

The first google entry on condor is the "condor project homepage"

[url]www.cs.wisc.edu/condor/[/url]

The other entries involve various vultures ("carrion-eaters"). The initial
purpose of the program being to scavenge for spare idle cycles. It's also
evolved into a scheduler on clusters. -Bruce

[QUOTE=bdodson;130419] ... I didn't want to leave the impression that only the quadcores
were running. So the initial experiment was 2+5 jobs; I'm waiting for
5+10 (amd+xeon_quad) to finish 123M; then will try ... uhm, c.100+300
available, so maybe 20+60 jobs, 1M+3M (but on 3p536)? That sounds like
the next sequence term would be 40+120, 2M+6M (these are 12-13 hour
runs), along with 20 on the new quads. Hmm, 60+180+20 might be pushing
things; maybe 50+150+20, which would double the 25% of Greg's max I
was considering before the first jobs ran. -Bruce[/QUOTE]

Uhm, post #17 was a report of completion of 92M-93M. It was intended
as a note of the file having been uploaded (though I didn't explicitly say so).
This is a report of completion on 123M-124M, also uploaded (92M hasn't been
counted yet, it seems?). These complete my 3+512 reservations, for the
moment. The relns in 123M are the first ones from condor (Opteron+old
quadcore).

Doesn't look to me like I'll need the mpi; at least, not for job submissions.
Just a bit of shell scripts and some "sed -e" goes most of the way. I
submitted 3M to the Opterons, 7M to the "old" quads, both over condor,
with another 1M on the new quads. That's 60+140+20 cpus, which will
take (well) under a day (presuming everything runs). The new number
was supposed to need 130M, so 11M/day ... well, even 10M/day would
be under two weeks for completely sieving 3p536. Guess we'll see (if
you don't mind the off-topic posts). Combination of Greg's binary(s) and
Tom's instructions (here and in the eleven-smooth thread) seems to have
done to get me running --- guess the new un-condor-ed cluster helped for
seeing what ought to happen. -Bruce

(9M so far for Greg's 3,536+ c252; so 130-9 = 121M left to go)

[QUOTE=bdodson;130384]Good thing, there's no ftp
(only sftp); automated ftp seems unlikely. -Bruce[/QUOTE]Perhaps it's a local LeHigh situation, but automated sftp (or scp, which is essentially the same thing) is very easy to set up. I use it in a number of situations, mostly cronjobs which transfer gigabytes of files (PostgreSQL dumps mostly) to and from Harvard every week.

Let me know if you would like more details of how to set up scripted sftp transfers.


Paul

[QUOTE=xilman;130538]Perhaps it's a local LeHigh situation, but automated sftp (or scp, which is essentially the same thing) is very easy to set up. I use it in a number of situations, mostly cronjobs which transfer gigabytes of files (PostgreSQL dumps mostly) to and from Harvard every week.

Let me know if you would like more details of how to set up scripted sftp transfers.

Paul[/QUOTE]

Sounds good; I'm not sure what Greg has in mind, data-wise (we
haven't gotten that far yet). On the 3+512 exchanges, there was
also a problem on Tom's side; the annon ftp wasn't accepting
sftp connections; or at least, I wasn't able to get one. Actually,
even if/when the new cluster gets ftp disabled, there's an old mail
server ...

I'm also interested in getting msieve assistance --- I ought to be
able to improve matters by removing duplicates, at least. If/when
I'm running complete sieving jobs (rather than partial ranges) I could
perhaps prepare a matrix to send off (as we used to do with Lehigh
factorizations finished on the MSR cluster for older nfs's, back in the
day); or perhaps the linux/quads here could be used? -Bruce

You've got enough resources there that I'd have thought you'd just run the msieve stage yourself. msieve is embarrassingly easy to run:

* put your number, unadorned, into a file worktodo.ini
* put 'N number' into a file msieve.dat, followed by the concatenation of all the relations
* write an msieve.fb file of the form
[code]
N number
A6 9
A0 1
R1 -1
R0 35917545547686059365808220080151141317043
[/code]
* do 'msieve -v -nc1 2500000,2500000' to do the filtering, 'msieve -v -t4 -nc2' to do the linear algebra on four threads, 'msieve -v -nc3' to do the square roots, 'msieve -v -t4 -ncr' to restart the linear algebra if your computer crashed. Count on maybe 15MB of memory use per million relations in the linear algebra stage, and maybe 20MB in the other two - the 225Mrel job for 6+383 just barely failed to fit on a 4G machine for those two stages, so I just ran nc1 on an 8G machine, copied the directory to a reliable 4G machine that I could leave for a week, ran nc2, copied the directory back to the 8G machine and ran nc3.

[QUOTE=fivemack;130548]
* do 'msieve -v -nc1 2500000,2500000' to do the filtering[/QUOTE]
Note that the latest version has much improved automatic sizing of filtering bounds, so that '-nc1' without numerical arguments should do the same job with much less memory

[QUOTE=fivemack;130548]msieve is embarrassingly easy to run:
[/QUOTE]Ditto on the "embarrassing" part. It has made factoring easier, and made me dumber, because I've fallen out of the habit of thinking about how some of the parameters affect the process. (This is really my fault, and not Jason's, but msieve is a crutch -- a well-made one..)

I'm not done with my current range, but I think I should get a new one on the books before fivemack takes off (assuming he won't be updating these pages while basking in the sun...)

Reserving 100-105

- ben.

I'm not fleeing the country until Saturday 12 April - I was just hoping that I could get results from a one-week processing job before I left on a jetplane. So you may want a bigger reservation ...

I [i]have[/i] posted to mersenneforum from a youth hostel in Serbia, but I don't have the ability to log into my compute farm from outside my house.

Sorry, got dates mixed up.

I don't want to monopolize the ranges (in the hopes that many others will see there are fertile q left and reserve them...). If all goes well I should be nearly done with what I have before you leave, so I can grab a new reservation then.

[quote=bsquared;130743]I'm not done with my current range, but I think I should get a new one on the books before fivemack takes off (assuming he won't be updating these pages while basking in the sun...)

Reserving 100-105

- ben.[/quote]

I'm on track to have this range done by saturday, so I'll go ahead and extend the reservation to 115. That will keep me busy for another couple weeks.

40-41M finished and currently uploading.

Running two instances of 15e slows down the core 2 duo too much, so I will wait with reserving the next range until I have finished 41-42M.

41-42M is finished, I will upload the file today or tomorrow.

reserving 42-43M

Range finished (through 115M) and uploaded.

I'll reserve 115-120.

- ben.

[QUOTE=Andi47;131217]40-41M finished and currently uploading.

Running two instances of 15e slows down the core 2 duo too much, so I will wait with reserving the next range until I have finished 41-42M.[/QUOTE]

I just see that I accidentally started 42M-43M instead of 41-42M. So I want to change this reservation to 42-43M.

[QUOTE=Andi47;132164]I just see that I accidentally started 42M-43M instead of 41-42M. So I want to change this reservation to 42-43M.[/QUOTE]

Hmmm... Seems I did 41-42M anyway, but the file somehow got missing. I will see, if I find it, otherwise I will redo the range as soon as 42-43 is finished.

[QUOTE=bsquared;132082]Range finished (through 115M) and uploaded.

I'll reserve 115-120.

- ben.[/QUOTE]

Thanks for that. I got my speed estimates slightly wrong and it'll be a few days before 72-90 is finished, I'll do the next ideal-counting run then since at this level they take all night.

[quote=bsquared;132082]Range finished (through 115M) and uploaded.

I'll reserve 115-120.

- ben.[/quote]

aaaand, done. I'll clean up the rest of the 120's stuff (120-123, 124-125)

Hey everyone, I think I may be having some trouble with gnfs-lasieve4I15e. For the past couple of days the "total yield" and "q=" values have been the same. I only really noticed today. Has anyone seen this before? Should I stop the sieve and restart it? If so, how do I tell what was the last q processed? Something else that makes this strange and possibly troublesome is that while .last_spq0 shows it was last modified a couple of days ago, the file where all the relations are going is still growing in size. Does this mean there will probably be a lot of duplicates at the end of the relations file? Or does it mean processing is continuing like normal, and just screen updates aren't happening? Any insights/suggestions greatly appreciated.

Hi WraithX. That's an interesting behaviour, which I haven't seen before.

Could you post the last few lines of the relations file?

Well, I ran "tail <relations_file> > output.txt" 3 times (a few seconds apart) and I couldn't see any repeats, but alot of the numbers are cryptic to me in that format so here they are:

45459707,59762453:134ec843,19f2d493,8EAF,B827,CCD1,39DDB3:41e6a2a5,DBAD,5C815,95ED5,12E1D3D,35A1,3930881
-60258599,153622043:4a0f9899,76D7D,1280E9,1A8E05,5CC809,A39E17,3209:33a95925,486c8569,163F1,5BDB1,6C12F1,136D,3930881
-59072547,74787011:96395e3,C461,17B31,49F5D,20DCE3,AC7647,3B4DFD5:131917c9,578D5,BB711,63C541,CA93A9,26DD,3930881
53699923,12114851:515a4a9,36e9d98f,75A7,89DB,1F8A3,3F00D,12B230F:3d150e61,294175,B521FD,DD80A5,313C8A1,3930881
62732297,94168379:10598513,EE67,9EC6B,11B0AB,1476D7,BEE4C9:1dabeb31,40BD,9749,507425,2B58411,1139,425,3930881
-78910789,128089875:84A9,23417,4C975,8BC01,1DE805,10F13BD,471FB29,D69:47a01ed,2ae6503d,41BA5,B24A5,322CCD,959,3930881
-84374441,119754587:a3c86c1,178d08b1,58AB,49B6D,10F3C7,223483,378D:d6a4eed,5035,3496CD,1076EA9,22E35ED,1CC9,3930881
75904207,13180611:426a3b01,7F79,9041,CEDAF,7E7C8B,2B33,BCB,905,607:c350ee5,6785,9EF5,20741,AC3BB9,2E01,3930881
-89621495,46426883:a401ca1,1175a751,39EA71,2E6B99D,4BDE5E7:11907dfd,586D,748D,90FFE9,1C2A4D1,955,3930881
-104350747,162716303:216ab481,23de5ad1,72D7,3D

-11246944,14240783:24760d6f,62b787f,12F91,164C779,3165D4B,279D,1445:52866b9,32A11,2F6015,3BD1,27C5,2395,3930881
14476978,779333:3fa09c43,29554c4b,7D1D,C5D89,33315D,1757B51:326dc221,ecd81e9,A86D19,D98AB1,3930881
22035002,11392815:52786523,19ff10fb,29731,2C7B9,F3A13F,AF3,49D:272c3521,106951,3475BC1,2B69,655,127D,3930881
-33368264,101303397:ce7579d,4D95,663205,A035A3,24F3875,130D,F47,595:47cf4ff1,28c71515,55B1,300A1,2544AB5,3930881
-35655824,127899513:1c4e9c87,9c4277f,6EBD,84E161,14D7F27,29ED,6F1:158bec79,d0eda95,5ADC49,A38235,150E869,3930881
-36312996,98990113:2d01b42b,7243,C8E7,BA007,10B5D2F,19BB63F,B29:1fc9f285,62cbbbc9,71C9,D559,2DC985,39F9C1,3930881
29539488,91532843:2e238375,54a6b52f,6B11,56179,88991B,A01EF3,15C5:6C65,7939,7DE9,1AA4D,C60B9,2BAB55,467A3D,3930881
-41029512,63944369:44024309,6a9df27,27E51,874B7,138745,4AE13B3,36D1:11c5ff5d,6756D,917FC9,15E5739,20132AD,7CD,3930881
-44753902,112143611:56a1f8d1,660c1453,2C8F87,250BC71,2EA45E9,1A2F,6CD:4ca2e721,1c085cc9,5D3D,330695,120BCAD,3930881
37858802,35441587:267f1483,e5a0a31,1CF01,1E743,2C7B9,19F039,295D:3baecf09,2CDA1,225D51,6F82C1,8C455

-60199978,82224235:3ea0c65b,7d5d5f9,101BD,2EBE3,831CD,B6E903,2351,8BD:5E69,14ED5,A0E35,AE101,12D951D,316D,1CE1,1AB1,3930881
70921746,48027139:62e155a1,39f2f88d,40B1,2F5217,7CC349,49E11B1:b9bec19,5486dc5,9D5C5,30D1,2659,135D,869,3930881
80754428,50753905:15dee91b,6F61,1B54D,90BD5,1BDC69,1DCC441,2BEB,1AB1:c6237c5,49a83b85,61B5,ACF9,34CE8F5,23F9,3930881
-85658948,19382021:3c0bf9c5,9D4C3,7225D9,18BBCB7,199D2BD,29CA7E7:118d0215,5cba695,1CCB49,1A7F8A1,33B9,2851,3930881
-85127666,104662827:37c48e8b,16935be3,7A1D3,328A2D,21D6769,38725E5,BE9:7577c989,5AA8D,6FF49,3376219,2F45,1139,3930881
40183275,7347188:16A33,31055,A1863,256717,24208CF,2E5A21D,BE1:4be4bc89,2b946c59,58BD,BB471,36DD149,39308C9
117219797,2102640:65a83e1,27e43527,A6CD,18E03,B74033,13AAA4F,137B:2abd9601,9dcc7ad,6964D,12CA61,1DC2E1,12D1,39308C9
-89034109,3555830:9b46d3d,215e19a5,AF8D7,17FA41,17B3A1B,1FE7,17B3:55da6fc1,3842a131,B18E9,13FB6D,D3ED49,103D,39308C9
-64467571,5776266:3644e39d,D62F1,32FCB1,4551A3,757741,1A6DD45:f6c0521,CC59,1288C1,11D9895,2DA1,4E1,C41,39308C9
-100860045,3155146:32adce23,5bdd5f5,28DF5,51D0D,2F0D067,24AF,7C

This particular instance is running the command:
./gnfs-lasieve4I15e.exe -f 58000000 -c 2000000 -o 3+512.58-60 -a 3+512.poly

The last screen output says (as it has since Friday afternoon):
total yield: 5920375, q=59780597 (0.36270 sec/rel)

If you need any more info please let me know.

OK, that is running along very happily; converting the last entry on the line from hex to decimal gives 59967617 and then it goes up to 59967689, so you're nearly finished, maybe seven hours left to go. I've no idea why the screen output and the .lastspq file are not being updated, you're obviously not out of disc space.

[QUOTE=Andi47;132222]Hmmm... Seems I did 41-42M anyway, but the file somehow got missing. I will see, if I find it, otherwise I will redo the range as soon as 42-43 is finished.[/QUOTE]

I found the file, it just somehow went to the wrong folder. (maybe I accidentally moved it with a wrong mouseclick.)

I will upload it in the next few days.

P.S.: 42-43 is approx. 76% done.

[quote=bsquared;132234]I'll clean up the rest of the 120's stuff (120-123, 124-125)[/quote]

done. I'll wait for a relation count update before committing to another range.

I just uploaded 41M-42M.

Thanks for all the recently-uploaded relations. I've started a count and a singleton-removal pass, will update the top post with the numbers when they pop out of the machine, either tomorrow morning or tomorrow evening. Currently sieving 67-72 with ETA about eight days from now; I wouldn't be amazed if we've got enough relations for a big ugly matrix already, but I would rather hold out for a small beautiful matrix that fits on my bijou 4GB computerette.

[quote=fivemack;132403]Currently sieving 67-72 with ETA about eight days from now;
[/quote]

I'll take 43-50 then, with ETA about 7 days from now.

This is going to end up amazingly over-sieved: I should have worked out that the rarity of usable special-Q would mean a comparable rarity of large prime ideals, and therefore not as many relations would be needed as I'd suspect. msieve is removing enormous numbers of cliques; I'll post the final output from the -nc1 run here when it completes.

I don't think any more sieving is needed, I'll stop my jobs when I get home, and put my farm onto Fibonacci numbers until we come up with another large-scale project to do. xilman's proposed 180-digit GNFS would be nicely pushing the borders of practicality, and what's the point in doing problems that we know we can do?

[QUOTE=fivemack;132433]This is going to end up amazingly over-sieved: I should have worked out that the rarity of usable special-Q would mean a comparable rarity of large prime ideals, and therefore not as many relations would be needed as I'd suspect. msieve is removing enormous numbers of cliques; I'll post the final output from the -nc1 run here when it completes.

I don't think any more sieving is needed, I'll stop my jobs when I get home, ...[/QUOTE]

@fivemack: My 42-43M is currently ~89% done, I expect it to finish during weekend, and so I could upload it not earlier than monday next week.

Alternatively, if You want to start postprocessing earlier, I could interrupt the job today in the afternoon and upload what I have tonight.

What do you prefer?

I would prefer, if you don't mind, for you to upload what you have tonight.

[QUOTE=fivemack;132436]I would prefer, if you don't mind, for you to upload what you have tonight.[/QUOTE]

OK, I think I can start the upload around 8 p.m. CEST (= 18 UTC)

I have run the range up to q=42897761.

I got through another 2M q overnight (43-45), about 7M relations, which I can upload now. Although this might only make the filtering harder, with little gain in matrix size. Let me know if you want them.

- ben.

Go ahead and upload them, I'll re-run the filtering overnight and we'll have another data point on d(matrix size)/d(relations). I don't have a machine available to run the matrix for at least a week.

[QUOTE=Andi47;132441]OK, I think I can start the upload around 8 p.m. CEST (= 18 UTC)

I have run the range up to q=42897761.[/QUOTE]

I was able to start the upload earlier, it is now complete.

[QUOTE=fivemack;132445]Go ahead and upload them, I'll re-run the filtering overnight and we'll have another data point on d(matrix size)/d(relations). I don't have a machine available to run the matrix for at least a week.[/QUOTE]
Do you still have the logs from the matrix runs that failed by finding only trivial dependencies? I have a hypothesis that failures of that type happen when the sparse portion of the final matrix has less than ~60 nonzeros per column on average, and two of the three failures I know about fit that profile. There are plenty of successes with >= 62 nonzeros in the sparse part, along with one failure.

I've only got two log files from matrix runs that failed, and only one with a recent version:

[code]
/safe/cunningham/5,775M/msieve.log-Wed Feb 13 22:30:22 2008 matrix is 5025987 x 5026229 (1294.7 MB) with weight 349634052 (69.56/col)
/safe/cunningham/5,775M/msieve.log-Wed Feb 13 22:30:22 2008 sparse part has weight 279094804 (55.53/col)
/safe/cunningham/5,775M/msieve.log-Wed Feb 13 22:30:22 2008 matrix includes 128 packed rows
/safe/cunningham/5,775M/msieve.log-Wed Feb 13 22:30:22 2008 using block size 65536 for processor cache size 4096 kB
/safe/cunningham/5,775M/msieve.log-Wed Feb 13 22:30:52 2008 commencing Lanczos iteration (2 threads)
/safe/cunningham/5,775M/msieve.log-Wed Feb 13 22:30:52 2008 memory use: 1319.4 MB
/safe/cunningham/5,775M/msieve.log-Sun Feb 17 03:28:48 2008 lanczos halted after 79479 iterations (dim = 5025576)
/safe/cunningham/5,775M/msieve.log:Sun Feb 17 03:29:01 2008 lanczos error: only trivial dependencies found
[/code]

which would seem to confirm your hypothesis.

On the other hand,
[code]
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:35:45 2008 matrix is 401907 x 402155 (103.5 MB) with weight 28957098 (72.00/col)
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:35:45 2008 sparse part has weight 22300832 (55.45/col)
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:35:45 2008 matrix includes 128 packed rows
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:35:45 2008 using block size 65536 for processor cache size 4096 kB
/safe/homcun/11+10.184/msieve/msieve.log:Fri Feb 15 00:35:48 2008 commencing Lanczos iteration
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:35:48 2008 memory use: 99.2 MB
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:37:54 2008 lanczos error: submatrix is not invertible
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:37:54 2008 lanczos halted after 540 iterations (dim = 34098)
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:37:54 2008 linear algebra failed; retrying...
/safe/homcun/11+10.184/msieve/msieve.log:Fri Feb 15 00:37:54 2008 commencing Lanczos iteration
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 00:37:54 2008 memory use: 99.2 MB
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 01:02:36 2008 lanczos halted after 6358 iterations (dim = 401902)
/safe/homcun/11+10.184/msieve/msieve.log-Fri Feb 15 01:02:37 2008 recovered 40 nontrivial dependencies
[/code]

Initial matrix for 3^512+1
 

[code]
Tue Apr 29 23:48:00 2008 Msieve v. 1.34
Tue Apr 29 23:48:00 2008 random seeds: 5f6708ca 737b360c
Tue Apr 29 23:48:00 2008 factoring 2652879528384736294387787089866884113161756949676609780113021980279955578028580515829763316598420245173034168388765124717208315443806148182904105317960270313646866242717807445467423472021744641 (193 digits)
Tue Apr 29 23:53:40 2008 restarting with 189612490 relations
Wed Apr 30 00:19:08 2008 found 48513490 hash collisions in 189611855 relations
Wed Apr 30 00:19:08 2008 commencing duplicate removal, pass 2
Wed Apr 30 00:35:48 2008 found 16908240 duplicates and 172703615 unique relations
Wed Apr 30 00:35:48 2008 memory use: 1911.0 MB
Wed Apr 30 00:35:49 2008 ignoring smallest 183072 rational and 182532 algebraic ideals
Wed Apr 30 00:35:49 2008 filtering rational ideals above 2500000
Wed Apr 30 00:35:49 2008 filtering algebraic ideals above 2500000
Wed Apr 30 00:35:49 2008 need 548406 more relations than ideals
Wed Apr 30 00:35:49 2008 commencing singleton removal, pass 1
Wed Apr 30 01:02:44 2008 relations with 0 large ideals: 0
Wed Apr 30 01:02:44 2008 relations with 1 large ideals: 742
Wed Apr 30 01:02:44 2008 relations with 2 large ideals: 25483
Wed Apr 30 01:02:44 2008 relations with 3 large ideals: 356357
Wed Apr 30 01:02:44 2008 relations with 4 large ideals: 2687291
Wed Apr 30 01:02:44 2008 relations with 5 large ideals: 11751148
Wed Apr 30 01:02:44 2008 relations with 6 large ideals: 30511295
Wed Apr 30 01:02:44 2008 relations with 7+ large ideals: 127371299
Wed Apr 30 01:02:44 2008 172703615 relations and about 96169746 large ideals
Wed Apr 30 01:02:44 2008 commencing singleton removal, pass 2
Wed Apr 30 01:29:41 2008 found 15878777 singletons
Wed Apr 30 01:29:41 2008 current dataset: 156824838 relations and about 79705795 large ideals
Wed Apr 30 01:29:42 2008 commencing singleton removal, pass 3
Wed Apr 30 01:54:36 2008 relations with 0 large ideals: 0
Wed Apr 30 01:54:36 2008 relations with 1 large ideals: 742
Wed Apr 30 01:54:36 2008 relations with 2 large ideals: 24904
Wed Apr 30 01:54:36 2008 relations with 3 large ideals: 341002
Wed Apr 30 01:54:36 2008 relations with 4 large ideals: 2520743
Wed Apr 30 01:54:36 2008 relations with 5 large ideals: 10848320
Wed Apr 30 01:54:36 2008 relations with 6 large ideals: 27860172
Wed Apr 30 01:54:36 2008 relations with 7+ large ideals: 115228955
Wed Apr 30 01:54:36 2008 156824838 relations and about 114324560 large ideals
Wed Apr 30 01:54:36 2008 commencing singleton removal, pass 4
Wed Apr 30 02:19:29 2008 found 24434594 singletons
Wed Apr 30 02:19:29 2008 current dataset: 132390244 relations and about 88294946 large ideals
Wed Apr 30 02:19:29 2008 commencing singleton removal, pass 5
Wed Apr 30 02:41:09 2008 relations with 0 large ideals: 0
Wed Apr 30 02:41:09 2008 relations with 1 large ideals: 742
Wed Apr 30 02:41:09 2008 relations with 2 large ideals: 23915
Wed Apr 30 02:41:09 2008 relations with 3 large ideals: 315754
Wed Apr 30 02:41:09 2008 relations with 4 large ideals: 2253955
Wed Apr 30 02:41:09 2008 relations with 5 large ideals: 9433948
Wed Apr 30 02:41:09 2008 relations with 6 large ideals: 23760197
Wed Apr 30 02:41:09 2008 relations with 7+ large ideals: 96601733
Wed Apr 30 02:41:09 2008 132390244 relations and about 107914977 large ideals
Wed Apr 30 02:41:09 2008 commencing singleton removal, pass 6
Wed Apr 30 03:02:51 2008 found 19388737 singletons
Wed Apr 30 03:02:51 2008 current dataset: 113001507 relations and about 87344686 large ideals
Wed Apr 30 03:02:51 2008 commencing singleton removal, pass 7
Wed Apr 30 03:21:45 2008 found 4530506 singletons
Wed Apr 30 03:21:45 2008 current dataset: 108471001 relations and about 82738277 large ideals
Wed Apr 30 03:21:45 2008 commencing singleton removal, pass 8
Wed Apr 30 03:40:00 2008 found 971921 singletons
Wed Apr 30 03:40:00 2008 current dataset: 107499080 relations and about 81762618 large ideals
Wed Apr 30 03:40:00 2008 commencing singleton removal, pass 9
Wed Apr 30 03:58:13 2008 found 198741 singletons
Wed Apr 30 03:58:13 2008 current dataset: 107300339 relations and about 81563729 large ideals
Wed Apr 30 03:58:13 2008 commencing singleton removal, final pass
Wed Apr 30 04:34:53 2008 memory use: 3808.2 MB
Wed Apr 30 04:34:56 2008 commencing in-memory singleton removal
Wed Apr 30 04:35:38 2008 begin with 107300339 relations and 92918933 unique ideals
Wed Apr 30 04:45:05 2008 reduce to 96734930 relations and 82148891 ideals in 16 passes
Wed Apr 30 04:45:05 2008 max relations containing the same ideal: 384
(clique removal)
Wed Apr 30 07:01:28 2008 removing 1529556 relations and 1129556 ideals in 400000 cliques
Wed Apr 30 07:01:30 2008 commencing in-memory singleton removal
Wed Apr 30 07:01:40 2008 begin with 31072254 relations and 31607946 unique ideals
Wed Apr 30 07:02:37 2008 reduce to 31021692 relations and 30427234 ideals in 6 passes
Wed Apr 30 07:02:37 2008 max relations containing the same ideal: 146
Wed Apr 30 07:03:23 2008 removing 249135 relations and 203084 ideals in 46051 cliques
Wed Apr 30 07:03:24 2008 commencing in-memory singleton removal
Wed Apr 30 07:03:35 2008 begin with 30772557 relations and 30427234 unique ideals
Wed Apr 30 07:04:12 2008 reduce to 30771559 relations and 30223150 ideals in 4 passes
Wed Apr 30 07:04:12 2008 max relations containing the same ideal: 145
Wed Apr 30 07:04:21 2008 filtering rational ideals above 750000
Wed Apr 30 07:04:21 2008 filtering algebraic ideals above 750000
Wed Apr 30 07:04:21 2008 need 120216 more relations than ideals
Wed Apr 30 07:04:21 2008 commencing singleton removal, final pass
Wed Apr 30 08:18:23 2008 keeping 73218132 ideals with weight <= 20, new excess is 9296364
Wed Apr 30 08:20:17 2008 memory use: 2639.6 MB
Wed Apr 30 08:20:18 2008 commencing in-memory singleton removal
Wed Apr 30 08:20:34 2008 begin with 96734931 relations and 73218132 unique ideals
Wed Apr 30 08:21:07 2008 reduce to 96734930 relations and 73218131 ideals in 2 passes
Wed Apr 30 08:21:07 2008 max relations containing the same ideal: 20
(more clique removal)
Wed Apr 30 09:26:13 2008 removing 1333448 relations and 993985 ideals in 339463 cliques
Wed Apr 30 09:26:14 2008 commencing in-memory singleton removal
Wed Apr 30 09:26:19 2008 begin with 35997998 relations and 26208200 unique ideals
Wed Apr 30 09:26:45 2008 reduce to 35964483 relations and 25180385 ideals in 5 passes
Wed Apr 30 09:26:45 2008 max relations containing the same ideal: 20
Wed Apr 30 09:27:14 2008 relations with 0 large ideals: 1594415
Wed Apr 30 09:27:14 2008 relations with 1 large ideals: 7302931
Wed Apr 30 09:27:14 2008 relations with 2 large ideals: 12706306
Wed Apr 30 09:27:14 2008 relations with 3 large ideals: 9939004
Wed Apr 30 09:27:14 2008 relations with 4 large ideals: 3708167
Wed Apr 30 09:27:14 2008 relations with 5 large ideals: 648403
Wed Apr 30 09:27:14 2008 relations with 6 large ideals: 61672
Wed Apr 30 09:27:14 2008 relations with 7+ large ideals: 3585
Wed Apr 30 09:27:14 2008 commencing 2-way merge
Wed Apr 30 09:27:49 2008 reduce to 24200819 relation sets and 13416721 unique ideals
Wed Apr 30 09:27:49 2008 commencing full merge
Wed Apr 30 09:31:27 2008 memory use: 1224.1 MB
Wed Apr 30 09:31:28 2008 found 12923421 cycles, need 11436921
Wed Apr 30 09:31:35 2008 weight of 11436921 cycles is about 743426050 (65.00/cycle)
Wed Apr 30 09:31:35 2008 distribution of cycle lengths:
Wed Apr 30 09:31:35 2008 1 relations: 1852171
Wed Apr 30 09:31:35 2008 2 relations: 1415558
Wed Apr 30 09:31:35 2008 3 relations: 1402932
Wed Apr 30 09:31:35 2008 4 relations: 1332850
Wed Apr 30 09:31:35 2008 5 relations: 1262651
Wed Apr 30 09:31:35 2008 6 relations: 1123342
Wed Apr 30 09:31:35 2008 7 relations: 983922
Wed Apr 30 09:31:35 2008 8 relations: 818904
Wed Apr 30 09:31:35 2008 9 relations: 659698
Wed Apr 30 09:31:35 2008 10+ relations: 584893
Wed Apr 30 09:31:35 2008 heaviest cycle: 12 relations
Wed Apr 30 09:31:36 2008 commencing cycle optimization
Wed Apr 30 09:32:04 2008 start with 52607046 relations
Wed Apr 30 09:33:45 2008 pruned 1197440 relations
Wed Apr 30 09:33:46 2008 memory use: 1822.9 MB
Wed Apr 30 09:33:46 2008 distribution of cycle lengths:
Wed Apr 30 09:33:46 2008 1 relations: 1852171
Wed Apr 30 09:33:46 2008 2 relations: 1448894
Wed Apr 30 09:33:46 2008 3 relations: 1454488
Wed Apr 30 09:33:46 2008 4 relations: 1370864
Wed Apr 30 09:33:46 2008 5 relations: 1305864
Wed Apr 30 09:33:46 2008 6 relations: 1153396
Wed Apr 30 09:33:46 2008 7 relations: 1000698
Wed Apr 30 09:33:46 2008 8 relations: 806218
Wed Apr 30 09:33:46 2008 9 relations: 605076
Wed Apr 30 09:33:46 2008 10+ relations: 439252
Wed Apr 30 09:33:46 2008 heaviest cycle: 12 relations
Wed Apr 30 09:34:11 2008 elapsed time 09:46:11
Wed Apr 30 18:27:33 2008
Wed Apr 30 18:27:33 2008
Wed Apr 30 18:27:33 2008 Msieve v. 1.34
Wed Apr 30 18:27:33 2008 random seeds: aed73ff8 776b3e7d
Wed Apr 30 18:27:33 2008 factoring 2652879528384736294387787089866884113161756949676609780113021980279955578028580515829763316598420245173034168388765124717208315443806148182904105317960270313646866242717807445467423472021744641 (193 digits)
Wed Apr 30 18:27:36 2008 no P-1/P+1/ECM available, skipping
Wed Apr 30 18:27:36 2008 commencing number field sieve (193-digit input)
Wed Apr 30 18:27:36 2008 R0: 35917545547686059365808220080151141317043
Wed Apr 30 18:27:36 2008 R1: -1
Wed Apr 30 18:27:36 2008 A0: 1
Wed Apr 30 18:27:36 2008 A1: 0
Wed Apr 30 18:27:36 2008 A2: 0
Wed Apr 30 18:27:36 2008 A3: 0
Wed Apr 30 18:27:36 2008 A4: 0
Wed Apr 30 18:27:36 2008 A5: 0
Wed Apr 30 18:27:36 2008 A6: 9
Wed Apr 30 18:27:36 2008 size score = 8.379415e-12, Murphy alpha = 1.425790, combined = 5.575653e-12
Wed Apr 30 18:27:36 2008
Wed Apr 30 18:27:36 2008 commencing linear algebra
Wed Apr 30 18:27:39 2008 read 11436921 cycles
Wed Apr 30 18:29:28 2008 cycles contain 31057959 unique relations
Wed Apr 30 18:36:05 2008 read 31057959 relations
Wed Apr 30 18:37:25 2008 using 32 quadratic characters above 2147480850
Wed Apr 30 18:43:03 2008 building initial matrix
Wed Apr 30 18:56:01 2008 memory use: 3942.0 MB
Wed Apr 30 18:56:11 2008 read 11436921 cycles
Wed Apr 30 18:56:35 2008 matrix is 11435826 x 11436921 (3204.8 MB) with weight 1001600646 (87.58/col)
Wed Apr 30 18:56:35 2008 sparse part has weight 714320576 (62.46/col)
Wed Apr 30 19:01:47 2008 filtering completed in 3 passes
Wed Apr 30 19:01:51 2008 matrix is 11400059 x 11400259 (3199.3 MB) with weight 999652442 (87.69/col)
Wed Apr 30 19:01:51 2008 sparse part has weight 713277612 (62.57/col)
Wed Apr 30 19:04:34 2008 read 11400259 cycles
Wed Apr 30 19:04:50 2008 matrix is 11400059 x 11400259 (3199.3 MB) with weight 999652442 (87.69/col)
Wed Apr 30 19:04:50 2008 sparse part has weight 713277612 (62.57/col)
Wed Apr 30 19:04:51 2008 saving the first 48 matrix rows for later
Wed Apr 30 19:04:57 2008 matrix is 11400011 x 11400259 (3083.8 MB) with weight 762539620 (66.89/col)
Wed Apr 30 19:04:57 2008 sparse part has weight 694386289 (60.91/col)
Wed Apr 30 19:04:57 2008 matrix includes 64 packed rows
Wed Apr 30 19:04:57 2008 using block size 65536 for processor cache size 4096 kB
Wed Apr 30 19:06:19 2008 commencing Lanczos iteration
Wed Apr 30 19:06:19 2008 memory use: 3047.6 MB
[/code]

and memory usage by msieve was fairly steady at 3771MB, so that would have just fitted on the 4GB machine. I'll be interested to see how the final numbers change with the ten or so million new relations I'm currently downloading.

Final matrix for 3+512
 

[code]
found 7795849 duplicates and 183336747 unique relations
memory use: 1911.0 MB
ignoring smallest 7108914 rational and 7107522 algebraic ideals
filtering rational ideals above 124977152
filtering algebraic ideals above 124977152
need 21324654 more relations than ideals
commencing singleton removal, pass 1
relations with 0 large ideals: 4720680
relations with 1 large ideals: 26532618
relations with 2 large ideals: 60196322
relations with 3 large ideals: 64203955
relations with 4 large ideals: 27674969
relations with 5 large ideals: 8203
relations with 6 large ideals: 0
relations with 7+ large ideals: 0
183336747 relations and about 92964614 large ideals
commencing singleton removal, pass 2
found 16723680 singletons
current dataset: 166613067 relations and about 75630287 large ideals
commencing singleton removal, pass 3
relations with 0 large ideals: 4720680
relations with 1 large ideals: 25504572
relations with 2 large ideals: 55639049
relations with 3 large ideals: 57071988
relations with 4 large ideals: 23669787
relations with 5 large ideals: 6991
relations with 6 large ideals: 0
relations with 7+ large ideals: 0
166613067 relations and about 107615586 large ideals
commencing singleton removal, pass 4
found 24157797 singletons
current dataset: 142455270 relations and about 82004924 large ideals
commencing singleton removal, pass 5
relations with 0 large ideals: 4720680
relations with 1 large ideals: 23872815
relations with 2 large ideals: 48775066
relations with 3 large ideals: 46864018
relations with 4 large ideals: 18217312
relations with 5 large ideals: 5379
relations with 6 large ideals: 0
relations with 7+ large ideals: 0
142455270 relations and about 99364914 large ideals
commencing singleton removal, pass 6
found 18002328 singletons
current dataset: 124452942 relations and about 80432277 large ideals
commencing singleton removal, pass 7
found 3972915 singletons
current dataset: 120480027 relations and about 76405418 large ideals
commencing singleton removal, pass 8
found 793186 singletons
current dataset: 119686841 relations and about 75609909 large ideals
commencing singleton removal, pass 9
found 148752 singletons
current dataset: 119538089 relations and about 75461071 large ideals
commencing singleton removal, final pass
memory use: 1948.8 MB
commencing in-memory singleton removal
begin with 119538089 relations and 85069326 unique ideals
reduce to 111164660 relations and 76572690 ideals in 13 passes
max relations containing the same ideal: 52
(clique removal)
commencing in-memory singleton removal
begin with 52911835 relations and 31837518 unique ideals
reduce to 52911003 relations and 31586348 ideals in 4 passes
max relations containing the same ideal: 31
filtering rational ideals above 750000
filtering algebraic ideals above 750000
need 120216 more relations than ideals
commencing singleton removal, final pass
keeping 80668853 ideals with weight <= 20, new excess is 10119748
memory use: 2671.2 MB
commencing in-memory singleton removal
begin with 111164660 relations and 80668853 unique ideals
reduce to 111156501 relations and 80660693 ideals in 8 passes
max relations containing the same ideal: 20
(clique removal)
commencing in-memory singleton removal
begin with 33196574 relations and 22370231 unique ideals
reduce to 33162215 relations and 21422807 ideals in 5 passes
max relations containing the same ideal: 17
relations with 0 large ideals: 2123367
relations with 1 large ideals: 8561800
relations with 2 large ideals: 12486331
relations with 3 large ideals: 7673775
relations with 4 large ideals: 2077904
relations with 5 large ideals: 225955
relations with 6 large ideals: 12656
relations with 7+ large ideals: 427
commencing 2-way merge
reduce to 22959819 relation sets and 11220411 unique ideals
commencing full merge
memory use: 1052.0 MB
found 12504356 cycles, need 10888611
weight of 10888611 cycles is about 707963521 (65.02/cycle)
distribution of cycle lengths:
1 relations: 2159899
2 relations: 1303681
3 relations: 1228402
4 relations: 1152853
5 relations: 1095856
6 relations: 994754
7 relations: 883124
8 relations: 755936
9 relations: 616168
10+ relations: 697938
heaviest cycle: 12 relations
commencing cycle optimization
start with 49500473 relations
pruned 1522481 relations
memory use: 1660.5 MB
distribution of cycle lengths:
1 relations: 2159899
2 relations: 1341861
3 relations: 1280474
4 relations: 1197979
5 relations: 1153379
6 relations: 1039012
7 relations: 914211
8 relations: 747326
9 relations: 561539
10+ relations: 492931
heaviest cycle: 12 relations
elapsed time 08:57:36
[/code]

So 10.6 million relations have gained us 550 thousand dimensions, or 6.2% more unique relations have saved us 4.8% of the matrix dimensions.

It's quite a large matrix but should fit in 4GB; there's a small problem of whether the matrix-build is memory-local enough to run swapping over NFS, since I know it will take >4GB and the machine I'm using has a 2G USB stick as local disc. I'll start the linalg on Saturday (fibonacci(1021) is running on that machine at the moment), it will probably take two weeks.

[quote=fivemack;133008]

So 10.6 million relations have gained us 550 thousand dimensions, or 6.2% more unique relations have saved us 4.8% of the matrix dimensions.

I'll start the linalg on Saturday (fibonacci(1021) is running on that machine at the moment), it will probably take two weeks.[/quote]

It took about 8 hours to get those 10.6M relations (well, it took 8 hours for my 7M, and I'm guessing about that long for the rest, working in parallel).

2 weeks = 336 hrs = 1209600 seconds or ~ 0.111 sec/dimension.

So reducing 550000 dimensions saved ~ 17 hours, less 8 hours of extra sieving, according to this napkin math :smile:.

It should be even better than that. The LA time is super-linear in the dimensionality of the matrix. The number of iterations is linear, but each iteration should be faster on the smaller matrix.

The linear algebra has started:

[code]
Sun May 11 02:39:47 2008 matrix is 10856877 x 10857125 (2895.2 MB) with weight 714095503 (65.77/col)
Sun May 11 02:39:47 2008 sparse part has weight 650401084 (59.91/col)
Sun May 11 02:39:47 2008 matrix includes 64 packed rows
Sun May 11 02:39:47 2008 using block size 65536 for processor cache size 4096 kB
Sun May 11 02:40:56 2008 commencing Lanczos iteration (4 threads)
Sun May 11 02:40:56 2008 memory use: 3112.7 MB
[/code]

ETA is afternoon of May 29th; whilst it said 'memory use: 3112.7MB', top says
[code]
7435 nfsslave 25 0 3574m 3.3g 440 R 341 87.7 3976:19 msieve
[/code]

Anyway, it fits in 4GB which was the important issue.

[QUOTE=fivemack;133234]The linear algebra has started:

[code]
Sun May 11 02:39:47 2008 matrix is 10856877 x 10857125 (2895.2 MB) with weight 714095503 (65.77/col)
Sun May 11 02:39:47 2008 sparse part has weight 650401084 (59.91/col)
Sun May 11 02:39:47 2008 matrix includes 64 packed rows
Sun May 11 02:39:47 2008 using block size 65536 for processor cache size 4096 kB
Sun May 11 02:40:56 2008 commencing Lanczos iteration (4 threads)
Sun May 11 02:40:56 2008 memory use: 3112.7 MB
[/code]

ETA is afternoon of May 29th; whilst it said 'memory use: 3112.7MB', top says
[code]
7435 nfsslave 25 0 3574m 3.3g 440 R 341 87.7 3976:19 msieve
[/code]

Anyway, it fits in 4GB which was the important issue.[/QUOTE]
The disparity between estimated and actual memory use is actually smaller than I've seen posted elsewhere; for comparison the actual memory use of the filtering step tends to be 30-50% larger than what the library estimates. Instead of measuring the size of data structures, I could instead use library functions like mallinfo(), but all of these assume 31-bit values for memory used.

Otherwise, the final matrix density looks dangerously low; I should change TARGET_DENSITY from 65.0 to 70.0 in the future, to force the final matrix to be more dense.

Not an ECM miss!
 

[code]
Sun May 11 23:47:01 2008 commencing Lanczos iteration (4 threads)
Sun May 11 23:47:01 2008 memory use: 3112.7 MB
Sun May 11 23:47:19 2008 restarting at iteration 7909 (dim = 500081)
Thu May 29 15:11:11 2008 lanczos halted after 171698 iterations (dim = 10856872)
Thu May 29 15:11:52 2008 recovered 50 nontrivial dependencies
Thu May 29 15:11:55 2008 elapsed time 423:28:20
Thu May 29 19:09:18 2008 Msieve v. 1.34
Thu May 29 19:09:19 2008 commencing square root phase
Thu May 29 19:09:19 2008 reading relations for dependency 1
Thu May 29 19:09:31 2008 read 5427551 cycles
Thu May 29 19:10:06 2008 cycles contain 17096763 unique relations
Thu May 29 19:15:57 2008 read 17096763 relations
Thu May 29 19:20:03 2008 multiplying 23949492 relations
Thu May 29 21:21:20 2008 multiply complete, coefficients have about 690.79 million bits
Thu May 29 21:21:35 2008 initial square root is modulo 1575811
Thu May 29 23:59:17 2008 prp93 factor: 185203545384014444998415700339182963094565346115682981302539679248192554448695865898272140289
Thu May 29 23:59:17 2008 prp101 factor: 14324129286424100231861565705311821806023066083407911748417199121054089089998098987789407767132602369
Thu May 29 23:59:17 2008 elapsed time 04:49:59
[/code]

Thanks for all the cycles!

[QUOTE=fivemack;134746]not an ECM miss[/QUOTE]

And NOT an p-1 or p+1 miss too, some ECM curves at the 15- and 20 digit levels show that both factors are waaaay out of p+/-1 range.

[quote=Andi47;134767]And NOT an p-1 or p+1 miss too, some ECM curves at the 15- and 20 digit levels show that both factors are waaaay out of p+/-1 range.[/quote]

Indeed:
[code]

prp93 - 1 = 2^10 * 10935391 * 873411622106606511242977261100967248293 * 18936340619807636319594684484400404592181649
prp93 + 1 = 2 * 3 * 5 * 13 * 661 * 12940947333474585331284139 * 8378869400326912691301842743 * 6625699555011851410351750036873963

prp101 - 1 = 2^15 * 3^7 * 2153 * 3691 * 121727 * 206630544613448030672025489880296233031839828693270211360359233883363407408743913
prp101 + 1 = 2 * 5 * 239049199913 * 5992126010728021621153697446264425308377152528306321229288237982195655176636574810316549

[/code]

Thanks, fivemack, for organizing this effort!