10^263-1 sieving
 

Here is a nice hefty project to keep your CPUs bushy-tailed and glossy-coated into the Spring. The polynomial file is

[code]
n: 276397778616959975917054244686348356245407334346331193552844312837949697033718836316510648552112872462175754206758265936948213056626234970255319654022011994534451487805497984097946614373924022283714690178871836462083
c6: 1
c0: -10
Y1: -1
Y0: 100000000000000000000000000000000000000000000
type: snfs
skew: 1.47
rlambda: 2.6
alambda: 2.6
lpbr: 31
lpba: 31
mfbr: 62
mfba: 62
alim: 125000000
rlim: 100000000
[/code]with parameters selected to minimise sieving time by sampling 10kQ at Q=75,100,125,150M and curve-fitting.

Start sieving at Q=25M on both sides (using gnfs-lasieve4I15e), and we'll run until we get a matrix of reasonable size, by which point I should have a computer of more-than-reasonable size on which to run the matrix. I guess we'll have to get to 125M or so, but the drop-off in speed and yield with increasing Q is quite perceptible:

150M .. 150M+1000
A total yield: 987, q=150001039 (0.80463 sec/rel)
R total yield: 1187, q=150001039 (0.68066 sec/rel)

25M .. 25M+1000
A total yield: 2580, q=25001077 (0.43460 sec/rel)
R total yield: 1799, q=25001029 (0.44244 sec/rel)

The city is Oslo.

[B]Reservations[/B]

Xyzzy 15M-20M (algebraic side) (done 04Feb)
JF 15M-20M (rational side) (done 30Jan)
fivemack 20M-25M (both sides) (done 06Feb)
Xyzzy 25M-30M (both sides) (done 27Jan)
batalov 30M-35M (both sides) (done by Xyzzy and batalov ~20 Jan)
JF 35M-40M (both sides) (done 03Jan)
andi47 40M-42M (algebraic side only) (done 04Feb)
bsquared 40M-42M (rational side only) (done 30Jan)
JF 42M-50M (both sides) (done 05Jan)
fivemack 50M-60M (both sides) (done 27Jan)
Syd 60M-65M (both sides) (done 30Jan)
batalov 65M-70M (both sides) (done 06Feb)
JF 70M-75M (both sides) (done 14Jan)
JF 75M-85M (both sides) (done 19Jan)
fivemack 85M-90M (both sides) (done 03Feb)
JF 90M-100M (both sides) (done 22Jan)
fivemack 100M-102M (both sides) (done 16Feb)
Xyzzy 102M-106M (both sides) (done 22Feb)
fivemack 106M-110M (both sides) (done 23Feb)
smh 110M-111M (both sides)
bsquared 111M-120M (both sides) (done 20Feb)
fivemack 120M-12xM (just killing time ...)

[B]Relation stock[/B]

A 15-121.3
R 15-121.3

[B]Incremental analysis[/B]

2015 14/01/2009 (50MQ): 63712027 relations, 54883524 unique.
0030 23/01/2009 (102MQ): 136304978 relations, 104394776 unique
2230 06/02/2009 (170MQ): 233613937 relations, 143698189 unique
2040 27/02/2009 (206.6MQ): 284849410 relations, 171805386 unique
Sat Feb 28 08:20:51 2009 matrix is 15785588 x 15785836 (4555.7 MB) with weight 1111879653 (70.44/col)
Sat Feb 28 08:20:51 2009 sparse part has weight 1036401865 (65.65/col)

skew: 1.47
 

Looks good, but maybe add [B]skew: 1.47[/B] (=10^1/6)
With and without this skew, I got --
[CODE]
-a 15e
total yield: 2456, q=25001029 (0.41596 sec/rel) (skew=1.47)
total yield: 2387, q=25001029 (0.42721 sec/rel) (skew=1)
-r 15e
total yield: 1837, q=25001029 (0.41020 sec/rel) (skew=1.47)
total yield: 1768, q=25001029 (0.45177 sec/rel) (skew=1)

-a 14e (my favorite blooper! but I ran it, so will show these as well)
total yield: 1129, q=25001029 (0.36877 sec/rel) (skew=1.47)
total yield: 1118, q=25001029 (0.37191 sec/rel) (skew=1)
-r 14e
total yield: 888, q=25001029 (0.35605 sec/rel) (skew=1.47)
total yield: 882, q=25001029 (0.37367 sec/rel) (skew=1)
[/CODE]
(This didn't match some of your numbers, though. Oh, I see what happened, your runs are not all stopped at q=25001029.)

Anyway, the difference is small, but skew=1.47 seems a bit better than 1.00. Would you like to bench it, too? your benchmark is more comprehesive.

I'll take 30-35M both sides.

Good point about the skew, I'm happy to believe your benchmark without running it myself.

We were thinking today, which is usually a dangerous thing.

We wanted to run a range on a core and have it survive a system crash, process crash or a shutdown with minimal loss. We also didn't want to have to watch over it too much.

So we came up with this very simple script:

[FONT=Courier New]for i in $(seq 30000000 10000 30990000); do ./gnfs-lasieve4I15e -r poly -f ${i} -c 10000 -o ${i}r && chmod 400 ${i}r; done[/FONT]

Basically, for each range of a million iterations, we create a folder. We reserved 30-35M so we used "30r", "31r", "32r", "33r" and "34r". We can run 4 ranges at a time on a quad core, so "34r" is going to have to wait for a while. We put a copy of gnfs-lasieve4I15e and a copy of the poly in each folder. We want all the work from 30,000,000 to 30,999,999 to stay in the "30" folder, and so on and so forth. Instead of doing all 1,000,000 iterations in one go, we break it up into 10,000 iteration chunks. We suppose you could use different sized chunks depending on your hardware and needs. There does seem to be a minute or so of lag when you start a range before you see any results so if you made the chunks too small you might lose a lot of CPU time. After each chunk is finished the results file is set to read only. The "&&" only allows this file to be set read only if the previous sieving command was successful. In the event of a crash or whatever, you just look at your folder and find the highest file, which should be writable, and then restart the script at that level, which is a matter of editing only 2 characters in the script. (The command will be in your history buffer so this is trivial.) As a sanity check you can ensure all the lower results files are read only. The script will, of course, overwrite the file that was not completed so there is no need to delete it. In the worst case scenario, you lose 9,999 iterations, or something like that.

We run mprime in the background to catch stray cycles and if a process fails mprime picks up the slack so there is no lost CPU time. We've played with "PauseWhenRunning=gnfs" in prime.txt, which works, but if one core goes down mprime will not kick in. If you run mprime without "PauseWhenRunning" and you do not "nice" your gnfs work, mprime will use ~5-6% out of 400% on a quad core box. One possible solution might be to rename each gnfs-lasieve4I15e binary to something unique and somehow have "PauseWhenRunning" deal with partial failures but right now it just gives us a headache thinking about it. We'll deal with the 5-6% hit.

We append "r" to the file name to indicate we are working the R side. We have 2 spare quad core boxes so we are running the R stuff on one box and the A stuff on the other. The A stuff has "a" appended to the file name. Our folders over there are "30a", "31a", "32a", "33a" and "34a". The changes in the script for the A side amount to changing 3 characters, all from "r" to "a". If the changes are not painfully obvious you probably should not be in the same room with sharp objects.

We use "screen" to keep track of all this. None of our Linux boxes have a GUI interface installed, and all have no monitor, keyboard or mouse. We use SSH to log in remotely.

We know there are probably much more elegant solutions out there but this is a start. We suspect there are people out there lurking, like we do, who appreciate having simple stuff explained. (The fact that this took us all day is a confirmation of our third-grade education level.) If there is a more optimal way to script things or divide the work up amongst cores please let us know. What would be really cool would be to have a master server issue out work units to the cores in your network and track them.

In the end, when the range is done, you just "cat" all the results files up and redirect them to one big file. When all the ranges are done you "cat" all the big files and redirect them to a final file, which can then have "bzip2 -9" applied to it.

One interesting benefit is you can look at the file modification time, related to the file modification time of the file that came before it, to get an idea of how long each chuck takes to compute. Another benefit is you can tell which chunk you are working on by looking at "top", rather than looking in each window or folder.

In other news, we are getting excellent iteration times compared to the last work we did. We are now happily using Linux rather than Windows. The Linux binary is very fast and much easier to use, now that we have access to the tools we are used to using.

:spot:

That is a VERY good start. No kidding!

I use a bit less granularity, say the "-c 50000" chunks.
Don't forget that with the latest binary (this reminds me to build it and repost it, the Opteron one) you will be able to finish failed chunks by adding [B]-R [/B](you cannot use it on a gzipped file, gunzip it).

P.S. I wonder how many people know that there's a [B]-z[/B] option in the siever?! That'll save you half the disk space. It was always there!
Note: libz is not linked, the output stream is simply piped into a [I]gzip[/I] child process; you need a [I]gzip[/I] binary in the [I]$path[/I]. And then the chunks may be concatenated into 1M chunks to be ftp'd (gzipped files allow that), but for even better compression, you can also gunzip them, cat together and bzip2 -9. Surely, you can do that on the fly using pipes, too.

[quote]That'll save you half the disk space.[/quote]We're not going to sweat disk space.

[code]$ df -h /dev/sda1
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 463G 2.8G 437G 1% /[/code]We remember what it was like using a cassette tape for mass storage.

We'd actually use around 700MiB for the system but we accidentally installed the basic system and the desktop. This will be addressed soon. (We don't bother with a swap partition or swap file either.)

Interesting note: When you have this much space to spare you can format EXT3 with "largefile4". It really doesn't make much of a difference, except you probably will never have to worry about fragmentation since each file is allocated, at the minimum, 4MiB. And you won't run out of inodes, either.

PS: Here is what "top" looks like. Note the ability to track the siever progress.

[code]top - 01:08:35 up 4 days, 7:22, 7 users, load average: 4.00, 4.00, 4.00
Tasks: 118 total, 6 running, 112 sleeping, 0 stopped, 0 zombie
Cpu0 : 99.0%us, 1.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 99.7%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 99.7%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 5987960k total, 1647444k used, 4340516k free, 83656k buffers
Swap: 0k total, 0k used, 0k free, 274212k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
17911 m 20 0 637m 280m 472 R 100 4.8 29:35.51 3 ./gnfs-lasieve4I15e -r poly -f 32020000 -c 10000 -o 32020000r
17933 m 20 0 637m 280m 472 R 100 4.8 27:57.22 1 ./gnfs-lasieve4I15e -r poly -f 33020000 -c 10000 -o 33020000r
18029 m 20 0 637m 280m 472 R 100 4.8 20:00.72 0 ./gnfs-lasieve4I15e -r poly -f 31020000 -c 10000 -o 31020000r
18057 m 20 0 637m 280m 472 R 100 4.8 17:48.94 2 ./gnfs-lasieve4I15e -r poly -f 30020000 -c 10000 -o 30020000r[/code]

[QUOTE=Xyzzy;156310]Interesting note: When you have this much space to spare you can format EXT3 with "largefile4". It really doesn't make much of a difference, except you probably will never have to worry about fragmentation since each file is allocated, at the minimum, 4MiB. And you won't run out of inodes, either.[/QUOTE]Thanks for posting this tip. I've learned something new!


Paul

I'll take 50-55M, both sides.
Being new and all, is there anything I should know about setting rlim, alim and such?

JF: would you mind taking 35-40 instead? The yield seems distinctly better for lower Q, and I'd prefer (perhaps only for aesthetic reasons) a contiguous range of sieving rather than lots of gaps to fill at the end.

rlim and alim: if you're sieving -r, make a copy of the .poly file and set rlim to the smallest Q that that run is using, if you're sieving -a do the same with alim.

I will take 40-42 (A side)

I will start sieving on Jan 7th or 8th, but I want to grab one of the low ranges which (presumably?) take less memory due to the lower alim, as long as these ranges are available. (please correct me if I'm wrong about memory useage.) (these huge GNFSes are almost maxing out what is the highest memory usage for the siever to run (almost) invisible on my office box, i.e. to not slow down other programs too much)

[quote=fivemack;156351]JF: would you mind taking 35-40 instead? The yield seems distinctly better for lower Q, and I'd prefer (perhaps only for aesthetic reasons) a contiguous range of sieving rather than lots of gaps to fill at the end.

rlim and alim: if you're sieving -r, make a copy of the .poly file and set rlim to the smallest Q that that run is using, if you're sieving -a do the same with alim.[/quote]

Ok, 35-40 it is.

[QUOTE=Andi47;156355]I will take 40-42 (A side)

I will start sieving on Jan 7th or 8th, but I want to grab one of the low ranges which (presumably?) take less memory due to the lower alim, as long as these ranges are available. (please correct me if I'm wrong about memory useage.) (these huge GNFSes are almost maxing out what is the highest memory usage for the siever to run (almost) invisible on my office box, i.e. to not slow down other programs too much)[/QUOTE]

I am unfamiliar with (is this msieve or GGNFS?) the application
specific terminology.

May I suggest that we ALL use a non-application specfic terminology
for these discussions instead of the application specific abbreviations and
TLA's? e.g. linear factor base bound, algebraic factor base bound,
sieve region size, linear large prime bound, etc. etc...

I have a *proof* (similar to the one I published for line sieving) that
the smaller special q's should have a LARGER sieve region and that the
size of the sieve region should gradually decrease as q increases.
Note that the yield rate decreases as q decreases.

The rate of decrease should be proportional to the rate of decrease
in the yield rate, with the proportionality constant being a certain
eigenvalue that pops out of the calc-of-variations optimization problem.
Unlike for line sieving, I don't have a geometric interpretation for the
value of this eigenvalue.

I realize of course that in practice one can not implement such a gradual
decrease because sieving works fastest (owing to how fast computers &
compilers can actually address memory) when the sieve region size is a
power of 2. However, one can implement a decreasing [b]step function[/b]
for the sieve region size.

I am guessing from context that 'alim' is somehow related to sieve region
size?? If so, the smaller q's should have a larger 'alim'.

Why?

Because the yield rate is higher for smaller q, and my published proof
shows that it is better to do more sieving where the yield rate is higher.
This should be intuititively obvious.

[QUOTE=R.D. Silverman;156381]
I am guessing from context that 'alim' is somehow related to sieve region
size?? If so, the smaller q's should have a larger 'alim'.

Why?

Because the yield rate is higher for smaller q, and my published proof
shows that it is better to do more sieving where the yield rate is higher.
This should be intuititively obvious.[/QUOTE]

GGNFS [i]requires[/i] alim to be smaller or equal than q if sieving on the algebraic side, and rlim to be smaller or equal than q if sieving on the rational side.

[QUOTE=Andi47;156390]GGNFS [i]requires[/i] alim to be smaller or equal than q if sieving on the algebraic side, and rlim to be smaller or equal than q if sieving on the rational side.[/QUOTE]

You still have not DEFINED 'alim'........

[quote=R.D. Silverman;156398]You still have not DEFINED 'alim'........[/quote]
algebraic factor base limit

[QUOTE=henryzz;156401]algebraic factor base limit[/QUOTE]

Ah. So it is not related to the size of the sieve region....

My comments about the size of the sieve region still apply.
Memory requirements should DECREASE as q increases.

Also, my siever sieves the entire factor base, regardless of the
special-q value. Sieving of the larger primes in the factor base
only takes a small fraction of the run time. I see no reason
to restrict what you call alim to be a function of the q values.
I use a static value for the size of both factor bases throughout
the sieving of all the q values. Note also that I use values of special q
that are inside the factor base.... I find that the increased yield rate more
than compensates for the extra duplicates you get this way.

OTOH:

Note that my siever does require more memory than what I have seen
reported for GGNFS. Currently, for 7,277+ with factor base bounds
of 35 million for both the linear and algebraic factor bases, and a large
prime bound of 800 million, my siever requires about 382Mbytes of
memory per instance.

35-40M is done. Fivemack, would you mind checking the result? Like Xyzzy, I also did some scripting to automate the process (generating jobs, poly input, postprocessing and such), and I just want to make sure everything is ok.

Reserving 42-50M.

I've checked the result (at least: there is a cliff in the distribution of primes in the right place in each file so you've been changing alim and q0 in harmony, the largest large primes are the right size, msieve doesn't give 1400000 'wrong relation' errors); thanks for the calculations, and carry on!

[quote][FONT=Courier New]for i in $(seq 30000000 10000 30990000); do ./gnfs-lasieve4I15e -r poly -f ${i} -c 10000 -o ${i}r && chmod 400 ${i}r; done[/FONT][/quote]Our updated script is below. The "-w" is important. We had a problem using large numbers with "seq" because they were output in scientific notation. We have one distro with "seq" that exhibits this behavior and one that does not. We just happen to prefer the distro that does the scientific notation stuff. Plus, this script is a little easier to edit when it is time to resume a job.

[FONT=Courier New]for i in $(seq -w 0 99); do ./gnfs-lasieve4I15e -r poly -f 30${i}0000 -c 10000 -o 30${i}0000r && chmod 400 30${i}0000r; done[/FONT]

We installed our preferred system with just the console stuff plus all of our special tools. The hard drive was formatted as EXT3 with the "largefile4" option. Roughly 110MiB of the space used are results files.

[code]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 466G 754M 442G 1% /
tmpfs 2.9G 0 2.9G 0% /lib/init/rw
udev 10M 44K 10M 1% /dev
tmpfs 2.9G 0 2.9G 0% /dev/shm

$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 119264 26361 92903 23% /
tmpfs 747112 2 747110 1% /lib/init/rw
udev 747112 421 746691 1% /dev
tmpfs 747112 1 747111 1% /dev/shm[/code]

[quote=Andi47;156355]I will take 40-42 (A side)

[/quote]

I'll take the R side.

Reserving 60-65, both sides

[QUOTE=Andi47;156355]I will take 40-42 (A side)

I will start sieving on Jan 7th or 8th, but I want to grab one of the low ranges which (presumably?) take less memory due to the lower alim, as long as these ranges are available. (please correct me if I'm wrong about memory useage.) (these huge GNFSes are almost maxing out what is the highest memory usage for the siever to run (almost) invisible on my office box, i.e. to not slow down other programs too much)[/QUOTE]

I have started my range today in the morning on my office PC. After running for ~6:40 hours I estimate that it will run for ~693 hours = 28-29 days (--> ETA: Feb. 6th). Is this too much? [i]Iff[/i] yes, than I could reduce my reservation accordingly.

I think a Feb 6th ETA is fine, this is quite a large project and I expect sieving to take more than a month.

Bad news?

We just noticed that we are almost finished with both sides of 30-35M, but we were supposed to be doing both sides of 25-30M.

Should we jump on 25-30M now? It will take us about 12 days to do the work.

(Even our examples earlier in this thread used the wrong range. Talk about dumb!)

:cry:

We uploaded a few of our files, in 1M increments. We used lowercase to set our files apart from the ones that are already there.

We noticed our files are significantly smaller than the other files that are in 1M increments.

We sure hope we are not doing something wrong.

In the meantime, we have started on the 25-30M range. We think we can complete it in time and not cause the project any delays.

it was my range but ##it happens. no problem
 

Ouch. I'll have a look at your files, and compare to mine. Yes, my files are bigger than yours and similar in size to JF's. So your files may be sparser than mine (my wild guess is that they were sieved with 14e, but I am not casting any stones - I've done that myself a few times) ...but it still doesn't make any sense for me to finish over-sieving this range with 15e, so I'll take another range instead.

I've already uploaded R30-31M.bz2 and A30-31M.bz2 and those are in line with sizes of JF's files, who this time led the pack with the first pancakes.

I also have sieved as far as 30-31.8 on both sides and will upload those and then will get another range.

It's OK. It happens to any of us.

<S>

Looking at the uploads, xyzzy's runs do appear to have been done with 14e rather than 15e; please use 15e for the 25-30 range. It's quite a lot slower than 14e, but the extra yield over 14e means that we can use a shorter Q-range without running into duplicate relation issues, and get the whole thing finished quicker.

The binary we downloaded says 15e. How can we know for sure we have the right one?

We am using the binary from this post:

[URL]http://www.mersenneforum.org/showpost.php?p=152310&postcount=5[/URL] ([URL]http://snp.gnf.org/gnfs-lasieve4I15e.zip[/URL])

We renamed our existing binary and tested it:

[code]$ mv gnfs-lasieve4I15e gnfs-lasieve4I15e~
$ wget -q http://snp.gnf.org/gnfs-lasieve4I15e.zip
$ unzip -q gnfs-lasieve4I15e.zip
$ rm *zip
$ ls -l g*
-rwxrwxrwx 1 m m 820000 2008-11-07 17:11 gnfs-lasieve4I15e
-rwxr-xr-x 1 m m 820000 2008-11-07 17:10 gnfs-lasieve4I15e~
$ md5sum g*
e2396ee3232c5b5eea53d785d4f89cb2 gnfs-lasieve4I15e
e2396ee3232c5b5eea53d785d4f89cb2 gnfs-lasieve4I15e~[/code]All we want is a binary that is optimized for a Phenom quad running 64-bit Linux.

Here are 2 questions that are less important than the stuff above:
[LIST][*]In the polynomial file we have "alim: 125000000" and "rlim: 100000000". Obviously, we need to lower the alim value when working the the "A" side and we need to lower the rlim value when working the "R" side, if, for example, we are working at 25M. Is it safe to lower both in the polynomial file at the same time, so we can use a shared polynomial file?[*]Is there any potential loss (other than some overhead time) if you use smaller chunks versus larger chunks? We've played with 100,000 and 10,000 chunks so far. We've added up the elapsed time from the 10,000 chunks and the total is pretty close to a 100,000 chunk. Are we losing an data in the process using a smaller chunk?[/LIST]

[QUOTE=Xyzzy;158139]Here are 2 questions that are less important than the stuff above:
[LIST][*]In the polynomial file we have "alim: 125000000" and "rlim: 100000000". Obviously, we need to lower the alim value when working the the "A" side and we need to lower the rlim value when working the "R" side, if, for example, we are working at 25M. Is it safe to lower both in the polynomial file at the same time, so we can use a shared polynomial file?[/LIST][/QUOTE]

I think this would result in a significant loss of relations found per q. (test for example running gnfs-lasieve4I15e -a inputfile -o outputfile -f 25000000 -c 5000 both with rlim = 100M and rlim = 25M)

fivemack didnt you have a gnfs-lasieve4I??e binary that would sieve below the factorbase bound at one point

[quote]I think this would result in a significant loss of relations found per q. (test for example running gnfs-lasieve4I15e -a inputfile -o outputfile -f 25000000 -c 5000 both with rlim = 100M and rlim = 25M)[/quote]We are running the test you suggested. It is not done yet, but something tells us that you are right. Look at the report from "top" below. Note the memory used difference. The "-a" argument in the command line shows what is in the polynomial file.

[code] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6947 m 25 0 540m 231m 536 R 100 4.0 5:52.57 ./gnfs-lasieve4I15e -a a025r100 -f 25000000 -c 5000 -o a
6949 m 25 0 620m 270m 512 R 100 4.6 4:29.90 ./gnfs-lasieve4I15e -r a125r025 -f 25000000 -c 5000 -o r
6960 m 25 0 263m 100m 508 R 100 1.7 2:19.26 ./gnfs-lasieve4I15e -a a025r025 -f 25000000 -c 5000 -o ax
6967 m 25 0 263m 100m 496 R 100 1.7 1:19.60 ./gnfs-lasieve4I15e -r a025r025 -f 25000000 -c 5000 -o rx[/code]

Ah, yes ... I have measured the distribution of the primes in the two relation files, and indeed Batalov's run on R30-31M has many more relations with algebraic primes in the 30-125M range than xyzzy's, and if I account for that then it does seem that xyzzy is using -15e. The problem is that a relation with one prime in the 30-125M range and two large primes would be found with Batalov's parameters and not with xyzzy's.

If you are running a chunk on the R side from 27.93 to 27.94 million, rlim should be 27930000 and alim should be 125000000; so you need to have a different .poly file for each chunk you run.

[quote]If you are running a chunk on the R side from 27.93 to 27.94 million, rlim should be 27930000 and alim should be 125000000; so you need to have a different .poly file for each chunk you run.[/quote]Okay. (See our edited post above.)

We have run the "R" side semi-properly, we think. On the "A" side we took the poly file from the "R" side and edited it to fix the "A" side without fixing the "R" side back to the right value. Doh!

If the chunk is 31M to 32M, and you are working the "R" side, would you raise the rlim to 31M or would a generic 30M be okay?

We used 30M for the rlim for the whole 30-35M "R" chunk.

What parts do we need to redo?

:smile:

The test results are in.

[code]-rw-r--r-- 1 m m 1031656 2009-01-11 14:09 a
-rw-r--r-- 1 m m 573416 2009-01-11 13:56 ax
-rw-r--r-- 1 m m 972593 2009-01-11 14:07 r
-rw-r--r-- 1 m m 599596 2009-01-11 13:50 rx[/code][LIST][*]"A" side sieving 25,000,000 to 25,000,4999[LIST][*]a = alim = 25M & rlim = 100M[*]ax = alim = 25M & rlim = 25M[/LIST][*]"R" side sieving 25,000,000 to 25,000,4999[LIST][*]r = alim = 125M & rlim = 25M[*]rx = alim = 25M & rlim = 25M[/LIST][/LIST]

ay, there's the rub; do not lower both limits
 

My posted Opteron binary is 15e (even though it is a bit old and doesn't have the -R option). I've just checked - I sieve with exactly the same binary. One of its benefits is that you don't have to manipulate the *.poly file [I]ever[/I] (it lowers the necessary limit itself, internally).

Do not lower [I]both[/I] limits - when using any binary.
Using this particular binary (and GGNFS-built after 332) - do not lower [I]any[/I] limits.

However, the bottom line is that no intervals are not worth redoing (they have enough valuable data as they are, the relations are compatible if fewer, and you will produce more relations by doing another interval (this goes for me, too).

I'll take 65-70M, both sides.

[quote]Do not lower [I]both[/I] limits - when using any binary.
Using this particular binary (and GGNFS-built after 332) - do not lower [I]any[/I] limits.[/quote]Thanks!

We just uploaded the "R" side for 30-34M. You can tell our files because we use lower case letters. The file sizes look close in size to the others so we expect we haven't mangled them up too bad.

We are currently (and properly) working the "A" and "R" side of 34-35M and the "A" and "R" side of 25-30M.

The "A" side of 30-34M that we uploaded earlier is obviously suboptimal, but we gather that it is a workable situation.

Thanks for letting us participate!

[quote=Batalov;158173]However, the bottom line is that no intervals are [strike]not[/strike] worth redoing ...(if they are >50% done) [/quote]
My semantic had already been noted to be abstruse (see above). Well I am glad that you saw through it.

Yes, those r30-34 look quite good, so don't [URL="http://lyricwiki.org/Baz_Luhrmann:Everybody's_Free_(To_Wear_Sunscreen)"]berate yourself[/URL] too hard.
It's all good.

_____
[SIZE=1]"...sometimes you're ahead, sometimes you're behind.[/SIZE]
[SIZE=1]The race is long and, in the end, it's only with yourself.[/SIZE]
[SIZE=1]Remember compliments you receive.[/SIZE]
[SIZE=1]Forget the insults; if you succeed in doing this, tell me how." (c) Mary Schmich[/SIZE]

Res. 75-85.

Reserving 90-100.

We can understand (and dream of) having a lot of horsepower to throw at a job, but how do you manage to keep track of everything?

[quote=Xyzzy;159612]We can understand (and dream of) having a lot of horsepower to throw at a job, but how do you manage to keep track of everything?[/quote]

I know you're not referring to me here, but from my POV the answer is "scripts".

I'm not sure what arrangement of computers J.F. has access to, but for me the cluster uses a queuing system and thus all jobs are faciliated by qsub. Once a set of scripts is in place to divide up a range and kick off jobs to qsub, which then distributes them to worker nodes, the management of files goes pretty easy.

Yep, a couple of shell scripts around qsub makes life pretty easy.

To illustrate: I work on a 200 node cluster, each a dualcore 2Ghz Opteron. I chose to take jobs to be 1e5 large, which are automatically split over both cores (this was actually a bit tricky to get to work).

To fire 100 jobs in the queue I just type:
~/factoring/runduals.sh poly.txt -a 900 999
This script does something like
[code]
for block in $blockindices; do
~/factoring/submit_dual_blocks.sh $polyfile $ratalg $block $projectname
done
[/code]The real work is done in submit_dual_blocks.sh:
* generate poly file from template
* generate job file, tailored to dualcores
* submit

At any moment a postprocessing script can be executed to bake the pancakes which have all ingredients in place.
Uploading (with lftp) is still a manual action, don't really need to automate that.

If anyone is interested, I could place the files online somewhere.

OK, we've now got 100MQ collected and the shape of the relations-against-Q curve is pretty clear.

Would anyone be able to run 15-25 both sides? My 16 cores are quite solidly committed, but it looks as if I misplaced the point of diminishing sieving returns and it's worth doing these small Q.

This isn't what you are asking for, but we will have 25-30M done in a few days.

[quote]Would anyone be able to run 15-25 both sides?[/quote]We can run the "A" side for 15-20, but it will take about 10 days. Is that acceptable?

[SIZE=1]Did the new upload for a25-30.bz2 work? We renamed the corrupted file with a "-" at the end and also included a MD5 checksum for the new file.
[/SIZE]

We started the "A" side for 15-20 last night. If this is not cool just let us know.

A 15-20 sounds excellent; I'll take both sides 20-25 and leave R15-20 for some enterprising individual.

[quote=fivemack;160553]... and leave R15-20 for some enterprising individual.[/quote]
That would be me, I guess.

I forgot that the partition for collecting relations was of finite size and that I had left the complete sets of relations for two large SNFS projects on it, so the disc filled up. There is now 17G of free space, which ought to suffice.

What is the current status - how many relations do we have collected so far? Are more ranges needed to be sieved?

BTW: my range is 99.85% done and should finish today - I can upload it tonight or tomorrow in the evening.

I think most users would be best to go onto the polynomial search for 109!+1 now; we've got 150MQ searched, I'll build a matrix once I've finished 20-25 and xyzzy has finished 15-20, and if the matrix is really ugly I'll politely beseech J.F. to do another 10MQ.

I've ordered a new PSU for the matrix-solving box that broke, but there may be a further delay if the PSU's death killed the motherboard. I am unsure as to the net economic disadvantage of cheap but unreliable PSUs; losing the computer for two days every six months is only a 1% slowdown, and I wouldn't have paid the £35 difference between a £15 and a £50 PSU in order to make the computer 1% faster.

my pancakes should have reached Oslo.

I've finished my A's, and the R's will be done by tomorrow.

[quote]…and xyzzy has finished 15-20…[/quote]The 15-20 "A" side has been sent along with checksum file. We think we have done all we have committed to do. If not let us know.

The 15-20 file arrived in perfect shape; there are a few sections of 20-25 left for me to complete - pesky people running actually-relevant jobs on the computers I was using - but I expect to make a matrix over the weekend.

Thanks very much to all the contributors!

[QUOTE=fivemack;161633]The 15-20 file arrived in perfect shape; there are a few sections of 20-25 left for me to complete - pesky people running actually-relevant jobs on the computers I was using - but I expect to make a matrix over the weekend.

Thanks very much to all the contributors![/QUOTE]

Have my files have properly arrived?

[b]fivemack[/b]: Yes, they're all here, I've merged them and cut out the small duplicated sections.

A short delay
 

The computer continues to do its almost perfect rock imitation with a new PSU. Bother. I've got an RMA for the motherboard, but it'll be a week or so before I get to know if the new motherboard helps.

(the motherboard is the next bit to replace, isn't it?)

Bother! You don't suspect the memory? If only you had a twin (or a cousin) computer to swap parts, one by one...
_________
[SIZE=1][/SIZE]
[SIZE=1]"Books and bother killed my mother. Books and bother killed my mother. And my father, too." (c) The Element of Crime[/SIZE]

Not quite enough relations
 

The duplicate rate is very high, so the 233M raw relations that have usually sufficed have turned into 144M unique relations, which wasn't enough.

So, we need to sieve 100M-115M, both sides. Any offers?

Sorry, I can't help for the next few days. As soon as the cluster opens up, I'll check back here if the ranges still need to be done.

We have several jobs running on our (puny) "cluster" but we could run some work in a few days.

However, it takes ~10 days for us to burn through 5M on both sides. What time frame are you wanting results in?

Ten days would be fine - I've only got four cores on 100-102, so that range will take about ten days. If a bigger cluster comes on line then we can just search further in the same time.

We will start up 102 to 106 on both sides later tomorrow.

[QUOTE=Batalov;161861]Bother! You don't suspect the memory? If only you had a twin (or a cousin) computer to swap parts, one by one...[/QUOTE]

I have a machine which resembles the dead one roughly as much as a gibbon does a marmoset, but let me confirm that the newly-purchased PSU was definitely capable of supplying power.

The memory is a little dubious - the sticks wiggled a lot more than I remember when it came to taking them out - but since the other board has a socket1366 and DDR3 memory, and the non-working one has a socketAM2 and DDR2, part exchange isn't going to help me much.

I'm not quite sure how I'll proceed (except for 'calmly, and without rancour') if the motherboard gets returned no-fault-found ...

We are about 30-35% through 102-106 on both sides. The job is taking a bit longer than anticipated due to some other work which has higher priority. We hope that this glacial progress is satisfactory.

The computer is now fixed and I can devote four more 2.5GHz K10 cores to finishing the sieving; I've increased my reservation.

It would be very nice to have some contributors to sieve say 110-120, if anyone happens to have spare cycles at the moment.

I'll take 110 both sides.

I don't know if i can keep both cores running during office hours.

I'll finish up 111-120 on both sides.

bsquared: Superb, thanks a million!

Looks like it's [B]reservations closed[/B] for now!

[SIZE=1]Wow, that seemed to go quickly![/SIZE]

[quote=fivemack;162928]bsquared: Superb, thanks a million![/quote]

No problem, the timing was good as there was a relative lull in the cluster usage that allowed me to go at this with 96 cpus, give or take.

I hope you're ready to rock because there are 1.3GB of gzipped relations heading across the atlantic as I type :smile:.

I'm only half way done with my range (110M both sides). Let me know if you want what i've got sofar. Until then, i'll continue running this range.

Xyzzy, smh: keep running your ranges until they are done. I'll crawl forward from 120 with my small cluster, and see where I've got to by the time your ranges are finished. I really don't know how many relations will be enough here, depends on what the duplication rate is like.

looks like the connection was lost sometime overnight and only 190MB made it there. I'll try resending the whole thing now.

[quote]Xyzzy, smh: keep running your ranges until they are done.[/quote]We are about 2 or 3 days from being done.

2354284 relations are on it's way to Oslo as i write this.

Excellent. I will start baking the relations this evening; I think with 15M-100M we were very close to a matrix, so there should be enough now for a reasonably-sized one.

ETA for the matrix

[code]
Sat Feb 28 08:20:51 2009 matrix is 15785588 x 15785836 (4555.7 MB) with weight 1111879653 (70.44/col)
Sat Feb 28 08:20:51 2009 sparse part has weight 1036401865 (65.65/col)
[/code]

is April 2nd; maybe this could have done with a bit more over-sieving, but it takes quite a lot of time for a request for over-sieving to get answered, so I doubt I'd have collected in a week enough more relations to get the linalg time down from 33 days to 26.

I doubt I'd have collected in a week enough more relations to get the linalg time down from 33 days to 26.

Tom,

Is the intent to produce a particular result ASAP? Or, as I view the NFSNet effort, to maximize the results over time. In the latter case, the ability to start, stop, and restart sieving effort (and providing work for the "next" project during the down time), is critical to the efficient use of resources. The largest problem that I currently encounter is that too few of the resources are participating in an automated manner. This causes me to expend far too much of the really critical resource (personal time) to incorporate potential automated resources.

Richard

[code]
40199712048008930571428131875902950777477607917 * 44098347924570538726555032711112035074551402751062500493727027459523924634267 * 6267758127577332259078470898197707371104920617239537855165700322603688636346683832426409007658616045014902930862175109617951881840830949049
%5 = 11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111
[/code]

Some highlights (full log attached):
[code]
Fri Feb 27 19:57:50 2009 commencing relation filtering
...
Fri Feb 27 21:20:54 2009 found 113044024 duplicates and 171805386 unique relations
...
Sat Feb 28 03:57:26 2009 weight of 15859152 cycles is about 1110269217 (70.01/cycle)
...
Sat Feb 28 08:20:51 2009 matrix is 15785588 x 15785836 (4555.7 MB) with weight 1111879653 (70.44/col)
Sat Feb 28 08:20:51 2009 sparse part has weight 1036401865 (65.65/col)
Sat Feb 28 08:20:51 2009 matrix includes 64 packed rows
Sat Feb 28 08:20:52 2009 using block size 65536 for processor cache size 2048 kB
Sat Feb 28 08:22:03 2009 commencing Lanczos iteration (4 threads)
Sat Feb 28 08:22:03 2009 memory use: 4861.9 MB
...
Thu Apr 2 08:13:41 2009 lanczos halted after 249630 iterations (dim = 15785585)
Thu Apr 2 08:14:30 2009 recovered 33 nontrivial dependencies
...
Thu Apr 2 21:18:10 2009 reading relations for dependency 5
Thu Apr 2 21:18:22 2009 read 7895094 cycles
Thu Apr 2 21:19:59 2009 cycles contain 27711405 unique relations
Thu Apr 2 21:29:50 2009 read 27711405 relations
Thu Apr 2 21:38:34 2009 multiplying 22852064 relations
Thu Apr 2 22:38:16 2009 multiply complete, coefficients have about 598.57 million bits
Thu Apr 2 22:38:32 2009 initial square root is modulo 234499
Fri Apr 3 00:22:06 2009 prp77 factor: 44098347924570538726555032711112035074551402751062500493727027459523924634267
Fri Apr 3 00:22:06 2009 prp139 factor: 626775812757733225907847089819770737110492061723953785516570032260368863634668383242640900765861604501490293086217510\
9617951881840830949049
[/code]

Quite a large matrix - a smidgen smaller than the C180 GNFS. 820 hours on four cores of 2.5GHz Phenom to do the linalg steps. The duplicate rate was quite high by the end of sieving; I guess total sieving effort was 46,000 C2/2400-hours = 165Msec (IE fifteen times longer than the linalg).

Wagstaff and Kamada have been informed.

Congratulations! 31-bit projects rule.

Page 110 gets pretty full. (I wonder if it will instead open the page 111.)

[QUOTE=fivemack;167854][code]
40199712048008930571428131875902950777477607917 * 44098347924570538726555032711112035074551402751062500493727027459523924634267 * 6267758127577332259078470898197707371104920617239537855165700322603688636346683832426409007658616045014902930862175109617951881840830949049

Wagstaff and Kamada have been informed.[/QUOTE]

Nicely done. Now if we could only tackle the 1st few holes in the
base 11 and 12 tables........ These have languished for some time.

[QUOTE=R.D. Silverman;167904]Nicely done. Now if we could only tackle the 1st few holes in the
base 11 and 12 tables........ These have languished for some time.[/QUOTE]You called?

I may be able to help out there. The current work on 3,508+ has reached the filtering stage. Depending on the matrix, factors may be ready before the end of the month.

In the mean time, my sievers are idle (actually, they are running ECM on the Cullen and Woodall tables) and could do with a new project. I'd been thinking of a 800-bit C or W SNFS but may change my mind.


Paul

[QUOTE=xilman;167933]You called?

I may be able to help out there. The current work on 3,508+ has reached the filtering stage. Depending on the matrix, factors may be ready before the end of the month.

In the mean time, my sievers are idle (actually, they are running ECM on the Cullen and Woodall tables) and could do with a new project. I'd been thinking of a 800-bit C or W SNFS but may change my mind.


Paul[/QUOTE]

My CPU resources max out at around 780 bits (unless I want to spend 6
months on one number)........