BOINC NFS sieving - RSALS

[jasonp]

Serge is fond of pointing out that gzip'ed files can be concatenated together and will still decompress properly.

In windows, 'copy /b file1+file2+...+filen output_file' should work

[debrouxl]

Serge is fond of pointing out that gzip'ed files can be concatenated together and will still decompress properly.

Indeed, RSALS has been using that property (shared by bzip2) for more than two years
But for F1887 and several other numbers, in order to save space on the server (routinely ~2 GB for a 30-bit LPs task), I froze a state of the .dat.gz file, ran remdups4 on it and pbzip2'ed the result, yielding a .dat.bz2 file, not directly usable by msieve.
Later, more results are returned by the clients, which get concatenated automatically to .dat.gz file that restarted from zero. It's best for the post-processor to get both the .dat.bz2 file and the .dat.gz file, but they do not mix and match together directly

[pinhodecarlos]

LA for F1887 started. Lionel, if you want you can delete the main file to free up some space on the server.

Edit: Why do you keep the two main files on the server for 11_339_minus1?

[debrouxl]

LA for F1887 started. Lionel, if you want you can delete the main file to free up some space on the server.

Done.

Edit: Why do you keep the two main files on the server for 11_339_minus1?

Because I forgot to remove one

[pinhodecarlos]

I quick note. I managed to merge the two files after a quick search on the forum on how to do it but I decompressed both before merging. Sorry for the question. Anyway, I didn't know about the concatenate gzip'ed files. Thank you.

[Dubslow]

Originally Posted by Dubslow

I'll do 160969_43_minus1 (it looks a bit closer to done than the one above it).

Well... this is awkward.

I used the relations file for this number as the test for this post about rel file format and compression.

I had forgotten to unzip it, and I did that when the LA completed (it was zipped with bzip2). Unfortunately, as always, the un/compression caused some corruption.

commencing square root phase
reading relations for dependency 1
read 3407441 cycles
cycles contain 9373226 unique relations
error: relation 45392007 corrupt

I manually looped over all 34 dependencies, and each time there was at least one corrupt relation.

The simplest method to fix this would be to remdups the rel file, tossing the bad rels, and redo the LA, but that would take a while and a lot of compute time.

A more efficient method would be to remdups the file to get the bad rels, and then fix them by hand, then re-add them to the rel file in the right location. This would be rather difficult, and I don't know how to construct rels.

To make matters worse, I still need to RMA my graphics card, and Linux won't boot without it (though Windows will), and I'm moving in to college tomorrow anyways, so my computer will be out of action anyways.

As such, I'm linking all the relevant files here. This is a link to the .cyc, .dep, and .fb files (compressed, 174 MB) this is a link to the uncompressed rel file (around 11.7 GB).

(Edit: Of course, these links won't work tomorrow, but they should work overnight and after tomorrow.)

[Batalov]

Originally Posted by jasonp

Serge is fond of pointing out that gzip'ed files can be concatenated together and will still decompress properly.

In windows, 'copy /b file1+file2+...+filen output_file' should work

Yes, under the condition that all parts 'gzip -tv' without errors.
(Otherwise there will be more hassle later to gzip repair.)

gzip standard was stream-ready at design; a gzip file (or a stream) is like a train of cars (proper chunks, identified by a valid header record), so if we connect two trains the result is a "train" again. Small gzipped files are single cars, or more. But if one train was chopped off (or was broken otherwise), the combined object will be repaired with more lost parts. The first car of the second train will stick to the broken off last car of the first train and both will be discarded. (the gzip repair kit splits the train into single cars and throws away invalid ones; then the user is free to reuse valid chunks. It is a bit of a pain to do but is a valuable life experience. Like a scar. ;-) )

[RichD]

GC_5_334 splits as ...

prp66 = 450956348716753108838639638208075170303357174618993330771280648693
prp169 = 1063506952024717121188111286584965562004890575356949552987937630434522275071306932720481262782749988010907589767937519608529842432033670755953761378482617292207519668293

[debrouxl]

I'll take care of 160969_43_minus1.

[xilman]

Originally Posted by RichD

GC_5_334 splits as ...

prp66 = 450956348716753108838639638208075170303357174618993330771280648693
prp169 = 1063506952024717121188111286584965562004890575356949552987937630434522275071306932720481262782749988010907589767937519608529842432033670755953761378482617292207519668293

Thanks!

Paul

[RichD]

1489_71_minus1 splits as ...

prp77 = 45896447589970773832931784986400490560268820308727411473025645680276882636169
oro146 = 27614776777951268487383853140458483138826979094458150355339368759958690289056222634422197634724899284008789933412583236893293328082409931239962079

I'll start the download of 601_83_minus1 later today.