v1.40 patch for massive NFS oversieving

[Batalov]

Let's simply do the gzipped relations. (This is all about 5,362+, btw.)
It will take a day to FTP; we have ~1Gb/hour transfer rate.

I will use the patch (if it isn't already in 1.41).
I will bump the starting weight to <=30 (not 20) right away, too.
Thanks, --S

[Jeff Gilchrist]

Originally Posted by Batalov

Let's simply do the gzipped relations. (This is all about 5,362+, btw.)
It will take a day to FTP; we have ~1Gb/hour transfer rate.

I was curious about various compression methods with sieve data so I took my 963MB msieve.dat file from a C121 and compressed them with a few different algorithms to see if it would make a big difference. Results:

gzip -9 = 457MB
bzip2 -9 = 398MB
7zip (LZMA Max) = 391MB
7zip (PPMd Max) = 366MB

So by using the free 7zip program which works in both Windows and UNIX, you can save some time in transferring the data compared to gzip.

Jeff.

[mdettweiler]

Originally Posted by Jeff Gilchrist

I was curious about various compression methods with sieve data so I took my 963MB msieve.dat file from a C121 and compressed them with a few different algorithms to see if it would make a big difference. Results:

gzip -9 = 457MB
bzip2 -9 = 398MB
7zip (LZMA Max) = 391MB
7zip (PPMd Max) = 366MB

So by using the free 7zip program which works in both Windows and UNIX, you can save some time in transferring the data compared to gzip.

Jeff.

One thing to keep in mind about 7-Zip/ppmd is that it uses a LOT of RAM for compression and decompression. The exact amount is configurable when compressing. If you have plenty of RAM, this wouldn't appear to be a problem; however, I've found that if you're decompressing under Windows, if it requires more than approximately 1.2-1.5 GB of RAM, you may run into the 2 GB per-process limit that Windows (32-bit) imposes. (The reason why you can't use the full 2 GB has something to do with how it reserves a certain chunk of RAM for the system, I think.)

Of course, you shouldn't have any problems if you're either running 64-bit Windows, or Linux (32- or 64-bit). For 64-bit Windows the limit is much higher, and if memory serves Linux will let you use everything that the respective 32-bit/64-bit stuff can see.

[Joshua2]

I had actually already run this test and discovered that my 7z ppmd could get 37% compression ratio, which was better than anything else. I had it set to ultra though, I talked about it over in the aliquot team sieve thread. PPMd is the best for text, so I guess this is text-like.

[Jeff Gilchrist]

Originally Posted by Joshua2

I had actually already run this test and discovered that my 7z ppmd could get 37% compression ratio, which was better than anything else. I had it set to ultra though, I talked about it over in the aliquot team sieve thread. PPMd is the best for text, so I guess this is text-like.

I don't follow aliquot stuff so I missed that thread. So PPMd Ultra takes it from 366MB to 362MB.

The msieve.dat file is a text file so that would explain why PPMd does so well.

Jeff.

[Batalov]

Originally Posted by jasonp

It's in v1.41, you're on the bleeding edge :)

Well, together with a lucky guess of w<=30 (otherwise we would have waited for the third pass), the filtering went smoothly and landed on the target density in a single pass, in only 11 hours. The matrix is rather good-looking for a complexity-253 number... 10.6Mx10.6M and the ETA for linalg is 9 days on 4 threads. The highest memory use watermark was 4.3Gb, and now in BL is 3.7Gb (3.9Gb virtual). With the OS, it would have gone swapping on a 4Gb machine (but this one is 8Gb).

We'll probably use .7z for the next transfer.

The filtering log is attached; I am not sure whether it shows that the massive patch was used, but Jason may spot it...

[jasonp]

Originally Posted by Batalov

The filtering log is attached; I am not sure whether it shows that the massive patch was used, but Jason may spot it...

It wasn't used (look for 'commencing heavy relation removal'); you had a huge number of cliques but apparently did not need to prune enough for the heavy relation removal to be needed.

Actually, if you had the memory for it then it may have been profitable to use an ideal weight of 35 instead of 30. If the estimated average weight after the merge phase is a lot higher than 70.0, then it's probably because the merge phase went and merged all the ideals; but if you had a higher merge weight then the merge phase could afford to be more selective. At least I've noticed that when there is a lot of oversieving, as the max ideal weight goes up the average matrix weight also goes up, until with a high enough weight it drops down to the expected average but remains at about the same dimensions. Could you retry with weight <= 35 after the LA finishes?

[Batalov]

Originally Posted by jasonp

Could you retry with weight <= 35 after the LA finishes?

OK. Will do. But in the meantime, here's the full log for the 2,1618L which went all the way up exactly to w<=35 and made a fantastic matrix, which was solved in mere 94 hours.