View Single Post
Old 2018-08-15, 17:27   #1
(loop (#_fork))
fivemack's Avatar
Feb 2006
Cambridge, England

13·491 Posts
Default Corrupt data files discovered during post-processing

When I run

 gunzip -c L3655A.dat.gz | nl | grep -a "000000[^0-9a-f,]" | tee every-millionth
it suggests that there is a large block of corrupt data in the middle of L3655A.dat.gz, meaning that there are only 378M or so lines in the file despite the log reporting 455M relations.

I'm having a bit of difficulty working out which Q-values are affected by this, because some versions of the sieving client put the special-Q at the end of the list of factors and some sort the factors numerically, so given a line it's not trivial to find the special-Q that it was.

My inclination is to write a better special-Q-finder, figure out the corrupt ranges, and sieve them myself with 16e; I'm not sure I'll get that done before I go to Australia in eight days.
fivemack is offline   Reply With Quote