mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > NFS@Home

Reply
 
Thread Tools
Old 2018-08-15, 17:27   #1
fivemack
(loop (#_fork))
 
fivemack's Avatar
 
Feb 2006
Cambridge, England

2×11×172 Posts
Default Corrupt data files discovered during post-processing

When I run

Code:
 gunzip -c L3655A.dat.gz | nl | grep -a "000000[^0-9a-f,]" | tee every-millionth
it suggests that there is a large block of corrupt data in the middle of L3655A.dat.gz, meaning that there are only 378M or so lines in the file despite the log reporting 455M relations.

I'm having a bit of difficulty working out which Q-values are affected by this, because some versions of the sieving client put the special-Q at the end of the list of factors and some sort the factors numerically, so given a line it's not trivial to find the special-Q that it was.

My inclination is to write a better special-Q-finder, figure out the corrupt ranges, and sieve them myself with 16e; I'm not sure I'll get that done before I go to Australia in eight days.
fivemack is offline   Reply With Quote
Old 2018-08-15, 17:41   #2
pinhodecarlos
 
pinhodecarlos's Avatar
 
"Carlos Pinho"
Oct 2011
Milton Keynes, UK

7×673 Posts
Default

Is it possible to cheat on sieving?
pinhodecarlos is online now   Reply With Quote
Old 2018-08-15, 21:38   #3
fivemack
(loop (#_fork))
 
fivemack's Avatar
 
Feb 2006
Cambridge, England

635810 Posts
Default

At a first analysis, there are no special-Q larger than 966392000 in the output file despite sieving supposedly having been done to 1200M

I don't see any suspiciously large gaps before Q=880M or so.
fivemack is offline   Reply With Quote
Old 2018-08-15, 21:46   #4
fivemack
(loop (#_fork))
 
fivemack's Avatar
 
Feb 2006
Cambridge, England

11000110101102 Posts
Default

With my embarrassed hat on, I also need to point out that I forgot to put an lss: 0 directive in the polynomial file so all the sieving was done on the lower-yielding side - I wondered why the yield was so low.

I will fix this, it will take a few tens of thousands of core hours at my side but that seems a reasonable penalty for me to pay.
fivemack is offline   Reply With Quote
Old 2018-08-15, 21:51   #5
RichD
 
RichD's Avatar
 
Sep 2008
Kansas

1100000101102 Posts
Default

Quote:
Originally Posted by fivemack View Post
I will fix this, it will take a few tens of thousands of core hours at my side but that seems a reasonable penalty for me to pay.
You are only human like the rest of us. I have no problem letting the grid help this number out...
RichD is offline   Reply With Quote
Old 2018-08-16, 00:20   #6
swellman
 
swellman's Avatar
 
Jun 2012

B2716 Posts
Default

Agreed. Penance is for sins, not simple mistakes. Use the grid.

No stones being thrown here!
swellman is offline   Reply With Quote
Old 2018-08-16, 01:11   #7
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

29×149 Posts
Default

On my home copies of lasieve4e, Qmax is near 1060M. I get errors above that and the sieve exits without trying any Q's. Perhaps someone might test-sieve Q=1100M or 1200M to see if that's even possible on the BOINC-ified sievers?
VBCurtis is offline   Reply With Quote
Old 2018-08-16, 07:04   #8
frmky
 
frmky's Avatar
 
Jul 2003
So Cal

2,039 Posts
Default

Quote:
Originally Posted by VBCurtis View Post
Perhaps someone might test-sieve Q=1100M or 1200M to see if that's even possible on the BOINC-ified sievers?
It's not for all 14e and 15e versions. IIRC the 64-bit Linux and FreeBSD clients will work, but the Mac and Windows ones will fail.
frmky is offline   Reply With Quote
Old 2018-08-16, 07:24   #9
fivemack
(loop (#_fork))
 
fivemack's Avatar
 
Feb 2006
Cambridge, England

635810 Posts
Default

OK: my computers are doing the small gaps and the grid job L3655Ab is taking the strain. I will have to manage some of this on an iPad from the other side of the planet, which might be less efficient than the most efficient protocol but should work.
fivemack is offline   Reply With Quote
Old 2018-08-17, 09:57   #10
fivemack
(loop (#_fork))
 
fivemack's Avatar
 
Feb 2006
Cambridge, England

2×11×172 Posts
Default

I have replaced L3655Ab with L3655Ac, which has a more sensible sieving region and, importantly, remembered to put the alim: and rlim: lines in the polynomial file.
fivemack is offline   Reply With Quote
Old 2018-10-13, 10:57   #11
fivemack
(loop (#_fork))
 
fivemack's Avatar
 
Feb 2006
Cambridge, England

2×11×172 Posts
Default

Another data quality issue: 5009_73m1 only has 345131720 usable relations despite a claim of 450653495, and so I can't yet build a matrix. I will investigate which regions are missing and put in a new sieving job.
fivemack is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
16e Post Processing Progress pinhodecarlos NFS@Home 8 2018-11-28 13:45
NFS@Home Post-Processing Rack Build pinhodecarlos NFS@Home 1 2016-09-27 12:34
Crash doing large post-processing job wombatman Msieve 22 2013-12-04 01:37
Update on 7^254+1 post processing dleclair NFSNET Discussion 4 2005-04-05 09:51
Post processing for 2,757- xilman NFSNET Discussion 3 2003-11-06 14:23

All times are UTC. The time now is 05:30.

Sat Sep 26 05:30:40 UTC 2020 up 16 days, 2:41, 0 users, load averages: 1.55, 1.60, 1.64

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.