Interesting Screw-up

R.D. Silverman · 2005-08-25, 00:11

The linear algebra just finished on 2,969+ and it failed to find any true
dependencies. It did find 64 pseudo dependencies.

I just found out why.

The filtering first does two passes, looking at
the number of excess relations after the first pass to adjust parameters
for the second.

The output files only contain primes above a certain threshold (typically
250k). It is therefore necessary to completely factor all relations before
doing a final filter pass and then building/solving the matrix.

Apparently, something caused the re-factor to fail. I suspect a power failure
in the middle of the computation.

I failed to notice that it had failed, and started up the final filter pass.
Therefore the final matrix only contained primes greater than 250K. Ooops!

I am re-doing the entire process now.

Maybe this will teach me to be more careful.

sean · 2005-08-25, 01:21

You have my sympathy.

Unfortunately, you can be almost certain that something similar will happen in the future. It's pretty hard to get everything right all the time.

When I did a C147 last year, I similarly wasted 20+ days solving a matrix that turned out to be broken. The cause ended up being a corrupt file left over from a machine crash that the filtering stage had picked up and included. It took me a while to track the cause down.

xilman · 2005-08-25, 08:44

Quote:

Originally Posted by R.D. Silverman

The linear algebra just finished on 2,969+ and it failed to find any true
dependencies. It did find 64 pseudo dependencies.

...

I am re-doing the entire process now.

Maybe this will teach me to be more careful.

Been there, done that.

When NFSNET factored 2,811- the dataset was so large I had serious problems filtering. One problem arose from a bug in Windows' IO library -- after reading more than 4G of text files (WIndows distinguishes between text and data) it lost track of where the line endings (CRLF on Windows, contrast with CR on MacOS and LF on Unices). The relations files became subtly corrupted and I was unable to build a "small" matrix. In desperation, I ran an enormous but very sparse matrix on the cluster. I forget the details, but it ran for a long time on a 32-cpu cluster, only to produce useless dependencies.

Paul

VJS · 2005-08-25, 15:48

Bob,

May I ask you:
- how long the entire process took
- Computers
- HD space
- memory requirements

2,969+ seems pretty big...

2005-08-25, 00:11	#1
R.D. Silverman Nov 2003 1110100100100₂ Posts	Interesting Screw-up The linear algebra just finished on 2,969+ and it failed to find any true dependencies. It did find 64 pseudo dependencies. I just found out why. The filtering first does two passes, looking at the number of excess relations after the first pass to adjust parameters for the second. The output files only contain primes above a certain threshold (typically 250k). It is therefore necessary to completely factor all relations before doing a final filter pass and then building/solving the matrix. Apparently, something caused the re-factor to fail. I suspect a power failure in the middle of the computation. I failed to notice that it had failed, and started up the final filter pass. Therefore the final matrix only contained primes greater than 250K. Ooops! I am re-doing the entire process now. Maybe this will teach me to be more careful.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Some interesting patterns regarding mod	CuriousKit	Miscellaneous Math	24	2015-04-06 18:40
An interesting series	Citrix	Math	0	2005-11-02 05:33
A new interesting thing about 15k	robert44444uk	15k Search	0	2005-04-06 23:00
very interesting reading	stippix	Hardware	0	2004-09-27 06:19
Something Interesting	clowns789	Hardware	1	2003-12-20 12:36

2005-08-25, 01:21	#2
sean Aug 2004 New Zealand DE₁₆ Posts	You have my sympathy. Unfortunately, you can be almost certain that something similar will happen in the future. It's pretty hard to get everything right all the time. When I did a C147 last year, I similarly wasted 20+ days solving a matrix that turned out to be broken. The cause ended up being a corrupt file left over from a machine crash that the filtering stage had picked up and included. It took me a while to track the cause down.

2005-08-25, 15:48	#4
VJS Dec 2004 299₁₀ Posts	Bob, May I ask you: - how long the entire process took - Computers - HD space - memory requirements 2,969+ seems pretty big...