Fixing the large dataset bug - Page 3

R.D. Silverman · 2014-06-25, 12:17

Quote:

Originally Posted by jasonp

p is 10743097, and when Pari failed to solve for the roots I realized p is composite, so this will never work. So it's a problem with the siever! The
offending relation is

Code:

368,487:42637,1d3a61:a3ed39,49f8d
                     ^^^^^

There is no primality test of factors that appear in relations; do we need one?

Perhaps. I have software that is run during the filtering stage.

My siever does not output all of the factors that appear in a relation.
One of the program inputs is a cutoff parameter that says to only output
primes greater than the cutoff.

It is therefore necessary, one a pass has been done to remove duplicates
and singletons (and perhaps do an initial filtering pass), to refactor all relations.

My code checks the norms, makes sure that all entries in a relation are
prime, and completes the factorizations before proceeding with full
filtering.

RichD · 2014-06-25, 12:25

Quote:

Originally Posted by jasonp

p is 10743097, and when Pari failed to solve for the roots I realized p is composite, ...

So is this what one would get if that relation survives dup and singleton removal?

Code:

error: relation nnnnnn corrupt

jasonp · 2014-06-25, 13:37

That's the message you get if a relation fails to correctly parse when starting the linear algebra; it should have been skipped earlier so this is a fatal error. I don't know why it would happen.

chris2be8 · 2014-06-25, 15:46

Quote:

Originally Posted by jasonp

There is no primality test of factors that appear in relations; do we need one?

Since the systems I'm sieving on don't have ECC memory I can't trust the sievers not to generate invalid relations. And I can't guarantee the are bug free. So a primality test to reject invalid relations would be useful.

Chris

jasonp · 2014-06-26, 02:49

Okay, if you have a small dataset that fails try running SVN 967 on it. This adds the primality test of relation factors in the first pass through the data file (code shamelessly stolen from SPWorley). The 107-digit test has seven relations containing composite factors.

Look in the log for a line 'skipped XXX relations with composite factors'

chris2be8 · 2014-06-26, 16:36

I've just got some very pleasant screen output:
prp38 factor: 19672027493011011110312010690534998063
prp75 factor: 330314774405936260487199694275045633281903844307841570763767379231675601507

That was from re-running filtering from scratch after removing free relations with "grep -v ':$'".

I didn't get a 'skipped XXX relations with composite factors' line though.

Next task, make this version live on all my systems.

Many thanks.

Chris

henryzz · 2014-06-26, 18:11

The big test is whether the large jobs that were failing now work. Anyone willing to try?

chris2be8 · 2014-06-26, 18:52

I'm now suspecting that the bug I hit isn't in msieve. The error occurred on an old system, but that doesn't have a recent version of SVN on it so I used a newer system to fetch msieve SVN 967. And I tested msieve SVN 967 on the newer system because that's faster. When I rechecked on the old system it failed again! Running on the new system with an old version of msieve I don't get the error.

Could using an old version of GMP cause the error? Or any other run time library? Were you able to reproduce the error on your system?

I'm sorry if this is all due to an old OS on my system.

Chris

jasonp · 2014-06-26, 20:45

Yes, I reproduced the error (though it didn't crash for me) and SVN 967 has several fixes for legitimate problems.

It may or may not have anything to do with the large dataset problems.

2014-06-26, 02:49	#27
jasonp Tribal Bullet Oct 2004 3,541 Posts	Okay, if you have a small dataset that fails try running SVN 967 on it. This adds the primality test of relation factors in the first pass through the data file (code shamelessly stolen from SPWorley). The 107-digit test has seven relations containing composite factors. Look in the log for a line 'skipped XXX relations with composite factors' Last fiddled with by jasonp on 2014-06-26 at 02:52

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
48-bit large primes!	jasonp	Msieve	24	2010-06-01 19:14
a^n mod m (with large n)	Romulas	Math	3	2010-05-08 20:11
NFS with 5 and 6 large primes	jasonp	Factoring	4	2007-12-04 18:32
Why only three large primes	fivemack	Factoring	18	2007-05-10 12:14
very large exponents	pacionet	Data	4	2005-11-04 20:10

2014-06-25, 13:37	#25
jasonp Tribal Bullet Oct 2004 3,541 Posts	That's the message you get if a relation fails to correctly parse when starting the linear algebra; it should have been skipped earlier so this is a fatal error. I don't know why it would happen.

2014-06-26, 16:36	#28
chris2be8 Sep 2009 100000011110₂ Posts	I've just got some very pleasant screen output: prp38 factor: 19672027493011011110312010690534998063 prp75 factor: 330314774405936260487199694275045633281903844307841570763767379231675601507 That was from re-running filtering from scratch after removing free relations with "grep -v ':$'". I didn't get a 'skipped XXX relations with composite factors' line though. Next task, make this version live on all my systems. Many thanks. Chris

2014-06-26, 18:11	#29
henryzz Just call me Henry "David" Sep 2007 Cambridge (GMT/BST) 2³×3×5×7² Posts	The big test is whether the large jobs that were failing now work. Anyone willing to try?

2014-06-26, 18:52	#30
chris2be8 Sep 2009 4036₈ Posts	I'm now suspecting that the bug I hit isn't in msieve. The error occurred on an old system, but that doesn't have a recent version of SVN on it so I used a newer system to fetch msieve SVN 967. And I tested msieve SVN 967 on the newer system because that's faster. When I rechecked on the old system it failed again! Running on the new system with an old version of msieve I don't get the error. Could using an old version of GMP cause the error? Or any other run time library? Were you able to reproduce the error on your system? I'm sorry if this is all due to an old OS on my system. Chris

2014-06-26, 20:45	#31
jasonp Tribal Bullet Oct 2004 3,541 Posts	Yes, I reproduced the error (though it didn't crash for me) and SVN 967 has several fixes for legitimate problems. It may or may not have anything to do with the large dataset problems.