In Progress? - Page 3

Raman · 2010-05-02, 13:42

Quote:

Originally Posted by bdodson

I'm still not entirely sure that a wake-up call for a pre-report is merited;
but am willing to be prepared for another factor report soon, while hoping
not to hear new/worrisome error reports.

3,581+ is due tomorrow, actually. You mean about that number only, right?

The only thing that I am afraid of, is that scarcity of relations, in order to not have a congruence of squares at all. Does filtering work out, thereby producing a matrix, even when there is deficiency of relations for running out of that dependencies, or that leads all the dependencies to be trivial?
Fortunately, that this has not occurred for any of my jobs till now, or others' jobs as well, so far. 80 to 85 million relations is sufficient to successfully complete any factorization with 31/29 bit large primes (or that 30/30 pair, of course)

But, that matrix size reduces as we add on more and more relations. Thus, merging more of partial relations, that decreases number of factor base primes for that matrix. For 10,351+ I had a 6.7M matrix, then that for just 3 digits of SNFS difficulty increase, simply, for that candidate 3,581+ I have a 8.96M size matrix right now. (May be with lesser number of relations, of course). How big a matrix, do you expect that for 5,413+? Or that the one that you had for 12,266+ case?

Three "conveyer" numbers = 10,363- 10,363+ 5,413+
that harder 16e snfs = 2,919- rather

Batalov · 2010-05-02, 17:50

Quote:

Originally Posted by Raman

5,400+ is certainly due tomorrow (May 1)
3,581+ falling upon May 3
This is for 100% sure, of course!

I think that these were particular statements that Bruce was alluding to. Nothing is 100% sure. You have seen that before (we remember your messages full of "what the hell"s), so you should know from your own experience.

But it was not math that gave you trouble and not even the particular programs. As far as I remember, you had not enough memory several times and you had data transfer errors on another occasion. Your latest logs are still showing a lot of errors -15; unlike -6/-11 errors, these are an indicator of hardware failures (and possibly overheating), maybe bad network.

While the reasons of your concern are valid, they would be probably somewhere under #4, #5 on my list of worries. Here's what I would worry about:
1. Human error. E.g. deleting the file, overflowing the disk with another computations, so that there's no space left and the data is lost; starting -nc2 when intending -ncr, etc etc etc.
2. Hardware failure: the hard drive, the computer, network glitch that would put a string of zeroes in the transferred file.
3. Unknown, never yet seen bugs in gnfs-lasieve or msieve. That's rare.
4, etc. Then, maybe, the "scarcity of relations", failing srqts, etc? (the failing of the "math"?) I wouldn't worry about it, because all of your polynomials are basically the same; you are in a relatively safe territory.

P.S. Your latest matrices should be anywhere from 7 to 11M in size depending on the amount of oversieving. These are not very large. Your question about the size was not very well posed: given any composite of you range, you can build a 13M matrix or you can build a 9M if you sieve for a week more.

Raman · 2010-05-02, 19:31

Quote:

Originally Posted by Batalov

While the reasons of your concern are valid, they would be probably somewhere under #4, #5 on my list of worries. Here's what I would worry about:
1. Human error. E.g. deleting the file, overflowing the disk with another computations, so that there's no space left and the data is lost; starting -nc2 when intending -ncr, etc etc etc.
2. Hardware failure: the hard drive, the computer, network glitch that would put a string of zeroes in the transferred file.
3. Unknown, never yet seen bugs in gnfs-lasieve or msieve. That's rare.
4, etc. Then, maybe, the "scarcity of relations", failing srqts, etc? (the failing of the "math"?) I wouldn't worry about it, because all of your polynomials are basically the same; you are in a relatively safe territory.

The other thing that I can fear about can be Linear algebra failures,
that require long runs again, instead of short recovery cases (rather than that way)
We have heard about "submatrix not invertible, linear algebra failed, retrying" or that "Newton iteration failed to converge" - the latter is for that square root, of course. But, I have as well as seen that cases within this forum where the matrix goes upto > 100%. How can it be so? Why, could it go on like that, thus?
If it is one of these failures that occur up, then the whole Linear Algebra has to be restarted up again, rather than from that checkpoint, maybe that filtering as well?

Batalov · 2010-05-02, 21:49

The latest data invariably shows that "submatrix not invertible, linear algebra failed, retrying" error was in every case a hardware error. Almost convincingly the same for >100% progress error, ...even though theoretically Block Lanczos is not guaranteed to converge, so this is debatable.

Thanks to Jasonp's orthogonality check implementation, this is now a thing of the past. (Unless you use ancient binaries, which I am afraid to say, you do - judging by the logs, you are still using 1.41?) Compile and use the latest binaries. Incidentally, after SVN 262 you will find errors -6/-11 gone. If you will still have errors -1, -15 and others - that also points to hardware problems.

The rule of thumb is that you can use even decidedly known bad computers to sieve. But do locate and use your best, most stable computer to filter and LA. It seems that you do that, so that's good.

Raman · 2010-05-02, 22:26

Quote:

Originally Posted by Batalov

Thanks to Jasonp's orthogonality check implementation, this is now a thing of the past. (Unless you use ancient binaries, which I am afraid to say, you do - judging by the logs, you are still using 1.41?)

I make use of msieve version 1.41 for square root, since you noticed that msieve 1.43 gave up a crash (segmentation fault) while multiplying up that relation sets. As well as I do use up with 1.41 for that linear algebra resuming, since with msieve 1.43 it sometimes gave up with that error "checkpoint recovery failed"
That msieve version 1.41 is always safe enough, in my opinion, only actually

Batalov · 2010-05-02, 23:25

Quote:

Originally Posted by Raman

I make use of msieve version 1.41 for square root, since you noticed that msieve 1.43 gave up a crash (segmentation fault)

That's bad hardware.

Quote:

Originally Posted by Raman

As well as I do use up with 1.41 for that linear algebra resuming, since with msieve 1.43 it sometimes gave up with that error "checkpoint recovery failed"

Bad hardware.

Quote:

Originally Posted by Raman

That msieve version 1.41 is always safe enough, in my opinion, only actually

Good luck, then.

bdodson · 2010-05-03, 00:26

Quote:

Originally Posted by R. Gerbicz

http://homes.cerias.purdue.edu/~ssw/cun/page117

At the moment, unless my browser isn't properly refreshing, this page seems
located under "old pages"! -bd

ps - I'm not sure I can get used to this much linearity.

pps - Hmm, on 2nd look, page 117 seems to be on the same level
as the main Cunningham page. Just there isn't a link on the main
page (before page 116) yet.

Raman · 2010-05-03, 04:41

LATE than I had expected up.

at the moment for 3,581+
linear algebra completed 8439570 of 8959019 dimensions (94.2%, ETA 16h25m)

16 hours remaining means that it will not be completed up today.
I am going out of station tomorrow
Thus, only 3,581+ upon May 5, 2010
certainly. Time widely being available to complete up that square root
to any number of dependencies.

Or that is it possible to send that mail, log file as that text (or thus as an attachment rather),
from the Linux terminal command line, by using a cron job
into my Gmail account directly, thus, at any time?

Quote:

Originally Posted by bdodson

I'm still not entirely sure that a wake-up call for a pre-report is merited

What do you mean by merited?

Raman · 2010-05-03, 11:40

I will not be available tomorrow at all.

Suppose that the linear algebra ends up at 2 am IST, I don't know how much time it will take up
for that square root, how many dependencies will it go upto, assuming that 2.5 hours per dependency rate

Somehow, I have set up a CRON job to automatically send mail to Prof.Sam with the factors of 3,581+
tomorrow a little before 3:30 pm IST. (GMT +05:30). I hope that he will receive the factors tomorrow morning according to his time.

The mail from the compute cluster goes to the department mail server, that forwards to my Gmail account, thereby forwards to that Prof.Sam's mailbox.
Let's see if it works out properly or not.
Tomorrow morning at that Purdue University (Tuesday morning)
Thus, saving up with that one day!

-----------------------------------------------------------------------------------------------------------------------------------------------------------------

This is a randomly generated RSA key from my Linux machine terminal.
What are the public keys, private keys, N within that decimal notation? How to find out all that things?

id_rsa.pub

Code:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAsoAWN4RCE+mgLAUB8y9NIALGEfuuzLqsLkcQQPIuF6E9cSVAoWEpifTGjp5SHJBksFrpMV5/28HDh2zXpu+siaNus2Rf/agmKSgbHYY6yfucwFgC7R9Uk4ONJ3ax/WvO/RdEWmYusvGHm82PoJPLxhs0n4cdKpA4VpTtPxwb5vAPuIVLygC0EatRW8f+hyIHPHf5lTqIiQ9E9Upzmvp+/1yf09Rz/JRFif05jNctRXFBLbcxgiugrdpJjFnkwuYMTvbToOQr3mFDjbAsf1stPysVP6gNTQOaXrAb1AhiVf3LR4RCMPXu76nD/1sBOghnqfVLbvCnnAFnHfxBpr2I2Q== cs09m038@leo0

id_rsa

Code:

jasonp · 2010-05-04, 10:00

Quote:

Originally Posted by Raman

I make use of msieve version 1.41 for square root, since you noticed that msieve 1.43 gave up a crash (segmentation fault) while multiplying up that relation sets. As well as I do use up with 1.41 for that linear algebra resuming, since with msieve 1.43 it sometimes gave up with that error "checkpoint recovery failed"
That msieve version 1.41 is always safe enough, in my opinion, only actually

Your previous checkpoint recovery problems were because you generated a checkpoint with v1.41 and the checkpoint file format changed in v1.42 to incorporate self-checking in the linear algebra. This means that if you
ever update the msieve version that you use, you need to do both the LA and the square root with versions >= 1.42

Andi47 · 2010-05-04, 12:03

Quote:

Originally Posted by Andi47

Yes, I have already started with 7,393+. By now I have sieved ~60% of the subrange of 20-30M and ~40% of the subrange of 30-40M, so it will still take quite a while.

Current status after sieving the subranges 20.1M-26.5M, 27M-29.5M and 30M-38M: 47798108 raw (=sieved per ggnfs) plus 812488 free relations, 45781554 unique. So I guess I'm approx. halfway through sieving.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Overall Project Progress	KEP	Conjectures 'R Us	54	2018-01-04 18:25
Progress	bsquared	YAFU	20	2014-05-22 16:52
Nice progress!	schickel	FactorDB	29	2012-07-18 17:03
Progress	R.D. Silverman	Factoring	0	2012-05-22 14:03
Picturing progress	ATH	Data	1	2006-06-22 23:04

2010-05-02, 21:49	#26
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 2⁴·593 Posts	The latest data invariably shows that "submatrix not invertible, linear algebra failed, retrying" error was in every case a hardware error. Almost convincingly the same for >100% progress error, ...even though theoretically Block Lanczos is not guaranteed to converge, so this is debatable. Thanks to Jasonp's orthogonality check implementation, this is now a thing of the past. (Unless you use ancient binaries, which I am afraid to say, you do - judging by the logs, you are still using 1.41?) Compile and use the latest binaries. Incidentally, after SVN 262 you will find errors -6/-11 gone. If you will still have errors -1, -15 and others - that also points to hardware problems. The rule of thumb is that you can use even decidedly known bad computers to sieve. But do locate and use your best, most stable computer to filter and LA. It seems that you do that, so that's good.