Current status

[fivemack]

All this mopping-up of old threads appears to have concealed any discussion of the current project status.

Both my machines have been working on 3_479P_1 for the last week or so, while nfsnet.org asserts that the NFSNET project is only 82% through 2_1466L. Am I misconfigured, or have the sievers now been split across two projects?

What's the current linear-algebra status?

[R.D. Silverman]

Originally Posted by fivemack

All this mopping-up of old threads appears to have concealed any discussion of the current project status.

Both my machines have been working on 3_479P_1 for the last week or so, while nfsnet.org asserts that the NFSNET project is only 82% through 2_1466L. Am I misconfigured, or have the sievers now been split across two projects?

What's the current linear-algebra status?

Clearly there is some kind of server problem.

The LA for 2,764+ *should* be close to done.

Has 3,479+ started? I thought 2,797+ was going to be done first????
George and Paul Zimmermann have the number "reserved"......

[Wacky]

I met with Paul in Houston this weekend and can provide some updated information.

2,1466L has finished sieving and the LA is more than 50% finished. We expect it to finish mid-week.

We have had some problems filtering 2,764+. Paul is returning home with a stack of CD's full of relations and will give it a try on his box. His "malloc" seems to work better than mine when it comes to pushing the 2GB limit.

2,797+ is progressing nicely with some select machines participating. The "lines" are MUCH longer than what we have normally been doing.
If you have a machine that sieves 24/7, or if you are willing to wait a number of minutes in order to gracefully stop sieving, I can accept some additional sievers. However, if you stop sieving frequently, or need to do so quickly on anything other than a "rare" basis, I would prefer that you continue to work on our "next" project instead.

We are currently doing sieving on 3,479+. I had e-mailed Sam and requested that he reserve it for us and, from his reply, thought that he had done so. I am presently requesting clarification from him.

If there is a mistake, we will get something else going ASAP.

(Bob: While we get a new project ready, could you use some "help" on something that is already set up and/or running?)

In any case, provided that we are using compatible parameters, I will offer any relations that we have found to the team that is actually going to complete the solution.

Richard

[R.D. Silverman]

Originally Posted by Wacky

I met with Paul in Houston this weekend and can provide some updated information.

2,1466L has finished sieving and the LA is more than 50% finished. We expect it to finish mid-week.

We have had some problems filtering 2,764+. Paul is returning home with a stack of CD's full of relations and will give it a try on his box. His "malloc" seems to work better than mine when it comes to pushing the 2GB limit.

2,797+ is progressing nicely with some select machines participating. The "lines" are MUCH longer than what we have normally been doing.
If you have a machine that sieves 24/7, or if you are willing to wait a number of minutes in order to gracefully stop sieving, I can accept some additional sievers. However, if you stop sieving frequently, or need to do so quickly on anything other than a "rare" basis, I would prefer that you continue to work on our "next" project instead.

We are currently doing sieving on 3,479+. I had e-mailed Sam and requested that he reserve it for us and, from his reply, thought that he had done so. I am presently requesting clarification from him.

If there is a mistake, we will get something else going ASAP.

(Bob: While we get a new project ready, could you use some "help" on something that is already set up and/or running?)

In any case, provided that we are using compatible parameters, I will offer any relations that we have found to the team that is actually going to complete the solution.

Richard

Hi,

I found that when running out of memory during filtering that one
could simply do the filtering in stages, lowering filtmin a little bit
at a time. I start with filtmin equal to the largest prime in the factor
base and mergelevel = 2. I then remove duplicates with mergelevel = 0.
I then do another pass with mergelevel = 2. Then I fully factor all
relations with primes greater than (say) 250K. Then I run another pass
with filtmin set at (say) 10 million. ONLY AFTER THAT do I increase
mergelevel. (and keep decreasing filtmin)

I am just now starting the final filter passes for 2,1406M. The LA should
start tommorow. 11,224+ is sieving and will take me 10 days. I have
a big logistics problem associated with accepting any outside help ....
I can't attach to any outside net and the amount of data is too big for
email. For a longer running project (11,224+ will take ~10 days to sieve)
I could accept independent data sent by snail mail on CD. But that
mechanism is quite clumsy.


3,479+ is a nice choice. And 2,797+ with GNFS should be an interesting
distributed experiment.

I am hoping that sometime in the next year or two we can ALL collaborate
on a kilobit SNFS factorization. But finding machines with enough memory
to run the lattice siever will be problematic. I doubt if it will fit on a 1G
machine.

[Wacky]

Bob,
Thank you for your comments.

Using techniques similar to those that you describe, I am able to reduce the initial set of relations and get to the stage where I need to run a relatively high mergelevel. It is in these last passes that I run into trouble. Once the data has merged a significant number of relations, each set basically falls into EVERY bucket, (at least when there are only 10's of them). This is where I get in trouble. "malloc" fails in trying to increase the bucket size from say 1.5GB to 1.75GB.

As for our current projects, all of the "heavy hitters" are working on 2,797+.
It is "spring break" and we are getting a large number of computer cycles. Next week that rate may well fall off. Because of the run requirements, and the logistics in qualifying the participants, I felt it more appropriate to simply send the "general populace" on to our "next" project where the rest will join them eventually. However, on an individual basis, I am quite willing to have others join in the sieving for 2,797+. WARNING: A single line takes about 15 minutes on one of my machines. If you abort in the middle of a line, quite a bit of work will be lost. If you wish to have a faster disengagement time, just stick to 3,479+. You will have a "head start" on the rest of us when we get there.

I recognize all too well the logistical difficulties in having "others" participate in your effort. I ask if you had some project on which "we" could help because I hate to "waste" computer cycles needlessly. You raised a question as to a possible conflict of efforts on 3,479+. It there is one, I want to move my contributors to another project where their contribution would clearly not be wasted. Meanwhile, we can sort out the potential conflicts and then redirect everyone to the work that they can most usefully do.

Subsequent to your posting, I have confirmed with Sam Wagstaff that he has assigned 3,479+ to our group. I don't understand why you made the statement "George and Paul Zimmermann have the number 'reserved'". If they are doing some ECM effort of that number, then I don't see a particular conflict. ECM might find a factor, and thereby negate the effort that we have, thus far, expended. However, it is also likely that it would not find a factor and any sieving that we have done will simply place us that much closer to an NFS solution. However, if they are working on the number, we need to stay in communication and coordinate our efforts in order to avoid irrational duplication of effort.

I am hoping that sometime in the next year or two we can ALL collaborate on a kilobit SNFS factorization. But finding machines with enough memory to run the lattice siever will be problematic. I doubt if it will fit on a 1G machine.

This is also my goal, and, I believe, Paul's. We have discussed it a number of times.
IMHO, there remain a number of hurdles. You allude to some of them. It is my opinion that the success of such a project will rely on many of us. You will help development efficient sievers. (The current ones do not scale to a problem of this size). I will worry about data coordination problems. Someone else will worry about the MASSIVE problems in doing the resulting LA. And, (here, I may differ from your opinion) we will leverage the contributions of a large number of individuals who don't, at all, understand the details of what we are doing. However, they will participate because they realize that they are contributing to a "good cause".

Richard

[xilman]

Originally Posted by Wacky

I met with Paul in Houston this weekend and can provide some updated information.

2,1466L has finished sieving and the LA is more than 50% finished. We expect it to finish mid-week.

I have much better internet connectivity on this trip than I am accustomed to. In particular, I can contribute to the forum and I can connect back home to my home systems.

The current state of play with the LA is that it has reached 3.84/4.15 of the way through the matrix. I make that 92.5% completed. It should finish late on Wednesday evening. I return home mid-morning on Thursday, airlines permitting. Assuming I'm not totally wiped out by a 9-hour flight and missing a night's sleep, I'll be able load up the relations on a couple of the CD's Richard gave me and then start the square root phase. With luck, factors should appear later on Thursday. It seems pretty likely that factors will appear by the weekend at the latest.

As for 2,764+, other CD's Richard gave me contain the raw relations. My present best guess is that the filtering will take a week or so, with the linear algebra taking somewhere between 2 and 4 weeks. We'll gett factors by early to mid May I guess.


Paul

[xilman]

Originally Posted by Wacky

Subsequent to your posting, I have confirmed with Sam Wagstaff that he has assigned 3,479+ to our group. I don't understand why you made the statement "George and Paul Zimmermann have the number 'reserved'".

I believe I can clarify the position. Each of Paul and George mark a number with the word "reserved" to flag that someone has reserved it for NFS factoring and so neither GIMPS nor ECMNET should perform any more ECM on it. It does not mean that either or both of them has reserved it for factoring themselves.

At least, that has always been the position in the past.


Paul

[R.D. Silverman]

Originally Posted by xilman

I believe I can clarify the position. Each of Paul and George mark a number with the word "reserved" to flag that someone has reserved it for NFS factoring and so neither GIMPS nor ECMNET should perform any more ECM on it. It does not mean that either or both of them has reserved it for factoring themselves.

At least, that has always been the position in the past.


Paul

Actually, I was referring to 2,797+, not 3,479+. However, Paul may
have marked the latter as 'reserved' by now.

Paul is correct with respect to the meaning of the word 'reserved'.

[xilman]

The linear algebra finished about 5 hours ago, apparently successfully.

As the dependencies are on my machine in Cambridge but I and the CDs holding the relations are still in Houston, it will be a day or so before any further progress can be made.


Paul

[Jeff Gilchrist]

Originally Posted by Wacky

I will worry about data coordination problems. Someone else will worry about the MASSIVE problems in doing the resulting LA.

For the time being, I have access to some large systems now including a 128CPU NUMA system with Itanium2 processors, and also a 1000+ CPU Opteron275 cluster running Linux with 8GB of RAM & 4 CPUs on each node using Myrinet interconnect. If any of this can be of use now or later, please let me know. I wouldn't be able to give you access to the machines though so I would have to run any code myself.

[R.D. Silverman]

Originally Posted by Jeff Gilchrist

For the time being, I have access to some large systems now including a 128CPU NUMA system with Itanium2 processors, and also a 1000+ CPU Opteron275 cluster running Linux with 8GB of RAM & 4 CPUs on each node using Myrinet interconnect. If any of this can be of use now or later, please let me know. I wouldn't be able to give you access to the machines though so I would have to run any code myself.

Wow! The things I could do with those kind of resources: