2^1466L done

[NeoGen]

Another one has reached 100%. This one was much faster than the last one.

What's next on the menu?

[Wacky]

Really finishing 2,1146L ...
Somebody entered incorrect data for the sieving effort parameters. We are really only about half done.

We are considering doing 2,797+.c150 which is what is left thanks to Bruce finding a 55 digit factor by ECM. This is now in the range where GNFS is more efficient that using the original SNFS polynomials.

However, there is one potential problem for our participants.
The typical SNFS project has something like 40M "lines" where each line is perhaps 90M long. This allows us to make assignments where the atomic assignment takes only a fraction of a minute and each client processes hundreds of them in a single task.

But with the GNFS polynomials, the sieving area is not so square.
Here we need perhaps only 250K lines, but each line is 18.9G long.
The difficulty is that the atomic assignment will now require perhaps 10 minutes, and we have no reasonable way to provide a finer granularity.

For many of the sievers, this is no problem. They sit there and sieve 24/7 at the lowest priority. So, they don't even notice that the OS may have suspended them while something of higher priority was run.

However, we also have another class of participants who frequently start and stop the siever in order to reboot or otherwise use the machine resources. Those users are often willing to wait a few seconds for the siever to stop gracefully. But asking them to wait a number of minutes would not be appropriate.

The best suggestion that I have at the moment is to sieve two projects simultaneously. Those who can process the longer assignments could work on 2,797+ while the others begin another SNFS project.

Comments?

Richard

[akruppa]

It should be possible to use reciprocal polynomials of opposite skewness, so a line would be only 250k long. This may hurt performance due to very frequent line changes, though.

Alex

[NeoGen]

Originally Posted by Wacky

Really finishing 2,1146L ...
Somebody entered incorrect data for the sieving effort parameters. We are really only about half done.

Can it be corrected? If not, it's no problem, but it's the first time I'll see it going to 200%.

[Jushi]

Originally Posted by Wacky

However, we also have another class of participants who frequently start and stop the siever in order to reboot or otherwise use the machine resources. Those users are often willing to wait a few seconds for the siever to stop gracefully. But asking them to wait a number of minutes would not be appropriate.

How about programming the nfsnetclient to immediatly stop sieving when the user aks to, losing the current line? This way sieving one line could take several minutes, but the program will still stop immediately.

I have to admit I don't know the internals of the code, but I'm sure this is not hard to implement.

[Wacky]

Originally Posted by Jushi

How about programming the nfsnetclient to immediatly stop sieving when the user asks to, losing the current line? This way sieving one line could take several minutes, but the program will still stop immediately.

I have to admit I don't know the internals of the code, but I'm sure this is not hard to implement.

It is more difficult than you think. Because of I/O buffering and "GUI" event loops, you cannot simply "abort" a running thread. In most (OS) cases, you end up having to "poll" to determine if the program should stop. The cost of polling is not insignificant. Therefore, you do not want do do it too often. Finding an appropriate location within the nested loops my not be easy to do.
If it is too low, not only do you waste the time polling, but you may also invalidate the caches, adding even more overhead. On the other hand, if the polling is done too far out, it may not be frequent enough to give satisfactory response. Considering that the length of a "line" may vary by a couple orders of magnitude, finding the right balance is not trivial.

[Jushi]

Originally Posted by Wacky

It is more difficult than you think. Because of I/O buffering and "GUI" event loops, you cannot simply "abort" a running thread. In most (OS) cases, you end up having to "poll" to determine if the program should stop. The cost of polling is not insignificant. Therefore, you do not want do do it too often. Finding an appropriate location within the nested loops my not be easy to do.
If it is too low, not only do you waste the time polling, but you may also invalidate the caches, adding even more overhead. On the other hand, if the polling is done too far out, it may not be frequent enough to give satisfactory response. Considering that the length of a "line" may vary by a couple orders of magnitude, finding the right balance is not trivial.

I don't know about Windows, but in Unices all this can be done quite easily using signals. You could have one process (nfsnetclient) doing the polling for stop.txt and sending a signal to the linesiever to stop it. The linesiever can then catch this signal to gracefully quit, but maybe this isn't even necessary. Maybe you can simply "abort" a running linesiever.

I know that Linux even has a mechanism using signals to notify when a new file is created in a directory, so then you don't even need to poll for stop.txt (this is probably Linux-specific).

[jasonp]

Originally Posted by Wacky

The best suggestion that I have at the moment is to sieve two projects simultaneously. Those who can process the longer assignments could work on 2,797+ while the others begin another SNFS project.

What about assigning 'panels', where each client gets maybe 100 lines but only in a specific range?

jasonp

[R.D. Silverman]

Originally Posted by jasonp

What about assigning 'panels', where each client gets maybe 100 lines but only in a specific range?

jasonp

This number is small enough so that NFSNET should not be too concerned
about an optimal (or even near-optimal) skewness. Just reduce the line
length and accept some loss in efficiency.

[Wacky]

Originally Posted by jasonp

What about assigning 'panels', where each client gets maybe 100 lines but only in a specific range?

I have already planned to do that. However, it is logistically difficult to gain even a full order-of-magnitude "improvement" by that technique.

I'll set up a separate server pool and invite those who do not feel that they would be adversely affected by the longer line calculation times to switch. Meanwhile, the rest of the participants can continue sieving as usual.