Sieving drive for k=301-400 n=1M-2M - Page 3

gd_barnes · 2010-02-20, 19:44

Well, after looking at what Mike (VBCurtis) has suggested, I'm still attempting to come to a decision as to what would be the best sieve depth to stop at. I've done the calculations he suggested and if all you are looking at is total project length, it makes the most sense to sieve the entire thing to the same depth. Unfortunately that depth is quite a lot deeper than where we are at. But...total project length (calulated statically; i.e. with just today's software/hardware) is not the only consideration here.

For the time being, it seems that we should sieve to P>120T but I don't want this to drag on past mid March because I feel this effort will generate quite a lot of interest and such a sieve depth is much deeper than most. As an example, if a person were to reserve a single k=300-400 himself and sieve just that k for n=1M-2M to its optimum depth by itself, that depth would likely not be nearly as deep except perhaps for the 2 or 3 very highest weight k's in the group. The point being that even at P=120T, the sieve files generated for the individual k's for this effort will be more effective for the average individual person to do searching on than they would had the person did the effort stand alone. Also, the searching for this effort will likely take 3 to 5 calendar years and computer hardware and software over that time will improve markedly over that time.

Based on this new info. coming to light; as of Monday, I'm going to put 3 quads on this effort, which should roll thru about P=~1T per day. So if I left them on for 23 days thru March 15th, that would put us near P=135T at that point. That would be sufficient but it wouldn't hurt if we could go a little further by then.

If anyone else can spare a few cores here, that would be great. Regardless, we will definitely be looking to begin the new drive by about March 20th.

Gary

VBCurtis · 2010-02-22, 08:38

Gary-
Thanks for reporting your findings. Like you, I question the use of the conclusions when running a truly massive sieve- 120T is indeed deeper than any individual (except me) is likely to achieve on his own. By that logic, the efficiency of the sieve has been achieved- candidates are sieved deeper than they would be by individuals.

As for your argument that future computer speeds justify stopping early: The question to ask is what task makes THAT machine's contribution to the project most effective? The fact Carlos owns a core i5 has absolutely no effect on what my P4 (or P3!) should be doing to contribute to a project. He may get any task done in 1/4 the time- so what? Similar thinking suggests that when you reach some satisfactory sieve depth to start LLR, you dedicate only the best sieve-machines to sieve remaining work. Athlon64s with 64bit OSes are the best sievers relative to their speed at LLR, and should thus never do LLR work in a managed army of multiple architectures. Perhaps there are some A64s in your group that could sieve after the project starts LLR? Software has not improved for LLR in many years- that code has been hand-tuned in assembly. Sieve code also is no longer getting more efficient- and if it did, it would serve to increase the optimal sieve depth, anyway.

Finally, I understand your interest in getting the project started due to demand. But with the calculations you have, you should break off only parts as-needed; perhaps 1.0-1.05M for March, and more blocks as demand requires. That way the sieve is as effective as possible at shortening the length of the entire project- assuming you plan to sieve the file deeper.

k=5 is done sieveing- Peter tested it to 3.05M already. I am sieving 11-13-31-45-99 on 2.1 to 4M at present, at 412T and counting. I'm surprised to find optimal sieve depth in excess of 500T for that sieve! Thanks for your suggestions.
-Curtis

mdettweiler · 2010-02-22, 15:32

Quote:

Originally Posted by VBCurtis

Software has not improved for LLR in many years- that code has been hand-tuned in assembly.

Not to nitpick, though the recently-released LLR 3.8.0 does increase testing times on base 2 by about 6-10%...

gd_barnes · 2010-02-22, 18:17

That's not my experience with sr2sieve. Sr2sieve has nearly tripled in speed in 3 years (with 64 vs. 32 bit and other efficiency gains) and computer speeds have likely close to doubled in that period of time and certainly increased their capacity. Even if LLR has only added 10% or whatever to its speed in the last 3 years or so, that's not the only effect. If sieving software continues to gain against primality searching software, then stopping short of the true mathematical optimal sieve depth today assuming a static scenario makes sense. The idea being to continue sieving further later when resources are cheaper per sieved candidate.

Another question to ask one's self is: Am I holding up the starting of an effort where there is a large amount of demand that would be dedicated towards the project that is currently being used elsewhere? I think we debated about that in the k=5 thread at one point at RPS. Like there, I know for a fact that we have 2 major searchers that will dedicate a large amount of cores towards this effort once we are ready to go but they currently have only a few cores on our drives that are ranging from n=670K to 750K right now.

From what I can tell, most of our searchers have machines that are either better primality testers (i.e. 32 bit machines) or machines that are equal at sieving and primality testing.

Breaking off small incremental pieces like n=1M-1.05M and continuing to sieve the rest at the same time would be a logical step if this drive would be like all of our regular drives. But this drive will be effectively a continuation of our individual-k drive where people will be able to reserve individual k's up to as high as n=1M-2M. That said, it's not a bad idea to have an ongoing sieve going after the drive has begun. So I think the best thing to do is to go ahead and leave sieving open while starting the drive in case there are some people who like sieving. We've somewhat avoided that in the past since we like to match up sieved k/n pairs with results and it's a little messy having different k's sieved to different depths but I think it is something we can manage here.

What I'm going to do since I know we're in the reasonable range of the optimum sieve depth at this point is to see what we can sieve up thru ~March 15th and start the drive on ~March 20th. 3 quads from me starting late today or early Tues. would put us at P=~135T by then.

Gary

mdettweiler · 2010-02-22, 19:17

Quote:

Originally Posted by gd_barnes

So I think the best thing to do is to go ahead and leave sieving open while starting the drive in case there are some people who like sieving. We've somewhat avoided that in the past since we like to match up sieved k/n pairs with results and it's a little messy having different k's sieved to different depths but I think it is something we can manage here.

Just a comment: if we do this, then I'd suggest that we keep track of which sieve depths were used for which ranges in a separate column of the reservations table, to avoid confusion.

gd_barnes · 2010-02-22, 20:25

Quote:

Originally Posted by mdettweiler

Just a comment: if we do this, then I'd suggest that we keep track of which sieve depths were used for which ranges in a separate column of the reservations table, to avoid confusion.

Completely agreed.

vaughan · 2010-02-23, 22:29

I'd like to reserve 112T to 113T.

gd_barnes · 2010-02-24, 10:26

Time to get busy: Reserving P=113T-125T.

vaughan · 2010-02-25, 01:39

What is wrong?

When I try to start SR2sieve.exe I see a DOS box briefly then it disapperas.

The log file shows this:

02/25/10 12:30:21 ERROR: Failed to open input file `sr2data.txt'.

My run.txt file is:

sr2sieve -p 112e12 -P 112500e9 -zz -i sieve300-400-1M-2M.txt

kar_bon · 2010-02-25, 01:48

Quote:

Originally Posted by vaughan

What is wrong?

When I try to start SR2sieve.exe I see a DOS box briefly then it disapperas.

The log file shows this:

02/25/10 12:30:21 ERROR: Failed to open input file `sr2data.txt'.

My run.txt file is:

sr2sieve -p 112e12 -P 112500e9 -zz -i sieve300-400-1M-2M.txt

sr2sieve needs an abcd-format file as input. command like

Code:

sr2sieve -v -f factors.txt -P 100000000000 -i sr_2.abcd

convert it with

Code:

srfile -a sieve300-400-1M-2M.txt

Mini-Geek · 2010-02-25, 02:06

Quote:

Originally Posted by vaughan

02/25/10 12:30:21 ERROR: Failed to open input file `sr2data.txt'.

My run.txt file is:

sr2sieve -p 112e12 -P 112500e9 -zz -i sieve300-400-1M-2M.txt

What do you intend run.txt to do? It looks to me like you've made that file but are doing nothing with it...are you intending it to do this?

Quote:

If no command line arguments are given but `sr2sieve-command-line.txt'
exists in the current directory, then the command line will be as if the
first line of this file had been used to invoke sr2sieve. This may be useful
on some GUI machines where the command shell and batch files have been
disabled for security reasons.

If so, it needs to be named "sr2sieve-command-line.txt", not "run.txt". Alternately, you can rename run.txt to run.bat and double click it (instead of sr2sieve.exe) to begin sieving.

This is separate from the issue of the input file's format. If sr2sieve has a problem with it (once it's looking at the right file), it should tell you and you can convert the sieve file to the ABCD format.

2010-02-20, 19:44	#23
gd_barnes May 2007 Kansas; USA 10100010011011₂ Posts	Well, after looking at what Mike (VBCurtis) has suggested, I'm still attempting to come to a decision as to what would be the best sieve depth to stop at. I've done the calculations he suggested and if all you are looking at is total project length, it makes the most sense to sieve the entire thing to the same depth. Unfortunately that depth is quite a lot deeper than where we are at. But...total project length (calulated statically; i.e. with just today's software/hardware) is not the only consideration here. For the time being, it seems that we should sieve to P>120T but I don't want this to drag on past mid March because I feel this effort will generate quite a lot of interest and such a sieve depth is much deeper than most. As an example, if a person were to reserve a single k=300-400 himself and sieve just that k for n=1M-2M to its optimum depth by itself, that depth would likely not be nearly as deep except perhaps for the 2 or 3 very highest weight k's in the group. The point being that even at P=120T, the sieve files generated for the individual k's for this effort will be more effective for the average individual person to do searching on than they would had the person did the effort stand alone. Also, the searching for this effort will likely take 3 to 5 calendar years and computer hardware and software over that time will improve markedly over that time. Based on this new info. coming to light; as of Monday, I'm going to put 3 quads on this effort, which should roll thru about P=~1T per day. So if I left them on for 23 days thru March 15th, that would put us near P=135T at that point. That would be sufficient but it wouldn't hurt if we could go a little further by then. If anyone else can spare a few cores here, that would be great. Regardless, we will definitely be looking to begin the new drive by about March 20th. Gary Last fiddled with by gd_barnes on 2010-02-20 at 19:45

2010-02-22, 08:38	#24
VBCurtis "Curtis" Feb 2005 Riverside, CA 11375₈ Posts	Gary- Thanks for reporting your findings. Like you, I question the use of the conclusions when running a truly massive sieve- 120T is indeed deeper than any individual (except me) is likely to achieve on his own. By that logic, the efficiency of the sieve has been achieved- candidates are sieved deeper than they would be by individuals. As for your argument that future computer speeds justify stopping early: The question to ask is what task makes THAT machine's contribution to the project most effective? The fact Carlos owns a core i5 has absolutely no effect on what my P4 (or P3!) should be doing to contribute to a project. He may get any task done in 1/4 the time- so what? Similar thinking suggests that when you reach some satisfactory sieve depth to start LLR, you dedicate only the best sieve-machines to sieve remaining work. Athlon64s with 64bit OSes are the best sievers relative to their speed at LLR, and should thus never do LLR work in a managed army of multiple architectures. Perhaps there are some A64s in your group that could sieve after the project starts LLR? Software has not improved for LLR in many years- that code has been hand-tuned in assembly. Sieve code also is no longer getting more efficient- and if it did, it would serve to increase the optimal sieve depth, anyway. Finally, I understand your interest in getting the project started due to demand. But with the calculations you have, you should break off only parts as-needed; perhaps 1.0-1.05M for March, and more blocks as demand requires. That way the sieve is as effective as possible at shortening the length of the entire project- assuming you plan to sieve the file deeper. k=5 is done sieveing- Peter tested it to 3.05M already. I am sieving 11-13-31-45-99 on 2.1 to 4M at present, at 412T and counting. I'm surprised to find optimal sieve depth in excess of 500T for that sieve! Thanks for your suggestions. -Curtis Last fiddled with by VBCurtis on 2010-02-22 at 08:42

2010-02-22, 18:17	#26
gd_barnes May 2007 Kansas; USA 289B₁₆ Posts	That's not my experience with sr2sieve. Sr2sieve has nearly tripled in speed in 3 years (with 64 vs. 32 bit and other efficiency gains) and computer speeds have likely close to doubled in that period of time and certainly increased their capacity. Even if LLR has only added 10% or whatever to its speed in the last 3 years or so, that's not the only effect. If sieving software continues to gain against primality searching software, then stopping short of the true mathematical optimal sieve depth today assuming a static scenario makes sense. The idea being to continue sieving further later when resources are cheaper per sieved candidate. Another question to ask one's self is: Am I holding up the starting of an effort where there is a large amount of demand that would be dedicated towards the project that is currently being used elsewhere? I think we debated about that in the k=5 thread at one point at RPS. Like there, I know for a fact that we have 2 major searchers that will dedicate a large amount of cores towards this effort once we are ready to go but they currently have only a few cores on our drives that are ranging from n=670K to 750K right now. From what I can tell, most of our searchers have machines that are either better primality testers (i.e. 32 bit machines) or machines that are equal at sieving and primality testing. Breaking off small incremental pieces like n=1M-1.05M and continuing to sieve the rest at the same time would be a logical step if this drive would be like all of our regular drives. But this drive will be effectively a continuation of our individual-k drive where people will be able to reserve individual k's up to as high as n=1M-2M. That said, it's not a bad idea to have an ongoing sieve going after the drive has begun. So I think the best thing to do is to go ahead and leave sieving open while starting the drive in case there are some people who like sieving. We've somewhat avoided that in the past since we like to match up sieved k/n pairs with results and it's a little messy having different k's sieved to different depths but I think it is something we can manage here. What I'm going to do since I know we're in the reasonable range of the optimum sieve depth at this point is to see what we can sieve up thru ~March 15th and start the drive on ~March 20th. 3 quads from me starting late today or early Tues. would put us at P=~135T by then. Gary Last fiddled with by gd_barnes on 2010-02-22 at 18:23

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
GPU sieving drive part III: k<10000 n=3M-6M	mdettweiler	No Prime Left Behind	19	2011-02-17 21:13
GPU sieving drive part II: k<10000 n=2M-3M	mdettweiler	No Prime Left Behind	44	2010-11-28 10:59
Bigger and better GPU sieving drive: Discussion	henryzz	No Prime Left Behind	75	2010-10-31 16:51
GPU sieving drive for k<=1001 n=1M-2M	mdettweiler	No Prime Left Behind	11	2010-10-04 22:45
Sieving drive for k=2000-3400 n=50K-1M	gd_barnes	No Prime Left Behind	145	2009-06-23 18:28

2010-02-23, 22:29	#29
vaughan Jan 2005 Sydney, Australia 5·67 Posts	I'd like to reserve 112T to 113T.

2010-02-24, 10:26	#30
gd_barnes May 2007 Kansas; USA 289B₁₆ Posts	Time to get busy: Reserving P=113T-125T.

2010-02-25, 01:39	#31
vaughan Jan 2005 Sydney, Australia 101001111₂ Posts	What is wrong? When I try to start SR2sieve.exe I see a DOS box briefly then it disapperas. The log file shows this: 02/25/10 12:30:21 ERROR: Failed to open input file `sr2data.txt'. My run.txt file is: sr2sieve -p 112e12 -P 112500e9 -zz -i sieve300-400-1M-2M.txt