Future direction of NPLB

[Brucifer]

I only mentioned the boinc thing as one option in approaching taking care of problem issues. In actuality there are other issues involved in it too, major ones, like whether or not the project owner would even care to do it, or to even merge into another project, etc. Then as Rytis touched at, there is the large issue of infrastructure which equates to bucks to obtain the hardware, etc., or what it would run moneywise for another project to add to it's overhead. Not to mention what the participants here think about it too. Some people like it other don't. As I mentioned in my previous post... pandora's box, cause that's just exactly what it is.

[gd_barnes]

Originally Posted by Anonymous

Unfortunately with the specific range Gary is talking about above, the k/n pairs take an extremely short time to test, and hence server load will be extremely high. However, yes, you're right, some of the higher parts of the proposed drive would be quite viable to run through LLRnet.

Anon,

I think what he means is that everyone should only put 1-3 cores on it and NO MORE. For n<100K, we wouldn't want more than about 20 cores on a server because the test times are so fast.


Gary

[Brucifer]

Originally Posted by gd_barnes

Anon,

I think what he means is that everyone should only put 1-3 cores on it and NO MORE. For n<100K, we wouldn't want more than about 20 cores on a server because the test times are so fast.


Gary

a place to use the "batching" version of llrnet to help keep the connection load down???

[gd_barnes]

Originally Posted by Rytis

I've been pointed to this thread by em99010pepe, so I'm posting a few thoughts.

* PrimeGrid is running on an Athlon 64 X2 4800+, 4GB RAM, and it barely holds, that's why we're upgrading to a two-server configuration in the nearest future.
* BOINC LLR isn't really suitable for n<200K because of server overhead.

What I can offer is not actual help setting up server (because it simply takes too much time, which I don't have; hundreds if not thousands of hours have been invested into PrimeGrid), but integrating NPLB as a subproject in PrimeGrid (it takes only about an hour for me, because all infrastructure is here). You would provide me LLR work in batches, I'd feed it into BOINC, and then give you results.

I won't rule out BOINC or PrimeGrid in the future but in the mean time, we need to get k=1003-2000 up to AT LEAST n=200K and preferrably into top-5000 territory before enlisting their effort.

What I see BOINC or PrimeGrid as more useful for is much higher n-ranges such as for n>~800K or so. That's what the 321 project did. They only utilized them once they were past n=~4M or so.

Asking for their help now would only complicate things for an n-range that many here will enjoy searching with manual files because so many smaller primes will be found.

Bruce, the initial ranges that we would be searching for k=1003-2000 would not have top-5000 primes. They would, as you mentioned, fill in gaps in ranges not already done BECAUSE they were NOT top-5000 ranges. NPLB is not all about top-5000 primes, it's about finding ALL primes that can be found.

This type of small n-range effort here is a large part of what NPLB is about; leaving no prime behind. A prime being left behind is of different sizes at different k's. Lower k's test faster and hence need to be searched higher than higher k's. Ideally all searched ranges would look approximately like this right now:

k=3 at n=3M (actually at n=5M)
k=5 at n=2.5M (actually at n=1.42M; needs to be searched faster but RPS has it reserved.)
k=7 & 9 at n=2M (k=9 is well behind but is reserved and Benson has 'pilfered' the k; not a pretty sight!)
All k=10-50 at n=1.5M
All k=50-100 at n=1.25M
All k=100-300 at n=1M
All k=300-400 at n=900K
All k=400-1001 at n=800K
All k=1003-1400 at n=600K
All k=1400-2000 at n=500K
All k=2000-3000 at n=400K
etc.

Note that this is just arbitrary but I have a graph of the actual search ranges for all k's with a least-squares fit exponential curve that shows where k's should be at. I haven't updated it in 3 months but it would come in close to the above.

I observed that many of the k=100-300 were much further 'behind' than most anything else vs. where they should be searched hence our project reserved 16 k's that are at only n=600K and will shortly make it a high priority. Our k=300-400 drive is at n>600K on several k's now with a few complete to n=1M so that is closing in on where it should be. Our k=400-1001 drive is only n=~270K behind where it should be. So the new k=100-300 effort, being n=400K behind where it should be was an obvious priority.

In other words, we're trying to 'catch up' the various k-ranges that are the most behind where they should be. That's part of the reason why I'm wanting to get this k=1003-2000 working. But to me the main reason is that k=1003-1400 is simply a flippin' mess!

Back to BOINC and PrimeGrid. I'm not sure about BOINC (don't know enough to speak reasonably about it), but IMHO where PrimeGrid is needed the most is at RPS, ESPECIALLY on k=5! Ever since I started this project, k=5 has just CRAWLED there in relation to what it should be doing. To give you an idea, k=5 for the +1 side is at n=4.5M and RPS is only at n=1.42M on the -1 side!!


Gary

[em99010pepe]

Why not put drive 3 on BOINC? Let's try something like that shall we? We could then concentrate our efforts on the 18 k's drive.

[gd_barnes]

Originally Posted by em99010pepe

Why not put drive 3 on BOINC? Let's try something like that shall we? We could then concentrate our efforts on the 18 k's drive.

See Rytis post above. BOINC/PrimeGrid isn't suited for n<200K due to fast testing time. Our double-check thread is for n=100K-260K. Like I mentioned previously, I think PrimeGrid would be good for testing 100's of k's at higher n-levels (likely n>800K).

On another note, I'm taking this opportunity to do a huge sieve for k=1005-2000 for n=50K-500K. (Anon already did k=1003 to n=500K on a stable machine.) I reached P=3G last night and will take it up to P=10G by Thursday morning on 2 slower cores. Bruce may assist with the sieving effort more at that point.

After getting enough sieving done, we can publicly decide how we'd like to LLR at least the smaller range of n=50K-100K on it. Mainly I just want to have the sieved files available for now.

It'll be nice to clean up part of this messy k-range. I'm all about clean ranges! lol


Gary

[em99010pepe]

I think you are trying to make more than you can handle.
Drive 3 is at n=530k so it is very suitable for BOINC, just my opinion.

[mdettweiler]

Originally Posted by gd_barnes

See Rytis post above. BOINC/PrimeGrid isn't suited for n<200K due to fast testing time. Our double-check thread is for n=100K-260K. Like I mentioned previously, I think PrimeGrid would be good for testing 100's of k's at higher n-levels (likely n>800K).

On another note, I'm taking this opportunity to do a huge sieve for k=1005-2000 for n=50K-500K. (Anon already did k=1003 to n=500K on a stable machine.) I reached P=3G last night and will take it up to P=10G by Thursday morning on 2 slower cores. Bruce may assist with the sieving effort more at that point.

After getting enough sieving done, we can publicly decide how we'd like to LLR at least the smaller range of n=50K-100K on it. Mainly I just want to have the sieved files available for now.

It'll be nice to clean up part of this messy k-range. I'm all about clean ranges! lol


Gary

Gary, I think Carlos was referring to Drive #3, i.e. 300<k<400, not the doublecheck drive, in his post.

Speaking of BOINC, though, Gary and I had discussed future BOINC options a bit via PM, and agreed that at this point, coordinating with PrimeGrid might prove to be too much for a small, relatively new project like NPLB--in fact, the immense participation level that PrimeGrid has would quite likely overwhelm our ability to provide sieved files. However, we also agreed that for NPLB to host its own BOINC server would be a viable option at some point in the future, since this would allow us to grow into a BOINC setup more easily.

With this in mind, I've been playing around with the BOINC server software on a Linux virtual machine. I was able to get it working (for Linux x86 clients at least) with the test app that does little filler tasks (designed so that projects can get the bugs worked out of their systems without having to muck around with a more complex "real" application). Now the trick will be to get LLR set up in it--probably the best place to start would be by asking PrimeGrid to share their BOINC LLR wrapper with us (like they already do with Riesel Sieve, which has been developing the app in parallel to PrimeGrid). Of course, I am a complete newbie at setting up BOINC server software (though it does help that I've used the client before), so I'm pretty much learning as I go. This should prove to be quite interesting.

Anon

[gd_barnes]

Originally Posted by em99010pepe

I think you are trying to make more than you can handle.
Drive 3 is at n=530k so it is very suitable for BOINC, just my opinion.

Oops my bad. Sorry. I get the blasted drives mixed up! lol

Regardless, I think Anon explained our thinking here.

Actually I AM taking more than we can handle in a year's time anyway. I'm looking at the big picture. This is a multi-year project to catch up and clean up all ranges for k<=1001 and n<=1M and k=1003-2000 and n<=500K.

I agree that at some point, perhaps as some of the ranges near n=800K and higher, we may want to enlist PrimeGrid. In the mean time, let's keep rockin' the ranges! The rallies can likely have a profound impact on how quickly we process things.


Gary

[em99010pepe]

Originally Posted by Anonymous

Speaking of BOINC, though, Gary and I had discussed future BOINC options a bit via PM, and agreed that at this point, coordinating with PrimeGrid might prove to be too much for a small, relatively new project like NPLB--in fact, the immense participation level that PrimeGrid has would quite likely overwhelm our ability to provide sieved files. However, we also agreed that for NPLB to host its own BOINC server would be a viable option at some point in the future, since this would allow us to grow into a BOINC setup more easily.

People from PrimeGrid have the ability to choose which project to run so that argument of the sieve file stands down, my thinking is towards the fact that we have another drive started (#4) but the CPU power is the same. So this means we need to get more CPU support and PrimeGrid is there for us. Put Drive #3 on Primegrid and let's concentrate on Drive #4. Imagine from time to time we feed PrimeGrid with some NPLB work. We by ourself can't work on three drives at the same time.

Carlos

[mdettweiler]

Originally Posted by em99010pepe

People from PrimeGrid have the ability to choose which project to run so that argument of the sieve file stands down, my thinking is towards the fact that we have another drive started (#4) but the CPU power is the same. So this means we need to get more CPU support and PrimeGrid is there for us. Put Drive #3 on Primegrid and let's concentrate on Drive #4. Imagine from time to time we feed PrimeGrid with some NPLB work. We by ourself can't work on three drives at the same time.

Carlos

Hmm...yeah, I see what you mean. But, didn't Rytis say something along the lines of us preferably having constant stream of work for PrimeGrid? Or was I reading something into his post that wasn't there? (I didn't see anything along those lines just now when I read through it...)