Follow up on LLRnet servers needed

[em99010pepe]

I'm tired of all this discussion about the new servers. We all are getting too excited. Let's make a pause and think if we are going through the right way.
I personally think we should only have less servers to handle and for now we should concentrate in cleaning the lower ranges. After this we can easily feed the servers with some new work instead of having new servers available or set up new servers. Don't forget that we are still small in size, in CPU power. I suppose you guys notice but the number of servers increased but not the people helping, we only jump from server to server to help, that's a nonsense!!! I'm not defending that my servers should be active, I rather have them off, I just think we should think wise...and clean Gary's mess!

Thanks,

Carlos

[Beyond]

This is getting a bit silly and creating more work for those handing out the files and gathering the results for the stats pages. All the lower range stuff should be on one server, the Top 5000 work on another, possibly a back-up server for each.

[kar_bon]

the same thoughts by myself.
i have to look which server got which k/n-pairs and what port and are they still working.
Carlos sent me right now results for lower ranges (n~ 280k-290k) to clear more ranges and compare with the original testfiles.
i wrote some scripts to handle this as easy as can be for me without a DB behind this all.
for example: only one prime found and i have to maintain 4 pages to be uptodated:
Prime, TeamDrive, Statistics and the 300<k<2000-page!
so i think 2 servers for the current work are enough to handle:
- lowrange n<333k
- Top5000 range n>333k

in sight of a new race it seems nessecary for another (or two) other servers.
but if IronBits server is ok this is the one for the race.

so my request to all: look at your workfiles, process them and then choose one or two servers only for LLRnetting!
and try not to grab too many pairs at once!
many ranges are incomplete and i have to wait for verifying them!

[Mini-Geek]

Gary's suggestions for servers (8 servers):
2 for 400<k<=1001 for n=333.3K-600K
1 for 400<k<=1001 for n=260K-320K
2 for 300<k<400 for n=333.3K-600K in early March (I would like to do the next rally on this one; after the rally 1 server may be sufficient)
1 for 300<k<400 for n=260K-333.3K in early March (smaller # of k's and n-range; shouldn't take too long)
1 for 300<k<400 for n=600K-1M in late March (will only be the few k's that have already been searched to n=600K)
1 for 300<k<400 for n=100K-260K double check

My suggestion for servers (4 servers, 6 if needed eventually):
2 for 400<k<=1001 for n=333.3K-600K ("set 1")
2 for 400<k<=1001 for n=260K-320K ("set 2")
add 300<k<400 for n=333.3K-600K in early March to set 1
add 300<k<400 for n=260K-333.3K in early March to set 2
add 300<k<400 for n=600K-1M in late March to set 1
add 300<k<400 for n=100K-260K double check to set 2

Yeah, it's pretty ridiculous to have so many servers. I think we should have 2 distinct, above and below Top 5000, 2 servers per range, so 4 (as others suggested but more specific, not just "backups" but multiple to split the load). As new sub-ranges, like 300<k<400 become ready to LLR, we just add them to the servers already here, based on whether it's Top 5000 or not.
The ports for the two servers in a set should be obviously related (e.g. port 300 and 301). If necessary as NPLB grows, we can add a third server to each range (to e.g. port 302), but I think it should be kept as simple as possible.
Or is there some way to have it be only one port people know and enter, but distribute the work load across multiple boxes? Something like how DNS has multiple IPs that it hands out in order, but obviously not exactly the same because then you'd return a result to the wrong server randomly.

Off-topic: Wow, we'll have 300<k<1001 100K-1M by the time we finish all these ranges planned.

[mdettweiler]

Originally Posted by Mini-Geek

...

My suggestion for servers (4 servers, 6 if needed eventually):
2 for 400<k<=1001 for n=333.3K-600K ("set 1")
2 for 400<k<=1001 for n=260K-320K ("set 2")
add 300<k<400 for n=333.3K-600K in early March to set 1
add 300<k<400 for n=260K-333.3K in early March to set 2
add 300<k<400 for n=600K-1M in late March to set 1
add 300<k<400 for n=100K-260K double check to set 2

Yeah, it's pretty ridiculous to have so many servers. I think we should have 2 distinct, above and below Top 5000, 2 servers per range, so 4 (as others suggested but more specific, not just "backups" but multiple to split the load). As new sub-ranges, like 300<k<400 become ready to LLR, we just add them to the servers already here, based on whether it's Top 5000 or not.
The ports for the two servers in a set should be obviously related (e.g. port 300 and 301). If necessary as NPLB grows, we can add a third server to each range (to e.g. port 302), but I think it should be kept as simple as possible.
Or is there some way to have it be only one port people know and enter, but distribute the work load across multiple boxes? Something like how DNS has multiple IPs that it hands out in order, but obviously not exactly the same because then you'd return a result to the wrong server randomly.

Off-topic: Wow, we'll have 300<k<1001 100K-1M by the time we finish all these ranges planned.

I'm kind of inclined to agree with Mini-Geek's suggestion. 8 servers is a bit much, both in terms of how much power we have to put on them and logistically.

I propose a slight variation on Mini-Geek's suggestion:

2 servers on 400<k<1001, n=370K-600K (i.e. 1st Drive). Carlos has already got one running; now we just need to find one more. Maybe AES can use his server for this?

2 servers on 400<k<1001, n=260K-320K (i.e. 2nd Drive). We've got three servers on this right now; we may as well leave them on it until this range is cleaned up, because between those three servers and the individually reserved ranges from the 2nd Drive, we've got all of its remaining ranges reserved, and it shouldn't take long to clean them all up.

1 server on 300<k<400, n=333K-600K. IronBits can host this, since his machine is sufficiently powerful so that it should be able to handle the pounding of a full rally by itself.

1 server on 300<k<400, n=260K-333K. How about Gary's upcoming server be put on this?

1 server on 3<k<1001, 100K<n<260K (doublecheck). I'll make a (tentative) offer to host this one; I've got a server that I might be able to get running 24/7 in the near future. Be forewarned, however, that this is a pretty "puny" server, with only 128MB of RAM, so if we do use it for the doublechecks, try not to hit it too hard.

Anyway, just my $0.02.

Anon

[gd_barnes]

OK, points taken guys. Sorry to create the confusion. Keep in mind that I was asking for opinions when I posted that and mentioned that it was posted off of the cuff so to speak. In restrospect, I should have gotten everyone's opinions first before making my own 'suggestions'. Also, we had some server issues on the last rally leading me to believe that we needed 2 servers on some ranges.

I will have k=300-400 for n=260K-600K available to post tomorrow. So here is what we will have:

k=300-400 tomorrow:
n=260K-333.3K
n=333.3K-600K

k=400-1001 now:
n=260K-333.3K
n=333.3K-600K

k=3-1001 for n=100K-260K double-checking in a few weeks.

So I will ask first this time: How does everyone think we should work the server situation on the above? The basic assumption being that we'll do a rally the weekend of the 8th on k=300-400 for n=333.3K-600K.

Is one server for each of the above the way to go? If so, that would be 4 now, 5 for a short while, and then down to 3 after n=260K-333.3K on both k-ranges is cleaned out.

I can set up a poll on this if you'd like. We'll just need to come up with some reasonable choices to choose from.

Edit: I'm also inclined to agree with Mini-Geek's suggestion. Nicely done.


Gary

[mdettweiler]

Originally Posted by gd_barnes

OK, points taken guys. Sorry to create the confusion. Keep in mind that I was asking for opinions when I posted that and mentioned that it was posted off of the cuff so to speak. In restrospect, I should have gotten everyone's opinions first before making my own 'suggestions'. Also, we had some server issues on the last rally leading me to believe that we needed 2 servers on some ranges.

I will have k=300-400 for n=260K-600K available to post tomorrow. So here is what we will have:

k=300-400 tomorrow:
n=260K-333.3K
n=333.3K-600K

k=400-1001 now:
n=260K-333.3K
n=333.3K-600K

k=3-1001 for n=100K-260K double-checking in a few weeks.

So I will ask first this time: How does everyone think we should work the server situation on the above? The basic assumption being that we'll do a rally the weekend of the 8th on k=300-400 for n=333.3K-600K.

Is one server for each of the above the way to go? If so, that would be 4 now, 5 for a short while, and then down to 3 after n=260K-333.3K on both k-ranges is cleaned out.

I can set up a poll on this if you'd like. We'll just need to come up with some reasonable choices to choose from.

Edit: I'm also inclined to agree with Mini-Geek's suggestion. Nicely done.


Gary

I suggest we leave Carlos's port 100 and IronBits's port 500 on the 400<k<1001, n=260K-333.3K--since they've both got big ranges reserved in that range, and shouldn't take long to clean it up.

We can leave Carlos's port 300 server and now AES's port 300 server on 400<k<1001, n=333.3K-600K.

Then, IronBits can start up a server for 300<k<400, n=333.3K-600K tomorrow when that range becomes available.

IronBits can start up another server, then, for 300<k<400, n=260K-333.3K. Then the 3 servers IronBits has offered to host will be maxed out (until his port 500 server on the 2nd Drive has run dry, that is).

As for the doublechecking--again, I might be able to host that, but Gary, if you'd rather run it on the new server you and I are trying to set up, I'm all for it. (In fact, that would probably be better than me hosting it, because if I hosted it, we'd have to wait until I can get my server set up to run 24/7 before we could start it.)

We can then tackle n>600K when that comes.

Anon

[Mini-Geek]

Originally Posted by gd_barnes

OK, points taken guys. Sorry to create the confusion. Keep in mind that I was asking for opinions when I posted that and mentioned that it was posted off of the cuff so to speak. In restrospect, I should have gotten everyone's opinions first before making my own 'suggestions'. Also, we had some server issues on the last rally leading me to believe that we needed 2 servers on some ranges.

I will have k=300-400 for n=260K-600K available to post tomorrow. So here is what we will have:

k=300-400 tomorrow:
n=260K-333.3K
n=333.3K-600K

k=400-1001 now:
n=260K-333.3K
n=333.3K-600K

k=3-1001 for n=100K-260K double-checking in a few weeks.

So I will ask first this time: How does everyone think we should work the server situation on the above? The basic assumption being that we'll do a rally the weekend of the 8th on k=300-400 for n=333.3K-600K.

Is one server for each of the above the way to go? If so, that would be 4 now, 5 for a short while, and then down to 3 after n=260K-333.3K on both k-ranges is cleaned out.

I can set up a poll on this if you'd like. We'll just need to come up with some reasonable choices to choose from.

Edit: I'm also inclined to agree with Mini-Geek's suggestion. Nicely done.


Gary

I think that 300<k<400 and 400<k<1001 are similar enough that they should be put together on the server. Come to think of it, I agree that the doublecheck is different enough that I think it warrants its own server.
Edit: So I now suggest three distinct: Top 5000, first-pass non-Top 5000, and Doublecheck non-Top 5000, with probably 2 servers on the first two distinct groups. You can work out which servers run what.
Part of my reasoning for having these three is the sort of work preference that people would have, while still being easy. After all, would someone that wanted the ease of LLRnet likely care if they got 300<k<400 work or 400<k<1001 work? I think not. Would they care if they were doing top 5000, and if they were running doublechecks compared to first-pass? I think so.
If part of the 300<k<400 is to be put into a manual reserving team drive, I think it should be a separate drive.

[gd_barnes]

Originally Posted by Anonymous

I suggest we leave Carlos's port 100 and IronBits's port 500 on the 400<k<1001, n=260K-333.3K--since they've both got big ranges reserved in that range, and shouldn't take long to clean it up.

We can leave Carlos's port 300 server and now AES's port 300 server on 400<k<1001, n=333.3K-600K.

Then, IronBits can start up a server for 300<k<400, n=333.3K-600K tomorrow when that range becomes available.

IronBits can start up another server, then, for 300<k<400, n=260K-333.3K. Then the 3 servers IronBits has offered to host will be maxed out (until his port 500 server on the 2nd Drive has run dry, that is).

As for the doublechecking--again, I might be able to host that, but Gary, if you'd rather run it on the new server you and I are trying to set up, I'm all for it. (In fact, that would probably be better than me hosting it, because if I hosted it, we'd have to wait until I can get my server set up to run 24/7 before we could start it.)

We can then tackle n>600K when that comes.

Anon

This sounds good to me and is close to what Mini-Geek suggested. Anon, yes, I can run the double-checking server after we get the issues worked out on it. One possible issue: My machine has 1 GB RAM and the low n-ranges for double checking will mean plenty of server hits. We'll just have to not have Beyond's ~70 machines on it! Running his ~70 machines on any server at such a low n-range would strain just about any server! Actually, it would be a waste of such large resources that could be better used on finishing n=260K-333.3K or top-5000 ranges or more preferrably in the future on n>600K ranges. (Although I would welcome 5-10 of his machines on the double-check effort in the future.)

What does everyone think of Anon's suggestion above? We would end up with 2 servers on k=400-1001 and n>333.3K. Karsten and Carlos, is that managable?


Gary

[em99010pepe]

With all these option servers with big range of work when are you guys thinking to fill all gaps, in 2020? The work will be dispersed...

So far we have three servers running lower ranges and one with Top 5000 candidates. Let's dry the first three ones, shall we? Stop talking of new servers, we only have 100 cores at our disposal and not 1000!!! One step at the time, not two...

Carlos

[gd_barnes]

I just now reread Mini-Geek's suggestion. There was a lot mentioned and I only had time to skim it the first time. If you're like me and didn't read the specific details of it the first time, then you might consider rereading it.

This looks like an excellent concept! Ports 300/301/302 etc. for "set one" on one n-range and ports 500/501/502 etc. for "set two" on another n-range.

Although we won't get this set up immediately, I'm thinking that it makes sense to work towards this concept. It will make it easier on Carlos/Karsten/Anon/me on accounting for results in the future.