Follow up on LLRnet servers needed

[em99010pepe]

After a few minutes of wise thinking I decided to leave this conversation. I really don't care how many servers will be available when I can manually reserve ranges. I'm hosting 2 NPLB servers, after they dry I am out. I'm reaching a point where I am tired of human stupidity so I prefer to walk away and silently crunch the manual ranges.

Carlos

[gd_barnes]

Originally Posted by em99010pepe

With all these option servers with big range of work when are you guys thinking to fill all gaps, in 2020? The work will be dispersed...

So far we have three servers running lower ranges and one with Top 5000 candidates. Let's dry the first three ones, shall we? Stop talking of new servers, we only have 100 cores at our disposal and not 1000!!! One step at the time, not two...

Carlos

It was my intent with this project to allow people a choice and to fill in the lower range as we searched for top-5000 primes. I'd still like to maintain that. Initially, the higher priority was the higher n-range but as we got out of the danger zone of primes > 333.3K not being in top 5000, it has shifted. In other words, at first 75/25% on higher range...now ~60/40% on lower range.

k=300-400 will be the same way. Higher priority on higher n-range at first for same reason followed by higher priority on the lower n-range to clean it up.

I wouldn't want to dictate to people that they have to search the lower range until we get the 'mess' cleaned up.

The main thing that I'm the happiest about is that people WANT to clean up the lower-range 'mess'. It was a concern of mine when the project started that it would be difficult to get people to search non-top-5000 ranges. THANKS TEAM!!


Gary

[gd_barnes]

I renamed this thread to 'Follow up on LLRnet servers needed' for future historical reference.

[mdettweiler]

Originally Posted by em99010pepe

After a few minutes of wise thinking I decided to leave this conversation. I really don't care how many servers will be available when I can manually reserve ranges. I'm hosting 2 NPLB servers, after they dry I am out. I'm reaching a point where I am tired of human stupidity so I prefer to walk away and silently crunch the manual ranges.

Carlos

Okay, I can understand that. You do have a point, we don't exactly NEED LLRnet servers for every single subproject and drive. Even the big projects such as PSP and SR5 do fine with only one or two servers.

Okay, here's a new idea I'm putting up for discussion:

Maybe here's what we should do: have one LLRnet server for a higher range, one LLRnet server for a lower range, and maybe one server for the n>600K testing later on. (The doublecheck server should probably be counted separately, because it's dealing with a wholly different type of work.) We don't necessarily need a server for each of those ranges on both 300<k<400 and 400<k<1001; instead, maybe we should have those three respective servers doing whichever k-range needs the most work for their designated n-range.

As long as we host our servers on machines such as IronBits' that can handle the stress of a rally without breaking a sweat, then we shouldn't need "backup" servers on any of the ranges.

That would keep logistical issues to a minimum, and avoid spreading our resources too thin, while still giving people plenty of choice as to n-ranges. Remember, as Mini-Geek said, most people won't care whether they're doing 300<k<400 or 400<k<1001, though they will care which n-range they're doing.

What does everyone think?

Anon

[Flatlander]

(There's so much server discussion on this forum so forgive me if this has been said.)

At the beginning of the double check, cores could be returning results every few seconds. Would a server cope and how much time would the client waste with all the comms?
Would reserving work units be better at the beginning?

[gd_barnes]

Originally Posted by Mini-Geek

I think that 300<k<400 and 400<k<1001 are similar enough that they should be put together on the server. Come to think of it, I agree that the doublecheck is different enough that I think it warrants its own server.
Edit: So I now suggest three distinct: Top 5000, first-pass non-Top 5000, and Doublecheck non-Top 5000, with probably 2 servers on the first two distinct groups. You can work out which servers run what.
Part of my reasoning for having these three is the sort of work preference that people would have, while still being easy. After all, would someone that wanted the ease of LLRnet likely care if they got 300<k<400 work or 400<k<1001 work? I think not. Would they care if they were doing top 5000, and if they were running doublechecks compared to first-pass? I think so.
If part of the 300<k<400 is to be put into a manual reserving team drive, I think it should be a separate drive.

Originally Posted by Anonymous

Okay, I can understand that. You do have a point, we don't exactly NEED LLRnet servers for every single subproject and drive. Even the big projects such as PSP and SR5 do fine with only one or two servers.

Okay, here's a new idea I'm putting up for discussion:

Maybe here's what we should do: have one LLRnet server for a higher range, one LLRnet server for a lower range, and maybe one server for the n>600K testing later on. (The doublecheck server should probably be counted separately, because it's dealing with a wholly different type of work.) We don't necessarily need a server for each of those ranges on both 300<k<400 and 400<k<1001; instead, maybe we should have those three respective servers doing whichever k-range needs the most work for their designated n-range.

As long as we host our servers on machines such as IronBits' that can handle the stress of a rally without breaking a sweat, then we shouldn't need "backup" servers on any of the ranges.

That would keep logistical issues to a minimum, and avoid spreading our resources too thin, while still giving people plenty of choice as to n-ranges. Remember, as Mini-Geek said, most people won't care whether they're doing 300<k<400 or 400<k<1001, though they will care which n-range they're doing.

What does everyone think?

Anon

I can go along with these ideas. This is my fault here. It was my ignorance of what constitues a 'large' prime search project when I made the initial suggestion of servers. Heck, my thinking is we are a large prime search project with 100+ cores searching and 50+ top-5000 primes in the first month. But clearly 100 cores does not constitute large in the prime-searching community.

We would want a server specifically on k=300-400 for n>333.3K for a rally the weekend of the 8th. But when n=300-400 'catches up' with n=400-1001, I can merge the big sieve file together and we can all work in harmony on the top-5000 range by itself with a single server for it (with a backup available if necessary). It may catch up with just the rally because it's < 1/6th as many k's. So perhaps temporarily, we could have 2 servers for n>333.3K; one for each k-range, with the idea of merging the range later. Or...as Anon said, just move the high-range server to k=300-400 until it gets caught up and not have one for k=400-1001.

For k=400-1001 n<333.3K, we could just leave one server on that range and when it's done, move the server over to k=300-400 n<333.3K until it is done.

Originally Posted by Flatlander

(There's so much server discussion on this forum so forgive me if this has been said.)

At the beginning of the double check, cores could be returning results every few seconds. Would a server cope and how much time would the client waste with all the comms?
Would reserving work units be better at the beginning?

It hadn't been brought up but I alluded to it when I offered my future server for double-checking (in another thread I think). You may be correct, we may want to do manual LLR reservations for a while. Alternatively, we could just simply ask people to limit the # of cores that they put on it.


Gary

[Mini-Geek]

Originally Posted by Flatlander

and how much time would the client waste with all the comms?
Would reserving work units be better at the beginning?

As long as the client had it set to cache at least, say, five (preferably higher), there wouldn't be any waste, assuming the server could take it. I doubt the server could take it, so I agree with Gary that we should do manual LLRing until it gets to a reasonable amount (I don't know FFT cutoffs for things this small, so I don't know when we could get it to an amount a server could handle), then maybe LLRnet/rally away the rest...or just keep doing it manual, if we prefer that when the time comes.

Originally Posted by gd_barnes

We would want a server specifically on k=300-400 for n>333.3K for a rally the weekend of the 8th. But when n=300-400 'catches up' with n=400-1001, I can merge the big sieve file together and we can all work in harmony on the top-5000 range by itself with a single server for it (with a backup available if necessary).

Would it be very difficult (or otherwise problematic) to sort the k=300-400 range into the knpairs.txt, then host a rally on that server? That way, once it's caught up, it'll start integrating the two ranges automatically and immediately.
I don't know how hard OOo or Excel would choke on a file as large as what we're dealing with, but you could always dump it in (make it .csv then make a space be the divider, or find/replace the space by a comma...this also works for looking at timings in lresults.txt), then sort it that way...Or you could just have it sort where the two ranges overlap, instead of dealing with the entire thing at once.

[gd_barnes]

Originally Posted by Mini-Geek

As long as the client had it set to cache at least, say, five (preferably higher), there wouldn't be any waste, assuming the server could take it. I doubt the server could take it, so I agree with Gary that we should do manual LLRing until it gets to a reasonable amount (I don't know FFT cutoffs for things this small, so I don't know when we could get it to an amount a server could handle), then maybe LLRnet/rally away the rest...or just keep doing it manual, if we prefer that when the time comes.

Would it be very difficult (or otherwise problematic) to sort the k=300-400 range into the knpairs.txt, then host a rally on that server? That way, once it's caught up, it'll start integrating the two ranges automatically and immediately.
I don't know how hard OOo or Excel would choke on a file as large as what we're dealing with, but you could always dump it in (make it .csv then make a space be the divider, or find/replace the space by a comma...this also works for looking at timings in lresults.txt), then sort it that way...Or you could just have it sort where the two ranges overlap, instead of dealing with the entire thing at once.

Actually, that would be no problem at all. I'd just have to use srfile with the -G parameter. It would automatically sort it by n. There would be too many rows to sort it in Excel.

Interesting idea. I like it. The only issue is that before we did that, I think we'd want to run the current LLRnet servers dry so that we aren't sitting there with a bunch of gaps for k=400-1001. I don't think we could finish that up by the 8th because most people are on n<320K right now.

So to incorporate and summarize your idea, here's what I think we'd need to do:
1. Have a temporary 2nd high-range server that exists only for the rally for k=300-400 n>333.3K AND until no more gaps exist in current LLRnet reservations for k=400-1001 n>333.3K.
2. After #1 is done, regardless of what n-range the two k-ranges are at, merge the two together and continue. Since they will be sorted by n, the k-range that has been searched the least will be searched until it is caught up.


Gary

[tnerual]

here are my 2 cents ... (i forgot to read everything above)

in the long term, the project only need 2 server

• first check
• double check

to go to this place, we need to level up all files to join the same level !

first thing to do:
sieve all sieve files to the same level so that we can merge all files to make only one big file.

we don't really need more than 2 server in every case.
we don't need to split the load if server are stable (IB one seems to handle the load)

at the moment, the two stable servers needed are:

- n=>333333 to find top 5000 primes (in range 1<k<1001)
- n=<333333 to complete all the gaps in range 1<k<1001

once all the gaps bellow 333333 are filled, we can switch this server to release double check tests ... (not really important at the moment)

this will limit the numbers of server to a maximum of 2 and the choice will be clear for everybody:
2 choices: top5000 primes or "clear Gary's mess" (this last one changing over time to "delation: mr X didn't check this file perfectly")

i think that's what carlos was trying to explain.

the division in k ranges (0-300 300-400 400+) is not usefull for llring (just complicating the reservation pages.) it's usefull for sieving (sr2sieve can't manage 500+ k numbers) but only for sieving !

we have to equalize everything to have a clear project !


as for the double check server: i will start one here limited to my home farm, it will clear all the low values (fast results hitting a server). once Gary's mess is cleared (every k/n pairs below n= 333333 tested once for every k< 1001) i will release the knpairs.txt file to the official llrnet double check server owner to let everybody crunch it.

but first: CLEAR GARY'S MESS

[kar_bon]

about Ex*el and the row-limits:
you have to think 3-dimensional.
i know E* can only load 65535 rows in one spreadsheet but what about more than one?

i make my VBAscripts to summarize the k/n-pairs for users/teams, ranges and daily stats in such a manner that all spreadsheets named 'ResultsXXX' are summurized. before that i convert the LLRnet pairs in semicolon-seperated text files with the data i need with gawk (easy and quick). import this in E* and call the macro: that's all. and without a DB it's the fastest way i can do. and the most independed one. the most work i got to do is to eliminate/correct 'false' usernames and make the changes in the html-pages. the last could be done by creating a spreadsheet/textfile to write the data i need in html-code. that's my next step to do.