PRPnet

[mdettweiler]

Originally Posted by gd_barnes

Outstanding!! Thanks for the great tip Tim.

Max, can you please make this change to the various PRPnet servers, recompile them, stop them, and restart them. Perhaps you can call the changed binary "prpservern", which indicates that the prpserver.candidates file would be sorted by n. The sorting of the candidates file has always annoyed me greatly. You can't quickly tell what candidates are handed out to people. You also can't quickly tell what the "true" lowest and highest n-value that are remaining.

You'd also have to do this with each new version of the PRPnet server that comes out unless you can talk Mark into making it the default sorting method in the code. For that reason, I would not suggest putting the version # in the name of the changed binary. We'd have too many of them very quickly.

Another thing, can we delete all of the previous temporary prpserver244 binaries from the various prpnet folders? I also see that one of them has a prpserver230, which I assume can be deleted. That way I'll always know to restart them with the prpserver binary as needed.


Thanks,
Gary

First of all, ditto, thanks for the tip Tim. I'd come across this once before when I was looking for a way to make the PRPnet server hand out k/n pairs by k instead of by decimal size (which usually means by n). To do that, you need to make the reverse change to another line which controls the hand-out order. That was a while back and I haven't tried compiling the tweak into any recent versions. But, yes, good idea. Since this version of PRPnet will hopefully stick around for a while before it needs replacing, it shouldn't be too hard to compile a tweaked version and stick with it. I'll do it as soon as I get the chance.

@Gary about the various prpserver binaries: I've gotten in the habit of keeping around variou older "milestone" versions as backups, primarily for use in case something goes majorly wrong with a new version that I'm testing. Now that 2.4.6 is out of the testing phase, yes, it would be OK to delete prpserver244 and prpserver230. I'll do that as soon as I get around to it.

[rogue]

It seems to me that you want a custom webpage. If you know how to write HTML, it shouldn't be hard for you to create a custom one by modify the appropriate class.

[mdettweiler]

Originally Posted by rogue

It seems to me that you want a custom webpage. If you know how to write HTML, it shouldn't be hard for you to create a custom one by modify the appropriate class.

The problem is, I know nothing of C++; the closest thing I know is Java, which was enough to allow me to make rudimentary changes like switching the BKCN and Length variables.

Also, one particular advantage of having the web pages generated by a separate script is that I can copy off the actual results files, convert them to LLRnet format, and post it to the web in incremental updates throughout the day. This is useful for providing updates to the NPLB stats database in 1-hour increments throughout the day (rather than just once a day when the results files are copied off). See the "Results since last copy-off" link at http://nplb-gb1.no-ip.org/prpnet/ for an example.

But, yes, it would be an interesting idea to check out. Possibly we could make a couple tweaks to the built-in web pages to add some extra functionality. Nontheless, probably the best solution for NPLB would still be to have a separate script-generated status page to complement PRPnet's built-in pages.

[gd_barnes]

Originally Posted by mdettweiler

@Gary about the various prpserver binaries: I've gotten in the habit of keeping around variou older "milestone" versions as backups, primarily for use in case something goes majorly wrong with a new version that I'm testing. Now that 2.4.6 is out of the testing phase, yes, it would be OK to delete prpserver244 and prpserver230. I'll do that as soon as I get around to it.

Excellent thinking about keeping older milestone versions. That's a given in the "real world" programming industry. If you mess up a current version beyond repair, you can fairly quickly start anew from the prior one.

Since the testing phase is done on 2.4.6, I'll go ahead and delete prpserver244 and prpserver230 from the folders. The machine's under my finger tips...no use for you to do it.

Thanks all for the tips and work on the PRPnet server. It's been a serious challenge but slowly and surely we've gotten close enough that I'm comfortable graduating all of our LLRnet servers to PRPnet servers as soon as the stats are interfacing with the DB. As Max had mentioned earlier in another thread, we'll probably still leave at least one drive on an LLRnet server for those who prefer them. NPLB has always been about choice; small, medium, and large candidates to test; and now manual, LLRnet, and PRPnet methods of searching.


Gary

[mdettweiler]

Originally Posted by gd_barnes

Thanks all for the tips and work on the PRPnet server. It's been a serious challenge but slowly and surely we've gotten close enough that I'm comfortable graduating all of our LLRnet servers to PRPnet servers as soon as the stats are interfacing with the DB. As Max had mentioned earlier in another thread, we'll probably still leave at least one drive on an LLRnet server for those who prefer them. NPLB has always been about choice; small, medium, and large candidates to test; and now manual, LLRnet, and PRPnet methods of searching.

I belive I mentioned over in the 7th Drive thread that that would probably be a good one to keep on LLRnet long-term, since eventually we'll have two other nearly identical drives running PRPnet, and Ian's already comfortably set up with that one. Do you concur?

You know, come to think of it, this is perfect timing to begin the PRPnet transition. As we're currently on the eve of having the new server ready, this way we can rather easily transition all the IB servers into new PRPnet GB servers--no downtime to worry about since the original servers will be drying out gracefully. And if G4000 is the one server we keep on LLRnet, that works fine since it's already running on dumpford and won't be too hard to copy over to a similar setup on the new server (as opposed to the rather different system David uses for administrating the ones he's hosting).

[gd_barnes]

Originally Posted by mdettweiler

I belive I mentioned over in the 7th Drive thread that that would probably be a good one to keep on LLRnet long-term, since eventually we'll have two other nearly identical drives running PRPnet, and Ian's already comfortably set up with that one. Do you concur?

Isn't that what I implied with "we'll probably still leave at least..."? You might wanna reread.

Originally Posted by mdettweiler

You know, come to think of it, this is perfect timing to begin the PRPnet transition. As we're currently on the eve of having the new server ready, this way we can rather easily transition all the IB servers into new PRPnet GB servers--no downtime to worry about since the original servers will be drying out gracefully. And if G4000 is the one server we keep on LLRnet, that works fine since it's already running on dumpford and won't be too hard to copy over to a similar setup on the new server (as opposed to the rather different system David uses for administrating the ones he's hosting).

I'm confused. How are we on the eve of having a new server ready? Isn't one already ready and running? Begin the PRPnet transition? What does that mean? To me, the plan all along was to wait for current ranges on the LLRnet servers to be about 3 days from drying and create a new PRPnet server in their place on my machines with the next n-range, like we did for port G3000.

But...one problem here. We can't just make everything (except the 7th drive) PRPnet servers without the stats interface and Email notification working. First things first...let's get that working for port 3000. Then we can transition more servers as they are near drying.

Here's how I see the transition going:

1. You coordinate with Dave on getting the stats interface and Email notification working. (this week?)
2. I get my machines built and a call put in on getting a commercial account set up. (call will be made Tues.; machines built by Fri.; actual changeover to commercial account will likely be 1-2 weeks)
3. I do the various techie stuff related to the Smoothwall router and the such and coordinate with you to test it to make sure it is working well. (by Tues. the 8th)
4. You/we move all of the current LLRnet/PRPnet servers from Dumpford to the new machine. No new servers are created at this point. (Weds.-Thurs. 8th-9th)
5. About 3 days before each LLRnet server dries, create an appropriate PRPnet server with the next higher n-range and update our threads as necessary to reflect them. Optionally this could be done well ahead of time with appropriate testing to make sure the stats interface and Email notification is working. (various after Dec. 9th)
6. After each PRPnet server from #5 is verified as working/interfacing/notifying correctly, once again update our threads to reflect the changeover and inform Ironbits to phase out the appropriate server.

To accomplish this in a reasonable time frame, we'll probably need to have Ironbits reduce the n-range for the k=1400-2000 drive and have someone like me put a bunch of cores on it to dry it out.

Ideally it'd be nice to have all of this done by year end but I know I've delayed things quite a bit by dragging my feet on building the machines. At this point, a realistic timeframe would be by ~Jan. 15th.

This effectively avoids stopping a server in the middle of processing an n-range in order to change it from LLRnet to PRPnet. IMHO, that is a potentially messy nightmare to make sure all pairs get processed.


Gary

[mdettweiler]

Originally Posted by gd_barnes

Isn't that what I implied with "we'll probably still leave at least..."? You might wanna reread.

Hmm...you're right, it was worded a bit unclearly. What I was intending to ask was whether you concurred that the 7th Drive/G4000 would be the one to keep on LLRnet.

I'm confused. How are we on the eve of having a new server ready? Isn't one already ready and running? Begin the PRPnet transition? What does that mean? To me, the plan all along was to wait for current ranges on the LLRnet servers to be about 3 days from drying and create a new PRPnet server in their place on my machines with the next n-range, like we did for port G3000.

But...one problem here. We can't just make everything (except the 7th drive) PRPnet servers without the stats interface and Email notification working. First things first...let's get that working for port 3000. Then we can transition more servers as they are near drying.

Here's how I see the transition going:

1. You coordinate with Dave on getting the stats interface and Email notification working. (this week?)
2. I get my machines built and a call put in on getting a commercial account set up. (call will be made Tues.; machines built by Fri.; actual changeover to commercial account will likely be 1-2 weeks)
3. I do the various techie stuff related to the Smoothwall router and the such and coordinate with you to test it to make sure it is working well. (by Tues. the 8th)
4. You/we move all of the current LLRnet/PRPnet servers from Dumpford to the new machine. No new servers are created at this point. (Weds.-Thurs. 8th-9th)
5. About 3 days before each LLRnet server dries, create an appropriate PRPnet server with the next higher n-range and update our threads as necessary to reflect them. Optionally this could be done well ahead of time with appropriate testing to make sure the stats interface and Email notification is working. (various after Dec. 9th)
6. After each PRPnet server from #5 is verified as working/interfacing/notifying correctly, once again update our threads to reflect the changeover and inform Ironbits to phase out the appropriate server.

To accomplish this in a reasonable time frame, we'll probably need to have Ironbits reduce the n-range for the k=1400-2000 drive and have someone like me put a bunch of cores on it to dry it out.

Ideally it'd be nice to have all of this done by year end but I know I've delayed things quite a bit by dragging my feet on building the machines. At this point, a realistic timeframe would be by ~Jan. 15th.

This effectively avoids stopping a server in the middle of processing an n-range in order to change it from LLRnet to PRPnet. IMHO, that is a potentially messy nightmare to make sure all pairs get processed.


Gary

Yes, that's essentially what I had in mind. Sorry for any confusion on that.

As for the email notification and DB import: I'm still waiting for Dave on that one. According to his forum profile, he hasn't been on since the 24th. My guess is he went away for Thanksgiving and still hasn't come back.

[Flatlander]

Originally Posted by gd_barnes

...
I think what I'll do is let it run halfway down and double that time. Hopefully the little indicators of power remaining are reasonably accurate.
...

I think that you have already found this out but there is far too much on this forum (and CRUS) for me to read properly :
My experience with electric wheelchairs (and the emergency power pack in my wagon) tells me that indicator lights are not to be trusted.
The 2nd 'half' is usually much less than the 1st 'half'! (I still don't fully understand that newfangled electrickery.)

[gd_barnes]

PRPnet is not ready for a full rollout at NPLB yet. Therefore we'll have only the 5th, 6th, 10th, and doublecheck drives on it for the foreseeable future. The 12th drive is currently on PRPnet. When the current k=2400-2600 range is near drying out, we'll be creating an LLRnet server for subsequent ranges. The small tests results in large loads are creating too many problems for PRPnet.

As a SWAG, my feeling is that PRPnet should be able to handle 100-120 cores on tests for n>700K. It may be more but to be safe, that would be best.

I've made the applicable modifications to the 1st post here to account for the current situation.

[mdettweiler]

Originally Posted by gd_barnes

PRPnet is not ready for a full rollout at NPLB yet. Therefore we'll have only the 5th, 6th, and doublecheck drives on it for the foreseeable future. The 12th drive is currently on PRPnet. When the current k=2400-2600 range is near drying out, we'll be creating an LLRnet server for subsequent ranges. The small tests results in large loads are creating too many problems for PRPnet.

As a SWAG, my feeling is that PRPnet should be able to handle 40-60 cores on tests for n>700K. It may be more but to be safe, that would be best.

I've made the applicable modifications to the 1st post here to account for the current situation.

Gary, see my post on this in the PRPnet servers for NPLB thread. I've observed that PRPnet can handle somewhat more than you're thinking, even if not the tiny n=50K-250K stuff.

[gd_barnes]

Originally Posted by mdettweiler

Gary, see my post on this in the PRPnet servers for NPLB thread. I've observed that PRPnet can handle somewhat more than you're thinking, even if not the tiny n=50K-250K stuff.

Based on it handling 30-50 cores from Buce at n=200K, I've upped my previous post to 100-120 cores at n=700K. Perhaps it can handle that and maybe it can't. You can't do a direct timing comparison. Example:

Let's say n=200K tests take 100 secs. That means n=700K tests take 100*(7/2)^2=~1200 secs. (Not actual; only an example.)

That doesn't mean it can handle 1200/100=12 times as many clients. That's because what if all clients hit the n=700K range at once? There's no easy way to do an easy apples-to-apples comparison.

From what I can tell, it's based off of how many clients concurrently hit the server and that is regardless of the n-value size. It one person starts 100 cores all at once at n=40M, the server may still barf, even though the tests take 20 days or more. We just have to ask ourselves: What is an acceptable level of risk. If we keep risking these problems, especially on an increasingly higher percentage of our servers, our higher-resourced folks will find testing to do elsewhere.


Gary