PRPnet servers for NPLB

[gd_barnes]

In case you missed it, all NPLB LLRnet and PRPnet servers are now up and running. My apologies for the bad hard drive that happened at the very beginning of an 8-day business trip. The holidays also delayed me getting it swapped out after getting back.

It might help to put a little bit of stress on the GB servers again to make sure that the different hard drive can handle it. Because we have a lot of logging set on in case there are server problems, there are a lot of reads and writes to the hard drive. So it's important to verify that it can handle them.

Interestingly the old "bad" hard drive works just fine on a quad running 4 LLRnet cores and no servers. With comparitively few reads and writes, it doesn't have any problem.


Gary

[mdettweiler]

Originally Posted by gd_barnes

Interestingly the old "bad" hard drive works just fine on a quad running 4 LLRnet cores and no servers. With comparitively few reads and writes, it doesn't have any problem.

I wonder if it has something to do with which portion of the hard drive is being read from/written to. Possibly one of the servers was on a bad portion that (somehow) caused it to go bonkers, while your LLRnet clients and all the other things being run now are on parts that work OK. In that case, it could start acting up again in the future if you tried to put something more on it, and happened to run into the bad part. But, at any rate, since it's working all right now, we may as well leave it be since there's nothing important on it any more. If it keeps working fine with the LLRnet cores, great, you saved $60 or so to replace the HD. If it eventually poops out, well, at least there's nothing important on there to lose.

[gd_barnes]

Originally Posted by mdettweiler

I wonder if it has something to do with which portion of the hard drive is being read from/written to. Possibly one of the servers was on a bad portion that (somehow) caused it to go bonkers, while your LLRnet clients and all the other things being run now are on parts that work OK. In that case, it could start acting up again in the future if you tried to put something more on it, and happened to run into the bad part. But, at any rate, since it's working all right now, we may as well leave it be since there's nothing important on it any more. If it keeps working fine with the LLRnet cores, great, you saved $60 or so to replace the HD. If it eventually poops out, well, at least there's nothing important on there to lose.

Well, after backing it up, I deleted all of the files, including server files, off of it and copied the other machine's files onto it. So who knows. It may be on such a different part of the hard drive now that it won't encounter the bad part for ages.

[gd_barnes]

GB servers are back online after a day's outage, likely due to a bad router.

The http://nplb-gb1.no-ip.org/prpnet/ web page still doesn't work. Max would have been unable to work on that since my servers were down. All other pages should work.

Sorry about the problems.

[mdettweiler]

Originally Posted by gd_barnes

The http://nplb-gb1.no-ip.org/prpnet/ web page still doesn't work. Max would have been unable to work on that since my servers were down. All other pages should work.

Actually, the news is worse than that. None of the pages work, whether LLRnet or PRPnet; they're affected by the same issue (which is in the webserver, apparently). I still need to fix this...stay tuned.

[gd_barnes]

Sorry. I have taken the servers back offline. It's more complex than just a simple router problem. It's very unusual and could be related to some sort of bad connections or bad splitter boxes on my other machines, all of which are wired.

[gd_barnes]

Holy crap, that was hard. ...been on the problem for ~5-6 hours. I've finally isolated my internet connection problem to one bad splitter box in the basement (out of the 2 down there). It "appears" that it was on-and-off flooding packets (I think that's close to the right term) into my router. Whenever I would plug the bad splitter box cable into the router, everything including my server machine would go off line immediately even though it was a different chord. With the downstairs machines unplugged, the main server machine and my wireless laptop connected just fine.

So after buying a new router and determining that wasn't the problem, I'll have to exchange that for a new splitter box. Unfortunately my one splitter only allows internet access to ~4-5 machines so half mine won't be running server work.

The good news is that without the flooding, the server quad, my laptop, and 4 of my downstairs quads have internet access so the servers should be good now.

That's about as difficult a technical problem that I can personally resolve with my limited knowledge of internet connections and servers. Even the tech guys at my ISP provider were convinced that I had a bad router. Of course they weren't here to see the setup but they did tell me there were unusual spurts/spikes of activity going thru the modem, which lead me to the packet flooding conclusion.

Max, how's that for a pain in the rear end? Once I changed the input to the router box from one downstairs splitter to the other, all machines connected to that splitter and the server machine connected directly to the router and my wireless laptop all worked.

One question for you: Have you heard of a bad splitter box doing such a thing? The above is the only conclusion I could draw from nearly a day's worth of trying different things.


Gary

[gd_barnes]

Originally Posted by mdettweiler

Actually, the news is worse than that. None of the pages work, whether LLRnet or PRPnet; they're affected by the same issue (which is in the webserver, apparently). I still need to fix this...stay tuned.

Hum. They all seem to work to me except the one that I previously noted. Here are the pages I'm referring to:

Status report: http://nplb-gb1.no-ip.org:3000/
Server user stats*: http://nplb-gb1.no-ip.org:3000/user_stats.html

(plus all reports and stats for each of the gb servers)

Are there others you are referring to?

[mdettweiler]

Originally Posted by gd_barnes

Max, how's that for a pain in the rear end? Once I changed the input to the router box from one downstairs splitter to the other, all machines connected to that splitter and the server machine connected directly to the router and my wireless laptop all worked.

One question for you: Have you heard of a bad splitter box doing such a thing? The above is the only conclusion I could draw from nearly a day's worth of trying different things.

No, I haven't heard of that before, though I haven't had much personal experience with splitter boxes (or switches, as the correct technical term would be, though splitter is an accurate approximation of its functionality in this use).

Sounds like your old router should be good, then--looks like this problem had absolutely nothing to do with it. Given your description of what your ISP saw on their end, though, I can see how they would have thought that was the case. In this case there's probably not much they could have done to diagnose it from their end.

Well, it's definitely good to know that it wasn't a bad router. That would have been a HUGE pain to get set up with all the static IPs we're using internally to facilitate the VNC stuff. Despite the pain of having all those machines offline for a while, the pain of switching out a splitter box is a comparative piece of cake--no configuration required, just plug it in.

Originally Posted by gd_barnes

Hum. They all seem to work to me except the one that I previously noted. Here are the pages I'm referring to:

Status report: http://nplb-gb1.no-ip.org:3000/
Server user stats*: http://nplb-gb1.no-ip.org:3000/user_stats.html

(plus all reports and stats for each of the gb servers)

Are there others you are referring to?

Ah, I see what you mean now. The PRPnet server reports are generated by the servers themselves, which are isolated from the problem causing the "403 Forbidden" errors. That seems to be somewhere in the web server, which only affects the main web site (i.e., pages like http://nplb-gb1.no-ip.org/llrnet/).

[gd_barnes]

Max,

Can you make it a priority to look into the problems with the http://nplb-gb1.no-ip.org/llrnet/ and http://nplb-gb1.no-ip.org/prpnet/ web pages today or Monday? More than anything, it would be nice for people to see their # of pairs processed on the hourly report.

Thanks a bunch.


Gary

[mdettweiler]

Originally Posted by gd_barnes

Max,

Can you make it a priority to look into the problems with the http://nplb-gb1.no-ip.org/llrnet/ and http://nplb-gb1.no-ip.org/prpnet/ web pages today or Monday? More than anything, it would be nice for people to see their # of pairs processed on the hourly report.

Thanks a bunch.


Gary

Well, that was easy. Turns out it was just a stupid little permissions error, which rather strangely showed up only in the GUI interface but not on the command line. Should be all fixed now.