Server outrages

[AMDave]

I have plotted a likely solution.
Testing will be done over the next 2 weeks before committing to NPLB.

[mdettweiler]

Originally Posted by AMDave

I don't think so.
That's the exact time that the daily refresh script ran the weekly result table optimize task.
This task has been in place since "# HISTORY : AMDave 20120524 original"
That keeps mysql very busy for more than a few minutes due to the size of the results table.
If mysql is very very busy it can impact the prpnet database I/O, as I think it may have done here.
I will turn the weekly task off for a while I analyse and try to find a 'softer' solution.

Ah, that's interesting. That would make sense to explain this "outage", since given the time of day, both the optimize task and the daily stats run were in progress.

Would that also explain the two earlier outages on the 18th (reported a few posts up)?

[AMDave]

Afraid not. That one is still a mystery.

[AMDave]

NPLB DR server planned outage: 24:00hrs 2014-08-07 AEST - 2 to 5 days.
WHAT: There will be a planned outage of the NPLB DR server for relocation of the hardware. This outage will not affect any normal NPLB services.
WHEN: from 24:00hrs 2014-08-07 AEST .
DURATION: The planned relocation window is 5 days, target is 2 days. The shortest possible turnaround on the critical path is 24 hours, depending on network build tasks, lunar and astral alignments, the lucky number 16, the market price of Longjing in Zhejiang Province and whether the roaring forties happens to be blowing at the time.

[gd_barnes]

Originally Posted by AMDave

Afraid not. That one is still a mystery.

I'm confident that those were outages were my provider monkeying around. I was online at my main Windows desktop machine when I saw the lights on the modem gradually disappear and then come back on a few minutes later as I waited for my pages to load for several minutes. I'm guessing that they occassionally do some random maintenance late at night.

[AMDave]

Originally Posted by AMDave

I have plotted a likely solution.
Testing will be done over the next 2 weeks before committing to NPLB.

Quick update:
The database patch script and refresh script changes are now designed and being tested.
Although the patch is simple it takes a long time to process the results table due to it's size.
There is no simple alternative, so a stats outage will need to be scheduled to deploy the patch.
I am still confirming timings & additional tests to verify that the desired outcomes will be achieved.
I am quite optimistic & will post again when I have the full set of results, to schedule the change.

[MyDogBuster]

Looks like we are down again.

[mdettweiler]

The server itself is still up and running (and the http://noprimeleftbehind.net/ web page is working just fine), but the LLR/PRPnet servers are all out.

It appears that Gary's power company came by and switched out his electric meter, as we'd been warned of over in the "News" thread. According to jeepford, its current uptime is about 5 hours, which means it partially survived the power outage: the UPS must have held it, but for whatever reason, the computer rebooted itself. (That often happens on semi-short power flickers, which is probably what happened since consumer-grade UPSes usually take a second or so to switch between AC and battery.)

That means that the computer is up and running, but not logged in. I can restart the LLR/PRPnet servers remotely...I'll go do that now.

[mdettweiler]

Well, after just attempting to get in through the "back door non-console-session VNC", I remembered why we never used that for jeepford. It's all coming back to me now...that solution worked fine on Gary's other (read: older OS) machines, but the magic incantations I'd put in the ~/.vnc/xstartup script didn't work for Ubuntu 9.04. Whenever you try to type anything, it comes out as gibberish because something's messed up the keyboard layout. (Yes, we're still on 9.04. Despite multiple non-starter attempts to upgrade to a newer version, and a totally overambitious plan I came up with to virtualize the entire server in a VM, we still have never actually finished the upgrade.)

So: long story short, the "back door style VNC" doesn't work on jeepford, and it never did. That's why we always used the "Ubuntu built-in console VNC" for jeepford. Trouble is, that one doesn't work until the system's been logged on locally from the console. I thought we solved that problem by setting it to log in automatically on boot-up, but, apparently that's not working for whatever reason.

Never fear, though! I can still start the LLR/PRPnet servers. I just have to do it from the SSH console...stand by.

[mdettweiler]

Quick upshot: all the servers are back online.

Long version for Gary:

Since I couldn't get graphical access to jeepford through VNC (well, I could, but the keyboard didn't work...), I had to start the servers through the textual SSH console. In order to multiplex the console (to allow starting multiple LLR/PRPnet server processes that would keep running when I logged out), I needed to use a program called "screen", which is very handy, and very powerful, but has a huge learning curve.

So, to make this easier (I have yet to learn screen properly myself), I used a handy little program called "screenie", which is a nice, simple wrapper around "screen" that doesn't require learning all those crazy keyboard shortcuts.

This is basically the old-fashioned (text-only) way of opening up multiple tabs on a terminal window like we usually do to start the servers.

In short, here's what you need to do to get to the servers I started (i.e., to shut them down, restart them, etc.):

• Open a terminal window. (Or connect remotely via SSH.)
• Run "./screenie". (This will only work from your home directory, so if you've cd'd to somewhere else, type "cd" to go back.)
• You will see a list of screens I've opened up - they're each labeled with the appropriate LLRnet/PRPnet port number.
• To go to one of the server screens, type the number next to it, and press Enter.
• You are now "inside" the terminal window associated with that screen. You can press Ctrl-C to stop the server, use the console as usual, restart the server, etc. If you type "exit", this screen window will go away, just like in the graphical terminal, and you'll be returned to the menu.
• To return to the menu without closing the terminal window - for instance, to leave an LLR/PRPnet server running - press Ctrl-A, then the D key. That will take you back to the menu.
• To quit screenie and return to the "master" terminal (the original window you opened, or your initial SSH session if you connected remotely), type "q", then press Enter at the screenie menu. Any screen windows shown as open on the menu will remain open, even when you log out.

BTW, if you want to restart your PRPnet clients on jeepford remotely, you can add additional terminal windows to screenie by typing "a" then pressing Enter at the menu. It will ask you to name the window, and then for a "job" - that's a bit confusing, but you should just enter "bash". (Don't ask. If you're wondering, though, that's the name of the Linux terminal, like cmd.exe on Windows.) From there, you have another terminal window you can use like any other - you can start up a PRPnet client (or sr2sieve, if that's what you have jeepford doing right now), and press "Ctrl-A d" to go back to the menu and leave it running, just like with the servers.

Confused yet? Sorry if this is a lot to take in...if you don't care about restarting whatever clients you had going on jeepford, don't worry, everything's up and running now. This is mainly for when you need to go in and stop the LLR/PRP servers, i.e. if you're going to reboot jeepford at some point.

[MyDogBuster]

Does all this mean they are up or down 'cause I still can't get to Port 1468.