Server machine Aug. 20th crash and backups

[kar_bon]

Originally Posted by mdettweiler

One thing to note: even though the pruning only can be triggered by an event on the server, all data since the last prune is not just stored in memory; it's also stored in joblist.txt as "appended" data. The prune just removes any unnecessary "appended" data (for instance, cancelling out corresponding entries for reservation and result) and cleans out knpairs.txt.

Yes, and that's what I can't understand:

I've got those 660 pairs before that issue (see Drive Progress from 20., 19:00-20:00, so these reservations should be stored in the joblist.txt file) and returned all pairs after the issue. The issue occured the hour after the reservation.
So what happened between those two events with the joblist.txt?
The knpairs.txt contain all pairs until the results were returned.

The joblist.txt is the only place where the server can check if a pair is reserved.

The rejected-file says all 660 pairs were returned between 18:00 and 19:00, but there was no issue with the connection, so the failure was before this.

What I think:

During the issue, something went wrong with the joblist.txt: not saved correctly or even lost at all. Because knpairs.txt contained still those pairs, the server offered these pairs to the clients again (for the server the first time, because joblist.txt was empty/corrupted/lost).

It seems, the joblist.txt was lost for this drive only and the server was started with an empty joblist then!

I only want to understand the whole thing here, otherwise I don't know what to make better to avoid this the next time.

[mdettweiler]

Originally Posted by kar_bon

Yes, and that's what I can't understand:

I've got those 660 pairs before that issue (see Drive Progress from 20., 19:00-20:00, so these reservations should be stored in the joblist.txt file) and returned all pairs after the issue. The issue occured the hour after the reservation.
So what happened between those two events with the joblist.txt?
The knpairs.txt contain all pairs until the results were returned.

The joblist.txt is the only place where the server can check if a pair is reserved.

The rejected-file says all 660 pairs were returned between 18:00 and 19:00, but there was no issue with the connection, so the failure was before this.

What I think:

During the issue, something went wrong with the joblist.txt: not saved correctly or even lost at all. Because knpairs.txt contained still those pairs, the server offered these pairs to the clients again (for the server the first time, because joblist.txt was empty/corrupted/lost).

It seems, the joblist.txt was lost for this drive only and the server was started with an empty joblist then!

I only want to understand the whole thing here, otherwise I don't know what to make better to avoid this the next time.

Yeah, that does sound about right--considering that entire MySQL databases got fried over on the PRPnet side of things, it's not inconceivable that joblist.txt went bye-bye as well. So that probably is what happened.

[kar_bon]

Originally Posted by mdettweiler

Yeah, that does sound about right--considering that entire MySQL databases got fried over on the PRPnet side of things, it's not inconceivable that joblist.txt went bye-bye as well. So that probably is what happened.

So what can we do, that this won't happen again?

Every server has it's own files for storing results in. The LLRnet server is here very simple:
3 files: joblist.txt, knpairs.txt, results.txt, that's all.

Saving these files every hour of a day with names like 'joblist_xx.txt' with xx the hour of the day, should store them in a different folder. So this would be enough time (24 hours before overwriting old copies) and in case of an issue (current files from server lost/damaged) only 1 or 2 hours of work will be lost!
Another point of view: If for example the knpairs.txt was damaged, it's timeconsuming to create a new one, taking all results done in account.

Also I don't know how the servers take care of their result-files?
The resultfile of the LLRnet server for Drive #11 would be great because of the amount of pairs done. If an issue occure while writing in this file, it could be damaged massively!

Other thoughts here?

[mdettweiler]

Originally Posted by kar_bon

So what can we do, that this won't happen again?

Every server has it's own files for storing results in. The LLRnet server is here very simple:
3 files: joblist.txt, knpairs.txt, results.txt, that's all.

Saving these files every hour of a day with names like 'joblist_xx.txt' with xx the hour of the day, should store them in a different folder. So this would be enough time (24 hours before overwriting old copies) and in case of an issue (current files from server lost/damaged) only 1 or 2 hours of work will be lost!
Another point of view: If for example the knpairs.txt was damaged, it's timeconsuming to create a new one, taking all results done in account.

Hmm...perhaps something a little simpler. What if each time the LLRnet server updated its joblist file, it first copied it to joblist.bak, then edited joblist.txt itself? That way, if it crashes and takes the joblist file down with it, we have a next-newest good copy on hand. A similar thing could be done for knpairs.txt as well.

Also I don't know how the servers take care of their result-files?
The resultfile of the LLRnet server for Drive #11 would be great because of the amount of pairs done. If an issue occure while writing in this file, it could be damaged massively!

We have two things happening to the results files (both LLRnet and PRPnet) on a regular basis:

-Every 15 minutes, the server's results.txt file is copied to /todayresults_portnum.txt on the web site. This is used for hourly updates by the DB and grows as the day goes on.
-Every day at 12:00 midnight, the results.txt file is moved to /llrnet/results/results_date_time_server_nplb_port.txt on the web site. This empties the local copy and, by extension, /todayresults_portnum.txt on the web site.

So if the local results.txt file is zapped, it can only take up to a day's worth of results with it.

[kar_bon]

Originally Posted by mdettweiler

Hmm...perhaps something a little simpler. What if each time the LLRnet server updated its joblist file, it first copied it to joblist.bak, then edited joblist.txt itself? That way, if it crashes and takes the joblist file down with it, we have a next-newest good copy on hand. A similar thing could be done for knpairs.txt as well.

But to get this working, the llrnet-server LUA-codes has to be changed first and have to be tested to be sure runable for UNIX and WIN!

The easiest way I think of, would be a cron-job (if I'm right, all NPLB servers running under UNIX).

[mdettweiler]

Originally Posted by kar_bon

But to get this working, the llrnet-server LUA-codes has to be changed first and have to be tested to be shure runable for UNIX and WIN!

The easiest way I think of, would be a cron-job (if I'm right, all NPLB servers running under UNIX).

Yeah, you're right, a cron job would be easier. Okay, I'll see what I can put together.

[vaughan]

Is the LLRnet server Port=3000 having any problems? My clients are running out of work and the GUI says "sleeping".

[mdettweiler]

Originally Posted by vaughan

Is the LLRnet server Port=3000 having any problems? My clients are running out of work and the GUI says "sleeping".

Looks like the server ran out of work. I'll fill it up again shortly.