LLRnet servers for NPLB

[IronBits]

Schweet!

sudo code

create vars for $date and $time for file names

setup for loops
for nplb(400 5000) do something
for llrnet(10000,10001) do something
for crus(4 5) do something

The full path on Server is: /home/ironbits/$typeserver$port (nplbserver400 or llrnetserver10000 or crusserver6) as an example

The path to store the files moved from each above would be /home/ironbits/results/results_yyyymmddhh_minsec_IB_$type_$port.org.txt

:something

mv /home/ironbits/$typeserver$port/results.txt /home/ironbits/results/results_$date_$time_IB_$type_$port.org.txt
etc.

Then when all the moving is done, we process all the .org.txt files and create .txt for web results dir and .csv for importing

@files = <$parsedir/*.org.txt>;
foreach $file (@files) {
print $file . "\n";
&processFile();
}
for each *.org.txt we would convert and create .txt and .csv files so when it is done, for each $type and $port we would have the following:

results_yyyymmddhh_minsec_IB_$type_$port.org.txt
results_yyyymmddhh_minsec_IB_$type_$port.txt
results_yyyymmddhh_minsec_IB_$type_$port.csv

I can take it from there to get the files over to the other server to place the files where they belong on the respective websites and import directory.

Thanks!

[mdettweiler]

Originally Posted by IronBits

Schweet!

sudo code

create vars for $date and $time for file names

setup for loops
for nplb(400 5000) do something
for llrnet(10000,10001) do something
for crus(4 5) do something

The full path on Server is: /home/ironbits/$typeserver$port (nplbserver400 or llrnetserver10000 or crusserver6) as an example

The path to store the files moved from each above would be /home/ironbits/results/results_yyyymmddhh_minsec_IB_$type_$port.org.txt

:something

mv /home/ironbits/$typeserver$port/results.txt /home/ironbits/results/results_$date_$time_IB_$type_$port.org.txt
etc.

Then when all the moving is done, we process all the .org.txt files and create .txt for web results dir and .csv for importing

@files = <$parsedir/*.org.txt>;
foreach $file (@files) {
print $file . "\n";
&processFile();
}
for each *.org.txt we would convert and create .txt and .csv files so when it is done, for each $type and $port we would have the following:

results_yyyymmddhh_minsec_IB_$type_$port.org.txt
results_yyyymmddhh_minsec_IB_$type_$port.txt
results_yyyymmddhh_minsec_IB_$type_$port.csv

I can take it from there to get the files over to the other server to place the files where they belong on the respective websites and import directory.

Thanks!

Okay...I think I see what you're saying. My scripts as they are don't quite work in the way you've described the proposed setup, but fortunately the CSV processing, at least, is somewhat modular, being a separate .pl file that takes the input and output files from the command line. As you can see from the latest llrnet-copyoff-results.pl file that I sent you via email, that script essentially copies off the results files from the LLRnet server folders, directly to the /var/www/llrnet/results/ directory (i.e. the backend for http://nplb-gb1.no-ip.org/llrnet/results/); then it feeds /var/www/llrnet/results/name_of_results_file.txt to csv.pl as input, and has it output to /var/www/llrnet/csv/name_of_results_file.csv. It isn't quite set up to deal with .org.txt files and all that stuff.

However, yes, the fundamental concepts of your idea would still work, modified of course to fit my script. How about this?

-llrnet-copyoff-results.pl moves the results.txt files to /tmp/llrnet/results_file_name.txt
-llrnet-copyoff-results.pl launches csv.pl, feeding it an input file name of /tmp/llrnet/results_file_name.txt and an output file name of /tmp/llrnet/results_file_name.csv. Repeat for each server.
-A script of some sort over on the "quad" box pulls down the results and csv files via scp or other such method once a day, 5 minutes or so after the files are generated by "server".

Then for the status page generator script:

-llrnet-status-pagegen.pl (in case you don't recognize the name, that's because I haven't sent this one to you yet ) looks at /home/ironbits/$typeserver$port/results.txt, knpairs.txt, rejected.txt, etc. and generates all the need info from it. This happens once an hour.
-llrnet-status-pagegen.pl outputs the file /tmp/llrnet/status_page.html
-A script over on "quad" pulls down status_page.html once an hour, and saves it as index.html (or whatever) in the respective folder. Again, this should probably be scheduled a minute or two after the file is generated so as to avoid any potential locking conflicts or stuff like that.

Maybe we'd be better off using a different directory instead of /tmp, such as /home/ironbits/llrnet_temp/ or something like that, so that the files don't get erased if the machine reboots unexpectedly in between the time when the files are generated and when they're picked up by "quad". Anyway, you get the picture.

Max

P.S.: You mentioned llrnet.ironbits.net ports 10000 and 10001, which reminded me: I just dried port 10001 earlier today. You can go ahead and dismantle port 10001 as soon as it's posted its results files for tomorrow. As discussed via email, you should probably leave port 10000 in place, though of course the server application itself doesn't need to be running as long as there's no work in it.

P.P.S.: You also mentioned crus.ironbits.net ports 4 and 5. I thought we only had port 6 (which, if I remember correctly, was set to port 6000 on your backend)? I know we used to have a port 4 also (but that's long since been dismantled), but never a port 5.

[IronBits]

Wow! Just need one script to run once every 24 hours, and one that runs hourly.
2 scripts to do it all.

I'll take a look at anything you want to send me so I have some sample working code to work with, and I'll get it all fixed up over here for multiple $types and $ports, and I'll send it to you when I'm done.

I'm planning on sprucing up the layout, just haven't wrapped my head around a decent layout for the data I'm putting up on the website...

It's gonna take me a little while, so don't hold your breath yet!

Glad you have a working system over there in linux.

[mdettweiler]

Originally Posted by IronBits

Wow! Just need one script to run once every 24 hours, and one that runs hourly.
2 scripts to do it all.

I'll take a look at anything you want to send me so I have some sample working code to work with, and I'll get it all fixed up over here for multiple $types and $ports, and I'll send it to you when I'm done.

I'm planning on sprucing up the layout, just haven't wrapped my head around a decent layout for the data I'm putting up on the website...

It's gonna take me a little while, so don't hold your breath yet!

Glad you have a working system over there in linux.

Okay, sounds good! I'll get the scripts all fixed up for you and send 'em your way tomorrow.

[mdettweiler]

Hi all,

Gary just installed some updates on the box that's hosting the G4000 server, and it will need to be rebooted in order to complete the process. We're currently holding off on the reboot until everyone running on this server has a chance to shift their resources elsewhere (at least temporarily).

Anyway, it appears that MyDogBuster and myself are the only two users currently crunching on G4000; Buster, could you please confirm here when you get your machines moved off G4000? Thanks.

Max

[em99010pepe]

Max,

You can reboot the server, don't worry about the clients, they will pick up the pace when they next connect.

Just trying IB code:

C443
Work done today by:
em99010pepe : 3828
Kman1293 : 1742
MrOzzy : 794
MyDogBuster : 718

Carlos

[mdettweiler]

Originally Posted by em99010pepe

Max,

You can reboot the server, don't worry about the clients, they will pick up the pace when they next connect.

Yeah, I'd thought of that...but I was a little wary of rebooting it just yet in case anybody connecting had a WUCacheSize of 1. That, and I'm also waiting for the OK from Gary to reboot the server, so I figured I may as well give everybody a warning so that they could either move to a different server, or increase their cache size if necessary, before I reboot it.

P.S.: Gary, I forgot to mention in my PM, when I reboot the server you should probably have crunchford plugged into a monitor, if it isn't already. I've noticed that if I reboot a computer without a monitor attached, usually when it starts back up it gets the resolution, refresh rate, etc. all wrong.

[MyDogBuster]

Sorry, Taking a nap. I'm set re-boot anytime.

[mdettweiler]

Originally Posted by MyDogBuster

Sorry, Taking a nap. I'm set re-boot anytime.

Okay, cool. Now I've just got to wait for Gary to give me the OK and I'll reboot the server.

[gd_barnes]

Originally Posted by mdettweiler

Okay, cool. Now I've just got to wait for Gary to give me the OK and I'll reboot the server.

Reboot away. That's weird about it needing to be connected to a monitor to get the refresh thing right. I have to drop my kids off now. If you wait until after 10:30 EST, I'll go down and connect it to a monitor. Usually it would be connected to one but right now, I have one monitor upstairs doing tests on my "down" machines.

I agree with Carlos...really people shouldn't have to move their machines off of the port. They should reconnect quickly if you reboot and restart the server quickly.


Gary

[mdettweiler]

Originally Posted by gd_barnes

Reboot away. That's weird about it needing to be connected to a monitor to get the refresh thing right. I have to drop my kids off now. If you wait until after 10:30 EST, I'll go down and connect it to a monitor. Usually it would be connected to one but right now, I have one monitor upstairs doing tests on my "down" machines.

I agree with Carlos...really people shouldn't have to move their machines off of the port. They should reconnect quickly if you reboot and restart the server quickly.


Gary

Okay, it's 10:30 EST right now--I'll go ahead and reboot the machine. As for it needing to be connected to a monitor: hmm, I think I may have been getting confused about that. You see, during the "testing phase" for my quad, I had it sitting right behind my dualcore, and was live-swapping the monitor, keyboard, and mouse connections when I wanted to have a console on one or the other. Usually when either of the machines was unattended, I would leave the stuff plugged in to my dualcore, but then if the quad rebooted unexpectedly (it did that from time to time because of a since-fixed hard drive cable issue), when I plugged the stuff back into it I would find it at 640x480 resolution and the keyboard and mouse wouldn't work.

Then again, maybe there's a difference in the fact that my quad runs Windows instead of Linux like yours do. Not to mention that, if memory serves, you have a keyboard and mouse plugged in to all of your quads even when they don't have a monitor on them, correct?

Anyway, I'll go reboot the server shortly...