GPU to 72 status...

[bayanne]

Is GPU72.com down for maintenance at present?

[storm5510]

Originally Posted by bayanne

Is GPU72.com down for maintenance at present?

It was at the time of this writing.

I found my Colab instance stopped again this morning. Paying more attention, the details read, "Connection timed out." So, I wrote a short batch file to "ping" an external IP address every 60 seconds. Perhaps this will keep it awake.

[chalsall]

Originally Posted by bayanne

Is GPU72.com down for maintenance at present?

Not ***scheduled*** maintenance... Another bad SeaCRAP HD...

GPU72 is back up for the moment. I need to schedule a swap-out with 1and1. There will be at least one additional offline period, hopefully only a half-hour or so. I will try to give as much notice as I can (1and1 only give about a four-hour window of possibility).

Going to be an amusing day...

[chalsall]

Originally Posted by chalsall

Going to be an amusing day...

OK... Quick update...

Because I've had so much unreliability with this machine's HDs over the years, I've decided to spin up a new dedicated server, and transition over. This is instead of swapping out (yet another) SeaCRAP drive, and having the RAID1 rebuild.

TL;DR: GPU72 should remain stable; its IPs will transition over the next few days. I'll let everyone know when that's (temporally) programmed for.

[chalsall]

I'm thinking I might want to document what happens when bringing a "virgin public-facing server" online. Most people have no idea what it's like.

I haven't even spun up the server which answers port 80 requests yet, and this is what I'm currently seeing:

[root@72116a7 ~]# tcpdump -nl port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:22:17.852926 IP 185.53.88.39.62644 > 74.208.169.89.http: Flags [S], seq 2761647998, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:20.846592 IP 185.53.88.39.62644 > 74.208.169.89.http: Flags [S], seq 2761647998, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:21.588242 IP 13.57.244.232.53726 > 74.208.27.191.http: Flags [S], seq 1435274884, win 29200, options [mss 1460,sackOK,TS val 1397441 ecr 0,nop,wscale 9], length 0
21:22:21.766143 IP 193.200.164.135.35414 > 74.208.169.89.http: Flags [S], seq 2070858583, win 32120, options [mss 1460,sackOK,TS val 12283524 ecr 1543503872,nop,wscale 0], length 0
21:22:21.839465 IP 13.57.244.232.53728 > 74.208.27.191.http: Flags [S], seq 3925785016, win 29200, options [mss 1460,sackOK,TS val 1397504 ecr 0,nop,wscale 9], length 0
21:22:22.007267 IP 13.52.104.84.50254 > 74.208.27.191.http: Flags [S], seq 1030701951, win 29200, options [mss 1460,sackOK,TS val 324447040 ecr 0,nop,wscale 9], length 0
21:22:22.238599 IP 193.200.164.135.35901 > 50.21.176.101.http: Flags [S], seq 347881487, win 32120, options [mss 1460,sackOK,TS val 13263582 ecr 218103808,nop,wscale 0], length 0
21:22:22.876213 IP 185.53.88.39.65490 > 74.208.169.89.http: Flags [S], seq 652498964, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:25.885735 IP 185.53.88.39.65490 > 74.208.169.89.http: Flags [S], seq 652498964, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:26.294492 IP 51.38.189.70.61000 > 70.35.195.244.http: Flags [S], seq 1925924080, win 1024, length 0
21:22:26.382673 IP 193.200.164.135.idp-infotrieve > 50.21.179.44.http: Flags [S], seq 3123960651, win 32120, options [mss 1460,sackOK,TS val 9465531 ecr 1577058304,nop,wscale 0], length 0
21:22:27.902216 IP 185.53.88.39.51850 > 74.208.169.89.http: Flags [S], seq 303222851, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:28.523814 IP 51.38.189.70.61000 > 70.35.195.244.http: Flags [S], seq 1925924080, win 1024, length 0
21:22:29.120483 IP 193.200.164.135.63547 > 62.151.183.227.http: Flags [S], seq 959754598, win 32120, options [mss 1460,sackOK,TS val 5323616 ecr 973078528,nop,wscale 0], length 0
21:22:30.911449 IP 185.53.88.39.51850 > 74.208.169.89.http: Flags [S], seq 303222851, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:32.923773 IP 185.53.88.39.54571 > 74.208.169.89.http: Flags [S], seq 749253661, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:34.506304 IP 193.200.164.135.35414 > 74.208.169.89.http: Flags [S], seq 2070858583, win 32120, options [mss 1460,sackOK,TS val 12283524 ecr 1543503872,nop,wscale 0], length 0
21:22:35.918028 IP 185.53.88.39.54571 > 74.208.169.89.http: Flags [S], seq 749253661, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:37.947237 IP 185.53.88.39.57365 > 74.208.169.89.http: Flags [S], seq 3087588616, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:39.771201 IP 193.200.164.135.35901 > 50.21.176.101.http: Flags [S], seq 347881487, win 32120, options [mss 1460,sackOK,TS val 13263582 ecr 218103808,nop,wscale 0], length 0
21:22:40.940817 IP 185.53.88.39.57365 > 74.208.169.89.http: Flags [S], seq 3087588616, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:41.034219 IP 193.200.164.135.51870 > 216.250.114.228.http: Flags [S], seq 2348093212, win 32120, options [mss 1460,sackOK,TS val 8450569 ecr 1023410176,nop,wscale 0], length 0
21:22:42.969197 IP 185.53.88.39.60118 > 74.208.169.89.http: Flags [S], seq 1478602993, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:43.349271 IP 193.200.164.135.63547 > 62.151.183.227.http: Flags [S], seq 959754598, win 32120, options [mss 1460,sackOK,TS val 5323616 ecr 973078528,nop,wscale 0], length 0
21:22:43.887361 IP 193.200.164.135.idp-infotrieve > 50.21.179.44.http: Flags [S], seq 3123960651, win 32120, options [mss 1460,sackOK,TS val 9465531 ecr 1577058304,nop,wscale 0], length 0
21:22:45.978833 IP 185.53.88.39.60118 > 74.208.169.89.http: Flags [S], seq 1478602993, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:47.990786 IP 185.53.88.39.63029 > 74.208.169.89.http: Flags [S], seq 2981964487, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:48.843445 IP 193.200.164.135.12072 > 70.35.207.84.http: Flags [S], seq 3654138408, win 32120, options [mss 1460,sackOK,TS val 6646050 ecr 16777216,nop,wscale 0], length 0
21:22:50.026514 IP 193.200.164.135.52503 > 74.208.82.104.http: Flags [S], seq 3524245071, win 32120, options [mss 1460,sackOK,TS val 971533 ecr 1962934272,nop,wscale 0], length 0
21:22:50.483613 IP 13.52.104.84.52384 > 74.208.27.191.http: Flags [S], seq 811130140, win 29200, options [mss 1460,sackOK,TS val 324454159 ecr 0,nop,wscale 9], length 0
21:22:51.000529 IP 185.53.88.39.63029 > 74.208.169.89.http: Flags [S], seq 2981964487, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:51.511316 IP 13.52.104.84.52384 > 74.208.27.191.http: Flags [S], seq 811130140, win 29200, options [mss 1460,sackOK,TS val 324454416 ecr 0,nop,wscale 9], length 0
21:22:52.051318 IP 193.200.164.135.35414 > 74.208.169.89.http: Flags [S], seq 2070858583, win 32120, options [mss 1460,sackOK,TS val 12283524 ecr 1543503872,nop,wscale 0], length 0
21:22:52.136877 IP 193.200.164.135.51870 > 216.250.114.228.http: Flags [S], seq 2348093212, win 32120, options [mss 1460,sackOK,TS val 8450569 ecr 1023410176,nop,wscale 0], length 0
21:22:52.306202 IP 193.200.164.135.35901 > 50.21.176.101.http: Flags [S], seq 347881487, win 32120, options [mss 1460,sackOK,TS val 13263582 ecr 218103808,nop,wscale 0], length 0
21:22:53.014729 IP 185.53.88.39.49467 > 74.208.169.89.http: Flags [S], seq 2751108481, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:53.527195 IP 13.52.104.84.52384 > 74.208.27.191.http: Flags [S], seq 811130140, win 29200, options [mss 1460,sackOK,TS val 324454920 ecr 0,nop,wscale 9], length 0
21:22:53.935789 IP 193.200.164.135.35901 > 50.21.176.101.http: Flags [S], seq 347881487, win 32120, options [mss 1460,sackOK,TS val 13263582 ecr 218103808,nop,wscale 0], length 0
21:22:55.631452 IP 13.57.244.232.53728 > 74.208.27.191.http: Flags [S], seq 3925785016, win 29200, options [mss 1460,sackOK,TS val 1405952 ecr 0,nop,wscale 9], length 0
21:22:55.636659 IP 13.57.244.232.53726 > 74.208.27.191.http: Flags [S], seq 1435274884, win 29200, options [mss 1460,sackOK,TS val 1405953 ecr 0,nop,wscale 9], length 0
21:22:56.023902 IP 185.53.88.39.49467 > 74.208.169.89.http: Flags [S], seq 2751108481, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:57.193133 IP 66.249.88.59.62591 > 216.250.114.228.http: Flags [S], seq 3033941043, win 62920, options [mss 1430,sackOK,TS val 2239715846 ecr 0,nop,wscale 8], length 0
21:22:57.591107 IP 13.52.104.84.52384 > 74.208.27.191.http: Flags [S], seq 811130140, win 29200, options [mss 1460,sackOK,TS val 324455936 ecr 0,nop,wscale 9], length 0
21:22:57.904296 IP 193.200.164.135.idp-infotrieve > 50.21.179.44.http: Flags [S], seq 3123960651, win 32120, options [mss 1460,sackOK,TS val 9465531 ecr 1577058304,nop,wscale 0], length 0
21:22:58.042139 IP 185.53.88.39.52241 > 74.208.169.89.http: Flags [S], seq 3834427809, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:22:58.125704 IP 66.249.88.58.61608 > 216.250.114.228.http: Flags [S], seq 3204750274, win 62920, options [mss 1430,sackOK,TS val 198912902 ecr 0,nop,wscale 8], length 0
21:22:58.193495 IP 66.249.88.59.62591 > 216.250.114.228.http: Flags [S], seq 3033941043, win 62920, options [mss 1430,sackOK,TS val 2239716846 ecr 0,nop,wscale 8], length 0
21:22:59.112125 IP 193.200.164.135.idp-infotrieve > 50.21.179.44.http: Flags [S], seq 3123960651, win 32120, options [mss 1460,sackOK,TS val 9465531 ecr 1577058304,nop,wscale 0], length 0
21:22:59.124828 IP 66.249.88.58.61608 > 216.250.114.228.http: Flags [S], seq 3204750274, win 62920, options [mss 1430,sackOK,TS val 198913902 ecr 0,nop,wscale 8], length 0
21:23:00.193657 IP 66.249.88.59.62591 > 216.250.114.228.http: Flags [S], seq 3033941043, win 62920, options [mss 1430,sackOK,TS val 2239718846 ecr 0,nop,wscale 8], length 0
21:23:00.878451 IP 193.200.164.135.63547 > 62.151.183.227.http: Flags [S], seq 959754598, win 32120, options [mss 1460,sackOK,TS val 5323616 ecr 973078528,nop,wscale 0], length 0
21:23:01.050719 IP 185.53.88.39.52241 > 74.208.169.89.http: Flags [S], seq 3834427809, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
21:23:01.126693 IP 66.249.88.58.61608 > 216.250.114.228.http: Flags [S], seq 3204750274, win 62920, options [mss 1430,sackOK,TS val 198915903 ecr 0,nop,wscale 8], length 0

I won't bore you with what's been asked on port 22...

[storm5510]

Originally Posted by chalsall

...I won't bore you with what's been asked on port 22...

I do not understand much of that nor do I want any explanation. I will simply add that I am pleased others do and leave it go there.

A theory: Some of us get P100's and other's get K80's in the Notebook. This may be nothing more than time sharing, equal allotment of resources, whatever. I am in error about my web page timing out. It happened again. It was a device disconnect in the Notebook and not a web timeout. An option to restart resulted in a K80 GPU where it had been a P100 before. I shut all of it down and logged out of Google. Logging back in later reconnected it to a P-100. All this was on my HP.

I created a second Notebook and started it on my primary i7 desktop. It decided on a P100. My HP has been running a P-100 instance as I wrote above. I suspect both will be disconnected by morning. If this ends up being the case then perhaps my theory may be partially correct, at least.

[James Heinrich]

Yes, as far as I can tell you randomly get a P100 or a K80, whatever happens to be available. I get a K80 about 80% of the time. The disconnection also appears normal, click reconnect and hit the play button and it should resume.

[storm5510]

Originally Posted by James Heinrich

Yes, as far as I can tell you randomly get a P100 or a K80, whatever happens to be available. I get a K80 about 80% of the time. The disconnection also appears normal, click reconnect and hit the play button and it should resume.

I agree that it is random, and also the amount of time a person has used a particular processor.

I realized late last night that it was the page itself I needed to keep active. After some searching, I found a Firefox extension called Session Alive which does exactly that. It asks for two things. The URL to the web page and the number of minutes to keep it going. Once supplied, a person simply starts it. Other browser most likely have something similar which can be attached.

I was sitting here early this morning when my HP disconnected at their end. The page was still alive so all I had to do was clear everything and restart it from the round icon on the left, near the top. I did not use "reconnect" in the small dialog that appeared. It went back to the P-100 which it had been running. I made a note of the time it stopped so I can get an interval of when this may happen.

============================

Edit/Update: Both my instances just shut down. Attempting to restart results in a message, "Failed to assign a backend." I'll hazard a guess and say someone is working on something somewhere.

[chalsall]

Originally Posted by storm5510

A theory: Some of us get P100's and other's get K80's in the Notebook. This may be nothing more than time sharing, equal allotment of resources, whatever.

Interesting... The P100s on Colab are a new offering, if I'm not mistaken.

For the last ~8 weeks, I've been getting only K80s. I didn't see a coveted T4 after a couple of days of running scripted mfaktc.

The last day or so I've been getting a P100 ~50% of the time.

Cool! These are the same as what Kaggle offer, constrained to ~38.9 hours a week...

[petrw1]

About one-third of the time.

[storm5510]

Originally Posted by chalsall

Interesting... The P100s on Colab are a new offering, if I'm not mistaken.

For the last ~8 weeks, I've been getting only K80s. I didn't see a coveted T4 after a couple of days of running scripted mfaktc.

The last day or so I've been getting a P100 ~50% of the time.

Cool! These are the same as what Kaggle offer, constrained to ~38.9 hours a week...

I think James Heinrich was correct in his K80 comment. The output says 393 GHz-d/Day. The progress seems to go much faster. James wrote that these are dual core GPU's. My HP has been running a K80 since noon at lower TF levels, end bits of 2⁷⁴ and 2⁷⁵.. My i7 is still running a P-100, and also running lower bit levels. Everything previous has been 2⁷⁶.

I had seen T4 in the options. I did not know what ti was until I did a search. An animation on Nvidia's web site shows a comparison to other models. It really goes!

If Google had written, "Failed to assign a backend" as "No GPU available," it would have been far better. I would have simply stopped and waited several hours instead of trying to figure out what was wrong on my end.