Google Diet Colab Notebook

[chalsall]

Originally Posted by James Heinrich

If Colab is truly "over", can someone volunteer to clean up the half-done exponents? I apparently have one at 91.6%, if Chris can send the checkpoint to whoever wants to finish it for me (I care nothing for credit).

I still have a single, old 560 online which I use for (slow) cleanup work. Unless anyone cares enough to ask me for the checkpoint files, I'll clear them out over the next few days. I should be able to bring a 1060 back online as well.

Just out of interest, today with an account that hasn't had a GPU backend for the last four days, I ran an experiment and Colab /did/ give me a CPU backend no problem, so we could still use Colab for some (slowish) CPU based work.

Secondly, does anyone out there have a Google account which they haven't used for the GPU72_TF Notebook in the past? An interesting experiment to run would be to see if a "virgin" account could get access, and if so, how long it lasted.

Because of the risk of being banned, I'd suggest one only do this with a "disposable" account.

[James Heinrich]

Originally Posted by chalsall

Secondly, does anyone out there have a Google account which they haven't used for the GPU72_TF Notebook in the past? An interesting experiment to run would be to see if a "virgin" account could get access, and if so, how long it lasted.

I was able to get connected to a GPU on a previously-unused account. I didn't actually try running any work, but it does look like it's account specific.

[PhilF]

Originally Posted by chalsall

Secondly, does anyone out there have a Google account which they haven't used for the GPU72_TF Notebook in the past? An interesting experiment to run would be to see if a "virgin" account could get access, and if so, how long it lasted.

Because of the risk of being banned, I'd suggest one only do this with a "disposable" account.

I use CPUs on Colab almost every day. Just for the heck of it, I just now connected, asked for a GPU, and got one immediately. I didn't use it for anything, but I did get one.

[chalsall]

Originally Posted by James Heinrich

I didn't actually try running any work, but it does look like it's account specific.

Huh... Well I'll be...

Based on this, I just created another GMail account, using the same browser context as my three previous (now non-GPU) Colab accounts, used the same phone number of SMS confirmation, etc. Trying to be as honest as I could be.

Logged into Colab using the just created account, copy-and-pasted in the GPU72_TF Notebook, and launched it. Got a P100 first try!

I'll let you all know how long it survives. But this supports George's theory that there's a "life-time GPU quota", but apparently by account.

Perhaps Colab isn't actually dead yet. Finger's crossed...

Edit: 25 minutes... "She's dead, Jim."

[kriesel]

Originally Posted by chalsall

Perhaps Colab isn't actually dead yet. Finger's crossed...

Edit: 25 minutes... "She's dead, Jim."

The P100 I got for gpuowl continuation on Colab at 2020-01-13 15:51:41 UTC is still chugging along, at least to 2020-01-13 19:56:54. That's on the first Colab account, that I've been very frequently using. (Without chalsall's tunnels etc; straight colab from browser session, with google drive use.) The other one, also used daily or more when I can get a gpu, got a K80 and is still going at 7 hr 18 min uptime. Sometimes I get 10 hours, sometimes I don't. One went to 10:47 recently.
But then again, I think some of you got started a month before I did.
FYI, I have more than 25 Colab days (24-hour days, 600 hours) of gpu time logged in benchmarking gpuowl P-1 alone, on the first account.
P100 16.8
P4 1.0
K80 7.6

[lycorn]

I keep using Colab on a daily basis (4 CPU instances) without any problem.
Two or three days ago I ran also two GPU mfaktc instances. May be because I had not use any GPU in a while (several weeks) I got two P100 straight away. But the fun only lasted for one hour or so. The accounts running only cpu instances were not affected. This is a pattern consistently observed lately: stick to cpus, and Colab will run like a charm; as soon as you touch a GPU, you're out.
So it seems that the availability of gpus is indeed a thing of the past, at least for most of us. I'll keep running mprime on 4 accounts. I have a few incomplete exponents from the gpu72 notebook that Chris may wish to finish off.

[kriesel]

(Oops. Had P100 and K80 totals switched earlier)

P10010.1
P4 1.0
K80 16.8

Nearly all of that was joint cpu & gpu sessions. There were also some cpu-only mprime sessions not included in those subtotals when I could not get a gpu for a time.

Sometimes gpu-using sessions terminate early, sometimes not. Here's the last top -d 120 output before session end, from one that ran overnight, showing a run time of just over 10 hours:

Htop - 09:20:46 up 10:02,  0 users,  load average: 1.05, 1.13, 1.10

%Cpu(s):  1.6 us,  1.7 sy, 49.9 ni, 46.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 13335180 total,  9070060 free,  1209308 used,  3055812 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 12249416 avail Mem 

    373 root      30  10  555192 408036   8848 S  99.4  3.1 516:16.59 mprime    
    365 root      20   0 12.875g 191792 145276 S   1.7  1.4   9:43.65 mfaktc.e+ 
    126 root      20   0  653984 152792  61752 S   0.5  1.1   3:05.81 python3   
     20 root      20   0  405212 101604  26672 S   0.1  0.8   0:50.33 jupyter-+ 
    305 root      20   0 1388332  68636  25152 S   0.1  0.5   0:36.99 drive     
     10 root      20   0  690324  61476  24908 S   0.0  0.5   0:18.31 node      
    118 root      20   0   35888   4880   3784 S   0.0  0.0   0:01.52 tail      
      1 root      20   0   39192   6368   4864 S   0.0  0.0   0:00.03 run.sh

Some times of day seem better for getting a gpu than others.

[kriesel]

21+ hours and running on dual instance, cpu and K80 gpu, mprime and mfaktc, top -d 120 last output:

Htop - 14:11:57 up 21:07,  0 users,  load average: 1.36, 1.13, 1.08

%Cpu(s):  1.1 us,  1.4 sy, 49.8 ni, 47.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 13335188 total,  9313584 free,  1118076 used,  2903528 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 12109812 avail Mem 


    421 root      30  10  550100 416388   7500 S  99.3  3.1 297:28.68 mprime    
    416 root      20   0 36.335g  94060  82872 S   0.7  0.7   2:08.60 mfaktc.e+ 
    127 root      20   0  653980 152760  62208 S   0.5  1.1   1:46.18 python3   
     27 root      20   0  404180 100632  26152 S   0.1  0.8   0:29.97 jupyter-+ 
    358 root      20   0 1388332  58440  25740 S   0.1  0.4   0:18.73 drive     
      9 root      20   0  691856  62776  24940 S   0.0  0.5   0:15.48 node      
    119 root      20   0   35884   4932   3840 S   0.0  0.0   0:02.87 tail      
    404 root      20   0    4568    876    812 S   0.0  0.0   0:00.66 tail      
      1 root      20   0   39192   6548   5044 S   0.0  0.0   0:00.04 run.sh

(No complaints, just have never seen it do that before.)

[chalsall]

So, yesterday Dylan managed to get a Colab session, which appears to have lasted about ten hours. Today, Storm got one; running currently.

None of my five different accounts have received a GPU backend for about a week now, nor anyone else with the exception of Wayne.

This tells me that while Colab have not blocked the GPU72_TF Notebook specifically, it does appear there is some kind of a "lifetime" GPU quote. It will be interesting to observe what happens to others doing GPU work in the future.

Very much a black box...

[petrw1]

Originally Posted by chalsall

with the exception of Wayne...

Colab: A couple days ago your Code stopped failing for me (post 788). But I haven't been able to get a GPU in those 2 days.

Kaggle: I consistently get my 30+ hours every week without fail.

I only run mfaktc via the code you provided

[axn]

Originally Posted by chalsall

None of my five different accounts have received a GPU backend for about a week now

Do you run CPU-only work when you don't get a GPU backend? If not, can you (on the theory that this will help the Google overmind unlearn your usage pattern)?