Google Diet Colab Notebook

[PhilF]

Originally Posted by axn

Once your commited job finished, you will be able to see all your files in the Output tab


That's exactly upside down. Only files are retained in Output tab. Screen output is lost (unless you redirect it to a file).


Yes. In the Output tab, all the files in the kaggle folder will be available once the session completes (either the code has run to completion, or session was killed after 9 hours (CPU) / 6 hrs (GPU))

That exactly opposite of my experience. Just now, in fact, I retrieved my commit V7, which was 9 curves run by gmp-ecm. ALL screen output was retained, including the complete output of my two cat commands.

EDIT TO ADD: I just discovered the output files available in the output tab as you described. However, all the screen output was saved also.

[Prime95]

Originally Posted by axn

In the Output tab, all the files in the kaggle folder will be available once the session completes (either the code has run to completion, or session was killed after 9 hours (CPU) / 6 hrs (GPU))/

Is there a way to feed those files into the next 9 hour run?

[axn]

Originally Posted by Prime95

Is there a way to feed those files into the next 9 hour run?

I haven't tried it. My use case finishes in 9 hrs. If for some reason it doesn't, I download the checkpoints and finishes off locally.

There is no obvious way to resume a committed session. In the output tab, there is a "Download all" button available. Perhaps, reverse engineering its URL and doing a wget of that within the notebook might be a (painful) way to do it?

[axn]

Originally Posted by PhilF

That exactly opposite of my experience. Just now, in fact, I retrieved my commit V7, which was 9 curves run by gmp-ecm. ALL screen output was retained, including the complete output of my two cat commands.

EDIT TO ADD: I just discovered the output files available in the output tab as you described. However, all the screen output was saved also.

Where to find the screen output?!

I close my kaggle session after all 10 batches are committed. When I come back to them, I don't see any obvious way to see the screen output.

[PhilF]

Originally Posted by axn

Where to find the screen output?!

I close my kaggle session after all 10 batches are committed. When I come back to them, I don't see any obvious way to see the screen output.

It is in the "Notebook" tab, just above the "Data" tab, which is just above the "Output" tab. It shows the original code, and below that all screen output.

[ATH]

Originally Posted by Prime95

Is there a way to feed those files into the next 9 hour run?

On the page of the finished committed kernel in the top right corner menu there is a clone or duplicate to new kernel, I have not tried that yet if it actually copies the files as well.


I have been using the Kaggle API to try and automate it.

https://www.kaggle.com/docs/api#crea...g-a-new-kernel
https://github.com/Kaggle/kaggle-api


Assuming you have python3 you install the API with:
sudo apt-get install python3-pip
pip3 install --user kaggle

It will install to ~/.local/bin/kaggle. You need to download "kaggle.json" from your account: https://www.kaggle.com/<username>/account
and place it in ~/.kaggle/

Then I have been able to download the output files from a finished kernel with:
~/.local/bin/kaggle kernels output <username>/<kernelname> -p tmpdir/

and then using "wput" to upload them to an ftp site (sudo apt-get install wput).

Inside the kernel notebooks I use wget to download the files from the ftp.


I tried to start a new kernel with the API but was not successful, have not spent much time on it yet. Once that works I think you can name kernels which makes everything much easier to automate.

[mnd9]

Today I'm attempting to commit on Kaggle and grab checkpoint/results files via the Output tab as others have suggested.

One last question: I assume the GPU quota usage for the committed sessions stops after the 6 hour limit is reached and there's no need to go in and turn it off manually?

[ATH]

It does not use GPU quota after it is stopped by itself.

But remember when you just committed the script, the "interactive" GPU session might use quota I think, if you do not turn it off. So close the sign saying it was committed and then choose "Run" - "Power Off".

Then you can check the committed version is running on:
https://www.kaggle.com/<username>/kernels

[xx005fs]

Originally Posted by Prime95

Is there a way to feed those files into the next 9 hour run?

On the top right corner of the output tab, there is a button that says "New Dataset Version", which would pop open a new window prompting you to name it a certain version number of the dataset you used in the previous file. Then as you go on to the next commit, delete the dataset from the input section by crossing the x next to the folder, and reimport the same dataset you just updated with the new version, and you are good to go.

[chalsall]

Originally Posted by xx005fs

Then as you go on to the next commit, delete the dataset from the input section by crossing the x next to the folder, and reimport the same dataset you just updated with the new version, and you are good to go.

Sweet!!!

[xx005fs]

Originally Posted by chalsall

Sweet!!!

What I am seeing recently is that updating the dataset with new versions from the output takes a long time, so make sure that it is finished before reimporting the dataset. The reimporting process will waste about 2-5 minutes of GPU quota but I think that's probably insignificant enough considering you are only doing that 4 times a week (yes ik quite a lot tbh).

After an exponent in gpuowl is say finished, I would delete everything but the executable in the dataset version. Then in the console scripts I would create a worktodo.txt file and echo the line of work into the file and work from there.