Google Diet Colab Notebook

[kriesel]

Originally Posted by kriesel

am no longer trying P-1 since the selftest failed.

That was in regard to CUDAPm1 v0.20.
Currently I'm running gpuowl (Fan Ming's executable), P-1 on Colab (Tesla K80). Its selftest passed, finding the known factor.

[ATH]

Anyone tried using 2 accounts on Kaggle? Running 2x 30 hours GPU per week and up to 20 CPU instances?

I'm tempted to do it since I scripted the restarting, but I do not want to be banned.

[xx005fs]

Originally Posted by ATH

Anyone tried using 2 accounts on Kaggle? Running 2x 30 hours GPU per week and up to 20 CPU instances?

I'm tempted to do it since I scripted the restarting, but I do not want to be banned.

Irrelevant to the question here but how do you manage 10 CPU instances? With a script or something? I am interested to use them to crank out some cofactor or ECM tests.

For the two accounts thing, I believe you need a phone number to access GPU so you would need multiple phone numbers and obv email accounts.

[ATH]

Yeah, I forgot about the extra phone number needed.

I started all 10 instances manually the first time, and then I use Kaggle API to download files after it is done and restart the instance, as I described in post #424 and #443.

I have an ftp I use to download the files inside the notebook scripts and I upload the worktodo.txt and temporary files afterwards.

The notebook scripts are:

import os
%cd /kaggle/working
!wget ftp://<ftp address>/kaggle/mprime.tar.gz
!tar -xzvf mprime.tar.gz
!wget ftp://<ftp address>/kaggle/kaggle1/*
!tar -xzvf files.tar.gz
!chmod 755 *
!./mprime -d >> output1.txt

The mprime.tar.gz is just the mprime program and I have 10 folders kaggle1 - kaggle10 with files.tar.gz, these files contains the individual prime.txt, local.txt, worktodo.txt, output.txt and the mprime temporary files.

In post #443 I explain how to rename the instance the first time, so my instances are called kaggle.com/<username>/mprime1 (to mprime10).
When they are done I use this Linux script to download and upload files and restart the instance:

#!/bin/sh
cd tmp
rm *
cd tmp
rm *
cd ~/kaggle/
~/.local/bin/kaggle kernels output <username>/mprime$1 -p tmp/
cd tmp
mv output$1.txt tmp/
mv worktodo.txt tmp/
mv results.txt tmp/
mv prime.log tmp/
mv prime.txt tmp/
mv local.txt tmp/
rm files.tar.gz
rm p*.bu*
rm prime*.*
mv p* tmp/
cd tmp
tar -czvf ~/kaggle/tmp/files.tar.gz *
cd ..
wput -u -nc files.tar.gz ftp://<ftp address>/kaggle/kaggle$1/files.tar.gz
cd ..
~/.local/bin/kaggle kernels push -p ~/kaggle/$1

Then I just run this script 10 times from another script with numbers 1 to 10 as input.

Edit:
The last line is where it restarts the instance: ~/.local/bin/kaggle kernels push -p ~/kaggle/$1

In the kaggle folder I have folders 1 to 10, and each folder contains mprime$1.ipynb and kernel-metadata.json, see post #443.

[kriesel]

Originally Posted by Dylan14

You need to have gmp installed on the instance first. Either you can compile it from source, or you can run

!apt install libgmp-dev

and then you should have gmp.h in the place gpuowl expects.

Thanks. That got closer to the goal, but only a little.

echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT GmpUtil.o -MMD -MP -MF .d/GmpUtil.Td -Wall -O2 -std=c++17   -c -o GmpUtil.o GmpUtil.cpp
g++ -MT Worktodo.o -MMD -MP -MF .d/Worktodo.Td -Wall -O2 -std=c++17   -c -o Worktodo.o Worktodo.cpp
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17   -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
 #include <filesystem>
          ^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1

[xx005fs]

Originally Posted by kriesel

Thanks. That got closer to the goal, but only a little.

echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT GmpUtil.o -MMD -MP -MF .d/GmpUtil.Td -Wall -O2 -std=c++17   -c -o GmpUtil.o GmpUtil.cpp
g++ -MT Worktodo.o -MMD -MP -MF .d/Worktodo.Td -Wall -O2 -std=c++17   -c -o Worktodo.o Worktodo.cpp
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17   -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
 #include <filesystem>
          ^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1

Looks like an older GCC/G++ version to me. I had the same issue until I updated my GCC & G++ to version 7 or above (I'm currently running gcc-9). I didn't compile mine on colab or kaggle, instead, I compiled it on my own Linux machine with ROCm on it, so I don't really know how to fix that on either cloud platforms.

I have changed it to #include <experimental-filesystem> before, which resolves this specific issue but others pop out.

[Dylan14]

Originally Posted by kriesel

Thanks. That got closer to the goal, but only a little.

echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT GmpUtil.o -MMD -MP -MF .d/GmpUtil.Td -Wall -O2 -std=c++17   -c -o GmpUtil.o GmpUtil.cpp
g++ -MT Worktodo.o -MMD -MP -MF .d/Worktodo.Td -Wall -O2 -std=c++17   -c -o Worktodo.o Worktodo.cpp
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17   -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
 #include <filesystem>
          ^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1

I had the same issue. You will need to have at least gcc version 8 for this to work. Luckily, gcc-8 is available in the Ubuntu repo:

!apt install gcc-8

Alternatively, you may replace #include <filesystem> with

#include <experimental/filesystem>

in the gpuowl files, and link with

-lstdc++fs

It might be a good idea to let preda know this so that he can make the appropriate changes.

[kracker]

Just got my account on Kaggle blocked - Not sure if it's worth making a new account or contacting support(doubt I'll get far with the latter).

[kriesel]

Originally Posted by Dylan14

I had the same issue. You will need to have at least gcc version 8 for this to work. Luckily, gcc-8 is available in the Ubuntu repo:

!apt install gcc-8

Alternatively, you may replace #include <filesystem> with

#include <experimental/filesystem>

in the gpuowl files, and link with

-lstdc++fs

It might be a good idea to let preda know this so that he can make the appropriate changes.

First approach, install gcc-8, does not change the outcome on the Colab compile session:

...Setting up libgcc-8-dev:amd64 (8.3.0-6ubuntu1~18.04.1) ...
Setting up gcc-8 (8.3.0-6ubuntu1~18.04.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17   -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
 #include <filesystem>
          ^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1

PM with link sent to Preda.

[preda]

did you check the g++ version,
g++ --version

I think that a reasonably recent version of gcc/g++ (8 probably) should find the #include<filesystem>. (I don't have another fix for that).

I didn't build myself on collab, so I don't have first-hand experience there. I'll review pull-request on github with fixes.

Originally Posted by kriesel

First approach, install gcc-8, does not change the outcome on the Colab compile session:

...Setting up libgcc-8-dev:amd64 (8.3.0-6ubuntu1~18.04.1) ...
Setting up gcc-8 (8.3.0-6ubuntu1~18.04.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17   -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
 #include <filesystem>
          ^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1

PM with link sent to Preda.

[kriesel]

Originally Posted by preda

did you check the g++ version,
g++ --version

I think that a reasonably recent version of gcc/g++ (8 probably) should find the #include<filesystem>. (I don't have another fix for that).

I didn't build myself on collab, so I don't have first-hand experience there. I'll review pull-request on github with fixes.

g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

So, anticipating maybe the next recommendation would be,
!apt install g++-8
did that, got

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  libstdc++-8-dev
Suggested packages:
  g++-8-multilib gcc-8-doc libstdc++6-8-dbg libstdc++-8-doc
The following NEW packages will be installed:
  g++-8 libstdc++-8-dev
0 upgraded, 2 newly installed, 0 to remove and 28 not upgraded.
Need to get 11.6 MB of archives.
After this operation, 45.2 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libstdc++-8-dev amd64 8.3.0-6ubuntu1~18.04.1 [1,543 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 g++-8 amd64 8.3.0-6ubuntu1~18.04.1 [10.1 MB]
Fetched 11.6 MB in 3s (3,461 kB/s)
Selecting previously unselected package libstdc++-8-dev:amd64.
(Reading database ... 132925 files and directories currently installed.)
Preparing to unpack .../libstdc++-8-dev_8.3.0-6ubuntu1~18.04.1_amd64.deb ...
Unpacking libstdc++-8-dev:amd64 (8.3.0-6ubuntu1~18.04.1) ...
Selecting previously unselected package g++-8.
Preparing to unpack .../g++-8_8.3.0-6ubuntu1~18.04.1_amd64.deb ...
Unpacking g++-8 (8.3.0-6ubuntu1~18.04.1) ...
Setting up libstdc++-8-dev:amd64 (8.3.0-6ubuntu1~18.04.1) ...
Setting up g++-8 (8.3.0-6ubuntu1~18.04.1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17   -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
 #include <filesystem>
          ^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1