mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Cloud Computing (https://www.mersenneforum.org/forumdisplay.php?f=134)
-   -   Google Diet Colab Notebook (https://www.mersenneforum.org/showthread.php?t=24646)

kriesel 2019-10-29 15:33

[QUOTE=kriesel;528911]am no longer trying P-1 since the selftest failed.
[/QUOTE]That was in regard to CUDAPm1 v0.20.
Currently I'm running gpuowl (Fan Ming's executable), P-1 on Colab (Tesla K80). Its selftest passed, finding the known factor.

ATH 2019-10-29 17:22

Anyone tried using 2 accounts on Kaggle? Running 2x 30 hours GPU per week and up to 20 CPU instances?

I'm tempted to do it since I scripted the restarting, but I do not want to be banned.

xx005fs 2019-10-29 23:21

[QUOTE=ATH;529208]Anyone tried using 2 accounts on Kaggle? Running 2x 30 hours GPU per week and up to 20 CPU instances?

I'm tempted to do it since I scripted the restarting, but I do not want to be banned.[/QUOTE]

Irrelevant to the question here but how do you manage 10 CPU instances? With a script or something? I am interested to use them to crank out some cofactor or ECM tests.

For the two accounts thing, I believe you need a phone number to access GPU so you would need multiple phone numbers and obv email accounts.

ATH 2019-10-30 06:28

Yeah, I forgot about the extra phone number needed. :sad:

I started all 10 instances manually the first time, and then I use Kaggle API to download files after it is done and restart the instance, as I described in post #424 and #443.

I have an ftp I use to download the files inside the notebook scripts and I upload the worktodo.txt and temporary files afterwards.

The notebook scripts are:

[CODE]import os
%cd /kaggle/working
!wget ftp://<ftp address>/kaggle/mprime.tar.gz
!tar -xzvf mprime.tar.gz
!wget ftp://<ftp address>/kaggle/kaggle1/*
!tar -xzvf files.tar.gz
!chmod 755 *
!./mprime -d >> output1.txt
[/CODE]

The mprime.tar.gz is just the mprime program and I have 10 folders kaggle1 - kaggle10 with files.tar.gz, these files contains the individual prime.txt, local.txt, worktodo.txt, output.txt and the mprime temporary files.

In post #443 I explain how to rename the instance the first time, so my instances are called kaggle.com/<username>/mprime1 (to mprime10).
When they are done I use this Linux script to download and upload files and restart the instance:
[CODE]#!/bin/sh
cd tmp
rm *
cd tmp
rm *
cd ~/kaggle/
~/.local/bin/kaggle kernels output <username>/mprime$1 -p tmp/
cd tmp
mv output$1.txt tmp/
mv worktodo.txt tmp/
mv results.txt tmp/
mv prime.log tmp/
mv prime.txt tmp/
mv local.txt tmp/
rm files.tar.gz
rm p*.bu*
rm prime*.*
mv p* tmp/
cd tmp
tar -czvf ~/kaggle/tmp/files.tar.gz *
cd ..
wput -u -nc files.tar.gz ftp://<ftp address>/kaggle/kaggle$1/files.tar.gz
cd ..
~/.local/bin/kaggle kernels push -p ~/kaggle/$1
[/CODE]

Then I just run this script 10 times from another script with numbers 1 to 10 as input.

Edit:
The last line is where it restarts the instance: ~/.local/bin/kaggle kernels push -p ~/kaggle/$1

In the kaggle folder I have folders 1 to 10, and each folder contains mprime$1.ipynb and kernel-metadata.json, see post #443.

kriesel 2019-11-01 19:07

[QUOTE=Dylan14;529113]You need to have gmp installed on the instance first. Either you can compile it from source, or you can run

[CODE]!apt install libgmp-dev[/CODE]and then you should have gmp.h in the place gpuowl expects.[/QUOTE]
Thanks. That got closer to the goal, but only a little.
[CODE]echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT GmpUtil.o -MMD -MP -MF .d/GmpUtil.Td -Wall -O2 -std=c++17 -c -o GmpUtil.o GmpUtil.cpp
g++ -MT Worktodo.o -MMD -MP -MF .d/Worktodo.Td -Wall -O2 -std=c++17 -c -o Worktodo.o Worktodo.cpp
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17 -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
#include <filesystem>
^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1[/CODE]

xx005fs 2019-11-01 22:14

[QUOTE=kriesel;529426]Thanks. That got closer to the goal, but only a little.
[CODE]echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT GmpUtil.o -MMD -MP -MF .d/GmpUtil.Td -Wall -O2 -std=c++17 -c -o GmpUtil.o GmpUtil.cpp
g++ -MT Worktodo.o -MMD -MP -MF .d/Worktodo.Td -Wall -O2 -std=c++17 -c -o Worktodo.o Worktodo.cpp
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17 -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
#include <filesystem>
^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1[/CODE][/QUOTE]

Looks like an older GCC/G++ version to me. I had the same issue until I updated my GCC & G++ to version 7 or above (I'm currently running gcc-9). I didn't compile mine on colab or kaggle, instead, I compiled it on my own Linux machine with ROCm on it, so I don't really know how to fix that on either cloud platforms.

I have changed it to #include <experimental-filesystem> before, which resolves this specific issue but others pop out.

Dylan14 2019-11-02 01:44

[QUOTE=kriesel;529426]Thanks. That got closer to the goal, but only a little.
[CODE]echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT GmpUtil.o -MMD -MP -MF .d/GmpUtil.Td -Wall -O2 -std=c++17 -c -o GmpUtil.o GmpUtil.cpp
g++ -MT Worktodo.o -MMD -MP -MF .d/Worktodo.Td -Wall -O2 -std=c++17 -c -o Worktodo.o Worktodo.cpp
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17 -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
#include <filesystem>
^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1[/CODE][/QUOTE]

I had the same issue. You will need to have at least gcc version 8 for this to work. Luckily, gcc-8 is available in the Ubuntu repo:

[CODE]!apt install gcc-8[/CODE]

Alternatively, you may replace #include <filesystem> with

[CODE]#include <experimental/filesystem>[/CODE]

in the gpuowl files, and link with

[CODE]-lstdc++fs[/CODE]
It might be a good idea to let preda know this so that he can make the appropriate changes.

kracker 2019-11-02 03:07

Just got my account on Kaggle blocked - Not sure if it's worth making a new account or contacting support(doubt I'll get far with the latter).

kriesel 2019-11-02 13:06

[QUOTE=Dylan14;529446]I had the same issue. You will need to have at least gcc version 8 for this to work. Luckily, gcc-8 is available in the Ubuntu repo:

[CODE]!apt install gcc-8[/CODE]Alternatively, you may replace #include <filesystem> with

[CODE]#include <experimental/filesystem>[/CODE]in the gpuowl files, and link with

[CODE]-lstdc++fs[/CODE]It might be a good idea to let preda know this so that he can make the appropriate changes.[/QUOTE]
First approach, install gcc-8, does not change the outcome on the Colab compile session:[CODE]...Setting up libgcc-8-dev:amd64 (8.3.0-6ubuntu1~18.04.1) ...
Setting up gcc-8 (8.3.0-6ubuntu1~18.04.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17 -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
#include <filesystem>
^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1[/CODE]PM with link sent to Preda.

preda 2019-11-02 13:49

did you check the g++ version,
g++ --version

I think that a reasonably recent version of gcc/g++ (8 probably) should find the #include<filesystem>. (I don't have another fix for that).

I didn't build myself on collab, so I don't have first-hand experience there. I'll review pull-request on github with fixes.

[QUOTE=kriesel;529465]First approach, install gcc-8, does not change the outcome on the Colab compile session:[CODE]...Setting up libgcc-8-dev:amd64 (8.3.0-6ubuntu1~18.04.1) ...
Setting up gcc-8 (8.3.0-6ubuntu1~18.04.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17 -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
#include <filesystem>
^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1[/CODE]PM with link sent to Preda.[/QUOTE]

kriesel 2019-11-02 18:50

[QUOTE=preda;529470]did you check the g++ version,
g++ --version

I think that a reasonably recent version of gcc/g++ (8 probably) should find the #include<filesystem>. (I don't have another fix for that).

I didn't build myself on collab, so I don't have first-hand experience there. I'll review pull-request on github with fixes.[/QUOTE][CODE]g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
[/CODE]So, anticipating maybe the next recommendation would be,
!apt install g++-8
did that, got [CODE]Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libstdc++-8-dev
Suggested packages:
g++-8-multilib gcc-8-doc libstdc++6-8-dbg libstdc++-8-doc
The following NEW packages will be installed:
g++-8 libstdc++-8-dev
0 upgraded, 2 newly installed, 0 to remove and 28 not upgraded.
Need to get 11.6 MB of archives.
After this operation, 45.2 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 libstdc++-8-dev amd64 8.3.0-6ubuntu1~18.04.1 [1,543 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 g++-8 amd64 8.3.0-6ubuntu1~18.04.1 [10.1 MB]
Fetched 11.6 MB in 3s (3,461 kB/s)
Selecting previously unselected package libstdc++-8-dev:amd64.
(Reading database ... 132925 files and directories currently installed.)
Preparing to unpack .../libstdc++-8-dev_8.3.0-6ubuntu1~18.04.1_amd64.deb ...
Unpacking libstdc++-8-dev:amd64 (8.3.0-6ubuntu1~18.04.1) ...
Selecting previously unselected package g++-8.
Preparing to unpack .../g++-8_8.3.0-6ubuntu1~18.04.1_amd64.deb ...
Unpacking g++-8 (8.3.0-6ubuntu1~18.04.1) ...
Setting up libstdc++-8-dev:amd64 (8.3.0-6ubuntu1~18.04.1) ...
Setting up g++-8 (8.3.0-6ubuntu1~18.04.1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-11-gfaaa2f2-dirty"
g++ -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17 -c -o common.o common.cpp
common.cpp:14:10: fatal error: filesystem: No such file or directory
#include <filesystem>
^~~~~~~~~~~~
compilation terminated.
Makefile:30: recipe for target 'common.o' failed
make: *** [common.o] Error 1[/CODE]


All times are UTC. The time now is 23:01.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.