Google Diet Colab Notebook

[kriesel]

Originally Posted by PhilF

There's a typo in that script. This line, about the 4th line down, has a missing leading slash:

%cd 'content/drive/My Drive/'

Try %cd '/content/drive/My Drive/'

Thanks. There's more to find.
Notebook script modified, with some echoes added to show progress including around errors:

#Notebook to run mprime on a Colab thing
import os.path
from google.colab import drive
import sys
print (sys.path)
if not os.path.exists('/content/drive/My Drive'):
  drive.mount('/content/drive')

!echo marker1 cd /content/drive/My Drive/ next
!cd '/content/drive/My Drive/'
!echo mark2 past cd to google drive
!ls -l
!chmod +w '/content/drive/My Drive'
if not os.path.exists('/content/drive/My Drive/mprime/'):
  !mkdir mprime
!echo mark3 md mprime
!cd '/content/drive/My Drive/mprime/'
! ls -l
#fetch mprime executable if we don't have it
if not os.path.exists('mprime'):
  !wget http://www.mersenne.org/ftp_root/gimps/p95v298b6.linux64.tar.gz
  !tar -zxvf p95v298b6.linux64.tar.gz

!echo mark4 retrieve mprime
#!ls
#run mprime
#first, create local.txt and prime.txt if they don't already exist
if not os.path.exists('prime.txt'):
  !echo V24OptionsConverted=1 > prime.txt
  !echo WGUID_version=2 >> prime.txt
  !echo StressTester=0 >> prime.txt
  !echo UsePrimenet=1 >> prime.txt
  !echo DialUp=0 >> prime.txt
  #change the user ID to your own or use ANOYNOMUS to work anonymously
  !echo V5UserID=Kriesel >> prime.txt
  !echo Priority=1 >> prime.txt
  #Since Drive is persistant, can set DaysOfWork as desired:
  !echo DaysOfWork=1 >> prime.txt
  #This comes from undoc.txt.
  !echo MaxExponents=1 >> prime.txt
  !echo RunOnBattery=1 >> prime.txt
  # (see http://v5.mersenne.org/v5design/v5webAPI_0.97.html, 7.3 GIMPS Work Preferences)
  # see also https://www.mersenneforum.org/showpost.php?p=505770&postcount=1 or prime95/mprime source code
  # 0 whatever makes sense (server decides)
  # 1 trial factoring LMH, not recommended for cpus, leave it to the much faster gpus
  # 2 trial factoring LMH, not recommended for cpus, leave it to the much faster gpus
  # 3 P-1 factoring small
  # 4 optimal P-1 factoring, large
  # 5 ECM factoring, smallish Mersennes
  # 6 factoring Fermat ECM
  # 7 factoring Cunningham ECM
  # 8-99 reserved
  # 100 LL first time test
  # 101 LL Double check
  # 102 LL test world-record
  # 103 LL test 10M digits (no longer relevant since even DC wavefront is 48M)
  # 104 LL test 100M digits
  # 105 LL first time test with no trial or P-1 factoring
  #106-149 reserved
  # 150 PRP first time test
  # 151 PRP double check
  # 152 PRP world record
  # 153 PRP 100M digit test
  # 154-159 reserved
  # 160 PRP cofactor test
  # 161 PRP cofactor double check
  #162-255 reserved
  #The next line sets the work preference. In this case it's set to 150, PRP first time test
  !echo WorkPreference=150 >> prime.txt
  !echo [PrimeNet] >> prime.txt
  !echo Debug=0 >> prime.txt
  !echo ProxyHost= >> prime.txt
if not os.path.exists('local.txt'):
  !echo WorkerThreads=1 >> local.txt
  !echo CoresPerTest=2 >> local.txt
  !echo ComputerID=colab >> local.txt
  !echo Memory=8192 during 7:30-23:30 else 8192 >> local.txt
#now run
!chmod +x mprime
!echo cat prime.txt:
!cat prime.txt
!echo
!echo cat local.txt:
!cat local.txt
!echo run ./mprime
 !./mprime

Resulting output seen in browser:

['', '/env/python', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '/usr/local/lib/python3.6/dist-packages/IPython/extensions', '/root/.ipython']
Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code

Enter your authorization code:
··········
Mounted at /content/drive
marker1 cd /content/drive/My Drive/ next
mark2 past cd to google drive
total 8
drwx------ 3 root root 4096 Oct 13 15:33 drive
drwxr-xr-x 1 root root 4096 Aug 27 16:17 sample_data
mark3 md mprime
/bin/bash: line 0: cd: /content/drive/My Drive/mprime/: No such file or directory
total 12
drwx------ 3 root root 4096 Oct 13 15:33 drive
drwxr-xr-x 2 root root 4096 Oct 13 15:33 mprime
drwxr-xr-x 1 root root 4096 Aug 27 16:17 sample_data
mark4 retrieve mprime
cat prime.txt:
V24OptionsConverted=1
WGUID_version=2
StressTester=0
UsePrimenet=1
DialUp=0
V5UserID=Kriesel
Priority=1
DaysOfWork=1
MaxExponents=1
RunOnBattery=1
WorkPreference=150
[PrimeNet]
Debug=0
ProxyHost=

cat local.txt:
WorkerThreads=1
CoresPerTest=2
ComputerID=colab
Memory=8192 during 7:30-23:30 else 8192
run ./mprime
/bin/bash: ./mprime: Is a directory

Google drive unaffected. CPU list unaffected.
ls -l output above is not consistent with changing directory to the Google drive, which has an mfaktc folder but not an mprime folder.

[PhilF]

Now that you have the mprime directory on your Google Drive, change to it and manually install mprime, since this only needs to be done once:

!wget http://www.mersenne.org/ftp_root/gim...linux64.tar.gz
!tar -zxvf p95v298b6.linux64.tar.gz

What I do at that point is manually edit prime.txt, worktodo.txt, etc, etc, on my Google Drive, rather than use the !echo method.

Then, when I want to run mprime, I connect and copy/paste this into the code line, all at once, then press control-Enter:

import os.path
from google.colab import drive

if not os.path.exists('/content/drive/My Drive'):
drive.mount('/content/drive')

%cd '/content/drive/My Drive/mprime//'

Once you then enter your Google Drive authentication, make mprime executable:

!chmod 700 mprime

Now you're ready to fly:

!./mprime -d

[Dylan14]

I have modified the boinc script. Now, you can specify the project and account key interactively. This script also uses the subprocess.run command in python, which cleans up stdout:

#@title BOINC test
import os.path
import subprocess
#Use apt-get to get boinc
!apt-get update
!apt-get install boinc boinc-client
#cp boinc, boinccmd to working directory
!cp /usr/bin/boinc /content
!cp /usr/bin/boinccmd /content
#create a slots directory if it doesn't exist(otherwise boinc doesn't work)
if not os.path.exists('/content/slots'):
  !mkdir slots
#launch the client
#attach to a project as desired
project = input("Enter a URL of a BOINC project: ")
acct_key = input("Enter your account key: ")
if not os.path.exists('/content/slots/0'):
  subprocess.run(['boinc', "--attach_project", project, acct_key])
else:
  subprocess.run('boinc')

[chalsall]

Originally Posted by Dylan14

Now, you can specify the project and account key interactively.

Very nice. Much cleaner than a Colab "Form". Do you know if this works on Kaggle as well (I don't have a session open at the moment to test)?

[kriesel]

wget and tar were not ever happening, because the test was finding the empty directory mprime. Revised code:

#Notebook to run mprime on a Colab thing
import os.path
from google.colab import drive
import sys
print (sys.path)
if not os.path.exists('/content/drive/My Drive'):
  drive.mount('/content/drive')

!echo marker1 cd /content/drive/My Drive/ next
!cd '/content/drive/My Drive/'
!echo marker2 past cd to google drive
!ls -l
!chmod +w '/content/drive/My Drive'
if not os.path.exists('/content/drive/My Drive/mprime/'):
  !mkdir mprime
!echo marker3 md mprime
!cd '/content/drive/My Drive/mprime/'
!ls -l
#fetch mprime executable if we don't have it
if not os.path.exists('/content/drive/My Drive/mprime/mprime'):
  !wget http://www.mersenne.org/ftp_root/gimps/p95v298b6.linux64.tar.gz
  !tar -zxvf p95v298b6.linux64.tar.gz

!echo marker4 past retrieve mprime
!ls -l
!echo ls -l /content/drive/My Drive/mprime/'
!ls -l /content/drive/My Drive/mprime/'
#next, create local.txt and prime.txt if they don't already exist
if not os.path.exists('prime.txt'):
  !echo V24OptionsConverted=1 > prime.txt
  !echo WGUID_version=2 >> prime.txt
  !echo StressTester=0 >> prime.txt
  !echo UsePrimenet=1 >> prime.txt
  !echo DialUp=0 >> prime.txt
  #change the user ID to your own or use ANOYNOMUS to work anonymously
  !echo V5UserID=Kriesel >> prime.txt
  !echo Priority=1 >> prime.txt
  #Since Drive is persistant, can set DaysOfWork as desired:
  !echo DaysOfWork=1 >> prime.txt
  #This comes from undoc.txt.
  !echo MaxExponents=1 >> prime.txt
  !echo RunOnBattery=1 >> prime.txt
  # (see http://v5.mersenne.org/v5design/v5webAPI_0.97.html, 7.3 GIMPS Work Preferences)
  # see also https://www.mersenneforum.org/showpost.php?p=505770&postcount=1 or prime95/mprime source code
  # 0 whatever makes sense (server decides)
  # 1 trial factoring LMH, not recommended for cpus, leave it to the much faster gpus
  # 2 trial factoring LMH, not recommended for cpus, leave it to the much faster gpus
  # 3 P-1 factoring small
  # 4 optimal P-1 factoring, large
  # 5 ECM factoring, smallish Mersennes
  # 6 factoring Fermat ECM
  # 7 factoring Cunningham ECM
  # 8-99 reserved
  # 100 LL first time test
  # 101 LL Double check
  # 102 LL test world-record
  # 103 LL test 10M digits (no longer relevant since even DC wavefront is 48M)
  # 104 LL test 100M digits
  # 105 LL first time test with no trial or P-1 factoring
  #106-149 reserved
  # 150 PRP first time test
  # 151 PRP double check
  # 152 PRP world record
  # 153 PRP 100M digit test
  # 154-159 reserved
  # 160 PRP cofactor test
  # 161 PRP cofactor double check
  #162-255 reserved
  #The next line sets the work preference. In this case it's set to 150, PRP first time test
  !echo WorkPreference=150 >> prime.txt
  !echo [PrimeNet] >> prime.txt
  !echo Debug=0 >> prime.txt
  !echo ProxyHost= >> prime.txt
if not os.path.exists('local.txt'):
  !echo WorkerThreads=1 >> local.txt
  !echo CoresPerTest=2 >> local.txt
  !echo ComputerID=colab >> local.txt
  !echo Memory=8192 during 7:30-23:30 else 8192 >> local.txt
#now run
!chmod +x mprime
!echo cat prime.txt:
!cat prime.txt
!echo
!echo cat local.txt:
!cat local.txt
!echo run ./mprime
!./mprime

Still appears to be not affecting the google drive. There is a colab cpu entry now, so some progress. It appears to be running mprime, on the VM drive, so presumably the checkpoint file will be lost when it terminates.

['', '/env/python', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages', '/usr/local/lib/python3.6/dist-packages/IPython/extensions', '/root/.ipython']
marker1 cd /content/drive/My Drive/ next
marker2 past cd to google drive
total 20
drwx------ 3 root root 4096 Oct 13 15:33 drive
-rw-r--r-- 1 root root   88 Oct 13 15:33 local.txt
drwxr-xr-x 2 root root 4096 Oct 13 15:33 mprime
-rw-r--r-- 1 root root  196 Oct 13 15:33 prime.txt
drwxr-xr-x 1 root root 4096 Aug 27 16:17 sample_data
mkdir: cannot create directory ‘mprime’: File exists
marker3 md mprime
/bin/bash: line 0: cd: /content/drive/My Drive/mprime/: No such file or directory
total 20
drwx------ 3 root root 4096 Oct 13 15:33 drive
-rw-r--r-- 1 root root   88 Oct 13 15:33 local.txt
drwxr-xr-x 2 root root 4096 Oct 13 15:33 mprime
-rw-r--r-- 1 root root  196 Oct 13 15:33 prime.txt
drwxr-xr-x 1 root root 4096 Aug 27 16:17 sample_data
--2019-10-13 15:53:11--  http://www.mersenne.org/ftp_root/gimps/p95v298b6.linux64.tar.gz
Resolving www.mersenne.org (www.mersenne.org)... 162.212.57.131
Connecting to www.mersenne.org (www.mersenne.org)|162.212.57.131|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7032597 (6.7M) [application/x-gzip]
Saving to: ‘p95v298b6.linux64.tar.gz’

p95v298b6.linux64.t 100%[===================>]   6.71M  3.72MB/s    in 1.8s    

2019-10-13 15:53:14 (3.72 MB/s) - ‘p95v298b6.linux64.tar.gz’ saved [7032597/7032597]

libgmp.so
libgmp.so.10
libgmp.so.10.3.2
license.txt
mprime
readme.txt
stress.txt
undoc.txt
whatsnew.txt
marker4 past retrieve mprime
total 44516
drwx------ 3 root root     4096 Oct 13 15:33 drive
lrwxrwxrwx 1  500  500       16 Aug 18 19:59 libgmp.so -> libgmp.so.10.3.2
lrwxrwxrwx 1  500  500       16 Aug 18 19:59 libgmp.so.10 -> libgmp.so.10.3.2
-rwxr-xr-x 1  500  500   644436 Aug 18 19:59 libgmp.so.10.3.2
-rw-r--r-- 1  500  500     2110 Aug 18 19:59 license.txt
-rw-r--r-- 1 root root       88 Oct 13 15:33 local.txt
-rwxrwxr-x 1  500  500 37748900 Aug 18 19:59 mprime
-rw-r--r-- 1 root root  7032597 Aug 18 20:03 p95v298b6.linux64.tar.gz
-rw-r--r-- 1 root root      196 Oct 13 15:33 prime.txt
-rw-r--r-- 1  500  500    20019 Aug 18 19:59 readme.txt
drwxr-xr-x 1 root root     4096 Aug 27 16:17 sample_data
-rw-r--r-- 1  500  500     9134 Aug 18 19:59 stress.txt
-rw-r--r-- 1  500  500    37331 Aug 18 19:59 undoc.txt
-rw-r--r-- 1  500  500    56172 Aug 18 19:59 whatsnew.txt
/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
/bin/bash: -c: line 1: syntax error: unexpected end of file
/bin/bash: -c: line 0: unexpected EOF while looking for matching `''
/bin/bash: -c: line 1: syntax error: unexpected end of file
cat prime.txt:
V24OptionsConverted=1
WGUID_version=2
StressTester=0
UsePrimenet=1
DialUp=0
V5UserID=Kriesel
Priority=1
DaysOfWork=1
MaxExponents=1
RunOnBattery=1
WorkPreference=150
[PrimeNet]
Debug=0
ProxyHost=

cat local.txt:
WorkerThreads=1
CoresPerTest=2
ComputerID=colab
Memory=8192 during 7:30-23:30 else 8192
run ./mprime

I tried +code with !top but got no output from it unless I first interrupted the main code block, which makes top output irrelevant. The same would apply to nvidia-smi. It looks like only one code block can be run at a time, so monitoring progress is not easy. That will also complicate running separate applications to use cpu and gpu at the same time on the same VM.

Sun Oct 13 16:06:47 2019        +-----------------------------------------------------------------------------+ | NVIDIA-SMI 430.40       Driver Version: 418.67       CUDA Version: 10.1     | |-------------------------------+----------------------+----------------------+ | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC | | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. | |===============================+======================+======================| |   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 | | N/A   49C    P8    30W / 149W |      0MiB / 11441MiB |      0%      Default | +-------------------------------+----------------------+----------------------+                                                                                 +-----------------------------------------------------------------------------+ | Processes:                                                       GPU Memory | |  GPU       PID   Type   Process name                             Usage      | |=============================================================================| |  No running processes found                                                 | +-----------------------------------------------------------------------------+

[kriesel]

Originally Posted by PhilF

Now that you have the mprime directory on your Google Drive, change to it and manually install mprime, since this only needs to be done once:

!wget http://www.mersenne.org/ftp_root/gim...linux64.tar.gz
!tar -zxvf p95v298b6.linux64.tar.gz

What I do at that point is manually edit prime.txt, worktodo.txt, etc, etc, on my Google Drive, rather than use the !echo method.

Then, when I want to run mprime, I connect and copy/paste this into the code line, all at once, then press control-Enter:



Once you then enter your Google Drive authentication, make mprime executable:

!chmod 700 mprime

Now you're ready to fly:

!./mprime -d

The cd to the google drive is silently failing. So wget, tar, ls -l, ./mprime all appear to be using the VM drive. The google drive folder mprime remains empty. Work in progress will be lost when the session terminates.


Ok, weird syntax there, in %cd '/content/drive/My Drive/mprime//' is that Python syntax? Now wget and tar are affecting google drive content, so mprime may also.

[Dylan14]

Originally Posted by chalsall

Very nice. Much cleaner than a Colab "Form". Do you know if this works on Kaggle as well (I don't have a session open at the moment to test)?

As it stands right now, no, it does not. My guess as to why may have to do with the fact that on Debian (which Kaggle uses), the BOINC version available is 7.6.33, whereas on Ubuntu (which Colab uses), the version available is 7.9.3.
So either I need to find a different repository for BOINC on Kaggle, or I would need to compile from source.

[PhilF]

It looks like you installed mprime in /content. I see the "drive" directory in your ls -l output.

After getting authenticated, you should be able to cd to your Google drive manually:

%cd /content/drive/My Drive

If that works, that is your Google Drive. Use !ls -l to see if your mprime directory is there. If it isn't, use !mkdir to create it. Then %cd to it, and execute the wget and tar from there.

You can always check your current working directory with !pwd

[xx005fs]

Originally Posted by chalsall

Yeah, that would very likely be flagged as a "crack attempt" by the AIs.

Just wondering how you intended to install and then run the custom kernel? You can't "shutdown -r now" within the instances.

Don't know exactly how it works but I was trying out to run OpenCL on the Nvidia GPU on kaggle, but unfortunately the clinfo shows number of platforms being 0 and that means OpenCL is not properly installed. Later I goes on to install the nvidia-opencl-icd, which somehow requires to reinstall some components of the drivers no matter if I select the 418 version (which is what's currently installed on kaggle), or the 430 version which is the newest available. The first installation would always fail and it would lead to a dpkg problem, but then when I reinstall the same package again the issue is immediately resolved, and it goes to completion. During the process of installation, it seems to try to change something in the kernel and is rebuilding it, but I am not sure about the exact process. But after installation the OpenCL device detection works immediately and the driver version shown by nvidia-smi is still unchanged, so idk if it actually changed anything within the driver stack to be considered a crack attempt or changed anything in kernels.

[Dylan14]

Originally Posted by Dylan14

As it stands right now, no, it does not. My guess as to why may have to do with the fact that on Debian (which Kaggle uses), the BOINC version available is 7.6.33, whereas on Ubuntu (which Colab uses), the version available is 7.9.3.
So either I need to find a different repository for BOINC on Kaggle, or I would need to compile from source.

Actually, what I stated here was wrong, as using the newest available version from http://ftp.us.debian.org/debian/dists/testing/ (7.16.1) did not rectify the issue. So, to see what's going on, I switched back to the old code (shown here), which shows all the output. By doing so, the issue appears to be with the Internet connection:

13-Oct-2019 18:31:57 [---] Project communication failed: attempting access to reference site 

13-Oct-2019 18:32:00 [---] BOINC can't access Internet - check network connection or proxy configuration.

But it is clear that I am able to connect to the Internet and ping on Kaggle:

--- 8.8.8.8 ping statistics --- 

84 packets transmitted, 84 received, 0% packet loss, time 84956ms 

rtt min/avg/max/mdev = 0.294/0.423/0.989/0.127 ms

so this would suggest Kaggle is using a proxy. I have no idea what the settings would be though.

[kriesel]

Originally Posted by PhilF

You can always check your current working directory with !pwd

I'm really on a roll now.

Click connect, get prompted to sign in, wait for connect to complete after that, +CODE, bigger scripts, or just !pwd fails. Or sign in first, then connect, same thing.
Is this what it does if no VM is available?