mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Hardware > Cloud Computing

Reply
 
Thread Tools
Old 2019-09-05, 15:02   #23
De Wandelaar
 
De Wandelaar's Avatar
 
"Yves"
Jul 2017
Belgium

73 Posts
Default

Thanks again, Chalsall : GPU mode was not set (Shame on me !).
Now it's working fine

Kitty mode is on !
De Wandelaar is offline   Reply With Quote
Old 2019-09-06, 15:50   #24
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

132×59 Posts
Default More adventures with CoLab...

Hey, so... I hate finding something like this when I have pressing deliverables, but...

Further research has shown that the CoLab VM is a Ubuntu 18.04.3 environment, within which all user processes run as root. (!)

This means that it is trivial to install software packages which don't already exist. For example, I was able to get my Perl submit_spider to run by first running this in a Section:
Code:
!apt-get install libcrypt-ssleay-perl
!apt-get install libwww-perl
This has to be done once each time the VM is reset (something like every 12 hours).

Over the weekend I'm going to put together a Notebook which automates the configuration of a VM, downloads work from GPU72, and then uploads the results back to GPU72 for the user(s) to submit to Primenet.

I'm going to use authentication keys (AKs) tied to each GPU72 account, so that GPU72 nor Primenet's login credentials are required. I'll set things up such that only the AK is needed to run work; access to a user's Google Drive won't be needed.

Lastly, each GPU72 account will be able to have multiple AKs, so people can ask "Friends and Family" to do work for them.

Thoughts?

And are there any GPU72 users who would like to help beta-test this? If so, please PM me and I'll give you the secret URLs to access.
chalsall is online now   Reply With Quote
Old 2019-09-07, 18:54   #25
De Wandelaar
 
De Wandelaar's Avatar
 
"Yves"
Jul 2017
Belgium

10010012 Posts
Default

Under Windows, you can easily integrate a Colab notebook in the Misfit workflow if you are working with a Google drive.
  • Create a mfaktc directory on your google drive.
  • Copy the executable provided by Chalsall & Hansl to this directory (see attach).
    Copy your mfaktc.ini file (nothing special in the parameters).
  • Create your colab notebook with this script (assuming you use the \mfaktc-0.21 directory) :
    Code:
    import os.path
    from google.colab import drive
    
    if not os.path.exists('/content/drive/My Drive'):
      drive.mount('/content/drive')
    
    %cd '/content/drive/My Drive/mfaktc-0.21/'
    
    !chmod 755 '/content/drive/My Drive/mfaktc-0.21/mfaktc.exe'
    !cd '.' && /content/drive/My\ Drive/mfaktc-0.21/mfaktc.exe
    The first lines install your Google drive as a local drive for the Virtual Machine.
    The next lines grant the necessary accesses to the virtual machine and launch the executable.

    Don't forget to set your runtime for GPU acceleration.
  • Parametrise your misfit configuration in order to refer to the google drive mfaktc directory.

It's done … the Google drive synchronisation mechanisms will ensure a lean communication between misfit and the VM.
Attached Files
File Type: tgz mfaktc_colab.tgz (622.1 KB, 220 views)
De Wandelaar is offline   Reply With Quote
Old 2019-09-07, 19:20   #26
Dylan14
 
Dylan14's Avatar
 
"Dylan"
Mar 2017

2·293 Posts
Default

Using the same type of script as chalsall did with mfaktc, I set up a script to handle CUDAPm1. Here's the procedure:
  • In your Google Drive, create a folder called cudapm1
  • Download CUDAPm1-0.22-cuda10-linux from the download mirror at https://download.mersenne.ca/CUDAPm1, rename it to cudapm1-linux.exe and upload it to the cudapm1 folder you made
  • (Optional) Provide a CudaPm1.ini file in the same folder
  • Run the following code in the notebook:
Code:
import os.path
from google.colab import drive

if not os.path.exists('/content/drive/My Drive'):
  drive.mount('/content/drive')

%cd '/content/drive/My Drive/cudapm1//'

#CUDAPm1 executable from https://download.mersenne.ca/CUDAPm1/CUDAPm1-0.22-cuda10-linux
!cp 'cudapm1-linux.exe' /usr/local/bin/
!chmod 755 '/usr/local/bin/cudapm1-linux.exe'

!mkdir /usr/local/bin/lib/
!cp 'lib/libcudart.so.6.5' /usr/local/bin/lib/

!cd '.' && LD_LIBRARY_PATH="lib:${LD_LIBRARY_PATH}" /usr/local/bin/cudapm1-linux.exe -selftest
and the self test will run.
I got one of the T4's, and it is fast! Attached is the results from the cufft benchmark.
Attached Files
File Type: txt TeslaT4_fft.txt (1.7 KB, 228 views)
Dylan14 is offline   Reply With Quote
Old 2019-09-07, 20:27   #27
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

100110111100112 Posts
Default

Quote:
Originally Posted by Dylan14 View Post
Using the same type of script as chalsall did with mfaktc, I set up a script to handle CUDAPm1. Here's the procedure:
Cool!

I'm really excited about this! It can be used for all kinds of things; just build a Notebook which automates things as much as possible, and then have people donate Google's cycles... The trick is coordinating the assignment of work units.

I'm just about ready to have beta testers try my initial attempt at a solution which doesn't use Google Drive, and doesn't need access credentials.

Are there any GPU72'ers out there who have a Google Account but aren't currently using Colaboratory for work, and would be willing to do some testing? If so, please post here, or PM me.

P.S. All the code I'm writing for this is GPL, so it might be useful to others to implement other compute spaces; I'm focusing on TF'ing initially. But in the back of my mind, I'm modeling other compute problems which could benifit.
chalsall is online now   Reply With Quote
Old 2019-09-07, 20:45   #28
Dylan14
 
Dylan14's Avatar
 
"Dylan"
Mar 2017

24A16 Posts
Default

Quote:
Originally Posted by chalsall View Post
P.S. All the code I'm writing for this is GPL, so it might be useful to others to implement other compute spaces; I'm focusing on TF'ing initially. But in the back of my mind, I'm modeling other compute problems which could benifit.

A true acid test would be to get BOINC set up on the notebook. It would prove handy for quite a lot of projects (thinking NFS@Home with the Xeon, or the PPS sieve on the GPU for PrimeGrid).
Dylan14 is offline   Reply With Quote
Old 2019-09-07, 21:13   #29
xx005fs
 
"Eric"
Jan 2018
USA

22×53 Posts
Default

Quote:
Using the same type of script as chalsall did with mfaktc, I set up a script to handle CUDAPm1
Similarly, I am using multiple instances of Colab to run gpuowl. I am using my google drive to store all the checkpoint files by first uploading the gpuowl file on there, and then running it off the drive instead of storing the checkpoints on the VM which disappears every 12 hours.

Code:
from google.colab import drive
drive.mount('/content/drive')
!chmod 777 '/content/drive/My Drive/gpuowl'
!cd '/content/drive/My Drive/gpuowl' && LD_LIBRARY_PATH="lib:${LD_LIBRARY_PATH}" && chmod 777 gpuowl && chmod 777 worktodo.txt && ./gpuowl -use ORIG_X2 -block 200 -log 120000
In this case, the Tesla K80 is the preferred GPU, as it is over 50% faster than the Tesla T4 in this workload. Whenever I am unable to obtain a Tesla T4 on Google Colab I would just use the K80s to run this. Surprisingly, the checkpoints do save on your drive permanently even after the session is over.

Last fiddled with by xx005fs on 2019-09-07 at 21:15
xx005fs is offline   Reply With Quote
Old 2019-09-07, 21:30   #30
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

132·59 Posts
Default

Quote:
Originally Posted by xx005fs View Post
Similarly, I am using multiple instances of Colab to run gpuowl.
What trick are you using to get multiple instances? Multiple Google accounts? Logged into by different browsers / local VMs?

Quote:
Originally Posted by xx005fs View Post
Surprisingly, the checkpoints do save on your drive permanently even after the session is over.
That isn't really surprising. Your permanent Google Drive is simply mounted in the CoLab's ephemeral filesystem. I have noticed quite a bit of latency between a file being uploaded to Drive and it appearing in the VM.

I just wish there was a way to mount only a subdirectory under the VM's FS. As it is a malicious Notebook could do a "!rm -rf /content/drive/"...
chalsall is online now   Reply With Quote
Old 2019-09-07, 22:38   #31
xx005fs
 
"Eric"
Jan 2018
USA

3248 Posts
Default

Quote:
Originally Posted by chalsall View Post
What trick are you using to get multiple instances? Multiple Google accounts? Logged into by different browsers / local VMs?



That isn't really surprising. Your permanent Google Drive is simply mounted in the CoLab's ephemeral filesystem. I have noticed quite a bit of latency between a file being uploaded to Drive and it appearing in the VM.

I just wish there was a way to mount only a subdirectory under the VM's FS. As it is a malicious Notebook could do a "!rm -rf /content/drive/"...
I just use multiple Google accounts, and I don't think a VM would work as when you log in to colab it will display the currently running things from your google account. The fact that it saved is surprising to me since I first thought it was read-only for a mounted google drive and there's no way for me to save the checkpoints.
xx005fs is offline   Reply With Quote
Old 2019-09-07, 23:22   #32
Corbeau
 
Corbeau's Avatar
 
Jul 2019

2×3 Posts
Default

Quote:
Originally Posted by xx005fs View Post
I just use multiple Google accounts, and I don't think a VM would work as when you log in to colab it will display the currently running things from your google account. The fact that it saved is surprising to me since I first thought it was read-only for a mounted google drive and there's no way for me to save the checkpoints.
Be careful using more than one Google account. Apparently people on the LCZero project were banned from using CoLab because they did that.
Corbeau is offline   Reply With Quote
Old 2019-09-07, 23:31   #33
xx005fs
 
"Eric"
Jan 2018
USA

21210 Posts
Default

Quote:
Originally Posted by Corbeau View Post
Be careful using more than one Google account. Apparently people on the LCZero project were banned from using CoLab because they did that.
Did they get banned because they were using multiple accounts or did they just get banned from long computes? Also, I am using my accounts that I actually use daily, and I didn't create them specifically for Colab and I only run 3-4 instances.

Last fiddled with by xx005fs on 2019-09-07 at 23:39
xx005fs is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Alternatives to Google Colab kriesel Cloud Computing 11 2020-01-14 18:45
Notebook enzocreti enzocreti 0 2019-02-15 08:20
Computer Diet causes Machine Check Exception -- need heuristics help Christenson Hardware 32 2011-12-25 08:17
Computer diet - Need help garo Hardware 41 2011-10-06 04:06
Workunit diet ? dsouza123 NFSNET Discussion 5 2004-02-27 00:42

All times are UTC. The time now is 18:33.


Thu Oct 28 18:33:46 UTC 2021 up 97 days, 13:02, 0 users, load averages: 0.95, 1.18, 1.31

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.