mersenneforum.org  

Go Back   mersenneforum.org > Extra Stuff > Blogorrhea > kriesel

Closed Thread
 
Thread Tools
Old 2019-10-19, 13:32   #12
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

150038 Posts
Default The Google drive access authorization sequence

Assuming a VM instance is obtained, and the script connects to and runs off a Google drive folder, there are several steps to making that Google drive connection from the Colaboratory script. Note, in the attached screen captures, my account names and some other data are blacked out afterward, but yours won't be when you use Google.
  1. A prompt is given to go to an accounts.google.com URL
  2. Click on that link. Another browser tab opens.
  3. Select the Google account you want to use to sign in with. Each has its own Google drive.
  4. A message that Google Drive File Streams wants to access your Google Account will appear. Click on "Allow" at the lower right.
  5. A Google Signin page with authorization code will appear, consisting of upper and lower case letters, digits, and other characters. Copy the line-wrapped code. (That's the part in the red box that I added for emphasis in the attachment.)
  6. Go back to the Colaboratory notebook web page and paste the authorization code into the prompt box. The code will be masked as a bunch of black dots. Press "enter" and the script will begin to run.
  7. A "Success" tab will appear. It does no harm to leave it, but they accumulate over time so close it eventually.
The above is the sequence if the script launches successfully.

But as of 2020-02-10, there's this, which should streamline things considerably: https://twitter.com/GoogleColab/stat...29213560610818
(Thanks for the notification, kracker)


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Thumbnails
Click image for larger version

Name:	auth request.png
Views:	294
Size:	15.1 KB
ID:	21140   Click image for larger version

Name:	google sign in.png
Views:	311
Size:	78.9 KB
ID:	21141   Click image for larger version

Name:	allow google drive access.png
Views:	282
Size:	23.3 KB
ID:	21142   Click image for larger version

Name:	copy the authorization code.png
Views:	285
Size:	6.4 KB
ID:	21143   Click image for larger version

Name:	paste the authorization code.png
Views:	282
Size:	15.2 KB
ID:	21144  


Last fiddled with by kriesel on 2020-02-18 at 01:26
kriesel is online now  
Old 2019-10-19, 13:40   #13
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

11010000000112 Posts
Default When a VM or GPU is not available

Either from overuse by one account, or due to general high demand, sometimes a Colaboratory gpu may not be available, or a VM not available.
A runtime configuration that requires a gpu will generate a message asking whether to connect and use a backend with no accelerator (gpu).

If no gpu is available, and the script does not require one, it may still be the case that the script can not be run because no backend is available.

Screen captures of the corresponding messages are attached.

I hit both conditions, in October 2019 and since, on my second account in use, while the first sometimes continues to run a script; sometimes neither can obtain a VM at all for many hours.

Launching a script that requires a gpu when one is not available may lead to a terminated script and a hidden background task running mprime on the cpu. This can be monitored with !top -d 60 or similar in a separate code section.


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Thumbnails
Click image for larger version

Name:	ifnogpuavailable.png
Views:	309
Size:	84.8 KB
ID:	21145   Click image for larger version

Name:	nobackendsavailable.png
Views:	303
Size:	92.4 KB
ID:	21146  

Last fiddled with by kriesel on 2020-02-09 at 17:43
kriesel is online now  
Old 2019-12-14, 00:13   #14
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

1A0316 Posts
Default Issues, questions, support

The Google Colaboratory project is using github for issue reporting and tracking. See https://github.com/googlecolab/colab...ssue+is%3Aopen

For questions, there is stackoverflow with the tag google-colaboratory. See https://stackoverflow.com/questions/...-colaboratory/

My question about preferring or requiring a specific gpu model has had dozens of views but no replies yet. https://stackoverflow.com/questions/...-or-to-require

Per the Colab FAQ, the way to send feedback, is in a Colab notebook, select the Help menu, then "Send feedback..." Example feedback (which can also include a screen shot, although this one did not):

It could sometimes be very useful to be able to determine in a script whether a gpu was available, and if so which one, and branch based on that. See post #16 in this thread for detect & branch code.

Also to require or request a particular gpu model would be useful; no solution in the Colab menus for that yet The detect and branch in python approach could be used. See also https://stackoverflow.com/questions/...-or-to-require

Connecting as chalsall describes in https://www.mersenneforum.org/showpo...&postcount=943 seems to help when Colab and Google Drive are operating normally; I see runs up to 10 or 12 hours with it; under 7 hours (and occasionally only minutes) without it.

Petrw1 reports an inability to obtain a gpu session over several days, and a link showing others experiencing that too, while others are not experiencing it. Cause and cure are unknown.


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2021-04-24 at 01:23
kriesel is online now  
Old 2019-12-19, 22:18   #15
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

665910 Posts
Default Gpu models available through Google Colab

The model allocated to a session may be any of the following. There is currently no way to select a model or indicate a gpu-model preference or requirement. (There are reports of V100 in the paid tier, but I have no data on it from the free tier.)

I think all prices below are used.

Code:
Tesla P100 https://www.techpowerup.com/gpu-specs/tesla-p100-pcie-16-gb.c2888
16GB HBM2 732 GB/sec dual-slot 250W FP64 4.763 TFLOPS (1/2) 
1175 GhzD/day TF, 173.4 LL(95M)
$600 on eBay
indicates 0 of 16280 MiB allocated at Colab notebook launch
 
Tesla P4 https://www.techpowerup.com/gpu-specs/tesla-p4.c2879
8GB 192 GB/sec single-slot 75W FP64 178.2 GFLOPS (1/32)
512 GhzD/day TF, 32.5 LL (95M)
$280 on eBay
indicates 0 of 7611 MiB allocated at Colab notebook launch

Tesla K80 (note, dual gpu, specs below are per card not per gpu; Colab free session may include single gpu but not dual)
12GBx2, 240.6 GB/sec x2, dual-slot 300W FP64 1371 GFLOPS (1/3)
766.7 GhzD/day TF, 115.1 LL (95M)
$230 on eBay
indicates 0 of 11441 MiB allocated at Colab notebook launch

Tesla T4
16GB 320 GB/sec single-slot 70W FP64 254.4 GFLOPS (1/32)
2467. GhzD/day TF, 59.3 LL (95M)
$355 on eBay
indicates 0 of 15079 MiB allocated at Colab notebook launch

Tesla V100
16GB HBM2 897 GB/sec mezzanine or dual-slot 250W FP64 7.834 TFLOPS (1/2) https://www.techpowerup.com/gpu-specs/tesla-v100-sxm2-16-gb.c3018
4162. GhzD/day TF, 221 LL (95M)
$2400 on eBay
(never seen one of these in Colab free myself)

NVIDIA Tesla A100 https://www.techpowerup.com/gpu-specs/a100-pcie.c3623
40GiB HBM2e vram, 1555 GB/sec, dual-slot 250W FP64 9.746 TFLOPS (1/2) 
6707 GHD/day TF, 746. GHD/day LL/PRP (100M)
$9400 on eBay
(never seen one of these in Colab free myself)
The above are all passively cooled (no integral fans), requiring either a server case with high pressure blowers, or aftermarket addon blowers & adapters.
Compare to:
Code:
Tesla C2075
6 GB 144 GB/sec dual-slot 247W FP64 515.2 GFLOPS (1/2)
282.2 GhzD/d TF, 22.2 LL (95M)
$175 on eBay

Radeon VII:
16 GB HBM2 1024 GB/sec dual-slot 295W FP64 3.36 TFLOPS (1/4)
1113.6 TF, 280.9 LL (95M); there are reports of >500 in PRP in Linux with ram overclocking to 1200 MHz; I've myself observed in Windows, ~1800 TF (Mfakto), ~400 PRP (Gpuowl v6.11-380)
$1200+ on eBay

RTX2080:
8GB 448 GB/sec dual-slot 215W FP64 314.6 GFLOPS (1/32)
2703 GHzD/d TF, 65 LL (95M)
$750 on eBay
Note: in gpuowl, use -maxAlloc m, where m is megabytes limit per gpuowl instance, g is free megabytes on the idle gpu, n is number of gpuowl instances per gpu, b = 1000 or perhaps more if there are problems at 1000;
m<=(g -b)/n.
Or go higher when using multiple instances per gpu and memlock and -pool in gpuowl V7
Note 2: the above prices are as of the original post date, and have changed considerably since.


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2022-04-03 at 14:36 Reason: minor formatting
kriesel is online now  
Old 2020-02-09, 17:40   #16
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

11010000000112 Posts
Default Multiple branches for cpu-only, or various gpu models

Thanks to a bit of code included in the "Taking advantage of colab pro" page, it's possible to have a Google Colaboratory script detect and branch based on gpu model or no-gpu condition. This allows making use of whatever gpu model one gets, with per-model separate folders and ini file, config.txt, etc. tuned for the specific gpu model or benchmarking for it, and avoid some error messages from the no-gpu-available case.

The attached example script implements that branching, running gpu and cpu as available, and identifying what's happening, as well as giving the user the choice to go ahead with a cpu-only session or quit. It also warns when the worktodo file is smaller than a settable threshold at startup for whichever gpu model is detected. (Mprime is presumed to be PrimeNet connected and getting new work that way as needed.)
A companion script section for building the latest commit of gpuowl is also included.

This could probably be tidied up some with use of Python functions.
If you see ways to make this more efficient, please respond, in a discussion thread https://www.mersenneforum.org/showthread.php?t=23383, clearly, with change "whatever" to "newstuff".


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Files
File Type: txt conditional gpu runs full.txt (5.6 KB, 310 views)
File Type: txt gpuowl-build.txt (2.3 KB, 309 views)

Last fiddled with by kriesel on 2020-09-18 at 18:20
kriesel is online now  
Old 2021-02-22, 01:56   #17
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

6,659 Posts
Default Mersenne.ca work assignment and results submit

dylan14 posted several versions of scripts for getting work assignments from and submitting results to mersenne.ca for mfaktc TF above exponent 1,000,000,000, including
https://www.mersenneforum.org/showpo...&postcount=598
https://www.mersenneforum.org/showpo...&postcount=600
https://www.mersenneforum.org/showpo...&postcount=650
https://www.mersenneforum.org/showpo...&postcount=654
https://www.mersenneforum.org/showpo...&postcount=711
https://www.mersenneforum.org/showpo...&postcount=768
https://www.mersenneforum.org/showpo...&postcount=915


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2021-02-22 at 16:47
kriesel is online now  
Old 2021-02-22, 01:57   #18
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

6,659 Posts
Default GMP-ECM for Colab

Fan Ming posted a compile of GMP-ECM for gpu. Not useful for wavefront GIMPS, haven't tried it, but there it is, at https://www.mersenneforum.org/showpo...&postcount=729


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2021-02-22 at 17:00
kriesel is online now  
Old 2021-02-22, 16:59   #19
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

6,659 Posts
Default Persistent storage

Without persistent storage, long runs would not be possible, and short runs might be repeated from the start.

Google Drive works. Other cloud storage might or might not. I haven't tried any other than free Google Drive.

Google Drive free capacity is 15 GB, including its trash folder. Note that Google offers multiple free mail, storage, etc accounts per person, so one's personal or other email and other cloud storage can be segregated by account, allowing multiple Colab-only accounts to be set up to use the full free 15GB of each. Mprime and Gpuowl clean up after themselves.

Cleaning out the trash https://mersenneforum.org/showpost.p...postcount=1025
"If you'd like to purchase more Drive space, visit Google Drive. Note that purchasing more space on Drive will not increase the amount of disk available on Colab VMs. Subscribing to Colab Pro will."
https://research.google.com/colaboratory/faq.html
Standard plan Google One (100GB) is $20/year; Advanced (200GB) $30/year; Premium (2TB) $100/year. https://one.google.com/about#upgrade


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2021-02-22 at 17:13
kriesel is online now  
Old 2021-09-30, 17:39   #20
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

150038 Posts
Default Moving Colab instances

On occasion it may be necessary or helpful to move a Colab instance from one browser or system to another.
(Easier access, free up a system for hardware maintenance, etc.)

For this post, it's the two-system case, with an associated Colab session, Google account & email, and Google drive, 1:1 mapping, Colab notebook and necessary software saved on the account's Google drive. (Actually I recently moved 3 to free a laptop in need of hardware maintenance.) Here's a suggested way how. (Not to be confused with moving work in progress to a different Colab instance or cloud storage etc.)

The system the instance is being moved from will be called source below; the system the instance is being moved to will be called destination.

0 Make a plan. What do you want where, and how do you want it laid out? If moving multiple instances, that may favor moving one particular instance first, another last, etc. Now might also be the easiest time to do a little of step 12 with the Colab accounts & related tabs already present on the destination system's browser, before adding more.

1 Establish some form of communication between the two systems. Windows remote desktop, VNC, ssh, a jointly reachable server file share, or even email may suffice.

2 Check the destination browser is set to reopen existing tabs upon restart.

For the first instance to be moved:

3 Wait for the source's Colab session to terminate, or force a stop, or confirm it's already stopped.

4 Save the source's Colab notebook, if any changes have been made since last save, or if you're not sure, to the associated Google drive.

5 At destination, log in to the associated Google account. That will later provide access to Google Drive, Colab, email, etc.

6 At source, copy the URL for the Google drive. On the destination open a new browser tab and "paste and go" that copied URL. Confirm you have access at the destination to the Google drive, such as by navigating it. Probably a good idea to bookmark it on the destination.

7 At source, copy the URL for The Colab notebook. On the destination open a new browser tab and paste and go that copied URL. Confirm the notebook loaded properly. Probably a good idea to bookmark it on the destination.
Run the notebook as usual. Confirm it is running properly.

8 If you have numerous Colab instances, and a scorecard of what's what and where, now is a good time to update that. I find that helpful with several. Also, grouping tabs by Google account.

9 After all is confirmed working on the destination, remove from the Colab instance etc. from the source without relaunching anything on the source:
Close the source Google drive browser tab.
Close the source Colab notebook browser tab.
If there are no other Google applications dependent on its presence, close the source Google authorization session or email session.

10 Repeat the above sequence restarting at step 3 for each additional instance to be moved, in order.

11 When you're really sure the destination instances are working, optionally remove source bookmarks and clear source browser cache.

12 Optionally, organize the tabs for clarity or efficiency. I find it most productive to have the various tab types of one account grouped together, isolated a bit from the other accounts, and in left-right order corresponding to the top-down order of account selection menu. In Firefox, tabs are drag-and-drop left or right.

It's faster than that description may make it seem.


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=24839
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2021-10-01 at 16:23
kriesel is online now  
Old 2021-11-11, 02:55   #21
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

150038 Posts
Default Embellishments

(draft)

Some possibly useful additions to the preceding posts' notebook code are (and some mayneed directory inserted)

ls -l /usr/lib/x86_64-linux-gnu/libstdc*
tail gpuowl.log
tail mfaktc-run.txt
cat worktodo.txt

Last fiddled with by kriesel on 2021-11-11 at 02:56
kriesel is online now  
Closed Thread

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Reference material discussion thread kriesel kriesel 88 2022-08-08 15:52
Google Diet Colab Notebook Corbeau Cloud Computing 1225 2022-07-31 13:51
Mlucas-specific reference thread kriesel kriesel 20 2021-11-18 19:22
Alternatives to anesthesia thread (because Google isn't helping) jasong jasong 16 2016-07-14 05:34
The thread for mundane questions that Google fails at jasong jasong 9 2014-02-08 01:54

All times are UTC. The time now is 02:31.


Thu Aug 11 02:31:34 UTC 2022 up 34 days, 21:18, 2 users, load averages: 1.17, 0.94, 0.95

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

≠ ± ∓ ÷ × · − √ ‰ ⊗ ⊕ ⊖ ⊘ ⊙ ≤ ≥ ≦ ≧ ≨ ≩ ≺ ≻ ≼ ≽ ⊏ ⊐ ⊑ ⊒ ² ³ °
∠ ∟ ° ≅ ~ ‖ ⟂ ⫛
≡ ≜ ≈ ∝ ∞ ≪ ≫ ⌊⌋ ⌈⌉ ∘ ∏ ∐ ∑ ∧ ∨ ∩ ∪ ⨀ ⊕ ⊗ 𝖕 𝖖 𝖗 ⊲ ⊳
∅ ∖ ∁ ↦ ↣ ∩ ∪ ⊆ ⊂ ⊄ ⊊ ⊇ ⊃ ⊅ ⊋ ⊖ ∈ ∉ ∋ ∌ ℕ ℤ ℚ ℝ ℂ ℵ ℶ ℷ ℸ 𝓟
¬ ∨ ∧ ⊕ → ← ⇒ ⇐ ⇔ ∀ ∃ ∄ ∴ ∵ ⊤ ⊥ ⊢ ⊨ ⫤ ⊣ … ⋯ ⋮ ⋰ ⋱
∫ ∬ ∭ ∮ ∯ ∰ ∇ ∆ δ ∂ ℱ ℒ ℓ
𝛢𝛼 𝛣𝛽 𝛤𝛾 𝛥𝛿 𝛦𝜀𝜖 𝛧𝜁 𝛨𝜂 𝛩𝜃𝜗 𝛪𝜄 𝛫𝜅 𝛬𝜆 𝛭𝜇 𝛮𝜈 𝛯𝜉 𝛰𝜊 𝛱𝜋 𝛲𝜌 𝛴𝜎𝜍 𝛵𝜏 𝛶𝜐 𝛷𝜙𝜑 𝛸𝜒 𝛹𝜓 𝛺𝜔