Google Diet Colab Notebook

[Corbeau]

@kriesel Can I run your script while running the GPU72 trial factoring script on Colab? If so, do I need to manually add work in the worktodo file, or will it get assignments from the GPU72-reserved LL assignments?

[kriesel]

One of the barriers, or rather a slowdown, I've run into initially in using Colab is that, handy as drag and drop of a file from a local host that I own to Google drive and vice versa is, compared to other methods, it still constitutes some overhead, and the file duplication seems likely to cause errors. What I'd really like is:

1. Ability to mount the relevant Google drives on a local system here
2. Easy setup for that
3. Multiple OS support for the host system OS
4. Low host system overhead
5. Multiple cloud storage type support, to unify usage of other cloud options such as Box, OneDrive, etc
6. Adequate security that it does not compromise the local host system
7. Acceptable convenience and performance
8. Low cost (free is ideal)
9. No requirement to encrypt the cloud storage, which could interfere with the Colab or other cloud application being able to usefully process the data and return useful results.
10. (What did I miss?)

I think CloudMounter qualifies, after a quick initial skim. There are alternatives.
https://www.nextofwindows.com/how-to...isk-on-windows
Windows https://cloudmounter.net/mount-cloud-drive-win.html
Mac https://cloudmounter.net/
linux https://cloudmounter.net/mount-cloud-drive-linux.html

Some of the following may qualify, while some are quite a stretch, or are useful utilities for other purposes alongside mounting a cloud drive. https://www.topbestalternatives.com/cloudmounter/

Favorite solutions? Thoughts? Experience?

(Please avoid OS-bashing. Different people use different OSes for different reasons, that make sense to them.)

[PhilF]

Originally Posted by kriesel

And the rapid progress of this collaborative effort has been something to see.

That's probably what Google and Kaggle are thinking too...

[kriesel]

Originally Posted by Corbeau

@kriesel Can I run your script while running the GPU72 trial factoring script on Colab? If so, do I need to manually add work in the worktodo file, or will it get assignments from the GPU72-reserved LL assignments?

I don't use GPU72, so have no idea about compatibility with it. I'm also unsure which script you're referring to; the mprime resume one at https://www.mersenneforum.org/showpo...0&postcount=3? If so, it uses primenet, as set up by a variation of Dylan14's initial setup script, gets work, reports work, on its own, given enough 12-hour sessions to finish an assignment. My first Colab 87M primality test is weeks away from completion at twice-daily restarts; it takes persistence. My understanding is if using cpu and gpu application script sections in Colab in the same work section on the Colab web page, one of them needs to be run as a subprocess, then the other run as the script process. I'm about to try that as ath has described.
If you're using chalsall's reverse tunneling, you could probably instead use that to run mprime.
It appeals to me to have one Colab script to relaunch, that runs both the subprocess and the other to occupy the VM cpu and the VM gpu for the duration.
For gpu applications, the same or similar client management applies in the Colab environment as when running on our own gpus. A single pass through a Python script for result reporting and ensuring adequate work seems like a natural fit to me. There are other possibilities.
I regard the Colab thread scripts I've posted on my blog as a collaborative effort, not "mine", and try to give credit there for the originator.

[kriesel]

Originally Posted by PhilF

That's probably what Google and Kaggle are thinking too...

Hence the disappearance of T4 availability? ("Give them only the older K80s, let's see what they come up with.")

[LaurV]

Well, we should try to "come up with" running cudaLucas on it.
K80 is a waste if used for TF. This card is flying like a rocket at LL.

[chalsall]

Originally Posted by kriesel

And the rapid progress of this collaborative effort has been something to see.

The family's asleep. Back At Console for a little bit...

A book that I seriously internalized as a "youngin" was The Mythical Man-Month by Brooks.

Empirically, I have found that small ad-hock teams can often do better work faster than larger groups with hierarchical communication channels can.

YMMV.

[chalsall]

Originally Posted by LaurV

Well, we should try to "come up with" running cudaLucas on it.

Don't disagree.

Why don't you code that up?

[axn]

Originally Posted by chalsall

Why don't you code that up?

BTDT. Got 3 DCs out of it. Major pain; Colab gets a conniption if you run it for long, and then you don't get GPU instance for a while.
Pretty fast, though. I estimated that if you run it full time, you could get about 60 GhzDay/day, which is pretty much in line with https://www.mersenne.ca/cudalucas.php

EDIT:- Even more impressive is Kaggle and its P100. That is not a 2x GPU like K80, and so in theory you should get the full 160 GhzDay/day. But for that, you need a file hosting location to download the appropriate CUDALucas executables and libraries (cudart and cufft) -- google drive won't work.

[bayanne]

Originally Posted by chalsall

Don't disagree.

Why don't you code that up?

Chapeau from me if you can :)

[LaurV]

Well, I am digging, but Linux was never my strong point, and here the pain is to store and retrieve the checkpoint files. But I am learning...