Google Diet Colab Notebook - Page 21

chalsall · 2019-10-06, 23:49

Quote:

Originally Posted by chalsall

I'll definitely be writing this one up! Probably easiest to do a Notebook on Github -- there are a few steps...

OK, this was a hoot to formalize and document! And it really works!

I have just published on GitHub a Notebook which should make setting up reverse SSH tunnels through which remote Bash shell access is possible as easy as it can be. A couple of manual steps are unavoidably needed.

For anyone so interested, could I please ask you to review the Notebook, and optionally give it a try?

So everyone knows, the "public-facing SSH servers" which makes this trick possible don't have to have anything special. So long as there's an open SSH port facing the Internet and you have (even unprivileged) shell access, you can use it to gain interactive shell access (with "root credentials") into one or more Colab instances.

Please note that because Kaggle doesn't support the "echo -e" trick, I haven't been able to get this to work there. Probably several different other tricks can be used to change the root password on the instance, but I didn't bother trying.

As always, feedback (including negative) welcomed.

petrw1 · 2019-10-08, 04:42

Based on my results page it would seem that both background sessions and the foreground session are all processing the same TF assignment.

… later ...

Until the kernel powered off and had to be restarted. Then it got a brand new assignment different from the backgrounds; rather than noticing there are other partial assignments.

… later …

Though it seems the 2 background sessions are still processing the same assignments.

Uncwilly · 2019-10-08, 05:52

Every time you restart the session, it should be the latest version.

chalsall · 2019-10-08, 14:27

Quote:

Originally Posted by petrw1

Based on my results page it would seem that both background sessions and the foreground session are all processing the same TF assignment.

Hmmm... Let me drill down on the logs, and see what's up. SPEs are /always/ possible!

But so you (and everyone) knows, if an instance shuts down and has to be relaunched, as far as GPU72 is concerned it's a brand new instance (even if it has the same assignment key name). I haven't figured out a way to manage this yet (there might not be one) -- each instance is an ***exact*** clone -- short of "uptime" there's no way to tell them apart.

And just for completeness... If an instance is stopped and then restarted (re-run), it's a different matter. The file system survives.

Edit: Further to Uncwilly... Yes, every time an instance is re-run (including, of course, launched), the latest version of the Bootstrap "payload" is downloaded.

chalsall · 2019-10-08, 16:12

Quote:

Originally Posted by chalsall

For anyone so interested, could I please ask you to review the [Reverse SSH Tunnel] Notebook, and optionally give it a try?

Just wondering if anyone has given this a try? Any feedback?

Because I plan to use Colab and Kaggle in some courses I'm developing, I've decided that deploying this trick should be fully automated. My students shouldn't need SSH access to a "public-facing server" to be able to SSH into a Notebook instance to do their work.

For anyone who might like to help me beta-test this, could I please ask that you PM me?

Basically, I need a few people who are comfortable at a Linux console, and who would like to be able to SSH into their Colab instances while they are running -- straight from your workstation.

I'll give links off to (currently hidden) URLs for you to set things up for automated tunnel building.

bayanne · 2019-10-08, 20:14

I think the whole colaboratory project is brilliant, and I started running it today.
Well done :)

These points may well have been answered already so please forgive me

I presume that I still have to submit the results manually to Priment

When the session stopped running, then I can I return back to the exponent that I was running?
The next session the project started with a fresh exponent, leaving 23.35% of 97413037 of the previous exponent still outstanding

Thanks

chalsall · 2019-10-08, 20:40

Quote:

Originally Posted by bayanne

I presume that I still have to submit the results manually to Priment

The next session the project started with a fresh exponent, leaving 23.35% of 97413037 of the previous exponent still outstanding

Thanks for the kind words. This was very much a collaborative effort -- many important suggestions and observations from the early beta-testers.

Yes, I'm afraid I haven't yet built the bridge back to Primenet to automatically submit the results. It's a more difficult problem space than I had initially hoped. Still in the queue.

And, yes, you will be given that assignment back in the (near) future. For technical reasons the server currently holds it for a while before handing it back out to another one of *your* instances to finish off.

I'm currently dealing with a stupid race-condition that I haven't figured out yet, so currently, this is back to 12 hours. But I'm hoping to have this "squished" later today when I should be able to bring the recycling period down to something like 20 minutes.

BTW, while I'm "talking"... Yesterday I tidied up the GPU72 TLP to be a bit more inviting (for years the language was written for someone who already knows all about mfakt* et al) -- I'm starting to see some traffic from "newbies"; don't want to scare them away with extreme context-specific geek.

Also, I've added some additional pages for those signed in, which should make drilling down on Instance AKeys easier. This includes a new Notebook Access Keys report, which will be expanded upon.

bayanne · 2019-10-09, 04:52

No problem about manual submission.

Is a file called results.txt generated at all?

And, yes, the unfinished exponent was given back to me to complete.

Uncwilly · 2019-10-09, 05:22

Quote:

Originally Posted by bayanne

Is a file called results.txt generated at all?

On the assignment page on GPU72, there is a link to your Colab results. Copy and paste to the primenet manual submission form (Chris put a handy link on the results page.)

bayanne · 2019-10-09, 05:46

Quote:

Originally Posted by Uncwilly

On the assignment page on GPU72, there is a link to your Colab results. Copy and paste to the primenet manual submission form (Chris put a handy link on the results page.)

Thanks, I had suspected that he might have this covered :)

ric · 2019-10-09, 09:27

I suspect a new factor-found malfunction: when TF results in a factor, this is not reported into the Instances Results page: the exponent remains assigned with a 100% completion percentage, and the subsequent communication to Primenet does not clear the relative assignment, nor the result. Attached (redacted) screenshot to hopefully make things clearer.
This behaviour is recent: until a few days ago, factors were reported flawlessly.

2019-10-08, 04:42	#222
petrw1 1976 Toyota Corona years forever! "Wayne" Nov 2006 Saskatchewan, Canada 1244₁₆ Posts	Is it me? Do I need to do something to get the latest version? Based on my results page it would seem that both background sessions and the foreground session are all processing the same TF assignment. … later ... Until the kernel powered off and had to be restarted. Then it got a brand new assignment different from the backgrounds; rather than noticing there are other partial assignments. … later … Though it seems the 2 background sessions are still processing the same assignments. Last fiddled with by petrw1 on 2019-10-08 at 05:25

2019-10-09, 09:27	#231
ric Jul 2004 Milan, Ita 2×7×13 Posts	Colab factors not recognized/posted I suspect a new factor-found malfunction: when TF results in a factor, this is not reported into the Instances Results page: the exponent remains assigned with a 100% completion percentage, and the subsequent communication to Primenet does not clear the relative assignment, nor the result. Attached (redacted) screenshot to hopefully make things clearer. This behaviour is recent: until a few days ago, factors were reported flawlessly. Attached Thumbnails Last fiddled with by ric on 2019-10-09 at 09:28 Reason: speLing

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Alternatives to Google Colab	kriesel	Cloud Computing	11	2020-01-14 18:45
Notebook	enzocreti	enzocreti	0	2019-02-15 08:20
Computer Diet causes Machine Check Exception -- need heuristics help	Christenson	Hardware	32	2011-12-25 08:17
Computer diet - Need help	garo	Hardware	41	2011-10-06 04:06
Workunit diet ?	dsouza123	NFSNET Discussion	5	2004-02-27 00:42

2019-10-08, 05:52	#223
Uncwilly 6809 > 6502 """"""""""""""""""" Aug 2003 101×103 Posts 9787₁₀ Posts	Every time you restart the session, it should be the latest version.

2019-10-08, 20:14	#226
bayanne "Tony Gott" Aug 2002 Yell, Shetland, UK 2²×83 Posts	I think the whole colaboratory project is brilliant, and I started running it today. Well done :) These points may well have been answered already so please forgive me I presume that I still have to submit the results manually to Priment When the session stopped running, then I can I return back to the exponent that I was running? The next session the project started with a fresh exponent, leaving 23.35% of 97413037 of the previous exponent still outstanding Thanks

2019-10-09, 04:52	#228
bayanne "Tony Gott" Aug 2002 Yell, Shetland, UK 2²×83 Posts	No problem about manual submission. Is a file called results.txt generated at all? And, yes, the unfinished exponent was given back to me to complete.