New Google Colab Notebooks For Primality Testing

[danc2]

Overview:
Hello fellow Crunchers,
Cruncher Teal Dulcet (@tdulcet) and I (@danc2) have been working on a project to expedite the computation of prime numbers. This project offers users the ability to run GIMPS programs on Google Colab for primality testing. This offers a cost effective way to utilize GPUs and CPUs to contribute to GIMPS without forking any money out for expensive hardware. Users may also increase their contribution by upgrading to Google Colab Pro, which offers a higher usage limit for GPU and CPU utilization than that of the 12 hours in the free version. This project is facilitated by the following noteworthy additions to the GIMPS project:

1. GPU-And-CPU-Powered Colab Jupyter Notebooks
Google Colab offers two free Jupyter notebooks for each Google Account, which can be run for a maximum of 12 hours per day without interruption. The GPU72 project previously created a notebook that can utilize assigned Nvidia GPUs and run mfaktc. Our CPU-powered notebook runs Prime95 and our GPU-powered notebook runs CUDALucas; with the added bonus of also using the CPU on that machine to run Prime95. On average, the time to finish a Category 4 First time LL test takes about 30 days when running CUDALucas on Google Colab. The results from these tests are highly reliable as Google Cloud uses ECC memory in its backend (source). Users can run multiple instances of each notebook while utilizing the same CUDALucas and/or MPrime binaries, but separate supporting files (e.g., worktodo). We also dynamically fix buffer overflows in CUDALucas, allowing it to utilize the P100 and V100 GPUs Google has to offer. Users interact with a form and run notebooks without the prerequisite need to know how to code.

2. Colab Autorun and Connect Browser Add-on/Extension
Our Colab Autorun and Connect Firefox and Chrome/Chromium add-on/extension will automatically connect, reconnect and run the first cell of notebooks in Google Colab. It can be used to automatically run both our GPU and CPU notebooks when the user opens their browser or wakes their computer. It will soon be published to Addons.mozilla.org (AMO) and possibly the Chrome Web Store.

3. PrimeNet Python Script For Mlucas and CUDALucas
Our script may also be used outside of these notebooks for other GIMPS projects. Notably, the script now uses the v5 API for all worktypes. We have also addressed a number of bugs from previous versions of the PrimeNet script.

Differences from the GPU72 Project notebook
1. Compute work types that are eligible for cash research discovery award of up to $50,000.
2. Choices are not limited to the Trial Factorization work type. Choose from any LL worktype for the GPU and all LL and PRP work types (not recommended due to ~3.5 GiB for current first time tests; ~23% of Drive space) on the CPU. Change these worktypes later if desired.
3. Assignments are sent from/to the PrimeNet server, without needing a third party intermediary server.
4. One can save state for long running work types (e.g., LL, PRP, etc.) using Google Drive.
5. Automatically obtain and submit jobs with PrimeNet, with little configuration.
6. Can utilize both the GPU and CPU to run CUDALucas and Prime95 (the GPU notebook).
7. Open Source.
8. Builds CUDALucas directly from source code rather than precompiled binaries.
9. Colab GPUs are much faster at primality testing (LL/PRP) than trial factoring (TF)(source).
10. Easier setup than existing Notebook implementations.
11. Includes counts of all previously received CPUs and GPUs.

How to Get Started
Use the GPU notebook.
Use the CPU notebook.
Or visit the repository for more detailed instructions.

Teal and I are available for further contact, questions, and feedback. Happy crunching!

[VBCurtis]

I'm glad you did this project, and starting a new thread for it is the right thing to do.
Please, please don't post links to your new thread in every possibly relevant (or not-so-relevant) thread you can think of. I've removed all those other posts.

Trust that users of this forum are smart enough to notice your new thread. Most users use the "new posts" feature in the menu bar at the top of the forum. Your topic will get plenty of notice there, and all your other links are redundant and clutter up that exact new-post feature.

[Uncwilly]

There is a notebook that Chris of GPU72 set up. I am guessing it does what you think yours does. His automatically uses the CPU and GPU (when available). The thing is, transferring big files associated with primality tests (and storing them) is not real practical. So he has his set to do P-1 on the CPU.
https://www.mersenneforum.org/showthread.php?t=24875
https://www.mersenneforum.org/showpo...5&postcount=19

[tdulcet]

Originally Posted by Uncwilly

There is a notebook that Chris of GPU72 set up. I am guessing it does what you think yours does.

See the "Differences from the GPU72 Project notebook" section of the first post, which addresses this. In short, his notebook is for trial factoring and our notebooks are for primality testing.

Note that users of the GPU72 notebook or any other notebooks can also use our browser add-on/extension.

[kriesel]

Please do not encourage people to run LL first tests. We're trying to phase LL out entirely as routine first tests. Assignment rules may prevent issuance of LL first test work at some point in the future. PRP with GEC and proof generation is far superior for first tests:

1. Error detection almost 100% by GEC on PRP, vs. only 50% probability of detection by Jacobi check (and CUDALucas does not have the Jacobi check)
2. Essentially equal effort for first primality test computation
3. Approx. 1% the effort of verification, so less than half the total effort for first-test and verification, more than twice the effective primality testing throughput (~1.01 tests equiv per PRP, proof generation, and Cert completion of an exponent, vs. ~2.04 tests per LL, DC, and occasional TC, QC, 5C, 6C)
4. Superior quality / standard of verification (certs can't be faked, even by the person who did the PRP test, while LL DC result reports could be; a successful cert also shows the primality test was done completely and correctly)
5. Far faster verification of a primality test (hours to weeks for PRP with proof generation via prompt brief Cert priority assignments, vs. several YEARS for LL DC, TC, etc.) Verification issues can be traced back to problem hardware or configuration usefully early in the hardware's useful life, instead of after it's removed from service.
6. Setting up for PRP from the start avoids a transition from LL to PRP later.

Those determined to run LL despite PRP's superiority are welcome to help with the approximately 8 year backlog of LL DC. There will be less of an issue with assignment expiration before completion on Google Colab, because LL DC assignments are much quicker than first-time LL assignments; about 1/4 the duration as a result of the nearly 2:1 ratio of exponent values at their respective wavefronts.

[chalsall]

Originally Posted by tdulcet

See the "Differences from the GPU72 Project notebook" section of the first post, which addresses this. In short, his notebook is for trial factoring and our notebooks are for primality testing.

Very nice work gentlemen.

Thanks for the reference in your docs. I had a hoot designing and implementing my particular "proof-of-concept".

And, yes. I made a conscious decision to only do Trial Factoring on the GPUs (although as you said, they are actually better at other work on some of the GPU offerings). I didn't want to get into the whole legal question of what is the situation if an MP was actually found.

Also... I decided very early on not to require the attachment of a Drive. At the time, it wasn't as easy as it is now, and so didn't "scale" well. And so the tiny TF checkpoint files mapped well (only later were the much larger P-1 contexts added).

Originally Posted by tdulcet

Note that users of the GPU72 notebook or any other notebooks can also use our browser add-on/extension.

I would advise against this.

The Colab Terms of Service are very clear that automation is *not* acceptable for Notebook startup / restarting. They want a human in the loop (I suspect part of this is actually a sociology experiment... 9-)

But, again... Nice work! I'm sure there are many out there who will use this.

[danc2]

Thank you for all the encouragement, congratulatory words, and constructive feedback. Below is a simple clarification on our intentions, thoughts, and motivations.

Trust that users of this forum are smart enough to notice your new thread

We hope no offense was taken to forum users' intellect. Posting in other places was an attempt to update users who are still interested in GIMPS, have become less active on forums, but still hold email subscriptions to popular threads (i.e., are still interested in the topics discussed therein). Nevertheless, we will honor moderator opinions.

Please do not encourage people to run LL first tests.

While we would love for this project to be optimal for PRP testing, the Google Drive size constraints make it difficult to do so, especially if Drive space is used for more than this project by the user. We are not encouraging users to do more LL testing as opposed to PRP testing on their other machines, but rather are trying to increase the net contribution with spare compute time offered by Google Cloud and not cause the project to fill up/bork the user's drive. I think that it is a very good point that this project could speed up DC checks. Note also that we started this project before GEC on PRP was a part of GIMPS.

The Colab Terms of Service are very clear that automation is *not* acceptable

I could not find anything besides the pro terms of service, but did not see anything about automation (I am probably missing it). The warning is noted, however. Though I will say that personally I think that rule would be a bit silly as the whole point of the project is to increase progress in research and machines can be disconnected quite often, even though another machine is available to reconnect to and continue progress on. This may be a rule that is more focused on miners of cryptocurrency then anything else (or maybe it is a sociological experiment). There are also a number of existing extensions (though they do not work) that attempt to do the same thing which Google has not yet barred from running on their machines thus far, perhaps because of their ineffectiveness.

Nice work! I'm sure there are many out there who will use this.:

Thanks! We really appreciate viewing and building off of your work!

Note: The link in #1 of overview is incorrect (my fault). The link should be GPU-And-CPU-Powered Colab Jupyter Notebooks.

[Chuck]

I used Chris' notebook setup successfully for several months (even signing up for the paid Colab Pro service), but Colab kept continually reducing the availability of service with no stated policy. It finally became so restricted that I just dumped the whole thing and cancelled the service.

[danc2]

It finally became so restricted that I just dumped the whole thing and cancelled the service.

I have not noticed any severe availability issues as of this present date. In fact I am so happy with the service, I actually have paid for three Colab Pro accounts ($9.99 each). Keep in mind also that some availability issues are made null by the AutoRun and Connect extension. That is, notebooks reconnect after (tight) usage limits and reconnect if your particular backend has been assigned to a Google Cloud customer.

[danc2]

I realize we did not post any output or pictures, just links.

Since we have this dedicated thread, here is example output from a GPU notebook running the Tesla V100-SMX2-16GB (a $6,195.00 GPU according to Amazon).

[Uncwilly]

Originally Posted by danc2

I have not noticed any severe availability issues as of this present date.

There are messages that pop up about usage limits.