Google Diet Colab Notebook

[lycorn]

It ran for 5h 31m.
Just started another run from a different account. Got a T4 as well. As I´m leaving home now, I won´t be around to click and extend the running time, so I guess it will stop after a bit more than 3 hours.
It´s weird that Colab is currently being more liberal on GPUs than on CPUs, but we should already be used to their oddities .

[kriesel]

last evening, same accounts as https://mersenneforum.org/showpost.p...postcount=1218
hh:mm, resolution ~3 or 4 minutes; scripts similar to https://www.mersenneforum.org/showthread.php?t=24839

mprime+mfaktc: 3:32; 3:45; 3:31
mprime+gpuowl: 1:58; 3:54
mprime only: no samples

That is quite different from the previous run cycle, in a good way. Maybe Google just had exceptionally high demand for paying compute for a while and was promptly kicking out us freeloaders during peak paid usage.

[slandrum]

All my CPU only sessions are running. They've passed the 6 hours 40 minutes mark, and no recaptcha popups. Usually at 6:01 to 6:10 there's a recaptcha, and if it's not responded to in time the sessions are killed at around 6:40.

[chalsall]

Originally Posted by slandrum

All my CPU only sessions are running. They've passed the 6 hours 40 minutes mark, and no recaptcha popups.

Yup. I can confirm the usage patterns are back to nominal (at least for the mfaktc/mprime and mprime runs).

Thanks, Colab / Google. You had us a little worried there for a bit...

[LaurV]

Originally Posted by chalsall

LaurV seemed to avoid the rath, but maybe he actually gives them money.

Yup. Long story. Don't jinx it!
How about a solution to my A100+Mfaktc problem?

[moebius]

2022-12-22 20:11:22 gpuowl v6.11-380-g79ea0cc
2022-12-22 20:11:22 Note: not found 'config.txt'
2022-12-22 20:11:22 config: -proof 6 -maxAlloc 1000 
2022-12-22 20:11:22 device 0, unique id ''
2022-12-22 20:11:23 Tesla T4-0 6307XXXX FFT: 3.25M 256:13:512 (18.51 bpw)
2022-12-22 20:11:23 Tesla T4-0 Expected maximum carry32: 4AA70000
2022-12-22 20:11:24 Tesla T4-0 OpenCL args "-DEXP=6307XXXXu -DWIDTH=256u -DSMALL_HEIGHT=512u -DMIDDLE=13u -DPM1=0 -DMM_CHAIN=1u -DMM2_CHAIN=1u -DMAX_ACCURACY=1 -DWEIGHT_STEP_MINUS_1=0x1.a07961fc2b3c1p-2 -DIWEIGHT_STEP_MINUS_1=-0x1.280fd972008b5p-2  -cl-unsafe-math-optimizations -cl-std=CL2.0 -cl-finite-math-only "
2022-12-22 20:11:29 Tesla T4-0 
2022-12-22 20:11:29 Tesla T4-0 OpenCL compilation in 4.63 s
2022-12-22 20:11:30 Tesla T4-0 6307XXXX LL  5500000 loaded: e119cb6604f86409
2022-12-22 20:17:36 Tesla T4-0 6307XXXX LL  5600000   8.88%; 3664 us/it; ETA 2d 10:30; 29d1cf173251008b
2022-12-22 20:23:44 Tesla T4-0 6307XXXX LL  5700000   9.04%; 3680 us/it; ETA 2d 10:39; e6b94ec44af21a67
2022-12-22 20:29:52 Tesla T4-0 6307XXXX LL  5800000   9.20%; 3682 us/it; ETA 2d 10:34; 6bea2de5e1b0657b
2022-12-22 20:36:00 Tesla T4-0 6307XXXX LL  5900000   9.35%; 3680 us/it; ETA 2d 10:27; 44c5ad247efa9cc5
2022-12-22 20:42:08 Tesla T4-0 6307XXXX LL  6000000   9.51%; 3682 us/it; ETA 2d 10:22; 34c6deb9b345c34c
2022-12-22 20:48:16 Tesla T4-0 6307XXXX LL  6100000   9.67%; 3681 us/it; ETA 2d 10:15; fdde8cf0b3989ee5
2022-12-22 20:48:16 Tesla T4-0 6307XXXX OK  6000000 (jacobi == -1)
2022-12-22 20:54:25 Tesla T4-0 6307XXXX LL  6200000   9.83%; 3682 us/it; ETA 2d 10:10; d63411de441a03bf
2022-12-22 21:00:32 Tesla T4-0 6307XXXX LL  6300000   9.99%; 3678 us/it; ETA 2d 10:00; e63b45a418fd52ff
2022-12-22 21:06:40 Tesla T4-0 6307XXXX LL  6400000  10.15%; 3678 us/it; ETA 2d 09:54; cdaaf32c36cdaf37
2022-12-22 21:12:48 Tesla T4-0 6307XXXX LL  6500000  10.31%; 3681 us/it; ETA 2d 09:51; bce890db7f74d259
2022-12-22 21:18:56 Tesla T4-0 6307XXXX LL  6600000  10.46%; 3679 us/it; ETA 2d 09:43; d6cae961ab66c91e
2022-12-22 21:18:56 Tesla T4-0 6307XXXX OK  6500000 (jacobi == -1)
2022-12-22 21:25:04 Tesla T4-0 6307XXXX LL  6700000  10.62%; 3679 us/it; ETA 2d 09:37; 4a2e5bac7da5da4b
2022-12-22 21:31:12 Tesla T4-0 6307XXXX LL  6800000  10.78%; 3678 us/it; ETA 2d 09:30; 2877d4b3b53c4da4
2022-12-22 21:37:20 Tesla T4-0 6307XXXX LL  6900000  10.94%; 3680 us/it; ETA 2d 09:25; 3971d1f107eaa7f0
2022-12-22 21:43:28 Tesla T4-0 6307XXXX LL  7000000  11.10%; 3681 us/it; ETA 2d 09:20; 5ab8e6938c09dab3
2022-12-22 21:49:36 Tesla T4-0 6307XXXX LL  7100000  11.26%; 3678 us/it; ETA 2d 09:11; aa31c242fed7003f
2022-12-22 21:49:36 Tesla T4-0 6307XXXX OK  7000000 (jacobi == -1)
2022-12-22 21:55:44 Tesla T4-0 6307XXXX LL  7200000  11.42%; 3678 us/it; ETA 2d 09:05; d399cb03d28c27c4
2022-12-22 22:01:52 Tesla T4-0 6307XXXX LL  7300000  11.57%; 3679 us/it; ETA 2d 09:00; 7abb6819f79a7f4f
2022-12-22 22:08:00 Tesla T4-0 6307XXXX LL  7400000  11.73%; 3678 us/it; ETA 2d 08:53; 90ba085c761c41d8
2022-12-22 22:14:07 Tesla T4-0 6307XXXX LL  7500000  11.89%; 3676 us/it; ETA 2d 08:44; 7add8c93408f525e
2022-12-22 22:20:15 Tesla T4-0 6307XXXX LL  7600000  12.05%; 3677 us/it; ETA 2d 08:40; 4c78d7163ae52c1c
2022-12-22 22:20:15 Tesla T4-0 6307XXXX OK  7500000 (jacobi == -1)
2022-12-22 22:26:23 Tesla T4-0 6307XXXX LL  7700000  12.21%; 3677 us/it; ETA 2d 08:33; cdcc182c6213bdbd
2022-12-22 22:32:31 Tesla T4-0 6307XXXX LL  7800000  12.37%; 3680 us/it; ETA 2d 08:30; 0767b59b14695bbd
2022-12-22 22:38:38 Tesla T4-0 6307XXXX LL  7900000  12.53%; 3678 us/it; ETA 2d 08:22; bdbab0229c41ae22
2022-12-22 22:44:46 Tesla T4-0 6307XXXX LL  8000000  12.68%; 3679 us/it; ETA 2d 08:17; 8aec0fbd145dcfc4
2022-12-22 22:50:54 Tesla T4-0 6307XXXX LL  8100000  12.84%; 3680 us/it; ETA 2d 08:12; 9417bf9acd26588a
2022-12-22 22:50:54 Tesla T4-0 6307XXXX OK  8000000 (jacobi == -1)
2022-12-22 22:57:02 Tesla T4-0 6307XXXX LL  8200000  13.00%; 3679 us/it; ETA 2d 08:04; 6465205228b50b17
2022-12-22 23:03:10 Tesla T4-0 6307XXXX LL  8300000  13.16%; 3678 us/it; ETA 2d 07:58; 2011ecb7cf2ed8ab
2022-12-22 23:09:18 Tesla T4-0 6307XXXX LL  8400000  13.32%; 3679 us/it; ETA 2d 07:52; b13f0c4cb0034dd0
2022-12-22 23:15:26 Tesla T4-0 6307XXXX LL  8500000  13.48%; 3678 us/it; ETA 2d 07:45; 6d886324cfbdd099
2022-12-22 23:21:34 Tesla T4-0 6307XXXX LL  8600000  13.64%; 3679 us/it; ETA 2d 07:40; 4f9cd69da0a45799
2022-12-22 23:21:34 Tesla T4-0 6307XXXX OK  8500000 (jacobi == -1)
2022-12-22 23:27:41 Tesla T4-0 6307XXXX LL  8700000  13.79%; 3680 us/it; ETA 2d 07:34; b4bc134553f77886
2022-12-22 23:33:49 Tesla T4-0 6307XXXX LL  8800000  13.95%; 3678 us/it; ETA 2d 07:27; e38e846a24a3317b
2022-12-22 23:39:57 Tesla T4-0 6307XXXX LL  8900000  14.11%; 3680 us/it; ETA 2d 07:22; d321cf34f01dc9ad
2022-12-22 23:46:05 Tesla T4-0 6307XXXX LL  9000000  14.27%; 3679 us/it; ETA 2d 07:15; 4b7f728c23f976df
\

[kriesel]

On Google Colab free, lscpu now includes a cpu vulnerabilities section. Sample captured:

Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable; SMT Host state unknown
Vulnerability Meltdown:          Vulnerable
Vulnerability Mmio stale data:   Vulnerable
Vulnerability Retbleed:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and use
                                 rcopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled, PB
                                 RSB-eIBRS: Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Vulnerable

Seems like a lot

[kriesel]

After successful runs up to and including 2023-01-11, all my Colab free session GPU app launches are consistently halting with error as follows, and there's no way I changed all of them! Or any that I recall.

./mmff-linux-colab.exe: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory

including the ones that include inline, just before the program launch,

!LD_LIBRARY_PATH="lib:${LD_LIBRARY_PATH" && chmod 777 mmff-linux-colab.exe && chmod 777 worktodo.txt
!apt-get install -y cuda-cudart-10-1

yielding log

Mounted at /content/drive
/content/drive/My Drive/mprime
mprime launched in background
code here for Tesla T4 case.
/content/drive/My Drive/mmff
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package cuda-cudart-10-1

or

./mfaktc.exe: error while loading shared libraries: libcudart.so.10.0: cannot open shared object file: No such file or directory

from

#section to continue mfaktc run in background
%cd '/content/drive/My Drive/mfaktc//'
!nvidia-smi
#as of about 2019 November 17, we need to install a cuda lib also
!apt-get install -y cuda-cudart-10-0
!chmod 755 '/content/drive/My Drive/mfaktc/mfaktc.exe'
!./mfaktc.exe >> mfaktc-run.txt 2>&1 &
print('mfaktc launched in background')

with resulting log

Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package cuda-cudart-10-0

And https://download.mersenne.ca/CUDA-DLLs does not include the Linux .so files for CUDA 10.0 or 10.1.

Are others seeing that too? Ideas for a fix? There are several lengthy runs stalled at ~40% in mmff.

[moebius]

[QUOTE=kriesel;622607]After successful runs up to and including 2023-01-11, all my Colab free session GPU app launches are consistently halting with error as follows, and there's no way I changed all of them! Or any that I recall.

Here are 2 guides to Installing CUDA 10.1 on Ubuntu 18.04/20.04. Don't know if they helps you?

https://medium.com/@stephengregory_6...4-e562a5e724a0

https://medium.com/@exesse/cuda-10-1...s-d04f89287130

[kriesel]

What a guide to install won't help with I think is why what used to work suddenly no longer does.
"Unable to locate package" sounds to me like the repository Google colab VMs will use became undefined or has removed the needed versions of cuda packages. (Had a similar problem with Centos 8.0-8.4 IIRC a while back; repositories removed packages for them.)
And isn't anything that requires a reboot likely to be useless for a Google Colab VM?

[chalsall]

Originally Posted by kriesel

Are others seeing that too? Ideas for a fix? There are several lengthy runs stalled at ~40% in mmff.

Yes. I had to recompile mfaktc against CUDA 11.0 for the GPU72 Colab TF Notebook. They also broke my Perl comms component, as a package that /used/ to be available from their repository (libcrypt-ssleay-perl) no longer is. Great fun (not!).

BTW... This /might/ mean that A100s work, but I have no way of testing this (Google doesn't consider Barbados a market worth selling to).