Glorious CCCG thread -- Cellphone Compute Cluster for GIMPS

[kriesel]

Originally Posted by Mark Rose

How do you reflash it without enabling USB debugging?

Perhaps this? https://mytabletguru.com/flash-dead-...android-phone/

[ewmayer]

Originally Posted by kriesel

Perhaps this? https://mytabletguru.com/flash-dead-...android-phone/

Thanks ... at this point I'm gonna wait for M344587487 to report how he fares with his broke-o-phones, hopefully at least one will need a workaround for the USB-enablement-via-screen issue in form of rooting the phone.

[M344587487]

After a lot of flailing about I managed to root a Samsung S7 Edge. There's a few dozen methods that mostly boil down to the same thing but it's a minefield of outdated and incompatible methods. The one I got to work was with CF-Auto-Root, YMMV:

• From phone turned off boot into recovery mode (on Samsung devices hold the home, power and volume up buttons until the samsung logo displays)
• It shows you exact model and android version. Find it on firmware.mobi and download the CF-Auto-root zip file
• The zip file contains the image to flash and the flashing program Odin (windows exe) to use. Haven't figured out a Linux-PC solution yet, there's a java port of Odin called JOdin but it may be out of date and I couldn't get it to work.
• Enable USB debugging on the phone and ensure the bootloader is unlocked (some models have an OEM unlocking toggle in the dev menu, there may be other steps so search online to find out)
• From phone turned off boot into download mode (on samsung devices hold the home, power and volume down buttons until a blue screen comes up, then press volume up to go into the mode)
• Follow the readme in the CF-Auto-Root zip file

Note that the bootloader needs to be unlocked otherwise trying to root may fail or brick the device.

Originally Posted by retina

That's why you need to root it or reflash it. Don't use the existing OS, it will get in your way and annoy you. Plus it will try to talk to google and just generally do lots of stuff that kills your throughput.

Reflash to what? All I've found so far are custom versions of android. Rooting applies to the existing OS only AFAIK. Searching is tricky because it just brings up guides to setting up a Linux-like environment on top of android.

Roadblocks identified so far:

• Completely dead screen: Enabling USB debugging blind is near impossible, it's a multi-step process involving scrolling and traversing nested menus which will be slightly different depending on model. The USB-HDMI connector I have requires USB debugging to work. AFAIK you'd need to temporarily replace the dead screen with a working one to have a chance of refurbing
• Factory Reset Protection (FRP): FRP is an anti-secondhand feature in which unless the phone has been reset from the android settings menu (not the recovery menu), it'll ask you for the previous account details to proceed after a factory reset. This means that even if the previous owner has done a factory reset and shows you a picture of the welcome screen, it may not have been reset properly so ask them how they did the reset. There are some models with ways to bypass FRP, if you can't you're out of luck unless you can convince the previous owner to tell you their google account password. Good luck with that. This largely rules out reviving phones with things like dead screens as you'll need to cooperate with the previous owner to get into the OS to enable USB debugging

FRP is the real killer as it means we can't easily hoover up bottom of the barrel spares and repairs even if we have the hardware to fix them.

Notes trying to run mlucas:

• A bit of weirdness in detecting cores, /proc/cpuinfo and mlucas show 6 cores when the SoC is an Exynos 8890 8 core. As such it won't let me run an instance with -cpu 4,5,6,7. It appears that the two missing cores are A53, perhaps reserved by android for totally-not-nefarious purposes
• As encountered with some SBC's you can compile with gcc but it seg-faults when trying to actually do something. Mlucas_c2simd works fine
• With the rooted phone I got Mlucas_c2simd to execute in an Ubuntu instance within UserLAnd, an app found in the F-Droid app repo. Couldn't get mlucas to execute in a simple terminal emulator even after chmod but I suspect the reason is that where mlucas was is blocked from execution somehow. UserLAnd creates an environment in the appropriate place so it works without issue
• The SoC is an Exynos 8890 which has M1 Mongoose cores. I thought the custom cores might require recompilation but it seems not
• The phone doesn't appear to be getting very hot or discharging the battery very quickly, yet it's spitting out results that seem to be pretty good. It may be ignoring the cpu flags or limiting real time spent executing. Still creating mlucas.cfg files so haven't gotten round to benchmarks yet
• Locking the screen doesn't appear to slow down execution

Originally Posted by ewmayer

Thanks ... at this point I'm gonna wait for M344587487 to report how he fares with his broke-o-phones, hopefully at least one will need a workaround for the USB-enablement-via-screen issue in form of rooting the phone.

There's one coming that does have a fully dead screen, it's unlikely I'll be able to fix it unfortunately.

[retina]

Originally Posted by M344587487

Reflash to what?

I've no personal experience here, but I know of people that have put Linux on them. They only get SSH access from what I can tell. No fancy graphics. No dumb graphics. No graphics at all. However the ones I saw were not second hand, they were working, and the original owners knew whatever passwords may have been necessary. So, perhaps I misunderstood the process. I didn't realise they might have needed a working screen and passwords to get Linux on them.

[ewmayer]

Wow, thanks for the detiled summary, M344587487 - that's a bummer re. the Factory Reset Protection, but it is what it is. Hopefully once we firm up the recipe (or set thereof) in terms of both the hardware and rootkit ingredients we'll have a better sense as to whether getting one's hands on a suitable number of eligible discard phones to build a compute farm with throughput similar to a cutting-edge Intel multicore-CPU system will be practicable.

Re. your self-tests, are you running one set with -cpu 0:3 and a second with -cpu 4:5, or what? (I assume the 4-cores-available CPU maps to cores 0-3 in proc/cpuinfo, is that right?) How long do you expect the self-tests to take? Did you stick any cooling fins on the CPUs, or are you using the phone as-is?

[M344587487]

Originally Posted by retina

I've no personal experience here, but I know of people that have put Linux on them. They only get SSH access from what I can tell. No fancy graphics. No dumb graphics. No graphics at all. However the ones I saw were not second hand, they were working, and the original owners knew whatever passwords may have been necessary. So, perhaps I misunderstood the process. I didn't realise they might have needed a working screen and passwords to get Linux on them.

A minimal custom android OS would be good, probably ideal honestly. There appear to be custom ROMs which are the OS, and custom kernels. Maybe there's a custom kernel for the S7 which exposes all 8 cores, and maybe there's a custom OS with a more suitable resource scheduler (that doesn't keep shunting work between clusters).

Originally Posted by ewmayer

...
Re. your self-tests, are you running one set with -cpu 0:3 and a second with -cpu 4:5, or what? (I assume the 4-cores-available CPU maps to cores 0-3 in proc/cpuinfo, is that right?)

That's right, haven't tried all 6 at once but I can if you like.

Originally Posted by ewmayer

...
How long do you expect the self-tests to take? Did you stick any cooling fins on the CPUs, or are you using the phone as-is?

Here's what I have so far, as you'll see there's some oddities that need addressing before the rest of the testing can happen. I'm using the phone as-is, it doesn't get very warm which I find very odd. It's certainly in a class beyond SBC's when it comes to heat, I suppose the entire phone is designed to be a heatsink which helps.

Some dodgy mlucas.cfg timings, looks like the processor preference was ignored. I kept unlocking the phone to check progress which probably didn't help but unless all cores are loaded I think this is normal behaviour. I used these configs in testing but changed the timings so that it'd pick the right FFTs:
mlucas.cfg -cpu 0,1,2,3 https://paste.ubuntu.com/p/rKWF8ZppBS/
mlucas.cfg -cpu 4,5 https://paste.ubuntu.com/p/hjdC9FxzPs/

/proc/cpuinfo http://paste.ubuntu.com/p/Bj5bzbJBhb/

cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq failed with permission denied even as root, but there are 8 cores there. A custom kernel may be in order.

The results are always cores 0,1,2,3 first, 4,5 second:
1024k battery locked http://paste.ubuntu.com/p/8vN9fNCFvH/
2560k battery locked http://paste.ubuntu.com/p/WNg9TqPPMS/

Where it gets weird is the 1024k test with battery charging:
1024k charging locked http://paste.ubuntu.com/p/pMw9z6fCZB/

In all three cases the two tests were started simultaneously, then the phone locked itself and wasn't unlocked until the test was stopped. The first two tests look stable. If you look at the 1024k charging timings not only are both clock timings way worse, but one of the clusters must have been turned off judging by the realtime timings. The test was also turned off at past midnight as I assumed more results were present, unknown if the tests were shelved or just executing slowly. I'm doing more testing at 1024k to try and figure out what's going on.

Just tried "1024k charged locked" for an hour and no results had been spat out, both clusters should have had half a dozen by then. Now trying "1024k charged unlocked", which means plugged in at full battery with the screen forced on. It seems to be spitting out marginally better timings than "1024k battery locked" but will let it run overnight and see if the scheduler goes nuts. It seems to be warmer but it's likely mostly due to the screen being on. Tomorrow I'll see if there's a way to get more accurate config files, putting the other cluster under load may work but who knows if it'll play nice.

[ewmayer]

Thanks for the data - very interesting. I doubt there's much point in trying any -cpu 0:5 runs, we already know from e.g. testing on prototype Odroid N1 than trying to run 1 job using both the big+little CPUs kills performance ... but feel free to try a single-FFT-length self-test @1024k using all 6 cores:

./Mlucas -fftlen 1024 -iters 100 -cpu 0:5

and see what the resulting line appended to mlucas.cfg says re. timing.

Some observations/questions:

o Is there a way to tell if the CPUs are throttling under load? Looking at the cfg-file data, it's almost a sawtooth kind of pattern, as if at the start of the self-test the CPUs are cold and unthrottled, then after a few minutes and several FFT lengths throttling kicks in to allow things to cool off, a few minutes of that and things are cool enough to unthrottle, lather, rinse, repeat. I'm thinking maybe the phone-not-getting-very-hot-under-load is due not to fabulous heat transfer design but rather aggressive throttling-temperature settings, the kind which would make sense for a device people hold in their hands while using. That might also explain the dramatic slowdown you see during charging - even a elatively modest amount of extra heat from the charging process could result in appreciably greater CPU throttling-under-load. (Or maybe the OS is designed to reduce charging time by throttling CPU loads, or some combination of both.)

o Throttling could also explain why the 2-jobs running 'production' tests @1024k have timings so much slower than the 1024k entries in the corresponding cfg-files: e.g. @1024k we have a timing or 31ms/iter in cfg1 (-cpu 0:3) and 20 ms/iter in cfg2 (-cpu 4:5), but in your actual simultaneous svereal-checkpoint runs @1024 the -cpu 0:3 job was getting 59 msec/iter and the -cpu 4:5 job 71 msec/iter. On the Odroid N1 prototype last year the slowdown from running -cpu 0:3 and -cpu 4:5 jobs simultaneously was only in the 10-20% range.

o Is there a simple way for you to increase the cooling of the CPUs? Maybe stick the phone in some kind of air stream (perhaps next to one of the case fans from an ATX-cased PC), or if you're in cool climate, outside into cold air?

o Re. a custom kernel that would enable access to all 8 cores, I didn't even know such hackage is possible - what would that involve, and would one need to manually add core 6/7 entries to /proc/cpuinfo analogous to those for cores 4/5?

[M344587487]

Originally Posted by ewmayer

Thanks for the data - very interesting. I doubt there's much point in trying any -cpu 0:5 runs, we already know from e.g. testing on prototype Odroid N1 than trying to run 1 job using both the big+little CPUs kills performance ... but feel free to try a single-FFT-length self-test @1024k using all 6 cores:

./Mlucas -fftlen 1024 -iters 100 -cpu 0:5

and see what the resulting line appended to mlucas.cfg says re. timing.

1024  msec/iter =   25.18  ROE[avg,max] = [0.283482143, 0.375000000]  radices =  32 16 32 32  0  0  0  0  0  0    100-iteration Res mod 2^64, 2^35-1, 2^36-1 = 77E8A2AB109EC324, 21535019539, 66963127211

Originally Posted by ewmayer

o Is there a way to tell if the CPUs are throttling under load? Looking at the cfg-file data, it's almost a sawtooth kind of pattern, as if at the start of the self-test the CPUs are cold and unthrottled, then after a few minutes and several FFT lengths throttling kicks in to allow things to cool off, a few minutes of that and things are cool enough to unthrottle, lather, rinse, repeat. I'm thinking maybe the phone-not-getting-very-hot-under-load is due not to fabulous heat transfer design but rather aggressive throttling-temperature settings, the kind which would make sense for a device people hold in their hands while using. That might also explain the dramatic slowdown you see during charging - even a elatively modest amount of extra heat from the charging process could result in appreciably greater CPU throttling-under-load. (Or maybe the OS is designed to reduce charging time by throttling CPU loads, or some combination of both.)

Miraculous heat dissipation is wishful thinking, it is definitely aggressively throttling. I'm on the lookout for a custom kernel and/or OS with a more aggressive governor and ideally exposes it simply for tweaking as I'm not familiar with how to do it on Linux let alone android which may be different. There are sensors which appear to be exposed via the program 'sensors', there's probably an easier way but I'll try and dump it periodically during a run.

Originally Posted by ewmayer

o Throttling could also explain why the 2-jobs running 'production' tests @1024k have timings so much slower than the 1024k entries in the corresponding cfg-files: e.g. @1024k we have a timing or 31ms/iter in cfg1 (-cpu 0:3) and 20 ms/iter in cfg2 (-cpu 4:5), but in your actual simultaneous svereal-checkpoint runs @1024 the -cpu 0:3 job was getting 59 msec/iter and the -cpu 4:5 job 71 msec/iter. On the Odroid N1 prototype last year the slowdown from running -cpu 0:3 and -cpu 4:5 jobs simultaneously was only in the 10-20% range.

1024k charged unlocked http://paste.ubuntu.com/p/GhSxK8RrCZ/

These are better timings and stable, by leaving it unlocked the phone thinks the user is using it so it allows the CPU to stretch its legs more. It still barely gets warm and I suspect much of that is due to the screen. All testing in different states does is indirectly alter what the governor does and clearly charged and unlocked is the best so I'll stick with it for now. It's interesting that the two clusters have timings that are so close, I wonder if the jobs are constantly getting swapped between clusters averaging out the timings while also slowing things down. A 0:5 test is pending.

Originally Posted by ewmayer

o Is there a simple way for you to increase the cooling of the CPUs? Maybe stick the phone in some kind of air stream (perhaps next to one of the case fans from an ATX-cased PC), or if you're in cool climate, outside into cold air?

It's cold outside, I'll try that.

Originally Posted by ewmayer

o Re. a custom kernel that would enable access to all 8 cores, I didn't even know such hackage is possible - what would that involve, and would one need to manually add core 6/7 entries to /proc/cpuinfo analogous to those for cores 4/5?

I don't know if it's possible I'm just spitballing, but it is sold as an 8 core SoC and we can run a custom OS and kernels so there's a possibility. It's a Samsung exclusive chip so there's no competitive downside to them restricting the SoC in software, they may restrict it for load balancing. Hopefully someone has created a custom whatever so it would just be a case of finding it and flashing it with Odin AFAIK.

[kriesel]

Originally Posted by M344587487

1024  msec/iter =   25.18  ROE[avg,max] = [0.283482143, 0.375000000]  radices =  32 16 32 32  0  0  0  0  0  0    100-iteration Res mod 2^64, 2^35-1, 2^36-1 = 77E8A2AB109EC324, 21535019539, 66963127211

Miraculous heat dissipation is wishful thinking, it is definitely aggressively throttling. I'm on the lookout for a custom kernel and/or OS with a more aggressive governor and ideally exposes it simply for tweaking as I'm not familiar with how to do it on Linux let alone android which may be different. There are sensors which appear to be exposed via the program 'sensors', there's probably an easier way but I'll try and dump it periodically during a run.

1024k charged unlocked http://paste.ubuntu.com/p/GhSxK8RrCZ/

These are better timings and stable, by leaving it unlocked the phone thinks the user is using it so it allows the CPU to stretch its legs more. It still barely gets warm and I suspect much of that is due to the screen. All testing in different states does is indirectly alter what the governor does and clearly charged and unlocked is the best so I'll stick with it for now. It's interesting that the two clusters have timings that are so close, I wonder if the jobs are constantly getting swapped between clusters averaging out the timings while also slowing things down. A 0:5 test is pending.

It's cold outside, I'll try that.

I don't know if it's possible I'm just spitballing, but it is sold as an 8 core SoC and we can run a custom OS and kernels so there's a possibility. It's a Samsung exclusive chip so there's no competitive downside to them restricting the SoC in software, they may restrict it for load balancing. Hopefully someone has created a custom whatever so it would just be a case of finding it and flashing it with Odin AFAIK.

Nice work!
Timings for 1024k fft length may give quick data for variable sensitivity testing.
Is it your intention that these cell phone based compute farms would be used on such low fft lengths & exponents in production? If planning use near the first-test or DC wavefront, I'd like to see some longer timings for that eventually. I have quadro 2000s running tests of over a month duration, with good results, so long runs 0.1 to 0.5 years per completed result, perhaps even longer, should not be completely ruled out. Judging from http://paste.ubuntu.com/p/WNg9TqPPMS/ they may be capable of a few DC a year. I think that compares favorably to some of the older silicon space heaters I have still running, both in duration and compute/joule.

[M344587487]

I plan to do simultaneous testing at 1024K, 2560K, 4608K, 7680K and 18432K with every SBC and phone I have that uses big.LITTLE so that they can be compared (following the RK3399 testing https://www.mersenneforum.org/showpo...&postcount=177 ). I picked these to be a good spread and hitting the main lengths of interest. Having both clusters do the same FFT allows for a synthetic iteration time to be made from the combined throughput so that these SoCs can be more easily compared to traditional CPU architectures. Still figuring out the best way to go about benching, ewmayer is right that cooling makes a big difference with phones so figuring out a fair way to test is the plan today. I'm leaning towards testing different temps at 1024K to show the temp scaling, and using a standardised ambient with airflow environment to bench the different FFT lengths. When that's done I'll hunt for a lean custom kernel and/or OS that focuses on performance to see if there's something to be gained there.

Don't know what others plans are but I plan on putting some phones on DC LL, maybe first time PRP if and when.

[ewmayer]

Originally Posted by M344587487

I don't know if it's possible I'm just spitballing, but it is sold as an 8 core SoC and we can run a custom OS and kernels so there's a possibility. It's a Samsung exclusive chip so there's no competitive downside to them restricting the SoC in software, they may restrict it for load balancing. Hopefully someone has created a custom whatever so it would just be a case of finding it and flashing it with Odin AFAIK.

Are you able to edit /proc/cpuinfo? I don't know how the OS exposes/hides cores from Joe User, but it might be worth a shot to see if simply exposing them via that system file allows a user app to use them. I would need to send you a little hack of the Mlucas util.c file to test this - one of the pending changes in v18 is to work around portability issues by having the program read the relevant data directly from that file rather than relying on the ABI for it. I would be surprised if said hack actually worked in terms of giving you access to cores 6:7, but it's easy enough to try.

I'll be interested to see your "baby, it's cold outside" timings, as well.

Originally Posted by kriesel

Nice work!
Timings for 1024k fft length may give quick data for variable sensitivity testing.
Is it your intention that these cell phone based compute farms would be used on such low fft lengths & exponents in production? If planning use near the first-test or DC wavefront, I'd like to see some longer timings for that eventually. I have quadro 2000s running tests of over a month duration, with good results, so long runs 0.1 to 0.5 years per completed result, perhaps even longer, should not be completely ruled out. Judging from http://paste.ubuntu.com/p/WNg9TqPPMS/ they may be capable of a few DC a year. I think that compares favorably to some of the older silicon space heaters I have still running, both in duration and compute/joule.

My little Odroid C2, which has just a single a53 quadcore CPU, needs ~ 2 months for an LL DC. My hope is that ARM-based phones of several-years-old vintage and of the big.little dual-CPU variety, might yield comparable throughput when properly cooled with small finned heatsinks stuck on each CPU and in decent airflow. And if such setups prove sufficiently reliable, no reason one couldn't do first-time tests on them, as well, assuming such wouldn't take so long that the assignment expires. Strength in numbers is the idea here ... we shall see!

For those of you with teenaged kids interested in such stuff, note that putting together such a cluster would make for a great science fair project.