ideas for a cloud based pet project?

[aurashift]

Hi,


So I'm working on a new on-prem private cloud redesign POC and I have an opportunity to run whatever the hell tickles my fancy to test it out, and I want to produce something of interest (we use gimps for stability testing on idle hardware, but I want to do something more ram/storage intensive than cranking out a bunch of LL's). I was thinking of running that software that was in the news somewhere in a post on here (that now I can't find) that generated a record number of digits of pi using several nodes and ~170TB of storage over a month.


If anyone knows where that post/software is that'd be nice, but I also am open to ideas on what else would be cool.


Once it gets off the ground (will be months from now) I'll likely have 6 nodes split cross-country with a 64c EPYC2 cpu, multi-terabytes of RAM, fast 100+Gbit interconnect, and a fast NVMe fabric with anywhere from several to a few hundred TBs of storage.


I don't do a lot of HPC stuff but I'm definitely looking into their playground to steal ideas.

[a1call]

You could run PRP tests without reducing the exponentiation via modular arithmetic.
This will enable you to perform tests on multiple candidates at 1/gazillions of The time it would take you to redo the modular arithmetic otherwise.
Please let me know if you are interested.

[hansl]

Originally Posted by aurashift

If anyone knows where that post/software is that'd be nice

Good news, the dev posts on this forum and it even has its own subforum:
https://mersenneforum.org/forumdisplay.php?f=159

[aurashift]

Originally Posted by hansl

Good news, the dev posts on this forum and it even has its own subforum:
https://mersenneforum.org/forumdisplay.php?f=159

noice...noiiiiiice

[aurashift]

Originally Posted by a1call

You could run PRP tests without reducing the exponentiation via modular arithmetic.
This will enable you to perform tests on multiple candidates at 1/gazillions of The time it would take you to redo the modular arithmetic otherwise.
Please let me know if you are interested.

So I'm a bit of an outsider here in that I'm not a math nerd at all, so I have no idea what you just said.


edit: You'd have to explain it like I'm five, sans the sesame street characters.

[a1call]

Originally Posted by aurashift

So I'm a bit of an outsider here in that I'm not a math nerd at all, so I have no idea what you just said.


edit: You'd have to explain it like I'm five, sans the sesame street characters.

Well, simplest form of Probabilistic-Primality test (PRP) is as follows:

Suppose you want to see if p=5 is prime. You would use the formula:

p | b^(p-1)-1

b is just any integer other than 1 or p and coprime to p, say 2.
So 5 divides 2^4-1=15
This tells you that 5 is a probable prime. And then you would perform a deterministic test if you could to prove it prime once and for all.

Now suppose you want to test a larger candidate say one with 1M decimal digits.
The formula is the same but raising b to the power of (p-1) will take perhaps a few hours and the result will not fit in any (in the market PC) RAM.
The remedy is to perform modular Arithmetic which still takes hours but needs only enough RAM to keep track of the reminder of b^(p-1)/p which is always smaller than p and sufficient information for the test. Now let's say you wanted to test a number of candidates which are close enough such that once you did one exponentiation you could get the next one quickly by a single multiplication.
Modular Arithmetic does not allow for this but if you had unlimited RAM you could register the whole number rather than just the reminder and the multiplication to get the next exponentiation would work.
Since there is no such thing as unlimited RAM, you could perform tests without Modular arithmetic only as much as your RAM would allow. But with Arbitrary-Precision programs used for large numbers the more RAM you have the bigger the numbers you could test without Modular Arithmetic.

[aurashift]

Originally Posted by a1call

Well, simplest form of Probabilistic-Primality test (PRP) is as follows:

Suppose you want to see if p=5 is prime. You would use the formula:

p | b^(p-1)-1

b is just any integer other than 1 or p and coprime to p, say 2.
So 5 divides 2^4-1=15
This tells you that 5 is a probable prime. And then you would perform a deterministic test if you could to prove it prime once and for all.

Now suppose you want to test a larger candidate say one with 1M decimal digits.
The formula is the same but raising b to the power of (p-1) will take perhaps a few hours and the result will not fit in any (in the market PC) RAM.
The remedy is to perform modular Arithmetic which still takes hours but needs only enough RAM to keep track of the reminder of b^(p-1)/p which is always smaller than p and sufficient information for the test. Now let's say you wanted to test a number of candidates which are close enough such that once you did one exponentiation you could get the next one quickly by a single multiplication.
Modular Arithmetic does not allow for this but if you had unlimited RAM you could register the whole number rather than just the reminder and the multiplication to get the next exponentiation would work.
Since there is no such thing as unlimited RAM, you could perform tests without Modular arithmetic only as much as your RAM would allow. But with Arbitrary-Precision programs used for large numbers the more RAM you have the bigger the numbers you could test without Modular Arithmetic.

How much RAM capacity/throughput are you talking?
And lets say you were swapping that to storage due to lack of RAM, what kind of numbers there? That might be harder to do, but maybe not.

[Prime95]

Originally Posted by aurashift

How much RAM capacity/throughput are you talking?

Well if every atom in the universe can hold one bit....

[aurashift]

Originally Posted by Prime95

Well if every atom in the universe can hold one bit....

well lets say you weren't going infinite just yet, and you ended up swapping to disk...what'd a few hundred terabytes get ya?

[a1call]

Originally Posted by aurashift

How much RAM capacity/throughput are you talking?
And lets say you were swapping that to storage due to lack of RAM, what kind of numbers there? That might be harder to do, but maybe not.

That's a good question.
The smallest Top-10 Twin-Prime has 52k decimal digits as of now:
My apologies, you would need virtually unlimited RAM for that.
Wolfram Alpha doesn't even dignify that calculation with a sane result.
https://www.wolframalpha.com/input/?i=2%5E(10%5E52000)

Should have done the math before I posted.

[Prime95]

Originally Posted by aurashift

well lets say you weren't going infinite just yet, and you ended up swapping to disk...what'd a few hundred terabytes get ya?

By my reckoning, you could primality test numbers of this size:

log2 (500 terrabytes) = log2 (500 * 10^12 * 8 bits)

Thus, numbers about 55 bits long.