mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   News (https://www.mersenneforum.org/forumdisplay.php?f=151)
-   -   12 machines permanently sidelined: LL was better than PRP on storage space (https://www.mersenneforum.org/showthread.php?t=27093)

ssybesma 2021-08-21 22:52

12 machines permanently sidelined: LL was better than PRP on storage space
 
Unless you guys can come up with a way to limit the amount of storage PRP wants to eat up, I'm done with the GIMPS project as far as most of my machines. I have 20 machines and now 12 have been forced out.

For a long time I was running Intel Compute Sticks (1GB RAM and 8GB storage) with a stripped down version of Peppermint OS and I had many completed LL tests doing it that way. It took about 4 months...and that was fine with me.

Forcing PRP is the end of first-time testing for me...12 machines have to be taken offline permanently that can't do first-time tests.

I didn't get into this project to do double-checks exclusively.

Do you have any suggestion to limit what PRP uses to no more than 512MB of space?

LL did not use that much storage space for its temp files. It was a fraction of what PRP uses.

paulunderwood 2021-08-21 23:32

Have you thought about PXE booting SLAX Linux from one HDD?

ssybesma 2021-08-21 23:48

PXE booting
 
[QUOTE=paulunderwood;586217]Have you thought about PXE booting SLAX Linux from one HDD?[/QUOTE]

Ugh...I wouldn't even know how to begin doing that. You're saying one machine acts as a server for the smaller machines to run mprime on?

All I really need is a way to put a limitation on how much disk space the PRP checks consume.

If I can do that, I can do PRP checks with all my machines but they are still a little slower, but at least they won't freeze up my machines. A few months ago I figured out out to slim down Peppermint OS even more but it's not really enough to make PRP happy. It almost eats up everything you have, even if you make more space.

If I can put a hard limit on PRP disk space consumption to say 400MB, that would be ideal...that's all the temp files combined.

I think now that amount is way over 700MB and I've already removed 9 machines from the project so all the files are gone.

I still have 11 machines finishing up but those will all be forced onto PRP after they complete.

paulunderwood 2021-08-22 00:11

[QUOTE=ssybesma;586218]Ugh...I wouldn't even know how to begin doing that. You're saying one machine acts as a server for the smaller machines to run mprime on?
[/quote]

Yes my first suggestion was a too drastic. Since you have Peppermint OS installed, why not set up a [URL="https://linuxconfig.org/how-to-configure-nfs-on-linux"]NFS[/URL] to and from an HDD'd machine?

Prime95 2021-08-22 00:23

You can put "ProofPower=5" in prime.txt. This would limit disk usage to about 32*14MB = 448MB which will go up slowly as the first-test wavefront advances.
You may also need to add "ProofPowerMult=1"

Alternatively, if you have a reasonably sized network disk drive mounted, you can ask prime95 to store the temporary files on the network disk. I do this with several of my computers.

Also: Tell prime95 to create fewer save files.

ssybesma 2021-08-22 00:27

[QUOTE=paulunderwood;586220]Yes my first suggestion was a too drastic. Since you have Peppermint OS installed, why not set up a [URL="https://linuxconfig.org/how-to-configure-nfs-on-linux"]NFS[/URL] to and from an HDD'd machine?[/QUOTE]

OK, NFS being file sharing...that is setup. I can access files on any device from any another.

So how do I point mprime on each device to look at a folder on larger (more storage space) machine for its temp files?

That would certainly solve the problem.

Steve

paulunderwood 2021-08-22 00:37

If the server is set up with something like /nfs_share/stick1, /nfs_share/stick2 etc then stick machine 1 accesses its share which contains all it needs for running mprime including the executable,

kriesel 2021-08-22 00:37

I think you have some choices. (Some of this is already covered somewhat by previous posts made by others while I was writing this)

Assuming 1GB ram, 8GB SD storage, Linux, mprime, & wifi equipped (802.11b/g/n):

A) mprime & PRP / GEC at low proof power is better than none. Run a low proof power to reduce storage space required for proof generation.
At power 7, that's ~3.2GB temp space, 400MB proof file, for p=100M, proportional to exponent (so ~3.5GB & 440MB at p=110M) both of which can get cleared automatically after proof is uploaded to the server. Still saves ~99.2% of DC effort.
At power 6, that's 1.6GB temp space, ~800MB proof file, and still saves ~98.5% of DC effort. (~1.76GB & 880MB =2.64GB at p=110M)
Minimum proof power in prime95/mprime is 5; 800MB temp space, & still saves ~97% of DC effort.
(George please correct any wrong numbers given)
How many GB of storage does your minimum install use up?

B) use other options to limit mprime's local footprint.
Put the executable, the temp file or proof file or all on a LAN server, and perhaps the more actively used working files stay on the stick's working directory.
Use Wifi or other connectivity options to access server storage. Use the separate settings for "optional directory to hold large temporary files" and "optional directory to hold archived proofs"
And as George said, reduce mprime number of backup files. Maybe you already have.

C) Use Windows which can support bigger SD cards (32GB) as local storage (but maybe that requires swapping out the Intel compute sticks or more cost than tolerable)

D) (Not recommended due to low efficiency, ~50%, sort of a waste of time compared to PRP/GEC/proof-gen once) Run PRP/GEC without proof, and PRP DC your own PRP first tests or partner with another contributor to DC each other's.
Recent Mlucas, or mprime V29.8 don't implement proof generation, so could be run on hardware that lacks space for proof gen and also lacks access to network shares with sufficient space. No proof generation means a full PRP DC is required.

E) Sell your idled 12 sticks, and buy newer hardware with adequate capacity for today's workloads.
I haven't used them, but some like Raspberry Pi. Recent Raspberry Pis have gigabit wired ethernet jacks.
Those could probably be network booted and use server hosted working directories, so make use of TB space available for sharing, temp space, proof file, backup files, etc. They're ARM based so would involve using Mlucas.
Mlucas does not have PRP proof file generation, yet.
Another way to go is a modest priced AMD GPU which could do the equivalent of 12-20 sticks of computing or more depending on models, in one unit to manage.
(For scale, 1 RadeonVII GPU can do about a wavefront PRP a day. An RX480 in ~5 days; RX550 ~18 days.)

F) Run some P-1 factoring. There's always a need for that at the wavefront. If I could do it with 1GB ram in a Windows laptop, it should be feasible in 1GB ram in Linux.

G) (Not recommended because it's very inefficient and less reliable)
Convert legitimate PRP first test assignments into bootleg LL first test,
settle for the 50% detection rate of errors that the Jacobi symbol check provides,
and stick the project with 102% or higher DC overhead.


Have fun!

ssybesma 2021-08-22 00:52

[QUOTE=paulunderwood;586223]If the server is set up with something like /nfs_share/stick1, /nfs_share/stick2 etc then stick machine 1 accesses its share which contains all it needs for running mprime including the executable,[/QUOTE]


I get that part.
I haven't seen how to setup where mprime looks.
I assume that prime.txt or local.txt would hold the path to the individual destination folder where it will keep its files.
That seems like a VERY clean option you gave me and that way the PRP temp files can be as big as they want.
I assume not all of them are being accessed constantly at once but I have excellent network speed and not worried about the host machine's resources.

kriesel 2021-08-22 00:55

[QUOTE=ssybesma;586225]I get that part.
I haven't seen how to setup where mprime looks.
[/QUOTE]./mprime -d
14
scroll through the choices, then at Advanced, say N don't skip

As a test I did this:
[CODE]Your choice: 14

Consult readme.txt prior to changing any of these settings.

Temporary disk space limit in GB/worker (6.000000):
Upload bandwidth limit in Mbps (0.250000):
Upload large files time period start (00:00):
Upload large files time period end (24:00):
Download limit for certification work in MB/day (40):
Skip advanced resource settings (Y): n
Optional directory to hold large temporary files: ./largetmp
Optional directory to hold archived proofs: ./archivedproofs
Daytime P-1/ECM stage 2 memory in GB (0.300000): 1
Nighttime P-1/ECM stage 2 memory in GB (0.300000): 1
Max emergency memory in GB/worker (1.000000):
Priority -- 1 is highly recommended, see readme.txt (1):
Certification work limit in % of CPU time (10):
Use hyperthreading for trial factoring (recommended) (Y):
Use hyperthreading for PRP, LL, P-1, ECM (not recommended) (N):

Accept the answers above? (Y): y[/CODE] and afterward found in local.txt:
[CODE]ProofResiduesDir=./largetmp
ProofArchiveDir=./archivedproofs
[/CODE]I did not find those Dir entries referenced in readme.txt or undoc.txt. Maybe I just missed them.
That's for splitting up the mprime storage space.
If you want it all on a file share, which would be simpler but maybe not quite the performance a split setup may offer, set up the share, cd to the share, as in a shell script, run from there, & don't set separate archive, temporary, or working directories in the mprime local.txt or prime.txt.

paulunderwood 2021-08-22 00:59

[QUOTE=ssybesma;586225]I get that part.
I haven't seen how to setup where mprime looks.
I assume that prime.txt or local.txt would hold the path to the individual destination folder where it will keep its files.
That seems like a VERY clean option you gave me and that way the PRP temp files can be as big as they want.
I assume not all of them are being accessed constantly at once but I have excellent network speed and not worried about the host machine's resources.[/QUOTE]

The share becomes part of the local file system for the stick. With this extension, everything needed to run mprime -- ini, local,, GMP, temp files, proofs mprime -- are all stored on the share. From the client stick you just see another directory. -- set up to share from the server.; It is mounted by the client -- tested with the mount command and automated by an entry in the /etc/fstab.


All times are UTC. The time now is 10:45.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.