Parallel sieving with newpgen

fivemack · 2017-05-16, 14:57

Is there a way that I can sieve different ranges of factors (for, say, k*2^131072+1, k=2^27..2^28) on different cores?

Splitting up the range across cores is pointless, because each core's still having to generate all the primes; I'd like to be able to sieve p=2e14..3e14 on one core, 3e14..4e14 on another, and merge the lists of eliminated k values afterwards.

Batalov · 2017-05-16, 16:31

What you described is exactly how some of us had done it for years.
It is easy to script (compete with core assignments); each process would need its small separate sandbox (e.g. a folder).

fivemack · 2017-05-16, 16:46

Does the first number on the first line of the file have to have any non-obvious congruence properties? Or is it really just editing the file to say

Code:

100000000000:P:1:2:16385

then running

Code:

../snewpgen -ol -ci=file -co=file-100-200 -osp=200000000000

(and with -ol I get to record the factors for posterity and I don't have to parse diff output :) )

I am almost certainly an idiot, but I can't work out what the option required to make two save files is - that it exists is mentioned in several places, but I can't see where in the documentation it's actually stated!

Batalov · 2017-05-16, 17:55

One thing that we have to keep in mind is that this is a tool that will not be modified by anyone except enthusiasts (well, ok, me for example). I can certify that it is quite cumbersome to recompile and that the 32-bit-ness is built into the source to the core. (If you build it as a 64-bit binary it will not work properly. If you try to build it not on 32-bit AWS instance, it will not work properly. ...almost anything, step to the left, step to the right, - and it will not work properly.)

That said, it is still quite useful.

1. Indeed, you guess was right and the first number in the input file is "sieve from here". So you simply edit it.

2. It is important to have NewPGen.ini prepared (for example by running it once, in an empty dir); then doctored to your needs and copied into work dirs. One thing that I definitely do for my origami projects is add another zero to the BitmapThreshold; another - if you want to do TwoSaveFiles, see below:

Code:

vi NewPGen.ini
TrayIcon = 1
WorkerThreadPriorityLowest = 1
Priority = Normal
BitmapThreshold = 4000000000
MinutesBetweenWrites = 30
DSM_TwoSaveFiles = 1
DSM_AlwaysSave = 0
TwoSaveFiles = 0
DSM_LogDeleted = 0

2017-05-16, 14:57	#1
fivemack (loop (#_fork)) Feb 2006 Cambridge, England 7²×131 Posts	Parallel sieving with newpgen Is there a way that I can sieve different ranges of factors (for, say, k*2^131072+1, k=2^27..2^28) on different cores? Splitting up the range across cores is pointless, because each core's still having to generate all the primes; I'd like to be able to sieve p=2e14..3e14 on one core, 3e14..4e14 on another, and merge the lists of eliminated k values afterwards.

2017-05-16, 16:46	#3
fivemack (loop (#_fork)) Feb 2006 Cambridge, England 7²·131 Posts	Does the first number on the first line of the file have to have any non-obvious congruence properties? Or is it really just editing the file to say Code: 100000000000:P:1:2:16385 then running Code: ../snewpgen -ol -ci=file -co=file-100-200 -osp=200000000000 (and with -ol I get to record the factors for posterity and I don't have to parse diff output :) ) I am almost certainly an idiot, but I can't work out what the option required to make two save files is - that it exists is mentioned in several places, but I can't see where in the documentation it's actually stated! Last fiddled with by fivemack on 2017-05-16 at 16:49

2017-05-16, 17:55	#4
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 3⁶·13 Posts	One thing that we have to keep in mind is that this is a tool that will not be modified by anyone except enthusiasts (well, ok, me for example). I can certify that it is quite cumbersome to recompile and that the 32-bit-ness is built into the source to the core. (If you build it as a 64-bit binary it will not work properly. If you try to build it not on 32-bit AWS instance, it will not work properly. ...almost anything, step to the left, step to the right, - and it will not work properly.) That said, it is still quite useful. 1. Indeed, you guess was right and the first number in the input file is "sieve from here". So you simply edit it. 2. It is important to have NewPGen.ini prepared (for example by running it once, in an empty dir); then doctored to your needs and copied into work dirs. One thing that I definitely do for my origami projects is add another zero to the BitmapThreshold; another - if you want to do TwoSaveFiles, see below: Code: vi NewPGen.ini TrayIcon = 1 WorkerThreadPriorityLowest = 1 Priority = Normal BitmapThreshold = 4000000000 MinutesBetweenWrites = 30 DSM_TwoSaveFiles = 1 DSM_AlwaysSave = 0 TwoSaveFiles = 0 DSM_LogDeleted = 0

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Deep Sieving MM49 in parallel	ET_	Operazione Doppi Mersennes	22	2016-07-28 11:23
Which bits of gmp-ecm are now parallel?	fivemack	GMP-ECM	5	2010-09-05 06:49
Faster sieving with NewPGen and srsieve	geoff	Sierpinski/Riesel Base 5	11	2006-07-10 01:10
Sieving multiple NewPGen files in a row(How?)	jasong	Software	18	2005-12-30 20:23
Sieving with NewPGen	Cruelty	15k Search	41	2005-10-27 10:28

2017-05-16, 16:31	#2
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 3⁶·13 Posts	What you described is exactly how some of us had done it for years. It is easy to script (compete with core assignments); each process would need its small separate sandbox (e.g. a folder).