CRUS sieving questions

[chalsall]

Originally Posted by storm5510

So, I begin, again. Live and learn...

It happens. From time to time. It is how we learn... 8^)

[storm5510]

So, here is what I have set up:

Base 60

Machine 1 runs -p 3 to -P 4e11
Machine 2 runs -p 4e11 to -P 650e9
Machine 3 runs -p 650e9 to -P 135e10

They all finish within an hour of each other. Once I got the first two to finish relatively close. I had to adjust the -P on the GPU machine up so it would match the first two. They finish in about 3 hours. Caveat: As the k's grow, the time will increase.

Each will produce a .abcd file with slightly different names. I can then merge them together into a single file.

Once I conform this setup works, then I can start "stacking" the k's in a single batch file. This will require more batch file editing, but they can run far longer without interaction from me.

I hope this is what everyone has in mind.

Edit: The below is what I referred to as "stacking:"

srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "36*60^n-1"
srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "1700*60^n-1"
srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "4708*60^n-1"
srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "5317*60^n-1"
srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "5611*60^n-1"

[rogue]

Originally Posted by storm5510

So, here is what I have set up:

Base 60

Machine 1 runs -p 3 to -P 4e11
Machine 2 runs -p 4e11 to -P 650e9
Machine 3 runs -p 650e9 to -P 135e10

They all finish within an hour of each other. Once I got the first two to finish relatively close. I had to adjust the -P on the GPU machine up so it would match the first two. They finish in about 3 hours. Caveat: As the k's grow, the time will increase.

Each will produce a .abcd file with slightly different names. I can then merge them together into a single file.

Once I conform this setup works, then I can start "stacking" the k's in a single batch file. This will require more batch file editing, but they can run far longer without interaction from me.

I hope this is what everyone has in mind.

Yes, something like this:

srsieve2cl -n 100e3 -N 250e3 -P 135e10 -g 16 -M 9000 -s "36*60^n-1" -s "1700*60^n-1" -s"4708*60^n-1" -s "5317*60^n-1" -s "5611*60^n-1"

or

srsieve2cl -n 100e3 -N 250e3 -P 135e10 -g 16 -M 9000 -s b60.in

where b60.in is a file where each line is a separate sequence for the same base.

If you are going to sieve the same base across multiple computers, pre-sieve to 1e9 to eliminate 90% of the factors. Use that output file as input to each machine so you get

srsieve2 -p1e9 -P4e11 -i b60_n.abcd -O f1.txt
srsieve2 -p4e11 -P8e11 -i b60_n.abcd -O f2.txt
srsieve2 -p8e11 -P12e11 -i b60_n.abcd -O f3.txt

When all of those are done:

srsieve2 -A -i b60_n.abcd -I f1.txt
srsieve2 -A -i b60_n.abcd -I f2.txt
srsieve2 -A -i b60_n.abcd -I f3.txt

srsieve2 and srsieve2cl are interchangeable, but if the machines are different, speed wise, then the range of p will differ.

My recommendation is to run different bases on different machines so that you don't need to piece together factor files. I think you will be surprised at how quickly you can sieve to 1e12 or 1e13 when you have dozens or hundreds of sequences on a GPU. I could sieve over 2000 sequences to 1e12 in less than 3 days on a GPU.

[gd_barnes]

Originally Posted by storm5510

So, here is what I have set up:

Base 60

Machine 1 runs -p 3 to -P 4e11
Machine 2 runs -p 4e11 to -P 650e9
Machine 3 runs -p 650e9 to -P 135e10

They all finish within an hour of each other. Once I got the first two to finish relatively close. I had to adjust the -P on the GPU machine up so it would match the first two. They finish in about 3 hours. Caveat: As the k's grow, the time will increase.

Each will produce a .abcd file with slightly different names. I can then merge them together into a single file.

Once I conform this setup works, then I can start "stacking" the k's in a single batch file. This will require more batch file editing, but they can run far longer without interaction from me.

I hope this is what everyone has in mind.

Edit: The below is what I referred to as "stacking:"

srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "36*60^n-1"
srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "1700*60^n-1"
srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "4708*60^n-1"
srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "5317*60^n-1"
srsieve2cl -n 100e3 -N 250e3 -p 650e9 -P 135e10 -g 16 -M 9000 -s "5611*60^n-1"

No. It is not what we have in mind. Without using the -O command, the first part of it will not even work properly. You are not creating factor files but instead sieve files. How will you be able to "merge" together sieve files for the same base that have different tests remaining? Merging them won't logically work. You have to create factor files and merge those together and then use those to remove terms from the sieve file.

Why do you insist on running one k at a time as you show in the last part of your process? We don't want to continue helping you if you insist on running one k at a time no matter how many batch jobs you set up. Please trust us that the below is the way to do this.

I will create an input file for you and spell it out as best as I can.

1. Using the attached file of k's called R60-remain.txt, run the following command on machine 1:
srsieve2cl -n 100e3 -N 250e3 -P 1e9 -g 16 -M 9000 -s R60-remain.txt

This will run quickly and is a precursor to running the big jobs next. It will create an output file. I suggest renaming it to something relevant. Let's call it R60-sieve.txt

The following assumes that you have 3 equal machines. You can tweak the sieving ranges to make them complete at about the same time.

2. Using the examples that Mark gave, run the following command on machine 1:
srsieve2cl -p1e9 -P4e11 -i R60-sieve.txt -O factors1.txt
(tweak as needed for the relevant number of cores/threads)

3. Run the following command on machine 2:
srsieve2cl -p4e11 -P8e11 -i R60-sieve.txt -O factors2.txt

4. Run the following command on machine 3:
srsieve2cl -p8e11 -P12e11 -i R60-sieve.txt -O factors3.txt

5. You now have 3 factor files factors1.txt, factors2.txt, and factors3.txt. Merge them all together. I just copy and paste each one to the end of the first one. To check yourself, add up how many total lines are in all of the files. Then after the copy and paste, make sure your final file has that many lines. You can also use a DOS command (or there's probably a myriad of other ways) to merge all of the files together. Rename that merged file as factors-all.txt

6. Run the following command:
srsieve2cl -A -i R60-sieve-txt -I factors-all.txt

This will create a new output file with all of the factors removed.

7. Rename the output file from #6 to something relevant again like R60-sieveb.txt

8. Correct the sieve depth in R60-sieveb.txt to show 120000000000 since you sieved to 12e11.

9. Run an LLR test at 60% of the n-range using approximately the median k-value in the file (60% of n=100000 to 250000 = 190000 and the median k is 12061, so use 12061*60^190000-1 in this case) to see how long a test takes.

10. Begin a short trial sieve using R60-sieveb.txt to see what the removal rate is. If you are removing factors at twice the rate of an LLR test, then you need to sieve to slightly less than double the current sieve depth. Note that this is not exact math-wise but it gets you close. If you are way off, say you are removing factors 10 times faster than the LLR test, don't automatically sieve 10 times as far. Do something less than that, say 5-7 times as far, and then see what the removal rate is again. Nothing is ever quite a straight-line when it comes to a removal rate.

11. Delete all extraneous factor or sieve files except for R60-sieveb.txt

12. Repeat steps 2 thru 11 as many times as necessary until you've reached approximately the correct optimum sieve depth. On the 2nd time through, use the new file R60-sieveb.txt. Each time you go through it, continue using the final file from step 8 as input to the new step 2.

Have you looked at some of the documentation that comes with the sieving programs, such as README? If not, that can be very helpful in understanding the difference between factor files and sieve files and how to merge and remove one from the other.

[storm5510]

@gd_barnes

OK, all of this is sailing over my head so it would probably be best to just drop the assignment. Please!

[gd_barnes]

Originally Posted by storm5510

@gd_barnes

OK, all of this is sailing over my head so it would probably be best to just drop the assignment. Please!

OK well...sorry to hear that.

What part of it is confusing you? Have you done factor removal from sieve files before? Once you've done it a couple of times, it becomes a lot easier.

On the optimum sieve depth part, if you want to skip that and just provide us with a preliminary sieve file, say, sieved to 1e12 or 5e12 or 10e12, then that would be OK too. Others can then pick up the sieve file, determine optimum sieve depth, and finish it off.

[storm5510]

Originally Posted by gd_barnes

OK well...sorry to hear that.

What part of it is confusing you? Have you done factor removal from sieve files before? Once you've done it a couple of times, it becomes a lot easier.

On the optimum sieve depth part, if you want to skip that and just provide us with a preliminary sieve file, say, sieved to 1e12 or 5e12 or 10e12, then that would be OK too. Others can then pick up the sieve file, determine optimum sieve depth, and finish it off.

Scratch my request! It was a decision made in haste because of being really tired.

What I need to do is study your last instructional post in detail, follow it, and see what I can come up with.

I had an issue with srsieve2cl which I have submitted for rogue to look into. I found a condition where it refused to start and displayed an error message.

Now, sleep!

[gd_barnes]

The instructions that I gave are essentially a turn-key operation of a complete sieving process using multiple machines. Doing all of that, the file would be completely ready for LLR testing in the most efficient way possible, both from a personal time and CPU time perspective.

Mark had suggested that you might find it easier to run one base on only one machine. That is true. But I had the impression that you wanted to use multiple machines for a single base.

If you want to skip the optimum sieve calculations, that is OK. I can guarantee that the optimum sieve depth is > 10e12, likely > 20e12. srsieve2cl is amazingly fast for multiple k's on multiple cores so it could be higher than that but I cannot say for sure without setting it up myself.

To make it worth your while, you should at least sieve to 1e12. If you can sieve to 10e12, that would be great. That's the maximum I'd suggest that you do without computing an optimum sieve depth.

There is an alternative that you might find easier that is only a little less efficient CPU-wise. Split that k-file up that I posted (R60-remain.txt) into 3 files, 1 for each machine, of 11 k's each or whatever ratio works since you have machines of varying capabilities. You still get the efficiency of multiple k's in each sieve without having to copy factor and sieve files across multiple machines, albeit somewhat less efficient.

Doing this, you'd have 3 separate sieves. You'd have separate factor files to remove from the separate sieve files each time they finish. That might make it a little easier to wrap your head around.

Then when all 3 sieves are done sieving to the same depth, you can use srfile to combine them into one big sieve file to send to us.

[storm5510]

I was not able to sleep last night so I got back up to work on this. I was up until 5 AM EST.

I took a lot from the post rogue made. I created a list as he mentioned and called it "input.txt"

36*60^n-1
1700*60^n-1
4708*60^n-1
5317*60^n-1
5611*60^n-1
6101*60^n-1
6162*60^n-1
6274*60^n-1
7060*60^n-1
7870*60^n-1

I developed my batch process as I read along, one step at a time. and testing each line individually. Below is the batch from Machine 3: 1 and 2 as different as they are not running a GPU version. The basics are the same.

@echo off
cls
srsieve2cl -n 100e3 -N 250e3 -p 1e12 -P 2e12 -g 16 -M 9000 -s input.txt
srsieve2cl -p 1e12 -P 2e12 -g 16 -M 9000 -i b60_n.abcd -O f3.txt
srsieve2cl -A -g 16 -M 9000 -i b60_n.abcd -I f3.txt

I wrote these in such a way to make them reusable by simply changing the input file. Below are the "p" settings for each.

Machine 1: -p 3 -P 45e10.
Machine 2: -p 45e10 to -P 1e12. 
Machine 3: -p 1e12 to 2e12.

All three should finish in a 3-hour window tomorrow.

[rogue]

Once you do the initial sieving to 1e9, then the first instance will start with 1e9. All three instances will the output from the the initial sieving as their input.

[storm5510]

Originally Posted by rogue

Once you do the initial sieving to 1e9, then the first instance will start with 1e9. All three instances will the output from the the initial sieving as their input.

You lose me here. I am not doing any sieving which stops at 1e9. My first batch is from 3 to 45e10.