A few polyselect questions

EdH · 2021-08-25, 13:26

I'm working on some scripts in which I'm doing the individual steps separately.

poly_deadline doesn't seem to be working for the individual steps even though it's listed correctly in the log. Does that only function when invoking the whole process?

I'm seting up ranges for the coefficients and some ranges immediately end. Does this mean there is no appropriate leading coefficient for this range or does the program check a small number of random coefficients and then give up?

I'm getting hundreds of thousands of -np1 hits, listed in msieve.dat.m and therefore -nps takes hours to complete, even if -np1 only ran for a few minutes. Would the minimum e value setting do any good in limiting the hits in the first stage and if so, would I use the minimum Murphy_E or something else at this point?

Thanks for any help.

chris2be8 · 2021-08-25, 16:04

poly_deadline=X is the number of CPU seconds that particular step should run for. I find it useful for GPU assisted searching, I do -np1 and -nps in parallel, using poly_deadline to make it stop after about 2% of estimated CPU time to do the whole job. But it's not much use if you don't know how many cores msieve will actually be using.

I then sort the stage 2 hits and put the top 100 or so through -npr. I don't know any way to do that with stage 1 output since you don't have the e-score at that point.

Chris

firejuggler · 2021-08-25, 16:09

Run a few minute of -np1 -nps, then get the best e-score... re-run with the new parameter.

EdH · 2021-08-25, 16:31

Quote:

Originally Posted by chris2be8

poly_deadline=X is the number of CPU seconds that particular step should run for. I find it useful for GPU assisted searching, I do -np1 and -nps in parallel, using poly_deadline to make it stop after about 2% of estimated CPU time to do the whole job. But it's not much use if you don't know how many cores msieve will actually be using.

I then sort the stage 2 hits and put the top 100 or so through -npr. I don't know any way to do that with stage 1 output since you don't have the e-score at that point.

Chris

I've been trying poly_deadline=300 and it says such in the log file

Code:

Tue Aug 24 15:23:29 2021  poly select deadline: 300
Tue Aug 24 15:23:29 2021  time limit set to 0.08 CPU-hours

but, it never stops on its own. I have to CTRL-C or pkill msieve.

Additionally, I haven't had any success with getting more than one thread engaged. Will msieve poly selection use multi-threading?

Quote:

Originally Posted by firejuggler

Run a few minute of -np1 -nps, then get the best e-score... re-run with the new parameter.

I hadn't tried doing both steps at once. I'll "give it a go."

EdH · 2021-08-25, 18:01

Maybe this is why it doesn't stop at 300 seconds:

Code:

$ ../msieve -t 12 -np "poly_deadline=300 min_coeff=0 max_coeff=1000000 polydegree=6"
deadline: 8640000 CPU-seconds per coefficient
coeff 120120 specialq 1 - 1030050632 other 484419 - 1162605
aprogs: 19335 entries, 75936 roots
line minimize failed
save 1.477489e-16 -8.5039 6685374.54 9.085104e-17 rroots 4

even though the log says:

Code:

Wed Aug 25 13:46:42 2021  poly select deadline: 300
 Wed Aug 25 13:46:42 2021  time limit set to 0.08 CPU-hours

So I'm looking at 1/3 of a year (CPU-time) for each coefficient instead of the, "0.08 CPU-hours?"

Edit: Also note that -t 12 did nothing (as expected). Only one thread is running.

swellman · 2021-08-25, 21:36

Quote:

Originally Posted by EdH

I'm getting hundreds of thousands of -np1 hits, listed in msieve.dat.m and therefore -nps takes hours to complete, even if -np1 only ran for a few minutes. Would the minimum e value setting do any good in limiting the hits in the first stage and if so, would I use the minimum Murphy_E or something else at this point?

Thanks for any help.

Ed -

Are you limiting the max_stage_1_norm (and 2)? With no limits (or bad limits) defined for these values you can collect a LOT of junk.

EdH · 2021-08-26, 00:13

Quote:

Originally Posted by swellman

Ed -

Are you limiting the max_stage_1_norm (and 2)? With no limits (or bad limits) defined for these values you can collect a LOT of junk.

I left them at default:

Code:

Wed Aug 25 13:46:42 2021  max stage 1 norm: 2.88e+28
 Wed Aug 25 13:46:42 2021  max stage 2 norm: 1.12e+28

because I don't know anything about them and couldn't locate anything recognizable in the docs.

Edit: Where can I find what I should use for these options?

swellman · 2021-08-26, 01:47

I use 5.5% of the stage1 default value, and 6% of the default stage2.

I find this gives good results with not too many polys to process. The msieve parser needs double quotes around all your arguments in the invocation line e.g.

Code:

msieve -v -g 0 -np1 -nps “1,1000000 stage1_norm=[value1] stage2_norm=[value2]” -s [output file name]

YMMV. Others use various fractions of the state 1 and 2 norms.

VBCurtis · 2021-08-26, 02:15

I used to try to pick stage 1 norm tight enough to keep the -nps part just able to keep up. I used "top" to judge this- I wanted msieve to be near 100% cpu use, but not pinned at 100. This gave me some sense that size-opt was keeping up with the output of the GPU.

I pick stage 2 norm to be default divided by 30, but that doesn't give many hits on smaller inputs; one might need a looser bound for regular use and a tighter bound for large (C170+) inputs.

swellman · 2021-08-26, 11:30

Ed -

Just to add to the mix, I have some old notes:

Poly search 5%-12.5% of Stg1, 1%-10% of a Stg2. 5.5%/6% seems best speed/yield mix. 3%/10% works 2x faster! (But it can miss good polys.)

Note the above works for deg 5 but not really deg 6. Msieve doesn’t work great for larger GNFS, say in low 190s. I tend to use CADO for deg 6 or >C190.

FWIW.

EdH · 2021-08-26, 12:09

Thanks for all the help, everyone.

I still can't get poly_deadline to work, even though it says it's set in the log. As for the norms, I tried the 5.5/6% and ran a session overnight with 20 threads running different ranges. I hard stopped msieve at four hours, but that should have been roughly equal to 80 hours work. I woke this morning to an empty .ms file.

Of note, msieve is defaulting to degree 5 for this 220 digit number, so I've been setting it to degree 6.

Also, I haven't been able to get my arch 3.0 card to work with msieve (or GMP-ECM)* so I'm just running via CPUs ATM.

*CUDA 10.X is supposed to support arch 3.0, but something causes a "trickle" of CUDA 11.X to sneak in. I guess the trouble was in my keeping up with the Ubuntu OSs.

2021-08-25, 13:26	#1
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2²·7·197 Posts	A few polyselect questions I'm working on some scripts in which I'm doing the individual steps separately. poly_deadline doesn't seem to be working for the individual steps even though it's listed correctly in the log. Does that only function when invoking the whole process? I'm seting up ranges for the coefficients and some ranges immediately end. Does this mean there is no appropriate leading coefficient for this range or does the program check a small number of random coefficients and then give up? I'm getting hundreds of thousands of -np1 hits, listed in msieve.dat.m and therefore -nps takes hours to complete, even if -np1 only ran for a few minutes. Would the minimum e value setting do any good in limiting the hits in the first stage and if so, would I use the minimum Murphy_E or something else at this point? Thanks for any help.

2021-08-26, 01:47	#8
swellman Jun 2012 3·1,327 Posts	I use 5.5% of the stage1 default value, and 6% of the default stage2. I find this gives good results with not too many polys to process. The msieve parser needs double quotes around all your arguments in the invocation line e.g. Code: msieve -v -g 0 -np1 -nps “1,1000000 stage1_norm=[value1] stage2_norm=[value2]” -s [output file name] YMMV. Others use various fractions of the state 1 and 2 norms. Last fiddled with by swellman on 2021-08-26 at 01:49

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Msieve failed to polyselect	firejuggler	Msieve	3	2018-10-01 13:47
Base method of montgomery polyselect implementation	csg	Homework Help	2	2017-12-13 07:04
Playing with CADO's individual polyselect binaries	Dubslow	CADO-NFS	22	2016-07-04 19:54
To CPU or to GPU: stage1 polyselect research	Karl M Johnson	Msieve	8	2011-11-12 19:46
Questions	joblack	Math	10	2009-05-01 19:52

2021-08-25, 16:04	#2
chris2be8 Sep 2009 11×223 Posts	poly_deadline=X is the number of CPU seconds that particular step should run for. I find it useful for GPU assisted searching, I do -np1 and -nps in parallel, using poly_deadline to make it stop after about 2% of estimated CPU time to do the whole job. But it's not much use if you don't know how many cores msieve will actually be using. I then sort the stage 2 hits and put the top 100 or so through -npr. I don't know any way to do that with stage 1 output since you don't have the e-score at that point. Chris

2021-08-25, 16:09	#3
firejuggler "Vincent" Apr 2010 Over the rainbow 2×3×5×97 Posts	Run a few minute of -np1 -nps, then get the best e-score... re-run with the new parameter.

2021-08-26, 02:15	#9
VBCurtis "Curtis" Feb 2005 Riverside, CA 13242₈ Posts	I used to try to pick stage 1 norm tight enough to keep the -nps part just able to keep up. I used "top" to judge this- I wanted msieve to be near 100% cpu use, but not pinned at 100. This gave me some sense that size-opt was keeping up with the output of the GPU. I pick stage 2 norm to be default divided by 30, but that doesn't give many hits on smaller inputs; one might need a looser bound for regular use and a tighter bound for large (C170+) inputs.

2021-08-26, 11:30	#10
swellman Jun 2012 3·1,327 Posts	Ed - Just to add to the mix, I have some old notes: Poly search 5%-12.5% of Stg1, 1%-10% of a Stg2. 5.5%/6% seems best speed/yield mix. 3%/10% works 2x faster! (But it can miss good polys.) Note the above works for deg 5 but not really deg 6. Msieve doesn’t work great for larger GNFS, say in low 190s. I tend to use CADO for deg 6 or >C190. FWIW.

2021-08-26, 12:09	#11
EdH "Ed Hall" Dec 2009 Adirondack Mtns 1010110001100₂ Posts	Thanks for all the help, everyone. I still can't get poly_deadline to work, even though it says it's set in the log. As for the norms, I tried the 5.5/6% and ran a session overnight with 20 threads running different ranges. I hard stopped msieve at four hours, but that should have been roughly equal to 80 hours work. I woke this morning to an empty .ms file. Of note, msieve is defaulting to degree 5 for this 220 digit number, so I've been setting it to degree 6. Also, I haven't been able to get my arch 3.0 card to work with msieve (or GMP-ECM)* so I'm just running via CPUs ATM. *CUDA 10.X is supposed to support arch 3.0, but something causes a "trickle" of CUDA 11.X to sneak in. I guess the trouble was in my keeping up with the Ubuntu OSs.