mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Msieve (https://www.mersenneforum.org/forumdisplay.php?f=83)
-   -   A few polyselect questions (https://www.mersenneforum.org/showthread.php?t=27100)

EdH 2021-08-25 13:26

A few polyselect questions
 
I'm working on some scripts in which I'm doing the individual steps separately.

poly_deadline doesn't seem to be working for the individual steps even though it's listed correctly in the log. Does that only function when invoking the whole process?

I'm seting up ranges for the coefficients and some ranges immediately end. Does this mean there is no appropriate leading coefficient for this range or does the program check a small number of random coefficients and then give up?

I'm getting hundreds of thousands of -np1 hits, listed in msieve.dat.m and therefore -nps takes hours to complete, even if -np1 only ran for a few minutes. Would the minimum e value setting do any good in limiting the hits in the first stage and if so, would I use the minimum Murphy_E or something else at this point?

Thanks for any help.

chris2be8 2021-08-25 16:04

poly_deadline=X is the number of CPU seconds that particular step should run for. I find it useful for GPU assisted searching, I do -np1 and -nps in parallel, using poly_deadline to make it stop after about 2% of estimated CPU time to do the whole job. But it's not much use if you don't know how many cores msieve will actually be using.

I then sort the stage 2 hits and put the top 100 or so through -npr. I don't know any way to do that with stage 1 output since you don't have the e-score at that point.

Chris

firejuggler 2021-08-25 16:09

:unsure:

Run a few minute of -np1 -nps, then get the best e-score... re-run with the new parameter.

EdH 2021-08-25 16:31

[QUOTE=chris2be8;586489]poly_deadline=X is the number of CPU seconds that particular step should run for. I find it useful for GPU assisted searching, I do -np1 and -nps in parallel, using poly_deadline to make it stop after about 2% of estimated CPU time to do the whole job. But it's not much use if you don't know how many cores msieve will actually be using.

I then sort the stage 2 hits and put the top 100 or so through -npr. I don't know any way to do that with stage 1 output since you don't have the e-score at that point.

Chris[/QUOTE]I've been trying poly_deadline=300 and it says such in the log file[code]Tue Aug 24 15:23:29 2021 poly select deadline: 300
Tue Aug 24 15:23:29 2021 time limit set to 0.08 CPU-hours[/code]but, it never stops on its own. I have to CTRL-C or pkill msieve.

Additionally, I haven't had any success with getting more than one thread engaged. Will msieve poly selection use multi-threading?

[QUOTE=firejuggler;586491]:unsure:

Run a few minute of -np1 -nps, then get the best e-score... re-run with the new parameter.[/QUOTE]I hadn't tried doing both steps at once. I'll "give it a go."

EdH 2021-08-25 18:01

Maybe this is why it doesn't stop at 300 seconds:[code]$ ../msieve -t 12 -np "poly_deadline=300 min_coeff=0 max_coeff=1000000 polydegree=6"
[B]deadline: 8640000 CPU-seconds per coefficient[/B]
coeff 120120 specialq 1 - 1030050632 other 484419 - 1162605
aprogs: 19335 entries, 75936 roots
line minimize failed
save 1.477489e-16 -8.5039 6685374.54 9.085104e-17 rroots 4[/code]even though the log says:[code]Wed Aug 25 13:46:42 2021 poly select deadline: 300
Wed Aug 25 13:46:42 2021 time limit set to 0.08 CPU-hours[/code]So I'm looking at 1/3 of a year (CPU-time) for each coefficient instead of the, "0.08 CPU-hours?"


Edit: Also note that -t 12 did nothing (as expected). Only one thread is running.

swellman 2021-08-25 21:36

[QUOTE=EdH;586473]
I'm getting hundreds of thousands of -np1 hits, listed in msieve.dat.m and therefore -nps takes hours to complete, even if -np1 only ran for a few minutes. Would the minimum e value setting do any good in limiting the hits in the first stage and if so, would I use the minimum Murphy_E or something else at this point?

Thanks for any help.[/QUOTE]

Ed -

Are you limiting the max_stage_1_norm (and 2)? With no limits (or bad limits) defined for these values you can collect a LOT of junk.

EdH 2021-08-26 00:13

[QUOTE=swellman;586523]Ed -

Are you limiting the max_stage_1_norm (and 2)? With no limits (or bad limits) defined for these values you can collect a LOT of junk.[/QUOTE]I left them at default:[code]Wed Aug 25 13:46:42 2021 max stage 1 norm: 2.88e+28
Wed Aug 25 13:46:42 2021 max stage 2 norm: 1.12e+28[/code]because I don't know anything about them and couldn't locate anything recognizable in the docs.

Edit: Where can I find what I should use for these options?

swellman 2021-08-26 01:47

I use 5.5% of the stage1 default value, and 6% of the default stage2.

I find this gives good results with not too many polys to process. The msieve parser needs double quotes around all your arguments in the invocation line e.g.

[CODE]
msieve -v -g 0 -np1 -nps “1,1000000 stage1_norm=[value1] stage2_norm=[value2]” -s [output file name]
[/CODE]

YMMV. Others use various fractions of the state 1 and 2 norms.

VBCurtis 2021-08-26 02:15

I used to try to pick stage 1 norm tight enough to keep the -nps part just able to keep up. I used "top" to judge this- I wanted msieve to be near 100% cpu use, but not pinned at 100. This gave me some sense that size-opt was keeping up with the output of the GPU.

I pick stage 2 norm to be default divided by 30, but that doesn't give many hits on smaller inputs; one might need a looser bound for regular use and a tighter bound for large (C170+) inputs.

swellman 2021-08-26 11:30

Ed -

Just to add to the mix, I have some old notes:

Poly search 5%-12.5% of Stg1, 1%-10% of a Stg2. 5.5%/6% seems best speed/yield mix. 3%/10% works 2x faster! (But it can miss good polys.)

Note the above works for deg 5 but not really deg 6. Msieve doesn’t work great for larger GNFS, say in low 190s. I tend to use CADO for deg 6 or >C190.

FWIW.

EdH 2021-08-26 12:09

Thanks for all the help, everyone.

I still can't get poly_deadline to work, even though it says it's set in the log. As for the norms, I tried the 5.5/6% and ran a session overnight with 20 threads running different ranges. I hard stopped msieve at four hours, but that should have been roughly equal to 80 hours work. I woke this morning to an empty .ms file.

Of note, msieve is defaulting to degree 5 for this 220 digit number, so I've been setting it to degree 6.

Also, I haven't been able to get my arch 3.0 card to work with msieve (or GMP-ECM)* so I'm just running via CPUs ATM.

*CUDA 10.X is supposed to support arch 3.0, but something causes a "trickle" of CUDA 11.X to sneak in. I guess the trouble was in my keeping up with the Ubuntu OSs.

swellman 2021-08-26 12:18

What was command line for msieve-GPU?

VBCurtis 2021-08-26 16:34

CPU search on msieve is hopeless- all our norm suggestions were intended to manage the firehose of output that a GPU generates.
A CPU core finds about 2% of the raw polys of a ~2017 GPU (say, a 980 or 1070 level card), so your overnight run was maybe 90 minutes worth of GPU searching.

msieve does not ever default to degree 6.

Also, the norm guidance doesn't work so well for deg 6- I thought you were trying to automate msieve poly select for small numbers, not run a search for the 4788 C220. size-opt for degree 6 is quite slow, what with an extra term to cycle through.

EdH 2021-08-27 01:57

[QUOTE=swellman;586548]What was command line for msieve-GPU?[/QUOTE]If you mean the build command for my Msieve trials, I think it was the basic:[code]make all ECM=1 CUDA=1 (NO_ZLIB=1 - tried with and without)[/code]Msieve returned without errors, but there were no .so files, so after it started, it would say it couldn't find them. I have a thread somewhere on my failures.

VBCurtis 2021-08-27 05:14

No, he meant the invocation you used to call msieve. We've all had empty output files unexpectedly, and oftentimes there's a typo to be found in the chain-of-flags on the command line that explains the output.

Then again, sometimes it's just too-tight norm settings!

EdH 2021-08-27 12:17

[QUOTE=VBCurtis;586622]No, he meant the invocation you used to call msieve. We've all had empty output files unexpectedly, and oftentimes there's a typo to be found in the chain-of-flags on the command line that explains the output.

Then again, sometimes it's just too-tight norm settings![/QUOTE]I haven't gotten the GPU version to run, I'm trying to script something to use all the threads of my farm with Msieve as an experiment. The command is within a script. I relaxed stage2_norm and that gave me some results.

VBCurtis 2021-08-27 18:30

Well, the reason we control either stage1 or stage2 norms from the command line is because msieve's defaults are from the era before GPUs, while GPUs produce 50-100x more data from stage1.

So, if you're running CPU only, I'd relax stage1 norm to half of default, or maybe even default. That should yield enough raw polys for size-opt to have useful work to do.

EdH 2021-08-27 19:07

[QUOTE=VBCurtis;586673]Well, the reason we control either stage1 or stage2 norms from the command line is because msieve's defaults are from the era before GPUs, while GPUs produce 50-100x more data from stage1.

So, if you're running CPU only, I'd relax stage1 norm to half of default, or maybe even default. That should yield enough raw polys for size-opt to have useful work to do.[/QUOTE]But, my problem was that I was piling up too many hits from stage 1 - several hundred thousand in five minutes. Maybe I really needed to restrict stage 1 via its norm, but leave stage 2 default in place.

Another query: I'm engaging all the threads of my machines via a script and providing the ranges via the script. I find that many ranges quit almost immediately with no results and those that continue always have the same coefficients. I thought that Msieve chooses random coefficients within the given range. Is the randomness only in the multiple of the small primes? And, does this mean if the ranges are too small, there will be no appropriate coefficients within some? I've written my script to take these skipped instances into account when assigning threads.

Additionally, once Msieve has chosen a coefficient within a range, it never releases that coefficient to move to another.* Should I choose ranges based on expecting each to hold only one appropriate coefficient, perhaps 120120 (2^3*3*5*7*11*13), which has been the first coefficient for every run?

I know Msieve README.nfs says each range can be run on multiple machines and the randomness will minimize collisions, maybe my understanding of randomness is(was) incorrect and two machines (threads) will work the same coefficient, but the randomness is in the work within that coefficient. Am I learning or losing it?

* I'm guessing that's because of the "deadline: 8640000 CPU-seconds per coefficient" and it really would move on after a few months.

VBCurtis 2021-08-27 20:09

Far as I know (it has been years since I paid any attention to msieve poly select, so some of this is as vague as my memory):
Leading coeff is not randomized. Multiples of 12 at the very smallest searches, 60 often, sometimes larger. I forget if it's the larger the input size or the larger the coeff itself that determines what the msieve version of "incr" is. It also may be the case that msieve samples some property other than divisibility of small primes to filter which leading coeff's to search deeply on.

On big poly select jobs, the space of second and third coeffs within a leading coeff is so large that a slice of the space of a leading coeff is searched. For C190+, there can be 30 or more slices, so many machines can work on the same coefficient with little work duplicated.

charybdis 2021-08-27 20:31

[QUOTE=VBCurtis;586678]Leading coeff is not randomized. Multiples of 12 at the very smallest searches, 60 often, sometimes larger. I forget if it's the larger the input size or the larger the coeff itself that determines what the msieve version of "incr" is.[/QUOTE]

From a cursory glance of the code, it's 420 for degree 4 searches, and then it's 12 up to 120 digits, 60 for 121-200 digits, and 120120 for >200 digits.

EdH 2021-08-27 20:41

[QUOTE=VBCurtis;586678]. . .
On big poly select jobs, the space of second and third coeffs within a leading coeff is so large that a slice of the space of a leading coeff is searched. For C190+, there can be 30 or more slices, so many machines can work on the same coefficient with little work duplicated.[/QUOTE]Ah Ha! This gives me a definite adjustment for my experiments. I'll try to work all threads of a single machine against a single coefficient.

[QUOTE=charybdis;586681]From a cursory glance of the code, it's 420 for degree 4 searches, and then it's 12 up to 120 digits, 60 for 121-200 digits, and 120120 for >200 digits.[/QUOTE]I had noticed all the coefficients were multiples of 120120, but hesitated stating so. If I base my ranges such that each encloses a multiple, I should be able to set my scripts such that I can run lots of threads against those multiples and maybe even have some of my 8-thread machines working on the same coefficients.

Thanks!

EdH 2021-08-27 22:20

Hmm. . . I guess the random threads didn't work:[code]120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763
120120 442190148297076679 593919502199669884690565060176108763[/code]for my 24 threads. They must have all seeded tthe same or they aren't random. . .

chris2be8 2021-09-16 15:40

I've miserably failed to find a poly for this c113:
[code]
13388164029832178605174690994702863628597531043917231628629707028750338026569517034134903882342648685574248816951
[/code]

I tried several times on two systems, one with Msieve v. 1.54 (SVN 1043) and a GTX 970 and one with Msieve v. 1.54 (SVN 1030) and a GTX 760. In every case msieve.dat.ms ended up zero bytes long (my script runs -np1 -nps as the first step, then splits msieve.dat.ms and runs -npr on several cores, if it has anything to run).

Has anyone any idea if it's something about this c113 (it's part of 45^275-1, Aurifeuillian M factor)? Or how to make poly selection work? In this size range I just want something I can throw numbers at and expect them to get factored without manual intervention.

Kvasir 2021-09-16 19:27

[QUOTE=chris2be8;587976]I've miserably failed to find a poly for this c113:
[code]
13388164029832178605174690994702863628597531043917231628629707028750338026569517034134903882342648685574248816951
[/code]

I tried several times on two systems, one with Msieve v. 1.54 (SVN 1043) and a GTX 970 and one with Msieve v. 1.54 (SVN 1030) and a GTX 760. In every case msieve.dat.ms ended up zero bytes long (my script runs -np1 -nps as the first step, then splits msieve.dat.ms and runs -npr on several cores, if it has anything to run).

Has anyone any idea if it's something about this c113 (it's part of 45^275-1, Aurifeuillian M factor)? Or how to make poly selection work? In this size range I just want something I can throw numbers at and expect them to get factored without manual intervention.[/QUOTE]

I don't think there's anything special with it. Using yafu, I factored it as
[code]
P74 = 23253040074809224483773311933322964126803314914606413239037385569218598601
P39 = 575759728051043642495256241438391908351
[/code]

chris2be8 2021-09-17 15:35

Did yafu use QS or GNFS? If GNFS what version of msieve did it use? I suspect the msieve versions I tried have some bounds set wrongly for number in this size range.

I did manage to find a poly for it with Msieve v. 1.54 (SVN 1022M) and a GTX 460. So you've saved me the job of factoring it with that. But I'm more concerned that other GNFS jobs in this size range will fail and I don't want to have to manually intervene every time it happens.

Kvasir 2021-09-18 11:14

I *think* it uses msieve 1.53. At least that's what in the ggnfs_dir that yafu.ini points to.

EdH 2021-09-18 13:05

[QUOTE=Kvasir;588099]I *think* it uses msieve 1.53. At least that's what in the ggnfs_dir that yafu.ini points to.[/QUOTE]I "believe" YAFU compiles its own Msieve from the location given in the Makefile. I also think, that adding a "-v" to your YAFU operation will increase the verbosity to include the Msieve version in use.

Edit: The above is for linux. For supplied Windows binaries, I would guess you'd ned to see if the "-v" option works.


All times are UTC. The time now is 08:43.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.