mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > YAFU

Reply
 
Thread Tools
Old 2021-04-08, 20:02   #12
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

43·83 Posts
Default

Quote:
Originally Posted by charybdis View Post
Perhaps it's time to put a new precompiled Windows binary on sourceforge? The current one is 8 years old.
I have been working toward this for months now... and it's finally here! See here.
bsquared is offline   Reply With Quote
Old 2021-04-09, 00:12   #13
Kvasir
 
Dec 2020

37 Posts
Default

The yafu I have doesn't accept psearch avg or psearch good; only psearch fast, wide or deep.
Kvasir is offline   Reply With Quote
Old 2021-04-09, 00:59   #14
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

43·83 Posts
Default

Quote:
Originally Posted by Kvasir View Post
The yafu I have doesn't accept psearch avg or psearch good; only psearch fast, wide or deep.
Go to https://github.com/bbuhrow/yafu and try the new exe.
bsquared is offline   Reply With Quote
Old 2021-07-10, 15:47   #15
bur
 
bur's Avatar
 
Aug 2020
5*10398e-4;3*2539e-3

1100001012 Posts
Default

Recently I find that yafu takes ages on poly selection. I just aborted polyselect on a c119 after one hour. In yafu.ini I have psearch=fast, but the same happened often with psearch=avg.


I noticed that polyselect runs on less and less threads, in the end it's only one threads left which tends to take very long to complete. Is there anything that can be done about it?


It's the linux build, yafu 2.03
bur is offline   Reply With Quote
Old 2021-07-10, 16:06   #16
charybdis
 
charybdis's Avatar
 
Apr 2020

463 Posts
Default

This isn't a perfect solution, but I set pbatch=60 in yafu.ini so that each thread gets smaller chunks of work.
charybdis is offline   Reply With Quote
Old 2021-07-10, 19:03   #17
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

DF116 Posts
Default

I thought that was what the change from a few weeks ago tried to address... making the deadline per coefficient much smaller. Assuming you have that version, I guess it doesn't quite work or needs more adjustment for numbers of that size. If I get some time I'll try some more tests.

Making the batch size smaller should help.
bsquared is offline   Reply With Quote
Old 2021-07-11, 03:46   #18
LaurV
Romulan Interpreter
 
LaurV's Avatar
 
Jun 2011
Thailand

3×3,251 Posts
Default

I work around this (for years by now) by making poly selection in less cores. What I witnessed is the fact that when many cores are used, like 10, 12, 16, or 18, then there are EVERY time 2 or 3 cores that are "lazier". While the rest of the cores finish their job in the promised amount of time, the lazier cores will take a lot more, sometimes more than double, and the main problem is that there is NO WAY (known to me) to resume or to interrupt without losing the whole job. Sometimes you can see (from the temporary files remaining in the folder, because yafu will clean the files for the cores that finished the work) which cores remained and you can stop with ctrl+c, and you restart, and if lucky (enough polys in the file, or whatever) yafu will move to sieving. But this is extremely risky business, if it doesn't move to sieving it will start from scratch.

So, my "plan B" is to run the poly selection in 4 or 6 cores - it will take proportionally longer, but I have the other cores free to do other jobs, or to search polys for more composites in the same time, if I have enough of them collected (different yafu copies in different folders). When less cores are used, they will all finish in the promised amount of time, with a higher probability, or there can be one core taking a bit longer, but I can avoid the situation where most of the cores sit idle for one day or longer.

When I am really in hurry, I will ask Max to get a poly for me, which he did repeatedly, or I will ask on the public forum, but this method doesn't always work. For example, last adventure was getting a poly for aliq_814848 i2066 C156, which pissed me off sooo much. It took yafu longer than 2 days (it says in factor log that needs 60000 seconds in 18 cores, which is about 16 hours) with three different trials, and I still didn't get any useful poly, because during the monsoon season in Thai, we have rain and thunderstorms and electricity breaks every day, and I had to repeatedly restart. So I asked Max and he provided me a poly in an hour or so, and I wondered wadahak server he has there with a million cores, to which he replied that "it took cado 30 minutes to generate, in 4 cores". He almost made me install cado, except I seem to be too stupid and have too less time to be able to handle anything else except yafu (win7 OS here, still). I still dream to be able to run msieve version which can generate polys using the GPU, as my CPU resources are very scarce in comparison, but never found the time to go through the initial setup process (compile? install? learning curve? what?). But anyhow, Max's poly didn't sieve right either, so I just decided this C156 is cursed, and didn't bother him again, I just searched for another poly "normal way", in 6 cores, in a smaller computer with a UPS (I do not have such a large UPS to keep alive the big machines, such toys are bloody expensive), which took three days (and guess what? the freaking electricity didn't break even once, in the three days involved!), and now the C156 is in sieving in the 18-cores machine (about 16M relations from the 44M needed, about two days to go).

That was just an example. A possibility to resume poly selection, or to just stop it and tell yafu "I am done with you, just select the best poly from what you already filed up to now, and move to sieving" (assuming you, the user, know what you are doing), should worth millions in this corner of the flat earth But I know your time is limited too.

Edit: Just for clarity, I am still using this, as it is by far the fastest in my machines, from what I tried. I didn't, however, tried "most newest" versions. So, if my "problem" is already solved, please enlighten me.
Click image for larger version

Name:	yafu0.png
Views:	28
Size:	17.5 KB
ID:	25256

Last fiddled with by LaurV on 2021-07-11 at 04:43
LaurV is offline   Reply With Quote
Old 2021-07-11, 04:57   #19
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

67618 Posts
Default

As best as I can tell, here is the situation:

Msieve poly select has two deadlines, one per coefficient and one per job. The coefficient deadline is huge, 100 days or something, so if a coefficient gets stuck it will stay stuck until the job deadline expires. I think that is why some threads get lazy, as you say... they are waiting for the job deadline to expire. Prior to the latest change I made, that deadline was msieve's internal job deadline, which for c120's is 4 hours or so and for c156 I think 300+ hours. Yafu's "fast" mode divides the job deadline by the number of threads, but msieve itself isn't threaded and doesn't know anything about this "fast" deadline so if a coefficient gets stuck it still stays stuck until the 300 hour (for example) job deadline expires. At least that's what I think was happening.

In the most recent change for yafu (made a couple weeks ago or so), I was able to modify msieve's job deadline to match yafu's divided down "fast" deadline, so if a coefficient gets stuck it should expire close to the same time as the rest of the threads. At the time I ran a bunch of tests on C110's and C120's with lots of threads and things worked much better. This is yafu 2.03, although that version number may not have changed for every update (I am not so good at remembering to do that every time). So bur's 2.03 may or may not have that change in it.

One thing I have mentioned already, and which I suspect bur might be seeing too, is that for some input sizes the quality demands in msieve's poly select seem very strict. I have run 80-thread poly select for 15 minutes or more on smallish numbers (don't remember exactly but less than c120), i.e. dozens of thread hours, without msieve reporting a single poly it thinks is good enough, even though there are hundreds of polys below its threshold that will take that 80-thread monster cpu 10 minutes to sieve. I am not trying to complain, and I would like to be able to help, but I haven't and won't have time to figure out better numbers for msieve to use. For most size inputs it is fine... but for other sizes it seems really strict. I suspect most people don't even notice because most people seem to use gpu's or cado now.

Long story short: the newest yafu is trying to get better at this for those of us (me) that still only use msieve's cpu poly select, but I am of course still heavily leaning on msieve which has all of the actual smarts and does all of the actual work. LaurV your idea to have a ctrl-C during poly select do a consolidation and save the best current poly is a good one, I can look into that.
bsquared is offline   Reply With Quote
Old 2021-08-04, 09:14   #20
bur
 
bur's Avatar
 
Aug 2020
5*10398e-4;3*2539e-3

1100001012 Posts
Default

To be honest, I'm not 100% sure I have the newest version since I am not really good with updating via repositories. Do I just follow the same steps as for a fresh install?

I'm currently running a lengthy ECM, but as soon as the next B1 value is finished (about 10 h) I can abort and test whether it was just an outdated version.
bur is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Poly select and planning for 2,2330M VBCurtis Cunningham Tables 70 2021-02-16 06:32
Poly select and planning for 2,2210M swellman Cunningham Tables 51 2020-03-22 22:09
YAFU Poly Select Deadline amphoria YAFU 22 2016-09-17 09:47
msieve poly select: choosing Stage1norm VBCurtis Msieve 0 2016-04-11 21:33
Starting NFS skipping poly select jux YAFU 5 2016-01-02 01:01

All times are UTC. The time now is 14:22.


Tue Sep 21 14:22:20 UTC 2021 up 60 days, 8:51, 0 users, load averages: 1.22, 1.50, 1.61

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.