Yafu - Page 37

jasonp · 2010-08-16, 22:13

QS postprocessing has not changed in literally years; the LA is much more efficient now but only for matrices much larger than QS would generate.

bsquared · 2010-08-17, 16:39

Quote:

Originally Posted by bsquared

It took a while to find one, but I now have a repeatable test case which locks up. I'll start looking into what's causing that.

The bug which was causing large SIQS jobs to hang occasionally should now be fixed in version 1.19.2.

The other issues mentioned were
1.) multi-threaded loses cores
2.) batch files don't resume as expected.

I'm not sure what to do about 1.), but I'll keep looking into it. Maybe the thread pool architecture needs to be re-thought...

As for 2.), I guess I need to be educated as to how people expect resuming work in a batchfile to behave. Do you want/expect it to pick up where it left off? Would you be ok with the program modifying the batchfile (deleting rows as they are completed, for example)? Right now, manual modification of the batchfile to remove lines which have already been completed is the correct thing to do.

em99010pepe · 2010-08-17, 19:55

Quote:

Originally Posted by bsquared

As for 2.), I guess I need to be educated as to how people expect resuming work in a batchfile to behave. Do you want/expect it to pick up where it left off? Would you be ok with the program modifying the batchfile (deleting rows as they are completed, for example)?

I expect the client to pick up the work where it left off just by deleting the numbers (row) completed so far.

xilman · 2010-08-17, 20:03

Quote:

Originally Posted by bsquared

As for 2.), I guess I need to be educated as to how people expect resuming work in a batchfile to behave. Do you want/expect it to pick up where it left off? Would you be ok with the program modifying the batchfile (deleting rows as they are completed, for example)? Right now, manual modification of the batchfile to remove lines which have already been completed is the correct thing to do.

I don't know how applicable the analogy may be, but the CWI LA "just works". That is, no-one needs to edit anything and the computation continues from the last checkpoint with no human intervention other than re-starting with the relevant checkpoint.

Paul

em99010pepe · 2010-08-18, 20:51

Quote:

Originally Posted by bsquared

The bug which was causing large SIQS jobs to hang occasionally should now be fixed in version 1.19.2.

I'm sorry but could you upload yafu-1.19.2.zip with the 64-bit versions?

bsquared · 2010-08-18, 21:00

Quote:

Originally Posted by em99010pepe

I'm sorry but could you upload yafu-1.19.2.zip with the 64-bit versions?

Sorry, I just got them from Brian Gladman today, in fact. I'm still having trouble getting my express edition 2010 on win7 to compile 64 bit code.

Anyway, they should now be in the 1.19.2 zip file for download.

em99010pepe · 2010-08-19, 21:14

Thank you. I'm running some Yafu tasks, I'll let you know more about the losing cores issue. We need to understand why it happens and fix it.

bsquared · 2010-08-19, 21:29

Quote:

Originally Posted by em99010pepe

Thank you. I'm running some Yafu tasks, I'll let you know more about the losing cores issue. We need to understand why it happens and fix it.

Agreed. Thanks, that will help.

Here is some interesting data that is maybe related.

On a machine running windows server 2008 and a nehalem based CPU (x5570), the scheduler seems to do a horrible job with yafu and performance really suffers. Looking at the task manager I see that every core is partially utilized, no matter how many threads I specify.

On a machine running windows server 2008 and a core2 based CPU (xeon 5160), the scheduler seems to do a decent job with yafu and performance is fine. Looking at the task manager I see that every core is partially utilized, same as in the nehalem case.

Is the problem that windows doesn't know the difference between a hyperthread and a physical core? Is linux smarter than this, or just lucky in the way it enumerates cores?

bsquared · 2010-08-19, 21:36

As Brian stated earlier, we can fix the scheduling issue for nehalems by programmatically assigning an affinity mask for the thread. But nothing informs this decision, so what if we assign the thread to an already loaded core? It doesn't seem like a very graceful fix, but the only other fix seems to be inside the windows scheduler. I'm open to ideas here... can we detect the utlization of a core during runtime in order to inform the affinity mask (i.e. try to hack in a scheduler within yafu)? is there a way to detect in windows whether a core is a hyperthread or not?

em99010pepe: do you see "core loss" behavior if you disable hyperthreading in the BIOS?

em99010pepe · 2010-08-19, 21:40

Quote:

Originally Posted by bsquared

em99010pepe: do you see "core loss" behavior if you disable hyperthreading in the BIOS?

My machines don't have hyperthreading.

bsquared · 2010-08-19, 21:42

Quote:

Originally Posted by em99010pepe

My machines don't have hyperthreading.

oh, sorry. nevermind.

2010-08-19, 21:14	#403
em99010pepe Sep 2004 2·5·283 Posts	Thank you. I'm running some Yafu tasks, I'll let you know more about the losing cores issue. We need to understand why it happens and fix it. Last fiddled with by em99010pepe on 2010-08-19 at 21:15

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Running YAFU via Aliqueit doesn't find yafu.ini	EdH	YAFU	8	2018-03-14 17:22
YAFU-1.34	bsquared	YAFU	119	2015-11-05 16:24
Yafu bug.	storflyt32	YAFU	2	2015-06-29 05:19
yafu-1.33	bsquared	YAFU	12	2012-11-08 04:12
yafu-1.32.1	bsquared	YAFU	21	2012-09-04 19:44

2010-08-16, 22:13	#397
jasonp Tribal Bullet Oct 2004 3543₁₀ Posts	QS postprocessing has not changed in literally years; the LA is much more efficient now but only for matrices much larger than QS would generate.

2010-08-19, 21:36	#405
bsquared "Ben" Feb 2007 2·3·587 Posts	As Brian stated earlier, we can fix the scheduling issue for nehalems by programmatically assigning an affinity mask for the thread. But nothing informs this decision, so what if we assign the thread to an already loaded core? It doesn't seem like a very graceful fix, but the only other fix seems to be inside the windows scheduler. I'm open to ideas here... can we detect the utlization of a core during runtime in order to inform the affinity mask (i.e. try to hack in a scheduler within yafu)? is there a way to detect in windows whether a core is a hyperthread or not? em99010pepe: do you see "core loss" behavior if you disable hyperthreading in the BIOS?