AffinityScramble issues

willmore · 2009-09-09, 22:09

Hello, all. I'm not sure if this is the best place for this, so moderators, please feel free to move this if you feel it would be better elsewhere.

I have a Q6600 which I have running 25.9 (32 bit XP). I recently upgraded the motherboard. Before I did so, the timings of Prime95 were always stable--they varried very little from reboot to reboot. That configuration ran for some 1.5 years.

Once I changed out the motherboard, things got strange. Now, each boot seems to change how fast my tests run. I have two workers each with two threads doing first time LL tests. So, that's 2560K FFTs right now. I found this while stability testing during the overclocking testing phase of the MB install. My benchmarks showed that running the CPU at 2.8GHz was slower than running it at 2.4GHz. That struck me as odd. On a later reboot after some other tweaking, it ran *faster*.

Further investigation showed that if I altered the AffinityScramble setting, I could restore the timings to the 'best' times for that clock speed. But, on almost every reboot, I had to twiddle with the settings to get it back to good tune.

Is it possible that my new MBs BIOS is doing something strange when enumerating the cores? Why, oh, why would it do such a thing?

Is this a known issue? If not, I'd be glad to provide whatever assistance I can. I may even have a C compiler on this thing....

Cheers!

cheesehead · 2009-09-10, 01:43

I can't answer your question directly, but it might help for you to include details (brand and model designation, at least) about your previous and current motherboards. Some here may know about problems with particular motherboards.

willmore · 2009-09-10, 04:07

The old MB was just fine--at stock clocks and if I didn't use the last SATA port. I was an ECS G31T-M. It ran the Q6600 at the stock 2.4GHz with the DDR2 memory at 400MHz (DDR2-800). I think the FSB was stock--can't remember that off the top of my head.

The new MB is a Gigabyte EP45-UD3LR. DRAM runs at 533MHz (1066). CPU runs at 3.2GHz. FSB is 400MHz. Everything runs perfectly.

OS is, as mentioned, 32 bit XP (SP3). Prime95 is 25.9 build 4.

I'm running two workers, each with two threads. I do first time LL tests which are currently using 2560K FFTs. On them, I get around 28.5ms/i.

Every part of the system has been beaten on extensively--the CPU ran for a few days running the stress test, the memory ran for a few days under both memtest86 and memtest86+. No errors detected in the current config--just couldn't get it to run stresstest at 3.6GHz--darn.

I've seen in the HW forum that there has been some strangeness seen with quad cores and thread mapping. One poster had some odd issues with core number enumeration at differing clock speeds. But, I'm staying at *one* config and just rebooting the machine. So, this is a bit different. Maybe this does belong in the hardware forum. Being a computer engineer, I see the BIOS as firmware which falls on the software side of the software/hardware line, IMHO.

Thanks to anyone who can shed some light on this.

lfm · 2009-09-10, 04:24

It would be considered a BIOS bug I'd say. I can see where starting up the CPUs would result in a race condition so that the numbering is inconsistent. Its only serious performance tweekers like GIMPS people that would notice at all.

lfm · 2009-09-10, 05:34

Indeed - I have 5 different systems with various quad CPUs running Linux and on one of them /proc/cpuinfo shows the processor numbers 0, 1, 2, 3 have core id numbers 0, 1, 3, 2. The other four match normally. I'm not sure if it changes with reboots but I wouldn't be surprised.

Nelson · 2009-09-10, 08:55

This probably belongs in hardware as it relates to a particular manufacturer. Do you also have issues with the restart function? Any settings over about 1% of standard results in the settings being saved Video resets but the system doesn't restart until I use the reset switch, unless I have made the changes beginning with default. I've contacted Gigabyte about the issue but get no useful/applicable response. Seems there is a timing issue which could be BIOS, Chipset or board design. Question is which?

I have the same problem with a EP45-UD3R Board however it only occurs when I make some adjustments to voltages or clocks in the M.I.T. menu. After a second reboot (using the reset switch or shutdown restart) the affinity settles down to 0123. Immediately after a "tweak" however all bets are off.

I noticed the behaviour first when the iteration timings for my MSI P35 Neo-F Board with Q6600 were about 50% better than my new UD3R Board with Q9550. Adjusting affinity scramble helped until the next reboot then re-adjusting the Affinity scramble was necessary. There may be an issue with how XP retrieves the CPU order. I haven't been able to reproduce the problem with Windows 7 but haven't really tried making any changes lately.

the following was a summary of what was going on with XP:
Iteration tim.txt

willmore · 2009-09-10, 13:17

Nelson,

I don't have problems if I stop and restart the workers in prime95. It's always stable until I reboot.

I'm not--to my knowledge--changing any settings in the BIOS when I reboot. Maybe the PC/BIOS is doing something behind my back that I'm unaware of.

LFM,

I wonder if there is a facility in windows similar to /proc/cpuinfo is on linux. Maybe that would shed some light on my situation. I it turns out that core id numbers are the stable value, I'd still hate to ask George for yet another configuration option. I'd be more tempted to see if I could rig up a little perl script to set AffinityScramble for me. :)

Nelson · 2009-09-10, 20:43

I'm referring only to system restart not prime95. If you don't have the issue maybe it's due to the slight difference in board version. Still the Scramble problem is definitely something only a Gimpster would notice. Only if using two or more threads per worker. More than two with the core2 doesn't scale well at all. The only reason two works well is memory bandwidth is not as heavily taxed as it is with more workers.

Changing BIOS settings was what caused the rearrangement of the core sequence in my case otherwise rebooting again brought it back to normal. The first boot after changes was always unpredictable. After setting a workable scramble it would revert to normal and I'd have to redo it again. Sometimes it would not change doing a warm reboot but shutdown and turning on again would cause changes.

It may be too that Windows XP gets the ordering from BIOS and Linux doesn't depend on the BIOS I'm not sure. That would indicate that there is definitely a BIOS instruction sequence that is not getting the information correctly and may be related to my resstart issues. I installed the Test version of Windows 7 and it doesn't seem to have issues with ordering, still has restart issues though, perhaps for the same reason that the OS discovers the ordering itself rather than taking it from BIOS. I think the CPUID instruction gives the correct identity of each core when used correctly. Somebody was working on that in the v25.11 thread. Specifically in reference to i7 however the information must also be available on the core2 architecture as well. You might try that.

Check out SysInternals suite on the microsoft website. A search fo SysInternals should give you the information you need. The suite includes a number of Utilities that I use regularly. Process Explorer, PageDefrag and contig to name a few. Process explorer gives a little more information that the task manager however CPU ordering is not included only a graph of CPU usage that can be set to display each core separately but what exactly is running on each core is not clear. Other utilities may be useful to you as well.

SysInternals was written by Mark Russinovich before he was hired by Microsoft, guess the competition was too hefty.

nelson

willmore · 2009-09-11, 01:07

Nelson,

I have not looked into what the core ordering looks like in terms of what kind of start/restart the system just performed--BIOS change first, BIOS change second, warm, or cold. I guess I'll start keeping track of that. If that shows a pattern, I guess I could make a set of batch files that set the scramble right for that kind of boot: "Oh, I just warm rebooted, I guess I should click on warm.bat" :) Thanks for the hint that the restart type might effect it.

I'm guessing that XP uses the multiprocessor tables and that the new kernel in Vista and '7' uses a different one. I know that linux doesn't trust the tables and uses CPUID to explore the core layout.

And, yes, I use a bunch of the sysinternals suite all of the time. And their reason for buying him was "If you can't do better, buy them out."

I'm following the 25.11 thread about the CPUID issues. I'll see if someone involved can make a binary that just enumerates what it sees. Maybe that'll shed some light on my situation.

Thanks for the help and the suggestions!

Cheers.

willmore · 2009-10-26, 20:47

Hello, all. It's been a while since I last posted about this and nothing much has changed. I have found out one interesting bit of info. I was playing with the SysInternals Suite of tools on a new PC I'm bringing into service and I saw a little program that had escaped my notice before: Coreinfo

What it tells you is some pretty basic information:

Code:

Logical to Physical Processor Map:
*---  Physical Processor 0
-*--  Physical Processor 1
--*-  Physical Processor 2
---*  Physical Processor 3

Logical Processor to Socket Map:
****  Socket 0

Logical Processor to NUMA Node Map:
****  NUMA Node 0

Logical Processor to Cache Map:
*---  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
*---  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
-*--  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
-*--  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--*-  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
--*-  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
*-*-  Unified Cache       0, Level 2,    4 MB, Assoc  16, LineSize  64
---*  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
---*  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
-*-*  Unified Cache       1, Level 2,    4 MB, Assoc  16, LineSize  64

It's pretty boring, but take a note of the two lines under the "Logical Processor to Cache Map:" section. Specifically the ones that show the Level 2 Unified Cache to Logical Processor mapping.

Right now, my AffinityScramble is 0213. Look familiar? That would correspond to
*-*- Unified Cache 0, Level 2, 4 MB, Assoc 16, LineSize 64
-*-* Unified Cache 1, Level 2, 4 MB, Assoc 16, LineSize 64
^^^^ this part

Normally, one would expect that to look more like:
**--
--**
Which would say that the first two cores share the same physical L2 as do the third and fourth cores. But that's not what this shows.

So, this seems to be a better way for me to explicitly find out the Logical Core to Cache mapping so that I can properly set my AffinityScramble. But, better than that, it shows that there is a clean way to *programatically* set it.

Does any Windows programmer out there know if the tool just parses a standard table/function call return value or is the clever folks at SysInternals came up with this on thier own? If the latter. Maybe they can be persuaded to share the 'secret sauce' with us.

Yay, progress at last.

2009-09-09, 22:09	#1
willmore Aug 2002 61 Posts	AffinityScramble issues Hello, all. I'm not sure if this is the best place for this, so moderators, please feel free to move this if you feel it would be better elsewhere. I have a Q6600 which I have running 25.9 (32 bit XP). I recently upgraded the motherboard. Before I did so, the timings of Prime95 were always stable--they varried very little from reboot to reboot. That configuration ran for some 1.5 years. Once I changed out the motherboard, things got strange. Now, each boot seems to change how fast my tests run. I have two workers each with two threads doing first time LL tests. So, that's 2560K FFTs right now. I found this while stability testing during the overclocking testing phase of the MB install. My benchmarks showed that running the CPU at 2.8GHz was slower than running it at 2.4GHz. That struck me as odd. On a later reboot after some other tweaking, it ran faster. Further investigation showed that if I altered the AffinityScramble setting, I could restore the timings to the 'best' times for that clock speed. But, on almost every reboot, I had to twiddle with the settings to get it back to good tune. Is it possible that my new MBs BIOS is doing something strange when enumerating the cores? Why, oh, why would it do such a thing? Is this a known issue? If not, I'd be glad to provide whatever assistance I can. I may even have a C compiler on this thing.... Cheers! Last fiddled with by willmore on 2009-09-09 at 22:11 Reason: spelling errors

2009-09-10, 04:07	#3
willmore Aug 2002 61 Posts	The old MB was just fine--at stock clocks and if I didn't use the last SATA port. I was an ECS G31T-M. It ran the Q6600 at the stock 2.4GHz with the DDR2 memory at 400MHz (DDR2-800). I think the FSB was stock--can't remember that off the top of my head. The new MB is a Gigabyte EP45-UD3LR. DRAM runs at 533MHz (1066). CPU runs at 3.2GHz. FSB is 400MHz. Everything runs perfectly. OS is, as mentioned, 32 bit XP (SP3). Prime95 is 25.9 build 4. I'm running two workers, each with two threads. I do first time LL tests which are currently using 2560K FFTs. On them, I get around 28.5ms/i. Every part of the system has been beaten on extensively--the CPU ran for a few days running the stress test, the memory ran for a few days under both memtest86 and memtest86+. No errors detected in the current config--just couldn't get it to run stresstest at 3.6GHz--darn. I've seen in the HW forum that there has been some strangeness seen with quad cores and thread mapping. One poster had some odd issues with core number enumeration at differing clock speeds. But, I'm staying at one config and just rebooting the machine. So, this is a bit different. Maybe this does belong in the hardware forum. Being a computer engineer, I see the BIOS as firmware which falls on the software side of the software/hardware line, IMHO. Thanks to anyone who can shed some light on this. Last fiddled with by willmore on 2009-09-10 at 04:09 Reason: I need a personal editor

2009-09-10, 08:55	#6
Nelson Apr 2008 Regensburg..^~^..Plzeň 5×17 Posts	other issues too? This probably belongs in hardware as it relates to a particular manufacturer. Do you also have issues with the restart function? Any settings over about 1% of standard results in the settings being saved Video resets but the system doesn't restart until I use the reset switch, unless I have made the changes beginning with default. I've contacted Gigabyte about the issue but get no useful/applicable response. Seems there is a timing issue which could be BIOS, Chipset or board design. Question is which? I have the same problem with a EP45-UD3R Board however it only occurs when I make some adjustments to voltages or clocks in the M.I.T. menu. After a second reboot (using the reset switch or shutdown restart) the affinity settles down to 0123. Immediately after a "tweak" however all bets are off. I noticed the behaviour first when the iteration timings for my MSI P35 Neo-F Board with Q6600 were about 50% better than my new UD3R Board with Q9550. Adjusting affinity scramble helped until the next reboot then re-adjusting the Affinity scramble was necessary. There may be an issue with how XP retrieves the CPU order. I haven't been able to reproduce the problem with Windows 7 but haven't really tried making any changes lately. the following was a summary of what was going on with XP: Iteration tim.txt

2009-10-26, 20:47	#10
willmore Aug 2002 3D₁₆ Posts	Ahah! Sysinternals Suite Coreinfo hints... Hello, all. It's been a while since I last posted about this and nothing much has changed. I have found out one interesting bit of info. I was playing with the SysInternals Suite of tools on a new PC I'm bringing into service and I saw a little program that had escaped my notice before: Coreinfo What it tells you is some pretty basic information: Code: Logical to Physical Processor Map: --- Physical Processor 0 --- Physical Processor 1 --- Physical Processor 2 --- Physical Processor 3 Logical Processor to Socket Map: ** Socket 0 Logical Processor to NUMA Node Map: ** NUMA Node 0 Logical Processor to Cache Map: --- Data Cache 0, Level 1, 32 KB, Assoc 8, LineSize 64 --- Instruction Cache 0, Level 1, 32 KB, Assoc 8, LineSize 64 --- Data Cache 1, Level 1, 32 KB, Assoc 8, LineSize 64 --- Instruction Cache 1, Level 1, 32 KB, Assoc 8, LineSize 64 --- Data Cache 2, Level 1, 32 KB, Assoc 8, LineSize 64 --- Instruction Cache 2, Level 1, 32 KB, Assoc 8, LineSize 64 -- Unified Cache 0, Level 2, 4 MB, Assoc 16, LineSize 64 ---* Data Cache 3, Level 1, 32 KB, Assoc 8, LineSize 64 ---* Instruction Cache 3, Level 1, 32 KB, Assoc 8, LineSize 64 -- Unified Cache 1, Level 2, 4 MB, Assoc 16, LineSize 64 It's pretty boring, but take a note of the two lines under the "Logical Processor to Cache Map:" section. Specifically the ones that show the Level 2 Unified Cache to Logical Processor mapping. Right now, my AffinityScramble is 0213. Look familiar? That would correspond to -- Unified Cache 0, Level 2, 4 MB, Assoc 16, LineSize 64 -- Unified Cache 1, Level 2, 4 MB, Assoc 16, LineSize 64 ^^^^ this part Normally, one would expect that to look more like: -- -- Which would say that the first two cores share the same physical L2 as do the third and fourth cores. But that's not what this shows. So, this seems to be a better way for me to explicitly find out the Logical Core to Cache mapping so that I can properly set my AffinityScramble. But, better than that, it shows that there is a clean way to programatically set it. Does any Windows programmer out there know if the tool just parses a standard table/function call return value or is the clever folks at SysInternals came up with this on thier own? If the latter. Maybe they can be persuaded to share the 'secret sauce' with us. Yay, progress at last.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
ECM RAM issues	yoyo	GMP-ECM	7	2018-04-28 05:51
Windows 10 Issues	Sulamandora	Software	5	2015-08-13 06:47
Mersenne.org Issues	SiliconSentry	Information & Answers	3	2014-05-21 22:36
New GPU; new issues...	chalsall	GPU Computing	18	2013-06-12 19:28
Quick AffinityScramble question	Smorg	Hardware	0	2009-11-17 20:38

2009-09-10, 01:43	#2
cheesehead "Richard B. Woods" Aug 2002 Wisconsin USA 7692₁₀ Posts	I can't answer your question directly, but it might help for you to include details (brand and model designation, at least) about your previous and current motherboards. Some here may know about problems with particular motherboards.

2009-09-10, 04:24	#4
lfm Jul 2006 Calgary 5²×17 Posts	It would be considered a BIOS bug I'd say. I can see where starting up the CPUs would result in a race condition so that the numbering is inconsistent. Its only serious performance tweekers like GIMPS people that would notice at all.

2009-09-10, 05:34	#5
lfm Jul 2006 Calgary 5²×17 Posts	Indeed - I have 5 different systems with various quad CPUs running Linux and on one of them /proc/cpuinfo shows the processor numbers 0, 1, 2, 3 have core id numbers 0, 1, 3, 2. The other four match normally. I'm not sure if it changes with reboots but I wouldn't be surprised.

2009-09-10, 13:17	#7
willmore Aug 2002 61 Posts	Nelson, I don't have problems if I stop and restart the workers in prime95. It's always stable until I reboot. I'm not--to my knowledge--changing any settings in the BIOS when I reboot. Maybe the PC/BIOS is doing something behind my back that I'm unaware of. LFM, I wonder if there is a facility in windows similar to /proc/cpuinfo is on linux. Maybe that would shed some light on my situation. I it turns out that core id numbers are the stable value, I'd still hate to ask George for yet another configuration option. I'd be more tempted to see if I could rig up a little perl script to set AffinityScramble for me. :)

2009-09-10, 20:43	#8
Nelson Apr 2008 Regensburg..^~^..Plzeň 55₁₆ Posts	I'm referring only to system restart not prime95. If you don't have the issue maybe it's due to the slight difference in board version. Still the Scramble problem is definitely something only a Gimpster would notice. Only if using two or more threads per worker. More than two with the core2 doesn't scale well at all. The only reason two works well is memory bandwidth is not as heavily taxed as it is with more workers. Changing BIOS settings was what caused the rearrangement of the core sequence in my case otherwise rebooting again brought it back to normal. The first boot after changes was always unpredictable. After setting a workable scramble it would revert to normal and I'd have to redo it again. Sometimes it would not change doing a warm reboot but shutdown and turning on again would cause changes. It may be too that Windows XP gets the ordering from BIOS and Linux doesn't depend on the BIOS I'm not sure. That would indicate that there is definitely a BIOS instruction sequence that is not getting the information correctly and may be related to my resstart issues. I installed the Test version of Windows 7 and it doesn't seem to have issues with ordering, still has restart issues though, perhaps for the same reason that the OS discovers the ordering itself rather than taking it from BIOS. I think the CPUID instruction gives the correct identity of each core when used correctly. Somebody was working on that in the v25.11 thread. Specifically in reference to i7 however the information must also be available on the core2 architecture as well. You might try that. Check out SysInternals suite on the microsoft website. A search fo SysInternals should give you the information you need. The suite includes a number of Utilities that I use regularly. Process Explorer, PageDefrag and contig to name a few. Process explorer gives a little more information that the task manager however CPU ordering is not included only a graph of CPU usage that can be set to display each core separately but what exactly is running on each core is not clear. Other utilities may be useful to you as well. SysInternals was written by Mark Russinovich before he was hired by Microsoft, guess the competition was too hefty. nelson

2009-09-11, 01:07	#9
willmore Aug 2002 61₁₀ Posts	Nelson, I have not looked into what the core ordering looks like in terms of what kind of start/restart the system just performed--BIOS change first, BIOS change second, warm, or cold. I guess I'll start keeping track of that. If that shows a pattern, I guess I could make a set of batch files that set the scramble right for that kind of boot: "Oh, I just warm rebooted, I guess I should click on warm.bat" :) Thanks for the hint that the restart type might effect it. I'm guessing that XP uses the multiprocessor tables and that the new kernel in Vista and '7' uses a different one. I know that linux doesn't trust the tables and uses CPUID to explore the core layout. And, yes, I use a bunch of the sysinternals suite all of the time. And their reason for buying him was "If you can't do better, buy them out." I'm following the 25.11 thread about the CPUID issues. I'll see if someone involved can make a binary that just enumerates what it sees. Maybe that'll shed some light on my situation. Thanks for the help and the suggestions! Cheers.