Here's my CPU config: Intel Core i3-3110M Dual-core (2 Threads/core)

Dual socket Xeon E5-2690 v3 (total 24c/48t), DDR4 1866MHz, Windows Server 2012 R2, P95 28.6 b1
10 iterations, had to shorten the log.

[CODE][May 22 11:23] Worker starting
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K
[May 22 11:23] p: 332220523. Time: 122.030 ms.
[May 22 11:23] p: 332220523. Time: 121.961 ms.
[May 22 11:23] Iterations: 10. Total time: 1.223 sec.
[May 22 11:23] Estimated time to complete this exponent: 470 days, 7 hours, 5 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 2 threads
[May 22 11:23] p: 332220523. Time: 62.094 ms.
[May 22 11:23] p: 332220523. Time: 62.174 ms.
[May 22 11:23] Iterations: 10. Total time: 0.623 sec.
[May 22 11:23] Estimated time to complete this exponent: 239 days, 12 hours, 16 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 3 threads
[May 22 11:23] p: 332220523. Time: 41.851 ms.
[May 22 11:23] p: 332220523. Time: 41.861 ms.
[May 22 11:23] Iterations: 10. Total time: 0.420 sec.
[May 22 11:23] Estimated time to complete this exponent: 161 days, 8 hours, 20 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 4 threads
[May 22 11:23] p: 332220523. Time: 31.730 ms.
[May 22 11:23] p: 332220523. Time: 31.771 ms.
[May 22 11:23] Iterations: 10. Total time: 0.319 sec.
[May 22 11:23] Estimated time to complete this exponent: 122 days, 13 hours, 24 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 5 threads
[May 22 11:23] p: 332220523. Time: 25.590 ms.
[May 22 11:23] p: 332220523. Time: 26.036 ms.
[May 22 11:23] Iterations: 10. Total time: 0.261 sec.
[May 22 11:23] Estimated time to complete this exponent: 100 days, 4 hours, 35 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 6 threads
[May 22 11:23] p: 332220523. Time: 21.533 ms.
[May 22 11:23] p: 332220523. Time: 21.913 ms.
[May 22 11:23] Iterations: 10. Total time: 0.220 sec.
[May 22 11:23] Estimated time to complete this exponent: 84 days, 17 hours, 11 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 7 threads
[May 22 11:23] p: 332220523. Time: 19.003 ms.
[May 22 11:23] p: 332220523. Time: 19.342 ms.
[May 22 11:23] Iterations: 10. Total time: 0.202 sec.
[May 22 11:23] Estimated time to complete this exponent: 77 days, 18 hours, 38 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 8 threads
[May 22 11:23] p: 332220523. Time: 17.213 ms.
[May 22 11:23] p: 332220523. Time: 17.579 ms.
[May 22 11:23] Iterations: 10. Total time: 0.176 sec.
[May 22 11:23] Estimated time to complete this exponent: 67 days, 19 hours, 58 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 9 threads
[May 22 11:23] p: 332220523. Time: 16.028 ms.
[May 22 11:23] p: 332220523. Time: 16.308 ms.
[May 22 11:23] Iterations: 10. Total time: 0.166 sec.
[May 22 11:23] Estimated time to complete this exponent: 63 days, 16 hours, 39 minutes.
[May 22 11:23] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 10 threads
[May 22 11:23] p: 332220523. Time: 15.189 ms.
[May 22 11:23] p: 332220523. Time: 15.479 ms.
[May 22 11:23] Iterations: 10. Total time: 0.156 sec.
[May 22 11:23] Estimated time to complete this exponent: 59 days, 23 hours, 7 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 11 threads
[May 22 11:24] p: 332220523. Time: 14.908 ms.
[May 22 11:24] p: 332220523. Time: 15.202 ms.
[May 22 11:24] Iterations: 10. Total time: 0.154 sec.
[May 22 11:24] Estimated time to complete this exponent: 59 days, 5 hours, 51 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 12 threads
[May 22 11:24] p: 332220523. Time: 14.600 ms.
[May 22 11:24] p: 332220523. Time: 14.798 ms.
[May 22 11:24] Iterations: 10. Total time: 0.153 sec.
[May 22 11:24] Estimated time to complete this exponent: 58 days, 16 hours, 51 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 13 threads
[May 22 11:24] p: 332220523. Time: 13.697 ms.
[May 22 11:24] p: 332220523. Time: 13.905 ms.
[May 22 11:24] Iterations: 10. Total time: 0.141 sec.
[May 22 11:24] Estimated time to complete this exponent: 54 days, 4 hours, 43 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 14 threads
[May 22 11:24] p: 332220523. Time: 13.027 ms.
[May 22 11:24] p: 332220523. Time: 13.659 ms.
[May 22 11:24] Iterations: 10. Total time: 0.141 sec.
[May 22 11:24] Estimated time to complete this exponent: 54 days, 8 hours, 13 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 15 threads
[May 22 11:24] p: 332220523. Time: 13.066 ms.
[May 22 11:24] p: 332220523. Time: 13.436 ms.
[May 22 11:24] Iterations: 10. Total time: 0.139 sec.
[May 22 11:24] Estimated time to complete this exponent: 53 days, 11 hours, 52 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 16 threads
[May 22 11:24] p: 332220523. Time: 13.143 ms.
[May 22 11:24] p: 332220523. Time: 13.518 ms.
[May 22 11:24] Iterations: 10. Total time: 0.148 sec.
[May 22 11:24] Estimated time to complete this exponent: 56 days, 18 hours, 13 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 17 threads
[May 22 11:24] p: 332220523. Time: 13.221 ms.
[May 22 11:24] p: 332220523. Time: 13.453 ms.
[May 22 11:24] Iterations: 10. Total time: 0.143 sec.
[May 22 11:24] Estimated time to complete this exponent: 54 days, 22 hours, 13 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 18 threads
[May 22 11:24] p: 332220523. Time: 13.256 ms.
[May 22 11:24] p: 332220523. Time: 13.609 ms.
[May 22 11:24] Iterations: 10. Total time: 0.137 sec.
[May 22 11:24] Estimated time to complete this exponent: [COLOR=red]52 days, 13 hours, 37 minutes.[/COLOR]
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 19 threads
[May 22 11:24] p: 332220523. Time: 13.319 ms.
[May 22 11:24] p: 332220523. Time: 13.633 ms.
[May 22 11:24] Iterations: 10. Total time: 0.145 sec.
[May 22 11:24] Estimated time to complete this exponent: 55 days, 14 hours, 36 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 20 threads
[May 22 11:24] p: 332220523. Time: 13.494 ms.
[May 22 11:24] p: 332220523. Time: 13.775 ms.
[May 22 11:24] Iterations: 10. Total time: 0.138 sec.
[May 22 11:24] Estimated time to complete this exponent: 53 days, 4 hours, 19 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 21 threads
[May 22 11:24] p: 332220523. Time: 13.401 ms.
[May 22 11:24] p: 332220523. Time: 13.770 ms.
[May 22 11:24] Iterations: 10. Total time: 0.148 sec.
[May 22 11:24] Estimated time to complete this exponent: 56 days, 21 hours, 43 minutes.
[May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 22 threads
[May 22 11:25] p: 332220523. Time: 13.430 ms.
[May 22 11:25] p: 332220523. Time: 13.876 ms.
[May 22 11:25] Iterations: 10. Total time: 0.143 sec.
[May 22 11:25] Estimated time to complete this exponent: 54 days, 23 hours, 20 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 23 threads
[May 22 11:25] p: 332220523. Time: 13.610 ms.
[May 22 11:25] p: 332220523. Time: 13.968 ms.
[May 22 11:25] Iterations: 10. Total time: 0.148 sec.
[May 22 11:25] Estimated time to complete this exponent: 56 days, 23 hours, 49 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 24 threads
[May 22 11:25] p: 332220523. Time: 13.430 ms.
[May 22 11:25] p: 332220523. Time: 13.880 ms.
[May 22 11:25] Iterations: 10. Total time: 0.139 sec.
[May 22 11:25] Estimated time to complete this exponent: 53 days, 8 hours, 10 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 25 threads
[May 22 11:25] p: 332220523. Time: 13.628 ms.
[May 22 11:25] p: 332220523. Time: 13.945 ms.
[May 22 11:25] Iterations: 10. Total time: 0.142 sec.
[May 22 11:25] Estimated time to complete this exponent: 54 days, 16 hours, 9 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 26 threads
[May 22 11:25] p: 332220523. Time: 13.686 ms.
[May 22 11:25] p: 332220523. Time: 13.950 ms.
[May 22 11:25] Iterations: 10. Total time: 0.140 sec.
[May 22 11:25] Estimated time to complete this exponent: 53 days, 20 hours, 0 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 27 threads
[May 22 11:25] p: 332220523. Time: 13.872 ms.
[May 22 11:25] p: 332220523. Time: 14.139 ms.
[May 22 11:25] Iterations: 10. Total time: 0.141 sec.
[May 22 11:25] Estimated time to complete this exponent: 54 days, 9 hours, 36 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 28 threads
[May 22 11:25] p: 332220523. Time: 13.684 ms.
[May 22 11:25] p: 332220523. Time: 14.018 ms.
[May 22 11:25] Iterations: 10. Total time: 0.141 sec.
[May 22 11:25] Estimated time to complete this exponent: 54 days, 1 hour, 9 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 29 threads
[May 22 11:25] p: 332220523. Time: 13.710 ms.
[May 22 11:25] p: 332220523. Time: 14.031 ms.
[May 22 11:25] Iterations: 10. Total time: 0.141 sec.
[May 22 11:25] Estimated time to complete this exponent: 54 days, 2 hours, 5 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 30 threads
[May 22 11:25] p: 332220523. Time: 13.747 ms.
[May 22 11:25] p: 332220523. Time: 14.035 ms.
[May 22 11:25] Iterations: 10. Total time: 0.141 sec.
[May 22 11:25] Estimated time to complete this exponent: 54 days, 4 hours, 32 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 31 threads
[May 22 11:25] p: 332220523. Time: 13.818 ms.
[May 22 11:25] p: 332220523. Time: 14.099 ms.
[May 22 11:25] Iterations: 10. Total time: 0.141 sec.
[May 22 11:25] Estimated time to complete this exponent: 54 days, 9 hours, 2 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 32 threads
[May 22 11:25] p: 332220523. Time: 13.883 ms.
[May 22 11:25] p: 332220523. Time: 14.123 ms.
[May 22 11:25] Iterations: 10. Total time: 0.145 sec.
[May 22 11:25] Estimated time to complete this exponent: 55 days, 15 hours, 26 minutes.
[May 22 11:25] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 33 threads
[May 22 11:25] p: 332220523. Time: 14.050 ms.
[May 22 11:25] p: 332220523. Time: 14.180 ms.
[May 22 11:25] Iterations: 10. Total time: 0.141 sec.
[May 22 11:25] Estimated time to complete this exponent: 54 days, 9 hours, 35 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 34 threads
[May 22 11:26] p: 332220523. Time: 14.102 ms.
[May 22 11:26] p: 332220523. Time: 14.117 ms.
[May 22 11:26] Iterations: 10. Total time: 0.144 sec.
[May 22 11:26] Estimated time to complete this exponent: 55 days, 10 hours, 58 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 35 threads
[May 22 11:26] p: 332220523. Time: 13.813 ms.
[May 22 11:26] p: 332220523. Time: 14.208 ms.
[May 22 11:26] Iterations: 10. Total time: 0.145 sec.
[May 22 11:26] Estimated time to complete this exponent: 55 days, 17 hours, 52 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 36 threads
[May 22 11:26] p: 332220523. Time: 14.163 ms.
[May 22 11:26] p: 332220523. Time: 14.703 ms.
[May 22 11:26] Iterations: 10. Total time: 0.148 sec.
[May 22 11:26] Estimated time to complete this exponent: 56 days, 19 hours, 24 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 37 threads
[May 22 11:26] p: 332220523. Time: 14.186 ms.
[May 22 11:26] p: 332220523. Time: 14.243 ms.
[May 22 11:26] Iterations: 10. Total time: 0.146 sec.
[May 22 11:26] Estimated time to complete this exponent: 56 days, 5 hours, 17 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 38 threads
[May 22 11:26] p: 332220523. Time: 13.934 ms.
[May 22 11:26] p: 332220523. Time: 14.211 ms.
[May 22 11:26] Iterations: 10. Total time: 0.146 sec.
[May 22 11:26] Estimated time to complete this exponent: 56 days, 6 hours, 47 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 39 threads
[May 22 11:26] p: 332220523. Time: 14.041 ms.
[May 22 11:26] p: 332220523. Time: 14.789 ms.
[May 22 11:26] Iterations: 10. Total time: 0.146 sec.
[May 22 11:26] Estimated time to complete this exponent: 56 days, 5 hours, 19 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 40 threads
[May 22 11:26] p: 332220523. Time: 13.901 ms.
[May 22 11:26] p: 332220523. Time: 14.326 ms.
[May 22 11:26] Iterations: 10. Total time: 0.144 sec.
[May 22 11:26] Estimated time to complete this exponent: 55 days, 11 hours, 31 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 41 threads
[May 22 11:26] p: 332220523. Time: 15.000 ms.
[May 22 11:26] p: 332220523. Time: 14.641 ms.
[May 22 11:26] Iterations: 10. Total time: 0.149 sec.
[May 22 11:26] Estimated time to complete this exponent: 57 days, 9 hours, 57 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 42 threads
[May 22 11:26] p: 332220523. Time: 14.043 ms.
[May 22 11:26] p: 332220523. Time: 14.311 ms.
[May 22 11:26] Iterations: 10. Total time: 0.146 sec.
[May 22 11:26] Estimated time to complete this exponent: 56 days, 0 hours, 42 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 43 threads
[May 22 11:26] p: 332220523. Time: 14.197 ms.
[May 22 11:26] p: 332220523. Time: 14.537 ms.
[May 22 11:26] Iterations: 10. Total time: 0.145 sec.
[May 22 11:26] Estimated time to complete this exponent: 55 days, 17 hours, 12 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 44 threads
[May 22 11:26] p: 332220523. Time: 14.619 ms.
[May 22 11:26] p: 332220523. Time: 14.992 ms.
[May 22 11:26] Iterations: 10. Total time: 0.151 sec.
[May 22 11:26] Estimated time to complete this exponent: 57 days, 23 hours, 39 minutes.
[May 22 11:26] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 45 threads
[May 22 11:27] p: 332220523. Time: 14.122 ms.
[May 22 11:27] p: 332220523. Time: 14.688 ms.
[May 22 11:27] Iterations: 10. Total time: 0.149 sec.
[May 22 11:27] Estimated time to complete this exponent: 57 days, 2 hours, 49 minutes.
[May 22 11:27] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 46 threads
[May 22 11:27] p: 332220523. Time: 14.076 ms.
[May 22 11:27] p: 332220523. Time: 14.591 ms.
[May 22 11:27] Iterations: 10. Total time: 0.148 sec.
[May 22 11:27] Estimated time to complete this exponent: 56 days, 18 hours, 36 minutes.
[May 22 11:27] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 47 threads
[May 22 11:27] p: 332220523. Time: 14.412 ms.
[May 22 11:27] p: 332220523. Time: 14.569 ms.
[May 22 11:27] Iterations: 10. Total time: 0.149 sec.
[May 22 11:27] Estimated time to complete this exponent: 57 days, 5 hours, 56 minutes.
[May 22 11:27] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 48 threads
[May 22 11:27] p: 332220523. Time: 14.113 ms.
[May 22 11:27] p: 332220523. Time: 14.788 ms.
[May 22 11:27] Iterations: 10. Total time: 0.149 sec.
[May 22 11:27] Estimated time to complete this exponent: 57 days, 3 hours, 38 minutes.
[May 22 11:27] Worker stopped.
[/CODE]

[QUOTE=LookAS;402991]Dual socket Xeon E5-2690 v3 (total 24c/48t), DDR4 1866MHz, Windows Server 2012 R2, P95 28.6 b1
10 iterations, had to shorten the log.

[CODE][May 22 11:23][May 22 11:24] Using FMA3 FFT length 18M, Pass1=1536, Pass2=12K, 18 threads
[May 22 11:24] p: 332220523. Time: 13.256 ms.
[May 22 11:24] p: 332220523. Time: 13.609 ms.
[May 22 11:24] Iterations: 10. Total time: 0.137 sec.
[May 22 11:24] Estimated time to complete this exponent: [COLOR=red]52 days, 13 hours, 37 minutes.[/COLOR]
[/CODE][/QUOTE]

Interesting that it peaked at 18 threads. I do wonder though, in the benchmarking when hyperthreads are enabled on a system, does that "18 threads" consist of 9 physical and 9 HT cores, or how exactly does it split them up?

When I was benchmarking my dual E5-2690 v2 (just 10 cores) I did it old school. Set my affinity scramble and other options to make sure it was only running on physical cores and then just let it process for about 1000 iterations with different core counts.

There's an undoc option to disable testing HT cores in the benchmark. I don't know if the "Time" test will honor those benchmark settings or not though:
BenchHyperthreads=0

(That's assuming the system can properly identify your HT cores, which I seemed to have some problems with on some systems.)

Maybe in a few months I'll have a new system with dual E5-2697 v3 CPUs (14-core, 2.6 GHz) so it'll be interesting to see how that does.

At least on my 10-core systems, it would improve the per-iteration times up through all 10 physical cores, and even improve a bit more using an 11th one on the other CPU, but beyond that the QPI link probably became saturated. That's why I'm suspecting the 18-thread peak time you saw may consist of a mix of physical/HT. Then again, the QPI links on Haswell are faster than Ivy Bridge.

Multi-socket systems ... it matters which socket you use
 

Here's a weird thing I've noticed lately:

When testing larger exponents on a system with 2 CPU's, it actually makes a difference which socket I run the test on. I suppose it's true for smaller exponents too but it's harder to notice.

Maybe this is specific to HP Proliant servers, I don't know.

For example, when testing a 67M exponent on a dual 10-core system, on all the cores of CPU #1 it's faster by about 2 hours than when running on all the cores of CPU #2.

Weird? I'll chalk that up to something on the hardware side... Prime95 won't care. But I have noticed it on multiple systems now that it caught my attention. All of them are setup so each CPU has identical memory in it's banks, so I'm pretty sure that's not it.

Anyway, if you're doing big crunching on a dual-socket system, try both of them and see if one actually runs faster than the other. In my case it was always the first CPU that did better. Go figure.

[QUOTE=Madpoo;403852]Here's a weird thing I've noticed lately:

When testing larger exponents on a system with 2 CPU's, it actually makes a difference which socket I run the test on. I suppose it's true for smaller exponents too but it's harder to notice.

Maybe this is specific to HP Proliant servers, I don't know.

For example, when testing a 67M exponent on a dual 10-core system, on all the cores of CPU #1 it's faster by about 2 hours than when running on all the cores of CPU #2.

Weird? I'll chalk that up to something on the hardware side... Prime95 won't care. But I have noticed it on multiple systems now that it caught my attention. All of them are setup so each CPU has identical memory in it's banks, so I'm pretty sure that's not it.

Anyway, if you're doing big crunching on a dual-socket system, try both of them and see if one actually runs faster than the other. In my case it was always the first CPU that did better. Go figure.[/QUOTE]Interesting. But I wonder if it could be software only/also. It might be that one CPU is handling system functions like SMM and OS interrupts etc. while the other has no such responsibilities. Perhaps even a different OS would give different results.

It goes for GPUs too, mfaktc is faster on the one which is the first in the list. I knew that "for years", and I found it in two systems already, one with 2x580 and one with 3x580, the first card produced few GHzD more that the others. As the first card was also the display card at that time, I tried to switch the cards and switch the display outputs to other cards, and yet, the first card was faster (like ~430-440, against ~420-430 of the other cards). The only thing that "kept" with the card was the temperature: due to the position of the slots, the first card was also the coolest (2-3 degrees) so I always assumed that was temperature-related. Now, as you describe, it could be chain-related too... I don't know, but it may worth a try to look for your temperatures, "just in case". All these newer and cleverer CPUs are also "trickyer", they do "minuscule" throttling :wink: if they get hot

[QUOTE=retina;403853]Interesting. But I wonder if it could be software only/also. It might be that one CPU is handling system functions like SMM and OS interrupts etc. while the other has no such responsibilities. Perhaps even a different OS would give different results.[/QUOTE]

Maybe. It would take more analysis than I care about right now. :smile: From what I could tell though, the system wasn't doing any unusual stuff on CPU #2. The disk activity isn't huge on the one I looked at in particular and there's not a lot of interrupt handling in general. Plus I think that the array controller is affined to CPU #1 so if anything I'd expect that one to be slower, but no, it's faster.

Well, it's a mystery. It'd be nice to get some extra performance out of CPU #2 if I knew what was happening. Maybe next time I do maintenance and can afford "quiet time" with all services stopped, I can do a quick test for a few minutes.

After all, it could be something else running on the system that just happens to prefer CPU #2 or the memory on that NUMA node, for whatever odd reason. I've seen it on multiple systems but then I'm a stickler for consistency and I have each of these bad boys configured the same.

[QUOTE=LaurV;403857]...I don't know, but it may worth a try to look for your temperatures, "just in case". All these newer and cleverer CPUs are also "trickyer", they do "minuscule" throttling :wink: if they get hot[/QUOTE]

Good idea. I have the servers setup for max performance which would ideally have them all doing full turbo mode all the time. On an E5-2690 V2 that's 3x133MHz. I guess if one CPU or the other was a little hotter it might reduce the turbo boost to 2x... I monitor all of the temps every 10-15 minutes so I could check the graph and see if the CPU temps show any variation.

On these systems, both CPU's pull air through the drive cage up front, over the CPU's and memory. CPU #2 then tends to airflow out past the power supplies, while CPU #1 has the PCI slots behind it which, who knows, maybe helps keep it cooler or whatever.

Hmm... yeah, in fact, looking at one of them now, I see CPU #1 is at 104 F and CPU #2 is 129 F. The diff of 15 F could do it? The memory temps (8 different points) range from 90-99 F and are probably fine. For everyone outside the US, that's 40C and 54C for the CPU temps. :smile: Well within normal operating range, but it wouldn't surprise me if it did throttle a bit more at 54C than it would at 40C.

I know this will sound like bragging, but an HP DL380p Gen8 server has 40 different temp sensors inside... it's crazy. But helpful crazy. :smile:

I don't have any utils that show the C-states of the CPU in real time. Not even sure what's available, besides CPU-Z I guess. Maybe WMI has the info and I could create a custom poller into our collection system.

[QUOTE=Madpoo;403861]... it wouldn't surprise me if it did throttle a bit more at 54C than it would at 40C.[/QUOTE]It would surprise me if any CPU is throttling at only 54C. I can touch 54C with my hand without too much discomfort, the CPUs should be very happy at that temperature. I would think if they get to 90C+ then thermal throttling could be a consideration.

[QUOTE=Madpoo;403861]Hmm... yeah, in fact, looking at one of them now, I see CPU #1 is at 104 F and CPU #2 is 129 F. The diff of 15 F could do it? The memory temps (8 different points) range from 90-99 F and are probably fine. For everyone outside the US, that's 40C and 54C for the CPU temps. :smile: Well within normal operating range, but it wouldn't surprise me if it did throttle a bit more at 54C than it would at 40C.[/QUOTE]

A plot of the clockspeeds should clear it up.

[QUOTE=retina;403862]It would surprise me if any CPU is throttling at only 54C. I can touch 54C with my hand without too much discomfort, the CPUs should be very happy at that temperature. I would think if they get to 90C+ then thermal throttling could be a consideration.[/QUOTE]

Yeah, I know what you mean... I've been outside in temps that were that high (130 degrees F in Death Valley, California).

Is it enough to throttle? Yeah, who knows. The difference in performance I see is, for example:
3456K FFT size (65.8M sized exponent)
2.9 ms/iter on CPU #1
3.1 ms/iter on CPU #2

What is that... like 6.5% increase in speed on CPU #1 over CPU #2?

The stock clock of the chip is 3.0 GHz, and with all 10 cores enabled the max turbo boost is 3 x 133MHz.

If I assume CPU #1 is going at a full 3.4 GHz and CPU #2 is perhaps throttled to 2x boost at 3.26 GHz that's only a 4% difference. Maybe it's even throttled to a 1x boost for 3.13 GHz which is ~8% slower than 3.4.

I don't know... maybe... when I find the time I'll use CPU-Z to at least get a snapshot of the CPU speed of each.