[QUOTE=Madpoo;398562]Very thorough. So the "ocRAM" corresponds to... ? I'd guess overclock RAM but I know on the HP's you really can't. :)[/QUOTE]

You can on the gen8's. went from 1600 to 1833. Let me know what you'd like me to try on that other 32-80 core thread and I'll give it a shot tomorrow. Kind of experiencing a lull at work right now anyway.

[QUOTE=aurashift;398563]You can on the gen8's. went from 1600 to 1833. Let me know what you'd like me to try on that other 32-80 core thread and I'll give it a shot tomorrow. Kind of experiencing a lull at work right now anyway.[/QUOTE]

Yeah, I just got done looking at one of my Gen8's.

I could set it to high performance mode through ILO no problem, no reboot needed. The time test I did before and after showed a huge difference and I was pretty psyched about it, but then I realized in the regular tests those workers were doing, there wasn't as big a difference.

I think with only 5 iterations like you had done for the tests, if you're in "balanced performance" mode, there's just not enough time for the cores to go into c0 state and kick into turbo mode before it's done? Just guessing. If I'd tried it for more like 1000 iterations maybe I'd see the sustained use drive it into c0.

At any rate, in high performance mode it "sticks" in turbo mode, dependent only on thermal constraints, and it's doing pretty well.

One test that was showing roughly 17 days, 16 hours is now showing 16 days, 21 hours. So yeah, it's a pretty nice bump. (exponent in the 191M range).

I rebooted and looked around in the BIOS. I couldn't find the thing it mentioned in the doc about switching to 1x memory refresh mode or whatever. Maybe that's not available on the 300 series Gen8, only 500 series? Or Gen9?

While I was in there I put it into max cooling since I now have a space heater in the datacenter. I noticed it's ambient temp reading went up by about 10 degrees F before I did that.

I also set the memory mode to max performance instead of power saving. I think all my RAM is 1.5V anyway (I don't have any LV dimms, I don't think), but just in case.

I'm going to keep an eye on this server for a few days, running like that, full bore. I mean, it's designed for the stress... I'm glad it's in a colo somewhere and not on my desk because with all those fans going they can sound like a jet engine revving up...I'm sure you know exactly what I mean. :smile:

EDIT: Oh, regarding the mem speed, I saw a setting to let it run at the highest speed the DIMMs are capable of. I'm not sure if that will overclock something like a 1600 part to 1866, or if it's still going to top out at whatever your fastest stick is? It made it sound like the latter, but I only glanced at it.

[QUOTE=Madpoo;398545]I'd be curious to see if there was any significant difference in your timings if you followed some/all of the HP guidelines on optimizing for low latency. Things like setting the mem refresh to 1x instead of 2x, static high performance mode, disabling low power states, etc.

Here's the doc on it:
[URL="http://h10032.www1.hp.com/ctg/Manual/c01804533.pdf"]http://h10032.www1.hp.com/ctg/Manual/c01804533.pdf[/URL]

I've personally never felt the need to tweak a server to that extent, but I am curious. One in particular raised my eyebrows:
"Disabling processor power and utilization monitoring and memory pre-failure notification SMIs"

Having the system query processor state frequently (and the interrupts I guess it causes) might make a slight difference. It did say that disabling the mem pre-failure checks isn't as big a deal in Gen8/Gen9 since it's only done every few minutes (5 or something). And that's probably worth keeping enabled.[/QUOTE]

Aha! I actually stumbled across something earlier today... something I'd totally forgotten about since the days of Compaq servers.

In the BIOS settings (RBSU), if the first thing you do is hit Ctrl-A (for "Advanced" option) you will then see a new "Service Options" selection at the bottom of the main list.

I remember doing Ctrl-A all the stinkin' time on Compaq servers because that's where they hid a lot of the really useful stuff. When the merger with HP happened, the new BIOS's tended to float those useful things into the regular settings and I've totally erased Ctrl-A from memory... but it's still there.

So that lets you disable the processor and memory monitoring to get a little better latency out of the system (with the risks mentioned in the document, about not having certain memory pre-failure warnings). And it also lets you change the memory refresh from the default 2x to just 1x (or 3x if you want the extra reliability, not better latency).

I'm a little chicken to try this on any production systems I have... I'd better not in fact. I do have some servers in our dev environment I may try it out with...do some before/after timings. They're "lowly" DL3x0 G7 boxes but it would be interesting to see what effect it has, if any.

[QUOTE=Madpoo;399864]I'm a little chicken to try this on any production systems I have... I'd better not in fact. I do have some servers in our dev environment I may try it out with...do some before/after timings. They're "lowly" DL3x0 G7 boxes but it would be interesting to see what effect it has, if any.[/QUOTE]

Okay, reporting back.

With all 3 things (memory monitoring, cpu monitoring, and refresh rate set to 1x) it improved per iteration times by roughly 2.25%.

With just the cpu/memory monitoring disabled and refresh at 2x, it was about 1% improvement.

With just the memory refresh setting at 1x it helped to the tune of about 0.5%.

I know, the 2 things separately don't add up to the improvement of all of it together but this wasn't an exact and scientific measurement.

Basically I'd reboot and let the system stabilize, with absolutely nothing else running on the server. I'd let Prime95 chug along for 4-5 minutes and then see how the per iteration times were averaging.

Anyway, those settings did seem to help ever so slightly. Not as much as what I saw when I merely changed to high performance mode (along with increased cooling).

[QUOTE=Madpoo;399868]Anyway, those settings did seem to help ever so slightly. Not as much as what I saw when I merely changed to high performance mode (along with increased cooling).[/QUOTE]

And while I'm thinking of it, I just changed that last of my dev servers over to that high performance mode. Increased the performance of Prime95 by about 13%. :smile:

That could be applicable to any other system, not just these HP servers. Disable any default power saving crud in the BIOS and see if that doesn't help.

For which settings is the memory being refreshed more or less frequently? I am not sure what 1X and 2X actually mean. My non-server board has two settings; I think they are timings, but I don't remember exactly. Just curious. :-}

[QUOTE=kladner;399907]For which settings is the memory being refreshed more or less frequently? I am not sure what 1X and 2X actually mean. My non-server board has two settings; I think they are timings, but I don't remember exactly. Just curious. :-}[/QUOTE]

From what I understand from the HP document, those settings are intended to cut down on potential memory errors. Setting it to 2x refresh will double the refresh cycle which adds some latency.

It may be more reliable, but from everything I've experienced with official HP branded memory modules, they're usually "top bin" stuff. That's how HP can get away with only allowing some features with official HP sticks (like running 3-dimm per channel at full speed).

Setting to 1x refresh will help with the latency but if you happened to be running with some marginal modules you might start getting more ECC issues, I'd guess.

Heck, now that I know that's there, if I ever do run across a situation where some module is being difficult with me, I could try setting it to 3x and see if it helps stabilize. The systems I manage are in remote datacenters in other states, so if a problem comes up it could be a while before I'm onsite to handle it. We have multiple layers of redundancy all along the way to handle that but it still bugs me when a drive has failed and we're running in RAID 5 recovery mode or something. Right now I have a system with a bad array controller battery so the write-caching is disabled. I could enable it anyway but if the system lost power we'd lose data and that would be worse. :smile:

Last time I had a system with bad RAM, it only crashed when running SQL for over 24 hours and it triggered the uncorrectable ECC condition. It was a SQL cluster member. I had to keep that system as a passive SQL node for about 6 months, only running it for short periods of time during scheduled maintenance, until I could get onsite to swap out the bad module. Yuck.

Thanks for the explanation. I looked at my BIOS and determined that the settings are Auto, 7.8 ms, and 3.9 ms. Set to Auto, it runs at 7.8 ms. I am thinking that, as far as the analogy can be carried, that 7.8 ms corresponds to 1X, and 3.9 ms to 2X.

This could make sense on an "enthusiast" board, which would be more concerned with speed. I cannot say if any instabilities I have encountered are related to this setting. I have always run it on Auto. Truthfully, most problems I have had named the graphics driver. With two 580 cards cranking all the time, I am not surprised.

[CODE][Apr 13 17:11] Worker starting
[Apr 13 17:11] Using AVX FFT length 19200K, Pass1=768, Pass2=25K
[Apr 13 17:11] p: 332220523. Time: 210.282 ms.
[Apr 13 17:11] p: 332220523. Time: 225.131 ms.
[Apr 13 17:11] p: 332220523. Time: 232.764 ms.
[Apr 13 17:11] p: 332220523. Time: 211.308 ms.
[Apr 13 17:11] p: 332220523. Time: 212.410 ms.
[Apr 13 17:11] p: 332220523. Time: 211.125 ms.
[Apr 13 17:11] p: 332220523. Time: 237.940 ms.
[Apr 13 17:11] p: 332220523. Time: 209.893 ms.
[Apr 13 17:11] p: 332220523. Time: 211.920 ms.
[Apr 13 17:11] p: 332220523. Time: 212.411 ms.
[Apr 13 17:11] Iterations: 10. Total time: 2.175 sec.
[Apr 13 17:11] Estimated time to complete this exponent: 836 days, 9 hours, 21 minutes.
[Apr 13 17:11] Using AVX FFT length 19200K, Pass1=768, Pass2=25K, 2 threads
[Apr 13 17:11] p: 332220523. Time: 106.892 ms.
[Apr 13 17:11] p: 332220523. Time: 116.139 ms.
[Apr 13 17:11] p: 332220523. Time: 109.861 ms.
[Apr 13 17:11] p: 332220523. Time: 109.957 ms.
[Apr 13 17:11] p: 332220523. Time: 109.866 ms.
[Apr 13 17:11] p: 332220523. Time: 109.691 ms.
[Apr 13 17:11] p: 332220523. Time: 121.754 ms.
[Apr 13 17:11] p: 332220523. Time: 119.575 ms.
[Apr 13 17:11] p: 332220523. Time: 109.451 ms.
[Apr 13 17:11] p: 332220523. Time: 111.075 ms.
[Apr 13 17:11] Iterations: 10. Total time: 1.124 sec.
[Apr 13 17:11] Estimated time to complete this exponent: 432 days, 7 hours, 4 minutes.
[Apr 13 17:11] Using AVX FFT length 19200K, Pass1=768, Pass2=25K, 3 threads
[Apr 13 17:11] p: 332220523. Time: 115.798 ms.
[Apr 13 17:11] p: 332220523. Time: 110.985 ms.
[Apr 13 17:11] p: 332220523. Time: 121.072 ms.
[Apr 13 17:11] p: 332220523. Time: 120.144 ms.
[Apr 13 17:11] p: 332220523. Time: 120.129 ms.
[Apr 13 17:11] p: 332220523. Time: 120.921 ms.
[Apr 13 17:11] p: 332220523. Time: 131.042 ms.
[Apr 13 17:11] p: 332220523. Time: 120.566 ms.
[Apr 13 17:11] p: 332220523. Time: 120.063 ms.
[Apr 13 17:11] p: 332220523. Time: 119.674 ms.
[Apr 13 17:11] Iterations: 10. Total time: 1.200 sec.
[Apr 13 17:11] Estimated time to complete this exponent: 461 days, 13 hours, 40 minutes.
[Apr 13 17:11] Using AVX FFT length 19200K, Pass1=768, Pass2=25K, 4 threads
[Apr 13 17:11] p: 332220523. Time: 128.676 ms.
[Apr 13 17:12] p: 332220523. Time: 137.578 ms.
[Apr 13 17:12] p: 332220523. Time: 132.595 ms.
[Apr 13 17:12] p: 332220523. Time: 130.425 ms.
[Apr 13 17:12] p: 332220523. Time: 130.105 ms.
[Apr 13 17:12] p: 332220523. Time: 130.985 ms.
[Apr 13 17:12] p: 332220523. Time: 146.978 ms.
[Apr 13 17:12] p: 332220523. Time: 132.284 ms.
[Apr 13 17:12] p: 332220523. Time: 131.660 ms.
[Apr 13 17:12] p: 332220523. Time: 130.219 ms.
[Apr 13 17:12] Iterations: 10. Total time: 1.332 sec.
[Apr 13 17:12] Estimated time to complete this exponent: 511 days, 23 hours, 35 minutes.
(~1.4017627557946661008325518890418 years)
[Apr 13 17:12] Worker stopped.
[/CODE]

Stargate38, Hm, and what CPU is it?

Most probably is a 2 cores HT (which proves again that using HT is not beneficial) or the memory is very crappy that bottlenecks at two threads.