Ordered E5-2683v3

[mackerel]

I was doing my infrequent survey on ebay in the hopes of finding a cheap LGA2011 mobo for the E5-2570 that everyone is practically giving away, but came across the E5-2683v3 instead. Now, it wasn't as cheap at about 3x the cost, but the spec got my attention. 14 cores at 2 GHz base, with AVX 2 (FMA). Yes please! So now I wait for Tuesday when hopefully it will arrive along with a new Asrock mobo. At least as a v3 CPU, I can use more easy to find new X99 boards. Other than another cooler also ordered, I think I have enough other spare parts to get a working system up without buying more right away.

Benchmarking it will be interesting. I've only really benched with 28.7 until this week, when I tried 28.9 for the first time. The bench output looks different but I've not dug into this yet, and may stick to 28.7 to keep things comparable with what I know. A particular question I have will be if there is some optimal number of effective workers less than the maximum (for similar work outside P95). If so, I might use this system to feed GPUs in the winter also.

I did debate if it was worth paying more for a dual board but thought this is enough power to play with for a while.

[VBCurtis]

Originally Posted by mackerel

A particular question I have will be if there is some optimal number of effective workers less than the maximum (for similar work outside P95). If so, I might use this system to feed GPUs in the winter also.

This depends entirely on how memory-bound the similar work is. I run LLR rather than P95 on a Haswell-E, and find that whether it's memory-bound or CPU-bound depends on how large the tests are (FFT size, just like with P95).

If I run ECM or srsieve or NFS sieving, using full hyperthreading is most efficient (12 threads on my 6-core). In fact, I generally run 4-6 threads of ECM or srsieve in addition to 6 threads of LLR; the combination helps to mask memory bandwidth saturation, as the LLR threads effectively have twice as long to wait for data from memory.

Nice purchase!

[mackerel]

I'm also mainly thinking of LLR. The ram bandwidth to CPU ratio is a bit worse than I like, lower even than my desktop (Skylake 4.2 GHz, dual channel, dual rank, 3200 ram).

Since I mostly run PrimeGrid, there is a first to return gets prime credit policy there, so I don't want units taking any longer than necessary. The low clock is already a concern there so I might have to stick to shorter units, but there will be quite some throughput!

[xtreme2k]

This is a slower brother to the E5-2697V3 14C/28T 35MB L3

P95 LL very well on this on a single worker with 12-14 threads.

[VBCurtis]

Originally Posted by xtreme2k

P95 LL very well on this on a single worker with 12-14 threads.

Since LLR does not have the feature to multi-thread a single test, this is not relevant to his question. Also, most folks using LLR are doing tests in the 2-6Mbit range, which is FFT too small to realize much gain from multithreading.

[Madpoo]

Originally Posted by mackerel

I was doing my infrequent survey on ebay in the hopes of finding a cheap LGA2011 mobo for the E5-2570 that everyone is practically giving away, but came across the E5-2683v3 instead. Now, it wasn't as cheap at about 3x the cost, but the spec got my attention. 14 cores at 2 GHz base, with AVX 2 (FMA). Yes please! So now I wait for Tuesday when hopefully it will arrive along with a new Asrock mobo. At least as a v3 CPU, I can use more easy to find new X99 boards. Other than another cooler also ordered, I think I have enough other spare parts to get a working system up without buying more right away.

Benchmarking it will be interesting. I've only really benched with 28.7 until this week, when I tried 28.9 for the first time. The bench output looks different but I've not dug into this yet, and may stick to 28.7 to keep things comparable with what I know. A particular question I have will be if there is some optimal number of effective workers less than the maximum (for similar work outside P95). If so, I might use this system to feed GPUs in the winter also.

I did debate if it was worth paying more for a dual board but thought this is enough power to play with for a while.

You're going to love it... my top of the line box is a dual Xeon E5-2697v3 which is similar... 14-cores but at 2.6 GHz instead of 2 GHz. As a bonus, I *think* you should be able to upgrade to v4 CPUs whenever those become more affordable. At least, with the Proliant Gen9 servers it lets you do that, so it may be BIOS dependent.

As far as # of workers and threads, I just run with all 14 cores of each CPU in one worker (so 2 workers total for the 2 CPU system).

It runs like a champ with that, and it actually continues to improve timings if I have just a single worker and start adding cores from the other chip... the faster QPI helps feed the crunching... But of course you do start to see diminishing returns with that approach. Still, I think when I verified M49 I was using 20-22 cores out of the total 28 in order to get it to finish as fast as possible... it was a difference of something like 42-45+ hours with only 14 cores or 34 hours with 20-22 cores (I forget the actual amount where I found the sweet spot).

And remember that the v3 is a quad mem channel CPU so be sure to have 4 DIMMs installed (and *only* 4 per channel...any more and it may force it to run at a slower clock). And the FASTEST memory your motherboard can handle. Sadly (and logically) HP servers don't have overclocking or anything, which makes total sense, so I'm locked in at DDR4-2133 (ECC). I can only imagine what something like that would do if allowed to run with 3200 modules at full tilt...

Even the v4 CPUs only officially support 2400 modules.

Oh, I was just looking at this... is this the motherboard you're getting?
http://www.newegg.com/Product/Produc...82E16813157543

If so (and if I can believe Newegg's specs), it does have overclocking options for the memory, up to 3000. DDR4 modules in ECC/Registered aren't exactly cheap (the 288-pin dimms) but you will get accurate results... the ECC will generally bug you if things are pushed too far.

[henryzz]

Does ECC allow for higher overclocking? Is it possible to get better performance out of ECC dimms? Is all memory ECC the same?

[mackerel]

Ok, one step at a time. I've had shipping notification on both mobo and CPU. The mobo is the Asrock X99 extreme 3, which is closely related to the one linked earlier but even cheaper. As an enthusiast board not server board, I'm aware it has OC abilities, but I don't know how locked down the CPU might be in preventing fun.

I already have a quad memory kit bought previously and misused in a dual channel system so will be moving that over. It is rated far above the 2133 officially supported by the CPU, so I hope there is room to move it up some more there. I don't have any plans to go ECC on this system but in theory it is supported.

v4 Xeons are supported with bios update, but I'm in no hurry. I wasn't after absolute performance, but the low cost of this Xeon did get my attention, and has now cured me of my E5-2670 desires.

[Madpoo]

Originally Posted by mackerel

Ok, one step at a time. I've had shipping notification on both mobo and CPU. The mobo is the Asrock X99 extreme 3, which is closely related to the one linked earlier but even cheaper. As an enthusiast board not server board, I'm aware it has OC abilities, but I don't know how locked down the CPU might be in preventing fun.

I already have a quad memory kit bought previously and misused in a dual channel system so will be moving that over. It is rated far above the 2133 officially supported by the CPU, so I hope there is room to move it up some more there. I don't have any plans to go ECC on this system but in theory it is supported.

v4 Xeons are supported with bios update, but I'm in no hurry. I wasn't after absolute performance, but the low cost of this Xeon did get my attention, and has now cured me of my E5-2670 desires.

All Xeon E5-26xx CPUs are locked... there's really no overclocking except maybe a MHz or two through pushing the FSB, but I doubt it'd be worth it.

I was trying to figure out how it can handle ECC and non-ECC memory... I thought the modules were keyed differently and had diff # of contacts, so I don't know how that works. There are ECC modules in registered and unbuffered flavors (RDIMMs and UDIMMs) but I think those are both 288-pin ECCs... at least, in the server world that's how it shakes out.

Your basic ECC is able to detect and correct single-bit errors and can detect-only a double-bit error. Advanced ECC as implemented by HP (and Dell also, I think) can detect and correct double-bit errors in many cases (when the failure is isolated to a single chip, which I think is the most common cause of errors like that).

HP servers have nifty features like online spare memory... let's say a module goes bad, you have a spare module installed that replaces the failed one. Or there's a mirrored memory option that does just what it says... you might have 2 dimms per channel and all writes go to the primary and secondary module per channel. In the event of failure you have another copy to work with.

Some of those increase latency a bit so I tend not to use them unless I know I have a flaky system... I might configure one of those modes remotely until I can get onsite and swap out the faulty DIMM, but it is nice to have the option. Especially if you're working in an industry where max uptime is the goal, those things let a server keep running despite failed modules. ECC/Advanced ECC are great for catching most problems but require downtime to swap out the failed device.

Anyway, since I only have experience in the server world when it comes to ECC, I've never thought about how it affects overclocking. Do mem issues from overclocking usually involve bits failing randomly, or is it more systemic than that (the controller itself puking, or getting massive multi-bit errors)? I don't know... it sure does help on a regular basis. I just had a server last week throw an error when an entire module pooped out. It caught it and wound up shutting down the box (no online spare/mirroring on there) but at least it caught the error and didn't just keep chugging along with whatever changed.

Half the time when that happens, the server is fine afterwards and I have no idea what caused it. I note the module that failed in case it happens again (which is great, knowing exactly which one was buggy). The other half of the time, that same module will throw another error in days, weeks, even months, and then it's time to dig into the spare parts.

[VBCurtis]

Originally Posted by Madpoo

I was trying to figure out how it can handle ECC and non-ECC memory... I thought the modules were keyed differently and had diff # of contacts, so I don't know how that works. There are ECC modules in registered and unbuffered flavors (RDIMMs and UDIMMs) but I think those are both 288-pin ECCs... at least, in the server world that's how it shakes out.

This is my understanding as well. I believe the CPU can handle ECC or non-ECC motherboards, but each board is one or the other. At some point in the past, they may have been interchangeable (DDR or SDRAM era) in boards compatible with ECC, but these days it's a board-by-board decision.

I haven't seen a server board (that is, an ECC-supporting board) that allows memory overclocking, but perhaps one step is a BIOS option (2133 to 2400, for example). ECC memory is usually not clocked as high as standard dimms of contemporary production, so one would definitely not choose an ECC setup for extra memory bandwidth!

[kladner]

I am puzzled on the ECC front. My board, Asus Crosshair V Formula Z (AMD 990FX chipset) says it supports:

4 x DIMM, Max. 32GB, DDR3 2400(O.C.)/2000(O.C.)/1600/1333/1066 MHz ECC, Non-ECC, Un-buffered Memory

There is a BIOS setting to select ECC or non-ECC. I have never looked into what sort of ECC RAM fits the slots on the board. I have only ever run it with DDR3-1600 non-ECC RAM.