Haswell Preview Benchmark

[Madpoo]

Originally Posted by Prime95

Yes, results are disappointing. I don't know if it is the chip or Apple's OS hampering it in some way.

My Haswell at 4GHz, DDR3-2400 does 4 2880K FFTs at 14.55 ms/iter. A 2880K uses 2880K * 8 bytes of memory, which is read twice and written twice plus about 5MB of sin/cos/weights data. That is, 8 * 2880K * 4 + 5 = 95MB, bandwidth = 95MB/iter * 4 workers / (.01455 s/iter) = 26.12GB/sec.

The Macbook Pro 2.3GHz, DDR3-1600 does 4 2560K FFTs in 16.8 ms/iter. Bandwidth = (8*2560K*4+5) * 4 / .0168 = 20.23GB/s.

One would expect the DDR3-1600 to deliver 1600/2400 * 26.12GB/s = 17.4GB/s. So the L4 cache does help, but much less than I hoped for.

Oh, and yes at the slow 2.3GHz clock speed the chip is bandwidth limited:

Timings for 2560K FFT length (1 cpu, 1 worker): 12.62 ms.  Throughput: 79.24 iter/sec.
Timings for 2560K FFT length (2 cpus, 2 workers): 12.49, 12.41 ms.  Throughput: 160.66 iter/sec.
Timings for 2560K FFT length (3 cpus, 3 workers): 14.14, 14.10, 14.17 ms.  Throughput: 212.27 iter/sec.
Timings for 2560K FFT length (4 cpus, 4 workers): 16.46, 16.03, 17.33, 17.24 ms.  Throughput: 238.83 iter/sec.
Timings for 2560K FFT length (4 cpus hyperthreaded, 4 workers): 16.85, 16.78, 16.82, 16.76 ms.  Throughput: 238.05 iter/sec.

I'm visiting some old ground here... I was just reading about Crystalwell and the L4 cache, etc.

It *seems* to me that the L4 cache should help, but only if an LL test is memory limited and not CPU limited.

Has anyone run a benchmark and ramped up the # of threads in a single worker to see how it scales? The issue I always see on many-core systems is that the 2nd core might come close to doubling iteration times, but beyond that you start to see smaller and smaller gains. The working assumption I have is that memory starts to be the bottleneck.

With a 128MB L4 cache and the higher bandwidth it offers, it seems like you could have 4 threads in a single worker and get closer to 4 times the performance of a single core?

I was trying to lookup the specs on Crystalwell... 1600 MHz, 128-bit path, so it's not super fast or anything (not as fast as the L2/L3) but still decent bandwidth compared to main memory I gather?

The one thing that stuck out to me was that the eDRAM is still shared with the GPU, so if you really expect to use *ALL* of that for the CPU, you need to have a discrete GPU installed on the system so the L4 can be just for CPU.

I was mostly just curious about the technology... I don't have plans to buy a Haswell with that feature, but I ran across something about it and it caught my eye.

[Prime95]

I have a Crystalwell, bandwidth benchmarks attached (running in Windows using VMware). Highlights:

L3 (3rd Level) Data/Unified Cache : 91.27GB/s
L4 (4th Level) Data/Unified Cache : 33GB/s
256MB Data Set : 10.88GB/s (8.7GB/s - 10.88GB/s) (DDR3-1600 memory)


Alas, the L4 cache did not substantially change prime95's performance. I still get more throughput running one worker per core rather than 1 or 2 multithreaded workers. This may because it is a Mac laptop -- Apple has been known to do weird things with OS tuning and users are not allowed to tamper with any BIOS configurations.

Also, does anyone know how to measure or estimate delays introduced by maintaining cache coherence? I'm wondering if that could be one of the problems with prime95 multi-thread performance.

[Mark Rose]

Originally Posted by Prime95

Also, does anyone know how to measure or estimate delays introduced by maintaining cache coherence? I'm wondering if that could be one of the problems with prime95 multi-thread performance.

It depends on what exactly you're doing with the cache. You may find Ulrich Drepper's "What Every Programmer Should Know About Memory" (PDF) insightful, specifically sections 3, 6, and 7. I'm sure you already know most of what is discussed, but I got a lot out of it. The author used to be the chief developer of glibc.

[LaurV]

That is a brilliant document sir! Thanks for sharing it!

edit: To bring a bit of contribution, this paper (about floating numbers) is referred few times (and given in the citations), I googled it and it is very interesting to read too. It explains what the floats are, what they do, why they do it, and if their parents know... It is one of the first links google gives, I assume that a deeper google dig can find a better format (this format is a bit difficult to read due to the fact that almost all text is underlined, at least in my pdf viewer).

[Dubslow]

Originally Posted by Mark Rose

It depends on what exactly you're doing with the cache. You may find Ulrich Drepper's "What Every Programmer Should Know About Memory" (PDF) insightful, specifically sections 3, 6, and 7. I'm sure you already know most of what is discussed, but I got a lot out of it. The author used to be the chief developer of glibc.

Quite the link indeed. Thanks for sharing.