Benchmark Variances

[Fred]

I'm having trouble deciding between 1 worker and 4 workers in terms of overall efficiency on a particular machine. I ran 30 minute benchmarks (BenchTime=1800) twice, and get close (but different) results. Is this the equivalent to a "statistical tie", and I should just randomly pick a configuration (1, 2, or 4 workers) and just forget about it?

Timings for 4096K FFT length (4 cpus, 1 worker):  5.85 ms.  Throughput: 170.91 iter/sec.
Timings for 4096K FFT length (4 cpus, 2 workers): 11.81, 11.74 ms.  Throughput: 169.85 iter/sec.
Timings for 4096K FFT length (4 cpus, 4 workers): 23.23, 23.36, 23.19, 23.23 ms.  Throughput: 172.04 iter/sec.

Timings for 4096K FFT length (4 cpus, 1 worker):  5.83 ms.  Throughput: 171.43 iter/sec.
Timings for 4096K FFT length (4 cpus, 2 workers): 11.57, 11.57 ms.  Throughput: 172.89 iter/sec.
Timings for 4096K FFT length (4 cpus, 4 workers): 23.56, 23.44, 23.46, 23.89 ms.  Throughput: 169.60 iter/sec.

[retina]

If there is no clear difference then it doesn't matter what you choose. Just choose any one. Or choose the one that is the most "pretty". Or go to random.org and let noise choose for you.

[axn]

Pick the 1 worker setup. That way, if you want to take a break, you need to finish up just the 1 test (quickly) and you're done. Also, you'll become eligible for cat 0 (if it is ever created ;) ).

Or follow retina's advice. It's all good.

[Xyzzy]

If you run enough benchmarks you will "waste" more time than any optimization will make up for.

Personally, we would run four cores on one job, so the work gets done faster. Our theory is the shorter the run is the less likely there will be a cosmic ray bit-flip.

[Prime95]

Originally Posted by Fred

I'm having trouble deciding between 1 worker and 4 workers in terms of overall efficiency on a particular machine. I ran 30 minute benchmarks (BenchTime=1800) twice, and get close (but different) results. Is this the equivalent to a "statistical tie", and I should just randomly pick a configuration (1, 2, or 4 workers) and just forget about it?

Your example is a toss-up. What OS did you use? I found that Windows 7 gave me wide-ranging results. Ubuntu server was very consistent. Ubuntu with GUI was also pretty steady. Personally, I use 20 or 30 second throughput benchmarks.

[Fred]

Originally Posted by Prime95

Your example is a toss-up. What OS did you use? I found that Windows 7 gave me wide-ranging results. Ubuntu server was very consistent. Ubuntu with GUI was also pretty steady. Personally, I use 20 or 30 second throughput benchmarks.

Windows 10. I did do shorter benchmarks, with just as much variance. I kicked off a couple 30 minute benches to see if hopefully that gave more consistent (averaged) results.

1 worker it is!