mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Software (https://www.mersenneforum.org/forumdisplay.php?f=10)
-   -   Benchmark Variances (https://www.mersenneforum.org/showthread.php?t=21168)

Fred 2016-04-01 14:44

Benchmark Variances
 
I'm having trouble deciding between 1 worker and 4 workers in terms of overall efficiency on a particular machine. I ran 30 minute benchmarks (BenchTime=1800) twice, and get close (but different) results. Is this the equivalent to a "statistical tie", and I should just randomly pick a configuration (1, 2, or 4 workers) and just forget about it?

[CODE]Timings for 4096K FFT length (4 cpus, 1 worker): 5.85 ms. Throughput: 170.91 iter/sec.
Timings for 4096K FFT length (4 cpus, 2 workers): 11.81, 11.74 ms. Throughput: 169.85 iter/sec.
Timings for 4096K FFT length (4 cpus, 4 workers): 23.23, 23.36, 23.19, 23.23 ms. Throughput: 172.04 iter/sec.[/CODE]

[CODE]Timings for 4096K FFT length (4 cpus, 1 worker): 5.83 ms. Throughput: 171.43 iter/sec.
Timings for 4096K FFT length (4 cpus, 2 workers): 11.57, 11.57 ms. Throughput: 172.89 iter/sec.
Timings for 4096K FFT length (4 cpus, 4 workers): 23.56, 23.44, 23.46, 23.89 ms. Throughput: 169.60 iter/sec.[/CODE]

retina 2016-04-01 14:56

If there is no clear difference then it doesn't matter what you choose. Just choose any one. Or choose the one that is the most "pretty". Or go to random.org and let noise choose for you.

axn 2016-04-01 15:05

Pick the 1 worker setup. That way, if you want to take a break, you need to finish up just the 1 test (quickly) and you're done. Also, you'll become eligible for cat 0 (if it is ever created ;) ).

Or follow retina's advice. It's all good.

Xyzzy 2016-04-01 15:06

If you run enough benchmarks you will "waste" more time than any optimization will make up for.

Personally, we would run four cores on one job, so the work gets done faster. Our theory is the shorter the run is the less likely there will be a cosmic ray bit-flip.

:mike:

Prime95 2016-04-01 16:41

[QUOTE=Fred;430533]I'm having trouble deciding between 1 worker and 4 workers in terms of overall efficiency on a particular machine. I ran 30 minute benchmarks (BenchTime=1800) twice, and get close (but different) results. Is this the equivalent to a "statistical tie", and I should just randomly pick a configuration (1, 2, or 4 workers) and just forget about it?
[/QUOTE]

Your example is a toss-up. What OS did you use? I found that Windows 7 gave me wide-ranging results. Ubuntu server was very consistent. Ubuntu with GUI was also pretty steady. Personally, I use 20 or 30 second throughput benchmarks.

Fred 2016-04-01 18:15

[QUOTE=Prime95;430551]Your example is a toss-up. What OS did you use? I found that Windows 7 gave me wide-ranging results. Ubuntu server was very consistent. Ubuntu with GUI was also pretty steady. Personally, I use 20 or 30 second throughput benchmarks.[/QUOTE]

Windows 10. I did do shorter benchmarks, with just as much variance. I kicked off a couple 30 minute benches to see if hopefully that gave more consistent (averaged) results.

1 worker it is!


All times are UTC. The time now is 20:12.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.