Only Best Times for Benchmarks?

jinydu · 2006-11-15, 05:12

I may have hinted at this question before, but I would like to ask it directly here.

Quote:

Originally Posted by 'www.mersenne.org/bench.htm'[/QUOTE

Where more than one user has reported timings for a given CPU, anomalous timings are discarded and the best times are reported.

This seems to be a contradiction. One part of the sentence says that anomalous times are discarded, whereas the other part says that the best times are reported. What if the best time is much faster than all of the other reported times; wouldn't it then count as a anomalous?

Also, why use the best time? Why not use the average time?

cheesehead · 2006-11-15, 22:54

Quote:

Originally Posted by jinydu

Quote:

Originally Posted by www.mersenne.org/bench.htm

Where more than one user has reported timings for a given CPU, anomalous timings are discarded and the best times are reported.

I think the last part should be read as "... are discarded, and then the best times are reported."

Quote:

Also, why use the best time? Why not use the average time?

Since the (elapsed) timings necessarily include the time needed to run any other software (OS or applications) as well as the Prime95 time, the best non-anomalous time probably includes the least non-Prime95 contribution and thus most accurately reflects Prime95 performance. (A best time that is significantly better than clustered next-best times would probably be interpreted as anomalous in the preceding step.)

Xyzzy · 2006-11-15, 22:59

I doubt the benchmark maintainer wants to spend a lot of time averaging stuff.

S485122 · 2006-11-16, 06:51

I did a lot of benchmarks in safe mode. No software running execpt Prime95. There is no significant difference between running one or two instances of Prime95. But the differences between "best times" from one run to another can go up to 2% on LL tests and up to 6% on factorisation tests.

This is why I think that an average of times would be more significant if enough iterations are used for each test.

It is possible that the policy of "best times" has to do with the development of the software see ftp://mersenne.org/gimps/p4notes.doc specially the part "Trace cache and branch prediction" (there are some parts of the algorithm that use a variable number of cycles...)

A possible compromise solution could be that the benchmarks outputs:
- best time.
- worst time.
- average time.
-standard deviation.

But this would give a lot of data to process.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
LLR times are not precise	carpetpool	Information & Answers	12	2018-03-24 08:48
Iteration times in i5 and i7	Jud McCranie	Information & Answers	53	2013-08-17 19:09
TF times	Unregistered	Information & Answers	7	2012-04-14 11:45
What are your per-iteration times?	LiquidNitrogen	Hardware	22	2011-07-12 23:15
How many times around the loop?	nibble4bits	Puzzles	2	2006-02-27 09:34

2006-11-15, 22:59	#3
Xyzzy "Mike" Aug 2002 2⁵·257 Posts	I doubt the benchmark maintainer wants to spend a lot of time averaging stuff.

2006-11-16, 06:51	#4
S485122 Sep 2006 Brussels, Belgium 2×3×281 Posts	I did a lot of benchmarks in safe mode. No software running execpt Prime95. There is no significant difference between running one or two instances of Prime95. But the differences between "best times" from one run to another can go up to 2% on LL tests and up to 6% on factorisation tests. This is why I think that an average of times would be more significant if enough iterations are used for each test. It is possible that the policy of "best times" has to do with the development of the software see ftp://mersenne.org/gimps/p4notes.doc specially the part "Trace cache and branch prediction" (there are some parts of the algorithm that use a variable number of cycles...) A possible compromise solution could be that the benchmarks outputs: - best time. - worst time. - average time. -standard deviation. But this would give a lot of data to process.