mersenneforum.org  

Go Back   mersenneforum.org > Fun Stuff > Lounge

Reply
 
Thread Tools
Old 2006-11-15, 05:12   #1
jinydu
 
jinydu's Avatar
 
Dec 2003
Hopefully Near M48

2×3×293 Posts
Default Only Best Times for Benchmarks?

I may have hinted at this question before, but I would like to ask it directly here.

Quote:
Originally Posted by 'www.mersenne.org/bench.htm'[/QUOTE
Where more than one user has reported timings for a given CPU, anomalous timings are discarded and the best times are reported.
This seems to be a contradiction. One part of the sentence says that anomalous times are discarded, whereas the other part says that the best times are reported. What if the best time is much faster than all of the other reported times; wouldn't it then count as a anomalous?

Also, why use the best time? Why not use the average time?
jinydu is offline   Reply With Quote
Old 2006-11-15, 22:54   #2
cheesehead
 
cheesehead's Avatar
 
"Richard B. Woods"
Aug 2002
Wisconsin USA

1E0C16 Posts
Default

Quote:
Originally Posted by jinydu View Post
Quote:
Originally Posted by www.mersenne.org/bench.htm
Where more than one user has reported timings for a given CPU, anomalous timings are discarded and the best times are reported.
I think the last part should be read as "... are discarded, and then the best times are reported."

Quote:
Also, why use the best time? Why not use the average time?
Since the (elapsed) timings necessarily include the time needed to run any other software (OS or applications) as well as the Prime95 time, the best non-anomalous time probably includes the least non-Prime95 contribution and thus most accurately reflects Prime95 performance. (A best time that is significantly better than clustered next-best times would probably be interpreted as anomalous in the preceding step.)

Last fiddled with by cheesehead on 2006-11-15 at 23:05
cheesehead is offline   Reply With Quote
Old 2006-11-15, 22:59   #3
Xyzzy
 
Xyzzy's Avatar
 
"Mike"
Aug 2002

11111111010012 Posts
Default

I doubt the benchmark maintainer wants to spend a lot of time averaging stuff.
Xyzzy is offline   Reply With Quote
Old 2006-11-16, 06:51   #4
S485122
 
S485122's Avatar
 
Sep 2006
Brussels, Belgium

68616 Posts
Default

I did a lot of benchmarks in safe mode. No software running execpt Prime95. There is no significant difference between running one or two instances of Prime95. But the differences between "best times" from one run to another can go up to 2% on LL tests and up to 6% on factorisation tests.

This is why I think that an average of times would be more significant if enough iterations are used for each test.

It is possible that the policy of "best times" has to do with the development of the software see ftp://mersenne.org/gimps/p4notes.doc specially the part "Trace cache and branch prediction" (there are some parts of the algorithm that use a variable number of cycles...)

A possible compromise solution could be that the benchmarks outputs:
- best time.
- worst time.
- average time.
-standard deviation.

But this would give a lot of data to process.
S485122 is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
LLR times are not precise carpetpool Information & Answers 12 2018-03-24 08:48
Iteration times in i5 and i7 Jud McCranie Information & Answers 53 2013-08-17 19:09
TF times Unregistered Information & Answers 7 2012-04-14 11:45
What are your per-iteration times? LiquidNitrogen Hardware 22 2011-07-12 23:15
How many times around the loop? nibble4bits Puzzles 2 2006-02-27 09:34

All times are UTC. The time now is 00:35.

Wed May 19 00:35:10 UTC 2021 up 40 days, 19:16, 0 users, load averages: 2.04, 2.13, 2.03

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.