help designing prime95 28.10 feature

[CRGreathouse]

Originally Posted by TObject

Can figure out what makes certain algorithms run faster on certain architectures, so that no benchmarking is necessary?

No doubt, more information would help making better informed guesses. But to do that we'd want to write benchmarking software and have lots of people with different hardware setups run it, right? So we end up in the same place.

[ATH]

Maybe at the start of an LL test you can run 20 sec benchmark on 2 or 3 possible FFTs based on the exponent.

Even if the computer is in use the relative difference in speed between FFTs should be the same unless the user starts or stops anything during those 40-60 sec?

[0PolarBearsHere]

Originally Posted by Prime95

1) The benchmark really wants to run when nothing else is going on. Even so, the OS may fire up some process that skews a run. So, I need a good algorithm that tracks multiple runs and throws out outlier data.
2) My first thought was to have a menu choice to run the throughput benchmark. I suspect it will not get run on many machines. Alternatively, I could run the benchmark when prime95 is launched or at a late night hour or both until we have enough runs we are confident in our throughput data. Ideas? How many runs before we are confident in the results?
3) Any ideas on how best to detect a significant change (CPU, memory, whatever?) and discard the accumulated data?

1) and 3) could be partially worked around by having P95 generate cpu load itself, and then have the benchmark results track, size, cpu usage, and performance.

2) If you're only looking at 20 second benchmarks, then there should be no issue doing them not only when P95 starts, but also each time a worker process is unpaused/resumed. Ultimately, even pausing every hour for a 20second benchmark will add less than 5 hours to a one month run-time. And this would only be until enough has been run anyway, after which, if successful, the improved results could make up for a lot of that.

[Madpoo]

Originally Posted by Prime95

I was going to use 20 second benchmarks running on all cores to determine a machine's throughput for each possible FFT implementation.

I may be crazy for saying so (and feel free), but what about bumping the priority up from "idle" when doing that initial benchmark. Maybe not quite "normal", but something in the 5-6 range, just a notch or two down from normal priority? If it's only 20 seconds... I mean, I could live with that, but maybe not everyone would be on board. Prime95 running in idle-priority is kind of a big selling feature, so even a higher-than-idle for a short duration could be a bad move.

As for when to run the benchmark, my opinion is that running a benchmark at the beginning of each LL test to determine the beset FFT method/size. If things change on the system mid-run, well... it's not the end of the world, and it might make a better choice when it starts the next one.

If the benchmark runs at idle priority and happens to run when the system is busy, then it probably doesn't matter one way or the other if it picks a slightly less than efficient method/FFT size... the system is apparently used for other things and won't be running Prime95 at full speed anyway, so a percent decrease in throughput in a "pure" benchmark is probably far less of a loss in real world use.

[retina]

Originally Posted by Madpoo

I may be crazy for saying so (and feel free), but what about bumping the priority up from "idle" when doing that initial benchmark. Maybe not quite "normal", but something in the 5-6 range, just a notch or two down from normal priority? If it's only 20 seconds... I mean, I could live with that, but maybe not everyone would be on board.

You are crazy for the following reason. If during normal usage it runs at lowest priority, then it is pointless to get false readings by doing benchmarks at a different priority.

My car can do 300mph on a test track with a professional driver^[1], so that makes it the best car for me to drive around town in a traffic jam when I am sleepy.

^[1] Not really, of course. It can really only reach 299mph.

[chalsall]

Originally Posted by retina

You are crazy for the following reason. If during normal usage it runs at lowest priority, then it is pointless to get false readings by doing benchmarks at a different priority.

I disagree. (Not that Aaron isn't crazy; that's an orthogonal question...)

If the very short benchmarks are dependant on "wall time" then bumping up the priority might better help determine which FFT parameters are optimal. Although only a 20 second temporal sample seems too small to me.

This statement assumes that optimal performance of the FFT will be the same under load as not, and it is simply the benchmarking which might be impacted. This seems like a reasonable Ass-u-me'tion....

[lycorn]

Originally Posted by retina

If during normal usage it runs at lowest priority, then it is pointless to get false readings by doing benchmarks at a different priority.

This vaguely reminds me of the VW emissions scandal...

[Madpoo]

Originally Posted by chalsall

I disagree. (Not that Aaron isn't crazy; that's an orthogonal question...)

If the very short benchmarks are dependant on "wall time" then bumping up the priority might better help determine which FFT parameters are optimal. Although only a 20 second temporal sample seems too small to me.

This statement assumes that optimal performance of the FFT will be the same under load as not, and it is simply the benchmarking which might be impacted. This seems like a reasonable Ass-u-me'tion....

Yeah, you get it. The point being, if you're doing A/B benchmarking, you really want to ensure that everything else is essentially equal, and it's only your code branches that differ.

If you leave it at idle priority and you're spiking the CPU with some other app while benchmark A is going, then it'll give you false readings if you stop whatever it was while benchmark B is going.

You want to compare apples to apples and bumping the priority up from idle during the benchmark is a good way of doing that, I just don't know if that's the right/appropriate/best way to do it.

[ATH]

Originally Posted by Madpoo

If you leave it at idle priority and you're spiking the CPU with some other app while benchmark A is going, then it'll give you false readings if you stop whatever it was while benchmark B is going.

But even if there is other cpu usage the same FFT should still be fastest as in an idle benchmark. Except if the cpu usage varies a lot during those 20 sec?

[axn]

Originally Posted by ATH

Except if the cpu usage varies a lot during those 20 sec?

Exactly. Once you allow uncontrollable variables, everything's a crapshoot.

[Madpoo]

Originally Posted by ATH

But even if there is other cpu usage the same FFT should still be fastest as in an idle benchmark. Except if the cpu usage varies a lot during those 20 sec?

That's exactly the problem... the CPU usage can vary wildly.

Well, at least in my experience... that's the whole reason I decided to setup the affinity scramble stuff in the first place because the routine that tries to autodetect which CPU threads are pairs of the same core kept failing and doing weird things. And that was just from server activity within a few seconds time period. All it takes is for something like IIS to have a spike in CPU for a split second while compiling something, and things are thrown off.

If there was a way to guarantee a quiet-period when nothing else was running while these benchmarks happened, that'd be great, but you know that'll never happen. Even something like moving the mouse around will throw things off a bit (old interrupt driven hardware drivers sure, but still...)