False lead about fluctuations of iteration timings with Prime95

[S485122]

Prime95 v30.8b17
I run LL doublechecks on an I9-10920X which is AVX512 capable. My problem is that some runs show very variable timings (even with computer otherwise unoccupied.) ms per iteration will be stable and correct most the time. But some runs see the timings fluctuate wildly from the normal time per iteration to 150% more. See the copy of the worker window included as attachment as an example. The first run suffered from that slowness, the subsequent runs don't.

My impression is that this happens when the order in which threads 10 and 11 (running on cores 11 and 12) are stared puts them in both in the the first six or both in the second 6. I can, now that I have an idea of the circumstances, stop and restart until threads 10 and 11 are well apart. Setting affinity is not a solution : it doesn't determine the order in which the threads are started. (The CPU is limited to 2,2 GHz for power-saving reasons.)

[kriesel]

Originally Posted by S485122

some runs see the timings fluctuate wildly from the normal time per iteration to 150% more. See the copy of the worker window included as attachment

What I saw there is ms/iter max value 1.540 / min value 1.361 = 1.1315, so +/- ~6.4% of the midpoint.

However; same prime95 version, and 4 workers on a Xeon Phi 7210, worker 1 is drastically slower even though it's running the smallest exponent; see attachment.
Worker 1: AVX512 FFT 9M, Pass1 1K, Pass2 9K, clm 1, 16 threads, ~20.7 msec/iter!
Worker 2: FFT identical to worker 1, but 11.2 msec/iter
Worker 3: AVX512 FFT 9600K, Pass1 960, Pass2 10K, clm 1, 16 threads, ~11.9 msec/iter;
Worker 4: AVX512 FFT 10M, Pass1 1k, Pass2 10K, clm 1, 16 threads, ~12.7 msec/iter

[S485122]

Originally Posted by kriesel

What I saw there is ms/iter max value 1.540 / min value 1.361 = 1.1315, so +/- ~6.4% of the midpoint.

Since I run Prime95 and not mprime, I can't log those stats. I copied the window just then and used that data, it doesn't show an extreme situation. I should have collected examples since the moment I became aware of the problem.
I remember for another FFT size the average iteration fluctuated between 0,92 ms and 0,96 ms, when things went "wrong" I they could go up to 1,70 ms...

Off topic since it does not concern fluctuations over time but differences between cores (and thus not fluctuations but differences ;-)

Originally Posted by kriesel

However; same prime95 version, and 4 workers on a Xeon Phi 7210, worker 1 is drastically slower even though it's running the smallest exponent; see attachment.
Worker 1: AVX512 FFT 9M, Pass1 1K, Pass2 9K, clm 1, 16 threads, ~20.7 msec/iter!
Worker 2: FFT identical to worker 1, but 11.2 msec/iter
Worker 3: AVX512 FFT 9600K, Pass1 960, Pass2 10K, clm 1, 16 threads, ~11.9 msec/iter;
Worker 4: AVX512 FFT 10M, Pass1 1k, Pass2 10K, clm 1, 16 threads, ~12.7 msec/iter

I run the processor at 2,2 GHz instead of the nominal 3,5 GHz. I first tried limiting the processor by capping the power to 100 W maximum, but the unstable timings and, for instance, the fact that Prime95 providing a constant load makes adjustments unnecessary, made me choose to limit the processor by clock and not power.

On topic again :
And, I repeat, my problem is that on some runs the timings fluctuate so much (even if we only consider a 6% variation it is not normal on a idle computer : other runs don't show that behaviour. I also checked my primary suspect : Windows with all its background processes (after a fresh install I disable syncing with devices, cloud accounts or storage and other crap...) (I regret my tuned Windows XP with about 12 processes running including Prime95, I had to abandon it : it wouldn't support modern hardware)

[kriesel]

Originally Posted by kriesel

same prime95 version, and 4 workers on a Xeon Phi 7210, worker 1 is drastically slower even though it's running the smallest exponent; see attachment.
Worker 1: AVX512 FFT 9M, Pass1 1K, Pass2 9K, clm 1, 16 threads, ~20.7 msec/iter! 48.3 iters/sec
Worker 2: FFT identical to worker 1, but 11.2 msec/iter = 89.3 iters/sec
Worker 3: AVX512 FFT 9600K, Pass1 960, Pass2 10K, clm 1, 16 threads, ~11.9 msec/iter; 84.03 iters/sec
Worker 4: AVX512 FFT 10M, Pass1 1k, Pass2 10K, clm 1, 16 threads, ~12.7 msec/iter; 78.7 iters/sec; total 300.4 iters/sec

(underlined added) To add to the puzzle, yesterday the timings in the same uninterrupted run were much more nearly equal:
W1: 16.1 msec/iter; 62.1 iters/sec
W2: 15.6 msec/iter; 64.1 iters/sec
W3: 15.8 msec/iter; 63.3 iters/sec
W4: 16.7 msec/iter; 59.9 iters/sec
total 249.4 iters/sec

It's hard to know, in a typical Windows OS, what else may be loading cores; AV scans or update downloads, OS update checks, browser update checks & downloads, file indexing, servicing interrupts, scheduled backups, etc.

[S485122]

Originally Posted by kriesel

(underlined added) To add to the puzzle, yesterday the timings in the same uninterrupted run were much more nearly equal:
W1: 16.1 msec/iter; 62.1 iters/sec
W2: 15.6 msec/iter; 64.1 iters/sec
W3: 15.8 msec/iter; 63.3 iters/sec
W4: 16.7 msec/iter; 59.9 iters/sec
total 249.4 iters/sec

It's hard to know, in a typical Windows OS, what else may be loading cores; AV scans or update downloads, OS update checks, browser update checks & downloads, file indexing, servicing interrupts, scheduled backups, etc.

On your system the timings differ from run to run not during a run, but the problem might be related.

I know Windows runs a lot of parasitic processes (your "etc"), but having worked for more than 30 years as software support in an IT company that used and sold Microsoft, I can analyse what is running and when, like for instance when the timings are high. I know when my files are indexed, when the antivirus runs, when my scheduled backups are running, and I know their impact on Prime95. My browsers and other applications don't run unless I use them. As I said I checked about Windows processes, I know what's happening on my computer and the impact of certain operations. Anyway on a system with 12 hyper-threaded cores with enough RAM those processes don't have much impact on Prime95 as I can verify daily. Knowing what's running on a typical Windows installation is not so hard, there are a lot of tools, anyway my installation certainly isn't typical.

I forgot : your 6,4 % from the midpoint calculation is not relevant, if you look at the data the median is 1,375 m.

[kriesel]

Originally Posted by S485122

On your system the timings differ from run to run not during a run

The timings for the four workers on a xeon phi 7210 I gave are earlier and later IN THE SAME EXPONENTS RUN. I did not interrupt it, prime95 automatic benchmarking did while I was asleep. Worker 1 and 2 console contents attached. Timings differ versus time for the same worker & same exponent & computation type, and between workers running the same FFT specification.

An interesting feature though is the aggregate iteration rate increased considerably on my 7210 Xeon Phi, while worker 1 slowed considerably. Worker 1 gets first pick of cores and memory footprint.

Stopping worker 1 manually later made only a few percent difference in the other workers' speed. I'm guessing cache efficiency went up. Clock rate did not change.

The phi referenced above runs Windows, AV, prime95, & occasional Taskmgr or notepad, and remote desktop accessed by a laptop. Not much going on there. I've turned off as many Win10 extras as I easily could. Sometimes Windows seems confused by the wealth of 64 cores, 256 hyperthreads and spending much effort switching from logical processor to logical processor. It's also subject to ambient temp fluctuations.

I know Windows runs a lot of parasitic processes (your "etc"), but having worked for more than 30 years as software support in an IT company that used and sold Microsoft, I can analyse what is running and when... I know when my files are indexed, when the antivirus runs, when my scheduled backups are running, and I know their impact on Prime95. My browsers and other applications don't run unless I use them. As I said I checked about Windows processes, I know what's happening on my computer and the impact of certain operations. Anyway on a system with 12 hyper-threaded cores with enough RAM those processes don't have much impact on Prime95 as I can verify daily. Knowing what's running on a typical Windows installation is not so hard, there are a lot of tools, anyway my installation certainly isn't typical.

Ok, sounds like you're very aware. Other readers may not be.

your 6,4 % from the midpoint calculation is not relevant, if you look at the data the median is 1,375 m.

I wrote midpoint, meaning middle of the span of samples (max+min)/2, what's usually called mid-range, for a reason. (Mainly, to minimize calculations. & my bad, not looking up the proper statistics terminology for it initially.) That's not median, mode, or mean, in general. https://en.wikipedia.org/wiki/Mid-range
But the main point was, you referred to increases of over 150%, and that was not present in the data you shared. It does happen sometimes that someone slips a decimal place in their posts.

None of what I wrote before or here is intended to provoke, belittle, or anything else negative. Just trying to promote understanding including my own.

When you figure out what's causing your prime95 timings to fluctuate, please update us.

[S485122]

Originally Posted by kriesel

...
When you figure out what's causing your prime95 timings to fluctuate, please update us.

I posted in this thread because I don't know what causes the fluctuations !
I waited a long time before doing so.

Anyway the only constant I found (the order of starting threads) is a false lead : today I have a stable run where the threads 10 and 11 are started amongst the last. I am in a mood to remove my part of the discussion. I am still under the impression that the phenomenon has something to do with AVX512.

I moved the discussion about this to a new thread.