Variance of the Count of Primes in an Interval

[wblipp]

Are there any theoretical or heurisitic results on the variance of the number of primes in an interval? Empirically it's much smaller than a Poisson model would suggest.

Or equivalently, any theoretical or heuristic results on the probability that two randomly chosen numbers in an interval are both prime? The variance could be calculated from that.

Googling around, I have noted that the Twin Prime and related conjectures can be used for the probability that two numbers separated by "k" are both prime, but I haven't found anything more closely related to this question.

Recently I proposed a heuristic based on the Poisson model, and Mark Underwood pointed out that his calculations appeared to have smaller variance than this model. To test this, we created 100 intervals starting with one billion, each interval expected to have 500,000 primes based on the log integral. The average number of primes found was 499,999.4. The Poisson model predicts the variance should equal the mean, but the observed variance , 95661, was smaller by more than a factor 5.

[R.D. Silverman]

Originally Posted by wblipp

Are there any theoretical or heurisitic results on the variance of the number of primes in an interval? Empirically it's much smaller than a Poisson model would suggest.

Or equivalently, any theoretical or heuristic results on the probability that two randomly chosen numbers in an interval are both prime? The variance could be calculated from that.

Googling around, I have noted that the Twin Prime and related conjectures can be used for the probability that two numbers separated by "k" are both prime, but I haven't found anything more closely related to this question.

Recently I proposed a heuristic based on the Poisson model, and Mark Underwood pointed out that his calculations appeared to have smaller variance than this model. To test this, we created 100 intervals starting with one billion, each interval expected to have 500,000 primes based on the log integral. The average number of primes found was 499,999.4. The Poisson model predicts the variance should equal the mean, but the observed variance , 95661, was smaller by more than a factor 5.

There are some heuristical results, but the true answer isn't even known
or conjectured, let alone proven. Work by a variety of number theorists
have hinted at an answer (Goldston, Hildebrand, Maier, others) but we
still don't even know today even whether Cramer's conjecture is correct.

You also need to be precise about your value of 'k' relative to the height
of the primes. When you say "intervals", do you mean very short, short,
medium, long etc. intervals? e.g. [x, x + (log x)^2), (x, x + (log x)^k),
(x, x + x^epsilon), (x, x + O(sqrt(x)), (x, x + epsilon*x) etc.

[wblipp]

Do any of these works deal with the typical variation? Cramer's Conjecture, for example, deals with gaps - this is perhaps related to the extreme value statistics, but doesn't have an obvious connection to the typical variation.

As for size of intervals, I gave an example from the numerical investigation - we looked at 100 intervals sized to average 500,000 primes (based on the log integral) starting at 10⁹. I would classify those as medium to long intervals, but it depends on the context. I didn't specify the asymptotic growth of the interval because I was curious about whether anything is known for any method.

A heuristic is known, of course. for the probability that two numbers separated by "k" are both prime - the twin prime heuristic being a special case. I could sum this over all possible separations in the interval, weighted by the probability a random selection would have that separation. If nobody points out a better approach, I'll give that a try and see how well it matches the observations.