Statics Question, in the form of a storey

[markr]

Originally Posted by jinydu

But wasn't the Binomial Normal approximation only valid when p = 0.5? Because for any other value of p, the distribution is skewed, which doesn't happen in the normal distribution.

It's best at 0.5, but it's fine as long as p is not too close to 0 or 1 as long as n is large enough, and it actually doesn't have to be very large. That just screams for quantifying ... a common rule apparently is np > 5 & n(1-p) > 5, but I'd opt for a slightly larger bound. As well, for small n you should use a continuity correction. This web page has a nice, instructive interactive graphic.

[wblipp]

Originally Posted by jinydu

Just checking in to this thread again...

Perhaps I should have been even more explicit. I'm looking for the name of a specific test that allows one to accept or reject the null hypothesis, which is that Alice and Bob have the same probability of winning.

Google Statistical Comparison of Means. One of the first links explains this as a t-test for unpaired data.

In keeping with the OP approach, a more interesting story comes from the Bayesian approach. Here you start with a prior distribution, which reperesents your knowledge and uncertainty about the parameters in the model, then you update that with your observations. One of the pitfalls of this approach is that the results depend on the assumptions even more than in other statistical approaches.

Let us select the model is "Alice has constant independent probability p of winning each game, Bob has constant independent probabilty q of winning each game." The prior distribution represents our belief of knowledge about what these values might be. If Bob's Prior Distribtion is p=q, with zero probabilty for any unequal choices of p and q, then his insistence that Alice is merely unlucky is completely consistent with the Bayesian statistics. Of course that isn't really surprising - it says if you assume the probabilitiies must be equal, then they must be equal. But dressing this up in statistical language can hide the obviousness of that.

[davieddy]

Don't mess with Bill

[CRGreathouse]

Originally Posted by markr

It's best at 0.5, but it's fine as long as p is not too close to 0 or 1 as long as n is large enough, and it actually doesn't have to be very large. That just screams for quantifying ... a common rule apparently is np > 5 & n(1-p) > 5, but I'd opt for a slightly larger bound. As well, for small n you should use a continuity correction. This web page has a nice, instructive interactive graphic.

You have good intuition. That common rule of thumb is actually pretty bad.

Brown, Cai, & DasGupta have a good paper on the topic. They recommend using the Agresti-Coull interval instead of the normal approximation.