P-values explained with an exact coin-test calculation

A p-value measures how incompatible the observed test statistic is with a specified null model, using the test's definition of outcomes at least as extreme. It is not the probability that the null hypothesis is true. To interpret one, you need the null, sampling assumptions, statistic and tail rule.

Consider an invented experiment: toss a coin 20 times and observe 16 heads. We will test the null model of independent tosses with heads probability 0.5. This is a mathematical teaching example, not data from a tested physical coin.

State the experiment before calculating

Under the null, the number of heads K follows a binomial distribution with n = 20 and p = 0.5. The probability of exactly k heads is comb(20, k) × 0.5²⁰.

For this symmetric null, define a two-sided test that treats deviations at least as far from 10 as the observed 16. That includes 16 through 20 heads and, on the opposite side, 0 through 4 heads. The p-value adds those probabilities. The probability of exactly 16 heads alone would omit other outcomes the test defines as at least as extreme.

from math import comb, isclose

n = 20
observed = 16
null_probability = 0.5

def probability(k):
    return comb(n, k) * null_probability**k * (1-null_probability)**(n-k)

distance = abs(observed - n * null_probability)
extreme = [k for k in range(n + 1)
           if abs(k - n * null_probability) >= distance]
p_value = sum(probability(k) for k in extreme)

print("extreme counts:", extreme)
print(f"observed proportion: {observed / n:.2f}")
print(f"two-sided p-value: {p_value:.6f}")
assert isclose(sum(probability(k) for k in range(n + 1)), 1.0)
assert isclose(p_value, 2 * sum(probability(k) for k in range(16, 21)))

Expected output:

extreme counts: [0, 1, 2, 3, 4, 16, 17, 18, 19, 20]
observed proportion: 0.80
two-sided p-value: 0.011818

The result says that the specified fair, independent-toss model assigns about 0.0118 probability to this test's set of extreme counts. It does not say there is a 1.18% probability that the coin is fair or a 98.82% probability that it is biased.

Keep the tail rule attached to the result

For this symmetric case, doubling the upper-tail probability matches our two-sided definition. That shortcut is not a universal recipe for every discrete or asymmetric null. Tests can define extremeness in different ways. A software output is interpretable only with the corresponding test specification.

A one-sided alternative asking specifically whether the probability of heads exceeds 0.5 uses the upper tail. Choose that question before seeing which direction the data happen to favor. Switching tail rules after inspecting results changes the procedure being evaluated.

Statistical evidence and effect size answer different questions

The observed proportion is 0.8, a difference of 0.3 from the null value. That describes this sample. An interval can communicate uncertainty about the underlying probability under a suitable estimation model. A p-value alone does not tell you how large, stable or practically relevant the effect is.

Increasing the sample size can make a small difference easier to detect. A large p-value can also reflect limited data rather than equivalence. To argue that a difference is small enough to ignore, define a meaningful margin and use an appropriate estimation or equivalence procedure.

The NIST binomial distribution reference gives the model used in our calculation. Real tests require checking the data-generating assumptions, not only substituting numbers into its formula.

Account for the way the data were collected

Independent identical tosses are a strong simplification. Dependence, changes in the process, selection of favorable observations or stopping only after a small p-value appears can invalidate the interpretation of a fixed-sample calculation.

Testing many different outcomes also creates more opportunities for apparently surprising results. Specify the analysis and report relevant tests, not only the favorable one. Appropriate multiple-testing or sequential procedures depend on the actual design; a single arbitrary threshold cannot repair an undisclosed search.

Try a nearby outcome

Change the observed number of heads to 15, recompute the extreme set and compare the p-value. Then change n and consider how the same observed proportion interacts with sample size. Keep the null and tail rule explicit in each case.

Continue with bootstrap confidence intervals to separate estimation uncertainty from testing a null model. The Statistics resources connect both to sampling, data interpretation and model evaluation.