Misleading statistics: 7 examples and how to spot each one
Seven common ways true numbers mislead: truncated axes, missing denominators, base rates, Simpson's paradox, survivorship bias, cherry-picked windows and correlation sold as cause. With a checklist for reading any claim.
Most misleading statistics are not false. The numbers are real, but the way they are chosen, compared or drawn leads you to a conclusion the data does not support. The same handful of patterns turns up again and again in news stories, business slides and model reports, and once you can name them you can check any claim in a minute or two.
Below are seven of the most common, each with how to spot it and the question that exposes it. Every one is taught in full, with data and code, in the free Statistics track.
1. A truncated axis that turns a small change into a cliff
A bar chart whose y axis starts at 95 instead of 0 makes a move from 96 to 98 look like a doubling. The numbers on the axis are honest; the picture is not, because a bar's length is read as its size.
How to spot it: look at where the value axis starts. Bars must start at zero, because their length is the message. A line chart may start elsewhere, since it shows change rather than size, but the scale should be labeled and the change put in proportion.
Ask: "How big is this change compared with the total?"
The lesson on how charts can mislead rebuilds misleading charts step by step: truncated axes, cherry-picked date ranges, smoothing that hides a change, and units nobody can check.
2. A count where a rate was needed
"Most accidents happen within five miles of home" sounds like a warning about local roads. But most driving happens within five miles of home too. A count of events means little until you divide it by how many chances there were for the event to happen.
How to spot it: a claim compares raw totals between groups of different sizes: cities, product lines, hospitals, user segments.
Ask: "Out of how many?" Then compare the rates, not the counts.
See categorical counts, rates and bar plots and denominators, base rates and missing groups.
3. Ignoring the base rate
Suppose a screening test catches 99% of people who have a rare condition and wrongly flags 5% of people who do not. If 1 person in 1,000 has the condition, then in a group of 100,000 people:
| Group | People | Flagged by the test |
|---|---|---|
| Has the condition | 100 | 99 |
| Does not have the condition | 99,900 | 4,995 |
| Total flagged | 5,094 |
These are illustrative numbers, not a real test. Of the 5,094 people flagged, only 99 have the condition: under 2%. A "99% accurate" test is mostly wrong when it says yes, because the condition is rare. The same arithmetic applies to fraud alerts, spam filters and any classifier that hunts for rare cases.
How to spot it: a claim quotes a test's accuracy without saying how common the thing being tested for is.
Ask: "How common is this in the first place?" Then count every way a positive result can happen.
Bayes' theorem works this calculation from counts, and conditional probability and independence explains why "the chance of a positive given the condition" is not "the chance of the condition given a positive".
4. Simpson's paradox: the trend that reverses inside every group
A treatment can look better overall and worse in every subgroup. A frequently cited example comes from a 1986 study of kidney stone treatments (Charig and colleagues, in the BMJ):
| Stones | Treatment A | Treatment B |
|---|---|---|
| Small | 93% (81/87) | 87% (234/270) |
| Large | 73% (192/263) | 69% (55/80) |
| All | 78% (273/350) | 83% (289/350) |
Treatment A succeeds more often for small stones and for large stones, yet B looks better overall. The reason is the mix: A was given mostly to large stones, which are harder to treat, and B mostly to small ones. Pooling the groups compares treatments on different cases.
How to spot it: an overall comparison between groups that differ in a hidden variable (severity, age, region, device).
Ask: "Does the pattern hold within each subgroup, and are the groups made up of the same mix?"
The lesson on Simpson's paradox and subgroup comparisons rebuilds a reversal and shows how to compare on one standard mix.
5. Survivorship bias: only the winners are left to count
"Successful founders dropped out of college" is drawn from founders who succeeded. The dropouts whose companies failed are not interviewed, so the data cannot say whether dropping out helped. The classic illustration is from the Second World War, when the statistician Abraham Wald pointed out that the bullet holes on returning aircraft showed where a plane could be hit and still fly home. The planes hit elsewhere did not come back to be counted.
How to spot it: the data comes only from cases that passed some filter: surviving companies, funds still open, users who are still active, patients who completed a study.
Ask: "Who dropped out before this data was collected, and why?"
See selective reporting and survivorship and who is in the dataset?
6. A cherry-picked window or a selectively reported result
Start a chart at a low point and any series looks like growth. Test twenty metrics and report the one that moved, and you will usually find something "significant" by chance alone: with a 5% threshold, about one in twenty tests of a true null comes out significant.
How to spot it: an unusual start date, a single metric with no mention of others, or a result reported without how many comparisons were made.
Ask: "What does the full period look like, and what else was measured?"
Using tests well covers multiple comparisons and p-hacking, and hypothesis testing and the p-value explains what a p-value can and cannot tell you.
7. Correlation presented as cause
Ice cream sales and drownings rise together, because both rise in hot weather. A correlation between two measures can come from a third factor driving both (a confounder), from the effect running the other way, or from chance.
How to spot it: observational data plus a causal verb: "causes", "drives", "boosts", "leads to".
Ask: "What else changed at the same time, and was anything randomly assigned?" Only an experiment, such as a well-run A/B test, or careful causal analysis can support the causal claim.
See correlation, confounding and causation and what the evidence can claim.
A checklist for reading any statistic
Before you believe or repeat a number, run through these:
- Axis: where does the chart's scale start, and are the units stated?
- Denominator: is this a count or a rate, and out of how many?
- Base rate: how common is the thing in the first place?
- Subgroups: does the pattern hold within each group, with the same mix?
- Survivors: who is missing from the data, and why?
- Window and choice: why this time range, and what else was measured?
- Cause: was anything randomly assigned, or could a third factor explain it?
Why this matters for machine learning
Every one of these traps appears in model work. A model's accuracy means little without the base rate of the class it predicts. A dataset built from active users carries survivorship bias into every prediction. A metric that improves overall can get worse for every segment. Reading data critically is the first skill in the Statistics track for that reason: the module on EDA, evidence and misleading statistics ends with a project where you audit a business claim and repair the chart behind it.
Your next step
Read the lessons above in order, starting with how charts can mislead. Lesson parts and key terms are free to read. A free account adds the quizzes and saves your progress, with no payment. If you are learning machine learning seriously, the learner community below is where course access for selected learners is shared.