Data and modeling
Machine learning for uncertainty quantification
Uncertainty quantification gives a machine learning prediction an honest range, not just a single number. Practitioners use conformal prediction for intervals with coverage guarantees, quantile regression for asymmetric ranges, and Bayesian models to separate noise from parameter uncertainty.
How uncertainty quantification is used in machine learning
Prediction intervals with coverage guarantees
Conformal prediction wraps any trained model and uses a calibration set to produce intervals or sets that contain the true value at a chosen rate, such as 90 percent. It needs only that new data are exchangeable with the calibration data.
Real exampleA Gentle Introduction to Conformal Prediction (Angelopoulos and Bates, 2021)The tutorial explains conformal prediction as a distribution-free way to create statistically rigorous uncertainty sets for the predictions of any model.
Ensemble weather forecasts
Forecasters run many forecasts with slightly different starting conditions to estimate the range of possible weather and the chance of extremes. Machine learning ensembles now produce these probabilistic forecasts much faster.
Real exampleGenCast (Google DeepMind, Nature 2024)DeepMind presents GenCast as a high-resolution AI ensemble model and reports better forecasts of day-to-day weather and extreme events than ECMWF's ENS, up to 15 days ahead.
Uncertainty from neural network ensembles
Training several networks with different random starts and averaging them gives both a better prediction and a spread that grows on unfamiliar inputs. It is a simple, strong baseline for deep learning uncertainty.
Real exampleDeep Ensembles (Lakshminarayanan, Pritzel and Blundell, 2017)The paper proposes ensembles of neural networks as a simple and scalable way to estimate predictive uncertainty.
Approximate Bayesian deep learning
Bayesian methods treat model weights as uncertain, which separates noise in the data from uncertainty about the model. Cheap approximations make this practical for large networks.
Real exampleDropout as a Bayesian Approximation (Gal and Ghahramani, 2016)The paper shows that a network trained with dropout can be interpreted as approximate Bayesian inference, giving a way to represent model uncertainty.
Can you learn uncertainty quantification for machine learning?
Yes, with a good grounding in probability. Learn distributions and estimation first, then build models and wrap them with quantile, conformal and Bayesian methods. Most practical tools are available in Python libraries such as MAPIE and PyMC.
- Probability and distributionsLearn distributions, quantiles, expectation and variance, the vocabulary every uncertainty method uses.Learn it in Statistics
- Estimation and intervalsUnderstand confidence intervals, the bootstrap and the difference between a confidence interval and a prediction interval.Learn it in Statistics
- Models to wrapTrain regression and classification models and check their calibration on held-out data.Learn it in Machine Learning
- Conformal, quantile and Bayesian methodsBuild split conformal intervals with MAPIE, quantile gradient boosting and a small Bayesian model in PyMC, and compare coverage and width.
Moving into uncertainty quantification work
If you already work in the field
Statisticians, engineers and scientists already reason about error bars and sources of uncertainty. Add modern ML models, then wrap one with conformal prediction and compare its coverage with the intervals you use now.
If you come from data science or ML
If you build ML models, add the statistics: calibration, coverage, proper scoring rules and the difference between noise and model uncertainty. Report an interval and its measured coverage alongside every point prediction you ship.
Roles that do this workMachine learning scientistStatisticianReliability engineerForecasting scientistResearch scientist
Tools you will use
| Tool | What you use it for |
|---|---|
| MAPIE | Wrap any scikit-learn model to produce conformal prediction intervals and sets. |
| scikit-learn | Fit quantile gradient boosting and check the calibration of predicted probabilities. |
| PyMC | Build Bayesian models that return full posterior distributions for parameters and predictions. |
| ArviZ | Diagnose Bayesian fits and summarise posterior uncertainty with plots and statistics. |
Public data to practise on
- Concrete Compressive Strength (UCI): Concrete mix ingredients and age with measured compressive strength, for regression with prediction intervals.
- Energy Efficiency (UCI): Simulated building shapes with heating and cooling loads, for interval estimates on two related targets.
Your first project
Download the Concrete Compressive Strength dataset, build 90% prediction intervals with split conformal prediction around a random forest, and compare them with quantile gradient boosting using held-out coverage and mean interval width.
Common questions
What is the difference between aleatoric and epistemic uncertainty?
Aleatoric uncertainty is noise in the data that more data cannot remove. Epistemic uncertainty comes from limited data or knowledge about the model, and shrinks as you gather more relevant data.
What is conformal prediction?
It is a method that turns any model's predictions into intervals or sets with a guaranteed coverage rate, using a held-out calibration set. It makes few assumptions about the model or the data distribution.
Are predicted probabilities from a classifier reliable?
Not always. Many models are over- or under-confident, so check calibration with a reliability plot and recalibrate with methods such as isotonic regression or temperature scaling.
Courses for these skills
- Statistics and Probability for Machine LearningAssess sampling variability and uncertainty.41 published lessons · all planned lessons open
- Practical Machine Learning in PythonDesign model comparisons that isolate the intended change.51 published lessons · more lessons in production
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.