Business and operations
Machine learning for variance reduction in online experiments
Variance reduction uses pre-experiment data, such as each user's past spend or activity, to explain away noise in an A/B test. Methods like CUPED, regression adjustment and machine-learning predicted covariates give tighter confidence intervals from the same randomised traffic.
How machine learning reduces variance in online experiments
Adjusting for pre-experiment behaviour (CUPED)
Each user's value of the metric before the test explains much of their value during it. Regressing on that pre-period value and subtracting the predicted part removes noise without biasing the treatment effect. This is a single-covariate linear regression.
Real exampleCUPED (Microsoft, WSDM 2013)Deng, Xu, Kohavi and Walker propose CUPED, which uses pre-experiment data to reduce metric variability, and report reducing variance by about 50% on Bing's experimentation system.
Using machine learning predictions as the covariate
A model trained on many pre-experiment features predicts each user's outcome, and that prediction is used as the regression covariate. Cross-fitting keeps the model from overfitting the same data used to estimate the effect.
Real exampleMLRATE (Meta)MLRATE uses machine learning predictors of the outcome with cross-fitting and, in A/A tests on 48 Facebook metrics, the authors report over 70% lower variance than the simple difference in means.
Measuring which metrics benefit
Variance reduction helps some metrics much more than others, depending on how well the pre-period predicts the outcome. Teams report the gain per metric as equivalent extra traffic to decide where it is worth applying.
Real exampleMicrosoft Experimentation Platform: deep dive into variance reductionMicrosoft's experimentation team reports an effective traffic multiplier for variance-reduced metrics and shows that the gain differs widely between product surfaces.
Built into experimentation platforms
Commercial and in-house experimentation platforms apply CUPED-style adjustment automatically to supported metrics. Analysts still need to check the pre-period window and that the covariate is unaffected by the treatment.
Real exampleStatsig CUPEDStatsig's documentation describes adjusting each user's metric with that user's own value over the 7 days before exposure.
Can you learn variance reduction for A/B testing?
Yes, if you are comfortable with basic statistics. Variance reduction is regression applied to randomised experiments, so the order is hypothesis testing and confidence intervals, then linear regression, then machine learning predictions with cross-fitting.
- Estimation and hypothesis testingLearn the difference-in-means estimator, its standard error and how sample size sets the smallest detectable effect.Learn it in Statistics
- Linear regressionRegression adjustment and CUPED are regressions of the outcome on a pre-period covariate, so learn least squares and its assumptions.Learn it in Machine Learning
- Python and pandas for experiment dataBuild one row per user with assignment, pre-period metrics and outcome, and simulate A/A tests to check interval coverage.Learn it in Data Tools
- Prediction models with cross-fittingTrain gradient-boosted trees to predict the outcome from pre-period features, using separate folds for fitting and adjustment.Learn it in Machine Learning
Moving into experimentation and variance reduction
If you already work in the field
If you already analyse A/B tests, you have the core: randomisation, standard errors and metric definitions. Add regression adjustment, then cross-fitted ML covariates, and check them with A/A simulations. A realistic first move is to apply CUPED to a past experiment and compare interval widths with your usual analysis.
If you come from data science or ML
If you come from ML, the new part is causal inference: why randomisation matters, why covariates must be measured before treatment and how to check coverage. Read the CUPED and MLRATE papers and work with your experimentation team on one metric.
Roles that do this workData scientist, experimentationProduct analystStatisticianEconomist
Tools you will use
| Tool | What you use it for |
|---|---|
| statsmodels | Fit regression-adjusted treatment effects with robust standard errors. |
| scikit-learn | Predict outcomes from pre-experiment features to use as a variance-reducing covariate. |
| DoubleML | Estimate treatment effects with cross-fitted machine-learning adjustments and valid intervals. |
| EconML | Estimate average and heterogeneous treatment effects with ML-based estimators. |
| SciPy | Run t-tests, bootstraps and simulations to check interval coverage. |
Public data to practise on
- Hillstrom MineThatData E-Mail Challenge: A randomised email test with prior-year spend and recency; estimate visit, conversion and spend effects.
- Criteo Uplift Prediction Dataset: Randomised advertising incrementality tests with anonymised features; study adjusted and heterogeneous effects. Non-commercial licence.
Your first project
Download the Hillstrom MineThatData email data, estimate the Womens e-mail effect on spend with CUPED using prior-year spend, and compare it with the unadjusted difference in means using confidence-interval width.
Common questions
What is CUPED in A/B testing?
CUPED stands for Controlled-experiment Using Pre-Experiment Data. It adjusts each user's metric using a regression on their pre-experiment value of a related metric, which narrows confidence intervals while keeping the effect estimate unbiased.
Does variance reduction bias the experiment result?
Not when the covariate is measured before treatment and the adjustment is done correctly, because randomisation makes the covariate independent of assignment. Covariates affected by the treatment, or models fitted on the same data without cross-fitting, can introduce bias.
What if users have no pre-experiment data?
New users have no history, so the adjustment cannot help them. A common approach is to fill missing pre-period values with a constant and include an indicator, or to use other features known at assignment time.
Is CUPED the same as regression adjustment?
They are closely related. CUPED with one covariate is equivalent to regression adjustment, also called ANCOVA, and machine learning methods extend it to many covariates through a predicted outcome.
Courses for these skills
- Statistics and Probability for Machine LearningDistinguish confidence, prediction uncertainty and calibration.41 published lessons · all planned lessons open
- Practical Machine Learning in PythonCompare losses and models for continuous outcomes.51 published lessons · more lessons in production
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.