Data leakage in machine learning: examples and prevention

Data leakage happens when training or model selection uses information that would not be available for a genuine new prediction. It can produce impressive evaluation results without a model that works in its intended setting. Common examples are future-derived features, preprocessing fitted before the split and related records appearing on both sides of an evaluation.

In this article, leakage means an invalid ML experiment. It is different from a security incident involving exposed personal data. scikit-learn's common-pitfalls guide explains why test information must stay out of fitting, including fitted preprocessing.

Three data leakage examples

These are illustrative dataset designs, not results from SchoolWhool experiments.

Before choosing an algorithm, write one sentence specifying when a prediction is made, what it predicts and what information exists at that moment. Every feature should pass that check.

For example: “At order placement, predict whether this order will be returned within the stated observation window, using information available at placement.” A field can be present in an exported database and still have been unavailable then.

Target leakage: an answer hidden in an input

A suspicious field does not need to contain the target literally. A status updated after an outcome, a later administrative action or an aggregate calculated using the target can reveal it indirectly.

Audit the table by adding three notes for each candidate feature: what it means, when it becomes available and whether the outcome influences its value. Drop fields that fail the availability check and investigate unclear ones with whoever owns the data.

Inspect your columns in Series and DataFrames. Then use correlation, confounding and causation to avoid treating a strong association as proof that the feature can support a valid prediction.

Preprocessing leakage: learning from the full dataset

Scaling, imputation and feature selection can learn properties from the data. If they are fitted on the full dataset before holding examples out, the fitted transformation has already used information from those examples.

Split first. Fit learned preprocessing on training data and use the fitted transformation on held-out data. During cross-validation, fitting must happen separately inside each training fold. scikit-learn's pipeline documentation explains how combining transformations with an estimator helps preserve this separation.

Our pipelines lesson builds on fit, predict and transform. Study those two operations before trying to repair leakage by moving lines around at random.

A Python example that keeps scaling inside fitting

This small example uses the built-in Iris dataset. Install scikit-learn in a project environment, then run the code as a Python script. It demonstrates separation of fitted preprocessing; it does not establish performance on a real application.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
validation_scores = cross_val_score(model, X_train, y_train, cv=folds)
print("Cross-validation scores:", validation_scores)

# For a real project, finalize model choices using training/validation only.
model.fit(X_train, y_train)
print("Final held-out accuracy:", model.score(X_test, y_test))

The outer split reserves a final test set. Cross-validation receives only the training portion and fits a fresh pipeline for each fold, including its scaler. The final fit learns the scaler and classifier from the full training portion before using the held-out test.

The random stratified split suits this introductory example. Do not copy that split blindly into time-dependent or grouped data. A pipeline cannot repair an inappropriate partition or remove a feature that reveals the answer.

If package setup is unfamiliar, start with uv, packages and environments. For a longer first-model walkthrough, read machine learning code in Python.

Train-test contamination from time and related records

The intended prediction determines what must be held out. If the goal is to predict later events, develop and validate on earlier periods before evaluating on later ones. scikit-learn's TimeSeriesSplit reference describes ordered folds; the timing of labels and features still needs a separate audit.

If the goal is to generalize to new customers or objects, keep each relevant group entirely on one side of a partition. GroupShuffleSplit is one tool for partitions based on groups.

For a photograph collection, write a manifest with the image identifier, object identifier and assigned split. Check that no object intended to be held out appears in training. For near-duplicates with different filenames, inspect the content rather than trusting filenames alone.

Data leakage vs overfitting

Overfitting describes a model learning training details that do not generalize. Leakage describes information crossing a boundary that your experiment requires. A leaking evaluation may conceal poor generalization by making the task artificially easy.

Fix leakage before comparing model settings. Making the model smaller cannot make a future-derived field legitimate. Once the split and features are sound, use overfitting vs underfitting to diagnose the remaining training behaviour.

A practical review before you publish a result

For the project described above, record the prediction moment, feature availability rules, partition method and fitted preprocessing. Include those in the README beside your training command.

Check that the final test did not guide model selection, that duplicates were investigated and that labels have had enough time to become observable. If you discover leakage, rerun the experiment and replace the invalid result; keep the correction clear in your notes.

Common questions

Can leakage happen without using the target column?

Yes. Related records, later events and learned transformations can expose information without including the target itself as an input.

Does a pipeline prevent all leakage?

No. It helps keep fitted transformations within training. You still need valid features, correctly timed labels, appropriate splits and a test set kept out of model selection.

Does unusually high accuracy prove leakage?

No. It is a reason to audit the experiment, not proof. Check the baseline, data provenance, duplicate records, feature availability and evaluation design.

Learn the data workflow behind the model

Follow the Data Tools course for arrays, tables and estimators, then the Statistics course for sampling and evidence. The Machine Learning course provides the wider model-development syllabus.

Primary references were checked on 12 October 2026.