Cross-validation for groups and time: check what a split holds out

Choose a validation split by the kind of new case the model must handle. Random folds can suit exchangeable observations. Grouped folds keep related records together when the goal is to generalize to unseen groups. Chronological folds prevent training on later records when the intended use predicts later events.

A split is an experimental claim about what “unseen” means. Running more folds does not repair the wrong claim. We will inspect an invented table with repeated customer records in time order and check the boundaries each splitter actually enforces.

Define three different prediction questions

Imagine twelve ordered records, two each from customers 0 through 5. A random-row evaluation asks about other rows under its sampling assumptions. Holding customers out asks about customers absent from training. Holding later times out asks about later observations, which may include familiar customers.

Those are distinct tasks. The code below does not train a model or report accuracy; it audits the split before any model score can distract from the design.

import numpy as np
from sklearn.model_selection import GroupKFold, KFold, TimeSeriesSplit

X = np.arange(12).reshape(-1, 1)
groups = np.repeat(np.arange(6), 2)

def shared_groups(train, test):
    return set(groups[train]) & set(groups[test])

random_folds = list(KFold(n_splits=3, shuffle=True, random_state=42).split(X))
has_overlap = any(shared_groups(train, test) for train, test in random_folds)
print("random folds share customers:", has_overlap)

group_folds = list(GroupKFold(n_splits=3).split(X, groups=groups))
assert all(not shared_groups(train, test) for train, test in group_folds)
print("group folds isolate customers:", True)

time_folds = list(TimeSeriesSplit(n_splits=3).split(X))
assert all(train.max() < test.min() for train, test in time_folds)
print("time folds train before test:", True)
print("first time fold:", time_folds[0][0].tolist(), time_folds[0][1].tolist())

Expected output:

random folds share customers: True
group folds isolate customers: True
time folds train before test: True
first time fold: [0, 1, 2] [3, 4, 5]

The shared-customer finding is not proof that every repeated-customer task requires grouped evaluation. It shows that random rows do not test unseen-customer generalization. The intended deployment question decides whether that matters.

Read what each splitter guarantees

GroupKFold prevents a group's records from appearing on both sides of a fold. It does not impose chronological order. A grouped split can train on later events from other customers while testing earlier events from the held-out customers.

TimeSeriesSplit uses row order. Our rows are already ordered, and the assertion checks the index relationship. In a real table, sort by the correct timestamp and verify that it represents information availability. An ordered event timestamp alone does not prove the labels or features were known at that time.

The example even places part of customer 1 in the first chronological training set and another part in its test set. That is acceptable only for a task that allows familiar customers. If the task requires both unseen customers and future periods, design and audit a split that enforces both constraints; neither default splitter automatically solves the combined requirement.

Feature and label timing are separate boundaries

A record can have an early timestamp but a feature calculated from later events. Likewise, a target may become observable only after a delay. Specify the prediction moment, feature lookback window and label availability before selecting train/test dates.

Some temporal problems require a gap to avoid overlapping windows or delayed labels. Choosing that gap requires knowledge of the task, not a conventional number of rows. Irregular sampling also affects how to interpret equally sized index-based test blocks.

Fit preprocessing inside each fold

After the split is correct, fit imputers, scalers, encoders and feature selectors on the training portion of each fold. A scikit-learn pipeline can enforce that fitting boundary when used correctly. It cannot remove an inappropriate future-derived feature or repair a split that mixes groups you intended to hold out.

Use cross-validation for model choices, then reserve an appropriate final test for the intended use. Record fold definitions and inspect performance variation rather than reporting only a mean. The scikit-learn cross-validation guide documents the splitters and their assumptions.

Make the audit part of the experiment

Add a timestamp array and assert max(train_time) < min(test_time) rather than relying on row indices. Add a duplicate-record identifier and check whether the same underlying observation crosses the boundary. Finally, write down which groups are allowed to recur at deployment.

These checks turn a vague instruction to “avoid leakage” into properties you can inspect. Continue with data leakage examples and ROC versus PR evaluation to connect valid splits with meaningful metrics.