Permutation feature importance: measure reliance and inspect correlated features

Permutation feature importance measures how a fixed model's evaluation score changes when one feature is shuffled. It describes reliance on that feature under a particular perturbation and evaluation dataset. It does not measure a feature's causal effect or an intrinsic importance independent of the model.

Our invented regression target depends on a signal feature. A second feature is a noisy copy of that signal, and a third is independent noise. This construction makes it possible to compare individual shuffles with a joint shuffle of the correlated pair.

Fit the model before interpreting it

We use separate training and held-out rows, a seeded random forest and R-squared as the evaluation score. Importance is the decrease in R-squared after shuffling; changing the scorer changes the meaning and units of the result.

A weak model may assign small importance to genuinely useful information simply because it failed to learn the relationship. Check the baseline score first. Repeated shuffles describe variability from the permutation procedure on this fixed test set; they are not a complete confidence interval for future populations.

Compare individual and joint perturbations

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance
from sklearn.metrics import r2_score
rng = np.random.default_rng(30)
signal = rng.normal(size=800)
X = np.column_stack([signal, signal + rng.normal(0, .05, 800), rng.normal(size=800)])
y = 3*signal + rng.normal(0, .2, 800)
model = RandomForestRegressor(n_estimators=100, min_samples_leaf=3,
                              random_state=30, n_jobs=1).fit(X[:600], y[:600])
held = X[600:]
base = r2_score(y[600:], model.predict(held))
importance = permutation_importance(model, held, y[600:], n_repeats=20,
                                    random_state=30, scoring='r2', n_jobs=1)
print(f'held-out R2={base:.3f}')
for name, mean, std in zip(['signal', 'copy', 'noise'],
                           importance.importances_mean, importance.importances_std):
    print(name, f'decrease={mean:.3f} shuffle_std={std:.3f}')
joint = []
for _ in range(20):
    changed = held.copy()
    order = rng.permutation(len(held))
    changed[:, :2] = held[order, :2]
    joint.append(base - r2_score(y[600:], model.predict(changed)))
print(f'joint signal+copy decrease={np.mean(joint):.3f}')
assert base > .9

Checked output:

held-out R2=0.993
signal decrease=1.592 shuffle_std=0.107
copy decrease=0.030 shuffle_std=0.003
noise decrease=0.000 shuffle_std=0.000
joint signal+copy decrease=1.937

Explain the correlated-feature problem

The model can distribute useful splits between the signal and its copy. Shuffling either one leaves another informative column available, so a single-column importance may understate the importance of the information shared by the pair.

The joint shuffle uses the same row permutation for both columns. It preserves their internal pairing while breaking their relationship to the target and other records. That is a different perturbation, so its score decrease need not equal the sum of the individual decreases.

Shuffling one correlated column also creates combinations that may be rare or impossible in the real data. A measured drop can partly reflect this distribution change. Conditional permutation, grouped perturbations and domain-specific interventions ask related but different questions; choose the one matching the interpretation you need.

Do not convert reliance into causality

A postcode, sensor proxy or downstream measurement may predict a target without causing it. A leaky feature may be extremely important to an invalid experiment. Investigate feature timing and availability before treating an importance plot as an explanation suitable for decisions.

A negative importance means the chosen shuffle improved the measured score in that run. It may reflect noise, overfitting or estimation variability. It does not establish that reversing or deleting the feature will improve a newly trained model. Test that separate intervention using development data.

Extend the experiment

Remove the correlated copy, retrain and compare the signal's importance. Then calculate training-set importance and contrast it with held-out importance. Finally, change the target so both features contribute distinct information and predict how the joint result should change.

The Machine Learning course connects model inspection with valid evaluation. Read tree split criteria for impurity-based behavior and cross-validation splits to ensure the held-out population matches the actual prediction task.

References

scikit-learn permutation importance — Score dependence, repeated permutations and correlated-feature interpretation.