Decision-tree split criteria: calculate weighted Gini impurity

A classification tree chooses a split by comparing how much it reduces impurity. For Gini impurity, a node with class proportions p_k has impurity 1 - sum(p_k²). A candidate split is scored using the child impurities weighted by their record counts.

We will enumerate every possible threshold for one invented feature, then check the best threshold against a fitted one-level scikit-learn tree. A separate four-record test fixture lets us inspect predictions without mistaking training purity for generalization.

Calculate a split by hand

The six training labels are [0, 0, 1, 0, 1, 1] at feature values 1 through 6. The parent contains three records of each class, so its Gini impurity is 0.5.

At threshold 2.5, the left child has two zeros and impurity 0. The right child has one zero and three ones, giving 1 - (1/4)² - (3/4)² = 0.375. The weighted impurity is (2/6)×0 + (4/6)×0.375 = 0.25, a reduction of 0.25.

Threshold 4.5 ties with that result. Tie handling can affect the selected structure. The code reports all threshold scores and checks that the library selects a best score, rather than pretending the optimum is unique.

Verify the search and the fitted stump

import numpy as np
from sklearn.tree import DecisionTreeClassifier
X = np.arange(1., 7.).reshape(-1, 1)
y = np.array([0, 0, 1, 0, 1, 1])
def gini(labels):
    proportions = np.bincount(labels, minlength=2) / len(labels)
    return 1 - np.sum(proportions**2)
scores = {}
for threshold in np.arange(1.5, 6, 1):
    mask = X[:, 0] <= threshold
    score = (mask.sum()*gini(y[mask]) + (~mask).sum()*gini(y[~mask])) / len(y)
    scores[float(threshold)] = float(score)
print('threshold weighted Gini:', {k: round(v, 4) for k, v in scores.items()})
model = DecisionTreeClassifier(max_depth=1, criterion='gini', random_state=29).fit(X, y)
chosen = float(model.tree_.threshold[0])
print('chosen threshold:', chosen, 'weighted impurity:', round(scores[chosen], 4))
X_test = np.array([[1.2], [2.2], [4.8], [5.8]])
y_test = np.array([0, 0, 1, 1])
majority = np.bincount(y).argmax()
print(f'test accuracy={model.score(X_test, y_test):.2f}',
      f'constant baseline={np.mean(y_test == majority):.2f}')
assert np.isclose(scores[chosen], min(scores.values()))

Checked output:

threshold weighted Gini: {1.5: 0.4, 2.5: 0.25, 3.5: 0.4444, 4.5: 0.25, 5.5: 0.4}
chosen threshold: 2.5 weighted impurity: 0.25
test accuracy=1.00 constant baseline=0.50

Separate a good local split from a good tree

The impurity reduction describes these training labels. It does not prove that the threshold will predict unseen observations well. The deliberately simple held-out fixture is useful for checking the code, but four points cannot establish reliable real-world performance.

A greedy tree chooses the best available split at each node; it does not search every possible complete tree. Increasing depth can fit training idiosyncrasies. Compare depth, minimum leaf size and pruning choices on development data, then evaluate the selected model on a reserved test set.

Class weights and sample weights change the effective proportions and child weights. Entropy and log-loss criteria use a different impurity expression. Before reproducing a split from a report, identify the criterion, weights, missing-value behavior and stopping constraints.

Understand why feature importance needs another check

A feature used near the root can receive large impurity-based importance. That importance reflects the fitted model's training reductions, and can favor features with many possible split points. It is not a causal effect or proof that the feature is necessary in another population.

Permutation importance measures a different question: how much a chosen evaluation score changes when a feature is shuffled for a fixed model. Correlated features can complicate that interpretation too.

Extend the tree

Change one training label and recalculate the threshold table. Then add a second feature with many unique random values and compare training fit with held-out results as depth grows. Keep the test set untouched while deciding the depth.

The Machine Learning course develops trees and evaluation. Review overfitting versus underfitting to interpret the depth experiment, and data leakage before accepting an unusually pure split.

References

scikit-learn decision trees — Classification criteria, weighted impurity reduction, pruning and implementation considerations.