Scaling vs normalization: feature statistics or row lengths

Feature scaling changes columns using feature-level statistics or ranges. Row normalization changes each observation to a chosen vector norm. These operations answer different questions, even though “normalization” is sometimes used informally for all of them.

StandardScaler centers each feature and scales by its training standard deviation. MinMaxScaler maps training feature extrema to a selected range. Normalizer rescales each row to a unit norm and does not estimate a training mean or range. Name the actual transformation rather than relying on the ambiguous word.

Inspect an explicit change in geometry

Our invented records have two measurements on different scales. A new record extends beyond the training range. We will fit both feature scalers on training rows only, inspect the transformed record and compare simple nearest-neighbor pipelines on a separate deterministic classification task.

Removing magnitude can discard useful information. If total quantity matters, making every row length one changes the problem rather than simply improving its units.

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler, Normalizer
from sklearn.pipeline import make_pipeline
from sklearn.neighbors import KNeighborsClassifier
X = np.array([[1., 100.], [2., 200.], [3., 300.]])
new = np.array([[4., 400.]])
for scaler in [StandardScaler(), MinMaxScaler(), Normalizer()]:
    scaler.fit(X)
    print(type(scaler).__name__, np.round(scaler.transform(new), 4).tolist())
rng = np.random.default_rng(28)
raw = rng.normal(size=(400, 2))
y = (raw[:, 0] > 0).astype(int)
raw[:, 1] *= 100
for name, steps in [('raw', []), ('standardized', [StandardScaler()]),
                    ('unit_rows', [Normalizer()])]:
    model = make_pipeline(*steps, KNeighborsClassifier(n_neighbors=7))
    model.fit(raw[:300], y[:300])
    print(name, f'held-out accuracy={model.score(raw[300:], y[300:]):.3f}')
assert MinMaxScaler().fit(X).transform(new)[0, 0] > 1

Checked output:

StandardScaler [[2.4495, 2.4495]]
MinMaxScaler [[1.5, 1.5]]
Normalizer [[0.01, 1.0]]
raw held-out accuracy=0.510
standardized held-out accuracy=0.960
unit_rows held-out accuracy=0.980

Explain the held-out comparison

The synthetic target depends only on the first unscaled feature, while the second is enlarged by a factor of 100. Raw Euclidean distance can therefore be dominated by an irrelevant coordinate. Standardizing columns restores comparable numerical scales for this construction.

The measured accuracies describe this seed, task and split. They are not a universal ranking of preprocessing methods. Unit-row normalization changes both coordinates jointly and can remove or distort magnitude information; whether that helps depends on the representation and target.

Trees based on single-feature thresholds are often much less sensitive to positive linear rescaling than distance-based models. That does not mean every preprocessing operation is irrelevant to trees. Nonlinear transformations, missing-value policies and feature construction can still change the model and its inputs.

Keep fitted statistics inside evaluation

The pipeline ensures the scaler learns from the training partition. In cross-validation, pass the entire pipeline to the splitter and model-selection procedure so each fold learns its own statistics. Fitting a scaler on all rows before cross-validation leaks evaluation information.

Outliers can strongly influence standard deviations and extrema. A robust scaler may be useful, but choose it using development data and the actual measurement process. Scaling an impossible sensor value does not make the value valid.

A constant feature also needs care: its standard deviation is zero. Library scalers have documented handling for that case; a handwritten division by the raw standard deviation can instead create nonfinite values. Add that boundary case when implementing preprocessing yourself.

Extend the comparison

Multiply only the informative feature by 100 and rerun the experiment. Then create a target that depends on vector magnitude and check whether row normalization removes the signal. Write down the geometry your model should respect before selecting a transform.

The Machine Learning course connects preprocessing with model behavior. Follow fit, transform and predict for state inspection and cosine versus Euclidean similarity for the relationship between direction, length and distance.

References

scikit-learn preprocessing guide — Feature scaling, normalization, robust alternatives and fitted transformation behavior.