Derivatives for machine learning: check a squared-error gradient

A derivative describes a local rate of change. For a model loss with several parameters, the gradient collects those rates into a vector. Training uses that vector to choose a parameter update, so a sign, scaling or shape error can send every subsequent step in the wrong direction.

We will differentiate a two-feature linear model with three invented observations. Then we will check the result using central finite differences. The numerical check tests an implementation at one parameter setting; it is useful evidence, not a substitute for deriving the expression.

Derive the gradient from the residual

Write predictions as p = Xw + b, where X is (n, d), w is (d,) and b is scalar. Let r = p - y. Choose the loss L = sum(r²) / (2n). The factor of two in the denominator cancels the derivative of the square.

For weight w_j, the derivative is sum_i(r_i X_ij) / n. In vector form, grad_w = X.T @ r / n. The bias gradient is mean(r), because each prediction changes by one unit for a one-unit change in b.

With three rows and two features, X.T has shape (2, 3), the residual has shape (3,) and the weight gradient has shape (2,). A gradient for two weights must have two components. Averaging over features instead of observations would produce a different scale and potentially a different shape.

Check against central differences

For parameter theta_j, the central difference is (L(theta + h e_j) - L(theta - h e_j)) / (2h). We perturb one coordinate at a time, leaving the others fixed. This costs two loss evaluations per parameter, which is manageable here but expensive for a large network.

import numpy as np
X = np.array([[1., 0.], [0., 2.], [1., 1.]])
y = np.array([1., 2., 3.])
theta = np.array([0.4, -0.2, 0.1])  # two weights, then bias

def loss(t):
    residual = X @ t[:2] + t[2] - y
    return np.mean(residual ** 2) / 2

r = X @ theta[:2] + theta[2] - y
gradient = np.r_[X.T @ r / len(y), r.mean()]
h = 1e-5
numerical = np.array([(loss(theta + h * e) - loss(theta - h * e)) / (2 * h)
                      for e in np.eye(3)])
print('weight shape:', gradient[:2].shape, 'bias shape:', gradient[2].shape)
print('analytic:', np.round(gradient, 6).tolist())
print('numerical:', np.round(numerical, 6).tolist())
print('small step decreases loss:', bool(loss(theta - .01 * gradient) < loss(theta)))
assert np.allclose(gradient, numerical, atol=1e-9)

Checked output:

weight shape: (2,) bias shape: ()
analytic: [-1.066667, -2.433333, -1.833333]
numerical: [-1.066667, -2.433333, -1.833333]
small step decreases loss: True

Interpret the sign without promising convergence

A negative gradient component means increasing that parameter decreases the loss locally. Gradient descent subtracts the gradient, so a negative component leads to a positive parameter update. The small update in the example reduces the loss, consistent with the derivative.

A very large step can increase the loss even when the derivative is correct. The gradient gives local information, and the loss changes as you move. The learning-rate experiment isolates that issue on a quadratic objective.

The residual convention matters. If you define r = y - prediction, the derivative needs the corresponding negative sign. Mixing one convention in the loss explanation with another in the gradient is a common source of incorrect updates.

Choose a useful finite-difference check

An extremely small h can cause subtractive cancellation; a large h may poorly approximate local behavior for a nonlinear function. Test several values, use floating-point precision appropriate to the check and compare both absolute and relative error. Our squared loss is unusually friendly to central differences, so its agreement should not be generalized to every objective.

Nondifferentiable points need separate care. At a ReLU kink or an absolute-value corner, a symmetric difference may not match the derivative convention used by an automatic differentiation library. Select a smooth test point before treating a mismatch as a library bug.

Extend to batches and networks

Add a fourth row and predict how the average changes. Then remove the 1/2 from the loss and derive the new gradient scale. Finally, add a nonlinear activation and write the chain-rule factors before asking an autodiff library for the result.

The Maths course develops derivatives and optimization. Continue with matrix multiplication shapes if the transpose is unclear, or verify the chain rule in a small neural network. Those checks prepare you to distinguish calculus mistakes from training choices.

References

NumPy finite differences — Distinguish numerical differentiation from an analytically derived parameter gradient.

PyTorch autograd mechanics — How automatic differentiation records operations and propagates gradients.