Linear regression and gradient descent cheat sheet
The fitted line, gradient descent, batch and mini-batch updates, multiple regression, conditioning, diagnostics and collinearity, with worked values from Auto MPG. Free to read and share, no account needed.
The line (3.1)
| formula | on the cars | |
|---|---|---|
| model | ŷ = w·x + b | |
| objective | mean squared error, mean((y − ŷ)²) | |
| slope | w = Σ(x − x̄)(y − ȳ) / Σ(x − x̄)² = covariance / variance | −7.80 mpg per 1,000 lb |
| intercept | b = ȳ − w·x̄ (the line passes through the point of means) | 46.88 |
| RMSE | √MSE, in mpg | mean (baseline) 8.01; line 4.34 |
| R² | 1 − SSE_model / SSE_mean; with one feature, r² | 0.706 = (−0.840)² |
Read a line only inside its data: 7,000 lb predicts −7.72 mpg (extrapolation).
Gradient descent (3.2)
new parameters = old − η × gradient. Stop when the gradient's length is below
the tolerance (0.001 here), with an update limit as a backstop.
- On
(w − 3)²each step multiplies the distance to 3 by1 − 2η: it converges for 0 < η < 1, lands in one step at 0.5, diverges at 1.1. - On the cars: η 0.01 / 0.05 / 0.09 take 5,699 / 1,137 / 630 updates to (−7.80,
46.88). The limit is
2/λmax= 0.0962: at 0.1 the loss falls for five updates, then climbs. - Watch the whole loss curve; never judge a rate by its first updates.
The gradient, and how many rows per update (3.3)
∂L/∂w = −2·mean(r·x), ∂L/∂b = −2·mean(r), with r = y − ŷ.
| method | rows per update | updates per epoch (318 rows) |
|---|---|---|
| batch | all 318 | 1 |
| stochastic | 1 | 318 |
| mini-batch (32) | 32 | 10 (the last holds 30) |
A constant rate makes stochastic and mini-batch hover; a shrinking rate
lets them settle. LinearRegression solves the closed form; check any
descent fit against it (SGDRegressor() defaults: −7.217, adaptive schedule:
−7.797).
Several features (3.4)
ŷ = Xw, X = a ones column, then one column per feature: (318, 3) for
weight and year. Least squares solves the normal equations XᵀXw = Xᵀy
(np.linalg.lstsq).
- Weight and year: b = −12.088, −6.851 per 1,000 lb, +0.7364 per year; RMSE 3.50, R² 0.809.
- A coefficient is the change in the prediction per unit of its feature with every other feature held fixed: weight went from −7.80 alone to −6.85 with year, because newer cars were lighter.
- Compare sizes only after standardising: −5.907 and +2.697 per standard deviation.
- Categories: one 0/1 column per category except a reference (Europe +1.98, Japan +2.38 against an American car of the same weight and year).
- A better training score is not evidence: adding a feature can never raise the training squared error.
Collinearity (3.7)
| check | engine columns (313 cars) |
|---|---|
| correlations | 0.847 to 0.951 for every pair |
| bootstrap, 200 refits | displacement's coefficient negative in only 60.5%; predictions move by 0.505 mpg |
VIF = 1 / (1 − R²_j) | displacement 19.01, cylinders 10.84, weight 8.59, horsepower 5.44 |
| exact copy | LinAlgError: Singular matrix; LinearRegression splits it, −3.8923 each |
Coefficients swing, predictions hold. Remedies: one feature per group, combine the group, or penalise the weights (ridge, Module 4). Drop one category column (the dummy-variable trap).
Conditioning (3.5, second pass)
Hessian 2XᵀX / n; κ = λmax / λmin; at the rate 1/λmax, updates ≈
log(10⁻⁶) / log(1 − 1/κ) ≈ 13.8 κ.
| features | κ | updates |
|---|---|---|
| weight in 1,000 lb | 145.4 | 2,001 run (2,002 predicted) |
| weight in lb, year | 5.11 × 10⁹ | 7.06 × 10¹⁰ predicted |
| standardised weight, year | 1.87 | 15 run (18 predicted) |
Standardise on the training rows, and divide each weight by its standard deviation to read it per unit again.
Diagnostics (3.6, second pass)
| pattern on the residual plot | meaning | the line's version | action |
|---|---|---|---|
| curve | wrong shape | group means +1.06 … +1.91 | add x², or predict 100 / mpg |
| fan | error size depends on the prediction | spreads 2.23 → 5.08 | report errors by range |
large residual, ordinary x | outlier | Mazda GLC +16.17, slope moves to −7.742 | look at it |
far in x | leverage | Pontiac Safari 0.0234 (mean 0.0063) | check it |
| both | influence (Cook's distance) | typo on the Safari: 2.70, slope −6.72 | fix the data |
Huber and RANSAC limit one bad row, but they are different objectives.
The code
X = np.column_stack([np.ones(len(train)), train["weight_k"], train["model_year"]])
w = np.linalg.lstsq(X, y, rcond=None)[0] # [-12.0878 -6.8509 0.7364]
LinearRegression().fit(train[["weight_k", "model_year"]], y) # the same, intercept apart
g = 2 * X.T @ (X @ w - y) / len(y) # the gradient of the MSE
lam = np.linalg.eigvalsh(2 * X.T @ X / len(y)) # curvature; kappa = lam[-1] / lam[0]
leverage = np.diag(X @ np.linalg.inv(X.T @ X) @ X.T)