Machine Learning · Module 3

Linear Regression and Gradient Descent

Take each lesson, answer its quiz, then do its practice. After the last lesson come the module quiz, the project and the interview questions.

Lessons

3.1Simple Linear RegressionPredict a number from one feature with a straight line, choose the line that minimises squared error, and read what its slope and intercept say.
3.2Gradient DescentFind the minimum of a loss without solving equations, by stepping against the gradient, and tell from the loss curve whether a run converges, crawls or diverges.
3.3Gradient Descent for Linear RegressionDerive the gradient of the squared-error loss, see why one update reads every row, and know what stochastic and mini-batch descent buy.
3.4Multiple Linear RegressionPredict from several features with one weight each, and read a coefficient as the change in the prediction with the other features held fixed.
3.5Why Gradient Descent Is Slow: Conditioning and Feature ScalingPredict from the data's shape how many updates descent will need, and see why scaling the features fixes it.
3.6Regression Diagnostics and Robust RegressionRead residual plots to find what a linear model gets wrong, tell an outlier from a point that controls the fit, and choose a loss that one bad row cannot dominate.
3.7CollinearitySee why correlated features make individual coefficients unstable while the predictions stay good, and measure how unstable they are.

Module material

Take these after the last lesson, in this order. Next on each page leads through them.

Glossary

Every term the module introduces, with the lesson that introduces it.

baseline (reminder, ML 1.1): Here, the mean mpg for every car: RMSE 8.01. 3.1
RMSE: The square root of the mean squared error, in the units of the label. 3.1
linear regression: Predicting a number with a straight line, or with a weighted sum of features. 3.1
slope: The change in the prediction per unit of the feature. 3.1
intercept: The prediction when every feature is zero. 3.1
residual: The actual value minus the prediction. 3.1
squared error: A residual multiplied by itself. 3.1
mean squared error: The average of the squared residuals. 3.1
least squares: Choosing the parameters that minimise the squared error. 3.1
contour map (reminder, Maths 4.4): Lines of equal loss over the plane of two parameters. 3.1
R²: One minus the model's squared error over the mean's squared error: the fraction of the mean's squared error the model removes. 3.1
extrapolation: Predicting outside the range of the training data. 3.1
gradient descent: Repeatedly stepping the parameters against the gradient of the loss. 3.2
update rule: w ← w − η · L′(w). 3.2
learning rate: The hyperparameter that scales each step, η. 3.2
iteration: One application of the update rule; also called an update. 3.2
convergence: The parameters settling at the minimum. 3.2
divergence: Each step landing further from the minimum. 3.2
tolerance: How small the gradient must be before descent stops. 3.2
loss curve: The loss plotted after every update. 3.2
local minimum (narration): A low point that is not the lowest, where descent can end on a loss with several. 3.2
learning-rate schedule: A rule that changes the learning rate during training. 3.2
batch gradient descent: Every update uses every training row. 3.3
stochastic gradient descent: Each update uses one training row, in shuffled order. 3.3
epoch: One pass through every training row. 3.3
mini-batch: A small group of training rows used for one update. 3.3
batch size: The number of rows in a mini-batch. 3.3
closed form: A formula that gives the minimum directly, without iterating. 3.3
design matrix: The feature table the model is fitted on, one row per sample, X. 3.4
intercept column: A column of ones whose weight is the intercept. 3.4
multiple linear regression: A linear model with more than one feature. 3.4
normal equations: XᵀX w = Xᵀy: every partial derivative of the squared error set to zero. 3.4
coefficient: The weight on one feature in a linear model. 3.4
held fixed: Read with every other feature in the model unchanged. 3.4
standardised coefficient: The change in the prediction for a one-standard-deviation change in the feature. 3.4
reference category: The category with no column of its own, which the other categories' coefficients are compared against. 3.4
Hessian (reminder, Maths 4.5): The matrix of second derivatives; for the mean squared error, 2XᵀX / n. 3.5
curvature along a direction: How fast the slope of the loss changes as one moves that way. 3.5
condition number: The largest eigenvalue of the Hessian over the smallest, κ. 3.5
ill-conditioned: Having a large condition number. 3.5
contraction factor: The number the remaining error is multiplied by at each update. 3.5
centring: Subtracting each feature's mean. 3.5
standardising: Centring each feature, then dividing it by its standard deviation. 3.5
fitted value: The model's prediction for a training row. 3.6
residual plot: Residuals plotted against fitted values. 3.6
heteroscedasticity: Residuals whose spread changes with the fitted value. 3.6
outlier: A row far from the model's prediction. 3.6
leverage: How far a row's features are from the other rows', which sets how much it can turn the fit. 3.6
hat matrix: The matrix that turns y into the fitted values; its diagonal is the leverage. 3.6
influence: How much a row moves the fit. 3.6
Cook's distance: How much the fitted values change when one row is left out. 3.6
absolute loss: The size of the residual. 3.6
Huber loss: Squared inside a threshold and linear outside it. 3.6
robust regression: Fitting with a loss that a few bad rows can not dominate. 3.6
median (reminder, Maths 4.3): The value that minimises the absolute loss. 3.6
RANSAC: Fitting on random minimal samples and keeping the one most rows agree with. 3.6
consensus set: The rows within the threshold of the chosen fit. 3.6
collinearity: Features that are close to linear functions of each other. 3.7
multicollinearity: Collinearity among more than two features. 3.7
bootstrap resample: A sample of the same size drawn from the data with replacement. 3.7
variance inflation factor: How many times larger a coefficient's variance is because the feature can be predicted from the other features: 1 / (1 − R²). 3.7
perfect collinearity: A feature that is an exact linear function of other features (singular matrices: Maths 3.3, second pass). 3.7
dummy-variable trap: One 0/1 column for every category, beside an intercept, which makes the columns perfectly collinear. 3.7