Machine Learning · Module 4

The First Classifier, Generalisation and Regularisation

Take each lesson, answer its quiz, then do its practice. After the last lesson come the module quiz, the project and the interview questions.

Lessons

4.1Distance and SimilarityMeasure how alike two rows are, and see how the choice of measure and the feature scales change who counts as a neighbour.
4.2k-Nearest NeighboursClassify a new point by a vote of its nearest neighbours, and see what K and the distance do to the boundary.
4.3Overfitting, Underfitting and the Test SetKnow why a score on the training data is not evidence, and read the train/test gap as a diagnosis.
4.4Cross-ValidationEstimate how a model will do on unseen data from the training set alone, with a split that matches how the data was collected.
4.5Hyperparameter SearchChoose hyperparameters with a search that spends a fixed budget well, and report a score the search did not bias.
4.6Learning Curves and Validation CurvesDecide from two curves whether more data, more capacity or less capacity will help.
4.7Polynomial Regression: Choosing Model CapacityFit curves with a linear model through basis functions, and choose the degree by validation.
4.8Ridge RegressionPenalise the size of the weights to trade a little bias for less variance, and choose the penalty's strength by cross-validation.
4.9Lasso and Elastic NetUse an L1 penalty to drive some weights exactly to zero, and know when elastic net is the better choice.
4.10k-NN at Scale: kd-Trees and HashingFind neighbours without comparing a query against every row.In production

Module material

Take these after the last lesson, in this order. Next on each page leads through them.

Glossary

Every term the module introduces, with the lesson that introduces it.

norm (reminder, Maths 2.2): The length of a vector; a distance is the norm of a difference. 4.1
standardising (reminder, Maths 2.2): Subtracting a feature's training mean and dividing by its training standard deviation. 4.1
nearest neighbor: The row with the smallest distance to a given row. 4.1
Euclidean distance: The square root of the sum of squared differences. 4.1
Manhattan distance: The sum of the absolute differences. 4.1
Chebyshev distance: The largest absolute difference. 4.1
Minkowski distance: (Σ | dᵢ | ᵖ)^(1/p); Manhattan at p = 1, Euclidean at p = 2; a distance only for p ≥ 1. 4.1
Hamming distance: The number of positions at which two rows differ. 4.1
cosine similarity (reminder, Maths 2.3): The cosine of the angle between two vectors. 4.1
cosine distance: One minus cosine similarity. 4.1
baseline (reminder, ML 1.1): Here, benign for every training tumour: 62.72%. 4.2
k-nearest neighbors: A classifier that predicts the majority label of the K nearest training rows. 4.2
vote: The count of each label among them. 4.2
hyperparameter (reminder, ML 1.1): A choice made before fitting: here K and the distance. 4.2
lazy learner: A model that stores the training rows and does its work when it predicts. 4.2
probabilistic label: The fraction of the K neighbors with each label. 4.2
tie: Equal votes for two labels. 4.2
weighted k-NN: Votes weighted by one over distance. 4.2
decision surface: The predicted label at every point of the feature space. 4.2
Voronoi cell: The region nearest to one training row. 4.2
k-NN regression: Predicting the mean label of the K nearest rows. 4.2
training set: The rows the model is fitted on. 4.3
validation set: Rows held out from fitting, used to choose a hyperparameter. 4.3
test set: Rows used once, to report the chosen model. 4.3
stratified split: A split that keeps each class's share the same in every part. 4.3
stratified sample (reminder, Stats 4.1): A sample drawn within groups, in proportion. 4.3
pipeline (reminder, Tools 4.3): Preprocessing and a model fitted together, so each step learns from the fitting rows only. 4.3
overfitting: Fitting the training rows much better than new rows. 4.3
underfitting: Fitting both poorly. 4.3
variance: How much the model changes with the training rows. 4.3
bias: Error from a model too simple for the pattern. 4.3
capacity: How closely a model can follow its training rows. 4.3
cross-validation: Estimating a model's score by fitting and validating on several splits of the training rows. 4.4
k-fold: k splits, each part validating once. 4.4
fold: One of the k parts. 4.4
stratified k-fold: Folds that keep each class's share. 4.4
standard error (reminder, Stats 5.1): How much an estimate moves from one sample to the next. 4.4
standard error of a cross-validated score: The fold scores' standard deviation over the square root of the number of folds. 4.4
group k-fold: Folds that keep each group's rows together. 4.4
time-based split: Fitting on earlier rows and validating on later ones. 4.4
grid search: Trying every combination of a set of hyperparameter values. 4.5
random search: Trying randomly sampled settings. 4.5
search budget: The number of fits a search may use. 4.5
successive halving: Running many settings on little data and giving more data to the best. 4.5
Bayesian optimization: Choosing each next setting from the scores of the settings tried so far. 4.5
nested cross-validation: An outer cross-validation that scores a whole search run inside each outer fold. 4.5
one-standard-error rule: Choose the simplest setting within one standard error of the best. 4.5
learning curve: Training and validation score against the number of training rows. 4.6
high-variance signature: A training curve well above the validation curve. 4.6
high-bias signature: Training and validation curves that meet at a low score. 4.6
validation curve: Training and validation score against a hyperparameter. 4.6
polynomial regression: Linear regression on powers of a feature. 4.7
polynomial features: The powers added as columns. 4.7
basis function: A transformed feature used as a column. 4.7
degree: The highest power in a polynomial model. 4.7
capacity (reminder, ML 4.3): Here, the degree. 4.7
one-standard-error rule (reminder, ML 4.5): Here, the smallest degree within one standard error of the best. 4.7
interaction term: The product of two features. 4.7
spline: A curve built from joined low-degree pieces. 4.7
bias, variance (reminder, ML 4.3): The two parts of the error that the penalty trades. 4.8
ridge regression: Linear regression with a penalty on the squared size of the weights. 4.8
penalty strength: The hyperparameter that sets how much the penalty counts: alpha in scikit-learn, λ in papers. 4.8
regularization: Adding a penalty on the weights to the objective. 4.8
L2 penalty: The sum of squared weights. 4.8
L2 norm (reminder, Maths 2.2): The square root of the sum of squares. 4.8
shrinkage: Pulling the weights toward zero. 4.8
coefficient path: Every weight plotted against the penalty strength. 4.8
collinearity (reminder, ML 3.7): Features that move together. 4.8
lasso: Linear regression with a penalty on the absolute size of the weights. 4.9
L1 penalty: The sum of absolute weights. 4.9
L1 norm (reminder, Maths 2.2): The sum of absolute values; its unit ball is a diamond. 4.9
sparsity: Many weights exactly zero. 4.9
coordinate descent: Minimizing over one weight at a time, with the others held fixed. 4.9
soft thresholding: Shrinking a value toward zero by α and setting it to zero when it is within α of zero. 4.9
lasso path: Every weight plotted as α decreases. 4.9
elastic net: A penalty mixing the L1 and L2 penalties. 4.9
L1 ratio: The share of the L1 penalty in that mix. 4.9
selection instability: A different sample choosing different features. 4.9