Maths · Module 3

Matrices

Take each lesson, answer its quiz, then do its practice. After the last lesson come the module quiz, the project and the interview questions.

Lessons

3.1Matrices, the Matrix–Vector Product and the TransposeSee a dataset as a matrix, and a linear model's predictions as one matrix–vector product.
3.2Matrix Multiplication as TransformationRead a matrix as something that moves space: rotate, stretch, shear, project.
3.3Identity, Inverse and Singular MatricesUse the inverse as the transformation that undoes another, and know when it cannot exist.
3.4Linear Systems, Rank and Least SquaresSolve `Ax = b` exactly when it can be solved, and in the least-squares sense when it cannot. Derive the normal equations.
3.5Eigenvalues and EigenvectorsFind the directions a matrix only stretches, and read the eigenvalues as how much.
3.6Singular Value DecompositionFactor any matrix into rotate–stretch–rotate, and keep only the largest parts.
3.7The Covariance MatrixBuild the covariance matrix from a centred data matrix and read its entries and its shape.

Module material

Take these after the last lesson, in this order. Next on each page leads through them.

Glossary

Every term the module introduces, with the lesson that introduces it.

matrix: a rectangular table of numbers, treated as one object; rows are samples, columns are features | X = [[2, 1], [0, 3], [4, 2]] 3.1
shape: the number of rows, then the number of columns | (3, 2) 3.1
entry: one number in a matrix, xᵢⱼ | x₃₂ = 2 3.1
matrix–vector product: each row of the matrix dotted with the vector, one output per row | Xw = (5, −3, 10) 3.1
linear combination: vectors, each multiplied by a number, then added | 3·col₁ − 1·col₂ 3.1
shape rule: the matrix's number of columns must equal the vector's length; the result has one entry per row | (3, 2) @ (2,) → (3,) 3.1
transpose: rows become columns; an n × d matrix becomes d × n | Xᵀ, (2, 3) 3.1
column mean: the average of one column, one number per feature | (2, 2) 3.1
centred matrix: each column minus its mean, so every column mean is zero | [[0, −1], [−2, 1], [2, 0]] 3.1
basis vectors: e₁ = (1, 0) and e₂ = (0, 1); every point is a combination of them | (1, 2) = e₁ + 2e₂ 3.2
columns of a matrix: where e₁ and e₂ land | [[2, 1], [1, 3]]: (2, 1) and (1, 3) 3.2
linear transformation: moves the plane so grid lines stay straight, parallel and evenly spaced, with the origin fixed | every matrix 3.2
projection: a transformation that flattens the plane onto a line | [[1, 0], [0, 0]] 3.2
determinant: the factor a matrix scales every area by; ad − bc for a 2 × 2 matrix | 5 3.2
sign of the determinant: negative when the transformation flips the plane over | the flip, −1 3.2
composition: one transformation applied after another | shear, then rotate 3.2
matrix product: the single matrix that does both | RH = [[0, −1], [1, 1]] 3.2
shape rule for products: (m, k) @ (k, n) → (m, n) | (2, 3) @ (3, 4) → (2, 4) 3.2
identity matrix: I, ones on the diagonal and zeros elsewhere; it leaves every vector unchanged | np.eye(2) 3.3
inverse: A⁻¹, the matrix that undoes A: A⁻¹A = AA⁻¹ = I | [[1, −1], [−1, 2]] 3.3
singular matrix: a square matrix with no inverse; it flattens at least one direction, and its determinant is zero | [[1, 2], [2, 4]] 3.3
rank: the number of dimensions a matrix's output fills | 1 for the singular example 3.3
condition number: how much a matrix can magnify a small relative change; 1 is perfect, and large means nearly singular | 25,008 for [[1, 2], [2, 4.001]] 3.3
linear system: equations in which each unknown is only multiplied by a number and added | x₁ + x₂ = 3 3.4
span: every combination of a set of vectors | a plane through the origin 3.4
column space: the span of a matrix's columns, every possible Ax | 3.4
linear independence: no column is a combination of the others | (1, 1, 1) and (0, 1, 2) 3.4
rank: the number of independent columns, the dimension of the column space | 2 3.4
residual: b − Ax, what the fit misses | (−0.5, 1, −0.5) 3.4
least squares: the x that makes the squared length of the residual smallest | x̂ = (1.5, 0.5) 3.4
normal equations: AᵀAx̂ = Aᵀb, from the residual being perpendicular to every column | [[3, 3], [3, 5]] x̂ = (6, 7) 3.4
eigenvector: a nonzero vector the matrix only stretches, Av = λv | (1, 1) 3.5
eigenvalue: λ, the factor it is stretched by | 3 3.5
symmetric matrix: equal to its own transpose; real eigenvalues, perpendicular eigenvectors | XᵀX 3.5
eigendecomposition: A = QΛQᵀ: rotate, stretch along the axes, rotate back | Λ = diag(1, 3) 3.5
quadratic form: xᵀAx, a quadratic built from a symmetric matrix | 2x² + 2xy + 2y² 3.5
singular value: the length of a semi-axis of the ellipse; how much the matrix stretches one direction | 6.7082, 2.2361 3.6
singular vectors: vᵢ, the input directions, and uᵢ, where they land | v₁ at 45°, u₁ at 71.57° 3.6
SVD: A = UΣVᵀ: rotate, stretch along the axes, rotate | every matrix 3.6
low-rank approximation: keep the k largest singular values; the best possible approximation of rank k | digits, k = 21 3.6
covariance matrix: (1/n) XᵀX for centred X; one row and column per feature | [[5, 4], [4, 5]] 3.7
variance: the mean squared distance from the mean | 5 3.7
covariance: the mean of the products of two centred features; positive when they rise together | 4 3.7
correlation: covariance divided by both standard deviations, between −1 and 1 | 0.8 3.7
covariance ellipse: axes along the covariance matrix's eigenvectors, semi-axes the square roots of its eigenvalues | 3 and 1 3.7
correlation matrix: the covariance matrix of standardised features; ones on the diagonal, correlations off it | [[1, 0.9588], [0.9588, 1]] 3.7