Maths · Module 4

Calculus and Optimisation

Take each lesson, answer its quiz, then do its practice. After the last lesson come the module quiz, the project and the interview questions.

Lessons

4.1Derivatives: Rate of Change and the TangentSay what a derivative measures, and read its sign and size off a graph.
4.2Derivative Rules and the Chain RuleDifferentiate the functions ML uses, including functions of functions.
4.3Maxima, Minima and Setting the Derivative to ZeroFind where a function is lowest or highest, and know when that cannot be done by solving an equation.
4.4Partial Derivatives and the GradientDifferentiate a function of several variables, and read the gradient as the direction of steepest increase.
4.5Curvature: Second Derivatives, Taylor and NewtonUse the second derivative to see how a function bends, and step straight to the bottom of a local quadratic.
4.6Integrals as AreaRead an integral as accumulated area, enough for densities and AUC.
4.7Convexity, Local Minima and Saddle PointsTell a convex function from a non-convex one, and know what a zero gradient does and does not guarantee.
4.8The Chain Rule Through a ComputationCompute the gradient of a loss that is built from many steps, by passing derivatives backwards through a graph of operations.
4.9Constrained Optimisation and Lagrange MultipliersMinimise a function subject to a constraint, and read the multiplier.

Module material

Take these after the last lesson, in this order. Next on each page leads through them.

Glossary

Every term the module introduces, with the lesson that introduces it.

rate of change: Δy/Δx, the change in the output per unit change in the input | x² from 1 to 2: 3 4.1
secant: a line through two points of a curve | through (1, 1) and (2, 4), slope 3 4.1
limit: the value a quantity approaches as h goes to zero | 2 + h → 2 4.1
tangent: the line that touches the curve at a point with the curve's slope there | y = 2x − 1 at x = 1 4.1
derivative: the slope of the tangent, the limit of Δy/Δx | 2 for x² at 1 4.1
f′(x): the derivative of f, a function of x | x² → 2x 4.1
dy/dx: the same derivative, written as a ratio of small changes | d(x²)/dx = 2x 4.1
power rule: the derivative of xⁿ is n xⁿ⁻¹ | x³ → 3x², 12 at 2 4.2
chain rule: the derivative of f(g(x)) is f′(g(x)) · g′(x) | (3 − 2x)² → 2(3 − 2x)(−2) 4.2
partial derivative: the derivative with respect to one variable, the others held fixed | ∂/∂w (y − wx − b)² = −9.6 for the 4-year-old car 4.2
stationary point: a point where the derivative is zero | x² − 3x + 2 at 1.5 4.3
minimum: the derivative changes from negative to positive | f′(1) = −1, f′(2) = 1 4.3
maximum: the derivative changes from positive to negative | x⁴ − 4x² + x at 0.126 4.3
local minimum: lower than every nearby point | 1.347, value −2.6186 4.3
global minimum: lower than every point | −1.473, value −5.4442 4.3
bisection: halve an interval that contains a zero, keeping the half where the sign changes | 2x + eˣ = 0 on [−1, 0] 4.3
iterative method: a method that improves an answer step by step | bisection, gradient descent 4.3
median: the middle value; it minimises the sum of absolute errors | 13.2 for the six hero prices 4.3
partial derivative: the derivative with respect to one variable, with every other variable held fixed | ∂(x² + 3y²)/∂x = 4 at (2, 1) 4.4
gradient: the vector of all the partial derivatives | (4, 6) 4.4
contour: a curve joining points where the function has the same value | the ellipse x² + 3y² = 7 4.4
directional derivative: the slope of f in the direction of a unit vector u, ∇f · u | 6.4641 at 30° 4.4
second derivative: the derivative of the derivative, f″(x) | x³ − 3x → 6x 4.5
curvature: how sharply a curve bends; its sign says up or down | +6 at 1, −6 at −1 4.5
Hessian: the matrix of all second partial derivatives | diag(2, 6) for x² + 3y² 4.5
Taylor approximation: a polynomial built from a function's derivatives at one point, matching it nearby | 1 + x + 1.5x² for x² + eˣ at 0 4.5
Newton's method: repeatedly jump to the minimum of the local parabola, x − f′/f″ | three steps to −0.351734 4.5
strip sum: the total area of thin rectangles under a curve | 0.46875 with four right-edge strips 4.6
integral: the limit of the strip sums as the strips get thinner | ∫₀¹ x² dx = 1/3 4.6
antiderivative: a function whose derivative is the given function | x³/3 for x² 4.6
fundamental theorem of calculus: the area's rate of growth is the curve's height | A′(x) = x² 4.6
density: a curve with total area 1, whose area over an interval is the probability of landing there | 2x on [0, 1] 4.6
ROC curve: the true positive rate against the false positive rate as the threshold moves | the tiny ward's seven points 4.6
AUC: the area under the ROC curve | 0.7656 on the tiny ward 4.6
chord: a straight line between two points of a curve | x² from −2 to 1 4.7
convex function: every chord lies on or above the curve | x², log(1 + eˣ) 4.7
strictly convex: every chord lies strictly above the curve; at most one minimiser | x² (one), log(1 + eˣ) (none) 4.7
saddle point: a point with zero gradient that is neither a minimum nor a maximum | the origin of x² − y² 4.7
computation graph: a calculation drawn as nodes, one operation each, joined by the values they pass | u = 3a, v = u + b, f = v² 4.8
forward pass: compute each node's value from its inputs, in order | 3, 5, 25 4.8
local derivative: the derivative of one node with respect to its own input | ∂f/∂v = 2v = 10 4.8
backward pass: start from the output and multiply local derivatives back along each edge | ∂f/∂a = 10 · 1 · 3 = 30 4.8
backpropagation: the backward pass on a computation graph, each local derivative used once | how deep learning libraries compute gradients 4.8
constraint: a condition the answer must satisfy, such as g(x, y) = c | x + y = 1 4.9
Lagrange multiplier: λ in ∇f = λ∇g at a constrained optimum | λ = −2 4.9
Lagrangian: L = f − λ(g − c); its stationary point gives the constrained optimum | three equations for x, y, λ 4.9
active constraint: an inequality that holds with equality at the optimum | x + y ≤ 1 at (1, 0) 4.9
complementary slackness: either the constraint is tight or its multiplier is zero | x + y ≤ 4: multiplier 0 4.9
KKT conditions: the conditions for an optimum with inequality constraints | gradient condition, feasibility, μ ≥ 0, slackness 4.9