Data and modeling
Machine learning for linear algebra
Linear algebra is the maths machine learning runs on: a dataset is a matrix, linear regression solves a system of equations, PCA and recommenders use matrix factorisation, and neural network layers are matrix multiplications. Embeddings and similarity search rely on dot products and norms.
How linear algebra is used in machine learning
Matrix factorisation for recommendations
A ratings table of users by items is approximated as the product of two thin matrices, one describing users and one describing items. Missing ratings are then predicted from the dot product of a user's and an item's vectors.
Real exampleSimon Funk, 'Netflix Update: Try This at Home' (2006)Funk explains how he approximated the Netflix Prize rating matrix as the product of a movie-aspect matrix and a user-preference matrix using singular value decomposition.
Word and item embeddings
Embeddings represent words, products or users as vectors, so that similar things have a large dot product. They feed search, recommendations and language models.
Real exampleword2vec (Mikolov et al., 2013)The paper proposes model architectures for computing continuous vector representations of words from very large datasets.
Similarity search over vectors
Retrieval systems find the nearest vectors to a query among millions or billions, using distances and dot products computed with fast matrix operations on GPUs.
Real exampleBillion-scale similarity search with GPUs (Johnson, Douze and Jégou, 2017)The paper presents GPU methods for k-nearest-neighbour similarity search on high-dimensional vectors, scaling to a graph connecting 1 billion vectors, and open-sources the approach.
Attention in transformers
Transformer attention multiplies matrices of queries, keys and values to decide how much each token draws on every other. Language models run these products at every layer.
Real exampleAttention Is All You Need (Vaswani et al., 2017)The paper introduces the Transformer, an architecture based solely on attention mechanisms, computed as matrix products of queries, keys and values.
Low-rank adaptation of large models
Fine-tuning a large model can be done by learning a small low-rank update to each weight matrix instead of changing all its entries. This uses the idea of matrix rank directly.
Real exampleLoRA (Hu et al., 2021)LoRA freezes pretrained weights and injects trainable rank decomposition matrices into each layer of the Transformer.
Can you learn linear algebra for machine learning?
Yes. The linear algebra machine learning needs is a focused subset: vectors, dot products, matrix multiplication, and a working idea of eigenvectors and the SVD. Learn it alongside NumPy so every idea becomes code you can run.
- Python and NumPyWrite vectors and matrices as NumPy arrays and use them to check every calculation.Learn it in Python
- VectorsLearn vector addition, the dot product, norms and cosine similarity, the basis of embeddings and distances.Learn it in Maths
- MatricesLearn matrix multiplication, inverses, rank, eigenvectors and the SVD, and see how a dataset is a matrix.Learn it in Maths
- Linear algebra inside modelsSee linear regression as a matrix equation, PCA as an SVD and a network layer as a matrix product.Learn it in Machine Learning
Where to go after linear algebra for ML
If you already work in the field
If you studied linear algebra in maths, physics or engineering, you already have the theory. Connect it to ML by implementing linear regression, PCA and a small network layer in NumPy, then compare with scikit-learn and PyTorch.
If you come from data science or ML
If you use ML libraries without the maths, revisit vectors, matrices and the SVD with a course that pairs each idea with code. It makes embeddings, attention and dimensionality reduction much easier to reason about.
Roles that do this workMachine learning engineerResearch scientistData scientistQuantitative analyst
Tools you will use
| Tool | What you use it for |
|---|---|
| NumPy linear algebra | Multiply matrices, solve linear systems and compute eigenvalues and SVDs. |
| SciPy linalg | Use further decompositions and solvers when problems grow larger. |
| scikit-learn decomposition | Run PCA, truncated SVD and non-negative matrix factorisation on data. |
| JAX | Write array code that differentiates and compiles, to see gradients as linear algebra. |
| Faiss | Search large collections of vectors by similarity using efficient indexes. |
Public data to practise on
- Optical Recognition of Handwritten Digits (UCI): Small handwritten-digit images stored as numeric vectors, ideal for PCA, SVD and nearest-neighbour practice.
Your first project
Download the Optical Recognition of Handwritten Digits dataset, compress the images to their top 10 SVD components, classify with k-nearest neighbours, and compare it with k-nearest neighbours on the raw pixels using test accuracy.
Common questions
How much linear algebra do I need for machine learning?
For applied work, vectors, dot products, matrix multiplication and an intuitive grasp of eigenvectors and SVD. Research and building new methods need more, including matrix calculus and decompositions.
Should I learn linear algebra or calculus first for machine learning?
Either order works, but linear algebra is useful sooner because data, weights and layers are all vectors and matrices. Calculus becomes essential when you study how models are trained with gradients.
Why are GPUs used for machine learning?
Training and running models is dominated by large matrix multiplications, and GPUs perform many of these arithmetic operations in parallel.
What is the SVD used for in machine learning?
The singular value decomposition underlies PCA, low-rank approximation, latent semantic analysis and some recommender methods. It finds the directions that capture most of the variation in a matrix.
Courses for these skills
- Maths for Machine LearningUnderstand derivatives and matrix shapes.26 published lessons · all planned lessons open
- Generative AI: LLMs, RAG and AgentsConnect embeddings and retrieval to language-model applications.Syllabus available · lessons open as produced
- Practical Machine Learning in PythonLearn regression, loss functions and generalisation on measured features.51 published lessons · more lessons in production
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnInspect numeric columns, missing values and sample-level tables.23 published lessons · all planned lessons open
- Deep Learning with PyTorchConnect gradients to neural training objectives.Syllabus available · lessons open as produced
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.