Data and modeling
Machine learning for linear regression
Linear regression is the first machine learning model most people fit: it predicts a number as a weighted sum of features. It is used to estimate prices, demand, quality scores and measured responses, and it is the baseline every more flexible model should beat.
Where linear regression is used in machine learning
Predicting a measured quantity from many features
Linear regression with a penalty, such as ridge, lasso or elastic net, can predict a number from thousands of inputs while keeping only the useful ones. It is fast, stable and easy to inspect, so it is used even when the inputs are very high-dimensional.
Real exampleHorvath's DNA methylation age (Genome Biology, 2013)Horvath developed a multi-tissue predictor of age from DNA methylation data using 8,000 samples from 82 datasets, with a penalised (elastic net) linear regression.
Marketing mix modelling
Advertisers regress sales on spend per channel, with transformed terms for delayed and diminishing effects, to estimate how much each channel contributes. Regularised and Bayesian versions handle the strong correlation between channels.
Real exampleBayesian Methods for Media Mix Modeling with Carryover and Shape Effects (Google)Google researchers describe a Bayesian regression model of media effects that includes carryover and shape (diminishing returns) terms.
Regularising correlated inputs
When inputs move together, ordinary least squares gives unstable coefficients. Ridge regression shrinks them towards zero, trading a little bias for much lower variance, which is why it is a common default in applied tools.
Real exampleRobyn (Meta, open source)Meta's marketing mix package uses ridge regression to regularise multicollinearity between channels and prevent overfitting.
A baseline for prediction problems
Before fitting boosted trees or neural networks, practitioners fit a linear model to see how much a simple, explainable model already achieves. Any more complex model has to beat it on held-out data to earn its place.
Real exampleMachine Learning: An Applied Econometric Approach (Mullainathan and Spiess, 2017)The authors set out how machine learning prediction relates to the regression tools economists already use, and where each fits in empirical work.
Can you learn linear regression for machine learning?
Yes. Linear regression is the usual first model and needs only school algebra to start. Learn to fit and evaluate it in Python, then the matrix form and gradient descent, then regularisation.
- Python and NumPyWrite Python and use NumPy arrays so you can compute predictions and errors yourself.Learn it in Python
- Vectors, matrices and derivativesLearn the dot product, matrix multiplication and the derivative of squared error, which is all the maths behind least squares and gradient descent.Learn it in Maths
- Fitting and evaluating regressionFit linear regression, ridge and lasso in scikit-learn, compare them with a mean baseline and read residual plots.Learn it in Machine Learning
- Inference and assumptionsUse statsmodels to read coefficients, standard errors and confidence intervals, and learn when those intervals can be trusted.Learn it in Statistics
Where to go after linear regression
If you already work in the field
If you already use regression in research, economics or engineering, you know the model well. Add the machine learning habits: held-out test sets, cross-validation, regularisation and comparison with nonlinear models such as gradient boosting.
If you come from data science or ML
If you mostly use complex models, revisit linear regression as a diagnostic tool: its coefficients, residuals and failures reveal leakage, scaling problems and missing features quickly. Learn the statistical side too, since many stakeholders will ask for confidence intervals.
Roles that do this workData scientistStatisticianQuantitative analystEconometricianMarketing scientist
Tools you will use
| Tool | What you use it for |
|---|---|
| scikit-learn | Fit ordinary least squares, ridge and lasso models inside a preprocessing pipeline. |
| statsmodels | Report coefficients with standard errors, confidence intervals and diagnostic tests. |
| NumPy | Solve least squares directly to see the matrix algebra behind the fit. |
| Matplotlib | Plot residuals against predictions to spot curvature and unequal variance. |
Public data to practise on
- Wine Quality (UCI): Physicochemical measurements of red and white wines with a sensory quality score to predict.
- Auto MPG (UCI): Car attributes such as weight, horsepower and cylinders, for predicting fuel consumption.
Your first project
Download the Wine Quality dataset, predict the quality score with ridge regression, and compare it with a baseline that always predicts the training mean using mean absolute error on a held-out test set.
Common questions
Is linear regression machine learning?
Yes, when it is fitted to data to make predictions on new examples and judged on held-out data. The same model is also a classical statistics tool, where the focus is usually on interpreting coefficients.
What is the difference between linear regression and logistic regression?
Linear regression predicts a number. Logistic regression predicts the probability of a class by passing a linear score through the logistic function, and is used for classification.
When should I use ridge or lasso instead of plain linear regression?
Use ridge when you have many correlated features, and lasso when you want the model to set some coefficients to exactly zero. Choose the penalty strength with cross-validation.
Does linear regression need normally distributed data?
Not for making predictions. The normality assumption concerns the errors and matters mainly for the standard errors and intervals used in classical inference.
Courses for these skills
- Practical Machine Learning in PythonUnderstand boosting, regularisation and model comparison.51 published lessons · more lessons in production
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnUnderstand table columns, array shapes and row alignment.23 published lessons · all planned lessons open
- Python for AI and Machine LearningRead estimator inputs, outputs and errors in Python.33 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.