Linear regression in machine learning: a worked Python example with a baseline
Learn linear regression with a worked prediction, runnable Python, a held-out test and a baseline. Explain coefficients and check where a line can fail.
Machine learning for linear regression means learning coefficients from examples so a model can predict a numerical target. With one input, the prediction is intercept + slope × input. The model chooses the slope and intercept from training data; you then check its predictions on separate examples.
This tutorial uses a deliberately simple, invented relationship so you can verify every number. You need basic Python and an understanding of a table with input and output columns. If those are new, start with Python projects for beginners.
What a fitted line predicts
Imagine a teaching dataset with an input x and target y:
| x | y | Role in the example |
|---|---|---|
| 1 | 12 | Training |
| 2 | 14 | Training |
| 3 | 16 | Training |
| 4 | 18 | Training |
| 5 | 20 | Training |
| 6 | 22 | Held-out check |
| 7 | 24 | Held-out check |
The constructed rule is y = 10 + 2x. A fitted intercept of 10 and slope of 2 therefore give 22 when x = 6. A residual is the observed value minus the predicted value. Here, a prediction of 22 for an observed 22 has residual zero.
In ordinary least squares, fitting chooses coefficients to reduce the sum of squared training residuals. This objective is separate from the test metric we choose to report. We will report mean absolute error, or MAE, because it reads directly as an average error in target units.
Run linear regression in Python
Use Python 3 in a virtual environment. In a terminal, install scikit-learn, which also installs NumPy:
python -m venv .venv
source .venv/bin/activate
python -m pip install scikit-learn
On Windows, activate with .venv\Scripts\activate instead. Copy the Python block into a .py file and run it with python filename.py, or use a notebook cell. The example was run with Python 3.12.3, NumPy 2.5.4 and scikit-learn 1.9.1. All data is invented and included inline.
import numpy as np
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
# Invented data: y = 10 + 2*x. The last two rows are held out.
X_train = np.array([[1], [2], [3], [4], [5]])
y_train = np.array([12, 14, 16, 18, 20])
X_test = np.array([[6], [7]])
y_test = np.array([22, 24])
baseline = DummyRegressor(strategy="mean").fit(X_train, y_train)
model = LinearRegression().fit(X_train, y_train)
predictions = model.predict(X_test)
print("Slope:", round(float(model.coef_[0]), 2))
print("Intercept:", round(float(model.intercept_), 2))
print("Predictions:", predictions.round(2).tolist())
print("Baseline MAE:", round(mean_absolute_error(y_test, baseline.predict(X_test)), 2))
print("Model MAE:", round(mean_absolute_error(y_test, predictions), 2))
assert np.allclose(predictions, [22, 24])
Expected output:
Slope: 2.0
Intercept: 10.0
Predictions: [22.0, 24.0]
Baseline MAE: 7.0
Model MAE: 0.0
Read the result against a baseline
The baseline learns only the training-target mean: (12 + 14 + 16 + 18 + 20) / 5 = 16. It predicts 16 for both held-out rows. The absolute errors are 6 and 8, so its MAE is 7.
The line predicts 22 and 24. Both match the invented held-out targets, giving MAE zero. The result checks that the code recovered this known straight-line relationship. It is not evidence that a model will predict real measurements perfectly.
Keeping rows apart is useful even in a small demonstration: it distinguishes fitting a line from checking its predictions. A real evaluation needs enough representative data, a split appropriate to the problem and uncertainty about the reported error. If observations arrive in time order, keep future outcomes out of the training process.
Try it yourself: replace the training targets with [12, 15, 15, 19, 20]. Before running the program, predict whether the recovered slope will still be exactly 2. Remove or revise the final assertion because it checks the original noiseless example. Compare the new test MAE with the unchanged mean baseline.
What the coefficient means, and what it does not
The slope says how the fitted prediction changes when the input increases by one unit. It does not establish that changing the input will cause the target to change. The correlation and causation lesson explains why an association can have another explanation.
With several inputs, each coefficient belongs to a particular set of other inputs in the model. Strongly related columns can make individual coefficients unstable. A model can also fit a line where a curve or a different feature representation would be more suitable.
The test inputs 6 and 7 are beyond the training range of 1 to 5. That is extrapolation. It works here because we invented the relationship. With real data, extending a fitted trend beyond the observed range needs evidence, not just a successful call to predict.
Checks before using real data
| Check | Why it matters | A practical action |
|---|---|---|
| Prediction timing | Future information can create a misleading score | List when each input is known |
| Units | A coefficient depends on how inputs are measured | Record units beside column names |
| Residual pattern | A line may miss systematic structure | Plot residuals against predictions |
| Baseline | A complex explanation may add little predictive value | Compare on identical held-out rows |
| Coverage | The test set may miss important groups or ranges | Break error down by relevant subsets |
For more on distinguishing fitting from useful prediction, read overfitting vs underfitting.
Common questions
Is linear regression supervised learning?
Yes. Training examples include both input features and observed numerical targets. The fitted model predicts the target for an input whose outcome is not yet known.
Is logistic regression the same thing?
No. Logistic regression is used for classification, despite its name. This example predicts a number with linear regression. The logistic regression implementation fits a classification probability with binary cross entropy and reports separate held-out errors.
Do I need gradient descent to fit this example?
You do not have to implement it to run scikit-learn's LinearRegression. The squared-error gradient calculation and learning-rate experiment explain iterative fitting. Learning gradient descent helps explain how many models are trained, but this estimator's fitting implementation is a separate detail from the prediction formula.
Learn the model from geometry to code
Start with SchoolWhool's published geometric interpretation of linear regression, then study gradient descent for linear regression. The Maths course supplies vectors, derivatives and least squares.
Use the Machine Learning course syllabus to see the wider sequence and which lessons are available. A free account adds quizzes and saved progress. See Pro and current availability for practice, projects and weekly live sessions with Gaurav.
References
For the estimator's parameters and prediction interface, see the official LinearRegression documentation. For the objective and correlated-feature caveat, see the linear models guide. The numerical dataset and exercise are original.