Regression models in machine learning: 8 types compared in Python

A regression model in machine learning predicts a number, such as a price, a temperature or a delivery time, from input features. The main types are linear regression and its regularised versions (ridge, lasso and elastic net), polynomial regression for curves, k-nearest neighbours, decision trees, and tree ensembles such as random forests and gradient boosting. They differ in how much shape they can learn, how much data they need, and how easy they are to explain.

There is no single best regression model. The reliable way to choose is to fit a few on the same data, score them on rows they did not see, and compare them with a baseline that always predicts the average. Below, eight models are compared that way in about 50 lines of Python with scikit-learn.

What makes a problem regression

Regression means the answer is a number on a scale, so being off by 1 is better than being off by 10. If the answer is a category (spam or not, which digit), it is classification instead. The kinds of machine learning lesson shows how to tell from the question being asked.

Logistic regression is the classic source of confusion: despite the name, it is a classification model. It predicts the probability of a class, not a quantity.

The 8 regression models at a glance

Other regression models worth knowing are support vector regression, neural networks, and models with different losses such as Poisson and quantile regression, which predict counts or ranges instead of an average.

Compare them in Python

The data is invented for teaching: 800 used cars with age, mileage and engine size, and price in thousands. Price falls quickly in the first years and then levels off, so the true relationship is a curve. Mileage rises with age, as it does in real cars. A fourth column is pure noise, to see which models ignore it. Every model is trained on 600 cars and scored on the other 200.

import numpy as np
from sklearn.dummy import DummyRegressor
from sklearn.ensemble import HistGradientBoostingRegressor, RandomForestRegressor
from sklearn.linear_model import Lasso, LinearRegression, Ridge
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.tree import DecisionTreeRegressor

# Invented used-car data: price in thousands from age, mileage, engine size
# and a column that has nothing to do with price.
rng = np.random.default_rng(7)
n = 800
age = rng.uniform(0, 15, n)
mileage = np.clip(age * 12 + rng.normal(0, 15, n), 0, None)  # thousand km
engine = rng.choice([1.0, 1.4, 1.6, 2.0, 3.0], n)
noise_column = rng.normal(0, 1, n)
price = (32 * np.exp(-0.12 * age) - 0.03 * mileage + 4 * engine
         + rng.normal(0, 1.5, n))
X = np.column_stack([age, mileage, engine, noise_column])
X_train, X_test, y_train, y_test = train_test_split(X, price, test_size=0.25, random_state=0)

models = {
    "mean baseline": DummyRegressor(),
    "linear": make_pipeline(StandardScaler(), LinearRegression()),
    "ridge": make_pipeline(StandardScaler(), Ridge(alpha=1.0)),
    "lasso": make_pipeline(StandardScaler(), Lasso(alpha=0.1)),
    "polynomial (degree 2)": make_pipeline(PolynomialFeatures(2), StandardScaler(), Ridge(alpha=1.0)),
    "k-nearest neighbours": make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=10)),
    "decision tree": DecisionTreeRegressor(max_depth=5, random_state=0),
    "random forest": RandomForestRegressor(n_estimators=200, random_state=0),
    "gradient boosting": HistGradientBoostingRegressor(random_state=0),
}
print(f"{'model':22} {'MAE':>5} {'RMSE':>5} {'R2':>5}")
for name, model in models.items():
    prediction = model.fit(X_train, y_train).predict(X_test)
    print(f"{name:22} {mean_absolute_error(y_test, prediction):5.2f} "
          f"{root_mean_squared_error(y_test, prediction):5.2f} {r2_score(y_test, prediction):5.2f}")

names = ["age", "mileage", "engine", "noise"]
linear = models["linear"][-1].coef_
lasso = models["lasso"][-1].coef_
print("linear coefficients:", dict(zip(names, np.round(linear, 3).tolist())))
print("lasso coefficients: ", dict(zip(names, np.round(lasso, 3).tolist())))

Checked output:

model                    MAE  RMSE    R2
mean baseline           7.79  9.17 -0.02
linear                  1.68  2.06  0.95
ridge                   1.68  2.06  0.95
lasso                   1.67  2.05  0.95
polynomial (degree 2)   1.20  1.49  0.97
k-nearest neighbours    1.57  1.96  0.95
decision tree           1.68  2.23  0.94
random forest           1.32  1.65  0.97
gradient boosting       1.34  1.68  0.97
linear coefficients: {'age': -7.928, 'mileage': -1.082, 'engine': 2.637, 'noise': -0.001}
lasso coefficients:  {'age': -7.869, 'mileage': -1.042, 'engine': 2.54, 'noise': 0.0}

Read the results

The baseline sets the floor. Predicting the average price for every car has an RMSE of 9.17 (thousand) and an R² of about zero. Every model beats it by a wide margin, so the features carry real information. Always include this line: a model that cannot beat the average has learned nothing useful.

The best possible score is about 1.5. The data was generated with random noise of standard deviation 1.5, which no model can predict. The polynomial model's RMSE of 1.49 is at that limit, so on this data it is as good as any model can be.

Shape matters more than sophistication here. Linear, ridge and lasso all score an RMSE of about 2.05: a flat plane cannot follow a price that drops fast and then levels off. Adding squared terms lets the polynomial model bend, and it wins. Random forest (1.65) and gradient boosting (1.68) also learn the curve, without being told its shape. Repeating the comparison with 5-fold cross-validation gave the same order (RMSE 1.50 polynomial, 1.66 random forest, 1.69 gradient boosting, 2.21 linear), so the ranking does not come from one lucky split.

A single shallow tree is the weakest learner. A decision tree with depth 5 predicts in steps, so nearby cars on either side of a split get quite different prices. Averaging hundreds of trees (random forest) or adding them in sequence (gradient boosting) smooths those steps out.

Lasso removes the useless feature. The linear model gives the noise column a tiny weight (−0.001); lasso sets it to exactly 0. With many features, that built-in selection is lasso's main appeal. Because inputs were standardised, each coefficient is the change in price per standard deviation of that feature, so age (−7.9) matters far more than mileage (−1.1). Age and mileage are strongly related, which makes their separate weights less reliable than their combined effect. The collinearity lesson explains why.

On your own data the order can be completely different. Tree ensembles often do well on tables with many mixed features and interactions; linear models often win when data is small or the relationship really is close to straight. That is why you compare instead of guessing.

How to choose a regression model

1. Start with a mean baseline and linear regression. They take seconds and tell you how much there is to gain.
2. Plot residuals against each feature. A curve in the residuals means a linear model is missing shape. The regression diagnostics lesson shows how to read them.
3. Need to explain every prediction? Use linear, ridge or lasso, adding polynomial or engineered features where the residuals show curves.
4. Need the most accuracy on a table? Try random forest and gradient boosting, and tune them with cross-validation, never on the test set.
5. Pick the metric before the model. MAE treats every error equally, RMSE punishes big misses more, and R² compares with the mean baseline. Evaluating a regression model explains which to report.

Learn each model properly

The Machine Learning course teaches each of these models in order, with the maths and code:

• Simple linear regression and multiple linear regression
• Polynomial regression and model capacity
• Ridge regression and lasso and elastic net
• k-nearest neighbours
• Decision trees, random forests and gradient boosting
• A full regression project in the house prices case study

For a single model worked through in detail, read linear regression in machine learning. To check what you can explain, work through the linear regression and gradient descent interview questions and the matrices interview questions, which cover the algebra behind the linear models. The linear regression cheat sheet is free to read.

Common questions

What is the most common regression model?

Linear regression. It is fast, needs little data and is easy to explain, so it is the usual first model and the baseline that more complex models must beat.

Is logistic regression a regression model?

Not in the machine learning sense. It predicts the probability of a class, so it is used for classification. The name comes from statistics, where it is a regression on the log-odds.

Which regression model is most accurate?

It depends on the data. On this curved used-car example, polynomial regression won, with random forest and gradient boosting close behind. Compare several on held-out data, with a mean baseline, before choosing.

Do I need to scale features for regression?

For linear models with regularisation, k-nearest neighbours and anything trained by gradient descent, yes. Trees and tree ensembles split on one feature at a time, so scaling does not change them. Scaling vs normalization covers which scaler to use.

References

scikit-learn: linear models — Ordinary least squares, ridge, lasso and elastic net.

scikit-learn: ensembles — Random forests and histogram-based gradient boosting.

scikit-learn: regression metrics — MAE, RMSE and R².