The Estimator Pattern

Here are six used cars. Their ages are 1, 2, 4, 6, 8 and 9 years. Their prices, in thousands of dollars, are 23.2, 19.6, 15.2, 11.2, 7.6 and 7.2. The older the car, the lower its price.

Store them as two lists, one for the ages and one for the prices:

ages = [1, 2, 4, 6, 8, 9]
prices = [23.2, 19.6, 15.2, 11.2, 7.6, 7.2]
len(ages), len(prices)

(6, 6)

Six ages and six prices, one of each per car.

Three steps

Every prediction model in scikit-learn is used in the same three steps:

1. First, you create the model, with any settings it needs.
2. Then you call its fitKey termThe method that learns from examples. It stores what it learned in attributes ending in _, and returns self. method with examples, which here are the ages and their prices. That is where the model learns.
3. And then you call its predictKey termThe method that gives answers for new inputs, using what fit learned. method with new ages, and it returns a price for each one.

scikit-learn's name for an object that is used in these three steps is an estimatorKey termAn object that learns from data in fit and answers in predict. Every scikit-learn model is one., and that is the word this lesson uses too.

So what does a class need so it can be used in those three steps? And what goes into each method? You will answer both questions by writing two models of your own.

Start with the simplest model there is. It does not use the age at all. It just predicts the average price for every car.

It is called MeanPredictor, and it has two methods. fit takes the ages, called X, and the prices, called y, and computes the mean of the prices. It stores that as self.mean_, and then returns self. And predict takes a list of ages, and returns the stored mean once for each of them:

class MeanPredictor:
    def fit(self, X, y):
        self.mean_ = sum(y) / len(y)
        return self

    def predict(self, X):
        return [self.mean_ for x in X]

The value that fit stores on the object is the model's learned state: what it worked out from the examples, kept so that predict can use it later.

Fitting the model

Create a MeanPredictor and fit it on the ages and prices. fit returns the model itself, so model.fit(ages, prices) is model gives True:

model = MeanPredictor()
model.fit(ages, prices) is model

True

And model.mean_ is the average price:

model.mean_

14.0

Predicting

Chaining fit and predict

Because fit returns self, you can chain the calls on one line. Create the model, fit it, and predict for a five-year-old car:

MeanPredictor().fit(ages, prices).predict([5])

[14.0]

MeanPredictor() creates the object, .fit(ages, prices) returns that same object, and .predict([5]) is called on it.

Now, why does mean_ end in an underscore?

In scikit-learn, and in any class written in its style, an attribute that ends in an underscore was learned from data, in fit. So vars of the fitted model shows one attribute, mean_, and it was fit that put it there:

vars(model)

{'mean_': 14.0}

Predicting before fitting

That convention also makes a common mistake easy to read. Call predict on a new MeanPredictor, before it has been fitted:

MeanPredictor().predict([3])

AttributeError: 'MeanPredictor' object has no attribute 'mean_'

Python raises an AttributeError, because fit never ran, so self.mean_ was never created. The underscore shows you that the missing value is one that fit was supposed to learn.

scikit-learn gives a clearer message for the same mistake. It raises a NotFittedError, which says the estimator is not fitted yet, and to call fit before using it. You will see it once scikit-learn is in the project, later in this lesson.

So far

Here is what you have so far:

• An estimator learns in fit, and stores what it learned in attributes that end in an underscore.
• fit returns the estimator itself.
• And predict uses the stored values to give an answer for each input.

The mean predictor ignores the age. A model that does use it is called k nearest neighbors, and here is the idea. To predict the price of a car, you find the cars in the data whose ages are nearest to its age, and take the average of their prices. k is how many of those nearest cars you use.

So k is a setting of the model, and it is chosen before the model is fitted on any data. A setting like that is called a hyperparameterKey termA setting of a model chosen before training, not learned in fit: k in KNNRegressor(k=2)..

KNNRegressor

KNNRegressor takes k in __init__, and stores it as self.k, with no underscore, because it was not learned:

class KNNRegressor:
    def __init__(self, k=3):
        self.k = k

    def fit(self, X, y):
        self.X_ = list(X)
        self.y_ = list(y)
        return self

    def predict(self, X):
        predictions = []
        for x in X:
            pairs = sorted(zip(self.X_, self.y_), key=lambda pair: abs(pair[0] - x))
            nearest = pairs[:self.k]
            predictions.append(round(sum(price for age, price in nearest) / self.k, 2))
        return predictions

vars of a new KNNRegressor with k equal to 2 shows only k:

vars(KNNRegressor(k=2))

{'k': 2}

What fit and predict do

Its fit stores copies of the ages and the prices, as X_ and y_, and returns self. That is everything this model learns: the data itself.

It is predict that does the work. For each age it is given, it sorts the stored cars by how far their age is from that age, using sorted with a key, the way you did with lambdas. Then it keeps the first k of them, and averages their prices.

One prediction by hand

The model agrees

And that is exactly what the model gives. KNNRegressor with k equal to 2, fitted on the cars, predicts 17.4 for age 3, 13.2 for age 5, and 9.4 for age 7:

knn = KNNRegressor(k=2).fit(ages, prices)
knn.predict([3, 5, 7])

[17.4, 13.2, 9.4]

vars of the fitted model now shows k, and also X_ and y_, holding the six ages and the six prices:

vars(knn)

{'k': 2, 'X_': [1, 2, 4, 6, 8, 9], 'y_': [23.2, 19.6, 15.2, 11.2, 7.6, 7.2]}

Changing k changes the model. With k equal to 3, the predictions at ages 3 and 7 are 19.33 and 8.67, because each one now averages three prices:

KNNRegressor(k=3).fit(ages, prices).predict([3, 7])

[19.33, 8.67]

k equal to six

A straight line

Machine learning also fits a straight line to these same cars: 24 minus 2 times the age. At ages 3, 5 and 7, that gives 18, 14 and 10:

[24 - 2 * age for age in [3, 5, 7]]

[18, 14, 10]

That is the next kind of model you will meet, and it is used in exactly the same three steps.

So how does your model compare with scikit-learn's own? Add scikit-learn to the project the way you added packages when you set up an environment:

uv add scikit-learn

scikit-learn's k nearest neighbors model for prices is called KNeighborsRegressor, and its k is a parameter called n_neighbors. Create one with n_neighbors equal to 2, fit it, and predict at ages 3, 5 and 7, rounding each prediction to two decimal places the same way KNNRegressor does:

from sklearn.neighbors import KNeighborsRegressor
X = [[age] for age in ages]
sk = KNeighborsRegressor(n_neighbors=2).fit(X, prices)
[round(float(p), 2) for p in sk.predict([[3], [5], [7]])]

[17.4, 13.2, 9.4]

You get 17.4, 13.2 and 9.4, the same numbers as yours. And with n_neighbors equal to 3, at ages 3 and 7, it gives 19.33 and 8.67, matching yours again:

sk3 = KNeighborsRegressor(n_neighbors=3).fit(X, prices)
[round(float(p), 2) for p in sk3.predict([[3], [7]])]

[19.33, 8.67]

X is a list of rows

There is one difference in how you call it. scikit-learn's X is a list of rows, with one list per car, so the ages are written as [1], [2], and so on:

X

[[1], [2], [4], [6], [8], [9]]

That is because a model can use several features for each car, like its age and its mileage, and each row holds all of them. A feature is one input value the model uses for each example.

Learned attributes in scikit-learn

Its learned attributes end in an underscore too. Here are the attributes fit added whose names end in an underscore:

sorted(name for name in vars(sk) if name.endswith("_") and not name.startswith("_"))

['effective_metric_',
 'effective_metric_params_',
 'n_features_in_',
 'n_samples_fit_']

They include n_features_in_ and n_samples_fit_. Here, one feature per car, and six cars:

sk.n_features_in_, sk.n_samples_fit_

(1, 6)

DummyRegressor and NotFittedError

scikit-learn even has its own mean predictor, called DummyRegressor, and it predicts 14 for a three-year-old car:

from sklearn.dummy import DummyRegressor
DummyRegressor().fit(X, prices).predict([[3]])

array([14.])

scikit-learn returns its predictions as a NumPy array rather than a list, which is why the result shows array([14.]).

And here is the clearer message from earlier. Calling predict before fit raises a NotFittedError:

KNeighborsRegressor(n_neighbors=2).predict([[3]])

NotFittedError: This KNeighborsRegressor instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

There is one mistake in fit that is easy to make, and KNNRegressor avoids it on purpose. Its fit stores list(X), which is a copy.

Here is a version that stores X itself instead. It inherits everything else from KNNRegressor and replaces only fit:

class KNNReference(KNNRegressor):
    def fit(self, X, y):
        self.X_ = X
        self.y_ = y
        return self

Storing the list itself

Refitting replaces

Two more things to keep in mind. First, calling fit again replaces what the model learned, and does not add to it. So a model fitted on two cars, and then fitted on the six, has only the six ages, and predicts 17.4:

knn = KNNRegressor(k=2).fit([1, 2], [10.0, 20.0])
knn.fit(ages, prices)
knn.X_, knn.predict([3])

([1, 2, 4, 6, 8, 9], [17.4])

And second, anything learned belongs in fit, not in __init__, because __init__ runs before fit has been called with any data.

So, back to the question. What does a class need so it can be used in scikit-learn's three steps? You wrote two estimators to find out.

You create an estimator with its hyperparameters, which __init__ stores, and which are never learned. fit learns from the examples, stores what it learned in attributes that end in an underscore, and returns the estimator itself, which is why you can chain fit and predict. And predict uses what fit stored to give an answer for each new input.

The mean predictor is the simplest estimator there is, and k nearest neighbors averages the prices of the k nearest cars, which gives the mean predictor again when k is every car. Your numbers matched scikit-learn's on these examples because both models average the targets of the nearest neighbors, and both use the same three calling steps. And whatever fit keeps, it should store as its own copy.

Your challenge

Here is a final challenge for this module. Write a MedianPredictor, with the same methods as the MeanPredictor.

1. Its fit should store the median price as median_, and return self.
2. Its predict should give that median for every input.
3. On the six cars, median_ should be 13.2, halfway between the two middle prices, 15.2 and 11.2.

The end of the module

That is the end of this module. You started with a class that holds a review's data and the methods that use it, and saw that every value in Python is an object of some class. Then a dataclass generated the lines you used to write by hand, from a list of fields. And now you have written two models with the same methods as every scikit-learn model, so every scikit-learn model you use from here on will work the way these two do.

But look at what your k nearest neighbors model does when it predicts. For every new car, it computes the distance to every stored car, and then sorts them all. With six cars, that is quick. But with six million cars, the time that takes matters a lot. Working out how that time grows, before you run anything, is what the next module explores.