ML 4.3Lesson outlineLesson 3 of 9 in Module 4
Overfitting, Underfitting and the Test Set
Know why a score on the training data is not evidence, and read the train/test gap as a diagnosis.
Video in production.
Lesson outline: Overfitting, Underfitting and the Test Set
In this lesson9 parts
- 01One hundred percent
- 02The sealed tumors
- 03Choosing K by training accuracy
- 04Three sets, three jobs
- 05Choosing K on the validation set
- 06Capacity, bias and variance
- 07Why the test set is used once
- 08Code
- 09Recap, your challenge, and what comes next
Builds on
This lesson stands on its own. These go deeper into what it uses.
Key terms: Overfitting, Underfitting and the Test Set
The words this lesson introduces, each in one line. The module’s glossary collects them all, and the AI glossary has every term across the courses.
- training set
- The rows the model is fitted on.
- validation set
- Rows held out from fitting, used to choose a hyperparameter.
- test set
- Rows used once, to report the chosen model.
- stratified split
- A split that keeps each class's share the same in every part.
- stratified sample (reminder, Stats 4.1)
- A sample drawn within groups, in proportion.
- pipeline (reminder, Tools 4.3)
- Preprocessing and a model fitted together, so each step learns from the fitting rows only.
- overfitting
- Fitting the training rows much better than new rows.
- underfitting
- Fitting both poorly.
- variance
- How much the model changes with the training rows.
- bias
- Error from a model too simple for the pattern.
- capacity
- How closely a model can follow its training rows.
Quiz: Overfitting, Underfitting and the Test Set
Your first pick on each question is the one that counts. Getting one wrong here is how the lesson sticks.