Everything else is ready: study the parts, the key terms and the quiz below.
Long Tails and Log Transforms
Recognise long-tailed data and test whether a simple transform helps the model or the interpretation.
In this lesson6 parts
- 01Decide what may enter a logarithm
- 02Derive the tail from positive invoices
- 03Follow each target through its inverse
- 04Seal time before fitting either model
- 05Let the objective choose the comparison
- 06Reproduce and defend one decision
Key terms
The words this lesson introduces, each in one line. The module’s glossary collects them all.
- invoice value
- Sum of included line quantities times their unit prices for one invoice.
- right skew
- Observations extend farther toward large values than small values.
- long right tail
- Relatively few observations occupy a wide high-value range.
- log transform
- Applying a logarithm to each declared target value before fitting.
- transformed scale
- Units of log(1 + pounds), not pounds.
- inverse transform
- Operation that maps a transformed value back to the original scale; here
expm1. - ordinary least squares
- Coefficients chosen to minimize summed squared deviations on the fitted target scale.
- held-out error
- Prediction error measured on rows withheld from fitting.
- MAE
- Mean absolute error, average absolute prediction miss in the original unit.
- RMSE
- Square root of mean squared prediction error, in the original unit.
Quiz 5 questions
Your first pick on each question is the one that counts, and a right one earns a coin. Getting one wrong here is how the lesson sticks.
Practice
Problems to solve in your own notebook. Each states the problem, not the steps: working out the steps is the exercise. Level A applies the lesson, B combines it with earlier ones, C stretches it.
The self-checking notebook for this lesson is Long tails, log transforms and the multivariate normal.
Common mistakes
What you will see when it goes wrong, why it happens, and the fix.
Where it’s used
Where this lesson’s ideas turn up in real work.