Model evaluation: Machine Learning worked examples

Choose metrics, splits and diagnostic experiments that support valid conclusions.

• Cross-validation for groups and time: check what a split holds out

  Audit random, grouped and chronological folds on the same small table, and choose a split that matches the prediction you intend to make.

  tutorial · 4 min read

• ROC AUC vs PR AUC: inspect what changes with class balance

  Compare ROC AUC, average precision and trapezoidal PR area on an explicit ranking, then change prevalence while preserving class-conditional scores.

  comparison · 4 min read

• Data leakage in machine learning: examples and prevention

  Spot target leakage, preprocessing leakage and train-test overlap. See concrete examples and a Python pipeline that keeps fitted transformations separate.

  troubleshooting · 6 min read

• Overfitting vs underfitting: examples and how to fix each

  Tell overfitting from underfitting using training and validation error. Work through a demand example, diagnose the cause and choose a sensible next experiment.

  troubleshooting · 6 min read