Data and modeling
Machine learning for survival analysis
Survival analysis predicts how long until an event happens, such as death, relapse, machine failure or customer churn, while correctly using records whose event has not happened yet. Machine learning adds survival forests, boosted models and neural networks to the classic Kaplan-Meier and Cox methods.
How machine learning is used in survival analysis
Nonlinear survival models with forests
Random survival forests grow trees that split on how event times differ, then combine them into a survival curve for each record. They capture interactions without specifying them by hand and handle censored records correctly.
Real exampleRandom Survival Forests (Ishwaran et al., 2008)The paper introduces random survival forests, an ensemble tree method for analysing right-censored survival data.
Deep learning extensions of the Cox model
Neural networks can replace the linear part of a Cox proportional hazards model to learn complex relationships between features and risk. In medical research these models are studied for risk stratification, not as clinical tools on their own.
Real exampleDeepSurv (Katzman et al., 2018)DeepSurv is a Cox proportional hazards deep neural network, studied in research as a personalised treatment recommender.
Competing risks
Sometimes several different events can end follow-up, such as different causes of failure. Competing-risks models estimate the chance of each event over time instead of treating the others as ordinary censoring.
Real exampleDeepHit (Lee et al., AAAI 2018)DeepHit is a deep learning approach to survival analysis that handles competing risks by learning the distribution of event times directly.
Predicting equipment failure
Engineers model time until a component fails from its age and health readings, so they can replace parts before failure. Drives that are still working at the end of the data are censored observations, which is exactly what survival models handle.
Real exampleBackblaze hard drive test dataBackblaze publishes a daily snapshot of each operational drive in its data centres, including SMART statistics, as open data on drive reliability.
Can you learn machine learning for survival analysis?
Yes, if you are comfortable with probability and regression. Learn the classical survival tools first, Kaplan-Meier curves, the Cox model and censoring, then machine learning extensions in scikit-survival and pycox.
- Python and data handlingBuild tables with one row per subject, a duration and an event indicator, which is the format every survival library expects.Learn it in Data Tools
- Probability and estimationLearn distributions, hazard ideas and estimation so survival curves and hazard ratios make sense.Learn it in Statistics
- Classical survival analysisFit Kaplan-Meier curves and Cox models in lifelines and learn why dropping censored records biases results.
- Machine learning survival modelsTrain random survival forests and gradient-boosted survival models, and compare them with Cox using the concordance index on held-out data.Learn it in Machine Learning
Moving into survival and time-to-event modelling
If you already work in the field
Biostatisticians, epidemiologists and reliability engineers already know censoring and the Cox model. Add cross-validated ML models such as survival forests, and compare them with your usual Cox specification on held-out data.
If you come from data science or ML
If you come from ML, learn why time-to-event data cannot be treated as plain regression or classification, and how censoring and competing risks change both models and metrics. Work through a lifelines or scikit-survival tutorial on a public dataset.
Roles that do this workBiostatisticianEpidemiologistReliability engineerData scientist (churn and retention)Actuarial analyst
Tools you will use
| Tool | What you use it for |
|---|---|
| lifelines | Fit Kaplan-Meier curves and Cox models, and check proportional-hazards assumptions. |
| scikit-survival | Train random survival forests and boosted survival models with a scikit-learn interface. |
| pycox | Train neural-network survival models in PyTorch. |
| pandas | Build the duration and event columns from dated records. |
Public data to practise on
- Heart Failure Clinical Records (UCI): Clinical features of heart-failure patients with follow-up time and whether death occurred, for survival modelling.
- Cirrhosis Patient Survival Prediction (UCI): Mayo Clinic trial records of patients with primary biliary cirrhosis, with follow-up time and survival status.
Your first project
Download the Heart Failure Clinical Records dataset, fit a Cox model on the clinical features, and compare it with an age-only Cox model using Harrell's concordance index on held-out patients.
Common questions
What is censoring in survival analysis?
A record is censored when the event has not happened by the end of observation, for example a patient still well at the last follow-up. You know the event time is at least that long, and survival methods use that partial information.
Can I use ordinary classification for time-to-event data?
Only for a fixed horizon, such as an event within one year, and only if records followed for less than a year are handled carefully. Survival models use all records and give a full curve over time.
What is the concordance index?
The C-index measures how often a model ranks pairs of subjects correctly, giving higher risk to the one whose event happens first. 0.5 is random ordering and 1.0 is perfect.
Is survival analysis only for medicine?
No. The same methods model customer churn, time until a machine fails, time to loan default and time to hire.
Courses for these skills
- Statistics and Probability for Machine LearningDistinguish repeated observations from independent evidence.41 published lessons · all planned lessons open
- Practical Machine Learning in PythonChoose an evaluation design before tuning a model.51 published lessons · more lessons in production
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.