Data and modeling
Machine learning XAI
Explainable AI (XAI) shows why a model makes its predictions. Teams use it to find which features drive a credit or risk score, explain an individual decision to a customer or reviewer, and catch models relying on leaked or biased inputs.
How explainable AI (XAI) is used in machine learning
Attributing a prediction to its features
Feature attribution splits a single prediction into contributions from each input, such as income or payment history in a credit score. Summed over many predictions, it shows which features the model relies on overall.
Real exampleSHAP (Lundberg and Lee, NeurIPS 2017)The paper presents SHAP, a unified framework that assigns each feature an importance value for a particular prediction.
Explaining any model locally
Local surrogate methods fit a simple model around one prediction to approximate how the complex model behaves nearby. They work with any classifier and help reviewers decide whether to trust it.
Real exampleLIME (Ribeiro, Singh and Guestrin, KDD 2016)LIME explains the predictions of any classifier by learning an interpretable model locally around the prediction.
Explaining deep networks
Gradient-based attribution highlights which pixels, words or inputs most affected a neural network's output. It is used to debug image and text models.
Real exampleIntegrated Gradients (Sundararajan, Taly and Yan, 2017)The paper proposes Integrated Gradients, an attribution method that needs no change to the network and only a few calls to the standard gradient operator.
Explaining decisions to the people affected
Where AI assists decisions about people, organisations need to explain the process and outcome in terms those people understand. Technical attributions are one input to that explanation.
Real exampleExplaining decisions made with AI (ICO and The Alan Turing Institute)The co-badged guidance gives organisations practical advice on explaining processes, services and decisions delivered or assisted by AI to the individuals affected.
Can you learn explainable AI?
Yes. Explainability builds on ordinary model training, so learn to build and evaluate models first. Then learn global methods such as permutation importance, local methods such as SHAP and LIME, and inherently interpretable models.
- Python and data toolsPrepare data and plot results, since most explanations are read as charts.Learn it in Data Tools
- Models to explainTrain linear models, trees and gradient boosting and evaluate them on held-out data.Learn it in Machine Learning
- Explaining modelsUse permutation importance, partial dependence, SHAP and LIME, and learn where each can mislead.Learn it in Machine Learning
- Explaining deep networksApply gradient-based attribution with Captum to an image or text model.Learn it in Deep Learning
Moving into explainable AI work
If you already work in the field
Risk, compliance and audit professionals already know what an explanation must cover for regulators and customers. Learn enough modelling to run SHAP and permutation importance yourself, and review one model's explanations against your policies.
If you come from data science or ML
If you build models, learn what different audiences need from an explanation and the limits of attribution methods. Read the ICO and Alan Turing Institute guidance and practise explaining one model to a non-technical reviewer.
Roles that do this workMachine learning engineerModel risk analystResponsible AI specialistData scientistModel validation analyst
Tools you will use
| Tool | What you use it for |
|---|---|
| SHAP | Attribute each prediction to its features and summarise global importance. |
| scikit-learn permutation importance | Measure how much held-out performance drops when each feature is shuffled. |
| InterpretML | Train glass-box explainable boosting machines and compare them with black-box explanations. |
| Captum | Attribute neural-network predictions to input features, pixels or tokens in PyTorch. |
| LIME | Explain single predictions by fitting a simple local model around them. |
Public data to practise on
- Statlog German Credit Data (UCI): Loan applicants described by account, employment and personal attributes, labelled good or bad credit risk.
- ProPublica COMPAS analysis data: Criminal risk scores with later recorded outcomes, for studying explanations alongside fairness.
Your first project
Download the Statlog German Credit dataset, fit gradient boosting and compare it with logistic regression using ROC AUC, then compare the boosted model's permutation-importance and SHAP feature rankings.
Common questions
What is the difference between interpretability and explainability?
Interpretability usually means a model is understandable by design, such as a short decision tree or linear model. Explainability usually means methods applied after training to describe a complex model's behaviour.
What is the difference between SHAP and LIME?
Both explain individual predictions. SHAP is based on Shapley values and has consistency guarantees, while LIME fits a simple local model around the prediction and is faster but can vary between runs.
Does explainable AI show causation?
No. Attributions describe how the model uses its inputs, not how the world works. A feature can matter to the model because it is correlated with the true cause.
Is explainable AI required by law?
Requirements vary by country and sector, and this is not legal advice. In the UK, the ICO's guidance with The Alan Turing Institute covers explaining AI-assisted decisions under data protection law.
Courses for these skills
- Practical Machine Learning in PythonUnderstand predictive nuisance models and overfitting.51 published lessons · more lessons in production
- Statistics and Probability for Machine LearningBuild probability, inference and experimental-design foundations.41 published lessons · all planned lessons open
- Generative AI: LLMs, RAG and AgentsStudy retrieval-augmented generation and evaluating grounded outputs.Syllabus available · lessons open as produced
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnPreserve document identifiers, revisions and access metadata.23 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.