Data and modeling
Machine learning for outlier detection
Outlier detection uses machine learning to score how unusual each record is, so people review the right ones first. Common uses include flagging suspicious card transactions, spotting faults in sensor or process data, and catching unusual network traffic.
How machine learning is used for outlier detection
Flagging fraudulent payments
Payment processors score each transaction for fraud risk from features such as amount, device, location and past behaviour, and block or review the riskiest. Supervised models learn from confirmed fraud, while unsupervised scores catch new patterns with no labels yet.
Real exampleStripe RadarStripe describes Radar as fraud detection built on AI and machine learning to detect and prevent fraud on payments.
Monitoring business and system metrics
Metrics such as sign-ups, error counts or traffic have trends and weekly cycles, so a fixed threshold either misses problems or raises false alarms. Seasonal decomposition and robust statistics flag points that are unusual for that time.
Real exampleAnomalyDetection (Twitter, open source)Twitter released an R package that detects anomalies in time series in the presence of seasonality and an underlying trend.
Scoring unusual records without labels
Tree-based methods such as Isolation Forest and Random Cut Forest score how easily a record is separated from the rest, with no labels needed. They work on mixed tabular data and scale to large streams.
Real exampleRandom Cut Forest in Amazon SageMakerAWS documents Random Cut Forest as an unsupervised algorithm for detecting anomalous data points within a dataset.
Detecting faults in sensor data
Engineers model what a healthy signal looks like and flag readings that deviate from the prediction. Forecasting-based detectors are common for equipment, vehicles and spacecraft telemetry.
Real exampleDetecting Spacecraft Anomalies Using LSTMs (NASA JPL, KDD 2018)The paper demonstrates LSTM networks with nonparametric dynamic thresholding on expert-labelled telemetry anomaly data from the SMAP satellite and the Curiosity rover.
Can you learn machine learning for outlier detection?
Yes. Outlier detection builds on ordinary statistics and a few machine learning methods, so it is a good early specialism. Learn Python and data handling, then distributions and robust statistics, then unsupervised and supervised detectors and how to evaluate them with rare labels.
- Python and pandasLoad event or transaction data, build per-entity features and plot distributions to see what normal looks like.Learn it in Data Tools
- Statistics of the normal rangeLearn distributions, z-scores, medians and quantiles, which are the basis of simple and robust outlier rules.Learn it in Statistics
- Anomaly detection methodsApply Isolation Forest, local outlier factor and one-class models, and compare them with simple statistical rules.Learn it in Machine Learning
- Evaluating with rare labelsMeasure performance with precision at the top of the ranking and average precision, since accuracy is meaningless when outliers are rare.Learn it in Machine Learning
Moving into anomaly detection work
If you already work in the field
Fraud analysts, site reliability engineers and quality engineers already know what a real incident looks like, which is the hardest knowledge to acquire. Add Python and two or three detectors, then test them on past incidents your team has already investigated.
If you come from data science or ML
If you come from general ML, learn the domain's costs: how many alerts reviewers can handle, what a missed fraud or fault costs, and how labels arrive late or never. Sit with the team that triages alerts before tuning any model.
Roles that do this workFraud data scientistRisk analystMachine learning engineerSite reliability engineerSecurity data scientist
Tools you will use
| Tool | What you use it for |
|---|---|
| scikit-learn outlier detection | Fit Isolation Forest, Local Outlier Factor and One-Class SVM detectors. |
| PyOD | Compare dozens of outlier detectors through one consistent interface. |
| River | Score anomalies on streaming data one observation at a time. |
| pandas | Prepare features and review the records a detector ranks highest. |
Public data to practise on
- Credit Card Fraud Detection (ULB, on Kaggle): Anonymised card transactions labelled as fraud or genuine, for ranking rare cases; needs a free Kaggle account.
- Numenta Anomaly Benchmark (NAB): Labelled streaming time series from servers, traffic and sensors, for testing real-time anomaly detectors.
Your first project
Download the Credit Card Fraud Detection dataset, fit an Isolation Forest without using the labels, and compare its ranking with a transaction-amount z-score baseline using average precision against the fraud labels.
Common questions
What is the difference between an outlier and an anomaly?
The terms are often used interchangeably. Outlier usually means a data point far from the rest, while anomaly usually means something that matters, such as fraud or a fault, which may or may not look extreme.
Is outlier detection supervised or unsupervised?
It can be either. Unsupervised methods score how unusual each record is without labels, and supervised models learn from confirmed cases when enough exist. Many production systems combine both.
Should I remove outliers before training a model?
Only when they are errors, such as impossible values or duplicated records. Genuine extreme cases are often the ones that matter, so investigate before deleting and consider robust models instead.
Courses for these skills
- Practical Machine Learning in PythonStudy anomaly detection and rare-event classification.51 published lessons · more lessons in production
- Statistics and Probability for Machine LearningUnderstand tails, robust summaries and base-rate effects.41 published lessons · all planned lessons open
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnPreserve capture groups and clean traffic-feature tables.23 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.