Finance and economics
Machine learning for algorithmic trading
In algorithmic trading, machine learning predicts short-horizon price moves from order-book and trade data, filters or sizes signals from simpler rules, and models execution costs. Sound research reports the prediction, the trading rule and the cost assumptions separately, tested on later data.
How machine learning is used in algorithmic trading
Predicting short-term price moves from the order book
Models read recent snapshots of the limit order book, the queue of resting buy and sell orders, and predict whether the mid-price will rise, fall or stay flat over the next few events. Convolutional and recurrent networks can learn features directly from the book.
Real exampleDeepLOB (University of Oxford)Combined convolutional layers and LSTM modules to predict price movements from limit order book data, tested on a public benchmark and on a year of London Stock Exchange quotes.
Learning from individual order messages
Market-by-order feeds record every order placed, changed or cancelled, which is more detailed than book snapshots. Sequence models can use this feed as an extra source of short-horizon information.
Real exampleDeep learning for market by order data (Zhang, Lim and Zohren)Showed that market-by-order data, the feed of individual trade instructions, can be used to forecast high-frequency price movements as a source of information alongside order book snapshots.
Executing large orders
Execution algorithms split a large order over time to limit market impact. Reinforcement learning can learn when and how much to trade from the state of the order book, and is compared with simple schedules such as TWAP.
Real exampleDouble deep Q-learning for optimal execution (Ning, Lin and Jaimungal)Trained a Double Deep Q-Network on limit order book features to choose trading actions when executing an order, as a model-free alternative to stochastic control.
Benchmarking on shared data
Order book data is expensive and hard to share, so public benchmarks let researchers compare models on the same inputs and splits before testing on their own data.
Real exampleFI-2010 limit order book benchmark (Ntakaris et al.)Published the first public benchmark of high-frequency limit order book data for mid-price prediction, from five NASDAQ Nordic stocks over ten days, with an anchored cross-validation protocol.
Guarding against backtest overfitting
Trying many strategy variants on the same history makes it likely that the best one fits noise. Researchers estimate how likely a chosen strategy is to be overfit before trusting its backtest.
Real exampleThe probability of backtest overfitting (Bailey, Borwein, López de Prado and Zhu)Proposes a framework to estimate the probability that a strategy selected by backtesting is overfit, noting that standard hold-out methods are unreliable for investment backtests.
Can you learn machine learning for algorithmic trading?
Yes, as a research skill. Algorithmic trading combines programming, statistics, market microstructure and machine learning, and the sound starting point is reproducible research on public data rather than live trading. Learn Python and statistics first, then supervised learning, then order book data and backtesting.
- Python for tick and bar dataProcess trades and quotes with pandas or Polars, resample by time or events, and keep every timestamp in one time zone.Learn it in Data Tools
- Statistics for noisy returnsHeavy tails, autocorrelation, multiple testing and confidence intervals, which decide whether a backtest result means anything.Learn it in Statistics
- Supervised learning with time-aware validationClassification and regression on lagged features, with walk-forward splits and a gap so training never overlaps the test period.Learn it in Machine Learning
- Deep learning for order book sequencesConvolutional and recurrent networks for order book inputs, trained and compared on a public benchmark such as FI-2010.Learn it in Deep Learning
- Market microstructure and costsHow limit order books, spreads, fees and market impact work, so every backtest includes realistic costs.
Moving into machine learning for algorithmic trading
If you already work in the field
Traders and execution specialists already understand order types, costs and how strategies fail in live markets. Add Python, statistics and time-aware validation, then reproduce a published order book study such as DeepLOB on public data before testing your own features.
If you come from data science or ML
ML engineers should study market microstructure, transaction costs and why financial data breaks the independence assumptions behind random cross-validation. A market microstructure textbook and the literature on backtest overfitting are the core reading.
Roles that do this workQuantitative researcherQuantitative developerAlgorithmic traderExecution quantMachine learning researcher, trading
Tools you will use
| Tool | What you use it for |
|---|---|
| pandas | Resample tick and bar data, build lagged features and align timestamps. |
| scikit-learn | Fit baseline classifiers and run time-series cross-validation splits. |
| LightGBM | Train fast gradient-boosted models on many engineered market features. |
| vectorbt | Run vectorised backtests of signal-based rules with fees across many parameter settings. |
| NautilusTrader | Simulate event-driven strategies with order types and fills, using the same code path for research and live. |
Public data to practise on
- Binance public market data: Free bulk downloads of Binance klines, trades and aggregated trades, suited to building and testing short-horizon signals.
- LOBSTER sample files: Free sample message and order-book files reconstructed from Nasdaq data, for studying order flow and price impact.
Your first project
Download one month of Binance 1-minute BTCUSDT klines, predict the sign of the next 5-minute return with logistic regression on lagged returns and volume, and compare it with an always-up baseline using log loss on the following month.
Common questions
Does machine learning work for algorithmic trading?
Firms use it for short-horizon prediction, execution and risk, but results depend heavily on data quality, costs and latency, and many published effects shrink after costs. This page covers research methods for learning and is not trading or investment advice.
What is a backtest and why can it mislead?
A backtest simulates how a rule would have behaved on historical data. Testing many variations on the same history makes it likely that the best one is fitted to noise, which is why researchers hold out later data and account for the number of trials.
Which programming language is used for algorithmic trading?
Python is standard for research and prototyping. Latency-sensitive production systems are often written in C++, Java or Rust, with specialised time-series databases for tick data.
Can I practise without risking money?
Yes. Public limit order book datasets, market simulators such as ABIDES and paper trading accounts let you test ideas without capital. Treat any result as a research finding, not evidence that a strategy will make money.
Courses for these skills
- Deep Learning with PyTorchUnderstand learned representations of sequential market observations.Syllabus available · lessons open as produced
- Practical Machine Learning in PythonSeparate predictive evaluation from a decision policy.51 published lessons · more lessons in production
- Statistics and Probability for Machine LearningUnderstand dependence, uncertainty and baseline comparisons.41 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.