Finance and economics
Machine learning for realised volatility forecasting
Machine learning forecasts realised volatility from past realised variance, intraday order-book features and implied volatility, and is compared with HAR and GARCH benchmarks. The target is built from high-frequency returns over a fixed window, so its definition comes first.
How machine learning is used to forecast realised volatility
Measuring the target from intraday data
Realised volatility sums squared intraday returns over a day or a shorter window, which turns volatility into a measured series. Once measured, it can be forecast with time-series and machine learning models.
Real exampleModeling and forecasting realized volatility (Andersen, Bollerslev, Diebold and Labys)Set out a framework for building realised volatility from high-frequency intraday returns so that standard time-series methods can be used to model and forecast it.
Improving on the HAR benchmark
The HAR model forecasts tomorrow's realised variance from its daily, weekly and monthly averages. Tree models and neural networks use the same lags, and often extra features, to test whether nonlinear models improve on it.
Real exampleA machine learning approach to volatility forecasting (Christensen, Siggaard and Veliyev)Compared regularised regression, regression trees and neural networks with HAR models for Dow Jones constituents, and found machine learning competitive, with larger gains at longer horizons.
Pooling data across stocks
Volatility in different stocks moves together during the day. Training one model on many stocks, with a measure of market volatility as an input, gives it far more data than a separate model per stock.
Real exampleVolatility forecasting with machine learning and intraday commonality (Zhang, Zhang, Cucuringu and Qian)Forecast intraday realised volatility by pooling stocks and adding a market volatility proxy, found neural networks outperformed linear regressions and tree models, and found the results held for stocks outside the training set.
Using order book features
Spreads, depth and trade activity in the order book carry information about short-term volatility. Features built from them are added to lagged realised volatility for very short horizons.
Real exampleOptiver Realized Volatility Prediction (Kaggle competition)Optiver's competition provided one-second order book snapshots and trades for hundreds of stocks and asked for the realised volatility of the following 10-minute window, scored by root mean square percentage error.
Can you learn machine learning for volatility forecasting?
Yes. Volatility forecasting is one of the most learnable areas of financial machine learning, because the target is measurable and well-studied benchmarks exist. Learn Python and time-series statistics, build the realised volatility target and the HAR baseline, then add tree and neural network models.
- Python for intraday dataResample tick or second-level prices, compute log returns and sum squared returns per window.Learn it in Data Tools
- Time-series statisticsAutocorrelation, persistence and loss functions such as QLIKE and mean squared error for comparing volatility forecasts.Learn it in Statistics
- HAR and GARCH baselinesFit HAR on daily, weekly and monthly realised variance, and a GARCH model on daily returns with the arch package.
- Tree and neural network modelsGradient boosting and small networks on lagged realised variance and order book features, validated on later periods.Learn it in Machine Learning
- Sequence modelsRecurrent or Transformer models on intraday data, compared with HAR on the same test dates.Learn it in Deep Learning
Moving into machine learning for volatility forecasting
If you already work in the field
Risk managers, volatility traders and econometricians already know GARCH, HAR and how volatility clusters. Add Python and machine learning, then extend a HAR model with nonlinear terms or extra features and compare forecasts on the same out-of-sample period.
If you come from data science or ML
Data scientists should learn how realised volatility is defined, why measurement noise affects evaluation, and which loss functions are robust to that noise. The realised volatility literature and the HAR model give the benchmarks every forecast is judged against.
Roles that do this workVolatility quantQuantitative risk analystQuantitative researcherMarket risk modeller
Tools you will use
| Tool | What you use it for |
|---|---|
| arch | Fit GARCH-family volatility models as baselines for learned forecasts. |
| statsmodels | Fit HAR regressions of realised variance on daily, weekly and monthly lags. |
| pandas | Resample intraday prices and aggregate squared returns into realised variance. |
| LightGBM | Train gradient-boosted volatility forecasts on lagged realised and order-book features. |
| scikit-learn | Run time-ordered validation and compare model errors. |
Public data to practise on
- Optiver Realized Volatility Prediction: Order-book and trade snapshots for many stocks, used to predict realised volatility over a following window; needs a free Kaggle account.
- Binance public market data: Free 1-minute klines and trades for crypto pairs, enough to compute daily realised variance.
- Cboe VIX historical data: Daily VIX history of S&P 500 implied volatility, useful as a predictor or comparison for realised volatility.
Your first project
Download Binance 1-minute BTCUSDT klines, compute daily realised variance from 5-minute returns, forecast the next day with a HAR regression, and compare it with yesterday's value using QLIKE and RMSE.
Common questions
What is realised volatility?
Realised volatility is volatility measured from actual intraday returns, usually the square root of the sum of squared returns over a day or window. Unlike implied volatility, it is computed from prices that have already traded.
What is the HAR model?
The heterogeneous autoregressive model, proposed by Fulvio Corsi, forecasts realised variance from its averages over the past day, week and month. It is simple, hard to beat and the standard benchmark in this field.
Is volatility easier to predict than returns?
Yes. Volatility clusters, so calm and turbulent periods tend to persist and past volatility carries information about future volatility. Returns themselves show far weaker persistence.
What is the difference between realised and implied volatility?
Implied volatility is backed out from option prices and reflects the market's expectation, plus a risk premium, over the option's life. Realised volatility is measured after the fact, and implied volatility is often used as a feature to forecast it.
Courses for these skills
- Statistics and Probability for Machine LearningUnderstand distributions, dependence and uncertain estimates.41 published lessons · all planned lessons open
- Practical Machine Learning in PythonCompare forecasting models under a temporal split.51 published lessons · more lessons in production
- Python for AI and Machine LearningWrite repeatable functions for feature construction and evaluation.33 published lessons · all planned lessons open
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnWork with dates, missing observations and lagged tables.23 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.