Science and engineering
Reinforcement learning for optimal control
Reinforcement learning finds control policies by trial and error in a simulator, which suits systems with nonlinear dynamics, contact or hard-to-model costs. It is used for robot locomotion, process control and energy management, often alongside classical controllers like LQR and MPC.
How reinforcement learning is used for optimal control
Industrial and building energy control
An agent learns which setpoints reduce energy use while keeping temperatures and equipment within limits. Safety constraints and human override remain part of the system.
Real exampleAutonomous data centre cooling (Google DeepMind, 2018)Every five minutes, deep neural networks predict how candidate cooling actions affect future energy use, and the system chooses actions that minimise energy while satisfying safety constraints.
Controlling plasma in fusion experiments
Tokamaks need fast, coordinated control of many magnetic coils to hold the plasma in shape. A policy trained in simulation can replace a set of hand-designed controllers.
Real exampleMagnetic control of tokamak plasmas (Degrave et al., Nature, 2022)Trained a deep reinforcement learning controller in simulation that controlled the TCV tokamak's magnetic coils and produced a range of plasma configurations, including two separate plasmas held at once.
Navigation under uncertainty
When dynamics depend on changing conditions such as winds, a learned policy can plan with forecasts and react to what it observes, where fixed rules struggle.
Real exampleStratospheric balloon navigation (Bellemare et al., Nature, 2020)Used deep reinforcement learning to station Loon superpressure balloons, including a 39-day controlled experiment over the Pacific Ocean.
Legged robot locomotion
Walking robots have contact-rich dynamics that are hard to model for classical controllers. Policies trained in simulation, with randomised physics, can transfer to real hardware.
Real exampleAgile legged locomotion (Hwangbo et al., 2019)Trained neural network policies in simulation and deployed them on the ANYmal quadruped, which followed velocity commands, ran faster than before and recovered from falls.
Can you learn reinforcement learning for control?
Yes, with a solid base in control or machine learning. Learn Python, probability and optimisation, then Markov decision processes and dynamic programming, then policy-gradient and actor-critic methods in simulation, comparing everything with LQR and MPC baselines.
- PythonWrite simulations and training loops, and use environments from Gymnasium.Learn it in Python
- Linear algebra, calculus and optimisationThe foundation for LQR, policy gradients and value functions.Learn it in Maths
- ProbabilityExpectations, variance and sampling drive reinforcement learning updates and explain their noise.Learn it in Statistics
- Neural networksPolicies and value functions in deep reinforcement learning are neural networks trained with gradients.Learn it in Deep Learning
- Reinforcement learning methodsMarkov decision processes, Q-learning, PPO and SAC, then sim-to-real techniques such as domain randomisation.
Moving into reinforcement learning for control
If you already work in the field
Control engineers already know system dynamics, stability and MPC, and know when a controller is unsafe. Learn deep reinforcement learning in simulation, train a policy for a system you can also control with MPC, and compare robustness and constraint handling. Robotics, energy and process industries use reinforcement learning alongside classical control.
If you come from data science or ML
Machine learning engineers should learn classical control (PID, LQR and MPC) and system identification, since these are strong baselines and the language of the field. A control textbook and a simulator such as MuJoCo give the practical base.
Roles that do this workControl systems engineerReinforcement learning engineerRobotics engineerResearch scientist, reinforcement learning
Tools you will use
| Tool | What you use it for |
|---|---|
| Gymnasium | Define control environments with a standard interface for RL agents. |
| Stable-Baselines3 | Train reliable implementations of PPO, SAC and other RL algorithms. |
| MuJoCo | Simulate rigid-body dynamics with contact for control and robotics tasks. |
| Python Control Systems Library | Design LQR and other classical controllers as baselines. |
| CasADi | Formulate model predictive control as a numerical optimisation problem. |
Public data to practise on
- Minari: Offline datasets of agent trajectories for Gymnasium and MuJoCo tasks, for offline RL and behaviour cloning.
Your first project
Download Minari's MuJoCo inverted pendulum expert dataset, train a behaviour-cloning policy from it, and compare it with an SAC agent trained online in the same environment using mean episode return and episode length.
Common questions
What is the difference between reinforcement learning and optimal control?
Both look for actions that minimise a cost over time. Optimal control usually starts from a known model and solves for the controller, while reinforcement learning learns from interaction or simulation, often without an explicit model.
Is reinforcement learning better than MPC?
Not in general. MPC handles constraints explicitly and is well understood. Reinforcement learning can help with dynamics or costs that are hard to model and is fast at run time once trained, and many systems combine the two.
Why is reinforcement learning usually trained in simulation?
It needs many trials, and early trials can be unsafe or damage hardware. Simulators allow millions of attempts, and techniques such as domain randomisation help policies transfer to the real system.
Which algorithms are used for continuous control?
Proximal policy optimisation (PPO) and soft actor-critic (SAC) are common starting points, with model-based methods used when data is expensive to collect.
Courses for these skills
- Deep Learning with PyTorchStudy sequence models and neural optimisation.Syllabus available · lessons open as produced
- Maths for Machine LearningConnect state vectors, derivatives and dynamical behaviour.26 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.