Science and engineering
Machine learning for robotics
Robots use machine learning to perceive objects and poses from cameras, to learn manipulation skills from human demonstrations, and to train locomotion and control policies in simulation before transfer to hardware.
How machine learning is used in robotics
Planning grasps from perception
A network looks at a depth image and predicts which grasp positions are likely to succeed, so a robot can pick up objects it has never seen. Training data can be generated in simulation at scale.
Real exampleDex-Net 2.0 (Mahler et al., 2017)Trained a Grasp Quality Convolutional Neural Network on 6.7 million synthetic point clouds to predict the probability that grasps succeed from depth images.
Learning skills from demonstrations
Imitation learning trains a policy on recorded human demonstrations, often collected by teleoperation. It is now a common way to teach manipulation tasks that are hard to program by hand.
Real exampleDiffusion Policy (Chi et al., 2023)Represented a robot's visuomotor policy as a conditional denoising diffusion process and reported an average 46.9% improvement over prior methods across 12 tasks from four manipulation benchmarks.
Training in simulation, running on hardware
Policies trained in simulation with randomised physics and visuals can transfer to real robots, avoiding the cost and risk of millions of real trials.
Real exampleSolving Rubik's Cube with a robot hand (OpenAI, 2019)Trained models only in simulation with automatic domain randomisation and used them to solve a Rubik's Cube with a real robot hand.
Following instructions with vision-language models
Models pretrained on web images and text can be trained to output robot actions, so a robot can follow natural-language instructions and generalise to new objects.
Real exampleRT-2 (Google DeepMind, 2023)Trained vision-language-action models that map robot observations to actions while drawing on web-scale vision and language pretraining to improve generalisation.
Can you learn machine learning for robotics?
Yes. Robotics engineers can add learning methods to the kinematics and control they know, and machine learning engineers can learn robotics through simulation. The order is Python, maths, machine learning and deep learning, then robotics foundations, imitation learning and reinforcement learning.
- PythonMost robot learning code and simulators have Python interfaces.Learn it in Python
- Linear algebra and calculusTransforms, kinematics and optimisation are the maths of robotics.Learn it in Maths
- Machine learning foundationsSupervised learning, validation and the habits that stop you fooling yourself with small datasets.Learn it in Machine Learning
- Deep learning and visionConvolutional networks and transformers for perception and policies.Learn it in Deep Learning
- Robotics and robot learningROS, control, simulators such as MuJoCo or Isaac Sim, imitation learning and reinforcement learning.
Moving into robot learning
If you already work in the field
Robotics engineers already know kinematics, control and hardware, which learning-based systems still rely on. Add deep learning and imitation learning, collect demonstrations on a robot you use and train a policy with an open framework such as LeRobot. Robotics companies and research labs increasingly hire for this combination.
If you come from data science or ML
Machine learning engineers should learn robot kinematics, control, sensors and ROS, and get hands-on with a simulator. Real robots fail in ways benchmarks do not show, so time on hardware matters.
Roles that do this workRobotics engineerRobot learning engineerPerception engineerResearch scientist, robotics
Tools you will use
| Tool | What you use it for |
|---|---|
| ROS 2 | Connect sensors, models and controllers on real and simulated robots. |
| MuJoCo | Simulate robot dynamics and contact for training and testing policies. |
| LeRobot | Train imitation-learning policies and load shared robot demonstration datasets. |
| Gymnasium-Robotics | Use standard simulated robot arm and hand tasks for RL experiments. |
| PyTorch | Build perception and policy networks. |
Public data to practise on
- LeRobot PushT: Human demonstrations of pushing a T-shaped block, with a matching simulator for evaluating policies.
- Open X-Embodiment: Robot demonstrations pooled from many labs and robot types for learning general manipulation policies.
Your first project
Download LeRobot's PushT dataset, train a behaviour-cloning policy with LeRobot, and compare it with a policy trained on a small fraction of the demonstrations using success rate in the PushT simulator.
Common questions
What is robot learning?
Robot learning covers methods that let robots acquire skills from data: imitation from demonstrations, reinforcement learning and learned perception and models. It complements classical planning and control.
Do I need a robot to learn robot learning?
Not to start. Simulators such as MuJoCo, PyBullet and Isaac Sim, plus public demonstration datasets, are enough to learn. A low-cost arm helps once you want to deal with real-world noise.
What is sim-to-real transfer?
It means training a policy in simulation and then running it on a real robot. Because simulators are imperfect, methods such as domain randomisation vary physics and visuals during training so the policy copes with reality.
Is ROS still needed with machine learning?
Usually, yes. ROS handles drivers, communication and tooling, while learned components run as nodes inside it.
Courses for these skills
- Deep Learning with PyTorchBuild neural policy and optimisation foundations.Syllabus available · lessons open as produced
- Maths for Machine LearningConnect objectives, state variables and optimisation.26 published lessons · all planned lessons open
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnKeep annotation coordinates and image identifiers aligned.23 published lessons · all planned lessons open
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.