Images, language and games
Machine learning for video analysis
Machine learning for video analysis recognises actions in clips, tracks people and objects across frames and finds when events start and end. It supports sports analysis, traffic counting, safety monitoring and searching video archives.
How machine learning is used in video analysis
Recognising actions
Action recognition models look at short clips and label the activity, such as running, falling or a goal being scored. They extend image models to learn motion over time.
Real exampleKinetics and I3D (DeepMind)DeepMind introduced the Kinetics human action video dataset and a two-stream inflated 3D ConvNet (I3D) for action recognition.
Tracking people and objects
A detector finds objects in each frame and a tracker links them across frames, giving each one an identity. This supports counting, path analysis and safety monitoring.
Real exampleByteTrackByteTrack is a multi-object tracker that associates almost every detection box, including low-score ones, to recover occluded objects.
Sports broadcast analysis
Models spot key moments in matches, such as goals and cards, and segment camera shots and replays. This speeds up highlight production and match analysis.
Real exampleSoccerNet-v2SoccerNet-v2 provides around 300,000 annotations on 500 broadcast football videos, with benchmarks for action spotting, camera shot segmentation and replay grounding.
Traffic and smart city video
Cameras at junctions and in warehouses feed models that count vehicles, track movement across cameras and flag unsafe situations. Public challenges benchmark these tasks on shared data.
Real exampleAI City ChallengeThe AI City Challenge runs annual benchmark tracks on video analytics for transportation, smart cities and multi-camera environments.
Tagging and searching video archives
Models label what appears in each video so large collections can be searched and recommended. Precomputed features make this practical at scale.
Real exampleYouTube-8MGoogle's YouTube-8M dataset contains millions of YouTube video IDs with machine-generated labels from a vocabulary of more than 3,800 visual entities.
Can you learn machine learning for video analysis?
Yes. Video analysis builds directly on image models, so the order is Python, image deep learning, then detection, tracking and temporal models. A pretrained detector and tracker give useful results on your own footage quickly.
- Python and video handlingRead frames, sample clips and write annotated video with OpenCV or PyAV.Learn it in Python
- Image models firstLearn CNNs, transfer learning and object detection on single frames before adding time.Learn it in Deep Learning
- Detection and trackingRun a pretrained detector, link boxes with a tracker such as ByteTrack, and measure with IDF1 and MOTA.
- Temporal modelsTrain action recognition models on clips and learn how 3D convolutions and video transformers model motion.
Moving into video analysis
If you already work in the field
If you work in sports analysis, security, traffic engineering or broadcasting, you know which events matter and how footage is captured. Learn Python and a detection and tracking library, then count or track something you already measure by hand.
If you come from data science or ML
If you come from image or tabular machine learning, add video decoding, frame sampling, tracking metrics and the compute costs of video. Learn the privacy and consent rules that apply to filming people where you work.
Roles that do this workComputer vision engineer, videoVideo analytics engineerMachine learning engineerSports data analyst
Tools you will use
| Tool | What you use it for |
|---|---|
| OpenCV | Read video files and streams, step through frames and draw results. |
| PyAV | Decode video with precise frame timestamps through Python bindings to FFmpeg. |
| Ultralytics | Run object detection with built-in multi-object tracking on video frames. |
| Supervision | Track detections across frames, count objects crossing zones and annotate output video. |
| MMAction2 | Train and evaluate action recognition and temporal action detection models. |
| TrackEval | Evaluate multi-object tracking with metrics such as HOTA, MOTA and IDF1. |
Public data to practise on
Your first project
Download the MOT17 training sequences, detect people with a pretrained YOLO model and link them with ByteTrack in Supervision, and compare with simple frame-to-frame box matching using IDF1 and identity switches from TrackEval.
Common questions
How is video analysis different from image analysis?
Video adds time. Models must track identities across frames and recognise motion and events, and they process far more data.
Do I need to process every frame?
Often not. Sampling a few frames per second is enough for many tasks, while fast motion and tracking need higher rates.
Can video analysis run in real time?
Yes, with efficient detectors and trackers on a GPU or edge device. Heavier action recognition models are often run offline.
Courses for these skills
- Deep Learning with PyTorchExplore sequence representations after a baseline.Syllabus available · lessons open as produced
- Data Tools: NumPy, pandas, Matplotlib and scikit-learnPrepare arrays and preserve sampling metadata.23 published lessons · all planned lessons open
- Practical Machine Learning in PythonCompare feature-based classifiers.51 published lessons · more lessons in production
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.