Images, language and games
Machine learning for inverse graphics
Machine learning for inverse graphics recovers the 3D shape, camera pose, materials and lighting that explain one or more photos. Differentiable renderers and neural radiance fields fit a scene so its rendered images match the observed ones, which powers 3D capture and novel-view synthesis.
How machine learning is used in inverse graphics
Novel view synthesis from photos
A neural field is fitted to photos of a scene taken from known camera positions, so it can render the scene from new viewpoints. The model learns colour and density at each point in space.
Real exampleNeRFNeRF represents a scene as a neural radiance field optimised from input images and renders photorealistic novel views.
Real-time 3D capture
Newer scene representations replace a slow neural network with many small 3D Gaussians that render quickly. This makes captured scenes practical for interactive viewing.
Real example3D Gaussian SplattingResearchers at Inria and partners presented 3D Gaussian Splatting for real-time radiance field rendering of captured scenes.
Indoor views in mapping products
Companies capture venues with many photos and reconstruct them so users can move through a space virtually. The pipeline combines camera pose estimation, neural fields and rendering.
Real exampleGoogle Maps Immersive ViewGoogle Research describes how it used NeRF to deliver indoor views of restaurants, cafes and other venues in Immersive View.
Recovering cameras and geometry
Learned models estimate camera poses and dense 3D points directly from image pairs, a step that traditionally needed calibrated cameras. The output can seed reconstruction or be used on its own.
Real exampleDUSt3RDUSt3R reconstructs dense 3D geometry from arbitrary image collections without prior camera calibration or viewpoint poses.
Depth from a single image
Monocular depth models predict how far each pixel is from the camera from one photo. They are used for 3D effects, robotics and as a starting point for full reconstruction.
Real exampleDepth AnythingDepth Anything is a monocular depth estimation model trained with large-scale automatically annotated unlabelled images.
Can you learn machine learning for inverse graphics?
Yes, though it is one of the more maths-heavy areas of vision. You need comfortable Python, linear algebra and some deep learning, plus camera geometry. Learn those in order, then fit your first scene with Nerfstudio before reading the papers in depth.
- Python and PyTorchWrite tensor code, run training loops and use a GPU, since every inverse graphics tool is built on these.Learn it in Python
- Linear algebra and calculusRotations, projections and gradients are the core of camera models and differentiable rendering.Learn it in Maths
- Deep learning foundationsMultilayer perceptrons, optimisers and positional encodings are the building blocks of neural fields.Learn it in Deep Learning
- Camera geometry and renderingLearn pinhole cameras, poses, ray marching and structure from motion with a tool such as COLMAP.
- Fit and evaluate a sceneTrain a NeRF or Gaussian splatting model on a public scene and measure held-out views with PSNR.
Moving into inverse graphics and 3D vision
If you already work in the field
If you work in graphics, visual effects, photogrammetry or surveying, you already know cameras, meshes and rendering. Add PyTorch and gradient-based optimisation, then reproduce a NeRF or Gaussian splatting result on your own capture.
If you come from data science or ML
If you come from machine learning, the gap is 3D geometry: camera models, poses and rendering. Work through a graphics or multi-view geometry text alongside fitting scenes in Nerfstudio.
Roles that do this work3D computer vision engineerResearch scientist, neural renderingGraphics engineerPhotogrammetry specialist
Tools you will use
| Tool | What you use it for |
|---|---|
| PyTorch3D | Render meshes and point clouds differentiably and optimise shape, pose and texture from images. |
| Mitsuba 3 | Physically based differentiable rendering for recovering materials, lighting and geometry from photos. |
| nvdiffrast | Fast GPU rasterisation primitives for differentiable mesh rendering in PyTorch. |
| Nerfstudio | Train and view neural radiance fields and Gaussian splats from your own photos or standard datasets. |
| Kaolin | NVIDIA's PyTorch library for 3D deep learning, with differentiable rendering and mesh operations. |
Public data to practise on
- NeRF synthetic and real scenes: Rendered objects such as a Lego model with known camera poses, plus real scenes, linked from the NeRF project page.
- Mip-NeRF 360: Real scenes photographed from all around, for testing reconstruction and novel-view synthesis.
- DTU MVS: Objects photographed by a robot arm from many positions with structured-light reference scans, for checking recovered geometry.
Your first project
Download the NeRF synthetic Lego scene, fit it with Nerfstudio's nerfacto method, and compare its held-out test views with the nearest training image as a baseline using PSNR.
Common questions
What is inverse graphics?
Graphics turns a 3D scene into an image. Inverse graphics goes the other way, recovering shape, materials, lighting and camera from images.
What is the difference between NeRF and Gaussian splatting?
NeRF stores a scene inside a neural network queried along camera rays. Gaussian splatting stores it as many explicit 3D Gaussians, which renders much faster.
Do I need a powerful GPU for NeRF?
Fitting a scene is practical on a single modern consumer GPU or a cloud notebook GPU. Large scenes and high resolutions need more memory and time.
Can I capture a scene with a phone?
Yes. Many projects start from a phone video, estimate camera poses with structure from motion, and then fit a radiance field or Gaussian splatting model.
Courses for these skills
- Maths for Machine LearningWork with norms, dot products and vector geometry.26 published lessons · all planned lessons open
- Deep Learning with PyTorchLearn image representations and reconstruction objectives.Syllabus available · lessons open as produced
- Generative AI: LLMs, RAG and AgentsConnect embeddings and retrieval to language-model applications.Syllabus available · lessons open as produced
New to programming or maths? Start with Python, or follow the machine learning path in order. Published lessons are free to read; compare Free and Pro.