Bundle Adjustment: Jointly Refining 3D Scenes and Cameras

Why it exists

Initial estimates of 3D structure and camera poses from multiple images are always noisy and inconsistent. If a 3D point is seen in several photos, projecting its estimated 3D position back into each camera view won't perfectly match where it was originally detected. Bundle adjustment was created to minimize this collective 'reprojection error' across the entire system.

The mental model

Bundle adjustment is a large-scale optimization that acts as a final polishing step. Imagine you have a rough 3D model, estimates of where your camera was for each photo, and a guess about your camera's lens. Bundle adjustment tweaks everything simultaneously—nudging the 3D points, shifting the camera positions, and refining the lens parameters—until the 3D model, when projected back into each camera view, aligns as closely as possible with the original photos.

How it works

The process minimizes a single cost function: the total reprojection error. This error is the sum of squared distances between observed feature locations in the images and the predicted locations from projecting the 3D points into the cameras. Because it adjusts the 3D points, camera motion parameters (rotation and translation), and camera intrinsic parameters (focal length, distortion) all at once, it is a non-linear least squares problem. Its name comes from adjusting the geometric 'bundles' of light rays that travel from each 3D point to each camera's optical center.

When to use it

Use bundle adjustment as the final, high-accuracy optimization stage in any multi-view 3D reconstruction pipeline. It is the gold standard for Structure from Motion (SfM), which generates 3D models from unordered image collections. It is also critical in visual SLAM (Simultaneous Localization and Mapping) systems for robots and augmented reality to maintain a consistent map and track the camera's location accurately.

When not to use it

Avoid bundle adjustment for real-time applications that cannot tolerate high computational latency, as it is very expensive. It is typically an offline or background process, not a fast initial estimation technique. For simple two-view reconstruction, basic triangulation is a sufficient starting point, though bundle adjustment can still be used to refine that result.

One canonical example

You take 100 photos of a statue from all angles. A Structure from Motion algorithm finds matching points and makes a rough guess of the 3D point cloud and camera poses. This initial result is sparse and distorted. Bundle adjustment then takes all 100 camera poses and all 3D points as input and jointly optimizes them, producing a geometrically correct and precise 3D model of the statue.