Structure from Motion (SfM): Rebuilding 3D from 2D Photos

WHY IT EXISTS: A single 2D photograph loses all depth information. We need a way to recover the third dimension from commodity cameras without specialized hardware like LiDAR or stereo rigs. Structure from Motion solves this by using the one thing we have in abundance: multiple, simple 2D images taken from different positions.

THE MENTAL MODEL: Imagine you're standing still, looking at a chair and a distant tree. Now, take a step to the side. The closer object (the chair) will appear to move more against the background than the farther object (the tree). This effect is called motion parallax. SfM is the algorithmic version of this phenomenon. It takes a set of photos, finds common points between them, and uses their apparent shift to simultaneously calculate their 3D positions and the camera's position for each shot.

HOW IT WORKS: The process is a pipeline. First, it detects keypoints (like corners or distinct patterns) in every image. Second, it matches these keypoints across different images, identifying the same real-world point seen from multiple angles. Third, using the 2D positions of these matched points and the camera's estimated motion, a process called triangulation calculates the 3D position of each point. This is all refined using an optimization technique called bundle adjustment, which fine-tunes the 3D point cloud and all camera positions to minimize projection errors. The output is a sparse 3D point cloud and the camera poses for each input image.

WHEN TO USE IT: Use SfM when you need to create a 3D model from a set of unstructured photos and you don't have control over the camera setup. It's perfect for photogrammetry with a handheld camera or a drone, digitizing large outdoor scenes or buildings, and in robotics for visual odometry (estimating a robot's movement through a scene).

WHEN NOT TO USE IT: Avoid SfM for scenes with little to no texture (white walls, flat-colored objects), highly reflective or transparent surfaces (glass, chrome), or dynamic scenes where objects are moving independently. It also struggles if the camera motion is too small (not enough parallax) or if the images have no overlap. For real-time, high-precision needs, dedicated hardware like LiDAR or structured light scanners are often superior.

ONE CANONICAL EXAMPLE: A common use case is creating a 3D map of a city block using a drone. The drone flies a grid pattern, capturing hundreds of overlapping photos. An SfM pipeline processes these images to generate a sparse point cloud of key features (building corners, windows) and the drone's flight path. This sparse cloud is then densified to create a detailed 3D mesh, which can be textured using the original photos, resulting in a photorealistic digital model of the area.