Epipolar Geometry: Finding 3D Points from 2D Images

WHY IT EXISTS A single 2D photograph loses all depth information; a point in an image could correspond to any object along a line of sight into the 3D world. To recover depth and reconstruct a 3D scene, you need at least two different viewpoints. Epipolar geometry provides the mathematical framework to relate these two views and triangulate points in 3D space.

THE MENTAL MODEL Imagine looking at an object with your left eye, then your right. The object appears to shift relative to the background. Epipolar geometry is the math that describes this shift. It tells you that if you identify a point with your left eye, you don't need to scan the entire world with your right eye to find it again. You only need to search along a specific line.

HOW IT WORKS The geometry is defined by two camera centers and a 3D point. These three points form a flat plane, known as the epipolar plane. The line where this plane intersects each camera's image sensor is called an epipolar line. The key constraint is this: for any point 'p' in the first image, its corresponding point 'p'' in the second image must lie on that epipolar line. This reduces the 2D search for a matching point down to a 1D search, dramatically speeding up the process.

WHEN TO USE IT This is fundamental for any system that computes 3D information from 2D images. Its primary use is in stereo vision for calculating depth maps, Structure from Motion (SfM) for building 3D models from photographs, and visual odometry for robot navigation, where a device tracks its own movement by observing features in the environment.

WHEN NOT TO USE IT The geometry breaks down if the two camera positions are identical (zero baseline), as there is no parallax shift. It's also ineffective in scenes without distinct, trackable features, like a uniform white wall. The model assumes a rigid scene, so it will produce errors if objects move independently between the two camera captures.

ONE CANONICAL EXAMPLE A self-driving car's stereo camera system uses two cameras a fixed distance apart. When the left camera sees a cyclist, the system knows the corresponding pixels for that cyclist in the right camera's image must lie on a specific horizontal line. By finding the matching pixels and measuring their horizontal offset (disparity), the car can instantly calculate the cyclist's distance.