Disparity Map: Seeing Depth from Two 2D Images

WHY IT EXISTS To extract 3D information from simple 2D images without using expensive active sensors like LiDAR. By using two standard cameras, a system can mimic biological vision to perceive depth, making 3D awareness more accessible and affordable.

THE MENTAL MODEL Hold your finger close to your face. Close your left eye, then your right. Your finger appears to jump a large distance against the background. Now do the same for a distant object; it barely moves. A disparity map is an image that records the size of that "jump" for every point in a scene. A big jump (high disparity) means the point is close. A small jump (low disparity) means it's far away. The map itself is often a grayscale image where brighter means closer.

HOW IT WORKS A stereo vision system captures two images of a scene from two cameras placed a known distance apart. An algorithm then takes a pixel from the left image and searches for its corresponding pixel in the right image. The horizontal distance (in pixels) between the pixel's position in the left image and its position in the right is its disparity value. This process is repeated across the image to build a 2D map of disparity values.

WHEN TO USE IT Use disparity maps as a foundational step for any task requiring depth perception from stereo cameras. This is common in autonomous driving for obstacle detection, in robotics for grasping and navigation, and in 3D reconstruction to create models of real-world objects and scenes.

WHEN NOT TO USE IT Avoid relying on disparity maps for scenes with large, textureless surfaces (like a blank wall) or highly repetitive patterns. The matching algorithm fails when it cannot find unique points to compare between the two images. It is also sensitive to poor lighting and camera misalignment. In these cases, active sensors like LiDAR are more reliable.

ONE CANONICAL EXAMPLE A self-driving car's perception system uses a forward-facing stereo camera to continuously generate disparity maps. This provides a real-time depth estimate of the road ahead, allowing the system to identify the distance to other cars, pedestrians, and curbs, which is crucial for path planning and collision avoidance.