Optical Flow: Tracking Motion by Watching Pixels

WHY IT EXISTS Computers process video as a series of static images. To understand motion, they need a way to quantify how things have moved between frames. Optical flow provides a mathematical method to estimate this motion from the changing brightness patterns in an image sequence.

THE MENTAL MODEL Think of optical flow as a vector field laid over a video. It doesn't track specific objects, but rather the apparent movement of brightness and texture. For each pixel, it asks: "Where did this pixel's brightness pattern move to in the next frame?" The answer is a vector representing a direction and a speed. The collection of all these vectors is the optical flow field.

HOW IT WORKS The core assumption is "brightness constancy": a point on an object will appear to have the same brightness in two consecutive frames, even if it moves. Algorithms search in the second frame for the best match for a pixel or patch from the first frame. The displacement between the original pixel's location and its new location gives the motion vector. This process is repeated across the image to build up a map of apparent motion.

WHEN TO USE IT Use optical flow when you need to estimate motion from a camera. Key applications include: first, video compression, to encode motion with vectors instead of re-drawing entire pixel blocks; second, robotics and autonomous vehicles, to perceive motion for navigation and obstacle avoidance; and third, video stabilization, to calculate and counteract camera shake.

WHEN NOT TO USE IT Optical flow fails when its core assumptions are broken. Avoid using it in scenes with major lighting changes (like a flickering light), as it will misinterpret brightness shifts as motion. It also performs poorly on textureless surfaces (like a blank wall) because there are no distinct patterns to track. Finally, it is unreliable when objects become occluded or leave the frame.

ONE CANONICAL EXAMPLE A drone uses optical flow to hold its position in the wind. Its downward-facing camera continuously calculates the flow field of the ground below. If the drone is pushed sideways by a gust, the ground will appear to move in the opposite direction. The drone's flight controller detects this motion via the optical flow vectors and applies thrust to counteract the drift, keeping it stationary.