Epipolar constraint for correspondence search

WHAT THIS TESTS The interviewer wants the core insight of multi-view geometry: a single point in one view does not pin its match to a point in the other view, but it does pin it to a line.

A GOOD ANSWER COVERS Given a point p1 in the first image, the corresponding 3D world point lies somewhere along the ray from the first camera center through p1. When you project that entire ray into the second image, it appears as a line, the epipolar line. Therefore the correct match p2 must lie on this epipolar line, which reduces the correspondence search from a two-dimensional sweep over the whole second image to a one-dimensional search along a single line. This relationship is epipolar geometry, and the fundamental matrix is the algebraic operator that, given p1, produces its epipolar line in the second image.

COMMON WRONG ANSWERS Saying you must search the entire second image. Confusing the epipole with the epipolar line. Believing a single point match is determined uniquely by geometry alone, when geometry only restricts it to a line.

LIKELY FOLLOW-UPS What are the epipoles and where do they lie. How does the fundamental matrix differ from the essential matrix. How does rectification make the epipolar lines horizontal.

ONE CONCRETE EXAMPLE Imagine a stereo pair of a tabletop. You pick the tip of a pencil, p1, in the left image. You do not know how far the pencil tip is from the cameras, so its 3D position could be anywhere along the viewing ray from the left camera through that pixel. Projecting that ray into the right image traces a line across it; the actual pencil tip in the right image, p2, must fall somewhere on that line. So instead of comparing p1's neighborhood against every pixel in the right image, you only compare along this epipolar line, which is both faster and far less error-prone. In practice you compute the fundamental matrix from several known correspondences, use it to draw the epipolar line for any new p1, and search there. If the stereo pair is rectified, the epipolar lines become horizontal rows, so the match simply lies on the same scanline, which is why stereo matching scans horizontally.