Triangulating 3D points from two views

WHAT THIS TESTS The interviewer checks understanding of how 2D-to-3D reconstruction works given calibration and pose, and crucially how to handle measurement noise.

A GOOD ANSWER COVERS Geometry: a single 2D point in one image only constrains the 3D point to lie along a ray from that camera's center through the pixel. With two cameras whose projection matrices are known, the same 3D point lies on a ray from each camera, and the two rays should intersect at the point. Method: form equations from the projection relation x equals P times X for each view, stack them, and solve the homogeneous linear system using the Direct Linear Transform, typically via SVD, to get an initial 3D estimate. Refinement: because pixel measurements are noisy, the two rays generally do not meet exactly, so you refine the point by minimizing the reprojection error, the distance between the projected estimate and the observed points in both images, often as part of bundle adjustment.

COMMON WRONG ANSWERS Claiming the two rays always intersect exactly, ignoring noise and the need for least squares. Or saying one view alone suffices for depth. Or forgetting that triangulation requires baseline; pure rotation gives no parallax and fails.

LIKELY FOLLOW-UPS Why does pure rotation make triangulation degenerate. Why minimize reprojection error rather than the midpoint of the rays. How does baseline length affect depth accuracy. How does triangulation fit into structure from motion and bundle adjustment.

ONE CONCRETE EXAMPLE Two cameras a known distance apart both see a corner of a table. Each observed pixel defines a ray from its camera center, and the DLT stacks the projection equations and solves via SVD for the rays' best intersection, giving an initial estimate of the corner's 3D coordinates. Because the detected pixels carry noise the rays do not meet exactly, so reprojection-error minimization nudges the point until its reprojections best match the two observed pixels. Note the baseline matters: a wider separation between the cameras gives more parallax and a more accurate depth estimate, while pure rotation with zero baseline would make the triangulation degenerate.