Incremental Structure from Motion pipeline

WHAT THIS TESTS The interviewer wants a clear, ordered understanding of how unordered 2D photos become camera poses and a 3D point cloud, including the role of optimization.

A GOOD ANSWER COVERS The input is a collection of overlapping images of a static scene. The pipeline begins with feature extraction and matching: detect keypoints such as SIFT in every image and match descriptors between image pairs, filtering with the ratio test and geometric verification using the fundamental or essential matrix and RANSAC. Then choose a good initial image pair, estimate their relative pose from the essential matrix, and triangulate the matched points to seed an initial sparse 3D structure. The incremental step repeats: select the next image with enough correspondences to existing 3D points, estimate its pose by solving the Perspective-n-Point problem, triangulate newly visible points, and periodically run bundle adjustment, a nonlinear least-squares optimization that jointly refines all camera poses and 3D point positions to minimize reprojection error. The outputs are the camera poses, intrinsics if also solved, and a sparse 3D point cloud.

COMMON WRONG ANSWERS Omitting bundle adjustment, which is what keeps drift in check. Confusing the sparse SfM point cloud with the dense reconstruction produced later by multi-view stereo. Forgetting geometric verification of matches.

LIKELY FOLLOW-UPS Why is bundle adjustment necessary and what does it minimize. How does PnP register a new camera. How does incremental SfM differ from global SfM.

ONE CONCRETE EXAMPLE Suppose you have a hundred photos of a statue taken while walking around it. The pipeline detects and matches features across overlapping shots, picks two images with a wide-enough baseline and many shared points as the seed, recovers their relative rotation and translation from the essential matrix, and triangulates the shared features into an initial cloud of 3D points on the statue. It then adds a third image by matching its features to the existing 3D points and solving PnP for that camera's pose, triangulates the points it newly sees, and runs bundle adjustment to nudge every camera and point so that all 3D points reproject as closely as possible to their detected image locations. Repeating this across all hundred images yields the pose of every camera and a sparse 3D point cloud outlining the statue. A subsequent multi-view stereo stage can densify this into a full surface, but classical SfM itself outputs the poses and the sparse structure.