The PnP problem in Structure from Motion

WHAT THIS TESTS Whether you understand how 3D reconstruction grows one camera at a time and the distinct geometric subproblems involved. PnP is the workhorse of incremental Structure from Motion.

A GOOD ANSWER COVERS PnP estimates the pose, meaning the rotation and translation, of a calibrated camera given a set of known 3D points in the world and their corresponding 2D projections in the image. The minimal case needs three correspondences, called P3P, which yields up to four solutions disambiguated by a fourth point. In SfM, the first two frames are reconstructed using the essential matrix and triangulation, producing an initial sparse point cloud. To add a third camera you cannot use the essential matrix alone reliably, so you match the new image's features to already-triangulated 3D points and solve PnP to place it in the same coordinate frame. New points are then triangulated and bundle adjustment refines everything.

COMMON WRONG ANSWERS Saying PnP triangulates 3D points; it does the opposite, recovering pose from known 3D points. Forgetting that real correspondences contain outliers, so PnP is wrapped in RANSAC. Ignoring that the camera intrinsics must be known.

LIKELY FOLLOW-UPS How many points does the minimal P3P solver need and why are there multiple solutions. How does RANSAC integrate with PnP. Why is registering new frames via PnP more stable than chaining pairwise essential matrices. How does bundle adjustment relate.

ONE CONCRETE EXAMPLE Reconstructing a building from a hundred photos. Frames one and two give an initial cloud of facade corners. For photo three, you match its detected features to those reconstructed corners, find say forty 2D to 3D matches, run RANSAC PnP to estimate where the camera stood, then triangulate new corners visible in three. Repeating this for every remaining photo, with periodic bundle adjustment, scales the model to the full structure.