What is Bundle Adjustment and why is it tractable?

WHAT THIS TESTS The interviewer probes whether you understand the central refinement step of structure from motion and SLAM, what it minimizes, and the linear-algebra trick that scales it.

A GOOD ANSWER COVERS Bundle Adjustment is the joint nonlinear refinement of scene structure and camera parameters. It optimizes the 3D positions of scene points and the pose, and sometimes intrinsics, of every camera simultaneously, minimizing the total reprojection error, the sum over all observations of the squared distance between each observed image point and where its estimated 3D point projects into that camera. It is usually solved with Levenberg-Marquardt, an iterative nonlinear least-squares method. It is expensive because the number of parameters is enormous, thousands of points and many cameras, and the unknowns are coupled, so a naive solve of the normal equations is cubic in the parameter count. The key insight that makes large-scale BA tractable is sparsity: each 3D point is observed by only a few cameras, so the Jacobian and the resulting Hessian have a sparse, structured block pattern. Exploiting this with the Schur complement marginalizes out the many point variables to form a much smaller reduced camera system, which is then solved efficiently, often with sparse Cholesky factorization.

COMMON WRONG ANSWERS Saying it optimizes only camera poses or only points, not both. Not naming reprojection error. Missing the sparsity and Schur complement insight, the actual answer to the tractability question.

LIKELY FOLLOW-UPS Why reprojection error and not 3D error, robust loss functions for outliers, local versus global BA in SLAM, and gauge freedom and fixing the reference frame.

ONE CONCRETE EXAMPLE Reconstructing a building from hundreds of photos, BA jointly nudges all 3D points and camera poses so every feature reprojects as close as possible to where it was detected; because each corner appears in only a handful of images, the Schur complement reduces the giant system to a manageable camera-only solve.