Eigenvalues, eigenvectors, and their role in PCA

WHAT THIS TESTS The interviewer wants proof that you grasp the linear algebra under common data science tools, not just the API call to fit PCA. They are checking whether you can move between an equation and its geometric meaning.

A GOOD ANSWER COVERS An eigenvector of a square matrix A is a nonzero vector v whose direction is unchanged when A is applied: A times v equals lambda times v. The scalar lambda is the eigenvalue and tells you how much v is stretched or shrunk. Geometrically, eigenvectors are the special directions a linear transformation only scales. For PCA you compute the covariance matrix of mean-centered features, then take its eigenvectors and eigenvalues. The eigenvectors are the principal components, mutually orthogonal directions, and each eigenvalue is the variance of the data projected onto its eigenvector. Sorting eigenvalues descending and keeping the top k gives the subspace that preserves the most variance.

COMMON WRONG ANSWERS Reciting the definition with no link to variance. Confusing eigenvalues with eigenvectors. Claiming PCA needs the correlation matrix when standardization was already done, or forgetting to center the data. Saying components are chosen to maximize accuracy rather than variance.

LIKELY FOLLOW-UPS How does the singular value decomposition relate to this, and why is SVD on the data matrix often preferred numerically. How do you pick k using the explained-variance ratio. Why must features be scaled first.

ONE CONCRETE EXAMPLE Take a two-feature dataset where height and weight are strongly correlated, forming a tilted ellipse cloud. The covariance matrix yields two eigenvectors: the first points along the ellipse's long axis with a large eigenvalue, the second is perpendicular with a small one. Projecting onto the first eigenvector compresses two columns into one while retaining most of the spread, which is exactly the dimensionality reduction PCA delivers.