Eigenvalues, eigenvectors, and their role in PCA
linear algebra intuition behind dimensionality reduction.
an eigenvector keeps direction under a matrix, its eigenvalue scales it; PCA finds eigenvectors of the covariance matrix as principal axes.
What's really being asked
The interviewer wants proof that you grasp the linear algebra under common data science tools, not just the API call to fit PCA. They are checking whether you can move between an equation and its geometric meaning.
The full answer
An eigenvector of a square matrix A is a nonzero vector v whose direction is unchanged when A is applied: A times v equals lambda times v. The scalar lambda is the eigenvalue and tells you how much v is stretched or shrunk. Geometrically, eigenvectors are the special directions a linear transformation only scales. For PCA you compute the covariance matrix of mean-centered features, then take its eigenvectors and eigenvalues. The eigenvectors are the principal components, mutually orthogonal directions, and each eigenvalue is the variance of the data projected onto its eigenvector. Sorting eigenvalues descending and keeping the top k gives the subspace that preserves the most variance.
The mistakes people make
Reciting the definition with no link to variance. Confusing eigenvalues with eigenvectors. Claiming PCA needs the correlation matrix when standardization was already done, or forgetting to center the data. Saying components are chosen to maximize accuracy rather than variance.
What usually comes next
How does the singular value decomposition relate to this, and why is SVD on the data matrix often preferred numerically. How do you pick k using the explained-variance ratio. Why must features be scaled first.
A concrete example
Take a two-feature dataset where height and weight are strongly correlated, forming a tilted ellipse cloud. The covariance matrix yields two eigenvectors: the first points along the ellipse's long axis with a large eigenvalue, the second is perpendicular with a small one. Projecting onto the first eigenvector compresses two columns into one while retaining most of the spread, which is exactly the dimensionality reduction PCA delivers.
Interview question
In PCA on mean-centered data, what does the eigenvalue associated with a principal component directly represent?
- a.The number of samples lying on that axis
- b.The correlation between the original two features
- c.The classification accuracy gained by keeping that component
- d.The variance of the data projected onto that component's directionCorrect
Why? this is the answer
Each eigenvalue equals the variance captured along its eigenvector, which is why components are ranked by eigenvalue. Accuracy is a downstream model metric, not what the eigenvalue measures.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #pca
- #linear-algebra
- #dimensionality-reduction
- #eigenvectors
- #machine-learning
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles