Describe the bias-variance tradeoff and how model complexity affects bias and variance

WHAT THIS TESTS: This question probes whether you see model flexibility as a dial that trades off two distinct sources of error rather than a knob that simply makes predictions better. Interviewers want to hear that you know the expected squared prediction error decomposes into three pieces: bias squared, variance, and irreducible noise. They also want to see if you can explain why the tradeoff matters for model selection and whether you can separate the concept from simply adding more training data.

A GOOD ANSWER COVERS four things in order. First, define bias as the error from overly strong assumptions in the learning algorithm, leading to underfitting when the model is too simple. Second, define variance as the error from excessive sensitivity to small fluctuations in the training set, leading to overfitting when the model is too complex. Third, state that increasing model complexity, such as adding polynomial degrees, hidden units, or features, reduces bias because the model can fit the training data more closely, but simultaneously increases variance because the fit becomes dependent on the specific sample. Fourth, mention that total test error traces a U-shaped curve as complexity grows, so the goal is to find the sweet spot that minimizes the sum of squared bias and variance.

COMMON WRONG ANSWERS include several red flags. One is claiming that both bias and variance decrease as you add parameters, which violates the fundamental tension. Another is saying that collecting more training data reduces both bias and variance in the same way that lowering complexity does; while more data can reduce variance, it does not change the bias-variance tradeoff itself. A third red flag is describing the tradeoff only in vague terms like overfitting versus underfitting without ever mentioning the mathematical decomposition or the U-shaped error curve.

LIKELY FOLLOW-UPS include asking how regularization affects the tradeoff, where you should explain that regularization effectively reduces variance at the cost of slightly increasing bias. Another follow-up is asking how the tradeoff manifests in specific algorithms, such as why a deep decision tree has low bias but high variance, or why linear regression has high bias but low variance. Interviewers may also ask how ensemble methods like bagging and boosting interact with the tradeoff.

ONE CONCRETE EXAMPLE: Consider predicting house prices. A model that always predicts the global mean has high bias because it ignores square footage and location, but near-zero variance because the mean does not change much across different training samples. A gradient boosted tree with thousands of splits has low bias because it fits every quirk of the training set, but high variance because a different sample of houses would produce a radically different tree structure. The optimal model might be a regularized linear model or a shallow ensemble that sits in the middle of the U-curve.

Source: Wikipedia: Bias–variance tradeoff