What is cross-validation and why is it more robust than a holdout split?

What's really being asked

This question probes whether you understand the statistical purpose of model evaluation, specifically the bias-variance tradeoff in performance estimation and data efficiency. Interviewers want to see that you know a single train-test split is just one noisy sample of model performance, and that you can explain k-fold cross-validation as a resampling procedure that reduces variance without requiring more data. It also checks if you treat CV as an evaluation protocol rather than a training algorithm.

The full answer

First, define cross-validation as a model evaluation technique that partitions data into multiple complementary subsets so the model can be trained and validated across different divisions. Second, explain why a simple train-test split is less robust: the score depends heavily on which specific rows land in the test set, giving high variance and wasting data because the test set is never used for training. Third, describe k-fold cross-validation step by step: shuffle the dataset, split it into k equal folds, iterate k times so each fold serves exactly once as the test set while the remaining k-1 folds form the training set, then compute and average the k performance scores. Fourth, note the benefits: every data point gets used for both training and testing, the averaged score is a more stable estimate of out-of-sample performance, and with typical values like k equals 5 or 10 the procedure remains computationally tractable.

The mistakes people make

A major red flag is describing cross-validation as a way to train a single final model; CV evaluates procedures and hyperparameters, it does not output one production-ready estimator trained on all folds. Another mistake is claiming that cross-validation prevents overfitting; it only measures generalization more reliably, and overfitting can still occur during the model fitting process inside each fold. Candidates also err by ignoring the computational cost: k-fold requires k times more training than a single split. Finally, applying CV without preserving group structure or time ordering, as in medical or time-series data, leaks information and invalidates the scores.

What usually comes next

The interviewer may ask how to choose k, prompting a discussion of the bias-variance tradeoff for the performance estimator itself, where k equals the sample size is leave-one-out and k equals 2 is a single holdout repeated once. They might ask about stratified k-fold to preserve class distribution, or about nested CV when you are both tuning hyperparameters and estimating generalization error. Time-series splits or group k-fold often come up as domain-specific alternatives.

A concrete example

Suppose you have 150 iris samples and use 5-fold cross-validation. You divide the data into five groups of 30 samples each. In round one you train on folds two through five and score on fold one. In round two you train on folds one, three, four, and five and score on fold two. You repeat this until every fold has been tested once, yielding five accuracy scores. The final reported metric is the mean of those five scores, giving a more reliable estimate than a single 70-30 split that might accidentally place most of the hard-to-classify flowers in the test set.