Validation loss increases while training loss decreases: what is this?

What's really being asked

The interviewer wants to see if you can distinguish memorization from generalization in large model training. They care whether you understand that divergence between training and validation loss signals the model is extracting noise or idiosyncrasies from the training set rather than learning transferable structure. They also want evidence that you have a toolbox of regularization techniques and know when to apply them.

The full answer

First, name the phenomenon immediately as overfitting. Second, explain that the gap between decreasing training loss and increasing validation loss means the model is memorizing training examples instead of learning patterns that generalize. Third, give three distinct and concrete techniques. Early stopping is the first: monitor validation loss and halt training when it rises for a sustained number of steps. Dropout or weight decay is the second: dropout randomly zeroes activations during training to prevent co-adaptation, while weight decay adds an L2 penalty to the loss function to keep parameters small. Data augmentation or additional training data is the third: for language models this can mean back-translation, synonym replacement, or simply collecting more diverse text so the training distribution better matches the target distribution. A senior candidate might also mention reducing model capacity or using label smoothing as valid alternatives.

The mistakes people make

A major red flag is suggesting to train longer or increase model size without qualification, since both actions typically worsen overfitting. Another weak response is proposing a lower learning rate as the primary fix; while scheduling matters, it does not directly address the memorization gap. Candidates who only know one technique or who confuse overfitting with underfitting by claiming the model needs more parameters will also signal shallow experience.

What usually comes next

The interviewer may ask how early stopping interacts with learning rate schedulers that decay over time. They might probe whether dropout is still used in modern transformer architectures or if alternatives like DropPath or stochastic depth are preferred. Another common follow-up is to ask how you would detect overfitting when validation loss is noisy rather than monotonically increasing, or how regularization trade-offs change with model scale.

A concrete example

Suppose you are pretraining a one billion parameter decoder-only transformer on fifty billion tokens and you see validation perplexity rise after the first one hundred thousand steps while training perplexity keeps falling. You would implement early stopping with a patience of five thousand steps, add weight decay set to one tenth of the learning rate, and augment the corpus with filtered web text from a different domain to broaden coverage. You would then retrain and verify that the validation curve flattens rather than climbs.