What is catastrophic forgetting in LLMs and how do you mitigate it?

WHAT THIS TESTS: This question probes whether you understand the stability-plasticity dilemma in neural network training. Interviewers want to see that you recognize fine-tuning is not just about fitting new data but about preserving existing capabilities. They are looking for awareness of how gradient updates can overwrite general knowledge and whether you can name concrete guardrails mentioned in the literature.

A GOOD ANSWER COVERS: First, define catastrophic forgetting as the phenomenon where a model loses previously learned skills after being trained on new tasks or domains. Second, explain parameter-efficient fine-tuning approaches such as LoRA which restrict updates to a smaller subset of parameters rather than the full weight set. Third, mention regularization techniques that constrain fine-tuned parameters from drifting far from their original values. Fourth, reference continual learning strategies that organize training to protect critical knowledge. Fifth, stress the importance of evaluation on both the new target task and legacy holdout tasks to detect regression.

COMMON WRONG ANSWERS: A major red flag is confusing catastrophic forgetting with overfitting. Another is claiming that freezing the first few transformer layers fully solves the problem. Proposing unconstrained full fine-tuning on a small domain-specific dataset without any regularization or parameter-efficient method is also a weak signal. Finally, suggesting that simply lowering the learning rate eliminates forgetting shows a shallow understanding of the underlying mechanics.

LIKELY FOLLOW-UPS: Expect the interviewer to ask how you would quantitatively measure forgetting in a production LLM. They may also ask when parameter-efficient methods are insufficient and full fine-tuning becomes necessary. Another common thread is how to balance regularization strength against convergence speed on the new task. You might also be asked how continual learning scales to large models or how to select which layers to target with restricted updates.

ONE CONCRETE EXAMPLE: Suppose you adapt a generalist model for medical record summarization. A naive full fine-tuning run causes the model to degrade on coding and reasoning benchmarks. Instead, you apply LoRA while adding a regularization term that penalizes large deviations from the base checkpoint. You also reserve a slice of general-domain prompts for continual evaluation. This setup allows the model to master medical terminology and formatting while retaining its original software engineering and commonsense reasoning capabilities.