Detecting catastrophic forgetting in continual fine-tuning

WHAT THIS TESTS This checks whether you understand catastrophic forgetting, that fine-tuning on new data can silently erode previously learned abilities, and can design a guardrail to detect it before it reaches users.

A GOOD ANSWER COVERS The core idea is a stable, frozen evaluation set that represents the model's original capabilities and never changes between updates, so scores are comparable over time. Curate it to cover the breadth of what the chatbot already does well: general knowledge and reasoning, the original domain tasks, instruction following, formatting, tone, and a few safety and refusal checks. Critically, keep this benchmark strictly held out from the continual training data so it is not leaked and contaminated. After each fine-tuning round, run the full suite and record per-capability scores, then compute the delta against the immediately prior version and against the original baseline. A drop on old tasks while new-task performance rises is the signature of catastrophic forgetting. Set regression thresholds that gate deployment, so an update that degrades any key capability beyond tolerance is blocked or rolled back. Track trends on a dashboard to catch slow erosion across many small updates. Mitigations to mention include rehearsal by mixing in original data, regularization, and parameter-efficient tuning that limits weight drift.

COMMON WRONG ANSWERS Evaluating only on the new data; assuming old capabilities persist; letting the benchmark drift or leak into training; checking only aggregate score instead of per-capability deltas; having no deployment gate or rollback.

LIKELY FOLLOW-UPS How do you prevent benchmark leakage? What threshold blocks a release? How does rehearsal or LoRA reduce forgetting? How do you catch slow erosion over many updates?

ONE CONCRETE EXAMPLE A support bot is fine-tuned weekly on new tickets. You maintain a frozen suite of three hundred prompts spanning original FAQs, reasoning, formatting, and safety. After each weekly fine-tune you run it, compare per-category scores to the baseline, and find that math reasoning dropped ten points while ticket accuracy rose. The deployment gate flags the regression, and you add rehearsal data to recover the lost skill before shipping.