What causes sudden loss spikes in long pre-training runs?

WHAT THIS TESTS: The interviewer is testing your ability to prioritize under pressure when weeks of GPU time are at risk. They want to see that you understand distributed transformer training mechanics, can distinguish root causes from symptoms, and know how to recover without starting from scratch. The focus is on practical debugging instincts and systems-level thinking rather than theoretical optimization.

A GOOD ANSWER COVERS: First, enumerate the common culprits in order of likelihood. These include sudden gradient norm explosions in attention or feed-forward layers, learning rate or batch size mismatches that violate scaling laws, FP16 overflow due to its limited dynamic range, and rare poison batches that produce anomalous loss. Second, describe an immediate triage workflow. A senior candidate will mention checking gradient norm logs first because a spike there confirms an explosion. Next, roll back to the last stable checkpoint rather than letting the run continue. Then, apply mitigations such as lowering the learning rate, enabling or tightening gradient clipping, switching from FP16 to BF16 if hardware allows, or inspecting the data shard around the failure step. Third, mention prevention. This includes using pre-layer normalization instead of post-layer normalization, adopting optimizers designed for large batches like LAMB or Adafactor, and maintaining robust logging of per-layer gradient norms and activation scales.

COMMON WRONG ANSWERS: A weak answer blames random hardware bit-flips or non-determinism without evidence. Another red flag is suggesting to restart training from scratch without attempting checkpoint recovery. Candidates who only mention lowering the learning rate but ignore gradient clipping or numerical precision show shallow experience. Similarly, proposing to increase the batch size to smooth out the spike demonstrates a misunderstanding of the stability-efficiency trade-off, since larger batches can actually narrow the safe learning rate range.

LIKELY FOLLOW-UPS: The interviewer may ask how you would identify which layer caused the explosion, prompting a discussion of per-layer gradient norm tracking. They might also ask about the differences between FP16 and BF16 overflow behavior, or how pre-training stability techniques change when moving to RLHF fine-tuning where policy drift and reward hacking create different spike patterns. Another common follow-up is how you would modify the optimizer or learning rate schedule if you had to keep the current batch size.

ONE CONCRETE EXAMPLE: Suppose you are pre-training a 7B parameter transformer and at step 42,000 the loss jumps from 2.1 to 8.7. You check the gradient norm monitor and see it spiked from 1.2 to 650. You immediately pause the job, restore the checkpoint from step 41,950, enable gradient clipping at norm 1.0, switch the mixed-precision backend to BF16, and resume with the learning rate dropped by half. Over the next 500 steps the loss returns to the prior trajectory and gradient norms stay below 2.0, confirming the recovery.

Source: rohan-paul.com