Explain vanishing and exploding gradients and common mitigation techniques.

WHAT THIS TESTS: This question tests whether you understand the mechanics of backpropagation in deep architectures, specifically how the chain rule creates multiplicative effects across layers. Interviewers want to see that you can connect mathematical properties like derivative ranges to training instability, and that you know which engineering tools address root causes rather than symptoms.

A GOOD ANSWER COVERS: Four things in order. First, define vanishing gradients as the exponential shrinkage of early layer gradients because backpropagation computes partial derivatives through repeated multiplication; mention that with activation functions like hyperbolic tangent whose derivatives live in the range 0 to 1, the product across many layers collapses toward zero. Second, define exploding gradients as the inverse problem where gradients grow exponentially, typically when weight initializations or activations produce derivatives whose magnitudes exceed one. Third, explicitly link the issue to depth rather than width, since the number of multiplications scales with the number of layers. Fourth, list concrete mitigation techniques and match them to the problem they solve: ReLU and its variants avoid the 0 to 1 derivative saturation of sigmoid or tanh; residual connections create shortcut paths that bypass multiplicative chains; batch normalization stabilizes the distribution of inputs to each layer; careful initialization schemes like He or Xavier keep initial gradient magnitudes in a controlled range; and gradient clipping imposes a hard ceiling to prevent explosion during training.

COMMON WRONG ANSWERS: Three red flags appear often. One is blaming the problem solely on activation choice without mentioning depth or the multiplicative chain rule effect. Another is confusing vanishing gradients with simple slow convergence or underfitting; the key is the exponential divergence between early and late layer updates, not just overall speed. A third is listing mitigations without explaining which problem each addresses, such as suggesting batch normalization for exploding gradients without noting it also helps vanishing by stabilizing activations.

LIKELY FOLLOW-UPS: An interviewer might ask why ReLU does not completely solve vanishing gradients, which lets you discuss dying ReLU and leaky variants. They might ask you to compare LSTM gating mechanisms to residual connections as alternative ways to preserve gradient flow. Another common pivot is asking how batch normalization changes the effective learning rate or why gradient clipping uses a norm threshold rather than an absolute value cap.

ONE CONCRETE EXAMPLE: Consider a ten layer network using tanh activations. Because each tanh derivative falls between 0 and 1, the gradient reaching the first layer is roughly the product of ten numbers each at most one, giving a magnitude bounded by an exponentially small upper limit. If the network instead uses large random weights with sigmoid activations in a recurrent setting, the same multiplicative chain can produce gradients with norms in the hundreds or thousands, forcing learning rate reductions or halting training unless clipping is applied.