Vanishing Gradients and Why ReLU Helps

WHAT THIS TESTS This probes foundational knowledge of how gradients propagate in deep networks and why activation choice matters, a core concept behind modern deep learning's feasibility.

A GOOD ANSWER COVERS During backpropagation, the gradient at an early layer is a product of many per-layer derivatives chained through the network. Sigmoid and tanh are saturating: for large positive or negative inputs their outputs flatten and their derivatives approach zero, and even at best sigmoid's derivative peaks around 0.25. Multiplying many such small numbers makes the gradient shrink exponentially with depth, so early layers receive almost no learning signal, the vanishing gradient problem. ReLU, defined as max of zero and x, has a derivative of exactly one for all positive inputs, so it does not attenuate the gradient in its active region, letting signal propagate through many layers. It is also cheap to compute.

COMMON WRONG ANSWERS Confusing vanishing with exploding gradients, the opposite failure where products grow without bound. Claiming ReLU completely solves the problem; it helps the saturation cause but networks still need residual connections, normalization, and good initialization at great depth. Forgetting ReLU's downside: a neuron stuck in the negative region outputs zero with zero gradient and can die permanently.

LIKELY FOLLOW-UPS How do residual connections and batch or layer normalization help independently? What is the dying ReLU problem and how do LeakyReLU or GELU address it? How does weight initialization, such as He or Xavier, interact with this? Why might tanh still be chosen in some recurrent settings?

ONE CONCRETE EXAMPLE In a twenty-layer network using sigmoid, suppose each layer's local derivative is around 0.2. Chained across twenty layers the gradient scales by roughly 0.2 to the twentieth power, an astronomically tiny number, so the first layers effectively stop updating. Swapping to ReLU, the positive-region derivative of one means the chained product no longer collapses toward zero, and the early layers receive a usable gradient and actually learn.