Vanishing Gradients and Why ReLU Helps
grasp of deep-network training dynamics.
saturating activations shrink gradients across layers, ReLU's flat-one derivative preserves them.
confusing it with exploding gradients or ignoring ReLU's dead-neuron downside.
WHAT THIS TESTS This probes foundational knowledge of how gradients propagate in deep networks and why activation choice matters, a core concept behind modern deep learning's feasibility.
A GOOD ANSWER COVERS During backpropagation, the gradient at an early layer is a product of many per-layer derivatives chained through the network. Sigmoid and tanh are saturating: for large positive or negative inputs their outputs flatten and their derivatives approach zero, and even at best sigmoid's derivative peaks around 0.25. Multiplying many such small numbers makes the gradient shrink exponentially with depth, so early layers receive almost no learning signal, the vanishing gradient problem. ReLU, defined as max of zero and x, has a derivative of exactly one for all positive inputs, so it does not attenuate the gradient in its active region, letting signal propagate through many layers. It is also cheap to compute.
COMMON WRONG ANSWERS Confusing vanishing with exploding gradients, the opposite failure where products grow without bound. Claiming ReLU completely solves the problem; it helps the saturation cause but networks still need residual connections, normalization, and good initialization at great depth. Forgetting ReLU's downside: a neuron stuck in the negative region outputs zero with zero gradient and can die permanently.
LIKELY FOLLOW-UPS How do residual connections and batch or layer normalization help independently? What is the dying ReLU problem and how do LeakyReLU or GELU address it? How does weight initialization, such as He or Xavier, interact with this? Why might tanh still be chosen in some recurrent settings?
ONE CONCRETE EXAMPLE In a twenty-layer network using sigmoid, suppose each layer's local derivative is around 0.2. Chained across twenty layers the gradient scales by roughly 0.2 to the twentieth power, an astronomically tiny number, so the first layers effectively stop updating. Swapping to ReLU, the positive-region derivative of one means the chained product no longer collapses toward zero, and the early layers receive a usable gradient and actually learn.
Interview question
Why does replacing sigmoid with ReLU in a deep network reduce vanishing gradients?
- a.ReLU's output is bounded between 0 and 1, stabilizing the gradient
- b.ReLU normalizes activations to zero mean and unit variance each layer
- c.ReLU has a derivative of 1 for positive inputs, so it does not shrink gradients thereCorrect
- d.ReLU multiplies gradients by a factor greater than 1, amplifying them
Why? this is the answer
ReLU's unit derivative in its active region avoids the sub-one multiplications that cause sigmoid's gradients to vanish across depth. ReLU is unbounded above, does not normalize, and does not amplify gradients beyond one.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #deep-learning
- #vanishing-gradient
- #relu
- #activation-functions
- #backpropagation
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles