The Vanishing Gradient Problem
Training a deep network is like a game of telephone; the error signal (gradient) gets weaker as it's passed back through layers. This happens in deep networks using sigmoid or tanh activations.
Why it exists
Deep neural networks learn by propagating an error signal from the final layer all the way back to the first. This process, backpropagation, uses the chain rule of calculus, which involves multiplying gradients from each layer. If these gradients are consistently small numbers, their product shrinks exponentially, causing the signal to disappear.
The mental model
Think of training a deep network like a game of telephone. The last person in line (the output layer) detects an error and whispers a correction to the person before them. That person relays the message backward, and so on. With each step, the message (the gradient) gets a little distorted and fainter. By the time it reaches the first few people (the input layers), it's too quiet to understand, so they don't change their behavior.
How it works
During backpropagation, the update for a weight in an early layer is proportional to the product of derivatives from all subsequent layers. Activation functions like the hyperbolic tangent (tanh) or sigmoid have derivatives in the range [0, 1]. Multiplying many numbers in this range results in an exponentially smaller value. Consequently, the gradients for the earliest layers become so small they effectively vanish, and those layers stop learning.
When to use it
This is not a technique to use, but a problem to be aware of. It's most prominent in very deep feed-forward networks or in Recurrent Neural Networks (RNNs) unrolled over many time steps. It is particularly triggered by using saturating activation functions like sigmoid and tanh, whose derivatives are small across most of their domain.
When not to use it
The problem is less severe in shallower networks. Modern architectures actively mitigate it. For example, using non-saturating activation functions like ReLU (Rectified Linear Unit), whose derivative is a constant 1 for positive inputs, prevents the signal from shrinking. Another solution is architectural, like the residual connections in ResNets which create a 'shortcut' for the gradient to bypass layers, allowing it to flow more freely.
One canonical example
Imagine a 20-layer network using the tanh activation function. The derivative of tanh is always ≤ 1. To update a weight in the first layer, backpropagation multiplies at least 19 of these derivative values. Even if each derivative is a relatively high 0.5, the resulting gradient for the first layer is scaled by 0.5^19, a number close to zero. The resulting weight update is minuscule, and the layer effectively stops training.
Interview question
What is the most significant consequence of the vanishing gradient problem in deep neural networks?
- a.The backpropagation algorithm becomes computationally inefficient and slow.
- b.Early layers of the network fail to learn effective features from the input data.Correct
- c.The network's output layer struggles to make accurate predictions.
- d.The network becomes prone to overfitting due to reduced regularization.
Why? this is the answer
The card states that vanishing gradients cause 'the gradients for the earliest layers become so small they effectively vanish, and those layers stop learning,' meaning they cannot learn effective features. Option D is incorrect because vanishing gradients lead to a failure to learn, not typically overfitting. Options C and D are general problems but not the specific, direct consequence on the learning of early layers.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #machine learning
- #neural networks
- #backpropagation
- #deep learning
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on machine learning — each one lists the topics its interview covers.
See open roles