ResNet residual blocks and the degradation problem

WHAT THIS TESTS Whether you understand the specific failure ResNet addresses. Many people say it fixes vanishing gradients only; the deeper motivation is the degradation problem.

A GOOD ANSWER COVERS Before ResNet, simply stacking more layers in a plain network eventually made training accuracy worse, not just test accuracy. This is the degradation problem, and it is not overfitting because even training error rises. A residual block reformulates each block to learn a residual function: instead of the block directly producing the desired mapping H(x), it computes F(x) and then adds the input via an identity shortcut, so the output is F(x) plus x. If the optimal mapping is close to identity, the network only needs to drive F toward zero, which is far easier to optimize. The skip connection also provides a clean highway for gradients to flow backward, easing vanishing gradients in very deep stacks.

COMMON WRONG ANSWERS Saying ResNet's main benefit is preventing overfitting; it primarily enables optimization of depth. Claiming it only fixes vanishing gradients; the degradation problem is the headline. Forgetting that when dimensions change, the shortcut uses a one by one projection.

LIKELY FOLLOW-UPS Why is learning the identity hard for a plain stack but easy with a residual. How are dimension mismatches handled in the shortcut. What is a bottleneck block. How does this connect to highway networks.

ONE CONCRETE EXAMPLE Consider a plain thirty-four-layer net that trains worse than its eighteen-layer counterpart. Convert it to a ResNet by adding identity shortcuts around every pair of conv layers. Now the thirty-four-layer version trains to lower error than the eighteen-layer one, because each extra block can default to passing its input through unchanged, so the added depth can only help. This is exactly the result that let ResNets scale to one hundred and even one thousand layers.