Residual Networks (ResNets): Go Deeper, Not Worse

WHY IT EXISTS As neural networks get deeper, they can model more complex functions. But a paradox emerged: adding more layers to an already deep "plain" network caused accuracy to saturate and then degrade rapidly. This wasn't overfitting; the training error itself got worse. ResNets were created to solve this degradation problem, enabling networks to be built with hundreds or even thousands of layers.

THE MENTAL MODEL Instead of forcing a stack of layers to learn a complex mapping H(x) from scratch, a ResNet reformulates the problem. It assumes the input to the layers, x, is already a good approximation. The layers only need to learn the small correction, or residual, F(x), to get to the desired output. The final output is then the original input plus the learned correction: H(x) = F(x) + x. It's easier for a network to learn a small adjustment than to reproduce the entire identity function.

HOW IT WORKS A ResNet is built from residual blocks. Each block has two paths for its input, x. One path sends x through a series of transformations (like convolutions and activation functions) to produce a result, F(x). The other path is a 'skip connection' or 'shortcut' that bypasses these layers, carrying x directly to the end of the block. The outputs of both paths are then added together: F(x) + x. This simple addition allows gradients to flow more directly to earlier layers during training, mitigating the vanishing gradient problem and making it easier for layers to be 'skipped' by learning to output zero.

WHEN TO USE IT ResNets are the default choice for very deep computer vision models. They are foundational for tasks like image classification, object detection, and semantic segmentation where capturing features at many different scales requires significant network depth (e.g., 50, 101, or 152 layers).

WHEN NOT TO USE IT For shallower networks or simpler problems, the degradation problem that ResNets solve may not even appear. In these cases, the added complexity of skip connections might be unnecessary. A simpler architecture like a plain Convolutional Neural Network might suffice and be easier to implement and debug.

ONE CANONICAL EXAMPLE The original ResNet architecture won the 2015 ImageNet competition by a significant margin. A 152-layer ResNet demonstrated lower error than much shallower VGG networks, proving that training could be made effective at depths previously considered impractical.