Residual Connections & Layer Norm: The Transformer's Stabilizers

WHY IT EXISTS: Training very deep neural networks is notoriously difficult. As layers stack up, gradients can vanish (become zero) or explode (become huge), halting the learning process. Furthermore, deep networks can suffer from a "degradation" problem where adding more layers actually increases the training error because it's hard for the network to learn a simple identity mapping (i.e., just passing the input through unchanged).

THE MENTAL MODEL: Think of a deep network as a tall skyscraper. A residual connection is an express elevator that lets information from a lower floor (an early layer) skip many intermediate floors and get added directly to the information on a higher floor. This ensures the original signal is never lost. Layer Normalization is the climate control on each floor, ensuring the "environment" (the scale of numbers) is always stable and within a comfortable range, regardless of what's happening on other floors.

HOW IT WORKS: A residual connection takes the input to a block of layers, x, and adds it directly to the output of that block, F(x). The final output is F(x) + x. This structure makes it trivial for the block to learn an identity function by simply driving the weights in F(x) towards zero. Layer Normalization computes the mean and variance across all the features for a single training example within a layer. It then uses these statistics to rescale the activations for that example to have a mean of 0 and a standard deviation of 1, stabilizing the inputs to the next layer.

WHEN TO USE IT: These are standard, almost mandatory, components in modern deep learning, especially for any network with more than a dozen layers. They are the backbone of architectures like Transformers (GPT, BERT) and ResNets (for computer vision), enabling stable training at depths previously thought impossible.

WHEN NOT TO USE IT: For very shallow networks (e.g., fewer than 10-15 layers), the complexity might not be necessary. The degradation and gradient stability problems are less pronounced in shallow models, so the benefits may not outweigh the added architectural complexity.

ONE CANONICAL EXAMPLE: In a Transformer block, the input first passes through a multi-head attention mechanism. The output of this attention is then added to the original input (this is the residual connection). The result of this addition is then passed through Layer Normalization. This "Add & Norm" sequence is a fundamental repeating motif in the Transformer architecture, appearing after both the attention and the feed-forward sub-layers in every block.