Layer Norm and Residuals in Transformer Blocks

WHAT THIS TESTS This is an advanced probe of why Transformer blocks train stably at great depth and why a specific normalization was chosen for sequence models.

A GOOD ANSWER COVERS A Transformer block has two sublayers, multi-head self-attention and a position-wise feed-forward network. Each is wrapped with a residual connection: the sublayer's input is added to its output, so the block computes input plus sublayer of input. Residuals give gradients a direct path to earlier layers, mitigating vanishing gradients and letting very deep stacks train. Layer normalization is applied around each sublayer, in the original post-norm design after the residual add, and in many modern models in a pre-norm position before the sublayer for better stability. Layer norm normalizes each token's activation vector across its feature dimension to zero mean and unit variance, then applies learned scale and shift, keeping activation magnitudes well-behaved.

WHY NOT BATCH NORM Batch normalization computes statistics across the batch dimension per feature, which is problematic for NLP: sequences have variable length and padding, batch statistics are noisy with small or variable batches, and train-time batch statistics differ from inference running averages. Layer norm computes statistics per individual token over features, so it is independent of batch size and sequence length and behaves identically in training and inference.

COMMON WRONG ANSWERS Swapping which dimension each normalizes. Claiming residuals are merely optional shortcuts rather than essential for deep training. Ignoring the pre-norm versus post-norm distinction and its effect on stability.

LIKELY FOLLOW-UPS What is the practical difference between pre-norm and post-norm and why did pre-norm become common? What is RMSNorm and why do recent models use it? How do residuals interact with initialization and warmup? Why does padding make batch norm statistics unreliable here?

ONE CONCRETE EXAMPLE In a 24-layer Transformer, the residual connections let the loss gradient reach layer one almost undiminished, so all layers learn. Layer norm rescales each token's 768-dimensional vector independently, so a batch mixing a 5-token sentence with a 200-token document is unaffected by their differing lengths, whereas batch norm would compute unstable per-feature statistics across that ragged, padded batch.