Batch Normalization: Stabilizing Neural Network Training

Why it exists

During training, as the weights of a neural network are updated, the distribution of each layer's inputs changes. This phenomenon, known as internal covariate shift, forces subsequent layers to constantly adapt to a moving target, which slows down training and makes it unstable. Batch Normalization was invented to reduce this shift.

The mental model

Think of Batch Normalization as a series of checkpoints inside a deep network. At each checkpoint, it inspects the data flowing through, recalibrates it to a standard 'normal' distribution, and then lets it proceed. This prevents the signal from becoming too weak (vanishing) or too strong (exploding) as it passes through many layers.

How it works

For each mini-batch of data during training, Batch Normalization first calculates the mean and variance of the activations. It then uses these statistics to normalize the activations, giving them a mean of zero and a variance of one. Crucially, it then applies a learned scaling factor (gamma) and a learned shifting factor (beta). These two parameters allow the network to learn the optimal scale and mean for each layer's inputs. During inference, it uses a running average of the mean and variance calculated during training, not the stats of the single inference example.

When to use it

Batch Norm is a default choice in many deep feed-forward networks, particularly for computer vision tasks. It is typically placed between a convolutional or linear layer and its activation function (e.g., Conv -> BatchNorm -> ReLU). Use it to accelerate training, use higher learning rates, and reduce the model's sensitivity to the initial weight configuration.

When not to use it

Its effectiveness can be limited with very small batch sizes, as the batch statistics become noisy and unreliable estimators of the true data distribution. In these scenarios, or in recurrent networks where applying it is complex, alternatives like Layer Normalization or Group Normalization are often preferred.

One canonical example

In a modern convolutional neural network like a ResNet, a typical block consists of a convolutional layer, followed by Batch Normalization, and then a ReLU activation function. The BatchNorm layer operates on the output channels of the convolution, normalizing the activations for each channel across all the examples in the mini-batch before they are passed to the ReLU.