Learning Rate Scheduling: A Gearbox for Model Training

WHY IT EXISTS: A static learning rate is a blunt instrument. Too high, and the optimizer overshoots the optimal weights, causing the loss to diverge. Too low, and training is impractically slow. Learning rate scheduling solves this by adjusting the rate during training for faster convergence and better final performance.

THE MENTAL MODEL: A learning rate schedule is a pre-planned 'gearbox' for your optimizer. Imagine parking a car: you approach the spot quickly at first, but slow down for the final, precise maneuvers to fit perfectly without hitting the curb. A schedule similarly starts with larger learning steps and reduces them as the model gets closer to a good solution.

HOW IT WORKS: A scheduler is an object that modifies the optimizer's learning rate at each training step or epoch based on a predefined rule. Common strategies include: first, a linear decay, where the rate decreases steadily; second, a cosine decay, which follows a cosine curve; and third, a warmup phase. A warmup starts the rate low, increases it over a set number of steps, and then begins the decay. This prevents early, large gradients from destabilizing a large model.

WHEN TO USE IT: Schedules are standard practice for training and fine-tuning large models, especially transformers like T5. They help the model quickly adapt to new data in the initial phases of training and then settle into a fine-grained local minimum for optimal performance. They are essential for achieving state-of-the-art results.

WHEN NOT TO USE IT: For simpler models, a fixed learning rate may be sufficient and avoids the complexity of tuning the schedule itself. More importantly, some optimizers like AdaFactor have built-in adaptive learning rate mechanisms. Using an external schedule with them can cause conflicts unless you explicitly disable the internal logic, for example by setting scale_parameter=False and relative_step=False in Hugging Face's AdaFactor implementation.

ONE CANONICAL EXAMPLE: When fine-tuning a T5 model, a common approach is a linear schedule with warmup. For 10,000 total training steps, you might use the first 1,000 steps for warmup, where the learning rate increases from 0 to its peak. For the remaining 9,000 steps, the learning rate then linearly decays back down to 0.