LLM Scaling Laws: Match Parameters to Tokens

WHY IT EXISTS: For years, the default strategy for improving language models was to increase parameter counts while keeping training datasets relatively static. Researchers assumed that bigger models absorbed more knowledge even with similar token counts. This created a fleet of massive but undertrained models that consumed enormous compute during training, fine-tuning, and inference without delivering proportional gains.

THE MENTAL MODEL: Think of compute budget as a pie that must be split between two ingredients: model capacity and data exposure. If you double the oven size but keep the same amount of dough, you get a thin crust. Scaling laws tell us that for every doubling of model size, you must also double training tokens to stay compute-optimal. The sweet spot is equal scaling, not pouring everything into parameters.

HOW IT WORKS: The Chinchilla paper trained over 400 models ranging from 70 million to 16 billion parameters on 5 to 500 billion tokens to map the loss landscape across model sizes and data volumes. They found that loss improves predictably when parameters and tokens scale equally for a fixed FLOP budget. This means a smaller model trained on proportionally more data can match or beat a much larger model trained on less data using the same total compute, while being dramatically cheaper to serve.

WHEN TO USE IT: Apply this when planning pretraining runs under a fixed compute envelope, choosing between model sizes for a new foundation model, or arguing for longer training runs instead of larger clusters. It is especially valuable when inference costs matter because smaller, better-trained models slash serving expenses.

WHEN NOT TO USE IT: Do not treat this as a universal law if you have an effectively infinite data budget and care only about absolute capability regardless of cost. If your application requires emergent abilities that appear only above certain parameter thresholds, or if you cannot source enough high-quality tokens, pure parameter scaling may still be the pragmatic path.

ONE CANONICAL EXAMPLE: DeepMind trained Chinchilla with 70 billion parameters on four times the data used for Gopher, despite both models sharing the same training compute budget. Chinchilla outperformed Gopher, GPT-3, Jurassic-1, and the 530 billion parameter Megatron-Turing NLG across downstream tasks, achieving 67.5 percent accuracy on MMLU, a 7 percent improvement over Gopher. It also used substantially less compute for fine-tuning and inference.