How did Chinchilla change compute allocation between model size and data size?

WHAT THIS TESTS: The interviewer wants to see if you understand the shift from parameter-centric scaling to compute-optimal scaling, and whether you can translate academic findings into pre-training strategy. This is about knowing that FLOPs are a product of model size and data size, and that prior work got the balance wrong.

A GOOD ANSWER COVERS: First, the historical approach: earlier models like GPT-3 and Gopher scaled parameters aggressively while keeping training tokens relatively constant, leading to undertrained models. Second, Chinchilla's finding: for a fixed compute budget, model parameters and training tokens should be scaled in equal proportions, so if you double parameters you must double data. Third, the practical implications for a pre-training team: choose a smaller model than legacy rules of thumb suggest, budget for a much larger dataset, and enjoy lower inference and fine-tuning costs. Fourth, the empirical validation: Chinchilla at 70B parameters trained on 1.4 trillion tokens outperformed Gopher at 280B parameters on the same compute budget, reaching 67.5 percent on MMLU versus Gopher's lower score.

COMMON WRONG ANSWERS: A red flag is claiming that Chinchilla simply says more data is better without the equal-scaling rule. Another mistake is saying the paper recommends the biggest possible model and the most possible data independently, ignoring the fixed compute constraint. Some candidates also confuse Chinchilla with DeepSeek or other efficiency papers, or claim it only applies to inference optimization rather than training.

LIKELY FOLLOW-UPS: The interviewer might ask how you would estimate the optimal model size for a new 10e21 FLOP budget, or how the scaling laws change when data is scarce. They may also ask about the trade-off between training cost and inference cost, or whether these laws hold for mixture-of-experts architectures. Another common follow-up is how to source and deduplicate enough tokens to feed a Chinchilla-optimal model.

ONE CONCRETE EXAMPLE: Suppose your team has a budget equivalent to what Gopher used. Instead of training a 280B parameter model on 300 billion tokens, Chinchilla says you should train a 70B parameter model on roughly 1.4 trillion tokens. You would need to invest heavily in data pipelines, filtering, and deduplication, but you would get better downstream accuracy, faster inference, and cheaper fine-tuning.