How MMLU works and the contamination problem

WHAT THIS TESTS This checks whether you understand both the mechanics of a popular benchmark and why a single leaderboard number can mislead. Senior engineers must reason about whether a reported score reflects genuine ability.

A GOOD ANSWER COVERS MMLU mechanics: a large bank of multiple-choice questions, about sixteen thousand, spanning 57 subjects from history and law to college mathematics and medicine. The model is given a question and four options, often with a few in-context examples, and must pick a letter. The metric is plain accuracy, averaged per subject and overall. It probes breadth of knowledge across difficulty levels. Then the limitations: data contamination, since pretraining corpora scraped from the web frequently contain the exact benchmark items, so a high score can mean the model memorized the test rather than reasoning. Other issues include prompt sensitivity, scores swing with formatting and ordering, ambiguous or mislabeled questions in the dataset, and saturation as frontier models approach the ceiling, compressing differences into noise.

COMMON WRONG ANSWERS Describing MMLU as open-ended generation rather than multiple choice. Treating the score as an unimpeachable capability measure. Ignoring contamination entirely, or claiming held-out test sets fully prevent it when web leakage is hard to exclude.

LIKELY FOLLOW-UPS How do you detect contamination, n-gram overlap, canary strings, or performance gaps on rewritten variants. What mitigations exist, private holdout sets, freshly authored questions, and dynamic benchmarks. Why is accuracy alone insufficient, it hides reasoning quality.

ONE CONCRETE EXAMPLE A model scores 88 on MMLU but drops to 70 when the same questions are paraphrased and options reordered. That gap is a strong contamination and memorization signal, undermining the headline number.