MMLU Benchmark

WHY IT EXISTS Before MMLU, most language model benchmarks tested narrow skills, like reading comprehension on one dataset or sentiment classification on another. As models started to look broadly capable, researchers needed a way to check whether that breadth was real. MMLU, short for Measuring Massive Multitask Language Understanding, was built to test a model the way a battery of school and professional exams would test a person, across dozens of unrelated subjects at once, so a high score is hard to fake by being good at just one thing.

THE MENTAL MODEL Treat MMLU like a stack of exam papers pulled from real high school, undergraduate, and professional tests, covering everything from elementary mathematics to US law to clinical medicine. A model that only memorized patterns in one domain will do well on that domain's questions and poorly everywhere else. A model with genuinely broad, transferable understanding scores consistently across the whole stack. The single MMLU number reported for a model is essentially an average grade across 57 very different final exams.

HOW IT WORKS MMLU is roughly 16,000 multiple choice questions, four options each, spread across 57 subjects grouped into humanities, social sciences, STEM, and other professional areas, sourced from real practice exams and textbooks rather than written for the benchmark. Models are typically evaluated few shot, seeing a handful of worked examples before answering new questions, and accuracy at picking the correct letter becomes the score. Because grading is unambiguous, labs adopted it as a standard leaderboard metric.

WHEN IT MATTERS MMLU matters when comparing the general knowledge and reasoning breadth of different model releases, which is why it became a standard line in nearly every model announcement. It matters less once models cluster near the high 80s or 90s in accuracy, since the remaining gap is small and can hide larger real differences that a multiple choice format does not capture. The footgun is contamination: because the questions come from public exams and textbooks, some of them plausibly leaked into newer models' training data, inflating scores in ways that do not reflect genuine improvement. That concern, plus known labeling errors in the original question set, motivated follow ups like MMLU-Pro and MMLU-Redux.

ONE CONCRETE EXAMPLE Two models might both report a 78 percent overall MMLU score, but one gets there by excelling at STEM subjects like abstract algebra while struggling with moral scenarios, and the other is evenly strong across both, a difference invisible in the headline number but visible the moment the score is broken down by subject category.