Why Elo Ratings for LLMs Can Be Misleading
Elo ranks LLMs like chess players, but models have fixed skills, not dynamic ones. This method powers leaderboards but produces volatile scores, meaning a model's rank can be an unstable estimate of its true, unchanging ability.
WHY IT EXISTS Evaluating LLMs is expensive and complex. Running massive benchmark suites is slow, and comparing models head-to-head requires significant effort. The Elo system, borrowed from chess, offers a seemingly simple way to create a single ranked list from many pairwise human judgments, like "Model A is better than Model B."
THE MENTAL MODEL Think of using Elo for LLMs as running a chess tournament where the players never practice, learn, or get worse. Their skill level is fixed. The Elo system, designed for players whose skill changes over time, is being used to estimate this fixed skill. The problem is that the path to that estimation can be rocky, and the final score can be sensitive to the order of the games played.
HOW IT WORKS In systems like the Chatbot Arena, users are shown responses from two anonymous models (A and B) for the same prompt and vote for the better one. Each win or loss updates the Elo scores of the two models involved. A model gains more points for beating a higher-ranked opponent and loses more for being defeated by a lower-ranked one. Over thousands of such comparisons, a ranked leaderboard emerges.
WHEN TO USE IT Use Elo leaderboards as a general, high-level signal of model performance, especially for subjective qualities like helpfulness or writing style that traditional benchmarks miss. It's a useful heuristic for the relative standing of models based on broad human preference.
WHEN NOT TO USE IT Do not treat Elo scores as absolute, precise measures of capability. Research shows these scores can be volatile and may not satisfy transitivity (where if A > B and B > C, then A > C). A small difference in Elo score between two models is often statistically meaningless. Avoid making critical decisions based on a model being ranked #3 instead of #4 on a public leaderboard.
ONE CANONICAL EXAMPLE The LMSYS Chatbot Arena is the most famous implementation. It collects millions of human votes from anonymous A/B tests to generate a live, Elo-based leaderboard of popular LLMs. While widely cited, its rankings are subject to the volatility and reliability issues inherent in applying a dynamic rating system to static entities.
Read the original → arxiv.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.