Perplexity: Measuring a Model's Uncertainty
Perplexity frames a model's uncertainty as the effective number of choices it's considering. For a fair die with six outcomes, the perplexity is 6, reflecting perfect confusion among six options. When evaluating language models, a lower perplexity score indicates a better ability to predict a sequence of text. The footgun is judging the score in a vacuum; a 'good' perplexity is always relative to the task's inherent randomness.
The mental model
Perplexity measures the uncertainty of a probability distribution by telling you the "effective number of choices" it represents. A low perplexity means a model is confident and considering few options for the next outcome. A high perplexity means it's confused and spread thinly across many possibilities. It answers the question: "How many options are you equally confused about?"
How it works
Perplexity is defined as the exponentiation of information entropy. For a discrete distribution with N equally likely outcomes (like a fair die), the probability of each is 1/N, and the perplexity is simply N. • A fair coin has 2 outcomes with 1/2 probability each. Its perplexity is 2. • A fair six-sided die has 6 outcomes with 1/6 probability each. Its perplexity is 6. If a distribution is not uniform (e.g., a loaded die), the perplexity will be lower than the total number of outcomes, reflecting the increased predictability.
When to use it
• To quantify uncertainty: It provides a single, intuitive number for how hard it is to predict the outcome of a random process. • To evaluate predictive models: In machine learning, especially for language models, perplexity is calculated on a test set. A lower score means the model's probability distribution is closer to the empirical distribution of the test data. • To compare models: Given the same test data, the model with the lower perplexity is generally considered better at that specific prediction task.
When not to use it
• As an absolute score without context: A perplexity of 50 might be excellent for a complex task (predicting the next word in a novel) but terrible for a simple one (predicting the next character in "ababab..."). The value is only meaningful relative to the inherent randomness of the problem.
One canonical example
Consider a fair, six-sided die. There are 6 possible outcomes, each with a probability of 1/6. The perplexity of this system is exactly 6. This means that any prediction is as difficult as choosing one option from a set of 6 equally likely choices. If the die were loaded so that '6' appeared 50% of the time, the perplexity would drop significantly below 6, because the outcome is now more predictable and the model is less "perplexed."
Interview question
A data scientist observes that Model A has a perplexity of 10, while Model B has a perplexity of 20 on the same language modeling task. What is the most accurate conclusion?
- a.A. Model A is inherently more complex than Model B.
- b.B. Model A is better at predicting the sequence of text than Model B.Correct
- c.C. Model B is considering fewer effective choices for its predictions than Model A.
- d.D. Both models are performing poorly, as perplexity scores should ideally be closer to 1.
Why? this is the answer
The card states that 'a lower perplexity score indicates a better ability to predict a sequence of text' and 'the model with the lower perplexity is generally considered better.' Therefore, Model A, with a perplexity of 10, is performing better than Model B with a perplexity of 20. Option D is incorrect because the card warns against judging perplexity 'as an absolute score without context,' emphasizing that a 'good' perplexity is relative to the task's inherent randomness, not necessarily close to 1.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles