Perplexity: Measuring a Model's Uncertainty
Perplexity frames a model's uncertainty as the effective number of choices it's considering. For a fair die with six outcomes, the perplexity is 6, reflecting perfect confusion among six options. When evaluating language models, a lower perplexity score indicates a better ability to predict a sequence of text. The footgun is judging the score in a vacuum; a 'good' perplexity is always relative to the task's inherent randomness.
Perplexity translates a probability distribution's uncertainty into an intuitive number: the effective number of choices a model is considering. For example, a fair coin has a perplexity of 2, and a fair die has a perplexity of 6. This concept is crucial for evaluating language models, where a lower perplexity on a test set indicates the model is less 'surprised' by the text and thus better at prediction. The main footgun is interpreting the score without context. A perplexity of 6 isn't 'bad' for a die roll—it's expected. The value is only meaningful relative to the problem's complexity.
Read the original → Wikipedia: Perplexity
- #information theory
- #llms
- #probability
- #metrics
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.