Perplexity versus BLEU for LMs

WHAT THIS TESTS The question checks whether you distinguish intrinsic from extrinsic evaluation and can pick the appropriate metric for a goal.

A GOOD ANSWER COVERS Perplexity is intrinsic: it is the exponential of the average negative log-likelihood the model assigns to held-out text, so it measures how well the model predicts the next token over a corpus. It needs only the model and test text, no reference outputs, and lower perplexity means better predictive language modeling. It is ideal for comparing models, tracking pretraining progress, and measuring fit on a domain. BLEU is extrinsic and reference-based: it compares a generated sequence against one or more human references by counting overlapping n-grams with a brevity penalty, originally for machine translation. It evaluates the quality of produced outputs for a specific task and requires references. You prioritize perplexity when assessing core language-modeling ability or training health, and BLEU when you have reference outputs and care about generation quality on tasks like translation or summarization.

COMMON WRONG ANSWERS Treating them as interchangeable, or using BLEU for open-ended generation like chat where many valid answers share few n-grams with any reference. Another error is thinking high perplexity is good; lower is better. Perplexity also says nothing about whether outputs are useful, only how predictable the text is.

LIKELY FOLLOW-UPS Expect questions on why perplexity is tokenizer-dependent and not comparable across vocabularies, on BLEU's weaknesses for semantics, on alternatives like ROUGE, BERTScore, and human or LLM-judge evaluation.

ONE CONCRETE EXAMPLE To decide which of two base models fits a medical corpus better, you compare perplexity on held-out medical text, needing no references. To grade a translation system, you compute BLEU against professional reference translations, since you care about the produced output matching known-good text, not raw next-token predictability.