BLEU Score: Judging Translation by Human Overlap
The BLEU score judges a machine translation by how closely its text matches a professional human translation. It's a popular, automated, and inexpensive way to benchmark translation systems, like comparing different versions of a model. The main footgun is that a high score indicates high textual overlap, not necessarily better fluency or meaning, as it's just a proxy for human judgment.
The mental model
BLEU (Bilingual Evaluation Understudy) is an algorithm that evaluates machine translation quality by measuring its similarity to one or more high-quality human translations. The core idea is simple: the more overlap a machine's output has with a professional's, the better it is. It's an 'understudy' because it approximates human judgment without the cost and time of actual human evaluation.
How it works
BLEU calculates a score based on the level of correspondence between the machine-generated text (the 'candidate') and a set of high-quality human translations (the 'references'). It essentially checks how many words and short phrases from the candidate translation also appear in the reference translations. A higher degree of match results in a higher score, indicating a translation that is considered closer to human quality. The metric was one of the first to demonstrate a high correlation with human judgments of quality.
When to use it
• Benchmarking: Comparing different machine translation systems against each other. • Regression Testing: Tracking the performance of a single translation model over time as it's being trained or updated. • Rapid Evaluation: Quickly and inexpensively getting a quality signal without needing to hire human evaluators for every iteration.
When not to use it
• As the sole measure of quality: Since it's based on textual correspondence, it can unfairly penalize creative or nuanced translations that use different wording (e.g., synonyms) than the specific reference text. • For judging fluency or creativity: BLEU is designed to measure precision and recall against a reference, not the aesthetic or grammatical elegance of the generated text on its own.
Canonical example
Imagine a human reference translation is: The cat sat on the mat.
• Machine Translation A: The cat sat on the mat. -> This would receive a very high BLEU score because it's a perfect match.
• Machine Translation B: A cat was sitting on the rug. -> This would receive a lower BLEU score. Even though the meaning is preserved, the words A, was, sitting, rug do not correspond to the reference text, so the overlap is lower.
This demonstrates the core footgun: BLEU rewards literal correspondence, not necessarily semantic equivalence.
Interview question
Which of the following best describes a significant limitation of using the BLEU score for evaluating machine translation?
- a.It is a costly and time-intensive process, making it impractical for frequent use.
- b.It cannot effectively compare the performance of different translation models.
- c.It may undervalue translations that are semantically accurate but diverge in wording from the reference.Correct
- d.It requires multiple human reference translations to achieve any meaningful result.
Why? this is the answer
The card explicitly states that BLEU's 'footgun' is that it can unfairly penalize creative or nuanced translations that use different wording (e.g., synonyms) than the specific reference text, even if the meaning is preserved. Options A and B are incorrect because the card highlights BLEU as an inexpensive, rapid, and effective method for benchmarking different translation systems.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #llms
- #machine translation
- #evaluation metrics
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llms — each one lists the topics its interview covers.
See open roles