BLEU Score: Judging Translation by Human Overlap
The BLEU score judges a machine translation by how closely its text matches a professional human translation. It's a popular, automated, and inexpensive way to benchmark translation systems, like comparing different versions of a model. The main footgun is that a high score indicates high textual overlap, not necessarily better fluency or meaning, as it's just a proxy for human judgment.
### The Mental Model BLEU (Bilingual Evaluation Understudy) is an algorithm that evaluates machine translation quality by measuring its similarity to one or more high-quality human translations. The core idea is simple: the more overlap a machine's output has with a professional's, the better it is. It's an 'understudy' because it approximates human judgment without the cost and time of actual human evaluation.
### How It Works BLEU calculates a score based on the level of correspondence between the machine-generated text (the 'candidate') and a set of high-quality human translations (the 'references'). It essentially checks how many words and short phrases from the candidate translation also appear in the reference translations. A higher degree of match results in a higher score, indicating a translation that is considered closer to human quality. The metric was one of the first to demonstrate a high correlation with human judgments of quality.
### When to Use It * **Benchmarking:** Comparing different machine translation systems against each other. * **Regression Testing:** Tracking the performance of a single translation model over time as it's being trained or updated. * **Rapid Evaluation:** Quickly and inexpensively getting a quality signal without needing to hire human evaluators for every iteration.
### When NOT to Use It * **As the sole measure of quality:** Since it's based on textual correspondence, it can unfairly penalize creative or nuanced translations that use different wording (e.g., synonyms) than the specific reference text. * **For judging fluency or creativity:** BLEU is designed to measure precision and recall against a reference, not the aesthetic or grammatical elegance of the generated text on its own.
### Canonical Example Imagine a human reference translation is: `The cat sat on the mat.`
* **Machine Translation A:** `The cat sat on the mat.` -> This would receive a very high BLEU score because it's a perfect match. * **Machine Translation B:** `A cat was sitting on the rug.` -> This would receive a lower BLEU score. Even though the meaning is preserved, the words `A`, `was`, `sitting`, `rug` do not correspond to the reference text, so the overlap is lower.
This demonstrates the core footgun: BLEU rewards literal correspondence, not necessarily semantic equivalence.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.