BERTScore: Judging AI Text on Meaning, Not Just Words

WHY IT EXISTS Traditional text evaluation metrics like BLEU or ROUGE rely on n-gram overlap, meaning they count matching sequences of words. This approach penalizes valid paraphrases and fails to capture semantic similarity. A model could generate "the vehicle is fast" when the reference is "the car is quick," and a simple metric would score it poorly despite the meaning being nearly identical. BERTScore was created to solve this by judging meaning over exact phrasing.

THE MENTAL MODEL Think of BERTScore as an expert human judge instead of a simple keyword checker. A keyword checker just counts matching words and phrases between a generated text and a reference text. The expert judge, BERTScore, reads both texts, understands the contextual meaning of each word, and then gives a score based on how closely the core meanings align, even if the specific words are different.

HOW IT WORKS BERTScore takes a candidate sentence (the AI's output) and a reference sentence (the ground truth). It uses a pre-trained model like BERT to generate a contextual embedding—a numerical vector representing meaning—for every token in both sentences. Then, for each token in the candidate, it finds the most similar token in the reference by calculating their embedding similarity. These individual token-level similarity scores are then aggregated to produce a final score that reflects how well the candidate captures the semantics of the reference.

WHEN TO USE IT Use BERTScore when evaluating text generation tasks where semantic correctness is more important than lexical overlap. This is ideal for machine translation, abstractive summarization, image captioning, and dialogue systems. It provides a score that correlates more closely with human perception of quality and is more robust to adversarial paraphrasing.

WHEN NOT TO USE IT Avoid BERTScore when computational resources are extremely limited, as generating embeddings is much more expensive than simple n-gram counting. It's also less suitable for tasks where specific keyword matching is critical, such as evaluating a system that must extract specific named entities without alteration. The choice of the underlying language model can also influence the scores.

ONE CANONICAL EXAMPLE Consider a machine translation system. The reference text is "The astronaut floated in zero gravity." A model generates, "The spaceman drifted in weightlessness." A classic metric would give this a very low score due to no matching word pairs. BERTScore, however, would recognize the high semantic similarity between "astronaut"/"spaceman," "floated"/"drifted," and "zero gravity"/"weightlessness," assigning it a high score that reflects its accurate meaning.