Evaluating RAG Systems: Metrics for Retrieval and Generation

WHY IT EXISTS A Retrieval-Augmented Generation (RAG) system has two failure points: finding information and using it. A simple 'is the answer good?' evaluation is not enough because it doesn't tell you where the system broke. We need separate metrics to diagnose whether the retriever failed to find the right context or the generator failed to use it correctly.

THE MENTAL MODEL Think of evaluating a RAG system like grading an open-book exam. You grade two things separately. First, did the student find the right page in the textbook for the question? That's retrieval evaluation. Second, did the student use the information on that page to write a correct answer without making things up? That's generation evaluation.

HOW IT WORKS Evaluation is a two-stage process targeting the distinct components of the RAG pipeline. First, you evaluate the Retrieval component. The key metric here is relevance, which measures if the retrieved text chunks are pertinent to the user's query. High relevance means the system is good at finding the right information.

Second, you evaluate the Generation component. This involves several metrics. Faithfulness measures whether the generated answer is directly supported by the retrieved context and avoids hallucination. Accuracy compares the generated answer to a known 'ground truth' to determine if it is factually correct. These metrics ensure the generator is not just fluent, but also truthful and grounded in its sources.

WHEN TO USE IT Use these metrics when building, testing, or fine-tuning any RAG application. They are crucial for diagnosing performance bottlenecks. For example, if your final answers are poor but faithfulness is high, the problem likely lies in the retrieval step; you need to improve your document chunking or embedding model. Conversely, if retrieval relevance is high but faithfulness is low, the generator needs to be prompted or fine-tuned to stick to the provided context.

WHEN NOT TO USE IT Relying solely on these automated metrics can be misleading. They often require a 'ground truth' dataset for comparison, which is not always available for novel or open-ended queries. Furthermore, they don't capture subjective qualities like tone, conciseness, or overall helpfulness, which often require supplemental human evaluation to assess properly.

ONE CANONICAL EXAMPLE A user asks a RAG system, 'What was the key finding of the Auepora paper?'. The retriever pulls a relevant abstract. The generator then produces an answer stating, 'The paper concludes RAG is flawless.' This answer is unfaithful, as the source text only discusses evaluation challenges. The retrieval relevance is high, but the faithfulness score is low, pinpointing the generator as the source of the error.