How to evaluate a RAG system end to end

WHAT THIS TESTS The interviewer wants to see whether you treat RAG as two coupled subsystems, retriever and generator, each with its own failure mode, instead of grading only the final text. Component isolation is what makes debugging tractable.

A GOOD ANSWER COVERS Retrieval evaluation first: given a query with known relevant documents, measure context recall, did we fetch the chunks that contain the answer, and context precision, how much retrieved content is actually relevant. Use ranking metrics like Mean Reciprocal Rank and NDCG when order matters. Generation evaluation second, assuming context is fixed: faithfulness or groundedness, does every claim in the answer trace back to the provided context, and answer relevance, does the answer address the question. These are commonly scored with an LLM-as-judge or natural language inference models. Finally, end-to-end correctness compares the final answer to a gold answer. Separating the layers lets you localize failures: high retrieval recall but low faithfulness means the generator ignores context; low recall means the retriever is the bottleneck.

COMMON WRONG ANSWERS Reporting only a single end-to-end accuracy or BLEU score. Conflating relevance, the answer fits the question, with faithfulness, the answer is grounded in context. Ignoring retrieval entirely and blaming the model.

LIKELY FOLLOW-UPS How do you build the gold set, annotate query-document relevance. How do you detect a faithfulness failure, NLI between answer and context. What if recall is high but precision is low, noisy context can degrade generation.

ONE CONCRETE EXAMPLE Users report wrong answers. Retrieval recall is 0.95, so the right chunks are present, but faithfulness is 0.4. The diagnosis is a generation problem; the model is overriding context with parametric knowledge, not a retriever fix.