Evaluating a RAG system end to end

WHAT THIS TESTS The interviewer wants to see that you treat RAG as a pipeline with a retriever and a generator, each needing its own metrics so failures can be attributed correctly.

A GOOD ANSWER COVERS For the retrieval stage, context recall measures whether all the passages needed to answer the question were actually retrieved, and context precision measures how much of the retrieved context was relevant versus noise. Low recall means the answer cannot be grounded; low precision means the generator is distracted by irrelevant text. For the generation stage, faithfulness or groundedness measures whether the generated answer is supported by the retrieved context, catching hallucinations even when retrieval succeeded. Answer relevance measures whether the response actually addresses the user's question. Many teams compute these with an LLM-as-judge using frameworks like RAGAS, supplemented by human spot checks and a curated test set with reference answers.

COMMON WRONG ANSWERS Reporting only a single end-to-end score such as BLEU or a vague quality rating. That cannot distinguish a retrieval miss from a generation hallucination, so it gives no actionable signal. Another error is assuming high fluency implies correctness; a fluent but unsupported answer is a faithfulness failure.

LIKELY FOLLOW-UPS Expect questions on LLM-as-judge reliability and bias, on building a golden evaluation set, on detecting when retrieval recall is the bottleneck, and on online metrics like user thumbs and citation click-through.

ONE CONCRETE EXAMPLE A support bot answers a billing question incorrectly. Context recall is high, so the right document was retrieved, but faithfulness is low, revealing the model ignored the context and hallucinated. The fix targets generation, perhaps prompt grounding, rather than the retriever, a diagnosis impossible from a single overall score.