Why does your RAG ignore or contradict retrieved context?

What's really being asked

This question probes your understanding of the full RAG stack beyond the LLM itself. Interviewers want to see if you can decompose a vague symptom, ignoring or contradicting context, into specific component failures across retrieval, context assembly, prompting, and model inference. They are looking for systematic debugging instincts rather than guesses.

The full answer

First, retrieval quality issues such as low relevance scores or embedding drift where the top-k documents are only 60 to 80 percent relevant instead of 95 percent, causing the model to distrust the context. Second, context assembly failures including chunk ordering that buries the most relevant passage at the end, token limit truncation that cuts off critical evidence, or poor delimiter formatting that makes documents blend together. Third, prompt design problems where instructions to use only the provided context are weak, ambiguous, or overridden by system prompts emphasizing helpfulness. Fourth, model-level grounding failures where parametric knowledge dominates retrieved evidence, especially in older models or when temperature is set too high. Fifth, a systematic debugging workflow: establish a baseline query with known correct retrieval, inspect retrieval scores and document content, swap in a golden context to isolate generation behavior, test prompt variations, and validate the full trace end to end.

The mistakes people make

Blaming hallucination without inspecting retrieval traces is the biggest red flag. Another weak pattern is suggesting retraining the embedding model or switching vector databases as a first step without evidence that retrieval is actually broken. Candidates who only talk about prompt engineering while ignoring chunking strategy or reranking pipelines also signal shallow RAG experience. Finally, proposing aggregate metrics like overall accuracy instead of per-trace inspection shows a lack of production debugging rigor.

What usually comes next

How would you measure retrieval relevance independently from generation quality? What changes if the contradiction only happens on edge cases or long documents? How do you prevent prompt injection from retrieved documents? Would you use a reranker, and where in the pipeline? How do you set up observability so the next failure is caught in hours instead of days?

A concrete example

Suppose a legal RAG system answers that the statute of limitations is two years when the retrieved document clearly states three years. The debug trace shows the correct document was retrieved at rank four with a score of 0.71 while three irrelevant FAQ chunks scored 0.74 and were placed first, causing the model to fixate on the wrong information. The fix is to add a cross-encoder reranker boosting the statute document to rank one, increase the prompt emphasis on the highest-ranked excerpt, and add a citation requirement so contradictions are explicit.