How would you architect a multi-turn conversational RAG system?

WHAT THIS TESTS: This question evaluates whether you understand that single-turn RAG breaks down in real conversations. Interviewers want to see you design explicit mechanisms for memory, query reformulation, and context-aware retrieval rather than hoping the LLM prompt window handles everything.

A GOOD ANSWER COVERS: First, conversation memory architecture with short-term windows of the last 5 to 10 turns for immediate context, summary-based compression for longer dialogues to stay within token limits, and optional long-term memory across sessions. Second, query reformulation strategies: LLM-based rewriting that replaces pronouns with referents and injects implicit context to create standalone queries, rule-based pattern matching for pronoun replacement and topic continuation for speed, and a hybrid approach that routes common cases to rules and falls back to LLM rewriting for complex references. Third, contextual retrieval mechanisms such as history-aware embeddings, filter refinement based on established conversation scope, and re-ranking that boosts results aligned with the conversation direction. Fourth, memory management tradeoffs: window-based memory is simple and cheap but abruptly forgets after N turns, while summary-based memory preserves key information but risks losing details during compression.

COMMON WRONG ANSWERS: Treating every user turn as an isolated query and stuffing raw conversation history into the retrieval prompt without reformulation. Relying solely on the LLM context window to infer meaning from pronouns like that one. Ignoring topic shifts and letting stale context pollute retrieval for new subjects. Proposing only window-based memory without summarization for long conversations, which hits context limits quickly. Suggesting a single monolithic rewrite without considering latency and cost tradeoffs.

LIKELY FOLLOW-UPS: How do you detect when a user shifts topics and previous context should be discarded? What is your fallback if query rewriting hallucinates or changes user intent? How do you measure whether reformulation actually improves retrieval metrics? When would you choose vector memory versus structured storage for conversation state? How do you handle multi-user sessions where different participants introduce separate contexts?

ONE CONCRETE EXAMPLE: A customer asks What is the return policy for electronics? then follows up with What about after that? and finally Can I get store credit instead? A naive system retrieves nothing for after that because the pronoun lacks meaning alone. A conversational system reformulates the second query to What is the return policy for electronics after 30 days? using LLM-based rewriting with coreference resolution. It maintains a short-term memory of the last 5 turns so the third query inherits the electronics topic, and it uses filter refinement to scope retrieval to the returns document set rather than the entire knowledge base.

Source: zenvanriel.com