How would you architect a multi-turn conversational RAG system?
This tests memory and query reformulation design beyond single-turn RAG. A strong answer covers 5-10 turn windows, LLM-based rewriting with coreference resolution, hybrid fallbacks, and summarized memory.
What's really being asked
This question evaluates whether you understand that single-turn RAG breaks down in real conversations. Interviewers want to see you design explicit mechanisms for memory, query reformulation, and context-aware retrieval rather than hoping the LLM prompt window handles everything.
The full answer
First, conversation memory architecture with short-term windows of the last 5 to 10 turns for immediate context, summary-based compression for longer dialogues to stay within token limits, and optional long-term memory across sessions. Second, query reformulation strategies: LLM-based rewriting that replaces pronouns with referents and injects implicit context to create standalone queries, rule-based pattern matching for pronoun replacement and topic continuation for speed, and a hybrid approach that routes common cases to rules and falls back to LLM rewriting for complex references. Third, contextual retrieval mechanisms such as history-aware embeddings, filter refinement based on established conversation scope, and re-ranking that boosts results aligned with the conversation direction. Fourth, memory management tradeoffs: window-based memory is simple and cheap but abruptly forgets after N turns, while summary-based memory preserves key information but risks losing details during compression.
The mistakes people make
Treating every user turn as an isolated query and stuffing raw conversation history into the retrieval prompt without reformulation. Relying solely on the LLM context window to infer meaning from pronouns like that one. Ignoring topic shifts and letting stale context pollute retrieval for new subjects. Proposing only window-based memory without summarization for long conversations, which hits context limits quickly. Suggesting a single monolithic rewrite without considering latency and cost tradeoffs.
What usually comes next
How do you detect when a user shifts topics and previous context should be discarded? What is your fallback if query rewriting hallucinates or changes user intent? How do you measure whether reformulation actually improves retrieval metrics? When would you choose vector memory versus structured storage for conversation state? How do you handle multi-user sessions where different participants introduce separate contexts?
A concrete example
A customer asks What is the return policy for electronics? then follows up with What about after that? and finally Can I get store credit instead? A naive system retrieves nothing for after that because the pronoun lacks meaning alone. A conversational system reformulates the second query to What is the return policy for electronics after 30 days? using LLM-based rewriting with coreference resolution. It maintains a short-term memory of the last 5 turns so the third query inherits the electronics topic, and it uses filter refinement to scope retrieval to the returns document set rather than the entire knowledge base.
Interview question
Why is a hybrid query reformulation pipeline—using both rules and an LLM—preferred over an LLM-only rewriter in a multi-turn RAG system?
- a.Hybrid systems eliminate the need for conversation memory by generating fully standalone queries from single turns
- b.Rules handle high-volume simple references with low latency, while the LLM fallback handles complex coreference without rewriting every queryCorrect
- c.LLM-only approaches cannot resolve pronouns or implicit context in user queries
- d.Rule-based reformulation achieves higher accuracy than LLM rewriting on complex multi-hop conversational references
Why? this is the answer
The card describes LLM rewriting as effective for coreference resolution but advocates a hybrid to route common cases to fast rule-based matching and reserve LLM calls for complex references, managing latency and cost. Option C is tempting but wrong because the card explicitly credits LLMs with resolving pronouns; the hybrid exists for efficiency, not because LLMs lack capability.
Just read this? Test yourself on what you have been reading.
Read the original → zenvanriel.com
- #rag
- #llm
- #conversational-ai
- #retrieval
- #system-design
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on rag — each one lists the topics its interview covers.
See open roles