Trade-offs between dense and sparse retrieval in RAG?

### What this tests This question probes your understanding of the fundamental information retrieval layer in RAG. It's not just about knowing the definitions; it's about your ability to analyze the engineering trade-offs: accuracy, latency, cost, and operational complexity. The interviewer wants to see if you can choose the right tool for the job, or know when to combine them.

### A good answer covers * **Core Distinction**: Clearly define the two approaches. Dense retrieval uses embeddings to find semantically similar text, even with different words (e.g., 'cold weather gear' finds 'winter coats'). Sparse retrieval uses inverted indices and algorithms like TF-IDF or BM25 to match exact keywords and weigh their importance. * **Accuracy & Performance Trade-offs**: * **Dense**: Superior for conceptual, conversational, or poorly-defined queries. It handles synonyms and paraphrasing well. It fails on queries requiring exact keyword matches, like product SKUs, error codes, or specific acronyms that get lost in the semantic embedding. * **Sparse**: Excels at keyword-heavy queries, identifiers, and acronyms. It's highly interpretable and fast. It fails when the query uses different vocabulary than the documents (e.g., 'car' won't match 'automobile'). * **Operational & Cost Trade-offs**: * **Dense**: High upfront and operational cost. Requires a powerful embedding model, significant compute to embed the entire corpus, and specialized vector databases (e.g., Pinecone, Weaviate) which can be expensive. Storage for vectors is large (e.g., 1M documents x 1536-dim vectors x 4 bytes/dim ≈ 6.1 GB). * **Sparse**: Low cost and mature technology. Indexing is computationally cheaper and storage is generally smaller. It can be run on standard, well-understood systems like Elasticsearch or OpenSearch. * **The Hybrid Solution (Advanced Answer)**: The best approach is often hybrid search. This involves running both dense and sparse queries in parallel and merging the results. Mentioning techniques like Reciprocal Rank Fusion (RRF) to combine the ranked lists demonstrates senior-level awareness of production-grade solutions.

### Common wrong answers * **"Dense is always better because it's AI."**: A massive red flag. This shows a superficial, buzzword-driven understanding and ignores the fundamental trade-offs. * **Describing only one method**: Failing to compare and contrast shows a lack of breadth. The question is explicitly about the trade-offs. * **Ignoring cost and latency**: A senior engineer must think about the practical implications. An answer that only covers retrieval accuracy is incomplete.

### Follow-up the interviewer might ask * "How would you implement and tune a hybrid search system? How do you decide the weighting between the sparse and dense scores?" * "Describe a scenario where you would *only* use sparse retrieval, even with unlimited budget."

### One concrete example Imagine an e-commerce site's help documentation RAG bot: * **User Query 1**: "My order is delayed, what do I do?" * **Dense** excels here, matching the concept of 'delayed order' to documents about 'late shipments' or 'tracking issues'. * **User Query 2**: "What is the return policy for product SKU `ABC-123-XYZ`?" * **Sparse** is critical. It will precisely find the SKU `ABC-123-XYZ`. A dense model might gloss over this specific identifier, retrieving documents about return policies in general. * **Hybrid search** handles both cases reliably by combining the semantic power of dense search with the precision of sparse search.