Identify RAG latency bottlenecks and propose optimizations

WHAT THIS TESTS: This question evaluates whether you can treat a production RAG pipeline as a distributed system with distinct latency budgets rather than a monolithic black box. Interviewers want to see you map wall-clock time to specific components, justify trade-offs between retrieval quality and speed, and propose optimizations that are concrete and measurable rather than generic.

A GOOD ANSWER COVERS: A strong answer walks through the pipeline in chronological order and assigns bottlenecks and fixes to each stage. First, embedding inference: mention that model size and dimensionality directly impact serialization and compute, so candidates should suggest caching frequent query embeddings, using lighter models like E5 or BGE, and batching requests. Second, vector search: identify HNSW index parameters as a tuning lever, advocate for hybrid dense-plus-sparse retrieval with BM25 to reduce the candidate set before vector scoring, and mention hierarchical clustering or multi-level indexing for knowledge bases beyond millions of documents. Third, chunk retrieval and network overhead: flag oversized chunks and poor chunking as silent latency killers, propose semantic chunking with 10 to 20 percent overlap to preserve context without inflating payload size, and add metadata pre-filtering to shrink the retrieved set. Fourth, generation: note that autoregressive decoding dominates tail latency, so propose load balancing across endpoints, intelligent caching of similar responses, circuit breakers and fallbacks for external APIs, and model distillation or speculative decoding to cut token-level latency. Fifth, cross-cutting concerns: mention API orchestration overhead, duplicate detection in preprocessing pipelines, and the cost of deserializing large context windows into the prompt.

COMMON WRONG ANSWERS: The biggest red flag is treating the vector database as a magic box and simply saying add more GPUs or scale horizontally without naming index types or chunking strategies. Another weak pattern is focusing exclusively on LLM generation while ignoring that retrieval and embedding can consume 30 to 50 percent of total latency in large knowledge bases. Suggesting fixed-size chunking without overlap or failing to mention hybrid search pruning also signals prototype-level experience rather than production scaling.

LIKELY FOLLOW-UPS: Expect the interviewer to ask how you would instrument each stage to measure its exact latency contribution, what you would cut first if the total budget drops to 200 milliseconds, how you prevent relevance degradation when moving to approximate search or smaller models, and whether you would choose an open-source model like Llama 2 or a proprietary API for high-volume generation.

ONE CONCRETE EXAMPLE: Imagine an enterprise RAG system serving thousands of concurrent users over a five-million-document knowledge base. You set HNSW ef_construction and ef_search based on a 95th percentile recall target, implement hybrid retrieval that uses BM25 to pre-filter to the top 100 candidates before vector re-ranking, chunk documents semantically with a 15 percent overlap ratio, cache the top 20 percent of query embeddings in Redis, and route generation through a distilled 7-billion-parameter model with a circuit breaker that falls back to a smaller endpoint if latency exceeds 500 milliseconds.