Why RAG persists despite million-token context windows

WHAT THIS TESTS Whether you can separate can the model technically accept N tokens from will it use them reliably and economically in production.

A GOOD ANSWER COVERS COST AND LATENCY. Prefill cost and latency grow with the number of input tokens, so feeding a full corpus on every request is far more expensive and slower than retrieving a few relevant chunks. RELIABILITY. Long-context models show lost-in-the-middle behavior: recall is strong for information at the start and end of the window but degrades for facts buried in the middle, so a needle in a large haystack can be missed even when present. RAG sidesteps this by retrieving the relevant passages and placing them in high-attention positions. CAPABILITIES RAG ADDS INDEPENDENTLY. Freshness, since you can update the index without retraining; access control, since retrieval can filter by user permissions before anything reaches the model; attribution, since you return the source chunks for citations; and scale beyond any finite window across millions of documents. RAG and long context are also complementary: retrieve a focused set, then let a long window reason over it.

COMMON WRONG ANSWERS RAG is dead because the window is huge. Assuming uniform attention across the whole context. Ignoring per-token cost and latency. Forgetting access control, freshness, and citation needs that a raw window cannot satisfy.

LIKELY FOLLOW-UPS What is lost-in-the-middle and what causes it? How do cost and latency scale with context length? When would you actually prefer long context over RAG? How do you combine them?

ONE CONCRETE EXAMPLE A support assistant must answer from a 50-million-token knowledge base that changes daily and is permission-scoped per customer. You cannot fit it in any window, you must reflect edits immediately, and you must not show one tenant another tenant's data. RAG retrieves the handful of relevant, permission-filtered chunks, returns citations, and stays cheap per query, while a long-context-only approach would be impossible to fit, stale, and unable to enforce access control.