Advanced RAG: Long-Context, Compression & Self-Correction

Long-Context vs Retrieval

Models now accept 200K–2M token context windows. A reasonable question: if you can fit the whole knowledge base in the prompt, why retrieve at all? The honest answer is that long context and RAG are complements, and knowing which to reach for is an advanced skill.

What Long Context Is Good At

• Whole-document reasoning. When the answer requires synthesizing across an entire contract or codebase that fits in the window, dumping it in beats chunked retrieval, which can sever the connections.
• Low-volume, high-value queries. If you ask a handful of questions against one 100-page document, the simplicity of "just put it in the prompt" is worth it.

Why Retrieval Still Wins at Scale

• Cost. You pay per input token on every call. A 500K-token prompt at scale is brutally expensive; retrieving 5 chunks of 500 tokens is ~2.5K tokens.
• Latency. Time-to-first-token grows with input length. A multi-hundred-K prompt is slow before the model says a word.
• Corpus size. A company wiki is hundreds of millions of tokens. It will never fit, no matter how big the window gets.
• "Lost in the middle" doesn't disappear. Even with a 1M-token window, models attend less to the middle of a long context. Filling the window does not mean the model uses all of it.

The Rule of Thumb

Use long context to reason over a small, already-selected set of documents. Use retrieval to select from a large corpus. The strongest systems do both: retrieve a generous candidate set, then let a long-context model reason over it.

Retrieval is a selection mechanism; long context is a reasoning surface. They sit at different stages of the same pipeline, not in competition.