Long-Context Windows vs RAG: When 1M Tokens Actually Beats Retrieval

Gemini 2.x and Claude both advertise context windows measured in millions of tokens. Every quarter, someone on a client call asks the same question: "If the model can read the whole codebase, why are we still building a RAG pipeline?" It's a fair question, and the answer is more nuanced than either camp wants to admit.

This is the framework we actually use when deciding between long context, RAG, or a hybrid — and where each one quietly falls apart.

What long-context models really give you

Google's Gemini 2.5 Pro documents a 1M-token context window (with 2M in limited preview per their model card), and Anthropic's Claude Sonnet 4 offers a 1M-token beta tier. OpenAI's GPT-4.1 family also pushed to 1M. On paper, you can drop an entire product manual, a mid-sized codebase, or a quarter of support tickets into a single prompt.

What you get in practice:

• Zero retrieval infrastructure. No vector DB, no chunker, no embedding refresh job.
• Full cross-document reasoning. The model sees everything at once, so it can join facts across files without a retriever guessing which chunks matter.
• Simpler eval surface. Fewer moving parts means fewer places a regression can hide.

What you don't get, despite the marketing:

• Uniform recall across the window. Vendors publish "needle in a haystack" scores that look great, but multi-needle and reasoning-over-context benchmarks (RULER, LongBench) show meaningful degradation past a few hundred thousand tokens on most models. In our experience, accuracy on "find and combine three facts from different parts of the document" drops noticeably once you cross ~200K tokens, even on frontier models.
• Free compute. You pay per input token every call unless you're using caching (more on that below).
• Fast responses. Time-to-first-token on a 500K-token prompt is measured in seconds, not milliseconds.

The cost math nobody wants to do

Let's put rough numbers on it. Assume you have a 300K-token knowledge base and 10,000 queries a day.

Naïve long-context approach: 300K input tokens × 10,000 queries = 3B input tokens/day. Even at aggressive frontier pricing, you're spending real money — often four figures per day — just re-reading the same documents.

RAG approach: Embed once (cheap, one-time), retrieve ~4K tokens of relevant chunks per query. That's 40M input tokens/day. Roughly two orders of magnitude cheaper.

Long context with caching: Gemini's context caching and Anthropic's prompt caching both let you pay a reduced rate on cached prefixes (Anthropic's docs quote up to 90% savings on cache reads; Gemini's caching has a storage fee plus reduced input pricing). If your 300K-token corpus is stable and hit rate is high, cached long context can get within striking distance of RAG on cost — but only if you're actually re-hitting the same prefix.

The honest heuristic: if your context changes per user or per session, caching doesn't save you and RAG wins on cost by a wide margin.

A quick decision snippet

Here's the rough gate we use in code review when someone proposes "just put it all in context":

def should_use_long_context(corpus_tokens, queries_per_day,
                            corpus_stability, cross_doc_reasoning):
    # corpus_stability: fraction of queries that hit the same prefix
    # cross_doc_reasoning: bool, does the task need many facts joined?

    if corpus_tokens > 500_000 and not cross_doc_reasoning:
        return "RAG"  # recall degrades, no reasoning payoff

    if corpus_stability > 0.8 and corpus_tokens < 400_000:
        return "long_context_with_cache"

    if cross_doc_reasoning and corpus_tokens < 300_000:
        return "long_context"  # RAG will miss the joins

    if queries_per_day > 5_000 and corpus_stability < 0.3:
        return "RAG"  # per-query cost dominates

    return "hybrid"

This isn't a law of physics. It's a starting point that has saved us from a few expensive rewrites.

Where long context genuinely wins

There are workloads where RAG is the wrong tool and we stop pretending otherwise:

1. Code understanding across a repo

When a task requires reasoning about how three services call each other, a retriever fed on symbol-level chunks will miss the graph. Loading the relevant service directories directly — even 200–400K tokens of them — produces materially better refactors and bug analyses. Claude in particular has been strong here in our testing.

2. Legal, compliance, and contract review

A 90-page master services agreement isn't a retrieval problem. It's a "read the whole thing and find the interactions between clauses" problem. Chunking a contract is how you miss the indemnity clause that references a schedule that references an appendix.

3. Long transcripts and meeting analysis

A 4-hour sales call transcript is maybe 60K tokens. Just put it in the window. Building RAG for something you'll query twice is engineering theatre.

4. One-shot document synthesis

Generating an audit report from a fixed set of source documents, once, is a job for long context. You're not amortising retrieval infrastructure across millions of queries — you're doing it once and shipping the artefact.

Where RAG still wins, decisively

1. Corpora bigger than the window

Obvious but worth saying: if your knowledge base is 50M tokens, no window saves you. You need retrieval.

2. Freshness-sensitive data

A support bot answering questions about product state, inventory, or ticket status needs current data on every query. Stuffing yesterday's snapshot into context is a bug factory.

3. Per-user or per-tenant data

Multi-tenant SaaS with any kind of access control is a RAG problem. You cannot dump every customer's data into a shared context, and per-tenant caching gets ugly fast.

4. High-QPS consumer apps

Latency matters. A well-tuned retrieval step (embedding + vector search + rerank) runs in a couple hundred milliseconds. A million-token prompt does not.

The hybrid pattern we actually ship

Most real systems end up somewhere in the middle. The pattern that has held up across the last dozen or so LLM projects we've delivered:

1. Retrieve broadly. Pull 30–80 candidate chunks with hybrid search (BM25 + dense embeddings).
2. Rerank. Trim to the top 10–20 with a cross-encoder or an LLM reranker.
3. Pack a medium context. Assemble 30–80K tokens of retrieved material, plus stable system context (schemas, style guides, tool definitions) that can be cached.
4. Let the model reason across it. At 30–80K tokens, recall is still strong on frontier models, and you're paying a fraction of the full-corpus cost.

This gives you most of the cross-document reasoning benefit of long context, without paying to re-read the world on every query. The cached stable prefix is where prompt caching earns its keep.

Eval implications

Switching architectures without changing evals is how you ship regressions. When we move a workload between RAG and long context, we re-run:

• Recall@k on gold questions. Did we lose the ability to find the right facts?
• Multi-hop reasoning tests. Does the model still join facts across sources?
• Latency at P50 and P95. Long context blows up the tail.
• Cost per resolved query. Not per token — per successful outcome, which is the only number that matters.

We've written before about building agent regression tests in CI; the same discipline applies here. Architecture changes are eval events.

The trap: benchmarks lie about your workload

Vendor benchmarks measure needle-in-haystack retrieval on synthetic data. Your workload is probably closer to "given these five overlapping policy documents, answer a question that requires reconciling two of them." That is a much harder task, and it's where long-context models degrade first.

Before you commit to an architecture, build a small eval set — 50 to 200 real questions with known answers — and run both approaches. We have watched teams pick long context because a keynote said 1M tokens, then quietly rebuild as RAG six months later when accuracy on real questions stalled at 70%.

Where we'd start

If you're staring at this decision right now:

• Measure your corpus size, query volume, and how often the same context is reused. Those three numbers decide most of the argument.
• Build a 100-question eval set from actual user queries before you write any retrieval code.
• Prototype both a naive long-context version and a basic RAG version. Neither should take more than a few days.
• Compare accuracy, P95 latency, and cost per successful query. Pick the winner, then invest in making it production-grade.
• Assume you'll end up with the hybrid pattern eventually. Design your data layer so you can add retrieval later without a rewrite.

Long context is a genuine capability jump, not a marketing artefact. It's just not a replacement for thinking about your data — and the teams shipping the best LLM products in 2026 are the ones who still treat retrieval, caching, and context assembly as first-class engineering problems. If you want a second opinion on your architecture, our AI engineering team does this for a living.