Context Window Budgeting: Stop Paying for Tokens You Don't Need

A million-token context window sounds like a cheat code until the invoice arrives and latency creeps past six seconds. We've watched teams stuff entire wikis into every request, then wonder why answers got worse and slower at the same time. Context is a budget, not a bucket — and treating it that way is one of the highest-leverage changes you can make to a production LLM app.

Why big context windows made things worse for us

When Anthropic shipped 200K context on Claude and Google pushed Gemini past a million tokens, a lot of engineering teams — ours included, briefly — replaced retrieval logic with "just send everything." It felt like progress. It wasn't.

Three things went sideways:

• Cost per request quietly 10x'd. Input tokens are cheaper than output, but not free. A 300K-token prompt hit twice per user session adds up fast.
• Latency became unpredictable. Time-to-first-token scales with input size on most providers. Users notice.
• Answer quality dropped on some tasks. The "lost in the middle" effect is real. Anthropic and Google both document degradation on needle-in-a-haystack style retrieval as context grows, especially with multiple needles or reasoning across them (see Anthropic's long-context evaluations and Google's Gemini technical reports).

The fix isn't to abandon long context. It's to spend it deliberately.

The four budgets inside every prompt

We carve every request into four named budgets. Each has a hard ceiling, and if a component wants more it has to justify it with an eval.

1. System / instructions — role, format rules, tool descriptions, guardrails
2. Retrieved context — RAG chunks, docs, code snippets
3. Conversation history — prior turns, tool call results
4. Working room — headroom for the model's reasoning + output

For a typical support-assistant style app on Claude Sonnet or GPT-4.1, our starting split looks roughly like this:

Total budget:        16,000 tokens
├── System:           1,500  (9%)
├── RAG context:      6,000  (38%)
├── History:          4,000  (25%)
└── Working room:     4,500  (28%)  <- includes output

Those percentages aren't sacred. A code-review agent will push RAG and working room up and history down. A long-running research agent inverts that. The discipline is that somebody owns each number and changes require a diff.

Why "working room" matters more than people think

If you're using a reasoning model (o-series, Claude with extended thinking, Gemini thinking mode), the model burns tokens on internal reasoning that you're billed for and that count against the output limit. We've seen teams set max_tokens: 1024 on a reasoning model and get truncated JSON because the model spent 900 tokens thinking. Leave real headroom.

Enforcing the budget in code

Budgets that live in a Notion doc get violated by Friday. Put them in code.

interface PromptBudget {
  system: number;
  rag: number;
  history: number;
  working: number;
}

const BUDGETS: Record<string, PromptBudget> = {
  "support-agent": { system: 1500, rag: 6000, history: 4000, working: 4500 },
  "code-review":   { system: 2000, rag: 10000, history: 1000, working: 8000 },
};

function assemblePrompt(
  task: string,
  parts: { system: string; rag: string[]; history: Msg[] },
  countTokens: (s: string) => number,
) {
  const b = BUDGETS[task];

  const system = truncate(parts.system, b.system, countTokens);
  const rag = packChunks(parts.rag, b.rag, countTokens);
  const history = trimHistory(parts.history, b.history, countTokens);

  const used = countTokens(system) + countTokens(rag) +
               history.reduce((n, m) => n + countTokens(m.content), 0);

  if (used > b.system + b.rag + b.history) {
    throw new BudgetExceededError({ task, used, budget: b });
  }

  return { system, rag, history, maxTokens: b.working };
}

A few things this buys you:

• Predictable costs. You can price a request before you send it.
• Failing loud. A regression that suddenly balloons RAG output throws instead of quietly costing $400/day.
• Cleaner evals. You can A/B different splits without touching business logic.

Picking the right token counter

Don't estimate tokens with text.length / 4. It's wrong enough at scale to matter, especially with code, CJK text, or heavy markdown.

• OpenAI: use tiktoken with the model-specific encoding (o200k_base for GPT-4o/4.1 family).
• Anthropic: use the count_tokens endpoint — Claude's tokenizer isn't public, and approximations drift.
• Gemini: the SDK exposes countTokens on the model client.

Cache counts per chunk. Token counting is cheap but not free, and RAG pipelines call it a lot.

RAG-specific budgeting patterns

Most of our budget fights happen inside the RAG slice. A few patterns that consistently pay off:

Rank, then pack — don't just top-K

Top-K by similarity gives you K chunks of wildly varying size. Instead, rank by score, then greedily pack into the RAG budget until the next chunk would overflow. You end up with more small, relevant chunks or fewer big ones — whichever the retriever thinks is better.

def pack_chunks(ranked, budget, count):
    used, kept = 0, []
    for chunk in ranked:
        cost = count(chunk.text)
        if used + cost > budget:
            continue  # try smaller chunks further down
        kept.append(chunk)
        used += cost
    return kept

Summarize the long tail

When the top 3 chunks are strong but chunks 4–10 are marginal, don't waste tokens including them verbatim. Run a cheap model (Haiku, GPT-4.1-mini, Flash) to produce a 200-token synthesis of the tail and include that. Quality on ambiguous queries goes up; token count goes down.

Kill boilerplate at index time

We once shaved 22% off average RAG token count by stripping repeated headers, footers, and "Was this article helpful?" widgets from indexed docs. That's free money you're paying for on every retrieval.

Conversation history: the silent budget killer

History is where most apps leak tokens. A 20-turn conversation with tool calls can easily hit 30K tokens of history alone.

Three tactics, roughly in order of ROI:

1. Drop tool call payloads once consumed. After the model has read a tool result and moved on, the raw JSON usually doesn't need to sit in history. Replace with a one-line summary: [searched knowledge base for "refund policy": 3 results used].
2. Rolling summarization. Every N turns, summarize turns 1..N-4 into a compact recap and keep the last 4 verbatim. Anthropic and OpenAI both recommend variants of this in their docs.
3. Prompt caching where available. If your system prompt and long RAG context are stable across turns, providers like Anthropic and OpenAI offer prompt caching that dramatically cuts input cost for the cached prefix. It doesn't shrink the context — it just makes it cheaper. Combine with, don't substitute for, budgeting.

Model choice changes the budget math

A 6,000-token RAG slice isn't the same decision on every model.

• Claude Sonnet / Opus: strong at long context, tolerates unstructured RAG dumps better, but input tokens are pricier than the cheap tier.
• GPT-4.1 / 4o family: good instruction-following on tightly structured contexts; we tend to be stricter with formatting when using them.
• Gemini 2.x Pro / Flash: genuinely useful huge contexts, but our evals show quality varies more with position of the key info. Put the important stuff near the end.
• Small models (Haiku, 4.1-mini, Flash): the right home for summarization, routing, and RAG tail-compression steps.

We usually run a cascade: cheap model does query rewriting and reranking inside the RAG pipeline, the expensive model gets a lean, well-packed prompt. If you want more on that pattern, our team writes about related tradeoffs on the 72Technologies blog.

Evals that catch budget regressions

Budgets without evals are just vibes. What we track per task:

• Answer quality on a fixed golden set (LLM-judged, spot-checked by humans).
• P50 and P95 input tokens. Sudden shifts mean something upstream changed.
• Cost per successful task, not per request. A cheap prompt that fails and retries isn't cheap.
• Time to first token.

When we tighten a budget, we require: quality doesn't drop more than 1 point on the golden set, and cost-per-task drops at least 10%. Otherwise the change reverts.

Where we'd start

If you're staring at an LLM app that's slower and more expensive than it should be, don't refactor the whole thing. Do this, in order:

1. Add real token counting and log input/output tokens per request, tagged by task.
2. Look at your P95 prompt for one week. You'll find one component eating 60%+ of the budget. That's your target.
3. Write down explicit budgets for that task's four slices and enforce them in code.
4. Add prompt caching for stable prefixes if your provider supports it.
5. Only then start moving parts of the pipeline to cheaper models.

Every team we've seen do this in that order has come out with lower cost, faster responses, and — usually — better answers. Context is a resource. Spend it like one.