Token Budgets Per Request: How to Stop Your Agent From Bankrupting a Feature

The first time an agent we shipped hit a tool-calling loop in production, it burned through about a month's projected inference budget in a single afternoon. No crash, no alert — just a very patient model politely re-planning itself into oblivion. Since then, every agent we build gets a hard token budget per request before it gets a system prompt.

This is the pattern we wish someone had handed us two years ago.

Why per-request budgets, not just monthly caps

Vendor dashboards give you monthly spend caps. Those are useful for accounting and roughly useless for engineering. By the time a monthly cap trips, you've already served thousands of degraded requests, and the cap kills everyone's traffic, not just the pathological ones.

What you actually want is a budget that lives at the same scope as a user action:

• A single chat turn
• A single background job
• A single agent "task" (which may involve many tool calls)

The budget is denominated in tokens, but it maps cleanly to money. If your average blended cost on Claude Sonnet is roughly $3–$15 per million tokens depending on input/output mix (see Anthropic's pricing page for current numbers), a 50k-token budget per request puts a hard ceiling of a fraction of a cent to a few cents on that request. Multiply by your traffic, and you can finally answer the question product keeps asking: what does this feature cost per user per month?

The three failure modes a budget prevents

1. Runaway loops. An agent that keeps calling search then plan then search again because a tool returned ambiguous output.
2. Context bloat. Each turn appends the previous turn's tool outputs. Without pruning, a 10-turn conversation can carry 200k tokens of stale JSON.
3. Prompt injection amplification. A malicious document tells the model to "summarise all previous documents in full". Without a budget, it will try.

Sizing the budget

We start from unit economics, not from the model's context window. The question is not "how much can Gemini 2.5 fit?" — it's "how much can this feature afford?"

A rough working formula:

budget_tokens = (target_cost_per_request / blended_price_per_token) * safety_factor

Where safety_factor is usually 0.7 — you want headroom for the occasional legitimately hard request. If a feature needs to cost under $0.02 per call and your blended price is $8/M tokens, that's a budget around 1,750 tokens. Which sounds tiny, until you realise most chat turns fit comfortably.

We usually define three tiers per feature:

• Soft budget (e.g. 60% of hard): log a warning, emit a metric
• Hard budget (100%): stop the agent, return best-effort output
• Kill budget (150%): raise, page on-call if this fires more than N times per hour

The soft budget is what makes this useful in practice. It gives you a leading indicator that a prompt change is drifting expensive before it starts failing.

Enforcing it in code

Most SDKs don't give you a first-class "stop at N tokens" primitive across a whole agent run. You have to build it. Here's the shape we use — a thin wrapper around whichever provider we're on that tracks a running total and short-circuits the loop.

type BudgetState = {
  hardLimit: number;
  softLimit: number;
  spent: number;
  onSoft?: () => void;
};

export class TokenBudget {
  constructor(private state: BudgetState) {}

  charge(inputTokens: number, outputTokens: number) {
    this.state.spent += inputTokens + outputTokens;
    if (
      this.state.spent >= this.state.softLimit &&
      this.state.spent - inputTokens - outputTokens < this.state.softLimit
    ) {
      this.state.onSoft?.();
    }
  }

  assertUnder() {
    if (this.state.spent >= this.state.hardLimit) {
      throw new BudgetExceeded(this.state.spent, this.state.hardLimit);
    }
  }

  remaining() {
    return Math.max(0, this.state.hardLimit - this.state.spent);
  }
}

The agent loop then looks like this:

async function runAgent(task: string, budget: TokenBudget) {
  const messages: Message[] = [{ role: 'user', content: task }];

  for (let step = 0; step < MAX_STEPS; step++) {
    budget.assertUnder();

    const response = await model.complete({
      messages,
      tools,
      max_tokens: Math.min(2048, budget.remaining()),
    });

    budget.charge(response.usage.input_tokens, response.usage.output_tokens);

    if (response.stop_reason === 'end_turn') return response;

    const toolResults = await runTools(response.tool_calls);
    messages.push(response.message, ...toolResults);
  }
}

Two details that matter more than they look:

• max_tokens is clamped to budget.remaining(). This is what prevents the last call in a nearly-exhausted budget from over-shooting by 4k tokens.
• assertUnder runs before the next call, not after. Once you've spent the money, throwing doesn't refund it. Check first.

Handling the hard-limit throw gracefully

A BudgetExceeded should not look like a 500 to the user. Catch it at the feature boundary and return whatever partial state the agent had. For a research agent, that might be "here's what I found so far, I stopped before completing step 4". For a code-writing agent, it might be the current draft plus a note.

Users tolerate truncation far better than they tolerate spinners that die.

Where the budget number comes from

We don't guess. We run an eval set — the same one we use for regression testing — with cost tracking on, and look at the distribution.

Percentile	Tokens used
p50	8,400
p90	22,100
p95	31,800
p99	74,500

The hard budget goes somewhere between p95 and p99. Below p95, you'll reject too many legitimate requests. Above p99, you're paying for the tail. In our experience, p97 is a sensible starting point, then you tune based on how often the soft budget fires in prod.

Re-run this every time you change the system prompt, add a tool, or swap models. Prompt changes can shift the p95 by 30% in either direction without touching the p50.

Model choice interacts with the budget

A budget expressed in tokens hides a real tradeoff: cheaper models often need more tokens to reach the same answer, because they lean harder on tool calls and self-correction. We've seen tasks where switching from a top-tier model to a mid-tier one saved 60% on per-token price but increased token usage by 2.5x — net loss.

Some rough heuristics from projects we've shipped:

• For structured extraction with clear schemas, smaller models (Haiku, GPT-4o mini, Gemini Flash) usually stay well inside budget.
• For multi-step agents with ambiguous tools, frontier models typically finish in fewer steps and end up cheaper despite the higher sticker price.
• For long-document Q&A, prompt caching (documented by Anthropic and OpenAI) changes the maths entirely — the input side of your budget can drop by 80%+ on cache hits.

Benchmark this on your workload before committing. Vendor comparison tables won't tell you how your specific agent behaves.

What to log

Every agent run should emit, at minimum:

• budget_hard, budget_spent, budget_soft_tripped
• Tokens per step (so you can see which tool call blew up)
• stop_reason including your own budget_exceeded
• Model, prompt version, feature ID

With that in a warehouse, you can answer questions like "which prompt version has the fattest tail?" and "is feature X's p95 creeping up week over week?" Those are the questions that catch cost regressions before they show up on the invoice.

Where we'd start

If you're retrofitting this into an existing agent, do it in this order:

1. Instrument first. Add token counting to every model call and ship it to your logs. Don't enforce anything yet — just watch for a week.
2. Set the hard budget at p99 of what you observe. This will almost never fire, but it'll catch the truly pathological cases immediately.
3. Add the soft budget at p90 and wire it to a dashboard. Watch which features drift.
4. Tighten from there. Once you trust the metric, pull the hard budget down toward p97, then p95, and negotiate with product about what to do when it trips.

The goal isn't to make the model cheap. It's to make the worst case predictable, so you can ship features with confidence instead of hoping nobody asks the agent to do something silly on a Friday afternoon. If you want a hand pressure-testing this on a real workload, our team does this work as part of our AI engineering services.