All articles
AI & LLMsNovember 5, 2026 6 min read

Model Routing in Production: How to Stop Paying Opus Prices for Haiku Work

Most teams pick one model and call it a day. That's how you end up paying $15 per million tokens to classify a support ticket. Here's how to route requests between Claude, GPT, and Gemini without turning your stack into a science project.

Most teams pick one model, wire it into every endpoint, and move on. Six months later they're staring at a bill where 70% of the spend went to classifying support tickets with a frontier model. Model routing — sending each request to the cheapest model that can actually do the job — is the single biggest cost lever we've pulled on client projects this year, and it's also the one teams most consistently botch.

This is a working engineer's guide to doing it without over-engineering.

Why a single-model stack stops making sense

When you're prototyping, one model is correct. You want to isolate the variable of "does the idea work" from "which model should we use." Reach for Claude Sonnet or GPT-4-class and keep moving.

The problem shows up around the time you have real traffic:

  • A huge chunk of your requests are trivial: classify, extract, rewrite, summarize a short span.
  • A smaller slice needs real reasoning: multi-step planning, code generation, tricky RAG synthesis.
  • A tiny slice needs the best model on earth: legal reasoning, hard math, agentic tool use with long horizons.

If you pay frontier prices for all three buckets, you're lighting money on fire. Anthropic's own pricing page shows Haiku at a fraction of Sonnet, and Sonnet at a fraction of Opus; the same shape holds for OpenAI's mini/nano tiers and Google's Flash vs Pro split. The gap between the cheapest and most expensive model in a vendor's lineup is typically 20–50x on input tokens. That's not a rounding error.

The three routing strategies that actually work

There are a lot of clever routing papers. In production, three patterns cover almost everything.

1. Static routing by task type

The simplest and often the best. You know, at code time, what kind of task each endpoint is doing. A classifier endpoint goes to Haiku or GPT-nano. A code-gen endpoint goes to Sonnet or GPT-class. A "deep research agent" goes to Opus or Gemini Pro.

This is boring and it works. Do this first.

type TaskType = 'classify' | 'extract' | 'summarize' | 'generate' | 'reason';

const MODEL_FOR_TASK: Record<TaskType, string> = {
  classify:  'claude-haiku-4',
  extract:   'claude-haiku-4',
  summarize: 'gpt-5-mini',
  generate:  'claude-sonnet-4',
  reason:    'claude-opus-4',
};

export function pickModel(task: TaskType): string {
  return MODEL_FOR_TASK[task];
}

That's the whole router. If you never grow past this, you're fine.

2. Cascade routing (cheap-first with escalation)

When you can't predict difficulty ahead of time — think open-ended user queries in a chat product — run the cheap model first, inspect its output, and escalate only if it fails a quality check.

The quality check is where teams get this wrong. "Ask the model if it's confident" doesn't work; models are famously bad at self-calibration. Better signals:

  • Structured output validation: did the JSON parse? Did required fields get filled?
  • Grounding checks: does the answer cite retrieved chunks? Do the citations exist?
  • A tiny judge model trained or prompted to score "is this answer complete given the question."
  • Explicit refusal detection: the model says "I don't have enough information" — escalate.
async function answer(query: string, context: Chunk[]) {
  const cheap = await callModel('gemini-flash', { query, context });
  if (passesQualityGate(cheap, context)) return cheap;

  // escalate
  return callModel('claude-sonnet-4', { query, context });
}

In our experience, a well-tuned cascade lets 60–80% of traffic resolve on the cheap tier for conversational RAG. Your numbers will vary wildly based on query distribution — measure before you believe anyone's headline number, including ours.

3. Classifier-based routing

Use a tiny model (or even a logistic regression on embeddings) to classify the incoming query's difficulty, then route. This is more work than cascade routing and usually not worth it unless you're at serious scale. Skip until you've exhausted the first two.

Picking which vendor for which tier

We're not going to tell you Claude is better than GPT-5 or Gemini. That changes every quarter and it depends on your task. But here's the shape of the tradeoffs we see repeatedly on real projects:

  • Cheap tier (classify, extract, short summarize): Haiku, GPT-5-mini/nano, and Gemini Flash are all close enough that latency and your existing SDK investment matter more than raw quality. Gemini Flash tends to win on raw throughput; Haiku tends to win on instruction-following consistency.
  • Mid tier (generation, RAG synthesis, moderate reasoning): this is the most competitive bracket. Claude Sonnet and GPT-5 class models trade blows. Pick based on your eval suite, not vibes.
  • Top tier (agents, hard reasoning, long-horizon planning): Opus and the frontier GPT models are what we reach for when a task genuinely needs it. Gemini Pro with its long context is the pick when the task is "reason over this 400-page document" rather than "think very hard about a short problem."

Multi-vendor routing adds real operational cost: three SDKs, three sets of rate limits, three billing dashboards. Don't do it until you've proven your single-vendor routing is maxed out. When you do go multi-vendor, do it because one vendor is meaningfully better at a specific task type — not for abstract resilience reasons.

The guardrails you need before you ship a router

Fallbacks that don't silently degrade

If your cheap model is down, falling back to the expensive model is fine — once. Doing it 50,000 times because your provider is having a bad hour will wreck your margin. Track fallback rate as a first-class metric and alert when it spikes.

async function callWithFallback(primary: string, fallback: string, req: Req) {
  try {
    return await callModel(primary, req);
  } catch (e) {
    metrics.increment('llm.fallback', { primary, fallback });
    if (isRateLimit(e) || isServerError(e)) {
      return callModel(fallback, req);
    }
    throw e;
  }
}

Per-tenant cost ceilings

If you're B2B, one customer's runaway integration can eat your monthly budget by Wednesday. Enforce per-tenant token ceilings at the router layer, not in a nightly cron. Reject or downgrade requests when a tenant blows past their cap for the hour.

Eval parity across tiers

Every model in your router needs to pass your eval suite for the tasks it's assigned. When you swap Haiku for a newer cheap model, re-run the suite. We've seen teams silently regress by 15 points on extraction accuracy because someone bumped a model string in config without re-running evals.

If you don't have an eval suite yet, start there before you start routing. We've written about building evals your team will actually maintain — the short version is: 50 real examples beats 5,000 synthetic ones.

The war story: when routing backfired

On one project we got cute and routed a customer support summarization task to the cheap tier because the eval suite said it was fine. It was fine — for English. The eval set was 95% English. When a wave of German and Polish tickets came in, the cheap model started producing summaries that were technically correct but missed nuance a human agent would catch, and CSAT dipped before anyone noticed.

Two lessons we paid for:

  1. Your eval set must mirror your actual traffic distribution. Language, length, domain, difficulty. All of it.
  2. Monitor downstream business metrics, not just LLM metrics. Token count and latency looked perfect the whole time.

We fixed it by segmenting the router: English stayed on the cheap tier, non-English escalated to Sonnet. Cost went up 8%, CSAT recovered within a week.

Where we'd start

If you're staring at a single-model stack and a growing bill, do this in order:

  1. Instrument first. Log task type, input/output tokens, and cost per request. You can't route what you can't see.
  2. Static route by endpoint. Move obvious cheap-tier tasks (classification, extraction, short summarization) to the cheapest model from your current vendor. Don't add a second vendor yet.
  3. Run your eval suite against the cheap model for every task you moved. Fix or revert anything that regresses meaningfully.
  4. Add a cascade for the one or two endpoints where request difficulty is genuinely unpredictable. Build a real quality gate, not a self-confidence prompt.
  5. Only then, consider multi-vendor routing — and only for tasks where you have evidence another vendor is materially better.

Skip the clever stuff until the boring stuff is done. A static router with good evals will beat a sophisticated classifier-based router with bad evals every single time.

#AI#LLMs#RAG#Cost Optimization#Architecture

Want a team like ours?

72Technologies builds production software for the kind of teams who actually read this blog.

Start a project