Cascading Models: Route Cheap First, Escalate Only When You Must

Most teams pick one model and pay for it on every request. That's fine until your bill hits five figures and you notice 80% of prompts are trivial classification or short rewrites a model a tenth of the price could handle. Cascading — try cheap first, escalate on doubt — is one of the highest-leverage cost patterns we deploy, and it barely gets discussed outside vendor blog posts.

What cascading actually is

A model cascade is a routing pattern where a request hits a small, cheap model first. If the output passes a confidence check, you return it. If not, you escalate to a stronger, more expensive model. Optionally you add a third tier — a frontier model like Claude Opus or GPT-4.1 — for the hard 5%.

This is different from a router that picks one model up-front based on prompt classification. A router guesses which model will be needed. A cascade observes whether the cheap model succeeded and only pays for more capacity when it didn't.

The economics are brutal in your favour. If Haiku handles 70% of traffic cleanly, Sonnet handles the next 25%, and Opus mops up the last 5%, your blended cost per request often lands 3–5x lower than sending everything to Sonnet, and 10x+ lower than everything-to-Opus. In our experience across support-triage and document-extraction workloads, blended savings sit in the 40–70% range without measurable quality loss on offline evals.

Why not just use the cheap model everywhere?

Because the cheap model fails on the long tail, and the long tail is where users get angry. A cascade lets you keep frontier-model quality on the hard cases while paying small-model prices on the easy ones. You're not choosing between cost and quality — you're choosing which requests deserve which tier.

The three tiers we actually use

We usually stand up cascades with three tiers. Two is often enough, but three gives you room to breathe.

• Tier 1 (fast/cheap): Claude Haiku 3.5, GPT-4o mini, or Gemini 2.0 Flash. Sub-second latency, pennies per million tokens. Handles the clear-cut majority.
• Tier 2 (workhorse): Claude Sonnet 4, GPT-4.1, or Gemini 2.5 Pro. This is where most production teams already sit. Good at reasoning, tool use, and structured outputs.
• Tier 3 (frontier): Claude Opus 4 or GPT-4.1 with extended thinking. Reserved for genuinely hard tasks — ambiguous extraction, multi-step reasoning, safety-sensitive edge cases.

Cross-vendor cascades are fine and often desirable (you get some resilience against a single provider having a bad day), but they complicate prompt engineering because system prompts and tool-use semantics differ. Start with one vendor's ladder and add cross-vendor fallbacks later.

The confidence check is the whole game

The entire pattern lives or dies on how well you decide this cheap answer isn't good enough, escalate. Get this wrong and you either escalate everything (no savings) or escalate nothing (quality tanks).

Here are the signals that actually work:

1. Self-reported confidence

Ask the small model to return a confidence score alongside its answer. Simple, cheap, surprisingly effective for classification and extraction:

response_schema = {
    "type": "object",
    "properties": {
        "answer": {"type": "string"},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1},
        "reasoning": {"type": "string"}
    },
    "required": ["answer", "confidence"]
}

result = call_haiku(prompt, response_schema)
if result["confidence"] < 0.75:
    result = call_sonnet(prompt, response_schema)

Calibrate the threshold on real data. Small models tend to be overconfident, so 0.9 from Haiku is not the same as 0.9 from Opus. We usually find the right threshold by plotting confidence against eval accuracy on a few hundred graded examples.

2. Structural validation

If the task has a machine-checkable output — valid JSON matching a schema, an SQL query that parses, a citation that exists in the source documents — validate it. Failed validation is a hard escalation signal. This is often stronger than self-reported confidence because it's objective.

3. Logprobs and answer-token probability

OpenAI exposes logprobs; Anthropic doesn't at the time of writing (check the Anthropic API docs for current status). Where available, low token probability on the answer span is a good escalation signal, especially for classification.

4. A tiny judge model

Run a second cheap call — a different small model, or the same one with a critic prompt — that scores the first answer. This adds latency and a bit of cost but catches confidence-calibration failures. Only worth it when self-reporting proves unreliable on your task.

A working sketch

Here's the shape of a cascade in production code. Real implementations add retries, timeouts, telemetry, and per-tenant overrides, but the core is small:

from dataclasses import dataclass
from typing import Callable, Optional

@dataclass
class Tier:
    name: str
    call: Callable
    escalate_if: Callable  # returns True if we should escalate

def run_cascade(prompt: str, tiers: list[Tier]) -> dict:
    last_result = None
    for tier in tiers:
        result = tier.call(prompt)
        result["_tier"] = tier.name
        last_result = result
        if not tier.escalate_if(result):
            return result
    return last_result  # top tier answer, even if it also failed the check

tiers = [
    Tier("haiku", call_haiku,
         lambda r: r["confidence"] < 0.8 or not r["valid_json"]),
    Tier("sonnet", call_sonnet,
         lambda r: r["confidence"] < 0.6),
    Tier("opus", call_opus,
         lambda _: False),  # never escalate past the top
]

answer = run_cascade(user_prompt, tiers)

Instrument every escalation. You need to know, per tier, what percent of traffic is passing through, what the escalation rate is, and — critically — whether the top-tier answers actually differ from the tier-below answers. If Opus and Sonnet agree 98% of the time on escalated cases, your Sonnet threshold is too aggressive.

Evals are non-negotiable

You cannot ship a cascade without an eval set. The whole pattern trades quality on some subset of requests for cost, and you need to prove that subset is small and unimportant.

Minimum viable eval loop:

1. Collect 200–500 real requests, graded by a human or a strong judge model.
2. Run each request through Tier 1 only, Tier 2 only, and the full cascade.
3. Compare accuracy and cost. The cascade should be within 1–2 percentage points of Tier 2 alone while costing significantly less.
4. Re-run this every time you change thresholds, prompts, or model versions.

Model versions matter more than people expect. When Anthropic or OpenAI ships a point-release, cheap-model calibration shifts and your thresholds drift. Bake the eval into CI. We covered eval harness design in more depth on the 72Technologies blog.

The escalation-rate alarm

Set a monitor on tier-1 pass rate. If it drops from 70% to 45% overnight, something changed — an upstream prompt template, a new customer segment, or a model version bump — and your bill is about to spike. Escalation rate is a leading indicator of cost incidents.

Where cascades don't work

Be honest about the failure modes:

• Latency-sensitive paths. Every escalation adds a round-trip. If you need P99 under 800ms, a three-tier cascade with sequential calls will hurt. Consider parallel speculative calls (fire Tier 1 and Tier 2 simultaneously, use Tier 2 only if Tier 1 fails the check) — you spend more, but latency stays flat.
• Long-context tasks. If your prompt is 100k tokens of RAG context, the cheap-model call isn't cheap any more. Cascades shine on short-to-medium prompts.
• Agentic loops. In a multi-step agent, cascading per step compounds decision errors. Better to pick the right model for the whole trajectory, or cascade only on specific tool calls (extraction, classification) rather than the planning loop.
• Streaming UX. If you stream tier-1 output to the user and then need to escalate, you've either shown them a wrong answer or held the stream. Neither is good. Cascades work best on non-streamed request/response endpoints.

Where we'd start

Pick one high-volume endpoint — support classification, tag extraction, summarisation, something with clear success criteria. Build a 200-example eval set graded by a strong model or a human. Wire up a two-tier cascade with self-reported confidence plus schema validation. Ship it behind a feature flag to 10% of traffic and watch the escalation rate and eval accuracy for a week.

If tier-1 is handling more than 50% of traffic and eval accuracy is within noise of the tier-2-only baseline, roll it out and move to the next endpoint. If not, tune the threshold or improve the tier-1 prompt before adding a third tier. Most of the win is in the first cascade you ship; the rest is repetition.

If you want help designing routing and eval infrastructure for a production LLM workload, our AI engineering services team does this kind of work day in, day out.