Fallback Chains for LLM APIs: Designing for Provider Outages Without Melting Your Latency Budget

Every team running LLMs in production eventually has the same bad afternoon: a provider goes sideways, your queue backs up, and your on-call is refreshing a status page while customers refresh your app. Multi-provider fallback isn't paranoia — it's basic hygiene. The question is how to do it without turning your inference path into a Rube Goldberg machine.

This is the pattern we've settled on after shipping a handful of Claude, GPT, and Gemini-backed products. It's opinionated, and it deliberately trades a bit of purity for operational calm.

Why single-provider setups keep failing

The major LLM APIs are more reliable than they were in 2023, but they still have bad days. Anthropic, OpenAI, and Google Cloud all publish status pages, and if you scroll back a year you'll see multi-hour incidents on every one of them. Beyond full outages, you get the more common failure modes:

• Regional latency spikes where p95 doubles for 20 minutes with no incident declared.
• Rate limit tightening during vendor-side capacity events, even for paying customers.
• Model-specific regressions where a new snapshot changes output shape and breaks your parser.
• Tool-call format drift when a provider ships a schema tweak on a Friday.

A single-provider architecture treats all of these as outages. A well-designed fallback chain treats most of them as routing decisions.

The trap of naive failover

The first instinct is usually: "if Claude 500s, retry on GPT." That works for about a week, until you notice three things. Your prompts were tuned for one model and produce worse output on the other. Your JSON schemas don't match because tool-calling semantics differ. And your cost dashboard is on fire because you fell back to a more expensive model during a transient blip that a 200ms retry would have solved.

Fallback is a design problem, not an if-statement.

The three-tier model

We structure fallback into three tiers, and we recommend you do the same before writing any routing code.

Tier 1 — Same provider, same model, retry. For transient 5xx, 429, and timeout errors under a threshold. Exponential backoff with jitter, capped at 2–3 attempts. This handles maybe 70% of real-world failures in our experience.

Tier 2 — Same provider, cheaper or faster model. If Claude Sonnet is timing out, try Haiku. If GPT-4 class is rate-limited, drop to a smaller sibling. Same tokenizer family, same tool-calling conventions, minimal prompt drift.

Tier 3 — Different provider entirely. Only when Tier 1 and 2 are exhausted or the whole provider is clearly down. This is the expensive tier, both in latency and in prompt-compatibility risk.

The key insight: most incidents resolve at Tier 1. Don't design your system around Tier 3 heroics if you haven't nailed Tier 1 retries.

A concrete implementation

Here's the shape of a router we've used in TypeScript. It's deliberately boring.

type Provider = 'anthropic' | 'openai' | 'google';

interface RouteStep {
  provider: Provider;
  model: string;
  timeoutMs: number;
  maxRetries: number;
}

const chain: RouteStep[] = [
  { provider: 'anthropic', model: 'claude-sonnet-4', timeoutMs: 8000, maxRetries: 2 },
  { provider: 'anthropic', model: 'claude-haiku-4', timeoutMs: 5000, maxRetries: 1 },
  { provider: 'openai',    model: 'gpt-4.1-mini',   timeoutMs: 6000, maxRetries: 1 },
];

async function route(prompt: PromptPayload): Promise<LLMResponse> {
  const errors: Error[] = [];
  for (const step of chain) {
    if (circuitOpen(step.provider)) continue;
    try {
      return await callWithRetry(step, prompt);
    } catch (err) {
      errors.push(err as Error);
      recordFailure(step.provider, err);
    }
  }
  throw new AggregateError(errors, 'All providers failed');
}

A few things to notice. The chain is data, not code — you can reorder it per-tenant, per-endpoint, or per-request-type without touching the router. Each step has its own timeout, because you want to fail fast on Tier 1 to leave budget for Tier 2. And there's a circuit breaker check before every call.

Circuit breakers matter more than retries

Without circuit breakers, a provider outage means every single request pays the full timeout before failing over. If your Tier 1 timeout is 8 seconds and Claude is fully down, every user waits 8+ seconds before you even try OpenAI.

Open the circuit after N consecutive failures (we usually start at 5 in a 30-second window), keep it open for 30–60 seconds, then let a single probe request through. Standard half-open pattern. Skip the tier entirely while the circuit is open.

This is what turns fallback from a nice idea into something that actually holds your p95 during an incident.

Prompt compatibility is the real problem

The engineering effort in fallback isn't the routing. It's making sure the fallback model produces usable output. A few patterns that help:

Keep system prompts provider-agnostic. Avoid vendor-specific instruction idioms like <thinking> tags tuned for one model. Write prompts that any capable model can follow, then add provider-specific wrappers at the last mile.

Normalize tool-calling. Anthropic, OpenAI, and Google all support structured tool use, but the schemas differ. Define your tools once in your own schema, then translate at the adapter layer. Anthropic's tool use docs, OpenAI's function-calling reference, and Google's Gemini function-calling guide are the sources of truth here — read them side by side once and it'll save you weeks.

Validate outputs, don't trust them. Every response goes through a schema validator (Zod, Pydantic, whatever). If validation fails, that's a failure signal for the router — try the next step. This catches model regressions and format drift automatically.

Test fallback output quality offline. If your Tier 3 provider produces meaningfully worse results, you need to know that before an incident, not during one. Run your eval suite against every model in the chain at least weekly.

When not to fall back

Some workloads shouldn't fall back at all. If you're generating legal summaries and only Claude has been validated for your use case, silently downgrading to a different model during an outage could be worse than failing. In those cases, fall back to a graceful degradation — a cached response, a simpler heuristic, or an honest "we're having trouble, try again in a minute" — rather than a different model.

This is a product decision, not an engineering one. Talk to whoever owns the feature before you wire it up.

Cost guardrails on the fallback path

A fallback chain can silently blow your budget. If your primary is a cheap small model and your fallback is a frontier model, a 30-minute rate-limit event could 10x your daily spend.

Two guardrails we always add:

1. Per-tier cost caps. Track spend per tier per hour. If Tier 3 usage exceeds a threshold, page someone. Something is wrong, either with the primary provider or with your circuit breaker logic.
2. Fallback rate alerting. In steady state, Tier 2 and Tier 3 should account for a small fraction of requests. If that ratio changes, you want to know within minutes, not at the end of the month when the invoice arrives.

We log every request with its final tier and provider, then chart the tier distribution on the same dashboard as latency and error rate. It's the single most useful graph for LLM reliability.

What we'd do first

If you're starting from a single-provider setup today, don't rewrite everything. In order:

1. Add proper retry-with-backoff to your existing provider. Measure how many failures that alone resolves — probably most of them.
2. Add a circuit breaker. This alone will save your p95 during the next incident.
3. Introduce a Tier 2 fallback to a smaller model from the same provider. Same tokenizer, same tool schemas, minimal prompt work.
4. Only then, and only if your reliability targets demand it, add a cross-provider Tier 3 with its own eval coverage and cost alerts.

Each step is a week or two of work and pays for itself the next time a provider has a bad afternoon. If you'd rather have someone else do this scaffolding for you, that's the kind of infra work our AI engineering team does day in and day out.

The goal isn't zero downtime. It's making provider outages boring.