Retry Logic for LLM APIs: Beyond Exponential Backoff

The default retry advice you'll find in most LLM SDK docs is roughly: catch the error, sleep with exponential backoff, try again. That works for a demo. It falls apart the moment you have a paying customer waiting on a streaming response, a tool call that half-executed, or a provider having a bad afternoon.

This is the retry playbook we actually use when shipping features on top of Claude, GPT, and Gemini — the failure modes that matter, and the code shape that handles them without lighting money on fire.

The Errors You Actually Need to Classify

Before you retry anything, you need to know what you're retrying. LLM APIs surface a much wider error taxonomy than a typical REST service, and treating them all the same is where budgets die.

Here's the rough classification we use:

• Transient infra errors — 500, 502, 503, connection resets. Retry aggressively.
• Provider overload — Anthropic's 529 overloaded_error, OpenAI's 503, Gemini's RESOURCE_EXHAUSTED when it's server-side. Retry with real jitter and a cap.
• Rate limits (your fault) — 429 with headers telling you when to try again. Respect the header, don't guess.
• Context or input errors — 400s about token limits, invalid tool schemas, content policy. Never retry. These are permanent until code changes.
• Auth errors — 401, 403. Never retry. Page someone.
• Streaming mid-flight failures — connection dropped after 40 tokens streamed. Special case, covered below.
• Partial tool call failures — the model returned a tool call, your tool errored, and now you have to decide whether to retry the tool or ask the model again.

If your retry wrapper treats a 400 the same as a 529, you'll retry three times, waste ~4 seconds, and still return an error. Multiply that by every request in a bad hour and you'll notice on the invoice.

Reading the vendor signals

Each provider gives you different hints. Per Anthropic's error docs, a 529 explicitly means the API is overloaded and you should back off; a 429 includes retry-after in some cases. OpenAI's rate-limit responses include x-ratelimit-reset-requests and x-ratelimit-reset-tokens headers you should actually parse. Gemini surfaces RetryInfo in the error details for RESOURCE_EXHAUSTED. Use what they give you before inventing your own timing.

Backoff That Doesn't Herd

Straight exponential backoff (2^n seconds) creates thundering herds: every client that failed at 12:00:00.000 wakes up at 12:00:02.000 and hammers the API together. During a real provider incident, this is what turns a five-minute blip into a thirty-minute outage on your side.

We use decorrelated jitter, which AWS popularised for their SDKs and which works just as well here:

import random
import time

def decorrelated_jitter(prev_delay: float, base: float = 1.0, cap: float = 30.0) -> float:
    return min(cap, random.uniform(base, prev_delay * 3))

delay = 1.0
for attempt in range(max_attempts):
    try:
        return call_llm()
    except RetryableError as e:
        if attempt == max_attempts - 1:
            raise
        # Prefer the provider's hint if present
        delay = e.retry_after or decorrelated_jitter(delay)
        time.sleep(delay)

Two things to notice: we cap the delay (30s is our default; longer than that and the user has left), and we always prefer the provider's retry-after over our own math.

Streaming Failures Are a Different Animal

A non-streaming call is atomic: it either returned or it didn't. A streaming call can fail after emitting 200 tokens to your UI. Retrying the whole request means the user sees the first sentence twice, or worse, contradicting text.

Three patterns we use, roughly in order of user-visible quality:

1. Silent restart — if fewer than N tokens have been emitted (we use ~20), clear the buffer, restart the request, and pretend it never happened. Fine for chat UIs.
2. Continue from last token — send a follow-up request with the partial output as an assistant message and instruct the model to continue. Works for long generation, but the seam is sometimes visible.
3. Fail forward — show the partial output plus an inline error affordance ("regenerate"). Best for creative tools where the user might like what they saw.

What you should almost never do is retry a streamed request from scratch and keep streaming to the same open connection. Users notice.

Idempotency Keys Are Not Optional

If your request has side effects — writing to a DB, charging a card, sending an email via a tool call — retries can double-execute. OpenAI supports an Idempotency-Key header on the Responses API for exactly this. Anthropic and Gemini don't have first-class idempotency keys at the model API level as of writing, so you handle it on your side: hash the request, cache the result for a short TTL, and short-circuit on retry.

const key = hash({ model, messages, tools, seed });
const cached = await cache.get(key);
if (cached) return cached;

const result = await callLLM({ model, messages, tools });
await cache.set(key, result, { ttl: 60 });
return result;

This also doubles as a poor-man's dedupe when a user double-clicks a button.

Retrying Tool Calls Without Looping Forever

Agent loops are where retry logic gets dangerous. The model calls a tool, the tool fails, you feed the error back, the model tries again, and again, and again. We've seen loops rack up a hundred requests before someone noticed.

Guardrails we always add:

• Per-turn tool retry cap — a single tool can fail at most twice before we surface a hard error to the model and let it choose an alternative.
• Total step budget — the whole agent run has a max step count (usually 8–15 depending on the task).
• Token budget — a hard ceiling on cumulative input+output tokens per run, checked before each step.
• Repeat detection — if the model calls the same tool with the same arguments twice in a row, we break the loop and return.

Repeat detection has caught more runaway costs than any other single guardrail we've shipped. Models will absolutely call search("foo") five times in a row when confused.

When the model itself is the flake

Sometimes the request succeeds — 200 OK — but the output is broken: malformed JSON, a tool call with invalid arguments, a refusal when you know the input was fine. These aren't API errors, they're content errors, and they need their own retry path.

For structured outputs, we retry once with a repair prompt ("Your previous output failed schema validation with error X — return corrected JSON only"). Beyond one repair attempt, we escalate: fall back to a stronger model, or return a clean error to the caller. Retrying the same prompt on the same model three times almost never fixes it — you're just paying for the same mistake.

Fallback Between Providers

If your product genuinely can't be down when Anthropic is having a bad day, you need a cross-provider fallback. This is more work than it sounds because prompts, tool schemas, and output formats differ enough that you can't just swap endpoints.

What has worked for us:

• Keep a capability tier, not a model name. "Reasoning tier" might map to Claude Sonnet primarily and GPT as a fallback.
• Maintain prompt variants per provider — same intent, different phrasing where it matters (Gemini prefers slightly different system-prompt structure than Claude, for instance).
• Only fall back on sustained failures (e.g., 3 failures in 30 seconds), not the first 529. Otherwise you'll flap between providers and confuse your own logs.
• Log which provider served each request so your evals stay honest.

This is not free. Maintaining two prompt paths costs real engineering time, and for most features it's not worth it. Reserve it for the flows where downtime = churn.

Observability That Makes Retries Debuggable

A retry you can't see is worse than no retry at all. At minimum, log per attempt: model, provider, error class, HTTP status, retry-after if present, attempt number, and the final outcome. Emit a metric on retry attempts by error class so you can see when a provider's 529 rate spikes before your users tweet about it.

We also tag every request with a client_request_id that stays constant across retries, so you can trace one user's five attempts as a single logical operation in your APM.

Where We'd Start

If you're bolting this onto an existing codebase, don't try to build the whole thing at once. In order:

1. Classify errors properly. Stop retrying 400s. This alone will cut wasted spend.
2. Add decorrelated jitter and respect retry-after. One afternoon of work.
3. Add a per-run step and token budget to any agent loop. This prevents the worst outages.
4. Add idempotency for anything with side effects.
5. Only then think about cross-provider fallback — and only for the flows that need it.

Most teams over-invest in fancy fallback logic and under-invest in error classification and budget caps. Fix the boring stuff first; it's what actually keeps the pager quiet.