Prompt Caching in Production: What It Actually Saves and Where It Bites

Prompt caching went from vendor curiosity to a line item on every serious LLM budget in about eighteen months. The pitch is irresistible: pay full price for your 8k-token system prompt once, then pay 10% (or less) on every follow-up call. The reality, once you ship it behind a real product, is more interesting — and occasionally infuriating.

This is a field report on what prompt caching actually does for cost and latency, how the three big vendors differ, and the specific places we've watched it silently stop working.

What prompt caching actually is

Every major provider now offers some flavor of it:

• Anthropic exposes explicit cache_control breakpoints on message blocks, with a 5-minute default TTL and a longer 1-hour option (see Anthropic's prompt caching docs).
• OpenAI does automatic caching on prompts over 1024 tokens, keyed on prefix, with no manual markers required (see OpenAI's prompt caching guide).
• Google Gemini offers explicit context caching via a separate cachedContents resource you create, reference by name, and manage the TTL on yourself.

All three share the same underlying idea: the KV cache from a previous forward pass over a prefix is retained, so repeated prefixes skip the prefill compute. That's why cached input tokens cost less and — often more importantly — return their first token faster.

The differences matter more than the marketing suggests.

Implicit vs explicit caching

OpenAI's automatic caching is convenient but opaque. You get a cached_tokens field in the usage response and that's about it. There's no way to say "cache this, not that" — the system decides based on prefix match.

Anthropic makes you commit. You put a cache_control marker on a block and that block plus everything before it becomes a cache breakpoint. You get up to four breakpoints per request. This is more work, but it also means you know exactly what you're paying to store.

Gemini's approach is the most explicit and, honestly, the most awkward. You create a cache object, get back a resource name, then pass that name in subsequent requests. It's designed for large, stable contexts (think: an entire codebase or a 200-page PDF) that you'll query repeatedly over an hour or more.

The cost math nobody puts in the pitch deck

Here's where teams get burned. Cached reads are cheap, yes. But cache writes are not free — and on Anthropic they cost more than a normal input token (roughly 1.25x for the 5-minute cache, higher for the 1-hour variant).

So the break-even isn't "one cache hit." It's more like:

breakeven_hits = (write_multiplier - 1) / (1 - read_discount)

For Anthropic's 5-minute cache at ~1.25x write and ~0.1x read, that's roughly:

(1.25 - 1) / (1 - 0.1) ≈ 0.28

Meaning you need about one hit for every ~3.5 writes to come out ahead. Sounds easy, but consider:

• A user opens a chat, sends one message, and closes the tab. One write, zero hits. Net loss.
• A background job processes 1,000 documents against the same system prompt. One write, 999 hits. Enormous win.
• A conversation that idles for 6 minutes between turns with the default TTL. Write, expire, write again. Repeated loss.

Caching is a bet on session shape. If your traffic is bursty and short, you can quietly increase your bill by turning it on naively.

Where we cache, and where we don't

After shipping this across several products, we've landed on a rough rubric:

Good caching candidates:

• Long, stable system prompts (agent instructions, tool schemas, style guides)
• RAG contexts that are reused within a session (a document the user is chatting with)
• Few-shot examples that don't rotate per request
• Multi-turn conversations where the running history grows monotonically

Bad caching candidates:

• Anything with a timestamp or request ID injected at the top
• Prompts where the retrieved chunks change every turn (classic RAG)
• One-shot classification endpoints with high fan-out and no session
• A/B experiments where the system prompt changes per user

The timestamp one gets people constantly. Someone adds Current time: {now} to the top of the system prompt for date-aware answers, and every single request misses the cache. We've seen this cut effective hit rate from 80% to under 5% in a single deploy.

Ordering matters more than you'd think

Cache prefixes are prefix-matched, not set-matched. Put the volatile stuff at the end:

# Bad: user_id changes every request, invalidates everything after
system = f"User: {user_id}\n{BIG_STABLE_PROMPT}\n{TOOLS}"

# Good: stable prefix first, volatile suffix last
system = f"{BIG_STABLE_PROMPT}\n{TOOLS}\nUser: {user_id}"

This sounds obvious until you find it in a codebase where three different teams have been appending to the same prompt string for a year.

A concrete example with Anthropic

Here's roughly how we structure a cached RAG call when the document is stable for the session but the user question isn't:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": AGENT_INSTRUCTIONS,  # ~2k tokens, never changes
            "cache_control": {"type": "ephemeral"},
        },
        {
            "type": "text",
            "text": f"<document>{doc_text}</document>",  # per-session
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=[
        {"role": "user", "content": user_question}
    ],
)

usage = response.usage
print(
    "input:", usage.input_tokens,
    "cache_read:", usage.cache_read_input_tokens,
    "cache_write:", usage.cache_creation_input_tokens,
)

Two breakpoints: one for the instructions (reused across all sessions and users), one for the document (reused across turns in a session). The user message stays uncached because it's tiny and changes every turn — no point paying the write premium.

Log those three usage fields into your metrics pipeline from day one. If you don't, you'll never know whether caching is actually working.

TTL traps

Anthropic's 5-minute TTL refreshes on hit, which is friendlier than it sounds — an active conversation keeps its cache warm. But there are two failure modes:

1. Idle users. A user walks away for lunch, comes back, and the next message pays a full cache-write again. In some products this is the majority of sessions.
2. Load balancer stickiness. Caches are typically per-region and sometimes per-server. If your traffic gets routed to a different backend, you can miss even within the TTL window. This is undocumented behaviour that varies by provider and we've only ever confirmed it by staring at usage metrics.

Gemini's explicit caching sidesteps the first problem (you set the TTL) but introduces the second one more visibly: you're managing a stateful resource with a lifecycle, and if you forget to delete stale caches, you pay storage fees for content nobody is reading.

Caching plays badly with some patterns

A few combinations to watch for:

• Streaming + short outputs. The TTFT win from caching is real, but if your output is 50 tokens the perceived latency difference is small. Save the engineering for endpoints where prefill dominates.
• Tool-heavy agents. Every tool call round-trip is a new request. If your tool results get appended in a way that doesn't preserve the prefix exactly (whitespace, ordering, IDs), you'll thrash the cache. Test this specifically.
• Evals. Cached responses aren't different responses, but if you're benchmarking latency, caching will make your numbers look better than production. Run eval suites with caching disabled or with cache-cold runs sampled in.

What we measure

For any endpoint using caching, we track four things per request and aggregate them:

• cache_read_tokens / total_input_tokens — the actual hit ratio
• cache_write_tokens per session — are we paying the premium repeatedly?
• Cost per successful task, not per request — the only number that matters
• p50 and p95 TTFT split by cache-hit vs cache-miss

Without the split TTFT metric, a drop in hit rate looks like a latency regression with no obvious cause. We've spent afternoons chasing ghosts because of this.

Where we'd start

If you haven't turned on caching yet: instrument first, cache second. Add the usage fields to your logging today, even without any caching enabled. Then pick your single highest-volume endpoint with a stable system prompt over ~2k tokens and put one cache breakpoint on it. Ship it behind a flag, watch the hit ratio for a week, and only then decide whether the second breakpoint on retrieved context is worth the complexity.

And if your prompt starts with the current timestamp — fix that before you do anything else. If you'd like a hand auditing an existing LLM stack for this kind of thing, that's the sort of work we do on our AI engagements.