Prompt Caching in Production: What Actually Saves Money vs What Just Looks Clever

Prompt caching is one of those features that looks like a free win in the pricing table and quietly turns into a mess in production. The vendors all shipped their own flavour — Anthropic, OpenAI, and Google — and the fine print differs enough that a naive port between them will either overspend or silently stop caching altogether.

This is what we've learned shipping cached prompts across Claude, GPT, and Gemini in real client workloads: where the money actually shows up, where the traps are, and how to design a prompt so the cache does what you think it does.

What each vendor actually caches

Before we get to patterns, the mechanics matter, because they're not the same thing wearing three logos.

Anthropic (Claude) exposes explicit cache_control breakpoints in the messages API. You mark up to four points in the prompt as cacheable, and Anthropic stores the prefix for a short TTL (five minutes by default, with a longer one-hour option documented in their prompt caching docs). Cache writes cost more than a normal input token; cache reads cost roughly a tenth of a normal input token. You pay to fill the cache, then you save every time you hit it.

OpenAI does prompt caching automatically for prompts above a token threshold (1024 tokens at time of writing, per their docs). There's no API flag — they hash the prefix, and if it matches a recent request routed to the same backend, you get a discount on the cached portion. No write premium, but also no explicit control.

Google (Gemini) offers context caching as a separate resource: you create a cached content object with its own TTL and pay a storage fee per hour, then reference it by name in generate calls. It's closer to a materialised view than a transparent cache.

Three very different mental models. Same marketing word.

Where the savings actually come from

In our experience, prompt caching pays off in exactly three shapes of workload. Everything else is noise.

1. Long system prompts hit many times per minute

This is the canonical case. You have a 4k–20k token system prompt — tool definitions, style guides, few-shot examples, policy — and thousands of short user turns. The system prompt is stable, so the prefix is stable, so it caches.

On Claude, put the cache_control breakpoint at the end of the system block and the end of any large tool schema. On OpenAI, just make sure the stable content is at the start of the messages array and nothing dynamic sneaks in before it. On Gemini, create a cached content resource for the system + tools and reference it.

2. Multi-turn agents with growing history

Agents replay their whole trajectory on each step. Turn 8 sends turns 1–7 plus the new observation. That's a growing prefix — and every new turn is a cache hit on everything that came before.

The trick here is not touching older turns. If you rewrite history (summarising, compressing, or reordering tool results) you invalidate the prefix and pay full price to refill the cache. Do compression at fixed breakpoints, not every turn.

3. RAG with a stable instruction block and volatile context

Counter-intuitively, RAG benefits less than people expect, because the retrieved chunks change per query. What caches well is everything before the retrieved chunks: instructions, output schema, few-shots. Structure the prompt so those come first, retrieval comes last, and mark the cache breakpoint just before retrieval.

# Claude example — cache the stable prefix, not the retrieved chunks
messages = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": SYSTEM_INSTRUCTIONS + FEW_SHOTS + OUTPUT_SCHEMA,
                "cache_control": {"type": "ephemeral"},
            }
        ],
    },
    {
        "role": "user",
        "content": f"Context:\n{retrieved_chunks}\n\nQuestion: {question}",
    },
]

If you flip the order and put retrieved chunks before instructions, you've built a cache that never hits.

The traps we keep seeing in code review

Most of the caching bugs we find on client codebases fall into the same handful of patterns.

Timestamps and request IDs in the prefix

Someone adds "Current time: {now}" to the system prompt for a good reason (time-aware answers), and every request becomes a cache miss. If you need the time, put it at the end of the prompt, after the cache breakpoint, or round it to the hour.

Non-deterministic tool schema serialisation

Python dictionaries are ordered now, but if any part of your tool schema comes from a set, or from a JSON blob that isn't sorted, you'll get different byte sequences from otherwise identical schemas. Cache hits are byte-exact. json.dumps(schema, sort_keys=True) is a cheap insurance policy.

Personalisation glued to the top

"Hi {user_name}, you are a helpful assistant" — congratulations, every user has their own cache line, and none of them hit. Personalisation goes after the cached block, or into the user message, not into the system prefix.

Cache TTL vs traffic shape

Anthropic's default TTL is five minutes. If your traffic is spiky — a batch job every 15 minutes, say — you'll pay the cache-write premium every single run and never amortise it. Either switch to the longer TTL tier, batch requests inside the TTL window, or accept that caching isn't the right lever for that workload.

Streaming and cache metrics

Cache hit stats come back in the response metadata (usage.cache_read_input_tokens on Anthropic, usage.prompt_tokens_details.cached_tokens on OpenAI). If you're not logging these per request, you have no idea whether caching is working. We've seen teams "enable caching" and see zero cost change for weeks because a single dynamic field was killing every prefix.

A rough decision framework

Here's the shortlist we run through when a client asks whether prompt caching will help them.

1. Is your stable prefix longer than ~1500 tokens? If not, skip it. The overhead isn't worth it, and OpenAI won't cache below its threshold anyway.
2. Do you hit the same prefix more than 2–3 times within the TTL window? If not, the write premium (on Claude) or storage cost (on Gemini) eats the savings.
3. Can you guarantee byte-exact prefix stability? If your prompt is assembled from a template with variable data anywhere before the breakpoint, fix that first.
4. Are you measuring cache read tokens per request? If not, instrument before you optimise.

If all four are yes, cached prefixes routinely cut input-token costs by 50–80% in our workloads. If any are no, you're probably better off looking at model selection, smaller retrieval, or a semantic cache in front of the whole call.

When not to cache

A few workloads where we've explicitly turned prompt caching off:

• One-shot analytical calls where each prompt is unique (document summarisation over a long tail of documents). No prefix reuse, no gain.
• Cheap models on short prompts. Haiku or GPT-4o-mini on a 500-token prompt: the accounting overhead isn't worth the engineering time.
• Highly personalised prompts where per-user context dominates the token count. Move personalisation into retrieval instead, and cache the instructions.
• Compliance-sensitive prompts where you'd rather not have any provider-side prefix retention, even ephemeral. Check the vendor's data retention terms against your DPA.

Where we'd start

If you're introducing prompt caching to an existing product, don't refactor everything on day one. Pick your single highest-traffic endpoint, log the current input token cost for a week, then:

1. Sort your prompt so all stable content is at the top.
2. Move every dynamic field (timestamps, user IDs, retrieved chunks) below a clear breakpoint.
3. Add the vendor's cache marker (or, on OpenAI, just verify you're above the threshold).
4. Ship it behind a feature flag and compare cached_tokens against total input tokens on the same traffic mix.

If the hit rate on that endpoint isn't north of 60% after a day, the prompt structure is wrong, not the cache. Fix the structure before you roll it out anywhere else.

If you want a second pair of eyes on a specific workload, our team does this kind of LLM cost and architecture review as part of our AI engineering services — and we've written more on the surrounding tradeoffs over on the blog.