All articles
AI & LLMsNovember 10, 2026 7 min read

Prompt Caching in Practice: What Actually Lands on the Bill

Prompt caching sounds like free money until you read the invoice. Here's how Anthropic, OpenAI, and Gemini caches actually behave, and the patterns that make them pay off.

Prompt caching is one of those features that reads like a cheat code in the vendor blog post and then quietly underperforms in your billing dashboard. The mechanics are simple; the economics are not. If you're building anything with long system prompts, tool schemas, or document grounding, the difference between using caching well and using it badly is roughly the difference between a healthy gross margin and a bad Monday.

This is a field guide to what actually happens when you turn caching on across Anthropic, OpenAI, and Google — where it helps, where it quietly costs more, and the patterns we reach for when shipping real products.

Why caching exists, and why the pricing is weird

LLM inference is dominated by the prefill step: the model has to attend over every token in your prompt before it generates the first output token. For a 20k-token system prompt plus retrieved context, prefill is where most of your latency and most of your input cost live. Prompt caching lets the provider reuse the KV cache from a previous request if the prefix matches exactly, skipping that work.

The pricing wrinkle is that writing to the cache is usually more expensive than a normal input token, while reading from it is much cheaper. So the question isn't "should I cache?" It's "will this prefix be hit often enough, within the TTL, to amortize the write?"

The three implementations, briefly

  • Anthropic (Claude): Explicit. You mark up to four cache_control breakpoints in your messages. Writes cost ~1.25x base input (5-minute TTL) or ~2x (1-hour TTL); reads cost ~0.1x. See Anthropic's prompt caching docs for current numbers and minimum cacheable sizes (1024 tokens on Sonnet-class models, 2048 on Haiku-class at time of writing).
  • OpenAI: Automatic for prompts over 1024 tokens on supported models. No code changes required; the router hashes your prefix and reuses it when it can. Cached input tokens are discounted (currently 50–75% off depending on model). TTL is short and not guaranteed.
  • Google (Gemini): Two modes. Implicit caching works like OpenAI's — automatic, best-effort. Explicit context caching is a separate API where you create a CachedContent object with its own TTL and per-hour storage fee, then reference it by name.

The mental model that helps: Anthropic makes you design for the cache, OpenAI hides it, and Gemini gives you a storage primitive you can manage like any other resource.

Where caching actually earns its keep

In our experience, four workload shapes get real savings. Everything else is marginal.

  1. Long, stable system prompts. Agent frameworks with 5–15k tokens of instructions, tool definitions, and few-shot examples. The prefix is identical across millions of calls.
  2. Document Q&A over a hot set. A legal or support tool where the same 50 documents get asked about all day. Cache the document, vary the question.
  3. Multi-turn conversations. Each turn appends to the previous; the growing prefix is a perfect cache candidate on every turn after the first.
  4. Batch evals and regression runs. Running your golden set against a new prompt — same prompt prefix, hundreds of test inputs.

Where it disappoints:

  • RAG with high retrieval diversity. If every query pulls a different set of chunks, the prefix changes every time. You're paying write cost with no reads.
  • Low-traffic endpoints. If your cache TTL is 5 minutes and your endpoint gets a request every 20 minutes, you're always writing, never reading.
  • Prompts under the minimum. Anthropic won't cache under 1024 tokens; OpenAI won't bother either. Short classification prompts get no benefit.

The prefix discipline problem

Caching is prefix-based. One byte different at position 500 and the cache miss cascades through everything after it. This sounds obvious until you audit a real codebase and find the timestamp someone helpfully injected into the system prompt "for debugging."

Things we've seen silently kill cache hit rates:

  • datetime.now() in the system prompt
  • User ID or session ID interpolated near the top
  • Tool definitions reordered because they're stored in a dict
  • Dynamic few-shot selection that reshuffles examples per request
  • A/B test flags that flip a single sentence

The fix is boring: make the cacheable portion of your prompt a pure function of a small, versioned set of inputs, and shove everything volatile to the end.

# Bad: cache-hostile
system = f"""You are an assistant for {user.company}.
Current time: {datetime.now().isoformat()}
User tier: {user.tier}

{LONG_INSTRUCTIONS}
{TOOL_DEFINITIONS}
"""

# Better: cacheable prefix, volatile suffix
system_cacheable = f"""{LONG_INSTRUCTIONS}
{TOOL_DEFINITIONS}
"""

user_message = f"""<context>
company: {user.company}
tier: {user.tier}
time: {datetime.now().isoformat()}
</context>

{actual_user_query}"""

For Anthropic specifically, you want the cache_control breakpoint right at the end of the stable block:

messages = [{
    "role": "user",
    "content": [
        {
            "type": "text",
            "text": LONG_INSTRUCTIONS + TOOL_DEFINITIONS,
            "cache_control": {"type": "ephemeral"}
        },
        {
            "type": "text",
            "text": volatile_context + user_query
        }
    ]
}]

Measuring hit rate before you celebrate

All three providers return cache usage in the response. Log it. If you're not logging cache_read_input_tokens and cache_creation_input_tokens (Anthropic) or prompt_tokens_details.cached_tokens (OpenAI) on every call, you don't know whether caching is working. We've audited projects where the team was convinced caching was saving them 60% and the actual read rate was 11%.

A minimal dashboard needs three numbers per endpoint: cache hit rate (reads / total input tokens eligible for caching), effective input cost per request, and TTL-adjusted write amortization (did this write get read enough times to pay for itself before it expired?).

TTL strategy: ephemeral vs explicit

Anthropic's 5-minute and 1-hour TTLs look like a simple choice until you do the math. The 1-hour cache costs ~2x input to write but survives 12x longer. The breakeven depends on how your traffic clusters.

Rough rule we use: if the prefix will be hit more than ~8–10 times in an hour, the 1-hour TTL wins. Below that, stick with 5-minute. For chat sessions, 5-minute is almost always right because conversations are bursty. For a batch eval run, the 1-hour TTL is a no-brainer.

Gemini's explicit context caching is the odd one out because it charges storage per hour. That makes it attractive for genuinely large, genuinely stable content — think a 500k-token codebase or a book you're answering questions about for a day — and unattractive for anything smaller or shorter-lived. Do the arithmetic against the base per-token cost before committing; Google's docs have the current storage rate.

Caching and RAG: the uncomfortable truth

Here's the pattern we see fail most often. Team builds a RAG system. Each query retrieves 8 chunks from a 100k-document corpus. They turn on prompt caching and expect a big win. They get almost nothing.

The reason: the retrieved chunks sit inside the prompt, usually before the question, and they change every query. The cacheable prefix is just the system instructions — maybe 2k tokens — and everything after the first chunk is a miss.

Two patterns that actually work:

  • Hot-document caching. For the top 20% of documents that get 80% of queries (yes, it's usually this skewed), build per-document prompts and cache them individually. Route queries that match a hot document to the cached variant; fall back to standard RAG for the long tail.
  • Query-time reordering. Put the retrieved context after the question, not before. You lose some of the cache benefit on the system prompt extension, but you preserve the ability to cache the stable instructional prefix. Sometimes this is worse for accuracy — test it.

If you're building RAG seriously, this intersects with chunking and retrieval strategy in ways that deserve their own treatment. Our AI engineering services team has spent a lot of 2025 and 2026 unpicking exactly these tradeoffs for clients.

Agents are the best caching customers you'll ever have

Agent loops are where caching goes from "nice optimization" to "the thing that makes the economics work at all." A ReAct-style agent with a 10k-token system prompt and tool schemas, running 8 turns to complete a task, would normally pay for that prefix 8 times. With caching, it pays once (plus small deltas for the growing message history).

The trick is being disciplined about where you put the cache breakpoints. For Claude, we typically use two of the four available: one after the system prompt + tools (rarely changes), and one after the most recent stable turn of the conversation. The last turn stays uncached because it's being written to.

On OpenAI, the implicit cache usually just works for this shape, which is a mercy.

What we'd do

If you're turning caching on this week: start by logging cache token usage on every call for two days before changing anything. You need a baseline. Then pick your single highest-volume endpoint and refactor its prompt into a strict stable-prefix + volatile-suffix shape. Measure again. If you're not seeing cache read rates above 50% on that endpoint within a week, the problem is prefix discipline, not the cache itself.

For new builds, design the prompt as two concatenated strings from day one — stable and volatile — and make it a lint rule that nothing dynamic goes in stable. It's a two-minute convention that pays for itself every month.

#LLMs#Cost Optimization#RAG#Anthropic#OpenAI#Gemini

Want a team like ours?

72Technologies builds production software for the kind of teams who actually read this blog.

Start a project