Prompt Caching in Production: What Actually Saves Money on Claude and Gemini

Prompt caching is one of those features that looks like a knob you turn once and forget. It isn't. Used well, it drops repeat-call latency by a third and cuts input token costs by 50–90% on the right workloads. Used badly, it's a rounding error on your invoice and a source of subtle bugs.

This is what we've learned shipping caching on Anthropic's Claude and Google's Gemini across RAG systems, coding agents, and long-context document tools.

What prompt caching actually is

Both Anthropic and Google let you mark a prefix of your prompt as cacheable. On a cache hit, the model skips re-processing those tokens and charges you a fraction of the normal input rate. The mechanics differ:

• Anthropic's prompt caching uses cache_control breakpoints inside your messages. You can place up to 4 breakpoints and pick a TTL (5 minutes or 1 hour, per Anthropic's docs). Cache writes cost more than base input tokens; cache reads cost significantly less.
• Gemini context caching is an explicit resource. You create a CachedContent object with a TTL, then reference it by name in subsequent generateContent calls. Google bills for cache storage per token-hour plus a reduced rate for cached input tokens (see Google's Gemini API docs).

The mental model matters. Anthropic's caching is implicit and ephemeral — a side effect of your request shape. Gemini's is explicit and managed — you create it, you delete it, you pay rent while it lives.

Why this changes your prompt architecture

Caching only works on exact prefix matches. One byte different at position 12, and you miss the cache for everything after position 12. That single constraint dictates everything else in this article.

The prefix discipline that actually pays off

Here's the layout we use for cacheable prompts, in strict order:

1. System instructions (rarely change)
2. Tool/function definitions (change on deploy)
3. Few-shot examples (change on prompt version bump)
4. Retrieved context or large documents (per-session or per-document)
5. Conversation history (grows over time)
6. The current user turn (always fresh)

Whatever is most stable goes first. Whatever changes per request goes last. This is the opposite of how a lot of teams naturally write prompts — they front-load the user question because it feels important.

A concrete Claude example:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,  # stable, ~2k tokens
        },
        {
            "type": "text",
            "text": LARGE_STYLE_GUIDE,  # ~15k tokens
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": document_context,  # per-document, ~30k tokens
                    "cache_control": {"type": "ephemeral"},
                },
                {
                    "type": "text",
                    "text": user_question,
                },
            ],
        }
    ],
)

Two breakpoints, two cache layers. The style guide cache survives across users. The document cache survives across a single user's session.

The invisible cache-buster: timestamps and IDs

We've watched teams add a Current time: 2026-03-14T09:31:22Z line to their system prompt and wonder why their hit rate is zero. Same story with request IDs, trace IDs, and user names injected into the system block. If you need these, put them after the last cache breakpoint or drop them entirely and pass them as tool inputs instead.

Picking TTLs without lighting money on fire

Anthropic's 5-minute TTL refreshes every time you hit the cache, so a busy endpoint keeps the cache warm indefinitely. The 1-hour TTL costs more to write but is the right call when traffic is bursty.

A rough decision tree we use:

• Request-to-request gap under ~2 minutes and steady traffic → 5-minute TTL.
• Bursty traffic, or an agent that pauses for tool execution → 1-hour TTL.
• Analytical workloads with predictable batch windows → 1-hour, kicked off with a warmup request.

For Gemini, the calculation is different because you're paying storage rent. The break-even is roughly: (cache_write_cost + storage_cost_over_lifetime) < (normal_input_cost × expected_reads). If you're not going to hit the cache at least a handful of times before it expires, don't create it.

Measuring whether it's actually working

Both APIs return usage stats that tell you exactly what happened. Log them. If you're not logging cache metrics per request, you're guessing.

Anthropic's response includes:

{
  "usage": {
    "input_tokens": 512,
    "cache_creation_input_tokens": 15234,
    "cache_read_input_tokens": 30112,
    "output_tokens": 487
  }
}

The metric that matters is cache read ratio: cache_read_input_tokens / (cache_read_input_tokens + cache_creation_input_tokens + input_tokens). On a healthy RAG endpoint we typically see 0.7 – 0.95 after warmup. Below 0.5 means your prefixes aren't stable.

Build a simple dashboard:

• Cache read ratio, per route
• Effective input cost per request (weighted by write/read/base rates)
• Cache write count per hour (spikes here mean cache-busting bugs)

We've caught two production regressions with this dashboard alone — both were engineers appending a debug_context field into the system prompt behind a feature flag.

Where caching quietly fails

Streaming and cache misses

Cache lookups happen before generation starts, so a miss adds latency you can't stream around. When p99 latency matters, keep an eye on your miss rate during deploys — a prompt template change invalidates every cache entry that referenced the old prefix.

Multi-tenant prefix pollution

If you interpolate a tenant name into your system prompt ("You are assisting Acme Corp employees..."), you get one cache per tenant. That's fine for large tenants and disastrous for a long tail of small ones. Consider moving tenant-specific context after the cache breakpoint, or using a two-tier prompt where the shared system block is cached and tenant details live in the user message.

Agents with dynamic tool sets

An agent that adds or removes tools mid-conversation is going to blow through your cache every turn. Freeze the tool list per session, or accept that agent workloads will have lower hit rates than RAG workloads.

The "we cached the wrong thing" trap

Caching a 2k-token system prompt saves you almost nothing. Caching a 50k-token document that's queried three times saves you real money. Rank your cache candidates by tokens × expected_reads and put breakpoints where that number is largest.

Claude vs Gemini for cached workloads

We don't think there's a universal winner — pick based on shape of your workload:

• High-frequency, short-lived context (support bot with a 5k-token knowledge base loaded per session): Claude's implicit caching is easier to wire up and the 5-minute auto-refresh handles conversational patterns naturally.
• Long-lived, expensive context (a 500k-token document you'll query dozens of times over an hour): Gemini's explicit context caching, combined with its large context window, tends to be cheaper and more predictable. You know the cache exists because you created it.
• Mixed workloads with tool use: Claude's cache_control on tool definitions is convenient. Just don't mutate the tool list.

For cost specifics, check current pricing on Anthropic's and Google's docs — the ratios move, and we've seen them shift meaningfully between model generations.

A quick war story

A client's RAG endpoint was costing roughly $8k/month. Their prompt structure was: user question, then retrieved chunks, then system instructions, then conversation history. Backwards. We inverted it — system, tools, a cached breakpoint after the (session-stable) retrieval context, then history and the user turn. Cache read ratio went from effectively zero to about 0.82 within a day. Monthly cost dropped to somewhere around $2.1k. No model change, no retrieval change, no quality regression on their eval set. Just prefix discipline.

That's not a benchmark — it's one workload with a particular traffic pattern. But the shape of the win is common enough that if you're spending real money on repeated long-context calls and haven't audited your prefix layout, you're probably leaving 40–70% on the table.

Where we'd start

If you're staring at an LLM bill and wondering whether caching is worth the engineering time, do these three things this week:

1. Log cache_creation_input_tokens and cache_read_input_tokens (or Gemini's equivalents) on every call. You can't optimize what you can't see.
2. Rewrite your top-cost endpoint so stable content comes first and volatile content last. Add one breakpoint after your largest stable block.
3. Measure the cache read ratio for 48 hours across normal traffic. If it's above 0.6, extend the pattern to other endpoints. If it's below 0.3, find the cache-buster (usually a timestamp, ID, or per-user field sneaking into a shared prefix).

If you'd rather have us audit a production LLM workload end-to-end, that's the kind of thing we do in our AI engineering practice. But most teams can grab the first big win themselves in an afternoon.