Prompt Caching in Production: When Claude and Gemini Actually Save You Money

Prompt caching is one of those features that reads like free money in the vendor blog post and then quietly underperforms in production. We've shipped it on both Claude and Gemini across RAG pipelines, long-context agents, and document-heavy workflows — and the economics only work when you design for them.

This is the guide we wish we had before we started measuring cache hit rates in Grafana at 2am.

What prompt caching actually is (and isn't)

Prompt caching lets you tell the provider: "I'm going to send this large prefix many times — store it server-side so you don't have to re-process it on every call." The provider keeps the attention computation warm for a short window and charges you a fraction of the normal input price on cache hits.

Two important things it is not:

• It is not a semantic cache. It won't return a previous response. Every call still runs the model.
• It is not free. You pay a write premium to put something into the cache, and a reduced read price on hits. If your hit rate is low, caching costs you more than not caching.

Anthropic documents Claude's prompt caching with a write cost roughly 1.25x the base input token price and reads at roughly 0.1x (see Anthropic's prompt caching docs for current numbers per model). Google's Gemini context caching has a different shape: you pay a storage cost per million tokens per hour, plus a reduced input price on cached tokens (see Google's Gemini API context caching docs).

That pricing shape difference is the whole game.

The break-even math nobody shows you

For Claude, the break-even is a function of hit rate. Rough model:

cost_no_cache = N * T * P_in
cost_with_cache = T * P_write + (N - 1) * T * P_read

where:
  N = number of calls reusing the prefix
  T = tokens in the cached prefix
  P_in = base input price
  P_write ≈ 1.25 * P_in
  P_read ≈ 0.1 * P_in

Solve for when caching wins: you need roughly N ≥ 2 calls against the same prefix within the TTL window for Claude's 5-minute default cache. Easy to hit in a chat session, hard to hit in a cold-start batch job.

For Gemini, it's different. You pay storage per hour regardless of hits:

cost_with_cache = (T * P_cache_write) 
                + (T * P_storage_per_hour * H) 
                + (N * T * P_cached_input)

So Gemini caching wins when traffic density against the prefix is high enough that storage amortizes. For a cached 100K-token system prompt sitting idle for an hour at low QPS, you can easily lose money.

A quick decision rule we use

• Bursty, session-scoped reuse (chat, agent loops): Claude's ephemeral cache.
• Steady, high-QPS reuse of a stable prefix (document Q&A over the same corpus for hours): Gemini's explicit cache often wins.
• Low-volume, unpredictable traffic: don't cache at all. Measure first.

Structuring prompts for actual cache hits

This is where most teams leak savings. Caches are prefix-based. Any byte that differs before your cache breakpoint invalidates the hit.

Typical mistakes we've seen (and shipped):

• Injecting current_time or a request ID at the top of the system prompt.
• Shuffling the order of tool definitions between calls.
• Re-serializing JSON with different key ordering.
• Including the user's name or locale in the system message instead of the user turn.

The fix is boring and mechanical: lock the prefix, push volatility to the suffix.

# Bad: timestamp invalidates every call
system = f"You are an assistant. Current time: {now()}. Rules: ..."

# Good: static prefix, dynamic data in user turn
system = STATIC_SYSTEM_PROMPT  # byte-identical across calls
messages = [
    {"role": "user", "content": f"[time={now()}] {user_query}"}
]

For Claude specifically, you mark cache breakpoints explicitly:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_STATIC_INSTRUCTIONS,  # 20K+ tokens
            "cache_control": {"type": "ephemeral"}
        },
        {
            "type": "text",
            "text": retrieved_docs,  # changes per query — don't cache
        }
    ],
    messages=[{"role": "user", "content": user_query}]
)

You get up to four breakpoints on Claude. We typically use them for: tool schemas, system instructions, few-shot examples, and the top of a long conversation history. Everything downstream of a breakpoint is implicitly cached up to it.

Gemini's explicit cache object

Gemini makes you create the cache as a first-class resource:

from google import genai
from google.genai import types

client = genai.Client()

cache = client.caches.create(
    model="gemini-2.5-pro",
    config=types.CreateCachedContentConfig(
        system_instruction=STATIC_SYSTEM_PROMPT,
        contents=[large_document_corpus],
        ttl="3600s",
    )
)

response = client.models.generate_content(
    model="gemini-2.5-pro",
    contents=user_query,
    config=types.GenerateContentConfig(cached_content=cache.name)
)

The upside: you control TTL explicitly and can reference the cache by name across processes. The downside: you're on the hook for storage even when nobody calls.

RAG with caching: a pattern that works

For RAG, the naive instinct is "cache the retrieved chunks." That's wrong — retrieved chunks change per query, so they'll never hit.

The right pattern is a two-tier prompt:

1. Cached tier: system prompt, tool definitions, output schema, and a corpus-level summary or ontology that doesn't change per query.
2. Live tier: retrieved chunks and the user question.

For a legal doc assistant we built, the cached tier was about 18K tokens (instructions + a stable glossary + output JSON schema + 6 few-shot examples). The live tier averaged 4K tokens of retrieved passages. On Claude Sonnet with roughly 70% session reuse, our effective input cost dropped by about 55% in our testing — not the 90% the marketing math implies, because the live tier is still full price.

We write about related RAG tradeoffs in more depth over on the 72Technologies blog if you want the longer version.

Agents: where caching really pays

Agent loops are the ideal caching workload. The system prompt, tool definitions, and conversation history grow monotonically, and each tool-call iteration re-sends almost everything.

Without caching, a 10-step agent on a 50K-token working context pays for roughly 275K input tokens (the triangular sum). With caching and a well-placed breakpoint after the tool definitions, you pay full price once for the growing tail and cached price for the rest. In our agent builds, this is where we see the biggest absolute savings — often more than half the input bill.

One gotcha: if your agent framework re-serializes the message history on each turn (looking at you, several popular SDKs), you may be invalidating the cache without knowing it. We log the byte hash of the first 10K characters of each outgoing request and alert if it changes unexpectedly between turns in the same session.

Measuring what's actually happening

Trust nothing. Both providers return cache metrics in the response:

• Claude: usage.cache_creation_input_tokens and usage.cache_read_input_tokens.
• Gemini: usage_metadata.cached_content_token_count.

Ship these to your metrics backend from day one. The dashboard we care about:

• Cache hit rate = cache_read / (cache_read + cache_write + uncached_input)
• Effective $/1K input tokens = total_input_cost / total_input_tokens
• Write amplification = cache_write / cache_read (should trend toward zero; if it's above 0.5, your prefixes aren't stable)

If hit rate is below ~40%, caching is probably costing you money. Turn it off for that route and investigate.

The TTL trap

Claude's default ephemeral cache is 5 minutes. Anthropic also offers a 1-hour cache tier at a higher write premium. The 1-hour tier is tempting but only pays off if you have genuine traffic across that window — otherwise you're paying 2x write cost for prefixes that would've been reused in the first five minutes anyway.

We default to 5-minute and only move to 1-hour after we've measured sustained traffic.

Where we'd start

If you're adding caching to an existing LLM product, do this in order:

1. Measure first. Add cache-hit metrics to your existing calls. You can't improve what you can't see.
2. Freeze your prefixes. Audit every call for timestamps, request IDs, and shuffled JSON. Fix those before touching cache_control.
3. Start with agents and chat. They have the highest reuse density and the clearest win.
4. Pick the right provider per workload. Bursty session reuse → Claude ephemeral. Steady high-QPS → Gemini explicit cache with a tuned TTL.
5. Set an alert on write amplification. It's the earliest signal that something upstream broke your prefix stability.

Prompt caching isn't magic, but once your prefixes are clean and your metrics are honest, it's one of the few optimizations that pays for itself in the first week. If you'd like help wiring this into a production LLM stack, that's the kind of work we do on our AI engineering services.