Semantic Cache Hits Are Lying to You: Building a Similarity Layer That Actually Works

Semantic caching sounds like free money: embed the query, look up something 95% similar, return the cached answer, skip the model call. Then a support engineer notices the bot told one customer their refund window is 30 days and another it's 14 — from the same cached response. That's the failure mode nobody demos.

We've shipped semantic caches on top of Claude, GPT-4-class, and Gemini deployments for chat, search, and internal copilots. The pattern works, but only if you stop treating cosine similarity as a truth oracle. This is what we've learned about making it safe.

Why naive semantic caching fails

The standard recipe is three lines of pseudo-code: embed the incoming query, run a nearest-neighbour lookup against a vector store of past queries, and if the top match clears some threshold (0.92, 0.95, whatever), return its cached answer. It's cheap, it's fast, and it's wrong in ways that are hard to see in aggregate metrics.

The core problem is that embedding models were trained to cluster topically similar text, not semantically equivalent questions. "How do I cancel my subscription?" and "How do I pause my subscription?" sit shockingly close in embedding space across every major model we've tested — OpenAI's text-embedding-3-large, Google's text-embedding-004, Voyage, Cohere. The vectors say they're the same question. The correct answers are completely different.

A few specific ways this bites:

• Negation collapse. "Can I get a refund?" and "Can I not get a refund?" often land within 0.02 cosine of each other. Embeddings underweight function words.
• Entity swaps. "What's the API rate limit for the Pro plan?" vs "...for the Enterprise plan?" — high similarity, wrong answer.
• Temporal drift. A cached answer from March that says "the current version is 4.2" is still returned in November when 5.0 shipped.
• User context bleed. Two users with different permissions ask the same question. The cache doesn't know their auth scope.

If your evals only measure aggregate answer quality on a static test set, you won't catch any of this. The cache hit rate looks great and the offline eval passes because you're comparing cached answers to themselves.

Choose the right similarity space

The first fix is admitting that a single embedding is doing too much work. It's simultaneously supposed to encode topic, intent, entities, constraints, and negation. It can't.

What's worked for us is decomposing the cache key into layers:

Layer 1: Canonicalized intent

Before embedding, run the query through a cheap normalizer. This can be a small model (Haiku, GPT-4o-mini, Gemini Flash) with a tight prompt that extracts an intent label from a fixed taxonomy plus the salient entities. Something like:

{
  "intent": "refund_policy_lookup",
  "entities": {"plan": "pro", "region": "eu"},
  "polarity": "affirmative",
  "time_sensitive": true
}

Now your cache key isn't a vector — it's a structured tuple. Two queries only collide if their intent, entities, and polarity match exactly. Embeddings become a secondary signal used inside a bucket, not the primary lookup.

Yes, this costs a model call. In our experience the normalizer runs in 200–400ms on a fast small model, and it's cacheable itself with a simple hash on the raw input. Net effect on p50 latency has been small; net effect on wrong-answer rate has been large.

Layer 2: Embedding within a bucket

Inside each intent bucket, then you can use cosine similarity to find near-duplicates. The threshold can be more aggressive (0.88 or so) because the intent gate has already caught the dangerous mismatches. Negation and entity swaps can't cross bucket boundaries.

Layer 3: Freshness and scope guards

Every cache entry gets metadata: created-at, source-doc-versions-referenced, user-scope-hash, model-version. A hit only counts if all of these are still valid. This is the layer that stops November-you from serving March-answers.

A concrete lookup flow

Here's a stripped-down version of the flow we use, in Python-ish pseudocode:

def semantic_cache_lookup(query: str, user_ctx: UserContext) -> Optional[CachedAnswer]:
    # 1. Normalize
    norm = normalizer_model(query)  # returns intent, entities, polarity
    bucket_key = (norm.intent, tuple(sorted(norm.entities.items())), norm.polarity)

    # 2. Fetch candidates in bucket
    candidates = cache.get_bucket(bucket_key, user_scope=user_ctx.scope_hash)
    if not candidates:
        return None

    # 3. Embed and rank within bucket
    q_vec = embed(query)
    ranked = sorted(candidates, key=lambda c: cosine(q_vec, c.vec), reverse=True)
    top = ranked[0]
    if cosine(q_vec, top.vec) < 0.88:
        return None

    # 4. Freshness + version guards
    if top.created_at < freshness_floor(norm.intent):
        return None
    if any(v.is_stale() for v in top.source_versions):
        return None
    if top.model_version != current_model_version(norm.intent):
        return None

    return top

The important thing is the order. Cheap deterministic checks first, embedding math only inside a safe partition, then invalidation guards.

Sizing the freshness window

One knob that's easy to get wrong is TTL. A blanket 24-hour TTL is a decent default but wastes cache on stable content and holds volatile content too long.

We tag intents with a freshness class:

• Static (product concepts, definitions, how-tos on stable APIs): weeks to months
• Semi-static (pricing, feature lists, policies): hours to a day
• Volatile (inventory, account state, anything user-specific): don't cache, or cache with a per-entity invalidation hook

For RAG-backed answers specifically, tie the TTL to the underlying document version. When a source doc gets re-indexed, every cache entry that cited it gets invalidated. This is where storing source_versions on the cache entry pays off — invalidation becomes a set intersection, not a wall-clock check.

Measuring whether it actually works

Hit rate is the wrong headline metric. The metrics we track:

1. True hit rate — hits that pass a re-verification eval. We periodically sample cache hits, run them through the full non-cached pipeline, and diff the answers. Divergence over ~3–5% means the similarity layer is too loose.
2. Cost-per-resolved-query — total spend (models + embeddings + normalizer) divided by resolved queries. This is what actually matters. A cache that lifts hit rate but adds a normalizer call can still win or lose here depending on model prices.
3. Wrong-answer complaints per 1000 sessions — the ground truth. If this ticks up after a caching change, roll back regardless of what the offline numbers say.

We've had cases where tightening the similarity threshold from 0.90 to 0.94 halved the hit rate but cut wrong-answer complaints by roughly 70%. The cost-per-resolved-query barely moved because the wrong answers were generating expensive follow-up sessions.

A note on prompt caching vs semantic caching

Don't confuse the two. Anthropic's prompt caching and OpenAI's automatic prompt caching (both documented in their respective API references) cache prefixes of the prompt at the provider level. They give you a discount on repeated system prompts and long contexts. That's real money, and it composes fine with semantic caching — but it doesn't help you skip model calls entirely. Semantic caching is application-layer and it's the only one that can turn a $0.02 call into a $0.00 call.

When to just not cache

Some workloads shouldn't have a semantic cache at all. If your assistant is stateful, personalized, or acting on live data, the risk of stale or cross-user leakage outweighs the savings. In those systems we cache the retrieval step (which chunks match a query) and the tool-call planning step, but let the final generation run fresh every time. That still cuts cost meaningfully because retrieval and planning are where the expensive context lives.

For RAG systems where answers are grounded in documents you control, semantic caching is worth the effort. For agentic systems making side-effectful tool calls, the answer is almost always no — cache the reads, never the writes.

Where we'd start

If you're adding a semantic cache to an existing LLM product this quarter, do it in this order: ship the normalizer and intent bucketing first with caching disabled, so you can measure how often queries actually share intent. Then turn on caching for one or two static-freshness intents with aggressive re-verification sampling. Only widen the scope after you've watched the true-hit-rate and complaint metrics for a couple of weeks.

The teams that get burned are the ones who ship semantic caching as a one-week infra project. It's not infra — it's a product surface, and every hit is you speaking on behalf of your system without checking. Treat it that way and it pays for itself. Treat it as a Redis with vectors and it will quietly embarrass you.

If you want a sharper look at how this fits into a broader LLM stack, our notes on RAG and evaluation patterns go deeper on the retrieval side.