Rerankers in RAG: When a Second Pass Actually Earns Its Latency

Every RAG stack eventually hits the same wall: retrieval returns technically-relevant chunks, the LLM writes a technically-correct answer, and users still say it feels off. The instinct is to swap embedding models or crank up top_k. The cheaper win is usually a reranker — but only if you know what you're buying.

This is a walkthrough of when a second-pass reranker earns its latency, when it's dead weight, and how we decide on client projects.

What a reranker actually does

A vector search returns the top N chunks by cosine similarity between a query embedding and document embeddings. That's a bi-encoder: query and doc are encoded independently, then compared. It's fast and cacheable, but it never lets the query and the document "look at each other" during encoding.

A reranker fixes that. You take the top 20–100 candidates from vector search and feed each (query, chunk) pair into a second model that scores relevance jointly. The two common flavours:

• Cross-encoder rerankers — models like bge-reranker-v2-m3, Cohere Rerank 3, or Voyage's rerank models. They output a single relevance score per pair. Fast enough for production (tens of ms per batch on GPU, ~100–300ms via API for 50 candidates).
• LLM-as-reranker — you prompt a small LLM (Haiku, GPT-4.1-mini, Gemini Flash) to score or rank the candidates. More flexible, more expensive, higher latency, and — in our experience — noisier unless you constrain the output tightly.

Cohere and Voyage both document their rerank endpoints as drop-in second-stage models designed exactly for this pattern; check their current docs for token limits and pricing before committing.

When a reranker actually helps

Rerankers pay off when your retrieval is recall-decent but precision-weak. Concretely, that looks like:

• The right chunk is usually somewhere in your top 20, but rarely in the top 3.
• Queries are short or ambiguous ("how do I cancel?") and embeddings surface too many surface-similar chunks.
• Your corpus has near-duplicates — policy docs with versioned language, product SKUs with tiny variants, FAQ entries that paraphrase each other.
• You're doing hybrid search (BM25 + vectors) and need a principled way to merge scores.

In those cases, a cross-encoder can lift nDCG@5 or MRR meaningfully — enough that end-users notice fewer "why did it cite that?" moments.

When it doesn't help

Skip the reranker (or at least deprioritise it) when:

• Your top_k=5 recall is already above ~90% on a real eval set. There's nothing to rerank.
• Your chunks are long (2k+ tokens) and few. The bi-encoder is already doing most of the work.
• Latency budget is tight and the query is well-formed (structured filters, exact IDs, code lookups).
• Your bottleneck is chunking or query rewriting, not ranking. Fix upstream first.

The most expensive mistake we see: teams add a reranker before they have an eval set. They "feel" it's better, ship it, and eat 300ms of p95 latency for a benefit they can't measure.

Build the eval before you build the reranker

You need a labelled set. It doesn't have to be huge — 100–300 real queries with the correct chunk (or chunks) marked is enough to make decisions.

A minimal eval loop looks like this:

from dataclasses import dataclass

@dataclass
class EvalCase:
    query: str
    relevant_chunk_ids: set[str]

def recall_at_k(retrieved_ids: list[str], relevant: set[str], k: int) -> float:
    hits = sum(1 for cid in retrieved_ids[:k] if cid in relevant)
    return hits / max(len(relevant), 1)

def mrr(retrieved_ids: list[str], relevant: set[str]) -> float:
    for i, cid in enumerate(retrieved_ids, start=1):
        if cid in relevant:
            return 1.0 / i
    return 0.0

def evaluate(pipeline, cases: list[EvalCase], k: int = 5):
    r_at_k, mrrs = [], []
    for case in cases:
        ids = pipeline(case.query)
        r_at_k.append(recall_at_k(ids, case.relevant_chunk_ids, k))
        mrrs.append(mrr(ids, case.relevant_chunk_ids))
    return {
        f"recall@{k}": sum(r_at_k) / len(r_at_k),
        "mrr": sum(mrrs) / len(mrrs),
    }

Run this against two pipelines: vector-only, and vector + reranker. If MRR jumps from, say, 0.42 to 0.71 and recall@5 goes from 0.78 to 0.93, the reranker is earning its keep. If MRR moves by 0.03, you have your answer.

Don't skip the failure buckets

Aggregate numbers hide the interesting cases. Split your eval set by query type — navigational, comparative, long-tail, multi-hop — and look at each bucket. We've had projects where a reranker lifted overall MRR by 0.15 but hurt long-tail queries because the model was over-weighting lexical overlap. That's the kind of thing you only see if you slice.

The latency math

Here's the tradeoff nobody wants to do on the whiteboard.

Assume vector search takes 40ms, generation takes 1.8s, and a hosted reranker adds 250ms for 50 candidates. That's a ~12% increase in end-to-end latency. Whether that's acceptable depends on:

• Streaming: if you're streaming tokens, users perceive time-to-first-token, not total time. Reranker latency lands before generation starts, so it does delay TTFT. Users notice.
• Parallelism: you can run reranking in parallel with query expansion, safety checks, or metadata lookups. Free wins.
• Batching: if multiple queries hit within a window (multi-turn agents, batch jobs), rerankers batch beautifully. Per-query cost drops.

A rough rule we use: if the reranker doesn't measurably move a metric users care about (answer correctness, citation accuracy, follow-up rate) by at least a couple of percentage points, it's not worth the p95 hit.

Hosted vs self-hosted

Three real options:

1. Hosted API (Cohere Rerank, Voyage rerank, Jina). Fastest to integrate, predictable quality, per-1k-search pricing. Good default for teams that don't already run GPU inference.
2. Self-hosted open weights (BGE reranker family, mxbai-rerank). Cheaper at scale, but you own the GPU, the autoscaling, and the eval-when-they-release-a-new-version problem. Worth it above ~500k rerank calls/day, roughly.
3. LLM-as-reranker. Use a small, fast model with a tight prompt returning a JSON array of ranked IDs. Flexible — you can inject business rules ("prefer chunks tagged official") — but slower and pricier per call than a purpose-built reranker. Reasonable when your ranking logic isn't purely semantic.

Check each vendor's current docs for max tokens per document, batch limits, and regional endpoints — these change often and can force architectural decisions.

A pattern that works well

For most production RAG stacks we build, the shape is:

query -> query rewrite (optional) -> hybrid retrieval (BM25 + vectors, top 50)
      -> cross-encoder rerank (top 5)
      -> LLM answer with citations

The query rewrite step matters more than people think — a reranker can only sort what retrieval gave it. Garbage top-50, garbage top-5.

Cost guardrails

A few things we always instrument:

• Per-request cost breakdown: log the cost of embed, retrieve, rerank, and generate separately. Reranker cost is small per call but scales with top_k_before_rerank.
• Rerank-off canary: keep a small percentage of traffic running without the reranker so you have a live control group. If your quality metric degrades, you'll see it before users do.
• Cap top_n_input: reranking 100 candidates is rarely worth it over 30–50. Diminishing returns kick in fast.
• Cache aggressively: identical (query, candidate_set) pairs happen more often than you'd think, especially in agent loops. A short-TTL cache on rerank results is cheap insurance.

Where we'd start

If you're staring at a RAG system that feels almost-right and wondering whether a reranker is the fix:

1. Build the 100–300 query eval set first. No reranker decision is defensible without it.
2. Measure recall@20 and MRR on your current pipeline. If recall@20 is low, fix retrieval before you rerank — a reranker can't recover chunks it never sees.
3. Drop in a hosted reranker (Cohere or Voyage) as the fastest A/B. One afternoon of work.
4. Compare MRR, recall@5, and — critically — a human-graded sample of 30 answers. Numbers can move without answers improving.
5. If the win is real, decide on hosted vs self-hosted based on volume, not vibes. Below a few hundred thousand calls a day, hosted almost always wins on total cost of ownership.

Rerankers aren't magic, and they aren't free. But when retrieval is the bottleneck and you've done the boring work of measurement, a second pass is one of the highest-leverage changes you can make to a RAG stack. If you'd like help pressure-testing yours, our team does this kind of work on AI engagements regularly.