Rerankers in RAG: When Cross-Encoders Actually Earn Their Latency

Every team that ships a RAG system eventually hits the same wall: the retriever returns 20 chunks, the top one looks great, and the actual answer is sitting at position 14. The default fix is a reranker. The real question is whether the extra 150–400ms is worth it for your product — and in a surprising number of cases, it isn't.

This is a field guide based on what we've seen building RAG for support bots, internal knowledge tools, and document-heavy B2B workflows. We'll cover when rerankers help, when bi-encoders are enough, how to wire up Cohere, Voyage, or an open-source cross-encoder without wrecking latency, and the eval setup that tells you whether the thing is actually earning its keep.

What a reranker actually does

A typical RAG retriever uses a bi-encoder: the query and each document are embedded independently, and similarity is computed with a dot product or cosine. It's fast because embeddings are precomputed and ANN indexes (HNSW, IVF, ScaNN) make nearest-neighbor lookup sub-10ms at millions of vectors.

A cross-encoder reranker is different. It takes (query, document) as a single input and runs a transformer over the pair, producing a relevance score. There's no shortcut — you pay a forward pass per candidate. That's why rerankers sit on top of retrieval: you grab the top 50–100 with the bi-encoder, then rerank down to 5–10.

The quality gap is real. Cross-encoders see token-level interaction between the query and the document, so they catch things like negation, qualifier mismatches ("Python 2 vs Python 3"), and entity disambiguation that pure embedding similarity misses.

The vendors worth knowing in 2026

• Cohere Rerank 3.5 — multilingual, strong on long documents, hosted API. See the Cohere rerank docs for current context limits.
• Voyage rerank-2 — competitive quality, often cheaper per 1K tokens.
• BGE reranker v2 (BAAI) — open weights, run on your own GPU. Good when you can't ship data to a third party.
• Jina Reranker v2 — small, fast, decent quality for the price.

We're deliberately not quoting benchmark numbers. Public MTEB leaderboards shift monthly and your corpus is not their corpus. Run your own evals.

When a reranker is worth it

In our experience, rerankers earn their latency in three situations:

1. High-recall retrieval, low-precision ranking. If you need top_k=5 for the LLM but the right chunk is often in positions 10–30 of your bi-encoder results, a reranker is the cleanest fix. This is common when your corpus has many near-duplicates (product variants, versioned docs, policy pages).
2. Long or jargon-heavy queries. Bi-encoders compress queries into a single vector and lose nuance. "Does our refund policy apply to annual subscriptions cancelled within 14 days under the EU directive?" is the kind of thing cross-encoders handle noticeably better.
3. Multilingual or domain-shifted corpora where your embedding model wasn't trained on your vocabulary. Rerankers (especially multilingual ones) often rescue weak embeddings without the pain of fine-tuning.

When it's a waste

Skip the reranker — or at least don't start with one — if:

• Your corpus is small (under ~50K chunks) and your bi-encoder recall@20 is already above ~0.95 on your eval set. The reranker has nothing to fix.
• Your product has strict sub-second latency budgets and you're already streaming. Adding 300ms before the first token is noticeable.
• You're using hybrid search (BM25 + vector + reciprocal rank fusion) and it's already giving good top-5 precision. Hybrid is cheaper and often closes most of the gap.
• Your main failure mode is bad chunks, not bad ranking. A reranker can't fix a chunk that doesn't contain the answer. Fix chunking first.

That last one is the most common mistake we see. Teams add a reranker to paper over retrieval problems that are really chunking or indexing problems. Rerankers reorder; they don't conjure.

A sane pipeline

Here's roughly what we deploy when a reranker makes sense. Hybrid retrieval first, rerank second, then feed the LLM.

from typing import TypedDict
import cohere
from qdrant_client import QdrantClient

class Chunk(TypedDict):
    id: str
    text: str
    score: float

co = cohere.ClientV2()
qdrant = QdrantClient(url="...")

async def retrieve(query: str, embed_vec: list[float]) -> list[Chunk]:
    # 1. Hybrid search: vector + BM25, fused server-side
    hits = qdrant.query_points(
        collection_name="docs",
        query=embed_vec,
        limit=50,
        with_payload=True,
    ).points
    return [{"id": h.id, "text": h.payload["text"], "score": h.score} for h in hits]

async def rerank(query: str, candidates: list[Chunk], top_n: int = 6) -> list[Chunk]:
    if len(candidates) <= top_n:
        return candidates  # don't pay for a no-op

    resp = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=[c["text"] for c in candidates],
        top_n=top_n,
    )
    return [candidates[r.index] | {"score": r.relevance_score} for r in resp.results]

A few things worth calling out:

• Skip the rerank call when candidates ≤ top_n. Easy win, often forgotten.
• Cap candidates at 50–100. Reranker latency scales roughly linearly with document count. Going from 50 to 200 rarely improves end-to-end quality enough to justify the extra time.
• Truncate documents to the reranker's effective context. Most hosted rerankers handle 4K–8K tokens per document, but quality degrades on very long inputs. If your chunks are big, consider reranking on a summary field.

Latency budgeting

A realistic budget for a chat RAG endpoint with time-to-first-token under 1.5s:

• Embed query: 20–60ms
• Vector + BM25 retrieve (top 50): 30–80ms
• Rerank 50 → 6: 150–400ms (hosted) or 80–200ms (local BGE on a warm GPU)
• LLM first token: 400–900ms

If you're already near the ceiling, the reranker is the first thing to cut or move async. One pattern we like: show the LLM's answer using the bi-encoder results, then quietly rerank in the background to warm a cache for follow-up turns in the same conversation.

Evaluating whether it actually helps

Don't ship a reranker on vibes. Build a small eval set — 100–300 real queries with human-labeled relevant chunk IDs is enough to see signal — and measure:

• Recall@k before and after reranking (should stay flat; reranking doesn't add candidates)
• nDCG@k or MRR before and after (this is where you want to see improvement)
• Answer correctness from an LLM-as-judge or human eval, measured end-to-end

The end-to-end number is the only one that matters for shipping. We've seen cases where nDCG improved meaningfully but answer quality didn't move, because the LLM was already robust to a slightly-worse top-5. If the user-visible metric doesn't move, don't pay the latency.

We wrote more about this kind of regression testing in our piece on LLM eval suites — the same discipline applies to retrieval changes.

Cost guardrails

Hosted rerankers are usually priced per search (where "search" = one query against N documents, up to a cap). Rough mental model for 2026:

• Cohere / Voyage rerank: a fraction of a cent per search at typical candidate counts
• Self-hosted BGE on a shared L4 or A10: effectively free marginal cost, but you own the GPU bill

For high-QPS products (support chat, in-product search), self-hosting a small open reranker often wins on total cost once you're past a few hundred thousand searches per month. For low-QPS internal tools, hosted is almost always cheaper than babysitting a GPU.

Two guardrails we always put in:

• Per-tenant rate limits on rerank calls, separate from LLM rate limits. A runaway crawler on your search endpoint can rack up rerank bills fast.
• Short-TTL cache keyed on (query_hash, candidate_ids_hash). Same query over the same candidate set returns the same ranking; no need to pay twice within a few minutes.

Where we'd start

If you're adding retrieval today, don't start with a reranker. Build hybrid retrieval (vector + BM25), get your chunking right, and measure recall@20 on a real eval set. If recall is low, fix retrieval. If recall is high but precision at top-5 is poor, then bolt on a reranker — hosted first for speed to ship, with a plan to migrate to a self-hosted open model if volume justifies it. Measure end-to-end answer quality, not just ranking metrics, and be honest about whether the latency is buying you anything a user will notice. If it isn't, rip it out. The best RAG pipeline is the simplest one that hits your quality bar.