Reranking in RAG: When a Cross-Encoder Actually Pays for Itself

Rerankers get sold like magic dust: sprinkle one on top of your vector search and answers get better. Sometimes they do. Often they add 200ms of latency and a per-query bill for a lift you can't measure. This is a walk through when a cross-encoder reranker actually earns its slot in a RAG pipeline, and when you're better off fixing retrieval upstream.

What a reranker actually does

Your first-stage retriever — usually a bi-encoder vector search, BM25, or a hybrid of both — is optimised for recall at speed. It embeds the query once, does an ANN lookup over millions of chunks, and returns the top k candidates. It's fast because the query and documents are encoded independently.

A cross-encoder reranker is the opposite tradeoff. It takes the query and each candidate document together as a single input to a transformer, and outputs a relevance score. Because attention runs across the concatenated pair, it can catch nuance a bi-encoder misses — negation, entity mismatch, question–answer structure. The cost: you now do k forward passes per query instead of one.

Typical stack:

query
  -> hybrid retrieval (BM25 + vector) -> top 50 candidates
  -> cross-encoder rerank            -> top 5
  -> LLM answer synthesis

The common options in 2026: Cohere Rerank 3.5, Voyage's rerank-2, Jina Reranker v2, and open-weight BGE and mxbai rerankers you can self-host. Vendor feature claims (multilingual support, context length, latency SLAs) change often — check their current docs before you commit.

When a reranker is worth it

In our experience, rerankers pay off in a fairly narrow band of situations. The clearest wins:

• Your corpus has a lot of near-duplicates. Policy documents, product SKUs, legal clauses, support articles that were forked and lightly edited. Bi-encoders collapse these into similar embedding neighbourhoods; a cross-encoder can tell which variant actually matches.
• Queries are short and ambiguous. "reset password mobile" against a knowledge base with fifty password-related articles. The reranker uses token-level interaction to break ties.
• You need high precision at k=3 to 5. Anything downstream that shows citations to a user (chat with sources, agent tool grounding) benefits when the top few are actually the top few.
• Recall is already good, ranking is bad. If the right answer is in your top 50 but rarely in your top 5, that's textbook reranker territory.

And when it usually isn't worth it:

• Long, well-formed queries where the bi-encoder has enough signal already.
• Small corpora (a few thousand chunks) where you can just feed the top 20 to a capable long-context model and let it sort things out.
• Latency-critical paths — voice agents, autocomplete, anything under a 500ms budget.
• You haven't fixed chunking yet. A reranker cannot rescue a pipeline that's splitting mid-sentence or losing headers.

Prove it with a tiny eval before you ship it

The number one mistake we see: teams add a reranker, spot-check three queries, feel good, and move on. Then six months later nobody can say what it's doing. Build a small labelled set first.

You need maybe 100–300 real queries with the correct chunk IDs labelled. Pull them from logs, have a subject-matter expert mark them, and store them as JSON. Then measure NDCG@5 and Recall@5 with and without the reranker. Something like this is enough to start:

import numpy as np

def dcg(rels):
    return sum(r / np.log2(i + 2) for i, r in enumerate(rels))

def ndcg_at_k(predicted_ids, relevant_ids, k=5):
    rels = [1 if pid in relevant_ids else 0 for pid in predicted_ids[:k]]
    ideal = sorted(rels, reverse=True)
    return dcg(rels) / dcg(ideal) if dcg(ideal) > 0 else 0.0

def evaluate(pipeline, dataset, k=5):
    scores = []
    for row in dataset:
        preds = pipeline(row["query"])  # returns ordered chunk_ids
        scores.append(ndcg_at_k(preds, set(row["relevant_ids"]), k))
    return float(np.mean(scores))

baseline = evaluate(retrieve_only, eval_set)
with_rerank = evaluate(retrieve_then_rerank, eval_set)
print(f"NDCG@5: {baseline:.3f} -> {with_rerank:.3f}")

If the lift is under a few points of NDCG, you're paying latency for noise. If it's meaningfully higher on the queries you care about, you have a case.

Segment your eval, don't just average

Averages hide the interesting behaviour. Bucket your eval set by query type: short vs long, factoid vs procedural, in-domain vs edge case. Rerankers usually shine on one or two buckets and do nothing on the rest. That segmentation tells you whether to rerank conditionally — only for short queries, for example — instead of on every request.

The latency and cost math

Be honest about the second-stage bill. A hosted reranker call for 50 candidates typically adds 100–400ms end-to-end, depending on document length and provider. Self-hosted BGE-reranker-v2 on a warm GPU can be faster but you're paying for the GPU. Per-query pricing from hosted vendors sits in the fractions-of-a-cent range for small candidate sets and climbs with k and document length — check the current price sheet, they've moved several times.

A rough decision frame we use:

• If your p95 answer latency budget is > 3s (chat with synthesis), a reranker fits comfortably.
• If it's 1–3s, rerank a smaller candidate set (top 20, not top 100).
• Under 1s, skip it or precompute.

Don't rerank 100 candidates to pick 5. The marginal gain past k≈25 is usually noise, and you pay linearly.

Alternatives that often beat "just add a reranker"

Before reaching for the cross-encoder, try the boring stuff:

1. Hybrid retrieval. Combine BM25 with dense vectors using reciprocal rank fusion. In many corpora this closes most of the gap a reranker would.
2. Better chunking. Respect document structure — headers, list items, code blocks stay together. We've covered why fixed-size splitting fails elsewhere on the blog.
3. Query rewriting. Have a cheap LLM expand or decompose the query before retrieval. This helps bi-encoders more than reranking does, because it fixes the input side.
4. Metadata filters. If you know the user is asking about product X, filter to product X first. No amount of reranking substitutes for knowing which shelf to look on.
5. Long-context stuffing. For small corpora, retrieve top 20 with generous chunks and let a long-context model (Gemini 2.x, Claude, GPT with extended context) do the selection during synthesis. You're trading reranker cost for prompt tokens — sometimes it comes out cheaper, especially with prompt caching.

Rerank when these are exhausted, not before.

A realistic production pattern

Here's a shape that has worked for us on customer-facing RAG:

def answer(query: str, user_ctx: dict) -> Answer:
    filters = build_filters(user_ctx)  # tenant, product, locale
    candidates = hybrid_search(query, filters=filters, k=30)

    if should_rerank(query, candidates):
        candidates = rerank(query, candidates, top_n=5)
    else:
        candidates = candidates[:5]

    return synthesize(query, candidates)

def should_rerank(query: str, candidates: list) -> bool:
    if len(query.split()) < 6:
        return True
    top_scores = [c.score for c in candidates[:5]]
    if max(top_scores) - min(top_scores) < 0.05:  # tight cluster
        return True
    return False

Two things worth noting. First, should_rerank is a cheap gate — short queries and tight score clusters are exactly where the cross-encoder helps. Second, we cap candidates at 30, not 100. Diminishing returns kick in fast.

Watch for silent degradation

Rerankers get retrained. Providers push new versions. Your corpus drifts. Add the reranker's version to your logs, run your eval set weekly against production, and alert on NDCG drops. Without this, you'll one day discover the reranker has been hurting results for a month.

Where we'd start

If you're staring at a RAG pipeline wondering whether to add a reranker, do this in order. Build a 100-query labelled eval set from real logs — a day of work. Measure NDCG@5 and Recall@5 on your current retrieval. Try hybrid search and query rewriting first; re-measure. Only then bolt on a reranker, start with top 20 candidates, and compare on the same eval. Segment the results by query type. Ship the reranker only for the segments where it clearly wins, and log the version so you can catch regressions. If you'd like help wiring this into a production stack, our AI services team does exactly this kind of retrieval work.