Hybrid Search for RAG: BM25 + Vectors Without the Duct Tape

Every RAG system we've inherited from another team has the same bug: someone searches for an exact error code, a SKU, or a function name, and the vector index returns three semantically related but useless chunks. The fix is almost always hybrid search — BM25 for lexical precision, dense vectors for meaning, fused into one ranked list. It sounds simple. In practice, the details are where retrieval quality lives or dies.

Why pure vector search keeps failing you

Dense embeddings are trained to collapse synonyms and paraphrases into nearby points. That's exactly what you want when a user asks "how do I cancel my plan?" and the doc says "subscription termination steps." It's exactly what you don't want when the user pastes ERR_CERT_AUTHORITY_INVALID and expects the doc containing that literal string.

The failure modes we see most often on client audits:

• Identifier queries: SKUs, error codes, function names, ISO codes, model numbers. Embeddings smear these across neighbours.
• Rare tokens: A proper noun that only appears in one document will still lose to five documents with generically similar phrasing.
• Negation and quantifiers: "without SSO" often retrieves the SSO setup guide.
• Short queries: Under ~4 tokens, dense retrieval quality drops noticeably because there's not enough signal for the encoder.

BM25 handles the first three natively because it rewards exact term matches weighted by inverse document frequency. It also has zero inference cost and updates instantly when you add a document. The tradeoff: it's blind to synonyms and paraphrasing, which is why nobody ships pure BM25 for a modern assistant either.

What hybrid search actually means

Hybrid search runs both retrievers in parallel and merges their results. There are three merger strategies worth knowing:

1. Reciprocal Rank Fusion (RRF)

The default for a reason. You throw away raw scores (which are on incompatible scales) and use only the rank position from each list.

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: -x[1])

The k constant (typically 60, per the original Cormack et al. paper) dampens the influence of top ranks so a single retriever can't dominate. RRF is boring, robust, and needs almost no tuning. Start here.

2. Weighted score fusion

You normalise BM25 and cosine scores (min-max or z-score) and take a weighted sum: final = α * bm25_norm + (1 - α) * vector_norm. This can beat RRF when you have labelled data to tune α per query type, but normalisation is fragile — a single outlier score skews the whole batch. We only reach for this when we have real eval data and a query classifier.

3. Learned fusion (rerankers)

Run both retrievers, take the union of the top 20 – 50, then pass everything to a cross-encoder reranker. This is the strongest option and increasingly the default. It also adds latency and cost, which we've covered separately in our post on rerankers in RAG.

A concrete pipeline

Here's the shape of a hybrid retrieval step we've shipped multiple times, using OpenSearch (which has BM25 and a k-NN plugin in one engine) or Postgres with pgvector plus tsvector. The pattern is the same either way.

async def hybrid_retrieve(query: str, k: int = 10):
    # Run both retrievers concurrently
    bm25_task = asyncio.create_task(bm25_search(query, top_k=50))
    vector_task = asyncio.create_task(
        vector_search(embed(query), top_k=50)
    )
    bm25_hits, vector_hits = await asyncio.gather(bm25_task, vector_task)

    # Fuse with RRF
    fused = rrf([
        [h.doc_id for h in bm25_hits],
        [h.doc_id for h in vector_hits],
    ])

    # Hydrate top-k documents
    top_ids = [doc_id for doc_id, _ in fused[:k]]
    return await fetch_docs(top_ids)

A few things worth calling out:

• Retrieve deep, return shallow. Pull 50 from each side, return 10 (or fewer, after a reranker). Fusion only helps if both lists have enough candidates to agree or disagree on.
• Run them in parallel. BM25 typically returns in single-digit milliseconds; vector search takes 20 – 80ms depending on your index. Sequential calls waste your latency budget.
• Cache the query embedding, not just the results. If a user rephrases slightly, you're often recomputing the same vector.

The chunking question, briefly

BM25 and vector search have different opinions about chunk size. Dense retrievers work best on 200 – 500 token chunks — long enough for semantic context, short enough that the embedding isn't averaged into mush. BM25 is happier with longer chunks because more terms mean more IDF signal.

In practice we index the same chunks for both, sized for the vector side (~400 tokens with modest overlap), and accept that BM25 is slightly suboptimal. Maintaining two chunk sets doubles your indexing complexity and, in our experience, moves recall by low single digits — not worth it unless you're already at the ceiling on everything else.

Vendor reality check

A few things worth knowing before you pick an engine:

• OpenSearch and Elasticsearch both ship native hybrid search with RRF built in (see the OpenSearch hybrid query and Elastic's rank clause). This is the least-friction option for most teams.
• Postgres + pgvector works fine for under ~10M chunks. You'll write your own fusion in application code. Use ts_rank_cd for the lexical side and an HNSW index for vectors.
• Pinecone added sparse-dense hybrid via SPLADE-style sparse vectors rather than classic BM25. Read their docs carefully — the sparse encoder is a separate model you have to run and version.
• Weaviate exposes hybrid search with a tunable α parameter directly in the query. Convenient, but you still need evals to pick α.
• Vertex AI Search and Azure AI Search both offer managed hybrid with reranking. Fine if you're already in that cloud; expect to hit customisation limits on tokenisation eventually.

Don't take our word for any of this — vendor capabilities change every quarter. Check the current docs before you commit.

How to know it's actually working

Hybrid search is one of those changes that looks good in a demo and can quietly regress on real traffic. Before you ship, build a small eval set — 50 to 200 queries is enough to start — that specifically stresses the failure modes we listed above.

A useful breakdown:

• Lexical-heavy queries: error codes, product IDs, exact phrases in quotes.
• Semantic queries: paraphrases, questions in different words than the docs use.
• Mixed queries: "how do I fix ERR_CERT_AUTHORITY_INVALID on Chrome?"
• Adversarial: negations, misspellings, very short queries.

Measure recall@10 and MRR for each bucket, not just an aggregate number. We've seen hybrid improve overall recall by 5 – 15% while quietly dropping performance on one bucket — usually because RRF's k constant was wrong for that query shape, or the BM25 analyzer was stripping something important (stemming "iOS" to "io" is a classic).

Tokenisation is where the bodies are buried

If you take one thing from this article: audit your BM25 analyzer before blaming the fusion logic. Default analyzers lowercase, stem, and drop punctuation. That means React.useState, react usestate, and React use state all collapse to the same tokens — sometimes what you want, sometimes catastrophic. For code and identifier-heavy corpora, we usually run a custom analyzer that preserves case, splits on camelCase, and keeps dots and underscores as token boundaries rather than deletions.

Where we'd start

If you already have a vector-only RAG that's underperforming on specific queries, don't rebuild anything. Add BM25 alongside your existing index (Postgres full-text is fine for a proof of concept), fuse with RRF, retrieve 50 from each, return 10. Ship it behind a flag, run it on your eval set, and look at the per-bucket numbers before you look at the average. Nine times out of ten, that's enough to close the gap. The reranker and the α-tuning can wait until you have data telling you they're worth the latency.

If you're starting a new RAG project in 2026, just build hybrid from day one. The extra complexity is a few dozen lines of code; the recovery cost when you discover pure-vector isn't good enough is a migration.