Hybrid Search for RAG: Why BM25 Still Beats Pure Vectors on Half Your Queries

Pure vector search demos beautifully. Then a user types ERR_CONN_RESET, or searches for part number A1-884-X, or looks up a colleague named Ng, and your beautiful cosine similarity returns a thesis on networking philosophy. This is the gap hybrid search exists to close, and in 2026 it's still the single highest-ROI change most RAG systems can make.

The failure mode nobody shows in the demo

Embedding models are trained to capture semantic similarity. That's great for "how do I cancel my subscription" matching a doc titled "Ending your plan." It's terrible for anything where the exact token is the meaning: SKUs, error codes, function names, legal citations, drug names, person names, acronyms the model has never seen.

In our experience building retrieval for support bots and internal search, roughly a third to half of real user queries contain at least one token where lexical match matters more than semantic match. If your retriever is 100% dense vectors, those queries silently degrade. Users don't file bugs — they just stop trusting the product.

BM25, the ranking function that's been powering Lucene and Elasticsearch for two decades, handles exact-token queries natively. It's cheap, interpretable, and has no training cost. The question isn't whether to use it — it's how to combine it with vectors without creating a tuning nightmare.

What hybrid search actually means

Hybrid search = run both retrievers, then fuse the results. That's it. The interesting choices are:

1. Which lexical retriever — BM25 (classic), BM25F (field-weighted), or SPLADE (learned sparse)
2. Which dense retriever — your embedding model of choice, usually something like OpenAI text-embedding-3-large, Cohere embed-v3, or Voyage
3. How to fuse — score normalization + weighted sum, or Reciprocal Rank Fusion (RRF)
4. What to rerank — optional cross-encoder pass on the merged top-k

Most teams overcomplicate step 1 and underthink step 3. Start with vanilla BM25 and RRF. You can always upgrade later.

Why RRF usually wins over weighted score fusion

Weighted fusion looks clean on paper: final_score = 0.6 * vector_score + 0.4 * bm25_score. In practice, BM25 scores are unbounded and depend on corpus statistics, while cosine similarities sit in [-1, 1]. Normalizing across queries is fiddly, and the "right" weight drifts as your corpus grows.

Reciprocal Rank Fusion sidesteps the whole problem by only using rank position:

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: -x[1])

That k=60 constant comes from the original Cormack et al. paper and is a reasonable default. Elasticsearch, OpenSearch, and Weaviate all ship RRF implementations now, and the LangChain and LlamaIndex retriever wrappers expose it directly. See the Elasticsearch RRF docs for the engine-side version.

RRF has one real downside: it throws away score magnitude, so you lose the ability to threshold "this result is actually good" vs "this is the best of a bad batch." If you need confidence scores downstream (e.g., to decide whether to call the LLM at all), keep the raw scores alongside the RRF rank.

A minimal implementation that actually ships

Here's the pattern we've used on several production RAG systems, Postgres-flavoured because most teams already have Postgres:

import asyncio
from typing import List

async def hybrid_search(query: str, k: int = 20) -> List[dict]:
    # Run both retrievers concurrently
    vector_task = vector_search(query, k=k)
    bm25_task = bm25_search(query, k=k)
    vector_hits, bm25_hits = await asyncio.gather(vector_task, bm25_task)

    # RRF fusion
    fused = rrf([
        [h['id'] for h in vector_hits],
        [h['id'] for h in bm25_hits],
    ])

    # Hydrate top results with full documents
    doc_map = {h['id']: h for h in vector_hits + bm25_hits}
    return [doc_map[doc_id] for doc_id, _ in fused[:k]]

For the BM25 half in Postgres, tsvector with a GIN index gets you 80% of the way there and ranks reasonably with ts_rank_cd. If you need true BM25 scoring, the pg_search extension (ParadeDB) or a sidecar OpenSearch cluster is the move. For dense vectors, pgvector with HNSW indexes is the default in 2026.

If you're on a managed stack, Weaviate, Qdrant, and Elasticsearch all do hybrid natively — you pass the query once and get fused results back. Pinecone added sparse-dense hybrid a while ago; check the current Pinecone hybrid docs for the exact index config, as the API has shifted.

The query-side gotcha

One subtle trap: your dense retriever wants the full natural-language query ("how do I reset my password on mobile"), but BM25 benefits from stopword removal and sometimes stemming. Don't pre-process the query once and pass it to both. Keep two query representations:

vector_query = raw_query                      # untouched
bm25_query = preprocess_for_bm25(raw_query)   # stopwords, stemming, maybe synonyms

Same for indexing: store the original text for embeddings, store the analyzed form for BM25. Most hybrid systems that "don't work" have one side of this backwards.

When hybrid is overkill, and when it's not enough

Skip hybrid if

• Your corpus is small (<10k docs) and queries are conversational. Pure dense is fine; the retrieval error floor is dominated by chunking, not ranking.
• Your users never type identifiers, codes, or proper nouns. Rare, but it happens — think meditation-app content search.
• You're latency-bound below 50ms end-to-end. Running two retrievers in parallel is cheap but not free, and the fusion step adds a hop.

Hybrid alone isn't enough if

• You have strong recency requirements (news, logs, tickets). Add a time-decay boost on top of the fused score.
• Results need to respect per-user permissions. Filter before retrieval, not after — otherwise your top-k is mostly documents the user can't see.
• You're hitting a precision ceiling around 0.7–0.8. That's when a cross-encoder reranker on the top 20–50 fused results starts earning its latency. We've written about when rerankers earn their keep in more detail.

Evaluating hybrid vs dense honestly

Don't trust vibes. Build a labelled set of 100–300 real queries with known-good document IDs, then measure Recall@10 and MRR@10 for three configurations: dense only, BM25 only, and hybrid. We've consistently seen hybrid beat the better of the two single retrievers by 5–15 points of Recall@10 on heterogeneous corpora. On homogeneous conversational corpora, the lift can be as small as 1–2 points — still worth it, but don't promise your PM a miracle.

Split your eval set by query type: natural language, keyword-heavy, identifier-containing, misspelled. Hybrid's gains are concentrated in the last three categories, and that's exactly where single-retriever systems quietly fail in production.

A note on cost

Hybrid adds roughly one extra index to maintain and one extra query per request. BM25 queries on a well-indexed corpus are typically sub-10ms; the dominant cost stays the embedding call for the query itself. If you're already caching query embeddings for repeated searches (and you should be), the marginal cost of adding BM25 is noise.

Where we'd start

If you have an existing dense-only RAG system, don't rewrite anything. Stand up a BM25 index over the same chunks, run both retrievers in parallel, fuse with RRF, and ship it behind a feature flag. Build a 150-query eval set from real logs — bucket by query type — and compare. You'll know within a day whether hybrid earns its keep on your corpus.

If you're starting fresh, pick a store that does hybrid natively (Weaviate, Qdrant, OpenSearch, or Elasticsearch) so you don't have to orchestrate two systems. Default to RRF with k=60. Add a reranker only after you've exhausted the easy wins in chunking and query rewriting. And log the per-retriever ranks alongside the final result — six months from now, when something regresses, you'll want to know which half of the pipeline moved.