Hybrid Search Combining BM25 and Dense Retrieval

A customer types "SKU AZ-4471" into a support chatbot. The document with that exact string sitting in the knowledge base never got surfaced. A retrieval system that relies on dense vector search alone misses exact-string matches like this, and it's one half of a two-sided failure pattern that shows up across nearly every production retrieval-augmented generation system.

Dense retrieval fails this way because of how embedding models work. But that same compression erases the identity of specific strings. A product SKU, an error code like ERR_SSL_VERSION_OR_CIPHER_MISMATCH, a function signature, an invoice number, a regulatory clause reference: none of these have meaning in the way a sentence does. Averaged into a vector alongside the rest of a passage's tokens, the identifier dissolves into a representation of "document about this general topic" rather than retaining "document containing this exact string.

BM25 fails in the opposite direction. A user asks "how do I return a damaged item?" and the relevant article is titled "Refund Policy for Defective Products." Not a single token overlaps between the query and the title. Because BM25 works off an inverted index that matches literal tokens, it finds nothing, even though the article answers the question perfectly. The same gap opens up for synonyms, for different phrasings of the same intent, for queries in a second language.

The two failure modes don't overlap, so a system built on only one of these retrievers doesn't fail randomly. It fails on a predictable, describable class of input, every time.

The dense failure mode deserves more attention because it's the dangerous one. Nothing in the logs flags that anything went wrong.

This silence compounds in an especially damaging way when eval sets get built from production query logs. The fix is a different eval set, one built to include the failure mode in the first place.

How BM25's design choices create its failure

BM25 works off an inverted index: for every term that appears anywhere in the corpus, the index stores which documents contain it and how often. Two design choices inside that scoring function determine what BM25 is good at and what it can never do.

The first is term frequency saturation. Each additional occurrence adds diminishing value to the score. This keeps a document from gaming the ranking through repetition and keeps the score grounded in whether the term is present and relevant, not how many times it was stuffed in.

The second is length normalization. A ten-thousand-word document that happens to contain a query term shouldn't automatically outrank a two-hundred-word document that's actually about the query, just because it has more words to work with.

Together, these mechanics produce scores that are unbounded positive numbers, sensitive to document length and to the size and composition of the corpus. That property matters later: it's the root of the score incompatibility problem that breaks naive fusion with dense retrieval.

The deeper consequence of how BM25 matches terms: a term is either present or it isn't. There's no blurring, no partial credit for something close in meaning. A rare identifier appearing in five documents out of a million-document corpus gets matched with exactly the same mechanical precision as a common term appearing in five million documents. That precision is BM25's strength on exact-match queries and its hard limit everywhere else. BM25 has no built-in concept of synonymy or conceptual equivalence. It only knows whether a string is there.

That precision comes at a low operational cost. BM25 needs no neural inference at query time, running entirely on CPU against the inverted index, which makes it fast and cheap even at large scale. BM25 has been the default for keyword search in Lucene and Elasticsearch since around 2016, though the underlying algorithm was first published in the 1990s, and it remains the backbone of Elasticsearch, OpenSearch, Solr, and Lucene today.

What the embedding compression in dense retrieval trades away

Dense retrieval replaces token matching with geometry. A neural encoder maps text, both the query and every document chunk, into a high-dimensional embedding space. Retrieval becomes a similarity search: find the documents whose vectors sit closest to the query vector, measured by cosine similarity or dot product.

Embedding dimensions typically run from 384 to 3072. The same compression averages away the one thing BM25 preserves perfectly: the identity of a specific string. A serial number or an error code gets folded into the same vector as the surrounding sentence, and whatever made that string unique gets diluted into a general topic representation.

Searching that space at production scale introduces another layer. Pinecone uses its own proprietary indexing approach. Whichever implementation is in play, the trade is the same: a small loss in recall in exchange for search speed that makes dense retrieval viable in production at all.

The output of that search is a bounded number, cosine similarity typically falls between -1 and 1. These two distributions don't share a scale, a unit, or a range. That mismatch means combining the two retrievers is a matter of reconciling two numbers that were never designed to be compared, not a matter of picking weights and averaging.

What score incompatibility means in production

The instinct, once a team has both a BM25 retriever and a dense retriever running, is to combine their scores: multiply each by a weight, add them together, rank by the sum. This fails in production, and it fails for a structural reason rather than a tuning oversight. BM25 scores are unbounded and shaped by corpus-specific factors like document length and term rarity. Dense similarity scores are bounded, usually within -1 to 1. A weighted average of the two numbers gets dominated by whichever distribution happens to produce the larger magnitude for a given query or a given corpus, and that's not something a fixed weight can reliably correct for.

Assembled.com ran into this directly. The company switched to a rank-based fusion method after struggling to find a workable alpha.

The ranking itself absorbs the practical damage. The combination produces a result that's worse than just trusting either retriever on its own, which defeats the entire reason for combining them in the first place.

Score normalization, scaling each retriever's scores into a common range before averaging, looks like a fix but introduces a different instability. The fix moves the problem elsewhere in the pipeline.

How Reciprocal Rank Fusion resolves score incompatibility

Reciprocal Rank Fusion sidesteps the entire problem by refusing to compare scores at all. Instead of asking how high a document scored, RRF asks where it ranked, position one, position five, position forty, in each retriever's list, and does its arithmetic on those positions.

The formula is simple. Each document's contribution from a given retriever is 1/(k + rank), with k set to 60 by convention. The constant k softens the advantage of the very top rank so a single first-place finish doesn't overwhelm everything else in the merge.

Because the entire calculation runs on rank position rather than raw score, the incompatibility between BM25's unbounded range and dense retrieval's bounded range stops mattering. The units never enter the equation, so the units never need to match.

RRF also needs no labeled data and no parameter tuning beyond the conventional k value, which makes it a reasonable default for teams starting from zero, with no query-relevance labels to calibrate against. Elasticsearch 8.x (on its Enterprise tier), OpenSearch 2.12 and later, Weaviate, and Qdrant all ship RRF as a built-in option.

RRF has a real limitation: the confidence gap between those two results disappears entirely.

That cost is visible in benchmark numbers. Past that point, a learned weighting scheme or a small reranker sitting over the merged list tends to perform better than stacking more retrievers into the same RRF formula.

The three-stage production pipeline that connects retrieval to the LLM

A production hybrid retrieval system runs in three sequential stages, dual retrieval, fusion, and reranking, and each stage does a job the other two can't substitute for.

Stage one sends the query to both indexes at once. The sparse index, BM25 or SPLADE, and the dense ANN index each process the query independently and return their own ranked candidate list. Running these in parallel rather than one after the other keeps the total latency bounded by the slower of the two, not the sum of both. This is also the stage where ingestion-time decisions pay off or don't: documents get chunked using recursive character splitting, typically into windows of 256 to 512 tokens with 10 to 15 percent overlap between chunks. That range gives dense embedding models enough surrounding context to produce a meaningful vector, while keeping term density concentrated enough for BM25's inverse-document-frequency calculations to still carry signal.

Stage two merges the two ranked lists using RRF, with k set to 60 by convention. This is the step where the score incompatibility problem from the earlier section gets resolved, producing a single unified ranking built from rank position rather than from two incompatible score distributions. The computational cost in the pipeline sits in stage one's parallel retrieval, not in the rank arithmetic that follows it.

Stage three reranks. A cross-encoder reranker reads the query and each candidate passage together, rather than comparing two pre-computed vectors the way dense retrieval does, and produces a fresh relevance score for each one. Against a multi-hundred-millisecond LLM call, that added latency is often acceptable given what it buys: cross-encoders deliver real precision gains on top of hybrid retrieval alone, and this is the stage that earns back the token budget spent on the LLM call that follows it.

The separation across these three stages is deliberate. Hybrid retrieval maximizes the odds that the correct document made it into the set at all. Reranking maximizes the odds that it reaches the language model's context window in a position where the model actually uses it.

BM25 vs. SPLADE: choosing the sparse component for your constraints

The sparse half of this pipeline isn't limited to BM25. SPLADE, Sparse Lexical and Expansion model, is a learned alternative that addresses much of what BM25 can't do, at the cost of infrastructure BM25 doesn't need.

It's CPU-only, requires zero training, and fits cleanly into offline-first or edge deployments where a GPU dependency isn't an option.

SPLADE works differently. The output stays sparse and interpretable, compatible with the same inverted-index infrastructure BM25 uses, but it closes the synonym gap that BM25 structurally can't close.

That capability comes with a cost. CPU execution and inference-free variants of SPLADE do exist, so the dependency isn't absolute. For most deployments, it's a real infrastructure decision.

The choice between the two comes down to latency budget, infrastructure constraints, and whether enough in-domain labeled data exists to make SPLADE's learned expansion worth the added complexity, not purely to which one scores better in isolation. SPLADE earns its keep specifically where the infrastructure and the data line up to support it.

The limits of hybrid search degradation

Hybrid search improves retrieval reliably when both component retrievers are reasonably strong on their own. When one of them is weak, the fusion step doesn't rescue it, and the combined result can end up worse than just using the stronger retriever by itself.

The degradation originates in candidate generation, before fusion or reranking ever enter the picture, and no amount of sophistication at the reranking stage fixes a problem that started with a weak retriever feeding bad candidates into the merge in the first place.

The implication is direct: before committing to a hybrid architecture, both retrieval paths need to be benchmarked independently against the target corpus. It dilutes the sparse signal that was working fine on its own, and the fused result inherits that dilution.

Domain and language matter here too. Research on non-English hybrid retrieval in the legal domain, published at COLING 2025 under the title "Know When to Fuse: Investigating Non-English Hybrid Retrieval in the Legal Domain," found that the benefits of fusion aren't uniform across languages or across different legal text structures. Financial documents are a useful case in point: heavy use of identifiers, numbers, and exact terminology is exactly where BM25 already performs substantially better than dense retrieval on its own, so the marginal value hybrid adds there is smaller than it is in general-domain corpora where both retrievers are already competitive with each other.

This doesn't settle the RRF-versus-weighted-fusion question in RRF's favor by default. Teams sitting on labeled query-relevance data should benchmark weighted fusion before committing to RRF long-term. RRF is a safe starting point, not a ceiling on what the system can achieve.

Native hybrid search implementation and adoption

Native hybrid search support is now standard across the major retrieval platforms, which shifts where the engineering effort actually goes: less time spent implementing fusion from scratch, more time spent configuring and evaluating it correctly for a specific corpus.

OpenSearch combines BM25, k-NN, and neural sparse retrieval through search pipelines with RRF built in natively. Weaviate exposes dual retrieval and fusion behind a single hybrid query call. Qdrant stabilized weighted RRF in its v1.17 release. Elasticsearch has made Better Binary Quantization the default for dense vector fields of 384 dimensions or more, a platform-level optimization that cuts down the memory cost of maintaining two indexes side by side. Milvus has introduced lake-native indexing alongside SINDI, plus Block-Max WAND and Block-Max MaxScore for sparse execution. Chroma now supports native lexical retrieval, BM25 and SPLADE both, through a built-in Search() API that handles hybrid search and RRF fusion inside the platform itself.

What's left, once a platform is chosen, isn't the mechanics of fusion. A representative eval set has to deliberately include queries with exact identifiers, rare named entities, and domain-specific jargon, the same query types that motivated the move to hybrid search to begin with.

The practical decision sequence for building a hybrid retrieval system

Everything above points to a specific order of operations. Building hybrid search out of sequence, picking a fusion strategy before checking whether both retrievers are individually sound, tends to produce a system optimized against the wrong baseline from the start.

Start by establishing baselines independently. Fusion doesn't rescue a weak component.

Choose the sparse component next. SPLADE's advantage over BM25 only appears when those three conditions are met together.

Choose a fusion strategy third. Move to weighted alpha fusion only after labeled query-relevance data has accumulated and nDCG@10 improvement has been benchmarked against the RRF baseline.

Add reranking on top of the fused list fourth. Pick the specific reranker model only after benchmarking its latency on the actual hardware it will run on, since reranker choice is the dominant lever on total latency once the candidate set size is fixed.

Build an eval set that reflects the real failure modes last, though in practice it should run alongside every step before it. Without that kind of eval set, tuning alpha or swapping reranker models produces changes that look neutral on the metrics while being quietly better or worse for the actual users typing actual queries into the system.

At ingestion, carry the chunking decision through all of it: 256 to 512 token windows, 10 to 15 percent overlap, a single range that serves both the dense embedding model and BM25's term-frequency calculations without requiring two separate chunking strategies for two separate indexes.