How I Reproduced BM25, Dense Retrieval, and SPLADE on a 16GB MacBook

(RAG) programs always say issues like “dense retrieval beats BM25” or “hybrid search works higher,” and more often than not, no person concerned has really reproduced the baseline being in contrast towards. BM25 is a key phrase scoring technique that has been used for many years and nonetheless reveals up because the baseline in almost each retrieval paper printed right this moment. Dense retrieval is now commonplace in manufacturing search. It makes use of a skilled mannequin to transform textual content into embeddings, then ranks paperwork by how carefully their embeddings match the question. Discovered sparse retrieval, a more recent strategy that sits conceptually between the 2, is spreading quick. Many individuals repeat claims about BM25, dense retrieval, and SPLADE, however far fewer have run all three themselves on actual benchmark knowledge and checked whether or not their outcomes match printed scores.

I did, on a MacBook with 16GB of reminiscence, utilizing a public set of workouts known as the Foundations of Retrieval onboarding path, maintained by the Castorini data retrieval analysis group. The trail makes use of two open-source toolkits from Castorini: Anserini, which is Java-based, and Pyserini, its Python counterpart. It exams them on two actual doc collections and offers precise anticipated scores for every train.

Each quantity matched. Getting there took an out-of-memory crash, a local library crash, and a gated mannequin entry request {that a} legitimate software programming interface (API) key alone didn’t fulfill. None of that reveals up within the official documentation, so this text covers it as fastidiously because the numbers themselves. By the top, you’ll know the right way to run the three strategies regionally, what can break on regular {hardware}, and the right way to learn the scores.

The sensible software is simple: if you’re constructing a RAG system, a doc search product, or an inside information base, retrieval high quality is the very first thing to confirm. Earlier than altering prompts, swapping massive language fashions, or including brokers, you might want to know whether or not the fitting paperwork are being discovered in any respect. Reproducing these baselines offers you a reference level: what BM25 can do, what dense retrieval provides, what realized sparse retrieval prices, and the right way to verify whether or not your personal scoring pipeline is behaving accurately.

What BM25, dense retrieval, and SPLADE really do

All three strategies remedy the identical downside: given a search question and a big assortment of paperwork, return the paperwork probably to reply it, ranked greatest first. They differ solely in how they flip textual content into one thing a pc can evaluate and rank.

BM25 reads a doc and a question, counts which phrases they share, and weighs every shared phrase by how uncommon and the way frequent it’s. It produces a sparse vector, an extended listing of numbers the place nearly each entry is zero, and the nonzero entries are phrase weights. It wants no coaching knowledge. The weights are decided by a hard and fast system, tuned by two constants (k1 and b), slightly than realized from examples.

Dense retrieval replaces that system with a skilled neural community. A doc encoder reads the doc and outputs a fixed-length listing of numbers, an embedding, meant to seize which means slightly than precise wording. A question encoder does the identical for the question. The 2 embeddings are in contrast with a dot product. Meaning multiplying matching positions and including the outcomes. The best-scoring paperwork win. This requires a mannequin skilled on massive quantities of labeled query-document pairs.

Discovered sparse retrieval, the class SPLADE belongs to, is a center path. Like BM25, it produces a sparse vector the place most entries are zero. Not like BM25, a skilled mannequin determines which phrases matter and by how a lot, slightly than counting on a hard and fast system. SPLADE model 3 is the precise mannequin used on this article.

All three observe the identical broad sample: encode the question, encode the doc, then evaluate the 2. That is known as a bi-encoder structure. Jimmy Lin’s paper on a representational framework for information retrieval lays this out formally. Part 4 of this text verifies it by hand.

Two collections, one full scale and one small on objective

The workouts use two check collections. MS MARCO passage ranking is a Microsoft benchmark of 8,841,823 passages with 6,980 improvement queries, launched for non-commercial analysis. It’s full-scale: the identical core operations a manufacturing search system depends on- indexing, looking, and evaluating ranked results- run as soon as right here as an alternative of constantly. The BM25 end result right here matches the official MS MARCO leaderboard entry for “BM25 (Lucene8, tuned)”, printed on June 26, 2019. It’s a publicly recorded benchmark end result, not a quantity picked to make the article look good.

NFCorpus is a smaller medical retrieval dataset with 3,633 paperwork and three,237 queries. As a result of it’s small, all three strategies can run on a laptop computer CPU in minutes or hours as an alternative of requiring a GPU. It’s used right here for a direct, similar assortment comparability of BM25, dense retrieval, and SPLADE aspect by aspect. Deal with the NFCorpus numbers as an actual end result on this particular dataset, not a common rating of which technique wins in every single place. Its small dimension is what makes the comparability sensible on a laptop computer, not a shortcut round actual analysis.

All instructions beneath assume you might be working from the foundation of a cloned Anserini or Pyserini checkout, matching the official onboarding information’s personal conference.

Setup friction the official docs don’t point out

Three issues confirmed up earlier than a single retrieval command ran.

Cloning each repositories with full git historical past timed out repeatedly. Anserini and Pyserini have years of commit historical past; a shallow clone mounted it instantly:

git clone --depth 1 https://github.com/castorini/anserini.git

A required submodule (anserini-tools, shared between each tasks) failed mid clone with a connection reset. Elevating git’s HTTP buffer dimension resolved it:

git config http.postBuffer 524288000
git submodule replace --init --recursive --depth 1

Each toolkits require Java 21 particularly. The default put in JDK was Java 11 (by way of Homebrew), which prompted complicated failures till JAVA_HOME was pointed explicitly on the proper model, in each terminal session:

export JAVA_HOME=$(/usr/libexec/java_home -v 21)
export PATH="$JAVA_HOME/bin:$PATH"

None of those three issues is unique. Anybody reproducing this on a machine that isn’t already configured precisely just like the maintainers’ will probably hit a minimum of one in every of them.

BM25 on the total MS MARCO passage assortment, and the reminiscence crash

The primary actual train builds a Lucene inverted index over all 8,841,823 MS MARCO passages, then searches it with BM25. An inverted index maps every phrase to the paperwork that include it. It’s the similar fundamental construction utilized by full-text engines like google.

The indexing command is simple:

bin/run.sh io.anserini.index.IndexCollection 
  -collection JsonCollection 
  -input collections/msmarco-passage/collection_jsonl 
  -index indexes/msmarco-passage/lucene-index-msmarco 
  -generator DefaultLuceneDocumentGenerator 
  -threads 9 -storePositions -storeDocvectors -storeRaw

On a 16GB machine, this command fails. bin/run.sh hardcodes a 192GB Java heap ceiling:

java -cp `ls goal/*-fatjar.jar` -Xms512M -Xmx192G ...

The working system’s out-of-memory killer ends the method partway by way of:

2026-07-12 10:15:49 - Beginning to index...
2026-07-12 10:16:49 - 0.00% of recordsdata accomplished, 3,400,000 paperwork listed
bin/run.sh: line 3: 95895 Killed: 9  java -cp `ls goal/*-fatjar.jar` -Xms512M -Xmx192G ...

This captured terminal output reveals macOS killing the indexing course of after 3.4 million of 8.8 million paperwork, with no error message past Killed: 9. That is what a silent out-of-memory kill appears to be like like within the terminal: no Java exception, solely the method disappearing.

Reducing the heap ceiling to 10GB and the thread depend from 9 to 4 mounted it. The second run listed all 8,841,823 paperwork in two minutes and sixteen seconds with zero errors, matching Anserini’s personal documentation of roughly 4 gigabytes of index dimension on disk.

Retrieval and analysis observe:

bin/run.sh io.anserini.search.SearchCollection 
  -index indexes/msmarco-passage/lucene-index-msmarco 
  -topics collections/msmarco-passage/queries.dev.small.tsv 
  -topicReader TsvInt 
  -output runs/run.msmarco-passage.dev.bm25.tsv 
  -format msmarco -parallelism 4 
  -bm25 -bm25.k1 0.82 -bm25.b 0.68 -hits 1000

python instruments/scripts/msmarco/msmarco_passage_eval.py 
  collections/msmarco-passage/qrels.dev.small.tsv 
  runs/run.msmarco-passage.dev.bm25.tsv

The end result was an MRR at 10 of 0.18741227770955546, matching the official anticipated worth from the Castorini information to each digit. MRR at 10, or imply reciprocal rank, is the official MS MARCO metric. It measures the place the primary right passage seems within the ranked listing for every question, so greater is best.

Proving the rating is actually a dot product

That is the part that issues most, as a result of it turns “BM25 gave a rating of 17.9” from a truth you settle for right into a truth you’ll be able to verify. Take doc 7187158 (a solution to the question “what’s paula deen’s brother”) and reconstruct its BM25 weight vector immediately from the index:

from pyserini.index.lucene import LuceneIndexReader

index_reader = LuceneIndexReader("indexes/lucene-index-msmarco-passage")
term_frequencies = index_reader.get_document_vector("7187158")
bm25_weights = {
    time period: index_reader.compute_bm25_term_weight("7187158", time period, analyzer=None)
    for time period in term_frequencies.keys()
}

Construct the question as a easy vector: one for each question phrase, zero in every single place else.

from pyserini.evaluation import Analyzer, get_lucene_analyzer

analyzer = Analyzer(get_lucene_analyzer())
query_tokens = analyzer.analyze("what's paula deen's brother")
query_weights = {token: 1 for token in query_tokens}

Compute the dot product by hand, summing the burden of each phrase that seems in each:

rating = sum(
    bm25_weights[term]
    for time period in bm25_weights.keys() & query_weights.keys()
)

The hand computed result’s 17.949487686157227. Lucene’s personal search returns 17.94950 for a similar question and doc, the identical quantity to the precision Lucene itself studies. BM25 search is, by building, evaluating a question vector towards a doc vector and rating by the best dot product, so the 2 numbers matching is neither a coincidence nor an approximation. When you reproduce that rating with a brief script, it stops feeling mysterious. It turns into a quantity you’ll be able to audit. The identical verify works for dense retrieval and SPLADE. You’ll be able to reconstruct the doc vector, encode the question, compute the dot product, and evaluate it with the engine’s reported rating. The runnable scripts for all three are within the companion repository.

Dense retrieval, and a crash between two libraries that have no idea about one another

Dense retrieval swaps BM25’s phrase counting system for BGE-base-en-v1.5, a printed embedding mannequin from the Beijing Academy of Synthetic Intelligence (BAAI), launched underneath the MIT License, which explicitly permits business use. Working it towards the identical MS MARCO queries makes use of a prebuilt HNSW index (Hierarchical Navigable Small World, a graph construction constructed for quick approximate search over dense vectors, downloaded slightly than constructed regionally):

bin/run.sh io.anserini.search.SearchHnswDenseVectors 
  -index msmarco-v1-passage.bge-base-en-v1.5.hnsw 
  -topics collections/msmarco-passage/queries.dev.small.tsv 
  -topicReader TsvInt 
  -output runs/run.msmarco-passage.dev.bge.txt 
  -encoder BgeBaseEn15 -hits 1000 -threads 4

Working this with a couple of search thread crashed the method outright, not with a Java exception however a segmentation fault inside a local threading library:

# A deadly error has been detected by the Java Runtime Setting:
# SIGSEGV (0xb) at laptop=0x0000000133938734, pid=5863, tid=259
# Problematic body:
# C  [libomp.dylib+0x58734]  void __kmp_suspend_64<false, true>(...)

The Java digital machine and PyTorch (used internally by the encoder) every load their very own copy of the OpenMP threading runtime, and on macOS the 2 copies collide. The repair is one setting variable plus operating search on a single thread:

export KMP_DUPLICATE_LIB_OK=TRUE

With that set, the identical command accomplished cleanly. MRR at 10 got here out to 0.3521, an 87.9 p.c enchancment over BM25’s 0.1874 (greater is best).

BM25, dense retrieval, and SPLADE aspect by aspect on NFCorpus

NFCorpus is sufficiently small to match all three strategies on the identical laptop computer CPU. I used nDCG at 10, a metric that rewards strategies for putting related paperwork close to the highest of the ranked listing.

BM25 with default parameters scored 0.3218. BGE-base dense retrieval scored 0.3808, an 18.3 p.c enchancment. SPLADE model 3, a realized sparse retrieval mannequin from Naver launched underneath a non business Artistic Commons license (extra on that beneath), scored 0.3624, touchdown between the 2: 12.6 p.c above BM25 and about 4.8 p.c beneath dense retrieval.

SPLADE launched its personal friction. naver/splade-v3 is a gated mannequin on Hugging Face: a sound API token alone doesn’t grant entry. A Hugging Face token just isn’t sufficient for this mannequin. The account should additionally request entry on the mannequin web page earlier than the token works, and the error returned earlier than entry is granted (GatedRepoError) doesn’t clarify that distinction. After entry was granted, SPLADE took roughly 68 minutes to encode NFCorpus on CPU. BGE-base took about 4 minutes on the identical assortment. That may be a actual value distinction to plan round should you should not have a GPU.

What this really takes, in follow

Reproducing BM25, dense retrieval, and SPLADE accurately, at actual scale, on a traditional laptop computer, is genuinely doable. It takes express reminiscence tuning on the indexing step, one setting variable to work round a local library battle, and clearing a gated mannequin entry request the documentation doesn’t totally clarify. None of those issues are unique. They’re frequent sufficient that anybody making an attempt that is prone to hit a minimum of one in every of them, which is strictly why they’re value writing down as an alternative of quietly working round and by no means mentioning.

In the event you already run a retrieval or RAG system, strive the dot-product verify by yourself index. Use your personal scoring perform and evaluate one query-document rating by hand. It takes a couple of minutes, and it will possibly catch scoring bugs earlier than they attain manufacturing.

That’s the half value carrying into any actual retrieval mission. Don’t begin by asking whether or not your RAG system feels good. Begin with three checks: can your retriever beat a identified baseline, does your metric match the product conduct you care about, and may you manually confirm one query-document rating? In the event you can not clarify one retrieved end result, you can’t belief a thousand of them.

There’s a actual open query beneath these numbers. BM25 wanted no coaching knowledge and nonetheless landed inside roughly half the rating of a mannequin skilled on tons of of hundreds of labeled examples. Does a hand designed, untrained scoring perform nonetheless deserve a spot in a contemporary retrieval stack, or is it nonetheless round largely as a result of it’s low cost and acquainted? The hole between 0.1874 and 0.3521 on MS MARCO suggests one reply. The hole between 0.3218 and 0.3808 on a ten instances smaller, out of area assortment suggests a smaller one.

Information and mannequin licenses

This text studies benchmark outcomes from public analysis datasets and fashions. It doesn’t redistribute any of the underlying knowledge or mannequin weights, and doesn’t construct or promote a business product utilizing them.

MS MARCO (Microsoft) is launched for non business analysis functions solely, offered as is with out guarantee. The outcomes listed here are reported solely as a replica of a public analysis benchmark; this text doesn’t redistribute the dataset or use it to construct a business services or products. Readers ought to evaluation the MS MARCO phrases earlier than utilizing the dataset or any analysis outputs in services or products. The numbers reported right here come from operating printed, commonplace analysis benchmarks on that dataset, the identical follow adopted by basically each retrieval paper and weblog put up that cites an MS MARCO leaderboard end result.

NFCorpus is launched underneath CC BY-SA 4.0, distributed by way of the BEIR benchmark, which allows reuse and adaptation with attribution and share alike phrases.

BGE-base-en-v1.5 (BAAI) is launched underneath the MIT License, which explicitly permits business use.

SPLADE model 3 (Naver) is launched underneath CC BY-NC-SA 4.0, a non business license. It’s used right here solely to compute and report a benchmark rating, not redistributed, modified, or used as a part of a business services or products.

Glossary

BM25. A key phrase based mostly scoring system that ranks paperwork by matching question phrases, weighted by how uncommon and the way frequent every phrase is. Wants no coaching knowledge.

MRR (imply reciprocal rank). A rating metric that scores 1 divided by the place of the primary right end result, averaged throughout queries. Greater is best; 1.0 is an ideal rating.

nDCG (normalized discounted cumulative achieve). A rating metric that rewards related outcomes showing close to the highest of the listing greater than additional down. Greater is best.

Dense retrieval. Search utilizing embeddings, lists of numbers produced by a skilled mannequin, in contrast by dot product as an alternative of matching precise phrases.

Sparse retrieval. Search utilizing vectors the place most entries are zero and the nonzero entries correspond to particular phrases or phrase like models. BM25 is unsupervised sparse retrieval; SPLADE is realized sparse retrieval.

Discovered sparse retrieval. Sparse retrieval the place a skilled mannequin decides time period weights, as an alternative of a hard and fast system like BM25’s.

Bi encoder. The shared structure behind all three strategies: one encoder for paperwork, one for queries, and a comparability perform that scores the pair.

HNSW (Hierarchical Navigable Small World). A graph based mostly knowledge construction used to look dense vector collections shortly with out evaluating a question to each single doc.

Embedding. The output of an encoder: a hard and fast size listing of numbers meant to symbolize the which means of a chunk of textual content.

Inverted index. A knowledge construction mapping every phrase to the listing of paperwork that include it, the core construction behind key phrase engines like google like Lucene.

Chosen sources

[1] J. Lin, “A Proposed Conceptual Framework for a Representational Strategy to Data Retrieval” (2021), arXiv:2110.01529

[2] R. Nogueira and J. Lin, “Doc Growth by Question Prediction” (2019), arXiv:1904.08375

[3] N. Thakur, N. Reimers, A. Rückle, A. Srivastava, and I. Gurevych, “BEIR: A Heterogeneous Benchmark for Zero-shot Analysis of Data Retrieval Fashions” (2021), arXiv:2104.08663

[4] C. Lassance, H. Déjean, T. Formal, and S. Clinchant, “SPLADE-v3: New baselines for SPLADE” (2024), arXiv:2403.06789

[5] S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff, “C-Pack: Packaged Assets To Advance Common Chinese language Embedding” (2023), arXiv:2309.07597

[6] Castorini analysis group, “Foundations of Retrieval onboarding path,” github.com/castorini/onboarding

[7] Castorini analysis group, Anserini “begin right here” documentation, github.com/castorini/anserini

[8] Microsoft, “MS MARCO Passage Rating Leaderboard,” microsoft.github.io/MSMARCO-Passage-Ranking-Submissions

All instructions, verification scripts, and the precise reproduced numbers referenced on this article can be found in a companion repository.