Haystack benchmark on one GPU: BM25 alone beat dense search, and 200-word chunks answered 27 of 40

Part of RAG & Vector Databases — retrieval-augmented generation and vector search.

This Haystack benchmark is the second library episode in the series, one after LlamaIndex, and it was set up to answer a question that episode left open. LlamaIndex measured hybrid retrieval against dense retrieval, so the share of the lexical half on its own stayed unknown. Haystack treats BM25 as a first-class component, so here it gets its own row, its own container and its own forty questions.

The answer turned out to be larger than the question. BM25 on its own answered more questions than dense search, the fusion of the two matched the best result of the previous episode, and a fifth generation on Haystack’s own default splitter passed it: 27 correct answers out of 40, the most any configuration in this series has returned. Nothing changed except which chunks reached the model.

What Haystack is, and why the pipeline is the artifact

Haystack by deepset is a library, like LlamaIndex one episode earlier. There is no image to pull, no panel and no server. What sets it apart is how a retrieval system is assembled: as a Pipeline, a graph of named components connected by explicit sockets, executed by Pipeline.run and written out as YAML by Pipeline.dumps.

That last property shapes this Haystack benchmark. Every recipe measured here is a real Pipeline object, and the YAML it serialises to ships with the raw results. A reader can therefore see exactly which components ran, with which parameters, without reading any of my code.

The version matters more than usual. pip install haystack-ai installs 3.1.1 as of September 2026, and the 3.x line moved every component built on sentence-transformers out of the core package. The cross-encoder reranker this episode needs lives in sentence-transformers-haystack, and almost every tutorial online still imports it from the old path.

The Haystack benchmark setup, command by command

Two model endpoints must answer before anything starts. Generation runs on a local llama-server over HTTPS with a certificate issued to bestin-it.test, which no container hostname can match, so a small relay puts plain HTTP on port 8084 in front of it. Embeddings run on a second llama-server on 8083, already plain HTTP.

C:/Tools/run_infrastructure/start-llama-server.ps1 -Model qwen3.8   # :8082 HTTPS
.benchmarkharnessstart_embed_server.ps1                          # :8083 HTTP
python benchmark/harness/tls_bridge.py                              # :8084 -> :8082

Check them with curl rather than PowerShell’s Invoke-RestMethod. Windows PowerShell 5.1 has no -SkipCertificateCheck, so reading the HTTPS endpoint fails on parameter binding and every server looks dead while all three are running.

curl.exe -sk -o NUL -w "8082 -> %{http_code}`n" https://127.0.0.1:8082/health
curl.exe -s  -o NUL -w "8083 -> %{http_code}`n" http://127.0.0.1:8083/v1/models
curl.exe -s  -o NUL -w "8084 -> %{http_code}`n" http://127.0.0.1:8084/v1/models

Then the container. Haystack ships no image, so the first run builds one from a Containerfile. The pinned packages are the whole recipe:

pip install "haystack-ai==3.1.1" 
            "sentence-transformers-haystack==0.1.1" 
            "sentence-transformers==6.0.1" 
            "tiktoken==0.14.0" "nltk==3.10.3"

The image also bakes in two things the framework would otherwise fetch on first use: the BAAI/bge-reranker-base weights and NLTK’s punkt_tab tokenizer data. The container has no route to the internet, so a lazy download would not be slow, it would fail — twenty minutes into a measured run.

.articlesE-07-haystackrun.ps1 -Recreate
curl.exe http://127.0.0.1:9624/health

The start script passes /dev/dxg and the WSL libdxcore.so into the container, which is how the cross-encoder reaches the AMD Radeon PRO W7900 through ROCm. The container prints which device it resolved to before serving a single request. In every measured generation that line read AMD Radeon PRO W7900.

Defaults that would have changed what this measures

Every value below was read from the installed package before the first measurement. Two of them reproduce in one snippet:

from haystack.components.embedders import OpenAIDocumentEmbedder
from haystack.document_stores.in_memory import InMemoryDocumentStore
store = InMemoryDocumentStore()
print(store.bm25_algorithm, store.embedding_similarity_function)   # BM25L dot_product

The default splitter cuts 200 words

DocumentSplitter() splits by word, 200 words per chunk, no overlap. On this corpus that is a median of 265 tokens per chunk and 2,543 chunks in total. The LlamaIndex episode used 1,024-token chunks, and the naive baseline from episode 2 sent about 6,400 tokens of evidence per question. Eight 265-token chunks would have handed the model a third of that.

Therefore the main index uses RecursiveDocumentSplitter(split_unit="token", split_length=1024, split_overlap=200), the one Haystack splitter that states its size in tokens. It produced 848 chunks with a median of 1,025 tokens, against 874 chunks and a median of 988 for LlamaIndex’s splitter. The default word splitter is not discarded: it gets its own generation of the stack, with the token budget matched, later in the article.

The token splitter downloads NLTK data on first use

Its sentence separator runs NLTK’s punkt tokenizer, and warm_up() calls nltk.download("punkt_tab") when the data is missing. Inside a container without network access that is a crash on the first index build. The fix is to download the data at image build time and point NLTK_DATA at it.

The embedder batches 32 documents

OpenAIDocumentEmbedder sends 32 chunks per request by default. Thirty-two chunks of 1,024 tokens exceed the embedding server’s 8,192-token context. The benchmark sets batch_size=4, the same value the LlamaIndex episode needed for the same reason.

The store scores by dot product, and BM25 is BM25L

InMemoryDocumentStore ranks dense results by dot product unless told otherwise. That equals cosine similarity only when every vector has unit length. The preflight check measures the norm of two live embeddings from the nomic server before each run; both came back at exactly 1.0, so the default stays. The sparse side uses BM25L rather than the textbook BM25Okapi — a detail worth knowing when comparing scores with another system.

Telemetry is on, and a token secret will not serialise

Two smaller traps. Haystack reports usage to Posthog by default; HAYSTACK_TELEMETRY_ENABLED=False turns that off, which matters in a container with no internet. In addition, Secret.from_token("local") works at run time but raises inside Pipeline.dumps(), because a token secret refuses to be written to a file. The OpenAI components therefore read OPENAI_API_KEY from the environment, and the local servers ignore its value.

Four recipes, one evidence budget, one prompt

Four pipelines were measured on the token-unit index, each on its own container generation:

Recipe	Components	Pool	Chunks to the model
embedding	InMemoryEmbeddingRetriever	8	8
bm25	InMemoryBM25Retriever	8	8
hybrid	both, then DocumentJoiner with reciprocal rank fusion	8	8
hybrid_rerank	both at 30, fused to 30, SentenceTransformersSimilarityRanker cuts to 8	30	8

Eight chunks is the top-k the naive baseline in episode 2 answered under, so the floor and the framework see the same evidence budget. The bm25 recipe is the one the LlamaIndex episode could not supply: there, hybrid was only ever measured against dense search, and the contribution of the lexical half on its own stayed unknown.

The prompt is the floor’s prompt, byte for byte. Haystack renders it through a Jinja template in ChatPromptBuilder, and a template is where whitespace changes silently. Consequently the preflight rebuilds the episode-2 prompt from the same chunks and compares SHA-256 digests with what the container actually sent. Every probe in every generation matched.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

{{ head }}

CONTEXT
{% for doc in documents %}[{{ doc.meta.file_name[:-3] }}]
{{ doc.content }}{% if not loop.last %}

---

{% endif %}{% endfor %}

QUESTION
{{ query }}

{{ tail }}

Every recipe gets its own stack and its own freshly loaded corpus

The obvious way to run four recipes is to build one index and query it four times. This benchmark does not do that. Each recipe gets a container created from scratch, a fresh volume and the 33 documents loaded again from disk. The build is splitting and embedding only — the adapter records llm_calls: 0 — and it took under a minute each time.

That buys a claim about the build rather than about a mode. The LlamaIndex episode could only compare independent builds by a content digest, because that framework draws every node id from uuid4. Haystack derives a document’s id from a SHA-256 of its content and metadata:

# haystack/dataclasses/document.py - the id is a hash of what the document is
data = f"{text}{dataframe}{blob!r}{mime_type}{meta}{embedding}{sparse_embedding}"
return hashlib.sha256(data.encode("utf-8")).hexdigest()

So here both digests — the one with ids and the one over content alone — are expected to agree across builds. Expected is not measured, which is why compare_builds.py reports each separately.

What the Haystack benchmark scored across five recipes

The floor for this benchmark is the naive baseline from episode 2: 70.0% document recall and 6 correct answers out of 40. Every row below faces the same 40 questions, the same prompt and the same blind judge. Evidence tokens are the median token count of the chunks actually sent, counted with the same tokenizer for every row.

Recipe	Evidence tokens	Document recall	Quotes in prompt	Correct	Refusals	Numeric exact	Median latency
naive baseline (E-02)	—	70.0%	—	15.0% (6/40)	30	60.0% of 10	17.57 s
embedding	8,206	72.5%	15/75	25.0% (10/40)	23	64.7% of 17	20.02 s
bm25	8,220	60.0%	28/75	37.5% (15/40)	19	71.4% of 21	19.95 s
hybrid	8,213	87.5%	30/75	47.5% (19/40)	15	80.0% of 25	20.17 s
hybrid_rerank	8,215	90.0%	28/75	40.0% (16/40)	16	70.8% of 24	20.45 s
hybrid_rerank, 200-word chunks, 31 to the model	8,560	100.0%	51/75	67.5% (27/40)	10	93.3% of 30	25.25 s

No run produced an error, and no run invented a number in any of the 200 answers. Plain dense retrieval lands four answers above the naive floor. BM25 alone lands nine above it. Fusing the two lands thirteen above it, with no rebuild and 0.15 s of extra latency.

Document recall picked the wrong winner, twice

Read the table by document recall and embedding beats bm25, 72.5% to 60.0%. By correct answers, however, bm25 wins, 15 to 10. The same thing happens one row lower: hybrid_rerank has the best document recall in the episode at 90.0%, and by correct answers it trails plain hybrid, 16 to 19.

The quote count predicted both outcomes. BM25 put 28 of 75 gold passages in the prompt against 15 for dense search, and answered five more questions. The cross-encoder took the passage count from 30 to 28 while raising the document count, and answered three fewer questions. Ordered by quotes in the prompt, the four token-unit recipes land in the same order as by correct answers. Ordered by document recall, they do not.

The LlamaIndex episode found the same thing with its reranker, and this episode was set up to see whether the result would repeat on another framework. It did, and it added a second case: a retriever that finds the right file less often can still answer more questions, if the passages it finds are the ones the question is about.

Two ways to count retrieval, and they disagree again

Strict document recall asks whether the right file was represented in the prompt. The LlamaIndex episode showed that on this corpus that is a weak statement, because one 10-K runs past a hundred thousand tokens and eight chunks cover a few percent of it. So this episode counts the passages as well: the golden set carries 75 evidence quotes, all 75 confirmed verbatim in their source documents before any run was scored.

By documents, embedding scores 72.5% and bm25 60.0%. Counted in quotes, the order flips: bm25 put 28 of 75 gold passages in front of the model, embedding only 15 of 75. The dense retriever finds the right file about the right topic. The lexical retriever finds the paragraph that carries the literal words of the question. On a corpus where every filing repeats the same segment names and product lines, those are different skills.

Fusing the two does not average them. hybrid reaches 87.5% of documents and 30 of 75 quotes, more than either half alone on both counts. Consequently the refusal count falls from 23 and 19 to 14: the model declines less often because it is shown more of the evidence.

Dense search owns D5, BM25 owns D7

The per-pattern split is where the two retrievers stop looking like competitors. Document recall is out of five questions per pattern:

Recipe	D1	D2	D3	D4	D5	D6	D7	D8
embedding	5	3	5	4	5	3	0	4
bm25	5	5	3	1	1	1	4	4
hybrid	5	5	5	4	4	3	4	5
hybrid_rerank	5	5	5	5	3	4	4	5

D5, event to consequence, lives in short 8-K reports about one event. Dense search found all five, BM25 one. The 8-K describes the event in its own words, and the question paraphrases it, so lexical overlap is thin while the topic is unmistakable.

D7, retroactive restatement, is the opposite case. It needs one 10-K from before the 10-for-1 stock split and one from after. Those two filings are nearly identical in vocabulary, and dense similarity gave the older one no place in eight: 0 of 5, the same zero every flat LlamaIndex mode produced. BM25 found four of five, because the D7 questions name the split and the restated figures literally, and those words sit in a few specific paragraphs.

Therefore the hybrid does not win because fusion is clever. It wins because the two failure sets barely overlap. Dense search fails on D7 and BM25 fails on D5, and each covers the other’s blind spot.

Who actually found the evidence

Haystack returns the output of every component in a pipeline, so the pool audit can say which retriever placed each gold document. Of the 55 gold document slots present in the hybrid pool, 35 came from both retrievers, 11 from dense search alone and 9 from BM25 alone. A fifth of the evidence arrived from exactly one side. The LlamaIndex episode could not make that decomposition, because there hybrid was only ever measured against dense search.

Rank fusion at top-k 8 dropped a document one retriever had at position six

Five gold slots did not enter the hybrid pool of eight. Two of them tell an uncomfortable story: the 8-K behind question D5-05 and the 10-K behind D6-03 were both at position six on the dense retriever’s own list. BM25 did not have them. After fusion, they were gone.

The arithmetic is simple. Haystack’s DocumentJoiner scores a candidate as 1 / (61 + rank) per list and sums. A document at rank 6 on one list scores about 0.0149. A document at rank 8 on both lists scores 2 / 69, about 0.0290, nearly twice as much. With input lists of eight, every document both retrievers found outranks every document only one found, however confident that one was.

# haystack/components/joiners/document_joiner.py - the fusion rule
k = 61
for rank, doc in enumerate(documents):
    scores_map[doc.id] += 1 / (k + rank)

Widening the retrievers to fifty candidates each and fusing again changed the top eight in 40 of 40 questions, exactly as it did in the LlamaIndex episode. The depth parameter of a fused retriever is an input to the algorithm, not a cut applied after it, so a wider page is a different configuration rather than a deeper look at the measured one.

What the cross-encoder returned, and what it cost

hybrid_rerank pulls thirty candidates from each retriever, fuses them to thirty and lets bge-reranker-base pick eight. The reranker runs on the AMD Radeon PRO W7900 inside the container, through the same ROCm chain the LlamaIndex episode needed.

Counted over gold documents present in the fused pool of thirty, the cross-encoder promoted 11 into the prompt and dropped 1, keeping 45. Document recall rose from 87.5% to 90.0%, the highest in the episode. Quote recall, however, fell from 30 of 75 to 28 of 75.

Ten of the eleven promotions were chunks of one document, the 10-K filed on 2026-02-25. That is the newest annual report in the corpus, and it holds 35 of the 60 gold document slots, so pulling it in is the right move more often than not. All five D7 questions gained a chunk of it. The one loss was the 8-K of 2025-04-15 behind question D5-02, which stood at position three after fusion and at position nine after the cross-encoder. A second D5 slot stopped at position nine. As a result, D5 went from 4 of 5 to 3 of 5 while D4 and D6 gained a document each.

Cost of the reranking pass

Timed by the pipeline itself, the ranker component took a median of 293 ms for thirty pairs, against roughly 20 s for generating the answer. The first call after loading took 1,575 ms, because the kernels compile on the first forward pass. The container therefore runs one dummy pair after warm_up(), or that number would have entered the first of forty latencies.

The same caveat as one episode earlier applies. Every sampled chunk in every generation was longer than the cross-encoder’s 512-token window: 100% of 32 sampled chunks, median 1,025 tokens. The model scored the first half of each candidate. It picked the right file and not necessarily the right passage, which is what the falling quote count says.

Five builds, one digest, ids included

Each recipe got its own container generation, so the corpus was split and embedded five times from nothing, 848 chunks each time in 50.9 to 51.1 seconds. The content digest was identical in every build. So was the digest that includes chunk ids, f09f08a7…, because Haystack derives an id from a SHA-256 of the content and metadata. The LlamaIndex episode could only compare content, since its ids were random UUIDs.

python articles/E-07-haystack/compare_builds.py --modes embedding,bm25,hybrid,hybrid_rerank
# VERDICT: REPRODUCIBLE
# id-bearing digests agree: True

Build wall time is reported and never compared, since four timings are four timings. The index is 18.7 MB on disk and the container held about 1.3 GB of memory during the build. No generative model was called at any point of any build.

The default splitter, budget matched: every document found, twice the passages

The fifth generation put the same recipe, hybrid retrieval with the cross-encoder, on an index cut by Haystack’s default DocumentSplitter: 200 words, no overlap. That produced 2,543 chunks of about 265 tokens instead of 848 of about 1,025. Sending eight of them would have handed the model a quarter of the evidence, so the budget was matched in tokens rather than in pieces: 31 chunks per question, a median of 8,560 evidence tokens against 8,206 to 8,220 in the four token-unit recipes.

The build took 40.8 s and 47.2 MB, again without a generative model. Retrieval widened in proportion, 116 candidates per retriever fused and reranked to 31, and the cross-encoder pass grew to a median of 702 ms for 116 pairs.

Retrieval changed more than any algorithm did. Strict document recall reached 100%, 40 of 40. Evidence quotes in the prompt reached 51 of 75, against 28 of 75 for the same recipe on 1,024-token chunks. D7 went from 2 of 10 quotes to 9 of 10: with 31 places instead of eight, the pre-split and post-split 10-K both fit. D3 went from 4 to 9 and D5 from 7 to 9.

This is the same result the LlamaIndex episode reached with 128-token leaves, from a different framework and a different splitter. At a fixed token budget, smaller pieces let the same number of tokens point at many more places in the corpus, and questions whose evidence is scattered by design are exactly the ones that need more pointers.

What the cross-encoder did on short chunks

The cross-encoder had little to do with it. On this index it read whole candidates for the first time in two episodes, since a 277-token median chunk fits its 512-token window, yet it promoted one gold document and dropped none. Fifty-nine of the sixty gold slots were already inside the fused top 31 before it ran. The gain came from the number of places, not from the ranking of them. Whether the reranker matters for answer quality on short chunks is a separate question, and a word-unit index without it was not measured.

How to read a score of 67.5%

First of all, that number means 27 of 40 questions answered correctly, judged by a model that never saw which system produced which answer. It does not mean 67.5% of some quality scale.

The rest breaks down as follows: 10 questions were refused, and 5 of those refusals were correct, because five questions in the golden set have no answer in the corpus and this run refused all five. The remaining three answers were partial. On the numeric questions the same run matched 28 of 30 values exactly, and invented none. Per pattern it answered every D2, D3 and D5 question, four of five D1, three of five D4 and D6, and two of five D7. So the evidence for D7 was in the prompt for all five questions, 9 of 10 quotes, and the model turned that into two correct answers. Retrieval on this pattern is now solved. Reading two restated figures side by side is not.

In the series scorecard, which weights cross-document retrieval at 30 points, in-document retrieval at 10 and answer correctness at 20, this run scores 85.5 of 100. Forty of those points came from finding every gold document. Where that places Haystack against the other frameworks, per dispersion pattern, is episode 10’s question, not this article’s.

Traps worth knowing before you start

Symptom	Cause	Fix
ImportError on SentenceTransformersSimilarityRanker	Haystack 3.x moved sentence-transformers components to sentence-transformers-haystack	Pin all three packages and import from haystack_integrations.components.rankers.sentence_transformers
First index build in an offline container fails at warm_up()	The token splitter downloads NLTK punkt_tab on first use	Download at image build time and set NLTK_DATA
Embedding server rejects a batch	OpenAIDocumentEmbedder sends 32 chunks per request, about 33,000 tokens	batch_size=4 for 1,024-token chunks on an 8,192-token server
Pipeline.dumps() raises ValueError on a working pipeline	Secret.from_token() refuses to serialise	Secret.from_env_var("OPENAI_API_KEY") with the variable set in the container
Two identical pipelines give two different YAML files	Each InMemoryDocumentStore carries a random index UUID	Exclude index when comparing pipeline YAML across generations
Widening the retrievers’ top-k changes the fused top eight	Reciprocal rank fusion recomputes over the longer input lists	Treat the wider page as a separate configuration
A curl.exe -o line in a PowerShell loop writes nothing, without an error	A non-ASCII repository path reaches the native tool in the console code page	Write artefacts through Python and check the exit code after every write

Frequently asked questions

Is BM25 or embedding retrieval better in Haystack?

On 33 SEC filings they were good at different things. Dense retrieval found the right document for 29 of 40 questions and BM25 for 24. Counted by evidence passages in the prompt, BM25 delivered 28 of 75 and dense retrieval 15 of 75. Dense search found all five event-to-consequence questions and none of the five restatement questions; BM25 found one and four respectively. Neither is better in general. Fused, they reached 35 of 40 documents and 30 of 75 passages.

Which Haystack splitter matches 1,024-token chunks?

RecursiveDocumentSplitter(split_unit="token", split_length=1024, split_overlap=200). It is the only Haystack splitter that states its size in tokens, and on this corpus it produced 848 chunks with a median of 1,025 tokens. The default DocumentSplitter() cuts 200 words with no overlap, a median of 265 tokens here, so eight of its chunks carry a quarter of the evidence that eight token-unit chunks do.

Why does Haystack’s DocumentJoiner drop a document one retriever ranked highly?

Reciprocal rank fusion sums 1 / (61 + rank) over the lists a document appears in. A document on one list at rank 6 scores about 0.015. A document on both lists at rank 8 scores about 0.029. With eight candidates per list, any document both retrievers found outranks any document only one found. Two gold documents at position six on the dense list were lost this way in this benchmark. Longer input lists, as in the reranking recipe with thirty per side, leave room for both.

Can I run the Haystack cross-encoder on an AMD GPU inside podman on Windows?

Yes, and every reranked number in this article was measured that way. The container reports rerank_device: cuda, which is what a ROCm build of torch calls the AMD Radeon PRO W7900. A podman machine is WSL2 and exposes /dev/dxg rather than /dev/kfd, so four pieces from three repositories have to line up: the WSL build of the HSA runtime from the driver repository, the hsa-amd-aqlprofile library from the ROCm repository, the ROCm torch wheel, and a copy of AMD’s libhsa-runtime64.so.1 over the one torch bundles. The reranking pass then takes a median of 293 ms for thirty pairs.

Why does Pipeline.dumps() fail with a ValueError?

One of the components holds a Secret.from_token(), and token secrets refuse to be written to a file. The pipeline still runs. Use Secret.from_env_var() instead; the YAML then records the variable’s name, and a local llama-server ignores the key anyway.

How reproducible is a Haystack index between builds?

Fully, on both digests. Five independent builds of the same 33 documents produced the same content digest and the same digest over chunk ids, because Haystack derives a document id from a SHA-256 of its content and metadata. LlamaIndex, one episode earlier, agreed only on content, since its node ids are random UUIDs.

Summary

Five recipes of one framework, five separate builds of the same 33 SEC filings, 40 questions whose answers sit in more than one document. Correct answers went from 10 to 27 out of 40 without changing the model, the prompt or the corpus.

Three things decided that range. The first is which retriever finds which kind of evidence: dense search finds the right file about the right topic and BM25 finds the paragraph carrying the question’s words, and on this corpus the paragraph mattered more, 15 correct answers against 10. The second is that document recall does not measure that. It ranked dense search above BM25 and the cross-encoder above plain fusion, and the answers went the other way both times, while the count of gold passages in the prompt got both orderings right. The third is chunk size at a fixed token budget: 31 chunks of 200 words let the same tokens point at four times as many places as eight chunks of 1,024 tokens, and every gold document fit.

Notably, the recipe that scored best on the retrieval metric everybody reports, the cross-encoder at 90.0% document recall, answered fewer questions than the recipe before it. It is worth being precise about why, because its 293 ms on the GPU makes cost a poor reason to skip it. On 1,024-token chunks it scored the first half of every candidate and traded two gold passages for one gold file. On 277-token chunks it read everything and had almost nothing left to do: one promotion, no drops. In practice, that repeats the previous episode’s order with a second framework: chunk size and token budget first, lexical search second, reranking last.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.