LlamaIndex benchmark on one GPU: smaller chunks beat the reranker, and document recall picked the wrong winner

Part of RAG & Vector Databases — retrieval-augmented generation and vector search.

This LlamaIndex benchmark is the one episode in the series with no structure built on top of the text. No knowledge graph, no summary tree, no entity extraction. The index is chunks and vectors, and building it calls no generative model at all. That makes LlamaIndex the reference point the whole series needs: whatever the graph frameworks gained, they have to beat plain retrieval done carefully.

It also turned into a different kind of experiment than planned. Five retrieval configurations ran against five independently built indexes. The gap between the worst and the best is five correct answers against nineteen, out of forty. Nothing changed except which chunks reached the model.

What LlamaIndex is, and why that changes the setup

LlamaIndex is a library, not a deployment. RAGFlow, measured one episode earlier, was five containers with a web interface and a task queue. LlamaIndex is an import statement. There is no server, no admin panel and no image to pull.

Consequently the container here is built rather than downloaded. That is the same arrangement the GraphRAG episode used, and for the same reason: the framework has to be pinned and isolated from the host Python. Otherwise “which version produced this number” stops having an answer.

Being a library has one consequence that shapes the rest of the article. Nothing is configured for you. Chunk size, retrieval depth, whether the client speaks to a chat endpoint or a completion endpoint — all of it comes from defaults, and the defaults are written for a hosted API rather than for a local llama-server. Four of them would have quietly changed what this benchmark measures.

The LlamaIndex benchmark setup, command by command

Two model endpoints must answer before anything starts. Generation runs on a local llama-server over HTTPS with a certificate issued to bestin-it.test, which no container hostname can match, so a small relay puts plain HTTP on port 8084 in front of it. Embeddings run on a second llama-server on 8083, already plain HTTP.

C:/Tools/run_infrastructure/start-llama-server.ps1 -Model qwen3.8   # :8082 HTTPS
.benchmarkharnessstart_embed_server.ps1                          # :8083 HTTP
python benchmark/harness/tls_bridge.py                              # :8084 -> :8082

Check them with curl rather than PowerShell’s Invoke-RestMethod. Windows PowerShell 5.1 has no -SkipCertificateCheck, so reading the HTTPS endpoint fails on parameter binding and every server looks dead while all three are running.

curl.exe -sk -o NUL -w "8082 -> %{http_code}`n" https://127.0.0.1:8082/health
curl.exe -s  -o NUL -w "8083 -> %{http_code}`n" http://127.0.0.1:8083/v1/models
curl.exe -s  -o NUL -w "8084 -> %{http_code}`n" http://127.0.0.1:8084/v1/models

Then the stack. The first run builds the image, which pulls torch from the CPU index and bakes in the reranker weights — about 3 GB in total. Later starts take seconds.

.articlesE-06-llamaindexrun.ps1 -Recreate
curl.exe http://127.0.0.1:9623/health

The container reaches the host through a name rather than host.docker.internal. Inside a podman WSL machine that name resolves to the WSL bridge, not to Windows, so the start script reads the machine’s default route and injects the real address as llmhost.

Four defaults that would have changed what this measures

Every value below comes from the installed package, and you can reproduce the first two in one snippet:

from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(model="q", api_base="http://x/v1", api_key="local")
print(llm.metadata.context_window, llm.metadata.is_chat_model)   # 3900 False

is_chat_model defaults to False, and that breaks two things at once

With the default, the request goes to /v1/completions. Qwen3.8 reads a raw prompt as a document to continue and answers with an immediate end-of-sequence token, which arrives as an empty string. In addition, the TLS relay injects enable_thinking: false only on paths ending in /chat/completions. A completions call therefore skips that normalisation and spends its whole budget inside a thinking block.

context_window defaults to 3900 tokens

This one causes no error, which is what makes it costly. Eight 1,024-token chunks do not fit in 3,900 tokens, so the compact synthesiser splits the context and builds the answer through a refine chain. The run succeeds and the numbers look normal. However, the measured object has changed: that is no longer an answer from eight chunks, it is the result of several successive rewrites.

QueryFusionRetriever generates four query variants by default

Left alone, the hybrid mode would change two things at once — adding BM25 and rewriting the question with a model. Neither result could then be attributed. Setting num_queries=1 turns the rewriting off. Query rewriting is a separate research question, not part of this episode.

The cross-encoder reads half of each chunk

SentenceTransformerRerank constructs the model with a hard constant:

DEFAULT_SENTENCE_TRANSFORMER_MAX_LENGTH = 512      # sbert_rerank.py
self._model = CrossEncoder(model, max_length=512, device=device, ...)

Chunks are 1,024 tokens, which is the splitter’s own default. Consequently the cross-encoder scores a pair of (question, chunk) truncated to roughly the first half of the chunk. Measured on the flat index: 93.8% of sampled chunks exceed that window, with a median chunk of 988 tokens. This is not a LlamaIndex defect — bge-reranker-base has 512 position embeddings and will not take a longer input.

One evidence budget for every mode, and where that broke

Five configurations ran: dense vector search, hybrid fusion of BM25 and dense, a rerank mode that pulls thirty candidates and cuts them to eight with the cross-encoder, and two modes on a hierarchical index where hitting a small chunk can pull in its parent.

The plan said all of them hand the model exactly eight chunks. For the three flat modes that held. For the hierarchical recipe it did not, and the latency gave it away: 3.26 s per question against roughly 17 s everywhere else. Counting the tokens explained it.

SentenceSplitter produces 1,024-token chunks. HierarchicalNodeParser produces leaves of 128 tokens, and retrieval runs over the leaves. Eight chunks is therefore the same count and a ninefold difference in evidence. A control written in chunks stops being a control the moment two parsers with different chunk sizes are in play.

Therefore a fifth generation was added: automerge_wide, the same index and the same retriever at 56 leaves = 7,168 tokens, inside the range the flat modes actually sent. Both hierarchical runs stay in the results. The first describes the default recipe; the second allows a comparison on equal evidence.

Every mode gets its own stack and its own freshly loaded corpus

The obvious way to run five retrieval modes is to build one index and query it five times. This benchmark does not do that. Each mode gets a container created from scratch, a fresh volume and the 33 documents loaded again from disk.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

That costs a few extra minutes of embedding per mode, and it buys a different class of evidence. With one index, a fingerprint proves only that no mode wrote to it. With independent builds, the same fingerprint answers a bigger question: does this corpus produce the same index every time it is built?

The previous episode is why that question is worth paying for. There, the same configuration built twice produced a different prompt on 37 of 40 questions, because a generative model was writing summaries during the build. This episode has no model in the build at all — the adapter records llm_calls: 0 in its ingestion statistics, so the claim has a file behind it rather than a paragraph.

The fingerprint that would have reported the wrong answer

The first version of the digest hashed node ids together with content. Two independent builds of the same corpus then produced different digests, which read as “the build does not reproduce” — a much bigger story than anything else in this episode. It was wrong, and two lines inside the container showed why:

from llama_index.core.schema import TextNode
print(TextNode(text='identical text').node_id)   # 9551829c-e36b-4f53-8eb9-113dc359b007
print(TextNode(text='identical text').node_id)   # 48944ffe-c95f-4f39-a22a-599b9481010d

LlamaIndex draws every node_id from uuid4 at construction. A digest containing them differs between any two builds by construction, whatever the corpus. It was measuring a random number generator.

The endpoint now returns two digests, because they answer two different questions. One covers ids and content, and it is the strict identity of a single build: unchanged before and after a run means the retrieval mode wrote nothing. The other covers content alone, and it is the only one that may be compared across builds.

curl.exe -s http://127.0.0.1:9623/fingerprint
# build 1: {"nodes": 9275, "sha256": "1a00652727...", "content_sha256": "d5d89d0ef4..."}
# build 2: {"nodes": 9275, "sha256": "7a02682999...", "content_sha256": "d5d89d0ef4..."}

Two independent builds, different identities, identical content. The build reproduces, and the digest before and after each run never moved. Both gates passed on all five generations.

What the LlamaIndex benchmark scored across five configurations

The floor for this benchmark is the naive baseline from episode 2: 70.0% document recall and 6 correct answers out of 40. Every row below faces the same 40 questions and the same judge.

Mode	Evidence tokens	Document recall	Correct	Refusals	Numeric exact	Median latency
naive baseline (E-02)	—	70.0%	15.0% (6/40)	30	—	—
vector	7,312	67.5%	17.5% (7/40)	27	61.5% of 13	17.14 s
hybrid	7,503	70.0%	35.0% (14/40)	16	66.7% of 24	17.65 s
rerank	7,365	72.5%	17.5% (7/40)	23	47.1% of 17	17.71 s
automerge (top-k 8)	815	67.5%	12.5% (5/40)	33	85.7% of 7	3.26 s
automerge_wide	5,968	100.0%	47.5% (19/40)	17	87.0% of 23	15.25 s

No run produced an error, and no run invented a number in any answer. Plain dense retrieval lands one answer above the naive floor. The best configuration lands thirteen above it, and answers faster than the flat modes.

Document recall picked the wrong winner

First, read that table by document recall and rerank wins among the flat modes. Read it by correct answers and rerank ties with the cheapest mode. The two orderings disagree, so at least one of them is not measuring what a reader assumes.

Strict document recall asks whether the right file was represented in the prompt. On this corpus a single 10-K runs to well over a hundred thousand tokens, and eight chunks of 1,024 tokens cover a few percent of it. Naming the right file is therefore a much weaker statement than showing the right passage.

So I counted the passages. The golden set carries 75 evidence quotes across the 40 questions, and each one appears verbatim in its own source document — all 75 of 75, checked before anything else, because a matcher that cannot find a quote in its own file would make the rest meaningless.

In this run, dense search scored 67.5% on documents and 13.3% on quotes. Across 32 of the 40 questions the prompt held not a single complete gold quote. Twenty questions had the right document present and its decisive passage missing.

The quote count also tracks the answers. Ordered by quotes: automerge_wide 52.0%, hybrid 36.0%, automerge 22.7%, rerank 18.7%, vector 13.3%. Ordered by correct answers, the top two and the bottom one land in the same places. As a result, the refusals stop looking like stubbornness: with 27 refusals and zero invented numbers, dense search was declining to answer from evidence it had not been shown.

Where the missing evidence actually was

A recall number merges two different events. The ranking may never have contained the gold document, or it may have contained it below the position that top-k keeps. Those point at different next steps, and the answer alone cannot tell them apart.

The container therefore exposes retrieval without generation, so the pool and the cut can both be read for all 40 questions at the cost of one embedding each.

python articles/E-06-llamaindex/pool_audit.py --mode vector
python articles/E-06-llamaindex/pool_audit.py --mode vector --pool 50

At the measured depth of eight, 16 of 60 gold document slots never appeared. At a depth of fifty, on the same index in the same minute, that number is zero. Every missing document was in the ranking.

In practice, twelve of the sixteen sat between positions 9 and 16. The furthest sat at 31. In addition, the pattern that lost most was D7, retroactive restatement — eight of the sixteen — where an answer needs one filing from before a 10-for-1 stock split and one from after. Two documents about the same event compete for the same eight places, and the newer filing wins.

One expectation did not survive. Going in, the suspect was the short 8-K reports: 20 of the 33 documents are 8-Ks with a median of 1,277 tokens, and a corpus-wide top-k should starve them. Only two of the sixteen were 8-Ks. Thirteen were long 10-K and 10-Q filings. What a fixed budget squeezed out was not the short document — it was the second document about the same matter.

What the reranker returned, and what it cost

The cross-encoder runs on the GPU, in the container, next to a llama-server that is using the same card for generation and embeddings throughout. Getting it there is not a one-liner — a podman machine has no /dev/kfd, so ROCm binds through /dev/dxg instead, and four pieces from three repositories have to line up. The FAQ has the recipe.

Timed apart from generation, the reranking pass takes a median of 312 ms per question for 30 pairs, against 17.3 s for generating the answer. The card is doing about 39 times less waiting than a CPU would: this episode was first measured end to end on an image with a CPU-only torch, where the same pass took 12.19 s and the mode’s median latency was 29.29 s. Moving the cross-encoder to the card brought that down to 17.71 s, within a second of plain dense search, and changed nothing else — both regimes returned the same 72.5% recall and the same seven correct answers.

In total five documents were gained, and that is exactly the 67.5% to 72.5% move in document recall. Meanwhile quote recall went from 13.3% to 18.7% and correct answers did not move at all — seven of forty, the same as dense search alone.

Why the reranker found files rather than passages

The reason is the window from earlier: the model scores a pair truncated at 512 tokens while 96.9% of the chunks in that generation were longer. It reliably picks the right file and unreliably picks the right passage, because it never read the second half. Note that this is a property of the chunk size, not of the device — the GPU run selected exactly the same documents as the CPU one, down to the same 72.5% recall.

Worth stating plainly, because it is a property of this recipe rather than of reranking: on the hierarchical index the same measurement returns 0.0% of chunks over the window. Chunks there have a median of 109 tokens. A reranker on that index would read every candidate in full.

So the honest summary of the cross-encoder here is not that it is expensive. On the card it is nearly free. It is that it was asked to judge text it could only half see, and half a chunk was enough to identify the document and not enough to find the sentence.

Smaller chunks, same evidence budget, twice the answers

The hierarchical index is the one place this episode built something other than a flat list. HierarchicalNodeParser cuts 2048 / 512 / 128, embeds only the leaves, and AutoMergingRetriever replaces a group of leaves with their parent when enough siblings come back together.

However, it costs more to build: 136 s and 151.7 MB against 32.3 s and 20.2 MB for the flat index, from 33 documents. Still no generative model, so the whole build stays in minutes rather than the hours the graph episodes needed.

At the default depth of eight the result sits below the naive floor: 5 correct answers, 33 refusals, 815 tokens of evidence. At 56 leaves, the same index and the same retriever produced the best result in the episode.

Why the smaller pieces reached more of the evidence

Two results stand out. First, D7 finally moved: 4 of 10 quotes against 0 of 10 in every flat mode. That is the same mechanism as the rank cliff, seen from the other side — with dozens of places available, both filings fit. Second, D3 went from 1 of 10 to 8 of 10, and D5 from 6 to 9.

Ultimately the explanation is budget arithmetic rather than anything clever. A 1,024-token chunk carries its whole neighbourhood into the prompt and occupies one of eight places. A 128-token chunk occupies an eighth of a place. For the same number of tokens you can therefore point at dozens of separate spots in the corpus instead of eight. When the evidence is scattered by design, the number of independent pointers mattered more than the size of each one.

One caution about what this does not establish. automerge_wide changes two things at once — chunk size and parent merging — and this run does not separate them. Doing that needs a flat index with 128-token chunks, which is a separate experiment and was not run.

How to read a score of 47.5%

First of all, that number means 19 of 40 questions answered correctly, judged by a model that never saw which system produced which answer. It does not mean 47.5% of some quality scale.

The rest breaks down as follows: 17 questions were refused, and 5 of those refusals were correct, because five questions in the golden set have no answer in the corpus at all. The remaining four answers were partial or wrong. On the numeric questions the same run matched 87.0% of 23 values exactly, and invented nothing.

Traps worth knowing before you start

Symptom	Cause	Fix
Empty answers, or very slow ones with a thinking block	is_chat_model=False sends prompts to /v1/completions	Set it to True explicitly
Answers look fine, but they came from a refine chain	context_window=3900 cannot hold eight 1,024-token chunks	Set the real window of your model
Two independent builds produce different digests	The digest includes node_id, a fresh uuid4 per build	Hash content alone for cross-build comparison
Widening top-k in a fusion retriever changes the top eight	Reciprocal rank fusion recomputes over longer input lists	Treat the wider page as a separate configuration
A JSON result file will not parse	A PowerShell redirect wrote UTF-8 with a BOM	Read with utf-8-sig, or write with curl -o

Frequently asked questions

Is a reranker worth adding to a RAG pipeline?

On this corpus it changed nothing that mattered. The cross-encoder improved document recall by five documents and left correct answers unchanged at seven of forty. Cost is not the objection — on the GPU the pass takes 312 ms, roughly two percent of the time spent generating the answer. The objection is the window: chunks were 1,024 tokens and the model reads 512, so it scored 96.9% of candidates having seen only the first half. Match the reranker’s window to your chunk size before concluding anything about rerankers in general.

Why does my LlamaIndex retrieval return only two chunks?

VectorIndexRetriever.similarity_top_k defaults to 2. That value comes from the five-line starter example rather than from a production recipe. This benchmark set it to 8, which is the depth the naive baseline in episode 2 answered under.

Does chunk size matter more than the retrieval algorithm?

Here it did. Moving from 1,024-token chunks to 128-token leaves at a matched token budget took correct answers from 7 of 40 to 19 of 40 — a larger change than adding BM25, and far larger than adding a cross-encoder. Latency also fell, because the total token count fell.

Can I run the reranker on an AMD GPU inside podman on Windows?

Yes, and every number in this article was measured that way. Inside the benchmark itself the reranking pass takes a median of 312 ms on the W7900 against 12.19 s on CPU for the same 30 pairs — about 39 times faster, and it drops the mode’s end-to-end latency from 29.29 s to 17.71 s. The retrieval results are unchanged: both devices selected the same documents and scored the same 72.5% recall.

A podman machine is WSL2, so ROCm binds through /dev/dxg rather than the /dev/kfd it normally uses. Four pieces are needed and they come from three different places:

# the driver repo, for the WSL build of the HSA runtime
hsa-runtime-rocr4wsl-amdgpu     # repo.radeon.com/amdgpu/6.4.2/ubuntu
# the ROCm repo, for a profiler library the runtime loads while enumerating devices
hsa-amd-aqlprofile              # repo.radeon.com/rocm/apt/6.4.2
# the ROCm wheels
torch 2.6.0+rocm6.4.2           # repo.radeon.com/rocm/manylinux
# then point torch at AMD's runtime instead of its own bundled one
cp $(dpkg -L hsa-runtime-rocr4wsl-amdgpu | grep 'libhsa-runtime64.so.1.')    .../site-packages/torch/lib/libhsa-runtime64.so.1

Run it with --device=/dev/dxg and libdxcore.so mounted from /usr/lib/wsl/lib. Skip any one of the four and the symptom is the same every time: the device node is visible, torch reports ROCm, and device_count is zero.

How reproducible is a LlamaIndex index between builds?

Fully, at the level of content. Two independent builds of the same 33 documents — new container, new volume, documents staged again — produced byte-identical content digests. Their id-bearing digests differed, because node ids are random UUIDs. Compare content, not identity.

Summary

Five configurations of one framework, five separate builds, one corpus of 33 SEC filings and 40 questions whose answers sit in more than one document. Correct answers went from 5 to 19 out of 40 without changing the model, the prompt or the corpus.

Two things decided that range, and neither was the retrieval algorithm. The first is how much of the evidence actually reaches the prompt, which document-level recall does not measure — it reported 72.5% for a configuration that showed the model 18.7% of the gold passages. The second is chunk size at a fixed token budget: smaller pieces let the same number of tokens point at far more places in the corpus, which is exactly what scattered evidence needs.

Notably, the cheapest changes returned the most. Adding BM25 doubled correct answers for half a second of extra latency. Meanwhile changing the chunk size nearly tripled them and made answers faster. By contrast the cross-encoder returned nothing beyond the base mode — and it is worth being precise here, because its 312 ms on the GPU makes cost a poor reason to skip it. The reason to skip it is that on 1,024-token chunks it cannot read what it is judging. In practice, that suggests an order: chunk size and token budget first, lexical search second, reranking last.

Where LlamaIndex lands against the other frameworks in this series is not this article’s question. That comparison, per dispersion pattern, belongs to episode 10.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.