R2R benchmark on one GPU: hybrid search returned semantic search, and the search limit decided 11 or 22 of 40

Part of RAG & Vector Databases — retrieval-augmented generation and vector search.

Why an R2R benchmark, and what it asked

This R2R benchmark is the seventh framework episode of the series. It is the first one where the framework arrives as a product: an image to pull, a REST API, a PostgreSQL schema it manages, a dashboard. The master plan filed R2R under “RAG as a ready-made API” and asked one question of it. Does the thirty-minute setup survive a local model server, a Windows host and forty questions whose answers sit in more than one filing?

The answer is yes, with four defaults that had to be found first. The result depends almost entirely on one number R2R does not advertise: how much evidence its default search limit hands the model. Read the way it ships, R2R answered 11 of 40. Read with the same evidence budget as the two previous episodes, on the same index, it answered 22. That placed it second on the series’ frozen scorecard.

What R2R is, and why the image is the artifact

R2R by SciPhi is the first framework in this series that arrives as a product rather than a library. There is an image to pull, a REST API with versioned routes, a PostgreSQL schema it manages itself, and a dashboard. The master plan filed it under “RAG as a ready-made API”. The question for this R2R benchmark was whether the thirty-minute setup the project promises survives contact with a local model server on Windows.

That shape changes what is measured. In the LlamaIndex and Haystack episodes the container held my thin HTTP wrapper around the library. Here the adapter speaks R2R’s own /v3 API and nothing of mine runs inside the container except one configuration file. What a reader gets from podman pull is what this episode scored.

The version is 3.6.6, the newest tag on Docker Hub in September 2026. It is also a slightly odd one. The repository’s last release tag is 3.6.5, and the 3.6.6 image was built from a commit that carries no tag. Eleven commits landed on main after it. One of them adds the api_base key to the embedding section that this episode had to work around.

The R2R benchmark setup, command by command

Two model endpoints must answer before anything starts. Generation runs on a local llama-server over HTTPS with a certificate issued to bestin-it.test. No container hostname can match that name, so a small relay puts plain HTTP on port 8084 in front of it. Embeddings run on a second llama-server on 8083.

C:/Tools/run_infrastructure/start-llama-server.ps1 -Model qwen3.8   # :8082 HTTPS
.benchmarkharnessstart_embed_server.ps1                          # :8083 HTTP
python benchmark/harness/tls_bridge.py                              # :8084 -> :8082

Then the stack. R2R needs PostgreSQL with pgvector. The two run in one podman pod, so the server reaches the database as localhost, the way the project’s own compose file arranges it. The image is the published one plus a single file:

podman pull docker.io/sciphiai/r2r:3.6.6
podman pull docker.io/pgvector/pgvector:0.8.6-pg16
podman build -t nvda-r2r:3.6.6 -f articles/E-08-r2r/Containerfile articles/E-08-r2r

# Containerfile, in full
FROM docker.io/sciphiai/r2r:3.6.6
COPY r2r.toml /app/config/r2r.toml
ENV R2R_CONFIG_PATH=/app/config/r2r.toml R2R_PORT=7272 R2R_HOST=0.0.0.0

The start script creates the pod, waits for PostgreSQL, and starts the server with the model endpoints as environment variables. There are two of them. R2R routes by model prefix, and each prefix carries one base URL:

podman pod create --name nvda-r2r -p 7272:7272 --add-host llmhost:192.168.64.1
podman run -d --pod nvda-r2r --name nvda-r2r-postgres 
    -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=postgres 
    -v nvda-r2r-pgdata:/var/lib/postgresql/data docker.io/pgvector/pgvector:0.8.6-pg16
podman run -d --pod nvda-r2r --name nvda-r2r-server 
    -e R2R_POSTGRES_HOST=localhost -e R2R_POSTGRES_PORT=5432 
    -e R2R_POSTGRES_USER=postgres -e R2R_POSTGRES_PASSWORD=postgres 
    -e R2R_POSTGRES_DBNAME=postgres -e R2R_PROJECT_NAME=nvda 
    -e LMSTUDIO_API_BASE=http://llmhost:8084/v1 -e LMSTUDIO_API_KEY=local 
    -e OPENAI_API_BASE=http://llmhost:8083/v1 -e OPENAI_API_KEY=local 
    nvda-r2r:3.6.6
curl.exe -s http://127.0.0.1:7272/v3/health

The address 192.168.64.1 is the Windows host as the podman machine sees it, read from the machine’s default route. The name host.docker.internal resolves to the WSL bridge instead and never reaches the model servers. All of this is run.ps1 -Recreate in the code bundle.

Defaults that would have changed what this measures

Every value below was read from the image before the first measurement. The configuration file in the bundle carries each one with its reason next to it.

Chunks are cut in characters, not tokens

R2R’s default ingestion is chunking_strategy = "recursive" with chunk_size = 1024 and chunk_overlap = 512. The splitter behind those numbers is a RecursiveCharacterTextSplitter. A chunk is at most 1,024 characters, and half of every chunk repeats in the next one. On this corpus that is roughly a quarter of the 1,024-token chunks the two previous episodes used. The default search limit is 10, so a default prompt carries about a quarter of the evidence the floor was given. Both defaults are measured as shipped, and two further generations match the budget in tokens rather than in chunks.

The model routing goes by prefix, and one of the prefixes is misspelt in the docs

R2R’s own completion provider picks an HTTP client by the model name’s prefix. lmstudio/<model> goes to LMSTUDIO_API_BASE; the shipped r2r.env spells that variable LM_STUDIO_API_BASE, and the code never reads that spelling. The api_base key under [completion.generation_config] is ignored by this provider. Embeddings take a different road. The openai embedding provider validates model names against a fixed table and rejects anything that is not an OpenAI model. A local embedder therefore has to go through LiteLLM, with the openai/ prefix and OPENAI_API_BASE.

[app]
fast_llm    = "lmstudio/Qwen3.8-27B-UD-Q4_K_XL.gguf"
quality_llm = "lmstudio/Qwen3.8-27B-UD-Q4_K_XL.gguf"

[completion]
provider = "r2r"
concurrent_request_limit = 1

[embedding]
provider = "litellm"
base_model = "openai/nomic-embed-text-v1.5.Q4_K_M.gguf"
base_dimension = nan
batch_size = 16

A dimension you are not allowed to state

The first upload failed with a message about a model that was not in use. It read: Setting dimensions is not supported for OpenAI text-embedding-3 and later models. R2R forwards base_dimension as the OpenAI dimensions parameter on every embedding call. LiteLLM 1.75.8 refuses that parameter for any openai/ model whose name is not text-embedding-3-*. The llama-server itself accepts the parameter. The only way out in 3.6.6 is base_dimension = nan. R2R then drops the parameter and creates the pgvector column without a declared dimension. The project’s own lm_studio.toml does the same.

The default ingest calls the chat model 33 times

Ingestion is not embeddings only. With skip_document_summary at its default of false, every document is sent to the chat model for a summary before it is chunked. The input can be up to 100,000 characters per document. The summary is embedded and stored for R2R’s agent, which plain retrieval never reads. It stays on here because it is what the default ingest builds, and it is counted as build cost.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

Two switches for a graph that was never built

automatic_extraction = true would send every chunk to the chat model for entity extraction during ingest. Under the light deployment that is a no-op, with a log line saying so. It is set to false anyway. Less visible: graph_settings.enabled is true in every search request by default. A stack with no graph still queries its empty graph tables on every question. The adapter switches it off so the flat modes measure the chunk search alone.

The embedding batch that would not fit

The shipped batch_size is 128 texts per request. With 1,024-character chunks that is about 32,000 tokens against the embedding server’s 8,192-token context, the same failure class the RAGFlow episode paid for. The configuration sets 16, and the preflight measures the worst case against the actual chunk sizes.

Four R2R benchmark modes, one corpus uploaded four times

Four configurations were measured, each on its own pod. That means a fresh PostgreSQL volume, the server started again, and the 33 filings uploaded again through the API. The first two are R2R as shipped. The other two hand the model the same evidence budget in tokens that the LlamaIndex and Haystack episodes used. The LlamaIndex episode showed that “ten chunks” is not a budget when chunk sizes differ four-fold.

Mode	Search	Chunks to the model	Evidence tokens, median
semantic	pgvector, cosine distance	10 (R2R default)	2,238
hybrid	pgvector + PostgreSQL full-text, reciprocal rank fusion at the shipped weights (5.0 semantic, 1.0 full-text, rrf_k 50)	10	2,259
semantic_wide	as semantic	37	8,573
hybrid_wide	as hybrid	37	8,573

Thirty-seven is measured rather than chosen. The default generation put a median of 2,238 evidence tokens into the prompt at the default limit of 10. The token-unit recipes of the previous episode put 8,206, so 10 × 8,206 / 2,238 rounds to 37. The matched generations landed at 8,573 tokens, within five percent of the target.

What the R2R index cost to build, four times

Every build produced the same 4,997 chunks and the same content digest. The digest over chunk ids agreed as well, because R2R derives a chunk’s id deterministically from the document id and its position. The four builds took 1,298 to 1,318 seconds each, just under 22 minutes, and the database weighed 45.7 to 45.9 MB afterwards.

That is not embedding time. Splitting and embedding 4,997 chunks of about 224 tokens takes well under two minutes on this server. The rest is the 33 document summaries the default ingest requests from the chat model, one per document, with up to 100,000 characters of input each. The median document took 14.9 seconds from upload to indexed, the slowest 95.9. For comparison, the Haystack build of the same corpus, embeddings only, took 51 seconds.

What the R2R benchmark retrieved

Document recall at the default limit was 80.0%: 32 of 40 questions had every gold document represented in the prompt. The figure was identical in semantic and hybrid. The matched-budget generations reached 97.5%, 39 of 40, again identical between the two searches. The one remaining document was absent from the 37-chunk page but present at a page of 50. Nothing was unreachable; it was ranked, and cut.

Counting quotes rather than documents tells the usual harsher story. At the default limit the prompt carried 24 of the 75 gold evidence passages in semantic and 25 in hybrid. At the matched budget it carried 34 in both. Per pattern, the default limit found every D5 passage: an event in an 8-K and its consequence, ten of ten. It found none of the D6 passages, governance in the proxy against operations in the 10-K. It found none of D7 either, a figure before and after the ten-for-one split. Widening the budget lifted D1 from two passages to five and D3 from three to six, and left D7 at zero of ten.

Hybrid search that is semantic search

The hybrid and semantic generations handed the model an identical list of chunks in 39 of the 40 questions. The two matched-budget generations did the same. That is not a rounding coincidence. R2R reports, for every chunk on a hybrid page, its rank on the semantic list and its rank on the full-text list. The pool audit read those ranks for every question.

Of 60 gold document slots on the 37-chunk page, semantic search alone found 58, both searches found one, and full-text search alone found none. Over the whole page, 1,480 chunks across forty questions, the full-text search returned two chunks, in one question. At a page of 50 it was two of 2,000.

The reason is one line of SQL:

-- core/providers/database/chunks.py, full_text_search
WHERE fts @@ websearch_to_tsquery('english', $1)

The whole question is the argument. websearch_to_tsquery joins every word of it with AND, and a benchmark question averages 22.6 words. A 1,024-character chunk that contains all of them barely exists, so the full-text list comes back empty. Reciprocal rank fusion over one non-empty list is that list’s order. The shipped weight of 5.0 for semantic against 1.0 for full-text never gets a chance to matter. The Haystack episode could measure BM25 on its own and found it answered more questions than dense retrieval. Here there is nothing to measure on its own, because the sparse half of the hybrid never returns.

Where the evidence sat, and what cut it

The pool audit separates two failures a recall number merges. At the default limit, 50 of 60 gold document slots reached the prompt and 10 did not. At a page of 50, all ten were present, ranked between 11 and 50. Nothing was unreachable by the embedder. Every miss at the default limit was a cut, which points at the limit and not at the index.

Widening the page under rank fusion changed the fused top ten in 2 of 40 questions, and the top 37 in 1 of 40. The LlamaIndex episode found that a widened page under fusion is a different retrieval configuration. Here the fusion runs over one list, so widening it behaves like a plain deeper view. That is one more way of saying the same thing about the full-text half.

What the R2R benchmark scored across four modes

Every run answered all forty questions without an error. None of the 160 answers contained a number absent from its own retrieval. The blind judge’s verdicts:

Mode	Documents	Quotes	Correct	Partial	Wrong	Refusals	Numeric exact	Median latency
the floor (episode 2)	70.0%	—	6/40	7	—	27	—	17.6 s
semantic	80.0%	24/75	11/40	8	3	18	59.1%	6.83 s
hybrid	80.0%	25/75	12/40	8	2	18	63.6%	6.88 s
semantic_wide	97.5%	34/75	22/40	8	3	7	66.7%	21.91 s
hybrid_wide	97.5%	34/75	22/40	8	3	7	66.7%	21.83 s

At the default limit R2R refused 18 questions, only five of which have no answer in the corpus. Thirteen refusals were answerable questions whose evidence the ten small chunks did not carry. Widening the budget to the token count the earlier episodes used cut the refusals to seven, five of them the right call. It doubled the correct answers, from 11 to 22. Nothing else changed: same index, same search, same prompt shape, same model.

Determinism, measured for free

The semantic and hybrid generations sent the model an identical prompt in 39 of 40 questions. In those 39 the model’s answers were identical to the byte, and the judge’s verdicts with them. The single question with a different prompt, D2-02, went from WRONG to CORRECT, which is the entire one-answer difference between the two modes. The matched-budget pair repeated the pattern: 36 identical prompts, 38 identical answers, identical verdict counts.

This is worth a paragraph because it settles a question the reader would otherwise have to ask. At temperature zero on one llama-server slot, with a prompt that contains nothing time-dependent, the same evidence produces the same answer. A difference between two R2R modes in this episode is a difference in retrieval, not noise in generation. The kotaemon episode that follows will show the opposite case.

Where the answers came from, and where they did not

Per pattern, at the matched budget: D2 and D3, the in-document footnote and definition patterns, went five for five. D4, D5 and D6 answered three of five each. D1, the table-against-narrative pattern, answered two: the model saw the table rows but not the sentence that explains them in three cases. D7, the restatement across the stock split, answered one of five with one quote out of ten in the prompt. D7-04 shows the shape of the failure. Asked for the original and the restated share count, the model saw only the restated filing and reported 24,940 million shares in both the originally reported and restated amounts. The number is real and the claim is wrong. The floor in episode 2 answered the same question with a share count that exists in no filing. R2R stayed on real numbers and still missed the restatement, because the pre-split filing was not in front of it.

D8, the five questions with no answer in the corpus, was refused five times out of five in every mode. That is the one pattern where the default limit and the matched budget agree completely. R2R does not invent an answer when the evidence is thin; it declines. The cost of that discipline is the thirteen answerable questions it declined at the default limit.

How to read a score of 80.1

On the series scorecard, frozen before any framework was measured, R2R’s matched-budget generations score 80.1 of 100. The default generations score 61.4 and 62.5. The points are not a percentage of correct answers. Retrieval is worth 40 of the 100 and correctness 20, so the restatement in questions is the one to remember:

• for 24 of the 25 questions whose answer sits in more than one document, it retrieved every evidence document (28.8 of 30 points);
• for 15 of the 15 questions settled inside a single document, it retrieved every evidence document (10 of 10);
• the blind judge called 22 of 40 answers fully correct (11 of 20);
• 22 of the 33 numeric questions hit the value within two percent (6.7 of 10);
• 38 of 40 answer-or-refuse decisions were the right call (9.5 of 10);
• 68.8% of answers stand on evidence the system retrieved itself, with no invented numbers to subtract (6.9 of 10);
• 0.36 hours to build the index once (4.9 of 5) and a median 21.9 seconds per answer (2.3 of 5).

Just under 39 of those points came from finding the right documents; 11 came from answering correctly, 22 times out of 40. A reader who needs one number should take the 22.

Summary of the R2R benchmark

R2R is the thirty-minute setup it claims to be, once four defaults are known. The chat model is routed by a prefix the docs misspell. The embedder must go through LiteLLM without a stated dimension. The default ingest calls the chat model 33 times for summaries. And hybrid search is semantic search, because the full-text query ANDs every word of the question. Measured as shipped, with 1,024-character chunks and a limit of 10, it answered 11 of 40. Handed the same evidence budget as the previous two episodes, on the same index, it answered 22 of 40. That came with 97.5% document recall, 95% correct refusal decisions and no invented numbers, the second-best result of the series on the frozen scorecard. The lesson is the one the LlamaIndex and Haystack episodes reached from the other direction. The number of chunks a framework shows the model is a budget, and the budget decides more than the retriever does.

Traps worth knowing before your own R2R benchmark

Each entry gives the symptom first, because the symptom is what you have when you start searching. Messages are reproduced literally.

• “‘2024-11-07_8-k_…’ is not a valid DocumentType” (HTTP 415) on upload. A title key in the upload’s metadata replaces the file name, and the document type is then read from the extension of the replacement. A title without a dot becomes an extension the size of the whole name. Leave title out, or give it the extension.
• “Setting dimensions is not supported for OpenAI text-embedding-3 and later models” on the first embedding call. The model in use is nomic, on a server that accepts the parameter. LiteLLM refuses dimensions for any openai/ model whose name is not text-embedding-3-*, and R2R forwards base_dimension as that parameter. Set base_dimension = nan; the pgvector column is created without a declared dimension, which is what the project’s own LM Studio config does.
• “Failed to truncate texts: This model isn’t mapped yet” on every batch. R2R’s guard against inputs longer than the embedder’s window looks the model up in LiteLLM’s price table. An unmapped local model disables the guard silently. Harmless at the default 1,024-character chunks, not harmless at a chunk size chosen to match a token-unit episode.

Model routing and ingest cost

• The chat model never gets called, or gets called at api.openai.com. The r2r completion provider routes by model prefix and ignores api_base in the TOML. Use lmstudio/<model> with LMSTUDIO_API_BASE, no underscore; the shipped r2r.env spells the variable with one, and that spelling is never read.
• A build that takes twenty minutes for two minutes of embeddings. The default ingest asks the chat model for a summary of every document, up to 100,000 characters of input each. ingestion_mode=fast skips it. Here it stayed on and is counted as build cost, because it is what the default builds.
• Hybrid search that returns exactly what semantic search returned. The full-text half runs websearch_to_tsquery over the whole question, which ANDs every word. Across 1,480 page chunks in forty questions it returned two. If you want the sparse half to contribute on long questions, it needs a query it can match. R2R offers no hook for one in the search settings.

R2R benchmark FAQ

Does R2R’s hybrid search improve on its semantic search?

Not on this corpus with the shipped weights. The two modes handed the model an identical chunk list in 39 of 40 questions. The full-text search returned two chunks across 1,480 on the page. The fusion is arithmetic over one list.

Why are R2R’s chunks so small?

The default splitter is a recursive character splitter at 1,024 with 512 of overlap. A chunk is about 224 tokens, and half of it repeats in the next one. At the default search limit of 10 the model sees about 2,200 tokens of evidence. That is a quarter of what the previous two episodes gave it.

Can R2R 3.6.6 use a local embedding server?

Yes, through LiteLLM with the openai/ prefix and OPENAI_API_BASE, and with base_dimension = nan. The built-in openai embedding provider rejects any model that is not in its table of OpenAI models. The TOML has no api_base key for embeddings in this version.

How long does R2R take to index 33 filings?

About 22 minutes for 4,997 chunks, four times out of four, with 33 chat-model calls for document summaries. Embedding alone would take under two minutes; the summaries are the cost.

Is the R2R index reproducible between builds?

Four independent builds of the corpus produced the same chunk count, the same content digest and the same id-bearing digest. A chunk’s id is derived from the document id and its position rather than drawn at random.

Why was the knowledge graph not measured?

R2R’s graph extraction is one chat-model call per chunk, and the default chunking yields 4,997 chunks. That is an order of magnitude more calls than the GraphRAG episode’s 13-hour build. Community building also needs a separate clustering container. The series already has two measured graphs to compare in the finale; this episode measured the product a reader gets in thirty minutes.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.