kotaemon benchmark on one GPU: hybrid mode is full-text search, and four builds gave four indexes

Part of RAG & Vector Databases — retrieval-augmented generation and vector search.

Why a kotaemon benchmark, and what it asked

This kotaemon benchmark is the last framework episode of the series. It is the bonus the master plan filed under “RAG with a ready-made UI for non-programmers”. The question it asked was the series’ own. Put the same 33 filings and the same forty questions through a framework whose selling point is that nobody needs to write code. Then see what its defaults deliver.

Getting an answer meant getting behind the interface, because kotaemon has no other door. What came out is a lexical index that does one thing no dense retriever in the series managed. It also showed a hybrid mode that is not one, and a build that will not repeat.

What kotaemon is, and why the chat page had to be taken apart

kotaemon by Cinnamon is the framework the master plan filed under “RAG with a ready-made UI for non-programmers”. It is a Gradio application over two Python packages, kotaemon for the components and ktem for the application. It has a file index, a chat page, a settings page and optional graph indexes that wrap Microsoft GraphRAG and LightRAG. The version measured is v0.12.0, released on 31 May 2026, the newest tag at the time of writing.

Two facts decided how this kotaemon benchmark could be run at all. The package is not on PyPI, so pip install kotaemon resolves to nothing and the framework is installed from its git checkout. And there is no API: app.py launches Gradio and nothing else, and the chat page names no endpoint that gradio_client could call. Sending a question is a chain of twenty-five .then() steps over browser state.

Driving a browser forty times per generation is not a measurement anyone can reproduce. So this episode does what the LlamaIndex and Haystack episodes did with their libraries. A small HTTP face inside the container constructs the same objects the chat page constructs. The file index’s indexing pipeline, its retrieval pipeline, and the “Simple QA” reasoning pipeline are kotaemon’s own classes. They are wired the way ktem wires them, with the application’s flattened settings passed in. The Gradio page still runs on the same volume for screenshots, after a measured build, never for a measurement.

The kotaemon benchmark setup, command by command

The same two model endpoints as every container episode, and the same relay in front of the HTTPS generation server:

C:/Tools/run_infrastructure/start-llama-server.ps1 -Model qwen3.8   # :8082 HTTPS
.benchmarkharnessstart_embed_server.ps1                          # :8083 HTTP
python benchmark/harness/tls_bridge.py                              # :8084 -> :8082

Then an image built from the release tag and the project’s frozen lock file, which is how kotaemon’s own Dockerfile installs it:

git clone --depth 1 --branch v0.12.0 https://github.com/Cinnamon/kotaemon /app
pip install uv && uv sync --frozen --no-cache
python -c "import tiktoken; tiktoken.encoding_for_model('gpt-3.5-turbo')"   # baked in

The last line matters. ktem calls tiktoken.encoding_for_model at import time. tiktoken fetches the encoding from the network on first use, and the container has no route to the internet. Baked in at build, the first import does not fail twenty seconds into a measured run.

podman build -t nvda-kotaemon:0.12.0 -f articles/E-09-kotaemon/Containerfile articles/E-09-kotaemon
podman run -d --name nvda-kotaemon --add-host llmhost:192.168.64.1 
    -v nvda-kotaemon-data:/app/ktem_app_data -p 9625:9625 nvda-kotaemon:0.12.0
curl.exe -s http://127.0.0.1:9625/health

Four environment switches turn off what a benchmark cannot use. They are the login page (KH_FEATURE_USER_MANAGEMENT), the first-run wizard that pulls Ollama models (KH_ENABLE_FIRST_SETUP), and the two graph index types whose packages the lite install does not carry (USE_LIGHTRAG, USE_MS_GRAPHRAG). All of this is run.ps1 -Recreate in the bundle; run.ps1 -UI starts the Gradio page on the same volume.

Defaults that would have changed what this measures

One base URL for chat and embeddings, and timeouts of 20 and 10 seconds

First, kotaemon fills its model pools from environment variables in flowsettings.py. The OpenAI block uses one OPENAI_API_BASE for both the chat model and the embedder. This series runs them on two servers. The chat entry also carries timeout: 20 and the embedding entry timeout: 10. A 27B model on one slot exceeds the first on the first long prompt. The episode’s own flowsettings.py imports the upstream module whole and replaces only the two pools:

KH_LLMS = {"local": {"spec": {
    "__type__": "kotaemon.llms.ChatOpenAI", "temperature": 0, "max_tokens": 512,
    "base_url": "http://llmhost:8084/v1", "api_key": "local",
    "model": "Qwen3.8-27B-UD-Q4_K_XL.gguf", "timeout": 1800}, "default": True}}
KH_EMBEDDINGS = {"local": {"spec": {
    "__type__": "kotaemon.embeddings.OpenAIEmbeddings",
    "base_url": "http://llmhost:8083/v1", "api_key": "local",
    "model": "nomic-embed-text-v1.5.Q4_K_M.gguf", "timeout": 600}, "default": True}}

The pools are copied into SQLite on first start. Only names not already in the database are written. A volume that survived an earlier configuration keeps the earlier configuration. That is one more reason every generation here gets a fresh volume.

200 chunks per embedding request

The indexing pipeline embeds chunk_batch_size = 200 chunks per request, and its chunks are 1,024 tokens. One request would carry about 200,000 tokens against the embedding server’s 8,192-token context. The number lives in one place, the pipeline object that route() builds per file. The HTTP face subclasses the indexing pipeline with one overridden method that sets it to 4. Nothing else in the framework’s indexing is touched.

Hybrid is concatenation, not fusion

The file index offers three retrieval modes: vector, text and the default hybrid. Reading VectorRetrieval.run for hybrid shows what the word means here:

result = [RetrievedDocument(**doc.to_dict(), score=-1.0)
          for doc in ds_docs if doc not in vs_ids]      # full-text hits first
result += [RetrievedDocument(**doc.to_dict(), score=score)
           for doc, score in zip(vs_docs, vs_scores)]   # vector hits after
...
return documents[:top_k]                                # cut from the top

The full-text hits go first with a placeholder score. The vector hits follow, and finally the list is cut at top_k. There is no rank fusion. If the full-text search returns ten or more hits, the prompt is full-text only and the vector search contributed nothing. The dedup test on the first line compares a Document object to a list of id strings, so it never removes anything either. The episode measures this rather than describing it: every answer records how many of its prompted chunks came from each search.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

Two LLM calls per question that never change the answer

With the shipped settings the chat page makes two extra kinds of model call per question. use_llm_reranking is on. It installs a relevance scorer that sends every retrieved chunk to the LLM for a 0 to 10 grade, ten concurrent requests, after the answer has been generated. In other words, the grades reach the panel, not the prompt. highlight_citation is on, and it runs a citation extractor with tool calling in a thread parallel to the answer. Neither changes the answer text, both change the latency. The fourth generation of this episode measures them as shipped; the other three switch them off.

The default reranker is a service you do not have

use_reranking is on by default and the reranker in the default pool is Cohere’s rerank-v4.0-fast. Without a key, the component prints Cohere API key not found. Skipping rerankings. and returns the list unchanged. There is no local cross-encoder in kotaemon’s ranking classes. Consequently, the reranking stage in this episode is a documented no-op rather than a switched-off feature.

Four kotaemon benchmark generations, one corpus staged four times

Four configurations were measured, each on its own container: a fresh volume, the 33 filings staged again, the index built again. The first three are the file index’s three retrieval modes with the chat page’s two LLM side-calls switched off. The fourth is the chat page as shipped.

kotaemon cuts 1,024-token chunks with 256 of overlap and retrieves ten of them by default. The model therefore saw about 7,700 to 8,000 evidence tokens per question, within five percent of the token-unit budget of the two previous episodes. No budget-matching generation was needed; this is the first framework of the series whose defaults land there on their own.

What the kotaemon index cost to build, four times

Each build took 151 seconds and no chat-model calls. The four builds did not, however, produce the same index: 1,103, 1,108, 1,102 and 1,104 chunks, and four different content digests. That is the episode’s build finding and it has a section of its own below.

What the kotaemon benchmark retrieved

Strict document recall is the share of questions whose every gold document was represented in the prompt. It was 72.5% for vector (29 of 40), 50.0% for text (20 of 40), 47.5% for hybrid (19 of 40) and 55.0% for the shipped default (22 of 40). Counting gold evidence quotes instead of documents reverses the order. Vector search put 17 of the 75 gold passages into the prompt, full-text search 30, the shipped default 30. The Haystack episode saw the same reversal between dense and lexical retrieval, and there too the answers followed the passages.

Per pattern, the lexical index found what the dense one missed. D6 needs the proxy statement and the 10-K together: full-text search put five of its ten passages in the prompt, vector search one. D4, the quarterly-against-annual pattern, went from two passages to five. D5, the event-and-consequence pattern that dense retrieval owned in every earlier episode, was the one place vector search kept up. Namely, four of ten against six. D7, the restatement across the stock split, stayed at zero of ten in all four generations. It has done so in every flat retrieval mode of the series.

Hybrid is full-text with a vector search that never reaches the prompt

Every one of the 400 chunks the hybrid generation put in front of the model came from the full-text search. Likewise, the same holds for the 400 of hybrid_default. Not one vector hit reached a prompt in eighty questions. The mechanism was read from the code before the run and is measured here rather than assumed:

# kotaemon/indices/vectorindex.py, VectorRetrieval.run, retrieval_mode == "hybrid"
result = [RetrievedDocument(**doc.to_dict(), score=-1.0)
          for doc in ds_docs if doc not in vs_ids]     # full-text hits, marked -1.0
result += [RetrievedDocument(**doc.to_dict(), score=score)
           for doc, score in zip(vs_docs, vs_scores)]  # vector hits after them
...
documents = documents[:top_k]                          # keep the first ten

The full-text search is asked for a hundred candidates and returns at least ten for every question in this corpus. As a result, the cut never reaches the vector list. On a standing index, the text and hybrid modes return the same ten chunks for the same question. The container records which search each prompted chunk came from, by the placeholder score kotaemon assigns to full-text hits. In total, the pool audit sums them to full-text 400 and vector 0, in both hybrid generations.

Same corpus, same settings, four different indexes

The four builds produced 1,103, 1,108, 1,102 and 1,104 chunks. A fifth, built afterwards for the screenshots in this article, produced 1,105. kotaemon’s splitter is deterministic. For example, run on its own over the 33 files, in two separate processes, it produced 950 chunks with the same content digest both times. The indexing pipeline adds something the splitter alone does not have.

It adds metadata. Before a file is split, IndexPipeline attaches llama-index’s default file metadata to it: path, name, type, size, and the creation and modification dates of the staged copy. In addition, it adds a file_id drawn from uuid4(). kotaemon’s TokenSplitter wraps llama-index’s TokenTextSplitter, which counts the tokens of that metadata against the chunk budget. Fed pipeline-shaped metadata, the same splitter produced 1,096 chunks instead of 950. Moreover, with a different random file_id it produced a different content digest at the same count. The token count of a random hexadecimal string is not constant. Every file therefore gets a slightly different effective chunk size in every build, and the file dates add a second moving part.

The consequence for reading the results is this. The text, hybrid and hybrid_default generations retrieve through the same full-text index over the same recipe. Their document recall of 20, 19 and 22 of 40 is three samples of one recipe on three different cuts of the corpus, not three configurations. Their prompts share the same ten chunks in only three of forty questions. The LlamaIndex episode found random ids in the digest. The RAGFlow episode found a build that would not repeat. This is the third form, a random id inside the token budget.

A page that changes when it is widened

The pool audit asks the container for a wider page than the mode measures, fifty chunks instead of ten. Then it checks whether the first ten stayed the same. Under vector search they did, in 39 of 40 questions. Under full-text search they did not: the top ten changed in 32 of 40 questions for text, 34 for hybrid and 33 for hybrid_default. kotaemon’s full-text query goes to LanceDB with an id in (...) filter over every chunk id in scope and a limit that grows with the page. Because of that, the first ten it returns depend on that limit. The widened audit therefore reads as a different retrieval configuration, the way the LlamaIndex episode found for rank fusion. A full-text “top ten” in kotaemon is a property of the query and the depth together.

What the kotaemon benchmark scored across four generations

Every run answered all forty questions without an error. None of the 160 answers contained a number absent from its own retrieval. The blind judge’s verdicts:

Document recall and correct answers point in opposite directions, for the third framework in a row. Vector search represented the most gold documents in the prompt, 72.5%, and answered the fewest questions, 12. Full-text search represented the fewest, 50.0%, and answered the most, 15. The quote count had said so before the judge did: 17 gold passages in the prompt for vector search, 30 for full-text. The Haystack episode found the same reversal between dense and lexical retrieval, and the LlamaIndex episode introduced the measure that makes it visible.

Refusals were the largest verdict class in every generation: 16 to 18 of 40, against five questions that have no answer in the corpus. The right answer-or-refuse decision was made in 62.5% to 70.0% of questions. kotaemon did not invent a figure once, but it declined eleven to thirteen answerable questions per generation.

What the shipped defaults cost

The hybrid_default generation is the chat page as it ships. It is the same hybrid retrieval, plus the LLM relevance scorer over the ten retrieved chunks and the LLM citation extractor, both on the same generation server as the answer. The median latency went from 20.46 seconds to 69.93 seconds. In other words, three and a half times, for the same recipe and the same evidence budget. The relevance scores reach the panel, not the prompt; the citations reach the highlighting, not the answer. Retrieval itself took 1.4 seconds per question in this generation against 0.75 in hybrid. This is because the scorer’s ten concurrent calls queue on the one slot the answer also needs.

Whether the answers are “the same” is a question this pair can only half answer. The two generations share the same ten chunks in one question out of forty, because their indexes differ. Even so, 25 of their 40 answers are identical to the byte. Short numeric answers converge from different evidence; the pair measures the cost of the defaults cleanly and the determinism of the model only loosely. The R2R episode, with a reproducible index, measured the determinism cleanly: 39 identical prompts, 39 identical answers.

Where the answers came from, and where they did not

Full-text search answered every D2 question, the footnote pattern, five of five, where vector search answered one. It also answered D7, the restatement across the ten-for-one split, in two of five questions. The hybrid generation answered four of five, the best D7 result of any flat retrieval mode in the series so far. Vector search answered none, as no dense mode has. The D7 questions name the split and the restated figures literally. Hence a lexical index finds the sentence that a similarity search cannot tell from its pre-split twin.

The price of the lexical index shows on D4 and D5, the quarterly-against-annual and event-to-consequence patterns. There it was one correct answer each for text and hybrid, against two each for vector. D8, the five questions with no answer, was refused four or five times out of five in every generation. Unfortunately, the refusals did not stop there.

How to read a score of 58.1

On the series scorecard, frozen before any framework was measured, kotaemon’s vector generation scores 58.1 of 100. text scores 51.6, the shipped default 49.3 and hybrid 48.9. Retrieval is worth 40 of the 100 points and correctness 20, which is why the generation with the fewest correct answers scores highest. In questions, for the vector generation:

• for 16 of the 25 questions whose answer sits in more than one document, it retrieved every evidence document (19.2 of 30 points);
• for 13 of the 15 questions settled inside a single document, it retrieved every evidence document (8.7 of 10);
• the blind judge called 12 of 40 answers fully correct (6 of 20);
• 14 of the 22 numeric questions it answered hit the value within two percent (6.4 of 10);
• 25 of 40 answer-or-refuse decisions were the right call (6.2 of 10);
• 42.5% of answers stand on evidence the system retrieved itself, with no invented numbers to subtract (4.2 of 10);
• 0.04 hours to build the index (5 of 5) and a median 19.6 seconds per answer (2.4 of 5).

A reader who needs one number should take the 15 of 40 that full-text search answered, and read it next to the 20 of 40 recall that came with it. In short, the answers came from the passages. The passages came from a lexical index that found what the dense one did not.

Summary of the kotaemon benchmark

kotaemon is an interface, and everything this episode had to do was get behind it. There is no API and no package. The framework installs from a git tag into a 10 GB image, and its retrieval objects can be driven only once one Gradio dependency is worked around. Measured that way, on 1,024-token chunks at the same evidence budget as the previous episodes, its three retrieval modes answered 12, 15 and 14 of 40 questions. Two of its defaults do not do what their names say. Hybrid retrieval is full-text retrieval: the vector half never reaches the prompt. And the same corpus builds a different index every time, because a random file id is counted against the chunk budget. The shipped chat page, with its relevance scorer and citation extractor, answers the same questions in three and a half times the time. What the episode measured best is the one thing kotaemon does that the dense retrievers before it did not. Its lexical index answered every footnote question and four of five restatement questions.

Traps worth knowing before your own kotaemon benchmark

Each entry gives the symptom first, because the symptom is what you have when you start searching. Messages are reproduced literally.

• “No matching distribution found for kotaemon”. The package is not on PyPI. Install from the git checkout with uv sync --frozen at a release tag, the way the project’s Dockerfile does. The published images track the main branch, and the only versioned tags are a year old.
• A 10 GB image for a “lite” install. Instead, the frozen lock resolves torch 2.8.0 with the full set of nvidia-*-cu12 wheels. That is 4.1 GB of CUDA libraries and 1.7 GB of torch in a virtual environment of 8.5 GB, on a machine with an AMD card. Meanwhile, nothing in the benchmark calls torch. The reranker is a Cohere service and both models are reached over HTTP.
• “‘NoneType’ object has no attribute ‘get_selected_ids'” when asking for retrievers. FileIndex.get_retriever_pipelines resolves the selected files through the chat page’s file selector, which only exists once the Gradio interface is built. Without a page, read every file id from the index’s Source table and call the retriever class’s get_pipeline with them. Apart from that, the rest of the method is unchanged.

Settings, batches and defaults

• The first long prompt times out. The shipped OpenAI entries carry timeout: 20 for chat and timeout: 10 for embeddings, and one OPENAI_API_BASE for both. Replace the two pools in flowsettings.py. The pools are copied into SQLite on first start and only for names not already there, so a reused volume keeps the old ones.
• An embedding request of 200 chunks of 1,024 tokens. IndexPipeline.chunk_batch_size is 200 and lives in the object route() builds per file. Subclass the indexing pipeline and set it in route(); no setting reaches it.
• “Cohere API key not found. Skipping rerankings.” Not an error. use_reranking is on by default and the reranker is a remote service; without a key the stage returns its input unchanged. kotaemon ships no local cross-encoder.
• Hybrid mode returns only full-text hits. By construction: the full-text list comes first, the vector list after, and the result is documents[:top_k]. There is no fusion, and the dedup test compares a Document to a list of strings, so it never fires.
• tiktoken reaches for the network at import. ktem calls encoding_for_model("gpt-3.5-turbo") when its pipelines module is imported. In a container without a route out, bake the encoding into the image with TIKTOKEN_CACHE_DIR.

kotaemon benchmark FAQ

Does kotaemon have a REST API?

No. The application is a Gradio page, and the page names no endpoint that gradio_client could call. This episode drove the framework’s own ktem pipeline objects from a small HTTP face inside the container. One Gradio dependency was replaced by a query on the index’s own table.

Is kotaemon’s hybrid retrieval better than vector or full-text alone?

On this corpus it is full-text retrieval under another name. The full-text hits come first in the merged list, the vector hits after, and the list is cut at ten. Every prompted chunk in eighty hybrid questions came from the full-text search. Its results, 14 and 14 of 40, sit between full-text alone (15) and vector alone (12).

Why do two kotaemon builds of the same files give different chunk counts?

The indexing pipeline attaches a random file_id and the file’s dates to every document before splitting. The splitter counts that metadata’s tokens against the chunk budget. A random hexadecimal string tokenises to a variable number of tokens, so the effective chunk size differs per file per build. The splitter alone is deterministic.

What do the LLM relevance scorer and citation highlighting cost?

Three and a half times the latency: a median of 69.9 seconds per question against 20.5 for the same retrieval with both switched off. Neither changes what reaches the prompt.

Which kotaemon retrieval mode answered the restatement questions?

The lexical ones. Full-text search answered two of the five D7 questions and the hybrid generation four. By contrast, vector search answered none, like every dense mode in the series. The questions name the stock split and the restated figures literally.

How long does kotaemon take to index 33 filings?

About 151 seconds for roughly 1,100 chunks of 1,024 tokens, with no chat-model calls, in every generation. The embedding request size had to be reduced from 200 chunks to 4 to fit the embedding server’s context.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.