GraphRAG benchmark on one GPU: 268 communities, 12 hours, one answer in five

Part of RAG & Vector Databases — retrieval-augmented generation and vector search.

This GraphRAG benchmark asks whether Microsoft’s community graph earns the day it takes to build. I indexed 33 NVIDIA SEC filings with a local 27B model, which cost 12.41 hours of model time and produced 1,400 entities, 3,616 relationships and 268 communities. Then I asked 40 cross-document questions whose answers sit in more than one filing. Local search retrieved every evidence document for every question — and answered one in five correctly.

What a GraphRAG community graph is supposed to buy you

GraphRAG does not stop at extracting entities. After it builds the graph, it clusters that graph into communities and writes a report for each one, at several levels of a hierarchy. Those reports are what global search reads instead of text chunks. The promise is a system that can answer questions no single passage contains, because somebody — the model, during indexing — already wrote the summary.

That promise costs real time. In this run the community layer alone took 3.07 hours, on top of the 9.34 hours the graph itself needed. Therefore the question is not whether the graph works. It is whether the hours come back as correct answers.

The shape of that hierarchy decides what global search costs later. Level 0 holds a few broad communities, the middle levels hold the most, and level 4 holds seven small ones. Since the search reads a whole level rather than a selection, the width of the middle is the query bill.

The benchmark is built to answer exactly that. Every filing comes from NVIDIA’s EDGAR archive, downloaded once and never edited. Forty questions, each tagged with a knowledge dispersion pattern and with the number of filings needed to answer it. No question can be answered from a single passage. In addition, five of them have no answer in the corpus at all, so a system that never refuses cannot score well.

The GraphRAG benchmark setup, command by command

Microsoft publishes GraphRAG as a Python package, not as an image. As a result there is no podman pull graphrag, and the first job is building one. In practice the image pins version 3.1.2 and adds exactly three files of ours: the configuration, an entrypoint and an HTTP wrapper.

# Containerfile - GraphRAG 3.1.2, nothing else added to the framework
FROM docker.io/library/python:3.12-slim
RUN pip install --no-cache-dir graphrag==3.1.2

WORKDIR /opt/nvda
COPY serve.py settings.yaml entrypoint.sh /opt/nvda/
RUN chmod +x /opt/nvda/entrypoint.sh

# GRAPHRAG_ROOT holds input/, output/, cache/, logs/ and prompts/.
# It is a volume, so an interrupted index survives a container restart.
ENV GRAPHRAG_ROOT=/data SERVE_PORT=9622 PYTHONUNBUFFERED=1
EXPOSE 9622
VOLUME ["/data"]
ENTRYPOINT ["/opt/nvda/entrypoint.sh"]

# Build the image, then start the container with the host address injected
podman build -t nvda-graphrag:3.1.2 -f Containerfile .

# Inside a podman WSL machine, host.docker.internal points at the WSL bridge,
# not at Windows. The real address comes from the machine's default route.
$hostIp = (podman machine ssh "ip route | awk '/default/ {print `$3}'").Trim()

podman run -d --name nvda-graphrag 
  --add-host "llmhost:$hostIp" 
  -v nvda-graphrag-data:/data 
  -p 9622:9622 nvda-graphrag:3.1.2

Why the GraphRAG CLI is not enough for a benchmark

At first glance the obvious command is graphrag query. It prints the answer and nothing else. However, a benchmark needs to know what the system retrieved, not only what it said, because recall computed from citations measures the model’s willingness to cite rather than the retriever’s reach.

The context comes back only through the Python API, as the second element of a tuple:

# graphrag.api returns (response, context_data); the CLI discards the second.
response, context = await graphrag.api.local_search(
    config=config, entities=entities, communities=communities,
    community_reports=reports, text_units=text_units,
    relationships=relationships, covariates=None,
    community_level=2, response_type="Multiple Paragraphs", query=question,
)

So the image ships serve.py, a thin HTTP layer over graphrag.api that returns the answer together with its context. That is lesson one from the previous episode, applied before the first measurement rather than after it.

Resolving retrieval back to documents

In addition, one more problem appears immediately. No element of GraphRAG’s retrieval names a file. Entities, relationships and community reports all carry text_unit_ids, and only text_units carries a document_id. Consequently the wrapper builds a map once, for four kinds of row, and resolves every context line through it.

The GraphRAG settings that matter

GraphRAG 3.x moved from a single models: block to completion_models: and embedding_models:, with LiteLLM underneath. A local llama-server therefore connects as an ordinary OpenAI-compatible endpoint.

completion_models:
  default_completion_model:
    type: litellm
    model_provider: openai
    model: Qwen3.8-27B-UD-Q4_K_XL.gguf
    api_base: http://llmhost:8084/v1
    api_key: local-llama-server   # the validator wants a non-empty value
    call_args:
      temperature: 0.0
      max_tokens: 2048

concurrent_requests: 1   # one model-server slot; the default of 25 queues inside it
input:
  type: text
  file_pattern: ".*.md$$"   # the reader defaults to *.txt only
vector_store:
  vector_size: 768           # nomic-embed, not OpenAI's 3072

concurrent_requests: 1 is the only performance setting that departs from the default, and it departs for a measurable reason. The generation server has a single slot. With the default of 25, the queue forms inside llama-server and GraphRAG measures its own waiting as indexing time.

What local, global and basic search actually read

The three search modes differ in one thing, and everything else follows from it: the unit they retrieve. Local search retrieves entities, global search retrieves community reports, and basic search retrieves text chunks. Consequently their costs are not variations of one number — they are different arithmetic.

	local	global	basic
Unit of retrieval	entity neighbourhood	community report	text chunk
Graph tables used	entities, relationships, text units, reports	communities, community reports	none
How content is chosen	vector similarity to entity descriptions	every report at the level	vector similarity to chunks
Model calls per question	1	~206 map + 1 reduce	1
Median latency	26.7 s	see the results table

In this run, local search pulled a median of 17.6 entities, 3.1 relationships, one community report and 4.1 source chunks per question, then made a single generation call over a 12,000-token context. Global search takes a different route entirely. With dynamic_community_selection=False it reads every community report at level 2 and above — 23 + 86 + 97 = 206 of this corpus’s 268 communities — scoring each batch for relevance in a map step before one final reduce call writes the answer.

Why global search costs twenty times more

That is why global search costs what it does. Local and basic scale with the question, because they take a fixed top-k. Global scales with the graph, because the question does not change how many reports have to be read. As a result, the hours spent building the community layer are not paid once at indexing time. They are paid again on every query.

Meanwhile, basic search is the control. It receives no graph table at all, which is exactly the point: same index, same model, same prompt, no graph. Without it there is no way to say how much of a result came from the graph and how much came from the corpus simply being well chunked.

# The three calls, as the benchmark wrapper makes them. Note what basic omits.
if method == "global":
    response, context = await api.global_search(
        community_level=2, dynamic_community_selection=False, **common)
elif method == "local":
    response, context = await api.local_search(
        text_units=..., relationships=..., community_level=2, **common)
elif method == "basic":
    # no entities, no relationships, no communities - vector search over chunks
    response, context = await api.basic_search(
        config=config, text_units=..., response_type=..., query=question)

What the GraphRAG benchmark build actually cost

The headline is 12.41 hours of model time. That number is useless on its own, because it hides the fact that the pipeline pays three separate bills, and each one counts something different.

Stage	Counts	Units	Per call	Model time
extract graph	text chunks	627	30.5 s	5.31 h
summarise descriptions	entities and relations	5,079	9.2 s	4.03 h
summarise communities	communities, 5 levels	268	41.4 s	3.07 h

Notably, the corpus fixes the first denominator. It does not fix the other two. Extraction produced those 5,079 entities and relations; summarisation produced those 268 communities. In practice that means an estimate built from page count prices roughly one third of the run.

One detail matters for anyone reading the middle column. Only 31% of the description steps actually called the model, because 652 of the 1,400 entities appear exactly once in the corpus and a single description has nothing to merge. A projection taken early in that stage therefore overestimates: the queue starts with the most-repeated entities and ends with singletons.

The most expensive filing is not the longest one

Meanwhile, every chunk gets the same 1,200-token budget, so a chunk of an annual report and a chunk of a proxy statement look identical going in. They do not cost the same coming out.

SEC form	Chunks	Median per chunk
DEF 14A (proxy statement)	115	38.0 s
8-K (current report)	29	26.0 s
10-K (annual report)	210	25.0 s
10-Q (quarterly report)	273	24.0 s

In essence, a proxy statement is a list of people, committees, roles and compensation figures. The extractor therefore pulls several times more entities from it than from a paragraph of narrative about data-centre demand. As a result, graph build time scales with entity density per page, not with page count — which is worth knowing if your own corpus holds registers, contracts or catalogues.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

GraphRAG’s progress counter never names a document, so this table needed a join to text_units.parquet by row order. That assumption is tested rather than assumed: document identity explains 0.148 of the variance in per-chunk cost, against a maximum of 0.113 across 200 shuffles of the same labels.

Resuming an interrupted GraphRAG build

That split matters when you size the machine. GraphRAG itself is a Python process doing bookkeeping between model calls, so a container with two gigabytes runs it. The 48 GB card is fully occupied for twelve hours by the model answering the extraction prompts, which is the resource you are actually renting.

As it happens, the build was interrupted once, by the model server dying six hours in. Nothing was lost. GraphRAG caches every model call by prompt hash under /data/cache, so a resumed run replays what it already paid for.

Overall, measured across the whole build: a replayed step costs 0.34 s against 18.0 s for a live call, across 4,761 replayed and 2,478 live steps. That is a factor of 53. On a multi-hour local pipeline, prompt caching is not an optimisation; it is what makes the run feasible at all.

GraphRAG benchmark results

All three search modes ran against the same finished index, one after another, on the same single-slot server. Local search took 18 minutes for the 40 questions and basic search took 20. Global search took 7.1 hours. Basic search is the control here: it gets no graph table at all, so whatever separates it from the two graph modes is what those 12.41 hours of building bought.

Measure	local	global	basic (no graph)
Questions asked	40	40	40
Strict document recall questions where every evidence document was retrieved	100.0%	100.0%	60.0%
Answers judged correct	20.0%	15.0%	17.5%
Answers judged correct, of 40	8	6	7
Answers partly correct	21	8	19
Answers plainly wrong	2	3	3
Questions refused	9	23	11
Figures exactly right of the numeric questions the system attempted	52.0% (13 of 25)	47.1% (8 of 17)	29.0% (9 of 31)
Answer-or-refuse decisions right	62.5%	55.0%	82.5%
Answers grounded in retrieval	19 of 40	6 of 40	25 of 40
Answers not grounded	1	4	4
Answers containing an invented number	0	0	0
Median latency	26.7 s	636.2 s	27.8 s
Hours to build the index one build, shared by every mode of this framework	13.08 h	13.08 h	13.08 h
Scorecard total, frozen rubric	65.3	57.9	46.3

One sentence carries the whole table. The graph bought exactly one correct answer in forty — eight for local search against seven for the control that ignores it. Those 12.41 hours of model time are the price of that one answer.

Local search: every document retrieved, one answer in five

Local search walks the entity neighbourhood around the question. On these 40 questions it reached 100% strict document recall: for every single question, every filing the golden set names as evidence was in the retrieved context. The answers tell a different story.

However, the verdict breakdown explains the gap better than the headline does. The judge marked 8 answers correct, 21 partial, 9 abstentions and only 2 wrong. In other words, the system is not inventing things. It is answering half of each question.

Global search: twenty-four times the cost, fewer right answers

Global search is the mode the community layer exists for. It reads 206 community reports per question, scores them in a map step and writes one answer in a reduce call. On this corpus it lost to local search on every accuracy measure, and it cost 24 times more time per question.

The behaviour behind that number is more interesting than the number. Global search refused 23 of the 40 questions, against 9 refusals from local search. It reads summaries rather than text, so it finds a specific figure less often and says so. That caution is badly aimed, though: only 55.0% of its answer-or-refuse decisions were right, which is worse than local search manages.

One measurement limit belongs with these numbers. The blind judge reads each answer next to that answer’s own retrieved context, capped at 160,000 characters. For 17 of the 40 global-search questions the context hit that cap, because 206 community reports do not fit. Correctness and numeric agreement do not depend on it, since those compare the answer with the golden set. The grounding score for global search rests on a truncated view of its evidence, and it therefore reads as a lower bound.

Basic search: the control that beats the graph

Basic search receives the same index without any graph table. It retrieved the evidence for only 60% of questions — a 40-point gap against both graph modes — and still answered 17.5% correctly, against local search’s 20.0%.

The third row of that comparison is more striking still. Basic search has the best answer-or-refuse accuracy in the episode, at 82.5%. The system that sees the least is the one that most often knows it does not know. Meanwhile, both graph modes see everything and answer anyway.

Where the community graph actually pays

The gain does not spread evenly across the benchmark. Broken out by dispersion pattern, correctness looks like this. Each pattern holds five questions, so one step is one answer.

Pattern	local	global	basic
D1 table ↔ narrative	20%	20%	20%
D2 footnote dependency	0%	0%	40%
D3 definition → use	40%	40%	40%
D4 granularity mismatch	0%	0%	0%
D5 event → consequence	0%	0%	0%
D6 governance vs operations	20%	60%	0%
D7 retrospective restatement	20%	0%	0%
D8 absence (refusal is correct)	60%	0%*	40%

The graph wins in two places only. Global search takes D6, governance against operations, at 60% against the control’s zero. Local search takes D7, retrospective restatement, at 20% against zero. Both are patterns whose answer is descriptive and spread across filings. Where the answer is a figure in a footnote, the graph adds nothing — and on D2 it actively hurts, since plain vector search scores 40% there and both graph modes score zero.

Retrieval, by contrast, no longer separates anything. Both graph modes hit 100% on all eight patterns, so the chart below stays flat where the interesting variation used to be.

Why it found everything and still answered wrong

In short, two mechanisms produce that result, and both are visible in the data rather than inferred from it.

One company, two nodes

First, the two highest-degree nodes in the entire graph are NVIDIA and NVIDIA CORPORATION. They are the same company. Together they have 832 distinct neighbours, of which only 153 are shared — a Jaccard overlap of 0.184.

This is not a bug in one implementation. Asking a language model to name entities chunk by chunk produces whatever surface form each chunk used, and nothing in the pipeline afterwards decides that two of them are one company. An LLM-built graph with no separate entity-resolution step therefore splits the corpus’s main subject — and in filings, that subject is what almost every question is about.

Too much evidence is its own failure mode

With 100% recall, local search puts a median of 32 of the corpus’s 33 documents into context. Document precision is therefore 5.1%, and the right passage is a needle in a stack of the system’s own sources.

For example, one question shows the consequence exactly. It asks for total assets at 25 January 2026 and for inventory provisions recorded in cost of revenue for fiscal 2026. The gold answer is $206,803 million and $4.0 billion. GraphRAG got the assets right and reported $2.3 billion in provisions. That figure was genuinely in the retrieved context — in this sentence:

“Our maximum loss exposure under these investments, including invested and future committed amounts, was $2.3 billion as of April 26, 2026.”

It describes a different concept, from the following quarter. Only the shape matched: billions of dollars, near a balance-sheet topic. Therefore the failure is not retrieval and not hallucination. The evidence was there, alongside a great deal of plausible-looking company.

That is worth stating plainly, because it is the opposite of the usual advice. GraphRAG answers badly here not for want of evidence, but because it hands the model too much to choose between, with nothing in the context marking which quarter or which concept each figure belongs to.

GraphRAG benchmark pitfalls and limits

In total, the episode logged 30 integration findings before any number was worth publishing. These five are the ones most likely to cost you a day.

• A single $ in settings.yaml aborts the start, including inside a comment: ValueError: Invalid placeholder in string: line 68, col 25. GraphRAG runs the whole file through string.Template before parsing it. Write $$.
• The text reader takes only *.txt. A corpus of .md files produces an empty index, zero documents and zero errors.
• The default vector size is 3072. A local embedder returning 768 fails with Vector for document '...' has dimension 768, but index 'entity_description' is configured with vector_size 3072 — and it fails after entity extraction, the most expensive step.
• A client-side max_tokens is a quality parameter, not a cost guard. At 2048 it collides with GraphRAG’s own 2000-token report body: one community report of 268 failed to parse and that community has no report at all.
• Numpy arrays break the usual list(value or []) idiom. The id columns in the parquet files are arrays, so that idiom raises instead of returning a default — and a system whose document maps are silently empty still answers questions, while reporting 0% recall.

Two more traps belong to the hardware rather than to GraphRAG, and a twelve-hour local run meets both. First, the AMD driver watchdog reset the card at 15:01 and took the model server with it — seven LiveKernelEvent a1000001 entries in the Windows event log, and llama-server.exe dying as BEX64 in ucrtbase.dll. The card logged 42 such resets in six hours; only one of them killed the process. Second, the embedding server died unnoticed at some point after 15:05, because global search does not use embeddings and nothing asked for it for seven hours.

Neither had a cause until the model server started writing a log file. One line in the start script — --log-file instead of --log-disable — turned the second crash from a mystery into a diagnosis. On a pipeline this long, that is the cheapest insurance available.

GraphRAG benchmark FAQ

Does GraphRAG work with a local model instead of OpenAI?

Yes. GraphRAG 3.x routes model calls through LiteLLM, so a local llama-server connects as an ordinary OpenAI-compatible endpoint with model_provider: openai and an api_base pointing at your host. The API key must be non-empty, because the config validator checks it, but the server ignores its value.

How long does GraphRAG take to index 33 filings on one GPU?

12.41 hours of model time on an AMD W7900 with a 27B model at concurrent_requests: 1: 5.31 h of entity extraction, 4.03 h of description summarisation and 3.07 h of community reports.

Can you resume a GraphRAG index after a crash?

Yes, and it is cheap. Every model call is cached by prompt hash in the project’s cache/ directory, so re-running the pipeline replays finished work at 0.34 s per step instead of 18.0 s. There are no per-document states to repair — the pipeline simply runs again and hits the cache.

What is the difference between local, global and basic search?

They differ in what they retrieve, and the numbers make the difference concrete. Local search pulled a median of 17.6 entities per question and called the model once. Global search reads 206 community reports per question — every report at level 2 and above — through a map step and one reduce call. Basic search receives no graph table at all and is therefore the control: the same index without the graph.

Why does GraphRAG retrieve almost every document in the corpus?

Because its unit of retrieval is an entity, not a chunk. An entity description is built from text units across many filings, so resolving retrieved entities back to documents touches nearly all of them. Document-level recall consequently stops discriminating between systems, and precision becomes the number to watch.

How slow is GraphRAG global search?

On this corpus, a median of 636.2 seconds per question — 24 times local search, and 7.1 hours for all 40 questions. The cause is structural, not a tuning error. With dynamic_community_selection=False, global search reads every community report at level 2 and above, which here means 206 reports scored in a map step before a single reduce call. Its cost therefore scales with the size of the graph, not with the question, so a bigger corpus makes every query slower.

Do GraphRAG community reports pay off?

On a corpus of financial filings, measured here: no. The community layer cost 3.07 hours to build and 636 seconds per query, and global search answered 15.0% of questions correctly against 17.5% for basic search, which ignores the graph entirely. It won one pattern — governance against operations, at 60% against zero — where the answer is descriptive and spread across filings. Where the answer is a figure from a table or a footnote, the summaries have already dropped it.

Summary: what the GraphRAG benchmark measured

This GraphRAG benchmark cost 12.41 hours of model time and produced a graph of 1,400 entities, 3,616 relationships and 268 communities across five levels. Three search modes then ran against it. Local search retrieved the evidence for all 40 questions and answered 20% correctly. Global search, the mode the community layer exists for, answered 15% and took 24 times longer. Basic search, which ignores the graph, answered 17.5% while retrieving 40 points less.

The series scorecard puts those three at 65.3, 57.9 and 46.3 points on a rubric frozen before this framework ran. Ranking frameworks against each other is the final episode’s job; this is GraphRAG measured against fixed anchors.

How to read a GraphRAG benchmark score of 65.3

That number is not the share of questions answered correctly. It is a weighted sum of eight dimensions, and on this rubric retrieval is worth 40 points while answer correctness is worth 20. A system that puts the evidence in front of the model and fails to use it therefore scores above one that never found the evidence — because the first problem is a prompt away, and the second needs the index rebuilt.

Where the 65.3 points came from

Read row by row, GraphRAG local search means this on the 40 questions:

Dimension	What it means on the 40 questions	Points	Of
Cross-document retrieval (D4-D8)	all evidence retrieved for 25 of the 25 multi-document questions	30.0	30
In-document retrieval (D1-D3)	all evidence retrieved for 15 of 15 single-document questions	10.0	10
Answer correctness (blind judge)	8 of 40 answers fully correct	4.0	20
Numeric agreement	13 of 25 numeric questions hit the figure within 2%	5.2	10
Knowing when not to answer	25 of 40 answer-or-refuse decisions right	6.2	10
Grounding in its own retrieval	share of answers standing on evidence the system itself retrieved	5.5	10
Cost to index the corpus	13.08 h once; the anchors are 24 h for zero and 0 h for ten	2.3	5
Cost to answer one question	median 26.71 s; the anchors are 300 s for zero and 1 s for ten	2.1	5
Total		65.3	100

So 40 of those 65.3 points came from finding the right documents, and 4 of a possible 20 from answering — eight questions out of forty. Read that way, a high total is not a recommendation. It is a statement that the evidence problem is solved and the answering problem is not.

Finally, the lesson is not that GraphRAG is worse. It is that retrieval stopped being the bottleneck some time ago. A system that finds every relevant document and still answers one question in five is telling you where the remaining work sits — in entity resolution, in period disambiguation, and in giving the model less rather than more.

Everything behind these numbers

The full result set ships with this article rather than as a summary you have to trust. Inside the bundle below, appendix-40-answers.html lists all 40 questions with the gold answer, what each of the three search modes replied, the judge’s verdict and its reasoning, whether the figures matched and how long each answer took. It opens in a browser; nothing has to run first.

The download bundle carries the Containerfile, the HTTP wrapper, the adapter, the shared benchmark runner, the evaluator, the chart scripts and every raw result file, plus a SHA-256 manifest so you can confirm the inputs are the ones measured here. Its folder layout matches the commands above, so they run verbatim after unzipping: nvda-rag-e04-graphrag.zip.

Not investment advice. NVIDIA filings are test material for a retrieval system, nothing more.

Companion code — the fragments above are the working parts: the --add-host line that lets a container find the Windows host, the serve.py call that returns retrieval alongside the answer, and the three settings that decide whether an index finishes at all. The commands in this article reproduce the result from scratch.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.