LightRAG benchmark on local hardware: 33 SEC filings, one knowledge graph

Part of RAG & Vector Databases — retrieval-augmented generation and vector search.

This LightRAG benchmark answers one question: does a knowledge graph actually help when the answer is scattered across several filings? I built the graph from 33 NVIDIA SEC filings on a local 27B model, then ran the same 40 questions twice — once through the graph, once through plain vector search over the same index. The graph won on exactly two of eight question types. On those two it was also still wrong, for a reason that turns out to be the same mechanism that made it win.

What a knowledge graph is supposed to buy you

Plain vector RAG splits documents into chunks and retrieves the chunks that look similar to your question. In practice that works while the answer sits in one place. However, it struggles when the answer needs a number from a 10-K, a footnote from a 10-Q and an event from an 8-K.

By contrast, LightRAG (from HKUDS) takes a different route. During indexing it asks a model to name the entities in every passage — the people, companies, products and figures it mentions — along with the relationships between them, then stores both as a graph. Then, at query time, it walks that graph as well as the vectors. In theory the graph carries the links a chunk index cannot: this segment belongs to that fiscal year, this charge follows that announcement.

Theory is cheap, so I measured it instead. Specifically, the corpus is 33 NVIDIA filings from SEC EDGAR, built and analysed in the corpus episode — 10-K, 10-Q, 8-K and DEF 14A, covering fiscal years 2024 to 2026. The benchmark is 40 questions, tagged by dispersion pattern — our label for how the knowledge is scattered, five questions per pattern. The golden-set episode covers how a model proposed those questions and which gates they had to pass. Patterns D1 to D3 hide the answer inside one document. Patterns D4 to D8 spread it across several, or leave it out entirely to see whether the system admits it.

Importantly, the model never competes with the retrieval systems here. It extracts entities during indexing and grades answers afterwards. It never answers benchmark questions from raw context.

The LightRAG benchmark setup, command by command

Everything runs locally. Specifically, generation and embeddings live on two llama.cpp servers, and LightRAG runs in a podman container that talks to both.

# 1. Generation: Qwen3.8-27B, 204,800-token context, one slot
llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf --host 0.0.0.0 --port 8082 `
             -ngl 99 -fa on --ctx-size 204800 -np 1

# 2. Embeddings: a second, much smaller server
llama-server -m nomic-embed-text-v1.5.Q4_K_M.gguf --embedding `
             --host 0.0.0.0 --port 8083 -ngl 99 --ctx-size 8192

Next comes the container. Above all, one detail decides whether it works at all: inside a podman machine, host.docker.internal resolves to the WSL bridge, not to Windows. Therefore the real host address has to be read from the machine and injected by name.

$hostIp = (podman machine ssh "ip route | awk '/default/ {print `$3}'").Trim()

podman run -d --name nvda-lightrag `
    --add-host "llmhost:$hostIp" `
    --env-file lightrag.env `
    -v nvda-lightrag-data:/app/data `
    -p 9621:9621 `
    ghcr.io/hkuds/lightrag:latest

curl -s http://127.0.0.1:9621/health

Finally, the environment file carries the wiring, and two values in it matter more than the rest:

LLM_BINDING=openai
LLM_BINDING_HOST=http://llmhost:8084/v1
EMBEDDING_BINDING_HOST=http://llmhost:8083/v1
EMBEDDING_DIM=768
TEMPERATURE=0.0

MAX_ASYNC=1              # the generation server runs one slot
LLM_TIMEOUT=900          # NOT the same variable as TIMEOUT
CHUNK_SIZE=1200
CHUNK_OVERLAP_SIZE=100

What the embedding model does here

Two models run in this setup, and they do different jobs. The 27B model extracts entities during indexing and writes the answers. The embedding model turns text into vectors so that similar passages can be found by distance rather than by keyword, and that is a smaller, much faster job.

I used nomic-embed-text-v1.5, quantised to Q4_K_M, on its own llama.cpp server with 768 dimensions and an 8,192-token context. It matters more than its size suggests: in naive mode the embedding model is the whole retrieval, because that mode ignores the graph and ranks chunks by vector distance alone.

Crucially, the same embedding model served every system compared in this article — the graph mode, the vector mode and the independent baseline from the previous episode. Therefore the differences you see later come from retrieval strategy, not from a better or worse embedder.

LightRAG stores the result in plain files rather than an external database: a NetworkX graph, a NanoVectorDB index for the vectors, and JSON for key-value state. Consequently the whole index is a podman volume you can copy, and there is no database to run alongside it.

Loading 33 SEC filings into the graph

Loading is a submit-then-poll job. First, each document goes in through /documents/text, and the server indexes it in the background while you watch the status counters. Indexing starts by cutting every filing into chunks — passages of roughly 1,200 tokens, which is the unit everything downstream is priced in. The 33 filings became 629 of them.

curl -s http://127.0.0.1:9621/documents/status_counts
podman logs --timestamps --tail 20 nvda-lightrag | grep "Chunk"

What the build actually cost

Loading all 33 filings into the graph took 10.97 hours on this machine. That is the time the pipeline spent working: 629 chunk extractions and 627 description merges, ending in a graph of 3,237 entities with no failed documents.

The cost is wildly uneven, and that is a property of the corpus rather than a quirk of the run. Eleven quarterly and annual reports carry 30 to 70 chunks each and take half an hour or more. Meanwhile twenty of the filings are 8-Ks — one to three chunks apiece — and all twenty together cost 22.5 minutes, or 3.4% of the build.

Why the cost per chunk keeps rising

In fact, two operations call the model during indexing, not one. Extraction runs once per chunk, and its cost stays flat throughout. Merging, by contrast, runs whenever an entity already in the graph appears again, and it grows more common with every document.

Chunk 12 of 37 extracted 18 Ent + 17 Rel     entity extraction, flat cost per chunk
LLMmrg: `U.S. Government` | 7+2              description merge, rising cost

As a result, seconds per chunk climbed from 49.1 s on the first large filing to 69.8 s on the twenty-ninth, a 42% increase. Merging ended up as 17.7% of all model time. In practice, this means an estimate built from your first document will understate the real build by tens of percent.

Resuming an interrupted build

Sooner or later, an eleven-hour build will be interrupted anyway. However, a killed run leaves documents in PARSING, ANALYZING or PROCESSING, and nothing restarts them by itself. Re-sending the same text does not help either, because LightRAG answers a repeat as a duplicate. Consequently a naive “submit everything again” resume indexes nothing and reports success.

Fortunately one call fixes it, and it is the call nobody mentions:

curl -X POST http://127.0.0.1:9621/documents/reprocess_failed

Consequently it re-queues pending and interrupted records without touching what is already done. When I resumed after a night, only four chunks were re-extracted, and all four came back from the LLM cache in about two seconds.

How every answer was scored

A benchmark is only as trustworthy as its marking, so here is the whole marking scheme. Scoring runs in two layers: arithmetic that cannot be argued with, and a model acting as a marker for everything else.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

The layers that need no model

• Strict document recall — did the system pull back every document the golden set names as evidence? Two documents needed, one retrieved, and the question counts as a miss. This is set arithmetic, not judgement.
• Numeric agreement — do the figures in the gold answer appear in the system’s answer? A tolerance handles rounding and scale, so $1.5 billion and $1,500 million match.
• Invented figures — does the answer state a number that appears nowhere in the context the system itself retrieved? Arithmetic the system performed on two retrieved numbers is allowed; a figure from nowhere is not.
• Abstention — the golden set marks which questions the corpus genuinely cannot answer. Silence on one of those counts as correct, and silence anywhere else counts as a miss.

The marker is the same local model, and it never sees a name

Everything else is graded by Qwen3.8-27B on the same llama.cpp server that answered the questions, at temperature 0. That is what “blind judge” means in this article, and it is worth being precise about: the marker is never told which framework produced the answer. It grades an anonymous “System A” every time, so it cannot favour a name it recognises. It also never sees the retrieval, the mode or the episode — only the question, the reference answer and the text under test.

This is the exact prompt, verbatim from benchmark/evaluate/evaluate.py. Every framework in the series is graded with these words, so the comparison measures retrieval rather than prompt wording.

You are grading one answer produced by an automated document
retrieval system, called System A. Grade only what is written.

QUESTION
{question}

REFERENCE ANSWER (correct by construction)
{gold}

SYSTEM A ANSWER
{answer}

Rules:
- CORRECT: states the same fact as the reference. The same number written at a
  different scale or with different rounding is still correct ($1.5 billion =
  $1,500 million). Extra unrequested detail does not make it wrong.
- PARTIAL: part of the reference fact is right and part is missing or wrong.
- WRONG: contradicts the reference, or answers a different question.
- ABSTAIN: System A says it cannot answer or has no information.

Reply with JSON only: {"verdict": "CORRECT|PARTIAL|WRONG|ABSTAIN", "reason": "one short sentence"}

The reference answer comes from the golden set built in the previous episode, where every figure was traced back to the filing it came from before the questions were frozen. Consequently the marker compares two fixed texts and returns one of four verdicts plus a one-sentence reason, which is why the appendix can show you its reasoning for all 40 questions.

A second pass checks grounding, not truth

Being right and being grounded are different properties, so a second prompt asks a separate question: is this answer actually supported by the context the system retrieved? That check ignores whether the answer is true in the world. An answer can be wrong and grounded, which is exactly what happened on the employee-count question later in this article.

Decide whether an answer is supported by the context it was
given. Judge support only, never whether the answer is true in the world.

CONTEXT
{context}

ANSWER
{answer}

Rules:
- SUPPORTED: every factual claim in the answer can be traced to the context.
- PARTIAL: some claims are traceable, at least one is not.
- UNSUPPORTED: the central claim appears nowhere in the context.

Reply with JSON only: {"verdict": "SUPPORTED|PARTIAL|UNSUPPORTED", "reason": "one short sentence"}

One setting matters here more than it looks. The marker sees up to 160,000 characters of context, which is about 40,000 tokens against the server’s 204,800-token window. That ceiling has to be identical for every framework, because a system that hands over more evidence would otherwise be judged on a smaller share of it. In this run nothing was truncated for either mode.

LightRAG benchmark results: the graph finds everything, then answers wrong

In short, the headline pair of numbers is the whole story. LightRAG retrieved every required document for every one of the 40 questions, yet the marker called only 45% of the answers fully correct. The first number is strict: a question counts only when all of its evidence documents came back, not one of two.

Measure	LightRAG hybrid	What it means
Strict document recall	100.0%	every evidence document reached, for all 40 questions
Answers judged correct	45.0%	18 correct, 13 partial, 3 wrong, 6 abstentions
Figures exactly right	58.8%	of 34 numeric questions
Answer-or-refuse decisions right	77.5%	it refused 4 times
Answers grounded in retrieval	31 of 40	traceable to the text the system itself pulled back
Invented figures	0	no number appeared that was absent from its context
Median answer latency	79.2 s	IQR 4.0 s, one slot, 27B model

Above all, zero invented figures deserves a note. For comparison, the naive baseline from the previous episode cited “24,930 million shares”, a value present in no filing. LightRAG never did that. In other words, it stays inside its evidence and simply picks the wrong item from it.

Does the graph beat plain vector search?

LightRAG ships a mode that turns the graph off. In naive mode it runs plain vector search over the same chunks, with the same model and the same prompt, so nothing changes except the retrieval strategy. That makes it the only controlled comparison available, and I ran it.

Fortunately the control is clean. In naive mode LightRAG pulled 0 entities, 0 relationships and exactly 20 chunks per question. In hybrid mode it pulled a median of 31 entities and 137 relationships alongside just 12 chunks. Consequently the graph retrieves less text and more structure.

Dispersion pattern	Graph (hybrid)	Vector only (naive)	Gain
D1 table vs narrative	100%	100%	—
D2 footnote dependency	100%	100%	—
D3 definition to use	100%	100%	—
D4 granularity mismatch	100%	100%	—
D5 event to consequence	100%	0%	+100 pts
D6 governance vs operations	100%	100%	—
D7 retroactive restatement	100%	60%	+40 pts
D8 absence	100%	100%	—
All 40 questions	100.0%	82.5%	+17.5 pts

Six patterns out of eight show no difference at all. After all, vector search already finds a footnote, a definition or a table inside one filing. The graph earns its entire advantage on two patterns: an 8-K announcing something and a 10-Q recording its effect, and a figure restated after a stock split.

What those 17.5 points bought in answers

Retrieval is not the product, though. Specifically, here is what the extra evidence turned into once the model wrote an answer.

Measure	Graph (hybrid)	Vector only (naive)	Difference
Documents retrieved	100.0%	82.5%	+17.5 pts
Answers judged correct	45.0%	42.5%	+2.5 pts
Figures exactly right	58.8%	52.9%	+5.9 pts
Answers grounded in retrieval	31 of 40	23 of 40	+8
Answers not grounded	3	7	−4
Answers plainly wrong	3	5	−2
Answer-or-refuse decisions right	77.5%	85.0%	−7.5 pts
Median latency	79.2 s	66.0 s	+13.2 s

Above all, read the second row carefully. Seventeen and a half points of extra retrieval became one more correct answer out of forty, at a price of 10.97 hours of indexing and 13 extra seconds per query. That is the honest exchange rate for a knowledge graph on this corpus.

Two rows point the other way, however. The graph grounded far more answers in real evidence — 31 supported against 23 — and produced fewer plainly wrong ones. Therefore it buys citation quality more than it buys correctness.

Why the graph wins on those two patterns

The D5 failures were not scattered. All five vector-mode misses were the same missing file: the 8-K of 15 April 2025 that announced the H20 export restrictions.

The reason is a budget, not a ranking accident. Naive mode retrieves a fixed 20 chunks, and for one of those questions they came from eight documents: five chunks from the 2026 10-K, four from a 10-Q, three from another, and so on. Every one of them discusses H20, because long filings discuss it at length. Meanwhile the 8-K that announced it holds 2 chunks out of 629 in the whole corpus, so it never makes the cut.

In other words, vector search with a fixed chunk budget is biased towards verbose documents. An event is short; the discussion of that event is long; therefore the discussion crowds out the event. A corpus with twenty 8-Ks next to eleven quarterly and annual reports walks into this systematically.

The graph sidesteps it entirely. Entities extracted from that 8-K sit in the graph whatever their chunks rank, so a relationship hop reaches a document vector search would never surface. That is the concrete answer to “what does a graph buy me”, and it applies to exactly two of the eight patterns.

Four answers worth reading in full

Naturally, aggregate scores hide the interesting part. Therefore here are four questions that each explain something the totals cannot.

The clean win: governance across two documents

For instance, question D6-01 asks who NVIDIA’s Lead Director is and what net income was for fiscal 2026. Naturally, those facts live in different places. LightRAG answered “Stephen C. Neal” and “$120,067 million”, both correct, both cited. All five governance questions came back correct, which is the pattern where the graph clearly earns its build time.

The near miss: an event and its consequence

Question D5-02 asks for the maximum estimated H20 charge announced in April 2025, plus H20 revenue before the new rules. The gold answer is “up to $5.5 billion” and “$4.6 billion”. In this case LightRAG returned the revenue correctly, then gave the charge as $4.5 billion — the amount actually recorded in the later 10-Q, not the estimate announced in the 8-K. The announcement and its consequence collapsed into one fact. Every D5 question came back partial for a variant of this reason.

The instructive failure: a stock split

Question D7-05 asks for the dividend per share as first disclosed and after restatement: $0.16, then $0.016 following the 10-for-1 split. LightRAG answered $0.004 on both sides — the post-split quarterly rate, used for both halves of a before-and-after question. Notably, it did not hallucinate the number. Rather, it had no way to tell which fiscal period the number belonged to.

Wrong, yet correctly grounded

Similarly, question D8-02 asks how many employees NVIDIA had in the United States. The corpus never says. Nevertheless, LightRAG replied “approximately 36,000 employees globally”, and the grounding check marked that answer supported — supported meaning traceable to the retrieved text, not true. The figure 36,000 really is in the context. It comes from an entity description whose sources start with a 10-Q filed in 2023.

Why the graph found everything and still answered wrong

As it turns out, those three failures share one cause, and it is measurable. LightRAG merges an entity’s description every time it meets that entity again. In short, merging is what keeps the graph small and queries cheap. Unfortunately it is also what destroys the time dimension.

Indeed, of the 1,033 entity descriptions retrieved across the benchmark, 547 (53.0%) fuse filings from more than one fiscal year into a single paragraph carrying no date. The widest merge spans 28 filings. Here is the entity for the company itself:

[entity | 2023-11-21_10-Q <SEP> 2024-02-21_10-K <SEP> 2024-05-29_10-Q
        <SEP> ... <SEP> 2026-02-25_10-K <SEP> 2026-05-20_10-Q]
NVIDIA (organization): NVIDIA Corporation, commonly known as NVIDIA ...

Consequently, ask that description for a fiscal-2026 headcount and it will hand back a number from some year. As a result, grounding stays high, because the number is genuinely there. Meanwhile correctness collapses, because nothing marks which year it belongs to.

The same entity, four times over

In addition, the graph has a second weakness that you can see without any measurement at all.

In total, across all 3,237 labels, 301 (9.3%) are a second spelling of an entity already present, differing only in case or spacing. That figure understates the problem, since it cannot catch splits by abbreviation. The panel’s own node search shows the effect plainly.

LightRAG benchmark pitfalls and limits

Meanwhile, five things cost me real time, and none of them produced a useful error message.

• host.docker.internal does not point at Windows. Inside a podman machine it resolves to the WSL bridge at 10.88.0.1. Read the default gateway instead.
• A self-signed certificate blocks every container. My llama-server serves HTTPS under a name no container uses, so a small plain-HTTP relay on localhost solved it once for every framework.
• LLM_TIMEOUT defaults too low for local models. A hosted API answers in seconds; a 27B model on one GPU does not.
• An interrupted ingest fails silently. It neither resumes nor complains, and a naive retry reports success while indexing nothing.
• Three answers in forty were citations with no answer. After stripping the References block, nothing remained. The judge scored two as abstentions and one as wrong, so the defect quietly lowered the score.

Finally, one more limit belongs here. Answer latency is a median 79.2 seconds per question on a single-slot 27B model. That is fine for a benchmark and slow for a product.

When LightRAG is the right choice

• Document sets full of named things that stay stable — contracts, product catalogues, technical documentation, org charts.
• Questions about relationships — who reports to whom, which component belongs to which system, what connects two parties.
• Small hardware — the container held a median 551 MiB of RAM, and the graph is a set of JSON files on a volume.
• Not for documents that restate the same figures over time — quarterly reports, versioned specifications, anything where the same name changes value from one period to the next.

Frequently asked questions

Does LightRAG work with a local model instead of OpenAI?

Yes. It speaks the OpenAI wire format, so any local server that implements it works. I pointed it at two llama.cpp servers, one for generation and one for embeddings, and changed nothing else.

How long did the LightRAG benchmark take to index 33 filings?

In this run, 2.97 million characters became 629 chunks in 10.97 hours on a W7900 with a 27B model. That is roughly 63 seconds per chunk on average, and the rate gets slower as the graph grows.

Why does LightRAG get slower the more documents you add?

Because entity merging is a model call, and it fires more often as entities repeat. Extraction cost per chunk stayed flat in my run, while total cost per chunk rose 42%.

Can you resume a LightRAG ingest after stopping the container?

Yes, but not by re-sending the documents. Call POST /documents/reprocess_failed, which re-queues records left mid-pipeline. The index lives in the volume and survives container restarts.

Which embedding model does LightRAG use?

Whichever one you point it at. I used nomic-embed-text-v1.5 in Q4_K_M, served by a second local llama.cpp instance at 768 dimensions. Set EMBEDDING_BINDING_HOST and EMBEDDING_DIM to match your own server, and note that changing the model later means rebuilding the index.

Is graph RAG better than vector RAG?

Not automatically. I ran both modes over the same index: the graph reached the evidence for 100.0% of these cross-document questions against 82.5% for plain vector search. However, correct answers moved only from 42.5% to 45.0%. In other words, the graph found much more and answered barely better.

What does the “Worker execution timeout after 480s” error mean?

It means LLM_TIMEOUT is still at its 240-second default and the worker timeout is twice that. Raise LLM_TIMEOUT, not TIMEOUT.

Summary

This LightRAG benchmark separated two things people usually treat as one. Retrieval worked: 100.0% strict document recall against 82.5% for plain vector search over the same index, with zero invented figures and 31 of 40 answers grounded in evidence. Answering barely moved, however: 45.0% fully correct against 42.5%. Consequently the graph bought one extra correct answer out of forty.

Fortunately the cause is specific rather than general. LightRAG merges entity descriptions across documents, which is what makes the graph affordable, and 53% of the descriptions it retrieved fused several fiscal years with no date attached. For a corpus of periodic reports, that is the wrong trade. For a corpus of stable entities and relationships, it is probably the right one.

Meanwhile, building that graph cost 10.97 hours of GPU time and 3,237 entities, and the cost per chunk rose 42% along the way. Therefore budget for the build, and measure the per-pattern profile rather than a single accuracy number. In this run the profile said something a single score would have hidden: the graph mattered on two question types out of eight, and nowhere else.

How to read a LightRAG benchmark score

The series scores every framework on one rubric, frozen before any of them ran, so an episode can be published the day it is measured. That total is not the share of questions answered correctly. It is a weighted sum of eight dimensions, and on this rubric retrieval is worth 40 points while answer correctness is worth 20. A system that puts the evidence in front of the model and fails to use it therefore scores above one that never found the evidence — because the first problem is a prompt away, and the second needs the index rebuilt.

Row by row, on the 40 questions:

Dimension	What it means on the 40 questions	Points	Of
Cross-document retrieval (D4-D8)	all evidence retrieved for 25 of the 25 multi-document questions	30.0	30
In-document retrieval (D1-D3)	all evidence retrieved for 15 of 15 single-document questions	10.0	10
Answer correctness (blind judge)	18 of 40 answers fully correct	9.0	20
Numeric agreement	20 of 34 numeric questions hit the figure within 2%	5.9	10
Knowing when not to answer	31 of 40 answer-or-refuse decisions right	7.8	10
Grounding in its own retrieval	share of answers standing on evidence the system itself retrieved	7.9	10
Cost to index the corpus	10.97 h once; the anchors are 24 h for zero and 0 h for ten	2.7	5
Cost to answer one question	median 79.18 s; the anchors are 300 s for zero and 1 s for ten	1.2	5
Total		74.4	100

Read that way, a high total is not a recommendation. It is a statement about which half of the problem is solved.

Everything behind these numbers

The full result set ships with this article rather than as a summary you have to trust. Inside the bundle below, appendix-40-answers.html lists all 40 questions with the gold answer, what each retrieval mode replied, the judge’s verdict and its reasoning, whether the figures matched and how long each answer took. It opens in a browser; nothing has to run first.

The download bundle (3.6 MB) carries the adapter, the shared benchmark runner, the evaluator, the chart scripts and every raw result file, plus a SHA-256 manifest so you can confirm the inputs are the ones measured here. Its folder layout matches the commands above, so they run verbatim after unzipping: nvda-rag-e03-lightrag.zip.

Not investment advice. NVIDIA filings are test material for a retrieval system, nothing more.

Companion code — the fragments above are the working parts: the --add-host line that lets a container find the Windows host, the two timeout settings that decide whether extraction finishes, and the one API call that resumes an interrupted build. The commands in this article reproduce the result from scratch.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.