RAG framework ranking on SEC filings: seven frameworks, 28 modes, one frozen scorecard

Part of RAG & Vector Databases — retrieval-augmented generation and vector search.

Why a RAG framework ranking, and why only now

This RAG framework ranking is the tenth and last article of a series that measured seven retrieval frameworks on the same corpus, the same questions, the same local model and the same blind judge, one framework at a time. Every episode refused to rank its framework against the others. Consequently, this article is where that refusal pays off: the numbers below were produced under one frozen protocol over several weeks, and none of them was known when the scoring rubric was written.

The ranking is the least interesting result. The pattern behind it is not. In particular, the three best modes of the series are plain retrievers that show the model more, smaller pieces of the corpus. The two frameworks that build a knowledge graph earned their hours on exactly two of the eight ways knowledge scatters across financial filings.

What this RAG framework ranking measures, and what it does not

Every episode in this series measured one framework against the same forty questions and refused to rank it against the others. That was deliberate: a number produced on Tuesday is not comparable with a number produced three weeks later unless the corpus, the questions, the judge and the generation server were frozen in between, and the whole point of freezing them was to make this article possible. Hence the ranking lives here and nowhere else.

Specifically, the benchmark is 33 NVIDIA filings from SEC EDGAR: three annual reports, eight quarterly reports, two proxy statements and twenty current reports, 749,859 tokens in all, cleaned from Inline XBRL into Markdown with the tables kept. On top of it, in turn, sits a golden set of 40 questions that the model proposed and ten machine-checked rules approved, five for each of eight ways knowledge scatters across such documents. In addition, half the questions need two filings to answer; five have no answer in the corpus at all, and the correct response to those is a refusal.

The eight dispersion patterns

The eight patterns are the axis every result below is reported on:

What was held constant

Three things were held constant for every framework. First, the generation model is Qwen3.8-27B (Q4_K_XL) on llama.cpp over Vulkan, on one AMD Radeon PRO W7900, one request at a time. Second, the embedder is nomic-embed-text-v1.5 on a second llama-server. Third, the answer instruction is one paragraph, byte for byte the same in every episode, delivered through whatever hook each framework offers: a prompt template, a task prompt, a QA prompt. Consequently, what differs between rows of the tables below is retrieval and nothing else.

How every answer was scored

Four layers, in order of how much they can be argued with. For example, retrieval recall is set arithmetic: did every document holding the gold evidence come back? Likewise, numeric agreement is deterministic: do the figures in the answer match the gold figures within two percent, with dates excluded? Then, invented numbers are figures that appear in no retrieved context and cannot be reached from any two of them by one arithmetic step. Finally, only the fourth layer is a model: a blind judge that grades “System A” and never learns which framework produced the answer, returning one of CORRECT, PARTIAL, WRONG or ABSTAIN.

In addition, two more measures came out of the series rather than going into it. The share of gold evidence quotes that actually reached the prompt, as opposed to the share of gold documents, was introduced in the LlamaIndex episode after document recall and correct answers parted ways, and it has been recomputed for every run since. Furthermore, every episode’s numbers pass through a claims ledger: a script that regenerates each published figure from the result files and refuses to mark it publishable while any input is still moving. The consolidation this article is drawn from reads only rows whose ledger is clean.

The scorecard rubric, frozen before the results existed

Indeed, a comparison invented after the numbers are in is a justification. The rubric was written on 22 August 2026, when only the naive baseline and LightRAG had been measured, and has not been edited since. Eight dimensions map onto 0 to 10 against fixed anchors, so a framework’s score does not move when another one is added later:

To be clear, retrieval is worth 40 of the 100 points and correctness 20, on purpose: a system that fetched the evidence and failed to use it is one prompt away from a fix, while a system that never fetched it needs a new index. Of course, the price of that choice is that the total invites misreading. A score of 85 is not 85% correct answers. For that reason, every score below is restated in questions out of forty, the way the series has done since the RAGFlow episode.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

The RAG framework ranking on the frozen scorecard

Seven frameworks, twenty-eight measured modes, one floor. Note that the scorecard ranks modes, not frameworks, because a framework’s modes are as far apart as different frameworks are. The best mode of each one:

Three things the table says at once

First, the top three are flat retrievers with no graph. The top three are flat retrievers with no graph, and all three reached the top by one move: showing the model more, smaller pieces of the corpus for the same number of tokens. Second, the two graph frameworks sit fourth and fifth on the strength of perfect document recall, and answered 18 and 8 questions after 11 and 13 hours of indexing. Third, the spread between the best and the worst measured mode of the same framework is wider than the spread between frameworks: Haystack ranges from 10 to 27 correct, R2R from 11 to 22, LlamaIndex from 5 to 19.

How to read a score of 85.5

The scorecard’s top entry, Haystack’s hybrid-plus-reranker recipe on its default 200-word splitter at a matched token budget, scores 85.5. In questions:

• for 25 of the 25 questions whose answer sits in more than one document, it retrieved every evidence document (30 of 30 points);
• for 15 of the 15 questions settled inside a single document, it retrieved every evidence document (10 of 10);
• the blind judge called 27 of 40 answers fully correct (13.5 of 20);
• 28 of the 30 numeric questions hit the value within two percent (9.3 of 10);
• 35 of 40 answer-or-refuse decisions were the right call (8.8 of 10);
• 67.5% of answers stand on evidence it retrieved itself, with no invented numbers to subtract (6.8 of 10);
• 0.01 hours to build the index and a median 25.3 seconds per answer (5 and 2.2 of 5).

In sum, forty of those 85.5 points came from finding the right documents; 13.5 came from answering correctly, 27 times out of 40. Similarly, the same sentence holds for every row: the score rewards retrieval twice as much as answering, on purpose, and the count of correct answers is the number to carry away.

Which dispersion pattern breaks which RAG framework

Above all, the series exists for this table rather than the one above. Correct answers per pattern, in questions out of five, for the best mode of every framework and the floor:

Why the D8 column reads backwards

D8 is the column that reads backwards. Here, a correct answer is a refusal phrased as a statement, and the judge marks a bare refusal as ABSTAIN, so the flat retrievers’ zeros mean they refused all five, which is the right behaviour. Notably, the column is where the graph frameworks score and the flat ones do not, and it is also the pattern where LightRAG was measured refusing less reliably than plain vector search on the answerable questions.

Three groups of patterns

Read across the rows, the patterns sort into three groups. D2, D3 and D5, the footnote, the definition distance and the event-to-consequence patterns, are solved: the best flat retrievers answer them five for five. Next, D1, D4 and D6, the table-against-narrative, granularity and governance patterns, are half solved, three of five at best, and the best is never the same framework for all three. Lastly, D7, the restatement across the ten-for-one stock split, is not solved by anything: the best result of the series is four of five, from kotaemon’s full-text index in hybrid mode, and no dense retriever managed more than two. This is because the two annual reports before and after the split are near-identical in vocabulary, and only a lexical match on the restated figure tells them apart.

Retrieval is not the hard part: the evidence across 28 modes

To recap, two measures of retrieval were kept for every mode: whether the right documents were represented in the prompt, and whether the right passages were. The first is what a framework reports; the second, however, is what the model reads. For instance, they parted ways in the LlamaIndex episode, where 67.5% document recall came with 13.3% of the gold passages in the prompt, and they parted ways in every episode since. Moreover, in three frameworks the mode with the highest document recall answered fewer questions than a mode with lower recall. As a result, in the chart above the correct answers follow the passages, not the documents.

Moreover, the move that carried the top three frameworks is the same move each time: hold the evidence budget in tokens fixed and cut the corpus into smaller pieces, so the same tokens point at more places. For example, LlamaIndex went from 5 to 19 correct between the same auto-merging index at top-k 8 and at a matched budget. Haystack’s default 200-word splitter at a matched budget went from 16 to 27. R2R’s default limit of ten 224-token chunks answered 11; the same index at the matched budget answered 22. Meanwhile, nothing about the retriever changed in any of the three.

What a graph cost, and what it returned

Specifically, four pairs in the series isolate a structure against its own flat mode on the same framework. LightRAG’s graph against its naive mode: +2.5 points of correct answers overall, +60 on D6, −40 on D2 and D3. GraphRAG local against basic: +2.5 overall. RAGFlow’s knowledge graph compiled into the same index: +2.5 overall, every point on D4. RAGFlow’s RAPTOR tree: +10 overall, spread over D2, D3, D4 and D8. By comparison, the graph builds cost 11 and 13 hours; RAPTOR cost 3.3. The best flat retriever was built in 41 seconds and answered more questions than any graph mode by nine.

However, that is not the same as saying graphs do not work. For example, where a question needs the proxy statement joined to the annual report, D6, LightRAG’s graph answered five of five and no flat retriever passed three. Similarly, where the answer is not in the corpus, D8, the graph frameworks produced a statement of absence where the flat ones refused. In short, a graph earns its hours on the two patterns where the link between documents has to be written down in advance; on the six others it is a tax, and on the in-document patterns it is a penalty.

Which RAG framework for what

Finally, the scorecard answers “which one should I try first”. By contrast, the decision matrix answers the question a reader actually has, which is “which one for my documents”:

Three things the series measured that no single episode could

The same configuration is not the same index. For instance, RAGFlow’s RAPTOR built twice from the same settings changed the prompt in 37 of 40 questions. LlamaIndex draws node ids from uuid4, so its id-bearing digest never repeats. kotaemon counts a random file id against the chunk budget, so four builds gave four chunk counts. By contrast, Haystack and R2R reproduced both digests every time. Therefore, before comparing two runs of any framework, compare their digests; the series’ code bundle does it in one script per episode.

“Hybrid” is a word, not a mechanism. Haystack fuses two ranked lists by reciprocal rank and the fusion has a measured edge case at position six. Meanwhile, R2R fuses two lists of which the full-text one is empty, because its query ANDs every word of the question. And kotaemon concatenates the two lists and cuts from the top, so the vector list never reaches the prompt. In other words, three frameworks, three different things behind the same setting, and only one of them was a hybrid.

The judge was blind and the numbers were checked twice. Across the 28 modes and 1,120 judged answers, no measured mode contained a figure absent from its own retrieval. The check exists because the first version of the floor, in episode 2, once cited a share count that appears in no filing. Finally, every published number in every episode was regenerated from the result files by a script that refuses to publish a moving value, and the seven framework ledgers behind this article hold 610 verified claims and none pending.

Summary of the RAG framework ranking

Seven retrieval frameworks were measured on the same 33 SEC filings, the same 40 cross-document questions, the same local model and the same blind judge, one framework at a time, each mode on a freshly built stack. In the end, on the frozen scorecard Haystack’s default splitter at a matched budget scored 85.5 with 27 of 40 correct, R2R 80.1 with 22, LlamaIndex 77.7 with 19, LightRAG 74.4 with 18, GraphRAG 65.3 with 8, kotaemon 58.1 with 12 and RAGFlow 57.9 with 13. Meanwhile, the floor answered 6. Above all, the result that matters more than the order is the shape of it: the three best modes are flat retrievers that show the model more, smaller pieces for the same tokens, and the two graph frameworks earned their hours on exactly two of the eight patterns. In short, retrieval was not the hard part. Instead, getting the right passage, rather than the right document, in front of the model was.

RAG framework ranking FAQ

Which RAG framework scored best on the benchmark?

Haystack, with its default 200-word splitter and a hybrid-plus-reranker recipe at a matched token budget: 85.5 on the frozen scorecard and 27 of 40 correct answers, 100% document recall, no invented numbers. Next, R2R was second with 22 of 40 at a matched budget.

Is a knowledge graph worth building for RAG on financial filings?

Only for two of the eight dispersion patterns. LightRAG’s graph answered every governance-against-operations question and the graph frameworks handled absence better; on the six other patterns a graph was a tax of 11 to 13 hours, and the best flat retriever answered nine more questions than any graph mode.

Why does document recall not predict correct answers?

Because a framework can put the right file in the prompt without the right passage. Therefore the two measures diverge. In fact, measured across 28 modes, correct answers follow the share of gold passages in the prompt, and in three frameworks the mode with the highest document recall answered fewer questions than a mode with lower recall.

What single change improved results the most?

Above all, smaller chunks at the same token budget. LlamaIndex went from 5 to 19 correct, Haystack from 16 to 27 and R2R from 11 to 22, each on the same index and the same retriever, by changing only how many pieces reach the prompt.

Which framework handles restated figures across a stock split?

As yet, none reliably. So far, the best result on the D7 pattern is four of five, from kotaemon’s full-text index; Haystack’s BM25 managed three; every dense retriever in the series managed at most two, because the pre-split and post-split annual reports are near-identical in vocabulary.

Can the benchmark be reproduced?

Yes. Indeed, each episode ships a code bundle with the harness, the golden set, the container definitions and every raw result file, and the ebook assembles the appendix of all 40 questions with every system’s answer from those files. The ledgers hold 610 verified claims across the seven framework episodes.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.