RAG on SEC filings: anatomy of a benchmark corpus

Part of RAG & Vector Databases — retrieval-augmented generation and vector search.

Running RAG on SEC filings punishes a pipeline that looked perfect on demo PDFs. However, the reason is not document length. Rather, the reason is that a single answer rarely sits in a single place: it is split between a table and the paragraph explaining it, between a footnote and the line it qualifies, and between filings published two years apart. This article measures that dispersion on a real corpus of 33 NVIDIA documents, names the eight patterns it produces, and shows how to build the same corpus for your own documents. Every filing referenced here is linked in full under source documents.

What sits inside the NVDA-RAG-40 corpus

The corpus covers one company across three fiscal years. NVIDIA works well as a subject for a retrieval benchmark, because its filings are long, numerically dense and heavily cross-referenced. Moreover, every document comes from SEC EDGAR, the public filing system of the U.S. Securities and Exchange Commission.

Altogether four filing types appear, and each contributes a different kind of difficulty. Annual reports (10-K) carry the audited financial statements. In addition, quarterly reports (10-Q) repeat the same metrics at a finer granularity. Proxy statements (DEF 14A) hold governance and compensation data that never appears anywhere else. Finally, current reports (8-K) announce events, usually without quantifying them.

In practice, the size spread matters more than the total. As a result, a 10-K is roughly 65 times an 8-K, so any retriever that ranks by similarity alone will drown the short documents. Therefore the corpus deliberately keeps 20 small 8-K filings against 3 large 10-K filings.

Why RAG on SEC filings starts with 85% markup

To begin with, SEC filings are not PDFs, and they are not plain HTML either. They arrive as Inline XBRL: machine-readable accounting tags embedded inside the presentation markup. As a result, a raw filing is mostly scaffolding. Across this corpus, 85.3% of the raw bytes are markup, and the readable text is the remaining 14.7%.

Moreover, that scaffolding is where most pipelines quietly lose data. Meanwhile, a naive text extraction returns prose in which every financial table has collapsed into a run of loose numbers. The digits survive, although the row labels and the column headers do not, so the number stops meaning anything.

Consequently the conversion step deserves real attention. Specifically, our converter drops the hidden XBRL context block, renders each table as a Markdown pipe table, and promotes Item 1A. style lines to headings so the section structure survives into the chunker. It runs on the standard library plus BeautifulSoup:

# 1) Download the filings from SEC EDGAR (stdlib only, writes SHA-256 manifest)
python benchmark/corpus/download_edgar.py

# 2) Inline XBRL -> Markdown, preserving tables and Item structure
python benchmark/corpus/clean_filings.py

# 3) Measure the corpus (exact tokens via llama-server /tokenize)
python benchmark/corpus/corpus_stats.py

All five scripts, plus the chart and screenshot generators and the measured JSON behind every figure here, are packaged together: download the corpus pipeline (ZIP). Unzip it and the commands above run as written, because the folder layout matches the paths.

Eight ways RAG on SEC filings loses the answer

In short, dispersion is not one problem. Instead it is eight, and they fail differently. Four patterns keep the knowledge inside a single document, while four scatter it across documents. Furthermore, naming them matters, because a framework that solves one may be helpless against another.

For that reason, the sections below walk the patterns that do most of the damage in practice. Each one comes with the measurement that proves it exists in this corpus.

D1: why RAG on SEC filings misreads tables

Financial tables look like grids. However, they are laid out for print rather than for parsing. A currency row spends separate cells on the dollar sign and the closing parenthesis; a percentage row does not. As a result, the same visual column lands on a different cell index depending on the row.

Therefore we measured this on the source markup of all 33 filings. In total, out of 913 multi-row tables, only 2 keep a consistent cell count across every row. In 643 tables — 70.4% — the data rows disagree with each other, not merely with the header. Worse still, one table in the FY2026 annual report contains rows of nine different widths.

In other words, this is what it looks like in the filing itself. Notice how the dollar sign occupies its own cell on the Revenue and Net income rows, yet disappears on the rows between them.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

Therefore, keeping such a table usable means dropping the spacer columns rather than the data. Consequently a column survives only when at least one row puts something meaningful in it:

# A cell that carries no information: currency symbols, spacers, rules.
NOISE_CELL_RE = re.compile(r"^[s$%()—–.:,* |-]*$")

# Drop a column only when every row agrees it is noise.
keep = [c for c in range(width)
        if not all(NOISE_CELL_RE.match(r[c]) for r in rows)]
rows = [[r[c] for c in keep] for r in rows]

Importantly, the count itself comes from the source markup, not from our converted copy, because conversion normalises the very geometry we want to measure:

counts = [len(tr.find_all(["td", "th"]))
          for tr in table.find_all("tr") if tr.find_all(["td", "th"])]

if len(set(counts)) == 1:          # every row agrees
    uniform += 1
elif len(set(counts[1:])) == 1:    # only the header differs
    header_only += 1
else:                              # data rows disagree with each other
    body_ragged += 1

D2: the footnote that changes the number

Similarly, a figure in a financial statement is frequently qualified elsewhere. In practice, the corpus contains 311 explicit “Note N” references, concentrated in the quarterly reports (196) and annual reports (107). In effect, each one is a pointer from a number to the text that gives it conditions.

A chunker typically splits the table and its note into different chunks, and a retriever then returns only one of them. As a result, the answer looks correct while remaining quietly incomplete. Note that the 8-K filings contain zero footnote references, which is precisely why they cannot answer quantitative questions on their own.

D4, D5 and D6: the answer lives in another document

Three patterns move the answer out of the document entirely. First, granularity mismatch (D4): a question about an annual figure may require summing four quarterly reports, or reconciling them against the audited annual one. This corpus holds 8 quarterly filings against 3 annual filings precisely so that reconciliation is testable.

Second, the event-then-consequence pattern (D5). An 8-K announces something — a leadership change, a buyback authorisation, a material agreement — and a later 10-K or 10-Q quantifies its effect. However, the 8-K filings here carry a median of just 1,277 tokens and, as noted, zero footnote references. In other words, they are pointers rather than sources.

Third, governance sits apart from operations (D6). Executive compensation, board composition and shareholder proposals appear only in the DEF 14A proxy statement, which contributes 72,122 median tokens but only 8 footnote references. Consequently a question about pay ratios cannot be answered from any 10-K, no matter how good the retriever is.

These three patterns share a failure signature. The retriever returns a confident, well-formed chunk from the wrong document class, because that chunk is topically similar. Therefore recall@k measured on a single document type systematically overstates how well a framework will perform.

D3 and D7: the answer that expired

By contrast, the sharpest failure in this corpus is not a missing number. It is a stale one. In June 2024 NVIDIA executed a ten-for-one stock split, so every per-share figure filed before that date was retroactively divided by ten in later filings.

To confirm this, our extraction script pulled the diluted EPS row from all three annual reports. Fiscal 2024 diluted EPS appears as $11.93 in the 10-K filed on 2024-02-21, and as $1.19 in the 10-K filings from 2025 and 2026. Likewise, the diluted share count moved the same way, from 2,494 million to 24,940 million. Fiscal 2023 shows the identical pattern, at $1.74 against $0.17.

Building a RAG corpus from your own documents

Fortunately, the method transfers to any regulated or versioned document set: contracts, clinical protocols, engineering specifications. First of all, choose one entity and a time span long enough to contain at least one revision. Second, gather more than one document type, because single-type corpora hide the between-document patterns entirely.

Next, convert with the tables intact and record a SHA-256 manifest, so a re-run is verifiable. After that, measure before you build anything: token counts, table geometry, cross-reference density. Finally, walk the eight patterns above and mark which ones your corpus actually exhibits. A corpus lacking D7 cannot test whether a framework handles superseded facts.

Finally, one practical note on scale. For instance, a 10-K in this corpus is roughly 82,600 tokens, so two full annual reports fit simultaneously inside the 204,800-token context window we serve. That capacity is what makes automated cross-document question extraction possible, which is the subject of the next episode.

Why this RAG corpus is test material, not investment advice

In summary, these documents were chosen because they are public, stable, numerically rich and awkward to retrieve from. Therefore the benchmark measures retrieval systems, not NVIDIA. No statement in this series is a claim about the company’s prospects, and no figure here should inform a financial decision. Anyone needing NVIDIA’s actual numbers should read the filings directly on SEC EDGAR.

Frequently asked questions

What is the hardest part of RAG on SEC filings?

Dispersion, not length. A single answer is typically split across a table, its footnote and a narrative paragraph, and often across two filings published a year apart. Retrieving the right document is rarely the failing step.

How many documents does a RAG benchmark corpus need?

Fewer than people expect, provided that the types vary. This corpus uses 33 documents and 749,859 tokens, but it spans four filing types and three fiscal years, which is what produces the between-document patterns.

Why not just use the XBRL tags instead of the text?

Because the tags answer only numeric questions on their own. Governance facts, risk-factor language and event narratives carry no XBRL tagging, and roughly 86% of a filing’s raw volume is markup that contains no prose at all.

Do SEC filings really change after publication?

Yes — per-share figures do, routinely. NVIDIA’s ten-for-one split in June 2024 restated fiscal 2024 diluted EPS from $11.93 to $1.19 across later filings, and both versions remain publicly available.

Which tokenizer produced these token counts?

The counts come from the /tokenize endpoint of llama-server running Qwen3.8-27B-UD-Q4_K_XL, the same model used throughout this series. Token counts differ between tokenizers, so a count is only meaningful next to its model.

Can I reproduce this corpus exactly?

Yes. The download script records a SHA-256 manifest for every filing, so a later run can verify byte-for-byte that it fetched the same documents from SEC EDGAR.

Summary

The NVDA-RAG-40 corpus contains 33 SEC filings and 749,859 tokens, covering 2023-11-21 to 2026-07-02. Its raw bytes are 85.3% markup. Of its 913 multi-row tables, just 2 are cleanly rectangular, and 70.4% have data rows that disagree with each other. Above all, it contains a genuine retroactive restatement: fiscal 2024 diluted EPS reads $11.93 in one annual report and $1.19 in the next two.

Above all, those properties are the point. A corpus without them tests nothing that production data will throw at a retriever. In the next episode we turn this corpus into a golden set, using the long context window to extract questions that genuinely require combining documents.

The eight dispersion patterns above are what that extraction targets, one pattern at a time. The golden-set episode publishes the complete prompt for each of them, together with the structural rule that decides whether a proposed question really exhibits its pattern.

Further reading on why structural chunking beats paragraph chunking for financial documents: Financial Report Chunking for Effective Retrieval Augmented Generation.

Source documents for RAG on SEC filings

In short, every number above traces back to one of the 33 filings listed here. Furthermore, each row links to the document exactly as NVIDIA filed it, so any claim in this article can be checked at the source. Note also that token counts come from the serving model’s own tokenizer.

Periodic and proxy filings

Together, these 13 documents carry the audited figures, the footnotes and the governance data. Consequently they supply most of the evidence for the dispersion patterns above.

Current reports

Meanwhile, the 20 8-K filings announce events as they happen. Individually they are tiny, yet they are what makes the event-then-consequence pattern testable.

Finally, the full filing history sits on NVIDIA’s SEC EDGAR company page. Because the download script records a SHA-256 hash for every file, a later run can prove it fetched exactly these documents.

Companion code — the fragments above are the working parts of the pipeline: the noise-column filter that keeps financial tables readable, and the row-width check behind the raggedness count. Everything runs from one archive — nvda-rag-e01-corpus-scripts.zip — which also carries the measured JSON, so the charts rebuild without re-downloading a single filing.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.