Part of RAG & Vector Databases — retrieval-augmented generation and vector search.
extract_questions.py, pattern_conformance.py, verify_answers.py, rebuild_patterns.py and run_benchmark.py · 33 NVIDIA SEC filings · Qwen3.8-27B on llama-server at temperature 0 · nomic-embed-text-v1.5 · measured 2026-08-21. Nothing is estimated.
A RAG golden set is only worth building if the questions are hard in the right way. Most people write benchmark questions from a single paragraph. A plain vector search then answers them, and every framework scores the same. This article shows how we used a long-context model to propose 214 questions, how ten machine checks cut those to 40, and what a naive baseline scored on the result.
The short version: the model is a proposal generator, not an author. Four of every five proposals were thrown away, and the interesting part is what threw them away.
- The model proposed 214 questions. 189 survived quote verification, 142 passed their pattern contract, and 40 entered the frozen set.
- Of 400 evidence quotes it cited, 25 existed nowhere in the sources — a 6.2% fabrication rate under ideal conditions.
- Sharpening the prompt moved conformance from 46.9% to 71.9%, measured over the same excerpts with the same checker.
- Adding rules to a prompt has a price: in two patterns the fabrication rate jumped from under 2% to about 40%.
- A person still had to read all 40. Machine checks cannot prove a negative, and four absence questions claimed the corpus lacked a figure it states in a table row.
Why most RAG golden set questions are too easy
Ask a model to “write hard questions about this document” and it writes questions about one paragraph. The paragraph is right there in its context, so the question it invents is answerable from that paragraph alone. As a result, the benchmark measures whether a retriever can find one chunk, which every retriever can do.
By contrast, our approach inverts the instruction. Instead of asking for difficulty, we target a specific dispersion pattern — one of the eight ways knowledge scatters through a filing corpus that the previous episode identified. We tell the model which pattern to produce. We also hand it material aligned to that pattern rather than a single document.
However, that still leaves the hard question: how do you know the question you got back really exhibits the pattern you asked for? Tagging is not evidence. This episode is mostly about answering that question with code.
Aligning documents for the RAG golden set
First of all, cross-document questions need cross-document material in the prompt. Therefore the pipeline splits every filing into its Item sections first. It then hands the model the same section from different filings, or two sections that sit far apart inside one filing.
In addition, section splitting has one wrinkle worth knowing. A 10-Q repeats its item numbers, because Part I carries the financial statements as “Item 1” and Part II carries legal proceedings as “Item 1” again. Sections are therefore keyed by part as well as number.
Similarly, head-trimming a long section does not work, and this cost us a full iteration. For the governance pattern we paired the proxy statement with the annual report and got nothing usable twice. The first pairing used Item 11 of the 10-K — which is only a cross-reference pointing at the proxy, so there was nothing to combine. The second trimmed both documents from the start, and the start of a proxy statement is the meeting agenda, not the compensation tables.
Centring each excerpt on an anchor phrase fixed it:
def focus_window(text: str, anchor: str, chars: int,
lead: float = 0.25, stop: str | None = None) -> str:
position = text.lower().find(anchor.lower())
if position == -1:
return text[:chars]
start = max(0, position - int(chars * lead))
end = start + chars
if stop:
cut = text.lower().find(stop.lower(), position)
if cut != -1:
end = min(end, cut)
return text[start:end]Two windows that carried each other’s text
The stop and lead arguments exist because of one failure that took a while to see. The footnote pattern needs a figure from the financial statements and the Note that qualifies it. Both live in Item 15, so both windows were cut from Item 15 — 72,000 characters each, centred a few thousand characters apart.
As a result, the two source blocks carried almost the same text. Asked for one quote from each side of the boundary where the notes begin, the model had no boundary to work with: six of eleven verified candidates quoted the notes twice, and two in five of its quotes were invented outright.
Finally, cutting the windows so they cannot overlap — the statements up to where Note 1 starts, the notes from there on — changed that round from 1 usable question to 20, with fabrication falling from 40.4% to 1.8%.
Three long-context traps when building a RAG golden set
Getting the extraction to run at all took three fixes that are easy to miss. Each one produced an empty or useless result rather than an error message, which is what made them expensive.
A raw completion prompt answers with silence
Our first attempt sent a 241,000-character prompt to the /completion endpoint. The server evaluated all 71,409 tokens and returned exactly one token: end-of-sequence. Without a chat template the model treats the prompt as a document to continue, and a document ending in a JSON schema is complete. Switching to the chat endpoint fixed it.
The reasoning block eats the whole budget
Qwen3.8 ignores the /no_think directive that works on Qwen3.6. At 71,000 tokens of context it spent all 12,288 generation tokens inside its reasoning block and returned an empty content field. The supported switch is a template argument:
{
"model": "Qwen3.8-27B-UD-Q4_K_XL.gguf",
"messages": [ ... ],
"max_tokens": 8192,
"temperature": 0,
"chat_template_kwargs": { "enable_thinking": false }
}More context is worse context
The server window is 204,800 tokens, so we filled it. That was the mistake. Cutting the excerpt budget from 90,000 to 36,000 tokens took one pattern from zero usable questions in 535 seconds to seven verified questions in 52 seconds. Aligned, trimmed sections beat raw volume.
The prompt behind the RAG golden set
In practice, every pattern uses one template. Three substitutions change: the pattern definition, the structural requirement, and worked examples mined from the previous round. The complete instruction set is below, exactly as it was sent.
Every pattern is extracted with the same template. Only three substitutions change: the pattern definition, the structural requirement and the worked examples mined from the previous round.
You are building a retrieval benchmark over SEC filings by NVIDIA.
DISPERSION PATTERN: {pattern_id} -- {pattern_name}
{pattern_definition}
Below are excerpts from the corpus. Each excerpt is introduced by a SOURCE line.
[... aligned excerpts, each introduced by a SOURCE line ...]
TASK
Write exactly {n} benchmark questions that exhibit the {pattern_id} pattern.
{multi_doc_rule}{structure_rule}
{worked_examples}
RULES
1. A question must NOT be answerable from a single short passage. It must require
combining material that sits in different places.
2. NEVER name a specific filing, Item number, section or page in the question. Do not
write "according to the 2026 10-K" or "between the two annual reports". The person
asking does not know how the corpus is organised, and a question that says where to
look is not testing retrieval. Ask for the fact itself.
2b. This includes the names of financial statements and note sections. Do not write
"according to the notes to the consolidated financial statements", "in the
consolidated balance sheet" or "in the statement of shareholders' equity". Those
phrases tell a retriever which part of the document to open, which is precisely
the work being measured.
3. Write the question in PLAIN ENGLISH, as one sentence of at most 35 words. A reader
who is not an accountant must be able to follow it on the first read. Prefer a
common word to a technical one where both are exact. Do not stack three
subordinate clauses; if the question needs two parts, join them with one "and".
3b. Keep the gold answer plain too: short sentences, none longer than 30 words.
3c. ALWAYS say which period the question is about - a fiscal year, a quarter, or an
as-of date. This corpus holds three annual reports and eight quarterly ones, so
"what is the total inventory balance" has eleven different correct answers and
grades an honest system as wrong. A question that asks for an absent fact is the
only exception.
4. Give the gold answer exactly as the sources support it. Be numerically precise, and
state the unit. Where two filings disagree, the gold answer is the CURRENT one, and
the reason the other is superseded belongs in the answer.
4b. If you write "increased", "decreased", "higher" or "lower", check the two figures
agree with the word. "Decreased from 69.3% to 71.1%" is an increase.
4c. If the answer is a list, list everything the evidence names. A partial list marks
a complete answer wrong.
5. For every question, quote the evidence VERBATIM from the excerpts above. Copy the
characters exactly. Never paraphrase a quote. Never invent a quote.
6. Name the SOURCE document each quote came from.
7. If the pattern is D8, the gold answer must say the corpus does not contain it, and
the evidence quotes should show the closest related material that still fails to answer.
OUTPUT
Return ONLY a JSON array, no prose before or after. Each element:
{"question": "...", "answer_type": "number|string|date|list|refusal",
"gold_answer": "...", "evidence": [{"document": "...", "quote": "..."}],
"requires_documents": 1}The structural requirement is what the A/B test moved. Below is what each pattern substitutes into {structure_rule}, next to the source windows it is given.
| Pattern | Source windows | Structural requirement |
|---|---|---|
| D1 Table vs narrative split | 2026-02-25_10-K:P1-Item15:Consolidated Statements of Income 2026-02-25_10-K:P1-Item7:Gross margin | |
| D2 Footnote dependency | 2026-02-25_10-K:P1-Item15:{‘anchor’: ‘Consolidated Statements of Income’, ‘lead’: 0.02, ‘stop’: ‘Note 1 – Organization’, ‘label’: ‘the financial statements’} 2026-02-25_10-K:P1-Item15:{‘anchor’: ‘Note 1 – Organization’, ‘lead’: 0.0, ‘label’: ‘the notes to the financial statements’} | |
| D3 Definition-then-use distance | 2026-02-25_10-K:P1-Item1:Compute & Networking 2026-02-25_10-K:P1-Item8 2026-02-25_10-K:P1-Item7:Compute & Networking | |
| D4 Granularity mismatch | 2026-02-25_10-K:P1-Item7 2025-08-27_10-Q:P1-Item2 2025-11-19_10-Q:P1-Item2 | |
| D5 Event then consequence | 2025-04-15_8-K 2025-01-17_8-K 2025-05-28_10-Q:P1-Item2 2026-02-25_10-K:P1-Item7 | |
| D6 Governance vs operations | 2026-05-12_DEF14A:Summary Compensation Table 2026-02-25_10-K:P1-Item15:Stock-based compensation expense | |
| D7 Retroactive restatement | 2024-02-21_10-K:P1-Item15 2026-02-25_10-K:P1-Item15 | |
| D8 Absence | 2026-02-25_10-K:P1-Item1 2026-02-25_10-K:P1-Item7 | none — definition only |
How much prompt wording moves a RAG golden set
We measured it rather than guessing. The same excerpts and the same checker were run three times, varying only how the requirement was expressed: v1 gave the pattern definition and the output schema, v2 added the structural requirement in prose, and v3 added one accepted and one rejected example from the previous round, the rejection carrying the checker’s own words.
| Prompt version | Accepted, of 96 proposals |
|---|---|
| v1 — definition and schema only | 46.9% |
| v2 — plus the requirement in prose | 59.4% |
| v3 — plus worked examples | 71.9% |
The average hides the interesting part. Four patterns inject their requirement as a hard rule enforced in code — “cite at least two different documents” — and they score 75% to 100% even on v1. Prompt wording has nothing to add there.
The three within-document patterns, where the requirement could only be described in words, score zero on v1: thirty-six proposals, not one usable question. All of the value of prompt engineering sat exactly where a rule could not be enforced mechanically.
A prompt that quotes text the model cannot see
The worked examples in v3 come from the previous round of the same pattern, which creates a trap worth naming. After the footnote pattern’s source windows moved, its mined example still carried a sentence from the old window — text no longer in the excerpts.
Consequently, the model copied that example wholesale: its question, its answer and its quote appeared in all 26 proposals of the next round. Quote verification rejected 26 of 26. The filter worked, and the round was still wasted.
quotes_visible = (not context or all(
_norm(ev.get("quote", "")) in _norm(context)
for ev in candidate.get("evidence", [])))
if result["conforms"] and candidate.get("verified") and quotes_visible and good is None:
good = candidateAn example is now only used when its quotes are still visible in the excerpts being sent. A prompt that quotes text the model cannot see teaches it to cite text it cannot see.
Ten gates every RAG golden set question passes
Each gate exists because something got through the gates before it. That is the honest way to read the list: it is a record of mistakes, not a design drawn up in advance.
| Gate | What it rejects |
|---|---|
| Quote verification | Evidence that exists nowhere in the corpus |
| Pattern contract | A question that does not structurally exhibit its own dispersion pattern |
| Answer support | A figure that does not follow from the quoted evidence |
| Source leak | A question that names the filing, the section or the note holding the answer |
| Plain language | Questions over 35 words, more than one sentence, or answer sentences over 30 words |
| Named period | A question about a balance with no date, which eleven filings answer differently |
| Direction check | “Decreased from 69.3% to 71.1%” |
| Idle quote | A quoted passage the answer would survive without |
| Idle document | A second filing cited so the question can claim to span two |
| Deduplication | Two questions with the same distinctive words, or the same figures in their answers |
Verifying the evidence rather than trusting it
To begin with, each proposal must quote its evidence verbatim. The pipeline then searches for that quote in the source texts, after normalising whitespace and quotation marks. A question resting on invented evidence is rejected before a human reads it.
Consequently, that search found 375 of 400 quotes. Twenty-five it did not find anywhere, a 6.2% fabrication rate. Conditions were otherwise ideal: the source text sat in the model’s own context window, under an explicit instruction to copy characters exactly.
The document each quote came from is resolved by that search rather than read from the model’s label, because the model labels inconsistently and sometimes omits the field. That resolution is also what makes the cross-document requirement enforceable at all.
Checking that the question is what it claims to be
A tag is a claim. pattern_conformance.py turns each of the eight claims into conditions that can be measured on the source text:
elif pattern == "D2":
notes_start = notes_region_start(unique_documents[0])
located = [entry["offset"] for entry in offsets if entry["found"]]
if not any(offset >= notes_start for offset in located):
failures.append("no quote comes from the notes to the financial statements")
elif not any(offset < notes_start for offset in located):
failures.append("every quote sits inside the notes; none is the figure they qualify")Moreover, distances are character counts in the cleaned filing, not impressions. Two quotes count as dispersed within a document when they sit at least 4,000 characters apart — roughly a printed page, so anything closer is material one retrieved chunk could plausibly carry.
The answer has to follow from the evidence
Even so, a real quote can still be read wrongly. So a second checker takes every number in every gold answer and asks whether it appears in a quoted span, or sits one arithmetic step from two quoted numbers, with a 2% tolerance for rounding and for the millions-versus-billions switch that filings invite.
Free ebook
Free AI Video, Generated Locally
Working scripts and measured benchmarks. Free.
No spam. Unsubscribe at any time.
That tolerance turned out to be wrong in one direction, and the fix is worth stating because it is the kind of error that quietly deletes good data. The evidence gives 62,556 minus 60,608 = 1,948 million. The answer says “$1.9 billion”. That is correct rounding, and it is a 2.5% deviation — outside a 2% band.
if decimals is not None and abs(a - b) <= 0.5 * 10.0 ** (-decimals):
return True
if b == 0:
return abs(a) < 1e-9
return abs(a - b) / abs(b) <= TOLERANCETherefore the tolerance now has a floor set by how the answer is written: half a step of its last written digit. At $215,938 million that floor is far tighter than 2% and changes nothing. At “$1.9 billion” it is what stops the checker from rejecting an answer for being correctly rounded.
Every quote and every document must do work
Meanwhile, the subtlest gate is the last one added. A question can cite two documents, pass every structural test, and still be a lookup wearing a costume:
The 8-K is cited. The 8-K does nothing: the answer is one figure out of the 10-Q, and the announcement is scenery.
The test asks what each part contributes. A quote or a document is idle when every figure in the answer is still written out somewhere else, and it brings fewer than two of the answer’s own content words that no other part brings. Deliberately, that test uses direct quotation only, with no arithmetic: with eight numbers in a table row, a 2% tolerance and five operations at three scales, some combination hits almost any target, and an earlier version of this check rejected twenty-five sound questions in a row because of it.
A question may not name its own source
In the same way, naming the document hands over the retrieval task. The rule started as a list of form names and grew four times, each time because something got through:
10-K,proxy statement,the filing— the original list;the notes to the consolidated financial statements,balance sheet— section names point just as precisely;Note 9,the note on leases— a note number is the finest-grained address in the filing, and the footnote pattern invites it;the original filing,the restated filing,filings— an adjective in front of the noun hides the same leak.
The last group is the reason this rule now runs during selection instead of afterwards. In the previous version of the set, four questions had to be edited by hand at freeze time to remove exactly those phrases. This version needed no hand edits at all.
Plain English, and a period
Two thresholds keep the questions readable: one sentence of at most 35 words, and no answer sentence over 30 words. They sit just below what the extractor produces unprompted, so they compress rather than reject. Average question length fell from 31.4 words to 22.6.
Similarly, the period rule is not cosmetic. “What is the total intangible assets balance?” has eleven correct answers in this corpus, one per filing date. A system that returns the FY2025 figure would be marked wrong for a question that never said which date it meant.
What building a RAG golden set taught us about prompting
Building this set meant running the same extraction dozens of times with one variable changed. Three results generalise beyond SEC filings.
Adding rules to a prompt is not free
Every gate that could not be enforced in code had to be described in the prompt, and the prompt grew. The cost showed up in the fabrication rate, measured on the same patterns before and after:
| Pattern | Invented quotes, short prompt | Invented quotes, long prompt |
|---|---|---|
| Footnote dependency | 1.8% | 40.4% |
| Governance vs operations | 0.0% | 39.3% |
| Definition-then-use, restatement, granularity | 0.0% | 0.0% |
In other words, a model pressed by more conditions than it can satisfy does not refuse. It invents a quote that would satisfy them. The filter catches that, so the set stays clean — but the yield collapses, and in the governance round only 6 of 28 proposals survived.
A prohibition is weaker than a format
Three rounds in a row, the granularity pattern returned answers of the shape “$68.1 billion, which is 31.5% of the full year”. That percentage is two arithmetic steps from anything quoted, so the answer gate rejected it. Three rounds in a row, a plainer and plainer prohibition failed to stop it.
By contrast, what stopped it was replacing the prohibition with a template:
ANSWER FORMAT, no exceptions: state the two period figures and nothing else.
'Nine-month revenue was $147,811 million and full-year revenue was $215,938
million.' NEVER add a percentage of your own.The same thing had already worked once, for the table-versus-narrative pattern: “quote 1 must contain a | character, quote 2 must not contain a single one” succeeded where three prose descriptions of the same requirement had failed. Give the model the shape of a correct output, not a description of an incorrect one.
The model cannot audit its own work, except when it can
The previous version of this set ran the same adversarial check — a model told to argue that each question is defective — and the conclusion was that it produced noise. Twenty-five of forty items were flagged “major”, and every class of complaint turned out to be a misunderstanding of the benchmark’s design.
On a cleaner set the same check earned its place. It found four real defects that no rule caught: a direction contradiction, three questions with no period, an incomplete list, and an answer whose figure matched the evidence only by coincidence — a $4.5 billion charge “confirmed” by a segment revenue of $4,532 million.
In short, the useful statement is conditional. A judge model applied to a careless set drowns in false positives and tells you nothing. Applied to a set that has already passed its rules, its remaining complaints are worth reading one by one.
What the corpus refused to give the RAG golden set
One dispersion pattern nearly failed to survive contact with the data, and the reason is a fact about SEC filings rather than about our pipeline.
For instance, the event-then-consequence pattern assumes an 8-K announces something without quantifying it, and a later periodic report quantifies the effect. Both documents should be required. In this corpus, they usually are not: of twenty 8-K filings, only four carry a figure that no 10-Q or 10-K repeats. The quarterly report restates its own announcement almost in full.
For the April 2025 export-licence announcement, exactly three facts are exclusive to the 8-K: the $5.5 billion estimated charge, the April 14 confirmation date, and the phrase “indefinite future”. The reason for the restriction, the products covered, the memory-bandwidth threshold and the country group all reappear verbatim in the 10-Q.
Above all, this matters for scoring rather than for taste. “How much H20 revenue was there before the licence requirement announced in April 2025?” looks cross-document and is not: the 10-Q answers it alone. A system that retrieves only the 10-Q would answer correctly and still be marked down for missing a document. The contract rejects those questions, which is why this pattern has one genuinely causal question and four compound ones.
Compound questions, and why they stay
Admittedly, several cross-document questions ask for two facts at once: “what charge was announced in April 2025, and what was the net income for that quarter”. No analyst phrases a question that way. It is still the right shape for this benchmark, because the two halves live in different filings and no single retrieval can satisfy both.
The governance pattern was designed that way from the start — a name from the proxy, a figure from the annual report. What is new is that the event pattern ended up there too, pushed by the corpus rather than by choice. The article states it plainly so that nobody reads those five questions as a claim about how people ask questions.
Where a RAG golden set stops being machine-checkable
In the end, ten machine checks catch everything that can be counted on the source text. They do not catch everything.
For that reason a person read all forty items before the freeze. Two defects surfaced that no rule saw: an answer that said “decreased” about an increase, and a question that stapled two unrelated facts together. Both then became rules, which is the pattern of this whole episode — the eye finds it once, the rule catches it forever.
By comparison, the third finding could not become a rule at all. Absence questions ask the system to refuse, and the model writes them by asserting that the corpus does not contain something. It sees two excerpts and makes a claim about thirty-three filings:
| Question claimed the corpus lacks… | The corpus states |
|---|---|
| the total inventory balance | | Inventories | 21,403 | |
| total deferred tax assets | Total deferred tax assets | 11,604 |
| Automotive revenue for Q1 FY2026 | a segment row in the Q1 10-Q |
| Data Center networking revenue for FY2025 | | Networking | $ | 47,405 | |
Above all, these are the most damaging defects a golden set can carry: a system that retrieves the right figure is marked wrong for not refusing. One of them was caught by a rule that searches the corpus for the question’s distinctive words next to a financial figure. The other three were not, and tightening that rule far enough to catch them also rejected a sound question. They are recorded as a hand-written rejection list with the reason attached, so the selection stays reproducible.
Making the human check on a RAG golden set cheap
No rule proves a negative over a corpus. A person has to look — so the pipeline makes looking cheap. For every refusal question, absence_review.py prints the corpus passages carrying its distinctive words next to a figure, table rows first:
NVDA-D8-0x What is the total amount of NVIDIA's inventory as of January 25, 2026?
claims: The corpus does not contain the total inventory balance.
[inventory] 2024-02-21_10-K ... | total inventories | $ | 5,282 | $ | 5,159 | ...
DECIDE: does any passage above answer the question?Two minutes of reading per question replaces a research task. It is not a gate, because it does not decide; it is the fastest way we found to let a person decide. Rejections go into a list in the selection script, with the reason attached, so the build stays reproducible.
What the frozen RAG golden set contains
What survives is 40 questions, five per dispersion pattern, averaging 22.7 words each. Half need a single document and half need two, which is the balance the whole series is built to measure.
| Property | Value |
|---|---|
| Questions | 40 (5 per pattern D1–D8) |
| Answer types | 25 numeric, 10 string, 5 refusal |
| Documents required | 20 single-document, 20 two-document |
| Evidence quotes | 75, every one located in the cleaned filing |
| Proposals behind them | 214, of which 189 quote-verified and 142 pattern-conforming |
| Hand edits at freeze | 0 |
The last row is the one that changed. The previous version needed five edits at freeze time: four questions that named the document holding the answer, and one gold answer that mentioned a product its own quotes never named. Those rules now run during selection, so a question that would need such an edit never enters the set.
Calibrating the RAG golden set with a naive baseline
A golden set nobody has scored is an untested instrument. We therefore ran a deliberately unsophisticated pipeline against it: fixed 3,200-character chunks, dense retrieval with top-k of 8, one generation call, no reranker and no metadata filter.
As a result, it retrieved every required document for 28 of 40 questions and refused to answer 30 of 40, with a median latency of 17.6 seconds. That is the profile of a well-calibrated set: the retriever often finds the material, and the naive pipeline still cannot turn it into an answer.
Retrieval halves once the answer spans two filings
Splitting the results by depth gives the headline number. Where the answer sits in one document the baseline found everything it needed 18 times out of 20. Where the answer spans two documents that fell to 10 out of 20.
The pattern a vector search cannot see
One number stands out from the rest: the restatement pattern scored zero out of five on document recall. Not low — zero.
The reason is structural, and it is therefore the clearest illustration of what this series is about. A restatement question needs the fiscal 2024 figure as it was first published and as it reads now, which means retrieving two annual reports at once. Those two documents are near-identical in wording: the same statement headings, the same line-item labels, the same accounting language. A dense retriever scores them almost equally, fills its top-k with chunks from whichever one wins, and never brings the other.
No amount of top-k tuning fixes that, because the second document is not slightly less similar — it is a near-duplicate whose only distinguishing feature is a date the query never mentions. Recency awareness and metadata filtering are the remedies, and both are framework features rather than retrieval-quality features. That is exactly why the series reports per pattern.
Compare it with the governance pattern at 2 of 5, which fails for the opposite reason: the proxy statement is a completely different document that a compensation query never reaches. One failure is a routing problem, the other a de-duplication problem, and a single averaged score would hide both.
Finding the document is not answering the question
Document recall of 70% and a correct-answer rate of 15% are the same run. A blind judge — it grades “System A” and never learns which pipeline produced the answer — scored 6 answers correct, 7 partially correct and 27 abstentions.
| Layer | Naive baseline on NVDA-RAG-40 |
|---|---|
| Strict document recall | 70.0% (28 of 40) |
| Judge: correct | 15.0% (6 of 40) |
| Numeric answers exactly right | 60% of the 10 it attempted |
| Abstentions | 30 of 40, of which 5 are correct refusals |
| Answers containing invented numbers | 0 |
| Median latency | 17.6 s |
The failure mode changed, and that is a result
The previous version of this set caught the baseline inventing a figure outright: asked about a restated share count, it answered “24,930 million shares”, a number that appears in no filing anywhere. This version caught nothing of the kind. Zero answers contain an invented number.
Instead of fabrication, what shows up now is mis-selection, and it is worth seeing side by side:
Baseline answer — “Stephen C. Neal, and the net cash provided by operating activities for fiscal 2026 was $27,414 million.”
Gold answer — $102,718 million. The figure the baseline gave is real: it is the operating cash flow for one quarter, lifted from a 10-Q it retrieved instead of the annual report.
The same shape appears in the granularity pattern, where the baseline answered $58.8 billion for a balance of $60,608 million. That figure is in its context too — it is a different line of the same table.
In every case the answers are fluent, precise, sourced and wrong. Our reading is that the questions changed rather than the model: every question now names its period, so the pipeline finds something plausible with the right label and stops looking. When the question was vague, it had nothing to anchor on and made a number up. Neither behaviour is acceptable in financial retrieval, and only one of them is visible to a fabrication detector.
All 40 questions in the RAG golden set
Below is the complete frozen set, grouped by dispersion pattern. Each group opens with what makes that pattern hard and how many filings a correct answer needs. Answers appear in their essential form; the frozen file carries each one in full, next to the verbatim quote it rests on.
D1 — Table vs narrative split
Within a single filing. Here the figure sits in a table, while the sentence explaining why it moved sits in the narrative pages away. A chunker normally keeps one and drops the other, so the answer arrives without its reason. One document is enough here, which is why a plain retriever copes best with this pattern.
The five questions
| ID | Question | Gold answer |
|---|---|---|
| D1-01 | What was the total revenue for the fiscal year ended January 25, 2026, and what specific charge impacted gross margin due to H20 inventory? | Total revenue was $215,938 million. The specific charge was $4.5 billion associated with H20 excess inventory and purchase obligations. |
| D1-02 | What was the net income for the fiscal year ended January 25, 2026, and what was the primary driver of the increase in other income, net? | Net income was $120,067 million. The increase in other income, net was primarily driven by unrealized gains in non-marketable and publicly-held… |
| D1-03 | What was the total assets balance as of January 25, 2026, and what was the total amount of inventory provisions recorded in cost of revenue for fiscal 2026? | Total assets were $206,803 million. The total amount of inventory provisions recorded in cost of revenue for fiscal 2026 was $4.0 billion. |
| D1-04 | What was the total shareholders’ equity as of January 25, 2026, and what was the weighted average period for recognizing unearned stock-based compensation expense for RSUs? | Total shareholders’ equity was $157,293 million. The weighted average period for recognizing unearned stock-based compensation expense for RSUs was… |
| D1-05 | What was the net cash provided by operating activities for the fiscal year ended January 25, 2026, and what was the primary reason for the increase in cash used in investing activities? | Net cash provided by operating activities was $102,718 million. The increase in cash used in investing activities was primarily driven by higher… |
D2 — Footnote dependency
Within a single filing. Similarly, a line in a financial statement is qualified by a numbered Note that changes what it means. Retrieve the figure without its Note and the answer reads as correct while being incomplete. One document suffices, but the retriever has to follow a pointer that the chunk boundary usually severs.
The five questions
| ID | Question | Gold answer |
|---|---|---|
| D2-01 | What was the total inventory balance as of January 25, 2026, and what provision expense was recorded in cost of revenue for that year? | Inventories were $21,403 million, after $4.0 billion of provisions recorded in cost of revenue. |
| D2-02 | What was the total accrued and other current liabilities as of January 25, 2026, and what expense was recorded for excess inventory purchase obligations? | Accrued and other current liabilities were $21,352 million, including $3.2 billion of expense for excess inventory purchase obligations. |
| D2-03 | What was the total goodwill balance as of January 25, 2026, and how much of that increase was allocated to the Compute & Networking reporting unit? | Goodwill was $20,832 million, with $15.6 billion of the increase allocated to the Compute & Networking reporting unit. |
| D2-04 | What was the total intangible assets balance as of January 25, 2026, and what was the amortization expense for fiscal year 2026? | Intangible assets were $3,306 million, with amortization expense of $488 million for fiscal year 2026. |
| D2-05 | What was the total property and equipment balance as of January 25, 2026, and what was the depreciation expense for fiscal year 2026? | Property and equipment were $10,383 million, with depreciation expense of $2.4 billion for fiscal year 2026. |
D3 — Definition-then-use distance
Within a single filing. Again, a term or segment is defined once and used far away, and some sections are nothing but cross-references. Item 8 of an annual report, nominally the financial statements, contains one sentence redirecting the reader to Item 15. One document, provided the system follows the redirect instead of answering from the stub.
The five questions
| ID | Question | Gold answer |
|---|---|---|
| D3-01 | What is the name of the autonomous driving platform and what was the revenue growth for the Automotive market in fiscal year 2026? | The platform is DRIVE Hyperion. Automotive revenue grew 39% in fiscal year 2026. |
| D3-02 | What is the name of the AI software suite for enterprise applications and what was the total revenue for fiscal year 2026? | The suite is NVIDIA AI Enterprise. Total revenue for fiscal year 2026 was $215,938 million. |
| D3-03 | What is the name of the cloud gaming service and what was the Gaming revenue growth in fiscal year 2026? | The service is GeForce NOW. Gaming revenue grew 41% in fiscal year 2026. |
| D3-04 | What is the name of the data center architecture launched in fiscal year 2025 and what was the Data Center revenue growth in fiscal year 2026? | The architecture is Blackwell. Data Center revenue grew 68% in fiscal year 2026. |
| D3-05 | What is the name of the virtual world simulation software and what was the gross margin in fiscal year 2026? | The software is Omniverse. The gross margin in fiscal year 2026 was 71.1%. |
D4 — Granularity mismatch
Across two filings. In this case the question is annual while the evidence is quarterly, or the reverse. Answering means reconciling a 10-Q against a 10-K and noticing that a nine-month figure is not a full year. Retrieval that ranks on topical similarity will happily return the quarterly passage and stop there.
The five questions
| ID | Question | Gold answer |
|---|---|---|
| D4-01 | What was the total revenue for the first nine months of fiscal year 2026 and the full fiscal year 2026? | Nine-month revenue was $147,811 million and full-year revenue was $215,938 million. |
| D4-02 | What was the gross margin percentage for the first nine months of fiscal year 2026 and the full fiscal year 2026? | Nine-month gross margin was 69.3% and full-year gross margin was 71.1%. |
| D4-03 | What was the net cash provided by operating activities for the first nine months of fiscal year 2026 and the full fiscal year 2026? | Nine-month operating cash flow was $66,530 million and full-year operating cash flow was $102,718 million. |
| D4-04 | What was the Compute & Networking segment revenue for the first nine months of fiscal year 2026 and the full fiscal year 2026? | Nine-month Compute & Networking revenue was $131,828 million and full-year revenue was $193,479 million. |
| D4-05 | What was the total cash, cash equivalents, and marketable securities as of October 26, 2025 and January 25, 2026? | The balance was $60,608 million as of October 26, 2025, and $62,556 million as of January 25, 2026. |
D5 — Event then consequence
Across two filings. An 8-K announces an event, and a later periodic report says what it did. The April 2025 export-licence announcement is the only clean case in this corpus: it estimated a charge of up to $5.5 billion, a figure no later filing repeats, and the quarterly report that followed recorded $4.5 billion. Neither document states both numbers, which is what makes the pair mandatory.
The five questions
| ID | Question | Gold answer |
|---|---|---|
| D5-01 | What was the maximum estimated charge for H20 products announced in April 2025 and the actual charge recorded in the first quarter of fiscal year 2026? | The estimated charge was up to $5.5 billion, while the actual charge was $4.5 billion. |
| D5-02 | What was the maximum estimated H20 charge announced in April 2025, and what was the total H20 revenue for Q1 FY2026 prior to the new requirements? | The estimated charge was up to $5.5 billion; H20 revenue was $4.6 billion. |
| D5-03 | What was the estimated H20 charge in the April 2025 announcement, and what was the net unfavorable impact on gross margin in Q1 FY2026? | The estimated charge was up to $5.5 billion; the net unfavorable impact on gross margin was 11.0%. |
| D5-04 | What was the estimated H20 charge in the April 2025 announcement, and what was the total provision for inventory and excess purchase obligations in Q1 FY2026? | The estimated charge was up to $5.5 billion; the total provision was $5.3 billion. |
| D5-05 | What was the estimated H20 charge in the April 2025 announcement, and what was the net income for Q1 FY2026? | The estimated charge was up to $5.5 billion; the net income was $18,775 million. |
D6 — Governance vs operations
Across two filings. Notably, compensation, board composition and meeting mechanics appear only in the proxy statement, while the operating figures they pair with sit in the annual report. No amount of reranking inside a 10-K reaches them, because the answer is not in that document class at all.
The five questions
| ID | Question | Gold answer |
|---|---|---|
| D6-01 | Who is the Lead Director of NVIDIA and what was the company’s net income for fiscal 2026? | The Lead Director is Stephen C. Neal. The net income for fiscal 2026 was $120,067 million. |
| D6-02 | What is the name of the independent registered public accounting firm for fiscal 2027 and what was the total revenue for fiscal 2026? | The independent registered public accounting firm is PricewaterhouseCoopers LLP. The total revenue for fiscal 2026 was $215,938 million. |
| D6-03 | Who is the Chairperson of the Compensation Committee and what was the operating income for fiscal 2026? | The Chairperson of the Compensation Committee is Dawn Hudson. The operating income for fiscal 2026 was $130,387 million. |
| D6-04 | Who is the Chairperson of the Audit Committee and what was the gross profit for fiscal 2026? | The Chairperson of the Audit Committee is A. Brooke Seawell. The gross profit for fiscal 2026 was $153,463 million. |
| D6-05 | Who is the Chairperson of the Nominating and Corporate Governance Committee and what was the net cash provided by operating activities for fiscal 2026? | The Chairperson of the Nominating and Corporate Governance Committee is Stephen C. Neal. The net cash provided by operating activities for fiscal… |
D7 — Retroactive restatement
Across two filings. Specifically, the June 2024 ten-for-one stock split divided every per-share figure filed before it. Both the old value and the new one are correct inside their own document, so the system must know which filing supersedes which. Deduplication cannot resolve this, since neither number is an error.
The five questions
| ID | Question | Gold answer |
|---|---|---|
| D7-01 | What was the basic net income per share for the fiscal year ended January 28, 2024, when it was first published, and what is the restated value? | The initial report was $12.05, and the restated value is $1.21. |
| D7-02 | How did the diluted net income per share for the fiscal year ended January 28, 2024, change between its initial publication and the later restatement? | It changed from $11.93 to $1.19. |
| D7-03 | What was the basic weighted average share count for the fiscal year ended January 28, 2024, at the time of its first report, and what is the restated figure? | The initial count was 2,469 million shares, and the restated count is 24,690 million shares. |
| D7-04 | Compare the diluted weighted average share count for the fiscal year ended January 28, 2024, as originally reported versus the restated amount. | The original report showed 2,494 million shares, while the restated amount is 24,940 million shares. |
| D7-05 | What was the cash dividend per common share declared and paid during the fiscal year ended January 28, 2024, when first disclosed, and what is the restated rate? | The initial disclosure was $0.16 per share, and the restated rate is $0.016 per share. |
D8 — Absence
In this group, nothing in the corpus answers the question, so refusal is the only correct response. Each one is built from material that looks related without supporting an answer, which means a system matching on topic alone will respond confidently and wrongly. These questions measure restraint rather than recall.
The five questions
| ID | Question | Gold answer |
|---|---|---|
| D8-01 | What is the specific dollar amount of the investment NVIDIA is finalizing with OpenAI? | The corpus does not contain the specific dollar amount of the investment with OpenAI. It only states that NVIDIA is finalizing an investment and… |
| D8-02 | How many employees did NVIDIA have in the United States as of the end of fiscal year 2026? | The corpus does not contain the number of employees in the United States. It only states that NVIDIA had approximately 42,000 employees in 38… |
| D8-03 | What was the exact revenue generated from the H200 licensing program in fiscal year 2026? | The corpus does not contain the exact revenue figure for the H200 licensing program. It states that NVIDIA has not generated any revenue under this… |
| D8-04 | What was the specific revenue contribution from the AI research and deployment company mentioned in fiscal year 2026? | The corpus does not contain the specific revenue contribution from that company. It only states that the company contributed to a meaningful amount… |
| D8-05 | What was the specific dollar amount of the $3.5 billion in land, power, and shell guarantees allocated to each partner? | The corpus does not contain the allocation of the $3.5 billion guarantee to individual partners. It only states the total amount provided to… |
The bias in an LLM-built RAG golden set
The same model extracts the questions and, from the next episode onward, judges the answers. That is a real bias risk and it deserves stating plainly rather than burying.
Three things limit it. Gold answers are anchored to quoted source spans rather than to the model’s opinion, so a judge that drifts can be checked against the filing. Ten machine gates and one human pass stand between a proposal and the set. And the judge never learns which framework produced an answer.
What none of that fixes is a systematic blind spot: questions this model finds natural to write may also be ones it finds natural to grade. The abstention numbers deserve particular caution for that reason, which is why the absence pattern is reported separately from the rest.
What synthetic benchmarks are known to get wrong
Our result sits inside a documented problem. Research on synthetic RAG evaluation reports that LLM-generated question sets tend toward simple, single-constraint queries, and that they rarely stop the model from writing questions it can already answer from memory rather than from retrieval. A benchmark built that way measures the generator, not the retriever.
Two design choices push against that. Targeting a dispersion pattern rules out single-constraint questions by construction, and the idle-document test makes a second filing mandatory rather than decorative on half the set. Nevertheless, the caution in Can we Evaluate RAGs with Synthetic Data? applies: synthetic sets are more trustworthy for tuning retrieval than for ranking generators. Since this series holds the generator fixed and varies only the framework, that is exactly the use they support.
Reproduce this RAG golden set pipeline
# 1) Propose questions for one pattern, with the prompt version that won the A/B test
python benchmark/golden/extract_questions.py --pattern D1 --n 28 --prompt-version v3
# 2) Can this pattern fill its five slots, and what killed the rest?
python benchmark/golden/candidate_yield.py
# 3) Select five per pattern: quotes verified, contract passed, answer supported,
# near-duplicates dropped
python benchmark/golden/rebuild_patterns.py --patterns D1 D2 D3 D4 D5 D6 D7 D8 --fresh
# 4) The gates, in the order they run
python benchmark/golden/pattern_conformance.py # 40/40 or it is not a set
python benchmark/golden/verify_answers.py # every number traced to evidence
python benchmark/golden/adversarial_check.py # rules plus a model told to find fault
python benchmark/golden/absence_review.py # the one gate that needs a person
python benchmark/golden/freeze_v1.py # refuses to stamp a set that fails
# 5) Calibrate difficulty with the naive baseline, through the shared harness
python benchmark/harness/run_benchmark.py --adapter articles/E-02-golden/adapter.py
--episode E-02 --runtime native
python benchmark/evaluate/evaluate.py --run production/raw-results/E-02/naive-rag_run.jsonThe whole pipeline ships as one archive, together with the converted corpus it reads: download the golden-set pipeline (ZIP). It contains the 33 filings already converted to Markdown with their tables intact, the frozen question set, every prompt exactly as it was sent, and the raw model output behind every number in this article. No EDGAR download is needed to reproduce the result.
Frequently asked questions
Can an LLM build its own RAG golden set?
It can propose one. It cannot approve one. Of 214 proposals, 25 cited evidence that does not exist and 72 more did not exhibit the pattern they were written for. What makes the surviving 40 usable is that every one of those failures is detected by a script rather than by trust.
How many questions does a RAG benchmark need?
Forty was enough to separate behaviour here, because the set is balanced: five questions for each of eight dispersion patterns, and an even split between single-document and two-document answers. Balance matters more than volume.
Does a better prompt fix a bad benchmark question?
Partly, and only where a rule cannot. Prompt refinement moved acceptance from 46.9% to 71.9% overall, but on the patterns whose requirement is enforced in code it changed nothing at all. It also raised the fabrication rate in two patterns from under 2% to about 40%, so the filter has to stay regardless.
Why not fill the whole context window when extracting questions?
Because it makes results worse. At 71,000 tokens the model spent its entire generation budget reasoning and returned nothing; at 36,000 tokens the same pattern produced seven verified questions in 52 seconds.
Is refusing to answer a good sign?
Only when the corpus genuinely lacks the answer — and checking that is harder than it sounds. Four proposed absence questions declared a figure missing that the filings state in a table row. A benchmark that ships those punishes systems for being right, so every refusal question in this set was read by a person against the corpus before the freeze.
Does the same model judging its own questions invalidate the benchmark?
It is a real risk, mitigated rather than removed. Gold answers are anchored to verbatim source quotes, ten rules and a person stand between a proposal and the set, and the judge never learns which framework wrote an answer.
Summary
The NVDA-RAG-40 golden set holds 40 questions, five for each dispersion pattern, split evenly between single-document and two-document answers. Building it took 214 model proposals: 189 survived quote verification, 142 exhibited the pattern they were written for, and 40 were selected.
A naive baseline puts the set in context. It retrieves everything it needs for 18 of 20 single-document questions and 10 of 20 two-document questions, and it scores zero on the restatement pattern, where the two filings it must combine are near-duplicates a vector search cannot tell apart. That gap is what the next seven episodes exist to close, starting with LightRAG.
Related reading on how retrieval fails on long financial documents: Decomposing Retrieval Failures in RAG for Long-Document Financial Question Answering. The filings themselves are public at SEC EDGAR.
Companion code — the fragments above are the working parts: the prompt template, the anchored excerpt window, the tolerance floor and the idle-quote test. They ship with the converted corpus in nvda-rag-e02-golden-set.zip, so the commands above rebuild the golden set offline.
Free ebook
Free AI Video, Generated Locally
Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.
No spam. Unsubscribe at any time.
Leave a Reply