Document chunk boundaries
Measure correct passages interpreted the wrong way
Retrieval can find the right words and still produce the wrong conclusion. Add interpretation checks for the context that document splitting can remove.
In this article
Define the facts that must travel together
Choose a set of questions whose answers depend on a relationship in the source. A number may depend on a unit. A recommendation may depend on a condition. A date may identify the period covered by a table rather than the date the report was published.
For each question, record the necessary evidence components. This is more precise than labelling a whole page as relevant. It lets the test distinguish finding the right row from finding the row with the labels needed to interpret it.
Keep the expected answer narrow. If the test asks for a quantity and its scope, the judgement should say whether those facts are correct. A general rating of how helpful the answer sounds can miss a specific unit error.
Separate retrieval, context and answer errors
Classify a failure according to where the needed information was lost. The source may be missing from the candidates. The correct passage may be selected without its heading or note. Or the complete evidence may reach the model and still be interpreted incorrectly.
These categories point to different work. Candidate failures call for retrieval investigation. Missing context calls for extraction, splitting or assembly changes. Incorrect interpretation with complete evidence may require a different answer method, clearer input representation or a human review path.
A simple result record can carry the question identifier, required evidence components, selected passage identifiers and the observed error category. Use a restricted evidence store if the fixture contains material that cannot be shared broadly.
Compare chunking changes against the same fixture
When testing a new split strategy, keep the document versions and questions fixed. Measure how often the selected evidence contains all required components. Then measure how often the final answer gets the fact and its qualification right.
Report results by structure type. Paragraphs, nested lists, tables, footnotes and scanned pages can respond differently to a change. An overall improvement dominated by easy prose can conceal a regression on the tables the business relies on.
Inspect cases where a larger chunk improves interpretation but introduces conflicting material. More context is not always better. A passage containing both an old rule and its later replacement can make an answer less reliable unless their relationship is clear.
Use the results to choose a bounded rule
Suppose repeating table headers fixes several unit errors without increasing unrelated content much. That is evidence for a specific structural rule. It is not evidence that every paragraph should receive a larger overlap window.
Record the extra indexed text and answer-context size created by the change. Quality and cost belong in the same decision, but keep them as separate measurements. Do not turn them into a weighted score that nobody can explain.
Finish the evaluation with a few source-to-passage-to-answer examples. They make the failure mechanism visible to people who do not work on the parser and give the implementation team a clearer target than an instruction to improve chunk quality.
Primary sources
Microsoft Learn: chunking strategies and contextMicrosoft Learn: structured document extractionReferences checked 11 September 2026.