Find the development example hiding in the test set

Start with a suspiciously easy result

Suppose a prompt change produces a large score increase, but most improved cases ask about the same policy paragraph used during development. The wording differs, yet the task and required answer are nearly identical.

This does not make the cases useless. They can verify that the known behaviour remains stable. It does mean the score offers limited evidence about new policies, unfamiliar exceptions or different task types.

Build an inventory linking each case to its source, scenario and origin. Record whether it was used in demonstrations, prompt tuning, fine-tuning or earlier manual debugging.

Search for related cases

Check exact duplicates first, then inspect near-duplicates by shared source passages, entities and expected claims. Automated similarity can help prioritise review, but a human should decide whether two cases exercise meaningfully different behaviour.

For example, asking the same threshold question with a different employee name is probably closely related. Asking about a nested exception in another section may test a different mechanism even if much of the vocabulary overlaps.

Review generated examples carefully. A generator can produce many fluent variations while preserving the same answer structure and source dependency. Counting each as independent coverage exaggerates the breadth of the dataset.

Split at the level you want to generalise

If the goal is performance on unseen documents, hold out entire documents or document families. If the goal is unfamiliar customer scenarios, group related scenario variants together. The split should match the claim being made about the system.

Keep a separate known-case regression suite rather than deleting valuable development examples. Label its purpose honestly. A release can require both stable known behaviour and acceptable performance on held-out work.

Do not continually inspect and tune against the held-out set after every small change. Once it becomes part of the optimisation loop, it no longer serves the same role.

Re-run and explain the difference

Compare results before and after grouping related cases. A lower score may be a more realistic estimate, not evidence that the application suddenly got worse. Report the changed dataset composition with the result.

Add metadata checks so future imports retain origin and source relationships. Periodically review examples promoted from incidents into the regression suite. The aim is to know what the evaluation demonstrates: familiarity with known cases, robustness to variants or performance on genuinely different work.