Version the evaluation set alongside the release

Treat the dataset as a maintained artifact

Assign a version to the case collection and retain a manifest of case identifiers, source snapshots and rubric versions. Each evaluation run should reference that manifest rather than a mutable folder called "latest".

Record why cases are added, corrected or retired. A new product capability may need additional tasks, while a policy correction may invalidate an old expected answer. These are legitimate changes, but they affect how scores can be compared.

Keep source dependencies stable where possible. A fixture pointing at a live document can change meaning between two runs even when the dataset file itself is unchanged.

Separate corrections from coverage expansion

If a label was wrong, correct it and identify prior results affected by the error. Do not describe the resulting score increase as an application improvement. If new cases broaden coverage, report their results separately during the transition.

Retain a common subset between versions where useful. Running both candidate and baseline on that subset helps distinguish system change from dataset composition. The full new set then establishes the broader release evidence.

Avoid silently removing difficult cases because a candidate fails them. Retire a case only when its task is no longer relevant, its evidence is invalid or it duplicates coverage deliberately retained elsewhere.

Update the pipeline and review process

Make release jobs record dataset, source, evaluator and application versions together. Validate missing references and duplicate identifiers before spending money on model runs.

If the schema changes, migrate fixtures with explicit field mappings and review any meaning that cannot be preserved automatically. A field renamed from "expected answer" to "required claims" may require editorial judgement, not just a string replacement.

Keep old reports readable. Reviewers should be able to inspect the case and rubric that produced a historical result even after the current dataset evolves.

Close the comparison honestly

For the first release using the new set, explain which metrics are comparable and which establish a new baseline. Include category counts and examples of new failure coverage.

Archive obsolete source snapshots according to their data rules, retaining enough metadata to explain historical results. A versioned evaluation library gives the team a stable record of what was tested and why, so a rising score can be interpreted rather than merely celebrated.