Keep sensitive examples out of unnecessary review systems

Trace an example through the evaluation path

A support conversation copied into a fixture may include names, account details and internal notes unrelated to the behaviour under review. Once submitted to an evaluator, that material may appear in traces, scoring explanations and exported reports.

Map each destination before importing production examples. Identify which systems receive raw inputs, retrieved context and outputs, and which people can inspect them. An evaluation platform is another data-processing path, not automatically a safe extension of the application.

Use synthetic cases for mechanisms such as permission boundaries, exceptions and retries where real data is unnecessary. A fictional record can often reproduce the defect precisely.

Preserve meaning while reducing exposure

When a real example is needed, remove unrelated fields and replace identifiers consistently. Check that the transformation has not changed the property being tested. An address-format defect may depend on punctuation or locality structure, so careless replacement can erase the failure.

Keep the original under the appropriate restricted process if it must be retained, and link the sanitised fixture through controlled metadata. Do not place the original in a repository merely to make the example easier to reproduce.

Review free-text fields carefully. Removing obvious names does not guarantee that a distinctive incident or combination of details is no longer identifying.

Apply access rules to outputs too

Evaluator explanations can quote the sensitive input. Logs and reports therefore need the same review as fixture storage. Limit reviewer access by the actual task and data classification.

Set retention for runs, traces and exports. A deleted fixture may remain in old evaluation reports or local downloads. Document those copies and provide a supported removal process where required.

Keep credentials and production authorisation tokens out of fixtures entirely. Tool tests should use isolated identities and controlled effects rather than replaying live access.

Verify the reduced fixture still works

Run the original failure mechanism against the sanitised or synthetic case and have an appropriate reviewer confirm the expected behaviour. If the defect disappears, inspect which removed detail mattered and retain only the minimum necessary structure.

Record provenance, permitted use and retention with the case. This allows future maintainers to reuse it responsibly without reconstructing the original data decision. A good evaluation set contains enough reality to test the product while avoiding unnecessary copies of people's information.