# Spend expert review on the cases that separate candidates

Human judgement is most valuable where correctness is consequential or automation is uncertain. Use deterministic checks and targeted sampling to make that time count.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Remove mechanical work first

Check schema validity, required identifiers and exact numeric fields in code where the contract permits it. An expert should not spend time deciding whether a returned order identifier matches the expected one when the test runner can establish that directly.

Keep semantic review for properties such as whether a policy exception was preserved or an explanation justifies a recommendation. Model-based evaluators can assist, but they need calibration against reviewed cases and should not obscure their own uncertainty.

Separate harness failures from answer failures before sending work to reviewers. Broken source references and timeouts create noise that consumes attention without helping compare candidates.

## Review disagreements and consequential cases

Prioritise cases where baseline and candidate differ, where evaluators disagree and where a failure would change an important action. Include a random sample of apparently passing cases so systematic false passes do not remain invisible.

Do not review only failures reported by the same automated judge you are trying to validate. That selection misses errors the judge consistently accepts. A mixed sampling strategy provides a better picture.

Present the input, relevant evidence and disputed property together. Reviewers should not have to search through a long trace to discover which sentence or tool target needs judgement.

## Estimate the real workload

Measure review time by case type, including adjudication. A short classification case and a multi-document answer can require very different effort. Use those observations to plan release review rather than multiplying all cases by one optimistic average.

For repeated evaluations, retain stable labels and reasoning where the source and task remain unchanged. Re-review when a rubric, source or consequential behaviour changes. Blindly relabelling every familiar case wastes expertise, while never revisiting labels allows stale assumptions to persist.

Track how often reviewers need additional information. That often points to incomplete fixtures or unclear product expectations.

## Keep coverage from narrowing too far

A small discriminating set is useful for rapid iteration, but it should not become the only release evidence. Maintain broader coverage for common tasks, rare boundaries and new capabilities.

Use different suites for quick development feedback and more thorough release assessment, with their purposes clearly stated. Compare the cost of review with the consequences of missed failures and the frequency of releases. The aim is a repeatable process that uses expert judgement where it changes the decision, rather than an expensive ritual of reading every answer the same way.

## Sources

- [Microsoft Learn: evaluation and observability](https://learn.microsoft.com/en-us/azure/foundry/concepts/observability)
- [Microsoft Learn: retrieval and answer evaluators](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators)