Investigate disagreement between human evaluators

Preserve independent judgements

When two reviewers disagree, keep their original labels and reasons. Do not immediately replace them with a majority vote or the more senior person's preference. The disagreement itself is useful evidence about the task or rubric.

Confirm both reviewers saw the same input, source version and answer. Different context can produce reasonable but incompatible decisions. A missing appendix or a changed policy may explain the result before any scoring debate begins.

Check whether the question has more than one plausible interpretation. If so, the expected behaviour may need to include clarification rather than one supposedly obvious answer.

Identify the disputed property

Ask whether the disagreement concerns factual correctness, completeness, relevance, tone or permitted action. Reviewers often use the same overall label while judging different properties.

For example, one reviewer may mark an answer correct because every sentence is supported. Another may reject it because the answer omits a critical exception. Both observations can be valid if the rubric never separated support from completeness.

Rewrite the criterion in terms of observable behaviour. Specify the required claim or prohibited effect, while allowing harmless differences in wording.

Adjudicate with the appropriate owner

Bring in the source or business-rule owner when the underlying requirement is unclear. Engineering should not settle a policy question by choosing whichever label makes the model score higher.

Record the resolution and reasoning. If the source itself is ambiguous, mark the fixture unresolved or revise it after the authoritative material is corrected. Do not retain a forced label that future reviewers cannot justify.

If the answer exposes a valid edge case, add it to the rubric examples. This helps later reviewers apply the same distinction without repeating the entire discussion.

Recheck affected results

Find other cases scored under the ambiguous criterion and reassess the relevant subset. Version the rubric change so results from before and after are not presented as directly comparable without explanation.

Measure disagreement by category over time. Repeated disputes may indicate that the product's expected behaviour is underspecified, not that reviewers need more reminders. A useful evaluation process improves both the labels and the underlying task definition when disagreement reveals a real gap.