Ask reviewers to inspect refusals and escalations

Look beyond successful answers

Select legitimate tasks the system is expected to complete and inspect cases where it refuses or asks for help. Determine whether information is genuinely missing, authority is absent or the candidate is avoiding an answer it should provide.

A refusal can be the correct result for an unsupported or unauthorised request. The same wording can be a product failure when the necessary evidence and permission are present. The dataset needs examples of both.

Check whether the evaluation score rewards silence too generously. A groundedness measure alone may favour a response that makes no factual claims, even when it fails the user's task.

Review the escalation package

When the assistant refers a task to a person, inspect what information accompanies it. The recipient needs the unresolved question, relevant evidence and work already completed. A vague "manual review required" message transfers effort without making the next step easier.

Verify that escalation does not expose data to an unauthorised reviewer. The handoff is another access path and should follow the same target and data rules as the main application.

Measure whether the person can resolve the case from the package or must repeat the entire investigation. This affects the practical value of a seemingly safe escalation strategy.

Inspect dataset balance

Ask how many cases require an answer, clarification, refusal or approval. Confirm that the mix reflects both ordinary use and important boundaries. A set dominated by unsupported requests can make a hesitant model appear ideal for a product meant to complete useful work.

Check the reasons behind expected labels. A case should not require refusal merely because the current model happened to refuse it during development.

Include ambiguous requests that become answerable after one clarification. The product should preserve the user's task through that exchange rather than treating every ambiguity as a terminal failure.

Compare changed outcomes

Review tasks that the baseline completed but the candidate now declines, and the reverse. Determine whether the change improves authority handling, reveals a regression or reflects a revised product scope.

Record acceptance criteria for useful completion and appropriate escalation separately. A release comparison is stronger when it shows not only that answers are correct, but that the system knows when to answer, when to ask and when another person is actually needed.