# Compare candidate and baseline on identical tasks

A fair release comparison holds the task and evidence steady, then examines changed outcomes. Separate quality, useful completion and operational cost.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Pair each result with the same case

Run the current configuration and candidate against the same dataset version and source snapshot. Pair results by case identifier so reviewers can inspect exactly what changed.

Keep retrieval or tool outputs fixed when isolating generation behaviour, then run an end-to-end comparison for the complete application. These answer different questions. A fixed-context test can reveal model differences, while the full path includes retrieval variability and integration effects.

Record failed runs and timeouts. Dropping them from the candidate's sample makes the comparison unfair and can hide an operational regression.

## Report transitions between outcomes

Count cases that improved, regressed, stayed correct and stayed incorrect. A net score alone can conceal a trade in which several easy improvements offset one important new failure.

Review regressions by task and consequence. For structured extraction, compare field-level correctness. For an agent, inspect target selection and actual effects. For question answering, separate support from completeness and relevance.

Include refusal and escalation transitions. A candidate that declines more tasks may reduce unsupported answers while reducing the product's usefulness.

## Measure the full request cost

Record end-to-end latency, model usage, retries, validation failures and human intervention. Compare equivalent successful tasks rather than only initial model responses.

Use the same concurrency and workload conditions when measuring performance. A quiet candidate endpoint and a busy baseline endpoint can produce a misleading latency comparison.

Report distribution and sample size rather than only an average. A slower tail can matter to interactive users even when the typical response improves. Where provider behaviour varies, repeat enough runs to understand the uncertainty relevant to the release decision.

## Make the acceptance rule visible

Set criteria before selecting the winner. Some categories may require no observed prohibited effects in the tested cases, while ordinary answer quality can use an agreed tolerance. Keep deterministic authority tests separate from model scores.

Publish a concise comparison with examples of material changes and known limitations. The decision should explain why the candidate is acceptable for the intended audience, not merely that one number is higher. Preserve the paired results so a later production report can be investigated against the same evidence.

## Sources

- [Microsoft Learn: evaluation and observability](https://learn.microsoft.com/en-us/azure/foundry/concepts/observability)
- [Microsoft Learn: retrieval and answer evaluators](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators)