Model change releases
Require a reproducible release comparison
A reviewer should be able to identify the candidate, rerun its important cases and understand every material difference from the baseline.
In this article
Inspect the release manifest
Ask for model reference, prompt version, tool schemas, retrieval configuration and validators. Confirm the evaluation report points to those exact artifacts rather than mutable development settings.
Record any provider alias or service behaviour that prevents exact replay. Reproducibility can have practical limits, but those limits should be visible. Retained inputs, outputs and configuration still help explain the original decision.
Check that secrets and production payloads have not been bundled into a broadly accessible release artifact.
Examine the comparison set
Verify both configurations ran on the same task and source versions. Inspect category coverage, held-out cases and known regression fixtures. Ask which examples influenced prompt tuning and which assess unfamiliar work.
Look at failed and timed-out runs as well as scored responses. Confirm the report does not silently drop inconvenient cases or select the best of several attempts for one candidate.
Review changed outcomes directly. A small set of important regressions can matter more than a higher overall score, especially for field extraction, access boundaries and external actions.
Test the restoration path
Have the team load the prior configuration and run an essential task. Confirm compatibility with current tool schemas and paused workflow state. A configuration file that no longer points to an available endpoint is not a usable rollback.
Inspect cohort routing, caches and in-flight work. The release should have a defined way to stop new effects and resolve tasks already accepted.
Check observability for the affected behaviours. Generic uptime metrics will not reveal that the model now chooses the wrong date field while returning valid JSON.
Record the decision and its limits
The review should state which tasks improved, which regressed and why the remaining behaviour is acceptable. Include the evidence owner and the conditions that trigger restoration or further restriction.
Keep the report concise but reproducible, with links to cases and traces rather than pages of unexamined outputs. A release comparison is complete when another engineer can follow the evidence and repeat the important checks without reconstructing the original team's environment from memory.
Primary sources
Microsoft Learn: evaluation and observabilityMicrosoft Learn: retrieval and answer evaluatorsReferences checked 11 September 2026.