# Choose a release cohort that includes difficult work

A small rollout is useful only if it exercises the behaviours you need to observe. Select the cohort by tasks and risk, not just by a random fraction of traffic.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Define what the first cohort should reveal

Identify the task categories most likely to change: ambiguous extraction, long context, tool selection, refusals or multi-step recovery. Choose a controlled audience and workload that includes those cases without exposing an unjustified level of consequence.

A cohort containing only short, clear questions may confirm basic availability while missing the reason for the evaluation. Conversely, putting the most consequential mutations into the first unproven cohort may create unnecessary exposure.

Use proposal-only or controlled test execution where it provides useful evidence, and label its limits. A dry-run proposal does not establish the reliability of a real downstream write.

## Route consistently and record the choice

Associate each task with the release configuration selected at its start. For multi-turn work, decide whether the conversation remains on that configuration. Switching models mid-task without a rule makes comparisons and incident diagnosis harder.

Record cohort assignment in internal traces. Keep it out of unnecessary user-facing clutter, but make it available to support and evaluation tooling.

Ensure baseline and candidate use comparable source and tool conditions. Otherwise the cohort may measure a routing or data difference rather than the intended model change.

## Watch outcomes with a stopping rule

Monitor task completion, material errors, policy denials, uncertain effects, latency and recovery work. Use reviewed samples to inspect errors that automated metrics may miss.

Set stopping and expansion criteria before observing the results. A clear prohibited effect may justify immediate containment even when the sample is small. Ordinary quality variation may need more evidence before a decision.

Give the team a tested route to restore or restrict the capability. Include cached answers and in-flight work in that plan rather than treating traffic routing as the entire rollback.

## Expand coverage deliberately

After the first cohort meets its criteria, add task types or audiences that were not yet represented. Keep an eye on distribution changes. Performance on internal expert users may not transfer directly to people who phrase requests differently.

Retain the prior configuration through the agreed observation period, then retire it only when compatibility and recovery requirements permit. The rollout should produce evidence about the real range of work, not simply a growing percentage beside a release flag.

## Sources

- [Microsoft Learn: evaluation and observability](https://learn.microsoft.com/en-us/azure/foundry/concepts/observability)