# Make the canary fail before trusting automatic promotion

Test the release controller with bad results, missing results and a healthy control group. The exercise should prove the decision path, not just deployment mechanics.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Start with a measurable synthetic fault

In a safe environment, make the candidate return a controlled error for a known test operation while the stable release remains healthy. Route a defined cohort through both paths.

Verify that telemetry identifies the candidate correctly. If metrics combine both versions, the healthy stable traffic can dilute the failure and conceal it from the analysis.

Use a fault with no harmful external effect. A deliberately rejected synthetic request is sufficient to test many control paths.

## Observe the abort action

Confirm that the configured failure condition stops expansion and performs the intended traffic or capacity action. Inspect the actual request path rather than only the controller's status.

Check requests already in flight. Aborting a rollout does not necessarily cancel work that has begun, so the application still needs safe operation handling.

Verify that stable capacity can accept the returned load without a second incident.

## Remove the evidence source

Repeat with the metrics query returning no observations, then with the telemetry provider unavailable. Confirm that these states follow the documented hold or review policy.

A zero error count with zero relevant requests should not be confused with a demonstrated healthy release. Record the denominator and observation window.

Test delayed metrics too. If the analysis reads data from before the candidate received traffic, it may appear healthy for the wrong reason.

## Exercise recovery after a committed effect

Use a synthetic record written by the candidate before the abort. Read and update it through the stable release after traffic returns.

If the candidate changed the data shape or business state in an incompatible way, routing rollback will expose the problem. Resolve that compatibility requirement before production.

Finish with a healthy run to verify that the policy can promote under sufficient evidence. The full exercise should show a correct failure decision, an honest inconclusive state and a successful progression, each tied to observable request behaviour.

## Sources

- [Argo Rollouts: canary deployments](https://argo-rollouts.readthedocs.io/en/stable/features/canary/)
- [Argo Rollouts: analysis](https://argo-rollouts.readthedocs.io/en/stable/features/analysis/)