An emergency cloud fix needs a path back into code

Three records can disagree

An infrastructure definition describes intended resources. The management tool keeps state about what it manages. The cloud provider holds the actual resources. These can diverge after a console edit, failed deployment or change made by another controller.

Consider an incident responder narrowing a network rule to contain unwanted traffic. The checked-in definition still contains the broader rule. An automatic apply can undo the containment while appearing to restore consistency.

The important question is which state is now intended. Drift detection identifies a difference, but the service owner must decide whether to retain, modify or reverse it.

Separate observation from reconciliation

Run detection through a controlled identity and preserve enough evidence to understand the change. Compare the actual resource, declared configuration and relevant change history.

Terraform's refresh-only planning can help inspect changes to recorded state without proposing the usual configuration reconciliation. Updating state alone does not rewrite the configuration to adopt a manual change.

Define which controller owns each field

Some values are intentionally managed by another system, such as an autoscaler. If two controllers repeatedly set different values, the result is an ownership conflict rather than an isolated manual mistake.

Document those boundaries and configure the infrastructure workflow accordingly. Ignoring a field can be appropriate when another maintained controller owns it, but it should not become a blanket way to hide unexplained changes.

Detection tools also have coverage limits. A clean report applies to the resources and properties actually checked, not every aspect of the environment.

Review the effect of correction

A seemingly small difference can require replacement, restart or loss of an address. Inspect the plan and the provider's behaviour before applying it.

For stateful resources, establish data protection and recovery appropriate to the proposed action. A corrected configuration does not compensate for a database recreated with no usable data.

Keep the actual plan tied to the reviewed code and current state. A plan produced before another incident change may no longer describe the action the operator expects.

Close the loop after incidents

Record emergency changes while they happen, then reconcile them into the maintained definition or remove them deliberately. Assign an owner and a completion condition.

The objective is a live environment whose intended state can be reproduced and explained. A zero-drift dashboard is useful only when it reflects that understanding, rather than suppressing differences or repeatedly undoing necessary work.