Secrets rotation

New connections fail after a rotation

Existing sessions can hide a stale credential until the pool reconnects. Compare target validity, secret version and consumer refresh before rotating again.

In this article

Identify the failing connection path

Determine which consumer is failing and whether failures affect new connections, existing sessions or both. Record the timing relative to rotation.

Check the secret reference and safe version metadata used by the process. Do not copy the credential into logs or incident chat to compare it manually.

Separate authentication failures from network, certificate and permission errors. They need different recovery actions.

Verify the target and published version

Use the approved diagnostic path to test the intended credential with a safe operation. Confirm that the secret-store version and target state agree.

If rotation stopped partway through, inspect that state before restarting it. Another rotation can make the mismatch harder to understand.

Check the automation's authority and access to the target, especially after network or role changes.

A useful diagnostic sequence compares three non-secret facts: the target credential version expected by the rotation workflow, the version currently published by the secret store and the version the process last loaded. If the first two agree and the process is older, consumer refresh is the likely missing step. If the first two disagree, restarting every consumer may spread the failure.

Test with a newly opened connection because an existing authenticated database session can continue working. A health check that reuses that session may remain green until the pool eventually recycles.

Refresh the affected consumers

Follow the application's supported refresh or restart procedure and confirm a newly established connection works. Existing pooled sessions are not sufficient evidence.

Apply bounded recovery so all instances do not reconnect simultaneously and overwhelm the target or secret service.

Inspect queued and in-flight operations. Retry only under the operation's safe recovery policy, preserving identity for work with uncertain outcomes.

Complete or repair the transition

Verify other consumers, including scheduled jobs and recovery environments. The visible application may be only one user of the credential.

Retire temporary access and confirm the previous credential's intended status. Do not restore a credential suspected of compromise simply to remove authentication errors.

Record the missing refresh behaviour or partial-state failure and add it to the drill. The incident is resolved when new connections and the relevant business operations work under the intended replacement authority.

Primary sources

AWS Secrets Manager: rotation strategies

References checked 11 September 2026.