# Interrupt the run on both sides of a commit

Recovery tests need precise fault locations. Stopping a process at a random time rarely proves how it handles the gap between an external effect and its local record.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Prepare one observable mutation

Use a synthetic order and a tool that adds a uniquely identifiable note. Record the run, proposal and operation identifiers. The expected final state is one note on the correct order, with one completed operation record.

Add test hooks around the boundaries you need to exercise. A controlled pause before dispatch and another after downstream commit make the test reproducible. Keep these hooks isolated from normal production behaviour.

Use a downstream test service that can record accepted requests and actual effects separately. This lets the test distinguish harmless repeated attempts from duplicate mutations.

## Stop before the request is sent

Persist the operation intent, stop the worker before dispatch and restart it. The resumed worker should find the existing pending operation and execute it using the same identifier.

Assert that it does not create a second proposal or ask the model to invent a replacement payload. The user's task and approval should remain tied to the saved operation, subject to current eligibility checks.

This case establishes that a durable intent survives process loss. It does not yet test uncertain outcomes because no external effect occurred before the interruption.

## Stop after the target changes

Now allow the downstream mutation to commit, then terminate the worker before the local result is saved. On restart, the application should recognise uncertainty and reconcile the existing operation.

If the downstream contract supports idempotent replay, verify the same identifier and payload are used. If recovery relies on a status lookup, confirm that lookup identifies this operation rather than merely observing a coincidentally matching value.

Deliver the original response late after recovery completes. The final state should remain consistent and should not trigger another step twice.

## Add competing recovery workers

Start two workers against the same interrupted run. Check the ownership mechanism and any stale-worker protection. A local lease can expire while an old process is still alive, so test that the old owner cannot overwrite newer progress.

Repeat with a permission or approval change during the pause. Completed work remains part of history, but the next mutation must follow the documented current-authority rule.

## Inspect business state, not just run status

A run marked completed can still have produced duplicate notes. Count effects in the target service and compare them with operation records. Verify the final user message reflects the actual outcome.

Keep each interruption point as a named regression case. When workflow libraries, queues or retry policies change, these tests provide evidence about the boundaries that matter rather than a generic claim that restart testing passed.

## Sources

- [AWS Builders' Library: idempotent APIs](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/)
- [PostgreSQL: explicit locking](https://www.postgresql.org/docs/17/explicit-locking.html)