Make the cheap model retry until it costs more

Use a failure that retries cannot repair

Create a synthetic task whose candidate consistently returns an output incompatible with the required contract. For example, the fixture can require a field the supplied evidence does not contain, while the repair loop keeps demanding a complete value.

The correct product behaviour should be to report missing evidence or escalate, not to generate indefinitely. Keep the test isolated and use bounded provider or test-double responses so it cannot create an uncontrolled bill.

Record the baseline configuration's accepted outcome and total work. The purpose is to compare completed-task behaviour, not merely prove that one model has a lower listed price.

Follow every repair attempt

Inspect model calls, validator results and retry decisions under the same task identifier. Confirm failed attempts consume the relevant task limits and do not reset the allowance.

Check whether the retry input changes meaningfully. Repeating an identical request after a permanent validation failure is often wasted work. A useful repair path should distinguish transient service failure from missing evidence or an impossible contract.

Count evaluator calls as well. A loop that generates and judges every candidate can cost more than the generation trace alone suggests.

Add concurrency and interruption

Start two repair workers against the same task and verify they cannot both reserve the last available capacity. Then stop a worker after dispatch but before usage settlement.

On restart, the application should find the unresolved reservation and reconcile it. Creating a new budget record or releasing the old reservation optimistically can allow more work than intended.

Deliver a duplicate usage callback to verify settlement occurs once. This tests accounting correctness independently of model quality.

Inspect the stopping result

Confirm the application prevents further calls when the limit is reached and retains any useful intermediate result. The user should receive a clear explanation of the unresolved requirement rather than another invitation to retry the same impossible task.

Compare total calls, latency and estimated or confirmed cost with the baseline under the same acceptance rule. Report the assumptions behind any currency calculation. The test demonstrates whether cheaper routing is actually economical and whether the budget control contains the failure when it is not.