Timeout after the write, before the tool response
The hardest retry case is a successful mutation whose response never reaches the agent. Inject that failure deliberately and inspect the business record.
Read articleCobnex blogs / Australia
Cobnex blog
AI implementation, software architecture, cloud operations and Australian technology policy.
150 articles in AI engineering
Page 3 of 9
Change one field after the reviewer approves
A small payload mutation is enough to test whether approval is bound to the action. Use a material field and verify that execution cannot reuse the earlier decision.
Read articleInterrupt the run on both sides of a commit
Recovery tests need precise fault locations. Stopping a process at a random time rarely proves how it handles the gap between an external effect and its local record.
Read articleFind the development example hiding in the test set
Near-duplicate cases can make an evaluation look independent when it is testing familiar material. Inspect relationships between examples, not just identical strings.
Read articleUse a fluent response with the wrong extracted field
Schema checks will accept many semantically wrong values. Test a document where nearby fields are easy to confuse and assert the business meaning of each output.
Read articlePut subtotal and payable total next to each other
Test field selection with several plausible amounts in the same region. A parser that reads the digits correctly can still choose the wrong business value.
Read articleTrigger an error and inspect where its payload goes
Error paths often log more than successful requests. Use a synthetic marker to find diagnostic copies and unexpected fallbacks without exposing real information.
Read articleMake the cheap model retry until it costs more
A lower initial request cost can disappear through repair loops. Test a persistent validation failure and verify that the task stops within its total allowance.
Read articleAsk a familiar question the corpus cannot answer
A model may know a plausible answer from general knowledge while the product requires approved evidence. Test that distinction with a question just outside the source collection.
Read articleCount forbidden passages before scoring relevance
Search quality and access safety answer different questions. Measure both so a useful-looking answer cannot hide a failed permission check.
Read articleSeparate candidate recall from final ranking quality
A missing document and a badly ordered document need different fixes. Measure the retrieval stages separately before changing the model or increasing the context window.
Read articleMeasure correct passages interpreted the wrong way
Retrieval can find the right words and still produce the wrong conclusion. Add interpretation checks for the context that document splitting can remove.
Read articleScore claim support separately from polished writing
A fluent answer can still misstate its evidence. Measure whether its claims are supported, whether important conditions survived and whether readers can inspect the source.
Read articleMeasure freshness from the business change
A healthy ingestion job can coexist with stale answers. Measure the interval a reader experiences, then use pipeline timings to explain it.
Read articleDistinguish suspicious output from a successful attack
An assistant can repeat hostile text without executing it, or claim to refuse while a tool still runs. Security results need observable application outcomes.
Read articleCount attempted actions as well as completed tasks
Task completion does not show how much unnecessary or prohibited work an agent tried along the way. Measure the action path as well as the final result.
Read articleMeasure approvals that actually change the outcome
A high approval rate does not show that review is useful. Look at corrections, missed errors and the time people need to understand a proposal.
Read articleMeasure recovery by the final business state
A restarted worker is not a recovered task until its effects are correct and its status is understandable. Measure duplication, unresolved work and operator effort together.
Read article