Review the tool schema against the business rule
A typed tool can still expose too much authority. Compare every accepted field and side effect with the operation the user is actually permitted to request.
Read articleCobnex blogs / Australia
Cobnex blog
AI implementation, software architecture, cloud operations and Australian technology policy.
150 articles in AI engineering
Page 8 of 9
Compare the approval screen with the submitted payload
Review the values a person saw against the values the executor used. This catches hidden fields, stale state and post-approval changes that a polished screen can conceal.
Read articleReview what the transcript cannot prove
A conversation can describe a completed action without proving its effect. Review the records and state transitions used to recover when messages and mutations diverge.
Read articleAsk reviewers to inspect refusals and escalations
A candidate can improve correctness by answering less. Review what it declines, what it passes to people and whether those decisions fit the product's intended scope.
Read articleRequire a reproducible release comparison
A reviewer should be able to identify the candidate, rerun its important cases and understand every material difference from the baseline.
Read articleReview one accepted invoice all the way to posting
Trace a record that bypassed manual review from its source page to the downstream payload. This reveals whether acceptance means more than a high extraction score.
Read articleReview the architecture with one real request trace
Use a controlled request to compare the documented data path with the deployed one. Then exercise an error and a fallback to expose routes the main diagram may omit.
Read articleReview the cost of one unsuccessful task
Failed tasks expose missing accounting, uncontrolled retries and weak stopping states. Trace one from its first request through its final resolution.
Read articleReview the next step after the assistant says no
A refusal should have a reason and a useful consequence. Inspect whether the product clarifies, escalates or denies access according to the actual gap.
Read articleDocument who owns access revocation
An access boundary depends on several teams. A useful handover names who changes the source, who updates retrieval and who handles a stale result.
Read articleMaintain a query collection that explains the ranking
Search quality needs an owned set of examples. Keep the expected sources and the reasons behind them so future changes can be judged without relying on memory.
Read articleGive ingestion owners a structural regression set
Parser upgrades can change the meaning of retrieved passages without changing the source files. Hand over examples that show which document relationships must survive.
Read articleMaintain the evidence mapping, not just the prompt
Citation quality depends on source records, access rules, rendering and review criteria. Give the next team enough information to maintain all four.
Read articleName the owner of a stale-answer incident
Source editors, ingestion engineers and application teams each control part of freshness. Give support one route to coordinate them when an old answer is still being served.
Read articleKeep an injection regression corpus with the tool owner
The team that changes an assistant's capabilities should own the tests showing how those capabilities resist untrusted instructions.
Read articleGive operators a map of every agent effect
Support needs to know what an agent can change, where the result is recorded and how to stop or recover it. A list of model prompts does not provide that map.
Read articleHand over the rules for stale approvals
The receiving team needs to know when an approved proposal can still run, when it needs review again and how to explain the difference to users.
Read articleLet the receiving team recover a paused run
Recovery documentation is ready when another team can use it to restore a task and explain its final business state. Make that exercise part of the handover.
Read article