Count attempted actions as well as completed tasks

Define completion in the business system

For a delivery-note task, completion means the intended note is attached to the correct eligible order and the user receives an accurate result. A final assistant message saying "done" is not sufficient evidence.

Use the operation record and target state to determine the outcome. Keep rejected requests, uncertain results and partial completion separate. If the workflow changes several records, define whether all are required or whether a partial result can be useful.

This definition prevents a model that writes confident summaries from appearing more reliable than one that honestly reports unresolved work.

Measure the route to completion

Count proposed tool calls, accepted calls, policy denials, retries and actual mutations. A run that completes correctly after twenty irrelevant lookups has different cost and exposure from one that uses two appropriate calls.

Track duplicate business effects separately from duplicate network attempts. An idempotent operation may safely receive several requests. The undesirable outcome is an extra effect or an incorrect result, not necessarily a second HTTP call.

Also inspect target selection errors. A tool can successfully update a record while failing the user's task because it chose the wrong order. Schema validity and API success rates do not catch that distinction.

Keep refusal and uncertainty visible

A restrictive candidate can reduce bad actions by refusing useful work. Report legitimate completion alongside denied and abandoned tasks, using a reviewed set of requests the agent should be able to handle.

Measure how often uncertain operations are resolved without human intervention and how long resolution takes. An agent that avoids duplicates by leaving every timeout unresolved may be safe in one respect but operationally expensive.

Break down results by capability and consequence. Read-only lookups and external mutations should not share one undifferentiated success rate. Include sample sizes and the kinds of faults exercised.

Set acceptance around permitted effects

Choose thresholds based on the workflow, then investigate exceptions with traces. Deterministic access rules need direct tests even when end-to-end task scores look strong. A prohibited mutation is not made acceptable by many successful ordinary tasks.

Use a stable fixture set to compare model or prompt changes, and add cases from actual failure mechanisms using synthetic data. The resulting report should show whether the agent does the right work, how efficiently it gets there and whether its mistakes remain inside the intended boundaries.