Service level indicators
Break the workflow while leaving the health endpoint green
A controlled dependency failure can show whether monitoring detects user impact or only process availability.
In this article
Choose a safe failure after acceptance
Use a test environment and a synthetic operation that passes the initial API check but fails later. For example, block the worker's access to a safe test destination while leaving the web process running.
Confirm that the health endpoint still behaves as expected for its intended scope. The purpose is not to make health checks meaningless, but to demonstrate that they cover a different question.
Record the operation identity and expected final outcome before introducing the fault.
Observe the user-level measure
Submit the operation through the normal test path and wait until the defined deadline. The service indicator should count the missed outcome even if the API returned an accepted response.
Inspect the denominator. If the measurement only counts completed jobs, the unfinished operation may disappear instead of reducing the result.
Verify that retries do not multiply the customer operation or hide the original acceptance time.
Check the alert and diagnosis path
Confirm that the configured alert or review signal reflects the degradation according to policy. Avoid requiring a particular page for every single synthetic failure if the alert is designed around sustained impact.
Use the linked runbook to find the affected operation and the failed dependency. The operator should not have to infer the backlog from a generic server alarm.
Test missing telemetry separately. An unavailable metric source should produce a visible monitoring problem rather than a falsely perfect service result.
Restore the dependency and inspect history
Resume processing and confirm eventual completion. The late operation should remain late in its original measurement while appearing completed in the recovery view.
Run a new synthetic operation to show that current service has recovered. This separates historical objective consumption from present health.
Keep the exercise's expected counts and timings as regression evidence. It proves that the measurement notices a specific user-visible failure even when infrastructure remains superficially healthy.
Primary sources
Google SRE: alerting on service objectivesReferences checked 11 September 2026.