Measure freshness from the business change

Choose a meaningful starting event

For a policy update, the useful clock may start when the approved revision becomes effective. For a document upload, it may start when the source accepts the file. For a withdrawal, it may start when the source owner requests that the material stop being used.

These events are not interchangeable. A draft can exist for days before publication, and a future-dated policy should not become authoritative merely because it was indexed early. Record the event that matches the business requirement and retain the other timestamps for diagnosis.

Where the source cannot provide an event time, label the measurement as observation-to-visibility. Do not report it as end-to-end freshness. The missing detection interval is precisely where a polling system can accumulate delay.

Measure success and absence separately

For an update, check when the expected revision becomes available through the normal query path. For a withdrawal, check when the old revision can no longer be served. A successful query for the new version does not prove the old version has disappeared from every cache or passage set.

Use controlled probes with known source identities. Natural-language answer wording is too variable to be the only signal. Inspect the evidence revision recorded by the application and verify relevant caches as part of the probe.

Report failures as well as completed timings. If a document never becomes current, excluding it from latency calculations makes the average look better as reliability worsens. Track the count and age of unresolved changes alongside completed changes.

Break results down by source and consequence

A large document archive and an urgent operational policy feed may have different freshness requirements. Separate them in reports so a high-volume, easy source cannot hide a small but important failing connector.

Useful measures include the oldest pending change, the proportion completed within the agreed interval and the number of answers served from superseded evidence. Include the sample size and observation period. A percentile calculated from a handful of changes is a weak basis for a service commitment.

Permission revocations deserve their own view where the access model depends on copied metadata. Their consequence differs from a harmless delay in indexing a new reference manual.

Use the metric to choose an action

Connect each threshold to a response. A growing detection delay may require restoring a connector. Slow processing may call for capacity changes or isolating malformed documents. A current index with stale answers points toward cache and conversation handling.

Review the requirement with source owners using actual examples. A phrase such as "near real time" leaves too much room for disagreement. A measurable interval, defined starting event and specified behaviour during delay give both engineering and the business something concrete to accept.