Revoke access while an answer is still cached

Prepare a result that is easy to recognise

Create a test document containing a unique phrase that does not occur elsewhere in the collection. Give one test user access and deny another. Use synthetic content rather than a real confidential contract. The phrase acts as a marker that lets you distinguish a leaked passage from a plausible answer the model produced independently.

Record the source identifier, source version and permission version. Confirm that the allowed user can retrieve the marker and that the denied user cannot. If that baseline fails, stop there. A revocation test is difficult to interpret when ordinary access filtering is already inconsistent.

Run the first query through the same application endpoint people use. Testing the search service directly is useful for diagnosis, but it misses identity handling, application caches and conversation storage.

Warm the paths that can retain content

Ask the allowed user to retrieve the marker, then ask a follow-up question in the same conversation. Open the citation. If the system stores generated answers, repeat the first question until you have evidence of a cache hit.

Write down what is actually cached. A query-result cache, a passage cache and a completed-answer cache can have different keys and expiry rules. The browser may also retain conversation text. These copies do not disappear simply because the search index has changed.

Do not log the full test conversation by default just to make the test convenient. The synthetic marker and request identifiers usually provide enough evidence to follow the path. Reserve detailed payload capture for a controlled test environment.

Remove access without closing the session

Revoke the user's document permission in the source system. Record the time at which the source confirms the change. Keep the application session and conversation open. Then repeat the original query, submit a follow-up that asks for more detail, and open the old citation.

Check again through a new browser session. This separates a stale session problem from an index or shared-cache problem. Repeat until the system's documented revocation window has elapsed, then continue long enough to catch a delayed retry that might accidentally republish old permissions.

A useful result record distinguishes each surface.

Source revocation confirmed: 10:00:00
Fresh query denied:         10:00:03
Existing conversation denied: 10:00:04
Cached answer denied:       10:00:04
Old citation denied:        10:00:02

These times are an example of the record to collect, not a suggested service target. Choose the target from the sensitivity of the content and the source system's actual capabilities.

Inspect the request before judging the answer

An answer that omits the marker is not enough to pass the test. Inspect the selected passages and the model request. If restricted text entered the model context but did not appear in the response, the access boundary still failed.

Also inspect fallback behaviour. Some applications retry a failed filtered query with a broader search. Others use an older cached answer when a dependency is unavailable. Both paths need the same access rules as the successful path.

When a test fails, retain the request identifiers and the relevant version metadata. Change one layer at a time during diagnosis. Clearing every cache and rebuilding the index at once may restore correct behaviour, but it hides which component was responsible and leaves the team without a reliable regression test.