Permission aware retrieval
Count forbidden passages before scoring relevance
Search quality and access safety answer different questions. Measure both so a useful-looking answer cannot hide a failed permission check.
In this article
Keep the two scorecards separate
A relevance score tells you whether retrieved material helps answer a question. It does not tell you whether the person asking should see that material. A system can produce an excellent answer from a document the user is forbidden to read.
Create an access scorecard before combining quality results into a release summary. Count unauthorised passages selected for the model, unauthorised text shown to the user and citation links that open content outside the user's access. Treat each as a separate observation. A citation failure can exist even when the generated answer is harmless.
Do not average an access failure away with thousands of successful relevance tests. A release decision needs an explicit rule for a confirmed unauthorised result. The appropriate response is usually to investigate the boundary and stop the affected release path, rather than accept a small percentage in an overall quality score.
Build a labelled access fixture
Use a small set of accounts and documents with known relationships. Include two organisations, two teams within one organisation, a public document and a document whose access changes during the test. Write the expected result for each account and document pair.
The labels should come from the intended policy, not from the search application's current answer. Otherwise, an implementation mistake can become the test's definition of correct behaviour. Have the data owner review the fixture and keep it small enough that someone can understand it without a separate tool.
Run both exact-marker queries and ordinary questions. Exact markers make leakage easy to diagnose. Natural questions exercise the ranking and fallback paths people actually use. Add a no-answer question so the system has an opportunity to abstain instead of broadening the search beyond the allowed collection.
Give every denominator a name
"No access failures in 500 tests" is useful only when those tests include attempts to cross a boundary. Separate the number of allowed queries from the number of denied queries. Report the accounts, document states and serving paths covered.
For example, a test set might include 120 allowed queries, 80 denied queries and 40 queries after access removal. That is a description of the fixture, not proof that the remaining application is safe. A new export endpoint or cache implementation may require new coverage even if the total test count stays unchanged.
Track the delay between a confirmed permission change and its effect on each serving path. Report the distribution and the slowest observed case. A mean can conceal one stale result that remains available much longer than the rest.
Pair the numbers with traceable examples
For each failed test, keep the request identifier, expected access decision, observed document version and component that supplied the passage. Avoid putting confidential text into a general metrics dashboard. A restricted evidence store can hold the detail needed for investigation.
Once access checks pass, assess relevance within the permitted document set. This matters because restrictive filters can reduce the number of useful candidates. If a result becomes less relevant, improve retrieval within the boundary rather than weakening the boundary to raise the score.
The release record should name the fixture version and the unresolved limitations. That makes a later comparison meaningful. A score without its test data, permission state and configuration is difficult to reproduce and easy to misread.
Primary sources
Microsoft Learn: vector query filteringOWASP: authorisation testing and enforcementReferences checked 11 September 2026.