Search relevance testing
Search relevance must be judged inside the user's permission boundary
Private documents should never compete for visibility with public results. Evaluate ranking only after the access model defines what the user may receive.
In this article
Define eligible content before scoring quality
For each test audience, establish which documents are authorised and current. A highly relevant restricted document must not appear merely because it would answer the question well.
Apply the application's supported permission filtering at the appropriate retrieval boundary. Check every output surface, including snippets, suggestions, related results and counts where they can reveal protected information.
Do not treat a low rank as protection. A document outside the first page may still be accessible through pagination, a direct API request or a more specific query.
Protect the evaluation dataset
Queries can contain customer names, internal project details or copied incident text. Review and sanitise production-derived examples before sharing them with reviewers or external evaluation services.
Keep access to judgments and retrieved content consistent with their sensitivity. A benchmark export can become an unintended collection of private documents even when the production interface is correctly restricted.
Use synthetic identities and content for boundary tests where possible. They can prove isolation without repeatedly distributing real customer material.
Test changes in permission and identity
Remove a user's access to a document and verify when search stops exposing it under the application's freshness contract. Test cached results as well as new queries.
Switch accounts in the same browser and confirm that previous results and snippets do not reappear under the new session. Query caches need an appropriate account boundary.
For a synthetic case, two organisations have similarly named recovery procedures. The result should come from the authorised organisation, not whichever document has a higher lexical score.
Separate relevance exceptions from access exceptions
A product owner may decide to promote a document for a particular query. That rule must still operate within current permissions and content eligibility.
Record who can change synonyms, boosts and pinned results, because those settings influence what people see. Keep the change history available for investigations into unexpected exposure or misleading results.
Acceptance should include deliberate denial cases alongside useful-result judgments. A search system is not successful if its strongest answer is one the user should never have received.
Primary sources
Microsoft: relevance and ranking overviewReferences checked 11 September 2026.