Search relevance testing
Search for the procedure using the words a new starter knows
Relevance failures often begin with vocabulary, outdated content or overbroad synonyms. Test those mechanisms directly instead of relying on exact-title queries.
In this article
Start with a task and an expected useful document
Choose a synthetic operational question such as recovering a deployment after a database change. Identify the current procedure and write why it answers the task.
Create query variants using ordinary language, an internal abbreviation and a plausible misspelling. Keep the expected intent stable while the wording changes.
Run the baseline and record the full first page. A relevant result at position twenty is a different failure from a document absent from the index entirely.
Introduce a distracting near match
Add an archived procedure with a highly similar title but obsolete steps. Test whether the system's eligibility and ranking rules handle its status as intended.
Then add a long document that repeats the query terms without providing the procedure. This exposes configurations that reward lexical overlap more than the useful content structure.
Do not repair the test by manually pinning the expected document before understanding the cause. A pin may be appropriate for a deliberate navigation rule, but it does not explain a broader retrieval defect.
Challenge synonym and identifier handling
Try an abbreviation with two meanings in the organisation. An overbroad synonym can improve one query while flooding another with unrelated results.
Test exact identifiers separately from topic search. Punctuation stripping or tokenisation can damage a reference such as a job number even when ordinary word queries work well.
Include a query with no suitable authorised answer. The interface should offer an honest empty state or useful next step rather than presenting unrelated content as a strong match.
Follow the result to the answer
Open the top relevant result and locate the needed section. Check the title, snippet and destination for stale or misleading information.
Run the same query under a restricted test account. Relevant private content must remain excluded, including its title, snippet and count implications where the access model requires that protection.
Record the failure mechanism and the smallest fixture that demonstrates it. This makes the eventual tuning change reviewable and protects against sacrificing one query group to improve another.
Primary sources
Microsoft: relevance and ranking overviewElastic: ranking evaluationReferences checked 11 September 2026.