Search relevance testing
Diagnose a missing search result before boosting its title
Check eligibility, ingestion and query interpretation before changing ranking. A score adjustment cannot retrieve a document that never entered the candidate set.
In this article
Reproduce the query in its original context
Capture the query text, filters, account context, time and expected document reference through an appropriate support process. Avoid collecting unrelated private search history.
Confirm that the document should be visible to that user and is current enough for the task. A deliberately restricted or archived record may be absent for the correct reason.
Run the same query against the relevant index version and preserve the ordered results. A later index update can otherwise make the original failure impossible to explain.
Verify the document's path into search
Check ingestion status, indexed fields and any deletion or access metadata. If the document is missing or its useful text was never indexed, repair that stage before tuning relevance.
Inspect tokenisation and query interpretation for exact identifiers, punctuation and abbreviations. A reference split into the wrong tokens can fail even though the source text looks correct in the document.
For a synthetic example, a procedure title is indexed but its main body is omitted after a parser change. Boosting the title may help one exact query while leaving ordinary-language questions broken.
Examine ranking only after candidacy is established
Compare the relevant document with the results above it. Look at searchable fields, weights, freshness rules and the chosen retrieval stages.
Do not compare raw scores across unrelated queries as though they were probabilities. Use the engine's explanation tools and a judged query set to evaluate a proposed change.
Test the fix against neighbouring query categories so a broad synonym or boost does not create a new failure elsewhere.
Close the user-visible path
Open the corrected result and confirm that its snippet and destination help the user find the answer. Repair stale titles or broken links separately from retrieval configuration.
Add the sanitised query and intent to the regression set with reviewed judgments. Record whether the cause was missing content, access, ingestion, interpretation or ranking.
If the organisation has no document answering the question, assign a content owner rather than manufacturing a relevance tweak. Search cannot recover knowledge that has not been recorded.
Primary sources
Microsoft: BM25 relevance scoringMicrosoft: relevance and ranking overviewReferences checked 11 September 2026.