Hybrid knowledge search
Maintain a query collection that explains the ranking
Search quality needs an owned set of examples. Keep the expected sources and the reasons behind them so future changes can be judged without relying on memory.
In this article
Store the question with its intended use
A query string alone is often ambiguous. "Reset controller" may mean a restart, a factory reset or clearing a warning. Record the user's task and any context needed to identify the intended answer.
For each example, keep the expected source identifiers, relevant versions and a short explanation of why those sources are useful. If several sources are acceptable, say so. A fixture that recognises only one document can penalise a valid alternative and encourage the team to tune towards an unnecessarily narrow result.
Include the caller's access profile when permissions affect the answer. Use synthetic account labels or a controlled fixture rather than storing personal credentials with the query collection.
Give labels an owner and a review process
A domain specialist should own the meaning of relevance, while the engineering team owns the execution and storage of the tests. These responsibilities can belong to the same person in a small team, but they should still be explicit.
When reviewers disagree, preserve the reason for the disagreement. A document may be relevant but outdated, or correct for one product version and wrong for another. These distinctions often reveal a missing field or version rule in the application.
Do not silently replace expected results after a failed test. Record whether the product requirement changed, the source collection changed or the implementation was wrong. Otherwise, the test suite can gradually redefine every regression as acceptable behaviour.
Refresh the collection without erasing its history
Add examples from real support reports after removing information that should not be retained. Keep important failures as regression cases, but avoid filling the collection with many near-identical questions from one incident.
Review examples when documents are retired or permissions change. An unsupported question may become answerable, and a previously correct source may no longer be approved. Version the collection so an old result can still be interpreted against the labels used at the time.
A practical record might contain the query, its category, the source snapshot, expected document identifiers, a relevance rationale and the date of review. Add implementation-specific data only when it helps reproduce the result.
Make the collection part of ordinary delivery
Run it before changing analysers, embedding models, candidate limits or ranking settings. Show changes by query category and include a few examples of the largest gains and losses. An unexplained aggregate score is difficult for a product owner to use.
Keep the process lightweight enough that people will maintain it. A carefully reviewed set of representative questions is more useful than a huge unlabelled export that nobody trusts. Expand coverage where the product's risk or usage justifies the work.
During handover, ask the receiving team to add one query, explain its expected result and run a comparison. If they cannot do that without the original developer, the collection is still a project artifact rather than an operating tool.
Primary sources
Microsoft Learn: relevance and ranking overviewMicrosoft Learn: hybrid search rankingReferences checked 11 September 2026.