Hybrid knowledge search
Separate candidate recall from final ranking quality
A missing document and a badly ordered document need different fixes. Measure the retrieval stages separately before changing the model or increasing the context window.
In this article
Ask whether the right source was available to the ranker
Candidate recall measures how much relevant material appears in the set proposed for later selection. If a question has four labelled relevant sources and the candidate set contains three, recall for that question is three divided by four. The labels and the size of the candidate set must be stated with the result.
That number does not describe the order of the documents. A useful source can be present at the bottom of a long list and still never reach the answer. Conversely, a ranker can order an incomplete set perfectly while the most important document remains absent.
Report candidate coverage separately for the keyword branch, vector branch and merged set. This shows whether the second branch adds genuinely useful sources or mostly duplicates the first.
Judge the result the reader will actually use
After candidate retrieval, evaluate the final selected passages. A simple starting measure is precision among the first few results: how many selected items are relevant to the question. If relevance has several grades, a rank-sensitive measure can give more credit when the best sources appear earlier.
Do not choose a measure only because a library calculates it. Define what counts as relevant for the workflow. A current policy may be more useful than an older policy with a closer wording match. A passage that mentions the product but does not answer the requested question may deserve a lower grade.
Have a domain reviewer label disputed examples. Preserve the reason for the judgement. Otherwise, a future engineer cannot tell whether a score changed because the search improved or because the labels were interpreted differently.
Split the results into meaningful groups
Create separate views for exact identifiers, abbreviations, ordinary prose and unsupported questions. Add access-restricted queries if the collection contains protected documents. A single aggregate can hide a regression concentrated in a small but important group.
For example, a configuration might improve 70 broad questions while making five exact-code questions wrong. Whether that is acceptable depends on the workflow, but the decision should be visible. It should not disappear inside an average calculated over all 75 questions.
Use the same source snapshot when comparing configurations. If a manual was added between runs, the improvement may come from better coverage rather than better retrieval. Record the index, embedding and query configuration alongside the test-set version.
Connect search evidence to answer evidence
Once the selected passages look correct, examine whether the answer uses them faithfully. Retrieval metrics do not measure unsupported claims, incorrect units or a conclusion drawn from the wrong version. Keep those answer-level checks separate so the team knows which stage needs attention.
Include latency and failure rate in the release record. A ranking improvement that regularly exceeds the application's deadline may not help the person waiting for an answer. Compare the quality improvement with the cost of achieving it.
Finish with representative wins and losses, not only a score. A small set of traceable examples helps reviewers understand what the number means and prevents a technically impressive metric from replacing a useful product decision.
Primary sources
Microsoft Learn: search relevance conceptsMicrosoft Learn: hybrid result rankingReferences checked 11 September 2026.