Hybrid knowledge search
Does the reranker earn its latency?
A second ranking stage should improve the passages people use. Measure its contribution separately from the cost of retrieving more candidates.
In this article
Find the error the reranker is meant to fix
Inspect a sample of poor results before adding another model. If the right source is missing from the candidate set, the problem is retrieval coverage. If the right source is present but buried below weaker candidates, reranking may help.
This distinction matters because a reranker works on the material it receives. Giving it a larger candidate set can increase the chance of including the right source, but that also changes the cost and timing of the experiment. Compare configurations that make the source of the improvement clear.
Keep a version that uses the original ordering. Without that baseline, the team cannot tell whether the extra stage is earning its place or simply making the pipeline more complicated.
Measure the critical path rather than adding every timer
Keyword and vector retrieval often run in parallel. When both are required, the time before merging is driven largely by the slower branch, not the sum of the two branch durations. Reranking then adds work after candidates are available.
Record the actual request timeline, including queueing, network time and retries. A fast model invocation can still sit behind a long queue during peak traffic. Average timings from an idle development environment are a poor guide to that behaviour.
Use a hypothetical budget to make the discussion concrete, then replace it with measurements. If an application allows two seconds for retrieval and answer preparation, decide how much can be spent on reranking without crowding out the answer step. The budget is a product choice, not a universal target.
Compare incremental quality with incremental cost
Run the same labelled questions with and without reranking. Count corrected results and newly introduced errors. Check exact identifiers separately because a semantically appealing passage can still refer to the wrong product.
Estimate monthly cost from the expected query volume and the provider's current billing unit. Include retries and any requests that are reranked but abandoned before the answer is read. Use current pricing from the chosen service when making a purchasing decision rather than copying an old price into the architecture record.
Also consider engineering cost. The extra stage needs monitoring, timeouts, version control and an understood fallback. For a small collection where keyword search already finds the right document, these obligations may outweigh a minor relevance gain.
Define what happens when the stage is slow
Set a deadline and decide whether the application can return the earlier ordering when reranking is unavailable. That fallback should be evaluated as a real serving mode, with its own quality evidence. It should not happen silently and remain invisible to operators.
A request that requires strong evidence may need to stop instead of using a weaker result. A simple document lookup may reasonably return the original candidate list. Choose the behaviour by task rather than applying one rule to every query.
Keep the reranker when the measured improvement justifies its operational and financial cost. Remove or limit it when it does not. A shorter pipeline that reliably returns the right source is a better outcome than a more elaborate one that merely looks advanced.
Primary sources
Microsoft Learn: semantic ranking capabilitiesMicrosoft Learn: hybrid retrieval rankingReferences checked 11 September 2026.