Search relevance testing
One average relevance score is not a release decision
Compare query groups and individual regressions. Match the metric to whether the user needs one known item or several useful results.
In this article
Choose the depth people actually inspect
If the interface shows five results before scrolling, evaluate that first group as well as a deeper candidate set. A metric measured at fifty results can look healthy while the visible screen is poor.
For an illustrative top-five list with two relevant documents, precision at five is 0.4 under a binary judgment scheme. That number says how much of the visible list is useful, not whether every relevant document in the collection was found.
Recall requires a meaningful set of known relevant documents. An incomplete judgment pool limits what the resulting recall estimate can establish.
Keep known-item queries distinct
For exact lookup, inspect the rank of the first relevant result. If that result is third, its reciprocal rank is one third. Averaging that measure across queries is useful for this task, but it does not evaluate the usefulness of later results.
Do not let common easy lookups dominate rare critical procedures. Report categories separately and identify any query whose regression would materially affect the business workflow.
Record no-result queries and failed requests rather than excluding them from evaluation. Missing data can make the remaining average look artificially strong.
Review judgments and uncertainty
Track the proportion of returned documents that remain unjudged. A ranking change often introduces new candidates, so the evaluation may need review before its score is comparable.
Use clear rating guidance and resolve important reviewer disagreements. The metric is only as meaningful as the intent and judgments behind it.
Keep a held-out set where practical so repeated tuning does not merely memorise the small development benchmark.
Add latency and task evidence
Measure the time to usable results and verify that users can open a relevant destination. A better ranking that routinely times out may not improve the search experience.
Treat clicks and reformulations as behavioural signals with known ambiguity. A user may click the first result because of its position or reformulate because the underlying content is missing.
A release decision should name gains, important losses, unjudged cases and operating cost. The aggregate score is a summary of that evidence, not a replacement for it.
Primary sources
Elastic: ranking evaluationReferences checked 11 September 2026.