# Ask which queries became worse after the search change

A credible relevance review shows regressions, unjudged results and corpus assumptions. A single improved demo query is not enough.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Inspect the evaluation material

Read the query intents and rating guidance. Confirm that the set includes ordinary user wording, exact references, ambiguous terms and no-answer cases.

Check whether judgments were made against the same corpus and access context used in the experiment. Stale or unauthorised expected documents can make the evaluation misleading.

Look for a distinction between unjudged and irrelevant results. New candidates need review rather than an automatic label chosen to favour the proposed configuration.

## Compare individual result lists

Review the largest gains and losses alongside aggregate metrics. Open the relevant documents and confirm that they actually satisfy the recorded task.

Ask whether the metric matches the interface depth and user need. A first-relevant-result measure can miss poor later results in a discovery task.

Examine critical queries separately even when they are rare. Their business consequence may justify a stricter acceptance threshold than the average query.

## Follow the technical boundaries

Verify corpus completeness, indexed fields, tokenisation and filter behaviour before attributing every difference to ranking. A parser change can masquerade as a search-model regression.

Inspect configuration versioning and tie behaviour. Reproducing the experiment should not depend on remembering an undocumented boost added in an admin console.

Test access restrictions, cached results and destination links. Useful ranking is only part of the user-visible search contract.

## Review the operating tradeoff

Compare latency, timeouts and cost at the chosen candidate depth. Include the fallback result when an optional reranking stage is unavailable.

Check the planned rollout and rollback route, including the freshness of any fallback index. A technically reversible switch can still return obsolete content if the old index stopped receiving updates.

Approve the change with named gains, accepted losses and unresolved judgments. “Search feels smarter” may describe a demo impression, but it does not provide a maintainable basis for a production configuration.

## Sources

- [Elastic: ranking evaluation](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/search-rank-eval)
- [Microsoft: BM25 relevance scoring](https://learn.microsoft.com/en-us/azure/search/index-similarity-and-scoring)