# Diagnose a relevance regression from both candidate lists

When search gets worse, identify the stage that changed. Preserve the question, source versions and branch results before tuning the final answer.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Capture one reproducible bad result

Start with a question that previously returned useful material and now does not. Record the expected source and why it should answer the question. A report that search "feels worse" is a useful signal, but it is not yet a reproducible failure.

Capture the application release, index version, embedding configuration and query options. If the question contains sensitive information, retain it through the approved diagnostic process rather than copying it into a broadly accessible issue tracker.

Check whether the expected document still exists and is currently accessible to the reporting user. A deleted source or a legitimate permission change is not a ranking regression, although the application may need a clearer explanation for the user.

## Compare the branches before the final list

Run the same question against the keyword and vector branches separately in a controlled environment. Record source identifiers and positions in each list. Then compare the merged list and the passages selected for the model.

If the expected source is absent from keyword results, inspect field mapping, tokenisation and query normalisation. If it is absent only from vector results, inspect the embedding version, indexed passage coverage and the query embedding. A model change can alter the neighbourhood of similar passages without changing any visible document text.

If the source is present in both branches but missing from the final selection, inspect deduplication, fusion and reranking. Check whether candidate limits changed. A later stage cannot consider a passage that an earlier truncation removed.

## Look for partial failures disguised as normal results

A slow vector branch may be timing out while the application quietly serves keyword results. A failed reranker may cause a fallback ordering. These behaviours can be reasonable if designed deliberately, but operators need to see when they occur.

Compare timing and error information with the bad queries. A deployment that introduced a tighter deadline may appear to change relevance even though the ranking configuration is unchanged. Similarly, an ingestion backlog may leave the newest sources unavailable to both branches.

Avoid changing several weights at once. Save the failing fixture, alter one suspected cause and compare the stage outputs again. Otherwise, a temporary improvement can leave the actual defect unidentified.

## Restore a known path and retain the learning

If the regression is tied to a recent configuration, consider returning to the last verified version. Confirm that its index and embedding configuration still match and that it respects current permissions. Reverting only the query model while keeping incompatible stored vectors is not a complete rollback.

After restoration, run the failing question and a small representative set from other query types. This catches a fix that improves one example by damaging exact identifiers or no-answer handling elsewhere.

Add the diagnosed case to the maintained query collection with its expected source and reason. Record the stage that failed. The result is a regression test the next engineer can understand, rather than an unexplained weight change left in a configuration file.

## Sources

- [Microsoft Learn: hybrid search ranking](https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking)
- [Microsoft Learn: semantic ranking](https://learn.microsoft.com/en-us/azure/search/semantic-search-overview)