Search relevance testing
Change search engines without losing the queries people depend on
Compare candidate coverage and ranked results under the same access context. A successful reindex does not establish equivalent search behaviour.
In this article
Preserve a baseline before rebuilding
Save representative queries, intent notes, judgments and the current ordered results. Record index schema, analyzers, synonyms and filter behaviour.
Include exact identifiers and critical operational questions, not just popular broad topics. A new engine may improve natural-language discovery while damaging precise lookup.
Keep document identities stable or provide an explicit mapping so old judgments and links remain useful during comparison.
Validate the new corpus before ranking it
Reconcile source counts, deletions, current versions and access metadata. Inspect parsed content from difficult formats and verify that the fields intended for search contain the expected text.
An index with the right document count can still omit the body of every PDF. Candidate coverage and content fidelity need direct checks.
Run restricted-account tests before exposing the new index. Ranking quality cannot compensate for a permissions regression.
Compare results without duplicate user effects
Use shadow queries or a controlled preview to compare the new system with the baseline. Keep latency and errors in the report alongside relevance metrics.
Review unjudged new candidates and per-query losses. Do not reject every unfamiliar result automatically, but do not assume semantic similarity establishes task usefulness either.
Test snippets, highlighting and destination links in the actual interface. Their contracts may differ even when the underlying document IDs match.
Cut over with a reversible query route
Keep a documented way to route queries back to the previous supported index while the new release is assessed. Continue necessary content updates to that fallback during the rollback window or explain its freshness limits.
Record configuration versions and the time of cutover so support can reproduce a reported result. A moving index and an undocumented synonym change make incident comparison difficult.
Retire the old system after relevance, access and operating evidence meet the agreed acceptance criteria. The migration is complete when users can still find the required knowledge, not when the last document finishes indexing.
Primary sources
Elastic: ranking evaluationMicrosoft: relevance and ranking overviewReferences checked 11 September 2026.