# Almost the same product code is the wrong product

Similar manuals are a useful stress test for semantic retrieval. Build examples where a one-character difference changes the correct answer.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Create a pair that should not be interchangeable

Use two synthetic product manuals with similar language and one important difference. For example, AX-140 may allow one connection sequence while AX-141 requires another. Make the differing instruction explicit in each source. The test is easier to judge when the correct result is unambiguous.

Add an exact identifier field to the fixture. Then include the identifier in ordinary body text as well. This lets you test whether the system honours an explicit product request or merely finds the most semantically similar passage.

Do not use a real safety-critical instruction as an invented example. The purpose is to test identity handling, not publish an installation recommendation. Synthetic product codes and harmless configuration differences are sufficient.

## Vary the query without changing the required product

Try the code alone, the code with a natural-language question and the code in a sentence containing terminology from the other manual. Include punctuation and case variations that the business actually treats as equivalent.

Also query a code that does not exist. The expected behaviour should be defined before running the test. If the application offers a possible alternative, it must make the substitution clear rather than answering as though the requested product were present.

```text
AX-140                         Find the AX-140 source
How is AX-140 configured?       Answer only from AX-140 evidence
AX-140 shutdown procedure       Do not silently switch to AX-141
AX-149                         Report the missing product or ask for clarification
```

These expectations belong to the example application. A different product may choose a different clarification flow, but it still needs an explicit rule.

## Inspect each stage of the result

Record the candidates returned by keyword search and vector search. Next record the merged list and the final passages supplied to the model. If the correct manual never appears, investigate ingestion, fields, filters and candidate retrieval. If it appears and is then displaced, investigate fusion or reranking.

Look for identity being lost during deduplication. Two passages with similar text are not necessarily duplicates if they refer to different products or versions. Deduplicating by text alone can erase the very distinction the query depends on.

Inspect the answer for blended instructions. The model may cite the correct manual while incorporating a detail from the other one. A citation's presence is not evidence that every sentence came from the cited product.

## Turn the failure into a narrow correction

If the query contains a recognised exact identifier, the application can use it as a constraint or a separate lookup signal. The correct choice depends on how identifiers are represented and whether users sometimes enter incomplete codes. Avoid an unreviewed rule that treats every number in a question as a product identifier.

After a correction, rerun natural-language questions that contain no code. An aggressive exact-match rule should not prevent users from finding a document when they only know the symptom.

Keep the paired manuals in the regression fixture. Add a third version when the application gains version-specific behaviour. This makes the test reflect the distinction users care about rather than becoming a one-off demonstration that one query now works.

## Sources

- [Microsoft Learn: relevance and ranking behaviour](https://learn.microsoft.com/en-us/azure/search/search-relevance-overview)
- [Microsoft Learn: semantic ranking scope](https://learn.microsoft.com/en-us/azure/search/semantic-search-overview)