# Ask a familiar question the corpus cannot answer

A model may know a plausible answer from general knowledge while the product requires approved evidence. Test that distinction with a question just outside the source collection.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Build a close but incomplete source set

Create synthetic manuals for two product versions, but omit the installation rule for a third version. Ask a question about that missing version using familiar terminology shared by the other manuals.

The expected behaviour should identify the evidence gap or ask a useful clarification. It should not silently apply a rule from another version merely because the wording looks relevant.

Keep the task's evidence policy explicit. Some products may allow clearly labelled general guidance, while others require source-backed instructions. The fixture should test the policy actually intended.

## Inspect retrieval before judging the answer

Confirm which passages were returned and whether they identify their product version. If the metadata was lost during extraction, the generation stage may not have enough information to recognise the mismatch.

Then inspect whether the answer acknowledges that mismatch. A citation to a real manual does not justify applying its instructions to an unsupported version.

Add a passage that mentions the missing version without giving the requested rule. This tests whether topical relevance is being mistaken for sufficient evidence.

## Make the missing detail available

In a second case, ask an ambiguous question without a version and let the user supply a supported version after clarification. The assistant should then answer from the matching source. This verifies that abstention is a useful intermediate state rather than a dead end.

In a third case, keep the requested version unsupported after clarification. The system should stop asking for details that cannot fill the source gap and offer the defined escalation or limited answer.

Also test a restricted manual that the user cannot access. The system should not resolve uncertainty by retrieving information outside the caller's authority.

## Assert the actual claims

Check that the final response does not state the unsupported installation rule as fact. Allow different wording, but require the relevant limitation and prohibit the cross-version instruction.

Inspect citations and any proposed actions. A cautious sentence is not enough if a tool still performs a change based on the unsupported rule.

Keep the fixture with its source snapshot and expected outcome. It is especially useful after model, retrieval or prompt changes, because a more fluent candidate may become better at filling the gap in a way that sounds convincing but violates the product's evidence requirement.

## Sources

- [Microsoft Learn: retrieval and answer evaluators](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators)
- [OWASP: authorisation guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html)