Review the query that has no correct answer

Define an unsupported question precisely

Choose a question whose answer is absent from the approved collection. It might refer to an unavailable product, a policy that has not been published or a condition the documents never discuss. Record why the sources are insufficient so reviewers are judging the same thing.

Distinguish this from a question that is merely difficult to phrase. If the answer exists but retrieval misses it, the system has a coverage problem. If no source supports the answer, returning a confident response creates a different failure.

Include a source that looks close but is wrong. For example, a manual for a neighbouring model makes the exercise more revealing than an empty index, because the application has a plausible candidate it must avoid treating as authoritative.

Watch how the pipeline fills the gap

Inspect the keyword candidates, vector candidates and final passages. Then read the answer against those passages. Does the application acknowledge the missing evidence, ask a useful clarification question or invent a conclusion from similar material?

Pay attention to defaults such as always returning a fixed number of passages. A request for five results can produce five weak candidates even when none is suitable. The existence of a full result list does not establish that the collection contains an answer.

Do not interpret a retrieval score as a calibrated probability of correctness. Scores depend on the method and configuration. A fusion score can also change when the number of contributing lists changes. A threshold needs evaluation in the actual pipeline, not a reassuring label such as confidence.

Check that the explanation helps the reader

An unhelpful refusal says only that the system cannot answer. A better response identifies the missing source or distinction without pretending to know more than it does. For a product-code ambiguity, it may ask the user to confirm the code. For an unpublished policy, it may point to the current policy and state its limits.

Avoid revealing the existence of a restricted document while explaining the absence of an answer. The application should not say that a confidential contract contains the answer if the caller is not entitled to know about that contract.

Keep the user's task in view. Sometimes a link to the closest relevant source, clearly labelled as incomplete, is more useful than a generated answer. The interface does not need to produce prose for every query.

Retain the unsupported cases in evaluation

Store the question, the collection version and the expected behaviour. Unsupported questions can become supported when a new document is published, so the fixture needs maintenance. The label is a statement about a particular source set, not a permanent property of the words in the question.

After changing retrieval or prompts, review these cases alongside the successful ones. A configuration that answers more questions may have improved coverage, or it may simply have become more willing to guess. The evidence should make that distinction visible.