Distinguish four gaps
A system may lack a relevant document, retrieve the wrong passage, retrieve a passage that does not support the requested claim, or hide a document because the user lacks permission. These produce different user experiences and debugging tasks. Do not tell a user a policy does not exist when it exists but is access restricted. Do not force an answer from a nearby passage because it contains similar words. Design internal reason codes for no retrieval, low relevance, insufficient support, and conflicting versions without disclosing restricted content.
Judge evidence claim by claim
Split the proposed answer into factual claims and check each against the retrieved passages. Ragas faithfulness defines a way to evaluate whether claims are supported by context. A source link alone is not enough: it may point to a page that discusses the topic but not the specific rule. If only part of a question is supported, answer that part with citation and clearly mark the rest unknown. Never fill the gap from a model's general knowledge when the product promises answers from company documents.
Build a no-answer test set
Include questions about facts absent from the corpus, future policy changes, obsolete versions, unapproved drafts, ambiguous dates, and documents a test identity cannot read. Include near matches whose wording tempts the model to infer an answer. Annotate whether the correct response is full answer, partial answer, clarification request, or no answer. Run the same cases through keyword, vector, and hybrid retrieval; Azure's hybrid overview explains why each can find different candidate passages. The no-answer threshold should be chosen on this realistic distribution.
A benefits-policy example
An employee asks whether a dependent is covered after turning 26. The retrieved policy discusses dependent coverage but gives no age rule. A grounded answer should say that the available passage does not establish an age limit, cite the relevant section, and direct the employee to the benefits team or a more specific policy. Saying yes because another insurer uses that limit is an unsupported claim. If a current and an old policy conflict, the system should state the version issue and request confirmation rather than choose by similarity score.
Make abstention actionable
Show the documents searched only where the user may access them, the date or version used, and a concise next step. Log internal failure reason, query, retrieval IDs, and reviewer outcome to improve the pipeline. Measure unsupported-answer rate and unnecessary-abstention rate together; a system that never answers is safe-looking but useless. Set stricter evidence requirements for actions that affect money, employment, or customer records. Review no-answer cases regularly to discover missing ingestion or policy documentation, not only model behavior. Designing refusals and fallbacks is part of how we build RAG and semantic search solutions that users can trust.
Technical references: Ragas faithfulness metric, Azure hybrid retrieval overview. Related guide. Check the cited documentation against your own data, permissions and release version before applying this design.