What is RAG, and why is a citation insufficient by itself?
Retrieval-augmented generation, or RAG, supports a language model's answer with text retrieved from external sources. A citation tells the reader which document and passage can be used to check a claim. A working link is the first check; whether the passage supports the full statement, applies on the relevant date and is accessible to the user are separate questions. A real document can still be the wrong paragraph, an obsolete version or evidence for only half of a sentence.
A study on correctness and faithfulness in RAG attribution distinguishes textual support from genuine model reliance on the cited source. A model can attach a plausible citation after producing an answer from prior knowledge. A correct attribution therefore does not establish actual source use by itself. In an enterprise assistant, first make the support relationship inspectable by the reader, and design source-reliance experiments separately. A supported answer can also remain invalid for a different region, date or product. Move from asking whether a citation exists to asking whether this passage verifies this claim for this user and event.
Citation-grounding test cases
Giving an entire answer one correct or incorrect label hides the failure's location. The following cases provide a starter set that assesses the link, support and applicability independently. Preserve the question, controlling source version, exact passage and expected behaviour for each case. Split paragraphs containing several claims: a correct number does not establish correct scope. For partial support, identify the supported and unsupported portions rather than merely awarding a score that leaves the same misleading sentence visible to the user.
| Test case | Expected label | Acceptance behaviour |
|---|---|---|
| Correct version, full support | Supported and applicable | Show the claim with its passage |
| Right document, wrong paragraph | Incorrect citation mapping | Link the exact supporting passage |
| Source supports half the sentence | Partially supported | Split claims and remove unsupported content |
| Old policy contains the same number | Supported but inapplicable | Retrieve the effective version |
| Exception footnote was not retrieved | Incomplete evidence coverage | Retrieve footnote or limit the answer |
| Citation is outside the user's permissions | Access violation | Exclude the passage before retrieval |
Connect labels to the component that needs correction. A missing new policy is an ingestion or retrieval issue. Ignoring a retrieved new policy is a generation issue. Showing the wrong link for the correct passage is a citation-mapping issue. An access violation remains unacceptable even if the answer is fluent and factually correct. Keep these outcomes separate so teams do not spend time tuning prompts to fix a broken indexing or permission process.
How to preserve the claim-to-source relationship
During ingestion, retain stable document identity, version identity, title, validity interval, page or paragraph range and passage identity. Associate the source URL with those fields. A URL can later hold a different file, so preserve a content hash or immutable version copy to help confirm which text was used. Keep the relationship between table cells, headings and controlling footnotes. If the user cannot open the source, the citation resolver must not widen access just to make the link work.
Give the generator an explicit list of permitted source identifiers. Associate every claim with one or more passages on that list; reject invented document identifiers and arbitrary source URLs. Identifier membership is the first check and semantic support is the second. Divide a sentence such as “All hotels above 4,000 TRY require approval” into the amount threshold, accommodation scope and approval requirement. Then ask whether the passage supports the universal wording and relevant exceptions, rather than checking only that the number appears in the text.
Define support labels as supported, partially supported, unsupported and uncheckable. The last category is useful when the source is unavailable or evidence is missing; it must not become a way to count an unsupported answer as correct. Store freshness, scope and access as separate dimensions alongside support. An accurate historical passage may support a claim but be inapplicable to a current decision. When several passages jointly support an answer, inspect the relationship between them as well. Combining two individually correct clauses can still produce a general conclusion that neither source establishes.
What does automated evaluation measure?
The Ragas faithfulness metric checks claims against retrieved context. That context-level check does not establish that each displayed citation points to the right evidence. The correct source can be in the context while the interface links another passage. Textual support is also different from validity for the current event. Record overall context support, individual claim-to-citation mapping and version applicability as distinct outcomes.
The ALCE citation evaluation research treats fluency, correctness and citation quality as separate evaluation dimensions. Your acceptance criteria should likewise reflect the decisions the assistant supports. You can measure working citation pointers, fully supported important claims, answers using the applicable version and appropriate abstention when evidence is absent. Do not invent universal target percentages for these measures. Agree thresholds with the product owner according to the consequence of a misleading answer, and preserve the failure categories behind the aggregate rates.
Give an automated evaluator the question, atomic claim, exact cited passage and relevant scope. Preserve its decision and a short explanation. Compare ambiguous cases and a random sample of other decisions with human review, because the evaluator can make mistakes too. An evaluator that omits claims during decomposition can overlook unsupported details. Include targeted cases for conditions, negation, dates and units. The user interface should expose the passage and version easily rather than hiding the evidence behind a document title. Showing more source links does not compensate for a claim whose actual support remains unclear.
Hypothetical example: the right number with the wrong scope
Suppose a fictional company's travel policy version 3, effective in January, requires manager approval for domestic hotels above 4,000 TRY per night. Version 4, effective on 1 July, raises the limit to 5,000 TRY and adds approval for international reservations regardless of amount. Assume the policy uses the accommodation date. A user asks whether a 4,500 TRY domestic stay taking place in August requires approval. The assistant answers, “Yes, all hotels above 4,000 TRY require approval,” citing version 3. The link works and the number exists in that source, but the historical version is inapplicable and its passage does not support the word all.
Switching to version 4 is still insufficient. “No, all hotels below 5,000 TRY need no approval” uses the current number but makes an invalid generalisation because of the international exception. Only the domestic, amount-based part of the sentence is supported. A correct answer says that the 4,500 TRY domestic stay does not require approval because of this amount threshold, while other approval conditions may still need assessment, and links the controlling clause in version 4. An international query should instead use the clause requiring approval at any amount.
Keep both incorrect answers in the same test set: the first catches obsolete-version selection and the second catches partial support. The review record should preserve each claim, its cited passage, version and failure label. All amounts and policy terms here are hypothetical, not a customer case or legal rule. A number-matching check could miss both scope errors. A reviewer should be able to see exactly which part of the proposed answer must change and which retrieval or mapping step produced the defect.
A citation verification test plan
- Prepare difficult source pairs. Include nearly identical product manuals, old and new policies, a table footnote, a missing exception and an inaccessible document. Label the correct version and passage for each question.
- Inspect raw retrieval. Check whether necessary evidence is present and restricted passages are absent. Record missing-source failures separately from generation failures rather than scoring only the final text.
- Bind claims to citations. Check full support for each important number, condition and scope term. When evidence is absent, expect a limited answer or a clear statement that the claim cannot be verified.
- Audit the evaluator. Track disagreement with human labels, partial support wrongly accepted as complete and missed exceptions. An automated score should not become the sole acceptance gate without this calibration.
- Re-run after changes. Repeat tests when ingestion, chunking, retrieval, models or interface mapping change. Include opening citations with the actual user's permissions, rather than testing them only with an administrator account.
Common mistakes
- Treating a working source link as proof of full claim support and applicability.
- Checking the number while overlooking universal wording, restrictions and exceptions.
- Confusing overall context support with correct mapping of an individual claim and citation.
- Relying on a mutable URL without preserving document version and passage range.
- Accepting automated evaluation decisions without checking a human-reviewed sample.
How Hatson Tech builds this
We start by identifying important business questions and the sources that can verify their answers. We design passage provenance, claim-to-citation mapping and acceptance tests together in our RAG and semantic search solutions. For retrieval measures, see our RAG evaluation metrics guide; for access boundaries, see permission-aware retrieval. A pilot should demonstrate that readers can verify every important claim from an applicable, authorised passage. That criterion gives the team a practical way to inspect evidence quality alongside the usefulness of the final answer.