Diagnosing a Bad RAG Answer

Save the user question, permitted identity, document versions, retrieved passages, prompt version, and answer. Use the fault tree to locate the failed stage and verify its repair.

What is RAG, and why does debugging need a separate process?

When an internal knowledge assistant gives a wrong answer, the language model is only one possible cause. A current document may never have reached the index, a paragraph may have been lost during extraction, or permissions may have excluded the relevant passage. Retrieval augmented generation, or RAG, supplies information retrieved from documents as input to a language model. Microsoft's explanation of RAG separates retrieval, augmentation and generation. Debugging should follow those boundaries: find the first point where the evidence becomes missing, unsuitable or misinterpreted.

Suppose an employee asks about a travel allowance and receives last year's amount. If the answer faithfully repeats an old document, generation may be working as configured while the business outcome is wrong. In another incident, the current amount reaches the model, but the answer omits a qualifying condition. These failures look similar in the interface and need different repairs. One belongs to document versions and ingestion; the other requires inspecting how the supplied context is used.

RAG debugging means reproducing a failed question with its evidence, then testing ingestion, access, retrieval, ranking, context assembly, generation and citations separately. It gives you a reason to change a setting rather than experimenting at random. A fluent response is insufficient: it must address the question, use valid evidence and stay within the caller's access scope.

A retrieval and generation fault tree for bad answers

Use this table as the decision tool attached to an incident. Work downwards and capture evidence at the first failed check. Later stages may also contain defects, but ranking cannot recover a sentence that never entered the index. A source being present means its valid version has searchable text and an indexed identity, not merely that somebody can find the file in a shared folder.

Check	Failure evidence	Inspect first
Is the source indexed?	Current document or necessary sentence missing	Ingestion, extracted text, document version
May the caller read it?	Permitted passage excluded by filtering	Identity, groups, tenant, permission version
Does search find the passage?	Absent from candidates or ranked too low	Query, chunk boundary, retrieval method
Does it reach final context?	Dropped or summarised after ranking	Context budget, assembly, summarisation
Does the answer use it correctly?	Date, condition or exception ignored	Generation with fixed evidence
Does the citation support the claim?	Wrong page or neighbouring section cited	Claim, passage and link alignment

The branching rule is simple: if correct, permitted evidence is absent from final context, investigate the retrieval chain. If it is present, investigate generation and citation handling. A source the caller may not read should remain absent; this is expected access control, not a retrieval defect. Run the same question with authorised and unauthorised test identities so improving the former never exposes evidence to the latter. Once you identify the failing stage, keep the other settings fixed while testing its repair.

How to capture a reproducible RAG trace

Give the request a persistent trace ID before investigating it. Link the question, search calls and final model input through that ID. Logging only the answer leaves too little evidence to explain the failure later. Separate the following fields in every incident record:

Question and conversation: Original question, previous messages actually used, language and any rewritten search query.
Access scope: Test user, effective groups, tenant, applied filter and permission synchronisation time.
Document provenance: Source ID, version, effective date, ingestion time and extracted-text version.
Retrieval trace: Candidate passage IDs, search method, ranks, scores and the list after reranking.
Generation input: Passages actually sent to the model, their order, prompt version, model version and settings.
Output and assessment: Answer, citations, expected evidence, failure category, latency and the attempted repair.

Do not assume the sources shown in the interface are identical to the context the model received. Your application may shorten, summarise or merge passages between retrieval and generation. Keep the material needed for replay in an authorised location with a defined retention period, and remove personal data before sharing a case. Redaction must preserve what the test needs to decide. Replacing a customer name consistently can work; deleting a relevant date or amount can invalidate the comparison. When replaying an old incident, identify the index and permission versions from that time rather than silently substituting current access.

Is the problem retrieval or generation?

First locate the expected passage by document ID and compare the extracted text with the source file. PDF columns can be interleaved, and a table can lose the heading that explains its values. In either case, the embedding, a numerical representation of text meaning, was built from incomplete input. Search settings cannot repair that text. Microsoft's document chunking guidance considers sentence structure, section context and overlap. Before changing chunk size, check that a necessary qualification remains attached to the statement it qualifies.

If the text is intact, run keyword, vector and hybrid retrieval with the same query and filter. A product code or policy number might rank poorly in vector search but match exactly in text search. Azure AI Search's hybrid search documentation describes running text and vector queries in parallel and merging their results with RRF. Do not treat raw scores from different ranking methods as interchangeable thresholds. Compare the expected passage's rank and whether it survives into final context.

Next, provide only the verified, permitted passage to the generator, keeping the question, prompt and model settings unchanged. If the answer becomes correct, return to retrieval, ranking or context assembly. If it remains wrong, test whether the generator misses a date, a negative statement or an exception. This experiment narrows the defect; success with a short, clean passage does not by itself explain behaviour with the original longer context.

For a third comparison, replay the original context in the same order and budget. If it contains both the right passage and a contradictory old version, inspect effective dates and version selection. When changing a prompt, specify the required behaviour: preserve conditions, identify contradictions and abstain when evidence is insufficient. Evaluate citations independently, because a correct amount linked to the wrong page still leaves the user unable to verify the decision.

Example: diagnosing an outdated travel allowance

Consider a hypothetical organisation with 1,200 documents and a 100-question evaluation set. Its new travel policy changes the accommodation limit from TRY 1,500 to TRY 2,000 for travel within the city. These amounts and all results below are illustrative. The assistant supplies both a correct answer and a correct citation on 70 questions. If the remaining 30 are grouped under a single “wrong answer” label, the team might propose moving directly to a larger model.

The fault tree reveals that 10 questions lack the current document in the index. For eight, the correct passage is retrieved as a candidate but falls outside the first five results. Seven have the necessary evidence in final context, yet the model drops the “within the city only” condition. Five give the correct amount but cite a page from the old policy. The first two categories total 18 cases, or 60% of failures. A model replacement does not address their underlying cause.

Complete the missing ingestion first. Then test a ranking change against its eight target cases and the other 92 questions. Examine the seven generation cases with fixed evidence, and the five citation cases through claim-to-page alignment. Measure each repair within its own failure category. Also check whether sending more passages increases latency or damages previously correct answers. These figures are neither a product benchmark nor a Hatson customer result.

Which RAG evaluation metrics show whether a fix works?

LangSmith's RAG evaluation tutorial evaluates answer correctness, response relevance, groundedness and retrieval relevance separately. Use those distinctions in a regression plan instead of relying on one overall score:

Label expected evidence: Record permitted source versions, required passages and an acceptable answer for each question. A main statement without its qualifying condition is incomplete evidence.
Measure retrieval: Count required passages present in the first five results. If one of two required passages is present, evidence coverage for that question is 50%; keep this separate from answer correctness.
Measure generation: Score preservation of dates, amounts and exceptions, support for each claim in final context, and citation alignment independently.
Add negative cases: Define expected behaviour for absent sources, contradictory versions and unauthorised questions. A fluent answer where the system should abstain is a failure.
Compare versions: Change one setting against the same fixed dataset. Report differences for Turkish, English, newly ingested documents and access roles rather than hiding them in an average.
Set a release gate: Agree acceptance limits before testing, block release on leakage, and track latency and cost alongside quality. Have people review a sample of model-graded cases.

Common mistakes

Saving the answer while losing the actual model context, making the incident impossible to reproduce.
Changing the model, chunk size and ranking together, so you cannot identify which change helped.
Disabling the permission filter to recover a document and introducing a leakage risk.
Combining answer accuracy and citation accuracy, allowing an unsupported reference to pass.
Testing only the failed question and missing regressions across languages or document versions.

How Hatson Tech builds this

For our RAG and semantic search solutions, we start with a source inventory, access scope and verified questions. We link retrieval traces, model inputs and citation alignment in the same incident record, then compare repairs against a versioned test set. See our guide to RAG evaluation metrics for measurement choices, permission-aware retrieval guide for access-related failures and hybrid search and reranking article for ranking options. Start with one failed question captured in this format and identify the stage that needs repair.

Frequently asked questions

What is RAG, and why can a RAG answer still be wrong?

RAG adds retrieved document content to a model's input so it can answer from that evidence. It does not guarantee that the documents are current, the right passages were found or every condition was preserved. Inspect the evidence reaching the model before attributing the failure to the model itself.

How do I debug a RAG application?

Start with a failed question and record its source versions, access scope, retrieved candidates and final context. Follow the fault tree to identify the first missing or misused piece of evidence. Keep other settings fixed when testing that stage's repair.

Is switching models enough to fix bad RAG answers?

A different model cannot retrieve a document missing from the index or restore text lost during extraction. Test the current generator with verified, permitted evidence first. Consider a model change only when the remaining defect concerns how valid context is interpreted, and check the full regression set.

Which RAG evaluation metrics should I track?

Separate retrieval evidence coverage, answer correctness, response relevance, groundedness and citation alignment. Track latency and cost alongside these quality measures. Report results by language and access role so a good average cannot hide a failing group.

Does a correct answer with a wrong citation pass the test?

It should fail the citation check even if its facts are correct. The link must identify the permitted source version and passage supporting the claim. Keep this failure category separate so a retrieval or generation score cannot conceal it.