The Demo Works, So What Does That Prove?
A RAG pilot almost always demos well. The team picks ten questions they know the corpus can answer, the retriever returns the right passages, the model writes a clean paragraph, and the room agrees the thing works. Two weeks later the same system sits in front of two hundred real users asking questions nobody rehearsed, and the complaints start arriving. It missed a document everybody knows exists. It cited the 2019 version of a policy that was superseded twice. It answered confidently about something that is not in the corpus at all. Nothing in the code changed. What changed is that the question distribution stopped being curated, and the demo had measured nothing, so it predicted nothing.
The uncomfortable part is that most teams cannot say how much worse production is, because they never produced a number in the first place. Ask what retrieval recall was during the pilot and you get a shrug. Ask whether last Thursday's prompt edit made answers better or worse and you get an opinion delivered with confidence. Without measurement, every later decision, bigger chunks, a different embedding model, adding a reranker, a longer system prompt, is a coin flip dressed as engineering. We have watched teams spend six weeks tuning a pipeline and land, measurably, exactly where they started, because the only feedback loop in the room was how the last three answers happened to feel.
Evaluation is not a phase you run once before launch. It is the instrument panel that makes every later change safe to attempt. The investment is real: assembling a usable evaluation set for a mid-sized corpus typically takes three to six person-days up front, plus a few hours a month to keep it alive. In exchange you can say that a change lifted recall@10 from 0.71 to 0.84 while faithfulness stayed flat, which is a sentence you can take to a steering committee. This article is deliberately narrower than evaluating models in general, which compares model families on public benchmarks. Here the system under test is your pipeline, over your documents, answering your users.
Two Systems, Two Failure Modes
A RAG system is a compound machine with at least two independent stages. The retriever decides which passages enter the context window. The generator decides what to say given those passages. These stages fail for completely different reasons and respond to completely different fixes, and if you only ever measure the final answer you cannot tell them apart. The arithmetic is unforgiving because the stages multiply. If the retriever places a sufficient passage in context 80 percent of the time, and the model uses that passage faithfully 90 percent of the time when it is present, your ceiling is 72 percent, and no amount of prompt engineering will move it past the retrieval limit.
That multiplication is why end-to-end accuracy is a bad first metric. It is a product of factors, so a drop tells you the system got worse without telling you which factor moved. Worse, the two factors trade against each other in ways that hide problems. Widening the candidate set raises recall and lowers precision, which usually lifts end-to-end quality a little and quietly makes faithfulness harder because the model now has more plausible-looking irrelevant text to anchor on. You see a flat end-to-end number and conclude nothing changed, when in fact two things changed in opposite directions and you have no idea which one to keep.
The practical consequence is that you build two harnesses, not one. The retrieval harness needs no language model at all. It takes a question, runs the retriever, and compares the returned chunk identifiers against a labelled set of relevant chunks. It costs nothing per run, finishes in seconds on a few hundred questions, and can therefore run on every commit. The generation harness is slower and costs money per run because it involves inference and usually a judge model, so it runs less often, on a smaller set, and against fixed retrieved contexts so that retrieval noise does not contaminate the generation signal.
Retrieval Metrics: recall@k, precision@k, MRR and NDCG
Recall@k is the headline number for RAG, and if you only track one retrieval metric, track this one. It asks a simple question: for each test query, did at least one, or in multi-hop cases all, of the labelled relevant chunks appear in the top k results? The value of k is not arbitrary. It should match how many chunks you actually place in the context window after reranking, typically five to eight, plus a second measurement at your pre-rerank candidate depth, typically 20 to 100. The gap between recall@50 and recall@5 is precisely the amount of quality your ranking stage is throwing away, and it is usually the cheapest thing in the whole system to fix.
Precision@k matters less than newcomers expect and more than veterans admit. It matters less because a modern model tolerates one or two irrelevant passages without much damage. It matters more because context is a budget, and every irrelevant chunk you pass costs tokens, costs latency, and adds a surface for the model to anchor on the wrong thing. Precision degradation is also the mechanism behind the long-context temptation: teams stop tuning retrieval and just pass more text, which is exactly the trade-off examined in the long-context versus RAG comparison. Passing 60 chunks where 6 would do is not robustness, it is an unmeasured cost.
MRR and NDCG capture something recall cannot: where in the list the good passage landed. Mean reciprocal rank rewards putting the first relevant result at position one rather than position eight. NDCG additionally handles graded relevance, so a chunk that fully answers the question counts more than one that merely touches it, and it discounts by position so late hits contribute less. Rank matters even when recall is fine, because models attend unevenly across a long context and a relevant passage buried at position twelve is materially less likely to be used than the same passage at position two. Track recall for coverage and NDCG for ordering; they move independently and both are actionable.
Context Precision and Context Recall: Judging Passages, Not Ranks
Rank metrics assume you have a clean binary label for every chunk-query pair, which is expensive to produce and brittle when your chunking changes. Context precision and context recall, the pair popularised by frameworks like Ragas, sidestep that by working from the reference answer instead. Context recall decomposes the ground-truth answer into individual claims and asks what fraction of those claims is supported somewhere in the retrieved context. Context precision asks whether the chunks that do carry supporting material are ranked ahead of the ones that do not. Both are computed by a judge model rather than by string matching, which is what makes them robust to rewording and to chunk-boundary changes.
The cost profile is different, and you should treat these as a second tier rather than a replacement. A judged context-recall run over 300 questions with five chunks each involves roughly 1,500 judge calls, which takes minutes and costs real money, so it belongs in a nightly job rather than a pre-commit hook. The compensation is that these metrics survive re-chunking. When you change chunk size from 800 to 400 tokens, your hand-labelled chunk identifiers become meaningless overnight and recall@k has to be relabelled, while context recall keeps working because it was never tied to chunk identity in the first place.
The most useful signal comes from where the two families disagree. A query with recall@10 of 1.0 but context recall of 0.4 means you found one relevant document while the reference answer needed three: a multi-hop coverage failure that binary recall marked as a success. The reverse pattern, low recall@k and high context recall, usually means your relevance labels are stale or too strict, because the retriever found material that supports the answer through a document you never labelled. Both disagreements are worth reading manually. In our experience the first ten disagreements you inspect teach you more about your corpus than the next hundred aggregate runs.
Generation Metrics: Faithfulness, Relevance and Citations
Faithfulness, sometimes called groundedness, is the generation-side headline. The measurement pattern is claim decomposition: split the generated answer into atomic factual statements, then check each statement against the retrieved context and score the fraction that is supported. A five-sentence answer might decompose into eight claims of which seven are grounded, giving 0.875. This granularity matters because a single unsupported clause inside an otherwise correct answer is exactly the failure mode that destroys user trust, and a whole-answer binary score cannot see it. For systems where a wrong statement carries legal or financial consequence, we treat anything below 0.95 aggregate faithfulness as not shippable, and we look at the distribution rather than the mean.
Answer relevance is a separate axis and teams routinely conflate the two. A perfectly grounded answer can still fail by answering an adjacent question, by burying the answer in three paragraphs of preamble, or by refusing to commit when the context clearly supports a direct statement. The usual measurement is a judge scoring whether the response addresses the specific question asked, ideally on a three-point scale rather than one to ten, because judges are far more consistent on coarse scales. Track it alongside faithfulness, because the two pull against each other: prompts that push hard for grounding tend to produce hedged, evasive, low-relevance answers, and you need both numbers to see that trade happening.
Citation correctness is really three measurements wearing one name. First, does the cited span actually exist in the retrieved context, or did the model fabricate a reference identifier? Second, does the cited passage genuinely support the claim it is attached to, or is it merely topically nearby? Third, is it the best available source, or did the model cite a summary when the authoritative clause was also in context? We score the first two as citation precision and citation recall and track them separately, because fabricated identifiers are a hard blocker while suboptimal source selection is a tuning issue. In regulated domains, citation precision below 0.98 tends to be treated as a defect rather than a metric.
Refusals and the Out-of-Corpus Set
Every evaluation set we have seen that was built by an enthusiastic team contains only questions the corpus can answer, and every such set overstates quality by a wide margin. Real users ask about things you never indexed, things that were deleted, things adjacent to your domain, and things that sound like your domain but are not. You need a negative set: questions the system should decline. We aim for roughly 15 to 20 percent of the evaluation set to be unanswerable by design, split between clearly out-of-scope questions and near-miss questions whose vocabulary overlaps heavily with the corpus but whose answer is genuinely absent.
Negative questions produce two error rates that must be tracked separately because they have opposite fixes. The false answer rate is the fraction of unanswerable questions the system answered anyway, which is the hallucination surface that ends up in a screenshot on social media. The over-refusal rate is the fraction of answerable questions the system declined, which is the quieter failure that makes people stop using the tool without ever filing a complaint. Tightening the prompt or raising a similarity threshold trades one for the other almost linearly, and without both numbers you will optimise one into the ground.
Near-miss questions are where the real engineering happens, and they are worth constructing deliberately. Take a policy that covers full-time staff and ask about contractors. Take a specification that documents model A and ask about model B. Take a regulation that applies from one date and ask about the period before it. In each case the retriever will return high-similarity passages, because the vocabulary matches almost perfectly, and the generator must notice that the retrieved text does not actually cover the case asked about. Systems that score well on standard questions frequently collapse here, and this is the single highest-yield block of test cases we add to a maturing evaluation set.
The Golden Dataset as a Living Asset
Size first, because everyone asks. For a first production release we target 150 to 300 labelled items, which is enough to move a metric outside the noise band for a change of any real size. A mature system running weekly regressions wants 500 to 1,000. Below roughly 100 items your confidence intervals are so wide that a five-point movement means nothing, and you will chase noise. Stratify rather than sample uniformly: bucket by document type, by question type, by difficulty, and by whether the answer requires one source or several. Report metrics per stratum as well as in aggregate, because an average hides that you are excellent on policies and useless on tables.
Source the questions from reality, not from imagination. The best sources in order are support tickets and helpdesk logs, the search queries already running against whatever system this replaces, transcripts of the interviews you ran during scoping, and structured sessions where five to ten domain experts write the questions they actually get asked. Generating questions with a language model is acceptable for padding coverage of quiet corners of the corpus, but if more than about a third of your set is synthetic you are measuring how well your system answers questions written by a model that read the answer first, which is a much easier task than the real one.
Labelling deserves more rigour than it usually gets. Each item should carry the question, a reference answer, the identifiers of passages that support it, and a difficulty and category tag. Have two annotators independently label an overlapping 20 to 30 percent of the set and measure agreement before you trust any of it. If Cohen's kappa comes in below about 0.6, your relevance definition is ambiguous rather than your annotators being careless, and the fix is a sharper rubric with worked examples, not more training. Adjudicate disagreements in a short session and fold the resolutions back into the rubric.
Then treat the set as a living asset with an owner. Every production failure that gets escalated becomes a test case within a week, with the correct answer and the correct sources attached. Every new document type added to the corpus brings five to ten new items with it. A healthy set grows by roughly 10 to 20 percent per quarter in the first year and then stabilises. Version it in the same repository as the code, review changes to it in pull requests, and never quietly delete an item because it started failing. That last habit is the one thing that reliably turns an evaluation suite into decoration.
LLM-as-a-Judge, and How to Trust It
Faithfulness, relevance and citation support cannot be scored by string matching, and hiring humans to score every regression run is not viable at the cadence you need. So you use a model as the judge, which works far better than intuition suggests and far worse than the people selling it suggest. The design rules that matter are simple. Give the judge the question, the retrieved context and the answer, and ask for one narrow decision at a time rather than a global quality verdict. Use coarse scales, binary or three-point, because judges are noticeably more self-consistent on them. Require a short justification before the label, and log it, because those justifications are how you debug the judge later.
Calibration is the step teams skip and then regret. Take 50 to 100 items, have a human expert label them against the same rubric, and measure how often the judge agrees. Below about 80 percent agreement on a binary decision the judge is not usable and the rubric needs rewriting, not the model swapping. Above roughly 90 percent you can run it unattended between periodic checks. Watch for the standard biases: judges reward longer answers, they reward answers that echo the question's phrasing, and they are more generous to text written by the same model family. Position bias in pairwise comparisons is severe enough that you should always run both orderings and average.
Judges drift, and a drifting judge is worse than no judge because it moves your numbers without moving your system. Pin the judge model to a specific version and treat a version change as a code change requiring re-calibration. Keep the human-labelled calibration set frozen and re-run it whenever the judge version, the rubric or the prompt template changes, and record the agreement figure alongside every reported metric so nobody compares two numbers produced by different judges. This is one of several disciplines that belong to running the system rather than building it, which is the subject of evals and observability after launch.
Ablation and Regression Gates in CI
When an answer is wrong, the first question is always the same: did the retriever fail to supply the evidence, or did the generator fail to use it? The oracle-context ablation settles this in one run. Take the failing questions, replace the retrieved context with the labelled ground-truth passages, and regenerate. If the answers become correct, retrieval is your bottleneck and prompt work is wasted effort. If they stay wrong with perfect evidence in hand, the problem is generation: the prompt, the instruction hierarchy, the model choice, or a context window so crowded that the relevant passage is being ignored. We run this ablation on every failure batch before anyone is allowed to propose a fix.
When retrieval is the bottleneck, a second ablation localises it further. Measure recall at your full candidate depth, typically 50 or 100, and compare it against recall at your final depth of five to eight. If recall@100 is high but recall@5 is poor, the right document is being found and then ranked away, which is a ranking and fusion problem addressed directly in hybrid search and reranking. If recall@100 is also poor, the document is not in the candidate set at all, and the cause is upstream in parsing, chunking or indexing. These two branches lead to entirely different weeks of work, and guessing between them is the most common way teams waste a sprint.
Wire the results into CI as gates. The retrieval suite is cheap and deterministic, so it runs on every pull request that touches retrieval code, the index configuration or the chunking pipeline, and it finishes in under a minute on a few hundred questions. The generation suite runs nightly and on release candidates. Use two kinds of threshold: absolute floors that encode what the business needs, for example recall@5 at or above 0.85 and faithfulness at or above 0.95, and relative no-regression deltas that fail the build if any tracked metric drops more than two or three points against the previous release, which catches slow erosion that floors never notice.
A gate only works if failing it costs something. Decide in advance who can override, require the override to name the metric and the reason in the pull request, and put a hard expiry on it. The failure mode we see most often is not a bad gate, it is a gate that everyone learned to skip because it was flaky, usually because the generation suite was run with a non-zero temperature. Fix that by pinning temperature to zero for evaluation runs, pinning model and prompt versions, and re-running any borderline result three times before you call it a regression. A gate people trust is worth more than a gate that is technically stricter.
How We Run RAG Evaluation at HatsonTech
On our own products we build the evaluation set before the retriever. On caseon.ai and DiligenceAI the first deliverable in a retrieval workstream is a stratified question set with reference answers and labelled source passages, agreed with the domain experts who will eventually judge the system anyway. It feels slow for about a week and then pays for itself continuously, because from that point every architectural argument has a resolution procedure. Somebody proposes a different embedding model, we run it against the set, and the discussion is over in an afternoon instead of running for a fortnight on competing anecdotes.
What we consistently see is that retrieval fixes outrank model upgrades. On Turkish corpora in particular, the largest single jumps in recall have come from parsing and segmentation work rather than from swapping the generator, which is why we treat chunking at scale as a first-class engineering problem and why the domain-specific handling described in building RAG on Turkish legal data exists as a separate discipline. The second most common finding is that a modest reranker over a wider candidate set beats a larger generator, at a fraction of the running cost.
When we take on a RAG engagement, evaluation infrastructure is part of the scope rather than an optional line item, and the client keeps it: the dataset, the harness, the CI configuration and the calibration records, all in their repository, all runnable without us. A first evaluation build on a 50,000 to 200,000 document corpus typically takes two to three weeks alongside the pipeline work. If you are running a RAG system today and cannot state its recall@5, that is the gap worth closing first, and our RAG and semantic search practice is built around closing it.