Launch Is Where the Work Starts
Conventional software has a comforting property: it does the same thing tomorrow that it did today. An LLM system does not. The model behind it can be updated by a vendor without your involvement. The documents it retrieves go stale as the business changes around them. The questions users ask drift. And the system itself is non-deterministic at the sentence level, so the notion of a fixed correct output that a test can assert against does not straightforwardly exist. Every one of those properties breaks an assumption that ordinary monitoring is built on.
The result is a failure pattern we see repeatedly. A team launches a well-received assistant. For six weeks it works. Then quality erodes, slowly enough that nobody can name the day it changed. Usage drops without a corresponding rise in complaints, because users who lose confidence in a tool quietly stop using it rather than filing tickets. When somebody finally asks what happened, there is no answer available, because nothing was recorded that would let anyone compare this month's behaviour with the launch month's. The system was shipped and then, in the operational sense, abandoned.
Day-two discipline for AI is the combination of two things that are usually built by different people. Observability tells you what the system did: every request, its inputs, its retrieved documents, its cost, its latency, its outcome. Evaluation tells you whether what it did was any good. Neither is sufficient alone. Observability without evaluation gives you dashboards full of traffic and token counts that cannot answer the only question that matters. Evaluation without observability gives you a clean score on a test set that may no longer resemble what your users are actually asking.
This article is about wiring both together and keeping them honest. It assumes you already have an evaluation set, or are building one, and it deliberately does not repeat the metric definitions covered in RAG evaluation metrics. The last section connects the operational evidence to regulation, because the logs, evaluation records and human-oversight trails this discipline produces are close to what the EU AI Act's technical documentation and post-market monitoring expect, and they support accountability under Turkish data protection law as well. That link is the subject of the final section.
The Trace Is the Unit of Truth
The atomic record in an AI system is the trace: everything that happened in service of one user request, as a tree of spans. Retrieval is a span. Each tool call is a span. Each model call is a span. Reranking is a span. This is the same idea as distributed tracing in ordinary backend systems, and OpenTelemetry's semantic conventions for generative AI let you carry it on infrastructure your operations team already runs, rather than adopting a separate observability stack that only the AI team knows how to read.
What the trace must carry is specific, and the omissions are always the same. Record the prompt template identifier and its version, not the rendered prompt alone. Record the exact model identifier including version, and the sampling parameters. Record the identifiers of every retrieved document or chunk, with their scores and their final position in the context. Record input, output and, where applicable, reasoning token counts, the computed cost, and per-span latency. Record tool calls with their arguments and results. Record the outcome: completed, refused, filtered by a guardrail, failed, or abandoned by the user.
Two fields are worth calling out because teams skip them and regret it. The first is a stable conversation and user identifier, hashed if privacy requires, which is the only way to reconstruct a multi-turn failure or to answer a data subject's question about what the system did with their data. The second is the guardrail decision log: which checks ran, what each returned, and whether anything was blocked or rewritten. Without it you cannot distinguish a model that behaved well from a model that behaved badly and was caught, and those two situations demand completely different responses.
Volume forces sampling decisions, and the sensible default is asymmetric. Keep structural trace metadata, timings, costs, document identifiers, decisions, for every single request, because it is small. Sample full prompt and response payloads at something like 5 to 20 percent for routine traffic, and at 100 percent for anything that errored, was refused, was flagged by a guardrail or received negative user feedback. Retention of 30 to 90 days for full payloads and considerably longer for aggregates is a common shape, and the exact numbers should be set by your data protection position rather than by storage cost.
Cost, Latency and the Numbers That Move
Cost in an LLM system is not a monthly line item, it is a per-request property that varies by an order of magnitude across requests. Attribute it at the trace level and aggregate upward: cost per request, per conversation, per feature, per tenant and per user cohort. The distribution matters far more than the total, because these distributions are heavily skewed. It is routine for a small percentage of conversations, typically the long multi-turn ones with large retrieved contexts and repeated tool calls, to account for a large share of total spend.
Latency needs percentiles, and means are actively misleading here because generation time depends on output length, which has a long tail. Track time to first token separately from total completion time. Break the budget down by span so you can see how much of your p95 is retrieval, how much is reranking, how much is the model, and how much is your own orchestration code. In our experience the surprise is usually not the model: it is a serial chain of avoidable round trips, or a reranking stage that was never batched.
Set thresholds on the aggregates and alert on movement rather than on absolutes alone. A sudden 30 percent jump in average prompt tokens usually means a retrieval change is pulling in more context than intended, or a prompt edit added examples that nobody costed. A sudden drop in average output tokens often means the model started refusing or truncating. A rising rate of guardrail blocks may mean an attack, or it may mean a legitimate new use case that your rules were not written for. None of these are visible from quality metrics; they are visible from operational metrics.
Eval-Driven Development and the Pre-Merge Gate
Eval-driven development is the practical response to a system with no assertable correct output. You test that the answer scores above a threshold on a set of graded criteria, across a fixed set of cases, and that the score has not regressed against the last release. The evaluation set becomes the specification, and writing new cases becomes part of writing the feature. The discipline is the same as test-driven development, and the cultural resistance is the same too: it feels slow for a fortnight and then it is the only reason anyone can change anything with confidence.
The gate has to be fast enough to survive contact with a real development cycle. A pre-merge suite that takes 40 minutes will be skipped within a month. Our target is a tiered structure: a fast tier of 100 to 200 cases that runs on every pull request in under ten minutes; a full tier of 500 or more that runs nightly and on release candidates; and a specialised tier for safety and injection cases that runs whenever the prompt, the tool definitions or the retrieval scope changes, which connects directly to the attack surface described in AI security and prompt injection.
Everything that can be pinned must be pinned, or the gate becomes flaky and flaky gates get disabled. Temperature to zero for evaluation runs. Explicit model version, never a floating alias that a vendor can repoint. Prompt template version recorded in the run. Retrieval index snapshot identifier, so a change in the corpus does not masquerade as a change in the code. Judge model and rubric version. When a result sits near a threshold, re-run it three times before calling it a regression, and log the variance, because a case with high run-to-run variance is itself a defect worth fixing.
Treat the prompt as source code, because it is. It lives in the repository, it is versioned, changes to it go through review, and it is deployed through the same pipeline with the same ability to roll back. Teams that edit prompts in a vendor console and then wonder why quality moved are not doing engineering, and no observability stack can compensate for a change that was never recorded. The same applies to tool definitions, retrieval parameters and guardrail rules: if it changes behaviour, it is code, and it belongs behind the gate.
Judges, Calibration and Judge Drift
Automated evaluation at production cadence means a model grades the outputs, because human graders cannot run on every pull request. This works, with conditions. Ask the judge one narrow question at a time rather than for a global quality verdict. Use binary or three-point scales, because judges are markedly more self-consistent on coarse scales than on one-to-ten. Require a short written justification before the label and store it, since those justifications are how you diagnose a judge that has quietly started grading something other than what you asked. Consistency beats sophistication.
Calibration against human labels is not optional, and it is the step that separates a measurement from a number. Have a domain expert label 50 to 100 cases against the same rubric the judge receives, then measure agreement. Below roughly 80 percent on a binary decision the judge is not usable, and the correct response is almost always to sharpen the rubric with worked examples rather than to change models. Above about 90 percent you can run it unattended between periodic re-checks. Re-calibrate on a schedule, quarterly is a reasonable default, and immediately whenever the judge model, the rubric or the prompt template changes.
Judges drift in ways that are easy to miss because the failure looks like a change in your system rather than in your instrument. A vendor updates the judge model and your faithfulness scores move two points with no code change. A rubric is edited to clarify an edge case and quietly widens what counts as grounded. The defences are mechanical: pin the judge version, keep a frozen human-labelled calibration set, record the agreement figure alongside every reported metric, and refuse to compare two numbers produced by different judge configurations.
Three Kinds of Drift and How to Detect Each
The first kind is input drift: what users ask changes. New product lines generate questions your corpus never covered. A regulatory change shifts the topic mix. A marketing campaign brings a different population with different vocabulary. Detection is straightforward and cheap: embed incoming queries, maintain a rolling centroid and a topic-cluster distribution, and compare each week against a trailing baseline. Alert on distance beyond a threshold, and more usefully, alert on the appearance of a new cluster carrying more than a small percentage of traffic. Then look at it manually, because the interesting part is always which new cluster, not how far the centroid moved.
The second kind is model drift, which for most teams means the vendor changed something. Model behaviour changes with version updates, with silent routing changes behind an alias. If you pinned versions, you control when this happens. Detection is a canary evaluation: run a small fixed set, perhaps 50 cases, against the model on a schedule, several times a day if the system matters, and alert on movement. When you do choose to upgrade, run the full suite on both versions side by side and treat the migration as a release with a rollback plan.
The third kind is world drift, and it is the one nobody instruments. The model is fine and the users are the same, but the corpus no longer describes reality. A policy was replaced and the old version is still indexed. A price list is six months stale. A regulation changed and the internal guidance has not caught up. The system will answer these questions confidently and correctly with respect to its documents, and wrongly with respect to the world, which is the most dangerous hallucination category precisely because it is well grounded in a stale source.
World drift needs corpus-side instrumentation rather than model-side. Track the age distribution of retrieved chunks and alert when the median age of cited documents rises. Track the proportion of answers citing documents past their review date. Reconcile the index against source systems on a schedule and alert on documents present in the index but deleted or superseded upstream. And maintain a small set of evaluation cases whose correct answers are known to change over time, review them quarterly, and treat a passing score on a question whose real-world answer has changed as a failure rather than a success.
Online Evaluation, Feedback and the Living Dataset
Offline evaluation runs a fixed set of cases against a candidate version in a controlled environment. It is reproducible, cheap and safe, and it cannot tell you how the system performs on the questions your users will ask next week. Online evaluation runs against live traffic: sampled judging of real conversations, A/B or canary comparison between versions, and behavioural signals such as follow-up rate, conversation abandonment and whether the user rephrased the same question twice. You need both, and they answer different questions. Each one answers a question the other simply cannot.
Canary releases are the practical bridge. Route 5 to 10 percent of traffic to the new version, hold for long enough to accumulate a meaningful sample, which for a moderately used internal tool is usually days rather than hours, and compare quality and cost metrics between arms before proceeding. Define the rollback trigger before you start, in numbers, and automate it. A canary with no pre-agreed abort threshold is a slow release, not a safety mechanism, because the discussion about whether the numbers are bad enough will always be won by whoever wants to ship.
User feedback is the cheapest signal available and the most commonly wasted. A thumbs-down with no structure tells you almost nothing; a thumbs-down with three optional reasons, wrong information, missing information, or unhelpful phrasing, turns each one into a routable defect. Explicit feedback rates are typically low, often only a few percent of interactions, so treat implicit signals as first-class: repeated rephrasing, immediate escalation to a human, copying the answer, or abandoning mid-response. None of these is conclusive alone, and all of them are useful in aggregate for deciding which conversations a human should read.
Close the loop by feeding production failures back into the evaluation set. Every escalated incident becomes a labelled case within a week, with the correct answer and correct sources attached. Every new failure mode gets three or four cases, not one, so the fix cannot be overfitted to a single example. The dataset stops being a snapshot taken at launch and becomes the accumulated institutional memory of everything the system has got wrong, which is by a wide margin the most valuable artefact the project produces and the one most likely to be lost when the team changes.
Incident Response When the Model Belongs to Someone Else
AI incidents do not look like outages. The service is up, latency is normal, error rates are flat, and the system is confidently producing wrong or harmful answers. Your existing alerting will not fire, because nothing it measures has changed. That means AI systems need their own detection paths: quality-metric alerts on sampled online judging, guardrail-block rate anomalies, sudden shifts in refusal rate in either direction, and a low-friction internal reporting route so that the first person to notice can raise it without opening a formal ticket. Detection is the hard part.
The runbook needs to answer a question ordinary runbooks never face: what does rollback mean when the model is not yours? In practice you have four levers. Roll back the prompt and tool definitions to the previous version, which you can do immediately if they are in the repository. Roll back to the previously pinned model version, which you can do only if the vendor still serves it. Disable the specific capability, tool or retrieval scope implicated while leaving the rest running. Or fall back to a degraded but safe mode: retrieval with citations and no generated answer.
Write the post-incident record, and include the fields that are specific to this class of system: the prompt and model versions in effect, the retrieved document identifiers, the guardrail decisions, and the evaluation cases added as a result. That last item is the one that turns an incident into an improvement. An AI incident that closes without new cases in the evaluation set has taught the organisation nothing. This is also, close to the record-keeping a serious governance framework asks for, which is treated in depth in enterprise AI governance.
The Compliance Bridge: The Same Evidence, Twice
Here is the part that rarely appears in engineering write-ups and almost never in Turkish ones. The artefacts this discipline produces, versioned evaluation results, traces with model and prompt versions, drift monitoring, incident records and human-oversight logs, are substantially the same artefacts that AI regulation asks for. Teams usually build them twice: once as engineering, once as a compliance exercise conducted by a different department against a spreadsheet. Building them once, in the pipeline, is cheaper and produces evidence that is actually true, because it is generated by the system rather than described after the fact.
The EU AI Act is Regulation (EU) 2024/1689. It entered into force on 1 August 2024 and became applicable on 2 August 2026, with exceptions. Prohibited practices and AI-literacy obligations have applied since 2 February 2025, and GPAI model provider obligations since 2 August 2025. The Digital Omnibus on AI, Regulation (EU) 2026/1744 of 8 July 2026, entered into force on 27 July 2026 and moved Annex III standalone high-risk systems to 2 December 2027 and Annex I product-embedded high-risk systems to 2 August 2028. Article 50 transparency obligations were not delayed and took effect on 2 August 2026, with a grace period to 2 December 2026 for marking synthetic content in systems already on the market. The authoritative source is the European Commission's regulatory framework page.
What the high-risk regime expects, in engineering terms, is recognisable. Technical documentation describing what the system does and how it was validated maps onto your evaluation methodology, dataset description and recorded results. Automatic logging of events over the system's lifetime maps onto tracing with retention. Human oversight maps onto the review queues, escalation paths and the record that a person actually looked. Post-market monitoring maps onto drift detection, online evaluation and the incident process. The deferral to 2027 and 2028 is time to build this properly rather than a reason to wait, particularly since procurement questionnaires from EU customers already ask for it.
In Türkiye there is no dedicated AI law in force; regulation proceeds through existing instruments, principally Law No. 6698 on the protection of personal data. Articles 10 and 12 on the duty to inform and on data security are hard to satisfy without knowing what your system logged, retained and disclosed, and Article 11(g), the right not to be subject to a decision produced solely by automated processing, is unanswerable without an oversight record showing where a human intervened. KVKK's generative AI guide and its March 2026 note on agentic AI are guidance rather than binding rules, but they point at the same evidence.
How We Run Day Two at HatsonTech
Our default is that observability and the evaluation harness are built during the project, not afterwards, and that they are handed over with everything else. Concretely that means OpenTelemetry-based tracing with generative-AI attributes flowing into whatever the client's operations team already uses, an experiment and evaluation record so that runs are comparable months later, prompts and tool definitions versioned in the client's repository, and a tiered evaluation suite wired into their CI. None of it is exotic tooling. The value is that it exists on day one of production rather than being retrofitted after the first quality incident.
The pattern we see most consistently is that operational instrumentation catches problems before quality metrics do. Token counts and refusal rates move days before anyone notices that answers got worse, because users tolerate a lot before they complain. The second pattern is that world drift, the corpus going stale, causes more real damage in document-heavy systems than model changes ever have, which is why we instrument corpus age and superseded-document citation rates alongside the model metrics, and why the versioning discipline described in chunking at scale is an operational concern rather than an ingestion detail.
The honest cost: on a typical engagement this work adds roughly 10 to 20 percent to the initial build and pays it back the first time somebody needs to change the prompt without guessing. If you are running an AI system in production and cannot say which prompt version served a given answer last month, or cannot tell whether quality today differs from quality at launch, that gap is worth closing before the next feature. Our custom software engineering practice builds these systems with the instrumentation in place from the start, and can retrofit it onto a system that shipped without it.