Use AI on Our Contracts Is Not a Task Definition
The most common opening line in an AI project is also the most expensive one: we want to use AI on our contracts. Six months and a real budget later there is a working retrieval pipeline, a demo everyone applauded, and no agreement on whether the output is good enough to put in front of the legal team. The team calls this a model problem and starts swapping models. It is not a model problem. Nobody ever wrote the sentence that says what correct means, so there is nothing the system can pass and nothing it can fail.
A scope document is a test, not a wish list. Hand it, together with the delivered system, to an engineer who attended none of the meetings, and ask them to accept or reject the delivery. If they can decide in an afternoon without asking a single clarifying question, you have a scope. If they come back asking what counts as correct here, you have a memo, and that gap is where most AI budget quietly dies. A testable task definition answers four questions: what goes in, what comes out, who consumes the output, and what a wrong answer costs.
Here is the rewrite, and its length is the point. For supplier agreements in Turkish and English, arriving as PDFs including scans, extract fourteen named fields: counterparty legal name, effective date, term length, renewal type, notice period in days, governing law, termination for convenience yes or no, liability cap value and currency, payment terms in days, price indexation clause present or absent, assignment restriction, confidentiality term in months, exclusivity flag, and signature date. Emit one record per agreement into the contract register, each field carrying a confidence score and a page reference to the sentence it came from.
The last two questions change the build more than the first two. If a contract manager reviews every record before it is saved, the system needs citations, a review queue and a fast correction path, and its acceptance bar can sit lower. If the record flows straight into a renewals calendar with no human in between, you are building a different system with a different risk profile. Price each wrong-answer class separately: a missing notice period is a queue item worth ten minutes, while a wrong notice period silently renews an agreement for another year. The asymmetry is sharper in a finance back office, as we describe in AI work in financial services.
Acceptance Criteria Is a Number Agreed Before Anyone Writes Code
It should be accurate is not an acceptance criterion, because it names no metric, no dataset and no threshold. Accurate on which measure: field-level exact match, record-level all-fields-correct, or recall on the single field that carries the money? Accurate on which documents: the twelve clean files in the demo folder, or the scanned 2019 addenda nobody wants to open? And accurate to what level: better than today, better than a junior analyst, or perfect? Until those three blanks are filled in, every progress meeting is a negotiation about impressions, and the most senior person in the room wins it.
Set the number against the current human baseline, never against 100 percent. Measuring the baseline is a two-day job, not a research project: take 100 to 200 items that people already processed, have a senior reviewer re-check them blind, and count the errors. In the projects we run, manual field extraction from messy documents lands around 88 to 95 percent at field level, and two competent reviewers agree on subjective classifications maybe 70 to 85 percent of the time. If your measured baseline is 91 percent, an acceptance target of 99 percent is not ambition. It is a plan to fail in month five.
A defensible target has four parts: metric, dataset, threshold and tolerance. Written out, it reads like this. Field-level accuracy of at least 93 percent across the fourteen fields, measured on the frozen 220-document evaluation set by exact match after normalisation of dates and amounts, with recall of at least 98 percent on notice period, and no more than 2 percent of records reaching the register with a wrong value at high confidence. The fields do not share one threshold. The threshold follows the cost of the error, and one uniform number across every field means nobody priced them.
Scope two numbers, not one: the quality threshold and the automation rate. A system that answers 40 percent of cases at 97 percent accuracy and defers the rest is a good system; a system that answers 100 percent of cases at 82 percent accuracy is usually a liability. Only the pair tells you what the thing is worth, because the automation rate is the multiplier inside every benefit calculation you will later make about measuring return on an AI investment. Write both into the acceptance criteria, along with the minimum automation rate below which the project does not pay for itself.
Build the Evaluation Set During Scoping, Not After
The evaluation set is a scoping deliverable, and its timing is not a matter of preference. An eval set written after a model exists is contaminated, and not subtly. The people writing it have seen outputs. They pick examples that feel representative, which in practice means examples the current system already handles, and they quietly resolve ambiguous cases in the direction the system leans. You end up with a ruler made of the same rubber as the thing you are measuring. Writing the set first also surfaces the moment a team discovers it does not agree on the task at all.
For a first evaluation set we typically build 150 to 300 items, enough for a five-point difference between two approaches to mean something and small enough that domain experts will finish it. Composition matters more than size. Roughly 60 percent should mirror the everyday distribution, about 25 percent should be known hard cases the team can name from experience, and the remaining 15 percent should be deliberately degenerate: a scan photographed at an angle, two agreements concatenated into one PDF, a document in the wrong language, an empty page. Write the strata into the scope so nobody can later call the set unfair.
Domain experts write the ground truth, not engineers, and the disagreement procedure is agreed before the first item is labelled. Have two annotators work independently over at least 20 to 30 percent of the set, then measure how often they differ. If two experienced people disagree on more than roughly 10 to 15 percent of items, the annotators are not the problem: the task definition is ambiguous and you rewrite it. Name one adjudicator with authority to settle the remainder and record the reasoning behind every adjudicated item, because those notes become the real specification. Budget 3 to 6 person-days of expert time for a 200-item set.
Freeze the set with a date and a version, store it where the build team cannot silently edit it, and hold back 20 to 30 percent as a blind set that one named person opens. Everything a team learns from a set it can see leaks into the system, which is why the visible portion tells you about progress and only the blind portion tells you about acceptance. The same artefact keeps earning after go-live, becoming the regression suite that anchors evaluation and observability in production and the first thing you rerun when a model or a prompt changes.
The Human-in-the-Loop Boundary, Priced by the Cost of a Wrong Answer
Draw the autonomy boundary as three bands and put every decision the system touches into exactly one of them: decides alone, drafts for a human to approve, never touches. The exercise takes about an hour and removes the most common source of late-stage conflict, which is a compliance or legal stakeholder discovering in month four that the system was designed to act rather than to suggest. Do it as a list of decisions, not features, because features hide decisions. One ordinary document-processing feature usually contains four or five separate decisions with genuinely different risk profiles.
The band is a function of three things: what a wrong answer costs, whether the action is reversible, and whether the error is detectable after the fact. Routing an inbound message to the right queue is cheap, reversible and visible within minutes, so it decides alone. Drafting a reply to a customer is expensive to get wrong but caught before it leaves the building, so it drafts for review. Approving a payment, changing a credit limit or asserting that a clause is compliant is expensive, slow to reverse and hard to detect, so it never decides. Accuracy does not buy its way across that line.
The mechanism that implements the boundary is confidence routing, and it needs a number in the scope document. Calibrate on the evaluation set, then route the lowest-confidence band to a human; in the systems we build that is commonly the bottom 15 to 25 percent of items at launch. Then size the review load in hours: if 6,000 documents a month arrive and 20 percent are routed for review at four minutes each, you have asked for roughly 80 hours a month of somebody's time. Review capacity nobody staffed is the most common reason a successful pilot never becomes a production process.
Write the boundary so it can move, and write the evidence that moves it. A line such as: the renewal-date decision may move from draft-for-review to automatic once 5,000 reviewed items have accumulated with an override rate under 0.5 percent across three consecutive months, on the signature of the process owner. Without that clause the boundary you drew for safety on day one becomes permanent, the review cost becomes permanent with it, and the benefit case dissolves. With it, autonomy is something the system earns with evidence rather than something a steering committee grants under schedule pressure.
Naming What Is Explicitly Out of Scope
The out-of-scope list is the cheapest document in the project and reliably the highest return per minute spent on it. It takes thirty to sixty minutes to write, it usually runs ten to twenty lines, and its only job is to make future disappointment impossible rather than merely regrettable. The rule is simple: put on the list every capability a reasonable person could assume is included after reading your task definition. Not the absurd requests, the plausible ones. Those are precisely the items that arrive as a small ask in week nine and consume three weeks of a fully committed team.
For the contract extraction example the list reads: no legal advice or risk opinion, only fields located in the text; no handwritten annotations; no languages beyond Turkish and English; no contracts before 2018 unless already digitised; no writes back into the ERP, register only; no clause drafting or redlining; no comparison against a clause library; no obligations calendar; no mobile interface; no bulk backfill of the archive in phase one. Each line is a sentence somebody would otherwise have assumed in good faith, and each has an implicit second half: not in this phase, and here is what it would take.
Two disciplines keep the list honest. First, each item carries a rough cost if someone wants it back in, even if that cost is a one-word band such as small, medium or large; a request with a price attached gets a decision, a request without one gets a debate. Second, the list is reviewed at every phase boundary and items either move in with budget or stay out with a written reason. Out-of-scope is not a refusal, it is a queue, and treating it as a queue stops the same argument restarting every fortnight with the same three stakeholders.
The Integration Inventory: Systems, Auth, Limits, Owners
Model work is rarely what makes a schedule slip. Access is. The integration inventory is one row per system with five columns you can fill in a single session: system name, what you need from it (read, write or both), authentication method, rate or volume limit, and the named human who can approve access. Add a sixth column most teams omit and later regret: the lead time to receive working credentials in a non-production environment, plus the date you filed the request. A row without an owner's name is not an inventory row, it is a hope, and hopes do not answer follow-up emails.
The lead times are the part worth taking seriously at scoping time. In the environments we work in, a read-only service account on an internal database typically takes 1 to 3 weeks, a service account on a core banking or hospital information system 4 to 8 weeks, a site-to-site VPN or private link 2 to 4 weeks, and single sign-on integration 3 to 6 weeks once somebody owns it. These run in parallel with everything else, but only if they start in week one. Filed in week six, they become the critical path and the team builds against mock data it will have to redo.
Rate limits deserve arithmetic in the scope document, not a note saying to be confirmed. A legacy endpoint capped at five requests per second gives you 432,000 calls a day in theory and far fewer in practice, so a 400,000-record backfill is a multi-day operation needing a checkpointed job and a conversation about maintenance windows. Add a data-readiness note to each row as well, because reachability and usability are different problems, and the gap between them is the subject of data preparation for AI projects. A system you can reach whose contents you cannot trust is a second project.
The Non-Functional Envelope and What Happens When It Fails
The non-functional envelope is four numbers, and vagueness in any of them gets the system redesigned later at full price. Latency: state a median and a tail, because the tail is what users remember. For an interactive retrieval-and-generation answer we typically design to a p50 of 2 to 4 seconds and a p95 under 8 to 10 seconds, and a p95 target under 2 seconds usually means streaming the answer, shrinking the candidate set, or dropping a reranking stage. Throughput: state the peak, not the average. Twenty thousand documents overnight in a six-hour window is a sustained one document per second with no room for a failed batch.
Availability: name the window and the degraded mode. There is a large cost difference between 99.9 percent around the clock and 99.5 percent between 08:00 and 20:00 on working days with a written manual fallback, and the second is what most internal processes actually need. Cost ceiling: agree a maximum spend per request or per document before the build starts, and treat it as a design constraint rather than a number you discover in the first invoice. A ceiling turns retrieval depth, model choice and reranking into trade-offs with a budget, the same discipline behind what an AI project actually costs.
Failure behaviour belongs in the scope document because it is a product decision, not an implementation detail. Three cases must be answered in writing. When retrieval returns nothing relevant, the system says it found nothing and hands the case to a human; it does not answer from the model's general knowledge, which is precisely how a confident and wrong answer reaches a user who cannot check it. When the model provider is unavailable, you choose in advance between queueing the work with a stated maximum delay, failing over to a smaller local model at a stated lower quality, or refusing gracefully and saying when to come back.
The third case is low confidence, and it quietly decides how much the system is trusted after month three. Define the threshold on the evaluation set, decide what happens below it, and make the behaviour visible: a partial result with the uncertain fields flagged is more useful to a reviewer than a blank screen or a silent guess. Add two operational rules while you are there: a per-user and per-tenant rate cap so one runaway loop cannot consume the monthly budget in an afternoon, and a kill switch a named on-call person can pull without a deployment.
The Phased Plan and a Kill Gate That Can Actually Fire
Three phases are enough for most projects. Discovery runs 2 to 4 calendar weeks and 4 to 8 person-weeks and produces the scope document, the frozen evaluation set, the measured baseline and the integration inventory. A pilot runs 6 to 10 weeks against real data with real users and either meets the acceptance number or does not. Hardening to production takes another 8 to 14 weeks, and that is where security review, monitoring, failover, review tooling and handover live. Anyone who tells you these collapse into one six-week engagement is selling a demo, and the demo was never the hard part.
The gate at the end of discovery only counts if it can fail. That means three things written down in advance. The evidence: which numbers, measured how, on which frozen set. The threshold: what result closes the gate as a stop rather than opening it. And the person: one named individual on the customer side with authority to halt the project and no incentive that depends on it continuing. A gate whose owner is the person whose bonus depends on go-live is theatre with a checklist attached. If nobody is named, a phase boundary is a presentation date, not a decision point.
Concretely, the discovery gate closes as a stop if any of the following is true: the measured human baseline is higher than the acceptance target you can plausibly reach, the annotators disagreed on more than 15 percent of items even after adjudication, field completeness in the source data sits below the level the task requires, two or more integrations have credential lead times outside the pilot window, or the benefit case depends on an automation rate the confidence distribution will not support. Each of those is a fact you can hold in week four, not an opinion.
Stopping at that gate is a success rather than a write-off, and it costs roughly 5 to 10 percent of the programme budget instead of all of it. Gartner's June 2025 press release, based on a poll of over 3,400 respondents, projected that over 40 percent of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Cancellation after real spend is expensive; cancellation at a gate is cheap. The same discipline governs the second gate, covered in moving a pilot into production.
The One-Page Scoping Template, Field by Field
Here is the template as plain prose, sixteen fields, in the order we fill them. Use case name: one line, a verb and an object, no adjectives, for example extract renewal terms from supplier agreements. Trigger and volume: what starts a run, and how many arrive per day, per month and at peak, taken from the historical number rather than an estimate. Input specification: the exact artefacts, their formats, their languages, their typical and worst-case sizes, and where they physically come from. Output specification: the exact shape of what is produced, field by field, including confidence and provenance, and where it lands.
Consumer and next action: who reads the output and what they do immediately afterwards, because that single answer determines both the interface and the acceptance bar. Cost of a wrong answer: written separately for each error class, in money, in hours or in regulatory exposure, as an order of magnitude rather than a paragraph of adjectives. Current baseline: the measured human performance, the method used, the sample size and the date. Acceptance threshold: the metric, the dataset, the number and any per-field exceptions, plus the minimum automation rate below which the project does not pay for itself.
Evaluation set: size, composition by stratum, who wrote the ground truth, how disagreements were resolved, the freeze date and the share held back as a blind set. Autonomy map: every decision the system participates in, each assigned to decides alone, drafts for review or never touches, with the evidence that would let it move up a band. Out of scope: ten to twenty explicit lines, each carrying a rough cost to bring it back in. Integration inventory: one row per system with what you need, authentication method, rate limit, named owner, credential lead time, request date and a data-readiness note.
Non-functional envelope: p50 and p95 latency, peak throughput, the availability window with its degraded mode spelled out, and the cost ceiling per request or per document. Failure and fallback behaviour: what happens on empty retrieval, on provider outage and on low confidence, plus rate caps and who may pull the kill switch. Data and compliance constraints: the lawful basis under Law No. 6698, where data may be processed and stored, retention periods, what must be masked before anything leaves the perimeter, and whether the use case touches special-category personal data. Phase plan and gates: three phases with dates, the evidence that closes each gate, and the named person allowed to stop.
How We Scope Projects at HatsonTech
We run scoping as a paid discovery of 2 to 4 weeks, and we do not quote a build before it, because a number produced without opening the data is a guess with a decimal point attached. The team is deliberately small: an engineer, a data person and someone who sits with the users for a day. The deliverables are the ones described above, and the one we care about most is the frozen evaluation set, because it is the only artefact that survives every later argument. If a client wants the build without the discovery, we usually say the scope is not ready yet.
What we have learned doing this repeatedly on RAG, extraction and classification systems is unglamorous. The task definition gets rewritten two or three times during discovery, and the second rewrite is where the real requirement appears. Annotation disagreement is the most informative signal we receive in the first fortnight. Credential lead time, not model quality, is the most frequent cause of a slipped pilot date. And the out-of-scope list, which takes under an hour, prevents more conflict than any steering committee. None of this requires a new model; it requires writing things down while they are still cheap to change.
We build these systems as custom software delivered against a written specification, which means the scope document is a contractual artefact rather than a slide in a kickoff deck. If discovery shows the human baseline is already strong, the data is not ready, or the benefit case depends on an automation rate the confidence distribution will not support, we say so and stop there. That is a smaller invoice than the alternative and a better outcome for both sides. If you already have a task definition written down, bring it, and we will spend the first session trying to break it.