Building an Agent Evaluation Dataset

An agent can write a convincing answer while taking the wrong action or changing the wrong record. Use task templates, held-out cases and state checks to detect this before release.

What is AI agent evaluation, and why build a dataset?

When a support agent tells a customer “I have corrected your address”, how do you know the job is done? Fluent wording is insufficient evidence. You need to see that the right order changed, other records stayed intact and any required approval was obtained. AI agent evaluation means running realistic tasks in a controlled environment and checking the results against agreed criteria. An evaluation dataset holds the requests, starting conditions and success rules needed to repeat those checks.

A case can span several tool calls, a clarification question or a handoff to a person. Which response is appropriate depends on the business policy. Anthropic's agent evaluation guide distinguishes the task, individual trial, execution trace and final environment state. That distinction makes it possible to identify an agent that gives a convincing answer while completing the wrong operation.

Agree on the intended outcome before collecting examples. Is the agent meant to complete a transaction or prepare a proposal for review? Asking a necessary question can be a success, as can refusing an unauthorised request. Clear definitions help you choose the version that completes the work within its boundaries and give reviewers something concrete to inspect.

Which behaviour should each check measure?

A single aggregate score can conceal different kinds of failure. Apply the following decision table to each case. A task may need several checks: proposing the right refund and making an unauthorised payment are separate observations. Keep serious violations as explicit release conditions rather than points that excellent wording can compensate for.

Situation	Success check	Additional control
Reading a record	Correct record and fields	No unauthorised records returned
Changing a record	Expected final database state	Other records remain unchanged
Ambiguous request	Necessary clarification requested	No change before clarification
Missing data or tool failure	Safe stop or handoff	No fabricated success claim
Adversarial tool output	Content treated as data	No forbidden call or leakage

Use code for objective state checks wherever possible. Give subjective dimensions, such as communication quality, a clear grading rubric. The original τ-bench paper evaluates tool-using agents by comparing the database state after a conversation with the annotated goal state and also measures consistency across trials. Adapt that outcome-checking principle to your order, CRM or document workflow. Its benchmark score does not set the acceptance target for your application.

How to write a task and failure case template

The following record structure provides a task/failure case template you can use with a spreadsheet or a versioned dataset. Keep each case's data, checks and explanation together. A reviewer should understand what would pass before running the agent. Rename the suggested fields to match your existing infrastructure, while preserving the information they hold:

Case ID and group: A permanent identifier, task family, language, risk and difficulty; group different descriptions of the same underlying incident.
Request and starting environment: The user message, relevant records, document version and tool responses; the initial conditions must be reproducible.
Identity and permissions: The user's role, accessible records, available tools and operations that require approval.
Expected outcome: The correct final state, required clarification or handoff; allow more than one valid outcome where appropriate.
Forbidden actions: Explicit failure conditions such as unapproved sending, changing another record, sharing data or repeating a payment.
Verification rule: State assertions, a response rubric and the check's version; keep the reference result out of the agent's input.
Failure record: The observed call, departure from expected behaviour, failure category, severity and reproduction steps.
Provenance and maintenance: The source and date of the incident, policy version, dataset split and maintenance owner.

For example, case ADDRESS-017 starts with two orders placed on the same day; the fixture has no creation times to establish their sequence. “Change the address on my latest order” does not identify which one they mean. The expected outcome is to ask for the order ID and leave both addresses unchanged. If a record changes, the failure entry stores the wrong-record category, affected order ID and call arguments. Add a counterpart in which the customer supplies a valid ID and the permitted operation can succeed; refusing every request must not earn a good score.

Test the verifier before the agent. Deliberately change the wrong record, change the correct one and produce only a success message. If the check cannot distinguish these outcomes, fix it before using the score. Where an LLM grades response quality, compare its decisions with human judgements on traces containing only the necessary data. Resolve meaningful disagreements before relying on automated grades for a release decision.

How to collect realistic cases and keep a held-out set

Start with support requests, operator corrections and previous incident records. Remove names, contact details and live identifiers while preserving the relationships required by the task; reconstruct records with artificial values where needed. Build a coverage list for task type, language, ambiguity, tool availability and risk. Common routine work should be represented alongside rare failures with significant consequences.

Anthropic's tool-design guidance recommends realistic tasks paired with verifiable outcomes. A question that merely asks the agent to name a tool offers little evidence about whether it can do the job. Include retrieval, record matching and decision-making in the cases. Record confusing tool descriptions as a separate failure category so a model change does not obscure a tool-design problem.

Keep examples used to refine prompts and tool descriptions apart from the held-out cases used for release decisions. A translation or paraphrase of one support incident should not enter a different split. Split by incident group. Once you tune against a held-out example, treat it as development data and replace it appropriately. LangSmith's evaluation concepts describe storing inputs, reference outputs and metadata with examples, and distinguish curated offline evaluation from monitoring production runs.

Restore the database, tool responses, memory and caches to their starting conditions for every trial. Label results with dataset, model, prompt, tool-contract and grader versions. An environment that failed to initialise should be reported as an infrastructure error, separately from a poor agent decision. Give the baseline and candidate the same information and permissions; otherwise the comparison measures more than the change you intended to assess.

Hypothetical example: comparing versions with 120 cases

Suppose you prepare 120 independent case families for a support agent: 60 routine lookups, 25 ambiguous requests, 20 tool failures and 15 adversarial inputs. These are illustrative numbers, not customer results or measured industry performance. Assign 80 cases to development and 40 to release testing at group level, with all four categories represented in both splits.

Run each of the 40 held-out cases three times for both versions, producing 120 trials per version. Suppose the previous version passes 102 trials and the candidate passes 111: trial-level success is 85% and 92.5%. However, if the candidate sends an email without approval in two trials, the aggregate improvement is insufficient for acceptance. Zero unauthorised sends is the release condition agreed in advance for this example.

If the previous version passes all three trials on 29 of the 40 cases and the candidate does so on 32, observed all-trial consistency is 72.5% and 80%. Do not present the repetitions as 120 independent tasks: they belong to the same 40 cases. Inspect which tasks regressed even if latency and cost improved. A small held-out set alone cannot establish the probability of failure across live traffic, particularly for uncommon events.

An agent evaluation test plan

Validate the checks: Use known correct and incorrect outcomes, including valid alternatives, to identify verifiers that accept mistakes or reject legitimate solutions.
Compare under equal conditions: Run the previous version or a fixed workflow alongside the candidate, using the same trial count and budget.
Report by slice: Show success, attempted forbidden actions, completed violations, tool errors, handoffs, latency and cost separately for each task category.
Read traces: Review failures and a selection of passes; distinguish retrieval, tool selection, argument, planning and grading errors before deciding on a fix.
Repeat security cases: Rerun adversarial tasks after prompt, tool or policy changes and turn understood production incidents into regression cases.

OWASP's AI agent security cheat sheet recommends repeatable tests for tool misuse, approval bypass and data exfiltration. A denied forbidden call may show that the execution control worked; track the agent's attempt separately. When business policy changes, review the expected outcomes of existing cases and record why they changed. Stable scores on obsolete cases provide little assurance about the new workflow.

Common mistakes

Treating a fluent success message as proof of a completed operation.
Placing paraphrases of one incident in development and release splits.
Requiring one exact tool sequence when other valid routes exist.
Counting safe clarification or refusal as failure by default.
Hiding forbidden actions inside the average success score.
Reusing an old dataset after policy changes without reviewing expectations.

How Hatson Tech builds this

We start AI agent development work by defining task boundaries, the starting environment and acceptance criteria. We design evaluation records together with tool contracts and the release process. For connecting these checks to monitoring, see our guide to AI observability and evaluation; for narrowing tool responsibilities, see our article on agent tool selection. The first deliverable is an agreement on the evidence the team needs to accept the next version.

Frequently asked questions

What is AI agent evaluation?

It is the process of checking a tool-using agent against expected outcomes and permission rules on defined tasks. An apparently correct response is insufficient to pass a case; relevant record states and external effects also need verification.

How do I build an agent evaluation dataset?

List common tasks and failures with significant consequences, then turn their starting conditions into reproducible case records. Specify success and failure conditions for every record. Validate the checks with known outcomes before splitting examples into development and release sets.

How many tasks are needed for agent evaluation?

There is no universal number: task diversity and the size of the error you want to detect determine the coverage needed. Begin with a representative core and add missing slices as you learn. A high score on a small set does not prove rare failures will be absent in production.

How does a held-out test set differ from a development set?

Use the development set to refine prompts and tool descriptions; the held-out set provides an independent check of the candidate release. Do not put differently worded copies of one incident on both sides. Once you tune against a held-out example, treat it as development data.

Is LLM-as-judge sufficient for agent evaluation?

It can help with subjective dimensions such as communication quality, but it cannot by itself establish that a record changed correctly or approval existed. Use direct system checks for those properties. Compare model scores with clear rubrics and human decisions to investigate false accepts and false rejects.