# Evals before shipping: grade the rate, not the run (/docs/agent/evals-before-shipping) --- type: Procedure title: "Evals before shipping: grade the rate, not the run" description: "The eval discipline from The AI Agent Factory applied to one worker: why tests are not enough, the three depths of a run, the golden set built from real failures, the judge as a model that needs calibrating, the bar as a decision, the regression baseline and the drift schedule, hold-outs against Goodhart, an unscripted set of questions nobody wrote in advance, and what evals cannot prove." status: stable order: 12 generated: { by: "process:claude-code", at: 2026-09-09T07:45:00Z } verified: - { by: "human:rrizwan98", at: 2026-09-03T08:20:00Z } - { by: "human:rrizwan98", at: 2026-09-09T07:47:26Z } sources: - id: af-49 title: "Eval-Driven Development for AI Employees: A Multi-Track Crash Course. The AI Agent Factory, Panaversity, 2026" resource: https://agentfactory.panaversity.org/docs/eval-driven-development-crash-course - id: af-checker title: "Trusting the Checker: An Evals Crash Course. The AI Agent Factory, Panaversity, 2026" resource: https://agentfactory.panaversity.org/docs/trusting-the-checker-crash-course - id: raza-test-build title: "Owner's review of a support agent built from this record as a test, 9 September 2026 (attestation)" resource: "attestation: Raza Rizwan, owner; review conducted with process:claude-code on 2026-09-09 of a test build made by a coding agent from this record and its skill; not client work" ksor: audience: [public] owner: human:rrizwan98 approval: { by: "human:rrizwan98", at: 2026-09-09T07:47:26Z } trust_tier: human-reviewed build_id: sha256:4d4bff09bd0a1dcff0a0a50f20f7ef514fc471e2b1f9e81e0e7b6150b93c0793 dirty: true --- The second promotion turns your own check into an eval. The tests from [tests first](tests-first.md) verify code; an eval verifies behaviour. The two are complementary, not substitutes. [^af-49] The book's one-line summary of the whole discipline, kept here as the rule of the page: **an agent is a distribution, so grade the rate, not the run.** [^af-checker] ## A test verifies a property; an eval estimates behaviour An ordinary test verifies a specific expected property: this input gives that output. Run it twice and it answers the same way, so one green result carries real information. An agent breaks that assumption: give the same task to the same model twice and you can get two different runs, different tool calls, different phrasing, sometimes a different answer. So a single green run tells you almost nothing about the next run. An eval estimates how well a probabilistic system performs across representative cases, usually over repeated runs, and the estimate to start with is the pass rate. [^af-checker] None of the assumptions a unit test rests on hold for an agent. It is probabilistic, multi-step (a test that checks only the final response can pass on a run where every intermediate step did the wrong thing), tool-using, context-sensitive, and connected to systems with side effects. The book's estimate is blunt: a green suite of unit, integration and end-to-end tests catches roughly 10 to 30 percent of agent failures in production, because it catches code bugs and misses reasoning bugs. [^af-49] And the weakest evidence there is: "it worked in the demo". A demo is one run, on a task chosen because it demos well, watched by someone hoping it works. The arithmetic behind that: a loop whose steps each succeed 95 percent of the time finishes a 20-step run cleanly only about 36 percent of the time, so any single clean demo can come from a system that fails most real tasks. Demos persuade; evals inform. [^af-checker] ## Three depths of one run The agent's behaviour is its full execution path, not just its final response. Grading only the response is grading an exam by reading its last paragraph. [^af-49] | Depth | What it grades | What it catches | | --- | --- | --- | | 1. The answer | What the agent said or produced | Wrong answers, broken formats, made-up claims, wrong refusals | | 2. The actions | Which tools ran, with which arguments, in what order | The wrong file edited, the wrong command run, the lookup that never happened, the wrong argument passed | | 3. The trace | Everything observable: messages, tool calls in order, retries, wandering, the visible rationale | Bad process that happened to produce correct actions this time and will not next time | The book's example for depth 2: a support agent looks up a customer by email, gets three matches, picks the first without checking, and refunds the wrong customer. The output looks perfect. Only a tool-use eval asking "was the right argument passed?" catches it. [^af-49] One caution on depth 3: the visible rationale is evidence of what the agent did, not a reliable window into what it thought, because a model's written reasoning can leave out the factors that decided the answer. Grade the trace for observable process, wandering, missing evidence, unsafe ordering, never for the "true" reasoning behind it. [^af-checker] Cheap cases grade depth 1; the cases that protect you grade depths 2 and 3. The SDK records the trace by default, which is what makes the deeper depths possible. [^af-49] ## The judge is a model too For any grading richer than "did the tests run", your judge is a model reading text and giving an opinion, and it has failure modes of its own, worth knowing by name: [^af-checker] - **Leniency drift.** Judges pass borderline work, especially when the rubric is vague. A judge with no anchors says 8 out of 10 to everything. - **Self-preference.** A model grades output from its own family more kindly. Judging with a different model family is a safeguard, not a cure. - **Surface bias.** Longer, more confident, better-formatted answers score higher than they should, unless the rubric forces the judge to check specific facts. - **Drift.** The judge model updates underneath you, and yesterday's 95 is not today's 95. The bar did not move; the ruler did. None of this makes model judges useless. It makes them instruments that need calibration. [^af-checker] ## Every caught failure becomes a case Beginners invent test cases, and invented cases test what you imagined, not what happens. The right source is the ratchet: every caught failure becomes a permanent fix and also an eval case, so that next month's change cannot silently undo it. Three sourcing rules keep the set sharp: **failures first**, because real caught failures are proven to be reachable, with near-misses second; **cover the categories, not the volume**, twenty to forty cases spread across difficulty, a few easy ones the agent must never miss, a solid middle, and the hard ones (the disambiguations, the injections, the false greens); and **version-control the folder**, because the golden set is code. [^af-checker] What makes a dataset good, in the book's longer course, in order: representativeness weighted like real traffic; edge-case coverage, about 70 percent representative plus 30 percent edge cases carved out on purpose; difficulty tags so "85 percent overall" becomes "95 on easy, 80 on medium, 60 on hard"; ground-truth quality; source diversity; and version control. Five ways datasets fail: the imagination trap, easy-mode bias (the one that produces false confidence), the single-author problem, the stale dataset, and threshold inflation. [^af-49] For the worker in this room the first cases are the proven by-hand runs from [from plan to worker](from-plan-to-worker.md), and the hard cases are the failures the delegation map on [understand the work](understand-the-work.md) predicted. A case is one small file: the input, the depth to judge, the expected behaviour, and the patterns that must never appear, plus an origin line that points back at the failure that earned it. [^af-checker] The book's lab keeps the dataset in the repo as `datasets/golden.json`, with a README that documents its conventions and changes treated like API contract changes. Each example carries: [^af-49] | Field | What it holds | | --- | --- | | `task_id` | Unique id | | `category` | One of the agent's task categories | | `input` | The user's message | | `context` | The case facts the agent would have: account, plan, status | | `expected_behavior` | What the agent should do, in plain language | | `expected_tools` | Ordered list; names must come from the agent's tool registry | | `expected_response_traits` | Rubric items the response should satisfy | | `unacceptable_patterns` | Specific things the response must not contain | | `difficulty` | easy, medium or hard | | `origin` | The real failure or proven run this case came from, with its date | Write a validation script that checks every example has all required fields, `expected_tools` references only tools that exist, no two examples share an identical input, and the category distribution matches the target within 5 percent, and run it before every eval run. [^af-49] For the worker in this room the categories are the rows of your step table and the tool registry is the list of tool names on the [build page](build-the-text-agent.md). ## The runner: a loop, three runs per case, verdicts as files The runner that turns a folder of cases into a pass rate is not a framework. It is a loop, three runs per case, with a small program grading each result against the expected verdict. Three details are deliberate. The judge writes its verdict to a file and the runner grades the file, because parsing a verdict out of a friendly prose summary is fragile. Errors and fails are counted apart, because a judge that broke protocol and a judge that judged wrong are different problems with different fixes: one is a harness bug, the other a calibration finding. And the eval job runs nearly read-only, allowed to read the fixture and write that one verdict file and nothing else, so a fixture that tries to steer the judge can touch nothing. [^af-checker] The book's repo-level shape for output evals is a pytest-like framework with LLM-graded metrics and thresholds: a test case with the input, the agent's actual output, the expected behaviour and the context, asserted against relevancy and hallucination thresholds. The mental model is pytest, except that instead of `assert result == expected` the assertions are graded behaviour metrics with thresholds. [^af-49] Deterministic runs of the loop with a scripted model and no provider call are on [tests first](tests-first.md); they exercise the tool-use depth in an ordinary unit test before any graded eval runs. ## Calibrate the judge A judge without a rubric grades by mood. Two rules do most of the work. **Anchor every score with an example.** "4 = mostly correct" constrains nothing; "4 = the action and amount are right, but the timeline is vague" constrains everything, and the best anchors are real past runs pasted into the rubric. **Make the judge check facts, not impressions.** "Is this response good?" invites surface bias; "does the diff remove any test? does the fix touch only the named function?" are questions with findable answers. [^af-checker] Then the bar. **The bar is a decision, not a discovery.** There is no natural law that 95 is good and 94 is bad. Choose it per category by what a miss costs: for the cases where one miss ships damage, false greens and injections, the bar is all of them, every time; for tone and style, 8 of 10 may be fine. Writing the bar down, with the reasoning, is what turns "the checker said PASS" into a policy someone chose on purpose. [^af-checker] **Grade the grader.** The protocol is one afternoon: sample twenty graded items, composed on purpose with a deliberate share of fails and borderline items, because agreement on an all-obvious sample is high by chance; hide the judge's verdicts; grade them blind with the same rubric; then compare, in a four-cell table of judge-said against you-said. For a checker the cell that matters most is the **false pass**, bad work the judge approved, because that is the work that ships. Report four things: agreement overall, agreement by category, the false-pass count, and the false-fail count. The rough guide: above 9 in 10 overall with zero false passes on high-severity items, the judge is earning its place; any false pass on a case you consider obvious, stop trusting the number until you fix it. **Fix the rubric before the judge:** most disagreement is the rubric's fault. And you are the reference, not the gold standard; for high-stakes categories upgrade the reference to two people grading independently, or a domain expert. Re-run the protocol whenever the judge's model changes. [^af-checker] ## The regression suite, the baseline, and drift Any change to the system re-runs the set before the change ships: a new rule, an edited constitution, a reworded instruction, a model swap, a new skill version. For a personal project that is a habit; for a shared repo it is a job in CI that fails below the committed baseline, enforced by the same merge gate as the tests, so behaviour regresses as loudly as code does. This is the half most teams skip: the industry watches its agents and does not test them. A dashboard tells you the loop failed last night; a regression suite tells you the change would fail before you ship it. [^af-checker] A baseline records more than one number: the date, the judge model, the rubric version, the overall rate, the rate by category, and who approved it. Re-baseline whenever the set itself changes, in the same commit, because a new hard case makes the rate drop for the right reason. Two controls stop the rule from being misused: a baseline may only move down with an explicit written approval in the commit, and the old baseline stays in history next to the reasoning. A baseline is a reference measurement, not a passing line that follows the score around. [^af-checker] Agent behaviour has a second failure channel with no local cause: the model underneath you updated. Same prompt, same rules, different behaviour. The defence is scheduled measurement: run the full set on a schedule, nightly for active loops and weekly for quiet ones; commit the baseline and alert on drops, so silence means "still at baseline"; and re-calibrate the judge on model changes, because a drifted judge can report a steady 95 while the meaning of 95 changes under it. [^af-checker] ## Reading the numbers A rate arrived: 31 of 36, down from 34. Three habits keep it honest. **Re-run before you panic**, with the policy decided before you see the result (for example, any first-run miss triggers four more attempts, and the report shows 3 of 5, never just the final pass), and record every attempt. Flakiness that persists is its own finding. **Know what three runs can tell you:** three runs per case is a development setting, cheap, fast and rough, a smoke signal, not a stable estimate; scale the sample to the decision, more runs until the result stops moving for a release decision. **Read which cases failed before reading how many:** 31 of 36 with three tone cases down barely matters; 35 of 36 with the deleted-test case down is an emergency. Tier the suite by cost: a smoke set of the five or six highest-stakes cases on every change, the full set nightly, the hold-outs weekly. [^af-checker] The step most teams skip is naming the failure mode before changing anything: output level, tool level, trace level, retrieval level or safety level. Most production agent failures are not prompt failures; they are tool, retrieval or workflow failures, which is why changing prompts repeatedly does not help. Then rerun the whole suite, not just the failing case: "fixed the case" without "no regressions" is not a fix, it is a trade. [^af-49] The four causes by timing on [diagnose misbehaviour](diagnose-misbehaviour.md) are the same move for a whole worker. ## Staying honest The discipline has one enemy left, and it is the discipline itself. Goodhart's law: when a measure becomes a target, it stops being a good measure. The moment "keep the suite above 33 of 36" becomes the goal, prompts get tuned to the cases, rules get shaped to the fixtures, and the number climbs while the behaviour it was supposed to represent quietly stops being measured. Three cheap defences: [^af-checker] - **Hold-outs.** A handful of cases the loop's authors never tune against: written, sealed, run only on the weekly schedule. A gap opening between the tuned set and the hold-outs is Goodhart's law showing itself. - **Refresh from production.** New real failures keep becoming new cases. Retire a case only when the behaviour it tests no longer exists, a stronger case covers it, or the requirement changed; the high-severity ones stay permanently, however long they stay green. - **Never let the agent see the answer key.** The cases and fixtures live outside the loop's working context: not in the rules file, not in a skill the maker loads. And attack fixtures are live ammunition: an injection case contains real attack text, and a session that wanders into the fixtures folder can be steered by your own test data. Keep the folder out of everyday context; a deny rule in the harness is one line. What evals cannot prove: an eval suite bounds your confidence on the situations it contains. It cannot speak about the genuinely novel input, the failure shaped like nothing in the folder, the day the world changes faster than the set refreshes. A calibrated 35 of 36 is a strong statement about known territory and total silence about unknown territory. That is why nothing in this room removes the human gate. Evals shrink what reaches the gate and sharpen what the gate sees; they do not replace the person standing at it. [^af-checker] ## The unscripted set A test build from this record, a support worker on a fictional shop, passed every one of its scripted cases, 28 of 28, and then failed the natural questions its owner asked next, because every scripted case started with an identifier and no case had ever been written by someone who did not know the script. The scripted set was correct and the worker was not natural. What an eval set contains is what a worker becomes. [^raza-test-build] So beside the golden set, this room keeps a second folder: fifteen to twenty questions that nobody scripted, written by a person who has not read the spec, in the words a real user would use. Multi-intent messages that ask two things at once, one private and one public. Vague opening lines: "where is my order", "I have a problem", "can you help". Off-topic questions the worker should decline gracefully. Questions that mention a record from earlier in the conversation without repeating its identifier. The pass rate on this folder is reported separately from the golden set, and one rule is absolute: **a well-formed question that ends in a hard error is a fail**, whatever the answer would have been. The folder is refreshed from real conversations once the worker is live, and it is graded at depth 2, because "asked for an identifier it could have listed" is an action failure, not an answer failure. [^raza-test-build] The rules those failures produced are on [design the experience](design-the-experience.md). ## Your basic eval, in eight steps 1. Write the golden set first: ten to twenty cases from your proven runs, tagged easy, medium or hard, each with an origin line, and a few written to be hard, including one injection and one false green. 2. Write the unscripted set beside it, by someone who has not read the spec. 3. Grade at two depths at least: the answer, and the tool calls with their arguments. Grade the injection and false-green cases at the trace. 4. Anchor the rubric with real examples and make it ask about facts. Set a bar per category and write down why; the damage categories are all of them, every time. 5. Grade the grader once: twenty blind items, the four-cell table, zero false passes on the cases that matter. 6. Run several times per case and read which failed before how many. 7. Commit the baseline with its date, judge model and rubric version. Re-run on every change; move the baseline down only with a written reason. 8. Schedule the full set, seal a few hold-outs, and re-calibrate the judge when its model changes. Then continue to [diagnose misbehaviour](diagnose-misbehaviour.md), which is what you do with the cases that failed. [^af-49]: Eval-Driven Development for AI Employees: A Multi-Track Crash Course, Concepts 1, 2, 3, 5, 6, 9, 11 and 12, and Decision 1 of the lab. The AI Agent Factory, Panaversity, 2026. [^af-checker]: Trusting the Checker: An Evals Crash Course, Parts 1 to 6. The AI Agent Factory, Panaversity, 2026. [^raza-test-build]: Owner's review of a support agent built from this record as a test, 9 September 2026. Attestation recorded in this document's sources; the build was a test on fictional data, not client work.