agent evaluation framework
Six gates between a working demo and an agent you trust.
Each gate answers one question, produces evidence a risk committee can read, and has a bar agreed before the numbers come in. The gates keep running after go-live, because models, prompts and inputs keep changing.
the six gates
In order. A gate that fails sends you back, usually to the golden set or the step it points at, not forward with a caveat. The bars are starting points; agree yours with the people who own the risk before you measure.
- 01
define done
What does a correct output look like, and who says so?
Write down, per step and end to end, what correct means: which fields must match exactly, which can be paraphrased, which errors are critical and which are cosmetic. Then build the golden set: real cases, labelled by the people who do the work today, with the hard ones in it on purpose.
Two people label a sample independently first. Their agreement with each other is the ceiling: no model can be expected to agree with one reviewer more than a second reviewer does.
- measure
- Labeller agreement (Cohen's kappa) on a double-labelled sample; golden set size and mix.
- passes when
- Kappa between labellers of 0.8 or more on critical fields. At least 300 cases, with every known exception type represented, and between a fifth and a third of them hard.
- evidence
- The rubric, the golden set with its version, and the double-labelling results.
- work it out with
- AI accuracy proof calculator: how many cases prove the accuracy you need
- 02
offline evals
Is each step good enough on its own, and is the chain?
Run every step against the golden set and score it: field-level accuracy for extraction, precision and recall for classification, the share of critical errors for judgement steps. Then run the whole pipeline end to end, because errors compound: five steps at 95% are right about 77% of the time together.
Report the lower end of the confidence interval, not the point estimate. A step that is 97% right on 100 cases may be 92% right in truth.
- measure
- Per-step accuracy with Wilson intervals; end-to-end straight-through, escalation and silent-error rates; cost and latency per case.
- passes when
- Lower bound of each critical step at or above its target. Silent-error rate end to end below the rate the current manual process achieves.
- evidence
- An eval report per step and for the chain, tied to model, prompt and golden-set versions.
- work it out with
- Agent reliability calculator: how step accuracy, checks and retries add up end to end
- 03
calibrate the judge
Can an automated grader be trusted to score what people can't read?
Golden sets cover hundreds of cases; production runs thousands. An LLM judge scores the rest, but only after it has been measured against people on the same cases. Score agreement, and above all the share of real failures the judge catches.
Agreement looks high when most cases pass. Kappa corrects for that; the catch rate tells you directly how many failures slip past.
- measure
- Cohen's kappa between judge and human labels, with an interval; judge catch rate on known failures; false alarm rate.
- passes when
- Kappa of 0.6 or more for a judge that informs people; 0.8 or more, with a catch rate of 90% or more, before a judge gates anything automatically.
- evidence
- The judge prompt and version, the agreement report, and the disagreements reviewed by a person.
- work it out with
- LLM judge agreement calculator: kappa, catch rate and how many cases to label
- 04
shadow run
Does it hold up on live traffic, next to the people who do it today?
Run the pipeline on real incoming cases for two to four weeks without acting on its output. People keep doing the work; the system's answers are compared with theirs. This is where data drift, odd formats and the cases nobody put in the golden set show up.
Use the shadow data to set confidence bands: measure how often the model is right at each confidence level, and choose the threshold below which cases go to a person.
- measure
- Agreement with human outcomes by case type; accuracy per confidence band; escalation rate; cost and cycle time per case.
- passes when
- No case type worse than the manual baseline on critical errors. Accuracy falls as confidence falls (the model is calibrated). Projected review load fits the team.
- evidence
- The shadow comparison report, the confidence-band table, and the threshold chosen with its cost.
- work it out with
- Escalation threshold calculator: where to set the hand-off to a person
- 05
staged rollout
What happens when it starts acting on real cases?
Let the system act on a small slice first (one case type, one team, or 5% of volume) with every output reviewed. Widen the slice only when the bar for the current one has held for an agreed period. Each stage has written stop conditions, and anyone on the team can pull it.
Staff the review queue for the escalation rate you measured, plus a margin, or the queue becomes the new bottleneck.
- measure
- Reviewer override rate; critical errors found in review; escalation rate against forecast; queue wait time.
- passes when
- Override rate below the agreed bar (often 2% to 5%) for two consecutive weeks before each widening. Zero unresolved critical errors.
- evidence
- The rollout log: each stage's dates, volumes, metrics and the sign-off to widen.
- work it out with
- Human review staffing calculator: reviewers needed for the escalation volume
- 06
monitor and re-prove
Is it still as good as the day it went live?
Sample live outputs for human review every week, score the rest with the calibrated judge, and watch the input mix for drift. Every change to a model, a prompt, a tool or a policy re-runs the golden set before release, in CI, and fails the build if a critical metric drops.
Add every production error that review finds to the golden set. Over a year the set becomes the best description of the work there is.
- measure
- Weekly sampled accuracy; judge-scored accuracy; input drift; escalation and override rates; cost per case.
- passes when
- Alert when sampled accuracy's lower bound drops below the go-live bar, or escalation moves more than a third from its baseline. Golden-set regression blocks a release.
- evidence
- The monitoring dashboard, the weekly sample results, and the regression report attached to each release.
- work it out with
- Agent loop cost calculator: catch cost drift as tasks take more turns
what goes in the golden set
Every gate leans on it, so it decides how much the others are worth.
- Real cases from the last few months, not synthetic ones, with personal data handled under the same rules as production.
- Every exception type the team knows about, even the rare ones, at a higher share than in live traffic.
- Cases the current process got wrong, with the corrected answer: these show whether the system repeats people's mistakes.
- Labels from the people who do the work, against a written rubric, with the disagreements resolved and recorded.
- A frozen version for go-live decisions and a growing version for monitoring. Never tune prompts on the cases you report on.
what to re-run when something changes
Passing once is not the end. These are the changes that most often break an agent that used to work, and the gates each one sends you back through.
| change | re-run gates |
|---|---|
| New model or model version | 02, 03, 04 (short), 06 baseline reset |
| Prompt or tool definition change | 02, and 03 if the judge reads that step |
| New document type or case type | 01 (extend the golden set), 02, 04 for that type |
| Policy or threshold change | 02 end to end, 04's band table re-read |
| Drift alert or error spike | 06 investigation, then 01 and 02 with the new cases |
where it fits
delivery methodology
The four-stage pipeline these gates test: assess, govern, retrieve, orchestrate. Gates 02 to 05 run in its last delivery phase.
agentic use cases
The governance checklist for multi-agent banking systems. The evidence from these gates is what answers it.
regulation
High-risk systems under the EU AI Act need documented accuracy, robustness and post-market monitoring. Gates 02 and 06 produce that record; check your system's tier first.
the teardowns
Every agent architecture post on the blog ends with an evaluation plan built on these gates, with a golden set and an acceptance bar.
bring an agent that needs proving
A pilot that looks good, a vendor claiming 95%, or a risk committee asking for evidence. In 15 minutes we can work out which gates you have already passed and which you haven't.