agent reliability calculator
A pipeline of five steps that are each 95% right is only 77% right end to end. Checks and retries win most of that back, but only where they catch errors, and an error nobody catches is worse than one that goes to a person. List your pipeline's steps with how often each is right, how many of its errors the check after it catches, and how many retries it gets. You get the share of cases that come out right untouched, go to a person, or come out wrong without anyone noticing, and which step causes most of the damage.
89.0% right end to end, 8.1% wrong without anyone noticing
| step | right | missed | to a person | of wrong outputs | of escalations |
|---|---|---|---|---|---|
| Classify the document | 98.7% | 1.16% | 0.16% | 14% | 5% |
| Extract the fields | 98.0% | 1.62% | 0.36% | 19% | 12% |
| Match to the system of record | 98.8% | 0.39% | 0.78% | 5% | 26% |
| Decide (weakest) | 95.1% | 3.88% | 1.03% | 46% | 35% |
| Draft the response | 97.9% | 1.48% | 0.64% | 17% | 21% |
Fix first: Decide. It starts 46% of the wrong outputs. A check that catches 95% of its errors would cut unseen errors from 404 to 244 a month and send 76 more cases to a person.
Errors compound: 5 steps at 95% each are right 77.4% of the time end to end with no checks. Without checks, every step would need 99.8% accuracy for the whole pipeline to reach 99%. A missed error is assumed to carry through to a wrong output unless a later step sends the case to a person, and people are assumed to fix what they are sent.
questions
what counts as a check?
Anything after a step that can tell its output is wrong: a JSON schema, a sum that must add up, a lookup against the system of record, a rule, or a second model acting as a judge. Measure its catch rate the same way as accuracy: run it on outputs you know are wrong and count how many it flags. A judge's catch rate is the recall from the LLM judge agreement calculator.
why does a missed error count as a wrong output?
Because later steps build on it. A wrong field extracted at step two flows into the decision and the letter, and each looks fine on its own. The calculator assumes a missed error reaches the output unless a later step happens to send the case to a person, and that a person fixes what they are sent.
why is a retry not a fresh attempt?
A retry sees the same input, often with the same prompt, so it tends to fail the same way. Set how independent a retry is below 100% to reflect that: at 70%, a retry succeeds 70% as often as a first attempt. Feeding the check's error message back into the retry raises it; retrying an unreadable scan does not.
where should i add a check first?
At the step marked weakest: it starts the largest share of wrong outputs. That is usually a judgement step with no deterministic check after it. The result shows what a check catching 95% of its errors would change, and how many more cases it would send to a person, so you can size the review queue with the human review staffing calculator.
more free tools
- ai roi & token cost estimatorMonthly Claude API cost, cost per task and ROI for a workload
- eu ai act compliance auditorRisk tier, obligations and dates for one AI system
- agent workflow fit scorerIs this process a good fit for AI agents, and what shape?
- ai build vs buy calculatorThree-year cost: SaaS tool, custom agents, or keep it manual
- ai accuracy proof calculatorHow many test cases prove a vendor's accuracy claim?
- ai model routing calculatorWhat sending each request to the cheapest capable model saves
- human review staffing calculatorHow many reviewers an AI workflow's escalations need
- ai vendor scorecardWeighted comparison of AI vendors, with must-haves and a swing check
- gdpr dpia screener for aiDoes this AI use of personal data need a DPIA?
- agent loop cost calculatorWhat a tool-calling agent costs per task as its context grows
- llm judge agreement calculatorDoes your LLM judge agree with people beyond chance? Cohen's kappa
- escalation threshold calculatorAt what confidence should an AI workflow hand a case to a person?
- rag vs long context cost calculatorWhole corpus in a cached prompt, or retrieve chunks per query?
Want the numbers checked against your own process? Book a 20-minute call.