ai accuracy proof calculator
"95% accurate" means little without the size of the test. Enter what a vendor measured and the calculator shows the range the true accuracy plausibly sits in, whether that proves their claim, and how many cases a fair test would need.
the test is too small to tell
Wilson score interval. The cases must look like your real work, including the hard ones, or the number proves nothing.
questions
why the wilson interval?
It stays accurate near 0% and 100%, where AI accuracy figures usually sit and where the simple normal approximation gives impossible ranges.
what is the zero-failure rule?
If a system passes every one of n tests, you can claim accuracy of at least p with confidence c once n ≥ ln(1−c)/ln(p). At 95% confidence that is 59 straight passes for 95% accuracy, and 299 for 99%.
does a big test set make the claim true?
Only if the cases look like your real work. A thousand easy cases prove nothing about your hard ones. Build the test set from your own data, including the exceptions.
more free tools
- ai roi & token cost estimatorMonthly Claude API cost, cost per task and ROI for a workload
- eu ai act compliance auditorRisk tier, obligations and dates for one AI system
- agent workflow fit scorerIs this process a good fit for AI agents, and what shape?
- ai build vs buy calculatorThree-year cost: SaaS tool, custom agents, or keep it manual
Want the numbers checked against your own process? Book a 20-minute call.