llm eval a/b test calculator
A new prompt scores 93.8% on the golden set and the old one 90.3%. Is that a real gain or the luck of these 400 cases? Because both ran on the same cases, only the cases where they disagree tell you anything. Count the four boxes and the calculator gives the difference with its range, an exact test, whether a model swap made no difference that matters, and how many cases a fair comparison needs.
B is better, beyond chance
Read the 9 cases B broke before you ship. An average can rise while B fails cases A handled. If they are the expensive ones, a better score is still a regression; add them to the golden set so the next change is tested on them too.
Only the 32 cases where A and B disagree carry information about the difference; the 368 they agree on cancel out. The p-value is McNemar's exact test on those cases, and the range is the Bonett–Price adjusted Wald interval. Cases needed use Connor's formula at the disagreement rate you measured: changes that rarely flip an answer need fewer cases to judge.
questions
why a paired test?
Because A and B answered the same cases. A case both get right, or both get wrong, says nothing about which is better; only the cases where they disagree do, and McNemar's test uses only those. Treating the two scores as if they came from separate test sets throws the pairing away and needs about twice as many cases or more to see the same difference.
what does no difference that matters mean?
The whole range for B − A sits inside plus or minus the smallest difference you care about. That is the answer you want before switching to a cheaper model or a new model version: not that no difference was found, which a small test always says, but that any difference is smaller than your bar.
why read the cases b broke?
An average can rise while B fails cases A handled. If the broken cases are the expensive ones, such as a missed fraud flag or a wrong dosage, a better score is still a regression. Read every one and add them to the golden set so the next change is tested on them too.
what about run-to-run noise?
Model outputs can change between runs of the same prompt. Run A twice and count how often it disagrees with itself. If A against A flips about as many cases as A against B, the difference is noise. Fix the settings you can, such as temperature, and grade with the same judge or rubric for both.
how many cases do i need?
It depends on how often A and B disagree, not on accuracy. The calculator uses Connor's formula with the disagreement rate you measured: a change that flips 8% of answers needs about 1,600 cases to find a 2-point gain at 95% confidence and 80% power. A change that flips 2% needs about 400.
more free tools
- ai roi & token cost estimatorMonthly Claude API cost, cost per task and ROI for a workload
- eu ai act compliance auditorRisk tier, obligations and dates for one AI system
- agent workflow fit scorerIs this process a good fit for AI agents, and what shape?
- ai build vs buy calculatorThree-year cost: SaaS tool, custom agents, or keep it manual
- ai accuracy proof calculatorHow many test cases prove a vendor's accuracy claim?
- ai model routing calculatorWhat sending each request to the cheapest capable model saves
- human review staffing calculatorHow many reviewers an AI workflow's escalations need
- ai vendor scorecardWeighted comparison of AI vendors, with must-haves and a swing check
- gdpr dpia screener for aiDoes this AI use of personal data need a DPIA?
- agent loop cost calculatorWhat a tool-calling agent costs per task as its context grows
- llm judge agreement calculatorDoes your LLM judge agree with people beyond chance? Cohen's kappa
- escalation threshold calculatorAt what confidence should an AI workflow hand a case to a person?
- rag vs long context cost calculatorWhole corpus in a cached prompt, or retrieve chunks per query?
- agent reliability calculatorEnd-to-end accuracy of a multi-step agent, and which step to fix
Want the numbers checked against your own process? Book a 20-minute call.