all free tools

llm eval a/b test calculator

A new prompt scores 93.8% on the golden set and the old one 90.3%. Is that a real gain or the luck of these 400 cases? Because both ran on the same cases, only the cases where they disagree tell you anything. Count the four boxes and the calculator gives the difference with its range, an exact test, whether a model swap made no difference that matters, and how many cases a fair comparison needs.

free, no sign-up, runs in your browseropen source on GitHub
the same cases, run through both

Run variant A (today's prompt or model) and variant B (the change) on the same golden set, grade both the same way, and count the cases in each box.

the bar

B is better, beyond chance

+3.5 ptsB − A
+0.7 pts to +6.3 ptsrange (95% confidence)
90.3%accuracy A
93.8%accuracy B
0.02exact McNemar p-value
32 (8.0%)cases where A and B disagree
A better0B better
23cases B fixed
9cases B broke
1,568cases to find a 2-point difference
3,133the same with two separate test sets, each

Read the 9 cases B broke before you ship. An average can rise while B fails cases A handled. If they are the expensive ones, a better score is still a regression; add them to the golden set so the next change is tested on them too.

Only the 32 cases where A and B disagree carry information about the difference; the 368 they agree on cancel out. The p-value is McNemar's exact test on those cases, and the range is the Bonett–Price adjusted Wald interval. Cases needed use Connor's formula at the disagreement rate you measured: changes that rarely flip an answer need fewer cases to judge.

questions

why a paired test?

Because A and B answered the same cases. A case both get right, or both get wrong, says nothing about which is better; only the cases where they disagree do, and McNemar's test uses only those. Treating the two scores as if they came from separate test sets throws the pairing away and needs about twice as many cases or more to see the same difference.

what does no difference that matters mean?

The whole range for B − A sits inside plus or minus the smallest difference you care about. That is the answer you want before switching to a cheaper model or a new model version: not that no difference was found, which a small test always says, but that any difference is smaller than your bar.

why read the cases b broke?

An average can rise while B fails cases A handled. If the broken cases are the expensive ones, such as a missed fraud flag or a wrong dosage, a better score is still a regression. Read every one and add them to the golden set so the next change is tested on them too.

what about run-to-run noise?

Model outputs can change between runs of the same prompt. Run A twice and count how often it disagrees with itself. If A against A flips about as many cases as A against B, the difference is noise. Fix the settings you can, such as temperature, and grade with the same judge or rubric for both.

how many cases do i need?

It depends on how often A and B disagree, not on accuracy. The calculator uses Connor's formula with the disagreement rate you measured: a change that flips 8% of answers needs about 1,600 cases to find a 2-point gain at 95% confidence and 80% power. A change that flips 2% needs about 400.

more free tools

Want the numbers checked against your own process? Book a 20-minute call.