ai vendor scorecard

Vendor selections usually end in a weighted spreadsheet that nobody questions. This one does two extra things. A must-have rules a vendor out however well it scores elsewhere, and a swing check tells you which criteria decide the result, so you argue about the weights before the demos rather than after.

free, no sign-up, runs in your browseropen source on GitHub

the options

Option 1Option 2Option 3

score each one, 1 to 5

criterion	weight (0–5)	must-have	Vendor A	Vendor B	Build in-house
Accuracy on your own dataMeasured on a test set you built, not the vendor's demo
Security and complianceSOC 2 / ISO 27001, data residency, retention, no training on your data
Fit with your systemsAPIs, SSO, the systems of record it reads and writes
Three-year total costLicences, usage, implementation, the labour it leaves behind
Time to valueWeeks to a pilot on real cases
Audit trail and explainabilityCan you show why it produced each answer, and who approved it?
ConfigurabilityCan your team change prompts, rules and thresholds without a change request?
Exit and portabilityExport of data, prompts and evaluation sets; notice periods
Vendor viability and supportFunding, customers like you, SLA and named support

Build in-house leads

80/100 Build in-house
65/100 Vendor B
57/100 Vendor A

The winner holds if any single weight drops by a point.

A must-have scored 2 or less rules an option out, whatever its total. Scores are rescaled so all 1s is 0 and all 5s is 100.

questions

how is the score calculated?

A weighted average of the 1–5 scores, rescaled so all 1s is 0 and all 5s is 100. A weight of 0 leaves a criterion out.

what does the swing check do?

For each criterion it lowers the weight by one point and reranks. If the winner changes or ties, that criterion is listed: the decision rests on it, so it deserves the most evidence.

how should i score accuracy?

On a test set built from your own cases, with the hard ones included, run by every vendor under the same conditions. The accuracy proof calculator tells you how many cases you need for the result to mean something.

should building in-house be one of the options?

Usually, yes. It sets the bar on control, audit trail and exit, and the build vs buy calculator gives you its three-year cost.

more free tools

Want the numbers checked against your own process? Book a 20-minute call.