AI vendor evaluation checklist

Evaluating an AI vendor is not the same exercise as evaluating a database or a CRM, and running a general software checklist against one is how organisations end up owning risks nobody scored. The difference is that an AI system's output is probabilistic, its behaviour changes when the vendor updates a model, and the data you send it may end up somewhere you did not agree to.

This is the checklist for those differences. It assumes you already have a weighted scorecard method and adds the criteria that method does not know to ask about.

What are the key criteria for evaluating AI vendors?

Seven categories cover it: capability against your real inputs, model and data provenance, evaluation evidence, data handling, security and abuse controls, commercial terms, and exit. The first and the last are the ones most evaluations shortchange, and they are the two that decide whether you are still comfortable in eighteen months.

Score these as ordinary weighted criteria, with one exception noted below: some of them are gates rather than criteria, and a gate that trades off against price has stopped being a gate.

1. Capability against your inputs

Demos run on inputs chosen because they work. The only capability evidence worth scoring comes from your own data.

Can you run a proof of value on your own inputs before signing, including the ugly ones?
What is the measured accuracy on your evaluation set, not the vendor's benchmark?
How does it behave on inputs it should refuse or escalate?
What happens at the edges of your domain: unusual formats, mixed languages, low-quality scans, adversarial users?
Does performance hold at your volume and latency requirement?

If a vendor will not run a pilot against your data, that is itself the finding. Treat it as a gate.

2. Model and data provenance

You are inheriting decisions the vendor made before you arrived.

Which models sit underneath, from which providers, and are they named in the contract?
Are you notified before a model version changes, and can you pin a version?
What was the system trained or fine-tuned on, and can the vendor speak to the licensing of that data?
Who bears IP indemnity if an output infringes?
If the vendor is a thin layer over a frontier model, what do they add that you could not buy directly?

That last question is uncomfortable and worth asking plainly. Sometimes the answer is excellent (workflow, integrations, evaluation, domain data) and sometimes it reveals a margin with no engineering behind it.

3. Evaluation evidence

This is the category that separates serious AI vendors from the rest, and almost no generic checklist contains it.

Ask	A good answer looks like	A bad answer looks like
How do you measure quality?	A named metric, a held-out set, and a threshold they hold themselves to	Public benchmark scores, or customer satisfaction
How do you catch regressions?	Automated evals in CI, run before every model or prompt change	"Customers tell us"
What is your error rate?	A number, by category, with the tail cases described	Reluctance to name one
What happens when it is wrong?	Confidence signals, escalation path, human review hooks	The output is returned indistinguishably from a correct one
Can we see your evals?	Methodology shared, sometimes the set itself	Treated as an odd request

A vendor without an internal evaluation practice is not being dishonest when they say the product works. They genuinely do not know, and neither will you.

4. Data handling

Score these individually, because the answers are frequently different for each and vendors summarise them into one reassuring sentence.

Is your data used for training, by the vendor or by their upstream model provider? Get this in the contract, not in the FAQ.
What is the retention period for prompts, outputs and logs, and can you set it to zero?
Which sub-processors see the data, and in which jurisdictions?
Is there tenant isolation, and what exactly is isolated?
Can you meet deletion requests end to end, including from logs and caches?
Does the system's memory or retrieval index respect your existing permissions model, or does it flatten them?

That last point causes more incidents than any other item on this list. A retrieval system indexed with a service account will happily surface documents the asking user was never allowed to open.

5. Security and abuse controls

Standard assurances first: SOC 2 Type II or ISO 27001, penetration test summary, vulnerability disclosure process.
Prompt injection defences, especially if the system ingests untrusted content such as email, web pages or uploaded files.
What actions can the system take autonomously, and what requires a human? Write the blast radius down.
Are tool and API permissions scoped to least privilege?
Is there an audit log of inputs, outputs and actions taken, and is it exportable?
Rate limits and cost controls, so a runaway loop is a bounded incident rather than an invoice.

6. Commercial terms

How is it priced: seats, usage, outcomes? Model your cost at three times current volume before signing.
What happens to price if upstream model costs move, in either direction?
Is there a cap on overage, and who is notified?
What are the SLAs on availability, latency, and support response?
Is the vendor's funding and runway consistent with a multi-year dependency? For an early-stage AI vendor this is a real criterion, not a formality.

Model the three-year total, not the first-year price. The failure mode is identical to the one in a build versus buy analysis, where integration and second-year costs are omitted on the buy side and the maintenance tail is omitted on the build side.

7. Exit

The least glamorous section and the one that determines your negotiating position at every future renewal.

Can you export your data, your configuration, your prompts and your evaluation set, in a usable format?
Who owns fine-tuned models or embeddings derived from your data?
What is the notice period, and what happens to your data after termination?
How much re-integration work would switching require? Estimate it now, while you still have leverage.

Which of these are gates rather than scored criteria?

A gate eliminates a vendor regardless of how well they score elsewhere. Deciding which items are gates before you meet any vendor is what stops a favourite from being scored past a real problem.

How to run the evaluation

Write your evaluation set first, before you shortlist. Fifty to two hundred real inputs with known-good outputs. This is the single highest-leverage artifact in the whole process, it makes vendor comparison objective, and you keep it afterwards.
Set weights before demos. Weighting after you have seen the product is how a scorecard becomes a justification.
Run the same pilot with each finalist, same inputs, same success measure, scored independently by three to five evaluators.
Discuss only where scores diverge by two points or more.
Run a sensitivity check. If a 10 per cent shift in one weight changes the winner, that weight is your real decision and it should be defended explicitly.

The evaluation set is the part worth insisting on. Vendors change, models change, and the product you buy today will not behave identically in a year. A set of real inputs with known-good outputs is the only asset that lets you notice, and it is the most durable thing the whole exercise produces.

Frequently asked questions

What are the key criteria for evaluating an AI vendor?

Seven categories cover it: capability measured against your own inputs rather than a demo, model and data provenance, evaluation evidence, data handling, security and abuse controls, commercial terms, and exit. The first and last are the ones most evaluations shortchange, and they are the two that determine whether you are still comfortable eighteen months later.

How is evaluating an AI vendor different from evaluating other software?

Three differences. The output is probabilistic, so capability has to be measured statistically against real inputs rather than confirmed by a feature checklist. The behaviour changes when the vendor updates a model, so version pinning and change notification become contract terms. And the data you send may be used for training or retained in logs, so data handling needs its own scored section rather than a line in a security questionnaire.

What should be in an AI vendor questionnaire?

Ask whether your data is used for training by the vendor or their upstream model provider, what the retention period is for prompts, outputs and logs, which sub-processors see the data and in which jurisdictions, which models sit underneath and whether you can pin a version, how the vendor measures quality and catches regressions, what happens when the system is wrong, what actions it can take autonomously, whether there is an exportable audit log, and what you can take with you on exit.

Which evaluation criteria should be gates rather than scored?

For most organisations the gates are: training on your data without consent, inability to meet a jurisdictional requirement you are actually bound by, no audit log where the process is regulated, and refusal to run a pilot on your own inputs. A gate eliminates a vendor regardless of how well they score elsewhere, and deciding which items are gates before meeting any vendor is what stops a favourite from being scored past a real problem.

An AI vendor evaluation checklist