Ask for records created during delivery, not for the sale
The proof worth requesting from an AI engineering partner is evidence produced while doing the work, not evidence assembled to win it. Run records, evaluation failures, reversed decisions, and a handover another team has actually used reveal conduct that a polished case study cannot.
That distinction should shape the due diligence checklist, the contract, and acceptance. A proposal explains what the partner wants you to believe; work records show what happened when the system failed, when an assumption stopped holding, and when responsibility for a bad answer moved from one team to another. Picture the moment concretely: a support engineer opens a ticket weeks after go-live because a customer got a confident wrong answer, and the only way to find out whether that failure mode was known in advance is the run record from the week it first appeared, not a summary written after the complaint.
Request that underlying evidence before award, and give the partner reasonable redaction rights without surrendering your right to pick the samples. A seller who chooses every example is still just performing well. The exercise tests presentation, not delivery.
Sales material conceals the decisions that create delivery risk
Case studies and demonstrations are selected after the outcome is already known. They omit the abandoned approaches, the awkward failures, and the operational work that happened after launch. A demo deck built around the query that finally worked says nothing about the queries that were quietly cut from the script. Even an honest presentation gives the buyer little basis for telling repeatable practice apart from a single unusually successful project.
Meaningful scrutiny requires access to development information, internal evaluation findings, and documentation generated during the work itself. Output-only access cannot expose how a team selected its tests, interpreted a failure, or changed its approach midstream.
Research on AI audit access reaches the same conclusion: examining the development record, rather than the system's outputs alone, is what allows a targeted evaluation of the process behind it. 1
This matters because the buyer is contracting for judgment under uncertainty, and the only evidence of that judgment is the trail it leaves behind. If the engagement is still at the sourcing stage, How should you test an AI vendor before hiring? explains how to create a bounded exercise that produces one.
The evidence set must connect execution, evaluation, decisions, and transfer
Ask for the following evidence set from a completed engagement or a paid selection exercise, and require the related artifacts to share identifiers, dates, or references, a run record tagged with the same ID as the evaluation case it failed, say, or a decision log that cites the ticket where the reversal was recorded, so the story can be traced across them.
| Evidence to require | Acceptance test | Why fabrication is difficult | What absence tells you |
|---|---|---|---|
| Run records | The buyer selects a run from an index and traces its input, configuration, output, status, and follow-up action. Sensitive values may be redacted, but the sequence must remain inspectable. | A credible record must agree with evaluation results, incident notes, and decision dates. Producing a consistent history after the fact requires more than writing a success narrative. | The partner may be unable to reproduce outcomes, explain changes, or separate a tested result from a remembered result. |
| The evaluation set, including failures | The partner supplies the test cases, expected and observed outcomes, selection rationale, failure labels, and disposition. The buyer chooses samples for replay or review. | Failures expose the boundary between the original claim and actual performance. They also reveal whether the team changed the test after seeing the answer. | A score without its cases cannot establish coverage. A set containing only passing cases suggests selection for display. |
| A decision that was reversed | The record identifies the original choice, the evidence that contradicted it, who changed the decision, and the linked work before and after the change. | A reversal carries internal cost and often crosses several records. It is less useful as sales theatre because it admits that the initial judgment was wrong. | The partner may not preserve decision history, or may treat plans as commitments that evidence cannot overturn. |
| A handover another team has used | With appropriate permission, the buyer reviews the handover package and speaks with its recipient without the sales team directing the answers. The recipient explains a task completed from the material and a gap discovered during use. | Real use produces access requests, corrections, operating questions, and updates. A handover assembled for procurement lacks that residue. | The documentation may describe the system while leaving control with the original builders. |
The artifacts should reinforce each other rather than stand alone. A failed evaluation ought to point back to the run that produced it, a reversed decision to the failure that forced it, and a handover to the run or failure a receiving team would need to investigate something similar.
A folder containing every requested filename is still weak evidence if none of those cross-references actually hold up.
Case studies, demonstrations, pilots, and assurance reports answer narrower questions
A case study establishes only that the partner has permission to describe a project. It says nothing about how representative that result was, or how the team behaved when something failed.
A demonstration shows that a prepared path works in a prepared environment. Whether the result can be reproduced from retained records once an assumption changes is a different question, and one the demonstration cannot answer.
A proof of concept can generate stronger evidence if the buyer controls acceptance cases and preserves failed attempts. If success is the only permitted commercial outcome, the exercise rewards hiding uncertainty. An AI pilot must be allowed to fail covers the contractual reason to preserve that possibility.
An assurance report can answer questions about controls within its stated scope and period. It should not be treated as proof that a particular delivery team evaluated a particular AI system competently. General-use SOC reporting also carries less detail than the restricted report intended for informed parties, so procurement should request the report whose scope and exceptions can actually be reviewed. 2
Each of these has a purpose. The mistake is letting one stand in for the delivery record instead.
Put sampling rights and usable formats into the contract
The evidence requirement should survive procurement. A promise to provide documentation is too broad to accept or enforce.
Use language of this shape:
The supplier shall maintain run records, evaluation cases and results, material decision records, and operating handover materials created during the services. The customer may select samples for review. The supplier may redact protected information only where the remaining material preserves sequence, provenance, and the basis for the decision. Delivery in a format the customer can inspect is an acceptance condition.
Define the review event, the responsible recipient, the permitted redactions, the retention period, and the remedy for missing evidence. Name that recipient as an actual role, a contracts manager or a technical reviewer reporting to procurement, rather than leaving it as "the customer" in the abstract; a clause naming no one tends to go unenforced. The remedy might be correction before acceptance, payment retention tied to delivery, or an exit obligation. Choose it before work begins, while producing the records can still be built into normal delivery rather than bolted on afterward.
Public procurement guidance follows the same underlying principle: a contracting authority should verify supplier claims through evidence, certifications, visits, or audits where proportionate. 3
Self-attestation can remain the baseline for lower-risk claims. NIST guidance expressly treats it as one validation method alongside certification, site visits, and third-party assessment, with escalation based on risk. 4 Apply that escalation to delivery evidence: increase independent verification when failure would create greater operational, legal, or financial exposure.
For handover, acceptance should depend on demonstrated use by the receiving team. Can another team operate what your AI vendor built? provides the corresponding operating test.
Regulated systems require a stricter retention clause
For a high-risk system within the EU AI Act, evidence management can be a legal design requirement. Technical documentation must demonstrate compliance in a form authorities can assess. 5
The same regulation requires high-risk systems to support automatic event logging throughout the system lifetime. 5 Providers must retain automatically generated logs under their control for at least six months, subject to rules applicable to the intended purpose. 5
Do not paste those duties into every contract. Confirm the system classification, the parties' legal roles, and any longer retention requirement with counsel, then make the engineering partner identify which records it controls, which records the buyer controls, and how each will be exported before access ends.
By-product evidence can still be staged or misread
A disciplined seller can manufacture artifacts that look realistic. The defense is procedural, not visual: choose samples only after the artifact index has already been disclosed, check that identifiers actually cross-reference between records, replay a selected evaluation, and interview the handover recipient directly. A tidy folder that answers every item on the checklist proves nothing unless the links between its documents survive that kind of checking.
Confidentiality can also stop a partner from sharing prior work. Redaction solves part of that problem, but it cannot preserve every useful detail, and a partner bound by a previous client's confidentiality terms may be telling the truth when it says it cannot show you that handover. Where that is the case, commission a bounded paid exercise whose contract requires the evidence set as an output, rather than accepting a newly written case study as a substitute for records the seller genuinely cannot disclose.
A new partner may have no prior handover that fits your context at all. That is a real limitation, not grounds for automatic rejection. Replace the historical proof with an observed transfer exercise, in which a separate team uses the supplied material to complete an operating task while you watch what breaks.
Records still only prove past conduct within a defined scope. They do not guarantee future performance, legal compliance, or fitness for a different domain, so the contract still needs outcome acceptance, named responsibilities, access rights, and exit terms of its own. Evidence earns its keep when a failed acceptance test carries a consequence. A document sitting unread in the procurement file does not.