Hire the vendor that can prove production operation
Hire the AI vendor that can prove it can evaluate, deploy, operate, recover, and hand over the system you are buying. A polished demonstration, familiar logo, or senior team biography does not show who responds when output quality falls or a release fails.
The commercial risk appears after acceptance. Research on software economics reports estimates that maintenance absorbs about 1. That figure is not an AI project forecast, but it identifies the procurement error: scoring the build while leaving most operating work undefined.
A useful evaluation therefore treats production capability as evidence, not reputation. The vendor must pass representative evaluation, failure handling, deployment evidence, operating ownership, and exit tests.
A production proof scorecard replaces presentation quality
Use pass, conditional pass, and fail instead of weighted points. Weighting lets strong presentation compensate for missing operating evidence. A gate prevents that trade.
| Proof area | Acceptance test | Evidence the vendor must produce | What a bad answer sounds like |
|---|---|---|---|
| Representative evaluation | Run the proposed system against buyer-approved cases, including difficult and unacceptable outcomes | Case definitions, expected outcomes, raw results, error analysis, and approval record | "The demo shows the model understands your domain." |
| Failure handling | Trigger agreed dependency, data, and output failures in a controlled environment | Detection record, alert route, recovery steps, elapsed timeline, and incident record | "The platform handles failures automatically." |
| Deployment evidence | Release a change through the proposed delivery process and restore the prior state | Release record, approvals, checks, rollback evidence, and current operating measures | "Our engineers use industry best practices." |
| Operating ownership | Resolve a simulated production incident using the proposed support arrangement | Named decision owner, escalation route, communication record, and post-incident actions | "Support is included." |
| Exit | Give another team the materials needed to restore, release, and operate the system | Rebuild instructions, access inventory, operating procedures, exportable records, and completed handover test | "You own the code, so there is no lock-in." |
A vendor passes by producing the artifact under observed conditions. A document prepared before the exercise can support the test, but it cannot replace the event.
Picture a vendor whose only artifact for the operating-ownership row is a slide deck dated eight months earlier, built for a different prospect's pitch. The deck is evidence that the vendor once wrote a proposal, not that anyone can name who gets paged tonight.
This scorecard also makes comparison fair. A buyer evaluating several suppliers can give each the same cases, injected failures, evidence request, and exit condition. The decision then rests on observable delivery risk rather than presentation style.
Representative evaluation tests the purchased outcome
The evaluation set should reflect the work named in the contract. It needs ordinary cases, costly edge cases, prohibited outcomes, and inputs for which the system should decline or route work to a person.
The buyer owns approval of those cases. If the vendor selects the test material, the test measures what the vendor chose to demonstrate. Research into voluntary vendor disclosures found that self-authored material serving both evaluation and marketing purposes could not operate as a 2.
Acceptance should state what happens to every failed case. The choices are correction before launch, an agreed operational control, removal from scope, or buyer acceptance of the residual risk. "Accuracy is good" is not an acceptance decision.
A short procurement exercise can establish this discipline before the main contract. How should you test an AI vendor before hiring? describes how to make that exercise comparable without turning it into unpaid project delivery.
Failure exercises reveal whether recovery is real
Ask the vendor to demonstrate failures that matter to the purchased service. Suitable tests include unavailable dependencies, malformed input, delayed data, unacceptable output, failed releases, and revoked access.
The contract should define the expected observable sequence: detection, containment, communication, recovery, verification, and follow-up. It should also identify who can stop automated processing, approve a fallback, communicate with affected users, and restore service.
Require a sample incident record and a completed record from the exercise. Established production practice treats a postmortem as expected after a significant undesirable event rather than as punishment, according to Google's 3. A supplier that refuses to document a simulated failure is unlikely to become more transparent during a disputed outage.
When the vendor's engineer, mid-exercise, cannot say who has the authority to approve the fallback, the buyer has just observed the actual failure mode: not the outage, but the absence of anyone empowered to act during it.
Bad evidence is a generic disaster-recovery document with no relation to the proposed system. Good evidence shows what the vendor observed, what decision it made, how the service recovered, and what remains unresolved.
Deployment evidence shows whether changes are controllable
A production supplier should demonstrate a release and a reversal. The buyer needs the release record, approval path, automated and manual checks, configuration history, rollback result, and confirmation that the restored service behaves as expected.
Ask for operating figures from systems the proposed delivery team actually controls, with customer-sensitive details removed. DORA identifies 4 for measuring delivery outcomes. A vendor should be able to define its reporting window, show the underlying records, and explain what changed after poor results.
A slide containing industry benchmarks is not equivalent evidence. It says nothing about the team, service, or working practices covered by the proposed contract.
The cost model exposes obligations hidden by the build price
Compare bids using a modelled operating estimate alongside the implementation fee. The estimate below assumes a build fee of $300,000, an operating retainer of $8,000 per month, buyer operations labour costing $100 per hour for 40 hours per month, buyer incident labour costing $150 per hour across 8 incidents requiring 16 hours each, business fallback involving 20 employees for 6 hours per incident at $50 per hour, and an exit rehearsal requiring 160 buyer hours at $100 per hour. These are stated assumptions, not measurements from production hardware or a CalQuity Labs project.
| Cost item | Open arithmetic | Modelled estimate |
|---|---|---|
| Initial build | Assumed fee | $300,000 |
| Vendor operation | $8,000 × 12 months | $96,000 |
| Buyer operating labour | $100 × 40 hours × 12 months | $48,000 |
| Buyer incident labour | $150 × 8 incidents × 16 hours | $19,200 |
| Business fallback | $50 × 20 employees × 6 hours × 8 incidents | $48,000 |
| Exit rehearsal | $100 × 160 hours | $16,000 |
| Operating and exit total | $96,000 + $48,000 + $19,200 + $48,000 + $16,000 | $227,200 |
| Build plus modelled operating and exit cost | $300,000 + $227,200 | $527,200 |
The model does not predict the winning vendor's cost. It shows which assumptions procurement must test. Replace every assumption with the bidder's committed fee, staffing requirement, incident history range, recovery obligation, and handover effort.
A lower build bid can become the expensive choice if it transfers monitoring, incident coordination, manual fallback, and reconstruction work to the buyer. If the vendor will not price those duties, record them as buyer-held cost rather than treating them as free.
Demos, references, and résumés leave different costs unpriced
A demo is useful for confirming that the proposed interaction can exist. It does not price recovery, monitoring, change control, or exit. Use it as a scope check, not as proof of production operation.
References reveal how a supplier describes selected relationships. They can confirm commercial conduct, but the supplier chooses whom to offer. A reference who answers every question with a version of "they were great to work with" has told the buyer about the relationship, not about what happened the night a release broke production. Ask references about named obligations, missed service commitments, incident communication, disputed ownership, and handover. Do not convert a positive conversation into evidence that an untested system will meet your acceptance criteria.
Past performance requirements also have a cost. US federal rulemaking recognises qualifying past performance as a 5. Requiring a famous customer list can exclude capable suppliers before they can produce evidence.
The remedy is a paid, bounded proof stage with the same tests for every bidder. Newer firms can compete through observed performance, while established firms cannot substitute reputation for evidence.
Put operating ownership and incident clocks in the contract
"Post-launch support" is not an obligation. The contract should name the monitored outcomes, alert recipient, response authority, communication route, service restoration duty, reporting cadence, evidence-retention duty, and conditions that trigger corrective work.
For high-risk systems in scope of the EU AI Act, providers must establish documented post-market monitoring and actively analyse performance data 6. Even where that law does not apply, the mechanism is commercially useful: monitoring must identify the data collected, the person reviewing it, the decision threshold, and the required response.
Incident language needs clocks. NIS2 requires an early warning within 7 and a fuller notification within 7 for covered significant incidents. Those legal deadlines do not automatically govern every contract. They show why "promptly" is weak drafting. A buyer should set supplier notification early enough to meet its own operational and regulatory duties.
Ownership must survive staff changes. Can another team operate what your AI vendor built? provides the handover test. If custom components are included, require the reproducibility evidence in Require vendors to prove every custom model can be rebuilt.
Production proof cannot repair an undefined purchase
This method fails when the buyer cannot define representative work, unacceptable outcomes, or authority during an incident. A vendor cannot prove compliance with an acceptance test that procurement has left subjective.
A controlled exercise also cannot prove every production condition. It samples behaviour. Vendors can rehearse the known scenario, and rare failures can remain unseen. Continued monitoring, evidence retention, and periodic recovery exercises are still required after acceptance.
Strict track-record gates can exclude smaller suppliers with strong operating discipline. A bounded paid test is more informative than a logo threshold, but the buyer must fund it and provide realistic cases.
Exit evidence also decays. Instructions that worked at handover can become obsolete after service changes. Make restoration and handover repeatable contract events, with payment or renewal tied to successful completion. Otherwise the buyer learns whether exit works only after the relationship has already failed.