Match the exercise to the delivery model
Test an AI vendor with a paid technical exercise that matches the delivery model you intend to buy. Use a detached exercise when the supplier will receive a brief and return completed artifacts. If its engineers will work inside your operation, run it there instead.
That distinction changes what the test can predict. A detached exercise shows whether a supplier can interpret a bounded brief, make reasonable assumptions, produce evidence, and hand over usable work: the conditions of arms-length delivery.
Embedded delivery runs on different conditions. The engineers must get context from busy operators, work through an approval path not built with them in mind, and keep moving when ownership is unclear. Picture someone partway through an embedded exercise who discovers the schema change they need signed off sits with a compliance reviewer nobody told them about: chase the reviewer, guess and proceed, or escalate to the buyer. That is what a detached brief can never surface.
A useful exercise reproduces the work relationship
A technical exercise for selecting an AI vendor should be a small contract, not a presentation disguised as one. The buyer supplies the problem, constraints, acceptance method, access rules, required artifacts, and deadline. The vendor supplies the work and the evidence needed to inspect it.
The brief should resemble the expected commercial interface. For arms-length delivery, give the supplier a fixed brief, controlled inputs, a question channel, and an artifact package to return. Do not rescue weak requirements analysis through informal coaching; the ability to identify ambiguity and record assumptions is part of the test.
For embedded delivery, place the supplier inside the relevant operating loop. Give controlled access to the people, policies, approval queues, existing interfaces, and incomplete information that shape the real work. Keep the technical scope small, but preserve the organizational conditions.
Do not score the demonstration alone. A polished demonstration can conceal manual preparation, excluded cases, or work that only the presenting engineer can operate. Federal AI procurement guidance follows the stronger principle: contract for enough information to evaluate vendor claims and manage risk rather than accepting those claims on faith. 1
Ask for the brief response, assumptions, evaluation inputs and outputs, error record, operating instructions, and handover package, the real one, naming who gets paged when the retrieval system goes down overnight, not a generic template. If the proposed system falls within the EU AI Act's high-risk regime, Article 13 requires instructions that are concise, complete, correct, and clear. Those instructions must also state the accuracy level and metrics against which the system was tested and validated. 2
Outside that regime, the same artifacts remain useful acceptance criteria. They let the buyer compare what was promised, what was tested, and what another team would need to operate.
The delivery boundary determines the right test
Use the intended contract shape as the decision rule.
If your team will specify outcomes and inspect deliveries, keep the exercise detached. Evaluate whether the vendor can turn a brief into an acceptable artifact without becoming dependent on continuous buyer intervention.
If the supplier will share planning, diagnosis, and operational decisions with your staff, embed the exercise instead. Evaluate how its engineers acquire context, surface disagreement, handle access delays, and leave decisions legible to the people accountable for the service.
A hybrid engagement needs both observations. Require a detached artifact delivery, then place a bounded change or investigation inside the operation, and do not average the scores together. A supplier that passes one mode and fails the other belongs only in the mode it passed.
Before selecting this route, decide whether a supplier is the right commercial form at all. Should you build, buy, or partner for enterprise AI? frames that earlier decision. The exercise should test a chosen relationship, not compensate for an unresolved sourcing strategy.
Questionnaires, pitches, and pilots answer different questions
A capability questionnaire tests whether the vendor can describe sensible practices. It is useful for excluding suppliers that cannot explain ownership, evaluation, security, or handover, but it does not show that the team assigned to your contract follows them.
References describe previous engagements under previous conditions, as the vendor's account team tells them. They expose patterns in commercial behavior, but do not test the specific engineers who would do your work.
Unpaid speculative work creates the wrong incentive: the vendor minimizes effort, withholds reusable material, or treats the exercise as sales support, and the buyer ends up inferring delivery quality from work neither party treated as delivery. UK employment guidance concerns applicants rather than vendor procurement, but its boundary is instructive: an unpaid work trial should not usually exceed one day, and work that is not genuinely for recruitment attracts a payment obligation. 3 The commercial rule is simpler. If you prescribe work and expect decision-grade output, pay for it.
A pilot tests more than a selection exercise. It can involve live users, operating costs, internal change, and production risk. Use it after identifying a preferred supplier, with explicit failure conditions. An AI pilot must be allowed to fail explains why a pilot without a credible stop decision becomes an implementation commitment.
The paid exercise sits between a sales process and a pilot. Its cost includes the supplier fee and the buyer's review time. A detached exercise consumes less operational attention; an embedded one consumes more, because staff must grant access, answer questions, and observe behavior. That additional cost buys evidence about embedded work only when the exercise preserves real operating constraints.
Score failures by what they reveal
A single pass or fail hides the reason a supplier struggled. Use a failure taxonomy that connects observed behavior to a buying decision.
| Test area | Evidence required | Failure signal | Contract response |
|---|---|---|---|
| Brief interpretation | Written assumptions and traceability from requirement to output | Silent assumptions or material requirements omitted | Reject arms-length delivery or require tighter buyer control |
| Evaluation | Reproducible test inputs, expected outcomes, actual outputs, and exceptions | A demonstration with no inspectable record | Do not accept performance claims |
| Failure handling | Recorded failure cases, impact, containment, and escalation route | Errors removed from the presentation or described as unlikely | Reject work affecting consequential decisions |
| Operating instructions | A nominated operator can follow the supplied procedure | The vendor says its engineers will operate everything | Price continuing dependency explicitly |
| Handover | Buyer-held artifacts can be inspected and transferred | Documentation is deferred until the main contract | Withhold award until handover succeeds |
| Embedded conduct | Decisions, blockers, and ownership are visible to operational staff | Progress depends on private conversations or constant executive intervention | Reject embedded delivery |
| Scope control | Variations and exclusions are recorded before acceptance | Unrequested features substitute for missed requirements | Treat as a delivery failure, not added value |
Define gate failures before the work begins. Security breaches, unauthorized data use, fabricated evidence, and inability to reproduce the submitted result should not be offset by strengths elsewhere.
Listen closely to how the vendor answers an adverse result. A useful answer identifies the failed case, the evidence available, the operational effect, and the proposed disposition. "The demonstration worked" is a bad answer because it does not address the failed case. "That edge case will be fixed later" is also weak, unless the vendor can define ownership, acceptance, and the condition that permits release.
Put payment, evidence, ownership, and exit in the exercise contract
The exercise agreement should state the fixed fee, payment trigger, scope, permitted access, acceptance procedure, and rework boundary. Payment should not depend on winning the main engagement, or the exercise remains speculative work with a different label.
Settle ownership before work starts: the buyer's data, the vendor's existing materials, newly created work, and any reusable components. Whether the evaluation scripts a vendor writes for your brief belong to you, or revert to their library once the exercise ends, goes unstated. The software engineering code of ethics calls for a fair agreement on ownership of contributed software, processes, research, writing, and other intellectual property before the contribution is made. 4
Require the vendor to return or destroy restricted material after the exercise. Prohibit reuse of buyer data, examples, and outputs outside the evaluation, and state whether either party may reuse general methods that contain no confidential information.
Make evidence delivery an acceptance condition. The buyer should receive enough material to reproduce the result, inspect excluded cases, understand assumptions, and identify manual steps. What proof should an AI engineering partner provide? gives the broader evidence package to carry into the main procurement.
Test exit during selection rather than promising it for later. Government procurement guidance recommends adding anti-lock-in principles when contractual requirements are established. 1 Lock-in exists when changing provider becomes difficult, slow, and disproportionately expensive, and long, inflexible contracts can create commercial lock-in independently of technical design. 5
A nominated internal team or replacement supplier should be able to inspect the submission, repeat the acceptance procedure, and identify what remains vendor-dependent. Can another team operate what your AI vendor built? turns that requirement into a handover test.
A detached exercise fails when context is part of the job
A detached paid exercise does not establish that engineers can work effectively inside your operation. It removes the access negotiations, shifting priorities, fragmented ownership, and operator judgment that embedded teams must handle, so strong artifact production in a clean environment provides weak evidence about embedded performance.
An embedded exercise also has limits. Its narrow scope cannot establish long-term staffing continuity, incident performance, sustained operating cost, or conduct during a prolonged failure. Those risks need separate contract terms, references, and production evidence. Make AI vendors prove they can run production systems addresses the evidence needed for that claim.
Protect the operation with restricted permissions, approved data, named decision owners, and a reversible scope, and preserve real queues and collaboration paths. If the buyer quietly accelerates approvals, hand-picks a dataset with no edge cases, or assigns the senior engineer who will disappear once the contract is signed, the exercise measures a temporary showcase rather than the engagement being purchased.