A signed manifest is necessary and insufficient
A vendor must rebuild every custom model from the handed-over inputs and pass a declared tolerance before the handover is accepted. A signed manifest proves the integrity and traceability of what it records. It does not prove that the record is complete, the training workflow still executes, or the reproduced model is equivalent to the delivered model.
This distinction belongs in the custom model handover requirements, not in an internal MLOps checklist. The buyer needs evidence that survives the vendor relationship. Repository access, a container image, model weights, and an experiment record can all be valid inputs to that evidence. None constitutes a successful rebuild.
The acceptance boundary is simple: the vendor identifies and signs the inputs, an executor rebuilds the model from those inputs, and the resulting model passes criteria agreed before the run.
The constraint map separates identity, provenance, execution, and equivalence
Treat rebuildability as a set of constraints with different proof artifacts. Combining them under "reproducibility" hides the exact failure the handover must catch.
| Constraint | Required evidence | Failure it catches |
|---|---|---|
| Artifact identity | Content digests for source, training data, base images, dependencies, configuration, evaluation data, and delivered weights | A path, tag, or filename resolves to different bytes |
| Provenance | A verified signature over a manifest connecting those digests to the declared build process | A recorded input or process description changed after attestation |
| Executability | A completed rebuild in a buyer-controlled or independently controlled environment, using only declared inputs | Missing credentials, undocumented preprocessing, unavailable packages, hidden services, and stale entrypoints |
| Model equivalence | A pass report against a predeclared metric, evaluation set, run policy, and tolerance | The pipeline executes but produces a materially different model |
The signed manifest covers artifact identity and provenance only when its references are content-addressed and its signature verifies. The in-toto specification defines an attestation as authenticated metadata about software artifacts, a definition that stops at metadata and never reaches a claim about whether the attested artifact is correct or reproducible. 1
Sigstore cosign makes the integrity boundary equally explicit. Verification checks that the container digest matches the signed digest. That confirms the identity of the container being verified, not that running it will reconstruct a model. 2
SLSA v1.2 provenance records the builder, process, and inputs. Its assurance levels then add protection against tampering with the build, provenance, or artifact. The lowest level requires only that provenance exist, so the presence of a provenance document cannot serve as rebuild acceptance. 3
Content digests must replace names wherever bytes affect training
A manifest entry such as dataset: training-final.parquet is documentation, not a pin. The same applies to a Git branch, container tag, object-store path, package range, or model alias. Each can continue resolving while its content changes.
A buyer's MLOps engineer, reviewing the handover package months later, finds that exact filename in the manifest and traces it to an object-store path that a vendor analyst has since overwritten with a corrected version of the same file, believing the old one to be an obsolete draft. The manifest still names it correctly. The bytes behind the name are no longer the ones the delivered model was trained on, and nothing in the manifest would have caught the substitution.
Pin every training-relevant object by content digest. DVC retrieves tracked data according to the file's content hash, which also determines its cache location. That is the required semantic for a handover: the identifier changes when the bytes change. 4
Apply the same rule to foundation-model revisions. Hugging Face Hub requires a full-length commit hash when a download is pinned to a commit; a short commit reference is insufficient. 5 A branch such as main records where to look, while a full commit identifies the selected revision.
The manifest should bind the source commit, data objects, preprocessing artifacts, tokenizer, starting checkpoint, dependency lockfiles, base image, training configuration, evaluation corpus, and final weights. It should also record the command or workflow entrypoint and the compute assumptions that affect execution.
For high-risk systems within the scope of Regulation (EU) 2024/1689, Annex IV specifically requires technical documentation to include the computational resources used for development, training, testing, and validation. 6 Even outside that regulatory scope, accelerator family, runtime, kernel libraries, and distributed topology belong in the manifest when changing them can alter the result.
A manifest cannot establish its own completeness. The reproduction run supplies that check by attempting to build from the declared set without relying on the vendor's original workspace.
The reproduction test must start outside the vendor's working environment
The executor should begin with an empty workspace and obtain every required artifact through the manifest. Undeclared network retrievals should fail the run. Secrets required only to reach the handover package can be supplied by the buyer, but a hidden vendor service cannot remain part of the training path.
The executor then verifies signatures and digests before invoking the declared entrypoint. A successful command is not yet a successful reproduction. The run record must preserve the resolved artifacts, environment identity, emitted model digest, evaluation outputs, and comparison against the acceptance profile.
Buyer control matters because a vendor can repeatedly demonstrate a rebuild inside an environment containing undeclared caches, local datasets, patched packages, or credentials. The test should be run by the buyer, by an independent assessor, or in infrastructure where the buyer controls initialization and captures the evidence.
An independent assessor running the declared entrypoint from a freshly provisioned machine watches the build reach for a preprocessing package from an internal registry the manifest never mentioned. On the vendor's own laptop, the same run had always succeeded silently, because that registry was already configured there and nobody had asked whether it should be.
This rebuild test is one component of the evidence package described in What proof should an AI engineering partner provide?. It should occur before handover acceptance while the vendor can still correct missing artifacts and instructions.
Declared tolerance turns stochastic training into a testable obligation
Bit-for-bit model identity is an unsuitable default for many training systems. PyTorch 2.8 states that complete reproducibility is not guaranteed across releases, commits, or platforms, even with identical seeds. It also identifies cuDNN convolution operations as a source of nondeterminism across separate executions. 7
The contract therefore needs an equivalence definition rather than an unqualified promise to produce "the same model." The acceptance profile should name the evaluation corpus by digest, the metric implementation by revision, the aggregation method, the direction of comparison, the permitted tolerance, the seed policy, and the allowed execution environment. These terms must be frozen before the reproduction output is evaluated.
The tolerance should follow the downstream decision the model supports. A classifier might be constrained by task metrics and class-specific error limits. A ranking model might require ordering metrics on a fixed query set. A generative system might require a suite containing deterministic checks, model-based evaluations, and human review. In every case, the metric code and evaluation inputs are part of the signed handover package.
Where run-to-run variance affects the acceptance result, the reproduction test should contain multiple independent runs and a declared statistical pass rule. MLPerf Training uses a set of N independent runs, with N selected according to observed benchmark variation, rather than treating one run as sufficient evidence. Its rules also identify high evaluation variance in LLM post-training because evaluation is nondeterministic and GRPO training is sensitive to small changes. 8
The required tolerance is not permission for an arbitrary regression. It is a boundary selected before execution. If the vendor can choose the metric, evaluation sample, or tolerance after seeing the rebuilt model, the exercise records an outcome without testing an obligation.
Picture a vendor's delivery lead presenting the rebuild results to the buyer's engineering director and proposing, only once the comparison numbers are in front of both of them, that the accuracy metric agreed at kickoff be swapped for a calibration metric the rebuilt model happens to pass. Whatever the merits of that metric in general, approving the swap after the run means the buyer never learned whether the originally agreed obligation was met.
Registry lineage and containers solve narrower problems
An MLflow Model Registry entry can connect a registered model to the experiment and run that produced it, while retaining version history and metadata. 9 That lineage is useful for locating the claimed origin. It does not show that the run remains executable from the transferred assets.
A container freezes a substantial part of the user-space environment. It can still depend on external datasets, mounted code, remote APIs, credentials, host drivers, accelerator behavior, or mutable image tags. The rebuild test exposes those undeclared dependencies because the container must operate in a newly initialized environment.
Enforcing deterministic algorithms can narrow variation, but it has a cost and does not create portability across every framework release or platform. PyTorch documents that deterministic operations are often slower than the nondeterministic operations they replace. 7 A buyer should require deterministic mode where exactness is technically available and useful, then use a declared behavioral tolerance where it is not.
Escrowing final weights provides continuity for inference, but only if the serving stack around them remains usable, and that continuity is narrower than it looks: the buyer still cannot retrain after data changes, repair a preprocessing defect, or produce an auditable successor model from a weights file alone. Weight escrow can be part of handover, but it cannot substitute for rebuilding.
Documentation review has the lowest execution cost because nobody has to run the training path. It also leaves the central question unanswered. A syntactically complete runbook may describe dependencies that are missing, inaccessible, or incompatible.
Put the rebuild obligation into the acceptance clause
A usable handover clause should define the evidence and the pass condition in operational terms:
The supplier shall deliver a signed manifest identifying by immutable digest every artifact required to build and evaluate the custom model, including source, data, preprocessing outputs, starting checkpoints, dependencies, configuration, execution environment, and evaluation assets. The supplier shall support an executed reproduction from a newly initialized, buyer-controlled or independently controlled environment. The reproduced model shall satisfy the evaluation procedure and tolerance agreed before execution. A verified manifest without a passing reproduction run does not satisfy handover acceptance.
Attach a machine-readable acceptance profile to that clause. It should declare the manifest schema, permitted artifact stores, signature verification method, build entrypoint, prohibited undeclared dependencies, environment constraints, evaluation procedure, variance policy, tolerance, and evidence-retention location.
Make failed execution an acceptance failure even if the delivered weights continue serving traffic. Otherwise the vendor can hand over an operational binary while leaving the buyer unable to create its successor.
Rebuildability also differs from operational transfer. On-call procedures, observability, rollback, capacity planning, and incident response are covered by Can another team operate what your AI vendor built?. A model can be reproducible and still be impractical for another team to operate.
A passing rebuild does not prove completeness beyond the tested boundary
The reproduction test proves that the declared package worked under the tested environment and met the declared tolerance. It does not prove data ownership, license compliance, privacy compliance, security, fairness, or fitness for workloads absent from the evaluation corpus.
A passing run also cannot guarantee future rebuilds if registries disappear, artifact access expires, hardware becomes unavailable, or external services change behavior. Long-term handover therefore requires the buyer to retain the referenced artifacts or hold enforceable access rights to them. A digest can detect replacement, but it cannot retrieve bytes that no longer exist.
The test can miss a hidden dependency if the supposedly clean environment already contains it. Environment initialization and network policy are part of the acceptance evidence for that reason. Vendor-controlled execution weakens the result even when every visible check passes.
Finally, the proof expires when a training input, workflow revision, execution environment, evaluation procedure, or tolerance changes. Each accepted model version needs its own signed manifest and executed reproduction record. Reusing an earlier model's paperwork preserves traceability to an obsolete build, not rebuildability of the model being handed over.