Keep generative AI outside deterministic control loops

Put a hard authority boundary between inference and actuation

Keep every generative model outside any path that can directly actuate equipment. It may summarize process state, recommend a response, or draft a configuration change. A deterministic control system must execute the response, and an independently verified protection layer must retain final authority over hazardous equipment.

The AI OT control loop boundary is an authority boundary. Network segmentation helps enforce it, but the decisive question is whether any model-produced event can become a PLC output, distributed control system setpoint, safety instrumented system input, motion command, or protection bypass without a separate human decision.

This distinction is missing from broad lists of manufacturing AI use cases. Predictive maintenance and digital twins can remain observational. They cross the boundary when their outputs acquire write authority. Generative AI, predictive AI, and automation solve different problems explains why text generation, prediction, and deterministic execution require different control contracts.

The constraint map assigns authority before selecting a model

Treat the architecture as five layers with different evidence requirements. Do not begin with a model and bolt on controls after integration.

The first two layers can use generative AI. The last three cannot depend on it for a safety claim.

Consider a reliability engineer reviewing an overnight alert that flags rising vibration on a feed pump bearing. The model's recommendation can open a maintenance case, attach the supporting vibration trace, and rank the finding against other open work. It must not suppress the pump's protection, increase an operating limit, or restart the equipment on its own initiative, even when the engineer is offsite and the case sits unopened until morning. The economic decision belongs to the workflow described in Does predictive maintenance AI actually save money?; the actuation boundary remains unchanged whether the recommendation saves money or not.

Non-deterministic output cannot support a repeatable safety claim

A safety function needs a testable relationship between an input condition, logic, output action, and failure state. The protection team must be able to inject specified conditions, observe the response, detect latent failures, and repeat that procedure after maintenance or modification.

HSE guidance makes proof testing part of the integrity argument because the probability of failure on demand or failure rate depends critically on test frequency and the test's ability to reveal previously hidden failures. ¹

Generative APIs provide a different contract. OpenAI documents that Chat Completions and Completions are non-deterministic by default, so identical requests can return different outputs. ² Seed and system fingerprint controls still leave a stated chance of different responses even when request parameters and the fingerprint match. ²

Temperature zero does not convert that contract into a safety-certified logic solver. A JSON schema can constrain syntax. A tool allowlist can constrain which function names are available. Neither proves that a generated setpoint is correct for the current process state or that every hazardous input produces the required safe output.

If a deterministic guard completely validates the model's proposed action, the guard is the controller carrying the safety burden. The model contributes optimization or explanation inside the guard's permitted region. If the guard does not completely validate the action, some safety-relevant behavior still depends on probabilistic output.

That distinction decides where certification work belongs.

The protection layer must survive loss of both AI and basic control

The safety instrumented system should have its own specified inputs, logic, outputs, diagnostics, proof-test procedure, and safe failure behavior. It must not call the model, consume the model's interpretation as its sole process input, or depend on the proposal database being available.

HSE states the separation requirement directly: protective systems should be sufficiently independent of the control system or other protective systems. ¹ US federal risk-management rules likewise name alarms, safety instrumented systems, and detection hardware as active measures that detect and respond to process deviations. ³

Independence is more than separate application processes. A generative service and a protection function can still share failure modes through a sensor, network switch, identity provider, time source, power supply, deployment pipeline, or write-capable gateway.

The architecture therefore needs a negative dependency test: remove the model service, its data store, its credentials, and its network segment. The protection function must still detect its specified conditions and drive the process to its defined safe state.

The basic control system also needs predictable degradation. Losing AI assistance may remove a diagnosis or recommendation. It must not prevent the control system from holding its operating state, entering a deterministic fallback, or accepting an operator command through the approved interface.

Drafted changes need a transaction boundary, not a tool call

A model can draft a change safely when the draft enters the same controlled path as a human-authored change. The artifact should identify the target, current value, proposed value, evidence used, applicable operating mode, expiry condition, and rollback action.

The model should write that artifact to a staging system with no route to the controller. A deterministic service then validates identifiers, units, ranges, operating-mode constraints, incompatible settings, authorization, and change state. Validation failure rejects the proposal rather than asking the model to reinterpret the error and retry against production.

An authorized operator or engineer reviews the proposal against raw process information, not only the model's summary. Approval creates a new authenticated transaction. The control system applies that transaction through its existing change mechanism and records the resulting state.

A process engineer asked to sign off on a proposed setpoint change for a reactor cooling loop, for instance, pulls the raw trend behind the staged proposal rather than trusting the model's one-line justification, and finds the trend flat over a window where the justification describes a rising trend. Rejecting the proposal at that point costs a few minutes. Approving it on the strength of the summary alone would have pushed a setpoint change into production against evidence that never actually supported it.

For software-backed control configuration, the equivalent pattern is a versioned diff, deterministic checks, independent approval, scheduled deployment, and tested rollback. A chat session with a PLC tool is not an acceptable substitute.

Access control still matters on the advisory side. A model that can retrieve operating procedures or incident records must preserve source permissions throughout retrieval and generation, as specified in How to preserve RAG access controls end to end. Those controls contain information exposure. They do not make model output suitable for actuation.

Four integration shortcuts move the model back inside the loop

Direct write access is the clearest failure. Giving an agent an OPC write tool, controller API, engineering workstation session, or robotic command endpoint makes generated output part of the actuation path. Requiring well-formed arguments changes the command grammar, not its safety semantics.

A single-click acceptance button can create the same problem operationally. If the interface hides the underlying signal, preselects the model's action, and makes approval the fastest way to clear the queue, the nominal human boundary becomes a confirmation step. A control room operator working through a backlog of proposed setpoint changes near the end of a shift, with the raw trend one screen away and the approve button in front of her, is not exercising the independent judgment the design assumed; she is clearing a queue.

Using the model as a second channel in a trip decision also fails the independence test. A protection function cannot claim independence from a component whose output participates in whether the trip occurs. Disagreement handling, model timeouts, malformed output, and service upgrades become safety-function behavior that must be specified and verified.

Placing the model on premises changes confidentiality and availability exposure, but none of those infrastructure gains touch the underlying behavior: the model still samples, its open-ended output still resists exhaustive testing, and neither fact moves toward a safety integrity claim.

A deterministic envelope around generated commands is valid only when every command is safe anywhere inside that envelope. In that design, the envelope owns actuation authority. If safe action depends on context the envelope cannot represent, automatic execution remains outside the defensible boundary.

Regulation follows the function assigned to the model

The EU AI Act defines a safety component by function and consequence: it performs a safety function, or its failure or malfunction endangers health and safety. ⁴

Article 6 classifies an AI system as high-risk when it is intended as a safety component of a product covered by listed Union harmonisation legislation and that product requires third-party conformity assessment. ⁴ Annex I includes Directive 2006/42/EC on machinery. ⁴

Calling the model an assistant does not determine its legal or engineering role. Its assigned function and effective authority do. If equipment safety depends on its answer, the architecture has treated it as a safety component regardless of the product label.

High-risk classification also does not supply a safety integrity claim. The protection case still needs specified behavior, independence, verification, and proof testing under the applicable machinery and functional-safety regime.

Apply the boundary with an authority-tracing review

Use the following procedure before procurement, during design review, and after every material integration change.

1. Start from each physical output that can alter equipment state. Trace its command path backward through the PLC, distributed control system, gateway, broker, workflow, and user interface.

2. Mark every point where model output can select a target, value, timing, operating mode, bypass, reset, or retry. A generated argument passed through deterministic transport is still generated authority.

3. Remove the model and all shared services used only by it. Verify that deterministic control reaches its defined fallback and that the protection layer can still trip or reach the safe state.

4. Inject malformed, stale, contradictory, and unauthorized proposals. Confirm that rejection is deterministic and cannot trigger an autonomous model retry against production.

5. Examine the human decision surface. Show primary signals, proposal provenance, assumptions, conflicts, and rollback information. Do not make generated prose the operator's only view of plant state.

6. Put the boundary into the safety requirements, network rules, identity policy, change process, test plan, and supplier contract. An architecture diagram without enforceable interfaces will drift when later teams add tools.

7. Repeat the authority trace after model, prompt, retrieval, tool, controller, or workflow changes. A new convenience action can turn an advisory path into a control path without changing the model itself.

An advisory boundary does not eliminate automation bias

Keeping generative AI outside actuation prevents it from carrying the deterministic safety claim. It does not make its advice harmless.

NASA-funded human-factors research defines automation bias as errors caused when decision makers substitute automated cues for vigilant information seeking and processing. ⁵ A human in the workflow therefore provides no automatic assurance of independent judgment.

The Milford Haven refinery investigation shows why presentation belongs in the safety review. HSE found that an excessive number of alarms during the emergency reduced operator effectiveness. It also found a control valve that was shut while the control system indicated it was open. ⁶

A generative assistant can reproduce both pressures. It can flood the operator with plausible diagnoses, or confidently summarize an incorrect instrument state. Rate limits, prioritization, source visibility, stale-data indicators, and suppression of duplicate advice are operational controls, but they do not establish that the remaining advice is correct.

The boundary also fails when staffing and timing make review fictional. If proposals arrive faster than operators can inspect their evidence, or the process demands action faster than deliberate review permits, that function needs deterministic automation with defined limits. It is not a suitable generative-AI task.

The final acceptance test is physical: disconnect the generative system during a proof test and demonstrate that the independent protection path still detects the specified condition and drives the equipment to its defined safe state. If that demonstration fails, the model is already inside the loop.