Classify the system by what it can change
A chatbot, a system that answers through conversation, a copilot, a system that prepares work beside a user, a workflow, a system that follows coded steps, and an AI agent, a system that selects steps toward a goal, should be classified by the external state they may change, meaning records, messages, money, permissions, or physical conditions outside the system. The product label does not determine the required controls. A chatbot allowed to issue refunds needs stronger safeguards than an AI agent that can only search a handbook.
This classification solves a practical problem. Buyers need to decide who may authorize an action, how much access the system receives, and how an incorrect action will be stopped or reversed. Labels based on interface or autonomy, the degree to which software selects actions without a person choosing each one, do not answer those questions.
The same terms also change meaning between providers. Anthropic calls fixed, developer-coded paths workflows and reserves agent for systems that direct their own tool use 1. OpenAI instead defines an agent as a component from which a workflow is built 2.
A procurement or governance decision cannot depend on which definition a vendor selected. It needs a boundary that survives a change of product name.
The highest-consequence write determines the control class
A write is any action that creates, updates, deletes, sends, purchases, moves, or triggers something outside the AI system. Reading a customer record is not a write. Changing its status, sending the customer a message, or initiating a refund is.
Picture a support lead reviewing a chatbot's draft reply to an angry customer. Pulling up the account and drafting the reply are reads, whatever the interface looks like. The moment the tool sends that reply or credits a refund, it has crossed into a write, and that is the point where the classification, and the controls around it, have to change.
A permission control is the technical rule that limits which reads and writes the system can perform. An approval control determines which proposed actions require authorization before execution. A recovery control stops, reverses, or contains an action after execution begins.
Classify the system by its most consequential permitted write, even if that action is rare.
| Permitted external effect | Classification test | Controls the design should provide |
|---|---|---|
| No external write | Can the system only retrieve information and return an answer? | Read-only access, limits on sensitive data, and records of what was retrieved |
| Proposal awaiting commitment | Does a person or separate policy mechanism perform the actual write? | A clear preview, an identified approver, and assurance that viewing a proposal cannot execute it |
| Reversible write | Can the system commit a change that can be reliably undone? | Narrow permissions, an action record, an undo mechanism, and a way to stop further writes |
| Hard-to-reverse write | Can it move money, publish externally, change access, place an order, or trigger an irreversible downstream process? | Pre-action authorization, scope limits, staged execution, an independent stop mechanism, and a tested recovery procedure |
The row is determined by capability, not the intended use described in a presentation. A system marketed as a drafting copilot belongs in the hard-to-reverse row if its credentials also let it publish. A model-directed agent belongs in the read-only row if it has no write path.
This rule also covers conventional automation. Knight Capital's order router was not an AI agent, yet live market access made its failure consequential. A coding error caused it to send more than 4 million orders while attempting to fill 212 customer orders during the first 45 minutes after the market opened, producing a loss of more than $460 million 3. What made that failure dangerous was not the architecture, a fixed, human-coded routing system rather than anything that chose its own steps. It was the live market access sitting behind it, the same kind of authority a badly scoped AI agent could just as easily be handed.
Autonomy, interface, and blanket approval hide different costs
Classifying by autonomy creates false negatives, meaning dangerous systems assigned to a safer class. A fixed workflow can execute a destructive database command without discretion. An AI agent can choose its own research steps while remaining unable to change any external record. Autonomy affects how predictable the path is, but permission determines which consequences that path can produce.
Classifying by interface fails for the same reason. A chat window can sit in front of read-only search, a draft generator, or a payment tool. Calling all three chatbots collapses materially different control requirements into one category.
Requiring human approval for every action appears safer, but routine prompts train people to approve without examining the proposed change. Anthropic reported that users approved roughly 93% of permission prompts as its coding system asked for permission more often 4. More prompts therefore do not necessarily produce more scrutiny.
Blanket approval also transfers operating cost to the reviewer. Low-consequence actions queue behind high-consequence ones, and the reviewer must reconstruct context for every prompt. Approval should sit at a consequence boundary, with enough evidence to make the decision, rather than appearing after every model-selected step.
Step one: list every verb that crosses the system boundary
Begin with the verbs, not the product category. Ask the owner to complete one sentence: “The system can read, draft, create, update, send, delete, purchase, move, grant, or revoke...” Then name the destination of each action.
A good answer is concrete: “It reads support tickets, drafts replies, updates ticket priority, and can send a reply after approval.” That answer exposes separate capabilities that can receive separate controls.
A bad answer is “It helps the support team resolve cases.” This conceals whether the system advises an employee, changes the case record, communicates with the customer, or issues compensation.
Include capabilities available through credentials even when the current prompt does not request them. Intended behavior can change with instructions, tool selection, or manipulated input. Technical authority is the classification boundary.
Step two: rank each write by consequence and reversibility
For each verb, identify what can be affected, how broadly the action can spread, whether another system consumes it, and whether the original state can be restored.
A good answer distinguishes a local correction from a propagated event. Editing an internal draft may be reversible. Sending that text to a customer is harder to reverse. Publishing it, notifying downstream systems, and using it to trigger fulfilment creates several recovery problems from one write.
A bad answer is “low risk” without naming the permitted action. It hides the difference between low expected frequency and low consequence. An uncommon payment, deletion, or access change still determines the control class if the system is allowed to perform it.
Use the most consequential allowed action as the system classification. Then apply narrower controls within that class where actions differ.
Step three: grant permissions per action, not per assistant
Give each action only the authority needed to complete it. Searching orders, changing an address, issuing a refund, and cancelling an order should not inherit one broad account merely because one assistant can request all four.
A good answer names the resource, action, scope, and operating condition: “The system may update the shipping address on the active order after the customer has authenticated, but it cannot change the account owner or payment destination.”
A bad answer is “It uses the service account.” This conceals which systems the account reaches, whether it can write, and whether unrelated actions share the same credential.
An on-call engineer chasing a stuck order at midnight may reach for that same service account, because it is the only credential in reach with enough access to see the order. That shortcut is exactly what per-action permissions are meant to close off, whether it is an assistant or a tired engineer using one broad account to do four unrelated things.
Separating permissions also makes classification useful during procurement. If a packaged product cannot divide read, proposal, and commit authority, the buyer cannot align controls with consequence. That limitation belongs in the decision record before integration begins.
Step four: place approval before the consequential commit
Approval must occur after the proposed action is fully specified and before the external write. OpenAI documents human review as a control that pauses a run before a sensitive tool call and allows a person or policy to approve or reject it 5.
A good approval request shows the target, proposed change, reason, evidence, expected downstream effects, and available recovery action. The reviewer should know what approval will cause without opening engineering logs.
A bad approval request says “Allow tool?” It moves accountability to a person while withholding the information needed to exercise it.
If approval transfers work from the system to an operator, preserve the identity, intent, evidence, actions already taken, and unresolved decision. What must an AI support handoff preserve? treats that continuity as part of the control, not presentation detail.
Reserve mandatory approval for consequence thresholds. Routine, reversible writes can use policy checks and sampling if their permissions are narrow and recovery works. A hard-to-reverse write should remain blocked until its designated authority approves it.
Step five: prove that stopping and recovery are separate mechanisms
Stopping prevents the next action. Recovery addresses actions already committed. A design needs both when a run can produce several writes.
A good answer names who can stop the system, the mechanism they use, what state the system enters, which credentials lose authority, how committed actions are identified, and how each one is reversed or contained. The stop control should remain available when the model, workflow, or primary interface is malfunctioning.
A bad answer is “An administrator can disable it.” This conceals whether the administrator can act during an incident, whether queued actions continue, and whether disabling the interface also revokes the underlying credentials.
An incident commander who switches off the chat interface still needs to know whether the refunds it queued moments earlier are going to fire anyway. If the credentials behind those actions stay live after the window goes dark, turning off the interface bought the appearance of stopping, not the fact of it.
For high-risk AI systems covered by the EU AI Act, Article 14 requires a person to be able to intervene or interrupt operation through a stop mechanism that brings the system to a safe state 6. Even where that legal category does not apply, “safe state” is a useful acceptance criterion because it forces the team to define what remains active after interruption.
A stop button without a defined safe state can leave jobs queued, temporary permissions active, or partial transactions unresolved. Your AI agent needs a safe way to stop explains why interruption must be designed as a system behavior rather than an interface control.
Consequence classification does not catch every failure
A read-only system can still disclose confidential data, produce harmful advice, or expose retrieved information in logs. External write classification does not replace privacy controls, output review, or restrictions on what the system may read.
Correct classification also does not prove that permissions are correctly implemented. OWASP traces excessive agency, a vulnerability in which a model can take damaging actions beyond what the task requires, to excessive functionality, permissions, or autonomy 7. Teams must test the actual tool and credential boundaries rather than accept the permission design on paper.
Reversibility can also be overstated. A record may support undo while its change has already triggered an email, shipment, access grant, or external export. Recovery testing must follow the entire chain of effects.
Human approval remains a fallible control when requests are frequent, context is missing, or reviewers face pressure to clear a queue. High-consequence systems need narrow technical permissions and independent recovery even when every commit has an approver. If a recovery exercise cannot identify committed actions, stop pending ones, and restore or contain each external effect, the system is not ready for that permission class.