Your AI agent needs a safe way to stop

Decision: route each failure class to its own exit contract

An AI agent needs a safe way to stop because policy refusal, dependency failure, incomplete work, and unresolved judgment require different downstream behavior. Converting every uncertain exit into a generic error, a fluent answer, or a human ticket destroys information that the caller needs to recover safely.

The decision is to define refusal, deterministic fallback, bounded partial output, and human review as separate terminal states. Each state gets its own eligibility rules, response schema, side-effect policy, retry semantics, telemetry, and reversal path.

This is an application-level decision. Model providers expose useful stop signals, but they do not know whether a failed tool call can be retried, whether omitted data makes an answer unsafe, or whether an operator has authority to approve the requested action.

A single error shape makes unsafe recovery look reasonable

Consider an agent that validates an invoice, checks a purchase order, and schedules payment. Several failures can occur:

• The request violates an authorization policy.
• The purchase-order service times out before returning data.
• An optional supplier-enrichment service is unavailable.
• The invoice and purchase order disagree in a way that requires judgment.
• The payment request reaches the processor, but the connection closes before the caller receives the result.

These failures must not share a recovery policy. Retrying an authorization refusal only wastes capacity, since permission was never the problem it can fix. Sending every enrichment timeout to an operator buries the one urgent case inside a queue of routine ones. A payment-state ambiguity is the sharpest case: if the connection to the processor drops before the response returns, the accounts-payable clerk who queued that payment needs to know whether the wire already left the account before she decides whether to resubmit it, and a partial answer that glosses over that gap can conceal a duplicate payment.

Provider APIs already expose distinct control signals. Anthropic returns tool_use when Claude expects the caller to execute a tool, refusal as a normal HTTP 200 response when generation is declined, and pause_turn when a server-side tool loop reaches its default limit of 10 iterations per request. These are different protocol states, not interchangeable error strings. ¹

OpenAI Structured Outputs similarly separates a safety refusal from schema-conforming content through a dedicated refusal field. The caller does not need to infer refusal by parsing prose. ²

Those signals feed the exit decision, but they do not make it.

The exit policy is a table, not a catch block

The following table is the core of the architecture decision record. Each row defines a distinct promise to downstream systems.

Classification happens before presentation. A model must not choose an exit merely by writing “I cannot continue” or “Here is what I found.” The orchestrator assigns the terminal state from machine-readable evidence such as policy decisions, tool results, budget state, and side-effect records.

The decision order is strict. A prohibited action becomes a refusal even if a fallback exists. An ambiguous mutation goes to review even if partial descriptive output is available. Partial output is allowed only after the system proves that omitted work cannot change the meaning or safety of the completed subset.

A terminal envelope keeps recovery semantics machine-readable

Represent the exit as a discriminated union rather than an optional error field attached to an otherwise successful response.

type AgentExit =
  | {
      kind: "refusal";
      reasonCode: string;
      policyId: string;
      explanation: string;
      sideEffects: SideEffectRecord[];
    }
  | {
      kind: "fallback";
      fallbackId: string;
      fallbackVersion: string;
      result: unknown;
      idempotencyKey?: string;
      sideEffects: SideEffectRecord[];
    }
  | {
      kind: "partial";
      result: unknown;
      completed: string[];
      omitted: Omission[];
      completeness: "partial";
      sideEffects: SideEffectRecord[];
    }
  | {
      kind: "human_queue";
      queue: string;
      reviewPacketId: string;
      allowedDecisions: string[];
      deadline: string;
      sideEffects: SideEffectRecord[];
    };

type Omission = {
  component: string;
  reasonCode: string;
  retryable: boolean;
};

type SideEffectRecord = {
  operationId: string;
  tool: string;
  status: "not_started" | "confirmed" | "ambiguous" | "compensated";
};

The envelope belongs outside the natural-language response. User-facing text can change without altering workflow behavior. The dispatcher switches on kind, while policy tests assert which kinds are legal for each failure class.

Persist the evidence used for classification. The trace needs the policy result, tool-call arguments, tool responses, model stop reason, remaining budget, selected exit, and side-effect ledger. How to govern AI agent traces used for evaluation covers the controls needed when those traces later become evaluation data.

Deterministic fallback requires an operation identity

A retry is a fallback only when repeated execution has defined semantics. “Try again” is unsafe for a mutation whose first outcome is unknown.

Stripe’s API illustrates the necessary contract. The client generates a unique idempotency key for an operation. Stripe saves the status code and body from the first request associated with that key, whether the request succeeded or failed, and subsequent retries receive that stored result. ³

An agent executor should create the operation identity before the model or tool adapter attempts the mutation. Store it with the run state. Every retry of that logical operation must reuse the same identity. Generating a fresh key during recovery describes a new operation and defeats duplicate suppression.

A deterministic fallback also needs versioned inputs. A rules-engine decision is reproducible only if the rule-set version, normalized inputs, and relevant reference-data version are recorded. A cached response needs its origin and freshness. A read-only fallback must remain read-only unless its mutation semantics are separately defined.

Circuit breaking belongs at this layer. The circuit breaker stops calls to an operation that is likely to keep failing. Its open state can return a meaningful default where the application permits one. ⁴

That default is bounded partial output or deterministic fallback depending on its contract. A cached tax rate selected by jurisdiction and effective date can be a deterministic fallback. Omitting optional supplier metadata while marking the fields unavailable is partial output. Returning an empty object with HTTP success is neither, because the caller cannot tell absence from failure.

Bounded partial output declares what is missing

Partial output is safe only when completeness is part of the schema. The response must identify completed components, omitted components, and whether any omitted component can invalidate the completed result.

Consider a due-diligence agent pulling the most recent supplier contracts for review: if it retrieves the documents but times out before extracting the indemnification clauses, returning the contracts with the clause extraction marked omitted still lets the analyst open the source pages herself, instead of a completed-looking summary that quietly skipped the clause that mattered. For an approval agent, returning the completed evidence checks without an approval decision may be useful in the same way. Returning “approved” while one required check is unavailable is not a partial result. It is an unsupported decision.

Keep partial output monotonic. Later enrichment may add missing fields, but it must not silently reinterpret an earlier decision. If new evidence can reverse the result, the earlier object needs a status such as pending_review, not complete.

Side effects also bound the output. A response that says “three steps completed” is insufficient if one completed step changed an external system. The caller needs the operation identity and confirmation state for every attempted mutation.

A human queue is an executable continuation

Human escalation must preserve enough state for an authorized reviewer to continue, reject, override, or compensate. A screenshot of the final chat message is not a review packet.

An underwriter opening the queue to review a declined policy change needs the policy record and the rule that fired, not a transcript of what the agent said to the applicant. The packet should include the original request, authenticated actor, delegated permissions, policy decisions, relevant model and tool outputs, attempted mutations, ambiguous effects, available actions, and the deadline after which those actions are invalid. The preservation requirements overlap directly with What must an AI support handoff preserve?.

For systems covered as high-risk under Regulation (EU) 2024/1689, Article 14 makes the control requirement explicit. Human overseers must be able to disregard, override, or reverse output, and must be able to interrupt operation through a stop button or similar procedure. Article 12 also requires technical support for automatic event logging over the system’s operating lifetime. ⁵

A queue that exposes information without granting the reviewer an effective action does not satisfy this architecture. Neither does a stop control that prevents future tool calls while leaving an ambiguous external mutation unresolved.

Human review introduces its own failure mode. The same Article 14 tells overseers to remain aware of automation bias, including automatic or excessive reliance on system output. ⁵ The review interface should therefore show source evidence and policy constraints before any model-generated recommendation, require a reason for override, and distinguish confirmed facts from the agent’s proposal.

Deploy the classifier beside the current exit path

Begin deployment in classify-only mode. For every existing terminal event, run the proposed exit classifier and record its selected kind without changing the user-visible behavior. Compare the proposed classification with the actual side effects and the recovery action taken by operators.

Build fixtures from failure injection rather than waiting for production incidents. Cover policy denial, dependency timeout, malformed tool output, exhausted tool-loop budget, optional-data loss, mutation timeout before acknowledgement, and contradictory evidence. Each fixture needs an expected exit and prohibited exits.

Enforcement should proceed by reversibility. Route clear policy denials to structured refusal first. Enable deterministic fallbacks for read-only operations and mutations protected by verified idempotency. Add partial output only after consumers handle completeness: "partial" explicitly. Move ambiguous or high-impact decisions into the human queue after the review packet and override path have been exercised end to end.

Run the legacy and new dispatchers from the same persisted terminal event during migration. The terminal event is the source of truth. This prevents model text, HTTP status, and queue routing from drifting into separate interpretations of the same failure.

Operational alerts should use exit-specific conditions. A rise in refusals suggests a policy, request-mix, or classifier change. A rise in fallback use indicates preferred-path degradation. Old partial results indicate failed enrichment. An aging human queue indicates insufficient review capacity or overly broad escalation rules. Combining them into one agent-failure counter hides the remediation owner.

The rejected alternatives fail in different ways

A single generic error envelope keeps client code superficially small. Its cost appears in recovery: callers retry denials, discard useful partial work, and escalate dependency outages as judgment cases. The error string becomes an undocumented protocol.

Always retry treats uncertainty as transient infrastructure failure. It cannot resolve missing authority, contradictory evidence, or a model loop that has exhausted its budget. It also risks repeating mutations unless operation identity and idempotency were established before the first attempt.

Always return best effort preserves responsiveness by concealing incompleteness. It is acceptable only where the consumer already treats every field as optional and no decision implies that required checks ran. Most action-oriented agents do not meet that condition.

Always escalate transfers classification work to operators. Routine outages then compete with genuine judgment calls, while the queue inherits automation-bias risk. Human review is the correct exit when the missing ingredient is authority or judgment, not when a circuit breaker can return an approved default.

A model-selected refusal is also insufficient as the universal stop mechanism. A benchmark covering 20 frontier models found that abstention remains unsolved and that increasing model scale offers little help. ⁶ Application policy must verify the conditions for refusal instead of trusting the model’s confidence or willingness to answer.

This design does not make uncertain work safe by itself

The exit taxonomy does not repair a tool that lacks idempotency, transaction boundaries, or a status lookup. If a mutation is ambiguous, the classifier can preserve and escalate that ambiguity. It cannot determine an external outcome that the tool cannot report.

It also cannot make a weak fallback trustworthy. A fallback that shares the failed dependency, stale reference data, or the same model judgment may reproduce the original failure under a different label. Each fallback needs independent preconditions and a versioned contract.

Bounded partial output breaks when consumers ignore completeness metadata. Schema validation should reject code paths that treat partial as complete, but downstream reporting and human interpretation still require testing.

A stop button cannot undo an irreversible side effect. Safe stopping therefore starts before execution: establish operation identity, record intent, constrain tool authority, and define compensation where the domain permits it. The terminal state then reports what stopped, what completed, and who has authority to decide the remainder.