Measure adoption, not deployment

Deployment is not evidence of adoption

A shipped AI system has no value when people route around it, abandon its output, or spend more time correcting it than the system saves. An enterprise rollout therefore needs task-level measures of usage, abandonment, override, and rework alongside model quality.

Licences, enabled accounts, and weekly active users answer whether employees can open the product. They do not show whether the product completes work.

That distinction changes the apparent result of a rollout. Quarterly telemetry covering more than 120,000 workers found that 82% of AI users returned quarter over quarter, while only about 2% reached usage consistently embedded in their workflows. The same research found that 57% of users spent less than 1% of their working hours in AI tools. A retention dashboard could call the first result healthy while missing the depth problem exposed by the other two measures. ¹

The unit of measurement must therefore be the task, not the account or chat session. An account can be active while the employee copies the result into another tool, rewrites it from scratch, or returns to the previous process.

A task event must survive beyond the model call

Model telemetry usually ends at the response boundary. It records latency, token consumption, model identifier, retrieval calls, and perhaps an evaluation score. Those fields describe inference. They do not describe whether the work was accepted.

Instrument the business task as a state transition that contains the model call. The task record needs an identifier shared by the workflow UI, orchestration layer, model gateway, and downstream system of record. It should also record the workflow name, task type, model release, prompt template version, retrieval configuration, policy version, and pseudonymous actor or team identifier.

The outcome fields carry the adoption signal:

task_id
workflow_id
task_type
actor_scope
started_at
completed_at

ai_invoked
model_id
prompt_template_version
retrieval_snapshot
response_id

output_presented
user_action: accepted | edited | rejected
override_reason
abandoned_stage
downstream_artifact_id
rework_reason
rework_at
baseline_path_used

OpenTelemetry's GenAI semantic conventions at commit 0c87594975195608dc91b3f702e250a7b240c151 define a completion-request event that can include chat history and parameters separately from a trace. That event is useful for linking inference details to the task, but it is not the task record itself. ²

Do not make raw prompts and responses the default adoption dataset. They can contain customer data, source code, legal text, or employee information. Store references, hashes, classifications, and evaluation outputs where they answer the measurement question. Put any retained content behind a separate access and retention policy. This is where AI governance should produce evidence, not only policy becomes operational: the governed event is evidence of what happened, under which model and policy version.

Four failure classes explain why an active user is not an adopted workflow

The dashboard should classify failure before aggregating it. The same activity count can be produced by incompatible conditions that require different interventions.

Route-around includes work completed through an old manual process, an unsanctioned model, or an informal handoff to another employee. Product telemetry alone cannot see it. The surrounding workflow must emit an eligible-task event before the user chooses a path.

Picture a billing analyst who opens the AI-drafted dispute letter, decides it will not survive a supervisor's review, and reopens the case in the legacy ticketing macro instead of editing the draft. The task was eligible, the tool was available, and the case still closed through the old path. Unless the ticketing system emits its own eligible-task event, the rollout dashboard never learns that the case existed.

Abandonment is different from rejection. A user may leave because the response is slow, authentication fails, required context is missing, or the interface offers no usable next action. A model-quality programme cannot repair those failures.

Consider a support agent who opens the drafting panel mid-call, watches the spinner while the customer waits on the line, and closes the panel to type a reply by hand before the model ever returns a token. The AI was invoked. No output was presented, and the model-quality team has nothing to review, because the failure happened before quality was ever tested.

Override records an explicit decision against the system output. A systematic review of clinical decision-support alerts found a 96.2% override rate, with rates varying sharply by alert type: a prescriber overriding an interaction alert because the record has not caught up with a medication the patient already stopped looks identical in the log to a prescriber overriding a warning they never read. The useful lesson is not that a particular override rate is acceptable, but that override must be segmented by task and reason before anyone interprets it. ³

Rework happens after apparent success. The user accepts a draft, submits a classification, or approves a recommendation. Someone later corrects the artifact, reopens the case, or repeats the task manually. Without the downstream artifact identifier, this cost disappears from AI telemetry.

DORA provides a useful precedent. Its five software delivery metrics are explicitly outcome measures of the delivery process, and deployment rework rate is defined as the ratio of unplanned deployments caused by a production incident. AI delivery needs the equivalent connection between initial throughput and later repair. ⁴

Compute adoption metrics from eligible work, not generated text

Start with the population of tasks for which the system is intended. If the denominator is AI sessions, every route-around disappears before calculation.

Task coverage is the share of eligible tasks in which the sanctioned AI path was invoked. It reveals whether the integration reaches the work, but high coverage alone can be produced by mandatory invocation.

Completion measures invoked tasks that produce the required downstream artifact. Abandonment is the complement within started AI-assisted tasks, classified by the last observed workflow stage.

Override rate is the share of presented outputs that users reject or materially change. The definition of material change belongs to the task. A text workflow might classify edits by changed claims or sections, while a coding workflow might use discarded patches, failed validation, or replacement commits. Character distance is rarely a sufficient proxy.

Rework rate is the share of completed AI-assisted tasks that later receive a correction attributable to the output. Attribute it through explicit reason codes where possible, then audit a sample. A reopened case is not automatically AI-caused.

These measures must be sliced by task type, team, model release, prompt version, retrieval configuration, and workflow stage. An aggregate override rate can fall because the model improved, because easier tasks entered the mix, or because users stopped checking. Acceptance is an observation, not a correctness label.

Model quality remains a separate axis. Offline evaluations can test factuality, policy compliance, extraction accuracy, or code behavior against controlled cases. Production task telemetry shows whether users complete work and what happens afterward. Joining the axes can reveal that a model scores well on a test set while failing inside the workflow, or that an imperfect model remains useful because users can detect and cheaply correct its errors.

The cost of checking belongs in the same record. How much of my team's time will checking the AI take? explains why review cannot be treated as free. A completion that requires extensive verification may still be worth shipping, but its economics differ from a completion that passes through unchanged.

Seat counts, token volume, and surveys fail for different reasons

Seat activation measures procurement and provisioning. It is useful for licence management. It cannot distinguish embedded use from occasional experimentation.

Weekly active users improve on activation by requiring observed activity. They still reward shallow use. One employee opening the tool for a disposable query and another completing daily operational work both count as active.

Request and token volume measure system load. They help with capacity and cost forecasting. They can rise because users retry failed prompts, split work into more calls, or generate output that never reaches a downstream artifact.

Self-reported time saved and satisfaction surveys capture perceived utility. They miss tasks that users never attempt, and responses are vulnerable to recall error and incentives around the rollout. Surveys are diagnostic context, not the adoption denominator.

Raw acceptance rate appears closer to value, but it has two opposing interpretations. High acceptance can mean the output is good. It can also mean users are rubber-stamping it. Low acceptance can mean poor output, or it can mean a high-value review process is catching consequential errors.

A single adoption score compresses these conflicts until the organization cannot tell why it moved. Keep the component measures visible. If executives require a summary indicator, pair it with the failure-class distribution and model-quality slice that produced it.

Instrument the rollout in the order failures become observable

Define eligible tasks before launch. Use the workflow's existing business objects, such as tickets, cases, pull requests, invoices, or reports. If eligibility is inferred only after an AI invocation, route-around cannot be measured.

Emit task-state events from the workflow application rather than relying on the chat interface. The application knows that a case was opened, an output was presented, and an artifact was submitted. The model gateway knows only that a request received a response.

Require structured reasons at the moments that matter, but keep the taxonomy short enough to use. Override reasons might distinguish missing evidence, incorrect content, policy conflict, unsuitable format, and user preference. Rework reasons should identify whether the prior AI-assisted artifact contributed to the correction.

Join downstream outcomes after completion. The observation window should follow the business process, not an arbitrary analytics session. A code suggestion can appear accepted before tests or review reject it. A support response can appear complete before the case reopens.

Compare cohorts at the task level. Segment by workflow and task difficulty before comparing teams or model releases. A team receiving harder cases should not look less adopted because its override rate is higher.

Set rollout gates around failure modes. A release may proceed when model-quality checks pass while abandonment remains under observation. It should pause when rework appears in a consequential task, even if weekly active use continues to rise. The operating decision follows the failure, not the activity total.

Task telemetry still misses coercion, gaming, and invisible work

Measurement changes behavior. Researchers studying enterprise adoption warn that a population can perform adoption through invocations without reliance and long sessions without workflow integration once people know the metric is tracked. ¹

Mandatory invocation makes coverage especially easy to game. Employees can open the tool, generate an answer, and continue through the manual path. Instrumenting both paths reveals some of this behavior, but private tools, copied text, and work completed outside managed systems remain partially invisible.

Override telemetry also breaks when users lack authority or confidence to disagree. A low override rate cannot establish trust, accuracy, or value. Pair it with sampled quality review, downstream rework, and task outcomes.

Rework attribution is inherently incomplete. Corrections can result from changed requirements, new information, reviewer preference, or defects unrelated to the model. Reason codes and audit samples reduce ambiguity; they do not eliminate it.

Privacy constraints can prevent collection of the content required to classify failures. In that case, retain less content and accept lower diagnostic resolution. Do not convert adoption measurement into indiscriminate employee surveillance.

Task-level telemetry is therefore a diagnostic system, not a productivity ranking. Use it to decide whether to repair the model, redesign the workflow, change the task boundary, or stop the rollout. When should a business not use AI? is the relevant decision when route-around and rework persist after the integration problems are removed.