Evaluation traces are a second production dataset
AI agent traces retained for evaluation must be governed as a second production dataset, because they often reproduce the prompts, tool arguments, retrieved records, model outputs, and user identifiers from the system they measure.
The common mistake is treating the trace backend as observability infrastructure and the evaluation corpus as engineering metadata. That classification is convenient, but it does not change what the trace contains. If a support agent trace includes a customer account number, a free-text complaint, a tool response from a CRM, and the final answer, copying that trace into an evaluator creates another store of the same sensitive material.
Picture the evaluation engineer who exports recent escalations to score a new prompt version. The export is a spreadsheet of account numbers, complaint text, and CRM responses sitting on a laptop, built for one afternoon's scoring session and still there long after because nobody's job was to delete it.
The trace copy is often weaker than the primary system. The primary CRM may have role-specific access, retention jobs, subject-deletion workflows, and regional controls. The tracing project may instead grant broad read access to engineers, retain runs indefinitely, export JSON into notebooks, and allow evaluators to download examples locally. Evaluation then turns production content into an uncontrolled dataset precisely because the team wanted to measure production behavior.
This matters before an evaluation framework, judge model, or trajectory rubric is selected. A trace that cannot be retained and accessed lawfully cannot become a benchmark by being assigned a score.
A trace becomes governed data when evaluation needs its content
A trace identifier, latency, model name, token count, and error status can support operational diagnosis without recreating the conversation. Evaluation usually needs more. It needs the user request, the agent's sequence of tool calls, retrieved context, tool results, model output, and often the rubric or human annotation used to judge it.
OpenTelemetry's GenAI semantic conventions make the distinction explicit. Instrumentations should not capture prompt and completion content by default, and should require an explicit opt-in to do so.1 The specification even names an example switch, OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT.
That default exists for a reason. The gen_ai.input.messages and gen_ai.output.messages attributes are described as likely to contain sensitive information, including user and personally identifiable data.1 Once those attributes are captured, the trace stops being just an execution record and becomes a content store with an execution graph attached to it.
Evaluation compounds this. A scored trace commonly gains labels such as grounded, unsafe, incomplete, correct, or failed handoff. OpenTelemetry models the evaluation result as a distinct event associated with the GenAI span evaluated.2 The useful lineage is therefore bidirectional:
| Dataset element | Lineage it must retain | Control that follows |
|---|---|---|
| Evaluation item | Source trace and source content record | Deletion and retention linkage |
| Trace content reference | Tenant, region, and consent or processing basis | Access and residency enforcement |
| Evaluation label | Evaluator, rubric version, and evaluated response | Auditability and reproducibility |
| Exported evaluation set | Item identifiers and extraction time | Recall, expiry, and contamination control |
Without those links, an evaluation set is a detached copy. The team cannot determine which primary record authorized it, which deletion request reaches it, which region applies, or whether an evaluator scored the actual response rather than a filtered surrogate.
The failure taxonomy is about lineage breaks, not missing dashboards
The central operational problem is not whether every span has a trace ID. It is whether the trace, the evaluation item, and the underlying content record stay connected while each moves through different systems.
| Failure | What engineers typically build | What breaks | Required control |
|---|---|---|---|
| Content capture without classification | Full prompt and output attributes enabled globally | Production content enters a telemetry project with operational defaults | Classify content-bearing attributes before export and require named opt-in |
| Orphaned evaluation copy | JSONL or warehouse table made from traces | The copy no longer receives source retention or erasure actions | Store source record IDs and process deletion as a cascade |
| Access widening through evaluators | Broad project read permission for graders and developers | People who could not access the primary system can read its copied content | Separate metadata access from content access, with evaluator-specific grants |
| Consent or purpose drift | Production traces reused for a new eval objective | The retained copy is used beyond the purpose represented to the user or customer | Record purpose and eligibility at extraction time |
| Residency drift | Trace export crosses regions for centralized scoring | Human or model evaluation runs outside the constraints attached to the original traffic | Bind region to the item and reject incompatible evaluation jobs |
| Redaction divergence | Trace text is filtered after generation | The evaluator sees a different response from the provider record | Preserve a redaction manifest and mark the evaluation as partial |
| Benchmark contamination | High-value traces are reused in training or prompt tuning | The score ceases to measure generalization | Partition evaluation data and record every downstream use |
A team can have complete distributed tracing and still fail every row in this table. Trace correlation answers how one span relates to another, but data lineage is answering something else entirely: where the underlying content came from, which obligations travel with it, and where every derivative copy now lives.
The distinction becomes acute for agent systems with human escalation. A handoff record can contain the same conversation history, decisions, and unresolved task state as the evaluation trace. What must an AI support handoff preserve? is useful here because preserving operational context and preserving an uncontrolled copy are different requirements. The handoff needs enough state to continue work. The evaluation system needs only the fields necessary to assess the behavior under test.
Full trace retention is convenient, but it moves the access boundary
The fastest implementation is to retain complete prompts, completions, tool payloads, and retrieved documents in the trace platform. It makes failed runs easy to replay and lets an evaluator query one system.
It also turns the trace project into a general-purpose content repository. If that project has a different identity provider group, different regional storage, or a longer retention window than the source system, its controls now determine the actual exposure path.
OpenTelemetry recommends a different production pattern when sensitive content needs secure handling: keep content in external storage and use the trace to reference it, because external storage permits separate access controls.1 The trace backend can then retain execution metadata while a content service resolves authorized fields for a specific evaluator.
That choice costs engineering work. Evaluation jobs need a resolver rather than a direct span query. The resolver must enforce tenant, region, purpose, and expiry checks. Reproducibility requires recording the content version or immutable evidence reference used by the scorer. Those costs are real, but they correspond to the controls that the primary system already needed.
A second alternative is permanent redaction before trace export. This lowers trace-store exposure, but it can make the evaluation invalid. OpenTelemetry warns that if output messages are filtered or truncated, gen_ai.response.finish_reasons still describes the provider's original generation rather than the filtered content left on the span.1 An evaluator can then attribute a failure to the agent when it is actually assessing a transformed record.
The third alternative is retaining no content, only metadata and aggregate scores. This is appropriate for some reliability measures, such as tool failure rate or execution duration. It cannot support faithfulness, policy, correctness, or trajectory evaluation that depends on the actual request and response. Do not pretend a metadata-only record supports content-level evaluation.
Retention and deletion must propagate from the source record
GDPR Article 5(1)(e) requires personal data to be kept in identifiable form no longer than necessary for the processing purpose.3 A trace store holding identifiable prompts or tool results is within that problem, even if the primary database has already deleted its copy.
Article 17 adds a practical systems requirement: where its conditions apply, the controller must erase the relevant personal data without undue delay.3 For an AI agent evaluation system, deletion cannot stop at the customer table. It must locate retained raw traces, extracted evaluation rows, annotations, cached evaluator inputs, and exported files.
Use a source-to-derivative deletion contract:
-
Assign each content-bearing trace a source record ID, tenant ID, region, collection purpose, and expiry policy at ingestion.
-
Make every evaluation item reference the trace ID and source record ID. Do not duplicate raw content into an item without this lineage.
-
When the source record expires or is erased, emit a deletion event that tombstones the trace content reference, removes evaluable content, and invalidates derivative items and cached inputs.
-
Retain non-identifying aggregate measurements only where they cannot be joined back to the subject through the remaining fields.
-
Make evaluation jobs fail when an item is deleted, expired, outside the evaluator's region, or lacks a valid purpose record.
The last step prevents a common repair failure. Teams often add a nightly deletion job, while a queued batch evaluator has already read the trace and written another copy. A job-time eligibility check closes that path.
Vendor defaults deserve the same inventory treatment. OpenAI states that abuse-monitoring logs are generated by default for API usage and retained for up to 30 days.4 Its /v1/evals entry is listed as ineligible for Zero Data Retention and Eyes Off, with 30-day abuse-monitoring retention and application-state retention until deletion.4 A team using that endpoint needs to distinguish vendor-side controls from its own trace retention policy. One does not replace the other.
Consent and human review belong in the trace eligibility record
Consent, contractual scope, and processing purpose are not fields that can be inferred from prompt text later. Capture the applicable eligibility record when the trace is created or when content is selected into an evaluation pool.
This becomes concrete when humans review examples. Azure's abuse-monitoring documentation restricts access to flagged prompts and completions to authorized employees using Secure Access Workstations and just-in-time manager approval.5 For EEA deployments, it also states that authorized human reviewers are located in the EEA.5
Those are examples of controls attached to content review. Internal evaluation workflows need equivalent decisions: who can inspect raw traces, who can inspect redacted views, whether a reviewer can export content, and where review occurs. A labeler who only needs to judge whether an agent selected the right tool does not necessarily need the customer's raw prompt or the full retrieved document.
The same restriction applies to stop behavior. An agent that reaches a refusal, escalation, or safe termination state can produce a trace that is valuable for evaluation, but the reason for that state may contain protected context. Your AI agent needs a safe way to stop explains why the stop mechanism must preserve enough state for safe control flow. The evaluation representation should retain the minimum evidence needed to test that mechanism.
Evaluation data needs contamination controls as well as privacy controls
A trace set can be compliant and still be useless as an evaluation set. If examples used for acceptance evaluation later enter fine-tuning, prompt optimization, retrieval curation, or test-case authoring, the score measures exposure to the set rather than the agent's behavior on unseen work.
The cost can be extreme. Research on benchmark contamination found that a 13-billion-parameter model could overfit a test benchmark and perform on par with GPT-4.6 The point for an agent team is narrower: an evaluation trace is a controlled asset because its predictive value depends on keeping its use history.
Track at least three states for each item: eligible for evaluation, reserved evaluation item, and excluded from evaluation. Move an item to excluded when it enters training, prompt tuning, retrieval corpus construction, or a human-authored test guide that makes its expected answer available to the system builders.
A trace that lacks source lineage, purpose eligibility, deletion propagation, and use history should be rejected from evaluation. Retaining it as an operational incident record may still be justified under its own policy, but a score built from it should not be treated as evidence about production quality.