Decision: assign recovery objectives to four recovery units
Write the disaster recovery plan as four linked recovery contracts: one each for the vector index, model weights, trace store, and queue. A workload-level RTO and RPO hides the fact that these stores take different paths back to service and lose different kinds of data.
The worked example below is a regional AI application using Pinecone Serverless for retrieval, a versioned model registry backed by pinned Hugging Face Hub artifacts, DynamoDB for traces, and Amazon SQS for pending jobs. The same reasoning survives swapping any of those products, provided the four recovery units stay distinct.
The decision follows the component-level meaning of RTO. NIST defines it as the time system components can remain in recovery before mission harm occurs, rather than as a single duration attached to an application 1. RPO is the point in time to which data must be recovered after an outage 2.
Those definitions produce a practical requirement. Each component needs an objective based on its recovery mechanism and the consequence of losing its state.
One workload-level RTO conceals incompatible recovery mechanisms
A vector index is a derived serving structure. If its source documents, chunking configuration, embedding model revision, tenant identifiers, and authorization metadata survive, the index can be reconstructed. Its RPO is therefore governed by the recoverable source-data cursor, while its RTO is governed by restore or re-embedding throughput.
Model weights behave differently. A released model should be an immutable artifact named by a registry version and content revision. Recovery means resolving and loading that exact artifact. It should not mean retraining the model or downloading whatever currently occupies a mutable main branch.
Trace rows are neither derived serving state nor deployable artifacts. They record prompts, retrieved context, tool calls, responses, policy decisions, and evaluation inputs. Missing rows cannot be recreated faithfully from the recovered application. The trace RPO therefore expresses permanent evidence loss.
A queue contains short-lived obligations. Losing a pending summarization job is different from losing a trace, but treating the queue as disposable can silently drop accepted work. Restoring every message without an idempotency rule can instead repeat tool calls, notifications, or writes.
Calling all of this "RTO: six hours, RPO: five minutes" creates an ambiguous promise. Hand that single line to the on-call engineer during an actual outage and it still leaves them guessing: can inference resume without retrieval, are the traces investigators will need actually available, do accepted jobs get replayed, and do the weights that come back up still match the index they were serving against. One number cannot carry four independent answers.
Recovery objectives should vary with how data is used. Google Cloud gives the concrete example that frequently modified user data can require an RPO of minutes while less critical, infrequently modified data can tolerate hours 3. An AI system needs that distinction inside the workload boundary.
The deployment manifest makes model recovery deterministic
The weights recovery unit includes the model artifact, tokenizer, inference configuration, adapter artifacts, and the identifiers required to retrieve them. The release manifest records an MLflow model version and a Hugging Face commit hash. It is replicated with deployment configuration into the recovery region.
MLflow Model Registry tracks model versions and supports rollback to a previous state 4. The registry entry identifies the release, but the deployer must still avoid resolving a moving artifact reference. Hugging Face Hub downloads the latest revision from the main branch by default; restoring a specific artifact requires a branch, tag, pull request, or commit hash in the revision parameter 5.
A usable recovery test starts the model from the manifest in an isolated recovery account, not from whatever revision happens to be sitting in the deploy engineer's local cache. It verifies artifact checksums, tokenizer compatibility, adapter attachment, runtime configuration, and a fixed inference probe before marking the weights unit recovered.
This contract gives weights an RPO of zero for committed releases. That objective means every production release must remain addressable and replicated. It does not claim that in-flight training state has the same protection.
The vector index objective must include restore time and validation time
The index recovery unit includes vectors, metadata filters, namespace or tenant boundaries, embedding revision, source-record identifiers, and an ingestion cursor. Recovering vectors without the metadata that enforces document visibility restores a search structure that must not receive traffic.
This is why the index sits downstream of the authoritative document pipeline described in How enterprise RAG works across two pipelines. The serving index can be replaced only if the ingestion pipeline can replay every transformation from an identified source point.
Pinecone documents that backup and restore duration for a serverless index scales with vector count. A serverless index containing one hundred million vectors can take up to five hours to back up or restore 6. That duration belongs in the index RTO calculation. It says nothing about the recovery time of weights, traces, or queued work.
Consider a modelled index RTO for an index of that documented size. Assume the vendor restore consumes the full five-hour allowance, and the team reserves one additional hour for metadata sampling, retrieval probes, and authorization checks. The modelled estimate is six hours because five hours plus one validation hour equals six hours. It has not been measured on production hardware.
The index RPO is evaluated against the ingestion cursor, not the backup timestamp alone. If the backup contains source changes through cursor CURSOR_A and the authoritative document store can recover through CURSOR_B, the recovery process replays the interval after CURSOR_A before opening retrieval traffic.
Authorization metadata must follow the same cursor. Restoring content to a newer point than its access-control projection can expose records that should be hidden or hide records that should be visible. How to preserve RAG access controls end to end defines the enforcement boundary that the index validation gate must test.
Trace recovery preserves evidence that recomputation cannot reproduce
The trace store recovery unit receives writes from online inference and queue workers. Its recovery contract covers the raw trace record, stable request identifier, model and prompt revisions, retrieval references, tool activity, policy results, and timestamps needed for evaluation.
For the worked deployment, DynamoDB point-in-time recovery is enabled on the trace table. AWS states that LatestRestorableDateTime is typically five minutes before the current time 7. A five-minute trace RPO is therefore a plausible service-aligned objective, subject to a restore drill proving the application can operate against the restored table.
That five-minute window has a different cost from a five-minute index lag. Index changes can be replayed from source records. Lost traces remove the evidence needed to reconstruct exactly what the system sent, retrieved, or executed during the missing interval. A five-minute trace RPO sounds abstract until an investigator needs the exact passage a customer was shown just before a complaint was filed, and that request falls inside the lost window.
Trace restoration must preserve the controls described in How to govern AI agent traces used for evaluation. A copied table that loses tenant partitions, retention state, or access policy is not a recovered trace store.
The trace writer should pass its health gate before queue replay begins. Otherwise replayed jobs create new business effects without producing the evidence expected from normal execution.
Queue recovery is a reconciliation problem, not a backup checkbox
The queue recovery unit consists of pending messages and the state required to decide whether each message is still actionable. Its contract names the producer request identifier, deduplication key, submission time, handler version, retry state, and any downstream idempotency key.
A queue RPO is a business decision about accepted work. The worked policy assumes the product owner accepts loss of jobs submitted during the final fifteen minutes before failure. Fifteen minutes is a chosen policy input, not a measured broker guarantee.
The recovery procedure must compare restored messages with durable completion records. A message whose downstream effect committed before the outage should be acknowledged rather than executed again. A message with no completion record can be replayed after its handler version and dependencies pass their recovery gates.
This is why queue traffic opens after weights, trace writes, and any required retrieval path. Starting consumers as soon as the broker is reachable converts infrastructure recovery into uncontrolled application replay.
If a queued operation cannot be made idempotent, the plan needs a manual disposition state. Operators then choose whether to abandon, inspect, or replay the operation with knowledge of its possible external effects, because backup configuration cannot make that decision for them. Consider a drill where a queued refund message replays because its completion record lived in a table that fell outside the recovery scope: the refund goes out a second time, and the reconciliation step that would have caught it is exactly the one skipped to save time during the incident.
A recovery matrix turns the decision into an executable contract
The following matrix is a worked policy, not evidence of measured production performance. Its business assumptions are explicit: released weights must not be lost, the trace store uses DynamoDB point-in-time recovery, queued work can lose a fifteen-minute submission window, and an index with one hundred million vectors receives one hour of validation after the vendor’s documented five-hour restore allowance.
| Recovery unit | Recovery point objective | Recovery time objective | Recovery mechanism | Gate before use |
|---|---|---|---|---|
| Index | Recover through the authoritative ingestion cursor | Modelled six-hour estimate | Restore Pinecone backup, replay later source records, verify embedding and access metadata | Retrieval probes and tenant-filter tests pass |
| Weights | Zero loss of committed releases | Chosen thirty-minute target | Resolve MLflow version and pinned Hub revision, verify checksum, load runtime | Fixed inference probe passes |
| Trace store | Five minutes | Chosen two-hour target | Restore DynamoDB table to the selected recovery point | Trace write, read, partition, and retention checks pass |
| Queue | Chosen fifteen-minute loss window | Chosen thirty-minute target after dependencies recover | Reconcile pending messages against completion records | Consumers remain paused until dependency gates pass |
The values should live beside the deployment manifest and runbook rather than in a general continuity document with no machine identifiers. Each row needs an owner, a health gate, the backup or replay source, and evidence from the latest drill.
The objectives are independent, but the recovery points cannot be selected independently. Google’s SRE guidance notes that restoring separate datastores becomes difficult when restored and live data retain mismatched relationships 8. The recovery controller therefore records one recovery epoch containing the trace restore time, source-data cursor, index backup identifier, model revision, and queue reconciliation cutoff.
Restore in dependency order and keep consumers paused
The incident commander first declares the recovery epoch and blocks new queue consumption, freezing the cutoffs the four recovery contracts will use.
Weights and the trace table can then recover in parallel because neither depends on the vector index. The model receives its fixed inference probe. The trace table receives write and read probes using the recovery application identity.
The index restore also starts immediately because it carries the longest modelled path in the worked deployment. After the backup is available, ingestion replays source changes through the declared cursor. Retrieval tests verify document identity, embedding compatibility, and authorization filters.
Application traffic can resume only in a declared mode. If the product supports a retrieval-disabled mode, the runbook states which requests may use it and which must fail closed. It must not silently answer retrieval-dependent requests without their evidence source.
Queue reconciliation follows. Consumers start only after every dependency used by their handlers has passed its gate. Replayed operations retain their original request and deduplication identifiers so traces and completion records remain joinable.
This ordering avoids adding the four RTOs together. The actual service recovery path is the longest parallel dependency plus any serial validation and reconciliation steps. The runbook should calculate that path from drill timestamps rather than treating the largest row value as proof that the whole system met its objective.
A single objective and universal active-active replication both lose useful information
The cheapest documentation option is one workload-level RTO and RPO. It requires fewer rows in a continuity plan, but it leaves operators to invent recovery priorities during the incident. It also makes a successful test easy to misstate. Restoring the model endpoint does not prove that the index is authorized, traces are writable, or accepted jobs remain accountable.
The opposite option is active-active replication for every recovery unit. It can reduce restore work, but it does not remove the four contracts. Model revision skew, vector ingestion lag, trace write conflicts, and duplicate queue execution still have separate failure modes. The organization also operates a second serving stack and the replication paths required to keep it usable.
A third option is to rebuild every component from source. That is valid for a derived index when the source data and transformation revisions survive. It is invalid for missing traces and completed-but-unacknowledged jobs because those records describe past events that replay cannot recover with certainty.
The selected design spends operational effort on four objectives and one coordination epoch. In return, each recovery claim names the artifact or data boundary it covers.
The plan fails when recovery points are individually valid but mutually inconsistent
Per-store RTO and RPO do not solve cross-store consistency by themselves. A restored index can reference documents beyond the recovered source cursor, a queue can replay a request against weights that have since changed, and a trace can name a retrieval result that no longer exists because a newer index snapshot replaced it. Each of those is a different way for the four recovery points to be individually correct and jointly wrong.
The recovery epoch and validation gates reduce this risk, but only a drill proves that the identifiers survive every restore path. The drill must include a queued request that crosses the chosen recovery cutoff, a trace near the latest restorable point, an index update after the selected backup, and a deployment manifest pinned to a noncurrent model revision.
The approach also fails if the registry and backup control plane share the incident domain. A pinned commit hash is useful only while the artifact remains retrievable. The release process therefore needs an independently recoverable copy of every production artifact and its checksum.
It fails when teams test backup creation but never execute a restore. Regulation (EU) 2022/2554, Article 11, requires covered financial entities to test ICT continuity and response and recovery plans at least yearly and after substantive changes to systems supporting critical or important functions 9. Even where that rule does not apply, a substantive change to embedding generation, trace schema, model packaging, or queue handlers invalidates prior drill evidence.
Finally, this decision does not make replayed side effects safe. If a handler sends money, changes permissions, or invokes an external tool without an idempotency boundary, queue recovery can repeat the effect. The correct recovery state for that message is manual review until the operator can establish whether the original execution committed.