Enterprise RAG runs on two clocks
Enterprise retrieval-augmented generation, or RAG, a method that grounds model answers in selected company sources, works as two pipelines, meaning two sequences of processing steps. An ingestion pipeline builds and refreshes a searchable index, while a query pipeline retrieves from that index and prepares evidence for an answer.
This split solves a practical problem. A company may have more documents than it can place into every model request. RAG prepares those documents in advance, then selects a smaller set of relevant passages when someone asks a question.
The two pipelines run on different clocks. A source document can change without the index changing, and the index can change without the query service exposing that update immediately. Treating retrieval as one request path hides this gap, which is where stale answers, undeleted records, incompatible representations, and access failures accumulate.
Take an illustrative case: a company edits its refund policy document on a Tuesday afternoon. If the index rebuilds on its normal overnight schedule and the edit reaches the source system after that night's run has already started, a customer asking about the refund policy on Wednesday morning still gets an answer built from the old wording, even though the document itself has already changed.
The ingestion pipeline decides what can be found later
The ingestion pipeline starts with source discovery. Connectors read files, database records, knowledge pages, or other approved material. The system records identifiers and change metadata so that a later run can distinguish new, modified, and deleted content.
The pipeline then parses each document and divides it into chunks, which are bounded passages stored as individual retrieval units. Chunking happens before any user query. Amazon Bedrock, for example, splits documents during ingestion, converts the chunks into embeddings, and writes them to a vector index. An embedding is a numeric representation designed to place semantically related text near each other for search. 1
Chunk boundaries are therefore product behavior, not an internal implementation detail. A passage that omits the heading above it may become ambiguous. A larger passage may preserve context while adding unrelated text that weakens its match to a narrow question.
The trade-off changes with the question type. In one retrieval study, TechQA recall at the top result rose from 16.5% with 128-token chunks to 61.3% with 512-token chunks. A token is a small unit of text processed by a language model. The result does not establish a universal best size. It shows that an ingestion-time choice can determine whether the later query pipeline receives the relevant evidence at all. 2
After chunking, the pipeline attaches metadata such as source identity, revision, owner, and permissions. It then writes the chunks and their embeddings to the index, a separate store optimized for retrieval. Publishing that index makes a particular ingestion state available to queries.
Source permissions must survive every transformation. Filtering only after retrieval can expose restricted text to intermediate services or logs. The identity, document permission, chunk record, and cited answer need one continuous control path, as described in How to preserve RAG access controls end to end.
The query pipeline can only search what ingestion published
The query pipeline begins when a user submits a request. It authenticates the user, derives any permission filters, and converts the request into a form compatible with the index.
Retrieval then selects candidate chunks. A system may compare query and document embeddings, use keyword matching, apply metadata filters, or combine those methods. A reranker, which is a component that reorders retrieved candidates using a second relevance calculation, may narrow the set before generation.
The language model receives the user request, selected chunks, and instructions governing how to answer. It produces a response and, where the application supports it, references the source records associated with those chunks.
This path can be functioning exactly as designed while returning obsolete material. It reads the index that exists, not the source system as it exists now. A successful query therefore proves that serving worked. It does not prove that ingestion observed the latest source state, processed it correctly, or published it to the serving index.
That distinction changes incident diagnosis. If the right passage never entered the index, prompt edits cannot recover it. If the passage is present but ranked below irrelevant material, reranking or query construction deserves attention. If the model receives the passage and still answers incorrectly, the fault is in generation or answer policy.
Picture a support agent who trusts the assistant's answer because it cites a source document by name. The citation gives no way to tell that the cited version was pulled into the index well before a compliance team retired the document, and no error message tells the agent that the source underneath the citation is already gone.
Sync completion and query availability are separate states
An index is usually refreshed by a schedule, event, or explicit operation. It is not automatically identical to its source.
Azure AI Search documents a minimum indexer schedule of 5 minutes. For data volumes that cannot finish inside its typical 2-hour processing window, its guidance is to schedule the indexer on a 2-hour cadence so it can work through the volume. The operational consequence is bounded staleness under ideal conditions and growing delay when ingestion falls behind. 3
Amazon Bedrock requires a sync after files are added, modified, or removed. The sync processes changes since the preceding run. Its documentation also warns that newly created embeddings can take a few minutes to become queryable after the sync reports completion for some vector stores. 4
A useful operating model therefore needs more than a sync status. It needs the source revision observed, the ingestion run that processed it, the index generation that received it, and the query service generation currently active. Without those identifiers, “sync succeeded” cannot establish what users can retrieve.
The failure taxonomy starts at the gap between pipelines
The fastest diagnosis is to classify the failure before changing prompts or retrieval settings.
| Observed behavior | Likely failure class | Evidence to inspect |
|---|---|---|
| A new document cannot be found | Discovery or freshness lag | Source revision, connector checkpoint, ingestion queue, active index generation |
| An edited passage returns its old wording | Publish or propagation lag | Chunk revision in the built index and revision visible to query serving |
| A deleted document still appears | Deletion handling | Source tombstone, deletion policy, index record, cached results |
| Relevant text exists but is split across results | Representation failure | Parser output, chunk boundaries, inherited headings, table handling |
| Relevant chunks rank poorly | Retrieval failure | Candidate set, filters, similarity scores, reranker output |
| A user sees restricted material | Authorization failure | Identity mapping, source permissions, chunk metadata, filter application |
| The model ignores retrieved evidence | Generation failure | Exact supplied context, instructions, citation mapping, final response |
Deletion deserves its own class because change detection does not imply deletion detection. Azure AI Search states that its indexer does not track object deletion in data sources. Without a separate deletion policy, removing a source object does not establish that its indexed representation has disappeared. 5
A records manager who deletes a former employee's expense reports from the source system, for example, has no guarantee the index heard about it. Unless a deletion policy explicitly tells the index to drop that record, the assistant can go on citing figures from a document the records manager believes no longer exists anywhere.
Representation failures also survive healthy infrastructure. Changing chunk policy or the embedding model changes the stored representation of the corpus. Old and new records cannot be assumed to behave alike. A migration needs a rebuilt index, evaluation against representative questions, controlled publication, and a rollback target. The index should be treated as a derived production asset in Write a disaster recovery plan for your AI system, even though its contents can theoretically be recreated.
The alternatives move cost onto the request path or operations
A separate retrieval index is not mandatory when the relevant source material fits comfortably in each request. Long-context processing places the document directly into the model input instead of chunking and indexing it in advance. One empirical comparison found that full-document context generally outperformed RAG on its question-answering benchmarks, especially for Wikipedia-based questions. 6
This alternative removes index freshness and chunk-boundary failures for the supplied document. It moves document selection, permission checks, transfer, and repeated processing onto the query path. It also stops being sufficient once the application must choose among more material than it can reasonably supply per request.
Live search against source systems avoids maintaining a secondary index. Its cost is dependency on each source's availability, latency, search semantics, and authorization interface during every request. It works best when a small number of systems already expose suitable search and stable permission checks.
A manually curated corpus gives owners direct control over publication. Its cost is an operating process for review, removal, and urgent updates. The approach is credible only when the organization accepts that manual publication defines the freshness clock.
Apply the two-pipeline model as an operating contract
Before choosing retrieval technology, write down the contract between ingestion and serving. The contract should answer these questions in terms an operator can verify:
- Which source revision counts as current, and how is that revision identified?
- What triggers ingestion after a creation, edit, permission change, or deletion?
- Which parser, chunk policy, embedding configuration, and metadata schema produced each indexed record?
- What condition marks an index generation as ready for queries?
- How does the query service report which generation answered a request?
- How are failed records, delayed runs, and deletions surfaced to an owner?
- How is a rebuilt index evaluated, published, and rolled back?
Store these answers as operational evidence rather than policy prose. A query trace should connect the answer to retrieved chunk identifiers, the active index generation, and the source revisions behind those chunks. That record makes the governance approach in AI governance should produce evidence, not only policy concrete for RAG.
Tests should cross the pipeline boundary. Add a document, modify it, revoke access, delete it, and observe what a permitted and prohibited user can retrieve at each state. Prompt-only tests cover the final segment of the system and miss the failure classes that motivated the architecture.
Two pipelines do not repair bad sources or guarantee good answers
This model does not make incomplete, contradictory, or poorly structured source material reliable. Ingestion can preserve provenance and revision state, but it cannot decide which conflicting policy is authoritative unless the organization supplies that rule.
Retrieval quality is not guaranteed either: chunking can discard context, parsing can damage tables, metadata filters can exclude the correct passage, and ranking can favor text that resembles the question without answering it.
Generation is the third gap. A model can receive the correct passage and still misstate it, which is why evaluation has to inspect the retrieved evidence separately from the final answer, so a generation error does not get misclassified as an indexing failure.
The architecture also breaks down as the default choice when the relevant corpus can be supplied directly and governed more simply. In that case, adding ingestion, synchronization, index publication, and deletion machinery creates failure states the application did not need. The correct boundary is operational: use the separate index when repeated selection across a large or distributed corpus earns the cost of maintaining a second, explicitly observable clock.