RAG Data Pipeline

Skip key takeaways

Key takeaways: RAG data pipeline 5

Updated August 16, 2026

Why a RAG corpus behaves like a live system with state rather than a build artifact, where vendor documentation for five major vector stores gives contradictory answers to the same event and why no source supports a refresh cadence.

• Retrieval failures that read as model problems are write-side omissions in the corpus Retrieval failures that read as model problems are, in this piece's argument, write-side omissions in the corpus: an unresolved identity, an unpropagated delete, an unmigrated embedding space. Diagnosing the write side first is the actual claim of this article.
• The claim that a vector store catches duplicates automatically is false for two of three products checked The claim that a vector store will catch a duplicate document automatically is false for two of the three products documented in this piece, Pinecone and Qdrant, both of which silently overwrite a repeated id rather than rejecting it. Only Weaviate, among the products checked, treats the same event as an error.
• Whether an embedding model change forces a rebuild depends entirely on the product Whether changing an embedding model forces rebuilding the index depends on the product. Qdrant supports migrating in place since a specific documented version; Azure AI Search, Weaviate, Elasticsearch and Pinecone all document that it effectively requires a rebuild, by four different mechanisms.
• Deleted is not one contract: six products define it six different ways Deleted is not one contract. Six products documented in this piece define it six different ways, from an asynchronous graph cleanup to a compaction trigger the caller does not control, and at least two cap how many records a single delete request can remove.
• No vendor documentation recommends a refresh cadence; the alternative is a declared staleness budget No vendor documentation found for this piece recommends a refresh cadence for a retrieval corpus. The sourced alternative is a declared staleness budget set per source class, chosen deliberately rather than left as an unstated trade.

Copy takeaways

If you are building a RAG system that carries production traffic, we can help you resolve duplicate identities and plan an embedding migration before corpus drift reaches your readers.

See our RAG knowledge systems services

A retrieval system that answered correctly for a year can start answering confidently and wrongly with the model untouched and the prompt untouched. The read side gets blamed first: a weak reranker, a bad prompt template, a top-k that is too small. The corpus itself gets treated as a build artifact, assembled once and now just sitting there. It is not one. A RAG corpus is a live system with mutable state, and the state changes whether or not anyone updates the query pipeline that reads it. This piece follows one document through its life inside that corpus, from the moment it first arrives to the moment every vector derived from it must be re-derived because the model that produced it retired, and asks at each point what write the system had to perform, and what wrong answer a reader saw when that write did not happen.

Seven vendors, one write-side diagnosis

In short: a retrieval failure that looks like a model problem can trace back to a write-side omission: an upsert that never fired, a delete that never propagated to every place holding a copy of the vector, a duplicate that was never collapsed at the identity level, an embedding space invalidated by a model retirement nobody was tracking. This piece checked vendor documentation for seven products: Pinecone, Qdrant, Weaviate, Elasticsearch, Azure AI Search, Vertex AI Vector Search and Amazon Bedrock Knowledge Bases. Not every event is sourced for every product; each section below states its own subset. Where documentation was checked, the answers differ, and one disagreement is stark enough to state here directly: Weaviate refuses a duplicate id outright as an error, while Pinecone and Qdrant both silently overwrite it. Building a corpus that stays correct means picking a documented answer for each event, not assuming one.

The answer was wrong and the model was fine

Call the failure by its actual name. We use index drift to mean a stored representation, a chunk, an embedding or a metadata field, that no longer matches the source document it was derived from. That is our working definition, not a term any vendor documentation uses; none of the products checked here name the phenomenon, only its mechanisms. Retrieval infrastructure has no governing specification either: no standards body publishes a primary reference for how a vector store must behave, and no published benchmark measures any of the events this piece traces, so every fact below is vendor documentation about a named product, not a general standard. It is also a different thing from the drift this site covers elsewhere as model or output drift, a change in what the model produces for the same input over time. Index drift is a change in what the corpus holds for the same source document, and a system can have one without the other. A reader chasing the wrong kind of drift will spend a release cycle retuning a model that was never the problem.

Five failures, one causal order

The five failures this piece traces, an unresolved identity, an unindexed change, an uncollapsed duplicate, an unpropagated deletion and an unmigrated embedding space, are causally ordered rather than independent. An identity failure at ingestion makes a later change undetectable, which is why the trace below starts with identity rather than with chunking, even though chunking happens first in most descriptions of a pipeline.

This article assumes the retrieval system already exists. Whether to build one at all, and how it compares to fine-tuning, is a decision this site covers separately at RAG versus fine-tuning; that piece ends where this one begins, with the system already shipped and already carrying production traffic.

The document arrives and nothing downstream knows what "the same document" means

The first write a corpus needs is not an embedding. It is an identity. Every event that follows, a change, a duplicate arrival, a withdrawal, only becomes detectable if the system can already answer "have I seen this exact document before". Get that wrong on the first ingestion and there is nothing stable to key a change, a duplicate check or a deletion against for any event that follows. To a reader, that failure looks the same every time: an answer grounded in a copy of the document that nobody downstream can tell apart from any other copy.

Two vendors, two ways to answer it

Weaviate documents the version that a content-derived key makes possible: "Object IDs are not randomly generated. The same value always generates the same ID." (Weaviate object creation docs), with a stated reason: "Weaviate throws an error if you provide a duplicate ID. Use deterministic IDs to avoid inserting duplicate objects." (Weaviate object creation docs). An id that is a function of content, rather than an autoincrement or a random identifier assigned at write time, makes the identity question answerable before a single vector gets computed.

Elasticsearch approaches the same problem from a different angle, closer to a digest than to an id: "Every write operation run on a document, deletes included, causes its version to be incremented." (Elasticsearch delete API docs), and that version number can gate a write directly: "Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document." (Elasticsearch delete API docs). That turns the store itself into the place a source system's own version travels, rather than something the ingestion pipeline has to track separately.

Between them these two mechanisms are the whole of what this event's write has to accomplish: a stable key derived from the document, and a way to tell whether the content behind that key has moved since it was last written. Neither vendor names it as an identity strategy. Both build it because nothing else in their product works without it.

The cut that separates a rule from its exception

A token count copied from someone else's benchmark is a number about their documents, not the reader's. What the vendor documentation checked for this piece actually supports is narrower and more useful: a boundary criterion, not a size, for where a chunk should end.

Amazon Bedrock Knowledge Bases states the criterion directly, as a constraint its chunker respects even when a size budget would tolerate more: "The chunker respects logical document boundaries (such as pages or sections) and does not merge content across these boundaries, even when increasing the maximum token size would otherwise allow for larger chunks." (Amazon Bedrock chunking docs) The source document's own structure outranks whatever the token budget would technically allow. A chunk boundary placed inside a clause that carries an exception, rather than at the section break preceding it, is how a rule gets retrieved without its qualification and reads as a confident wrong answer instead of a missing one.

The other half of the write is carrying structure forward rather than discarding it at chunk time. The same product names the pattern: "Hierarchical chunking involves organizing information into nested structures of child and parent chunks." (Amazon Bedrock chunking docs) A chunk record carrying its parent id and the source document's own offsets lets a later process reconstruct where a retrieved fragment sat, which matters most exactly when that fragment is the exception to a rule stated two paragraphs earlier in the source.

The source changed and the store did not

A reader experiences this event as retrieval failure: the source document is edited, the vector store keeps serving the version it embedded earlier, and the answer that comes back is grounded in text that is no longer true. The corpus was not wrong when it was written. It went wrong later, silently, when something else changed and nothing told it.

What re-ingestion actually does when a vendor owns it

Amazon Bedrock Knowledge Bases publishes the fullest documented statement of this write found for this piece. Its sync is incremental: "Syncing is incremental, so Amazon Bedrock only processes added, modified, or deleted documents since the last sync." (Amazon Bedrock sync docs) For anything flagged as changed, the product states plainly what processing means: "The document is re-ingested (re-parsed, re-chunked, re-embedded, and re-indexed)." (Amazon Bedrock sync docs) The default is to re-derive the entire chain, not to patch a field. The one documented optimization skips the embedding call, and only under three named conditions at once: the change touched only metadata, the content file is not CSV, and no custom transformation function is configured. Even within those conditions the vendor's own description stays narrow: "This optimization retrieves existing vector embeddings from the vector store, merges the new metadata, and writes the updated embeddings back, which avoids calls to the embedding model." (Amazon Bedrock sync docs) Full re-derivation is the default; skipping the embedding call is the deliberately narrow exception.

Two architectures, one fork

Not every product owns this step. Weaviate re-embeds automatically when a previously vectorised property changes: "If you update the value of a previously vectorized property, Weaviate re-vectorizes the object automatically. This also reindexes the updated object." (Weaviate update docs), but draws a sharp line: "Weaviate doesn't re-vectorize and re-index existing objects when a new property is defined, only when an existing property is updated." (Weaviate update docs). The trigger is an update event on an existing field, not a general comparison against the source.

Pinecone, Qdrant, Elasticsearch and Vertex AI Vector Search sit on the other side of that fork. They store the vector they are handed and have no view of the document it came from, so a source change produces nothing on its own, because no component has that as its job. Vertex documents its partial-update contract in exactly those terms: "Only the vectors specified in Index.metadata.contentsDeltaUri are updated, inserted, or deleted. The other existing embeddings in the index remain." (Google Cloud Vector Search update docs) Everything else keeps answering from the old vector until an external pipeline detects the change and writes an upsert keyed to the identity established above. Skipping that upsert is not a bug in the vector store. It is the pipeline never having built the piece whose job that was.

The same document arrived twice by two paths

A document that arrives twice, once from a scheduled crawl and once from a manual re-upload, produces two chunks with identical content and two separate embeddings unless something collapses them before they reach the index. Left uncollapsed, duplicates crowd the top of a similarity search and one document ends up answering questions it was never the best source for.

Collapsing the pair depends entirely on the identity decision made at first ingestion. If the id is a function of the content, as in Weaviate's deterministic derivation described above, a duplicate arrival is the same id twice, and identity catches it before a vector is ever computed. If the id is assigned freshly on each write, nothing catches it, because two writes with different ids look like two different documents to every downstream component.

Three products, three different responses

What happens next diverges sharply by product, against what an engineer new to the field would assume. Weaviate treats a repeated id as loud: "Weaviate throws an error if you provide a duplicate ID." (Weaviate object creation docs) Pinecone treats it as a silent replace: "If a record ID already exists, upserting overwrites the entire record." (Pinecone upsert docs) Qdrant states the same silent behavior as a design feature: "All APIs in Qdrant, including point loading, are idempotent." (Qdrant points docs) Its own documentation clarifies what that means concretely: "points with the same id will be overwritten when re-uploaded." (Qdrant points docs)

That is a genuine three-way split on the same event, not a difference in wording. One product refuses the write outright. Two others accept it and quietly discard whichever version arrived second, with no error and no signal that a collision happened at all. The assumption that a vector store will catch a duplicate for you is false of two of the three products checked here, Pinecone and Qdrant, and true only of the third, Weaviate, which treats a repeat as an error.

The document was withdrawn and the deletion did not propagate

A source document gets pulled, a page gets taken down, a contract expires, and the corpus is still citing it. "Deleted" sounds like one well-defined operation. Reading what six of the seven vendors actually document for it, it is not.

Six products, six different contracts

Weaviate marks an object deleted and leaves it connected to the search structure until a background pass catches up: "Cleanup is an async process runs that rebuilds the HNSW graph after deletes and updates. Prior to cleanup, objects are marked as deleted, but they are still connected to the HNSW graph." (Weaviate vector indexing docs) That is a documented window in which a deleted document is still part of what a query searches against.

Elasticsearch keeps a version tombstone rather than a graph edge, and it expires: "The version number of a deleted document remains available for a short time after deletion to allow for control of concurrent operations." (Elasticsearch delete API docs) The tombstone's lifetime is left to a configurable setting, index.gc_deletes. The tombstone exists to arbitrate concurrent writes, not to filter search results, and once it lapses the store has no memory the document ever existed, which opens a separate door: a late, stale write can resurrect a record the pipeline believes is gone.

Vertex AI Vector Search defers a batch index's delete until a rebuild the caller does not directly trigger: "For batch indexes, datapoint deletion is delayed until compaction occurs." (Google Cloud Vector Search update docs) Qdrant keeps deleted points around for a while as part of routine background optimization. Amazon Bedrock's entire published contract for the event is one sentence with no timing attached: "The document is removed from the vector store." (Amazon Bedrock sync docs) Pinecone states only that writes are eventually consistent, with nothing said about how long a deleted vector stays physically present. Six products, six different answers, and one has no answer at all. Any sentence describing what a delete does in general is wrong about at least five of these six.

The caps nobody notices until a big withdrawal

Deletion also has a size limit in more than one product, where a large withdrawal partially fails without anyone noticing. Weaviate caps how many objects a single delete query can remove, 10,000 by default, and states plainly that the caller is responsible for the rest: "To delete more objects than the limit, run the same query multiple times until no objects are matched anymore." (Weaviate delete docs) Pinecone caps a delete-by-id request at 1,000 ids per call. A withdrawal producing more chunks than either cap is a multi-request operation, and a caller that issues one request and checks for a non-error response has removed a prefix of the set with no signal the rest survived.

None of the products checked here document a tombstone-and-filter pattern, keeping the record in place and excluding it from results with a metadata flag, as a supported feature. That separation between gone from the index and gone from search results is the pipeline's own job to build. That is our position, not a vendor claim.

The day every vector in the store must be re-derived

Every other event in this piece touches one document. This one touches all of them at once, because every vector in the store was derived from a model, and when that model retires, every one of those vectors needs to be re-derived from whatever replaces it. That is not hypothetical. Model providers retire embedding models on a published schedule, and retirements have already happened. OpenAI shut down its first generation of text embedding models on a fixed date: "They will be shut down on January 04, 2024." (OpenAI deprecations page) Cohere states retirement as ongoing policy: "As Cohere launches safer and more capable models, we will regularly retire old models." (Cohere deprecations page) It has a date already announced for its previous generation: "Effective April 4th, 2026, the following models will be retired" (Cohere deprecations page). Google commits only to not moving a date earlier: "While retirement timelines may be extended, they won't be moved to an earlier date than what is listed." (Google Cloud model versions page) A corpus built on any of these models has a hard deadline on a full re-embed, set by a third party on its own schedule.

Two vendors publish a migration procedure

Qdrant is the only product here that documents embedding model migration by that name, and it offers two strategies. One is a shadow migration inside a live collection: "Named vectors can be added to or removed from an existing collection without having to recreate the collection. This is useful for embedding model migration" (Qdrant collections docs), followed by the procedure: "you can add a new vector for the new model, re-embed points in the background, and then remove the old vector when you’re ready." (Qdrant collections docs) Qdrant also documents the gap in the middle of that migration: "Existing points will not have values for the newly added vector until they are upserted again. The new vector can be queried immediately, but will return no results until it is populated." (Qdrant collections docs) That is the silent half-migration failure in a vendor's own words: an empty result and a genuine non-match look identical from outside, so a partially backfilled corpus fails quietly rather than loudly.

Qdrant's second strategy is a parallel build with an atomic pointer swap: "In a production environment, it is sometimes necessary to switch different versions of vectors seamlessly. For example, when upgrading to a new version of the neural network." (Qdrant collections docs) The documentation then states: "it is possible to build a second collection in the background and then switch alias from the old to the new collection." (Qdrant collections docs) The switch itself is atomic: "no concurrent requests will be affected during the switch." (Qdrant collections docs) Azure AI Search documents the same shape independently: "For production schema changes, create and test a new index side by side, then use an index alias to swap indexes without changing application code." (Azure AI Search reindex docs) A straight rebuild is destructive: "Queries targeting the index are immediately dropped. Remember that deleting an index is irreversible" (Azure AI Search reindex docs).

Two vendors publish that they cannot do it in place

Weaviate documents the opposite for its own product, and scopes the claim carefully: the capability is about adding a property to an existing collection, not about migrating an embedding model, with a workaround stated as a full round trip: "Export the existing data from the collection. Re-create it with the new property. Import the data into the updated collection." (Weaviate collection operations docs) It adds that an in-place path does not exist yet: "We are working on a re-indexing API to allow you to re-index the data after adding a property. This will be available in a future release." (Weaviate collection operations docs) Elasticsearch documents a structural reason a mapped field cannot move once written: "you can’t change the mapping or field type of an existing field. Changing an existing field could invalidate data that’s already indexed." (Elasticsearch explicit mapping docs) The remedy is named directly: "create a new index with the correct mapping and reindex your data into that index." (Elasticsearch explicit mapping docs) Pinecone documents no partial path for a full clear: "To remove all records from an index, delete the index and recreate it" (Pinecone delete docs).

Put together, that is a direct contradiction on the same question, not a difference of emphasis. Whether an embedding model change forces rebuilding the store: Qdrant says no, since a specific documented version. Azure AI Search, Weaviate, Elasticsearch and Pinecone all say yes, by four different routes. The sentence "changing your embedding model means rebuilding your index" is a fair description of four of the five vector stores compared for this question and a false description of the fifth.

A second, sharper contradiction sits underneath it. Whether two models' vectors can live in one store at all: Azure AI Search states flatly that one field holds embeddings from a single model, "The embedding space consists of all vector fields populated with embeddings from the same embedding model." (Azure AI Search index docs) Mixing dimensions there is an indexing error rather than a degraded state. Qdrant states the opposite for its own architecture: a collection enforces one dimensionality per named vector, and a single stored point can carry several named vectors, each with its own dimensionality and its own distance metric. Both statements are correct descriptions of their own products and neither generalises to the other's.

What the vendors do not say, and what we say instead

No vendor documentation fetched for this piece states that vectors produced by two different embedding models are mathematically incomparable, and that claim does not appear anywhere in this article as a fact. What the vendors document instead is the operational shape of the problem: mixed dimensions are rejected outright in at least one product, a half-populated new vector space returns nothing for the documents not yet re-embedded, and every product examined either forbids two models in one space or builds a specific, separately documented mechanism to accommodate it. Our own reading of that pattern, stated as our position and not as a vendor claim, is that a corpus is safest treated as belonging to exactly one embedding model at a time, with a migration as a deliberate, bounded event carrying its own cutover rather than a background drift nobody scheduled. Building that cutover correctly, for a corpus already carrying production traffic, is the scope our RAG and knowledge systems service is designed to cover.

Refresh cadence as a stated budget rather than a cron line

Every event traced above, a stale write, a duplicate, an unpropagated delete, a retired model, eventually gets asked the same question: how fresh does this have to be. The honest answer is that no vendor documentation checked for this piece recommends an interval, not a refresh frequency, not a reindex schedule, not a re-embed cadence, across any of the products above. The nearest thing found is Vertex AI Vector Search's own compaction heuristic for its own batch indexes, triggered internally once accumulated changes cross a threshold or a pending change reaches a fixed age. That is Google's internal trigger for its own storage engine, not advice to anyone building on top of it, and treating it as a benchmark cadence elsewhere would be exactly the number-laundering this piece has avoided throughout.

A budget instead of a cadence

What can be stated instead is a mechanism, not a number: a staleness budget, declared per source class rather than assumed globally. A source that changes daily and a source that changes once a year do not need the same freshness guarantee, and a single global cadence is either wasteful for the slow source or dangerous for the fast one. Declaring the budget explicitly converts "how fresh is the index" from a question nobody on the team can answer into a stated trade chosen on purpose. Our position is that the budget belongs on paper before the migration in the previous section, not derived after an incident, because that section shows how expensive an undeclared trade becomes once a model retirement forces the question anyway.

What you have to be able to see before any of this is debuggable

Every failure traced above produces the same first question during an incident: is this document even correctly represented in the store right now. Answering it needs telemetry, and the two instruments an engineer reaches for first both turn out to be documented as unreliable by the vendors that ship them.

The write that is not yet visible

A successful write is not necessarily a visible one. Qdrant is explicit that its default acknowledgement is not a completion guarantee: "This response does not mean that the data is available for retrieval yet ... it is possible that such request eventually fails." (Qdrant points docs), and offers an opt-in flag that blocks until the write is actually searchable. Pinecone documents the same shape more softly: "Pinecone is eventually consistent, so there can be a slight delay before new or changed records are visible to queries." (Pinecone delete docs) Elasticsearch documents a default refresh with a condition easy to leave out of a summary: roughly one second, but "only on indices that have received one search request or more in the last 30 seconds" (Elasticsearch near real-time search docs); a quiet index is not refreshing on that cadence at all. Amazon Bedrock documents a latency two orders of magnitude larger for some configurations: "it could take a few minutes for the vector embeddings of the newly synced data to reflect in your knowledge base" (Amazon Bedrock sync docs), depending on which underlying vector store was chosen at creation. A pipeline that assumes a fixed visibility latency is assuming a number true of at most one vendor's default configuration.

Pinecone is the one product here that publishes an actual mechanism for checking rather than assuming: every write gets a monotonically increasing sequence number, carried in a response header, and a query response carries one too, so a caller can confirm a specific write is reflected in a specific query rather than guessing from elapsed time.

The counters that lie about freshness

The other obvious instrument, a record count, is worse than merely imprecise. Pinecone states the arithmetic reason directly: "if you delete the same number of records that you insert, the expected record count may remain the same" (Pinecone freshness docs). Qdrant is blunter about its own point counters: "The above counts are not exact, but should be considered approximate." (Qdrant collections docs), and: "It’s therefore important not to rely on them." (Qdrant collections docs) Both vendors are naming the same cause from different sides: internal storage accounting lags, temporarily duplicates and retains deleted points for reasons that have nothing to do with what a caller actually wants to know, whether a specific document's representation currently matches its source.

What a vendor's own answer to this problem looks like

Amazon Bedrock built a per-sync telemetry surface for its own managed pipeline: documents scanned, newly indexed, modified-and-indexed, deleted, failed and skipped, counted separately for content and metadata. Failed and skipped are kept apart deliberately, since a document that could not be processed and one correctly determined not to need processing are different states. None of that surface is per document, though. It answers how the last sync went, not what the current state of one document is. A per-document view, last ingested, last embedded, embedding model version, whether the record carries a tombstone, is not documented by any product checked here. Building it is our own position, and it is the natural companion to the per-call telemetry this site already covers for model-side observability and cost, which instruments what the model did rather than what the corpus currently holds.

The smallest evaluation that proves a refresh actually helped

Measuring retrieval quality is a saturated topic elsewhere, recall@k, nDCG, the RAGAS and ARES families of evaluation, and repeating that coverage here would add nothing. The only question this piece needs answered is narrower: after a change above lands, a re-embed, a deletion, a duplicate collapse, did retrieval actually get better for the documents that change touched. That needs a small, fixed evaluation set built from real queries whose correct source document is known, re-run before and after the change, scoped to the affected documents rather than the whole corpus. No benchmark or published methodology was found in the research behind this piece for that narrow check specifically, so this stays a pointer: build the smallest set that exercises the documents just touched, and re-run it every time one of the events above fires, rather than trusting that the write succeeded.

What an agent does to a corpus that a chatbot does not

Everything traced above assumes a human, or a scheduled job, is the only thing writing to the corpus. An agent breaks that assumption. An agent that reads the corpus repeatedly over the course of one task, and writes findings or intermediate state back into it, is a second writer with none of the discipline this piece has argued for: no stable identity strategy, no digest to key an upsert against, no declared staleness budget, no telemetry watching what it just did. Every failure traced above gets a second, less supervised source the moment a corpus stops being read-only.

Our position, stated as one

This is our own position, stated as one, because no vendor documentation checked for this piece addresses agent memory write-back directly. The same identity, change-detection and deletion contracts a corpus needs for human and scheduled writers apply without exception to a machine writer. The architecture patterns that specify how an agent reads a corpus repeatedly during a task are covered separately at AI agent architecture patterns; what happens to the corpus itself once that agent starts writing back into it is the open half of the question, and it is where this piece ends.

Sources: vendor documentation retrieved 2026-08-16. OpenAI Platform (Deprecations, Embeddings guide); Cohere (Deprecations); Google Cloud (Model versions and lifecycle, Vertex AI Vector Search); Pinecone (Create an index, Upsert data, Delete records, Check data freshness); Qdrant (Collections, Points); Weaviate (Create, Update and Delete objects, Collection operations, Vector indexing concepts); Elasticsearch (Explicit mapping, dense_vector, Near real-time search, Delete document API); Azure AI Search (Create a vector index, Update or rebuild an index); Microsoft Foundry (Model lifecycle and retirements); Amazon Bedrock Knowledge Bases (Sync your data, chunking, IngestionJobStatistics). This is engineering guidance drawn from vendor documentation as it read on the retrieval date above, not a benchmark result, since vendor pages change without a version marker or a changelog.

FAQ

Last updated: August 16, 2026

Quick answers to common questions about custom software development, pricing, process and technology.

Expand all Collapse all

Search FAQs

Popular

Type to filter questions and answers. Use Topic to narrow the list.

Showing all 7

• ★ Featured question. What is index drift in a RAG system?

  Copy link Copies a direct link to this answer to your clipboard.

  We use index drift to mean a stored representation, a chunk, an embedding or a metadata field, that no longer matches the source document it was derived from. No vendor documentation checked for this piece names the phenomenon; the term is ours.

  It is a different thing from model or output drift, a change in what the model produces for the same input over time, which is the drift this site covers elsewhere in its production AI engineering coverage. A corpus can drift while the model stays fixed, and diagnosing the wrong one wastes a release cycle.

• ★ Featured question. Do vector stores catch duplicate documents automatically?

  Copy link Copies a direct link to this answer to your clipboard.

  Not consistently, and the split is worth knowing before it costs an incident. Weaviate treats a repeated deterministic id as an error and refuses the write.

  Pinecone and Qdrant both treat a repeated id as a silent full overwrite, with Qdrant documenting the behavior as a deliberate idempotence feature rather than an edge case. The assumption that a vector store will catch a duplicate automatically is false of two of the three products checked here. Collapsing duplicates has to happen at the identity layer, before the write, if the store in use is one of the two that overwrite silently.

• How do you delete a document from a vector store so it stops being retrieved?

  Copy link Copies a direct link to this answer to your clipboard.

  There is no single documented answer, because deleted means something different in each product examined. Weaviate marks an object deleted but keeps it connected to the search graph until an asynchronous cleanup runs.

  Elasticsearch keeps a tombstone with a configurable lifetime. Vertex AI Vector Search defers a batch index's delete until an internally triggered compaction. Two products also cap how many records a single delete request can remove, so a large withdrawal can partially succeed without an error. Verifying a deletion actually took effect, rather than assuming the delete call did, is the safer default across every product checked.

• Do you have to re-embed everything when you change the embedding model?

  Copy link Copies a direct link to this answer to your clipboard.

  It depends entirely on which product holds the corpus, and the vendors disagree directly. Azure AI Search, Weaviate, Elasticsearch and Pinecone all document that a model change effectively forces a rebuild, by four different routes.

  Qdrant is the exception: since a specific documented version it supports adding a new named vector to a live collection, backfilling it in the background and cutting over, without recreating the collection. A blanket claim either way is wrong about at least one of the five products checked here.

• How long until a newly written vector is visible to a query?

  Copy link Copies a direct link to this answer to your clipboard.

  Every product examined documents some form of delay, and the documented defaults differ by roughly two orders of magnitude. Elasticsearch's default refresh is close to one second, but only on indices that have been searched in the last 30 seconds.

  Amazon Bedrock documents a delay of a few minutes for some vector store configurations. Pinecone and Qdrant decline to state a fixed number at all; Pinecone instead publishes a sequence number a caller can check directly, and Qdrant offers an opt-in flag that blocks until a write is confirmed searchable. There is no single safe number to assume.

• How often should a RAG index be refreshed?

  Copy link Copies a direct link to this answer to your clipboard.

  No vendor documentation checked for this piece states a refresh cadence, a reindex frequency or a re-embed schedule, for any of the products examined. That absence is itself the finding, not a gap in the research.

  What we recommend instead of a number is a declared staleness budget per source class, a stated answer to how out of date a given kind of document is allowed to get, set deliberately rather than left as an unstated trade nobody chose on purpose.

• Why does a RAG system return an answer from a document that was deleted?

  Copy link Copies a direct link to this answer to your clipboard.

  Because in most of the products checked here, a delete does not remove the vector immediately or does not remove it in a way retrieval can see right away. Weaviate keeps a deleted object connected to its search graph until a background cleanup runs.

  Qdrant's own storage layer may keep deleted points around briefly as part of routine optimization. Vertex AI Vector Search defers a batch index's delete until compaction. A system answering from a document withdrawn minutes ago has not necessarily failed. In several of the products examined, that is the documented behavior.

I work with startup founders who need a dedicated software development team but don’t want to gamble on hiring, random outsourcing, or opaque delivery.
Most founders face the same problem sooner or later.
Early technical and team decisions lock the product into tech debt, slow delivery, missed milestones and constant re-hiring. By the time this becomes visible, fixing it is already expensive.

As a CTO and software architect, I help founders design, build and run dedicated development teams that work as a true extension of the startup. Not as a black-box vendor.

My focus is on complex products where mistakes are costly:

• Web3 and blockchain platforms
• FinTech and regulated products
• High-load startup systems
• MVP → scale transitions

We don’t do body-shopping.
We don’t sell generic outsourcing.

Instead, we help founders:

• build the right team structure from day one
• keep technical ownership and transparency
• scale delivery without losing control
• avoid vendor lock-in and hidden risks

Teams are aligned with the product roadmap, business goals and long-term architecture. Not just short-term velocity.