Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

docs(vector-stores): state that an already-live add_id must be replaced (#5767)

LearningCircuit7af5ff56a4ba
diff --git a/changelog.d/+dedup-contract-replace-on-collision.misc.md b/changelog.d/+dedup-contract-replace-on-collision.misc.md
new file mode 100644
index 0000000000..bee9ada781
--- /dev/null
+++ b/changelog.d/+dedup-contract-replace-on-collision.misc.md
@@ -0,0 +1 @@
+Clarified the `VectorStore.apply()` contract: `dedup` de-duplicates ids within one call and never means "skip an add whose id is already live" — an already-live id must be replaced, because SQLite recycles AUTOINCREMENT ids on ROLLBACK and keeping the stale vector makes search rehydrate the wrong document's text.
diff --git a/src/local_deep_research/vector_stores/base.py b/src/local_deep_research/vector_stores/base.py
index df3a276224..ab5c287477 100644
--- a/src/local_deep_research/vector_stores/base.py
+++ b/src/local_deep_research/vector_stores/base.py
@@ -215,10 +215,25 @@ class BaseVectorStore(ABC):
server-backed store issues upserts/deletes) and is not part of this
contract.
- Contract: removals are applied before additions, and ``dedup`` skips
- additions whose id is already present after removal — so a "replace" is
- expressed as ``remove_ids`` + ``add_ids`` of the same id. On failure the
- store must not be left durably half-applied.
+ Contract: removals are applied before additions. A "replace" is
+ expressed as ``remove_ids`` + ``add_ids`` of the same id.
+
+ ``dedup`` de-duplicates ids WITHIN one call (a repeated id in
+ ``add_ids`` is written once). It does NOT mean "skip an add whose id
+ is already live in the store" — an already-live ``add_id`` MUST be
+ replaced, and the NEW vector must win.
+
+ That is not a nicety. An ``add_id`` that is already live is either an
+ idempotent re-add or — critically — a REUSED ``DocumentChunk.id``
+ whose prior row was rolled back: SQLite recycles AUTOINCREMENT ids on
+ ROLLBACK, so a failed ``index()`` commit can leave a stale orphan
+ vector under an id that a later, unrelated chunk then reuses. Keeping
+ the old vector makes search rank on one document's embedding and then
+ rehydrate a DIFFERENT document's text by id. Nothing downstream
+ detects it: a reconciler that checks ``live_ids()`` membership sees a
+ healthy index, because the skipped add left the id live.
+
+ On failure the store must not be left durably half-applied.
Returns a stats dict (e.g. ``{"added": n, "removed": m}``). ids +
vectors ONLY — never text (see the module "SECURITY INVARIANT").