~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
docs(vector-stores): state that an already-live add_id must be replaced (#5767)
LearningCircuit7af5ff56a4ba
diff --git a/changelog.d/+dedup-contract-replace-on-collision.misc.md b/changelog.d/+dedup-contract-replace-on-collision.misc.mdnew file mode 100644index 0000000000..bee9ada781--- /dev/null+++ b/changelog.d/+dedup-contract-replace-on-collision.misc.md@@ -0,0 +1 @@+Clarified the `VectorStore.apply()` contract: `dedup` de-duplicates ids within one call and never means "skip an add whose id is already live" — an already-live id must be replaced, because SQLite recycles AUTOINCREMENT ids on ROLLBACK and keeping the stale vector makes search rehydrate the wrong document's text.diff --git a/src/local_deep_research/vector_stores/base.py b/src/local_deep_research/vector_stores/base.pyindex df3a276224..ab5c287477 100644--- a/src/local_deep_research/vector_stores/base.py+++ b/src/local_deep_research/vector_stores/base.py@@ -215,10 +215,25 @@ class BaseVectorStore(ABC):server-backed store issues upserts/deletes) and is not part of thiscontract.- Contract: removals are applied before additions, and ``dedup`` skips- additions whose id is already present after removal — so a "replace" is- expressed as ``remove_ids`` + ``add_ids`` of the same id. On failure the- store must not be left durably half-applied.+ Contract: removals are applied before additions. A "replace" is+ expressed as ``remove_ids`` + ``add_ids`` of the same id.++ ``dedup`` de-duplicates ids WITHIN one call (a repeated id in+ ``add_ids`` is written once). It does NOT mean "skip an add whose id+ is already live in the store" — an already-live ``add_id`` MUST be+ replaced, and the NEW vector must win.++ That is not a nicety. An ``add_id`` that is already live is either an+ idempotent re-add or — critically — a REUSED ``DocumentChunk.id``+ whose prior row was rolled back: SQLite recycles AUTOINCREMENT ids on+ ROLLBACK, so a failed ``index()`` commit can leave a stale orphan+ vector under an id that a later, unrelated chunk then reuses. Keeping+ the old vector makes search rank on one document's embedding and then+ rehydrate a DIFFERENT document's text by id. Nothing downstream+ detects it: a reconciler that checks ``live_ids()`` membership sees a+ healthy index, because the skipped add left the id live.++ On failure the store must not be left durably half-applied.Returns a stats dict (e.g. ``{"added": n, "removed": m}``). ids +vectors ONLY — never text (see the module "SECURITY INVARIANT").