~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""Abstract base class for vector stores.
Mirrors the embeddings/LLM provider pattern (``BaseEmbeddingProvider``,
``BaseLLM`` + factory): a small typed interface plus a registry so a future
release can add backends (Qdrant, pgvector, Milvus, ...) selectable by a
setting. **Only FAISS is implemented today** — there is intentionally no
settings selector wired yet.
Scope / honest boundary
-----------------------
This abstraction covers the *vector-storage operations* — create / add /
search / delete / reconstruct / persist / count. Local-file constructors accept
injected persistence resources, but this interface deliberately does **not**
prescribe their per-user paths, integrity checksums, or reload/merge locking
model that :class:`LibraryRAGService` layers on top. Those are *local-file*
concerns (see :attr:`is_local_file`): a future server-backed store (e.g.
Qdrant) would set ``is_local_file = False``, no-op :meth:`persist` /
:meth:`load`, implement :meth:`drop_collection` (the deletion path cannot
``unlink()`` a remote collection), and the service layer would skip the file
lock + integrity machinery for it. So this base class makes adding a backend *easier* (a clean,
verified query surface), not *free* — a real server backend still requires
refactoring the file/lock/integrity layer in the service.
Identity model
--------------
Vectors are keyed by an application-supplied **int64 id**. In this codebase
that id is ``DocumentChunk.id`` (the encrypted-DB primary key); the store holds
only vectors + ids, and all text/metadata is rehydrated from the DB by id.
============================================================================
SECURITY INVARIANT — NEVER pass document text to a vector store
============================================================================
Every method here accepts ONLY an integer id and its embedding vector.
Callers MUST NOT pass document/chunk text (or any other user content) into a
vector-store method, and implementations MUST NOT persist such text.
The per-user database is encrypted; a vector index is not (it is a local file
for FAISS, or an external service for a future backend). Keeping user content
out of the vector store keeps the encrypted database the sole home of that
content. Because the interface has no text parameter, the protection is
STRUCTURAL rather than a matter of discipline — no backend (FAISS today, or a
swapped-in Qdrant/pgvector tomorrow) can receive or store the text. The
authoritative text lives solely in the encrypted DB
(``DocumentChunk.chunk_text``), keyed by the same int id; retrieval rehydrates
snippets from there by id (see ``LibraryRAGService`` / the search engines).
============================================================================
"""
from abc import ABC, abstractmethod
from pathlib import Path
from types import TracebackType
from typing import Callable, List, Optional, Protocol, Self, Sequence, Tuple
import numpy as np
IntegrityRecord = Callable[[Path], None]
IntegrityVerify = Callable[[Path], Tuple[bool, Optional[str]]]
class VectorStoreIntegrityError(ValueError):
"""An authenticated index references corrupt or missing component data."""
class WriteLock(Protocol):
"""Minimal lock capability used to guard a vector store's file writes.
Implementations MUST be reentrant. Write paths nest acquisition on a
single thread — the service layer holds the lock across a document
operation while the store's own reload/apply/persist choreography
re-acquires it — so a plain ``threading.Lock`` satisfies this Protocol
structurally but self-deadlocks at runtime. Use ``threading.RLock`` (or
a wrapper around one, as the production tracked lock is).
"""
def __enter__(self) -> bool | Self: ...
def __exit__(
self,
exception_type: type[BaseException] | None,
exception: BaseException | None,
traceback: TracebackType | None,
) -> None: ...
class BaseVectorStore(ABC):
"""Abstract interface for a per-collection vector index.
An instance wraps a single collection's index (stateful, in-memory, backed
by a persisted file for local-file stores). Construct via :meth:`create`
(new, empty) or :meth:`load` (from disk).
"""
# Override in subclasses.
provider_key: str = "base" # unique id; matches the (future) setting value
provider_name: str = "Base" # display name for logs/UI
# True when the store persists to a local file this process owns end to end
# — i.e. the service's file write-lock + integrity checksum + reload/merge
# model applies. A server-backed store sets this False.
is_local_file: bool = True
# Settings key holding this store's endpoint URI, for a store that is NOT
# a local file. The egress policy reads it exactly as it reads a search
# engine's ``url_setting`` (``security/egress/policy.py``): a
# server-backed store whose configured host resolves to a PUBLIC address
# ships the collection's embeddings and every query off the machine, so
# the library/collection engines fail up to exposing/public and
# PRIVATE_ONLY denies them. A non-local-file store that declares no key
# cannot be shown to be on-box and is therefore treated as remote —
# declaring it is how a backend proves the opposite. Unused for a
# local-file store (FAISS), which is contained by construction.
uri_setting: Optional[str] = None
# True when the store can invert id -> vector (needed for the in-place
# format migration, which re-keys vectors without re-embedding).
supports_reconstruct: bool = True
# Embedding dimension this instance was constructed/loaded for. Concrete
# subclasses set this in __init__/create()/load(); declared here (no
# default — there is no sensible universal value) so callers like the
# facade can read it via the abstract interface.
dimension: int
# ------------------------------------------------------------------ #
# Construction
# ------------------------------------------------------------------ #
@classmethod
@abstractmethod
def create(
cls,
*,
dimension: int,
index_type: str,
metric: str,
normalize: bool,
path: Optional[Path] = None,
lock: Optional[WriteLock] = None,
integrity_record: Optional[IntegrityRecord] = None,
integrity_verify: Optional[IntegrityVerify] = None,
) -> "BaseVectorStore":
"""Create a new, empty store for vectors of ``dimension``.
Args:
dimension: Embedding dimension.
index_type: Backend index family (e.g. ``"flat"``, ``"hnsw"``).
metric: Distance metric (``"l2"``, ``"cosine"``, ``"dot_product"``).
normalize: Whether query/doc vectors are L2-normalized before use
(cosine similarity via inner product on normalized vectors).
path: Optional local persistence path.
lock: Optional reentrant lock for local persistence (see
:class:`WriteLock`).
integrity_record: Optional persisted-file integrity callback.
integrity_verify: Optional persisted-file integrity check.
"""
@classmethod
@abstractmethod
def load(
cls,
path: Path,
*,
dimension: int,
index_type: str,
metric: str,
normalize: bool,
lock: Optional[WriteLock] = None,
integrity_record: Optional[IntegrityRecord] = None,
integrity_verify: Optional[IntegrityVerify] = None,
) -> "BaseVectorStore":
"""Load a persisted store from ``path`` (local-file stores only).
``dimension`` / ``index_type`` / ``metric`` / ``normalize`` describe how
the index was built (some backends persist this, some do not); callers
pass the values from settings so search-time normalization matches
build-time normalization. ``lock`` / ``integrity_record`` /
``integrity_verify`` are the same injected persistence resources as on
:meth:`create`.
"""
# ------------------------------------------------------------------ #
# Vector operations
# ------------------------------------------------------------------ #
@abstractmethod
def add(self, ids: List[int], vectors: np.ndarray) -> None:
"""Add vectors under the given int64 ids (aligned 1:1, same length).
ids + vectors ONLY — never text (see the module-level "SECURITY
INVARIANT" block). The text stays in the encrypted DB.
"""
@abstractmethod
def search(
self, query_vector: np.ndarray, k: int
) -> List[Tuple[int, float]]:
"""Return up to ``k`` ``(id, distance)`` pairs, nearest first.
``distance`` is the backend's raw score (L2 distance or inner product);
callers map it to a relevance score. Absent/empty slots are filtered.
"""
@abstractmethod
def delete(self, ids: List[int]) -> int:
"""Remove the given ids. Returns the number actually removed.
May raise for index families that do not support removal (e.g. FAISS
HNSW) — callers handle that the same way they did pre-abstraction.
"""
@abstractmethod
def live_ids(self) -> List[int]:
"""Return every id currently stored (the authoritative membership)."""
def iter_id_batches(self):
"""Stream membership for cleanup without retaining a corpus-sized list."""
ids = self.live_ids()
for start in range(0, len(ids), 500):
yield ids[start : start + 500]
@abstractmethod
def count(self) -> int:
"""Return the number of vectors currently stored."""
@abstractmethod
def apply(
self,
*,
add_ids: List[int],
add_vectors: Optional[np.ndarray],
remove_ids: Sequence[int] = (),
dedup: bool = True,
) -> dict:
"""Durably apply a batch of removals + additions as one atomic unit.
This is the single write primitive. The caller computes *which* ids to
remove and add (from its own authoritative source — for LDR, the
encrypted DB); the store applies them and persists so that a concurrent
writer to the same logical index cannot lose either party's changes.
*How* that atomicity/isolation is achieved is backend-specific (a
local-file backend takes a per-index lock and reload-merges; a
server-backed store issues upserts/deletes) and is not part of this
contract.
Contract: removals are applied before additions. A "replace" is
expressed as ``remove_ids`` + ``add_ids`` of the same id.
``dedup`` de-duplicates ids WITHIN one call (a repeated id in
``add_ids`` is written once). It does NOT mean "skip an add whose id
is already live in the store" — an already-live ``add_id`` MUST be
replaced, and the NEW vector must win.
That is not a nicety. An ``add_id`` that is already live is either an
idempotent re-add or — critically — a REUSED ``DocumentChunk.id``
whose prior row was rolled back: SQLite recycles AUTOINCREMENT ids on
ROLLBACK, so a failed ``index()`` commit can leave a stale orphan
vector under an id that a later, unrelated chunk then reuses. Keeping
the old vector makes search rank on one document's embedding and then
rehydrate a DIFFERENT document's text by id. Nothing downstream
detects it: a reconciler that checks ``live_ids()`` membership sees a
healthy index, because the skipped add left the id live.
On failure the store must not be left durably half-applied.
Returns a stats dict (e.g. ``{"added": n, "removed": m}``). ids +
vectors ONLY — never text (see the module "SECURITY INVARIANT").
"""
# ------------------------------------------------------------------ #
# Optional (local-file / reconstructable stores)
# ------------------------------------------------------------------ #
def reconstruct(self, id: int) -> Optional[np.ndarray]:
"""Return the stored vector for ``id`` (migration helper).
Only meaningful when :attr:`supports_reconstruct` is True.
"""
raise NotImplementedError(
f"{type(self).__name__} does not support reconstruct()"
)
def persist(self, path: Path) -> None:
"""Durably write the store to ``path`` (local-file stores only)."""
raise NotImplementedError(
f"{type(self).__name__} does not support persist()"
)
# ------------------------------------------------------------------ #
# Lifecycle (server-backed stores)
# ------------------------------------------------------------------ #
@classmethod
def drop_collection(
cls,
*,
username: str,
index_hash: str,
settings_snapshot: Optional[dict] = None,
) -> bool:
"""Permanently remove the backing collection for one RAG index.
The deletion counterpart of :meth:`create`, so the library deletion
cascade can dispose of vectors it cannot simply ``unlink()``. It is a
**classmethod on purpose**: a deletion path must be able to drop a
collection without first loading it (loading a possibly-corrupt or
already-absent index just to delete it is both wasteful and
failure-prone).
Identity
--------
The collection is addressed by ``(username, index_hash)`` rather than
by an instance, because those two values are the only stable handle
available at deletion time: ``index_hash`` lives on the ``RAGIndex``
row, and that row is deleted by the same cascade. An implementation
MUST derive the backend collection name from these same two inputs at
BOTH creation and deletion, or the name it computes here will not
match the one it created and drop_collection will silently miss.
Nothing enforces that yet: :meth:`create` does not currently receive
``username`` or ``index_hash`` (it takes ``dimension``/``index_type``/
``metric``/``normalize``/``path``/``lock``/``integrity_record``/
``integrity_verify``), so there is no shared derivation to reuse. The
first server-backed provider MUST extend the creation path to accept
both, thread them through into whatever it uses to name the remote
collection, and use that exact derivation here too (see the follow-up
design issue, #6769).
**Ordering contract**: callers MUST invoke this while the ``RAGIndex``
row still exists. Once the row is gone ``index_hash`` is unrecoverable
and the remote collection becomes *unaddressable* -- orphaned vectors
that no longer have a name (see issue #5765).
Contract for implementations:
* Idempotent -- return ``False`` (not raise) when the collection is
already absent, ``True`` when one was actually dropped.
* Raise on *transport / permission* failure. Returning ``False`` for an
unreachable backend would let the caller record a successful deletion
while the user's vectors survive on third-party infrastructure.
* Make the failure classifiable. The cascade tells the user to retry
only for a *transient* failure: a ``ConnectionError`` /
``TimeoutError`` (or subclass), or any exception carrying a boolean
``retryable`` attribute set to True. Everything else -- auth or
permission errors, missing connection config, unrecognised client
exceptions -- is reported as permanent, so translate the backend
client's own "unreachable" / "timed out" exceptions accordingly.
* Bound each call with a client timeout: the cascade calls this inside
the user's database transaction.
* Read connection config from ``settings_snapshot`` -- the deletion
callers pass the user's snapshot. Only the exception's type name is
surfaced to users and logs, never its message.
Local-file stores (:attr:`is_local_file` True) do not implement this --
their deletion path unlinks the index files instead -- so the base
implementation raises :class:`NotImplementedError`. Server-backed
stores MUST override it.
"""
raise NotImplementedError(
f"{cls.__name__} does not support drop_collection()"
)