Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

16.2 KB
Raw Blame
"""Abstract base class for vector stores.

Mirrors the embeddings/LLM provider pattern (``BaseEmbeddingProvider``,
``BaseLLM`` + factory): a small typed interface plus a registry so a future
release can add backends (Qdrant, pgvector, Milvus, ...) selectable by a
setting. **Only FAISS is implemented today** — there is intentionally no
settings selector wired yet.

Scope / honest boundary
-----------------------
This abstraction covers the *vector-storage operations* — create / add /
search / delete / reconstruct / persist / count. Local-file constructors accept
injected persistence resources, but this interface deliberately does **not**
prescribe their per-user paths, integrity checksums, or reload/merge locking
model that :class:`LibraryRAGService` layers on top. Those are *local-file*
concerns (see :attr:`is_local_file`): a future server-backed store (e.g.
Qdrant) would set ``is_local_file = False``, no-op :meth:`persist` /
:meth:`load`, implement :meth:`drop_collection` (the deletion path cannot
``unlink()`` a remote collection), and the service layer would skip the file
lock + integrity machinery for it. So this base class makes adding a backend *easier* (a clean,
verified query surface), not *free* — a real server backend still requires
refactoring the file/lock/integrity layer in the service.

Identity model
--------------
Vectors are keyed by an application-supplied **int64 id**. In this codebase
that id is ``DocumentChunk.id`` (the encrypted-DB primary key); the store holds
only vectors + ids, and all text/metadata is rehydrated from the DB by id.

============================================================================
SECURITY INVARIANT — NEVER pass document text to a vector store
============================================================================
Every method here accepts ONLY an integer id and its embedding vector.
Callers MUST NOT pass document/chunk text (or any other user content) into a
vector-store method, and implementations MUST NOT persist such text.

The per-user database is encrypted; a vector index is not (it is a local file
for FAISS, or an external service for a future backend). Keeping user content
out of the vector store keeps the encrypted database the sole home of that
content. Because the interface has no text parameter, the protection is
STRUCTURAL rather than a matter of discipline — no backend (FAISS today, or a
swapped-in Qdrant/pgvector tomorrow) can receive or store the text. The
authoritative text lives solely in the encrypted DB
(``DocumentChunk.chunk_text``), keyed by the same int id; retrieval rehydrates
snippets from there by id (see ``LibraryRAGService`` / the search engines).
============================================================================
"""

from abc import ABC, abstractmethod
from pathlib import Path
from types import TracebackType
from typing import Callable, List, Optional, Protocol, Self, Sequence, Tuple

import numpy as np


IntegrityRecord = Callable[[Path], None]
IntegrityVerify = Callable[[Path], Tuple[bool, Optional[str]]]


class VectorStoreIntegrityError(ValueError):
    """An authenticated index references corrupt or missing component data."""


class WriteLock(Protocol):
    """Minimal lock capability used to guard a vector store's file writes.

    Implementations MUST be reentrant. Write paths nest acquisition on a
    single thread — the service layer holds the lock across a document
    operation while the store's own reload/apply/persist choreography
    re-acquires it — so a plain ``threading.Lock`` satisfies this Protocol
    structurally but self-deadlocks at runtime. Use ``threading.RLock`` (or
    a wrapper around one, as the production tracked lock is).
    """

    def __enter__(self) -> bool | Self: ...

    def __exit__(
        self,
        exception_type: type[BaseException] | None,
        exception: BaseException | None,
        traceback: TracebackType | None,
    ) -> None: ...


class BaseVectorStore(ABC):
    """Abstract interface for a per-collection vector index.

    An instance wraps a single collection's index (stateful, in-memory, backed
    by a persisted file for local-file stores). Construct via :meth:`create`
    (new, empty) or :meth:`load` (from disk).
    """

    # Override in subclasses.
    provider_key: str = "base"  # unique id; matches the (future) setting value
    provider_name: str = "Base"  # display name for logs/UI
    # True when the store persists to a local file this process owns end to end
    # — i.e. the service's file write-lock + integrity checksum + reload/merge
    # model applies. A server-backed store sets this False.
    is_local_file: bool = True
    # Settings key holding this store's endpoint URI, for a store that is NOT
    # a local file. The egress policy reads it exactly as it reads a search
    # engine's ``url_setting`` (``security/egress/policy.py``): a
    # server-backed store whose configured host resolves to a PUBLIC address
    # ships the collection's embeddings and every query off the machine, so
    # the library/collection engines fail up to exposing/public and
    # PRIVATE_ONLY denies them. A non-local-file store that declares no key
    # cannot be shown to be on-box and is therefore treated as remote —
    # declaring it is how a backend proves the opposite. Unused for a
    # local-file store (FAISS), which is contained by construction.
    uri_setting: Optional[str] = None
    # True when the store can invert id -> vector (needed for the in-place
    # format migration, which re-keys vectors without re-embedding).
    supports_reconstruct: bool = True
    # Embedding dimension this instance was constructed/loaded for. Concrete
    # subclasses set this in __init__/create()/load(); declared here (no
    # default — there is no sensible universal value) so callers like the
    # facade can read it via the abstract interface.
    dimension: int

    # ------------------------------------------------------------------ #
    # Construction
    # ------------------------------------------------------------------ #
    @classmethod
    @abstractmethod
    def create(
        cls,
        *,
        dimension: int,
        index_type: str,
        metric: str,
        normalize: bool,
        path: Optional[Path] = None,
        lock: Optional[WriteLock] = None,
        integrity_record: Optional[IntegrityRecord] = None,
        integrity_verify: Optional[IntegrityVerify] = None,
    ) -> "BaseVectorStore":
        """Create a new, empty store for vectors of ``dimension``.

        Args:
            dimension: Embedding dimension.
            index_type: Backend index family (e.g. ``"flat"``, ``"hnsw"``).
            metric: Distance metric (``"l2"``, ``"cosine"``, ``"dot_product"``).
            normalize: Whether query/doc vectors are L2-normalized before use
                (cosine similarity via inner product on normalized vectors).
            path: Optional local persistence path.
            lock: Optional reentrant lock for local persistence (see
                :class:`WriteLock`).
            integrity_record: Optional persisted-file integrity callback.
            integrity_verify: Optional persisted-file integrity check.
        """

    @classmethod
    @abstractmethod
    def load(
        cls,
        path: Path,
        *,
        dimension: int,
        index_type: str,
        metric: str,
        normalize: bool,
        lock: Optional[WriteLock] = None,
        integrity_record: Optional[IntegrityRecord] = None,
        integrity_verify: Optional[IntegrityVerify] = None,
    ) -> "BaseVectorStore":
        """Load a persisted store from ``path`` (local-file stores only).

        ``dimension`` / ``index_type`` / ``metric`` / ``normalize`` describe how
        the index was built (some backends persist this, some do not); callers
        pass the values from settings so search-time normalization matches
        build-time normalization. ``lock`` / ``integrity_record`` /
        ``integrity_verify`` are the same injected persistence resources as on
        :meth:`create`.
        """

    # ------------------------------------------------------------------ #
    # Vector operations
    # ------------------------------------------------------------------ #
    @abstractmethod
    def add(self, ids: List[int], vectors: np.ndarray) -> None:
        """Add vectors under the given int64 ids (aligned 1:1, same length).

        ids + vectors ONLY — never text (see the module-level "SECURITY
        INVARIANT" block). The text stays in the encrypted DB.
        """

    @abstractmethod
    def search(
        self, query_vector: np.ndarray, k: int
    ) -> List[Tuple[int, float]]:
        """Return up to ``k`` ``(id, distance)`` pairs, nearest first.

        ``distance`` is the backend's raw score (L2 distance or inner product);
        callers map it to a relevance score. Absent/empty slots are filtered.
        """

    @abstractmethod
    def delete(self, ids: List[int]) -> int:
        """Remove the given ids. Returns the number actually removed.

        May raise for index families that do not support removal (e.g. FAISS
        HNSW) — callers handle that the same way they did pre-abstraction.
        """

    @abstractmethod
    def live_ids(self) -> List[int]:
        """Return every id currently stored (the authoritative membership)."""

    def iter_id_batches(self):
        """Stream membership for cleanup without retaining a corpus-sized list."""
        ids = self.live_ids()
        for start in range(0, len(ids), 500):
            yield ids[start : start + 500]

    @abstractmethod
    def count(self) -> int:
        """Return the number of vectors currently stored."""

    @abstractmethod
    def apply(
        self,
        *,
        add_ids: List[int],
        add_vectors: Optional[np.ndarray],
        remove_ids: Sequence[int] = (),
        dedup: bool = True,
    ) -> dict:
        """Durably apply a batch of removals + additions as one atomic unit.

        This is the single write primitive. The caller computes *which* ids to
        remove and add (from its own authoritative source — for LDR, the
        encrypted DB); the store applies them and persists so that a concurrent
        writer to the same logical index cannot lose either party's changes.
        *How* that atomicity/isolation is achieved is backend-specific (a
        local-file backend takes a per-index lock and reload-merges; a
        server-backed store issues upserts/deletes) and is not part of this
        contract.

        Contract: removals are applied before additions. A "replace" is
        expressed as ``remove_ids`` + ``add_ids`` of the same id.

        ``dedup`` de-duplicates ids WITHIN one call (a repeated id in
        ``add_ids`` is written once). It does NOT mean "skip an add whose id
        is already live in the store" — an already-live ``add_id`` MUST be
        replaced, and the NEW vector must win.

        That is not a nicety. An ``add_id`` that is already live is either an
        idempotent re-add or — critically — a REUSED ``DocumentChunk.id``
        whose prior row was rolled back: SQLite recycles AUTOINCREMENT ids on
        ROLLBACK, so a failed ``index()`` commit can leave a stale orphan
        vector under an id that a later, unrelated chunk then reuses. Keeping
        the old vector makes search rank on one document's embedding and then
        rehydrate a DIFFERENT document's text by id. Nothing downstream
        detects it: a reconciler that checks ``live_ids()`` membership sees a
        healthy index, because the skipped add left the id live.

        On failure the store must not be left durably half-applied.

        Returns a stats dict (e.g. ``{"added": n, "removed": m}``). ids +
        vectors ONLY — never text (see the module "SECURITY INVARIANT").
        """

    # ------------------------------------------------------------------ #
    # Optional (local-file / reconstructable stores)
    # ------------------------------------------------------------------ #
    def reconstruct(self, id: int) -> Optional[np.ndarray]:
        """Return the stored vector for ``id`` (migration helper).

        Only meaningful when :attr:`supports_reconstruct` is True.
        """
        raise NotImplementedError(
            f"{type(self).__name__} does not support reconstruct()"
        )

    def persist(self, path: Path) -> None:
        """Durably write the store to ``path`` (local-file stores only)."""
        raise NotImplementedError(
            f"{type(self).__name__} does not support persist()"
        )

    # ------------------------------------------------------------------ #
    # Lifecycle (server-backed stores)
    # ------------------------------------------------------------------ #
    @classmethod
    def drop_collection(
        cls,
        *,
        username: str,
        index_hash: str,
        settings_snapshot: Optional[dict] = None,
    ) -> bool:
        """Permanently remove the backing collection for one RAG index.

        The deletion counterpart of :meth:`create`, so the library deletion
        cascade can dispose of vectors it cannot simply ``unlink()``. It is a
        **classmethod on purpose**: a deletion path must be able to drop a
        collection without first loading it (loading a possibly-corrupt or
        already-absent index just to delete it is both wasteful and
        failure-prone).

        Identity
        --------
        The collection is addressed by ``(username, index_hash)`` rather than
        by an instance, because those two values are the only stable handle
        available at deletion time: ``index_hash`` lives on the ``RAGIndex``
        row, and that row is deleted by the same cascade. An implementation
        MUST derive the backend collection name from these same two inputs at
        BOTH creation and deletion, or the name it computes here will not
        match the one it created and drop_collection will silently miss.

        Nothing enforces that yet: :meth:`create` does not currently receive
        ``username`` or ``index_hash`` (it takes ``dimension``/``index_type``/
        ``metric``/``normalize``/``path``/``lock``/``integrity_record``/
        ``integrity_verify``), so there is no shared derivation to reuse. The
        first server-backed provider MUST extend the creation path to accept
        both, thread them through into whatever it uses to name the remote
        collection, and use that exact derivation here too (see the follow-up
        design issue, #6769).

        **Ordering contract**: callers MUST invoke this while the ``RAGIndex``
        row still exists. Once the row is gone ``index_hash`` is unrecoverable
        and the remote collection becomes *unaddressable* -- orphaned vectors
        that no longer have a name (see issue #5765).

        Contract for implementations:

        * Idempotent -- return ``False`` (not raise) when the collection is
          already absent, ``True`` when one was actually dropped.
        * Raise on *transport / permission* failure. Returning ``False`` for an
          unreachable backend would let the caller record a successful deletion
          while the user's vectors survive on third-party infrastructure.
        * Make the failure classifiable. The cascade tells the user to retry
          only for a *transient* failure: a ``ConnectionError`` /
          ``TimeoutError`` (or subclass), or any exception carrying a boolean
          ``retryable`` attribute set to True. Everything else -- auth or
          permission errors, missing connection config, unrecognised client
          exceptions -- is reported as permanent, so translate the backend
          client's own "unreachable" / "timed out" exceptions accordingly.
        * Bound each call with a client timeout: the cascade calls this inside
          the user's database transaction.
        * Read connection config from ``settings_snapshot`` -- the deletion
          callers pass the user's snapshot. Only the exception's type name is
          surfaced to users and logs, never its message.

        Local-file stores (:attr:`is_local_file` True) do not implement this --
        their deletion path unlinks the index files instead -- so the base
        implementation raises :class:`NotImplementedError`. Server-backed
        stores MUST override it.
        """
        raise NotImplementedError(
            f"{cls.__name__} does not support drop_collection()"
        )