Skip to content

armgit / ente

Public Imported from GitHub
0

πŸ’š End-to-end encrypted cloud for everything.

vecdb: drop in-repo design docs

laurenspriemc671d595e57b
diff --git a/rust/crates/ml/CONTEXT.md b/rust/crates/ml/CONTEXT.md
deleted file mode 100644
index 5adbfb55f5..0000000000
--- a/rust/crates/ml/CONTEXT.md
+++ /dev/null
@@ -1,23 +0,0 @@
-# ente-ml β€” Context
-
-Glossary of terms for the ML crate's domain. Definitions describe meaning, not implementation.
-
-## Vector DB (vecdb)
-
-- **VecDb** β€” A single persistent collection of embeddings sharing one dimensionality, addressed by a single filesystem path. The app runs many independent VecDb instances (one per embedding space). Intended to become the sole store for embeddings, not a derived cache.
-- **Key** β€” The caller-chosen string identity of an embedding within one VecDb (non-empty UTF-8, ≀ 256 bytes): a file ID, face ID, cluster ID, etc., used directly β€” no integer mapping layer. Adding an existing key replaces its embedding (upsert semantics).
-- **Vector Log** β€” The durable source of truth of a VecDb: the append-only record of every embedding entrusted to it. An acknowledged write survives process kill and power loss.
-- **Graph Snapshot** β€” A persisted copy of the search index (HNSW graph). Purely derived state: losing or corrupting it costs rebuild time, never data.
-- **Tombstone** β€” A logged record marking a key as removed. Tombstoned entries are dead but still occupy log and graph space until compaction.
-- **Compaction** β€” Rewriting the vector log to contain only live entries and rebuilding the graph, triggered when dead entries exceed a threshold.
-- **Writer** β€” The single live instance per index path within a process, holding the OS lock that keeps other processes out. Every in-process handle aliases it, so all handles see every write immediately.
-- **Handle** β€” A caller's reference to a VecDb. Handles to one path share the process's one live instance (writable, or a read-only view of it); a read-only open with no live writer in the process gets its own point-in-time copy instead.
-- **Flush** β€” Explicitly persisting the current graph snapshot so the next open replays no records into the graph; every open still scans the log to rebuild the arena.
-- **Exact search** β€” Brute-force scan over all live embeddings; results are ground truth.
-- **Approximate search** β€” Index-backed (HNSW) search; may miss neighbours in exchange for speed.
-- **Distance** β€” 1 βˆ’ inner product of L2-normalized embeddings, in [0, 2]; lower is closer. The only metric vecdb speaks.
-- **Embedding space** β€” A model-specific vector space (e.g. CLIP image 512-d, face 192-d, pet face 128-d). Embeddings from different spaces are never comparable, hence one VecDb per space.
-
-## Legacy
-
-- **VectorDB (usearch)** β€” The existing usearch-backed engine, kept intact while vecdb is validated by internal users. Same conceptual model, but persists by rewriting the entire index per mutation.
diff --git a/rust/crates/ml/docs/adr/0001-vecdb-durable-log-plus-derived-graph.md b/rust/crates/ml/docs/adr/0001-vecdb-durable-log-plus-derived-graph.md
deleted file mode 100644
index 54b1c53563..0000000000
--- a/rust/crates/ml/docs/adr/0001-vecdb-durable-log-plus-derived-graph.md
+++ /dev/null
@@ -1,23 +0,0 @@
-# vecdb: durable append-only vector log + derived HNSW snapshot
-
-We replaced usearch (C++ under the hood, broken on the napi desktop toolchain, rewrites the entire index file on every mutation) with a pure-Rust engine tailored to Ente's on-device scale of 1k–200k embeddings per index. The persistence architecture splits durability from search: an append-only log of (key, vector) records is the sole source of truth β€” one checksummed, fsynced append per write operation gives sqlite-grade durability β€” while the HNSW graph is derived state, persisted as a sidecar snapshot that can always be rebuilt from the log. Losing the snapshot costs rebuild time, never data. The DB is designed to be trustworthy as the *sole* store for embeddings, so the duplicate embedding blobs in sqlite can eventually be dropped.
-
-## Considered options
-
-- **Incremental on-disk graph mutation (true sqlite-style paging/journaling)** β€” rejected: months of complexity and corruption risk for no functional gain at our scale.
-- **Log only, rebuild graph on every open** β€” rejected: tens of seconds of rebuild at 200kΓ—512-d on every app start.
-- **Converting existing usearch files** β€” rejected: would mean maintaining a parser for a foreign versioned binary format for one-off use; instead indexes are refilled from sqlite via the existing migration machinery.
-
-## Consequences
-
-- One caller-visible path maps to three files: `<path>` (log), `<path>.graph` (snapshot), and `<path>.lock` (writer-lock sidecar); the Rust layer owns all three, consumers never see the sidecars.
-- Snapshots are written lazily (mutation threshold + quiet-gap debounce + hard cap), on compaction, on explicit flush, and on open after tail replay β€” so a killed process pays at most one bounded replay at next open.
-- Deletes are tombstones; dead entries are compacted away (log rewrite + graph rebuild) past a ~10% dead ratio with a small absolute floor.
-- Exactly one writer instance per index file, enforced by an OS advisory lock; read-only opens hold no lock and no fds, and see a point-in-time view β€” which is what makes rename-based compaction safe on Windows.
-- Single metric (inner product on L2-normalized vectors, distance = 1 βˆ’ dot) and single scalar (f32) are hardcoded; the file header carries scalar/metric tags and the distance kernel sits behind one internal seam so smaller scalars (f16/i8) can be added without a format break.
-- Keys are strings (non-empty UTF-8, ≀ 256 bytes), not integers: file/face/cluster IDs are used directly, which clears the way to retire the four AUTOINCREMENT key-mapping tables the app kept in sqlite solely for usearch's u64 keys once vecdb replaces usearch. Internally keys are interned to u32 slots; the graph, arena, and log replay speak slots.
-- Vectors live on disk as raw little-endian f32 payloads in the framed log records, and in memory in a chunked, 32-byte-aligned arena (chunks of SIMD lanes; free-list slot reuse; growth never reallocates existing chunks). Memory-mapping the log was rejected: it breaks wasm compatibility, fights rename-based compaction on Windows, and framed records give no alignment guarantee.
-- vecdb is layered as an IO-free core (graph, arena, distance kernel β€” wasm32-compatible) beneath a native-only storage layer (fs, fsync, locking). The writer lock is std file locking (`File::try_lock`) on the `<path>.lock` sidecar β€” no `libc`, no `windows-sys` β€” and is held continuously across compaction, so there is never an unlock window; locking the sidecar rather than the log keeps Windows whole-file lock semantics away from readers and log renames. vecdb has zero direct dependencies: locking is std file locking, CRC32 is hand-rolled in-module, and the distance kernel is pure std.
-- SIMD comes from auto-vectorization by construction, not a SIMD crate: the dot product keeps 8 independent partial accumulators over 32-byte-aligned 8-float lanes and reduces them in a fixed sequential order. This loop shape is load-bearing β€” a single running sum cannot legally auto-vectorize because float addition is non-associative β€” and an A/B against an explicit-SIMD (`wide`) kernel measured it within ~10% on the exact-scan path on baseline NEON, parity elsewhere, with bit-identical results (same reduction order, pinned by the kernel tests). Do not collapse the accumulators.
-- usearch stays in the tree, feature-gated, while vecdb runs as a parallel engine for internal users; removal is a separate later decision.
-- Derived-state failures degrade instead of failing writes: snapshot and compaction errors are logged as warnings and never fail an acknowledged write, open never fails for snapshot reasons, and only an explicit `flush()` surfaces snapshot errors to the caller.
diff --git a/rust/crates/ml/docs/adr/0002-vecdb-process-shared-handles.md b/rust/crates/ml/docs/adr/0002-vecdb-process-shared-handles.md
deleted file mode 100644
index c69de9def9..0000000000
--- a/rust/crates/ml/docs/adr/0002-vecdb-process-shared-handles.md
+++ /dev/null
@@ -1,20 +0,0 @@
-# vecdb: process-shared handles via a per-path registry
-
-The app holds many handles to one index at once β€” every Dart isolate opens its own, all inside a single process. Under the original design that meant a second in-process writer open failed `Locked`, and read-only opens were stale point-in-time copies, each holding a private copy of every vector (~230 MB at 100kΓ—512-d per copy). This mirrors the handle-staleness bugs the app fought with usearch. vecdb now keeps a process-global registry (canonical path β†’ weak reference) guaranteeing at most one live instance per index path per process; `open()` either creates that instance or returns another handle aliasing it, the sqlite-shared-cache model. Handles use interior locking instead of `&mut`: a writer mutex (log fd, OS lock, snapshot/compaction policy) serializes all mutation, and a read-write lock over the search state (arena + graph) lets searches run concurrently; the lock order is always writer mutex before state lock.
-
-## Considered options
-
-- **Keep per-handle instances, add cross-handle invalidation** β€” rejected: reintroduces the staleness window it is meant to close, and still duplicates vectors per handle.
-- **Force the app to funnel all access through one isolate** β€” rejected: pushes the hardest part (lifecycle and routing) onto every consumer, which is how the usearch bugs happened.
-
-## Consequences
-
-- Handle staleness is structurally impossible in-process: every handle reads the same arena and graph, so a write is visible to all handles the moment it lands.
-- One copy of the vectors per process, regardless of handle count.
-- A second in-process `open()` joins the live instance instead of failing `Locked`; the OS advisory lock now guards purely against other processes and is held for the instance's lifetime.
-- Concurrent opens serialize per path, not globally: simultaneous opens of one path coalesce onto a single load (the losers wait and then join it), while opens of different paths load in parallel β€” the app opening its dozen indexes at startup pays only the slowest load, not the sum.
-- Bulk writes become visible per vector, not batch-atomically: the batch is durable after its single fsync, but a concurrent search may observe it partially applied (each vector individually consistent).
-- `delete()` closes the shared instance: surviving handles get `Closed` on every subsequent operation, and reopening the path creates a fresh empty db. `reset()` keeps all handles live on the emptied instance.
-- `open_read_only()` aliases the live writer instance when one exists (a live view that rejects mutation); with no live writer it stays a standalone point-in-time load, unregistered, for the cross-process case.
-- Compaction stages and rebuilds off to the side and swaps the new state in under one short write lock, so searches keep answering from the old state for the whole rebuild.
-- The last handle dropped drops the instance: fds close, the OS lock releases, and the registry entry is removed.