Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(embeddings): the use case - semantic recall without data egress

AkitaOnRails10a4ce5da043
diff --git a/docs/ROADMAP-2.0.md b/docs/ROADMAP-2.0.md
index 3c0a919a..83cf8821 100644
--- a/docs/ROADMAP-2.0.md
+++ b/docs/ROADMAP-2.0.md
@@ -134,7 +134,7 @@ worth having. Builds on item 3's schema work.
- Migration: additive columns; backfill `valid_from` from the page
version's `created_at` - a one-shot data migration inside refinery.
-## Item 5 - Local embeddings (ONNX, all-MiniLM class)
+## Item 5 - Local embeddings (all-MiniLM class; shipped via candle, not ONNX)
*Why:* zero-config hybrid retrieval with no provider; `models/` has been
reserved for this since M9.5. Competitors ship it by default.
@@ -151,7 +151,10 @@ reserved for this since M9.5. Competitors ship it by default.
configured provider either way.
- Watch: `ort`/onnxruntime licensing + build weight on all release
targets (incl. Windows); a `local-embeddings` cargo feature if the
- dependency is heavy.
+ dependency is heavy. **As built: this watch item decided the
+ implementation — candle (pure Rust) instead of ort, no native
+ runtime, feature `local-embeddings` (default on). Same model, same
+ use case; see `docs/local-embeddings.md`.**
## Item 6 - Cross-session abstraction ("Experience" stage)
diff --git a/docs/local-embeddings.md b/docs/local-embeddings.md
index 55e6f97f..ae9e073a 100644
--- a/docs/local-embeddings.md
+++ b/docs/local-embeddings.md
@@ -11,6 +11,47 @@ comparable memory servers ship by default.
embedding_provider = "local" # model/dim default correctly
```
+## Why: semantic recall without data egress
+
+Search is hybrid: FTS5 (lexical), entities, graph, and — when an
+embedder is configured — vectors. The vector stream is what lets
+*"how do we deploy"* find the page that says *"release procedure"*:
+paraphrase recall, no shared keywords required. Before this provider,
+enabling it cost one of two things:
+
+- **an API key** (OpenAI / Voyage / Google): per-call spend, and every
+ page body and every search query leaves your machine. For a system
+ whose job is recording everything you do, that is not a small ask;
+- **a self-hosted engine** (Ollama / LM Studio via `openai-compat`):
+ keyless, but another server to run, warm, and keep on the same
+ network as ai-memory.
+
+`local` removes both. Use it when any of these describe you:
+
+- you run the zero-LLM path and want better recall without handing a
+ provider your memory;
+- the install is offline or air-gapped (drop the model files in
+ manually — see below);
+- a homelab/team server where "one binary, one volume" is the whole
+ operational story and adding an Ollama sidecar just for embeddings
+ is not worth it;
+- you want reproducible retrieval: same model files (checksum-pinned),
+ same vectors, forever — no provider-side model deprecations.
+
+Stick with a hosted or self-hosted provider when you already run one
+happily, want a larger/multilingual model, or want embedding compute
+off the memory server's CPU.
+
+### Why not ONNX?
+
+ONNX is a model format plus a native runtime (`onnxruntime`) — one
+*mechanism* for local inference, and the one the comparable servers
+use. We ship the same model through **candle** (pure Rust) instead:
+identical capability, but no native C++ library to build, license, and
+debug across every release target (Windows, macOS, the sandboxed nix
+build). If a future model genuinely requires an ONNX-only runtime, the
+`Embedder` trait is where it would slot in.
+
## The model files
The binary does not bundle the model (~87 MB). On the first start with