Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust
5.9 KB
Raw Blame

Local embeddings

2.0 item 5. AI_MEMORY_EMBEDDING_PROVIDER=local runs sentence embeddings in-process, with no API key, external server, or GPU required. It uses pure-Rust BERT inference (candle) with all-MiniLM-L6-v2 (384-dim), the sentence-transformers model the comparable memory servers ship by default.

As of 2.0 this is the default: an install with no embedding_provider configured gets local embeddings automatically, on a best-effort basis. The model downloads in the background on the first start (hybrid search enables on the next restart) and existing pages are backfilled automatically. A host that cannot fetch the model (or a slim build without the feature) keeps the FTS-only behaviour with a warning. Opt out with embedding_provider = "none"; an explicitly configured provider is never overridden.

# config.toml — all optional as of 2.0
embedding_provider = "local"     # explicit: hard-fails if unavailable
# embedding_provider = "none"    # opt out of vectors entirely

Why: semantic recall without data egress

Search is hybrid: FTS5 (lexical), entities, graph, and, when an embedder is configured, vectors. The vector stream lets "how do we deploy" find the page that says "release procedure" (paraphrase recall, with no shared keywords). Before this provider, enabling it cost one of two things:

  • an API key (OpenAI / Voyage / Google): per-call spend, and every page body and every search query leaves your machine, which is a lot to ask of a system that records everything you do;
  • a self-hosted engine (Ollama / LM Studio via openai-compat): keyless, but another server to run, warm, and keep on the same network as ai-memory.

local removes both. Use it when any of these describe you:

  • you run the zero-LLM path and want better recall without handing a provider your memory;
  • the install is offline or air-gapped (drop the model files in manually, as described below);
  • a homelab/team server where "one binary, one volume" is the whole operational story and adding an Ollama sidecar just for embeddings is not worth it;
  • you want reproducible retrieval: same model files (checksum-pinned), same vectors, forever, with no provider-side model deprecations.

Stick with a hosted or self-hosted provider when you already run one happily, want a larger/multilingual model, or want embedding compute off the memory server's CPU.

Why not ONNX?

ONNX is a model format plus a native runtime (onnxruntime). It is one way to run local inference, and the one the comparable servers use. We ship the same model through candle (pure Rust) instead. The capability is identical, and there is no native C++ library to build, license, and debug across every release target (Windows, macOS, the sandboxed nix build). If a future model requires an ONNX-only runtime, the Embedder trait is where it would slot in.

The model files

The binary does not bundle the model (~87 MB). On the first start with local configured, the server fetches three files into <data_dir>/models/all-MiniLM-L6-v2/. Each is verified against a sha256 pinned in the source, so a drifted or tampered upstream file fails loudly instead of silently changing every vector:

file sha256 (pinned 2026-09-01)
model.safetensors 53aa5117…
tokenizer.json be50c362…
config.json 953f9c0d…

Offline installs: download the three files from https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/ on any machine and drop them into the directory above; the loader verifies the same checksums and never touches the network. The model is Apache-2.0.

Coexistence and migration

Nothing is forced. (provider, model, dim) is stored on every embedding row, and hybrid search ignores vectors whose triple does not match the configured embedder. Local vectors therefore sit beside any provider vectors you already have, and switching back is a config change. ai-memory embed --force re-embeds a project under the current provider when you want one consistent set.

Existing installs keep their configured provider; local is opt-in. Fresh-install defaults are decided by benchmark numbers. docs/benchmarks/ has the zero-LLM vs local-embeddings LongMemEval rows, reproducible via:

cargo run --release -p ai-memory-eval -- retrieval --embeddings local

Operational notes

  • Inference is CPU, off the async runtime (spawn_blocking); first-load reads ~87 MB into memory once per server process.
  • Local inference runs through a small process-wide scheduler sized from the CPU: half the available parallelism, clamped to between 2 and 6 slots (local_capacity in ai-memory-llm). all-MiniLM inference already parallelizes across cores inside one embed, so this bounds concurrent calls while mild oversubscription keeps the CPU busy. Document embeddings may use all but one slot, so page writes and administrative backfill embed several documents at once instead of serializing, and one slot stays reserved for search queries. A query that arrives while every slot is busy waits up to 250 ms for one before falling back to lexical/entity/graph retrieval, so a query that merely races a document embed keeps its vector stream; only sustained saturation degrades. Hosted embedding providers are not subject to this local scheduler.
  • The local-embeddings cargo feature (default on) carries the ML dependency tree; a slim build can disable it and keep every other provider.
  • Tokenization truncates at 512 tokens, the model's positional limit. Longer page bodies contribute only their head, the same contract as the hosted embedders.

The opt-in contention probe compares the raw local embedder with that capacity scheduler. It needs the downloaded model and is not part of the routine test tier:

AI_MEMORY_TEST_MODELS_DIR=$PWD/evals/models \
AI_MEMORY_EMBED_BENCH_WORKERS=8 \
cargo test -p ai-memory-consolidate \
  local_query_latency_under_document_load -- --ignored --nocapture