Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs: align SECURITY.md and DATA_HANDLING.md with the code (#1561, #1560)
AkitaOnRailsb1f622838f54
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7a7d5377..3ffee8b8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -44,6 +44,31 @@ version in [`docs/changelog/`](docs/changelog/):container (the wrapper sets `AI_MEMORY_RUN_AUTOWIRE=false` for theintercepted commands unless the operator set it). (#1545)+- Corrected four statements in `SECURITY.md` that no longer matched the+ code, plus two clarifications in the same sections (#1561). The+ `AI_MEMORY_ALLOWED_HOSTS` default is `localhost`, `127.0.0.1`, and `::1`;+ request bodies are capped at 10 MB on `/mcp`, `/hook` and `/workstream/*`,+ 32 MB on `/admin/*`, and 2 MB elsewhere; the auth ladder is anonymous,+ root bearer, proxy-asserted user, and `aim_` database-user tokens (the+ server itself never validates OIDC tokens, and the doc no longer says it+ does); and the "no per-user or per-project ACLs" claim became the current+ truth about per-project access modes and grants, including that+ proxy-asserted identities are not subject to them. The bearer-check note+ now lists the routes that are public by design (`/healthz`, `/auth/login`,+ `/auth/recovery`, `/favicon.ico`, custom `/web` static assets).+- Completed the egress inventory in `DATA_HANDLING.md` and documented the+ default first-start behavior it omitted (#1560). A default install now+ says plainly that the local embedder downloads its three model files+ (~87 MB, sha256-pinned) from `huggingface.co` in the background on first+ start, that nothing stored is sent, and how to avoid it+ (`embedding_provider = "none"` or offline pre-seeding). The export table+ gained the rows it was missing: the `llm_provider` pipeline (including+ the `codex`, `copilot` and `cursor` CLI-backed providers), the `copilot`+ embedding provider, and `[[admission_webhooks]]`, plus notes on+ `ai-memory upgrade`'s GitHub release query, the OAuth/OIDC login flows,+ and remote `server_url` topologies. `docs/airgapped-install.md`'s+ runtime summary was corrected to match.+## [3.2.0] - 2026-10-21### Addeddiff --git a/DATA_HANDLING.md b/DATA_HANDLING.mdindex 1413f3e8..3a82e25b 100644--- a/DATA_HANDLING.md+++ b/DATA_HANDLING.md@@ -4,72 +4,122 @@ This page is for reviewers who need one place to answer "what data doesai-memory collect, where does it live, and what (if anything) leaves themachine", usually for a security, legal, or data-protection review ahead ofinternal approval. It restates facts from [`SECURITY.md`](SECURITY.md), cited-by section below, and makes no new claims; if the two ever disagree,-`SECURITY.md` is authoritative.+by section below, alongside the operational docs it names; if the two ever+disagree, `SECURITY.md` is authoritative.-## Default: fully local, no telemetry+## Default: local, no telemetry, one model downloadai-memory does not phone home. There is no analytics, crash reporting, or-usage telemetry built into the binary. Everything it stores (wiki pages and-the SQLite database) lives under a single, operator-controlled data-directory on the local filesystem (`SECURITY.md`, "Local data-confidentiality"). No data leaves the host unless one of the three opt-in-paths below is explicitly enabled. The **embedding provider** is the one-most likely to matter to a reviewer, since it decides whether stored content-reaches a cloud API at all.--The out-of-the-box default is local-only: `config.default.toml` configures-in-process local embeddings (all-MiniLM-L6-v2, 384-dim) with no API key and-no data egress. See [`docs/local-embeddings.md`](docs/local-embeddings.md).+usage telemetry built into the binary, and no background update check.+Everything it stores (wiki pages and the SQLite database) lives under a+single, operator-controlled data directory on the local filesystem+(`SECURITY.md`, "Local data confidentiality"). No stored data leaves the+host unless one of the opt-in paths below is explicitly enabled. The+**embedding provider** is the one most likely to matter to a reviewer,+since it decides whether stored content reaches a cloud API at all.++The out-of-the-box default is local embeddings: the config template leaves+`embedding_provider` unset, which selects in-process `local` embeddings+(all-MiniLM-L6-v2, 384-dim) with no API key and no cloud calls. That+default carries one network behavior: on the first start, when the model+files are not already present, the server downloads three files (~87 MB+total, sha256-pinned in source) from `huggingface.co` in the background,+and hybrid search enables on the next restart. Nothing stored is sent; the+transfer is a download, so your host's address and the timing of the+request are visible to Hugging Face and its CDN, and no content goes with+them. To avoid the download entirely, set `embedding_provider = "none"`,+or pre-seed the files offline as+[`docs/airgapped-install.md`](docs/airgapped-install.md) and+[`docs/local-embeddings.md`](docs/local-embeddings.md) describe.There is no encryption at rest in v1; ai-memory relies on OS filesystempermissions (owner-only directories/files on Unix, ACLs on Windows;`SECURITY.md`, same section).-## The three opt-in paths that send data externally+## The opt-in paths that send data externally-All three are off by default (local embeddings need no key or network call)-and each requires a deliberate config change to turn on.+Each is off by default and requires a deliberate config change or command+to turn on.| path | what leaves the host ||---|---|-| `embedding_provider = openai\|voyage\|google\|openai-compat` | the text of every stored page |+| `embedding_provider = openai\|voyage\|google\|openai-compat\|copilot` | the text of every stored page |+| `llm_provider` (and `llm_fallbacks`), off by default | session observations and page text on consolidation, lint, bootstrap, dream, profile and auto-improve runs || `capture_assistant` (double opt-in) | the assistant's final-turn text || `AI_MEMORY_RERANKER=llm` | each live query + bounded page titles/snippets |+| `[[admission_webhooks]]` | every write's page path, frontmatter and body, POSTed to the URL you configured |1. **Embedding provider.** This is the broadest path by far, and the onemost relevant to a reviewer's usual question ("does our sourcematerial reach a cloud provider?"). `OpenAiEmbedder`, `VoyageEmbedder`,- and `GoogleEmbedder` each call their provider's embedding endpoint for- every page that needs a vector. This isn't limited to new writes:- switching a running install from `local` to a cloud provider triggers a- backfill of the **existing corpus** (`POST /admin/embed`, `ai-memory embed- --force`), so changing this setting later can retroactively send content- that was previously local-only. Leave `embedding_provider` on `local` (the- default) to keep this off; see- [`docs/local-embeddings.md`](docs/local-embeddings.md).-2. **Assistant/Stop capture** (`SECURITY.md`, "Assistant/Stop capture is- opt-in and sanitized"). Persisting the coding assistant's final-turn text- requires a double opt-in: `capture_assistant` on the server *and*- `install-hooks --capture-assistant` on the client. Once enabled, captured- text rides in the session's automatic handoff to the next session and- flows into consolidation/reviewer prompts. Only if you have separately- configured a cloud LLM provider is it sent to that provider. The- flag is global to the install; there is no per-project exclusion once it's- on.-3. **LLM-based search reranking** (`SECURITY.md`, "Search reranking is an+ `GoogleEmbedder`, and `CopilotEmbedder` each call their provider's+ embedding endpoint for every page that needs a vector. This isn't limited+ to new writes: switching a running install from `local` to a cloud+ provider triggers a backfill of the **existing corpus** (`POST+ /admin/embed`, `ai-memory embed --force`), so changing this setting+ later can retroactively send content that was previously local-only.+ Leave `embedding_provider` on `local` (the default) to keep this off;+ see [`docs/local-embeddings.md`](docs/local-embeddings.md).+ `openai-compat` points at a self-hosted engine, so the destination is+ your own server.+2. **LLM provider.** Setting `llm_provider` (or `llm_fallbacks`) turns on+ the model-assisted pipeline: consolidation, lint, bootstrap, dream, the+ cross-project profile digest, and the auto-improvement loop each send+ session observations and page text to the configured provider and+ receive generated pages and proposals back. The `codex`, `copilot`, and+ `cursor` provider options spawn the vendor's own CLI, which talks to+ that vendor's cloud under your existing login. With no provider+ configured, every one of these features either runs rule-based locally+ or stays off; see `docs/ARCHITECTURE.md` ("Zero-LLM default path").+3. **Assistant/Stop capture** (`SECURITY.md`, "Assistant/Stop capture is+ opt-in and sanitized"). Persisting the coding assistant's final-turn+ text requires a double opt-in: `capture_assistant` on the server *and*+ `install-hooks --capture-assistant` on the client. Once enabled,+ captured text rides in the session's automatic handoff to the next+ session and flows into consolidation/reviewer prompts, so it reaches a+ cloud provider only through path 2. The flag is global to the install;+ there is no per-project exclusion once it's on.+4. **LLM-based search reranking** (`SECURITY.md`, "Search reranking is anoutbound-data opt-in"). Setting `AI_MEMORY_RERANKER=llm` sends each livequery plus bounded page titles/snippets to the configured LLM provider.Leave this unset, or point it at a local provider, to keep queries andrecalled snippets on-host.--Paths 2 and 3 pass through a sanitizer that strips obvious credentials before-anything is transmitted. The sanitizer is documented as best-effort, not a-guarantee; see the "Stored-content prompt injection" and reranking notes in-`SECURITY.md` for the exact caveats. The embedding path (1) sends full page-text to the provider's embedding endpoint and is not sanitized the same way,-so it is the path to leave on `local` when stored content must not leave the-server.+5. **Admission webhooks.** Every wiki write (create, update, delete, and+ the handoff lifecycle events) runs the configured webhook chain: each+ `[[admission_webhooks]]` entry receives the page's path, frontmatter,+ and full body as a JSON POST, and may mutate or reject the write. The+ request carries only an `X-Memory-Op` header naming the operation; no+ credentials are attached. Deletes send the path with no body. The URL+ is one you configured, so this sends content to infrastructure you+ chose, over whatever TLS it has+ ([`docs/admission-webhooks.md`](docs/admission-webhooks.md)).++Paths 3 and 4 pass through a sanitizer that strips obvious credentials+before anything is transmitted. The sanitizer is documented as+best-effort, not a guarantee; see the "Stored-content prompt injection"+and reranking notes in `SECURITY.md` for the exact caveats. The embedding+path (1) sends full page text to the provider's embedding endpoint and is+not sanitized the same way, and neither are the LLM provider (2) and+admission webhook (5) payloads, so they are the paths to leave off when+stored content must not leave the server.++Two more network behaviors sit outside the table because they carry no+stored content:++- `ai-memory upgrade`, when you run it, queries `api.github.com` and+ downloads from `github.com`. Nothing checks for updates in the+ background.+- `auth login oidc-device` (and the `openai-oauth`, `anthropic-oauth`,+ `copilot` logins) talks to the issuer or vendor token endpoints during+ login and token refresh. These calls carry credentials, never memory+ content.++Finally, a topology choice rather than a config key: with a remote+`server_url`, the lifecycle hooks and the external relay ship every+captured event to the host you point them at+(`docs/external-lifecycle.md`). The data handling is everything on this+page, but the host, and whoever administers it, are now outside your+machine.## Is this "personal data" under GDPR?@@ -83,8 +133,8 @@ to process personal data as such. In practice this means:commit message with a name, or free text a developer typed), that data isstored and retained exactly like any other memory content, under your ownorganization's control on infrastructure you run.-- ai-memory does not transmit that content anywhere on its own; the three- opt-in paths above are the only export mechanisms, and all are under your+- ai-memory does not transmit that content anywhere on its own; the opt-in+ paths above are the only export mechanisms, and all are under yourcontrol.- Treat this the same way you'd treat any local database your teamoperates: your organization is the data controller and decides retention,diff --git a/SECURITY.md b/SECURITY.mdindex 038401b4..28bfb1cf 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -17,8 +17,10 @@ maintainer will follow up; do not put vulnerability details in that issue.## Threat modelai-memory is a **single-tenant workstation/homelab service**. It supports-multiple attributed users, but every authenticated user belongs to the same-trust domain and can read the same project memory. The following describes+multiple attributed users. Projects are `open` by default, so every+authenticated user can read the same project memory; a root operator can+mark a project `restricted` so only root, its creator, and granted database+users reach it (see "Per-project isolation" below). The following describeswhat the project is and is not designed to defend against.### In scope@@ -36,8 +38,11 @@ what the project is and is not designed to defend against.- **Network exposure when binding to non-loopback addresses.** If you run`ai-memory serve --bind 0.0.0.0:…` you are exposing the MCP and adminroutes to your local network. Protect this with:- - `AI_MEMORY_AUTH_TOKEN` / `ai-memory generate-auth-token` (bearer token- checked on every request).+ - `AI_MEMORY_AUTH_TOKEN` / `ai-memory generate-auth-token` (a bearer+ token checked on every authenticated surface). `/healthz`, `/auth/login`,+ `/auth/recovery`, `/favicon.ico`, and the custom `/web` static assets are+ public by design; the builtin wiki browser and `/api/v1` stay behind+ auth.- Firewall rules or a reverse proxy with TLS.The server fails closed before serving unauthenticated non-loopback HTTP.@@ -62,17 +67,28 @@ what the project is and is not designed to defend against.access; browsers intentionally withhold Secure cookies over HTTP.- **Host-header DNS rebinding.** The HTTP server enforces an- `AI_MEMORY_ALLOWED_HOSTS` allowlist (defaulting to `127.0.0.1` and- `localhost`). Requests with a `Host` header not in the list are rejected+ `AI_MEMORY_ALLOWED_HOSTS` allowlist (defaulting to `localhost`, `127.0.0.1`,+ and `::1`). Requests with a `Host` header not in the list are rejectedwith 403.-- **Request body size.** Inbound HTTP bodies are capped at 10 MB to prevent- trivial memory exhaustion.--- **Authentication and administrative authorization.** A static root bearer- token, database-user tokens, and optional OIDC hook-edge tokens form the- documented auth ladder. The first database user makes every `/admin/*`- route root-only; DB-user tokens provide attribution but never admin access.+- **Request body size.** Inbound HTTP bodies are capped to prevent trivial+ memory exhaustion: 10 MB on `/mcp`, `/hook`, and `/workstream/*`, and+ 32 MB on the `/admin/*` routes, whose `/admin/bootstrap` can carry a large+ JSON array of sources. The remaining routes (web, `/api/v1`, `/auth/*`)+ use axum's 2 MB default.++- **Authentication and administrative authorization.** The documented auth+ ladder (`docs/users.md`) is: anonymous when no root bearer is configured,+ the static root bearer, a proxy-asserted user (a distinct+ `[auth].actor_proxy_bearer_token` plus trusted `X-Memory-Actor-*` headers,+ which is root only on an exact `root_issuer` + `root_subject` match), and+ database-user `aim_` tokens, which attribute writes but never grant admin+ access; an unknown bearer gets 401. The server itself does not validate+ OIDC tokens. `ai-memory auth login oidc-device` stores a per-developer+ token for an external OIDC-aware gateway or bridge in front of the server,+ which must translate it into one of the rungs above (`docs/sso.md`). The+ first database user (or a configured proxy bearer) makes every `/admin/*`+ route root-only.- **Per-project isolation.** Wiki files and SQLite rows are namespaced by`(workspace_id, project_id)`. A purge operation for project A cannot@@ -87,6 +103,17 @@ what the project is and is not designed to defend against.claim descriptor-relative protection against a concurrently malicious processrunning as the same operating-system user.+ On a multi-user server, per-project access modes gate entry on top of that+ boundary: every project is `open` until an operator restricts it+ (`ai-memory project access --mode restricted`), after which only root, the+ project's creator, and users holding a grant (`ai-memory user grant`) may+ reach it, and refusals are explicit 403s rather than empty results. Grants+ apply to database users. Root and single-operator installs are never+ gated, and a user asserted by the trusted-proxy bearer is not subject to+ per-project grants: that identity maps to no database `UserId` to key a+ grant on, so in a trusted-proxy deployment the proxy itself is the+ authorization boundary (`docs/users.md`).+- **Entity text remains bounded local data.** Consolidator output andhand-edited `entities:` frontmatter cross the same normalization boundary:at most 10 names per page, 64 characters per name, with control characters@@ -147,8 +174,11 @@ what the project is and is not designed to defend against.### Out of scope for v1-- **Tenant isolation and per-user ACLs.** Database users provide attribution,- not private memory. There are no per-user or per-project ACLs; run separate+- **Private memory and per-page ACLs.** Database users provide attribution,+ not private memory. Per-project access modes and grants decide which+ projects a database user may reach, but inside a project every reader sees+ every page: there is no per-page RBAC, no private per-user pages, and no+ per-project restriction applied to proxy-asserted identities. Run separateservers/data directories for users who must not see one another's data.- **Encryption at rest.** The data directory is a plain filesystem tree.- **Remote sync security.** If you push the wiki git repository to a remote,diff --git a/docs/airgapped-install.md b/docs/airgapped-install.mdindex 31db7d05..96954def 100644--- a/docs/airgapped-install.md+++ b/docs/airgapped-install.md@@ -28,8 +28,10 @@ into the air-gapped environment.Covered in full in [`DATA_HANDLING.md`](../DATA_HANDLING.md), summarizedhere: ai-memory has no telemetry, analytics, or phone-home behavior. Theserver, CLI, and lifecycle hooks run entirely against the local data-directory. The only things that make a network call are things you-explicitly configure:+directory. The things that make a network call are the ones you explicitly+configure, plus one default: the local embedding model download described+below, which fires on the first start when `embedding_provider` is unset,+unless the files are pre-seeded.- **A cloud LLM/embedding provider**, if you want AI-assisted consolidation,search, or provider-based embeddings. Skip this and use the `local`