Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust
33.7 KB
Raw Blame

LLM Providers

Provider configuration for consolidation and embeddings. Moved verbatim from the README front page.

ai-memory runs without an LLM: hooks still capture sessions, search uses FTS5 + declared entities + graph neighbors, and summaries fall back to rule-based output. Add an LLM provider when you want LLM consolidation (on PreCompact / PostCompaction, on demand via memory_consolidate, or opt-in at session end with AI_MEMORY_CONSOLIDATE_ON_SESSION_END), richer linting, and bootstrap. Substantive session ends always write a rule-based summary page + handoff either way. A session containing only SessionStart / SessionEnd boundaries is closed without a page, handoff, or provider job. When that empty session had accepted startup context, its session-bound handoff is returned to the open pool for the next receiver instead of being lost. PreCompact / PostCompaction write a rule-based checkpoint on the hook request. Provider work for those checkpoints, and for opt-in SessionEnd, is durably queued after those deterministic writes and handled by one bounded server worker, so hook drain latency does not cancel it. The worker starts whenever a consolidator is configured; SessionEnd enqueue stays behind the opt-in flag. A checkpoint rewrite still queued when the session ends runs once anyway, even with the flag off. One already running when the session ends finishes after the SessionEnd summary and replaces it, so with the flag off the latest page can leave out the end of the session; the summary stays reachable as the superseded version. Every queued job runs as the session owner recorded at session start, not anonymously. Failed jobs retry with backoff and survive a server restart. A resumed native session is ended again only after its observation generation advances; the persisted generation watermark makes duplicate SessionEnd delivery and system clock skew converge without repeated provider work. The end watermark and automatic handoff commit atomically, and an interrupted keyed replay finishes the wiki commit, queue insert, and key completion without duplicating that handoff. On the next SessionStart, the newest cwd-eligible automatic handoff wins; accepting it expires older eligible automatic handoffs without consuming manual or sibling-directory work. A new automatic handoff also expires prior open automatic handoffs from its exact cwd, so repeated SessionEnds cannot accumulate there before a receiver starts.

To keep consolidation style project-specific, write _prompts/consolidation.md in that project's wiki. Its body can express preferences such as "prefer Portuguese titles" or "omit routine CI noise". Automatic, single-page, and multi-page consolidation use the page; a manual memory_consolidate call can pass instructions to override it once. ai-memory sanitizes and caps the value at 2,000 characters, JSON-encodes it in the user message, and treats it as untrusted advisory data. It cannot supply facts, request tool use or disclosure, or override the consolidation schema and faithfulness rules. TTL-expired preference pages are ignored. With no active page or argument, no preference block is appended.

Recommended defaults:

Provider Default Use when
anthropic claude-haiku-4-5 Best default for consolidation quality and rule classification.
anthropic-oauth claude-sonnet-4-6 Use a Claude Pro/Max subscription via claude setup-token, no API key. Currently refused by Anthropic for third-party apps (usually 429); see below.
openai gpt-5.4-mini Cheaper and faster hosted option.
openai-oauth gpt-5.5 ChatGPT Pro/Plus/Codex backend via ai-memory auth login openai-oauth; no Platform API key.
codex gpt-5.6-luna Reuse the Codex CLI-owned auth.json; access-token refresh remains owned by codex app-server.
cursor cursor-grok-4.6-high Cursor subscription via the logged-in agent CLI. Runs agent --print --mode ask (read-only). Does not pass --yolo. The default id is the one agent --list-models labels "Grok 4.6".
copilot gpt-5.5 GitHub Copilot Chat backend via ai-memory auth login copilot or COPILOT_GITHUB_TOKEN; requires a Copilot subscription.
gemini gemini-3.5-flash Google-hosted option with a generous free tier.
opencode mimo-v2.6-flash OpenCode Go or Zen via OPENCODE_API_KEY. Go is the default endpoint; AI_MEMORY_LLM_BASE_URL selects Zen. Set AI_MEMORY_LLM_MODEL to an id the chosen endpoint serves.
openai-compat no default OpenRouter, Atlas Cloud, OrcaRouter, Cheaper Inference, API Route, FutureInfra, Ollama, vLLM, LM Studio, and other compatible endpoints.
openai-compat + AI_MEMORY_LLM_BASE_URL=https://openrouter.ai/api/v1 no default (recommended: anthropic/claude-haiku-4.5) Hosted access to a large model catalogue through one key. See OpenRouter below and the empirical comparison in llm-provider-comparison.md.

openai-oauth stores a refresh token in <data_dir>/auth.json and talks to the ChatGPT/Codex Responses backend, not api.openai.com. For Docker quick starts, run ai-memory auth login openai-oauth with the wrapper so the token lands in the same ai-memory-data volume as the server.

codex is independent from openai-oauth: it never copies credentials into ai-memory's data directory and never reads a refresh token or ID token. It reads only tokens.access_token and tokens.account_id from $CODEX_HOME/auth.json, or from the platform home's .codex/auth.json when CODEX_HOME is unset or empty. Each request reloads the file. A first 401 may trigger one serialized codex app-server --stdio recovery followed by one retry; further 401 responses fail with a reauthentication hint. Override the binary with AI_MEMORY_CODEX_EXECUTABLE when codex is not on PATH.

export AI_MEMORY_LLM_PROVIDER=codex
export AI_MEMORY_LLM_MODEL=gpt-5.6-luna
export AI_MEMORY_LLM_REASONING_EFFORT=medium
ai-memory llm-test --provider codex --model gpt-5.6-luna --prompt "Reply with OK"
ai-memory llm-test --provider codex --model gpt-5.6-luna --structured --prompt "Return a short answer"

cursor uses the Cursor Agent already logged in on this machine. Set AI_MEMORY_LLM_PROVIDER=cursor. The default model is cursor-grok-4.6-high ("Grok 4.6" in agent --list-models); any other id from that list works. AI_MEMORY_CURSOR_AGENT overrides the binary. The provider writes the prompt into a temporary workspace and calls agent --print --mode ask --trust. It does not pass --yolo or --force: consolidation text is untrusted captured content, and those flags would allow tool execution. The Cursor CLI has no JSON-schema mode, so the schema travels in the prompt. The answer is parsed leniently (code fences and surrounding prose are stripped) and then deserialised into the expected type; nothing checks it against the schema itself, so a value that deserialises is accepted even if it breaks a constraint serde does not enforce. The agent process does not inherit the server's environment. It gets PATH, HOME, user and locale variables (LANG, LC_*), temp and XDG directories, DBUS_SESSION_BUS_ADDRESS, proxy and CA settings, the Windows process basics, and every CURSOR_* variable, so AI_MEMORY_AUTH_TOKEN and other providers' keys never reach it. The per-call workspace is created under the system temp directory captured at startup.

export AI_MEMORY_LLM_PROVIDER=cursor
export AI_MEMORY_LLM_MODEL=cursor-grok-4.6-high
ai-memory llm-test --provider cursor --model cursor-grok-4.6-high --prompt "Reply with OK"

Codex credential storage mode file is supported. auto works only when its effective credential is present in auth.json; keyring-only and ephemeral credentials are not read and produce an actionable missing-auth-file error. Docker is not configured automatically: the Codex executable, CODEX_HOME, and its credential file must all exist in the same container/environment as ai-memory.

opencode talks to OpenCode's gateway with the sk-... key from opencode.ai/auth, read from OPENCODE_API_KEY only; it does not fall back to LLM_API_KEY. It defaults to the Go endpoint, https://opencode.ai/zen/go/v1: a subscription with a per-model monthly allowance rather than per-token billing. Set AI_MEMORY_LLM_BASE_URL=https://opencode.ai/zen/v1 (or llm_base_url in config.toml) for Zen's pay-per-token catalogue. Model ids are per catalogue and written plainly (mimo-v2.5, glm-5.3-flash), not in the opencode-go/<model> form OpenCode's own client config uses. The built-in default is mimo-v2.6-flash, served by Go. When selecting Zen, set AI_MEMORY_LLM_MODEL to a model from its catalogue, such as claude-sonnet-5-5. The provider preserves the chosen catalogue and does not switch billing products automatically. Published Responses models, including gpt-5.6-luna and gpt-6-luna, use /responses. The wire API follows the selected catalogue: minimax-m3, minimax-m2.7, and qwen3.8-max use /messages on Go and Chat Completions on Zen. Go's muse-spark-1.3-contributor and muse-spark-1.2-contributor use /responses. Supported Claude and the other mapped Qwen models use /messages; remaining models retain Chat Completions. Changing the base URL reselects the transport and preserves timeout, reasoning effort, and operator headers. Go is identified by a URL path ending in /zen/go or /zen/go/v1, optionally including a complete endpoint; custom proxy URLs with other paths use the Zen mapping. See the official Go endpoints and Zen endpoints, whose source tables are go.mdx and zen.mdx. The provider sends the x-opencode-session correlation header OpenCode asks for, one id per logical operation, and its own User-Agent; AI_MEMORY_LLM_HEADERS overrides either. AI_MEMORY_LLM_REASONING_EFFORT and AI_MEMORY_LLM_TIMEOUT_SECS apply as for every other provider. AI_MEMORY_LLM_PROVIDER / llm_provider also accept the historical aliases opencode-zen and opencode_zen for the same provider (llm-test --provider takes only opencode); the endpoint is chosen by the base URL, not the alias.

export AI_MEMORY_LLM_PROVIDER=opencode
export AI_MEMORY_LLM_MODEL=mimo-v2.6-flash
ai-memory llm-test --provider opencode --model mimo-v2.6-flash --prompt "Reply with OK"

For Claude through Zen:

export AI_MEMORY_LLM_PROVIDER=opencode
export AI_MEMORY_LLM_BASE_URL=https://opencode.ai/zen/v1
export AI_MEMORY_LLM_MODEL=claude-sonnet-5-5
ai-memory llm-test --provider opencode --model claude-sonnet-5-5 --prompt "Reply with OK"

OpenRouter is used through the generic openai-compat provider, the same wiring as Atlas Cloud, OrcaRouter, Ollama, vLLM, and LM Studio. There is no dedicated openrouter provider name, and setting AI_MEMORY_LLM_PROVIDER=openrouter fails at startup with AI_MEMORY_LLM_PROVIDER=openrouter is not one of anthropic|openai|gemini|openai-compat|.... Select it by pointing the compat base URL at OpenRouter and supplying its sk-or-v1-... key through LLM_API_KEY:

A minimal working env-file (drop into ~/.config/ai-memory/openrouter.env or the Docker --env-file argument):

LLM_API_KEY=sk-or-v1-...

AI_MEMORY_LLM_PROVIDER=openai-compat
AI_MEMORY_LLM_BASE_URL=https://openrouter.ai/api/v1
AI_MEMORY_LLM_MODEL=deepseek/deepseek-v4-flash
AI_MEMORY_LLM_COMPAT_STRICT=true

Then verify:

ai-memory llm-test --provider openai-compat \
    --model deepseek/deepseek-v4-flash --prompt "Reply with OK"

AI_MEMORY_LLM_COMPAT_STRICT=true sends OpenAI-style response_format=json_schema for every structured call. Modern hosted gateways honour it; some model backends (DeepSeek among them) otherwise wrap their JSON output in markdown fences that ai-memory's tolerant parser has to strip, which occasionally drops a field. Setting it to true at the env-file level is safer for consolidation reliability. Turn it off (AI_MEMORY_LLM_COMPAT_STRICT=false) only for a specific model backend that explicitly rejects the schema-constrained shape.

The docker/.env.production.example file in the repo ships with an OpenRouter setup pre-filled, with moonshotai/kimi-k2.6 as the sample model (see the gotchas below for why reasoning models are ineligible). Model ids follow OpenRouter's provider/model convention. There is no built-in default: AI_MEMORY_LLM_MODEL is required, as it is for every openai-compat endpoint.

Choosing a model

For LLM selection guidance, see llm-provider-comparison.md, where the project keeps its own A/B tests of consolidation quality, latency, and cost. That comparison uses English consolidation fixtures and covers the recommended default (anthropic/claude-haiku-4.5), the cheaper alternative (openai/gpt-5.4-mini), and the reasoning-model ineligibility note; the OpenRouter subsection here defers to it rather than duplicating rankings.

For workloads dominated by non-English content, community reports suggest that a higher-recall multilingual embedder (for example baai/bge-m3 via the openai-compat embedder) noticeably improves retrieval quality relative to the built-in English-only local embedder. Configuration is in the embedding paragraph below.

ai-memory automatically layers OpenRouter's app-attribution headers (HTTP-Referer and X-Title) whenever the base URL points at openrouter.ai, so ai-memory traffic is credited on OpenRouter's app leaderboard. AI_MEMORY_LLM_HEADERS=HTTP-Referer=https://example.com,X-Title=my-app overrides them per operator. AI_MEMORY_LLM_REASONING_EFFORT is honoured on this path: OpenRouter hosts receive reasoning: { effort, exclude: true }.

OpenRouter's LLM/completions API is what ai-memory uses above. It does not reliably expose an OpenAI-compatible /v1/embeddings endpoint (historically it has not), so do not assume an OpenRouter base URL works for the embedder. If a given deployment of OpenRouter does offer an embeddings endpoint, verify it answers the standard /v1/embeddings shape (curl the endpoint directly) before relying on it. When it does, configure the embedder through the openai-compat path rather than the openai path, which has an OpenAI-specific key precedence chain:

export AI_MEMORY_EMBEDDING_PROVIDER=openai-compat
export AI_MEMORY_EMBEDDING_BASE_URL=https://openrouter.ai/api/v1
export AI_MEMORY_EMBEDDING_MODEL=baai/bge-m3
export AI_MEMORY_EMBEDDING_DIM=1024
# LLM_API_KEY is reused automatically when EMBEDDING_API_KEY is unset on
# the openai-compat embedder path; no second key is required.

AI_MEMORY_EMBEDDING_DIM is mandatory for openai-compat embeddings and must match the model. For English-only content the built-in local sentence-transformer (AI_MEMORY_EMBEDDING_PROVIDER=local, see local-embeddings.md) is a valid alternative that avoids hosted egress; for non-English content, a multilingual hosted model (baai/bge-m3, intfloat/multilingual-e5-large, or similar) served by any provider that exposes a real /v1/embeddings endpoint is the practical choice. The two endpoints are still independent, so setting AI_MEMORY_LLM_BASE_URL alone does not redirect embeddings; set AI_MEMORY_EMBEDDING_BASE_URL too.

[!TIP] openai-compat sends X-Request-Id on every chat attempt. Each HTTP attempt of one logical operation, including the strict-to-tolerant fallback, carries the same operation id (a 36-character UUID v7) in the X-Request-Id header, so a gateway that records it (for example as req=<id>) can correlate every attempt of the same operation and forward it to the engine (vLLM). The caller of the operation (the consolidation invocation or the auto-improve review) generates the id fresh, and it is never the agent's session id, so a gateway never sees two operations of one session, or a crash-resumed run, as one. Crash re-entry is exactly one new operation: a process restart or a queue re-claim mints a new id for the calls it makes. The official openai provider and opencode do not send this header; opencode keeps its own x-opencode-session contract. Because ai-memory owns the header on the openai-compat path, a static x-request-id entry in AI_MEMORY_LLM_HEADERS is refused at startup for that provider; the dynamic value must not be shadowed or duplicated.

Probing the server's own provider

ai-memory llm-test builds a provider from the shell it runs in. That is the right check before a key goes into the config, and it works with the server down — but it is the wrong check when the server runs in a container or on another host, because the keys exist there and not in your shell. A failure like building LLM provider (no key) or a provider 400 Please pass a valid API key says nothing about the server in that case.

ai-memory llm-test --server asks the server to run the probe itself, with its own configured provider, fallback chain and keys, and prints the provider, model, latency and the error class:

ai-memory llm-test --server

It is the same probe status and the health panel name when a provider call fails, it runs one fixed short prompt, and it takes no --provider, --model, --prompt, --base-url or --api-key: a client cannot hand the server a key nor point it at another URL. On a multi-user server the route is root-only, and a provider's error body is scrubbed before it is printed, so a key the provider quoted back never reaches your terminal. Without a configured provider the server answers 503 and says so.

Security and gotchas

  • Bind address vs. port publish. When running inside a container, the process-level bind (--bind 0.0.0.0:49374 or similar) does not decide who can reach the port; the container's publish spec does. -p 127.0.0.1:49374:49374 restricts to loopback on the host; -p 49374:49374 publishes on every host interface. If the port is ever reachable beyond loopback, generate a bearer token first:
    ai-memory generate-auth-token
    export AI_MEMORY_AUTH_TOKEN=<token>
    export AI_MEMORY_ALLOWED_HOSTS=<host,host,...>
    
    The server logs a loud startup warning about non-loopback exposure without a token; treat it as blocking. AI_MEMORY_ALLOWED_HOSTS guards against DNS-rebinding. TLS termination is expected to sit in front of the process; see docs/https-via-proxy.md.
  • Secret hygiene. The env file that holds LLM_API_KEY (and any provider-specific key you set) should be chmod 600 and outside any directory a version-control tool sees. If you back up ~/.config/ai-memory/ (see docs/backup.md), exclude the env file. The sample script and mirror .gitignore shipped there already exclude *.env by default.
  • Cheap models can drift. LLM consolidation writes durable wiki pages, so a model that occasionally fabricates a kind: decision page for a session that made no decision leaves a page in the wiki that ai-memory will treat as first-class evidence on subsequent reads. This is a known cost of running a smaller / cheaper model on the consolidation prompt. Spot-check <data_dir>/wiki/<workspace>/<project>/decisions/ during the first week on a new model for pages you do not recognize; if any appear, either raise the provider tier for consolidation only (keep the cheaper model for retrieval reranking) or unset AI_MEMORY_LLM_PROVIDER to fall back to the deterministic rule-based path.
  • :free-suffix models draw from a shared quota. Any model id ending in :free on OpenRouter is served from a pool shared across all users of the free tier, not the caller's paid budget. When that pool is exhausted, requests fail with an upstream-shared-pool rate-limit error. They are fine for evaluation but unreliable for production; prefer a model listed with multiple upstream providers on its OpenRouter page.
  • Reasoning-mode models are ineligible for consolidation. Consolidation and lint use a strict-JSON output contract; reasoning models route most of their token budget through internal reasoning and either stall past the 300 s per-request ceiling or return empty strings. docs/install.md#llm-provider-tiers has the standing "what we don't recommend" note listing the pattern; the same applies to any OpenRouter model flagged as a reasoning/thinking variant. Prefer a non-reasoning peer for AI_MEMORY_LLM_MODEL.

anthropic-oauth hits the same /v1/messages endpoint as anthropic but authenticates with an OAuth bearer token instead of an API key. Run claude setup-token once; it prints the token and does not save it anywhere, so export it yourself as ANTHROPIC_OAUTH_TOKEN=<token> (or CLAUDE_CODE_OAUTH_TOKEN) and set AI_MEMORY_LLM_PROVIDER=anthropic-oauth. No ANTHROPIC_API_KEY is needed. The Docker wrappers forward either token by name to short-lived helper commands such as llm-test; configure the long-lived server container separately as shown in the installation guide.

For both Anthropic providers, ai-memory omits temperature for Claude 4.7 and later models and Claude Mythos Preview because those models reject non-default sampling parameters. llm-test sends the same representative 0.2 value as the normal pipeline before the provider applies that compatibility rule.

⚠️ Unofficial and against Anthropic's usage policies. Use at your own risk; it may get your account rate-limited or banned. See the warning in docs/install.md.

Anthropic currently refuses subscription tokens from third-party apps (legal and compliance), so expect anthropic-oauth to fail on every request, typically with provider error 429 and rate_limit_error, even though the same token works in the claude CLI. That 429 is a policy refusal, not a real rate limit. Use the anthropic provider with an Anthropic Console API key, or another provider from the table above.

copilot stores a GitHub user token in the same auth file, exchanges it for a short-lived Copilot API token via GitHub's /copilot_internal/v2/token, and uses the Copilot Chat endpoint with vscode-chat integration headers. You can also set COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN on the server.

[!TIP] For the OAuth/subscription backends, prefer a small, fast model via AI_MEMORY_LLM_MODEL where the backend lets you choose one. ai-memory's LLM work (consolidation, lint, explore) is summarisation, not hard reasoning, so a Haiku/mini-class model is plenty and is much easier on subscription rate limits. Save the high-effort thinking models for your coding agent. Per backend:

  • anthropic-oauth: set claude-haiku-4-5.
  • openai-oauth / codex: leave the provider default (gpt-5.5). The Codex/ChatGPT backend only accepts a small server-defined set of model ids and rejects others (e.g. gpt-5-mini) with a deterministic 400, so do not override the model here.
  • copilot: a mini-class id such as gpt-5-mini may work, but Copilot's accepted model set is unverified. Check before relying on it, and fall back to the default if the endpoint rejects your choice.

[!TIP] OpenAI-compatible structured output is schema-constrained by default. ai-memory sends each operation's JSON Schema through response_format=json_schema, which recent Ollama, vLLM, LM Studio, and llama.cpp releases honour. It falls back to the tolerant parser when an endpoint explicitly rejects that field or returns a malformed shape. Set AI_MEMORY_LLM_COMPAT_STRICT=false only for an incompatible endpoint.

The tolerant parser removes <think>, <thinking>, <analysis> and <reasoning> wrappers outside the JSON answer and unwraps Markdown fences. Literal tags inside JSON strings remain intact, so summaries can quote prompt templates or other tagged text. Anthropic's structured text replies use the same parsing rules when the provider uses automatic tool choice.

[!TIP] Thinking-capable local engines (vLLM / SGLang serving Qwen3-class models). Such engines spend the output budget on a reasoning pass before the structured payload and can truncate the JSON mid-object (finish_reason = "length"). Set AI_MEMORY_LLM_COMPAT_DISABLE_THINKING=true to send chat_template_kwargs: {"enable_thinking": false} on every openai-compat request so the engine spends the budget on the payload instead. The knob is opt-in, openai-compat only, and off by default: existing vLLM / Ollama / LM Studio setups are unchanged unless it is set, and every other provider ignores it. When the engine still truncates a structured response, or returns HTTP 2xx with an empty message.content, the call fails fast with the terminal errors truncated-response / empty-content. There is no retry and no second HTTP call, and the error text carries no response content.

For small-context local models, configure both consolidation limits. The input target accounts for the complete rendered prompt, including bounded slot and current-page context plus the structured-output schema; the output limit is sent to the provider. Their sum must fit the model context window, with extra headroom because provider tokenizers differ:

[consolidation]
max_input_tokens = 10000
max_output_tokens = 1000

The equivalent environment variables are AI_MEMORY_CONSOLIDATION__MAX_INPUT_TOKENS and AI_MEMORY_CONSOLIDATION__MAX_OUTPUT_TOKENS. Provider failures during an automatic PreCompact/PostCompaction checkpoint fall back to the deterministic rule-based page; admission, storage, and scope errors still fail closed. The validated minimums are 8,350 input and 1,000 output tokens at the default input_token_safety_margin of 0.8; a lower margin shrinks the character budget and needs a proportionally larger input target.

Every chat provider bounds each completion request at 300 seconds (AI_MEMORY_LLM_TIMEOUT_SECS to override, or llm_timeout_secs = 900 in config.toml; the quick openai-oauth token refresh keeps the default ceiling). The default tolerates a local engine cold-loading a large model; slow hosted gateways whose long completions exceed the ceiling fail every request with http: error sending request, so raise the value there instead of watching consolidation exhaust its retries.

Reranking is optional and off by default. With an LLM provider configured, AI_MEMORY_RERANKER=llm makes project and explicit-scope memory_query calls over-fetch from the hybrid stage, fuse scopes, and make at most one LLM call to reorder the best candidates. This can promote a relevant page that RRF ranked below the requested cut, at the cost of LLM latency and usage. The request sends the query plus at most 30 bounded page titles and search snippets to the configured provider; all values are JSON-encoded and treated as untrusted data. A timeout, provider error, or incomplete/invalid score set preserves the normal order. The timeout is 20 seconds per query; raise it with AI_MEMORY_RERANK_TIMEOUT_SECS (or rerank_timeout_secs in config.toml) when a slow provider or a fallback chain needs longer to answer. global=true and supplemental global-preference hits keep their existing non-RRF ranking. Concurrent provider calls are capped at four; saturated queries keep their local ranking without waiting.

Workload telemetry

ai-memory status --json (the root-protected GET /admin/status response) includes providers.llm_workloads whenever an LLM provider is configured. Its closed labels are unspecified, consolidation, rerank, auto-improve, bootstrap, query-answer, lint, profile, dream, and experience. Each row reports process-lifetime calls, successes and errors, total/max latency, fixed-bucket p50/p95 upper bounds, last outcome, and token counts. The counters are process-local and reset on restart; the server constructs the configured provider wrapper once at startup and has no runtime provider reload.

estimated_input_tokens and estimated_output_tokens use a deliberately coarse four-characters-per-token estimate so structured and plain calls remain comparable. reported_input_tokens and reported_output_tokens count only responses for which the provider supplied usage; usage_reported_calls shows that coverage. Structured-output provider adapters currently return decoded JSON without usage, so their reported counters can remain zero even when the estimate is nonzero.

The telemetry stores no prompt, completion, scope, project, or caller-supplied label. A fallback chain is measured as one provider-interface invocation, not as one call per internal candidate attempt; an outer caller retry is another invocation. Workload labels do not select a provider or model: all operations still use the configured global provider and fallback chain. Per-workload routing remains measure-first and has no public configuration surface.

Embeddings are optional and separate from the LLM provider. Set AI_MEMORY_EMBEDDING_PROVIDER=openai, voyage, google/gemini, openai-compat, or copilot when you want vector retrieval in addition to FTS5 + entity + graph-neighbor retrieval. openai-compat targets self-hosted engines (Ollama, LM Studio, vLLM): it needs no API key and requires explicit AI_MEMORY_EMBEDDING_BASE_URL, AI_MEMORY_EMBEDDING_MODEL, and AI_MEMORY_EMBEDDING_DIM. The optional EMBEDDING_API_KEY credentials the embedder alone and is checked before OPENAI_API_KEY and LLM_API_KEY, so embeddings can run on a different provider than the LLM. Because the two endpoints are independent, AI_MEMORY_LLM_BASE_URL redirects only the LLM; set AI_MEMORY_EMBEDDING_BASE_URL as well or embedding traffic still goes to the embedding provider's default endpoint. Both the FTS-only and hybrid paths apply the same bounded page-authority adjustment after candidate generation; embeddings improve relevance recall but do not decide which source is canonical.

Asymmetric embedding models, trained with a different instruction for queries than for the text being indexed, need that instruction added to the text itself; the OpenAI-compatible /v1/embeddings wire format has no field for it. AI_MEMORY_EMBEDDING_QUERY_PREFIX and AI_MEMORY_EMBEDDING_DOCUMENT_PREFIX (only applied by the openai and openai-compat embedders) are prepended to query and document text respectively, before the existing truncation, so a long body is still truncated to the same overall input cap with the prefix included. Both are empty by default (no behaviour change when unset) and are not trimmed, so a publisher's trailing space is preserved exactly. For example, NVIDIA's nvidia/Nemotron-3-Embed-1B-BF16 (2048-dim, served locally through vLLM/openai-compat) specifies "query: " for queries and "passage: " for documents per its model card:

embedding_provider = "openai-compat"
embedding_model = "nvidia/Nemotron-3-Embed-1B-BF16"
embedding_base_url = "http://localhost:8000/v1"
embedding_dim = 2048
embedding_query_prefix = "query: "
embedding_document_prefix = "passage: "

Base E5 models (intfloat/e5-base-v2, e5-large-v2, multilingual E5, …) use the same "query: " / "passage: " convention. Instruction-tuned E5 variants and Qwen3-Embedding instead need a task-instruction string on the query side only (their documents are embedded plain, with no document prefix), and the two use different exact spacing, confirmed against each model card:

  • intfloat/e5-mistral-7b-instruct: embedding_query_prefix = "Instruct: {task description}\nQuery: ", with a trailing space after Query:.
  • Qwen/Qwen3-Embedding-0.6B (and the other Qwen3-Embedding sizes): embedding_query_prefix = "Instruct: {task description}\nQuery:", with no trailing space; the query text follows the colon directly.

Fill in your own task description for {task description}, leave embedding_document_prefix unset for both, and don't copy one model's exact string for the other. The trailing-space difference is publisher-specified, not a typo.

Changing embedding_query_prefix alone never requires re-embedding: only embedding_document_prefix is folded into the stored embedding identity (a non-empty value changes what ai-memory treats as the current {provider, model, dim} triple for that page's vectors), so a document prefix change makes memory_query stop matching the old vectors and the next ai-memory embed (scheduled or manual) pass re-embed them automatically, without --force. --force remains useful to re-embed everything immediately rather than waiting for scheduled backfill, or to force a re-embed for a reason the triple alone doesn't capture (e.g. touching up the vLLM server's tokenizer or quantization).

AI_MEMORY_EMBEDDING_PROVIDER=copilot reuses the same Copilot OAuth login as the copilot LLM provider (ai-memory auth login copilot or COPILOT_GITHUB_TOKEN/GITHUB_COPILOT_API_TOKEN), with no separate API key. It defaults to model text-embedding-3-small, dim 1536, and calls Copilot's /embeddings endpoint following the same OpenAI-compatible contract Copilot documents for chat. That endpoint's exact shape is not covered by a live test against Copilot here; treat it as needing a real-Copilot smoke test before relying on it in production.

AI_MEMORY_EMBEDDING_PROVIDER=local needs no key and no server at all: sentence embeddings run in-process (pure-Rust all-MiniLM-L6-v2, 384-dim), with the model fetched once into <data_dir>/models/ under pinned checksums; see docs/local-embeddings.md.

See docs/install.md#llm-provider-tiers for env vars and Ollama/OpenRouter/Atlas Cloud/OrcaRouter/Cheaper Inference/API Route/FutureInfra examples, and docs/llm-provider-comparison.md for the empirical model comparison.