Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
LLM Providers
Provider configuration for consolidation and embeddings. Moved verbatim from the README front page.
ai-memory runs without an LLM: hooks still capture sessions, search uses
FTS5 + declared entities + graph neighbors, and summaries fall back to
rule-based output. Add an LLM provider
when you want LLM consolidation (on PreCompact / PostCompaction, on demand via
memory_consolidate, or opt-in at session end with
AI_MEMORY_CONSOLIDATE_ON_SESSION_END), richer linting, and bootstrap.
Substantive session ends always write a rule-based summary page + handoff either
way. A session containing only SessionStart / SessionEnd boundaries is
closed without a page, handoff, or provider job. When that empty session had
accepted startup context, its session-bound handoff is returned to the open
pool for the next receiver instead of being lost.
PreCompact / PostCompaction write a rule-based checkpoint on the hook request.
Provider work for those checkpoints, and for opt-in SessionEnd, is durably
queued after those deterministic writes and handled by one bounded server
worker, so hook drain latency does not cancel it. The worker starts whenever a
consolidator is configured; SessionEnd enqueue stays behind the opt-in flag.
A checkpoint rewrite still queued when the session ends runs once anyway, even
with the flag off. One already running when the session ends finishes after the
SessionEnd summary and replaces it, so with the flag off the latest page can
leave out the end of the session; the summary stays reachable as the superseded
version. Every queued job runs as the session owner recorded at session start,
not anonymously. Failed jobs retry with backoff and survive a server restart. A resumed native session is ended again only after its
observation generation advances; the persisted generation watermark makes
duplicate SessionEnd delivery and system clock skew converge without repeated
provider work. The end watermark and automatic handoff commit atomically, and
an interrupted keyed replay finishes the wiki commit, queue insert, and key
completion without duplicating that handoff. On the next SessionStart, the
newest cwd-eligible automatic handoff wins; accepting it expires older eligible
automatic handoffs without consuming manual or sibling-directory work. A new
automatic handoff also expires prior open automatic handoffs from its exact
cwd, so repeated SessionEnds cannot accumulate there before a receiver starts.
To keep consolidation style project-specific, write
_prompts/consolidation.md in that project's wiki. Its body can express
preferences such as "prefer Portuguese titles" or "omit routine CI noise".
Automatic, single-page, and multi-page consolidation use the page; a manual
memory_consolidate call can pass instructions to override it once. ai-memory
sanitizes and caps the value at 2,000 characters, JSON-encodes it in the user
message, and treats it as untrusted advisory data. It cannot supply facts,
request tool use or disclosure, or override the consolidation schema and
faithfulness rules. TTL-expired preference pages are ignored. With no active
page or argument, no preference block is appended.
Recommended defaults:
| Provider | Default | Use when |
|---|---|---|
anthropic |
claude-haiku-4-5 |
Best default for consolidation quality and rule classification. |
anthropic-oauth |
claude-sonnet-4-6 |
Use a Claude Pro/Max subscription via claude setup-token, no API key. Currently refused by Anthropic for third-party apps (usually 429); see below. |
openai |
gpt-5.4-mini |
Cheaper and faster hosted option. |
openai-oauth |
gpt-5.5 |
ChatGPT Pro/Plus/Codex backend via ai-memory auth login openai-oauth; no Platform API key. |
codex |
gpt-5.6-luna |
Reuse the Codex CLI-owned auth.json; access-token refresh remains owned by codex app-server. |
cursor |
cursor-grok-4.6-high |
Cursor subscription via the logged-in agent CLI. Runs agent --print --mode ask (read-only). Does not pass --yolo. The default id is the one agent --list-models labels "Grok 4.6". |
copilot |
gpt-5.5 |
GitHub Copilot Chat backend via ai-memory auth login copilot or COPILOT_GITHUB_TOKEN; requires a Copilot subscription. |
gemini |
gemini-3.5-flash |
Google-hosted option with a generous free tier. |
opencode |
mimo-v2.6-flash |
OpenCode Go or Zen via OPENCODE_API_KEY. Go is the default endpoint; AI_MEMORY_LLM_BASE_URL selects Zen. Set AI_MEMORY_LLM_MODEL to an id the chosen endpoint serves. |
openai-compat |
no default | OpenRouter, Atlas Cloud, OrcaRouter, Cheaper Inference, API Route, FutureInfra, Ollama, vLLM, LM Studio, and other compatible endpoints. |
openai-compat + AI_MEMORY_LLM_BASE_URL=https://openrouter.ai/api/v1 |
no default (recommended: anthropic/claude-haiku-4.5) |
Hosted access to a large model catalogue through one key. See OpenRouter below and the empirical comparison in llm-provider-comparison.md. |
openai-oauth stores a refresh token in <data_dir>/auth.json and talks to
the ChatGPT/Codex Responses backend, not api.openai.com. For Docker quick
starts, run ai-memory auth login openai-oauth with the wrapper so the token
lands in the same ai-memory-data volume as the server.
codex is independent from openai-oauth: it never copies credentials into
ai-memory's data directory and never reads a refresh token or ID token. It
reads only tokens.access_token and tokens.account_id from
$CODEX_HOME/auth.json, or from the platform home's .codex/auth.json when
CODEX_HOME is unset or empty. Each request reloads the file. A first 401 may
trigger one serialized codex app-server --stdio recovery followed by one
retry; further 401 responses fail with a reauthentication hint. Override the
binary with AI_MEMORY_CODEX_EXECUTABLE when codex is not on PATH.
export AI_MEMORY_LLM_PROVIDER=codex
export AI_MEMORY_LLM_MODEL=gpt-5.6-luna
export AI_MEMORY_LLM_REASONING_EFFORT=medium
ai-memory llm-test --provider codex --model gpt-5.6-luna --prompt "Reply with OK"
ai-memory llm-test --provider codex --model gpt-5.6-luna --structured --prompt "Return a short answer"
cursor uses the Cursor Agent already logged in on this machine. Set
AI_MEMORY_LLM_PROVIDER=cursor. The default model is cursor-grok-4.6-high
("Grok 4.6" in agent --list-models); any other id from that list works.
AI_MEMORY_CURSOR_AGENT overrides the binary. The provider writes the prompt
into a temporary workspace and calls agent --print --mode ask --trust. It
does not pass --yolo or --force: consolidation text is untrusted captured
content, and those flags would allow tool execution. The Cursor CLI has no
JSON-schema mode, so the schema travels in the prompt. The answer is parsed
leniently (code fences and surrounding prose are stripped) and then
deserialised into the expected type; nothing checks it against the schema
itself, so a value that deserialises is accepted even if it breaks a constraint
serde does not enforce. The agent process does not inherit the server's
environment. It gets PATH, HOME, user and locale variables (LANG,
LC_*), temp and XDG directories, DBUS_SESSION_BUS_ADDRESS, proxy and CA
settings, the Windows process basics, and every CURSOR_* variable, so
AI_MEMORY_AUTH_TOKEN and other providers' keys never reach it. The per-call
workspace is created under the system temp directory captured at startup.
export AI_MEMORY_LLM_PROVIDER=cursor
export AI_MEMORY_LLM_MODEL=cursor-grok-4.6-high
ai-memory llm-test --provider cursor --model cursor-grok-4.6-high --prompt "Reply with OK"
Codex credential storage mode file is supported. auto works only when its
effective credential is present in auth.json; keyring-only and ephemeral
credentials are not read and produce an actionable missing-auth-file error.
Docker is not configured automatically: the Codex executable, CODEX_HOME,
and its credential file must all exist in the same container/environment as
ai-memory.
opencode talks to OpenCode's gateway with the sk-... key from
opencode.ai/auth, read from OPENCODE_API_KEY only; it does not fall back
to LLM_API_KEY. It defaults to the Go endpoint,
https://opencode.ai/zen/go/v1: a subscription with a per-model monthly
allowance rather than per-token billing. Set
AI_MEMORY_LLM_BASE_URL=https://opencode.ai/zen/v1 (or llm_base_url in
config.toml) for Zen's pay-per-token catalogue. Model ids are per
catalogue and written plainly (mimo-v2.5, glm-5.3-flash), not in the
opencode-go/<model> form OpenCode's own client config uses. The built-in
default is mimo-v2.6-flash, served by Go. When selecting Zen, set
AI_MEMORY_LLM_MODEL to a model from its catalogue, such as claude-sonnet-5-5.
The provider preserves the chosen catalogue and does not switch billing products
automatically. Published Responses models, including gpt-5.6-luna and
gpt-6-luna, use /responses. The wire API follows the selected catalogue:
minimax-m3, minimax-m2.7, and qwen3.8-max use /messages on Go and
Chat Completions on Zen. Go's muse-spark-1.3-contributor and
muse-spark-1.2-contributor use /responses. Supported Claude and the other
mapped Qwen models use /messages; remaining models retain Chat Completions.
Changing the base URL reselects the transport and preserves timeout, reasoning
effort, and operator headers. Go is identified by a URL path ending in
/zen/go or /zen/go/v1, optionally including a complete endpoint;
custom proxy URLs with other paths use the Zen mapping.
See the official Go endpoints and
Zen endpoints, whose source tables
are go.mdx
and zen.mdx.
The provider sends the
x-opencode-session correlation header OpenCode asks for, one id per logical
operation, and its own User-Agent; AI_MEMORY_LLM_HEADERS overrides
either. AI_MEMORY_LLM_REASONING_EFFORT and AI_MEMORY_LLM_TIMEOUT_SECS
apply as for every other provider. AI_MEMORY_LLM_PROVIDER / llm_provider
also accept the historical aliases opencode-zen and opencode_zen for the
same provider (llm-test --provider takes only opencode); the endpoint is
chosen by the base URL, not the alias.
export AI_MEMORY_LLM_PROVIDER=opencode
export AI_MEMORY_LLM_MODEL=mimo-v2.6-flash
ai-memory llm-test --provider opencode --model mimo-v2.6-flash --prompt "Reply with OK"
For Claude through Zen:
export AI_MEMORY_LLM_PROVIDER=opencode
export AI_MEMORY_LLM_BASE_URL=https://opencode.ai/zen/v1
export AI_MEMORY_LLM_MODEL=claude-sonnet-5-5
ai-memory llm-test --provider opencode --model claude-sonnet-5-5 --prompt "Reply with OK"
OpenRouter is used through the generic openai-compat
provider, the same wiring as Atlas Cloud, OrcaRouter, Ollama, vLLM, and LM
Studio. There is no dedicated openrouter provider name, and setting
AI_MEMORY_LLM_PROVIDER=openrouter fails at startup with
AI_MEMORY_LLM_PROVIDER=openrouter is not one of anthropic|openai|gemini|openai-compat|....
Select it by pointing the compat base URL at OpenRouter and supplying its
sk-or-v1-... key through LLM_API_KEY:
A minimal working env-file (drop into ~/.config/ai-memory/openrouter.env
or the Docker --env-file argument):
LLM_API_KEY=sk-or-v1-...
AI_MEMORY_LLM_PROVIDER=openai-compat
AI_MEMORY_LLM_BASE_URL=https://openrouter.ai/api/v1
AI_MEMORY_LLM_MODEL=deepseek/deepseek-v4-flash
AI_MEMORY_LLM_COMPAT_STRICT=true
Then verify:
ai-memory llm-test --provider openai-compat \
--model deepseek/deepseek-v4-flash --prompt "Reply with OK"
AI_MEMORY_LLM_COMPAT_STRICT=true sends OpenAI-style
response_format=json_schema for every structured call. Modern hosted
gateways honour it; some model backends (DeepSeek among them) otherwise wrap
their JSON output in markdown fences that ai-memory's tolerant parser has to
strip, which occasionally drops a field. Setting it to true at the
env-file level is safer for consolidation reliability. Turn it off
(AI_MEMORY_LLM_COMPAT_STRICT=false) only for a specific model backend that
explicitly rejects the schema-constrained shape.
The docker/.env.production.example file in the repo ships with an OpenRouter
setup pre-filled, with moonshotai/kimi-k2.6 as the sample model (see the
gotchas below for why reasoning models are ineligible). Model ids follow
OpenRouter's provider/model convention. There is no built-in default:
AI_MEMORY_LLM_MODEL is required, as it is for every openai-compat endpoint.
Choosing a model
For LLM selection guidance, see
llm-provider-comparison.md, where the project
keeps its own A/B tests of consolidation quality, latency, and cost. That comparison uses English consolidation fixtures and covers the
recommended default (anthropic/claude-haiku-4.5), the cheaper alternative
(openai/gpt-5.4-mini), and the reasoning-model ineligibility note; the
OpenRouter subsection here defers to it rather than duplicating rankings.
For workloads dominated by non-English content, community reports suggest
that a higher-recall multilingual embedder (for example baai/bge-m3 via
the openai-compat embedder) noticeably improves retrieval quality
relative to the built-in English-only local embedder. Configuration is in
the embedding paragraph below.
ai-memory automatically layers OpenRouter's app-attribution headers
(HTTP-Referer and X-Title) whenever the base URL points at
openrouter.ai, so ai-memory traffic is credited on OpenRouter's app
leaderboard. AI_MEMORY_LLM_HEADERS=HTTP-Referer=https://example.com,X-Title=my-app
overrides them per operator. AI_MEMORY_LLM_REASONING_EFFORT is honoured on
this path: OpenRouter hosts receive reasoning: { effort, exclude: true }.
OpenRouter's LLM/completions API is what ai-memory uses above. It does not
reliably expose an OpenAI-compatible /v1/embeddings endpoint (historically it
has not), so do not assume an OpenRouter base URL works for the embedder. If a
given deployment of OpenRouter does offer an embeddings endpoint, verify it
answers the standard /v1/embeddings shape (curl the endpoint directly)
before relying on it. When it does, configure the embedder through the
openai-compat path rather than the openai path, which has an OpenAI-specific
key precedence chain:
export AI_MEMORY_EMBEDDING_PROVIDER=openai-compat
export AI_MEMORY_EMBEDDING_BASE_URL=https://openrouter.ai/api/v1
export AI_MEMORY_EMBEDDING_MODEL=baai/bge-m3
export AI_MEMORY_EMBEDDING_DIM=1024
# LLM_API_KEY is reused automatically when EMBEDDING_API_KEY is unset on
# the openai-compat embedder path; no second key is required.
AI_MEMORY_EMBEDDING_DIM is mandatory for openai-compat embeddings and must
match the model. For English-only content the built-in local
sentence-transformer (AI_MEMORY_EMBEDDING_PROVIDER=local, see
local-embeddings.md) is a valid alternative that
avoids hosted egress; for non-English content, a multilingual hosted model
(baai/bge-m3, intfloat/multilingual-e5-large, or similar) served by any
provider that exposes a real /v1/embeddings endpoint is the practical choice.
The two endpoints are still independent, so setting AI_MEMORY_LLM_BASE_URL
alone does not redirect embeddings; set AI_MEMORY_EMBEDDING_BASE_URL too.
[!TIP]
openai-compatsendsX-Request-Idon every chat attempt. Each HTTP attempt of one logical operation, including the strict-to-tolerant fallback, carries the same operation id (a 36-character UUID v7) in theX-Request-Idheader, so a gateway that records it (for example asreq=<id>) can correlate every attempt of the same operation and forward it to the engine (vLLM). The caller of the operation (the consolidation invocation or the auto-improve review) generates the id fresh, and it is never the agent's session id, so a gateway never sees two operations of one session, or a crash-resumed run, as one. Crash re-entry is exactly one new operation: a process restart or a queue re-claim mints a new id for the calls it makes. The officialopenaiprovider andopencodedo not send this header;opencodekeeps its ownx-opencode-sessioncontract. Because ai-memory owns the header on theopenai-compatpath, a staticx-request-identry inAI_MEMORY_LLM_HEADERSis refused at startup for that provider; the dynamic value must not be shadowed or duplicated.
Probing the server's own provider
ai-memory llm-test builds a provider from the shell it runs in. That is the
right check before a key goes into the config, and it works with the server
down — but it is the wrong check when the server runs in a container or on
another host, because the keys exist there and not in your shell. A failure
like building LLM provider (no key) or a provider 400 Please pass a valid API key says nothing about the server in that case.
ai-memory llm-test --server asks the server to run the probe itself, with its
own configured provider, fallback chain and keys, and prints the provider,
model, latency and the error class:
ai-memory llm-test --server
It is the same probe status and the health panel name when a provider call
fails, it runs one fixed short prompt, and it takes no --provider, --model,
--prompt, --base-url or --api-key: a client cannot hand the server a key
nor point it at another URL. On a multi-user server the route is root-only, and
a provider's error body is scrubbed before it is printed, so a key the provider
quoted back never reaches your terminal. Without a configured provider the
server answers 503 and says so.
Security and gotchas
- Bind address vs. port publish. When running inside a container, the
process-level bind (
--bind 0.0.0.0:49374or similar) does not decide who can reach the port; the container's publish spec does.-p 127.0.0.1:49374:49374restricts to loopback on the host;-p 49374:49374publishes on every host interface. If the port is ever reachable beyond loopback, generate a bearer token first:
The server logs a loud startup warning about non-loopback exposure without a token; treat it as blocking.ai-memory generate-auth-token export AI_MEMORY_AUTH_TOKEN=<token> export AI_MEMORY_ALLOWED_HOSTS=<host,host,...>AI_MEMORY_ALLOWED_HOSTSguards against DNS-rebinding. TLS termination is expected to sit in front of the process; seedocs/https-via-proxy.md. - Secret hygiene. The env file that holds
LLM_API_KEY(and any provider-specific key you set) should bechmod 600and outside any directory a version-control tool sees. If you back up~/.config/ai-memory/(seedocs/backup.md), exclude the env file. The sample script and mirror.gitignoreshipped there already exclude*.envby default. - Cheap models can drift. LLM consolidation writes durable wiki
pages, so a model that occasionally fabricates a
kind: decisionpage for a session that made no decision leaves a page in the wiki that ai-memory will treat as first-class evidence on subsequent reads. This is a known cost of running a smaller / cheaper model on the consolidation prompt. Spot-check<data_dir>/wiki/<workspace>/<project>/decisions/during the first week on a new model for pages you do not recognize; if any appear, either raise the provider tier for consolidation only (keep the cheaper model for retrieval reranking) or unsetAI_MEMORY_LLM_PROVIDERto fall back to the deterministic rule-based path. :free-suffix models draw from a shared quota. Any model id ending in:freeon OpenRouter is served from a pool shared across all users of the free tier, not the caller's paid budget. When that pool is exhausted, requests fail with an upstream-shared-pool rate-limit error. They are fine for evaluation but unreliable for production; prefer a model listed with multiple upstream providers on its OpenRouter page.- Reasoning-mode models are ineligible for consolidation.
Consolidation and lint use a strict-JSON output contract; reasoning
models route most of their token budget through internal reasoning
and either stall past the 300 s per-request ceiling or return empty
strings.
docs/install.md#llm-provider-tiershas the standing "what we don't recommend" note listing the pattern; the same applies to any OpenRouter model flagged as a reasoning/thinking variant. Prefer a non-reasoning peer forAI_MEMORY_LLM_MODEL.
anthropic-oauth hits the same /v1/messages endpoint as anthropic but
authenticates with an OAuth bearer token instead of an API key. Run
claude setup-token once; it prints the token and does not save it anywhere, so
export it yourself as ANTHROPIC_OAUTH_TOKEN=<token> (or
CLAUDE_CODE_OAUTH_TOKEN) and set AI_MEMORY_LLM_PROVIDER=anthropic-oauth. No
ANTHROPIC_API_KEY is needed. The Docker
wrappers forward either token by name to short-lived helper commands such as
llm-test; configure the long-lived server container separately as shown in the
installation guide.
For both Anthropic providers, ai-memory omits temperature for Claude
4.7 and later models and Claude Mythos Preview because those models reject
non-default sampling parameters. llm-test sends the same representative 0.2
value as the normal pipeline before the provider applies that compatibility
rule.
⚠️ Unofficial and against Anthropic's usage policies. Use at your own risk;
it may get your account rate-limited or banned. See
the warning in docs/install.md.
Anthropic currently refuses subscription tokens from third-party apps
(legal and compliance),
so expect anthropic-oauth to fail on every request, typically with
provider error 429 and rate_limit_error, even though the same token works
in the claude CLI. That 429 is a policy refusal, not a real rate limit. Use
the anthropic provider with an Anthropic Console API key, or another provider
from the table above.
copilot stores a GitHub user token in the same auth file, exchanges it for a
short-lived Copilot API token via GitHub's /copilot_internal/v2/token, and
uses the Copilot Chat endpoint with vscode-chat integration headers. You can
also set COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN on the server.
[!TIP] For the OAuth/subscription backends, prefer a small, fast model via
AI_MEMORY_LLM_MODELwhere the backend lets you choose one. ai-memory's LLM work (consolidation, lint, explore) is summarisation, not hard reasoning, so a Haiku/mini-class model is plenty and is much easier on subscription rate limits. Save the high-effort thinking models for your coding agent. Per backend:
anthropic-oauth: setclaude-haiku-4-5.openai-oauth/codex: leave the provider default (gpt-5.5). The Codex/ChatGPT backend only accepts a small server-defined set of model ids and rejects others (e.g.gpt-5-mini) with a deterministic 400, so do not override the model here.copilot: a mini-class id such asgpt-5-minimay work, but Copilot's accepted model set is unverified. Check before relying on it, and fall back to the default if the endpoint rejects your choice.
[!TIP] OpenAI-compatible structured output is schema-constrained by default. ai-memory sends each operation's JSON Schema through
response_format=json_schema, which recent Ollama, vLLM, LM Studio, and llama.cpp releases honour. It falls back to the tolerant parser when an endpoint explicitly rejects that field or returns a malformed shape. SetAI_MEMORY_LLM_COMPAT_STRICT=falseonly for an incompatible endpoint.
The tolerant parser removes <think>, <thinking>, <analysis> and
<reasoning> wrappers outside the JSON answer and unwraps Markdown fences.
Literal tags inside JSON strings remain intact, so summaries can quote prompt
templates or other tagged text. Anthropic's structured text replies use the
same parsing rules when the provider uses automatic tool choice.
[!TIP] Thinking-capable local engines (vLLM / SGLang serving Qwen3-class models). Such engines spend the output budget on a reasoning pass before the structured payload and can truncate the JSON mid-object (
finish_reason = "length"). SetAI_MEMORY_LLM_COMPAT_DISABLE_THINKING=trueto sendchat_template_kwargs: {"enable_thinking": false}on every openai-compat request so the engine spends the budget on the payload instead. The knob is opt-in, openai-compat only, and off by default: existing vLLM / Ollama / LM Studio setups are unchanged unless it is set, and every other provider ignores it. When the engine still truncates a structured response, or returns HTTP 2xx with an emptymessage.content, the call fails fast with the terminal errorstruncated-response/empty-content. There is no retry and no second HTTP call, and the error text carries no response content.
For small-context local models, configure both consolidation limits. The input target accounts for the complete rendered prompt, including bounded slot and current-page context plus the structured-output schema; the output limit is sent to the provider. Their sum must fit the model context window, with extra headroom because provider tokenizers differ:
[consolidation]
max_input_tokens = 10000
max_output_tokens = 1000
The equivalent environment variables are
AI_MEMORY_CONSOLIDATION__MAX_INPUT_TOKENS and
AI_MEMORY_CONSOLIDATION__MAX_OUTPUT_TOKENS. Provider failures during an
automatic PreCompact/PostCompaction checkpoint fall back to the deterministic
rule-based page; admission, storage, and scope errors still fail closed. The
validated minimums are 8,350 input and 1,000 output tokens at the default
input_token_safety_margin of 0.8; a lower margin shrinks the character budget
and needs a proportionally larger input target.
Every chat provider bounds each completion request at 300 seconds
(AI_MEMORY_LLM_TIMEOUT_SECS to override, or llm_timeout_secs = 900 in
config.toml; the quick openai-oauth token refresh keeps the default ceiling).
The default tolerates a local engine cold-loading a large model; slow hosted
gateways whose long completions exceed the ceiling fail every request with
http: error sending request, so raise the value there instead of watching
consolidation exhaust its retries.
Reranking is optional and off by default. With an LLM provider configured,
AI_MEMORY_RERANKER=llm makes project and explicit-scope memory_query
calls over-fetch from the hybrid stage, fuse scopes, and make at most one LLM
call to reorder the best candidates. This can promote a relevant page that
RRF ranked below the requested cut, at the cost of LLM latency and usage. The
request sends the query plus at most 30 bounded page titles and search snippets
to the configured provider; all values are JSON-encoded and treated as
untrusted data. A timeout, provider error, or incomplete/invalid score set
preserves the normal order. The timeout is 20 seconds per query; raise it with
AI_MEMORY_RERANK_TIMEOUT_SECS (or rerank_timeout_secs in config.toml)
when a slow provider or a fallback chain needs longer to answer. global=true and supplemental global-preference
hits keep their existing non-RRF ranking. Concurrent provider calls are capped
at four; saturated queries keep their local ranking without waiting.
Workload telemetry
ai-memory status --json (the root-protected GET /admin/status response)
includes providers.llm_workloads whenever an LLM provider is configured. Its
closed labels are unspecified, consolidation, rerank, auto-improve,
bootstrap, query-answer, lint, profile, dream, and experience.
Each row reports process-lifetime calls, successes and errors, total/max
latency, fixed-bucket p50/p95 upper bounds, last outcome, and token counts. The
counters are process-local and reset on restart; the server constructs the
configured provider wrapper once at startup and has no runtime provider reload.
estimated_input_tokens and estimated_output_tokens use a deliberately
coarse four-characters-per-token estimate so structured and plain calls remain
comparable. reported_input_tokens and reported_output_tokens count only
responses for which the provider supplied usage; usage_reported_calls shows
that coverage. Structured-output provider adapters currently return decoded
JSON without usage, so their reported counters can remain zero even when the
estimate is nonzero.
The telemetry stores no prompt, completion, scope, project, or caller-supplied label. A fallback chain is measured as one provider-interface invocation, not as one call per internal candidate attempt; an outer caller retry is another invocation. Workload labels do not select a provider or model: all operations still use the configured global provider and fallback chain. Per-workload routing remains measure-first and has no public configuration surface.
Embeddings are optional and separate from the LLM provider. Set
AI_MEMORY_EMBEDDING_PROVIDER=openai, voyage, google/gemini,
openai-compat, or copilot when you want vector retrieval in addition to
FTS5 + entity + graph-neighbor retrieval. openai-compat targets self-hosted
engines (Ollama, LM Studio, vLLM): it needs no API key and requires explicit
AI_MEMORY_EMBEDDING_BASE_URL, AI_MEMORY_EMBEDDING_MODEL, and
AI_MEMORY_EMBEDDING_DIM. The optional EMBEDDING_API_KEY credentials the
embedder alone and is checked before OPENAI_API_KEY and LLM_API_KEY, so
embeddings can run on a different provider than the LLM. Because the two
endpoints are independent, AI_MEMORY_LLM_BASE_URL redirects only the LLM;
set AI_MEMORY_EMBEDDING_BASE_URL as well or embedding traffic still goes to
the embedding provider's default endpoint. Both the FTS-only and
hybrid paths apply the same bounded page-authority adjustment after candidate
generation; embeddings improve relevance recall but do not decide which source
is canonical.
Asymmetric embedding models, trained with a different instruction for
queries than for the text being indexed, need that instruction added to the
text itself; the OpenAI-compatible /v1/embeddings wire format has no field
for it. AI_MEMORY_EMBEDDING_QUERY_PREFIX and
AI_MEMORY_EMBEDDING_DOCUMENT_PREFIX (only applied by the openai and
openai-compat embedders) are prepended to query and document text
respectively, before the existing truncation, so a long body is still
truncated to the same overall input cap with the prefix included. Both are
empty by default (no behaviour change when unset) and are not trimmed, so
a publisher's trailing space is preserved exactly. For example, NVIDIA's
nvidia/Nemotron-3-Embed-1B-BF16
(2048-dim, served locally through vLLM/openai-compat) specifies
"query: " for queries and "passage: " for documents per its model card:
embedding_provider = "openai-compat"
embedding_model = "nvidia/Nemotron-3-Embed-1B-BF16"
embedding_base_url = "http://localhost:8000/v1"
embedding_dim = 2048
embedding_query_prefix = "query: "
embedding_document_prefix = "passage: "
Base E5 models (intfloat/e5-base-v2, e5-large-v2, multilingual E5, …)
use the same "query: " / "passage: " convention. Instruction-tuned E5
variants and Qwen3-Embedding instead need a task-instruction string on the
query side only (their documents are embedded plain, with no document
prefix), and the two use different exact spacing, confirmed against
each model card:
intfloat/e5-mistral-7b-instruct:embedding_query_prefix = "Instruct: {task description}\nQuery: ", with a trailing space afterQuery:.Qwen/Qwen3-Embedding-0.6B(and the other Qwen3-Embedding sizes):embedding_query_prefix = "Instruct: {task description}\nQuery:", with no trailing space; the query text follows the colon directly.
Fill in your own task description for {task description}, leave
embedding_document_prefix unset for both, and don't copy one model's
exact string for the other. The trailing-space difference is
publisher-specified, not a typo.
Changing embedding_query_prefix alone never requires re-embedding: only
embedding_document_prefix is folded into the stored embedding identity
(a non-empty value changes what ai-memory treats as the current
{provider, model, dim} triple for that page's vectors), so a document
prefix change makes memory_query stop matching the old vectors and the
next ai-memory embed (scheduled or manual) pass re-embed them
automatically, without --force. --force remains useful to
re-embed everything immediately rather than waiting for scheduled
backfill, or to force a re-embed for a reason the triple alone doesn't
capture (e.g. touching up the vLLM server's tokenizer or quantization).
AI_MEMORY_EMBEDDING_PROVIDER=copilot reuses the same Copilot OAuth login as
the copilot LLM provider (ai-memory auth login copilot or
COPILOT_GITHUB_TOKEN/GITHUB_COPILOT_API_TOKEN), with no separate API key.
It defaults to model text-embedding-3-small, dim 1536, and calls Copilot's
/embeddings endpoint following the same OpenAI-compatible contract Copilot
documents for chat. That endpoint's exact shape is not covered by a live
test against Copilot here; treat it as needing a real-Copilot smoke test
before relying on it in production.
AI_MEMORY_EMBEDDING_PROVIDER=local needs no key and no server at all:
sentence embeddings run in-process (pure-Rust all-MiniLM-L6-v2,
384-dim), with the model fetched once into <data_dir>/models/ under
pinned checksums; see docs/local-embeddings.md.
See docs/install.md#llm-provider-tiers
for env vars and Ollama/OpenRouter/Atlas Cloud/OrcaRouter/Cheaper Inference/API Route/FutureInfra
examples, and
docs/llm-provider-comparison.md
for the empirical model comparison.