Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

Merge main into release/2.4: rmcp 2.x security bump + Windows/sanitize/wiki fixes batch

AkitaOnRailse83ef8a4d5be
diff --cc CHANGELOG.md
index 96d73fc5,9c59817a..b04e9a1e
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,220 -7,77 +7,293 @@@ and this project adheres to [Semantic V
## [Unreleased]
+ ### Security
+ - Bumped `rmcp` to 2.x (2.2.0), resolving three MCP transport advisories:
+ GHSA-9pj6-vhgr-3mwh (unauthenticated Streamable-HTTP session-table leak /
+ DoS), GHSA-33f5-2c5q-wgwj (missing OAuth resource-field validation), and
+ GHSA-9g45-5xwm-f3wc (custom headers leaking to cross-origin redirect
+ targets). Behavior-preserving: the only source change is the
+ `rmcp::model::Content` → `ContentBlock` rename (imported under the prior
+ name), the feature set is unchanged, and the 23-tool MCP surface is
+ unaffected. (#794)
+
+### Added
+- LLM "dream" pass — cross-session rewrite/merge of cold clusters, scheduled on
+ idle (design-memory-aging.md buckets B2/B3/B4). Where A3 collapses
+ near-duplicate cold clusters *extractively* (zero-LLM, keep-token union), the
+ dream pass hands each cold cluster to the configured provider to be rewritten
+ into ONE coherent page. It is **opt-in LLM, OFF by default, and gated on an R2
+ number before it may default on**: it runs only when the new `[dream] enabled`
+ flag is set AND a provider AND an embedder are configured — a provider-less
+ store keeps the zero-LLM A3 path untouched (invariant #13). **It never deletes
+ a source** (invariant #16): the highest-retention member is rewritten and every
+ merged-away member is *superseded* with a merge-note stub pointing at it, so the
+ full pre-merge body stays reachable via the supersession chain + git and
+ `restore-page` recovers it; `page_evidence` (`reconsolidation` +
+ `b2_dream:<id>`) records which members fed each merge (the hallucinated-merge
+ guard). The rewrite routes through the existing gated apply path
+ (`preflight_admission(Consolidate)` → `Wiki::apply_batch`, single-writer actor,
+ invariant #2) with **`dry_run` first** (a dry run returns the plan and calls
+ neither the LLM nor the writer), and uses **JSON-schema structured output only**
+ (invariant #7). Scheduling (B3) runs the pass only after a configurable idle
+ window with no client activity and **cancels it the moment the operator
+ returns** (a cheap cancellation flag polled between clusters), bounded to a
+ capped number of clusters per run (invariant #5). Work is ordered
+ **surprisal-first** (B4): most-novel clusters — those farthest from the nearest
+ existing page — first. Every run returns an observable `DreamReport` (clusters
+ considered, merged, pages rewritten/superseded, skipped, cancelled) so a bad
+ run is never silent. New `[dream]` config section; no new migration (reuses
+ `page_evidence` + supersession); no new MCP tool (still 23) (#816).
+- Belief-strength confidence over the `page_evidence` substrate
+ (design-memory-aging.md bucket B1 / design-hindsight-borrowings.md §3): a
+ read-time, **zero-LLM** `confidence` derived per page version from its
+ evidence — distinct supporting sessions (breadth, not raw count), recency of
+ the newest sighting, and live `contradicts` count — bounded to
+ `[0.0, 0.95]`. It is **exposed inertly** everywhere it helps diagnosis:
+ `memory_query(explain=true)` now reports `confidence` and `evidence_count`
+ per hit (and `belief_factor` when folding is on), and `memory_status` reports
+ the project's `evidence_rows` count — none of which changes ranking. It can
+ optionally be **folded into ranking authority** as one more bounded factor
+ inside the existing `[0.55, 1.50]` clamp (never a new multiplier tower) via
+ the new `[retrieval] belief_authority_weight` config key, which **defaults to
+ `0.0` (OFF)** so upgrades rank byte-identically. The anti-entrenchment guards
+ are baked in: breadth weighting by distinct sessions, recency shading, a
+ hard confidence cap, and — the caller-side guard for invariant #16 — **a
+ supersession always wins regardless of evidence** (a superseded version's
+ stale evidence never boosts it, and confidence never gates whether a write or
+ correction takes). Turning the authority factor on is **gated on a positive
+ R2 delta** (retrieval-triple / QA), not yet performed (#815).
+- Zero-LLM contradiction detection surfaced through `memory_lint`
+ (design-memory-aging.md bucket A5): the lint pass now flags likely-conflicting
+ pages by cosine-similarity band. Cold knowledge pages (semantic / procedural)
+ whose already-stored embeddings sit in the **0.4–0.75 cosine-similarity band** —
+ "same topic, but not a near-duplicate", the shape of a likely contradiction
+ (a pair ≥ 0.75 is A3 dedup territory; < 0.4 is unrelated) — get an advisory
+ `contradiction` lint finding naming both pages, with timestamp-based
+ resolution advice (the newer page supersedes on a timestamp basis; reconcile).
+ It is fully **zero generative LLM**: it reads only existing embeddings and
+ cosine (invariant #13), so with no embedder configured — or no embeddings for
+ the configured `(provider, model, dim)` triple — it is a clean no-op, not an
+ error, and never a provider call. It runs on the user-invoked `memory_lint`
+ (MCP and admin) and is **advisory-only and non-destructive**: it emits a
+ finding and never deletes, edits, or supersedes a page (invariant #16), and
+ never persists an edge (the `links` table's `contradicts` edges are
+ body-derived and rewritten on every page write, so a programmatic edge would
+ be silently wiped) — hence **no new migration**, and no new MCP tool (still
+ 23). The scan is bounded: one embeddings load over the already-bounded cold
+ set, capped page and finding counts, deterministic ordering (invariant #2)
+ (#814).
+- Cold-cluster dedup of near-duplicate episodic pages (design-memory-aging.md
+ bucket A3): the forget-sweep can now cluster near-duplicate cold episodic
+ pages by embedding (cosine-distance DBSCAN with an adaptive k-distance eps,
+ `minPts = 2`) and collapse each cluster to one survivor — the highest-retention
+ member, its body the *extractive union* of the cluster's keep-tokens, so every
+ member's durable facts survive — superseding the other members with a merge
+ note that points at the survivor. It runs only over the bounded cold-episodic
+ candidate set the sweep already materialises (never O(N²) over the whole
+ corpus), is **opt-in and off by default** via `[decay] dedup_cold_clusters`
+ (a `false` default), and fully **zero generative LLM**: it reads only
+ already-stored embeddings, so with no embedder configured — or no embeddings
+ for the configured `(provider, model, dim)` triple — it is a clean no-op, not
+ an error. The eps is clamped to a conservative ceiling (`[decay] dedup_max_eps`,
+ default cosine distance ≈ 0.15) so it errs toward NOT merging. **Non-destructive
+ and reversible**: no source is ever hard-deleted — every merged-away member
+ stays reachable via the supersession chain and git history and is recoverable
+ with `restore-page` (invariant #16) — and the merge provenance is recorded in
+ `page_evidence`. Every run reports its collapses in the `SweepReport`. Reuses
+ existing tables: **no new migration**, and no new MCP tool (still 23). Ships
+ opt-in/off; the R2 recall no-regression proof is the gate before any future
+ default-on (#812).
+- Entropy / boilerplate pre-filter before consolidation (design-memory-aging.md
+ bucket A4): a pure, zero-LLM Shannon-entropy + boilerplate gate that skips
+ low-information session pages (near-empty, whitespace, single-character, or
+ highly-repetitive boilerplate) from the cross-session experience consolidation
+ pass *before* they reach the LLM prompt, the eval gate, or `apply_batch`.
+ It is **advisory and non-destructive** — a skipped page is not consolidated,
+ never deleted (invariant #16) — and **opt-in / off by default** via
+ `[auto_improve.scheduler.experience_entropy_filter]` (a `false` default with
+ conservative, validated thresholds tuned so a terse-but-informative note with
+ a file path and an error code is KEPT), so an upgrade changes no consolidation
+ output until an operator opts in. Every run surfaces the skip count in the
+ experience report warnings. No schema change and no new MCP tool (still 23)
+ (#812).
+- Extractive tier-down of cold episodic pages (design-memory-aging.md bucket
+ A2): instead of evicting a cold episodic page, the forget-sweep can now
+ *compact* it — keeping the L0 frontmatter `abstract:`, an L1 first-paragraph
+ summary, and an L2 regex-mined keep-token set (file paths, URLs, inline-code
+ spans, error codes, `UPPER_SNAKE` constants and long identifiers), and
+ dropping the prose body. Tier-down beats eviction because the durable facts
+ survive while the expensive, low-signal prose does not. It is **opt-in and
+ off by default** via `[decay] compact_cold_episodic` (a `false` default, so an
+ upgrade changes nothing until an operator opts in), fully zero-LLM (regex
+ only), and **reversible and non-destructive**: the rewrite goes through the
+ wiki layer, so the full pre-compaction body stays reachable in git history and
+ the supersession chain and is recoverable with `restore-page`. A new `V65`
+ migration adds a nullable `pages.compacted_at` marker (additive `ADD COLUMN`,
+ no backfill; populated lazily by the sweep from a `compacted: true` frontmatter
+ mirror) so the sweep and the curator tell a deliberately-short compacted page
+ from a cold one — a compacted page is never re-compacted, re-evicted, or
+ re-reported as cold. Only unpinned episodic pages compact; pinned/semantic/
+ procedural pages are never touched. Every run reports what it compacted in the
+ `SweepReport`. Ships opt-in/off; the R2 recall no-regression proof is the gate
+ before any future default-on. No new MCP tool (still 23) (#808).
+- Per-tier retention half-life curves (design-memory-aging.md bucket A1): the
+ forget-sweep's decay rate can now be tuned per memory tier via an opt-in
+ `[decay.half_life_days]` config table, replacing the single global λ. Each
+ key (`working` / `episodic` / `semantic` / `procedural`) is a half-life in
+ *days*, converted internally to `λ = ln(2) / days`, so an operator can keep
+ episodic session history longer and working-tier scratch shorter (the
+ mcp-memory-service 365/180/90/30 shape). An omitted key falls back to the
+ scalar `[decay] lambda`, so the default (no table) is byte-identical to the
+ previous single-λ behaviour — an upgrade changes no score and mass-evicts
+ nothing on the first post-upgrade sweep. Pure math + config: no new column,
+ no migration, and no new MCP tool (still 23) (#807).
+- Access reinforcement on the remaining read paths (design-memory-aging.md
+ bucket C1): `memory_read_page` (a direct by-path/by-query read), its
+ `include_related` link-graph walk (the walked neighbours, not just the seed),
+ and `memory_explore` (the pages it surfaces — rules, slots, recent, pinned,
+ settled) now bump `access_count` + `last_accessed_at` exactly as
+ `memory_query` and `memory_recent` already do. A page a human opens directly,
+ or one the graph surfaces, is *used* and now resists decay like a search hit.
+ Reuses the sanctioned reinforcement path: fire-and-forget on the single-writer
+ actor, throttled to ≤1 per (page, operator) per minute, and FTS-exempt. It is
+ strictly additive — reinforcement only raises retention scores, never blocks,
+ never touches the response payloads, and adds no new MCP tool (still 23)
+ (#798).
+- Reasoning tier on the LLM synthesis paths: an opt-in `reasoning` argument on
+ `memory_query` (its `answer` path) and `memory_explore` (borrowed from
+ Honcho's reasoning-effort ladder; targets the 2.4 line). The knob is a schema
+ enum `minimal` (default) / `low` / `medium` / `high` / `max`; an unknown value
+ is rejected. Because the provider-neutral `ChatRequest` carries no per-request
+ reasoning/effort field (the provider-level `reasoning_effort` is fixed at
+ construction from config), the tier maps honestly to a per-tier max-token
+ budget scaled off each path's base budget (answer 2 000, explore 16 000):
+ `minimal` = 1x, `low` = 1.5x, `medium` = 2x, `high` = 3x, `max` = 4x — a
+ higher tier gives the model more room to reason before its output is
+ truncated. It only tunes the answer path: `reasoning` is inert unless the LLM
+ path actually runs (`answer: true` with a provider, or `memory_explore` with a
+ provider), so the zero-LLM default path is untouched. Omitting `reasoning`, or
+ passing `minimal`, is byte-identical to before. No new MCP tool (still 23)
+ (#783).
+- Dialectic answer on `memory_query`: an opt-in, off-by-default `answer`
+ argument (borrowed from Honcho's dialectic endpoint; targets the 2.4 line).
+ When `answer: true` AND the server has an LLM provider configured, the query
+ synthesizes a concise, cited natural-language answer over the top retrieved
+ hits and attaches it as `answer: { text, citations }`, where `citations` are
+ the page paths the answer drew from (JSON-schema structured output, grounded
+ strictly in the retrieved snippets). When `answer: true` but no provider is
+ configured, the normal hits are returned plus a short `answer_unavailable`
+ note rather than an error. With `answer` omitted/`false` (the default), no LLM
+ provider is accessed and the response is byte-identical to before, so the
+ zero-LLM default path is untouched. Applies to the normal single-project /
+ `scopes` search; `global` and `as_of` queries ignore it. Honest caveat: the
+ feature is new and its answer quality is not yet eval-validated — treat the
+ synthesized answer as a convenience over the same hits and still open the
+ cited pages before acting (#782).
+- "Pin before search": `memory_query` gained an opt-in `pin_first` argument and
+ `memory_briefing` now carries a bounded `pinned` list (default off/absent;
+ targets the 2.4 line). Pinned pages previously earned only a small post-RRF
+ authority bump; they were never surfaced *ahead of* the search, and the
+ briefing never listed them by the `pinned` column. With `pin_first: true`, a
+ single-project `memory_query` prepends the project's bounded pinned latest
+ pages (newest first, cap 10) ahead of the fused hits, deduped by page id so a
+ pinned page that also matches the query appears once (marked `pinned: true`),
+ and re-truncates to the requested limit; `scopes`, `global`, and `as_of`
+ queries ignore it. A project-scoped `memory_briefing` snapshot now includes a
+ bounded `pinned` list of pinned latest pages (distinct from the `_slots/`
+ path-prefixed `slots`) so SessionStart hot-context can show standing context.
+ Both are backed by the new `ReaderPool::list_pinned_pages`; default off/empty
+ is byte-identical to the previous query ordering and briefing shape (#780).
+- `memory_read_page` gained an opt-in related-pages graph walk (default false;
+ targets the 2.4 line). Passing `include_related: true` adds a `related` array
+ of the pages reachable from the read page through the link graph — a bounded
+ breadth-first walk that reuses the single-hop link primitive per node,
+ following both outgoing links and incoming back-links out to `related_depth`
+ hops (default 1, hard-capped at 3). Each entry carries its
+ path/title/kind/workspace/project plus the hop `depth` and edge `direction`
+ (`link`/`backlink`) it was reached by; the walk is cross-project aware,
+ dedup- and cycle-safe via a global visited set, and bounded by a total-node
+ cap. Default-off behaviour is byte-identical to the previous single-page
+ response (no `related` field) (#775).
+- `memory_query` gained an opt-in `include_superseded` argument (default false;
+ targets the 2.4 line). When set, project and explicit-scope searches also
+ return superseded (older) page versions across the FTS/entity/vector/graph
+ streams, each hit labelled `superseded: true` so callers can tell historical
+ versions from the current one; the current version is never marked. Default-off
+ behaviour is byte-identical to the previous latest-only retrieval, and
+ `global=true` and `as_of` time-travel are unaffected (#773).
+- `memory_status` now reports which project answered: a `scope` object with
+ `workspace`, `project`, and `resolved_by` (`explicit`, `session`,
+ `shared_slot`, `startup_seed`, `default`, or `default_after_mismatch`). An
+ unscoped call from a static MCP client, whose transport session id is not a
+ lifecycle-hook session id, returned plausible counts for a project it never
+ named, with nothing in the response to question them; `resolved_by` now makes
+ that visible. The server also logs a warning whenever an unscoped MCP read is
+ resolved by the startup seed or by the default after a session mismatch,
+ rather than by the caller's own hook session (#757, #774).
++||||||| 353841d9
++
+ ### Docs
+ - `docs/llm-providers.md` now covers the `opencode` LLM provider, which has
+ shipped since 1.x but was missing from the recommended-defaults table:
+ `OPENCODE_API_KEY` as the only credential, Go as the default endpoint, Zen
+ via `AI_MEMORY_LLM_BASE_URL`, the built-in default model, per-catalogue
+ model ids, and which model goes through the Responses endpoint (#763).
+
+ ### Fixed
+ - The Windows Docker wrapper (`bin/ai-memory.ps1`) now forwards the same
+ provider credentials and host-config env vars as the POSIX wrapper into the
+ helper container. A host-exported `GEMINI_API_KEY` / `GOOGLE_API_KEY`,
+ Copilot token, `OPENCODE_API_KEY`, `CLAUDE_CONFIG_DIR`, or
+ `AI_MEMORY_WORKSTREAM_ID` previously never reached `Config::load`, so
+ `llm-test`, Copilot auth, OpenCode, and a relocated Claude config all
+ reported "not configured" on native Windows Docker Desktop even though the
+ same export worked through `bin/ai-memory`. The POSIX wrapper also now
+ forwards `OPENCODE_API_KEY`. (#803)
+ - `Config::load` now treats Windows `%USERPROFILE%` (then `dirs::home_dir`) as
+ the operator home when `AI_MEMORY_HOME` and `$HOME` are unset. Native
+ Windows often has no `HOME`, so the #103 catch-all guard — skip a stored
+ `repo_path` equal to the user profile so it cannot prefix-match every
+ project beneath it — was inert there and a home-directory project could
+ swallow unrelated cwds. (#804)
+ - Automatic handoff selection and cwd-prefix project matching now treat
+ Windows drive-letter and UNC paths as case-insensitive. A Linux server
+ (the Docker Desktop helper) comparing host cwds from Explorer, Git, and
+ PowerShell previously required a byte-exact match, so `C:\Users\…\repo`
+ vs `c:\users\…\repo` missed the pending auto-handoff and could mint a
+ fragment project. Unix paths stay case-sensitive. (#806)
+ - Fixed `Ctrl+C` at the native-session chooser leaving the launcher alive and
+ renewing its workstream lease. Cancelling now releases the acquired run and
+ exits without waiting for Enter or linking a native session (#795).
+ - The privacy strip now redacts Windows credential paths (`C:\Users\…\.ssh`,
+ `.aws`, `.kube`, `.gnupg`, `.config\gcloud`). The previous patterns required
+ a POSIX `/` separator, so a captured tool result that echoed a native
+ Windows path stored the profile directory and key file name verbatim. (#805)
+ - The privacy strip now redacts secrets written in JSON. The quote before a
+ value put it outside the value character class, so `{"db_password":"..."}`
+ was stored verbatim while the identical YAML form was redacted, and JSON is
+ the shape most captured tool payloads arrive in. The same rule now also
+ accepts an auth scheme word before the value, so
+ `Authorization: Basic <base64>` (which carries `user:password`) is redacted
+ like the `Bearer` form already was, and covers two unprefixed names the
+ generic env rule missed: Azure `AccountKey=` and npm `_authToken=`. (#800)
+ - Terminal escape sequences, NUL and bidi override characters are stripped
+ from captured text instead of being stored. Page bodies and observations are
+ replayed to a terminal by `ai-memory read-page` and `ai-memory search`, where
+ an escape rewrites the screen or the window title and a bidi override
+ reverses what the reader sees; a NUL additionally made the markdown file
+ binary, costing it `grep` and git diffs. Tabs, newlines and carriage returns
+ are kept. (#800)
+ - A page write is refused when another live page in the same project differs
+ from it only by case or Unicode normalization. Such a pair is one file on
+ macOS (APFS) and Windows (NTFS), so creating the second silently overwrote
+ the first page's file while the index kept both rows, so reads for either path
+ then returned the survivor's body, and the wiki watcher superseded the
+ overwritten row, losing the original content from disk and index alike. The
+ refusal names both paths, applies on every platform (the wiki is synced
+ between them), and leaves supersedes of an existing path untouched.
+ `reindex` skips such a pair instead of failing the whole rebuild, reports the
+ count, and logs each one. (#799)
## [2.3.2] - 2026-09-20