Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

Merge remote-tracking branch 'origin/main' into test-557

AkitaOnRails95edbe034e3b
diff --cc CHANGELOG.md
index 4f2fcb26,2e5e74f9..098b90ac
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -8,17 -8,260 +8,271 @@@ and this project adheres to [Semantic V
## [Unreleased]
### Added
+ - `ai-memory handoffs --expire-all --confirm` and `POST /admin/handoffs/expire`
+ clear a stale open-handoff backlog for one scope. The listing added in
+ v1.35.0 made a backlog visible and gave `memory_handoff_cancel` the ids it
+ needs, but clearing one required a call per handoff (#513).
+
+ It deliberately does **not** honour the exemptions the automatic sweep
+ applies. `expire_superseded_auto_handoffs` spares manual handoffs
+ (`from_session_id IS NULL`) and ones whose `cwd` does not match the accepting
+ session — correct for a sweep triggered by an unrelated accept, and the
+ reason a backlog accumulates at all. The leftover backlog is therefore made,
+ by construction, of exactly the handoffs that sweep will never touch, so an
+ operator-driven expiry honouring the same exemptions would clear nothing. A
+ test pins that: applying the sweep's manual-handoff exemption leaves the
+ backlog behind.
+
+ A state change rather than a delete — the summary, provenance and timestamps
+ survive and the handoff simply stops being consumable, which is also why it
+ is recoverable in a way a delete would not be. Owner scoping lives inside the
+ `UPDATE` so expiring another user's baton is a zero-row no-op rather than a
+ read-then-write race, matching `cancel_handoff`. `--older-than-days N` keeps
+ recent batons; `--confirm` is required and the count is audited.
+ - `ai-memory purge-session --session-id <uuid> --confirm` and
+ `POST /admin/purge-session` delete one session by UUID: its row, its
+ observations, the handoffs it authored, its `sessions/<id>.md` page and
+ every superseded version, their embeddings, and its auto-improve runs.
+ `purge-project` was too broad and `delete-page` removed a single wiki path,
+ so an application promising to forget one conversation had nothing to call
+ (#387).
+
+ Scope is enforced structurally rather than trusted: every statement filters
+ on `workspace_id` and `project_id` alongside `session_id`, a session outside
+ the named scope is a `404` with nothing deleted, and derived pages are
+ deleted by id rather than by path — two projects can hold the same
+ `sessions/<uuid>.md`, and deleting by path would take the other one with it.
+ Targets are collected before anything is cut, because
+ `auto_improve_runs.session_id`, `handoffs.from_session_id`,
+ `handoffs.accepted_by_session` and `pages.supersedes` are all
+ `ON DELETE SET NULL`: cutting the session first destroys the pointers needed
+ to find what it produced. Handoffs the session only *accepted* are
+ deliberately kept — that text belongs to the session that authored them. The
+ admission chain runs before any row is touched, so a `reject`-policy webhook
+ can still abort the purge.
+
+ The purge is terminal. Events that produced a session can sit undelivered in
+ a client hook spool for days, and `begin_session` would otherwise recreate
+ the session row when that spool drained — silently undoing a deletion an
+ application had already reported to its user. A `purged_sessions` tombstone
+ (V52) is written in the same transaction as the delete, and ingest refuses to
+ recreate a session that carries one. The tombstone is scoped, so purging an
+ id in one project cannot suppress capture for that id elsewhere.
+
+ The purge is a **logical delete** by default: the session is unreachable
+ through the API and through search, but its bytes stay in free pages of the
+ database file, as with any SQLite delete. `--compact` additionally rebuilds
+ the affected FTS5 indexes and `VACUUM`s. The rebuild is the operative step —
+ measured, an ordinary delete leaves the tokens inside the index segments and
+ `VACUUM` alone does not clear them. `--compact` rewrites the whole database,
+ needs free disk space of about its size, and is **not** forensic erasure: the
+ wiki git history keeps page content in its objects and commit messages, and
+ earlier backups still contain it. `docs/lifecycle-ops.md` states that
+ boundary in a table so it is not inferred from the command name.
+
+ - `ai-memory purge-project --compact` and `{"compact": true}` on
+ `POST /admin/delete-workspace` reclaim the bytes a delete frees, reusing the
+ `Compaction` opt-in that `purge-session` gained in the same cycle (#540). Both
+ responses now report `compacted`.
+
+ Compaction rebuilds **all three** FTS5 indexes rather than the two a session
+ purge needs. A managed workstream's `workstream_events` rows leave through the
+ `projects → workstreams → workstream_events` cascade, and a cascade does not
+ fire the `AFTER DELETE` trigger that would drop them from
+ `workstream_events_fts` — so the tokens survive an ordinary delete, and a
+ `VACUUM` alone does not clear them. A test pins this: removing the third
+ rebuild leaves a purged agent transcript's text in the database file.
+
+ - `ai-memory compact --confirm` and `POST /admin/compact` reclaim free
+ database pages on demand, deleting nothing, and `ai-memory status` now
+ reports the figure that makes the decision possible: database size,
+ reclaimable bytes, and the share of the file they represent (#549).
+
+ Until now compaction was only reachable by *deleting* something — it existed
+ solely as `--compact` on the destructive commands — so a store that had
+ accumulated free pages through a retention sweep or months of superseded page
+ versions had no way to return the space except by purging data worth keeping.
+ There was also no figure anywhere saying whether a `VACUUM` would reclaim
+ anything at all, which made scheduling one guesswork.
+
+ `docs/deploy.md` documents the operational side, including why an
+ unconditional nightly `VACUUM` is the wrong default: it takes an exclusive
+ lock and rewrites the whole database, so every write blocks — and because
+ SQLite reuses free pages, a store in steady use usually has almost nothing to
+ reclaim, paying the full cost for no benefit. The recipe there is conditional
+ on the reported figure and runs in off hours.
+
+ `status` only advises compaction once the backlog is worth the stall (≥20% of
+ the file *and* ≥64 MiB); the raw numbers are always reported.
+
+ ### Fixed
+ - A failed or skipped embed now leaves a durable record. `page_embeddings`
+ stores only successes, and its upsert rewrites `created_at`, so a global
+ `embed --force` erased any signal about which row had been stale. A failure
+ left nothing at all: the inline write path warned and returned success, and
+ the backfill's per-page warnings lived only in container logs that a restart
+ took with them (#528).
+
+ A `page_embed_failures` ledger (V53) records the last attempt that produced
+ no embedding, across all four sites: the inline `write_page` embed, and the
+ backfill's unreadable-page, empty-body and provider-error branches. The
+ empty-body case matters most — a page skipped on every pass was previously
+ indistinguishable from an idle one, which is what made #509 undiagnosable.
+
+ Failure-only by design. A success writes nothing, because a `page_embeddings`
+ row already records it, so the common path pays no extra write and
+ `embed --force` cannot erase the history — it never touches this table.
+ `status` joins the two to report unresolved failures separately from ones a
+ later pass recovered, so a page that failed once and was repaired does not
+ read as an outstanding problem forever. One row per page, cascading with it.
+
+ Designed to @barrosohub's three constraints, who also supplied the
+ measurement that made the gap concrete: of 1,491 embedding rows on their
+ instance, zero predated the `--force` that had overwritten every timestamp.
+
+ - The hook drain no longer stalls the whole spool behind one undeliverable
+ entry. A spooled event carries the URL it was captured against, so a spool
+ can hold entries addressed to a port nothing listens on any more — an old
+ `--bind`, or a server that came back on a different port. The drain stopped
+ at the first such entry, charged it a single retry attempt and returned, so
+ every deliverable event queued behind it was never attempted. With a dead
+ head of queue this is not a delay but silent loss: those events sat until the
+ 7-day spool window discarded them undelivered, while the drain reported the
+ same `0 sent, N queued, 0 dropped` a healthy idle pass reports.
+
+ A transport failure is now distinguished from a server rejection
+ (`PostOutcome::Unreachable` / `BatchOutcome::Unreachable`). When an endpoint
+ cannot be reached the drain charges one attempt to the entry it actually
+ tried, records the address, skips its remaining siblings without charging
+ them for an attempt they never got, and carries on to entries addressed
+ somewhere that answers. A total outage behaves exactly as before — every
+ entry shares the one dead endpoint, so the pass still ends having burnt a
+ single attempt. Reported by @rob-prado, who traced it to the head of their
+ own 7,949-entry spool being 100% dead-port, and by @swhite1122, whose
+ stand-in courier avoided the stall by continuing past failures (#493).
+
+ An entry frozen against a dead **loopback** port is also now retried once
+ against the address in the store's own `config.toml`, so a server that came
+ back on a different port delivers its backlog instead of merely skipping it.
+ Restricted to loopback because a loopback authority can only ever have meant
+ "this machine", making a stale port unambiguous; a remote host is left alone
+ rather than silently re-pointed. The target is read from that file directly
+ and not through the usual config load, which also merges the environment — a
+ drain honouring `AI_MEMORY_SERVER_URL` would send captured events to whatever
+ address happened to be exported into the process that ran it.
+
+ - `purge-project` and `delete-workspace` no longer overstate what they delete
+ (#540). The help text promised "Permanently delete a project and ALL its
+ data", which reads as byte-level removal neither command performed.
+
+ The deletion was never the defect. Rows go and bytes stay in free pages until
+ the file is rewritten — ordinary SQLite behaviour, and the same boundary
+ `purge-session` already documented. Measured on a canary token: delete alone
+ leaves the text on disk, delete + `VACUUM` still leaves it, and only
+ delete + FTS `rebuild` + `VACUUM` removes it. The claim was what needed
+ fixing, so both commands now describe the boundary they actually enforce and
+ offer the same `--compact` opt-in, and `docs/lifecycle-ops.md` covers all
+ three destructive commands in one place instead of documenting the limits of
+ one and overstating the other two.
+
+ - The hook-spool drain can now recover events stranded by a rotated auth token
+ (#542). A spooled envelope freezes the bearer it was captured with, so
+ rotating the token left every already-spooled entry authenticating with the
+ old one and failing `401` forever; OIDC already re-resolved per pass, static
+ tokens did not.
+
+ The drain now retries a rejected entry once with the token it was handed by
+ the hook that spawned it, which is current by construction. Bounded three
+ ways: only after the server has already answered `401`, only for a `Static`
+ entry, and only when the live token actually differs from the one that just
+ failed — so a healthy drain never pays for it and an identical credential is
+ not re-sent.
+
+ The token reaches the drain in the child process's **environment**, never on
+ its command line: an argument would publish the credential to the process
+ table for every local user for as long as the drain runs. A test pins that it
+ stays out of argv. `401` also became a distinct `Unauthorized` outcome —
+ previously indistinguishable from any other refusal, so the drain could not
+ tell a stale credential from a bad event.
+
+ No new credential is stored at rest: the drain is handed a token that already
+ exists rather than reading one from a file ai-memory would have to write and
+ protect.
+
+ - Gave `ai-memory mcp-bridge` a TLS backend so it can reach an `https://` MCP
+ server. `ai-memory-cli` enabled rmcp's `transport-streamable-http-client-reqwest`
+ feature, which pulls reqwest in via `__reqwest = ["dep:reqwest"]` and stops
+ there — no `rustls`, no `native-tls`, no TLS at all. reqwest with no TLS
+ feature refuses any non-`http` scheme, so every bridge run against an HTTPS
+ server URL failed at connect with `invalid URL, scheme is not http`, and the
+ only URL that worked was plaintext `http://` — which is also the transport
+ the bridge attaches its bearer token to. Enabling rmcp's `reqwest` feature
+ the bridge now uses rmcp's `reqwest-tls-no-provider`
+ (`reqwest?/rustls-no-provider`), which supplies the platform certificate
+ verifier without pinning a crypto provider, and installs `ring` — already
+ compiled for this binary via the workspace's reqwest 0.12 — as the process
+ default. rmcp's plain `reqwest` feature would have pinned `aws-lc-rs`,
+ adding a compiled C dependency to every build for a bridge most users never
+ enable. The workspace-client half of #492 landed in #496; this is the MCP
+ transport, which is a second, independent copy of reqwest ([#497]).
+
+ ### Docs
+
+ - `docs/windows.md` gains **Scenario E**, a persistent-server story for native
+ Windows. Scenarios C and D both ended at `ai-memory serve` in a foreground
+ terminal, while Linux got `Restart=on-failure` from the packaged systemd
+ units; WinSW is now documented as the equivalent supervisor (#530).
+
+ It leads with the trap rather than the recipe. A Scheduled Task whose action
+ is a PowerShell script calling `Start-Process` looks correct and is silently
+ killed at the next reboot: `Start-Process` returns immediately, the script
+ exits, and Task Scheduler tears down the job object the still-running server
+ was never detached from. The only symptom is a permanently green
+ `LastTaskResult = 0`, which is why it costs hours. Reported and root-caused
+ with event-log evidence by @gabrielscharb.
+
+ Also documented: `sc create` against `ai-memory.exe` cannot work (no SCM
+ control-code dispatcher, and none is planned — the resiliency belongs in the
+ supervisor, as it does on Linux); the corrected Task Scheduler shape for
+ anyone who wants one anyway; and why the WinSW config must use absolute paths
+ — a service runs as `LocalSystem`, so `%LOCALAPPDATA%` would resolve to the
+ system profile and quietly serve an empty data directory.
+
+ - Scenario E's WinSW steps were executed end to end on real Windows hardware,
+ and the doc now records what they were true for: Windows 11 25H2 (build
+ 26200.9168), WinSW v2.12.0, ai-memory v1.38.0, from an elevated PowerShell
+ session against a throwaway service, port and data directory. Install, start,
+ `LocalSystem` + `Automatic`, an MCP `initialize` answered over the bound
+ port, the absolute `--data-dir` honoured, crash recovery ~8s after
+ force-killing the wrapped process, and a clean stop/uninstall all held;
+ the service was configured with `StartType Automatic`, but boot-time
+ startup was not independently exercised (#530).
+ - Added `ai-memory rename-workstream`, a checkout-local rename for managed
+ workstreams selectable by current name or by the stable id `workstreams`
+ prints. Names were fixed at `run --new` time and had no correction path, so
+ a typo outlived the work it labelled. The rename is metadata only: the
+ ledger, managed runs, and linked native sessions all key on the workstream
+ id, and `selected_at` and `updated_at` are deliberately left untouched, so
+ neither the listing order nor the workstream a bare `ai-memory run` resumes
+ moves as a side effect of relabelling. The destination is validated exactly
+ like a `--new` name and refused with a named conflict when another
+ workstream in the same checkout already holds it; renaming a workstream to
+ the name it already has writes nothing and is not an error. Both selectors
+ repeat the checkout predicate, so an id belonging to another workspace,
+ project, or worktree reads as absent rather than renamable. The Docker
+ wrapper routes the command through its native host client, since repository
+ identity is a host resource.
+- `llm_reasoning_effort` / `AI_MEMORY_LLM_REASONING_EFFORT` is a typed
+ enum (`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`,
+ `ultra`, `persistent`); unknown values fail at config load. Unset keeps
+ the model default. Each chat provider maps the value to its native
+ field: OpenAI `reasoning_effort`, OpenRouter `reasoning.effort`,
+ xAI Grok `reasoning_effort`, Anthropic `output_config.effort` (plus
+ adaptive/disabled thinking on models that accept it), and Codex
+ Responses `reasoning.effort`. Gemini and Copilot ignore the key.
+ Host-unsupported values are clamped (`ultra`/`persistent` → `max` on
+ OpenAI/OpenRouter/Codex; Anthropic `xhigh`/`max` per model; `none`
+ does not send `thinking: disabled` on always-on Claude models).
## [1.38.0] - 2026-08-30
diff --cc Cargo.lock
index 0c1661e4,0d4aefc6..bf138494
--- a/Cargo.lock
+++ b/Cargo.lock
@@@ -56,7 -56,7 +56,8 @@@ dependencies =
"jsonc-parser",
"reqwest 0.12.28",
"rmcp",
+ "rstest",
+ "rustls",
"secrecy",
"serde",
"serde_json",
@@@ -2664,6 -2650,33 +2730,33 @@@ dependencies =
"zeroize",
]
+ [[package]]
+ name = "rustls-platform-verifier"
+ version = "0.7.0"
+ source = "registry+https://github.com/rust-lang/crates.io-index"
+ checksum = "26d1e2536ce4f35f4846aa13bff16bd0ff40157cdb14cc056c7b14ba41233ba0"
+ dependencies = [
+ "core-foundation",
+ "core-foundation-sys",
+ "jni",
+ "log",
+ "once_cell",
+ "rustls",
+ "rustls-native-certs",
+ "rustls-platform-verifier-android",
+ "rustls-webpki",
+ "security-framework",
+ "security-framework-sys",
+ "webpki-root-certs",
- "windows-sys 0.52.0",
++ "windows-sys 0.61.2",
+ ]
+
+ [[package]]
+ name = "rustls-platform-verifier-android"
+ version = "0.1.1"
+ source = "registry+https://github.com/rust-lang/crates.io-index"
+ checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f"
+
[[package]]
name = "rustls-webpki"
version = "0.103.13"