Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
docs: regenerate AGENTS.md contributor guide via /init
AkitaOnRails5ee5f1ec34c2
diff --git a/AGENTS.md b/AGENTS.mdindex 6785f7ee..5a9aa28d 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -68,118 +68,294 @@ Both are idempotent: re-runs replace the block delimited by the ai-memorystart/end HTML-comment markers, without disturbing the rest of the file.<!-- ai-memory:end -->-## Canonical Agent Instructions--- `AGENTS.md` is the single canonical instruction file for this repository,- including Claude Code. Keep `CLAUDE.md` as a short pointer to this file;- do not duplicate project rules there.-- Read `docs/ARCHITECTURE.md` when you need the current operational map, and- `docs/design-decisions.md` when you need historical rationale.-- Read `docs/auto-improvement-loop.md` before changing auto-improvement review,- pending proposal storage, approval flows, or prompt routing for learning- review.--## Project Summary--ai-memory is a self-contained Rust binary providing long-term memory for AI-coding agents over MCP and lifecycle hooks. Markdown-in-git is the wiki source-of truth; SQLite is the derived index for search, sessions, observations,-handoffs, users, audit, and embeddings. Capture is automatic through hooks;-durable retrieval follows the Karpathy-style LLM Wiki pattern.--## Stack And Layout--- Runtime: Rust edition 2024, `tokio`, workspace resolver 3.-- MCP/HTTP: `rmcp` plus `axum` for MCP HTTP, hooks, admin, and web routes.-- Store: `rusqlite`, `refinery` migrations, FTS5, sqlite-vec-compatible- embeddings, one SQLite file, one writer actor, read pool.-- Wiki: markdown on disk, `notify-debouncer-full` watcher, atomic writes,- `git2` checkpoints.-- LLM: typed providers in `ai-memory-llm`; provider-specific behavior belongs- there, not in CLI/admin handlers.-- Config: `figment`; runtime behavior resolves config once and threads typed- settings through call sites.-- Crates: `ai-memory-core`, `ai-memory-store`, `ai-memory-wiki`,- `ai-memory-mcp`, `ai-memory-hooks`, `ai-memory-llm`,- `ai-memory-consolidate`, `ai-memory-cli`, plus `ai-memory-web` and `evals/`.--## Workflow Rules--- Keep changes small and scoped. Do not start adjacent feature work unless the- current task requires it.-- No dead code or half-built public surface. If something is future work,- document it in docs/design notes rather than shipping unreachable stubs.-- Document constraints and incidents, not line-by-line mechanics.-- Add focused regression tests for bug fixes and behavior changes.--## Project Maintenance Rules--- Keep `CLAUDE.md` as a pointer to `AGENTS.md`; this avoids split-brain- instructions between Claude Code and AGENTS-aware harnesses.-- Any change affecting user-visible behavior, installation, supported- platforms, supported agents, providers, deployment, env/config, or public- tool/admin surfaces must update `CHANGELOG.md` and the relevant README/docs- references in the same commit.-- Do not bump crate/package versions, minor versions, or cut release tags- automatically. Ask the user before any version bump or release tag; prefer no- version change unless the user explicitly approves it.-- When asked to evaluate a PR, report the pros, cons, and recommended fix,- then ask the user for approval before merging or pushing PR changes. Do not- merge PRs during evaluation unless the user explicitly approves that action.-- When the MCP tool surface changes, update `MEMORY_INSTRUCTIONS`,- `ai_memory_core::SNIPPET_BODY`, README/docs tool references, and regression- tests that assert every tool appears in both prompt surfaces.--## Rust Engineering Rules--- Prefer small, behavior-preserving changes. Do not add compatibility- branches, new abstractions, or new public surface unless a shipped caller,- persisted data, or explicit requirement needs them.-- Optimize the real bottleneck class first: algorithm, query shape, batching,- allocation count, IO boundaries, and container choice. Avoid clever- micro-optimizations without evidence.-- Keep SQLite writes behind the single writer actor. For hot paths, batch work- into one command/transaction instead of spawning many writer messages or- opening per-row transactions.-- Avoid N+1 store reads. Prefer reader methods that return the data shape the- caller actually needs, and use cached/prepared statements for repeated- queries.-- Keep hook ingestion fire-and-forget and bounded. Do not introduce unbounded- `tokio::spawn` fan-out, unbounded queues, or synchronous agent-facing waits.-- Keep CLI commands thin: parse arguments, resolve config once, call typed- library functions, render output. Provider-specific behavior belongs in the- provider module, not in command handlers.-- Treat typed boundaries as load-bearing: IDs, `PagePath`, `AgentKind`,- sanitization, workspace/project resolution, auth capability, and provider- dialects should be parsed or normalized once and reused.-- Preserve workspace/project isolation through the shared scope framework.- New MCP/admin/web routes must use `ai_memory_store::ScopeResolver` or its- explicit helpers (`lookup_existing_scope`, `create_explicit_scope`,- `resolve_many_existing_scopes`) instead of hand-rolled workspace/project- lookup chains. Read, search, embed, retention, and destructive paths must use- no-create lookups and fail closed on partial or missing scope; only explicit- write/create paths may create workspaces or projects. PRs touching scope- resolution need table-driven tests for partial scope, missing explicit scope,- active-project precedence, and cross-workspace isolation.-- Preserve auth boundaries through `AuthLevel::authorize(Capability::...)`.- Do not open-code username comparisons or ad hoc root checks in handlers. In- multi-user mode, every `/admin/*` route is root-only; DB-user tokens are for- normal MCP/API read/write attribution and must not bypass admin gates or- admission webhooks. PRs touching permissions need tests for root, DB-user,- and anonymous behavior.-- Treat markdown as the source of truth and SQLite as the derived index. Wiki- mutations must go through `Wiki::write_page`, `Wiki::apply_batch`, or the- existing destructive helpers so sanitization, admission, attribution,- rollback, and index updates stay together. Do not write wiki files directly- from handlers; add recovery/rollback tests for any new disk+SQL mutation.+# AGENTS.md — ai-memory contributor guide++This file is the single canonical instruction file for AI coding agents+working in this repository (Claude Code, Codex, OpenCode, Cursor, Gemini+CLI, Kimi Code, and other AGENTS-aware harnesses). `CLAUDE.md` is only a+short pointer here — do not duplicate rules into it.++## Project overview++ai-memory is a self-contained Rust binary that gives AI coding agents+long-term, cross-session memory over MCP and lifecycle hooks. Quit Claude+Code mid-task, open Codex in the same directory, and continue without+re-explaining context.++Core design:++- **Markdown-in-git is the source of truth.** The wiki lives at+ `<data_dir>/wiki/`, is editable by hand, and every consolidation pass+ produces a git commit (via `git2`).+- **SQLite is the derived index** (`<data_dir>/db/memory.sqlite`, WAL+ mode): FTS5 search, sessions, observations, handoffs, users, audit log,+ embeddings, and the optional managed-workstream ledger. One writer+ actor owns the writer connection; reads go through a read-only pool.+- **Capture is automatic** through agent lifecycle hooks that POST+ sanitized, bounded observations to the server (`/hook`). The server+ compiles session observations into durable wiki pages (Karpathy-style+ "compile, not retrieve").+- **Retrieval** is FTS5 + link-neighbor RRF, with optional vector RRF+ when an embedding provider is configured, plus bounded raw-observation+ fallback.+- **LLM is opt-in.** Zero-LLM mode still captures, searches (FTS5), and+ writes rule-based summaries. Providers (Anthropic, OpenAI, OpenAI/Codex+ OAuth, GitHub Copilot, Gemini, OpenAI-compatible endpoints) enable+ consolidation, lint, and the auto-improvement loop.+- **Per-project isolation by construction**: every row and page is keyed+ by `(workspace_id, project_id, path)`, resolved from the caller's cwd,+ a `.ai-memory.toml` marker file, or explicit scope arguments.++The full operational map is [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md);+historical rationale is in [`docs/design-decisions.md`](docs/design-decisions.md).+Read [`docs/auto-improvement-loop.md`](docs/auto-improvement-loop.md) before+changing auto-improvement review, pending proposal storage, approval flows,+or prompt routing for learning review.++## Technology stack++- **Rust edition 2024, toolchain 1.95** (pinned in `rust-toolchain.toml`),+ workspace resolver 3. Current workspace version: 1.17.3.+- **Async runtime:** `tokio` (full features).+- **MCP/HTTP:** `rmcp` 1.7 (server SDK) + `axum` 0.8 for MCP HTTP, hooks,+ admin, `/api/v1`, and the built-in `/web` UI; `tower` / `tower-http`.+- **Store:** `rusqlite` (bundled SQLite, backup API), `refinery`+ migrations, FTS5, `parking_lot`.+- **Wiki:** atomic markdown writes (tmp + rename + fsync),+ `notify-debouncer-full` watcher, `git2` with vendored libgit2+ checkpoints.+- **LLM:** typed providers in `ai-memory-llm` behind `LlmProvider` /+ `Embedder` traits; `reqwest` (rustls) for provider HTTP.+- **Config:** `figment` (TOML + `AI_MEMORY_*` env); CLI via `clap` 4+ derive; `clap_complete` for shell completions.+- **Auth/secrets:** `secrecy`, `subtle` (constant-time compare),+ `getrandom`, `base64`, `sha2`.+- **Time/IDs:** `jiff`, `uuid` (v4/v5/v7).+- The build is self-contained (bundled SQLite, vendored libgit2); only a+ standard C toolchain is needed.++Workspace lints (`Cargo.toml`): `unsafe_code = "forbid"`,+`missing_docs = "warn"`, all default clippy lints at warn. Release+profile: thin LTO, `codegen-units = 1`, stripped symbols.++## Repository layout++```+crates/+├── ai-memory-core/ domain types, errors, ids. NO IO.+├── ai-memory-store/ SQLite + writer actor + reader pool + decay math.+├── ai-memory-wiki/ atomic markdown writes, file watcher, git.+├── ai-memory-mcp/ rmcp transport + tool router + admin routes.+├── ai-memory-hooks/ payload schemas, sanitizer, /hook ingress.+├── ai-memory-llm/ provider auth boundary + LlmProvider / Embedder traits.+├── ai-memory-consolidate/ Karpathy ingest / lint / sweep / auto-improve pipeline.+├── ai-memory-web/ read-only /web UI and /api/v1 JSON routes.+├── ai-memory-workstream/ read-only native transcript + launch adapters (`ai-memory run`).+└── ai-memory-cli/ `ai-memory` binary entry point + thin HTTP subcommands.+evals/ live A/B harness; workspace member, not shipped.+companions/ai-memory-importer/ standalone OMC wiki importer; NOT a root+ workspace member — build/test it with+ `--manifest-path companions/ai-memory-importer/Cargo.toml`.+hooks/ per-agent lifecycle hook bundles (shell/native).+bin/ host wrapper scripts (`ai-memory`, `deploy`, `release`).+docker/ Dockerfile, compose files, TLS proxy templates.+packaging/ AUR/systemd/sysusers/tmpfiles native packaging assets.+scripts/ packaging checks, hook installer, acceptance scripts.+tests/ e2e smoke (`e2e/handoff_smoke.sh`), hook shell tests, fixtures.+docs/ architecture, design decisions, install/deploy/usage guides.+```++Each crate has a single responsibility and exposes a typed API; no+circular dependencies. Inter-crate boundaries enforce the invariants+below.++## Build and test commands++Rust 1.95 is required (the pinned toolchain installs automatically via+rustup). Before claiming any Rust change is ready, run the full local+gate — the same gates CI (`.github/workflows/ci.yml`) and `bin/release`+enforce:++```bash+cargo fmt --all -- --check # formatting+git diff --check # whitespace+TAILWIND_SKIP=1 cargo test --workspace # tests+TAILWIND_SKIP=1 cargo clippy --workspace --all-targets -- -D warnings+cargo deny check # dependency policy (if installed)+```++- `TAILWIND_SKIP=1` skips the Tailwind asset build in `ai-memory-web`'s+ build script; use it for local test/clippy runs. CI's Linux/macOS test+ job runs the full build without it.+- Run the companion importer separately:+ `cargo test --manifest-path companions/ai-memory-importer/Cargo.toml`+ (plus fmt/clippy on the same manifest). Root `--workspace` commands do+ not cover it.+- Useful focused runs: `cargo test -p ai-memory-store`, etc.+- Shell-level checks: `tests/hooks/test_lib.sh`,+ `tests/e2e/handoff_smoke.sh`, `scripts/check-native-packaging.sh`.+- CI additionally runs `cargo build --release --bin ai-memory` on+ Linux/macOS, a Docker image smoke test, `cargo audit` (with the ignores+ listed in `ci.yml`), gitleaks secret scanning, and a non-gating Windows+ test job.++## Code style guidelines++- Match the surrounding file's conventions: naming, comment density,+ module structure. Comments explain *why*, never restate *what*.+- Small, scoped, behavior-preserving changes. No adjacent feature work,+ no opportunistic refactors, no speculative abstractions or new public+ surface without a shipped caller, persisted data, or an explicit+ requirement.+- No dead code or half-built public surface. Future work is documented in+ `docs/` design notes, not shipped as unreachable stubs.- Prefer explicit fallbacks over `unwrap`, `expect`, or `unreachable!` in- runtime paths. Panics are acceptable in tests only.-- Do not use `unsafe` for performance work in this project unless profiling- proves it is necessary and the safety argument is documented in the code.-- Add focused regression tests for bug fixes and behavior changes. For- filesystem tests, use temp dirs or injected roots; never depend on the real- user home directory being writable.-- Run the full local gate before claiming a Rust change is ready:- `cargo fmt --check`, `git diff --check`,- `TAILWIND_SKIP=1 cargo test --workspace`, and- `TAILWIND_SKIP=1 cargo clippy --workspace --all-targets -- -D warnings`.+ runtime paths; panics are acceptable in tests only.+- `unsafe` is forbidden by workspace lint; do not add it.+- Typed boundaries are load-bearing: IDs, `PagePath`, `AgentKind`,+ sanitization, workspace/project resolution, auth capability, and+ provider dialects are parsed/normalized once and reused.+- Keep CLI commands thin: parse args, resolve config once, call typed+ library functions, render output. Provider-specific behavior belongs in+ `ai-memory-llm`, not in CLI/admin handlers.++## Cross-cutting invariants (do not violate)++These are carved into the architecture; each traces to a documented+prior-art bug (see `docs/ARCHITECTURE.md` and `docs/issues-*.md`):++1. **One config-read path.** `Config::load()` runs once at startup; never+ call `std::env::var` outside it.+2. **Single-writer SQLite actor.** All writes go through one `mpsc`+ channel to one dedicated thread (`WriterHandle`). Batch hot-path work+ into one command/transaction; avoid N+1 reads.+3. **Indexes commit in the same transaction as the data.** No+ index-after-return background tasks.+4. **Typed 3-tuple identity** `(workspace_id, project_id, path)` on every+ domain row.+5. **Hooks are fire-and-forget and bounded.** Script hooks hard-timeout+ at ≤200 ms; the server returns 202 immediately or 429 when saturated.+ No unbounded `tokio::spawn` fan-out or queues on hook paths.+6. **Privacy strip is a typed boundary.** `Sanitized<NewObservation>` has+ no constructor other than `sanitize()`; the hook router's sanitizer is+ the only path from untrusted text into the store.+7. **JSON-schema structured outputs only** for LLM calls; no XML or+ wrapper libraries.+8. **`{provider, model, dim}` denormalized next to every embedding**;+ stale vectors are warned about and ignored on config mismatch.+9. **Live-process check before destructive ops** (`reset`, `backup`,+ `restore` consult `sysinfo`).+10. **Atomic file writes** (tmp + rename + fsync); the watcher ignores+ its own writes by filename prefix.+11. **Absolute canonical data dir**, logged loudly at startup.+12. **No global singletons / `lazy_static` configs**; all dependencies+ explicit.+13. **Zero-LLM default path**; the system fully works with no provider.+14. **Provider auth resolves before provider construction**; provider+ clients consume typed `ProviderAuth` material and never read env vars+ directly.+15. **Tracing subscribers explicitly filter their own module** — no+ feedback loops.++Additional boundary rules:++- **Scope resolution:** new MCP/admin/web routes must use+ `ai_memory_store::ScopeResolver` or its explicit helpers+ (`lookup_existing_scope`, `create_explicit_scope`,+ `resolve_many_existing_scopes`) — never hand-rolled workspace/project+ lookup chains. Read/search/embed/retention/destructive paths use+ no-create lookups and fail closed on partial or missing scope; only+ explicit write/create paths may create workspaces or projects.+- **Auth:** preserve boundaries through+ `AuthLevel::authorize(Capability::...)`; do not open-code username+ comparisons or ad-hoc root checks. In multi-user mode every `/admin/*`+ route is root-only; DB-user tokens never bypass admin gates or+ admission webhooks.+- **Wiki mutations** must go through `Wiki::write_page`,+ `Wiki::apply_batch`, or the existing destructive helpers so+ sanitization, admission, attribution, rollback, and index updates stay+ together. Never write wiki files directly from handlers.++## Testing instructions++- Add focused regression tests for every bug fix and behavior change.+ Parsers, ID derivation, and retention/decay math especially.+- Filesystem tests use temp dirs or injected roots (`tempfile`); never+ depend on the real user home directory being writable.+- PRs touching scope resolution need table-driven tests for partial+ scope, missing explicit scope, active-project precedence, and+ cross-workspace isolation.+- PRs touching permissions need tests for root, DB-user, and anonymous+ behavior.+- New disk+SQL mutations need recovery/rollback tests.+- The recall-eval framework lives at+ `crates/ai-memory-consolidate/tests/recall_eval.rs`.+- Tests run with `cargo test --workspace` (use `TAILWIND_SKIP=1` locally).++## Security considerations++- **Default posture:** loopback-only bind (`127.0.0.1:49374`), no auth —+ safe for a single-user machine. Any non-loopback bind should set a+ bearer token (`AI_MEMORY_AUTH_TOKEN`) and `AI_MEMORY_ALLOWED_HOSTS`+ (DNS-rebinding guard). TLS is deliberately delegated to a reverse+ proxy (see `docs/https-via-proxy.md`).+- **Never commit secrets.** gitleaks runs in CI with `.gitleaks.toml`;+ keep real tokens out of docs, fixtures, and tests.+- **Sanitization is the trust boundary:** all untrusted hook payload text+ passes through the `ai-memory-hooks` sanitizer before storage; do not+ create paths that bypass it (or hook backpressure, or the single-writer+ actor).+- **Capture exclusions** (`[capture] ignore_paths` in the nearest+ `.ai-memory.toml` marker) drop recognized file-tool events before they+ reach spool, transport, logs, or storage — preserve this behavior in+ native hook commands and generated integrations.+- **Auth ladder:** static root bearer token → DB-user tokens+ (attribution only, no admin) → OIDC device tokens at the hook edge.+ `/admin/*` becomes root-only the moment the first DB user exists.+- **Dependency policy:** `cargo deny check --all-features` and+ `cargo audit` run in CI; do not add dependencies without checking the+ project doesn't already have the capability, and match existing+ versions/idioms.+- **Destructive operations** (`purge-project`, `reset`, `restore`) must+ keep their confirmation flags and live-process checks.++## Project maintenance rules++- **CHANGELOG is a merge gate.** Any change affecting user-visible+ behavior, installation, supported platforms/agents/providers,+ deployment, env/config, or public tool/admin surfaces must add a+ `CHANGELOG.md` entry under `## [Unreleased]` (correct+ `Added`/`Changed`/`Fixed` heading, past-tense, trailing `(#NNN)`+ reference) and update the relevant README/docs references in the same+ commit. Internal refactors and test-only churn are exempt.+- **No version bumps or release tags without explicit user approval.**+ Do not bump crate/package versions automatically.+- **PR evaluation:** report pros, cons, and recommended fix, then ask for+ approval before merging or pushing PR changes.+- **MCP tool surface changes** require updating `MEMORY_INSTRUCTIONS`,+ `ai_memory_core::SNIPPET_BODY`, README/docs tool references, and the+ regression tests asserting every tool appears in both prompt surfaces.+ The tool count is currently 16 (see `docs/ARCHITECTURE.md`).+- **Semantic versioning:** patch = fixes; minor = additive (new CLI+ subcommands, MCP tools, config keys); major = breaking (on-disk format+ without migration, removed subcommands, breaking MCP schema changes).+- Keep `CLAUDE.md` as a pointer to this file.++## Documentation map++- [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — operational map: data+ flow, crate breakdown, schema, invariants, config reference.+- [`docs/design-decisions.md`](docs/design-decisions.md) — full v1 spec+ and milestone plan.+- [`docs/install.md`](docs/install.md) — installation cookbook for every+ supported agent client.+- [`docs/lifecycle-ops.md`](docs/lifecycle-ops.md) — read before touching+ purge/rename/backup/restore/reset/reindex/restore-page.+- [`docs/auto-improvement-loop.md`](docs/auto-improvement-loop.md) —+ learning-loop design, approval gates, curator boundaries.+- [`docs/users.md`](docs/users.md) — multi-user attribution and the+ four-rung auth ladder.+- [`docs/managed-workstreams.md`](docs/managed-workstreams.md) —+ `ai-memory run` cross-harness continuity.+- [`docs/companion-crates.md`](docs/companion-crates.md) — boundary for+ optional companion projects (e.g. the importer).