Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs(design): opt-in file path index (#1843)

Yuri Alves44a387c6de64
diff --git a/docs/design-file-path-index.md b/docs/design-file-path-index.md
new file mode 100644
index 00000000..edf39b64
--- /dev/null
+++ b/docs/design-file-path-index.md
@@ -0,0 +1,352 @@
+# Design proposal: opt-in file path index (#1843)
+
+**Status: open design proposal, not implemented, with no assigned release.**
+Issue [#1843](https://github.com/akitaonrails/ai-memory/issues/1843) remains
+open. Any implementation must rebase the migration number and branch
+assumptions below against the then-current tree.
+
+The maintainer set the order in
+[#1843](https://github.com/akitaonrails/ai-memory/issues/1843#issuecomment-6508464269):
+specify the opt-in path index, then design delivery separately, then measure
+before expanding. This note covers only the index. Section 10 lists the
+decisions for the maintainer.
+
+## 1. Problem
+
+A file-tool observation records that a file tool ran, but not which file. The
+body is `tool_family`, `outcome`, `tool_call_id` and the bounded output, and the
+table has no path column. So on the zero-LLM path the store cannot answer "what
+do we know about `src/lib.rs`" except by matching incidental text. Text matching
+is unreliable for paths (section 2), and the maintainer ruled it out as a basis
+for delivery.
+
+Non-goals: delivery into any harness, calls that name more than one file
+(including a multi-file `apply_patch`), shell commands, legacy shell and
+PowerShell hook bundles, any new MCP tool or HTTP route (section 6).
+
+## 2. Current state
+
+Checked against `main` at `ac699b56`. References name the file and the symbol.
+
+| Part | Where | Fact |
+|---|---|---|
+| Table | `crates/ai-memory-store/migrations/V01__init.sql`, `V10__observation_extension_events.sql` | `observations` keys every row by `session_id`, `workspace_id` and `project_id`, all `ON DELETE CASCADE`. No path column. V10 added two nullable columns (`extension`, `source_event`) and a partial index `WHERE extension IS NOT NULL AND source_event IS NOT NULL` scoped by `(workspace_id, project_id, ...)`. The highest migration is V96. |
+| Body | `crates/ai-memory-hooks/src/payload.rs` `safe_tool_body`, `closed_tool_agent`; `capture_policy.rs` `tool_observation_metadata` | Closed schema for the agents in `closed_tool_agent`. `tool_observation_metadata` keeps family, call id and MCP names; it never reads a path field. |
+| FTS | `crates/ai-memory-store/migrations/V12__fts_remove_diacritics.sql` (`observations_fts`) | The tokenizer keeps `/`, `_` and `-` inside tokens. An absolute path in tool output (`/repo/src/lib.rs`) does not match a query for `src/lib.rs`. The CJK index added in V87 covers only unspaced Han, Kana and Hangul. |
+| Client policy | `crates/ai-memory-cli/src/commands/hook.rs` `run_with_payload_in_home` | The native client applies `[capture] ignore_paths` before the spool: Drop sends nothing, MetadataOnly rebuilds the body, Keep under an active or invalid policy attaches `_ai_memory_capture`. Keep under an **inactive** policy (no marker, or no `ignore_paths`, the common case) attaches nothing. |
+| TS policy | `crates/ai-memory-cli/src/commands/render_shared.rs` `ts_capture_policy_v1` | The generated integrations (OpenCode, OMP, Pi, OpenClaw) implement the same table and return early on an inactive policy, also without a claim. The template is generated from `capture_policy.rs` so the two front doors cannot drift. |
+| `apply_patch` | `capture_policy.rs` `extract`, `patch_paths` | Codex `apply_patch` and Droid `ApplyPatch` carry no direct path field. `patch_paths` reads the `Add File`, `Update File`, `Delete File` and `Move to` headers, and feeds only the cross-checkout lookup. A file call without a direct path is `MissingOrMalformed`, so under an active policy it becomes metadata-only, whatever the paths. |
+| Protocol | `capture_policy.rs` `CaptureProtocol` | Parses through a `deny_unknown_fields` wire struct with a fixed version. |
+| Reserved fields | `crates/ai-memory-hooks/src/assistant_capture.rs` `apply_assistant_backstop`; `router.rs` (`_ai_memory_usage`) | Precedent for a separate top-level field: `_ai_memory_assistant` is versioned, parsed with `deny_unknown_fields`, and always removed from the raw payload before storage, whether or not its gates accept it. |
+| Server check | `crates/ai-memory-hooks/src/router.rs` `inspect_capture_envelope` | An unparsable claim is treated as invalid; for a file tool that rewrites the event to metadata-only. A parsed claim is re-inspected against the payload; the server never receives the client's patterns. |
+| Checkout root | `crates/ai-memory-cli/src/marker.rs` `checkout_root` | The nearest ancestor holding `.git` (a worktree's `.git` file counts). `ignore_paths` patterns root at the marker directory, not the checkout root. |
+| Normalization | `capture_policy.rs` `normalize_segments`, `marker.rs` `absolute_normalized` | Lexical; `.` and `..` folded; symlinks never resolved. |
+| Sanitizer | `crates/ai-memory-core/src/sanitize.rs` `scrub` | Credential directories (`.ssh`, `.aws`, `.kube`, `.config/gcloud`, `.gnupg`) are redacted as `credential_path`; the patterns need a separator before the directory. |
+| Ingest config | `docs/config-properties.md` (`[ingest]`) | Holds the server-side capture filters, among them `record_tool_names`, off by default. |
+| Ownership | `crates/ai-memory-core/src/actor.rs` `OwnerFilter`; `crates/ai-memory-store/src/reader.rs` `sessions_hidden_from` | Observations are owned through `sessions.actor_user` (V40). The observation stream hides other operators' sessions with `sessions_hidden_from`. `session_observations_scoped` takes no owner and relies on its callers checking the session first. |
+
+## 3. Proposal
+
+### 3.1 Storage
+
+One nullable column and one partial index, in a single additive migration:
+
+```sql
+ALTER TABLE observations ADD COLUMN file_path TEXT;
+CREATE INDEX idx_observations_file_path
+ ON observations(workspace_id, project_id, file_path, created_at)
+ WHERE file_path IS NOT NULL;
+```
+
+The column follows the V10 precedent. A row has at most one path in this
+version (3.2), so a side table would only add a join. Because the path lives on
+the observation row, every lifecycle operation in 3.5 applies to it with no new
+code.
+
+### 3.2 Where the path comes from
+
+Only the two front doors that implement capture policy v1 produce a path: the
+native client and the generated TS integrations. They add a reserved top-level
+field, versioned like `_ai_memory_assistant`:
+
+```json
+"_ai_memory_file": { "version": 1, "path": "src/lib.rs" }
+```
+
+**When the field is sent.** All three must hold:
+
+- the disposition is Keep, under an active or an inactive policy;
+- the family is `file`;
+- the call names exactly one file: a direct path (extraction state
+ `extracted`, `path_count` 1), or an `apply_patch` / `ApplyPatch` whose
+ envelope has exactly one file header, read with the existing `patch_paths`.
+ A `Move to` header names two files and is skipped.
+
+Drop and MetadataOnly never carry the field, so an ignored path never leaves
+the machine through it. Under an inactive policy the kept payload already
+carries the raw path, so the field exposes nothing new; it only carries the
+normalized form the server cannot compute (3.3). An `apply_patch` only reaches
+this point under an inactive policy (section 2); section 10, question 6, asks
+whether that is acceptable.
+
+The client never reads server config, so the field goes out on every such
+event even when the server key is off. The cost is one short JSON object per
+kept file-tool post, which the server removes on arrival (3.4).
+
+**Why a separate field.** `CaptureProtocol` parses with `deny_unknown_fields`,
+so a new key inside `_ai_memory_capture` would make an older server treat every
+claim as invalid and rewrite every file event to metadata-only. An unknown
+top-level field is ignored by older servers, and older clients never send it.
+
+**Who never sends it.** Legacy shell and PowerShell bundles, manual Docker
+script bundles and older clients, so they never index.
+
+### 3.3 Normalization
+
+The client computes the path, because only the client knows the checkout root
+of the event:
+
+1. Make the tool path absolute against the event's cwd with the existing
+ lexical helpers (`absolute_normalized` in the CLI crate,
+ `normalize_segments` in the hooks crate). Both are private today; the
+ implementation widens the visibility it needs and adds no second
+ normalizer.
+2. Strip the `checkout_root` prefix. A path outside the checkout is not indexed.
+3. Use `/` as the separator; keep case as given.
+4. Refuse: an empty result, a leading `/` or drive letter, any `..` segment, more
+ than 512 characters (the bound `HANDOFF_FILE_MAX_CHARS` uses for stored
+ paths).
+
+Relative to the checkout root rather than the marker directory, so two worktrees
+or two clones of the same repository produce the same key.
+
+On the client, step 1 already folds `..`, so the `..` refusal matters on the
+server: it applies to the string the server receives. The server repeats every
+refusal in step 4 and drops only the field, never the event. It cannot repeat
+step 2, because it never learns the checkout root; a path from outside the
+checkout reaches it only as absolute or with `..`, which step 4 already
+refuses.
+
+### 3.4 Server acceptance
+
+The server always removes `_ai_memory_file` from the raw payload, as
+`apply_assistant_backstop` does with `_ai_memory_assistant`, on both routes
+that build an envelope: `handle_hook` and the batch route. It stores
+`file_path` only when all of these hold:
+
+- `[ingest] index_file_paths = true` (default `false`; 3.7);
+- the field parses with `deny_unknown_fields` and the current version;
+- the server's own inspection of the event yields Keep, family `file` and
+ exactly one file. Under an inactive policy the client attaches no claim, and
+ `inspect_capture_envelope` returns such an event without inspecting it, so
+ the server runs `capture_inspector(...).inspect` itself, as
+ `inspect_capture_envelope` already does for a claim: with the claimed policy
+ state, or `Inactive` when no claim is attached. For an `apply_patch`, the
+ single file is checked with `patch_paths`. This runs only when the key is on
+ and the field is present;
+- the path passes the step 4 checks;
+- `scrub` leaves the path unchanged, checked on the path with a leading `/`
+ prepended, because the credential-directory patterns need a separator before
+ the directory (`.ssh/config` at the checkout root would otherwise pass).
+
+The path enters the store through the typed boundary: `NewObservation` gains
+`file_path: Option<String>`, and `Sanitized::<NewObservation>::new` runs the
+checks above and clears the field when one fails. A failed check drops the field and keeps the
+observation exactly as today.
+
+### 3.5 Lifecycle
+
+The column inherits the observation's lifecycle (`docs/lifecycle-ops.md`):
+
+| Operation | Effect on `file_path` |
+|---|---|
+| `observation_retention_days` prune (`prune_consolidated_observations`) | deleted with the row |
+| purge-project, purge-session, delete-workspace | deleted by the existing cascade; purge tombstones also stop spool replay from recreating it |
+| move-session, move-project, merge-project | moves with the row (the existing `UPDATE observations` re-stamps it) |
+| reset, restore, backup, `uninstall --purge-data` | follows `db/` |
+| `reindex` | not rebuilt; `reindex` never reconstructs observations |
+| turning the key off | stops new writes; existing values stay until their rows go (section 10, question 3) |
+
+### 3.6 Read authorization
+
+One internal store function, with no MCP tool, HTTP route or CLI command in
+this step:
+
+```rust
+observations_for_file(scope: (WorkspaceId, ProjectId), owner: &OwnerFilter,
+ file_path: &str, limit: usize) -> Vec<ObservationRef>
+```
+
+- `ObservationRef` is a new type: the observation id and its `created_at`.
+- Scope comes from the no-create lookup in `ScopeResolver`; no `global`,
+ `scopes` or `[recall] default_global` broadening.
+- The owner filter reuses `sessions_hidden_from`: another operator's session is
+ hidden, shared (`NULL` owner) sessions are visible.
+- `limit` is bounded; the result carries ids and timestamps, not bodies.
+
+It lives in `reader.rs`, next to the private `sessions_hidden_from`. It exists
+to prove isolation now, as #1843 asks, and is the function a later delivery
+design would call. Until then its only callers are tests. AGENTS.md allows new
+surface without a shipped caller when there is an explicit requirement, and
+the #1843 sequence is that requirement; section 10, question 5, lets the
+maintainer defer it instead.
+
+### 3.7 Opt-in
+
+A server key in `[ingest]`, off by default like `record_tool_names`. The
+operator owns the store, so the operator decides whether paths are kept. The
+RFC pointed at the marker's `[recall]` table, but for delivery, which spends
+tokens per repository; keeping data is a decision about the store. A
+repository keeps opting out of capture with `ignore_paths`, which applies
+before the field exists. The key lands in `config.default.toml` and
+`docs/config-properties.md` in the implementation PR.
+
+## 4. Invariants
+
+Numbered as in the cross-cutting invariants of AGENTS.md.
+
+- **1 (one config read):** the key is read by `Config::load` on the server; the
+ clients read nothing new.
+- **2 and 3 (single writer, same transaction):** `file_path` is written by the
+ existing observation insert, in the same statement.
+- **4 (typed identity):** `file_path` is an attribute of the observation, not
+ part of its identity; the row keeps its `(workspace_id, project_id)` and no
+ new domain row is created.
+- **5 (bounded hooks):** no new synchronous call; the field rides the existing
+ post.
+- **6 (sanitizer boundary):** the field is validated and scrub-checked inside
+ `Sanitized::<NewObservation>::new`, so it reaches the store only through
+ `Sanitized<NewObservation>`.
+- **16 (pages shared, batons owned):** the index belongs to observations, which
+ are owned through their session; pages are untouched.
+
+## 5. Trust and security
+
+New rows for `docs/security-boundaries.md`. Each test fails with its guard
+removed and passes with it:
+
+| Guard | Enforcing code | Adversarial test |
+|---|---|---|
+| A path never crosses projects | `observations_for_file` scope predicate | Same `file_path` in two projects; a query from one never returns the other's rows. Control: same project returns them. |
+| A path never crosses operators | `sessions_hidden_from` in `observations_for_file` | Operator B's session is hidden from A; owner and root see it; a shared session is visible to both. |
+| An ignored path is never stored | client Drop and MetadataOnly carry no field | Marker ignores `secret/`; a Read there stores no row and no `file_path`. |
+| A forged or malformed field is not stored | server checks in 3.4 | Absolute, `..`, over-long, credential-directory, wrong-version and non-file-family fields are dropped, as is a field on a patch that names two files; the observation is kept and the field never reaches storage. |
+| Off means off | 3.4 key check | With the key off, a valid field stores nothing. |
+
+A client can still lie about a path inside its own project, as it can about any
+captured text today. The server cannot verify that a path was not ignored,
+because it never receives the patterns; this is the same trust the capture
+policy already places in the client.
+
+## 6. Compatibility
+
+- Key off (the default): existing columns are unchanged and `file_path` stays
+ `NULL`; the field is removed and ignored.
+- Migration is additive: one nullable column, existing rows read `NULL`, and
+ no current query changes. Like every migration it is one-way: an older binary
+ refuses the upgraded store (`StoreError::DataSchemaAhead`), so a downgrade
+ needs a `restore` from the pre-upgrade backup.
+- New client with an older server: the field is ignored. Older client with a
+ newer server: no field, no path.
+- No MCP, HTTP, SDK or routing-snippet change.
+
+## 7. Tests
+
+**Unit**
+
+- Rust: normalization (relative, absolute, Windows spelling, worktree), each
+ refusal in 3.3, the scrub check including `.ssh/config` at the root. A
+ symlinked cwd is not resolved: a cwd whose lexical form is outside the
+ checkout yields no field.
+- Rust, `apply_patch`: one `Add File`, `Update File` or `Delete File` header
+ yields the field; two files, a `Move to` header, an empty or malformed patch,
+ and an active policy yield none. The same table for Droid `ApplyPatch`.
+- Generated TS: the Node checks `render_shared.rs` already runs for
+ `ts_capture_policy_v1`, extended to the field under active and inactive
+ policies, and to its absence on Drop and MetadataOnly.
+
+**Integration**
+
+- `crates/ai-memory-store/tests/suite/multi_session.rs`, `project_authz.rs`: the
+ project and operator rows in section 5.
+- Hooks router tests, on `handle_hook` and the batch route: forged and
+ malformed fields, key off, no field from a legacy-shaped payload, a field on
+ an event without a claim that the server's inspection rejects, the
+ observation kept in every refusal; a Codex-shaped `apply_patch` PostToolUse
+ with one file stores its path, one with two files stores none.
+- `observation_prune.rs`, `session_move.rs`: prune, purge and move take the path
+ with the row.
+- Recovery: the migration applied to a store with existing observations, and a
+ failed observation insert that leaves no `file_path` behind (AGENTS.md: new
+ disk+SQL mutations need recovery and rollback tests).
+- `crates/ai-memory-cli/tests/suite/external_capture_ts.rs`: the generated
+ plugin, run in Node against a real server, stores the path.
+
+**Live harness (manual, recorded in the implementation PR's test plan).** The
+existing smoke tests fire no tool event: `tests/e2e/claude_plugin_smoke.py`
+runs without credentials, `codex_plugin_smoke.py` points the provider at a
+closed port, and `handoff_smoke.sh` records why OpenCode is not automated.
+
+- Claude Code: `claude -p` with credentials in a fixture checkout reads and
+ edits one file each; the SQLite rows carry the expected `file_path`.
+- Codex: `codex exec` edits one file through `apply_patch`; the row carries
+ `file_path`. A second run edits two files in one patch; no row carries a
+ path. Codex reads go through the shell and stay out of scope.
+- OpenCode: the same through `opencode run`.
+
+**Measurement** (before any delivery design): on a real store with the key on,
+report the share of live file-tool events that got a path (backfill imports
+never carry the field, so they stay out of the denominator), the number of paths
+touched in two or more sessions, and the stored bytes per thousand
+observations. Delivery is designed only if the share reaches the threshold in
+section 10, question 7.
+
+## 8. Pull request plan
+
+1. This note, against `main` because it is docs only (as #1319).
+2. Index, against the standing feature branch (currently `release/3.4`):
+ migration, both front doors, server acceptance, `observations_for_file`,
+ config key and docs, the section 5 rows and the section 7 tests, CHANGELOG.
+ Off by default.
+3. Measurement on a real store, published with the PR or right after it.
+4. Delivery: a separate design note, starting with PostToolUse in one harness,
+ once per path per session, untrusted-data framing, no global broadening.
+
+## 9. Alternatives
+
+- **Text matching only (option A in #1843):** no storage change, but the
+ tokenizer misses absolute paths in tool output and coverage without LLM
+ consolidation is low. Rejected by the maintainer as a basis for delivery.
+- **A side table:** fits several paths per call (a multi-file `apply_patch`), at
+ the cost of a join and its own lifecycle. Deferred until multi-file calls are
+ in scope.
+- **The path inside `_ai_memory_capture`:** breaks older servers (3.2).
+- **A protocol version bump:** same effect on older servers.
+- **A hash instead of the path:** a repository path has little entropy, so the
+ hash hides little, and it makes the index unreadable for diagnosis.
+- **Server-side normalization:** the server sees the cwd, not the checkout
+ root, so worktrees would not share keys.
+
+## 10. Decisions for the maintainer
+
+Each question carries the default this note assumes.
+
+1. **Key name and section.** Default: `[ingest] index_file_paths`, next to the
+ other server-side capture filters.
+2. **Reads and writes, or writes only.** Default: both, because the later
+ delivery is about a file the agent opens.
+3. **Turning the key off.** Default: keep existing values until their rows go;
+ retention and purge already remove them, and a clearing sweep would be new
+ code.
+4. **Path length bound.** Default: reuse 512. `HANDOFF_FILE_MAX_CHARS` is
+ private to `ai-memory-mcp`, so the hooks crate needs its own constant or a
+ shared one in `ai-memory-core`.
+5. **`observations_for_file` in step 2 or with delivery.** Default: step 2, with
+ test-only callers, because #1843 asks for isolation to be proven before
+ delivery. Deferring it moves the two isolation rows of section 5 to the
+ delivery PR.
+6. **Single-file `apply_patch` only under an inactive policy.** Default: accept
+ the asymmetry. Codex writes go through `apply_patch`, so leaving it out
+ drops every Codex write from the index.
+7. **Measurement threshold.** The share of file-tool events with a path below
+ which delivery is not designed. No default.