Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

docs: design proposal for boot-time backfill of pre-hook local history

AkitaOnRails18740c60e078
diff --git a/docs/design-boot-backfill.md b/docs/design-boot-backfill.md
new file mode 100644
index 00000000..43405b6f
--- /dev/null
+++ b/docs/design-boot-backfill.md
@@ -0,0 +1,213 @@
+# Design proposal: boot-time backfill of pre-hook local history
+
+**Status: proposal for review — not implemented.** This is the design pass for the
+"pre-load existing session history when the store is new" idea. It adds a new default-on
+capture behavior and touches the trust boundary (retroactive untrusted text enters the
+store), so it is deliberately separated from implementation.
+
+## Problem
+
+Capture is forward-only. The lifecycle hooks record events *from the moment they are
+installed*. If you have already been working in a project for weeks — a long Claude or
+Codex session on disk — and only now install ai-memory, none of that prior context is
+in memory. You get a "cold start": the tool that exists to give cross-session continuity
+begins with amnesia about the very session you are resuming.
+
+The `ai-memory doctor` command (shipped alongside this proposal) makes the related gap
+visible — "this harness ran here but captured nothing." This proposal fills a specific
+instance of it: **when a project's store is brand new (empty) and the harness is resuming
+a session that already has local transcript history, import that history once, so the
+first consolidated pages reflect what you actually did rather than nothing.**
+
+## What already exists (do not rebuild)
+
+The hard part — read a native transcript, sanitize it, get it into the store — is built
+twice already. This proposal is a *trigger and guards* around existing ingest, not a new
+importer.
+
+- **`ai-memory-workstream` transcript adapters** read every supported harness's native
+ session store read-only, keyed by cwd, with per-harness path/id encoding in one place
+ (`export_transcript`, `list_native_sessions`, `discover_native_session`). `doctor`
+ already reuses `list_native_sessions`; boot-backfill reuses `export_transcript`.
+- **`export_transcript`** returns an `ExportedTranscript { native_session_id, events:
+ Vec<NewWorkstreamEvent>, source_cursor, losses }` — a bounded, sanitizable visible-event
+ ledger plus an **opaque incremental cursor**. `ai-memory run` calls it after a managed
+ child exits and POSTs the events in size-bounded batches (`FinishManagedRunRequest`,
+ `event_batches`) to the managed-workstream finish endpoint; the server persists
+ `source_cursor` so the next read is incremental. This is a working backfill engine that
+ only ever runs *after* a managed launch today.
+- **The companion importer** (`companions/ai-memory-importer`) has `ExternalConversation`
+ — "replay a generic external conversation through ai-memory's hook API" — with exactly
+ the bounding discipline this needs: `MAX_CONVERSATION_MESSAGES = 128`,
+ `MAX_USER_MESSAGE_BYTES = 16 KiB`, `MAX_CONVERSATION_TOTAL_BYTES = 1 MiB`, a dedicated
+ `external-import` agent kind, all through the sanitizer.
+- **The sanitizer** (`ai-memory-hooks`, `Sanitized<NewObservation>`) is the only path from
+ untrusted text into the store, and it is enforced by construction (invariant #6).
+- **Detached background work from the hook path** already exists: the native hook spools
+ events and drains them via a separate detached process (`hook_drain_process`), so hooks
+ stay within their ≤200 ms budget (invariant #5). Boot-backfill borrows this pattern.
+
+## Decision (confirmed with the operator)
+
+- **Default on**, with a clean opt-out.
+- **Full project bootstrap**: when the store is empty on first boot in a project,
+ import *all* local sessions for this cwd across every supported harness, not just the
+ current resumed session — hard-capped, with a one-line notice printed.
+
+## Design
+
+### 1. Trigger — the SessionStart boot handshake
+
+The SessionStart hook already fires and already contacts the server (it fetches the
+hot-context/handoff block). Extend that one round-trip to also learn, cheaply:
+
+- **Is this project's store empty?** A scoped count — the same family as doctor's
+ `GET /admin/sessions/by-agent` for `(workspace, project)`; sum of sessions == 0 means
+ empty. Emptiness is the gate: an established project is never retro-imported (that would
+ duplicate/rewrite real history and confuse consolidation). Boot-backfill is a strictly
+ **one-time bootstrap**.
+- **Backfill enabled?** `[capture] backfill_on_start` (default true), overridable by
+ `AI_MEMORY_BACKFILL_ON_START` and a `.ai-memory.toml` marker key.
+
+If empty AND enabled, the hook **spawns a detached `ai-memory backfill` process** and
+returns immediately. The hook never blocks boot and never does the import inline
+(invariant #5). If not empty, or disabled, the hook does nothing new.
+
+### 2. Worker — `ai-memory backfill` (new thin CLI subcommand)
+
+A detached, read-only-on-the-local-side worker. For the resolved `(workspace, project)`
+and cwd:
+
+1. Re-check emptiness server-side and take a **bootstrap lease** (a single-writer guarded
+ claim, like handoff/message claim-once) so two harnesses booting at once cannot both
+ import. Lease keyed by `(workspace_id, project_id)`; second claimant no-ops.
+2. Enumerate local sessions for the cwd across every supported harness (`doctor`'s
+ `scan_local` set), newest first.
+3. For each session, up to the caps below, `export_transcript(...)` and ingest the
+ events, attaching to the harness-native session id so the imported session id matches
+ what forward capture would have used (Claude/Codex local id == server session id).
+4. Persist each session's `source_cursor` so a later real resume continues incrementally
+ from where the backfill stopped — no double-capture of the overlap.
+5. Print one notice to the operator's next surface: `📼 ai-memory imported N prior local
+ sessions (~M events) for this project. Opt out with AI_MEMORY_BACKFILL_ON_START=0.`
+
+Manual control: `ai-memory backfill` can be run by hand (idempotent), with
+`--all-sessions` / `--session <id>` / `--dry-run` / `--force` (re-import a non-empty
+store, explicit) for operators who want it outside the boot path.
+
+### 3. Ingest path — two options
+
+The events from `export_transcript` are `NewWorkstreamEvent`s. Two ways to land them:
+
+- **(A) Reuse the managed-workstream finish endpoint** (`FinishManagedRunRequest` +
+ `source_cursor`, batched by `event_batches`). Richest fit: the cursor and incremental
+ semantics already exist and the server already consolidates workstream ledgers. Cost: it
+ is currently coupled to a managed *run* lifecycle (a `WorkstreamCheckpoint`, an
+ `exit_code`); backfill has no live child, so it needs a "finish without a run" entry —
+ a synthetic/adopted workstream for the pre-hook history.
+- **(B) Replay as sanitized hook observations** (the importer's approach). Simpler and
+ lifecycle-free; integrates with the ordinary capture/consolidation path. Cost: needs a
+ `NewWorkstreamEvent → NewObservation` mapping and its own idempotency (no built-in
+ cursor).
+
+**Recommendation: (A)**, extended with a lifecycle-free "import" variant of the finish
+endpoint. It preserves the incremental `source_cursor` contract end to end (critical for
+step 4 — the overlap between backfilled history and the resumed live session must not be
+captured twice), and reuses the server's existing workstream consolidation rather than
+inventing an observation mapping. The cursor is the load-bearing reason to prefer A.
+
+### 4. Idempotency and overlap
+
+Three guards, all reusing existing mechanisms:
+
+- **Emptiness gate** — only an empty store bootstraps; a populated store is never touched
+ by the automatic path.
+- **Bootstrap lease** — claim-once per project (handoff/message pattern) prevents
+ concurrent double-import across simultaneous boots.
+- **Per-session `source_cursor`** — the imported session's cursor is persisted, so when
+ the operator's live resume then produces new events, forward capture continues *after*
+ the cursor. No event is both backfilled and live-captured.
+
+### 5. Caps (hard, non-negotiable)
+
+Full-project bootstrap can be large (a single real Codex session was ~18.5k events).
+Bound everything, defaults chosen to protect boot and the single-writer actor:
+
+- `max_sessions` (e.g. 25 newest local sessions per project).
+- `max_events_total` and `max_bytes_total` across the whole bootstrap.
+- `max_events_per_session` (favor breadth over one giant session).
+- Per-event/message byte caps inherited from the sanitizer + importer constants.
+- Ingest in the existing size-bounded batches; never one unbounded POST.
+- On hitting a cap: import what fits (newest-first), record the truncation in `losses`,
+ print it in the notice. Never silently drop without a trace.
+
+### 6. Security
+
+Retroactive text is exactly as untrusted as live hook text — it is old model/tool output.
+
+- **No sanitizer bypass.** Every imported record passes the same
+ `Sanitized<NewObservation>` / workstream sanitize boundary and the same caps. The e2e
+ canary secret must be scrubbed from a backfilled body just as from a live one.
+- **Capture exclusions honored.** `[capture] ignore_paths` from the nearest
+ `.ai-memory.toml` must drop matching file-tool events in the backfill path too, not just
+ live hooks.
+- **Emptiness + lease** prevent a backfill from overwriting or racing an established
+ project's real history.
+- **Multi-user posture.** On a shared server, default-on retro-import of a long local
+ history is more sensitive than on a single-user box. Backfill attributes to the booting
+ operator, imports only into an *empty* project scope, and is opt-outable per project via
+ the marker. Consider gating the automatic path to the default (loopback/single-operator)
+ posture and requiring explicit `ai-memory backfill` on `deployment_distinguishes_operators()`
+ servers — an open question below.
+
+### 7. Opt-out and configuration
+
+- `[capture] backfill_on_start = true` (default) in config.
+- `AI_MEMORY_BACKFILL_ON_START=0` env override (one config-read path — read in
+ `Config::load`, never ad hoc).
+- `.ai-memory.toml` marker key so a specific project can opt out regardless of global
+ config (mirrors how `ignore_paths` is project-scoped).
+- Manual `ai-memory backfill` is always available regardless of the auto setting.
+
+### 8. Harness coverage
+
+Exactly the set with read-only native transcript adapters (doctor's `SCANNED_HARNESSES`).
+A harness whose store is unreadable/absent contributes nothing — backfill never invents
+history it cannot read, same conservative stance as doctor.
+
+## Relationship to `ai-memory doctor`
+
+Complementary: **doctor detects** the coverage gap (a harness ran here but captured
+nothing); **backfill fills** the specific first-boot instance of it. The docs should
+cross-link them — doctor is the ongoing check, backfill is the one-time bootstrap.
+
+## Testing plan
+
+- **Pure**: cap enforcement (sessions/events/bytes, newest-first truncation), emptiness
+ gate decision, enabled/disabled resolution, overlap/cursor bookkeeping.
+- **Detector reuse**: backfill's local enumeration shares doctor's `scan_local`; the
+ planted-transcript fixture pattern extends to "export + ingest a planted Claude session
+ into an empty in-memory store, assert observations land, sanitized, once."
+- **Idempotency**: run backfill twice → second run no-ops (lease + non-empty gate); a
+ live resume after backfill does not double-capture the overlap (cursor).
+- **Security**: a canary secret in a planted transcript is scrubbed; an `ignore_paths`
+ match in a planted file-tool event is dropped.
+- **Multi-session**: two concurrent boots → exactly one import (lease claim-once), the
+ other no-ops — the invariant-#16 concurrency shape, proven at integration level.
+
+## Open questions for the plan
+
+1. **Ingest path A vs B** — confirm the lifecycle-free workstream-import endpoint is worth
+ adding versus an observation replay. (Recommend A for the cursor.)
+2. **Shared-server default** — auto-on everywhere, or auto-on only in the single-operator
+ posture with explicit `ai-memory backfill` required on multi-user servers?
+3. **Notice delivery** — Claude discards SessionStart stdout; reuse the same
+ "deliver on first user prompt" path the hot-context block already uses for Kimi/Claude?
+4. **Consolidation quality** — should a first pass gate on a small eval that backfilled
+ pre-hook ledgers produce pages of comparable quality to forward capture before making
+ full-project bootstrap the default, or ship behind the cap and iterate?
+
+## Semver
+
+Additive (new subcommand, new config key, new default-on behavior, no breaking change) →
+**minor**, targeting `release/2.3`.