Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(cli): boot-time backfill of pre-hook local history (on by default)

AkitaOnRailsc380278ef677
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 22e63013..95fed696 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- Boot-time backfill of pre-hook local history, on by default. When a project's
+ ai-memory store is brand new (empty), the SessionStart hook triggers a
+ one-time, bounded import of that project's existing local harness transcripts
+ so installing hooks mid-project no longer starts amnesiac about the very
+ session you are resuming. It replays each local transcript through `/hook` —
+ the same ingress live capture uses — so the history becomes real sessions and
+ observations that consolidate into pages and are searchable via `memory_query`,
+ **sanitized and bounded on the server exactly like live capture** and
+ attributed to the original harness. It only ever bootstraps an empty project
+ (never overwrites an established one; live capture from install-time forward
+ and backfill of before-install history do not overlap), is hard-capped (newest
+ 25 sessions, 50k events), and runs detached so session start is never blocked.
+ New `ai-memory backfill`
+ subcommand runs it by hand (`--dry-run`, `--force`, `--session`, `--json`).
+ Opt out with `AI_MEMORY_BACKFILL_ON_START=false` / `backfill_on_start = false`.
- New `ai-memory doctor` command: a capture-coverage check that, for the current
project, compares every known harness's local (on-disk) native session store
against what the server actually captured (`GET /admin/sessions/by-agent`) and
diff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rs
index 35b709ed..7cbf5f58 100644
--- a/crates/ai-memory-cli/src/cli.rs
+++ b/crates/ai-memory-cli/src/cli.rs
@@ -37,6 +37,11 @@ pub enum Command {
/// project against what the server captured, and warn when a harness ran
/// here recently but has no captured sessions (its hook is likely missing).
Doctor(DoctorArgs),
+ /// One-time import of this project's existing local harness session history
+ /// into a brand-new (empty) ai-memory store, so installing hooks
+ /// mid-project doesn't start amnesiac. No-op once the store has any
+ /// sessions unless `--force`.
+ Backfill(BackfillArgs),
/// Launch an agent in an opt-in, cross-harness managed workstream.
/// Native arguments are forwarded except exact wrapper flags such as
/// `--yolo` and `--fresh`.
@@ -1368,6 +1373,43 @@ pub struct StatusArgs {
pub json: bool,
}
+/// Arguments for `backfill`.
+#[derive(Debug, Args)]
+pub struct BackfillArgs {
+ /// Workspace name. Defaults to the current project's resolved scope.
+ #[arg(long)]
+ pub workspace: Option<String>,
+ /// Project name. Defaults to the current project's resolved scope.
+ #[arg(long)]
+ pub project: Option<String>,
+ /// Import only this one harness-native session id instead of every local
+ /// session for the project.
+ #[arg(long)]
+ pub session: Option<String>,
+ /// Import even when the store already has sessions. Off by default: the
+ /// automatic path is a one-time bootstrap of an empty project only.
+ #[arg(long)]
+ pub force: bool,
+ /// Report what would be imported without importing anything.
+ #[arg(long)]
+ pub dry_run: bool,
+ /// Import at most this many of the newest local sessions.
+ #[arg(long, default_value_t = 25)]
+ pub max_sessions: usize,
+ /// Emit the report as JSON instead of human-readable text.
+ #[arg(long)]
+ pub json: bool,
+ /// Suppress the human summary line (used by the automatic SessionStart
+ /// trigger, which runs detached).
+ #[arg(long)]
+ pub quiet: bool,
+ /// Internal: this run was spawned by the SessionStart trigger. Honors the
+ /// `backfill_on_start` opt-out and records that the automatic bootstrap has
+ /// been attempted for this checkout. Not for interactive use.
+ #[arg(long, hide = true)]
+ pub auto: bool,
+}
+
/// Arguments for `doctor`.
#[derive(Debug, Args)]
pub struct DoctorArgs {
diff --git a/crates/ai-memory-cli/src/commands/backfill.rs b/crates/ai-memory-cli/src/commands/backfill.rs
new file mode 100644
index 00000000..75898dae
--- /dev/null
+++ b/crates/ai-memory-cli/src/commands/backfill.rs
@@ -0,0 +1,790 @@
+//! `ai-memory backfill` — one-time import of a project's pre-hook local history.
+//!
+//! Capture is forward-only: hooks record events only from the moment they are
+//! installed. If you have worked in a project for weeks and only now install
+//! ai-memory, the tool that exists for cross-session continuity starts
+//! amnesiac about the very session you are resuming. This command fills that
+//! first-boot gap: when the project's store is **brand new (empty)**, it
+//! imports the existing local harness transcript history once.
+//!
+//! For each local native session it **replays the transcript through `/hook`**,
+//! the same ingress live capture uses (as the companion importer does for
+//! external conversations):
+//!
+//! - `export_transcript` (ai-memory-workstream) reads the harness's native
+//! transcript read-only into a normalized, bounded visible-event ledger,
+//! doing all the per-harness parsing in one place;
+//! - each event is posted to `/hook/batch` as a session-start, `user-prompt`,
+//! or backfill-extension observation, attributed to the original harness and
+//! its native session id — so it becomes a real session + observations that
+//! consolidate into pages and are searchable via `memory_query`, exactly like
+//! live capture. The server **sanitizes and bounds every event**, so
+//! retroactive text crosses the same trust boundary as live capture.
+//!
+//! (An earlier revision imported through the managed-workstream begin/finish
+//! endpoints, but that populates the `ai-memory run` continuity ledger, not the
+//! searchable memory pipeline — verified by a live smoke where the observation
+//! count stayed zero. `/hook` replay is what makes the history recall-able.)
+//!
+//! Safety: the automatic path only ever bootstraps an **empty** project (never
+//! overwrites an established one — hook capture from install-time forward and
+//! backfill of before-install history do not overlap), and it runs at most once
+//! per checkout (a local sentinel).
+
+use std::path::Path;
+use std::path::PathBuf;
+use std::time::{Duration, SystemTime};
+
+use anyhow::{Context, Result, bail};
+use serde::{Deserialize, Serialize};
+
+use ai_memory_core::{NewWorkstreamEvent, WorkstreamEventKind};
+use ai_memory_workstream::{
+ ManagedHarness, build_launch_plan, export_transcript, list_native_sessions,
+ wait_for_transcript_flush,
+};
+
+use super::doctor::SCANNED_HARNESSES;
+use super::run;
+use crate::config::Config;
+use crate::http_client::{ServerEndpoint, get_json, post_json};
+
+/// Circuit breaker on total imported events across the whole bootstrap. Once
+/// reached, no further sessions are started; the remaining sessions are
+/// reported as skipped rather than partially imported.
+const MAX_EVENTS_TOTAL: usize = 50_000;
+
+/// Newest local sessions to enumerate per harness before applying `--max-sessions`.
+const PER_HARNESS_SCAN_LIMIT: usize = 200;
+
+/// One `POST /hook/batch` request carries at most this many events — matches the
+/// server's `MAX_HOOK_BATCH_ITEMS` so a batch is never rejected for size.
+const HOOK_BATCH_ITEMS: usize = 256;
+
+/// The extension label backfilled non-user events (assistant/tool) are recorded
+/// under, mirroring the companion importer's replay-through-`/hook` mechanism.
+const BACKFILL_EXTENSION: &str = "ai-memory-backfill";
+
+/// A local native session eligible for import.
+#[derive(Debug, Clone)]
+pub(crate) struct SessionRef {
+ harness: ManagedHarness,
+ native_session_id: String,
+ updated_at: SystemTime,
+}
+
+/// The outcome of a backfill run, and the JSON output shape.
+#[derive(Debug, Default, Serialize)]
+struct BackfillReport {
+ workspace: String,
+ project: String,
+ /// Sessions selected for import (after `--max-sessions`).
+ selected: usize,
+ /// Sessions actually imported.
+ imported_sessions: usize,
+ /// Events imported across all sessions.
+ imported_events: usize,
+ /// Sessions skipped because the total-events circuit breaker tripped.
+ skipped_for_cap: usize,
+ /// Per-session failures (import errors); the run continues past them.
+ failed_sessions: usize,
+ /// True when the store was already populated (or `--force` off) and nothing
+ /// was imported.
+ skipped_non_empty: bool,
+ /// True for a `--dry-run` (planning only, no import).
+ dry_run: bool,
+}
+
+/// The captured side: `GET /admin/sessions/by-agent` — summing its counts tells
+/// us whether the project store is empty.
+#[derive(Debug, Deserialize)]
+struct ByAgentResponse {
+ by_agent: Vec<AgentCount>,
+}
+
+#[derive(Debug, Deserialize)]
+struct AgentCount {
+ #[allow(dead_code)]
+ agent: String,
+ sessions: u64,
+}
+
+/// Newest-first, then cap to `max_sessions`. Pure, so the selection/cap policy
+/// is unit-tested without a server.
+pub(crate) fn select_sessions(mut all: Vec<SessionRef>, max_sessions: usize) -> Vec<SessionRef> {
+ all.sort_by(|a, b| {
+ b.updated_at
+ .cmp(&a.updated_at)
+ .then_with(|| a.native_session_id.cmp(&b.native_session_id))
+ });
+ all.truncate(max_sessions);
+ all
+}
+
+/// Filename under `<data_dir>/backfill-state/` marking that the automatic
+/// backfill has already been attempted for this checkout on this machine, so
+/// the SessionStart trigger does not re-spawn on every boot. Keyed by a hash of
+/// the working directory — which both the hook (before it spawns) and the
+/// worker (which inherits the hook's cwd) compute identically without resolving
+/// scope — so arbitrary paths stay path-safe.
+pub(crate) fn sentinel_path(data_dir: &Path, cwd: &Path) -> PathBuf {
+ use sha2::{Digest, Sha256};
+ let mut hasher = Sha256::new();
+ hasher.update(cwd.as_os_str().as_encoded_bytes());
+ let digest = hasher.finalize();
+ data_dir.join("backfill-state").join(format!("{digest:x}"))
+}
+
+fn write_sentinel(data_dir: &Path, cwd: &Path) {
+ let path = sentinel_path(data_dir, cwd);
+ if let Some(parent) = path.parent() {
+ let _ = std::fs::create_dir_all(parent);
+ }
+ // Best-effort: a missing sentinel only means a redundant future probe, not
+ // incorrect data, so a write failure is not worth failing the import over.
+ let _ = std::fs::write(&path, b"");
+}
+
+/// Run the backfill.
+///
+/// # Errors
+/// Returns an error when the scope cannot be resolved, the working directory
+/// cannot be read, or the server is unreachable for the emptiness check.
+pub async fn run(config: &Config, args: crate::cli::BackfillArgs) -> Result<()> {
+ let cwd = std::env::current_dir().context("resolving the current working directory")?;
+
+ // The automatic (SessionStart-spawned) path honors the opt-out and records
+ // that it has been attempted for this checkout, so it runs at most once per
+ // machine regardless of outcome. Manual runs ignore both.
+ if args.auto && !config.backfill_on_start {
+ write_sentinel(&config.data_dir, &cwd);
+ return Ok(());
+ }
+
+ let (workspace, project) =
+ super::resolve_scope(config, args.workspace.as_deref(), args.project.as_deref())?;
+ let home = run::native_home(config).context("locating the local harness session stores")?;
+ let endpoint = ServerEndpoint::from_config_resolving_auth(config).await;
+
+ let mut report = BackfillReport {
+ workspace: workspace.clone(),
+ project: project.clone(),
+ dry_run: args.dry_run,
+ ..BackfillReport::default()
+ };
+
+ // Emptiness gate. Only bootstrap a brand-new store; an established project
+ // is never retro-imported (that would duplicate hook-captured history).
+ if !args.force {
+ let empty = project_is_empty(&endpoint, &workspace, &project).await?;
+ if !empty {
+ report.skipped_non_empty = true;
+ // Mark attempted so the auto-trigger stops probing this checkout.
+ write_sentinel(&config.data_dir, &cwd);
+ return finish(&args, &report);
+ }
+ }
+
+ // Enumerate local sessions for this cwd across every supported harness.
+ let candidates = collect_local_sessions(&home, &cwd, args.session.as_deref()).await;
+ let selected = select_sessions(candidates, args.max_sessions.max(1));
+ report.selected = selected.len();
+
+ if args.dry_run {
+ write_sentinel(&config.data_dir, &cwd);
+ return finish(&args, &report);
+ }
+
+ // Import each selected session, oldest-first so the imported history reads
+ // chronologically, until the total-events circuit breaker trips.
+ for session in selected.into_iter().rev() {
+ if report.imported_events >= MAX_EVENTS_TOTAL {
+ report.skipped_for_cap += 1;
+ continue;
+ }
+ match import_one(&endpoint, &workspace, &project, &home, &cwd, &session).await {
+ Ok(events) => {
+ report.imported_sessions += 1;
+ report.imported_events += events;
+ }
+ Err(error) => {
+ report.failed_sessions += 1;
+ if !args.quiet {
+ eprintln!(
+ "ai-memory: backfill of {} session {} failed: {error:#}",
+ session.harness.as_str(),
+ display_id(&session.native_session_id)
+ );
+ }
+ }
+ }
+ }
+
+ write_sentinel(&config.data_dir, &cwd);
+ finish(&args, &report)
+}
+
+/// Sum the server's per-agent session counts for this scope; zero means empty.
+async fn project_is_empty(
+ endpoint: &ServerEndpoint,
+ workspace: &str,
+ project: &str,
+) -> Result<bool> {
+ match get_json::<ByAgentResponse>(
+ endpoint,
+ "/admin/sessions/by-agent",
+ &[("workspace", workspace), ("project", project)],
+ )
+ .await
+ {
+ Ok(response) => Ok(response.by_agent.iter().map(|c| c.sessions).sum::<u64>() == 0),
+ // A project that has never been written to does not exist server-side
+ // yet, so the no-create lookup answers 404 — which is the strongest
+ // possible "empty", and the common case for a first-time backfill.
+ Err(error) if super::is_scope_not_found(&error) => Ok(true),
+ Err(error) => Err(error).with_context(|| {
+ format!("checking whether {workspace}/{project} already has captured sessions")
+ }),
+ }
+}
+
+/// Enumerate local native sessions for `cwd` across every scanned harness.
+/// Read-only; a harness whose store is absent/unreadable contributes nothing.
+async fn collect_local_sessions(
+ home: &Path,
+ cwd: &Path,
+ only_session: Option<&str>,
+) -> Vec<SessionRef> {
+ let mut out = Vec::new();
+ for &harness in SCANNED_HARNESSES {
+ let session_dir = build_launch_plan(harness, None, Vec::new(), None)
+ .ok()
+ .and_then(|plan| plan.session_dir);
+ let Ok(sessions) = list_native_sessions(
+ harness,
+ home,
+ cwd,
+ session_dir.as_deref(),
+ PER_HARNESS_SCAN_LIMIT,
+ )
+ .await
+ else {
+ continue;
+ };
+ for session in sessions {
+ if only_session.is_some_and(|want| want != session.native_session_id) {
+ continue;
+ }
+ out.push(SessionRef {
+ harness,
+ native_session_id: session.native_session_id,
+ updated_at: session.updated_at,
+ });
+ }
+ }
+ out
+}
+
+/// One item in a `POST /hook/batch` request: the full hook URL (whose query the
+/// server parses for event/agent/scope/session) plus the JSON body.
+#[derive(Debug, Serialize)]
+struct HookItem {
+ url: String,
+ body: serde_json::Value,
+}
+
+/// The server's `/hook/batch` acknowledgement (subset we act on).
+#[derive(Debug, Deserialize)]
+struct HookBatchAck {
+ /// Contiguous leading prefix committed, oldest-first.
+ accepted: usize,
+}
+
+/// Import one native session by replaying its transcript through `/hook`, the
+/// same ingress live capture uses — so the events become real sanitized
+/// observations that consolidate into pages and are searchable, attributed to
+/// the original harness and its native session id. Returns the number of
+/// content events (messages/tools) imported. The server sanitizes and bounds
+/// every event, so retroactive text crosses the same trust boundary as live
+/// capture.
+async fn import_one(
+ endpoint: &ServerEndpoint,
+ workspace: &str,
+ project: &str,
+ home: &Path,
+ cwd: &Path,
+ session: &SessionRef,
+) -> Result<usize> {
+ let session_dir = build_launch_plan(session.harness, None, Vec::new(), None)
+ .ok()
+ .and_then(|plan| plan.session_dir);
+ // These are historical sessions, so the flush wait is a quick no-op; ignore
+ // its result and read whatever is on disk.
+ let _ = wait_for_transcript_flush(
+ session.harness,
+ home,
+ cwd,
+ session_dir.as_deref(),
+ &session.native_session_id,
+ )
+ .await;
+ let transcript = export_transcript(
+ session.harness,
+ home,
+ cwd,
+ session_dir.as_deref(),
+ &session.native_session_id,
+ None,
+ )
+ .await
+ .with_context(|| format!("reading the {} transcript", session.harness.as_str()))?;
+
+ let sid = &session.native_session_id;
+ let agent = session.harness.agent_kind().as_str();
+ let mut items = Vec::with_capacity(transcript.events.len() + 2);
+ items.push(hook_item(
+ endpoint,
+ workspace,
+ project,
+ agent,
+ "session-start",
+ sid,
+ &format!("{sid}:session-start"),
+ None,
+ serde_json::json!({ "session_id": sid }),
+ )?);
+ let mut content = 0usize;
+ for event in &transcript.events {
+ if let Some(mapped) = map_event(sid, event) {
+ items.push(hook_item(
+ endpoint,
+ workspace,
+ project,
+ agent,
+ &mapped.event,
+ sid,
+ &mapped.ingest_key,
+ mapped.source_event.as_deref(),
+ mapped.body,
+ )?);
+ content += 1;
+ }
+ }
+ items.push(hook_item(
+ endpoint,
+ workspace,
+ project,
+ agent,
+ "session-end",
+ sid,
+ &format!("{sid}:session-end"),
+ None,
+ serde_json::json!({ "session_id": sid }),
+ )?);
+
+ post_hook_items(endpoint, &items).await?;
+ Ok(content)
+}
+
+/// A transcript event mapped to its `/hook` shape.
+struct MappedEvent {
+ event: String,
+ source_event: Option<String>,
+ ingest_key: String,
+ body: serde_json::Value,
+}
+
+/// Map one transcript event to a hook event. User messages become the canonical
+/// `user-prompt` observation; every other content-bearing event is recorded as
+/// a backfill extension observation. Non-content boundary events (compaction,
+/// checkpoint, annotation) are dropped — they are not session content.
+fn map_event(session_id: &str, event: &NewWorkstreamEvent) -> Option<MappedEvent> {
+ let content = event.content.trim();
+ if content.is_empty() {
+ return None;
+ }
+ let role = event.role.as_deref().unwrap_or("");
+ let ingest_key = format!("{session_id}:{}", event.event_id);
+ match event.kind {
+ WorkstreamEventKind::Message if role == "user" || role == "human" => Some(MappedEvent {
+ event: "user-prompt".to_string(),
+ source_event: None,
+ ingest_key,
+ body: serde_json::json!({ "session_id": session_id, "prompt": event.content }),
+ }),
+ WorkstreamEventKind::Message
+ | WorkstreamEventKind::ToolCall
+ | WorkstreamEventKind::ToolResult => {
+ let source_event = match event.kind {
+ WorkstreamEventKind::ToolCall => "tool_call".to_string(),
+ WorkstreamEventKind::ToolResult => "tool_result".to_string(),
+ _ if role.is_empty() => "message".to_string(),
+ _ => format!("{role}-message"),
+ };
+ Some(MappedEvent {
+ event: format!("backfill.{source_event}"),
+ source_event: Some(source_event),
+ ingest_key,
+ body: serde_json::json!({
+ "session_id": session_id,
+ "title": first_line(content),
+ "message": event.content,
+ }),
+ })
+ }
+ WorkstreamEventKind::Compaction
+ | WorkstreamEventKind::Checkpoint
+ | WorkstreamEventKind::Annotation => None,
+ }
+}
+
+/// A short one-line title from the event content.
+fn first_line(content: &str) -> String {
+ let line = content.lines().next().unwrap_or("").trim();
+ let capped: String = line.chars().take(120).collect();
+ if capped.is_empty() {
+ "(backfilled event)".to_string()
+ } else {
+ capped
+ }
+}
+
+/// Build a `/hook` URL whose query the server parses for event/agent/scope. The
+/// URL is data the batch item carries, not the request target (that is
+/// `/hook/batch`), so it is built against this server's origin exactly as a live
+/// hook would have spooled it.
+#[allow(clippy::too_many_arguments)]
+fn hook_item(
+ endpoint: &ServerEndpoint,
+ workspace: &str,
+ project: &str,
+ agent: &str,
+ event: &str,
+ session_id: &str,
+ ingest_key: &str,
+ source_event: Option<&str>,
+ body: serde_json::Value,
+) -> Result<HookItem> {
+ let mut url = reqwest::Url::parse(&endpoint.build_url("/hook"))
+ .context("building the hook URL for backfill")?;
+ {
+ let mut query = url.query_pairs_mut();
+ query
+ .append_pair("event", event)
+ .append_pair("agent", agent)
+ .append_pair("workspace", workspace)
+ .append_pair("project", project)
+ .append_pair("project_src", "marker")
+ .append_pair("session_id", session_id)
+ .append_pair("ingest_key", ingest_key);
+ if let Some(source_event) = source_event {
+ query
+ .append_pair("extension", BACKFILL_EXTENSION)
+ .append_pair("source_event", source_event);
+ }
+ }
+ Ok(HookItem {
+ url: url.into(),
+ body,
+ })
+}
+
+/// POST the events in server-sized batches, retrying the unaccepted suffix of a
+/// partially accepted batch (per-source rate limiting can skip a tail) before
+/// failing. Events are oldest-first, so a leading-prefix ack is safe to resume.
+async fn post_hook_items(endpoint: &ServerEndpoint, items: &[HookItem]) -> Result<()> {
+ for chunk in items.chunks(HOOK_BATCH_ITEMS) {
+ let mut offset = 0usize;
+ let mut stalled = 0u32;
+ while offset < chunk.len() {
+ let batch: Vec<&HookItem> = chunk[offset..].iter().collect();
+ let ack: HookBatchAck = post_json(endpoint, "/hook/batch", &batch)
+ .await
+ .context("replaying transcript events through /hook/batch")?;
+ if ack.accepted == 0 {
+ stalled += 1;
+ if stalled >= 5 {
+ bail!(
+ "server accepted none of a hook batch after {stalled} attempts \
+ (rate limited or saturated); rerun to resume"
+ );
+ }
+ tokio::time::sleep(Duration::from_millis(200 * u64::from(stalled))).await;
+ continue;
+ }
+ stalled = 0;
+ offset += ack.accepted.min(chunk.len() - offset);
+ }
+ }
+ Ok(())
+}
+
+fn finish(args: &crate::cli::BackfillArgs, report: &BackfillReport) -> Result<()> {
+ if args.json {
+ println!("{}", serde_json::to_string_pretty(report)?);
+ return Ok(());
+ }
+ if args.quiet {
+ return Ok(());
+ }
+ if report.skipped_non_empty {
+ println!(
+ "ai-memory: {}/{} already has captured sessions — nothing to backfill (use --force to import anyway).",
+ report.workspace, report.project
+ );
+ return Ok(());
+ }
+ if report.dry_run {
+ println!(
+ "ai-memory: would import {} local session(s) into {}/{} (dry run).",
+ report.selected, report.workspace, report.project
+ );
+ return Ok(());
+ }
+ if report.imported_sessions == 0 && report.failed_sessions == 0 {
+ println!(
+ "ai-memory: no local session history found to backfill for {}/{}.",
+ report.workspace, report.project
+ );
+ return Ok(());
+ }
+ let mut line = format!(
+ "📼 ai-memory imported {} prior local session(s) (~{} events) for {}/{}.",
+ report.imported_sessions, report.imported_events, report.workspace, report.project
+ );
+ if report.skipped_for_cap > 0 {
+ line.push_str(&format!(
+ " {} older session(s) skipped (import cap).",
+ report.skipped_for_cap
+ ));
+ }
+ line.push_str(" Opt out with AI_MEMORY_BACKFILL_ON_START=false.");
+ println!("{line}");
+ Ok(())
+}
+
+fn display_id(id: &str) -> &str {
+ id.get(..8).unwrap_or(id)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use std::time::Duration;
+
+ fn session(id: &str, secs_ago: u64) -> SessionRef {
+ SessionRef {
+ harness: ManagedHarness::Claude,
+ native_session_id: id.to_string(),
+ updated_at: SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000 - secs_ago),
+ }
+ }
+
+ #[test]
+ fn select_sessions_keeps_the_newest_up_to_the_cap() {
+ let all = vec![session("old", 300), session("new", 10), session("mid", 100)];
+ let picked = select_sessions(all, 2);
+ assert_eq!(
+ picked
+ .iter()
+ .map(|s| s.native_session_id.as_str())
+ .collect::<Vec<_>>(),
+ vec!["new", "mid"],
+ "newest first, capped to 2"
+ );
+ }
+
+ #[test]
+ fn select_sessions_cap_zero_is_raised_to_one_by_caller_contract() {
+ // The caller passes max_sessions.max(1); select_sessions itself honors
+ // whatever it is given, so a literal 0 yields nothing.
+ assert!(select_sessions(vec![session("a", 1)], 0).is_empty());
+ assert_eq!(select_sessions(vec![session("a", 1)], 1).len(), 1);
+ }
+
+ #[test]
+ fn sentinel_path_is_cwd_specific_and_path_safe() {
+ let dir = Path::new("/data");
+ let a = sentinel_path(dir, Path::new("/home/me/projects/app"));
+ let b = sentinel_path(dir, Path::new("/home/me/projects/other"));
+ let c = sentinel_path(dir, Path::new("/home/me/projects/app"));
+ assert_eq!(a, c, "same cwd -> same path");
+ assert_ne!(a, b, "different cwd -> different path");
+ assert!(
+ a.starts_with("/data/backfill-state/"),
+ "under the data dir: {a:?}"
+ );
+ // A path with separators hashes to a single flat, path-safe leaf.
+ assert_eq!(
+ a.strip_prefix("/data/backfill-state/")
+ .unwrap()
+ .components()
+ .count(),
+ 1,
+ "the hashed leaf must be a single path component: {a:?}"
+ );
+ }
+
+ fn event(kind: WorkstreamEventKind, role: Option<&str>, content: &str) -> NewWorkstreamEvent {
+ NewWorkstreamEvent {
+ event_id: "evt-1".to_string(),
+ agent: ai_memory_core::AgentKind::ClaudeCode,
+ native_session_id: "sid".to_string(),
+ source_record_id: None,
+ kind,
+ role: role.map(str::to_string),
+ content: content.to_string(),
+ occurred_at: None,
+ metadata: serde_json::Value::Null,
+ }
+ }
+
+ #[test]
+ fn user_message_maps_to_the_canonical_user_prompt() {
+ let m = map_event(
+ "sid",
+ &event(WorkstreamEventKind::Message, Some("user"), "do the thing"),
+ )
+ .expect("user message maps");
+ assert_eq!(m.event, "user-prompt");
+ assert!(
+ m.source_event.is_none(),
+ "user-prompt is a lifecycle event, not an extension"
+ );
+ assert_eq!(m.body["prompt"], "do the thing");
+ assert_eq!(
+ m.ingest_key, "sid:evt-1",
+ "ingest key is stable per source event"
+ );
+ }
+
+ #[test]
+ fn assistant_and_tool_events_map_to_backfill_extension_observations() {
+ let a = map_event(
+ "sid",
+ &event(
+ WorkstreamEventKind::Message,
+ Some("assistant"),
+ "here is the plan\nline2",
+ ),
+ )
+ .expect("assistant maps");
+ assert_eq!(a.event, "backfill.assistant-message");
+ assert_eq!(a.source_event.as_deref(), Some("assistant-message"));
+ assert_eq!(
+ a.body["title"], "here is the plan",
+ "title is the first line"
+ );
+ assert_eq!(a.body["message"], "here is the plan\nline2");
+
+ let t = map_event(
+ "sid",
+ &event(WorkstreamEventKind::ToolCall, None, "grep foo"),
+ )
+ .expect("tool call maps");
+ assert_eq!(t.event, "backfill.tool_call");
+ assert_eq!(t.source_event.as_deref(), Some("tool_call"));
+ }
+
+ #[test]
+ fn empty_and_boundary_events_are_dropped() {
+ assert!(
+ map_event(
+ "sid",
+ &event(WorkstreamEventKind::Message, Some("user"), " ")
+ )
+ .is_none(),
+ "whitespace-only content is not an observation"
+ );
+ for kind in [
+ WorkstreamEventKind::Compaction,
+ WorkstreamEventKind::Checkpoint,
+ WorkstreamEventKind::Annotation,
+ ] {
+ assert!(
+ map_event("sid", &event(kind, None, "boundary")).is_none(),
+ "{kind:?} is not session content"
+ );
+ }
+ }
+
+ /// End-to-end proof that `collect_local_sessions` wires to the real
+ /// workstream path encoder: a Claude transcript planted under the actual
+ /// `~/.claude/projects/<enc-cwd>/` layout for this cwd is discovered, and a
+ /// transcript for a different cwd is ignored.
+ #[tokio::test]
+ async fn collect_local_sessions_finds_a_planted_claude_session_for_this_cwd() {
+ let home = tempfile::tempdir().unwrap();
+ let cwd = tempfile::tempdir().unwrap();
+ let other_cwd = tempfile::tempdir().unwrap();
+
+ let session_dir = home
+ .path()
+ .join(".claude")
+ .join("projects")
+ .join(cwd.path().to_string_lossy().replace('/', "-"));
+ std::fs::create_dir_all(&session_dir).unwrap();
+ let header = serde_json::json!({
+ "sessionId": "11111111-2222-3333-4444-555555555555",
+ "cwd": cwd.path().to_string_lossy(),
+ });
+ std::fs::write(session_dir.join("sess.jsonl"), format!("{header}\n")).unwrap();
+ let foreign = serde_json::json!({
+ "sessionId": "99999999-8888-7777-6666-555555555555",
+ "cwd": other_cwd.path().to_string_lossy(),
+ });
+ std::fs::write(session_dir.join("foreign.jsonl"), format!("{foreign}\n")).unwrap();
+
+ let found = collect_local_sessions(home.path(), cwd.path(), None).await;
+ let claude: Vec<_> = found
+ .iter()
+ .filter(|s| s.harness == ManagedHarness::Claude)
+ .collect();
+ assert_eq!(claude.len(), 1, "only the matching-cwd session: {found:?}");
+ assert_eq!(
+ claude[0].native_session_id,
+ "11111111-2222-3333-4444-555555555555"
+ );
+
+ // `--session` narrows to one id.
+ let only = collect_local_sessions(home.path(), cwd.path(), Some("nope")).await;
+ assert!(only.is_empty(), "no session matches the filter: {only:?}");
+ }
+
+ /// The automatic path must honor the `backfill_on_start` opt-out: it records
+ /// the attempt (so it never re-spawns) and returns without contacting the
+ /// server at all.
+ #[tokio::test]
+ async fn auto_run_opted_out_writes_sentinel_and_makes_no_request() {
+ let home = tempfile::tempdir().unwrap();
+ let data_dir = tempfile::tempdir().unwrap();
+ let mut config =
+ crate::config::Config::load(None, Some(home.path().to_path_buf())).unwrap();
+ config.data_dir = data_dir.path().to_path_buf();
+ config.backfill_on_start = false;
+ // An unroutable server would make any request hang/fail; the opt-out
+ // must return before we ever build the endpoint.
+ config.server_url = "http://127.0.0.1:9".to_string();
+
+ let args = crate::cli::BackfillArgs {
+ workspace: None,
+ project: None,
+ session: None,
+ force: false,
+ dry_run: false,
+ max_sessions: 25,
+ json: false,
+ quiet: true,
+ auto: true,
+ };
+ run(&config, args)
+ .await
+ .expect("opted-out auto run must succeed without contacting the server");
+
+ let cwd = std::env::current_dir().unwrap();
+ assert!(
+ sentinel_path(&config.data_dir, &cwd).exists(),
+ "the opt-out must still record the attempt so it does not re-spawn"
+ );
+ }
+}
diff --git a/crates/ai-memory-cli/src/commands/doctor.rs b/crates/ai-memory-cli/src/commands/doctor.rs
index 0e9ddc16..5bfbddbb 100644
--- a/crates/ai-memory-cli/src/commands/doctor.rs
+++ b/crates/ai-memory-cli/src/commands/doctor.rs
@@ -36,7 +36,7 @@ use crate::http_client::{ServerEndpoint, get_json};
/// an [`AgentKind`] (Kiro v2/v3, OpenCode v1/v2) are both scanned — they read
/// distinct on-disk stores — and their local counts fold together under the
/// one agent kind the server records.
-const SCANNED_HARNESSES: &[ManagedHarness] = &[
+pub(crate) const SCANNED_HARNESSES: &[ManagedHarness] = &[
ManagedHarness::Claude,
ManagedHarness::Codex,
ManagedHarness::OpenCode,
@@ -236,7 +236,7 @@ pub async fn run(config: &Config, args: crate::cli::DoctorArgs) -> Result<()> {
super::run::native_home(config).context("locating the local harness session stores")?;
let ep = ServerEndpoint::from_config_resolving_auth(config).await;
- let captured: BTreeMap<String, u64> = get_json::<ByAgentResponse>(
+ let captured: BTreeMap<String, u64> = match get_json::<ByAgentResponse>(
&ep,
"/admin/sessions/by-agent",
&[
@@ -245,13 +245,22 @@ pub async fn run(config: &Config, args: crate::cli::DoctorArgs) -> Result<()> {
],
)
.await
- .with_context(|| {
- format!("asking the server for captured session counts for {workspace}/{project}")
- })?
- .by_agent
- .into_iter()
- .map(|c| (c.agent, c.sessions))
- .collect();
+ {
+ Ok(response) => response
+ .by_agent
+ .into_iter()
+ .map(|c| (c.agent, c.sessions))
+ .collect(),
+ // A project that has never been captured into does not exist
+ // server-side yet (404) — that is "nothing captured", not an error, so
+ // every local harness correctly reads as uncaptured.
+ Err(error) if super::is_scope_not_found(&error) => BTreeMap::new(),
+ Err(error) => {
+ return Err(error).with_context(|| {
+ format!("asking the server for captured session counts for {workspace}/{project}")
+ });
+ }
+ };
let local = scan_local(&home, &cwd, args.since_days).await;
let rows = build_rows(&local, &captured);
diff --git a/crates/ai-memory-cli/src/commands/hook.rs b/crates/ai-memory-cli/src/commands/hook.rs
index 7ec3ae45..d241446b 100644
--- a/crates/ai-memory-cli/src/commands/hook.rs
+++ b/crates/ai-memory-cli/src/commands/hook.rs
@@ -710,6 +710,19 @@ where
.await;
}
+ // session-start: if this checkout has never had the one-time boot backfill
+ // attempted, spawn it detached. It self-gates (config opt-out, empty-store
+ // check) and records the attempt, so this is at most one extra process the
+ // first time a project is opened after installing hooks — never inline, so
+ // the SessionStart budget is untouched. Best-effort; a spawn failure must
+ // not affect session start.
+ if args.event == "session-start"
+ && let Ok(trigger_cwd) = std::env::current_dir()
+ && !super::backfill::sentinel_path(&dd, &trigger_cwd).exists()
+ {
+ let _ = hook_drain_process::spawn_backfill(&dd);
+ }
+
// session-start: drain any backlog (e.g. from a previous session that ended
// abruptly), then fetch + inject the pending handoff for the resuming agent.
if args.event == "session-start" {
diff --git a/crates/ai-memory-cli/src/commands/hook_drain_process.rs b/crates/ai-memory-cli/src/commands/hook_drain_process.rs
index 7f38cc84..d3b046fb 100644
--- a/crates/ai-memory-cli/src/commands/hook_drain_process.rs
+++ b/crates/ai-memory-cli/src/commands/hook_drain_process.rs
@@ -92,6 +92,36 @@ pub fn command_spec(data_dir: &Path, live_token: Option<&str>) -> io::Result<Dra
})
}
+/// Build `ai-memory --data-dir <dir> backfill --auto --quiet`.
+///
+/// The one-time boot backfill of pre-hook local history is potentially long
+/// (it reads native transcripts and imports them), so it runs detached exactly
+/// like the drainer rather than inline in the SessionStart hook's tight budget.
+/// It carries no bearer on its argv or in its environment: `backfill` resolves
+/// auth from config / the persisted hook token itself, and the automatic path
+/// only runs in the loopback single-operator posture where no token is needed.
+pub fn backfill_command_spec(data_dir: &Path) -> io::Result<DrainCommandSpec> {
+ Ok(DrainCommandSpec {
+ exe: std::env::current_exe()?,
+ args: vec![
+ OsString::from("--data-dir"),
+ data_dir.as_os_str().to_os_string(),
+ OsString::from("backfill"),
+ OsString::from("--auto"),
+ OsString::from("--quiet"),
+ ],
+ stderr_log: data_dir.join("logs").join("backfill.log"),
+ live_token: None,
+ })
+}
+
+/// Spawn the detached one-time boot backfill without inheriting hook stdio.
+/// Best-effort: a spawn failure must never break session start.
+pub fn spawn_backfill(data_dir: &Path) -> io::Result<()> {
+ let spec = backfill_command_spec(data_dir)?;
+ spawn_spec(&spec)
+}
+
/// Build the stdio/detach shape used when spawning.
#[must_use]
pub fn spawn_config(spec: &DrainCommandSpec, try_breakaway: bool) -> SpawnConfig {
diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rs
index 9cfd610c..bc164b67 100644
--- a/crates/ai-memory-cli/src/commands/mod.rs
+++ b/crates/ai-memory-cli/src/commands/mod.rs
@@ -4,12 +4,26 @@ use anyhow::{Context, Result, anyhow, bail};
use crate::config::Config;
+/// Whether a server error is a no-create scope lookup missing its
+/// `(workspace, project)` — a `404`. Read/count endpoints fail closed on an
+/// unknown scope (invariant: reads never create), so a project that has never
+/// been written to answers `404` rather than "empty". Callers that ask "does
+/// this project have anything?" must treat that as **nothing there**, not a
+/// hard error — otherwise the brand-new-project case (exactly the one
+/// `doctor` and `backfill` exist for) fails on itself.
+pub(crate) fn is_scope_not_found(error: &anyhow::Error) -> bool {
+ error
+ .downcast_ref::<crate::http_client::ServerResponseError>()
+ .is_some_and(|response| response.status() == reqwest::StatusCode::NOT_FOUND)
+}
+
pub mod api_key;
pub mod apply_shared;
pub mod audit_contamination;
pub mod auth;
pub mod auto_improve;
pub mod auto_improve_report;
+pub mod backfill;
pub mod backup;
pub mod bootstrap;
pub mod checkpoints;
diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rs
index 48cbd269..f0b33875 100644
--- a/crates/ai-memory-cli/src/config.rs
+++ b/crates/ai-memory-cli/src/config.rs
@@ -264,6 +264,15 @@ pub struct Config {
/// `install-hooks --capture-assistant`. Set with
/// `AI_MEMORY_CAPTURE_ASSISTANT=true`.
pub capture_assistant: bool,
+ /// On by default. When the project's ai-memory store is brand new (empty),
+ /// the SessionStart hook triggers a one-time, bounded import of this
+ /// project's existing local harness session history so installing hooks
+ /// mid-project does not start amnesiac. The import is read-only on the local
+ /// side, sanitized on the server exactly like live capture, and only ever
+ /// bootstraps an empty project (never overwrites an established one). Turn
+ /// off with `AI_MEMORY_BACKFILL_ON_START=false` or `backfill_on_start =
+ /// false`; `ai-memory backfill` remains available to run it by hand.
+ pub backfill_on_start: bool,
/// Strip root-level `anyOf`/`oneOf`/`allOf` from MCP tool input
/// schemas (e.g. `memory_read_page`'s "exactly one of path/query"
/// contract) on every `tools/list`, regardless of client or `?flavor=`
@@ -718,6 +727,7 @@ impl Default for Config {
llm_fallback_configs: Vec::new(),
consolidate_on_session_end: false,
capture_assistant: false,
+ backfill_on_start: true,
strip_root_combinators: false,
gemini_safe_schemas: false,
reranker: None,
diff --git a/crates/ai-memory-cli/src/lib.rs b/crates/ai-memory-cli/src/lib.rs
index 25f6430e..ab2ece38 100644
--- a/crates/ai-memory-cli/src/lib.rs
+++ b/crates/ai-memory-cli/src/lib.rs
@@ -85,6 +85,7 @@ pub async fn run() -> Result<()> {
Command::Init(args) => commands::init::run(&config, args, config_path.as_deref()),
Command::Status(args) => commands::status::run(&config, args).await,
Command::Doctor(args) => commands::doctor::run(&config, args).await,
+ Command::Backfill(args) => commands::backfill::run(&config, args).await,
Command::Run(args) => {
let exit_code = commands::run::run(&config, args).await?;
if exit_code != 0 {
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index a7687eb8..f7161f07 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -484,7 +484,7 @@ move-project move-session uninstall
auth user completions
handoffs purge-session compact
api-key export-okf message
-doctor
+doctor backfill
```
Run `ai-memory --help` for the full tree.
diff --git a/docs/cookbook.md b/docs/cookbook.md
index 8c82c3f7..f63c4f50 100644
--- a/docs/cookbook.md
+++ b/docs/cookbook.md
@@ -94,6 +94,7 @@ Most people never need these — the agent does it — but they exist:
```bash
ai-memory status # counts, paths, health
ai-memory doctor # is every harness that ran here captured?
+ai-memory backfill # import prior local history into an empty store
ai-memory write-page … # save a durable page
ai-memory handoffs # list open handoffs for the project
ai-memory message list # cross-project inbox
@@ -110,6 +111,13 @@ ai-memory serve # run the server
captured them — so a harness you rotated in without installing its hook (a
silent gap: it keeps its own local history while capturing nothing) shows up
as a warning with the exact `install-hooks` command to fix it.
+- **I just installed hooks in a project I've worked in for a while**: the first
+ time you open the project after installing, ai-memory imports your existing
+ local session history once (bounded, sanitized on the server, only into an
+ empty store) so it isn't starting from nothing. It runs automatically in the
+ background; you can trigger or preview it yourself with `ai-memory backfill`
+ (`--dry-run` to see what it would import), or turn it off with
+ `AI_MEMORY_BACKFILL_ON_START=false`.
- **A search misses something you saved**: confirm the scope — memory is
per-project; a page saved in project A is not returned in project B unless it
was written to the global scope or you query with an explicit scope.
diff --git a/docs/design-boot-backfill.md b/docs/design-boot-backfill.md
index 43405b6f..770897e7 100644
--- a/docs/design-boot-backfill.md
+++ b/docs/design-boot-backfill.md
@@ -1,9 +1,32 @@
-# Design proposal: boot-time backfill of pre-hook local history
-
-**Status: proposal for review — not implemented.** This is the design pass for the
-"pre-load existing session history when the store is new" idea. It adds a new default-on
-capture behavior and touches the trust boundary (retroactive untrusted text enters the
-store), so it is deliberately separated from implementation.
+# Design: boot-time backfill of pre-hook local history
+
+**Status: implemented on `release/2.3`.** This is the design pass for the "pre-load
+existing session history when the store is new" idea; it adds a default-on capture
+behavior and touches the trust boundary (retroactive untrusted text enters the store).
+
+**As-built note.** Two deviations from the approved plan, both verified by a live smoke:
+
+1. **Ingest path is B, not A.** The plan recommended path A (reuse the managed-workstream
+ begin → link → finish endpoints, for the incremental cursor). A live smoke proved A
+ populates the *workstream continuity ledger*, not the `sessions`/`observations`/`pages`
+ pipeline — `observation` count stayed zero after import, so the history was not
+ recall-able via `memory_query`. The implementation therefore **replays each transcript
+ through `/hook/batch`** (the same ingress live capture uses, as the companion importer
+ does): `export_transcript` normalizes the native transcript, then each event is posted
+ as a session-start / `user-prompt` / backfill-extension observation, attributed to the
+ original harness + native session id. The re-run smoke confirmed
+ `sessions 0→1, observations 0→5, pages 0→1`, by-agent `claude-code: 1`, and a correct
+ no-op on the second run.
+2. **No new lease table (V65 dropped).** "Import exactly once" is held by the emptiness
+ gate (only bootstrap a project with zero captured sessions) plus a per-checkout local
+ sentinel (`<data_dir>/backfill-state/<hash(cwd)>`) that stops the SessionStart trigger
+ re-spawning. Live hook capture (install-time forward) and backfill (before-install
+ history) do not overlap, so there is no double-capture to reconcile.
+
+The automatic path runs detached and silent (like Claude Code's own auto-memory); the
+`📼 imported N session(s)` summary is shown on a **manual** `ai-memory backfill` only.
+Surfacing the count in the next session's on-start context is a possible follow-up
+(§ "Notice delivery" describes the delivery path it would reuse).
## Problem