Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(cli): boot-time backfill of pre-hook local history (on by default)
AkitaOnRailsc380278ef677
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 22e63013..95fed696 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Added+- Boot-time backfill of pre-hook local history, on by default. When a project's+ ai-memory store is brand new (empty), the SessionStart hook triggers a+ one-time, bounded import of that project's existing local harness transcripts+ so installing hooks mid-project no longer starts amnesiac about the very+ session you are resuming. It replays each local transcript through `/hook` —+ the same ingress live capture uses — so the history becomes real sessions and+ observations that consolidate into pages and are searchable via `memory_query`,+ **sanitized and bounded on the server exactly like live capture** and+ attributed to the original harness. It only ever bootstraps an empty project+ (never overwrites an established one; live capture from install-time forward+ and backfill of before-install history do not overlap), is hard-capped (newest+ 25 sessions, 50k events), and runs detached so session start is never blocked.+ New `ai-memory backfill`+ subcommand runs it by hand (`--dry-run`, `--force`, `--session`, `--json`).+ Opt out with `AI_MEMORY_BACKFILL_ON_START=false` / `backfill_on_start = false`.- New `ai-memory doctor` command: a capture-coverage check that, for the currentproject, compares every known harness's local (on-disk) native session storeagainst what the server actually captured (`GET /admin/sessions/by-agent`) anddiff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rsindex 35b709ed..7cbf5f58 100644--- a/crates/ai-memory-cli/src/cli.rs+++ b/crates/ai-memory-cli/src/cli.rs@@ -37,6 +37,11 @@ pub enum Command {/// project against what the server captured, and warn when a harness ran/// here recently but has no captured sessions (its hook is likely missing).Doctor(DoctorArgs),+ /// One-time import of this project's existing local harness session history+ /// into a brand-new (empty) ai-memory store, so installing hooks+ /// mid-project doesn't start amnesiac. No-op once the store has any+ /// sessions unless `--force`.+ Backfill(BackfillArgs),/// Launch an agent in an opt-in, cross-harness managed workstream./// Native arguments are forwarded except exact wrapper flags such as/// `--yolo` and `--fresh`.@@ -1368,6 +1373,43 @@ pub struct StatusArgs {pub json: bool,}+/// Arguments for `backfill`.+#[derive(Debug, Args)]+pub struct BackfillArgs {+ /// Workspace name. Defaults to the current project's resolved scope.+ #[arg(long)]+ pub workspace: Option<String>,+ /// Project name. Defaults to the current project's resolved scope.+ #[arg(long)]+ pub project: Option<String>,+ /// Import only this one harness-native session id instead of every local+ /// session for the project.+ #[arg(long)]+ pub session: Option<String>,+ /// Import even when the store already has sessions. Off by default: the+ /// automatic path is a one-time bootstrap of an empty project only.+ #[arg(long)]+ pub force: bool,+ /// Report what would be imported without importing anything.+ #[arg(long)]+ pub dry_run: bool,+ /// Import at most this many of the newest local sessions.+ #[arg(long, default_value_t = 25)]+ pub max_sessions: usize,+ /// Emit the report as JSON instead of human-readable text.+ #[arg(long)]+ pub json: bool,+ /// Suppress the human summary line (used by the automatic SessionStart+ /// trigger, which runs detached).+ #[arg(long)]+ pub quiet: bool,+ /// Internal: this run was spawned by the SessionStart trigger. Honors the+ /// `backfill_on_start` opt-out and records that the automatic bootstrap has+ /// been attempted for this checkout. Not for interactive use.+ #[arg(long, hide = true)]+ pub auto: bool,+}+/// Arguments for `doctor`.#[derive(Debug, Args)]pub struct DoctorArgs {diff --git a/crates/ai-memory-cli/src/commands/backfill.rs b/crates/ai-memory-cli/src/commands/backfill.rsnew file mode 100644index 00000000..75898dae--- /dev/null+++ b/crates/ai-memory-cli/src/commands/backfill.rs@@ -0,0 +1,790 @@+//! `ai-memory backfill` — one-time import of a project's pre-hook local history.+//!+//! Capture is forward-only: hooks record events only from the moment they are+//! installed. If you have worked in a project for weeks and only now install+//! ai-memory, the tool that exists for cross-session continuity starts+//! amnesiac about the very session you are resuming. This command fills that+//! first-boot gap: when the project's store is **brand new (empty)**, it+//! imports the existing local harness transcript history once.+//!+//! For each local native session it **replays the transcript through `/hook`**,+//! the same ingress live capture uses (as the companion importer does for+//! external conversations):+//!+//! - `export_transcript` (ai-memory-workstream) reads the harness's native+//! transcript read-only into a normalized, bounded visible-event ledger,+//! doing all the per-harness parsing in one place;+//! - each event is posted to `/hook/batch` as a session-start, `user-prompt`,+//! or backfill-extension observation, attributed to the original harness and+//! its native session id — so it becomes a real session + observations that+//! consolidate into pages and are searchable via `memory_query`, exactly like+//! live capture. The server **sanitizes and bounds every event**, so+//! retroactive text crosses the same trust boundary as live capture.+//!+//! (An earlier revision imported through the managed-workstream begin/finish+//! endpoints, but that populates the `ai-memory run` continuity ledger, not the+//! searchable memory pipeline — verified by a live smoke where the observation+//! count stayed zero. `/hook` replay is what makes the history recall-able.)+//!+//! Safety: the automatic path only ever bootstraps an **empty** project (never+//! overwrites an established one — hook capture from install-time forward and+//! backfill of before-install history do not overlap), and it runs at most once+//! per checkout (a local sentinel).++use std::path::Path;+use std::path::PathBuf;+use std::time::{Duration, SystemTime};++use anyhow::{Context, Result, bail};+use serde::{Deserialize, Serialize};++use ai_memory_core::{NewWorkstreamEvent, WorkstreamEventKind};+use ai_memory_workstream::{+ ManagedHarness, build_launch_plan, export_transcript, list_native_sessions,+ wait_for_transcript_flush,+};++use super::doctor::SCANNED_HARNESSES;+use super::run;+use crate::config::Config;+use crate::http_client::{ServerEndpoint, get_json, post_json};++/// Circuit breaker on total imported events across the whole bootstrap. Once+/// reached, no further sessions are started; the remaining sessions are+/// reported as skipped rather than partially imported.+const MAX_EVENTS_TOTAL: usize = 50_000;++/// Newest local sessions to enumerate per harness before applying `--max-sessions`.+const PER_HARNESS_SCAN_LIMIT: usize = 200;++/// One `POST /hook/batch` request carries at most this many events — matches the+/// server's `MAX_HOOK_BATCH_ITEMS` so a batch is never rejected for size.+const HOOK_BATCH_ITEMS: usize = 256;++/// The extension label backfilled non-user events (assistant/tool) are recorded+/// under, mirroring the companion importer's replay-through-`/hook` mechanism.+const BACKFILL_EXTENSION: &str = "ai-memory-backfill";++/// A local native session eligible for import.+#[derive(Debug, Clone)]+pub(crate) struct SessionRef {+ harness: ManagedHarness,+ native_session_id: String,+ updated_at: SystemTime,+}++/// The outcome of a backfill run, and the JSON output shape.+#[derive(Debug, Default, Serialize)]+struct BackfillReport {+ workspace: String,+ project: String,+ /// Sessions selected for import (after `--max-sessions`).+ selected: usize,+ /// Sessions actually imported.+ imported_sessions: usize,+ /// Events imported across all sessions.+ imported_events: usize,+ /// Sessions skipped because the total-events circuit breaker tripped.+ skipped_for_cap: usize,+ /// Per-session failures (import errors); the run continues past them.+ failed_sessions: usize,+ /// True when the store was already populated (or `--force` off) and nothing+ /// was imported.+ skipped_non_empty: bool,+ /// True for a `--dry-run` (planning only, no import).+ dry_run: bool,+}++/// The captured side: `GET /admin/sessions/by-agent` — summing its counts tells+/// us whether the project store is empty.+#[derive(Debug, Deserialize)]+struct ByAgentResponse {+ by_agent: Vec<AgentCount>,+}++#[derive(Debug, Deserialize)]+struct AgentCount {+ #[allow(dead_code)]+ agent: String,+ sessions: u64,+}++/// Newest-first, then cap to `max_sessions`. Pure, so the selection/cap policy+/// is unit-tested without a server.+pub(crate) fn select_sessions(mut all: Vec<SessionRef>, max_sessions: usize) -> Vec<SessionRef> {+ all.sort_by(|a, b| {+ b.updated_at+ .cmp(&a.updated_at)+ .then_with(|| a.native_session_id.cmp(&b.native_session_id))+ });+ all.truncate(max_sessions);+ all+}++/// Filename under `<data_dir>/backfill-state/` marking that the automatic+/// backfill has already been attempted for this checkout on this machine, so+/// the SessionStart trigger does not re-spawn on every boot. Keyed by a hash of+/// the working directory — which both the hook (before it spawns) and the+/// worker (which inherits the hook's cwd) compute identically without resolving+/// scope — so arbitrary paths stay path-safe.+pub(crate) fn sentinel_path(data_dir: &Path, cwd: &Path) -> PathBuf {+ use sha2::{Digest, Sha256};+ let mut hasher = Sha256::new();+ hasher.update(cwd.as_os_str().as_encoded_bytes());+ let digest = hasher.finalize();+ data_dir.join("backfill-state").join(format!("{digest:x}"))+}++fn write_sentinel(data_dir: &Path, cwd: &Path) {+ let path = sentinel_path(data_dir, cwd);+ if let Some(parent) = path.parent() {+ let _ = std::fs::create_dir_all(parent);+ }+ // Best-effort: a missing sentinel only means a redundant future probe, not+ // incorrect data, so a write failure is not worth failing the import over.+ let _ = std::fs::write(&path, b"");+}++/// Run the backfill.+///+/// # Errors+/// Returns an error when the scope cannot be resolved, the working directory+/// cannot be read, or the server is unreachable for the emptiness check.+pub async fn run(config: &Config, args: crate::cli::BackfillArgs) -> Result<()> {+ let cwd = std::env::current_dir().context("resolving the current working directory")?;++ // The automatic (SessionStart-spawned) path honors the opt-out and records+ // that it has been attempted for this checkout, so it runs at most once per+ // machine regardless of outcome. Manual runs ignore both.+ if args.auto && !config.backfill_on_start {+ write_sentinel(&config.data_dir, &cwd);+ return Ok(());+ }++ let (workspace, project) =+ super::resolve_scope(config, args.workspace.as_deref(), args.project.as_deref())?;+ let home = run::native_home(config).context("locating the local harness session stores")?;+ let endpoint = ServerEndpoint::from_config_resolving_auth(config).await;++ let mut report = BackfillReport {+ workspace: workspace.clone(),+ project: project.clone(),+ dry_run: args.dry_run,+ ..BackfillReport::default()+ };++ // Emptiness gate. Only bootstrap a brand-new store; an established project+ // is never retro-imported (that would duplicate hook-captured history).+ if !args.force {+ let empty = project_is_empty(&endpoint, &workspace, &project).await?;+ if !empty {+ report.skipped_non_empty = true;+ // Mark attempted so the auto-trigger stops probing this checkout.+ write_sentinel(&config.data_dir, &cwd);+ return finish(&args, &report);+ }+ }++ // Enumerate local sessions for this cwd across every supported harness.+ let candidates = collect_local_sessions(&home, &cwd, args.session.as_deref()).await;+ let selected = select_sessions(candidates, args.max_sessions.max(1));+ report.selected = selected.len();++ if args.dry_run {+ write_sentinel(&config.data_dir, &cwd);+ return finish(&args, &report);+ }++ // Import each selected session, oldest-first so the imported history reads+ // chronologically, until the total-events circuit breaker trips.+ for session in selected.into_iter().rev() {+ if report.imported_events >= MAX_EVENTS_TOTAL {+ report.skipped_for_cap += 1;+ continue;+ }+ match import_one(&endpoint, &workspace, &project, &home, &cwd, &session).await {+ Ok(events) => {+ report.imported_sessions += 1;+ report.imported_events += events;+ }+ Err(error) => {+ report.failed_sessions += 1;+ if !args.quiet {+ eprintln!(+ "ai-memory: backfill of {} session {} failed: {error:#}",+ session.harness.as_str(),+ display_id(&session.native_session_id)+ );+ }+ }+ }+ }++ write_sentinel(&config.data_dir, &cwd);+ finish(&args, &report)+}++/// Sum the server's per-agent session counts for this scope; zero means empty.+async fn project_is_empty(+ endpoint: &ServerEndpoint,+ workspace: &str,+ project: &str,+) -> Result<bool> {+ match get_json::<ByAgentResponse>(+ endpoint,+ "/admin/sessions/by-agent",+ &[("workspace", workspace), ("project", project)],+ )+ .await+ {+ Ok(response) => Ok(response.by_agent.iter().map(|c| c.sessions).sum::<u64>() == 0),+ // A project that has never been written to does not exist server-side+ // yet, so the no-create lookup answers 404 — which is the strongest+ // possible "empty", and the common case for a first-time backfill.+ Err(error) if super::is_scope_not_found(&error) => Ok(true),+ Err(error) => Err(error).with_context(|| {+ format!("checking whether {workspace}/{project} already has captured sessions")+ }),+ }+}++/// Enumerate local native sessions for `cwd` across every scanned harness.+/// Read-only; a harness whose store is absent/unreadable contributes nothing.+async fn collect_local_sessions(+ home: &Path,+ cwd: &Path,+ only_session: Option<&str>,+) -> Vec<SessionRef> {+ let mut out = Vec::new();+ for &harness in SCANNED_HARNESSES {+ let session_dir = build_launch_plan(harness, None, Vec::new(), None)+ .ok()+ .and_then(|plan| plan.session_dir);+ let Ok(sessions) = list_native_sessions(+ harness,+ home,+ cwd,+ session_dir.as_deref(),+ PER_HARNESS_SCAN_LIMIT,+ )+ .await+ else {+ continue;+ };+ for session in sessions {+ if only_session.is_some_and(|want| want != session.native_session_id) {+ continue;+ }+ out.push(SessionRef {+ harness,+ native_session_id: session.native_session_id,+ updated_at: session.updated_at,+ });+ }+ }+ out+}++/// One item in a `POST /hook/batch` request: the full hook URL (whose query the+/// server parses for event/agent/scope/session) plus the JSON body.+#[derive(Debug, Serialize)]+struct HookItem {+ url: String,+ body: serde_json::Value,+}++/// The server's `/hook/batch` acknowledgement (subset we act on).+#[derive(Debug, Deserialize)]+struct HookBatchAck {+ /// Contiguous leading prefix committed, oldest-first.+ accepted: usize,+}++/// Import one native session by replaying its transcript through `/hook`, the+/// same ingress live capture uses — so the events become real sanitized+/// observations that consolidate into pages and are searchable, attributed to+/// the original harness and its native session id. Returns the number of+/// content events (messages/tools) imported. The server sanitizes and bounds+/// every event, so retroactive text crosses the same trust boundary as live+/// capture.+async fn import_one(+ endpoint: &ServerEndpoint,+ workspace: &str,+ project: &str,+ home: &Path,+ cwd: &Path,+ session: &SessionRef,+) -> Result<usize> {+ let session_dir = build_launch_plan(session.harness, None, Vec::new(), None)+ .ok()+ .and_then(|plan| plan.session_dir);+ // These are historical sessions, so the flush wait is a quick no-op; ignore+ // its result and read whatever is on disk.+ let _ = wait_for_transcript_flush(+ session.harness,+ home,+ cwd,+ session_dir.as_deref(),+ &session.native_session_id,+ )+ .await;+ let transcript = export_transcript(+ session.harness,+ home,+ cwd,+ session_dir.as_deref(),+ &session.native_session_id,+ None,+ )+ .await+ .with_context(|| format!("reading the {} transcript", session.harness.as_str()))?;++ let sid = &session.native_session_id;+ let agent = session.harness.agent_kind().as_str();+ let mut items = Vec::with_capacity(transcript.events.len() + 2);+ items.push(hook_item(+ endpoint,+ workspace,+ project,+ agent,+ "session-start",+ sid,+ &format!("{sid}:session-start"),+ None,+ serde_json::json!({ "session_id": sid }),+ )?);+ let mut content = 0usize;+ for event in &transcript.events {+ if let Some(mapped) = map_event(sid, event) {+ items.push(hook_item(+ endpoint,+ workspace,+ project,+ agent,+ &mapped.event,+ sid,+ &mapped.ingest_key,+ mapped.source_event.as_deref(),+ mapped.body,+ )?);+ content += 1;+ }+ }+ items.push(hook_item(+ endpoint,+ workspace,+ project,+ agent,+ "session-end",+ sid,+ &format!("{sid}:session-end"),+ None,+ serde_json::json!({ "session_id": sid }),+ )?);++ post_hook_items(endpoint, &items).await?;+ Ok(content)+}++/// A transcript event mapped to its `/hook` shape.+struct MappedEvent {+ event: String,+ source_event: Option<String>,+ ingest_key: String,+ body: serde_json::Value,+}++/// Map one transcript event to a hook event. User messages become the canonical+/// `user-prompt` observation; every other content-bearing event is recorded as+/// a backfill extension observation. Non-content boundary events (compaction,+/// checkpoint, annotation) are dropped — they are not session content.+fn map_event(session_id: &str, event: &NewWorkstreamEvent) -> Option<MappedEvent> {+ let content = event.content.trim();+ if content.is_empty() {+ return None;+ }+ let role = event.role.as_deref().unwrap_or("");+ let ingest_key = format!("{session_id}:{}", event.event_id);+ match event.kind {+ WorkstreamEventKind::Message if role == "user" || role == "human" => Some(MappedEvent {+ event: "user-prompt".to_string(),+ source_event: None,+ ingest_key,+ body: serde_json::json!({ "session_id": session_id, "prompt": event.content }),+ }),+ WorkstreamEventKind::Message+ | WorkstreamEventKind::ToolCall+ | WorkstreamEventKind::ToolResult => {+ let source_event = match event.kind {+ WorkstreamEventKind::ToolCall => "tool_call".to_string(),+ WorkstreamEventKind::ToolResult => "tool_result".to_string(),+ _ if role.is_empty() => "message".to_string(),+ _ => format!("{role}-message"),+ };+ Some(MappedEvent {+ event: format!("backfill.{source_event}"),+ source_event: Some(source_event),+ ingest_key,+ body: serde_json::json!({+ "session_id": session_id,+ "title": first_line(content),+ "message": event.content,+ }),+ })+ }+ WorkstreamEventKind::Compaction+ | WorkstreamEventKind::Checkpoint+ | WorkstreamEventKind::Annotation => None,+ }+}++/// A short one-line title from the event content.+fn first_line(content: &str) -> String {+ let line = content.lines().next().unwrap_or("").trim();+ let capped: String = line.chars().take(120).collect();+ if capped.is_empty() {+ "(backfilled event)".to_string()+ } else {+ capped+ }+}++/// Build a `/hook` URL whose query the server parses for event/agent/scope. The+/// URL is data the batch item carries, not the request target (that is+/// `/hook/batch`), so it is built against this server's origin exactly as a live+/// hook would have spooled it.+#[allow(clippy::too_many_arguments)]+fn hook_item(+ endpoint: &ServerEndpoint,+ workspace: &str,+ project: &str,+ agent: &str,+ event: &str,+ session_id: &str,+ ingest_key: &str,+ source_event: Option<&str>,+ body: serde_json::Value,+) -> Result<HookItem> {+ let mut url = reqwest::Url::parse(&endpoint.build_url("/hook"))+ .context("building the hook URL for backfill")?;+ {+ let mut query = url.query_pairs_mut();+ query+ .append_pair("event", event)+ .append_pair("agent", agent)+ .append_pair("workspace", workspace)+ .append_pair("project", project)+ .append_pair("project_src", "marker")+ .append_pair("session_id", session_id)+ .append_pair("ingest_key", ingest_key);+ if let Some(source_event) = source_event {+ query+ .append_pair("extension", BACKFILL_EXTENSION)+ .append_pair("source_event", source_event);+ }+ }+ Ok(HookItem {+ url: url.into(),+ body,+ })+}++/// POST the events in server-sized batches, retrying the unaccepted suffix of a+/// partially accepted batch (per-source rate limiting can skip a tail) before+/// failing. Events are oldest-first, so a leading-prefix ack is safe to resume.+async fn post_hook_items(endpoint: &ServerEndpoint, items: &[HookItem]) -> Result<()> {+ for chunk in items.chunks(HOOK_BATCH_ITEMS) {+ let mut offset = 0usize;+ let mut stalled = 0u32;+ while offset < chunk.len() {+ let batch: Vec<&HookItem> = chunk[offset..].iter().collect();+ let ack: HookBatchAck = post_json(endpoint, "/hook/batch", &batch)+ .await+ .context("replaying transcript events through /hook/batch")?;+ if ack.accepted == 0 {+ stalled += 1;+ if stalled >= 5 {+ bail!(+ "server accepted none of a hook batch after {stalled} attempts \+ (rate limited or saturated); rerun to resume"+ );+ }+ tokio::time::sleep(Duration::from_millis(200 * u64::from(stalled))).await;+ continue;+ }+ stalled = 0;+ offset += ack.accepted.min(chunk.len() - offset);+ }+ }+ Ok(())+}++fn finish(args: &crate::cli::BackfillArgs, report: &BackfillReport) -> Result<()> {+ if args.json {+ println!("{}", serde_json::to_string_pretty(report)?);+ return Ok(());+ }+ if args.quiet {+ return Ok(());+ }+ if report.skipped_non_empty {+ println!(+ "ai-memory: {}/{} already has captured sessions — nothing to backfill (use --force to import anyway).",+ report.workspace, report.project+ );+ return Ok(());+ }+ if report.dry_run {+ println!(+ "ai-memory: would import {} local session(s) into {}/{} (dry run).",+ report.selected, report.workspace, report.project+ );+ return Ok(());+ }+ if report.imported_sessions == 0 && report.failed_sessions == 0 {+ println!(+ "ai-memory: no local session history found to backfill for {}/{}.",+ report.workspace, report.project+ );+ return Ok(());+ }+ let mut line = format!(+ "📼 ai-memory imported {} prior local session(s) (~{} events) for {}/{}.",+ report.imported_sessions, report.imported_events, report.workspace, report.project+ );+ if report.skipped_for_cap > 0 {+ line.push_str(&format!(+ " {} older session(s) skipped (import cap).",+ report.skipped_for_cap+ ));+ }+ line.push_str(" Opt out with AI_MEMORY_BACKFILL_ON_START=false.");+ println!("{line}");+ Ok(())+}++fn display_id(id: &str) -> &str {+ id.get(..8).unwrap_or(id)+}++#[cfg(test)]+mod tests {+ use super::*;+ use std::time::Duration;++ fn session(id: &str, secs_ago: u64) -> SessionRef {+ SessionRef {+ harness: ManagedHarness::Claude,+ native_session_id: id.to_string(),+ updated_at: SystemTime::UNIX_EPOCH + Duration::from_secs(1_000_000 - secs_ago),+ }+ }++ #[test]+ fn select_sessions_keeps_the_newest_up_to_the_cap() {+ let all = vec![session("old", 300), session("new", 10), session("mid", 100)];+ let picked = select_sessions(all, 2);+ assert_eq!(+ picked+ .iter()+ .map(|s| s.native_session_id.as_str())+ .collect::<Vec<_>>(),+ vec!["new", "mid"],+ "newest first, capped to 2"+ );+ }++ #[test]+ fn select_sessions_cap_zero_is_raised_to_one_by_caller_contract() {+ // The caller passes max_sessions.max(1); select_sessions itself honors+ // whatever it is given, so a literal 0 yields nothing.+ assert!(select_sessions(vec![session("a", 1)], 0).is_empty());+ assert_eq!(select_sessions(vec![session("a", 1)], 1).len(), 1);+ }++ #[test]+ fn sentinel_path_is_cwd_specific_and_path_safe() {+ let dir = Path::new("/data");+ let a = sentinel_path(dir, Path::new("/home/me/projects/app"));+ let b = sentinel_path(dir, Path::new("/home/me/projects/other"));+ let c = sentinel_path(dir, Path::new("/home/me/projects/app"));+ assert_eq!(a, c, "same cwd -> same path");+ assert_ne!(a, b, "different cwd -> different path");+ assert!(+ a.starts_with("/data/backfill-state/"),+ "under the data dir: {a:?}"+ );+ // A path with separators hashes to a single flat, path-safe leaf.+ assert_eq!(+ a.strip_prefix("/data/backfill-state/")+ .unwrap()+ .components()+ .count(),+ 1,+ "the hashed leaf must be a single path component: {a:?}"+ );+ }++ fn event(kind: WorkstreamEventKind, role: Option<&str>, content: &str) -> NewWorkstreamEvent {+ NewWorkstreamEvent {+ event_id: "evt-1".to_string(),+ agent: ai_memory_core::AgentKind::ClaudeCode,+ native_session_id: "sid".to_string(),+ source_record_id: None,+ kind,+ role: role.map(str::to_string),+ content: content.to_string(),+ occurred_at: None,+ metadata: serde_json::Value::Null,+ }+ }++ #[test]+ fn user_message_maps_to_the_canonical_user_prompt() {+ let m = map_event(+ "sid",+ &event(WorkstreamEventKind::Message, Some("user"), "do the thing"),+ )+ .expect("user message maps");+ assert_eq!(m.event, "user-prompt");+ assert!(+ m.source_event.is_none(),+ "user-prompt is a lifecycle event, not an extension"+ );+ assert_eq!(m.body["prompt"], "do the thing");+ assert_eq!(+ m.ingest_key, "sid:evt-1",+ "ingest key is stable per source event"+ );+ }++ #[test]+ fn assistant_and_tool_events_map_to_backfill_extension_observations() {+ let a = map_event(+ "sid",+ &event(+ WorkstreamEventKind::Message,+ Some("assistant"),+ "here is the plan\nline2",+ ),+ )+ .expect("assistant maps");+ assert_eq!(a.event, "backfill.assistant-message");+ assert_eq!(a.source_event.as_deref(), Some("assistant-message"));+ assert_eq!(+ a.body["title"], "here is the plan",+ "title is the first line"+ );+ assert_eq!(a.body["message"], "here is the plan\nline2");++ let t = map_event(+ "sid",+ &event(WorkstreamEventKind::ToolCall, None, "grep foo"),+ )+ .expect("tool call maps");+ assert_eq!(t.event, "backfill.tool_call");+ assert_eq!(t.source_event.as_deref(), Some("tool_call"));+ }++ #[test]+ fn empty_and_boundary_events_are_dropped() {+ assert!(+ map_event(+ "sid",+ &event(WorkstreamEventKind::Message, Some("user"), " ")+ )+ .is_none(),+ "whitespace-only content is not an observation"+ );+ for kind in [+ WorkstreamEventKind::Compaction,+ WorkstreamEventKind::Checkpoint,+ WorkstreamEventKind::Annotation,+ ] {+ assert!(+ map_event("sid", &event(kind, None, "boundary")).is_none(),+ "{kind:?} is not session content"+ );+ }+ }++ /// End-to-end proof that `collect_local_sessions` wires to the real+ /// workstream path encoder: a Claude transcript planted under the actual+ /// `~/.claude/projects/<enc-cwd>/` layout for this cwd is discovered, and a+ /// transcript for a different cwd is ignored.+ #[tokio::test]+ async fn collect_local_sessions_finds_a_planted_claude_session_for_this_cwd() {+ let home = tempfile::tempdir().unwrap();+ let cwd = tempfile::tempdir().unwrap();+ let other_cwd = tempfile::tempdir().unwrap();++ let session_dir = home+ .path()+ .join(".claude")+ .join("projects")+ .join(cwd.path().to_string_lossy().replace('/', "-"));+ std::fs::create_dir_all(&session_dir).unwrap();+ let header = serde_json::json!({+ "sessionId": "11111111-2222-3333-4444-555555555555",+ "cwd": cwd.path().to_string_lossy(),+ });+ std::fs::write(session_dir.join("sess.jsonl"), format!("{header}\n")).unwrap();+ let foreign = serde_json::json!({+ "sessionId": "99999999-8888-7777-6666-555555555555",+ "cwd": other_cwd.path().to_string_lossy(),+ });+ std::fs::write(session_dir.join("foreign.jsonl"), format!("{foreign}\n")).unwrap();++ let found = collect_local_sessions(home.path(), cwd.path(), None).await;+ let claude: Vec<_> = found+ .iter()+ .filter(|s| s.harness == ManagedHarness::Claude)+ .collect();+ assert_eq!(claude.len(), 1, "only the matching-cwd session: {found:?}");+ assert_eq!(+ claude[0].native_session_id,+ "11111111-2222-3333-4444-555555555555"+ );++ // `--session` narrows to one id.+ let only = collect_local_sessions(home.path(), cwd.path(), Some("nope")).await;+ assert!(only.is_empty(), "no session matches the filter: {only:?}");+ }++ /// The automatic path must honor the `backfill_on_start` opt-out: it records+ /// the attempt (so it never re-spawns) and returns without contacting the+ /// server at all.+ #[tokio::test]+ async fn auto_run_opted_out_writes_sentinel_and_makes_no_request() {+ let home = tempfile::tempdir().unwrap();+ let data_dir = tempfile::tempdir().unwrap();+ let mut config =+ crate::config::Config::load(None, Some(home.path().to_path_buf())).unwrap();+ config.data_dir = data_dir.path().to_path_buf();+ config.backfill_on_start = false;+ // An unroutable server would make any request hang/fail; the opt-out+ // must return before we ever build the endpoint.+ config.server_url = "http://127.0.0.1:9".to_string();++ let args = crate::cli::BackfillArgs {+ workspace: None,+ project: None,+ session: None,+ force: false,+ dry_run: false,+ max_sessions: 25,+ json: false,+ quiet: true,+ auto: true,+ };+ run(&config, args)+ .await+ .expect("opted-out auto run must succeed without contacting the server");++ let cwd = std::env::current_dir().unwrap();+ assert!(+ sentinel_path(&config.data_dir, &cwd).exists(),+ "the opt-out must still record the attempt so it does not re-spawn"+ );+ }+}diff --git a/crates/ai-memory-cli/src/commands/doctor.rs b/crates/ai-memory-cli/src/commands/doctor.rsindex 0e9ddc16..5bfbddbb 100644--- a/crates/ai-memory-cli/src/commands/doctor.rs+++ b/crates/ai-memory-cli/src/commands/doctor.rs@@ -36,7 +36,7 @@ use crate::http_client::{ServerEndpoint, get_json};/// an [`AgentKind`] (Kiro v2/v3, OpenCode v1/v2) are both scanned — they read/// distinct on-disk stores — and their local counts fold together under the/// one agent kind the server records.-const SCANNED_HARNESSES: &[ManagedHarness] = &[+pub(crate) const SCANNED_HARNESSES: &[ManagedHarness] = &[ManagedHarness::Claude,ManagedHarness::Codex,ManagedHarness::OpenCode,@@ -236,7 +236,7 @@ pub async fn run(config: &Config, args: crate::cli::DoctorArgs) -> Result<()> {super::run::native_home(config).context("locating the local harness session stores")?;let ep = ServerEndpoint::from_config_resolving_auth(config).await;- let captured: BTreeMap<String, u64> = get_json::<ByAgentResponse>(+ let captured: BTreeMap<String, u64> = match get_json::<ByAgentResponse>(&ep,"/admin/sessions/by-agent",&[@@ -245,13 +245,22 @@ pub async fn run(config: &Config, args: crate::cli::DoctorArgs) -> Result<()> {],).await- .with_context(|| {- format!("asking the server for captured session counts for {workspace}/{project}")- })?- .by_agent- .into_iter()- .map(|c| (c.agent, c.sessions))- .collect();+ {+ Ok(response) => response+ .by_agent+ .into_iter()+ .map(|c| (c.agent, c.sessions))+ .collect(),+ // A project that has never been captured into does not exist+ // server-side yet (404) — that is "nothing captured", not an error, so+ // every local harness correctly reads as uncaptured.+ Err(error) if super::is_scope_not_found(&error) => BTreeMap::new(),+ Err(error) => {+ return Err(error).with_context(|| {+ format!("asking the server for captured session counts for {workspace}/{project}")+ });+ }+ };let local = scan_local(&home, &cwd, args.since_days).await;let rows = build_rows(&local, &captured);diff --git a/crates/ai-memory-cli/src/commands/hook.rs b/crates/ai-memory-cli/src/commands/hook.rsindex 7ec3ae45..d241446b 100644--- a/crates/ai-memory-cli/src/commands/hook.rs+++ b/crates/ai-memory-cli/src/commands/hook.rs@@ -710,6 +710,19 @@ where.await;}+ // session-start: if this checkout has never had the one-time boot backfill+ // attempted, spawn it detached. It self-gates (config opt-out, empty-store+ // check) and records the attempt, so this is at most one extra process the+ // first time a project is opened after installing hooks — never inline, so+ // the SessionStart budget is untouched. Best-effort; a spawn failure must+ // not affect session start.+ if args.event == "session-start"+ && let Ok(trigger_cwd) = std::env::current_dir()+ && !super::backfill::sentinel_path(&dd, &trigger_cwd).exists()+ {+ let _ = hook_drain_process::spawn_backfill(&dd);+ }+// session-start: drain any backlog (e.g. from a previous session that ended// abruptly), then fetch + inject the pending handoff for the resuming agent.if args.event == "session-start" {diff --git a/crates/ai-memory-cli/src/commands/hook_drain_process.rs b/crates/ai-memory-cli/src/commands/hook_drain_process.rsindex 7f38cc84..d3b046fb 100644--- a/crates/ai-memory-cli/src/commands/hook_drain_process.rs+++ b/crates/ai-memory-cli/src/commands/hook_drain_process.rs@@ -92,6 +92,36 @@ pub fn command_spec(data_dir: &Path, live_token: Option<&str>) -> io::Result<Dra})}+/// Build `ai-memory --data-dir <dir> backfill --auto --quiet`.+///+/// The one-time boot backfill of pre-hook local history is potentially long+/// (it reads native transcripts and imports them), so it runs detached exactly+/// like the drainer rather than inline in the SessionStart hook's tight budget.+/// It carries no bearer on its argv or in its environment: `backfill` resolves+/// auth from config / the persisted hook token itself, and the automatic path+/// only runs in the loopback single-operator posture where no token is needed.+pub fn backfill_command_spec(data_dir: &Path) -> io::Result<DrainCommandSpec> {+ Ok(DrainCommandSpec {+ exe: std::env::current_exe()?,+ args: vec![+ OsString::from("--data-dir"),+ data_dir.as_os_str().to_os_string(),+ OsString::from("backfill"),+ OsString::from("--auto"),+ OsString::from("--quiet"),+ ],+ stderr_log: data_dir.join("logs").join("backfill.log"),+ live_token: None,+ })+}++/// Spawn the detached one-time boot backfill without inheriting hook stdio.+/// Best-effort: a spawn failure must never break session start.+pub fn spawn_backfill(data_dir: &Path) -> io::Result<()> {+ let spec = backfill_command_spec(data_dir)?;+ spawn_spec(&spec)+}+/// Build the stdio/detach shape used when spawning.#[must_use]pub fn spawn_config(spec: &DrainCommandSpec, try_breakaway: bool) -> SpawnConfig {diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rsindex 9cfd610c..bc164b67 100644--- a/crates/ai-memory-cli/src/commands/mod.rs+++ b/crates/ai-memory-cli/src/commands/mod.rs@@ -4,12 +4,26 @@ use anyhow::{Context, Result, anyhow, bail};use crate::config::Config;+/// Whether a server error is a no-create scope lookup missing its+/// `(workspace, project)` — a `404`. Read/count endpoints fail closed on an+/// unknown scope (invariant: reads never create), so a project that has never+/// been written to answers `404` rather than "empty". Callers that ask "does+/// this project have anything?" must treat that as **nothing there**, not a+/// hard error — otherwise the brand-new-project case (exactly the one+/// `doctor` and `backfill` exist for) fails on itself.+pub(crate) fn is_scope_not_found(error: &anyhow::Error) -> bool {+ error+ .downcast_ref::<crate::http_client::ServerResponseError>()+ .is_some_and(|response| response.status() == reqwest::StatusCode::NOT_FOUND)+}+pub mod api_key;pub mod apply_shared;pub mod audit_contamination;pub mod auth;pub mod auto_improve;pub mod auto_improve_report;+pub mod backfill;pub mod backup;pub mod bootstrap;pub mod checkpoints;diff --git a/crates/ai-memory-cli/src/config.rs b/crates/ai-memory-cli/src/config.rsindex 48cbd269..f0b33875 100644--- a/crates/ai-memory-cli/src/config.rs+++ b/crates/ai-memory-cli/src/config.rs@@ -264,6 +264,15 @@ pub struct Config {/// `install-hooks --capture-assistant`. Set with/// `AI_MEMORY_CAPTURE_ASSISTANT=true`.pub capture_assistant: bool,+ /// On by default. When the project's ai-memory store is brand new (empty),+ /// the SessionStart hook triggers a one-time, bounded import of this+ /// project's existing local harness session history so installing hooks+ /// mid-project does not start amnesiac. The import is read-only on the local+ /// side, sanitized on the server exactly like live capture, and only ever+ /// bootstraps an empty project (never overwrites an established one). Turn+ /// off with `AI_MEMORY_BACKFILL_ON_START=false` or `backfill_on_start =+ /// false`; `ai-memory backfill` remains available to run it by hand.+ pub backfill_on_start: bool,/// Strip root-level `anyOf`/`oneOf`/`allOf` from MCP tool input/// schemas (e.g. `memory_read_page`'s "exactly one of path/query"/// contract) on every `tools/list`, regardless of client or `?flavor=`@@ -718,6 +727,7 @@ impl Default for Config {llm_fallback_configs: Vec::new(),consolidate_on_session_end: false,capture_assistant: false,+ backfill_on_start: true,strip_root_combinators: false,gemini_safe_schemas: false,reranker: None,diff --git a/crates/ai-memory-cli/src/lib.rs b/crates/ai-memory-cli/src/lib.rsindex 25f6430e..ab2ece38 100644--- a/crates/ai-memory-cli/src/lib.rs+++ b/crates/ai-memory-cli/src/lib.rs@@ -85,6 +85,7 @@ pub async fn run() -> Result<()> {Command::Init(args) => commands::init::run(&config, args, config_path.as_deref()),Command::Status(args) => commands::status::run(&config, args).await,Command::Doctor(args) => commands::doctor::run(&config, args).await,+ Command::Backfill(args) => commands::backfill::run(&config, args).await,Command::Run(args) => {let exit_code = commands::run::run(&config, args).await?;if exit_code != 0 {diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex a7687eb8..f7161f07 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -484,7 +484,7 @@ move-project move-session uninstallauth user completionshandoffs purge-session compactapi-key export-okf message-doctor+doctor backfill```Run `ai-memory --help` for the full tree.diff --git a/docs/cookbook.md b/docs/cookbook.mdindex 8c82c3f7..f63c4f50 100644--- a/docs/cookbook.md+++ b/docs/cookbook.md@@ -94,6 +94,7 @@ Most people never need these — the agent does it — but they exist:```bashai-memory status # counts, paths, healthai-memory doctor # is every harness that ran here captured?+ai-memory backfill # import prior local history into an empty storeai-memory write-page … # save a durable pageai-memory handoffs # list open handoffs for the projectai-memory message list # cross-project inbox@@ -110,6 +111,13 @@ ai-memory serve # run the servercaptured them — so a harness you rotated in without installing its hook (asilent gap: it keeps its own local history while capturing nothing) shows upas a warning with the exact `install-hooks` command to fix it.+- **I just installed hooks in a project I've worked in for a while**: the first+ time you open the project after installing, ai-memory imports your existing+ local session history once (bounded, sanitized on the server, only into an+ empty store) so it isn't starting from nothing. It runs automatically in the+ background; you can trigger or preview it yourself with `ai-memory backfill`+ (`--dry-run` to see what it would import), or turn it off with+ `AI_MEMORY_BACKFILL_ON_START=false`.- **A search misses something you saved**: confirm the scope — memory isper-project; a page saved in project A is not returned in project B unless itwas written to the global scope or you query with an explicit scope.diff --git a/docs/design-boot-backfill.md b/docs/design-boot-backfill.mdindex 43405b6f..770897e7 100644--- a/docs/design-boot-backfill.md+++ b/docs/design-boot-backfill.md@@ -1,9 +1,32 @@-# Design proposal: boot-time backfill of pre-hook local history--**Status: proposal for review — not implemented.** This is the design pass for the-"pre-load existing session history when the store is new" idea. It adds a new default-on-capture behavior and touches the trust boundary (retroactive untrusted text enters the-store), so it is deliberately separated from implementation.+# Design: boot-time backfill of pre-hook local history++**Status: implemented on `release/2.3`.** This is the design pass for the "pre-load+existing session history when the store is new" idea; it adds a default-on capture+behavior and touches the trust boundary (retroactive untrusted text enters the store).++**As-built note.** Two deviations from the approved plan, both verified by a live smoke:++1. **Ingest path is B, not A.** The plan recommended path A (reuse the managed-workstream+ begin → link → finish endpoints, for the incremental cursor). A live smoke proved A+ populates the *workstream continuity ledger*, not the `sessions`/`observations`/`pages`+ pipeline — `observation` count stayed zero after import, so the history was not+ recall-able via `memory_query`. The implementation therefore **replays each transcript+ through `/hook/batch`** (the same ingress live capture uses, as the companion importer+ does): `export_transcript` normalizes the native transcript, then each event is posted+ as a session-start / `user-prompt` / backfill-extension observation, attributed to the+ original harness + native session id. The re-run smoke confirmed+ `sessions 0→1, observations 0→5, pages 0→1`, by-agent `claude-code: 1`, and a correct+ no-op on the second run.+2. **No new lease table (V65 dropped).** "Import exactly once" is held by the emptiness+ gate (only bootstrap a project with zero captured sessions) plus a per-checkout local+ sentinel (`<data_dir>/backfill-state/<hash(cwd)>`) that stops the SessionStart trigger+ re-spawning. Live hook capture (install-time forward) and backfill (before-install+ history) do not overlap, so there is no double-capture to reconcile.++The automatic path runs detached and silent (like Claude Code's own auto-memory); the+`📼 imported N session(s)` summary is shown on a **manual** `ai-memory backfill` only.+Surfacing the count in the next session's on-start context is a possible follow-up+(§ "Notice delivery" describes the delivery path it would reuse).## Problem