Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(run): recover missed history after server downtime
AkitaOnRails96d8d21a335a
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 09477532..863bfdf8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,6 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]+### Added+- Added `ai-memory recover` and a bounded local recovery journal for server+ outages. Offline managed launches now record only uniquely correlated native+ session locators; recovery quarantines overlapping spool entries before exact+ `/hook/batch` transcript replay, journals harnesses without transcript export+ as spool-only without calling their exporter, retains bounded recovery dedup+ keys with their sessions,+ fails closed on terminal drops, and lets only the original owner finish an+ expired unsuperseded managed run through exact idempotent replay. Backfill+ and recovery replay now use reserved `recovery_*` idempotency keys, so+ re-running `backfill --force` over sessions imported by ai-memory ≤2.6.0 (or+ plain-backfilling a recovery-replayed session) can duplicate those older+ observations. (#1125)+### Fixed- Fixed the hook spool charging a spooled event's retry budget while theserver was unreachable: an endpoint-level delivery failure (connection@@ -18,8 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0file cap and 7-day spool TTL bounds are unchanged. Also corrected thestale `hooks/_lib.sh` comment that claimed the backlog is drained atsession boundaries only (a piggyback drain also runs after any- successful 2xx POST). (#NNN)-+ successful 2xx POST). (#1121)## [2.6.0] - 2026-10-07diff --git a/Cargo.lock b/Cargo.lockindex ff1c7e62..3df5a291 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -57,6 +57,7 @@ dependencies = ["fs2","jiff","jsonc-parser",+ "libc","reqwest 0.12.28","rmcp","rstest",diff --git a/Cargo.toml b/Cargo.tomlindex 3ea82278..c7d81eaa 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -126,6 +126,7 @@ notify = "8"notify-debouncer-full = "0.6"sysinfo = "0.32"fs2 = "0.4"+libc = "0.2"# Windows NTFS file-index (inode equivalent) for own-write detection.winapi-util = "0.1"diff --git a/README.md b/README.mdindex 91b4cb28..568dd0fb 100644--- a/README.md+++ b/README.md@@ -373,9 +373,9 @@ no separate `install-hooks`/`install-mcp` step to forget), it wires the rightproject scope by construction, and it adds cross-harness *session* continuity ontop of shared memory. Everything is idempotent and one-time per harness andconfig home. If the server is unreachable, `run` warns and launches anyway-with local capture spooling (see [Degraded offline-launches](docs/managed-workstreams.md#degraded-offline-launches)); pass-`--require-server` to fail closed instead.+with local capture spooling; run `ai-memory recover` after the server returns+(see [Degraded offline launches](docs/managed-workstreams.md#degraded-offline-launches)).+Pass `--require-server` to fail closed instead.```bashai-memory run claudediff --git a/crates/ai-memory-cli/Cargo.toml b/crates/ai-memory-cli/Cargo.tomlindex bb26825c..7e0c6b51 100644--- a/crates/ai-memory-cli/Cargo.toml+++ b/crates/ai-memory-cli/Cargo.toml@@ -48,6 +48,7 @@ dirs.workspace = truefigment.workspace = trueflate2.workspace = truefs2.workspace = true+libc.workspace = true# `transport-streamable-http-client-reqwest` pulls reqwest in with no TLS backend# (`__reqwest = ["dep:reqwest"]` and nothing more), so the bridge could not reach an# `https://` MCP server at all.diff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rsindex 75a72da7..9b6a06f1 100644--- a/crates/ai-memory-cli/src/cli.rs+++ b/crates/ai-memory-cli/src/cli.rs@@ -258,6 +258,10 @@ pub enum Command {AutoImprove(AutoImproveArgs),/// Manually finalize the latest open session for one agent in this project.FinalizeSession(FinalizeSessionArgs),+ /// Recover history a server outage missed: drain the local hook spool,+ /// replay degraded-run transcripts through the hook backfill path, and+ /// finish managed runs whose transcript import failed. Safe to re-run.+ Recover(RecoverArgs),/// Review, approve, or reject staged auto-improvement proposals.PendingWrites(PendingWritesArgs),/// Compute + store embeddings for every latest page (M9).@@ -2450,6 +2454,14 @@ fn parse_finalizable_agent(s: &str) -> Result<ai_memory_core::AgentKind, String>Ok(kind)}+/// Arguments for `recover`.+#[derive(Debug, Args)]+pub struct RecoverArgs {+ /// Print the machine-readable report as JSON on stdout.+ #[arg(long)]+ pub json: bool,+}+/// Arguments for `finalize-session`.#[derive(Debug, Args)]pub struct FinalizeSessionArgs {diff --git a/crates/ai-memory-cli/src/commands/backfill.rs b/crates/ai-memory-cli/src/commands/backfill.rsindex 4c2d8a22..74195a85 100644--- a/crates/ai-memory-cli/src/commands/backfill.rs+++ b/crates/ai-memory-cli/src/commands/backfill.rs@@ -41,7 +41,8 @@ use serde::{Deserialize, Serialize};use ai_memory_core::{NewWorkstreamEvent, WorkstreamEventKind};use ai_memory_workstream::{LaunchRoots, ManagedHarness, build_launch_plan_with_env, export_transcript,- list_native_sessions, wait_for_transcript_flush,+ export_transcript_delta, export_transcript_range, list_native_sessions,+ wait_for_transcript_flush,};use super::doctor::{SCANNED_HARNESSES, relocated_session_dir};@@ -60,6 +61,7 @@ const PER_HARNESS_SCAN_LIMIT: usize = 200;/// One `POST /hook/batch` request carries at most this many events — matches the/// server's `MAX_HOOK_BATCH_ITEMS` so a batch is never rejected for size.const HOOK_BATCH_ITEMS: usize = 256;+const HOOK_BATCH_BYTES: usize = 8 * 1024 * 1024;/// The extension label backfilled non-user events (assistant/tool) are recorded/// under, mirroring the companion importer's replay-through-`/hook` mechanism.@@ -430,7 +432,7 @@ pub(crate) async fn collect_local_sessions_with(/// One item in a `POST /hook/batch` request: the full hook URL (whose query the/// server parses for event/agent/scope/session) plus the JSON body.-#[derive(Debug, Serialize)]+#[derive(Clone, Debug, Serialize)]struct HookItem {url: String,body: serde_json::Value,@@ -439,8 +441,61 @@ struct HookItem {/// The server's `/hook/batch` acknowledgement (subset we act on).#[derive(Debug, Deserialize)]struct HookBatchAck {- /// Contiguous leading prefix committed, oldest-first.accepted: usize,+ results: Vec<HookBatchResult>,+ #[serde(default)]+ accepted_indices: Option<Vec<usize>>,+ #[serde(default)]+ failed_index: Option<usize>,+}++#[derive(Debug, Deserialize)]+struct HookBatchResult {+ index: usize,+ outcome: String,+}++fn acknowledged_items(ack: &HookBatchAck, items: &[&HookItem]) -> Result<Vec<usize>> {+ let indices = ack+ .accepted_indices+ .clone()+ .unwrap_or_else(|| (0..ack.accepted).collect());+ if ack.accepted > items.len()+ || indices.iter().any(|index| *index >= items.len())+ || indices.windows(2).any(|pair| pair[0] >= pair[1])+ || ack.accepted+ != indices+ .iter()+ .enumerate()+ .take_while(|(pos, index)| *pos == **index)+ .count()+ || ack.results.len() != indices.len()+ || ack+ .failed_index+ .is_some_and(|index| index >= items.len() || indices.contains(&index))+ {+ bail!("invalid hook acknowledgement; keeping transcript for retry");+ }+ for (result, index) in ack.results.iter().zip(&indices) {+ if result.index != *index {+ bail!("hook acknowledgement result indices disagree");+ }+ let is_end = reqwest::Url::parse(&items[*index].url)?+ .query_pairs()+ .any(|(key, value)| key == "event" && value == "session-end");+ match result.outcome.as_str() {+ "stored" | "replayed" => {}+ "ignored_end" if is_end => {}+ "resumed" => bail!("hook processing resumed; rerun to confirm durable replay"),+ outcome => {+ bail!("hook item {index} was not durably stored ({outcome}); keeping transcript")+ }+ }+ }+ if let Some(index) = ack.failed_index {+ bail!("hook processing failed at item {index}; keeping transcript for retry");+ }+ Ok(indices)}/// Import one native session by replaying its transcript through `/hook`, the@@ -463,29 +518,105 @@ async fn import_one(build_launch_plan_with_env(session.harness, None, Vec::new(), None, &[], Some(roots)).ok().and_then(|plan| plan.session_dir);- // These are historical sessions, so the flush wait is a quick no-op; ignore- // its result and read whatever is on disk.- let _ = wait_for_transcript_flush(- session.harness,+ import_exact_session(+ endpoint,+ workspace,+ project,home,cwd,- session_dir.as_deref(),+ session.harness,&session.native_session_id,+ session_dir.as_deref(),+ None,)- .await;- let transcript = export_transcript(- session.harness,+ .await+}++#[allow(clippy::too_many_arguments)]+pub(crate) async fn import_exact_session_range(+ endpoint: &ServerEndpoint,+ workspace: &str,+ project: &str,+ home: &Path,+ cwd: &Path,+ harness: ManagedHarness,+ native_session_id: &str,+ recovery_interval_id: &str,+ session_dir: Option<&Path>,+ source_cursor: Option<&str>,+ final_cursor: &str,+ expected_digests: &[String],+) -> Result<usize> {+ let _ = wait_for_transcript_flush(harness, home, cwd, session_dir, native_session_id).await;+ let transcript = export_transcript_range(+ harness,home,cwd,- session_dir.as_deref(),- &session.native_session_id,+ session_dir,+ native_session_id,+ source_cursor,+ final_cursor,+ )+ .await+ .with_context(|| format!("reading the {} transcript", harness.as_str()))?;+ let actual = ai_memory_workstream::transcript_interval_digests(&transcript.events);+ if actual != expected_digests {+ bail!("native transcript range no longer matches its exact ordered semantic identity");+ }+ import_exported_transcript(+ endpoint,+ workspace,+ project,+ harness,+ native_session_id,+ Some(recovery_interval_id),+ transcript,+ )+ .await+}++#[allow(clippy::too_many_arguments)]+pub(crate) async fn import_exact_session(+ endpoint: &ServerEndpoint,+ workspace: &str,+ project: &str,+ home: &Path,+ cwd: &Path,+ harness: ManagedHarness,+ native_session_id: &str,+ session_dir: Option<&Path>,+ source_cursor: Option<&str>,+) -> Result<usize> {+ let _ = wait_for_transcript_flush(harness, home, cwd, session_dir, native_session_id).await;+ let transcript = if let Some(cursor) = source_cursor {+ export_transcript_delta(harness, home, cwd, session_dir, native_session_id, cursor).await+ } else {+ export_transcript(harness, home, cwd, session_dir, native_session_id, None).await+ }+ .with_context(|| format!("reading the {} transcript", harness.as_str()))?;+ import_exported_transcript(+ endpoint,+ workspace,+ project,+ harness,+ native_session_id,None,+ transcript,).await- .with_context(|| format!("reading the {} transcript", session.harness.as_str()))?;+}- let sid = &session.native_session_id;- let agent = session.harness.agent_kind().as_str();+async fn import_exported_transcript(+ endpoint: &ServerEndpoint,+ workspace: &str,+ project: &str,+ harness: ManagedHarness,+ native_session_id: &str,+ recovery_interval_id: Option<&str>,+ transcript: ai_memory_workstream::ExportedTranscript,+) -> Result<usize> {+ let sid = native_session_id;+ let agent = harness.agent_kind().as_str();let resolved = resolve_occurred_at(&transcript.events);let mut items = Vec::with_capacity(transcript.events.len() + 2);items.push(hook_item(@@ -495,13 +626,13 @@ async fn import_one(agent,"session-start",sid,- &format!("{sid}:session-start"),+ &replay_ingest_key(recovery_interval_id, sid, "session-start", "boundary"),None,serde_json::json!({ "session_id": sid, "occurred_at": resolved.earliest }),)?);let mut content = 0usize;for (event, occurred_at) in transcript.events.iter().zip(&resolved.per_event) {- if let Some(mapped) = map_event(sid, event, occurred_at.as_deref()) {+ if let Some(mapped) = map_event(recovery_interval_id, sid, event, occurred_at.as_deref()) {items.push(hook_item(endpoint,workspace,@@ -523,7 +654,7 @@ async fn import_one(agent,"session-end",sid,- &format!("{sid}:session-end"),+ &replay_ingest_key(recovery_interval_id, sid, "session-end", "boundary"),None,serde_json::json!({ "session_id": sid, "occurred_at": resolved.latest }),)?);@@ -621,6 +752,7 @@ struct MappedEvent {/// body so the imported observation is dated at the transcript's own event/// time rather than at import time.fn map_event(+ recovery_interval_id: Option<&str>,session_id: &str,event: &NewWorkstreamEvent,occurred_at: Option<&str>,@@ -630,7 +762,12 @@ fn map_event(return None;}let role = event.role.as_deref().unwrap_or("");- let ingest_key = format!("{session_id}:{}", event.event_id);+ let ingest_key = replay_ingest_key(+ recovery_interval_id,+ session_id,+ event.kind.as_str(),+ &event.event_id,+ );match event.kind {WorkstreamEventKind::Message if role == "user" || role == "human" => Some(MappedEvent {event: "user-prompt".to_string(),@@ -680,6 +817,27 @@ fn first_line(content: &str) -> String {}}+fn replay_ingest_key(+ recovery_interval_id: Option<&str>,+ session_id: &str,+ kind: &str,+ event_id: &str,+) -> String {+ use sha2::{Digest as _, Sha256};+ let mut hasher = Sha256::new();+ for value in [+ recovery_interval_id.unwrap_or_default(),+ session_id,+ kind,+ event_id,+ ] {+ hasher.update(value.len().to_be_bytes());+ hasher.update(value.as_bytes());+ }+ let digest = format!("{:x}", hasher.finalize());+ format!("recovery_{}", &digest[..55])+}+/// Build a `/hook` URL whose query the server parses for event/agent/scope. The/// URL is data the batch item carries, not the request target (that is/// `/hook/batch`), so it is built against this server's origin exactly as a live@@ -720,32 +878,58 @@ fn hook_item(})}-/// POST the events in server-sized batches, retrying the unaccepted suffix of a-/// partially accepted batch (per-source rate limiting can skip a tail) before-/// failing. Events are oldest-first, so a leading-prefix ack is safe to resume.+/// POST events in server-sized batches. Recovery requires a modern per-item+/// acknowledgement and fails closed on partial or terminal-drop outcomes.+fn hook_batch_end(items: &[HookItem], offset: usize) -> Result<usize> {+ let mut end = offset;+ let mut bytes = 2_usize;+ while end < items.len() && end - offset < HOOK_BATCH_ITEMS {+ let item_bytes = serde_json::to_vec(&items[end])?.len().saturating_add(1);+ if item_bytes > HOOK_BATCH_BYTES {+ bail!("one sanitized hook item exceeds the recovery request byte limit");+ }+ if end > offset && bytes.saturating_add(item_bytes) > HOOK_BATCH_BYTES {+ break;+ }+ bytes = bytes.saturating_add(item_bytes);+ end += 1;+ }+ Ok(end)+}+async fn post_hook_items(endpoint: &ServerEndpoint, items: &[HookItem]) -> Result<()> {- for chunk in items.chunks(HOOK_BATCH_ITEMS) {- let mut offset = 0usize;+ let mut offset = 0_usize;+ while offset < items.len() {+ let end = hook_batch_end(items, offset)?;+ let chunk = &items[offset..end];let mut stalled = 0u32;- while offset < chunk.len() {- let batch: Vec<&HookItem> = chunk[offset..].iter().collect();+ loop {+ let batch: Vec<&HookItem> = chunk.iter().collect();let ack: HookBatchAck = post_json(endpoint, "/hook/batch", &batch).await.context("replaying transcript events through /hook/batch")?;- if ack.accepted == 0 {+ let accepted = acknowledged_items(&ack, &batch)?;+ if accepted.len() == batch.len() {+ break;+ }+ if accepted.is_empty() {stalled += 1;if stalled >= 5 {bail!(- "server accepted none of a hook batch after {stalled} attempts \+ "server acknowledged none of a hook batch after {stalled} attempts \(rate limited or saturated); rerun to resume");}tokio::time::sleep(Duration::from_millis(200 * u64::from(stalled))).await;continue;}- stalled = 0;- offset += ack.accepted.min(chunk.len() - offset);+ bail!(+ "server acknowledged only {} of {} transcript items; keeping transcript for a safe retry",+ accepted.len(),+ batch.len()+ );}+ offset = end;}Ok(())}@@ -888,6 +1072,7 @@ mod tests {#[test]fn user_message_maps_to_the_canonical_user_prompt() {let m = map_event(+ None,"sid",&event(WorkstreamEventKind::Message, Some("user"), "do the thing"),Some("2026-09-10T12:00:00Z"),@@ -901,14 +1086,545 @@ mod tests {assert_eq!(m.body["prompt"], "do the thing");assert_eq!(m.body["occurred_at"], "2026-09-10T12:00:00Z");assert_eq!(- m.ingest_key, "sid:evt-1",+ m.ingest_key,+ replay_ingest_key(None, "sid", "message", "evt-1"),"ingest key is stable per source event");+ assert_eq!(m.ingest_key.len(), 64);+ assert!(+ m.ingest_key+ .bytes()+ .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-' || byte == b'_')+ );+ }++ #[test]+ fn replay_keys_are_deterministic_valid_and_bounded_for_long_native_ids() {+ let session = "native:".repeat(200);+ let event = "event/with spaces:".repeat(200);+ let first = replay_ingest_key(None, &session, "message", &event);+ let second = replay_ingest_key(None, &session, "message", &event);+ assert_eq!(first, second);+ assert_eq!(first.len(), 64);+ assert!(first.starts_with("recovery_"));+ assert!(+ first+ .bytes()+ .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-' || byte == b'_')+ );+ assert_ne!(+ first,+ replay_ingest_key(None, &session, "tool_call", &event)+ );+ let other_interval = replay_ingest_key(Some("journal-2"), &session, "message", &event);+ assert_ne!(first, other_interval);+ assert_eq!(+ other_interval,+ replay_ingest_key(Some("journal-2"), &session, "message", &event),+ "one journal interval must replay stably"+ );+ }++ #[test]+ fn recovery_hook_batches_respect_count_and_serialized_byte_limits() {+ let endpoint = ServerEndpoint::for_hook_target("http://127.0.0.1:1".into(), None);+ let make = |size| HookItem {+ url: endpoint.build_url("/hook?event=user-prompt"),+ body: serde_json::json!({"prompt": "x".repeat(size)}),+ };+ let items = (0..HOOK_BATCH_ITEMS + 1)+ .map(|_| make(1))+ .collect::<Vec<_>>();+ assert_eq!(hook_batch_end(&items, 0).unwrap(), HOOK_BATCH_ITEMS);++ let large = vec![make(HOOK_BATCH_BYTES / 2), make(HOOK_BATCH_BYTES / 2)];+ assert_eq!(hook_batch_end(&large, 0).unwrap(), 1);+ assert!(+ hook_batch_end(&[make(HOOK_BATCH_BYTES)], 0)+ .unwrap_err()+ .to_string()+ .contains("exceeds")+ );+ }++ #[test]+ fn acknowledgement_requires_results_and_rejects_terminal_drops() {+ let endpoint = ServerEndpoint::for_hook_target("http://127.0.0.1:1".into(), None);+ let item = hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ "user-prompt",+ "sid",+ "key",+ None,+ serde_json::json!({"session_id": "sid", "prompt": "hello"}),+ )+ .unwrap();+ let items = [&item];+ for outcome in [+ "dropped_policy",+ "dropped_subagent",+ "dropped_unauthorized",+ "dropped_collision",+ "dropped_invalid",+ "resumed",+ ] {+ let ack = HookBatchAck {+ accepted: 1,+ results: vec![HookBatchResult {+ index: 0,+ outcome: outcome.into(),+ }],+ accepted_indices: None,+ failed_index: None,+ };+ assert!(acknowledged_items(&ack, &items).is_err(), "{outcome}");+ }+ let legacy = HookBatchAck {+ accepted: 1,+ results: Vec::new(),+ accepted_indices: None,+ failed_index: None,+ };+ assert!(acknowledged_items(&legacy, &items).is_err());+ }++ #[test]+ fn acknowledgement_allows_stored_replayed_and_only_terminal_ignored_end() {+ let endpoint = ServerEndpoint::for_hook_target("http://127.0.0.1:1".into(), None);+ let make = |event| {+ hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ event,+ "sid",+ "key",+ None,+ serde_json::json!({"session_id": "sid"}),+ )+ .unwrap()+ };+ let prompt = make("user-prompt");+ let end = make("session-end");+ for outcome in ["stored", "replayed"] {+ let ack = HookBatchAck {+ accepted: 1,+ results: vec![HookBatchResult {+ index: 0,+ outcome: outcome.into(),+ }],+ accepted_indices: Some(vec![0]),+ failed_index: None,+ };+ assert_eq!(acknowledged_items(&ack, &[&prompt]).unwrap(), vec![0]);+ }+ let ignored = HookBatchAck {+ accepted: 1,+ results: vec![HookBatchResult {+ index: 0,+ outcome: "ignored_end".into(),+ }],+ accepted_indices: Some(vec![0]),+ failed_index: None,+ };+ assert!(acknowledged_items(&ignored, &[&prompt]).is_err());+ assert_eq!(acknowledged_items(&ignored, &[&end]).unwrap(), vec![0]);+ }++ #[tokio::test]+ async fn real_hook_router_replay_is_idempotent_and_reports_outcomes() {+ use ai_memory_core::{ActiveProject, IngestMetrics, Sanitizer, SessionId};+ use ai_memory_hooks::{+ HookState, IngestGates, IngestRateLimiter, ProjectCacheStore, SubagentSessionSet,+ hook_router,+ };+ use ai_memory_store::Store;+ use ai_memory_wiki::Wiki;+ use std::sync::Arc;+ use tower::ServiceExt as _;++ let temp = tempfile::tempdir().unwrap();+ let store = Store::open(temp.path()).unwrap();+ let workspace_id = store.writer.get_or_create_workspace("ws").await.unwrap();+ let project_id = store+ .writer+ .get_or_create_project(workspace_id, "project", None)+ .await+ .unwrap();+ let state = HookState {+ workspace_id,+ project_id,+ writer: store.writer.clone(),+ reader: store.reader.clone(),+ wiki: Wiki::new(temp.path(), store.writer.clone()).unwrap(),+ consolidator: None,+ sanitizer: Sanitizer::default(),+ project_cache: Arc::new(tokio::sync::Mutex::new(ProjectCacheStore::default())),+ active_project: ActiveProject::new(),+ ingest_metrics: Arc::new(IngestMetrics::default()),+ ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(16)),+ ingest_gates: IngestGates::default(),+ ingest_rate: Arc::new(tokio::sync::Mutex::new(IngestRateLimiter::disabled())),+ consolidate_on_session_end: false,+ session_consolidation_notify: None,+ profile_notify: None,+ capture_assistant_enabled: false,+ claim_handoff_on_session_start: true,+ create_handoff_on_session_end: false,+ subagent_sessions: Arc::new(tokio::sync::Mutex::new(SubagentSessionSet::default())),+ home_dir: None,+ trusted_proxy_identity: false,+ per_user_slots: false,+ mid_session_routing: ai_memory_core::MidSessionRouting::default(),+ profile: ai_memory_core::profile::ProfileSettings::default(),+ };+ let endpoint = ServerEndpoint::for_hook_target("http://localhost".into(), None);+ let sid = "native-session-with-a-very-long-identifier-that-would-break-the-old-key";+ let recovery_interval_id = Some("journal");+ let items = vec![+ hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ "session-start",+ sid,+ &replay_ingest_key(recovery_interval_id, sid, "session-start", "boundary"),+ None,+ serde_json::json!({"session_id": sid, "cwd": "/repo"}),+ )+ .unwrap(),+ hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ "user-prompt",+ sid,+ &replay_ingest_key(recovery_interval_id, sid, "message", "event-1"),+ None,+ serde_json::json!({"session_id": sid, "cwd": "/repo", "prompt": "same prompt"}),+ )+ .unwrap(),+ hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ "session-end",+ sid,+ &replay_ingest_key(recovery_interval_id, sid, "session-end", "boundary"),+ None,+ serde_json::json!({"session_id": sid, "cwd": "/repo"}),+ )+ .unwrap(),+ ];+ let spool = crate::commands::hook_spool::spool_dir(temp.path());+ let overlapping = hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ "user-prompt",+ sid,+ "random_live_key",+ None,+ serde_json::json!({"session_id": sid, "cwd": "/repo", "prompt": "same prompt"}),+ )+ .unwrap();+ crate::commands::hook_spool::enqueue(+ &spool,+ &crate::commands::hook_spool::entry_for(+ overlapping.url,+ overlapping.body.to_string(),+ None,+ false,+ ),+ )+ .unwrap();+ let quarantine = crate::commands::hook_spool::quarantine_session_entries(+ &Config::default(),+ &spool,+ "journal",+ 0,+ u64::MAX,+ &endpoint.identity(),+ "claude-code",+ "ws",+ "project",+ Path::new("/repo"),+ sid,+ )+ .unwrap();+ assert_eq!(crate::commands::hook_spool::spool_len(&spool), 0);++ let body = serde_json::to_vec(&items).unwrap();+ let first = hook_router(state.clone())+ .oneshot(+ axum::http::Request::post("/hook/batch")+ .header("content-type", "application/json")+ .body(axum::body::Body::from(body.clone()))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(first.status(), axum::http::StatusCode::OK);+ let first: HookBatchAck = serde_json::from_slice(+ &axum::body::to_bytes(first.into_body(), usize::MAX)+ .await+ .unwrap(),+ )+ .unwrap();+ assert_eq!(+ acknowledged_items(&first, &items.iter().collect::<Vec<_>>())+ .unwrap()+ .len(),+ 3+ );+ let second = hook_router(state.clone())+ .oneshot(+ axum::http::Request::post("/hook/batch")+ .header("content-type", "application/json")+ .body(axum::body::Body::from(body))+ .unwrap(),+ )+ .await+ .unwrap();+ let second: HookBatchAck = serde_json::from_slice(+ &axum::body::to_bytes(second.into_body(), usize::MAX)+ .await+ .unwrap(),+ )+ .unwrap();+ assert!(+ second+ .results+ .iter()+ .any(|result| result.outcome == "replayed")+ );+ assert!(+ second+ .results+ .iter()+ .any(|result| result.outcome == "ignored_end")+ );+ let observations = store+ .reader+ .observations_for_session(SessionId::from_native(sid))+ .await+ .unwrap();+ assert_eq!(observations.len(), 3);+ assert_eq!(+ observations+ .iter()+ .filter(|observation| observation.body == "same prompt")+ .count(),+ 1+ );+ quarantine.complete().unwrap();+ }++ /// Two sequential offline intervals on one resumed native session: the+ /// second interval's journal id gives its boundary and content keys their+ /// own identity, so its session-end is STORED (re-closing the session with+ /// its end-of-session effects) instead of being short-circuited as a+ /// replay of the first interval's end, while re-running either interval+ /// stays a pure replay.+ #[tokio::test]+ async fn two_sequential_recovery_intervals_on_one_resumed_session_replay_exactly_once() {+ use ai_memory_core::{ActiveProject, IngestMetrics, ObservationKind, Sanitizer, SessionId};+ use ai_memory_hooks::{+ HookState, IngestGates, IngestRateLimiter, ProjectCacheStore, SubagentSessionSet,+ hook_router,+ };+ use ai_memory_store::Store;+ use ai_memory_wiki::Wiki;+ use std::sync::Arc;+ use tower::ServiceExt as _;++ let temp = tempfile::tempdir().unwrap();+ let store = Store::open(temp.path()).unwrap();+ let workspace_id = store.writer.get_or_create_workspace("ws").await.unwrap();+ let project_id = store+ .writer+ .get_or_create_project(workspace_id, "project", None)+ .await+ .unwrap();+ let state = HookState {+ workspace_id,+ project_id,+ writer: store.writer.clone(),+ reader: store.reader.clone(),+ wiki: Wiki::new(temp.path(), store.writer.clone()).unwrap(),+ consolidator: None,+ sanitizer: Sanitizer::default(),+ project_cache: Arc::new(tokio::sync::Mutex::new(ProjectCacheStore::default())),+ active_project: ActiveProject::new(),+ ingest_metrics: Arc::new(IngestMetrics::default()),+ ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(16)),+ ingest_gates: IngestGates::default(),+ ingest_rate: Arc::new(tokio::sync::Mutex::new(IngestRateLimiter::disabled())),+ consolidate_on_session_end: false,+ session_consolidation_notify: None,+ profile_notify: None,+ capture_assistant_enabled: false,+ claim_handoff_on_session_start: true,+ create_handoff_on_session_end: false,+ subagent_sessions: Arc::new(tokio::sync::Mutex::new(SubagentSessionSet::default())),+ home_dir: None,+ trusted_proxy_identity: false,+ per_user_slots: false,+ mid_session_routing: ai_memory_core::MidSessionRouting::default(),+ profile: ai_memory_core::profile::ProfileSettings::default(),+ };+ let endpoint = ServerEndpoint::for_hook_target("http://localhost".into(), None);+ let sid = "native-session-resumed-across-two-offline-intervals";+ let interval_items = |journal: &str, prompt: &str, event_id: &str| {+ vec![+ hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ "session-start",+ sid,+ &replay_ingest_key(Some(journal), sid, "session-start", "boundary"),+ None,+ serde_json::json!({"session_id": sid, "cwd": "/repo"}),+ )+ .unwrap(),+ hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ "user-prompt",+ sid,+ &replay_ingest_key(Some(journal), sid, "message", event_id),+ None,+ serde_json::json!({"session_id": sid, "cwd": "/repo", "prompt": prompt}),+ )+ .unwrap(),+ hook_item(+ &endpoint,+ "ws",+ "project",+ "claude-code",+ "session-end",+ sid,+ &replay_ingest_key(Some(journal), sid, "session-end", "boundary"),+ None,+ serde_json::json!({"session_id": sid, "cwd": "/repo"}),+ )+ .unwrap(),+ ]+ };+ let router_state = state.clone();+ let post = move |items: Vec<HookItem>| {+ let state = router_state.clone();+ async move {+ let response = hook_router(state)+ .oneshot(+ axum::http::Request::post("/hook/batch")+ .header("content-type", "application/json")+ .body(axum::body::Body::from(serde_json::to_vec(&items).unwrap()))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(response.status(), axum::http::StatusCode::OK);+ let ack: HookBatchAck = serde_json::from_slice(+ &axum::body::to_bytes(response.into_body(), usize::MAX)+ .await+ .unwrap(),+ )+ .unwrap();+ assert_eq!(+ acknowledged_items(&ack, &items.iter().collect::<Vec<_>>())+ .unwrap()+ .len(),+ items.len()+ );+ ack.results+ .into_iter()+ .map(|result| result.outcome)+ .collect::<Vec<_>>()+ }+ };++ let first = interval_items("journal-a", "first interval prompt", "event-1");+ assert_eq!(+ post(first.clone()).await,+ vec!["stored", "stored", "stored"],+ "the first offline interval recovers exactly once"+ );+ let second = interval_items("journal-b", "second interval prompt", "event-2");+ assert_eq!(+ post(second.clone()).await,+ vec!["stored", "stored", "stored"],+ "the resumed interval's content and session-end must be stored, not replayed"+ );++ let observations = store+ .reader+ .observations_for_session(SessionId::from_native(sid))+ .await+ .unwrap();+ assert_eq!(observations.len(), 6, "two start/end pairs and two prompts");+ assert_eq!(+ observations+ .iter()+ .filter(|observation| observation.body == "first interval prompt")+ .count(),+ 1+ );+ assert_eq!(+ observations+ .iter()+ .filter(|observation| observation.body == "second interval prompt")+ .count(),+ 1+ );+ assert_eq!(+ observations+ .iter()+ .filter(|observation| observation.kind == ObservationKind::SessionEnd)+ .count(),+ 2,+ "each interval's session-end is its own stored observation"+ );++ for rerun in [first, second] {+ assert!(+ rerun+ .iter()+ .zip(post(rerun.clone()).await)+ .all(|(item, outcome)| {+ let is_end = reqwest::Url::parse(&item.url)+ .unwrap()+ .query_pairs()+ .any(|(key, value)| key == "event" && value == "session-end");+ outcome == "replayed" || (is_end && outcome == "ignored_end")+ }),+ "re-running a recovered interval is a pure replay"+ );+ }+ let observations = store+ .reader+ .observations_for_session(SessionId::from_native(sid))+ .await+ .unwrap();+ assert_eq!(observations.len(), 6, "replays add no observations");}#[test]fn assistant_and_tool_events_map_to_backfill_extension_observations() {let a = map_event(+ None,"sid",&event(WorkstreamEventKind::Message,@@ -931,6 +1647,7 @@ mod tests {);let t = map_event(+ None,"sid",&event(WorkstreamEventKind::ToolCall, None, "grep foo"),None,@@ -1050,6 +1767,7 @@ mod tests {fn empty_and_boundary_events_are_dropped() {assert!(map_event(+ None,"sid",&event(WorkstreamEventKind::Message, Some("user"), " "),None,@@ -1063,7 +1781,7 @@ mod tests {WorkstreamEventKind::Annotation,] {assert!(- map_event("sid", &event(kind, None, "boundary"), None).is_none(),+ map_event(None, "sid", &event(kind, None, "boundary"), None).is_none(),"{kind:?} is not session content");}diff --git a/crates/ai-memory-cli/src/commands/hook.rs b/crates/ai-memory-cli/src/commands/hook.rsindex ed2228c2..a666d2e7 100644--- a/crates/ai-memory-cli/src/commands/hook.rs+++ b/crates/ai-memory-cli/src/commands/hook.rs@@ -902,7 +902,7 @@ wherelet entry =hook_spool::entry_for(event_url, payload.clone(), effective_token, oidc_present).routed_to(profile_name);- if hook_spool::enqueue(&spool, &entry).is_err() {+ if hook_spool::enqueue_with_config(&spool, &entry, Some(&spool_policy)).is_err() {eprintln!("ai-memory hook warning: failed to spool lifecycle event; capture for this event was skipped");@@ -1591,6 +1591,7 @@ mod tests {hook_spool::DrainResult::default(),hook_spool::DrainResult {sent: 12,+ durable: 12,remaining: 0,dropped: 0,},@@ -1611,6 +1612,7 @@ mod tests {let clean = drain_report(&hook_spool::LockedDrainResult::Drained(hook_spool::DrainResult {sent: 5,+ durable: 5,remaining: 0,dropped: 0,},@@ -1618,6 +1620,7 @@ mod tests {let lossy = drain_report(&hook_spool::LockedDrainResult::Drained(hook_spool::DrainResult {sent: 0,+ durable: 0,remaining: 0,dropped: 7,},@@ -1651,6 +1654,7 @@ mod tests {let report = drain_report(&hook_spool::LockedDrainResult::Drained(hook_spool::DrainResult {sent: 1,+ durable: 1,remaining: 4,dropped: 0,},diff --git a/crates/ai-memory-cli/src/commands/hook_capture.rs b/crates/ai-memory-cli/src/commands/hook_capture.rsindex 9adb4e91..fc31cae2 100644--- a/crates/ai-memory-cli/src/commands/hook_capture.rs+++ b/crates/ai-memory-cli/src/commands/hook_capture.rs@@ -808,6 +808,13 @@ pub async fn post_hook(/// edge auth hop over the whole batch instead of paying it per event.#[derive(Debug, Clone, PartialEq, Eq)]pub enum BatchOutcome {+ /// Modern server returned one terminal result for each acknowledged item.+ Detailed {+ /// Per-item processing outcomes in request order.+ outcomes: Vec<String>,+ /// Item that failed after any earlier acknowledged items.+ failed_index: Option<usize>,+ },/// Server committed the leading `usize` items (contiguous prefix, oldest/// first). Equals the request length on full success; a smaller value means/// the server stopped on that item (fail-fast) — the caller deletes the@@ -872,7 +879,12 @@ pub async fn post_batch(if status.is_success() {match resp.json::<serde_json::Value>().await {Ok(v) => {- if let Some(indices) = accepted_indices(&v) {+ if let Some(outcomes) = batch_outcomes(&v) {+ BatchOutcome::Detailed {+ outcomes,+ failed_index: failed_index(&v),+ }+ } else if let Some(indices) = accepted_indices(&v) {BatchOutcome::AcceptedIndices {indices,failed_index: failed_index(&v),@@ -915,6 +927,18 @@ pub async fn post_batch(}}+fn batch_outcomes(v: &serde_json::Value) -> Option<Vec<String>> {+ let results = v.get("results")?.as_array()?;+ let mut outcomes = Vec::with_capacity(results.len());+ for (position, result) in results.iter().enumerate() {+ if result.get("index")?.as_u64()? as usize != position {+ return None;+ }+ outcomes.push(result.get("outcome")?.as_str()?.to_owned());+ }+ Some(outcomes)+}+fn failed_index(v: &serde_json::Value) -> Option<usize> {v.get("failed_index")?.as_u64().map(|n| n as usize)}diff --git a/crates/ai-memory-cli/src/commands/hook_spool.rs b/crates/ai-memory-cli/src/commands/hook_spool.rsindex 77843b4e..73816322 100644--- a/crates/ai-memory-cli/src/commands/hook_spool.rs+++ b/crates/ai-memory-cli/src/commands/hook_spool.rs@@ -66,8 +66,23 @@ const MAX_BATCH_ITEMS: usize = 256;/// `DefaultBodyLimit` with margin for JSON framing. A chunk always carries at/// least one event even if that event alone exceeds this.const MAX_BATCH_BYTES: usize = 8 * 1024 * 1024;+const RECOVERY_DIR: &str = ".recovery";+const RECOVERY_MATCH_FILE: &str = ".active";+const RECOVERY_COMPLETED_FILE: &str = ".completed";+const SPOOL_CAPACITY_LOCK_FILE: &str = ".capacity.lock";+const SPOOL_COORD_LOCK_WAIT: Duration = Duration::from_millis(5);+const MAX_SPOOL_ENTRY_BYTES: u64 = 10 * 1024 * 1024;+const RECOVERY_TOMBSTONE_AGE_MS: u64 = 24 * 60 * 60 * 1000;+const MAX_RECOVERY_TOMBSTONES: usize = 512;+const MAX_RECOVERY_SWEEP_DIRS: usize = 1_024;static ENQUEUE_SEQ: AtomicU64 = AtomicU64::new(0);+#[cfg(test)]+static PAUSE_ENQUEUE_AFTER_MAIN_RENAME: std::sync::atomic::AtomicBool =+ std::sync::atomic::AtomicBool::new(false);+#[cfg(test)]+static ENQUEUE_REACHED_RECOVERY_WINDOW: std::sync::atomic::AtomicBool =+ std::sync::atomic::AtomicBool::new(false);/// How a spooled event authenticates to the server when drained.#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]@@ -188,7 +203,7 @@ pub fn spool_health(spool: &Path) -> SpoolHealth {retries_total: 0,};for (_, path) in &files {- if let Ok(bytes) = std::fs::read(path)+ if let Ok(bytes) = read_spool_entry_bytes(path)&& let Ok(entry) = serde_json::from_slice::<SpoolEntry>(&bytes){health.retries_total += u64::from(entry.attempts);@@ -207,12 +222,9 @@ pub fn spool_health(spool: &Path) -> SpoolHealth {/// entry enqueued at the Unix epoch — see [`spool_health`].fn created_ms_from_name(file_name: &str) -> Option<u64> {let (stamp, rest) = file_name.split_once('-')?;- // Require the full zero-padded width the writer emits, so a name that- // merely *starts* with digits is not accepted.if stamp.len() != 13 || !stamp.bytes().all(|b| b.is_ascii_digit()) {return None;}- // And require the remainder to look like `{pid}-{seq}.json`.if !rest.ends_with(".json") || !rest.contains('-') {return None;}@@ -235,11 +247,21 @@ fn now_ms() -> u64 {////// # Errors/// Returns an error only when the spool file cannot be written.+#[cfg(test)]pub fn enqueue(spool: &Path, entry: &SpoolEntry) -> std::io::Result<()> {+ enqueue_with_config(spool, entry, None)+}++pub fn enqueue_with_config(+ spool: &Path,+ entry: &SpoolEntry,+ config: Option<&crate::config::Config>,+) -> std::io::Result<()> {create_spool_dir(spool)?;let seq = ENQUEUE_SEQ.fetch_add(1, Ordering::Relaxed);+ let recovery_key = recovery_coordinate_key_for_entry(entry, config)?;let name = format!(- "{:013}-{}-{seq:016x}.json",+ "{:013}-{}-{seq:016x}-{recovery_key}.json",entry.created_ms,std::process::id());@@ -247,8 +269,53 @@ pub fn enqueue(spool: &Path, entry: &SpoolEntry) -> std::io::Result<()> {let final_path = spool.join(&name);let bytes = serde_json::to_vec(entry)?;write_private(&tmp, &bytes)?;+ let capacity_lock = open_spool_capacity_lock(spool)?;+ let started = Instant::now();+ loop {+ match capacity_lock.try_lock_exclusive() {+ Ok(()) => break,+ Err(error)+ if is_drain_lock_busy_error(&error)+ && started.elapsed() < SPOOL_COORD_LOCK_WAIT =>+ {+ std::thread::sleep(Duration::from_millis(1));+ }+ Err(error) if is_drain_lock_busy_error(&error) => {+ let _ = std::fs::remove_file(&tmp);+ return Ok(());+ }+ Err(error) => return Err(error),+ }+ }+ if total_spool_entries(spool)? >= MAX_SPOOL_FILES {+ let _ = std::fs::remove_file(&tmp);+ eprintln!(+ "ai-memory: hook-spool at capacity ({MAX_SPOOL_FILES}); dropping one new UNDELIVERED event"+ );+ return Ok(());+ }std::fs::rename(&tmp, &final_path)?;- prune_spool_file_count(spool);+ #[cfg(test)]+ if PAUSE_ENQUEUE_AFTER_MAIN_RENAME.load(Ordering::SeqCst) {+ ENQUEUE_REACHED_RECOVERY_WINDOW.store(true, Ordering::SeqCst);+ while PAUSE_ENQUEUE_AFTER_MAIN_RENAME.load(Ordering::SeqCst) {+ std::thread::yield_now();+ }+ }+ let coord_lock = open_recovery_coord_lock(spool)?;+ if coord_lock.try_lock_exclusive().is_ok() {+ match recovery_target_for_entry(spool, entry, config)? {+ Some(RecoveryTarget::Active(dir)) => {+ std::fs::rename(&final_path, dir.join(&name))?;+ }+ Some(RecoveryTarget::Completed(_)) => match std::fs::remove_file(&final_path) {+ Ok(()) => {}+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}+ Err(error) => return Err(error),+ },+ None => {}+ }+ }Ok(())}@@ -260,6 +327,38 @@ fn create_spool_dir(spool: &Path) -> std::io::Result<()> {super::path_util::create_private_dir(spool)}+fn open_spool_capacity_lock(spool: &Path) -> std::io::Result<File> {+ create_spool_dir(spool)?;+ std::fs::OpenOptions::new()+ .create(true)+ .truncate(false)+ .read(true)+ .write(true)+ .open(spool.with_extension(SPOOL_CAPACITY_LOCK_FILE))+}++fn total_spool_entries(spool: &Path) -> std::io::Result<usize> {+ let mut total = list_entries(spool).map_or(0, |(files, _)| files.len());+ let recovery = spool.join(RECOVERY_DIR);+ let directories = match std::fs::read_dir(recovery) {+ Ok(entries) => entries,+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(total),+ Err(error) => return Err(error),+ };+ for (index, directory) in directories.enumerate() {+ if index >= MAX_RECOVERY_SWEEP_DIRS {+ return Ok(MAX_SPOOL_FILES);+ }+ let directory = directory?;+ if directory.file_type()?.is_dir() {+ total = total.saturating_add(+ list_entries(&directory.path()).map_or(0, |(files, _)| files.len()),+ );+ }+ }+ Ok(total)+}+fn write_private(path: &Path, bytes: &[u8]) -> std::io::Result<()> {use std::io::Write as _;let mut opts = std::fs::OpenOptions::new();@@ -315,6 +414,8 @@ impl SpoolEntry {pub struct DrainResult {/// Events delivered (server answered 2xx) and removed from the spool.pub sent: usize,+ /// Events whose modern per-item acknowledgement proves durable storage.+ pub durable: usize,/// Events still queued (failed this pass, or skipped when the budget ran out).pub remaining: usize,/// Events discarded as undeliverable (too old or too many failed attempts).@@ -335,6 +436,587 @@ pub struct DrainLock {_file: File,}+/// Outcome of [`QuarantinedSpool::restore`].+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum RestoreOutcome {+ /// Quarantined events were moved back to the plain spool (or the+ /// quarantine directory no longer existed).+ Restored,+ /// The interval was already completed, so its correlated evidence is+ /// durably delivered and there is nothing left to restore or re-deliver.+ AlreadyDelivered,+}++#[derive(Debug, Clone, Serialize, Deserialize)]+struct RecoveryMatch {+ journal_id: String,+ coordinate_key: String,+ interval_started_ms: u64,+ interval_ended_ms: u64,+ server_identity: String,+ agent: String,+ workspace: String,+ project: String,+ cwd: String,+ native_session_id: String,+}++pub struct QuarantinedSpool {+ dir: PathBuf,+ spool: PathBuf,+ config: crate::config::Config,+}++#[derive(Debug, Default, Clone, Copy)]+pub struct CorrelatedDrainResult {+ pub durable: usize,+ pub remaining: usize,+ pub dropped: usize,+}++impl QuarantinedSpool {+ pub fn len(&self) -> usize {+ list_entries(&self.dir).map_or(0, |(files, _)| files.len())+ }++ pub fn complete(self) -> std::io::Result<()> {+ if self.dir.exists() {+ let lock = open_recovery_coord_lock(&self.spool)?;+ lock.lock_exclusive()?;+ let active = self.dir.join(RECOVERY_MATCH_FILE);+ let completed = self.dir.join(RECOVERY_COMPLETED_FILE);+ let spec = read_recovery_match(&active)?;+ std::fs::rename(&active, &completed)?;+ move_late_matching_entries(&self.spool, &self.dir, &self.config, &spec)?;+ for item in std::fs::read_dir(&self.dir)? {+ let item = item?;+ if item.path().extension().and_then(|value| value.to_str()) == Some("json") {+ std::fs::remove_file(item.path())?;+ }+ }+ sweep_recovery_tombstones(&self.spool)?;+ }+ Ok(())+ }++ pub async fn drain_strict(+ &self,+ data_dir: &Path,+ total_budget: Duration,+ per_event_timeout: Duration,+ max_attempts: MaxAttempts,+ ) -> CorrelatedDrainResult {+ let Ok((mut files, unreadable)) = list_entries(&self.dir) else {+ return CorrelatedDrainResult {+ remaining: 1,+ ..CorrelatedDrainResult::default()+ };+ };+ files.sort();+ let client = build_client();+ let started = Instant::now();+ let mut oidc_cache = None;+ let mut result = DrainResult {+ remaining: unreadable,+ ..DrainResult::default()+ };+ for path in files {+ if started.elapsed() >= total_budget {+ result.remaining += 1;+ continue;+ }+ let Some(entry) = load_live_entry(&path, &mut result) else {+ continue;+ };+ if body_is_malformed(&entry) {+ bump_or_drop(&path, &entry, max_attempts.0, &mut result);+ continue;+ }+ let bearer = entry_bearer(&entry, &client, data_dir, &mut oidc_cache).await;+ let Some(payload) = batch_payload(&[(path.clone(), entry.clone())]) else {+ bump_or_drop(&path, &entry, max_attempts.0, &mut result);+ continue;+ };+ let remaining_budget = total_budget.saturating_sub(started.elapsed());+ let timeout = per_event_timeout.min(remaining_budget);+ match post_batch(+ &client,+ &batch_endpoint(&entry.url),+ &payload,+ bearer.as_deref(),+ timeout,+ )+ .await+ {+ BatchOutcome::Detailed {+ outcomes,+ failed_index: None,+ } if outcomes.len() == 1 => {+ let outcome = outcomes[0].as_str();+ let durable = matches!(outcome, "stored" | "replayed")+ || outcome == "ignored_end"+ && entry_event(&entry).as_deref() == Some("session-end");+ if durable {+ let _ = std::fs::remove_file(path);+ result.sent += 1;+ result.durable += 1;+ } else if outcome.starts_with("dropped_") {+ let _ = std::fs::remove_file(path);+ result.dropped += 1;+ } else {+ bump_or_drop(&path, &entry, max_attempts.0, &mut result);+ }+ }+ _ => bump_or_drop(&path, &entry, max_attempts.0, &mut result),+ }+ }+ CorrelatedDrainResult {+ durable: result.durable,+ remaining: result.remaining,+ dropped: result.dropped,+ }+ }++ pub fn complete_spool_only(&self) -> std::io::Result<bool> {+ let lock = open_recovery_coord_lock(&self.spool)?;+ lock.lock_exclusive()?;+ let active = self.dir.join(RECOVERY_MATCH_FILE);+ let completed = self.dir.join(RECOVERY_COMPLETED_FILE);+ let spec = read_recovery_match(&active)?;+ move_late_matching_entries(&self.spool, &self.dir, &self.config, &spec)?;+ if self.len() != 0 {+ return Ok(false);+ }+ std::fs::rename(active, completed)?;+ Ok(true)+ }++ pub fn restore(self, spool: &Path) -> std::io::Result<RestoreOutcome> {+ if !self.dir.exists() {+ return Ok(RestoreOutcome::Restored);+ }+ let lock = open_recovery_coord_lock(spool)?;+ lock.lock_exclusive()?;+ let active = self.dir.join(RECOVERY_MATCH_FILE);+ let completed = self.dir.join(RECOVERY_COMPLETED_FILE);+ // Completion is terminal: a `.completed` marker is durable proof the+ // interval's correlated evidence reached the server, so a restore has+ // nothing to put back. A `.active` beside the tombstone is a rerun+ // rewriting the same interval, not a re-opened one.+ if completed.is_file() {+ match std::fs::remove_file(&active) {+ Ok(()) => {}+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}+ Err(error) => return Err(error),+ }+ move_quarantined_entries_to_spool(&self.dir, spool)?;+ return Ok(RestoreOutcome::AlreadyDelivered);+ }+ if !active.is_file() {+ return Err(std::io::Error::new(+ std::io::ErrorKind::NotFound,+ format!(+ "recovery interval {} has neither an active nor a completed marker",+ self.dir.display()+ ),+ ));+ }+ let spec = read_recovery_match(&active)?;+ move_late_matching_entries(spool, &self.dir, &self.config, &spec)?;+ let _ = std::fs::remove_file(active);+ move_quarantined_entries_to_spool(&self.dir, spool)?;+ std::fs::remove_dir(self.dir)?;+ Ok(RestoreOutcome::Restored)+ }+}++/// Move every quarantined `*.json` event back to the plain spool, leaving the+/// marker files where they are.+fn move_quarantined_entries_to_spool(dir: &Path, spool: &Path) -> std::io::Result<()> {+ for item in std::fs::read_dir(dir)? {+ let item = item?;+ if item.path().extension().and_then(|value| value.to_str()) != Some("json") {+ continue;+ }+ let destination = spool.join(item.file_name());+ std::fs::rename(item.path(), destination)?;+ }+ Ok(())+}++fn read_recovery_match(path: &Path) -> std::io::Result<RecoveryMatch> {+ serde_json::from_slice(&read_spool_entry_bytes(path)?)+ .map_err(|error| std::io::Error::new(std::io::ErrorKind::InvalidData, error))+}++fn move_late_matching_entries(+ spool: &Path,+ dir: &Path,+ config: &crate::config::Config,+ spec: &RecoveryMatch,+) -> std::io::Result<()> {+ let (files, _) = list_entries(spool)?;+ for path in files {+ let Ok(bytes) = read_spool_entry_bytes(&path) else {+ continue;+ };+ let Ok(entry) = serde_json::from_slice::<SpoolEntry>(&bytes) else {+ continue;+ };+ if entry.created_ms >= spec.interval_started_ms+ && entry.created_ms <= spec.interval_ended_ms+ && spool_entry_matches_session(+ &entry,+ &spec.server_identity,+ &spec.agent,+ Some(config),+ &spec.workspace,+ &spec.project,+ Path::new(&spec.cwd),+ &spec.native_session_id,+ )+ && recovery_target_for_entry(spool, &entry, Some(config))?+ .is_some_and(|target| target.dir() == dir)+ && let Some(name) = path.file_name()+ {+ std::fs::rename(&path, dir.join(name))?;+ }+ }+ Ok(())+}++#[allow(clippy::too_many_arguments)]+pub fn quarantine_session_entries(+ config: &crate::config::Config,+ spool: &Path,+ journal_id: &str,+ interval_started_ms: u64,+ interval_ended_ms: u64,+ server_identity: &str,+ agent: &str,+ workspace: &str,+ project: &str,+ cwd: &Path,+ native_session_id: &str,+) -> std::io::Result<QuarantinedSpool> {+ let coord_lock = open_recovery_coord_lock(spool)?;+ coord_lock.lock_exclusive()?;+ sweep_recovery_tombstones(spool)?;+ let coordinate_key = recovery_coordinate_key(+ server_identity,+ agent,+ workspace,+ project,+ cwd.to_string_lossy().as_ref(),+ native_session_id,+ )?;+ let dir = spool+ .join(RECOVERY_DIR)+ .join(recovery_interval_key(journal_id)?);+ create_spool_dir(&dir)?;+ let match_spec = RecoveryMatch {+ journal_id: journal_id.to_owned(),+ coordinate_key,+ interval_started_ms,+ interval_ended_ms,+ server_identity: server_identity.to_owned(),+ agent: agent.to_owned(),+ workspace: workspace.to_owned(),+ project: project.to_owned(),+ cwd: cwd.to_string_lossy().into_owned(),+ native_session_id: native_session_id.to_owned(),+ };+ write_private(+ &dir.join(RECOVERY_MATCH_FILE),+ &serde_json::to_vec(&match_spec)?,+ )?;+ if let Err(error) = move_late_matching_entries(spool, &dir, config, &match_spec) {+ let _ = std::fs::remove_file(dir.join(RECOVERY_MATCH_FILE));+ return Err(error);+ }+ Ok(QuarantinedSpool {+ dir,+ spool: spool.to_path_buf(),+ config: config.clone(),+ })+}++fn open_recovery_coord_lock(spool: &Path) -> std::io::Result<File> {+ create_spool_dir(spool)?;+ std::fs::OpenOptions::new()+ .create(true)+ .truncate(false)+ .read(true)+ .write(true)+ .open(spool.with_extension("recovery.lock"))+}++fn sweep_recovery_tombstones(spool: &Path) -> std::io::Result<()> {+ let recovery = spool.join(RECOVERY_DIR);+ let entries = match std::fs::read_dir(&recovery) {+ Ok(entries) => entries,+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()),+ Err(error) => return Err(error),+ };+ let now = SystemTime::now();+ let mut completed = Vec::new();+ for entry in entries.take(MAX_RECOVERY_SWEEP_DIRS) {+ let entry = entry?;+ if !entry.file_type()?.is_dir() {+ continue;+ }+ let marker = entry.path().join(RECOVERY_COMPLETED_FILE);+ let Ok(metadata) = marker.metadata() else {+ continue;+ };+ let modified = metadata.modified().unwrap_or(UNIX_EPOCH);+ if now.duration_since(modified).unwrap_or_default().as_millis() as u64+ > RECOVERY_TOMBSTONE_AGE_MS+ {+ std::fs::remove_dir_all(entry.path())?;+ } else {+ completed.push((modified, entry.path()));+ }+ }+ if completed.len() > MAX_RECOVERY_TOMBSTONES {+ completed.sort_by_key(|(modified, _)| *modified);+ let excess = completed.len() - MAX_RECOVERY_TOMBSTONES;+ for (_, path) in completed.into_iter().take(excess) {+ std::fs::remove_dir_all(path)?;+ }+ }+ Ok(())+}++enum RecoveryTarget {+ Active(PathBuf),+ Completed(PathBuf),+}++impl RecoveryTarget {+ fn dir(&self) -> &Path {+ match self {+ Self::Active(dir) | Self::Completed(dir) => dir,+ }+ }+}++fn recovery_target_for_entry(+ spool: &Path,+ entry: &SpoolEntry,+ config: Option<&crate::config::Config>,+) -> std::io::Result<Option<RecoveryTarget>> {+ let coordinate_key = recovery_coordinate_key_for_entry(entry, config)?;+ let recovery = spool.join(RECOVERY_DIR);+ let directories = match std::fs::read_dir(recovery) {+ Ok(entries) => entries,+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),+ Err(error) => return Err(error),+ };+ let mut matched = None;+ for directory in directories.take(MAX_RECOVERY_SWEEP_DIRS) {+ let directory = directory?;+ if !directory.file_type()?.is_dir() {+ continue;+ }+ let dir = directory.path();+ let marker = if dir.join(RECOVERY_MATCH_FILE).is_file() {+ RECOVERY_MATCH_FILE+ } else if dir.join(RECOVERY_COMPLETED_FILE).is_file() {+ RECOVERY_COMPLETED_FILE+ } else {+ continue;+ };+ let Ok(spec) = read_recovery_match(&dir.join(marker)) else {+ continue;+ };+ if spec.coordinate_key != coordinate_key+ || entry.created_ms < spec.interval_started_ms+ || entry.created_ms > spec.interval_ended_ms+ {+ continue;+ }+ let candidate = if marker == RECOVERY_MATCH_FILE {+ RecoveryTarget::Active(dir)+ } else {+ RecoveryTarget::Completed(dir)+ };+ if matched.is_some() {+ return Ok(None);+ }+ matched = Some(candidate);+ }+ Ok(matched)+}++fn recovery_coordinate_key_for_entry(+ entry: &SpoolEntry,+ config: Option<&crate::config::Config>,+) -> std::io::Result<String> {+ let url = reqwest::Url::parse(&entry.url)+ .map_err(|error| std::io::Error::new(std::io::ErrorKind::InvalidData, error))?;+ let mut identity = url.clone();+ identity.set_query(None);+ if identity.path().ends_with("/hook") {+ let path = identity.path().trim_end_matches("/hook").to_owned();+ identity.set_path(&path);+ }+ let mut agent = None;+ let mut workspace = None;+ let mut project = None;+ let mut session = None;+ let mut cwd = None;+ for (key, value) in url.query_pairs() {+ match key.as_ref() {+ "agent" => agent = Some(value.into_owned()),+ "workspace" => workspace = Some(value.into_owned()),+ "project" => project = Some(value.into_owned()),+ "session_id" => session = Some(value.into_owned()),+ "cwd" => cwd = Some(value.into_owned()),+ _ => {}+ }+ }+ if let Ok(body) = serde_json::from_str::<serde_json::Value>(&entry.body) {+ let context = ai_memory_hooks::hook_payload_context(&body);+ session = context.session_id.or(session);+ cwd = context.cwd.or(cwd);+ }+ let (workspace, project) = match (config, cwd.as_deref()) {+ (Some(config), Some(cwd)) => super::resolve_scope_for_path_with_explicit(+ config,+ Path::new(cwd),+ workspace.as_deref(),+ project.as_deref(),+ )+ .unwrap_or_else(|_| {+ (+ workspace.clone().unwrap_or_default(),+ project.clone().unwrap_or_default(),+ )+ }),+ _ => (workspace.unwrap_or_default(), project.unwrap_or_default()),+ };+ recovery_coordinate_key(+ identity.as_str().trim_end_matches('/'),+ agent.as_deref().unwrap_or_default(),+ &workspace,+ &project,+ cwd.as_deref().unwrap_or_default(),+ session.as_deref().unwrap_or_default(),+ )+}++fn recovery_interval_key(journal_id: &str) -> std::io::Result<String> {+ if journal_id.is_empty()+ || journal_id.len() > 128+ || !journal_id+ .bytes()+ .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_'))+ {+ return Err(std::io::Error::new(+ std::io::ErrorKind::InvalidInput,+ "invalid recovery journal interval id",+ ));+ }+ use sha2::{Digest as _, Sha256};+ Ok(format!("{:x}", Sha256::digest(journal_id.as_bytes())))+}++fn recovery_coordinate_key(+ server_identity: &str,+ agent: &str,+ workspace: &str,+ project: &str,+ cwd: &str,+ native_session_id: &str,+) -> std::io::Result<String> {+ use sha2::{Digest as _, Sha256};+ let cwd = crate::marker::absolute_normalized(Path::new(cwd));+ let mut hasher = Sha256::new();+ for value in [+ server_identity,+ agent,+ workspace,+ project,+ cwd.to_string_lossy().as_ref(),+ native_session_id,+ ] {+ hasher.update(value.len().to_be_bytes());+ hasher.update(value.as_bytes());+ }+ Ok(format!("{:x}", hasher.finalize()))+}++#[allow(clippy::too_many_arguments)]+fn spool_entry_matches_session(+ entry: &SpoolEntry,+ server_identity: &str,+ agent: &str,+ config: Option<&crate::config::Config>,+ workspace: &str,+ project: &str,+ cwd: &Path,+ native_session_id: &str,+) -> bool {+ let Ok(url) = reqwest::Url::parse(&entry.url) else {+ return false;+ };+ let mut identity = url.clone();+ identity.set_query(None);+ if identity.path().ends_with("/hook") {+ let path = identity.path().trim_end_matches("/hook").to_owned();+ identity.set_path(&path);+ }+ let identity = identity.as_str().trim_end_matches('/');+ if identity != server_identity.trim_end_matches('/') {+ return false;+ }+ let mut session = None;+ let mut event_agent = None;+ let mut ws = None;+ let mut proj = None;+ let mut event_cwd = None;+ for (key, value) in url.query_pairs() {+ match key.as_ref() {+ "agent" => event_agent = Some(value.into_owned()),+ "session_id" => session = Some(value.into_owned()),+ "workspace" => ws = Some(value.into_owned()),+ "project" => proj = Some(value.into_owned()),+ "cwd" => event_cwd = Some(value.into_owned()),+ _ => {}+ }+ }+ if let Ok(body) = serde_json::from_str::<serde_json::Value>(&entry.body) {+ let context = ai_memory_hooks::hook_payload_context(&body);+ session = context.session_id.or(session);+ event_cwd = context.cwd.or(event_cwd);+ }+ let expected_cwd = crate::marker::absolute_normalized(cwd);+ let same_cwd = event_cwd+ .as_deref()+ .is_some_and(|value| crate::marker::absolute_normalized(Path::new(value)) == expected_cwd);+ let explicit_match = ws.as_deref() == Some(workspace) && proj.as_deref() == Some(project);+ let scope_matches = if explicit_match {+ true+ } else if let (Some(config), Some(event_cwd)) = (config, event_cwd.as_deref()) {+ super::resolve_scope_for_path_with_explicit(+ config,+ Path::new(event_cwd),+ ws.as_deref(),+ proj.as_deref(),+ )+ .is_ok_and(|resolved| resolved == (workspace.to_owned(), project.to_owned()))+ } else {+ ws.as_deref().is_none_or(|value| value == workspace)+ && proj.as_deref().is_none_or(|value| value == project)+ && ((ws.is_some() && proj.is_some()) || same_cwd)+ };+ event_agent.as_deref() == Some(agent)+ && session.as_deref() == Some(native_session_id)+ && scope_matches+ && same_cwd+}+/// Windows can report a contended fs2 byte-range lock as this native OS code/// instead of mapping it to `WouldBlock`.#[cfg(windows)]@@ -485,43 +1167,87 @@ pub async fn drain_until_quiescent(per_event_timeout: Duration,wait: DrainLockWait,max_attempts: MaxAttempts,+) -> std::io::Result<LockedDrainResult> {+ drain_until_quiescent_with_live_token(+ spool,+ data_dir,+ total_budget,+ per_event_timeout,+ wait,+ live_static_token().as_deref(),+ max_attempts,+ )+ .await+}++pub async fn drain_until_quiescent_with_live_token(+ spool: &Path,+ data_dir: &Path,+ total_budget: Duration,+ per_event_timeout: Duration,+ wait: DrainLockWait,+ live_token: Option<&str>,+ max_attempts: MaxAttempts,) -> std::io::Result<LockedDrainResult> {let Some(_lock) = acquire_drain_lock(spool, wait)? else {return Ok(LockedDrainResult::LockBusy);};+ Ok(LockedDrainResult::Drained(+ drain_until_quiescent_locked_with_live_token(+ spool,+ data_dir,+ total_budget,+ per_event_timeout,+ live_token,+ max_attempts,+ )+ .await,+ ))+}++pub async fn drain_until_quiescent_locked_with_live_token(+ spool: &Path,+ data_dir: &Path,+ total_budget: Duration,+ per_event_timeout: Duration,+ live_token: Option<&str>,+ max_attempts: MaxAttempts,+) -> DrainResult {let started = Instant::now();let mut combined = DrainResult::default();loop {let Some(remaining_budget) = total_budget.checked_sub(started.elapsed()) else {combined.remaining = spool_len(spool);- return Ok(LockedDrainResult::Drained(combined));+ return combined;};if remaining_budget.is_zero() {combined.remaining = spool_len(spool);- return Ok(LockedDrainResult::Drained(combined));+ return combined;}- let result = drain_with_max_attempts(+ let result = drain_with_live_token(spool,data_dir,remaining_budget,per_event_timeout,+ live_token,max_attempts,).await;combined.sent += result.sent;+ combined.durable += result.durable;combined.dropped += result.dropped;combined.remaining = result.remaining;if result.remaining > 0 {- return Ok(LockedDrainResult::Drained(combined));+ return combined;}let queued = spool_len(spool);if queued == 0 {combined.remaining = 0;- return Ok(LockedDrainResult::Drained(combined));+ return combined;}combined.remaining = queued;}@@ -725,6 +1451,49 @@ pub async fn drain_with_live_token(let batch_timeout =batch_request_timeout(per_event_timeout, remaining_budget, chunk.len());match post_batch(&client, &base, &payload, bearer.as_deref(), batch_timeout).await {+ BatchOutcome::Detailed {+ outcomes,+ failed_index,+ } => {+ if outcomes.len() > chunk.len()+ || failed_index.is_some_and(|index| index >= chunk.len())+ {+ bump_or_drop(&chunk[0].0, &chunk[0].1, max_attempts, &mut result);+ result.remaining +=+ chunk.len().saturating_sub(1) + files.len().saturating_sub(idx);+ break;+ }+ for (position, outcome) in outcomes.iter().enumerate() {+ let terminal = matches!(outcome.as_str(), "stored" | "replayed")+ || outcome == "ignored_end"+ && entry_event(&chunk[position].1).as_deref()+ == Some("session-end");+ if terminal {+ let _ = std::fs::remove_file(&chunk[position].0);+ result.sent += 1;+ result.durable += 1;+ } else {+ bump_or_drop(+ &chunk[position].0,+ &chunk[position].1,+ max_attempts,+ &mut result,+ );+ }+ }+ if let Some(failed_index) = failed_index {+ bump_or_drop(+ &chunk[failed_index].0,+ &chunk[failed_index].1,+ max_attempts,+ &mut result,+ );+ }+ result.remaining += chunk+ .len()+ .saturating_sub(outcomes.len())+ .saturating_sub(usize::from(failed_index.is_some()));+ }BatchOutcome::Accepted(k) => {let k = k.min(chunk.len());for (path, _) in &chunk[..k] {@@ -909,6 +1678,24 @@ pub async fn drain_with_live_token().await{+ BatchOutcome::Detailed { outcomes, .. } => {+ let durable = outcomes+ .iter()+ .enumerate()+ .filter(|(position, outcome)| {+ matches!(outcome.as_str(), "stored" | "replayed")+ || outcome.as_str() == "ignored_end"+ && entry_event(&chunk[*position].1).as_deref()+ == Some("session-end")+ })+ .map(|(position, _)| position)+ .collect::<Vec<_>>();+ let sent = delete_accepted_indices(&chunk, &durable);+ result.sent += sent;+ result.durable += sent;+ result.remaining += chunk.len().saturating_sub(sent);+ continue;+ }BatchOutcome::Accepted(k) => {let k = k.min(chunk.len());for (path, _) in &chunk[..k] {@@ -978,6 +1765,24 @@ pub async fn drain_with_live_token(match post_batch(&client, &base, &payload, Some(&token), batch_timeout).await{+ BatchOutcome::Detailed { outcomes, .. } => {+ let durable = outcomes+ .iter()+ .enumerate()+ .filter(|(position, outcome)| {+ matches!(outcome.as_str(), "stored" | "replayed")+ || outcome.as_str() == "ignored_end"+ && entry_event(&chunk[*position].1).as_deref()+ == Some("session-end")+ })+ .map(|(position, _)| position)+ .collect::<Vec<_>>();+ let sent = delete_accepted_indices(&chunk, &durable);+ result.sent += sent;+ result.durable += sent;+ result.remaining += chunk.len().saturating_sub(sent);+ continue;+ }BatchOutcome::Accepted(k) => {let k = k.min(chunk.len());for (path, _) in &chunk[..k] {@@ -1115,8 +1920,71 @@ fn batch_request_timeout(.min(remaining_budget)}+fn completed_tombstone_for_path(path: &Path) -> bool {+ let Some(name) = path.file_name().and_then(|name| name.to_str()) else {+ return false;+ };+ let Some(stem) = name.strip_suffix(".json") else {+ return false;+ };+ let Some(coordinate_key) = stem.rsplit('-').next() else {+ return false;+ };+ let Some(created_ms) = created_ms_from_name(name) else {+ return false;+ };+ let Some(spool) = path.parent() else {+ return false;+ };+ let Ok(directories) = std::fs::read_dir(spool.join(RECOVERY_DIR)) else {+ return false;+ };+ let mut matches = 0usize;+ for directory in directories.take(MAX_RECOVERY_SWEEP_DIRS).flatten() {+ let marker = directory.path().join(RECOVERY_COMPLETED_FILE);+ let Ok(spec) = read_recovery_match(&marker) else {+ continue;+ };+ if spec.coordinate_key == coordinate_key+ && created_ms >= spec.interval_started_ms+ && created_ms <= spec.interval_ended_ms+ {+ matches += 1;+ if matches > 1 {+ return false;+ }+ }+ }+ matches == 1+}++fn read_spool_entry_bytes(path: &Path) -> std::io::Result<Vec<u8>> {+ use std::io::Read as _;+ let file = File::open(path)?;+ if file.metadata()?.len() > MAX_SPOOL_ENTRY_BYTES {+ return Err(std::io::Error::new(+ std::io::ErrorKind::InvalidData,+ "spool entry exceeds its byte limit",+ ));+ }+ let mut bytes = Vec::new();+ file.take(MAX_SPOOL_ENTRY_BYTES.saturating_add(1))+ .read_to_end(&mut bytes)?;+ if bytes.len() as u64 > MAX_SPOOL_ENTRY_BYTES {+ return Err(std::io::Error::new(+ std::io::ErrorKind::InvalidData,+ "spool entry exceeds its byte limit",+ ));+ }+ Ok(bytes)+}+fn load_live_entry(path: &Path, result: &mut DrainResult) -> Option<SpoolEntry> {- let Ok(bytes) = std::fs::read(path) else {+ if completed_tombstone_for_path(path) {+ let _ = std::fs::remove_file(path);+ return None;+ }+ let Ok(bytes) = read_spool_entry_bytes(path) else {result.remaining += 1;return None;};@@ -1317,6 +2185,13 @@ fn configured_server_url(data_dir: &Path) -> Option<String> {/// The `/hook/batch` URL for a spooled per-event URL: strip the `?…` query and/// append `/batch` (a spooled URL ends in `…/hook` before its query). Entries/// whose endpoint string matches can ride one batch request.+fn entry_event(entry: &SpoolEntry) -> Option<String> {+ reqwest::Url::parse(&entry.url)+ .ok()?+ .query_pairs()+ .find_map(|(key, value)| (key == "event").then(|| value.into_owned()))+}+fn batch_endpoint(url: &str) -> String {let path = url.split('?').next().unwrap_or(url);format!("{path}/batch")@@ -1395,28 +2270,6 @@ pub async fn resolve_bearer(crate::auth_bearer::resolve_bearer(client, &data_dir.join("auth.json"), auth_token).await}-fn prune_spool_file_count(spool: &Path) {- let Ok((mut files, _)) = list_entries(spool) else {- return;- };- let excess = files.len().saturating_sub(MAX_SPOOL_FILES);- if excess == 0 {- return;- }- files.sort();- // The spool is at its hard cap: the oldest events are about to be deleted- // WITHOUT ever reaching the server — silent capture loss otherwise. Surface- // it on stderr (never stdout, which carries the hook's JSON protocol output)- // so a sustained backlog dropping events is visible, not invisible.- eprintln!(- "ai-memory: hook-spool at capacity ({} > {MAX_SPOOL_FILES}); evicting {excess} oldest UNDELIVERED event(s)",- files.len()- );- for path in files.into_iter().take(excess) {- let _ = std::fs::remove_file(path);- }-}-/// List `*.json` spool files (ignoring in-flight `*.json.tmp`), or None when the/// directory doesn't exist yet./// Spool files awaiting delivery, or the IO error that prevented listing them.@@ -1441,6 +2294,9 @@ fn list_entries(spool: &Path) -> std::io::Result<(Vec<PathBuf>, usize)> {let path = ent.path();if path.extension().and_then(|e| e.to_str()) == Some("json") {out.push(path);+ if out.len() > MAX_SPOOL_FILES {+ break;+ }}}Err(_) => unreadable += 1,@@ -1453,6 +2309,457 @@ fn list_entries(spool: &Path) -> std::io::Result<(Vec<PathBuf>, usize)> {mod tests {use super::*;+ #[test]+ fn correlated_session_entries_quarantine_and_restore_without_touching_others() {+ let temp = tempfile::tempdir().unwrap();+ let spool = temp.path().join("hook-spool");+ let matching = SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project&session_id=native-1".into(),+ body: r#"{"session_id":"native-1","cwd":"/repo","prompt":"same prompt"}"#.into(),+ created_ms: now_ms(),+ auth_mode: AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ let other = SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&workspace=ws&project=project&session_id=native-2".into(),+ body: r#"{"session_id":"native-2","cwd":"/repo","prompt":"other"}"#.into(),+ ..matching.clone()+ };+ enqueue(&spool, &matching).unwrap();+ enqueue(&spool, &other).unwrap();+ let quarantine = quarantine_session_entries(+ &crate::config::Config::default(),+ &spool,+ "journal-1",+ matching.created_ms.saturating_sub(1),+ matching.created_ms.saturating_add(1),+ "http://127.0.0.1:49374",+ "claude-code",+ "ws",+ "project",+ Path::new("/repo"),+ "native-1",+ )+ .unwrap();+ assert_eq!(spool_len(&spool), 1);+ quarantine.restore(&spool).unwrap();+ assert_eq!(spool_len(&spool), 2);+ }++ /// A quarantine whose `.active` marker is gone because the interval+ /// already completed restores as already-delivered instead of erroring —+ /// including when a rerun rewrote a fresh `.active` over the completed+ /// tombstone — while a directory with neither marker is still an error.+ #[test]+ fn restore_treats_a_completed_interval_as_already_delivered() {+ let temp = tempfile::tempdir().unwrap();+ let spool = temp.path().join("hook-spool");+ let quarantine = |journal: &str| {+ quarantine_session_entries(+ &crate::config::Config::default(),+ &spool,+ journal,+ 0,+ u64::MAX,+ "http://127.0.0.1:49374",+ "claude-code",+ "ws",+ "project",+ Path::new("/repo"),+ "native-1",+ )+ };+ let interval_dir = |journal: &str| {+ spool+ .join(RECOVERY_DIR)+ .join(recovery_interval_key(journal).unwrap())+ };+ let completed = quarantine("journal-done").unwrap();+ assert!(completed.complete_spool_only().unwrap());+ assert_eq!(+ completed.restore(&spool).unwrap(),+ RestoreOutcome::AlreadyDelivered,+ "a missing .active beside a completed marker means durable delivery"+ );+ let rerun = quarantine("journal-done").unwrap();+ assert!(+ interval_dir("journal-done")+ .join(RECOVERY_MATCH_FILE)+ .is_file(),+ "the rerun rewrites .active over the completed tombstone"+ );+ assert_eq!(+ rerun.restore(&spool).unwrap(),+ RestoreOutcome::AlreadyDelivered,+ "a rerun's rewritten .active never re-opens a completed interval"+ );+ assert!(+ interval_dir("journal-done")+ .join(RECOVERY_COMPLETED_FILE)+ .is_file()+ );+ assert!(+ !interval_dir("journal-done")+ .join(RECOVERY_MATCH_FILE)+ .is_file()+ );++ let broken = quarantine("journal-broken").unwrap();+ std::fs::remove_file(interval_dir("journal-broken").join(RECOVERY_MATCH_FILE)).unwrap();+ assert!(+ broken.restore(&spool).is_err(),+ "a directory with neither marker is still an error"+ );+ }++ #[test]+ fn quarantine_matching_reuses_every_canonical_nested_hook_context_shape() {+ let base = SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),+ body: String::new(),+ created_ms: now_ms(),+ auth_mode: AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ let matching = [+ serde_json::json!({"info": {"id": "native-1", "directory": "/repo"}}),+ serde_json::json!({"properties": {"session_id": "native-1", "cwd": "/repo"}}),+ serde_json::json!({"properties": {"sessionID": "native-1", "cwd": "/repo"}}),+ serde_json::json!({"properties": {"info": {"id": "native-1", "directory": "/repo"}}}),+ serde_json::json!({"event": {"sessionId": "native-1", "working_dir": "/repo"}}),+ serde_json::json!({"event": {"properties": {"sessionID": "native-1", "cwd": "/repo"}}}),+ serde_json::json!({"event": {"properties": {"info": {"id": "native-1", "directory": "/repo"}}}}),+ serde_json::json!({"payload": {"conversationId": "native-1", "directory": "/repo"}}),+ serde_json::json!({"payload": {"path": {"session": "native-1", "cwd": "/repo"}}}),+ serde_json::json!({"payload": {"info": {"id": "native-1", "directory": "/repo"}}}),+ serde_json::json!({"payload": {"properties": {"sessionID": "native-1", "cwd": "/repo"}}}),+ serde_json::json!({"payload": {"properties": {"info": {"id": "native-1", "directory": "/repo"}}}}),+ ];+ for body in matching {+ let entry = SpoolEntry {+ body: body.to_string(),+ ..base.clone()+ };+ assert!(spool_entry_matches_session(+ &entry,+ "http://127.0.0.1:49374",+ "claude-code",+ None,+ "ws",+ "project",+ Path::new("/repo"),+ "native-1",+ ));+ }+ for body in [+ serde_json::json!({"payload": {"info": {"id": "other", "directory": "/repo"}}}),+ serde_json::json!({"payload": {"info": {"id": "native-1", "directory": "/other"}}}),+ serde_json::json!({"payload": {"info": {"id": "native-1"}}, "unrelated": {"cwd": "/repo"}}),+ ] {+ let entry = SpoolEntry {+ body: body.to_string(),+ ..base.clone()+ };+ assert!(!spool_entry_matches_session(+ &entry,+ "http://127.0.0.1:49374",+ "claude-code",+ None,+ "ws",+ "project",+ Path::new("/repo"),+ "native-1",+ ));+ }+ }++ #[test]+ fn quarantine_matches_partial_scope_and_captures_late_enqueue() {+ let temp = tempfile::tempdir().unwrap();+ let repo = temp.path().join("project");+ std::fs::create_dir_all(&repo).unwrap();+ std::fs::write(+ repo.join(".ai-memory.toml"),+ "workspace = \"ws\"\nproject = \"project\"\n",+ )+ .unwrap();+ let spool = temp.path().join("hook-spool");+ let config = crate::config::Config::default();+ let base = SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws".into(),+ body: serde_json::json!({"session_id":"native-1","cwd":repo,"prompt":"one"})+ .to_string(),+ created_ms: now_ms(),+ auth_mode: AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ enqueue_with_config(&spool, &base, Some(&config)).unwrap();+ let quarantine = quarantine_session_entries(+ &config,+ &spool,+ "journal-partial",+ base.created_ms.saturating_sub(1),+ base.created_ms.saturating_add(1),+ "http://127.0.0.1:49374",+ "claude-code",+ "ws",+ "project",+ &repo,+ "native-1",+ )+ .unwrap();+ assert_eq!(+ quarantine.len(),+ 1,+ "workspace-only event is correlated by cwd"+ );+ enqueue_with_config(+ &spool,+ &SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=stop&agent=claude-code&project=project"+ .into(),+ body: serde_json::json!({"session_id":"native-1","cwd":repo}).to_string(),+ ..base.clone()+ },+ Some(&config),+ )+ .unwrap();+ assert_eq!(+ quarantine.len(),+ 2,+ "late enqueue joins the active quarantine"+ );+ assert_eq!(spool_len(&spool), 0, "neither correlated event can drain");+ quarantine.restore(&spool).unwrap();+ assert_eq!(spool_len(&spool), 2);+ }++ #[test]+ fn enqueue_is_bounded_when_recovery_coordination_lock_is_held() {+ let temp = tempfile::tempdir().unwrap();+ let spool = temp.path().join("hook-spool");+ let lock = open_recovery_coord_lock(&spool).unwrap();+ lock.lock_exclusive().unwrap();+ let entry = entry_for(+ "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),+ r#"{"session_id":"native-1","cwd":"/repo","prompt":"late"}"#.into(),+ None,+ false,+ );+ let started = Instant::now();+ enqueue(&spool, &entry).unwrap();+ assert!(started.elapsed() < Duration::from_millis(100));+ assert_eq!(spool_len(&spool), 1, "content is retained for recovery");+ }++ #[test]+ fn completion_race_leaves_contended_enqueue_safe_for_tombstone_cleanup() {+ let temp = tempfile::tempdir().unwrap();+ let spool = temp.path().join("hook-spool");+ let config = crate::config::Config::default();+ let entry = SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),+ body: r#"{"session_id":"native-1","cwd":"/repo","prompt":"late"}"#.into(),+ created_ms: now_ms(),+ auth_mode: AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ let quarantine = quarantine_session_entries(+ &config,+ &spool,+ "journal",+ entry.created_ms.saturating_sub(1),+ entry.created_ms.saturating_add(1),+ "http://127.0.0.1:49374",+ "claude-code",+ "ws",+ "project",+ Path::new("/repo"),+ "native-1",+ )+ .unwrap();+ PAUSE_ENQUEUE_AFTER_MAIN_RENAME.store(true, Ordering::SeqCst);+ ENQUEUE_REACHED_RECOVERY_WINDOW.store(false, Ordering::SeqCst);+ let enqueue_spool = spool.clone();+ let enqueue_config = config.clone();+ let enqueue = std::thread::spawn(move || {+ enqueue_with_config(&enqueue_spool, &entry, Some(&enqueue_config)).unwrap()+ });+ while !ENQUEUE_REACHED_RECOVERY_WINDOW.load(Ordering::SeqCst) {+ std::thread::yield_now();+ }+ quarantine.complete().unwrap();+ PAUSE_ENQUEUE_AFTER_MAIN_RENAME.store(false, Ordering::SeqCst);+ enqueue.join().unwrap();+ let mut result = DrainResult::default();+ if let Some(path) = list_entries(&spool).unwrap().0.into_iter().next() {+ assert!(load_live_entry(&path, &mut result).is_none());+ }+ assert_eq!(spool_len(&spool), 0, "completed tombstone removes overlap");+ }++ #[test]+ fn completed_recovery_tombstone_retires_late_overlap() {+ let temp = tempfile::tempdir().unwrap();+ let spool = temp.path().join("hook-spool");+ let entry = SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),+ body: r#"{"session_id":"native-1","cwd":"/repo","prompt":"late"}"#.into(),+ created_ms: now_ms(),+ auth_mode: AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ let quarantine = quarantine_session_entries(+ &crate::config::Config::default(),+ &spool,+ "journal",+ entry.created_ms.saturating_sub(1),+ entry.created_ms.saturating_add(1),+ "http://127.0.0.1:49374",+ "claude-code",+ "ws",+ "project",+ Path::new("/repo"),+ "native-1",+ )+ .unwrap();+ quarantine.complete().unwrap();+ enqueue(&spool, &entry).unwrap();+ assert_eq!(spool_len(&spool), 0, "late overlap must not be delivered");+ }++ /// The completed tombstone is bounded by its interval: a hook from a+ /// LATER resume of the same native session is fresh evidence and must be+ /// delivered, never swallowed by the earlier interval's tombstone.+ #[test]+ fn completed_tombstone_keeps_a_later_resume_of_the_same_session() {+ let temp = tempfile::tempdir().unwrap();+ let spool = temp.path().join("hook-spool");+ let interval_ended = now_ms().saturating_sub(60_000);+ let quarantine = quarantine_session_entries(+ &crate::config::Config::default(),+ &spool,+ "journal",+ interval_ended.saturating_sub(1_000),+ interval_ended,+ "http://127.0.0.1:49374",+ "claude-code",+ "ws",+ "project",+ Path::new("/repo"),+ "native-1",+ )+ .unwrap();+ quarantine.complete().unwrap();+ let later_resume = SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),+ body: r#"{"session_id":"native-1","cwd":"/repo","prompt":"from a later resume"}"#.into(),+ created_ms: now_ms(),+ auth_mode: AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ enqueue(&spool, &later_resume).unwrap();+ assert_eq!(+ spool_len(&spool),+ 1,+ "a hook from a later resume must stay queued for delivery"+ );+ let path = list_entries(&spool).unwrap().0.into_iter().next().unwrap();+ let mut result = DrainResult::default();+ assert!(+ load_live_entry(&path, &mut result).is_some(),+ "the drain path must deliver, not drop, post-interval evidence"+ );+ assert_eq!(result.dropped, 0);+ }++ /// Completing one interval only ever consumes its own correlated entries:+ /// another interval's quarantine keeps its evidence, keeps capturing its+ /// late hooks, and an entry attributable to two intervals at once fails+ /// closed into the main spool instead of being attributed by guesswork.+ #[test]+ fn completing_one_recovery_interval_leaves_the_other_intervals_entries_intact() {+ let temp = tempfile::tempdir().unwrap();+ let spool = temp.path().join("hook-spool");+ let config = crate::config::Config::default();+ let first_ms = now_ms().saturating_sub(120_000);+ let second_ms = first_ms + 60_000;+ let entry = |created_ms: u64, prompt: &str| {+ SpoolEntry {+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),+ body: format!(r#"{{"session_id":"native-1","cwd":"/repo","prompt":"{prompt}"}}"#),+ created_ms,+ auth_mode: AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ }+ };+ let quarantine = |journal: &str, started: u64, ended: u64| {+ quarantine_session_entries(+ &config,+ &spool,+ journal,+ started,+ ended,+ "http://127.0.0.1:49374",+ "claude-code",+ "ws",+ "project",+ Path::new("/repo"),+ "native-1",+ )+ };+ enqueue(&spool, &entry(first_ms, "first interval")).unwrap();+ enqueue(&spool, &entry(second_ms, "second interval")).unwrap();+ let first = quarantine("journal-a", first_ms - 1, first_ms + 1).unwrap();+ let second = quarantine("journal-b", second_ms - 1, second_ms + 1).unwrap();+ assert_eq!(first.len(), 1);+ assert_eq!(second.len(), 1);+ assert_eq!(spool_len(&spool), 0);++ first.complete().unwrap();+ assert_eq!(+ second.len(),+ 1,+ "completing one interval never deletes another interval's correlated entries"+ );+ enqueue(&spool, &entry(second_ms, "second interval late hook")).unwrap();+ assert_eq!(+ second.len(),+ 2,+ "the still-active interval keeps capturing its own late hooks"+ );+ assert_eq!(spool_len(&spool), 0);++ let overlapping = quarantine("journal-c", second_ms - 1, second_ms + 1).unwrap();+ assert_eq!(+ overlapping.len(),+ 0,+ "the new interval claims nothing already attributed to journal-b"+ );+ enqueue(&spool, &entry(second_ms, "attributable to both")).unwrap();+ assert_eq!(+ spool_len(&spool),+ 1,+ "an entry two intervals could own fails closed into the main spool"+ );+ second.restore(&spool).unwrap();+ assert_eq!(spool_len(&spool), 3);+ }+#[test]fn entry_for_picks_auth_mode() {let s = entry_for("u".into(), "{}".into(), Some("tok"), false);@@ -1625,7 +2932,7 @@ mod tests {}#[test]- fn enqueue_prunes_oldest_files_when_spool_exceeds_limit() {+ fn enqueue_refuses_new_entries_when_spool_reaches_limit() {let tmp = tempfile::tempdir().unwrap();let spool = spool_dir(tmp.path());for i in 0..(MAX_SPOOL_FILES + 2) {@@ -1646,7 +2953,77 @@ mod tests {.iter().map(|path| serde_json::from_slice(&std::fs::read(path).unwrap()).unwrap()).collect();- assert!(bodies.iter().all(|entry| entry.created_ms >= 2));+ assert!(+ bodies+ .iter()+ .all(|entry| entry.created_ms < MAX_SPOOL_FILES as u64)+ );+ }++ #[test]+ fn capacity_tracks_current_global_occupancy_across_restart_and_quarantine() {+ let temp = tempfile::tempdir().unwrap();+ let spool = spool_dir(temp.path());+ for cycle in 0..(MAX_SPOOL_FILES * 3) {+ let entry = entry_for(+ format!("https://x/hook?event={cycle}"),+ "{}".into(),+ None,+ false,+ );+ enqueue(&spool, &entry).unwrap();+ let path = list_entries(&spool).unwrap().0.into_iter().next().unwrap();+ std::fs::remove_file(path).unwrap();+ }+ let accepted = entry_for(+ "https://x/hook?event=accepted".into(),+ "{}".into(),+ None,+ false,+ );+ enqueue(&spool, &accepted).unwrap();+ assert_eq!(total_spool_entries(&spool).unwrap(), 1);+ std::fs::remove_file(list_entries(&spool).unwrap().0[0].clone()).unwrap();++ let quarantine = spool.join(RECOVERY_DIR).join("full");+ std::fs::create_dir_all(&quarantine).unwrap();+ for index in 0..MAX_SPOOL_FILES {+ std::fs::write(quarantine.join(format!("{index}.json")), b"{}").unwrap();+ }+ enqueue(&spool, &accepted).unwrap();+ assert_eq!(+ spool_len(&spool),+ 0,+ "active quarantines count toward the cap"+ );+ }++ #[test]+ fn oversized_spool_entry_is_refused_before_read_allocation() {+ let temp = tempfile::tempdir().unwrap();+ let path = temp.path().join("oversized.json");+ let file = File::create(&path).unwrap();+ file.set_len(MAX_SPOOL_ENTRY_BYTES + 1).unwrap();+ let error = read_spool_entry_bytes(&path).unwrap_err();+ assert_eq!(error.kind(), std::io::ErrorKind::InvalidData);+ }++ #[test]+ fn tombstone_sweep_is_global_and_bounded_by_age_and_count() {+ let temp = tempfile::tempdir().unwrap();+ let spool = spool_dir(temp.path());+ let recovery = spool.join(RECOVERY_DIR);+ std::fs::create_dir_all(&recovery).unwrap();+ for index in 0..(MAX_RECOVERY_TOMBSTONES + 2) {+ let dir = recovery.join(format!("done-{index:04}"));+ std::fs::create_dir(&dir).unwrap();+ write_private(&dir.join(RECOVERY_COMPLETED_FILE), b"done").unwrap();+ }+ sweep_recovery_tombstones(&spool).unwrap();+ assert!(+ std::fs::read_dir(&recovery).unwrap().count() <= MAX_RECOVERY_TOMBSTONES,+ "completed tombstones are globally count-bounded"+ );}#[tokio::test]diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rsindex 02f18239..6f07049c 100644--- a/crates/ai-memory-cli/src/commands/mod.rs+++ b/crates/ai-memory-cli/src/commands/mod.rs@@ -80,6 +80,7 @@ pub mod purge_project;pub mod purge_session;pub mod read_page;pub mod reclaim_ledger_versions;+pub mod recovery;pub mod reindex;pub mod rename_project;pub mod rename_workstream;@@ -256,6 +257,15 @@ pub(crate) fn resolve_scope_with_marker(pub(crate) fn resolve_scope_for_path(config: &Config,cwd: &std::path::Path,+) -> Result<(String, String)> {+ resolve_scope_for_path_with_explicit(config, cwd, None, None)+}++pub(crate) fn resolve_scope_for_path_with_explicit(+ config: &Config,+ cwd: &std::path::Path,+ explicit_ws: Option<&str>,+ explicit_proj: Option<&str>,) -> Result<(String, String)> {if !cwd.exists() {return Err(anyhow!(@@ -266,39 +276,44 @@ pub(crate) fn resolve_scope_for_path(let cwd = crate::marker::absolute_normalized(cwd);let identity = cwd.to_string_lossy().into_owned();let marker = crate::marker::read_scope(&identity, &config.runtime_env)?;- let workspace = marker- .as_ref()- .and_then(|scope| scope.workspace.clone())+ let workspace = explicit_ws+ .filter(|value| !value.is_empty())+ .map(str::to_owned)+ .or_else(|| marker.as_ref().and_then(|scope| scope.workspace.clone())).unwrap_or_else(|| crate::config::DEFAULT_WORKSPACE.to_string());- let project = match marker.as_ref() {- Some(scope) => scope- .project- .clone()- .or_else(|| scope.canonical_remote_project())- .or_else(|| {- if scope.is_repo_root() {- crate::marker::repo_root_project(&identity)- } else {+ let project = if let Some(project) = explicit_proj.filter(|value| !value.is_empty()) {+ project.to_owned()+ } else {+ match marker.as_ref() {+ Some(scope) => scope+ .project+ .clone()+ .or_else(|| scope.canonical_remote_project())+ .or_else(|| {+ if scope.is_repo_root() {+ crate::marker::repo_root_project(&identity)+ } else {+ ai_memory_consolidate::derive_project_name(+ &cwd,+ ai_memory_consolidate::ProjectNameStrategy::Basename,+ )+ .map(|(name, _)| name)+ }+ })+ .ok_or_else(|| anyhow!("could not derive project name from {}", cwd.display()))?,+ None => crate::marker::discover_remote_identity(&identity)+ .and_then(|repository| {+ ai_memory_core::repository_identity::path_style_name(&repository)+ })+ .or_else(|| {ai_memory_consolidate::derive_project_name(&cwd,ai_memory_consolidate::ProjectNameStrategy::Basename,).map(|(name, _)| name)- }- })- .ok_or_else(|| anyhow!("could not derive project name from {}", cwd.display()))?,- None => crate::marker::discover_remote_identity(&identity)- .and_then(|repository| {- ai_memory_core::repository_identity::path_style_name(&repository)- })- .or_else(|| {- ai_memory_consolidate::derive_project_name(- &cwd,- ai_memory_consolidate::ProjectNameStrategy::Basename,- )- .map(|(name, _)| name)- })- .ok_or_else(|| anyhow!("could not derive project name from {}", cwd.display()))?,+ })+ .ok_or_else(|| anyhow!("could not derive project name from {}", cwd.display()))?,+ }};Ok((workspace, project))}diff --git a/crates/ai-memory-cli/src/commands/recovery.rs b/crates/ai-memory-cli/src/commands/recovery.rsnew file mode 100644index 00000000..ee639e90--- /dev/null+++ b/crates/ai-memory-cli/src/commands/recovery.rs@@ -0,0 +1,2141 @@+//! Downtime recovery — the offline journal and `ai-memory recover`.+//!+//! When the server is unreachable, `ai-memory run` degrades instead of+//! failing: hooks spool locally and a degraded launch still runs the harness.+//! The spool returns by itself, but two kinds of history used to need a+//! hand-typed repair and, for the transcript, were simply lost:+//!+//! - a run whose lease existed but whose transcript `finish` could not reach+//! the dead server (`finish-failed`);+//! - a degraded launch's session, which no server run ever recorded+//! (`degraded-run`).+//!+//! Both now append a bounded, metadata-only entry to a local journal+//! (`<data_dir>/recovery-journal.json`). `ai-memory recover`, run once the+//! server is back, drains the spool and then works the journal: it re-exports+//! each `finish-failed` session's transcript from the native store through the+//! ordinary managed-run finish path, and replays each degraded launch's exact+//! native transcript through the ordinary sanitized `/hook/batch` backfill+//! path. Harnesses without transcript export are journaled explicitly as+//! spool-only and recovery never invokes their exporter. Stable ingest and+//! event keys make both paths idempotent. Entries clear as they succeed;+//! failures stay journaled with their reason.+//!+//! The journal never holds transcript content or secrets — only locators,+//! ordered semantic digests, and delivery evidence. Recovery never regenerates+//! the original repository checkpoint, so later repository state is not+//! attributed to the interrupted run. Transcripts are re-exported from the native store+//! at recover time instead of being copied into the data dir: the store is+//! the same durable local source the online path reads seconds after the+//! child exits, and re-export reuses the exact adapter path rather than+//! maintaining a second serialized copy of private conversation content.++use std::collections::HashMap;+use std::fs::{self, File, OpenOptions};+use std::path::{Path, PathBuf};+use std::str::FromStr as _;+use std::time::{Duration, Instant};++use ai_memory_core::{ManagedRunId, Sanitizer};+use ai_memory_workstream::TranscriptCapability;+use anyhow::{Context as _, Result, anyhow, bail};+use fs2::FileExt as _;+use serde::{Deserialize, Serialize};++use crate::cli::RecoverArgs;+use crate::commands::hook_spool::{self, DrainLockWait, MaxAttempts, RestoreOutcome};+use crate::config::Config;+use crate::http_client::{ServerEndpoint, ServerProbe, probe_server};++/// Journal file name inside the data dir.+const JOURNAL_FILE: &str = "recovery-journal.json";+/// Exclusive-mutation lock file name inside the data dir.+const JOURNAL_LOCK_FILE: &str = "recovery-journal.lock";+/// Journal schema version.+const JOURNAL_VERSION: u32 = 1;+/// Most entries kept. Appends at capacity are refused so recovery work is+/// never silently evicted.+pub(crate) const MAX_JOURNAL_ENTRIES: usize = 256;+/// Refuse to load or overwrite a journal larger than this, mirroring the+/// client project registry's guard: a swollen file is corruption, not backlog.+const MAX_JOURNAL_BYTES: u64 = 16 * 1024 * 1024;+/// Reasons remembered per entry across recover attempts.+const MAX_ENTRY_FAILURES: usize = 3;+const MAX_FAILURE_REASON_BYTES: usize = 2_048;++/// How the entry's history was cut off from the server.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "kebab-case")]+pub(crate) enum JournalKind {+ /// A degraded launch made while the server was unreachable: no managed+ /// run ever existed, so recovery replays its exact native transcript+ /// through the hook backfill path after draining the spool.+ DegradedRun,+ /// A run whose transcript import failed because the server died after the+ /// child exited: the run still exists server-side, so recovery re-exports+ /// the transcript and finishes it again.+ FinishFailed,+}++impl JournalKind {+ /// Stable label for reports.+ #[must_use]+ pub(crate) fn as_str(self) -> &'static str {+ match self {+ Self::DegradedRun => "degraded-run",+ Self::FinishFailed => "finish-failed",+ }+ }+}++/// One bounded, metadata-only record of history the server missed.+///+/// Constructed by `run` at the moment the loss is detected; consumed by+/// `recover` once the server answers again.+#[derive(Debug, Clone, Serialize, Deserialize)]+pub(crate) struct JournalEntry {+ /// Unique entry id; recovery clears per entry.+ pub(crate) id: String,+ /// When the loss was recorded (RFC 3339, UTC).+ pub(crate) recorded_at: String,+ /// Inclusive lower bound for hook events belonging to this launch.+ pub(crate) interval_started_ms: u64,+ /// Inclusive upper bound for hook events belonging to this launch.+ pub(crate) interval_ended_ms: u64,+ /// How the run was cut off.+ pub(crate) kind: JournalKind,+ /// Managed harness name (`ManagedHarness::as_str`), including the exact+ /// Kiro engine flavor so recovery reads the right native store.+ pub(crate) harness: String,+ /// Native session locator for transcript re-export.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub(crate) native_session_id: Option<String>,+ /// Session-store override the launch resolved (`--session-dir` and kin),+ /// without which the native transcript may be unreachable.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub(crate) session_dir: Option<String>,+ /// Canonical checkout directory the run launched from.+ pub(crate) cwd: String,+ /// Workspace name for degraded transcript replay and reporting.+ pub(crate) workspace: String,+ /// Project name for degraded transcript replay and reporting.+ pub(crate) project: String,+ /// Server URL the run targeted, for a mismatch warning at recover time.+ pub(crate) server_url: String,+ /// Managed run id (`finish-failed` only): the recovery import target.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub(crate) run_id: Option<String>,+ /// Ordered semantic digests for this exact transcript interval. `Some([])`+ /// proves a successful empty export; `None` means export was unavailable.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub(crate) event_digests: Option<Vec<String>>,+ /// Whether this typed harness capability permits spool evidence only and+ /// must never invoke a native transcript exporter.+ #[serde(default)]+ pub(crate) spool_only: bool,+ /// Exact native adapter cursor captured before a resumed launch.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub(crate) source_cursor: Option<String>,+ /// Exact native adapter cursor captured immediately after child exit.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub(crate) final_cursor: Option<String>,+ /// Whether correlated hook events have ever been observed locally.+ #[serde(default)]+ pub(crate) correlated_events_seen: bool,+ /// Whether a prior pass durably delivered all correlated hook evidence.+ #[serde(default)]+ pub(crate) correlated_delivery_confirmed: bool,+ /// Whether a prior pass lost potentially correlated hook evidence.+ #[serde(default)]+ pub(crate) correlated_loss: bool,+ /// Child exit code, replayed with the recovered import.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub(crate) exit_code: Option<i32>,+ /// Reasons from previous recover attempts, newest last, bounded.+ #[serde(default, skip_serializing_if = "Vec::is_empty")]+ pub(crate) failures: Vec<String>,+}++/// Wire shape of the journal file.+#[derive(Debug, Serialize, Deserialize)]+struct JournalFile {+ version: u32,+ entries: Vec<JournalEntry>,+}++/// `<data_dir>/recovery-journal.json`.+#[must_use]+pub(crate) fn journal_path(data_dir: &Path) -> PathBuf {+ data_dir.join(JOURNAL_FILE)+}++/// Read the journal; a missing file is an empty journal, a corrupt or swollen+/// one is an error the caller surfaces (never silently reset: the file is the+/// only record of what the server missed).+pub(crate) fn load_journal(data_dir: &Path) -> Result<Vec<JournalEntry>> {+ let path = journal_path(data_dir);+ refuse_symlink(&path)?;+ let mut file = match open_journal_read(&path) {+ Ok(file) => file,+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),+ Err(error) => {+ return Err(error).with_context(|| format!("reading {}", path.display()));+ }+ };+ if file.metadata()?.len() > MAX_JOURNAL_BYTES {+ bail!(+ "recovery journal {} exceeds the {} byte limit",+ path.display(),+ MAX_JOURNAL_BYTES+ );+ }+ let mut bytes = Vec::new();+ use std::io::Read as _;+ file.read_to_end(&mut bytes)?;+ if bytes.len() as u64 > MAX_JOURNAL_BYTES {+ bail!(+ "recovery journal {} exceeds the {} byte limit",+ path.display(),+ MAX_JOURNAL_BYTES+ );+ }+ let journal: JournalFile = serde_json::from_slice(&bytes).with_context(|| {+ format!(+ "parsing {}; refusing to discard the recorded recovery work",+ path.display()+ )+ })?;+ if journal.version != JOURNAL_VERSION {+ bail!(+ "unsupported recovery journal version {} in {} (expected {JOURNAL_VERSION})",+ journal.version,+ path.display()+ );+ }+ if journal.entries.len() > MAX_JOURNAL_ENTRIES {+ bail!(+ "recovery journal {} exceeds the {} entry limit",+ path.display(),+ MAX_JOURNAL_ENTRIES+ );+ }+ Ok(journal.entries)+}++/// Append one entry under the exclusive lock. Atomic (tmp + rename),+/// owner-only, and symlink-refusing so a swapped file cannot be trampled.+pub(crate) fn append_journal_entry(data_dir: &Path, entry: JournalEntry) -> Result<()> {+ mutate_journal(data_dir, |entries| {+ if entries.len() >= MAX_JOURNAL_ENTRIES {+ bail!(+ "recovery journal is full ({MAX_JOURNAL_ENTRIES} entries); run `ai-memory recover` before another degraded launch"+ );+ }+ entries.push(entry);+ Ok(())+ })+}++/// Rewrite the journal under the exclusive lock with `mutate` applied. The+/// lock spans the whole read-modify-write so two finishing launches cannot+/// lose each other's entry.+fn mutate_journal(+ data_dir: &Path,+ mutate: impl FnOnce(&mut Vec<JournalEntry>) -> Result<()>,+) -> Result<()> {+ validate_journal_entry_limit(data_dir)?;+ let path = journal_path(data_dir);+ let lock = open_private_lock(&data_dir.join(JOURNAL_LOCK_FILE))?;+ lock.lock_exclusive()+ .with_context(|| format!("locking {}", data_dir.join(JOURNAL_LOCK_FILE).display()))?;+ let mut entries = load_journal(data_dir)?;+ mutate(&mut entries)?;+ let journal = JournalFile {+ version: JOURNAL_VERSION,+ entries,+ };+ let mut rendered =+ serde_json::to_vec_pretty(&journal).context("serializing the recovery journal")?;+ rendered.push(b'\n');+ refuse_symlink(&path)?;+ if rendered.len() as u64 > MAX_JOURNAL_BYTES {+ bail!(+ "recovery journal {} would exceed the {} byte limit",+ path.display(),+ MAX_JOURNAL_BYTES+ );+ }+ write_journal_atomic(&path, &rendered)+ .with_context(|| format!("writing {}", path.display()))?;+ make_private(&path)?;+ Ok(())+}++fn write_journal_atomic(path: &Path, bytes: &[u8]) -> Result<()> {+ let parent = path+ .parent()+ .ok_or_else(|| anyhow!("recovery journal path has no parent"))?;+ refuse_symlink(parent)?;+ let mut temp = tempfile::Builder::new()+ .prefix(".recovery-journal.")+ .tempfile_in(parent)?;+ {+ use std::io::Write as _;+ temp.write_all(bytes)?;+ temp.as_file().sync_data()?;+ }+ refuse_symlink(path)?;+ temp.persist(path).map_err(|error| error.error)?;+ if let Ok(directory) = File::open(parent) {+ let _ = directory.sync_all();+ }+ Ok(())+}++fn validate_journal_entry_limit(data_dir: &Path) -> Result<()> {+ let path = journal_path(data_dir);+ refuse_symlink(&path)?;+ match fs::metadata(&path) {+ Ok(metadata) if metadata.len() > MAX_JOURNAL_BYTES => bail!(+ "recovery journal {} exceeds the {} byte limit",+ path.display(),+ MAX_JOURNAL_BYTES+ ),+ Ok(_) => Ok(()),+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),+ Err(error) => Err(error).with_context(|| format!("inspecting {}", path.display())),+ }+}++fn open_journal_read(path: &Path) -> std::io::Result<File> {+ let mut options = OpenOptions::new();+ options.read(true);+ #[cfg(unix)]+ {+ use std::os::unix::fs::OpenOptionsExt as _;+ options.custom_flags(libc::O_NOFOLLOW);+ }+ options.open(path)+}++fn open_private_lock(path: &Path) -> Result<File> {+ if let Some(parent) = path.parent() {+ refuse_symlink(parent)?;+ crate::commands::path_util::create_private_dir(parent)+ .with_context(|| format!("creating {}", parent.display()))?;+ refuse_symlink(parent)?;+ }+ refuse_symlink(path)?;+ let mut options = OpenOptions::new();+ options.create(true).read(true).write(true);+ #[cfg(unix)]+ {+ use std::os::unix::fs::OpenOptionsExt as _;+ options.mode(0o600).custom_flags(libc::O_NOFOLLOW);+ }+ let file = options+ .open(path)+ .with_context(|| format!("opening {}", path.display()))?;+ refuse_symlink(path)?;+ make_private(path)?;+ Ok(file)+}++fn refuse_symlink(path: &Path) -> Result<()> {+ let mut current = PathBuf::new();+ for component in path.components() {+ current.push(component.as_os_str());+ match fs::symlink_metadata(¤t) {+ Ok(metadata) if metadata.file_type().is_symlink() => {+ bail!("refusing recovery journal symlink {}", current.display());+ }+ Ok(_) => {}+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => break,+ Err(error) => {+ return Err(error).with_context(|| format!("inspecting {}", current.display()));+ }+ }+ }+ Ok(())+}++#[cfg(unix)]+fn make_private(path: &Path) -> Result<()> {+ use std::os::unix::fs::PermissionsExt as _;+ fs::set_permissions(path, fs::Permissions::from_mode(0o600))+ .with_context(|| format!("restricting permissions on {}", path.display()))+}++#[cfg(windows)]+fn make_private(path: &Path) -> Result<()> {+ let identity = std::process::Command::new("whoami")+ .output()+ .context("resolving the Windows recovery-journal owner")?;+ if !identity.status.success() {+ bail!("could not resolve the Windows recovery-journal owner");+ }+ let owner =+ String::from_utf8(identity.stdout).context("reading the Windows recovery-journal owner")?;+ let grant = format!("{}:(F)", owner.trim());+ let restricted = std::process::Command::new("icacls")+ .arg(path)+ .args(["/inheritance:r", "/grant:r"])+ .arg(grant)+ .output()+ .with_context(|| format!("restricting permissions on {}", path.display()))?;+ if !restricted.status.success() {+ bail!("could not restrict permissions on {}", path.display());+ }+ Ok(())+}++#[cfg(not(any(unix, windows)))]+fn make_private(_path: &Path) -> Result<()> {+ Ok(())+}++/// Count journal entries for status lines; an unreadable journal reports zero+/// with its error carried back, never blocking status output.+pub(crate) fn journal_summary(data_dir: &Path) -> (usize, usize, usize, Option<String>) {+ match load_journal(data_dir) {+ Ok(entries) => {+ let degraded = entries+ .iter()+ .filter(|entry| entry.kind == JournalKind::DegradedRun)+ .count();+ let finish_failed = entries.len() - degraded;+ (entries.len(), degraded, finish_failed, None)+ }+ Err(error) => (0, 0, 0, Some(format!("{error:#}"))),+ }+}++/// Budgets for one `recover` invocation; injectable so tests can prove the+/// bounds bite.+#[derive(Debug, Clone, Copy)]+struct RecoverBudgets {+ /// Total spool-drain budget. Generous but bounded, matching the hook+ /// background-drain default: recovery must not hang forever on a wedged+ /// server, but a large outage backlog must fully flush.+ drain_total: Duration,+ /// Per-event POST timeout while draining.+ drain_event_timeout: Duration,+ /// Per-journal-entry transcript HTTP budget.+ entry_timeout: Duration,+ /// Total journal HTTP budget.+ journal_total: Duration,+ /// How long recovery waits on another process's drain lock.+ drain_lock_wait: Duration,+}++impl Default for RecoverBudgets {+ fn default() -> Self {+ Self {+ drain_total: Duration::from_secs(5 * 60),+ drain_event_timeout: Duration::from_secs(30),+ entry_timeout: Duration::from_secs(60),+ journal_total: Duration::from_secs(10 * 60),+ drain_lock_wait: Duration::from_secs(30),+ }+ }+}++/// What recovery did for one journal entry.+#[derive(Debug)]+enum EntryRecovery {+ /// Re-exported and imported; the count is newly imported content events.+ Imported(usize),+ /// The harness has no transcript exporter and its complete spool was delivered.+ RecoveredViaSpool,+ /// Recovery failed; the entry stays journaled with the reason.+ Failed(String),+}++/// Spool-drain counts for the report.+#[derive(Debug, Default, Clone, Copy, Serialize)]+struct SpoolOutcome {+ sent: usize,+ remaining: usize,+ dropped: usize,+}++/// One recovered (or failed) session, for the report.+#[derive(Debug, Serialize)]+struct SessionOutcome {+ kind: &'static str,+ harness: String,+ native_session_id: Option<String>,+ workspace: String,+ project: String,+ recorded_at: String,+ outcome: &'static str,+ imported_events: usize,+ #[serde(skip_serializing_if = "Option::is_none")]+ detail: Option<String>,+}++/// The full `ai-memory recover` report.+#[derive(Debug, Serialize)]+struct RecoverReport {+ server: String,+ reachable: bool,+ spool: SpoolOutcome,+ spool_lock_busy: bool,+ spool_incomplete: bool,+ journal_pending: usize,+ journal_degraded: usize,+ journal_finish_failed: usize,+ sessions: Vec<SessionOutcome>,+ journal_remaining: usize,+ #[serde(skip_serializing_if = "Option::is_none")]+ journal_error: Option<String>,+}++impl RecoverReport {+ fn failures(&self) -> usize {+ self.sessions+ .iter()+ .filter(|session| session.outcome == "failed")+ .count()+ }++ fn imported_total(&self) -> usize {+ self.sessions.iter().map(|s| s.imported_events).sum()+ }+}++/// Run the `recover` subcommand and return its process exit code.+///+/// # Errors+/// Returns an error only for local infrastructure failures.+pub async fn run(config: &Config, args: RecoverArgs) -> Result<i32> {+ let report = recover(config, &RecoverBudgets::default()).await?;+ if args.json {+ println!("{}", serde_json::to_string_pretty(&report)?);+ } else {+ print_report(&report);+ }+ Ok(i32::from(+ report.reachable+ && (report.spool_incomplete || report.failures() > 0 || report.journal_error.is_some()),+ ))+}++async fn recover(config: &Config, budgets: &RecoverBudgets) -> Result<RecoverReport> {+ let endpoint = ServerEndpoint::from_config_resolving_auth(config).await;+ let spool_dir = hook_spool::spool_dir(&config.data_dir);+ let (mut journal, journal_error) = match load_journal(&config.data_dir) {+ Ok(entries) => (entries, None),+ // A broken journal still leaves the spool drain: the two recover+ // independent halves of the outage.+ Err(error) => (Vec::new(), Some(format!("{error:#}"))),+ };+ let journal_degraded = journal+ .iter()+ .filter(|entry| entry.kind == JournalKind::DegradedRun)+ .count();+ let journal_finish_failed = journal.len() - journal_degraded;++ if let ServerProbe::Unreachable(_) = probe_server(&endpoint).await {+ return Ok(RecoverReport {+ server: endpoint.url.clone(),+ reachable: false,+ spool: SpoolOutcome {+ remaining: hook_spool::spool_len(&spool_dir),+ ..SpoolOutcome::default()+ },+ spool_lock_busy: false,+ spool_incomplete: false,+ journal_pending: journal.len(),+ journal_degraded,+ journal_finish_failed,+ sessions: Vec::new(),+ journal_remaining: journal.len(),+ journal_error,+ });+ }++ let sanitizer = Sanitizer::new(&config.sanitize).map_err(|error| anyhow!(error.to_string()))?;+ let journal_pending = journal.len();+ let mut quarantined = HashMap::new();+ let drain_lock = match hook_spool::acquire_drain_lock(+ &spool_dir,+ DrainLockWait::Bounded(budgets.drain_lock_wait),+ ) {+ Ok(Some(lock)) => lock,+ Ok(None) => {+ return Ok(lock_busy_report(+ &endpoint,+ &spool_dir,+ journal_pending,+ journal_degraded,+ journal_finish_failed,+ journal_error,+ ));+ }+ Err(error) => {+ return Err(error).context("acquiring the recovery spool lock");+ }+ };+ let restore_quarantine = |quarantined: HashMap<String, hook_spool::QuarantinedSpool>| {+ let mut first_error = None;+ for (_, quarantine) in quarantined {+ if let Err(error) = quarantine.restore(&spool_dir)+ && first_error.is_none()+ {+ first_error = Some(error);+ }+ }+ first_error.map_or(Ok(()), Err)+ };+ for entry in &journal {+ if entry.kind != JournalKind::DegradedRun && !entry.spool_only {+ continue;+ }+ let Some(native_session_id) = entry.native_session_id.as_deref() else {+ continue;+ };+ match hook_spool::quarantine_session_entries(+ config,+ &spool_dir,+ &entry.id,+ entry.interval_started_ms,+ entry.interval_ended_ms,+ &entry.server_url,+ super::run::harness_from_journal_name(&entry.harness)+ .map(|harness| harness.agent_kind().as_str())+ .unwrap_or(&entry.harness),+ &entry.workspace,+ &entry.project,+ Path::new(&entry.cwd),+ native_session_id,+ ) {+ Ok(quarantine) => {+ quarantined.insert(entry.id.clone(), quarantine);+ }+ Err(error) => {+ restore_quarantine(quarantined)+ .context("restoring previously quarantined hook events")?;+ return Err(error).context("quarantining correlated hook events");+ }+ }+ }+ let mut evidence_started = Vec::new();+ for entry in &mut journal {+ if entry.kind != JournalKind::DegradedRun && !entry.spool_only {+ continue;+ }+ let correlated_count = quarantined.get(&entry.id).map_or(0, |value| value.len());+ if correlated_count > 0 {+ entry.correlated_events_seen = true;+ entry.correlated_loss = true;+ evidence_started.push(entry.id.clone());+ }+ }+ if !evidence_started.is_empty() {+ update_journal_evidence(&config.data_dir, &journal)?;+ }+ for entry in &mut journal {+ if !entry_supports_transcript_export(entry)+ && let Some(quarantine) = quarantined.get(&entry.id)+ && quarantine.len() > 0+ {+ let outcome = quarantine+ .drain_strict(+ &config.data_dir,+ budgets.drain_total,+ budgets.drain_event_timeout,+ MaxAttempts::new(config.hook_spool.max_attempts),+ )+ .await;+ if outcome.durable > 0 {+ entry.correlated_events_seen = true;+ }+ if outcome.dropped > 0 {+ entry.correlated_loss = true;+ }+ if outcome.remaining == 0+ && outcome.dropped == 0+ && outcome.durable > 0+ && quarantine.complete_spool_only()?+ {+ entry.correlated_delivery_confirmed = true;+ entry.correlated_loss = false;+ }+ }+ }+ update_journal_evidence(&config.data_dir, &journal)?;+ let drained = hook_spool::drain_until_quiescent_locked_with_live_token(+ &spool_dir,+ &config.data_dir,+ budgets.drain_total,+ budgets.drain_event_timeout,+ config.auth.bearer_token.as_deref(),+ MaxAttempts::new(config.hook_spool.max_attempts),+ )+ .await;+ let spool = SpoolOutcome {+ sent: drained.sent,+ remaining: drained.remaining,+ dropped: drained.dropped,+ };+ let spool_incomplete = spool_is_incomplete(&spool);+ if spool.dropped > 0 {+ eprintln!(+ "ai-memory: recovery: {count} hook event(s) were dropped; recovery remains incomplete unless an exact correlated transcript replaces them",+ count = spool.dropped+ );+ }++ let mut sessions = Vec::with_capacity(journal.len());+ let mut cleared: Vec<String> = Vec::with_capacity(journal.len());+ let mut annotated: Vec<(String, String)> = Vec::new();+ let journal_started = Instant::now();+ for entry in journal {+ let mut recovery = if spool_incomplete {+ EntryRecovery::Failed(format!(+ "hook spool recovery was incomplete ({} queued, {} dropped); transcript recovery was deferred",+ spool.remaining, spool.dropped+ ))+ } else if let Some(timeout) = entry_budget(journal_started, budgets) {+ if timeout.is_zero() {+ failed_reason(+ &sanitizer,+ "journal entry recovery timed out; rerun to continue",+ )+ } else {+ match tokio::time::timeout(+ timeout,+ recover_entry(config, &endpoint, &sanitizer, &entry),+ )+ .await+ {+ Ok(recovery) => recovery,+ Err(_) => failed_reason(+ &sanitizer,+ "journal entry recovery timed out; rerun to continue",+ ),+ }+ }+ } else {+ failed_reason(+ &sanitizer,+ "total recovery budget exhausted; rerun to continue",+ )+ };+ if let Some(quarantine) = quarantined.remove(&entry.id) {+ let restored = match &recovery {+ EntryRecovery::Imported(_) => {+ quarantine.complete().map(|()| RestoreOutcome::Restored)+ }+ EntryRecovery::RecoveredViaSpool => Ok(RestoreOutcome::Restored),+ EntryRecovery::Failed(_) => quarantine.restore(&spool_dir),+ }+ .context("restoring quarantined hook events")?;+ if matches!(restored, RestoreOutcome::AlreadyDelivered) {+ // The completed marker is durable proof this interval's+ // correlated evidence reached the server on a prior pass;+ // the failure only deferred work there is none left of, so+ // the entry clears instead of staying journaled behind it.+ recovery = EntryRecovery::RecoveredViaSpool;+ }+ }+ let outcome = match &recovery {+ EntryRecovery::Imported(_) => "imported",+ EntryRecovery::RecoveredViaSpool => "recovered-via-spool",+ EntryRecovery::Failed(_) => "failed",+ };+ let imported_events = match &recovery {+ EntryRecovery::Imported(events) => *events,+ _ => 0,+ };+ let detail = match &recovery {+ EntryRecovery::Imported(_) | EntryRecovery::RecoveredViaSpool => None,+ EntryRecovery::Failed(note) => Some(note.clone()),+ };+ if let EntryRecovery::Failed(reason) = &recovery {+ annotated.push((entry.id.clone(), reason.clone()));+ } else {+ cleared.push(entry.id.clone());+ }+ sessions.push(SessionOutcome {+ kind: entry.kind.as_str(),+ harness: entry.harness.clone(),+ native_session_id: entry.native_session_id.clone(),+ workspace: entry.workspace.clone(),+ project: entry.project.clone(),+ recorded_at: entry.recorded_at.clone(),+ outcome,+ imported_events,+ detail,+ });+ }++ restore_quarantine(quarantined)?;+ drop(drain_lock);++ // (d) Clear succeeded entries; failures stay with their reason.+ let journal_remaining = finish_journal(&config.data_dir, &sanitizer, &cleared, &annotated)?;++ Ok(RecoverReport {+ server: endpoint.url.clone(),+ reachable: true,+ spool,+ spool_lock_busy: false,+ spool_incomplete,+ journal_pending,+ journal_degraded,+ journal_finish_failed,+ sessions,+ journal_remaining,+ journal_error,+ })+}++fn entry_budget(started: Instant, budgets: &RecoverBudgets) -> Option<Duration> {+ if budgets.entry_timeout.is_zero() {+ return Some(Duration::ZERO);+ }+ let remaining = budgets.journal_total.checked_sub(started.elapsed())?;+ (!remaining.is_zero()).then(|| budgets.entry_timeout.min(remaining))+}++fn spool_is_incomplete(spool: &SpoolOutcome) -> bool {+ spool.remaining > 0 || spool.dropped > 0+}++fn lock_busy_report(+ endpoint: &ServerEndpoint,+ spool_dir: &Path,+ journal_pending: usize,+ journal_degraded: usize,+ journal_finish_failed: usize,+ journal_error: Option<String>,+) -> RecoverReport {+ RecoverReport {+ server: endpoint.url.clone(),+ reachable: true,+ spool: SpoolOutcome {+ remaining: hook_spool::spool_len(spool_dir),+ ..SpoolOutcome::default()+ },+ spool_lock_busy: true,+ spool_incomplete: true,+ journal_pending,+ journal_degraded,+ journal_finish_failed,+ sessions: Vec::new(),+ journal_remaining: journal_pending,+ journal_error,+ }+}++async fn recover_entry(+ config: &Config,+ endpoint: &ServerEndpoint,+ sanitizer: &Sanitizer,+ entry: &JournalEntry,+) -> EntryRecovery {+ let recorded = entry.server_url.trim_end_matches('/');+ let configured_identity = endpoint.identity();+ let configured = configured_identity.trim_end_matches('/');+ if recorded.is_empty() || recorded != configured {+ return failed_reason(+ sanitizer,+ &format!("refusing to replay an entry recorded for {recorded:?} against {configured}"),+ );+ }+ if entry.spool_only {+ return recover_spool_only(sanitizer, entry);+ }+ match entry.kind {+ JournalKind::FinishFailed => {+ recover_finish_failed(config, endpoint, sanitizer, entry).await+ }+ JournalKind::DegradedRun => recover_degraded_run(config, endpoint, sanitizer, entry).await,+ }+}++/// Re-export the transcript from the native store and finish the run again.+/// Reuses `run`'s import machinery verbatim: the server dedups by stable event+/// id, so a full re-export cannot duplicate history a partial import landed.+fn recovery_session_dir(entry: &JournalEntry) -> Option<PathBuf> {+ entry.session_dir.as_deref().map(|path| {+ let path = Path::new(path);+ if path.is_absolute() {+ path.to_path_buf()+ } else {+ Path::new(&entry.cwd).join(path)+ }+ })+}++async fn recover_finish_failed(+ config: &Config,+ endpoint: &ServerEndpoint,+ sanitizer: &Sanitizer,+ entry: &JournalEntry,+) -> EntryRecovery {+ let Some(run_id) = entry.run_id.as_deref() else {+ return failed_reason(sanitizer, "finish-failed entry records no run id");+ };+ if ManagedRunId::from_str(run_id).is_err() {+ return failed_reason(+ sanitizer,+ &format!("journal records an invalid run id {run_id}"),+ );+ }+ let Some(harness) = super::run::harness_from_journal_name(&entry.harness) else {+ return failed_reason(+ sanitizer,+ &format!("journal names unsupported harness '{}'", entry.harness),+ );+ };+ let Some(home) = super::run::native_home(config) else {+ return failed_reason(+ sanitizer,+ "the native harness home directory could not be located",+ );+ };+ let Some(expected_digests) = entry.event_digests.as_ref() else {+ return failed_reason(+ sanitizer,+ "finish-failed entry has no exact ordered transcript identity; refusing full transcript replay",+ );+ };+ let Some(final_cursor) = entry.final_cursor.as_deref() else {+ return failed_reason(+ sanitizer,+ "finish-failed entry has no exact post-run transcript cursor; refusing replay",+ );+ };+ let transcript = match entry.native_session_id.as_deref() {+ Some(native_session_id) => {+ match ai_memory_workstream::export_transcript_range(+ harness,+ &home,+ Path::new(&entry.cwd),+ recovery_session_dir(entry).as_deref(),+ native_session_id,+ entry.source_cursor.as_deref(),+ final_cursor,+ )+ .await+ {+ Ok(transcript) => transcript,+ Err(error) => {+ return failed_reason(+ sanitizer,+ &format!("native transcript re-export failed: {error:#}"),+ );+ }+ }+ }+ None => {+ return failed_reason(+ sanitizer,+ "finish-failed entry records no native transcript locator",+ );+ }+ };+ let run_path = format!("/workstream/runs/{run_id}/recover");+ let actual = ai_memory_workstream::transcript_interval_digests(&transcript.events);+ if &actual != expected_digests {+ return failed_reason(+ sanitizer,+ "native transcript changed since the failed finish; refusing non-identical replay",+ );+ }+ let transcript = ai_memory_workstream::ExportedTranscript {+ source_cursor: Some(final_cursor.to_owned()),+ ..transcript+ };+ match super::run::import_batches(+ endpoint,+ &run_path,+ transcript,+ ai_memory_core::WorkstreamCheckpoint::default(),+ entry.exit_code,+ sanitizer,+ )+ .await+ {+ Ok(imported) => EntryRecovery::Imported(imported),+ Err(error) => failed_reason(sanitizer, &format!("managed finish failed: {error:#}")),+ }+}++fn failed_reason(sanitizer: &Sanitizer, reason: &str) -> EntryRecovery {+ EntryRecovery::Failed(ai_memory_core::truncate_utf8_bytes(+ &sanitizer.scrub(reason),+ MAX_FAILURE_REASON_BYTES,+ ))+}++/// A degraded launch had no server run, so recover its exact native transcript+/// through the ordinary backfill ingress. That path emits stable keyed+/// session-start/content/session-end items through `/hook/batch`, making a+/// repeated recovery idempotent while still crossing the normal sanitizer and+/// consolidation boundary.+fn entry_supports_transcript_export(entry: &JournalEntry) -> bool {+ !entry.spool_only+ && super::run::harness_from_journal_name(&entry.harness)+ .is_some_and(|harness| harness.transcript_capability() == TranscriptCapability::Export)+}++fn recover_spool_only(sanitizer: &Sanitizer, entry: &JournalEntry) -> EntryRecovery {+ if entry.correlated_delivery_confirmed && !entry.correlated_loss {+ EntryRecovery::RecoveredViaSpool+ } else {+ failed_reason(+ sanitizer,+ "no durably delivered correlated hook evidence exists; retain this entry and repair the native session manually",+ )+ }+}++async fn recover_degraded_run(+ config: &Config,+ endpoint: &ServerEndpoint,+ sanitizer: &Sanitizer,+ entry: &JournalEntry,+) -> EntryRecovery {+ let Some(harness) = super::run::harness_from_journal_name(&entry.harness) else {+ return failed_reason(+ sanitizer,+ &format!("journal names unsupported harness '{}'", entry.harness),+ );+ };+ if harness.transcript_capability() == TranscriptCapability::SpoolOnly {+ return recover_spool_only(sanitizer, entry);+ }+ let Some(native_session_id) = entry.native_session_id.as_deref() else {+ return failed_reason(+ sanitizer,+ "no native session was recorded for this degraded run",+ );+ };+ let Some(home) = super::run::native_home(config) else {+ return failed_reason(+ sanitizer,+ "the native harness home directory could not be located",+ );+ };+ let Some(expected_digests) = entry.event_digests.as_deref() else {+ return failed_reason(+ sanitizer,+ "degraded-run entry has no exact ordered transcript identity; repair the native session manually",+ );+ };+ let Some(final_cursor) = entry.final_cursor.as_deref() else {+ return failed_reason(+ sanitizer,+ "degraded-run entry has no exact post-run transcript cursor; repair the native session manually",+ );+ };+ match super::backfill::import_exact_session_range(+ endpoint,+ &entry.workspace,+ &entry.project,+ &home,+ Path::new(&entry.cwd),+ harness,+ native_session_id,+ &entry.id,+ recovery_session_dir(entry).as_deref(),+ entry.source_cursor.as_deref(),+ final_cursor,+ expected_digests,+ )+ .await+ {+ Ok(imported) => EntryRecovery::Imported(imported),+ Err(error) => failed_reason(+ sanitizer,+ &format!("native transcript replay failed: {error:#}"),+ ),+ }+}++/// Apply clears and failure annotations in one locked rewrite; returns the+/// remaining entry count.+fn update_journal_evidence(data_dir: &Path, current: &[JournalEntry]) -> Result<()> {+ mutate_journal(data_dir, |entries| {+ for entry in entries {+ if let Some(updated) = current.iter().find(|candidate| candidate.id == entry.id) {+ entry.correlated_events_seen = updated.correlated_events_seen;+ entry.correlated_delivery_confirmed = updated.correlated_delivery_confirmed;+ entry.correlated_loss = updated.correlated_loss;+ }+ }+ Ok(())+ })+}++fn finish_journal(+ data_dir: &Path,+ sanitizer: &Sanitizer,+ cleared: &[String],+ annotated: &[(String, String)],+) -> Result<usize> {+ if cleared.is_empty() && annotated.is_empty() {+ return Ok(load_journal(data_dir).map_or(0, |entries| entries.len()));+ }+ let mut remaining = 0_usize;+ mutate_journal(data_dir, |entries| {+ entries.retain(|entry| !cleared.contains(&entry.id));+ for (id, reason) in annotated {+ if let Some(entry) = entries.iter_mut().find(|entry| &entry.id == id) {+ let reason = ai_memory_core::truncate_utf8_bytes(+ &sanitizer.scrub(reason),+ MAX_FAILURE_REASON_BYTES,+ );+ if !entry.failures.contains(&reason) {+ entry.failures.push(reason);+ }+ let overflow = entry.failures.len().saturating_sub(MAX_ENTRY_FAILURES);+ if overflow > 0 {+ entry.failures.drain(..overflow);+ }+ }+ }+ remaining = entries.len();+ Ok(())+ })?;+ Ok(remaining)+}++fn print_report(report: &RecoverReport) {+ if !report.reachable {+ println!(+ "ai-memory: server at {} is unreachable — nothing to recover yet.",+ report.server+ );+ if let Some(error) = &report.journal_error {+ println!(" journal: unreadable ({error})");+ } else {+ println!(+ " journal: {} entr{} await{} recovery ({} degraded launch(es), {} failed \+ transcript import(s))",+ report.journal_pending,+ if report.journal_pending == 1 {+ "y"+ } else {+ "ies"+ },+ if report.journal_pending == 1 { "s" } else { "" },+ report.journal_degraded,+ report.journal_finish_failed,+ );+ }+ println!(+ " spool: {} hook event(s) queued locally",+ report.spool.remaining+ );+ println!(+ "Start the server (or fix AI_MEMORY_SERVER_URL), then run `ai-memory recover` again."+ );+ return;+ }+ println!("ai-memory: recovering against {}…", report.server);+ if report.spool_lock_busy {+ println!(" spool: another drainer held the lock; transcript recovery was deferred");+ } else {+ println!(+ " spool: {} hook event(s) delivered, {} remain queued, {} dropped",+ report.spool.sent, report.spool.remaining, report.spool.dropped+ );+ }+ if report.spool_incomplete {+ println!(+ " recovery is incomplete because hook events remain queued or were dropped; exact correlated transcripts may still replace quarantined events"+ );+ }+ for session in &report.sessions {+ let target = session+ .native_session_id+ .as_deref()+ .unwrap_or("(no session id)");+ let mut line = format!(+ " [{}] {} {} in {}/{} — {}",+ session.kind,+ session.harness,+ target,+ session.workspace,+ session.project,+ session.outcome,+ );+ if session.imported_events > 0 {+ line.push_str(&format!(" ({} events)", session.imported_events));+ }+ println!("{line}");+ if let Some(detail) = &session.detail {+ println!(" {detail}");+ }+ }+ println!(+ "recovered {} event(s) across {} journal entr{}; {} failed; {} remain",+ report.imported_total(),+ report.sessions.len(),+ if report.sessions.len() == 1 {+ "y"+ } else {+ "ies"+ },+ report.failures(),+ report.journal_remaining,+ );+ if let Some(error) = &report.journal_error {+ println!("recovery journal error: {error}");+ }+ if report.failures() > 0 {+ println!(+ "failed entries stay journaled with their reason; fix the cause and re-run \+ `ai-memory recover` (re-running is safe — imports dedup)"+ );+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn test_entry(id: &str, kind: JournalKind) -> JournalEntry {+ JournalEntry {+ id: id.to_string(),+ recorded_at: "2026-10-07T12:00:00Z".to_string(),+ interval_started_ms: 1_000,+ interval_ended_ms: 2_000,+ kind,+ harness: "claude".to_string(),+ native_session_id: Some("sess-1".to_string()),+ session_dir: None,+ cwd: "/tmp/repo".to_string(),+ workspace: "ws".to_string(),+ project: "proj".to_string(),+ server_url: "http://127.0.0.1:49374".to_string(),+ run_id: (kind == JournalKind::FinishFailed)+ .then(|| "12345678-1234-4234-9234-123456789abe".to_string()),+ event_digests: Some(Vec::new()),+ spool_only: false,+ source_cursor: None,+ final_cursor: None,+ correlated_events_seen: false,+ correlated_delivery_confirmed: false,+ correlated_loss: false,+ exit_code: Some(0),+ failures: Vec::new(),+ }+ }++ #[test]+ fn append_then_load_round_trips() {+ let temp = tempfile::tempdir().unwrap();+ let entry = test_entry("run-1", JournalKind::FinishFailed);+ append_journal_entry(temp.path(), entry.clone()).unwrap();+ let loaded = load_journal(temp.path()).unwrap();+ assert_eq!(loaded.len(), 1);+ assert_eq!(loaded[0].id, "run-1");+ assert_eq!(loaded[0].kind, JournalKind::FinishFailed);+ assert_eq!(loaded[0].harness, entry.harness);+ assert_eq!(loaded[0].native_session_id.as_deref(), Some("sess-1"));+ }++ #[test]+ fn missing_journal_is_empty_and_append_creates_it_private() {+ let temp = tempfile::tempdir().unwrap();+ assert!(load_journal(temp.path()).unwrap().is_empty());+ append_journal_entry(temp.path(), test_entry("a", JournalKind::DegradedRun)).unwrap();+ assert_eq!(load_journal(temp.path()).unwrap().len(), 1);+ #[cfg(unix)]+ {+ use std::os::unix::fs::PermissionsExt as _;+ let mode = fs::metadata(journal_path(temp.path()))+ .unwrap()+ .permissions()+ .mode();+ assert_eq!(mode & 0o777, 0o600, "the journal is owner-only");+ }+ }++ #[test]+ fn full_journal_refuses_append_without_evicting() {+ let temp = tempfile::tempdir().unwrap();+ for index in 0..MAX_JOURNAL_ENTRIES {+ append_journal_entry(+ temp.path(),+ test_entry(&format!("entry-{index}"), JournalKind::DegradedRun),+ )+ .unwrap();+ }+ let error = append_journal_entry(+ temp.path(),+ test_entry("must-not-fit", JournalKind::DegradedRun),+ )+ .unwrap_err();+ assert!(format!("{error:#}").contains("ai-memory recover"));+ let loaded = load_journal(temp.path()).unwrap();+ assert_eq!(loaded.len(), MAX_JOURNAL_ENTRIES);+ assert_eq!(loaded[0].id, "entry-0");+ assert_eq!(+ loaded.last().unwrap().id,+ format!("entry-{}", MAX_JOURNAL_ENTRIES - 1)+ );+ }++ #[test]+ fn appends_are_atomic_and_leave_no_temporary_files() {+ // The writer only ever renames a complete tmp file into place, so a+ // crash between appends cannot leave a half-written journal: the+ // previous complete file is still there and parseable.+ let temp = tempfile::tempdir().unwrap();+ append_journal_entry(temp.path(), test_entry("first", JournalKind::DegradedRun)).unwrap();+ let before = fs::read(journal_path(temp.path())).unwrap();+ append_journal_entry(temp.path(), test_entry("second", JournalKind::DegradedRun)).unwrap();+ let after = fs::read(journal_path(temp.path())).unwrap();+ assert!(before.len() < after.len());+ assert_eq!(load_journal(temp.path()).unwrap().len(), 2);+ assert!(+ !fs::read_dir(temp.path()).unwrap().any(|entry| entry+ .unwrap()+ .file_name()+ .to_string_lossy()+ .contains(".tmp")),+ "no temporary file is left behind"+ );+ }++ #[test]+ fn corrupt_oversized_or_wrong_version_journal_is_refused_not_reset() {+ let temp = tempfile::tempdir().unwrap();+ let path = journal_path(temp.path());+ fs::write(&path, b"{ not json").unwrap();+ let error = load_journal(temp.path()).unwrap_err();+ assert!(+ format!("{error:#}").contains("refusing to discard"),+ "{error:#}"+ );+ let error = append_journal_entry(temp.path(), test_entry("x", JournalKind::DegradedRun))+ .unwrap_err();+ assert!(+ format!("{error:#}").contains("refusing to discard"),+ "{error:#}"+ );+ assert_eq!(fs::read(&path).unwrap(), b"{ not json");++ let entries = (0..=MAX_JOURNAL_ENTRIES)+ .map(|index| test_entry(&index.to_string(), JournalKind::DegradedRun))+ .collect();+ fs::write(+ &path,+ serde_json::to_vec(&JournalFile {+ version: JOURNAL_VERSION,+ entries,+ })+ .unwrap(),+ )+ .unwrap();+ let error = load_journal(temp.path()).unwrap_err();+ assert!(format!("{error:#}").contains("entry limit"), "{error:#}");+ }++ #[cfg(unix)]+ #[test]+ fn symlinked_lock_is_refused_without_touching_its_target() {+ let temp = tempfile::tempdir().unwrap();+ let outside = temp.path().join("outside-lock");+ fs::write(&outside, b"canary").unwrap();+ std::os::unix::fs::symlink(&outside, temp.path().join(JOURNAL_LOCK_FILE)).unwrap();+ let error =+ append_journal_entry(temp.path(), test_entry("outside", JournalKind::DegradedRun))+ .unwrap_err();+ assert!(format!("{error:#}").contains("symlink"), "{error:#}");+ assert_eq!(fs::read(outside).unwrap(), b"canary");+ }++ #[cfg(unix)]+ #[test]+ fn symlinked_journal_and_ancestor_are_refused() {+ let temp = tempfile::tempdir().unwrap();+ let real = temp.path().join("real.json");+ fs::write(&real, b"{}").unwrap();+ std::os::unix::fs::symlink(&real, journal_path(temp.path())).unwrap();+ let error = load_journal(temp.path()).unwrap_err();+ assert!(format!("{error:#}").contains("symlink"), "{error:#}");++ let outside = tempfile::tempdir().unwrap();+ let linked_data = temp.path().join("linked-data");+ std::os::unix::fs::symlink(outside.path(), &linked_data).unwrap();+ let error = append_journal_entry(+ &linked_data,+ test_entry("outside", JournalKind::DegradedRun),+ )+ .unwrap_err();+ assert!(format!("{error:#}").contains("symlink"), "{error:#}");+ assert!(!outside.path().join(JOURNAL_FILE).exists());+ }++ #[cfg(windows)]+ #[test]+ fn journal_acl_is_restricted_to_the_current_owner() {+ let temp = tempfile::tempdir().unwrap();+ append_journal_entry(temp.path(), test_entry("acl", JournalKind::DegradedRun)).unwrap();+ let output = std::process::Command::new("icacls")+ .arg(journal_path(temp.path()))+ .output()+ .unwrap();+ assert!(output.status.success());+ let acl = String::from_utf8_lossy(&output.stdout).to_ascii_lowercase();+ assert!(!acl.contains("everyone"), "{acl}");+ assert!(!acl.contains("authenticated users"), "{acl}");+ }++ #[test]+ fn clearing_entries_and_annotating_failures_rewrites_the_journal() {+ let temp = tempfile::tempdir().unwrap();+ for id in ["keep", "clear", "fail"] {+ append_journal_entry(temp.path(), test_entry(id, JournalKind::DegradedRun)).unwrap();+ }+ finish_journal(+ temp.path(),+ &Sanitizer::builtin(),+ &["clear".to_string()],+ &[("fail".to_string(), "server said no".to_string())],+ )+ .unwrap();+ let loaded = load_journal(temp.path()).unwrap();+ assert_eq!(loaded.len(), 2);+ assert!(loaded.iter().all(|entry| entry.id != "clear"));+ let failed = loaded.iter().find(|entry| entry.id == "fail").unwrap();+ assert_eq!(failed.failures, vec!["server said no".to_string()]);+ // The same failure reason is not recorded twice.+ finish_journal(+ temp.path(),+ &Sanitizer::builtin(),+ &[],+ &[("fail".to_string(), "server said no".to_string())],+ )+ .unwrap();+ let loaded = load_journal(temp.path()).unwrap();+ assert_eq!(+ loaded+ .iter()+ .find(|entry| entry.id == "fail")+ .unwrap()+ .failures,+ vec!["server said no".to_string()]+ );+ }++ #[test]+ fn failure_reasons_are_sanitized_and_bounded_per_entry() {+ let temp = tempfile::tempdir().unwrap();+ append_journal_entry(temp.path(), test_entry("e", JournalKind::DegradedRun)).unwrap();+ for index in 0..(MAX_ENTRY_FAILURES + 2) {+ finish_journal(+ temp.path(),+ &Sanitizer::builtin(),+ &[],+ &[(+ "e".to_string(),+ format!(+ "reason {index} token sk-{} {}",+ "a".repeat(32),+ "x".repeat(MAX_FAILURE_REASON_BYTES * 2)+ ),+ )],+ )+ .unwrap();+ }+ let loaded = load_journal(temp.path()).unwrap();+ assert_eq!(loaded[0].failures.len(), MAX_ENTRY_FAILURES);+ assert!(loaded[0].failures[0].starts_with("reason 2 token [REDACTED:api_key]"));+ assert!(+ loaded[0]+ .failures+ .iter()+ .all(|reason| reason.len() <= MAX_FAILURE_REASON_BYTES)+ );+ }++ #[test]+ fn unknown_and_known_empty_event_sets_remain_distinct() {+ let mut unknown = test_entry("unknown", JournalKind::FinishFailed);+ unknown.event_digests = None;+ let unknown_json = serde_json::to_value(&unknown).unwrap();+ assert!(unknown_json.get("event_digests").is_none());++ let empty = test_entry("empty", JournalKind::FinishFailed);+ let empty_json = serde_json::to_value(&empty).unwrap();+ assert_eq!(empty_json["event_digests"], serde_json::json!([]));+ }++ #[test]+ fn recovery_journal_omits_repository_state() {+ let bytes = serde_json::to_vec(&test_entry("id", JournalKind::FinishFailed)).unwrap();+ let text = String::from_utf8(bytes).unwrap();+ assert!(!text.contains("checkpoint"));+ assert!(!text.contains("changed_paths"));+ assert!(!text.contains("branch"));+ }++ #[test]+ fn legacy_relative_session_dir_resolves_against_recorded_cwd() {+ let mut entry = test_entry("legacy", JournalKind::DegradedRun);+ entry.cwd = "/original/repo".into();+ entry.session_dir = Some("stores/native".into());+ assert_eq!(+ recovery_session_dir(&entry),+ Some(PathBuf::from("/original/repo/stores/native"))+ );+ }++ #[test]+ fn journal_entry_serializes_without_optional_noise() {+ let bytes = serde_json::to_vec(&test_entry("id", JournalKind::DegradedRun)).unwrap();+ let text = String::from_utf8(bytes).unwrap();+ assert!(text.contains("\"kind\":\"degraded-run\""));+ assert!(+ !text.contains("run_id"),+ "absent fields are omitted: {text}"+ );+ }++ #[test]+ fn journal_summary_counts_kinds_and_carries_errors() {+ let temp = tempfile::tempdir().unwrap();+ append_journal_entry(temp.path(), test_entry("d1", JournalKind::DegradedRun)).unwrap();+ append_journal_entry(temp.path(), test_entry("f1", JournalKind::FinishFailed)).unwrap();+ assert_eq!(+ journal_summary(temp.path()),+ (2, 1, 1, None),+ "(total, degraded, finish-failed, error)"+ );+ fs::write(journal_path(temp.path()), b"nope").unwrap();+ let (total, _, _, error) = journal_summary(temp.path());+ assert_eq!(total, 0);+ assert!(error.is_some());+ }++ #[test]+ fn dropped_spool_events_make_recovery_incomplete() {+ assert!(spool_is_incomplete(&SpoolOutcome {+ sent: 3,+ remaining: 0,+ dropped: 1,+ }));+ assert!(!spool_is_incomplete(&SpoolOutcome {+ sent: 3,+ remaining: 0,+ dropped: 0,+ }));+ }++ #[tokio::test]+ async fn failed_exact_replay_restores_correlated_spool_entry() {+ let temp = tempfile::tempdir().unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new().route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ ),+ )+ .await+ .unwrap()+ });+ let server_url = format!("http://{address}");+ let spool = hook_spool::spool_dir(temp.path());+ let created_ms = u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap();+ let correlated = hook_spool::SpoolEntry {+ url: format!(+ "{server_url}/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=proj&session_id=sess-1"+ ),+ body: r#"{"session_id":"sess-1","cwd":"/tmp/repo","prompt":"same prompt"}"#.into(),+ created_ms,+ auth_mode: hook_spool::AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ hook_spool::enqueue(&spool, &correlated).unwrap();+ let entry = JournalEntry {+ server_url: server_url.clone(),+ interval_started_ms: created_ms.saturating_sub(1),+ interval_ended_ms: created_ms.saturating_add(1),+ ..test_entry("restore", JournalKind::DegradedRun)+ };+ append_journal_entry(temp.path(), entry).unwrap();+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url,+ ..Config::default()+ };+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert_eq!(report.failures(), 1);+ assert_eq!(report.journal_remaining, 1);+ assert_eq!(hook_spool::spool_len(&spool), 1);+ server.abort();+ }++ #[tokio::test]+ async fn spool_only_harness_clears_only_after_a_complete_drain() {+ let temp = tempfile::tempdir().unwrap();+ let cwd = temp.path().join("proj");+ fs::create_dir_all(&cwd).unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new()+ .route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ )+ .route(+ "/hook/batch",+ axum::routing::post(|| async {+ axum::Json(serde_json::json!({+ "accepted": 1,+ "results": [{"index": 0, "outcome": "stored"}]+ }))+ }),+ ),+ )+ .await+ .unwrap()+ });+ let server_url = format!("http://{address}");+ let spool = hook_spool::spool_dir(temp.path());+ let created_ms = u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap();+ hook_spool::enqueue(+ &spool,+ &hook_spool::SpoolEntry {+ url: format!(+ "{server_url}/hook?event=user-prompt-submit&agent=antigravity-cli&workspace=ws&project=proj"+ ),+ body: serde_json::json!({+ "conversationId": "sess-1",+ "cwd": cwd,+ "prompt": "captured"+ })+ .to_string(),+ created_ms,+ auth_mode: hook_spool::AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ },+ )+ .unwrap();+ let entry = JournalEntry {+ harness: "antigravity".into(),+ cwd: cwd.to_string_lossy().into_owned(),+ project: "proj".into(),+ server_url: server_url.clone(),+ interval_started_ms: created_ms.saturating_sub(1),+ interval_ended_ms: created_ms.saturating_add(1),+ ..test_entry("spool-only", JournalKind::DegradedRun)+ };+ append_journal_entry(temp.path(), entry).unwrap();+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url,+ ..Config::default()+ };+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert_eq!(+ report.spool.sent, 0,+ "correlated evidence drains separately"+ );+ assert_eq!(report.sessions[0].outcome, "recovered-via-spool");+ assert_eq!(report.journal_remaining, 0);+ assert!(load_journal(temp.path()).unwrap().is_empty());+ server.abort();+ }++ /// A spool-only entry dispositioned Failed because an *unrelated* queued+ /// event kept the drain incomplete must not wedge recovery after its own+ /// interval already completed: restoring the quarantine treats the+ /// completed marker as durable delivery, the rerun neither errors nor+ /// re-delivers, and the journal clears.+ #[tokio::test]+ async fn completed_spool_only_interval_survives_an_unrelated_incomplete_drain() {+ use axum::response::IntoResponse as _;++ let temp = tempfile::tempdir().unwrap();+ let cwd = temp.path().join("proj");+ fs::create_dir_all(&cwd).unwrap();+ let deliveries = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));+ let handler_deliveries = deliveries.clone();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new()+ .route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ )+ .route(+ "/hook/batch",+ axum::routing::post(move |body: axum::body::Bytes| {+ let deliveries = handler_deliveries.clone();+ async move {+ let Ok(items) =+ serde_json::from_slice::<Vec<serde_json::Value>>(&body)+ else {+ return (+ axum::http::StatusCode::BAD_REQUEST,+ "malformed batch",+ )+ .into_response();+ };+ let prompts: Vec<String> = items+ .iter()+ .filter_map(|item| {+ item.get("body")?.get("prompt")?.as_str().map(str::to_owned)+ })+ .collect();+ if prompts.iter().any(|prompt| prompt == "unrelated") {+ return (+ axum::http::StatusCode::INTERNAL_SERVER_ERROR,+ "unrelated event refused",+ )+ .into_response();+ }+ deliveries.lock().unwrap().extend(prompts);+ axum::Json(serde_json::json!({+ "accepted": items.len(),+ "results": (0..items.len())+ .map(|index| serde_json::json!({+ "index": index,+ "outcome": "stored"+ }))+ .collect::<Vec<_>>()+ }))+ .into_response()+ }+ }),+ ),+ )+ .await+ .unwrap()+ });+ let server_url = format!("http://{address}");+ let spool = hook_spool::spool_dir(temp.path());+ let created_ms = u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap();+ let correlated = hook_spool::SpoolEntry {+ url: format!(+ "{server_url}/hook?event=user-prompt-submit&agent=antigravity-cli&workspace=ws&project=proj"+ ),+ body: serde_json::json!({+ "conversationId": "sess-1",+ "cwd": cwd,+ "prompt": "captured"+ })+ .to_string(),+ created_ms,+ auth_mode: hook_spool::AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ let unrelated = hook_spool::SpoolEntry {+ url: format!(+ "{server_url}/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=proj&session_id=unrelated-1"+ ),+ body: r#"{"session_id":"unrelated-1","cwd":"/tmp/repo","prompt":"unrelated"}"#.into(),+ created_ms: created_ms.saturating_add(1),+ auth_mode: hook_spool::AuthMode::Anonymous,+ token: None,+ attempts: 0,+ profile: None,+ };+ hook_spool::enqueue(&spool, &correlated).unwrap();+ hook_spool::enqueue(&spool, &unrelated).unwrap();+ let entry = JournalEntry {+ harness: "antigravity".into(),+ spool_only: true,+ native_session_id: Some("sess-1".into()),+ cwd: cwd.to_string_lossy().into_owned(),+ workspace: "ws".into(),+ project: "proj".into(),+ server_url: server_url.clone(),+ interval_started_ms: created_ms.saturating_sub(1),+ interval_ended_ms: created_ms.saturating_add(2),+ ..test_entry("completed-rerun", JournalKind::DegradedRun)+ };+ append_journal_entry(temp.path(), entry).unwrap();+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url,+ ..Config::default()+ };+ let first = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert_eq!(first.sessions[0].outcome, "recovered-via-spool");+ assert_eq!(first.journal_remaining, 0);+ assert!(first.spool_incomplete, "the unrelated event still queues");+ assert!(load_journal(temp.path()).unwrap().is_empty());++ let rerun = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert!(rerun.sessions.is_empty());+ assert_eq!(rerun.failures(), 0);+ assert_eq!(rerun.journal_remaining, 0);+ assert!(rerun.spool_incomplete);+ assert!(+ load_journal(temp.path()).unwrap().is_empty(),+ "the rerun must clear the journal"+ );+ assert_eq!(+ deliveries+ .lock()+ .unwrap()+ .iter()+ .filter(|prompt| *prompt == "captured")+ .count(),+ 1,+ "the completed interval's evidence is never re-delivered"+ );+ assert_eq!(+ hook_spool::spool_len(&spool),+ 1,+ "only the unrelated event stays queued"+ );+ server.abort();+ }++ #[tokio::test]+ async fn spool_only_without_correlated_evidence_never_auto_clears() {+ let temp = tempfile::tempdir().unwrap();+ let mut entry = test_entry("no-hooks", JournalKind::DegradedRun);+ entry.harness = "antigravity".into();+ append_journal_entry(temp.path(), entry).unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new().route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ ),+ )+ .await+ .unwrap();+ });+ let server_url = format!("http://{address}");+ mutate_journal(temp.path(), |entries| {+ entries[0].server_url.clone_from(&server_url);+ Ok(())+ })+ .unwrap();+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url,+ ..Config::default()+ };+ for _ in 0..2 {+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert_eq!(report.failures(), 1);+ assert_eq!(report.journal_remaining, 1);+ assert!(+ report.sessions[0]+ .detail+ .as_deref()+ .unwrap()+ .contains("no durably delivered")+ );+ }+ server.abort();+ }++ #[tokio::test]+ async fn a_dropped_spool_event_stays_lost_on_the_next_empty_pass() {+ let temp = tempfile::tempdir().unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new().route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ ),+ )+ .await+ .unwrap();+ });+ let server_url = format!("http://{address}");+ let spool = hook_spool::spool_dir(temp.path());+ let created_ms = u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap();+ hook_spool::enqueue(+ &spool,+ &hook_spool::SpoolEntry {+ url: format!(+ "{server_url}/hook?event=user-prompt-submit&agent=antigravity-cli&workspace=ws&project=proj"+ ),+ body: r#"{"conversationId":"sess-1","cwd":"/tmp/repo","prompt":"lost"}"#.into(),+ created_ms,+ auth_mode: hook_spool::AuthMode::Anonymous,+ token: None,+ attempts: 7,+ profile: None,+ },+ )+ .unwrap();+ let entry = JournalEntry {+ harness: "antigravity".into(),+ server_url: server_url.clone(),+ interval_started_ms: created_ms.saturating_sub(1),+ interval_ended_ms: created_ms.saturating_add(1),+ ..test_entry("dropped", JournalKind::DegradedRun)+ };+ append_journal_entry(temp.path(), entry).unwrap();+ mutate_journal(temp.path(), |entries| {+ entries[0].correlated_events_seen = true;+ entries[0].correlated_loss = true;+ Ok(())+ })+ .unwrap();+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url,+ ..Config::default()+ };+ let first = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert_eq!(+ first.spool.sent, 0,+ "correlated evidence never rides the main drain"+ );+ assert_eq!(+ first.failures(),+ 1,+ "the correlated loss keeps the journal entry"+ );+ assert!(load_journal(temp.path()).unwrap()[0].correlated_loss);+ let second = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert_eq!(second.spool.dropped, 0, "the dropped evidence stays lost");+ assert_eq!(second.journal_remaining, 1);+ assert!(load_journal(temp.path()).unwrap()[0].correlated_loss);+ server.abort();+ }++ #[tokio::test]+ async fn a_lossy_spool_pass_keeps_the_recovery_journal() {+ let temp = tempfile::tempdir().unwrap();+ let mut entry = test_entry("keep", JournalKind::DegradedRun);+ entry.harness = "antigravity".into();+ append_journal_entry(temp.path(), entry).unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new().route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ ),+ )+ .await+ .unwrap();+ });+ let spool = hook_spool::spool_dir(temp.path());+ hook_spool::enqueue(+ &spool,+ &hook_spool::SpoolEntry {+ url: format!("http://{address}/hook?event=user-prompt-submit"),+ body: "not-json".into(),+ created_ms: u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap(),+ auth_mode: hook_spool::AuthMode::Anonymous,+ token: None,+ attempts: 7,+ profile: None,+ },+ )+ .unwrap();+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url: format!("http://{address}"),+ ..Config::default()+ };+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert_eq!(report.spool.dropped, 1);+ assert!(report.spool_incomplete);+ assert_eq!(report.journal_remaining, 1);+ assert!(+ report.sessions[0]+ .detail+ .as_deref()+ .is_some_and(|detail| detail.contains("1 dropped"))+ );+ server.abort();+ }++ #[tokio::test]+ async fn per_entry_budget_times_out_and_keeps_the_entry() {+ let temp = tempfile::tempdir().unwrap();+ append_journal_entry(temp.path(), test_entry("one", JournalKind::DegradedRun)).unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new().route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ ),+ )+ .await+ .unwrap();+ });+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url: format!("http://{address}"),+ ..Config::default()+ };+ let report = recover(+ &config,+ &RecoverBudgets {+ entry_timeout: Duration::ZERO,+ ..RecoverBudgets::default()+ },+ )+ .await+ .unwrap();+ assert_eq!(report.failures(), 1);+ assert!(+ report.sessions[0]+ .detail+ .as_deref()+ .is_some_and(|detail| detail.contains("timed out"))+ );+ assert_eq!(load_journal(temp.path()).unwrap().len(), 1);+ server.abort();+ }++ #[tokio::test]+ async fn total_journal_budget_keeps_unattempted_entries() {+ let temp = tempfile::tempdir().unwrap();+ append_journal_entry(temp.path(), test_entry("one", JournalKind::DegradedRun)).unwrap();+ append_journal_entry(temp.path(), test_entry("two", JournalKind::DegradedRun)).unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new().route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ ),+ )+ .await+ .unwrap();+ });+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url: format!("http://{address}"),+ ..Config::default()+ };+ let report = recover(+ &config,+ &RecoverBudgets {+ journal_total: Duration::ZERO,+ ..RecoverBudgets::default()+ },+ )+ .await+ .unwrap();+ assert_eq!(report.failures(), 2);+ assert!(report.sessions.iter().all(|session| {+ session+ .detail+ .as_deref()+ .is_some_and(|detail| detail.contains("budget exhausted"))+ }));+ assert_eq!(load_journal(temp.path()).unwrap().len(), 2);+ server.abort();+ }++ #[tokio::test]+ async fn a_journal_entry_cannot_be_replayed_to_another_server() {+ let temp = tempfile::tempdir().unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new().route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ ),+ )+ .await+ .unwrap();+ });+ append_journal_entry(+ temp.path(),+ JournalEntry {+ server_url: "http://127.0.0.1:1".to_string(),+ ..test_entry("foreign-server", JournalKind::DegradedRun)+ },+ )+ .unwrap();+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url: format!("http://{address}"),+ ..Config::default()+ };+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();+ assert_eq!(report.failures(), 1);+ assert!(+ report.sessions[0]+ .detail+ .as_deref()+ .unwrap()+ .contains("refusing to replay")+ );+ assert_eq!(load_journal(temp.path()).unwrap().len(), 1);+ server.abort();+ }++ #[tokio::test]+ async fn failed_recovery_returns_a_testable_nonzero_code_without_exiting() {+ let temp = tempfile::tempdir().unwrap();+ append_journal_entry(+ temp.path(),+ JournalEntry {+ native_session_id: None,+ ..test_entry("bad", JournalKind::FinishFailed)+ },+ )+ .unwrap();+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let address = listener.local_addr().unwrap();+ let server = tokio::spawn(async move {+ axum::serve(+ listener,+ axum::Router::new().route(+ "/healthz",+ axum::routing::get(|| async { axum::http::StatusCode::OK }),+ ),+ )+ .await+ .unwrap();+ });+ let config = Config {+ data_dir: temp.path().to_path_buf(),+ server_url: format!("http://{address}"),+ ..Config::default()+ };+ let code = run(&config, RecoverArgs { json: false }).await.unwrap();+ assert_eq!(code, 1);+ assert_eq!(load_journal(temp.path()).unwrap().len(), 1);+ server.abort();+ }+}diff --git a/crates/ai-memory-cli/src/commands/run.rs b/crates/ai-memory-cli/src/commands/run.rsindex e7ce3f01..d443ef58 100644--- a/crates/ai-memory-cli/src/commands/run.rs+++ b/crates/ai-memory-cli/src/commands/run.rs@@ -25,7 +25,8 @@ use ai_memory_workstream::{kiro_selects_v3_engine, kiro_v3_resume_uses_default_store, list_native_sessions,marked_choices, native_session_exists, native_session_in_checkout, native_store_root,omp_profile_flag, omp_profile_flag_env, parse_jail_toggles, store_override_vars,- usable_ai_jail_here, wait_for_transcript_flush,+ transcript_baseline, transcript_interval_digests, usable_ai_jail_here,+ wait_for_transcript_flush,};use anyhow::{Context as _, Result, anyhow};use clap::ValueEnum as _;@@ -629,6 +630,8 @@ async fn run_once_with_wiring(harness: provisional_harness,executable: executable.clone(),native_args: &native_args,+ workspace: &workspace,+ project: &project,force_fresh,yolo_modes,jail_plan: &jail_plan,@@ -1217,25 +1220,43 @@ async fn run_once_with_wiring().await);- let transcript = if plan.mode == LaunchMode::Session {- let source_cursor = if native_session_id.as_deref() == prepared.native_session_id.as_deref()- {- prepared.source_cursor.as_deref()+ let recovery_start_cursor =+ if native_session_id.as_deref() == prepared.native_session_id.as_deref() {+ prepared.source_cursor.clone()} else {None};- export_after_flush(+ let (transcript, exact_export_available) = if plan.mode == LaunchMode::Session+ && harness.transcript_capability() == ai_memory_workstream::TranscriptCapability::Export+ {+ let exported = export_after_flush(harness,&home,&repository.cwd,plan.session_dir.as_deref(),native_session_id.as_deref(),- source_cursor,+ recovery_start_cursor.as_deref(),)- .await+ .await;+ let exact = exported.losses.iter().all(|loss| {+ !loss.starts_with("native session id could not be discovered")+ && !loss.starts_with("native transcript import failed")+ }) && exported.source_cursor.is_some();+ (exported, exact)} else {- ExportedTranscript::default()+ (+ ExportedTranscript {+ native_session_id: native_session_id.clone().unwrap_or_default(),+ ..ExportedTranscript::default()+ },+ false,+ )};+ let recovery_event_digests =+ exact_export_available.then(|| transcript_interval_digests(&transcript.events));+ let recovery_cursor = exact_export_available+ .then(|| transcript.source_cursor.clone())+ .flatten();let checkpoint = inspect_repository(&repository.cwd).map(|identity| identity.checkpoint).unwrap_or(repository.checkpoint);@@ -1266,6 +1287,23 @@ async fn run_once_with_wiring(&error,));+ journal_finish_failed_run(+ config,+ &endpoint,+ harness,+ native_session_id.as_deref(),+ &plan,+ &workspace,+ &project,+ &repository.cwd,+ started_at,+ exit_code,+ prepared.run_id,+ recovery_event_digests.as_deref(),+ recovery_start_cursor.as_deref(),+ recovery_cursor.as_deref(),+ &privacy,+ );interrupt_task.abort();return Ok(RunOutcome {exit_code,@@ -1348,6 +1386,8 @@ struct DegradedLaunch<'a> {harness: ManagedHarness,executable: Option<OsString>,native_args: &'a [OsString],+ workspace: &'a str,+ project: &'a str,force_fresh: bool,yolo_modes: YoloModes,jail_plan: &'a JailPlan,@@ -1470,6 +1510,41 @@ async fn degraded_launch(for name in blank_home_overrides(launch.harness, &launch_env) {command.env_remove(name);}+ let prelaunch_sessions = snapshot_native_sessions(+ launch.harness,+ launch.home,+ launch.cwd,+ plan.session_dir.as_deref(),+ )+ .await;+ let prelaunch_cursor = if let Some(session) = plan.expected_session_id.as_deref()+ && launch.harness.transcript_capability()+ == ai_memory_workstream::TranscriptCapability::Export+ {+ match native_session_exists(+ launch.harness,+ launch.home,+ launch.cwd,+ plan.session_dir.as_deref(),+ session,+ ) {+ Ok(true) => Some(+ transcript_baseline(+ launch.harness,+ launch.home,+ launch.cwd,+ plan.session_dir.as_deref(),+ session,+ )+ .await,+ ),+ Ok(false) => None,+ Err(error) => Some(Err(error)),+ }+ } else {+ None+ };+ let started_at = SystemTime::now();let child = command.spawn().map_err(|spawn_error| {let spawn_message = spawn_error.to_string();anyhow!(spawn_message).context(format!(@@ -1486,6 +1561,16 @@ async fn degraded_launch("ai-memory: this run was not recorded on the server; {}",spooled_events_note(&spool));+ journal_degraded_run(+ config,+ launch,+ &plan,+ exit_code,+ started_at,+ prelaunch_sessions,+ prelaunch_cursor,+ )+ .await;Ok(RunOutcome {exit_code,harness: launch.harness,@@ -1496,6 +1581,255 @@ async fn degraded_launch(})}+/// Journal a degraded session launch so `ai-memory recover` can close it out+/// once the server returns. Best-effort by contract: the child has already+/// exited and its exit code is preserved above everything else, so a journal+/// problem may only warn. Passthrough launches are not journaled — they hold+/// no session and no transcript, so there is nothing to recover.+async fn journal_degraded_run(+ config: &Config,+ launch: &DegradedLaunch<'_>,+ plan: &LaunchPlan,+ exit_code: i32,+ started_at: SystemTime,+ prelaunch_sessions: Option<std::collections::HashSet<String>>,+ prelaunch_cursor: Option<Result<String>>,+) {+ if plan.mode != LaunchMode::Session {+ return;+ }+ let native_session_id = match Sanitizer::new(&config.sanitize) {+ Ok(privacy) => resolve_degraded_session(+ plan,+ launch.harness,+ launch.home,+ launch.cwd,+ started_at,+ prelaunch_sessions.as_ref(),+ &privacy,+ )+ .await+ .unwrap_or_else(|error| {+ eprintln!(+ "ai-memory: could not uniquely identify the session this run used ({error:#}); \+ nothing was journaled; preserve the local transcript and run `ai-memory \+ backfill --force --session <id>` after identifying it"+ );+ None+ }),+ Err(error) => {+ eprintln!(+ "ai-memory: could not record this run for recovery (privacy config: {error}); \+ nothing was journaled"+ );+ return;+ }+ };+ let Some(native_session_id) = native_session_id else {+ eprintln!(+ "ai-memory: no native session was discovered, so this run could not be journaled; \+ retain the harness transcript for manual backfill"+ );+ return;+ };+ let source_cursor = match prelaunch_cursor {+ Some(Ok(cursor)) => Some(cursor),+ Some(Err(error)) => {+ eprintln!(+ "ai-memory: could not record a safe delta baseline for resumed session {} ({error:#}); nothing was journaled; retain the native transcript for manual repair",+ display_session_id(&native_session_id)+ );+ return;+ }+ None => None,+ };+ let spool_only = launch.harness.transcript_capability()+ == ai_memory_workstream::TranscriptCapability::SpoolOnly;+ let (event_digests, final_cursor) = if spool_only {+ (None, None)+ } else {+ let exact_export = match export_exact_after_run(+ launch.harness,+ launch.home,+ launch.cwd,+ plan.session_dir.as_deref(),+ &native_session_id,+ source_cursor.as_deref(),+ )+ .await+ {+ Ok(export) => export,+ Err(error) => {+ eprintln!(+ "ai-memory: could not capture an exact post-run transcript bound for session {} ({error:#}); nothing was journaled; retain the native transcript for manual repair",+ display_session_id(&native_session_id)+ );+ return;+ }+ };+ let Some(final_cursor) = exact_export.source_cursor.clone() else {+ eprintln!(+ "ai-memory: the {} transcript adapter produced no post-run cursor; nothing was journaled; retain the native transcript for manual repair",+ launch.harness.as_str()+ );+ return;+ };+ (+ Some(transcript_interval_digests(&exact_export.events)),+ Some(final_cursor),+ )+ };+ let entry = super::recovery::JournalEntry {+ id: uuid::Uuid::new_v4().to_string(),+ recorded_at: jiff::Timestamp::now().to_string(),+ interval_started_ms: system_time_ms(started_at),+ interval_ended_ms: system_time_ms(SystemTime::now()),+ kind: super::recovery::JournalKind::DegradedRun,+ harness: launch.harness.as_str().to_string(),+ native_session_id: Some(native_session_id),+ session_dir: absolute_session_dir(plan.session_dir.as_deref(), launch.cwd)+ .map(|path| path.to_string_lossy().into_owned()),+ cwd: launch.cwd.to_string_lossy().into_owned(),+ workspace: launch.workspace.to_string(),+ project: launch.project.to_string(),+ server_url: launch.endpoint.identity(),+ run_id: None,+ event_digests,+ spool_only,+ source_cursor,+ final_cursor,+ correlated_events_seen: false,+ correlated_delivery_confirmed: false,+ correlated_loss: false,+ exit_code: Some(exit_code),+ failures: Vec::new(),+ };+ match super::recovery::append_journal_entry(&config.data_dir, entry) {+ Ok(()) => {+ eprintln!(+ "ai-memory: recorded for recovery — run `ai-memory recover` once the server \+ is back to replay this session's exact native transcript"+ );+ }+ Err(error) => {+ eprintln!(+ "ai-memory: could not record this run for recovery ({error:#}); the native \+ transcript remains in the harness store and can be imported with `ai-memory \+ backfill --force --session <id>`"+ );+ }+ }+}++const DEGRADED_SESSION_SNAPSHOT_LIMIT: usize = 2_000;++async fn snapshot_native_sessions(+ harness: ManagedHarness,+ home: &Path,+ cwd: &Path,+ session_dir: Option<&Path>,+) -> Option<std::collections::HashSet<String>> {+ let sessions = list_native_sessions(+ harness,+ home,+ cwd,+ session_dir,+ DEGRADED_SESSION_SNAPSHOT_LIMIT + 1,+ )+ .await+ .ok()?;+ if sessions.len() > DEGRADED_SESSION_SNAPSHOT_LIMIT {+ return None;+ }+ Some(+ sessions+ .into_iter()+ .map(|session| session.native_session_id)+ .collect(),+ )+}++#[allow(clippy::too_many_arguments)]+async fn resolve_degraded_session(+ plan: &LaunchPlan,+ harness: ManagedHarness,+ home: &Path,+ cwd: &Path,+ started_at: SystemTime,+ before: Option<&std::collections::HashSet<String>>,+ sanitizer: &Sanitizer,+) -> Result<Option<String>> {+ if let Some(expected) = plan.expected_session_id.as_deref() {+ return NativeSessionIdentity::parse(expected, sanitizer)+ .map(NativeSessionIdentity::into_string)+ .map(Some)+ .map_err(anyhow::Error::from);+ }+ let Some(before) = before else {+ return Err(anyhow!(+ "pre-launch native-session snapshot was unavailable"+ ));+ };+ let after = list_native_sessions(+ harness,+ home,+ cwd,+ plan.session_dir.as_deref(),+ DEGRADED_SESSION_SNAPSHOT_LIMIT + 1,+ )+ .await?;+ if after.len() > DEGRADED_SESSION_SNAPSHOT_LIMIT {+ return Err(anyhow!(+ "post-launch native-session snapshot exceeded its {}-record limit",+ DEGRADED_SESSION_SNAPSHOT_LIMIT+ ));+ }+ let native_session_id = unique_new_session(after, before, started_at)?;+ NativeSessionIdentity::parse(&native_session_id, sanitizer)+ .map(NativeSessionIdentity::into_string)+ .map(Some)+ .map_err(anyhow::Error::from)+}++fn unique_new_session(+ after: Vec<NativeSessionCandidate>,+ before: &std::collections::HashSet<String>,+ started_at: SystemTime,+) -> Result<String> {+ let mut new = after+ .into_iter()+ .filter(|candidate| {+ candidate.updated_at + Duration::from_secs(2) >= started_at+ && !before.contains(&candidate.native_session_id)+ })+ .collect::<Vec<_>>();+ if new.len() != 1 {+ return Err(anyhow!(+ "expected one new native session after launch, found {}",+ new.len()+ ));+ }+ Ok(new.remove(0).native_session_id)+}++fn system_time_ms(time: SystemTime) -> u64 {+ time.duration_since(std::time::UNIX_EPOCH)+ .unwrap_or_default()+ .as_millis()+ .try_into()+ .unwrap_or(u64::MAX)+}++fn absolute_session_dir(session_dir: Option<&Path>, cwd: &Path) -> Option<PathBuf> {+ session_dir.map(|path| {+ if path.is_absolute() {+ clean_path(path)+ } else {+ clean_path(&cwd.join(path))+ }+ })+}+/// The one-line account of what remains in the local hook spool after a/// degraded run, mirroring `ai-memory status`'s offline-spool phrasing.fn spooled_events_note(spool: &super::hook_spool::SpoolHealth) -> String {@@ -1526,9 +1860,10 @@ fn offline_launch_warning(format!("ai-memory: WARNING: the ai-memory server at {} is unreachable — launching {} \WITHOUT server support.\n {diagnosis:#}\n Lost this run: no workstream lease or \- cross-harness context, no transcript import, no handoff delivery.\n Still works: \- lifecycle hooks spool events locally and drain when the server returns; an existing \- MCP registration degrades to no-recall rather than blocking the session.\n Spooled \+ cross-harness context, no immediate transcript import, no handoff delivery.\n Still \+ works: lifecycle hooks spool events locally, and the native transcript is journaled \+ after session discovery. Run `ai-memory recover` when the server returns to drain the \+ spool and replay the exact transcript through the sanitized backfill path.\n Spooled \events are dropped after {} failed drain passes, 7 days, or the 10,000-file spool \cap (set AI_MEMORY_HOOK_SPOOL_MAX_ATTEMPTS=0 for a planned outage). Pass \--require-server to fail instead of degrading.",@@ -1548,23 +1883,86 @@ fn offline_import_warning(project: &str,error: &anyhow::Error,) -> String {- let repair = match native_session_id {- Some(session) => format!(- "ai-memory finalize-session --agent {} --reopen --session-id {} --workspace {} --project {}",- harness.agent_kind().as_str(),- SessionId::from_native(session),- super::render_shared::shell_quote(workspace),- super::render_shared::shell_quote(project),- ),- None => "ai-memory finalize-session (see --help)".to_string(),- };+ let session = native_session_id+ .map(SessionId::from_native)+ .map_or_else(|| "unknown".to_string(), |session| session.to_string());format!(- "ai-memory: warning: {error:#}; the transcript was not imported and the orphaned lease \- will expire automatically within 90 seconds. The harness exit code is preserved. \- Repair it once the server is back with `{repair}`."+ "ai-memory: warning: {error:#}; the transcript for {} session {session} in {}/{} was \+ not imported and the orphaned lease will expire automatically within 90 seconds. The \+ harness exit code is preserved. Run `ai-memory recover` once the server is back.",+ harness.as_str(),+ workspace,+ project,)}+/// Journal a run whose transcript import failed on an unreachable server, so+/// `ai-memory recover` can re-export and re-import it once the server+/// returns. The journal keeps only locators and recomputes the repository+/// checkpoint during recovery — never transcript content. Best-effort: the+/// child's exit code is already preserved, so a journal problem may only warn.+#[allow(clippy::too_many_arguments)]+fn journal_finish_failed_run(+ config: &Config,+ endpoint: &ServerEndpoint,+ harness: ManagedHarness,+ native_session_id: Option<&str>,+ plan: &LaunchPlan,+ workspace: &str,+ project: &str,+ cwd: &Path,+ started_at: SystemTime,+ exit_code: i32,+ run_id: ai_memory_core::ManagedRunId,+ event_digests: Option<&[String]>,+ source_cursor: Option<&str>,+ final_cursor: Option<&str>,+ sanitizer: &Sanitizer,+) {+ let entry = super::recovery::JournalEntry {+ id: uuid::Uuid::new_v4().to_string(),+ recorded_at: jiff::Timestamp::now().to_string(),+ interval_started_ms: system_time_ms(started_at),+ interval_ended_ms: system_time_ms(SystemTime::now()),+ kind: super::recovery::JournalKind::FinishFailed,+ harness: harness.as_str().to_string(),+ native_session_id: native_session_id+ .and_then(|id| NativeSessionIdentity::parse(id, sanitizer).ok())+ .map(NativeSessionIdentity::into_string),+ session_dir: absolute_session_dir(plan.session_dir.as_deref(), cwd)+ .map(|path| path.to_string_lossy().into_owned()),+ cwd: cwd.to_string_lossy().into_owned(),+ workspace: workspace.to_string(),+ project: project.to_string(),+ server_url: endpoint.identity(),+ run_id: Some(run_id.to_string()),+ event_digests: event_digests.map(<[String]>::to_vec),+ spool_only: harness.transcript_capability()+ == ai_memory_workstream::TranscriptCapability::SpoolOnly,+ source_cursor: source_cursor.map(str::to_owned),+ final_cursor: final_cursor.map(str::to_owned),+ correlated_events_seen: false,+ correlated_delivery_confirmed: false,+ correlated_loss: false,+ exit_code: Some(exit_code),+ failures: Vec::new(),+ };+ match super::recovery::append_journal_entry(&config.data_dir, entry) {+ Ok(()) => {+ eprintln!(+ "ai-memory: recorded for recovery — run `ai-memory recover` once the server \+ is back to re-import this transcript"+ );+ }+ Err(error) => {+ eprintln!(+ "ai-memory: could not record this run for recovery ({error:#}); retain the \+ native transcript and original run id for manual repair"+ );+ }+ }+}+/// Whether a managed-run request failed because the server could not be/// reached at all (connect error or timeout) rather than answering with a/// protocol-level failure. Only the former may be downgraded to a warning@@ -3278,6 +3676,26 @@ fn session_age(updated_at: SystemTime, now: SystemTime) -> String {super::humanize_age_secs(i64::try_from(secs).unwrap_or(i64::MAX))}+async fn export_exact_after_run(+ harness: ManagedHarness,+ home: &std::path::Path,+ cwd: &std::path::Path,+ session_dir: Option<&std::path::Path>,+ native_session_id: &str,+ source_cursor: Option<&str>,+) -> Result<ExportedTranscript> {+ wait_for_transcript_flush(harness, home, cwd, session_dir, native_session_id).await?;+ export_transcript(+ harness,+ home,+ cwd,+ session_dir,+ native_session_id,+ source_cursor,+ )+ .await+}+async fn export_after_flush(harness: ManagedHarness,home: &std::path::Path,@@ -3319,7 +3737,7 @@ async fn export_after_flush(}}-async fn import_batches(+pub(crate) async fn import_batches(endpoint: &ServerEndpoint,run_path: &str,transcript: ExportedTranscript,@@ -3730,6 +4148,29 @@ const fn managed_harness_from_agent(agent: AgentKind) -> Option<ManagedHarness>}}+/// Parse the exact harness names the recovery journal writes+/// (`ManagedHarness::as_str`). The user-facing `from_name` table deliberately+/// has no "kiro-v3" spelling (no one types that), but the journal must round+/// trip its own exact value, engine flavor included.+pub(crate) fn harness_from_journal_name(name: &str) -> Option<ManagedHarness> {+ match name {+ "claude" => Some(ManagedHarness::Claude),+ "codex" => Some(ManagedHarness::Codex),+ "opencode" => Some(ManagedHarness::OpenCode),+ "opencode2" => Some(ManagedHarness::OpenCode2),+ "pi" => Some(ManagedHarness::Pi),+ "crush" => Some(ManagedHarness::Crush),+ "omp" => Some(ManagedHarness::Omp),+ "kimi" => Some(ManagedHarness::Kimi),+ "command-code" => Some(ManagedHarness::CommandCode),+ "kiro" => Some(ManagedHarness::Kiro),+ "kiro-v3" => Some(ManagedHarness::KiroV3),+ "grok" => Some(ManagedHarness::Grok),+ "antigravity" => Some(ManagedHarness::Antigravity),+ _ => None,+ }+}+#[cfg(test)]mod tests {use std::ffi::{OsStr, OsString};@@ -4612,6 +5053,43 @@ mod tests {]}+ #[test]+ fn degraded_session_correlation_requires_one_new_session() {+ let started = SystemTime::UNIX_EPOCH + Duration::from_secs(100);+ let before = std::collections::HashSet::from(["old".to_string()]);+ let candidate = |id: &str| NativeSessionCandidate {+ native_session_id: id.into(),+ updated_at: started + Duration::from_secs(1),+ };+ assert_eq!(+ unique_new_session(vec![candidate("old"), candidate("new")], &before, started).unwrap(),+ "new"+ );+ assert!(+ unique_new_session(+ vec![candidate("first"), candidate("second")],+ &before,+ started+ )+ .unwrap_err()+ .to_string()+ .contains("found 2")+ );+ assert!(unique_new_session(vec![candidate("old")], &before, started).is_err());+ }++ #[test]+ fn relative_session_dir_is_persisted_against_launch_cwd() {+ assert_eq!(+ absolute_session_dir(Some(Path::new("stores/native")), Path::new("/repo")),+ Some(PathBuf::from("/repo/stores/native"))+ );+ assert_eq!(+ absolute_session_dir(Some(Path::new("/stores/native")), Path::new("/repo")),+ Some(PathBuf::from("/stores/native"))+ );+ }+#[test]fn native_grok_rules_flags_suppress_context_injection() {for args in [@@ -6128,8 +6606,7 @@ mod tests {}/// A fake Claude executable: captures its environment and argv, writes- /// the (empty-session) transcript the launcher waits for, and exits with- /// `code`.+ /// the transcript the launcher waits for, and exits with `code`.#[cfg(unix)]fn capture_env_claude(repo: &Path, home: &Path, code: i32) -> PathBuf {use std::os::unix::fs::PermissionsExt as _;@@ -6403,8 +6880,8 @@ mod tests {}/// The degraded-launch warning names everything the operator needs: the- /// server URL, what is lost, what still works (spool + no-recall MCP),- /// the bounded retention, and the opt-outs.+ /// server URL, what is unavailable, what remains recoverable, the bounded+ /// retention, and the opt-outs.#[tokio::test]async fn offline_launch_warning_names_the_tradeoffs() {let url = closed_port_url().await;@@ -6423,7 +6900,8 @@ mod tests {assert!(warning.contains("Lost this run"), "{warning}");assert!(warning.contains("Still works"), "{warning}");assert!(warning.contains("spool"), "{warning}");- assert!(warning.contains("no-recall"), "{warning}");+ assert!(warning.contains("ai-memory recover"), "{warning}");+ assert!(warning.contains("sanitized backfill"), "{warning}");assert!(warning.contains("8 failed drain passes"), "{warning}");assert!(warning.contains("AI_MEMORY_HOOK_SPOOL_MAX_ATTEMPTS=0"),@@ -6478,7 +6956,7 @@ mod tests {}#[test]- fn offline_import_warning_points_at_finalize_session() {+ fn offline_import_warning_points_at_recovery() {let warning = offline_import_warning(ManagedHarness::Claude,Some("session-123"),@@ -6487,15 +6965,10 @@ mod tests {&anyhow!("connection refused"),);assert!(warning.contains("exit code is preserved"), "{warning}");+ assert!(warning.contains("ai-memory recover"), "{warning}");+ assert!(warning.contains("in ws/proj"), "{warning}");assert!(- warning.contains(&format!(- "ai-memory finalize-session --agent claude-code --reopen --session-id {}",- SessionId::from_native("session-123")- )),- "{warning}"- );- assert!(- warning.contains("--workspace ws --project proj"),+ warning.contains(&SessionId::from_native("session-123").to_string()),"{warning}");@@ -6506,10 +6979,8 @@ mod tests {"proj",&anyhow!("timed out"),);- assert!(- generic.contains("finalize-session (see --help)"),- "{generic}"- );+ assert!(generic.contains("session unknown"), "{generic}");+ assert!(generic.contains("ai-memory recover"), "{generic}");}/// A server that dies between prepare and finish: the child's exit codediff --git a/crates/ai-memory-cli/src/commands/status.rs b/crates/ai-memory-cli/src/commands/status.rsindex aa4c6c8c..c435d9c5 100644--- a/crates/ai-memory-cli/src/commands/status.rs+++ b/crates/ai-memory-cli/src/commands/status.rs@@ -188,6 +188,29 @@ fn report_offline_spool(spool: &SpoolHealth, json: bool) {}}+/// One line pointing at `ai-memory recover` while the server is down, when+/// there is actually something waiting: queued hook events or journaled+/// outage runs. Goes to stderr like the spool block above, so `--json`+/// consumers' stdout stays a single object.+fn report_offline_recovery_hint(config: &Config, spool: &SpoolHealth, json: bool) {+ let (journal_pending, _, _, journal_error) = super::recovery::journal_summary(&config.data_dir);+ if spool.pending == 0 && journal_pending == 0 && journal_error.is_none() {+ return;+ }+ if json {+ eprintln!(+ "recovery hint (server unreachable): {{\"journal_pending\": {journal_pending},\+ \"spool_pending\": {}}}",+ spool.pending+ );+ return;+ }+ eprintln!(+ " run `ai-memory recover` once the server is back to deliver queued events and \+ repair unrecorded runs ({journal_pending} journaled)"+ );+}+/// Which capture mode the hook would enforce for this install (#446).////// Reads the same file the hook reads rather than keeping a second source of@@ -221,32 +244,26 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {Ok(report) => report,Err(err) => {report_offline_spool(&spool, args.json);+ report_offline_recovery_hint(config, &spool, args.json);return Err(err);}};+ let (journal_pending, journal_degraded, journal_finish_failed, journal_error) =+ super::recovery::journal_summary(&config.data_dir);if args.json {println!("{}",- serde_json::to_string_pretty(&serde_json::json!({- "version": report.version,- "data_dir": report.data_dir,- "bind": report.bind,- "db_path": report.db_path,- "counts": {- "pages_latest": report.counts.pages_latest,- "pages_all": report.counts.pages_all,- "sessions": report.counts.sessions,- "observations": report.counts.observations,- },- "derived": report.derived,- "storage": report.storage,- "providers": report.providers,- "spool": spool,- "capture_mode": capture_mode,- "ingest": report.ingest,- "client": { "server_url": ep.url, "auth": ep.auth_token.is_some() },- }))?+ serde_json::to_string_pretty(&json_report(+ report,+ spool,+ journal_pending,+ journal_degraded,+ journal_finish_failed,+ journal_error,+ capture_mode,+ &ep,+ ))?);} else {println!("ai-memory {} (server)", report.version);@@ -355,6 +372,16 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {println!(" pending: {}", spool.pending);println!(" oldest: {}", spool_age_line(spool.oldest_age_ms));println!(" retries: {}", spool.retries_total);+ if journal_pending > 0 || journal_error.is_some() {+ println!(" recovery:");+ println!(+ " journal: {journal_pending} pending ({journal_degraded} degraded, {journal_finish_failed} finish-failed)"+ );+ if let Some(error) = &journal_error {+ println!(" error: {error}");+ }+ println!(" action: run `ai-memory recover`");+ }if let Some(ingest) = &report.ingest {println!(" ingest (server, this process):");println!(" accepted: {}", ingest.accepted);@@ -453,6 +480,44 @@ pub(super) fn spool_age_line(age_ms: Option<u64>) -> String {}}+#[allow(clippy::too_many_arguments)]+fn json_report(+ report: Report,+ spool: SpoolHealth,+ journal_pending: usize,+ journal_degraded: usize,+ journal_finish_failed: usize,+ journal_error: Option<String>,+ capture_mode: &str,+ endpoint: &ServerEndpoint,+) -> serde_json::Value {+ serde_json::json!({+ "version": report.version,+ "data_dir": report.data_dir,+ "bind": report.bind,+ "db_path": report.db_path,+ "counts": {+ "pages_latest": report.counts.pages_latest,+ "pages_all": report.counts.pages_all,+ "sessions": report.counts.sessions,+ "observations": report.counts.observations,+ },+ "derived": report.derived,+ "storage": report.storage,+ "providers": report.providers,+ "spool": spool,+ "recovery": {+ "journal_pending": journal_pending,+ "degraded_runs": journal_degraded,+ "finish_failed_runs": journal_finish_failed,+ "journal_error": journal_error,+ },+ "capture_mode": capture_mode,+ "ingest": report.ingest,+ "client": { "server_url": endpoint.url, "auth": endpoint.auth_token.is_some() },+ })+}+fn provider_health_line(role: &ProviderRoleHealthSnapshot) -> String {match role.status {ProviderHealthStatus::Disabled => "disabled".to_string(),@@ -507,6 +572,64 @@ fn error_detail(role: &ProviderRoleHealthSnapshot) -> String {mod tests {use super::*;+ #[test]+ fn status_json_keeps_existing_fields_and_adds_recovery_compatibly() {+ let report: Report = serde_json::from_value(serde_json::json!({+ "version": "2.6.0",+ "data_dir": "/data",+ "bind": "127.0.0.1:49374",+ "db_path": "/data/db/memory.sqlite",+ "counts": {"pages_latest": 1, "pages_all": 2, "sessions": 3, "observations": 4},+ "derived": {+ "pages_rows": 0, "pages_fts_rows": 0,+ "observations_rows": 0, "observations_fts_rows": 0,+ "latest_pages_missing_embeddings": 0,+ "embedding_rows": 0, "embedding_triples": [],+ "links_from_latest_pages": 0,+ "unresolved_links_from_latest_pages": 0,+ "stale_links_from_latest_pages": 0+ },+ "storage": {+ "page_size": 0, "page_count": 0, "freelist_count": 0,+ "database_bytes": 0, "reclaimable_bytes": 0+ },+ "providers": ProviderHealthSnapshot::default(),+ "ingest": null,+ "write_queue": null,+ "wiki_format": null+ }))+ .unwrap();+ let endpoint = ServerEndpoint::from_pair(Some("http://127.0.0.1:49374".into()), None);+ let value = json_report(+ report,+ SpoolHealth::default(),+ 2,+ 1,+ 1,+ None,+ "denylist",+ &endpoint,+ );+ for key in [+ "version",+ "data_dir",+ "bind",+ "db_path",+ "counts",+ "derived",+ "storage",+ "providers",+ "spool",+ "capture_mode",+ "ingest",+ "client",+ ] {+ assert!(value.get(key).is_some(), "missing legacy status key {key}");+ }+ assert_eq!(value["recovery"]["journal_pending"], 2);+ assert_eq!(value["counts"]["observations"], 4);+ }+#[test]fn ingest_additions_default_for_older_servers_and_roundtrip() {let old = serde_json::json!({"accepted": 2, "dropped_by_policy": 1, "shed_saturated": 0, "shed_rate_limited": 0, "last_persisted_ms": null});diff --git a/crates/ai-memory-cli/src/lib.rs b/crates/ai-memory-cli/src/lib.rsindex b03dfd54..8cdc0d23 100644--- a/crates/ai-memory-cli/src/lib.rs+++ b/crates/ai-memory-cli/src/lib.rs@@ -160,6 +160,13 @@ pub async fn run() -> Result<()> {Command::AutoImproveReport(args) => commands::auto_improve_report::run(&config, args).await,Command::AutoImprove(args) => commands::auto_improve::run(&config, args).await,Command::FinalizeSession(args) => commands::finalize_session::run(&config, args).await,+ Command::Recover(args) => {+ let exit_code = commands::recovery::run(&config, args).await?;+ if exit_code != 0 {+ std::process::exit(exit_code);+ }+ Ok(())+ }Command::PendingWrites(args) => commands::pending_writes::run(&config, args).await,Command::Embed(args) => commands::embed::run(&config, args).await,Command::GenerateAuthToken(args) => commands::generate_auth_token::run(&config, args),diff --git a/crates/ai-memory-cli/tests/suite/backfill_e2e.rs b/crates/ai-memory-cli/tests/suite/backfill_e2e.rsindex 2473da11..b4a60b68 100644--- a/crates/ai-memory-cli/tests/suite/backfill_e2e.rs+++ b/crates/ai-memory-cli/tests/suite/backfill_e2e.rs@@ -79,6 +79,17 @@ mod slow {None}+ fn status_json(data_dir: &Path, home: &Path, cwd: &Path, base: &str) -> Value {+ serde_json::from_str(&run_cli(+ &["status", "--json"],+ data_dir,+ home,+ Some(cwd),+ base,+ ))+ .expect("status JSON remains compatible")+ }+/// Call a memory tool over the stateless Streamable-HTTP `/mcp` transport/// (the default `serve` mode: no `initialize` handshake needed) and return/// the joined text of the tool result.@@ -296,6 +307,41 @@ mod slow {"the re-run must not duplicate the imported session",);+ let observations_before = status_json(data_dir.path(), home.path(), &cwd, &base)["counts"]+ ["observations"]+ .as_u64()+ .unwrap();+ let forced: Value = serde_json::from_str(&run_cli(+ &[+ "backfill",+ "--workspace",+ WORKSPACE,+ "--project",+ PROJECT,+ "--force",+ "--json",+ ],+ data_dir.path(),+ home.path(),+ Some(&cwd),+ &base,+ ))+ .expect("forced replay report");+ assert_eq!(forced["imported_sessions"], 1);+ assert_eq!(+ session_count(&client, &base, WORKSPACE, PROJECT).await,+ 1,+ "a real second replay must reuse the session",+ );+ let status = status_json(data_dir.path(), home.path(), &cwd, &base);+ assert!(status["ingest"]["replayed"].as_u64().unwrap_or(0) >= 2);+ assert!(status["ingest"]["ignored_end"].as_u64().unwrap_or(0) >= 1);+ assert_eq!(+ status["counts"]["observations"].as_u64(),+ Some(observations_before),+ "the second replay must create no duplicate observations"+ );+drop(server);}}diff --git a/crates/ai-memory-cli/tests/suite/backfill_failures.rs b/crates/ai-memory-cli/tests/suite/backfill_failures.rsindex c5b6c29e..4370107f 100644--- a/crates/ai-memory-cli/tests/suite/backfill_failures.rs+++ b/crates/ai-memory-cli/tests/suite/backfill_failures.rs@@ -3,6 +3,69 @@use axum::{Json, Router, http::StatusCode, routing::post};use serde_json::{Value, json};+#[tokio::test]+async fn replay_refuses_legacy_partial_and_terminal_drop_acknowledgements() {+ for ack in [+ json!({"accepted": 3}),+ json!({"accepted": 1, "accepted_indices": [0], "failed_index": 1,+ "results": [{"index": 0, "outcome": "stored"}]}),+ json!({"accepted": 3, "results": [+ {"index": 0, "outcome": "stored"},+ {"index": 1, "outcome": "dropped_collision"},+ {"index": 2, "outcome": "ignored_end"}+ ]}),+ ] {+ let home = tempfile::tempdir().unwrap();+ let project = tempfile::tempdir().unwrap();+ let cwd = std::fs::canonicalize(project.path()).unwrap();+ let transcripts = home+ .path()+ .join(".claude/projects")+ .join(cwd.to_string_lossy().replace('/', "-"));+ std::fs::create_dir_all(&transcripts).unwrap();+ let sid = "11111111-2222-3333-4444-555555555555";+ crate::e2e_support::write_jsonl(+ &transcripts.join(format!("{sid}.jsonl")),+ &[+ json!({"sessionId": sid, "cwd": cwd}),+ json!({"type": "user", "message": {"role": "user", "content": [{"type":"text", "text":"fixture history"}]}}),+ ],+ );+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let addr = listener.local_addr().unwrap();+ let response = ack.clone();+ let app = Router::new().route(+ "/hook/batch",+ post(move || {+ let response = response.clone();+ async move { Json(response) }+ }),+ );+ let server = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });+ let data = tempfile::tempdir().unwrap();+ let output = tokio::process::Command::from(crate::e2e_support::hermetic(env!(+ "CARGO_BIN_EXE_ai-memory"+ )))+ .current_dir(&cwd)+ .env("AI_MEMORY_HOME", home.path())+ .env("AI_MEMORY_DATA_DIR", data.path())+ .env("AI_MEMORY_SERVER_URL", format!("http://{addr}"))+ .args([+ "backfill",+ "--workspace",+ "review",+ "--project",+ "fixture",+ "--force",+ ])+ .output()+ .await+ .unwrap();+ assert!(!output.status.success(), "ack was accepted: {ack}");+ server.abort();+ }+}+#[tokio::test]async fn import_failures_are_errors_in_human_json_and_quiet_modes() {for partial in [false, true] {@@ -32,7 +95,13 @@ async fn import_failures_are_errors_in_human_json_and_quiet_modes() {"/hook/batch",post(move |Json(items): Json<Vec<Value>>| async move {if partial && items[0]["url"].as_str().unwrap().contains("99999999") {- (StatusCode::OK, Json(json!({"accepted": items.len()})))+ let results = (0..items.len())+ .map(|index| json!({"index": index, "outcome": "stored"}))+ .collect::<Vec<_>>();+ (+ StatusCode::OK,+ Json(json!({"accepted": items.len(), "results": results})),+ )} else {(StatusCode::INTERNAL_SERVER_ERROR,diff --git a/crates/ai-memory-cli/tests/suite/run_offline.rs b/crates/ai-memory-cli/tests/suite/run_offline.rsindex 6213609c..1a0fce7a 100644--- a/crates/ai-memory-cli/tests/suite/run_offline.rs+++ b/crates/ai-memory-cli/tests/suite/run_offline.rs@@ -8,14 +8,25 @@#![cfg(unix)]use std::fs;+use std::io::Write as _;use std::os::unix::fs::PermissionsExt as _;use std::path::{Path, PathBuf};use std::process::{Output, Stdio};+use std::sync::{Arc, Mutex};-use axum::{Json, Router, http::StatusCode, routing::post};-use serde_json::json;+use axum::{+ Json, Router,+ extract::State,+ http::StatusCode,+ routing::{get, post},+};+use serde_json::{Value, json};const BIN: &str = env!("CARGO_BIN_EXE_ai-memory");+type RecoveryServerState = (+ Arc<Mutex<Vec<Vec<Value>>>>,+ Arc<Mutex<Vec<(String, Value)>>>,+);/// A loopback address nothing listens on: `free_port` binds and releases.fn closed_port_url() -> String {@@ -81,7 +92,7 @@ impl Fixture {"printf '%s\\n' \"$@\" > '{ran}'\nset > '{env_}'\n\while [ $# -gt 0 ]; do\n\if [ \"$1\" = --session-id ]; then\n\- printf '{{\"sessionId\":\"%s\",\"cwd\":\"%s\"}}\\n' \"$2\" '{repo}' > '{dir}'/\"$2\".jsonl\n\+ printf '{{\"sessionId\":\"%s\",\"cwd\":\"%s\"}}\\n{{\"type\":\"user\",\"message\":{{\"role\":\"user\",\"content\":[{{\"type\":\"text\",\"text\":\"offline exact prompt\"}}]}}}}\\n{{\"type\":\"assistant\",\"message\":{{\"role\":\"assistant\",\"content\":[{{\"type\":\"text\",\"text\":\"offline exact reply\"}}]}}}}\\n' \"$2\" '{repo}' > '{dir}'/\"$2\".jsonl\n\fi\nshift\ndone\nexit 0\n",ran = claude_ran.display(),env_ = claude_env.display(),@@ -131,10 +142,131 @@ async fn run(fixture: &Fixture, server: &str, args: &[&str]) -> Output {.expect("spawn ai-memory")}-fn stderr(output: &Output) -> String {+async fn recover(fixture: &Fixture, server: &str) -> Output {+ run(fixture, server, &["recover", "--json"]).await+}++fn journal_entries(fixture: &Fixture) -> Vec<Value> {+ let journal_path = fixture.home.join("data/recovery-journal.json");+ let journal: Value = serde_json::from_slice(&fs::read(journal_path).expect("recovery journal"))+ .expect("recovery journal JSON");+ journal["entries"]+ .as_array()+ .expect("journal entries")+ .clone()+}++fn stderr_text(output: &Output) -> String {String::from_utf8_lossy(&output.stderr).into_owned()}+/// Re-stamp the newest spool entry as a hook that fired inside the journal+/// interval `[0].interval_started_ms ..= interval_ended_ms`. Real harness+/// hooks spool while the child runs; the fake harness has none, so the test+/// enqueues after the fact and back-dates both the body and the file name+/// (whose leading milliseconds the tombstone cleaner trusts).+fn backdate_latest_spool_entry(fixture: &Fixture, ms: u64) {+ let spool = fixture.home.join("data/hook-spool");+ let mut newest: Option<(std::path::PathBuf, String)> = None;+ for entry in fs::read_dir(&spool).unwrap() {+ let path = entry.unwrap().path();+ if path.extension().and_then(|value| value.to_str()) != Some("json") {+ continue;+ }+ let name = path+ .file_name()+ .and_then(|name| name.to_str())+ .unwrap()+ .to_string();+ if newest.as_ref().is_none_or(|(_, other)| &name > other) {+ newest = Some((path, name));+ }+ }+ let Some((path, name)) = newest else {+ panic!("no spool entry to back-date");+ };+ let mut value: Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap();+ value["created_ms"] = json!(ms);+ fs::write(&path, serde_json::to_vec(&value).unwrap()).unwrap();+ let suffix = name.split_once('-').expect("spool name embeds its stamp").1;+ fs::rename(&path, spool.join(format!("{:013}-{suffix}", ms))).unwrap();+}++fn enqueue_hook(fixture: &Fixture, server: &str, session_id: &str, marker: &str) {+ let mut child = crate::e2e_support::hermetic(BIN)+ .args([+ "--data-dir",+ fixture.home.join("data").to_str().unwrap(),+ "hook",+ "--event",+ "user-prompt-submit",+ "--agent",+ "claude-code",+ "--server-url",+ server,+ ])+ .current_dir(&fixture.repo)+ .stdin(Stdio::piped())+ .stdout(Stdio::piped())+ .stderr(Stdio::piped())+ .spawn()+ .unwrap();+ child+ .stdin+ .take()+ .unwrap()+ .write_all(+ json!({+ "session_id": session_id,+ "cwd": fixture.repo,+ "prompt": marker,+ })+ .to_string()+ .as_bytes(),+ )+ .unwrap();+ let output = child.wait_with_output().unwrap();+ assert!(output.status.success(), "{}", stderr_text(&output));+}++async fn fake_recovery_server_on(+ listener: tokio::net::TcpListener,+ batches: Arc<Mutex<Vec<Vec<Value>>>>,+ finishes: Arc<Mutex<Vec<(String, Value)>>>,+) -> tokio::task::JoinHandle<()> {+ async fn health() -> StatusCode {+ StatusCode::OK+ }+ async fn batch(+ State((batches, _)): State<RecoveryServerState>,+ Json(items): Json<Vec<Value>>,+ ) -> Json<Value> {+ let accepted = items.len();+ let results = (0..accepted)+ .map(|index| json!({"index": index, "outcome": "stored"}))+ .collect::<Vec<_>>();+ batches.lock().unwrap().push(items);+ Json(json!({ "accepted": accepted, "results": results }))+ }+ async fn finish(+ State((_, finishes)): State<RecoveryServerState>,+ axum::extract::Path(run_id): axum::extract::Path<String>,+ Json(body): Json<Value>,+ ) -> Json<Value> {+ finishes.lock().unwrap().push((run_id, body));+ Json(json!({ "imported_events": 1, "latest_sequence": 1 }))+ }++ let state = (batches, finishes);+ let app = Router::new()+ .route("/healthz", get(health))+ .route("/hook/batch", post(batch))+ .route("/workstream/runs/{run_id}/finish", post(finish))+ .route("/workstream/runs/{run_id}/recover/finish", post(finish))+ .with_state(state);+ tokio::spawn(async move { axum::serve(listener, app).await.unwrap() })+}+/// A closed port downgrades the launch: one loud warning, the harness runs,/// no run/workstream ids are attributed to the offline child, and the/// not-recorded line accounts for the (empty) local spool.@@ -143,27 +275,48 @@ async fn offline_run_launches_the_harness_without_the_server() {let fixture = Fixture::new();let server = closed_port_url();let output = run(&fixture, &server, &["run", "--no-autowire", "claude"]).await;- assert!(output.status.success(), "{}", stderr(&output));+ assert!(output.status.success(), "{}", stderr_text(&output));- let stderr = stderr(&output);+ let output_stderr = stderr_text(&output);assert!(- stderr.contains("WARNING") && stderr.contains("unreachable"),- "the degraded warning is loud:\n{stderr}"+ output_stderr.contains("WARNING") && output_stderr.contains("unreachable"),+ "the degraded warning is loud:\n{output_stderr}");assert!(- stderr.contains(&server),- "the warning names the server URL:\n{stderr}"+ output_stderr.contains(&server),+ "the warning names the server URL:\n{output_stderr}");assert!(- stderr.contains("not recorded on the server"),- "the spool account is printed:\n{stderr}"+ output_stderr.contains("not recorded on the server"),+ "the spool account is printed:\n{output_stderr}");assert!(- stderr.contains("no hook events remain spooled locally"),- "the empty spool is reported as empty:\n{stderr}"+ output_stderr.contains("no hook events remain spooled locally"),+ "the empty spool is reported as empty:\n{output_stderr}"+ );++ assert!(+ fixture.claude_ran.exists(),+ "the harness ran:\n{output_stderr}");+ let entries = journal_entries(&fixture);+ assert_eq!(+ entries.len(),+ 1,+ "one offline run is journaled: {entries:?}"+ );+ assert_eq!(entries[0]["kind"], "degraded-run");+ assert_eq!(entries[0]["workspace"], "default");+ assert!(entries[0]["native_session_id"].as_str().is_some());+ let journal_text = serde_json::to_string(&entries).unwrap();+ assert!(!journal_text.contains("offline exact prompt"));+ let pending = recover(&fixture, &server).await;+ assert!(pending.status.success(), "{}", stderr_text(&pending));+ let report: Value = serde_json::from_slice(&pending.stdout).expect("recover JSON");+ assert_eq!(report["reachable"], false);+ assert_eq!(report["journal_remaining"], 1);+ assert_eq!(journal_entries(&fixture).len(), 1);- assert!(fixture.claude_ran.exists(), "the harness ran:\n{stderr}");let child_env = fs::read_to_string(&fixture.claude_env).unwrap();assert!(!child_env@@ -180,6 +333,203 @@ async fn offline_run_launches_the_harness_without_the_server() {);}+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn unsupported_exporter_is_journaled_by_a_real_degraded_run_as_spool_only() {+ let fixture = Fixture::new();+ let conversation = "12345678-1234-4234-9234-123456789abc";+ let store = fixture.home.join(".gemini/antigravity-cli/conversations");+ fs::create_dir_all(&store).unwrap();+ let db = store.join(format!("{conversation}.db"));+ let connection = rusqlite::Connection::open(&db).unwrap();+ connection+ .execute_batch("CREATE TABLE trajectory (trajectory_metadata_blob BLOB NOT NULL);")+ .unwrap();+ let uri = format!("file://{}", fixture.repo.display());+ let mut nested = Vec::new();+ nested.push(10);+ nested.push(u8::try_from(uri.len()).unwrap());+ nested.extend_from_slice(uri.as_bytes());+ let mut metadata = vec![10, u8::try_from(nested.len()).unwrap()];+ metadata.extend_from_slice(&nested);+ connection+ .execute(+ "INSERT INTO trajectory (trajectory_metadata_blob) VALUES (?1)",+ rusqlite::params![metadata],+ )+ .unwrap();+ drop(connection);+ write_script(&fixture.bin.join("agy"), "exit 0\n");++ let server = closed_port_url();+ let output = run(+ &fixture,+ &server,+ &[+ "run",+ "--no-autowire",+ "antigravity",+ "--conversation",+ conversation,+ ],+ )+ .await;+ assert!(output.status.success(), "{}", stderr_text(&output));+ let entries = journal_entries(&fixture);+ assert_eq!(entries.len(), 1, "real degraded run must be journaled");+ assert_eq!(entries[0]["harness"], "antigravity");+ assert_eq!(entries[0]["native_session_id"], conversation);+ assert_eq!(entries[0]["spool_only"], true);+ assert!(entries[0].get("event_digests").is_none());+ assert!(entries[0].get("final_cursor").is_none());+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn recovery_refuses_same_event_ids_with_mutated_content() {+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let server_url = format!("http://{}", listener.local_addr().unwrap());+ let fixture = Fixture::new();+ let output = run(&fixture, &server_url, &["run", "--no-autowire", "claude"]).await;+ assert!(output.status.success(), "{}", stderr_text(&output));+ let entries = journal_entries(&fixture);+ let session_id = entries[0]["native_session_id"].as_str().unwrap();+ let transcript = fixture+ .home+ .join(".claude/projects")+ .join(fixture.repo.to_string_lossy().replace('/', "-"))+ .join(format!("{session_id}.jsonl"));+ let original = fs::read_to_string(&transcript).unwrap();+ fs::write(+ &transcript,+ original.replace("offline exact reply", "mutated exact reply"),+ )+ .unwrap();++ let batches = Arc::new(Mutex::new(Vec::new()));+ let finishes = Arc::new(Mutex::new(Vec::new()));+ let server = fake_recovery_server_on(listener, batches.clone(), finishes).await;+ let recovered = recover(&fixture, &server_url).await;+ assert!(+ !recovered.status.success(),+ "semantic mutation must refuse replay"+ );+ let report: Value = serde_json::from_slice(&recovered.stdout).unwrap();+ assert_eq!(report["journal_remaining"], 1);+ assert!(batches.lock().unwrap().is_empty());+ server.abort();+}++#[tokio::test(flavor = "multi_thread", worker_threads = 2)]+async fn recover_drains_spool_then_replays_exact_degraded_transcript_idempotently() {+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let server_url = format!("http://{}", listener.local_addr().unwrap());+ let fixture = Fixture::new();+ let output = run(&fixture, &server_url, &["run", "--no-autowire", "claude"]).await;+ assert!(output.status.success(), "{}", stderr_text(&output));+ let entries = journal_entries(&fixture);+ let session_id = entries[0]["native_session_id"]+ .as_str()+ .unwrap()+ .to_string();+ enqueue_hook(&fixture, &server_url, &session_id, "offline exact prompt");+ backdate_latest_spool_entry(+ &fixture,+ entries[0]["interval_ended_ms"]+ .as_u64()+ .expect("interval end"),+ );+ let transcript = fixture+ .home+ .join(".claude/projects")+ .join(fixture.repo.to_string_lossy().replace('/', "-"))+ .join(format!("{session_id}.jsonl"));+ let mut transcript_file = fs::OpenOptions::new()+ .append(true)+ .open(transcript)+ .unwrap();+ writeln!(+ transcript_file,+ "{}",+ json!({"type":"user","message":{"role":"user","content":[{"type":"text","text":"later resumed work"}]}})+ )+ .unwrap();++ let batches = Arc::new(Mutex::new(Vec::new()));+ let finishes = Arc::new(Mutex::new(Vec::new()));+ let server = fake_recovery_server_on(listener, batches.clone(), finishes.clone()).await;+ let recovered = recover(&fixture, &server_url).await;+ assert!(recovered.status.success(), "{}", stderr_text(&recovered));+ let report: Value = serde_json::from_slice(&recovered.stdout).expect("recover JSON");+ assert_eq!(report["spool"]["sent"], 0);+ assert_eq!(report["journal_remaining"], 0);+ assert!(journal_entries(&fixture).is_empty());++ let requests = batches.lock().unwrap().clone();+ assert_eq!(+ requests.len(),+ 1,+ "the correlated spool entry must be quarantined and replaced by the transcript"+ );+ let replay = &requests[0];+ assert_eq!(replay.len(), 4, "start, exact user/reply, end: {replay:?}");+ assert!(+ replay[0]["url"]+ .as_str()+ .unwrap()+ .contains("event=session-start")+ );+ assert_eq!(replay[1]["body"]["prompt"], "offline exact prompt");+ assert_eq!(replay[2]["body"]["message"], "offline exact reply");+ assert!(+ replay.iter().all(|item| item["body"]+ .to_string()+ .find("later resumed work")+ .is_none()),+ "post-run resumed work must stay outside the captured upper bound: {replay:?}"+ );+ assert!(+ replay[3]["url"]+ .as_str()+ .unwrap()+ .contains("event=session-end")+ );+ assert!(replay.iter().all(|item| {+ item["url"]+ .as_str()+ .unwrap()+ .contains(&format!("session_id={session_id}"))+ }));+ assert!(finishes.lock().unwrap().is_empty());+ let recovery_root = fixture.home.join("data/hook-spool/.recovery");+ let recovery_dirs = fs::read_dir(&recovery_root)+ .map(|entries| {+ entries+ .map(|entry| entry.unwrap().path())+ .collect::<Vec<_>>()+ })+ .unwrap_or_default();+ assert_eq!(recovery_dirs.len(), 1, "a bounded tombstone remains");+ assert!(recovery_dirs[0].join(".completed").is_file());+ assert_eq!(+ fs::read_dir(&recovery_dirs[0])+ .unwrap()+ .filter(|entry| entry+ .as_ref()+ .unwrap()+ .path()+ .extension()+ .is_some_and(|ext| ext == "json"))+ .count(),+ 0,+ "confirmed replay deletes quarantined overlap but retains its tombstone"+ );++ let before = requests.len();+ let rerun = recover(&fixture, &server_url).await;+ assert!(rerun.status.success(), "{}", stderr_text(&rerun));+ assert_eq!(batches.lock().unwrap().len(), before);+ server.abort();+}+/// `--require-server` restores the fail-closed behavior: the old augmented/// connect error (including "the agent was not started"), and the harness/// never starts.@@ -194,18 +544,18 @@ async fn require_server_fails_closed_when_offline() {).await;assert!(!output.status.success());- let stderr = stderr(&output);+ let output_stderr = stderr_text(&output);assert!(- stderr.contains("the agent was not started"),- "the prepare context is preserved:\n{stderr}"+ output_stderr.contains("the agent was not started"),+ "the prepare context is preserved:\n{output_stderr}");assert!(- stderr.contains("could not reach"),- "the augmented connect diagnosis is reused:\n{stderr}"+ output_stderr.contains("could not reach"),+ "the augmented connect diagnosis is reused:\n{output_stderr}");assert!(!fixture.claude_ran.exists(),- "the harness must not run:\n{stderr}"+ "the harness must not run:\n{output_stderr}");}@@ -217,15 +567,15 @@ async fn offline_autowire_wires_hooks_but_not_mcp() {let fixture = Fixture::new();let server = closed_port_url();let output = run(&fixture, &server, &["run", "claude"]).await;- assert!(output.status.success(), "{}", stderr(&output));- let stderr = stderr(&output);+ assert!(output.status.success(), "{}", stderr_text(&output));+ let output_stderr = stderr_text(&output);assert!(- stderr.contains("wiring its ai-memory hooks"),- "the offline wiring is announced:\n{stderr}"+ output_stderr.contains("wiring its ai-memory hooks"),+ "the offline wiring is announced:\n{output_stderr}");assert!(- stderr.contains("MCP registration waits"),- "the deferred MCP is announced:\n{stderr}"+ output_stderr.contains("MCP registration waits"),+ "the deferred MCP is announced:\n{output_stderr}");let settings = fs::read_to_string(fixture.home.join(".claude/settings.json"))@@ -255,8 +605,8 @@ async fn offline_autowire_wires_hooks_but_not_mcp() {}/// The mid-run outage repair path: when the server dies after prepare, the-/// child's exit code survives, the warning points at `finalize-session`, and-/// the recorded exit status stays the child's own. The mock server answers+/// child's exit code survives, the warning points at `recover`, and the+/// original run id stays journaled without repository-state details. The mock server answers/// prepare, then closes its listener once the run is linked.#[tokio::test(flavor = "multi_thread", worker_threads = 2)]async fn a_mid_run_outage_preserves_the_child_exit_code() {@@ -305,8 +655,18 @@ async fn a_mid_run_outage_preserves_the_child_exit_code() {write_script(&fixture.bin.join("claude"),&format!(- "printf '%s\\n' \"$@\" > '{}'\nsleep 1\nexit 5\n",- fixture.claude_ran.display(),+ "printf '%s\\n' \"$@\" > '{ran}'\n\+ while [ $# -gt 0 ]; do\n\+ if [ \"$1\" = --session-id ]; then\n\+ printf '{{\"sessionId\":\"%s\",\"cwd\":\"%s\"}}\\n{{\"type\":\"user\",\"message\":{{\"role\":\"user\",\"content\":[{{\"type\":\"text\",\"text\":\"finish recovery prompt\"}}]}}}}\\n' \"$2\" '{repo}' > '{dir}'/\"$2\".jsonl\n\+ fi\nshift\ndone\nsleep 1\nexit 5\n",+ ran = fixture.claude_ran.display(),+ repo = fixture.repo.display(),+ dir = fixture+ .home+ .join(".claude/projects")+ .join(fixture.repo.to_string_lossy().replace('/', "-"))+ .display(),),);let server_url = format!("http://{address}");@@ -315,16 +675,53 @@ async fn a_mid_run_outage_preserves_the_child_exit_code() {output.status.code(),Some(5),"the child's exit code is preserved, not an ai-memory error:\n{}",- stderr(&output)+ stderr_text(&output));- let stderr = stderr(&output);+ let output_stderr = stderr_text(&output);assert!(- stderr.contains("was not imported"),- "the outage is reported:\n{stderr}"+ output_stderr.contains("was not imported"),+ "the outage is reported:\n{output_stderr}");assert!(- stderr.contains("ai-memory finalize-session"),- "the repair path is named:\n{stderr}"+ output_stderr.contains("ai-memory recover"),+ "the automatic repair path is named:\n{output_stderr}");- server.abort();+ let entries = journal_entries(&fixture);+ assert_eq!(entries.len(), 1);+ assert_eq!(entries[0]["kind"], "finish-failed");+ assert_eq!(entries[0]["run_id"], "12345678-1234-4234-9234-123456789abe");+ assert_eq!(entries[0]["exit_code"], 5);+ assert!(entries[0].get("checkpoint").is_none());+ assert!(entries[0].get("changed_paths").is_none());++ server.await.unwrap();+ let listener = tokio::net::TcpListener::bind(address).await.unwrap();+ let batches = Arc::new(Mutex::new(Vec::new()));+ let finishes = Arc::new(Mutex::new(Vec::new()));+ let recovery_server =+ fake_recovery_server_on(listener, batches.clone(), finishes.clone()).await;+ let recovered = recover(&fixture, &server_url).await;+ assert!(+ recovered.status.success(),+ "stdout: {}\nstderr: {}",+ String::from_utf8_lossy(&recovered.stdout),+ stderr_text(&recovered)+ );+ assert!(journal_entries(&fixture).is_empty());+ assert!(batches.lock().unwrap().is_empty());+ let finish_requests = finishes.lock().unwrap().clone();+ assert_eq!(finish_requests.len(), 1);+ assert_eq!(finish_requests[0].0, "12345678-1234-4234-9234-123456789abe");+ assert_eq!(finish_requests[0].1["complete"], true);+ assert_eq!(finish_requests[0].1["exit_code"], 5);+ assert!(finish_requests[0].1["checkpoint"].is_object());+ assert_eq!(+ finish_requests[0].1["events"][0]["content"],+ "finish recovery prompt"+ );++ let rerun = recover(&fixture, &server_url).await;+ assert!(rerun.status.success(), "{}", stderr_text(&rerun));+ assert_eq!(finishes.lock().unwrap().len(), 1);+ recovery_server.abort();}diff --git a/crates/ai-memory-hooks/src/lib.rs b/crates/ai-memory-hooks/src/lib.rsindex 9383a252..cc19cf15 100644--- a/crates/ai-memory-hooks/src/lib.rs+++ b/crates/ai-memory-hooks/src/lib.rs@@ -52,6 +52,7 @@ pub use capture_policy::{pub use payload::{HookEnvelope, HookEvent, NOTIFICATION_EXCERPT_MAX_BYTES, POST_COMPACTION_EXCERPT_MAX_BYTES,USER_PROMPT_EXCERPT_MAX_BYTES, agent_from_payload, cap_lifecycle_body_for_client,+ hook_payload_context,};pub use router::{DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, DEFAULT_INGEST_GATE_MAX_ENTRIES,diff --git a/crates/ai-memory-hooks/src/payload.rs b/crates/ai-memory-hooks/src/payload.rsindex 5703c963..c7865288 100644--- a/crates/ai-memory-hooks/src/payload.rs+++ b/crates/ai-memory-hooks/src/payload.rs@@ -500,6 +500,62 @@ pub fn agent_from_payload(raw: &serde_json::Value) -> Option<AgentKind> {extract_string(raw, &["cursor_version"]).map(|_| AgentKind::Cursor)}+/// Session and checkout fields extracted from the canonical hook payload shapes.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct HookPayloadContext {+ /// Harness-native session identity.+ pub session_id: Option<String>,+ /// Current checkout directory.+ pub cwd: Option<String>,+}++/// Extract the session and checkout exactly as [`HookEnvelope`] does.+#[must_use]+pub fn hook_payload_context(raw: &serde_json::Value) -> HookPayloadContext {+ let session_id = extract_string(+ raw,+ &[+ "session_id",+ "sessionId",+ "sessionID",+ "session",+ "conversationId",+ ],+ )+ .or_else(|| {+ extract_string_path(+ raw,+ &[+ &["info", "id"],+ &["properties", "sessionID"],+ &["properties", "info", "id"],+ &["event", "properties", "sessionID"],+ &["event", "properties", "info", "id"],+ &["payload", "info", "id"],+ &["payload", "properties", "sessionID"],+ &["payload", "properties", "info", "id"],+ ],+ )+ });+ let cwd = extract_string(raw, &["cwd", "current_dir", "working_dir", "directory"])+ .or_else(|| extract_first_string_array_item(raw, &["workspacePaths", "workspace_roots"]))+ .or_else(|| {+ extract_string_path(+ raw,+ &[+ &["path", "cwd"],+ &["info", "directory"],+ &["properties", "info", "directory"],+ &["event", "properties", "info", "directory"],+ &["payload", "path", "cwd"],+ &["payload", "info", "directory"],+ &["payload", "properties", "info", "directory"],+ ],+ )+ });+ HookPayloadContext { session_id, cwd }+}+impl HookEnvelope {/// Build an envelope from the parsed query + the body JSON. Performs/// best-effort extraction of `session_id` / `cwd` / a body excerpt@@ -510,66 +566,14 @@ impl HookEnvelope {let event = HookEvent::parse(&query.event);let agent = agent_from_payload(&raw).unwrap_or_else(|| query.agent.as_deref().map_or(AgentKind::Other, parse_agent));- // OpenCode's plugin SDK sends `sessionID` (capital `ID`) on the- // tool.execute.*/session.* events; Claude Code and native Codex use- // `session_id`, older Codex bridges `sessionId`, and Antigravity CLI- // uses `conversationId`.- // JSON keys are case-sensitive, so all spellings must be listed- // or tool events fail the router's "missing session_id" check.- let body_session_id = extract_string(- &raw,- &[- "session_id",- "sessionId",- "sessionID",- "session",- "conversationId",- ],- )- .or_else(|| {- extract_string_path(- &raw,- &[- &["info", "id"],- &["properties", "sessionID"],- &["properties", "info", "id"],- &["event", "properties", "sessionID"],- &["event", "properties", "info", "id"],- &["payload", "info", "id"],- &["payload", "properties", "sessionID"],- &["payload", "properties", "info", "id"],- ],- )- });- let session_id = body_session_id.or_else(|| query.session_id.filter(|s| !s.is_empty()));- // Cursor spells the workspace directory `workspace_roots` (an array,- // normally one entry; multi-root workspaces carry several) and never- // sends a usable top-level `cwd`: its `sessionStart` / `sessionEnd`- // payloads omit `cwd` entirely, and its tool events send `cwd: ""`.- // Without this spelling every Cursor session resolved to no cwd at all- // and landed in the server-default `default/scratch` bucket.- let body_cwd = extract_string(&raw, &["cwd", "current_dir", "working_dir", "directory"])- .or_else(|| {- extract_first_string_array_item(&raw, &["workspacePaths", "workspace_roots"])- })- .or_else(|| {- extract_string_path(- &raw,- &[- &["path", "cwd"],- &["info", "directory"],- &["properties", "info", "directory"],- &["event", "properties", "info", "directory"],- &["payload", "path", "cwd"],- &["payload", "info", "directory"],- &["payload", "properties", "info", "directory"],- ],- )- });+ let context = hook_payload_context(&raw);+ let session_id = context+ .session_id+ .or_else(|| query.session_id.filter(|s| !s.is_empty()));// Body cwd wins over the query-string fallback: the body is// what agent CLIs natively send, so any query-string `cwd` is// a bridge / test override that should defer to live data.- let cwd = body_cwd.or_else(|| query.cwd.filter(|s| !s.is_empty()));+ let cwd = context.cwd.or_else(|| query.cwd.filter(|s| !s.is_empty()));let workspace_override = query.workspace.filter(|s| !s.is_empty());let project_override = query.project.filter(|s| !s.is_empty());let project_strategy = ProjectStrategy::parse(query.project_strategy.as_deref());diff --git a/crates/ai-memory-hooks/src/workstream.rs b/crates/ai-memory-hooks/src/workstream.rsindex 3eefa347..227e5f87 100644--- a/crates/ai-memory-hooks/src/workstream.rs+++ b/crates/ai-memory-hooks/src/workstream.rs@@ -83,6 +83,10 @@ pub fn workstream_router(state: WorkstreamState) -> Router {).route("/workstream/runs/{run_id}/link", post(link_run)).route("/workstream/runs/{run_id}/finish", post(finish_run))+ .route(+ "/workstream/runs/{run_id}/recover/finish",+ post(recover_finish_run),+ ).route("/workstream/recent", post(list_recent_workstreams)).route("/workstream/rename", post(rename_workstream)).route("/workstream/{workstream_id}/events", get(search_events))@@ -887,6 +891,21 @@ async fn link_run(}}+async fn recover_finish_run(+ State(state): State<WorkstreamState>,+ level: Option<Extension<AuthLevel>>,+ viewer: Option<Extension<ai_memory_core::AuthorizedViewer>>,+ actor: Option<Extension<ai_memory_core::ActorContext>>,+ user_id: Option<Extension<ai_memory_core::UserId>>,+ AxumPath(raw_run_id): AxumPath<String>,+ Json(request): Json<FinishManagedRunRequest>,+) -> Response {+ finish_run_inner(+ state, level, viewer, actor, user_id, raw_run_id, request, true,+ )+ .await+}+async fn finish_run(State(state): State<WorkstreamState>,level: Option<Extension<AuthLevel>>,@@ -894,7 +913,24 @@ async fn finish_run(actor: Option<Extension<ai_memory_core::ActorContext>>,user_id: Option<Extension<ai_memory_core::UserId>>,AxumPath(raw_run_id): AxumPath<String>,- Json(mut request): Json<FinishManagedRunRequest>,+ Json(request): Json<FinishManagedRunRequest>,+) -> Response {+ finish_run_inner(+ state, level, viewer, actor, user_id, raw_run_id, request, false,+ )+ .await+}++#[allow(clippy::too_many_arguments)]+async fn finish_run_inner(+ state: WorkstreamState,+ level: Option<Extension<AuthLevel>>,+ viewer: Option<Extension<ai_memory_core::AuthorizedViewer>>,+ actor: Option<Extension<ai_memory_core::ActorContext>>,+ user_id: Option<Extension<ai_memory_core::UserId>>,+ raw_run_id: String,+ mut request: FinishManagedRunRequest,+ recover_expired: bool,) -> Response {if let Err(response) = authorize(level, Capability::NormalWrite) {return response.into_response();@@ -943,7 +979,7 @@ async fn finish_run(Ok(None) => return error(StatusCode::NOT_FOUND, "managed run not found"),Err(failure) => return error(StatusCode::INTERNAL_SERVER_ERROR, failure.to_string()),};- if status.state == "finished" {+ if status.state == "finished" && !recover_expired {return match state.writer.finish_workstream_run(@@ -969,8 +1005,10 @@ async fn finish_run(Err(failure) => store_error_response(failure),};}- if status.state != "active" {- return error(StatusCode::CONFLICT, "managed run is not active");+ if status.state != "active"+ && !(recover_expired && matches!(status.state.as_str(), "expired" | "finished"))+ {+ return error(StatusCode::CONFLICT, "managed run is not recoverable");}let native_identity = match request.native_session_id@@ -999,7 +1037,7 @@ async fn finish_run() {return error(StatusCode::BAD_REQUEST, message);}- if request.complete {+ if request.complete && status.state != "finished" {append_boundary_events(&state.sanitizer,run_id,@@ -1008,16 +1046,20 @@ async fn finish_run(&mut request,);}- let segment_path = match write_segment(- &state.data_dir,- status.workstream_id,- run_id,- &request.events,- ) {- Ok(path) => path,- Err(failure) => {- warn!(error = %failure, run = %run_id, "managed transcript segment write failed");- return error(StatusCode::INTERNAL_SERVER_ERROR, failure.to_string());+ let segment_path = if status.state == "finished" {+ None+ } else {+ match write_segment(+ &state.data_dir,+ status.workstream_id,+ run_id,+ &request.events,+ ) {+ Ok(path) => Some(path),+ Err(failure) => {+ warn!(error = %failure, run = %run_id, "managed transcript segment write failed");+ return error(StatusCode::INTERNAL_SERVER_ERROR, failure.to_string());+ }}};let input = FinishWorkstreamRun {@@ -1027,10 +1069,18 @@ async fn finish_run(source_cursor: request.source_cursor,events: request.events,complete: request.complete,- segment_path: Some(segment_path),+ segment_path,exit_code: request.exit_code,};- match state.writer.finish_workstream_run(authority, input).await {+ let result = if recover_expired {+ state+ .writer+ .recover_expired_workstream_run(authority, input)+ .await+ } else {+ state.writer.finish_workstream_run(authority, input).await+ };+ match result {Ok(result) => Json(FinishManagedRunResponse {imported_events: result.imported_events,latest_sequence: result.latest_sequence,@@ -4037,6 +4087,201 @@ mod tests {assert_eq!(finish_sql_snapshot(&store).await, before);}+ #[tokio::test]+ async fn expired_run_recovery_is_owner_only_and_refuses_superseded_runs() {+ let temp = TempDir::new().unwrap();+ let store = Store::open(temp.path()).unwrap();+ let mut state = test_state(&store, temp.path());+ state.trusted_proxy_identity = true;+ let (ws, project) = seed_scope(&store).await;+ let prepare = |name: &str| PrepareWorkstreamRun {+ workspace_id: ws,+ project_id: project,+ repo_fingerprint: name.into(),+ worktree_fingerprint: name.into(),+ cwd: "/repo".into(),+ agent: AgentKind::Codex,+ automatic_harness: false,+ available_agents: Vec::new(),+ selection: WorkstreamSelection::New(name.into()),+ lease_owner: "launcher".into(),+ };+ let alice_actor = ai_memory_core::ActorContext {+ user: Some("alice".into()),+ ..ai_memory_core::ActorContext::anonymous()+ };+ let bob_actor = ai_memory_core::ActorContext {+ user: Some("bob".into()),+ ..ai_memory_core::ActorContext::anonymous()+ };+ let recoverable = store+ .writer+ .prepare_workstream_run_owned(prepare("recoverable"), Some("user:alice".into()))+ .await+ .unwrap();+ store+ .writer+ .cancel_managed_run(recoverable.run_id)+ .await+ .unwrap();+ let request = || FinishManagedRunRequest {+ native_session_id: Some("native-recovery".into()),+ source_cursor: None,+ events: vec![NewWorkstreamEvent {+ event_id: "recovered-event".into(),+ agent: AgentKind::Codex,+ native_session_id: "native-recovery".into(),+ source_record_id: None,+ kind: WorkstreamEventKind::Message,+ role: Some("user".into()),+ content: "recovered".into(),+ occurred_at: None,+ metadata: serde_json::Value::Null,+ }],+ complete: true,+ checkpoint: Default::default(),+ losses: Vec::new(),+ exit_code: Some(0),+ };+ let refused = recover_finish_run(+ State(state.clone()),+ Some(Extension(AuthLevel::User)),+ None,+ Some(Extension(bob_actor)),+ None,+ AxumPath(recoverable.run_id.to_string()),+ Json(request()),+ )+ .await;+ assert_eq!(refused.status(), StatusCode::FORBIDDEN);+ let accepted = recover_finish_run(+ State(state.clone()),+ Some(Extension(AuthLevel::User)),+ None,+ Some(Extension(alice_actor.clone())),+ None,+ AxumPath(recoverable.run_id.to_string()),+ Json(request()),+ )+ .await;+ assert_eq!(accepted.status(), StatusCode::OK);+ let exact_retry = recover_finish_run(+ State(state.clone()),+ Some(Extension(AuthLevel::User)),+ None,+ Some(Extension(alice_actor.clone())),+ None,+ AxumPath(recoverable.run_id.to_string()),+ Json(request()),+ )+ .await;+ let exact_status = exact_retry.status();+ let exact_body = to_bytes(exact_retry.into_body(), usize::MAX).await.unwrap();+ assert_eq!(+ exact_status,+ StatusCode::OK,+ "{}",+ String::from_utf8_lossy(&exact_body)+ );+ let changed_checkpoint = recover_finish_run(+ State(state.clone()),+ Some(Extension(AuthLevel::User)),+ None,+ Some(Extension(alice_actor.clone())),+ None,+ AxumPath(recoverable.run_id.to_string()),+ Json(FinishManagedRunRequest {+ checkpoint: ai_memory_core::WorkstreamCheckpoint {+ head: Some("repository-changed-after-finish".into()),+ ..Default::default()+ },+ ..request()+ }),+ )+ .await;+ assert_eq!(changed_checkpoint.status(), StatusCode::OK);+ assert_eq!(+ store+ .reader+ .search_workstream_events(+ recoverable.workstream_id,+ "".into(),+ 100,+ Sanitizer::default(),+ )+ .await+ .unwrap()+ .len(),+ 2,+ "a finished retry must not append a regenerated checkpoint"+ );+ for changed in [+ FinishManagedRunRequest {+ native_session_id: Some("changed-native".into()),+ ..request()+ },+ FinishManagedRunRequest {+ events: {+ let mut events = request().events;+ events[0].event_id = "losing-event".into();+ events+ },+ ..request()+ },+ FinishManagedRunRequest {+ source_cursor: Some("uncommitted-cursor".into()),+ ..request()+ },+ FinishManagedRunRequest {+ exit_code: Some(9),+ ..request()+ },+ ] {+ let changed_finished = recover_finish_run(+ State(state.clone()),+ Some(Extension(AuthLevel::User)),+ None,+ Some(Extension(alice_actor.clone())),+ None,+ AxumPath(recoverable.run_id.to_string()),+ Json(changed),+ )+ .await;+ assert_eq!(changed_finished.status(), StatusCode::BAD_REQUEST);+ }++ let old = store+ .writer+ .prepare_workstream_run_owned(prepare("superseded"), Some("user:alice".into()))+ .await+ .unwrap();+ store.writer.cancel_managed_run(old.run_id).await.unwrap();+ store+ .writer+ .prepare_workstream_run_owned(+ PrepareWorkstreamRun {+ repo_fingerprint: "superseded".into(),+ worktree_fingerprint: "superseded".into(),+ selection: WorkstreamSelection::Named("superseded".into()),+ ..prepare("replacement")+ },+ Some("user:alice".into()),+ )+ .await+ .unwrap();+ let superseded = recover_finish_run(+ State(state),+ Some(Extension(AuthLevel::User)),+ None,+ Some(Extension(alice_actor)),+ None,+ AxumPath(old.run_id.to_string()),+ Json(request()),+ )+ .await;+ assert_eq!(superseded.status(), StatusCode::BAD_REQUEST);+ }+#[tokio::test]async fn native_identity_http_stored_identity_refuses_before_raw_segment() {let temp = TempDir::new().unwrap();diff --git a/crates/ai-memory-store/migrations/V77__recovery_ingest_keys.sql b/crates/ai-memory-store/migrations/V77__recovery_ingest_keys.sqlnew file mode 100644index 00000000..ec53de26--- /dev/null+++ b/crates/ai-memory-store/migrations/V77__recovery_ingest_keys.sql@@ -0,0 +1,7 @@+ALTER TABLE ingest_keys ADD COLUMN session_id BLOB REFERENCES sessions(id) ON DELETE CASCADE;+ALTER TABLE ingest_keys ADD COLUMN agent_kind TEXT;+ALTER TABLE ingest_keys ADD COLUMN observation_kind TEXT;+ALTER TABLE ingest_keys ADD COLUMN event_identity TEXT;+ALTER TABLE ingest_keys ADD COLUMN recovery_retained INTEGER NOT NULL DEFAULT 0 CHECK (recovery_retained IN (0, 1));++CREATE INDEX idx_ingest_keys_session ON ingest_keys (session_id);diff --git a/crates/ai-memory-store/src/api_credentials.rs b/crates/ai-memory-store/src/api_credentials.rsindex 471cdc6c..9fa0ba20 100644--- a/crates/ai-memory-store/src/api_credentials.rs+++ b/crates/ai-memory-store/src/api_credentials.rs@@ -411,7 +411,7 @@ mod tests {|row| row.get(0),).unwrap();- assert_eq!(version, 76, "update the pin when adding a migration");+ assert_eq!(version, 77, "update the pin when adding a migration");let cols: i64 = conn.query_row("SELECT COUNT(*) FROM pragma_table_info('users') WHERE name = 'token_hash'",diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rsindex f4e0429a..d6df5ec3 100644--- a/crates/ai-memory-store/src/lib.rs+++ b/crates/ai-memory-store/src/lib.rs@@ -3379,7 +3379,12 @@ mod tests {.unwrap();let session_a = SessionId::new();let session_b = SessionId::new();- for (session_id, project_id) in [(session_a, proj_a), (session_b, proj_b)] {+ let session_c = SessionId::new();+ for (session_id, project_id) in [+ (session_a, proj_a),+ (session_b, proj_b),+ (session_c, proj_a),+ ] {store.writer.begin_session(NewSession {@@ -3477,6 +3482,74 @@ mod tests {.await.unwrap();assert!(matches!(reused, IngestObservationOutcome::Inserted(_)));++ let recovery_key = format!("recovery_{}", "a".repeat(55));+ let recovery_first = store+ .writer+ .insert_observation_ingest(+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),+ recovery_key.clone(),+ )+ .await+ .unwrap();+ assert!(matches!(+ recovery_first,+ IngestObservationOutcome::Inserted(_)+ ));+ store+ .writer+ .complete_observation_ingest(proj_a, recovery_key.clone())+ .await+ .unwrap();+ let conn = Connection::open(store.db_path()).unwrap();+ conn.execute(+ "UPDATE ingest_keys SET seen_at = 1 WHERE project_id = ?1 AND key = ?2",+ params![proj_a.as_bytes(), recovery_key],+ )+ .unwrap();+ drop(conn);+ let recovery_replay = store+ .writer+ .insert_observation_ingest(+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),+ recovery_key.clone(),+ )+ .await+ .unwrap();+ assert_eq!(recovery_replay, IngestObservationOutcome::AlreadyComplete);+ assert!(matches!(+ store+ .writer+ .insert_observation_ingest(+ Sanitized::new(obs(session_c, proj_a), &Sanitizer::builtin()),+ recovery_key.clone(),+ )+ .await,+ Err(StoreError::SessionCollision)+ ));+ let mut changed = obs(session_a, proj_a);+ changed.body = "different event body".into();+ assert!(matches!(+ store+ .writer+ .insert_observation_ingest(+ Sanitized::new(changed, &Sanitizer::builtin()),+ recovery_key,+ )+ .await,+ Err(StoreError::SessionCollision)+ ));+ let conn = Connection::open(store.db_path()).unwrap();+ assert_eq!(+ conn.query_row(+ "SELECT COUNT(*) FROM observations WHERE project_id = ?1",+ params![proj_a.as_bytes()],+ |row| row.get::<_, i64>(0),+ )+ .unwrap(),+ 3,+ "a different session cannot preclaim or suppress a recovery key"+ );}#[tokio::test]diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rsindex d197ba15..4444e5ed 100644--- a/crates/ai-memory-store/src/ops.rs+++ b/crates/ai-memory-store/src/ops.rs@@ -2744,10 +2744,39 @@ pub fn insert_observation(}/// Ingest-keys older than this are swept opportunistically on every keyed-/// insert. Keys only need to outlive the client spool (7 days + retry-/// backoff); 30 days is a generous margin.+/// insert. Ordinary keys only need to outlive the client spool (7 days + retry+/// backoff); recovery keys remain tied to their stored session until it is purged.const INGEST_KEY_TTL_MICROS: i64 = 30 * 24 * 60 * 60 * 1_000_000;+fn recovery_ingest_key(key: &str) -> bool {+ key.strip_prefix("recovery_").is_some_and(|digest| {+ digest.len() == 55 && digest.bytes().all(|byte| byte.is_ascii_hexdigit())+ })+}++fn recovery_event_identity(obs: &NewObservation, agent: Option<&str>) -> String {+ let mut hasher = Sha256::new();+ let session_id = obs.session_id.to_string();+ for value in [+ Some(session_id.as_str()),+ agent,+ Some(obs.kind.as_str()),+ obs.extension.as_deref(),+ obs.source_event.as_deref(),+ Some(obs.title.as_str()),+ Some(obs.body.as_str()),+ ] {+ if let Some(value) = value {+ hasher.update(value.len().to_be_bytes());+ hasher.update(value.as_bytes());+ } else {+ hasher.update(usize::MAX.to_be_bytes());+ }+ }+ hasher.update(obs.occurred_at.unwrap_or_default().to_be_bytes());+ format!("{digest:x}", digest = hasher.finalize())+}+/// Claim a project-scoped ingest key and append its observation atomically.////// The key claim and the observation row commit in ONE transaction: either@@ -2766,18 +2795,48 @@ pub fn insert_observation_keyed(// Sweep before looking up the current key so an expired key can be reused// even when no unrelated keyed event arrived in the meantime.tx.execute(- "DELETE FROM ingest_keys WHERE seen_at < ?1",+ "DELETE FROM ingest_keys WHERE recovery_retained = 0 AND seen_at < ?1",params![now - INGEST_KEY_TTL_MICROS],)?;- let existing: Option<Option<i64>> = tx+ let recovery = recovery_ingest_key(ingest_key);+ let session_agent = if recovery {+ tx.query_row(+ "SELECT agent_kind FROM sessions WHERE id = ?1",+ params![obs.session_id.as_bytes()],+ |row| row.get::<_, String>(0),+ )+ .optional()?+ } else {+ None+ };+ let event_identity = recovery.then(|| recovery_event_identity(obs, session_agent.as_deref()));+ type IngestKeyRow = (+ Option<i64>,+ Option<Vec<u8>>,+ Option<String>,+ Option<String>,+ Option<String>,+ );+ if recovery && session_agent.is_none() {+ return Err(StoreError::SessionCollision);+ }+ let existing: Option<IngestKeyRow> = tx.query_row(- "SELECT completed_at FROM ingest_keys \+ "SELECT completed_at, session_id, agent_kind, observation_kind, event_identity FROM ingest_keys \WHERE project_id = ?1 AND key = ?2",params![obs.project_id.as_bytes(), ingest_key],- |row| row.get(0),+ |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?, row.get(4)?)),).optional()?;- if let Some(completed_at) = existing {+ if let Some((completed_at, session_id, agent, kind, identity)) = existing {+ if recovery+ && (session_id.as_deref() != Some(obs.session_id.as_bytes())+ || agent.as_deref() != session_agent.as_deref()+ || kind.as_deref() != Some(obs.kind.as_str())+ || identity.as_deref() != event_identity.as_deref())+ {+ return Err(StoreError::SessionCollision);+ }tx.commit()?;return Ok(if completed_at.is_some() {IngestObservationOutcome::AlreadyComplete@@ -2787,9 +2846,18 @@ pub fn insert_observation_keyed(}tx.execute(- "INSERT INTO ingest_keys (project_id, key, seen_at, completed_at) \- VALUES (?1, ?2, ?3, NULL)",- params![obs.project_id.as_bytes(), ingest_key, now],+ "INSERT INTO ingest_keys (project_id, key, seen_at, completed_at, session_id, agent_kind, observation_kind, event_identity, recovery_retained) \+ VALUES (?1, ?2, ?3, NULL, ?4, ?5, ?6, ?7, ?8)",+ params![+ obs.project_id.as_bytes(),+ ingest_key,+ now,+ recovery.then_some(obs.session_id.as_bytes()),+ session_agent,+ recovery.then_some(obs.kind.as_str()),+ event_identity,+ i64::from(recovery)+ ],)?;let id = insert_observation_row(&tx, obs)?;tx.commit()?;@@ -2945,18 +3013,36 @@ pub fn admit_hook_session_event(}let ingest = if let Some(key) = ingest_key {tx.execute(- "DELETE FROM ingest_keys WHERE seen_at < ?1",+ "DELETE FROM ingest_keys WHERE recovery_retained = 0 AND seen_at < ?1",params![now - INGEST_KEY_TTL_MICROS],)?;- let old: Option<Option<i64>> = tx+ let recovery = recovery_ingest_key(key);+ let event_identity =+ recovery.then(|| recovery_event_identity(obs, Some(session.agent_kind.as_str())));+ type IngestKeyRow = (+ Option<i64>,+ Option<Vec<u8>>,+ Option<String>,+ Option<String>,+ Option<String>,+ );+ let old: Option<IngestKeyRow> = tx.query_row(- "SELECT completed_at FROM ingest_keys WHERE project_id = ?1 AND key = ?2",+ "SELECT completed_at, session_id, agent_kind, observation_kind, event_identity FROM ingest_keys WHERE project_id = ?1 AND key = ?2",params![obs.project_id.as_bytes(), key],- |r| r.get(0),+ |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?, r.get(4)?)),).optional()?;match old {- Some(done) => {+ Some((done, keyed_session, keyed_agent, keyed_kind, keyed_identity)) => {+ if recovery+ && (keyed_session.as_deref() != Some(obs.session_id.as_bytes())+ || keyed_agent.as_deref() != Some(session.agent_kind.as_str())+ || keyed_kind.as_deref() != Some(obs.kind.as_str())+ || keyed_identity.as_deref() != event_identity.as_deref())+ {+ return Err(StoreError::SessionCollision);+ }if done.is_some() {IngestObservationOutcome::AlreadyComplete} else {@@ -2964,7 +3050,20 @@ pub fn admit_hook_session_event(}}None => {- tx.execute("INSERT INTO ingest_keys (project_id, key, seen_at, completed_at) VALUES (?1, ?2, ?3, NULL)", params![obs.project_id.as_bytes(), key, now])?;+ tx.execute(+ "INSERT INTO ingest_keys (project_id, key, seen_at, completed_at, session_id, agent_kind, observation_kind, event_identity, recovery_retained) \+ VALUES (?1, ?2, ?3, NULL, ?4, ?5, ?6, ?7, ?8)",+ params![+ obs.project_id.as_bytes(),+ key,+ now,+ recovery.then_some(obs.session_id.as_bytes()),+ recovery.then_some(session.agent_kind.as_str()),+ recovery.then_some(obs.kind.as_str()),+ event_identity,+ i64::from(recovery)+ ],+ )?;IngestObservationOutcome::Inserted(insert_observation_row(&tx, obs)?)}}@@ -8732,6 +8831,14 @@ pub(crate) mod tests {fn purge_session_removes_the_session_and_everything_derived_from_it() {let (_tmp, mut conn, ws, proj) = fresh_db();let (sid, _page) = seed_session(&mut conn, ws, proj, "target");+ let recovery_key = format!("recovery_{}", "a".repeat(55));+ conn.execute(+ "INSERT INTO ingest_keys \+ (project_id, key, seen_at, completed_at, session_id, recovery_retained) \+ VALUES (?1, ?2, 1, 1, ?3, 1)",+ params![proj.as_bytes(), recovery_key, sid.as_bytes()],+ )+ .unwrap();let summary = purge_session(&mut conn,@@ -8755,6 +8862,11 @@ pub(crate) mod tests {"observations=0");assert_eq!(count(&conn, "SELECT COUNT(*) FROM pages"), 0, "pages=0");+ assert_eq!(+ count(&conn, "SELECT COUNT(*) FROM ingest_keys"),+ 0,+ "session-bound recovery keys cascade with the purged session"+ );assert_eq!(summary.observations_deleted, 1);assert_eq!(summary.pages_deleted, 1);assert_eq!(diff --git a/crates/ai-memory-store/src/workstream.rs b/crates/ai-memory-store/src/workstream.rsindex 87227430..0834ec5d 100644--- a/crates/ai-memory-store/src/workstream.rs+++ b/crates/ai-memory-store/src/workstream.rs@@ -392,6 +392,7 @@ struct FinishRunRow {sync_after: i64,sync_through: i64,context_delivered: bool,+ exit_code: Option<i32>,}struct LinkRunRow {@@ -1053,6 +1054,23 @@ pub(crate) fn finish_run(conn: &mut Connection,authority: &ManagedRunAuthority,input: &FinishWorkstreamRun,+) -> StoreResult<FinishedWorkstreamRun> {+ finish_run_inner(conn, authority, input, false)+}++pub(crate) fn recover_run(+ conn: &mut Connection,+ authority: &ManagedRunAuthority,+ input: &FinishWorkstreamRun,+) -> StoreResult<FinishedWorkstreamRun> {+ finish_run_inner(conn, authority, input, true)+}++fn finish_run_inner(+ conn: &mut Connection,+ authority: &ManagedRunAuthority,+ input: &FinishWorkstreamRun,+ recover_expired: bool,) -> StoreResult<FinishedWorkstreamRun> {let now = Timestamp::now().as_microsecond();let tx = conn.transaction()?;@@ -1060,7 +1078,7 @@ pub(crate) fn finish_run(let run: Option<FinishRunRow> = tx.query_row("SELECT workstream_id, agent_kind, native_session_id, state, \- sync_after, sync_through, context_delivered \+ sync_after, sync_through, context_delivered, exit_code \FROM managed_runs WHERE id = ?1",params![input.run_id.as_bytes()],|row| {@@ -1072,6 +1090,7 @@ pub(crate) fn finish_run(sync_after: row.get(4)?,sync_through: row.get(5)?,context_delivered: row.get(6)?,+ exit_code: row.get(7)?,})},)@@ -1090,6 +1109,7 @@ pub(crate) fn finish_run(sync_after,sync_through,context_delivered,+ exit_code,} = run;let latest_before: i64 = tx.query_row("SELECT COALESCE(MAX(sequence), 0) FROM workstream_events WHERE workstream_id = ?1",@@ -1097,12 +1117,44 @@ pub(crate) fn finish_run(|row| row.get(0),)?;if state == "finished" {+ if recover_expired {+ validate_finished_recovery(+ &tx,+ &workstream,+ &agent_wire,+ linked_session.as_deref(),+ exit_code,+ input,+ )?;+ }+ tx.commit()?;return Ok(FinishedWorkstreamRun {imported_events: 0,latest_sequence: latest_before,});}- if state != "active" {+ if state == "expired" && recover_expired {+ let superseded: bool = tx.query_row(+ "SELECT EXISTS(SELECT 1 FROM managed_runs newer \+ JOIN managed_runs current ON current.id = ?2 \+ WHERE newer.workstream_id = ?1 AND \+ (newer.started_at > current.started_at OR \+ (newer.started_at = current.started_at AND newer.rowid > current.rowid)))",+ params![workstream, input.run_id.as_bytes()],+ |row| row.get(0),+ )?;+ if superseded {+ return Err(StoreError::InvalidState(format!(+ "managed run {} was superseded and cannot be recovered",+ input.run_id+ )));+ }+ tx.execute(+ "UPDATE managed_runs SET state = 'active', ended_at = NULL, lease_expires_at = ?1 \+ WHERE id = ?2 AND state = 'expired'",+ params![now + LEASE_MICROS, input.run_id.as_bytes()],+ )?;+ } else if state != "active" {return Err(StoreError::InvalidState(format!("managed run {} is {state}",input.run_id@@ -1272,6 +1324,108 @@ pub(crate) fn finish_run(})}+fn validate_finished_recovery(+ tx: &Transaction<'_>,+ workstream: &[u8],+ agent_wire: &str,+ linked_session: Option<&str>,+ exit_code: Option<i32>,+ input: &FinishWorkstreamRun,+) -> StoreResult<()> {+ if input.native_session_id.as_deref() != linked_session+ || (input.complete && input.exit_code != exit_code)+ || (!input.complete && (input.source_cursor.is_some() || input.exit_code.is_some()))+ {+ return Err(StoreError::InvalidState(+ "finished managed run recovery is not an exact replay".into(),+ ));+ }+ for event in &input.events {+ type StoredEvent = (+ String,+ String,+ String,+ Option<String>,+ Option<String>,+ String,+ Option<String>,+ String,+ );+ let stored: Option<StoredEvent> = tx+ .query_row(+ "SELECT agent_kind, native_session_id, kind, source_record_id, role, content, \+ occurred_at, metadata_json FROM workstream_events \+ WHERE workstream_id = ?1 AND event_id = ?2",+ params![workstream, event.event_id],+ |row| {+ Ok((+ row.get(0)?,+ row.get(1)?,+ row.get(2)?,+ row.get(3)?,+ row.get(4)?,+ row.get(5)?,+ row.get(6)?,+ row.get(7)?,+ ))+ },+ )+ .optional()?;+ let (source_record_id, metadata) = scrub_workstream_provenance(+ &input.sanitizer,+ event.source_record_id.as_deref(),+ &event.metadata,+ );+ let expected = (+ event.agent.as_str(),+ event.native_session_id.as_str(),+ event.kind.as_str(),+ source_record_id.as_deref(),+ event.role.as_deref(),+ ai_memory_core::truncate_utf8_bytes(&event.content, WORKSTREAM_CONTENT_MAX_BYTES),+ event.occurred_at.as_deref(),+ serde_json::to_string(&metadata)?,+ );+ let exact = stored.as_ref().is_some_and(|stored| {+ stored.0 == expected.0+ && stored.1 == expected.1+ && stored.2 == expected.2+ && stored.3.as_deref() == expected.3+ && stored.4.as_deref() == expected.4+ && stored.5 == expected.5+ && stored.6.as_deref() == expected.6+ && stored.7 == expected.7+ });+ if !exact || event.agent.as_str() != agent_wire {+ return Err(StoreError::InvalidState(+ "finished managed run recovery contains uncommitted or changed events".into(),+ ));+ }+ }+ let stored_cursor = if let Some(native_session) = linked_session {+ tx.query_row(+ "SELECT source_cursor FROM workstream_native_sessions \+ WHERE workstream_id = ?1 AND agent_kind = ?2 AND native_session_id = ?3",+ params![workstream, agent_wire, native_session],+ |row| row.get::<_, Option<String>>(0),+ )+ .optional()?+ .ok_or_else(|| {+ StoreError::InvalidState(+ "finished managed run recovery has no durable native session".into(),+ )+ })?+ } else {+ None+ };+ if input.complete && stored_cursor != input.source_cursor {+ return Err(StoreError::InvalidState(+ "finished managed run recovery cannot advance its source cursor".into(),+ ));+ }+ Ok(())+}+/// Reader-side managed-run status.pub(crate) fn run_status(conn: &Connection,diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rsindex 7801a9e5..64d668d5 100644--- a/crates/ai-memory-store/src/writer.rs+++ b/crates/ai-memory-store/src/writer.rs@@ -795,6 +795,7 @@ pub(crate) enum WriteCmd {FinishWorkstreamRun {authority: crate::ManagedRunAuthority,input: FinishWorkstreamRun,+ recover_expired: bool,reply: oneshot::Sender<StoreResult<FinishedWorkstreamRun>>,},RenameWorkstream {@@ -3401,11 +3402,32 @@ impl WriterHandle {&self,authority: crate::ManagedRunAuthority,input: FinishWorkstreamRun,+ ) -> StoreResult<FinishedWorkstreamRun> {+ self.finish_workstream_run_inner(authority, input, false)+ .await+ }++ /// Finish an expired run for its original owner when it was not superseded.+ pub async fn recover_expired_workstream_run(+ &self,+ authority: crate::ManagedRunAuthority,+ input: FinishWorkstreamRun,+ ) -> StoreResult<FinishedWorkstreamRun> {+ self.finish_workstream_run_inner(authority, input, true)+ .await+ }++ async fn finish_workstream_run_inner(+ &self,+ authority: crate::ManagedRunAuthority,+ input: FinishWorkstreamRun,+ recover_expired: bool,) -> StoreResult<FinishedWorkstreamRun> {let (tx, rx) = oneshot::channel();self.send(WriteCmd::FinishWorkstreamRun {authority,input,+ recover_expired,reply: tx,}).await?;@@ -4758,9 +4780,14 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {WriteCmd::FinishWorkstreamRun {authority,input,+ recover_expired,reply,} => {- let result = crate::workstream::finish_run(&mut conn, &authority, &input);+ let result = if recover_expired {+ crate::workstream::recover_run(&mut conn, &authority, &input)+ } else {+ crate::workstream::finish_run(&mut conn, &authority, &input)+ };send_or_warn(reply, result, "finish_workstream_run");}WriteCmd::RenameWorkstream { input, reply } => {diff --git a/crates/ai-memory-store/tests/suite/multi_session.rs b/crates/ai-memory-store/tests/suite/multi_session.rsindex 19c4a4ed..6f62abf0 100644--- a/crates/ai-memory-store/tests/suite/multi_session.rs+++ b/crates/ai-memory-store/tests/suite/multi_session.rs@@ -1582,6 +1582,37 @@ async fn owner_finish_store_refuses_another_writer_including_finished_retry() {assert_eq!(retry.imported_events, 0);assert_eq!(retry.latest_sequence, result.latest_sequence);assert_eq!(finish_snapshot(&f.store).await, before);+ for mutate in ["event-id", "content", "cursor", "exit-code"] {+ let mut changed = finish_input(run.run_id);+ match mutate {+ "event-id" => changed.events[0].event_id = "losing-event".into(),+ "content" => changed.events[0].content = "losing payload".into(),+ "cursor" => changed.source_cursor = Some("uncommitted-cursor".into()),+ "exit-code" => changed.exit_code = Some(9),+ _ => unreachable!(),+ }+ assert!(+ matches!(+ f.store+ .writer+ .recover_expired_workstream_run(f.alice(), changed)+ .await,+ Err(StoreError::InvalidState(_))+ ),+ "finished recovery must reject changed {mutate}"+ );+ assert_eq!(finish_snapshot(&f.store).await, before);+ }+ assert_eq!(+ f.store+ .writer+ .recover_expired_workstream_run(f.alice(), finish_input(run.run_id))+ .await+ .unwrap()+ .imported_events,+ 0,+ "an exact owner retry remains idempotent"+ );for (name, auth) in [("root-shared",@@ -1947,7 +1978,7 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {.authorize_managed_run(run.run_id, anonymous.clone()).await.unwrap();- store+ let owner_user = store.writer.create_human_user(NewUser {@@ -2038,6 +2069,92 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {0);assert_eq!(finish_snapshot(&store).await, before);+ // Explicit recovery admits the original owner only when no newer run exists.+ store+ .writer+ .grant_memory(+ owner_user,+ project,+ ai_memory_store::GrantLevel::Write,+ None,+ )+ .await+ .unwrap();+ let recoverable = store+ .writer+ .prepare_workstream_run_owned(+ PrepareWorkstreamRun {+ workspace_id: ws,+ project_id: project,+ repo_fingerprint: "recoverable".into(),+ worktree_fingerprint: "recoverable".into(),+ cwd: "/repo".into(),+ agent: AgentKind::Codex,+ automatic_harness: false,+ available_agents: Vec::new(),+ selection: WorkstreamSelection::New("recoverable".into()),+ lease_owner: "reload".into(),+ },+ Some("user:new-user".into()),+ )+ .await+ .unwrap();+ store+ .writer+ .cancel_managed_run(recoverable.run_id)+ .await+ .unwrap();+ let foreign_user = store+ .writer+ .create_human_user(+ NewUser {+ username: "foreign-recovery".into(),+ name: None,+ email: None,+ },+ UserRole::User,+ None,+ false,+ )+ .await+ .unwrap();+ store+ .writer+ .grant_memory(+ foreign_user,+ project,+ ai_memory_store::GrantLevel::Write,+ None,+ )+ .await+ .unwrap();+ let foreign = finish_authority(+ ai_memory_core::AuthLevel::User,+ Some(foreign_user),+ Some("foreign-recovery"),+ );+ assert!(matches!(+ store+ .writer+ .recover_expired_workstream_run(foreign, finish_input(recoverable.run_id))+ .await,+ Err(StoreError::Forbidden(_))+ ));+ let owner = finish_authority(+ ai_memory_core::AuthLevel::User,+ Some(owner_user),+ Some("new-user"),+ );+ assert_eq!(+ store+ .writer+ .recover_expired_workstream_run(owner, finish_input(recoverable.run_id))+ .await+ .unwrap()+ .imported_events,+ 1+ );+// Cancel and replacement both store the existing expired phase.for replaced in [false, true] {let old = store@@ -2062,7 +2179,7 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {[old.run_id.as_bytes()],).unwrap();- store+ let replacement = store.writer.prepare_workstream_run(PrepareWorkstreamRun {workspace_id: ws,@@ -2078,6 +2195,11 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {}).await.unwrap();+ conn.execute(+ "UPDATE managed_runs SET started_at = 7 WHERE id IN (?1, ?2)",+ rusqlite::params![old.run_id.as_bytes(), replacement.run_id.as_bytes()],+ )+ .unwrap();} else {store.writer.cancel_managed_run(old.run_id).await.unwrap();}@@ -2089,7 +2211,29 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {.await,Err(StoreError::InvalidState(_))));- assert_eq!(finish_snapshot(&store).await, before);+ if replaced {+ assert!(matches!(+ store+ .writer+ .recover_expired_workstream_run(root.clone(), finish_input(old.run_id))+ .await,+ Err(StoreError::InvalidState(_))+ ));+ } else {+ assert_eq!(+ store+ .writer+ .recover_expired_workstream_run(root.clone(), finish_input(old.run_id))+ .await+ .unwrap()+ .imported_events,+ 0,+ "the unsuperseded expired run is recoverable and stable event ids remain idempotent"+ );+ }+ if replaced {+ assert_eq!(finish_snapshot(&store).await, before);+ }}let before = finish_snapshot(&store).await;assert!(matches!(diff --git a/crates/ai-memory-workstream/src/harness.rs b/crates/ai-memory-workstream/src/harness.rsindex 5244075b..4126d91c 100644--- a/crates/ai-memory-workstream/src/harness.rs+++ b/crates/ai-memory-workstream/src/harness.rs@@ -113,6 +113,15 @@ fn valid_semver_identifiers(value: &str, reject_leading_zero_numeric: bool) -> b})}+/// Native transcript support available to recovery and backfill.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum TranscriptCapability {+ /// The native store can export a normalized visible-event transcript.+ Export,+ /// Recovery can only rely on lifecycle events retained in the hook spool.+ SpoolOnly,+}+/// Harnesses with native-session and transcript adapters.#[derive(Debug, Clone, Copy, PartialEq, Eq)]pub enum ManagedHarness {@@ -167,6 +176,15 @@ impl ManagedHarness {}}+ /// Recovery capability of this harness's native session store.+ #[must_use]+ pub const fn transcript_capability(self) -> TranscriptCapability {+ match self {+ Self::Antigravity => TranscriptCapability::SpoolOnly,+ _ => TranscriptCapability::Export,+ }+ }+/// Whether the harness has no native session-end hook, so its sessions/// stay open until `ai-memory finalize-session` runs (docs/support-matrix.md).#[must_use]@@ -3437,6 +3455,36 @@ mod tests {));}+ #[test]+ fn transcript_capability_marks_only_antigravity_spool_only() {+ for harness in [+ ManagedHarness::Claude,+ ManagedHarness::Codex,+ ManagedHarness::OpenCode,+ ManagedHarness::OpenCode2,+ ManagedHarness::Pi,+ ManagedHarness::Crush,+ ManagedHarness::Omp,+ ManagedHarness::Kimi,+ ManagedHarness::CommandCode,+ ManagedHarness::Kiro,+ ManagedHarness::KiroV3,+ ManagedHarness::Grok,+ ManagedHarness::Antigravity,+ ] {+ assert_eq!(+ harness.transcript_capability(),+ if harness == ManagedHarness::Antigravity {+ TranscriptCapability::SpoolOnly+ } else {+ TranscriptCapability::Export+ },+ "{}",+ harness.as_str()+ );+ }+ }+#[test]fn only_harnesses_without_a_session_end_hook_need_finalizing() {let hookless = [diff --git a/crates/ai-memory-workstream/src/lib.rs b/crates/ai-memory-workstream/src/lib.rsindex e00ad78f..c55094e8 100644--- a/crates/ai-memory-workstream/src/lib.rs+++ b/crates/ai-memory-workstream/src/lib.rs@@ -6,7 +6,7 @@ mod repository;mod transcript;pub use harness::{- LaunchMode, LaunchPlan, LaunchRoots, ManagedHarness, OpenCodeDialect,+ LaunchMode, LaunchPlan, LaunchRoots, ManagedHarness, OpenCodeDialect, TranscriptCapability,allows_native_session_adoption, apply_claude_true_yolo, apply_yolo, build_launch_plan,build_launch_plan_with_env, build_launch_plan_with_env_lookup, clean_path, crush_data_dir,crush_global_config_path, env_dir_override, has_native_session_selector,@@ -28,7 +28,8 @@ pub use transcript::{AmbiguousNativeSession, ExportedTranscript, NativeSessionCandidate,claude_attached_background_session, claude_live_background_attach_id,claude_session_ran_in_background, discover_native_session, export_transcript,- kiro_harness_from_source_cursor, kiro_v3_resume_uses_default_store, list_native_sessions,- native_memory_dir, native_session_exists, native_session_in_checkout, native_store_root,- wait_for_transcript_flush,+ export_transcript_delta, export_transcript_range, kiro_harness_from_source_cursor,+ kiro_v3_resume_uses_default_store, list_native_sessions, native_memory_dir,+ native_session_exists, native_session_in_checkout, native_store_root, transcript_baseline,+ transcript_interval_digests, wait_for_transcript_flush,};diff --git a/crates/ai-memory-workstream/src/transcript.rs b/crates/ai-memory-workstream/src/transcript.rsindex 20c9f03b..5319748c 100644--- a/crates/ai-memory-workstream/src/transcript.rs+++ b/crates/ai-memory-workstream/src/transcript.rs@@ -11,7 +11,7 @@ use ai_memory_core::{Sanitizer, WorkstreamEventKind,};use anyhow::{Context as _, Result, anyhow};-use rusqlite::{Connection, OpenFlags, params};+use rusqlite::{Connection, OpenFlags, OptionalExtension as _, params};use serde::{Deserialize, Serialize};use serde_json::{Value, json};use sha2::{Digest as _, Sha256};@@ -21,6 +21,13 @@ use crate::{ManagedHarness, clean_path};const MAX_SCAN_FILES: usize = 50_000;const MAX_EVENT_BYTES: usize = 128 * 1024;+const MAX_SESSION_HEADER_BYTES: usize = 128 * 1024;+const MAX_SOURCE_RECORD_BYTES: usize = 1024 * 1024;+const MAX_EXPORTED_EVENTS: usize = 4_096;+const MAX_EXPORTED_BYTES: usize = 32 * 1024 * 1024;+const MAX_SOURCE_RECORDS: usize = 32_768;+const MAX_SOURCE_BYTES: usize = 64 * 1024 * 1024;+const MAX_SESSION_METADATA_BYTES: u64 = 1024 * 1024;const LEGACY_MANAGED_WORKSTREAM_PACKET_PREFIX: &str = "> **ai-memory managed workstream:";/// Checkout-local native session that can seed an otherwise-empty workstream.@@ -45,6 +52,27 @@ pub struct ExportedTranscript {pub losses: Vec<String>,}+/// Ordered semantic identity for a bounded transcript interval.+#[must_use]+pub fn transcript_interval_digests(events: &[NewWorkstreamEvent]) -> Vec<String> {+ events.iter().map(transcript_event_digest).collect()+}++fn transcript_event_digest(event: &NewWorkstreamEvent) -> String {+ let mut hasher = Sha256::new();+ for value in [+ event.event_id.as_str(),+ event.kind.as_str(),+ event.role.as_deref().unwrap_or_default(),+ event.occurred_at.as_deref().unwrap_or_default(),+ event.content.as_str(),+ ] {+ hasher.update(value.len().to_be_bytes());+ hasher.update(value.as_bytes());+ }+ format!("{:x}", hasher.finalize())+}+#[derive(Debug, Clone, Serialize, Deserialize)]struct FileCursor {path: String,@@ -113,29 +141,419 @@ pub async fn export_transcript(native_session_id: &str,source_cursor: Option<&str>,) -> Result<ExportedTranscript> {- if harness == ManagedHarness::OpenCode {- return export_opencode(home, session_dir, native_session_id, source_cursor);- }- if harness == ManagedHarness::OpenCode2 {- return export_opencode2(home, session_dir, native_session_id, source_cursor);- }- if harness == ManagedHarness::Crush {- return export_crush(home, cwd, session_dir, native_session_id, source_cursor);- }- if harness == ManagedHarness::Antigravity {- // The conversation store keeps every step as an undocumented protobuf- // blob whose step-type enum is unversioned, so message text cannot be- // decoded without guessing at a schema that changes between `agy`- // releases. Conversation identity and workspace are read (they are- // stable fields observed in current metadata); the visible-event- // ledger for this harness comes from lifecycle-hook capture instead.+ let home = home.to_path_buf();+ let cwd = cwd.to_path_buf();+ let session_dir = session_dir.map(Path::to_path_buf);+ let native_session_id = native_session_id.to_owned();+ let source_cursor = source_cursor.map(str::to_owned);+ tokio::task::spawn_blocking(move || {+ export_transcript_blocking(+ harness,+ &home,+ &cwd,+ session_dir.as_deref(),+ &native_session_id,+ source_cursor.as_deref(),+ None,+ )+ })+ .await+ .map_err(|error| anyhow!("native transcript reader failed: {error}"))?+}++fn export_transcript_blocking(+ harness: ManagedHarness,+ home: &Path,+ cwd: &Path,+ session_dir: Option<&Path>,+ native_session_id: &str,+ source_cursor: Option<&str>,+ final_cursor: Option<&str>,+) -> Result<ExportedTranscript> {+ let transcript = if harness == ManagedHarness::OpenCode {+ export_opencode_range(+ home,+ session_dir,+ native_session_id,+ source_cursor,+ final_cursor,+ )?+ } else if harness == ManagedHarness::OpenCode2 {+ export_opencode2_range(+ home,+ session_dir,+ native_session_id,+ source_cursor,+ final_cursor,+ )?+ } else if harness == ManagedHarness::Crush {+ export_crush_range(+ home,+ cwd,+ session_dir,+ native_session_id,+ source_cursor,+ final_cursor,+ )?+ } else if harness == ManagedHarness::Antigravity {return Err(anyhow!("antigravity conversations expose no decodable transcript; this session's events come from hook capture"));+ } else {+ let path = locate_session_file(harness, home, cwd, session_dir, native_session_id)?+ .ok_or_else(|| anyhow!("native transcript for {native_session_id} was not found"))?;+ export_jsonl_range(+ harness,+ &path,+ native_session_id,+ source_cursor,+ final_cursor,+ )?+ };+ validate_export_bounds(transcript)+}++/// Export only records inside an exact previously captured cursor range.+/// Both cursor identities are validated against the adapter/session, and rows+/// or bytes appended after `final_cursor` are never read.+pub async fn export_transcript_range(+ harness: ManagedHarness,+ home: &Path,+ cwd: &Path,+ session_dir: Option<&Path>,+ native_session_id: &str,+ source_cursor: Option<&str>,+ final_cursor: &str,+) -> Result<ExportedTranscript> {+ let home = home.to_path_buf();+ let cwd = cwd.to_path_buf();+ let session_dir = session_dir.map(Path::to_path_buf);+ let native_session_id = native_session_id.to_owned();+ let source_cursor = source_cursor.map(str::to_owned);+ let final_cursor = final_cursor.to_owned();+ tokio::task::spawn_blocking(move || {+ if let Some(source_cursor) = source_cursor.as_deref() {+ validate_transcript_cursor(+ harness,+ &home,+ &cwd,+ session_dir.as_deref(),+ &native_session_id,+ source_cursor,+ )?;+ }+ validate_transcript_cursor(+ harness,+ &home,+ &cwd,+ session_dir.as_deref(),+ &native_session_id,+ &final_cursor,+ )?;+ validate_cursor_range(harness, source_cursor.as_deref(), &final_cursor)?;+ export_transcript_blocking(+ harness,+ &home,+ &cwd,+ session_dir.as_deref(),+ &native_session_id,+ source_cursor.as_deref(),+ Some(&final_cursor),+ )+ })+ .await+ .map_err(|error| anyhow!("native transcript range reader failed: {error}"))?+}++/// Export only records after a cursor previously returned for this exact+/// adapter/session. Invalid or stale cursors fail closed instead of replaying+/// the complete transcript.+pub async fn export_transcript_delta(+ harness: ManagedHarness,+ home: &Path,+ cwd: &Path,+ session_dir: Option<&Path>,+ native_session_id: &str,+ source_cursor: &str,+) -> Result<ExportedTranscript> {+ let home = home.to_path_buf();+ let cwd = cwd.to_path_buf();+ let session_dir = session_dir.map(Path::to_path_buf);+ let native_session_id = native_session_id.to_owned();+ let source_cursor = source_cursor.to_owned();+ tokio::task::spawn_blocking(move || {+ validate_transcript_cursor(+ harness,+ &home,+ &cwd,+ session_dir.as_deref(),+ &native_session_id,+ &source_cursor,+ )?;+ export_transcript_blocking(+ harness,+ &home,+ &cwd,+ session_dir.as_deref(),+ &native_session_id,+ Some(&source_cursor),+ None,+ )+ })+ .await+ .map_err(|error| anyhow!("native transcript delta reader failed: {error}"))?+}++fn validate_cursor_range(+ harness: ManagedHarness,+ source_cursor: Option<&str>,+ final_cursor: &str,+) -> Result<()> {+ if matches!(+ harness,+ ManagedHarness::OpenCode | ManagedHarness::OpenCode2 | ManagedHarness::Crush+ ) {+ let start = source_cursor+ .map(serde_json::from_str::<SqlCursor>)+ .transpose()+ .context("invalid native transcript SQL cursor")?+ .unwrap_or_default();+ let end: SqlCursor = serde_json::from_str(final_cursor)+ .context("invalid native transcript final SQL cursor")?;+ if (end.updated, end.id.as_str()) < (start.updated, start.id.as_str()) {+ return Err(anyhow!(+ "native transcript final cursor precedes its start cursor"+ ));+ }+ } else if let Some(source_cursor) = source_cursor {+ let start: FileCursor =+ serde_json::from_str(source_cursor).context("invalid native transcript file cursor")?;+ let end: FileCursor = serde_json::from_str(final_cursor)+ .context("invalid native transcript final file cursor")?;+ if start.path != end.path || start.flavor != end.flavor || end.offset < start.offset {+ return Err(anyhow!(+ "native transcript final cursor precedes or changes its start cursor"+ ));+ }+ }+ Ok(())+}++fn validate_transcript_cursor(+ harness: ManagedHarness,+ home: &Path,+ cwd: &Path,+ session_dir: Option<&Path>,+ native_session_id: &str,+ source_cursor: &str,+) -> Result<()> {+ if matches!(+ harness,+ ManagedHarness::OpenCode | ManagedHarness::OpenCode2 | ManagedHarness::Crush+ ) {+ serde_json::from_str::<SqlCursor>(source_cursor)+ .context("invalid native transcript SQL cursor")?;+ return Ok(());}+ let cursor: FileCursor =+ serde_json::from_str(source_cursor).context("invalid native transcript file cursor")?;let path = locate_session_file(harness, home, cwd, session_dir, native_session_id)?.ok_or_else(|| anyhow!("native transcript for {native_session_id} was not found"))?;- export_jsonl(harness, &path, native_session_id, source_cursor)+ if Path::new(&cursor.path) != path || cursor.flavor != file_flavor(harness) {+ return Err(anyhow!(+ "native transcript cursor does not match this adapter session"+ ));+ }+ let len = fs::metadata(&path)?.len();+ if cursor.offset > len {+ return Err(anyhow!(+ "native transcript cursor is past the current transcript end"+ ));+ }+ if journal_rewrites_in_place(harness) {+ let expected = cursor+ .prefix_sha256+ .as_deref()+ .ok_or_else(|| anyhow!("native transcript cursor has no prefix identity"))?;+ let mut file = File::open(path)?;+ let actual = hash_file_prefix(&mut file, cursor.offset)?+ .map(|hasher| format!("{:x}", hasher.finalize()))+ .ok_or_else(|| anyhow!("native transcript cursor prefix is incomplete"))?;+ if actual != expected {+ return Err(anyhow!(+ "native transcript changed before the recovery cursor"+ ));+ }+ }+ Ok(())+}++fn validate_export_bounds(transcript: ExportedTranscript) -> Result<ExportedTranscript> {+ if transcript.events.len() > MAX_EXPORTED_EVENTS {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_EXPORTED_EVENTS} event recovery limit"+ ));+ }+ let mut bytes = 0_usize;+ for event in &transcript.events {+ bytes = bytes.saturating_add(serde_json::to_vec(event)?.len());+ if bytes > MAX_EXPORTED_BYTES {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_EXPORTED_BYTES} byte recovery limit"+ ));+ }+ }+ Ok(transcript)+}++fn enforce_parse_bounds(+ events: &[NewWorkstreamEvent],+ checked_events: &mut usize,+ event_bytes: &mut usize,+ source_bytes: &mut usize,+ source_record_bytes: usize,+) -> Result<()> {+ *source_bytes = source_bytes.saturating_add(source_record_bytes);+ if *source_bytes > MAX_SOURCE_BYTES {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_SOURCE_BYTES} source-byte limit"+ ));+ }+ if events.len() > MAX_EXPORTED_EVENTS {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_EXPORTED_EVENTS} event recovery limit"+ ));+ }+ for event in &events[*checked_events..] {+ *event_bytes = event_bytes.saturating_add(serde_json::to_vec(event)?.len());+ if *event_bytes > MAX_EXPORTED_BYTES {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_EXPORTED_BYTES} byte recovery limit"+ ));+ }+ }+ *checked_events = events.len();+ Ok(())+}++/// Capture the exact adapter cursor at the current end of one native session.+/// Recovery may use it only for that same session and adapter.+pub async fn transcript_baseline(+ harness: ManagedHarness,+ home: &Path,+ cwd: &Path,+ session_dir: Option<&Path>,+ native_session_id: &str,+) -> Result<String> {+ let home = home.to_path_buf();+ let cwd = cwd.to_path_buf();+ let session_dir = session_dir.map(Path::to_path_buf);+ let native_session_id = native_session_id.to_owned();+ tokio::task::spawn_blocking(move || {+ transcript_baseline_blocking(+ harness,+ &home,+ &cwd,+ session_dir.as_deref(),+ &native_session_id,+ )+ })+ .await+ .map_err(|error| anyhow!("native transcript baseline reader failed: {error}"))?+}++fn transcript_baseline_blocking(+ harness: ManagedHarness,+ home: &Path,+ cwd: &Path,+ session_dir: Option<&Path>,+ native_session_id: &str,+) -> Result<String> {+ let sql_cursor = |db: PathBuf, sql: &str| -> Result<String> {+ let connection = Connection::open_with_flags(+ &db,+ OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,+ )?;+ let cursor = connection+ .query_row(sql, params![native_session_id], |row| {+ Ok(SqlCursor {+ updated: row.get(0)?,+ id: row.get(1)?,+ })+ })+ .optional()?+ .unwrap_or_default();+ Ok(serde_json::to_string(&cursor)?)+ };+ match harness {+ ManagedHarness::OpenCode => sql_cursor(+ opencode_db(home, session_dir),+ "SELECT p.time_updated, p.id FROM part p WHERE p.session_id = ?1 ORDER BY p.time_updated DESC, p.id DESC LIMIT 1",+ ),+ ManagedHarness::OpenCode2 => sql_cursor(+ opencode_db(home, session_dir),+ "SELECT time_updated, id FROM session_message WHERE session_id = ?1 ORDER BY time_updated DESC, id DESC LIMIT 1",+ ),+ ManagedHarness::Crush => sql_cursor(+ crush_db(home, cwd, session_dir),+ "SELECT updated_at, id FROM messages WHERE session_id = ?1 ORDER BY updated_at DESC, id DESC LIMIT 1",+ ),+ ManagedHarness::Antigravity => Err(anyhow!(+ "{} does not expose a reliable transcript cursor",+ harness.as_str()+ )),+ _ => {+ let path = locate_session_file(harness, home, cwd, session_dir, native_session_id)?+ .ok_or_else(|| {+ anyhow!("native transcript for {native_session_id} was not found")+ })?;+ let mut file = File::open(&path)?;+ let mut reader = BufReader::new(&mut file);+ let mut offset = 0_u64;+ let mut committed_offset = 0_u64;+ let mut prefix_hasher = Sha256::new();+ let mut line = Vec::new();+ let mut records = 0_usize;+ loop {+ line.clear();+ let read = std::io::Read::by_ref(&mut reader)+ .take((MAX_SOURCE_RECORD_BYTES + 1) as u64)+ .read_until(b'\n', &mut line)?;+ if read == 0 {+ break;+ }+ records += 1;+ if records > MAX_SOURCE_RECORDS {+ return Err(anyhow!(+ "native transcript baseline exceeds the {MAX_SOURCE_RECORDS} source-record limit"+ ));+ }+ if read > MAX_SOURCE_RECORD_BYTES {+ return Err(anyhow!(+ "native transcript baseline record exceeds the {MAX_SOURCE_RECORD_BYTES} byte limit"+ ));+ }+ offset = offset.saturating_add(read as u64);+ if offset > MAX_SOURCE_BYTES as u64 {+ return Err(anyhow!(+ "native transcript baseline exceeds the {MAX_SOURCE_BYTES} source-byte limit"+ ));+ }+ if !line.ends_with(b"\n") {+ break;+ }+ prefix_hasher.update(&line);+ committed_offset = offset;+ }+ Ok(serde_json::to_string(&FileCursor {+ path: path.to_string_lossy().into_owned(),+ offset: committed_offset,+ flavor: file_flavor(harness),+ prefix_sha256: journal_rewrites_in_place(harness)+ .then(|| format!("{:x}", prefix_hasher.finalize())),+ })?)+ }+ }}/// New sessions appeared in a store that records no launch identity and none@@ -582,11 +1000,22 @@ pub async fn wait_for_transcript_flush(Ok(())}+#[cfg(test)]fn export_jsonl(harness: ManagedHarness,path: &Path,native_session_id: &str,source_cursor: Option<&str>,+) -> Result<ExportedTranscript> {+ export_jsonl_range(harness, path, native_session_id, source_cursor, None)+}++fn export_jsonl_range(+ harness: ManagedHarness,+ path: &Path,+ native_session_id: &str,+ source_cursor: Option<&str>,+ final_cursor: Option<&str>,) -> Result<ExportedTranscript> {let flavor = file_flavor(harness);let cursor = source_cursor@@ -594,9 +1023,29 @@ fn export_jsonl(.filter(|cursor| {Path::new(&cursor.path) == path && (cursor.flavor.is_none() || cursor.flavor == flavor)});+ let final_cursor = final_cursor+ .map(serde_json::from_str::<FileCursor>)+ .transpose()+ .context("invalid native transcript final file cursor")?;+ if final_cursor+ .as_ref()+ .is_some_and(|cursor| Path::new(&cursor.path) != path || cursor.flavor != flavor)+ {+ return Err(anyhow!(+ "native transcript final cursor does not match this adapter session"+ ));+ }let mut file = File::open(path).with_context(|| format!("opening native transcript {}", path.display()))?;- let len = file.metadata()?.len();+ let file_len = file.metadata()?.len();+ let len = final_cursor+ .as_ref()+ .map_or(file_len, |cursor| cursor.offset);+ if len > file_len {+ return Err(anyhow!(+ "native transcript final cursor is past the current transcript end"+ ));+ }let (start, mut prefix_hasher) = if journal_rewrites_in_place(harness) {let validated = if let Some(cursor) = cursor.as_ref().filter(|cursor| cursor.offset <= len)&& let Some(expected) = cursor.prefix_sha256.as_deref()@@ -621,12 +1070,41 @@ fn export_jsonl(let mut line = Vec::new();let mut events = Vec::new();let mut losses = Vec::new();+ let mut records = 0_usize;+ let mut source_bytes = 0_usize;+ let mut checked_events = 0_usize;+ let mut event_bytes = 0_usize;loop {line.clear();- let read = reader.read_until(b'\n', &mut line)?;+ let remaining = len.saturating_sub(offset);+ if remaining == 0 {+ break;+ }+ let read_limit = remaining.min((MAX_SOURCE_RECORD_BYTES + 1) as u64);+ let read = std::io::Read::by_ref(&mut reader)+ .take(read_limit)+ .read_until(b'\n', &mut line)?;if read == 0 {break;}+ records += 1;+ if records > MAX_SOURCE_RECORDS {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_SOURCE_RECORDS} source-record limit"+ ));+ }+ if read > MAX_SOURCE_RECORD_BYTES {+ return Err(anyhow!(+ "native transcript record exceeds the {MAX_SOURCE_RECORD_BYTES} byte limit"+ ));+ }+ enforce_parse_bounds(+ &events,+ &mut checked_events,+ &mut event_bytes,+ &mut source_bytes,+ read,+ )?;offset += read as u64;if !line.ends_with(b"\n") {break;@@ -727,6 +1205,13 @@ fn export_jsonl());}}+ enforce_parse_bounds(+ &events,+ &mut checked_events,+ &mut event_bytes,+ &mut source_bytes,+ 0,+ )?;}if harness == ManagedHarness::Kimi {annotate_kimi_subagents(path, &mut losses);@@ -737,8 +1222,13 @@ fn export_jsonl(path: path.to_string_lossy().into_owned(),offset: committed_offset,flavor,- prefix_sha256: journal_rewrites_in_place(harness)- .then(|| format!("{:x}", prefix_hasher.finalize())),+ prefix_sha256: if journal_rewrites_in_place(harness) {+ final_cursor+ .and_then(|cursor| cursor.prefix_sha256.clone())+ .or_else(|| Some(format!("{:x}", prefix_hasher.finalize())))+ } else {+ None+ },})?),events,losses: deduplicate_losses(losses),@@ -2126,11 +2616,22 @@ fn codex_synthetic_context(agent: AgentKind, role: &str, text: &str) -> bool {}}+#[cfg(test)]fn export_opencode(home: &Path,session_dir: Option<&Path>,session: &str,source_cursor: Option<&str>,+) -> Result<ExportedTranscript> {+ export_opencode_range(home, session_dir, session, source_cursor, None)+}++fn export_opencode_range(+ home: &Path,+ session_dir: Option<&Path>,+ session: &str,+ source_cursor: Option<&str>,+ final_cursor: Option<&str>,) -> Result<ExportedTranscript> {let db = opencode_db(home, session_dir);let connection = Connection::open_with_flags(@@ -2146,25 +2647,57 @@ fn export_opencode(let cursor = source_cursor.and_then(|raw| serde_json::from_str::<SqlCursor>(raw).ok()).unwrap_or_default();+ let upper = final_cursor+ .map(serde_json::from_str::<SqlCursor>)+ .transpose()+ .context("invalid native transcript final SQL cursor")?+ .unwrap_or(SqlCursor {+ updated: i64::MAX,+ id: "\u{10ffff}".to_string(),+ });let mut statement = connection.prepare(- "SELECT p.id, p.time_updated, m.data, p.data+ "SELECT p.id, p.time_updated, length(m.data), length(p.data), m.data, p.dataFROM part p JOIN message m ON m.id = p.message_idWHERE p.session_id = ?1 AND (p.time_updated > ?2 OR (p.time_updated = ?2 AND p.id > ?3))- ORDER BY p.time_updated, p.id",+ AND (p.time_updated < ?4 OR (p.time_updated = ?4 AND p.id <= ?5))+ ORDER BY p.time_updated, p.id+ LIMIT ?6",)?;- let rows = statement.query_map(params![session, cursor.updated, cursor.id], |row| {- Ok((- row.get::<_, String>(0)?,- row.get::<_, i64>(1)?,- row.get::<_, String>(2)?,- row.get::<_, String>(3)?,- ))- })?;+ let mut rows = statement.query(params![+ session,+ cursor.updated,+ cursor.id,+ upper.updated,+ upper.id,+ (MAX_SOURCE_RECORDS + 1) as i64+ ])?;let mut events = Vec::new();let mut losses = Vec::new();let mut next_cursor = cursor;- for row in rows {- let (id, updated, message_raw, part_raw) = row?;+ let mut source_bytes = 0_usize;+ let mut checked_events = 0_usize;+ let mut event_bytes = 0_usize;+ let mut records = 0_usize;+ while let Some(row) = rows.next()? {+ records += 1;+ if records > MAX_SOURCE_RECORDS {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_SOURCE_RECORDS} source-record limit"+ ));+ }+ let updated = row.get::<_, i64>(1)?;+ let message_len = row.get::<_, usize>(2)?;+ let part_len = row.get::<_, usize>(3)?;+ enforce_parse_bounds(+ &events,+ &mut checked_events,+ &mut event_bytes,+ &mut source_bytes,+ message_len.saturating_add(part_len),+ )?;+ let id = row.get::<_, String>(0)?;+ let message_raw = row.get::<_, String>(4)?;+ let part_raw = row.get::<_, String>(5)?;next_cursor = SqlCursor {updated,id: id.clone(),@@ -2178,6 +2711,13 @@ fn export_opencode(continue;};parse_opencode(&message, &part, session, &id, &mut events, &mut losses);+ enforce_parse_bounds(+ &events,+ &mut checked_events,+ &mut event_bytes,+ &mut source_bytes,+ 0,+ )?;}Ok(ExportedTranscript {native_session_id: session.to_string(),@@ -2187,12 +2727,24 @@ fn export_opencode(})}+#[cfg(test)]fn export_crush(home: &Path,cwd: &Path,session_dir: Option<&Path>,session: &str,source_cursor: Option<&str>,+) -> Result<ExportedTranscript> {+ export_crush_range(home, cwd, session_dir, session, source_cursor, None)+}++fn export_crush_range(+ home: &Path,+ cwd: &Path,+ session_dir: Option<&Path>,+ session: &str,+ source_cursor: Option<&str>,+ final_cursor: Option<&str>,) -> Result<ExportedTranscript> {let db = crush_db(home, cwd, session_dir);let connection = Connection::open_with_flags(@@ -2203,27 +2755,57 @@ fn export_crush(let cursor = source_cursor.and_then(|raw| serde_json::from_str::<SqlCursor>(raw).ok()).unwrap_or_default();+ let upper = final_cursor+ .map(serde_json::from_str::<SqlCursor>)+ .transpose()+ .context("invalid native transcript final SQL cursor")?+ .unwrap_or(SqlCursor {+ updated: i64::MAX,+ id: "\u{10ffff}".to_string(),+ });let mut statement = connection.prepare(- "SELECT id, role, parts, updated_at, is_summary_message \+ "SELECT id, role, length(parts), parts, updated_at, is_summary_message \FROM messages \WHERE session_id = ?1 \AND (updated_at > ?2 OR (updated_at = ?2 AND id > ?3)) \- ORDER BY updated_at, id",+ AND (updated_at < ?4 OR (updated_at = ?4 AND id <= ?5)) \+ ORDER BY updated_at, id LIMIT ?6",)?;- let rows = statement.query_map(params![session, cursor.updated, cursor.id], |row| {- Ok((- row.get::<_, String>(0)?,- row.get::<_, String>(1)?,- row.get::<_, String>(2)?,- row.get::<_, i64>(3)?,- row.get::<_, i64>(4)?,- ))- })?;+ let mut rows = statement.query(params![+ session,+ cursor.updated,+ cursor.id,+ upper.updated,+ upper.id,+ (MAX_SOURCE_RECORDS + 1) as i64+ ])?;let mut events = Vec::new();let mut losses = Vec::new();let mut next_cursor = cursor;- for row in rows {- let (id, role, parts_raw, updated, is_summary) = row?;+ let mut source_bytes = 0_usize;+ let mut checked_events = 0_usize;+ let mut event_bytes = 0_usize;+ let mut records = 0_usize;+ while let Some(row) = rows.next()? {+ records += 1;+ if records > MAX_SOURCE_RECORDS {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_SOURCE_RECORDS} source-record limit"+ ));+ }+ let parts_len = row.get::<_, usize>(2)?;+ enforce_parse_bounds(+ &events,+ &mut checked_events,+ &mut event_bytes,+ &mut source_bytes,+ parts_len,+ )?;+ let id = row.get::<_, String>(0)?;+ let role = row.get::<_, String>(1)?;+ let parts_raw = row.get::<_, String>(3)?;+ let updated = row.get::<_, i64>(4)?;+ let is_summary = row.get::<_, i64>(5)?;next_cursor = SqlCursor {updated,id: id.clone(),@@ -2245,6 +2827,13 @@ fn export_crush(&mut events,&mut losses,);+ enforce_parse_bounds(+ &events,+ &mut checked_events,+ &mut event_bytes,+ &mut source_bytes,+ 0,+ )?;}Ok(ExportedTranscript {native_session_id: session.to_string(),@@ -2643,6 +3232,25 @@ fn temporary_transcript(path: &Path) -> bool {.is_some_and(|name| name.contains(".jsonl."))}+fn read_bounded_file(path: &Path, limit: u64) -> std::io::Result<Vec<u8>> {+ let file = File::open(path)?;+ if file.metadata()?.len() > limit {+ return Err(std::io::Error::new(+ std::io::ErrorKind::InvalidData,+ "native transcript metadata exceeds its byte limit",+ ));+ }+ let mut bytes = Vec::new();+ file.take(limit.saturating_add(1)).read_to_end(&mut bytes)?;+ if bytes.len() as u64 > limit {+ return Err(std::io::Error::new(+ std::io::ErrorKind::InvalidData,+ "native transcript metadata exceeds its byte limit",+ ));+ }+ Ok(bytes)+}+fn session_header(harness: ManagedHarness, path: &Path) -> Result<Option<(String, PathBuf)>> {if harness == ManagedHarness::Kimi {return kimi_session_header(path);@@ -2663,9 +3271,14 @@ fn session_header(harness: ManagedHarness, path: &Path) -> Result<Option<(Stringlet mut line = String::new();for _ in 0..64 {line.clear();- if reader.read_line(&mut line)? == 0 {+ let read = std::io::Read::take(&mut reader, (MAX_SESSION_HEADER_BYTES + 1) as u64)+ .read_line(&mut line)?;+ if read == 0 {break;}+ if read > MAX_SESSION_HEADER_BYTES {+ return Ok(None);+ }let Ok(value) = serde_json::from_str::<Value>(&line) else {continue;};@@ -2769,10 +3382,11 @@ fn kimi_session_header(path: &Path) -> Result<Option<(String, PathBuf)>> {let Some(session_dir) = path.ancestors().nth(3) else {return Ok(None);};- let Ok(raw) = fs::read_to_string(session_dir.join("state.json")) else {+ let Ok(raw) = read_bounded_file(&session_dir.join("state.json"), MAX_SESSION_METADATA_BYTES)+ else {return Ok(None);};- let Ok(state) = serde_json::from_str::<Value>(&raw) else {+ let Ok(state) = serde_json::from_slice::<Value>(&raw) else {return Ok(None);};let Some(id) = session_dir.file_name().and_then(|name| name.to_str()) else {@@ -2815,10 +3429,11 @@ fn kiro_session_header(path: &Path) -> Result<Option<(String, PathBuf)>> {if Uuid::parse_str(stem).is_err() {return Ok(None);}- let Ok(raw) = fs::read_to_string(path.with_extension("json")) else {+ let Ok(raw) = read_bounded_file(&path.with_extension("json"), MAX_SESSION_METADATA_BYTES)+ else {return Ok(None);};- let Ok(metadata) = serde_json::from_str::<Value>(&raw) else {+ let Ok(metadata) = serde_json::from_slice::<Value>(&raw) else {return Ok(None);};let Some(id) = metadata.get("session_id").and_then(Value::as_str) else {@@ -2852,10 +3467,13 @@ fn kiro_v3_session_header(path: &Path, cwd: &Path) -> Result<Option<(String, Patif Uuid::parse_str(uuid).is_err() {return Ok(None);}- let Ok(raw) = fs::read_to_string(session_dir.join("session.json")) else {+ let Ok(raw) = read_bounded_file(+ &session_dir.join("session.json"),+ MAX_SESSION_METADATA_BYTES,+ ) else {return Ok(None);};- let Ok(metadata) = serde_json::from_str::<Value>(&raw) else {+ let Ok(metadata) = serde_json::from_slice::<Value>(&raw) else {return Ok(None);};if metadata.get("schemaVersion").and_then(Value::as_str) != Some("1.0.0")@@ -2885,10 +3503,13 @@ fn grok_session_header(path: &Path) -> Result<Option<(String, PathBuf)>> {let Some(session_dir) = path.parent() else {return Ok(None);};- let Ok(raw) = fs::read_to_string(session_dir.join("summary.json")) else {+ let Ok(raw) = read_bounded_file(+ &session_dir.join("summary.json"),+ MAX_SESSION_METADATA_BYTES,+ ) else {return Ok(None);};- let Ok(summary) = serde_json::from_str::<Value>(&raw) else {+ let Ok(summary) = serde_json::from_slice::<Value>(&raw) else {return Ok(None);};let info = summary.get("info").unwrap_or(&Value::Null);@@ -2925,8 +3546,8 @@ fn antigravity_session_header(path: &Path) -> Result<Option<(String, PathBuf)>>return Ok(None);};let blob = connection.query_row(- "SELECT data FROM trajectory_metadata_blob WHERE id = 'main' LIMIT 1",- [],+ "SELECT data FROM trajectory_metadata_blob WHERE id = 'main' AND length(data) <= ?1 LIMIT 1",+ params![MAX_SESSION_METADATA_BYTES as i64],|row| row.get::<_, Vec<u8>>(0),);let Ok(blob) = blob else {@@ -3324,11 +3945,22 @@ fn opencode_directories(cwd: &Path) -> (String, String) {/// is encrypted and excluded like v1's hidden reasoning; anything else/// unrecognized becomes a bounded loss, never a guess — the beta schema is/// still changing.+#[cfg(test)]fn export_opencode2(home: &Path,session_dir: Option<&Path>,session: &str,source_cursor: Option<&str>,+) -> Result<ExportedTranscript> {+ export_opencode2_range(home, session_dir, session, source_cursor, None)+}++fn export_opencode2_range(+ home: &Path,+ session_dir: Option<&Path>,+ session: &str,+ source_cursor: Option<&str>,+ final_cursor: Option<&str>,) -> Result<ExportedTranscript> {let db = opencode_db(home, session_dir);let connection = Connection::open_with_flags(@@ -3344,24 +3976,54 @@ fn export_opencode2(let cursor = source_cursor.and_then(|raw| serde_json::from_str::<SqlCursor>(raw).ok()).unwrap_or_default();+ let upper = final_cursor+ .map(serde_json::from_str::<SqlCursor>)+ .transpose()+ .context("invalid native transcript final SQL cursor")?+ .unwrap_or(SqlCursor {+ updated: i64::MAX,+ id: "\u{10ffff}".to_string(),+ });let mut statement = connection.prepare(- "SELECT id, time_updated, type, data FROM session_message \+ "SELECT id, time_updated, type, length(data), data FROM session_message \WHERE session_id = ?1 AND (time_updated > ?2 OR (time_updated = ?2 AND id > ?3)) \- ORDER BY time_updated, id",+ AND (time_updated < ?4 OR (time_updated = ?4 AND id <= ?5)) \+ ORDER BY time_updated, id LIMIT ?6",)?;- let rows = statement.query_map(params![session, cursor.updated, cursor.id], |row| {- Ok((- row.get::<_, String>(0)?,- row.get::<_, i64>(1)?,- row.get::<_, String>(2)?,- row.get::<_, String>(3)?,- ))- })?;+ let mut rows = statement.query(params![+ session,+ cursor.updated,+ cursor.id,+ upper.updated,+ upper.id,+ (MAX_SOURCE_RECORDS + 1) as i64+ ])?;let mut events = Vec::new();let mut losses = Vec::new();let mut next_cursor = cursor;- for row in rows {- let (id, updated, message_type, data_raw) = row?;+ let mut source_bytes = 0_usize;+ let mut checked_events = 0_usize;+ let mut event_bytes = 0_usize;+ let mut records = 0_usize;+ while let Some(row) = rows.next()? {+ records += 1;+ if records > MAX_SOURCE_RECORDS {+ return Err(anyhow!(+ "native transcript exceeds the {MAX_SOURCE_RECORDS} source-record limit"+ ));+ }+ let data_len = row.get::<_, usize>(3)?;+ enforce_parse_bounds(+ &events,+ &mut checked_events,+ &mut event_bytes,+ &mut source_bytes,+ data_len,+ )?;+ let id = row.get::<_, String>(0)?;+ let updated = row.get::<_, i64>(1)?;+ let message_type = row.get::<_, String>(2)?;+ let data_raw = row.get::<_, String>(4)?;next_cursor = SqlCursor {updated,id: id.clone(),@@ -3371,6 +4033,13 @@ fn export_opencode2(continue;};parse_opencode2(&message_type, &data, session, &id, &mut events, &mut losses);+ enforce_parse_bounds(+ &events,+ &mut checked_events,+ &mut event_bytes,+ &mut source_bytes,+ 0,+ )?;}Ok(ExportedTranscript {native_session_id: session.to_string(),@@ -4069,6 +4738,294 @@ mod tests {assert_eq!(second.events[0].metadata["parent_id"], "u1");}+ #[tokio::test]+ async fn persisted_baseline_exports_only_offline_append_and_rejects_rewrite() {+ let temp = tempfile::tempdir().unwrap();+ let cwd = temp.path().join("repo");+ fs::create_dir_all(&cwd).unwrap();+ let session = "7c1d5698-204a-4c0f-ae9c-43db7fc4e41d";+ let store = temp.path().join("command-code");+ let bucket = store.join("bucket");+ fs::create_dir_all(&bucket).unwrap();+ let path = bucket.join(format!("{session}.jsonl"));+ fs::write(+ &path,+ format!(+ "{}\n{}\n",+ json!({"type":"session","version":3,"id":session,"timestamp":"2026-08-07T17:00:00Z","cwd":cwd}),+ json!({"type":"message","id":"before","timestamp":"2026-08-07T17:00:01Z","message":{"role":"user","content":[{"type":"text","text":"prior live history"}],"meta":{"source":"user"}}})+ ),+ )+ .unwrap();+ let baseline = transcript_baseline(+ ManagedHarness::CommandCode,+ temp.path(),+ &cwd,+ Some(&store),+ session,+ )+ .await+ .unwrap();+ let mut file = fs::OpenOptions::new().append(true).open(&path).unwrap();+ use std::io::Write as _;+ writeln!(+ file,+ "{}",+ json!({"type":"message","id":"after","timestamp":"2026-08-07T17:00:02Z","message":{"role":"assistant","content":[{"type":"text","text":"offline append"}],"meta":{"source":"model"}}})+ )+ .unwrap();+ let delta = export_transcript_delta(+ ManagedHarness::CommandCode,+ temp.path(),+ &cwd,+ Some(&store),+ session,+ &baseline,+ )+ .await+ .unwrap();+ assert_eq!(delta.events.len(), 1);+ assert_eq!(delta.events[0].content, "offline append");+ let final_cursor = delta.source_cursor.clone().unwrap();+ writeln!(+ file,+ "{}",+ json!({"type":"message","id":"later","timestamp":"2026-08-07T17:00:03Z","message":{"role":"user","content":[{"type":"text","text":"later resumed work"}],"meta":{"source":"user"}}})+ )+ .unwrap();+ let exact = export_transcript_range(+ ManagedHarness::CommandCode,+ temp.path(),+ &cwd,+ Some(&store),+ session,+ Some(&baseline),+ &final_cursor,+ )+ .await+ .unwrap();+ assert_eq!(exact.events.len(), 1);+ assert_eq!(exact.events[0].content, "offline append");++ fs::write(&path, b"{}\n").unwrap();+ let error = export_transcript_delta(+ ManagedHarness::CommandCode,+ temp.path(),+ &cwd,+ Some(&store),+ session,+ &baseline,+ )+ .await+ .unwrap_err();+ assert!(+ error+ .to_string()+ .contains("past the current transcript end")+ || error.to_string().contains("was not found"),+ "{error:#}"+ );+ }++ #[tokio::test]+ async fn transcript_baseline_refuses_oversized_records_before_unbounded_growth() {+ let temp = tempfile::tempdir().unwrap();+ let cwd = temp.path().join("repo");+ fs::create_dir_all(&cwd).unwrap();+ let session = "7c1d5698-204a-4c0f-ae9c-43db7fc4e41f";+ let store = temp.path().join("command-code");+ let bucket = store.join("bucket");+ fs::create_dir_all(&bucket).unwrap();+ let path = bucket.join(format!("{session}.jsonl"));+ let mut bytes = format!(+ "{}\n",+ json!({"type":"session","version":3,"id":session,"timestamp":"2026-08-07T17:00:00Z","cwd":cwd})+ )+ .into_bytes();+ bytes.extend(std::iter::repeat_n(b'x', MAX_SOURCE_RECORD_BYTES + 2));+ fs::write(&path, bytes).unwrap();+ let error = transcript_baseline(+ ManagedHarness::CommandCode,+ temp.path(),+ &cwd,+ Some(&store),+ session,+ )+ .await+ .unwrap_err();+ assert!(error.to_string().contains("record exceeds"), "{error:#}");+ }++ /// An oversized FIRST JSONL record is rejected by the byte caps before any+ /// unbounded allocation: the session-listing paths every degraded+ /// pre/post snapshot uses skip the transcript (a valid header on a later+ /// line is deliberately never reached) instead of reading the record.+ #[tokio::test]+ async fn an_oversized_first_record_is_skipped_by_session_listing_caps() {+ let temp = tempfile::tempdir().unwrap();+ let cwd = temp.path().join("repo");+ fs::create_dir_all(&cwd).unwrap();+ let session = "9c8d7a06-93e4-4a01-b14e-64f2f1f2b3c4";+ let transcripts = temp+ .path()+ .join(".claude/projects")+ .join(cwd.to_string_lossy().replace('/', "-"));+ fs::create_dir_all(&transcripts).unwrap();+ let claude_path = transcripts.join(format!("{session}.jsonl"));+ let oversized_header = json!({+ "sessionId": session,+ "cwd": cwd,+ "padding": "x".repeat(MAX_SOURCE_RECORD_BYTES + 2),+ });+ let valid_header = json!({"sessionId": session, "cwd": cwd});+ {+ use std::io::Write as _;+ let mut file = fs::File::create(&claude_path).unwrap();+ writeln!(file, "{oversized_header}").unwrap();+ writeln!(file, "{valid_header}").unwrap();+ }+ assert!(+ list_native_sessions(ManagedHarness::Claude, temp.path(), &cwd, None, 10)+ .await+ .unwrap()+ .is_empty(),+ "a transcript whose first record exceeds the header cap is skipped, header or not"+ );++ let store = temp.path().join("command-code");+ let bucket = store.join("bucket");+ fs::create_dir_all(&bucket).unwrap();+ let command_code_path = bucket.join(format!("{session}.jsonl"));+ {+ use std::io::Write as _;+ let mut file = fs::File::create(&command_code_path).unwrap();+ writeln!(+ file,+ "{}",+ json!({+ "type":"session","version":3,"id":session,+ "timestamp":"2026-08-07T17:00:00Z","cwd":cwd,+ "padding":"y".repeat(MAX_EVENT_BYTES + 2),+ })+ )+ .unwrap();+ }+ assert!(+ list_native_sessions(+ ManagedHarness::CommandCode,+ temp.path(),+ &cwd,+ Some(&store),+ 10+ )+ .await+ .unwrap()+ .is_empty(),+ "the tighter first-line cap rejects before the source-record cap is even reached"+ );+ }++ #[tokio::test]+ async fn exact_empty_range_stays_empty_after_later_append() {+ let temp = tempfile::tempdir().unwrap();+ let cwd = temp.path().join("repo");+ fs::create_dir_all(&cwd).unwrap();+ let session = "7c1d5698-204a-4c0f-ae9c-43db7fc4e41e";+ let store = temp.path().join("command-code");+ let bucket = store.join("bucket");+ fs::create_dir_all(&bucket).unwrap();+ let path = bucket.join(format!("{session}.jsonl"));+ fs::write(+ &path,+ format!(+ "{}\n",+ json!({"type":"session","version":3,"id":session,"timestamp":"2026-08-07T17:00:00Z","cwd":cwd})+ ),+ )+ .unwrap();+ let baseline = transcript_baseline(+ ManagedHarness::CommandCode,+ temp.path(),+ &cwd,+ Some(&store),+ session,+ )+ .await+ .unwrap();+ let final_cursor = baseline.clone();+ use std::io::Write as _;+ writeln!(+ fs::OpenOptions::new().append(true).open(&path).unwrap(),+ "{}",+ json!({"type":"message","id":"later","message":{"role":"user","content":[{"type":"text","text":"later resumed work"}]}})+ )+ .unwrap();+ let exact = export_transcript_range(+ ManagedHarness::CommandCode,+ temp.path(),+ &cwd,+ Some(&store),+ session,+ Some(&baseline),+ &final_cursor,+ )+ .await+ .unwrap();+ assert!(exact.events.is_empty());+ }++ #[test]+ fn jsonl_scan_stops_at_source_byte_cap() {+ let temp = tempfile::tempdir().unwrap();+ let path = temp.path().join("oversized.jsonl");+ let record = format!(+ "{}\n",+ json!({"type":"system","padding":"x".repeat(MAX_SOURCE_RECORD_BYTES / 2)})+ );+ let mut file = fs::File::create(&path).unwrap();+ use std::io::Write as _;+ for _ in 0..(MAX_SOURCE_BYTES / record.len() + 2) {+ file.write_all(record.as_bytes()).unwrap();+ }+ let error = export_jsonl(ManagedHarness::CommandCode, &path, "session", None).unwrap_err();+ assert!(error.to_string().contains("source-byte limit"), "{error:#}");+ }++ #[test]+ fn exported_transcript_caps_count_and_serialized_bytes() {+ let event = NewWorkstreamEvent {+ event_id: "event".into(),+ agent: AgentKind::ClaudeCode,+ native_session_id: "session".into(),+ source_record_id: None,+ kind: WorkstreamEventKind::Message,+ role: Some("user".into()),+ content: "x".repeat(MAX_EVENT_BYTES),+ occurred_at: None,+ metadata: Value::Null,+ };+ let too_many = ExportedTranscript {+ events: vec![event.clone(); MAX_EXPORTED_EVENTS + 1],+ ..ExportedTranscript::default()+ };+ assert!(+ validate_export_bounds(too_many)+ .unwrap_err()+ .to_string()+ .contains("event recovery limit")+ );+ let too_large = ExportedTranscript {+ events: vec![event; MAX_EXPORTED_BYTES / MAX_EVENT_BYTES + 1],+ ..ExportedTranscript::default()+ };+ assert!(+ validate_export_bounds(too_large)+ .unwrap_err()+ .to_string()+ .contains("byte recovery limit")+ );+ }+#[tokio::test]async fn opencode_candidate_discovery_is_newest_first_and_checkout_scoped() {let temp = tempfile::tempdir().unwrap();@@ -5012,6 +5969,36 @@ mod tests {assert!(second.losses.is_empty());}+ #[test]+ fn opencode_sql_scan_stops_at_source_byte_cap_before_json_parse() {+ let home = tempfile::tempdir().unwrap();+ let db = opencode_db(home.path(), None);+ fs::create_dir_all(db.parent().unwrap()).unwrap();+ let connection = Connection::open(&db).unwrap();+ connection+ .execute_batch(+ "CREATE TABLE message(id TEXT PRIMARY KEY, session_id TEXT, data TEXT);\+ CREATE TABLE part(id TEXT PRIMARY KEY, message_id TEXT, session_id TEXT, \+ time_updated INTEGER, data TEXT);",+ )+ .unwrap();+ connection+ .execute(+ "INSERT INTO message VALUES ('m1', 's1', ?1)",+ [json!({"role":"user"}).to_string()],+ )+ .unwrap();+ let oversized = "x".repeat(MAX_SOURCE_BYTES + 1);+ connection+ .execute(+ "INSERT INTO part VALUES ('p1', 'm1', 's1', 1, ?1)",+ [oversized],+ )+ .unwrap();+ let error = export_opencode(home.path(), None, "s1", None).unwrap_err();+ assert!(error.to_string().contains("source-byte limit"), "{error:#}");+ }+#[test]fn opencode2_updated_tracks_session_v2_rows() {let home = tempfile::tempdir().unwrap();@@ -5883,6 +6870,43 @@ mod tests {/// Every step payload is an undocumented protobuf blob, so the ledger for/// this harness comes from hook capture. The failure has to say so.+ #[test]+ fn transcript_interval_identity_is_ordered_and_semantic() {+ let event = |id: &str, role: &str, timestamp: &str, content: &str| NewWorkstreamEvent {+ event_id: id.into(),+ agent: AgentKind::ClaudeCode,+ native_session_id: "session".into(),+ source_record_id: None,+ kind: WorkstreamEventKind::Message,+ role: Some(role.into()),+ content: content.into(),+ occurred_at: Some(timestamp.into()),+ metadata: json!({}),+ };+ let first = event("one", "user", "2026-10-07T10:00:00Z", "hello");+ let second = event("two", "assistant", "2026-10-07T10:00:01Z", "world");+ let expected = transcript_interval_digests(&[first.clone(), second.clone()]);+ assert_eq!(+ transcript_interval_digests(&[first.clone(), second.clone()]),+ expected+ );+ assert_ne!(+ transcript_interval_digests(&[second.clone(), first.clone()]),+ expected,+ "row order is part of interval identity"+ );+ for changed in [+ event("one", "assistant", "2026-10-07T10:00:00Z", "hello"),+ event("one", "user", "2026-10-07T10:00:02Z", "hello"),+ event("one", "user", "2026-10-07T10:00:00Z", "changed"),+ ] {+ assert_ne!(+ transcript_interval_digests(&[changed, second.clone()]),+ expected+ );+ }+ }+#[tokio::test]async fn antigravity_transcript_export_explains_why_it_is_unavailable() {let temp = tempfile::TempDir::new().unwrap();diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.mdindex 65a664af..9664eb36 100644--- a/docs/ARCHITECTURE.md+++ b/docs/ARCHITECTURE.md@@ -68,7 +68,12 @@ from hook paths.2. Server's hook router sanitises the payload (the only path fromuntrusted text into the store), assigns an [`ObservationKind`], andenqueues a `WriteCmd` to the writer actor. For native keyed events, the- project-scoped key and observation commit together. The key is marked+ project-scoped key and observation commit together. Ordinary keys expire+ after 30 days; validated `recovery_<55 hex>` keys are bound to the actual+ session, agent, event kind, and sanitized event identity and retained until+ session/project deletion, so a writer cannot preclaim another recovery+ event and an old journal cannot duplicate observations after ordinary+ cleanup. The key is markedcomplete only after downstream processing: an incomplete replay resumeswiki/handoff effects without another observation, while a completed replayis acknowledged and skipped. A bounded per-project/key gate serializes an@@ -268,8 +273,15 @@ death remains bounded by the renewable lease expiry. An explicit`--force-unlock` recovery expires and replaces a selected active lease in thesame writer transaction, but only when its durable operator attribution equalsthe new run's attribution; the informational `host:pid` lease label is never an-authorization key. The old run can no longer heartbeat or finish. See [Managed-cross-harness workstreams](managed-workstreams.md).+authorization key. The old run can no longer heartbeat or finish normally.+`POST /workstream/runs/{run_id}/recover/finish` is the narrower outage path: it+reuses the normal owner and project-write checks and admits an expired run only+when no newer run in that workstream superseded it. A finished retry validates+only durable transcript/native/cursor/exit identity and does not regenerate a+repository checkpoint. Degraded resumed sessions persist a validated native+source cursor before launch, and bounded recovery export reads only the delta+on a blocking worker. See [Managed cross-harness+workstreams](managed-workstreams.md).## Hook event vocabulary@@ -674,17 +686,18 @@ hook install-mcp commitcheckpoints restore-page llm-testforget-sweep lint curatorauto-improve-report auto-improve finalize-session-pending-writes embed generate-auth-token-setup-agent bootstrap install-instructions-install-skills reorg purge-project-rename-project move-project move-session-uninstall upgrade auth-user completions handoffs-purge-session compact api-key-export-okf message doctor-backfill project reclaim-ledger-versions-repair-backfill-timestamps server backup-agents-restore-agents profile+recover pending-writes embed+generate-auth-token setup-agent bootstrap+install-instructions install-skills reorg+purge-project rename-project move-project+move-session uninstall upgrade+auth user completions+handoffs purge-session compact+api-key export-okf message+doctor backfill project+reclaim-ledger-versions repair-backfill-timestamps+server backup-agents restore-agents+profile```Run `ai-memory --help` for the full tree.diff --git a/docs/cookbook.md b/docs/cookbook.mdindex 4efa173b..d5c0551c 100644--- a/docs/cookbook.md+++ b/docs/cookbook.md@@ -387,6 +387,7 @@ ai-memory resume --search auth # pick a matching workstream in this checkoai-memory resume --all # pick from every linked checkout on this machineai-memory workstreams # list this checkout's managed workstreamsai-memory status # counts, paths, health+ai-memory recover # replay outage spool/transcripts safelyai-memory list-projects # every workspace/project the server knows aboutai-memory doctor # is every harness that ran here captured?ai-memory backfill # import prior local history into an empty store@@ -406,9 +407,23 @@ ai-memory serve # run the serverautomatically when the server returns), an existing MCP registrationdegrades to no-recall for the session, and the child's exit code isreturned. What you lose for that run is the workstream lease/context,- transcript import, and handoff delivery; sessions resume only through an- explicit native selector because no lease means no mutual exclusion. See- [Degraded offline+ immediate transcript import, and handoff delivery; sessions resume only+ through an explicit native selector because no lease means no mutual+ exclusion. An explicit resume is journaled with its exact pre-launch native+ cursor and recovery imports only the appended delta; a cursor that cannot be+ proven requires manual repair. A fresh run is journaled only when one new+ native session can be correlated; ambiguity prints a manual backfill action.+ Once the server returns, run `ai-memory recover`: for transcript-capable+ harnesses it quarantines correlated spool entries, drains unrelated events,+ and replays the bounded exact transcript through sanitized `/hook/batch`;+ Antigravity has no supported transcript exporter, so its degraded run is+ journaled as spool-only and clears only after correlated hook evidence has a+ durable-delivery confirmation; recovery never calls its exporter. Empty or+ previously correlated-loss spool state stays journaled for manual repair. Owner-authorized recovery+ finishes an expired original managed run only when it was not superseded. A full journal refuses+ new entries instead of evicting old work, and dropped spool events make the+ command fail while retaining every journal entry for a later clean pass.+ Re-running is safe. See [Degraded offlinelaunches](managed-workstreams.md#degraded-offline-launches). To fail insteadof degrading, pass `--require-server` (or set `run.require_server = true` /`AI_MEMORY_RUN_REQUIRE_SERVER=true`). For a planned outage, considerdiff --git a/docs/external-lifecycle.md b/docs/external-lifecycle.mdindex bb359823..9967ce33 100644--- a/docs/external-lifecycle.md+++ b/docs/external-lifecycle.md@@ -130,7 +130,10 @@ to unkeyed ingestion, so validate them before sending.The server claims the key and observation in one transaction. A completed replayis skipped; a pending replay resumes downstream work without inserting another-observation. Keys expire after **30 days**. This is bounded retry protection, not+observation. Ordinary client keys expire after **30 days**. The reserved,+server-validated `recovery_<55 hex>` namespace is used only by native transcript+recovery/backfill and remains bound to its actual stored session until that+session or project is deleted. This is bounded retry protection, notpermanent deduplication or an exactly-once guarantee for every wiki side effect.### Ordered batches and recoverydiff --git a/docs/managed-workstreams.md b/docs/managed-workstreams.mdindex 6394349e..b9151a6a 100644--- a/docs/managed-workstreams.md+++ b/docs/managed-workstreams.md@@ -512,21 +512,43 @@ the server first (any HTTP answer counts as reachable, so an older buildwithout `/healthz` still passes), prints one loud warning naming the serverURL and what the degraded run means, and launches the harness anyway:-- **Lost this run**: no workstream lease or cross-harness context packet, no- transcript import into the ledger, and no cross-harness handoff delivery.- Nothing that needs the server runs: no prepare, lease, link, heartbeat,- context fetch, status check, finish, or import.-- **Still works**: the harness's ai-memory lifecycle hooks (if installed) keep- capturing — events spool locally and drain automatically when the server- returns. An existing MCP registration degrades to no-recall for the session- rather than blocking it (the same principle as `docs/mcp-install.md`'s- optional-mode entries: an unreachable memory server costs you recall, not- the session).+- **Unavailable during the run**: no workstream lease or cross-harness context+ packet, immediate transcript import, or handoff delivery. Nothing that needs+ the server runs: no prepare, lease, link, heartbeat, context fetch, status+ check, finish, or import.+- **Recovered afterward**: recovery requires both the `ai-memory run` client+ and server to be upgraded to the release that provides `ai-memory recover`+ and modern per-item hook acknowledgements. The harness's ai-memory lifecycle+ hooks (if installed) spool events locally. After the child exits, the launcher+ discovers its native session and appends only bounded metadata (never+ transcript content or secrets) to an owner-only local recovery journal. Run+ `ai-memory recover` when the server returns: under the spool's exclusive+ drain lock it first quarantines entries correlated to each journaled session,+ drains unrelated entries, then re-exports the exact native transcript and+ replays session start, content, and session end through the existing+ sanitized `/hook/batch` backfill path. A successful replay deletes the+ quarantine; failure restores it. Antigravity's native store has no supported+ transcript exporter, so its degraded launch is still journaled explicitly as+ spool-only and recovery never invokes its exporter. Its correlated entries+ are quarantined and the journal clears as `recovered-via-spool` only after+ their durable delivery was confirmed.+ An empty spool, a correlated prior drop, or a launch with no hook evidence+ stays journaled with a manual-repair status. Stable recovery ingest keys retain the+ exact session, agent, event kind, and sanitized event identity until+ session/project deletion; an ordinary writer cannot preclaim one for another+ event. Ordinary hook keys still expire after 30 days. An existing MCP+ registration still degrades to no-recall during the outage rather than+ blocking the session.- **Sessions**: because no lease exists there is no mutual exclusion againstanother launcher in the same checkout, so a degraded launch never adopts or- resumes a session implicitly. An explicit native session selector- (`claude --resume <id>`, `codex continue`, …) still resumes — you named the- session — and everything else starts a fresh session.+ resumes a session implicitly. An explicit native session selector still+ resumes the named session; before such a resume the launcher persists the+ native adapter's exact pre-launch cursor, and recovery exports only records+ after it. If that cursor cannot be proven, the launch remains usable but is+ not journaled for automatic replay. Otherwise the launcher snapshots+ checkout-local candidates before spawn and journals only when exactly one new+ session appears. Ambiguous concurrent launches are not journaled and print+ the manual `backfill --force --session <id>` recovery action.- **Auto-wire**: the first-launch hook install still happens (it is local andidempotent, and capture must spool offline), but no MCP entry is registeredfor the now-unreachable server and no completion sentinel is written, so the@@ -539,17 +561,39 @@ URL and what the degraded run means, and launches the harness anyway:not an error.When the child exits, the launcher prints that the run was not recorded on the-server and how many hook events remain spooled locally (oldest first by age).-Spooled events are bounded: they are dropped after the configured number of-failed drain passes (8 by default), 7 days of age, or the 10,000-file spool-cap. For a planned outage set `AI_MEMORY_HOOK_SPOOL_MAX_ATTEMPTS=0` to disable-only the attempt-count drops (see `docs/install.md`).+server, how many hook events remain spooled locally, and that the run was+journaled for `ai-memory recover`. The journal is capped at 256 entries;+once full it refuses new entries with an actionable warning instead of evicting+unrecovered work. It is written atomically as owner-only local state; Unix lock opens use+`O_NOFOLLOW` and the atomic replacement rechecks the destination immediately+before rename. Windows applies an owner-only ACL and rejects observed symlinks,+but reparse-point race hardening still requires the PR `windows` CI lane before+merge. Spooled events remain separately+bounded: they are dropped after the configured number of failed drain passes+(8 by default), 7 days of age, or the 10,000-file spool cap. For a planned+outage set `AI_MEMORY_HOOK_SPOOL_MAX_ATTEMPTS=0` to disable only the+attempt-count drops (see `docs/install.md`).If the server dies *during* a run — after the lease was acquired — the exit-code is still preserved: the unimported transcript is reported as a warning-with the exact `ai-memory finalize-session` command that repairs the record-once the server is back, and the orphaned lease expires on its own within 90-seconds.+code is still preserved. The journal records the original run id, native+session locator, absolute store path, and exit code; repository branch and path+state are not persisted. Recovery retries validate the exact ordered semantic+transcript interval and native identity, and omit a regenerated checkpoint, so+repository changes during downtime are not claimed as the failed run's state and+a successful finish cannot create a second checkpoint event.+`ai-memory recover` uses the owner-authorized `/recover/finish` route to resume+an expired original run only when no newer run superseded it; ordering is the+explicit `(started_at, SQLite insertion order)` tuple, so timestamp ties cannot+revive an older run. Another operator, a superseded run, and any finished-run+payload that is not an exact replay of already indexed events/cursor/exit state+remain refused.+Successful entries are removed, while failed entries remain with a bounded+sanitized reason. Each journal request and the whole journal pass have time+budgets. A spool pass that drops anything is reported incomplete and keeps all+journal entries; correlated entries are quarantined before that pass and can+only be deleted after a later clean pass and exact transcript replay. If the+server is still unreachable, `recover` exits successfully+and prints the remaining spool/journal counts and startup guidance.To restore the strict behavior (fail with the usual "could not reach …"diagnosis and never start the agent), pass `--require-server`, setdiff --git a/docs/security-boundaries.md b/docs/security-boundaries.mdindex 3f43e2d6..416a330b 100644--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@ -72,6 +72,7 @@ boundary not yet built.| 11f | Native hook executable-path integrity after self-replacement (#1027) | `ai-memory-cli/src/commands/render_shared.rs` `hook_embedded_exe_path` is the single executable resolver for every persisted native hook form (POSIX/Windows command, Windows exec, Zero, ZCode, OpenCode 2 assistant capture): it prefers an existing literal path, strips Linux's trailing ` (deleted)` procfs marker only when the stripped installed binary exists, and otherwise falls back to the platform command name. Upgrade keeps the already-loaded config and refreshes in-process | `render_shared.rs` `embedded_exe_strips_the_marker_when_the_stripped_path_exists`, `embedded_exe_preserves_a_real_file_whose_name_has_the_marker_suffix`, `embedded_exe_falls_back_when_the_resolved_path_is_missing`, `hook_embedded_exe_path_names_an_existing_file_without_marker`; `upgrade_e2e.rs` `native_upgrade_refreshes_hooks_without_a_deleted_path_or_losing_cli_data_dir` reproduces the replacement window through the shipped binary, plants the corrupt suffix, and proves refresh removes it while preserving a CLI-only data dir | STRONG on Linux reproduction plus platform-neutral resolver/render coverage; Windows path rendering remains covered by its existing command/exec tests and full Windows CI || 11g | Cross-project file attribution cannot bypass destination capture or server routing (#932) | `ai-memory-hooks/src/capture_policy.rs` `absolute_file_tool_paths` accepts only bounded, recognized, absolute file-tool candidates; `ai-memory-cli/src/commands/hook.rs` `file_tool_routing_cwd` requires every path to prove the same repository/marker boundary and resolves that destination before capture policy, server profile, scope hints, or spool credentials. Relative, mixed-project, unsupported, and arbitrary non-project paths keep the payload cwd | `hook.rs` `absolute_file_target_uses_destination_scope_and_capture_policy` (hostile content under the destination exclusion is never spooled; a public control is routed there), `absolute_file_target_cannot_escape_a_destination_server_profile` (unregistered destination profile drops instead of falling back to source/install default), `file_target_routing_refuses_relative_external_and_mixed_project_paths` | STRONG for native `ai-memory hook`; generated and compatibility hooks retain cwd-only routing || 11h | Docker-wrapper capture-policy capability (#1002) | `bin/ai-memory` routes ordinary Linux/macOS `install-hooks` and upgrade refreshes through the stable checksum-verified native host client, so the hook process executes the capture-policy boundary in 11b before spool/network. Only an explicit `AI_MEMORY_HOOK_PLATFORM=posix|windows` retains the documented script compatibility path and its warning; `AI_MEMORY_NATIVE_BIN` may not point back to the wrapper | `scripts/check-native-packaging.sh` `Checking host-launch wrapper routing`: a default allowlist install reaches the fake native client while a Docker executable that always fails proves no container fallback occurred; the explicit `posix` control reaches fake Docker and adds no native invocation; the self-referential native override is refused | STRONG on the shipped Linux/macOS wrapper. Windows and remote/manual script installers remain explicitly outside capture-policy v1 |+| 11j | Offline recovery journal and transcript replay | `ai-memory-cli/src/commands/recovery.rs` keeps bounded locators, ordered semantic transcript digests, pre-launch cursors and persistent correlated-delivery/loss evidence in an atomic journal, refuses symlinked path components, sanitizes/caps failure reasons, and bounds each replay plus the whole pass; `hook_spool` coordinates enqueue and quarantine with short filesystem locks, canonically resolves partial scope from the recorded cwd, bounds per-file reads and current global occupancy across the main spool and recovery directories, sweeps completed tombstones, and holds correlated events outside drain until replay succeeds or restores them on failure; degraded resumes use `ai-memory-workstream`'s adapter-validated delta export, whose blocking parse and baseline scans are capped by record, event-count, source-byte and serialized-byte totals; `backfill::import_exact_session` chunks by item and byte limits and requires per-item durable acknowledgements; `run.rs` journals every session launch, including explicit spool-only evidence for unsupported exporters, and journals a resumed export only after capturing its exact pre-launch cursor; `workstream.rs` `/recover/finish` retains owner/project authorization, rejects superseded runs, and validates a finished retry by durable transcript/native/cursor/exit identity without regenerating its checkpoint; validated recovery ingest keys bind session, agent, event kind and sanitized event identity until session purge | `transcript.rs` `persisted_baseline_exports_only_offline_append_and_rejects_rewrite`, parser/baseline cap and ordered-semantic digest tests; CLI `run_offline.rs` real degraded spool-only and overlap/idempotency tests; `recovery.rs` `spool_only_without_correlated_evidence_never_auto_clears`, `a_dropped_spool_event_stays_lost_on_the_next_empty_pass`, loss/time-budget tests; `hook_spool.rs` partial-scope late enqueue, completion-race, occupancy churn/active-cap, bounded-read and tombstone-sweep tests, plus interval isolation (`completed_tombstone_keeps_a_later_resume_of_the_same_session` proves a post-interval resume hook is delivered, `completing_one_recovery_interval_leaves_the_other_intervals_entries_intact` proves one interval's completion never consumes another's evidence and ambiguous entries fail closed into the main spool); `backfill.rs` `two_sequential_recovery_intervals_on_one_resumed_session_replay_exactly_once` proves the second interval's content and session-end are stored with end-of-session effects while reruns stay pure replays; `transcript.rs` `an_oversized_first_record_is_skipped_by_session_listing_caps` proves the first-record byte caps bite before allocation; store `insert_observation_ingest_dedups_on_key` rejects cross-session and changed-event preclaims; hooks `expired_run_recovery_is_owner_only_and_refuses_superseded_runs` proves changed transcript refusal and changed-checkpoint no-op | STRONG on Unix integration coverage; Unix lock opens use no-follow and atomic replacement rechecks the destination. Windows owner-only ACL/reparse and lock semantics require the PR `windows` lane || 11i | OpenCode executable probing and generated-config cleanup (#1097) | `ai-memory-cli/src/config.rs` captures the full process environment once in typed `RuntimeEnv`; `run.rs` overlays explicit `run --env` values with platform key semantics and uses that same effective environment for executable resolution, probing, and launch without rereading process environment; `opencode_dialect.rs` invokes only direct argv `--version`, caps output and time, and rejects malformed/future majors. Automatic launch skips only OpenCode on probe failure; explicit OpenCode fails. `install_hooks.rs` quarantines the incompatible sibling before reading its exact ownership markers, restores unowned content with an atomic no-replace link, and never deletes or overwrites a concurrent replacement; `install_mcp.rs` owns the shared dual-schema endpoint/token inference used by both installers, rejects conflicting generated V1/V2 entries before mutation, and migrates an alternate-major entry only when endpoint, generated shape, and absent-or-Authorization-only headers match, preserving custom headers and ambiguous entries; auto-wire migrates MCP before hooks infer from that exact config path and skips hook mutation when transition preflight fails | `run.rs` `automatic_opencode_probe_failure_keeps_another_harness_available`, `opencode_executable_probe_selects_v2_for_launch_and_autowire`, effective-child-environment PATH casing controls, fake V1/V2/env-derived-major/malformed/future/timeout executable tests, `setup-agent` V1/V2 no-probe integration controls; `install_hooks.rs` `opencode_plugin_cleanup_removes_only_exact_owned_incompatible_sibling`, `opencode_plugin_cleanup_preserves_a_concurrent_user_replacement`, `opencode_plugin_restore_moves_quarantine_when_destination_is_free`, `opencode_plugin_restore_preserves_an_existing_replacement`, `opencode_plugin_restore_does_not_replace_a_file_created_at_restore_window`, `opencode_plugin_cleanup_does_not_escape_custom_target_directory`; `install_mcp.rs` `opencode_install_reuses_owned_endpoint_and_token_across_major_transitions`, `opencode_install_refuses_conflicting_owned_transition_entries`, `opencode_install_does_not_infer_from_a_user_owned_transition_entry`, `opencode_mcp_migration_moves_only_owned_alternate_shape`, `opencode_mcp_migration_preserves_ambiguous_user_entry`, `opencode_mcp_migration_preserves_custom_headers`, `opencode_mcp_migration_accepts_only_historical_generated_headers` | STRONG for Unix process probes and platform-neutral ownership gates; exact Windows executable resolution remains covered by `run.rs` resolver tests and requires the PR `windows` label || 12 | Network/auth posture | `config.rs` loopback `DEFAULT_BIND`; `serve.rs` `validate_http_exposure`, `require_allowed_host`; `auth.rs` `require_bearer`; `serve.rs` mounts `/identity` inside the host and machine-auth gates even with web disabled; `nix/nixos-module.nix` treats only literal `127.0.0.1`/`::1` as loopback, requires exactly one secret environment source for other binds, keeps secret auth and `llm_headers` values out of the Nix store, and brackets IPv6 in `ExecStart` | `serve.rs` host-guard (missing→400 / forged→403), non-loopback-requires-token; `auth.rs` wrong-token 401; `serve.rs` `machine_identity_remains_authenticated_with_web_disabled_and_expired_keys` (wrong and expired keys refused, valid machine key accepted, web remains disabled); `flake.nix` `nixos-module-eval` refuses `localhost` and `0.0.0.0` without a secret source, refuses two secret sources, `settings.auth` secrets, and `settings.llm_headers`, with literal IPv4/IPv6 loopback and one-source controls | STRONG || 12b | Managed-workstream provenance privacy, stable replay identity and server event namespace | `ai-memory-core/src/workstream.rs` `scrub_workstream_provenance`; `ai-memory-hooks/src/workstream.rs` `finish_run` reserved client-id check and `sanitize_events` before immutable segments; `ai-memory-store/src/workstream.rs` `finish_run` stable identity check and `stored_event`, using an explicitly supplied configured sanitizer; no changes to the typed observation boundary | `store/tests/suite/workstream_provenance.rs` `workstream_provenance_is_bounded_scrubbed_and_idempotent_before_storage` (hostile and oversized source ids, unknown metadata, legitimate correlation, retry and distinct event ids), `workstream_provenance_legacy_reads_strip_secrets_and_unknown_metadata` (search and tail, malformed/oversized legacy JSON), `workstream_provenance_reused_event_id_retains_data_and_refuses_identity_change` (first indexed version retained, identity refusal and batch rollback), `workstream_provenance_replay_survives_sanitizer_changes` (new extra patterns and removal of a built-in-pattern allowlist exemption, with identity-refusal control), `workstream_provenance_replay_preserves_legacy_rows_above_bounds`, `workstream_provenance_finished_run_ignores_late_events`, `workstream_provenance_retry_with_marker_matching_pattern_is_idempotent`; `hooks` `workstream_provenance_finish_scrubs_before_raw_segment_and_search` (custom sanitizer, immutable segment, SQL recovery and finished-run no-op), `workstream_provenance_client_cannot_use_server_event_ids` (reserved ids refused before raw writes, normal client event and server-generated checkpoint/loss controls); `cli/tests/suite/workstream_provenance.rs` (text/JSON and old server DTOs) | STRONG: privacy, stable-identity and reserved-namespace regressions fail with their guards bypassed; an isolated allowlist bypass fails on the unknown key, and bypassing the raw-id search authorization returns 200 where its test requires 403. No provenance is emitted in startup packets, and old raw segments are not rewritten |