Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat(run): recover missed history after server downtime

AkitaOnRails96d8d21a335a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 09477532..863bfdf8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Added
+- Added `ai-memory recover` and a bounded local recovery journal for server
+ outages. Offline managed launches now record only uniquely correlated native
+ session locators; recovery quarantines overlapping spool entries before exact
+ `/hook/batch` transcript replay, journals harnesses without transcript export
+ as spool-only without calling their exporter, retains bounded recovery dedup
+ keys with their sessions,
+ fails closed on terminal drops, and lets only the original owner finish an
+ expired unsuperseded managed run through exact idempotent replay. Backfill
+ and recovery replay now use reserved `recovery_*` idempotency keys, so
+ re-running `backfill --force` over sessions imported by ai-memory ≤2.6.0 (or
+ plain-backfilling a recovery-replayed session) can duplicate those older
+ observations. (#1125)
+
### Fixed
- Fixed the hook spool charging a spooled event's retry budget while the
server was unreachable: an endpoint-level delivery failure (connection
@@ -18,8 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
file cap and 7-day spool TTL bounds are unchanged. Also corrected the
stale `hooks/_lib.sh` comment that claimed the backlog is drained at
session boundaries only (a piggyback drain also runs after any
- successful 2xx POST). (#NNN)
-
+ successful 2xx POST). (#1121)
## [2.6.0] - 2026-10-07
diff --git a/Cargo.lock b/Cargo.lock
index ff1c7e62..3df5a291 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -57,6 +57,7 @@ dependencies = [
"fs2",
"jiff",
"jsonc-parser",
+ "libc",
"reqwest 0.12.28",
"rmcp",
"rstest",
diff --git a/Cargo.toml b/Cargo.toml
index 3ea82278..c7d81eaa 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -126,6 +126,7 @@ notify = "8"
notify-debouncer-full = "0.6"
sysinfo = "0.32"
fs2 = "0.4"
+libc = "0.2"
# Windows NTFS file-index (inode equivalent) for own-write detection.
winapi-util = "0.1"
diff --git a/README.md b/README.md
index 91b4cb28..568dd0fb 100644
--- a/README.md
+++ b/README.md
@@ -373,9 +373,9 @@ no separate `install-hooks`/`install-mcp` step to forget), it wires the right
project scope by construction, and it adds cross-harness *session* continuity on
top of shared memory. Everything is idempotent and one-time per harness and
config home. If the server is unreachable, `run` warns and launches anyway
-with local capture spooling (see [Degraded offline
-launches](docs/managed-workstreams.md#degraded-offline-launches)); pass
-`--require-server` to fail closed instead.
+with local capture spooling; run `ai-memory recover` after the server returns
+(see [Degraded offline launches](docs/managed-workstreams.md#degraded-offline-launches)).
+Pass `--require-server` to fail closed instead.
```bash
ai-memory run claude
diff --git a/crates/ai-memory-cli/Cargo.toml b/crates/ai-memory-cli/Cargo.toml
index bb26825c..7e0c6b51 100644
--- a/crates/ai-memory-cli/Cargo.toml
+++ b/crates/ai-memory-cli/Cargo.toml
@@ -48,6 +48,7 @@ dirs.workspace = true
figment.workspace = true
flate2.workspace = true
fs2.workspace = true
+libc.workspace = true
# `transport-streamable-http-client-reqwest` pulls reqwest in with no TLS backend
# (`__reqwest = ["dep:reqwest"]` and nothing more), so the bridge could not reach an
# `https://` MCP server at all.
diff --git a/crates/ai-memory-cli/src/cli.rs b/crates/ai-memory-cli/src/cli.rs
index 75a72da7..9b6a06f1 100644
--- a/crates/ai-memory-cli/src/cli.rs
+++ b/crates/ai-memory-cli/src/cli.rs
@@ -258,6 +258,10 @@ pub enum Command {
AutoImprove(AutoImproveArgs),
/// Manually finalize the latest open session for one agent in this project.
FinalizeSession(FinalizeSessionArgs),
+ /// Recover history a server outage missed: drain the local hook spool,
+ /// replay degraded-run transcripts through the hook backfill path, and
+ /// finish managed runs whose transcript import failed. Safe to re-run.
+ Recover(RecoverArgs),
/// Review, approve, or reject staged auto-improvement proposals.
PendingWrites(PendingWritesArgs),
/// Compute + store embeddings for every latest page (M9).
@@ -2450,6 +2454,14 @@ fn parse_finalizable_agent(s: &str) -> Result<ai_memory_core::AgentKind, String>
Ok(kind)
}
+/// Arguments for `recover`.
+#[derive(Debug, Args)]
+pub struct RecoverArgs {
+ /// Print the machine-readable report as JSON on stdout.
+ #[arg(long)]
+ pub json: bool,
+}
+
/// Arguments for `finalize-session`.
#[derive(Debug, Args)]
pub struct FinalizeSessionArgs {
diff --git a/crates/ai-memory-cli/src/commands/backfill.rs b/crates/ai-memory-cli/src/commands/backfill.rs
index 4c2d8a22..74195a85 100644
--- a/crates/ai-memory-cli/src/commands/backfill.rs
+++ b/crates/ai-memory-cli/src/commands/backfill.rs
@@ -41,7 +41,8 @@ use serde::{Deserialize, Serialize};
use ai_memory_core::{NewWorkstreamEvent, WorkstreamEventKind};
use ai_memory_workstream::{
LaunchRoots, ManagedHarness, build_launch_plan_with_env, export_transcript,
- list_native_sessions, wait_for_transcript_flush,
+ export_transcript_delta, export_transcript_range, list_native_sessions,
+ wait_for_transcript_flush,
};
use super::doctor::{SCANNED_HARNESSES, relocated_session_dir};
@@ -60,6 +61,7 @@ const PER_HARNESS_SCAN_LIMIT: usize = 200;
/// One `POST /hook/batch` request carries at most this many events — matches the
/// server's `MAX_HOOK_BATCH_ITEMS` so a batch is never rejected for size.
const HOOK_BATCH_ITEMS: usize = 256;
+const HOOK_BATCH_BYTES: usize = 8 * 1024 * 1024;
/// The extension label backfilled non-user events (assistant/tool) are recorded
/// under, mirroring the companion importer's replay-through-`/hook` mechanism.
@@ -430,7 +432,7 @@ pub(crate) async fn collect_local_sessions_with(
/// One item in a `POST /hook/batch` request: the full hook URL (whose query the
/// server parses for event/agent/scope/session) plus the JSON body.
-#[derive(Debug, Serialize)]
+#[derive(Clone, Debug, Serialize)]
struct HookItem {
url: String,
body: serde_json::Value,
@@ -439,8 +441,61 @@ struct HookItem {
/// The server's `/hook/batch` acknowledgement (subset we act on).
#[derive(Debug, Deserialize)]
struct HookBatchAck {
- /// Contiguous leading prefix committed, oldest-first.
accepted: usize,
+ results: Vec<HookBatchResult>,
+ #[serde(default)]
+ accepted_indices: Option<Vec<usize>>,
+ #[serde(default)]
+ failed_index: Option<usize>,
+}
+
+#[derive(Debug, Deserialize)]
+struct HookBatchResult {
+ index: usize,
+ outcome: String,
+}
+
+fn acknowledged_items(ack: &HookBatchAck, items: &[&HookItem]) -> Result<Vec<usize>> {
+ let indices = ack
+ .accepted_indices
+ .clone()
+ .unwrap_or_else(|| (0..ack.accepted).collect());
+ if ack.accepted > items.len()
+ || indices.iter().any(|index| *index >= items.len())
+ || indices.windows(2).any(|pair| pair[0] >= pair[1])
+ || ack.accepted
+ != indices
+ .iter()
+ .enumerate()
+ .take_while(|(pos, index)| *pos == **index)
+ .count()
+ || ack.results.len() != indices.len()
+ || ack
+ .failed_index
+ .is_some_and(|index| index >= items.len() || indices.contains(&index))
+ {
+ bail!("invalid hook acknowledgement; keeping transcript for retry");
+ }
+ for (result, index) in ack.results.iter().zip(&indices) {
+ if result.index != *index {
+ bail!("hook acknowledgement result indices disagree");
+ }
+ let is_end = reqwest::Url::parse(&items[*index].url)?
+ .query_pairs()
+ .any(|(key, value)| key == "event" && value == "session-end");
+ match result.outcome.as_str() {
+ "stored" | "replayed" => {}
+ "ignored_end" if is_end => {}
+ "resumed" => bail!("hook processing resumed; rerun to confirm durable replay"),
+ outcome => {
+ bail!("hook item {index} was not durably stored ({outcome}); keeping transcript")
+ }
+ }
+ }
+ if let Some(index) = ack.failed_index {
+ bail!("hook processing failed at item {index}; keeping transcript for retry");
+ }
+ Ok(indices)
}
/// Import one native session by replaying its transcript through `/hook`, the
@@ -463,29 +518,105 @@ async fn import_one(
build_launch_plan_with_env(session.harness, None, Vec::new(), None, &[], Some(roots))
.ok()
.and_then(|plan| plan.session_dir);
- // These are historical sessions, so the flush wait is a quick no-op; ignore
- // its result and read whatever is on disk.
- let _ = wait_for_transcript_flush(
- session.harness,
+ import_exact_session(
+ endpoint,
+ workspace,
+ project,
home,
cwd,
- session_dir.as_deref(),
+ session.harness,
&session.native_session_id,
+ session_dir.as_deref(),
+ None,
)
- .await;
- let transcript = export_transcript(
- session.harness,
+ .await
+}
+
+#[allow(clippy::too_many_arguments)]
+pub(crate) async fn import_exact_session_range(
+ endpoint: &ServerEndpoint,
+ workspace: &str,
+ project: &str,
+ home: &Path,
+ cwd: &Path,
+ harness: ManagedHarness,
+ native_session_id: &str,
+ recovery_interval_id: &str,
+ session_dir: Option<&Path>,
+ source_cursor: Option<&str>,
+ final_cursor: &str,
+ expected_digests: &[String],
+) -> Result<usize> {
+ let _ = wait_for_transcript_flush(harness, home, cwd, session_dir, native_session_id).await;
+ let transcript = export_transcript_range(
+ harness,
home,
cwd,
- session_dir.as_deref(),
- &session.native_session_id,
+ session_dir,
+ native_session_id,
+ source_cursor,
+ final_cursor,
+ )
+ .await
+ .with_context(|| format!("reading the {} transcript", harness.as_str()))?;
+ let actual = ai_memory_workstream::transcript_interval_digests(&transcript.events);
+ if actual != expected_digests {
+ bail!("native transcript range no longer matches its exact ordered semantic identity");
+ }
+ import_exported_transcript(
+ endpoint,
+ workspace,
+ project,
+ harness,
+ native_session_id,
+ Some(recovery_interval_id),
+ transcript,
+ )
+ .await
+}
+
+#[allow(clippy::too_many_arguments)]
+pub(crate) async fn import_exact_session(
+ endpoint: &ServerEndpoint,
+ workspace: &str,
+ project: &str,
+ home: &Path,
+ cwd: &Path,
+ harness: ManagedHarness,
+ native_session_id: &str,
+ session_dir: Option<&Path>,
+ source_cursor: Option<&str>,
+) -> Result<usize> {
+ let _ = wait_for_transcript_flush(harness, home, cwd, session_dir, native_session_id).await;
+ let transcript = if let Some(cursor) = source_cursor {
+ export_transcript_delta(harness, home, cwd, session_dir, native_session_id, cursor).await
+ } else {
+ export_transcript(harness, home, cwd, session_dir, native_session_id, None).await
+ }
+ .with_context(|| format!("reading the {} transcript", harness.as_str()))?;
+ import_exported_transcript(
+ endpoint,
+ workspace,
+ project,
+ harness,
+ native_session_id,
None,
+ transcript,
)
.await
- .with_context(|| format!("reading the {} transcript", session.harness.as_str()))?;
+}
- let sid = &session.native_session_id;
- let agent = session.harness.agent_kind().as_str();
+async fn import_exported_transcript(
+ endpoint: &ServerEndpoint,
+ workspace: &str,
+ project: &str,
+ harness: ManagedHarness,
+ native_session_id: &str,
+ recovery_interval_id: Option<&str>,
+ transcript: ai_memory_workstream::ExportedTranscript,
+) -> Result<usize> {
+ let sid = native_session_id;
+ let agent = harness.agent_kind().as_str();
let resolved = resolve_occurred_at(&transcript.events);
let mut items = Vec::with_capacity(transcript.events.len() + 2);
items.push(hook_item(
@@ -495,13 +626,13 @@ async fn import_one(
agent,
"session-start",
sid,
- &format!("{sid}:session-start"),
+ &replay_ingest_key(recovery_interval_id, sid, "session-start", "boundary"),
None,
serde_json::json!({ "session_id": sid, "occurred_at": resolved.earliest }),
)?);
let mut content = 0usize;
for (event, occurred_at) in transcript.events.iter().zip(&resolved.per_event) {
- if let Some(mapped) = map_event(sid, event, occurred_at.as_deref()) {
+ if let Some(mapped) = map_event(recovery_interval_id, sid, event, occurred_at.as_deref()) {
items.push(hook_item(
endpoint,
workspace,
@@ -523,7 +654,7 @@ async fn import_one(
agent,
"session-end",
sid,
- &format!("{sid}:session-end"),
+ &replay_ingest_key(recovery_interval_id, sid, "session-end", "boundary"),
None,
serde_json::json!({ "session_id": sid, "occurred_at": resolved.latest }),
)?);
@@ -621,6 +752,7 @@ struct MappedEvent {
/// body so the imported observation is dated at the transcript's own event
/// time rather than at import time.
fn map_event(
+ recovery_interval_id: Option<&str>,
session_id: &str,
event: &NewWorkstreamEvent,
occurred_at: Option<&str>,
@@ -630,7 +762,12 @@ fn map_event(
return None;
}
let role = event.role.as_deref().unwrap_or("");
- let ingest_key = format!("{session_id}:{}", event.event_id);
+ let ingest_key = replay_ingest_key(
+ recovery_interval_id,
+ session_id,
+ event.kind.as_str(),
+ &event.event_id,
+ );
match event.kind {
WorkstreamEventKind::Message if role == "user" || role == "human" => Some(MappedEvent {
event: "user-prompt".to_string(),
@@ -680,6 +817,27 @@ fn first_line(content: &str) -> String {
}
}
+fn replay_ingest_key(
+ recovery_interval_id: Option<&str>,
+ session_id: &str,
+ kind: &str,
+ event_id: &str,
+) -> String {
+ use sha2::{Digest as _, Sha256};
+ let mut hasher = Sha256::new();
+ for value in [
+ recovery_interval_id.unwrap_or_default(),
+ session_id,
+ kind,
+ event_id,
+ ] {
+ hasher.update(value.len().to_be_bytes());
+ hasher.update(value.as_bytes());
+ }
+ let digest = format!("{:x}", hasher.finalize());
+ format!("recovery_{}", &digest[..55])
+}
+
/// Build a `/hook` URL whose query the server parses for event/agent/scope. The
/// URL is data the batch item carries, not the request target (that is
/// `/hook/batch`), so it is built against this server's origin exactly as a live
@@ -720,32 +878,58 @@ fn hook_item(
})
}
-/// POST the events in server-sized batches, retrying the unaccepted suffix of a
-/// partially accepted batch (per-source rate limiting can skip a tail) before
-/// failing. Events are oldest-first, so a leading-prefix ack is safe to resume.
+/// POST events in server-sized batches. Recovery requires a modern per-item
+/// acknowledgement and fails closed on partial or terminal-drop outcomes.
+fn hook_batch_end(items: &[HookItem], offset: usize) -> Result<usize> {
+ let mut end = offset;
+ let mut bytes = 2_usize;
+ while end < items.len() && end - offset < HOOK_BATCH_ITEMS {
+ let item_bytes = serde_json::to_vec(&items[end])?.len().saturating_add(1);
+ if item_bytes > HOOK_BATCH_BYTES {
+ bail!("one sanitized hook item exceeds the recovery request byte limit");
+ }
+ if end > offset && bytes.saturating_add(item_bytes) > HOOK_BATCH_BYTES {
+ break;
+ }
+ bytes = bytes.saturating_add(item_bytes);
+ end += 1;
+ }
+ Ok(end)
+}
+
async fn post_hook_items(endpoint: &ServerEndpoint, items: &[HookItem]) -> Result<()> {
- for chunk in items.chunks(HOOK_BATCH_ITEMS) {
- let mut offset = 0usize;
+ let mut offset = 0_usize;
+ while offset < items.len() {
+ let end = hook_batch_end(items, offset)?;
+ let chunk = &items[offset..end];
let mut stalled = 0u32;
- while offset < chunk.len() {
- let batch: Vec<&HookItem> = chunk[offset..].iter().collect();
+ loop {
+ let batch: Vec<&HookItem> = chunk.iter().collect();
let ack: HookBatchAck = post_json(endpoint, "/hook/batch", &batch)
.await
.context("replaying transcript events through /hook/batch")?;
- if ack.accepted == 0 {
+ let accepted = acknowledged_items(&ack, &batch)?;
+ if accepted.len() == batch.len() {
+ break;
+ }
+ if accepted.is_empty() {
stalled += 1;
if stalled >= 5 {
bail!(
- "server accepted none of a hook batch after {stalled} attempts \
+ "server acknowledged none of a hook batch after {stalled} attempts \
(rate limited or saturated); rerun to resume"
);
}
tokio::time::sleep(Duration::from_millis(200 * u64::from(stalled))).await;
continue;
}
- stalled = 0;
- offset += ack.accepted.min(chunk.len() - offset);
+ bail!(
+ "server acknowledged only {} of {} transcript items; keeping transcript for a safe retry",
+ accepted.len(),
+ batch.len()
+ );
}
+ offset = end;
}
Ok(())
}
@@ -888,6 +1072,7 @@ mod tests {
#[test]
fn user_message_maps_to_the_canonical_user_prompt() {
let m = map_event(
+ None,
"sid",
&event(WorkstreamEventKind::Message, Some("user"), "do the thing"),
Some("2026-09-10T12:00:00Z"),
@@ -901,14 +1086,545 @@ mod tests {
assert_eq!(m.body["prompt"], "do the thing");
assert_eq!(m.body["occurred_at"], "2026-09-10T12:00:00Z");
assert_eq!(
- m.ingest_key, "sid:evt-1",
+ m.ingest_key,
+ replay_ingest_key(None, "sid", "message", "evt-1"),
"ingest key is stable per source event"
);
+ assert_eq!(m.ingest_key.len(), 64);
+ assert!(
+ m.ingest_key
+ .bytes()
+ .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-' || byte == b'_')
+ );
+ }
+
+ #[test]
+ fn replay_keys_are_deterministic_valid_and_bounded_for_long_native_ids() {
+ let session = "native:".repeat(200);
+ let event = "event/with spaces:".repeat(200);
+ let first = replay_ingest_key(None, &session, "message", &event);
+ let second = replay_ingest_key(None, &session, "message", &event);
+ assert_eq!(first, second);
+ assert_eq!(first.len(), 64);
+ assert!(first.starts_with("recovery_"));
+ assert!(
+ first
+ .bytes()
+ .all(|byte| byte.is_ascii_alphanumeric() || byte == b'-' || byte == b'_')
+ );
+ assert_ne!(
+ first,
+ replay_ingest_key(None, &session, "tool_call", &event)
+ );
+ let other_interval = replay_ingest_key(Some("journal-2"), &session, "message", &event);
+ assert_ne!(first, other_interval);
+ assert_eq!(
+ other_interval,
+ replay_ingest_key(Some("journal-2"), &session, "message", &event),
+ "one journal interval must replay stably"
+ );
+ }
+
+ #[test]
+ fn recovery_hook_batches_respect_count_and_serialized_byte_limits() {
+ let endpoint = ServerEndpoint::for_hook_target("http://127.0.0.1:1".into(), None);
+ let make = |size| HookItem {
+ url: endpoint.build_url("/hook?event=user-prompt"),
+ body: serde_json::json!({"prompt": "x".repeat(size)}),
+ };
+ let items = (0..HOOK_BATCH_ITEMS + 1)
+ .map(|_| make(1))
+ .collect::<Vec<_>>();
+ assert_eq!(hook_batch_end(&items, 0).unwrap(), HOOK_BATCH_ITEMS);
+
+ let large = vec![make(HOOK_BATCH_BYTES / 2), make(HOOK_BATCH_BYTES / 2)];
+ assert_eq!(hook_batch_end(&large, 0).unwrap(), 1);
+ assert!(
+ hook_batch_end(&[make(HOOK_BATCH_BYTES)], 0)
+ .unwrap_err()
+ .to_string()
+ .contains("exceeds")
+ );
+ }
+
+ #[test]
+ fn acknowledgement_requires_results_and_rejects_terminal_drops() {
+ let endpoint = ServerEndpoint::for_hook_target("http://127.0.0.1:1".into(), None);
+ let item = hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ "user-prompt",
+ "sid",
+ "key",
+ None,
+ serde_json::json!({"session_id": "sid", "prompt": "hello"}),
+ )
+ .unwrap();
+ let items = [&item];
+ for outcome in [
+ "dropped_policy",
+ "dropped_subagent",
+ "dropped_unauthorized",
+ "dropped_collision",
+ "dropped_invalid",
+ "resumed",
+ ] {
+ let ack = HookBatchAck {
+ accepted: 1,
+ results: vec![HookBatchResult {
+ index: 0,
+ outcome: outcome.into(),
+ }],
+ accepted_indices: None,
+ failed_index: None,
+ };
+ assert!(acknowledged_items(&ack, &items).is_err(), "{outcome}");
+ }
+ let legacy = HookBatchAck {
+ accepted: 1,
+ results: Vec::new(),
+ accepted_indices: None,
+ failed_index: None,
+ };
+ assert!(acknowledged_items(&legacy, &items).is_err());
+ }
+
+ #[test]
+ fn acknowledgement_allows_stored_replayed_and_only_terminal_ignored_end() {
+ let endpoint = ServerEndpoint::for_hook_target("http://127.0.0.1:1".into(), None);
+ let make = |event| {
+ hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ event,
+ "sid",
+ "key",
+ None,
+ serde_json::json!({"session_id": "sid"}),
+ )
+ .unwrap()
+ };
+ let prompt = make("user-prompt");
+ let end = make("session-end");
+ for outcome in ["stored", "replayed"] {
+ let ack = HookBatchAck {
+ accepted: 1,
+ results: vec![HookBatchResult {
+ index: 0,
+ outcome: outcome.into(),
+ }],
+ accepted_indices: Some(vec![0]),
+ failed_index: None,
+ };
+ assert_eq!(acknowledged_items(&ack, &[&prompt]).unwrap(), vec![0]);
+ }
+ let ignored = HookBatchAck {
+ accepted: 1,
+ results: vec![HookBatchResult {
+ index: 0,
+ outcome: "ignored_end".into(),
+ }],
+ accepted_indices: Some(vec![0]),
+ failed_index: None,
+ };
+ assert!(acknowledged_items(&ignored, &[&prompt]).is_err());
+ assert_eq!(acknowledged_items(&ignored, &[&end]).unwrap(), vec![0]);
+ }
+
+ #[tokio::test]
+ async fn real_hook_router_replay_is_idempotent_and_reports_outcomes() {
+ use ai_memory_core::{ActiveProject, IngestMetrics, Sanitizer, SessionId};
+ use ai_memory_hooks::{
+ HookState, IngestGates, IngestRateLimiter, ProjectCacheStore, SubagentSessionSet,
+ hook_router,
+ };
+ use ai_memory_store::Store;
+ use ai_memory_wiki::Wiki;
+ use std::sync::Arc;
+ use tower::ServiceExt as _;
+
+ let temp = tempfile::tempdir().unwrap();
+ let store = Store::open(temp.path()).unwrap();
+ let workspace_id = store.writer.get_or_create_workspace("ws").await.unwrap();
+ let project_id = store
+ .writer
+ .get_or_create_project(workspace_id, "project", None)
+ .await
+ .unwrap();
+ let state = HookState {
+ workspace_id,
+ project_id,
+ writer: store.writer.clone(),
+ reader: store.reader.clone(),
+ wiki: Wiki::new(temp.path(), store.writer.clone()).unwrap(),
+ consolidator: None,
+ sanitizer: Sanitizer::default(),
+ project_cache: Arc::new(tokio::sync::Mutex::new(ProjectCacheStore::default())),
+ active_project: ActiveProject::new(),
+ ingest_metrics: Arc::new(IngestMetrics::default()),
+ ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(16)),
+ ingest_gates: IngestGates::default(),
+ ingest_rate: Arc::new(tokio::sync::Mutex::new(IngestRateLimiter::disabled())),
+ consolidate_on_session_end: false,
+ session_consolidation_notify: None,
+ profile_notify: None,
+ capture_assistant_enabled: false,
+ claim_handoff_on_session_start: true,
+ create_handoff_on_session_end: false,
+ subagent_sessions: Arc::new(tokio::sync::Mutex::new(SubagentSessionSet::default())),
+ home_dir: None,
+ trusted_proxy_identity: false,
+ per_user_slots: false,
+ mid_session_routing: ai_memory_core::MidSessionRouting::default(),
+ profile: ai_memory_core::profile::ProfileSettings::default(),
+ };
+ let endpoint = ServerEndpoint::for_hook_target("http://localhost".into(), None);
+ let sid = "native-session-with-a-very-long-identifier-that-would-break-the-old-key";
+ let recovery_interval_id = Some("journal");
+ let items = vec![
+ hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ "session-start",
+ sid,
+ &replay_ingest_key(recovery_interval_id, sid, "session-start", "boundary"),
+ None,
+ serde_json::json!({"session_id": sid, "cwd": "/repo"}),
+ )
+ .unwrap(),
+ hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ "user-prompt",
+ sid,
+ &replay_ingest_key(recovery_interval_id, sid, "message", "event-1"),
+ None,
+ serde_json::json!({"session_id": sid, "cwd": "/repo", "prompt": "same prompt"}),
+ )
+ .unwrap(),
+ hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ "session-end",
+ sid,
+ &replay_ingest_key(recovery_interval_id, sid, "session-end", "boundary"),
+ None,
+ serde_json::json!({"session_id": sid, "cwd": "/repo"}),
+ )
+ .unwrap(),
+ ];
+ let spool = crate::commands::hook_spool::spool_dir(temp.path());
+ let overlapping = hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ "user-prompt",
+ sid,
+ "random_live_key",
+ None,
+ serde_json::json!({"session_id": sid, "cwd": "/repo", "prompt": "same prompt"}),
+ )
+ .unwrap();
+ crate::commands::hook_spool::enqueue(
+ &spool,
+ &crate::commands::hook_spool::entry_for(
+ overlapping.url,
+ overlapping.body.to_string(),
+ None,
+ false,
+ ),
+ )
+ .unwrap();
+ let quarantine = crate::commands::hook_spool::quarantine_session_entries(
+ &Config::default(),
+ &spool,
+ "journal",
+ 0,
+ u64::MAX,
+ &endpoint.identity(),
+ "claude-code",
+ "ws",
+ "project",
+ Path::new("/repo"),
+ sid,
+ )
+ .unwrap();
+ assert_eq!(crate::commands::hook_spool::spool_len(&spool), 0);
+
+ let body = serde_json::to_vec(&items).unwrap();
+ let first = hook_router(state.clone())
+ .oneshot(
+ axum::http::Request::post("/hook/batch")
+ .header("content-type", "application/json")
+ .body(axum::body::Body::from(body.clone()))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(first.status(), axum::http::StatusCode::OK);
+ let first: HookBatchAck = serde_json::from_slice(
+ &axum::body::to_bytes(first.into_body(), usize::MAX)
+ .await
+ .unwrap(),
+ )
+ .unwrap();
+ assert_eq!(
+ acknowledged_items(&first, &items.iter().collect::<Vec<_>>())
+ .unwrap()
+ .len(),
+ 3
+ );
+ let second = hook_router(state.clone())
+ .oneshot(
+ axum::http::Request::post("/hook/batch")
+ .header("content-type", "application/json")
+ .body(axum::body::Body::from(body))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ let second: HookBatchAck = serde_json::from_slice(
+ &axum::body::to_bytes(second.into_body(), usize::MAX)
+ .await
+ .unwrap(),
+ )
+ .unwrap();
+ assert!(
+ second
+ .results
+ .iter()
+ .any(|result| result.outcome == "replayed")
+ );
+ assert!(
+ second
+ .results
+ .iter()
+ .any(|result| result.outcome == "ignored_end")
+ );
+ let observations = store
+ .reader
+ .observations_for_session(SessionId::from_native(sid))
+ .await
+ .unwrap();
+ assert_eq!(observations.len(), 3);
+ assert_eq!(
+ observations
+ .iter()
+ .filter(|observation| observation.body == "same prompt")
+ .count(),
+ 1
+ );
+ quarantine.complete().unwrap();
+ }
+
+ /// Two sequential offline intervals on one resumed native session: the
+ /// second interval's journal id gives its boundary and content keys their
+ /// own identity, so its session-end is STORED (re-closing the session with
+ /// its end-of-session effects) instead of being short-circuited as a
+ /// replay of the first interval's end, while re-running either interval
+ /// stays a pure replay.
+ #[tokio::test]
+ async fn two_sequential_recovery_intervals_on_one_resumed_session_replay_exactly_once() {
+ use ai_memory_core::{ActiveProject, IngestMetrics, ObservationKind, Sanitizer, SessionId};
+ use ai_memory_hooks::{
+ HookState, IngestGates, IngestRateLimiter, ProjectCacheStore, SubagentSessionSet,
+ hook_router,
+ };
+ use ai_memory_store::Store;
+ use ai_memory_wiki::Wiki;
+ use std::sync::Arc;
+ use tower::ServiceExt as _;
+
+ let temp = tempfile::tempdir().unwrap();
+ let store = Store::open(temp.path()).unwrap();
+ let workspace_id = store.writer.get_or_create_workspace("ws").await.unwrap();
+ let project_id = store
+ .writer
+ .get_or_create_project(workspace_id, "project", None)
+ .await
+ .unwrap();
+ let state = HookState {
+ workspace_id,
+ project_id,
+ writer: store.writer.clone(),
+ reader: store.reader.clone(),
+ wiki: Wiki::new(temp.path(), store.writer.clone()).unwrap(),
+ consolidator: None,
+ sanitizer: Sanitizer::default(),
+ project_cache: Arc::new(tokio::sync::Mutex::new(ProjectCacheStore::default())),
+ active_project: ActiveProject::new(),
+ ingest_metrics: Arc::new(IngestMetrics::default()),
+ ingest_semaphore: Arc::new(tokio::sync::Semaphore::new(16)),
+ ingest_gates: IngestGates::default(),
+ ingest_rate: Arc::new(tokio::sync::Mutex::new(IngestRateLimiter::disabled())),
+ consolidate_on_session_end: false,
+ session_consolidation_notify: None,
+ profile_notify: None,
+ capture_assistant_enabled: false,
+ claim_handoff_on_session_start: true,
+ create_handoff_on_session_end: false,
+ subagent_sessions: Arc::new(tokio::sync::Mutex::new(SubagentSessionSet::default())),
+ home_dir: None,
+ trusted_proxy_identity: false,
+ per_user_slots: false,
+ mid_session_routing: ai_memory_core::MidSessionRouting::default(),
+ profile: ai_memory_core::profile::ProfileSettings::default(),
+ };
+ let endpoint = ServerEndpoint::for_hook_target("http://localhost".into(), None);
+ let sid = "native-session-resumed-across-two-offline-intervals";
+ let interval_items = |journal: &str, prompt: &str, event_id: &str| {
+ vec![
+ hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ "session-start",
+ sid,
+ &replay_ingest_key(Some(journal), sid, "session-start", "boundary"),
+ None,
+ serde_json::json!({"session_id": sid, "cwd": "/repo"}),
+ )
+ .unwrap(),
+ hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ "user-prompt",
+ sid,
+ &replay_ingest_key(Some(journal), sid, "message", event_id),
+ None,
+ serde_json::json!({"session_id": sid, "cwd": "/repo", "prompt": prompt}),
+ )
+ .unwrap(),
+ hook_item(
+ &endpoint,
+ "ws",
+ "project",
+ "claude-code",
+ "session-end",
+ sid,
+ &replay_ingest_key(Some(journal), sid, "session-end", "boundary"),
+ None,
+ serde_json::json!({"session_id": sid, "cwd": "/repo"}),
+ )
+ .unwrap(),
+ ]
+ };
+ let router_state = state.clone();
+ let post = move |items: Vec<HookItem>| {
+ let state = router_state.clone();
+ async move {
+ let response = hook_router(state)
+ .oneshot(
+ axum::http::Request::post("/hook/batch")
+ .header("content-type", "application/json")
+ .body(axum::body::Body::from(serde_json::to_vec(&items).unwrap()))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(response.status(), axum::http::StatusCode::OK);
+ let ack: HookBatchAck = serde_json::from_slice(
+ &axum::body::to_bytes(response.into_body(), usize::MAX)
+ .await
+ .unwrap(),
+ )
+ .unwrap();
+ assert_eq!(
+ acknowledged_items(&ack, &items.iter().collect::<Vec<_>>())
+ .unwrap()
+ .len(),
+ items.len()
+ );
+ ack.results
+ .into_iter()
+ .map(|result| result.outcome)
+ .collect::<Vec<_>>()
+ }
+ };
+
+ let first = interval_items("journal-a", "first interval prompt", "event-1");
+ assert_eq!(
+ post(first.clone()).await,
+ vec!["stored", "stored", "stored"],
+ "the first offline interval recovers exactly once"
+ );
+ let second = interval_items("journal-b", "second interval prompt", "event-2");
+ assert_eq!(
+ post(second.clone()).await,
+ vec!["stored", "stored", "stored"],
+ "the resumed interval's content and session-end must be stored, not replayed"
+ );
+
+ let observations = store
+ .reader
+ .observations_for_session(SessionId::from_native(sid))
+ .await
+ .unwrap();
+ assert_eq!(observations.len(), 6, "two start/end pairs and two prompts");
+ assert_eq!(
+ observations
+ .iter()
+ .filter(|observation| observation.body == "first interval prompt")
+ .count(),
+ 1
+ );
+ assert_eq!(
+ observations
+ .iter()
+ .filter(|observation| observation.body == "second interval prompt")
+ .count(),
+ 1
+ );
+ assert_eq!(
+ observations
+ .iter()
+ .filter(|observation| observation.kind == ObservationKind::SessionEnd)
+ .count(),
+ 2,
+ "each interval's session-end is its own stored observation"
+ );
+
+ for rerun in [first, second] {
+ assert!(
+ rerun
+ .iter()
+ .zip(post(rerun.clone()).await)
+ .all(|(item, outcome)| {
+ let is_end = reqwest::Url::parse(&item.url)
+ .unwrap()
+ .query_pairs()
+ .any(|(key, value)| key == "event" && value == "session-end");
+ outcome == "replayed" || (is_end && outcome == "ignored_end")
+ }),
+ "re-running a recovered interval is a pure replay"
+ );
+ }
+ let observations = store
+ .reader
+ .observations_for_session(SessionId::from_native(sid))
+ .await
+ .unwrap();
+ assert_eq!(observations.len(), 6, "replays add no observations");
}
#[test]
fn assistant_and_tool_events_map_to_backfill_extension_observations() {
let a = map_event(
+ None,
"sid",
&event(
WorkstreamEventKind::Message,
@@ -931,6 +1647,7 @@ mod tests {
);
let t = map_event(
+ None,
"sid",
&event(WorkstreamEventKind::ToolCall, None, "grep foo"),
None,
@@ -1050,6 +1767,7 @@ mod tests {
fn empty_and_boundary_events_are_dropped() {
assert!(
map_event(
+ None,
"sid",
&event(WorkstreamEventKind::Message, Some("user"), " "),
None,
@@ -1063,7 +1781,7 @@ mod tests {
WorkstreamEventKind::Annotation,
] {
assert!(
- map_event("sid", &event(kind, None, "boundary"), None).is_none(),
+ map_event(None, "sid", &event(kind, None, "boundary"), None).is_none(),
"{kind:?} is not session content"
);
}
diff --git a/crates/ai-memory-cli/src/commands/hook.rs b/crates/ai-memory-cli/src/commands/hook.rs
index ed2228c2..a666d2e7 100644
--- a/crates/ai-memory-cli/src/commands/hook.rs
+++ b/crates/ai-memory-cli/src/commands/hook.rs
@@ -902,7 +902,7 @@ where
let entry =
hook_spool::entry_for(event_url, payload.clone(), effective_token, oidc_present)
.routed_to(profile_name);
- if hook_spool::enqueue(&spool, &entry).is_err() {
+ if hook_spool::enqueue_with_config(&spool, &entry, Some(&spool_policy)).is_err() {
eprintln!(
"ai-memory hook warning: failed to spool lifecycle event; capture for this event was skipped"
);
@@ -1591,6 +1591,7 @@ mod tests {
hook_spool::DrainResult::default(),
hook_spool::DrainResult {
sent: 12,
+ durable: 12,
remaining: 0,
dropped: 0,
},
@@ -1611,6 +1612,7 @@ mod tests {
let clean = drain_report(&hook_spool::LockedDrainResult::Drained(
hook_spool::DrainResult {
sent: 5,
+ durable: 5,
remaining: 0,
dropped: 0,
},
@@ -1618,6 +1620,7 @@ mod tests {
let lossy = drain_report(&hook_spool::LockedDrainResult::Drained(
hook_spool::DrainResult {
sent: 0,
+ durable: 0,
remaining: 0,
dropped: 7,
},
@@ -1651,6 +1654,7 @@ mod tests {
let report = drain_report(&hook_spool::LockedDrainResult::Drained(
hook_spool::DrainResult {
sent: 1,
+ durable: 1,
remaining: 4,
dropped: 0,
},
diff --git a/crates/ai-memory-cli/src/commands/hook_capture.rs b/crates/ai-memory-cli/src/commands/hook_capture.rs
index 9adb4e91..fc31cae2 100644
--- a/crates/ai-memory-cli/src/commands/hook_capture.rs
+++ b/crates/ai-memory-cli/src/commands/hook_capture.rs
@@ -808,6 +808,13 @@ pub async fn post_hook(
/// edge auth hop over the whole batch instead of paying it per event.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum BatchOutcome {
+ /// Modern server returned one terminal result for each acknowledged item.
+ Detailed {
+ /// Per-item processing outcomes in request order.
+ outcomes: Vec<String>,
+ /// Item that failed after any earlier acknowledged items.
+ failed_index: Option<usize>,
+ },
/// Server committed the leading `usize` items (contiguous prefix, oldest
/// first). Equals the request length on full success; a smaller value means
/// the server stopped on that item (fail-fast) — the caller deletes the
@@ -872,7 +879,12 @@ pub async fn post_batch(
if status.is_success() {
match resp.json::<serde_json::Value>().await {
Ok(v) => {
- if let Some(indices) = accepted_indices(&v) {
+ if let Some(outcomes) = batch_outcomes(&v) {
+ BatchOutcome::Detailed {
+ outcomes,
+ failed_index: failed_index(&v),
+ }
+ } else if let Some(indices) = accepted_indices(&v) {
BatchOutcome::AcceptedIndices {
indices,
failed_index: failed_index(&v),
@@ -915,6 +927,18 @@ pub async fn post_batch(
}
}
+fn batch_outcomes(v: &serde_json::Value) -> Option<Vec<String>> {
+ let results = v.get("results")?.as_array()?;
+ let mut outcomes = Vec::with_capacity(results.len());
+ for (position, result) in results.iter().enumerate() {
+ if result.get("index")?.as_u64()? as usize != position {
+ return None;
+ }
+ outcomes.push(result.get("outcome")?.as_str()?.to_owned());
+ }
+ Some(outcomes)
+}
+
fn failed_index(v: &serde_json::Value) -> Option<usize> {
v.get("failed_index")?.as_u64().map(|n| n as usize)
}
diff --git a/crates/ai-memory-cli/src/commands/hook_spool.rs b/crates/ai-memory-cli/src/commands/hook_spool.rs
index 77843b4e..73816322 100644
--- a/crates/ai-memory-cli/src/commands/hook_spool.rs
+++ b/crates/ai-memory-cli/src/commands/hook_spool.rs
@@ -66,8 +66,23 @@ const MAX_BATCH_ITEMS: usize = 256;
/// `DefaultBodyLimit` with margin for JSON framing. A chunk always carries at
/// least one event even if that event alone exceeds this.
const MAX_BATCH_BYTES: usize = 8 * 1024 * 1024;
+const RECOVERY_DIR: &str = ".recovery";
+const RECOVERY_MATCH_FILE: &str = ".active";
+const RECOVERY_COMPLETED_FILE: &str = ".completed";
+const SPOOL_CAPACITY_LOCK_FILE: &str = ".capacity.lock";
+const SPOOL_COORD_LOCK_WAIT: Duration = Duration::from_millis(5);
+const MAX_SPOOL_ENTRY_BYTES: u64 = 10 * 1024 * 1024;
+const RECOVERY_TOMBSTONE_AGE_MS: u64 = 24 * 60 * 60 * 1000;
+const MAX_RECOVERY_TOMBSTONES: usize = 512;
+const MAX_RECOVERY_SWEEP_DIRS: usize = 1_024;
static ENQUEUE_SEQ: AtomicU64 = AtomicU64::new(0);
+#[cfg(test)]
+static PAUSE_ENQUEUE_AFTER_MAIN_RENAME: std::sync::atomic::AtomicBool =
+ std::sync::atomic::AtomicBool::new(false);
+#[cfg(test)]
+static ENQUEUE_REACHED_RECOVERY_WINDOW: std::sync::atomic::AtomicBool =
+ std::sync::atomic::AtomicBool::new(false);
/// How a spooled event authenticates to the server when drained.
#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq, Debug)]
@@ -188,7 +203,7 @@ pub fn spool_health(spool: &Path) -> SpoolHealth {
retries_total: 0,
};
for (_, path) in &files {
- if let Ok(bytes) = std::fs::read(path)
+ if let Ok(bytes) = read_spool_entry_bytes(path)
&& let Ok(entry) = serde_json::from_slice::<SpoolEntry>(&bytes)
{
health.retries_total += u64::from(entry.attempts);
@@ -207,12 +222,9 @@ pub fn spool_health(spool: &Path) -> SpoolHealth {
/// entry enqueued at the Unix epoch — see [`spool_health`].
fn created_ms_from_name(file_name: &str) -> Option<u64> {
let (stamp, rest) = file_name.split_once('-')?;
- // Require the full zero-padded width the writer emits, so a name that
- // merely *starts* with digits is not accepted.
if stamp.len() != 13 || !stamp.bytes().all(|b| b.is_ascii_digit()) {
return None;
}
- // And require the remainder to look like `{pid}-{seq}.json`.
if !rest.ends_with(".json") || !rest.contains('-') {
return None;
}
@@ -235,11 +247,21 @@ fn now_ms() -> u64 {
///
/// # Errors
/// Returns an error only when the spool file cannot be written.
+#[cfg(test)]
pub fn enqueue(spool: &Path, entry: &SpoolEntry) -> std::io::Result<()> {
+ enqueue_with_config(spool, entry, None)
+}
+
+pub fn enqueue_with_config(
+ spool: &Path,
+ entry: &SpoolEntry,
+ config: Option<&crate::config::Config>,
+) -> std::io::Result<()> {
create_spool_dir(spool)?;
let seq = ENQUEUE_SEQ.fetch_add(1, Ordering::Relaxed);
+ let recovery_key = recovery_coordinate_key_for_entry(entry, config)?;
let name = format!(
- "{:013}-{}-{seq:016x}.json",
+ "{:013}-{}-{seq:016x}-{recovery_key}.json",
entry.created_ms,
std::process::id()
);
@@ -247,8 +269,53 @@ pub fn enqueue(spool: &Path, entry: &SpoolEntry) -> std::io::Result<()> {
let final_path = spool.join(&name);
let bytes = serde_json::to_vec(entry)?;
write_private(&tmp, &bytes)?;
+ let capacity_lock = open_spool_capacity_lock(spool)?;
+ let started = Instant::now();
+ loop {
+ match capacity_lock.try_lock_exclusive() {
+ Ok(()) => break,
+ Err(error)
+ if is_drain_lock_busy_error(&error)
+ && started.elapsed() < SPOOL_COORD_LOCK_WAIT =>
+ {
+ std::thread::sleep(Duration::from_millis(1));
+ }
+ Err(error) if is_drain_lock_busy_error(&error) => {
+ let _ = std::fs::remove_file(&tmp);
+ return Ok(());
+ }
+ Err(error) => return Err(error),
+ }
+ }
+ if total_spool_entries(spool)? >= MAX_SPOOL_FILES {
+ let _ = std::fs::remove_file(&tmp);
+ eprintln!(
+ "ai-memory: hook-spool at capacity ({MAX_SPOOL_FILES}); dropping one new UNDELIVERED event"
+ );
+ return Ok(());
+ }
std::fs::rename(&tmp, &final_path)?;
- prune_spool_file_count(spool);
+ #[cfg(test)]
+ if PAUSE_ENQUEUE_AFTER_MAIN_RENAME.load(Ordering::SeqCst) {
+ ENQUEUE_REACHED_RECOVERY_WINDOW.store(true, Ordering::SeqCst);
+ while PAUSE_ENQUEUE_AFTER_MAIN_RENAME.load(Ordering::SeqCst) {
+ std::thread::yield_now();
+ }
+ }
+ let coord_lock = open_recovery_coord_lock(spool)?;
+ if coord_lock.try_lock_exclusive().is_ok() {
+ match recovery_target_for_entry(spool, entry, config)? {
+ Some(RecoveryTarget::Active(dir)) => {
+ std::fs::rename(&final_path, dir.join(&name))?;
+ }
+ Some(RecoveryTarget::Completed(_)) => match std::fs::remove_file(&final_path) {
+ Ok(()) => {}
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
+ Err(error) => return Err(error),
+ },
+ None => {}
+ }
+ }
Ok(())
}
@@ -260,6 +327,38 @@ fn create_spool_dir(spool: &Path) -> std::io::Result<()> {
super::path_util::create_private_dir(spool)
}
+fn open_spool_capacity_lock(spool: &Path) -> std::io::Result<File> {
+ create_spool_dir(spool)?;
+ std::fs::OpenOptions::new()
+ .create(true)
+ .truncate(false)
+ .read(true)
+ .write(true)
+ .open(spool.with_extension(SPOOL_CAPACITY_LOCK_FILE))
+}
+
+fn total_spool_entries(spool: &Path) -> std::io::Result<usize> {
+ let mut total = list_entries(spool).map_or(0, |(files, _)| files.len());
+ let recovery = spool.join(RECOVERY_DIR);
+ let directories = match std::fs::read_dir(recovery) {
+ Ok(entries) => entries,
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(total),
+ Err(error) => return Err(error),
+ };
+ for (index, directory) in directories.enumerate() {
+ if index >= MAX_RECOVERY_SWEEP_DIRS {
+ return Ok(MAX_SPOOL_FILES);
+ }
+ let directory = directory?;
+ if directory.file_type()?.is_dir() {
+ total = total.saturating_add(
+ list_entries(&directory.path()).map_or(0, |(files, _)| files.len()),
+ );
+ }
+ }
+ Ok(total)
+}
+
fn write_private(path: &Path, bytes: &[u8]) -> std::io::Result<()> {
use std::io::Write as _;
let mut opts = std::fs::OpenOptions::new();
@@ -315,6 +414,8 @@ impl SpoolEntry {
pub struct DrainResult {
/// Events delivered (server answered 2xx) and removed from the spool.
pub sent: usize,
+ /// Events whose modern per-item acknowledgement proves durable storage.
+ pub durable: usize,
/// Events still queued (failed this pass, or skipped when the budget ran out).
pub remaining: usize,
/// Events discarded as undeliverable (too old or too many failed attempts).
@@ -335,6 +436,587 @@ pub struct DrainLock {
_file: File,
}
+/// Outcome of [`QuarantinedSpool::restore`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum RestoreOutcome {
+ /// Quarantined events were moved back to the plain spool (or the
+ /// quarantine directory no longer existed).
+ Restored,
+ /// The interval was already completed, so its correlated evidence is
+ /// durably delivered and there is nothing left to restore or re-deliver.
+ AlreadyDelivered,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+struct RecoveryMatch {
+ journal_id: String,
+ coordinate_key: String,
+ interval_started_ms: u64,
+ interval_ended_ms: u64,
+ server_identity: String,
+ agent: String,
+ workspace: String,
+ project: String,
+ cwd: String,
+ native_session_id: String,
+}
+
+pub struct QuarantinedSpool {
+ dir: PathBuf,
+ spool: PathBuf,
+ config: crate::config::Config,
+}
+
+#[derive(Debug, Default, Clone, Copy)]
+pub struct CorrelatedDrainResult {
+ pub durable: usize,
+ pub remaining: usize,
+ pub dropped: usize,
+}
+
+impl QuarantinedSpool {
+ pub fn len(&self) -> usize {
+ list_entries(&self.dir).map_or(0, |(files, _)| files.len())
+ }
+
+ pub fn complete(self) -> std::io::Result<()> {
+ if self.dir.exists() {
+ let lock = open_recovery_coord_lock(&self.spool)?;
+ lock.lock_exclusive()?;
+ let active = self.dir.join(RECOVERY_MATCH_FILE);
+ let completed = self.dir.join(RECOVERY_COMPLETED_FILE);
+ let spec = read_recovery_match(&active)?;
+ std::fs::rename(&active, &completed)?;
+ move_late_matching_entries(&self.spool, &self.dir, &self.config, &spec)?;
+ for item in std::fs::read_dir(&self.dir)? {
+ let item = item?;
+ if item.path().extension().and_then(|value| value.to_str()) == Some("json") {
+ std::fs::remove_file(item.path())?;
+ }
+ }
+ sweep_recovery_tombstones(&self.spool)?;
+ }
+ Ok(())
+ }
+
+ pub async fn drain_strict(
+ &self,
+ data_dir: &Path,
+ total_budget: Duration,
+ per_event_timeout: Duration,
+ max_attempts: MaxAttempts,
+ ) -> CorrelatedDrainResult {
+ let Ok((mut files, unreadable)) = list_entries(&self.dir) else {
+ return CorrelatedDrainResult {
+ remaining: 1,
+ ..CorrelatedDrainResult::default()
+ };
+ };
+ files.sort();
+ let client = build_client();
+ let started = Instant::now();
+ let mut oidc_cache = None;
+ let mut result = DrainResult {
+ remaining: unreadable,
+ ..DrainResult::default()
+ };
+ for path in files {
+ if started.elapsed() >= total_budget {
+ result.remaining += 1;
+ continue;
+ }
+ let Some(entry) = load_live_entry(&path, &mut result) else {
+ continue;
+ };
+ if body_is_malformed(&entry) {
+ bump_or_drop(&path, &entry, max_attempts.0, &mut result);
+ continue;
+ }
+ let bearer = entry_bearer(&entry, &client, data_dir, &mut oidc_cache).await;
+ let Some(payload) = batch_payload(&[(path.clone(), entry.clone())]) else {
+ bump_or_drop(&path, &entry, max_attempts.0, &mut result);
+ continue;
+ };
+ let remaining_budget = total_budget.saturating_sub(started.elapsed());
+ let timeout = per_event_timeout.min(remaining_budget);
+ match post_batch(
+ &client,
+ &batch_endpoint(&entry.url),
+ &payload,
+ bearer.as_deref(),
+ timeout,
+ )
+ .await
+ {
+ BatchOutcome::Detailed {
+ outcomes,
+ failed_index: None,
+ } if outcomes.len() == 1 => {
+ let outcome = outcomes[0].as_str();
+ let durable = matches!(outcome, "stored" | "replayed")
+ || outcome == "ignored_end"
+ && entry_event(&entry).as_deref() == Some("session-end");
+ if durable {
+ let _ = std::fs::remove_file(path);
+ result.sent += 1;
+ result.durable += 1;
+ } else if outcome.starts_with("dropped_") {
+ let _ = std::fs::remove_file(path);
+ result.dropped += 1;
+ } else {
+ bump_or_drop(&path, &entry, max_attempts.0, &mut result);
+ }
+ }
+ _ => bump_or_drop(&path, &entry, max_attempts.0, &mut result),
+ }
+ }
+ CorrelatedDrainResult {
+ durable: result.durable,
+ remaining: result.remaining,
+ dropped: result.dropped,
+ }
+ }
+
+ pub fn complete_spool_only(&self) -> std::io::Result<bool> {
+ let lock = open_recovery_coord_lock(&self.spool)?;
+ lock.lock_exclusive()?;
+ let active = self.dir.join(RECOVERY_MATCH_FILE);
+ let completed = self.dir.join(RECOVERY_COMPLETED_FILE);
+ let spec = read_recovery_match(&active)?;
+ move_late_matching_entries(&self.spool, &self.dir, &self.config, &spec)?;
+ if self.len() != 0 {
+ return Ok(false);
+ }
+ std::fs::rename(active, completed)?;
+ Ok(true)
+ }
+
+ pub fn restore(self, spool: &Path) -> std::io::Result<RestoreOutcome> {
+ if !self.dir.exists() {
+ return Ok(RestoreOutcome::Restored);
+ }
+ let lock = open_recovery_coord_lock(spool)?;
+ lock.lock_exclusive()?;
+ let active = self.dir.join(RECOVERY_MATCH_FILE);
+ let completed = self.dir.join(RECOVERY_COMPLETED_FILE);
+ // Completion is terminal: a `.completed` marker is durable proof the
+ // interval's correlated evidence reached the server, so a restore has
+ // nothing to put back. A `.active` beside the tombstone is a rerun
+ // rewriting the same interval, not a re-opened one.
+ if completed.is_file() {
+ match std::fs::remove_file(&active) {
+ Ok(()) => {}
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
+ Err(error) => return Err(error),
+ }
+ move_quarantined_entries_to_spool(&self.dir, spool)?;
+ return Ok(RestoreOutcome::AlreadyDelivered);
+ }
+ if !active.is_file() {
+ return Err(std::io::Error::new(
+ std::io::ErrorKind::NotFound,
+ format!(
+ "recovery interval {} has neither an active nor a completed marker",
+ self.dir.display()
+ ),
+ ));
+ }
+ let spec = read_recovery_match(&active)?;
+ move_late_matching_entries(spool, &self.dir, &self.config, &spec)?;
+ let _ = std::fs::remove_file(active);
+ move_quarantined_entries_to_spool(&self.dir, spool)?;
+ std::fs::remove_dir(self.dir)?;
+ Ok(RestoreOutcome::Restored)
+ }
+}
+
+/// Move every quarantined `*.json` event back to the plain spool, leaving the
+/// marker files where they are.
+fn move_quarantined_entries_to_spool(dir: &Path, spool: &Path) -> std::io::Result<()> {
+ for item in std::fs::read_dir(dir)? {
+ let item = item?;
+ if item.path().extension().and_then(|value| value.to_str()) != Some("json") {
+ continue;
+ }
+ let destination = spool.join(item.file_name());
+ std::fs::rename(item.path(), destination)?;
+ }
+ Ok(())
+}
+
+fn read_recovery_match(path: &Path) -> std::io::Result<RecoveryMatch> {
+ serde_json::from_slice(&read_spool_entry_bytes(path)?)
+ .map_err(|error| std::io::Error::new(std::io::ErrorKind::InvalidData, error))
+}
+
+fn move_late_matching_entries(
+ spool: &Path,
+ dir: &Path,
+ config: &crate::config::Config,
+ spec: &RecoveryMatch,
+) -> std::io::Result<()> {
+ let (files, _) = list_entries(spool)?;
+ for path in files {
+ let Ok(bytes) = read_spool_entry_bytes(&path) else {
+ continue;
+ };
+ let Ok(entry) = serde_json::from_slice::<SpoolEntry>(&bytes) else {
+ continue;
+ };
+ if entry.created_ms >= spec.interval_started_ms
+ && entry.created_ms <= spec.interval_ended_ms
+ && spool_entry_matches_session(
+ &entry,
+ &spec.server_identity,
+ &spec.agent,
+ Some(config),
+ &spec.workspace,
+ &spec.project,
+ Path::new(&spec.cwd),
+ &spec.native_session_id,
+ )
+ && recovery_target_for_entry(spool, &entry, Some(config))?
+ .is_some_and(|target| target.dir() == dir)
+ && let Some(name) = path.file_name()
+ {
+ std::fs::rename(&path, dir.join(name))?;
+ }
+ }
+ Ok(())
+}
+
+#[allow(clippy::too_many_arguments)]
+pub fn quarantine_session_entries(
+ config: &crate::config::Config,
+ spool: &Path,
+ journal_id: &str,
+ interval_started_ms: u64,
+ interval_ended_ms: u64,
+ server_identity: &str,
+ agent: &str,
+ workspace: &str,
+ project: &str,
+ cwd: &Path,
+ native_session_id: &str,
+) -> std::io::Result<QuarantinedSpool> {
+ let coord_lock = open_recovery_coord_lock(spool)?;
+ coord_lock.lock_exclusive()?;
+ sweep_recovery_tombstones(spool)?;
+ let coordinate_key = recovery_coordinate_key(
+ server_identity,
+ agent,
+ workspace,
+ project,
+ cwd.to_string_lossy().as_ref(),
+ native_session_id,
+ )?;
+ let dir = spool
+ .join(RECOVERY_DIR)
+ .join(recovery_interval_key(journal_id)?);
+ create_spool_dir(&dir)?;
+ let match_spec = RecoveryMatch {
+ journal_id: journal_id.to_owned(),
+ coordinate_key,
+ interval_started_ms,
+ interval_ended_ms,
+ server_identity: server_identity.to_owned(),
+ agent: agent.to_owned(),
+ workspace: workspace.to_owned(),
+ project: project.to_owned(),
+ cwd: cwd.to_string_lossy().into_owned(),
+ native_session_id: native_session_id.to_owned(),
+ };
+ write_private(
+ &dir.join(RECOVERY_MATCH_FILE),
+ &serde_json::to_vec(&match_spec)?,
+ )?;
+ if let Err(error) = move_late_matching_entries(spool, &dir, config, &match_spec) {
+ let _ = std::fs::remove_file(dir.join(RECOVERY_MATCH_FILE));
+ return Err(error);
+ }
+ Ok(QuarantinedSpool {
+ dir,
+ spool: spool.to_path_buf(),
+ config: config.clone(),
+ })
+}
+
+fn open_recovery_coord_lock(spool: &Path) -> std::io::Result<File> {
+ create_spool_dir(spool)?;
+ std::fs::OpenOptions::new()
+ .create(true)
+ .truncate(false)
+ .read(true)
+ .write(true)
+ .open(spool.with_extension("recovery.lock"))
+}
+
+fn sweep_recovery_tombstones(spool: &Path) -> std::io::Result<()> {
+ let recovery = spool.join(RECOVERY_DIR);
+ let entries = match std::fs::read_dir(&recovery) {
+ Ok(entries) => entries,
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(()),
+ Err(error) => return Err(error),
+ };
+ let now = SystemTime::now();
+ let mut completed = Vec::new();
+ for entry in entries.take(MAX_RECOVERY_SWEEP_DIRS) {
+ let entry = entry?;
+ if !entry.file_type()?.is_dir() {
+ continue;
+ }
+ let marker = entry.path().join(RECOVERY_COMPLETED_FILE);
+ let Ok(metadata) = marker.metadata() else {
+ continue;
+ };
+ let modified = metadata.modified().unwrap_or(UNIX_EPOCH);
+ if now.duration_since(modified).unwrap_or_default().as_millis() as u64
+ > RECOVERY_TOMBSTONE_AGE_MS
+ {
+ std::fs::remove_dir_all(entry.path())?;
+ } else {
+ completed.push((modified, entry.path()));
+ }
+ }
+ if completed.len() > MAX_RECOVERY_TOMBSTONES {
+ completed.sort_by_key(|(modified, _)| *modified);
+ let excess = completed.len() - MAX_RECOVERY_TOMBSTONES;
+ for (_, path) in completed.into_iter().take(excess) {
+ std::fs::remove_dir_all(path)?;
+ }
+ }
+ Ok(())
+}
+
+enum RecoveryTarget {
+ Active(PathBuf),
+ Completed(PathBuf),
+}
+
+impl RecoveryTarget {
+ fn dir(&self) -> &Path {
+ match self {
+ Self::Active(dir) | Self::Completed(dir) => dir,
+ }
+ }
+}
+
+fn recovery_target_for_entry(
+ spool: &Path,
+ entry: &SpoolEntry,
+ config: Option<&crate::config::Config>,
+) -> std::io::Result<Option<RecoveryTarget>> {
+ let coordinate_key = recovery_coordinate_key_for_entry(entry, config)?;
+ let recovery = spool.join(RECOVERY_DIR);
+ let directories = match std::fs::read_dir(recovery) {
+ Ok(entries) => entries,
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),
+ Err(error) => return Err(error),
+ };
+ let mut matched = None;
+ for directory in directories.take(MAX_RECOVERY_SWEEP_DIRS) {
+ let directory = directory?;
+ if !directory.file_type()?.is_dir() {
+ continue;
+ }
+ let dir = directory.path();
+ let marker = if dir.join(RECOVERY_MATCH_FILE).is_file() {
+ RECOVERY_MATCH_FILE
+ } else if dir.join(RECOVERY_COMPLETED_FILE).is_file() {
+ RECOVERY_COMPLETED_FILE
+ } else {
+ continue;
+ };
+ let Ok(spec) = read_recovery_match(&dir.join(marker)) else {
+ continue;
+ };
+ if spec.coordinate_key != coordinate_key
+ || entry.created_ms < spec.interval_started_ms
+ || entry.created_ms > spec.interval_ended_ms
+ {
+ continue;
+ }
+ let candidate = if marker == RECOVERY_MATCH_FILE {
+ RecoveryTarget::Active(dir)
+ } else {
+ RecoveryTarget::Completed(dir)
+ };
+ if matched.is_some() {
+ return Ok(None);
+ }
+ matched = Some(candidate);
+ }
+ Ok(matched)
+}
+
+fn recovery_coordinate_key_for_entry(
+ entry: &SpoolEntry,
+ config: Option<&crate::config::Config>,
+) -> std::io::Result<String> {
+ let url = reqwest::Url::parse(&entry.url)
+ .map_err(|error| std::io::Error::new(std::io::ErrorKind::InvalidData, error))?;
+ let mut identity = url.clone();
+ identity.set_query(None);
+ if identity.path().ends_with("/hook") {
+ let path = identity.path().trim_end_matches("/hook").to_owned();
+ identity.set_path(&path);
+ }
+ let mut agent = None;
+ let mut workspace = None;
+ let mut project = None;
+ let mut session = None;
+ let mut cwd = None;
+ for (key, value) in url.query_pairs() {
+ match key.as_ref() {
+ "agent" => agent = Some(value.into_owned()),
+ "workspace" => workspace = Some(value.into_owned()),
+ "project" => project = Some(value.into_owned()),
+ "session_id" => session = Some(value.into_owned()),
+ "cwd" => cwd = Some(value.into_owned()),
+ _ => {}
+ }
+ }
+ if let Ok(body) = serde_json::from_str::<serde_json::Value>(&entry.body) {
+ let context = ai_memory_hooks::hook_payload_context(&body);
+ session = context.session_id.or(session);
+ cwd = context.cwd.or(cwd);
+ }
+ let (workspace, project) = match (config, cwd.as_deref()) {
+ (Some(config), Some(cwd)) => super::resolve_scope_for_path_with_explicit(
+ config,
+ Path::new(cwd),
+ workspace.as_deref(),
+ project.as_deref(),
+ )
+ .unwrap_or_else(|_| {
+ (
+ workspace.clone().unwrap_or_default(),
+ project.clone().unwrap_or_default(),
+ )
+ }),
+ _ => (workspace.unwrap_or_default(), project.unwrap_or_default()),
+ };
+ recovery_coordinate_key(
+ identity.as_str().trim_end_matches('/'),
+ agent.as_deref().unwrap_or_default(),
+ &workspace,
+ &project,
+ cwd.as_deref().unwrap_or_default(),
+ session.as_deref().unwrap_or_default(),
+ )
+}
+
+fn recovery_interval_key(journal_id: &str) -> std::io::Result<String> {
+ if journal_id.is_empty()
+ || journal_id.len() > 128
+ || !journal_id
+ .bytes()
+ .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_'))
+ {
+ return Err(std::io::Error::new(
+ std::io::ErrorKind::InvalidInput,
+ "invalid recovery journal interval id",
+ ));
+ }
+ use sha2::{Digest as _, Sha256};
+ Ok(format!("{:x}", Sha256::digest(journal_id.as_bytes())))
+}
+
+fn recovery_coordinate_key(
+ server_identity: &str,
+ agent: &str,
+ workspace: &str,
+ project: &str,
+ cwd: &str,
+ native_session_id: &str,
+) -> std::io::Result<String> {
+ use sha2::{Digest as _, Sha256};
+ let cwd = crate::marker::absolute_normalized(Path::new(cwd));
+ let mut hasher = Sha256::new();
+ for value in [
+ server_identity,
+ agent,
+ workspace,
+ project,
+ cwd.to_string_lossy().as_ref(),
+ native_session_id,
+ ] {
+ hasher.update(value.len().to_be_bytes());
+ hasher.update(value.as_bytes());
+ }
+ Ok(format!("{:x}", hasher.finalize()))
+}
+
+#[allow(clippy::too_many_arguments)]
+fn spool_entry_matches_session(
+ entry: &SpoolEntry,
+ server_identity: &str,
+ agent: &str,
+ config: Option<&crate::config::Config>,
+ workspace: &str,
+ project: &str,
+ cwd: &Path,
+ native_session_id: &str,
+) -> bool {
+ let Ok(url) = reqwest::Url::parse(&entry.url) else {
+ return false;
+ };
+ let mut identity = url.clone();
+ identity.set_query(None);
+ if identity.path().ends_with("/hook") {
+ let path = identity.path().trim_end_matches("/hook").to_owned();
+ identity.set_path(&path);
+ }
+ let identity = identity.as_str().trim_end_matches('/');
+ if identity != server_identity.trim_end_matches('/') {
+ return false;
+ }
+ let mut session = None;
+ let mut event_agent = None;
+ let mut ws = None;
+ let mut proj = None;
+ let mut event_cwd = None;
+ for (key, value) in url.query_pairs() {
+ match key.as_ref() {
+ "agent" => event_agent = Some(value.into_owned()),
+ "session_id" => session = Some(value.into_owned()),
+ "workspace" => ws = Some(value.into_owned()),
+ "project" => proj = Some(value.into_owned()),
+ "cwd" => event_cwd = Some(value.into_owned()),
+ _ => {}
+ }
+ }
+ if let Ok(body) = serde_json::from_str::<serde_json::Value>(&entry.body) {
+ let context = ai_memory_hooks::hook_payload_context(&body);
+ session = context.session_id.or(session);
+ event_cwd = context.cwd.or(event_cwd);
+ }
+ let expected_cwd = crate::marker::absolute_normalized(cwd);
+ let same_cwd = event_cwd
+ .as_deref()
+ .is_some_and(|value| crate::marker::absolute_normalized(Path::new(value)) == expected_cwd);
+ let explicit_match = ws.as_deref() == Some(workspace) && proj.as_deref() == Some(project);
+ let scope_matches = if explicit_match {
+ true
+ } else if let (Some(config), Some(event_cwd)) = (config, event_cwd.as_deref()) {
+ super::resolve_scope_for_path_with_explicit(
+ config,
+ Path::new(event_cwd),
+ ws.as_deref(),
+ proj.as_deref(),
+ )
+ .is_ok_and(|resolved| resolved == (workspace.to_owned(), project.to_owned()))
+ } else {
+ ws.as_deref().is_none_or(|value| value == workspace)
+ && proj.as_deref().is_none_or(|value| value == project)
+ && ((ws.is_some() && proj.is_some()) || same_cwd)
+ };
+ event_agent.as_deref() == Some(agent)
+ && session.as_deref() == Some(native_session_id)
+ && scope_matches
+ && same_cwd
+}
+
/// Windows can report a contended fs2 byte-range lock as this native OS code
/// instead of mapping it to `WouldBlock`.
#[cfg(windows)]
@@ -485,43 +1167,87 @@ pub async fn drain_until_quiescent(
per_event_timeout: Duration,
wait: DrainLockWait,
max_attempts: MaxAttempts,
+) -> std::io::Result<LockedDrainResult> {
+ drain_until_quiescent_with_live_token(
+ spool,
+ data_dir,
+ total_budget,
+ per_event_timeout,
+ wait,
+ live_static_token().as_deref(),
+ max_attempts,
+ )
+ .await
+}
+
+pub async fn drain_until_quiescent_with_live_token(
+ spool: &Path,
+ data_dir: &Path,
+ total_budget: Duration,
+ per_event_timeout: Duration,
+ wait: DrainLockWait,
+ live_token: Option<&str>,
+ max_attempts: MaxAttempts,
) -> std::io::Result<LockedDrainResult> {
let Some(_lock) = acquire_drain_lock(spool, wait)? else {
return Ok(LockedDrainResult::LockBusy);
};
+ Ok(LockedDrainResult::Drained(
+ drain_until_quiescent_locked_with_live_token(
+ spool,
+ data_dir,
+ total_budget,
+ per_event_timeout,
+ live_token,
+ max_attempts,
+ )
+ .await,
+ ))
+}
+
+pub async fn drain_until_quiescent_locked_with_live_token(
+ spool: &Path,
+ data_dir: &Path,
+ total_budget: Duration,
+ per_event_timeout: Duration,
+ live_token: Option<&str>,
+ max_attempts: MaxAttempts,
+) -> DrainResult {
let started = Instant::now();
let mut combined = DrainResult::default();
loop {
let Some(remaining_budget) = total_budget.checked_sub(started.elapsed()) else {
combined.remaining = spool_len(spool);
- return Ok(LockedDrainResult::Drained(combined));
+ return combined;
};
if remaining_budget.is_zero() {
combined.remaining = spool_len(spool);
- return Ok(LockedDrainResult::Drained(combined));
+ return combined;
}
- let result = drain_with_max_attempts(
+ let result = drain_with_live_token(
spool,
data_dir,
remaining_budget,
per_event_timeout,
+ live_token,
max_attempts,
)
.await;
combined.sent += result.sent;
+ combined.durable += result.durable;
combined.dropped += result.dropped;
combined.remaining = result.remaining;
if result.remaining > 0 {
- return Ok(LockedDrainResult::Drained(combined));
+ return combined;
}
let queued = spool_len(spool);
if queued == 0 {
combined.remaining = 0;
- return Ok(LockedDrainResult::Drained(combined));
+ return combined;
}
combined.remaining = queued;
}
@@ -725,6 +1451,49 @@ pub async fn drain_with_live_token(
let batch_timeout =
batch_request_timeout(per_event_timeout, remaining_budget, chunk.len());
match post_batch(&client, &base, &payload, bearer.as_deref(), batch_timeout).await {
+ BatchOutcome::Detailed {
+ outcomes,
+ failed_index,
+ } => {
+ if outcomes.len() > chunk.len()
+ || failed_index.is_some_and(|index| index >= chunk.len())
+ {
+ bump_or_drop(&chunk[0].0, &chunk[0].1, max_attempts, &mut result);
+ result.remaining +=
+ chunk.len().saturating_sub(1) + files.len().saturating_sub(idx);
+ break;
+ }
+ for (position, outcome) in outcomes.iter().enumerate() {
+ let terminal = matches!(outcome.as_str(), "stored" | "replayed")
+ || outcome == "ignored_end"
+ && entry_event(&chunk[position].1).as_deref()
+ == Some("session-end");
+ if terminal {
+ let _ = std::fs::remove_file(&chunk[position].0);
+ result.sent += 1;
+ result.durable += 1;
+ } else {
+ bump_or_drop(
+ &chunk[position].0,
+ &chunk[position].1,
+ max_attempts,
+ &mut result,
+ );
+ }
+ }
+ if let Some(failed_index) = failed_index {
+ bump_or_drop(
+ &chunk[failed_index].0,
+ &chunk[failed_index].1,
+ max_attempts,
+ &mut result,
+ );
+ }
+ result.remaining += chunk
+ .len()
+ .saturating_sub(outcomes.len())
+ .saturating_sub(usize::from(failed_index.is_some()));
+ }
BatchOutcome::Accepted(k) => {
let k = k.min(chunk.len());
for (path, _) in &chunk[..k] {
@@ -909,6 +1678,24 @@ pub async fn drain_with_live_token(
)
.await
{
+ BatchOutcome::Detailed { outcomes, .. } => {
+ let durable = outcomes
+ .iter()
+ .enumerate()
+ .filter(|(position, outcome)| {
+ matches!(outcome.as_str(), "stored" | "replayed")
+ || outcome.as_str() == "ignored_end"
+ && entry_event(&chunk[*position].1).as_deref()
+ == Some("session-end")
+ })
+ .map(|(position, _)| position)
+ .collect::<Vec<_>>();
+ let sent = delete_accepted_indices(&chunk, &durable);
+ result.sent += sent;
+ result.durable += sent;
+ result.remaining += chunk.len().saturating_sub(sent);
+ continue;
+ }
BatchOutcome::Accepted(k) => {
let k = k.min(chunk.len());
for (path, _) in &chunk[..k] {
@@ -978,6 +1765,24 @@ pub async fn drain_with_live_token(
match post_batch(&client, &base, &payload, Some(&token), batch_timeout)
.await
{
+ BatchOutcome::Detailed { outcomes, .. } => {
+ let durable = outcomes
+ .iter()
+ .enumerate()
+ .filter(|(position, outcome)| {
+ matches!(outcome.as_str(), "stored" | "replayed")
+ || outcome.as_str() == "ignored_end"
+ && entry_event(&chunk[*position].1).as_deref()
+ == Some("session-end")
+ })
+ .map(|(position, _)| position)
+ .collect::<Vec<_>>();
+ let sent = delete_accepted_indices(&chunk, &durable);
+ result.sent += sent;
+ result.durable += sent;
+ result.remaining += chunk.len().saturating_sub(sent);
+ continue;
+ }
BatchOutcome::Accepted(k) => {
let k = k.min(chunk.len());
for (path, _) in &chunk[..k] {
@@ -1115,8 +1920,71 @@ fn batch_request_timeout(
.min(remaining_budget)
}
+fn completed_tombstone_for_path(path: &Path) -> bool {
+ let Some(name) = path.file_name().and_then(|name| name.to_str()) else {
+ return false;
+ };
+ let Some(stem) = name.strip_suffix(".json") else {
+ return false;
+ };
+ let Some(coordinate_key) = stem.rsplit('-').next() else {
+ return false;
+ };
+ let Some(created_ms) = created_ms_from_name(name) else {
+ return false;
+ };
+ let Some(spool) = path.parent() else {
+ return false;
+ };
+ let Ok(directories) = std::fs::read_dir(spool.join(RECOVERY_DIR)) else {
+ return false;
+ };
+ let mut matches = 0usize;
+ for directory in directories.take(MAX_RECOVERY_SWEEP_DIRS).flatten() {
+ let marker = directory.path().join(RECOVERY_COMPLETED_FILE);
+ let Ok(spec) = read_recovery_match(&marker) else {
+ continue;
+ };
+ if spec.coordinate_key == coordinate_key
+ && created_ms >= spec.interval_started_ms
+ && created_ms <= spec.interval_ended_ms
+ {
+ matches += 1;
+ if matches > 1 {
+ return false;
+ }
+ }
+ }
+ matches == 1
+}
+
+fn read_spool_entry_bytes(path: &Path) -> std::io::Result<Vec<u8>> {
+ use std::io::Read as _;
+ let file = File::open(path)?;
+ if file.metadata()?.len() > MAX_SPOOL_ENTRY_BYTES {
+ return Err(std::io::Error::new(
+ std::io::ErrorKind::InvalidData,
+ "spool entry exceeds its byte limit",
+ ));
+ }
+ let mut bytes = Vec::new();
+ file.take(MAX_SPOOL_ENTRY_BYTES.saturating_add(1))
+ .read_to_end(&mut bytes)?;
+ if bytes.len() as u64 > MAX_SPOOL_ENTRY_BYTES {
+ return Err(std::io::Error::new(
+ std::io::ErrorKind::InvalidData,
+ "spool entry exceeds its byte limit",
+ ));
+ }
+ Ok(bytes)
+}
+
fn load_live_entry(path: &Path, result: &mut DrainResult) -> Option<SpoolEntry> {
- let Ok(bytes) = std::fs::read(path) else {
+ if completed_tombstone_for_path(path) {
+ let _ = std::fs::remove_file(path);
+ return None;
+ }
+ let Ok(bytes) = read_spool_entry_bytes(path) else {
result.remaining += 1;
return None;
};
@@ -1317,6 +2185,13 @@ fn configured_server_url(data_dir: &Path) -> Option<String> {
/// The `/hook/batch` URL for a spooled per-event URL: strip the `?…` query and
/// append `/batch` (a spooled URL ends in `…/hook` before its query). Entries
/// whose endpoint string matches can ride one batch request.
+fn entry_event(entry: &SpoolEntry) -> Option<String> {
+ reqwest::Url::parse(&entry.url)
+ .ok()?
+ .query_pairs()
+ .find_map(|(key, value)| (key == "event").then(|| value.into_owned()))
+}
+
fn batch_endpoint(url: &str) -> String {
let path = url.split('?').next().unwrap_or(url);
format!("{path}/batch")
@@ -1395,28 +2270,6 @@ pub async fn resolve_bearer(
crate::auth_bearer::resolve_bearer(client, &data_dir.join("auth.json"), auth_token).await
}
-fn prune_spool_file_count(spool: &Path) {
- let Ok((mut files, _)) = list_entries(spool) else {
- return;
- };
- let excess = files.len().saturating_sub(MAX_SPOOL_FILES);
- if excess == 0 {
- return;
- }
- files.sort();
- // The spool is at its hard cap: the oldest events are about to be deleted
- // WITHOUT ever reaching the server — silent capture loss otherwise. Surface
- // it on stderr (never stdout, which carries the hook's JSON protocol output)
- // so a sustained backlog dropping events is visible, not invisible.
- eprintln!(
- "ai-memory: hook-spool at capacity ({} > {MAX_SPOOL_FILES}); evicting {excess} oldest UNDELIVERED event(s)",
- files.len()
- );
- for path in files.into_iter().take(excess) {
- let _ = std::fs::remove_file(path);
- }
-}
-
/// List `*.json` spool files (ignoring in-flight `*.json.tmp`), or None when the
/// directory doesn't exist yet.
/// Spool files awaiting delivery, or the IO error that prevented listing them.
@@ -1441,6 +2294,9 @@ fn list_entries(spool: &Path) -> std::io::Result<(Vec<PathBuf>, usize)> {
let path = ent.path();
if path.extension().and_then(|e| e.to_str()) == Some("json") {
out.push(path);
+ if out.len() > MAX_SPOOL_FILES {
+ break;
+ }
}
}
Err(_) => unreadable += 1,
@@ -1453,6 +2309,457 @@ fn list_entries(spool: &Path) -> std::io::Result<(Vec<PathBuf>, usize)> {
mod tests {
use super::*;
+ #[test]
+ fn correlated_session_entries_quarantine_and_restore_without_touching_others() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = temp.path().join("hook-spool");
+ let matching = SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project&session_id=native-1".into(),
+ body: r#"{"session_id":"native-1","cwd":"/repo","prompt":"same prompt"}"#.into(),
+ created_ms: now_ms(),
+ auth_mode: AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ let other = SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&workspace=ws&project=project&session_id=native-2".into(),
+ body: r#"{"session_id":"native-2","cwd":"/repo","prompt":"other"}"#.into(),
+ ..matching.clone()
+ };
+ enqueue(&spool, &matching).unwrap();
+ enqueue(&spool, &other).unwrap();
+ let quarantine = quarantine_session_entries(
+ &crate::config::Config::default(),
+ &spool,
+ "journal-1",
+ matching.created_ms.saturating_sub(1),
+ matching.created_ms.saturating_add(1),
+ "http://127.0.0.1:49374",
+ "claude-code",
+ "ws",
+ "project",
+ Path::new("/repo"),
+ "native-1",
+ )
+ .unwrap();
+ assert_eq!(spool_len(&spool), 1);
+ quarantine.restore(&spool).unwrap();
+ assert_eq!(spool_len(&spool), 2);
+ }
+
+ /// A quarantine whose `.active` marker is gone because the interval
+ /// already completed restores as already-delivered instead of erroring —
+ /// including when a rerun rewrote a fresh `.active` over the completed
+ /// tombstone — while a directory with neither marker is still an error.
+ #[test]
+ fn restore_treats_a_completed_interval_as_already_delivered() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = temp.path().join("hook-spool");
+ let quarantine = |journal: &str| {
+ quarantine_session_entries(
+ &crate::config::Config::default(),
+ &spool,
+ journal,
+ 0,
+ u64::MAX,
+ "http://127.0.0.1:49374",
+ "claude-code",
+ "ws",
+ "project",
+ Path::new("/repo"),
+ "native-1",
+ )
+ };
+ let interval_dir = |journal: &str| {
+ spool
+ .join(RECOVERY_DIR)
+ .join(recovery_interval_key(journal).unwrap())
+ };
+ let completed = quarantine("journal-done").unwrap();
+ assert!(completed.complete_spool_only().unwrap());
+ assert_eq!(
+ completed.restore(&spool).unwrap(),
+ RestoreOutcome::AlreadyDelivered,
+ "a missing .active beside a completed marker means durable delivery"
+ );
+ let rerun = quarantine("journal-done").unwrap();
+ assert!(
+ interval_dir("journal-done")
+ .join(RECOVERY_MATCH_FILE)
+ .is_file(),
+ "the rerun rewrites .active over the completed tombstone"
+ );
+ assert_eq!(
+ rerun.restore(&spool).unwrap(),
+ RestoreOutcome::AlreadyDelivered,
+ "a rerun's rewritten .active never re-opens a completed interval"
+ );
+ assert!(
+ interval_dir("journal-done")
+ .join(RECOVERY_COMPLETED_FILE)
+ .is_file()
+ );
+ assert!(
+ !interval_dir("journal-done")
+ .join(RECOVERY_MATCH_FILE)
+ .is_file()
+ );
+
+ let broken = quarantine("journal-broken").unwrap();
+ std::fs::remove_file(interval_dir("journal-broken").join(RECOVERY_MATCH_FILE)).unwrap();
+ assert!(
+ broken.restore(&spool).is_err(),
+ "a directory with neither marker is still an error"
+ );
+ }
+
+ #[test]
+ fn quarantine_matching_reuses_every_canonical_nested_hook_context_shape() {
+ let base = SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),
+ body: String::new(),
+ created_ms: now_ms(),
+ auth_mode: AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ let matching = [
+ serde_json::json!({"info": {"id": "native-1", "directory": "/repo"}}),
+ serde_json::json!({"properties": {"session_id": "native-1", "cwd": "/repo"}}),
+ serde_json::json!({"properties": {"sessionID": "native-1", "cwd": "/repo"}}),
+ serde_json::json!({"properties": {"info": {"id": "native-1", "directory": "/repo"}}}),
+ serde_json::json!({"event": {"sessionId": "native-1", "working_dir": "/repo"}}),
+ serde_json::json!({"event": {"properties": {"sessionID": "native-1", "cwd": "/repo"}}}),
+ serde_json::json!({"event": {"properties": {"info": {"id": "native-1", "directory": "/repo"}}}}),
+ serde_json::json!({"payload": {"conversationId": "native-1", "directory": "/repo"}}),
+ serde_json::json!({"payload": {"path": {"session": "native-1", "cwd": "/repo"}}}),
+ serde_json::json!({"payload": {"info": {"id": "native-1", "directory": "/repo"}}}),
+ serde_json::json!({"payload": {"properties": {"sessionID": "native-1", "cwd": "/repo"}}}),
+ serde_json::json!({"payload": {"properties": {"info": {"id": "native-1", "directory": "/repo"}}}}),
+ ];
+ for body in matching {
+ let entry = SpoolEntry {
+ body: body.to_string(),
+ ..base.clone()
+ };
+ assert!(spool_entry_matches_session(
+ &entry,
+ "http://127.0.0.1:49374",
+ "claude-code",
+ None,
+ "ws",
+ "project",
+ Path::new("/repo"),
+ "native-1",
+ ));
+ }
+ for body in [
+ serde_json::json!({"payload": {"info": {"id": "other", "directory": "/repo"}}}),
+ serde_json::json!({"payload": {"info": {"id": "native-1", "directory": "/other"}}}),
+ serde_json::json!({"payload": {"info": {"id": "native-1"}}, "unrelated": {"cwd": "/repo"}}),
+ ] {
+ let entry = SpoolEntry {
+ body: body.to_string(),
+ ..base.clone()
+ };
+ assert!(!spool_entry_matches_session(
+ &entry,
+ "http://127.0.0.1:49374",
+ "claude-code",
+ None,
+ "ws",
+ "project",
+ Path::new("/repo"),
+ "native-1",
+ ));
+ }
+ }
+
+ #[test]
+ fn quarantine_matches_partial_scope_and_captures_late_enqueue() {
+ let temp = tempfile::tempdir().unwrap();
+ let repo = temp.path().join("project");
+ std::fs::create_dir_all(&repo).unwrap();
+ std::fs::write(
+ repo.join(".ai-memory.toml"),
+ "workspace = \"ws\"\nproject = \"project\"\n",
+ )
+ .unwrap();
+ let spool = temp.path().join("hook-spool");
+ let config = crate::config::Config::default();
+ let base = SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws".into(),
+ body: serde_json::json!({"session_id":"native-1","cwd":repo,"prompt":"one"})
+ .to_string(),
+ created_ms: now_ms(),
+ auth_mode: AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ enqueue_with_config(&spool, &base, Some(&config)).unwrap();
+ let quarantine = quarantine_session_entries(
+ &config,
+ &spool,
+ "journal-partial",
+ base.created_ms.saturating_sub(1),
+ base.created_ms.saturating_add(1),
+ "http://127.0.0.1:49374",
+ "claude-code",
+ "ws",
+ "project",
+ &repo,
+ "native-1",
+ )
+ .unwrap();
+ assert_eq!(
+ quarantine.len(),
+ 1,
+ "workspace-only event is correlated by cwd"
+ );
+ enqueue_with_config(
+ &spool,
+ &SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=stop&agent=claude-code&project=project"
+ .into(),
+ body: serde_json::json!({"session_id":"native-1","cwd":repo}).to_string(),
+ ..base.clone()
+ },
+ Some(&config),
+ )
+ .unwrap();
+ assert_eq!(
+ quarantine.len(),
+ 2,
+ "late enqueue joins the active quarantine"
+ );
+ assert_eq!(spool_len(&spool), 0, "neither correlated event can drain");
+ quarantine.restore(&spool).unwrap();
+ assert_eq!(spool_len(&spool), 2);
+ }
+
+ #[test]
+ fn enqueue_is_bounded_when_recovery_coordination_lock_is_held() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = temp.path().join("hook-spool");
+ let lock = open_recovery_coord_lock(&spool).unwrap();
+ lock.lock_exclusive().unwrap();
+ let entry = entry_for(
+ "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),
+ r#"{"session_id":"native-1","cwd":"/repo","prompt":"late"}"#.into(),
+ None,
+ false,
+ );
+ let started = Instant::now();
+ enqueue(&spool, &entry).unwrap();
+ assert!(started.elapsed() < Duration::from_millis(100));
+ assert_eq!(spool_len(&spool), 1, "content is retained for recovery");
+ }
+
+ #[test]
+ fn completion_race_leaves_contended_enqueue_safe_for_tombstone_cleanup() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = temp.path().join("hook-spool");
+ let config = crate::config::Config::default();
+ let entry = SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),
+ body: r#"{"session_id":"native-1","cwd":"/repo","prompt":"late"}"#.into(),
+ created_ms: now_ms(),
+ auth_mode: AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ let quarantine = quarantine_session_entries(
+ &config,
+ &spool,
+ "journal",
+ entry.created_ms.saturating_sub(1),
+ entry.created_ms.saturating_add(1),
+ "http://127.0.0.1:49374",
+ "claude-code",
+ "ws",
+ "project",
+ Path::new("/repo"),
+ "native-1",
+ )
+ .unwrap();
+ PAUSE_ENQUEUE_AFTER_MAIN_RENAME.store(true, Ordering::SeqCst);
+ ENQUEUE_REACHED_RECOVERY_WINDOW.store(false, Ordering::SeqCst);
+ let enqueue_spool = spool.clone();
+ let enqueue_config = config.clone();
+ let enqueue = std::thread::spawn(move || {
+ enqueue_with_config(&enqueue_spool, &entry, Some(&enqueue_config)).unwrap()
+ });
+ while !ENQUEUE_REACHED_RECOVERY_WINDOW.load(Ordering::SeqCst) {
+ std::thread::yield_now();
+ }
+ quarantine.complete().unwrap();
+ PAUSE_ENQUEUE_AFTER_MAIN_RENAME.store(false, Ordering::SeqCst);
+ enqueue.join().unwrap();
+ let mut result = DrainResult::default();
+ if let Some(path) = list_entries(&spool).unwrap().0.into_iter().next() {
+ assert!(load_live_entry(&path, &mut result).is_none());
+ }
+ assert_eq!(spool_len(&spool), 0, "completed tombstone removes overlap");
+ }
+
+ #[test]
+ fn completed_recovery_tombstone_retires_late_overlap() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = temp.path().join("hook-spool");
+ let entry = SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),
+ body: r#"{"session_id":"native-1","cwd":"/repo","prompt":"late"}"#.into(),
+ created_ms: now_ms(),
+ auth_mode: AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ let quarantine = quarantine_session_entries(
+ &crate::config::Config::default(),
+ &spool,
+ "journal",
+ entry.created_ms.saturating_sub(1),
+ entry.created_ms.saturating_add(1),
+ "http://127.0.0.1:49374",
+ "claude-code",
+ "ws",
+ "project",
+ Path::new("/repo"),
+ "native-1",
+ )
+ .unwrap();
+ quarantine.complete().unwrap();
+ enqueue(&spool, &entry).unwrap();
+ assert_eq!(spool_len(&spool), 0, "late overlap must not be delivered");
+ }
+
+ /// The completed tombstone is bounded by its interval: a hook from a
+ /// LATER resume of the same native session is fresh evidence and must be
+ /// delivered, never swallowed by the earlier interval's tombstone.
+ #[test]
+ fn completed_tombstone_keeps_a_later_resume_of_the_same_session() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = temp.path().join("hook-spool");
+ let interval_ended = now_ms().saturating_sub(60_000);
+ let quarantine = quarantine_session_entries(
+ &crate::config::Config::default(),
+ &spool,
+ "journal",
+ interval_ended.saturating_sub(1_000),
+ interval_ended,
+ "http://127.0.0.1:49374",
+ "claude-code",
+ "ws",
+ "project",
+ Path::new("/repo"),
+ "native-1",
+ )
+ .unwrap();
+ quarantine.complete().unwrap();
+ let later_resume = SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),
+ body: r#"{"session_id":"native-1","cwd":"/repo","prompt":"from a later resume"}"#.into(),
+ created_ms: now_ms(),
+ auth_mode: AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ enqueue(&spool, &later_resume).unwrap();
+ assert_eq!(
+ spool_len(&spool),
+ 1,
+ "a hook from a later resume must stay queued for delivery"
+ );
+ let path = list_entries(&spool).unwrap().0.into_iter().next().unwrap();
+ let mut result = DrainResult::default();
+ assert!(
+ load_live_entry(&path, &mut result).is_some(),
+ "the drain path must deliver, not drop, post-interval evidence"
+ );
+ assert_eq!(result.dropped, 0);
+ }
+
+ /// Completing one interval only ever consumes its own correlated entries:
+ /// another interval's quarantine keeps its evidence, keeps capturing its
+ /// late hooks, and an entry attributable to two intervals at once fails
+ /// closed into the main spool instead of being attributed by guesswork.
+ #[test]
+ fn completing_one_recovery_interval_leaves_the_other_intervals_entries_intact() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = temp.path().join("hook-spool");
+ let config = crate::config::Config::default();
+ let first_ms = now_ms().saturating_sub(120_000);
+ let second_ms = first_ms + 60_000;
+ let entry = |created_ms: u64, prompt: &str| {
+ SpoolEntry {
+ url: "http://127.0.0.1:49374/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=project".into(),
+ body: format!(r#"{{"session_id":"native-1","cwd":"/repo","prompt":"{prompt}"}}"#),
+ created_ms,
+ auth_mode: AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ }
+ };
+ let quarantine = |journal: &str, started: u64, ended: u64| {
+ quarantine_session_entries(
+ &config,
+ &spool,
+ journal,
+ started,
+ ended,
+ "http://127.0.0.1:49374",
+ "claude-code",
+ "ws",
+ "project",
+ Path::new("/repo"),
+ "native-1",
+ )
+ };
+ enqueue(&spool, &entry(first_ms, "first interval")).unwrap();
+ enqueue(&spool, &entry(second_ms, "second interval")).unwrap();
+ let first = quarantine("journal-a", first_ms - 1, first_ms + 1).unwrap();
+ let second = quarantine("journal-b", second_ms - 1, second_ms + 1).unwrap();
+ assert_eq!(first.len(), 1);
+ assert_eq!(second.len(), 1);
+ assert_eq!(spool_len(&spool), 0);
+
+ first.complete().unwrap();
+ assert_eq!(
+ second.len(),
+ 1,
+ "completing one interval never deletes another interval's correlated entries"
+ );
+ enqueue(&spool, &entry(second_ms, "second interval late hook")).unwrap();
+ assert_eq!(
+ second.len(),
+ 2,
+ "the still-active interval keeps capturing its own late hooks"
+ );
+ assert_eq!(spool_len(&spool), 0);
+
+ let overlapping = quarantine("journal-c", second_ms - 1, second_ms + 1).unwrap();
+ assert_eq!(
+ overlapping.len(),
+ 0,
+ "the new interval claims nothing already attributed to journal-b"
+ );
+ enqueue(&spool, &entry(second_ms, "attributable to both")).unwrap();
+ assert_eq!(
+ spool_len(&spool),
+ 1,
+ "an entry two intervals could own fails closed into the main spool"
+ );
+ second.restore(&spool).unwrap();
+ assert_eq!(spool_len(&spool), 3);
+ }
+
#[test]
fn entry_for_picks_auth_mode() {
let s = entry_for("u".into(), "{}".into(), Some("tok"), false);
@@ -1625,7 +2932,7 @@ mod tests {
}
#[test]
- fn enqueue_prunes_oldest_files_when_spool_exceeds_limit() {
+ fn enqueue_refuses_new_entries_when_spool_reaches_limit() {
let tmp = tempfile::tempdir().unwrap();
let spool = spool_dir(tmp.path());
for i in 0..(MAX_SPOOL_FILES + 2) {
@@ -1646,7 +2953,77 @@ mod tests {
.iter()
.map(|path| serde_json::from_slice(&std::fs::read(path).unwrap()).unwrap())
.collect();
- assert!(bodies.iter().all(|entry| entry.created_ms >= 2));
+ assert!(
+ bodies
+ .iter()
+ .all(|entry| entry.created_ms < MAX_SPOOL_FILES as u64)
+ );
+ }
+
+ #[test]
+ fn capacity_tracks_current_global_occupancy_across_restart_and_quarantine() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = spool_dir(temp.path());
+ for cycle in 0..(MAX_SPOOL_FILES * 3) {
+ let entry = entry_for(
+ format!("https://x/hook?event={cycle}"),
+ "{}".into(),
+ None,
+ false,
+ );
+ enqueue(&spool, &entry).unwrap();
+ let path = list_entries(&spool).unwrap().0.into_iter().next().unwrap();
+ std::fs::remove_file(path).unwrap();
+ }
+ let accepted = entry_for(
+ "https://x/hook?event=accepted".into(),
+ "{}".into(),
+ None,
+ false,
+ );
+ enqueue(&spool, &accepted).unwrap();
+ assert_eq!(total_spool_entries(&spool).unwrap(), 1);
+ std::fs::remove_file(list_entries(&spool).unwrap().0[0].clone()).unwrap();
+
+ let quarantine = spool.join(RECOVERY_DIR).join("full");
+ std::fs::create_dir_all(&quarantine).unwrap();
+ for index in 0..MAX_SPOOL_FILES {
+ std::fs::write(quarantine.join(format!("{index}.json")), b"{}").unwrap();
+ }
+ enqueue(&spool, &accepted).unwrap();
+ assert_eq!(
+ spool_len(&spool),
+ 0,
+ "active quarantines count toward the cap"
+ );
+ }
+
+ #[test]
+ fn oversized_spool_entry_is_refused_before_read_allocation() {
+ let temp = tempfile::tempdir().unwrap();
+ let path = temp.path().join("oversized.json");
+ let file = File::create(&path).unwrap();
+ file.set_len(MAX_SPOOL_ENTRY_BYTES + 1).unwrap();
+ let error = read_spool_entry_bytes(&path).unwrap_err();
+ assert_eq!(error.kind(), std::io::ErrorKind::InvalidData);
+ }
+
+ #[test]
+ fn tombstone_sweep_is_global_and_bounded_by_age_and_count() {
+ let temp = tempfile::tempdir().unwrap();
+ let spool = spool_dir(temp.path());
+ let recovery = spool.join(RECOVERY_DIR);
+ std::fs::create_dir_all(&recovery).unwrap();
+ for index in 0..(MAX_RECOVERY_TOMBSTONES + 2) {
+ let dir = recovery.join(format!("done-{index:04}"));
+ std::fs::create_dir(&dir).unwrap();
+ write_private(&dir.join(RECOVERY_COMPLETED_FILE), b"done").unwrap();
+ }
+ sweep_recovery_tombstones(&spool).unwrap();
+ assert!(
+ std::fs::read_dir(&recovery).unwrap().count() <= MAX_RECOVERY_TOMBSTONES,
+ "completed tombstones are globally count-bounded"
+ );
}
#[tokio::test]
diff --git a/crates/ai-memory-cli/src/commands/mod.rs b/crates/ai-memory-cli/src/commands/mod.rs
index 02f18239..6f07049c 100644
--- a/crates/ai-memory-cli/src/commands/mod.rs
+++ b/crates/ai-memory-cli/src/commands/mod.rs
@@ -80,6 +80,7 @@ pub mod purge_project;
pub mod purge_session;
pub mod read_page;
pub mod reclaim_ledger_versions;
+pub mod recovery;
pub mod reindex;
pub mod rename_project;
pub mod rename_workstream;
@@ -256,6 +257,15 @@ pub(crate) fn resolve_scope_with_marker(
pub(crate) fn resolve_scope_for_path(
config: &Config,
cwd: &std::path::Path,
+) -> Result<(String, String)> {
+ resolve_scope_for_path_with_explicit(config, cwd, None, None)
+}
+
+pub(crate) fn resolve_scope_for_path_with_explicit(
+ config: &Config,
+ cwd: &std::path::Path,
+ explicit_ws: Option<&str>,
+ explicit_proj: Option<&str>,
) -> Result<(String, String)> {
if !cwd.exists() {
return Err(anyhow!(
@@ -266,39 +276,44 @@ pub(crate) fn resolve_scope_for_path(
let cwd = crate::marker::absolute_normalized(cwd);
let identity = cwd.to_string_lossy().into_owned();
let marker = crate::marker::read_scope(&identity, &config.runtime_env)?;
- let workspace = marker
- .as_ref()
- .and_then(|scope| scope.workspace.clone())
+ let workspace = explicit_ws
+ .filter(|value| !value.is_empty())
+ .map(str::to_owned)
+ .or_else(|| marker.as_ref().and_then(|scope| scope.workspace.clone()))
.unwrap_or_else(|| crate::config::DEFAULT_WORKSPACE.to_string());
- let project = match marker.as_ref() {
- Some(scope) => scope
- .project
- .clone()
- .or_else(|| scope.canonical_remote_project())
- .or_else(|| {
- if scope.is_repo_root() {
- crate::marker::repo_root_project(&identity)
- } else {
+ let project = if let Some(project) = explicit_proj.filter(|value| !value.is_empty()) {
+ project.to_owned()
+ } else {
+ match marker.as_ref() {
+ Some(scope) => scope
+ .project
+ .clone()
+ .or_else(|| scope.canonical_remote_project())
+ .or_else(|| {
+ if scope.is_repo_root() {
+ crate::marker::repo_root_project(&identity)
+ } else {
+ ai_memory_consolidate::derive_project_name(
+ &cwd,
+ ai_memory_consolidate::ProjectNameStrategy::Basename,
+ )
+ .map(|(name, _)| name)
+ }
+ })
+ .ok_or_else(|| anyhow!("could not derive project name from {}", cwd.display()))?,
+ None => crate::marker::discover_remote_identity(&identity)
+ .and_then(|repository| {
+ ai_memory_core::repository_identity::path_style_name(&repository)
+ })
+ .or_else(|| {
ai_memory_consolidate::derive_project_name(
&cwd,
ai_memory_consolidate::ProjectNameStrategy::Basename,
)
.map(|(name, _)| name)
- }
- })
- .ok_or_else(|| anyhow!("could not derive project name from {}", cwd.display()))?,
- None => crate::marker::discover_remote_identity(&identity)
- .and_then(|repository| {
- ai_memory_core::repository_identity::path_style_name(&repository)
- })
- .or_else(|| {
- ai_memory_consolidate::derive_project_name(
- &cwd,
- ai_memory_consolidate::ProjectNameStrategy::Basename,
- )
- .map(|(name, _)| name)
- })
- .ok_or_else(|| anyhow!("could not derive project name from {}", cwd.display()))?,
+ })
+ .ok_or_else(|| anyhow!("could not derive project name from {}", cwd.display()))?,
+ }
};
Ok((workspace, project))
}
diff --git a/crates/ai-memory-cli/src/commands/recovery.rs b/crates/ai-memory-cli/src/commands/recovery.rs
new file mode 100644
index 00000000..ee639e90
--- /dev/null
+++ b/crates/ai-memory-cli/src/commands/recovery.rs
@@ -0,0 +1,2141 @@
+//! Downtime recovery — the offline journal and `ai-memory recover`.
+//!
+//! When the server is unreachable, `ai-memory run` degrades instead of
+//! failing: hooks spool locally and a degraded launch still runs the harness.
+//! The spool returns by itself, but two kinds of history used to need a
+//! hand-typed repair and, for the transcript, were simply lost:
+//!
+//! - a run whose lease existed but whose transcript `finish` could not reach
+//! the dead server (`finish-failed`);
+//! - a degraded launch's session, which no server run ever recorded
+//! (`degraded-run`).
+//!
+//! Both now append a bounded, metadata-only entry to a local journal
+//! (`<data_dir>/recovery-journal.json`). `ai-memory recover`, run once the
+//! server is back, drains the spool and then works the journal: it re-exports
+//! each `finish-failed` session's transcript from the native store through the
+//! ordinary managed-run finish path, and replays each degraded launch's exact
+//! native transcript through the ordinary sanitized `/hook/batch` backfill
+//! path. Harnesses without transcript export are journaled explicitly as
+//! spool-only and recovery never invokes their exporter. Stable ingest and
+//! event keys make both paths idempotent. Entries clear as they succeed;
+//! failures stay journaled with their reason.
+//!
+//! The journal never holds transcript content or secrets — only locators,
+//! ordered semantic digests, and delivery evidence. Recovery never regenerates
+//! the original repository checkpoint, so later repository state is not
+//! attributed to the interrupted run. Transcripts are re-exported from the native store
+//! at recover time instead of being copied into the data dir: the store is
+//! the same durable local source the online path reads seconds after the
+//! child exits, and re-export reuses the exact adapter path rather than
+//! maintaining a second serialized copy of private conversation content.
+
+use std::collections::HashMap;
+use std::fs::{self, File, OpenOptions};
+use std::path::{Path, PathBuf};
+use std::str::FromStr as _;
+use std::time::{Duration, Instant};
+
+use ai_memory_core::{ManagedRunId, Sanitizer};
+use ai_memory_workstream::TranscriptCapability;
+use anyhow::{Context as _, Result, anyhow, bail};
+use fs2::FileExt as _;
+use serde::{Deserialize, Serialize};
+
+use crate::cli::RecoverArgs;
+use crate::commands::hook_spool::{self, DrainLockWait, MaxAttempts, RestoreOutcome};
+use crate::config::Config;
+use crate::http_client::{ServerEndpoint, ServerProbe, probe_server};
+
+/// Journal file name inside the data dir.
+const JOURNAL_FILE: &str = "recovery-journal.json";
+/// Exclusive-mutation lock file name inside the data dir.
+const JOURNAL_LOCK_FILE: &str = "recovery-journal.lock";
+/// Journal schema version.
+const JOURNAL_VERSION: u32 = 1;
+/// Most entries kept. Appends at capacity are refused so recovery work is
+/// never silently evicted.
+pub(crate) const MAX_JOURNAL_ENTRIES: usize = 256;
+/// Refuse to load or overwrite a journal larger than this, mirroring the
+/// client project registry's guard: a swollen file is corruption, not backlog.
+const MAX_JOURNAL_BYTES: u64 = 16 * 1024 * 1024;
+/// Reasons remembered per entry across recover attempts.
+const MAX_ENTRY_FAILURES: usize = 3;
+const MAX_FAILURE_REASON_BYTES: usize = 2_048;
+
+/// How the entry's history was cut off from the server.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "kebab-case")]
+pub(crate) enum JournalKind {
+ /// A degraded launch made while the server was unreachable: no managed
+ /// run ever existed, so recovery replays its exact native transcript
+ /// through the hook backfill path after draining the spool.
+ DegradedRun,
+ /// A run whose transcript import failed because the server died after the
+ /// child exited: the run still exists server-side, so recovery re-exports
+ /// the transcript and finishes it again.
+ FinishFailed,
+}
+
+impl JournalKind {
+ /// Stable label for reports.
+ #[must_use]
+ pub(crate) fn as_str(self) -> &'static str {
+ match self {
+ Self::DegradedRun => "degraded-run",
+ Self::FinishFailed => "finish-failed",
+ }
+ }
+}
+
+/// One bounded, metadata-only record of history the server missed.
+///
+/// Constructed by `run` at the moment the loss is detected; consumed by
+/// `recover` once the server answers again.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub(crate) struct JournalEntry {
+ /// Unique entry id; recovery clears per entry.
+ pub(crate) id: String,
+ /// When the loss was recorded (RFC 3339, UTC).
+ pub(crate) recorded_at: String,
+ /// Inclusive lower bound for hook events belonging to this launch.
+ pub(crate) interval_started_ms: u64,
+ /// Inclusive upper bound for hook events belonging to this launch.
+ pub(crate) interval_ended_ms: u64,
+ /// How the run was cut off.
+ pub(crate) kind: JournalKind,
+ /// Managed harness name (`ManagedHarness::as_str`), including the exact
+ /// Kiro engine flavor so recovery reads the right native store.
+ pub(crate) harness: String,
+ /// Native session locator for transcript re-export.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) native_session_id: Option<String>,
+ /// Session-store override the launch resolved (`--session-dir` and kin),
+ /// without which the native transcript may be unreachable.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) session_dir: Option<String>,
+ /// Canonical checkout directory the run launched from.
+ pub(crate) cwd: String,
+ /// Workspace name for degraded transcript replay and reporting.
+ pub(crate) workspace: String,
+ /// Project name for degraded transcript replay and reporting.
+ pub(crate) project: String,
+ /// Server URL the run targeted, for a mismatch warning at recover time.
+ pub(crate) server_url: String,
+ /// Managed run id (`finish-failed` only): the recovery import target.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) run_id: Option<String>,
+ /// Ordered semantic digests for this exact transcript interval. `Some([])`
+ /// proves a successful empty export; `None` means export was unavailable.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) event_digests: Option<Vec<String>>,
+ /// Whether this typed harness capability permits spool evidence only and
+ /// must never invoke a native transcript exporter.
+ #[serde(default)]
+ pub(crate) spool_only: bool,
+ /// Exact native adapter cursor captured before a resumed launch.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) source_cursor: Option<String>,
+ /// Exact native adapter cursor captured immediately after child exit.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) final_cursor: Option<String>,
+ /// Whether correlated hook events have ever been observed locally.
+ #[serde(default)]
+ pub(crate) correlated_events_seen: bool,
+ /// Whether a prior pass durably delivered all correlated hook evidence.
+ #[serde(default)]
+ pub(crate) correlated_delivery_confirmed: bool,
+ /// Whether a prior pass lost potentially correlated hook evidence.
+ #[serde(default)]
+ pub(crate) correlated_loss: bool,
+ /// Child exit code, replayed with the recovered import.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub(crate) exit_code: Option<i32>,
+ /// Reasons from previous recover attempts, newest last, bounded.
+ #[serde(default, skip_serializing_if = "Vec::is_empty")]
+ pub(crate) failures: Vec<String>,
+}
+
+/// Wire shape of the journal file.
+#[derive(Debug, Serialize, Deserialize)]
+struct JournalFile {
+ version: u32,
+ entries: Vec<JournalEntry>,
+}
+
+/// `<data_dir>/recovery-journal.json`.
+#[must_use]
+pub(crate) fn journal_path(data_dir: &Path) -> PathBuf {
+ data_dir.join(JOURNAL_FILE)
+}
+
+/// Read the journal; a missing file is an empty journal, a corrupt or swollen
+/// one is an error the caller surfaces (never silently reset: the file is the
+/// only record of what the server missed).
+pub(crate) fn load_journal(data_dir: &Path) -> Result<Vec<JournalEntry>> {
+ let path = journal_path(data_dir);
+ refuse_symlink(&path)?;
+ let mut file = match open_journal_read(&path) {
+ Ok(file) => file,
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),
+ Err(error) => {
+ return Err(error).with_context(|| format!("reading {}", path.display()));
+ }
+ };
+ if file.metadata()?.len() > MAX_JOURNAL_BYTES {
+ bail!(
+ "recovery journal {} exceeds the {} byte limit",
+ path.display(),
+ MAX_JOURNAL_BYTES
+ );
+ }
+ let mut bytes = Vec::new();
+ use std::io::Read as _;
+ file.read_to_end(&mut bytes)?;
+ if bytes.len() as u64 > MAX_JOURNAL_BYTES {
+ bail!(
+ "recovery journal {} exceeds the {} byte limit",
+ path.display(),
+ MAX_JOURNAL_BYTES
+ );
+ }
+ let journal: JournalFile = serde_json::from_slice(&bytes).with_context(|| {
+ format!(
+ "parsing {}; refusing to discard the recorded recovery work",
+ path.display()
+ )
+ })?;
+ if journal.version != JOURNAL_VERSION {
+ bail!(
+ "unsupported recovery journal version {} in {} (expected {JOURNAL_VERSION})",
+ journal.version,
+ path.display()
+ );
+ }
+ if journal.entries.len() > MAX_JOURNAL_ENTRIES {
+ bail!(
+ "recovery journal {} exceeds the {} entry limit",
+ path.display(),
+ MAX_JOURNAL_ENTRIES
+ );
+ }
+ Ok(journal.entries)
+}
+
+/// Append one entry under the exclusive lock. Atomic (tmp + rename),
+/// owner-only, and symlink-refusing so a swapped file cannot be trampled.
+pub(crate) fn append_journal_entry(data_dir: &Path, entry: JournalEntry) -> Result<()> {
+ mutate_journal(data_dir, |entries| {
+ if entries.len() >= MAX_JOURNAL_ENTRIES {
+ bail!(
+ "recovery journal is full ({MAX_JOURNAL_ENTRIES} entries); run `ai-memory recover` before another degraded launch"
+ );
+ }
+ entries.push(entry);
+ Ok(())
+ })
+}
+
+/// Rewrite the journal under the exclusive lock with `mutate` applied. The
+/// lock spans the whole read-modify-write so two finishing launches cannot
+/// lose each other's entry.
+fn mutate_journal(
+ data_dir: &Path,
+ mutate: impl FnOnce(&mut Vec<JournalEntry>) -> Result<()>,
+) -> Result<()> {
+ validate_journal_entry_limit(data_dir)?;
+ let path = journal_path(data_dir);
+ let lock = open_private_lock(&data_dir.join(JOURNAL_LOCK_FILE))?;
+ lock.lock_exclusive()
+ .with_context(|| format!("locking {}", data_dir.join(JOURNAL_LOCK_FILE).display()))?;
+ let mut entries = load_journal(data_dir)?;
+ mutate(&mut entries)?;
+ let journal = JournalFile {
+ version: JOURNAL_VERSION,
+ entries,
+ };
+ let mut rendered =
+ serde_json::to_vec_pretty(&journal).context("serializing the recovery journal")?;
+ rendered.push(b'\n');
+ refuse_symlink(&path)?;
+ if rendered.len() as u64 > MAX_JOURNAL_BYTES {
+ bail!(
+ "recovery journal {} would exceed the {} byte limit",
+ path.display(),
+ MAX_JOURNAL_BYTES
+ );
+ }
+ write_journal_atomic(&path, &rendered)
+ .with_context(|| format!("writing {}", path.display()))?;
+ make_private(&path)?;
+ Ok(())
+}
+
+fn write_journal_atomic(path: &Path, bytes: &[u8]) -> Result<()> {
+ let parent = path
+ .parent()
+ .ok_or_else(|| anyhow!("recovery journal path has no parent"))?;
+ refuse_symlink(parent)?;
+ let mut temp = tempfile::Builder::new()
+ .prefix(".recovery-journal.")
+ .tempfile_in(parent)?;
+ {
+ use std::io::Write as _;
+ temp.write_all(bytes)?;
+ temp.as_file().sync_data()?;
+ }
+ refuse_symlink(path)?;
+ temp.persist(path).map_err(|error| error.error)?;
+ if let Ok(directory) = File::open(parent) {
+ let _ = directory.sync_all();
+ }
+ Ok(())
+}
+
+fn validate_journal_entry_limit(data_dir: &Path) -> Result<()> {
+ let path = journal_path(data_dir);
+ refuse_symlink(&path)?;
+ match fs::metadata(&path) {
+ Ok(metadata) if metadata.len() > MAX_JOURNAL_BYTES => bail!(
+ "recovery journal {} exceeds the {} byte limit",
+ path.display(),
+ MAX_JOURNAL_BYTES
+ ),
+ Ok(_) => Ok(()),
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),
+ Err(error) => Err(error).with_context(|| format!("inspecting {}", path.display())),
+ }
+}
+
+fn open_journal_read(path: &Path) -> std::io::Result<File> {
+ let mut options = OpenOptions::new();
+ options.read(true);
+ #[cfg(unix)]
+ {
+ use std::os::unix::fs::OpenOptionsExt as _;
+ options.custom_flags(libc::O_NOFOLLOW);
+ }
+ options.open(path)
+}
+
+fn open_private_lock(path: &Path) -> Result<File> {
+ if let Some(parent) = path.parent() {
+ refuse_symlink(parent)?;
+ crate::commands::path_util::create_private_dir(parent)
+ .with_context(|| format!("creating {}", parent.display()))?;
+ refuse_symlink(parent)?;
+ }
+ refuse_symlink(path)?;
+ let mut options = OpenOptions::new();
+ options.create(true).read(true).write(true);
+ #[cfg(unix)]
+ {
+ use std::os::unix::fs::OpenOptionsExt as _;
+ options.mode(0o600).custom_flags(libc::O_NOFOLLOW);
+ }
+ let file = options
+ .open(path)
+ .with_context(|| format!("opening {}", path.display()))?;
+ refuse_symlink(path)?;
+ make_private(path)?;
+ Ok(file)
+}
+
+fn refuse_symlink(path: &Path) -> Result<()> {
+ let mut current = PathBuf::new();
+ for component in path.components() {
+ current.push(component.as_os_str());
+ match fs::symlink_metadata(&current) {
+ Ok(metadata) if metadata.file_type().is_symlink() => {
+ bail!("refusing recovery journal symlink {}", current.display());
+ }
+ Ok(_) => {}
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => break,
+ Err(error) => {
+ return Err(error).with_context(|| format!("inspecting {}", current.display()));
+ }
+ }
+ }
+ Ok(())
+}
+
+#[cfg(unix)]
+fn make_private(path: &Path) -> Result<()> {
+ use std::os::unix::fs::PermissionsExt as _;
+ fs::set_permissions(path, fs::Permissions::from_mode(0o600))
+ .with_context(|| format!("restricting permissions on {}", path.display()))
+}
+
+#[cfg(windows)]
+fn make_private(path: &Path) -> Result<()> {
+ let identity = std::process::Command::new("whoami")
+ .output()
+ .context("resolving the Windows recovery-journal owner")?;
+ if !identity.status.success() {
+ bail!("could not resolve the Windows recovery-journal owner");
+ }
+ let owner =
+ String::from_utf8(identity.stdout).context("reading the Windows recovery-journal owner")?;
+ let grant = format!("{}:(F)", owner.trim());
+ let restricted = std::process::Command::new("icacls")
+ .arg(path)
+ .args(["/inheritance:r", "/grant:r"])
+ .arg(grant)
+ .output()
+ .with_context(|| format!("restricting permissions on {}", path.display()))?;
+ if !restricted.status.success() {
+ bail!("could not restrict permissions on {}", path.display());
+ }
+ Ok(())
+}
+
+#[cfg(not(any(unix, windows)))]
+fn make_private(_path: &Path) -> Result<()> {
+ Ok(())
+}
+
+/// Count journal entries for status lines; an unreadable journal reports zero
+/// with its error carried back, never blocking status output.
+pub(crate) fn journal_summary(data_dir: &Path) -> (usize, usize, usize, Option<String>) {
+ match load_journal(data_dir) {
+ Ok(entries) => {
+ let degraded = entries
+ .iter()
+ .filter(|entry| entry.kind == JournalKind::DegradedRun)
+ .count();
+ let finish_failed = entries.len() - degraded;
+ (entries.len(), degraded, finish_failed, None)
+ }
+ Err(error) => (0, 0, 0, Some(format!("{error:#}"))),
+ }
+}
+
+/// Budgets for one `recover` invocation; injectable so tests can prove the
+/// bounds bite.
+#[derive(Debug, Clone, Copy)]
+struct RecoverBudgets {
+ /// Total spool-drain budget. Generous but bounded, matching the hook
+ /// background-drain default: recovery must not hang forever on a wedged
+ /// server, but a large outage backlog must fully flush.
+ drain_total: Duration,
+ /// Per-event POST timeout while draining.
+ drain_event_timeout: Duration,
+ /// Per-journal-entry transcript HTTP budget.
+ entry_timeout: Duration,
+ /// Total journal HTTP budget.
+ journal_total: Duration,
+ /// How long recovery waits on another process's drain lock.
+ drain_lock_wait: Duration,
+}
+
+impl Default for RecoverBudgets {
+ fn default() -> Self {
+ Self {
+ drain_total: Duration::from_secs(5 * 60),
+ drain_event_timeout: Duration::from_secs(30),
+ entry_timeout: Duration::from_secs(60),
+ journal_total: Duration::from_secs(10 * 60),
+ drain_lock_wait: Duration::from_secs(30),
+ }
+ }
+}
+
+/// What recovery did for one journal entry.
+#[derive(Debug)]
+enum EntryRecovery {
+ /// Re-exported and imported; the count is newly imported content events.
+ Imported(usize),
+ /// The harness has no transcript exporter and its complete spool was delivered.
+ RecoveredViaSpool,
+ /// Recovery failed; the entry stays journaled with the reason.
+ Failed(String),
+}
+
+/// Spool-drain counts for the report.
+#[derive(Debug, Default, Clone, Copy, Serialize)]
+struct SpoolOutcome {
+ sent: usize,
+ remaining: usize,
+ dropped: usize,
+}
+
+/// One recovered (or failed) session, for the report.
+#[derive(Debug, Serialize)]
+struct SessionOutcome {
+ kind: &'static str,
+ harness: String,
+ native_session_id: Option<String>,
+ workspace: String,
+ project: String,
+ recorded_at: String,
+ outcome: &'static str,
+ imported_events: usize,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ detail: Option<String>,
+}
+
+/// The full `ai-memory recover` report.
+#[derive(Debug, Serialize)]
+struct RecoverReport {
+ server: String,
+ reachable: bool,
+ spool: SpoolOutcome,
+ spool_lock_busy: bool,
+ spool_incomplete: bool,
+ journal_pending: usize,
+ journal_degraded: usize,
+ journal_finish_failed: usize,
+ sessions: Vec<SessionOutcome>,
+ journal_remaining: usize,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ journal_error: Option<String>,
+}
+
+impl RecoverReport {
+ fn failures(&self) -> usize {
+ self.sessions
+ .iter()
+ .filter(|session| session.outcome == "failed")
+ .count()
+ }
+
+ fn imported_total(&self) -> usize {
+ self.sessions.iter().map(|s| s.imported_events).sum()
+ }
+}
+
+/// Run the `recover` subcommand and return its process exit code.
+///
+/// # Errors
+/// Returns an error only for local infrastructure failures.
+pub async fn run(config: &Config, args: RecoverArgs) -> Result<i32> {
+ let report = recover(config, &RecoverBudgets::default()).await?;
+ if args.json {
+ println!("{}", serde_json::to_string_pretty(&report)?);
+ } else {
+ print_report(&report);
+ }
+ Ok(i32::from(
+ report.reachable
+ && (report.spool_incomplete || report.failures() > 0 || report.journal_error.is_some()),
+ ))
+}
+
+async fn recover(config: &Config, budgets: &RecoverBudgets) -> Result<RecoverReport> {
+ let endpoint = ServerEndpoint::from_config_resolving_auth(config).await;
+ let spool_dir = hook_spool::spool_dir(&config.data_dir);
+ let (mut journal, journal_error) = match load_journal(&config.data_dir) {
+ Ok(entries) => (entries, None),
+ // A broken journal still leaves the spool drain: the two recover
+ // independent halves of the outage.
+ Err(error) => (Vec::new(), Some(format!("{error:#}"))),
+ };
+ let journal_degraded = journal
+ .iter()
+ .filter(|entry| entry.kind == JournalKind::DegradedRun)
+ .count();
+ let journal_finish_failed = journal.len() - journal_degraded;
+
+ if let ServerProbe::Unreachable(_) = probe_server(&endpoint).await {
+ return Ok(RecoverReport {
+ server: endpoint.url.clone(),
+ reachable: false,
+ spool: SpoolOutcome {
+ remaining: hook_spool::spool_len(&spool_dir),
+ ..SpoolOutcome::default()
+ },
+ spool_lock_busy: false,
+ spool_incomplete: false,
+ journal_pending: journal.len(),
+ journal_degraded,
+ journal_finish_failed,
+ sessions: Vec::new(),
+ journal_remaining: journal.len(),
+ journal_error,
+ });
+ }
+
+ let sanitizer = Sanitizer::new(&config.sanitize).map_err(|error| anyhow!(error.to_string()))?;
+ let journal_pending = journal.len();
+ let mut quarantined = HashMap::new();
+ let drain_lock = match hook_spool::acquire_drain_lock(
+ &spool_dir,
+ DrainLockWait::Bounded(budgets.drain_lock_wait),
+ ) {
+ Ok(Some(lock)) => lock,
+ Ok(None) => {
+ return Ok(lock_busy_report(
+ &endpoint,
+ &spool_dir,
+ journal_pending,
+ journal_degraded,
+ journal_finish_failed,
+ journal_error,
+ ));
+ }
+ Err(error) => {
+ return Err(error).context("acquiring the recovery spool lock");
+ }
+ };
+ let restore_quarantine = |quarantined: HashMap<String, hook_spool::QuarantinedSpool>| {
+ let mut first_error = None;
+ for (_, quarantine) in quarantined {
+ if let Err(error) = quarantine.restore(&spool_dir)
+ && first_error.is_none()
+ {
+ first_error = Some(error);
+ }
+ }
+ first_error.map_or(Ok(()), Err)
+ };
+ for entry in &journal {
+ if entry.kind != JournalKind::DegradedRun && !entry.spool_only {
+ continue;
+ }
+ let Some(native_session_id) = entry.native_session_id.as_deref() else {
+ continue;
+ };
+ match hook_spool::quarantine_session_entries(
+ config,
+ &spool_dir,
+ &entry.id,
+ entry.interval_started_ms,
+ entry.interval_ended_ms,
+ &entry.server_url,
+ super::run::harness_from_journal_name(&entry.harness)
+ .map(|harness| harness.agent_kind().as_str())
+ .unwrap_or(&entry.harness),
+ &entry.workspace,
+ &entry.project,
+ Path::new(&entry.cwd),
+ native_session_id,
+ ) {
+ Ok(quarantine) => {
+ quarantined.insert(entry.id.clone(), quarantine);
+ }
+ Err(error) => {
+ restore_quarantine(quarantined)
+ .context("restoring previously quarantined hook events")?;
+ return Err(error).context("quarantining correlated hook events");
+ }
+ }
+ }
+ let mut evidence_started = Vec::new();
+ for entry in &mut journal {
+ if entry.kind != JournalKind::DegradedRun && !entry.spool_only {
+ continue;
+ }
+ let correlated_count = quarantined.get(&entry.id).map_or(0, |value| value.len());
+ if correlated_count > 0 {
+ entry.correlated_events_seen = true;
+ entry.correlated_loss = true;
+ evidence_started.push(entry.id.clone());
+ }
+ }
+ if !evidence_started.is_empty() {
+ update_journal_evidence(&config.data_dir, &journal)?;
+ }
+ for entry in &mut journal {
+ if !entry_supports_transcript_export(entry)
+ && let Some(quarantine) = quarantined.get(&entry.id)
+ && quarantine.len() > 0
+ {
+ let outcome = quarantine
+ .drain_strict(
+ &config.data_dir,
+ budgets.drain_total,
+ budgets.drain_event_timeout,
+ MaxAttempts::new(config.hook_spool.max_attempts),
+ )
+ .await;
+ if outcome.durable > 0 {
+ entry.correlated_events_seen = true;
+ }
+ if outcome.dropped > 0 {
+ entry.correlated_loss = true;
+ }
+ if outcome.remaining == 0
+ && outcome.dropped == 0
+ && outcome.durable > 0
+ && quarantine.complete_spool_only()?
+ {
+ entry.correlated_delivery_confirmed = true;
+ entry.correlated_loss = false;
+ }
+ }
+ }
+ update_journal_evidence(&config.data_dir, &journal)?;
+ let drained = hook_spool::drain_until_quiescent_locked_with_live_token(
+ &spool_dir,
+ &config.data_dir,
+ budgets.drain_total,
+ budgets.drain_event_timeout,
+ config.auth.bearer_token.as_deref(),
+ MaxAttempts::new(config.hook_spool.max_attempts),
+ )
+ .await;
+ let spool = SpoolOutcome {
+ sent: drained.sent,
+ remaining: drained.remaining,
+ dropped: drained.dropped,
+ };
+ let spool_incomplete = spool_is_incomplete(&spool);
+ if spool.dropped > 0 {
+ eprintln!(
+ "ai-memory: recovery: {count} hook event(s) were dropped; recovery remains incomplete unless an exact correlated transcript replaces them",
+ count = spool.dropped
+ );
+ }
+
+ let mut sessions = Vec::with_capacity(journal.len());
+ let mut cleared: Vec<String> = Vec::with_capacity(journal.len());
+ let mut annotated: Vec<(String, String)> = Vec::new();
+ let journal_started = Instant::now();
+ for entry in journal {
+ let mut recovery = if spool_incomplete {
+ EntryRecovery::Failed(format!(
+ "hook spool recovery was incomplete ({} queued, {} dropped); transcript recovery was deferred",
+ spool.remaining, spool.dropped
+ ))
+ } else if let Some(timeout) = entry_budget(journal_started, budgets) {
+ if timeout.is_zero() {
+ failed_reason(
+ &sanitizer,
+ "journal entry recovery timed out; rerun to continue",
+ )
+ } else {
+ match tokio::time::timeout(
+ timeout,
+ recover_entry(config, &endpoint, &sanitizer, &entry),
+ )
+ .await
+ {
+ Ok(recovery) => recovery,
+ Err(_) => failed_reason(
+ &sanitizer,
+ "journal entry recovery timed out; rerun to continue",
+ ),
+ }
+ }
+ } else {
+ failed_reason(
+ &sanitizer,
+ "total recovery budget exhausted; rerun to continue",
+ )
+ };
+ if let Some(quarantine) = quarantined.remove(&entry.id) {
+ let restored = match &recovery {
+ EntryRecovery::Imported(_) => {
+ quarantine.complete().map(|()| RestoreOutcome::Restored)
+ }
+ EntryRecovery::RecoveredViaSpool => Ok(RestoreOutcome::Restored),
+ EntryRecovery::Failed(_) => quarantine.restore(&spool_dir),
+ }
+ .context("restoring quarantined hook events")?;
+ if matches!(restored, RestoreOutcome::AlreadyDelivered) {
+ // The completed marker is durable proof this interval's
+ // correlated evidence reached the server on a prior pass;
+ // the failure only deferred work there is none left of, so
+ // the entry clears instead of staying journaled behind it.
+ recovery = EntryRecovery::RecoveredViaSpool;
+ }
+ }
+ let outcome = match &recovery {
+ EntryRecovery::Imported(_) => "imported",
+ EntryRecovery::RecoveredViaSpool => "recovered-via-spool",
+ EntryRecovery::Failed(_) => "failed",
+ };
+ let imported_events = match &recovery {
+ EntryRecovery::Imported(events) => *events,
+ _ => 0,
+ };
+ let detail = match &recovery {
+ EntryRecovery::Imported(_) | EntryRecovery::RecoveredViaSpool => None,
+ EntryRecovery::Failed(note) => Some(note.clone()),
+ };
+ if let EntryRecovery::Failed(reason) = &recovery {
+ annotated.push((entry.id.clone(), reason.clone()));
+ } else {
+ cleared.push(entry.id.clone());
+ }
+ sessions.push(SessionOutcome {
+ kind: entry.kind.as_str(),
+ harness: entry.harness.clone(),
+ native_session_id: entry.native_session_id.clone(),
+ workspace: entry.workspace.clone(),
+ project: entry.project.clone(),
+ recorded_at: entry.recorded_at.clone(),
+ outcome,
+ imported_events,
+ detail,
+ });
+ }
+
+ restore_quarantine(quarantined)?;
+ drop(drain_lock);
+
+ // (d) Clear succeeded entries; failures stay with their reason.
+ let journal_remaining = finish_journal(&config.data_dir, &sanitizer, &cleared, &annotated)?;
+
+ Ok(RecoverReport {
+ server: endpoint.url.clone(),
+ reachable: true,
+ spool,
+ spool_lock_busy: false,
+ spool_incomplete,
+ journal_pending,
+ journal_degraded,
+ journal_finish_failed,
+ sessions,
+ journal_remaining,
+ journal_error,
+ })
+}
+
+fn entry_budget(started: Instant, budgets: &RecoverBudgets) -> Option<Duration> {
+ if budgets.entry_timeout.is_zero() {
+ return Some(Duration::ZERO);
+ }
+ let remaining = budgets.journal_total.checked_sub(started.elapsed())?;
+ (!remaining.is_zero()).then(|| budgets.entry_timeout.min(remaining))
+}
+
+fn spool_is_incomplete(spool: &SpoolOutcome) -> bool {
+ spool.remaining > 0 || spool.dropped > 0
+}
+
+fn lock_busy_report(
+ endpoint: &ServerEndpoint,
+ spool_dir: &Path,
+ journal_pending: usize,
+ journal_degraded: usize,
+ journal_finish_failed: usize,
+ journal_error: Option<String>,
+) -> RecoverReport {
+ RecoverReport {
+ server: endpoint.url.clone(),
+ reachable: true,
+ spool: SpoolOutcome {
+ remaining: hook_spool::spool_len(spool_dir),
+ ..SpoolOutcome::default()
+ },
+ spool_lock_busy: true,
+ spool_incomplete: true,
+ journal_pending,
+ journal_degraded,
+ journal_finish_failed,
+ sessions: Vec::new(),
+ journal_remaining: journal_pending,
+ journal_error,
+ }
+}
+
+async fn recover_entry(
+ config: &Config,
+ endpoint: &ServerEndpoint,
+ sanitizer: &Sanitizer,
+ entry: &JournalEntry,
+) -> EntryRecovery {
+ let recorded = entry.server_url.trim_end_matches('/');
+ let configured_identity = endpoint.identity();
+ let configured = configured_identity.trim_end_matches('/');
+ if recorded.is_empty() || recorded != configured {
+ return failed_reason(
+ sanitizer,
+ &format!("refusing to replay an entry recorded for {recorded:?} against {configured}"),
+ );
+ }
+ if entry.spool_only {
+ return recover_spool_only(sanitizer, entry);
+ }
+ match entry.kind {
+ JournalKind::FinishFailed => {
+ recover_finish_failed(config, endpoint, sanitizer, entry).await
+ }
+ JournalKind::DegradedRun => recover_degraded_run(config, endpoint, sanitizer, entry).await,
+ }
+}
+
+/// Re-export the transcript from the native store and finish the run again.
+/// Reuses `run`'s import machinery verbatim: the server dedups by stable event
+/// id, so a full re-export cannot duplicate history a partial import landed.
+fn recovery_session_dir(entry: &JournalEntry) -> Option<PathBuf> {
+ entry.session_dir.as_deref().map(|path| {
+ let path = Path::new(path);
+ if path.is_absolute() {
+ path.to_path_buf()
+ } else {
+ Path::new(&entry.cwd).join(path)
+ }
+ })
+}
+
+async fn recover_finish_failed(
+ config: &Config,
+ endpoint: &ServerEndpoint,
+ sanitizer: &Sanitizer,
+ entry: &JournalEntry,
+) -> EntryRecovery {
+ let Some(run_id) = entry.run_id.as_deref() else {
+ return failed_reason(sanitizer, "finish-failed entry records no run id");
+ };
+ if ManagedRunId::from_str(run_id).is_err() {
+ return failed_reason(
+ sanitizer,
+ &format!("journal records an invalid run id {run_id}"),
+ );
+ }
+ let Some(harness) = super::run::harness_from_journal_name(&entry.harness) else {
+ return failed_reason(
+ sanitizer,
+ &format!("journal names unsupported harness '{}'", entry.harness),
+ );
+ };
+ let Some(home) = super::run::native_home(config) else {
+ return failed_reason(
+ sanitizer,
+ "the native harness home directory could not be located",
+ );
+ };
+ let Some(expected_digests) = entry.event_digests.as_ref() else {
+ return failed_reason(
+ sanitizer,
+ "finish-failed entry has no exact ordered transcript identity; refusing full transcript replay",
+ );
+ };
+ let Some(final_cursor) = entry.final_cursor.as_deref() else {
+ return failed_reason(
+ sanitizer,
+ "finish-failed entry has no exact post-run transcript cursor; refusing replay",
+ );
+ };
+ let transcript = match entry.native_session_id.as_deref() {
+ Some(native_session_id) => {
+ match ai_memory_workstream::export_transcript_range(
+ harness,
+ &home,
+ Path::new(&entry.cwd),
+ recovery_session_dir(entry).as_deref(),
+ native_session_id,
+ entry.source_cursor.as_deref(),
+ final_cursor,
+ )
+ .await
+ {
+ Ok(transcript) => transcript,
+ Err(error) => {
+ return failed_reason(
+ sanitizer,
+ &format!("native transcript re-export failed: {error:#}"),
+ );
+ }
+ }
+ }
+ None => {
+ return failed_reason(
+ sanitizer,
+ "finish-failed entry records no native transcript locator",
+ );
+ }
+ };
+ let run_path = format!("/workstream/runs/{run_id}/recover");
+ let actual = ai_memory_workstream::transcript_interval_digests(&transcript.events);
+ if &actual != expected_digests {
+ return failed_reason(
+ sanitizer,
+ "native transcript changed since the failed finish; refusing non-identical replay",
+ );
+ }
+ let transcript = ai_memory_workstream::ExportedTranscript {
+ source_cursor: Some(final_cursor.to_owned()),
+ ..transcript
+ };
+ match super::run::import_batches(
+ endpoint,
+ &run_path,
+ transcript,
+ ai_memory_core::WorkstreamCheckpoint::default(),
+ entry.exit_code,
+ sanitizer,
+ )
+ .await
+ {
+ Ok(imported) => EntryRecovery::Imported(imported),
+ Err(error) => failed_reason(sanitizer, &format!("managed finish failed: {error:#}")),
+ }
+}
+
+fn failed_reason(sanitizer: &Sanitizer, reason: &str) -> EntryRecovery {
+ EntryRecovery::Failed(ai_memory_core::truncate_utf8_bytes(
+ &sanitizer.scrub(reason),
+ MAX_FAILURE_REASON_BYTES,
+ ))
+}
+
+/// A degraded launch had no server run, so recover its exact native transcript
+/// through the ordinary backfill ingress. That path emits stable keyed
+/// session-start/content/session-end items through `/hook/batch`, making a
+/// repeated recovery idempotent while still crossing the normal sanitizer and
+/// consolidation boundary.
+fn entry_supports_transcript_export(entry: &JournalEntry) -> bool {
+ !entry.spool_only
+ && super::run::harness_from_journal_name(&entry.harness)
+ .is_some_and(|harness| harness.transcript_capability() == TranscriptCapability::Export)
+}
+
+fn recover_spool_only(sanitizer: &Sanitizer, entry: &JournalEntry) -> EntryRecovery {
+ if entry.correlated_delivery_confirmed && !entry.correlated_loss {
+ EntryRecovery::RecoveredViaSpool
+ } else {
+ failed_reason(
+ sanitizer,
+ "no durably delivered correlated hook evidence exists; retain this entry and repair the native session manually",
+ )
+ }
+}
+
+async fn recover_degraded_run(
+ config: &Config,
+ endpoint: &ServerEndpoint,
+ sanitizer: &Sanitizer,
+ entry: &JournalEntry,
+) -> EntryRecovery {
+ let Some(harness) = super::run::harness_from_journal_name(&entry.harness) else {
+ return failed_reason(
+ sanitizer,
+ &format!("journal names unsupported harness '{}'", entry.harness),
+ );
+ };
+ if harness.transcript_capability() == TranscriptCapability::SpoolOnly {
+ return recover_spool_only(sanitizer, entry);
+ }
+ let Some(native_session_id) = entry.native_session_id.as_deref() else {
+ return failed_reason(
+ sanitizer,
+ "no native session was recorded for this degraded run",
+ );
+ };
+ let Some(home) = super::run::native_home(config) else {
+ return failed_reason(
+ sanitizer,
+ "the native harness home directory could not be located",
+ );
+ };
+ let Some(expected_digests) = entry.event_digests.as_deref() else {
+ return failed_reason(
+ sanitizer,
+ "degraded-run entry has no exact ordered transcript identity; repair the native session manually",
+ );
+ };
+ let Some(final_cursor) = entry.final_cursor.as_deref() else {
+ return failed_reason(
+ sanitizer,
+ "degraded-run entry has no exact post-run transcript cursor; repair the native session manually",
+ );
+ };
+ match super::backfill::import_exact_session_range(
+ endpoint,
+ &entry.workspace,
+ &entry.project,
+ &home,
+ Path::new(&entry.cwd),
+ harness,
+ native_session_id,
+ &entry.id,
+ recovery_session_dir(entry).as_deref(),
+ entry.source_cursor.as_deref(),
+ final_cursor,
+ expected_digests,
+ )
+ .await
+ {
+ Ok(imported) => EntryRecovery::Imported(imported),
+ Err(error) => failed_reason(
+ sanitizer,
+ &format!("native transcript replay failed: {error:#}"),
+ ),
+ }
+}
+
+/// Apply clears and failure annotations in one locked rewrite; returns the
+/// remaining entry count.
+fn update_journal_evidence(data_dir: &Path, current: &[JournalEntry]) -> Result<()> {
+ mutate_journal(data_dir, |entries| {
+ for entry in entries {
+ if let Some(updated) = current.iter().find(|candidate| candidate.id == entry.id) {
+ entry.correlated_events_seen = updated.correlated_events_seen;
+ entry.correlated_delivery_confirmed = updated.correlated_delivery_confirmed;
+ entry.correlated_loss = updated.correlated_loss;
+ }
+ }
+ Ok(())
+ })
+}
+
+fn finish_journal(
+ data_dir: &Path,
+ sanitizer: &Sanitizer,
+ cleared: &[String],
+ annotated: &[(String, String)],
+) -> Result<usize> {
+ if cleared.is_empty() && annotated.is_empty() {
+ return Ok(load_journal(data_dir).map_or(0, |entries| entries.len()));
+ }
+ let mut remaining = 0_usize;
+ mutate_journal(data_dir, |entries| {
+ entries.retain(|entry| !cleared.contains(&entry.id));
+ for (id, reason) in annotated {
+ if let Some(entry) = entries.iter_mut().find(|entry| &entry.id == id) {
+ let reason = ai_memory_core::truncate_utf8_bytes(
+ &sanitizer.scrub(reason),
+ MAX_FAILURE_REASON_BYTES,
+ );
+ if !entry.failures.contains(&reason) {
+ entry.failures.push(reason);
+ }
+ let overflow = entry.failures.len().saturating_sub(MAX_ENTRY_FAILURES);
+ if overflow > 0 {
+ entry.failures.drain(..overflow);
+ }
+ }
+ }
+ remaining = entries.len();
+ Ok(())
+ })?;
+ Ok(remaining)
+}
+
+fn print_report(report: &RecoverReport) {
+ if !report.reachable {
+ println!(
+ "ai-memory: server at {} is unreachable — nothing to recover yet.",
+ report.server
+ );
+ if let Some(error) = &report.journal_error {
+ println!(" journal: unreadable ({error})");
+ } else {
+ println!(
+ " journal: {} entr{} await{} recovery ({} degraded launch(es), {} failed \
+ transcript import(s))",
+ report.journal_pending,
+ if report.journal_pending == 1 {
+ "y"
+ } else {
+ "ies"
+ },
+ if report.journal_pending == 1 { "s" } else { "" },
+ report.journal_degraded,
+ report.journal_finish_failed,
+ );
+ }
+ println!(
+ " spool: {} hook event(s) queued locally",
+ report.spool.remaining
+ );
+ println!(
+ "Start the server (or fix AI_MEMORY_SERVER_URL), then run `ai-memory recover` again."
+ );
+ return;
+ }
+ println!("ai-memory: recovering against {}…", report.server);
+ if report.spool_lock_busy {
+ println!(" spool: another drainer held the lock; transcript recovery was deferred");
+ } else {
+ println!(
+ " spool: {} hook event(s) delivered, {} remain queued, {} dropped",
+ report.spool.sent, report.spool.remaining, report.spool.dropped
+ );
+ }
+ if report.spool_incomplete {
+ println!(
+ " recovery is incomplete because hook events remain queued or were dropped; exact correlated transcripts may still replace quarantined events"
+ );
+ }
+ for session in &report.sessions {
+ let target = session
+ .native_session_id
+ .as_deref()
+ .unwrap_or("(no session id)");
+ let mut line = format!(
+ " [{}] {} {} in {}/{} — {}",
+ session.kind,
+ session.harness,
+ target,
+ session.workspace,
+ session.project,
+ session.outcome,
+ );
+ if session.imported_events > 0 {
+ line.push_str(&format!(" ({} events)", session.imported_events));
+ }
+ println!("{line}");
+ if let Some(detail) = &session.detail {
+ println!(" {detail}");
+ }
+ }
+ println!(
+ "recovered {} event(s) across {} journal entr{}; {} failed; {} remain",
+ report.imported_total(),
+ report.sessions.len(),
+ if report.sessions.len() == 1 {
+ "y"
+ } else {
+ "ies"
+ },
+ report.failures(),
+ report.journal_remaining,
+ );
+ if let Some(error) = &report.journal_error {
+ println!("recovery journal error: {error}");
+ }
+ if report.failures() > 0 {
+ println!(
+ "failed entries stay journaled with their reason; fix the cause and re-run \
+ `ai-memory recover` (re-running is safe — imports dedup)"
+ );
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn test_entry(id: &str, kind: JournalKind) -> JournalEntry {
+ JournalEntry {
+ id: id.to_string(),
+ recorded_at: "2026-10-07T12:00:00Z".to_string(),
+ interval_started_ms: 1_000,
+ interval_ended_ms: 2_000,
+ kind,
+ harness: "claude".to_string(),
+ native_session_id: Some("sess-1".to_string()),
+ session_dir: None,
+ cwd: "/tmp/repo".to_string(),
+ workspace: "ws".to_string(),
+ project: "proj".to_string(),
+ server_url: "http://127.0.0.1:49374".to_string(),
+ run_id: (kind == JournalKind::FinishFailed)
+ .then(|| "12345678-1234-4234-9234-123456789abe".to_string()),
+ event_digests: Some(Vec::new()),
+ spool_only: false,
+ source_cursor: None,
+ final_cursor: None,
+ correlated_events_seen: false,
+ correlated_delivery_confirmed: false,
+ correlated_loss: false,
+ exit_code: Some(0),
+ failures: Vec::new(),
+ }
+ }
+
+ #[test]
+ fn append_then_load_round_trips() {
+ let temp = tempfile::tempdir().unwrap();
+ let entry = test_entry("run-1", JournalKind::FinishFailed);
+ append_journal_entry(temp.path(), entry.clone()).unwrap();
+ let loaded = load_journal(temp.path()).unwrap();
+ assert_eq!(loaded.len(), 1);
+ assert_eq!(loaded[0].id, "run-1");
+ assert_eq!(loaded[0].kind, JournalKind::FinishFailed);
+ assert_eq!(loaded[0].harness, entry.harness);
+ assert_eq!(loaded[0].native_session_id.as_deref(), Some("sess-1"));
+ }
+
+ #[test]
+ fn missing_journal_is_empty_and_append_creates_it_private() {
+ let temp = tempfile::tempdir().unwrap();
+ assert!(load_journal(temp.path()).unwrap().is_empty());
+ append_journal_entry(temp.path(), test_entry("a", JournalKind::DegradedRun)).unwrap();
+ assert_eq!(load_journal(temp.path()).unwrap().len(), 1);
+ #[cfg(unix)]
+ {
+ use std::os::unix::fs::PermissionsExt as _;
+ let mode = fs::metadata(journal_path(temp.path()))
+ .unwrap()
+ .permissions()
+ .mode();
+ assert_eq!(mode & 0o777, 0o600, "the journal is owner-only");
+ }
+ }
+
+ #[test]
+ fn full_journal_refuses_append_without_evicting() {
+ let temp = tempfile::tempdir().unwrap();
+ for index in 0..MAX_JOURNAL_ENTRIES {
+ append_journal_entry(
+ temp.path(),
+ test_entry(&format!("entry-{index}"), JournalKind::DegradedRun),
+ )
+ .unwrap();
+ }
+ let error = append_journal_entry(
+ temp.path(),
+ test_entry("must-not-fit", JournalKind::DegradedRun),
+ )
+ .unwrap_err();
+ assert!(format!("{error:#}").contains("ai-memory recover"));
+ let loaded = load_journal(temp.path()).unwrap();
+ assert_eq!(loaded.len(), MAX_JOURNAL_ENTRIES);
+ assert_eq!(loaded[0].id, "entry-0");
+ assert_eq!(
+ loaded.last().unwrap().id,
+ format!("entry-{}", MAX_JOURNAL_ENTRIES - 1)
+ );
+ }
+
+ #[test]
+ fn appends_are_atomic_and_leave_no_temporary_files() {
+ // The writer only ever renames a complete tmp file into place, so a
+ // crash between appends cannot leave a half-written journal: the
+ // previous complete file is still there and parseable.
+ let temp = tempfile::tempdir().unwrap();
+ append_journal_entry(temp.path(), test_entry("first", JournalKind::DegradedRun)).unwrap();
+ let before = fs::read(journal_path(temp.path())).unwrap();
+ append_journal_entry(temp.path(), test_entry("second", JournalKind::DegradedRun)).unwrap();
+ let after = fs::read(journal_path(temp.path())).unwrap();
+ assert!(before.len() < after.len());
+ assert_eq!(load_journal(temp.path()).unwrap().len(), 2);
+ assert!(
+ !fs::read_dir(temp.path()).unwrap().any(|entry| entry
+ .unwrap()
+ .file_name()
+ .to_string_lossy()
+ .contains(".tmp")),
+ "no temporary file is left behind"
+ );
+ }
+
+ #[test]
+ fn corrupt_oversized_or_wrong_version_journal_is_refused_not_reset() {
+ let temp = tempfile::tempdir().unwrap();
+ let path = journal_path(temp.path());
+ fs::write(&path, b"{ not json").unwrap();
+ let error = load_journal(temp.path()).unwrap_err();
+ assert!(
+ format!("{error:#}").contains("refusing to discard"),
+ "{error:#}"
+ );
+ let error = append_journal_entry(temp.path(), test_entry("x", JournalKind::DegradedRun))
+ .unwrap_err();
+ assert!(
+ format!("{error:#}").contains("refusing to discard"),
+ "{error:#}"
+ );
+ assert_eq!(fs::read(&path).unwrap(), b"{ not json");
+
+ let entries = (0..=MAX_JOURNAL_ENTRIES)
+ .map(|index| test_entry(&index.to_string(), JournalKind::DegradedRun))
+ .collect();
+ fs::write(
+ &path,
+ serde_json::to_vec(&JournalFile {
+ version: JOURNAL_VERSION,
+ entries,
+ })
+ .unwrap(),
+ )
+ .unwrap();
+ let error = load_journal(temp.path()).unwrap_err();
+ assert!(format!("{error:#}").contains("entry limit"), "{error:#}");
+ }
+
+ #[cfg(unix)]
+ #[test]
+ fn symlinked_lock_is_refused_without_touching_its_target() {
+ let temp = tempfile::tempdir().unwrap();
+ let outside = temp.path().join("outside-lock");
+ fs::write(&outside, b"canary").unwrap();
+ std::os::unix::fs::symlink(&outside, temp.path().join(JOURNAL_LOCK_FILE)).unwrap();
+ let error =
+ append_journal_entry(temp.path(), test_entry("outside", JournalKind::DegradedRun))
+ .unwrap_err();
+ assert!(format!("{error:#}").contains("symlink"), "{error:#}");
+ assert_eq!(fs::read(outside).unwrap(), b"canary");
+ }
+
+ #[cfg(unix)]
+ #[test]
+ fn symlinked_journal_and_ancestor_are_refused() {
+ let temp = tempfile::tempdir().unwrap();
+ let real = temp.path().join("real.json");
+ fs::write(&real, b"{}").unwrap();
+ std::os::unix::fs::symlink(&real, journal_path(temp.path())).unwrap();
+ let error = load_journal(temp.path()).unwrap_err();
+ assert!(format!("{error:#}").contains("symlink"), "{error:#}");
+
+ let outside = tempfile::tempdir().unwrap();
+ let linked_data = temp.path().join("linked-data");
+ std::os::unix::fs::symlink(outside.path(), &linked_data).unwrap();
+ let error = append_journal_entry(
+ &linked_data,
+ test_entry("outside", JournalKind::DegradedRun),
+ )
+ .unwrap_err();
+ assert!(format!("{error:#}").contains("symlink"), "{error:#}");
+ assert!(!outside.path().join(JOURNAL_FILE).exists());
+ }
+
+ #[cfg(windows)]
+ #[test]
+ fn journal_acl_is_restricted_to_the_current_owner() {
+ let temp = tempfile::tempdir().unwrap();
+ append_journal_entry(temp.path(), test_entry("acl", JournalKind::DegradedRun)).unwrap();
+ let output = std::process::Command::new("icacls")
+ .arg(journal_path(temp.path()))
+ .output()
+ .unwrap();
+ assert!(output.status.success());
+ let acl = String::from_utf8_lossy(&output.stdout).to_ascii_lowercase();
+ assert!(!acl.contains("everyone"), "{acl}");
+ assert!(!acl.contains("authenticated users"), "{acl}");
+ }
+
+ #[test]
+ fn clearing_entries_and_annotating_failures_rewrites_the_journal() {
+ let temp = tempfile::tempdir().unwrap();
+ for id in ["keep", "clear", "fail"] {
+ append_journal_entry(temp.path(), test_entry(id, JournalKind::DegradedRun)).unwrap();
+ }
+ finish_journal(
+ temp.path(),
+ &Sanitizer::builtin(),
+ &["clear".to_string()],
+ &[("fail".to_string(), "server said no".to_string())],
+ )
+ .unwrap();
+ let loaded = load_journal(temp.path()).unwrap();
+ assert_eq!(loaded.len(), 2);
+ assert!(loaded.iter().all(|entry| entry.id != "clear"));
+ let failed = loaded.iter().find(|entry| entry.id == "fail").unwrap();
+ assert_eq!(failed.failures, vec!["server said no".to_string()]);
+ // The same failure reason is not recorded twice.
+ finish_journal(
+ temp.path(),
+ &Sanitizer::builtin(),
+ &[],
+ &[("fail".to_string(), "server said no".to_string())],
+ )
+ .unwrap();
+ let loaded = load_journal(temp.path()).unwrap();
+ assert_eq!(
+ loaded
+ .iter()
+ .find(|entry| entry.id == "fail")
+ .unwrap()
+ .failures,
+ vec!["server said no".to_string()]
+ );
+ }
+
+ #[test]
+ fn failure_reasons_are_sanitized_and_bounded_per_entry() {
+ let temp = tempfile::tempdir().unwrap();
+ append_journal_entry(temp.path(), test_entry("e", JournalKind::DegradedRun)).unwrap();
+ for index in 0..(MAX_ENTRY_FAILURES + 2) {
+ finish_journal(
+ temp.path(),
+ &Sanitizer::builtin(),
+ &[],
+ &[(
+ "e".to_string(),
+ format!(
+ "reason {index} token sk-{} {}",
+ "a".repeat(32),
+ "x".repeat(MAX_FAILURE_REASON_BYTES * 2)
+ ),
+ )],
+ )
+ .unwrap();
+ }
+ let loaded = load_journal(temp.path()).unwrap();
+ assert_eq!(loaded[0].failures.len(), MAX_ENTRY_FAILURES);
+ assert!(loaded[0].failures[0].starts_with("reason 2 token [REDACTED:api_key]"));
+ assert!(
+ loaded[0]
+ .failures
+ .iter()
+ .all(|reason| reason.len() <= MAX_FAILURE_REASON_BYTES)
+ );
+ }
+
+ #[test]
+ fn unknown_and_known_empty_event_sets_remain_distinct() {
+ let mut unknown = test_entry("unknown", JournalKind::FinishFailed);
+ unknown.event_digests = None;
+ let unknown_json = serde_json::to_value(&unknown).unwrap();
+ assert!(unknown_json.get("event_digests").is_none());
+
+ let empty = test_entry("empty", JournalKind::FinishFailed);
+ let empty_json = serde_json::to_value(&empty).unwrap();
+ assert_eq!(empty_json["event_digests"], serde_json::json!([]));
+ }
+
+ #[test]
+ fn recovery_journal_omits_repository_state() {
+ let bytes = serde_json::to_vec(&test_entry("id", JournalKind::FinishFailed)).unwrap();
+ let text = String::from_utf8(bytes).unwrap();
+ assert!(!text.contains("checkpoint"));
+ assert!(!text.contains("changed_paths"));
+ assert!(!text.contains("branch"));
+ }
+
+ #[test]
+ fn legacy_relative_session_dir_resolves_against_recorded_cwd() {
+ let mut entry = test_entry("legacy", JournalKind::DegradedRun);
+ entry.cwd = "/original/repo".into();
+ entry.session_dir = Some("stores/native".into());
+ assert_eq!(
+ recovery_session_dir(&entry),
+ Some(PathBuf::from("/original/repo/stores/native"))
+ );
+ }
+
+ #[test]
+ fn journal_entry_serializes_without_optional_noise() {
+ let bytes = serde_json::to_vec(&test_entry("id", JournalKind::DegradedRun)).unwrap();
+ let text = String::from_utf8(bytes).unwrap();
+ assert!(text.contains("\"kind\":\"degraded-run\""));
+ assert!(
+ !text.contains("run_id"),
+ "absent fields are omitted: {text}"
+ );
+ }
+
+ #[test]
+ fn journal_summary_counts_kinds_and_carries_errors() {
+ let temp = tempfile::tempdir().unwrap();
+ append_journal_entry(temp.path(), test_entry("d1", JournalKind::DegradedRun)).unwrap();
+ append_journal_entry(temp.path(), test_entry("f1", JournalKind::FinishFailed)).unwrap();
+ assert_eq!(
+ journal_summary(temp.path()),
+ (2, 1, 1, None),
+ "(total, degraded, finish-failed, error)"
+ );
+ fs::write(journal_path(temp.path()), b"nope").unwrap();
+ let (total, _, _, error) = journal_summary(temp.path());
+ assert_eq!(total, 0);
+ assert!(error.is_some());
+ }
+
+ #[test]
+ fn dropped_spool_events_make_recovery_incomplete() {
+ assert!(spool_is_incomplete(&SpoolOutcome {
+ sent: 3,
+ remaining: 0,
+ dropped: 1,
+ }));
+ assert!(!spool_is_incomplete(&SpoolOutcome {
+ sent: 3,
+ remaining: 0,
+ dropped: 0,
+ }));
+ }
+
+ #[tokio::test]
+ async fn failed_exact_replay_restores_correlated_spool_entry() {
+ let temp = tempfile::tempdir().unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new().route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ ),
+ )
+ .await
+ .unwrap()
+ });
+ let server_url = format!("http://{address}");
+ let spool = hook_spool::spool_dir(temp.path());
+ let created_ms = u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap();
+ let correlated = hook_spool::SpoolEntry {
+ url: format!(
+ "{server_url}/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=proj&session_id=sess-1"
+ ),
+ body: r#"{"session_id":"sess-1","cwd":"/tmp/repo","prompt":"same prompt"}"#.into(),
+ created_ms,
+ auth_mode: hook_spool::AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ hook_spool::enqueue(&spool, &correlated).unwrap();
+ let entry = JournalEntry {
+ server_url: server_url.clone(),
+ interval_started_ms: created_ms.saturating_sub(1),
+ interval_ended_ms: created_ms.saturating_add(1),
+ ..test_entry("restore", JournalKind::DegradedRun)
+ };
+ append_journal_entry(temp.path(), entry).unwrap();
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url,
+ ..Config::default()
+ };
+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert_eq!(report.failures(), 1);
+ assert_eq!(report.journal_remaining, 1);
+ assert_eq!(hook_spool::spool_len(&spool), 1);
+ server.abort();
+ }
+
+ #[tokio::test]
+ async fn spool_only_harness_clears_only_after_a_complete_drain() {
+ let temp = tempfile::tempdir().unwrap();
+ let cwd = temp.path().join("proj");
+ fs::create_dir_all(&cwd).unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new()
+ .route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ )
+ .route(
+ "/hook/batch",
+ axum::routing::post(|| async {
+ axum::Json(serde_json::json!({
+ "accepted": 1,
+ "results": [{"index": 0, "outcome": "stored"}]
+ }))
+ }),
+ ),
+ )
+ .await
+ .unwrap()
+ });
+ let server_url = format!("http://{address}");
+ let spool = hook_spool::spool_dir(temp.path());
+ let created_ms = u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap();
+ hook_spool::enqueue(
+ &spool,
+ &hook_spool::SpoolEntry {
+ url: format!(
+ "{server_url}/hook?event=user-prompt-submit&agent=antigravity-cli&workspace=ws&project=proj"
+ ),
+ body: serde_json::json!({
+ "conversationId": "sess-1",
+ "cwd": cwd,
+ "prompt": "captured"
+ })
+ .to_string(),
+ created_ms,
+ auth_mode: hook_spool::AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ },
+ )
+ .unwrap();
+ let entry = JournalEntry {
+ harness: "antigravity".into(),
+ cwd: cwd.to_string_lossy().into_owned(),
+ project: "proj".into(),
+ server_url: server_url.clone(),
+ interval_started_ms: created_ms.saturating_sub(1),
+ interval_ended_ms: created_ms.saturating_add(1),
+ ..test_entry("spool-only", JournalKind::DegradedRun)
+ };
+ append_journal_entry(temp.path(), entry).unwrap();
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url,
+ ..Config::default()
+ };
+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert_eq!(
+ report.spool.sent, 0,
+ "correlated evidence drains separately"
+ );
+ assert_eq!(report.sessions[0].outcome, "recovered-via-spool");
+ assert_eq!(report.journal_remaining, 0);
+ assert!(load_journal(temp.path()).unwrap().is_empty());
+ server.abort();
+ }
+
+ /// A spool-only entry dispositioned Failed because an *unrelated* queued
+ /// event kept the drain incomplete must not wedge recovery after its own
+ /// interval already completed: restoring the quarantine treats the
+ /// completed marker as durable delivery, the rerun neither errors nor
+ /// re-delivers, and the journal clears.
+ #[tokio::test]
+ async fn completed_spool_only_interval_survives_an_unrelated_incomplete_drain() {
+ use axum::response::IntoResponse as _;
+
+ let temp = tempfile::tempdir().unwrap();
+ let cwd = temp.path().join("proj");
+ fs::create_dir_all(&cwd).unwrap();
+ let deliveries = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
+ let handler_deliveries = deliveries.clone();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new()
+ .route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ )
+ .route(
+ "/hook/batch",
+ axum::routing::post(move |body: axum::body::Bytes| {
+ let deliveries = handler_deliveries.clone();
+ async move {
+ let Ok(items) =
+ serde_json::from_slice::<Vec<serde_json::Value>>(&body)
+ else {
+ return (
+ axum::http::StatusCode::BAD_REQUEST,
+ "malformed batch",
+ )
+ .into_response();
+ };
+ let prompts: Vec<String> = items
+ .iter()
+ .filter_map(|item| {
+ item.get("body")?.get("prompt")?.as_str().map(str::to_owned)
+ })
+ .collect();
+ if prompts.iter().any(|prompt| prompt == "unrelated") {
+ return (
+ axum::http::StatusCode::INTERNAL_SERVER_ERROR,
+ "unrelated event refused",
+ )
+ .into_response();
+ }
+ deliveries.lock().unwrap().extend(prompts);
+ axum::Json(serde_json::json!({
+ "accepted": items.len(),
+ "results": (0..items.len())
+ .map(|index| serde_json::json!({
+ "index": index,
+ "outcome": "stored"
+ }))
+ .collect::<Vec<_>>()
+ }))
+ .into_response()
+ }
+ }),
+ ),
+ )
+ .await
+ .unwrap()
+ });
+ let server_url = format!("http://{address}");
+ let spool = hook_spool::spool_dir(temp.path());
+ let created_ms = u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap();
+ let correlated = hook_spool::SpoolEntry {
+ url: format!(
+ "{server_url}/hook?event=user-prompt-submit&agent=antigravity-cli&workspace=ws&project=proj"
+ ),
+ body: serde_json::json!({
+ "conversationId": "sess-1",
+ "cwd": cwd,
+ "prompt": "captured"
+ })
+ .to_string(),
+ created_ms,
+ auth_mode: hook_spool::AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ let unrelated = hook_spool::SpoolEntry {
+ url: format!(
+ "{server_url}/hook?event=user-prompt-submit&agent=claude-code&workspace=ws&project=proj&session_id=unrelated-1"
+ ),
+ body: r#"{"session_id":"unrelated-1","cwd":"/tmp/repo","prompt":"unrelated"}"#.into(),
+ created_ms: created_ms.saturating_add(1),
+ auth_mode: hook_spool::AuthMode::Anonymous,
+ token: None,
+ attempts: 0,
+ profile: None,
+ };
+ hook_spool::enqueue(&spool, &correlated).unwrap();
+ hook_spool::enqueue(&spool, &unrelated).unwrap();
+ let entry = JournalEntry {
+ harness: "antigravity".into(),
+ spool_only: true,
+ native_session_id: Some("sess-1".into()),
+ cwd: cwd.to_string_lossy().into_owned(),
+ workspace: "ws".into(),
+ project: "proj".into(),
+ server_url: server_url.clone(),
+ interval_started_ms: created_ms.saturating_sub(1),
+ interval_ended_ms: created_ms.saturating_add(2),
+ ..test_entry("completed-rerun", JournalKind::DegradedRun)
+ };
+ append_journal_entry(temp.path(), entry).unwrap();
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url,
+ ..Config::default()
+ };
+ let first = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert_eq!(first.sessions[0].outcome, "recovered-via-spool");
+ assert_eq!(first.journal_remaining, 0);
+ assert!(first.spool_incomplete, "the unrelated event still queues");
+ assert!(load_journal(temp.path()).unwrap().is_empty());
+
+ let rerun = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert!(rerun.sessions.is_empty());
+ assert_eq!(rerun.failures(), 0);
+ assert_eq!(rerun.journal_remaining, 0);
+ assert!(rerun.spool_incomplete);
+ assert!(
+ load_journal(temp.path()).unwrap().is_empty(),
+ "the rerun must clear the journal"
+ );
+ assert_eq!(
+ deliveries
+ .lock()
+ .unwrap()
+ .iter()
+ .filter(|prompt| *prompt == "captured")
+ .count(),
+ 1,
+ "the completed interval's evidence is never re-delivered"
+ );
+ assert_eq!(
+ hook_spool::spool_len(&spool),
+ 1,
+ "only the unrelated event stays queued"
+ );
+ server.abort();
+ }
+
+ #[tokio::test]
+ async fn spool_only_without_correlated_evidence_never_auto_clears() {
+ let temp = tempfile::tempdir().unwrap();
+ let mut entry = test_entry("no-hooks", JournalKind::DegradedRun);
+ entry.harness = "antigravity".into();
+ append_journal_entry(temp.path(), entry).unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new().route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ ),
+ )
+ .await
+ .unwrap();
+ });
+ let server_url = format!("http://{address}");
+ mutate_journal(temp.path(), |entries| {
+ entries[0].server_url.clone_from(&server_url);
+ Ok(())
+ })
+ .unwrap();
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url,
+ ..Config::default()
+ };
+ for _ in 0..2 {
+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert_eq!(report.failures(), 1);
+ assert_eq!(report.journal_remaining, 1);
+ assert!(
+ report.sessions[0]
+ .detail
+ .as_deref()
+ .unwrap()
+ .contains("no durably delivered")
+ );
+ }
+ server.abort();
+ }
+
+ #[tokio::test]
+ async fn a_dropped_spool_event_stays_lost_on_the_next_empty_pass() {
+ let temp = tempfile::tempdir().unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new().route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ ),
+ )
+ .await
+ .unwrap();
+ });
+ let server_url = format!("http://{address}");
+ let spool = hook_spool::spool_dir(temp.path());
+ let created_ms = u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap();
+ hook_spool::enqueue(
+ &spool,
+ &hook_spool::SpoolEntry {
+ url: format!(
+ "{server_url}/hook?event=user-prompt-submit&agent=antigravity-cli&workspace=ws&project=proj"
+ ),
+ body: r#"{"conversationId":"sess-1","cwd":"/tmp/repo","prompt":"lost"}"#.into(),
+ created_ms,
+ auth_mode: hook_spool::AuthMode::Anonymous,
+ token: None,
+ attempts: 7,
+ profile: None,
+ },
+ )
+ .unwrap();
+ let entry = JournalEntry {
+ harness: "antigravity".into(),
+ server_url: server_url.clone(),
+ interval_started_ms: created_ms.saturating_sub(1),
+ interval_ended_ms: created_ms.saturating_add(1),
+ ..test_entry("dropped", JournalKind::DegradedRun)
+ };
+ append_journal_entry(temp.path(), entry).unwrap();
+ mutate_journal(temp.path(), |entries| {
+ entries[0].correlated_events_seen = true;
+ entries[0].correlated_loss = true;
+ Ok(())
+ })
+ .unwrap();
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url,
+ ..Config::default()
+ };
+ let first = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert_eq!(
+ first.spool.sent, 0,
+ "correlated evidence never rides the main drain"
+ );
+ assert_eq!(
+ first.failures(),
+ 1,
+ "the correlated loss keeps the journal entry"
+ );
+ assert!(load_journal(temp.path()).unwrap()[0].correlated_loss);
+ let second = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert_eq!(second.spool.dropped, 0, "the dropped evidence stays lost");
+ assert_eq!(second.journal_remaining, 1);
+ assert!(load_journal(temp.path()).unwrap()[0].correlated_loss);
+ server.abort();
+ }
+
+ #[tokio::test]
+ async fn a_lossy_spool_pass_keeps_the_recovery_journal() {
+ let temp = tempfile::tempdir().unwrap();
+ let mut entry = test_entry("keep", JournalKind::DegradedRun);
+ entry.harness = "antigravity".into();
+ append_journal_entry(temp.path(), entry).unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new().route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ ),
+ )
+ .await
+ .unwrap();
+ });
+ let spool = hook_spool::spool_dir(temp.path());
+ hook_spool::enqueue(
+ &spool,
+ &hook_spool::SpoolEntry {
+ url: format!("http://{address}/hook?event=user-prompt-submit"),
+ body: "not-json".into(),
+ created_ms: u64::try_from(jiff::Timestamp::now().as_millisecond()).unwrap(),
+ auth_mode: hook_spool::AuthMode::Anonymous,
+ token: None,
+ attempts: 7,
+ profile: None,
+ },
+ )
+ .unwrap();
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url: format!("http://{address}"),
+ ..Config::default()
+ };
+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert_eq!(report.spool.dropped, 1);
+ assert!(report.spool_incomplete);
+ assert_eq!(report.journal_remaining, 1);
+ assert!(
+ report.sessions[0]
+ .detail
+ .as_deref()
+ .is_some_and(|detail| detail.contains("1 dropped"))
+ );
+ server.abort();
+ }
+
+ #[tokio::test]
+ async fn per_entry_budget_times_out_and_keeps_the_entry() {
+ let temp = tempfile::tempdir().unwrap();
+ append_journal_entry(temp.path(), test_entry("one", JournalKind::DegradedRun)).unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new().route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ ),
+ )
+ .await
+ .unwrap();
+ });
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url: format!("http://{address}"),
+ ..Config::default()
+ };
+ let report = recover(
+ &config,
+ &RecoverBudgets {
+ entry_timeout: Duration::ZERO,
+ ..RecoverBudgets::default()
+ },
+ )
+ .await
+ .unwrap();
+ assert_eq!(report.failures(), 1);
+ assert!(
+ report.sessions[0]
+ .detail
+ .as_deref()
+ .is_some_and(|detail| detail.contains("timed out"))
+ );
+ assert_eq!(load_journal(temp.path()).unwrap().len(), 1);
+ server.abort();
+ }
+
+ #[tokio::test]
+ async fn total_journal_budget_keeps_unattempted_entries() {
+ let temp = tempfile::tempdir().unwrap();
+ append_journal_entry(temp.path(), test_entry("one", JournalKind::DegradedRun)).unwrap();
+ append_journal_entry(temp.path(), test_entry("two", JournalKind::DegradedRun)).unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new().route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ ),
+ )
+ .await
+ .unwrap();
+ });
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url: format!("http://{address}"),
+ ..Config::default()
+ };
+ let report = recover(
+ &config,
+ &RecoverBudgets {
+ journal_total: Duration::ZERO,
+ ..RecoverBudgets::default()
+ },
+ )
+ .await
+ .unwrap();
+ assert_eq!(report.failures(), 2);
+ assert!(report.sessions.iter().all(|session| {
+ session
+ .detail
+ .as_deref()
+ .is_some_and(|detail| detail.contains("budget exhausted"))
+ }));
+ assert_eq!(load_journal(temp.path()).unwrap().len(), 2);
+ server.abort();
+ }
+
+ #[tokio::test]
+ async fn a_journal_entry_cannot_be_replayed_to_another_server() {
+ let temp = tempfile::tempdir().unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new().route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ ),
+ )
+ .await
+ .unwrap();
+ });
+ append_journal_entry(
+ temp.path(),
+ JournalEntry {
+ server_url: "http://127.0.0.1:1".to_string(),
+ ..test_entry("foreign-server", JournalKind::DegradedRun)
+ },
+ )
+ .unwrap();
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url: format!("http://{address}"),
+ ..Config::default()
+ };
+ let report = recover(&config, &RecoverBudgets::default()).await.unwrap();
+ assert_eq!(report.failures(), 1);
+ assert!(
+ report.sessions[0]
+ .detail
+ .as_deref()
+ .unwrap()
+ .contains("refusing to replay")
+ );
+ assert_eq!(load_journal(temp.path()).unwrap().len(), 1);
+ server.abort();
+ }
+
+ #[tokio::test]
+ async fn failed_recovery_returns_a_testable_nonzero_code_without_exiting() {
+ let temp = tempfile::tempdir().unwrap();
+ append_journal_entry(
+ temp.path(),
+ JournalEntry {
+ native_session_id: None,
+ ..test_entry("bad", JournalKind::FinishFailed)
+ },
+ )
+ .unwrap();
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = tokio::spawn(async move {
+ axum::serve(
+ listener,
+ axum::Router::new().route(
+ "/healthz",
+ axum::routing::get(|| async { axum::http::StatusCode::OK }),
+ ),
+ )
+ .await
+ .unwrap();
+ });
+ let config = Config {
+ data_dir: temp.path().to_path_buf(),
+ server_url: format!("http://{address}"),
+ ..Config::default()
+ };
+ let code = run(&config, RecoverArgs { json: false }).await.unwrap();
+ assert_eq!(code, 1);
+ assert_eq!(load_journal(temp.path()).unwrap().len(), 1);
+ server.abort();
+ }
+}
diff --git a/crates/ai-memory-cli/src/commands/run.rs b/crates/ai-memory-cli/src/commands/run.rs
index e7ce3f01..d443ef58 100644
--- a/crates/ai-memory-cli/src/commands/run.rs
+++ b/crates/ai-memory-cli/src/commands/run.rs
@@ -25,7 +25,8 @@ use ai_memory_workstream::{
kiro_selects_v3_engine, kiro_v3_resume_uses_default_store, list_native_sessions,
marked_choices, native_session_exists, native_session_in_checkout, native_store_root,
omp_profile_flag, omp_profile_flag_env, parse_jail_toggles, store_override_vars,
- usable_ai_jail_here, wait_for_transcript_flush,
+ transcript_baseline, transcript_interval_digests, usable_ai_jail_here,
+ wait_for_transcript_flush,
};
use anyhow::{Context as _, Result, anyhow};
use clap::ValueEnum as _;
@@ -629,6 +630,8 @@ async fn run_once_with_wiring(
harness: provisional_harness,
executable: executable.clone(),
native_args: &native_args,
+ workspace: &workspace,
+ project: &project,
force_fresh,
yolo_modes,
jail_plan: &jail_plan,
@@ -1217,25 +1220,43 @@ async fn run_once_with_wiring(
)
.await
);
- let transcript = if plan.mode == LaunchMode::Session {
- let source_cursor = if native_session_id.as_deref() == prepared.native_session_id.as_deref()
- {
- prepared.source_cursor.as_deref()
+ let recovery_start_cursor =
+ if native_session_id.as_deref() == prepared.native_session_id.as_deref() {
+ prepared.source_cursor.clone()
} else {
None
};
- export_after_flush(
+ let (transcript, exact_export_available) = if plan.mode == LaunchMode::Session
+ && harness.transcript_capability() == ai_memory_workstream::TranscriptCapability::Export
+ {
+ let exported = export_after_flush(
harness,
&home,
&repository.cwd,
plan.session_dir.as_deref(),
native_session_id.as_deref(),
- source_cursor,
+ recovery_start_cursor.as_deref(),
)
- .await
+ .await;
+ let exact = exported.losses.iter().all(|loss| {
+ !loss.starts_with("native session id could not be discovered")
+ && !loss.starts_with("native transcript import failed")
+ }) && exported.source_cursor.is_some();
+ (exported, exact)
} else {
- ExportedTranscript::default()
+ (
+ ExportedTranscript {
+ native_session_id: native_session_id.clone().unwrap_or_default(),
+ ..ExportedTranscript::default()
+ },
+ false,
+ )
};
+ let recovery_event_digests =
+ exact_export_available.then(|| transcript_interval_digests(&transcript.events));
+ let recovery_cursor = exact_export_available
+ .then(|| transcript.source_cursor.clone())
+ .flatten();
let checkpoint = inspect_repository(&repository.cwd)
.map(|identity| identity.checkpoint)
.unwrap_or(repository.checkpoint);
@@ -1266,6 +1287,23 @@ async fn run_once_with_wiring(
&error,
)
);
+ journal_finish_failed_run(
+ config,
+ &endpoint,
+ harness,
+ native_session_id.as_deref(),
+ &plan,
+ &workspace,
+ &project,
+ &repository.cwd,
+ started_at,
+ exit_code,
+ prepared.run_id,
+ recovery_event_digests.as_deref(),
+ recovery_start_cursor.as_deref(),
+ recovery_cursor.as_deref(),
+ &privacy,
+ );
interrupt_task.abort();
return Ok(RunOutcome {
exit_code,
@@ -1348,6 +1386,8 @@ struct DegradedLaunch<'a> {
harness: ManagedHarness,
executable: Option<OsString>,
native_args: &'a [OsString],
+ workspace: &'a str,
+ project: &'a str,
force_fresh: bool,
yolo_modes: YoloModes,
jail_plan: &'a JailPlan,
@@ -1470,6 +1510,41 @@ async fn degraded_launch(
for name in blank_home_overrides(launch.harness, &launch_env) {
command.env_remove(name);
}
+ let prelaunch_sessions = snapshot_native_sessions(
+ launch.harness,
+ launch.home,
+ launch.cwd,
+ plan.session_dir.as_deref(),
+ )
+ .await;
+ let prelaunch_cursor = if let Some(session) = plan.expected_session_id.as_deref()
+ && launch.harness.transcript_capability()
+ == ai_memory_workstream::TranscriptCapability::Export
+ {
+ match native_session_exists(
+ launch.harness,
+ launch.home,
+ launch.cwd,
+ plan.session_dir.as_deref(),
+ session,
+ ) {
+ Ok(true) => Some(
+ transcript_baseline(
+ launch.harness,
+ launch.home,
+ launch.cwd,
+ plan.session_dir.as_deref(),
+ session,
+ )
+ .await,
+ ),
+ Ok(false) => None,
+ Err(error) => Some(Err(error)),
+ }
+ } else {
+ None
+ };
+ let started_at = SystemTime::now();
let child = command.spawn().map_err(|spawn_error| {
let spawn_message = spawn_error.to_string();
anyhow!(spawn_message).context(format!(
@@ -1486,6 +1561,16 @@ async fn degraded_launch(
"ai-memory: this run was not recorded on the server; {}",
spooled_events_note(&spool)
);
+ journal_degraded_run(
+ config,
+ launch,
+ &plan,
+ exit_code,
+ started_at,
+ prelaunch_sessions,
+ prelaunch_cursor,
+ )
+ .await;
Ok(RunOutcome {
exit_code,
harness: launch.harness,
@@ -1496,6 +1581,255 @@ async fn degraded_launch(
})
}
+/// Journal a degraded session launch so `ai-memory recover` can close it out
+/// once the server returns. Best-effort by contract: the child has already
+/// exited and its exit code is preserved above everything else, so a journal
+/// problem may only warn. Passthrough launches are not journaled — they hold
+/// no session and no transcript, so there is nothing to recover.
+async fn journal_degraded_run(
+ config: &Config,
+ launch: &DegradedLaunch<'_>,
+ plan: &LaunchPlan,
+ exit_code: i32,
+ started_at: SystemTime,
+ prelaunch_sessions: Option<std::collections::HashSet<String>>,
+ prelaunch_cursor: Option<Result<String>>,
+) {
+ if plan.mode != LaunchMode::Session {
+ return;
+ }
+ let native_session_id = match Sanitizer::new(&config.sanitize) {
+ Ok(privacy) => resolve_degraded_session(
+ plan,
+ launch.harness,
+ launch.home,
+ launch.cwd,
+ started_at,
+ prelaunch_sessions.as_ref(),
+ &privacy,
+ )
+ .await
+ .unwrap_or_else(|error| {
+ eprintln!(
+ "ai-memory: could not uniquely identify the session this run used ({error:#}); \
+ nothing was journaled; preserve the local transcript and run `ai-memory \
+ backfill --force --session <id>` after identifying it"
+ );
+ None
+ }),
+ Err(error) => {
+ eprintln!(
+ "ai-memory: could not record this run for recovery (privacy config: {error}); \
+ nothing was journaled"
+ );
+ return;
+ }
+ };
+ let Some(native_session_id) = native_session_id else {
+ eprintln!(
+ "ai-memory: no native session was discovered, so this run could not be journaled; \
+ retain the harness transcript for manual backfill"
+ );
+ return;
+ };
+ let source_cursor = match prelaunch_cursor {
+ Some(Ok(cursor)) => Some(cursor),
+ Some(Err(error)) => {
+ eprintln!(
+ "ai-memory: could not record a safe delta baseline for resumed session {} ({error:#}); nothing was journaled; retain the native transcript for manual repair",
+ display_session_id(&native_session_id)
+ );
+ return;
+ }
+ None => None,
+ };
+ let spool_only = launch.harness.transcript_capability()
+ == ai_memory_workstream::TranscriptCapability::SpoolOnly;
+ let (event_digests, final_cursor) = if spool_only {
+ (None, None)
+ } else {
+ let exact_export = match export_exact_after_run(
+ launch.harness,
+ launch.home,
+ launch.cwd,
+ plan.session_dir.as_deref(),
+ &native_session_id,
+ source_cursor.as_deref(),
+ )
+ .await
+ {
+ Ok(export) => export,
+ Err(error) => {
+ eprintln!(
+ "ai-memory: could not capture an exact post-run transcript bound for session {} ({error:#}); nothing was journaled; retain the native transcript for manual repair",
+ display_session_id(&native_session_id)
+ );
+ return;
+ }
+ };
+ let Some(final_cursor) = exact_export.source_cursor.clone() else {
+ eprintln!(
+ "ai-memory: the {} transcript adapter produced no post-run cursor; nothing was journaled; retain the native transcript for manual repair",
+ launch.harness.as_str()
+ );
+ return;
+ };
+ (
+ Some(transcript_interval_digests(&exact_export.events)),
+ Some(final_cursor),
+ )
+ };
+ let entry = super::recovery::JournalEntry {
+ id: uuid::Uuid::new_v4().to_string(),
+ recorded_at: jiff::Timestamp::now().to_string(),
+ interval_started_ms: system_time_ms(started_at),
+ interval_ended_ms: system_time_ms(SystemTime::now()),
+ kind: super::recovery::JournalKind::DegradedRun,
+ harness: launch.harness.as_str().to_string(),
+ native_session_id: Some(native_session_id),
+ session_dir: absolute_session_dir(plan.session_dir.as_deref(), launch.cwd)
+ .map(|path| path.to_string_lossy().into_owned()),
+ cwd: launch.cwd.to_string_lossy().into_owned(),
+ workspace: launch.workspace.to_string(),
+ project: launch.project.to_string(),
+ server_url: launch.endpoint.identity(),
+ run_id: None,
+ event_digests,
+ spool_only,
+ source_cursor,
+ final_cursor,
+ correlated_events_seen: false,
+ correlated_delivery_confirmed: false,
+ correlated_loss: false,
+ exit_code: Some(exit_code),
+ failures: Vec::new(),
+ };
+ match super::recovery::append_journal_entry(&config.data_dir, entry) {
+ Ok(()) => {
+ eprintln!(
+ "ai-memory: recorded for recovery — run `ai-memory recover` once the server \
+ is back to replay this session's exact native transcript"
+ );
+ }
+ Err(error) => {
+ eprintln!(
+ "ai-memory: could not record this run for recovery ({error:#}); the native \
+ transcript remains in the harness store and can be imported with `ai-memory \
+ backfill --force --session <id>`"
+ );
+ }
+ }
+}
+
+const DEGRADED_SESSION_SNAPSHOT_LIMIT: usize = 2_000;
+
+async fn snapshot_native_sessions(
+ harness: ManagedHarness,
+ home: &Path,
+ cwd: &Path,
+ session_dir: Option<&Path>,
+) -> Option<std::collections::HashSet<String>> {
+ let sessions = list_native_sessions(
+ harness,
+ home,
+ cwd,
+ session_dir,
+ DEGRADED_SESSION_SNAPSHOT_LIMIT + 1,
+ )
+ .await
+ .ok()?;
+ if sessions.len() > DEGRADED_SESSION_SNAPSHOT_LIMIT {
+ return None;
+ }
+ Some(
+ sessions
+ .into_iter()
+ .map(|session| session.native_session_id)
+ .collect(),
+ )
+}
+
+#[allow(clippy::too_many_arguments)]
+async fn resolve_degraded_session(
+ plan: &LaunchPlan,
+ harness: ManagedHarness,
+ home: &Path,
+ cwd: &Path,
+ started_at: SystemTime,
+ before: Option<&std::collections::HashSet<String>>,
+ sanitizer: &Sanitizer,
+) -> Result<Option<String>> {
+ if let Some(expected) = plan.expected_session_id.as_deref() {
+ return NativeSessionIdentity::parse(expected, sanitizer)
+ .map(NativeSessionIdentity::into_string)
+ .map(Some)
+ .map_err(anyhow::Error::from);
+ }
+ let Some(before) = before else {
+ return Err(anyhow!(
+ "pre-launch native-session snapshot was unavailable"
+ ));
+ };
+ let after = list_native_sessions(
+ harness,
+ home,
+ cwd,
+ plan.session_dir.as_deref(),
+ DEGRADED_SESSION_SNAPSHOT_LIMIT + 1,
+ )
+ .await?;
+ if after.len() > DEGRADED_SESSION_SNAPSHOT_LIMIT {
+ return Err(anyhow!(
+ "post-launch native-session snapshot exceeded its {}-record limit",
+ DEGRADED_SESSION_SNAPSHOT_LIMIT
+ ));
+ }
+ let native_session_id = unique_new_session(after, before, started_at)?;
+ NativeSessionIdentity::parse(&native_session_id, sanitizer)
+ .map(NativeSessionIdentity::into_string)
+ .map(Some)
+ .map_err(anyhow::Error::from)
+}
+
+fn unique_new_session(
+ after: Vec<NativeSessionCandidate>,
+ before: &std::collections::HashSet<String>,
+ started_at: SystemTime,
+) -> Result<String> {
+ let mut new = after
+ .into_iter()
+ .filter(|candidate| {
+ candidate.updated_at + Duration::from_secs(2) >= started_at
+ && !before.contains(&candidate.native_session_id)
+ })
+ .collect::<Vec<_>>();
+ if new.len() != 1 {
+ return Err(anyhow!(
+ "expected one new native session after launch, found {}",
+ new.len()
+ ));
+ }
+ Ok(new.remove(0).native_session_id)
+}
+
+fn system_time_ms(time: SystemTime) -> u64 {
+ time.duration_since(std::time::UNIX_EPOCH)
+ .unwrap_or_default()
+ .as_millis()
+ .try_into()
+ .unwrap_or(u64::MAX)
+}
+
+fn absolute_session_dir(session_dir: Option<&Path>, cwd: &Path) -> Option<PathBuf> {
+ session_dir.map(|path| {
+ if path.is_absolute() {
+ clean_path(path)
+ } else {
+ clean_path(&cwd.join(path))
+ }
+ })
+}
+
/// The one-line account of what remains in the local hook spool after a
/// degraded run, mirroring `ai-memory status`'s offline-spool phrasing.
fn spooled_events_note(spool: &super::hook_spool::SpoolHealth) -> String {
@@ -1526,9 +1860,10 @@ fn offline_launch_warning(
format!(
"ai-memory: WARNING: the ai-memory server at {} is unreachable — launching {} \
WITHOUT server support.\n {diagnosis:#}\n Lost this run: no workstream lease or \
- cross-harness context, no transcript import, no handoff delivery.\n Still works: \
- lifecycle hooks spool events locally and drain when the server returns; an existing \
- MCP registration degrades to no-recall rather than blocking the session.\n Spooled \
+ cross-harness context, no immediate transcript import, no handoff delivery.\n Still \
+ works: lifecycle hooks spool events locally, and the native transcript is journaled \
+ after session discovery. Run `ai-memory recover` when the server returns to drain the \
+ spool and replay the exact transcript through the sanitized backfill path.\n Spooled \
events are dropped after {} failed drain passes, 7 days, or the 10,000-file spool \
cap (set AI_MEMORY_HOOK_SPOOL_MAX_ATTEMPTS=0 for a planned outage). Pass \
--require-server to fail instead of degrading.",
@@ -1548,23 +1883,86 @@ fn offline_import_warning(
project: &str,
error: &anyhow::Error,
) -> String {
- let repair = match native_session_id {
- Some(session) => format!(
- "ai-memory finalize-session --agent {} --reopen --session-id {} --workspace {} --project {}",
- harness.agent_kind().as_str(),
- SessionId::from_native(session),
- super::render_shared::shell_quote(workspace),
- super::render_shared::shell_quote(project),
- ),
- None => "ai-memory finalize-session (see --help)".to_string(),
- };
+ let session = native_session_id
+ .map(SessionId::from_native)
+ .map_or_else(|| "unknown".to_string(), |session| session.to_string());
format!(
- "ai-memory: warning: {error:#}; the transcript was not imported and the orphaned lease \
- will expire automatically within 90 seconds. The harness exit code is preserved. \
- Repair it once the server is back with `{repair}`."
+ "ai-memory: warning: {error:#}; the transcript for {} session {session} in {}/{} was \
+ not imported and the orphaned lease will expire automatically within 90 seconds. The \
+ harness exit code is preserved. Run `ai-memory recover` once the server is back.",
+ harness.as_str(),
+ workspace,
+ project,
)
}
+/// Journal a run whose transcript import failed on an unreachable server, so
+/// `ai-memory recover` can re-export and re-import it once the server
+/// returns. The journal keeps only locators and recomputes the repository
+/// checkpoint during recovery — never transcript content. Best-effort: the
+/// child's exit code is already preserved, so a journal problem may only warn.
+#[allow(clippy::too_many_arguments)]
+fn journal_finish_failed_run(
+ config: &Config,
+ endpoint: &ServerEndpoint,
+ harness: ManagedHarness,
+ native_session_id: Option<&str>,
+ plan: &LaunchPlan,
+ workspace: &str,
+ project: &str,
+ cwd: &Path,
+ started_at: SystemTime,
+ exit_code: i32,
+ run_id: ai_memory_core::ManagedRunId,
+ event_digests: Option<&[String]>,
+ source_cursor: Option<&str>,
+ final_cursor: Option<&str>,
+ sanitizer: &Sanitizer,
+) {
+ let entry = super::recovery::JournalEntry {
+ id: uuid::Uuid::new_v4().to_string(),
+ recorded_at: jiff::Timestamp::now().to_string(),
+ interval_started_ms: system_time_ms(started_at),
+ interval_ended_ms: system_time_ms(SystemTime::now()),
+ kind: super::recovery::JournalKind::FinishFailed,
+ harness: harness.as_str().to_string(),
+ native_session_id: native_session_id
+ .and_then(|id| NativeSessionIdentity::parse(id, sanitizer).ok())
+ .map(NativeSessionIdentity::into_string),
+ session_dir: absolute_session_dir(plan.session_dir.as_deref(), cwd)
+ .map(|path| path.to_string_lossy().into_owned()),
+ cwd: cwd.to_string_lossy().into_owned(),
+ workspace: workspace.to_string(),
+ project: project.to_string(),
+ server_url: endpoint.identity(),
+ run_id: Some(run_id.to_string()),
+ event_digests: event_digests.map(<[String]>::to_vec),
+ spool_only: harness.transcript_capability()
+ == ai_memory_workstream::TranscriptCapability::SpoolOnly,
+ source_cursor: source_cursor.map(str::to_owned),
+ final_cursor: final_cursor.map(str::to_owned),
+ correlated_events_seen: false,
+ correlated_delivery_confirmed: false,
+ correlated_loss: false,
+ exit_code: Some(exit_code),
+ failures: Vec::new(),
+ };
+ match super::recovery::append_journal_entry(&config.data_dir, entry) {
+ Ok(()) => {
+ eprintln!(
+ "ai-memory: recorded for recovery — run `ai-memory recover` once the server \
+ is back to re-import this transcript"
+ );
+ }
+ Err(error) => {
+ eprintln!(
+ "ai-memory: could not record this run for recovery ({error:#}); retain the \
+ native transcript and original run id for manual repair"
+ );
+ }
+ }
+}
+
/// Whether a managed-run request failed because the server could not be
/// reached at all (connect error or timeout) rather than answering with a
/// protocol-level failure. Only the former may be downgraded to a warning
@@ -3278,6 +3676,26 @@ fn session_age(updated_at: SystemTime, now: SystemTime) -> String {
super::humanize_age_secs(i64::try_from(secs).unwrap_or(i64::MAX))
}
+async fn export_exact_after_run(
+ harness: ManagedHarness,
+ home: &std::path::Path,
+ cwd: &std::path::Path,
+ session_dir: Option<&std::path::Path>,
+ native_session_id: &str,
+ source_cursor: Option<&str>,
+) -> Result<ExportedTranscript> {
+ wait_for_transcript_flush(harness, home, cwd, session_dir, native_session_id).await?;
+ export_transcript(
+ harness,
+ home,
+ cwd,
+ session_dir,
+ native_session_id,
+ source_cursor,
+ )
+ .await
+}
+
async fn export_after_flush(
harness: ManagedHarness,
home: &std::path::Path,
@@ -3319,7 +3737,7 @@ async fn export_after_flush(
}
}
-async fn import_batches(
+pub(crate) async fn import_batches(
endpoint: &ServerEndpoint,
run_path: &str,
transcript: ExportedTranscript,
@@ -3730,6 +4148,29 @@ const fn managed_harness_from_agent(agent: AgentKind) -> Option<ManagedHarness>
}
}
+/// Parse the exact harness names the recovery journal writes
+/// (`ManagedHarness::as_str`). The user-facing `from_name` table deliberately
+/// has no "kiro-v3" spelling (no one types that), but the journal must round
+/// trip its own exact value, engine flavor included.
+pub(crate) fn harness_from_journal_name(name: &str) -> Option<ManagedHarness> {
+ match name {
+ "claude" => Some(ManagedHarness::Claude),
+ "codex" => Some(ManagedHarness::Codex),
+ "opencode" => Some(ManagedHarness::OpenCode),
+ "opencode2" => Some(ManagedHarness::OpenCode2),
+ "pi" => Some(ManagedHarness::Pi),
+ "crush" => Some(ManagedHarness::Crush),
+ "omp" => Some(ManagedHarness::Omp),
+ "kimi" => Some(ManagedHarness::Kimi),
+ "command-code" => Some(ManagedHarness::CommandCode),
+ "kiro" => Some(ManagedHarness::Kiro),
+ "kiro-v3" => Some(ManagedHarness::KiroV3),
+ "grok" => Some(ManagedHarness::Grok),
+ "antigravity" => Some(ManagedHarness::Antigravity),
+ _ => None,
+ }
+}
+
#[cfg(test)]
mod tests {
use std::ffi::{OsStr, OsString};
@@ -4612,6 +5053,43 @@ mod tests {
]
}
+ #[test]
+ fn degraded_session_correlation_requires_one_new_session() {
+ let started = SystemTime::UNIX_EPOCH + Duration::from_secs(100);
+ let before = std::collections::HashSet::from(["old".to_string()]);
+ let candidate = |id: &str| NativeSessionCandidate {
+ native_session_id: id.into(),
+ updated_at: started + Duration::from_secs(1),
+ };
+ assert_eq!(
+ unique_new_session(vec![candidate("old"), candidate("new")], &before, started).unwrap(),
+ "new"
+ );
+ assert!(
+ unique_new_session(
+ vec![candidate("first"), candidate("second")],
+ &before,
+ started
+ )
+ .unwrap_err()
+ .to_string()
+ .contains("found 2")
+ );
+ assert!(unique_new_session(vec![candidate("old")], &before, started).is_err());
+ }
+
+ #[test]
+ fn relative_session_dir_is_persisted_against_launch_cwd() {
+ assert_eq!(
+ absolute_session_dir(Some(Path::new("stores/native")), Path::new("/repo")),
+ Some(PathBuf::from("/repo/stores/native"))
+ );
+ assert_eq!(
+ absolute_session_dir(Some(Path::new("/stores/native")), Path::new("/repo")),
+ Some(PathBuf::from("/stores/native"))
+ );
+ }
+
#[test]
fn native_grok_rules_flags_suppress_context_injection() {
for args in [
@@ -6128,8 +6606,7 @@ mod tests {
}
/// A fake Claude executable: captures its environment and argv, writes
- /// the (empty-session) transcript the launcher waits for, and exits with
- /// `code`.
+ /// the transcript the launcher waits for, and exits with `code`.
#[cfg(unix)]
fn capture_env_claude(repo: &Path, home: &Path, code: i32) -> PathBuf {
use std::os::unix::fs::PermissionsExt as _;
@@ -6403,8 +6880,8 @@ mod tests {
}
/// The degraded-launch warning names everything the operator needs: the
- /// server URL, what is lost, what still works (spool + no-recall MCP),
- /// the bounded retention, and the opt-outs.
+ /// server URL, what is unavailable, what remains recoverable, the bounded
+ /// retention, and the opt-outs.
#[tokio::test]
async fn offline_launch_warning_names_the_tradeoffs() {
let url = closed_port_url().await;
@@ -6423,7 +6900,8 @@ mod tests {
assert!(warning.contains("Lost this run"), "{warning}");
assert!(warning.contains("Still works"), "{warning}");
assert!(warning.contains("spool"), "{warning}");
- assert!(warning.contains("no-recall"), "{warning}");
+ assert!(warning.contains("ai-memory recover"), "{warning}");
+ assert!(warning.contains("sanitized backfill"), "{warning}");
assert!(warning.contains("8 failed drain passes"), "{warning}");
assert!(
warning.contains("AI_MEMORY_HOOK_SPOOL_MAX_ATTEMPTS=0"),
@@ -6478,7 +6956,7 @@ mod tests {
}
#[test]
- fn offline_import_warning_points_at_finalize_session() {
+ fn offline_import_warning_points_at_recovery() {
let warning = offline_import_warning(
ManagedHarness::Claude,
Some("session-123"),
@@ -6487,15 +6965,10 @@ mod tests {
&anyhow!("connection refused"),
);
assert!(warning.contains("exit code is preserved"), "{warning}");
+ assert!(warning.contains("ai-memory recover"), "{warning}");
+ assert!(warning.contains("in ws/proj"), "{warning}");
assert!(
- warning.contains(&format!(
- "ai-memory finalize-session --agent claude-code --reopen --session-id {}",
- SessionId::from_native("session-123")
- )),
- "{warning}"
- );
- assert!(
- warning.contains("--workspace ws --project proj"),
+ warning.contains(&SessionId::from_native("session-123").to_string()),
"{warning}"
);
@@ -6506,10 +6979,8 @@ mod tests {
"proj",
&anyhow!("timed out"),
);
- assert!(
- generic.contains("finalize-session (see --help)"),
- "{generic}"
- );
+ assert!(generic.contains("session unknown"), "{generic}");
+ assert!(generic.contains("ai-memory recover"), "{generic}");
}
/// A server that dies between prepare and finish: the child's exit code
diff --git a/crates/ai-memory-cli/src/commands/status.rs b/crates/ai-memory-cli/src/commands/status.rs
index aa4c6c8c..c435d9c5 100644
--- a/crates/ai-memory-cli/src/commands/status.rs
+++ b/crates/ai-memory-cli/src/commands/status.rs
@@ -188,6 +188,29 @@ fn report_offline_spool(spool: &SpoolHealth, json: bool) {
}
}
+/// One line pointing at `ai-memory recover` while the server is down, when
+/// there is actually something waiting: queued hook events or journaled
+/// outage runs. Goes to stderr like the spool block above, so `--json`
+/// consumers' stdout stays a single object.
+fn report_offline_recovery_hint(config: &Config, spool: &SpoolHealth, json: bool) {
+ let (journal_pending, _, _, journal_error) = super::recovery::journal_summary(&config.data_dir);
+ if spool.pending == 0 && journal_pending == 0 && journal_error.is_none() {
+ return;
+ }
+ if json {
+ eprintln!(
+ "recovery hint (server unreachable): {{\"journal_pending\": {journal_pending},\
+ \"spool_pending\": {}}}",
+ spool.pending
+ );
+ return;
+ }
+ eprintln!(
+ " run `ai-memory recover` once the server is back to deliver queued events and \
+ repair unrecorded runs ({journal_pending} journaled)"
+ );
+}
+
/// Which capture mode the hook would enforce for this install (#446).
///
/// Reads the same file the hook reads rather than keeping a second source of
@@ -221,32 +244,26 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {
Ok(report) => report,
Err(err) => {
report_offline_spool(&spool, args.json);
+ report_offline_recovery_hint(config, &spool, args.json);
return Err(err);
}
};
+ let (journal_pending, journal_degraded, journal_finish_failed, journal_error) =
+ super::recovery::journal_summary(&config.data_dir);
if args.json {
println!(
"{}",
- serde_json::to_string_pretty(&serde_json::json!({
- "version": report.version,
- "data_dir": report.data_dir,
- "bind": report.bind,
- "db_path": report.db_path,
- "counts": {
- "pages_latest": report.counts.pages_latest,
- "pages_all": report.counts.pages_all,
- "sessions": report.counts.sessions,
- "observations": report.counts.observations,
- },
- "derived": report.derived,
- "storage": report.storage,
- "providers": report.providers,
- "spool": spool,
- "capture_mode": capture_mode,
- "ingest": report.ingest,
- "client": { "server_url": ep.url, "auth": ep.auth_token.is_some() },
- }))?
+ serde_json::to_string_pretty(&json_report(
+ report,
+ spool,
+ journal_pending,
+ journal_degraded,
+ journal_finish_failed,
+ journal_error,
+ capture_mode,
+ &ep,
+ ))?
);
} else {
println!("ai-memory {} (server)", report.version);
@@ -355,6 +372,16 @@ pub async fn run(config: &Config, args: StatusArgs) -> Result<()> {
println!(" pending: {}", spool.pending);
println!(" oldest: {}", spool_age_line(spool.oldest_age_ms));
println!(" retries: {}", spool.retries_total);
+ if journal_pending > 0 || journal_error.is_some() {
+ println!(" recovery:");
+ println!(
+ " journal: {journal_pending} pending ({journal_degraded} degraded, {journal_finish_failed} finish-failed)"
+ );
+ if let Some(error) = &journal_error {
+ println!(" error: {error}");
+ }
+ println!(" action: run `ai-memory recover`");
+ }
if let Some(ingest) = &report.ingest {
println!(" ingest (server, this process):");
println!(" accepted: {}", ingest.accepted);
@@ -453,6 +480,44 @@ pub(super) fn spool_age_line(age_ms: Option<u64>) -> String {
}
}
+#[allow(clippy::too_many_arguments)]
+fn json_report(
+ report: Report,
+ spool: SpoolHealth,
+ journal_pending: usize,
+ journal_degraded: usize,
+ journal_finish_failed: usize,
+ journal_error: Option<String>,
+ capture_mode: &str,
+ endpoint: &ServerEndpoint,
+) -> serde_json::Value {
+ serde_json::json!({
+ "version": report.version,
+ "data_dir": report.data_dir,
+ "bind": report.bind,
+ "db_path": report.db_path,
+ "counts": {
+ "pages_latest": report.counts.pages_latest,
+ "pages_all": report.counts.pages_all,
+ "sessions": report.counts.sessions,
+ "observations": report.counts.observations,
+ },
+ "derived": report.derived,
+ "storage": report.storage,
+ "providers": report.providers,
+ "spool": spool,
+ "recovery": {
+ "journal_pending": journal_pending,
+ "degraded_runs": journal_degraded,
+ "finish_failed_runs": journal_finish_failed,
+ "journal_error": journal_error,
+ },
+ "capture_mode": capture_mode,
+ "ingest": report.ingest,
+ "client": { "server_url": endpoint.url, "auth": endpoint.auth_token.is_some() },
+ })
+}
+
fn provider_health_line(role: &ProviderRoleHealthSnapshot) -> String {
match role.status {
ProviderHealthStatus::Disabled => "disabled".to_string(),
@@ -507,6 +572,64 @@ fn error_detail(role: &ProviderRoleHealthSnapshot) -> String {
mod tests {
use super::*;
+ #[test]
+ fn status_json_keeps_existing_fields_and_adds_recovery_compatibly() {
+ let report: Report = serde_json::from_value(serde_json::json!({
+ "version": "2.6.0",
+ "data_dir": "/data",
+ "bind": "127.0.0.1:49374",
+ "db_path": "/data/db/memory.sqlite",
+ "counts": {"pages_latest": 1, "pages_all": 2, "sessions": 3, "observations": 4},
+ "derived": {
+ "pages_rows": 0, "pages_fts_rows": 0,
+ "observations_rows": 0, "observations_fts_rows": 0,
+ "latest_pages_missing_embeddings": 0,
+ "embedding_rows": 0, "embedding_triples": [],
+ "links_from_latest_pages": 0,
+ "unresolved_links_from_latest_pages": 0,
+ "stale_links_from_latest_pages": 0
+ },
+ "storage": {
+ "page_size": 0, "page_count": 0, "freelist_count": 0,
+ "database_bytes": 0, "reclaimable_bytes": 0
+ },
+ "providers": ProviderHealthSnapshot::default(),
+ "ingest": null,
+ "write_queue": null,
+ "wiki_format": null
+ }))
+ .unwrap();
+ let endpoint = ServerEndpoint::from_pair(Some("http://127.0.0.1:49374".into()), None);
+ let value = json_report(
+ report,
+ SpoolHealth::default(),
+ 2,
+ 1,
+ 1,
+ None,
+ "denylist",
+ &endpoint,
+ );
+ for key in [
+ "version",
+ "data_dir",
+ "bind",
+ "db_path",
+ "counts",
+ "derived",
+ "storage",
+ "providers",
+ "spool",
+ "capture_mode",
+ "ingest",
+ "client",
+ ] {
+ assert!(value.get(key).is_some(), "missing legacy status key {key}");
+ }
+ assert_eq!(value["recovery"]["journal_pending"], 2);
+ assert_eq!(value["counts"]["observations"], 4);
+ }
+
#[test]
fn ingest_additions_default_for_older_servers_and_roundtrip() {
let old = serde_json::json!({"accepted": 2, "dropped_by_policy": 1, "shed_saturated": 0, "shed_rate_limited": 0, "last_persisted_ms": null});
diff --git a/crates/ai-memory-cli/src/lib.rs b/crates/ai-memory-cli/src/lib.rs
index b03dfd54..8cdc0d23 100644
--- a/crates/ai-memory-cli/src/lib.rs
+++ b/crates/ai-memory-cli/src/lib.rs
@@ -160,6 +160,13 @@ pub async fn run() -> Result<()> {
Command::AutoImproveReport(args) => commands::auto_improve_report::run(&config, args).await,
Command::AutoImprove(args) => commands::auto_improve::run(&config, args).await,
Command::FinalizeSession(args) => commands::finalize_session::run(&config, args).await,
+ Command::Recover(args) => {
+ let exit_code = commands::recovery::run(&config, args).await?;
+ if exit_code != 0 {
+ std::process::exit(exit_code);
+ }
+ Ok(())
+ }
Command::PendingWrites(args) => commands::pending_writes::run(&config, args).await,
Command::Embed(args) => commands::embed::run(&config, args).await,
Command::GenerateAuthToken(args) => commands::generate_auth_token::run(&config, args),
diff --git a/crates/ai-memory-cli/tests/suite/backfill_e2e.rs b/crates/ai-memory-cli/tests/suite/backfill_e2e.rs
index 2473da11..b4a60b68 100644
--- a/crates/ai-memory-cli/tests/suite/backfill_e2e.rs
+++ b/crates/ai-memory-cli/tests/suite/backfill_e2e.rs
@@ -79,6 +79,17 @@ mod slow {
None
}
+ fn status_json(data_dir: &Path, home: &Path, cwd: &Path, base: &str) -> Value {
+ serde_json::from_str(&run_cli(
+ &["status", "--json"],
+ data_dir,
+ home,
+ Some(cwd),
+ base,
+ ))
+ .expect("status JSON remains compatible")
+ }
+
/// Call a memory tool over the stateless Streamable-HTTP `/mcp` transport
/// (the default `serve` mode: no `initialize` handshake needed) and return
/// the joined text of the tool result.
@@ -296,6 +307,41 @@ mod slow {
"the re-run must not duplicate the imported session",
);
+ let observations_before = status_json(data_dir.path(), home.path(), &cwd, &base)["counts"]
+ ["observations"]
+ .as_u64()
+ .unwrap();
+ let forced: Value = serde_json::from_str(&run_cli(
+ &[
+ "backfill",
+ "--workspace",
+ WORKSPACE,
+ "--project",
+ PROJECT,
+ "--force",
+ "--json",
+ ],
+ data_dir.path(),
+ home.path(),
+ Some(&cwd),
+ &base,
+ ))
+ .expect("forced replay report");
+ assert_eq!(forced["imported_sessions"], 1);
+ assert_eq!(
+ session_count(&client, &base, WORKSPACE, PROJECT).await,
+ 1,
+ "a real second replay must reuse the session",
+ );
+ let status = status_json(data_dir.path(), home.path(), &cwd, &base);
+ assert!(status["ingest"]["replayed"].as_u64().unwrap_or(0) >= 2);
+ assert!(status["ingest"]["ignored_end"].as_u64().unwrap_or(0) >= 1);
+ assert_eq!(
+ status["counts"]["observations"].as_u64(),
+ Some(observations_before),
+ "the second replay must create no duplicate observations"
+ );
+
drop(server);
}
}
diff --git a/crates/ai-memory-cli/tests/suite/backfill_failures.rs b/crates/ai-memory-cli/tests/suite/backfill_failures.rs
index c5b6c29e..4370107f 100644
--- a/crates/ai-memory-cli/tests/suite/backfill_failures.rs
+++ b/crates/ai-memory-cli/tests/suite/backfill_failures.rs
@@ -3,6 +3,69 @@
use axum::{Json, Router, http::StatusCode, routing::post};
use serde_json::{Value, json};
+#[tokio::test]
+async fn replay_refuses_legacy_partial_and_terminal_drop_acknowledgements() {
+ for ack in [
+ json!({"accepted": 3}),
+ json!({"accepted": 1, "accepted_indices": [0], "failed_index": 1,
+ "results": [{"index": 0, "outcome": "stored"}]}),
+ json!({"accepted": 3, "results": [
+ {"index": 0, "outcome": "stored"},
+ {"index": 1, "outcome": "dropped_collision"},
+ {"index": 2, "outcome": "ignored_end"}
+ ]}),
+ ] {
+ let home = tempfile::tempdir().unwrap();
+ let project = tempfile::tempdir().unwrap();
+ let cwd = std::fs::canonicalize(project.path()).unwrap();
+ let transcripts = home
+ .path()
+ .join(".claude/projects")
+ .join(cwd.to_string_lossy().replace('/', "-"));
+ std::fs::create_dir_all(&transcripts).unwrap();
+ let sid = "11111111-2222-3333-4444-555555555555";
+ crate::e2e_support::write_jsonl(
+ &transcripts.join(format!("{sid}.jsonl")),
+ &[
+ json!({"sessionId": sid, "cwd": cwd}),
+ json!({"type": "user", "message": {"role": "user", "content": [{"type":"text", "text":"fixture history"}]}}),
+ ],
+ );
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let addr = listener.local_addr().unwrap();
+ let response = ack.clone();
+ let app = Router::new().route(
+ "/hook/batch",
+ post(move || {
+ let response = response.clone();
+ async move { Json(response) }
+ }),
+ );
+ let server = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
+ let data = tempfile::tempdir().unwrap();
+ let output = tokio::process::Command::from(crate::e2e_support::hermetic(env!(
+ "CARGO_BIN_EXE_ai-memory"
+ )))
+ .current_dir(&cwd)
+ .env("AI_MEMORY_HOME", home.path())
+ .env("AI_MEMORY_DATA_DIR", data.path())
+ .env("AI_MEMORY_SERVER_URL", format!("http://{addr}"))
+ .args([
+ "backfill",
+ "--workspace",
+ "review",
+ "--project",
+ "fixture",
+ "--force",
+ ])
+ .output()
+ .await
+ .unwrap();
+ assert!(!output.status.success(), "ack was accepted: {ack}");
+ server.abort();
+ }
+}
+
#[tokio::test]
async fn import_failures_are_errors_in_human_json_and_quiet_modes() {
for partial in [false, true] {
@@ -32,7 +95,13 @@ async fn import_failures_are_errors_in_human_json_and_quiet_modes() {
"/hook/batch",
post(move |Json(items): Json<Vec<Value>>| async move {
if partial && items[0]["url"].as_str().unwrap().contains("99999999") {
- (StatusCode::OK, Json(json!({"accepted": items.len()})))
+ let results = (0..items.len())
+ .map(|index| json!({"index": index, "outcome": "stored"}))
+ .collect::<Vec<_>>();
+ (
+ StatusCode::OK,
+ Json(json!({"accepted": items.len(), "results": results})),
+ )
} else {
(
StatusCode::INTERNAL_SERVER_ERROR,
diff --git a/crates/ai-memory-cli/tests/suite/run_offline.rs b/crates/ai-memory-cli/tests/suite/run_offline.rs
index 6213609c..1a0fce7a 100644
--- a/crates/ai-memory-cli/tests/suite/run_offline.rs
+++ b/crates/ai-memory-cli/tests/suite/run_offline.rs
@@ -8,14 +8,25 @@
#![cfg(unix)]
use std::fs;
+use std::io::Write as _;
use std::os::unix::fs::PermissionsExt as _;
use std::path::{Path, PathBuf};
use std::process::{Output, Stdio};
+use std::sync::{Arc, Mutex};
-use axum::{Json, Router, http::StatusCode, routing::post};
-use serde_json::json;
+use axum::{
+ Json, Router,
+ extract::State,
+ http::StatusCode,
+ routing::{get, post},
+};
+use serde_json::{Value, json};
const BIN: &str = env!("CARGO_BIN_EXE_ai-memory");
+type RecoveryServerState = (
+ Arc<Mutex<Vec<Vec<Value>>>>,
+ Arc<Mutex<Vec<(String, Value)>>>,
+);
/// A loopback address nothing listens on: `free_port` binds and releases.
fn closed_port_url() -> String {
@@ -81,7 +92,7 @@ impl Fixture {
"printf '%s\\n' \"$@\" > '{ran}'\nset > '{env_}'\n\
while [ $# -gt 0 ]; do\n\
if [ \"$1\" = --session-id ]; then\n\
- printf '{{\"sessionId\":\"%s\",\"cwd\":\"%s\"}}\\n' \"$2\" '{repo}' > '{dir}'/\"$2\".jsonl\n\
+ printf '{{\"sessionId\":\"%s\",\"cwd\":\"%s\"}}\\n{{\"type\":\"user\",\"message\":{{\"role\":\"user\",\"content\":[{{\"type\":\"text\",\"text\":\"offline exact prompt\"}}]}}}}\\n{{\"type\":\"assistant\",\"message\":{{\"role\":\"assistant\",\"content\":[{{\"type\":\"text\",\"text\":\"offline exact reply\"}}]}}}}\\n' \"$2\" '{repo}' > '{dir}'/\"$2\".jsonl\n\
fi\nshift\ndone\nexit 0\n",
ran = claude_ran.display(),
env_ = claude_env.display(),
@@ -131,10 +142,131 @@ async fn run(fixture: &Fixture, server: &str, args: &[&str]) -> Output {
.expect("spawn ai-memory")
}
-fn stderr(output: &Output) -> String {
+async fn recover(fixture: &Fixture, server: &str) -> Output {
+ run(fixture, server, &["recover", "--json"]).await
+}
+
+fn journal_entries(fixture: &Fixture) -> Vec<Value> {
+ let journal_path = fixture.home.join("data/recovery-journal.json");
+ let journal: Value = serde_json::from_slice(&fs::read(journal_path).expect("recovery journal"))
+ .expect("recovery journal JSON");
+ journal["entries"]
+ .as_array()
+ .expect("journal entries")
+ .clone()
+}
+
+fn stderr_text(output: &Output) -> String {
String::from_utf8_lossy(&output.stderr).into_owned()
}
+/// Re-stamp the newest spool entry as a hook that fired inside the journal
+/// interval `[0].interval_started_ms ..= interval_ended_ms`. Real harness
+/// hooks spool while the child runs; the fake harness has none, so the test
+/// enqueues after the fact and back-dates both the body and the file name
+/// (whose leading milliseconds the tombstone cleaner trusts).
+fn backdate_latest_spool_entry(fixture: &Fixture, ms: u64) {
+ let spool = fixture.home.join("data/hook-spool");
+ let mut newest: Option<(std::path::PathBuf, String)> = None;
+ for entry in fs::read_dir(&spool).unwrap() {
+ let path = entry.unwrap().path();
+ if path.extension().and_then(|value| value.to_str()) != Some("json") {
+ continue;
+ }
+ let name = path
+ .file_name()
+ .and_then(|name| name.to_str())
+ .unwrap()
+ .to_string();
+ if newest.as_ref().is_none_or(|(_, other)| &name > other) {
+ newest = Some((path, name));
+ }
+ }
+ let Some((path, name)) = newest else {
+ panic!("no spool entry to back-date");
+ };
+ let mut value: Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap();
+ value["created_ms"] = json!(ms);
+ fs::write(&path, serde_json::to_vec(&value).unwrap()).unwrap();
+ let suffix = name.split_once('-').expect("spool name embeds its stamp").1;
+ fs::rename(&path, spool.join(format!("{:013}-{suffix}", ms))).unwrap();
+}
+
+fn enqueue_hook(fixture: &Fixture, server: &str, session_id: &str, marker: &str) {
+ let mut child = crate::e2e_support::hermetic(BIN)
+ .args([
+ "--data-dir",
+ fixture.home.join("data").to_str().unwrap(),
+ "hook",
+ "--event",
+ "user-prompt-submit",
+ "--agent",
+ "claude-code",
+ "--server-url",
+ server,
+ ])
+ .current_dir(&fixture.repo)
+ .stdin(Stdio::piped())
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped())
+ .spawn()
+ .unwrap();
+ child
+ .stdin
+ .take()
+ .unwrap()
+ .write_all(
+ json!({
+ "session_id": session_id,
+ "cwd": fixture.repo,
+ "prompt": marker,
+ })
+ .to_string()
+ .as_bytes(),
+ )
+ .unwrap();
+ let output = child.wait_with_output().unwrap();
+ assert!(output.status.success(), "{}", stderr_text(&output));
+}
+
+async fn fake_recovery_server_on(
+ listener: tokio::net::TcpListener,
+ batches: Arc<Mutex<Vec<Vec<Value>>>>,
+ finishes: Arc<Mutex<Vec<(String, Value)>>>,
+) -> tokio::task::JoinHandle<()> {
+ async fn health() -> StatusCode {
+ StatusCode::OK
+ }
+ async fn batch(
+ State((batches, _)): State<RecoveryServerState>,
+ Json(items): Json<Vec<Value>>,
+ ) -> Json<Value> {
+ let accepted = items.len();
+ let results = (0..accepted)
+ .map(|index| json!({"index": index, "outcome": "stored"}))
+ .collect::<Vec<_>>();
+ batches.lock().unwrap().push(items);
+ Json(json!({ "accepted": accepted, "results": results }))
+ }
+ async fn finish(
+ State((_, finishes)): State<RecoveryServerState>,
+ axum::extract::Path(run_id): axum::extract::Path<String>,
+ Json(body): Json<Value>,
+ ) -> Json<Value> {
+ finishes.lock().unwrap().push((run_id, body));
+ Json(json!({ "imported_events": 1, "latest_sequence": 1 }))
+ }
+
+ let state = (batches, finishes);
+ let app = Router::new()
+ .route("/healthz", get(health))
+ .route("/hook/batch", post(batch))
+ .route("/workstream/runs/{run_id}/finish", post(finish))
+ .route("/workstream/runs/{run_id}/recover/finish", post(finish))
+ .with_state(state);
+ tokio::spawn(async move { axum::serve(listener, app).await.unwrap() })
+}
+
/// A closed port downgrades the launch: one loud warning, the harness runs,
/// no run/workstream ids are attributed to the offline child, and the
/// not-recorded line accounts for the (empty) local spool.
@@ -143,27 +275,48 @@ async fn offline_run_launches_the_harness_without_the_server() {
let fixture = Fixture::new();
let server = closed_port_url();
let output = run(&fixture, &server, &["run", "--no-autowire", "claude"]).await;
- assert!(output.status.success(), "{}", stderr(&output));
+ assert!(output.status.success(), "{}", stderr_text(&output));
- let stderr = stderr(&output);
+ let output_stderr = stderr_text(&output);
assert!(
- stderr.contains("WARNING") && stderr.contains("unreachable"),
- "the degraded warning is loud:\n{stderr}"
+ output_stderr.contains("WARNING") && output_stderr.contains("unreachable"),
+ "the degraded warning is loud:\n{output_stderr}"
);
assert!(
- stderr.contains(&server),
- "the warning names the server URL:\n{stderr}"
+ output_stderr.contains(&server),
+ "the warning names the server URL:\n{output_stderr}"
);
assert!(
- stderr.contains("not recorded on the server"),
- "the spool account is printed:\n{stderr}"
+ output_stderr.contains("not recorded on the server"),
+ "the spool account is printed:\n{output_stderr}"
);
assert!(
- stderr.contains("no hook events remain spooled locally"),
- "the empty spool is reported as empty:\n{stderr}"
+ output_stderr.contains("no hook events remain spooled locally"),
+ "the empty spool is reported as empty:\n{output_stderr}"
+ );
+
+ assert!(
+ fixture.claude_ran.exists(),
+ "the harness ran:\n{output_stderr}"
);
+ let entries = journal_entries(&fixture);
+ assert_eq!(
+ entries.len(),
+ 1,
+ "one offline run is journaled: {entries:?}"
+ );
+ assert_eq!(entries[0]["kind"], "degraded-run");
+ assert_eq!(entries[0]["workspace"], "default");
+ assert!(entries[0]["native_session_id"].as_str().is_some());
+ let journal_text = serde_json::to_string(&entries).unwrap();
+ assert!(!journal_text.contains("offline exact prompt"));
+ let pending = recover(&fixture, &server).await;
+ assert!(pending.status.success(), "{}", stderr_text(&pending));
+ let report: Value = serde_json::from_slice(&pending.stdout).expect("recover JSON");
+ assert_eq!(report["reachable"], false);
+ assert_eq!(report["journal_remaining"], 1);
+ assert_eq!(journal_entries(&fixture).len(), 1);
- assert!(fixture.claude_ran.exists(), "the harness ran:\n{stderr}");
let child_env = fs::read_to_string(&fixture.claude_env).unwrap();
assert!(
!child_env
@@ -180,6 +333,203 @@ async fn offline_run_launches_the_harness_without_the_server() {
);
}
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn unsupported_exporter_is_journaled_by_a_real_degraded_run_as_spool_only() {
+ let fixture = Fixture::new();
+ let conversation = "12345678-1234-4234-9234-123456789abc";
+ let store = fixture.home.join(".gemini/antigravity-cli/conversations");
+ fs::create_dir_all(&store).unwrap();
+ let db = store.join(format!("{conversation}.db"));
+ let connection = rusqlite::Connection::open(&db).unwrap();
+ connection
+ .execute_batch("CREATE TABLE trajectory (trajectory_metadata_blob BLOB NOT NULL);")
+ .unwrap();
+ let uri = format!("file://{}", fixture.repo.display());
+ let mut nested = Vec::new();
+ nested.push(10);
+ nested.push(u8::try_from(uri.len()).unwrap());
+ nested.extend_from_slice(uri.as_bytes());
+ let mut metadata = vec![10, u8::try_from(nested.len()).unwrap()];
+ metadata.extend_from_slice(&nested);
+ connection
+ .execute(
+ "INSERT INTO trajectory (trajectory_metadata_blob) VALUES (?1)",
+ rusqlite::params![metadata],
+ )
+ .unwrap();
+ drop(connection);
+ write_script(&fixture.bin.join("agy"), "exit 0\n");
+
+ let server = closed_port_url();
+ let output = run(
+ &fixture,
+ &server,
+ &[
+ "run",
+ "--no-autowire",
+ "antigravity",
+ "--conversation",
+ conversation,
+ ],
+ )
+ .await;
+ assert!(output.status.success(), "{}", stderr_text(&output));
+ let entries = journal_entries(&fixture);
+ assert_eq!(entries.len(), 1, "real degraded run must be journaled");
+ assert_eq!(entries[0]["harness"], "antigravity");
+ assert_eq!(entries[0]["native_session_id"], conversation);
+ assert_eq!(entries[0]["spool_only"], true);
+ assert!(entries[0].get("event_digests").is_none());
+ assert!(entries[0].get("final_cursor").is_none());
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn recovery_refuses_same_event_ids_with_mutated_content() {
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let server_url = format!("http://{}", listener.local_addr().unwrap());
+ let fixture = Fixture::new();
+ let output = run(&fixture, &server_url, &["run", "--no-autowire", "claude"]).await;
+ assert!(output.status.success(), "{}", stderr_text(&output));
+ let entries = journal_entries(&fixture);
+ let session_id = entries[0]["native_session_id"].as_str().unwrap();
+ let transcript = fixture
+ .home
+ .join(".claude/projects")
+ .join(fixture.repo.to_string_lossy().replace('/', "-"))
+ .join(format!("{session_id}.jsonl"));
+ let original = fs::read_to_string(&transcript).unwrap();
+ fs::write(
+ &transcript,
+ original.replace("offline exact reply", "mutated exact reply"),
+ )
+ .unwrap();
+
+ let batches = Arc::new(Mutex::new(Vec::new()));
+ let finishes = Arc::new(Mutex::new(Vec::new()));
+ let server = fake_recovery_server_on(listener, batches.clone(), finishes).await;
+ let recovered = recover(&fixture, &server_url).await;
+ assert!(
+ !recovered.status.success(),
+ "semantic mutation must refuse replay"
+ );
+ let report: Value = serde_json::from_slice(&recovered.stdout).unwrap();
+ assert_eq!(report["journal_remaining"], 1);
+ assert!(batches.lock().unwrap().is_empty());
+ server.abort();
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+async fn recover_drains_spool_then_replays_exact_degraded_transcript_idempotently() {
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let server_url = format!("http://{}", listener.local_addr().unwrap());
+ let fixture = Fixture::new();
+ let output = run(&fixture, &server_url, &["run", "--no-autowire", "claude"]).await;
+ assert!(output.status.success(), "{}", stderr_text(&output));
+ let entries = journal_entries(&fixture);
+ let session_id = entries[0]["native_session_id"]
+ .as_str()
+ .unwrap()
+ .to_string();
+ enqueue_hook(&fixture, &server_url, &session_id, "offline exact prompt");
+ backdate_latest_spool_entry(
+ &fixture,
+ entries[0]["interval_ended_ms"]
+ .as_u64()
+ .expect("interval end"),
+ );
+ let transcript = fixture
+ .home
+ .join(".claude/projects")
+ .join(fixture.repo.to_string_lossy().replace('/', "-"))
+ .join(format!("{session_id}.jsonl"));
+ let mut transcript_file = fs::OpenOptions::new()
+ .append(true)
+ .open(transcript)
+ .unwrap();
+ writeln!(
+ transcript_file,
+ "{}",
+ json!({"type":"user","message":{"role":"user","content":[{"type":"text","text":"later resumed work"}]}})
+ )
+ .unwrap();
+
+ let batches = Arc::new(Mutex::new(Vec::new()));
+ let finishes = Arc::new(Mutex::new(Vec::new()));
+ let server = fake_recovery_server_on(listener, batches.clone(), finishes.clone()).await;
+ let recovered = recover(&fixture, &server_url).await;
+ assert!(recovered.status.success(), "{}", stderr_text(&recovered));
+ let report: Value = serde_json::from_slice(&recovered.stdout).expect("recover JSON");
+ assert_eq!(report["spool"]["sent"], 0);
+ assert_eq!(report["journal_remaining"], 0);
+ assert!(journal_entries(&fixture).is_empty());
+
+ let requests = batches.lock().unwrap().clone();
+ assert_eq!(
+ requests.len(),
+ 1,
+ "the correlated spool entry must be quarantined and replaced by the transcript"
+ );
+ let replay = &requests[0];
+ assert_eq!(replay.len(), 4, "start, exact user/reply, end: {replay:?}");
+ assert!(
+ replay[0]["url"]
+ .as_str()
+ .unwrap()
+ .contains("event=session-start")
+ );
+ assert_eq!(replay[1]["body"]["prompt"], "offline exact prompt");
+ assert_eq!(replay[2]["body"]["message"], "offline exact reply");
+ assert!(
+ replay.iter().all(|item| item["body"]
+ .to_string()
+ .find("later resumed work")
+ .is_none()),
+ "post-run resumed work must stay outside the captured upper bound: {replay:?}"
+ );
+ assert!(
+ replay[3]["url"]
+ .as_str()
+ .unwrap()
+ .contains("event=session-end")
+ );
+ assert!(replay.iter().all(|item| {
+ item["url"]
+ .as_str()
+ .unwrap()
+ .contains(&format!("session_id={session_id}"))
+ }));
+ assert!(finishes.lock().unwrap().is_empty());
+ let recovery_root = fixture.home.join("data/hook-spool/.recovery");
+ let recovery_dirs = fs::read_dir(&recovery_root)
+ .map(|entries| {
+ entries
+ .map(|entry| entry.unwrap().path())
+ .collect::<Vec<_>>()
+ })
+ .unwrap_or_default();
+ assert_eq!(recovery_dirs.len(), 1, "a bounded tombstone remains");
+ assert!(recovery_dirs[0].join(".completed").is_file());
+ assert_eq!(
+ fs::read_dir(&recovery_dirs[0])
+ .unwrap()
+ .filter(|entry| entry
+ .as_ref()
+ .unwrap()
+ .path()
+ .extension()
+ .is_some_and(|ext| ext == "json"))
+ .count(),
+ 0,
+ "confirmed replay deletes quarantined overlap but retains its tombstone"
+ );
+
+ let before = requests.len();
+ let rerun = recover(&fixture, &server_url).await;
+ assert!(rerun.status.success(), "{}", stderr_text(&rerun));
+ assert_eq!(batches.lock().unwrap().len(), before);
+ server.abort();
+}
+
/// `--require-server` restores the fail-closed behavior: the old augmented
/// connect error (including "the agent was not started"), and the harness
/// never starts.
@@ -194,18 +544,18 @@ async fn require_server_fails_closed_when_offline() {
)
.await;
assert!(!output.status.success());
- let stderr = stderr(&output);
+ let output_stderr = stderr_text(&output);
assert!(
- stderr.contains("the agent was not started"),
- "the prepare context is preserved:\n{stderr}"
+ output_stderr.contains("the agent was not started"),
+ "the prepare context is preserved:\n{output_stderr}"
);
assert!(
- stderr.contains("could not reach"),
- "the augmented connect diagnosis is reused:\n{stderr}"
+ output_stderr.contains("could not reach"),
+ "the augmented connect diagnosis is reused:\n{output_stderr}"
);
assert!(
!fixture.claude_ran.exists(),
- "the harness must not run:\n{stderr}"
+ "the harness must not run:\n{output_stderr}"
);
}
@@ -217,15 +567,15 @@ async fn offline_autowire_wires_hooks_but_not_mcp() {
let fixture = Fixture::new();
let server = closed_port_url();
let output = run(&fixture, &server, &["run", "claude"]).await;
- assert!(output.status.success(), "{}", stderr(&output));
- let stderr = stderr(&output);
+ assert!(output.status.success(), "{}", stderr_text(&output));
+ let output_stderr = stderr_text(&output);
assert!(
- stderr.contains("wiring its ai-memory hooks"),
- "the offline wiring is announced:\n{stderr}"
+ output_stderr.contains("wiring its ai-memory hooks"),
+ "the offline wiring is announced:\n{output_stderr}"
);
assert!(
- stderr.contains("MCP registration waits"),
- "the deferred MCP is announced:\n{stderr}"
+ output_stderr.contains("MCP registration waits"),
+ "the deferred MCP is announced:\n{output_stderr}"
);
let settings = fs::read_to_string(fixture.home.join(".claude/settings.json"))
@@ -255,8 +605,8 @@ async fn offline_autowire_wires_hooks_but_not_mcp() {
}
/// The mid-run outage repair path: when the server dies after prepare, the
-/// child's exit code survives, the warning points at `finalize-session`, and
-/// the recorded exit status stays the child's own. The mock server answers
+/// child's exit code survives, the warning points at `recover`, and the
+/// original run id stays journaled without repository-state details. The mock server answers
/// prepare, then closes its listener once the run is linked.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn a_mid_run_outage_preserves_the_child_exit_code() {
@@ -305,8 +655,18 @@ async fn a_mid_run_outage_preserves_the_child_exit_code() {
write_script(
&fixture.bin.join("claude"),
&format!(
- "printf '%s\\n' \"$@\" > '{}'\nsleep 1\nexit 5\n",
- fixture.claude_ran.display(),
+ "printf '%s\\n' \"$@\" > '{ran}'\n\
+ while [ $# -gt 0 ]; do\n\
+ if [ \"$1\" = --session-id ]; then\n\
+ printf '{{\"sessionId\":\"%s\",\"cwd\":\"%s\"}}\\n{{\"type\":\"user\",\"message\":{{\"role\":\"user\",\"content\":[{{\"type\":\"text\",\"text\":\"finish recovery prompt\"}}]}}}}\\n' \"$2\" '{repo}' > '{dir}'/\"$2\".jsonl\n\
+ fi\nshift\ndone\nsleep 1\nexit 5\n",
+ ran = fixture.claude_ran.display(),
+ repo = fixture.repo.display(),
+ dir = fixture
+ .home
+ .join(".claude/projects")
+ .join(fixture.repo.to_string_lossy().replace('/', "-"))
+ .display(),
),
);
let server_url = format!("http://{address}");
@@ -315,16 +675,53 @@ async fn a_mid_run_outage_preserves_the_child_exit_code() {
output.status.code(),
Some(5),
"the child's exit code is preserved, not an ai-memory error:\n{}",
- stderr(&output)
+ stderr_text(&output)
);
- let stderr = stderr(&output);
+ let output_stderr = stderr_text(&output);
assert!(
- stderr.contains("was not imported"),
- "the outage is reported:\n{stderr}"
+ output_stderr.contains("was not imported"),
+ "the outage is reported:\n{output_stderr}"
);
assert!(
- stderr.contains("ai-memory finalize-session"),
- "the repair path is named:\n{stderr}"
+ output_stderr.contains("ai-memory recover"),
+ "the automatic repair path is named:\n{output_stderr}"
);
- server.abort();
+ let entries = journal_entries(&fixture);
+ assert_eq!(entries.len(), 1);
+ assert_eq!(entries[0]["kind"], "finish-failed");
+ assert_eq!(entries[0]["run_id"], "12345678-1234-4234-9234-123456789abe");
+ assert_eq!(entries[0]["exit_code"], 5);
+ assert!(entries[0].get("checkpoint").is_none());
+ assert!(entries[0].get("changed_paths").is_none());
+
+ server.await.unwrap();
+ let listener = tokio::net::TcpListener::bind(address).await.unwrap();
+ let batches = Arc::new(Mutex::new(Vec::new()));
+ let finishes = Arc::new(Mutex::new(Vec::new()));
+ let recovery_server =
+ fake_recovery_server_on(listener, batches.clone(), finishes.clone()).await;
+ let recovered = recover(&fixture, &server_url).await;
+ assert!(
+ recovered.status.success(),
+ "stdout: {}\nstderr: {}",
+ String::from_utf8_lossy(&recovered.stdout),
+ stderr_text(&recovered)
+ );
+ assert!(journal_entries(&fixture).is_empty());
+ assert!(batches.lock().unwrap().is_empty());
+ let finish_requests = finishes.lock().unwrap().clone();
+ assert_eq!(finish_requests.len(), 1);
+ assert_eq!(finish_requests[0].0, "12345678-1234-4234-9234-123456789abe");
+ assert_eq!(finish_requests[0].1["complete"], true);
+ assert_eq!(finish_requests[0].1["exit_code"], 5);
+ assert!(finish_requests[0].1["checkpoint"].is_object());
+ assert_eq!(
+ finish_requests[0].1["events"][0]["content"],
+ "finish recovery prompt"
+ );
+
+ let rerun = recover(&fixture, &server_url).await;
+ assert!(rerun.status.success(), "{}", stderr_text(&rerun));
+ assert_eq!(finishes.lock().unwrap().len(), 1);
+ recovery_server.abort();
}
diff --git a/crates/ai-memory-hooks/src/lib.rs b/crates/ai-memory-hooks/src/lib.rs
index 9383a252..cc19cf15 100644
--- a/crates/ai-memory-hooks/src/lib.rs
+++ b/crates/ai-memory-hooks/src/lib.rs
@@ -52,6 +52,7 @@ pub use capture_policy::{
pub use payload::{
HookEnvelope, HookEvent, NOTIFICATION_EXCERPT_MAX_BYTES, POST_COMPACTION_EXCERPT_MAX_BYTES,
USER_PROMPT_EXCERPT_MAX_BYTES, agent_from_payload, cap_lifecycle_body_for_client,
+ hook_payload_context,
};
pub use router::{
DEFAULT_HOOK_INGEST_MAX_IN_FLIGHT, DEFAULT_INGEST_GATE_MAX_ENTRIES,
diff --git a/crates/ai-memory-hooks/src/payload.rs b/crates/ai-memory-hooks/src/payload.rs
index 5703c963..c7865288 100644
--- a/crates/ai-memory-hooks/src/payload.rs
+++ b/crates/ai-memory-hooks/src/payload.rs
@@ -500,6 +500,62 @@ pub fn agent_from_payload(raw: &serde_json::Value) -> Option<AgentKind> {
extract_string(raw, &["cursor_version"]).map(|_| AgentKind::Cursor)
}
+/// Session and checkout fields extracted from the canonical hook payload shapes.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct HookPayloadContext {
+ /// Harness-native session identity.
+ pub session_id: Option<String>,
+ /// Current checkout directory.
+ pub cwd: Option<String>,
+}
+
+/// Extract the session and checkout exactly as [`HookEnvelope`] does.
+#[must_use]
+pub fn hook_payload_context(raw: &serde_json::Value) -> HookPayloadContext {
+ let session_id = extract_string(
+ raw,
+ &[
+ "session_id",
+ "sessionId",
+ "sessionID",
+ "session",
+ "conversationId",
+ ],
+ )
+ .or_else(|| {
+ extract_string_path(
+ raw,
+ &[
+ &["info", "id"],
+ &["properties", "sessionID"],
+ &["properties", "info", "id"],
+ &["event", "properties", "sessionID"],
+ &["event", "properties", "info", "id"],
+ &["payload", "info", "id"],
+ &["payload", "properties", "sessionID"],
+ &["payload", "properties", "info", "id"],
+ ],
+ )
+ });
+ let cwd = extract_string(raw, &["cwd", "current_dir", "working_dir", "directory"])
+ .or_else(|| extract_first_string_array_item(raw, &["workspacePaths", "workspace_roots"]))
+ .or_else(|| {
+ extract_string_path(
+ raw,
+ &[
+ &["path", "cwd"],
+ &["info", "directory"],
+ &["properties", "info", "directory"],
+ &["event", "properties", "info", "directory"],
+ &["payload", "path", "cwd"],
+ &["payload", "info", "directory"],
+ &["payload", "properties", "info", "directory"],
+ ],
+ )
+ });
+ HookPayloadContext { session_id, cwd }
+}
+
impl HookEnvelope {
/// Build an envelope from the parsed query + the body JSON. Performs
/// best-effort extraction of `session_id` / `cwd` / a body excerpt
@@ -510,66 +566,14 @@ impl HookEnvelope {
let event = HookEvent::parse(&query.event);
let agent = agent_from_payload(&raw)
.unwrap_or_else(|| query.agent.as_deref().map_or(AgentKind::Other, parse_agent));
- // OpenCode's plugin SDK sends `sessionID` (capital `ID`) on the
- // tool.execute.*/session.* events; Claude Code and native Codex use
- // `session_id`, older Codex bridges `sessionId`, and Antigravity CLI
- // uses `conversationId`.
- // JSON keys are case-sensitive, so all spellings must be listed
- // or tool events fail the router's "missing session_id" check.
- let body_session_id = extract_string(
- &raw,
- &[
- "session_id",
- "sessionId",
- "sessionID",
- "session",
- "conversationId",
- ],
- )
- .or_else(|| {
- extract_string_path(
- &raw,
- &[
- &["info", "id"],
- &["properties", "sessionID"],
- &["properties", "info", "id"],
- &["event", "properties", "sessionID"],
- &["event", "properties", "info", "id"],
- &["payload", "info", "id"],
- &["payload", "properties", "sessionID"],
- &["payload", "properties", "info", "id"],
- ],
- )
- });
- let session_id = body_session_id.or_else(|| query.session_id.filter(|s| !s.is_empty()));
- // Cursor spells the workspace directory `workspace_roots` (an array,
- // normally one entry; multi-root workspaces carry several) and never
- // sends a usable top-level `cwd`: its `sessionStart` / `sessionEnd`
- // payloads omit `cwd` entirely, and its tool events send `cwd: ""`.
- // Without this spelling every Cursor session resolved to no cwd at all
- // and landed in the server-default `default/scratch` bucket.
- let body_cwd = extract_string(&raw, &["cwd", "current_dir", "working_dir", "directory"])
- .or_else(|| {
- extract_first_string_array_item(&raw, &["workspacePaths", "workspace_roots"])
- })
- .or_else(|| {
- extract_string_path(
- &raw,
- &[
- &["path", "cwd"],
- &["info", "directory"],
- &["properties", "info", "directory"],
- &["event", "properties", "info", "directory"],
- &["payload", "path", "cwd"],
- &["payload", "info", "directory"],
- &["payload", "properties", "info", "directory"],
- ],
- )
- });
+ let context = hook_payload_context(&raw);
+ let session_id = context
+ .session_id
+ .or_else(|| query.session_id.filter(|s| !s.is_empty()));
// Body cwd wins over the query-string fallback: the body is
// what agent CLIs natively send, so any query-string `cwd` is
// a bridge / test override that should defer to live data.
- let cwd = body_cwd.or_else(|| query.cwd.filter(|s| !s.is_empty()));
+ let cwd = context.cwd.or_else(|| query.cwd.filter(|s| !s.is_empty()));
let workspace_override = query.workspace.filter(|s| !s.is_empty());
let project_override = query.project.filter(|s| !s.is_empty());
let project_strategy = ProjectStrategy::parse(query.project_strategy.as_deref());
diff --git a/crates/ai-memory-hooks/src/workstream.rs b/crates/ai-memory-hooks/src/workstream.rs
index 3eefa347..227e5f87 100644
--- a/crates/ai-memory-hooks/src/workstream.rs
+++ b/crates/ai-memory-hooks/src/workstream.rs
@@ -83,6 +83,10 @@ pub fn workstream_router(state: WorkstreamState) -> Router {
)
.route("/workstream/runs/{run_id}/link", post(link_run))
.route("/workstream/runs/{run_id}/finish", post(finish_run))
+ .route(
+ "/workstream/runs/{run_id}/recover/finish",
+ post(recover_finish_run),
+ )
.route("/workstream/recent", post(list_recent_workstreams))
.route("/workstream/rename", post(rename_workstream))
.route("/workstream/{workstream_id}/events", get(search_events))
@@ -887,6 +891,21 @@ async fn link_run(
}
}
+async fn recover_finish_run(
+ State(state): State<WorkstreamState>,
+ level: Option<Extension<AuthLevel>>,
+ viewer: Option<Extension<ai_memory_core::AuthorizedViewer>>,
+ actor: Option<Extension<ai_memory_core::ActorContext>>,
+ user_id: Option<Extension<ai_memory_core::UserId>>,
+ AxumPath(raw_run_id): AxumPath<String>,
+ Json(request): Json<FinishManagedRunRequest>,
+) -> Response {
+ finish_run_inner(
+ state, level, viewer, actor, user_id, raw_run_id, request, true,
+ )
+ .await
+}
+
async fn finish_run(
State(state): State<WorkstreamState>,
level: Option<Extension<AuthLevel>>,
@@ -894,7 +913,24 @@ async fn finish_run(
actor: Option<Extension<ai_memory_core::ActorContext>>,
user_id: Option<Extension<ai_memory_core::UserId>>,
AxumPath(raw_run_id): AxumPath<String>,
- Json(mut request): Json<FinishManagedRunRequest>,
+ Json(request): Json<FinishManagedRunRequest>,
+) -> Response {
+ finish_run_inner(
+ state, level, viewer, actor, user_id, raw_run_id, request, false,
+ )
+ .await
+}
+
+#[allow(clippy::too_many_arguments)]
+async fn finish_run_inner(
+ state: WorkstreamState,
+ level: Option<Extension<AuthLevel>>,
+ viewer: Option<Extension<ai_memory_core::AuthorizedViewer>>,
+ actor: Option<Extension<ai_memory_core::ActorContext>>,
+ user_id: Option<Extension<ai_memory_core::UserId>>,
+ raw_run_id: String,
+ mut request: FinishManagedRunRequest,
+ recover_expired: bool,
) -> Response {
if let Err(response) = authorize(level, Capability::NormalWrite) {
return response.into_response();
@@ -943,7 +979,7 @@ async fn finish_run(
Ok(None) => return error(StatusCode::NOT_FOUND, "managed run not found"),
Err(failure) => return error(StatusCode::INTERNAL_SERVER_ERROR, failure.to_string()),
};
- if status.state == "finished" {
+ if status.state == "finished" && !recover_expired {
return match state
.writer
.finish_workstream_run(
@@ -969,8 +1005,10 @@ async fn finish_run(
Err(failure) => store_error_response(failure),
};
}
- if status.state != "active" {
- return error(StatusCode::CONFLICT, "managed run is not active");
+ if status.state != "active"
+ && !(recover_expired && matches!(status.state.as_str(), "expired" | "finished"))
+ {
+ return error(StatusCode::CONFLICT, "managed run is not recoverable");
}
let native_identity = match request
.native_session_id
@@ -999,7 +1037,7 @@ async fn finish_run(
) {
return error(StatusCode::BAD_REQUEST, message);
}
- if request.complete {
+ if request.complete && status.state != "finished" {
append_boundary_events(
&state.sanitizer,
run_id,
@@ -1008,16 +1046,20 @@ async fn finish_run(
&mut request,
);
}
- let segment_path = match write_segment(
- &state.data_dir,
- status.workstream_id,
- run_id,
- &request.events,
- ) {
- Ok(path) => path,
- Err(failure) => {
- warn!(error = %failure, run = %run_id, "managed transcript segment write failed");
- return error(StatusCode::INTERNAL_SERVER_ERROR, failure.to_string());
+ let segment_path = if status.state == "finished" {
+ None
+ } else {
+ match write_segment(
+ &state.data_dir,
+ status.workstream_id,
+ run_id,
+ &request.events,
+ ) {
+ Ok(path) => Some(path),
+ Err(failure) => {
+ warn!(error = %failure, run = %run_id, "managed transcript segment write failed");
+ return error(StatusCode::INTERNAL_SERVER_ERROR, failure.to_string());
+ }
}
};
let input = FinishWorkstreamRun {
@@ -1027,10 +1069,18 @@ async fn finish_run(
source_cursor: request.source_cursor,
events: request.events,
complete: request.complete,
- segment_path: Some(segment_path),
+ segment_path,
exit_code: request.exit_code,
};
- match state.writer.finish_workstream_run(authority, input).await {
+ let result = if recover_expired {
+ state
+ .writer
+ .recover_expired_workstream_run(authority, input)
+ .await
+ } else {
+ state.writer.finish_workstream_run(authority, input).await
+ };
+ match result {
Ok(result) => Json(FinishManagedRunResponse {
imported_events: result.imported_events,
latest_sequence: result.latest_sequence,
@@ -4037,6 +4087,201 @@ mod tests {
assert_eq!(finish_sql_snapshot(&store).await, before);
}
+ #[tokio::test]
+ async fn expired_run_recovery_is_owner_only_and_refuses_superseded_runs() {
+ let temp = TempDir::new().unwrap();
+ let store = Store::open(temp.path()).unwrap();
+ let mut state = test_state(&store, temp.path());
+ state.trusted_proxy_identity = true;
+ let (ws, project) = seed_scope(&store).await;
+ let prepare = |name: &str| PrepareWorkstreamRun {
+ workspace_id: ws,
+ project_id: project,
+ repo_fingerprint: name.into(),
+ worktree_fingerprint: name.into(),
+ cwd: "/repo".into(),
+ agent: AgentKind::Codex,
+ automatic_harness: false,
+ available_agents: Vec::new(),
+ selection: WorkstreamSelection::New(name.into()),
+ lease_owner: "launcher".into(),
+ };
+ let alice_actor = ai_memory_core::ActorContext {
+ user: Some("alice".into()),
+ ..ai_memory_core::ActorContext::anonymous()
+ };
+ let bob_actor = ai_memory_core::ActorContext {
+ user: Some("bob".into()),
+ ..ai_memory_core::ActorContext::anonymous()
+ };
+ let recoverable = store
+ .writer
+ .prepare_workstream_run_owned(prepare("recoverable"), Some("user:alice".into()))
+ .await
+ .unwrap();
+ store
+ .writer
+ .cancel_managed_run(recoverable.run_id)
+ .await
+ .unwrap();
+ let request = || FinishManagedRunRequest {
+ native_session_id: Some("native-recovery".into()),
+ source_cursor: None,
+ events: vec![NewWorkstreamEvent {
+ event_id: "recovered-event".into(),
+ agent: AgentKind::Codex,
+ native_session_id: "native-recovery".into(),
+ source_record_id: None,
+ kind: WorkstreamEventKind::Message,
+ role: Some("user".into()),
+ content: "recovered".into(),
+ occurred_at: None,
+ metadata: serde_json::Value::Null,
+ }],
+ complete: true,
+ checkpoint: Default::default(),
+ losses: Vec::new(),
+ exit_code: Some(0),
+ };
+ let refused = recover_finish_run(
+ State(state.clone()),
+ Some(Extension(AuthLevel::User)),
+ None,
+ Some(Extension(bob_actor)),
+ None,
+ AxumPath(recoverable.run_id.to_string()),
+ Json(request()),
+ )
+ .await;
+ assert_eq!(refused.status(), StatusCode::FORBIDDEN);
+ let accepted = recover_finish_run(
+ State(state.clone()),
+ Some(Extension(AuthLevel::User)),
+ None,
+ Some(Extension(alice_actor.clone())),
+ None,
+ AxumPath(recoverable.run_id.to_string()),
+ Json(request()),
+ )
+ .await;
+ assert_eq!(accepted.status(), StatusCode::OK);
+ let exact_retry = recover_finish_run(
+ State(state.clone()),
+ Some(Extension(AuthLevel::User)),
+ None,
+ Some(Extension(alice_actor.clone())),
+ None,
+ AxumPath(recoverable.run_id.to_string()),
+ Json(request()),
+ )
+ .await;
+ let exact_status = exact_retry.status();
+ let exact_body = to_bytes(exact_retry.into_body(), usize::MAX).await.unwrap();
+ assert_eq!(
+ exact_status,
+ StatusCode::OK,
+ "{}",
+ String::from_utf8_lossy(&exact_body)
+ );
+ let changed_checkpoint = recover_finish_run(
+ State(state.clone()),
+ Some(Extension(AuthLevel::User)),
+ None,
+ Some(Extension(alice_actor.clone())),
+ None,
+ AxumPath(recoverable.run_id.to_string()),
+ Json(FinishManagedRunRequest {
+ checkpoint: ai_memory_core::WorkstreamCheckpoint {
+ head: Some("repository-changed-after-finish".into()),
+ ..Default::default()
+ },
+ ..request()
+ }),
+ )
+ .await;
+ assert_eq!(changed_checkpoint.status(), StatusCode::OK);
+ assert_eq!(
+ store
+ .reader
+ .search_workstream_events(
+ recoverable.workstream_id,
+ "".into(),
+ 100,
+ Sanitizer::default(),
+ )
+ .await
+ .unwrap()
+ .len(),
+ 2,
+ "a finished retry must not append a regenerated checkpoint"
+ );
+ for changed in [
+ FinishManagedRunRequest {
+ native_session_id: Some("changed-native".into()),
+ ..request()
+ },
+ FinishManagedRunRequest {
+ events: {
+ let mut events = request().events;
+ events[0].event_id = "losing-event".into();
+ events
+ },
+ ..request()
+ },
+ FinishManagedRunRequest {
+ source_cursor: Some("uncommitted-cursor".into()),
+ ..request()
+ },
+ FinishManagedRunRequest {
+ exit_code: Some(9),
+ ..request()
+ },
+ ] {
+ let changed_finished = recover_finish_run(
+ State(state.clone()),
+ Some(Extension(AuthLevel::User)),
+ None,
+ Some(Extension(alice_actor.clone())),
+ None,
+ AxumPath(recoverable.run_id.to_string()),
+ Json(changed),
+ )
+ .await;
+ assert_eq!(changed_finished.status(), StatusCode::BAD_REQUEST);
+ }
+
+ let old = store
+ .writer
+ .prepare_workstream_run_owned(prepare("superseded"), Some("user:alice".into()))
+ .await
+ .unwrap();
+ store.writer.cancel_managed_run(old.run_id).await.unwrap();
+ store
+ .writer
+ .prepare_workstream_run_owned(
+ PrepareWorkstreamRun {
+ repo_fingerprint: "superseded".into(),
+ worktree_fingerprint: "superseded".into(),
+ selection: WorkstreamSelection::Named("superseded".into()),
+ ..prepare("replacement")
+ },
+ Some("user:alice".into()),
+ )
+ .await
+ .unwrap();
+ let superseded = recover_finish_run(
+ State(state),
+ Some(Extension(AuthLevel::User)),
+ None,
+ Some(Extension(alice_actor)),
+ None,
+ AxumPath(old.run_id.to_string()),
+ Json(request()),
+ )
+ .await;
+ assert_eq!(superseded.status(), StatusCode::BAD_REQUEST);
+ }
+
#[tokio::test]
async fn native_identity_http_stored_identity_refuses_before_raw_segment() {
let temp = TempDir::new().unwrap();
diff --git a/crates/ai-memory-store/migrations/V77__recovery_ingest_keys.sql b/crates/ai-memory-store/migrations/V77__recovery_ingest_keys.sql
new file mode 100644
index 00000000..ec53de26
--- /dev/null
+++ b/crates/ai-memory-store/migrations/V77__recovery_ingest_keys.sql
@@ -0,0 +1,7 @@
+ALTER TABLE ingest_keys ADD COLUMN session_id BLOB REFERENCES sessions(id) ON DELETE CASCADE;
+ALTER TABLE ingest_keys ADD COLUMN agent_kind TEXT;
+ALTER TABLE ingest_keys ADD COLUMN observation_kind TEXT;
+ALTER TABLE ingest_keys ADD COLUMN event_identity TEXT;
+ALTER TABLE ingest_keys ADD COLUMN recovery_retained INTEGER NOT NULL DEFAULT 0 CHECK (recovery_retained IN (0, 1));
+
+CREATE INDEX idx_ingest_keys_session ON ingest_keys (session_id);
diff --git a/crates/ai-memory-store/src/api_credentials.rs b/crates/ai-memory-store/src/api_credentials.rs
index 471cdc6c..9fa0ba20 100644
--- a/crates/ai-memory-store/src/api_credentials.rs
+++ b/crates/ai-memory-store/src/api_credentials.rs
@@ -411,7 +411,7 @@ mod tests {
|row| row.get(0),
)
.unwrap();
- assert_eq!(version, 76, "update the pin when adding a migration");
+ assert_eq!(version, 77, "update the pin when adding a migration");
let cols: i64 = conn
.query_row(
"SELECT COUNT(*) FROM pragma_table_info('users') WHERE name = 'token_hash'",
diff --git a/crates/ai-memory-store/src/lib.rs b/crates/ai-memory-store/src/lib.rs
index f4e0429a..d6df5ec3 100644
--- a/crates/ai-memory-store/src/lib.rs
+++ b/crates/ai-memory-store/src/lib.rs
@@ -3379,7 +3379,12 @@ mod tests {
.unwrap();
let session_a = SessionId::new();
let session_b = SessionId::new();
- for (session_id, project_id) in [(session_a, proj_a), (session_b, proj_b)] {
+ let session_c = SessionId::new();
+ for (session_id, project_id) in [
+ (session_a, proj_a),
+ (session_b, proj_b),
+ (session_c, proj_a),
+ ] {
store
.writer
.begin_session(NewSession {
@@ -3477,6 +3482,74 @@ mod tests {
.await
.unwrap();
assert!(matches!(reused, IngestObservationOutcome::Inserted(_)));
+
+ let recovery_key = format!("recovery_{}", "a".repeat(55));
+ let recovery_first = store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),
+ recovery_key.clone(),
+ )
+ .await
+ .unwrap();
+ assert!(matches!(
+ recovery_first,
+ IngestObservationOutcome::Inserted(_)
+ ));
+ store
+ .writer
+ .complete_observation_ingest(proj_a, recovery_key.clone())
+ .await
+ .unwrap();
+ let conn = Connection::open(store.db_path()).unwrap();
+ conn.execute(
+ "UPDATE ingest_keys SET seen_at = 1 WHERE project_id = ?1 AND key = ?2",
+ params![proj_a.as_bytes(), recovery_key],
+ )
+ .unwrap();
+ drop(conn);
+ let recovery_replay = store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(obs(session_a, proj_a), &Sanitizer::builtin()),
+ recovery_key.clone(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(recovery_replay, IngestObservationOutcome::AlreadyComplete);
+ assert!(matches!(
+ store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(obs(session_c, proj_a), &Sanitizer::builtin()),
+ recovery_key.clone(),
+ )
+ .await,
+ Err(StoreError::SessionCollision)
+ ));
+ let mut changed = obs(session_a, proj_a);
+ changed.body = "different event body".into();
+ assert!(matches!(
+ store
+ .writer
+ .insert_observation_ingest(
+ Sanitized::new(changed, &Sanitizer::builtin()),
+ recovery_key,
+ )
+ .await,
+ Err(StoreError::SessionCollision)
+ ));
+ let conn = Connection::open(store.db_path()).unwrap();
+ assert_eq!(
+ conn.query_row(
+ "SELECT COUNT(*) FROM observations WHERE project_id = ?1",
+ params![proj_a.as_bytes()],
+ |row| row.get::<_, i64>(0),
+ )
+ .unwrap(),
+ 3,
+ "a different session cannot preclaim or suppress a recovery key"
+ );
}
#[tokio::test]
diff --git a/crates/ai-memory-store/src/ops.rs b/crates/ai-memory-store/src/ops.rs
index d197ba15..4444e5ed 100644
--- a/crates/ai-memory-store/src/ops.rs
+++ b/crates/ai-memory-store/src/ops.rs
@@ -2744,10 +2744,39 @@ pub fn insert_observation(
}
/// Ingest-keys older than this are swept opportunistically on every keyed
-/// insert. Keys only need to outlive the client spool (7 days + retry
-/// backoff); 30 days is a generous margin.
+/// insert. Ordinary keys only need to outlive the client spool (7 days + retry
+/// backoff); recovery keys remain tied to their stored session until it is purged.
const INGEST_KEY_TTL_MICROS: i64 = 30 * 24 * 60 * 60 * 1_000_000;
+fn recovery_ingest_key(key: &str) -> bool {
+ key.strip_prefix("recovery_").is_some_and(|digest| {
+ digest.len() == 55 && digest.bytes().all(|byte| byte.is_ascii_hexdigit())
+ })
+}
+
+fn recovery_event_identity(obs: &NewObservation, agent: Option<&str>) -> String {
+ let mut hasher = Sha256::new();
+ let session_id = obs.session_id.to_string();
+ for value in [
+ Some(session_id.as_str()),
+ agent,
+ Some(obs.kind.as_str()),
+ obs.extension.as_deref(),
+ obs.source_event.as_deref(),
+ Some(obs.title.as_str()),
+ Some(obs.body.as_str()),
+ ] {
+ if let Some(value) = value {
+ hasher.update(value.len().to_be_bytes());
+ hasher.update(value.as_bytes());
+ } else {
+ hasher.update(usize::MAX.to_be_bytes());
+ }
+ }
+ hasher.update(obs.occurred_at.unwrap_or_default().to_be_bytes());
+ format!("{digest:x}", digest = hasher.finalize())
+}
+
/// Claim a project-scoped ingest key and append its observation atomically.
///
/// The key claim and the observation row commit in ONE transaction: either
@@ -2766,18 +2795,48 @@ pub fn insert_observation_keyed(
// Sweep before looking up the current key so an expired key can be reused
// even when no unrelated keyed event arrived in the meantime.
tx.execute(
- "DELETE FROM ingest_keys WHERE seen_at < ?1",
+ "DELETE FROM ingest_keys WHERE recovery_retained = 0 AND seen_at < ?1",
params![now - INGEST_KEY_TTL_MICROS],
)?;
- let existing: Option<Option<i64>> = tx
+ let recovery = recovery_ingest_key(ingest_key);
+ let session_agent = if recovery {
+ tx.query_row(
+ "SELECT agent_kind FROM sessions WHERE id = ?1",
+ params![obs.session_id.as_bytes()],
+ |row| row.get::<_, String>(0),
+ )
+ .optional()?
+ } else {
+ None
+ };
+ let event_identity = recovery.then(|| recovery_event_identity(obs, session_agent.as_deref()));
+ type IngestKeyRow = (
+ Option<i64>,
+ Option<Vec<u8>>,
+ Option<String>,
+ Option<String>,
+ Option<String>,
+ );
+ if recovery && session_agent.is_none() {
+ return Err(StoreError::SessionCollision);
+ }
+ let existing: Option<IngestKeyRow> = tx
.query_row(
- "SELECT completed_at FROM ingest_keys \
+ "SELECT completed_at, session_id, agent_kind, observation_kind, event_identity FROM ingest_keys \
WHERE project_id = ?1 AND key = ?2",
params![obs.project_id.as_bytes(), ingest_key],
- |row| row.get(0),
+ |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?, row.get(4)?)),
)
.optional()?;
- if let Some(completed_at) = existing {
+ if let Some((completed_at, session_id, agent, kind, identity)) = existing {
+ if recovery
+ && (session_id.as_deref() != Some(obs.session_id.as_bytes())
+ || agent.as_deref() != session_agent.as_deref()
+ || kind.as_deref() != Some(obs.kind.as_str())
+ || identity.as_deref() != event_identity.as_deref())
+ {
+ return Err(StoreError::SessionCollision);
+ }
tx.commit()?;
return Ok(if completed_at.is_some() {
IngestObservationOutcome::AlreadyComplete
@@ -2787,9 +2846,18 @@ pub fn insert_observation_keyed(
}
tx.execute(
- "INSERT INTO ingest_keys (project_id, key, seen_at, completed_at) \
- VALUES (?1, ?2, ?3, NULL)",
- params![obs.project_id.as_bytes(), ingest_key, now],
+ "INSERT INTO ingest_keys (project_id, key, seen_at, completed_at, session_id, agent_kind, observation_kind, event_identity, recovery_retained) \
+ VALUES (?1, ?2, ?3, NULL, ?4, ?5, ?6, ?7, ?8)",
+ params![
+ obs.project_id.as_bytes(),
+ ingest_key,
+ now,
+ recovery.then_some(obs.session_id.as_bytes()),
+ session_agent,
+ recovery.then_some(obs.kind.as_str()),
+ event_identity,
+ i64::from(recovery)
+ ],
)?;
let id = insert_observation_row(&tx, obs)?;
tx.commit()?;
@@ -2945,18 +3013,36 @@ pub fn admit_hook_session_event(
}
let ingest = if let Some(key) = ingest_key {
tx.execute(
- "DELETE FROM ingest_keys WHERE seen_at < ?1",
+ "DELETE FROM ingest_keys WHERE recovery_retained = 0 AND seen_at < ?1",
params![now - INGEST_KEY_TTL_MICROS],
)?;
- let old: Option<Option<i64>> = tx
+ let recovery = recovery_ingest_key(key);
+ let event_identity =
+ recovery.then(|| recovery_event_identity(obs, Some(session.agent_kind.as_str())));
+ type IngestKeyRow = (
+ Option<i64>,
+ Option<Vec<u8>>,
+ Option<String>,
+ Option<String>,
+ Option<String>,
+ );
+ let old: Option<IngestKeyRow> = tx
.query_row(
- "SELECT completed_at FROM ingest_keys WHERE project_id = ?1 AND key = ?2",
+ "SELECT completed_at, session_id, agent_kind, observation_kind, event_identity FROM ingest_keys WHERE project_id = ?1 AND key = ?2",
params![obs.project_id.as_bytes(), key],
- |r| r.get(0),
+ |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?, r.get(3)?, r.get(4)?)),
)
.optional()?;
match old {
- Some(done) => {
+ Some((done, keyed_session, keyed_agent, keyed_kind, keyed_identity)) => {
+ if recovery
+ && (keyed_session.as_deref() != Some(obs.session_id.as_bytes())
+ || keyed_agent.as_deref() != Some(session.agent_kind.as_str())
+ || keyed_kind.as_deref() != Some(obs.kind.as_str())
+ || keyed_identity.as_deref() != event_identity.as_deref())
+ {
+ return Err(StoreError::SessionCollision);
+ }
if done.is_some() {
IngestObservationOutcome::AlreadyComplete
} else {
@@ -2964,7 +3050,20 @@ pub fn admit_hook_session_event(
}
}
None => {
- tx.execute("INSERT INTO ingest_keys (project_id, key, seen_at, completed_at) VALUES (?1, ?2, ?3, NULL)", params![obs.project_id.as_bytes(), key, now])?;
+ tx.execute(
+ "INSERT INTO ingest_keys (project_id, key, seen_at, completed_at, session_id, agent_kind, observation_kind, event_identity, recovery_retained) \
+ VALUES (?1, ?2, ?3, NULL, ?4, ?5, ?6, ?7, ?8)",
+ params![
+ obs.project_id.as_bytes(),
+ key,
+ now,
+ recovery.then_some(obs.session_id.as_bytes()),
+ recovery.then_some(session.agent_kind.as_str()),
+ recovery.then_some(obs.kind.as_str()),
+ event_identity,
+ i64::from(recovery)
+ ],
+ )?;
IngestObservationOutcome::Inserted(insert_observation_row(&tx, obs)?)
}
}
@@ -8732,6 +8831,14 @@ pub(crate) mod tests {
fn purge_session_removes_the_session_and_everything_derived_from_it() {
let (_tmp, mut conn, ws, proj) = fresh_db();
let (sid, _page) = seed_session(&mut conn, ws, proj, "target");
+ let recovery_key = format!("recovery_{}", "a".repeat(55));
+ conn.execute(
+ "INSERT INTO ingest_keys \
+ (project_id, key, seen_at, completed_at, session_id, recovery_retained) \
+ VALUES (?1, ?2, 1, 1, ?3, 1)",
+ params![proj.as_bytes(), recovery_key, sid.as_bytes()],
+ )
+ .unwrap();
let summary = purge_session(
&mut conn,
@@ -8755,6 +8862,11 @@ pub(crate) mod tests {
"observations=0"
);
assert_eq!(count(&conn, "SELECT COUNT(*) FROM pages"), 0, "pages=0");
+ assert_eq!(
+ count(&conn, "SELECT COUNT(*) FROM ingest_keys"),
+ 0,
+ "session-bound recovery keys cascade with the purged session"
+ );
assert_eq!(summary.observations_deleted, 1);
assert_eq!(summary.pages_deleted, 1);
assert_eq!(
diff --git a/crates/ai-memory-store/src/workstream.rs b/crates/ai-memory-store/src/workstream.rs
index 87227430..0834ec5d 100644
--- a/crates/ai-memory-store/src/workstream.rs
+++ b/crates/ai-memory-store/src/workstream.rs
@@ -392,6 +392,7 @@ struct FinishRunRow {
sync_after: i64,
sync_through: i64,
context_delivered: bool,
+ exit_code: Option<i32>,
}
struct LinkRunRow {
@@ -1053,6 +1054,23 @@ pub(crate) fn finish_run(
conn: &mut Connection,
authority: &ManagedRunAuthority,
input: &FinishWorkstreamRun,
+) -> StoreResult<FinishedWorkstreamRun> {
+ finish_run_inner(conn, authority, input, false)
+}
+
+pub(crate) fn recover_run(
+ conn: &mut Connection,
+ authority: &ManagedRunAuthority,
+ input: &FinishWorkstreamRun,
+) -> StoreResult<FinishedWorkstreamRun> {
+ finish_run_inner(conn, authority, input, true)
+}
+
+fn finish_run_inner(
+ conn: &mut Connection,
+ authority: &ManagedRunAuthority,
+ input: &FinishWorkstreamRun,
+ recover_expired: bool,
) -> StoreResult<FinishedWorkstreamRun> {
let now = Timestamp::now().as_microsecond();
let tx = conn.transaction()?;
@@ -1060,7 +1078,7 @@ pub(crate) fn finish_run(
let run: Option<FinishRunRow> = tx
.query_row(
"SELECT workstream_id, agent_kind, native_session_id, state, \
- sync_after, sync_through, context_delivered \
+ sync_after, sync_through, context_delivered, exit_code \
FROM managed_runs WHERE id = ?1",
params![input.run_id.as_bytes()],
|row| {
@@ -1072,6 +1090,7 @@ pub(crate) fn finish_run(
sync_after: row.get(4)?,
sync_through: row.get(5)?,
context_delivered: row.get(6)?,
+ exit_code: row.get(7)?,
})
},
)
@@ -1090,6 +1109,7 @@ pub(crate) fn finish_run(
sync_after,
sync_through,
context_delivered,
+ exit_code,
} = run;
let latest_before: i64 = tx.query_row(
"SELECT COALESCE(MAX(sequence), 0) FROM workstream_events WHERE workstream_id = ?1",
@@ -1097,12 +1117,44 @@ pub(crate) fn finish_run(
|row| row.get(0),
)?;
if state == "finished" {
+ if recover_expired {
+ validate_finished_recovery(
+ &tx,
+ &workstream,
+ &agent_wire,
+ linked_session.as_deref(),
+ exit_code,
+ input,
+ )?;
+ }
+ tx.commit()?;
return Ok(FinishedWorkstreamRun {
imported_events: 0,
latest_sequence: latest_before,
});
}
- if state != "active" {
+ if state == "expired" && recover_expired {
+ let superseded: bool = tx.query_row(
+ "SELECT EXISTS(SELECT 1 FROM managed_runs newer \
+ JOIN managed_runs current ON current.id = ?2 \
+ WHERE newer.workstream_id = ?1 AND \
+ (newer.started_at > current.started_at OR \
+ (newer.started_at = current.started_at AND newer.rowid > current.rowid)))",
+ params![workstream, input.run_id.as_bytes()],
+ |row| row.get(0),
+ )?;
+ if superseded {
+ return Err(StoreError::InvalidState(format!(
+ "managed run {} was superseded and cannot be recovered",
+ input.run_id
+ )));
+ }
+ tx.execute(
+ "UPDATE managed_runs SET state = 'active', ended_at = NULL, lease_expires_at = ?1 \
+ WHERE id = ?2 AND state = 'expired'",
+ params![now + LEASE_MICROS, input.run_id.as_bytes()],
+ )?;
+ } else if state != "active" {
return Err(StoreError::InvalidState(format!(
"managed run {} is {state}",
input.run_id
@@ -1272,6 +1324,108 @@ pub(crate) fn finish_run(
})
}
+fn validate_finished_recovery(
+ tx: &Transaction<'_>,
+ workstream: &[u8],
+ agent_wire: &str,
+ linked_session: Option<&str>,
+ exit_code: Option<i32>,
+ input: &FinishWorkstreamRun,
+) -> StoreResult<()> {
+ if input.native_session_id.as_deref() != linked_session
+ || (input.complete && input.exit_code != exit_code)
+ || (!input.complete && (input.source_cursor.is_some() || input.exit_code.is_some()))
+ {
+ return Err(StoreError::InvalidState(
+ "finished managed run recovery is not an exact replay".into(),
+ ));
+ }
+ for event in &input.events {
+ type StoredEvent = (
+ String,
+ String,
+ String,
+ Option<String>,
+ Option<String>,
+ String,
+ Option<String>,
+ String,
+ );
+ let stored: Option<StoredEvent> = tx
+ .query_row(
+ "SELECT agent_kind, native_session_id, kind, source_record_id, role, content, \
+ occurred_at, metadata_json FROM workstream_events \
+ WHERE workstream_id = ?1 AND event_id = ?2",
+ params![workstream, event.event_id],
+ |row| {
+ Ok((
+ row.get(0)?,
+ row.get(1)?,
+ row.get(2)?,
+ row.get(3)?,
+ row.get(4)?,
+ row.get(5)?,
+ row.get(6)?,
+ row.get(7)?,
+ ))
+ },
+ )
+ .optional()?;
+ let (source_record_id, metadata) = scrub_workstream_provenance(
+ &input.sanitizer,
+ event.source_record_id.as_deref(),
+ &event.metadata,
+ );
+ let expected = (
+ event.agent.as_str(),
+ event.native_session_id.as_str(),
+ event.kind.as_str(),
+ source_record_id.as_deref(),
+ event.role.as_deref(),
+ ai_memory_core::truncate_utf8_bytes(&event.content, WORKSTREAM_CONTENT_MAX_BYTES),
+ event.occurred_at.as_deref(),
+ serde_json::to_string(&metadata)?,
+ );
+ let exact = stored.as_ref().is_some_and(|stored| {
+ stored.0 == expected.0
+ && stored.1 == expected.1
+ && stored.2 == expected.2
+ && stored.3.as_deref() == expected.3
+ && stored.4.as_deref() == expected.4
+ && stored.5 == expected.5
+ && stored.6.as_deref() == expected.6
+ && stored.7 == expected.7
+ });
+ if !exact || event.agent.as_str() != agent_wire {
+ return Err(StoreError::InvalidState(
+ "finished managed run recovery contains uncommitted or changed events".into(),
+ ));
+ }
+ }
+ let stored_cursor = if let Some(native_session) = linked_session {
+ tx.query_row(
+ "SELECT source_cursor FROM workstream_native_sessions \
+ WHERE workstream_id = ?1 AND agent_kind = ?2 AND native_session_id = ?3",
+ params![workstream, agent_wire, native_session],
+ |row| row.get::<_, Option<String>>(0),
+ )
+ .optional()?
+ .ok_or_else(|| {
+ StoreError::InvalidState(
+ "finished managed run recovery has no durable native session".into(),
+ )
+ })?
+ } else {
+ None
+ };
+ if input.complete && stored_cursor != input.source_cursor {
+ return Err(StoreError::InvalidState(
+ "finished managed run recovery cannot advance its source cursor".into(),
+ ));
+ }
+ Ok(())
+}
+
/// Reader-side managed-run status.
pub(crate) fn run_status(
conn: &Connection,
diff --git a/crates/ai-memory-store/src/writer.rs b/crates/ai-memory-store/src/writer.rs
index 7801a9e5..64d668d5 100644
--- a/crates/ai-memory-store/src/writer.rs
+++ b/crates/ai-memory-store/src/writer.rs
@@ -795,6 +795,7 @@ pub(crate) enum WriteCmd {
FinishWorkstreamRun {
authority: crate::ManagedRunAuthority,
input: FinishWorkstreamRun,
+ recover_expired: bool,
reply: oneshot::Sender<StoreResult<FinishedWorkstreamRun>>,
},
RenameWorkstream {
@@ -3401,11 +3402,32 @@ impl WriterHandle {
&self,
authority: crate::ManagedRunAuthority,
input: FinishWorkstreamRun,
+ ) -> StoreResult<FinishedWorkstreamRun> {
+ self.finish_workstream_run_inner(authority, input, false)
+ .await
+ }
+
+ /// Finish an expired run for its original owner when it was not superseded.
+ pub async fn recover_expired_workstream_run(
+ &self,
+ authority: crate::ManagedRunAuthority,
+ input: FinishWorkstreamRun,
+ ) -> StoreResult<FinishedWorkstreamRun> {
+ self.finish_workstream_run_inner(authority, input, true)
+ .await
+ }
+
+ async fn finish_workstream_run_inner(
+ &self,
+ authority: crate::ManagedRunAuthority,
+ input: FinishWorkstreamRun,
+ recover_expired: bool,
) -> StoreResult<FinishedWorkstreamRun> {
let (tx, rx) = oneshot::channel();
self.send(WriteCmd::FinishWorkstreamRun {
authority,
input,
+ recover_expired,
reply: tx,
})
.await?;
@@ -4758,9 +4780,14 @@ fn worker_loop(mut conn: Connection, mut rx: mpsc::Receiver<WriteCmd>) {
WriteCmd::FinishWorkstreamRun {
authority,
input,
+ recover_expired,
reply,
} => {
- let result = crate::workstream::finish_run(&mut conn, &authority, &input);
+ let result = if recover_expired {
+ crate::workstream::recover_run(&mut conn, &authority, &input)
+ } else {
+ crate::workstream::finish_run(&mut conn, &authority, &input)
+ };
send_or_warn(reply, result, "finish_workstream_run");
}
WriteCmd::RenameWorkstream { input, reply } => {
diff --git a/crates/ai-memory-store/tests/suite/multi_session.rs b/crates/ai-memory-store/tests/suite/multi_session.rs
index 19c4a4ed..6f62abf0 100644
--- a/crates/ai-memory-store/tests/suite/multi_session.rs
+++ b/crates/ai-memory-store/tests/suite/multi_session.rs
@@ -1582,6 +1582,37 @@ async fn owner_finish_store_refuses_another_writer_including_finished_retry() {
assert_eq!(retry.imported_events, 0);
assert_eq!(retry.latest_sequence, result.latest_sequence);
assert_eq!(finish_snapshot(&f.store).await, before);
+ for mutate in ["event-id", "content", "cursor", "exit-code"] {
+ let mut changed = finish_input(run.run_id);
+ match mutate {
+ "event-id" => changed.events[0].event_id = "losing-event".into(),
+ "content" => changed.events[0].content = "losing payload".into(),
+ "cursor" => changed.source_cursor = Some("uncommitted-cursor".into()),
+ "exit-code" => changed.exit_code = Some(9),
+ _ => unreachable!(),
+ }
+ assert!(
+ matches!(
+ f.store
+ .writer
+ .recover_expired_workstream_run(f.alice(), changed)
+ .await,
+ Err(StoreError::InvalidState(_))
+ ),
+ "finished recovery must reject changed {mutate}"
+ );
+ assert_eq!(finish_snapshot(&f.store).await, before);
+ }
+ assert_eq!(
+ f.store
+ .writer
+ .recover_expired_workstream_run(f.alice(), finish_input(run.run_id))
+ .await
+ .unwrap()
+ .imported_events,
+ 0,
+ "an exact owner retry remains idempotent"
+ );
for (name, auth) in [
(
"root-shared",
@@ -1947,7 +1978,7 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {
.authorize_managed_run(run.run_id, anonymous.clone())
.await
.unwrap();
- store
+ let owner_user = store
.writer
.create_human_user(
NewUser {
@@ -2038,6 +2069,92 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {
0
);
assert_eq!(finish_snapshot(&store).await, before);
+ // Explicit recovery admits the original owner only when no newer run exists.
+ store
+ .writer
+ .grant_memory(
+ owner_user,
+ project,
+ ai_memory_store::GrantLevel::Write,
+ None,
+ )
+ .await
+ .unwrap();
+ let recoverable = store
+ .writer
+ .prepare_workstream_run_owned(
+ PrepareWorkstreamRun {
+ workspace_id: ws,
+ project_id: project,
+ repo_fingerprint: "recoverable".into(),
+ worktree_fingerprint: "recoverable".into(),
+ cwd: "/repo".into(),
+ agent: AgentKind::Codex,
+ automatic_harness: false,
+ available_agents: Vec::new(),
+ selection: WorkstreamSelection::New("recoverable".into()),
+ lease_owner: "reload".into(),
+ },
+ Some("user:new-user".into()),
+ )
+ .await
+ .unwrap();
+ store
+ .writer
+ .cancel_managed_run(recoverable.run_id)
+ .await
+ .unwrap();
+ let foreign_user = store
+ .writer
+ .create_human_user(
+ NewUser {
+ username: "foreign-recovery".into(),
+ name: None,
+ email: None,
+ },
+ UserRole::User,
+ None,
+ false,
+ )
+ .await
+ .unwrap();
+ store
+ .writer
+ .grant_memory(
+ foreign_user,
+ project,
+ ai_memory_store::GrantLevel::Write,
+ None,
+ )
+ .await
+ .unwrap();
+ let foreign = finish_authority(
+ ai_memory_core::AuthLevel::User,
+ Some(foreign_user),
+ Some("foreign-recovery"),
+ );
+ assert!(matches!(
+ store
+ .writer
+ .recover_expired_workstream_run(foreign, finish_input(recoverable.run_id))
+ .await,
+ Err(StoreError::Forbidden(_))
+ ));
+ let owner = finish_authority(
+ ai_memory_core::AuthLevel::User,
+ Some(owner_user),
+ Some("new-user"),
+ );
+ assert_eq!(
+ store
+ .writer
+ .recover_expired_workstream_run(owner, finish_input(recoverable.run_id))
+ .await
+ .unwrap()
+ .imported_events,
+ 1
+ );
+
// Cancel and replacement both store the existing expired phase.
for replaced in [false, true] {
let old = store
@@ -2062,7 +2179,7 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {
[old.run_id.as_bytes()],
)
.unwrap();
- store
+ let replacement = store
.writer
.prepare_workstream_run(PrepareWorkstreamRun {
workspace_id: ws,
@@ -2078,6 +2195,11 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {
})
.await
.unwrap();
+ conn.execute(
+ "UPDATE managed_runs SET started_at = 7 WHERE id IN (?1, ?2)",
+ rusqlite::params![old.run_id.as_bytes(), replacement.run_id.as_bytes()],
+ )
+ .unwrap();
} else {
store.writer.cancel_managed_run(old.run_id).await.unwrap();
}
@@ -2089,7 +2211,29 @@ async fn owner_finish_store_topology_reload_and_phase_controls() {
.await,
Err(StoreError::InvalidState(_))
));
- assert_eq!(finish_snapshot(&store).await, before);
+ if replaced {
+ assert!(matches!(
+ store
+ .writer
+ .recover_expired_workstream_run(root.clone(), finish_input(old.run_id))
+ .await,
+ Err(StoreError::InvalidState(_))
+ ));
+ } else {
+ assert_eq!(
+ store
+ .writer
+ .recover_expired_workstream_run(root.clone(), finish_input(old.run_id))
+ .await
+ .unwrap()
+ .imported_events,
+ 0,
+ "the unsuperseded expired run is recoverable and stable event ids remain idempotent"
+ );
+ }
+ if replaced {
+ assert_eq!(finish_snapshot(&store).await, before);
+ }
}
let before = finish_snapshot(&store).await;
assert!(matches!(
diff --git a/crates/ai-memory-workstream/src/harness.rs b/crates/ai-memory-workstream/src/harness.rs
index 5244075b..4126d91c 100644
--- a/crates/ai-memory-workstream/src/harness.rs
+++ b/crates/ai-memory-workstream/src/harness.rs
@@ -113,6 +113,15 @@ fn valid_semver_identifiers(value: &str, reject_leading_zero_numeric: bool) -> b
})
}
+/// Native transcript support available to recovery and backfill.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum TranscriptCapability {
+ /// The native store can export a normalized visible-event transcript.
+ Export,
+ /// Recovery can only rely on lifecycle events retained in the hook spool.
+ SpoolOnly,
+}
+
/// Harnesses with native-session and transcript adapters.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ManagedHarness {
@@ -167,6 +176,15 @@ impl ManagedHarness {
}
}
+ /// Recovery capability of this harness's native session store.
+ #[must_use]
+ pub const fn transcript_capability(self) -> TranscriptCapability {
+ match self {
+ Self::Antigravity => TranscriptCapability::SpoolOnly,
+ _ => TranscriptCapability::Export,
+ }
+ }
+
/// Whether the harness has no native session-end hook, so its sessions
/// stay open until `ai-memory finalize-session` runs (docs/support-matrix.md).
#[must_use]
@@ -3437,6 +3455,36 @@ mod tests {
));
}
+ #[test]
+ fn transcript_capability_marks_only_antigravity_spool_only() {
+ for harness in [
+ ManagedHarness::Claude,
+ ManagedHarness::Codex,
+ ManagedHarness::OpenCode,
+ ManagedHarness::OpenCode2,
+ ManagedHarness::Pi,
+ ManagedHarness::Crush,
+ ManagedHarness::Omp,
+ ManagedHarness::Kimi,
+ ManagedHarness::CommandCode,
+ ManagedHarness::Kiro,
+ ManagedHarness::KiroV3,
+ ManagedHarness::Grok,
+ ManagedHarness::Antigravity,
+ ] {
+ assert_eq!(
+ harness.transcript_capability(),
+ if harness == ManagedHarness::Antigravity {
+ TranscriptCapability::SpoolOnly
+ } else {
+ TranscriptCapability::Export
+ },
+ "{}",
+ harness.as_str()
+ );
+ }
+ }
+
#[test]
fn only_harnesses_without_a_session_end_hook_need_finalizing() {
let hookless = [
diff --git a/crates/ai-memory-workstream/src/lib.rs b/crates/ai-memory-workstream/src/lib.rs
index e00ad78f..c55094e8 100644
--- a/crates/ai-memory-workstream/src/lib.rs
+++ b/crates/ai-memory-workstream/src/lib.rs
@@ -6,7 +6,7 @@ mod repository;
mod transcript;
pub use harness::{
- LaunchMode, LaunchPlan, LaunchRoots, ManagedHarness, OpenCodeDialect,
+ LaunchMode, LaunchPlan, LaunchRoots, ManagedHarness, OpenCodeDialect, TranscriptCapability,
allows_native_session_adoption, apply_claude_true_yolo, apply_yolo, build_launch_plan,
build_launch_plan_with_env, build_launch_plan_with_env_lookup, clean_path, crush_data_dir,
crush_global_config_path, env_dir_override, has_native_session_selector,
@@ -28,7 +28,8 @@ pub use transcript::{
AmbiguousNativeSession, ExportedTranscript, NativeSessionCandidate,
claude_attached_background_session, claude_live_background_attach_id,
claude_session_ran_in_background, discover_native_session, export_transcript,
- kiro_harness_from_source_cursor, kiro_v3_resume_uses_default_store, list_native_sessions,
- native_memory_dir, native_session_exists, native_session_in_checkout, native_store_root,
- wait_for_transcript_flush,
+ export_transcript_delta, export_transcript_range, kiro_harness_from_source_cursor,
+ kiro_v3_resume_uses_default_store, list_native_sessions, native_memory_dir,
+ native_session_exists, native_session_in_checkout, native_store_root, transcript_baseline,
+ transcript_interval_digests, wait_for_transcript_flush,
};
diff --git a/crates/ai-memory-workstream/src/transcript.rs b/crates/ai-memory-workstream/src/transcript.rs
index 20c9f03b..5319748c 100644
--- a/crates/ai-memory-workstream/src/transcript.rs
+++ b/crates/ai-memory-workstream/src/transcript.rs
@@ -11,7 +11,7 @@ use ai_memory_core::{
Sanitizer, WorkstreamEventKind,
};
use anyhow::{Context as _, Result, anyhow};
-use rusqlite::{Connection, OpenFlags, params};
+use rusqlite::{Connection, OpenFlags, OptionalExtension as _, params};
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use sha2::{Digest as _, Sha256};
@@ -21,6 +21,13 @@ use crate::{ManagedHarness, clean_path};
const MAX_SCAN_FILES: usize = 50_000;
const MAX_EVENT_BYTES: usize = 128 * 1024;
+const MAX_SESSION_HEADER_BYTES: usize = 128 * 1024;
+const MAX_SOURCE_RECORD_BYTES: usize = 1024 * 1024;
+const MAX_EXPORTED_EVENTS: usize = 4_096;
+const MAX_EXPORTED_BYTES: usize = 32 * 1024 * 1024;
+const MAX_SOURCE_RECORDS: usize = 32_768;
+const MAX_SOURCE_BYTES: usize = 64 * 1024 * 1024;
+const MAX_SESSION_METADATA_BYTES: u64 = 1024 * 1024;
const LEGACY_MANAGED_WORKSTREAM_PACKET_PREFIX: &str = "> **ai-memory managed workstream:";
/// Checkout-local native session that can seed an otherwise-empty workstream.
@@ -45,6 +52,27 @@ pub struct ExportedTranscript {
pub losses: Vec<String>,
}
+/// Ordered semantic identity for a bounded transcript interval.
+#[must_use]
+pub fn transcript_interval_digests(events: &[NewWorkstreamEvent]) -> Vec<String> {
+ events.iter().map(transcript_event_digest).collect()
+}
+
+fn transcript_event_digest(event: &NewWorkstreamEvent) -> String {
+ let mut hasher = Sha256::new();
+ for value in [
+ event.event_id.as_str(),
+ event.kind.as_str(),
+ event.role.as_deref().unwrap_or_default(),
+ event.occurred_at.as_deref().unwrap_or_default(),
+ event.content.as_str(),
+ ] {
+ hasher.update(value.len().to_be_bytes());
+ hasher.update(value.as_bytes());
+ }
+ format!("{:x}", hasher.finalize())
+}
+
#[derive(Debug, Clone, Serialize, Deserialize)]
struct FileCursor {
path: String,
@@ -113,29 +141,419 @@ pub async fn export_transcript(
native_session_id: &str,
source_cursor: Option<&str>,
) -> Result<ExportedTranscript> {
- if harness == ManagedHarness::OpenCode {
- return export_opencode(home, session_dir, native_session_id, source_cursor);
- }
- if harness == ManagedHarness::OpenCode2 {
- return export_opencode2(home, session_dir, native_session_id, source_cursor);
- }
- if harness == ManagedHarness::Crush {
- return export_crush(home, cwd, session_dir, native_session_id, source_cursor);
- }
- if harness == ManagedHarness::Antigravity {
- // The conversation store keeps every step as an undocumented protobuf
- // blob whose step-type enum is unversioned, so message text cannot be
- // decoded without guessing at a schema that changes between `agy`
- // releases. Conversation identity and workspace are read (they are
- // stable fields observed in current metadata); the visible-event
- // ledger for this harness comes from lifecycle-hook capture instead.
+ let home = home.to_path_buf();
+ let cwd = cwd.to_path_buf();
+ let session_dir = session_dir.map(Path::to_path_buf);
+ let native_session_id = native_session_id.to_owned();
+ let source_cursor = source_cursor.map(str::to_owned);
+ tokio::task::spawn_blocking(move || {
+ export_transcript_blocking(
+ harness,
+ &home,
+ &cwd,
+ session_dir.as_deref(),
+ &native_session_id,
+ source_cursor.as_deref(),
+ None,
+ )
+ })
+ .await
+ .map_err(|error| anyhow!("native transcript reader failed: {error}"))?
+}
+
+fn export_transcript_blocking(
+ harness: ManagedHarness,
+ home: &Path,
+ cwd: &Path,
+ session_dir: Option<&Path>,
+ native_session_id: &str,
+ source_cursor: Option<&str>,
+ final_cursor: Option<&str>,
+) -> Result<ExportedTranscript> {
+ let transcript = if harness == ManagedHarness::OpenCode {
+ export_opencode_range(
+ home,
+ session_dir,
+ native_session_id,
+ source_cursor,
+ final_cursor,
+ )?
+ } else if harness == ManagedHarness::OpenCode2 {
+ export_opencode2_range(
+ home,
+ session_dir,
+ native_session_id,
+ source_cursor,
+ final_cursor,
+ )?
+ } else if harness == ManagedHarness::Crush {
+ export_crush_range(
+ home,
+ cwd,
+ session_dir,
+ native_session_id,
+ source_cursor,
+ final_cursor,
+ )?
+ } else if harness == ManagedHarness::Antigravity {
return Err(anyhow!(
"antigravity conversations expose no decodable transcript; this session's events come from hook capture"
));
+ } else {
+ let path = locate_session_file(harness, home, cwd, session_dir, native_session_id)?
+ .ok_or_else(|| anyhow!("native transcript for {native_session_id} was not found"))?;
+ export_jsonl_range(
+ harness,
+ &path,
+ native_session_id,
+ source_cursor,
+ final_cursor,
+ )?
+ };
+ validate_export_bounds(transcript)
+}
+
+/// Export only records inside an exact previously captured cursor range.
+/// Both cursor identities are validated against the adapter/session, and rows
+/// or bytes appended after `final_cursor` are never read.
+pub async fn export_transcript_range(
+ harness: ManagedHarness,
+ home: &Path,
+ cwd: &Path,
+ session_dir: Option<&Path>,
+ native_session_id: &str,
+ source_cursor: Option<&str>,
+ final_cursor: &str,
+) -> Result<ExportedTranscript> {
+ let home = home.to_path_buf();
+ let cwd = cwd.to_path_buf();
+ let session_dir = session_dir.map(Path::to_path_buf);
+ let native_session_id = native_session_id.to_owned();
+ let source_cursor = source_cursor.map(str::to_owned);
+ let final_cursor = final_cursor.to_owned();
+ tokio::task::spawn_blocking(move || {
+ if let Some(source_cursor) = source_cursor.as_deref() {
+ validate_transcript_cursor(
+ harness,
+ &home,
+ &cwd,
+ session_dir.as_deref(),
+ &native_session_id,
+ source_cursor,
+ )?;
+ }
+ validate_transcript_cursor(
+ harness,
+ &home,
+ &cwd,
+ session_dir.as_deref(),
+ &native_session_id,
+ &final_cursor,
+ )?;
+ validate_cursor_range(harness, source_cursor.as_deref(), &final_cursor)?;
+ export_transcript_blocking(
+ harness,
+ &home,
+ &cwd,
+ session_dir.as_deref(),
+ &native_session_id,
+ source_cursor.as_deref(),
+ Some(&final_cursor),
+ )
+ })
+ .await
+ .map_err(|error| anyhow!("native transcript range reader failed: {error}"))?
+}
+
+/// Export only records after a cursor previously returned for this exact
+/// adapter/session. Invalid or stale cursors fail closed instead of replaying
+/// the complete transcript.
+pub async fn export_transcript_delta(
+ harness: ManagedHarness,
+ home: &Path,
+ cwd: &Path,
+ session_dir: Option<&Path>,
+ native_session_id: &str,
+ source_cursor: &str,
+) -> Result<ExportedTranscript> {
+ let home = home.to_path_buf();
+ let cwd = cwd.to_path_buf();
+ let session_dir = session_dir.map(Path::to_path_buf);
+ let native_session_id = native_session_id.to_owned();
+ let source_cursor = source_cursor.to_owned();
+ tokio::task::spawn_blocking(move || {
+ validate_transcript_cursor(
+ harness,
+ &home,
+ &cwd,
+ session_dir.as_deref(),
+ &native_session_id,
+ &source_cursor,
+ )?;
+ export_transcript_blocking(
+ harness,
+ &home,
+ &cwd,
+ session_dir.as_deref(),
+ &native_session_id,
+ Some(&source_cursor),
+ None,
+ )
+ })
+ .await
+ .map_err(|error| anyhow!("native transcript delta reader failed: {error}"))?
+}
+
+fn validate_cursor_range(
+ harness: ManagedHarness,
+ source_cursor: Option<&str>,
+ final_cursor: &str,
+) -> Result<()> {
+ if matches!(
+ harness,
+ ManagedHarness::OpenCode | ManagedHarness::OpenCode2 | ManagedHarness::Crush
+ ) {
+ let start = source_cursor
+ .map(serde_json::from_str::<SqlCursor>)
+ .transpose()
+ .context("invalid native transcript SQL cursor")?
+ .unwrap_or_default();
+ let end: SqlCursor = serde_json::from_str(final_cursor)
+ .context("invalid native transcript final SQL cursor")?;
+ if (end.updated, end.id.as_str()) < (start.updated, start.id.as_str()) {
+ return Err(anyhow!(
+ "native transcript final cursor precedes its start cursor"
+ ));
+ }
+ } else if let Some(source_cursor) = source_cursor {
+ let start: FileCursor =
+ serde_json::from_str(source_cursor).context("invalid native transcript file cursor")?;
+ let end: FileCursor = serde_json::from_str(final_cursor)
+ .context("invalid native transcript final file cursor")?;
+ if start.path != end.path || start.flavor != end.flavor || end.offset < start.offset {
+ return Err(anyhow!(
+ "native transcript final cursor precedes or changes its start cursor"
+ ));
+ }
+ }
+ Ok(())
+}
+
+fn validate_transcript_cursor(
+ harness: ManagedHarness,
+ home: &Path,
+ cwd: &Path,
+ session_dir: Option<&Path>,
+ native_session_id: &str,
+ source_cursor: &str,
+) -> Result<()> {
+ if matches!(
+ harness,
+ ManagedHarness::OpenCode | ManagedHarness::OpenCode2 | ManagedHarness::Crush
+ ) {
+ serde_json::from_str::<SqlCursor>(source_cursor)
+ .context("invalid native transcript SQL cursor")?;
+ return Ok(());
}
+ let cursor: FileCursor =
+ serde_json::from_str(source_cursor).context("invalid native transcript file cursor")?;
let path = locate_session_file(harness, home, cwd, session_dir, native_session_id)?
.ok_or_else(|| anyhow!("native transcript for {native_session_id} was not found"))?;
- export_jsonl(harness, &path, native_session_id, source_cursor)
+ if Path::new(&cursor.path) != path || cursor.flavor != file_flavor(harness) {
+ return Err(anyhow!(
+ "native transcript cursor does not match this adapter session"
+ ));
+ }
+ let len = fs::metadata(&path)?.len();
+ if cursor.offset > len {
+ return Err(anyhow!(
+ "native transcript cursor is past the current transcript end"
+ ));
+ }
+ if journal_rewrites_in_place(harness) {
+ let expected = cursor
+ .prefix_sha256
+ .as_deref()
+ .ok_or_else(|| anyhow!("native transcript cursor has no prefix identity"))?;
+ let mut file = File::open(path)?;
+ let actual = hash_file_prefix(&mut file, cursor.offset)?
+ .map(|hasher| format!("{:x}", hasher.finalize()))
+ .ok_or_else(|| anyhow!("native transcript cursor prefix is incomplete"))?;
+ if actual != expected {
+ return Err(anyhow!(
+ "native transcript changed before the recovery cursor"
+ ));
+ }
+ }
+ Ok(())
+}
+
+fn validate_export_bounds(transcript: ExportedTranscript) -> Result<ExportedTranscript> {
+ if transcript.events.len() > MAX_EXPORTED_EVENTS {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_EXPORTED_EVENTS} event recovery limit"
+ ));
+ }
+ let mut bytes = 0_usize;
+ for event in &transcript.events {
+ bytes = bytes.saturating_add(serde_json::to_vec(event)?.len());
+ if bytes > MAX_EXPORTED_BYTES {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_EXPORTED_BYTES} byte recovery limit"
+ ));
+ }
+ }
+ Ok(transcript)
+}
+
+fn enforce_parse_bounds(
+ events: &[NewWorkstreamEvent],
+ checked_events: &mut usize,
+ event_bytes: &mut usize,
+ source_bytes: &mut usize,
+ source_record_bytes: usize,
+) -> Result<()> {
+ *source_bytes = source_bytes.saturating_add(source_record_bytes);
+ if *source_bytes > MAX_SOURCE_BYTES {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_SOURCE_BYTES} source-byte limit"
+ ));
+ }
+ if events.len() > MAX_EXPORTED_EVENTS {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_EXPORTED_EVENTS} event recovery limit"
+ ));
+ }
+ for event in &events[*checked_events..] {
+ *event_bytes = event_bytes.saturating_add(serde_json::to_vec(event)?.len());
+ if *event_bytes > MAX_EXPORTED_BYTES {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_EXPORTED_BYTES} byte recovery limit"
+ ));
+ }
+ }
+ *checked_events = events.len();
+ Ok(())
+}
+
+/// Capture the exact adapter cursor at the current end of one native session.
+/// Recovery may use it only for that same session and adapter.
+pub async fn transcript_baseline(
+ harness: ManagedHarness,
+ home: &Path,
+ cwd: &Path,
+ session_dir: Option<&Path>,
+ native_session_id: &str,
+) -> Result<String> {
+ let home = home.to_path_buf();
+ let cwd = cwd.to_path_buf();
+ let session_dir = session_dir.map(Path::to_path_buf);
+ let native_session_id = native_session_id.to_owned();
+ tokio::task::spawn_blocking(move || {
+ transcript_baseline_blocking(
+ harness,
+ &home,
+ &cwd,
+ session_dir.as_deref(),
+ &native_session_id,
+ )
+ })
+ .await
+ .map_err(|error| anyhow!("native transcript baseline reader failed: {error}"))?
+}
+
+fn transcript_baseline_blocking(
+ harness: ManagedHarness,
+ home: &Path,
+ cwd: &Path,
+ session_dir: Option<&Path>,
+ native_session_id: &str,
+) -> Result<String> {
+ let sql_cursor = |db: PathBuf, sql: &str| -> Result<String> {
+ let connection = Connection::open_with_flags(
+ &db,
+ OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
+ )?;
+ let cursor = connection
+ .query_row(sql, params![native_session_id], |row| {
+ Ok(SqlCursor {
+ updated: row.get(0)?,
+ id: row.get(1)?,
+ })
+ })
+ .optional()?
+ .unwrap_or_default();
+ Ok(serde_json::to_string(&cursor)?)
+ };
+ match harness {
+ ManagedHarness::OpenCode => sql_cursor(
+ opencode_db(home, session_dir),
+ "SELECT p.time_updated, p.id FROM part p WHERE p.session_id = ?1 ORDER BY p.time_updated DESC, p.id DESC LIMIT 1",
+ ),
+ ManagedHarness::OpenCode2 => sql_cursor(
+ opencode_db(home, session_dir),
+ "SELECT time_updated, id FROM session_message WHERE session_id = ?1 ORDER BY time_updated DESC, id DESC LIMIT 1",
+ ),
+ ManagedHarness::Crush => sql_cursor(
+ crush_db(home, cwd, session_dir),
+ "SELECT updated_at, id FROM messages WHERE session_id = ?1 ORDER BY updated_at DESC, id DESC LIMIT 1",
+ ),
+ ManagedHarness::Antigravity => Err(anyhow!(
+ "{} does not expose a reliable transcript cursor",
+ harness.as_str()
+ )),
+ _ => {
+ let path = locate_session_file(harness, home, cwd, session_dir, native_session_id)?
+ .ok_or_else(|| {
+ anyhow!("native transcript for {native_session_id} was not found")
+ })?;
+ let mut file = File::open(&path)?;
+ let mut reader = BufReader::new(&mut file);
+ let mut offset = 0_u64;
+ let mut committed_offset = 0_u64;
+ let mut prefix_hasher = Sha256::new();
+ let mut line = Vec::new();
+ let mut records = 0_usize;
+ loop {
+ line.clear();
+ let read = std::io::Read::by_ref(&mut reader)
+ .take((MAX_SOURCE_RECORD_BYTES + 1) as u64)
+ .read_until(b'\n', &mut line)?;
+ if read == 0 {
+ break;
+ }
+ records += 1;
+ if records > MAX_SOURCE_RECORDS {
+ return Err(anyhow!(
+ "native transcript baseline exceeds the {MAX_SOURCE_RECORDS} source-record limit"
+ ));
+ }
+ if read > MAX_SOURCE_RECORD_BYTES {
+ return Err(anyhow!(
+ "native transcript baseline record exceeds the {MAX_SOURCE_RECORD_BYTES} byte limit"
+ ));
+ }
+ offset = offset.saturating_add(read as u64);
+ if offset > MAX_SOURCE_BYTES as u64 {
+ return Err(anyhow!(
+ "native transcript baseline exceeds the {MAX_SOURCE_BYTES} source-byte limit"
+ ));
+ }
+ if !line.ends_with(b"\n") {
+ break;
+ }
+ prefix_hasher.update(&line);
+ committed_offset = offset;
+ }
+ Ok(serde_json::to_string(&FileCursor {
+ path: path.to_string_lossy().into_owned(),
+ offset: committed_offset,
+ flavor: file_flavor(harness),
+ prefix_sha256: journal_rewrites_in_place(harness)
+ .then(|| format!("{:x}", prefix_hasher.finalize())),
+ })?)
+ }
+ }
}
/// New sessions appeared in a store that records no launch identity and none
@@ -582,11 +1000,22 @@ pub async fn wait_for_transcript_flush(
Ok(())
}
+#[cfg(test)]
fn export_jsonl(
harness: ManagedHarness,
path: &Path,
native_session_id: &str,
source_cursor: Option<&str>,
+) -> Result<ExportedTranscript> {
+ export_jsonl_range(harness, path, native_session_id, source_cursor, None)
+}
+
+fn export_jsonl_range(
+ harness: ManagedHarness,
+ path: &Path,
+ native_session_id: &str,
+ source_cursor: Option<&str>,
+ final_cursor: Option<&str>,
) -> Result<ExportedTranscript> {
let flavor = file_flavor(harness);
let cursor = source_cursor
@@ -594,9 +1023,29 @@ fn export_jsonl(
.filter(|cursor| {
Path::new(&cursor.path) == path && (cursor.flavor.is_none() || cursor.flavor == flavor)
});
+ let final_cursor = final_cursor
+ .map(serde_json::from_str::<FileCursor>)
+ .transpose()
+ .context("invalid native transcript final file cursor")?;
+ if final_cursor
+ .as_ref()
+ .is_some_and(|cursor| Path::new(&cursor.path) != path || cursor.flavor != flavor)
+ {
+ return Err(anyhow!(
+ "native transcript final cursor does not match this adapter session"
+ ));
+ }
let mut file = File::open(path)
.with_context(|| format!("opening native transcript {}", path.display()))?;
- let len = file.metadata()?.len();
+ let file_len = file.metadata()?.len();
+ let len = final_cursor
+ .as_ref()
+ .map_or(file_len, |cursor| cursor.offset);
+ if len > file_len {
+ return Err(anyhow!(
+ "native transcript final cursor is past the current transcript end"
+ ));
+ }
let (start, mut prefix_hasher) = if journal_rewrites_in_place(harness) {
let validated = if let Some(cursor) = cursor.as_ref().filter(|cursor| cursor.offset <= len)
&& let Some(expected) = cursor.prefix_sha256.as_deref()
@@ -621,12 +1070,41 @@ fn export_jsonl(
let mut line = Vec::new();
let mut events = Vec::new();
let mut losses = Vec::new();
+ let mut records = 0_usize;
+ let mut source_bytes = 0_usize;
+ let mut checked_events = 0_usize;
+ let mut event_bytes = 0_usize;
loop {
line.clear();
- let read = reader.read_until(b'\n', &mut line)?;
+ let remaining = len.saturating_sub(offset);
+ if remaining == 0 {
+ break;
+ }
+ let read_limit = remaining.min((MAX_SOURCE_RECORD_BYTES + 1) as u64);
+ let read = std::io::Read::by_ref(&mut reader)
+ .take(read_limit)
+ .read_until(b'\n', &mut line)?;
if read == 0 {
break;
}
+ records += 1;
+ if records > MAX_SOURCE_RECORDS {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_SOURCE_RECORDS} source-record limit"
+ ));
+ }
+ if read > MAX_SOURCE_RECORD_BYTES {
+ return Err(anyhow!(
+ "native transcript record exceeds the {MAX_SOURCE_RECORD_BYTES} byte limit"
+ ));
+ }
+ enforce_parse_bounds(
+ &events,
+ &mut checked_events,
+ &mut event_bytes,
+ &mut source_bytes,
+ read,
+ )?;
offset += read as u64;
if !line.ends_with(b"\n") {
break;
@@ -727,6 +1205,13 @@ fn export_jsonl(
));
}
}
+ enforce_parse_bounds(
+ &events,
+ &mut checked_events,
+ &mut event_bytes,
+ &mut source_bytes,
+ 0,
+ )?;
}
if harness == ManagedHarness::Kimi {
annotate_kimi_subagents(path, &mut losses);
@@ -737,8 +1222,13 @@ fn export_jsonl(
path: path.to_string_lossy().into_owned(),
offset: committed_offset,
flavor,
- prefix_sha256: journal_rewrites_in_place(harness)
- .then(|| format!("{:x}", prefix_hasher.finalize())),
+ prefix_sha256: if journal_rewrites_in_place(harness) {
+ final_cursor
+ .and_then(|cursor| cursor.prefix_sha256.clone())
+ .or_else(|| Some(format!("{:x}", prefix_hasher.finalize())))
+ } else {
+ None
+ },
})?),
events,
losses: deduplicate_losses(losses),
@@ -2126,11 +2616,22 @@ fn codex_synthetic_context(agent: AgentKind, role: &str, text: &str) -> bool {
}
}
+#[cfg(test)]
fn export_opencode(
home: &Path,
session_dir: Option<&Path>,
session: &str,
source_cursor: Option<&str>,
+) -> Result<ExportedTranscript> {
+ export_opencode_range(home, session_dir, session, source_cursor, None)
+}
+
+fn export_opencode_range(
+ home: &Path,
+ session_dir: Option<&Path>,
+ session: &str,
+ source_cursor: Option<&str>,
+ final_cursor: Option<&str>,
) -> Result<ExportedTranscript> {
let db = opencode_db(home, session_dir);
let connection = Connection::open_with_flags(
@@ -2146,25 +2647,57 @@ fn export_opencode(
let cursor = source_cursor
.and_then(|raw| serde_json::from_str::<SqlCursor>(raw).ok())
.unwrap_or_default();
+ let upper = final_cursor
+ .map(serde_json::from_str::<SqlCursor>)
+ .transpose()
+ .context("invalid native transcript final SQL cursor")?
+ .unwrap_or(SqlCursor {
+ updated: i64::MAX,
+ id: "\u{10ffff}".to_string(),
+ });
let mut statement = connection.prepare(
- "SELECT p.id, p.time_updated, m.data, p.data
+ "SELECT p.id, p.time_updated, length(m.data), length(p.data), m.data, p.data
FROM part p JOIN message m ON m.id = p.message_id
WHERE p.session_id = ?1 AND (p.time_updated > ?2 OR (p.time_updated = ?2 AND p.id > ?3))
- ORDER BY p.time_updated, p.id",
+ AND (p.time_updated < ?4 OR (p.time_updated = ?4 AND p.id <= ?5))
+ ORDER BY p.time_updated, p.id
+ LIMIT ?6",
)?;
- let rows = statement.query_map(params![session, cursor.updated, cursor.id], |row| {
- Ok((
- row.get::<_, String>(0)?,
- row.get::<_, i64>(1)?,
- row.get::<_, String>(2)?,
- row.get::<_, String>(3)?,
- ))
- })?;
+ let mut rows = statement.query(params![
+ session,
+ cursor.updated,
+ cursor.id,
+ upper.updated,
+ upper.id,
+ (MAX_SOURCE_RECORDS + 1) as i64
+ ])?;
let mut events = Vec::new();
let mut losses = Vec::new();
let mut next_cursor = cursor;
- for row in rows {
- let (id, updated, message_raw, part_raw) = row?;
+ let mut source_bytes = 0_usize;
+ let mut checked_events = 0_usize;
+ let mut event_bytes = 0_usize;
+ let mut records = 0_usize;
+ while let Some(row) = rows.next()? {
+ records += 1;
+ if records > MAX_SOURCE_RECORDS {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_SOURCE_RECORDS} source-record limit"
+ ));
+ }
+ let updated = row.get::<_, i64>(1)?;
+ let message_len = row.get::<_, usize>(2)?;
+ let part_len = row.get::<_, usize>(3)?;
+ enforce_parse_bounds(
+ &events,
+ &mut checked_events,
+ &mut event_bytes,
+ &mut source_bytes,
+ message_len.saturating_add(part_len),
+ )?;
+ let id = row.get::<_, String>(0)?;
+ let message_raw = row.get::<_, String>(4)?;
+ let part_raw = row.get::<_, String>(5)?;
next_cursor = SqlCursor {
updated,
id: id.clone(),
@@ -2178,6 +2711,13 @@ fn export_opencode(
continue;
};
parse_opencode(&message, &part, session, &id, &mut events, &mut losses);
+ enforce_parse_bounds(
+ &events,
+ &mut checked_events,
+ &mut event_bytes,
+ &mut source_bytes,
+ 0,
+ )?;
}
Ok(ExportedTranscript {
native_session_id: session.to_string(),
@@ -2187,12 +2727,24 @@ fn export_opencode(
})
}
+#[cfg(test)]
fn export_crush(
home: &Path,
cwd: &Path,
session_dir: Option<&Path>,
session: &str,
source_cursor: Option<&str>,
+) -> Result<ExportedTranscript> {
+ export_crush_range(home, cwd, session_dir, session, source_cursor, None)
+}
+
+fn export_crush_range(
+ home: &Path,
+ cwd: &Path,
+ session_dir: Option<&Path>,
+ session: &str,
+ source_cursor: Option<&str>,
+ final_cursor: Option<&str>,
) -> Result<ExportedTranscript> {
let db = crush_db(home, cwd, session_dir);
let connection = Connection::open_with_flags(
@@ -2203,27 +2755,57 @@ fn export_crush(
let cursor = source_cursor
.and_then(|raw| serde_json::from_str::<SqlCursor>(raw).ok())
.unwrap_or_default();
+ let upper = final_cursor
+ .map(serde_json::from_str::<SqlCursor>)
+ .transpose()
+ .context("invalid native transcript final SQL cursor")?
+ .unwrap_or(SqlCursor {
+ updated: i64::MAX,
+ id: "\u{10ffff}".to_string(),
+ });
let mut statement = connection.prepare(
- "SELECT id, role, parts, updated_at, is_summary_message \
+ "SELECT id, role, length(parts), parts, updated_at, is_summary_message \
FROM messages \
WHERE session_id = ?1 \
AND (updated_at > ?2 OR (updated_at = ?2 AND id > ?3)) \
- ORDER BY updated_at, id",
+ AND (updated_at < ?4 OR (updated_at = ?4 AND id <= ?5)) \
+ ORDER BY updated_at, id LIMIT ?6",
)?;
- let rows = statement.query_map(params![session, cursor.updated, cursor.id], |row| {
- Ok((
- row.get::<_, String>(0)?,
- row.get::<_, String>(1)?,
- row.get::<_, String>(2)?,
- row.get::<_, i64>(3)?,
- row.get::<_, i64>(4)?,
- ))
- })?;
+ let mut rows = statement.query(params![
+ session,
+ cursor.updated,
+ cursor.id,
+ upper.updated,
+ upper.id,
+ (MAX_SOURCE_RECORDS + 1) as i64
+ ])?;
let mut events = Vec::new();
let mut losses = Vec::new();
let mut next_cursor = cursor;
- for row in rows {
- let (id, role, parts_raw, updated, is_summary) = row?;
+ let mut source_bytes = 0_usize;
+ let mut checked_events = 0_usize;
+ let mut event_bytes = 0_usize;
+ let mut records = 0_usize;
+ while let Some(row) = rows.next()? {
+ records += 1;
+ if records > MAX_SOURCE_RECORDS {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_SOURCE_RECORDS} source-record limit"
+ ));
+ }
+ let parts_len = row.get::<_, usize>(2)?;
+ enforce_parse_bounds(
+ &events,
+ &mut checked_events,
+ &mut event_bytes,
+ &mut source_bytes,
+ parts_len,
+ )?;
+ let id = row.get::<_, String>(0)?;
+ let role = row.get::<_, String>(1)?;
+ let parts_raw = row.get::<_, String>(3)?;
+ let updated = row.get::<_, i64>(4)?;
+ let is_summary = row.get::<_, i64>(5)?;
next_cursor = SqlCursor {
updated,
id: id.clone(),
@@ -2245,6 +2827,13 @@ fn export_crush(
&mut events,
&mut losses,
);
+ enforce_parse_bounds(
+ &events,
+ &mut checked_events,
+ &mut event_bytes,
+ &mut source_bytes,
+ 0,
+ )?;
}
Ok(ExportedTranscript {
native_session_id: session.to_string(),
@@ -2643,6 +3232,25 @@ fn temporary_transcript(path: &Path) -> bool {
.is_some_and(|name| name.contains(".jsonl."))
}
+fn read_bounded_file(path: &Path, limit: u64) -> std::io::Result<Vec<u8>> {
+ let file = File::open(path)?;
+ if file.metadata()?.len() > limit {
+ return Err(std::io::Error::new(
+ std::io::ErrorKind::InvalidData,
+ "native transcript metadata exceeds its byte limit",
+ ));
+ }
+ let mut bytes = Vec::new();
+ file.take(limit.saturating_add(1)).read_to_end(&mut bytes)?;
+ if bytes.len() as u64 > limit {
+ return Err(std::io::Error::new(
+ std::io::ErrorKind::InvalidData,
+ "native transcript metadata exceeds its byte limit",
+ ));
+ }
+ Ok(bytes)
+}
+
fn session_header(harness: ManagedHarness, path: &Path) -> Result<Option<(String, PathBuf)>> {
if harness == ManagedHarness::Kimi {
return kimi_session_header(path);
@@ -2663,9 +3271,14 @@ fn session_header(harness: ManagedHarness, path: &Path) -> Result<Option<(String
let mut line = String::new();
for _ in 0..64 {
line.clear();
- if reader.read_line(&mut line)? == 0 {
+ let read = std::io::Read::take(&mut reader, (MAX_SESSION_HEADER_BYTES + 1) as u64)
+ .read_line(&mut line)?;
+ if read == 0 {
break;
}
+ if read > MAX_SESSION_HEADER_BYTES {
+ return Ok(None);
+ }
let Ok(value) = serde_json::from_str::<Value>(&line) else {
continue;
};
@@ -2769,10 +3382,11 @@ fn kimi_session_header(path: &Path) -> Result<Option<(String, PathBuf)>> {
let Some(session_dir) = path.ancestors().nth(3) else {
return Ok(None);
};
- let Ok(raw) = fs::read_to_string(session_dir.join("state.json")) else {
+ let Ok(raw) = read_bounded_file(&session_dir.join("state.json"), MAX_SESSION_METADATA_BYTES)
+ else {
return Ok(None);
};
- let Ok(state) = serde_json::from_str::<Value>(&raw) else {
+ let Ok(state) = serde_json::from_slice::<Value>(&raw) else {
return Ok(None);
};
let Some(id) = session_dir.file_name().and_then(|name| name.to_str()) else {
@@ -2815,10 +3429,11 @@ fn kiro_session_header(path: &Path) -> Result<Option<(String, PathBuf)>> {
if Uuid::parse_str(stem).is_err() {
return Ok(None);
}
- let Ok(raw) = fs::read_to_string(path.with_extension("json")) else {
+ let Ok(raw) = read_bounded_file(&path.with_extension("json"), MAX_SESSION_METADATA_BYTES)
+ else {
return Ok(None);
};
- let Ok(metadata) = serde_json::from_str::<Value>(&raw) else {
+ let Ok(metadata) = serde_json::from_slice::<Value>(&raw) else {
return Ok(None);
};
let Some(id) = metadata.get("session_id").and_then(Value::as_str) else {
@@ -2852,10 +3467,13 @@ fn kiro_v3_session_header(path: &Path, cwd: &Path) -> Result<Option<(String, Pat
if Uuid::parse_str(uuid).is_err() {
return Ok(None);
}
- let Ok(raw) = fs::read_to_string(session_dir.join("session.json")) else {
+ let Ok(raw) = read_bounded_file(
+ &session_dir.join("session.json"),
+ MAX_SESSION_METADATA_BYTES,
+ ) else {
return Ok(None);
};
- let Ok(metadata) = serde_json::from_str::<Value>(&raw) else {
+ let Ok(metadata) = serde_json::from_slice::<Value>(&raw) else {
return Ok(None);
};
if metadata.get("schemaVersion").and_then(Value::as_str) != Some("1.0.0")
@@ -2885,10 +3503,13 @@ fn grok_session_header(path: &Path) -> Result<Option<(String, PathBuf)>> {
let Some(session_dir) = path.parent() else {
return Ok(None);
};
- let Ok(raw) = fs::read_to_string(session_dir.join("summary.json")) else {
+ let Ok(raw) = read_bounded_file(
+ &session_dir.join("summary.json"),
+ MAX_SESSION_METADATA_BYTES,
+ ) else {
return Ok(None);
};
- let Ok(summary) = serde_json::from_str::<Value>(&raw) else {
+ let Ok(summary) = serde_json::from_slice::<Value>(&raw) else {
return Ok(None);
};
let info = summary.get("info").unwrap_or(&Value::Null);
@@ -2925,8 +3546,8 @@ fn antigravity_session_header(path: &Path) -> Result<Option<(String, PathBuf)>>
return Ok(None);
};
let blob = connection.query_row(
- "SELECT data FROM trajectory_metadata_blob WHERE id = 'main' LIMIT 1",
- [],
+ "SELECT data FROM trajectory_metadata_blob WHERE id = 'main' AND length(data) <= ?1 LIMIT 1",
+ params![MAX_SESSION_METADATA_BYTES as i64],
|row| row.get::<_, Vec<u8>>(0),
);
let Ok(blob) = blob else {
@@ -3324,11 +3945,22 @@ fn opencode_directories(cwd: &Path) -> (String, String) {
/// is encrypted and excluded like v1's hidden reasoning; anything else
/// unrecognized becomes a bounded loss, never a guess — the beta schema is
/// still changing.
+#[cfg(test)]
fn export_opencode2(
home: &Path,
session_dir: Option<&Path>,
session: &str,
source_cursor: Option<&str>,
+) -> Result<ExportedTranscript> {
+ export_opencode2_range(home, session_dir, session, source_cursor, None)
+}
+
+fn export_opencode2_range(
+ home: &Path,
+ session_dir: Option<&Path>,
+ session: &str,
+ source_cursor: Option<&str>,
+ final_cursor: Option<&str>,
) -> Result<ExportedTranscript> {
let db = opencode_db(home, session_dir);
let connection = Connection::open_with_flags(
@@ -3344,24 +3976,54 @@ fn export_opencode2(
let cursor = source_cursor
.and_then(|raw| serde_json::from_str::<SqlCursor>(raw).ok())
.unwrap_or_default();
+ let upper = final_cursor
+ .map(serde_json::from_str::<SqlCursor>)
+ .transpose()
+ .context("invalid native transcript final SQL cursor")?
+ .unwrap_or(SqlCursor {
+ updated: i64::MAX,
+ id: "\u{10ffff}".to_string(),
+ });
let mut statement = connection.prepare(
- "SELECT id, time_updated, type, data FROM session_message \
+ "SELECT id, time_updated, type, length(data), data FROM session_message \
WHERE session_id = ?1 AND (time_updated > ?2 OR (time_updated = ?2 AND id > ?3)) \
- ORDER BY time_updated, id",
+ AND (time_updated < ?4 OR (time_updated = ?4 AND id <= ?5)) \
+ ORDER BY time_updated, id LIMIT ?6",
)?;
- let rows = statement.query_map(params![session, cursor.updated, cursor.id], |row| {
- Ok((
- row.get::<_, String>(0)?,
- row.get::<_, i64>(1)?,
- row.get::<_, String>(2)?,
- row.get::<_, String>(3)?,
- ))
- })?;
+ let mut rows = statement.query(params![
+ session,
+ cursor.updated,
+ cursor.id,
+ upper.updated,
+ upper.id,
+ (MAX_SOURCE_RECORDS + 1) as i64
+ ])?;
let mut events = Vec::new();
let mut losses = Vec::new();
let mut next_cursor = cursor;
- for row in rows {
- let (id, updated, message_type, data_raw) = row?;
+ let mut source_bytes = 0_usize;
+ let mut checked_events = 0_usize;
+ let mut event_bytes = 0_usize;
+ let mut records = 0_usize;
+ while let Some(row) = rows.next()? {
+ records += 1;
+ if records > MAX_SOURCE_RECORDS {
+ return Err(anyhow!(
+ "native transcript exceeds the {MAX_SOURCE_RECORDS} source-record limit"
+ ));
+ }
+ let data_len = row.get::<_, usize>(3)?;
+ enforce_parse_bounds(
+ &events,
+ &mut checked_events,
+ &mut event_bytes,
+ &mut source_bytes,
+ data_len,
+ )?;
+ let id = row.get::<_, String>(0)?;
+ let updated = row.get::<_, i64>(1)?;
+ let message_type = row.get::<_, String>(2)?;
+ let data_raw = row.get::<_, String>(4)?;
next_cursor = SqlCursor {
updated,
id: id.clone(),
@@ -3371,6 +4033,13 @@ fn export_opencode2(
continue;
};
parse_opencode2(&message_type, &data, session, &id, &mut events, &mut losses);
+ enforce_parse_bounds(
+ &events,
+ &mut checked_events,
+ &mut event_bytes,
+ &mut source_bytes,
+ 0,
+ )?;
}
Ok(ExportedTranscript {
native_session_id: session.to_string(),
@@ -4069,6 +4738,294 @@ mod tests {
assert_eq!(second.events[0].metadata["parent_id"], "u1");
}
+ #[tokio::test]
+ async fn persisted_baseline_exports_only_offline_append_and_rejects_rewrite() {
+ let temp = tempfile::tempdir().unwrap();
+ let cwd = temp.path().join("repo");
+ fs::create_dir_all(&cwd).unwrap();
+ let session = "7c1d5698-204a-4c0f-ae9c-43db7fc4e41d";
+ let store = temp.path().join("command-code");
+ let bucket = store.join("bucket");
+ fs::create_dir_all(&bucket).unwrap();
+ let path = bucket.join(format!("{session}.jsonl"));
+ fs::write(
+ &path,
+ format!(
+ "{}\n{}\n",
+ json!({"type":"session","version":3,"id":session,"timestamp":"2026-08-07T17:00:00Z","cwd":cwd}),
+ json!({"type":"message","id":"before","timestamp":"2026-08-07T17:00:01Z","message":{"role":"user","content":[{"type":"text","text":"prior live history"}],"meta":{"source":"user"}}})
+ ),
+ )
+ .unwrap();
+ let baseline = transcript_baseline(
+ ManagedHarness::CommandCode,
+ temp.path(),
+ &cwd,
+ Some(&store),
+ session,
+ )
+ .await
+ .unwrap();
+ let mut file = fs::OpenOptions::new().append(true).open(&path).unwrap();
+ use std::io::Write as _;
+ writeln!(
+ file,
+ "{}",
+ json!({"type":"message","id":"after","timestamp":"2026-08-07T17:00:02Z","message":{"role":"assistant","content":[{"type":"text","text":"offline append"}],"meta":{"source":"model"}}})
+ )
+ .unwrap();
+ let delta = export_transcript_delta(
+ ManagedHarness::CommandCode,
+ temp.path(),
+ &cwd,
+ Some(&store),
+ session,
+ &baseline,
+ )
+ .await
+ .unwrap();
+ assert_eq!(delta.events.len(), 1);
+ assert_eq!(delta.events[0].content, "offline append");
+ let final_cursor = delta.source_cursor.clone().unwrap();
+ writeln!(
+ file,
+ "{}",
+ json!({"type":"message","id":"later","timestamp":"2026-08-07T17:00:03Z","message":{"role":"user","content":[{"type":"text","text":"later resumed work"}],"meta":{"source":"user"}}})
+ )
+ .unwrap();
+ let exact = export_transcript_range(
+ ManagedHarness::CommandCode,
+ temp.path(),
+ &cwd,
+ Some(&store),
+ session,
+ Some(&baseline),
+ &final_cursor,
+ )
+ .await
+ .unwrap();
+ assert_eq!(exact.events.len(), 1);
+ assert_eq!(exact.events[0].content, "offline append");
+
+ fs::write(&path, b"{}\n").unwrap();
+ let error = export_transcript_delta(
+ ManagedHarness::CommandCode,
+ temp.path(),
+ &cwd,
+ Some(&store),
+ session,
+ &baseline,
+ )
+ .await
+ .unwrap_err();
+ assert!(
+ error
+ .to_string()
+ .contains("past the current transcript end")
+ || error.to_string().contains("was not found"),
+ "{error:#}"
+ );
+ }
+
+ #[tokio::test]
+ async fn transcript_baseline_refuses_oversized_records_before_unbounded_growth() {
+ let temp = tempfile::tempdir().unwrap();
+ let cwd = temp.path().join("repo");
+ fs::create_dir_all(&cwd).unwrap();
+ let session = "7c1d5698-204a-4c0f-ae9c-43db7fc4e41f";
+ let store = temp.path().join("command-code");
+ let bucket = store.join("bucket");
+ fs::create_dir_all(&bucket).unwrap();
+ let path = bucket.join(format!("{session}.jsonl"));
+ let mut bytes = format!(
+ "{}\n",
+ json!({"type":"session","version":3,"id":session,"timestamp":"2026-08-07T17:00:00Z","cwd":cwd})
+ )
+ .into_bytes();
+ bytes.extend(std::iter::repeat_n(b'x', MAX_SOURCE_RECORD_BYTES + 2));
+ fs::write(&path, bytes).unwrap();
+ let error = transcript_baseline(
+ ManagedHarness::CommandCode,
+ temp.path(),
+ &cwd,
+ Some(&store),
+ session,
+ )
+ .await
+ .unwrap_err();
+ assert!(error.to_string().contains("record exceeds"), "{error:#}");
+ }
+
+ /// An oversized FIRST JSONL record is rejected by the byte caps before any
+ /// unbounded allocation: the session-listing paths every degraded
+ /// pre/post snapshot uses skip the transcript (a valid header on a later
+ /// line is deliberately never reached) instead of reading the record.
+ #[tokio::test]
+ async fn an_oversized_first_record_is_skipped_by_session_listing_caps() {
+ let temp = tempfile::tempdir().unwrap();
+ let cwd = temp.path().join("repo");
+ fs::create_dir_all(&cwd).unwrap();
+ let session = "9c8d7a06-93e4-4a01-b14e-64f2f1f2b3c4";
+ let transcripts = temp
+ .path()
+ .join(".claude/projects")
+ .join(cwd.to_string_lossy().replace('/', "-"));
+ fs::create_dir_all(&transcripts).unwrap();
+ let claude_path = transcripts.join(format!("{session}.jsonl"));
+ let oversized_header = json!({
+ "sessionId": session,
+ "cwd": cwd,
+ "padding": "x".repeat(MAX_SOURCE_RECORD_BYTES + 2),
+ });
+ let valid_header = json!({"sessionId": session, "cwd": cwd});
+ {
+ use std::io::Write as _;
+ let mut file = fs::File::create(&claude_path).unwrap();
+ writeln!(file, "{oversized_header}").unwrap();
+ writeln!(file, "{valid_header}").unwrap();
+ }
+ assert!(
+ list_native_sessions(ManagedHarness::Claude, temp.path(), &cwd, None, 10)
+ .await
+ .unwrap()
+ .is_empty(),
+ "a transcript whose first record exceeds the header cap is skipped, header or not"
+ );
+
+ let store = temp.path().join("command-code");
+ let bucket = store.join("bucket");
+ fs::create_dir_all(&bucket).unwrap();
+ let command_code_path = bucket.join(format!("{session}.jsonl"));
+ {
+ use std::io::Write as _;
+ let mut file = fs::File::create(&command_code_path).unwrap();
+ writeln!(
+ file,
+ "{}",
+ json!({
+ "type":"session","version":3,"id":session,
+ "timestamp":"2026-08-07T17:00:00Z","cwd":cwd,
+ "padding":"y".repeat(MAX_EVENT_BYTES + 2),
+ })
+ )
+ .unwrap();
+ }
+ assert!(
+ list_native_sessions(
+ ManagedHarness::CommandCode,
+ temp.path(),
+ &cwd,
+ Some(&store),
+ 10
+ )
+ .await
+ .unwrap()
+ .is_empty(),
+ "the tighter first-line cap rejects before the source-record cap is even reached"
+ );
+ }
+
+ #[tokio::test]
+ async fn exact_empty_range_stays_empty_after_later_append() {
+ let temp = tempfile::tempdir().unwrap();
+ let cwd = temp.path().join("repo");
+ fs::create_dir_all(&cwd).unwrap();
+ let session = "7c1d5698-204a-4c0f-ae9c-43db7fc4e41e";
+ let store = temp.path().join("command-code");
+ let bucket = store.join("bucket");
+ fs::create_dir_all(&bucket).unwrap();
+ let path = bucket.join(format!("{session}.jsonl"));
+ fs::write(
+ &path,
+ format!(
+ "{}\n",
+ json!({"type":"session","version":3,"id":session,"timestamp":"2026-08-07T17:00:00Z","cwd":cwd})
+ ),
+ )
+ .unwrap();
+ let baseline = transcript_baseline(
+ ManagedHarness::CommandCode,
+ temp.path(),
+ &cwd,
+ Some(&store),
+ session,
+ )
+ .await
+ .unwrap();
+ let final_cursor = baseline.clone();
+ use std::io::Write as _;
+ writeln!(
+ fs::OpenOptions::new().append(true).open(&path).unwrap(),
+ "{}",
+ json!({"type":"message","id":"later","message":{"role":"user","content":[{"type":"text","text":"later resumed work"}]}})
+ )
+ .unwrap();
+ let exact = export_transcript_range(
+ ManagedHarness::CommandCode,
+ temp.path(),
+ &cwd,
+ Some(&store),
+ session,
+ Some(&baseline),
+ &final_cursor,
+ )
+ .await
+ .unwrap();
+ assert!(exact.events.is_empty());
+ }
+
+ #[test]
+ fn jsonl_scan_stops_at_source_byte_cap() {
+ let temp = tempfile::tempdir().unwrap();
+ let path = temp.path().join("oversized.jsonl");
+ let record = format!(
+ "{}\n",
+ json!({"type":"system","padding":"x".repeat(MAX_SOURCE_RECORD_BYTES / 2)})
+ );
+ let mut file = fs::File::create(&path).unwrap();
+ use std::io::Write as _;
+ for _ in 0..(MAX_SOURCE_BYTES / record.len() + 2) {
+ file.write_all(record.as_bytes()).unwrap();
+ }
+ let error = export_jsonl(ManagedHarness::CommandCode, &path, "session", None).unwrap_err();
+ assert!(error.to_string().contains("source-byte limit"), "{error:#}");
+ }
+
+ #[test]
+ fn exported_transcript_caps_count_and_serialized_bytes() {
+ let event = NewWorkstreamEvent {
+ event_id: "event".into(),
+ agent: AgentKind::ClaudeCode,
+ native_session_id: "session".into(),
+ source_record_id: None,
+ kind: WorkstreamEventKind::Message,
+ role: Some("user".into()),
+ content: "x".repeat(MAX_EVENT_BYTES),
+ occurred_at: None,
+ metadata: Value::Null,
+ };
+ let too_many = ExportedTranscript {
+ events: vec![event.clone(); MAX_EXPORTED_EVENTS + 1],
+ ..ExportedTranscript::default()
+ };
+ assert!(
+ validate_export_bounds(too_many)
+ .unwrap_err()
+ .to_string()
+ .contains("event recovery limit")
+ );
+ let too_large = ExportedTranscript {
+ events: vec![event; MAX_EXPORTED_BYTES / MAX_EVENT_BYTES + 1],
+ ..ExportedTranscript::default()
+ };
+ assert!(
+ validate_export_bounds(too_large)
+ .unwrap_err()
+ .to_string()
+ .contains("byte recovery limit")
+ );
+ }
+
#[tokio::test]
async fn opencode_candidate_discovery_is_newest_first_and_checkout_scoped() {
let temp = tempfile::tempdir().unwrap();
@@ -5012,6 +5969,36 @@ mod tests {
assert!(second.losses.is_empty());
}
+ #[test]
+ fn opencode_sql_scan_stops_at_source_byte_cap_before_json_parse() {
+ let home = tempfile::tempdir().unwrap();
+ let db = opencode_db(home.path(), None);
+ fs::create_dir_all(db.parent().unwrap()).unwrap();
+ let connection = Connection::open(&db).unwrap();
+ connection
+ .execute_batch(
+ "CREATE TABLE message(id TEXT PRIMARY KEY, session_id TEXT, data TEXT);\
+ CREATE TABLE part(id TEXT PRIMARY KEY, message_id TEXT, session_id TEXT, \
+ time_updated INTEGER, data TEXT);",
+ )
+ .unwrap();
+ connection
+ .execute(
+ "INSERT INTO message VALUES ('m1', 's1', ?1)",
+ [json!({"role":"user"}).to_string()],
+ )
+ .unwrap();
+ let oversized = "x".repeat(MAX_SOURCE_BYTES + 1);
+ connection
+ .execute(
+ "INSERT INTO part VALUES ('p1', 'm1', 's1', 1, ?1)",
+ [oversized],
+ )
+ .unwrap();
+ let error = export_opencode(home.path(), None, "s1", None).unwrap_err();
+ assert!(error.to_string().contains("source-byte limit"), "{error:#}");
+ }
+
#[test]
fn opencode2_updated_tracks_session_v2_rows() {
let home = tempfile::tempdir().unwrap();
@@ -5883,6 +6870,43 @@ mod tests {
/// Every step payload is an undocumented protobuf blob, so the ledger for
/// this harness comes from hook capture. The failure has to say so.
+ #[test]
+ fn transcript_interval_identity_is_ordered_and_semantic() {
+ let event = |id: &str, role: &str, timestamp: &str, content: &str| NewWorkstreamEvent {
+ event_id: id.into(),
+ agent: AgentKind::ClaudeCode,
+ native_session_id: "session".into(),
+ source_record_id: None,
+ kind: WorkstreamEventKind::Message,
+ role: Some(role.into()),
+ content: content.into(),
+ occurred_at: Some(timestamp.into()),
+ metadata: json!({}),
+ };
+ let first = event("one", "user", "2026-10-07T10:00:00Z", "hello");
+ let second = event("two", "assistant", "2026-10-07T10:00:01Z", "world");
+ let expected = transcript_interval_digests(&[first.clone(), second.clone()]);
+ assert_eq!(
+ transcript_interval_digests(&[first.clone(), second.clone()]),
+ expected
+ );
+ assert_ne!(
+ transcript_interval_digests(&[second.clone(), first.clone()]),
+ expected,
+ "row order is part of interval identity"
+ );
+ for changed in [
+ event("one", "assistant", "2026-10-07T10:00:00Z", "hello"),
+ event("one", "user", "2026-10-07T10:00:02Z", "hello"),
+ event("one", "user", "2026-10-07T10:00:00Z", "changed"),
+ ] {
+ assert_ne!(
+ transcript_interval_digests(&[changed, second.clone()]),
+ expected
+ );
+ }
+ }
+
#[tokio::test]
async fn antigravity_transcript_export_explains_why_it_is_unavailable() {
let temp = tempfile::TempDir::new().unwrap();
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 65a664af..9664eb36 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -68,7 +68,12 @@ from hook paths.
2. Server's hook router sanitises the payload (the only path from
untrusted text into the store), assigns an [`ObservationKind`], and
enqueues a `WriteCmd` to the writer actor. For native keyed events, the
- project-scoped key and observation commit together. The key is marked
+ project-scoped key and observation commit together. Ordinary keys expire
+ after 30 days; validated `recovery_<55 hex>` keys are bound to the actual
+ session, agent, event kind, and sanitized event identity and retained until
+ session/project deletion, so a writer cannot preclaim another recovery
+ event and an old journal cannot duplicate observations after ordinary
+ cleanup. The key is marked
complete only after downstream processing: an incomplete replay resumes
wiki/handoff effects without another observation, while a completed replay
is acknowledged and skipped. A bounded per-project/key gate serializes an
@@ -268,8 +273,15 @@ death remains bounded by the renewable lease expiry. An explicit
`--force-unlock` recovery expires and replaces a selected active lease in the
same writer transaction, but only when its durable operator attribution equals
the new run's attribution; the informational `host:pid` lease label is never an
-authorization key. The old run can no longer heartbeat or finish. See [Managed
-cross-harness workstreams](managed-workstreams.md).
+authorization key. The old run can no longer heartbeat or finish normally.
+`POST /workstream/runs/{run_id}/recover/finish` is the narrower outage path: it
+reuses the normal owner and project-write checks and admits an expired run only
+when no newer run in that workstream superseded it. A finished retry validates
+only durable transcript/native/cursor/exit identity and does not regenerate a
+repository checkpoint. Degraded resumed sessions persist a validated native
+source cursor before launch, and bounded recovery export reads only the delta
+on a blocking worker. See [Managed cross-harness
+workstreams](managed-workstreams.md).
## Hook event vocabulary
@@ -674,17 +686,18 @@ hook install-mcp commit
checkpoints restore-page llm-test
forget-sweep lint curator
auto-improve-report auto-improve finalize-session
-pending-writes embed generate-auth-token
-setup-agent bootstrap install-instructions
-install-skills reorg purge-project
-rename-project move-project move-session
-uninstall upgrade auth
-user completions handoffs
-purge-session compact api-key
-export-okf message doctor
-backfill project reclaim-ledger-versions
-repair-backfill-timestamps server backup-agents
-restore-agents profile
+recover pending-writes embed
+generate-auth-token setup-agent bootstrap
+install-instructions install-skills reorg
+purge-project rename-project move-project
+move-session uninstall upgrade
+auth user completions
+handoffs purge-session compact
+api-key export-okf message
+doctor backfill project
+reclaim-ledger-versions repair-backfill-timestamps
+server backup-agents restore-agents
+profile
```
Run `ai-memory --help` for the full tree.
diff --git a/docs/cookbook.md b/docs/cookbook.md
index 4efa173b..d5c0551c 100644
--- a/docs/cookbook.md
+++ b/docs/cookbook.md
@@ -387,6 +387,7 @@ ai-memory resume --search auth # pick a matching workstream in this checko
ai-memory resume --all # pick from every linked checkout on this machine
ai-memory workstreams # list this checkout's managed workstreams
ai-memory status # counts, paths, health
+ai-memory recover # replay outage spool/transcripts safely
ai-memory list-projects # every workspace/project the server knows about
ai-memory doctor # is every harness that ran here captured?
ai-memory backfill # import prior local history into an empty store
@@ -406,9 +407,23 @@ ai-memory serve # run the server
automatically when the server returns), an existing MCP registration
degrades to no-recall for the session, and the child's exit code is
returned. What you lose for that run is the workstream lease/context,
- transcript import, and handoff delivery; sessions resume only through an
- explicit native selector because no lease means no mutual exclusion. See
- [Degraded offline
+ immediate transcript import, and handoff delivery; sessions resume only
+ through an explicit native selector because no lease means no mutual
+ exclusion. An explicit resume is journaled with its exact pre-launch native
+ cursor and recovery imports only the appended delta; a cursor that cannot be
+ proven requires manual repair. A fresh run is journaled only when one new
+ native session can be correlated; ambiguity prints a manual backfill action.
+ Once the server returns, run `ai-memory recover`: for transcript-capable
+ harnesses it quarantines correlated spool entries, drains unrelated events,
+ and replays the bounded exact transcript through sanitized `/hook/batch`;
+ Antigravity has no supported transcript exporter, so its degraded run is
+ journaled as spool-only and clears only after correlated hook evidence has a
+ durable-delivery confirmation; recovery never calls its exporter. Empty or
+ previously correlated-loss spool state stays journaled for manual repair. Owner-authorized recovery
+ finishes an expired original managed run only when it was not superseded. A full journal refuses
+ new entries instead of evicting old work, and dropped spool events make the
+ command fail while retaining every journal entry for a later clean pass.
+ Re-running is safe. See [Degraded offline
launches](managed-workstreams.md#degraded-offline-launches). To fail instead
of degrading, pass `--require-server` (or set `run.require_server = true` /
`AI_MEMORY_RUN_REQUIRE_SERVER=true`). For a planned outage, consider
diff --git a/docs/external-lifecycle.md b/docs/external-lifecycle.md
index bb359823..9967ce33 100644
--- a/docs/external-lifecycle.md
+++ b/docs/external-lifecycle.md
@@ -130,7 +130,10 @@ to unkeyed ingestion, so validate them before sending.
The server claims the key and observation in one transaction. A completed replay
is skipped; a pending replay resumes downstream work without inserting another
-observation. Keys expire after **30 days**. This is bounded retry protection, not
+observation. Ordinary client keys expire after **30 days**. The reserved,
+server-validated `recovery_<55 hex>` namespace is used only by native transcript
+recovery/backfill and remains bound to its actual stored session until that
+session or project is deleted. This is bounded retry protection, not
permanent deduplication or an exactly-once guarantee for every wiki side effect.
### Ordered batches and recovery
diff --git a/docs/managed-workstreams.md b/docs/managed-workstreams.md
index 6394349e..b9151a6a 100644
--- a/docs/managed-workstreams.md
+++ b/docs/managed-workstreams.md
@@ -512,21 +512,43 @@ the server first (any HTTP answer counts as reachable, so an older build
without `/healthz` still passes), prints one loud warning naming the server
URL and what the degraded run means, and launches the harness anyway:
-- **Lost this run**: no workstream lease or cross-harness context packet, no
- transcript import into the ledger, and no cross-harness handoff delivery.
- Nothing that needs the server runs: no prepare, lease, link, heartbeat,
- context fetch, status check, finish, or import.
-- **Still works**: the harness's ai-memory lifecycle hooks (if installed) keep
- capturing — events spool locally and drain automatically when the server
- returns. An existing MCP registration degrades to no-recall for the session
- rather than blocking it (the same principle as `docs/mcp-install.md`'s
- optional-mode entries: an unreachable memory server costs you recall, not
- the session).
+- **Unavailable during the run**: no workstream lease or cross-harness context
+ packet, immediate transcript import, or handoff delivery. Nothing that needs
+ the server runs: no prepare, lease, link, heartbeat, context fetch, status
+ check, finish, or import.
+- **Recovered afterward**: recovery requires both the `ai-memory run` client
+ and server to be upgraded to the release that provides `ai-memory recover`
+ and modern per-item hook acknowledgements. The harness's ai-memory lifecycle
+ hooks (if installed) spool events locally. After the child exits, the launcher
+ discovers its native session and appends only bounded metadata (never
+ transcript content or secrets) to an owner-only local recovery journal. Run
+ `ai-memory recover` when the server returns: under the spool's exclusive
+ drain lock it first quarantines entries correlated to each journaled session,
+ drains unrelated entries, then re-exports the exact native transcript and
+ replays session start, content, and session end through the existing
+ sanitized `/hook/batch` backfill path. A successful replay deletes the
+ quarantine; failure restores it. Antigravity's native store has no supported
+ transcript exporter, so its degraded launch is still journaled explicitly as
+ spool-only and recovery never invokes its exporter. Its correlated entries
+ are quarantined and the journal clears as `recovered-via-spool` only after
+ their durable delivery was confirmed.
+ An empty spool, a correlated prior drop, or a launch with no hook evidence
+ stays journaled with a manual-repair status. Stable recovery ingest keys retain the
+ exact session, agent, event kind, and sanitized event identity until
+ session/project deletion; an ordinary writer cannot preclaim one for another
+ event. Ordinary hook keys still expire after 30 days. An existing MCP
+ registration still degrades to no-recall during the outage rather than
+ blocking the session.
- **Sessions**: because no lease exists there is no mutual exclusion against
another launcher in the same checkout, so a degraded launch never adopts or
- resumes a session implicitly. An explicit native session selector
- (`claude --resume <id>`, `codex continue`, …) still resumes — you named the
- session — and everything else starts a fresh session.
+ resumes a session implicitly. An explicit native session selector still
+ resumes the named session; before such a resume the launcher persists the
+ native adapter's exact pre-launch cursor, and recovery exports only records
+ after it. If that cursor cannot be proven, the launch remains usable but is
+ not journaled for automatic replay. Otherwise the launcher snapshots
+ checkout-local candidates before spawn and journals only when exactly one new
+ session appears. Ambiguous concurrent launches are not journaled and print
+ the manual `backfill --force --session <id>` recovery action.
- **Auto-wire**: the first-launch hook install still happens (it is local and
idempotent, and capture must spool offline), but no MCP entry is registered
for the now-unreachable server and no completion sentinel is written, so the
@@ -539,17 +561,39 @@ URL and what the degraded run means, and launches the harness anyway:
not an error.
When the child exits, the launcher prints that the run was not recorded on the
-server and how many hook events remain spooled locally (oldest first by age).
-Spooled events are bounded: they are dropped after the configured number of
-failed drain passes (8 by default), 7 days of age, or the 10,000-file spool
-cap. For a planned outage set `AI_MEMORY_HOOK_SPOOL_MAX_ATTEMPTS=0` to disable
-only the attempt-count drops (see `docs/install.md`).
+server, how many hook events remain spooled locally, and that the run was
+journaled for `ai-memory recover`. The journal is capped at 256 entries;
+once full it refuses new entries with an actionable warning instead of evicting
+unrecovered work. It is written atomically as owner-only local state; Unix lock opens use
+`O_NOFOLLOW` and the atomic replacement rechecks the destination immediately
+before rename. Windows applies an owner-only ACL and rejects observed symlinks,
+but reparse-point race hardening still requires the PR `windows` CI lane before
+merge. Spooled events remain separately
+bounded: they are dropped after the configured number of failed drain passes
+(8 by default), 7 days of age, or the 10,000-file spool cap. For a planned
+outage set `AI_MEMORY_HOOK_SPOOL_MAX_ATTEMPTS=0` to disable only the
+attempt-count drops (see `docs/install.md`).
If the server dies *during* a run — after the lease was acquired — the exit
-code is still preserved: the unimported transcript is reported as a warning
-with the exact `ai-memory finalize-session` command that repairs the record
-once the server is back, and the orphaned lease expires on its own within 90
-seconds.
+code is still preserved. The journal records the original run id, native
+session locator, absolute store path, and exit code; repository branch and path
+state are not persisted. Recovery retries validate the exact ordered semantic
+transcript interval and native identity, and omit a regenerated checkpoint, so
+repository changes during downtime are not claimed as the failed run's state and
+a successful finish cannot create a second checkpoint event.
+`ai-memory recover` uses the owner-authorized `/recover/finish` route to resume
+an expired original run only when no newer run superseded it; ordering is the
+explicit `(started_at, SQLite insertion order)` tuple, so timestamp ties cannot
+revive an older run. Another operator, a superseded run, and any finished-run
+payload that is not an exact replay of already indexed events/cursor/exit state
+remain refused.
+Successful entries are removed, while failed entries remain with a bounded
+sanitized reason. Each journal request and the whole journal pass have time
+budgets. A spool pass that drops anything is reported incomplete and keeps all
+journal entries; correlated entries are quarantined before that pass and can
+only be deleted after a later clean pass and exact transcript replay. If the
+server is still unreachable, `recover` exits successfully
+and prints the remaining spool/journal counts and startup guidance.
To restore the strict behavior (fail with the usual "could not reach …"
diagnosis and never start the agent), pass `--require-server`, set
diff --git a/docs/security-boundaries.md b/docs/security-boundaries.md
index 3f43e2d6..416a330b 100644
--- a/docs/security-boundaries.md
+++ b/docs/security-boundaries.md
@@ -72,6 +72,7 @@ boundary not yet built.
| 11f | Native hook executable-path integrity after self-replacement (#1027) | `ai-memory-cli/src/commands/render_shared.rs` `hook_embedded_exe_path` is the single executable resolver for every persisted native hook form (POSIX/Windows command, Windows exec, Zero, ZCode, OpenCode 2 assistant capture): it prefers an existing literal path, strips Linux's trailing ` (deleted)` procfs marker only when the stripped installed binary exists, and otherwise falls back to the platform command name. Upgrade keeps the already-loaded config and refreshes in-process | `render_shared.rs` `embedded_exe_strips_the_marker_when_the_stripped_path_exists`, `embedded_exe_preserves_a_real_file_whose_name_has_the_marker_suffix`, `embedded_exe_falls_back_when_the_resolved_path_is_missing`, `hook_embedded_exe_path_names_an_existing_file_without_marker`; `upgrade_e2e.rs` `native_upgrade_refreshes_hooks_without_a_deleted_path_or_losing_cli_data_dir` reproduces the replacement window through the shipped binary, plants the corrupt suffix, and proves refresh removes it while preserving a CLI-only data dir | STRONG on Linux reproduction plus platform-neutral resolver/render coverage; Windows path rendering remains covered by its existing command/exec tests and full Windows CI |
| 11g | Cross-project file attribution cannot bypass destination capture or server routing (#932) | `ai-memory-hooks/src/capture_policy.rs` `absolute_file_tool_paths` accepts only bounded, recognized, absolute file-tool candidates; `ai-memory-cli/src/commands/hook.rs` `file_tool_routing_cwd` requires every path to prove the same repository/marker boundary and resolves that destination before capture policy, server profile, scope hints, or spool credentials. Relative, mixed-project, unsupported, and arbitrary non-project paths keep the payload cwd | `hook.rs` `absolute_file_target_uses_destination_scope_and_capture_policy` (hostile content under the destination exclusion is never spooled; a public control is routed there), `absolute_file_target_cannot_escape_a_destination_server_profile` (unregistered destination profile drops instead of falling back to source/install default), `file_target_routing_refuses_relative_external_and_mixed_project_paths` | STRONG for native `ai-memory hook`; generated and compatibility hooks retain cwd-only routing |
| 11h | Docker-wrapper capture-policy capability (#1002) | `bin/ai-memory` routes ordinary Linux/macOS `install-hooks` and upgrade refreshes through the stable checksum-verified native host client, so the hook process executes the capture-policy boundary in 11b before spool/network. Only an explicit `AI_MEMORY_HOOK_PLATFORM=posix|windows` retains the documented script compatibility path and its warning; `AI_MEMORY_NATIVE_BIN` may not point back to the wrapper | `scripts/check-native-packaging.sh` `Checking host-launch wrapper routing`: a default allowlist install reaches the fake native client while a Docker executable that always fails proves no container fallback occurred; the explicit `posix` control reaches fake Docker and adds no native invocation; the self-referential native override is refused | STRONG on the shipped Linux/macOS wrapper. Windows and remote/manual script installers remain explicitly outside capture-policy v1 |
+| 11j | Offline recovery journal and transcript replay | `ai-memory-cli/src/commands/recovery.rs` keeps bounded locators, ordered semantic transcript digests, pre-launch cursors and persistent correlated-delivery/loss evidence in an atomic journal, refuses symlinked path components, sanitizes/caps failure reasons, and bounds each replay plus the whole pass; `hook_spool` coordinates enqueue and quarantine with short filesystem locks, canonically resolves partial scope from the recorded cwd, bounds per-file reads and current global occupancy across the main spool and recovery directories, sweeps completed tombstones, and holds correlated events outside drain until replay succeeds or restores them on failure; degraded resumes use `ai-memory-workstream`'s adapter-validated delta export, whose blocking parse and baseline scans are capped by record, event-count, source-byte and serialized-byte totals; `backfill::import_exact_session` chunks by item and byte limits and requires per-item durable acknowledgements; `run.rs` journals every session launch, including explicit spool-only evidence for unsupported exporters, and journals a resumed export only after capturing its exact pre-launch cursor; `workstream.rs` `/recover/finish` retains owner/project authorization, rejects superseded runs, and validates a finished retry by durable transcript/native/cursor/exit identity without regenerating its checkpoint; validated recovery ingest keys bind session, agent, event kind and sanitized event identity until session purge | `transcript.rs` `persisted_baseline_exports_only_offline_append_and_rejects_rewrite`, parser/baseline cap and ordered-semantic digest tests; CLI `run_offline.rs` real degraded spool-only and overlap/idempotency tests; `recovery.rs` `spool_only_without_correlated_evidence_never_auto_clears`, `a_dropped_spool_event_stays_lost_on_the_next_empty_pass`, loss/time-budget tests; `hook_spool.rs` partial-scope late enqueue, completion-race, occupancy churn/active-cap, bounded-read and tombstone-sweep tests, plus interval isolation (`completed_tombstone_keeps_a_later_resume_of_the_same_session` proves a post-interval resume hook is delivered, `completing_one_recovery_interval_leaves_the_other_intervals_entries_intact` proves one interval's completion never consumes another's evidence and ambiguous entries fail closed into the main spool); `backfill.rs` `two_sequential_recovery_intervals_on_one_resumed_session_replay_exactly_once` proves the second interval's content and session-end are stored with end-of-session effects while reruns stay pure replays; `transcript.rs` `an_oversized_first_record_is_skipped_by_session_listing_caps` proves the first-record byte caps bite before allocation; store `insert_observation_ingest_dedups_on_key` rejects cross-session and changed-event preclaims; hooks `expired_run_recovery_is_owner_only_and_refuses_superseded_runs` proves changed transcript refusal and changed-checkpoint no-op | STRONG on Unix integration coverage; Unix lock opens use no-follow and atomic replacement rechecks the destination. Windows owner-only ACL/reparse and lock semantics require the PR `windows` lane |
| 11i | OpenCode executable probing and generated-config cleanup (#1097) | `ai-memory-cli/src/config.rs` captures the full process environment once in typed `RuntimeEnv`; `run.rs` overlays explicit `run --env` values with platform key semantics and uses that same effective environment for executable resolution, probing, and launch without rereading process environment; `opencode_dialect.rs` invokes only direct argv `--version`, caps output and time, and rejects malformed/future majors. Automatic launch skips only OpenCode on probe failure; explicit OpenCode fails. `install_hooks.rs` quarantines the incompatible sibling before reading its exact ownership markers, restores unowned content with an atomic no-replace link, and never deletes or overwrites a concurrent replacement; `install_mcp.rs` owns the shared dual-schema endpoint/token inference used by both installers, rejects conflicting generated V1/V2 entries before mutation, and migrates an alternate-major entry only when endpoint, generated shape, and absent-or-Authorization-only headers match, preserving custom headers and ambiguous entries; auto-wire migrates MCP before hooks infer from that exact config path and skips hook mutation when transition preflight fails | `run.rs` `automatic_opencode_probe_failure_keeps_another_harness_available`, `opencode_executable_probe_selects_v2_for_launch_and_autowire`, effective-child-environment PATH casing controls, fake V1/V2/env-derived-major/malformed/future/timeout executable tests, `setup-agent` V1/V2 no-probe integration controls; `install_hooks.rs` `opencode_plugin_cleanup_removes_only_exact_owned_incompatible_sibling`, `opencode_plugin_cleanup_preserves_a_concurrent_user_replacement`, `opencode_plugin_restore_moves_quarantine_when_destination_is_free`, `opencode_plugin_restore_preserves_an_existing_replacement`, `opencode_plugin_restore_does_not_replace_a_file_created_at_restore_window`, `opencode_plugin_cleanup_does_not_escape_custom_target_directory`; `install_mcp.rs` `opencode_install_reuses_owned_endpoint_and_token_across_major_transitions`, `opencode_install_refuses_conflicting_owned_transition_entries`, `opencode_install_does_not_infer_from_a_user_owned_transition_entry`, `opencode_mcp_migration_moves_only_owned_alternate_shape`, `opencode_mcp_migration_preserves_ambiguous_user_entry`, `opencode_mcp_migration_preserves_custom_headers`, `opencode_mcp_migration_accepts_only_historical_generated_headers` | STRONG for Unix process probes and platform-neutral ownership gates; exact Windows executable resolution remains covered by `run.rs` resolver tests and requires the PR `windows` label |
| 12 | Network/auth posture | `config.rs` loopback `DEFAULT_BIND`; `serve.rs` `validate_http_exposure`, `require_allowed_host`; `auth.rs` `require_bearer`; `serve.rs` mounts `/identity` inside the host and machine-auth gates even with web disabled; `nix/nixos-module.nix` treats only literal `127.0.0.1`/`::1` as loopback, requires exactly one secret environment source for other binds, keeps secret auth and `llm_headers` values out of the Nix store, and brackets IPv6 in `ExecStart` | `serve.rs` host-guard (missing→400 / forged→403), non-loopback-requires-token; `auth.rs` wrong-token 401; `serve.rs` `machine_identity_remains_authenticated_with_web_disabled_and_expired_keys` (wrong and expired keys refused, valid machine key accepted, web remains disabled); `flake.nix` `nixos-module-eval` refuses `localhost` and `0.0.0.0` without a secret source, refuses two secret sources, `settings.auth` secrets, and `settings.llm_headers`, with literal IPv4/IPv6 loopback and one-source controls | STRONG |
| 12b | Managed-workstream provenance privacy, stable replay identity and server event namespace | `ai-memory-core/src/workstream.rs` `scrub_workstream_provenance`; `ai-memory-hooks/src/workstream.rs` `finish_run` reserved client-id check and `sanitize_events` before immutable segments; `ai-memory-store/src/workstream.rs` `finish_run` stable identity check and `stored_event`, using an explicitly supplied configured sanitizer; no changes to the typed observation boundary | `store/tests/suite/workstream_provenance.rs` `workstream_provenance_is_bounded_scrubbed_and_idempotent_before_storage` (hostile and oversized source ids, unknown metadata, legitimate correlation, retry and distinct event ids), `workstream_provenance_legacy_reads_strip_secrets_and_unknown_metadata` (search and tail, malformed/oversized legacy JSON), `workstream_provenance_reused_event_id_retains_data_and_refuses_identity_change` (first indexed version retained, identity refusal and batch rollback), `workstream_provenance_replay_survives_sanitizer_changes` (new extra patterns and removal of a built-in-pattern allowlist exemption, with identity-refusal control), `workstream_provenance_replay_preserves_legacy_rows_above_bounds`, `workstream_provenance_finished_run_ignores_late_events`, `workstream_provenance_retry_with_marker_matching_pattern_is_idempotent`; `hooks` `workstream_provenance_finish_scrubs_before_raw_segment_and_search` (custom sanitizer, immutable segment, SQL recovery and finished-run no-op), `workstream_provenance_client_cannot_use_server_event_ids` (reserved ids refused before raw writes, normal client event and server-generated checkpoint/loss controls); `cli/tests/suite/workstream_provenance.rs` (text/JSON and old server DTOs) | STRONG: privacy, stable-identity and reserved-namespace regressions fail with their guards bypassed; an isolated allowlist bypass fails on the unknown key, and bypassing the raw-id search authorization returns 200 where its test requires 403. No provenance is emitted in startup packets, and old raw segments are not rewritten |