Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(hooks): scrub observation bodies before the per-event caps
Vinícius DS Andrade52b60a8987b9
diff --git a/crates/ai-memory-hooks/src/payload.rs b/crates/ai-memory-hooks/src/payload.rsindex 5703c963..e6de5040 100644--- a/crates/ai-memory-hooks/src/payload.rs+++ b/crates/ai-memory-hooks/src/payload.rs@@ -17,7 +17,32 @@ pub const POST_COMPACTION_EXCERPT_MAX_BYTES: usize = OBSERVATION_BODY_MAX_BYTES;/// Durable excerpt ceiling for notifications.pub const NOTIFICATION_EXCERPT_MAX_BYTES: usize = 2_000;-const TOOL_EXCERPT_MAX_BYTES: usize = 2_000;+/// Durable excerpt ceiling for tool I/O summaries and short excerpts+/// (tool results, extension bodies, Stop assistant excerpts).+pub const TOOL_EXCERPT_MAX_BYTES: usize = 2_000;++/// Durable body ceiling the ingest funnel applies to an event's excerpt+/// **after** the sanitizer has scrubbed the full text.+///+/// The cap used to run inside excerpt extraction, before the sanitizer ever+/// saw the text, so a secret straddling the cutoff was cut in half and the+/// surviving prefix too short to match a pattern — stored unredacted (#1114),+/// the body-side twin of the #980 title-hint leak. Extraction now returns the+/// uncapped text and the router applies this cap to the scrubbed output,+/// mirroring `ai_memory_core::sanitize::Sanitized::new`.+#[must_use]+pub fn durable_body_cap(event: HookEvent) -> usize {+ match event {+ HookEvent::UserPrompt => USER_PROMPT_EXCERPT_MAX_BYTES,+ HookEvent::PostCompaction => POST_COMPACTION_EXCERPT_MAX_BYTES,+ HookEvent::Notification+ | HookEvent::PreToolUse+ | HookEvent::PostToolUse+ | HookEvent::Stop+ | HookEvent::Other => TOOL_EXCERPT_MAX_BYTES,+ _ => OBSERVATION_BODY_MAX_BYTES,+ }+}/// Query-string parameters on `POST /hook`.#[derive(Debug, Clone, Default, Deserialize)]@@ -777,7 +802,7 @@ fn legacy_tool_body(event: HookEvent, agent: AgentKind, raw: &serde_json::Value)&["tool_response", "tool_output", "output", "result"],).or_else(|| extract_content(payload, &["error"]))?;- Some(truncate_excerpt(&format!("tool: {tool}\n---\n{result}")))+ Some(format!("tool: {tool}\n---\n{result}"))}const fn closed_tool_agent(agent: AgentKind) -> bool {@@ -886,7 +911,7 @@ fn safe_tool_body(.unwrap_or_else(|| "(no output captured)".into());summary.push_str("\n---\n");summary.push_str(&result);- Some(truncate_excerpt(&summary))+ Some(summary)}_ => None,}@@ -1051,7 +1076,6 @@ fn extension_body_excerpt(raw: &serde_json::Value) -> Option<String> {"details",],)- .map(|s| truncate_excerpt(&s))}/// Extract human-readable text content for an observation body, accepting the@@ -1106,8 +1130,7 @@ fn value_to_text(value: &serde_json::Value) -> Option<String> {fn best_body_excerpt(event: HookEvent, raw: &serde_json::Value) -> Option<String> {match event {- HookEvent::UserPrompt => extract_content(raw, &["prompt", "message", "text"])- .map(|body| truncate_utf8_bytes(&body, USER_PROMPT_EXCERPT_MAX_BYTES)),+ HookEvent::UserPrompt => extract_content(raw, &["prompt", "message", "text"]),HookEvent::PostToolUse => {let tool = extract_string(raw, &["tool", "tool_name", "name"]).or_else(|| extract_string_path(raw, &[&["toolCall", "name"]]))@@ -1118,12 +1141,10 @@ fn best_body_excerpt(event: HookEvent, raw: &serde_json::Value) -> Option<Stringextract_content(raw, &["tool_response", "tool_output", "output", "result"]).or_else(|| extract_content(raw, &["error"])).unwrap_or_else(|| "(no output captured)".into());- Some(format!("tool: {tool}\n---\n{}", truncate_excerpt(&result)))+ Some(format!("tool: {tool}\n---\n{result}"))}- HookEvent::Notification => extract_content(raw, &["message", "text"])- .map(|body| truncate_utf8_bytes(&body, NOTIFICATION_EXCERPT_MAX_BYTES)),- HookEvent::PostCompaction => extract_content(raw, &["summary"])- .map(|body| truncate_utf8_bytes(&body, POST_COMPACTION_EXCERPT_MAX_BYTES)),+ HookEvent::Notification => extract_content(raw, &["message", "text"]),+ HookEvent::PostCompaction => extract_content(raw, &["summary"]),_ => None,}}@@ -1150,10 +1171,6 @@ fn first_line(s: String) -> Option<String> {(!line.is_empty()).then_some(line)}-fn truncate_excerpt(s: &str) -> String {- truncate_utf8_bytes(s, TOOL_EXCERPT_MAX_BYTES)-}-/// Cap core lifecycle body fields before the native hook writes its local/// spool entry. The server independently reapplies the same per-event limits/// while constructing [`HookEnvelope`], and the typed persistence boundary has@@ -2415,11 +2432,19 @@ mod tests {},serde_json::json!({(field): body}),);- let excerpt = env.body_excerpt.expect("bounded body excerpt");- assert!(excerpt.len() <= cap, "{event} exceeded {cap} bytes");- assert!(excerpt.ends_with('…'), "{event} omitted truncation marker");+ // Extraction no longer caps (#1114): the sanitizer at the ingest+ // funnel must see the full text before any byte limit runs. The+ // scrub-then-cap order itself is asserted by the router's+ // `issue_1114_*` tests.+ let excerpt = env.body_excerpt.expect("body excerpt");+ assert_eq!(excerpt, body, "{event} excerpt was capped at extraction");+ let hook_event = HookEvent::parse(event);+ assert_eq!(durable_body_cap(hook_event), cap);+ let capped = truncate_utf8_bytes(&excerpt, durable_body_cap(hook_event));+ assert!(capped.len() <= cap, "{event} exceeded {cap} bytes");+ assert!(capped.ends_with('…'), "{event} omitted truncation marker");assert!(- !excerpt.contains("TAIL_SENTINEL"),+ !capped.contains("TAIL_SENTINEL"),"{event} retained content after the cap");}@@ -2732,9 +2757,12 @@ mod tests {},serde_json::json!({"tool_name": "Bash", "tool_use_id": "call-long", "tool_response": {"content": [{"type": "text", "text": "é".repeat(2_000)}]}}),);+ // Extraction no longer caps: the sanitizer must see the full text+ // before any byte limit runs (#1114). The cap itself is asserted at+ // the ingest funnel, where it is applied to the scrubbed body.let body = env.body_excerpt.unwrap();assert!(body.contains('é'));- assert!(body.len() <= TOOL_EXCERPT_MAX_BYTES);+ assert!(body.len() > TOOL_EXCERPT_MAX_BYTES);}#[test]diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rsindex 0ee17a63..e1ef083f 100644--- a/crates/ai-memory-hooks/src/router.rs+++ b/crates/ai-memory-hooks/src/router.rs@@ -17,7 +17,7 @@ use ai_memory_core::{ActiveProject, ActorKey, AgentKind, DEFAULT_WORKSPACE_NAME, Handoff, IdentityKey,MANAGED_WORKSTREAM_PACKET_MARKER, ManagedRunId, MidSessionRouting, NewHandoff, NewObservation,NewSession, ObservationKind, ProjectId, Sanitized, Sanitizer, SessionId, WorkspaceId,- WorkstreamEvent, WorkstreamEventKind,+ WorkstreamEvent, WorkstreamEventKind, truncate_utf8_bytes,};use ai_memory_store::{HookSessionAdmission, IngestObservationOutcome, StoreError, WriterHandle,@@ -42,7 +42,7 @@ use crate::capture_policy::{use crate::log;use crate::payload::{HookEnvelope, HookEvent, HookQuery, ProjectSource, ProjectStrategy, body_is_subagent,- parse_agent,+ durable_body_cap, parse_agent,};use crate::synth::synthesize_session_page;@@ -3292,6 +3292,21 @@ async fn process(}}+/// Durable body for a hook observation: scrub the **full** excerpt first,+/// then apply the per-event ceiling to the redacted text.+///+/// The cap used to run inside excerpt extraction in `payload`, before the+/// sanitizer ever saw the text — a secret straddling the cutoff was cut in+/// half, leaving a prefix too short to match a pattern, stored in clear text+/// (#1114). This is the body-side twin of the #980 title-hint fix: scrub and+/// cap together, in that order, at the persistence boundary.+fn durable_body(event: HookEvent, excerpt: Option<&str>, sanitizer: &Sanitizer) -> String {+ let Some(excerpt) = excerpt else {+ return String::new();+ };+ truncate_utf8_bytes(&sanitizer.scrub(excerpt), durable_body_cap(event))+}+async fn process_authorized(state: &HookState,env: HookEnvelope,@@ -3534,7 +3549,7 @@ async fn process_authorized(.title_hint.clone().unwrap_or_else(|| kind.as_str().to_string()),- body: env.body_excerpt.clone().unwrap_or_default(),+ body: durable_body(env.event, env.body_excerpt.as_deref(), &state.sanitizer),importance: importance_for(env.event),occurred_at: env.occurred_at_micros(),};@@ -16735,6 +16750,180 @@ mod tests {);}+ /// Shared shape for the #1114 body-side regressions: a secret straddling+ /// a per-event byte cap used to be cut in half at excerpt extraction,+ /// before the sanitizer ever ran, so the surviving prefix was too short+ /// to match and was stored unredacted. Each test drives the full ingest+ /// funnel with a `{20,}`-length pattern and asserts the stored body+ /// carries the redaction marker, no secret fragment, and the cap.+ async fn assert_body_scrubbed_before_cap(+ state: &HookState,+ event: &str,+ kind: ObservationKind,+ body_key: &str,+ padding: usize,+ cap: usize,+ ) {+ let secret = format!("SECRET{}", "Z".repeat(30));+ let sid = SessionId::new();+ let env = HookEnvelope::from_query_and_body(+ HookQuery {+ event: event.into(),+ agent: Some("claude-code".into()),+ ..Default::default()+ },+ serde_json::json!({+ "session_id": sid.to_string(),+ "cwd": "/repo",+ body_key: format!("{} {secret}", "x".repeat(padding)),+ }),+ );+ process(state, env, None, Vec::new()).await.unwrap();++ let observations = state.reader.observations_for_session(sid).await.unwrap();+ let body = observations+ .iter()+ .find(|o| o.kind == kind)+ .unwrap_or_else(|| panic!("{event} observation was recorded"))+ .body+ .clone();+ assert!(+ !body.contains(&secret) && !body.contains("SECRETZ"),+ "unredacted secret fragment survived the cap: {body:?}"+ );+ assert!(+ body.contains("REDACT"),+ "body carries no trace of redaction — the sanitizer never saw \+ enough of the secret to match it: {body:?}"+ );+ assert!(body.len() <= cap, "body exceeded the {cap}-byte cap: {body:?}");+ }++ #[tokio::test]+ async fn issue_1114_user_prompt_body_is_sanitized_before_the_16kib_cap() {+ let tmp = TempDir::new().unwrap();+ let mut state = make_state(&tmp).await;+ state.sanitizer = Sanitizer::new(&SanitizeConfig {+ extra_patterns: vec![r"SECRET[0-9A-Za-z]{20,}".into()],+ allowlist: Vec::new(),+ })+ .unwrap();+ assert_body_scrubbed_before_cap(+ &state,+ "user-prompt",+ ObservationKind::UserPrompt,+ "prompt",+ crate::payload::USER_PROMPT_EXCERPT_MAX_BYTES - 14,+ crate::payload::USER_PROMPT_EXCERPT_MAX_BYTES,+ )+ .await;+ }++ #[tokio::test]+ async fn issue_1114_post_compaction_body_is_sanitized_before_the_16kib_cap() {+ let tmp = TempDir::new().unwrap();+ let mut state = make_state(&tmp).await;+ state.sanitizer = Sanitizer::new(&SanitizeConfig {+ extra_patterns: vec![r"SECRET[0-9A-Za-z]{20,}".into()],+ allowlist: Vec::new(),+ })+ .unwrap();+ assert_body_scrubbed_before_cap(+ &state,+ "post-compaction",+ ObservationKind::PostCompaction,+ "summary",+ crate::payload::POST_COMPACTION_EXCERPT_MAX_BYTES - 14,+ crate::payload::POST_COMPACTION_EXCERPT_MAX_BYTES,+ )+ .await;+ }++ /// The tool excerpt surface: the cap runs on the whole durable body+ /// (`tool: …\n---\n…`), so the padding lands inside the tool result.+ #[tokio::test]+ async fn issue_1114_tool_body_is_sanitized_before_the_2kb_cap() {+ let tmp = TempDir::new().unwrap();+ let mut state = make_state(&tmp).await;+ state.sanitizer = Sanitizer::new(&SanitizeConfig {+ extra_patterns: vec![r"SECRET[0-9A-Za-z]{20,}".into()],+ allowlist: Vec::new(),+ })+ .unwrap();+ let secret = format!("SECRET{}", "Z".repeat(30));+ let sid = SessionId::new();+ let env = HookEnvelope::from_query_and_body(+ HookQuery {+ event: "PostToolUse".into(),+ agent: Some("claude-code".into()),+ ..Default::default()+ },+ serde_json::json!({+ "session_id": sid.to_string(),+ "cwd": "/repo",+ "tool_name": "Read",+ "tool_use_id": "call-1114",+ "tool_response": format!("{} {secret}", "x".repeat(1_940)),+ }),+ );+ process(&state, env, None, Vec::new()).await.unwrap();++ let observations = state.reader.observations_for_session(sid).await.unwrap();+ let body = observations+ .iter()+ .find(|o| o.kind == ObservationKind::PostToolUse)+ .expect("post-tool-use observation was recorded")+ .body+ .clone();+ assert!(+ !body.contains(&secret) && !body.contains("SECRETZ"),+ "unredacted secret fragment survived the cap: {body:?}"+ );+ assert!(body.contains("REDACT"), "no redaction marker: {body:?}");+ assert!(+ body.len() <= crate::payload::TOOL_EXCERPT_MAX_BYTES,+ "body exceeded the 2 KB cap: {body:?}"+ );+ }++ /// The per-event caps keep splitting on UTF-8 boundaries through the+ /// funnel (`truncate_utf8_bytes`), so a multibyte tool result is capped+ /// without producing invalid bytes.+ #[tokio::test]+ async fn tool_body_cap_keeps_the_utf8_boundary() {+ let tmp = TempDir::new().unwrap();+ let state = make_state(&tmp).await;+ let sid = SessionId::new();+ let env = HookEnvelope::from_query_and_body(+ HookQuery {+ event: "PostToolUse".into(),+ agent: Some("claude-code".into()),+ ..Default::default()+ },+ serde_json::json!({+ "session_id": sid.to_string(),+ "cwd": "/repo",+ "tool_name": "Read",+ "tool_use_id": "call-utf8",+ "tool_response": "é".repeat(2_000), // 2 bytes each: 2x the cap+ }),+ );+ process(&state, env, None, Vec::new()).await.unwrap();++ let observations = state.reader.observations_for_session(sid).await.unwrap();+ let body = observations+ .iter()+ .find(|o| o.kind == ObservationKind::PostToolUse)+ .expect("post-tool-use observation was recorded")+ .body;+ assert!(body.contains('é'));+ assert!(+ body.len() <= crate::payload::TOOL_EXCERPT_MAX_BYTES,+ "body exceeded the 2 KB cap: {body:?}"+ );+ assert!(body.ends_with('…'));+ }+#[test]fn codex_native_patch_capture_backstop_discards_unproven_output() {for protocol in [diff --git a/docs/security-boundaries.md b/docs/security-boundaries.mdindex 3f43e2d6..71293742 100644--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@ -49,7 +49,7 @@ boundary not yet built.| 6 | Active-project pointer (PerActor, no clobber) | `ai-memory-core/src/active_project.rs` `set_for`/`lookup_for` (fail-closed on `Mismatch`) | `active_project.rs` `parallel_harnesses_of_one_user_keep_separate_pointers`, `two_operators_never_read_each_others_pointer`, `a_session_mismatch_fails_closed_once_anything_has_been_keyed` | STRONG || 6b | Native `_meta` routing coordinate is routing-only (never grants identity) | `ai-memory-mcp/src/server.rs` `attach_native_session` reads only `_meta["ai.opencode/sessionID"]` into `ActorKey.session_id` (trimmed, non-empty-string-validated); `user` still comes from the authenticated `ActorContext`, so a forged `_meta` cannot cross a user/scope boundary — it only selects a session slot behind the existing `active_project` guard | `server.rs` `native_session_metadata_is_routing_only_and_preserves_precedence`, `native_session_metadata_validates_strings_without_granting_identity`; `autoscope_multiuser.rs` `native_metadata_routes_concurrent_reads_and_writes_to_hook_workspaces` (#864) | STRONG || 6c | Per-user slot namespaces stay per operator (`[slots] per_user`) | `ai-memory-mcp/src/server.rs` `place_slot_mutation`, shared by `place_slot_write` (`memory_write_page`) and `place_slot_delete` (`memory_delete_page`): a non-admin's generic `_slots/<name>.md` is redirected to their own `_slots/u-<operator>/` namespace, a path inside another operator's namespace is refused, admins bypass, and nothing changes with the flag off (#1083) | `slot_identity.rs` `oidc_operator_cannot_delete_another_operators_personal_slot`, `oidc_operator_deleting_shared_slot_path_deletes_their_personal_slot` (shared slot survives), legitimate controls `oidc_operator_can_delete_their_own_personal_slot_explicitly`, `admin_can_delete_shared_and_personal_slots` | STRONG |-| 7 | Sanitizer trust boundary (invariant #6) | `ai-memory-core/src/sanitize.rs` `Sanitized<T>` (private field, only `sanitize()` ctor); `Sanitized::new` scrubs the title *before* the 80-char display cap; `ai-memory-mcp/src/server.rs` `sanitize_feedback_reason` scrubs the reason *before* the 500-char cap; `WriterHandle::insert_observation` requires `Sanitized`; `ops` crate-private | Structural (compile-time) + `store/src/lib.rs` `insert_observation_boundary_scrubs_before_disk` + sanitizer scrub unit tests + `ai-memory-core::sanitize` `title_is_scrubbed_before_the_80_char_cap_runs` + `ai-memory-hooks::router` `issue_980_user_prompt_title_is_sanitized_before_truncated` (a secret straddling the 80-char boundary is redacted, not stored truncated; #980) + `ai-memory-mcp::server` `feedback_reason_is_scrubbed_before_the_500_char_cap` (#1109) | STRONG (structural) |+| 7 | Sanitizer trust boundary (invariant #6) | `ai-memory-core/src/sanitize.rs` `Sanitized<T>` (private field, only `sanitize()` ctor); `Sanitized::new` scrubs the title *before* the 80-char display cap; `ai-memory-mcp/src/server.rs` `sanitize_feedback_reason` scrubs the reason *before* the 500-char cap; `ai-memory-hooks/src/router.rs` `durable_body` scrubs the full excerpt *before* the per-event body caps (`payload::durable_body_cap`), which no longer run at excerpt extraction; `WriterHandle::insert_observation` requires `Sanitized`; `ops` crate-private | Structural (compile-time) + `store/src/lib.rs` `insert_observation_boundary_scrubs_before_disk` + sanitizer scrub unit tests + `ai-memory-core::sanitize` `title_is_scrubbed_before_the_80_char_cap_runs` + `ai-memory-hooks::router` `issue_980_user_prompt_title_is_sanitized_before_truncated` (a secret straddling the 80-char boundary is redacted, not stored truncated; #980) + `ai-memory-mcp::server` `feedback_reason_is_scrubbed_before_the_500_char_cap` (#1109) + `ai-memory-hooks::router` `issue_1114_user_prompt_body_is_sanitized_before_the_16kib_cap`, `issue_1114_post_compaction_body_is_sanitized_before_the_16kib_cap`, `issue_1114_tool_body_is_sanitized_before_the_2kb_cap`, `tool_body_cap_keeps_the_utf8_boundary` (each places a secret straddling the cap and asserts a `[REDACTED:…]` marker survives with no raw fragment; #1114) | STRONG (structural) || 7a | Eval observations: server provenance and reason privacy | `ai-memory-consolidate/src/auto_improve.rs` constructs private results from the existing execution, ignores extra model/scorer result fields and all eval results on report deserialization, hashes the evaluated request/bodies, and scrubs reasons before the 2,048-character cap and pending persistence; changed bodies cannot receive the earlier result | `auto_improve::tests::structured_eval_records_server_observation`; `tests/suite/eval_results.rs` `eval_reason_is_sanitized_before_pending_storage_with_legitimate_control` attempts forged deserialized reports and IDs/digests and malicious reasons with secrets/control characters, checks report and stored evidence before sidecar rendering, and preserves a legitimate reason and external evidence without independent results; fails with reason scrubbing or the report deserialization guard removed | STRONG || 7b | New public page metadata is bounded and cannot supply scope, author or admission controls; admin kind retains its legacy adapter | `ai-memory-core/src/page.rs` `PageWriteMetadata::into_frontmatter`; MCP validates all four fields; admin consumes legacy kind explicitly and shares entities/abstract/relations validation before scope creation; both use `Wiki::write_page` for sanitization/admission/authenticated attribution | `admin_write_page.rs` `write_page_metadata_rejects_invalid_payload_before_scope_creation` (invalid shapes, unknown relation kinds, raw-input over-limit values, untrimmed relation scopes and escaping paths refused; legitimate control admitted), `write_page_metadata_preserves_admin_auth_and_sanitization` (anonymous/DB-user refused, root control admitted with scrubbed metadata and authenticated attribution); `server.rs` `memory_write_page_metadata_rejects_invalid_before_scope_creation`, `memory_write_page_metadata_preserves_scope_author_and_sanitization` (foreign user denied; writer control admitted, forged frontmatter/author ignored, secrets scrubbed, invalid rewrite leaves disk and SQL latest unchanged); core `page_write_metadata_scope_components_refuse_surrounding_whitespace` and `page_write_metadata_limits_apply_before_trimming_and_normalization`; admin `write_page_metadata_admin_legacy_kind_matches_base` compares long/control-character kind behavior with the base adapter; entity-bound refusal bite-checked with its validation removed | STRONG || 8a | Messaging: recipient-only visibility | `ai-memory-store/src/ops.rs` pop target-select + `reader.rs` `list_messages` (`to_*` predicate) | `agent_messages.rs` `a_message_is_only_visible_to_its_recipient`; `agent_messages_tools.rs` non-recipient cannot pop | STRONG |