Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(hooks): scrub observation bodies before the per-event caps

Vinícius DS Andrade52b60a8987b9
diff --git a/crates/ai-memory-hooks/src/payload.rs b/crates/ai-memory-hooks/src/payload.rs
index 5703c963..e6de5040 100644
--- a/crates/ai-memory-hooks/src/payload.rs
+++ b/crates/ai-memory-hooks/src/payload.rs
@@ -17,7 +17,32 @@ pub const POST_COMPACTION_EXCERPT_MAX_BYTES: usize = OBSERVATION_BODY_MAX_BYTES;
/// Durable excerpt ceiling for notifications.
pub const NOTIFICATION_EXCERPT_MAX_BYTES: usize = 2_000;
-const TOOL_EXCERPT_MAX_BYTES: usize = 2_000;
+/// Durable excerpt ceiling for tool I/O summaries and short excerpts
+/// (tool results, extension bodies, Stop assistant excerpts).
+pub const TOOL_EXCERPT_MAX_BYTES: usize = 2_000;
+
+/// Durable body ceiling the ingest funnel applies to an event's excerpt
+/// **after** the sanitizer has scrubbed the full text.
+///
+/// The cap used to run inside excerpt extraction, before the sanitizer ever
+/// saw the text, so a secret straddling the cutoff was cut in half and the
+/// surviving prefix too short to match a pattern — stored unredacted (#1114),
+/// the body-side twin of the #980 title-hint leak. Extraction now returns the
+/// uncapped text and the router applies this cap to the scrubbed output,
+/// mirroring `ai_memory_core::sanitize::Sanitized::new`.
+#[must_use]
+pub fn durable_body_cap(event: HookEvent) -> usize {
+ match event {
+ HookEvent::UserPrompt => USER_PROMPT_EXCERPT_MAX_BYTES,
+ HookEvent::PostCompaction => POST_COMPACTION_EXCERPT_MAX_BYTES,
+ HookEvent::Notification
+ | HookEvent::PreToolUse
+ | HookEvent::PostToolUse
+ | HookEvent::Stop
+ | HookEvent::Other => TOOL_EXCERPT_MAX_BYTES,
+ _ => OBSERVATION_BODY_MAX_BYTES,
+ }
+}
/// Query-string parameters on `POST /hook`.
#[derive(Debug, Clone, Default, Deserialize)]
@@ -777,7 +802,7 @@ fn legacy_tool_body(event: HookEvent, agent: AgentKind, raw: &serde_json::Value)
&["tool_response", "tool_output", "output", "result"],
)
.or_else(|| extract_content(payload, &["error"]))?;
- Some(truncate_excerpt(&format!("tool: {tool}\n---\n{result}")))
+ Some(format!("tool: {tool}\n---\n{result}"))
}
const fn closed_tool_agent(agent: AgentKind) -> bool {
@@ -886,7 +911,7 @@ fn safe_tool_body(
.unwrap_or_else(|| "(no output captured)".into());
summary.push_str("\n---\n");
summary.push_str(&result);
- Some(truncate_excerpt(&summary))
+ Some(summary)
}
_ => None,
}
@@ -1051,7 +1076,6 @@ fn extension_body_excerpt(raw: &serde_json::Value) -> Option<String> {
"details",
],
)
- .map(|s| truncate_excerpt(&s))
}
/// Extract human-readable text content for an observation body, accepting the
@@ -1106,8 +1130,7 @@ fn value_to_text(value: &serde_json::Value) -> Option<String> {
fn best_body_excerpt(event: HookEvent, raw: &serde_json::Value) -> Option<String> {
match event {
- HookEvent::UserPrompt => extract_content(raw, &["prompt", "message", "text"])
- .map(|body| truncate_utf8_bytes(&body, USER_PROMPT_EXCERPT_MAX_BYTES)),
+ HookEvent::UserPrompt => extract_content(raw, &["prompt", "message", "text"]),
HookEvent::PostToolUse => {
let tool = extract_string(raw, &["tool", "tool_name", "name"])
.or_else(|| extract_string_path(raw, &[&["toolCall", "name"]]))
@@ -1118,12 +1141,10 @@ fn best_body_excerpt(event: HookEvent, raw: &serde_json::Value) -> Option<String
extract_content(raw, &["tool_response", "tool_output", "output", "result"])
.or_else(|| extract_content(raw, &["error"]))
.unwrap_or_else(|| "(no output captured)".into());
- Some(format!("tool: {tool}\n---\n{}", truncate_excerpt(&result)))
+ Some(format!("tool: {tool}\n---\n{result}"))
}
- HookEvent::Notification => extract_content(raw, &["message", "text"])
- .map(|body| truncate_utf8_bytes(&body, NOTIFICATION_EXCERPT_MAX_BYTES)),
- HookEvent::PostCompaction => extract_content(raw, &["summary"])
- .map(|body| truncate_utf8_bytes(&body, POST_COMPACTION_EXCERPT_MAX_BYTES)),
+ HookEvent::Notification => extract_content(raw, &["message", "text"]),
+ HookEvent::PostCompaction => extract_content(raw, &["summary"]),
_ => None,
}
}
@@ -1150,10 +1171,6 @@ fn first_line(s: String) -> Option<String> {
(!line.is_empty()).then_some(line)
}
-fn truncate_excerpt(s: &str) -> String {
- truncate_utf8_bytes(s, TOOL_EXCERPT_MAX_BYTES)
-}
-
/// Cap core lifecycle body fields before the native hook writes its local
/// spool entry. The server independently reapplies the same per-event limits
/// while constructing [`HookEnvelope`], and the typed persistence boundary has
@@ -2415,11 +2432,19 @@ mod tests {
},
serde_json::json!({(field): body}),
);
- let excerpt = env.body_excerpt.expect("bounded body excerpt");
- assert!(excerpt.len() <= cap, "{event} exceeded {cap} bytes");
- assert!(excerpt.ends_with('…'), "{event} omitted truncation marker");
+ // Extraction no longer caps (#1114): the sanitizer at the ingest
+ // funnel must see the full text before any byte limit runs. The
+ // scrub-then-cap order itself is asserted by the router's
+ // `issue_1114_*` tests.
+ let excerpt = env.body_excerpt.expect("body excerpt");
+ assert_eq!(excerpt, body, "{event} excerpt was capped at extraction");
+ let hook_event = HookEvent::parse(event);
+ assert_eq!(durable_body_cap(hook_event), cap);
+ let capped = truncate_utf8_bytes(&excerpt, durable_body_cap(hook_event));
+ assert!(capped.len() <= cap, "{event} exceeded {cap} bytes");
+ assert!(capped.ends_with('…'), "{event} omitted truncation marker");
assert!(
- !excerpt.contains("TAIL_SENTINEL"),
+ !capped.contains("TAIL_SENTINEL"),
"{event} retained content after the cap"
);
}
@@ -2732,9 +2757,12 @@ mod tests {
},
serde_json::json!({"tool_name": "Bash", "tool_use_id": "call-long", "tool_response": {"content": [{"type": "text", "text": "é".repeat(2_000)}]}}),
);
+ // Extraction no longer caps: the sanitizer must see the full text
+ // before any byte limit runs (#1114). The cap itself is asserted at
+ // the ingest funnel, where it is applied to the scrubbed body.
let body = env.body_excerpt.unwrap();
assert!(body.contains('é'));
- assert!(body.len() <= TOOL_EXCERPT_MAX_BYTES);
+ assert!(body.len() > TOOL_EXCERPT_MAX_BYTES);
}
#[test]
diff --git a/crates/ai-memory-hooks/src/router.rs b/crates/ai-memory-hooks/src/router.rs
index 0ee17a63..e1ef083f 100644
--- a/crates/ai-memory-hooks/src/router.rs
+++ b/crates/ai-memory-hooks/src/router.rs
@@ -17,7 +17,7 @@ use ai_memory_core::{
ActiveProject, ActorKey, AgentKind, DEFAULT_WORKSPACE_NAME, Handoff, IdentityKey,
MANAGED_WORKSTREAM_PACKET_MARKER, ManagedRunId, MidSessionRouting, NewHandoff, NewObservation,
NewSession, ObservationKind, ProjectId, Sanitized, Sanitizer, SessionId, WorkspaceId,
- WorkstreamEvent, WorkstreamEventKind,
+ WorkstreamEvent, WorkstreamEventKind, truncate_utf8_bytes,
};
use ai_memory_store::{
HookSessionAdmission, IngestObservationOutcome, StoreError, WriterHandle,
@@ -42,7 +42,7 @@ use crate::capture_policy::{
use crate::log;
use crate::payload::{
HookEnvelope, HookEvent, HookQuery, ProjectSource, ProjectStrategy, body_is_subagent,
- parse_agent,
+ durable_body_cap, parse_agent,
};
use crate::synth::synthesize_session_page;
@@ -3292,6 +3292,21 @@ async fn process(
}
}
+/// Durable body for a hook observation: scrub the **full** excerpt first,
+/// then apply the per-event ceiling to the redacted text.
+///
+/// The cap used to run inside excerpt extraction in `payload`, before the
+/// sanitizer ever saw the text — a secret straddling the cutoff was cut in
+/// half, leaving a prefix too short to match a pattern, stored in clear text
+/// (#1114). This is the body-side twin of the #980 title-hint fix: scrub and
+/// cap together, in that order, at the persistence boundary.
+fn durable_body(event: HookEvent, excerpt: Option<&str>, sanitizer: &Sanitizer) -> String {
+ let Some(excerpt) = excerpt else {
+ return String::new();
+ };
+ truncate_utf8_bytes(&sanitizer.scrub(excerpt), durable_body_cap(event))
+}
+
async fn process_authorized(
state: &HookState,
env: HookEnvelope,
@@ -3534,7 +3549,7 @@ async fn process_authorized(
.title_hint
.clone()
.unwrap_or_else(|| kind.as_str().to_string()),
- body: env.body_excerpt.clone().unwrap_or_default(),
+ body: durable_body(env.event, env.body_excerpt.as_deref(), &state.sanitizer),
importance: importance_for(env.event),
occurred_at: env.occurred_at_micros(),
};
@@ -16735,6 +16750,180 @@ mod tests {
);
}
+ /// Shared shape for the #1114 body-side regressions: a secret straddling
+ /// a per-event byte cap used to be cut in half at excerpt extraction,
+ /// before the sanitizer ever ran, so the surviving prefix was too short
+ /// to match and was stored unredacted. Each test drives the full ingest
+ /// funnel with a `{20,}`-length pattern and asserts the stored body
+ /// carries the redaction marker, no secret fragment, and the cap.
+ async fn assert_body_scrubbed_before_cap(
+ state: &HookState,
+ event: &str,
+ kind: ObservationKind,
+ body_key: &str,
+ padding: usize,
+ cap: usize,
+ ) {
+ let secret = format!("SECRET{}", "Z".repeat(30));
+ let sid = SessionId::new();
+ let env = HookEnvelope::from_query_and_body(
+ HookQuery {
+ event: event.into(),
+ agent: Some("claude-code".into()),
+ ..Default::default()
+ },
+ serde_json::json!({
+ "session_id": sid.to_string(),
+ "cwd": "/repo",
+ body_key: format!("{} {secret}", "x".repeat(padding)),
+ }),
+ );
+ process(state, env, None, Vec::new()).await.unwrap();
+
+ let observations = state.reader.observations_for_session(sid).await.unwrap();
+ let body = observations
+ .iter()
+ .find(|o| o.kind == kind)
+ .unwrap_or_else(|| panic!("{event} observation was recorded"))
+ .body
+ .clone();
+ assert!(
+ !body.contains(&secret) && !body.contains("SECRETZ"),
+ "unredacted secret fragment survived the cap: {body:?}"
+ );
+ assert!(
+ body.contains("REDACT"),
+ "body carries no trace of redaction — the sanitizer never saw \
+ enough of the secret to match it: {body:?}"
+ );
+ assert!(body.len() <= cap, "body exceeded the {cap}-byte cap: {body:?}");
+ }
+
+ #[tokio::test]
+ async fn issue_1114_user_prompt_body_is_sanitized_before_the_16kib_cap() {
+ let tmp = TempDir::new().unwrap();
+ let mut state = make_state(&tmp).await;
+ state.sanitizer = Sanitizer::new(&SanitizeConfig {
+ extra_patterns: vec![r"SECRET[0-9A-Za-z]{20,}".into()],
+ allowlist: Vec::new(),
+ })
+ .unwrap();
+ assert_body_scrubbed_before_cap(
+ &state,
+ "user-prompt",
+ ObservationKind::UserPrompt,
+ "prompt",
+ crate::payload::USER_PROMPT_EXCERPT_MAX_BYTES - 14,
+ crate::payload::USER_PROMPT_EXCERPT_MAX_BYTES,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn issue_1114_post_compaction_body_is_sanitized_before_the_16kib_cap() {
+ let tmp = TempDir::new().unwrap();
+ let mut state = make_state(&tmp).await;
+ state.sanitizer = Sanitizer::new(&SanitizeConfig {
+ extra_patterns: vec![r"SECRET[0-9A-Za-z]{20,}".into()],
+ allowlist: Vec::new(),
+ })
+ .unwrap();
+ assert_body_scrubbed_before_cap(
+ &state,
+ "post-compaction",
+ ObservationKind::PostCompaction,
+ "summary",
+ crate::payload::POST_COMPACTION_EXCERPT_MAX_BYTES - 14,
+ crate::payload::POST_COMPACTION_EXCERPT_MAX_BYTES,
+ )
+ .await;
+ }
+
+ /// The tool excerpt surface: the cap runs on the whole durable body
+ /// (`tool: …\n---\n…`), so the padding lands inside the tool result.
+ #[tokio::test]
+ async fn issue_1114_tool_body_is_sanitized_before_the_2kb_cap() {
+ let tmp = TempDir::new().unwrap();
+ let mut state = make_state(&tmp).await;
+ state.sanitizer = Sanitizer::new(&SanitizeConfig {
+ extra_patterns: vec![r"SECRET[0-9A-Za-z]{20,}".into()],
+ allowlist: Vec::new(),
+ })
+ .unwrap();
+ let secret = format!("SECRET{}", "Z".repeat(30));
+ let sid = SessionId::new();
+ let env = HookEnvelope::from_query_and_body(
+ HookQuery {
+ event: "PostToolUse".into(),
+ agent: Some("claude-code".into()),
+ ..Default::default()
+ },
+ serde_json::json!({
+ "session_id": sid.to_string(),
+ "cwd": "/repo",
+ "tool_name": "Read",
+ "tool_use_id": "call-1114",
+ "tool_response": format!("{} {secret}", "x".repeat(1_940)),
+ }),
+ );
+ process(&state, env, None, Vec::new()).await.unwrap();
+
+ let observations = state.reader.observations_for_session(sid).await.unwrap();
+ let body = observations
+ .iter()
+ .find(|o| o.kind == ObservationKind::PostToolUse)
+ .expect("post-tool-use observation was recorded")
+ .body
+ .clone();
+ assert!(
+ !body.contains(&secret) && !body.contains("SECRETZ"),
+ "unredacted secret fragment survived the cap: {body:?}"
+ );
+ assert!(body.contains("REDACT"), "no redaction marker: {body:?}");
+ assert!(
+ body.len() <= crate::payload::TOOL_EXCERPT_MAX_BYTES,
+ "body exceeded the 2 KB cap: {body:?}"
+ );
+ }
+
+ /// The per-event caps keep splitting on UTF-8 boundaries through the
+ /// funnel (`truncate_utf8_bytes`), so a multibyte tool result is capped
+ /// without producing invalid bytes.
+ #[tokio::test]
+ async fn tool_body_cap_keeps_the_utf8_boundary() {
+ let tmp = TempDir::new().unwrap();
+ let state = make_state(&tmp).await;
+ let sid = SessionId::new();
+ let env = HookEnvelope::from_query_and_body(
+ HookQuery {
+ event: "PostToolUse".into(),
+ agent: Some("claude-code".into()),
+ ..Default::default()
+ },
+ serde_json::json!({
+ "session_id": sid.to_string(),
+ "cwd": "/repo",
+ "tool_name": "Read",
+ "tool_use_id": "call-utf8",
+ "tool_response": "é".repeat(2_000), // 2 bytes each: 2x the cap
+ }),
+ );
+ process(&state, env, None, Vec::new()).await.unwrap();
+
+ let observations = state.reader.observations_for_session(sid).await.unwrap();
+ let body = observations
+ .iter()
+ .find(|o| o.kind == ObservationKind::PostToolUse)
+ .expect("post-tool-use observation was recorded")
+ .body;
+ assert!(body.contains('é'));
+ assert!(
+ body.len() <= crate::payload::TOOL_EXCERPT_MAX_BYTES,
+ "body exceeded the 2 KB cap: {body:?}"
+ );
+ assert!(body.ends_with('…'));
+ }
+
#[test]
fn codex_native_patch_capture_backstop_discards_unproven_output() {
for protocol in [
diff --git a/docs/security-boundaries.md b/docs/security-boundaries.md
index 3f43e2d6..71293742 100644
--- a/docs/security-boundaries.md
+++ b/docs/security-boundaries.md
@@ -49,7 +49,7 @@ boundary not yet built.
| 6 | Active-project pointer (PerActor, no clobber) | `ai-memory-core/src/active_project.rs` `set_for`/`lookup_for` (fail-closed on `Mismatch`) | `active_project.rs` `parallel_harnesses_of_one_user_keep_separate_pointers`, `two_operators_never_read_each_others_pointer`, `a_session_mismatch_fails_closed_once_anything_has_been_keyed` | STRONG |
| 6b | Native `_meta` routing coordinate is routing-only (never grants identity) | `ai-memory-mcp/src/server.rs` `attach_native_session` reads only `_meta["ai.opencode/sessionID"]` into `ActorKey.session_id` (trimmed, non-empty-string-validated); `user` still comes from the authenticated `ActorContext`, so a forged `_meta` cannot cross a user/scope boundary — it only selects a session slot behind the existing `active_project` guard | `server.rs` `native_session_metadata_is_routing_only_and_preserves_precedence`, `native_session_metadata_validates_strings_without_granting_identity`; `autoscope_multiuser.rs` `native_metadata_routes_concurrent_reads_and_writes_to_hook_workspaces` (#864) | STRONG |
| 6c | Per-user slot namespaces stay per operator (`[slots] per_user`) | `ai-memory-mcp/src/server.rs` `place_slot_mutation`, shared by `place_slot_write` (`memory_write_page`) and `place_slot_delete` (`memory_delete_page`): a non-admin's generic `_slots/<name>.md` is redirected to their own `_slots/u-<operator>/` namespace, a path inside another operator's namespace is refused, admins bypass, and nothing changes with the flag off (#1083) | `slot_identity.rs` `oidc_operator_cannot_delete_another_operators_personal_slot`, `oidc_operator_deleting_shared_slot_path_deletes_their_personal_slot` (shared slot survives), legitimate controls `oidc_operator_can_delete_their_own_personal_slot_explicitly`, `admin_can_delete_shared_and_personal_slots` | STRONG |
-| 7 | Sanitizer trust boundary (invariant #6) | `ai-memory-core/src/sanitize.rs` `Sanitized<T>` (private field, only `sanitize()` ctor); `Sanitized::new` scrubs the title *before* the 80-char display cap; `ai-memory-mcp/src/server.rs` `sanitize_feedback_reason` scrubs the reason *before* the 500-char cap; `WriterHandle::insert_observation` requires `Sanitized`; `ops` crate-private | Structural (compile-time) + `store/src/lib.rs` `insert_observation_boundary_scrubs_before_disk` + sanitizer scrub unit tests + `ai-memory-core::sanitize` `title_is_scrubbed_before_the_80_char_cap_runs` + `ai-memory-hooks::router` `issue_980_user_prompt_title_is_sanitized_before_truncated` (a secret straddling the 80-char boundary is redacted, not stored truncated; #980) + `ai-memory-mcp::server` `feedback_reason_is_scrubbed_before_the_500_char_cap` (#1109) | STRONG (structural) |
+| 7 | Sanitizer trust boundary (invariant #6) | `ai-memory-core/src/sanitize.rs` `Sanitized<T>` (private field, only `sanitize()` ctor); `Sanitized::new` scrubs the title *before* the 80-char display cap; `ai-memory-mcp/src/server.rs` `sanitize_feedback_reason` scrubs the reason *before* the 500-char cap; `ai-memory-hooks/src/router.rs` `durable_body` scrubs the full excerpt *before* the per-event body caps (`payload::durable_body_cap`), which no longer run at excerpt extraction; `WriterHandle::insert_observation` requires `Sanitized`; `ops` crate-private | Structural (compile-time) + `store/src/lib.rs` `insert_observation_boundary_scrubs_before_disk` + sanitizer scrub unit tests + `ai-memory-core::sanitize` `title_is_scrubbed_before_the_80_char_cap_runs` + `ai-memory-hooks::router` `issue_980_user_prompt_title_is_sanitized_before_truncated` (a secret straddling the 80-char boundary is redacted, not stored truncated; #980) + `ai-memory-mcp::server` `feedback_reason_is_scrubbed_before_the_500_char_cap` (#1109) + `ai-memory-hooks::router` `issue_1114_user_prompt_body_is_sanitized_before_the_16kib_cap`, `issue_1114_post_compaction_body_is_sanitized_before_the_16kib_cap`, `issue_1114_tool_body_is_sanitized_before_the_2kb_cap`, `tool_body_cap_keeps_the_utf8_boundary` (each places a secret straddling the cap and asserts a `[REDACTED:…]` marker survives with no raw fragment; #1114) | STRONG (structural) |
| 7a | Eval observations: server provenance and reason privacy | `ai-memory-consolidate/src/auto_improve.rs` constructs private results from the existing execution, ignores extra model/scorer result fields and all eval results on report deserialization, hashes the evaluated request/bodies, and scrubs reasons before the 2,048-character cap and pending persistence; changed bodies cannot receive the earlier result | `auto_improve::tests::structured_eval_records_server_observation`; `tests/suite/eval_results.rs` `eval_reason_is_sanitized_before_pending_storage_with_legitimate_control` attempts forged deserialized reports and IDs/digests and malicious reasons with secrets/control characters, checks report and stored evidence before sidecar rendering, and preserves a legitimate reason and external evidence without independent results; fails with reason scrubbing or the report deserialization guard removed | STRONG |
| 7b | New public page metadata is bounded and cannot supply scope, author or admission controls; admin kind retains its legacy adapter | `ai-memory-core/src/page.rs` `PageWriteMetadata::into_frontmatter`; MCP validates all four fields; admin consumes legacy kind explicitly and shares entities/abstract/relations validation before scope creation; both use `Wiki::write_page` for sanitization/admission/authenticated attribution | `admin_write_page.rs` `write_page_metadata_rejects_invalid_payload_before_scope_creation` (invalid shapes, unknown relation kinds, raw-input over-limit values, untrimmed relation scopes and escaping paths refused; legitimate control admitted), `write_page_metadata_preserves_admin_auth_and_sanitization` (anonymous/DB-user refused, root control admitted with scrubbed metadata and authenticated attribution); `server.rs` `memory_write_page_metadata_rejects_invalid_before_scope_creation`, `memory_write_page_metadata_preserves_scope_author_and_sanitization` (foreign user denied; writer control admitted, forged frontmatter/author ignored, secrets scrubbed, invalid rewrite leaves disk and SQL latest unchanged); core `page_write_metadata_scope_components_refuse_surrounding_whitespace` and `page_write_metadata_limits_apply_before_trimming_and_normalization`; admin `write_page_metadata_admin_legacy_kind_matches_base` compares long/control-character kind behavior with the base adapter; entity-bound refusal bite-checked with its validation removed | STRONG |
| 8a | Messaging: recipient-only visibility | `ai-memory-store/src/ops.rs` pop target-select + `reader.rs` `list_messages` (`to_*` predicate) | `agent_messages.rs` `a_message_is_only_visible_to_its_recipient`; `agent_messages_tools.rs` non-recipient cannot pop | STRONG |