Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

feat: share bounded companion capture privacy validation

Éverton Toffanettod7ba81976c35
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a253c87b..f3b0a53c 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -156,7 +156,7 @@ jobs:
strategy:
fail-fast: false
matrix:
- companion: [ai-memory-importer, ai-memory-relay]
+ companion: [ai-memory-client, ai-memory-importer, ai-memory-relay]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -355,6 +355,7 @@ jobs:
--ignore RUSTSEC-2024-0320
- run: cargo audit --file companions/ai-memory-relay/Cargo.lock
- run: cargo audit --file companions/ai-memory-importer/Cargo.lock
+ - run: cargo audit --file companions/ai-memory-client/Cargo.lock
gitleaks:
name: gitleaks (secret scan)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 762d7fbb..6670edd8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
+- Added shared private capture privacy helpers used by the relay and conversation
+ importer before body hashes, local persistence, previews and delivery. Sensitive
+ native identities were refused without renaming; nested credentials were scrubbed
+ with bounded body validation and an exact numeric token-metric allowlist. Unsafe
+ legacy relay heads retained their bytes and retry state while deferring only their
+ session, and importer byte caps preserved complete redaction markers. (#823)
- Added `install-hooks --agent claude-code --scope project`, which writes the
hook configuration to the checkout's gitignored `.claude/settings.local.json`
(where Claude Code reads it: the git root, or the launch directory on
diff --git a/companions/ai-memory-client/Cargo.lock b/companions/ai-memory-client/Cargo.lock
new file mode 100644
index 00000000..61644056
--- /dev/null
+++ b/companions/ai-memory-client/Cargo.lock
@@ -0,0 +1,151 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 4
+
+[[package]]
+name = "aho-corasick"
+version = "1.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "ai-memory-client"
+version = "0.1.0"
+dependencies = [
+ "anyhow",
+ "regex",
+ "serde_json",
+]
+
+[[package]]
+name = "anyhow"
+version = "1.0.104"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
+
+[[package]]
+name = "itoa"
+version = "1.0.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
+
+[[package]]
+name = "memchr"
+version = "2.8.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
+
+[[package]]
+name = "proc-macro2"
+version = "1.0.107"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
+dependencies = [
+ "unicode-ident",
+]
+
+[[package]]
+name = "quote"
+version = "1.0.47"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
+dependencies = [
+ "proc-macro2",
+]
+
+[[package]]
+name = "regex"
+version = "1.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.4.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-syntax"
+version = "0.8.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
+
+[[package]]
+name = "serde"
+version = "1.0.229"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
+dependencies = [
+ "serde_core",
+]
+
+[[package]]
+name = "serde_core"
+version = "1.0.229"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
+dependencies = [
+ "serde_derive",
+]
+
+[[package]]
+name = "serde_derive"
+version = "1.0.229"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn",
+]
+
+[[package]]
+name = "serde_json"
+version = "1.0.151"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
+dependencies = [
+ "itoa",
+ "memchr",
+ "serde",
+ "serde_core",
+ "zmij",
+]
+
+[[package]]
+name = "syn"
+version = "3.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "unicode-ident",
+]
+
+[[package]]
+name = "unicode-ident"
+version = "1.0.26"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"
+
+[[package]]
+name = "zmij"
+version = "1.0.23"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
diff --git a/companions/ai-memory-client/Cargo.toml b/companions/ai-memory-client/Cargo.toml
new file mode 100644
index 00000000..695f0615
--- /dev/null
+++ b/companions/ai-memory-client/Cargo.toml
@@ -0,0 +1,17 @@
+[workspace]
+
+[package]
+name = "ai-memory-client"
+version = "0.1.0"
+edition = "2024"
+publish = false
+license = "MIT"
+description = "Shared ingress privacy contract for ai-memory companions"
+
+[lints.rust]
+unsafe_code = "forbid"
+
+[dependencies]
+anyhow = "1"
+regex = "1"
+serde_json = "1"
diff --git a/companions/ai-memory-client/README.md b/companions/ai-memory-client/README.md
new file mode 100644
index 00000000..a2680ed5
--- /dev/null
+++ b/companions/ai-memory-client/README.md
@@ -0,0 +1,35 @@
+# ai-memory-client
+
+Private capture privacy helpers shared by the standalone relay and conversation
+importer. This package has its own Cargo workspace and is not published to crates.io.
+
+The four public functions have shipped callers:
+
+| Function | Caller and purpose |
+| --- | --- |
+| `reject_sensitive` | Relay binding and native event identities; importer source/native identities and delivery URLs. Rejects credentials and unsafe controls without renaming identities. |
+| `sanitize_external_text` | Importer conversation text before transcript hashes, event bodies, previews and manifests. |
+| `sanitize_external_value` | Relay ingress and importer event planning before body serialization, hashing or persistence. Recursively scrubs strings and opaque credential values. |
+| `check_body` | Both companions before delivery; relay before enqueue and importer before event hashing. Refuses unsafe content and top-level destination or actor authority. |
+
+Credential keys accept camelCase, acronym and separated spellings, including
+plural API, access, private and session key pairs. Only numeric values under
+these 11 normalized names retain their representation: `max_tokens`,
+`input_tokens`, `token_count`, `token_usage`, `output_tokens`, `total_tokens`,
+`cache_read_input_tokens`, `cache_creation_input_tokens`, `prompt_tokens`,
+`completion_tokens`, and `reasoning_tokens`. Strings or objects under these
+names are redacted. Other credential-like numeric fields are redacted too.
+
+Values are bounded to 64 nesting levels and individual strings to 256 KiB.
+Checked bodies must be objects and their serialized size cannot exceed 256 KiB.
+Complete redaction markers survive repeated scrubbing. Importer truncation
+preserves UTF-8 boundaries and drops a marker that would be cut in half.
+
+Native identity validation precedes sanitation. Already-pending relay bytes,
+digest, retry key and attempt state are never rewritten. An unsafe pending head
+defers only its session; clean sessions still advance. Body provenance remains
+untrusted data, and server scope, authorization and capture policy remain
+authoritative. Producers must apply capture exclusions before using the relay.
+
+Run this package's format, clippy and test checks separately, as with the other
+companions. Root workspace checks do not cover its tests.
diff --git a/companions/ai-memory-client/src/lib.rs b/companions/ai-memory-client/src/lib.rs
new file mode 100644
index 00000000..ff8ed040
--- /dev/null
+++ b/companions/ai-memory-client/src/lib.rs
@@ -0,0 +1,122 @@
+//! Privacy validation and ingress sanitation shared by both capture companions.
+
+use anyhow::{Result, bail};
+use serde_json::Value;
+
+mod scrub;
+pub use scrub::sanitize_external_text;
+
+/// Reject credentials and control characters without changing retry identities.
+pub fn reject_sensitive(value: &Value) -> Result<()> {
+ bounded(value, 0)?;
+ let mut safe = value.clone();
+ sanitize_external_value(&mut safe)?;
+ if safe != *value {
+ bail!("sensitive envelope rejected before persistence or delivery");
+ }
+ Ok(())
+}
+
+/// Sanitize new payload values before persistence; callers validate identities first.
+pub fn sanitize_external_value(value: &mut Value) -> Result<()> {
+ bounded(value, 0)?;
+ sanitize_value(value)
+}
+
+fn bounded(value: &Value, depth: usize) -> Result<()> {
+ if depth > 64 {
+ bail!("capture value exceeds nesting limit");
+ }
+ match value {
+ Value::String(text) if text.len() > 256 * 1024 => bail!("capture text exceeds byte limit"),
+ Value::Array(items) => items.iter().try_for_each(|item| bounded(item, depth + 1))?,
+ Value::Object(fields) => {
+ for (key, value) in fields {
+ bounded(&Value::String(key.clone()), depth + 1)?;
+ bounded(value, depth + 1)?;
+ }
+ }
+ _ => {}
+ }
+ Ok(())
+}
+
+fn sanitize_value(value: &mut Value) -> Result<()> {
+ match value {
+ Value::String(text) => *text = sanitize_external_text(text),
+ Value::Array(items) => items.iter_mut().try_for_each(sanitize_value)?,
+ Value::Object(fields) => {
+ for (key, value) in fields {
+ reject_sensitive(&Value::String(key.clone()))?;
+ if sensitive_key(key, value) {
+ *value = Value::String("[REDACTED]".into());
+ } else {
+ sanitize_value(value)?;
+ }
+ }
+ }
+ _ => {}
+ }
+ Ok(())
+}
+
+fn sensitive_key(key: &str, value: &Value) -> bool {
+ let chars: Vec<_> = key.chars().collect();
+ let mut name = String::new();
+ for (i, ch) in chars.iter().enumerate() {
+ if ch.is_ascii_uppercase()
+ && i > 0
+ && (chars[i - 1].is_ascii_lowercase()
+ || (chars[i - 1].is_ascii_uppercase()
+ && chars.get(i + 1).is_some_and(char::is_ascii_lowercase)))
+ {
+ name.push('_');
+ }
+ name.push(if ch.is_ascii_alphanumeric() {
+ ch.to_ascii_lowercase()
+ } else {
+ '_'
+ });
+ }
+ let metrics = concat!(
+ "max_tokens input_tokens token_count token_usage ",
+ "output_tokens total_tokens cache_read_input_tokens cache_creation_input_tokens ",
+ "prompt_tokens completion_tokens reasoning_tokens"
+ );
+ let is_metric = metrics.split(' ').any(|metric| metric == name);
+ if value.is_number() && is_metric {
+ return false;
+ }
+ let parts: Vec<_> = name.split('_').filter(|part| !part.is_empty()).collect();
+ let sensitive = concat!(
+ "authorization auth jwt pwd bearer password passwd secret token cookie ",
+ "credential apikey privatekey accesskey signature"
+ );
+ sensitive
+ .split_ascii_whitespace()
+ .any(|word| parts.iter().any(|part| part.trim_end_matches('s') == word))
+ || parts.windows(2).any(|pair| {
+ ["api", "private", "access", "session"].contains(&pair[0])
+ && matches!(pair[1], "key" | "keys")
+ })
+}
+
+/// Reject body-supplied authority and payloads requiring further sanitation.
+pub fn check_body(body: &Value) -> Result<()> {
+ if !body.is_object()
+ || ["workspace", "project", "actor", "author_id", "headers"]
+ .iter()
+ .any(|key| body.get(key).is_some())
+ {
+ bail!("hook body cannot supply destination or actor authority");
+ }
+ reject_sensitive(body)?;
+ if serde_json::to_vec(body)?.len() > 256 * 1024 {
+ bail!("capture body exceeds byte limit");
+ }
+ Ok(())
+}
+
+#[cfg(test)]
+#[path = "../tests/suite/mod.rs"]
+mod tests;
diff --git a/companions/ai-memory-client/src/scrub.rs b/companions/ai-memory-client/src/scrub.rs
new file mode 100644
index 00000000..bb9f4d3f
--- /dev/null
+++ b/companions/ai-memory-client/src/scrub.rs
@@ -0,0 +1,84 @@
+//! Local credential scrub shared by external capture clients.
+
+use regex::Regex;
+use std::sync::OnceLock;
+
+pub fn sanitize_external_text(input: &str) -> String {
+ let controls_removed: String = input
+ .chars()
+ .map(|ch| {
+ if ch == '\n'
+ || ch == '\r'
+ || ch == '\t'
+ || !(ch.is_control()
+ || matches!(ch, '\u{202a}'..='\u{202e}' | '\u{2066}'..='\u{2069}'))
+ {
+ ch
+ } else {
+ '\u{fffd}'
+ }
+ })
+ .collect();
+ secret_patterns()
+ .iter()
+ .fold(controls_removed, |text, rule| {
+ rule.regex
+ .replace_all(&text, |captures: &regex::Captures<'_>| {
+ let matched = captures.get(0).map_or("", |matched| matched.as_str());
+ if captures.get(3).is_some_and(|value| {
+ value.as_str().chars().count() < 6
+ || [
+ "[REDACTED]",
+ "[REDACTED CREDENTIAL]",
+ "[REDACTED PRIVATE KEY]",
+ ]
+ .contains(&value.as_str())
+ }) {
+ matched.to_owned()
+ } else {
+ let mut replacement = String::new();
+ captures.expand(rule.replacement, &mut replacement);
+ replacement
+ }
+ })
+ .into_owned()
+ })
+}
+
+struct RedactionRule {
+ regex: Regex,
+ replacement: &'static str,
+}
+
+fn secret_patterns() -> &'static [RedactionRule] {
+ static RULES: OnceLock<Vec<RedactionRule>> = OnceLock::new();
+ RULES.get_or_init(|| {
+ vec![
+ RedactionRule {
+ regex: Regex::new(
+ r"(?s)-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----",
+ )
+ .unwrap(),
+ replacement: "[REDACTED PRIVATE KEY]",
+ },
+ RedactionRule {
+ regex: Regex::new(
+ r"\b(?:github_pat_[A-Za-z0-9_]{20,}|gh[pousr]_[A-Za-z0-9]{20,}|sk-[A-Za-z0-9_-]{16,}|AKIA[0-9A-Z]{16})\b",
+ )
+ .unwrap(),
+ replacement: "[REDACTED CREDENTIAL]",
+ },
+ RedactionRule {
+ regex: Regex::new(r"(?i)\b(bearer\s+)[A-Za-z0-9._~+/=-]{8,}").unwrap(),
+ replacement: "$1[REDACTED]",
+ },
+ RedactionRule {
+ regex: Regex::new(
+ r#"(?i)\b(api[_-]?key|access[_-]?token|auth[_-]?token|token|secret|password)\b(\s*[:=]\s*)(\"[^\"\r\n]{6,}\"|'[^'\r\n]{6,}'|(?:\[REDACTED(?: CREDENTIAL| PRIVATE KEY)?\]|[^\s,;])+)"#,
+ )
+ .unwrap(),
+ replacement: "$1$2[REDACTED]",
+ },
+ ]
+ })
+}
diff --git a/companions/ai-memory-client/tests/suite/mod.rs b/companions/ai-memory-client/tests/suite/mod.rs
new file mode 100644
index 00000000..5a7b45ce
--- /dev/null
+++ b/companions/ai-memory-client/tests/suite/mod.rs
@@ -0,0 +1,337 @@
+use super::*;
+use serde_json::json;
+
+#[test]
+fn bounded_body_schema_refuses_authority_depth_and_size_with_control() {
+ let control = json!({"session_id":"sessão-界", "cwd":"/work", "_ai_memory_capture":{"actor":"untrusted provenance"}});
+ check_body(&control).unwrap();
+ assert!(check_body(&json!([control.clone()])).is_err());
+ for field in ["workspace", "project", "actor", "author_id", "headers"] {
+ let mut attack = control.clone();
+ attack[field] = json!("forged");
+ assert!(
+ check_body(&attack).is_err(),
+ "body authority must be refused"
+ );
+ }
+ let mut deep = control.clone();
+ for _ in 0..65 {
+ deep = json!({"nested": deep});
+ }
+ assert!(check_body(&deep).is_err(), "deep bodies must be refused");
+ let large = json!({"first":"x".repeat(140_000),"second":"x".repeat(140_000)});
+ assert!(
+ check_body(&large).is_err(),
+ "body byte cap must be enforced"
+ );
+ let mut large_text = json!("x".repeat(256 * 1024 + 1));
+ assert!(sanitize_external_value(&mut large_text).is_err());
+ check_body(&control).unwrap();
+}
+
+#[test]
+fn complete_literal_redaction_markers_survive_repeated_scrubbing() {
+ for marker in [
+ "[REDACTED]",
+ "[REDACTED CREDENTIAL]",
+ "[REDACTED PRIVATE KEY]",
+ ] {
+ let control = format!("界 token={marker} literal {marker}");
+ assert_eq!(sanitize_external_text(&control), control);
+ assert_eq!(
+ sanitize_external_text(&sanitize_external_text(&control)),
+ control
+ );
+ }
+}
+
+#[test]
+fn secrets_are_rejected_in_all_envelope_positions_without_exposing_them() {
+ for secret in [
+ "Bearer abcdefghijklmnop",
+ "github_pat_1234567890abcdefghijklmnop",
+ "password=abcdefghi",
+ "unsafe\u{1b}text",
+ ] {
+ assert_ne!(sanitize_external_text(secret), secret);
+ for envelope in [
+ json!(secret),
+ json!({"session_id": secret}),
+ json!({"cwd": secret}),
+ json!({"source": secret}),
+ json!({"metadata": [secret]}),
+ ] {
+ let error = reject_sensitive(&envelope).unwrap_err();
+ assert!(!error.to_string().contains(secret));
+ }
+ }
+ assert!(reject_sensitive(&json!({"nested": {"api_key": "short"}})).is_err());
+ assert!(
+ reject_sensitive(
+ &json!({"session_id": "native", "cwd": "/work", "prompt": "ordinary text"})
+ )
+ .is_ok()
+ );
+}
+
+#[test]
+fn redacted_nested_values_validate_idempotently() {
+ let body = json!({"session_id": "native", "cwd": "/work", "nested": {"token": "[REDACTED]", "api-key": "[REDACTED]"}});
+ reject_sensitive(&body).unwrap();
+ check_body(&body).unwrap();
+ let mut again = body.clone();
+ sanitize_external_value(&mut again).unwrap();
+ assert_eq!(again, body);
+}
+
+#[test]
+fn sanitation_keeps_native_identity_and_refuses_body_authority() {
+ let mut body = json!({"session_id":"native", "cwd":"/work", "output":"\u{1b}[32m token=getToken() Bearer abcdefghijklmnop", "nested":{"password":"process.env.X"}});
+ sanitize_external_value(&mut body).unwrap();
+ assert_eq!(body["session_id"], "native");
+ assert_eq!(body["cwd"], "/work");
+ assert_eq!(body["nested"]["password"], "[REDACTED]");
+ assert!(!body["output"].as_str().unwrap().contains(['\u{1b}']));
+ check_body(&body).unwrap();
+ for field in ["workspace", "project", "actor", "author_id", "headers"] {
+ let mut forged = body.clone();
+ forged[field] = "foreign".into();
+ assert!(check_body(&forged).is_err());
+ }
+}
+
+#[test]
+fn sensitive_key_forms_redact_opaque_values_and_preserve_numeric_metrics() {
+ let keys = [
+ "accessToken",
+ "refreshToken",
+ "clientSecret",
+ "privateKey",
+ "x-api-key",
+ "OPENAI_API_KEY",
+ "accessKey",
+ "HTTPAuthorization",
+ "cookie",
+ "sessionCookies",
+ "credential",
+ "credentials",
+ "password",
+ "token_count",
+ "token_usage",
+ "max_tokens",
+ "input_tokens",
+ ];
+ for key in keys {
+ let raw = json!({"nested": [{key: "fixture-value"}]});
+ assert_eq!(sanitize_external_text("fixture-value"), "fixture-value");
+ assert!(reject_sensitive(&raw).is_err());
+ let mut safe = raw;
+ sanitize_external_value(&mut safe).unwrap();
+ assert_eq!(safe["nested"][0][key], "[REDACTED]");
+ reject_sensitive(&safe).unwrap();
+ let once = safe.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert_eq!(safe, once);
+ }
+ let metrics = json!({"max_tokens": 12, "input_tokens": 123456789012345_u64, "token_count": 0, "token_usage": 1.25});
+ let mut safe = metrics.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert_eq!(safe, metrics);
+ assert_eq!(
+ serde_json::to_string(&safe).unwrap(),
+ serde_json::to_string(&metrics).unwrap()
+ );
+ reject_sensitive(&safe).unwrap();
+}
+
+#[test]
+fn numeric_token_metrics_preserve_representation_without_credential_bypass() {
+ let metrics = json!({"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25,"output_tokens":9007199254740993_u64,"total_tokens":89,"totalTokens":89,"cache_read_input_tokens":2,"cache_creation_input_tokens":3,"prompt_tokens":5,"completion_tokens":7,"reasoning_tokens":11});
+ let raw = json!({"usage":metrics,"nested":[metrics.clone()]});
+ let mut safe = raw.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(
+ safe == raw,
+ "numeric metric values and types must survive sanitation"
+ );
+ assert!(serde_json::to_vec(&safe).unwrap() == serde_json::to_vec(&raw).unwrap());
+ reject_sensitive(&safe).unwrap();
+ let once = safe.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe == once);
+ for key in metrics.as_object().unwrap().keys() {
+ let mut opaque = json!({key:"fixture-value"});
+ sanitize_external_value(&mut opaque).unwrap();
+ assert!(
+ opaque[key] == "[REDACTED]",
+ "opaque metric values must be redacted"
+ );
+ }
+ for key in [
+ "token",
+ "access_tokens",
+ "private_token_count",
+ "auth_token_usage",
+ "authorization_token_limit",
+ "private_key",
+ "accessKey",
+ ] {
+ let mut credential = json!({key:123456});
+ sanitize_external_value(&mut credential).unwrap();
+ assert!(
+ credential[key] == "[REDACTED]",
+ "numeric credential values must be redacted"
+ );
+ }
+ let mut usage_object = json!({"token_usage":{"input_tokens":12,"output_tokens":34}});
+ sanitize_external_value(&mut usage_object).unwrap();
+ assert!(usage_object["token_usage"] == "[REDACTED]");
+}
+
+#[test]
+fn common_credential_components_redact_opaque_values_with_author_controls() {
+ for key in [
+ "auth",
+ "jwt",
+ "pwd",
+ "bearer",
+ "session_key",
+ "signature",
+ "clientAuth",
+ "HTTP_JWT",
+ "HTTPJwt",
+ "JWT",
+ "client-pwd",
+ "Bearer",
+ "sessionKey",
+ "Session-Key",
+ "requestSignature",
+ ] {
+ for value in [json!("fixture-value"), json!(123456)] {
+ let raw = json!({"nested":[{key:value}]});
+ assert!(
+ reject_sensitive(&raw).is_err(),
+ "raw credential component must be refused"
+ );
+ let mut safe = raw;
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe["nested"][0][key] == "[REDACTED]");
+ reject_sensitive(&safe).unwrap();
+ let once = safe.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe == once);
+ }
+ }
+ let control =
+ json!({"author":"fixture-value","authority":"fixture-value","coauthor":"fixture-value"});
+ assert!(sanitize_external_text("fixture-value") == "fixture-value");
+ let mut safe = control.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(
+ safe == control,
+ "author and authority are legitimate components"
+ );
+ reject_sensitive(&safe).unwrap();
+}
+
+#[test]
+fn session_key_pairs_are_sensitive_with_native_session_metadata_control() {
+ for key in [
+ "session_key",
+ "sessionKey",
+ "Session-Key",
+ "HTTP_SESSION_KEY",
+ ] {
+ for value in [json!("fixture-value"), json!(123456)] {
+ let raw = json!({"nested":[{key:value}]});
+ assert!(reject_sensitive(&raw).is_err());
+ let mut safe = raw;
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe["nested"][0][key] == "[REDACTED]");
+ reject_sensitive(&safe).unwrap();
+ let once = safe.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe == once);
+ }
+ }
+ let control = json!({"session":"fixture-value","session_id":"native","key":"fixture-value"});
+ let mut safe = control.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe == control);
+ reject_sensitive(&safe).unwrap();
+}
+
+#[test]
+fn plural_credential_key_pairs_redact_opaque_values_recursively() {
+ assert!(sanitize_external_text("fixture-value") == "fixture-value");
+ for key in [
+ "api_keys",
+ "access_keys",
+ "private_keys",
+ "session_keys",
+ "apiKeys",
+ "accessKeys",
+ "privateKeys",
+ "sessionKeys",
+ "API_KEYS",
+ "session-keys",
+ ] {
+ for value in [
+ json!("fixture-value"),
+ json!(["fixture-value", {"label":"opaque-value"}]),
+ ] {
+ let raw = json!({"nested":[{"details":{key:value}}]});
+ assert!(
+ reject_sensitive(&raw).is_err(),
+ "plural key pair must be refused before sanitation"
+ );
+ let mut safe = raw;
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe["nested"][0]["details"][key] == "[REDACTED]");
+ check_body(&safe).unwrap();
+ let once = safe.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe == once);
+ }
+ }
+}
+
+#[test]
+fn plural_key_guard_preserves_lone_keys_native_ids_and_metrics() {
+ let control = json!({
+ "key":"fixture-value", "keys":["fixture-value"],
+ "session":"fixture-value", "session_id":"native-session", "event_id":"native-event",
+ "author":"fixture-value", "authority":"fixture-value",
+ "metrics":{"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25,"output_tokens":9007199254740993_u64,"total_tokens":89,"totalTokens":89,"cache_read_input_tokens":2,"cache_creation_input_tokens":3,"prompt_tokens":5,"completion_tokens":7,"reasoning_tokens":11}
+ });
+ let bytes = serde_json::to_vec(&control).unwrap();
+ check_body(&control).unwrap();
+ let mut safe = control.clone();
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe == control);
+ assert!(serde_json::to_vec(&safe).unwrap() == bytes);
+ sanitize_external_value(&mut safe).unwrap();
+ assert!(safe == control);
+}
+
+#[test]
+fn redaction_marker_suffix_never_exempts_a_credential() {
+ for marker in [
+ "[REDACTED]",
+ "[REDACTED CREDENTIAL]",
+ "[REDACTED PRIVATE KEY]",
+ ] {
+ let control = format!("token={marker}");
+ assert_eq!(sanitize_external_text(&control), control);
+ for attack in [
+ format!("token=abcdefghijklmnop{marker}"),
+ format!("token={marker}abcdefghijklmnop"),
+ ] {
+ assert!(reject_sensitive(&json!(attack)).is_err());
+ assert_eq!(sanitize_external_text(&attack), "token=[REDACTED]");
+ let safe = sanitize_external_text(&attack);
+ assert_eq!(sanitize_external_text(&safe), safe);
+ }
+ assert_eq!(sanitize_external_text(&control), control);
+ }
+}
diff --git a/companions/ai-memory-importer/Cargo.lock b/companions/ai-memory-importer/Cargo.lock
index 9f10bb3f..54e87ace 100644
--- a/companions/ai-memory-importer/Cargo.lock
+++ b/companions/ai-memory-importer/Cargo.lock
@@ -11,13 +11,22 @@ dependencies = [
"memchr",
]
+[[package]]
+name = "ai-memory-client"
+version = "0.1.0"
+dependencies = [
+ "anyhow",
+ "regex",
+ "serde_json",
+]
+
[[package]]
name = "ai-memory-importer"
version = "0.1.0"
dependencies = [
+ "ai-memory-client",
"anyhow",
"clap",
- "regex",
"reqwest",
"serde",
"serde_json",
diff --git a/companions/ai-memory-importer/Cargo.toml b/companions/ai-memory-importer/Cargo.toml
index 0fa7d1e4..14d7caf3 100644
--- a/companions/ai-memory-importer/Cargo.toml
+++ b/companions/ai-memory-importer/Cargo.toml
@@ -7,12 +7,12 @@ edition = "2024"
publish = false
[dependencies]
+ai-memory-client = { path = "../ai-memory-client" }
anyhow = "1"
clap = { version = "4", features = ["derive", "env"] }
# Native roots, matching the workspace: the importer POSTs to the same server
# an operator may front with a private CA (#492).
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }
-regex = "1"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
serde_yaml = "0.9"
diff --git a/companions/ai-memory-importer/README.md b/companions/ai-memory-importer/README.md
index c5cd13bd..78882427 100644
--- a/companions/ai-memory-importer/README.md
+++ b/companions/ai-memory-importer/README.md
@@ -84,6 +84,13 @@ stubs for them.
extension messages. The dry-run and manifest report the truncation count.
- Conversation bodies receive client-side credential redaction before replay
and then cross ai-memory's normal server sanitizer as a second boundary.
+ The [shared private privacy helpers](../ai-memory-client) scrub nested values
+ before event hashes, previews, manifests and delivery. Sensitive native source
+ identities are refused before deriving an imported session ID or writing a
+ manifest. Valid native session strings are preserved exactly. UTF-8 byte caps
+ keep complete redaction markers, and numeric token metrics use an exact
+ 11-name allowlist. Body provenance remains untrusted and cannot supply actor
+ or destination authority.
- Stable session IDs derive from `(workspace, project, source, session_id)`.
Each replay event has a stable `ingest_key`; a changed transcript gets a new
terminal generation key, so rerunning an interrupted import is safe and
diff --git a/companions/ai-memory-importer/src/main.rs b/companions/ai-memory-importer/src/main.rs
index 382d812c..2a96a70c 100644
--- a/companions/ai-memory-importer/src/main.rs
+++ b/companions/ai-memory-importer/src/main.rs
@@ -1,12 +1,11 @@
use std::collections::HashMap;
use std::fs;
use std::path::{Component, Path, PathBuf};
-use std::sync::OnceLock;
use std::time::Duration;
+use ai_memory_client::sanitize_external_text;
use anyhow::{Context, Result, anyhow, bail};
use clap::{Parser, Subcommand};
-use regex::Regex;
use reqwest::StatusCode;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
@@ -527,6 +526,11 @@ fn plan_external_conversation(
server_url: &str,
) -> Result<ConversationPlan> {
validate_label(workspace, "workspace", 128)?;
+ ai_memory_client::reject_sensitive(&serde_json::json!([
+ workspace,
+ server_url,
+ file.to_string_lossy().as_ref()
+ ]))?;
let metadata = fs::metadata(file)
.with_context(|| format!("read conversation source {}", file.display()))?;
if !metadata.is_file() {
@@ -540,12 +544,18 @@ fn plan_external_conversation(
}
let bytes = fs::read(file).with_context(|| format!("read {}", file.display()))?;
let source_sha256 = sha256_hex(&bytes);
- let mut envelope: ConversationEnvelope =
- serde_json::from_slice(&bytes).context("parse generic conversation JSON")?;
+ let mut envelope: ConversationEnvelope = serde_json::from_slice(&bytes)
+ .map_err(|_| anyhow!("parse generic conversation JSON: invalid schema"))?;
+ ai_memory_client::reject_sensitive(&serde_json::json!({
+ "project": envelope.project, "source": envelope.source, "session_id": envelope.session_id,
+ }))?;
envelope.project = validate_label(&envelope.project, "project", 128)?.to_owned();
envelope.source = validate_source_name(&envelope.source)?.to_owned();
- envelope.session_id = validate_label(&envelope.session_id, "session_id", 256)?.to_owned();
+ if envelope.session_id.len() > 256 || envelope.session_id.chars().any(char::is_control) {
+ bail!("session_id must be bounded and free of control characters");
+ }
+ validate_label(&envelope.session_id, "session_id", 256)?;
if envelope.messages.is_empty() {
bail!("conversation must contain at least one message");
}
@@ -704,8 +714,16 @@ fn planned_hook_event(
event: &str,
role: Option<ConversationRole>,
source_event: Option<&str>,
- body: serde_json::Value,
+ mut body: serde_json::Value,
) -> Result<PlannedHookEvent> {
+ ai_memory_client::reject_sensitive(
+ &serde_json::json!({"session_id": stable_session_id, "event": event, "source_event": source_event}),
+ )?;
+ if let Some(id) = body.get("session_id") {
+ ai_memory_client::reject_sensitive(id)?;
+ }
+ ai_memory_client::sanitize_external_value(&mut body)?;
+ ai_memory_client::check_body(&body)?;
let body_bytes = serde_json::to_vec(&body)?;
let ingest_key = event_ingest_key(stable_session_id, index, event, &body_bytes);
let url = build_hook_url(
@@ -717,6 +735,7 @@ fn planned_hook_event(
&ingest_key,
source_event,
)?;
+ ai_memory_client::reject_sensitive(&serde_json::json!(url))?;
Ok(PlannedHookEvent {
index,
event: event.to_owned(),
@@ -864,65 +883,15 @@ fn truncate_utf8(input: &str, max_bytes: usize) -> String {
while end > 0 && !input.is_char_boundary(end) {
end -= 1;
}
+ if let Some(start) = input[..end].rfind('[')
+ && input[start..].starts_with("[REDACTED")
+ && !input[start..end].contains(']')
+ {
+ end = start;
+ }
input[..end].to_owned()
}
-fn sanitize_external_text(input: &str) -> String {
- let controls_removed: String = input
- .chars()
- .map(|ch| {
- if ch == '\n' || ch == '\r' || ch == '\t' || !ch.is_control() {
- ch
- } else {
- '\u{fffd}'
- }
- })
- .collect();
- secret_patterns()
- .iter()
- .fold(controls_removed, |text, rule| {
- rule.regex.replace_all(&text, rule.replacement).into_owned()
- })
-}
-
-struct RedactionRule {
- regex: Regex,
- replacement: &'static str,
-}
-
-fn secret_patterns() -> &'static [RedactionRule] {
- static RULES: OnceLock<Vec<RedactionRule>> = OnceLock::new();
- RULES.get_or_init(|| {
- vec![
- RedactionRule {
- regex: Regex::new(
- r"(?s)-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----",
- )
- .unwrap(),
- replacement: "[REDACTED PRIVATE KEY]",
- },
- RedactionRule {
- regex: Regex::new(
- r"\b(?:github_pat_[A-Za-z0-9_]{20,}|gh[pousr]_[A-Za-z0-9]{20,}|sk-[A-Za-z0-9_-]{16,}|AKIA[0-9A-Z]{16})\b",
- )
- .unwrap(),
- replacement: "[REDACTED CREDENTIAL]",
- },
- RedactionRule {
- regex: Regex::new(r"(?i)\b(bearer\s+)[A-Za-z0-9._~+/=-]{8,}").unwrap(),
- replacement: "$1[REDACTED]",
- },
- RedactionRule {
- regex: Regex::new(
- r#"(?i)\b(api[_-]?key|access[_-]?token|auth[_-]?token|token|secret|password)\b(\s*[:=]\s*)(?:\"[^\"\r\n]{6,}\"|'[^'\r\n]{6,}'|[^\s,;]{6,})"#,
- )
- .unwrap(),
- replacement: "$1$2[REDACTED]",
- },
- ]
- })
-}
-
fn conversation_manifest(plan: &ConversationPlan) -> ConversationManifest {
ConversationManifest {
import_version: CONVERSATION_IMPORT_VERSION.into(),
@@ -1355,6 +1324,10 @@ impl ImportClient {
}
async fn post_hook_batch(&self, events: &[PlannedHookEvent]) -> Result<HookBatchDelivery> {
+ for event in events {
+ ai_memory_client::check_body(&event.body)?;
+ ai_memory_client::reject_sensitive(&serde_json::json!(event.url))?;
+ }
let items: Vec<_> = events
.iter()
.map(|event| HookBatchItem {
@@ -1884,4 +1857,303 @@ mod tests {
.accepted_every_event(3)
);
}
+ #[tokio::test]
+ async fn shared_key_sanitation_precedes_plan_manifest_transport_and_exact_retry() {
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ let base = format!("http://{}", listener.local_addr().unwrap());
+ let mut raw = serde_json::json!({"session_id":"native", "cwd":"/work/app", "output":"\u{1b}[32mresult\u{1b}[0m\u{202e} token=getToken() Bearer abcdefghijklmnop", "nested":[{"accessToken":"fixture-value", "refreshToken":"fixture-value", "clientSecret":"fixture-value", "privateKey":"fixture-value", "x-api-key":"fixture-value", "OPENAI_API_KEY":"fixture-value", "cookie":"fixture-value", "credentials":"fixture-value"}], "metrics":{"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25}});
+ raw["metrics"] = serde_json::json!({"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25,"output_tokens":9007199254740993_u64,"total_tokens":89,"totalTokens":89,"cache_read_input_tokens":2,"cache_creation_input_tokens":3,"prompt_tokens":5,"completion_tokens":7,"reasoning_tokens":11});
+ raw["key_forms"] = serde_json::json!([{"auth":"fixture-value","jwt":"fixture-value","pwd":"fixture-value","bearer":"fixture-value","session_key":"fixture-value","signature":"fixture-value"}]);
+ raw["plural_forms"] = serde_json::json!([{
+ "api_keys":"fixture-value", "access_keys":"fixture-value", "private_keys":"fixture-value", "session_keys":"fixture-value",
+ "apiKeys":["fixture-value"], "accessKeys":["fixture-value"], "privateKeys":["fixture-value"], "sessionKeys":["fixture-value"]
+ }]);
+ raw["controls"] = serde_json::json!({"author":"fixture-value","authority":"fixture-value","key":"fixture-value","keys":["fixture-value"]});
+ raw["numeric_cases"] = serde_json::json!({"token":123456,"access_tokens":123456,"private_token_count":123456,"auth_token_usage":123456,"token_usage":{"input_tokens":12}});
+ let planned = || {
+ planned_hook_event(
+ &base,
+ "w",
+ "p",
+ "native",
+ 0,
+ "post-tool-use",
+ None,
+ None,
+ raw.clone(),
+ )
+ .unwrap()
+ };
+ let event = planned();
+ assert_eq!(event.body["session_id"], raw["session_id"]);
+ assert_eq!(event.body["cwd"], raw["cwd"]);
+ assert_eq!(
+ event.body["output"],
+ sanitize_external_text(raw["output"].as_str().unwrap())
+ );
+ assert_eq!(event.body["metrics"], raw["metrics"]);
+ assert!(
+ event.body["nested"][0]
+ .as_object()
+ .unwrap()
+ .values()
+ .all(|value| value == "[REDACTED]")
+ );
+ assert_eq!(
+ event.ingest_key,
+ event_ingest_key(
+ "native",
+ 0,
+ "post-tool-use",
+ &serde_json::to_vec(&event.body).unwrap()
+ )
+ );
+ let serialized = serde_json::to_string(&event).unwrap();
+ assert!(
+ event.body["key_forms"][0]
+ .as_object()
+ .unwrap()
+ .values()
+ .all(|value| value == "[REDACTED]")
+ );
+ assert!(
+ event.body["plural_forms"][0]
+ .as_object()
+ .unwrap()
+ .values()
+ .all(|value| value == "[REDACTED]")
+ );
+ assert!(
+ event.body["numeric_cases"]
+ .as_object()
+ .unwrap()
+ .values()
+ .all(|value| value == "[REDACTED]")
+ );
+ assert!(event.body["controls"] == raw["controls"]);
+ assert!(
+ serde_json::to_vec(&event.body["metrics"]).unwrap()
+ == serde_json::to_vec(&raw["metrics"]).unwrap()
+ );
+ assert!(!serialized.contains("abcdefghijklmnop"));
+ ai_memory_client::check_body(&event.body).unwrap();
+ let expected = event.body.clone();
+ let server = thread::spawn(move || {
+ let mut bodies = Vec::new();
+ for attempt in 0..2 {
+ let (mut socket, _) = listener.accept().unwrap();
+ let bytes = read_http_request(&mut socket).1;
+ let wire: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
+ assert_eq!(wire[0]["body"], expected);
+ bodies.push(bytes);
+ if attempt == 1 {
+ respond_json(&mut socket, "200 OK", r#"{"accepted":1}"#);
+ }
+ }
+ assert_eq!(bodies[0], bodies[1]);
+ });
+ let client = ImportClient::new(&base).unwrap();
+ assert!(client.post_hook_batch(&[event]).await.is_err());
+ let retry = planned();
+ assert_eq!(serde_json::to_string(&retry).unwrap(), serialized);
+ assert_eq!(
+ client.post_hook_batch(&[retry]).await.unwrap().ack.accepted,
+ 1
+ );
+ server.join().unwrap();
+ }
+
+ #[test]
+ fn sanitized_redaction_markers_survive_utf8_byte_caps_in_bodies_and_titles() {
+ let root = tempdir().unwrap().keep();
+ for (role, cap) in [
+ ("user", MAX_USER_MESSAGE_BYTES),
+ ("assistant", MAX_EXTENSION_MESSAGE_BYTES),
+ ] {
+ let prefix = format!("{}{} token=", "界", "x".repeat(cap - 16));
+ let source = serde_json::json!({"project":"p","source":"chatgpt","session_id":"native", "messages":[
+ {"role":"user", "content":"clean control"},
+ {"role":role, "content":format!("{prefix}abcdefghijklmnop\u{1b}[0m\u{202e}")},
+ {"role":"assistant", "content":format!("{} token=abcdefghijklmnop", "x".repeat(145))}
+ ]});
+ let file = write_conversation(&root, source);
+ let plan = plan_external_conversation(&file, "w", DEFAULT_SERVER_URL).unwrap();
+ for event in &plan.events {
+ ai_memory_client::check_body(&event.body).unwrap();
+ for field in ["prompt", "message", "title"] {
+ if let Some(text) = event.body[field].as_str() {
+ assert!(!text.contains("abcdefghijklmnop"));
+ assert!(!text.contains(['\u{1b}', '\u{202e}']));
+ if let Some(at) = text.rfind('[') {
+ assert!(
+ text[at..].contains(']'),
+ "redaction marker must remain whole"
+ );
+ }
+ }
+ }
+ }
+ let body = &plan.events[2].body;
+ let text = body[if role == "user" { "prompt" } else { "message" }]
+ .as_str()
+ .unwrap();
+ assert!(text.len() <= cap);
+ assert!(plan.events[3].body["title"].as_str().unwrap().len() <= 160);
+ let again = plan_external_conversation(&file, "w", DEFAULT_SERVER_URL).unwrap();
+ assert_eq!(plan.transcript_sha256, again.transcript_sha256);
+ assert_eq!(plan.events[2].ingest_key, again.events[2].ingest_key);
+ }
+ }
+
+ #[tokio::test]
+ async fn sensitive_native_identity_is_refused_before_manifest_and_http_with_unicode_control() {
+ let root = tempdir().unwrap();
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ listener.set_nonblocking(true).unwrap();
+ let base = format!("http://{}", listener.local_addr().unwrap());
+ let mut control = sample_conversation();
+ control["session_id"] = " sessão-界-opaque ".into();
+ let file = write_conversation(root.path(), control.clone());
+ let plan = plan_external_conversation(&file, "w", &base).unwrap();
+ assert_eq!(plan.external_session_id, " sessão-界-opaque ");
+ assert_eq!(
+ plan.events[0].body["external_session_id"],
+ control["session_id"]
+ );
+ assert_eq!(
+ plan.stable_session_id,
+ stable_external_session_id("w", "memory-lab", "chatgpt", " sessão-界-opaque ")
+ );
+ for (field, value) in [
+ ("project", "sk-1234567890abcdefghijklmnop"),
+ ("source", "sk-1234567890abcdefghijklmnop"),
+ ("session_id", "sk-1234567890abcdefghijklmnop"),
+ ("session_id", "\tnative"),
+ ("session_id", "native\n"),
+ ("session_id", "\rnative"),
+ ] {
+ let mut source = control.clone();
+ source[field] = value.into();
+ let file = write_conversation(root.path(), source);
+ assert!(
+ plan_external_conversation(&file, "w", &base).is_err(),
+ "sensitive native identity must be refused before planning"
+ );
+ let manifest = root.path().join(format!("{field}-manifest.json"));
+ let error = run_conversation(ConversationArgs {
+ file,
+ workspace: Some("w".into()),
+ server_url: base.clone(),
+ apply: true,
+ manifest_out: Some(manifest.clone()),
+ create_destination: false,
+ show_body: true,
+ })
+ .await;
+ assert!(error.is_err(), "sensitive source identity must be refused");
+ assert!(!error.unwrap_err().to_string().contains("abcdefghijklmnop"));
+ assert!(
+ !manifest.exists(),
+ "refused identity cannot create a manifest"
+ );
+ assert!(
+ listener.accept().is_err(),
+ "refused identity cannot reach HTTP"
+ );
+ }
+ let mut oversized = control.clone();
+ oversized["session_id"] = format!("native{}", " ".repeat(256)).into();
+ let oversized_file = write_conversation(root.path(), oversized);
+ assert!(
+ plan_external_conversation(&oversized_file, "w", &base).is_err(),
+ "native ID bounds apply before trimming"
+ );
+ let file = write_conversation(root.path(), control);
+ let manifest = root.path().join("control-manifest.json");
+ run_conversation(ConversationArgs {
+ file,
+ workspace: Some("w".into()),
+ server_url: base,
+ apply: false,
+ manifest_out: Some(manifest.clone()),
+ create_destination: false,
+ show_body: true,
+ })
+ .await
+ .unwrap();
+ assert!(manifest.exists());
+ assert!(listener.accept().is_err());
+ }
+
+ #[tokio::test]
+ async fn preview_and_manifest_use_scrubbed_bodies_and_digest() {
+ let root = tempdir().unwrap();
+ let mut source = sample_conversation();
+ source["messages"][1]["content"] =
+ "界 token=abcdefghijklmnop[REDACTED] literal [REDACTED PRIVATE KEY]".into();
+ let file = write_conversation(root.path(), source.clone());
+ let plan = plan_external_conversation(&file, "w", DEFAULT_SERVER_URL).unwrap();
+ let preview = serde_json::to_string(&plan.events).unwrap();
+ assert!(!preview.contains("abcdefghijklmnop"));
+ assert!(preview.contains("literal [REDACTED PRIVATE KEY]"));
+ source["messages"][1]["content"] =
+ sanitize_external_text(source["messages"][1]["content"].as_str().unwrap()).into();
+ let safe_file = root.path().join("safe-conversation.json");
+ fs::write(&safe_file, serde_json::to_vec(&source).unwrap()).unwrap();
+ let safe = plan_external_conversation(&safe_file, "w", DEFAULT_SERVER_URL).unwrap();
+ assert_eq!(plan.transcript_sha256, safe.transcript_sha256);
+ assert_eq!(plan.events[2].ingest_key, safe.events[2].ingest_key);
+ let manifest = root.path().join("manifest.json");
+ run_conversation(ConversationArgs {
+ file,
+ workspace: Some("w".into()),
+ server_url: DEFAULT_SERVER_URL.into(),
+ apply: false,
+ manifest_out: Some(manifest.clone()),
+ create_destination: false,
+ show_body: true,
+ })
+ .await
+ .unwrap();
+ let stored = fs::read_to_string(manifest).unwrap();
+ assert!(!stored.contains("abcdefghijklmnop"));
+ assert_eq!(
+ serde_json::from_str::<serde_json::Value>(&stored).unwrap()["transcript_sha256"],
+ plan.transcript_sha256
+ );
+ }
+
+ #[tokio::test]
+ async fn forged_body_authority_is_refused_before_transport() {
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ listener.set_nonblocking(true).unwrap();
+ let base = format!("http://{}", listener.local_addr().unwrap());
+ let clean = planned_hook_event(
+ &base,
+ "w",
+ "p",
+ "sessão-界",
+ 0,
+ "session-start",
+ None,
+ None,
+ serde_json::json!({"session_id":"sessão-界"}),
+ )
+ .unwrap();
+ let mut client = ImportClient::new(&base).unwrap();
+ client.client = reqwest::Client::builder()
+ .timeout(Duration::from_millis(100))
+ .build()
+ .unwrap();
+ for field in ["workspace", "project", "actor", "author_id", "headers"] {
+ let mut attack = clean.clone();
+ attack.body[field] = "forged".into();
+ assert!(client.post_hook_batch(&[attack]).await.is_err());
+ assert!(
+ listener.accept().is_err(),
+ "forged body authority must be refused before POST"
+ );
+ }
+ ai_memory_client::check_body(&clean.body).unwrap();
+ }
}
diff --git a/companions/ai-memory-relay/Cargo.lock b/companions/ai-memory-relay/Cargo.lock
index 13ea88d2..e1a852c5 100644
--- a/companions/ai-memory-relay/Cargo.lock
+++ b/companions/ai-memory-relay/Cargo.lock
@@ -14,10 +14,29 @@ dependencies = [
"zerocopy",
]
+[[package]]
+name = "aho-corasick"
+version = "1.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"
+dependencies = [
+ "memchr",
+]
+
+[[package]]
+name = "ai-memory-client"
+version = "0.1.0"
+dependencies = [
+ "anyhow",
+ "regex",
+ "serde_json",
+]
+
[[package]]
name = "ai-memory-relay"
version = "0.1.0"
dependencies = [
+ "ai-memory-client",
"anyhow",
"clap",
"fs2",
@@ -872,6 +891,35 @@ dependencies = [
"rand_core",
]
+[[package]]
+name = "regex"
+version = "1.13.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-automata",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-automata"
+version = "0.4.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"
+dependencies = [
+ "aho-corasick",
+ "memchr",
+ "regex-syntax",
+]
+
+[[package]]
+name = "regex-syntax"
+version = "0.8.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"
+
[[package]]
name = "reqwest"
version = "0.12.28"
diff --git a/companions/ai-memory-relay/Cargo.toml b/companions/ai-memory-relay/Cargo.toml
index 8ae374d1..8ceec15e 100644
--- a/companions/ai-memory-relay/Cargo.toml
+++ b/companions/ai-memory-relay/Cargo.toml
@@ -17,6 +17,7 @@ path = "src/main.rs"
unsafe_code = "forbid"
[dependencies]
+ai-memory-client = { path = "../ai-memory-client" }
anyhow = "1"
# Read the bearer token separately to keep it out of clap's Debug output.
clap = { version = "4", features = ["derive"] }
diff --git a/companions/ai-memory-relay/README.md b/companions/ai-memory-relay/README.md
index ec964f20..83a000f0 100644
--- a/companions/ai-memory-relay/README.md
+++ b/companions/ai-memory-relay/README.md
@@ -81,8 +81,10 @@ ai-memory-relay status --queue-dir "$HOME/.ai-memory-relay"
`enqueue` validates the whole array and commits it in one transaction. Each body
must be an object with explicit `session_id` and `cwd` strings. Use the harness's
-native identity and canonical hook event names. Other body fields keep their
-values, including an `_ai_memory_capture` block. The relay does not translate
+native identity and canonical hook event names. Sensitive identities are refused
+without renaming them. Shared private capture helpers scrub new body text and
+credential values before serialization, hashing and enqueueing. An
+`_ai_memory_capture` block remains untrusted provenance. The relay does not translate
provider transcripts or infer parent agents and workflows.
For authenticated servers, supply `AI_MEMORY_AUTH_TOKEN` through the flush
@@ -151,7 +153,13 @@ print event bodies. Receipts count acknowledgements, including policy drops.
## Local data and recovery
-The queue contains producer-supplied event bodies before server sanitization.
+New queue bodies receive local credential sanitation before persistence, then
+cross the normal server sanitizer on delivery. Legacy pending bytes, digest,
+native identity, ingest key and attempt state remain exact. An unsafe stored
+head is retained with a payload-free `privacy/validation` diagnostic; only its
+session is deferred while clean sessions continue. No attempt or receipt is
+recorded for the refused item. See the [shared privacy helpers](../ai-memory-client).
+
The producer must apply its capture exclusions before enqueueing. The relay
does not load the project's `[capture] ignore_paths` settings or inspect tool
payloads for sensitive paths. Server sanitization cannot protect a local queue
diff --git a/companions/ai-memory-relay/src/identity.rs b/companions/ai-memory-relay/src/identity.rs
index 672e989d..6363022e 100644
--- a/companions/ai-memory-relay/src/identity.rs
+++ b/companions/ai-memory-relay/src/identity.rs
@@ -1,8 +1,8 @@
//! Envelope validation and the stable retry identity.
//!
//! Validate input before enqueueing so malformed events cannot block a batch.
-//! Body values, including `_ai_memory_capture`, are preserved for the server.
-//! These checks do not sanitize locally queued content.
+//! New body values are sanitized before hashing and persistence. Native identities
+//! are validated first; stored retry bytes are never rewritten.
use serde::Deserialize;
use sha2::{Digest, Sha256};
@@ -40,7 +40,7 @@ pub const TERMINAL_EVENTS: &[&str] = &["session-end"];
///
/// `deny_unknown_fields` applies to the envelope only. `body` is an opaque
/// object: unknown keys inside it (a harness payload field, the
-/// `_ai_memory_capture` protocol block) are preserved untouched.
+/// `_ai_memory_capture` protocol block) remain untrusted and are sanitized.
#[derive(Debug, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct InputEvent {
@@ -50,7 +50,7 @@ pub struct InputEvent {
pub agent: String,
/// Canonical lifecycle event name, used for both `event` and `source_event`.
pub event: String,
- /// Harness payload, delivered verbatim.
+ /// Harness payload, sanitized at new-event ingress.
pub body: serde_json::Value,
}
@@ -62,8 +62,7 @@ pub struct ValidEvent {
pub event: String,
pub session_id: String,
pub cwd: String,
- /// Canonical JSON of the body exactly as supplied (sorted keys, no reformatting
- /// of values). The bytes delivered later are these bytes.
+ /// Canonical JSON after ingress sanitation. Retries use these exact bytes.
pub body_json: String,
pub body_sha256: String,
pub ingest_key: String,
@@ -110,6 +109,8 @@ pub fn check_actor(actor: &str) -> Result<(), String> {
/// Validate a workspace or project name.
pub fn check_scope(label: &str, value: &str) -> Result<(), String> {
+ ai_memory_client::reject_sensitive(&serde_json::json!(value))
+ .map_err(|_| format!("unsafe --{label} rejected"))?;
is_plain(value, MAX_SCOPE_LEN)
.then_some(())
.ok_or_else(|| format!("--{label} must be 1..={MAX_SCOPE_LEN} printable characters"))
@@ -152,11 +153,34 @@ pub fn ingest_key(
/// quote body content: an invalid payload must not become a log line.
pub fn validate(
index: usize,
- input: InputEvent,
+ mut input: InputEvent,
producer: &str,
actor: &str,
) -> Result<ValidEvent, String> {
let at = |detail: &str| format!("item {index}: {detail}");
+ ai_memory_client::reject_sensitive(&serde_json::json!({
+ "event_id": input.event_id, "agent": input.agent, "event": input.event,
+ "producer": producer, "actor": actor,
+ }))
+ .map_err(|_| at("sensitive envelope rejected"))?;
+ let identities: Vec<_> = [
+ "session_id",
+ "cwd",
+ "producer",
+ "actor",
+ "event_id",
+ "agent",
+ "event",
+ "ingest_key",
+ ]
+ .iter()
+ .filter_map(|key| input.body.get(key))
+ .collect();
+ ai_memory_client::reject_sensitive(&serde_json::json!(identities))
+ .map_err(|_| at("unsafe body identity rejected"))?;
+ ai_memory_client::sanitize_external_value(&mut input.body)
+ .map_err(|_| at("unsafe hook body rejected"))?;
+ ai_memory_client::check_body(&input.body).map_err(|_| at("unsafe hook body rejected"))?;
if !is_plain(&input.event_id, MAX_EVENT_ID_LEN) {
return Err(at(&format!(
"event_id must be 1..={MAX_EVENT_ID_LEN} printable characters"
diff --git a/companions/ai-memory-relay/src/lib.rs b/companions/ai-memory-relay/src/lib.rs
index 7a1a3a36..f8a0bcb7 100644
--- a/companions/ai-memory-relay/src/lib.rs
+++ b/companions/ai-memory-relay/src/lib.rs
@@ -5,7 +5,7 @@
//! Pending events remain on disk when delivery fails or their retry window
//! expires; reaching a capacity limit rejects new input.
//!
-//! Bodies retain their input values before server sanitization. Producers must
+//! New bodies are sanitized before enqueueing; stored retries stay exact. Producers must
//! apply capture exclusions before enqueueing and protect the queue directory.
pub mod ack;
diff --git a/companions/ai-memory-relay/src/relay.rs b/companions/ai-memory-relay/src/relay.rs
index 06b562f8..646e68fc 100644
--- a/companions/ai-memory-relay/src/relay.rs
+++ b/companions/ai-memory-relay/src/relay.rs
@@ -96,6 +96,9 @@ pub fn init(
identity::check_actor(actor).map_err(anyhow::Error::msg)?;
identity::check_scope("workspace", workspace).map_err(anyhow::Error::msg)?;
identity::check_scope("project", project).map_err(anyhow::Error::msg)?;
+ ai_memory_client::reject_sensitive(&serde_json::json!({
+ "producer": producer, "actor": actor, "server_url": server_url,
+ }))?;
let binding = Binding {
server_url: http::normalize_server_url(server_url)?,
producer: producer.to_owned(),
@@ -274,6 +277,8 @@ pub fn flush(dir: &Path, options: &FlushOptions) -> Result<Report> {
fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {
let mut queue = Queue::open(dir)?;
let binding = queue.binding()?;
+ identity::check_scope("workspace", &binding.workspace).map_err(anyhow::Error::msg)?;
+ identity::check_scope("project", &binding.project).map_err(anyhow::Error::msg)?;
queue.prune(crate::now_ms())?;
let token = Token::from_env()?;
let authenticated = token.is_some();
@@ -297,7 +302,7 @@ fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {
let now = crate::now_ms();
let base = binding.server_url.clone();
let for_cost = binding.clone();
- let batch: Batch =
+ let mut batch: Batch =
queue.select_batch(MAX_BATCH_ITEMS, MAX_BATCH_BYTES, now, &deferred, |item| {
item_cost(&base, &for_cost, item)
})?;
@@ -305,22 +310,42 @@ fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {
if batch.items.is_empty() {
break;
}
+ let mut items = Vec::new();
+ for item in &batch.items {
+ let wire = serde_json::from_str(&item.body_json).ok().and_then(|body| {
+ ai_memory_client::reject_sensitive(&serde_json::json!([
+ item.event_id,
+ item.agent,
+ item.event,
+ item.session_id,
+ ]))
+ .ok()?;
+ ai_memory_client::check_body(&body).ok()?;
+ let url = http::item_url(&binding.server_url, &binding, item);
+ ai_memory_client::reject_sensitive(&serde_json::json!(url)).ok()?;
+ Some(BatchItem { url, body })
+ });
+ if let Some(wire) = wire {
+ items.push(wire);
+ } else {
+ queue.record_failure(&item.ingest_key, "privacy/validation")?;
+ deferred.insert(item.session());
+ failure.get_or_insert_with(|| {
+ "privacy/validation: unsafe stored body retained; its session deferred".into()
+ });
+ }
+ }
+ batch
+ .items
+ .retain(|item| !deferred.contains(&item.session()));
+ if items.is_empty() {
+ continue;
+ }
batches += 1;
let keys: Vec<String> = batch.items.iter().map(|i| i.ingest_key.clone()).collect();
// Durable, and before the request: a lost ack must not restart the
- // 30-day clock on the next run.
+ // 30-day clock on the next run. Unsafe items were already excluded.
queue.stamp_attempt(&keys, now)?;
- let items: Vec<BatchItem> = batch
- .items
- .iter()
- .map(|item| {
- Ok(BatchItem {
- url: http::item_url(&binding.server_url, &binding, item),
- body: serde_json::from_str(&item.body_json)
- .context("stored body is no longer valid JSON")?,
- })
- })
- .collect::<Result<Vec<_>>>()?;
// Selection checked the retry window once; a retry inside this batch can
// still cross it. Carry the batch's tightest expiry into the sender so
// every attempt is re-checked against it.
@@ -473,7 +498,7 @@ fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {
}
if !deferred.is_empty() {
summary.push(format!(
- "{} session(s) deferred to a later flush (a failed or rate-limited head); \
+ "{} session(s) deferred to a later flush (an unsafe, failed or rate-limited head); \
their later events were never sent ahead of it",
deferred.len()
));
diff --git a/companions/ai-memory-relay/tests/relay.rs b/companions/ai-memory-relay/tests/relay.rs
index 05900db2..17461312 100644
--- a/companions/ai-memory-relay/tests/relay.rs
+++ b/companions/ai-memory-relay/tests/relay.rs
@@ -1745,3 +1745,503 @@ fn results_match_sparse_partial_failure_and_empty_acks() {
assert_eq!(ack::validate(len, &parsed).unwrap(), expected);
}
}
+
+#[test]
+fn sensitive_envelope_is_rejected_before_enqueue_with_clean_control() {
+ let root = fixture("sensitive-envelope");
+ let dir = bind(&root, "http://127.0.0.1:49374");
+ enqueue(
+ &dir,
+ &root,
+ "before.json",
+ serde_json::json!([event("opaque-界", "codex", "session-start", "sessão-界")]),
+ )
+ .unwrap();
+ assert_eq!(pending_count(&dir), 1);
+ for field in [
+ "session_id",
+ "cwd",
+ "producer",
+ "actor",
+ "event_id",
+ "agent",
+ "event",
+ "ingest_key",
+ ] {
+ let mut input = event("a", "codex", "session-start", "a");
+ input["body"][field] = "Bearer abcdefghijklmnop".into();
+ let error = enqueue(&dir, &root, "input.json", serde_json::json!([input])).unwrap_err();
+ assert!(!error.to_string().contains("abcdefghijklmnop"));
+ assert_eq!(pending_count(&dir), 1);
+ }
+ enqueue(
+ &dir,
+ &root,
+ "control.json",
+ serde_json::json!([event("a", "codex", "session-start", "a")]),
+ )
+ .unwrap();
+ assert_eq!(pending_count(&dir), 2);
+}
+
+#[test]
+fn new_native_bodies_are_sanitized_before_hash_enqueue_and_exact_retry() {
+ let root = fixture("ingress-sanitized-retry");
+ let server = stub(vec![Reply::Close, Reply::AcceptAll]);
+ let dir = bind(&root, &server.url());
+ let mut raw = event("native-event", "codex", "post-tool-use", "native-session");
+ for (name, value) in [
+ ("producer", "example.runtime"),
+ ("event_id", "native-event"),
+ ("agent", "codex"),
+ ("event", "post-tool-use"),
+ ("ingest_key", "native-ingest-key"),
+ ] {
+ raw["body"][name] = value.into();
+ }
+ raw["body"]["output"] =
+ "\u{1b}[32mresult\u{1b}[0m\u{202e} token=getToken() Bearer abcdefghijklmnop".into();
+ raw["body"]["tool_input"] = serde_json::json!({"nested": {"password": "process.env.X", "api-key": "tiny", "items": ["password=abcdefghi", "token=abcdefghijklmnop[REDACTED]"]}});
+ raw["body"]["key_cases"] = serde_json::json!([{
+ "accessToken":"fixture-value", "refreshToken":"fixture-value", "clientSecret":"fixture-value", "privateKey":"fixture-value", "x-api-key":"fixture-value", "OPENAI_API_KEY":"fixture-value", "credentials":"fixture-value", "cookie":"fixture-value"
+ }]);
+ raw["body"]["key_forms"] = serde_json::json!([{"auth":"fixture-value","jwt":"fixture-value","pwd":"fixture-value","bearer":"fixture-value","sessionKey":"fixture-value","signature":"fixture-value"}]);
+ raw["body"]["plural_forms"] = serde_json::json!([{
+ "api_keys":"fixture-value", "access_keys":"fixture-value", "private_keys":"fixture-value", "session_keys":"fixture-value",
+ "apiKeys":["fixture-value"], "accessKeys":["fixture-value"], "privateKeys":["fixture-value"], "sessionKeys":["fixture-value"]
+ }]);
+ raw["body"]["controls"] = serde_json::json!({"author":"fixture-value","authority":"fixture-value","key":"fixture-value","keys":["fixture-value"]});
+ raw["body"]["numeric_cases"] = serde_json::json!({"token":123456,"access_tokens":123456,"private_token_count":123456,"auth_token_usage":123456,"token_usage":{"input_tokens":12}});
+ let metrics = serde_json::json!({"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25,"output_tokens":9007199254740993_u64,"total_tokens":89,"totalTokens":89,"cache_read_input_tokens":2,"cache_creation_input_tokens":3,"prompt_tokens":5,"completion_tokens":7,"reasoning_tokens":11});
+ raw["body"]["metrics"] = metrics.clone();
+ enqueue(&dir, &root, "native.json", serde_json::json!([raw.clone()])).unwrap();
+ enqueue(&dir, &root, "repeat.json", serde_json::json!([raw])).unwrap();
+ assert_eq!(pending_count(&dir), 1);
+ let conn = rusqlite::Connection::open(dir.join(ai_memory_relay::fsguard::DB_FILE)).unwrap();
+ let (body, digest, key, id, agent, event, session): (
+ String,
+ String,
+ String,
+ String,
+ String,
+ String,
+ String,
+ ) = conn
+ .query_row(
+ "SELECT body_json,body_sha256,ingest_key,event_id,agent,event,session_id FROM pending",
+ [],
+ |row| {
+ Ok((
+ row.get(0)?,
+ row.get(1)?,
+ row.get(2)?,
+ row.get(3)?,
+ row.get(4)?,
+ row.get(5)?,
+ row.get(6)?,
+ ))
+ },
+ )
+ .unwrap();
+ let safe: serde_json::Value = serde_json::from_str(&body).unwrap();
+ assert_eq!(
+ (&id[..], &agent[..], &event[..], &session[..]),
+ ("native-event", "codex", "post-tool-use", "native-session")
+ );
+ assert_eq!(
+ key,
+ identity::ingest_key(
+ "example.runtime",
+ "operator-a",
+ &agent,
+ &session,
+ &event,
+ &id
+ )
+ );
+ for (name, value) in [
+ ("producer", "example.runtime"),
+ ("event_id", "native-event"),
+ ("agent", "codex"),
+ ("event", "post-tool-use"),
+ ("ingest_key", "native-ingest-key"),
+ ] {
+ assert_eq!(safe[name], value);
+ }
+ assert_eq!(safe["cwd"], "/work/app");
+ assert!(
+ !safe["output"]
+ .as_str()
+ .unwrap()
+ .contains(['\u{1b}', '\u{202e}'])
+ );
+ assert!(!body.contains("abcdefghijklmnop"));
+ assert!(!body.contains("process.env.X"));
+ assert!(!body.contains("tiny"));
+ assert_eq!(safe["tool_input"]["nested"]["api-key"], "[REDACTED]");
+ assert!(
+ safe["key_forms"][0]
+ .as_object()
+ .unwrap()
+ .values()
+ .all(|value| value == "[REDACTED]")
+ );
+ assert!(
+ safe["plural_forms"][0]
+ .as_object()
+ .unwrap()
+ .values()
+ .all(|value| value == "[REDACTED]")
+ );
+ assert!(
+ safe["numeric_cases"]
+ .as_object()
+ .unwrap()
+ .values()
+ .all(|value| value == "[REDACTED]")
+ );
+ assert!(
+ safe["controls"]
+ == serde_json::json!({"author":"fixture-value","authority":"fixture-value","key":"fixture-value","keys":["fixture-value"]})
+ );
+ assert!(
+ safe["key_cases"][0]
+ .as_object()
+ .unwrap()
+ .values()
+ .all(|value| value == "[REDACTED]")
+ );
+ assert!(safe["metrics"] == metrics);
+ assert!(serde_json::to_vec(&safe["metrics"]).unwrap() == serde_json::to_vec(&metrics).unwrap());
+ assert_eq!(
+ safe["output"],
+ ai_memory_client::sanitize_external_text(
+ "\u{1b}[32mresult\u{1b}[0m\u{202e} token=getToken() Bearer abcdefghijklmnop"
+ )
+ );
+
+ ai_memory_client::check_body(&safe).unwrap();
+ let validated = identity::validate(
+ 0,
+ InputEvent {
+ event_id: id,
+ agent,
+ event,
+ body: safe.clone(),
+ },
+ "example.runtime",
+ "operator-a",
+ )
+ .unwrap();
+ assert_eq!(validated.body_json, body);
+ assert_eq!(validated.body_sha256, digest);
+ assert!(flush(&dir).failure.is_none());
+ assert_eq!(pending_count(&dir), 0);
+ let sent = server.bodies();
+ assert_eq!(sent.len(), 2);
+ assert_eq!(sent[0], sent[1]);
+ assert_eq!(sent[0][0]["body"], safe);
+ assert_eq!(serde_json::to_string(&sent[0][0]["body"]).unwrap(), body);
+ assert_eq!(
+ Queue::open(&dir)
+ .unwrap()
+ .stats(ai_memory_relay::now_ms())
+ .unwrap()
+ .receipts,
+ 1
+ );
+}
+
+#[test]
+fn unsafe_legacy_pending_is_retained_exactly_and_never_sent() {
+ let root = fixture("unsafe-legacy-pending");
+ let server = stub(vec![Reply::AcceptAll, Reply::AcceptAll]);
+ let dir = bind(&root, &server.url());
+ let mut legacy = valid("native-event", "codex", "session-start", "native-session");
+ legacy.body_json = "{ \"prompt\": \"Bearer abcdefghijklmnop\", \"cwd\": \"/native/path\", \"session_id\": \"native-session\" }".into();
+ let mut queue = Queue::open(&dir).unwrap();
+ queue
+ .enqueue(&[legacy.clone()], ai_memory_relay::now_ms())
+ .unwrap();
+ let first_attempt = ai_memory_relay::now_ms() - 1000;
+ queue
+ .stamp_attempt(&[legacy.ingest_key.clone()], first_attempt)
+ .unwrap();
+ queue
+ .stamp_attempt(&[legacy.ingest_key.clone()], first_attempt)
+ .unwrap();
+ let clean = valid("clean-event", "codex", "session-start", "clean-session");
+ queue
+ .enqueue(
+ &[
+ valid("later-dirty", "codex", "session-end", "native-session"),
+ clean.clone(),
+ ],
+ ai_memory_relay::now_ms(),
+ )
+ .unwrap();
+ drop(queue);
+ for _ in 0..2 {
+ let report = relay::flush(&dir, &FlushOptions::default()).unwrap();
+ assert!(
+ report
+ .failure
+ .as_ref()
+ .is_some_and(|failure| failure.contains("privacy/validation"))
+ );
+ assert!(!format!("{report:?}").contains("abcdefghijklmnop"));
+ assert!(report.pending);
+ assert_eq!(pending_count(&dir), 2);
+ }
+ let sent = server.bodies();
+ assert_eq!(sent.len(), 1);
+ assert_eq!(
+ sent[0][0]["body"],
+ serde_json::from_str::<serde_json::Value>(&clean.body_json).unwrap()
+ );
+ assert_eq!(
+ Queue::open(&dir)
+ .unwrap()
+ .stats(ai_memory_relay::now_ms())
+ .unwrap()
+ .receipts,
+ 1
+ );
+ let conn = rusqlite::Connection::open(dir.join(ai_memory_relay::fsguard::DB_FILE)).unwrap();
+ let preserved: (String, String, String, String, Option<i64>, i64) = conn
+ .query_row(
+ "SELECT ingest_key,session_id,body_json,body_sha256,first_attempt_ms,attempts FROM pending WHERE event_id=?1",
+ [&legacy.event_id],
+ |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?, row.get(4)?, row.get(5)?)),
+ )
+ .unwrap();
+ assert!(
+ preserved
+ == (
+ legacy.ingest_key,
+ legacy.session_id,
+ legacy.body_json,
+ legacy.body_sha256,
+ Some(first_attempt),
+ 2
+ ),
+ "unsafe legacy identity, bytes, digest and attempt state must remain unchanged"
+ );
+ let errors: Vec<(String, Option<String>, Option<i64>)> = conn
+ .prepare("SELECT event_id,last_error,first_attempt_ms FROM pending ORDER BY seq")
+ .unwrap()
+ .query_map([], |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)))
+ .unwrap()
+ .map(Result::unwrap)
+ .collect();
+ assert_eq!(
+ errors,
+ [
+ (
+ "native-event".into(),
+ Some("privacy/validation".into()),
+ Some(first_attempt)
+ ),
+ ("later-dirty".into(), None, None)
+ ]
+ );
+}
+
+#[test]
+fn legacy_binding_scope_spaces_are_preserved_on_delivery() {
+ let root = fixture("legacy-scope-spaces");
+ let server = stub(vec![Reply::AcceptAll]);
+ let dir = queue_dir(&root);
+ relay::init(
+ &dir,
+ &server.url(),
+ "example.runtime",
+ "operator-a",
+ " team ",
+ " app ",
+ )
+ .unwrap();
+ enqueue(
+ &dir,
+ &root,
+ "native.json",
+ serde_json::json!([event("e", "codex", "session-start", "s")]),
+ )
+ .unwrap();
+ assert!(flush(&dir).failure.is_none());
+ let bodies = server.bodies();
+ let url = url::Url::parse(bodies[0][0]["url"].as_str().unwrap()).unwrap();
+ let pairs: HashMap<_, _> = url.query_pairs().into_owned().collect();
+ assert_eq!(pairs["workspace"], " team ");
+ assert_eq!(pairs["project"], " app ");
+}
+
+#[test]
+fn opaque_unicode_native_identity_ships_unchanged() {
+ let root = fixture("privacy-unicode-control");
+ let server = stub(vec![Reply::AcceptAll]);
+ let dir = bind(&root, &server.url());
+ let mut input = event("event-界", "codex", "session-start", " sessão-界/opaque ");
+ input["body"]["cwd"] = "/work/界".into();
+ input["body"]["_ai_memory_capture"] =
+ serde_json::json!({"actor":"untrusted-label", "authority":"untrusted provenance"});
+ let original = input["body"].clone();
+ enqueue(&dir, &root, "input.json", serde_json::json!([input])).unwrap();
+ assert!(flush(&dir).failure.is_none());
+ assert_eq!(server.bodies()[0][0]["body"], original);
+ let bodies = server.bodies();
+ let url = url::Url::parse(bodies[0][0]["url"].as_str().unwrap()).unwrap();
+ assert_eq!(bodies[0][0]["body"]["session_id"], " sessão-界/opaque ");
+ assert_eq!(
+ url.query_pairs()
+ .find(|(key, _)| key == "ingest_key")
+ .unwrap()
+ .1,
+ identity::ingest_key(
+ "example.runtime",
+ "operator-a",
+ "codex",
+ " sessão-界/opaque ",
+ "session-start",
+ "event-界"
+ )
+ );
+ assert!(
+ server
+ .heads
+ .lock()
+ .unwrap()
+ .iter()
+ .all(|head| head.starts_with("POST /hook/batch "))
+ );
+}
+
+#[test]
+fn sensitive_outer_identity_and_body_authority_are_refused_before_enqueue() {
+ let root = fixture("privacy-outer-refusal");
+ let dir = bind(&root, "http://127.0.0.1:49374");
+ enqueue(
+ &dir,
+ &root,
+ "before.json",
+ serde_json::json!([event("before", "codex", "session-start", "control")]),
+ )
+ .unwrap();
+ for field in ["event_id", "agent", "event"] {
+ let mut attack = event("native", "codex", "session-start", "session");
+ attack[field] = "sk-1234567890abcdefghijklmnop".into();
+ let result = enqueue(&dir, &root, "attack.json", serde_json::json!([attack]));
+ assert!(result.is_err(), "sensitive outer identity must be refused");
+ assert!(!result.unwrap_err().to_string().contains("abcdefghijklmnop"));
+ assert_eq!(pending_count(&dir), 1);
+ }
+ for field in ["workspace", "project", "actor", "author_id", "headers"] {
+ let mut attack = event("native", "codex", "session-start", "session");
+ attack["body"][field] = "forged".into();
+ assert!(enqueue(&dir, &root, "attack.json", serde_json::json!([attack])).is_err());
+ assert_eq!(pending_count(&dir), 1);
+ }
+ enqueue(
+ &dir,
+ &root,
+ "after.json",
+ serde_json::json!([event("after", "codex", "session-end", "control")]),
+ )
+ .unwrap();
+ assert_eq!(pending_count(&dir), 2);
+}
+
+#[test]
+fn sensitive_binding_is_refused_before_queue_creation() {
+ let root = fixture("privacy-binding-refusal");
+ let safe = ["example.runtime", "operator-a", "team", "app"];
+ let before = root.join("control-before");
+ relay::init(
+ &before,
+ "http://127.0.0.1:49374",
+ safe[0],
+ safe[1],
+ safe[2],
+ safe[3],
+ )
+ .unwrap();
+ for field in 0..4 {
+ let mut attack = safe;
+ attack[field] = "sk-1234567890abcdefghijklmnop";
+ let dir = root.join(format!("attack-{field}"));
+ let result = relay::init(
+ &dir,
+ "http://127.0.0.1:49374",
+ attack[0],
+ attack[1],
+ attack[2],
+ attack[3],
+ );
+ assert!(result.is_err(), "sensitive binding must be refused");
+ assert!(!result.unwrap_err().to_string().contains("abcdefghijklmnop"));
+ assert!(!dir.exists(), "sensitive binding must not create the queue");
+ }
+ relay::init(
+ &root.join("control-after"),
+ "http://127.0.0.1:49374",
+ safe[0],
+ safe[1],
+ safe[2],
+ safe[3],
+ )
+ .unwrap();
+}
+
+#[test]
+fn unsafe_legacy_native_identity_defers_only_its_session() {
+ let root = fixture("privacy-legacy-native-id");
+ let server = stub(vec![Reply::AcceptAll, Reply::AcceptAll]);
+ let dir = bind(&root, &server.url());
+ let mut legacy = valid("old-event", "codex", "session-start", "old-session");
+ legacy.event_id = "sk-1234567890abcdefghijklmnop".into();
+ let mut queue = Queue::open(&dir).unwrap();
+ let before = valid("before", "codex", "session-start", "clean-before");
+ queue
+ .enqueue(&[legacy.clone(), before.clone()], ai_memory_relay::now_ms())
+ .unwrap();
+ drop(queue);
+ let report = flush(&dir);
+ assert!(
+ report
+ .failure
+ .is_some_and(|failure| failure.contains("privacy/validation")),
+ "unsafe legacy native identity must be refused"
+ );
+ assert_eq!(pending_count(&dir), 1);
+ enqueue(
+ &dir,
+ &root,
+ "after.json",
+ serde_json::json!([event("after", "codex", "session-start", "clean-after")]),
+ )
+ .unwrap();
+ let report = flush(&dir);
+ assert!(report.failure.is_some());
+ assert_eq!(pending_count(&dir), 1);
+ let sent = server.bodies();
+ assert_eq!(sent.len(), 2);
+ assert_eq!(
+ sent[0][0]["body"],
+ serde_json::from_str::<serde_json::Value>(&before.body_json).unwrap()
+ );
+ assert_eq!(sent[1][0]["body"]["session_id"], "clean-after");
+ let conn = rusqlite::Connection::open(dir.join(ai_memory_relay::fsguard::DB_FILE)).unwrap();
+ let stored: (String,String,String,String,String,Option<i64>,i64) = conn.query_row("SELECT event_id,session_id,ingest_key,body_json,body_sha256,first_attempt_ms,attempts FROM pending", [], |row| Ok((row.get(0)?,row.get(1)?,row.get(2)?,row.get(3)?,row.get(4)?,row.get(5)?,row.get(6)?))).unwrap();
+ assert_eq!(
+ stored,
+ (
+ legacy.event_id,
+ legacy.session_id,
+ legacy.ingest_key,
+ legacy.body_json,
+ legacy.body_sha256,
+ None,
+ 0
+ )
+ );
+}
diff --git a/docs/companion-crates.md b/docs/companion-crates.md
index 1ffd1dcc..1c2ca4d0 100644
--- a/docs/companion-crates.md
+++ b/docs/companion-crates.md
@@ -79,6 +79,17 @@ the public MCP write/delete tools. It must never open the wiki directory or
SQLite directly. The core read seam is `/api/v1` in API-only mode; the supported
MCP page arguments are documented in [programmatic memory](programmatic-memory.md).
+## `ai-memory-client`: shared private capture privacy
+
+[`ai-memory-client`](../companions/ai-memory-client) supplies four privacy
+functions consumed by both the relay and external-conversation importer. New
+bodies are scrubbed before serialization, hashes and local persistence; native
+identities containing credentials are refused without being renamed. Unsafe
+legacy relay heads retain their exact bytes and retry metadata and defer only
+their own session. The package has its own workspace and focused tests, with no
+server dependency. Server authorization and capture exclusions remain the
+authoritative boundaries.
+
## `ai-memory-relay`: external lifecycle delivery
[`ai-memory-relay`](../companions/ai-memory-relay) delivers events collected by
diff --git a/docs/security-boundaries.md b/docs/security-boundaries.md
index 53774ace..e2a8511b 100644
--- a/docs/security-boundaries.md
+++ b/docs/security-boundaries.md
@@ -64,6 +64,7 @@ boundary not yet built.
| 13a | Managed Codex stale-daemon run recovery (#987; concurrent launches, project/worktree isolation, operator isolation) | `managed_runs.owner_user` (V71) uses the topology-aware `IdentityKey` owner stamp; `router.rs` resolves and authorizes the SessionStart repository before handling the untrusted run id; store `link_or_adopt_native_session` selects and links in one writer transaction and admits only the sole live, undelivered, never-linked Codex run in the exact workspace/project/checkout-cwd/owner bucket. An active supplied id must itself match every boundary; another harness, an ambiguity, or a second native session is refused | `multi_session.rs` `stale_codex_run_recovery_cannot_cross_project_owner_or_session_boundaries` (foreign owner, foreign checkout, and cross-project candidates refused; project-local controls and racing rebind), `stale_codex_run_recovery_fails_closed_on_ambiguity_and_active_mismatch` (two candidates and active foreign id refused, exact control); `workstream.rs` `prepare_stamps_the_operator_bucket_used_by_session_start_recovery` (HTTP actor stamp with foreign and legitimate controls); `router.rs` `managed_session_start_delivers_ledger_and_pending_handoff` supplies a finished stale id and proves the new run is linked and delivered | STRONG for SessionStart recovery. No candidate is selected from a run id's stale scope, another checkout, or another operator bucket; ambiguity deliberately falls back to no managed packet |
| 13b | Managed-run lease exclusivity and explicit recovery (one active run per workstream, invariant #16) | `ai-memory-store/src/workstream.rs` `prepare_run` expires lapsed leases and otherwise refuses an `active` run inside the writer transaction (`StoreError::WorkstreamBusy`). The explicit `force_unlock` path may expire and replace that row in the same transaction only when its durable `owner_user` equals the incoming actor's attribution; different operators remain isolated. `heartbeat` renews only `active` rows. `ai-memory-cli/src/commands/run.rs` normally uses `wait_out_held_lease`; `--force-unlock` instead asks once and fails immediately when refused. Neither path treats the informational `lease_owner` label as authority | `multi_session.rs` `force_unlock_replaces_only_the_same_operators_active_run` (foreign operator refused with live control, same operator succeeds, old run loses heartbeat, unattributed single-user control); hooks `workstream.rs` `force_unlock_cannot_evict_another_operator` exercises the HTTP actor boundary; CLI `run.rs` `force_unlock_asks_once_instead_of_waiting_out_the_lease`; existing controls `managed_workstream_batches_are_idempotent_and_release_the_lease`, `a_renewed_lease_is_reported_as_a_live_owner_not_taken_over`, `interactive_launch_waits_out_a_lapsing_lease_then_proceeds`, and `ctrl_c_aborts_the_held_lease_wait_immediately` | STRONG for exclusivity. The `lease_owner` label (`host:pid`) is informational only and not unique inside ai-jail (every jailed launcher reports `ai-sandbox:<ns-pid>`), so it must never become an ownership key. A shared unattributed/root context is intentionally one operator bucket; deployments needing person-level refusal must authenticate DB users |
| 14 | Per-project authorization (#708) | `ai-memory-store/src/project_authz.rs` `authorize_project` / `ProjectAuthz::authorize` choke point (V68 `project_grants` + `projects.access_mode`, default `open`; V69 `projects.created_by` feeds `is_creator`); `scope.rs` `ScopeResolver::with_project_authz` (reader pool for reads, writer actor for writes) and its free forms `authorize_scope_for` / `*_guarded`, attached for every DB user by `ai-memory-mcp` `scope_resolver_as`, `ai-memory-web` `authorize_read` / `lookup_project`, and `ai-memory-hooks` (`grants.rs` `authorize_resolved` for run/workstream ids, the capture check in `router.rs`); read-shaped mutations resolve at `ProjectAccess::Write` (`resolve_existing_args(.., need)`); unscoped reads filtered before `LIMIT` by `reader.rs` `readable_repository_sql`; `WriterHandle::authorize_project` as defense in depth. Page ids are never taken from a caller (only derived from already-authorized hits), so there is no page-id entry point to guard; incremental `recent` resolves and authorizes the requested project before decoding its scope-bound cursor; batch outcomes do not bypass the capture grant or session owner check | `tests/suite/project_authz.rs` (decision matrix, ship-inert, resolver gate); `tests/suite/access_mode.rs` `every_caller_against_both_modes`, `a_restricted_project_admits_the_team_and_refuses_the_outsider`, `new_projects_follow_the_server_default_and_admit_their_creator`; `scope.rs` `the_argument_shape_does_not_decide_the_level`, `a_user_reaches_only_what_they_were_granted`, `creating_authorizes_against_a_project_that_already_exists`, `a_refused_scope_fails_the_search_instead_of_shortening_it`; `grants.rs` `search_finds_only_what_the_viewer_may_read`, `the_limit_counts_only_what_the_viewer_may_see`, `the_workspace_handoff_comes_only_from_readable_repositories`; `ai-memory-mcp` `server.rs` `a_reader_may_read_everything_and_change_nothing`, `bob_cannot_read_alices_page_in_a_restricted_project`, `bob_cannot_find_alices_page_by_searching_in_a_restricted_project`, `bob_cannot_consolidate_a_session_in_alices_repository`, `a_restricted_projects_queues_need_write`; `ai-memory-hooks` `a_capture_needs_writer_on_the_repository_it_lands_in`, `session_start_delivers_nothing_from_a_repository_the_viewer_cannot_read`, `run_and_workstream_ids_only_answer_someone_who_may_reach_the_repository`; `ai-memory-web` `web_reads_honour_grants_in_a_restricted_project`, `metadata_shows_only_what_the_viewer_may_read` — each with a granted or open-project control, and proven to fail with the choke point (18 tests) or the SQL filter (9 tests) neutralized; `routes.rs` `api_recent_incremental_rechecks_restricted_scope_on_cursor` (a valid cursor cannot bypass a revoked grant; granted control passes), `api_recent_incremental_paginates_over_120_pages_and_binds_cursor`; `mcp_stateless_http.rs` `generic_machine_client_writes_queries_and_claims_a_handoff` (real machine key, foreign scope and partial scope refused); `router.rs` `an_unauthorized_batch_item_is_consumed_rather_than_retried`, `batch_acknowledges_foreign_collision_then_commits_next_item` (refused foreign item followed by a legitimate stored item) | STRONG — slice 3 closed both bypass classes (unscoped reads, raw-id entry points) and added the root-only management surface. Out of scope by design: per-project administrators (granting/restricting is root-only), and access modes, creators and grants live only in SQLite, so `reindex` resets them |
+| 14aa | Companion capture privacy before persistence and delivery | `companions/ai-memory-client/src/lib.rs` `reject_sensitive`, `sanitize_external_value`, `check_body`; `src/scrub.rs` `sanitize_external_text`; relay `identity::validate`, `check_scope`, `relay::init` and `flush_locked`; importer `plan_external_conversation`, `planned_hook_event`, `truncate_utf8`, `ImportClient::post_hook_batch` | SDK secret-position, nested key, 11-name numeric metric, bounded schema and literal-marker tests; relay `sensitive_envelope_is_rejected_before_enqueue_with_clean_control`, `new_native_bodies_are_sanitized_before_hash_enqueue_and_exact_retry`, `unsafe_legacy_pending_is_retained_exactly_and_never_sent`, `legacy_binding_scope_spaces_are_preserved_on_delivery`; importer `sensitive_native_identity_is_refused_before_manifest_and_http_with_unicode_control`, `shared_key_sanitation_precedes_plan_manifest_transport_and_exact_retry`, `preview_and_manifest_use_scrubbed_bodies_and_digest`, `forged_body_authority_is_refused_before_transport`, `sanitized_redaction_markers_survive_utf8_byte_caps_in_bodies_and_titles`. Controls preserve opaque native Unicode IDs, metric representation and untrusted provenance. Unsafe legacy bodies keep exact bytes/digest/identity/key and existing attempt state; only their session is deferred while clean sessions ship. Guard-removal runtime probes target the final formatted sources and restore identical hashes. | Client-side privacy only; server scope, auth, admission and capture-policy guards remain authoritative |
| 14b | Repository identity routing (#708) | `ai-memory-store/src/ops.rs` `resolve_project_by_identity` — one transaction; an identity already on a project is never overwritten; an unclaimed project is claimed only when the capturing user may write it (the choke point's `resolve_project_authz` on the same transaction), otherwise it is returned unclaimed; a different identity under the same name splits into a new project with no shared `repo_path`; `ai-memory-hooks/src/router.rs` `cache_key_for` keys the path cache by identity too; `ai-memory-core/src/repository_identity.rs` strips credentials from remote URLs client-side, and the server accepts only the routing rungs (`explicit`, `git_remote`) from the wire | `tests/suite/identity_resolution.rs` `an_outsider_cannot_take_an_unclaimed_projects_identity` (control: `an_existing_project_is_claimed_in_place`), `two_unrelated_repositories_with_one_folder_name_stay_apart`, `a_created_or_split_project_admits_its_creator`; `router.rs` `one_path_with_two_remotes_is_two_projects`, `two_api_checkouts_with_different_remotes_get_two_projects` (a manifest rung on the wire is ignored); `repository_identity.rs` credential and normalisation tables — the claim guard and the cache key each proven to fail their test when removed | STRONG — first claimant wins: in a restricted workspace a user who creates a project under a remote identity first holds it until root grants others |
| 14c | Client-supplied event time is bounded and self-scoped (#919) | `ai-memory-hooks/src/payload.rs` `HookEnvelope::occurred_at_micros` — a `/hook` caller's optional `occurred_at` must be `> 0` and `<= now + 5min`; it only sets the caller's *own* admitted session's `started_at`/`ended_at`/`created_at`, never another session/project/user, and the ingest dedup `seen_at`/TTL stays on `now` | `payload.rs` `occurred_at_micros_rejects_a_far_future_timestamp`, `…_rejects_non_positive_values`, `…_rejects_garbage_strings` | MEDIUM — self-scoped numeric bound; no cross-tenant surface (a client already controls its own content) |
| 14d | `include_related` shows and walks only readable projects (#708) | `ai-memory-store/src/reader.rs` `related_walk(.., viewer)` — both the outgoing and incoming hop queries carry `readable_repository_predicate("pg.project_id", viewer)`, the same far-end filter `page_links` applies, so a page in a project the viewer cannot read is neither returned nor expanded; `ai-memory-mcp/src/server.rs` `memory_read_page` passes `viewer_from_parts`, and the access bump only touches the filtered nodes | `related_walk.rs` `the_walk_hides_and_does_not_cross_projects_the_viewer_cannot_read` (an outgoing and an incoming neighbour in an ungranted restricted project are absent, and so is a granted page reachable only through one; control: a granted restricted project is shown, and no viewer walks the whole graph) — fails with the predicate removed; `ai-memory-mcp` `server.rs` `bob_does_not_see_restricted_pages_through_the_related_walk` (an ungranted user's `memory_read_page{include_related}` omits the restricted page; control: no viewer still reaches it) — fails with `memory_read_page` passing `None` instead of `viewer_from_parts` | STRONG |