Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat: share bounded companion capture privacy validation
Éverton Toffanettod7ba81976c35
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex a253c87b..f3b0a53c 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -156,7 +156,7 @@ jobs:strategy:fail-fast: falsematrix:- companion: [ai-memory-importer, ai-memory-relay]+ companion: [ai-memory-client, ai-memory-importer, ai-memory-relay]runs-on: ubuntu-lateststeps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1@@ -355,6 +355,7 @@ jobs:--ignore RUSTSEC-2024-0320- run: cargo audit --file companions/ai-memory-relay/Cargo.lock- run: cargo audit --file companions/ai-memory-importer/Cargo.lock+ - run: cargo audit --file companions/ai-memory-client/Cargo.lockgitleaks:name: gitleaks (secret scan)diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 762d7fbb..6670edd8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0## [Unreleased]### Added+- Added shared private capture privacy helpers used by the relay and conversation+ importer before body hashes, local persistence, previews and delivery. Sensitive+ native identities were refused without renaming; nested credentials were scrubbed+ with bounded body validation and an exact numeric token-metric allowlist. Unsafe+ legacy relay heads retained their bytes and retry state while deferring only their+ session, and importer byte caps preserved complete redaction markers. (#823)- Added `install-hooks --agent claude-code --scope project`, which writes thehook configuration to the checkout's gitignored `.claude/settings.local.json`(where Claude Code reads it: the git root, or the launch directory ondiff --git a/companions/ai-memory-client/Cargo.lock b/companions/ai-memory-client/Cargo.locknew file mode 100644index 00000000..61644056--- /dev/null+++ b/companions/ai-memory-client/Cargo.lock@@ -0,0 +1,151 @@+# This file is automatically @generated by Cargo.+# It is not intended for manual editing.+version = 4++[[package]]+name = "aho-corasick"+version = "1.1.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"+dependencies = [+ "memchr",+]++[[package]]+name = "ai-memory-client"+version = "0.1.0"+dependencies = [+ "anyhow",+ "regex",+ "serde_json",+]++[[package]]+name = "anyhow"+version = "1.0.104"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"++[[package]]+name = "itoa"+version = "1.0.18"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"++[[package]]+name = "memchr"+version = "2.8.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"++[[package]]+name = "proc-macro2"+version = "1.0.107"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"+dependencies = [+ "unicode-ident",+]++[[package]]+name = "quote"+version = "1.0.47"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"+dependencies = [+ "proc-macro2",+]++[[package]]+name = "regex"+version = "1.13.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"+dependencies = [+ "aho-corasick",+ "memchr",+ "regex-automata",+ "regex-syntax",+]++[[package]]+name = "regex-automata"+version = "0.4.18"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"+dependencies = [+ "aho-corasick",+ "memchr",+ "regex-syntax",+]++[[package]]+name = "regex-syntax"+version = "0.8.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"++[[package]]+name = "serde"+version = "1.0.229"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"+dependencies = [+ "serde_core",+]++[[package]]+name = "serde_core"+version = "1.0.229"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"+dependencies = [+ "serde_derive",+]++[[package]]+name = "serde_derive"+version = "1.0.229"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"+dependencies = [+ "proc-macro2",+ "quote",+ "syn",+]++[[package]]+name = "serde_json"+version = "1.0.151"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"+dependencies = [+ "itoa",+ "memchr",+ "serde",+ "serde_core",+ "zmij",+]++[[package]]+name = "syn"+version = "3.0.6"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"+dependencies = [+ "proc-macro2",+ "quote",+ "unicode-ident",+]++[[package]]+name = "unicode-ident"+version = "1.0.26"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"++[[package]]+name = "zmij"+version = "1.0.23"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"diff --git a/companions/ai-memory-client/Cargo.toml b/companions/ai-memory-client/Cargo.tomlnew file mode 100644index 00000000..695f0615--- /dev/null+++ b/companions/ai-memory-client/Cargo.toml@@ -0,0 +1,17 @@+[workspace]++[package]+name = "ai-memory-client"+version = "0.1.0"+edition = "2024"+publish = false+license = "MIT"+description = "Shared ingress privacy contract for ai-memory companions"++[lints.rust]+unsafe_code = "forbid"++[dependencies]+anyhow = "1"+regex = "1"+serde_json = "1"diff --git a/companions/ai-memory-client/README.md b/companions/ai-memory-client/README.mdnew file mode 100644index 00000000..a2680ed5--- /dev/null+++ b/companions/ai-memory-client/README.md@@ -0,0 +1,35 @@+# ai-memory-client++Private capture privacy helpers shared by the standalone relay and conversation+importer. This package has its own Cargo workspace and is not published to crates.io.++The four public functions have shipped callers:++| Function | Caller and purpose |+| --- | --- |+| `reject_sensitive` | Relay binding and native event identities; importer source/native identities and delivery URLs. Rejects credentials and unsafe controls without renaming identities. |+| `sanitize_external_text` | Importer conversation text before transcript hashes, event bodies, previews and manifests. |+| `sanitize_external_value` | Relay ingress and importer event planning before body serialization, hashing or persistence. Recursively scrubs strings and opaque credential values. |+| `check_body` | Both companions before delivery; relay before enqueue and importer before event hashing. Refuses unsafe content and top-level destination or actor authority. |++Credential keys accept camelCase, acronym and separated spellings, including+plural API, access, private and session key pairs. Only numeric values under+these 11 normalized names retain their representation: `max_tokens`,+`input_tokens`, `token_count`, `token_usage`, `output_tokens`, `total_tokens`,+`cache_read_input_tokens`, `cache_creation_input_tokens`, `prompt_tokens`,+`completion_tokens`, and `reasoning_tokens`. Strings or objects under these+names are redacted. Other credential-like numeric fields are redacted too.++Values are bounded to 64 nesting levels and individual strings to 256 KiB.+Checked bodies must be objects and their serialized size cannot exceed 256 KiB.+Complete redaction markers survive repeated scrubbing. Importer truncation+preserves UTF-8 boundaries and drops a marker that would be cut in half.++Native identity validation precedes sanitation. Already-pending relay bytes,+digest, retry key and attempt state are never rewritten. An unsafe pending head+defers only its session; clean sessions still advance. Body provenance remains+untrusted data, and server scope, authorization and capture policy remain+authoritative. Producers must apply capture exclusions before using the relay.++Run this package's format, clippy and test checks separately, as with the other+companions. Root workspace checks do not cover its tests.diff --git a/companions/ai-memory-client/src/lib.rs b/companions/ai-memory-client/src/lib.rsnew file mode 100644index 00000000..ff8ed040--- /dev/null+++ b/companions/ai-memory-client/src/lib.rs@@ -0,0 +1,122 @@+//! Privacy validation and ingress sanitation shared by both capture companions.++use anyhow::{Result, bail};+use serde_json::Value;++mod scrub;+pub use scrub::sanitize_external_text;++/// Reject credentials and control characters without changing retry identities.+pub fn reject_sensitive(value: &Value) -> Result<()> {+ bounded(value, 0)?;+ let mut safe = value.clone();+ sanitize_external_value(&mut safe)?;+ if safe != *value {+ bail!("sensitive envelope rejected before persistence or delivery");+ }+ Ok(())+}++/// Sanitize new payload values before persistence; callers validate identities first.+pub fn sanitize_external_value(value: &mut Value) -> Result<()> {+ bounded(value, 0)?;+ sanitize_value(value)+}++fn bounded(value: &Value, depth: usize) -> Result<()> {+ if depth > 64 {+ bail!("capture value exceeds nesting limit");+ }+ match value {+ Value::String(text) if text.len() > 256 * 1024 => bail!("capture text exceeds byte limit"),+ Value::Array(items) => items.iter().try_for_each(|item| bounded(item, depth + 1))?,+ Value::Object(fields) => {+ for (key, value) in fields {+ bounded(&Value::String(key.clone()), depth + 1)?;+ bounded(value, depth + 1)?;+ }+ }+ _ => {}+ }+ Ok(())+}++fn sanitize_value(value: &mut Value) -> Result<()> {+ match value {+ Value::String(text) => *text = sanitize_external_text(text),+ Value::Array(items) => items.iter_mut().try_for_each(sanitize_value)?,+ Value::Object(fields) => {+ for (key, value) in fields {+ reject_sensitive(&Value::String(key.clone()))?;+ if sensitive_key(key, value) {+ *value = Value::String("[REDACTED]".into());+ } else {+ sanitize_value(value)?;+ }+ }+ }+ _ => {}+ }+ Ok(())+}++fn sensitive_key(key: &str, value: &Value) -> bool {+ let chars: Vec<_> = key.chars().collect();+ let mut name = String::new();+ for (i, ch) in chars.iter().enumerate() {+ if ch.is_ascii_uppercase()+ && i > 0+ && (chars[i - 1].is_ascii_lowercase()+ || (chars[i - 1].is_ascii_uppercase()+ && chars.get(i + 1).is_some_and(char::is_ascii_lowercase)))+ {+ name.push('_');+ }+ name.push(if ch.is_ascii_alphanumeric() {+ ch.to_ascii_lowercase()+ } else {+ '_'+ });+ }+ let metrics = concat!(+ "max_tokens input_tokens token_count token_usage ",+ "output_tokens total_tokens cache_read_input_tokens cache_creation_input_tokens ",+ "prompt_tokens completion_tokens reasoning_tokens"+ );+ let is_metric = metrics.split(' ').any(|metric| metric == name);+ if value.is_number() && is_metric {+ return false;+ }+ let parts: Vec<_> = name.split('_').filter(|part| !part.is_empty()).collect();+ let sensitive = concat!(+ "authorization auth jwt pwd bearer password passwd secret token cookie ",+ "credential apikey privatekey accesskey signature"+ );+ sensitive+ .split_ascii_whitespace()+ .any(|word| parts.iter().any(|part| part.trim_end_matches('s') == word))+ || parts.windows(2).any(|pair| {+ ["api", "private", "access", "session"].contains(&pair[0])+ && matches!(pair[1], "key" | "keys")+ })+}++/// Reject body-supplied authority and payloads requiring further sanitation.+pub fn check_body(body: &Value) -> Result<()> {+ if !body.is_object()+ || ["workspace", "project", "actor", "author_id", "headers"]+ .iter()+ .any(|key| body.get(key).is_some())+ {+ bail!("hook body cannot supply destination or actor authority");+ }+ reject_sensitive(body)?;+ if serde_json::to_vec(body)?.len() > 256 * 1024 {+ bail!("capture body exceeds byte limit");+ }+ Ok(())+}++#[cfg(test)]+#[path = "../tests/suite/mod.rs"]+mod tests;diff --git a/companions/ai-memory-client/src/scrub.rs b/companions/ai-memory-client/src/scrub.rsnew file mode 100644index 00000000..bb9f4d3f--- /dev/null+++ b/companions/ai-memory-client/src/scrub.rs@@ -0,0 +1,84 @@+//! Local credential scrub shared by external capture clients.++use regex::Regex;+use std::sync::OnceLock;++pub fn sanitize_external_text(input: &str) -> String {+ let controls_removed: String = input+ .chars()+ .map(|ch| {+ if ch == '\n'+ || ch == '\r'+ || ch == '\t'+ || !(ch.is_control()+ || matches!(ch, '\u{202a}'..='\u{202e}' | '\u{2066}'..='\u{2069}'))+ {+ ch+ } else {+ '\u{fffd}'+ }+ })+ .collect();+ secret_patterns()+ .iter()+ .fold(controls_removed, |text, rule| {+ rule.regex+ .replace_all(&text, |captures: ®ex::Captures<'_>| {+ let matched = captures.get(0).map_or("", |matched| matched.as_str());+ if captures.get(3).is_some_and(|value| {+ value.as_str().chars().count() < 6+ || [+ "[REDACTED]",+ "[REDACTED CREDENTIAL]",+ "[REDACTED PRIVATE KEY]",+ ]+ .contains(&value.as_str())+ }) {+ matched.to_owned()+ } else {+ let mut replacement = String::new();+ captures.expand(rule.replacement, &mut replacement);+ replacement+ }+ })+ .into_owned()+ })+}++struct RedactionRule {+ regex: Regex,+ replacement: &'static str,+}++fn secret_patterns() -> &'static [RedactionRule] {+ static RULES: OnceLock<Vec<RedactionRule>> = OnceLock::new();+ RULES.get_or_init(|| {+ vec![+ RedactionRule {+ regex: Regex::new(+ r"(?s)-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----",+ )+ .unwrap(),+ replacement: "[REDACTED PRIVATE KEY]",+ },+ RedactionRule {+ regex: Regex::new(+ r"\b(?:github_pat_[A-Za-z0-9_]{20,}|gh[pousr]_[A-Za-z0-9]{20,}|sk-[A-Za-z0-9_-]{16,}|AKIA[0-9A-Z]{16})\b",+ )+ .unwrap(),+ replacement: "[REDACTED CREDENTIAL]",+ },+ RedactionRule {+ regex: Regex::new(r"(?i)\b(bearer\s+)[A-Za-z0-9._~+/=-]{8,}").unwrap(),+ replacement: "$1[REDACTED]",+ },+ RedactionRule {+ regex: Regex::new(+ r#"(?i)\b(api[_-]?key|access[_-]?token|auth[_-]?token|token|secret|password)\b(\s*[:=]\s*)(\"[^\"\r\n]{6,}\"|'[^'\r\n]{6,}'|(?:\[REDACTED(?: CREDENTIAL| PRIVATE KEY)?\]|[^\s,;])+)"#,+ )+ .unwrap(),+ replacement: "$1$2[REDACTED]",+ },+ ]+ })+}diff --git a/companions/ai-memory-client/tests/suite/mod.rs b/companions/ai-memory-client/tests/suite/mod.rsnew file mode 100644index 00000000..5a7b45ce--- /dev/null+++ b/companions/ai-memory-client/tests/suite/mod.rs@@ -0,0 +1,337 @@+use super::*;+use serde_json::json;++#[test]+fn bounded_body_schema_refuses_authority_depth_and_size_with_control() {+ let control = json!({"session_id":"sessão-界", "cwd":"/work", "_ai_memory_capture":{"actor":"untrusted provenance"}});+ check_body(&control).unwrap();+ assert!(check_body(&json!([control.clone()])).is_err());+ for field in ["workspace", "project", "actor", "author_id", "headers"] {+ let mut attack = control.clone();+ attack[field] = json!("forged");+ assert!(+ check_body(&attack).is_err(),+ "body authority must be refused"+ );+ }+ let mut deep = control.clone();+ for _ in 0..65 {+ deep = json!({"nested": deep});+ }+ assert!(check_body(&deep).is_err(), "deep bodies must be refused");+ let large = json!({"first":"x".repeat(140_000),"second":"x".repeat(140_000)});+ assert!(+ check_body(&large).is_err(),+ "body byte cap must be enforced"+ );+ let mut large_text = json!("x".repeat(256 * 1024 + 1));+ assert!(sanitize_external_value(&mut large_text).is_err());+ check_body(&control).unwrap();+}++#[test]+fn complete_literal_redaction_markers_survive_repeated_scrubbing() {+ for marker in [+ "[REDACTED]",+ "[REDACTED CREDENTIAL]",+ "[REDACTED PRIVATE KEY]",+ ] {+ let control = format!("界 token={marker} literal {marker}");+ assert_eq!(sanitize_external_text(&control), control);+ assert_eq!(+ sanitize_external_text(&sanitize_external_text(&control)),+ control+ );+ }+}++#[test]+fn secrets_are_rejected_in_all_envelope_positions_without_exposing_them() {+ for secret in [+ "Bearer abcdefghijklmnop",+ "github_pat_1234567890abcdefghijklmnop",+ "password=abcdefghi",+ "unsafe\u{1b}text",+ ] {+ assert_ne!(sanitize_external_text(secret), secret);+ for envelope in [+ json!(secret),+ json!({"session_id": secret}),+ json!({"cwd": secret}),+ json!({"source": secret}),+ json!({"metadata": [secret]}),+ ] {+ let error = reject_sensitive(&envelope).unwrap_err();+ assert!(!error.to_string().contains(secret));+ }+ }+ assert!(reject_sensitive(&json!({"nested": {"api_key": "short"}})).is_err());+ assert!(+ reject_sensitive(+ &json!({"session_id": "native", "cwd": "/work", "prompt": "ordinary text"})+ )+ .is_ok()+ );+}++#[test]+fn redacted_nested_values_validate_idempotently() {+ let body = json!({"session_id": "native", "cwd": "/work", "nested": {"token": "[REDACTED]", "api-key": "[REDACTED]"}});+ reject_sensitive(&body).unwrap();+ check_body(&body).unwrap();+ let mut again = body.clone();+ sanitize_external_value(&mut again).unwrap();+ assert_eq!(again, body);+}++#[test]+fn sanitation_keeps_native_identity_and_refuses_body_authority() {+ let mut body = json!({"session_id":"native", "cwd":"/work", "output":"\u{1b}[32m token=getToken() Bearer abcdefghijklmnop", "nested":{"password":"process.env.X"}});+ sanitize_external_value(&mut body).unwrap();+ assert_eq!(body["session_id"], "native");+ assert_eq!(body["cwd"], "/work");+ assert_eq!(body["nested"]["password"], "[REDACTED]");+ assert!(!body["output"].as_str().unwrap().contains(['\u{1b}']));+ check_body(&body).unwrap();+ for field in ["workspace", "project", "actor", "author_id", "headers"] {+ let mut forged = body.clone();+ forged[field] = "foreign".into();+ assert!(check_body(&forged).is_err());+ }+}++#[test]+fn sensitive_key_forms_redact_opaque_values_and_preserve_numeric_metrics() {+ let keys = [+ "accessToken",+ "refreshToken",+ "clientSecret",+ "privateKey",+ "x-api-key",+ "OPENAI_API_KEY",+ "accessKey",+ "HTTPAuthorization",+ "cookie",+ "sessionCookies",+ "credential",+ "credentials",+ "password",+ "token_count",+ "token_usage",+ "max_tokens",+ "input_tokens",+ ];+ for key in keys {+ let raw = json!({"nested": [{key: "fixture-value"}]});+ assert_eq!(sanitize_external_text("fixture-value"), "fixture-value");+ assert!(reject_sensitive(&raw).is_err());+ let mut safe = raw;+ sanitize_external_value(&mut safe).unwrap();+ assert_eq!(safe["nested"][0][key], "[REDACTED]");+ reject_sensitive(&safe).unwrap();+ let once = safe.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert_eq!(safe, once);+ }+ let metrics = json!({"max_tokens": 12, "input_tokens": 123456789012345_u64, "token_count": 0, "token_usage": 1.25});+ let mut safe = metrics.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert_eq!(safe, metrics);+ assert_eq!(+ serde_json::to_string(&safe).unwrap(),+ serde_json::to_string(&metrics).unwrap()+ );+ reject_sensitive(&safe).unwrap();+}++#[test]+fn numeric_token_metrics_preserve_representation_without_credential_bypass() {+ let metrics = json!({"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25,"output_tokens":9007199254740993_u64,"total_tokens":89,"totalTokens":89,"cache_read_input_tokens":2,"cache_creation_input_tokens":3,"prompt_tokens":5,"completion_tokens":7,"reasoning_tokens":11});+ let raw = json!({"usage":metrics,"nested":[metrics.clone()]});+ let mut safe = raw.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert!(+ safe == raw,+ "numeric metric values and types must survive sanitation"+ );+ assert!(serde_json::to_vec(&safe).unwrap() == serde_json::to_vec(&raw).unwrap());+ reject_sensitive(&safe).unwrap();+ let once = safe.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe == once);+ for key in metrics.as_object().unwrap().keys() {+ let mut opaque = json!({key:"fixture-value"});+ sanitize_external_value(&mut opaque).unwrap();+ assert!(+ opaque[key] == "[REDACTED]",+ "opaque metric values must be redacted"+ );+ }+ for key in [+ "token",+ "access_tokens",+ "private_token_count",+ "auth_token_usage",+ "authorization_token_limit",+ "private_key",+ "accessKey",+ ] {+ let mut credential = json!({key:123456});+ sanitize_external_value(&mut credential).unwrap();+ assert!(+ credential[key] == "[REDACTED]",+ "numeric credential values must be redacted"+ );+ }+ let mut usage_object = json!({"token_usage":{"input_tokens":12,"output_tokens":34}});+ sanitize_external_value(&mut usage_object).unwrap();+ assert!(usage_object["token_usage"] == "[REDACTED]");+}++#[test]+fn common_credential_components_redact_opaque_values_with_author_controls() {+ for key in [+ "auth",+ "jwt",+ "pwd",+ "bearer",+ "session_key",+ "signature",+ "clientAuth",+ "HTTP_JWT",+ "HTTPJwt",+ "JWT",+ "client-pwd",+ "Bearer",+ "sessionKey",+ "Session-Key",+ "requestSignature",+ ] {+ for value in [json!("fixture-value"), json!(123456)] {+ let raw = json!({"nested":[{key:value}]});+ assert!(+ reject_sensitive(&raw).is_err(),+ "raw credential component must be refused"+ );+ let mut safe = raw;+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe["nested"][0][key] == "[REDACTED]");+ reject_sensitive(&safe).unwrap();+ let once = safe.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe == once);+ }+ }+ let control =+ json!({"author":"fixture-value","authority":"fixture-value","coauthor":"fixture-value"});+ assert!(sanitize_external_text("fixture-value") == "fixture-value");+ let mut safe = control.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert!(+ safe == control,+ "author and authority are legitimate components"+ );+ reject_sensitive(&safe).unwrap();+}++#[test]+fn session_key_pairs_are_sensitive_with_native_session_metadata_control() {+ for key in [+ "session_key",+ "sessionKey",+ "Session-Key",+ "HTTP_SESSION_KEY",+ ] {+ for value in [json!("fixture-value"), json!(123456)] {+ let raw = json!({"nested":[{key:value}]});+ assert!(reject_sensitive(&raw).is_err());+ let mut safe = raw;+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe["nested"][0][key] == "[REDACTED]");+ reject_sensitive(&safe).unwrap();+ let once = safe.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe == once);+ }+ }+ let control = json!({"session":"fixture-value","session_id":"native","key":"fixture-value"});+ let mut safe = control.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe == control);+ reject_sensitive(&safe).unwrap();+}++#[test]+fn plural_credential_key_pairs_redact_opaque_values_recursively() {+ assert!(sanitize_external_text("fixture-value") == "fixture-value");+ for key in [+ "api_keys",+ "access_keys",+ "private_keys",+ "session_keys",+ "apiKeys",+ "accessKeys",+ "privateKeys",+ "sessionKeys",+ "API_KEYS",+ "session-keys",+ ] {+ for value in [+ json!("fixture-value"),+ json!(["fixture-value", {"label":"opaque-value"}]),+ ] {+ let raw = json!({"nested":[{"details":{key:value}}]});+ assert!(+ reject_sensitive(&raw).is_err(),+ "plural key pair must be refused before sanitation"+ );+ let mut safe = raw;+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe["nested"][0]["details"][key] == "[REDACTED]");+ check_body(&safe).unwrap();+ let once = safe.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe == once);+ }+ }+}++#[test]+fn plural_key_guard_preserves_lone_keys_native_ids_and_metrics() {+ let control = json!({+ "key":"fixture-value", "keys":["fixture-value"],+ "session":"fixture-value", "session_id":"native-session", "event_id":"native-event",+ "author":"fixture-value", "authority":"fixture-value",+ "metrics":{"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25,"output_tokens":9007199254740993_u64,"total_tokens":89,"totalTokens":89,"cache_read_input_tokens":2,"cache_creation_input_tokens":3,"prompt_tokens":5,"completion_tokens":7,"reasoning_tokens":11}+ });+ let bytes = serde_json::to_vec(&control).unwrap();+ check_body(&control).unwrap();+ let mut safe = control.clone();+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe == control);+ assert!(serde_json::to_vec(&safe).unwrap() == bytes);+ sanitize_external_value(&mut safe).unwrap();+ assert!(safe == control);+}++#[test]+fn redaction_marker_suffix_never_exempts_a_credential() {+ for marker in [+ "[REDACTED]",+ "[REDACTED CREDENTIAL]",+ "[REDACTED PRIVATE KEY]",+ ] {+ let control = format!("token={marker}");+ assert_eq!(sanitize_external_text(&control), control);+ for attack in [+ format!("token=abcdefghijklmnop{marker}"),+ format!("token={marker}abcdefghijklmnop"),+ ] {+ assert!(reject_sensitive(&json!(attack)).is_err());+ assert_eq!(sanitize_external_text(&attack), "token=[REDACTED]");+ let safe = sanitize_external_text(&attack);+ assert_eq!(sanitize_external_text(&safe), safe);+ }+ assert_eq!(sanitize_external_text(&control), control);+ }+}diff --git a/companions/ai-memory-importer/Cargo.lock b/companions/ai-memory-importer/Cargo.lockindex 9f10bb3f..54e87ace 100644--- a/companions/ai-memory-importer/Cargo.lock+++ b/companions/ai-memory-importer/Cargo.lock@@ -11,13 +11,22 @@ dependencies = ["memchr",]+[[package]]+name = "ai-memory-client"+version = "0.1.0"+dependencies = [+ "anyhow",+ "regex",+ "serde_json",+]+[[package]]name = "ai-memory-importer"version = "0.1.0"dependencies = [+ "ai-memory-client","anyhow","clap",- "regex","reqwest","serde","serde_json",diff --git a/companions/ai-memory-importer/Cargo.toml b/companions/ai-memory-importer/Cargo.tomlindex 0fa7d1e4..14d7caf3 100644--- a/companions/ai-memory-importer/Cargo.toml+++ b/companions/ai-memory-importer/Cargo.toml@@ -7,12 +7,12 @@ edition = "2024"publish = false[dependencies]+ai-memory-client = { path = "../ai-memory-client" }anyhow = "1"clap = { version = "4", features = ["derive", "env"] }# Native roots, matching the workspace: the importer POSTs to the same server# an operator may front with a private CA (#492).reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }-regex = "1"serde = { version = "1", features = ["derive"] }serde_json = "1"serde_yaml = "0.9"diff --git a/companions/ai-memory-importer/README.md b/companions/ai-memory-importer/README.mdindex c5cd13bd..78882427 100644--- a/companions/ai-memory-importer/README.md+++ b/companions/ai-memory-importer/README.md@@ -84,6 +84,13 @@ stubs for them.extension messages. The dry-run and manifest report the truncation count.- Conversation bodies receive client-side credential redaction before replayand then cross ai-memory's normal server sanitizer as a second boundary.+ The [shared private privacy helpers](../ai-memory-client) scrub nested values+ before event hashes, previews, manifests and delivery. Sensitive native source+ identities are refused before deriving an imported session ID or writing a+ manifest. Valid native session strings are preserved exactly. UTF-8 byte caps+ keep complete redaction markers, and numeric token metrics use an exact+ 11-name allowlist. Body provenance remains untrusted and cannot supply actor+ or destination authority.- Stable session IDs derive from `(workspace, project, source, session_id)`.Each replay event has a stable `ingest_key`; a changed transcript gets a newterminal generation key, so rerunning an interrupted import is safe anddiff --git a/companions/ai-memory-importer/src/main.rs b/companions/ai-memory-importer/src/main.rsindex 382d812c..2a96a70c 100644--- a/companions/ai-memory-importer/src/main.rs+++ b/companions/ai-memory-importer/src/main.rs@@ -1,12 +1,11 @@use std::collections::HashMap;use std::fs;use std::path::{Component, Path, PathBuf};-use std::sync::OnceLock;use std::time::Duration;+use ai_memory_client::sanitize_external_text;use anyhow::{Context, Result, anyhow, bail};use clap::{Parser, Subcommand};-use regex::Regex;use reqwest::StatusCode;use serde::{Deserialize, Serialize};use sha2::{Digest, Sha256};@@ -527,6 +526,11 @@ fn plan_external_conversation(server_url: &str,) -> Result<ConversationPlan> {validate_label(workspace, "workspace", 128)?;+ ai_memory_client::reject_sensitive(&serde_json::json!([+ workspace,+ server_url,+ file.to_string_lossy().as_ref()+ ]))?;let metadata = fs::metadata(file).with_context(|| format!("read conversation source {}", file.display()))?;if !metadata.is_file() {@@ -540,12 +544,18 @@ fn plan_external_conversation(}let bytes = fs::read(file).with_context(|| format!("read {}", file.display()))?;let source_sha256 = sha256_hex(&bytes);- let mut envelope: ConversationEnvelope =- serde_json::from_slice(&bytes).context("parse generic conversation JSON")?;+ let mut envelope: ConversationEnvelope = serde_json::from_slice(&bytes)+ .map_err(|_| anyhow!("parse generic conversation JSON: invalid schema"))?;+ ai_memory_client::reject_sensitive(&serde_json::json!({+ "project": envelope.project, "source": envelope.source, "session_id": envelope.session_id,+ }))?;envelope.project = validate_label(&envelope.project, "project", 128)?.to_owned();envelope.source = validate_source_name(&envelope.source)?.to_owned();- envelope.session_id = validate_label(&envelope.session_id, "session_id", 256)?.to_owned();+ if envelope.session_id.len() > 256 || envelope.session_id.chars().any(char::is_control) {+ bail!("session_id must be bounded and free of control characters");+ }+ validate_label(&envelope.session_id, "session_id", 256)?;if envelope.messages.is_empty() {bail!("conversation must contain at least one message");}@@ -704,8 +714,16 @@ fn planned_hook_event(event: &str,role: Option<ConversationRole>,source_event: Option<&str>,- body: serde_json::Value,+ mut body: serde_json::Value,) -> Result<PlannedHookEvent> {+ ai_memory_client::reject_sensitive(+ &serde_json::json!({"session_id": stable_session_id, "event": event, "source_event": source_event}),+ )?;+ if let Some(id) = body.get("session_id") {+ ai_memory_client::reject_sensitive(id)?;+ }+ ai_memory_client::sanitize_external_value(&mut body)?;+ ai_memory_client::check_body(&body)?;let body_bytes = serde_json::to_vec(&body)?;let ingest_key = event_ingest_key(stable_session_id, index, event, &body_bytes);let url = build_hook_url(@@ -717,6 +735,7 @@ fn planned_hook_event(&ingest_key,source_event,)?;+ ai_memory_client::reject_sensitive(&serde_json::json!(url))?;Ok(PlannedHookEvent {index,event: event.to_owned(),@@ -864,65 +883,15 @@ fn truncate_utf8(input: &str, max_bytes: usize) -> String {while end > 0 && !input.is_char_boundary(end) {end -= 1;}+ if let Some(start) = input[..end].rfind('[')+ && input[start..].starts_with("[REDACTED")+ && !input[start..end].contains(']')+ {+ end = start;+ }input[..end].to_owned()}-fn sanitize_external_text(input: &str) -> String {- let controls_removed: String = input- .chars()- .map(|ch| {- if ch == '\n' || ch == '\r' || ch == '\t' || !ch.is_control() {- ch- } else {- '\u{fffd}'- }- })- .collect();- secret_patterns()- .iter()- .fold(controls_removed, |text, rule| {- rule.regex.replace_all(&text, rule.replacement).into_owned()- })-}--struct RedactionRule {- regex: Regex,- replacement: &'static str,-}--fn secret_patterns() -> &'static [RedactionRule] {- static RULES: OnceLock<Vec<RedactionRule>> = OnceLock::new();- RULES.get_or_init(|| {- vec![- RedactionRule {- regex: Regex::new(- r"(?s)-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----.*?-----END [A-Z0-9 ]*PRIVATE KEY-----",- )- .unwrap(),- replacement: "[REDACTED PRIVATE KEY]",- },- RedactionRule {- regex: Regex::new(- r"\b(?:github_pat_[A-Za-z0-9_]{20,}|gh[pousr]_[A-Za-z0-9]{20,}|sk-[A-Za-z0-9_-]{16,}|AKIA[0-9A-Z]{16})\b",- )- .unwrap(),- replacement: "[REDACTED CREDENTIAL]",- },- RedactionRule {- regex: Regex::new(r"(?i)\b(bearer\s+)[A-Za-z0-9._~+/=-]{8,}").unwrap(),- replacement: "$1[REDACTED]",- },- RedactionRule {- regex: Regex::new(- r#"(?i)\b(api[_-]?key|access[_-]?token|auth[_-]?token|token|secret|password)\b(\s*[:=]\s*)(?:\"[^\"\r\n]{6,}\"|'[^'\r\n]{6,}'|[^\s,;]{6,})"#,- )- .unwrap(),- replacement: "$1$2[REDACTED]",- },- ]- })-}-fn conversation_manifest(plan: &ConversationPlan) -> ConversationManifest {ConversationManifest {import_version: CONVERSATION_IMPORT_VERSION.into(),@@ -1355,6 +1324,10 @@ impl ImportClient {}async fn post_hook_batch(&self, events: &[PlannedHookEvent]) -> Result<HookBatchDelivery> {+ for event in events {+ ai_memory_client::check_body(&event.body)?;+ ai_memory_client::reject_sensitive(&serde_json::json!(event.url))?;+ }let items: Vec<_> = events.iter().map(|event| HookBatchItem {@@ -1884,4 +1857,303 @@ mod tests {.accepted_every_event(3));}+ #[tokio::test]+ async fn shared_key_sanitation_precedes_plan_manifest_transport_and_exact_retry() {+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();+ let base = format!("http://{}", listener.local_addr().unwrap());+ let mut raw = serde_json::json!({"session_id":"native", "cwd":"/work/app", "output":"\u{1b}[32mresult\u{1b}[0m\u{202e} token=getToken() Bearer abcdefghijklmnop", "nested":[{"accessToken":"fixture-value", "refreshToken":"fixture-value", "clientSecret":"fixture-value", "privateKey":"fixture-value", "x-api-key":"fixture-value", "OPENAI_API_KEY":"fixture-value", "cookie":"fixture-value", "credentials":"fixture-value"}], "metrics":{"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25}});+ raw["metrics"] = serde_json::json!({"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25,"output_tokens":9007199254740993_u64,"total_tokens":89,"totalTokens":89,"cache_read_input_tokens":2,"cache_creation_input_tokens":3,"prompt_tokens":5,"completion_tokens":7,"reasoning_tokens":11});+ raw["key_forms"] = serde_json::json!([{"auth":"fixture-value","jwt":"fixture-value","pwd":"fixture-value","bearer":"fixture-value","session_key":"fixture-value","signature":"fixture-value"}]);+ raw["plural_forms"] = serde_json::json!([{+ "api_keys":"fixture-value", "access_keys":"fixture-value", "private_keys":"fixture-value", "session_keys":"fixture-value",+ "apiKeys":["fixture-value"], "accessKeys":["fixture-value"], "privateKeys":["fixture-value"], "sessionKeys":["fixture-value"]+ }]);+ raw["controls"] = serde_json::json!({"author":"fixture-value","authority":"fixture-value","key":"fixture-value","keys":["fixture-value"]});+ raw["numeric_cases"] = serde_json::json!({"token":123456,"access_tokens":123456,"private_token_count":123456,"auth_token_usage":123456,"token_usage":{"input_tokens":12}});+ let planned = || {+ planned_hook_event(+ &base,+ "w",+ "p",+ "native",+ 0,+ "post-tool-use",+ None,+ None,+ raw.clone(),+ )+ .unwrap()+ };+ let event = planned();+ assert_eq!(event.body["session_id"], raw["session_id"]);+ assert_eq!(event.body["cwd"], raw["cwd"]);+ assert_eq!(+ event.body["output"],+ sanitize_external_text(raw["output"].as_str().unwrap())+ );+ assert_eq!(event.body["metrics"], raw["metrics"]);+ assert!(+ event.body["nested"][0]+ .as_object()+ .unwrap()+ .values()+ .all(|value| value == "[REDACTED]")+ );+ assert_eq!(+ event.ingest_key,+ event_ingest_key(+ "native",+ 0,+ "post-tool-use",+ &serde_json::to_vec(&event.body).unwrap()+ )+ );+ let serialized = serde_json::to_string(&event).unwrap();+ assert!(+ event.body["key_forms"][0]+ .as_object()+ .unwrap()+ .values()+ .all(|value| value == "[REDACTED]")+ );+ assert!(+ event.body["plural_forms"][0]+ .as_object()+ .unwrap()+ .values()+ .all(|value| value == "[REDACTED]")+ );+ assert!(+ event.body["numeric_cases"]+ .as_object()+ .unwrap()+ .values()+ .all(|value| value == "[REDACTED]")+ );+ assert!(event.body["controls"] == raw["controls"]);+ assert!(+ serde_json::to_vec(&event.body["metrics"]).unwrap()+ == serde_json::to_vec(&raw["metrics"]).unwrap()+ );+ assert!(!serialized.contains("abcdefghijklmnop"));+ ai_memory_client::check_body(&event.body).unwrap();+ let expected = event.body.clone();+ let server = thread::spawn(move || {+ let mut bodies = Vec::new();+ for attempt in 0..2 {+ let (mut socket, _) = listener.accept().unwrap();+ let bytes = read_http_request(&mut socket).1;+ let wire: serde_json::Value = serde_json::from_slice(&bytes).unwrap();+ assert_eq!(wire[0]["body"], expected);+ bodies.push(bytes);+ if attempt == 1 {+ respond_json(&mut socket, "200 OK", r#"{"accepted":1}"#);+ }+ }+ assert_eq!(bodies[0], bodies[1]);+ });+ let client = ImportClient::new(&base).unwrap();+ assert!(client.post_hook_batch(&[event]).await.is_err());+ let retry = planned();+ assert_eq!(serde_json::to_string(&retry).unwrap(), serialized);+ assert_eq!(+ client.post_hook_batch(&[retry]).await.unwrap().ack.accepted,+ 1+ );+ server.join().unwrap();+ }++ #[test]+ fn sanitized_redaction_markers_survive_utf8_byte_caps_in_bodies_and_titles() {+ let root = tempdir().unwrap().keep();+ for (role, cap) in [+ ("user", MAX_USER_MESSAGE_BYTES),+ ("assistant", MAX_EXTENSION_MESSAGE_BYTES),+ ] {+ let prefix = format!("{}{} token=", "界", "x".repeat(cap - 16));+ let source = serde_json::json!({"project":"p","source":"chatgpt","session_id":"native", "messages":[+ {"role":"user", "content":"clean control"},+ {"role":role, "content":format!("{prefix}abcdefghijklmnop\u{1b}[0m\u{202e}")},+ {"role":"assistant", "content":format!("{} token=abcdefghijklmnop", "x".repeat(145))}+ ]});+ let file = write_conversation(&root, source);+ let plan = plan_external_conversation(&file, "w", DEFAULT_SERVER_URL).unwrap();+ for event in &plan.events {+ ai_memory_client::check_body(&event.body).unwrap();+ for field in ["prompt", "message", "title"] {+ if let Some(text) = event.body[field].as_str() {+ assert!(!text.contains("abcdefghijklmnop"));+ assert!(!text.contains(['\u{1b}', '\u{202e}']));+ if let Some(at) = text.rfind('[') {+ assert!(+ text[at..].contains(']'),+ "redaction marker must remain whole"+ );+ }+ }+ }+ }+ let body = &plan.events[2].body;+ let text = body[if role == "user" { "prompt" } else { "message" }]+ .as_str()+ .unwrap();+ assert!(text.len() <= cap);+ assert!(plan.events[3].body["title"].as_str().unwrap().len() <= 160);+ let again = plan_external_conversation(&file, "w", DEFAULT_SERVER_URL).unwrap();+ assert_eq!(plan.transcript_sha256, again.transcript_sha256);+ assert_eq!(plan.events[2].ingest_key, again.events[2].ingest_key);+ }+ }++ #[tokio::test]+ async fn sensitive_native_identity_is_refused_before_manifest_and_http_with_unicode_control() {+ let root = tempdir().unwrap();+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();+ listener.set_nonblocking(true).unwrap();+ let base = format!("http://{}", listener.local_addr().unwrap());+ let mut control = sample_conversation();+ control["session_id"] = " sessão-界-opaque ".into();+ let file = write_conversation(root.path(), control.clone());+ let plan = plan_external_conversation(&file, "w", &base).unwrap();+ assert_eq!(plan.external_session_id, " sessão-界-opaque ");+ assert_eq!(+ plan.events[0].body["external_session_id"],+ control["session_id"]+ );+ assert_eq!(+ plan.stable_session_id,+ stable_external_session_id("w", "memory-lab", "chatgpt", " sessão-界-opaque ")+ );+ for (field, value) in [+ ("project", "sk-1234567890abcdefghijklmnop"),+ ("source", "sk-1234567890abcdefghijklmnop"),+ ("session_id", "sk-1234567890abcdefghijklmnop"),+ ("session_id", "\tnative"),+ ("session_id", "native\n"),+ ("session_id", "\rnative"),+ ] {+ let mut source = control.clone();+ source[field] = value.into();+ let file = write_conversation(root.path(), source);+ assert!(+ plan_external_conversation(&file, "w", &base).is_err(),+ "sensitive native identity must be refused before planning"+ );+ let manifest = root.path().join(format!("{field}-manifest.json"));+ let error = run_conversation(ConversationArgs {+ file,+ workspace: Some("w".into()),+ server_url: base.clone(),+ apply: true,+ manifest_out: Some(manifest.clone()),+ create_destination: false,+ show_body: true,+ })+ .await;+ assert!(error.is_err(), "sensitive source identity must be refused");+ assert!(!error.unwrap_err().to_string().contains("abcdefghijklmnop"));+ assert!(+ !manifest.exists(),+ "refused identity cannot create a manifest"+ );+ assert!(+ listener.accept().is_err(),+ "refused identity cannot reach HTTP"+ );+ }+ let mut oversized = control.clone();+ oversized["session_id"] = format!("native{}", " ".repeat(256)).into();+ let oversized_file = write_conversation(root.path(), oversized);+ assert!(+ plan_external_conversation(&oversized_file, "w", &base).is_err(),+ "native ID bounds apply before trimming"+ );+ let file = write_conversation(root.path(), control);+ let manifest = root.path().join("control-manifest.json");+ run_conversation(ConversationArgs {+ file,+ workspace: Some("w".into()),+ server_url: base,+ apply: false,+ manifest_out: Some(manifest.clone()),+ create_destination: false,+ show_body: true,+ })+ .await+ .unwrap();+ assert!(manifest.exists());+ assert!(listener.accept().is_err());+ }++ #[tokio::test]+ async fn preview_and_manifest_use_scrubbed_bodies_and_digest() {+ let root = tempdir().unwrap();+ let mut source = sample_conversation();+ source["messages"][1]["content"] =+ "界 token=abcdefghijklmnop[REDACTED] literal [REDACTED PRIVATE KEY]".into();+ let file = write_conversation(root.path(), source.clone());+ let plan = plan_external_conversation(&file, "w", DEFAULT_SERVER_URL).unwrap();+ let preview = serde_json::to_string(&plan.events).unwrap();+ assert!(!preview.contains("abcdefghijklmnop"));+ assert!(preview.contains("literal [REDACTED PRIVATE KEY]"));+ source["messages"][1]["content"] =+ sanitize_external_text(source["messages"][1]["content"].as_str().unwrap()).into();+ let safe_file = root.path().join("safe-conversation.json");+ fs::write(&safe_file, serde_json::to_vec(&source).unwrap()).unwrap();+ let safe = plan_external_conversation(&safe_file, "w", DEFAULT_SERVER_URL).unwrap();+ assert_eq!(plan.transcript_sha256, safe.transcript_sha256);+ assert_eq!(plan.events[2].ingest_key, safe.events[2].ingest_key);+ let manifest = root.path().join("manifest.json");+ run_conversation(ConversationArgs {+ file,+ workspace: Some("w".into()),+ server_url: DEFAULT_SERVER_URL.into(),+ apply: false,+ manifest_out: Some(manifest.clone()),+ create_destination: false,+ show_body: true,+ })+ .await+ .unwrap();+ let stored = fs::read_to_string(manifest).unwrap();+ assert!(!stored.contains("abcdefghijklmnop"));+ assert_eq!(+ serde_json::from_str::<serde_json::Value>(&stored).unwrap()["transcript_sha256"],+ plan.transcript_sha256+ );+ }++ #[tokio::test]+ async fn forged_body_authority_is_refused_before_transport() {+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();+ listener.set_nonblocking(true).unwrap();+ let base = format!("http://{}", listener.local_addr().unwrap());+ let clean = planned_hook_event(+ &base,+ "w",+ "p",+ "sessão-界",+ 0,+ "session-start",+ None,+ None,+ serde_json::json!({"session_id":"sessão-界"}),+ )+ .unwrap();+ let mut client = ImportClient::new(&base).unwrap();+ client.client = reqwest::Client::builder()+ .timeout(Duration::from_millis(100))+ .build()+ .unwrap();+ for field in ["workspace", "project", "actor", "author_id", "headers"] {+ let mut attack = clean.clone();+ attack.body[field] = "forged".into();+ assert!(client.post_hook_batch(&[attack]).await.is_err());+ assert!(+ listener.accept().is_err(),+ "forged body authority must be refused before POST"+ );+ }+ ai_memory_client::check_body(&clean.body).unwrap();+ }}diff --git a/companions/ai-memory-relay/Cargo.lock b/companions/ai-memory-relay/Cargo.lockindex 13ea88d2..e1a852c5 100644--- a/companions/ai-memory-relay/Cargo.lock+++ b/companions/ai-memory-relay/Cargo.lock@@ -14,10 +14,29 @@ dependencies = ["zerocopy",]+[[package]]+name = "aho-corasick"+version = "1.1.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "c982642fa9e8606056828ee9a8505737230110bb1099153c79efe865c59d12ba"+dependencies = [+ "memchr",+]++[[package]]+name = "ai-memory-client"+version = "0.1.0"+dependencies = [+ "anyhow",+ "regex",+ "serde_json",+]+[[package]]name = "ai-memory-relay"version = "0.1.0"dependencies = [+ "ai-memory-client","anyhow","clap","fs2",@@ -872,6 +891,35 @@ dependencies = ["rand_core",]+[[package]]+name = "regex"+version = "1.13.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d"+dependencies = [+ "aho-corasick",+ "memchr",+ "regex-automata",+ "regex-syntax",+]++[[package]]+name = "regex-automata"+version = "0.4.18"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"+dependencies = [+ "aho-corasick",+ "memchr",+ "regex-syntax",+]++[[package]]+name = "regex-syntax"+version = "0.8.11"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4"+[[package]]name = "reqwest"version = "0.12.28"diff --git a/companions/ai-memory-relay/Cargo.toml b/companions/ai-memory-relay/Cargo.tomlindex 8ae374d1..8ceec15e 100644--- a/companions/ai-memory-relay/Cargo.toml+++ b/companions/ai-memory-relay/Cargo.toml@@ -17,6 +17,7 @@ path = "src/main.rs"unsafe_code = "forbid"[dependencies]+ai-memory-client = { path = "../ai-memory-client" }anyhow = "1"# Read the bearer token separately to keep it out of clap's Debug output.clap = { version = "4", features = ["derive"] }diff --git a/companions/ai-memory-relay/README.md b/companions/ai-memory-relay/README.mdindex ec964f20..83a000f0 100644--- a/companions/ai-memory-relay/README.md+++ b/companions/ai-memory-relay/README.md@@ -81,8 +81,10 @@ ai-memory-relay status --queue-dir "$HOME/.ai-memory-relay"`enqueue` validates the whole array and commits it in one transaction. Each bodymust be an object with explicit `session_id` and `cwd` strings. Use the harness's-native identity and canonical hook event names. Other body fields keep their-values, including an `_ai_memory_capture` block. The relay does not translate+native identity and canonical hook event names. Sensitive identities are refused+without renaming them. Shared private capture helpers scrub new body text and+credential values before serialization, hashing and enqueueing. An+`_ai_memory_capture` block remains untrusted provenance. The relay does not translateprovider transcripts or infer parent agents and workflows.For authenticated servers, supply `AI_MEMORY_AUTH_TOKEN` through the flush@@ -151,7 +153,13 @@ print event bodies. Receipts count acknowledgements, including policy drops.## Local data and recovery-The queue contains producer-supplied event bodies before server sanitization.+New queue bodies receive local credential sanitation before persistence, then+cross the normal server sanitizer on delivery. Legacy pending bytes, digest,+native identity, ingest key and attempt state remain exact. An unsafe stored+head is retained with a payload-free `privacy/validation` diagnostic; only its+session is deferred while clean sessions continue. No attempt or receipt is+recorded for the refused item. See the [shared privacy helpers](../ai-memory-client).+The producer must apply its capture exclusions before enqueueing. The relaydoes not load the project's `[capture] ignore_paths` settings or inspect toolpayloads for sensitive paths. Server sanitization cannot protect a local queuediff --git a/companions/ai-memory-relay/src/identity.rs b/companions/ai-memory-relay/src/identity.rsindex 672e989d..6363022e 100644--- a/companions/ai-memory-relay/src/identity.rs+++ b/companions/ai-memory-relay/src/identity.rs@@ -1,8 +1,8 @@//! Envelope validation and the stable retry identity.//!//! Validate input before enqueueing so malformed events cannot block a batch.-//! Body values, including `_ai_memory_capture`, are preserved for the server.-//! These checks do not sanitize locally queued content.+//! New body values are sanitized before hashing and persistence. Native identities+//! are validated first; stored retry bytes are never rewritten.use serde::Deserialize;use sha2::{Digest, Sha256};@@ -40,7 +40,7 @@ pub const TERMINAL_EVENTS: &[&str] = &["session-end"];////// `deny_unknown_fields` applies to the envelope only. `body` is an opaque/// object: unknown keys inside it (a harness payload field, the-/// `_ai_memory_capture` protocol block) are preserved untouched.+/// `_ai_memory_capture` protocol block) remain untrusted and are sanitized.#[derive(Debug, Deserialize)]#[serde(deny_unknown_fields)]pub struct InputEvent {@@ -50,7 +50,7 @@ pub struct InputEvent {pub agent: String,/// Canonical lifecycle event name, used for both `event` and `source_event`.pub event: String,- /// Harness payload, delivered verbatim.+ /// Harness payload, sanitized at new-event ingress.pub body: serde_json::Value,}@@ -62,8 +62,7 @@ pub struct ValidEvent {pub event: String,pub session_id: String,pub cwd: String,- /// Canonical JSON of the body exactly as supplied (sorted keys, no reformatting- /// of values). The bytes delivered later are these bytes.+ /// Canonical JSON after ingress sanitation. Retries use these exact bytes.pub body_json: String,pub body_sha256: String,pub ingest_key: String,@@ -110,6 +109,8 @@ pub fn check_actor(actor: &str) -> Result<(), String> {/// Validate a workspace or project name.pub fn check_scope(label: &str, value: &str) -> Result<(), String> {+ ai_memory_client::reject_sensitive(&serde_json::json!(value))+ .map_err(|_| format!("unsafe --{label} rejected"))?;is_plain(value, MAX_SCOPE_LEN).then_some(()).ok_or_else(|| format!("--{label} must be 1..={MAX_SCOPE_LEN} printable characters"))@@ -152,11 +153,34 @@ pub fn ingest_key(/// quote body content: an invalid payload must not become a log line.pub fn validate(index: usize,- input: InputEvent,+ mut input: InputEvent,producer: &str,actor: &str,) -> Result<ValidEvent, String> {let at = |detail: &str| format!("item {index}: {detail}");+ ai_memory_client::reject_sensitive(&serde_json::json!({+ "event_id": input.event_id, "agent": input.agent, "event": input.event,+ "producer": producer, "actor": actor,+ }))+ .map_err(|_| at("sensitive envelope rejected"))?;+ let identities: Vec<_> = [+ "session_id",+ "cwd",+ "producer",+ "actor",+ "event_id",+ "agent",+ "event",+ "ingest_key",+ ]+ .iter()+ .filter_map(|key| input.body.get(key))+ .collect();+ ai_memory_client::reject_sensitive(&serde_json::json!(identities))+ .map_err(|_| at("unsafe body identity rejected"))?;+ ai_memory_client::sanitize_external_value(&mut input.body)+ .map_err(|_| at("unsafe hook body rejected"))?;+ ai_memory_client::check_body(&input.body).map_err(|_| at("unsafe hook body rejected"))?;if !is_plain(&input.event_id, MAX_EVENT_ID_LEN) {return Err(at(&format!("event_id must be 1..={MAX_EVENT_ID_LEN} printable characters"diff --git a/companions/ai-memory-relay/src/lib.rs b/companions/ai-memory-relay/src/lib.rsindex 7a1a3a36..f8a0bcb7 100644--- a/companions/ai-memory-relay/src/lib.rs+++ b/companions/ai-memory-relay/src/lib.rs@@ -5,7 +5,7 @@//! Pending events remain on disk when delivery fails or their retry window//! expires; reaching a capacity limit rejects new input.//!-//! Bodies retain their input values before server sanitization. Producers must+//! New bodies are sanitized before enqueueing; stored retries stay exact. Producers must//! apply capture exclusions before enqueueing and protect the queue directory.pub mod ack;diff --git a/companions/ai-memory-relay/src/relay.rs b/companions/ai-memory-relay/src/relay.rsindex 06b562f8..646e68fc 100644--- a/companions/ai-memory-relay/src/relay.rs+++ b/companions/ai-memory-relay/src/relay.rs@@ -96,6 +96,9 @@ pub fn init(identity::check_actor(actor).map_err(anyhow::Error::msg)?;identity::check_scope("workspace", workspace).map_err(anyhow::Error::msg)?;identity::check_scope("project", project).map_err(anyhow::Error::msg)?;+ ai_memory_client::reject_sensitive(&serde_json::json!({+ "producer": producer, "actor": actor, "server_url": server_url,+ }))?;let binding = Binding {server_url: http::normalize_server_url(server_url)?,producer: producer.to_owned(),@@ -274,6 +277,8 @@ pub fn flush(dir: &Path, options: &FlushOptions) -> Result<Report> {fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {let mut queue = Queue::open(dir)?;let binding = queue.binding()?;+ identity::check_scope("workspace", &binding.workspace).map_err(anyhow::Error::msg)?;+ identity::check_scope("project", &binding.project).map_err(anyhow::Error::msg)?;queue.prune(crate::now_ms())?;let token = Token::from_env()?;let authenticated = token.is_some();@@ -297,7 +302,7 @@ fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {let now = crate::now_ms();let base = binding.server_url.clone();let for_cost = binding.clone();- let batch: Batch =+ let mut batch: Batch =queue.select_batch(MAX_BATCH_ITEMS, MAX_BATCH_BYTES, now, &deferred, |item| {item_cost(&base, &for_cost, item)})?;@@ -305,22 +310,42 @@ fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {if batch.items.is_empty() {break;}+ let mut items = Vec::new();+ for item in &batch.items {+ let wire = serde_json::from_str(&item.body_json).ok().and_then(|body| {+ ai_memory_client::reject_sensitive(&serde_json::json!([+ item.event_id,+ item.agent,+ item.event,+ item.session_id,+ ]))+ .ok()?;+ ai_memory_client::check_body(&body).ok()?;+ let url = http::item_url(&binding.server_url, &binding, item);+ ai_memory_client::reject_sensitive(&serde_json::json!(url)).ok()?;+ Some(BatchItem { url, body })+ });+ if let Some(wire) = wire {+ items.push(wire);+ } else {+ queue.record_failure(&item.ingest_key, "privacy/validation")?;+ deferred.insert(item.session());+ failure.get_or_insert_with(|| {+ "privacy/validation: unsafe stored body retained; its session deferred".into()+ });+ }+ }+ batch+ .items+ .retain(|item| !deferred.contains(&item.session()));+ if items.is_empty() {+ continue;+ }batches += 1;let keys: Vec<String> = batch.items.iter().map(|i| i.ingest_key.clone()).collect();// Durable, and before the request: a lost ack must not restart the- // 30-day clock on the next run.+ // 30-day clock on the next run. Unsafe items were already excluded.queue.stamp_attempt(&keys, now)?;- let items: Vec<BatchItem> = batch- .items- .iter()- .map(|item| {- Ok(BatchItem {- url: http::item_url(&binding.server_url, &binding, item),- body: serde_json::from_str(&item.body_json)- .context("stored body is no longer valid JSON")?,- })- })- .collect::<Result<Vec<_>>>()?;// Selection checked the retry window once; a retry inside this batch can// still cross it. Carry the batch's tightest expiry into the sender so// every attempt is re-checked against it.@@ -473,7 +498,7 @@ fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {}if !deferred.is_empty() {summary.push(format!(- "{} session(s) deferred to a later flush (a failed or rate-limited head); \+ "{} session(s) deferred to a later flush (an unsafe, failed or rate-limited head); \their later events were never sent ahead of it",deferred.len()));diff --git a/companions/ai-memory-relay/tests/relay.rs b/companions/ai-memory-relay/tests/relay.rsindex 05900db2..17461312 100644--- a/companions/ai-memory-relay/tests/relay.rs+++ b/companions/ai-memory-relay/tests/relay.rs@@ -1745,3 +1745,503 @@ fn results_match_sparse_partial_failure_and_empty_acks() {assert_eq!(ack::validate(len, &parsed).unwrap(), expected);}}++#[test]+fn sensitive_envelope_is_rejected_before_enqueue_with_clean_control() {+ let root = fixture("sensitive-envelope");+ let dir = bind(&root, "http://127.0.0.1:49374");+ enqueue(+ &dir,+ &root,+ "before.json",+ serde_json::json!([event("opaque-界", "codex", "session-start", "sessão-界")]),+ )+ .unwrap();+ assert_eq!(pending_count(&dir), 1);+ for field in [+ "session_id",+ "cwd",+ "producer",+ "actor",+ "event_id",+ "agent",+ "event",+ "ingest_key",+ ] {+ let mut input = event("a", "codex", "session-start", "a");+ input["body"][field] = "Bearer abcdefghijklmnop".into();+ let error = enqueue(&dir, &root, "input.json", serde_json::json!([input])).unwrap_err();+ assert!(!error.to_string().contains("abcdefghijklmnop"));+ assert_eq!(pending_count(&dir), 1);+ }+ enqueue(+ &dir,+ &root,+ "control.json",+ serde_json::json!([event("a", "codex", "session-start", "a")]),+ )+ .unwrap();+ assert_eq!(pending_count(&dir), 2);+}++#[test]+fn new_native_bodies_are_sanitized_before_hash_enqueue_and_exact_retry() {+ let root = fixture("ingress-sanitized-retry");+ let server = stub(vec![Reply::Close, Reply::AcceptAll]);+ let dir = bind(&root, &server.url());+ let mut raw = event("native-event", "codex", "post-tool-use", "native-session");+ for (name, value) in [+ ("producer", "example.runtime"),+ ("event_id", "native-event"),+ ("agent", "codex"),+ ("event", "post-tool-use"),+ ("ingest_key", "native-ingest-key"),+ ] {+ raw["body"][name] = value.into();+ }+ raw["body"]["output"] =+ "\u{1b}[32mresult\u{1b}[0m\u{202e} token=getToken() Bearer abcdefghijklmnop".into();+ raw["body"]["tool_input"] = serde_json::json!({"nested": {"password": "process.env.X", "api-key": "tiny", "items": ["password=abcdefghi", "token=abcdefghijklmnop[REDACTED]"]}});+ raw["body"]["key_cases"] = serde_json::json!([{+ "accessToken":"fixture-value", "refreshToken":"fixture-value", "clientSecret":"fixture-value", "privateKey":"fixture-value", "x-api-key":"fixture-value", "OPENAI_API_KEY":"fixture-value", "credentials":"fixture-value", "cookie":"fixture-value"+ }]);+ raw["body"]["key_forms"] = serde_json::json!([{"auth":"fixture-value","jwt":"fixture-value","pwd":"fixture-value","bearer":"fixture-value","sessionKey":"fixture-value","signature":"fixture-value"}]);+ raw["body"]["plural_forms"] = serde_json::json!([{+ "api_keys":"fixture-value", "access_keys":"fixture-value", "private_keys":"fixture-value", "session_keys":"fixture-value",+ "apiKeys":["fixture-value"], "accessKeys":["fixture-value"], "privateKeys":["fixture-value"], "sessionKeys":["fixture-value"]+ }]);+ raw["body"]["controls"] = serde_json::json!({"author":"fixture-value","authority":"fixture-value","key":"fixture-value","keys":["fixture-value"]});+ raw["body"]["numeric_cases"] = serde_json::json!({"token":123456,"access_tokens":123456,"private_token_count":123456,"auth_token_usage":123456,"token_usage":{"input_tokens":12}});+ let metrics = serde_json::json!({"max_tokens":12,"input_tokens":34,"token_count":0,"token_usage":1.25,"output_tokens":9007199254740993_u64,"total_tokens":89,"totalTokens":89,"cache_read_input_tokens":2,"cache_creation_input_tokens":3,"prompt_tokens":5,"completion_tokens":7,"reasoning_tokens":11});+ raw["body"]["metrics"] = metrics.clone();+ enqueue(&dir, &root, "native.json", serde_json::json!([raw.clone()])).unwrap();+ enqueue(&dir, &root, "repeat.json", serde_json::json!([raw])).unwrap();+ assert_eq!(pending_count(&dir), 1);+ let conn = rusqlite::Connection::open(dir.join(ai_memory_relay::fsguard::DB_FILE)).unwrap();+ let (body, digest, key, id, agent, event, session): (+ String,+ String,+ String,+ String,+ String,+ String,+ String,+ ) = conn+ .query_row(+ "SELECT body_json,body_sha256,ingest_key,event_id,agent,event,session_id FROM pending",+ [],+ |row| {+ Ok((+ row.get(0)?,+ row.get(1)?,+ row.get(2)?,+ row.get(3)?,+ row.get(4)?,+ row.get(5)?,+ row.get(6)?,+ ))+ },+ )+ .unwrap();+ let safe: serde_json::Value = serde_json::from_str(&body).unwrap();+ assert_eq!(+ (&id[..], &agent[..], &event[..], &session[..]),+ ("native-event", "codex", "post-tool-use", "native-session")+ );+ assert_eq!(+ key,+ identity::ingest_key(+ "example.runtime",+ "operator-a",+ &agent,+ &session,+ &event,+ &id+ )+ );+ for (name, value) in [+ ("producer", "example.runtime"),+ ("event_id", "native-event"),+ ("agent", "codex"),+ ("event", "post-tool-use"),+ ("ingest_key", "native-ingest-key"),+ ] {+ assert_eq!(safe[name], value);+ }+ assert_eq!(safe["cwd"], "/work/app");+ assert!(+ !safe["output"]+ .as_str()+ .unwrap()+ .contains(['\u{1b}', '\u{202e}'])+ );+ assert!(!body.contains("abcdefghijklmnop"));+ assert!(!body.contains("process.env.X"));+ assert!(!body.contains("tiny"));+ assert_eq!(safe["tool_input"]["nested"]["api-key"], "[REDACTED]");+ assert!(+ safe["key_forms"][0]+ .as_object()+ .unwrap()+ .values()+ .all(|value| value == "[REDACTED]")+ );+ assert!(+ safe["plural_forms"][0]+ .as_object()+ .unwrap()+ .values()+ .all(|value| value == "[REDACTED]")+ );+ assert!(+ safe["numeric_cases"]+ .as_object()+ .unwrap()+ .values()+ .all(|value| value == "[REDACTED]")+ );+ assert!(+ safe["controls"]+ == serde_json::json!({"author":"fixture-value","authority":"fixture-value","key":"fixture-value","keys":["fixture-value"]})+ );+ assert!(+ safe["key_cases"][0]+ .as_object()+ .unwrap()+ .values()+ .all(|value| value == "[REDACTED]")+ );+ assert!(safe["metrics"] == metrics);+ assert!(serde_json::to_vec(&safe["metrics"]).unwrap() == serde_json::to_vec(&metrics).unwrap());+ assert_eq!(+ safe["output"],+ ai_memory_client::sanitize_external_text(+ "\u{1b}[32mresult\u{1b}[0m\u{202e} token=getToken() Bearer abcdefghijklmnop"+ )+ );++ ai_memory_client::check_body(&safe).unwrap();+ let validated = identity::validate(+ 0,+ InputEvent {+ event_id: id,+ agent,+ event,+ body: safe.clone(),+ },+ "example.runtime",+ "operator-a",+ )+ .unwrap();+ assert_eq!(validated.body_json, body);+ assert_eq!(validated.body_sha256, digest);+ assert!(flush(&dir).failure.is_none());+ assert_eq!(pending_count(&dir), 0);+ let sent = server.bodies();+ assert_eq!(sent.len(), 2);+ assert_eq!(sent[0], sent[1]);+ assert_eq!(sent[0][0]["body"], safe);+ assert_eq!(serde_json::to_string(&sent[0][0]["body"]).unwrap(), body);+ assert_eq!(+ Queue::open(&dir)+ .unwrap()+ .stats(ai_memory_relay::now_ms())+ .unwrap()+ .receipts,+ 1+ );+}++#[test]+fn unsafe_legacy_pending_is_retained_exactly_and_never_sent() {+ let root = fixture("unsafe-legacy-pending");+ let server = stub(vec![Reply::AcceptAll, Reply::AcceptAll]);+ let dir = bind(&root, &server.url());+ let mut legacy = valid("native-event", "codex", "session-start", "native-session");+ legacy.body_json = "{ \"prompt\": \"Bearer abcdefghijklmnop\", \"cwd\": \"/native/path\", \"session_id\": \"native-session\" }".into();+ let mut queue = Queue::open(&dir).unwrap();+ queue+ .enqueue(&[legacy.clone()], ai_memory_relay::now_ms())+ .unwrap();+ let first_attempt = ai_memory_relay::now_ms() - 1000;+ queue+ .stamp_attempt(&[legacy.ingest_key.clone()], first_attempt)+ .unwrap();+ queue+ .stamp_attempt(&[legacy.ingest_key.clone()], first_attempt)+ .unwrap();+ let clean = valid("clean-event", "codex", "session-start", "clean-session");+ queue+ .enqueue(+ &[+ valid("later-dirty", "codex", "session-end", "native-session"),+ clean.clone(),+ ],+ ai_memory_relay::now_ms(),+ )+ .unwrap();+ drop(queue);+ for _ in 0..2 {+ let report = relay::flush(&dir, &FlushOptions::default()).unwrap();+ assert!(+ report+ .failure+ .as_ref()+ .is_some_and(|failure| failure.contains("privacy/validation"))+ );+ assert!(!format!("{report:?}").contains("abcdefghijklmnop"));+ assert!(report.pending);+ assert_eq!(pending_count(&dir), 2);+ }+ let sent = server.bodies();+ assert_eq!(sent.len(), 1);+ assert_eq!(+ sent[0][0]["body"],+ serde_json::from_str::<serde_json::Value>(&clean.body_json).unwrap()+ );+ assert_eq!(+ Queue::open(&dir)+ .unwrap()+ .stats(ai_memory_relay::now_ms())+ .unwrap()+ .receipts,+ 1+ );+ let conn = rusqlite::Connection::open(dir.join(ai_memory_relay::fsguard::DB_FILE)).unwrap();+ let preserved: (String, String, String, String, Option<i64>, i64) = conn+ .query_row(+ "SELECT ingest_key,session_id,body_json,body_sha256,first_attempt_ms,attempts FROM pending WHERE event_id=?1",+ [&legacy.event_id],+ |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?, row.get(4)?, row.get(5)?)),+ )+ .unwrap();+ assert!(+ preserved+ == (+ legacy.ingest_key,+ legacy.session_id,+ legacy.body_json,+ legacy.body_sha256,+ Some(first_attempt),+ 2+ ),+ "unsafe legacy identity, bytes, digest and attempt state must remain unchanged"+ );+ let errors: Vec<(String, Option<String>, Option<i64>)> = conn+ .prepare("SELECT event_id,last_error,first_attempt_ms FROM pending ORDER BY seq")+ .unwrap()+ .query_map([], |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)))+ .unwrap()+ .map(Result::unwrap)+ .collect();+ assert_eq!(+ errors,+ [+ (+ "native-event".into(),+ Some("privacy/validation".into()),+ Some(first_attempt)+ ),+ ("later-dirty".into(), None, None)+ ]+ );+}++#[test]+fn legacy_binding_scope_spaces_are_preserved_on_delivery() {+ let root = fixture("legacy-scope-spaces");+ let server = stub(vec![Reply::AcceptAll]);+ let dir = queue_dir(&root);+ relay::init(+ &dir,+ &server.url(),+ "example.runtime",+ "operator-a",+ " team ",+ " app ",+ )+ .unwrap();+ enqueue(+ &dir,+ &root,+ "native.json",+ serde_json::json!([event("e", "codex", "session-start", "s")]),+ )+ .unwrap();+ assert!(flush(&dir).failure.is_none());+ let bodies = server.bodies();+ let url = url::Url::parse(bodies[0][0]["url"].as_str().unwrap()).unwrap();+ let pairs: HashMap<_, _> = url.query_pairs().into_owned().collect();+ assert_eq!(pairs["workspace"], " team ");+ assert_eq!(pairs["project"], " app ");+}++#[test]+fn opaque_unicode_native_identity_ships_unchanged() {+ let root = fixture("privacy-unicode-control");+ let server = stub(vec![Reply::AcceptAll]);+ let dir = bind(&root, &server.url());+ let mut input = event("event-界", "codex", "session-start", " sessão-界/opaque ");+ input["body"]["cwd"] = "/work/界".into();+ input["body"]["_ai_memory_capture"] =+ serde_json::json!({"actor":"untrusted-label", "authority":"untrusted provenance"});+ let original = input["body"].clone();+ enqueue(&dir, &root, "input.json", serde_json::json!([input])).unwrap();+ assert!(flush(&dir).failure.is_none());+ assert_eq!(server.bodies()[0][0]["body"], original);+ let bodies = server.bodies();+ let url = url::Url::parse(bodies[0][0]["url"].as_str().unwrap()).unwrap();+ assert_eq!(bodies[0][0]["body"]["session_id"], " sessão-界/opaque ");+ assert_eq!(+ url.query_pairs()+ .find(|(key, _)| key == "ingest_key")+ .unwrap()+ .1,+ identity::ingest_key(+ "example.runtime",+ "operator-a",+ "codex",+ " sessão-界/opaque ",+ "session-start",+ "event-界"+ )+ );+ assert!(+ server+ .heads+ .lock()+ .unwrap()+ .iter()+ .all(|head| head.starts_with("POST /hook/batch "))+ );+}++#[test]+fn sensitive_outer_identity_and_body_authority_are_refused_before_enqueue() {+ let root = fixture("privacy-outer-refusal");+ let dir = bind(&root, "http://127.0.0.1:49374");+ enqueue(+ &dir,+ &root,+ "before.json",+ serde_json::json!([event("before", "codex", "session-start", "control")]),+ )+ .unwrap();+ for field in ["event_id", "agent", "event"] {+ let mut attack = event("native", "codex", "session-start", "session");+ attack[field] = "sk-1234567890abcdefghijklmnop".into();+ let result = enqueue(&dir, &root, "attack.json", serde_json::json!([attack]));+ assert!(result.is_err(), "sensitive outer identity must be refused");+ assert!(!result.unwrap_err().to_string().contains("abcdefghijklmnop"));+ assert_eq!(pending_count(&dir), 1);+ }+ for field in ["workspace", "project", "actor", "author_id", "headers"] {+ let mut attack = event("native", "codex", "session-start", "session");+ attack["body"][field] = "forged".into();+ assert!(enqueue(&dir, &root, "attack.json", serde_json::json!([attack])).is_err());+ assert_eq!(pending_count(&dir), 1);+ }+ enqueue(+ &dir,+ &root,+ "after.json",+ serde_json::json!([event("after", "codex", "session-end", "control")]),+ )+ .unwrap();+ assert_eq!(pending_count(&dir), 2);+}++#[test]+fn sensitive_binding_is_refused_before_queue_creation() {+ let root = fixture("privacy-binding-refusal");+ let safe = ["example.runtime", "operator-a", "team", "app"];+ let before = root.join("control-before");+ relay::init(+ &before,+ "http://127.0.0.1:49374",+ safe[0],+ safe[1],+ safe[2],+ safe[3],+ )+ .unwrap();+ for field in 0..4 {+ let mut attack = safe;+ attack[field] = "sk-1234567890abcdefghijklmnop";+ let dir = root.join(format!("attack-{field}"));+ let result = relay::init(+ &dir,+ "http://127.0.0.1:49374",+ attack[0],+ attack[1],+ attack[2],+ attack[3],+ );+ assert!(result.is_err(), "sensitive binding must be refused");+ assert!(!result.unwrap_err().to_string().contains("abcdefghijklmnop"));+ assert!(!dir.exists(), "sensitive binding must not create the queue");+ }+ relay::init(+ &root.join("control-after"),+ "http://127.0.0.1:49374",+ safe[0],+ safe[1],+ safe[2],+ safe[3],+ )+ .unwrap();+}++#[test]+fn unsafe_legacy_native_identity_defers_only_its_session() {+ let root = fixture("privacy-legacy-native-id");+ let server = stub(vec![Reply::AcceptAll, Reply::AcceptAll]);+ let dir = bind(&root, &server.url());+ let mut legacy = valid("old-event", "codex", "session-start", "old-session");+ legacy.event_id = "sk-1234567890abcdefghijklmnop".into();+ let mut queue = Queue::open(&dir).unwrap();+ let before = valid("before", "codex", "session-start", "clean-before");+ queue+ .enqueue(&[legacy.clone(), before.clone()], ai_memory_relay::now_ms())+ .unwrap();+ drop(queue);+ let report = flush(&dir);+ assert!(+ report+ .failure+ .is_some_and(|failure| failure.contains("privacy/validation")),+ "unsafe legacy native identity must be refused"+ );+ assert_eq!(pending_count(&dir), 1);+ enqueue(+ &dir,+ &root,+ "after.json",+ serde_json::json!([event("after", "codex", "session-start", "clean-after")]),+ )+ .unwrap();+ let report = flush(&dir);+ assert!(report.failure.is_some());+ assert_eq!(pending_count(&dir), 1);+ let sent = server.bodies();+ assert_eq!(sent.len(), 2);+ assert_eq!(+ sent[0][0]["body"],+ serde_json::from_str::<serde_json::Value>(&before.body_json).unwrap()+ );+ assert_eq!(sent[1][0]["body"]["session_id"], "clean-after");+ let conn = rusqlite::Connection::open(dir.join(ai_memory_relay::fsguard::DB_FILE)).unwrap();+ let stored: (String,String,String,String,String,Option<i64>,i64) = conn.query_row("SELECT event_id,session_id,ingest_key,body_json,body_sha256,first_attempt_ms,attempts FROM pending", [], |row| Ok((row.get(0)?,row.get(1)?,row.get(2)?,row.get(3)?,row.get(4)?,row.get(5)?,row.get(6)?))).unwrap();+ assert_eq!(+ stored,+ (+ legacy.event_id,+ legacy.session_id,+ legacy.ingest_key,+ legacy.body_json,+ legacy.body_sha256,+ None,+ 0+ )+ );+}diff --git a/docs/companion-crates.md b/docs/companion-crates.mdindex 1ffd1dcc..1c2ca4d0 100644--- a/docs/companion-crates.md+++ b/docs/companion-crates.md@@ -79,6 +79,17 @@ the public MCP write/delete tools. It must never open the wiki directory orSQLite directly. The core read seam is `/api/v1` in API-only mode; the supportedMCP page arguments are documented in [programmatic memory](programmatic-memory.md).+## `ai-memory-client`: shared private capture privacy++[`ai-memory-client`](../companions/ai-memory-client) supplies four privacy+functions consumed by both the relay and external-conversation importer. New+bodies are scrubbed before serialization, hashes and local persistence; native+identities containing credentials are refused without being renamed. Unsafe+legacy relay heads retain their exact bytes and retry metadata and defer only+their own session. The package has its own workspace and focused tests, with no+server dependency. Server authorization and capture exclusions remain the+authoritative boundaries.+## `ai-memory-relay`: external lifecycle delivery[`ai-memory-relay`](../companions/ai-memory-relay) delivers events collected bydiff --git a/docs/security-boundaries.md b/docs/security-boundaries.mdindex 53774ace..e2a8511b 100644--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@ -64,6 +64,7 @@ boundary not yet built.| 13a | Managed Codex stale-daemon run recovery (#987; concurrent launches, project/worktree isolation, operator isolation) | `managed_runs.owner_user` (V71) uses the topology-aware `IdentityKey` owner stamp; `router.rs` resolves and authorizes the SessionStart repository before handling the untrusted run id; store `link_or_adopt_native_session` selects and links in one writer transaction and admits only the sole live, undelivered, never-linked Codex run in the exact workspace/project/checkout-cwd/owner bucket. An active supplied id must itself match every boundary; another harness, an ambiguity, or a second native session is refused | `multi_session.rs` `stale_codex_run_recovery_cannot_cross_project_owner_or_session_boundaries` (foreign owner, foreign checkout, and cross-project candidates refused; project-local controls and racing rebind), `stale_codex_run_recovery_fails_closed_on_ambiguity_and_active_mismatch` (two candidates and active foreign id refused, exact control); `workstream.rs` `prepare_stamps_the_operator_bucket_used_by_session_start_recovery` (HTTP actor stamp with foreign and legitimate controls); `router.rs` `managed_session_start_delivers_ledger_and_pending_handoff` supplies a finished stale id and proves the new run is linked and delivered | STRONG for SessionStart recovery. No candidate is selected from a run id's stale scope, another checkout, or another operator bucket; ambiguity deliberately falls back to no managed packet || 13b | Managed-run lease exclusivity and explicit recovery (one active run per workstream, invariant #16) | `ai-memory-store/src/workstream.rs` `prepare_run` expires lapsed leases and otherwise refuses an `active` run inside the writer transaction (`StoreError::WorkstreamBusy`). The explicit `force_unlock` path may expire and replace that row in the same transaction only when its durable `owner_user` equals the incoming actor's attribution; different operators remain isolated. `heartbeat` renews only `active` rows. `ai-memory-cli/src/commands/run.rs` normally uses `wait_out_held_lease`; `--force-unlock` instead asks once and fails immediately when refused. Neither path treats the informational `lease_owner` label as authority | `multi_session.rs` `force_unlock_replaces_only_the_same_operators_active_run` (foreign operator refused with live control, same operator succeeds, old run loses heartbeat, unattributed single-user control); hooks `workstream.rs` `force_unlock_cannot_evict_another_operator` exercises the HTTP actor boundary; CLI `run.rs` `force_unlock_asks_once_instead_of_waiting_out_the_lease`; existing controls `managed_workstream_batches_are_idempotent_and_release_the_lease`, `a_renewed_lease_is_reported_as_a_live_owner_not_taken_over`, `interactive_launch_waits_out_a_lapsing_lease_then_proceeds`, and `ctrl_c_aborts_the_held_lease_wait_immediately` | STRONG for exclusivity. The `lease_owner` label (`host:pid`) is informational only and not unique inside ai-jail (every jailed launcher reports `ai-sandbox:<ns-pid>`), so it must never become an ownership key. A shared unattributed/root context is intentionally one operator bucket; deployments needing person-level refusal must authenticate DB users || 14 | Per-project authorization (#708) | `ai-memory-store/src/project_authz.rs` `authorize_project` / `ProjectAuthz::authorize` choke point (V68 `project_grants` + `projects.access_mode`, default `open`; V69 `projects.created_by` feeds `is_creator`); `scope.rs` `ScopeResolver::with_project_authz` (reader pool for reads, writer actor for writes) and its free forms `authorize_scope_for` / `*_guarded`, attached for every DB user by `ai-memory-mcp` `scope_resolver_as`, `ai-memory-web` `authorize_read` / `lookup_project`, and `ai-memory-hooks` (`grants.rs` `authorize_resolved` for run/workstream ids, the capture check in `router.rs`); read-shaped mutations resolve at `ProjectAccess::Write` (`resolve_existing_args(.., need)`); unscoped reads filtered before `LIMIT` by `reader.rs` `readable_repository_sql`; `WriterHandle::authorize_project` as defense in depth. Page ids are never taken from a caller (only derived from already-authorized hits), so there is no page-id entry point to guard; incremental `recent` resolves and authorizes the requested project before decoding its scope-bound cursor; batch outcomes do not bypass the capture grant or session owner check | `tests/suite/project_authz.rs` (decision matrix, ship-inert, resolver gate); `tests/suite/access_mode.rs` `every_caller_against_both_modes`, `a_restricted_project_admits_the_team_and_refuses_the_outsider`, `new_projects_follow_the_server_default_and_admit_their_creator`; `scope.rs` `the_argument_shape_does_not_decide_the_level`, `a_user_reaches_only_what_they_were_granted`, `creating_authorizes_against_a_project_that_already_exists`, `a_refused_scope_fails_the_search_instead_of_shortening_it`; `grants.rs` `search_finds_only_what_the_viewer_may_read`, `the_limit_counts_only_what_the_viewer_may_see`, `the_workspace_handoff_comes_only_from_readable_repositories`; `ai-memory-mcp` `server.rs` `a_reader_may_read_everything_and_change_nothing`, `bob_cannot_read_alices_page_in_a_restricted_project`, `bob_cannot_find_alices_page_by_searching_in_a_restricted_project`, `bob_cannot_consolidate_a_session_in_alices_repository`, `a_restricted_projects_queues_need_write`; `ai-memory-hooks` `a_capture_needs_writer_on_the_repository_it_lands_in`, `session_start_delivers_nothing_from_a_repository_the_viewer_cannot_read`, `run_and_workstream_ids_only_answer_someone_who_may_reach_the_repository`; `ai-memory-web` `web_reads_honour_grants_in_a_restricted_project`, `metadata_shows_only_what_the_viewer_may_read` — each with a granted or open-project control, and proven to fail with the choke point (18 tests) or the SQL filter (9 tests) neutralized; `routes.rs` `api_recent_incremental_rechecks_restricted_scope_on_cursor` (a valid cursor cannot bypass a revoked grant; granted control passes), `api_recent_incremental_paginates_over_120_pages_and_binds_cursor`; `mcp_stateless_http.rs` `generic_machine_client_writes_queries_and_claims_a_handoff` (real machine key, foreign scope and partial scope refused); `router.rs` `an_unauthorized_batch_item_is_consumed_rather_than_retried`, `batch_acknowledges_foreign_collision_then_commits_next_item` (refused foreign item followed by a legitimate stored item) | STRONG — slice 3 closed both bypass classes (unscoped reads, raw-id entry points) and added the root-only management surface. Out of scope by design: per-project administrators (granting/restricting is root-only), and access modes, creators and grants live only in SQLite, so `reindex` resets them |+| 14aa | Companion capture privacy before persistence and delivery | `companions/ai-memory-client/src/lib.rs` `reject_sensitive`, `sanitize_external_value`, `check_body`; `src/scrub.rs` `sanitize_external_text`; relay `identity::validate`, `check_scope`, `relay::init` and `flush_locked`; importer `plan_external_conversation`, `planned_hook_event`, `truncate_utf8`, `ImportClient::post_hook_batch` | SDK secret-position, nested key, 11-name numeric metric, bounded schema and literal-marker tests; relay `sensitive_envelope_is_rejected_before_enqueue_with_clean_control`, `new_native_bodies_are_sanitized_before_hash_enqueue_and_exact_retry`, `unsafe_legacy_pending_is_retained_exactly_and_never_sent`, `legacy_binding_scope_spaces_are_preserved_on_delivery`; importer `sensitive_native_identity_is_refused_before_manifest_and_http_with_unicode_control`, `shared_key_sanitation_precedes_plan_manifest_transport_and_exact_retry`, `preview_and_manifest_use_scrubbed_bodies_and_digest`, `forged_body_authority_is_refused_before_transport`, `sanitized_redaction_markers_survive_utf8_byte_caps_in_bodies_and_titles`. Controls preserve opaque native Unicode IDs, metric representation and untrusted provenance. Unsafe legacy bodies keep exact bytes/digest/identity/key and existing attempt state; only their session is deferred while clean sessions ship. Guard-removal runtime probes target the final formatted sources and restore identical hashes. | Client-side privacy only; server scope, auth, admission and capture-policy guards remain authoritative || 14b | Repository identity routing (#708) | `ai-memory-store/src/ops.rs` `resolve_project_by_identity` — one transaction; an identity already on a project is never overwritten; an unclaimed project is claimed only when the capturing user may write it (the choke point's `resolve_project_authz` on the same transaction), otherwise it is returned unclaimed; a different identity under the same name splits into a new project with no shared `repo_path`; `ai-memory-hooks/src/router.rs` `cache_key_for` keys the path cache by identity too; `ai-memory-core/src/repository_identity.rs` strips credentials from remote URLs client-side, and the server accepts only the routing rungs (`explicit`, `git_remote`) from the wire | `tests/suite/identity_resolution.rs` `an_outsider_cannot_take_an_unclaimed_projects_identity` (control: `an_existing_project_is_claimed_in_place`), `two_unrelated_repositories_with_one_folder_name_stay_apart`, `a_created_or_split_project_admits_its_creator`; `router.rs` `one_path_with_two_remotes_is_two_projects`, `two_api_checkouts_with_different_remotes_get_two_projects` (a manifest rung on the wire is ignored); `repository_identity.rs` credential and normalisation tables — the claim guard and the cache key each proven to fail their test when removed | STRONG — first claimant wins: in a restricted workspace a user who creates a project under a remote identity first holds it until root grants others || 14c | Client-supplied event time is bounded and self-scoped (#919) | `ai-memory-hooks/src/payload.rs` `HookEnvelope::occurred_at_micros` — a `/hook` caller's optional `occurred_at` must be `> 0` and `<= now + 5min`; it only sets the caller's *own* admitted session's `started_at`/`ended_at`/`created_at`, never another session/project/user, and the ingest dedup `seen_at`/TTL stays on `now` | `payload.rs` `occurred_at_micros_rejects_a_far_future_timestamp`, `…_rejects_non_positive_values`, `…_rejects_garbage_strings` | MEDIUM — self-scoped numeric bound; no cross-tenant surface (a client already controls its own content) || 14d | `include_related` shows and walks only readable projects (#708) | `ai-memory-store/src/reader.rs` `related_walk(.., viewer)` — both the outgoing and incoming hop queries carry `readable_repository_predicate("pg.project_id", viewer)`, the same far-end filter `page_links` applies, so a page in a project the viewer cannot read is neither returned nor expanded; `ai-memory-mcp/src/server.rs` `memory_read_page` passes `viewer_from_parts`, and the access bump only touches the filtered nodes | `related_walk.rs` `the_walk_hides_and_does_not_cross_projects_the_viewer_cannot_read` (an outgoing and an incoming neighbour in an ungranted restricted project are absent, and so is a granted page reachable only through one; control: a granted restricted project is shown, and no viewer walks the whole graph) — fails with the predicate removed; `ai-memory-mcp` `server.rs` `bob_does_not_see_restricted_pages_through_the_related_walk` (an ungranted user's `memory_read_page{include_related}` omits the restricted page; control: no viewer still reaches it) — fails with `memory_read_page` passing `None` instead of `viewer_from_parts` | STRONG |