Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
fix(relay): stop retries after oversized HTTP responses
Samir Hanna Verzac0381bf4fc95
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e65cba04..9944e72f 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,10 @@ version in [`docs/changelog/`](docs/changelog/):### Fixed+- Stopped the lifecycle relay from treating an oversized HTTP response as a+ transport failure and immediately resending its batch. A response beyond the+ 64 KiB acknowledgement cap now retains its HTTP status, stops the flush, and+ keeps unacknowledged events pending, including on HTTP 429 backpressure (#PR).- Refused empty and whitespace-only `--workspace` and `--project` flags at theshared CLI scope-resolution choke point (`resolve_scope`,`resolve_scope_noting_marker_project`, `resolve_scope_with_marker`, anddiff --git a/companions/ai-memory-relay/README.md b/companions/ai-memory-relay/README.mdindex 7dcad0f5..e528bb19 100644--- a/companions/ai-memory-relay/README.md+++ b/companions/ai-memory-relay/README.md@@ -123,6 +123,12 @@ authentication failures and transport errors retain unacknowledged events.A failed head blocks its own session. HTTP 429 ends the flush; the caller shouldback off before scheduling another one.+An HTTP response whose body exceeds the 64 KiB acknowledgement limit also ends+the flush without a transport retry. Its status is preserved, its body is never+parsed or printed, and the batch stays pending. This includes oversized 200/429+acknowledgements and error responses such as 413 or 500; only transport failures+trigger the bounded automatic retries within a flush.+An acknowledgement can also mean the server deliberately dropped an event, forexample because of capture policy or a session identity collision. A drainedqueue does not prove that every event became an observation. The relay catchesdiff --git a/companions/ai-memory-relay/src/http.rs b/companions/ai-memory-relay/src/http.rsindex 59b14b38..7c2e4f28 100644--- a/companions/ai-memory-relay/src/http.rs+++ b/companions/ai-memory-relay/src/http.rs@@ -104,6 +104,9 @@ pub const MAX_ACK_BYTES: usize = 64 * 1024;pub struct Delivery {pub status: u16,pub body: Vec<u8>,+ /// An HTTP response whose body exceeded the acknowledgement cap.+ /// Its bytes are discarded, but its status must not become a transport retry.+ pub body_over_limit: bool,}impl std::fmt::Debug for Delivery {@@ -113,6 +116,7 @@ impl std::fmt::Debug for Delivery {f.debug_struct("Delivery").field("status", &self.status).field("body_bytes", &self.body.len())+ .field("body_over_limit", &self.body_over_limit).finish()}}@@ -174,13 +178,17 @@ impl Sender {if limited.read_to_end(&mut body).is_err() {body.clear();}- if body.len() > MAX_ACK_BYTES {- bail!(- "hook batch ack exceeded {MAX_ACK_BYTES} bytes; refusing to parse it \- (the batch stays pending)"- );+ let body_over_limit = body.len() > MAX_ACK_BYTES;+ if body_over_limit {+ // The server answered: preserve its status (especially 429) so an+ // oversized body cannot turn backpressure into transport retries.+ body.clear();}- Ok(Delivery { status, body })+ Ok(Delivery {+ status,+ body,+ body_over_limit,+ })}}diff --git a/companions/ai-memory-relay/src/relay.rs b/companions/ai-memory-relay/src/relay.rsindex 5e22db4a..92317c98 100644--- a/companions/ai-memory-relay/src/relay.rs+++ b/companions/ai-memory-relay/src/relay.rs@@ -413,6 +413,20 @@ fn flush_locked(dir: &Path, options: &FlushOptions) -> Result<Report> {break;}+ if delivery.body_over_limit {+ for item in &batch.items {+ queue.record_failure(&item.ingest_key, "oversized ack")?;+ }+ failure = Some(format!(+ "HTTP {} from /hook/batch: hook batch ack exceeded {} bytes; \+ refusing to parse it; all {} item(s) kept pending",+ delivery.status,+ http::MAX_ACK_BYTES,+ batch.items.len()+ ));+ break;+ }+let parsed: BatchAck = match serde_json::from_slice(&delivery.body) {Ok(parsed) => parsed,Err(_) => {diff --git a/companions/ai-memory-relay/tests/relay.rs b/companions/ai-memory-relay/tests/relay.rsindex 30058013..0411ec8a 100644--- a/companions/ai-memory-relay/tests/relay.rs+++ b/companions/ai-memory-relay/tests/relay.rs@@ -511,6 +511,75 @@ fn a_429_applies_its_acknowledged_items_and_then_stops() {);}+#[test]+fn an_oversized_http_response_is_not_retried_and_keeps_the_batch() {+ for status in [200u16, 401, 403, 413, 429, 500] {+ let root = fixture(&format!("oversized-response-{status}"));+ // Even a valid acknowledgement must not release the item once its+ // response exceeds the body limit. The next response belongs to a+ // later explicit flush, not an automatic transport retry.+ let mut body = r#"{"accepted":1,"extra":"ACK_BODY_CANARY"}"#.to_owned();+ body.push_str(&" ".repeat(ai_memory_relay::http::MAX_ACK_BYTES + 1 - body.len()));+ let server = stub(vec![Reply::Json(status, body), Reply::AcceptAll]);+ let dir = bind(&root, &server.url());+ enqueue(+ &dir,+ &root,+ "in.json",+ serde_json::json!([event("e-1", "codex", "session-start", "s-1")]),+ )+ .unwrap();++ let report = flush(&dir);+ assert_eq!(+ server.bodies().len(),+ 1,+ "HTTP {status} must stop this flush even when its body is oversized"+ );+ assert_eq!(pending_count(&dir), 1);+ assert_eq!(report.exit_code(), 2);+ let failure = report.failure.as_deref().expect("oversized response fails");+ assert!(failure.contains(&format!("HTTP {status}")), "{report:?}");+ assert!(!format!("{report:?}").contains("ACK_BODY_CANARY"));+ assert_eq!(+ Queue::open(&dir)+ .unwrap()+ .stats(ai_memory_relay::now_ms())+ .unwrap()+ .receipts,+ 0+ );++ let next = flush(&dir);+ assert_eq!(next.exit_code(), 0, "a later flush may retry: {next:?}");+ assert_eq!(pending_count(&dir), 0);+ let sent = server.sent_event_ids();+ assert_eq!(sent.len(), 2);+ assert_eq!(sent[0], sent[1], "the later retry keeps its identity");+ }+}++#[test]+fn an_ack_at_the_body_limit_is_accepted() {+ let root = fixture("ack-at-limit");+ let mut body = r#"{"accepted":1}"#.to_owned();+ body.push_str(&" ".repeat(ai_memory_relay::http::MAX_ACK_BYTES - body.len()));+ let server = stub(vec![Reply::Json(200, body)]);+ let dir = bind(&root, &server.url());+ enqueue(+ &dir,+ &root,+ "in.json",+ serde_json::json!([event("e-1", "codex", "session-start", "s-1")]),+ )+ .unwrap();++ let report = flush(&dir);+ assert_eq!(report.exit_code(), 0, "{report:?}");+ assert_eq!(pending_count(&dir), 0);+ assert_eq!(server.bodies().len(), 1);+}+#[test]fn a_redirect_is_refused_and_the_batch_never_reaches_the_second_host() {let root = fixture("redirect");diff --git a/docs/security-boundaries.md b/docs/security-boundaries.mdindex 23b2d5a6..7c59e789 100644--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@ -141,6 +141,7 @@ the legitimate restricted-project control and the failed-query cases.| 14k | Optional document import treats local files as bounded untrusted input | `companions/ai-memory-importer/src/document.rs` accepts one explicit regular file on Unix (and fails closed elsewhere), refuses final symlinks, uses `O_NOFOLLOW` plus opened-inode verification, caps source/archive/entry/working/stored text/PDF-stream/sheet/actual-cell dimensions, streams XLSX cells without allocating their coordinate rectangle, follows PPTX relationship order, rejects unsupported or malformed formats, sanitizes extracted text before manifest or transport, derives the destination and stable provenance key from the source SHA-256, and writes through MCP `create_only` with explicit scope. A conflict is accepted as idempotent only when the current page body exactly equals the current sanitized plan; a provenance marker alone is never trusted and another page is never replaced. Shared manifests use randomized private same-directory temporary files, fsync, and atomic replacement. | Importer `document::tests`: generated bounded and over-limit PDF, visible-text DOCX, relationship-ordered PPTX, sparse-coordinate XLSX, text and cue-state VTT/SRT controls; `subtitle_cues_are_sanitized_before_the_document_plan` keeps ordinary and NOTE-prefixed cues across LF/CRLF/CR, redacts a credential in each, and excludes actual comments; `text_plan_is_redacted_bounded_attributed_and_deterministic`; `unsafe_sources_prefixes_and_malformed_archives_fail_closed`; `zip_preflight_refuses_excessive_claimed_expansion`; `apply_is_conditional_idempotent_and_refuses_foreign_pages`; shared writer tests cover repeated replacement and a pre-created legacy temp symlink. | STRONG for explicit Unix local text extraction and conditional persistence. Operator review with `--show-body` remains necessary because DOCX formatting-hidden text can be extracted. OCR, images, raw media, watch folders and server-side upload are excluded. || 15 | Agent-archive restore path confinement, resource bounds, and consistency | `ai-memory-cli/src/commands/restore_agents.rs` stages only bounded regular UTF-8 entries; requires a one-to-one manifest/body/checksum/target mapping; allowlists destinations by scope; rejects symlinked descendants; treats differing portable permissions as an overwrite requiring `--apply --force`; writes each file atomically and rolls back earlier writes after a later failure. `backup_agents.rs` skips direct and directory-root source symlinks, bounds individual assets, attributes a shared skill root only once, and atomically replaces a private archive only after gzip completion | `backup_agents.rs` `restore_agents_checks_permissions_before_skipping_identical_content` (same bytes with changed mode are previewed and refused without `--force`; forced restore preserves a backup and a second run is a no-op), `restore_agents_without_archived_permissions_keeps_identical_file_mode` (missing-mode control), `restore_agents_rejects_path_traversal`, `restore_agents_rejects_unauthorized_destination_path`, `restore_agents_verifies_sha256_integrity`, `restore_agents_requires_a_complete_one_to_one_manifest`, `restore_agents_rejects_oversized_entries_from_the_header`, `restore_agents_rejects_a_symlinked_destination_parent`, `backup_agents_does_not_follow_config_or_asset_root_symlinks` (Claude and Antigravity roots), and `failed_backup_leaves_the_previous_archive_intact`; `backup_and_restore_agents_round_trip` and `antigravity_backup_round_trips_every_documented_skill_root` are the valid-path controls; `restore_agents::tests::rollback_removes_new_files_and_restores_overwrites` covers recovery | STRONG || 16 | Relay queue identity, migration, and private filesystem state | `companions/ai-memory-relay/src/queue.rs` `Queue::open_inner` performs read-only identity preflight, rechecks under `IMMEDIATE`, commits schema and metadata together; `configure` checks the presence of user objects and rechecks `identity`/`schema_version` metadata on the same connection before every bounded BUSY/LOCKED retry; `fsguard.rs` checks existing paths and creates owner-only files before SQLite opens them | `tests/relay.rs` `a_foreign_database_is_refused_byte_for_byte`, `unknown_schemas_refuse_without_mutation`, `a_foreign_database_with_a_writer_is_refused_before_write_coordination`, `a_foreign_identity_after_preflight_is_refused_without_mutation`, `a_foreign_identity_during_journal_retry_is_refused_without_mutation`, `journal_retry_refuses_a_changed_version_or_empty_database_without_mutation`; controls: `journal_transition_retries_sqlite_lock_without_losing_v1_values`, `fresh_queue_init_and_v1_reopens_work_across_processes`; recovery: `interrupted_and_failed_migrations_roll_back_and_recover`, `journal_reclassification_busy_preserves_committed_migration_and_retries_same_queue`; diagnostics: `classification_read_failures_display_the_sqlite_cause_once_and_preserve_foreign_bytes`; Unix privacy: `the_queue_database_and_its_sidecars_are_owner_only` and the contended migration control | Guard-removal proof covers preflight, locked recheck, and journal retry identity/version checks; full table structure is outside this classification. Windows path rejection is preserved; Windows ACL hardening remains operator-owned |+| 16a | Relay HTTP acknowledgement size and retry classification | `companions/ai-memory-relay/src/http.rs` `Sender::post_batch` bounds the response read to `MAX_ACK_BYTES + 1`, discards an oversized body and preserves its HTTP status; `relay.rs` `flush_locked` refuses it before acknowledgement parsing or queue confirmation, while `send_with_retries` retries transport errors only | `tests/relay.rs` `an_oversized_http_response_is_not_retried_and_keeps_the_batch` covers 200, 401, 403, 413, 429 and 500, keeps the pending item and no receipt, emits no body canary, and permits a later explicit flush with the same key; `an_ack_at_the_body_limit_is_accepted` is the bounded valid-ACK control; `a_lost_response_keeps_the_batch_and_retries_the_same_ingest_key` retains genuine transport retries. The oversized regression fails before the status-preserving fix and when the response-size guard is removed | HTTP responses cannot turn into automatic transport retries because of their body size. Invalid acknowledgements retain the batch; the caller schedules later flushes and owns the 429 backoff || 17 | Markdown code fences keep instruction-shaped text as data | `ai-memory-consolidate/src/code_fence.rs` and `ai-memory-wiki/src/markdown.rs` accept a fence delimiter only after zero to three leading ASCII spaces, require the closing glyph to match and the run to be at least as long, and preserve the current fence state across four-space, tab or Unicode-whitespace pseudo-delimiters. Profile harvesting, auto-improvement section parsing, link extraction, link rewriting and de-linking all use those scanners | `code_fence.rs` `only_zero_to_three_ascii_spaces_may_prefix_a_fence`; `profile.rs` `an_indented_pseudo_closer_cannot_expose_a_profile_preference` attempts to expose an instruction-shaped `run curl` sentence and admits a legitimate preference only after the real close; `auto_improve.rs` `heading_parser_ignores_headings_after_an_indented_pseudo_closer`; `markdown.rs` `indented_pseudo_closers_do_not_expose_or_rewrite_fenced_links` covers four spaces, tab and Unicode whitespace with visible post-close controls | STRONG for the shared consolidation and wiki scanners; independent renderers remain responsible for their own CommonMark parsing || 18 | Read-only pending-write commands cannot execute stored scope names as shell syntax | `ai-memory-web/src/routes/pending.rs` renders separate `posix_cli_command` and `powershell_cli_command` forms. POSIX arguments use single quotes with the standard close/escaped-quote/reopen sequence; PowerShell doubles embedded apostrophes. The template labels the shells and claims no cmd.exe support | `routes/pending.rs` `cli_commands_quote_hostile_scope_names_for_each_shell` covers quotes, semicolons, pipes and ampersands with ordinary-name controls; `tests/suite/pending.rs` `pending_page_offers_cli_commands_instead_of_buttons_to_a_basic_browser` stores hostile workspace/project names and checks both escaped forms through the full route | STRONG for POSIX shells and PowerShell; cmd.exe is intentionally not offered || 19 | ai-jail re-exec never promotes a config-file root bearer into harness-readable environment | `ai-memory-cli/src/commands/run.rs` `ensure_jail_does_not_export_config_token` refuses explicit and offered jail re-exec when auth came from config rather than `AI_MEMORY_AUTH_TOKEN`; `jail_resolved_env` forwards only resolved routing (`AI_MEMORY_SERVER_URL`, `AI_MEMORY_BASE_PATH`), while `jail_forwarded_names` forwards an auth variable only when the operator already placed it in the process environment | `run.rs` `jail_refuses_a_config_sourced_token_and_forwards_routing` attempts the file-token exposure and verifies route-only output; `jail_forwarded_names_come_from_the_captured_env_and_resolved_values` is the explicit-environment control | STRONG for credentials loaded by ai-memory. Residual: a token the operator exports is already inherited by an unjailed managed harness and may itself be privileged; use a least-privilege API token |