Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
Merge remote-tracking branch 'origin/pr/1799' into release/3.2.10
AkitaOnRails46ebb0bbb516
diff --cc CHANGELOG.mdindex 53ab951a,c41a0029..d55f6a4c--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -13,47 -13,10 +13,51 @@@ version in [`docs/changelog/`](docs/cha### Fixed+- Sorted recent page lists by timestamp rather than RFC3339 text, so updates+ with different fractional-second precision appear in chronological order+ in the legacy recent API, project activity, folder previews and namespace+ listings. Kept title-sort and equal-timestamp path tie-breaks (#1247).++- Fixed `restore` replacing live data when a backup without `config.toml`+ contains API keys that the configured pepper cannot verify. It now refuses+ before the swap, preserving the live wiki, database and config, and names+ the original configuration as the remedy (#1495).++- Preserved explicit DOCX/PPTX line breaks and DOCX tabs in document imports+ when XML uses paired tags, including formatted PowerPoint breaks in slides+ and speaker notes. DOCX carriage returns now also keep adjacent text runs+ on separate lines instead of joining their words. Paragraph tab-stop+ definitions no longer insert spurious tabs into the extracted text (#1786).++- Kept a wiki-sync server page when its repository file was recreated after+ the sync planned a deletion. `sync --propagate-deletes --apply` now rechecks+ local absence before sending the delete, preserves the page's baseline and+ reports the changed file for the next sync, including with `--prefer repo`+ (#1787).+- Transcript imports (`backfill`, the `run` ledger, `recover`) now drop Codex+ memory-writing agent runs by the same signals the live hook uses: the+ checkout cwd, an event's recorded cwd, or a Codex user prompt that opens+ with `## Memory Writing Agent`. A custom `CODEX_HOME` whose last path+ component is not `.codex` previously reimported those sessions (#1790).+- Admission webhook HTTP requests now follow redirects only within the+ configured `scheme://host:port` origin (up to 10 hops) and fail on+ cross-origin redirects without forwarding page payloads or applying foreign+ response mutations, surfacing the redirect refusal cause in logs and+ `reject`-policy errors (#1794).+- `ai-memory project access` and `ai-memory project grants` now refuse empty+ or whitespace-only `--workspace`, `--project`, and (for access) `--mode`+ values before sending any request to the server. These commands take+ required names and never call `resolve_scope`, so the #1774 choke point+ did not cover them (#1791).+- `ai-memory user grant` and `ai-memory user revoke` now refuse empty or+ whitespace-only `--user`, `--workspace`, `--project`, and (for grant)+ `--level` values before sending any request to the server. These commands+ take required names and never call `resolve_scope`, so the #1774 choke+ point did not cover them (#1792).+ - Stopped the lifecycle relay from treating an oversized HTTP response as a+ transport failure and immediately resending its batch. A response beyond the+ 64 KiB acknowledgement cap now retains its HTTP status, stops the flush, and+ keeps unacknowledged events pending, including on HTTP 429 backpressure (#1799).- Refused empty and whitespace-only `--workspace` and `--project` flags at theshared CLI scope-resolution choke point (`resolve_scope`,`resolve_scope_noting_marker_project`, `resolve_scope_with_marker`, anddiff --cc docs/security-boundaries.mdindex 496c48e3,7c59e789..aec51265--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@@ -138,9 -138,10 +138,10 @@@ the legitimate restricted-project contr| 14h | Wiki link graph pages and JSON show only readable projects (#708) | `ai-memory-web/src/routes/graph.rs` `global_handler` calls `ReaderPool::cross_project_edges(None, viewer)` so an edge needs both ends readable; `project_handler` resolves through `resolve_read_scope` (`lookup_existing_scope_guarded`, no-create, `ProjectAccess::Read`) then `project_link_graph`, which ranks and caps nodes in SQL before loading them and queries only edges whose endpoints are among those selected pages; `api.rs` `graph_handler` accepts `workspace`+`project` together (partial scope 400) and passes the same viewer filter, and `project_graph_handler` uses `lookup_project` | `routes.rs` `metadata_shows_only_what_the_viewer_may_read` (HTML `/web/graph`, `/web/w/…/graph`, `/api/v1/…/graph`, and scoped `/api/v1/graph?workspace=&project=`: bob cannot see alpha or client-work, a 403 on alpha's graph leaks neither title nor path, a scoped graph hides an edge whose far end he cannot read; carol and no-viewer controls keep the edge; partial scope 400); store `project_link_graph.rs` hides expired pages, omits cross-project and unresolved links, and verifies the 120-node SQL cap preserves high-degree nodes; `graph.rs` `graph_page_renders_hostile_titles_inert` | STRONG. Bite checks: passing no viewer to `cross_project_edges` in the HTML global handler, resolving the project graph through unguarded `lookup_existing_scope`, or removing the SQL limit fails the restricted-project or node-cap assertions || 14i | Decoded ordinary Markdown destinations stay inside their original project | `ai-memory-wiki/src/markdown.rs` `decode_markdown_destination_path` decodes URL path bytes once before `normalize_link_target` applies scheme, separator and project-root traversal checks; wikilinks remain literal. `ai-memory-web/src/markdown.rs` `scope_relative_link` checks the decoded path before route encoding; an encoded spelling cannot become a generated cross-scope route. | `markdown_url_paths_decode_once_before_link_resolution` (space, Unicode and literal-percent controls); `decoded_markdown_paths_cannot_escape_or_change_scope` (encoded root escape, backslash, external scheme, invalid UTF-8 and controls refused; legitimate in-root normalization passes); `routes.rs` `encoded_markdown_links_resolve_and_remain_in_the_source_scope` writes colliding targets in sibling projects and workspaces, verifies home resolution and no foreign backlinks, and follows the generated home URL; web `decoded_markdown_destinations_cannot_change_the_web_scope` (encoded traversal, root route, separator and scheme refused, encoded `w/` remains a page inside the current scope); `markdown_url_destinations_are_decoded_once_before_route_encoding` | STRONG for one-decode path handling and project confinement; URL grammar outside local Markdown destinations remains owned by the Markdown parser and existing safe-URL checks || 14j | Structured MCP query filters only narrow authorized page results | `ai-memory-mcp/src/server.rs` parses bounded `kind`, `tier`, literal `path_prefix` and exact `tag` values without changing the normal scope-shape or authorization paths, then passes one typed `PageFilter` through project, multi-scope, cross-project, reserved-global/profile, historical and pinned searches. `ai-memory-store/src/reader.rs` binds every filter value as a SQL parameter before each stream's candidate cut; path wildcard characters are escaped and an exact binary-prefix check preserves case, while tags match only string members of a JSON array. A filtered miss cannot fall through to raw observations, which carry no page metadata. | `server.rs` `memory_query_filters_apply_before_every_result_source` combines all filters and covers literal `%`/`_`, case-sensitive paths, tag arrays versus scalars, multi-scope and global search, `as_of`, superseded metadata, `pin_first`, the reserved-global/profile union, traversal-shaped input, invalid tiers, controls and Unicode length bounds; store `structured_filters_hold_in_vector_and_both_graph_directions` isolates path filtering in the vector stream and tag filtering in outgoing and incoming graph expansion; `page_filtering_precedes_the_observation_candidate_cut` puts 80 stronger excluded observations ahead of an eligible tagged session; `memory_query_returns_raw_hits_when_pages_miss` proves an unfiltered miss retains fallback while a filtered miss omits it; `prompts_and_tool_schema_document_query_options` pins the public schema. Existing web every-stream and authorization tests in row 14g exercise kind/tier across the same shared SQL. | STRONG: filters are candidate predicates, never an alternative scope resolver. Empty filter fields normalize to absent for backward compatibility; observation fallback remains available only to unfiltered queries. |-| 14k | Optional document import treats local files as bounded untrusted input | `companions/ai-memory-importer/src/document.rs` accepts one explicit regular file on Unix (and fails closed elsewhere), refuses final symlinks, uses `O_NOFOLLOW` plus opened-inode verification, caps source/archive/entry/working/stored text/PDF-stream/sheet/actual-cell dimensions, streams XLSX cells without allocating their coordinate rectangle, follows PPTX relationship order, rejects unsupported or malformed formats, sanitizes extracted text before manifest or transport, derives the destination and stable provenance key from the source SHA-256, and writes through MCP `create_only` with explicit scope. A conflict is accepted as idempotent only when the current page body exactly equals the current sanitized plan; a provenance marker alone is never trusted and another page is never replaced. Shared manifests use randomized private same-directory temporary files, fsync, and atomic replacement. | Importer `document::tests`: generated bounded and over-limit PDF, visible-text DOCX, relationship-ordered PPTX, sparse-coordinate XLSX, text and cue-state VTT/SRT controls; `subtitle_cues_are_sanitized_before_the_document_plan` keeps ordinary and NOTE-prefixed cues across LF/CRLF/CR, redacts a credential in each, and excludes actual comments; `text_plan_is_redacted_bounded_attributed_and_deterministic`; `unsafe_sources_prefixes_and_malformed_archives_fail_closed`; `zip_preflight_refuses_excessive_claimed_expansion`; `apply_is_conditional_idempotent_and_refuses_foreign_pages`; shared writer tests cover repeated replacement and a pre-created legacy temp symlink. | STRONG for explicit Unix local text extraction and conditional persistence. Operator review with `--show-body` remains necessary because DOCX formatting-hidden text can be extracted. OCR, images, raw media, watch folders and server-side upload are excluded. |+| 14k | Optional document import treats local files as bounded untrusted input | `companions/ai-memory-importer/src/document.rs` accepts one explicit regular file on Unix (and fails closed elsewhere), refuses final symlinks, uses `O_NOFOLLOW` plus opened-inode verification, caps source/archive/entry/working/stored text/PDF-stream/sheet/actual-cell dimensions, streams XLSX cells without allocating their coordinate rectangle, follows PPTX relationship order, rejects unsupported or malformed formats, sanitizes extracted text before manifest or transport, derives the destination and stable provenance key from the source SHA-256, and writes through MCP `create_only` with explicit scope. A conflict is accepted as idempotent only when the current page body exactly equals the current sanitized plan; a provenance marker alone is never trusted and another page is never replaced. Shared manifests use randomized private same-directory temporary files, fsync, and atomic replacement. | Importer `document::tests`: generated bounded and over-limit PDF, visible-text DOCX, relationship-ordered PPTX, sparse-coordinate XLSX, text and cue-state VTT/SRT controls; `docx_run_separators_are_preserved_and_sanitized` and `pptx_run_separators_are_preserved_and_sanitized` keep legitimate separators across empty/paired/formatted XML, redact credentials in document/slide/note text, exclude paragraph tab-stop definitions, and exclude DOCX deleted text, field instructions and separators in deleted runs; `subtitle_cues_are_sanitized_before_the_document_plan` keeps ordinary and NOTE-prefixed cues across LF/CRLF/CR, redacts a credential in each, and excludes actual comments; `text_plan_is_redacted_bounded_attributed_and_deterministic`; `unsafe_sources_prefixes_and_malformed_archives_fail_closed`; `zip_preflight_refuses_excessive_claimed_expansion`; `apply_is_conditional_idempotent_and_refuses_foreign_pages`; shared writer tests cover repeated replacement and a pre-created legacy temp symlink. | STRONG for explicit Unix local text extraction and conditional persistence. Operator review with `--show-body` remains necessary because DOCX formatting-hidden text can be extracted. OCR, images, raw media, watch folders and server-side upload are excluded. || 15 | Agent-archive restore path confinement, resource bounds, and consistency | `ai-memory-cli/src/commands/restore_agents.rs` stages only bounded regular UTF-8 entries; requires a one-to-one manifest/body/checksum/target mapping; allowlists destinations by scope; rejects symlinked descendants; treats differing portable permissions as an overwrite requiring `--apply --force`; writes each file atomically and rolls back earlier writes after a later failure. `backup_agents.rs` skips direct and directory-root source symlinks, bounds individual assets, attributes a shared skill root only once, and atomically replaces a private archive only after gzip completion | `backup_agents.rs` `restore_agents_checks_permissions_before_skipping_identical_content` (same bytes with changed mode are previewed and refused without `--force`; forced restore preserves a backup and a second run is a no-op), `restore_agents_without_archived_permissions_keeps_identical_file_mode` (missing-mode control), `restore_agents_rejects_path_traversal`, `restore_agents_rejects_unauthorized_destination_path`, `restore_agents_verifies_sha256_integrity`, `restore_agents_requires_a_complete_one_to_one_manifest`, `restore_agents_rejects_oversized_entries_from_the_header`, `restore_agents_rejects_a_symlinked_destination_parent`, `backup_agents_does_not_follow_config_or_asset_root_symlinks` (Claude and Antigravity roots), and `failed_backup_leaves_the_previous_archive_intact`; `backup_and_restore_agents_round_trip` and `antigravity_backup_round_trips_every_documented_skill_root` are the valid-path controls; `restore_agents::tests::rollback_removes_new_files_and_restores_overwrites` covers recovery | STRONG || 16 | Relay queue identity, migration, and private filesystem state | `companions/ai-memory-relay/src/queue.rs` `Queue::open_inner` performs read-only identity preflight, rechecks under `IMMEDIATE`, commits schema and metadata together; `configure` checks the presence of user objects and rechecks `identity`/`schema_version` metadata on the same connection before every bounded BUSY/LOCKED retry; `fsguard.rs` checks existing paths and creates owner-only files before SQLite opens them | `tests/relay.rs` `a_foreign_database_is_refused_byte_for_byte`, `unknown_schemas_refuse_without_mutation`, `a_foreign_database_with_a_writer_is_refused_before_write_coordination`, `a_foreign_identity_after_preflight_is_refused_without_mutation`, `a_foreign_identity_during_journal_retry_is_refused_without_mutation`, `journal_retry_refuses_a_changed_version_or_empty_database_without_mutation`; controls: `journal_transition_retries_sqlite_lock_without_losing_v1_values`, `fresh_queue_init_and_v1_reopens_work_across_processes`; recovery: `interrupted_and_failed_migrations_roll_back_and_recover`, `journal_reclassification_busy_preserves_committed_migration_and_retries_same_queue`; diagnostics: `classification_read_failures_display_the_sqlite_cause_once_and_preserve_foreign_bytes`; Unix privacy: `the_queue_database_and_its_sidecars_are_owner_only` and the contended migration control | Guard-removal proof covers preflight, locked recheck, and journal retry identity/version checks; full table structure is outside this classification. Windows path rejection is preserved; Windows ACL hardening remains operator-owned |+ | 16a | Relay HTTP acknowledgement size and retry classification | `companions/ai-memory-relay/src/http.rs` `Sender::post_batch` bounds the response read to `MAX_ACK_BYTES + 1`, discards an oversized body and preserves its HTTP status; `relay.rs` `flush_locked` refuses it before acknowledgement parsing or queue confirmation, while `send_with_retries` retries transport errors only | `tests/relay.rs` `an_oversized_http_response_is_not_retried_and_keeps_the_batch` covers 200, 401, 403, 413, 429 and 500, keeps the pending item and no receipt, emits no body canary, and permits a later explicit flush with the same key; `an_ack_at_the_body_limit_is_accepted` is the bounded valid-ACK control; `a_lost_response_keeps_the_batch_and_retries_the_same_ingest_key` retains genuine transport retries. The oversized regression fails before the status-preserving fix and when the response-size guard is removed | HTTP responses cannot turn into automatic transport retries because of their body size. Invalid acknowledgements retain the batch; the caller schedules later flushes and owns the 429 backoff || 17 | Markdown code fences keep instruction-shaped text as data | `ai-memory-consolidate/src/code_fence.rs` and `ai-memory-wiki/src/markdown.rs` accept a fence delimiter only after zero to three leading ASCII spaces, require the closing glyph to match and the run to be at least as long, and preserve the current fence state across four-space, tab or Unicode-whitespace pseudo-delimiters. Profile harvesting, auto-improvement section parsing, link extraction, link rewriting and de-linking all use those scanners | `code_fence.rs` `only_zero_to_three_ascii_spaces_may_prefix_a_fence`; `profile.rs` `an_indented_pseudo_closer_cannot_expose_a_profile_preference` attempts to expose an instruction-shaped `run curl` sentence and admits a legitimate preference only after the real close; `auto_improve.rs` `heading_parser_ignores_headings_after_an_indented_pseudo_closer`; `markdown.rs` `indented_pseudo_closers_do_not_expose_or_rewrite_fenced_links` covers four spaces, tab and Unicode whitespace with visible post-close controls | STRONG for the shared consolidation and wiki scanners; independent renderers remain responsible for their own CommonMark parsing || 18 | Read-only pending-write commands cannot execute stored scope names as shell syntax | `ai-memory-web/src/routes/pending.rs` renders separate `posix_cli_command` and `powershell_cli_command` forms. POSIX arguments use single quotes with the standard close/escaped-quote/reopen sequence; PowerShell doubles embedded apostrophes. The template labels the shells and claims no cmd.exe support | `routes/pending.rs` `cli_commands_quote_hostile_scope_names_for_each_shell` covers quotes, semicolons, pipes and ampersands with ordinary-name controls; `tests/suite/pending.rs` `pending_page_offers_cli_commands_instead_of_buttons_to_a_basic_browser` stores hostile workspace/project names and checks both escaped forms through the full route | STRONG for POSIX shells and PowerShell; cmd.exe is intentionally not offered || 19 | ai-jail re-exec never promotes a config-file root bearer into harness-readable environment | `ai-memory-cli/src/commands/run.rs` `ensure_jail_does_not_export_config_token` refuses explicit and offered jail re-exec when auth came from config rather than `AI_MEMORY_AUTH_TOKEN`; `jail_resolved_env` forwards only resolved routing (`AI_MEMORY_SERVER_URL`, `AI_MEMORY_BASE_PATH`), while `jail_forwarded_names` forwards an auth variable only when the operator already placed it in the process environment | `run.rs` `jail_refuses_a_config_sourced_token_and_forwards_routing` attempts the file-token exposure and verifies route-only output; `jail_forwarded_names_come_from_the_captured_env_and_resolved_values` is the explicit-environment control | STRONG for credentials loaded by ai-memory. Residual: a token the operator exports is already inherited by an unjailed managed harness and may itself be privileged; use a least-privilege API token |