Skip to content

Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors

Rust

fix(importer): preserve Office run separators

Samir Hanna Verza7e32d0d3c8cc
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e65cba04..c179648c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,6 +13,11 @@ version in [`docs/changelog/`](docs/changelog/):
### Fixed
+- Preserved explicit DOCX/PPTX line breaks and DOCX tabs in document imports
+ when XML uses paired tags, including formatted PowerPoint breaks in slides
+ and speaker notes. DOCX carriage returns now also keep adjacent text runs
+ on separate lines instead of joining their words. Paragraph tab-stop
+ definitions no longer insert spurious tabs into the extracted text (#PR).
- Refused empty and whitespace-only `--workspace` and `--project` flags at the
shared CLI scope-resolution choke point (`resolve_scope`,
`resolve_scope_noting_marker_project`, `resolve_scope_with_marker`, and
diff --git a/companions/ai-memory-importer/README.md b/companions/ai-memory-importer/README.md
index 2e3eb91b..436cb774 100644
--- a/companions/ai-memory-importer/README.md
+++ b/companions/ai-memory-importer/README.md
@@ -145,6 +145,11 @@ by formatting may still be extracted; deleted revisions and field instructions
are excluded. Review `--show-body` before every live import of a document whose
visibility rules matter.
+DOCX and PPTX extraction preserves explicit run-level line breaks, including
+DOCX carriage returns and formatted PPTX breaks, and DOCX tabs. Paired XML
+start/end tags and self-closing tags produce the same separators.
+Paragraph tab-stop definitions are formatting and do not insert text.
+
VTT/SRT files accept LF, CRLF, or CR line endings and any number of blank lines
between cues. WebVTT comments are omitted; identifiers such as `NOTE123` keep
their cue text.
diff --git a/companions/ai-memory-importer/src/document.rs b/companions/ai-memory-importer/src/document.rs
index 3622194b..8bb0c2ef 100644
--- a/companions/ai-memory-importer/src/document.rs
+++ b/companions/ai-memory-importer/src/document.rs
@@ -1198,8 +1198,14 @@ fn xml_text(xml: &str) -> Result<String> {
loop {
match reader.read_event().context("parse Office XML")? {
Event::Start(_) if skipped_depth > 0 => skipped_depth += 1,
+ // These tab lists define paragraph layout, not tab characters:
+ // https://learn.microsoft.com/en-us/dotnet/api/documentformat.openxml.wordprocessing.tabs
+ // https://learn.microsoft.com/en-us/dotnet/api/documentformat.openxml.drawing.tabstoplist
Event::Start(tag)
- if matches!(tag.local_name().as_ref(), "del" | "delText" | "instrText") =>
+ if matches!(
+ tag.local_name().as_ref(),
+ "del" | "delText" | "instrText" | "tabs" | "tabLst"
+ ) =>
{
skipped_depth = 1;
}
@@ -1217,11 +1223,18 @@ fn xml_text(xml: &str) -> Result<String> {
Event::GeneralRef(reference) if text_depth > 0 => {
push_bounded(&mut output, &xml_reference(&reference)?)?;
}
- Event::Empty(tag) if skipped_depth == 0 => match tag.local_name().as_ref() {
- "br" => push_bounded(&mut output, "\n")?,
- "tab" => push_bounded(&mut output, "\t")?,
- _ => {}
- },
+ // DrawingML breaks can contain run properties; empty WordprocessingML
+ // breaks and tabs may also use paired start/end tags. A `w:cr` is a
+ // line break, like `w:br`, rather than a new paragraph.
+ // https://learn.microsoft.com/en-us/dotnet/api/documentformat.openxml.drawing.break
+ // https://learn.microsoft.com/en-us/dotnet/api/documentformat.openxml.wordprocessing.carriagereturn
+ Event::Start(tag) | Event::Empty(tag) if skipped_depth == 0 => {
+ match tag.local_name().as_ref() {
+ "br" | "cr" => push_bounded(&mut output, "\n")?,
+ "tab" => push_bounded(&mut output, "\t")?,
+ _ => {}
+ }
+ }
Event::End(tag) if skipped_depth == 0 => match tag.local_name().as_ref() {
"p" | "tr" => push_bounded(&mut output, "\n")?,
"tc" => push_bounded(&mut output, "\t")?,
@@ -1581,6 +1594,30 @@ mod tests {
assert!(!plan.body.contains("field instruction"));
}
+ #[test]
+ fn docx_run_separators_are_preserved_and_sanitized() {
+ let temp = TempDir::new().unwrap();
+ let file = temp.path().join("separators.docx");
+ for line_break in ["<w:br/>", "<w:br></w:br>", "<w:cr/>", "<w:cr></w:cr>"] {
+ for tab in ["<w:tab/>", "<w:tab></w:tab>"] {
+ let document = format!(
+ r#"<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body><w:p><w:r><w:t>First line</w:t>{line_break}<w:t>Host</w:t>{tab}<w:t>8443 token=abcdefghijklmnop</w:t></w:r></w:p><w:p><w:pPr><w:tabs><w:tab w:val="left" w:pos="720"/><w:tab w:val="left" w:pos="1440"></w:tab></w:tabs></w:pPr><w:r><w:t>Kept</w:t></w:r><w:del><w:r><w:delText>DELETED_SENTINEL</w:delText>{line_break}{tab}</w:r></w:del><w:r><w:instrText>FIELD_SENTINEL</w:instrText><w:t> together</w:t></w:r></w:p></w:body></w:document>"#
+ );
+ write_zip(&file, &[("word/document.xml", &document)]);
+ let plan = plan_document(&file, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();
+ assert!(
+ plan.body
+ .contains("First line\nHost\t8443 token=[REDACTED]\nKept together"),
+ "break {line_break}, tab {tab}: {}",
+ plan.body
+ );
+ assert!(!plan.body.contains("abcdefghijklmnop"));
+ assert!(!plan.body.contains("DELETED_SENTINEL"));
+ assert!(!plan.body.contains("FIELD_SENTINEL"));
+ }
+ }
+ }
+
#[test]
fn pdf_extracts_text_by_page() {
let temp = TempDir::new().unwrap();
@@ -1723,6 +1760,64 @@ mod tests {
assert!(plan.body.contains("Say this"));
}
+ #[test]
+ fn pptx_run_separators_are_preserved_and_sanitized() {
+ let temp = TempDir::new().unwrap();
+ let file = temp.path().join("separators.pptx");
+ let tab_stops = r#"<a:tab pos="914400" algn="l"/><a:tab pos="1828800" algn="l"></a:tab>"#;
+ let list_style = format!(
+ "<a:lstStyle><a:defPPr><a:tabLst>{tab_stops}</a:tabLst></a:defPPr><a:lvl1pPr><a:tabLst>{tab_stops}</a:tabLst></a:lvl1pPr></a:lstStyle>"
+ );
+ // The first shape makes any injected tab in the next shape's paragraph
+ // properties visible instead of letting normalize_text trim it away.
+ let first_shape = "<p:sp><p:txBody><a:bodyPr/><a:lstStyle/><a:p><a:r><a:t>Heading</a:t></a:r></a:p></p:txBody></p:sp>";
+ for line_break in [
+ "<a:br/>",
+ "<a:br></a:br>",
+ r#"<a:br><a:rPr lang="en-US"/></a:br>"#,
+ ] {
+ let slide = format!(
+ r#"<p:sld xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main" xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"><p:cSld><p:spTree>{first_shape}<p:sp><p:txBody><a:bodyPr/>{list_style}<a:p><a:pPr><a:tabLst>{tab_stops}</a:tabLst></a:pPr><a:r><a:t>Use</a:t></a:r>{line_break}<a:r><a:t>TLS token=abcdefghijklmnop</a:t></a:r></a:p></p:txBody></p:sp></p:spTree></p:cSld></p:sld>"#
+ );
+ let notes = format!(
+ r#"<p:notes xmlns:p="http://schemas.openxmlformats.org/presentationml/2006/main" xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main"><p:cSld><p:spTree>{first_shape}<p:sp><p:txBody><a:bodyPr/>{list_style}<a:p><a:pPr><a:tabLst>{tab_stops}</a:tabLst></a:pPr><a:r><a:t>Say</a:t></a:r>{line_break}<a:r><a:t>this token=abcdefghijklmnop</a:t></a:r></a:p></p:txBody></p:sp></p:spTree></p:cSld></p:notes>"#
+ );
+ write_zip(
+ &file,
+ &[
+ (
+ "ppt/presentation.xml",
+ r#"<p:presentation xmlns:p="p" xmlns:r="r"><p:sldIdLst><p:sldId id="1" r:id="rId1"/></p:sldIdLst></p:presentation>"#,
+ ),
+ (
+ "ppt/_rels/presentation.xml.rels",
+ r#"<Relationships><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide" Target="slides/slide1.xml"/></Relationships>"#,
+ ),
+ ("ppt/slides/slide1.xml", &slide),
+ (
+ "ppt/slides/_rels/slide1.xml.rels",
+ r#"<Relationships><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/notesSlide" Target="../notesSlides/notesSlide1.xml"/></Relationships>"#,
+ ),
+ ("ppt/notesSlides/notesSlide1.xml", &notes),
+ ],
+ );
+ let plan = plan_document(&file, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();
+ assert!(
+ plan.body
+ .contains("## Slide 1\n\nHeading\nUse\nTLS token=[REDACTED]"),
+ "break {line_break}: {}",
+ plan.body
+ );
+ assert!(
+ plan.body
+ .contains("### Speaker notes\n\nHeading\nSay\nthis token=[REDACTED]"),
+ "break {line_break}: {}",
+ plan.body
+ );
+ assert!(!plan.body.contains("abcdefghijklmnop"));
+ }
+ }
+
#[test]
fn transcripts_drop_timing_metadata_and_markup() {
let vtt = b"WEBVTT\n\nNOTE private cue note\nskip me\n\nintro-id\n00:00:01.000 --> 00:00:02.000\n<v Alice>Hello</v> world\n\n00:00:03.000 --> 00:00:04.000\n1 < 2 and literal --> arrow\n";
diff --git a/docs/security-boundaries.md b/docs/security-boundaries.md
index 23b2d5a6..911ecae8 100644
--- a/docs/security-boundaries.md
+++ b/docs/security-boundaries.md
@@ -138,7 +138,7 @@ the legitimate restricted-project control and the failed-query cases.
| 14h | Wiki link graph pages and JSON show only readable projects (#708) | `ai-memory-web/src/routes/graph.rs` `global_handler` calls `ReaderPool::cross_project_edges(None, viewer)` so an edge needs both ends readable; `project_handler` resolves through `resolve_read_scope` (`lookup_existing_scope_guarded`, no-create, `ProjectAccess::Read`) then `project_link_graph`, which ranks and caps nodes in SQL before loading them and queries only edges whose endpoints are among those selected pages; `api.rs` `graph_handler` accepts `workspace`+`project` together (partial scope 400) and passes the same viewer filter, and `project_graph_handler` uses `lookup_project` | `routes.rs` `metadata_shows_only_what_the_viewer_may_read` (HTML `/web/graph`, `/web/w/…/graph`, `/api/v1/…/graph`, and scoped `/api/v1/graph?workspace=&project=`: bob cannot see alpha or client-work, a 403 on alpha's graph leaks neither title nor path, a scoped graph hides an edge whose far end he cannot read; carol and no-viewer controls keep the edge; partial scope 400); store `project_link_graph.rs` hides expired pages, omits cross-project and unresolved links, and verifies the 120-node SQL cap preserves high-degree nodes; `graph.rs` `graph_page_renders_hostile_titles_inert` | STRONG. Bite checks: passing no viewer to `cross_project_edges` in the HTML global handler, resolving the project graph through unguarded `lookup_existing_scope`, or removing the SQL limit fails the restricted-project or node-cap assertions |
| 14i | Decoded ordinary Markdown destinations stay inside their original project | `ai-memory-wiki/src/markdown.rs` `decode_markdown_destination_path` decodes URL path bytes once before `normalize_link_target` applies scheme, separator and project-root traversal checks; wikilinks remain literal. `ai-memory-web/src/markdown.rs` `scope_relative_link` checks the decoded path before route encoding; an encoded spelling cannot become a generated cross-scope route. | `markdown_url_paths_decode_once_before_link_resolution` (space, Unicode and literal-percent controls); `decoded_markdown_paths_cannot_escape_or_change_scope` (encoded root escape, backslash, external scheme, invalid UTF-8 and controls refused; legitimate in-root normalization passes); `routes.rs` `encoded_markdown_links_resolve_and_remain_in_the_source_scope` writes colliding targets in sibling projects and workspaces, verifies home resolution and no foreign backlinks, and follows the generated home URL; web `decoded_markdown_destinations_cannot_change_the_web_scope` (encoded traversal, root route, separator and scheme refused, encoded `w/` remains a page inside the current scope); `markdown_url_destinations_are_decoded_once_before_route_encoding` | STRONG for one-decode path handling and project confinement; URL grammar outside local Markdown destinations remains owned by the Markdown parser and existing safe-URL checks |
| 14j | Structured MCP query filters only narrow authorized page results | `ai-memory-mcp/src/server.rs` parses bounded `kind`, `tier`, literal `path_prefix` and exact `tag` values without changing the normal scope-shape or authorization paths, then passes one typed `PageFilter` through project, multi-scope, cross-project, reserved-global/profile, historical and pinned searches. `ai-memory-store/src/reader.rs` binds every filter value as a SQL parameter before each stream's candidate cut; path wildcard characters are escaped and an exact binary-prefix check preserves case, while tags match only string members of a JSON array. A filtered miss cannot fall through to raw observations, which carry no page metadata. | `server.rs` `memory_query_filters_apply_before_every_result_source` combines all filters and covers literal `%`/`_`, case-sensitive paths, tag arrays versus scalars, multi-scope and global search, `as_of`, superseded metadata, `pin_first`, the reserved-global/profile union, traversal-shaped input, invalid tiers, controls and Unicode length bounds; store `structured_filters_hold_in_vector_and_both_graph_directions` isolates path filtering in the vector stream and tag filtering in outgoing and incoming graph expansion; `page_filtering_precedes_the_observation_candidate_cut` puts 80 stronger excluded observations ahead of an eligible tagged session; `memory_query_returns_raw_hits_when_pages_miss` proves an unfiltered miss retains fallback while a filtered miss omits it; `prompts_and_tool_schema_document_query_options` pins the public schema. Existing web every-stream and authorization tests in row 14g exercise kind/tier across the same shared SQL. | STRONG: filters are candidate predicates, never an alternative scope resolver. Empty filter fields normalize to absent for backward compatibility; observation fallback remains available only to unfiltered queries. |
-| 14k | Optional document import treats local files as bounded untrusted input | `companions/ai-memory-importer/src/document.rs` accepts one explicit regular file on Unix (and fails closed elsewhere), refuses final symlinks, uses `O_NOFOLLOW` plus opened-inode verification, caps source/archive/entry/working/stored text/PDF-stream/sheet/actual-cell dimensions, streams XLSX cells without allocating their coordinate rectangle, follows PPTX relationship order, rejects unsupported or malformed formats, sanitizes extracted text before manifest or transport, derives the destination and stable provenance key from the source SHA-256, and writes through MCP `create_only` with explicit scope. A conflict is accepted as idempotent only when the current page body exactly equals the current sanitized plan; a provenance marker alone is never trusted and another page is never replaced. Shared manifests use randomized private same-directory temporary files, fsync, and atomic replacement. | Importer `document::tests`: generated bounded and over-limit PDF, visible-text DOCX, relationship-ordered PPTX, sparse-coordinate XLSX, text and cue-state VTT/SRT controls; `subtitle_cues_are_sanitized_before_the_document_plan` keeps ordinary and NOTE-prefixed cues across LF/CRLF/CR, redacts a credential in each, and excludes actual comments; `text_plan_is_redacted_bounded_attributed_and_deterministic`; `unsafe_sources_prefixes_and_malformed_archives_fail_closed`; `zip_preflight_refuses_excessive_claimed_expansion`; `apply_is_conditional_idempotent_and_refuses_foreign_pages`; shared writer tests cover repeated replacement and a pre-created legacy temp symlink. | STRONG for explicit Unix local text extraction and conditional persistence. Operator review with `--show-body` remains necessary because DOCX formatting-hidden text can be extracted. OCR, images, raw media, watch folders and server-side upload are excluded. |
+| 14k | Optional document import treats local files as bounded untrusted input | `companions/ai-memory-importer/src/document.rs` accepts one explicit regular file on Unix (and fails closed elsewhere), refuses final symlinks, uses `O_NOFOLLOW` plus opened-inode verification, caps source/archive/entry/working/stored text/PDF-stream/sheet/actual-cell dimensions, streams XLSX cells without allocating their coordinate rectangle, follows PPTX relationship order, rejects unsupported or malformed formats, sanitizes extracted text before manifest or transport, derives the destination and stable provenance key from the source SHA-256, and writes through MCP `create_only` with explicit scope. A conflict is accepted as idempotent only when the current page body exactly equals the current sanitized plan; a provenance marker alone is never trusted and another page is never replaced. Shared manifests use randomized private same-directory temporary files, fsync, and atomic replacement. | Importer `document::tests`: generated bounded and over-limit PDF, visible-text DOCX, relationship-ordered PPTX, sparse-coordinate XLSX, text and cue-state VTT/SRT controls; `docx_run_separators_are_preserved_and_sanitized` and `pptx_run_separators_are_preserved_and_sanitized` keep legitimate separators across empty/paired/formatted XML, redact credentials in document/slide/note text, exclude paragraph tab-stop definitions, and exclude DOCX deleted text, field instructions and separators in deleted runs; `subtitle_cues_are_sanitized_before_the_document_plan` keeps ordinary and NOTE-prefixed cues across LF/CRLF/CR, redacts a credential in each, and excludes actual comments; `text_plan_is_redacted_bounded_attributed_and_deterministic`; `unsafe_sources_prefixes_and_malformed_archives_fail_closed`; `zip_preflight_refuses_excessive_claimed_expansion`; `apply_is_conditional_idempotent_and_refuses_foreign_pages`; shared writer tests cover repeated replacement and a pre-created legacy temp symlink. | STRONG for explicit Unix local text extraction and conditional persistence. Operator review with `--show-body` remains necessary because DOCX formatting-hidden text can be extracted. OCR, images, raw media, watch folders and server-side upload are excluded. |
| 15 | Agent-archive restore path confinement, resource bounds, and consistency | `ai-memory-cli/src/commands/restore_agents.rs` stages only bounded regular UTF-8 entries; requires a one-to-one manifest/body/checksum/target mapping; allowlists destinations by scope; rejects symlinked descendants; treats differing portable permissions as an overwrite requiring `--apply --force`; writes each file atomically and rolls back earlier writes after a later failure. `backup_agents.rs` skips direct and directory-root source symlinks, bounds individual assets, attributes a shared skill root only once, and atomically replaces a private archive only after gzip completion | `backup_agents.rs` `restore_agents_checks_permissions_before_skipping_identical_content` (same bytes with changed mode are previewed and refused without `--force`; forced restore preserves a backup and a second run is a no-op), `restore_agents_without_archived_permissions_keeps_identical_file_mode` (missing-mode control), `restore_agents_rejects_path_traversal`, `restore_agents_rejects_unauthorized_destination_path`, `restore_agents_verifies_sha256_integrity`, `restore_agents_requires_a_complete_one_to_one_manifest`, `restore_agents_rejects_oversized_entries_from_the_header`, `restore_agents_rejects_a_symlinked_destination_parent`, `backup_agents_does_not_follow_config_or_asset_root_symlinks` (Claude and Antigravity roots), and `failed_backup_leaves_the_previous_archive_intact`; `backup_and_restore_agents_round_trip` and `antigravity_backup_round_trips_every_documented_skill_root` are the valid-path controls; `restore_agents::tests::rollback_removes_new_files_and_restores_overwrites` covers recovery | STRONG |
| 16 | Relay queue identity, migration, and private filesystem state | `companions/ai-memory-relay/src/queue.rs` `Queue::open_inner` performs read-only identity preflight, rechecks under `IMMEDIATE`, commits schema and metadata together; `configure` checks the presence of user objects and rechecks `identity`/`schema_version` metadata on the same connection before every bounded BUSY/LOCKED retry; `fsguard.rs` checks existing paths and creates owner-only files before SQLite opens them | `tests/relay.rs` `a_foreign_database_is_refused_byte_for_byte`, `unknown_schemas_refuse_without_mutation`, `a_foreign_database_with_a_writer_is_refused_before_write_coordination`, `a_foreign_identity_after_preflight_is_refused_without_mutation`, `a_foreign_identity_during_journal_retry_is_refused_without_mutation`, `journal_retry_refuses_a_changed_version_or_empty_database_without_mutation`; controls: `journal_transition_retries_sqlite_lock_without_losing_v1_values`, `fresh_queue_init_and_v1_reopens_work_across_processes`; recovery: `interrupted_and_failed_migrations_roll_back_and_recover`, `journal_reclassification_busy_preserves_committed_migration_and_retries_same_queue`; diagnostics: `classification_read_failures_display_the_sqlite_cause_once_and_preserve_foreign_bytes`; Unix privacy: `the_queue_database_and_its_sidecars_are_owner_only` and the contended migration control | Guard-removal proof covers preflight, locked recheck, and journal retry identity/version checks; full table structure is outside this classification. Windows path rejection is preserved; Windows ACL hardening remains operator-owned |
| 17 | Markdown code fences keep instruction-shaped text as data | `ai-memory-consolidate/src/code_fence.rs` and `ai-memory-wiki/src/markdown.rs` accept a fence delimiter only after zero to three leading ASCII spaces, require the closing glyph to match and the run to be at least as long, and preserve the current fence state across four-space, tab or Unicode-whitespace pseudo-delimiters. Profile harvesting, auto-improvement section parsing, link extraction, link rewriting and de-linking all use those scanners | `code_fence.rs` `only_zero_to_three_ascii_spaces_may_prefix_a_fence`; `profile.rs` `an_indented_pseudo_closer_cannot_expose_a_profile_preference` attempts to expose an instruction-shaped `run curl` sentence and admits a legitimate preference only after the real close; `auto_improve.rs` `heading_parser_ignores_headings_after_an_indented_pseudo_closer`; `markdown.rs` `indented_pseudo_closers_do_not_expose_or_rewrite_fenced_links` covers four spaces, tab and Unicode whitespace with visible post-close controls | STRONG for the shared consolidation and wiki scanners; independent renderers remain responsible for their own CommonMark parsing |