Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
feat(importer): add bounded document extraction
AkitaOnRails4cce726497d9
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bcaba705..dadedae1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -13,6 +13,14 @@ version in [`docs/changelog/`](docs/changelog/):### Added+- Added bounded local document extraction to the optional+ `ai-memory-importer` companion. Its dry-run-first `document` command accepts+ UTF-8 text/Markdown, text-layer PDF, DOCX, PPTX, XLSX, VTT, and+ SRT; emits attributed, client-sanitized markdown; caps source, archive,+ extracted text, PDF streams, sheets, and actual XLSX cells; and uses deterministic digest paths plus+ conditional MCP writes for idempotent reruns without overwriting an existing+ foreign page. OCR, images, and raw audio/video remain explicitly out of+ scope.- Added passive per-workload LLM telemetry to `ai-memory status --json` and`GET /admin/status`. The closed workload set covers consolidation, reranking,auto-improvement, bootstrap, query answers, lint, profile maintenance, dreamdiff --git a/README.md b/README.mdindex 9bef3009..d56a9232 100644--- a/README.md+++ b/README.md@@ -588,6 +588,9 @@ prompts, tool calls, and session boundaries; session end turns them intoreadable wiki pages; the next session starts with a handoff.- Ask "where did we leave off?" to continue from the pending handoff.+- Import a local PDF, DOCX, PPTX, XLSX, Markdown file, or VTT/SRT transcript as+ attributed, sanitized memory with the optional+ [`ai-memory-importer`](companions/ai-memory-importer) `document` command.- Ask "have we discussed X?" or "search memory for Y" to query the wiki.- Narrow a large wiki before ranking with `memory_query` filters such as`path_prefix: "procedures/"`, `tag: "rust"`, `kind: "decision"`, ordiff --git a/companions/ai-memory-importer/Cargo.lock b/companions/ai-memory-importer/Cargo.lockindex 7e67abd9..c3404b75 100644--- a/companions/ai-memory-importer/Cargo.lock+++ b/companions/ai-memory-importer/Cargo.lock@@ -2,6 +2,23 @@# It is not intended for manual editing.version = 4+[[package]]+name = "adler2"+version = "2.0.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"++[[package]]+name = "aes"+version = "0.9.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "35f0f96ce78e38c3dc6d8948aa8163d06385be74000f3c7a95bf1eef35d3ea32"+dependencies = [+ "cipher",+ "cpubits",+ "cpufeatures 0.3.1",+]+[[package]]name = "aho-corasick"version = "1.1.5"@@ -27,16 +44,35 @@ dependencies = ["ai-memory-client","anyhow","clap",+ "libc",+ "lopdf",+ "quick-xml","reqwest","serde","serde-saphyr","serde_json",- "sha2",+ "sha2 0.10.9","tempfile","tokio","url","uuid","walkdir",+ "zip",+]++[[package]]+name = "alloc-no-stdlib"+version = "2.0.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3"++[[package]]+name = "alloc-stdlib"+version = "0.2.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195"+dependencies = [+ "alloc-no-stdlib",][[package]]@@ -145,6 +181,34 @@ dependencies = ["generic-array",]+[[package]]+name = "block-buffer"+version = "0.12.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa"+dependencies = [+ "hybrid-array",+]++[[package]]+name = "block-padding"+version = "0.4.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "710f1dd022ef4e93f8a438b4ba958de7f64308434fa6a87104481645cc30068b"+dependencies = [+ "hybrid-array",+]++[[package]]+name = "brotli-decompressor"+version = "5.0.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583"+dependencies = [+ "alloc-no-stdlib",+ "alloc-stdlib",+]+[[package]]name = "bumpalo"version = "3.20.3"@@ -157,6 +221,15 @@ version = "1.12.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "8ae3f5d315924270530207e2a68396c3cc547f6dca3fbdca317cfb1a51edb593"+[[package]]+name = "cbc"+version = "0.2.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ce2dc9ee5f88d11e0beb842c88b33c8a5cf0d1329c4b19494af42b07dbfe8896"+dependencies = [+ "cipher",+]+[[package]]name = "cc"version = "1.2.65"@@ -190,6 +263,16 @@ dependencies = ["rand_core",]+[[package]]+name = "cipher"+version = "0.5.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e8cf2a2c93cd704877c0858356ed03480ff301ee950b43f1cbe4573b088bfa6c"+dependencies = [+ "crypto-common 0.2.2",+ "inout",+]+[[package]]name = "clap"version = "4.6.1"@@ -236,6 +319,12 @@ version = "1.0.5"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570"+[[package]]+name = "const-oid"+version = "0.10.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"+[[package]]name = "core-foundation"version = "0.10.1"@@ -258,6 +347,12 @@ version = "1.0.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "7f8f80099a98041a3d1622845c271458a2d73e688351bf3cb999266764b81d48"+[[package]]+name = "cpubits"+version = "0.1.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "15b85f9c39137c3a891689859392b1bd49812121d0d61c9caf00d46ed5ce06ae"+[[package]]name = "cpufeatures"version = "0.2.17"@@ -276,6 +371,15 @@ dependencies = ["libc",]+[[package]]+name = "crc32fast"+version = "1.5.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78"+dependencies = [+ "cfg-if",+]+[[package]]name = "crypto-common"version = "0.1.7"@@ -286,14 +390,34 @@ dependencies = ["typenum",]+[[package]]+name = "crypto-common"+version = "0.2.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"+dependencies = [+ "hybrid-array",+]+[[package]]name = "digest"version = "0.10.7"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"dependencies = [- "block-buffer",- "crypto-common",+ "block-buffer 0.10.4",+ "crypto-common 0.1.7",+]++[[package]]+name = "digest"+version = "0.11.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"+dependencies = [+ "block-buffer 0.12.1",+ "const-oid",+ "crypto-common 0.2.2",][[package]]@@ -307,6 +431,15 @@ dependencies = ["syn",]+[[package]]+name = "ecb"+version = "0.2.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "26f2a8b3e564eba0877223dc343703ad0385794e882e6d13f3a4dd5c6b1f41ac"+dependencies = [+ "cipher",+]+[[package]]name = "encoding_rs"version = "0.8.42"@@ -330,6 +463,12 @@ dependencies = ["encoding_rs",]+[[package]]+name = "equivalent"+version = "1.0.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "00d174d5400e5e8fd687ad1049e2f578285fa914201b1af7e8b112a4546bd826"+[[package]]name = "errno"version = "0.3.14"@@ -352,6 +491,17 @@ version = "0.1.9"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"+[[package]]+name = "flate2"+version = "1.1.10"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"+dependencies = [+ "crc32fast",+ "miniz_oxide",+ "zlib-rs",+]+[[package]]name = "form_urlencoded"version = "1.2.2"@@ -441,6 +591,12 @@ dependencies = ["smallvec",]+[[package]]+name = "hashbrown"+version = "0.17.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"+[[package]]name = "heck"version = "0.5.0"@@ -486,6 +642,15 @@ version = "1.10.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"+[[package]]+name = "hybrid-array"+version = "0.4.15"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "27f864f10dfb56725ce5ce5472bc52252c8f93a4ab86327122cebf62c5f59a17"+dependencies = [+ "typenum",+]+[[package]]name = "hyper"version = "1.10.1"@@ -648,6 +813,26 @@ dependencies = ["icu_properties",]+[[package]]+name = "indexmap"+version = "2.14.2"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855"+dependencies = [+ "equivalent",+ "hashbrown",+]++[[package]]+name = "inout"+version = "0.2.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3ce983bb6fb399067d101fbfc9e74174e07e9f29a2bc679efefeb8c645819c3f"+dependencies = [+ "block-padding",+ "hybrid-array",+]+[[package]]name = "ipnet"version = "2.12.0"@@ -697,9 +882,36 @@ checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0"[[package]]name = "log"-version = "0.4.33"+version = "0.4.34"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad"+checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"++[[package]]+name = "lopdf"+version = "0.45.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "bfffda0fe1ab0157e1a13c14bebd3f28671f2fccb7922f0722ec53926e6922d3"+dependencies = [+ "aes",+ "bitflags",+ "brotli-decompressor",+ "cbc",+ "ecb",+ "encoding_rs",+ "flate2",+ "getrandom 0.4.3",+ "indexmap",+ "itoa",+ "log",+ "md-5",+ "nom",+ "rand",+ "rangemap",+ "sha2 0.11.1",+ "stringprep",+ "thiserror",+ "weezl",+][[package]]name = "lru-slab"@@ -707,12 +919,32 @@ version = "0.1.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"+[[package]]+name = "md-5"+version = "0.11.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98"+dependencies = [+ "cfg-if",+ "digest 0.11.3",+]+[[package]]name = "memchr"version = "2.8.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"+[[package]]+name = "miniz_oxide"+version = "0.9.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"+dependencies = [+ "adler2",+ "simd-adler32",+]+[[package]]name = "mio"version = "1.2.1"@@ -730,6 +962,27 @@ version = "1.0.0"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "743fb55ba31b18fb1ecef6bdc9aa2743314978ac084044301a7eee33fb99a20d"+[[package]]+name = "nix"+version = "0.31.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d"+dependencies = [+ "bitflags",+ "cfg-if",+ "cfg_aliases",+ "libc",+]++[[package]]+name = "nom"+version = "8.0.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405"+dependencies = [+ "memchr",+]+[[package]]name = "num-traits"version = "0.2.19"@@ -787,6 +1040,15 @@ dependencies = ["unicode-ident",]+[[package]]+name = "quick-xml"+version = "0.42.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "41b1177fdf999d2321d3fb46ff47159d9c1fb9ad66a4879f8c50a0b504615e9b"+dependencies = [+ "memchr",+]+[[package]]name = "quinn"version = "0.11.11"@@ -884,6 +1146,12 @@ dependencies = ["rand_core",]+[[package]]+name = "rangemap"+version = "1.8.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "a611d15b50743feb4c76b7d03edcb0e64f399c26961e4efe6975bc398be6aa3d"+[[package]]name = "regex"version = "1.13.1"@@ -1173,7 +1441,18 @@ checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"dependencies = ["cfg-if","cpufeatures 0.2.17",- "digest",+ "digest 0.10.7",+]++[[package]]+name = "sha2"+version = "0.11.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "47d7069beb7d6ac7b9acd1039986e73443f24234f41074da099d6f994ac9ad19"+dependencies = [+ "cfg-if",+ "cpufeatures 0.3.1",+ "digest 0.11.3",][[package]]@@ -1182,6 +1461,12 @@ version = "2.0.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"+[[package]]+name = "simd-adler32"+version = "0.3.10"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"+[[package]]name = "simdutf8"version = "0.1.5"@@ -1196,9 +1481,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"[[package]]name = "smallvec"-version = "1.15.2"+version = "1.16.3"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90"+checksum = "5b3dc8af474f516a851ff4bd12db780f948b9250ad37211e4eec0bccea54e01b"[[package]]name = "socket2"@@ -1216,6 +1501,17 @@ version = "1.2.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"+[[package]]+name = "stringprep"+version = "0.1.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "7b4df3d392d81bd458a8a621b8bffbd2302a12ffe288a9d931670948749463b1"+dependencies = [+ "unicode-bidi",+ "unicode-normalization",+ "unicode-properties",+]+[[package]]name = "strsim"version = "0.11.1"@@ -1423,18 +1719,45 @@ version = "0.2.5"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"+[[package]]+name = "typed-path"+version = "0.12.3"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e"+[[package]]name = "typenum"version = "1.20.1"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"+[[package]]+name = "unicode-bidi"+version = "0.3.18"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5"+[[package]]name = "unicode-ident"version = "1.0.24"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"+[[package]]+name = "unicode-normalization"+version = "0.1.26"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "75cd3fb03d5288b3ba9a1ee103ac7e9cbb59f0b03d64f8ddfcc04dd42758a85d"+dependencies = [+ "tinyvec",+]++[[package]]+name = "unicode-properties"+version = "0.1.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d"+[[package]]name = "unicode-width"version = "0.2.2"@@ -1588,6 +1911,12 @@ dependencies = ["wasm-bindgen",]+[[package]]+name = "weezl"+version = "0.2.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "d4ca08e5ef825b65b056d9efbd95c8750683f0a6d0466d02e96dc2e4e360f3d2"+[[package]]name = "winapi-util"version = "0.1.11"@@ -1848,8 +2177,41 @@ dependencies = ["syn",]+[[package]]+name = "zip"+version = "9.0.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "6e04976da629d3b04204ba8105b396a193af8ad2583b2c79bfb3ab8af580a11e"+dependencies = [+ "crc32fast",+ "flate2",+ "indexmap",+ "memchr",+ "nix",+ "typed-path",+ "zopfli",+]++[[package]]+name = "zlib-rs"+version = "0.6.8"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "b268e58e7c693d7c271f93ffc4ba3b380412554231c85bf61ca7af91042a4112"+[[package]]name = "zmij"version = "1.0.21"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"++[[package]]+name = "zopfli"+version = "0.8.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "aaf7fc5d30c28483d93805c4a5e12b05bbb52407fa67c5f8bd552374cd01fb11"+dependencies = [+ "bumpalo",+ "crc32fast",+ "log",+ "simd-adler32",+]diff --git a/companions/ai-memory-importer/Cargo.toml b/companions/ai-memory-importer/Cargo.tomlindex 31fac97f..93d0eb22 100644--- a/companions/ai-memory-importer/Cargo.toml+++ b/companions/ai-memory-importer/Cargo.toml@@ -4,12 +4,17 @@name = "ai-memory-importer"version = "0.1.0"edition = "2024"+license = "MIT"publish = false[dependencies]ai-memory-client = { version = "0.1.0", path = "../ai-memory-client" }anyhow = "1"clap = { version = "4", features = ["derive", "env"] }+# Pure-Rust text-layer extraction. It does not OCR scanned/image-only PDFs.+lopdf = { version = "0.45", default-features = false }+quick-xml = "0.42"+libc = "0.2"# Native roots, matching the workspace: the importer POSTs to the same server# an operator may front with a private CA (#492).reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }@@ -18,10 +23,9 @@ serde_json = "1"serde-saphyr = { version = "1.3", default-features = false, features = ["deserialize"] }sha2 = "0.10"tokio = { version = "1", features = ["macros", "rt-multi-thread"] }+tempfile = "3"url = "2"# Stable ingest keys for agentmemory records (UUIDv5 of their ids).uuid = { version = "1", features = ["v5"] }walkdir = "2"--[dev-dependencies]-tempfile = "3"+zip = { version = "9.0.5", default-features = false, features = ["deflate"] }diff --git a/companions/ai-memory-importer/README.md b/companions/ai-memory-importer/README.mdindex f6bfc029..b6916a0f 100644--- a/companions/ai-memory-importer/README.md+++ b/companions/ai-memory-importer/README.md@@ -122,28 +122,53 @@ export page whose `pagination.hasMore` is true needs `--allow-partial`.Claude memory graph and Qdrant imports remain roadmap items; there are no codestubs for them.+### Local documents++`document --file <path>` extracts one explicitly selected local document into+one attributed page under `imported/documents/`. Supported inputs are UTF-8+text and Markdown, text-layer PDF, DOCX, PPTX (including speaker notes), XLSX,+WebVTT, and SRT. CSV and TSV are imported as text so their original delimiters+remain visible. Legacy XLS, XLSB, and ODS files must first be exported as XLSX+or delimiter-separated text.++The page records the source file's basename, format, SHA-256 digest, and+extractor version, followed by sanitized text. Its path includes the full+source digest, and its last line carries a deterministic provenance marker.+A rerun therefore performs no duplicate write. If another page already owns+that path, the import refuses to replace it.++This is text extraction, not a media pipeline. Image-only PDFs need OCR before+import. Images, raw audio/video, and proprietary media transcription are not+supported; provide a completed VTT or SRT transcript instead. The original+opaque file is never uploaded or stored in ai-memory. DOCX text marked hidden+by formatting may still be extracted; deleted revisions and field instructions+are excluded. Review `--show-body` before every live import of a document whose+visibility rules matter.+## Safety contract- Default mode is dry-run; live mode requires `--apply`.- Live mode requires explicit `--workspace`, `--project`, and`--manifest-out <path>`.-- Live writes use only `POST /admin/write-page`; the importer never opens- ai-memory SQLite or wiki files directly and never deletes pages.+- The OMC wiki route writes only through `POST /admin/write-page`. No route+ opens ai-memory SQLite or wiki files directly or deletes pages.- The destination workspace/project must already exist unless`--create-destination` is passed.-- Existing destination pages abort the import unless `--overwrite` is passed.- The importer also re-checks each page immediately before writing.+- Existing OMC destination pages abort the import unless `--overwrite` is+ passed. The importer also re-checks each page immediately before writing.This is best-effort protection: a concurrent writer could still race betweenthe check and `/admin/write-page`, so avoid running competing import/write jobsinto the same destination.-- It stops on the first live-write error and updates the manifest with completed- writes and the failed checkpoint.+- The OMC route stops on the first live-write error and updates the manifest+ with completed writes and the failed checkpoint.- Path handling fails closed: absolute paths, `..`, unsafe destination paths,and reserved/internal destination prefixes are rejected. Duplicate generateddestination paths abort planning.- Dry-run output does not print full page bodies unless `--show-body` is passed.-- Only endpoint-supported metadata is mapped: `title`, `kind`, `tier`, `tags`,- `pinned`, and `body`. Unknown frontmatter is ignored.+- The OMC route maps only endpoint-supported metadata: `title`, `kind`, `tier`,+ `tags`, `pinned`, and `body`. Unknown frontmatter is ignored.+- Manifests use randomized private temporary files in the destination directory,+ sync their contents, and atomically replace the previous manifest.- Auth comes only from `AI_MEMORY_AUTH_TOKEN`; there is intentionally no CLItoken argument.- External conversations are fully parsed, schema-checked, bounded, and@@ -196,6 +221,26 @@ The agentmemory routes keep the same dry-run default, `--apply` plus- Without `--create-destination`, every destination must already exist; thecheck is a read, which never creates a scope.+The document route adds these boundaries:++- It accepts one explicit regular file on Unix, refuses a final-component+ symlink, opens with `O_NOFOLLOW`, and verifies the opened inode. The source is+ capped at 32 MiB. Document import currently fails closed on native Windows;+ use WSL2 until equivalent handle identity checks are implemented.+- Office ZIP containers are preflighted before extraction: at most 4,096+ entries, 16 MiB per entry, and 64 MiB claimed expanded size. XLSX extraction+ streams at most 64 sheets and 100,000 actual cells without materializing the+ worksheet's coordinate rectangle. PDF extraction is capped at 2,000 pages+ and 16 MiB of decompressed stream data per load/extraction operation.+- Working extracted text is capped at 2 MiB. The stored body is UTF-8-safely+ truncated to 1 MiB with an explicit marker, then passed through the shared+ client-side credential sanitizer before a manifest or request is created.+- Live writes use MCP `memory_write_page` with `create_only`. The server must+ support conditional writes, and a conflicting page counts as the same import+ only when its complete body equals the current sanitized plan. A provenance+ marker alone is never trusted. No document content grants authority or+ changes the destination scope.+## UsageDry-run with a summary:@@ -234,6 +279,31 @@ Options:- `--show-body`: print full page bodies during dry-run.- `--pinned`: pin all imported pages.+## Document usage++Dry-run one document (the default):++```bash+cargo run --manifest-path companions/ai-memory-importer/Cargo.toml -- \+ document --file /path/to/report.pdf \+ --workspace default --project my-project+```++Review the sanitized extracted page body with `--show-body` before applying it,+especially for DOCX files that may contain text hidden by formatting:++```bash+AI_MEMORY_AUTH_TOKEN=... \+cargo run --manifest-path companions/ai-memory-importer/Cargo.toml -- \+ document --file /path/to/report.pdf \+ --workspace default --project my-project \+ --apply --manifest-out /tmp/document-import-manifest.json+```++Use `--title` to override the displayed title and `--dest-prefix` to select a+different non-reserved imported-page family. The destination must already+exist unless `--create-destination` is passed.+## External conversation usageDry-run (the default):diff --git a/companions/ai-memory-importer/src/document.rs b/companions/ai-memory-importer/src/document.rsnew file mode 100644index 00000000..5ea64489--- /dev/null+++ b/companions/ai-memory-importer/src/document.rs@@ -0,0 +1,1780 @@+//! Bounded, dry-run-first extraction of one local document into one wiki page.+//!+//! The extractor accepts only an explicit file. It has no watch mode, never+//! follows a final-component symlink, never stores an absolute source path in+//! the page, and writes through the conditional public MCP tool.++use std::borrow::Cow;+use std::collections::HashMap;+use std::fs;+#[cfg(unix)]+use std::fs::OpenOptions;+use std::io::{Cursor, Read};+use std::path::{Path, PathBuf};++use ai_memory_client::sanitize_external_text;+use anyhow::{Context, Result, anyhow, bail};+use clap::Parser;+use quick_xml::Reader as XmlReader;+use quick_xml::events::Event;+use serde::Serialize;+use sha2::{Digest, Sha256};+use uuid::Uuid;+use zip::ZipArchive;++use crate::ImportClient;+use crate::agentmemory::{validate_dest_prefix, validate_planned_path};+use crate::mcp::{McpWrite, Precondition, PreconditionFailed};++const IMPORT_VERSION: &str = "document-v1";+const DEFAULT_DEST_PREFIX: &str = "imported/documents";+const MAX_SOURCE_BYTES: u64 = 32 * 1024 * 1024;+const MAX_ARCHIVE_ENTRIES: usize = 4_096;+const MAX_ARCHIVE_ENTRY_BYTES: u64 = 16 * 1024 * 1024;+const MAX_ARCHIVE_EXPANDED_BYTES: u64 = 64 * 1024 * 1024;+const MAX_EXTRACTED_BYTES: usize = 1024 * 1024;+const MAX_WORKING_TEXT_BYTES: usize = 2 * MAX_EXTRACTED_BYTES;+const MAX_PDF_PAGES: usize = 2_000;+const MAX_PDF_STREAM_BYTES: usize = 16 * 1024 * 1024;+const MAX_SHEETS: usize = 64;+const MAX_SPREADSHEET_CELLS: usize = 100_000;+const MAX_SHARED_STRINGS: usize = MAX_SPREADSHEET_CELLS;++/// Extract one supported local document into one attributed wiki page.+#[derive(Parser, Debug, Clone)]+pub(crate) struct DocumentArgs {+ /// Local PDF, DOCX, PPTX, XLSX, text, Markdown, VTT, or SRT file.+ #[arg(long)]+ file: PathBuf,+ /// Explicit destination workspace. Required with --apply.+ #[arg(long)]+ workspace: Option<String>,+ /// Explicit destination project. Required with --apply.+ #[arg(long)]+ project: Option<String>,+ /// Destination family for imported pages.+ #[arg(long, default_value = DEFAULT_DEST_PREFIX)]+ dest_prefix: String,+ /// Override the page title. The destination path still derives from the file.+ #[arg(long)]+ title: Option<String>,+ #[arg(+ long,+ env = "AI_MEMORY_SERVER_URL",+ default_value = crate::DEFAULT_SERVER_URL+ )]+ server_url: String,+ /// Perform the conditional MCP write. Without this flag, only print a plan.+ #[arg(long)]+ apply: bool,+ /// Durable JSON manifest. Required with --apply.+ #[arg(long)]+ manifest_out: Option<PathBuf>,+ /// Permit the write to create a missing explicit destination.+ #[arg(long)]+ create_destination: bool,+ /// Print the sanitized, bounded page body during a dry run.+ #[arg(long)]+ show_body: bool,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]+#[serde(rename_all = "kebab-case")]+enum DocumentFormat {+ Text,+ Markdown,+ Pdf,+ Docx,+ Pptx,+ Xlsx,+ Vtt,+ Srt,+}++impl DocumentFormat {+ fn detect(path: &Path) -> Result<Self> {+ let extension = path+ .extension()+ .and_then(|value| value.to_str())+ .unwrap_or_default()+ .to_ascii_lowercase();+ match extension.as_str() {+ "txt" | "log" | "rst" | "csv" | "tsv" => Ok(Self::Text),+ "md" | "markdown" => Ok(Self::Markdown),+ "pdf" => Ok(Self::Pdf),+ "docx" => Ok(Self::Docx),+ "pptx" => Ok(Self::Pptx),+ "xlsx" => Ok(Self::Xlsx),+ "vtt" => Ok(Self::Vtt),+ "srt" => Ok(Self::Srt),+ _ => bail!(+ "unsupported document extension {extension:?}; supported: txt, log, rst, csv, tsv, md, markdown, pdf, docx, pptx, xlsx, vtt, srt"+ ),+ }+ }++ const fn as_str(self) -> &'static str {+ match self {+ Self::Text => "text",+ Self::Markdown => "markdown",+ Self::Pdf => "pdf",+ Self::Docx => "docx",+ Self::Pptx => "pptx",+ Self::Xlsx => "xlsx",+ Self::Vtt => "vtt",+ Self::Srt => "srt",+ }+ }++ const fn needs_zip_preflight(self) -> bool {+ matches!(self, Self::Docx | Self::Pptx | Self::Xlsx)+ }+}++#[derive(Debug)]+struct DocumentPlan {+ source_path: String,+ source_filename: String,+ source_sha256: String,+ format: DocumentFormat,+ workspace: String,+ project: String,+ destination_path: String,+ title: String,+ body: String,+ import_key: String,+ extracted_bytes: usize,+ truncated: bool,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]+#[serde(rename_all = "kebab-case")]+enum ManifestStatus {+ Planned,+ Written,+ Present,+ Failed,+}++#[derive(Debug, Serialize)]+struct DocumentManifest {+ import_version: &'static str,+ source_path: String,+ source_filename: String,+ source_sha256: String,+ format: DocumentFormat,+ workspace: String,+ project: String,+ destination_path: String,+ import_key: String,+ extracted_bytes: usize,+ truncated: bool,+ status: ManifestStatus,+ #[serde(skip_serializing_if = "Option::is_none")]+ page_id: Option<String>,+ #[serde(skip_serializing_if = "Option::is_none")]+ error: Option<String>,+}++impl DocumentManifest {+ fn from_plan(plan: &DocumentPlan) -> Self {+ Self {+ import_version: IMPORT_VERSION,+ source_path: plan.source_path.clone(),+ source_filename: plan.source_filename.clone(),+ source_sha256: plan.source_sha256.clone(),+ format: plan.format,+ workspace: plan.workspace.clone(),+ project: plan.project.clone(),+ destination_path: plan.destination_path.clone(),+ import_key: plan.import_key.clone(),+ extracted_bytes: plan.extracted_bytes,+ truncated: plan.truncated,+ status: ManifestStatus::Planned,+ page_id: None,+ error: None,+ }+ }++ fn write(&self, path: &Path) -> Result<()> {+ crate::atomic_write(path, serde_json::to_string_pretty(self)?.as_bytes())+ .with_context(|| format!("write document manifest {}", path.display()))+ }++ fn fail(&mut self, path: &Path, error: &anyhow::Error) -> Result<()> {+ self.status = ManifestStatus::Failed;+ self.error = Some(crate::mcp::preview(&format!("{error:#}")));+ self.write(path)+ }+}++pub(crate) async fn run(args: DocumentArgs) -> Result<()> {+ if args.apply {+ if args+ .workspace+ .as_deref()+ .is_none_or(|value| value.trim().is_empty())+ || args+ .project+ .as_deref()+ .is_none_or(|value| value.trim().is_empty())+ {+ bail!("--apply requires explicit --workspace and --project");+ }+ if args.manifest_out.is_none() {+ bail!("--apply requires --manifest-out <path>");+ }+ }++ let workspace = args+ .workspace+ .clone()+ .unwrap_or_else(|| "DRY-RUN-WORKSPACE".into());+ let project = args+ .project+ .clone()+ .unwrap_or_else(|| "DRY-RUN-PROJECT".into());+ let plan = plan_document(+ &args.file,+ &workspace,+ &project,+ &args.dest_prefix,+ args.title.as_deref(),+ )?;+ let mut manifest = DocumentManifest::from_plan(&plan);++ if !args.apply {+ if args.show_body {+ println!("--- {} ---\n{}", plan.destination_path, plan.body);+ }+ if let Some(path) = &args.manifest_out {+ manifest.write(path)?;+ } else {+ println!(+ "dry-run: {} ({}) -> {}/{}:{}; {} extracted bytes{}; no HTTP writes performed",+ plan.source_filename,+ plan.format.as_str(),+ plan.workspace,+ plan.project,+ plan.destination_path,+ plan.extracted_bytes,+ if plan.truncated { ", truncated" } else { "" }+ );+ }+ return Ok(());+ }++ let manifest_path = args+ .manifest_out+ .as_ref()+ .ok_or_else(|| anyhow!("--apply requires --manifest-out <path>"))?;+ manifest.write(manifest_path)?;+ let client = ImportClient::new(&args.server_url)?;+ if let Err(error) = preflight(&client, &plan, args.create_destination).await {+ manifest.fail(manifest_path, &error)?;+ return Err(error);+ }++ let tags = vec![+ "document-import".to_owned(),+ format!("format-{}", plan.format.as_str()),+ ];+ let write = McpWrite {+ workspace: &plan.workspace,+ project: &plan.project,+ path: &plan.destination_path,+ body: &plan.body,+ title: Some(&plan.title),+ kind: "document",+ tier: "semantic",+ tags: &tags,+ expires_at: None,+ };+ match client.mcp_write_page(&write, Precondition::Absent).await {+ Ok(page_id) => {+ manifest.status = ManifestStatus::Written;+ manifest.page_id = Some(page_id);+ manifest.write(manifest_path)?;+ }+ Err(error) if error.downcast_ref::<PreconditionFailed>().is_some() => {+ let current = client+ .mcp_read_page(&plan.workspace, &plan.project, &plan.destination_path)+ .await+ .with_context(|| {+ format!("reading {} after a refused write", plan.destination_path)+ })?;+ if current.body != plan.body {+ let error = anyhow!(+ "{} already holds a page this document import did not write; refusing to replace it",+ plan.destination_path+ );+ manifest.fail(manifest_path, &error)?;+ return Err(error);+ }+ manifest.status = ManifestStatus::Present;+ manifest.page_id = current.page_id;+ manifest.write(manifest_path)?;+ }+ Err(error) => {+ manifest.fail(manifest_path, &error)?;+ return Err(error.context(format!("writing {}", plan.destination_path)));+ }+ }+ let outcome = match manifest.status {+ ManifestStatus::Written => "wrote",+ ManifestStatus::Present => "already had",+ status => bail!("document import ended in unexpected status {status:?}"),+ };+ println!(+ "import complete: {} {}/{}:{}",+ outcome, plan.workspace, plan.project, plan.destination_path+ );+ Ok(())+}++async fn preflight(+ client: &ImportClient,+ plan: &DocumentPlan,+ create_destination: bool,+) -> Result<()> {+ client.ensure_conditional_writes().await?;+ if !create_destination+ && !client+ .mcp_scope_exists(&plan.workspace, &plan.project)+ .await+ {+ bail!(+ "destination {}/{} must already exist (or pass --create-destination)",+ plan.workspace,+ plan.project+ );+ }+ Ok(())+}++fn plan_document(+ path: &Path,+ workspace: &str,+ project: &str,+ dest_prefix: &str,+ title: Option<&str>,+) -> Result<DocumentPlan> {+ let format = DocumentFormat::detect(path)?;+ let bytes = read_source(path)?;+ if format.needs_zip_preflight() {+ preflight_zip(&bytes)?;+ }+ let source_sha256 = hex_sha256(&bytes);+ let source_filename = source_filename(path)?;+ let stem = path+ .file_stem()+ .and_then(|value| value.to_str())+ .map_or("document", |value| value);+ let title = one_line(&sanitize_external_text(title.unwrap_or(stem)), 200);+ let title = if title.is_empty() {+ "Imported document".to_owned()+ } else {+ title+ };+ let prefix = validate_dest_prefix(dest_prefix)?;+ let destination_path = format!(+ "{prefix}/{}-{source_sha256}.md",+ slugify(&one_line(stem, 64))+ );+ validate_planned_path(&destination_path, &prefix)?;++ let extracted = extract(format, path, &bytes)?;+ let sanitized = sanitize_external_text(&normalize_text(&extracted));+ if sanitized.trim().is_empty() {+ let hint = if format == DocumentFormat::Pdf {+ "; image-only PDFs need OCR before import"+ } else {+ ""+ };+ bail!("document contained no extractable text{hint}");+ }+ let extracted_bytes = sanitized.len();+ let truncated = extracted_bytes > MAX_EXTRACTED_BYTES;+ let extracted = if truncated {+ let suffix = format!(+ "\n\n[Document truncated by ai-memory-importer after {MAX_EXTRACTED_BYTES} bytes.]"+ );+ let budget = MAX_EXTRACTED_BYTES.saturating_sub(suffix.len());+ format!("{}{}", crate::truncate_utf8(&sanitized, budget), suffix)+ } else {+ sanitized+ };+ let import_key = document_key(format, &source_sha256);+ let body = render_page(+ &title,+ &source_filename,+ format,+ &source_sha256,+ &extracted,+ &import_key,+ );+ Ok(DocumentPlan {+ source_path: path.display().to_string(),+ source_filename,+ source_sha256,+ format,+ workspace: workspace.to_owned(),+ project: project.to_owned(),+ destination_path,+ title,+ body,+ import_key,+ extracted_bytes,+ truncated,+ })+}++#[cfg(unix)]+fn read_source(path: &Path) -> Result<Vec<u8>> {+ let metadata = fs::symlink_metadata(path)+ .with_context(|| format!("inspect source document {}", path.display()))?;+ if metadata.file_type().is_symlink() || !metadata.is_file() {+ bail!("source document must be a regular file, not a symlink");+ }+ if metadata.len() > MAX_SOURCE_BYTES {+ bail!(+ "source document is {} bytes; maximum is {MAX_SOURCE_BYTES}",+ metadata.len()+ );+ }+ let mut options = OpenOptions::new();+ options.read(true);+ use std::os::unix::fs::OpenOptionsExt as _;+ options.custom_flags(libc::O_NOFOLLOW | libc::O_NONBLOCK);+ let mut file = options+ .open(path)+ .with_context(|| format!("open {}", path.display()))?;+ let opened_metadata = file.metadata()?;+ if !opened_metadata.is_file() {+ bail!("source document changed before it could be read");+ }+ use std::os::unix::fs::MetadataExt as _;+ if metadata.dev() != opened_metadata.dev() || metadata.ino() != opened_metadata.ino() {+ bail!("source document changed before it could be read");+ }+ let mut bytes = Vec::with_capacity(usize::try_from(metadata.len()).unwrap_or(0));+ file.by_ref()+ .take(MAX_SOURCE_BYTES + 1)+ .read_to_end(&mut bytes)?;+ if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > MAX_SOURCE_BYTES {+ bail!("source document grew past the {MAX_SOURCE_BYTES}-byte maximum while reading");+ }+ Ok(bytes)+}++#[cfg(not(unix))]+fn read_source(_path: &Path) -> Result<Vec<u8>> {+ bail!(+ "document import is unavailable on this platform because stable no-follow source identity cannot be guaranteed"+ )+}++fn source_filename(path: &Path) -> Result<String> {+ let name = path+ .file_name()+ .ok_or_else(|| anyhow!("source document path needs a file name"))?+ .to_string_lossy();+ let name = one_line(&sanitize_external_text(&name), 200);+ if name.is_empty() {+ bail!("source document file name is empty after sanitization");+ }+ Ok(name)+}++fn extract(format: DocumentFormat, _path: &Path, bytes: &[u8]) -> Result<String> {+ match format {+ DocumentFormat::Text | DocumentFormat::Markdown => {+ if bytes.len() > MAX_WORKING_TEXT_BYTES {+ bail!("text document exceeds the {MAX_WORKING_TEXT_BYTES}-byte extraction limit");+ }+ String::from_utf8(bytes.to_vec()).context("text and Markdown documents must be UTF-8")+ }+ DocumentFormat::Pdf => extract_pdf(bytes),+ DocumentFormat::Docx => extract_docx(bytes),+ DocumentFormat::Pptx => extract_pptx(bytes),+ DocumentFormat::Xlsx => extract_xlsx(bytes),+ DocumentFormat::Vtt => extract_transcript(bytes, true),+ DocumentFormat::Srt => extract_transcript(bytes, false),+ }+}++fn extract_pdf(bytes: &[u8]) -> Result<String> {+ std::panic::catch_unwind(|| extract_pdf_inner(bytes))+ .map_err(|_| anyhow!("PDF text extractor panicked on the input; refusing the document"))?+}++fn extract_pdf_inner(bytes: &[u8]) -> Result<String> {+ extract_pdf_inner_with_limit(bytes, MAX_PDF_STREAM_BYTES)+}++fn extract_pdf_inner_with_limit(bytes: &[u8], max_stream_bytes: usize) -> Result<String> {+ let options = lopdf::LoadOptions::with_max_decompressed_size(max_stream_bytes);+ let document = lopdf::Document::load_mem_with_options(bytes, options).context("open PDF")?;+ let pages = document.get_pages();+ let page_count = pages.len();+ if page_count > MAX_PDF_PAGES {+ bail!("PDF has {page_count} pages; maximum is {MAX_PDF_PAGES}");+ }+ let mut output = String::new();+ for (index, page_number) in pages.keys().enumerate() {+ let page = document+ .extract_text_with_limit(&[*page_number], max_stream_bytes)+ .with_context(|| format!("extract PDF page {}", index + 1))?;+ push_bounded(&mut output, &format!("## Page {}\n\n", index + 1))?;+ push_bounded(&mut output, &page)?;+ push_bounded(&mut output, "\n\n")?;+ }+ Ok(output)+}++fn extract_docx(bytes: &[u8]) -> Result<String> {+ let mut archive = ZipArchive::new(Cursor::new(bytes)).context("open DOCX archive")?;+ let mut output = String::new();+ for (name, heading) in [+ ("word/document.xml", None),+ ("word/footnotes.xml", Some("## Footnotes\n\n")),+ ("word/endnotes.xml", Some("## Endnotes\n\n")),+ ] {+ let Some(xml) = read_zip_entry(&mut archive, name)? else {+ continue;+ };+ if let Some(heading) = heading {+ push_bounded(&mut output, heading)?;+ }+ push_bounded(&mut output, &xml_text(&xml)?)?;+ push_bounded(&mut output, "\n\n")?;+ }+ if output.is_empty() {+ bail!("DOCX archive has no word/document.xml text");+ }+ Ok(output)+}++fn extract_pptx(bytes: &[u8]) -> Result<String> {+ let mut archive = ZipArchive::new(Cursor::new(bytes)).context("open PPTX archive")?;+ let presentation = read_zip_entry(&mut archive, "ppt/presentation.xml")?+ .ok_or_else(|| anyhow!("PPTX archive has no presentation metadata"))?;+ let relationships = read_zip_entry(&mut archive, "ppt/_rels/presentation.xml.rels")?+ .ok_or_else(|| anyhow!("PPTX archive has no presentation relationships"))?;+ let slides = pptx_slide_order(&presentation, &relationships)?;+ let mut output = String::new();+ for (position, name) in slides.into_iter().enumerate() {+ let Some(xml) = read_zip_entry(&mut archive, &name)? else {+ bail!("PPTX slide relationship points to missing entry {name:?}");+ };+ push_bounded(&mut output, &format!("## Slide {}\n\n", position + 1))?;+ push_bounded(&mut output, &xml_text(&xml)?)?;+ push_bounded(&mut output, "\n\n")?;+ let number = numbered_part(&name, "ppt/slides/slide", ".xml")?;+ let relationships = format!("ppt/slides/_rels/slide{number}.xml.rels");+ if let Some(xml) = read_zip_entry(&mut archive, &relationships)?+ && let Some(notes) = pptx_notes_target(&xml)?+ && let Some(xml) = read_zip_entry(&mut archive, ¬es)?+ {+ let text = xml_text(&xml)?;+ if !text.trim().is_empty() {+ push_bounded(&mut output, "### Speaker notes\n\n")?;+ push_bounded(&mut output, &text)?;+ push_bounded(&mut output, "\n\n")?;+ }+ }+ }+ if output.is_empty() {+ bail!("PPTX archive has no slide text");+ }+ Ok(output)+}++fn extract_xlsx(bytes: &[u8]) -> Result<String> {+ let mut archive = ZipArchive::new(Cursor::new(bytes)).context("open XLSX archive")?;+ let workbook = read_zip_entry(&mut archive, "xl/workbook.xml")?+ .ok_or_else(|| anyhow!("XLSX archive has no workbook metadata"))?;+ let relationships = read_zip_entry(&mut archive, "xl/_rels/workbook.xml.rels")?+ .ok_or_else(|| anyhow!("XLSX archive has no workbook relationships"))?;+ let sheets = xlsx_sheet_order(&workbook, &relationships)?;+ if sheets.len() > MAX_SHEETS {+ bail!(+ "XLSX workbook has {} sheets; maximum is {MAX_SHEETS}",+ sheets.len()+ );+ }+ let shared_strings = match read_zip_entry(&mut archive, "xl/sharedStrings.xml")? {+ Some(xml) => xlsx_shared_strings(&xml)?,+ None => Vec::new(),+ };+ let mut output = String::new();+ let mut cells = 0_usize;+ for (name, path) in sheets {+ let xml = read_zip_entry(&mut archive, &path)?.ok_or_else(|| {+ anyhow!("XLSX worksheet relationship points to missing entry {path:?}")+ })?;+ push_bounded(+ &mut output,+ &format!("## Sheet: {}\n\n", one_line(&name, 120)),+ )?;+ extract_xlsx_sheet(&xml, &shared_strings, &mut cells, &mut output)?;+ push_bounded(&mut output, "\n")?;+ }+ Ok(output)+}++fn pptx_slide_order(presentation: &str, relationships: &str) -> Result<Vec<String>> {+ let targets = office_relationships(relationships, "/slide", pptx_slide_target)?;+ let mut reader = XmlReader::from_str(presentation);+ let mut slides = Vec::new();+ loop {+ match reader+ .read_event()+ .context("parse PPTX presentation metadata")?+ {+ Event::Start(tag) | Event::Empty(tag) if tag.local_name().as_ref() == "sldId" => {+ let id = required_qualified_attribute(&tag, "r:id", "PPTX slide id")?;+ let target = targets+ .get(&id)+ .ok_or_else(|| anyhow!("PPTX slide id {id:?} has no slide relationship"))?;+ slides.push(target.clone());+ }+ Event::Eof => break,+ _ => {}+ }+ }+ if slides.is_empty() {+ bail!("PPTX presentation contains no ordered slides");+ }+ Ok(slides)+}++fn xlsx_sheet_order(workbook: &str, relationships: &str) -> Result<Vec<(String, String)>> {+ let targets = office_relationships(relationships, "/worksheet", xlsx_sheet_target)?;+ let mut reader = XmlReader::from_str(workbook);+ let mut sheets = Vec::new();+ loop {+ match reader+ .read_event()+ .context("parse XLSX workbook metadata")?+ {+ Event::Start(tag) | Event::Empty(tag) if tag.local_name().as_ref() == "sheet" => {+ let name = required_attribute(&tag, "name", "XLSX sheet name")?;+ let id = required_qualified_attribute(&tag, "r:id", "XLSX sheet id")?;+ let target = targets+ .get(&id)+ .ok_or_else(|| anyhow!("XLSX sheet id {id:?} has no worksheet relationship"))?;+ sheets.push((name, target.clone()));+ }+ Event::Eof => break,+ _ => {}+ }+ }+ if sheets.is_empty() {+ bail!("XLSX workbook contains no worksheets");+ }+ Ok(sheets)+}++fn office_relationships(+ xml: &str,+ type_suffix: &str,+ normalize_target: fn(&str) -> Result<String>,+) -> Result<HashMap<String, String>> {+ let mut reader = XmlReader::from_str(xml);+ let mut relationships = HashMap::new();+ loop {+ match reader.read_event().context("parse Office relationships")? {+ Event::Start(tag) | Event::Empty(tag)+ if tag.local_name().as_ref() == "Relationship" =>+ {+ let relation_type = required_attribute(&tag, "Type", "relationship type")?;+ if !relation_type.ends_with(type_suffix) {+ continue;+ }+ let id = required_attribute(&tag, "Id", "relationship id")?;+ let target = required_attribute(&tag, "Target", "relationship target")?;+ if relationships+ .insert(id.clone(), normalize_target(&target)?)+ .is_some()+ {+ bail!("Office relationships contain duplicate id {id:?}");+ }+ }+ Event::Eof => break,+ _ => {}+ }+ }+ Ok(relationships)+}++fn pptx_slide_target(target: &str) -> Result<String> {+ let filename = target+ .strip_prefix("slides/")+ .or_else(|| target.strip_prefix("/ppt/slides/"))+ .ok_or_else(|| anyhow!("unsafe PPTX slide relationship target"))?;+ let number = numbered_part(filename, "slide", ".xml")?;+ Ok(format!("ppt/slides/slide{number}.xml"))+}++fn xlsx_sheet_target(target: &str) -> Result<String> {+ let filename = target+ .strip_prefix("worksheets/")+ .or_else(|| target.strip_prefix("/xl/worksheets/"))+ .ok_or_else(|| anyhow!("unsafe XLSX worksheet relationship target"))?;+ let number = numbered_part(filename, "sheet", ".xml")?;+ Ok(format!("xl/worksheets/sheet{number}.xml"))+}++fn numbered_part<'a>(value: &'a str, prefix: &str, suffix: &str) -> Result<&'a str> {+ value+ .strip_prefix(prefix)+ .and_then(|value| value.strip_suffix(suffix))+ .filter(|value| !value.is_empty() && value.bytes().all(|byte| byte.is_ascii_digit()))+ .ok_or_else(|| anyhow!("unsafe numbered Office part {value:?}"))+}++fn required_attribute(+ tag: &quick_xml::events::BytesStart<'_>,+ local_name: &str,+ description: &str,+) -> Result<String> {+ for attribute in tag.attributes() {+ let attribute = attribute.context("parse Office XML attribute")?;+ if attribute.key.local_name().as_ref() == local_name {+ return Ok(attribute+ .normalized_value(quick_xml::XmlVersion::Implicit1_0)?+ .into_owned());+ }+ }+ bail!("missing {description}")+}++fn required_qualified_attribute(+ tag: &quick_xml::events::BytesStart<'_>,+ name: &str,+ description: &str,+) -> Result<String> {+ for attribute in tag.attributes() {+ let attribute = attribute.context("parse Office XML attribute")?;+ if attribute.key.as_ref() == name {+ return Ok(attribute+ .normalized_value(quick_xml::XmlVersion::Implicit1_0)?+ .into_owned());+ }+ }+ bail!("missing {description}")+}++fn xlsx_shared_strings(xml: &str) -> Result<Vec<String>> {+ let mut reader = XmlReader::from_str(xml);+ let mut strings = Vec::new();+ let mut current = None::<String>;+ let mut in_text = false;+ loop {+ match reader.read_event().context("parse XLSX shared strings")? {+ Event::Start(tag) if tag.local_name().as_ref() == "si" => {+ if current.is_some() {+ bail!("nested XLSX shared string entry");+ }+ current = Some(String::new());+ }+ Event::Start(tag) if tag.local_name().as_ref() == "t" => in_text = true,+ Event::Text(text) if in_text => {+ if let Some(current) = current.as_mut() {+ push_bounded(current, &text.xml10_content())?;+ }+ }+ Event::CData(text) if in_text => {+ if let Some(current) = current.as_mut() {+ push_bounded(current, &text.xml10_content())?;+ }+ }+ Event::GeneralRef(reference) if in_text => {+ if let Some(current) = current.as_mut() {+ push_bounded(current, &xml_reference(&reference)?)?;+ }+ }+ Event::End(tag) if tag.local_name().as_ref() == "t" => in_text = false,+ Event::End(tag) if tag.local_name().as_ref() == "si" => {+ strings.push(one_line(¤t.take().unwrap_or_default(), 4_096));+ if strings.len() > MAX_SHARED_STRINGS {+ bail!("XLSX exceeds the {MAX_SHARED_STRINGS}-shared-string limit");+ }+ }+ Event::Eof => break,+ _ => {}+ }+ }+ if current.is_some() {+ bail!("unterminated XLSX shared string entry");+ }+ Ok(strings)+}++fn extract_xlsx_sheet(+ xml: &str,+ shared_strings: &[String],+ cells: &mut usize,+ output: &mut String,+) -> Result<()> {+ extract_xlsx_sheet_with_limit(xml, shared_strings, cells, output, MAX_WORKING_TEXT_BYTES)+}++fn extract_xlsx_sheet_with_limit(+ xml: &str,+ shared_strings: &[String],+ cells: &mut usize,+ output: &mut String,+ max_output_bytes: usize,+) -> Result<()> {+ let mut reader = XmlReader::from_str(xml);+ let mut row_cells = None::<usize>;+ let mut cell = None::<(Option<String>, String)>;+ let mut capture_value = false;+ loop {+ match reader.read_event().context("parse XLSX worksheet")? {+ Event::Start(tag) if tag.local_name().as_ref() == "row" => {+ if row_cells.is_some() {+ bail!("nested XLSX row");+ }+ row_cells = Some(0);+ }+ Event::Start(tag) if tag.local_name().as_ref() == "c" => {+ if cell.is_some() {+ bail!("nested XLSX cell");+ }+ *cells = cells+ .checked_add(1)+ .ok_or_else(|| anyhow!("XLSX cell count overflow"))?;+ if *cells > MAX_SPREADSHEET_CELLS {+ bail!("XLSX exceeds the {MAX_SPREADSHEET_CELLS}-cell extraction limit");+ }+ let mut kind = None;+ for attribute in tag.attributes() {+ let attribute = attribute.context("parse XLSX cell attribute")?;+ if attribute.key.local_name().as_ref() == "t" {+ kind = Some(+ attribute+ .normalized_value(quick_xml::XmlVersion::Implicit1_0)?+ .into_owned(),+ );+ }+ }+ cell = Some((kind, String::new()));+ }+ Event::Empty(tag) if tag.local_name().as_ref() == "c" => {+ *cells = cells+ .checked_add(1)+ .ok_or_else(|| anyhow!("XLSX cell count overflow"))?;+ if *cells > MAX_SPREADSHEET_CELLS {+ bail!("XLSX exceeds the {MAX_SPREADSHEET_CELLS}-cell extraction limit");+ }+ let row_cells = row_cells+ .as_mut()+ .ok_or_else(|| anyhow!("XLSX cell appears outside a row"))?;+ push_bounded_with_limit(+ output,+ if *row_cells == 0 { " " } else { "\t" },+ max_output_bytes,+ )?;+ *row_cells += 1;+ }+ Event::Start(tag)+ if cell.is_some() && matches!(tag.local_name().as_ref(), "v" | "t") =>+ {+ capture_value = true;+ }+ Event::Text(text) if capture_value => {+ if let Some((_, value)) = cell.as_mut() {+ push_bounded(value, &text.xml10_content())?;+ }+ }+ Event::CData(text) if capture_value => {+ if let Some((_, value)) = cell.as_mut() {+ push_bounded(value, &text.xml10_content())?;+ }+ }+ Event::GeneralRef(reference) if capture_value => {+ if let Some((_, value)) = cell.as_mut() {+ push_bounded(value, &xml_reference(&reference)?)?;+ }+ }+ Event::End(tag) if matches!(tag.local_name().as_ref(), "v" | "t") => {+ capture_value = false;+ }+ Event::End(tag) if tag.local_name().as_ref() == "c" => {+ let (kind, raw) = cell+ .take()+ .ok_or_else(|| anyhow!("XLSX cell ended without a start"))?;+ let value: Cow<'_, str> = if kind.as_deref() == Some("s") {+ let index = raw+ .parse::<usize>()+ .context("XLSX shared-string cell has a non-numeric index")?;+ Cow::Borrowed(+ shared_strings+ .get(index)+ .ok_or_else(|| {+ anyhow!("XLSX shared-string index {index} is out of range")+ })?+ .as_str(),+ )+ } else {+ Cow::Owned(one_line(&raw, 4_096))+ };+ let row_cells = row_cells+ .as_mut()+ .ok_or_else(|| anyhow!("XLSX cell appears outside a row"))?;+ push_bounded_with_limit(+ output,+ if *row_cells == 0 { " " } else { "\t" },+ max_output_bytes,+ )?;+ push_bounded_with_limit(output, &value, max_output_bytes)?;+ *row_cells += 1;+ }+ Event::End(tag) if tag.local_name().as_ref() == "row" => {+ let cells_in_row = row_cells+ .take()+ .ok_or_else(|| anyhow!("XLSX row ended without a start"))?;+ if cells_in_row == 0 {+ push_bounded_with_limit(output, " ", max_output_bytes)?;+ }+ push_bounded_with_limit(output, "\n", max_output_bytes)?;+ }+ Event::Eof => break,+ _ => {}+ }+ }+ if row_cells.is_some() || cell.is_some() {+ bail!("unterminated XLSX row or cell");+ }+ Ok(())+}++fn pptx_notes_target(relationships: &str) -> Result<Option<String>> {+ let mut reader = XmlReader::from_str(relationships);+ loop {+ match reader+ .read_event()+ .context("parse PPTX slide relationships")?+ {+ Event::Start(tag) | Event::Empty(tag)+ if tag.local_name().as_ref() == "Relationship" =>+ {+ let mut relation_type = None;+ let mut target = None;+ for attribute in tag.attributes() {+ let attribute = attribute.context("parse PPTX relationship attribute")?;+ match attribute.key.local_name().as_ref() {+ "Type" => {+ relation_type = Some(+ attribute+ .normalized_value(quick_xml::XmlVersion::Implicit1_0)?+ .into_owned(),+ )+ }+ "Target" => {+ target = Some(+ attribute+ .normalized_value(quick_xml::XmlVersion::Implicit1_0)?+ .into_owned(),+ )+ }+ _ => {}+ }+ }+ if relation_type+ .as_deref()+ .is_some_and(|value| value.ends_with("/notesSlide"))+ {+ let target =+ target.ok_or_else(|| anyhow!("PPTX notes relationship has no target"))?;+ let filename = target+ .strip_prefix("../notesSlides/")+ .or_else(|| target.strip_prefix("/ppt/notesSlides/"))+ .ok_or_else(|| anyhow!("unsafe PPTX notes relationship target"))?;+ let number = filename+ .strip_prefix("notesSlide")+ .and_then(|value| value.strip_suffix(".xml"))+ .filter(|value| {+ !value.is_empty() && value.bytes().all(|byte| byte.is_ascii_digit())+ })+ .ok_or_else(|| anyhow!("unsafe PPTX notes relationship target"))?;+ return Ok(Some(format!("ppt/notesSlides/notesSlide{number}.xml")));+ }+ }+ Event::Eof => return Ok(None),+ _ => {}+ }+ }+}++fn extract_transcript(bytes: &[u8], webvtt: bool) -> Result<String> {+ let text = std::str::from_utf8(bytes).context("transcript must be UTF-8")?;+ let mut output = String::new();+ let normalized = text.trim_start_matches('\u{feff}').replace("\r\n", "\n");+ for (block_index, block) in normalized.split("\n\n").enumerate() {+ let lines = block.lines().map(str::trim).collect::<Vec<_>>();+ let Some(first) = lines.first().copied().filter(|line| !line.is_empty()) else {+ continue;+ };+ if webvtt && block_index == 0 && first.starts_with("WEBVTT") {+ continue;+ }+ if webvtt && (matches!(first, "STYLE" | "REGION") || first.starts_with("NOTE")) {+ continue;+ }+ let timing_index = if is_timing_line(first) {+ Some(0)+ } else if lines.get(1).is_some_and(|line| is_timing_line(line)) {+ Some(1)+ } else {+ None+ };+ let payload_start = timing_index.map_or(0, |index| index + 1);+ for line in &lines[payload_start..] {+ if line.is_empty() {+ continue;+ }+ push_bounded(&mut output, clean_transcript_markup(line).trim())?;+ push_bounded(&mut output, "\n")?;+ }+ if !output.is_empty() && !output.ends_with("\n\n") {+ push_bounded(&mut output, "\n")?;+ }+ }+ Ok(output)+}++fn is_timing_line(line: &str) -> bool {+ let Some((start, end)) = line.split_once("-->") else {+ return false;+ };+ let looks_like_timestamp = |value: &str| {+ let value = value.split_whitespace().next().unwrap_or_default();+ value.contains(':')+ && value.bytes().any(|byte| byte.is_ascii_digit())+ && value+ .bytes()+ .all(|byte| byte.is_ascii_digit() || matches!(byte, b':' | b'.' | b','))+ };+ looks_like_timestamp(start) && looks_like_timestamp(end)+}++fn clean_transcript_markup(line: &str) -> String {+ let mut output = String::with_capacity(line.len());+ let mut rest = line;+ while let Some(start) = rest.find('<') {+ output.push_str(&rest[..start]);+ let after_start = &rest[start + 1..];+ let Some(end) = after_start.find('>') else {+ output.push_str(&rest[start..]);+ return output;+ };+ let candidate = &after_start[..end];+ if !is_transcript_tag(candidate) {+ output.push('<');+ rest = after_start;+ continue;+ }+ rest = &after_start[end + 1..];+ }+ output.push_str(rest);+ output+}++fn is_transcript_tag(value: &str) -> bool {+ let value = value.trim_start_matches('/');+ let name = value+ .split(|ch: char| ch.is_whitespace() || ch == '.' || ch == ':')+ .next()+ .unwrap_or_default();+ matches!(name, "b" | "c" | "i" | "lang" | "ruby" | "rt" | "u" | "v")+ || (value.contains(':')+ && value+ .bytes()+ .all(|byte| byte.is_ascii_digit() || matches!(byte, b':' | b'.')))+}++fn preflight_zip(bytes: &[u8]) -> Result<()> {+ preflight_zip_with_limits(+ bytes,+ MAX_ARCHIVE_ENTRIES,+ MAX_ARCHIVE_ENTRY_BYTES,+ MAX_ARCHIVE_EXPANDED_BYTES,+ )+}++fn preflight_zip_with_limits(+ bytes: &[u8],+ max_entries: usize,+ max_entry_bytes: u64,+ max_expanded_bytes: u64,+) -> Result<()> {+ let mut archive = ZipArchive::new(Cursor::new(bytes)).context("open Office ZIP container")?;+ if archive.len() > max_entries {+ bail!(+ "Office archive has {} entries; maximum is {max_entries}",+ archive.len()+ );+ }+ let mut expanded = 0_u64;+ for index in 0..archive.len() {+ let entry = archive.by_index(index)?;+ if entry.size() > max_entry_bytes {+ let name = entry.name().map_or_else(+ |_| "<invalid entry name>".to_owned(),+ |name| name.into_owned(),+ );+ bail!(+ "Office archive entry {name:?} expands to {} bytes; per-entry maximum is {max_entry_bytes}",+ entry.size(),+ );+ }+ expanded = expanded+ .checked_add(entry.size())+ .ok_or_else(|| anyhow!("Office archive expanded-size overflow"))?;+ if expanded > max_expanded_bytes {+ bail!("Office archive expands past the {max_expanded_bytes}-byte maximum");+ }+ }+ Ok(())+}++fn read_zip_entry<R: Read + std::io::Seek>(+ archive: &mut ZipArchive<R>,+ name: &str,+) -> Result<Option<String>> {+ let Ok(mut entry) = archive.by_name(name) else {+ return Ok(None);+ };+ if !entry.is_file() || entry.size() > MAX_ARCHIVE_ENTRY_BYTES {+ bail!("unsafe or oversized Office archive entry {name:?}");+ }+ let mut bytes = Vec::with_capacity(usize::try_from(entry.size()).unwrap_or(0));+ entry+ .by_ref()+ .take(MAX_ARCHIVE_ENTRY_BYTES + 1)+ .read_to_end(&mut bytes)?;+ if u64::try_from(bytes.len()).unwrap_or(u64::MAX) > MAX_ARCHIVE_ENTRY_BYTES {+ bail!("Office archive entry {name:?} exceeded its extraction limit");+ }+ Ok(Some(String::from_utf8(bytes).with_context(|| {+ format!("Office XML {name:?} is not UTF-8")+ })?))+}++fn xml_text(xml: &str) -> Result<String> {+ let mut reader = XmlReader::from_str(xml);+ reader.config_mut().trim_text(false);+ let mut output = String::new();+ let mut text_depth = 0_usize;+ let mut skipped_depth = 0_usize;+ loop {+ match reader.read_event().context("parse Office XML")? {+ Event::Start(_) if skipped_depth > 0 => skipped_depth += 1,+ Event::Start(tag)+ if matches!(tag.local_name().as_ref(), "del" | "delText" | "instrText") =>+ {+ skipped_depth = 1;+ }+ Event::End(_) if skipped_depth > 0 => skipped_depth -= 1,+ Event::Start(tag) if tag.local_name().as_ref() == "t" => text_depth += 1,+ Event::End(tag) if tag.local_name().as_ref() == "t" => {+ text_depth = text_depth.saturating_sub(1);+ }+ Event::Text(text) if text_depth > 0 => {+ push_bounded(&mut output, &text.xml10_content())?+ }+ Event::CData(text) if text_depth > 0 => {+ push_bounded(&mut output, &text.xml10_content())?+ }+ Event::GeneralRef(reference) if text_depth > 0 => {+ push_bounded(&mut output, &xml_reference(&reference)?)?;+ }+ Event::Empty(tag) if skipped_depth == 0 => match tag.local_name().as_ref() {+ "br" => push_bounded(&mut output, "\n")?,+ "tab" => push_bounded(&mut output, "\t")?,+ _ => {}+ },+ Event::End(tag) if skipped_depth == 0 => match tag.local_name().as_ref() {+ "p" | "tr" => push_bounded(&mut output, "\n")?,+ "tc" => push_bounded(&mut output, "\t")?,+ _ => {}+ },+ Event::Eof => break,+ _ => {}+ }+ }+ Ok(normalize_text(&output))+}++fn xml_reference(reference: &quick_xml::events::BytesRef<'_>) -> Result<String> {+ Ok(match reference.as_ref() {+ "amp" => "&".to_owned(),+ "lt" => "<".to_owned(),+ "gt" => ">".to_owned(),+ "quot" => "\"".to_owned(),+ "apos" => "'".to_owned(),+ _ => reference+ .resolve_char_ref()?+ .map(|ch| ch.to_string())+ .ok_or_else(|| anyhow!("Office XML contains an unsupported entity"))?,+ })+}++fn push_bounded(output: &mut String, value: &str) -> Result<()> {+ push_bounded_with_limit(output, value, MAX_WORKING_TEXT_BYTES)+}++fn push_bounded_with_limit(output: &mut String, value: &str, limit: usize) -> Result<()> {+ if output.len().saturating_add(value.len()) > limit {+ bail!("extracted document text exceeds the {limit}-byte working limit");+ }+ output.push_str(value);+ Ok(())+}++fn normalize_text(text: &str) -> String {+ let normalized = text.replace("\r\n", "\n").replace('\r', "\n");+ let mut output = String::with_capacity(normalized.len());+ let mut blank = 0;+ for line in normalized.lines() {+ let line = line.trim_end();+ if line.trim().is_empty() {+ blank += 1;+ if blank > 1 {+ continue;+ }+ } else {+ blank = 0;+ }+ output.push_str(line);+ output.push('\n');+ }+ output.trim().to_owned()+}++fn one_line(text: &str, max_chars: usize) -> String {+ one_line_chars(text.chars(), max_chars)+}++fn one_line_chars(chars: impl Iterator<Item = char>, max_chars: usize) -> String {+ if max_chars == 0 {+ return String::new();+ }+ let mut output = String::new();+ let mut pending_space = false;+ let mut emitted = 0_usize;+ for ch in chars {+ if emitted == max_chars {+ break;+ }+ if ch.is_whitespace() {+ pending_space = !output.is_empty();+ continue;+ }+ if pending_space {+ output.push(' ');+ emitted += 1;+ pending_space = false;+ if emitted == max_chars {+ break;+ }+ }+ output.push(ch);+ emitted += 1;+ if emitted == max_chars {+ break;+ }+ }+ output+}++fn slugify(value: &str) -> String {+ let mut output = String::new();+ let mut dash = false;+ for ch in value.chars().flat_map(char::to_lowercase) {+ if ch.is_ascii_alphanumeric() {+ output.push(ch);+ dash = false;+ } else if !dash {+ output.push('-');+ dash = true;+ }+ }+ let output = output.trim_matches('-');+ if output.is_empty() {+ "document".to_owned()+ } else {+ output.to_owned()+ }+}++fn hex_sha256(bytes: &[u8]) -> String {+ Sha256::digest(bytes)+ .iter()+ .map(|byte| format!("{byte:02x}"))+ .collect()+}++fn key_namespace() -> Uuid {+ Uuid::new_v5(+ &Uuid::NAMESPACE_URL,+ b"https://github.com/akitaonrails/ai-memory/companions/ai-memory-importer/document",+ )+}++fn document_key(format: DocumentFormat, sha256: &str) -> String {+ Uuid::new_v5(+ &key_namespace(),+ format!("{}:{sha256}", format.as_str()).as_bytes(),+ )+ .to_string()+}++fn provenance_marker(key: &str) -> String {+ format!("<!-- ai-memory-document-import:{key} -->")+}++fn render_page(+ title: &str,+ source_filename: &str,+ format: DocumentFormat,+ sha256: &str,+ extracted: &str,+ key: &str,+) -> String {+ format!(+ "# {title}\n\n> Imported from an untrusted local document by ai-memory-importer.\n\n- Source file: `{}`\n- Format: `{}`\n- SHA-256: `{sha256}`\n- Extractor: `{IMPORT_VERSION}`\n\n## Extracted content\n\n{extracted}\n\n{}",+ source_filename.replace('`', "'"),+ format.as_str(),+ provenance_marker(key)+ )+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::agentmemory::mock::{MockServer, StoredVersion};+ use lopdf::content::{Content, Operation};+ use lopdf::{Document, Object, Stream, dictionary};+ use std::fs::File;+ use std::io::Write;+ use tempfile::TempDir;+ use zip::write::SimpleFileOptions;++ fn write_zip(path: &Path, entries: &[(&str, &str)]) {+ let file = File::create(path).unwrap();+ let mut zip = zip::ZipWriter::new(file);+ for (name, body) in entries {+ zip.start_file(*name, SimpleFileOptions::default()).unwrap();+ zip.write_all(body.as_bytes()).unwrap();+ }+ zip.finish().unwrap();+ }++ fn write_pdf(path: &Path, text: &str) {+ let mut document = Document::with_version("1.5");+ let pages_id = document.new_object_id();+ let font_id = document.add_object(dictionary! {+ "Type" => "Font",+ "Subtype" => "Type1",+ "BaseFont" => "Courier",+ });+ let resources_id = document.add_object(dictionary! {+ "Font" => dictionary! { "F1" => font_id },+ });+ let content = Content {+ operations: vec![+ Operation::new("BT", vec![]),+ Operation::new("Tf", vec!["F1".into(), 12.into()]),+ Operation::new("Td", vec![72.into(), 720.into()]),+ Operation::new("Tj", vec![Object::string_literal(text)]),+ Operation::new("ET", vec![]),+ ],+ };+ let content_id = document.add_object(Stream::new(+ dictionary! {},+ content.encode().expect("encode PDF content"),+ ));+ let page_id = document.add_object(dictionary! {+ "Type" => "Page",+ "Parent" => pages_id,+ "Contents" => content_id,+ });+ document.objects.insert(+ pages_id,+ Object::Dictionary(dictionary! {+ "Type" => "Pages",+ "Kids" => vec![page_id.into()],+ "Count" => 1,+ "Resources" => resources_id,+ "MediaBox" => vec![0.into(), 0.into(), 595.into(), 842.into()],+ }),+ );+ let catalog_id = document.add_object(dictionary! {+ "Type" => "Catalog",+ "Pages" => pages_id,+ });+ document.trailer.set("Root", catalog_id);+ document.save(path).expect("write PDF fixture");+ }++ fn write_xlsx(path: &Path) {+ write_zip(+ path,+ &[+ (+ "[Content_Types].xml",+ r#"<?xml version="1.0" encoding="UTF-8"?><Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types"><Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/><Default Extension="xml" ContentType="application/xml"/><Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet.main+xml"/><Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml"/></Types>"#,+ ),+ (+ "_rels/.rels",+ r#"<?xml version="1.0" encoding="UTF-8"?><Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="xl/workbook.xml"/></Relationships>"#,+ ),+ (+ "xl/workbook.xml",+ r#"<?xml version="1.0" encoding="UTF-8"?><workbook xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships"><sheets><sheet name="Totals" sheetId="1" r:id="rId1"/></sheets></workbook>"#,+ ),+ (+ "xl/_rels/workbook.xml.rels",+ r#"<?xml version="1.0" encoding="UTF-8"?><Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/worksheet" Target="worksheets/sheet1.xml"/></Relationships>"#,+ ),+ (+ "xl/worksheets/sheet1.xml",+ r#"<?xml version="1.0" encoding="UTF-8"?><worksheet xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><sheetData><row r="1"><c r="A1" t="s"><v>0</v></c><c r="XFD1048576"><v>42</v></c></row></sheetData></worksheet>"#,+ ),+ (+ "xl/sharedStrings.xml",+ r#"<?xml version="1.0" encoding="UTF-8"?><sst xmlns="http://schemas.openxmlformats.org/spreadsheetml/2006/main"><si><t>Item</t></si></sst>"#,+ ),+ ],+ );+ }++ fn apply_args(file: &Path, server_url: &str, manifest: &Path) -> DocumentArgs {+ DocumentArgs {+ file: file.to_path_buf(),+ workspace: Some("w".into()),+ project: Some("p".into()),+ dest_prefix: DEFAULT_DEST_PREFIX.into(),+ title: None,+ server_url: server_url.into(),+ apply: true,+ manifest_out: Some(manifest.to_path_buf()),+ create_destination: false,+ show_body: false,+ }+ }++ #[test]+ fn text_plan_is_redacted_bounded_attributed_and_deterministic() {+ let temp = TempDir::new().unwrap();+ let file = temp.path().join("Release Notes.md");+ fs::write(+ &file,+ format!(+ "# Release\n\ntoken=abcdefghijklmnop\n{}",+ "x".repeat(MAX_EXTRACTED_BYTES)+ ),+ )+ .unwrap();+ let first = plan_document(&file, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();+ let second = plan_document(&file, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();+ assert_eq!(first.destination_path, second.destination_path);+ assert_eq!(first.import_key, second.import_key);+ assert!(+ first+ .destination_path+ .starts_with("imported/documents/release-notes-")+ );+ assert!(first.body.contains("token=[REDACTED]"));+ assert!(!first.body.contains("abcdefghijklmnop"));+ assert!(first.body.contains("Source file: `Release Notes.md`"));+ assert!(first.body.contains("SHA-256:"));+ assert!(+ first+ .body+ .trim_end()+ .ends_with(&provenance_marker(&first.import_key))+ );+ assert!(first.truncated);+ assert!(first.body.contains("Document truncated"));+ let titled = plan_document(+ &file,+ "w",+ "p",+ DEFAULT_DEST_PREFIX,+ Some("token=abcdefghijklmnop"),+ )+ .unwrap();+ assert_eq!(titled.title, "token=[REDACTED]");+ }++ #[test]+ fn docx_extracts_document_footnote_text_and_decodes_entities() {+ let temp = TempDir::new().unwrap();+ let file = temp.path().join("brief.docx");+ write_zip(+ &file,+ &[+ (+ "word/document.xml",+ r#"<w:document xmlns:w="w"><w:body><w:p><w:r><w:t>Hello & world</w:t></w:r><w:del><w:r><w:delText>deleted secret</w:delText></w:r></w:del><w:instrText>field instruction</w:instrText></w:p><w:tbl><w:tr><w:tc><w:p><w:r><w:t>A</w:t></w:r></w:p></w:tc><w:tc><w:p><w:r><w:t>B</w:t></w:r></w:p></w:tc></w:tr></w:tbl></w:body></w:document>"#,+ ),+ (+ "word/footnotes.xml",+ r#"<w:footnotes xmlns:w="w"><w:footnote><w:p><w:r><w:t>Foot</w:t></w:r></w:p></w:footnote></w:footnotes>"#,+ ),+ ],+ );+ let plan = plan_document(&file, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();+ assert!(plan.body.contains("Hello & world"), "{}", plan.body);+ assert!(plan.body.contains("A\n\tB"), "{}", plan.body);+ assert!(plan.body.contains("## Footnotes"));+ assert!(plan.body.contains("Foot"));+ assert!(!plan.body.contains("deleted secret"));+ assert!(!plan.body.contains("field instruction"));+ }++ #[test]+ fn pdf_extracts_text_by_page() {+ let temp = TempDir::new().unwrap();+ let file = temp.path().join("brief.pdf");+ write_pdf(&file, "Quarterly result");+ let plan = plan_document(&file, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();+ assert!(plan.body.contains("## Page 1"), "{}", plan.body);+ assert!(plan.body.contains("Quarterly result"), "{}", plan.body);+ }++ #[test]+ fn pdf_refuses_content_stream_expansion_past_limit() {+ let temp = TempDir::new().unwrap();+ let file = temp.path().join("large-stream.pdf");+ write_pdf(&file, &"x".repeat(4_096));+ let bytes = fs::read(file).unwrap();+ assert!(extract_pdf_inner_with_limit(&bytes, 64).is_err());+ }++ #[test]+ fn xlsx_extracts_sheet_names_and_cells() {+ let temp = TempDir::new().unwrap();+ let file = temp.path().join("totals.xlsx");+ write_xlsx(&file);+ let plan = plan_document(&file, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();+ assert!(plan.body.contains("## Sheet: Totals"), "{}", plan.body);+ assert!(plan.body.contains("Item\t42"), "{}", plan.body);+ }++ #[test]+ fn xlsx_row_hits_output_limit_before_parsing_later_cells() {+ let xml = r#"<worksheet><sheetData><row><c t="s"><v>0</v></c><c/><c><v>&invalid;</v></c></row></sheetData></worksheet>"#;+ let mut output = String::new();+ let mut cells = 0;+ let error =+ extract_xlsx_sheet_with_limit(xml, &["A".to_owned()], &mut cells, &mut output, 5)+ .unwrap_err();+ assert!(error.to_string().contains("5-byte working limit"));+ assert_eq!(output, " A");+ assert_eq!(cells, 2);+ }++ #[test]+ fn xlsx_normalizes_a_long_shared_string_once_before_repeated_references() {+ let shared_xml = format!("<sst><si><t>{}</t></si></sst>", "x".repeat(1_500_000));+ let shared = xlsx_shared_strings(&shared_xml).unwrap();+ assert_eq!(shared[0].len(), 4_096);++ let cells_xml = (0..100)+ .map(|_| r#"<c t="s"><v>0</v></c>"#)+ .collect::<String>();+ let sheet = format!("<worksheet><sheetData><row>{cells_xml}</row></sheetData></worksheet>");+ let mut output = String::new();+ let mut cells = 0;+ let error = extract_xlsx_sheet_with_limit(&sheet, &shared, &mut cells, &mut output, 9_000)+ .unwrap_err();+ assert!(error.to_string().contains("9000-byte working limit"));+ assert_eq!(cells, 3);+ assert_eq!(output.len(), 8_198);+ }++ #[test]+ fn one_line_stops_consuming_when_its_character_budget_is_full() {+ let consumed = std::cell::Cell::new(0_usize);+ let chars = std::iter::from_fn(|| {+ let next = consumed.get() + 1;+ consumed.set(next);+ assert!(next <= 4_096, "one_line consumed past its output budget");+ Some('x')+ });+ let output = one_line_chars(chars, 4_096);+ assert_eq!(output.len(), 4_096);+ assert_eq!(consumed.get(), 4_096);+ }++ #[test]+ fn pptx_uses_presentation_order_and_includes_speaker_notes() {+ let temp = TempDir::new().unwrap();+ let file = temp.path().join("deck.pptx");+ write_zip(+ &file,+ &[+ (+ "ppt/presentation.xml",+ r#"<p:presentation xmlns:p="p" xmlns:r="r"><p:sldIdLst><p:sldId id="10" r:id="rId10"/><p:sldId id="2" r:id="rId2"/></p:sldIdLst></p:presentation>"#,+ ),+ (+ "ppt/_rels/presentation.xml.rels",+ r#"<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide" Target="slides/slide2.xml"/><Relationship Id="rId10" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide" Target="slides/slide10.xml"/></Relationships>"#,+ ),+ (+ "ppt/slides/slide10.xml",+ "<p:sld xmlns:p=\"p\"><p><t>Ten</t></p></p:sld>",+ ),+ (+ "ppt/slides/slide2.xml",+ "<p:sld xmlns:p=\"p\"><p><t>Two</t></p></p:sld>",+ ),+ (+ "ppt/slides/_rels/slide2.xml.rels",+ r#"<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships"><Relationship Id="rId9" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/notesSlide" Target="../notesSlides/notesSlide7.xml"/></Relationships>"#,+ ),+ (+ "ppt/notesSlides/notesSlide7.xml",+ "<p:notes xmlns:p=\"p\"><p><t>Say this</t></p></p:notes>",+ ),+ ],+ );+ let plan = plan_document(&file, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();+ let ten = plan.body.find("Ten").unwrap();+ let two = plan.body.find("Two").unwrap();+ assert!(ten < two);+ assert!(plan.body.contains("## Slide 1"));+ assert!(plan.body.contains("## Slide 2"));+ assert!(plan.body.contains("### Speaker notes"));+ assert!(plan.body.contains("Say this"));+ }++ #[test]+ fn transcripts_drop_timing_metadata_and_markup() {+ let vtt = b"WEBVTT\n\nNOTE private cue note\nskip me\n\nintro-id\n00:00:01.000 --> 00:00:02.000\n<v Alice>Hello</v> world\n\n00:00:03.000 --> 00:00:04.000\n1 < 2 and literal --> arrow\n";+ let text = extract_transcript(vtt, true).unwrap();+ assert_eq!(text.trim(), "Hello world\n\n1 < 2 and literal --> arrow");+ assert!(!text.contains("intro-id"));+ let srt = b"1\n00:00:01,000 --> 00:00:02,000\n12345\n\n2\n00:00:03,000 --> 00:00:04,000\nSecond line\n";+ let text = extract_transcript(srt, false).unwrap();+ assert_eq!(text.trim(), "12345\n\nSecond line");+ }++ #[test]+ fn unsafe_sources_prefixes_and_malformed_archives_fail_closed() {+ let temp = TempDir::new().unwrap();+ let bad = temp.path().join("bad.docx");+ fs::write(&bad, b"not a zip").unwrap();+ assert!(plan_document(&bad, "w", "p", DEFAULT_DEST_PREFIX, None).is_err());+ let text = temp.path().join("a.txt");+ fs::write(&text, "hello").unwrap();+ assert!(plan_document(&text, "w", "p", "sessions", None).is_err());+ assert!(DocumentFormat::detect(Path::new("malware.exe")).is_err());++ let oversized = temp.path().join("oversized.txt");+ File::create(&oversized)+ .unwrap()+ .set_len(MAX_SOURCE_BYTES + 1)+ .unwrap();+ assert!(plan_document(&oversized, "w", "p", DEFAULT_DEST_PREFIX, None).is_err());++ assert!(+ extract(+ DocumentFormat::Text,+ Path::new("large.txt"),+ &vec![b'x'; MAX_WORKING_TEXT_BYTES + 1]+ )+ .is_err()+ );++ #[cfg(unix)]+ {+ std::os::unix::fs::symlink(&text, temp.path().join("link.txt")).unwrap();+ assert!(+ plan_document(+ &temp.path().join("link.txt"),+ "w",+ "p",+ DEFAULT_DEST_PREFIX,+ None+ )+ .is_err()+ );+ }+ }++ #[test]+ fn zip_preflight_refuses_excessive_claimed_expansion() {+ let temp = TempDir::new().unwrap();+ let file = temp.path().join("large.docx");+ write_zip(&file, &[("word/document.xml", &"x".repeat(65))]);+ let bytes = fs::read(file).unwrap();+ assert!(preflight_zip_with_limits(&bytes, 10, 64, 128).is_err());+ }++ #[tokio::test]+ async fn apply_is_conditional_idempotent_and_refuses_foreign_pages() {+ let temp = TempDir::new().unwrap();+ let file = temp.path().join("notes.md");+ fs::write(&file, "# Notes\n\nOne durable fact.").unwrap();+ let server = MockServer::start(&[("w", "p")]);+ let manifest = temp.path().join("manifest.json");++ run(apply_args(&file, &server.url, &manifest))+ .await+ .unwrap();+ assert_eq!(server.state().stored_versions(), 1);+ assert!(+ server+ .state()+ .pages+ .values()+ .flatten()+ .all(|version| version.arguments["create_only"] == true)+ );+ run(apply_args(&file, &server.url, &manifest))+ .await+ .unwrap();+ assert_eq!(server.state().stored_versions(), 1);+ let recorded: serde_json::Value =+ serde_json::from_slice(&fs::read(&manifest).unwrap()).unwrap();+ assert_eq!(recorded["status"], "present");++ let other = temp.path().join("foreign.md");+ fs::write(&other, "A different document.").unwrap();+ let planned = plan_document(&other, "w", "p", DEFAULT_DEST_PREFIX, None).unwrap();+ server.state().pages.insert(+ ("w".into(), "p".into(), planned.destination_path.clone()),+ vec![StoredVersion {+ page_id: "someone-else".into(),+ body: format!(+ "# Hand written\n\n{}",+ provenance_marker(&planned.import_key)+ ),+ arguments: serde_json::Value::Null,+ }],+ );+ let error = run(apply_args(&other, &server.url, &manifest))+ .await+ .unwrap_err();+ assert!(error.to_string().contains("did not write"), "{error:#}");+ assert_eq!(+ server+ .state()+ .chain("w", "p", &planned.destination_path)+ .len(),+ 1+ );+ let recorded: serde_json::Value =+ serde_json::from_slice(&fs::read(&manifest).unwrap()).unwrap();+ assert_eq!(recorded["status"], "failed");+ }+}diff --git a/companions/ai-memory-importer/src/main.rs b/companions/ai-memory-importer/src/main.rsindex 24573696..0fe0bc1f 100644--- a/companions/ai-memory-importer/src/main.rs+++ b/companions/ai-memory-importer/src/main.rs@@ -1,5 +1,6 @@use std::collections::HashMap;use std::fs;+use std::io::Write as _;use std::path::{Component, Path, PathBuf};use std::time::Duration;@@ -13,6 +14,7 @@ use url::Url;use walkdir::WalkDir;mod agentmemory;+mod document;mod mcp;#[cfg(test)]mod test_http;@@ -45,6 +47,8 @@ enum Commands {AgentmemoryVault(agentmemory::vault::VaultArgs),/// Import an agentmemory `ExportData` JSON file.AgentmemoryExport(agentmemory::export::ExportArgs),+ /// Extract one bounded local document into an attributed wiki page.+ Document(document::DocumentArgs),}#[derive(Parser, Debug, Clone)]@@ -308,6 +312,7 @@ async fn main() -> Result<()> {Commands::ExternalConversation(args) => run_conversation(args).await,Commands::AgentmemoryVault(args) => agentmemory::vault::run(args).await,Commands::AgentmemoryExport(args) => agentmemory::export::run(args).await,+ Commands::Document(args) => document::run(args).await,}}@@ -951,8 +956,10 @@ fn write_manifest(path: &Path, entries: &[ManifestEntry]) -> Result<()> {}fn atomic_write(path: &Path, bytes: &[u8]) -> Result<()> {- if let Some(parent) = path.parent()- && !parent.as_os_str().is_empty()+ let parent = path+ .parent()+ .filter(|parent| !parent.as_os_str().is_empty());+ if let Some(parent) = parent&& !parent.exists(){bail!(@@ -960,23 +967,22 @@ fn atomic_write(path: &Path, bytes: &[u8]) -> Result<()> {parent.display());}- let tmp = temp_path_for(path)?;- fs::write(&tmp, bytes)?;- match fs::rename(&tmp, path) {- Ok(()) => Ok(()),- Err(err) => {- let _ = fs::remove_file(&tmp);- Err(err.into())- }- }-}--fn temp_path_for(path: &Path) -> Result<PathBuf> {let file_name = path.file_name().and_then(|s| s.to_str()).ok_or_else(|| anyhow!("manifest path must include a file name"))?;- Ok(path.with_file_name(format!(".{file_name}.tmp")))+ let directory = parent.unwrap_or_else(|| Path::new("."));+ let mut tmp = tempfile::Builder::new()+ .prefix(&format!(".{file_name}."))+ .suffix(".tmp")+ .tempfile_in(directory)?;+ tmp.write_all(bytes)?;+ tmp.flush()?;+ tmp.as_file().sync_all()?;+ tmp.persist(path).map_err(|error| error.error)?;+ #[cfg(unix)]+ fs::File::open(directory)?.sync_all()?;+ Ok(())}fn plan_omc_wiki(@@ -1421,6 +1427,38 @@ mod tests {})}+ #[test]+ fn atomic_manifest_write_replaces_existing_file() {+ let temp = tempdir().unwrap();+ let path = temp.path().join("manifest.json");+ atomic_write(&path, b"first").unwrap();+ atomic_write(&path, b"second").unwrap();+ assert_eq!(fs::read(&path).unwrap(), b"second");+ #[cfg(unix)]+ {+ use std::os::unix::fs::PermissionsExt as _;+ assert_eq!(+ fs::metadata(path).unwrap().permissions().mode() & 0o777,+ 0o600+ );+ }+ }++ #[cfg(unix)]+ #[test]+ fn atomic_manifest_write_does_not_follow_legacy_predictable_temp_symlink() {+ let temp = tempdir().unwrap();+ let path = temp.path().join("manifest.json");+ let victim = temp.path().join("victim");+ fs::write(&victim, "unchanged").unwrap();+ std::os::unix::fs::symlink(&victim, temp.path().join(".manifest.json.tmp")).unwrap();++ atomic_write(&path, b"manifest").unwrap();++ assert_eq!(fs::read_to_string(victim).unwrap(), "unchanged");+ assert_eq!(fs::read(path).unwrap(), b"manifest");+ }+#[test]fn parses_omc_frontmatter() {let parsed = parse_markdown("---\ntitle: T\nkind: rule\ntier: procedural\ntags: [a, b]\npinned: true\nextra: ignored\n---\n# Body\ntext").unwrap();diff --git a/docs/companion-crates.md b/docs/companion-crates.mdindex cc8036da..d2ce9763 100644--- a/docs/companion-crates.md+++ b/docs/companion-crates.md@@ -142,6 +142,11 @@ Initial source support is intentionally narrow:Observations are dropped unless replayed as `external-import` sessionsthrough `/hook/batch`; records with no ai-memory counterpart are counted inthe manifest. Details are in the importer README.+- one explicitly selected local text/Markdown, text-layer PDF, DOCX, PPTX,+ XLSX, VTT, or SRT document (`document`, Unix-only). Extraction is bounded and+ sanitized, pages carry source digest and extractor provenance, and writes+ use MCP `create_only` so reruns are idempotent and conflicts never overwrite+ another page. OCR, images, and raw audio/video stay outside the companion.Future sources can include:diff --git a/docs/comparison.md b/docs/comparison.mdindex 62fc43e0..1eaa7986 100644--- a/docs/comparison.md+++ b/docs/comparison.md@@ -412,6 +412,11 @@ ships something ai-memory does not, it is named.provider-reported usage for each closed internal workload. It records noprompt or output text. Provider/model selection remains global until realmeasurements and quality evaluation justify a larger configuration surface.+- **Document extraction is optional and text-first.** The standalone importer+ turns one explicit local text/Markdown, text-layer PDF, DOCX, PPTX, XLSX,+ VTT, or SRT file into attributed, sanitized markdown with+ bounded parsing and a conditional write. Unlike hosted multimodal products,+ it has no connectors, OCR, image understanding, or raw media transcription.- **Marketplace reach is narrower than harness support.** ai-memory now shipsits own Claude Code and Codex marketplace plugins and `ai-memory setup`detects and wires installed agents across Linux, macOS, and Windows. It isdiff --git a/docs/competitive-parity.md b/docs/competitive-parity.mdindex 64b3b9a6..f71ca7e6 100644--- a/docs/competitive-parity.md+++ b/docs/competitive-parity.md@@ -65,7 +65,7 @@ coding-agent competitors as well as in their original camps.| **claude-mem** (hook capture, LLM observer) | **Yes** for zero-LLM, multi-harness or team use; **not obviously** for a solo Claude Code user happy with it | Zero-LLM capture (it compresses every observation with an LLM), git-markdown truth instead of SQLite FTS5 + Chroma, claim-once handoffs, cross-project messaging, per-project multi-user auth without a paid cloud tier | Its to-do tools, three-step progressive search, native-memory bridge, file observations, and v14's polished local “My Work” queue; TeamBrain adds paid shared-work/collision views || **MemPalace** (hook capture, team server) | **Partly**: overlapping goals, different substrate | Git-markdown truth, claim-once handoffs, typed cross-project messaging, the auth ladder | Its temporal graph and team server as shipped. Its 96.6% / 98.4% R@5 is retrieval-only like our hit@5, but on a different harness, so neither number ranks the tools || **mcp-memory-service** (fact rows + hook capture) | **Yes** if you liked hook capture + typed edges + published numbers | Files-as-truth (not opaque fact rows), handoffs, cross-project messaging, `as_of` temporal, a reproducible number, and (new on 2.4) its aging set at parity done zero-LLM/reversible/off-by-default (per-tier decay curves, extractive compression, DBSCAN cold-cluster dedup, access boosts, 0.4–0.75-band contradiction flagging) | Cloud/multi-backend replication (Cloudflare/Milvus), graph viz. (Belief-strength + clustering consolidation are no longer a give-up: ai-memory now ships both, clustering as a zero-LLM default-off pass, belief-strength off and eval-gated) |-| **basic-memory** (file-first markdown) | **Yes** for multi-harness coding continuity; **no** for a personal Obsidian KB with live team co-edit | Ambient capture (no `write_note` ceremony), full lifecycle, handoffs/messaging, git-versioned truth, published numbers, native OKF v0.2, conditional writes, temporal reads | Local zero-cost reranking, real-time collaborative editing, Postgres backend, `build_context` graph-walk, hosted mobile/web UX, document extraction, and broader structured filters. The latter two are in its unreleased v0.24 changelog, not its stable release |+| **basic-memory** (file-first markdown) | **Yes** for multi-harness coding continuity; **no** for a personal Obsidian KB with live team co-edit | Ambient capture (no `write_note` ceremony), full lifecycle, handoffs/messaging, git-versioned truth, published numbers, native OKF v0.2, conditional writes, temporal reads, structured query filters, and bounded document extraction in the optional importer | Local zero-cost reranking, real-time collaborative editing, Postgres backend, `build_context` graph-walk, hosted mobile/web UX, and its broader document/connectors surface. Its v0.24 features remain unreleased as of this comparison || **agentmemory** (hook capture, KV store) | **Yes** for operability + data ownership | Self-contained binary (no `iii-engine` sidecar, no ELv2 runtime), files-as-truth, SQL indexes committed in-txn, zero-LLM summaries and handoffs, typed claim-once handoffs, a multi-user auth ladder (DB tokens, OIDC), native Windows without a manual sidecar download (3.1 `install.ps1` + `setup`), and a 3.0 importer for its vault and `ExportData`. Both now default to auth on for new installs (theirs since 0.9.30, ours in 3.0) | Browser edit/forget (the 3.0 web UI adds page history/diff, a session timeline, a recall trace and a health panel, but stays read-only by design), the iii console's OTEL traces (we ship explain timings + Prometheus `/admin/metrics` in 3.0), a P2P mesh-sync primitive (`ai-memory-wikisync` is our git-based answer), the opt-in local cross-encoder reranker, an npm/`npx` install (ours is a two-command native install on Linux, macOS and Windows in 3.0/3.1, no npm package), ~31 more tools (54, all visible by default; mostly speculative). Benchmarks are not like-for-like: their 95.2% LongMemEval-S is a component measurement with the reranker disabled (their 2026-10-09 baseline adds 84.68% strict recall-all@5, which they say supports no competitive claim), ours is end-to-end || **Claude Code built-in memory** | **Yes** for teams / multi-machine / multi-harness; **not obviously** for a solo dev on one machine in Claude only | Cross-machine, cross-harness, team sharing, tool-lifecycle capture, real search | Zero setup: it's already on with no server to run (our one structural disadvantage for the solo case; 3.0's `install.sh` + `ai-memory setup` cut ours to two commands on Linux and macOS, but there is still a server) || **Hindsight** (pages-over-facts; coding-agent package) | **Yes** for zero-LLM, files-as-truth, or claim-once handoffs; **no** if you want its end-to-end QA numbers, git-history ingestion, or 16+ agent installer | Zero-LLM default (Hindsight needs an LLM, local or by subscription), markdown files as the source of truth (its markdown is a one-way mirror of Postgres), one binary, claim-once handoffs, cross-project messaging, and (new on 2.4) a belief-strength `confidence` + an opt-in "dream" rewrite that borrow their model but never delete a source and keep the zero-LLM default | Its published end-to-end QA accuracy, git-history ingestion, hooks for 16+ coding agents with one shared bank per repo, a reranker on by default, image/file ingestion, user/team/company scopes inside a bank, and belief-strength that *actually moves ranking by default* (ours ships but is off and eval-gated) |@@ -284,6 +284,7 @@ invariants), **ALREADY** (we have it and should say so).| Foreground query capacity isolated from embedding backfill (EverOS v1.4.2) | SHIPPED (3.2) for local inference | A real-model probe measured eight-worker query p50/p95 rising from 36/41 ms idle to 74/76 ms. Limiting local document inference to one slot while reserving one query slot reduced loaded latency to 39/51 ms. Hosted providers keep their existing concurrency; provider failures still degrade to lexical/entity/graph retrieval || Structured pre-rank filters (Mem0; basic-memory v0.24 unreleased) | SHIPPED (3.2), except author | `memory_query` applies bounded kind, tier, literal path-prefix and exact-array tag predicates before every page-ranking stream, including global, historical, profile and pinned results. `as_of` is the valid-time control. Author stays deferred until null root/anonymous attribution has explicit semantics || Per-operation LLM provider/model profiles (Graphiti v0.31; Hindsight `main`) | TELEMETRY SHIPPED (3.2); ROUTING MEASURE-FIRST | Root-only status now measures calls, outcomes, latency, estimated tokens, reported usage and health for a fixed workload set without storing content. Labels remain routing-inert. Add public profiles only after real measurements plus quality evaluation show a benefit worth the config/auth surface |+| Document extraction (basic-memory v0.24 unreleased; hosted memory APIs) | SHIPPED (3.2) as an optional companion | One explicit local text/Markdown, text-layer PDF, DOCX, PPTX, XLSX, VTT, or SRT file becomes attributed, sanitized markdown through a bounded, dry-run-first Unix importer and conditional MCP write. OCR, images, raw media, connectors, and multimodal embeddings remain out of scope || Retrieval recall through capture (several tools rank whole raw sessions) | SHIPPED (3.0) | The 2026-10-12 decomposition showed our zero-LLM session page indexed only 80-character turn titles; the observation stream recovered it (FTS hit@5 0.662 → 0.951, local 0.800 → 0.945, all 470). Chasing another tool's headline number is declined: those numbers measure different things || Local zero-LLM cross-encoder reranker | MEASURE-FIRST | Fits the zero-LLM invariant (a local model, not an LLM). Bar before building: hit@5 at least +0.03 on both FTS and local paths over the shipped ranking, p95 at most +100 ms on the reference host and measured on a low-end ARM host; default-on only if the keyless path gains. Measured headroom is small after the stream (component-mode FTS 0.972 vs pipeline 0.951) || Editable browser viewer (edit / forget / bulk forget) | DECLINE | The web UI is read-only by design (`frontend-api.md` §10); wiki mutations go only through `Wiki::write_page`/`apply_batch`; agents already edit and forget via MCP (`memory_write_page` with `expected_page_id`, `memory_delete_page`) and the CLI |diff --git a/docs/research-2026-landscape.md b/docs/research-2026-landscape.mdindex de52d51d..477e4083 100644--- a/docs/research-2026-landscape.md+++ b/docs/research-2026-landscape.md@@ -854,9 +854,11 @@ retrieval harness and accuracy/latency/context reporting exist. `memory_query`now exposes kind, tier, literal path-prefix, and exact-array tag predicatesbefore every page-ranking stream, including global, historical, profile, andpinned results. Author remains deferred until null root/anonymous attribution-has explicit semantics. Connectors and document-extraction, if adopted, stay in the importer companion so untrusted input still-crosses the sanitizer boundary and the server remains text-first.+has explicit semantics. Document extraction now stays in the importer+companion: explicit local text/Markdown, text-layer PDF, DOCX, PPTX, XLSX, and+VTT/SRT become attributed sanitized markdown through bounded+conditional writes. The server remains text-first; connectors, OCR, images,+and raw media remain out of scope.### Current plan after the 2026-10-18 pass@@ -889,14 +891,19 @@ Items not marked complete still need their own design and approval.not affect the provider wire request. Real measurements still need to showa quality/cost benefit before public per-workload provider/model settings ora cheap decision tier are added.-5. **P3 - Optional document importer.** If demand is real, extend the- importer companion to extract PDF/Office/media transcripts into attributed,- sanitized markdown. Keep opaque files and native multimodal embeddings out- of the server core.-6. **P3 - End-to-end QA evaluation.** Use the shipped answerer/judge path for- a deliberate full 470-question run and publish its receipt with accuracy,- p50/p95 latency, context tokens, and exact models. Do not build a duplicate- path or claim a cross-tool ranking without a shared harness.+5. **P3 - Optional document importer. Completed.** The companion extracts one+ explicit local text/Markdown, text-layer PDF, DOCX, PPTX, XLSX, VTT, or SRT+ file into attributed, sanitized markdown. Source, archive, extracted+ text, sheet, and cell limits fail closed; deterministic digest paths and+ conditional MCP writes make reruns idempotent without overwriting a foreign+ page. Opaque files, OCR, raw media, connectors, and native multimodal+ embeddings stay out of the server core.+6. **P3 - End-to-end QA evaluation. Deferred to an authorized paid run.** Use+ the shipped answerer/judge path for one deliberate full 470-question run and+ publish its receipt with accuracy, p50/p95 latency, context tokens, and exact+ models. This is not a release gate and needs explicit provider/budget+ approval because it makes roughly 940 paid model calls. Do not build a+ duplicate path or claim a cross-tool ranking without a shared harness.**Deliberately not recommended:** joining the memory-OS camp (agentself-editing its memory is token-expensive, and Letta's own successor nowdiff --git a/docs/security-boundaries.md b/docs/security-boundaries.mdindex a9483a75..10a3d6f0 100644--- a/docs/security-boundaries.md+++ b/docs/security-boundaries.md@@ -125,6 +125,7 @@ means the boundary is not built yet.| 14h | Wiki link graph pages and JSON show only readable projects (#708) | `ai-memory-web/src/routes/graph.rs` `global_handler` calls `ReaderPool::cross_project_edges(None, viewer)` so an edge needs both ends readable; `project_handler` resolves through `resolve_read_scope` (`lookup_existing_scope_guarded`, no-create, `ProjectAccess::Read`) then `project_link_graph`, which ranks and caps nodes in SQL before loading them and queries only edges whose endpoints are among those selected pages; `api.rs` `graph_handler` accepts `workspace`+`project` together (partial scope 400) and passes the same viewer filter, and `project_graph_handler` uses `lookup_project` | `routes.rs` `metadata_shows_only_what_the_viewer_may_read` (HTML `/web/graph`, `/web/w/…/graph`, `/api/v1/…/graph`, and scoped `/api/v1/graph?workspace=&project=`: bob cannot see alpha or client-work, a 403 on alpha's graph leaks neither title nor path, a scoped graph hides an edge whose far end he cannot read; carol and no-viewer controls keep the edge; partial scope 400); store `project_link_graph.rs` hides expired pages, omits cross-project and unresolved links, and verifies the 120-node SQL cap preserves high-degree nodes; `graph.rs` `graph_page_renders_hostile_titles_inert` | STRONG. Bite checks: passing no viewer to `cross_project_edges` in the HTML global handler, resolving the project graph through unguarded `lookup_existing_scope`, or removing the SQL limit fails the restricted-project or node-cap assertions || 14i | Decoded ordinary Markdown destinations stay inside their original project | `ai-memory-wiki/src/markdown.rs` `decode_markdown_destination_path` decodes URL path bytes once before `normalize_link_target` applies scheme, separator and project-root traversal checks; wikilinks remain literal. `ai-memory-web/src/markdown.rs` `scope_relative_link` checks the decoded path before route encoding; an encoded spelling cannot become a generated cross-scope route. | `markdown_url_paths_decode_once_before_link_resolution` (space, Unicode and literal-percent controls); `decoded_markdown_paths_cannot_escape_or_change_scope` (encoded root escape, backslash, external scheme, invalid UTF-8 and controls refused; legitimate in-root normalization passes); `routes.rs` `encoded_markdown_links_resolve_and_remain_in_the_source_scope` writes colliding targets in sibling projects and workspaces, verifies home resolution and no foreign backlinks, and follows the generated home URL; web `decoded_markdown_destinations_cannot_change_the_web_scope` (encoded traversal, root route, separator and scheme refused, encoded `w/` remains a page inside the current scope); `markdown_url_destinations_are_decoded_once_before_route_encoding` | STRONG for one-decode path handling and project confinement; URL grammar outside local Markdown destinations remains owned by the Markdown parser and existing safe-URL checks || 14j | Structured MCP query filters only narrow authorized page results | `ai-memory-mcp/src/server.rs` parses bounded `kind`, `tier`, literal `path_prefix` and exact `tag` values without changing the normal scope-shape or authorization paths, then passes one typed `PageFilter` through project, multi-scope, cross-project, reserved-global/profile, historical and pinned searches. `ai-memory-store/src/reader.rs` binds every filter value as a SQL parameter before each stream's candidate cut; path wildcard characters are escaped and an exact binary-prefix check preserves case, while tags match only string members of a JSON array. A filtered miss cannot fall through to raw observations, which carry no page metadata. | `server.rs` `memory_query_filters_apply_before_every_result_source` combines all filters and covers literal `%`/`_`, case-sensitive paths, tag arrays versus scalars, multi-scope and global search, `as_of`, superseded metadata, `pin_first`, the reserved-global/profile union, traversal-shaped input, invalid tiers, controls and Unicode length bounds; store `structured_filters_hold_in_vector_and_both_graph_directions` isolates path filtering in the vector stream and tag filtering in outgoing and incoming graph expansion; `page_filtering_precedes_the_observation_candidate_cut` puts 80 stronger excluded observations ahead of an eligible tagged session; `memory_query_returns_raw_hits_when_pages_miss` proves an unfiltered miss retains fallback while a filtered miss omits it; `prompts_and_tool_schema_document_query_options` pins the public schema. Existing web every-stream and authorization tests in row 14g exercise kind/tier across the same shared SQL. | STRONG: filters are candidate predicates, never an alternative scope resolver. Empty filter fields normalize to absent for backward compatibility; observation fallback remains available only to unfiltered queries. |+| 14k | Optional document import treats local files as bounded untrusted input | `companions/ai-memory-importer/src/document.rs` accepts one explicit regular file on Unix (and fails closed elsewhere), refuses final symlinks, uses `O_NOFOLLOW` plus opened-inode verification, caps source/archive/entry/working/stored text/PDF-stream/sheet/actual-cell dimensions, streams XLSX cells without allocating their coordinate rectangle, follows PPTX relationship order, rejects unsupported or malformed formats, sanitizes extracted text before manifest or transport, derives the destination and stable provenance key from the source SHA-256, and writes through MCP `create_only` with explicit scope. A conflict is accepted as idempotent only when the current page body exactly equals the current sanitized plan; a provenance marker alone is never trusted and another page is never replaced. Shared manifests use randomized private same-directory temporary files, fsync, and atomic replacement. | Importer `document::tests`: generated bounded and over-limit PDF, visible-text DOCX, relationship-ordered PPTX, sparse-coordinate XLSX, text and cue-state VTT/SRT controls; `text_plan_is_redacted_bounded_attributed_and_deterministic`; `unsafe_sources_prefixes_and_malformed_archives_fail_closed`; `zip_preflight_refuses_excessive_claimed_expansion`; `apply_is_conditional_idempotent_and_refuses_foreign_pages`; shared writer tests cover repeated replacement and a pre-created legacy temp symlink. | STRONG for explicit Unix local text extraction and conditional persistence. Operator review with `--show-body` remains necessary because DOCX formatting-hidden text can be extracted. OCR, images, raw media, watch folders and server-side upload are excluded. || 15 | Agent-archive restore path confinement, resource bounds, and consistency | `ai-memory-cli/src/commands/restore_agents.rs` stages only bounded regular UTF-8 entries; requires a one-to-one manifest/body/checksum/target mapping; allowlists destinations by scope; rejects symlinked descendants; writes each file atomically and rolls back earlier writes after a later failure. `backup_agents.rs` skips direct and directory-root source symlinks, bounds individual assets, attributes a shared skill root only once, and atomically replaces a private archive only after gzip completion | `backup_agents.rs` `restore_agents_rejects_path_traversal`, `restore_agents_rejects_unauthorized_destination_path`, `restore_agents_verifies_sha256_integrity`, `restore_agents_requires_a_complete_one_to_one_manifest`, `restore_agents_rejects_oversized_entries_from_the_header`, `restore_agents_rejects_a_symlinked_destination_parent`, `backup_agents_does_not_follow_config_or_asset_root_symlinks` (Claude and Antigravity roots), and `failed_backup_leaves_the_previous_archive_intact`; `backup_and_restore_agents_round_trip` and `antigravity_backup_round_trips_every_documented_skill_root` are the valid-path controls; `restore_agents::tests::rollback_removes_new_files_and_restores_overwrites` covers recovery | STRONG || 16 | Relay queue identity, migration, and private filesystem state | `companions/ai-memory-relay/src/queue.rs` `Queue::open_inner` performs read-only identity preflight, rechecks under `IMMEDIATE`, commits schema and metadata together; `configure` checks the presence of user objects and rechecks `identity`/`schema_version` metadata on the same connection before every bounded BUSY/LOCKED retry; `fsguard.rs` checks existing paths and creates owner-only files before SQLite opens them | `tests/relay.rs` `a_foreign_database_is_refused_byte_for_byte`, `unknown_schemas_refuse_without_mutation`, `a_foreign_database_with_a_writer_is_refused_before_write_coordination`, `a_foreign_identity_after_preflight_is_refused_without_mutation`, `a_foreign_identity_during_journal_retry_is_refused_without_mutation`, `journal_retry_refuses_a_changed_version_or_empty_database_without_mutation`; controls: `journal_transition_retries_sqlite_lock_without_losing_v1_values`, `fresh_queue_init_and_v1_reopens_work_across_processes`; recovery: `interrupted_and_failed_migrations_roll_back_and_recover`, `journal_reclassification_busy_preserves_committed_migration_and_retries_same_queue`; diagnostics: `classification_read_failures_display_the_sqlite_cause_once_and_preserve_foreign_bytes`; Unix privacy: `the_queue_database_and_its_sidecars_are_owner_only` and the contended migration control | Guard-removal proof covers preflight, locked recheck, and journal retry identity/version checks; full table structure is outside this classification. Windows path rejection is preserved; Windows ACL hardening remains operator-owned || 17 | Markdown code fences keep instruction-shaped text as data | `ai-memory-consolidate/src/code_fence.rs` and `ai-memory-wiki/src/markdown.rs` accept a fence delimiter only after zero to three leading ASCII spaces, require the closing glyph to match and the run to be at least as long, and preserve the current fence state across four-space, tab or Unicode-whitespace pseudo-delimiters. Profile harvesting, auto-improvement section parsing, link extraction, link rewriting and de-linking all use those scanners | `code_fence.rs` `only_zero_to_three_ascii_spaces_may_prefix_a_fence`; `profile.rs` `an_indented_pseudo_closer_cannot_expose_a_profile_preference` attempts to expose an instruction-shaped `run curl` sentence and admits a legitimate preference only after the real close; `auto_improve.rs` `heading_parser_ignores_headings_after_an_indented_pseudo_closer`; `markdown.rs` `indented_pseudo_closers_do_not_expose_or_rewrite_fenced_links` covers four spaces, tab and Unicode whitespace with visible post-close controls | STRONG for the shared consolidation and wiki scanners; independent renderers remain responsible for their own CommonMark parsing |