Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

feat: ingest that reads CSV, TSV and HTML into table and parts sections, names the formats it skips and previews with o2b brain extract (v1.69.0) (#228)

Serg Eroshed49c46979daa
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index 7ef0688f..6cdf893a 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.68.0",
+ "version": "1.69.0",
"description": "Plugin-first second brain package for AI agents and humans.",
"author": {
"name": "Open Second Brain contributors"
diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json
index 26ef83d7..c4c88646 100644
--- a/.codex-plugin/plugin.json
+++ b/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.68.0",
+ "version": "1.69.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index eeee56ad..0442b1d9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,52 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [1.69.0] - 2026-10-02
+
+Ingest now reads more than Markdown: one format registry maps a file's extension to a format, CSV, TSV and HTML sources get a derived `## Table` or `## Parts` section and their content hash on the summary page, which carries the source's visibility, PDF, Office, EPUB, RTF and image files are named as format skips instead of counted as unclassifiable, and the read-only `o2b brain extract` previews what ingest derives from a file; the daily log pages, the daily and weekly briefs, the analytics, claims, backlinks, log, topic, preference, MOC audit and doctor readers, the source-cleanup tools and the recall verdict's root coverage now answer at the caller's reach, private regions and table cells are redacted in linear time, and `brain_recall_gate` accepts the `turn_id` the Hermes plugin sends.
+
+### Added
+
+- **A source format registry.** One registry maps a file's extension to a format and each format to an extractor: Markdown and the lightweight markups (`.md`, `.markdown`, `.txt`, `.text`, `.rst`, `.org`) are read as they are, CSV, TSV and HTML (`.csv`, `.tsv`, `.html`, `.htm`) are extracted, and PDF, the Office formats (`docx`, `xlsx`, `pptx`, `odt`, `ods`, `odp`), EPUB, RTF and images are named but not extracted. No parser dependency and no external program is added.
+- **HTML parts on summary pages.** `brain_ingest_source` on an HTML file in the vault, in the trusted lane and readable at the caller's reach, reduces it to text with a linear scanner (comments, scripts, styles, embedded SVG and MathML dropped, entities decoded, `<private>` regions replaced by their placeholder, also inside a title or a textarea, no attribute value kept) and writes its headings under `## Parts` as one fenced block, one line per heading: `h<level> <trail> | lines <a>-<b>`. At most 256 parts; each heading and the title are redacted (`key=value` credentials and URL userinfo) over their first 4,096 code units, then capped at 200 characters, and a backslash and a `|` in a heading are escaped as `\\` and `\|` so no heading can forge a line span. The scan keeps at most one heading past the cap, so a page of millions of headings is read in bounded memory. The body text never reaches the page. The result gains `parts` (`{extracted: true, count, omitted?}` or `{extracted: false, reason}`).
+- **Table notes for CSV and TSV.** The first record is the header; quoted fields follow RFC 4180 leniently, a `.csv` file is read as semicolon-delimited when its first record has at least two fields on semicolons and more than on commas (so `name;"price, eur";qty` is three columns), and an unterminated quote is refused as `malformed-quoting` with the record number. Rows are rendered under `## Table` as fenced plain text in groups of at most 50 rows and 600 tokens, each repeating the header, so every search chunk of the page carries the column names and a `Table > Rows a-b` heading path. Caps: 1,000 rows, 64 columns, 256 characters per cell, 256 KiB per section, each cut named; only the first 4,096 code units of a cell are read for redaction, and a longer cell counts as cut. A `<private>` region is hidden before the records are split, so a region spanning records hides every row between its tags. A column whose header names a credential has its values replaced and every other cell goes through the output redactor with URL credentials included. Inside a cell a backslash, `|`, a backtick, a line feed, a carriage return and a tab are escaped, and every other control character (C0, DEL and C1) is written as `\u{XXXX}`, so no cell can end the fence or reach a terminal raw. The result gains `table` (`{rendered: true, format, delimiter, columns, rows, rows_rendered, truncated?, redacted_cells}` or `{rendered: false, format, reason, detail?}`).
+- **New frontmatter on derived pages.** `source_format` and `source_content_hash`, then for tables `table_delimiter`, `table_columns`, `table_rows`, `table_rows_rendered` and `table_truncated` (absent when nothing was cut).
+- **`o2b brain extract <file> [--json]`.** A read-only preview of what ingest derives from a file, chosen by its extension, up to 8 MiB: the HTML title and parts (with `--json`, the extracted text and each part's line span and source byte offset), or the table counts and the `## Table` section, or `not extracted: <reason>` (`format-read-verbatim`, `format-not-extractable`, `format-unknown`, `not-a-regular-file`, `source-too-large`, `not-utf8`). A named or unknown format is answered from its extension without reading the file, and a leading frontmatter block is left out as ingest leaves it out. Exit code 0 for every answer; a missing path or a symbolic link exits 1.
+- **`turn_id` on `brain_recall_gate`.** The optional argument (at most 512 characters) the Hermes plugin sends is accepted and recorded on the `gate_telemetry` record.
+
+### Changed
+
+- **CSV, TSV and HTML files are planned by default.** `o2b brain batch-plan` and `brain_ingest_batch_plan` discover them beside Markdown and plain text, and a planned file of a non-text format carries `format` after `status` (CLI: `- <path> (<status>, <bytes>B, <format>)`). The plan id of a tree holding such files changes, so a resume checkpoint written for it before the upgrade no longer matches; a Markdown-only tree plans byte for byte as before. The first ingest after the upgrade of a CSV, TSV or HTML source already ingested rewrites its summary page once with the derived section and keys.
+- **Named but unextracted formats are listed, not counted.** A PDF, Office, EPUB, RTF or image file moves from `unclassifiable` to `skipped_non_extractable` with reason `format-not-extractable` and its format as `detail`, ahead of the schema skips, and `skip_reason_counts` gains that key. An `extensions` override that lists such an extension still plans it.
+- **A summary page is at most as visible as its source.** A summary page with a derived section takes on the source's own `visibility` tokens; when both the page and the source declare tokens, the page gets the audience both the operator and the source allow, and when they share none, the page is withheld below local reach. The reserved token is kept whenever either side carries it. A leading frontmatter block in a CSV, TSV or HTML source is not data: it is never rendered as a row or as text.
+- **Over-long correlation arguments are refused up front.** `brain_recall_gate` refuses a `turn_id` or `session_id` over 512 characters or a `telemetry_host` over 200 with `INVALID_PARAMS` before it runs, with telemetry on or off; such a value used to be dropped from the telemetry record silently, or not checked at all with telemetry off.
+- **The daily and weekly briefs answer at the caller's reach.** Below local reach `brain_brief` `view="daily"` and `view="weekly"` leave out status transitions, retirements and contradictions naming a record the caller cannot read (under its `pref-` or `ret-` id), withhold a source pointer when every evidence event citing it is about such a record or when it names a page the caller cannot read, recompute `events_by_kind` and `vault_delta` from the events the caller may see, and take no report snapshot and show no `delta`.
+- **More readers treat a page or record the caller cannot read at its reach as absent.** A local caller and the CLI see no change.
+ - Below local reach the daily log pages (`Brain/log/`, the JSONL sidecars included) are not served by `brain_search` or any other generic page reader.
+ - `brain_analytics` `view="timeline"`, `view="belief_evolution"` and `view="concept_synthesis"`, `brain_brief` `view="today"`, `brain_query` with `since`, `topic` or `preference`, and the `osb://log/{date}`, `osb://topic/{topic}` and `osb://preference/{id}` resources judge every log entry and record under each id it answers to (`pref-` and `ret-`), drop an entry naming such a record before any limit or total, and answer a refused preference as an absent one; `osb://log/{date}` re-renders the day from the entries it keeps.
+ - `brain_claims` (current and history), `brain_backlinks` and `osb://backlinks/{id}` judge a record under both its spellings, so a retired record is not named under its earlier id.
+ - `brain_moc_audit` answers a hub the caller cannot read as `hub note not found`, and leaves such a member unlisted, unsized and out of the reference counts.
+ - Every `brain_doctor` finding is kept only when the caller can read each record it names; `orphan-evidence` and `malformed-evidence-range` now carry `sources` (the preference the evidence is about) and `removed-tool-reference` carries `path` (vault-relative on the wire), with the messages unchanged.
+ - `brain_search_by_source` and `brain_delete_by_source` honour the reach of a derived summary page: such a page, and the content-manifest entry of a source the caller cannot read, are neither listed nor deleted. A `brain_ingest_source` below local reach leaves a summary page the caller cannot read untouched and answers as a first ingest would.
+ - An entity intake of a source that is reserved against remote reads stays in the trusted lane but records no `source_content_hash` on any Brain page.
+ - The recall verdict's root coverage in `brain_search` and `brain_recall_gate` counts a root as reached only through a page the caller can read.
+- **An open private tag opens a region more readily.** An open `<private>` tag whose attributes hold a `<`, or whose `>` never comes, now opens a region in the redactor, and the session-recall and continuity detectors ask the redactor's own region walker, so an unclosed tag also flags a turn private there.
+- **The bounded source reader sizes its buffer to the file.** It used to allocate the full read limit for every source; it now starts at the file's size and grows once when the file grew under the limit.
+- **Docs:** `docs/mcp.md` gains two "Since v1.69.0" bullets (non-Markdown ingest, and the readers that answer at the caller's reach), `docs/cli-reference.md` the `extract` and `batch-plan` lines, `docs/how-it-works.md` a "Non-Markdown sources" section with the limits of table notes and of the redactor, the architect clause and the doctor finding fields, `docs/observability.md` the `turn_id` field, and `docs/updating.md` "Upgrading to 1.69.0"; the README names this release.
+
+### Fixed
+
+- **An operator-set `visibility` on a summary page survives a re-ingest.** It used to be dropped whenever the source was ingested again, widening a page the operator had narrowed; it is now kept like `created_at`, for every source format.
+- **A byte-order mark no longer hides a frontmatter block.** A page or source that starts with U+FEFF before its `---` block used to be read as having no frontmatter, so its `visibility` did not apply; the mark is now skipped before the block is read.
+- **Private regions and frontmatter blocks are read in linear time.** Stripping `<private>` regions, matching an open tag that never ends, detecting a private region in a session turn or a continuity record and reading a frontmatter block that opens and never closes each took time quadratic in the input on some text; each now scans its input once, so a page of a few megabytes no longer stalls a read.
+- **The digest and `brain_event_trace` keep a dream shared with a readable record.** Below local reach a dream that also names a record the caller can read used to be dropped whole; it is now kept with its readable transitions, and its `confirmed`, `retired` and `new_unconfirmed` lists keep only readable links.
+- **`brain_event_trace` drops evidence on a retired record the caller cannot read.** Below local reach an `apply-evidence` event logged before its preference was retired, which names the record only under its `pref-` id, used to be listed with its artifact and result once the record had moved to its reserved `ret-` page; each record id is now judged under both spellings.
+- **`o2b brain extract` refuses a symbolic link.** A path that is a symbolic link is an error (exit code 1, the message names the link) before any format is named, as the reference states; the verb used to follow the link.
+- **Empty HTML comments no longer hide the rest of a page.** The scanner closes `<!-->` and `<!--->` where they stand; either used to hide every heading and line of text up to the next `-->` or the end of the source.
+- **A part's `source_offset` is counted in the source file.** When a leading frontmatter block (and a byte-order mark before it) is left out, `o2b brain extract --json` used to count each part's byte offset from the end of that block; it now counts from the start of the file.
+- **The architecture overview no longer claims there are no module edges when it cannot draw them.** When every declared edge touches a module whose name a link cannot carry, the `module-dependencies` region names those edges instead of the no-edge sentence; with mixed edges one line under the diagram names the dropped ones, and the module note lists them under `Not linked:`.
+- **Multi-line prompts that start with a command name are admitted by the recall surfacing gate.** The newline check ran on the normalised prompt, where line breaks were already gone; it now tests the raw prompt, so `git status` followed by a question on the next line retrieves, while a single-line command (also with only a trailing line break) is still skipped as `shell_command`.
+
## [1.68.0] - 2026-10-02
The architecture scan now reads the dependency manifests a project declares, at its root and at each module, and turns them into dependency regions, declared module edges and a typed `depends_on` link on module notes; the pre-extractor reads Terraform files into block seeds with `depends_on` and `references` edges, and redacts credentials in import specifiers; and the active digest, the activity and lessons digests, the morning brief, the pre-compress pack, `brain_health` and the `brain_doctor` repair now answer at the caller's reach, while the generic page readers withhold the compiled digest pages from a remote caller.
@@ -8025,6 +8071,7 @@ plugin config (vault field)`, and exits with a clear
- Sandbox vault and plugin manifest fixtures for tests.
- GitHub release workflow for tag-based and manually dispatched releases.
+[1.69.0]: https://github.com/itechmeat/open-second-brain/compare/v1.68.0...v1.69.0
[1.68.0]: https://github.com/itechmeat/open-second-brain/compare/v1.67.0...v1.68.0
[1.67.0]: https://github.com/itechmeat/open-second-brain/compare/v1.66.0...v1.67.0
[1.66.0]: https://github.com/itechmeat/open-second-brain/compare/v1.65.0...v1.66.0
diff --git a/README.md b/README.md
index 1dcca790..e4c11830 100644
--- a/README.md
+++ b/README.md
@@ -113,7 +113,7 @@ The full router with readiness criteria is [`install.md`](install.md); native Wi
## What is new
-1.68.0 teaches the architecture scan to read the dependency manifests a project declares. `o2b brain architect` reads `package.json`, `pyproject.toml`, `Cargo.toml` and `go.mod` at the root and at each module, gives every manifest one status (`read`, `malformed`, `unreadable` or `unsupported`), lists runtime dependencies per ecosystem, and turns a dependency of one module on another into a declared edge: a `module-dependencies` diagram on the overview, a `dependencies` region on each module note and a generator-owned `depends_on` frontmatter key the indexer reads as typed links. The pre-extractor reads Terraform (`.tf`, `.tfvars`) into block seeds with `depends_on` and `references` edges and redacts credentials in import specifiers, and the active digest, the activity and lessons digests, the morning brief, the pre-compress pack, `brain_health` and the `brain_doctor` repair now treat a record the caller cannot read at its reach as absent. Every release is described in the [CHANGELOG](CHANGELOG.md).
+1.69.0 teaches ingest to read more than Markdown. One format registry decides what a file is: CSV, TSV and HTML files are planned by default and `brain_ingest_source` writes what it derives from them onto the summary page that search indexes - a `## Table` section of fenced plain-text row groups that each repeat the header, with named caps and redacted credential columns, or a `## Parts` section of HTML headings and their line spans, never the body - with `source_format`, `source_content_hash` and the table's counts in frontmatter. PDF, Office, EPUB, RTF and image files are named per file as `format-not-extractable` instead of counted as unknown, and `o2b brain extract` previews the extraction of any file without writing. The recall verdict's root coverage, the daily and weekly briefs, the daily log pages, the analytics, claims, backlinks, MOC audit and doctor readers and the source-cleanup tools now answer at the caller's reach, a summary page carries its source's visibility, private regions and table cells are redacted in linear time, `brain_recall_gate` accepts the `turn_id` the Hermes plugin sends, multi-line prompts that start with a command name are now admitted by the surfacing gate, and the architecture notes name the module edges a link cannot carry instead of claiming there are none. Every release is described in the [CHANGELOG](CHANGELOG.md).
## Documentation
diff --git a/docs/brainstorm/non-markdown-ingest/cli-output/claude.md b/docs/brainstorm/non-markdown-ingest/cli-output/claude.md
new file mode 100644
index 00000000..e39e47d6
--- /dev/null
+++ b/docs/brainstorm/non-markdown-ingest/cli-output/claude.md
@@ -0,0 +1,40 @@
+### Variant 1: Pre-extract gains a second lane, "text structure"
+
+- **Approach**: Extend the existing opt-in pre-extract pass (`runPreExtract` in ingest.ts, `pre_extract` on `brain_ingest_source`) with a sibling result shape, `PreExtractText`, that for `.html`/`.htm` returns extracted text and heading-derived parts (via `HTMLRewriter` with a text handler on the document plus element handlers for `h1-h6`/`title`, a hand-written entity decoder for the five XML entities plus numeric refs, and no `onEndTag`), and for `.csv`/`.tsv` returns a header row, a column count, a row count and a bounded row sample. PDF, Office and image extensions join `LANGUAGE_BY_EXTENSION`'s neighbour table as a closed `UNSUPPORTED_SOURCE_KIND` vocabulary so the pass answers `extracted: false, reason: <token>` by name. The calling agent still writes the summary page; the kernel only returns the structure it read, and the batch planner gains `.html .htm .csv .tsv` in `DEFAULT_INGESTIBLE_EXTENSIONS` plus a per-extension `unsupported` bucket beside `unclassifiable`.
+- **Trade-offs**:
+ - Pro: one read path, one bounded reader, one opt-in flag, no new tool and no description change on `brain_ingest_source` (already at 299).
+ - Pro: HTML text never lands in the vault; the page is still agent-authored, so provenance and redaction stay as today.
+ - Pro: the CSV table never enters the chunker, sidestepping the header-loss and `[[x]]`/`#tag` cell problems entirely.
+ - Con: a CSV's rows are not searchable unless the agent renders them, so the "table without rows" gap is only half closed; the kernel can offer a fenced sample but cannot force it onto the page.
+ - Con: overloading `pre_extract` (named "code-structure") with text structure muddies the result type and the CLI verb name; a rename or alias ripples through serializers and tests.
+ - Con: `HTMLRewriter` runs in Bun only; the Node-bundled build needs a conditional fallback (regex strip) or a named `unsupported-on-this-runtime` reason.
+- **Complexity**: medium
+- **Risk**: medium
+
+### Variant 2: A "source reading" kernel stage with a rendered table block on the page
+
+- **Approach**: Introduce a new ingest-side module `source-read.ts` that runs before the page write for every file source (not opt-in), classifies the source into a closed `SOURCE_FORMAT` vocabulary (`markdown-like`, `html`, `delimited-table`, `pdf`, `office`, `image`, `unknown`), and produces a `SourceReading` carried on the ingest result and, for `delimited-table`, written by the kernel into a fenced ```` ```table ```` block on the summary page (header + rows up to a byte cap, fence so links and tags stay inert, redactor's raw-output pass applied cell-wise). HTML yields heading parts only. Unsupported formats become a typed `SkippedPageReason`-style result and a batch-plan `unsupported` map.
+- **Trade-offs**:
+ - Pro: closes the CSV gap fully. Rows become searchable because they live on the only indexed page, and the fence keeps them out of the graph and out of the GFM chunker's header-splitting path.
+ - Pro: one closed vocabulary covers both planner counting and ingest reporting; the census test registers it once.
+ - Con: the kernel now writes source body onto a page it previously never touched. This crosses a line the task draws for HTML ("no source body copied") and changes the agent/kernel contract for a page the agent owns; existing text sources must be proven byte-identical by test.
+ - Con: a code fence in a table block defeats the chunker's table awareness, and the chunker's line-by-line split of oversize fences still drops the header in later chunks; closing that needs a chunker change (repeat the fence header per chunk) in the same release.
+ - Con: cell-wise redaction has to run on data the kernel never saw before; a false-negative leaks a secret into an indexed page.
+ - Con: not opt-in means every ingest pays the read; the 1 MiB allocation fold-in becomes mandatory, not optional.
+- **Complexity**: large
+- **Risk**: high
+
+### Variant 3: Planner-only honesty plus a sidecar reader verb
+
+- **Approach**: Keep ingest untouched for this release. The batch planner learns a closed `SOURCE_FORMAT` classification and reports `unsupported` per extension and reason (`binary-document`, `image`, `archive`) beside `unclassifiable`, and a new CLI-only verb `o2b brain source-read <path>` (no MCP tool) returns HTML text with heading parts or CSV/TSV header, dimensions and a sample, for the agent to fold into the page it writes. Fold-ins ship as independent commits.
+- **Trade-offs**:
+ - Pro: smallest blast radius. No ingest result shape change, no page content change, byte-identity of text sources is trivially preserved.
+ - Pro: the Node-bundled build is unaffected because the verb lives in the CLI only.
+ - Con: MCP callers (the primary surface) cannot reach the reader, so for the agent in an MCP session nothing changes about HTML or CSV. The release title promises ingest of non-Markdown sources and delivers it only to CLI users.
+ - Con: two format vocabularies drift (planner and verb) unless shared, and the sidecar invites a later MCP tool anyway, which the constraints discourage.
+ - Con: the CSV rows-in-the-index gap stays open.
+- **Complexity**: small
+- **Risk**: low
+
+### Recommended: Variant 1
+**Rationale**: It reaches the MCP caller through the existing `pre_extract` flag and bounded reader, so no new tool, no description budget and no second read path are needed, and it keeps the kernel out of the page body, which matches both the "no source body copied" rule for HTML and the byte-identity constraint. Variant 2 closes the CSV gap more fully but does so by having the kernel write unredacted-by-design source rows onto an agent-owned page, which couples the release to a chunker change and a redaction guarantee the project does not yet have; Variant 3 is safe but leaves MCP sessions exactly where they are. The CSV residue in Variant 1 is honest: the result names header, dimensions and a fenced sample, and the agent renders the rows it chooses, which is the same contract the project already uses for every other source.
diff --git a/docs/brainstorm/non-markdown-ingest/cli-output/expected-output.md b/docs/brainstorm/non-markdown-ingest/cli-output/expected-output.md
new file mode 100644
index 00000000..d3c275b3
--- /dev/null
+++ b/docs/brainstorm/non-markdown-ingest/cli-output/expected-output.md
@@ -0,0 +1,287 @@
+# Expected output, before and after
+
+Fixture vault `<vault>` with a folder `Clips/`:
+
+- `page.html` (138 bytes): `<html><head><title>Release notes</title></head><body><h1>Overview</h1><p>Fish &amp; chips</p><h2>Install</h2><p>Run it.</p></body></html>`
+- `parts.csv` (22 bytes): `name,qty` / `bolt,4` / `nut,7`
+- `report.pdf` (9 bytes, a PDF header only), `blob.bin` (3 bytes), `notes.md` (15 bytes).
+
+The "before" blocks were captured on f5c46615 with the CLI from the worktree (Bun 1.4.0). The "after" blocks are the target shape pinned by the plan; `<plan id>` stands for the new plan id (the discovered set gains the HTML and CSV files, so the id changes).
+
+## `o2b brain batch-plan Clips`
+
+Before:
+
+```
+batch-plan: Clips (plan 6e4905944b42be45)
+ 1 file(s) to ingest in 1 batch(es); 0 unchanged skipped
+ batch 0: 1 file(s), 15 byte(s)
+ - Clips/notes.md (new, 15B)
+ 4 unclassifiable file(s) not planned, by extension:
+ - .bin: 1
+ - .csv: 1
+ - .html: 1
+ - .pdf: 1
+```
+
+After:
+
+```
+batch-plan: Clips (plan <plan id>)
+ 3 file(s) to ingest in 1 batch(es); 0 unchanged skipped
+ batch 0: 3 file(s), 175 byte(s)
+ - Clips/notes.md (new, 15B)
+ - Clips/page.html (new, 138B, html)
+ - Clips/parts.csv (new, 22B, csv)
+ 1 non-extractable page(s) skipped:
+ - Clips/report.pdf (format-not-extractable: pdf)
+ by reason: format-not-extractable=1
+ 1 unclassifiable file(s) not planned, by extension:
+ - .bin: 1
+```
+
+## `o2b brain batch-plan Clips --json`
+
+Before:
+
+```json
+{
+ "ok": true,
+ "source_dir": "Clips",
+ "max_batch_bytes": 1048576,
+ "max_batch_files": 25,
+ "total_files": 1,
+ "total_bytes": 15,
+ "skipped": [],
+ "unclassifiable": {
+ "total": 4,
+ "by_extension": {
+ ".bin": 1,
+ ".csv": 1,
+ ".html": 1,
+ ".pdf": 1
+ }
+ },
+ "plan_id": "6e4905944b42be45",
+ "resumed_completed": 0,
+ "batches": [
+ {
+ "index": 0,
+ "total_bytes": 15,
+ "files": [
+ {
+ "path": "Clips/notes.md",
+ "bytes": 15,
+ "status": "new"
+ }
+ ]
+ }
+ ]
+}
+```
+
+After:
+
+```json
+{
+ "ok": true,
+ "source_dir": "Clips",
+ "max_batch_bytes": 1048576,
+ "max_batch_files": 25,
+ "total_files": 3,
+ "total_bytes": 175,
+ "skipped": [],
+ "skipped_non_extractable": [
+ {
+ "path": "Clips/report.pdf",
+ "reason": "format-not-extractable",
+ "detail": "pdf"
+ }
+ ],
+ "skip_reason_counts": {
+ "format-not-extractable": 1
+ },
+ "unclassifiable": {
+ "total": 1,
+ "by_extension": {
+ ".bin": 1
+ }
+ },
+ "plan_id": "<plan id>",
+ "resumed_completed": 0,
+ "batches": [
+ {
+ "index": 0,
+ "total_bytes": 175,
+ "files": [
+ { "path": "Clips/notes.md", "bytes": 15, "status": "new" },
+ { "path": "Clips/page.html", "bytes": 138, "status": "new", "format": "html" },
+ { "path": "Clips/parts.csv", "bytes": 22, "status": "new", "format": "csv" }
+ ]
+ }
+ ]
+}
+```
+
+## `o2b brain extract <vault>/Clips/page.html`
+
+Before (every form of the verb):
+
+```
+error: unknown brain verb: extract
+usage: o2b brain <verb> [args...]
+```
+
+After:
+
+```
+extract: <vault>/Clips/page.html (html)
+ title: Release notes
+ 2 part(s):
+ h1 Overview | lines 1-2
+ h2 Overview > Install | lines 3-4
+```
+
+After, `--json`:
+
+```json
+{
+ "ok": true,
+ "path": "<vault>/Clips/page.html",
+ "extracted": true,
+ "format": "html",
+ "title": "Release notes",
+ "text": "Overview\nFish & chips\nInstall\nRun it.",
+ "parts": [
+ { "index": 0, "level": 1, "heading": "Overview", "trail": "Overview", "line_start": 1, "line_end": 2, "source_offset": 53 },
+ { "index": 1, "level": 2, "heading": "Install", "trail": "Overview > Install", "line_start": 3, "line_end": 4, "source_offset": 93 }
+ ]
+}
+```
+
+## `o2b brain extract <vault>/Clips/parts.csv`
+
+After:
+
+````
+extract: <vault>/Clips/parts.csv (csv)
+ comma-delimited, 2 column(s), 2 row(s), 2 rendered, 0 redacted cell(s)
+## Table
+
+### Rows 1-2
+
+```table
+name | qty
+bolt | 4
+nut | 7
+```
+````
+
+After, `--json`:
+
+```json
+{
+ "ok": true,
+ "path": "<vault>/Clips/parts.csv",
+ "extracted": true,
+ "format": "csv",
+ "delimiter": "comma",
+ "columns": 2,
+ "rows": 2,
+ "rows_rendered": 2,
+ "redacted_cells": 0,
+ "section": "## Table\n\n### Rows 1-2\n\n```table\nname | qty\nbolt | 4\nnut | 7\n```"
+}
+```
+
+After, on `report.pdf` and on `notes.md` (`--json`):
+
+```json
+{ "ok": true, "path": "<vault>/Clips/report.pdf", "extracted": false, "format": "pdf", "reason": "format-not-extractable" }
+{ "ok": true, "path": "<vault>/Clips/notes.md", "extracted": false, "format": "text", "reason": "format-read-verbatim" }
+```
+
+## `brain_ingest_source` result
+
+Before, for `sourcePath: "Clips/page.html"` (and the same shape for `Clips/parts.csv`):
+
+```json
+{
+ "summary_path": "Brain/sources/<slug>-<hash12>.md",
+ "created": true,
+ "entities_created": [],
+ "entities_updated": [],
+ "connections": [],
+ "capture_scope": "full-local"
+}
+```
+
+After, HTML:
+
+```json
+{
+ "summary_path": "Brain/sources/<slug>-<hash12>.md",
+ "created": true,
+ "entities_created": [],
+ "entities_updated": [],
+ "connections": [],
+ "capture_scope": "full-local",
+ "parts": { "extracted": true, "count": 2 }
+}
+```
+
+The page gains `source_format: html` and `source_content_hash: <sha256>` and ends with:
+
+````
+## Parts
+
+```parts
+h1 Overview | lines 1-2
+h2 Overview > Install | lines 3-4
+```
+````
+
+After, CSV:
+
+```json
+{
+ "summary_path": "Brain/sources/<slug>-<hash12>.md",
+ "created": true,
+ "entities_created": [],
+ "entities_updated": [],
+ "connections": [],
+ "capture_scope": "full-local",
+ "table": { "rendered": true, "format": "csv", "delimiter": "comma", "columns": 2, "rows": 2, "rows_rendered": 2, "redacted_cells": 0 }
+}
+```
+
+The page gains `source_format: csv`, `source_content_hash: <sha256>`, `table_delimiter: comma`, `table_columns: 2`, `table_rows: 2`, `table_rows_rendered: 2`, and ends with the `## Table` section shown above. At remote reach, a hidden `.csv` source and an absent one both answer `"table": { "rendered": false, "format": "csv", "reason": "source-not-local" }` with no digest and no section; `notes.md` answers exactly as before (no `parts`, no `table`, page byte-identical).
+
+## `o2b brain architect` module-dependencies region, only unlinkable edges
+
+Fixture: module `web` depends only on `@mono/ab`, owned by module `a[b]`.
+
+Before (overview region, false):
+
+```
+No module's manifest names exactly one other module's manifest as a runtime dependency.
+```
+
+Before (module note `web`, false):
+
+```
+Depends on: no other module
+```
+
+After (overview region):
+
+```
+Every declared module edge touches a module whose name a link cannot carry, so none is drawn:
+- `web` -> `a[b]`
+```
+
+After (module note `web`):
+
+```
+Not linked: `a[b]`
+```
diff --git a/docs/brainstorm/non-markdown-ingest/cli-output/prompt.md b/docs/brainstorm/non-markdown-ingest/cli-output/prompt.md
new file mode 100644
index 00000000..e121961b
--- /dev/null
+++ b/docs/brainstorm/non-markdown-ingest/cli-output/prompt.md
@@ -0,0 +1,49 @@
+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.
+
+# Task
+
+One release that lets Open Second Brain ingest its first non-Markdown sources honestly, plus small fold-ins.
+
+1. Source artifact ingest, first slice. Ingest is text-only today: `DEFAULT_INGESTIBLE_EXTENSIONS` (src/core/brain/ingest/batch-plan.ts:78-85) is `.md .markdown .txt .text .rst .org`; HTML, PDF, Office files and images are only counted per extension as `unclassifiable`. The closed skip-reason vocabulary `SKIPPED_PAGE_REASON` (extractable-gate.ts:34-45) has one member, `schema-type-not-extractable`. The ingest pipeline never reads the source for the agent: the calling agent writes the summary page under `Brain/sources`; the only in-kernel read is the opt-in code-structure pre-extract pass (bounded read, 1 MiB). The search walker indexes `.md` only, so a part of an HTML file is searchable only if it is rendered onto the Markdown summary page. Wanted: HTML becomes text with heading-derived structural parts (no source body copied into the page), PDF, Office and images are named as unsupported instead of dropped silently, OCR stays caller-side, no new runtime dependency (Bun 1.4.0 has the `HTMLRewriter` global; it does not decode entities, misses document-level text in element handlers and throws on `onEndTag` for void elements; part of src/ is bundled for Node).
+
+2. CSV and TSV as plain-text table notes. Nothing in src parses tables. A table ingested without its rows is the gap. The summary page is the only indexed place where rows could live; the chunker splits oversize tables line by line so a GFM table loses its header in later chunks; `[[x]]` and `#tag` in table cells would become graph facts unless inside a code fence. The redactor has a key-name predicate and a raw-output redaction pass.
+
+3. Fold-ins: (a) `readSourceBounded` allocates the full cap for every source; (b) the recall verdict's root coverage counts a root as reached when it holds only pages the caller cannot read; (c) the daily and weekly brief views name preference ids from log events at remote reach; (d) the architect module-dependencies region says "no edges" when every declared edge touches a module a wikilink cannot carry; (e) the Hermes plugin sends `turn_id` to `brain_recall_gate`, whose closed input schema refuses it; (f) a dead newline branch in the surfacing gate's shell-only test.
+
+# Project context
+
+Open Second Brain, TypeScript on Bun 1.4.0 (also a Node-bundled OpenClaw build from part of src/), MCP server plus `o2b` CLI over an Obsidian-compatible Markdown vault.
+Recent commits:
+f5c46615 feat: dependency facts from project manifests, Terraform seeds in the pre-extractor and readers that answer at the caller's reach (v1.68.0)
+400b20ea feat: distillation pages that prove what they quote and say how much of their source the vault holds (v1.67.0)
+3ec2334a feat: stable error codes on the MCP wire, typed rerank degradation and a rerank doctor check (v1.66.0)
+a1ecfe7b feat: unattended maintenance recipes, install-owned lane tasks and an MCP fault guard (v1.65.0)
+4499698c feat: search that honours pins and event time, diagnostics that prove action, writes that name their residue (v1.64.0)
+Related files:
+src/core/brain/ingest/batch-plan.ts, src/core/brain/ingest/extractable-gate.ts, src/core/brain/ingest/ingest.ts, src/core/brain/ingest/pre-extract.ts, src/core/brain/intake/source-trust.ts, src/core/brain/provenance/capture-scope.ts, src/core/brain/research/extract.ts, src/core/search/chunker.ts, src/core/redactor.ts, src/mcp/brain/ingest-tools.ts, src/cli/brain/verbs/pre-extract.ts, src/core/search/indexer.ts, src/mcp/brain/brief-tools.ts, src/core/brain/architect/generate.ts, src/core/search/surfacing-gate.ts, src/mcp/search-tools.ts
+Conventions:
+- Closed reason vocabularies use a four-piece idiom (frozen object, derived union, member list, unknown guard) and are registered in a census test.
+- Errors surface by name; no silent fallbacks; "could not do it" is data with a reason.
+- Deterministic output, codepoint sort order, no natural-language word lists.
+- At remote reach a page withheld by visibility is treated as absent; every new reader gets an A/B test.
+- MCP tool descriptions are capped at 300 characters; `brain_ingest_source` is at 299.
+Constraints:
+- No new runtime dependency. Text sources and existing pages stay byte-identical.
+- No new MCP tool unless unavoidable. External extractor commands and OCR are out of scope.
+
+# Required output format
+
+Produce exactly 3 distinct architectural variants. For each variant:
+
+### Variant N: <short name>
+- **Approach**: 2-3 sentences describing the variant.
+- **Trade-offs**: bullet list of pros and cons.
+- **Complexity**: small | medium | large
+- **Risk**: low | medium | high
+
+After the three variants, add exactly one recommendation:
+
+### Recommended: Variant N
+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.
+
+Output nothing outside of these sections.
diff --git a/docs/brainstorm/non-markdown-ingest/design.md b/docs/brainstorm/non-markdown-ingest/design.md
new file mode 100644
index 00000000..440171be
--- /dev/null
+++ b/docs/brainstorm/non-markdown-ingest/design.md
@@ -0,0 +1,139 @@
+# Non-Markdown source ingest - HTML parts, CSV and TSV table notes, named unsupported formats, and readers that answer at the caller's reach
+
+**Status:** draft (phase 1 spec review applied)
+**Author:** release orchestrator (via feature-release-playbook)
+**Audience:** implementation
+**Base:** origin/main f5c46615 (v1.68.0), branch `feat/non-markdown-ingest`, target version 1.69.0 (minor)
+
+## Problem statement
+
+Ingest in Open Second Brain is text-only. `DEFAULT_INGESTIBLE_EXTENSIONS` (`src/core/brain/ingest/batch-plan.ts:78-85`) holds `.md .markdown .txt .text .rst .org`; every other file only adds to a per-extension `unclassifiable` count (`batch-plan.ts:600-604`). The search walker indexes `.md` files only (`src/core/search/walker.ts:118`), so an exported web page or a spreadsheet export in the vault never reaches search or recall. Nothing parses HTML into structure (the only reducer, `htmlToText` in `src/core/brain/research/extract.ts:27-37`, is a 4000-character web-finding helper with no headings and no entity decoding), nothing parses tables, and PDF, Office and image files are dropped by an anonymous count instead of being named. Six smaller defects ride along: the bounded source reader allocates its full cap for every file, the recall verdict's root coverage counts a root as reached through pages the caller cannot read, the daily and weekly brief views name preference ids from log events whatever the caller's reach, the architect says "no edges" when every declared edge touches a module a link cannot carry, the Hermes plugin's `turn_id` is refused by `brain_recall_gate`, and the surfacing gate has a dead newline branch.
+
+## Premise verification (live source, f5c46615)
+
+Full evidence: the reconnaissance reports of this wave (HTML extraction and format registry, CSV and TSV table notes, v1.68.0 left-overs, the surfacing-gate facts). The orchestrator re-opened the anchors this design builds on: `batch-plan.ts:75-100, 130-195, 255-330, 578-625`, `extractable-gate.ts:30-90`, `ingest.ts:168-240, 286-345`, `ingest-tools.ts:95-126, 255-275`, `registry-guard.ts:352`, `search-tools.ts:255-270, 566-610`, `surfacing-gate.ts:40-58`, `indexer.ts:1725-1752`, `brief-tools.ts:170-195`, `architect/generate.ts:335-345, 419-432`, `chunker.ts:28-36, 157`, the OpenClaw bundle's module list, and ran the current CLI on a fixture tree (outputs in `cli-output/expected-output.md`).
+
+### Card A (source artifact ingest) - partly false, narrowed and built as narrowed
+
+- False: "no general artifact indexer". Per-source summary pages with provenance (`ingestSource`, `ingest.ts:152-285`), the sources registry, the content-hash manifest (it hashes raw bytes, so a changed HTML or CSV file is re-planned with no change), the batch planner and a closed skip-reason union already exist.
+- False in its framing: the pipeline never reads a source's text for the agent; the calling agent writes the summary. A "searchable part" can therefore only exist as text on the indexed Markdown summary page.
+- Narrowed (validator ruling, kept): PDF, Office and images are named as unsupported, not extracted; OCR stays caller-side; watches and URL fetch are out.
+- Built: a format registry, HTML text with heading-derived parts (headings and line spans only, never body text, on the page), a named `format-not-extractable` skip, and a read-only `o2b brain extract` preview verb.
+
+### Card B (plain-text notes for tabular data) - confirmed, narrowed
+
+- Confirmed: nothing in `src/` parses or renders tables; `.csv` is the canonical `unclassifiable` example in two test files.
+- Narrowed: CSV and TSV only, no formulas or workbook metadata (those live in XLSX and ODS, named unsupported under card A), no new dependency.
+- New contract recorded: this is the first time the kernel writes derived source content onto a `kind: brain-source` page. It does so only for an in-vault file in the trusted lane, after the reach check, with two redaction passes, under named caps.
+
+### Fold-ins - confirmed, folded as narrowed
+
+1. `readSourceBounded` allocates `PRE_EXTRACT_MAX_SOURCE_BYTES + 1` for every source (`ingest.ts:329`): confirmed; folded (the `architect/manifests.ts:224-249` pattern).
+2. Coverage-divergent root flip (`indexRootCoverage`, `indexer.ts:1727-1750`, no reach filter): confirmed; folded.
+3. Whole-layer status and brief counts: confirmed and wider than recorded (daily and weekly envelopes name preference ids from log events). Only the id slice is folded; the counts are deferred (see Out of scope).
+4. Architect "no edges" sentence when only unlinkable edges exist (`generate.ts:419-432`, `NO_DEPENDS_ON` `:342`): confirmed, plus the same false sentence on the module note; folded.
+5. Hermes `turn_id` refused by the closed `RECALL_GATE_INPUT_SCHEMA` (`search-tools.ts:566-609`, `additionalProperties: false`): confirmed; folded. The sibling schema at `search-tools.ts:263` already declares the same `turn_id` (512), which is the pattern copied.
+6. Dead newline branch in `isShellOnlyPrompt` (`surfacing-gate.ts:53` tests `normalized`, already whitespace-collapsed at `:45`): confirmed. Premise corrected: testing the raw prompt for a line break admits multi-line prompts that start with a command name; a single-line prose prompt that starts with a command name ("git push keeps failing after the rebase") is still skipped as `shell_command`. The CHANGELOG and docs state exactly that, not "prose prompts are admitted".
+
+## Scope
+
+1. One format registry, `src/core/brain/ingest/source-formats.ts` (Node-safe leaf, no `Bun.*`): the `SOURCE_FORMAT` vocabulary, one spec per format with its extractor, the extension map, `sourceFormatOf`, and `DEFAULT_INGESTIBLE_EXTENSIONS` derived from it (re-exported from `batch-plan.ts`).
+2. Planner: `.csv .tsv .html .htm` planned by default; PDF, Office, EPUB, RTF and image files reported per file as `format-not-extractable` with the format as `detail`; a planned file whose format is not `text` carries `format`.
+3. HTML: a linear scanner (`extract-html.ts`) giving title, text and parts; `## Parts` on the summary page.
+4. CSV and TSV: `table-note.ts` giving a fenced plain-text `## Table` section and five `table_*` frontmatter keys.
+5. `brain_ingest_source`: automatic by format, `source_format` plus `source_content_hash` on derived pages, additive `parts` and `table` result fields (counts and tokens only).
+6. CLI `o2b brain extract <file> [--json]` for every registered format.
+7. Fold-ins 1, 2, 3 (id slice), 4, 5 and 6 above.
+
+## Out of scope (deferred, by name)
+
+- **PDF, Office (DOCX, XLSX, PPTX, ODT, ODS, ODP), EPUB, RTF and images**: named `format-not-extractable`, not extracted. PDF text needs content-stream and font decoding that is unreliable without a library; Office is feasible later through `node:zlib` but is its own change.
+- **OCR**: caller-side (validator ruling); the per-file skip with `detail: image` is the hook a caller routes on.
+- **External-extractor command gate**: a new subprocess capability reachable from MCP needs the `secrets/exec.ts` audit discipline, an apply gate and an allowlisted command form; the `format-not-extractable` entries are the hook a later gate claims (a config key on the `bench_judge_cmd` precedent, `config.ts:1230`).
+- **Watches**: overlap the maintenance lane; the content-hash manifest already re-plans changed bytes.
+- **URL fetch**: ingest stays network-free.
+- **A read-only `brain_extract_source` MCP tool**: a new capability (tool count, parity, probe catalogue, docs) for a model that assumes the agent holds the source.
+- **Whole-layer count aggregates of the status and brief views** (`computeBrainStatus`, `buildOperatorSnapshot`, monthly, operator and today views, `osb://status`): 12-16 files, a refusal-code decision and many tests that need a local reach; deferred as its own change. Only the daily and weekly slice ships now: their ids, and their `events_by_kind` and `vault_delta` counts recomputed from the events the caller may see (review round).
+- `htmlToText` in `research/extract.ts` stays unchanged (a different contract: a bounded web finding).
+
+## Chosen approach
+
+The consultant recommended its Variant 1 (a second lane of the opt-in `pre_extract` pass, nothing written to the page). The orchestrator overrides it for Variant 2's spine, adjusted (see `variants.md`): card B's deliverable is rows that search can find, and search indexes only the summary page, so a preview the agent may or may not copy leaves the gap open. The consultant's two objections to Variant 2 are answered in the design: the header-loss problem by token-bounded row groups that each repeat the header (no chunker rule change), and the redaction worry by two named passes with a pinned over-redaction test. HTML keeps the consultant's "no body copied" line: only headings and line spans reach the page.
+
+## Design decisions
+
+### Registry and planner (lane A)
+
+- **R1 one registry.** `SOURCE_FORMAT` (four-piece, census row) = `text, csv, tsv, html, pdf, docx, xlsx, pptx, odt, ods, odp, epub, rtf, image`. Markdown and the lightweight markups map to `text` (they are read as they are). Each spec is `{format, extractor, extensions}` with `extractor` from `SOURCE_EXTRACTOR` (`verbatim | html | table`) or `null`. Evidence: the pre-extract `LANGUAGE_BY_EXTENSION` single-home pattern; a Map, not prose, decides.
+- **R2 derived defaults.** `DEFAULT_INGESTIBLE_EXTENSIONS` = the extensions of every spec with a non-null extractor, in spec order: the existing six first, then `.csv .tsv .html .htm`. Same name and export from `batch-plan.ts`, so nothing that iterates it reorders.
+- **R3 skip routing.** In `collectIngestible`, after the ignore check and before the reach predicate: an extension in the effective set is discovered; an extension the registry names with `extractor: null` becomes `{reason: "format-not-extractable", detail: <format>}`; anything else stays `unclassifiable`. `format-not-extractable` is FIRST in `SKIPPED_PAGE_REASONS` ("in the order a page meets the gates"). Format skips sort by path and precede schema skips in `skippedNonExtractable`. Evidence: visibility is a Markdown rule (docs/mcp.md:2062-2064) and calling the predicate on a binary reads it whole (`vault.ts:200`); the unclassifiable count already runs before the predicate for the same reason. A caller's `extensions` override still wins: a named format the caller lists is planned.
+- **R4 planned format.** `PlannedFile.format?: SourceFormat`, set only when the format is not `text`; serialized as `format` beside `status`. A tree of Markdown files plans byte-identically. The plan id of a tree holding HTML, CSV or TSV changes (those files join the discovered set), disclosed in `docs/updating.md`.
+- **R5 bounded reader.** `readSourceBounded(absolute, maxBytes)` allocates `stat.size + 1` and grows once to `maxBytes + 1` when the buffer fills under the cap (the `manifests.ts` pattern), exported for the CLI verb; the pre-extract pass passes `PRE_EXTRACT_MAX_SOURCE_BYTES` unchanged.
+- **R6 one fence helper.** `fenceFor` moves from `provenance/capture-scope.ts` to `src/core/markdown-fence.ts`; the excerpt section and both new sections use it.
+
+### Dispatch and the ingest write (lane A)
+
+- **I1 dispatch.** `extract-source.ts` exports `extractSource(path, bytes)` with an exhaustive `switch (spec.extractor)` over `null`, `verbatim`, `html` and `table`. The orchestrator steer had lane A ship the `verbatim` and `null` arms and lanes B and C each add one arm. Overturned: an exhaustive switch over a union that already holds `html` and `table` does not type-check without those arms, and a placeholder arm would be a stub. Lane A writes all four arms in its last task (A4), which waits on the two pure modules (B2, C2) whose exported signatures this plan pins; `extract-source.ts` then has one owner.
+- **I2 trigger.** Automatic by format; no new MCP argument, no new tool, the `brain_ingest_source` description (299 of 300 characters) untouched. Evidence: the derived section is the page content the card asks for; the pre-extract pass is opt-in because its output never reaches the page.
+- **I3 reach order.** Identity is a vault-relative file whose format has an extractor of `html` or `table`; then the trusted lane; then `isSourceHidden(vault, source, readable)`; then one `readSourceOrigin` read (8 MiB, `O_NOFOLLOW`, digest of exactly the bytes returned). A URL, an absent file and a hidden file all answer `source-not-local` with no digest and no section. A source above 8 MiB is already refused by the intake before this step.
+- **I4 page.** The derived section comes after `## Connections to existing notes`. Frontmatter gains `source_format` and `source_content_hash` (via `sourceContentHashFrontmatter`) whenever the bytes were read, plus the format's own keys. A `text` source gains nothing: byte-identical.
+- **I5 visibility kept.** An operator-set `visibility` on an existing summary page is preserved on rewrite, the way `created_at` is (`ingest.ts:201-202`). Applied to every summary page: dropping it on re-ingest is the same defect for a text source, and the key is the only control a rendered table has. Disclosed in `docs/updating.md`.
+- **I6 wire.** `parts` (HTML) and `table` (CSV, TSV) on the result, each absent for other formats, counts and tokens only. `registry-guard.ts:352` reason becomes "write; returns the summary path, bounded id lists and derived-section counts".
+
+### HTML (lane B)
+
+- **H1 scanner, not HTMLRewriter.** A pure TypeScript linear scanner (index walk, no regex over unbounded input). Evidence: HTMLRewriter leaves entities undecoded, misses document-level text in element handlers, throws on `onEndTag` for void elements, and is Bun-only (recon probes on Bun 1.4.0 and 1.4.2).
+- **H2 grammar.** Comments, doctype and processing instructions dropped; `script style noscript template svg math` skipped whole; `<private>` content becomes `PRIVATE_REGION_PLACEHOLDER`; numeric entities (invalid code points become U+FFFD) and `amp lt gt quot apos nbsp` decoded, any other named entity kept verbatim; block elements emit line breaks; `pre` keeps whitespace, elsewhere whitespace collapses; `<title>` is the title; no attribute value is ever emitted; CRLF and CR normalised before lines are counted; strict UTF-8, refused by name.
+- **H3 parts.** `{index, level, heading, trail, lineStart, lineEnd, sourceOffset}`; level 0 is a non-empty preamble; `trail` joins ancestor headings with ` > `; headings folded with `oneLine` and capped at 200 characters; at most `HTML_PARTS_MAX = 256` parts, the rest counted in `partsOmitted`.
+- **H4 section.** `## Parts` holds one fenced block (info string `parts`, fence by `fenceFor`) with one line per part, `h<level> <trail> | lines <a>-<b>` (`preamble | lines <a>-<b>` for level 0). Inside a fence, `[[...]]` and `#word` from untrusted headings stay out of the link graph (`stripCode`, `tags.ts:38-52`) while FTS still indexes the text.
+- **H5 CLI.** `o2b brain extract <file> [--json]`, read-only, through `readSourceBounded(file, SOURCE_HASH_MAX_BYTES)`, prints the dispatch result for any registered format; text and named-unsupported formats print `{extracted: false, format, reason}`.
+
+### Tables (lane C)
+
+- **T1 section.** Fenced plain text, not a GFM table: one block per row group under `### Rows <a>-<b>`, info string `table`, the header record first in every group, cells joined by ` | `, cell escapes `\` -> `\\`, `|` -> `\|`, LF -> `\n`, CR -> `\r`, TAB -> `\t`, and (review round) a backtick -> ``\` `` so no row line can start a fence and every table body takes a three-backtick fence, and every other C0 control, DEL and C1 control -> `\u{XXXX}`. Evidence: a GFM table turns `[[x]]` and `#word` cells into graph facts and loses its header in later chunks (`splitBlock`, `chunker.ts:511`).
+- **T2 group size.** At most `TABLE_NOTE_ROWS_PER_GROUP = 50` rows AND at most `TABLE_NOTE_GROUP_MAX_TOKENS = 600` tokens counted by the chunker's own counter (exported as `countChunkTokens`), so a group block fits one 800-token chunk and every chunk carries a header line and a `Table > Rows a-b` heading path. Adjusted from the steer's fixed 50 rows: a 50-row group of a 15-column table exceeds one chunk and its later chunks lose the header. A single row over the budget forms its own group (residue: such a group can still split). `CHUNKER_VERSION` untouched.
+- **T3 parsing.** Delimiter by extension: `.tsv` TAB without quote processing; `.csv` comma with RFC 4180 lenient quoting, switching to semicolon when the first record has two or more fields with semicolon and more fields with semicolon than with comma (amended in the review round from "one field with comma": a comma-decimal export quotes only fields holding `;`, so its header `name;"price, eur";qty` has two comma fields; `"a;b",c` stays comma). BOM stripped, empty lines skipped, CRLF, LF and lone CR end records, bare quotes kept, an unterminated quoted field refused as `malformed-quoting` with the 1-based record number. The first record is always the header. Ragged rows keep their own cell count.
+- **T4 caps.** `TABLE_NOTE_MAX_ROWS = 1_000`, `TABLE_NOTE_MAX_COLUMNS = 64`, `TABLE_NOTE_MAX_CELL_CHARS = 256` (code points, U+2026 marks a cut), `TABLE_NOTE_MAX_BYTES = 262_144` for the section. Every cut is named in `table_truncated` and a closing line states rendered and total rows.
+- **T5 redaction.** A column whose header satisfies `isSecretKeyName` has every value replaced by `REDACTION_PLACEHOLDER` (header kept); every other cell goes through `redactRawOutput(cell, {redactUrlCredentials: true, maxInput: Number.POSITIVE_INFINITY})`; no `redactTokens` (it would erase order ids and hashes). `keyword` headers over-redact, pinned by a test. Count on `redacted_cells`.
+- **T6 refusals.** `TABLE_NOTE_SKIP_REASON`: `source-not-local`, `not-utf8`, `contains-nul`, `malformed-quoting`, `empty`. Data, not errors: the page and entities are still written.
+
+### Reach left-overs (lane D)
+
+- **L1 root coverage.** `indexRootCoverage(config, roots, admit?)` counts a root reached only through an admitted path; below local reach `admit` is `isPathReadableAtReach` over `indexedVisibilityByPaths` read in the same store session; `probeRetrievalCorpus(resolveConfig, reach?)` threads it; local reach unchanged. The `brain_search` and `brain_recall_gate` residual sentences leave the registry.
+- **L2 brief id slice.** Below local reach the daily and weekly views drop `status_transitions`, `retired`, `contradictions` and `source_pointers` rows whose ids fail `readerRefView`, and skip `captureReportDelta` (the digest precedent, `brief-tools.ts:190`). Counts stay as they are and the registry row says so.
+- Public wording: "answers at the caller's reach".
+
+### Small fixes (lane E; `turn_id` rides in lane D, which owns `search-tools.ts`)
+
+- **E1 architect.** When every declared edge touches an unlinkable module the region prints a distinct fixed sentence and lists the dropped edges as code spans (`codeSpanName(from) -> codeSpanName(to)`); with some edges drawn, one extra line under the diagram names the dropped ones; the module note says `Not linked:` with the code spans instead of `Depends on: no other module`. `depends_on` frontmatter unchanged.
+- **E2 `turn_id`.** Optional `turn_id` (string, max 512) on `RECALL_GATE_INPUT_SCHEMA`, recorded on the `gate_telemetry` payload under `CONTINUITY_TURN_ID_KEY`; `plugins/hermes/_schemas.py` re-vendored verbatim; a `prefetch(..., turn_id=...)` drive in `tests/python/test_static_schemas.py`.
+- **E3 newline branch.** `isShellOnlyPrompt` receives the raw prompt's line-break test before normalisation; the four-script contract test untouched.
+
+## File changes
+
+About 57-59 files: 19 new (6 source modules, 13 tests), about 32 changed source and test files, 6 docs; plus the version mirrors and the bundle outside the lanes. Exact ownership in `plan.md`.
+
+## Risks and open questions
+
+1. The plan id of a tree holding HTML, CSV or TSV changes; an in-flight resume checkpoint for such a tree no longer matches (disclosed).
+2. `.png` and the other named formats move from `unclassifiable` to `skipped_non_extractable`; a reader that summed `unclassifiable` sees fewer files there (disclosed).
+3. A re-ingest of an already-ingested CSV, TSV or HTML file rewrites its page once with the new section and keys (disclosed).
+4. Redaction of table cells is fail-safe by over-redaction on key-like headers; a credential that matches no pass stays in an indexed page, the same exposure the raw file already has in the vault, now also in embeddings (stated in docs/mcp.md).
+5. A summary page that carries a derived section is at most as visible as its source (review round): `deriveSourceSection` returns the source's own `visibility` tokens and `ingestSource` joins them with the page's kept `visibility`; a leading frontmatter block of the source is not handed to the extractor.
+6. Redaction residue of table cells (review round, kept as decided in T5): bare vendor tokens, base64 or high-entropy blobs, PEM blocks in quoted multi-line cells and credential-like values under headers that name no credential are not redacted; stated in docs/how-it-works.md.
+7. No lane edited a module of the OpenClaw bundle; the review round did (`redactor.ts` for linear private-region walking, `vault.ts` for a linear frontmatter opener), so `openclaw/index.js` is rebuilt and committed after the last `src/` change.
+
+## Follow-ups (review round)
+
+- **Chunker fence rule.** `readCodeBlock` (`src/core/search/chunker.ts`) closes a fence of four or more backticks on any line starting with three. A CommonMark-length-aware close (a run of the same character at least as long as the opening one, the rule `src/core/tags.ts` already implements) is deferred: it changes chunk boundaries and needs a `CHUNKER_VERSION` bump that re-chunks every vault. Table notes no longer depend on it, since cells escape every backtick.
+- **`@osb` marker fence rule.** `discoverMarkersDetailed` (`src/core/brain/inline.ts`) recognises only three-backtick fences. The same length-aware rule is deferred because it changes marker parsing; latent today, since both walkers that act on markers skip `Brain/`.
+- **Format skips before the reach predicate** (accepted residual). `brain_ingest_batch_plan` names a format-skipped file in `skipped_non_extractable` before asking the reach predicate. Visibility is a frontmatter rule, so a real PDF or image is always readable; only a non-text file that starts with a frontmatter block, or an unreadable one, would be named where a reserved page is not.
+- **Single read in `deriveSourceSection`.** The section is derived from a second read of the source; a source that grows past the ceiling between the intake's read and this one now answers `source-not-local`. Threading the intake's bytes through is the follow-up that removes the second read.
+- **`oneLine` leaf module.** `src/core/brain/ingest/extract-html.ts` and `src/cli/brain/verbs/extract.ts` import `oneLine` from `src/core/architect/manifests.ts`, a Bun-dependent module (safe to load in the Node build, since the Bun call sits inside a function). Moving `oneLine` to a leaf module, re-exported from `manifests.ts`, removes the coupling of ingest to the architect; deferred.
+- **Frontmatter pattern, one copy.** The leading-block strip of `extract-source.ts` copied the vault's frontmatter pattern; it now imports `FRONTMATTER_RE` from `src/core/vault.ts`, as do the two link-graph readers that held their own copies. The pattern's opener takes horizontal whitespace only, so a run of blank lines after an unclosed `---` is read in linear time.
+- **Other `<private>` detectors (resolved in the review rounds).** `src/core/brain/session-recall.ts` and `src/core/brain/continuity/redaction.ts` matched the open tag with their own regular expressions; both now ask the redactor's span walker (`privateRegionTexts`), so one linear rule decides what opens a region.
+- **Repeated remote re-ingest of a reserved summary page** (accepted residual). A second remote brain_ingest_source of a source whose deterministic summary page is reserved still reveals that page's existence. Both calls answer created: true where the control answers true then false, and brain_search_by_source totals differ (2 against 3). A remote caller's summary write is discarded. Closing it means a caller-lane summary path below local reach (the R2S11 follow-up).
+- **Provenance as the summary id (R2S11, the full form of R3-6).** An entity page from an intake still names the reserved source path in its `## Sources` section (no content, no digest). The full fix reads the source's visibility once from `origin.bytes`, hands it to both the intake and the derivation, writes the provenance as the summary id rather than the source path, and teaches `traceReferences` to follow summary-id provenance so a local `brain_delete_by_source` still cascades to the entity pages. Deferred because it changes the cascade and the provenance shape together.
+- **`removed-tool-reference` cap before the reach filter** (count residual). `REMOVED_TOOL_MAX_WARNINGS` counts before `brain_doctor` filters findings by the caller's reach, so a reserved page can take one of the capped slots. The check runs without the caller's view; moving the cap after the filter needs the view threaded into it.
+- **Stale-dependency note over the whole Brain layer** (count residual, released code). The `uncertain` stale-dependency note of `brain_doctor` counts "N state(s) stopped being current" over every record and moves by one when a reserved record is retired; it names no record. Answering it at the caller's reach needs the stale-dependency collector to take the reach view.
+- **Shared ranking statistics** (residual, stated on the `brain_search` registry row). The bm25 corpus statistics and the diversity rerank run over the shared index, so scores can differ by reach while the returned paths do not.
diff --git a/docs/brainstorm/non-markdown-ingest/plan.md b/docs/brainstorm/non-markdown-ingest/plan.md
new file mode 100644
index 00000000..4909f57e
--- /dev/null
+++ b/docs/brainstorm/non-markdown-ingest/plan.md
@@ -0,0 +1,217 @@
+# Non-Markdown source ingest - implementation plan
+
+Design: `docs/brainstorm/non-markdown-ingest/design.md` (decision ids R*, I*, H*, T*, L*, E* below refer to its "Design decisions").
+
+## How to run this plan
+
+Five lanes with DISJOINT file ownership run concurrently in the one worktree. Lane A (shared substrate) commits A1-A3 first; lanes B-E start at once and wait only on the lane A task named in "Depends on". Lane A's last task A4 waits on B2 and C2, the two pure modules whose signatures this contract pins. A lane edits only the files listed for it under "File ownership". Every task is TDD: write the named test first, run it and see it fail for the stated reason, implement, see it pass, then `bunx oxfmt` and `bunx oxlint -c oxlint.json` on the lane's own files only, then ONE step `git commit --only <own paths> -m "<conventional message>"` (retry on `index.lock`; after five hook rejections caused by another lane's files, check your own files and commit with `--no-verify`). Gates per task: `bun test <the task's test files>` and `bun run typecheck`, Bun 1.4.0 (`~/.cache/osb-pr-prepare/bun-1.4.0/bun-linux-x64`). Tests use forward-slash vault-relative paths, never compare absolute paths (Windows CI), use `tests/helpers/platform.ts` guards for permission and FIFO rows, and take credential-shaped literals (cells, `href`s) from `fakeCredential()`; no identifier contains `secret` before `=` or `:`.
+
+CHANGELOG, the version bump (1.69.0) and the OpenClaw bundle belong to no lane. No lane edits a bundled module (`openclaw/index.js` carries `redactor.ts` and `vault.ts`; lane C only imports `redactor.ts`); the integrator rebuilds the bundle after the last `src/` change and runs `tests/openclaw/bundle-node.test.ts`.
+
+Dependencies:
+
+- A1 -> A2 -> A3 (one lane, in order); B2 and C2 -> A4.
+- A1 -> B1 -> B2; A4 -> B3 -> B4.
+- A1 -> C1 -> C2; A4 -> C3.
+- D1 -> D2 -> D3 -> D4 (no lane A dependency; waits on A1 only so the census edit lands first).
+- E1 and E2 need nothing; E3 (docs) depends on A4, B3, C3, D4 and E2 (it describes and verifies their shipped behaviour).
+
+## Cross-lane contract (pinned now; no lane renames or re-spells any of this)
+
+### `src/core/brain/ingest/source-formats.ts` (lane A, A1; Node-safe leaf, no `Bun.*`, imports only types)
+
+- `SOURCE_FORMAT = Object.freeze({ text: "text", csv: "csv", tsv: "tsv", html: "html", pdf: "pdf", docx: "docx", xlsx: "xlsx", pptx: "pptx", odt: "odt", ods: "ods", odp: "odp", epub: "epub", rtf: "rtf", image: "image" } as const)`, `type SourceFormat`, `SOURCE_FORMATS` (frozen, this order), `isSourceFormat(value: unknown): value is SourceFormat`. Census row `SOURCE_FORMAT`.
+- `SOURCE_EXTRACTOR = Object.freeze({ verbatim: "verbatim", html: "html", table: "table" } as const)`, `type SourceExtractor` (no list, no guard: not a reason vocabulary).
+- `interface SourceFormatSpec { readonly format: SourceFormat; readonly extractor: SourceExtractor | null; readonly extensions: ReadonlyArray<string> }`
+- `SOURCE_FORMAT_SPECS: ReadonlyArray<SourceFormatSpec>`, this order: text `verbatim` (`.md .markdown .txt .text .rst .org`), csv `table` (`.csv`), tsv `table` (`.tsv`), html `html` (`.html .htm`), pdf `null` (`.pdf`), docx `null` (`.docx`), xlsx `null` (`.xlsx`), pptx `null` (`.pptx`), odt `null` (`.odt`), ods `null` (`.ods`), odp `null` (`.odp`), epub `null` (`.epub`), rtf `null` (`.rtf`), image `null` (`.png .jpg .jpeg .gif .webp .bmp .tif .tiff .heic`).
+- `SOURCE_FORMAT_BY_EXTENSION: ReadonlyMap<string, SourceFormat>`; `sourceFormatOf(path: string): SourceFormat | null` (lowercased extension of the last path segment, `""` never matches); `sourceFormatSpec(format: SourceFormat): SourceFormatSpec`.
+- `DEFAULT_INGESTIBLE_EXTENSIONS: readonly string[]` = `.md .markdown .txt .text .rst .org .csv .tsv .html .htm` (derived, frozen); `batch-plan.ts` re-exports it under the same name.
+- `SOURCE_EXTRACT_SKIP_REASON = Object.freeze({ formatReadVerbatim: "format-read-verbatim", formatNotExtractable: "format-not-extractable", formatUnknown: "format-unknown", sourceNotLocal: "source-not-local", sourceTooLarge: "source-too-large", notARegularFile: "not-a-regular-file", notUtf8: "not-utf8" } as const)`, `type SourceExtractSkipReason`, `SOURCE_EXTRACT_SKIP_REASONS`, `isSourceExtractSkipReason`. Census row `SOURCE_EXTRACT_SKIP_REASON`.
+
+### `src/core/brain/ingest/extractable-gate.ts` and the planner (lane A, A1-A2)
+
+- `SKIPPED_PAGE_REASON.formatNotExtractable = "format-not-extractable"`, FIRST in `SKIPPED_PAGE_REASONS`; `SkippedPage.detail` = the `SourceFormat` token. `parseSkippedPageReason` unchanged.
+- `PlannedFile.format?: SourceFormat` (absent when `text`); wire key `format` after `status` in `serializeBatchPlan`; CLI text line `- <path> (<status>, <bytes>B, <format>)` only when present.
+- `skippedNonExtractable` = format skips sorted by path, then schema skips. Existing wire keys `skipped_non_extractable` and `skip_reason_counts` reused.
+- Fixtures `.png`/`.csv` in `batch-plan.test.ts:289-330` and `brain-batch-plan-ignore-warnings.test.ts:205-242` become `.bin`/`.dat`.
+
+### Shared helpers (lane A, A3)
+
+- `src/core/markdown-fence.ts`: `export function fenceFor(text: string): string` (moved verbatim from `provenance/capture-scope.ts:178-182`).
+- `src/core/brain/ingest/ingest.ts`: `export function readSourceBounded(absolute: string, maxBytes: number): { readonly text: string; readonly unread?: undefined } | { readonly text: null; readonly unread: SourceUnread }`, `SourceUnread` exported with members `"not a regular file"` and `"larger than the read limit"` (unchanged strings). Callers map them to `not-a-regular-file` and `source-too-large`.
+
+### `src/core/brain/ingest/extract-html.ts` (lane B, B1-B2; pure, no I/O)
+
+- `HTML_EXTRACT_MAX_SOURCE_BYTES = SOURCE_HASH_MAX_BYTES`, `HTML_PARTS_MAX = 256`, `HTML_HEADING_MAX_CHARS = 200`.
+- `interface HtmlPart { readonly index: number; readonly level: number; readonly heading: string; readonly trail: string; readonly lineStart: number; readonly lineEnd: number; readonly sourceOffset: number }`
+- `interface HtmlExtraction { readonly extracted: true; readonly title: string | null; readonly text: string; readonly parts: ReadonlyArray<HtmlPart>; readonly partsOmitted: number }`
+- `type HtmlExtractResult = HtmlExtraction | { readonly extracted: false; readonly reason: typeof SOURCE_EXTRACT_SKIP_REASON.notUtf8 }`
+- `extractHtml(bytes: Uint8Array): HtmlExtractResult`
+- `renderPartsSection(extraction: HtmlExtraction): string` - `""` when there are no parts, else `## Parts`, blank line, one fence by `fenceFor` with info string `parts`, one line per part: `h<level> <trail> | lines <a>-<b>`, level 0 as `preamble | lines <a>-<b>`.
+
+### `src/core/brain/ingest/table-note.ts` (lane C, C1-C2; pure, no I/O)
+
+- Constants: `TABLE_NOTE_MAX_ROWS = 1_000`, `TABLE_NOTE_MAX_COLUMNS = 64`, `TABLE_NOTE_MAX_CELL_CHARS = 256`, `TABLE_NOTE_MAX_BYTES = 262_144`, `TABLE_NOTE_ROWS_PER_GROUP = 50`, `TABLE_NOTE_GROUP_MAX_TOKENS = 600`.
+- `TABLE_DELIMITER = Object.freeze({ comma: "comma", semicolon: "semicolon", tab: "tab" } as const)`, `type TableDelimiter`.
+- `TABLE_TRUNCATION = Object.freeze({ rows: "rows", columns: "columns", cells: "cells", bytes: "bytes" } as const)`, `TableTruncation`, `TABLE_TRUNCATIONS`, `isTableTruncation`.
+- `TABLE_NOTE_SKIP_REASON = Object.freeze({ sourceNotLocal: "source-not-local", notUtf8: "not-utf8", containsNul: "contains-nul", malformedQuoting: "malformed-quoting", empty: "empty" } as const)`, `TableNoteSkipReason`, `TABLE_NOTE_SKIP_REASONS`, `isTableNoteSkipReason`.
+- `type TableFormat = typeof SOURCE_FORMAT.csv | typeof SOURCE_FORMAT.tsv`
+- `interface TableNoteRendered { readonly rendered: true; readonly format: TableFormat; readonly delimiter: TableDelimiter; readonly columns: number; readonly rows: number; readonly rowsRendered: number; readonly truncated: ReadonlyArray<TableTruncation>; readonly redactedCells: number; readonly section: string }`
+- `interface TableNoteSkipped { readonly rendered: false; readonly format: TableFormat; readonly reason: TableNoteSkipReason; readonly detail?: string }`; `type TableNoteResult = TableNoteRendered | TableNoteSkipped`.
+- `tableNote(path: string, bytes: Uint8Array): TableNoteResult` (throws `TypeError` naming the path when its format is not csv or tsv: a caller error).
+- `tableNoteFrontmatter(result: TableNoteRendered): FrontmatterMap` -> `table_delimiter`, `table_columns`, `table_rows`, `table_rows_rendered`, `table_truncated` (absent when empty), in this order.
+- Section: `## Table`, then per group `### Rows <a>-<b>` (1-based data rows) and one fence by `fenceFor` with info string `table`, header record first, cells joined by ` | `, escapes per T1; last line `Rendered <n> of <m> rows.` only when `rowsRendered < rows`.
+- `src/core/search/chunker.ts` (lane C): `export const DEFAULT_CHUNK_MAX_TOKENS = 800` (the existing `DEFAULT_MAX_TOKENS` renamed and exported) and `export function countChunkTokens(text: string): number` (the existing `countTokens`, exported under this name). `CHUNKER_VERSION` untouched.
+
+### `src/core/brain/ingest/extract-source.ts` (lane A, A4)
+
+- `type SourceExtraction = { readonly extractor: typeof SOURCE_EXTRACTOR.html; readonly format: SourceFormat; readonly html: HtmlExtractResult } | { readonly extractor: typeof SOURCE_EXTRACTOR.table; readonly format: SourceFormat; readonly table: TableNoteResult } | { readonly extractor: null | typeof SOURCE_EXTRACTOR.verbatim; readonly format: SourceFormat | null; readonly reason: SourceExtractSkipReason }`
+- `extractSource(path: string, bytes: Uint8Array): SourceExtraction`; an unregistered extension answers `{extractor: null, format: null, reason: "format-unknown"}` before the switch. The switch arms, exactly:
+ - `case null: return { extractor: null, format: spec.format, reason: SOURCE_EXTRACT_SKIP_REASON.formatNotExtractable };`
+ - `case SOURCE_EXTRACTOR.verbatim: return { extractor: spec.extractor, format: spec.format, reason: SOURCE_EXTRACT_SKIP_REASON.formatReadVerbatim };`
+ - `case SOURCE_EXTRACTOR.html: return { extractor: spec.extractor, format: spec.format, html: extractHtml(bytes) };`
+ - `case SOURCE_EXTRACTOR.table: return { extractor: spec.extractor, format: spec.format, table: tableNote(path, bytes) };`
+- `type PartsOutcome = { readonly extracted: true; readonly count: number; readonly omitted?: number } | { readonly extracted: false; readonly reason: SourceExtractSkipReason }`
+- `type TableOutcome = Omit<TableNoteRendered, "section"> | TableNoteSkipped`
+- `interface SourceDerivation { readonly format: SourceFormat; readonly frontmatter: FrontmatterMap; readonly section: string; readonly parts?: PartsOutcome; readonly table?: TableOutcome }`
+- `deriveSourceSection(vault: string, source: string, trust: SourceTrust, readable?: (rel: string) => boolean): SourceDerivation | undefined` - `undefined` unless the format's extractor is `html` or `table` (text, unknown and named-unsupported sources: byte-identical page). Order: format, trusted lane and vault-relative file, `isSourceHidden`, `readSourceOrigin`. Not local, absent or hidden: `section ""`, `frontmatter {}`, outcome reason `source-not-local`. Read: `frontmatter` = `source_format`, then `sourceContentHashFrontmatter(contentHash)`, then the table keys when rendered.
+
+### Ingest result and wire (lane A, A4)
+
+- `IngestSourceResult.parts?: PartsOutcome`, `IngestSourceResult.table?: TableOutcome`.
+- `brain_ingest_source` result keys, after `capture_scope` and before `pre_extract`: `parts: {extracted, count, omitted?} | {extracted: false, reason}` and `table: {rendered: true, format, delimiter, columns, rows, rows_rendered, truncated?, redacted_cells} | {rendered: false, format, reason, detail?}`.
+- Frontmatter keys on derived pages: `source_format`, `source_content_hash`, then `table_delimiter`, `table_columns`, `table_rows`, `table_rows_rendered`, `table_truncated`. Body section order: summary, provenance, `## Entities`, `## Connections to existing notes`, then `## Parts` or `## Table`.
+- `registry-guard.ts:352`: `brain_ingest_source: "write; returns the summary path, bounded id lists and derived-section counts"`.
+
+### CLI `o2b brain extract <file> [--json]` (lane B, B3)
+
+- Manifest entry `extract` (flags `--json`), dispatch case `"extract"` in `src/cli/brain.ts`, export `runBrainExtract` from `verbs/index.ts`.
+- Reads `readSourceBounded(file, HTML_EXTRACT_MAX_SOURCE_BYTES)`; `unread` maps to `not-a-regular-file` or `source-too-large`.
+- JSON: html `{path, extracted: true, format: "html", title, text, parts: [{index, level, heading, trail, line_start, line_end, source_offset}], parts_omitted?}`; csv/tsv rendered `{path, extracted: true, format, delimiter, columns, rows, rows_rendered, truncated?, redacted_cells, section}`; anything else `{path, extracted: false, format, reason, detail?}` (`format` is `null` for an unknown extension). Exit code 0 in every case: "could not extract" is data.
+- Text: `extract: <path> (<format>)` then the title line, `<n> part(s)` with one `h<level> <trail> | lines <a>-<b>` line each, or the table counts line and the section, or `not extracted: <reason>`.
+
+### Reach (lane D)
+
+- `indexRootCoverage(config: ResolvedSearchConfig, roots: ReadonlyArray<string>, admit?: (path: string, indexedTags: ReadonlyArray<string>) => boolean): Promise<IndexRootCoverage>`; `probeRetrievalCorpus(resolveConfig, reach?: TransportReach)`.
+- Brief: rows dropped below local reach are filtered with `readerRefView`; no new wire key, no error code.
+- `RECALL_GATE_INPUT_SCHEMA.properties.turn_id = { type: "string", maxLength: 512, description: "Optional turn correlation id recorded on the telemetry record." }` (the `search-tools.ts:263` text); `GateTelemetryInput.turnId?: string`, payload key `CONTINUITY_TURN_ID_KEY`.
+
+### Census and docs headings
+
+- `tests/core/architecture/verdict-vocabulary-census.test.ts` (lane A): rows `SOURCE_FORMAT`, `SOURCE_EXTRACT_SKIP_REASON` (A1), `TABLE_NOTE_SKIP_REASON`, `TABLE_TRUNCATION` (A4). Lane C's own gates skip the census suite until A4.
+- Docs (lane E): `docs/mcp.md` bullet "Since v1.69.0"; `docs/updating.md` section "Upgrading to 1.69.0"; `docs/cli-reference.md` lines for `extract` and `batch-plan`; README release paragraph. Public wording "answers at the caller's reach"; never security wording.
+
+## File ownership (every file this plan touches, exactly one owner)
+
+| Lane | Files |
+|---|---|
+| A | `src/core/brain/ingest/source-formats.ts` (new), `tests/core/brain/ingest/source-formats.test.ts` (new), `src/core/brain/ingest/extractable-gate.ts`, `tests/core/brain/ingest/extractable-gate.test.ts`, `tests/core/architecture/verdict-vocabulary-census.test.ts`, `src/core/brain/ingest/batch-plan.ts`, `tests/core/brain/ingest/batch-plan.test.ts`, `tests/core/brain/ingest/batch-plan-extractable.test.ts`, `src/cli/brain/verbs/batch-plan.ts`, `tests/cli/brain-batch-plan-ignore-warnings.test.ts`, `tests/mcp/ingest-tool-ignore-warnings.test.ts`, `tests/mcp/ingest-plan-format-reach.test.ts` (new), `src/core/markdown-fence.ts` (new), `src/core/brain/provenance/capture-scope.ts`, `src/core/brain/ingest/ingest.ts`, `tests/core/brain/ingest/ingest.test.ts`, `src/core/brain/ingest/extract-source.ts` (new), `tests/core/brain/ingest/extract-source.test.ts` (new), `tests/core/brain/ingest/ingest-derived-section.test.ts` (new), `src/mcp/brain/ingest-tools.ts`, `src/mcp/registry-guard.ts` |
+| B | `src/core/brain/ingest/extract-html.ts` (new), `tests/core/brain/ingest/extract-html.test.ts` (new), `src/cli/brain/verbs/extract.ts` (new), `src/cli/brain/verbs/index.ts`, `src/cli/brain.ts`, `src/cli/command-manifest.ts`, `tests/cli/brain-extract.test.ts` (new), `tests/mcp/ingest-parts-reach.test.ts` (new) |
+| C | `src/core/brain/ingest/table-note.ts` (new), `tests/core/brain/ingest/table-note.test.ts` (new), `src/core/search/chunker.ts`, `tests/core/brain/ingest/table-note-chunks.test.ts` (new), `tests/mcp/ingest-tool-table.test.ts` (new) |
+| D | `src/core/search/indexer.ts`, `src/core/search/pipeline/outcome.ts`, `src/mcp/search-tools.ts`, `src/core/search/visibility-surface-registry.ts`, `tests/core/search/index-root-coverage.test.ts` (new), `tests/mcp/derived-digest-search-reach.test.ts`, `src/mcp/brain/brief-tools.ts`, `tests/mcp/brief-ids-reach.test.ts` (new), at most two existing brief tests that need `{ reach: TRANSPORT_REACH.local }` (named in the commit body), `src/core/brain/gate-telemetry.ts`, `plugins/hermes/_schemas.py`, `tests/python/test_static_schemas.py`, `tests/mcp/recall-gate-turn-id.test.ts` (new) |
+| E | `src/core/brain/architect/generate.ts`, `tests/core/brain/architect-dependencies.test.ts`, `src/core/search/surfacing-gate.ts`, `tests/core/search/surfacing-gate.test.ts`, `docs/mcp.md`, `docs/cli-reference.md`, `docs/how-it-works.md`, `docs/updating.md`, `docs/observability.md`, `README.md` |
+| none | `CHANGELOG.md`, `package.json` and the version mirrors (PR preparation); `openclaw/index.js` (integrator) |
+
+Expected size: about 57-59 changed files (19 new).
+
+## Tasks
+
+### Lane A - shared substrate (A1-A3 commit first)
+
+### Task A1: format registry, extract skip reasons, the new gate member
+- **Files**: `src/core/brain/ingest/source-formats.ts`, `tests/core/brain/ingest/source-formats.test.ts`, `src/core/brain/ingest/extractable-gate.ts`, `tests/core/brain/ingest/extractable-gate.test.ts`, `tests/core/architecture/verdict-vocabulary-census.test.ts`
+- **Acceptance**: the test pins the spec order, every extension's format, `sourceFormatOf` (case-insensitive, URL and extensionless paths give `null`), `DEFAULT_INGESTIBLE_EXTENSIONS` order, and that the module has no `Bun.` reference; `SKIPPED_PAGE_REASONS[0]` is `format-not-extractable`; the legacy parser still maps only to `schema-type-not-extractable`; census passes with both new rows.
+- **Depends on**: none
+
+### Task A2: planner routing, planned `format`, fixtures, planner A/B
+- **Files**: `src/core/brain/ingest/batch-plan.ts`, `tests/core/brain/ingest/batch-plan.test.ts`, `tests/core/brain/ingest/batch-plan-extractable.test.ts`, `src/cli/brain/verbs/batch-plan.ts`, `tests/cli/brain-batch-plan-ignore-warnings.test.ts`, `tests/mcp/ingest-tool-ignore-warnings.test.ts`, `tests/mcp/ingest-plan-format-reach.test.ts`
+- **Acceptance**: fixtures moved to `.bin`/`.dat` with unchanged totals; a tree with HTML, CSV, PDF, PNG and `.bin` plans the HTML and CSV with `format`, skips PDF and PNG as `format-not-extractable` (`detail` pdf, image) before schema skips, counts `.bin` unclassifiable; an `extensions` override that lists `.pdf` plans it; a Markdown-only tree serializes byte-identically; through the real MCPServer at remote reach a vault with a hidden Markdown page and one without give identical format-skip lists; the CLI text matches `cli-output/expected-output.md`.
+- **Depends on**: A1
+
+### Task A3: bounded reader grows on demand, shared fence helper
+- **Files**: `src/core/brain/ingest/ingest.ts`, `tests/core/brain/ingest/ingest.test.ts`, `src/core/markdown-fence.ts`, `src/core/brain/provenance/capture-scope.ts`
+- **Acceptance**: two `fstatReportsSize` spy rows (grows after the fstat but stays under the cap: read whole; grows past the cap: refused `larger than the read limit`); the over-cap and FIFO rows unchanged; the excerpt tests pass unchanged with the moved `fenceFor`.
+- **Depends on**: A2
+
+### Task A4: dispatch, derived section on the page, wire fields
+- **Files**: `src/core/brain/ingest/extract-source.ts`, `tests/core/brain/ingest/extract-source.test.ts`, `src/core/brain/ingest/ingest.ts`, `tests/core/brain/ingest/ingest-derived-section.test.ts`, `src/mcp/brain/ingest-tools.ts`, `src/mcp/registry-guard.ts`, `tests/core/architecture/verdict-vocabulary-census.test.ts`
+- **Acceptance**: `extractSource` returns each arm for `.md`, `.html`, `.csv`, `.pdf`, `.xyz`; an HTML and a CSV source get `source_format`, `source_content_hash` equal to `hashBytes` of the file, and their section last; a `.md` source page is byte-identical to before; a URL `.csv`, an absent and a hidden `.csv` all give `table.reason` `source-not-local`, no digest, no section; an operator `visibility` survives a re-ingest; census passes with the two table rows; `brain_ingest_source` description length unchanged.
+- **Depends on**: A3, B2, C2
+
+### Lane B - HTML
+
+### Task B1: linear scanner and text
+- **Files**: `src/core/brain/ingest/extract-html.ts`, `tests/core/brain/ingest/extract-html.test.ts`
+- **Acceptance**: entities (numeric, invalid code point, the six named, an unknown named kept), raw-text elements skipped, `<private>` placeholder, void elements, document-level text, `pre` whitespace, CRLF equals LF output, no attribute value emitted (a `fakeCredential()` `href`), title, malformed and unclosed tags, invalid UTF-8 refused as `not-utf8`, linear time on a 1 MiB adversarial input.
+- **Depends on**: A1
+
+### Task B2: parts, caps, `## Parts` section
+- **Files**: `src/core/brain/ingest/extract-html.ts`, `tests/core/brain/ingest/extract-html.test.ts`
+- **Acceptance**: levels, trail, 1-based line spans, `sourceOffset` of the start tag, preamble only when non-empty, 257 headings give 256 parts and `partsOmitted: 1`, a 300-character heading capped at 200, `renderPartsSection` output pinned, `[[x]]` and `#tag` in a heading yield no link or tag through `stripCode`.
+- **Depends on**: B1
+
+### Task B3: `o2b brain extract`
+- **Files**: `src/cli/brain/verbs/extract.ts`, `src/cli/brain/verbs/index.ts`, `src/cli/brain.ts`, `src/cli/command-manifest.ts`, `tests/cli/brain-extract.test.ts`
+- **Acceptance**: text and `--json` for an HTML, a CSV, a `.md`, a `.pdf` and an unknown extension match `cli-output/expected-output.md`; a FIFO answers `not-a-regular-file` (POSIX only); manifest-completeness and help-surface-parity suites pass.
+- **Depends on**: A4
+
+### Task B4: parts through the real MCPServer, reach A/B
+- **Files**: `tests/mcp/ingest-parts-reach.test.ts`
+- **Acceptance**: `brain_ingest_source` on an HTML vault file returns `parts {extracted: true, count}` and the page holds `## Parts`; at remote reach a vault with an owner-scoped HTML-named Markdown source and one without give identical normalised results and pages; a `.md` source result has no `parts` key.
+- **Depends on**: B3
+
+### Lane C - tables
+
+### Task C1: CSV and TSV parsing, delimiter, header
+- **Files**: `src/core/brain/ingest/table-note.ts`, `tests/core/brain/ingest/table-note.test.ts`
+- **Acceptance**: CRLF, LF, lone CR, BOM, quoted delimiters and line breaks, `""`, bare quotes, an unterminated quote refused `malformed-quoting` with the record number, ragged rows, a semicolon export, a TSV with quotes kept, `contains-nul`, `not-utf8`, `empty`.
+- **Depends on**: A1
+
+### Task C2: render, caps, redaction, chunk-sized groups
+- **Files**: `src/core/brain/ingest/table-note.ts`, `tests/core/brain/ingest/table-note.test.ts`, `src/core/search/chunker.ts`, `tests/core/brain/ingest/table-note-chunks.test.ts`
+- **Acceptance**: every cap named in `truncated`; groups of at most 50 rows and 600 tokens; cell escapes; a fence longer than a backtick run in a cell; a `fakeCredential()` value under a key-like header and a URL with userinfo redacted and counted; `keyword` over-redaction pinned; `chunkMarkdown` over a rendered 400-row, 15-column page gives chunks that each carry the header line and a `Table > Rows a-b` heading path; determinism.
+- **Depends on**: C1
+
+### Task C3: table through the real MCPServer, reach A/B
+- **Files**: `tests/mcp/ingest-tool-table.test.ts`
+- **Acceptance**: the `table` wire field and the five frontmatter keys; an unchanged re-ingest is inert, a changed CSV rewrites the page; at remote reach a vault with a hidden `.csv`-named source and one without give identical normalised results and pages (`source-not-local`, no digest); a text source has no `table` key.
+- **Depends on**: A4
+
+### Lane D - reach left-overs and the gate argument
+
+### Task D1: root coverage admits only readable paths
+- **Files**: `src/core/search/indexer.ts`, `tests/core/search/index-root-coverage.test.ts`
+- **Acceptance**: an `admit` that rejects every path leaves the root without documents; no `admit` is byte-identical to today.
+- **Depends on**: A1
+
+### Task D2: reach threaded into the corpus probe, registry, A/B
+- **Files**: `src/core/search/pipeline/outcome.ts`, `src/mcp/search-tools.ts`, `src/core/search/visibility-surface-registry.ts`, `tests/mcp/derived-digest-search-reach.test.ts`
+- **Acceptance**: a root holding only a reserved page and the same root empty give identical remote `brain_search` and `brain_recall_gate` verdicts; local stays `not_found`; the residual sentences leave both registry rows; visibility census passes.
+- **Depends on**: D1
+
+### Task D3: daily and weekly brief ids at reach
+- **Files**: `src/mcp/brain/brief-tools.ts`, `tests/mcp/brief-ids-reach.test.ts`, `src/core/search/visibility-surface-registry.ts`, at most two existing brief tests
+- **Acceptance**: with apply evidence inside BOTH the 24-hour and 30-day windows and a reserved preference confirmed and retired in the window, the two-vault remote A/B gives identical normalised daily and weekly envelopes and no delta snapshot; local still names the id and captures the delta; the `brain_brief` row states the remaining count residual.
+- **Depends on**: D2
+
+### Task D4: `turn_id` on the recall gate
+- **Files**: `src/mcp/search-tools.ts`, `src/core/brain/gate-telemetry.ts`, `tests/mcp/recall-gate-turn-id.test.ts`, `plugins/hermes/_schemas.py`, `tests/python/test_static_schemas.py`
+- **Acceptance**: the real MCPServer accepts `turn_id` and records it on `gate_telemetry` with telemetry on; 513 characters refused; the vendored schema equals the live projection; the Python suite drives `prefetch(..., turn_id=...)` through the conformance check.
+- **Depends on**: D3
+
+### Lane E - small fixes and docs
+
+### Task E1: architect names the edges it cannot draw
+- **Files**: `src/core/brain/architect/generate.ts`, `tests/core/brain/architect-dependencies.test.ts`
+- **Acceptance**: only-unlinkable fixture prints the new fixed sentence plus code-span edges on the overview and `Not linked:` on the module note; the mixed fixture gains one line naming the dropped edge; the no-edge sentence tests at :137 and :155 unchanged; second run `unchanged`.
+- **Depends on**: none
+
+### Task E2: newline test on the raw prompt
+- **Files**: `src/core/search/surfacing-gate.ts`, `tests/core/search/surfacing-gate.test.ts`
+- **Acceptance**: `"git status\nwhy does this fail"` is retrieved; `"git status"` and `"$ ls -la"` still skipped as `shell_command`; the single-line `"git push keeps failing after the rebase"` stays skipped (pinned, the documented residue); the four-script contract test unchanged.
+- **Depends on**: none
+
+### Task E3: reference docs
+- **Files**: `docs/mcp.md`, `docs/cli-reference.md`, `docs/how-it-works.md`, `docs/updating.md`, `docs/observability.md`, `README.md`
+- **Acceptance**: "Since v1.69.0" bullet (formats, the header rule, caps, frontmatter keys, `parts` and `table`, redaction, the `turn_id` argument row); cli-reference lines for `extract` and `batch-plan`; how-it-works ingest paragraph and the architect clause; "Upgrading to 1.69.0" (CSV, TSV and HTML planned by default, re-ingest adds the section, `.png` and friends move from `unclassifiable` to format skips, plan ids of trees holding these formats change, `visibility` now kept on re-ingest, multi-line prompts starting with a command name admitted); observability row; README paragraph; every statement checked against the shipped code; no card ids, no exclamation marks, "Open Second Brain" in full.
+- **Depends on**: A4, B3, C3, D4, E2
diff --git a/docs/brainstorm/non-markdown-ingest/variants.md b/docs/brainstorm/non-markdown-ingest/variants.md
new file mode 100644
index 00000000..13ba9469
--- /dev/null
+++ b/docs/brainstorm/non-markdown-ingest/variants.md
@@ -0,0 +1,38 @@
+# Non-Markdown source ingest - variants and decision
+
+Consultant: the primary CLI consultant (`claude -p`, prompt in `cli-output/prompt.md`, verbatim answer in `cli-output/claude.md`). Exit 0, three parseable variants; the fallback consultant was not run.
+
+## Variant 1: A second, text-structure lane of the opt-in pre-extract pass - lost (consultant's recommendation, overridden)
+
+Extend `runPreExtract` and the `pre_extract` flag so an HTML file returns text and heading parts (through `HTMLRewriter`) and a CSV returns a header, dimensions and a row sample; the agent decides what reaches the page.
+
+Why it lost: card B's deliverable is rows that search and recall can find, and search indexes only the summary page, so a sample the agent may or may not copy leaves the gap open. It also overloads a pass named for code structure, and `HTMLRewriter` is Bun-only, leaves entities undecoded, misses document-level text and throws on `onEndTag` for void elements.
+
+## Variant 2: A source-reading kernel stage that renders derived sections on the page - chosen, adjusted
+
+Classify every file source against one closed format vocabulary, write a derived section onto the summary page for formats that have an extractor, and name every other format.
+
+Adjustments by the orchestrator, with evidence in `design.md`:
+
+1. HTML writes headings and line spans only (`## Parts`), never body text: the consultant's own line for HTML holds.
+2. The chunker's header-loss objection is answered without changing chunking rules: row groups are bounded by 50 rows AND 600 tokens counted by the chunker's own counter, each repeating the header under a `Rows a-b` heading (design T2).
+3. The redaction objection is answered by two named passes with fail-safe over-redaction pinned by a test (T5), and by keeping operator `visibility` on rewrite (I5).
+4. The derived read happens only for an in-vault file in the trusted lane, after the reach check, so "every ingest pays the read" does not hold: text sources pay nothing and stay byte-identical (I3, I4).
+5. Named formats are a `SKIPPED_PAGE_REASON` member with the format as `detail`, not a separate `unsupported` map beside `unclassifiable` (R3).
+
+## Variant 3: Planner-only honesty plus a CLI-only reader verb - lost
+
+Teach the planner the format vocabulary and add a CLI verb that previews HTML and CSV structure; leave ingest untouched.
+
+Why it lost: an MCP session, the primary surface, gains nothing, and the CSV rows stay out of the index. Its CLI verb survives inside the chosen variant as `o2b brain extract`.
+
+## Alternatives considered inside the chosen variant
+
+- **HTMLRewriter vs the linear scanner**: the scanner won. Entity decoding is project code either way, the two silent-loss or abort traps above need workarounds, and a Bun-only module cannot be reused from the Node-bundled build.
+- **GFM table vs fenced plain text**: fenced text won. A GFM table turns `[[x]]` and `#word` cells into graph facts from untrusted data and loses its header in every chunk after the first; inside a fence `stripCode` keeps links and tags out while FTS still indexes the cells.
+- **Opt-in flag vs automatic by format**: automatic won. The derived section is the page content the cards ask for, the `brain_ingest_source` description has one character left for documenting a flag, and text sources stay byte-identical by construction.
+- **New MCP tool (`brain_extract_source`) vs CLI verb**: the CLI verb won. A tool is a new capability with a tool-count move, parity rows, a probe-catalogue entry and docs, for an ingest model in which the agent already holds the source.
+- **External extractor command gate now vs later**: later. It is a new subprocess capability reachable from MCP that needs the exec audit discipline and an apply gate; the per-file `format-not-extractable` entries are the hook it will claim.
+- **Lanes B and C adding their own switch arms vs lane A writing all four**: lane A writes all four in A4. An exhaustive switch over a union that already holds `html` and `table` does not type-check without those arms, and a placeholder arm would be a stub.
+- **Fixed 50-row groups vs token-bounded groups**: token-bounded. A fixed 50-row group of a wide table spans several chunks, and every chunk after the first loses the header.
+- **Whole-layer count aggregates of the status and brief views now vs later**: later, as their own change; the id slice of the daily and weekly views ships now.
diff --git a/docs/cli-reference.md b/docs/cli-reference.md
index c2eb5a2e..e6120181 100644
--- a/docs/cli-reference.md
+++ b/docs/cli-reference.md
@@ -1189,6 +1189,65 @@ unquoted. `--json` adds `capture_scope` (always: `full-local`,
`unpaired`, `findings` (each `{ claim, outcome, span }`, capped at 25 with
`total`, `returned` and `truncated`).
+### Non-Markdown sources (since v1.69.0)
+
+```text
+o2b brain extract <file> [--json] - read-only preview of what ingest derives from a file, chosen by its extension; writes nothing
+o2b brain batch-plan plans .csv .tsv .html .htm by default; a planned file of a non-text format shows it: "- <path> (<status>, <bytes>B, <format>)"; PDF, Office, EPUB, RTF and image files are listed as format-not-extractable with their format instead of counted as unclassifiable
+```
+
+`o2b brain extract` reads one file (a path on disk, inside a vault or
+not) up to 8 MiB and prints what `brain_ingest_source` would derive from
+it. For HTML it prints the title and one line per part:
+
+```text
+extract: <file> (html)
+ title: Release notes
+ 2 part(s):
+ h1 Overview | lines 1-2
+ h2 Overview > Install | lines 3-4
+```
+
+`--json` adds the extracted `text` and each part's `index`, `level`,
+`heading`, `trail`, `line_start`, `line_end` and `source_offset` (the
+byte offset of its start tag in the source file), plus `parts_omitted` when more than 256
+headings were found. A leading frontmatter block in the file is left
+out, as ingest leaves it out. Each heading and the title are redacted
+(`key=value` credentials and URL userinfo) over their first 4,096 code
+units before they are capped at 200 characters (a longer one ends in
+`…`), and in the parts list a backslash in a heading is written as `\\`
+and a `|` as `\|`.
+The `text` has every `<private>` region replaced by its placeholder and
+is otherwise not redacted; it is a preview of a file the caller already
+reads, and it never reaches a page. For CSV and TSV it prints the delimiter and the
+counts, then the `## Table` section the summary page would hold:
+
+````text
+extract: <file> (csv)
+ comma-delimited, 2 column(s), 2 row(s), 2 rendered, 0 redacted cell(s)
+## Table
+
+### Rows 1-2
+
+```table
+name | qty
+bolt | 4
+nut | 7
+```
+````
+
+Any other file prints `not extracted: <reason>`, and `--json` returns
+`{ok: true, path, extracted: false, format, reason, detail?}`:
+`format-read-verbatim`
+for Markdown and plain text, `format-not-extractable` for a named format
+such as PDF, `format-unknown` (with `format: null`) for an extension the
+registry does not know (a named or unknown format is answered from the
+extension, without reading the file), and `not-a-regular-file`, `source-too-large`,
+`not-utf8` or a table refusal for a file that could not be read or
+parsed. The exit code is `0` in every one of these cases: a file that
+cannot be extracted is a result, not an error. A path that does not
+exist or is a symbolic link is an error (exit code `1`).
+
### Knowledge packs
A knowledge pack is a selected subset of Brain knowledge - rules and the
diff --git a/docs/how-it-works.md b/docs/how-it-works.md
index 29f8ae20..f3942166 100644
--- a/docs/how-it-works.md
+++ b/docs/how-it-works.md
@@ -915,6 +915,12 @@ with a registered next command, `src/core/brain/doctor-exits.ts` for those
without) and is CI-enforced by `tests/core/brain/doctor-exit-census.test.ts`,
which fails on a code that appears in neither table.
+Since v1.69.0 the `orphan-evidence` and `malformed-evidence-range`
+findings carry `sources` (the preference the evidence is about) and
+`removed-tool-reference` carries `path` (vault-relative on the MCP
+wire), so every finding names its record structurally and `brain_doctor`
+leaves out a finding about a record the caller cannot read at its reach.
+
With `--strict`, warnings demote `ok` to `false` so CI can gate on
hygiene. `brain_doctor` itself stays read-only — auto-modifying state on
a plain doctor run would break the "explicit-driven" invariant. The
@@ -1351,6 +1357,14 @@ containment only. On an existing overview or module note the new
regions are appended at the end of the note, once, so the first run
after an upgrade reports those notes `updated`.
+Since v1.69.0, when modules declare edges but every one of them touches
+such a module, the `module-dependencies` region says so in a fixed
+sentence and lists the dropped edges as code spans instead of claiming
+that no module depends on another; with some edges drawn, one line under
+the diagram names the dropped ones. A module note names the modules it
+depends on but cannot link on a `Not linked:` line, and its
+`depends_on` key still holds only links.
+
## The agent write contract (since v0.41.0)
The Brain core stays deterministic - no LLM ever runs inside it. The write-session protocol is how that rule survives contact with agents that need to PROPOSE structured artifacts: schema-typed notes, handoffs, curated summaries, panel deliberations.
@@ -1485,6 +1499,87 @@ pages are skipped: they already carry the `untrusted_source` marker, which
names the same condition (and which the retrieval trust gate excludes when
`search_trust_gate_enabled` is on).
+## Non-Markdown sources (since v1.69.0)
+
+Ingest used to be text only: a file that was not Markdown or plain text
+was counted per extension and set aside. One format registry now maps a
+file's extension to a format and each format to an extractor. Markdown
+and the lightweight markups are read as they are; CSV, TSV and HTML are
+extracted; PDF, the Office formats, EPUB, RTF and images are named but
+not extracted, so the ingest plan lists each such file with reason
+`format-not-extractable` and its format, the hook a caller that runs
+its own converter or OCR routes on. Open Second Brain adds no parser
+dependency and runs no external program for this.
+
+The calling agent still writes the summary of a source. What the kernel
+adds is structure it can derive without judgement, written onto the
+summary page, because the summary page is what search indexes: the
+source file itself is not a Markdown page. `brain_ingest_source` does
+this by format, for a file in the vault, in the trusted lane, after
+the same reach check every reader applies, and reads the bytes once to
+derive the section and its `source_content_hash`.
+
+For HTML, a linear scanner (no regular expression over unbounded input)
+reduces the page to text: comments, scripts, styles and embedded SVG or
+MathML are dropped, entities are decoded, a `<private>` region becomes
+its placeholder and no attribute value is ever kept. The headings become
+parts, each with its level, its trail of ancestor headings and its line
+span in the extracted text. Only the parts reach the page, under
+`## Parts` inside one fenced block, so a heading that holds `[[...]]` or
+`#word` adds no link and no tag while full-text search still finds it.
+Each heading and the title are redacted (`key=value` credentials and URL
+userinfo) before they are capped, over their first 4,096 code units (a
+longer one ends in `…`), and in the parts list a backslash is written
+as `\\` and a `|` as `\|`, so a heading cannot forge a line span.
+The text itself stays out of the page; `o2b brain extract --json`
+returns it.
+
+For CSV and TSV, the first record is the header. Quoted fields follow
+RFC 4180 leniently, a semicolon export is recognised from its first
+record (at least two fields on semicolons and more than on commas, so a
+header such as `name;"price, eur";qty` reads as three columns), and an
+unterminated quote is refused by name with the record number. A leading
+frontmatter block in the source is skipped for HTML and tables alike, so
+it is never read as data. The rows are rendered as fenced plain text,
+not a Markdown table, in groups that each repeat the header and stay
+under one search chunk, so every chunk of the page carries the column
+names and a `Table > Rows a-b` heading path. Rows, columns, cell length
+and section size are capped, and every cut is named; only the first
+4,096 code units of a cell are read for redaction, and a longer cell
+counts as cut. A `<private>` region is hidden before the records are
+split, so a region spanning records hides every row between its tags. A column whose header names a
+credential has its values replaced, and every other cell goes through
+the output redactor. Inside a cell a backslash, `|`, a backtick, a line
+feed, a carriage return and a tab are escaped (`\\`, `\|`, ``\` ``,
+`\n`, `\r`, `\t`), and every other control character (C0, DEL and C1)
+is written as `\u{XXXX}` with its code point in four uppercase
+hexadecimal digits, so no row
+line can open or close a fence and no escape sequence reaches a
+terminal. One limit of the chunk alignment remains, and it loses no
+content: a single row longer than one group's token budget forms a
+group of its own and can still be split across search chunks.
+
+The redactor does not catch everything in a cell. It replaces whole
+columns under a credential-named header, `key=value` and `key: value`
+pairs, JSON entries, Bearer tokens, JWTs, private regions and URL
+userinfo. It does not redact a bare vendor token (an access key with a
+vendor prefix and nothing naming it), a base64 or other high-entropy
+blob, a PEM block inside a quoted multi-line cell, or a credential-like
+value under a header that does not name a credential. Bare tokens are
+left on purpose, because a token pass would also erase order ids, SKUs
+and hashes. Such a value reaches the summary page, full-text search and
+the embeddings, as it already sits in the source file in the vault.
+
+A summary page that carries a derived section is at most as visible as
+its source: it takes on the source's own `visibility` tokens, so a reader
+that cannot read the source cannot read what was derived from it. When
+the page already had a `visibility`, it gets the audience both the
+operator and the source allow; when they share none, the page is
+withheld below local reach.
+
+A Markdown or text source gets nothing new: its page is byte for byte
+what it was.
+
## Safety properties
These are invariants of the system, not configuration to enable.
diff --git a/docs/mcp.md b/docs/mcp.md
index 9058b362..816af477 100644
--- a/docs/mcp.md
+++ b/docs/mcp.md
@@ -508,7 +508,10 @@ to its owning vault at the configuration level. `brain_idea_lineage` accepts `id
`max_depth`. `brain_note_history` accepts `path` and optional `gap_hours` /
`max_count`.
`brain_recall_gate` accepts optional `previous_prompt` and
-`explicit`; `explicit: true` always returns `retrieve: true`.
+`explicit`; `explicit: true` always returns `retrieve: true`. Since
+v1.69.0 it also accepts an optional `turn_id` (a string of at most 512
+characters), recorded on the `gate_telemetry` record when that telemetry
+is on.
Optional decision-model `answerable` signal (see
[decision-models/answerable.md](decision-models/answerable.md)): with rerank
@@ -2094,3 +2097,126 @@ format characters), when it contains NUL, or when it exceeds the cap.
(`.tf`, `.tfvars`) and redacts credentials in import specifiers
(see `o2b brain pre-extract` in
[`cli-reference.md`](cli-reference.md)).
+- Since v1.69.0 ingest reads more than Markdown. One format registry
+ maps a file's extension to a format: Markdown and the lightweight
+ markups (`.md`, `.markdown`, `.txt`, `.text`, `.rst`, `.org`) are read
+ as they are, CSV, TSV and HTML (`.csv`, `.tsv`, `.html`, `.htm`) are
+ extracted, and PDF, the Office formats (`docx`, `xlsx`, `pptx`, `odt`,
+ `ods`, `odp`), EPUB, RTF and images are named but not extracted.
+ `brain_ingest_batch_plan` plans CSV, TSV and HTML files by default, and
+ a planned file whose format is not text carries `format` beside
+ `status`. A file of a named format that is not extracted is listed in
+ `skipped_non_extractable` with reason `format-not-extractable` and the
+ format as `detail` (before the schema skips, and counted in
+ `skip_reason_counts`) instead of adding to `unclassifiable`; an
+ `extensions` override that lists its extension still plans it.
+ `brain_ingest_source` derives a section from an HTML, CSV or TSV file
+ in the vault, by format and with no new argument, in the trusted lane
+ and only from a file the caller can read at its reach. The page gains
+ `source_format` and `source_content_hash`, and the derived section
+ comes after `## Connections to existing notes`. For HTML, `## Parts`
+ holds one fenced block with one line per heading, `h<level> <trail> |
+ lines <a>-<b>` (the trail joins the ancestor headings with ` > `, the
+ line span is 1-based in the extracted text), never body text: at most
+ 256 parts, each heading capped at 200 characters; the result gains
+ `parts` (`{extracted: true, count, omitted?}` or `{extracted: false,
+ reason}`). For CSV and TSV, `## Table` holds fenced plain-text row
+ groups under `### Rows <a>-<b>`. The first record is always the header,
+ and every group repeats it, so each search chunk of the page carries
+ it, except the later pieces of a single row longer than a group's
+ token budget, which can still be split across chunks; a group holds at
+ most 50 rows and 600 tokens. A `.csv` file is read as
+ semicolon-delimited when its first record has at least two fields on
+ semicolons and more than on commas. Cells are joined by ` | `; inside
+ a cell a backslash, `|`, a backtick, a line feed, a carriage return
+ and a tab are written as `\\`, `\|`, ``\` ``, `\n`, `\r` and `\t`, and
+ every other control character (C0, DEL and C1) as `\u{XXXX}` with its
+ code point in four uppercase hexadecimal digits, so no cell can end the fence or reach a
+ terminal raw. The caps are 1,000 rows, 64 columns, 256 characters per
+ cell and 256 KiB per section, and each cut is named in
+ `table_truncated`; only the first 4,096 code units of a cell are read
+ for redaction, and a longer cell counts as cut. A `<private>` region
+ is hidden before the records are split, so a region spanning records
+ hides every row between its tags. Each HTML heading is redacted
+ (`key=value` credentials and URL userinfo) before it is capped, only
+ its first 4,096 code units are read for that (a longer heading ends in
+ `…`), and in the parts list a backslash in a heading is written as
+ `\\` and a `|` as `\|`. The page gains
+ `table_delimiter`, `table_columns`, `table_rows`,
+ `table_rows_rendered` and `table_truncated` (absent when nothing was
+ cut); the result gains `table` (`{rendered: true, format, delimiter,
+ columns, rows, rows_rendered, truncated?, redacted_cells}` or
+ `{rendered: false, format, reason, detail?}`, with reasons
+ `source-not-local`, `not-utf8`, `contains-nul`, `malformed-quoting`
+ and `empty`). Every value in a column whose header names a credential
+ is replaced, and every other cell goes through the output redactor
+ with URL credentials included; `redacted_cells` counts both. A
+ credential no pass recognises stays on the indexed page and in its
+ embeddings, as it already sits in the file in the vault. A summary page
+ that carries a derived section is at most as visible as its source: it
+ takes on the source's own `visibility` tokens; when the page already
+ had a `visibility`, it gets the audience both the operator and the
+ source allow; when they share none, the page is withheld below local
+ reach. A leading frontmatter block in an
+ HTML, CSV or TSV source is not extracted, so it is never read as table
+ data or text. A URL source, an absent file and a file the caller
+ cannot read at its reach all answer `source-not-local`, with no digest and no section. A Markdown or
+ text source has no `parts` and no `table` key and its page is
+ unchanged. An operator-set `visibility` on a summary page is now kept
+ when its source is ingested again. `brain_recall_gate` accepts an
+ optional `turn_id` (a string of at most 512 characters), recorded on
+ the `gate_telemetry` record, and refuses a `turn_id` or `session_id`
+ over 512 characters or a `telemetry_host` over 200 with
+ `INVALID_PARAMS` before it runs, with telemetry on or off. The recall
+ verdict's root coverage and the ids of the daily and weekly
+ `brain_brief` views answer at the caller's reach: below local reach a
+ root counts as reached only through a page the caller can read, and
+ the daily and weekly views leave out the status transitions,
+ retirements and contradictions that name a record the caller cannot
+ read (under its `pref-` or its `ret-` id), withhold a source pointer
+ when every evidence event citing it is about such a record, or when
+ the pointer names a page the caller cannot read, recompute the
+ `events_by_kind` and `vault_delta` counts from the events the caller
+ may see, and take no report snapshot and show no `delta`. The monthly
+ and operator views, and the today view outside its recent activity,
+ still count over the whole Brain layer and name no record. The
+ read-only preview of the same
+ extraction is the CLI verb `o2b brain extract` (see
+ [`cli-reference.md`](cli-reference.md)).
+- Since v1.69.0 more readers treat a page or record the caller cannot
+ read at its reach as absent; a local caller and the CLI see no change.
+ Below local reach the daily log pages (`Brain/log/`, the JSONL
+ sidecars included) are not served by the generic page readers
+ (`brain_search` and every other tool that reads a page by path).
+ `brain_analytics` `view="timeline"`, `view="belief_evolution"` and
+ `view="concept_synthesis"`, `brain_brief` `view="today"`,
+ `view="digest"` and `brain_event_trace`, `brain_claims` (current and
+ history), `brain_backlinks` and `osb://backlinks/{id}`,
+ `osb://log/{date}`, `osb://topic/{topic}`, `osb://preference/{id}`,
+ `brain_query` with `since`, `topic` or `preference`, `brain_moc_audit`
+ and every `brain_doctor` finding answer at the caller's reach: a record
+ is judged under each id it answers to (`pref-` and `ret-`), a log
+ entry naming such a record is dropped before any limit or total, and a
+ dream shared with a readable record is kept with its readable
+ transitions only. `osb://log/{date}` re-renders the day from the kept
+ entries. A refused preference or hub answers as an absent one, and a
+ refused MOC member is neither listed nor sized. The `brain_doctor`
+ findings `orphan-evidence` and `malformed-evidence-range` now carry
+ `sources` (the preference the evidence is about) and
+ `removed-tool-reference` carries `path` (vault-relative on the wire),
+ so each names its record structurally; the messages are unchanged.
+ `brain_search_by_source` and `brain_delete_by_source` honour the reach
+ of a derived summary page: such a page, and the content-manifest entry
+ of a source the caller cannot read, are left out of the answer and
+ left in place. A `brain_ingest_source` below local reach leaves a
+ summary page the caller cannot read untouched and answers as a first
+ ingest would. An entity intake of a source that is reserved against
+ remote reads records no `source_content_hash`. A byte-order mark
+ before a frontmatter block no longer hides that block, so its
+ `visibility` applies to the page. The empty-search coverage verdict no
+ longer counts a root as reached through a page the caller cannot read.
+ Some counts still run over the whole Brain layer and name no record:
+ the monthly and operator views, the status views, the cap on
+ `removed-tool-reference` warnings and the `brain_doctor`
+ stale-dependency note; the ranking statistics of `brain_search` are
+ taken over the shared index.
diff --git a/docs/observability.md b/docs/observability.md
index 1290fb2a..2d25935f 100644
--- a/docs/observability.md
+++ b/docs/observability.md
@@ -111,7 +111,7 @@ Every continuity record shares one envelope, and exactly these eight fields in t
|---|---|---|
| `recall_telemetry` | opt-in per call (`telemetry` option/param); the prompt-time hook emitter is gated by `recall_inject_enabled` instead | `packContext` (`mode: context_pack`), `buildPreCompressPack` (`mode: pre_compress`), `brain_search` MCP handler (`mode: search`), `brain_query` MCP handler (`mode: query`, since v0.40.0 - payload carries the query kind only, never the supplied preference id / topic / timestamp), and since v1.46.0 the `UserPromptSubmit` recall-inject hook (`mode: search`, `channel: hook`, mapping its inject / abstain / error decision onto `ok` / `empty` / `error`; with the optional `recall_inject` decision-model use in `shadow` or `enforce`, `metadata` gains a `decision_model` object - `mode`, `outcome` (`ok`, `not_sent` or `degraded`), `latency_ms`, `notes_dropped`, `abstained` - and the abstain reason `decision_model_abstain`, never prompt or note text; see `docs/decision-models/recall-inject.md`) |
| `context_receipt` | opt-in per call (`receipt` option/param) | `packContext`, `buildPreCompressPack` |
-| `gate_telemetry` | config key `recall_gate_telemetry` (default off) | `brain_recall_gate` MCP handler; a call carrying `decision_answerable` while that decision-model use is on adds `adequacy_level` and `decision_answerable` (`probability`, `disagrees`) |
+| `gate_telemetry` | config key `recall_gate_telemetry` (default off) | `brain_recall_gate` MCP handler; a call carrying `decision_answerable` while that decision-model use is on adds `adequacy_level` and `decision_answerable` (`probability`, `disagrees`); since v1.69.0 a call carrying `turn_id` records it under `turn_id`, which the read-model lifts like every other turn id |
| `negative_recall` | config key `recall_gate_telemetry` (default off - the SAME gate as `gate_telemetry` above, so one config decision governs both observations the recall gate makes) | `brain_recall_gate` MCP handler on the zero-usable-result path only (`emitNegativeRecallTelemetry` in `src/core/brain/gate-telemetry.ts`) - one record per gate call whose caller reported no usable result, carrying the typed corpus verdict (`state`, `complete`, the named `unknown_reason` when the verdict is `unknown`, `reason`) plus the reduced coverage receipt when one could be built (`coverage_digest`, `documents`, `chunks`, `schema_version`, `embedding_signature`, `last_indexed_at`, `scope`, `unindexed_roots`). Two receipt fields are not copied into the payload: `index_path` is deliberately dropped - a host filesystem path is not vault content and these records sync between devices, and the digest already binds it; the `embeddings` census the receipt carries is likewise absent from the payload and likewise bound by `coverage_digest`, so a reader comparing two records still sees a keyword-only index differ from a fully embedded one. Same prompt invariant as `gate_telemetry`: a SHA-256 prefix (`prompt_hash`) and the length (`prompt_chars`), never the prompt. A call with usable results writes NO record, so a corpus statement is never invented for an attempt that had one |
| `generation_report` | opt-in per call (`enable`) or config key `generation_trace_enabled` (default off) | inbound agent report after a generation handoff (`brain_generation_reports` action `record`, `o2b brain generation-reports record`) |
| `mcp_route_latency` | config key `mcp_route_metrics_enabled` (default off) | the MCP server's tool-call seam (`MCPServer.invokeToolHandler` in `src/mcp/server.ts`) - one record per `tools/call` and per CLI-bridge `callTool`; payload carries the tool name, scope, status, duration, and argument KEY NAMES only, never argument values |
diff --git a/docs/updating.md b/docs/updating.md
index 95ab9d52..965712bf 100644
--- a/docs/updating.md
+++ b/docs/updating.md
@@ -47,6 +47,112 @@ instruction files such as `CLAUDE.md`/`AGENTS.md`, installed
`.claude/skills/`) and warns with the exact replacement for any stale
reference it finds (`removed-tool-reference`).
+## Upgrading to 1.69.0
+
+No step below is required. Ten changes are visible to an operator or a
+client.
+
+**CSV, TSV and HTML files are planned by default.** `o2b brain
+batch-plan` and `brain_ingest_batch_plan` now discover `.csv`, `.tsv`,
+`.html` and `.htm` files beside the Markdown and plain-text ones, and a
+planned file of one of these formats carries its `format`. The plan id
+of a tree holding such files therefore changes, and a resume checkpoint
+written for that tree before the upgrade no longer matches it; a tree of
+Markdown and plain-text files plans exactly as before. An `extensions`
+override is honoured as before.
+
+**PDF, Office, EPUB, RTF and image files are named, not counted.** A
+`.png`, `.pdf`, `.docx` or another file of a format the registry names
+but does not extract moves from the `unclassifiable` counts to
+`skipped_non_extractable`, one entry per file with reason
+`format-not-extractable` and the format as `detail`, so
+`skip_reason_counts` gains the `format-not-extractable` key and the plan
+id of a tree holding such files changes too. A reader that summed
+`unclassifiable` sees fewer files there and finds them in
+`skipped_non_extractable`.
+
+**Ingesting an HTML, CSV or TSV source adds a derived section.**
+`brain_ingest_source` writes a `## Parts` section (HTML headings and
+their line spans) or a `## Table` section (the rows as fenced plain
+text) onto the summary page of an in-vault source of these formats,
+with `source_format`, `source_content_hash` and, for a table, five
+`table_*` frontmatter keys, and returns `parts` or `table`. The first
+ingest after the upgrade of a CSV, TSV or HTML source already ingested
+rewrites its page once with the new section and keys. Text and Markdown
+sources are unchanged. A leading frontmatter block in such a source is
+not data: it is never rendered as a table row or as text. Table cells
+and headings pass the redactor; see [`mcp.md`](mcp.md) for what it
+catches and what it does not.
+
+**A summary page carries its source's visibility, and a re-ingest keeps
+the operator's.** A summary page with a derived section takes on its
+source's `visibility`, so a source an operator reserved yields a summary
+page reserved the same way. When both the page and the source declare
+tokens, the page gets the audience both the operator and the source
+allow; when they share none, the page is withheld below local reach. An
+operator-set `visibility` on any summary page used to be dropped when
+the source was ingested again; it is now kept, the way `created_at` is,
+for every source format. A byte-order mark before a frontmatter block
+no longer hides that block, on any page.
+
+**`brain_recall_gate` accepts `turn_id` and bounds its correlation
+arguments.** The optional argument (at most 512 characters) the Hermes
+plugin sends was refused by the closed input schema; it is now accepted
+and recorded on the `gate_telemetry` record. A `turn_id`, `session_id`
+or `telemetry_host` longer than its bound (512, 512 and 200 characters)
+is now refused with `INVALID_PARAMS` before the gate runs, with
+telemetry on or off; it used to be dropped from the telemetry record
+without a word, and not checked at all with telemetry off.
+
+**Multi-line prompts that start with a command name are now admitted.**
+The recall surfacing gate skipped a prompt whose first word is a command
+name such as `git` or `ls` as a shell command, even when the prompt went
+on over more lines. Such a multi-line prompt now retrieves. A
+single-line prompt that starts with a command name is still skipped as
+`shell_command`.
+
+**The daily and weekly briefs and the recall verdict answer at the
+caller's reach.** Below local reach the daily and weekly `brain_brief`
+views leave out rows and source pointers that only a record the caller
+cannot read accounts for, recompute `events_by_kind` and `vault_delta`
+from the events the caller may see, take no report snapshot and show no
+`delta`. The empty-search coverage verdict of `brain_search` and
+`brain_recall_gate` counts a root as reached only through a page the
+caller can read, so it no longer flips on a root that holds only pages
+the caller cannot read.
+
+**More readers treat a page or record the caller cannot read at its
+reach as absent.** Below local reach the daily log pages are no longer
+served by the generic page readers (`brain_search` and every tool that
+reads a page by path). `brain_analytics` `view="timeline"`,
+`view="belief_evolution"` and `view="concept_synthesis"`, the today,
+digest and `brain_event_trace` views, `brain_claims`, `brain_backlinks`,
+the `osb://backlinks`, `osb://log`, `osb://topic` and `osb://preference`
+resources, `brain_query` with `since`, `topic` or `preference`,
+`brain_moc_audit` and every `brain_doctor` finding answer at the
+caller's reach. `brain_search_by_source` and `brain_delete_by_source`
+honour the reach of a derived summary page, and a `brain_ingest_source`
+below local reach leaves a summary page the caller cannot read
+untouched. An entity intake of a source reserved against remote reads
+records no `source_content_hash`. The `brain_doctor` findings
+`orphan-evidence` and `malformed-evidence-range` gain `sources`, and
+`removed-tool-reference` gains `path` (vault-relative on the wire); the
+messages are unchanged. A local caller and the CLI see no change. See
+[`mcp.md`](mcp.md) for the full list.
+
+**Private regions open more readily.** An open `<private>` tag whose
+attributes hold a `<`, or that never closes, now opens a region, in the
+redactor, the session-recall externalisation and the continuity
+redaction alike; the text after it is hidden or flagged private where
+it used to pass. Private regions and table cells are now redacted in
+linear time.
+
+**Architecture notes name the edges a link cannot carry.**
+`o2b brain architect` no longer says that no module depends on another
+when every declared edge touches a module whose name a link cannot
+carry: it lists those edges as code spans, so an overview or module note
+in that state reports `updated` once.
+
## Upgrading to 1.68.0
No step below is required. Six changes are visible to an operator or a
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index 82c08a1e..b8d4c4d3 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -2,7 +2,7 @@
"id": "open-second-brain",
"name": "Open Second Brain",
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",
- "version": "1.68.0",
+ "version": "1.69.0",
"activation": { "onStartup": true },
"skills": ["./skills"],
"contracts": {
diff --git a/openclaw/index.js b/openclaw/index.js
index 2c0a38b2..27c45f0b 100644
--- a/openclaw/index.js
+++ b/openclaw/index.js
@@ -1943,7 +1943,7 @@ var SCAN_TRUNCATED_MARKER = `
${SCAN_TRUNCATED_SENTINEL} [redactor scan window exceeded (> 1 MiB); the unscanned tail was dropped. ` + `This payload was only partially scanned — treat it as unverified and inspect the raw source before sharing.]
`;
-var PRIVATE_OPEN_TAG_RE = /<private\b[^>]*>/gi;
+var PRIVATE_OPEN_TAG_RE = /<private\b[^<>]*(?:>|(?=<)|$)/gi;
var PRIVATE_CLOSE_TAG_RE = /<\/private>/gi;
var SECRET_KEYS = [
"api_key",
@@ -2087,42 +2087,55 @@ function quoteRedactedFrontmatter(text) {
return text;
return text.replace(FRONTMATTER_BLOCK_RE, (_match, open, body, close) => open + body.replace(FRONTMATTER_SCALAR_RE, (_m, prefix, value) => value.startsWith('"') ? `${prefix}${value}` : `${prefix}${quoteYamlScalar(value)}`).replace(FRONTMATTER_ITEM_RE, (_m, prefix, value) => value.startsWith('"') ? `${prefix}${value}` : `${prefix}${quoteYamlScalar(value)}`).replace(FRONTMATTER_FLOW_ITEM_RE, (_m, prefix, value) => `${prefix}${quoteYamlScalar(value)}`) + close);
}
-function stripPrivateRegions(text) {
- if (!text)
- return text;
- let output = "";
+function privateRegionSpans(text) {
+ const spans = [];
+ const open = new RegExp(PRIVATE_OPEN_TAG_RE.source, "gi");
+ const close = new RegExp(PRIVATE_CLOSE_TAG_RE.source, "gi");
+ const nextAt = (re, from) => {
+ re.lastIndex = from;
+ return re.exec(text);
+ };
let cursor = 0;
- PRIVATE_OPEN_TAG_RE.lastIndex = 0;
- PRIVATE_CLOSE_TAG_RE.lastIndex = 0;
while (cursor < text.length) {
- PRIVATE_OPEN_TAG_RE.lastIndex = cursor;
- const openMatch = PRIVATE_OPEN_TAG_RE.exec(text);
- if (!openMatch) {
- output += text.slice(cursor);
+ const first = nextAt(open, cursor);
+ if (!first)
break;
- }
- output += text.slice(cursor, openMatch.index);
- output += PRIVATE_REGION_PLACEHOLDER;
let depth = 1;
- let scan = PRIVATE_OPEN_TAG_RE.lastIndex;
+ let scan = first.index + first[0].length;
+ let nextOpen = nextAt(open, scan);
+ let nextClose = nextAt(close, scan);
while (depth > 0) {
- PRIVATE_OPEN_TAG_RE.lastIndex = scan;
- PRIVATE_CLOSE_TAG_RE.lastIndex = scan;
- const nextOpen = PRIVATE_OPEN_TAG_RE.exec(text);
- const nextClose = PRIVATE_CLOSE_TAG_RE.exec(text);
- if (!nextClose)
- return output;
+ if (!nextClose) {
+ spans.push({ start: first.index, end: text.length });
+ return spans;
+ }
if (nextOpen && nextOpen.index < nextClose.index) {
depth += 1;
- scan = PRIVATE_OPEN_TAG_RE.lastIndex;
+ scan = nextOpen.index + nextOpen[0].length;
} else {
depth -= 1;
- scan = PRIVATE_CLOSE_TAG_RE.lastIndex;
+ scan = nextClose.index + nextClose[0].length;
}
+ if (nextOpen && nextOpen.index < scan)
+ nextOpen = nextAt(open, scan);
+ if (nextClose.index < scan)
+ nextClose = nextAt(close, scan);
}
+ spans.push({ start: first.index, end: scan });
cursor = scan;
}
- return output;
+ return spans;
+}
+function stripPrivateRegions(text) {
+ if (!text)
+ return text;
+ let output = "";
+ let cursor = 0;
+ for (const span of privateRegionSpans(text)) {
+ output += text.slice(cursor, span.start) + PRIVATE_REGION_PLACEHOLDER;
+ cursor = span.end;
+ }
+ return output + text.slice(cursor);
}
function scanRawOutput(text, opts = {}) {
if (!text)
@@ -3308,7 +3321,8 @@ function isRemotelyReadable(pageTags, reach) {
}
// src/core/vault.ts
-var FRONTMATTER_RE = /^---\s*\n([\s\S]*?)\n---\s*\n?/;
+var FRONTMATTER_RE = /^---[^\S\n]*\n([\s\S]*?)\n---\s*\n?/;
+var BYTE_ORDER_MARK = 65279;
var FRONTMATTER_KEY_PATTERN = "[a-zA-Z_][a-zA-Z0-9_-]*";
var FRONTMATTER_KEY_RE = new RegExp(`^${FRONTMATTER_KEY_PATTERN}$`);
var KEY_VALUE_RE = new RegExp(`^(${FRONTMATTER_KEY_PATTERN})\\s*:\\s*(.*?)\\s*$`);
@@ -3358,9 +3372,10 @@ function parseFrontmatterWithNotices(path, opts = {}) {
}
return parseFrontmatterTextWithNotices(text, { site, path });
}
-function parseFrontmatterTextWithNotices(text, opts = {}) {
+function parseFrontmatterTextWithNotices(raw, opts = {}) {
const notices = [];
const site = opts.site ?? FRONTMATTER_SITE;
+ const text = raw.charCodeAt(0) === BYTE_ORDER_MARK ? raw.slice(1) : raw;
const match = FRONTMATTER_RE.exec(text);
if (!match) {
return [{}, text.trim(), notices];
diff --git a/package.json b/package.json
index e5916039..ce8985f1 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.68.0",
+ "version": "1.69.0",
"private": false,
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",
"keywords": [
diff --git a/plugin.yaml b/plugin.yaml
index 55ed56d2..95b02340 100644
--- a/plugin.yaml
+++ b/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.68.0"
+version: "1.69.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.json
index afdf9c62..ad4afe87 100644
--- a/plugins/codex/.codex-plugin/plugin.json
+++ b/plugins/codex/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "1.68.0",
+ "version": "1.69.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/plugins/codex/README.md b/plugins/codex/README.md
index 6ee35afe..6dda182c 100644
--- a/plugins/codex/README.md
+++ b/plugins/codex/README.md
@@ -113,7 +113,7 @@ The full router with readiness criteria is [`install.md`](https://github.com/ite
## What is new
-1.68.0 teaches the architecture scan to read the dependency manifests a project declares. `o2b brain architect` reads `package.json`, `pyproject.toml`, `Cargo.toml` and `go.mod` at the root and at each module, gives every manifest one status (`read`, `malformed`, `unreadable` or `unsupported`), lists runtime dependencies per ecosystem, and turns a dependency of one module on another into a declared edge: a `module-dependencies` diagram on the overview, a `dependencies` region on each module note and a generator-owned `depends_on` frontmatter key the indexer reads as typed links. The pre-extractor reads Terraform (`.tf`, `.tfvars`) into block seeds with `depends_on` and `references` edges and redacts credentials in import specifiers, and the active digest, the activity and lessons digests, the morning brief, the pre-compress pack, `brain_health` and the `brain_doctor` repair now treat a record the caller cannot read at its reach as absent. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).
+1.69.0 teaches ingest to read more than Markdown. One format registry decides what a file is: CSV, TSV and HTML files are planned by default and `brain_ingest_source` writes what it derives from them onto the summary page that search indexes - a `## Table` section of fenced plain-text row groups that each repeat the header, with named caps and redacted credential columns, or a `## Parts` section of HTML headings and their line spans, never the body - with `source_format`, `source_content_hash` and the table's counts in frontmatter. PDF, Office, EPUB, RTF and image files are named per file as `format-not-extractable` instead of counted as unknown, and `o2b brain extract` previews the extraction of any file without writing. The recall verdict's root coverage, the daily and weekly briefs, the daily log pages, the analytics, claims, backlinks, MOC audit and doctor readers and the source-cleanup tools now answer at the caller's reach, a summary page carries its source's visibility, private regions and table cells are redacted in linear time, `brain_recall_gate` accepts the `turn_id` the Hermes plugin sends, multi-line prompts that start with a command name are now admitted by the surfacing gate, and the architecture notes name the module edges a link cannot carry instead of claiming there are none. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).
## Documentation
diff --git a/plugins/hermes/_schemas.py b/plugins/hermes/_schemas.py
index 0f8132f7..5ba9980c 100644
--- a/plugins/hermes/_schemas.py
+++ b/plugins/hermes/_schemas.py
@@ -480,6 +480,10 @@ STATIC_TOOL_SCHEMAS: tuple[dict[str, Any], ...] = (
'description': 'Optional session correlation id '
'recorded on the telemetry '
'record.'},
+ 'turn_id': {'type': 'string',
+ 'maxLength': 512,
+ 'description': 'Optional turn correlation id '
+ 'recorded on the telemetry record.'},
'scores': {'type': 'array',
'maxItems': 200,
'items': {'type': 'number'},
diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yaml
index 55ed56d2..95b02340 100644
--- a/plugins/hermes/plugin.yaml
+++ b/plugins/hermes/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "1.68.0"
+version: "1.69.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/pyproject.toml b/pyproject.toml
index f3f35914..369c66c8 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
# CLI entry points (those moved to `package.json` `bin`).
[project]
name = "open-second-brain"
-version = "1.68.0"
+version = "1.69.0"
description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."
readme = "README.md"
requires-python = ">=3.11"
diff --git a/src/cli/brain.ts b/src/cli/brain.ts
index 2a4df38d..93aac3a7 100644
--- a/src/cli/brain.ts
+++ b/src/cli/brain.ts
@@ -93,6 +93,7 @@ import {
cmdBrainForgetSource,
cmdBrainBatchPlan,
cmdBrainPreExtract,
+ runBrainExtract,
cmdBrainDistill,
cmdBrainHandoff,
cmdBrainIntention,
@@ -382,6 +383,8 @@ export async function handleBrainSubcommand(argv: ReadonlyArray<string>): Promis
return await cmdBrainBatchPlan(rest);
case "pre-extract":
return await cmdBrainPreExtract(rest);
+ case "extract":
+ return await runBrainExtract(rest);
case "distill":
return await cmdBrainDistill(rest);
case "links":
diff --git a/src/cli/brain/verbs/batch-plan.ts b/src/cli/brain/verbs/batch-plan.ts
index bb19bf41..3e8b14cd 100644
--- a/src/cli/brain/verbs/batch-plan.ts
+++ b/src/cli/brain/verbs/batch-plan.ts
@@ -97,7 +97,10 @@ export async function cmdBrainBatchPlan(argv: string[]): Promise<number> {
);
for (const b of plan.batches) {
ok(` batch ${b.index}: ${b.files.length} file(s), ${b.totalBytes} byte(s)`);
- for (const f of b.files) ok(` - ${f.path} (${f.status}, ${f.bytes}B)`);
+ for (const f of b.files) {
+ const format = f.format !== undefined ? `, ${f.format}` : "";
+ ok(` - ${f.path} (${f.status}, ${f.bytes}B${format})`);
+ }
}
if (plan.skipped.length > 0) {
info(` ${plan.skipped.length} unchanged file(s) skipped:`);
diff --git a/src/cli/brain/verbs/extract.ts b/src/cli/brain/verbs/extract.ts
new file mode 100644
index 00000000..9cfb07b4
--- /dev/null
+++ b/src/cli/brain/verbs/extract.ts
@@ -0,0 +1,182 @@
+/**
+ * `o2b brain extract <file> [--json]`: preview what ingest derives from one
+ * source file, for every registered format.
+ *
+ * An HTML file answers its title, its text and its heading-derived parts; a
+ * CSV or TSV file answers its table note (the counts and the rendered
+ * section). Every other format - text read as it is, a named format with no
+ * extractor, an unknown extension - answers `extracted: false` with the
+ * reason by name, without reading the file. So does an HTML or table file
+ * the bounded reader refuses (not a regular file, larger than the read
+ * limit). "Could not extract" is data: every such answer exits 0. Read-only
+ * and deterministic; nothing is written.
+ */
+
+import { lstatSync } from "node:fs";
+
+import { oneLine } from "../../../core/brain/architect/manifests.ts";
+import {
+ formatPartLine,
+ HTML_EXTRACT_MAX_SOURCE_BYTES,
+ type HtmlPart,
+} from "../../../core/brain/ingest/extract-html.ts";
+import { extractSource, type SourceExtraction } from "../../../core/brain/ingest/extract-source.ts";
+import { readSourceBounded, type SourceUnread } from "../../../core/brain/ingest/ingest.ts";
+import {
+ SOURCE_EXTRACT_SKIP_REASON,
+ SOURCE_EXTRACTOR,
+ type SourceExtractor,
+ sourceFormatOf,
+ sourceFormatSpec,
+ type SourceExtractSkipReason,
+} from "../../../core/brain/ingest/source-formats.ts";
+import { fail, ok, okJson, parse, usageError } from "../helpers.ts";
+
+const USAGE = "usage: o2b brain extract <file> [--json]";
+/** The bytes handed to the dispatch for a format answered by name. */
+const NO_BYTES = new Uint8Array(0);
+/** How the text output names the format of an unregistered extension. */
+const UNKNOWN_FORMAT_LABEL = "unknown";
+
+/** The skip reason each bounded-read refusal answers. */
+const UNREAD_REASON: Readonly<Record<SourceUnread, SourceExtractSkipReason>> = Object.freeze({
+ "not a regular file": SOURCE_EXTRACT_SKIP_REASON.notARegularFile,
+ "larger than the read limit": SOURCE_EXTRACT_SKIP_REASON.sourceTooLarge,
+});
+
+/** The extractors that read a source's bytes; every other format is answered by name. */
+const BYTE_READING_EXTRACTORS: ReadonlySet<SourceExtractor> = new Set([
+ SOURCE_EXTRACTOR.html,
+ SOURCE_EXTRACTOR.table,
+]);
+
+/**
+ * The dispatch answer for `file`. The path is lstat-ed first, before any
+ * format is named: a missing path and a symbolic link are both errors. A
+ * format with no byte-reading extractor (text read as it is, a named format
+ * with no extractor, an unknown extension) is then answered by name without
+ * reading the file. An HTML or table source goes through the bounded
+ * reader, whose refusal answers in the same shape as a format skip.
+ */
+function extractFile(file: string): SourceExtraction {
+ if (lstatSync(file).isSymbolicLink()) throw new Error(`symbolic link: ${file}`);
+ const format = sourceFormatOf(file);
+ const extractor = format === null ? null : sourceFormatSpec(format).extractor;
+ if (extractor === null || !BYTE_READING_EXTRACTORS.has(extractor)) {
+ return extractSource(file, NO_BYTES);
+ }
+ const read = readSourceBounded(file, HTML_EXTRACT_MAX_SOURCE_BYTES);
+ if (read.unread !== undefined) {
+ return { extractor: null, format, reason: UNREAD_REASON[read.unread] };
+ }
+ return extractSource(file, read.bytes);
+}
+
+/** The wire shape of one HTML part. */
+function partJson(part: HtmlPart): Record<string, unknown> {
+ return {
+ index: part.index,
+ level: part.level,
+ heading: part.heading,
+ trail: part.trail,
+ line_start: part.lineStart,
+ line_end: part.lineEnd,
+ source_offset: part.sourceOffset,
+ };
+}
+
+/** The `--json` payload (without `ok`) for `path`. */
+function extractionJson(path: string, extraction: SourceExtraction): Record<string, unknown> {
+ if (extraction.extractor === SOURCE_EXTRACTOR.html) {
+ const html = extraction.html;
+ if (!html.extracted) {
+ return { path, extracted: false, format: extraction.format, reason: html.reason };
+ }
+ return {
+ path,
+ extracted: true,
+ format: extraction.format,
+ title: html.title,
+ text: html.text,
+ parts: html.parts.map(partJson),
+ ...(html.partsOmitted > 0 ? { parts_omitted: html.partsOmitted } : {}),
+ };
+ }
+ if (extraction.extractor === SOURCE_EXTRACTOR.table) {
+ const table = extraction.table;
+ if (!table.rendered) {
+ return {
+ path,
+ extracted: false,
+ format: table.format,
+ reason: table.reason,
+ ...(table.detail !== undefined ? { detail: table.detail } : {}),
+ };
+ }
+ return {
+ path,
+ extracted: true,
+ format: table.format,
+ delimiter: table.delimiter,
+ columns: table.columns,
+ rows: table.rows,
+ rows_rendered: table.rowsRendered,
+ ...(table.truncated.length > 0 ? { truncated: table.truncated } : {}),
+ redacted_cells: table.redactedCells,
+ section: table.section,
+ };
+ }
+ return { path, extracted: false, format: extraction.format, reason: extraction.reason };
+}
+
+/** The "not extracted" line, with the detail when there is one. */
+function notExtractedLine(reason: string, detail?: string): string {
+ return ` not extracted: ${reason}${detail !== undefined ? ` (${oneLine(detail)})` : ""}`;
+}
+
+/** The text output for `path`, one entry per line. */
+function extractionText(path: string, extraction: SourceExtraction): string[] {
+ const lines = [`extract: ${path} (${extraction.format ?? UNKNOWN_FORMAT_LABEL})`];
+ if (extraction.extractor === SOURCE_EXTRACTOR.html) {
+ const html = extraction.html;
+ if (!html.extracted) return [...lines, notExtractedLine(html.reason)];
+ if (html.title !== null) lines.push(` title: ${html.title}`);
+ lines.push(` ${html.parts.length} part(s)${html.parts.length > 0 ? ":" : ""}`);
+ for (const part of html.parts) lines.push(` ${formatPartLine(part)}`);
+ if (html.partsOmitted > 0) lines.push(` ${html.partsOmitted} more part(s) omitted`);
+ return lines;
+ }
+ if (extraction.extractor === SOURCE_EXTRACTOR.table) {
+ const table = extraction.table;
+ if (!table.rendered) return [...lines, notExtractedLine(table.reason, table.detail)];
+ const truncated =
+ table.truncated.length > 0 ? `, truncated: ${table.truncated.join(", ")}` : "";
+ lines.push(
+ ` ${table.delimiter}-delimited, ${table.columns} column(s), ${table.rows} row(s), ` +
+ `${table.rowsRendered} rendered, ${table.redactedCells} redacted cell(s)${truncated}`,
+ table.section,
+ );
+ return lines;
+ }
+ return [...lines, notExtractedLine(extraction.reason)];
+}
+
+export async function runBrainExtract(argv: string[]): Promise<number> {
+ const { flags, positional } = parse(argv, {
+ json: { type: "boolean" },
+ });
+ const file = positional[0];
+ if (!file) return usageError(USAGE);
+
+ try {
+ const extraction = extractFile(file);
+ if (flags["json"]) {
+ okJson(extractionJson(file, extraction));
+ return 0;
+ }
+ for (const line of extractionText(file, extraction)) ok(line);
+ return 0;
+ } catch (err) {
+ return fail((err as Error).message ?? String(err));
+ }
+}
diff --git a/src/cli/brain/verbs/index.ts b/src/cli/brain/verbs/index.ts
index de793c67..85bbe491 100644
--- a/src/cli/brain/verbs/index.ts
+++ b/src/cli/brain/verbs/index.ts
@@ -79,6 +79,7 @@ export { cmdBrainSource } from "./source.ts";
export { cmdBrainForgetSource } from "./forget-source.ts";
export { cmdBrainBatchPlan } from "./batch-plan.ts";
export { cmdBrainPreExtract } from "./pre-extract.ts";
+export { runBrainExtract } from "./extract.ts";
export { cmdBrainDistill } from "./distill.ts";
export { cmdBrainLinks } from "./links.ts";
export { cmdBrainProfile } from "./profile.ts";
diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.ts
index 60535784..d34c618d 100644
--- a/src/cli/command-manifest.ts
+++ b/src/cli/command-manifest.ts
@@ -448,6 +448,11 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({
"Extract deterministic code structure (classes/functions/imports/inheritance, Terraform blocks/references) from a source file as JSON seeds",
[flag("json", "boolean")],
),
+ command(
+ "extract",
+ "Preview what ingest derives from one source file (HTML title, text and parts; CSV or TSV table note), or why it derives nothing",
+ [flag("json", "boolean")],
+ ),
command("links", "Normalize wikilink path format across Brain notes"),
command("bridges", "Propose, accept, or dismiss embedding-near bridge links"),
command("clusters", "Detect link-graph communities and materialize cluster notes"),
diff --git a/src/core/brain/architect/generate.ts b/src/core/brain/architect/generate.ts
index b9daeacb..1c353c0e 100644
--- a/src/core/brain/architect/generate.ts
+++ b/src/core/brain/architect/generate.ts
@@ -338,9 +338,24 @@ const MODULE_DEPENDENCIES_CLAIM =
const NO_MODULE_DEPENDENCY =
"No module's manifest names exactly one other module's manifest as a runtime dependency.";
+/**
+ * What the `module-dependencies` region says, above the list of dropped
+ * edges, when modules declare edges but every one touches a module whose
+ * name a link cannot carry.
+ */
+const ONLY_UNLINKABLE_MODULE_DEPENDENCIES =
+ "Every declared module edge touches a module whose name a link cannot carry, " +
+ "so none is drawn:";
+
+/** What the `module-dependencies` region says under its diagram before the edges it dropped. */
+const UNDRAWN_EDGES_LEAD = "Not drawn (a module name holds a character a link cannot carry):";
+
/** What a module note says when its module declares no edge. */
const NO_DEPENDS_ON = "Depends on: no other module";
+/** What a module note says before the modules it declares an edge to but cannot link. */
+const NOT_LINKED_LEAD = "Not linked:";
+
/** One manifest line: its path, ecosystem, status and, when it has one, the detail. */
function manifestLine(reading: ManifestReading): string {
const detail = reading.detail === undefined ? "" : ` - ${oneLine(reading.detail)}`;
@@ -418,7 +433,14 @@ function modulesByName(facts: ProjectFacts): ReadonlyArray<ModuleFact> {
*/
function moduleDependenciesBody(facts: ProjectFacts): string {
const edges = linkableEdges(facts);
- if (edges.length === 0) return NO_MODULE_DEPENDENCY;
+ const undrawn = unlinkableEdges(facts);
+ if (edges.length === 0) {
+ if (undrawn.length === 0) return NO_MODULE_DEPENDENCY;
+ return [
+ ONLY_UNLINKABLE_MODULE_DEPENDENCIES,
+ ...undrawn.map((edge) => `- ${edgeSpan(edge)}`),
+ ].join("\n");
+ }
const ids = new Map(modulesByName(facts).map((module, index) => [module.name, `mod${index}`]));
const node = (name: string): string => `${ids.get(name)}["${mermaidLabel(name)}"]`;
return [
@@ -428,6 +450,9 @@ function moduleDependenciesBody(facts: ProjectFacts): string {
"graph LR",
...edges.map((edge) => ` ${node(edge.from)} --> ${node(edge.to)}`),
"```",
+ ...(undrawn.length === 0
+ ? []
+ : ["", `${UNDRAWN_EDGES_LEAD} ${undrawn.map(edgeSpan).join(", ")}`]),
].join("\n");
}
@@ -458,7 +483,21 @@ function codeSpanName(name: string): string {
/** The declared edges between modules a link can name; an edge touching any other is left out. */
function linkableEdges(facts: ProjectFacts): ReadonlyArray<ModuleDependency> {
- return facts.moduleDependencies.filter((edge) => isLinkable(edge.from) && isLinkable(edge.to));
+ return facts.moduleDependencies.filter(isLinkableEdge);
+}
+
+function isLinkableEdge(edge: ModuleDependency): boolean {
+ return isLinkable(edge.from) && isLinkable(edge.to);
+}
+
+/** The declared edges `linkableEdges` leaves out: each touches a module a link cannot name. */
+function unlinkableEdges(facts: ProjectFacts): ReadonlyArray<ModuleDependency> {
+ return facts.moduleDependencies.filter((edge) => !isLinkableEdge(edge));
+}
+
+/** One edge as two code spans, so a name no link can carry is still named, never linked. */
+function edgeSpan(edge: ModuleDependency): string {
+ return `${codeSpanName(edge.from)} -> ${codeSpanName(edge.to)}`;
}
/** The wikilink to one module's note, as the overview's module list writes it. */
@@ -582,18 +621,33 @@ function dependsOn(facts: ProjectFacts, module: ModuleFact): ReadonlyArray<strin
.map((edge) => edge.to);
}
-/** A module note's `dependencies` region: its manifests and the modules it depends on. */
+/** The modules `module` declares a dependency on whose edge no link can carry, sorted. */
+function notLinkedTargets(facts: ProjectFacts, module: ModuleFact): ReadonlyArray<string> {
+ return unlinkableEdges(facts)
+ .filter((edge) => edge.from === module.name)
+ .map((edge) => edge.to);
+}
+
+/**
+ * A module note's `dependencies` region: its manifests, the modules it
+ * depends on as links, and the ones it depends on that no link can carry
+ * as code spans.
+ */
function moduleDependenciesRegionBody(
key: string,
module: ModuleFact,
targets: ReadonlyArray<string>,
+ notLinked: ReadonlyArray<string>,
): string {
if (module.manifests.length === 0) return NO_MODULE_MANIFEST;
- const edges =
+ const linked =
targets.length === 0
- ? NO_DEPENDS_ON
- : ["Depends on:", ...targets.map((name) => `- ${moduleLink(key, name)}`)].join("\n");
- return `${dependencySections(module.manifests, new Set())}\n\n${edges}`;
+ ? []
+ : ["Depends on:", ...targets.map((name) => `- ${moduleLink(key, name)}`)];
+ const unlinked =
+ notLinked.length === 0 ? [] : [`${NOT_LINKED_LEAD} ${notLinked.map(codeSpanName).join(", ")}`];
+ const lines = linked.length + unlinked.length === 0 ? [NO_DEPENDS_ON] : [...linked, ...unlinked];
+ return `${dependencySections(module.manifests, new Set())}\n\n${lines.join("\n")}`;
}
function overviewRegions(
@@ -690,6 +744,7 @@ function moduleRegions(
key: string,
module: ModuleFact,
targets: ReadonlyArray<string>,
+ notLinked: ReadonlyArray<string>,
): ReadonlyArray<Region> {
const facts = [
`Path: ${oneLine(module.path)}`,
@@ -703,7 +758,7 @@ function moduleRegions(
return [
{ id: "facts", body: facts },
{ id: "files", body: files },
- { id: "dependencies", body: moduleDependenciesRegionBody(key, module, targets) },
+ { id: "dependencies", body: moduleDependenciesRegionBody(key, module, targets, notLinked) },
];
}
@@ -868,7 +923,7 @@ function renderNotes(
planNote(
modulePath(dir, module),
frontmatter("arch-module", key, [`module: ${yamlName(module.name)}`, ...owned.lines]),
- moduleRegions(key, module, targets),
+ moduleRegions(key, module, targets, notLinkedTargets(facts, module)),
owned,
),
);
diff --git a/src/core/brain/continuity/redaction.ts b/src/core/brain/continuity/redaction.ts
index c11cb374..32e229e7 100644
--- a/src/core/brain/continuity/redaction.ts
+++ b/src/core/brain/continuity/redaction.ts
@@ -1,4 +1,4 @@
-import { redactRawOutput, stripPrivateRegions } from "../../redactor.ts";
+import { privateRegionTexts, redactRawOutput, stripPrivateRegions } from "../../redactor.ts";
export interface SafePayloadResult {
readonly payload: Readonly<Record<string, unknown>>;
@@ -6,7 +6,6 @@ export interface SafePayloadResult {
readonly redacted: boolean;
}
-const PRIVATE_REGION_RE = /<private\b[^>]*>.*?<\/private>/is;
const REDACTED_RE = /\*\*\*REDACTED\*\*\*/;
export function safeContinuityPayload(
@@ -23,7 +22,10 @@ export function safeContinuityPayload(
function sanitizeValue(value: unknown): unknown {
if (typeof value === "string") {
- if (PRIVATE_REGION_RE.test(value)) sawPrivate = true;
+ // The redactor's own region scan, linear in the value's length and
+ // the same rule the strip below applies; a lazy-dot pattern here was
+ // quadratic on a run of open tags (21.8 s at 720 KB).
+ if (privateRegionTexts(value).length > 0) sawPrivate = true;
const stripped = stripPrivateRegions(value);
const redacted = redactRawOutput(stripped);
if (REDACTED_RE.test(redacted) && redacted !== stripped) sawRedaction = true;
diff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.ts
index 097270a5..e5a31a6a 100644
--- a/src/core/brain/diagnostics.ts
+++ b/src/core/brain/diagnostics.ts
@@ -63,7 +63,7 @@ import { scanDanglingWorkruns, WORKRUN_PHASE } from "./dream-workrun.ts";
import { extractWikilinkRichBodies, parseWikilinkRich } from "./link-graph/parse-wikilink.ts";
import { LINT_CONSOLIDATE_KIND } from "./lint-consolidate.ts";
import { appendLogEvent } from "./log.ts";
-import { everyArtifactRefView } from "./artifact-ref-view.ts";
+import { everyArtifactRefView, type ArtifactRef } from "./artifact-ref-view.ts";
import { ownerScopeView } from "./owner-scope-view.ts";
import { reachView } from "./reach-view.ts";
import { acquireLockSync } from "./sync-lockfile.ts";
@@ -71,6 +71,8 @@ import { isoSecond } from "./time.ts";
import { BRAIN_LOG_EVENT_KIND, type DoctorIssue } from "./types.ts";
import { isBrainArtifactId, normaliseWikilinkTarget } from "./wikilink.ts";
import { assertVaultIdentityForWrite } from "./vault-identity.ts";
+import { recordRefs } from "./log-events-at-reach.ts";
+import { extractId } from "./temporal/period-common.ts";
// ----- Diagnostics-signal model --------------------------------------------
@@ -1286,17 +1288,22 @@ export interface DoctorIssueNaming {
* structured target and name their subjects inside the message as
* `[[pref-x]]`; they are read through the shared wikilink lexer, a
* structural read of link syntax rather than a match on prose. A finding
- * survives a view only when every reference passes it.
+ * survives a view only when every reference passes it. A record id - the
+ * target, an id-shaped source, a message wikilink - is asked under every
+ * spelling it answers to ({@link recordRefs}), so a broken backlink to
+ * `pref-x` names the reserved `ret-x` it became.
*/
export function doctorIssueRefs(
vault: string,
issue: DoctorIssueNaming,
-): ReadonlyArray<string | undefined> {
+): ReadonlyArray<ArtifactRef> {
return [
issue.path === undefined ? undefined : vaultRelativeOrSelf(vault, issue.path),
- issue.target,
- ...(issue.sources ?? []),
- ...extractWikilinkRichBodies(issue.message).map((b) => parseWikilinkRich(b).target),
+ ...recordRefs(issue.target),
+ ...(issue.sources ?? []).flatMap((source) => [source, ...recordRefs(extractId(source))]),
+ ...extractWikilinkRichBodies(issue.message).flatMap((b) =>
+ recordRefs(parseWikilinkRich(b).target),
+ ),
];
}
diff --git a/src/core/brain/digest.ts b/src/core/brain/digest.ts
index c0c69152..21cbcf49 100644
--- a/src/core/brain/digest.ts
+++ b/src/core/brain/digest.ts
@@ -39,6 +39,7 @@ import { existsSync, readdirSync } from "node:fs";
import { join, posix } from "node:path";
import { readerRefView, type ArtifactRefView } from "./artifact-ref-view.ts";
+import { logEntriesAtReach, logEntryAtReach } from "./log-events-at-reach.ts";
import { buildBacklinkIndex, type BacklinkIndex } from "./backlinks.ts";
import { computeAgentSummary, type AgentSummaryEntry } from "./digest-agent-summary.ts";
import { findMergeCandidates } from "./merge-candidates.ts";
@@ -55,7 +56,7 @@ import {
resolveOwnerScopeDelivery,
} from "./preferences-collect.ts";
import { isPreferenceVisible } from "./owner-scoped-facts.ts";
-import { logEntryArtifactRefs, type BrainLogEntry } from "./log.ts";
+import type { BrainLogEntry } from "./log.ts";
import { listLogDates, readLogDay } from "./log-jsonl.ts";
import {
BRAIN_APPLY_RESULT,
@@ -516,7 +517,7 @@ function collectDigestData(
const idToPrinciple = new Map<string, string>();
for (const { pref } of preferences) idToPrinciple.set(pref.id, pref.principle);
for (const { ret } of retiredAll) idToPrinciple.set(ret.id, ret.principle);
- const logEntries = refs.keep(readLogsInWindow(vault, since, until), logEntryArtifactRefs);
+ const logEntries = logEntriesAtReach(refs, readLogsInWindow(vault, since, until));
// A shift or contradiction line is one string naming its preference
// inside other text, so the event-level filter above cannot resolve it;
// the parsed id is asked on its own.
@@ -564,7 +565,7 @@ function collectDigestData(
vault,
since,
until,
- refs.filtersNothing ? undefined : (e) => refs.row(...logEntryArtifactRefs(e)),
+ refs.filtersNothing ? undefined : (e) => logEntryAtReach(refs, e) !== null,
);
// Most-applied (Nd) — mirrors the section in `Brain/active.md`.
@@ -849,7 +850,7 @@ function findFirstAppliedArtifact(
): string | null {
for (const date of listLogDates(vault)) {
const { entries } = readLogDay(vault, date);
- for (const e of refs.keep(entries, logEntryArtifactRefs)) {
+ for (const e of logEntriesAtReach(refs, entries)) {
if (e.eventType !== BRAIN_LOG_EVENT_KIND.applyEvidence) continue;
if (e.body["result"] !== BRAIN_APPLY_RESULT.applied) continue;
const prefPayload = e.body["preference"];
diff --git a/src/core/brain/doctor/log-checks.ts b/src/core/brain/doctor/log-checks.ts
index 11e2f114..eb324f4c 100644
--- a/src/core/brain/doctor/log-checks.ts
+++ b/src/core/brain/doctor/log-checks.ts
@@ -10,7 +10,7 @@
import type { DegradationNotice } from "../../integrity/degradation.ts";
import { listVaultBasenames } from "../../vault.ts";
import { listLogMarkdownFiles } from "../log-jsonl.ts";
-import { parseLogDayFile } from "../log.ts";
+import { parseLogDayFile, type BrainLogEntry } from "../log.ts";
import { brainDirs } from "../paths.ts";
import { BRAIN_LOG_EVENT_KIND } from "../types.ts";
import { parseArtifactRef } from "../wikilink.ts";
@@ -76,6 +76,19 @@ function listShards(
}
}
+/**
+ * The preference an `apply-evidence` event was logged against, as the
+ * finding's `sources`. The message names only the evidence artifact, so
+ * without it a reader filtering findings by what they name could not
+ * tell that the finding is about a preference the caller may not read.
+ */
+function evidencePreferenceSources(e: BrainLogEntry): { readonly sources?: ReadonlyArray<string> } {
+ const preference = e.body["preference"];
+ if (typeof preference !== "string") return {};
+ const target = parseArtifactRef(preference).target;
+ return target ? { sources: [target] } : {};
+}
+
/**
* `malformed-evidence-range`: walks every `apply-evidence` event,
* runs the artifact wikilink through {@link parseArtifactRef}, and
@@ -95,6 +108,7 @@ export const evidenceRangeCheck: DoctorCheck = {
issues.push({
severity: "warning",
code: "malformed-evidence-range",
+ ...evidencePreferenceSources(e),
message:
`apply-evidence at ${e.timestamp} references artifact ${parsed.raw}` +
` with malformed range '${parsed.rangeText}'.` +
@@ -150,6 +164,7 @@ export const orphanEvidenceCheck: DoctorCheck = {
issues.push({
severity: "warning",
code: "orphan-evidence",
+ ...evidencePreferenceSources(e),
message:
`apply-evidence at ${e.timestamp} references artifact [[${target}]]` +
" but no file with that basename exists in the vault.",
diff --git a/src/core/brain/doctor/removed-tool-checks.ts b/src/core/brain/doctor/removed-tool-checks.ts
index 4d4c9f2a..25f3a5fc 100644
--- a/src/core/brain/doctor/removed-tool-checks.ts
+++ b/src/core/brain/doctor/removed-tool-checks.ts
@@ -105,6 +105,9 @@ export const removedToolReferenceCheck: DoctorCheck = {
issues.push({
severity: "warning",
code: "removed-tool-reference",
+ // The page, structurally: a reader filtering findings by what they
+ // name cannot judge a path that appears only in the prose.
+ path,
message:
// Vault-relative, forward-slash on every host: the note is named the
// way Obsidian and every other vault path in the report spell it.
diff --git a/src/core/brain/event-trace.ts b/src/core/brain/event-trace.ts
index 0d0184da..b11b4efe 100644
--- a/src/core/brain/event-trace.ts
+++ b/src/core/brain/event-trace.ts
@@ -41,6 +41,7 @@ import {
type ArtifactRefView,
} from "./artifact-ref-view.ts";
import { ownerScopeView } from "./owner-scope-view.ts";
+import { logEntryAtReach } from "./log-events-at-reach.ts";
import type { BrainLogEntry, BrainLogEntryPayload } from "./log.ts";
import { loadNormalizedContinuityRecords } from "./continuity/read-model.ts";
import type { NormalizedContinuityRecord } from "./continuity/read-model.ts";
@@ -240,8 +241,15 @@ export function resolveLogEventTraces(
// the count plus one).
const cap = selector.limit !== undefined ? Math.max(0, Math.floor(selector.limit)) : undefined;
const results: LogEventTrace[] = [];
- for (const entry of entries) {
- if (selector.kind !== undefined && entry.eventType !== selector.kind) continue;
+ for (const logged of entries) {
+ if (selector.kind !== undefined && logged.eventType !== selector.kind) continue;
+ // A dream shared with a withheld record is kept with its readable
+ // transitions only, and its correlation is read from that form, so
+ // neither the body nor the artifacts name the withheld record; each
+ // record id is asked under its pref- and ret- spellings, so evidence
+ // logged before a reserved record was retired is dropped too.
+ const entry = logEntryAtReach(view, logged);
+ if (entry === null) continue;
if (wantStamp !== undefined && entry.timestamp !== wantStamp) continue;
const correlation = extractEventCorrelation(entry);
if (selector.sessionId !== undefined && correlation.sessionId !== selector.sessionId) continue;
diff --git a/src/core/brain/gate-telemetry.ts b/src/core/brain/gate-telemetry.ts
index d0593eb6..0baa7881 100644
--- a/src/core/brain/gate-telemetry.ts
+++ b/src/core/brain/gate-telemetry.ts
@@ -22,7 +22,7 @@
import { createHash } from "node:crypto";
import { appendContinuityRecord, listContinuityRecords } from "./continuity/store.ts";
-import type { ContinuityRecord } from "./continuity/types.ts";
+import { CONTINUITY_TURN_ID_KEY, type ContinuityRecord } from "./continuity/types.ts";
import type { NegativeRecallVerdict } from "./negative-recall.ts";
export interface GateTelemetryInput {
@@ -31,6 +31,8 @@ export interface GateTelemetryInput {
readonly retrieve: boolean;
readonly reason: string;
readonly sessionId?: string;
+ /** The caller's turn correlation id, recorded under {@link CONTINUITY_TURN_ID_KEY}. */
+ readonly turnId?: string;
readonly createdAt?: string;
/**
* The adequacy verdict and the caller's advisory decision-model
@@ -72,6 +74,7 @@ export function emitGateTelemetry(vault: string, input: GateTelemetryInput): Con
payload: {
host: input.host,
...(input.sessionId !== undefined ? { session_id: input.sessionId } : {}),
+ ...(input.turnId !== undefined ? { [CONTINUITY_TURN_ID_KEY]: input.turnId } : {}),
decision: input.retrieve ? "retrieve" : "skip",
reason: input.reason,
prompt_hash: hashPrompt(input.prompt),
diff --git a/src/core/brain/ingest/batch-plan.ts b/src/core/brain/ingest/batch-plan.ts
index ce71b06f..ad5e096b 100644
--- a/src/core/brain/ingest/batch-plan.ts
+++ b/src/core/brain/ingest/batch-plan.ts
@@ -64,25 +64,28 @@ import { classifyPaths, readManifest } from "./content-manifest.ts";
import {
extractableAllowlist,
partitionExtractable,
+ SKIPPED_PAGE_REASON,
SKIPPED_PAGE_REASONS,
type SkippedPage,
type SkippedPageReason,
} from "./extractable-gate.ts";
+import {
+ DEFAULT_INGESTIBLE_EXTENSIONS,
+ SOURCE_FORMAT,
+ SOURCE_FORMAT_BY_EXTENSION,
+ sourceFormatOf,
+ sourceFormatSpec,
+ type SourceFormat,
+} from "./source-formats.ts";
/**
- * Default set of ingestible (text-bearing) extensions, lowercase with the dot.
- * Deliberately conservative - the pipeline runs no OCR/binary path, so only
- * plain-text and lightweight-markup document formats qualify. Callers may
- * override via {@link BatchPlanOptions.extensions}.
+ * Default set of ingestible extensions, lowercase with the dot: every
+ * extension the source format registry gives an extractor, the six text
+ * extensions first. Callers may override via
+ * {@link BatchPlanOptions.extensions}. Re-exported under the planner's
+ * historical name so nothing that iterates it reorders.
*/
-export const DEFAULT_INGESTIBLE_EXTENSIONS: readonly string[] = Object.freeze([
- ".md",
- ".markdown",
- ".txt",
- ".text",
- ".rst",
- ".org",
-]);
+export { DEFAULT_INGESTIBLE_EXTENSIONS };
export interface BatchPlanOptions {
/** Hard upper bound on the summed bytes of one batch (must be > 0). */
@@ -137,6 +140,12 @@ export interface PlannedFile {
readonly bytes: number;
/** Why the file is being ingested: absent from / differing in the manifest. */
readonly status: "new" | "modified";
+ /**
+ * The file's registered format when it is not `text` (HTML, CSV, TSV, or a
+ * named format a caller's `extensions` override admitted). Absent for text
+ * and for an unregistered extension, so a Markdown plan is unchanged.
+ */
+ readonly format?: SourceFormat;
}
/** One bounded batch the caller dispatches as a single parallel unit. */
@@ -173,9 +182,11 @@ export interface BatchPlan {
/** Canonical paths classified `unchanged` and therefore skipped, sorted. */
readonly skipped: readonly string[];
/**
- * Pages skipped before extraction because their `schema_type` is not in the
- * schema `extractable` allowlist (t_ed856388). Empty when the allowlist is
- * unset (the gate is off), keeping the plan byte-identical to before.
+ * Files skipped before extraction: first every file whose format the
+ * source format registry names with no extractor (`format-not-extractable`,
+ * `detail` the format), sorted by path; then every page whose
+ * `schema_type` is not in the schema `extractable` allowlist (t_ed856388).
+ * Empty when neither applies, keeping the plan byte-identical to before.
*/
readonly skippedNonExtractable: readonly SkippedPage[];
/**
@@ -295,6 +306,7 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOpt
// root inside an ignored subtree discovers nothing and says why on the plan.
const discovered: string[] = [];
const unclassifiable = new Map<string, number>();
+ const formatSkips: SkippedPage[] = [];
if (prunedWarning === null) {
collectIngestible(walkRoot, walkScope, {
vault,
@@ -303,6 +315,7 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOpt
out: discovered,
warnings: ignoreWarnings,
unclassifiable,
+ formatSkips,
...(opts.include !== undefined ? { include: opts.include } : {}),
});
}
@@ -316,7 +329,12 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOpt
// set (hence planId and the whole plan) stays byte-identical to before.
const partition = partitionExtractable(vault, discoveredRel, extractableAllowlist(vault));
const relPaths = partition.extractable;
- const skippedNonExtractable = partition.skipped;
+ // A file meets the format gate before any schema gate, so format skips
+ // come first, each group in path order.
+ const skippedNonExtractable: SkippedPage[] = [
+ ...formatSkips.toSorted((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0)),
+ ...partition.skipped,
+ ];
// The plan id keys on the FULL discovered set, so it is identical before and
// after an interruption regardless of how many items have completed.
@@ -342,11 +360,15 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOpt
const status = new Map<string, "new" | "modified">();
for (const p of classification.new) status.set(p, "new");
for (const p of classification.modified) status.set(p, "modified");
- const planned: PlannedFile[] = [...status.keys()].toSorted().map((path) => ({
- path,
- bytes: statSync(join(vault, path)).size,
- status: status.get(path)!,
- }));
+ const planned: PlannedFile[] = [...status.keys()].toSorted().map((path) => {
+ const format = sourceFormatOf(path);
+ return {
+ path,
+ bytes: statSync(join(vault, path)).size,
+ status: status.get(path)!,
+ ...(format !== null && format !== SOURCE_FORMAT.text ? { format } : {}),
+ };
+ });
const batches = packBatches(planned, opts.maxBatchBytes, opts.maxBatchFiles);
@@ -558,6 +580,8 @@ interface WalkContext {
readonly warnings: IgnoreWarning[];
/** Counts files dropped for a non-ingestible extension, per extension. */
readonly unclassifiable: Map<string, number>;
+ /** Accumulates files of a registered format that has no extractor. */
+ readonly formatSkips: SkippedPage[];
/** See {@link BatchPlanOptions.include}. Absent: every file is included. */
readonly include?: (rel: string) => boolean;
}
@@ -574,8 +598,11 @@ interface WalkContext {
* repository-declared stack no matter how deep the walk goes.
*
* A regular file the scope lets through whose extension is not ingestible is
- * counted on {@link WalkContext.unclassifiable} (P4) instead of vanishing:
- * the discovery outcome is unchanged, but the drop is no longer silent.
+ * either named on {@link WalkContext.formatSkips} - its format is one the
+ * source format registry names with no extractor (PDF, Office, images) - or
+ * counted on {@link WalkContext.unclassifiable} (P4). Both run before the
+ * reach predicate: visibility is a Markdown rule, and asking it about a binary
+ * would read the binary whole.
*/
function collectIngestible(dir: string, scope: IgnoreScope, ctx: WalkContext): void {
const effective = ctx.excludeLayer === null ? scope : scope.extend(ctx.excludeLayer);
@@ -599,7 +626,16 @@ function collectIngestible(dir: string, scope: IgnoreScope, ctx: WalkContext): v
if (effective.isIgnored(rel, false)) continue;
const ext = extname(entry.name).toLowerCase();
if (!ctx.extensions.has(ext)) {
- ctx.unclassifiable.set(ext, (ctx.unclassifiable.get(ext) ?? 0) + 1);
+ const format = SOURCE_FORMAT_BY_EXTENSION.get(ext);
+ if (format !== undefined && sourceFormatSpec(format).extractor === null) {
+ ctx.formatSkips.push({
+ path: canonicalNotePath(rel),
+ reason: SKIPPED_PAGE_REASON.formatNotExtractable,
+ detail: format,
+ });
+ } else {
+ ctx.unclassifiable.set(ext, (ctx.unclassifiable.get(ext) ?? 0) + 1);
+ }
continue;
}
// After the extension test, so the predicate reads only files the plan
diff --git a/src/core/brain/ingest/extract-html.ts b/src/core/brain/ingest/extract-html.ts
new file mode 100644
index 00000000..ba7b5e3d
--- /dev/null
+++ b/src/core/brain/ingest/extract-html.ts
@@ -0,0 +1,815 @@
+/**
+ * HTML source extraction: the title, the visible text and the
+ * heading-derived parts of one HTML file, from its bytes.
+ *
+ * A pure, linear scanner, not `HTMLRewriter`: that one leaves entities
+ * undecoded, misses document-level text in element handlers, throws on
+ * `onEndTag` for void elements and exists in Bun only, while this module
+ * also runs in the Node build. Every step moves the cursor forward and no
+ * regular expression runs over unbounded input, so the running time is
+ * linear in the size of the source whatever its shape.
+ *
+ * The grammar, by decision: comments, the doctype and processing
+ * instructions are dropped; raw-text elements and the elements in
+ * {@link SKIPPED_ELEMENTS} contribute nothing; a `<private>` region, also
+ * one inside a title or a textarea, becomes {@link PRIVATE_REGION_PLACEHOLDER};
+ * numeric character references and the six named entities are decoded, any
+ * other named entity is kept verbatim; block elements break lines, `pre`
+ * keeps its whitespace and everywhere else whitespace collapses; `<title>`
+ * is the title; no attribute value is ever emitted; CRLF and a lone CR
+ * count as one line break; bytes that are not UTF-8 are refused by name.
+ */
+
+import { fenceFor } from "../../markdown-fence.ts";
+import {
+ PRIVATE_REGION_PLACEHOLDER,
+ redactRawOutput,
+ stripPrivateRegions,
+} from "../../redactor.ts";
+import { oneLine } from "../architect/manifests.ts";
+import { SOURCE_HASH_MAX_BYTES } from "../intake/source-trust.ts";
+import { SOURCE_EXTRACT_SKIP_REASON } from "./source-formats.ts";
+
+/** The largest HTML source read for extraction, in bytes: the source hash ceiling. */
+export const HTML_EXTRACT_MAX_SOURCE_BYTES = SOURCE_HASH_MAX_BYTES;
+/** The most parts one extraction reports; the rest are counted in `partsOmitted`. */
+export const HTML_PARTS_MAX = 256;
+/**
+ * The most headings the scan keeps: the parts cap plus one, so the last kept
+ * part still takes its line span from the next heading. A heading past it is
+ * only counted, never built, so a source of many small headings under long
+ * ancestors costs no memory per omitted part.
+ */
+const HTML_SCANNED_MAX = HTML_PARTS_MAX + 1;
+/** The longest heading (and title) kept, in code points; a longer one is cut. */
+export const HTML_HEADING_MAX_CHARS = 200;
+
+/**
+ * One heading-derived part of the text. Level 0 is the preamble, the text
+ * before the first heading. `trail` joins the headings of the enclosing
+ * sections and this one with {@link TRAIL_SEPARATOR}. The line span is
+ * 1-based and inclusive over the lines of the extracted text; it runs from
+ * the heading to the line before the next part. `sourceOffset` is the UTF-8
+ * byte offset of the heading's start tag in the source file (0 for the
+ * preamble).
+ */
+export interface HtmlPart {
+ readonly index: number;
+ readonly level: number;
+ readonly heading: string;
+ readonly trail: string;
+ readonly lineStart: number;
+ readonly lineEnd: number;
+ readonly sourceOffset: number;
+}
+
+export interface HtmlExtraction {
+ readonly extracted: true;
+ readonly title: string | null;
+ readonly text: string;
+ readonly parts: ReadonlyArray<HtmlPart>;
+ readonly partsOmitted: number;
+}
+
+export type HtmlExtractResult =
+ | HtmlExtraction
+ | { readonly extracted: false; readonly reason: typeof SOURCE_EXTRACT_SKIP_REASON.notUtf8 };
+
+/** Elements whose content is raw text up to their own end tag, never markup. */
+const RAW_TEXT_ELEMENTS: ReadonlySet<string> = new Set([
+ "script",
+ "style",
+ "xmp",
+ "iframe",
+ "noembed",
+ "noframes",
+]);
+/** Elements whose content is text up to their own end tag, with entities decoded. */
+const RCDATA_ELEMENTS: ReadonlySet<string> = new Set(["title", "textarea"]);
+const TITLE_ELEMENT = "title";
+const PRIVATE_ELEMENT = "private";
+const PRE_ELEMENT = "pre";
+const LINE_BREAK_ELEMENT = "br";
+/** Elements whose whole subtree contributes no text and no part. */
+const SKIPPED_ELEMENTS: ReadonlySet<string> = new Set([
+ "noscript",
+ "template",
+ "svg",
+ "math",
+ PRIVATE_ELEMENT,
+]);
+/** Elements that start and end a line of text. */
+const BLOCK_ELEMENTS: ReadonlySet<string> = new Set([
+ "address",
+ "article",
+ "aside",
+ "blockquote",
+ "body",
+ "caption",
+ "dd",
+ "details",
+ "dialog",
+ "div",
+ "dl",
+ "dt",
+ "fieldset",
+ "figcaption",
+ "figure",
+ "footer",
+ "form",
+ "h1",
+ "h2",
+ "h3",
+ "h4",
+ "h5",
+ "h6",
+ "head",
+ "header",
+ "hgroup",
+ "hr",
+ "html",
+ "legend",
+ "li",
+ "main",
+ "nav",
+ "ol",
+ "option",
+ "p",
+ PRE_ELEMENT,
+ "section",
+ "summary",
+ "table",
+ "tbody",
+ "textarea",
+ "tfoot",
+ "thead",
+ "tr",
+ "ul",
+]);
+/** Heading elements by name, mapped to their level. */
+const HEADING_LEVELS: ReadonlyMap<string, number> = new Map([
+ ["h1", 1],
+ ["h2", 2],
+ ["h3", 3],
+ ["h4", 4],
+ ["h5", 5],
+ ["h6", 6],
+]);
+/** The level of the preamble part, the text before the first heading. */
+const PREAMBLE_LEVEL = 0;
+/** Joins the headings of a part's enclosing sections into its trail. */
+const TRAIL_SEPARATOR = " > ";
+
+/** The `## Parts` section heading and the info string of its fenced block. */
+const PARTS_SECTION_HEADING = "## Parts";
+const PARTS_FENCE_INFO = "parts";
+/** Separates a parts line's label from its span; escaped inside the label. */
+const SPAN_SEPARATOR = "|";
+const ESCAPED_SPAN_SEPARATOR = "\\|";
+const ESCAPE_CHAR = "\\";
+const ESCAPED_ESCAPE_CHAR = "\\\\";
+/** How the preamble part is named in a parts line. */
+const PREAMBLE_LABEL = "preamble";
+
+/** Table cells: a word boundary, not a line break, so a row reads as one line. */
+const CELL_ELEMENTS: ReadonlySet<string> = new Set(["td", "th"]);
+
+/** Character references decoded by name; every other name is kept as written. */
+const NAMED_ENTITIES: ReadonlyMap<string, string> = new Map([
+ ["amp", "&"],
+ ["lt", "<"],
+ ["gt", ">"],
+ ["quot", '"'],
+ ["apos", "'"],
+ ["nbsp", "\u00A0"],
+]);
+/** What an invalid numeric character reference decodes to. */
+const REPLACEMENT_CHARACTER = "�";
+const MAX_CODE_POINT = 0x10ffff;
+const SURROGATE_FIRST = 0xd800;
+const SURROGATE_LAST = 0xdfff;
+const BYTE_ORDER_MARK = 0xfeff;
+
+const LINE_FEED = "\n";
+const SPACE = " ";
+const COMMENT_OPEN = "<!--";
+const COMMENT_CLOSE = "-->";
+/** What closes an abruptly closed empty comment `<!--->` after its opening. */
+const EMPTY_COMMENT_TAIL = "->";
+const END_TAG_OPEN = "</";
+const TAG_CLOSE = ">";
+
+const CHAR_LT = 0x3c; // <
+const CHAR_GT = 0x3e; // >
+const CHAR_AMP = 0x26; // &
+const CHAR_SLASH = 0x2f; // /
+const CHAR_BANG = 0x21; // !
+const CHAR_QUESTION = 0x3f; // ?
+const CHAR_HASH = 0x23; // #
+const CHAR_SEMICOLON = 0x3b; // ;
+const CHAR_QUOTE = 0x22; // "
+const CHAR_APOSTROPHE = 0x27; // '
+const CHAR_EQUALS = 0x3d; // =
+const CHAR_CR = 0x0d;
+const CHAR_LF = 0x0a;
+
+const DECIMAL_RADIX = 10;
+const HEX_RADIX = 16;
+
+/** HTML whitespace: space, tab, line feed, form feed, carriage return. */
+function isHtmlSpace(code: number): boolean {
+ return code === 0x20 || code === 0x09 || code === 0x0a || code === 0x0c || code === 0x0d;
+}
+
+function isAsciiAlpha(code: number): boolean {
+ return (code >= 0x41 && code <= 0x5a) || (code >= 0x61 && code <= 0x7a);
+}
+
+function isAsciiDigit(code: number): boolean {
+ return code >= 0x30 && code <= 0x39;
+}
+
+function isAsciiHexDigit(code: number): boolean {
+ return isAsciiDigit(code) || (code >= 0x41 && code <= 0x46) || (code >= 0x61 && code <= 0x66);
+}
+
+function isAsciiAlphanumeric(code: number): boolean {
+ return isAsciiAlpha(code) || isAsciiDigit(code);
+}
+
+/** A character that ends a tag name: whitespace, `/` or `>`. */
+function endsTagName(code: number): boolean {
+ return isHtmlSpace(code) || code === CHAR_SLASH || code === CHAR_GT;
+}
+
+/** The strict decoder: a byte sequence that is not UTF-8 throws. The BOM is kept and skipped by the scanner, so byte offsets stay exact. */
+const STRICT_UTF8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
+
+/**
+ * The lines of extracted text as they are emitted. Outside `pre` a run of
+ * whitespace becomes one space and a line is trimmed when it ends; an
+ * empty line is never kept. Inside `pre` every line break ends a line,
+ * blank lines included.
+ */
+class LineSink {
+ readonly lines: string[] = [];
+ private current = "";
+ private pendingSpace = false;
+
+ /** Append text that collapses its whitespace. */
+ appendFlowing(text: string): void {
+ let start = 0;
+ for (let i = 0; i <= text.length; i++) {
+ const atEnd = i === text.length;
+ if (!atEnd && !isHtmlSpace(text.charCodeAt(i))) continue;
+ if (i > start) {
+ if (this.pendingSpace && this.current.length > 0) this.current += SPACE;
+ this.current += text.slice(start, i);
+ this.pendingSpace = false;
+ }
+ if (!atEnd) this.pendingSpace = true;
+ start = i + 1;
+ }
+ }
+
+ /** Append preformatted text: whitespace kept, CRLF and lone CR read as one line break. */
+ appendPreformatted(text: string): void {
+ let start = 0;
+ for (let i = 0; i < text.length; i++) {
+ const code = text.charCodeAt(i);
+ if (code !== CHAR_LF && code !== CHAR_CR) continue;
+ this.current += text.slice(start, i);
+ this.lines.push(this.current);
+ this.current = "";
+ if (code === CHAR_CR && text.charCodeAt(i + 1) === CHAR_LF) i++;
+ start = i + 1;
+ }
+ this.current += text.slice(start);
+ this.pendingSpace = false;
+ }
+
+ /** A word boundary that collapses with any whitespace around it. */
+ space(): void {
+ this.pendingSpace = true;
+ }
+
+ /** End the current line; an empty one is dropped. */
+ breakLine(): void {
+ if (this.current.length > 0) this.lines.push(this.current);
+ this.current = "";
+ this.pendingSpace = false;
+ }
+
+ text(): string {
+ this.breakLine();
+ return this.lines.join(LINE_FEED);
+ }
+}
+
+/** `text` with every whitespace run folded to one space, trimmed. */
+function collapseWhitespace(text: string): string {
+ const sink = new LineSink();
+ sink.appendFlowing(text);
+ return sink.text();
+}
+
+/**
+ * The value pass over a heading or the title: key=value credentials and
+ * URL userinfo. A heading is written to a summary page that search and
+ * embeddings read, while the HTML file itself is never indexed.
+ */
+const HEADING_REDACTION = Object.freeze({ redactUrlCredentials: true });
+
+/**
+ * The code units of a heading or title the value pass reads: far wider than
+ * {@link HTML_HEADING_MAX_CHARS}, so a credential the cap would split is
+ * still seen whole, and bounded, so one megabyte-sized heading costs no
+ * more than a short one.
+ */
+const HEADING_SCAN_MAX_CHARS = 4_096;
+
+/** The last code point of a heading cut by the window or the cap. */
+const CUT_MARK = "…";
+
+/**
+ * A folded heading or title as it is kept: its first
+ * {@link HEADING_SCAN_MAX_CHARS} code units redacted, folded onto one line
+ * again (the redactor's own notices carry line breaks), then cut to
+ * {@link HTML_HEADING_MAX_CHARS} code points. A heading longer than the
+ * window ends in the cut mark even when its redacted window is short.
+ */
+function keptHeading(folded: string): string {
+ const windowCut = folded.length > HEADING_SCAN_MAX_CHARS;
+ const window = windowCut ? codePointPrefix(folded, HEADING_SCAN_MAX_CHARS) : folded;
+ const kept = oneLine(redactRawOutput(window, HEADING_REDACTION));
+ return capCodePoints(windowCut ? `${kept}${CUT_MARK}` : kept, HTML_HEADING_MAX_CHARS);
+}
+
+/** The first `max` code units of `text`, one fewer when the cut would split a surrogate pair. */
+function codePointPrefix(text: string, max: number): string {
+ const last = text.charCodeAt(max - 1);
+ return text.slice(0, last >= 0xd800 && last <= 0xdbff ? max - 1 : max);
+}
+
+/** `text` cut to `max` code points, the last one the cut mark when anything was cut. */
+function capCodePoints(text: string, max: number): string {
+ const points = Array.from(text);
+ if (points.length <= max) return text;
+ return `${points.slice(0, max - 1).join("")}${CUT_MARK}`;
+}
+
+/**
+ * The character reference in `s` starting at `start` (an `&`): its decoded
+ * text and the index after it. Anything that is not a reference is the
+ * literal text it spans.
+ */
+function decodeReference(
+ s: string,
+ start: number,
+): { readonly text: string; readonly end: number } {
+ let k = start + 1;
+ if (s.charCodeAt(k) === CHAR_HASH) {
+ k++;
+ const hex = s.charCodeAt(k) === 0x78 || s.charCodeAt(k) === 0x58; // x or X
+ if (hex) k++;
+ const digitsStart = k;
+ let value = 0;
+ while (
+ k < s.length &&
+ (hex ? isAsciiHexDigit(s.charCodeAt(k)) : isAsciiDigit(s.charCodeAt(k)))
+ ) {
+ // Saturate above the largest code point: the digits are still consumed.
+ if (value <= MAX_CODE_POINT) {
+ value = value * (hex ? HEX_RADIX : DECIMAL_RADIX) + Number.parseInt(s[k]!, HEX_RADIX);
+ }
+ k++;
+ }
+ if (k === digitsStart) return { text: s.slice(start, k), end: k };
+ if (s.charCodeAt(k) === CHAR_SEMICOLON) k++;
+ const valid =
+ value !== 0 && value <= MAX_CODE_POINT && (value < SURROGATE_FIRST || value > SURROGATE_LAST);
+ return { text: valid ? String.fromCodePoint(value) : REPLACEMENT_CHARACTER, end: k };
+ }
+ while (k < s.length && isAsciiAlphanumeric(s.charCodeAt(k))) k++;
+ if (k === start + 1 || s.charCodeAt(k) !== CHAR_SEMICOLON) {
+ return { text: s.slice(start, k), end: k };
+ }
+ const named = NAMED_ENTITIES.get(s.slice(start + 1, k));
+ return { text: named ?? s.slice(start, k + 1), end: k + 1 };
+}
+
+/** `text` with its character references decoded. */
+function decodeAll(text: string): string {
+ let out = "";
+ let k = 0;
+ while (k < text.length) {
+ const amp = text.indexOf("&", k);
+ if (amp === -1) return out + text.slice(k);
+ out += text.slice(k, amp);
+ const decoded = decodeReference(text, amp);
+ out += decoded.text;
+ k = decoded.end;
+ }
+ return out;
+}
+
+/**
+ * UTF-8 byte offsets of increasing indices into a string decoded from
+ * valid UTF-8 (surrogates always paired). Each call resumes where the
+ * last stopped, so converting every offset of a scan stays linear.
+ */
+class ByteOffsets {
+ private index = 0;
+ private bytes = 0;
+
+ constructor(private readonly source: string) {}
+
+ at(index: number): number {
+ const s = this.source;
+ while (this.index < index) {
+ const code = s.charCodeAt(this.index);
+ if (code < 0x80) this.bytes += 1;
+ else if (code < 0x800) this.bytes += 2;
+ else if (code >= SURROGATE_FIRST && code < 0xdc00) {
+ this.bytes += 4;
+ this.index++;
+ } else this.bytes += 3;
+ this.index++;
+ }
+ return this.bytes;
+ }
+}
+
+/** A heading as the scan met it, before spans, the cap on parts and indices. */
+interface ScannedPart {
+ readonly level: number;
+ readonly heading: string;
+ readonly trail: string;
+ readonly lineStart: number;
+ readonly sourceOffset: number;
+}
+
+/** A heading whose end tag has not been met yet. */
+interface OpenHeading {
+ readonly level: number;
+ readonly sourceOffset: number;
+ /** The number of lines emitted before the heading's first line. */
+ readonly linesBefore: number;
+}
+
+/**
+ * The scanner state. One instance per call; `run` walks the source once.
+ */
+class HtmlScanner {
+ private readonly sink = new LineSink();
+ private title: string | null = null;
+ /** Names of the open skipped elements, innermost last. */
+ private readonly skipped: string[] = [];
+ private preDepth = 0;
+ private i: number;
+ private readonly offsets: ByteOffsets;
+ private openHeading: OpenHeading | null = null;
+ /** The enclosing sections of the next heading, outermost first. */
+ private readonly sections: { readonly level: number; readonly heading: string }[] = [];
+ private readonly scanned: ScannedPart[] = [];
+ /** Headings met past {@link HTML_SCANNED_MAX}, counted and never built. */
+ private unscanned = 0;
+
+ /**
+ * @param offsetBase the bytes of the file that precede `source` (a
+ * leading frontmatter block left out), added to every heading offset.
+ */
+ constructor(
+ private readonly source: string,
+ private readonly offsetBase: number,
+ ) {
+ this.i = source.charCodeAt(0) === BYTE_ORDER_MARK ? 1 : 0;
+ this.offsets = new ByteOffsets(source);
+ }
+
+ run(): HtmlExtraction {
+ const s = this.source;
+ while (this.i < s.length) {
+ const next = this.nextMarkup(this.i);
+ if (next > this.i) this.emitText(s.slice(this.i, next));
+ this.i = next;
+ if (this.i >= s.length) break;
+ if (s.charCodeAt(this.i) === CHAR_AMP) this.readReference();
+ else this.readMarkup();
+ }
+ this.sink.breakLine();
+ this.closeHeading();
+ const text = this.sink.text();
+ const all = this.partsWithSpans(this.sink.lines.length);
+ const parts = all.slice(0, HTML_PARTS_MAX);
+ return {
+ extracted: true,
+ title: this.title,
+ text,
+ parts,
+ partsOmitted: all.length - parts.length + this.unscanned,
+ };
+ }
+
+ /**
+ * Every part, the preamble first when text precedes the first heading,
+ * each spanning to the line before the next part (the last one to the
+ * last line of the text).
+ */
+ private partsWithSpans(totalLines: number): HtmlPart[] {
+ const first = this.scanned[0];
+ const ordered: ScannedPart[] =
+ first !== undefined && first.lineStart > 1
+ ? [
+ {
+ level: PREAMBLE_LEVEL,
+ heading: "",
+ trail: "",
+ lineStart: 1,
+ sourceOffset: 0,
+ },
+ ...this.scanned,
+ ]
+ : this.scanned;
+ return ordered.map((part, index) => ({
+ index,
+ level: part.level,
+ heading: part.heading,
+ trail: part.trail,
+ lineStart: part.lineStart,
+ lineEnd: (ordered[index + 1]?.lineStart ?? totalLines + 1) - 1,
+ sourceOffset: part.sourceOffset,
+ }));
+ }
+
+ /** End the open heading: an empty one is no part. */
+ private closeHeading(): void {
+ const open = this.openHeading;
+ if (open === null) return;
+ this.openHeading = null;
+ const folded = oneLine(this.sink.lines.slice(open.linesBefore).join(SPACE));
+ if (folded.length === 0) return;
+ if (this.scanned.length >= HTML_SCANNED_MAX) {
+ this.unscanned++;
+ return;
+ }
+ const heading = keptHeading(folded);
+ while ((this.sections.at(-1)?.level ?? PREAMBLE_LEVEL) >= open.level) this.sections.pop();
+ this.sections.push({ level: open.level, heading });
+ this.scanned.push({
+ level: open.level,
+ heading,
+ trail: this.sections.map((section) => section.heading).join(TRAIL_SEPARATOR),
+ lineStart: open.linesBefore + 1,
+ sourceOffset: open.sourceOffset,
+ });
+ }
+
+ /** The index of the next `<` or `&` at or after `from`, or the end. */
+ private nextMarkup(from: number): number {
+ const s = this.source;
+ for (let k = from; k < s.length; k++) {
+ const code = s.charCodeAt(k);
+ if (code === CHAR_LT || code === CHAR_AMP) return k;
+ }
+ return s.length;
+ }
+
+ private emitText(text: string): void {
+ if (this.skipped.length > 0) return;
+ if (this.preDepth > 0) this.sink.appendPreformatted(text);
+ else this.sink.appendFlowing(text);
+ }
+
+ /** Decode the character reference at `this.i` (an `&`) and emit it. */
+ private readReference(): void {
+ const decoded = decodeReference(this.source, this.i);
+ this.emitText(decoded.text);
+ this.i = decoded.end;
+ }
+
+ /** Read the markup at `this.i` (a `<`): a tag, a comment, a declaration, or text. */
+ private readMarkup(): void {
+ const s = this.source;
+ const start = this.i;
+ const after = s.charCodeAt(start + 1);
+ if (s.startsWith(COMMENT_OPEN, start)) {
+ const body = start + COMMENT_OPEN.length;
+ // `<!-->` and `<!--->` are empty comments (HTML: abrupt closing).
+ if (s.charCodeAt(body) === CHAR_GT) this.i = body + 1;
+ else if (s.startsWith(EMPTY_COMMENT_TAIL, body)) this.i = body + EMPTY_COMMENT_TAIL.length;
+ else this.i = this.indexAfter(COMMENT_CLOSE, body);
+ } else if (after === CHAR_BANG || after === CHAR_QUESTION) {
+ this.i = this.indexAfter(TAG_CLOSE, start + 2);
+ } else if (after === CHAR_SLASH) {
+ if (s.charCodeAt(start + 2) === CHAR_GT) {
+ this.i = start + 3; // `</>` is nothing
+ } else if (isAsciiAlpha(s.charCodeAt(start + 2))) {
+ const name = this.readTagName(start + 2);
+ this.i = this.indexAfter(TAG_CLOSE, start + 2 + name.length);
+ this.endTag(name);
+ } else {
+ this.i = this.indexAfter(TAG_CLOSE, start + 2); // a bogus comment
+ }
+ } else if (isAsciiAlpha(after)) {
+ const name = this.readTagName(start + 1);
+ const { end, selfClosing } = this.skipAttributes(start + 1 + name.length);
+ this.i = end;
+ this.startTag(name, start, selfClosing);
+ } else {
+ this.emitText("<");
+ this.i = start + 1;
+ }
+ }
+
+ /** The index just past the next `needle` at or after `from`, or the end. */
+ private indexAfter(needle: string, from: number): number {
+ const at = this.source.indexOf(needle, from);
+ return at === -1 ? this.source.length : at + needle.length;
+ }
+
+ /** The lowercased tag name starting at `from`. */
+ private readTagName(from: number): string {
+ const s = this.source;
+ let k = from;
+ while (k < s.length && !endsTagName(s.charCodeAt(k))) k++;
+ return s.slice(from, k).toLowerCase();
+ }
+
+ /**
+ * Walk the attributes of a start tag from `from` to its closing `>`,
+ * honouring quoted values (which may hold a `>`). Values are skipped,
+ * never read. An unterminated tag runs to the end of the source.
+ */
+ private skipAttributes(from: number): { readonly end: number; readonly selfClosing: boolean } {
+ const s = this.source;
+ let k = from;
+ let lastSignificant = -1;
+ while (k < s.length) {
+ const code = s.charCodeAt(k);
+ if (code === CHAR_GT) return { end: k + 1, selfClosing: lastSignificant === CHAR_SLASH };
+ if (code === CHAR_QUOTE || code === CHAR_APOSTROPHE) {
+ if (lastSignificant === CHAR_EQUALS) {
+ const close = s.indexOf(code === CHAR_QUOTE ? '"' : "'", k + 1);
+ if (close === -1) return { end: s.length, selfClosing: false };
+ k = close + 1;
+ lastSignificant = code;
+ continue;
+ }
+ }
+ if (!isHtmlSpace(code)) lastSignificant = code;
+ k++;
+ }
+ return { end: s.length, selfClosing: false };
+ }
+
+ /**
+ * The content of a raw-text or RCDATA element opened just before
+ * `this.i`, up to its own end tag (any case, then whitespace, `/` or
+ * `>`); the cursor moves past that end tag, or to the end of the source.
+ */
+ private readUntilEndTag(name: string): string {
+ const s = this.source;
+ let from = this.i;
+ for (;;) {
+ const at = s.indexOf(END_TAG_OPEN, from);
+ if (at === -1) {
+ const content = s.slice(this.i);
+ this.i = s.length;
+ return content;
+ }
+ const nameEnd = at + END_TAG_OPEN.length + name.length;
+ if (
+ s.slice(at + END_TAG_OPEN.length, nameEnd).toLowerCase() === name &&
+ (nameEnd >= s.length || endsTagName(s.charCodeAt(nameEnd)))
+ ) {
+ const content = s.slice(this.i, at);
+ this.i = this.indexAfter(TAG_CLOSE, nameEnd);
+ return content;
+ }
+ from = at + END_TAG_OPEN.length;
+ }
+ }
+
+ private startTag(name: string, offset: number, selfClosing: boolean): void {
+ if (RAW_TEXT_ELEMENTS.has(name)) {
+ this.readUntilEndTag(name);
+ return;
+ }
+ if (this.skipped.length > 0) {
+ if (SKIPPED_ELEMENTS.has(name) && !selfClosing) this.skipped.push(name);
+ return;
+ }
+ if (RCDATA_ELEMENTS.has(name)) {
+ // Stripped after decoding, so an encoded `&lt;private&gt;` hides too (fail-closed).
+ const content = stripPrivateRegions(decodeAll(this.readUntilEndTag(name)));
+ if (name === TITLE_ELEMENT) {
+ if (this.title === null) {
+ const folded = oneLine(collapseWhitespace(content));
+ if (folded.length > 0) this.title = keptHeading(folded);
+ }
+ return;
+ }
+ this.sink.breakLine();
+ this.sink.appendFlowing(content);
+ this.sink.breakLine();
+ return;
+ }
+ if (SKIPPED_ELEMENTS.has(name)) {
+ if (selfClosing) return;
+ if (name === PRIVATE_ELEMENT) this.sink.appendFlowing(PRIVATE_REGION_PLACEHOLDER);
+ this.skipped.push(name);
+ return;
+ }
+ if (name === LINE_BREAK_ELEMENT) {
+ this.sink.breakLine();
+ return;
+ }
+ if (CELL_ELEMENTS.has(name)) {
+ this.sink.space();
+ return;
+ }
+ if (BLOCK_ELEMENTS.has(name)) this.sink.breakLine();
+ const level = HEADING_LEVELS.get(name);
+ if (level !== undefined) {
+ this.closeHeading();
+ this.openHeading = {
+ level,
+ sourceOffset: this.offsetBase + this.offsets.at(offset),
+ linesBefore: this.sink.lines.length,
+ };
+ }
+ if (name === PRE_ELEMENT) {
+ this.preDepth++;
+ this.skipLeadingLineBreak();
+ }
+ }
+
+ /** A line break right after `<pre>` is not content (HTML parsing rule). */
+ private skipLeadingLineBreak(): void {
+ const s = this.source;
+ if (s.charCodeAt(this.i) === CHAR_CR) this.i++;
+ if (s.charCodeAt(this.i) === CHAR_LF) this.i++;
+ }
+
+ private endTag(name: string): void {
+ if (this.skipped.length > 0) {
+ if (this.skipped[this.skipped.length - 1] === name) this.skipped.pop();
+ return;
+ }
+ if (CELL_ELEMENTS.has(name)) {
+ this.sink.space();
+ return;
+ }
+ if (BLOCK_ELEMENTS.has(name)) this.sink.breakLine();
+ if (HEADING_LEVELS.has(name)) this.closeHeading();
+ if (name === PRE_ELEMENT && this.preDepth > 0) this.preDepth--;
+ }
+}
+
+/**
+ * The title, text and parts of an HTML source, or `not-utf8` when its
+ * bytes are not UTF-8. `offsetBase` is the number of file bytes that
+ * precede `bytes` (a leading frontmatter block left out), so each part's
+ * `sourceOffset` stays an offset in the source file. Pure: no I/O, no
+ * clock, the same bytes give the same answer.
+ */
+export function extractHtml(bytes: Uint8Array, offsetBase = 0): HtmlExtractResult {
+ let source: string;
+ try {
+ source = STRICT_UTF8.decode(bytes);
+ } catch {
+ return { extracted: false, reason: SOURCE_EXTRACT_SKIP_REASON.notUtf8 };
+ }
+ return new HtmlScanner(source, offsetBase).run();
+}
+
+/**
+ * One line of the parts list: `h<level> <trail> | lines <a>-<b>`, or
+ * `preamble | lines <a>-<b>`. A `\` in the trail is escaped as `\\` and
+ * then a `|` as `\|`, as table cells are, so a heading cannot forge the
+ * span that follows it, not even with a backslash of its own.
+ */
+export function formatPartLine(part: HtmlPart): string {
+ const label =
+ part.level === PREAMBLE_LEVEL
+ ? PREAMBLE_LABEL
+ : `h${part.level} ${part.trail
+ .replaceAll(ESCAPE_CHAR, ESCAPED_ESCAPE_CHAR)
+ .replaceAll(SPAN_SEPARATOR, ESCAPED_SPAN_SEPARATOR)}`;
+ return `${label} ${SPAN_SEPARATOR} lines ${part.lineStart}-${part.lineEnd}`;
+}
+
+/**
+ * The `## Parts` section of a summary page: the heading, a blank line and
+ * one fenced block (info string `parts`) with one line per part. Inside
+ * the fence a `[[link]]` or `#tag` in an untrusted heading stays out of
+ * the link graph while full-text search still reads it. Empty when the
+ * extraction has no parts.
+ */
+export function renderPartsSection(extraction: HtmlExtraction): string {
+ if (extraction.parts.length === 0) return "";
+ const body = extraction.parts.map(formatPartLine).join(LINE_FEED);
+ const fence = fenceFor(body);
+ return [PARTS_SECTION_HEADING, "", `${fence}${PARTS_FENCE_INFO}`, body, fence].join(LINE_FEED);
+}
diff --git a/src/core/brain/ingest/extract-source.ts b/src/core/brain/ingest/extract-source.ts
new file mode 100644
index 00000000..13f583b8
--- /dev/null
+++ b/src/core/brain/ingest/extract-source.ts
@@ -0,0 +1,283 @@
+/**
+ * Extraction dispatch and the derived section of a summary page.
+ *
+ * {@link extractSource} is the one place a source's bytes meet an extractor:
+ * an exhaustive switch over the registry's extractor for the source's
+ * format, so a format added to the registry with a new extractor does not
+ * type-check until it has an arm here. {@link deriveSourceSection} is what
+ * ingest writes from it onto a summary page: for an in-vault HTML, CSV or
+ * TSV file the caller may read, the format, the content hash and one
+ * section (`## Parts` or `## Table`); for every other source nothing.
+ */
+
+import { pageVisibility } from "../../graph/visibility.ts";
+import type { FrontmatterMap } from "../../types.ts";
+import { FRONTMATTER_RE, parseFrontmatterText } from "../../vault.ts";
+import { isSourceHidden, readSourceOrigin, SourceTrustError } from "../intake/source-trust.ts";
+import {
+ INTAKE_TRUST,
+ type IntakeTrust,
+ sourceContentHashFrontmatter,
+} from "../trust/untrusted-provenance.ts";
+import { extractHtml, type HtmlExtractResult, renderPartsSection } from "./extract-html.ts";
+import {
+ SOURCE_EXTRACT_SKIP_REASON,
+ SOURCE_EXTRACTOR,
+ SOURCE_FORMAT,
+ type SourceExtractSkipReason,
+ type SourceFormat,
+ sourceFormatOf,
+ sourceFormatSpec,
+} from "./source-formats.ts";
+import {
+ type TableFormat,
+ tableNote,
+ tableNoteFrontmatter,
+ type TableNoteRendered,
+ type TableNoteResult,
+ type TableNoteSkipped,
+} from "./table-note.ts";
+
+/** What the dispatch made of one source's bytes. */
+export type SourceExtraction =
+ | {
+ readonly extractor: typeof SOURCE_EXTRACTOR.html;
+ readonly format: SourceFormat;
+ readonly html: HtmlExtractResult;
+ }
+ | {
+ readonly extractor: typeof SOURCE_EXTRACTOR.table;
+ readonly format: SourceFormat;
+ readonly table: TableNoteResult;
+ }
+ | {
+ readonly extractor: null | typeof SOURCE_EXTRACTOR.verbatim;
+ readonly format: SourceFormat | null;
+ readonly reason: SourceExtractSkipReason;
+ };
+
+/**
+ * Dispatch `bytes` addressed by `path` to the extractor its format names.
+ * An unregistered extension answers `format-unknown` with no format; a text
+ * format is read verbatim and a named format without an extractor is
+ * `format-not-extractable`. Pure: no I/O.
+ */
+export function extractSource(path: string, bytes: Uint8Array): SourceExtraction {
+ const format = sourceFormatOf(path);
+ if (format === null) {
+ return { extractor: null, format: null, reason: SOURCE_EXTRACT_SKIP_REASON.formatUnknown };
+ }
+ const spec = sourceFormatSpec(format);
+ switch (spec.extractor) {
+ case null:
+ return {
+ extractor: null,
+ format: spec.format,
+ reason: SOURCE_EXTRACT_SKIP_REASON.formatNotExtractable,
+ };
+ case SOURCE_EXTRACTOR.verbatim:
+ return {
+ extractor: spec.extractor,
+ format: spec.format,
+ reason: SOURCE_EXTRACT_SKIP_REASON.formatReadVerbatim,
+ };
+ case SOURCE_EXTRACTOR.html:
+ return {
+ extractor: spec.extractor,
+ format: spec.format,
+ html: extractHtmlAfterFrontmatter(bytes),
+ };
+ case SOURCE_EXTRACTOR.table:
+ return {
+ extractor: spec.extractor,
+ format: spec.format,
+ table: tableNote(path, withoutLeadingFrontmatter(bytes).bytes),
+ };
+ }
+}
+
+/** The UTF-8 byte-order mark, which may precede a leading frontmatter block. */
+const UTF8_BOM = Object.freeze([0xef, 0xbb, 0xbf]);
+/** The first bytes of a frontmatter block. */
+const FRONTMATTER_FENCE = "---";
+const UTF8_STRICT = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
+/** Invalid bytes become U+FFFD, as in a page read as text; the mark is kept for the parser. */
+const UTF8_LENIENT = new TextDecoder("utf-8", { ignoreBOM: true });
+const UTF8_ENCODER = new TextEncoder();
+
+/**
+ * `bytes` without one leading frontmatter block (after an optional
+ * byte-order mark), so the block is never rendered as source data: a CSV
+ * header of `---` or a `visibility: private` row. `skipped` is the number
+ * of bytes left out (the mark and the block), 0 when nothing is. Bytes that
+ * are not UTF-8 or open with no closed block are returned as they are, and
+ * the extractor answers for them.
+ */
+function withoutLeadingFrontmatter(bytes: Uint8Array): {
+ readonly bytes: Uint8Array;
+ readonly skipped: number;
+} {
+ const whole = { bytes, skipped: 0 };
+ const bom = UTF8_BOM.every((byte, i) => bytes[i] === byte) ? UTF8_BOM.length : 0;
+ for (let i = 0; i < FRONTMATTER_FENCE.length; i++) {
+ if (bytes[bom + i] !== FRONTMATTER_FENCE.charCodeAt(i)) return whole;
+ }
+ let text: string;
+ try {
+ text = UTF8_STRICT.decode(bytes.subarray(bom));
+ } catch {
+ return whole;
+ }
+ // The vault's own frontmatter pattern, so the block the reach predicate
+ // reads a source's visibility from is exactly the block withheld here.
+ const block = FRONTMATTER_RE.exec(text);
+ if (block === null) return whole;
+ const skipped = bom + UTF8_ENCODER.encode(block[0]).length;
+ return { bytes: bytes.subarray(skipped), skipped };
+}
+
+/** The HTML extraction of `bytes` after any leading frontmatter block, offsets counted in the whole file. */
+function extractHtmlAfterFrontmatter(bytes: Uint8Array): HtmlExtractResult {
+ const rest = withoutLeadingFrontmatter(bytes);
+ return extractHtml(rest.bytes, rest.skipped);
+}
+
+/** The HTML outcome on an ingest result: counts only, never the text. */
+export type PartsOutcome =
+ | { readonly extracted: true; readonly count: number; readonly omitted?: number }
+ | { readonly extracted: false; readonly reason: SourceExtractSkipReason };
+
+/** The table outcome on an ingest result: the rendering's counts, never its rows. */
+export type TableOutcome = Omit<TableNoteRendered, "section"> | TableNoteSkipped;
+
+/** What ingest adds to a summary page for a source with a derived section. */
+export interface SourceDerivation {
+ readonly format: SourceFormat;
+ /** `source_format`, `source_content_hash`, then the table keys; empty when nothing was read. */
+ readonly frontmatter: FrontmatterMap;
+ /** The section written last on the page; `""` when there is none. */
+ readonly section: string;
+ readonly parts?: PartsOutcome;
+ readonly table?: TableOutcome;
+ /**
+ * The source's own visibility tokens, present only when it declares some:
+ * a page that copies source content is at most as visible as the source.
+ */
+ readonly visibility?: readonly string[];
+}
+
+/** Frontmatter key naming a derived page's source format. */
+export const SOURCE_FORMAT_FRONTMATTER_KEY = "source_format";
+
+/** Is this registered format one the table extractor reads? */
+function isTableFormat(format: SourceFormat): format is TableFormat {
+ return format === SOURCE_FORMAT.csv || format === SOURCE_FORMAT.tsv;
+}
+
+/** The derivation of a source whose bytes were not read: no keys, no section. */
+function notLocal(format: SourceFormat): SourceDerivation {
+ const reason = SOURCE_EXTRACT_SKIP_REASON.sourceNotLocal;
+ if (isTableFormat(format)) {
+ return { format, frontmatter: {}, section: "", table: { rendered: false, format, reason } };
+ }
+ return { format, frontmatter: {}, section: "", parts: { extracted: false, reason } };
+}
+
+/** The parts outcome and section of an HTML extraction. */
+function fromHtml(html: HtmlExtractResult): { parts: PartsOutcome; section: string } {
+ if (!html.extracted) return { parts: { extracted: false, reason: html.reason }, section: "" };
+ const parts: PartsOutcome =
+ html.partsOmitted > 0
+ ? { extracted: true, count: html.parts.length, omitted: html.partsOmitted }
+ : { extracted: true, count: html.parts.length };
+ return { parts, section: renderPartsSection(html) };
+}
+
+/** The table outcome, extra keys and section of a table note. */
+function fromTable(table: TableNoteResult): {
+ table: TableOutcome;
+ frontmatter: FrontmatterMap;
+ section: string;
+} {
+ if (!table.rendered) return { table, frontmatter: {}, section: "" };
+ const { section, ...counts } = table;
+ return { table: counts, frontmatter: tableNoteFrontmatter(table), section };
+}
+
+/**
+ * The derived section of the summary page for `source`, or `undefined` when
+ * its format has no derived section (text, an unregistered extension, a
+ * named format without an extractor): those pages are written exactly as
+ * before.
+ *
+ * The order is the reach order: the format first (it reads nothing), then
+ * the trusted lane, then the caller's reach, then ONE read whose digest
+ * covers exactly the bytes extracted. A URL, an absent file and a file the
+ * caller may not read all answer `source-not-local` with no digest and no
+ * section, so the page cannot tell them apart.
+ */
+export function deriveSourceSection(
+ vault: string,
+ source: string,
+ trust: IntakeTrust,
+ readable?: (rel: string) => boolean,
+): SourceDerivation | undefined {
+ const format = sourceFormatOf(source);
+ if (format === null) return undefined;
+ const extractor = sourceFormatSpec(format).extractor;
+ if (extractor !== SOURCE_EXTRACTOR.html && extractor !== SOURCE_EXTRACTOR.table) {
+ return undefined;
+ }
+ if (trust !== INTAKE_TRUST.trusted || isSourceHidden(vault, source, readable)) {
+ return notLocal(format);
+ }
+ let origin: ReturnType<typeof readSourceOrigin>;
+ try {
+ origin = readSourceOrigin(vault, source);
+ } catch (cause) {
+ // The intake already read this source; a refusal now (it grew past the
+ // digest ceiling, or stopped being readable, between the two reads) is
+ // the source not being there to derive from, not a reason to fail an
+ // ingest whose entity pages are already written.
+ if (cause instanceof SourceTrustError) return notLocal(format);
+ throw cause;
+ }
+ if (origin.bytes === undefined || origin.contentHash === undefined) return notLocal(format);
+
+ const base: FrontmatterMap = {
+ [SOURCE_FORMAT_FRONTMATTER_KEY]: format,
+ ...sourceContentHashFrontmatter(origin.contentHash),
+ };
+ const visibility = sourceVisibility(origin.bytes);
+ const reach = visibility.length > 0 ? { visibility } : {};
+ const extraction = extractSource(source, origin.bytes);
+ if ("html" in extraction) {
+ const { parts, section } = fromHtml(extraction.html);
+ return { format, frontmatter: base, section, parts, ...reach };
+ }
+ if ("table" in extraction) {
+ const derived = fromTable(extraction.table);
+ return {
+ format,
+ frontmatter: { ...base, ...derived.frontmatter },
+ section: derived.section,
+ table: derived.table,
+ ...reach,
+ };
+ }
+ // The registry gave this format an html or table extractor above, so the
+ // dispatch answered with that arm; anything else is a registry defect.
+ throw new Error(`extractor ${extractor} answered without its arm for ${format}`);
+}
+
+/**
+ * The visibility tokens the source's own frontmatter block declares, read
+ * from the bytes the extractor strips that block from (so the two agree by
+ * construction) and decoded the way the reach predicate
+ * (`isPathReadableAtReach`) decodes a page: leniently, through the vault's
+ * frontmatter parser.
+ */
+function sourceVisibility(bytes: Uint8Array): readonly string[] {
+ const [meta] = parseFrontmatterText(UTF8_LENIENT.decode(bytes));
+ return pageVisibility(meta);
+}
diff --git a/src/core/brain/ingest/extractable-gate.ts b/src/core/brain/ingest/extractable-gate.ts
index cdcf7f7e..7a12f205 100644
--- a/src/core/brain/ingest/extractable-gate.ts
+++ b/src/core/brain/ingest/extractable-gate.ts
@@ -32,6 +32,11 @@ const PAGE_TYPE_FIELD = "schema_type";
* does not understand instead of parsing free text.
*/
export const SKIPPED_PAGE_REASON = Object.freeze({
+ /**
+ * The file's format is one the source format registry names but has no
+ * extractor for (PDF, Office, EPUB, RTF, images); `detail` is the format.
+ */
+ formatNotExtractable: "format-not-extractable",
/** The page's `schema_type` is not in the schema `extractable` allowlist. */
notExtractable: "schema-type-not-extractable",
} as const);
@@ -41,6 +46,7 @@ export type SkippedPageReason = (typeof SKIPPED_PAGE_REASON)[keyof typeof SKIPPE
/** Membership list of the closed union, in the order a page meets the gates. */
export const SKIPPED_PAGE_REASONS: ReadonlyArray<SkippedPageReason> = Object.freeze([
+ SKIPPED_PAGE_REASON.formatNotExtractable,
SKIPPED_PAGE_REASON.notExtractable,
]);
@@ -80,9 +86,10 @@ export interface SkippedPage {
/** Typed reason token from the closed {@link SKIPPED_PAGE_REASONS} union. */
readonly reason: SkippedPageReason;
/**
- * The value behind the skip - the page's declared `schema_type` - so the
- * reason is checkable without re-reading the page (the page-lint `detail`
- * pattern: identifiers cross the boundary, never prose).
+ * The value behind the skip - the page's declared `schema_type`, or the
+ * `SourceFormat` token of a format skip - so the reason is checkable
+ * without re-reading the page (the page-lint `detail` pattern:
+ * identifiers cross the boundary, never prose).
*/
readonly detail: string;
}
diff --git a/src/core/brain/ingest/ingest.ts b/src/core/brain/ingest/ingest.ts
index 25fb6acf..2623278f 100644
--- a/src/core/brain/ingest/ingest.ts
+++ b/src/core/brain/ingest/ingest.ts
@@ -32,6 +32,7 @@ import {
import { dirname, join, relative } from "node:path";
import type { FrontmatterMap } from "../../types.ts";
+import { normToken, REMOTE_DENY_VISIBILITY_TOKEN } from "../../graph/visibility.ts";
import { canonicalNotePath, ensureInsideVault } from "../../path-safety.ts";
import { assertCheckpointId } from "../checkpoint-store.ts";
import {
@@ -58,6 +59,7 @@ import {
} from "../provenance/provenance.ts";
import { PLAN_ID_LABEL, recordCompleted } from "./checkpoint.ts";
import { readManifest, updateManifest } from "./content-manifest.ts";
+import { deriveSourceSection, type PartsOutcome, type TableOutcome } from "./extract-source.ts";
import {
isCodeStructureSource,
preExtractCodeStructure,
@@ -105,7 +107,8 @@ export interface IngestSourceOptions {
/**
* May the caller read the vault file at this vault-relative path? Handed to
* the intake: a source the predicate refuses is classified as one with no
- * local bytes (untrusted lane, no digest, `url-only`). A local caller passes
+ * local bytes (untrusted lane, no digest, `url-only`). A summary page it
+ * refuses is left as it is and answered as absent. A local caller passes
* nothing.
*/
readonly readable?: (rel: string) => boolean;
@@ -137,8 +140,27 @@ export interface IngestSourceResult {
* never a fake empty success. Absent entirely when the pass was off.
*/
readonly preExtract?: PreExtractResult;
+ /**
+ * What the HTML extractor made of an HTML source: a part count, or the
+ * named reason there is no `## Parts` section. Absent for every other
+ * format.
+ */
+ readonly parts?: PartsOutcome;
+ /**
+ * What the table note made of a CSV or TSV source: its counts, or the
+ * named reason there is no `## Table` section. Absent for every other
+ * format.
+ */
+ readonly table?: TableOutcome;
}
+/**
+ * Frontmatter key an operator sets to scope a page's visibility. Kept on a
+ * rewrite like `created_at`: dropping it on re-ingest would widen a page the
+ * operator narrowed.
+ */
+const VISIBILITY_FRONTMATTER_KEY = "visibility";
+
function renderLinkSection(heading: string, ids: readonly string[]): string {
if (ids.length === 0) return "";
return [`## ${heading}`, "", ...ids.map((id) => `- [[${id}]]`)].join("\n");
@@ -188,6 +210,9 @@ export function ingestSource(
const captureScope = captureScopeForTrust(trust);
const connections = intake.entitiesUpdated;
const allEntities = [...intake.entitiesCreated, ...intake.entitiesUpdated];
+ // After the intake, which decided the lane: the derived section reads the
+ // source only in the trusted lane and only at the caller's reach.
+ const derivation = deriveSourceSection(vault, canonicalSource, trust, opts.readable);
// The page filename keys on the source-identity hash, not just the slug:
// two distinct non-ASCII / symbol-only source paths can slugify to the same
@@ -196,10 +221,19 @@ export function ingestSource(
// source path always yields the same hash, hence the same file).
const sourceHash = sourceIdentityHash([canonicalSource]);
const absPath = sourcePagePath(vault, `${slugify(canonicalSource)}-${sourceHash.slice(0, 12)}`);
- const existed = existsSync(absPath);
+ // A summary page the caller may not read (it inherited a reserved
+ // source's visibility) is neither read nor rewritten, and the answer is
+ // the one an absent page gets: its path is deterministic, so `created`
+ // would otherwise tell the caller the page exists.
+ const summaryPath = canonicalNotePath(relative(vault, absPath));
+ const withheld = existsSync(absPath) && opts.readable?.(summaryPath) === false;
+ const existed = !withheld && existsSync(absPath);
const stamp = isoSecond(opts.now);
- // Preserve the original created_at on a re-ingest; bump updated_at.
- const createdAt = existed ? readCreatedAt(absPath, stamp) : stamp;
+ // Preserve the original created_at and an operator-set visibility on a
+ // re-ingest; bump updated_at.
+ const kept: KeptFrontmatter = existed
+ ? readKeptFrontmatter(absPath, stamp)
+ : { createdAt: stamp };
const meta: FrontmatterMap = {
kind: BRAIN_SOURCE_KIND,
@@ -214,9 +248,14 @@ export function ingestSource(
// Says how much of the source was captured when it is less than the
// whole file; a full-local source adds nothing, for the same reason.
...captureScopeFrontmatter(captureScope),
- created_at: createdAt,
+ // An HTML, CSV or TSV source read in the trusted lane names its format,
+ // the digest of the bytes the section was derived from, and its own
+ // keys. Every other source adds nothing.
+ ...derivation?.frontmatter,
+ created_at: kept.createdAt,
updated_at: stamp,
tags: ["brain", "brain/source"],
+ ...visibilityFrontmatter(kept.visibility, derivation?.visibility),
};
const body = [
@@ -224,6 +263,7 @@ export function ingestSource(
renderProvenanceSection(provenance),
renderLinkSection("Entities", allEntities),
renderLinkSection("Connections to existing notes", connections),
+ derivation?.section ?? "",
]
.filter((section) => section.length > 0)
.join("\n\n");
@@ -234,7 +274,7 @@ export function ingestSource(
// unchanged source truly inert.
const nextContents = formatFrontmatter(meta, body);
const unchanged = existed && readFileSync(absPath, "utf8") === nextContents;
- if (!unchanged) {
+ if (!withheld && !unchanged) {
mkdirSync(dirname(absPath), { recursive: true });
writeFrontmatterAtomic(absPath, meta, body, { overwrite: true });
}
@@ -250,7 +290,10 @@ export function ingestSource(
// Only in the trusted lane, which the intake grants only to a file the
// caller may read at its reach: a source answered as absent records no
// digest anywhere, the manifest included.
+ // Nor when the page was withheld: the manifest would then record a
+ // summary as current that this call did not write.
if (
+ !withheld &&
trust === INTAKE_TRUST.trusted &&
resolvesInsideVault(vault, canonicalSource) &&
existsSync(join(vault, canonicalSource))
@@ -274,13 +317,15 @@ export function ingestSource(
}
return {
- summaryPath: canonicalNotePath(relative(vault, absPath)),
+ summaryPath,
created: !existed,
entitiesCreated: intake.entitiesCreated,
entitiesUpdated: intake.entitiesUpdated,
connections,
captureScope,
...(preExtract !== undefined ? { preExtract } : {}),
+ ...(derivation?.parts !== undefined ? { parts: derivation.parts } : {}),
+ ...(derivation?.table !== undefined ? { table: derivation.table } : {}),
};
}
@@ -297,7 +342,8 @@ const SOURCE_UNREAD = Object.freeze({
tooLarge: "larger than the read limit",
} as const);
-type SourceUnread = (typeof SOURCE_UNREAD)[keyof typeof SOURCE_UNREAD];
+/** Why {@link readSourceBounded} read no text. */
+export type SourceUnread = (typeof SOURCE_UNREAD)[keyof typeof SOURCE_UNREAD];
/**
* How a source is opened: read-only and never blocking, so a FIFO met
@@ -310,32 +356,37 @@ const SOURCE_OPEN_FLAGS = fsConstants.O_RDONLY | (fsConstants.O_NONBLOCK ?? 0);
/**
* Read a source through ONE descriptor, so the checks hold for the bytes
- * read: the descriptor must be a regular file, and no more than
- * {@link PRE_EXTRACT_MAX_SOURCE_BYTES} is read however large the file has
- * grown since the `fstat`.
+ * read: the descriptor must be a regular file, and no more than `maxBytes`
+ * is read however large the file has grown since the `fstat`. The buffer is
+ * sized to the file plus one byte, so a small source costs a small buffer;
+ * a file that grew under the cap is read on into one buffer of the cap plus
+ * one byte. `bytes` are exactly the bytes read, for a caller that must judge
+ * their encoding itself.
*/
-function readSourceBounded(
+export function readSourceBounded(
absolute: string,
+ maxBytes: number,
):
- | { readonly text: string; readonly unread?: undefined }
- | { readonly text: null; readonly unread: SourceUnread } {
+ | { readonly text: string; readonly bytes: Uint8Array; readonly unread?: undefined }
+ | { readonly text: null; readonly bytes?: undefined; readonly unread: SourceUnread } {
const fd = openSync(absolute, SOURCE_OPEN_FLAGS);
try {
const stat = fstatSync(fd);
if (!stat.isFile()) return { text: null, unread: SOURCE_UNREAD.notAFile };
- if (stat.size > PRE_EXTRACT_MAX_SOURCE_BYTES)
- return { text: null, unread: SOURCE_UNREAD.tooLarge };
- // One byte past the cap, so growth since the fstat is seen.
- const buffer = Buffer.allocUnsafe(PRE_EXTRACT_MAX_SOURCE_BYTES + 1);
+ if (stat.size > maxBytes) return { text: null, unread: SOURCE_UNREAD.tooLarge };
+ // One byte past the size, so growth since the fstat is seen.
+ let buffer = Buffer.allocUnsafe(stat.size + 1);
let filled = 0;
for (;;) {
+ // A full buffer is under the cap here: past it, the loop has returned.
+ if (filled === buffer.length) buffer = Buffer.concat([buffer], maxBytes + 1);
const count = readSync(fd, buffer, filled, buffer.length - filled, null);
if (count === 0) break;
filled += count;
- if (filled > PRE_EXTRACT_MAX_SOURCE_BYTES)
- return { text: null, unread: SOURCE_UNREAD.tooLarge };
+ if (filled > maxBytes) return { text: null, unread: SOURCE_UNREAD.tooLarge };
}
- return { text: buffer.toString("utf8", 0, filled) };
+ const bytes = buffer.subarray(0, filled);
+ return { text: bytes.toString("utf8"), bytes };
} finally {
closeSync(fd);
}
@@ -380,7 +431,7 @@ function runPreExtract(
};
let content: string;
try {
- const read = readSourceBounded(join(vault, canonicalSource));
+ const read = readSourceBounded(join(vault, canonicalSource), PRE_EXTRACT_MAX_SOURCE_BYTES);
if (read.text === null) return read.unread === SOURCE_UNREAD.tooLarge ? tooLarge : noBytes;
content = read.text;
} catch {
@@ -415,9 +466,53 @@ function resolvesInsideVault(vault: string, canonicalSource: string): boolean {
}
}
-/** Read a stable `created_at` from an existing summary page, else fall back. */
-function readCreatedAt(absPath: string, fallback: string): string {
+/** The frontmatter a rewrite keeps from the page it replaces. */
+interface KeptFrontmatter {
+ readonly createdAt: string;
+ readonly visibility?: FrontmatterMap[string];
+}
+
+/**
+ * Read the stable `created_at` and an operator-set `visibility` from an
+ * existing summary page; `created_at` falls back to `fallback`.
+ */
+function readKeptFrontmatter(absPath: string, fallback: string): KeptFrontmatter {
const [meta] = parseFrontmatter(absPath);
const value = meta["created_at"];
- return typeof value === "string" && value.length > 0 ? value : fallback;
+ const createdAt = typeof value === "string" && value.length > 0 ? value : fallback;
+ const visibility = meta[VISIBILITY_FRONTMATTER_KEY];
+ return visibility === undefined || visibility === null
+ ? { createdAt }
+ : { createdAt, visibility };
+}
+
+/**
+ * The summary page's `visibility`. The page's audience is never wider than
+ * the source's or the operator's: when only one side declares tokens, that
+ * side's tokens are written (a page with neither gains none); when both do,
+ * the page keeps the tokens both allow, and when they share none it gets
+ * the reserved token, so it is withheld below local reach rather than
+ * widened. The reserved token is kept whenever either side carries it.
+ */
+function visibilityFrontmatter(
+ kept: KeptFrontmatter["visibility"],
+ source: readonly string[] | undefined,
+): FrontmatterMap {
+ if (source === undefined || source.length === 0) {
+ return kept !== undefined ? { [VISIBILITY_FRONTMATTER_KEY]: kept } : {};
+ }
+ let keptTokens: string[] = [];
+ if (Array.isArray(kept)) keptTokens = kept.map((k) => normToken(String(k)));
+ else if (kept !== undefined) keptTokens = [normToken(String(kept))];
+ keptTokens = keptTokens.filter((k) => k.length > 0);
+ if (keptTokens.length === 0) return { [VISIBILITY_FRONTMATTER_KEY]: [...source] };
+ const merged = source.filter((token) => keptTokens.includes(token));
+ const reserved =
+ merged.length === 0 ||
+ keptTokens.includes(REMOTE_DENY_VISIBILITY_TOKEN) ||
+ source.includes(REMOTE_DENY_VISIBILITY_TOKEN);
+ if (reserved && !merged.includes(REMOTE_DENY_VISIBILITY_TOKEN)) {
+ merged.push(REMOTE_DENY_VISIBILITY_TOKEN);
+ }
+ return { [VISIBILITY_FRONTMATTER_KEY]: merged };
}
diff --git a/src/core/brain/ingest/source-formats.ts b/src/core/brain/ingest/source-formats.ts
new file mode 100644
index 00000000..3e040425
--- /dev/null
+++ b/src/core/brain/ingest/source-formats.ts
@@ -0,0 +1,210 @@
+/**
+ * The one registry of source formats ingest knows by extension.
+ *
+ * Every question "what is this file, and can ingest read it" is answered
+ * here, from a table, never from prose: the batch planner derives its default
+ * extension set and its per-file format skips from it, and the extraction
+ * dispatch (`extract-source.ts`) switches on a spec's extractor. A format
+ * the registry names with no extractor (PDF, Office, EPUB, RTF, images) is
+ * reported by name as `format-not-extractable` instead of being counted as
+ * an anonymous `unclassifiable` extension.
+ *
+ * A Node-safe leaf: no Bun runtime API, no I/O, only type imports, because part of
+ * `src/` is bundled for Node and both the planner and the CLI read it.
+ */
+
+/**
+ * Every format the registry names. Markdown and the lightweight markups map
+ * to `text`: they are read as they are. Closed four-piece vocabulary
+ * (census row `SOURCE_FORMAT`): the token crosses the MCP wire and the CLI
+ * JSON verbatim.
+ */
+export const SOURCE_FORMAT = Object.freeze({
+ text: "text",
+ csv: "csv",
+ tsv: "tsv",
+ html: "html",
+ pdf: "pdf",
+ docx: "docx",
+ xlsx: "xlsx",
+ pptx: "pptx",
+ odt: "odt",
+ ods: "ods",
+ odp: "odp",
+ epub: "epub",
+ rtf: "rtf",
+ image: "image",
+} as const);
+
+/** The closed union of format tokens. */
+export type SourceFormat = (typeof SOURCE_FORMAT)[keyof typeof SOURCE_FORMAT];
+
+/** Membership list of {@link SOURCE_FORMAT}, in registry order. */
+export const SOURCE_FORMATS: ReadonlyArray<SourceFormat> = Object.freeze(
+ Object.values(SOURCE_FORMAT),
+);
+
+/** Membership guard of {@link SOURCE_FORMATS}. */
+export function isSourceFormat(value: unknown): value is SourceFormat {
+ return typeof value === "string" && (SOURCE_FORMATS as ReadonlyArray<string>).includes(value);
+}
+
+/**
+ * How ingest reads a format: `verbatim` (the text is the source), `html` (the
+ * linear HTML scanner) or `table` (the CSV and TSV table note). Not a reason
+ * vocabulary - it never crosses a wire as a verdict - so it has no member
+ * list and no guard.
+ */
+export const SOURCE_EXTRACTOR = Object.freeze({
+ verbatim: "verbatim",
+ html: "html",
+ table: "table",
+} as const);
+
+/** The closed union of extractor tokens. */
+export type SourceExtractor = (typeof SOURCE_EXTRACTOR)[keyof typeof SOURCE_EXTRACTOR];
+
+/** One registered format: its extractor (`null`: named, not extracted) and extensions. */
+export interface SourceFormatSpec {
+ readonly format: SourceFormat;
+ readonly extractor: SourceExtractor | null;
+ /** Lowercase, dot-prefixed. */
+ readonly extensions: ReadonlyArray<string>;
+}
+
+function spec(
+ format: SourceFormat,
+ extractor: SourceExtractor | null,
+ extensions: ReadonlyArray<string>,
+): SourceFormatSpec {
+ return Object.freeze({ format, extractor, extensions: Object.freeze([...extensions]) });
+}
+
+/** The registry, one spec per format, in {@link SOURCE_FORMAT} order. */
+export const SOURCE_FORMAT_SPECS: ReadonlyArray<SourceFormatSpec> = Object.freeze([
+ spec(SOURCE_FORMAT.text, SOURCE_EXTRACTOR.verbatim, [
+ ".md",
+ ".markdown",
+ ".txt",
+ ".text",
+ ".rst",
+ ".org",
+ ]),
+ spec(SOURCE_FORMAT.csv, SOURCE_EXTRACTOR.table, [".csv"]),
+ spec(SOURCE_FORMAT.tsv, SOURCE_EXTRACTOR.table, [".tsv"]),
+ spec(SOURCE_FORMAT.html, SOURCE_EXTRACTOR.html, [".html", ".htm"]),
+ spec(SOURCE_FORMAT.pdf, null, [".pdf"]),
+ spec(SOURCE_FORMAT.docx, null, [".docx"]),
+ spec(SOURCE_FORMAT.xlsx, null, [".xlsx"]),
+ spec(SOURCE_FORMAT.pptx, null, [".pptx"]),
+ spec(SOURCE_FORMAT.odt, null, [".odt"]),
+ spec(SOURCE_FORMAT.ods, null, [".ods"]),
+ spec(SOURCE_FORMAT.odp, null, [".odp"]),
+ spec(SOURCE_FORMAT.epub, null, [".epub"]),
+ spec(SOURCE_FORMAT.rtf, null, [".rtf"]),
+ spec(SOURCE_FORMAT.image, null, [
+ ".png",
+ ".jpg",
+ ".jpeg",
+ ".gif",
+ ".webp",
+ ".bmp",
+ ".tif",
+ ".tiff",
+ ".heic",
+ ]),
+]);
+
+const SPEC_BY_FORMAT: ReadonlyMap<SourceFormat, SourceFormatSpec> = new Map(
+ SOURCE_FORMAT_SPECS.map((s) => [s.format, s]),
+);
+
+/** Every registered extension mapped to its format. */
+export const SOURCE_FORMAT_BY_EXTENSION: ReadonlyMap<string, SourceFormat> = new Map(
+ SOURCE_FORMAT_SPECS.flatMap((s) => s.extensions.map((ext) => [ext, s.format] as const)),
+);
+
+/** The forward-slash path separator a Windows backslash is folded to. */
+const PATH_SEPARATOR = "/";
+const EXTENSION_MARK = ".";
+
+/**
+ * The lowercased extension of the last path segment, `""` when it has none
+ * (a dotfile name such as `.env` has none either). Both separators count, so
+ * a Windows path reads the same as a POSIX one.
+ */
+function extensionOf(path: string): string {
+ const base = path.slice(path.replaceAll("\\", PATH_SEPARATOR).lastIndexOf(PATH_SEPARATOR) + 1);
+ const dot = base.lastIndexOf(EXTENSION_MARK);
+ return dot <= 0 ? "" : base.slice(dot).toLowerCase();
+}
+
+/**
+ * The registered format of `path`, by the lowercased extension of its last
+ * segment, or `null` when the extension is absent or unregistered. A URL is
+ * judged by its last segment like any path, so one whose last segment carries
+ * no registered extension answers `null`.
+ */
+export function sourceFormatOf(path: string): SourceFormat | null {
+ const ext = extensionOf(path);
+ if (ext.length === 0) return null;
+ return SOURCE_FORMAT_BY_EXTENSION.get(ext) ?? null;
+}
+
+/** The registry spec of `format`. */
+export function sourceFormatSpec(format: SourceFormat): SourceFormatSpec {
+ const found = SPEC_BY_FORMAT.get(format);
+ if (found === undefined) {
+ // Unreachable for a typed caller; a cast value that is no format is a
+ // caller error and is named, never mapped to a guess.
+ throw new TypeError(`unknown source format: ${String(format)}`);
+ }
+ return found;
+}
+
+/**
+ * The extensions batch planning discovers by default: every extension of a
+ * spec with an extractor, in spec order (the six text extensions first).
+ */
+export const DEFAULT_INGESTIBLE_EXTENSIONS: readonly string[] = Object.freeze(
+ SOURCE_FORMAT_SPECS.filter((s) => s.extractor !== null).flatMap((s) => s.extensions),
+);
+
+/**
+ * Why a source's content was not extracted. Closed four-piece vocabulary
+ * (census row `SOURCE_EXTRACT_SKIP_REASON`): "could not extract" is data on
+ * the wire, never an error.
+ */
+export const SOURCE_EXTRACT_SKIP_REASON = Object.freeze({
+ /** A text format: the source is read as it is, nothing to extract. */
+ formatReadVerbatim: "format-read-verbatim",
+ /** A format the registry names but has no extractor for. */
+ formatNotExtractable: "format-not-extractable",
+ /** An extension the registry does not name. */
+ formatUnknown: "format-unknown",
+ /** A URL, an absent file, a file outside the vault or one hidden from the caller. */
+ sourceNotLocal: "source-not-local",
+ /** Larger than the read limit. */
+ sourceTooLarge: "source-too-large",
+ /** A directory, FIFO or other non-regular file. */
+ notARegularFile: "not-a-regular-file",
+ /** The bytes are not valid UTF-8. */
+ notUtf8: "not-utf8",
+} as const);
+
+/** The closed union of extract skip reasons. */
+export type SourceExtractSkipReason =
+ (typeof SOURCE_EXTRACT_SKIP_REASON)[keyof typeof SOURCE_EXTRACT_SKIP_REASON];
+
+/** Membership list of {@link SOURCE_EXTRACT_SKIP_REASON}. */
+export const SOURCE_EXTRACT_SKIP_REASONS: ReadonlyArray<SourceExtractSkipReason> = Object.freeze(
+ Object.values(SOURCE_EXTRACT_SKIP_REASON),
+);
+
+/** Membership guard of {@link SOURCE_EXTRACT_SKIP_REASONS}. */
+export function isSourceExtractSkipReason(value: unknown): value is SourceExtractSkipReason {
+ return (
+ typeof value === "string" &&
+ (SOURCE_EXTRACT_SKIP_REASONS as ReadonlyArray<string>).includes(value)
+ );
+}
diff --git a/src/core/brain/ingest/table-note.ts b/src/core/brain/ingest/table-note.ts
new file mode 100644
index 00000000..174335dd
--- /dev/null
+++ b/src/core/brain/ingest/table-note.ts
@@ -0,0 +1,519 @@
+/**
+ * CSV and TSV table notes: source bytes in, a fenced plain-text `## Table`
+ * section out. Pure, no I/O: the caller decides whether the bytes may be
+ * read at all (the reach order lives in `extract-source.ts`).
+ *
+ * The section is fenced plain text, not a GFM table: inside a fence a
+ * `[[x]]` or `#word` cell stays out of the link graph, and the chunker
+ * never splits a table it does not recognise as one. Each row group sits
+ * under its own `### Rows a-b` heading and repeats the header record, so
+ * every chunk carries both the column names and a heading path.
+ *
+ * Parsing follows the delimiter the extension names: `.tsv` is TAB with no
+ * quote processing (fields cannot hold a TAB or a line break); `.csv` is a
+ * comma with lenient RFC 4180 quoting, read with semicolons only when the
+ * first record has two or more fields with semicolons and more with
+ * semicolons than with commas (the spreadsheet export whose decimal
+ * separator is a comma). The first
+ * record is always the header: a file on disk carries no `header=` MIME
+ * parameter, and any heuristic guesses wrong on all-text tables.
+ */
+
+import { fenceFor } from "../../markdown-fence.ts";
+import {
+ isSecretKeyName,
+ REDACTION_PLACEHOLDER,
+ redactRawOutput,
+ stripPrivateRegions,
+} from "../../redactor.ts";
+import { countChunkTokens } from "../../search/chunker.ts";
+import type { FrontmatterMap } from "../../types.ts";
+import { SOURCE_FORMAT, sourceFormatOf } from "./source-formats.ts";
+
+/** Data rows rendered at most; the rest are counted, not shown. */
+export const TABLE_NOTE_MAX_ROWS = 1_000;
+/** Columns rendered at most; wider records are cut at the right. */
+export const TABLE_NOTE_MAX_COLUMNS = 64;
+/** Code points kept per cell; a longer cell is cut and marked. */
+export const TABLE_NOTE_MAX_CELL_CHARS = 256;
+/** UTF-8 bytes of the rendered section at most; rows stop at the last whole group that fits. */
+export const TABLE_NOTE_MAX_BYTES = 262_144;
+/** Data rows per group at most. */
+export const TABLE_NOTE_ROWS_PER_GROUP = 50;
+/** Chunker tokens per group block at most, so one group fits one chunk. */
+export const TABLE_NOTE_GROUP_MAX_TOKENS = 600;
+
+/** The delimiter a table was read with, recorded on the page as a word. */
+export const TABLE_DELIMITER = Object.freeze({
+ comma: "comma",
+ semicolon: "semicolon",
+ tab: "tab",
+} as const);
+
+export type TableDelimiter = (typeof TABLE_DELIMITER)[keyof typeof TABLE_DELIMITER];
+
+/** Which cap cut the rendering. Closed four-piece vocabulary (census row `TABLE_TRUNCATION`). */
+export const TABLE_TRUNCATION = Object.freeze({
+ /** More data rows than {@link TABLE_NOTE_MAX_ROWS}. */
+ rows: "rows",
+ /** A record wider than {@link TABLE_NOTE_MAX_COLUMNS}. */
+ columns: "columns",
+ /** A cell longer than {@link TABLE_NOTE_MAX_CELL_CHARS}. */
+ cells: "cells",
+ /** Row groups dropped to keep the section within {@link TABLE_NOTE_MAX_BYTES}. */
+ bytes: "bytes",
+} as const);
+
+export type TableTruncation = (typeof TABLE_TRUNCATION)[keyof typeof TABLE_TRUNCATION];
+
+/** Membership list, in the order the caps are applied. */
+export const TABLE_TRUNCATIONS: ReadonlyArray<TableTruncation> = Object.freeze(
+ Object.values(TABLE_TRUNCATION),
+);
+
+export function isTableTruncation(value: unknown): value is TableTruncation {
+ return typeof value === "string" && (TABLE_TRUNCATIONS as ReadonlyArray<string>).includes(value);
+}
+
+/**
+ * Why no table was rendered. Closed four-piece vocabulary (census row
+ * `TABLE_NOTE_SKIP_REASON`). Data, not an error: the summary page and its
+ * entities are written either way.
+ */
+export const TABLE_NOTE_SKIP_REASON = Object.freeze({
+ /** A URL, an absent file or a file hidden at the caller's reach: one token for all three. */
+ sourceNotLocal: "source-not-local",
+ /** The bytes are not strict UTF-8. */
+ notUtf8: "not-utf8",
+ /** The text holds a NUL character (a binary or UTF-16 file). */
+ containsNul: "contains-nul",
+ /** A quoted field is still open at the end of the input; `detail` is its 1-based record number. */
+ malformedQuoting: "malformed-quoting",
+ /** No data record after the header. */
+ empty: "empty",
+} as const);
+
+export type TableNoteSkipReason =
+ (typeof TABLE_NOTE_SKIP_REASON)[keyof typeof TABLE_NOTE_SKIP_REASON];
+
+/** Membership list, in the order a source meets the checks. */
+export const TABLE_NOTE_SKIP_REASONS: ReadonlyArray<TableNoteSkipReason> = Object.freeze(
+ Object.values(TABLE_NOTE_SKIP_REASON),
+);
+
+export function isTableNoteSkipReason(value: unknown): value is TableNoteSkipReason {
+ return (
+ typeof value === "string" && (TABLE_NOTE_SKIP_REASONS as ReadonlyArray<string>).includes(value)
+ );
+}
+
+export type TableFormat = typeof SOURCE_FORMAT.csv | typeof SOURCE_FORMAT.tsv;
+
+export interface TableNoteRendered {
+ readonly rendered: true;
+ readonly format: TableFormat;
+ readonly delimiter: TableDelimiter;
+ /** Rendered column count: the widest rendered record, capped. */
+ readonly columns: number;
+ /** Data records in the source, header excluded. */
+ readonly rows: number;
+ readonly rowsRendered: number;
+ /** Every cap that cut the rendering, in {@link TABLE_TRUNCATIONS} order; empty when none. */
+ readonly truncated: ReadonlyArray<TableTruncation>;
+ readonly redactedCells: number;
+ readonly section: string;
+}
+
+export interface TableNoteSkipped {
+ readonly rendered: false;
+ readonly format: TableFormat;
+ readonly reason: TableNoteSkipReason;
+ readonly detail?: string;
+}
+
+export type TableNoteResult = TableNoteRendered | TableNoteSkipped;
+
+const TABLE_HEADING = "## Table";
+const GROUP_HEADING_PREFIX = "### Rows ";
+const FENCE_INFO = "table";
+const CELL_SEPARATOR = " | ";
+const NEWLINE = "\n";
+const BLOCK_SEPARATOR = "\n\n";
+
+const QUOTE = '"';
+const CR = "\r";
+const LF = "\n";
+const NUL = "\u0000";
+
+const DELIMITER_CHAR: Readonly<Record<TableDelimiter, string>> = Object.freeze({
+ comma: ",",
+ semicolon: ";",
+ tab: "\t",
+});
+
+/**
+ * Cell characters escaped so one record stays one line, a cell boundary
+ * stays unambiguous, no rendered line opens with a backtick run (the
+ * chunker closes a fence on any line that starts with three backticks,
+ * whatever the opening fence's length) and no control character reaches a
+ * page or a terminal. Line breaks and TAB keep their short form; every
+ * other C0 control, DEL and C1 control becomes `\u{XXXX}`. Reversible:
+ * `\` is escaped too.
+ */
+const CELL_ESCAPES: Readonly<Record<string, string>> = Object.freeze({
+ "\\": "\\\\",
+ "|": "\\|",
+ "`": "\\`",
+ "\n": "\\n",
+ "\r": "\\r",
+ "\t": "\\t",
+});
+// oxlint-disable-next-line no-control-regex -- matching control characters is the point
+const CELL_ESCAPE_RE = /[\\|`\u0000-\u001f\u007f-\u009f]/g;
+const CODE_POINT_HEX_DIGITS = 4;
+
+/** `\u{XXXX}`: a control character by its code point. */
+function codePointEscape(ch: string): string {
+ const hex = ch.codePointAt(0)!.toString(16).toUpperCase().padStart(CODE_POINT_HEX_DIGITS, "0");
+ return `\\u{${hex}}`;
+}
+
+/** Marks a cell cut at {@link TABLE_NOTE_MAX_CELL_CHARS}. */
+const CUT_MARKER = "\u2026";
+
+/**
+ * Code units of a cell the value pass reads at most. A cell is cut to
+ * {@link TABLE_NOTE_MAX_CELL_CHARS} anyway, so the window only has to be
+ * wide enough that a credential the cap would split is still seen whole;
+ * bounding it keeps one megabyte-sized cell from costing seconds.
+ */
+const CELL_SCAN_MAX_CHARS = 4_096;
+
+/**
+ * The value pass over a cell window: key=value credentials, private
+ * regions and URL userinfo, never truncated by the redactor itself (the
+ * window bounds the input and the cell cap applies after the pass).
+ */
+const CELL_REDACTION = Object.freeze({
+ redactUrlCredentials: true,
+ maxInput: Number.POSITIVE_INFINITY,
+});
+
+const UTF8_STRICT = new TextDecoder("utf-8", { fatal: true });
+const UTF8_ENCODER = new TextEncoder();
+
+function utf8Length(text: string): number {
+ return UTF8_ENCODER.encode(text).length;
+}
+
+type Records = string[][];
+type ParseOutcome =
+ | { readonly ok: true; readonly records: Records }
+ | { readonly ok: false; readonly record: number };
+
+/**
+ * Split `text` into records. Records end at CRLF, LF or a lone CR outside
+ * quotes; a fully empty line is skipped; a final line break makes no empty
+ * record. With `quoting`, a field that opens with `"` is quoted (`""` is a
+ * literal quote, delimiters and line breaks are literal), text after the
+ * closing quote is appended literally, and a bare `"` elsewhere is kept.
+ * Stops after `maxRecords` records. Linear in the input.
+ */
+function parseRecords(
+ text: string,
+ delimiter: string,
+ quoting: boolean,
+ maxRecords = Number.POSITIVE_INFINITY,
+): ParseOutcome {
+ const records: Records = [];
+ let record: string[] = [];
+ let field = "";
+ let recordStarted = false;
+ let fieldStarted = false;
+ let inQuotes = false;
+ let i = 0;
+ const endRecord = (): void => {
+ record.push(field);
+ records.push(record);
+ record = [];
+ field = "";
+ recordStarted = false;
+ fieldStarted = false;
+ };
+ while (i < text.length && records.length < maxRecords) {
+ const ch = text[i]!;
+ if (inQuotes) {
+ if (ch === QUOTE) {
+ if (text[i + 1] === QUOTE) {
+ field += QUOTE;
+ i += 2;
+ continue;
+ }
+ inQuotes = false;
+ } else {
+ field += ch;
+ }
+ i += 1;
+ continue;
+ }
+ if (ch === CR || ch === LF) {
+ i += ch === CR && text[i + 1] === LF ? 2 : 1;
+ if (recordStarted) endRecord();
+ continue;
+ }
+ recordStarted = true;
+ if (ch === delimiter) {
+ record.push(field);
+ field = "";
+ fieldStarted = false;
+ } else if (quoting && ch === QUOTE && !fieldStarted) {
+ inQuotes = true;
+ fieldStarted = true;
+ } else {
+ field += ch;
+ fieldStarted = true;
+ }
+ i += 1;
+ }
+ if (inQuotes) return { ok: false, record: records.length + 1 };
+ if (recordStarted && records.length < maxRecords) endRecord();
+ return { ok: true, records };
+}
+
+/**
+ * The CSV delimiter: a comma, unless the first record has two or more
+ * fields with semicolons and more with semicolons than with commas. A
+ * semicolon export quotes only fields holding a semicolon, so its header
+ * may carry a bare comma (`Name;Price, EUR;Qty`).
+ */
+function csvDelimiter(text: string): TableDelimiter {
+ const first = (delimiter: TableDelimiter): number => {
+ const parsed = parseRecords(text, DELIMITER_CHAR[delimiter], true, 1);
+ return parsed.ok ? (parsed.records[0]?.length ?? 0) : 0;
+ };
+ const comma = first(TABLE_DELIMITER.comma);
+ const semicolon = first(TABLE_DELIMITER.semicolon);
+ if (semicolon >= 2 && semicolon > comma) return TABLE_DELIMITER.semicolon;
+ return TABLE_DELIMITER.comma;
+}
+
+/** A cell cut to {@link TABLE_NOTE_MAX_CELL_CHARS} code points, the last one the marker. */
+function capCell(cell: string): { readonly text: string; readonly cut: boolean } {
+ // A UTF-16 length within the cap is a code-point count within it.
+ if (cell.length <= TABLE_NOTE_MAX_CELL_CHARS) return { text: cell, cut: false };
+ const points = Array.from(cell);
+ if (points.length <= TABLE_NOTE_MAX_CELL_CHARS) return { text: cell, cut: false };
+ return { text: points.slice(0, TABLE_NOTE_MAX_CELL_CHARS - 1).join("") + CUT_MARKER, cut: true };
+}
+
+function escapeCell(cell: string): string {
+ return cell.replace(CELL_ESCAPE_RE, (ch) => CELL_ESCAPES[ch] ?? codePointEscape(ch));
+}
+
+/** One record as it is rendered: its line and what the caps and redaction did to it. */
+interface PreparedLine {
+ readonly line: string;
+ readonly width: number;
+ readonly redacted: number;
+ readonly columnsCut: boolean;
+ readonly cellsCut: boolean;
+ /** Chunker tokens of {@link line}. */
+ readonly tokens: number;
+}
+
+/**
+ * Cut, redact, cap and escape one record. Only the first
+ * {@link CELL_SCAN_MAX_CHARS} code units of a cell are read, and a cell
+ * longer than that counts as cut. A cell under a column whose
+ * header names a credential is replaced whole (the header is kept: names
+ * only); every other cell, header cells included (the first record may be
+ * data), goes through the value pass. No bare-token pass: it would erase
+ * order ids, SKUs and hashes, which are table data.
+ */
+function prepareLine(
+ record: ReadonlyArray<string>,
+ credentialColumns: ReadonlyArray<boolean>,
+): PreparedLine {
+ const kept = record.slice(0, TABLE_NOTE_MAX_COLUMNS);
+ let redacted = 0;
+ let cellsCut = false;
+ const cells = kept.map((raw, column) => {
+ const windowCut = raw.length > CELL_SCAN_MAX_CHARS;
+ const window = windowCut ? raw.slice(0, CELL_SCAN_MAX_CHARS) : raw;
+ const clean =
+ credentialColumns[column] === true && raw.length > 0
+ ? REDACTION_PLACEHOLDER
+ : redactRawOutput(window, CELL_REDACTION);
+ if (clean !== window) redacted += 1;
+ const capped = capCell(clean);
+ if (capped.cut || windowCut) cellsCut = true;
+ return escapeCell(capped.text);
+ });
+ const line = cells.join(CELL_SEPARATOR);
+ return {
+ line,
+ width: kept.length,
+ redacted,
+ columnsCut: record.length > kept.length,
+ cellsCut,
+ tokens: countChunkTokens(line),
+ };
+}
+
+function groupHeading(first: number, last: number): string {
+ return `${GROUP_HEADING_PREFIX}${first}-${last}`;
+}
+
+function renderGroup(header: string, lines: ReadonlyArray<string>, first: number): string {
+ const body = [header, ...lines].join(NEWLINE);
+ const fence = fenceFor(body);
+ const heading = groupHeading(first, first + lines.length - 1);
+ return `${heading}${BLOCK_SEPARATOR}${fence}${FENCE_INFO}${NEWLINE}${body}${NEWLINE}${fence}`;
+}
+
+function closingLine(rendered: number, total: number): string {
+ return `Rendered ${rendered} of ${total} rows.`;
+}
+
+/** Data rows split into groups of at most {@link TABLE_NOTE_ROWS_PER_GROUP} rows and {@link TABLE_NOTE_GROUP_MAX_TOKENS} tokens. */
+function groupRows(
+ header: PreparedLine,
+ rows: ReadonlyArray<PreparedLine>,
+): ReadonlyArray<ReadonlyArray<PreparedLine>> {
+ // Tokens are whitespace-delimited, so a group's count is the sum of its
+ // lines plus the fixed lines around them (heading, fences, header).
+ const overhead = countChunkTokens(renderGroup(header.line, [], 1));
+ const groups: PreparedLine[][] = [];
+ let current: PreparedLine[] = [];
+ let tokens = overhead;
+ for (const row of rows) {
+ const full =
+ current.length === TABLE_NOTE_ROWS_PER_GROUP ||
+ (current.length > 0 && tokens + row.tokens > TABLE_NOTE_GROUP_MAX_TOKENS);
+ if (full) {
+ groups.push(current);
+ current = [];
+ tokens = overhead;
+ }
+ current.push(row);
+ tokens += row.tokens;
+ }
+ if (current.length > 0) groups.push(current);
+ return groups;
+}
+
+function tableFormatOf(path: string): TableFormat {
+ const format = sourceFormatOf(path);
+ if (format === SOURCE_FORMAT.csv || format === SOURCE_FORMAT.tsv) return format;
+ throw new TypeError(`not a table source (csv or tsv): ${path}`);
+}
+
+/**
+ * Render the table note of the CSV or TSV file at `path` from its bytes.
+ * Throws `TypeError` naming `path` when its extension is not a table
+ * format: that is a caller error, not a property of the bytes.
+ */
+export function tableNote(path: string, bytes: Uint8Array): TableNoteResult {
+ const format = tableFormatOf(path);
+ const skip = (reason: TableNoteSkipReason, detail?: string): TableNoteSkipped =>
+ detail === undefined
+ ? { rendered: false, format, reason }
+ : { rendered: false, format, reason, detail };
+
+ let text: string;
+ try {
+ text = UTF8_STRICT.decode(bytes);
+ } catch {
+ return skip(TABLE_NOTE_SKIP_REASON.notUtf8);
+ }
+ if (text.includes(NUL)) return skip(TABLE_NOTE_SKIP_REASON.containsNul);
+ // A private region follows the vault-wide rule before any parsing, so a
+ // region that spans records hides every row between its tags, not only
+ // the cells that hold a tag.
+ text = stripPrivateRegions(text);
+
+ const delimiter = format === SOURCE_FORMAT.tsv ? TABLE_DELIMITER.tab : csvDelimiter(text);
+ const parsed = parseRecords(text, DELIMITER_CHAR[delimiter], format === SOURCE_FORMAT.csv);
+ if (!parsed.ok) return skip(TABLE_NOTE_SKIP_REASON.malformedQuoting, String(parsed.record));
+ const [headerRecord, ...dataRecords] = parsed.records;
+ if (headerRecord === undefined || dataRecords.length === 0) {
+ return skip(TABLE_NOTE_SKIP_REASON.empty);
+ }
+
+ const credentialColumns = headerRecord.map((name) => isSecretKeyName(name));
+ const header = prepareLine(headerRecord, []);
+ const rows = dataRecords
+ .slice(0, TABLE_NOTE_MAX_ROWS)
+ .map((record) => prepareLine(record, credentialColumns));
+
+ // Groups are added while the section, with room for the closing line,
+ // stays within the byte cap; the last group needs no closing line when
+ // nothing else was cut.
+ const total = dataRecords.length;
+ const reserve = utf8Length(BLOCK_SEPARATOR + closingLine(total, total));
+ const groups = groupRows(header, rows);
+ const blocks: string[] = [TABLE_HEADING];
+ let bytesUsed = utf8Length(TABLE_HEADING);
+ let rowsRendered = 0;
+ let redactedCells = header.redacted;
+ let columns = header.width;
+ let columnsCut = header.columnsCut;
+ let cellsCut = header.cellsCut;
+ let bytesCut = false;
+ for (const [index, group] of groups.entries()) {
+ const block = renderGroup(
+ header.line,
+ group.map((row) => row.line),
+ rowsRendered + 1,
+ );
+ const size = utf8Length(BLOCK_SEPARATOR + block);
+ const completes = index === groups.length - 1 && rows.length === total;
+ if (bytesUsed + size + (completes ? 0 : reserve) > TABLE_NOTE_MAX_BYTES) {
+ bytesCut = true;
+ break;
+ }
+ blocks.push(block);
+ bytesUsed += size;
+ rowsRendered += group.length;
+ for (const row of group) {
+ redactedCells += row.redacted;
+ columns = Math.max(columns, row.width);
+ columnsCut ||= row.columnsCut;
+ cellsCut ||= row.cellsCut;
+ }
+ }
+ if (rowsRendered < total) blocks.push(closingLine(rowsRendered, total));
+
+ const cuts: Readonly<Record<TableTruncation, boolean>> = {
+ rows: total > rows.length,
+ columns: columnsCut,
+ cells: cellsCut,
+ bytes: bytesCut,
+ };
+ return {
+ rendered: true,
+ format,
+ delimiter,
+ columns,
+ rows: total,
+ rowsRendered,
+ truncated: TABLE_TRUNCATIONS.filter((kind) => cuts[kind]),
+ redactedCells,
+ section: blocks.join(BLOCK_SEPARATOR),
+ };
+}
+
+/**
+ * The frontmatter keys a rendered table adds to its summary page, in the
+ * pinned order; `table_truncated` only when a cap cut the rendering.
+ */
+export function tableNoteFrontmatter(result: TableNoteRendered): FrontmatterMap {
+ const keys: FrontmatterMap = {
+ table_delimiter: result.delimiter,
+ table_columns: result.columns,
+ table_rows: result.rows,
+ table_rows_rendered: result.rowsRendered,
+ };
+ if (result.truncated.length > 0) keys.table_truncated = [...result.truncated];
+ return keys;
+}
diff --git a/src/core/brain/intake/extract-intake.ts b/src/core/brain/intake/extract-intake.ts
index 1ff67a2e..b3437969 100644
--- a/src/core/brain/intake/extract-intake.ts
+++ b/src/core/brain/intake/extract-intake.ts
@@ -33,6 +33,8 @@
*/
import { isKnownRelation, normalizeRelation } from "../../graph/relation-vocab.ts";
+import { TRANSPORT_REACH } from "../../graph/transport-reach.ts";
+import { isPathReadableAtReach, type FrontmatterCache } from "../../search/result-filters.ts";
import { validateEntityCategory, normalizeEntityName } from "../entities/canonical.ts";
import { relateEntities, upsertEntity } from "../entities/registry.ts";
import { renderProvenanceSection, type Provenance } from "../provenance/provenance.ts";
@@ -169,7 +171,9 @@ function validateIntake(intake: ExtractionIntake): void {
* source. With several, there is no single set of bytes this extraction came
* from, and stamping the first one would record a file the entities may have
* nothing to do with - a misleading audit record is worse than none, because
- * only the second can be read as "not recorded".
+ * only the second can be read as "not recorded". Nor is it recorded for a
+ * source a remote reader may not read: the entity pages carrying it are
+ * readable where the source is not.
*
* An intake citing NO source throws instead of picking a lane. It cannot be
* classified, and both lanes are wrong for it: trusting it makes the omission
@@ -197,9 +201,14 @@ function resolveIntakeOrigin(
// The reach question comes first, before any stat or read, so neither a
// refused stat nor the size ceiling can name a page the caller may not read.
if (only !== undefined) {
- return isSourceHidden(vault, only, readable)
- ? UNTRUSTED_ORIGIN
- : classifySourceOrigin(vault, only);
+ if (isSourceHidden(vault, only, readable)) return UNTRUSTED_ORIGIN;
+ const origin = classifySourceOrigin(vault, only);
+ // The digest lands on entity pages, which do not inherit the source's
+ // visibility: a source a remote reader may not read keeps its lane but
+ // records no digest, or the digest would confirm a guess of its bytes.
+ return origin.contentHash !== undefined && isSourceHidden(vault, only, remotelyReadable(vault))
+ ? { trust: origin.trust }
+ : origin;
}
// `some` rather than a full map: the first source outside the vault (or
@@ -213,6 +222,12 @@ function resolveIntakeOrigin(
return { trust: untrusted ? INTAKE_TRUST.untrusted : INTAKE_TRUST.trusted };
}
+/** May a remote reader read the vault file at this vault-relative path? */
+function remotelyReadable(vault: string): (rel: string) => boolean {
+ const cache: FrontmatterCache = new Map();
+ return (rel) => isPathReadableAtReach(vault, rel, TRANSPORT_REACH.remote, cache);
+}
+
/**
* Intake a typed extraction into the entity registry. Validates the full
* payload first (throwing {@link IntakeValidationError} with no write on
diff --git a/src/core/brain/lessons.ts b/src/core/brain/lessons.ts
index d90658cc..e407d6d0 100644
--- a/src/core/brain/lessons.ts
+++ b/src/core/brain/lessons.ts
@@ -40,8 +40,9 @@ import { readGeneratedAt } from "./active.ts";
import { readerRefView } from "./artifact-ref-view.ts";
import { decayWeight } from "./continuity/usage-signal.ts";
import { listDeadEnds, type DeadEndEntry } from "./dead-ends.ts";
+import { logEntryAtReach } from "./log-events-at-reach.ts";
import { listLogDates, readLogDay } from "./log-jsonl.ts";
-import { logEntryArtifactRefs, type BrainLogEntry } from "./log.ts";
+import type { BrainLogEntry } from "./log.ts";
import { brainDirs, brainDirsForWrite, brainLessonsPath, vaultRelative } from "./paths.ts";
import { parsePreference } from "./preference.ts";
import {
@@ -435,7 +436,7 @@ function renderLessons(vault: string, opts: RenderLessonsOptions): LessonsRender
limit,
...(refs.filtersNothing
? {}
- : { keepEvent: (e: BrainLogEntry) => refs.row(...logEntryArtifactRefs(e)) }),
+ : { keepEvent: (e: BrainLogEntry) => logEntryAtReach(refs, e) !== null }),
},
);
diff --git a/src/core/brain/link-graph/moc-audit.ts b/src/core/brain/link-graph/moc-audit.ts
index 8faaf4ac..9f6e0d2a 100644
--- a/src/core/brain/link-graph/moc-audit.ts
+++ b/src/core/brain/link-graph/moc-audit.ts
@@ -25,9 +25,10 @@
import { existsSync, readFileSync } from "node:fs";
import { join } from "node:path";
+import { FRONTMATTER_RE } from "../../vault.ts";
import { buildBacklinkIndex } from "../backlinks.ts";
import { ownerScopeView } from "../owner-scope-view.ts";
-import { brainDirs } from "../paths.ts";
+import { brainDirs, vaultRelative } from "../paths.ts";
import { BRAIN_LINK_GRAPH_DEFAULTS, loadBrainConfig, resolveLinkGraph } from "../policy.ts";
import { normaliseWikilinkTarget } from "../wikilink.ts";
import { extractWikilinkRichBodies, parseWikilinkRich } from "./parse-wikilink.ts";
@@ -86,6 +87,14 @@ export interface AuditMocOptions {
* see, so both follow the member out.
*/
readonly ownerScope?: string | null;
+ /**
+ * May the caller read the page at this vault-relative path? Absent
+ * reads every page. A hub it refuses answers as an absent hub, and a
+ * member it refuses is dropped exactly as an owner-hidden member is:
+ * not bucketed, not sized, and its links not counted toward a missing
+ * target's `referenceCount`.
+ */
+ readonly readable?: (rel: string) => boolean;
}
/** The structural hub thresholds (from `link_graph` config unless overridden). */
@@ -181,13 +190,17 @@ export function resolveHubThresholds(vault: string): HubBodyThresholds {
};
}
+/** The {@link AuditMocOptions.readable} answer when none is given. */
+const READ_EVERY_PAGE = (): boolean => true;
+
export function auditMoc(vault: string, hubId: string, opts: AuditMocOptions = {}): MocAuditReport {
const hubCanonical = normaliseWikilinkTarget(hubId);
const view = ownerScopeView(vault, opts.ownerScope ?? null);
const hubPath = locateArtifact(vault, hubCanonical);
+ const readable = opts.readable ?? READ_EVERY_PAGE;
// A hub the caller may not see reads as a hub that is not there: the
// same sentence an absent hub produces, so the two are indistinguishable.
- if (!hubPath || !view.visible(hubCanonical)) {
+ if (!hubPath || !view.visible(hubCanonical) || !readable(vaultRelative(hubPath, vault))) {
throw new MocAuditError(`hub note not found: ${hubCanonical}`);
}
@@ -220,7 +233,9 @@ export function auditMoc(vault: string, hubId: string, opts: AuditMocOptions = {
const memberPath = locateArtifact(vault, target);
// A member this caller may not see leaves no trace in any bucket -
// not even the missing one, which would decide exists-vs-absent for it.
- if (memberPath && !view.visible(target)) continue;
+ if (memberPath && (!view.visible(target) || !readable(vaultRelative(memberPath, vault)))) {
+ continue;
+ }
if (!memberPath) {
missingCounts.set(target, (missingCounts.get(target) ?? 0) + 1);
continue;
@@ -255,7 +270,7 @@ export function auditMoc(vault: string, hubId: string, opts: AuditMocOptions = {
if (!missingCounts.has(target)) continue;
for (const member of outboundTargets) {
const path = locateArtifact(vault, member);
- if (!path || !view.visible(member)) continue;
+ if (!path || !view.visible(member) || !readable(vaultRelative(path, vault))) continue;
const body = stripFrontmatter(readFileSync(path, "utf8"));
for (const bracketBody of extractWikilinkRichBodies(body)) {
const t = parseWikilinkRich(bracketBody).target;
@@ -294,7 +309,7 @@ function locateArtifact(vault: string, id: string): string | null {
}
function stripFrontmatter(text: string): string {
- const m = /^---\s*\n([\s\S]*?)\n---\s*\n?/.exec(text);
+ const m = FRONTMATTER_RE.exec(text);
if (!m) return text;
return text.slice(m[0].length);
}
diff --git a/src/core/brain/link-graph/unlinked-mentions.ts b/src/core/brain/link-graph/unlinked-mentions.ts
index 70999e15..eb8a6ed9 100644
--- a/src/core/brain/link-graph/unlinked-mentions.ts
+++ b/src/core/brain/link-graph/unlinked-mentions.ts
@@ -21,7 +21,7 @@
import { existsSync, readdirSync, readFileSync } from "node:fs";
import { join } from "node:path";
-import { parseFrontmatter } from "../../vault.ts";
+import { FRONTMATTER_RE, parseFrontmatter } from "../../vault.ts";
import { ownerScopeView } from "../owner-scope-view.ts";
import { UNFILTERED_ARTIFACT_REFS, type ArtifactRefView } from "../artifact-ref-view.ts";
import { brainDirs } from "../paths.ts";
@@ -179,7 +179,7 @@ function scanDir(
}
function stripFrontmatter(text: string): string {
- const m = /^---\s*\n([\s\S]*?)\n---\s*\n?/.exec(text);
+ const m = FRONTMATTER_RE.exec(text);
if (!m) return text;
return text.slice(m[0].length);
}
diff --git a/src/core/brain/log-events-at-reach.ts b/src/core/brain/log-events-at-reach.ts
new file mode 100644
index 00000000..7c598846
--- /dev/null
+++ b/src/core/brain/log-events-at-reach.ts
@@ -0,0 +1,175 @@
+/**
+ * One rule for every reader that renders Brain log events to a caller
+ * who may not read every record those events name.
+ *
+ * A log event names records by id: an evidence event its preference and
+ * the artifact the rule was applied to, a dream the preferences it
+ * created, confirmed and retired, any other event the record it scopes
+ * to. Below local reach a reader shows the caller the log as if a record
+ * it cannot read had never been logged:
+ *
+ * - an event naming such a record is dropped, and nothing counts it;
+ * - a dream is the exception, because it is shared: a vault that never
+ * had the withheld record still logged the dream for the others. It
+ * is kept while at least one of its transitions is readable, and it
+ * shows only those. Dropping the whole dream was an oracle of its own
+ * (a count fell when a reserved preference shared a dream).
+ *
+ * A record is judged under every spelling it answers to across its
+ * retirement ({@link recordRefs}), so a transition logged as `pref-x`
+ * cannot pass once `ret-x` is the reserved page on disk.
+ *
+ * The rule is asked through an {@link ArtifactRefView}, so the per-path
+ * decision stays with the reach predicate and the ownership predicate;
+ * this module adds no visibility rule of its own.
+ */
+
+import type { ArtifactRef, ArtifactRefView } from "./artifact-ref-view.ts";
+import { PREF_ID_PREFIX, RETIRED_ID_PREFIX } from "./dream-plan.ts";
+import { logEntryArtifactRefs, type BrainLogEntry, type BrainLogEntryPayload } from "./log.ts";
+import { isBrainLogRel } from "./paths.ts";
+import { extractId } from "./temporal/period-common.ts";
+import type { DreamSummarySlots, TemporalEvent } from "./temporal/types.ts";
+import { BRAIN_LOG_EVENT_KIND } from "./types.ts";
+import { stripBrainIdPrefix } from "./wikilink.ts";
+
+/**
+ * Every spelling a preference record answers to across its retirement:
+ * `pref-<slug>` before `moveToRetired` renamed it and `ret-<slug>` after.
+ * A row naming either must be judged by whichever page is on disk, or a
+ * transition logged under `pref-x` would name nothing once `ret-x` is the
+ * reserved page, and pass.
+ */
+export function recordRefs(id: string | undefined): ReadonlyArray<ArtifactRef> {
+ if (id === undefined) return [];
+ const slug = stripBrainIdPrefix(id);
+ return [id, `${PREF_ID_PREFIX}${slug}`, `${RETIRED_ID_PREFIX}${slug}`];
+}
+
+/** May the reader see one dream transition, spelled as the dream logged it? */
+export function transitionReadable(refs: ArtifactRefView, link: string): boolean {
+ return refs.row(link, ...recordRefs(extractId(link)));
+}
+
+/** The dream body keys that carry transitions, as the dream pass logs them. */
+const DREAM_TRANSITION_KEYS = Object.freeze(["new_unconfirmed", "confirmed", "retired"] as const);
+
+/** The {@link DreamSummarySlots} keys, in the order of {@link DREAM_TRANSITION_KEYS}. */
+const DREAM_SUMMARY_SLOTS = Object.freeze(["newUnconfirmed", "confirmed", "retired"] as const);
+
+/**
+ * Trim transition lists to the readable links. Answers `null` when the
+ * lists held at least one transition and none is readable (the dream is
+ * dropped), the input itself when nothing was withheld, and otherwise the
+ * trimmed lists with every emptied list left out, as a dream that never
+ * named the withheld record would have logged it.
+ */
+function trimTransitions<K extends string>(
+ refs: ArtifactRefView,
+ lists: ReadonlyArray<readonly [K, ReadonlyArray<string> | undefined]>,
+): ReadonlyMap<K, ReadonlyArray<string>> | null | "unchanged" {
+ let total = 0;
+ let kept = 0;
+ const out = new Map<K, ReadonlyArray<string>>();
+ for (const [key, links] of lists) {
+ if (links === undefined) continue;
+ const readable = links.filter((link) => transitionReadable(refs, link));
+ total += links.length;
+ kept += readable.length;
+ if (readable.length > 0) out.set(key, Object.freeze(readable));
+ }
+ if (kept === total) return "unchanged";
+ return kept === 0 ? null : out;
+}
+
+/**
+ * The event as a reader behind `refs` may see it, or `null` when it names
+ * a record the reader cannot read. See the module docblock for the rule.
+ *
+ * The page an event was read from is asked too, except a Brain log page:
+ * every log event comes from one, and its own verdict below local reach
+ * is the reason this per-event rule exists.
+ */
+export function eventAtReach(refs: ArtifactRefView, ev: TemporalEvent): TemporalEvent | null {
+ if (refs.filtersNothing) return ev;
+ if (!isBrainLogRel(ev.source.path) && !refs.visible(ev.source.path)) return null;
+ if (ev.kind === BRAIN_LOG_EVENT_KIND.applyEvidence) {
+ return refs.row(ev.artifact, ...recordRefs(ev.prefId)) ? ev : null;
+ }
+ if (ev.kind !== BRAIN_LOG_EVENT_KIND.dream) {
+ return refs.row(...recordRefs(ev.prefId)) ? ev : null;
+ }
+ const summary = ev.dreamSummary;
+ if (summary === undefined) return ev;
+ const trimmed = trimTransitions(
+ refs,
+ DREAM_SUMMARY_SLOTS.map((slot) => [slot, summary[slot]] as const),
+ );
+ if (trimmed === "unchanged") return ev;
+ if (trimmed === null) return null;
+ const dreamSummary: DreamSummarySlots = Object.freeze(Object.fromEntries(trimmed));
+ return Object.freeze({ ...ev, dreamSummary });
+}
+
+/**
+ * A dream log entry as a reader behind `refs` may see it: its transition
+ * lists trimmed to the readable links, or `null` when it named
+ * transitions and none is readable. Every other entry is returned as it
+ * is; the caller still asks its own row rule over the result, which for a
+ * trimmed dream no longer names the withheld record.
+ */
+export function dreamEntryAtReach(
+ refs: ArtifactRefView,
+ entry: BrainLogEntry,
+): BrainLogEntry | null {
+ if (refs.filtersNothing || entry.eventType !== BRAIN_LOG_EVENT_KIND.dream) return entry;
+ const body = entry.body;
+ const trimmed = trimTransitions(
+ refs,
+ DREAM_TRANSITION_KEYS.map((key) => {
+ const value = body[key];
+ return [key, Array.isArray(value) ? value : undefined] as const;
+ }),
+ );
+ if (trimmed === "unchanged") return entry;
+ if (trimmed === null) return null;
+ // Rebuilt in the body's own key order, so a trimmed dream serialises
+ // as the dream that never named the withheld record.
+ const trimmedBody: Record<string, string | ReadonlyArray<string>> = {};
+ for (const [key, value] of Object.entries(body)) {
+ if (!(DREAM_TRANSITION_KEYS as ReadonlyArray<string>).includes(key)) {
+ trimmedBody[key] = value;
+ continue;
+ }
+ const links = trimmed.get(key as (typeof DREAM_TRANSITION_KEYS)[number]);
+ if (links !== undefined) trimmedBody[key] = links;
+ }
+ return Object.freeze({ ...entry, body: Object.freeze(trimmedBody) as BrainLogEntryPayload });
+}
+
+/**
+ * A log entry as a reader behind `refs` may see it, or `null`: a dream is
+ * trimmed to its readable transitions ({@link dreamEntryAtReach}), and the
+ * entry must then name nothing the reader cannot read, every id-shaped
+ * reference judged under each spelling its record answers to
+ * ({@link recordRefs}). Without the spellings, `[[pref-x|rule]]` named no
+ * page once `ret-x` was the reserved page on disk, and passed.
+ */
+export function logEntryAtReach(refs: ArtifactRefView, entry: BrainLogEntry): BrainLogEntry | null {
+ if (refs.filtersNothing) return entry;
+ const shown = dreamEntryAtReach(refs, entry);
+ if (shown === null) return null;
+ const named = logEntryArtifactRefs(shown).flatMap((ref) =>
+ ref === undefined ? [] : [ref, ...recordRefs(extractId(ref))],
+ );
+ return refs.row(...named) ? shown : null;
+}
+
+/** The entries a reader behind `refs` may see, each in the form it may see it, in order. */
+export function logEntriesAtReach(
+ refs: ArtifactRefView,
+ entries: ReadonlyArray<BrainLogEntry>,
+): ReadonlyArray<BrainLogEntry> {
+ if (refs.filtersNothing) return entries;
+ return entries.flatMap((entry) => logEntryAtReach(refs, entry) ?? []);
+}
diff --git a/src/core/brain/log.ts b/src/core/brain/log.ts
index 026102ae..e2f2c198 100644
--- a/src/core/brain/log.ts
+++ b/src/core/brain/log.ts
@@ -418,6 +418,15 @@ function renderFileHeader(date: string): string {
return lines.join("\n") + "\n";
}
+/**
+ * One parsed entry rendered as the markdown block {@link appendLogEvent}
+ * writes for it, so a reader that drops or trims entries can serve the
+ * rest in the shape the shard would have had without them.
+ */
+export function renderLogEntryBlock(entry: BrainLogEntry): string {
+ return renderEventBlock(entry, parseIsoUtc(entry.timestamp).hms);
+}
+
function renderEventBlock(event: BrainLogEntry, hms: string): string {
const lines: string[] = [`## ${hms}Z — ${event.eventType}`];
// Stable iteration order: keys in the order they appear in the
diff --git a/src/core/brain/path-constants.ts b/src/core/brain/path-constants.ts
index 2db36d68..8433e3aa 100644
--- a/src/core/brain/path-constants.ts
+++ b/src/core/brain/path-constants.ts
@@ -299,6 +299,18 @@ export function isCompiledDigestRel(relPath: string): boolean {
return [BRAIN_ACTIVE_FILE, BRAIN_LESSONS_FILE].some((file) => foldSegment(file) === name);
}
+/**
+ * Does `relPath` name a file under the Brain log ({@link BRAIN_LOG_REL}),
+ * in any spelling the filesystem resolves to it? The daily log pages name
+ * preference ids, retired ids and evidence artifacts in their event bodies
+ * and carry no visibility of their own, so, like a compiled digest page,
+ * they cannot say that a record they name is reserved. Read with the same
+ * normalization and case-folding as {@link brainRootSegments}.
+ */
+export function isBrainLogRel(relPath: string): boolean {
+ return isInBrainLane(relPath, [BRAIN_LOG_REL]);
+}
+
/** Does `relPath` land at or under `Brain/`? See {@link brainRootSegments}. */
export function isUnderBrainRoot(relPath: string): boolean {
return brainRootSegments(relPath) !== null;
diff --git a/src/core/brain/provenance/capture-scope.ts b/src/core/brain/provenance/capture-scope.ts
index 3624297e..1cc447c5 100644
--- a/src/core/brain/provenance/capture-scope.ts
+++ b/src/core/brain/provenance/capture-scope.ts
@@ -31,6 +31,7 @@ import { createHash } from "node:crypto";
import { posix } from "node:path";
import type { FrontmatterMap } from "../../types.ts";
+import { FENCE_CHAR, fenceFor, MIN_FENCE_LENGTH } from "../../markdown-fence.ts";
import {
classifySourceTrust,
normalizeSourceIdentity,
@@ -90,15 +91,6 @@ export const CAPTURE_EXCERPT_HEADING = "## Excerpt";
*/
export const CAPTURE_EXCERPT_FENCE_INFO = "excerpt";
-/** The fence character; a backtick fence admits any info string. */
-const FENCE_CHAR = "`";
-
-/** The shortest fence CommonMark recognises. */
-const MIN_FENCE_LENGTH = 3;
-
-/** Every run of the fence character, to size a fence that contains them all. */
-const FENCE_CHAR_RUN_RE = /`+/g;
-
/** Line separator of a rendered section. */
const NEWLINE = "\n";
@@ -174,13 +166,6 @@ export function excerptDigest(excerpt: string): string {
return createHash("sha256").update(excerpt, "utf8").digest("hex");
}
-/** A backtick fence longer than every backtick run in `text`. */
-function fenceFor(text: string): string {
- let longest = 0;
- for (const run of text.matchAll(FENCE_CHAR_RUN_RE)) longest = Math.max(longest, run[0].length);
- return FENCE_CHAR.repeat(Math.max(MIN_FENCE_LENGTH, longest + 1));
-}
-
/**
* The excerpt section: the heading, a blank line, then a fenced block whose
* fence is longer than any backtick run in the excerpt. One newline always
diff --git a/src/core/brain/session-recall.ts b/src/core/brain/session-recall.ts
index afd63cab..5dd1a66f 100644
--- a/src/core/brain/session-recall.ts
+++ b/src/core/brain/session-recall.ts
@@ -2,6 +2,7 @@ import { createHash } from "node:crypto";
import { charSpanToLineSpan } from "../search/line-numbering.ts";
import { caseInsensitiveIndex, NO_MATCH_OFFSET, snippetAround } from "../search/snippet-window.ts";
+import { privateRegionTexts } from "../redactor.ts";
import {
appendContinuityRecord,
clipPayloadToBudget,
@@ -421,9 +422,6 @@ function loadSessionPayloadPolicy(vault: string): ResolvedSessionPayloadPolicy {
}
}
-/** A `<private>` opening tag, as the continuity sanitiser detects one. */
-const PRIVATE_OPEN_RE = /<private\b[^>]*>/i;
-
function importRawTurn(
vault: string,
sessionId: string,
@@ -473,7 +471,9 @@ function appendTurnRow(
sourceRefs: sourceRefs(sessionId, turn.turnId),
// The registry strips `<private>` regions before it writes anything,
// which leaves the store nothing to detect; the verdict is carried.
- ...(externalized.payloads.length > 0 && PRIVATE_OPEN_RE.test(original)
+ // It is the redactor's own region detection, so a region the registry
+ // stripped (an unclosed tag included) is the region that flags the row.
+ ...(externalized.payloads.length > 0 && privateRegionTexts(original).length > 0
? { private: true }
: {}),
payload: {
diff --git a/src/core/brain/source-cleanup.ts b/src/core/brain/source-cleanup.ts
index a79ec928..7031077b 100644
--- a/src/core/brain/source-cleanup.ts
+++ b/src/core/brain/source-cleanup.ts
@@ -60,6 +60,7 @@ import {
brainDirs,
} from "./paths.ts";
import { BRAIN_SOURCE_KIND } from "./ingest/ingest.ts";
+import { isSourceHidden } from "./intake/source-trust.ts";
import { manifestPath, readManifest, writeManifestAtomic } from "./ingest/content-manifest.ts";
import { appendContinuitySourceInvalidation } from "./continuity/store.ts";
import { isoSecond } from "./time.ts";
@@ -184,10 +185,12 @@ export interface DeleteBySourceOptions {
/** Agent identity recorded in the audit reason. */
readonly agent?: string;
/**
- * May the caller read the original at this vault-relative path? Asked
- * before the original is stat-ed, so one it may not read is planned,
- * counted and left on disk exactly as an absent one would be. Absent
- * includes every original (the CLI, which reads the vault directly).
+ * May the caller read the page or the original at this vault-relative
+ * path? Asked before anything is stat-ed or deleted, so a derived page,
+ * a mention, an original or a manifest entry whose source it may not
+ * read is planned, counted and left on disk exactly as an absent one
+ * would be. Absent includes everything (the CLI, which reads the vault
+ * directly).
*/
readonly include?: (rel: string) => boolean;
}
@@ -685,9 +688,19 @@ export function deleteBySource(
// Dry-run (the default) plans and writes nothing, so it stays ungated.
if (confirm) assertVaultIdentityForWrite(vault);
- const { derived, mentions } = traceReferences(vault, identity);
+ // A page or a source the caller may not read is planned, counted and
+ // left on disk exactly as an absent one: a remote caller can neither
+ // learn of a reserved summary page nor delete it.
+ const included = (rel: string): boolean => opts.include?.(rel) !== false;
+ const traced = traceReferences(vault, identity);
+ const derived = traced.derived.filter((entry) => included(entry.path));
+ const mentions = traced.mentions.filter((entry) => included(entry.path));
const originals = findOriginals(vault, canonical, opts.include);
- const manifestKey = readManifest(vault).entries[canonical] !== undefined ? canonical : null;
+ const manifestKey =
+ readManifest(vault).entries[canonical] !== undefined &&
+ !isSourceHidden(vault, canonical, opts.include)
+ ? canonical
+ : null;
const blastRadius = derived.length + mentions.length + originals.length;
/**
diff --git a/src/core/brain/today-dashboard.ts b/src/core/brain/today-dashboard.ts
index 9bd75e1f..31c24fe9 100644
--- a/src/core/brain/today-dashboard.ts
+++ b/src/core/brain/today-dashboard.ts
@@ -29,6 +29,7 @@ import { scanOpenLoops, type OpenLoopScan } from "./open-loops.ts";
import { isoSecond } from "./time.ts";
import { buildActivityTimeline, type ActivityTimeline } from "./temporal/activity-timeline.ts";
import { buildTimelineIndex } from "./temporal/build-index.ts";
+import type { TemporalEvent, TimelineIndex } from "./temporal/types.ts";
/** Options accepted by {@link buildTodayDashboard}. */
export interface TodayDashboardOptions {
@@ -38,6 +39,13 @@ export interface TodayDashboardOptions {
readonly activityLookbackDays?: number;
/** Max recent-activity entries to keep, newest first. Default 20. */
readonly activityLimit?: number;
+ /**
+ * The form of one log event the caller may see, or `null` when it may
+ * not see it at all. Applied to the recent-activity events before the
+ * limit and the totals, so a withheld event moves neither. Omitted, every
+ * event is shown as logged (the operator's own shell).
+ */
+ readonly eventAtReach?: (ev: TemporalEvent) => TemporalEvent | null;
}
/** One obligation row surfaced on the dashboard. */
@@ -176,10 +184,44 @@ function buildRecentActivitySection(
now: Date,
lookbackDays: number,
limit: number,
+ eventAtReach: TodayDashboardOptions["eventAtReach"],
): ActivityTimeline {
const since = isoSecond(new Date(now.getTime() - lookbackDays * DAY_MS));
const index = buildTimelineIndex(vault, { now, since });
- return buildActivityTimeline(index, { now, since, limit });
+ const shown = eventAtReach === undefined ? index : indexAtReach(index, eventAtReach);
+ return buildActivityTimeline(shown, { now, since, limit });
+}
+
+/**
+ * The index holding only the events the caller may see, each in its
+ * visible form, regrouped so every lookup the timeline builder may take
+ * agrees with the event list.
+ */
+function indexAtReach(
+ index: TimelineIndex,
+ eventAtReach: (ev: TemporalEvent) => TemporalEvent | null,
+): TimelineIndex {
+ const events = index.events.flatMap((ev) => eventAtReach(ev) ?? []);
+ const group = <K>(
+ key: (ev: TemporalEvent) => K | undefined,
+ ): ReadonlyMap<K, ReadonlyArray<TemporalEvent>> => {
+ const out = new Map<K, TemporalEvent[]>();
+ for (const ev of events) {
+ const k = key(ev);
+ if (k === undefined) continue;
+ const bucket = out.get(k);
+ if (bucket === undefined) out.set(k, [ev]);
+ else bucket.push(ev);
+ }
+ return out;
+ };
+ return Object.freeze({
+ events: Object.freeze(events),
+ eventsByKind: group((ev) => ev.kind),
+ eventsByPrefId: group((ev) => ev.prefId),
+ eventsByTopic: group((ev) => ev.topic),
+ window: index.window,
+ });
}
function computeTotals(
@@ -289,7 +331,7 @@ export function buildTodayDashboard(vault: string, opts: TodayDashboardOptions):
const recentActivity = computeSection(
"recentActivity",
errors,
- () => buildRecentActivitySection(vault, opts.now, lookbackDays, limit),
+ () => buildRecentActivitySection(vault, opts.now, lookbackDays, limit, opts.eventAtReach),
EMPTY_ACTIVITY_TIMELINE,
);
const totals = computeSection(
diff --git a/src/core/markdown-fence.ts b/src/core/markdown-fence.ts
new file mode 100644
index 00000000..cbe8153b
--- /dev/null
+++ b/src/core/markdown-fence.ts
@@ -0,0 +1,24 @@
+/**
+ * Fenced code blocks that hold arbitrary text safely.
+ *
+ * A fence must be longer than every backtick run inside the block, or a run
+ * in the text closes the block early and what follows it is read as
+ * Markdown. One helper, shared by every section that fences text it did not
+ * write (the capture excerpt, the HTML parts list, the table note).
+ */
+
+/** The fence character; a backtick fence admits any info string. */
+export const FENCE_CHAR = "`";
+
+/** The shortest fence CommonMark recognises. */
+export const MIN_FENCE_LENGTH = 3;
+
+/** Every run of the fence character, to size a fence that contains them all. */
+const FENCE_CHAR_RUN_RE = /`+/g;
+
+/** A backtick fence longer than every backtick run in `text`. */
+export function fenceFor(text: string): string {
+ let longest = 0;
+ for (const run of text.matchAll(FENCE_CHAR_RUN_RE)) longest = Math.max(longest, run[0].length);
+ return FENCE_CHAR.repeat(Math.max(MIN_FENCE_LENGTH, longest + 1));
+}
diff --git a/src/core/redactor.ts b/src/core/redactor.ts
index 815a0c23..eb86c2bf 100644
--- a/src/core/redactor.ts
+++ b/src/core/redactor.ts
@@ -104,7 +104,15 @@ export function wasScanTruncated(text: string): boolean {
return typeof text === "string" && text.includes(SCAN_TRUNCATED_SENTINEL);
}
-const PRIVATE_OPEN_TAG_RE = /<private\b[^>]*>/gi;
+/**
+ * An open tag's attributes stop at the next `<` as well as at `>`: with
+ * `[^>]*` a run of `<private ` with no closing bracket made every attempt
+ * scan to the end of the text, quadratic in its length. A tag whose
+ * attributes reach a `<` or the end of the text still opens a region
+ * there, so the bound never narrows what counts as private: the rule
+ * fails closed and stays linear.
+ */
+const PRIVATE_OPEN_TAG_RE = /<private\b[^<>]*(?:>|(?=<)|$)/gi;
const PRIVATE_CLOSE_TAG_RE = /<\/private>/gi;
/**
@@ -739,46 +747,65 @@ function quoteRedactedFrontmatter(text: string): string {
);
}
-export function stripPrivateRegions(text: string): string {
- if (!text) return text;
+/** One outermost private region: `[start, end)`, tags included. */
+interface PrivateRegionSpan {
+ readonly start: number;
+ readonly end: number;
+}
- let output = "";
+/**
+ * Every outermost `<private>` region of `text`, in order. Nested open tags
+ * deepen the region; an unclosed region runs to the end of the text and
+ * ends the walk. Linear: the next open and the next close are each found
+ * once and reused until the scan passes them, so a run of nested opens
+ * before one far close no longer re-scans to that close per open.
+ */
+function privateRegionSpans(text: string): PrivateRegionSpan[] {
+ const spans: PrivateRegionSpan[] = [];
+ const open = new RegExp(PRIVATE_OPEN_TAG_RE.source, "gi");
+ const close = new RegExp(PRIVATE_CLOSE_TAG_RE.source, "gi");
+ const nextAt = (re: RegExp, from: number): RegExpExecArray | null => {
+ re.lastIndex = from;
+ return re.exec(text);
+ };
let cursor = 0;
- PRIVATE_OPEN_TAG_RE.lastIndex = 0;
- PRIVATE_CLOSE_TAG_RE.lastIndex = 0;
-
while (cursor < text.length) {
- PRIVATE_OPEN_TAG_RE.lastIndex = cursor;
- const openMatch = PRIVATE_OPEN_TAG_RE.exec(text);
- if (!openMatch) {
- output += text.slice(cursor);
- break;
- }
-
- output += text.slice(cursor, openMatch.index);
- output += PRIVATE_REGION_PLACEHOLDER;
-
+ const first = nextAt(open, cursor);
+ if (!first) break;
let depth = 1;
- let scan = PRIVATE_OPEN_TAG_RE.lastIndex;
+ let scan = first.index + first[0].length;
+ let nextOpen = nextAt(open, scan);
+ let nextClose = nextAt(close, scan);
while (depth > 0) {
- PRIVATE_OPEN_TAG_RE.lastIndex = scan;
- PRIVATE_CLOSE_TAG_RE.lastIndex = scan;
- const nextOpen = PRIVATE_OPEN_TAG_RE.exec(text);
- const nextClose = PRIVATE_CLOSE_TAG_RE.exec(text);
- if (!nextClose) return output;
-
+ if (!nextClose) {
+ spans.push({ start: first.index, end: text.length });
+ return spans;
+ }
if (nextOpen && nextOpen.index < nextClose.index) {
depth += 1;
- scan = PRIVATE_OPEN_TAG_RE.lastIndex;
+ scan = nextOpen.index + nextOpen[0].length;
} else {
depth -= 1;
- scan = PRIVATE_CLOSE_TAG_RE.lastIndex;
+ scan = nextClose.index + nextClose[0].length;
}
+ if (nextOpen && nextOpen.index < scan) nextOpen = nextAt(open, scan);
+ if (nextClose.index < scan) nextClose = nextAt(close, scan);
}
+ spans.push({ start: first.index, end: scan });
cursor = scan;
}
+ return spans;
+}
- return output;
+export function stripPrivateRegions(text: string): string {
+ if (!text) return text;
+ let output = "";
+ let cursor = 0;
+ for (const span of privateRegionSpans(text)) {
+ output += text.slice(cursor, span.start) + PRIVATE_REGION_PLACEHOLDER;
+ cursor = span.end;
+ }
+ return output + text.slice(cursor);
}
/**
@@ -789,39 +816,8 @@ export function stripPrivateRegions(text: string): string {
* slice carries private text whose tags fell outside the slice.
*/
export function privateRegionTexts(text: string): string[] {
- const regions: string[] = [];
- if (!text) return regions;
- const open = new RegExp(PRIVATE_OPEN_TAG_RE.source, "gi");
- const close = new RegExp(PRIVATE_CLOSE_TAG_RE.source, "gi");
- let cursor = 0;
- while (cursor < text.length) {
- open.lastIndex = cursor;
- const openMatch = open.exec(text);
- if (!openMatch) break;
- const start = openMatch.index;
- let depth = 1;
- let scan = open.lastIndex;
- while (depth > 0) {
- open.lastIndex = scan;
- close.lastIndex = scan;
- const nextOpen = open.exec(text);
- const nextClose = close.exec(text);
- if (!nextClose) {
- regions.push(text.slice(start));
- return regions;
- }
- if (nextOpen && nextOpen.index < nextClose.index) {
- depth += 1;
- scan = open.lastIndex;
- } else {
- depth -= 1;
- scan = close.lastIndex;
- }
- }
- regions.push(text.slice(start, scan));
- cursor = scan;
- }
- return regions;
+ if (!text) return [];
+ return privateRegionSpans(text).map((span) => text.slice(span.start, span.end));
}
export interface RedactRawOutputOptions {
diff --git a/src/core/search/chunker.ts b/src/core/search/chunker.ts
index 1dccf3b3..e51c9770 100644
--- a/src/core/search/chunker.ts
+++ b/src/core/search/chunker.ts
@@ -29,7 +29,12 @@
*/
export const CHUNKER_VERSION = 2;
-const DEFAULT_MAX_TOKENS = 800;
+/**
+ * Token budget of one chunk when the caller names none. Exported so a
+ * renderer that sizes blocks to fit one chunk (the table note's row
+ * groups) states its budget against this one rather than a copy.
+ */
+export const DEFAULT_CHUNK_MAX_TOKENS = 800;
const DEFAULT_MIN_TOKENS = 100;
const DEFAULT_OVERLAP_TOKENS = 100;
@@ -154,7 +159,12 @@ function walkTokens(text: string, starts: number[] | null): number {
return count;
}
-function countTokens(text: string): number {
+/**
+ * The chunker's token count of `text`, the same count every budget in this
+ * module is measured with. Exported so a renderer can size a block to fit
+ * one chunk without re-deriving the rules.
+ */
+export function countChunkTokens(text: string): number {
if (text.length === 0) return 0;
return walkTokens(text, null);
}
@@ -176,7 +186,7 @@ function tailTokens(text: string, budget: number): string {
function tokensOfLines(lines: ReadonlyArray<Line>): number {
let total = 0;
- for (const l of lines) total += countTokens(l.text);
+ for (const l of lines) total += countChunkTokens(l.text);
return total;
}
@@ -371,7 +381,7 @@ function tailLines(lines: ReadonlyArray<Line>, budget: number): Line[] {
const tail: Line[] = [];
for (let i = lines.length - 1; i >= 0; i--) {
const line = lines[i]!;
- const t = countTokens(line.text);
+ const t = countChunkTokens(line.text);
if (acc + t > budget) {
if (tail.length === 0) {
const cut = tailTokens(line.text, budget);
@@ -407,7 +417,7 @@ function emitChunk(
? tailLines(overlap, maxTokens - draft.tokens)
: overlap;
const allText = [...fitted.map((l) => l.text), ...bodyLines.map((l) => l.text)].join("\n");
- const tokenCount = countTokens(allText);
+ const tokenCount = countChunkTokens(allText);
return Object.freeze({
chunkIndex: index,
content: allText,
@@ -483,7 +493,7 @@ function splitLine(text: string, budget: number): string[] {
const parts: string[] = [];
let acc = "";
for (const s of sentences(text)) {
- if (countTokens(s) > budget) {
+ if (countChunkTokens(s) > budget) {
if (acc !== "") parts.push(acc);
const pieces = hardCut(s, budget);
parts.push(...pieces.slice(0, -1));
@@ -491,7 +501,7 @@ function splitLine(text: string, budget: number): string[] {
continue;
}
const merged = acc + s;
- if (acc !== "" && countTokens(merged) > budget) {
+ if (acc !== "" && countChunkTokens(merged) > budget) {
parts.push(acc);
acc = s;
} else {
@@ -511,7 +521,7 @@ function splitLine(text: string, budget: number): string[] {
function splitBlock(block: Block, budget: number): Block[] {
const units: Block[] = [];
for (const line of block.lines) {
- const t = countTokens(line.text);
+ const t = countChunkTokens(line.text);
if (t <= budget) {
units.push({ kind: block.kind, lines: [line], tokenCount: t });
continue;
@@ -520,7 +530,7 @@ function splitBlock(block: Block, budget: number): Block[] {
units.push({
kind: block.kind,
lines: [{ num: line.num, text: part }],
- tokenCount: countTokens(part),
+ tokenCount: countChunkTokens(part),
});
}
}
@@ -696,7 +706,7 @@ export function chunkMarkdown(
filenameBase: string | null,
opts?: ChunkOptions,
): ChunkResult {
- const maxTokens = opts?.maxTokens ?? DEFAULT_MAX_TOKENS;
+ const maxTokens = opts?.maxTokens ?? DEFAULT_CHUNK_MAX_TOKENS;
const minTokens = opts?.minTokens ?? DEFAULT_MIN_TOKENS;
const overlapTokens = opts?.overlapTokens ?? DEFAULT_OVERLAP_TOKENS;
diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.ts
index 8fb97cf3..52f35953 100644
--- a/src/core/search/indexer.ts
+++ b/src/core/search/indexer.ts
@@ -1688,12 +1688,15 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexSt
* statement - see {@link indexRootCoverage}.
*/
export interface IndexRootCoverage {
- /** Roots carrying at least one indexed document. */
+ /** Roots carrying at least one indexed (and, when filtered, admitted) document. */
readonly rootsWithDocuments: ReadonlyArray<string>;
- /** Roots the index holds no document under at all. */
+ /** Roots the index holds no (admitted) document under at all. */
readonly rootsWithoutDocuments: ReadonlyArray<string>;
}
+/** The visibility an unmeasured indexed document answers `admit` with. */
+const NO_INDEXED_TAGS: ReadonlyArray<string> = Object.freeze([]);
+
/**
* Partition note roots by whether the index carries any document under
* them (silence-is-not-an-answer, U2).
@@ -1719,14 +1722,25 @@ export interface IndexRootCoverage {
* index exists first, and an all-absent answer over a missing index would
* read like a measurement.
*
- * Cost: one read-mode open and one full document-path scan. It runs only
- * on the recall gate's zero-result path, where a search has already
- * failed to produce anything, and the alternative - a per-root prefix
- * query - would buy microseconds for a new store surface.
+ * `admit`, when given, decides which indexed documents count: a root is
+ * reached only through a document `admit` accepts. It receives the path
+ * and the visibility tokens the index measured for it (empty when none
+ * were measured), so a caller below local reach can count only the pages
+ * it may read and a root holding nothing else answers exactly like an
+ * empty one. It is asked lazily, root by root, and stops at the first
+ * admitted document. Omitted, every indexed document counts and nothing
+ * beyond the path scan is read.
+ *
+ * Cost: one read-mode open and one full document-path scan, plus one
+ * visibility lookup per candidate asked of `admit`. It runs only on the
+ * recall gate's zero-result path, where a search has already failed to
+ * produce anything, and the alternative - a per-root prefix query - would
+ * buy microseconds for a new store surface.
*/
export async function indexRootCoverage(
config: ResolvedSearchConfig,
roots: ReadonlyArray<string>,
+ admit?: (path: string, indexedTags: ReadonlyArray<string>) => boolean,
): Promise<IndexRootCoverage> {
const store = await Store.open(config, { mode: "read" });
try {
@@ -1737,7 +1751,12 @@ export async function indexRootCoverage(
// Segment-wise: a root of `Notes` is not reached by a document in
// `Notes-archive/`. Roots arrive from `resolveNoteRoots`, which
// normalises them, and indexed paths are what the walker wrote.
- const reached = paths.some((path) => pathCovers(root, path));
+ const reached = paths.some(
+ (path) =>
+ pathCovers(root, path) &&
+ (admit === undefined ||
+ admit(path, store.indexedVisibilityByPaths([path]).get(path) ?? NO_INDEXED_TAGS)),
+ );
(reached ? withDocuments : withoutDocuments).push(root);
}
return Object.freeze({
diff --git a/src/core/search/pipeline/outcome.ts b/src/core/search/pipeline/outcome.ts
index de3c7134..ab5a8742 100644
--- a/src/core/search/pipeline/outcome.ts
+++ b/src/core/search/pipeline/outcome.ts
@@ -25,6 +25,7 @@ import { fnv1aHex } from "../feedback.ts";
import {
attachTrustMetadata,
buildTerminalPaths,
+ isPathReadableAtReach,
type FrontmatterCache,
} from "../result-filters.ts";
import type {
@@ -289,9 +290,18 @@ const UNRESOLVED_COVERAGE_SCOPE: CoverageScope = Object.freeze({
* sit above this module in the graph, and the sanctioned cure for that in
* this tree is an import evaluated at call time. It is called only on the
* zero-result path, so the cost lands where the answer is owed.
+ *
+ * `reach` decides which indexed pages reach a root: below local reach a
+ * root counts as reached only through a page that caller may read
+ * ({@link isPathReadableAtReach}, honouring the visibility the index
+ * measured), so a root holding nothing that caller can read answers
+ * exactly like an empty one. An absent reach is remote, the convention
+ * {@link corpusVerdictAtReach} keeps. At local reach nothing is read
+ * beyond the index.
*/
export async function probeRetrievalCorpus(
resolveConfig: () => ResolvedSearchConfig,
+ reach?: TransportReach,
): Promise<NegativeRecallVerdict> {
let snapshot: CoverageIndexSnapshot | null = null;
let scope: CoverageScope = UNRESOLVED_COVERAGE_SCOPE;
@@ -305,7 +315,8 @@ export async function probeRetrievalCorpus(
// index to read, or no note root the operator authorized.
const indexedRoots =
status.exists && authorizedRoots.length > 0
- ? (await indexRootCoverage(config, authorizedRoots)).rootsWithDocuments
+ ? (await indexRootCoverage(config, authorizedRoots, rootAdmitAtReach(config, reach)))
+ .rootsWithDocuments
: [];
snapshot = status;
scope = { authorizedRoots, indexedRoots };
@@ -315,6 +326,22 @@ export async function probeRetrievalCorpus(
return classifyNegativeRecall({ snapshot, scope });
}
+/**
+ * The page filter root coverage applies at `reach`: none at local reach,
+ * otherwise the per-path reach verdict over one frontmatter cache for the
+ * whole probe.
+ */
+function rootAdmitAtReach(
+ config: ResolvedSearchConfig,
+ reach: TransportReach | undefined,
+): ((path: string, indexedTags: ReadonlyArray<string>) => boolean) | undefined {
+ const resolved = resolvedTransportReach(reach);
+ if (resolved === TRANSPORT_REACH.local) return undefined;
+ const cache: FrontmatterCache = new Map();
+ return (path, indexedTags) =>
+ isPathReadableAtReach(config.vault, path, resolved, cache, indexedTags);
+}
+
/**
* The corpus statement a zero-result window owes, or `null` when it owes
* none. One decision point for both outcome shapes: a window with rows in
@@ -328,7 +355,9 @@ export async function corpusStatementForEmptyWindow(
reach: TransportReach | undefined,
): Promise<RetrievalCorpusStatement | null> {
if (retrieved > 0 || degraded.length > 0) return null;
- return corpusStatementFor(corpusVerdictAtReach(await probeRetrievalCorpus(resolveConfig), reach));
+ return corpusStatementFor(
+ corpusVerdictAtReach(await probeRetrievalCorpus(resolveConfig, reach), reach),
+ );
}
/**
diff --git a/src/core/search/result-filters.ts b/src/core/search/result-filters.ts
index 35d1cec1..d1a3db51 100644
--- a/src/core/search/result-filters.ts
+++ b/src/core/search/result-filters.ts
@@ -10,7 +10,7 @@ import { join } from "node:path";
import { parseFrontmatterWithNotices } from "../vault.ts";
import { pathIsInside } from "../path-safety.ts";
-import { BRAIN_STATE_REL, isCompiledDigestRel } from "../brain/paths.ts";
+import { BRAIN_STATE_REL, isBrainLogRel, isCompiledDigestRel } from "../brain/paths.ts";
import { DEGRADATION_CODE } from "../integrity/degradation.ts";
import type { FrontmatterMap } from "../types.ts";
import {
@@ -371,6 +371,12 @@ export function applyVisibilityScope(
* records that may be reserved, and its own tags cannot say so. Its
* dedicated readers render it per reader and never ask this predicate
* about the page itself.
+ *
+ * A Brain log page ({@link isBrainLogRel}) answers `false` below local
+ * reach for the same reason: its event bodies name reserved preference
+ * ids, retired ids and evidence artifacts, and the page carries no
+ * visibility that could say so. The log's dedicated readers render its
+ * events per reader instead.
*/
export function isPathReadableAtReach(
vault: string,
@@ -380,7 +386,9 @@ export function isPathReadableAtReach(
indexedTags: ReadonlyArray<string> = [],
): boolean {
if (!pathIsInside(join(vault, path), vault)) return false;
- if (reach !== TRANSPORT_REACH.local && isCompiledDigestRel(path)) return false;
+ if (reach !== TRANSPORT_REACH.local && (isCompiledDigestRel(path) || isBrainLogRel(path))) {
+ return false;
+ }
const entry = readCachedFrontmatterEntry(frontmatterCache, vault, path);
const tags = entry.unreadable ? UNMEASURABLE_VISIBILITY : pageVisibility(entry.meta);
return isRemotelyReadable(tags, reach) && isRemotelyReadable(indexedTags, reach);
diff --git a/src/core/search/surfacing-gate.ts b/src/core/search/surfacing-gate.ts
index 77b65ca7..3f896c5e 100644
--- a/src/core/search/surfacing-gate.ts
+++ b/src/core/search/surfacing-gate.ts
@@ -49,8 +49,20 @@ function isSlashCommand(normalized: string): boolean {
return normalized.startsWith("/") && !normalized.includes(" ");
}
-function isShellOnlyPrompt(normalized: string): boolean {
- if (normalized.includes("?") || normalized.includes("\n")) return false;
+/** A line break (LF or CR) inside the trimmed raw prompt. */
+const LINE_BREAK = /[\n\r]/u;
+
+/**
+ * Whether the raw prompt spans more than one line. Tested on the raw
+ * prompt because normalisation collapses every whitespace run, line
+ * breaks included, into one space.
+ */
+function isMultiLinePrompt(prompt: string): boolean {
+ return LINE_BREAK.test(prompt.trim());
+}
+
+function isShellOnlyPrompt(normalized: string, multiLine: boolean): boolean {
+ if (multiLine || normalized.includes("?")) return false;
const firstToken = normalized.replace(/^\$\s*/u, "").split(/\s+/u)[0] ?? "";
return SHELL_COMMANDS.has(firstToken);
}
@@ -62,7 +74,7 @@ function isShellOnlyPrompt(normalized: string): boolean {
* words against any natural-language vocabulary, so a prompt in any
* language is treated identically. Only structural signals suppress
* retrieval — an empty prompt, a verbatim repeat of the previous one, a
- * slash command, or a single shell command (command names, not human
+ * slash command, or a one-line shell command (command names, not human
* language). Everything else FAILS OPEN: we retrieve and let ranking
* decide relevance, because a missed recall is worse than a cheap
* no-result search.
@@ -79,7 +91,7 @@ export function evaluateSurfacingGate(input: SurfacingGateInput): SurfacingGateD
}
if (isSlashCommand(normalized))
return Object.freeze({ retrieve: false, reason: "slash_command" });
- if (isShellOnlyPrompt(normalized)) {
+ if (isShellOnlyPrompt(normalized, isMultiLinePrompt(input.prompt))) {
return Object.freeze({ retrieve: false, reason: "shell_command" });
}
return Object.freeze({ retrieve: true, reason: "default_retrieve" });
diff --git a/src/core/search/visibility-surface-registry.ts b/src/core/search/visibility-surface-registry.ts
index 087c2f85..6bcecf4b 100644
--- a/src/core/search/visibility-surface-registry.ts
+++ b/src/core/search/visibility-surface-registry.ts
@@ -96,9 +96,15 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
"the caller passes no visibility argument at all. A zero-result answer below local " +
"reach carries no coverage receipt and no index count: a not_found names the index time " +
"only and an unknown states the fixed reason of its unknown_reason " +
- "(corpusVerdictAtReach in pipeline/outcome.ts, withoutCorpusCounts). Residual: an " +
- "authorized note root that holds only pages the caller cannot read is still a reached " +
- "root, so its presence can still move the verdict to coverage-divergent.",
+ "(corpusVerdictAtReach in pipeline/outcome.ts, withoutCorpusCounts). Its root coverage " +
+ "answers at the caller's reach: below local reach an authorized note root counts as " +
+ "reached only through a page that caller may read (probeRetrievalCorpus threads the " +
+ "reach into indexRootCoverage's admit predicate), so a root holding nothing else answers " +
+ "like an empty one. Its ranking statistics are a stated residual: the bm25 corpus " +
+ "statistics and the diversity rerank are computed over the shared index, which counts " +
+ "pages the caller may not read (reserved pages, the Brain log), so two vaults that " +
+ "return the same paths remotely can return different scores and rerank reasons - a weak " +
+ "count signal across the whole index, not a page or its content.",
},
{
surface: "brain_file_context",
@@ -276,11 +282,13 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
{
surface: "brain_claims",
kind: K.mcpTool,
- category: C.excluded,
+ category: C.covered,
reason:
"toolBrainClaims's own docblock (knowledge-tools.ts) states a claim row carries the " +
"artifact's id, vault-relative path, topic and full principle text; every row-returning " +
- "operation is filtered by gatedOwnerScopeView (agent-scope), never by visibility.",
+ "operation and the rebuild count keep a row only when it passes gatedOwnerScopeView and, " +
+ "at the caller's reach, reachView over its page, its id under the pref- and ret- " +
+ "spellings, and the records that superseded or contest it.",
},
{
surface: "brain_truth",
@@ -360,8 +368,9 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
reason:
"its hygiene count asks the owner view ANDed with reachView and its cited pages answer " +
"readableAtContextReach(ctx), but the doctor error and warning counts and the preference " +
- "counts are taken over the whole Brain layer, so they move with a reserved record; it " +
- "returns counts and problem labels, never a note path, title, or body.",
+ "counts are taken over the whole Brain layer, so they move with a reserved record " +
+ "(a stated, deferred residual: no count is recomputed per reader yet); it returns counts " +
+ "and problem labels, never a note path, title, or body.",
},
{
surface: "brain_maintenance",
@@ -468,7 +477,11 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
"resolveLogEventTraces (core/brain/event-trace.ts) takes the handler's " +
"reachView(ctx.vault, contextReach(ctx)) as its view option and drops, before the limit " +
"and the totals, every event whose artifacts or body strings name a page the caller may " +
- "not read at its reach, and every attached trace whose handoff reference it hides.",
+ "not read at its reach, and every attached trace whose handoff reference it hides; a " +
+ "dream shared with a withheld record is kept with its readable transitions only " +
+ "(log-events-at-reach.ts), its body and artifacts read from that form, so the count " +
+ "does not fall when a reserved preference shares a dream; each record id is judged " +
+ "under its pref- and ret- spellings.",
},
{
surface: "brain_foresight",
@@ -503,10 +516,37 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
reason:
"view=digest (brief-tools.ts) is rendered for the caller below local reach - renderDigest " +
"is handed readableAtContextReach(ctx), so a reserved preference or retired record is " +
- "absent from its rows and counts, and no report snapshot is taken or delta shown; " +
+ "absent from its rows and counts (a dream shared with one is counted with its readable " +
+ "transitions only), and no report snapshot is taken or delta shown; " +
"view=morning is handed the same predicate and leaves such a preference out, and shows " +
- "no pending trigger or trigger-queue failure and marks nothing delivered - but the " +
- "daily, weekly, monthly, operator and today views count over the whole Brain layer.",
+ "no pending trigger or trigger-queue failure and marks nothing delivered; view=daily " +
+ "and view=weekly answer at the caller's reach for the ids they name - a status " +
+ "transition, retirement or contradiction naming a record the caller cannot read (under " +
+ "its pref- or ret- spelling) is dropped through readerRefView, source_pointers are " +
+ "recollected from the evidence events the caller may see, events_by_kind and vault_delta " +
+ "are recomputed from that same event selection (evidence on a withheld record, a dream " +
+ "whose every transition is withheld and any other event scoped to a withheld record are " +
+ "not counted), and no report snapshot is taken or delta shown; view=today renders its " +
+ "recent activity through the shared log-event rule (log-events-at-reach.ts via " +
+ "reach-events.ts) before the limit and the totals, so an event naming a withheld record " +
+ "is absent and a shared dream shows only its readable transitions. Residual: the monthly " +
+ "and operator views still count over the whole Brain layer (summary events, status " +
+ "transitions, retired and contradiction counts; preference, retired and doctor counts " +
+ "and the trust verdict), and name no id.",
+ },
+ {
+ surface: "brain_analytics",
+ kind: K.mcpTool,
+ category: C.excluded,
+ reason:
+ "view=timeline, view=belief_evolution and view=concept_synthesis answer at the caller's " +
+ "reach (analytics-tools.ts via reach-events.ts): a timeline event naming a record the " +
+ "caller cannot read under its pref- or ret- spelling is dropped before the limit and the " +
+ "total, a shared dream is kept while one transition is readable; a belief-evolution row " +
+ "is asked over every record it names and a refused pref_id target answers as an absent " +
+ "one; a concept-synthesis linker or mention from an unreadable page is dropped and a " +
+ "refused target answers as an empty cluster. Residual: view=attention_flows and " +
+ "view=dedup are filtered by the gated owner view only, never by reach.",
},
{
surface: "brain_pre_compress_pack",
@@ -534,8 +574,9 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
"a diagnostic classifier over caller-supplied scores/match_quality - it runs no search and " +
"returns no note content; included for completeness of the file-level sweep only. Its " +
"corpus statement below local reach carries no coverage receipt and no index count, for " +
- "every state (corpusVerdictAtReach, withoutCorpusCounts); the coverage-divergent residual " +
- "named on brain_search applies here too.",
+ "every state (corpusVerdictAtReach, withoutCorpusCounts), and its root coverage answers " +
+ "at the caller's reach the same way brain_search's does (probeRetrievalCorpus with " +
+ "contextReach).",
},
{
surface: "brain_recall_feedback",
@@ -650,7 +691,8 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
reason:
"reports install/config/vault status blocks - no note path, title, or body crosses this " +
"surface, but its Brain block is computeBrainStatus over the whole Brain layer, so the " +
- "preference counts and the last apply-evidence time move with a reserved record; included " +
+ "preference counts and the last apply-evidence time move with a reserved record (a " +
+ "stated, deferred residual); included " +
"for the file-level sweep (tools.ts imports listVaultPages for second_brain_query, defined " +
"in the same file).",
},
@@ -852,7 +894,9 @@ export const VISIBILITY_SURFACE_REGISTRY: ReadonlyArray<VisibilitySurfaceEntry>
kind: K.mcpResource,
category: C.excluded,
reason:
- "computeBrainStatus() output, same unfiltered whole-vault-reader class; no visibility check.",
+ "computeBrainStatus() output, same unfiltered whole-vault-reader class; no visibility " +
+ "check, so its preference and retired counts move with a reserved record (a stated, " +
+ "deferred residual).",
},
{
surface: "osb://preference/{id}",
diff --git a/src/core/vault.ts b/src/core/vault.ts
index 98b0af0d..3944e9eb 100644
--- a/src/core/vault.ts
+++ b/src/core/vault.ts
@@ -72,7 +72,16 @@ import {
} from "./graph/visibility.ts";
import type { FrontmatterMap, FrontmatterValue, VaultPage } from "./types.ts";
-const FRONTMATTER_RE = /^---\s*\n([\s\S]*?)\n---\s*\n?/;
+/**
+ * A leading frontmatter block, its body captured. The opening fence takes
+ * horizontal whitespace only before its line end: with `\s*` there, a run
+ * of blank lines after an unclosed `---` was re-scanned to the end of the
+ * text once per blank line, quadratic in its length. Blank lines after the
+ * fence still open the block; they are the first lines of its body.
+ */
+export const FRONTMATTER_RE = /^---[^\S\n]*\n([\s\S]*?)\n---\s*\n?/;
+/** U+FEFF, which may precede a page's opening fence. */
+const BYTE_ORDER_MARK = 0xfeff;
/**
* The key grammar, written once and used by BOTH directions.
*
@@ -244,11 +253,15 @@ export function parseFrontmatterWithNotices(
* dropped, exactly as before; only the trace is new.
*/
export function parseFrontmatterTextWithNotices(
- text: string,
+ raw: string,
opts: FrontmatterNoticeOptions = {},
): FrontmatterParseWithNotices {
const notices: DegradationNotice[] = [];
const site = opts.site ?? FRONTMATTER_SITE;
+ // A byte-order mark before the opening fence (a spreadsheet export, an
+ // editor's UTF-8 setting) must not hide the block: a hidden block is a
+ // hidden `visibility`, and the page would read as open.
+ const text = raw.charCodeAt(0) === BYTE_ORDER_MARK ? raw.slice(1) : raw;
const match = FRONTMATTER_RE.exec(text);
if (!match) {
return [{}, text.trim(), notices];
diff --git a/src/mcp/brain/analytics-tools.ts b/src/mcp/brain/analytics-tools.ts
index f5e05afc..e8e1d27c 100644
--- a/src/mcp/brain/analytics-tools.ts
+++ b/src/mcp/brain/analytics-tools.ts
@@ -35,6 +35,8 @@ import {
requiredStringArg,
} from "./shared.ts";
import { unknownOperationError } from "../coerce.ts";
+import type { ArtifactRef } from "../../core/brain/artifact-ref-view.ts";
+import { eventsAtReach, recordReadable, recordRefs, requestRefView } from "./reach-events.ts";
function coerceEventKind(tool: string, raw: unknown): BrainLogEventKind | undefined {
if (raw === undefined || raw === null) return undefined;
@@ -79,8 +81,16 @@ async function toolBrainTimeline(
// U3). Filtered BEFORE the limit and before `total`, so the count is
// the visible one - a total that still included the hidden rows would
// leak their number.
+ //
+ // The same holds one rule over at the caller's reach: an event naming a
+ // record the caller cannot read (under its pref- or ret- spelling) is
+ // dropped, and a dream shared with one is kept while a transition it
+ // names is readable.
const view = gatedOwnerScopeView(ctx.vault, ctx.agentName);
- const visible = view.keep(events, (ev) => [ev.source.path, ev.prefId, ev.artifact]);
+ const visible = eventsAtReach(
+ requestRefView(ctx),
+ view.keep(events, (ev) => [ev.source.path, ev.prefId, ev.artifact]),
+ );
const sliced = limit !== undefined ? visible.slice(0, limit) : visible;
return {
vault_path: vaultPathField(ctx),
@@ -130,18 +140,41 @@ async function toolBrainBeliefEvolution(
// transition by `prefId` and by the wikilink the dream summary used,
// an evidence row by `prefId` and by the `artifact` it was applied to,
// a retirement by its whole supersession chain.
+ //
+ // At the caller's reach every row is asked under each spelling its
+ // records answer to, and a pref_id target the caller cannot read is
+ // answered as an absent one: no row at all.
const view = gatedOwnerScopeView(ctx.vault, ctx.agentName);
+ const refs = requestRefView(ctx);
+ const targetShown = !hasPref || recordReadable(refs, (prefIdRaw as string).trim());
+ const shown = <T>(
+ rows: ReadonlyArray<T>,
+ ownerRefs: (row: T) => ReadonlyArray<ArtifactRef>,
+ reachRefs: (row: T) => ReadonlyArray<ArtifactRef>,
+ ): ReadonlyArray<T> => (targetShown ? refs.keep(view.keep(rows, ownerRefs), reachRefs) : []);
return {
vault_path: vaultPathField(ctx),
target: evo.target,
- transitions: view.keep(evo.transitions, (t) => [t.prefId, t.link]),
- evidence: view.keep(evo.evidence, (e) => [e.prefId, e.artifact]),
- retirements: view.keep(evo.retirements, (r) => [
- r.prefId,
- r.retiredBy,
- r.supersededBy,
- r.supersedes,
- ]),
+ transitions: shown(
+ evo.transitions,
+ (t) => [t.prefId, t.link],
+ (t) => [t.link, ...recordRefs(t.prefId)],
+ ),
+ evidence: shown(
+ evo.evidence,
+ (e) => [e.prefId, e.artifact],
+ (e) => [e.artifact, ...recordRefs(e.prefId)],
+ ),
+ retirements: shown(
+ evo.retirements,
+ (r) => [r.prefId, r.retiredBy, r.supersededBy, r.supersedes],
+ (r) => [
+ ...recordRefs(r.prefId),
+ ...recordRefs(r.retiredBy),
+ ...recordRefs(r.supersededBy),
+ ...recordRefs(r.supersedes),
+ ],
+ ),
generated_at: evo.generatedAt,
};
}
@@ -181,12 +214,18 @@ async function toolBrainConceptSynthesis(
includeUnlinked,
ownerScope: gatedOwnerScopeView(ctx.vault, ctx.agentName).scope,
});
+ // At the caller's reach a linker or a mention whose source page the
+ // caller cannot read is dropped, and a target it cannot read (under its
+ // pref- or ret- spelling) is answered as an absent one: its own id as
+ // the title and an empty cluster.
+ const refs = requestRefView(ctx);
+ const targetShown = recordReadable(refs, cluster.targetId);
return {
vault_path: vaultPathField(ctx),
target_id: cluster.targetId,
- target_title: cluster.targetTitle,
- linkers: cluster.linkers,
- unlinked_mentions: cluster.unlinkedMentions,
+ target_title: targetShown ? cluster.targetTitle : cluster.targetId,
+ linkers: targetShown ? refs.keep(cluster.linkers, (l) => [l.source]) : [],
+ unlinked_mentions: targetShown ? refs.keep(cluster.unlinkedMentions, (m) => [m.source]) : [],
generated_at: cluster.generatedAt,
};
}
diff --git a/src/mcp/brain/brief-tools.ts b/src/mcp/brain/brief-tools.ts
index 2dffa971..2a1492f3 100644
--- a/src/mcp/brain/brief-tools.ts
+++ b/src/mcp/brain/brief-tools.ts
@@ -19,6 +19,20 @@ import {
type TriggerQueueFailures,
} from "../../core/brain/triggers/store.ts";
import { buildTimelineIndex } from "../../core/brain/temporal/build-index.ts";
+import {
+ collectSourcePointers,
+ computeVaultDelta,
+ countByKind,
+ type PeriodStatusTransition,
+} from "../../core/brain/temporal/period-common.ts";
+import { selectEvents } from "../../core/brain/temporal/select-events.ts";
+import type { TimelineIndex } from "../../core/brain/temporal/types.ts";
+import {
+ readerRefView,
+ type ArtifactRef,
+ type ArtifactRefView,
+} from "../../core/brain/artifact-ref-view.ts";
+import { PREF_ID_PREFIX } from "../../core/brain/dream-plan.ts";
import { buildTodayDashboard } from "../../core/brain/today-dashboard.ts";
import { buildDailyBrief } from "../../core/brain/temporal/daily-brief.ts";
import { buildWeeklySynthesis } from "../../core/brain/temporal/weekly-brief.ts";
@@ -36,6 +50,7 @@ import { TRANSPORT_REACH } from "../../core/graph/transport-reach.ts";
import { INVALID_PARAMS, MCPError } from "../protocol.ts";
import { contextReach, type ServerContext, type ToolDefinition } from "../tool-contract.ts";
import { readableAtContextReach, readableAtContextReachOrUndefined } from "./reach-readable.ts";
+import { eventAtReach, eventsAtReach, recordRefs, requestRefView } from "./reach-events.ts";
import { vaultPathField } from "../vault-path-field.ts";
import { MCP_PREVIEW_BUDGET } from "../preview-budget.ts";
import {
@@ -217,6 +232,67 @@ async function toolBrainDigest(
// ----- brain_query ---------------------------------------------------------
+/**
+ * The rows of a daily or weekly envelope a reader at this request's
+ * reach may see, or `null` at local reach, where the envelope is the
+ * builder's own (the digest precedent: a reader below local reach is
+ * shown only what it may read, and takes no report snapshot).
+ *
+ * Status transitions, retirements and contradictions are dropped when a
+ * record they name is withheld. Source pointers are deduplicated
+ * artifacts with no record left on them, so they are recollected from
+ * the window's events the reader may see: an artifact cited only by
+ * evidence on a withheld preference is absent. The counts
+ * (`events_by_kind`, `vault_delta`) are recomputed from the same
+ * selection, so a withheld record moves no count either.
+ */
+function periodRowsAtReach(
+ ctx: ServerContext,
+ index: TimelineIndex,
+ window: { readonly since: string; readonly until: string },
+): PeriodRowsView | null {
+ if (contextReach(ctx) === TRANSPORT_REACH.local) return null;
+ const refs = readerRefView(ctx.vault, readableAtContextReach(ctx));
+ const events = eventsAtReach(refs, selectEvents(index, window));
+ return {
+ refs,
+ sourcePointers: collectSourcePointers(events),
+ eventsByKind: countByKind(events),
+ vaultDelta: (transitions) => computeVaultDelta(events, transitions),
+ };
+}
+
+interface PeriodRowsView {
+ readonly refs: ArtifactRefView;
+ readonly sourcePointers: ReadonlyArray<string>;
+ readonly eventsByKind: ReturnType<typeof countByKind>;
+ /** The window's delta over the visible events and the transitions kept for the reader. */
+ readonly vaultDelta: (
+ transitions: ReadonlyArray<PeriodStatusTransition>,
+ ) => ReturnType<typeof computeVaultDelta>;
+}
+
+/** A transition or retirement row: the record id and the link it was logged with. */
+function transitionRefs(row: {
+ readonly prefId: string;
+ readonly link: string;
+}): ReadonlyArray<ArtifactRef> {
+ return [row.link, ...recordRefs(row.prefId)];
+}
+
+/** A contradiction row: its record, the topic's record and its evidence artifact. */
+function contradictionRefs(row: {
+ readonly prefId?: string;
+ readonly topic?: string;
+ readonly artifact?: string;
+}): ReadonlyArray<ArtifactRef> {
+ return [
+ row.artifact,
+ ...recordRefs(row.prefId),
+ ...(row.topic !== undefined ? recordRefs(`${PREF_ID_PREFIX}${row.topic}`) : []),
+ ];
+}
+
/**
* `brain_daily_brief` - structured counters + transitions + source
* pointers for one day. Defaults `date` to today UTC when omitted.
@@ -233,18 +309,23 @@ async function toolBrainDailyBrief(
const brief = buildDailyBrief(index, ctx.vault, date, {
offsetHours: cfg.daily_window_offset_hours,
});
+ const atReach = periodRowsAtReach(ctx, index, brief.window);
+ const statusTransitions =
+ atReach?.refs.keep(brief.statusTransitions, transitionRefs) ?? brief.statusTransitions;
const envelope: Record<string, unknown> = {
vault_path: vaultPathField(ctx),
date: brief.date,
window: brief.window,
- events_by_kind: brief.eventsByKind,
- status_transitions: brief.statusTransitions,
- vault_delta: brief.vaultDelta,
- source_pointers: brief.sourcePointers,
+ events_by_kind: atReach?.eventsByKind ?? brief.eventsByKind,
+ status_transitions: statusTransitions,
+ vault_delta: atReach?.vaultDelta(statusTransitions) ?? brief.vaultDelta,
+ source_pointers: atReach?.sourcePointers ?? brief.sourcePointers,
generated_at: brief.generatedAt,
};
- // Dual-output (t_00eece5d): persist a machine snapshot and report
- // the run-over-run delta when report snapshots are enabled.
+ // Below local reach no snapshot is taken and no delta reported.
+ if (atReach !== null) return envelope;
+ // Dual-output: persist a machine snapshot and report the run-over-run
+ // delta when report snapshots are enabled.
const delta = captureReportDelta(
ctx.vault,
"daily",
@@ -274,18 +355,27 @@ async function toolBrainWeeklySynthesis(
const cfg = loadTemporalConfigSafe(ctx.vault);
const index = buildTimelineIndex(ctx.vault, {});
const synth = buildWeeklySynthesis(index, ctx.vault, weekEnd, cfg);
+ const atReach = periodRowsAtReach(ctx, index, {
+ since: synth.windowStart,
+ until: synth.windowEnd,
+ });
+ const statusTransitions =
+ atReach?.refs.keep(synth.statusTransitions, transitionRefs) ?? synth.statusTransitions;
const envelope: Record<string, unknown> = {
vault_path: vaultPathField(ctx),
window_start: synth.windowStart,
window_end: synth.windowEnd,
- events_by_kind: synth.eventsByKind,
- status_transitions: synth.statusTransitions,
- retired: synth.retired,
- contradictions: synth.contradictions,
- vault_delta: synth.vaultDelta,
- source_pointers: synth.sourcePointers,
+ events_by_kind: atReach?.eventsByKind ?? synth.eventsByKind,
+ status_transitions: statusTransitions,
+ retired: atReach?.refs.keep(synth.retired, transitionRefs) ?? synth.retired,
+ contradictions:
+ atReach?.refs.keep(synth.contradictions, contradictionRefs) ?? synth.contradictions,
+ vault_delta: atReach?.vaultDelta(statusTransitions) ?? synth.vaultDelta,
+ source_pointers: atReach?.sourcePointers ?? synth.sourcePointers,
generated_at: synth.generatedAt,
};
+ // Below local reach no snapshot is taken and no delta reported.
+ if (atReach !== null) return envelope;
const delta = captureReportDelta(
ctx.vault,
"weekly",
@@ -447,8 +537,13 @@ async function toolBrainToday(
args["lookback_days"],
);
const limit = coerceNonNegativeInteger("brain_brief view=today", "limit", args["limit"]);
+ // The recent-activity section renders log events; below local reach
+ // (or under the ownership gate) it shows them as this caller may see
+ // them, before the limit and the totals.
+ const refs = requestRefView(ctx);
const dashboard = buildTodayDashboard(ctx.vault, {
now: new Date(),
+ ...(refs.filtersNothing ? {} : { eventAtReach: (ev) => eventAtReach(refs, ev) }),
...(lookbackDays !== undefined ? { activityLookbackDays: lookbackDays } : {}),
...(limit !== undefined ? { activityLimit: limit } : {}),
});
diff --git a/src/mcp/brain/ingest-tools.ts b/src/mcp/brain/ingest-tools.ts
index 2919f286..9ced9aae 100644
--- a/src/mcp/brain/ingest-tools.ts
+++ b/src/mcp/brain/ingest-tools.ts
@@ -14,6 +14,7 @@ import { planBatches, type BatchPlan } from "../../core/brain/ingest/batch-plan.
import { clearCheckpoint, PLAN_ID_LABEL } from "../../core/brain/ingest/checkpoint.ts";
import { assertCheckpointId } from "../../core/brain/checkpoint-store.ts";
import { ingestSource } from "../../core/brain/ingest/ingest.ts";
+import type { PartsOutcome, TableOutcome } from "../../core/brain/ingest/extract-source.ts";
import type { CodeEdgeSeed, PreExtractResult } from "../../core/brain/ingest/pre-extract.ts";
import { reconcilePlan } from "../../core/brain/ingest/reconcile.ts";
import { IntakeValidationError } from "../../core/brain/intake/extract-intake.ts";
@@ -103,6 +104,10 @@ async function toolBrainIngestSource(
// it. Always present: absence would read as "full-local" to a caller
// that never learned the key exists.
capture_scope: res.captureScope,
+ // Only for an HTML or CSV/TSV source, so a text source's payload is
+ // byte-identical to before. Counts and tokens only, never content.
+ ...(res.parts !== undefined ? { parts: serializeParts(res.parts) } : {}),
+ ...(res.table !== undefined ? { table: serializeTable(res.table) } : {}),
// Only emitted when the pre-extract pass ran, so a call without it is
// byte-identical to before (P4).
...(res.preExtract !== undefined ? { pre_extract: serializePreExtract(res.preExtract) } : {}),
@@ -110,6 +115,38 @@ async function toolBrainIngestSource(
});
}
+/** The HTML outcome on the wire: `{extracted, count, omitted?}` or `{extracted: false, reason}`. */
+function serializeParts(parts: PartsOutcome): Record<string, unknown> {
+ if (!parts.extracted) return { extracted: false, reason: parts.reason };
+ return {
+ extracted: true,
+ count: parts.count,
+ ...(parts.omitted !== undefined ? { omitted: parts.omitted } : {}),
+ };
+}
+
+/** The table outcome on the wire, snake_case, `truncated` only when a cap cut it. */
+function serializeTable(table: TableOutcome): Record<string, unknown> {
+ if (!table.rendered) {
+ return {
+ rendered: false,
+ format: table.format,
+ reason: table.reason,
+ ...(table.detail !== undefined ? { detail: table.detail } : {}),
+ };
+ }
+ return {
+ rendered: true,
+ format: table.format,
+ delimiter: table.delimiter,
+ columns: table.columns,
+ rows: table.rows,
+ rows_rendered: table.rowsRendered,
+ ...(table.truncated.length > 0 ? { truncated: [...table.truncated] } : {}),
+ redacted_cells: table.redactedCells,
+ };
+}
+
/** The pre-extract result with its edge keys in the payload's snake_case. */
function serializePreExtract(result: PreExtractResult): Record<string, unknown> {
if (!result.extracted) return { ...result };
@@ -183,7 +220,10 @@ async function toolBrainSearchBySource(
args: Record<string, unknown>,
): Promise<Record<string, unknown>> {
const sourceFile = coerceStr(args, "source_file", true)!;
- const hits = searchBySourceFile(ctx.vault, sourceFile);
+ // A page the caller may not read at its reach (a summary page that
+ // inherited a reserved source's visibility) is answered as an absent one.
+ const readable = readableAtContextReach(ctx);
+ const hits = searchBySourceFile(ctx.vault, sourceFile).filter((entry) => readable(entry.path));
// Owner-scope isolation (context-integrity-gates, Unit A). Every entry
// names a Brain page, so the ownership rule applies through the same
// path-based resolver the ranked search path uses - including its
@@ -254,8 +294,9 @@ export function serializeBatchPlan(plan: BatchPlan): Record<string, unknown> {
skipped: [...plan.skipped],
// Only emitted when the extractable gate skipped something, so a plan with
// no extractable declaration serializes byte-identically to before. The
- // reason is the typed token (P4) and `detail` carries the schema_type
- // value behind it, so a reader can check the skip without re-reading.
+ // reason is the typed token (P4) and `detail` carries the value behind
+ // it - the schema_type, or the format of a format skip - so a reader can
+ // check the skip without re-reading.
...(plan.skippedNonExtractable.length > 0
? {
skipped_non_extractable: plan.skippedNonExtractable.map((s) => ({
@@ -298,7 +339,13 @@ export function serializeBatchPlan(plan: BatchPlan): Record<string, unknown> {
batches: plan.batches.map((b) => ({
index: b.index,
total_bytes: b.totalBytes,
- files: b.files.map((f) => ({ path: f.path, bytes: f.bytes, status: f.status })),
+ // `format` only for a non-text file, so a Markdown plan is unchanged.
+ files: b.files.map((f) => ({
+ path: f.path,
+ bytes: f.bytes,
+ status: f.status,
+ ...(f.format !== undefined ? { format: f.format } : {}),
+ })),
})),
};
}
diff --git a/src/mcp/brain/knowledge-tools.ts b/src/mcp/brain/knowledge-tools.ts
index c5df9b52..32788180 100644
--- a/src/mcp/brain/knowledge-tools.ts
+++ b/src/mcp/brain/knowledge-tools.ts
@@ -40,6 +40,7 @@ import { auditMoc, MocAuditError } from "../../core/brain/link-graph/moc-audit.t
import { reachView } from "../../core/brain/reach-view.ts";
import { everyArtifactRefView } from "../../core/brain/artifact-ref-view.ts";
import { gatedOwnerScopeView } from "../../core/brain/owner-scope-view.ts";
+import { recordRefs } from "../../core/brain/log-events-at-reach.ts";
import { normaliseWikilinkTarget } from "../../core/brain/wikilink.ts";
import { isoSecond } from "../../core/brain/time.ts";
import { normalizeAgentArgument } from "../../core/agent-identity.ts";
@@ -407,6 +408,7 @@ async function toolBrainMocAudit(
// (a-label-is-not-a-boundary, U3, recon C4).
const report = auditMoc(ctx.vault, targetId, {
ownerScope: gatedOwnerScopeView(ctx.vault, ctx.agentName).scope,
+ readable: readableAtContextReach(ctx),
});
return {
vault_path: vaultPathField(ctx),
@@ -772,9 +774,20 @@ function toolBrainClaims(
// recon C6 to the "this bucket is metadata" label. Every row-returning
// operation is filtered, and so is the rebuild count: a count the
// caller cannot decompose still says how many claims exist.
- const view = gatedOwnerScopeView(ctx.vault, ctx.agentName);
- const visible = (rows: ReadonlyArray<ClaimNode>): ReadonlyArray<ClaimNode> =>
- view.keep(rows, (n) => [n.path, n.id]);
+ // At the caller's reach a row is asked over every record it names: its
+ // page, its id under the pref- and ret- spellings, and the records that
+ // superseded or contest it.
+ const owner = gatedOwnerScopeView(ctx.vault, ctx.agentName);
+ const reach = reachView(ctx.vault, contextReach(ctx));
+ const shown = (n: ClaimNode): boolean =>
+ owner.row(n.path, n.id) &&
+ reach.row(
+ n.path,
+ ...recordRefs(n.id),
+ ...recordRefs(n.superseded_by ?? undefined),
+ ...n.contradicts.flatMap((id) => recordRefs(id)),
+ );
+ const visible = (rows: ReadonlyArray<ClaimNode>): ReadonlyArray<ClaimNode> => rows.filter(shown);
switch (operation) {
case "rebuild": {
const graph = rebuildClaimGraph(ctx.vault);
@@ -788,8 +801,7 @@ function toolBrainClaims(
case "replaced": {
const id = coerceStr(args, "id", true)!;
const tip = whatReplaced(resolveClaimGraph(ctx.vault), id);
- const shown = tip !== null && view.row(tip.path, tip.id) ? tip : null;
- return { operation, id, tip: shown ? renderClaimNode(shown) : null };
+ return { operation, id, tip: tip !== null && shown(tip) ? renderClaimNode(tip) : null };
}
case "contests": {
const id = coerceStr(args, "id", true)!;
diff --git a/src/mcp/brain/query-tools.ts b/src/mcp/brain/query-tools.ts
index e7a2f6be..1b122999 100644
--- a/src/mcp/brain/query-tools.ts
+++ b/src/mcp/brain/query-tools.ts
@@ -70,7 +70,7 @@ import { loadGuardrailsConfigSafe } from "../../core/brain/policy.ts";
import { normalizeAgentScope } from "../../core/graph/agent-scope.ts";
import { isPreferenceVisible } from "../../core/brain/owner-scoped-facts.ts";
import { reachView } from "../../core/brain/reach-view.ts";
-import { logEntryArtifactRefs } from "../../core/brain/log.ts";
+import { logEntriesAtReach, recordRefs } from "../../core/brain/log-events-at-reach.ts";
/** Accepted `at` forms, named in every refusal so the exit is actionable. */
const AS_OF_FORMS = "an ISO-8601 instant or YYYY-MM-DD date";
@@ -193,6 +193,12 @@ async function toolBrainQuery(
try {
if (preference !== null) {
try {
+ // Asked before the page is parsed: a withheld page that fails to
+ // parse would otherwise answer with the parser's error, which
+ // names the page an absent id never could.
+ if (!queryView.row(...recordRefs(preference))) {
+ throw new BrainNotFoundError(preference);
+ }
const res = queryByPreference(ctx.vault, preference);
// Fail closed: an owner-private fact outside the requested scope is
// indistinguishable from absent, so it cannot leak across owners.
@@ -244,7 +250,7 @@ async function toolBrainQuery(
// tells the caller how many rows it was not shown - a count oracle
// reached through a second tool rather than this one.
const signals = queryView.keep(res.signals, (sig) => [sig.id]);
- const logEvents = queryView.keep(res.all_log_events, (e) => logEntryArtifactRefs(e));
+ const logEvents = logEntriesAtReach(queryView, res.all_log_events);
const resultCount = signals.length + logEvents.length;
emitQueryTelemetry(resultCount > 0 ? "ok" : "empty", resultCount);
const topicPrefOwner = res.preference?.owner;
@@ -263,7 +269,7 @@ async function toolBrainQuery(
// since
const res = queryByLogSince(ctx.vault, since!);
- const events = queryView.keep(res, (e) => logEntryArtifactRefs(e));
+ const events = logEntriesAtReach(queryView, res);
emitQueryTelemetry(events.length > 0 ? "ok" : "empty", events.length);
return {
mode: "since",
@@ -532,8 +538,12 @@ async function toolBrainBacklinks(
// one - the empty backlink document - rather than refusing, because an
// unknown target is a legitimate zero here and a refusal would be the
// one response shape that proves the page exists.
+ //
+ // The target is asked under every spelling its record answers to, so a
+ // retired record (on disk only as `ret-x`) asked for as `pref-x` is the
+ // reserved page it is, not an absent one whose log backlinks are listed.
const view = reachView(ctx.vault, contextReach(ctx));
- const refs = view.visible(target)
+ const refs = view.row(...recordRefs(target))
? (index.get(target) ?? []).filter((r) => view.visible(r.source))
: [];
const unparsed = index.unparsed
diff --git a/src/mcp/brain/reach-events.ts b/src/mcp/brain/reach-events.ts
new file mode 100644
index 00000000..c207b7c0
--- /dev/null
+++ b/src/mcp/brain/reach-events.ts
@@ -0,0 +1,45 @@
+/**
+ * The Brain log event rule bound to one request.
+ *
+ * Every MCP reader that renders log events - the daily and weekly briefs,
+ * the today view, and the timeline, belief-evolution and
+ * concept-synthesis analytics views - asks the same question: which of
+ * these events may this caller see, and in what form? The rule itself
+ * lives in `core/brain/log-events-at-reach.ts`, where the core readers
+ * (the digest, the event trace) reach it too; this module binds it to
+ * the request's reach and owner predicate, so no reader rebuilds the
+ * reference view by hand.
+ */
+
+import { readerRefView, type ArtifactRefView } from "../../core/brain/artifact-ref-view.ts";
+import { eventAtReach, recordRefs } from "../../core/brain/log-events-at-reach.ts";
+import type { TemporalEvent } from "../../core/brain/temporal/types.ts";
+import type { ServerContext } from "../tool-contract.ts";
+import { readableAtContextReachOrUndefined } from "./reach-readable.ts";
+
+export { eventAtReach, recordRefs };
+
+/**
+ * The reference view for one request: unfiltered at local reach with the
+ * ownership gate off, the request's reach and owner predicate otherwise.
+ */
+export function requestRefView(ctx: ServerContext): ArtifactRefView {
+ return readerRefView(ctx.vault, readableAtContextReachOrUndefined(ctx));
+}
+
+/** The events this caller may see, each in the form it may see it, in order. */
+export function eventsAtReach(
+ refs: ArtifactRefView,
+ events: ReadonlyArray<TemporalEvent>,
+): ReadonlyArray<TemporalEvent> {
+ if (refs.filtersNothing) return events;
+ return events.flatMap((ev) => eventAtReach(refs, ev) ?? []);
+}
+
+/**
+ * May this caller read the record `id` names, under every spelling it
+ * answers to? A record refused here is answered as an absent one.
+ */
+export function recordReadable(refs: ArtifactRefView, id: string): boolean {
+ return refs.row(...recordRefs(id));
+}
diff --git a/src/mcp/registry-guard.ts b/src/mcp/registry-guard.ts
index b87040ca..2e2aa318 100644
--- a/src/mcp/registry-guard.ts
+++ b/src/mcp/registry-guard.ts
@@ -349,7 +349,8 @@ export const PREVIEW_BUDGET_EXEMPT: Readonly<Record<string, string>> = Object.fr
brain_switch_vault: "write; returns a small profile ack",
brain_write_session: "lifecycle ops return one fixed-shape envelope; prompts are kernel-bounded",
brain_intake_entities: "write; returns created/updated id lists and a relation count",
- brain_ingest_source: "write; returns the summary path plus bounded id lists",
+ brain_ingest_source:
+ "write; returns the summary path, bounded id lists and derived-section counts. Below local reach a summary page the caller cannot read is left untouched and the call answers as a first ingest would, so a repeated call answers created each time where a vault without that page answers created once; closing that needs a caller-lane summary path",
brain_distill_source:
"write; returns the distillation path, a claim count, source hash, capture scope, and a quotes report whose finding list is capped and self-declaring its truncation",
brain_research_report:
diff --git a/src/mcp/resources.ts b/src/mcp/resources.ts
index 30d282fb..6875ed5a 100644
--- a/src/mcp/resources.ts
+++ b/src/mcp/resources.ts
@@ -88,9 +88,9 @@ import {
resolvedTransportReach,
type TransportReach,
} from "../core/graph/transport-reach.ts";
-import { extractWikilinkRichBodies } from "../core/brain/link-graph/parse-wikilink.ts";
import { normaliseWikilinkTarget } from "../core/brain/wikilink.ts";
-import { logEntryArtifactRefs } from "../core/brain/log.ts";
+import { logEntriesAtReach, recordRefs } from "../core/brain/log-events-at-reach.ts";
+import { parseLogDayFile, renderLogEntryBlock, type ParseLogDayResult } from "../core/brain/log.ts";
import type { BrainPreference, BrainRetired } from "../core/brain/types.ts";
import { INTERNAL_ERROR, INVALID_PARAMS, MCPError } from "./protocol.ts";
@@ -438,6 +438,10 @@ function readPreference(
}
const normalized =
rawId.startsWith("pref-") || rawId.startsWith("ret-") ? rawId : `pref-${rawId}`;
+ // Asked before the page is parsed: a withheld page that fails to parse
+ // would otherwise answer with the parser's error, which names the page
+ // an absent id never could.
+ if (!view.refs.row(...recordRefs(normalized))) throw notFound(normalized);
let result;
try {
result = queryByPreference(ctx.vault, normalized);
@@ -508,7 +512,7 @@ function readTopic(
const scoped = Object.freeze({
...result,
signals: view.refs.keep(result.signals, (s) => [s.id]),
- all_log_events: view.refs.keep(result.all_log_events, (e) => logEntryArtifactRefs(e)),
+ all_log_events: logEntriesAtReach(view.refs, result.all_log_events),
});
return {
uri,
@@ -533,7 +537,9 @@ function readLog(
throw new MCPError(INTERNAL_ERROR, `no log file for date '${rawDate}'`);
}
const shard = (f: { path: string }): string =>
- withVisibleLogEvents(readFileSync(f.path, "utf8"), view.refs);
+ shardAtReach(readFileSync(f.path, "utf8"), view.refs, () =>
+ parseLogDayFile(ctx.vault, rawDate, f.path),
+ );
// The single-shard read stays verbatim (trailing newline included), so
// a vault with the gate off is byte-identical to the pre-filter shape.
if (files.length === 1) return { uri, mimeType: MIME_MARKDOWN, text: shard(files[0]!) };
@@ -543,40 +549,36 @@ function readLog(
/** Start of one rendered log event: `## <time> — <kind>`. */
const LOG_EVENT_HEADING = "\n## ";
+/** The blank line {@link appendLogEvent} keeps between a shard's blocks. */
+const LOG_BLOCK_SEPARATOR = "\n\n";
+
/**
- * Drop the rendered log events naming an artifact this caller may not
- * see, keeping the shard's frontmatter and day heading.
+ * One shard as a caller behind `view` may read it: the frontmatter and
+ * day heading verbatim, then the entries the Brain log event rule keeps
+ * ({@link logEntriesAtReach}), each rendered as the appender writes it.
+ * A shared dream keeps its readable transitions, and a retired record is
+ * judged under each of its spellings, so the shard reads as it would in a
+ * vault that never logged the withheld record.
*
* A shard is shared by construction - it is named by date and carries no
- * `owner:`, which is why `owner-scope-view.ts` leaves `Brain/log/` out
- * of its id-resolution directories. Its CONTENTS are not: every
- * `apply-evidence`, `promote` and `retire` event names the preference it
- * was about, by id and by vault-relative path, and `brain_event_trace`
- * already filters exactly those rows. Serving the same events as raw
- * markdown was the tool's boundary re-opened one protocol verb along.
+ * `owner:` - but its CONTENTS name preferences by id and by path, which
+ * is why the events are filtered rather than the page.
*
- * The split is structural - the `## ` heading the log renderer emits per
- * event, and the wikilink / `.md` path tokens inside the section - not a
- * match on any event's prose.
+ * A block the parser cannot read is not served below local reach: what
+ * it names cannot be judged, so it fails closed.
*/
-function withVisibleLogEvents(markdown: string, view: ArtifactRefView): string {
+function shardAtReach(
+ markdown: string,
+ view: ArtifactRefView,
+ parse: () => ParseLogDayResult,
+): string {
if (view.filtersNothing) return markdown;
- const parts = markdown.split(LOG_EVENT_HEADING);
- const head = parts[0]!;
- const kept = parts
- .slice(1)
- .filter((section) => view.row(...logSectionRefs(section)))
- .map((section) => LOG_EVENT_HEADING + section);
- return head + kept.join("");
-}
-
-/** Every artifact one rendered log section names, id- or path-shaped. */
-function logSectionRefs(section: string): ReadonlyArray<string> {
- const refs = [...extractWikilinkRichBodies(section)];
- for (const token of section.split(/\s+/)) {
- if (token.endsWith(".md")) refs.push(token);
- }
- return refs;
+ const firstEvent = markdown.indexOf(LOG_EVENT_HEADING);
+ const head = firstEvent < 0 ? markdown : markdown.slice(0, firstEvent);
+ const blocks = logEntriesAtReach(view, parse().entries).map((e) =>
+ renderLogEntryBlock(e).trimEnd(),
+ );
+ return [head.trimEnd(), ...blocks].join(LOG_BLOCK_SEPARATOR) + "\n";
}
function readBacklinks(
@@ -602,7 +604,10 @@ function readBacklinks(
// refusal here would be the one response shape that proves the page
// exists. The echoed target is the caller's own argument.
const index = buildBacklinkIndex(ctx.vault, view.ownerScope);
- const refs = view.refs.visible(target)
+ // Every spelling the target answers to across its retirement: a
+ // retired reserved record is otherwise asked as `pref-x`, which names
+ // no page once `ret-x` is the one on disk.
+ const refs = view.refs.row(...recordRefs(target))
? (index.get(target) ?? []).filter((ref) => view.refs.visible(ref.source))
: [];
return {
diff --git a/src/mcp/search-tools.ts b/src/mcp/search-tools.ts
index 5843d41c..81aaf52b 100644
--- a/src/mcp/search-tools.ts
+++ b/src/mcp/search-tools.ts
@@ -563,6 +563,11 @@ const SEARCH_OUTPUT_SCHEMA: NonNullable<ToolDefinition["outputSchema"]> = {
*/
const RECALL_SCORES_ARG_NAME = "scores";
+/** The bound on the gate's `session_id` and `turn_id` correlation arguments. */
+const GATE_CORRELATION_ID_MAX_CHARS = 512;
+/** The bound on the gate's `telemetry_host` label. */
+const GATE_TELEMETRY_HOST_MAX_CHARS = 200;
+
const RECALL_GATE_INPUT_SCHEMA: Record<string, unknown> = {
type: "object",
properties: {
@@ -585,14 +590,19 @@ const RECALL_GATE_INPUT_SCHEMA: Record<string, unknown> = {
},
telemetry_host: {
type: "string",
- maxLength: 200,
+ maxLength: GATE_TELEMETRY_HOST_MAX_CHARS,
description: "Optional host/client label recorded on the telemetry record.",
},
session_id: {
type: "string",
- maxLength: 512,
+ maxLength: GATE_CORRELATION_ID_MAX_CHARS,
description: "Optional session correlation id recorded on the telemetry record.",
},
+ turn_id: {
+ type: "string",
+ maxLength: GATE_CORRELATION_ID_MAX_CHARS,
+ description: "Optional turn correlation id recorded on the telemetry record.",
+ },
[RECALL_SCORES_ARG_NAME]: RECALL_SCORES_SCHEMA,
[MATCH_QUALITY_ARG_NAME]: matchQualitySchema(RECALL_SCORES_ARG_NAME),
[DECISION_ANSWERABLE_ARG_NAME]: decisionAnswerableSchema(RECALL_SCORES_ARG_NAME),
@@ -1256,6 +1266,14 @@ async function toolBrainRecallGate(
}
const previousPrompt = coerceStringOptional(args, "previous_prompt", 4000);
const explicit = coerceBoolOptional(args, "explicit") ?? false;
+ // The telemetry correlation arguments are checked here, before anything
+ // runs, rather than inside the telemetry thunks below: those are
+ // fail-open, so a bound broken there was dropped without a word, and
+ // with telemetry off it was never checked at all.
+ const telemetryHost =
+ coerceStringOptional(args, "telemetry_host", GATE_TELEMETRY_HOST_MAX_CHARS) ?? "mcp";
+ const sessionId = coerceStringOptional(args, "session_id", GATE_CORRELATION_ID_MAX_CHARS);
+ const turnId = coerceStringOptional(args, "turn_id", GATE_CORRELATION_ID_MAX_CHARS);
const decision = evaluateSurfacingGate({
prompt,
previousPrompt: previousPrompt ?? null,
@@ -1296,15 +1314,14 @@ async function toolBrainRecallGate(
// emit kernel (t_5d7aa7c5) - the payload thunk never runs with the
// config off, and a broken continuity store never breaks the gate's
// pure-diagnostic contract (fail-open).
- emitGatedTelemetry(resolveRecallGateTelemetry(ctx.configPath ?? undefined), () => {
- const host = coerceStringOptional(args, "telemetry_host", 200) ?? "mcp";
- const sessionId = coerceStringOptional(args, "session_id", 512);
- return emitGateTelemetry(ctx.vault, {
- host,
+ emitGatedTelemetry(resolveRecallGateTelemetry(ctx.configPath ?? undefined), () =>
+ emitGateTelemetry(ctx.vault, {
+ host: telemetryHost,
prompt,
retrieve: decision.retrieve,
reason: decision.reason,
...(sessionId !== undefined ? { sessionId } : {}),
+ ...(turnId !== undefined ? { turnId } : {}),
...(verdict !== undefined && answerableVerdict !== undefined
? {
decisionAnswerable: {
@@ -1314,8 +1331,8 @@ async function toolBrainRecallGate(
},
}
: {}),
- });
- });
+ }),
+ );
if (refused !== null) throw refused;
if (verdict === undefined) return { ...decision };
// signals-that-survive, unit 6: an unmet verdict is stamped onto the
@@ -1338,14 +1355,13 @@ async function toolBrainRecallGate(
// same fail-open kernel as everything else this handler writes.
const negative = verdict.resultCount === 0 ? await assessNegativeRecall(ctx) : null;
if (negative !== null) {
- emitGatedTelemetry(resolveRecallGateTelemetry(ctx.configPath ?? undefined), () => {
- const sessionId = coerceStringOptional(args, "session_id", 512);
- return emitNegativeRecallTelemetry(ctx.vault, {
+ emitGatedTelemetry(resolveRecallGateTelemetry(ctx.configPath ?? undefined), () =>
+ emitNegativeRecallTelemetry(ctx.vault, {
prompt,
verdict: negative,
...(sessionId !== undefined ? { sessionId } : {}),
- });
- });
+ }),
+ );
}
return {
...decision,
@@ -1392,10 +1408,12 @@ async function toolBrainRecallGate(
* that reads no claim graph has no grounds for `did_not_happen`.
*/
async function assessNegativeRecall(ctx: ServerContext): Promise<NegativeRecallVerdict> {
- const verdict = await probeRetrievalCorpus(() =>
- resolveSearchConfig({ vault: ctx.vault, configPath: ctx.configPath ?? undefined }),
+ const reach = contextReach(ctx);
+ const verdict = await probeRetrievalCorpus(
+ () => resolveSearchConfig({ vault: ctx.vault, configPath: ctx.configPath ?? undefined }),
+ reach,
);
- return corpusVerdictAtReach(verdict, contextReach(ctx));
+ return corpusVerdictAtReach(verdict, reach);
}
const RECALL_FEEDBACK_INPUT_SCHEMA: Record<string, unknown> = {
diff --git a/tests/cli/brain-batch-plan-ignore-warnings.test.ts b/tests/cli/brain-batch-plan-ignore-warnings.test.ts
index 68ccbfd3..430d50b3 100644
--- a/tests/cli/brain-batch-plan-ignore-warnings.test.ts
+++ b/tests/cli/brain-batch-plan-ignore-warnings.test.ts
@@ -208,14 +208,14 @@ describe("o2b brain batch-plan skip-reason counts (P4)", () => {
describe("o2b brain batch-plan unclassifiable counts (P4)", () => {
test("the human surface renders the per-extension counts", async () => {
write("mono/a.md");
- write("mono/pic.png", "binary");
- write("mono/data.csv", "a,b\n");
+ write("mono/pic.bin", "binary");
+ write("mono/data.dat", "a,b\n");
const res = await runCli(["brain", "batch-plan", "mono"], { env: ENV() });
expect(res.returncode).toBe(0);
expect(res.stdout).toContain("2 unclassifiable file(s)");
- expect(res.stdout).toContain(".png: 1");
- expect(res.stdout).toContain(".csv: 1");
+ expect(res.stdout).toContain(".bin: 1");
+ expect(res.stdout).toContain(".dat: 1");
});
test("a tree with no unclassifiable files renders no count and carries no key on the wire", async () => {
@@ -233,12 +233,66 @@ describe("o2b brain batch-plan unclassifiable counts (P4)", () => {
test("the --json payload carries the total and per-extension counts", async () => {
write("mono/a.md");
- write("mono/pic.png", "binary");
- write("mono/data.csv", "a,b\n");
+ write("mono/pic.bin", "binary");
+ write("mono/data.dat", "a,b\n");
const res = await runCli(["brain", "batch-plan", "mono", "--json"], { env: ENV() });
expect(res.returncode).toBe(0);
const plan = JSON.parse(res.stdout);
- expect(plan.unclassifiable).toEqual({ total: 2, by_extension: { ".png": 1, ".csv": 1 } });
+ expect(plan.unclassifiable).toEqual({ total: 2, by_extension: { ".bin": 1, ".dat": 1 } });
+ });
+});
+
+describe("o2b brain batch-plan source formats", () => {
+ const PAGE_HTML =
+ "<html><head><title>Release notes</title></head><body><h1>Overview</h1><p>Fish &amp; chips</p><h2>Install</h2><p>Run it.</p></body></html>\n";
+
+ function clips(): void {
+ write("Clips/page.html", PAGE_HTML);
+ write("Clips/parts.csv", "name,qty\nbolt,4\nnut,7\n");
+ write("Clips/report.pdf", "%PDF-1.7\n");
+ write("Clips/blob.bin", "abc");
+ write("Clips/notes.md", "# Notes\n\nHello\n");
+ }
+
+ test("the text surface lists the planned format and names the format skip", async () => {
+ clips();
+ const res = await runCli(["brain", "batch-plan", "Clips"], { env: ENV() });
+ expect(res.returncode).toBe(0);
+ const body = res.stdout.replace(/\(plan [0-9a-f]+\)/, "(plan <plan id>)");
+ expect(body).toContain(
+ [
+ "batch-plan: Clips (plan <plan id>)",
+ " 3 file(s) to ingest in 1 batch(es); 0 unchanged skipped",
+ " batch 0: 3 file(s), 175 byte(s)",
+ " - Clips/notes.md (new, 15B)",
+ " - Clips/page.html (new, 138B, html)",
+ " - Clips/parts.csv (new, 22B, csv)",
+ " 1 non-extractable page(s) skipped:",
+ " - Clips/report.pdf (format-not-extractable: pdf)",
+ " by reason: format-not-extractable=1",
+ " 1 unclassifiable file(s) not planned, by extension:",
+ " - .bin: 1",
+ ].join("\n"),
+ );
+ });
+
+ test("--json carries format on the planned file and the format skip", async () => {
+ clips();
+ const res = await runCli(["brain", "batch-plan", "Clips", "--json"], { env: ENV() });
+ expect(res.returncode).toBe(0);
+ const plan = JSON.parse(res.stdout);
+ expect(plan.total_files).toBe(3);
+ expect(plan.total_bytes).toBe(175);
+ expect(plan.skipped_non_extractable).toEqual([
+ { path: "Clips/report.pdf", reason: "format-not-extractable", detail: "pdf" },
+ ]);
+ expect(plan.skip_reason_counts).toEqual({ "format-not-extractable": 1 });
+ expect(plan.unclassifiable).toEqual({ total: 1, by_extension: { ".bin": 1 } });
+ expect(plan.batches[0].files).toEqual([
+ { path: "Clips/notes.md", bytes: 15, status: "new" },
+ { path: "Clips/page.html", bytes: 138, status: "new", format: "html" },
+ { path: "Clips/parts.csv", bytes: 22, status: "new", format: "csv" },
+ ]);
});
});
diff --git a/tests/cli/brain-extract.test.ts b/tests/cli/brain-extract.test.ts
new file mode 100644
index 00000000..bbe2bd6a
--- /dev/null
+++ b/tests/cli/brain-extract.test.ts
@@ -0,0 +1,285 @@
+/**
+ * `o2b brain extract <file> [--json]`: the read-only preview of what ingest
+ * derives from one source file, for every registered format. The text and
+ * JSON shapes are pinned against `docs/brainstorm/non-markdown-ingest/
+ * cli-output/expected-output.md`; "could not extract" is data, so every
+ * answer exits 0.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { spawnSync } from "node:child_process";
+import { mkdtempSync, rmSync, symlinkSync, truncateSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { CLI_COMMAND_MANIFEST, type CliCommandManifest } from "../../src/cli/command-manifest.ts";
+import { HTML_EXTRACT_MAX_SOURCE_BYTES } from "../../src/core/brain/ingest/extract-html.ts";
+import { IS_WINDOWS } from "../helpers/platform.ts";
+import { runCli } from "../helpers/run-cli.ts";
+
+/** The fixture files of the expected-output document. */
+const PAGE_HTML =
+ "<html><head><title>Release notes</title></head><body><h1>Overview</h1>" +
+ "<p>Fish &amp; chips</p><h2>Install</h2><p>Run it.</p></body></html>";
+const PARTS_CSV = "name,qty\nbolt,4\nnut,7\n";
+const TABLE_SECTION = "## Table\n\n### Rows 1-2\n\n```table\nname | qty\nbolt | 4\nnut | 7\n```";
+
+/** The manifest entry of `o2b brain extract`. */
+function extractManifest(): CliCommandManifest | undefined {
+ const brain = CLI_COMMAND_MANIFEST.commands.find((c) => c.name === "brain");
+ return brain?.commands?.find((c) => c.name === "extract");
+}
+
+let work: string;
+
+/** Write `content` to `name` in the work folder and answer its path. */
+function fixture(name: string, content: string): string {
+ const path = join(work, name);
+ writeFileSync(path, content, "utf8");
+ return path;
+}
+
+/** Run the verb with `--json` on `path`, expecting exit 0, and parse its output. */
+async function extractJson(path: string): Promise<Record<string, unknown>> {
+ const res = await runCli(["brain", "extract", path, "--json"]);
+ expect(res.returncode).toBe(0);
+ return JSON.parse(res.stdout) as Record<string, unknown>;
+}
+
+/** Run the verb in text mode on `path`, expecting exit 0, and answer stdout. */
+async function extractText(path: string): Promise<string> {
+ const res = await runCli(["brain", "extract", path]);
+ expect(res.returncode).toBe(0);
+ return res.stdout;
+}
+
+beforeEach(() => {
+ work = mkdtempSync(join(tmpdir(), "o2b-extract-cli-"));
+});
+
+afterEach(() => {
+ rmSync(work, { recursive: true, force: true });
+});
+
+describe("o2b brain extract", () => {
+ test("is in the CLI manifest with a --json flag", () => {
+ expect(extractManifest()?.flags).toEqual([{ name: "json", type: "boolean" }]);
+ });
+
+ test("an HTML file prints its title and parts", async () => {
+ const path = fixture("page.html", PAGE_HTML);
+ expect(await extractText(path)).toBe(
+ [
+ `extract: ${path} (html)`,
+ " title: Release notes",
+ " 2 part(s):",
+ " h1 Overview | lines 1-2",
+ " h2 Overview > Install | lines 3-4",
+ "",
+ ].join("\n"),
+ );
+ });
+
+ test("an HTML file as JSON gives the text and every part", async () => {
+ const path = fixture("page.html", PAGE_HTML);
+ expect(await extractJson(path)).toEqual({
+ ok: true,
+ path,
+ extracted: true,
+ format: "html",
+ title: "Release notes",
+ text: "Overview\nFish & chips\nInstall\nRun it.",
+ parts: [
+ {
+ index: 0,
+ level: 1,
+ heading: "Overview",
+ trail: "Overview",
+ line_start: 1,
+ line_end: 2,
+ source_offset: 53,
+ },
+ {
+ index: 1,
+ level: 2,
+ heading: "Install",
+ trail: "Overview > Install",
+ line_start: 3,
+ line_end: 4,
+ source_offset: 93,
+ },
+ ],
+ });
+ });
+
+ test("an HTML file whose bytes are not UTF-8 is not extracted, by name", async () => {
+ const path = join(work, "bad.html");
+ writeFileSync(path, new Uint8Array([0x3c, 0x70, 0x3e, 0xff, 0xfe]));
+ expect(await extractJson(path)).toEqual({
+ ok: true,
+ path,
+ extracted: false,
+ format: "html",
+ reason: "not-utf8",
+ });
+ });
+
+ test("a CSV file prints its counts and the table section", async () => {
+ const path = fixture("parts.csv", PARTS_CSV);
+ expect(await extractText(path)).toBe(
+ [
+ `extract: ${path} (csv)`,
+ " comma-delimited, 2 column(s), 2 row(s), 2 rendered, 0 redacted cell(s)",
+ TABLE_SECTION,
+ "",
+ ].join("\n"),
+ );
+ });
+
+ test("a CSV file as JSON gives the counts and the section", async () => {
+ const path = fixture("parts.csv", PARTS_CSV);
+ const out = await extractJson(path);
+ expect(out).toEqual({
+ ok: true,
+ path,
+ extracted: true,
+ format: "csv",
+ delimiter: "comma",
+ columns: 2,
+ rows: 2,
+ rows_rendered: 2,
+ redacted_cells: 0,
+ section: TABLE_SECTION,
+ });
+ expect(Object.keys(out)).toEqual([
+ "ok",
+ "path",
+ "extracted",
+ "format",
+ "delimiter",
+ "columns",
+ "rows",
+ "rows_rendered",
+ "redacted_cells",
+ "section",
+ ]);
+ });
+
+ test("a CSV with an unterminated quote is not extracted and names the record", async () => {
+ const path = fixture("broken.csv", 'a,b\n"open,1\n');
+ const out = await extractJson(path);
+ expect(out).toMatchObject({
+ ok: true,
+ path,
+ extracted: false,
+ format: "csv",
+ reason: "malformed-quoting",
+ });
+ expect(typeof out["detail"]).toBe("string");
+ expect(await extractText(path)).toStartWith(
+ `extract: ${path} (csv)\n not extracted: malformed-quoting (`,
+ );
+ });
+
+ test("a PDF and a Markdown file are named, not extracted", async () => {
+ const pdf = fixture("report.pdf", "%PDF-1.7");
+ const md = fixture("notes.md", "# Notes\n\nbody\n");
+ expect(await extractJson(pdf)).toEqual({
+ ok: true,
+ path: pdf,
+ extracted: false,
+ format: "pdf",
+ reason: "format-not-extractable",
+ });
+ expect(await extractJson(md)).toEqual({
+ ok: true,
+ path: md,
+ extracted: false,
+ format: "text",
+ reason: "format-read-verbatim",
+ });
+ expect(await extractText(pdf)).toBe(
+ `extract: ${pdf} (pdf)\n not extracted: format-not-extractable\n`,
+ );
+ });
+
+ test("an unknown extension answers a null format", async () => {
+ const path = fixture("blob.xyz", "abc");
+ expect(await extractJson(path)).toEqual({
+ ok: true,
+ path,
+ extracted: false,
+ format: null,
+ reason: "format-unknown",
+ });
+ expect(await extractText(path)).toBe(
+ `extract: ${path} (unknown)\n not extracted: format-unknown\n`,
+ );
+ });
+
+ test.skipIf(IS_WINDOWS)("a FIFO is answered as not a regular file without blocking", async () => {
+ const path = join(work, "pipe.html");
+ expect(spawnSync("mkfifo", [path]).status).toBe(0);
+ expect(await extractJson(path)).toEqual({
+ ok: true,
+ path,
+ extracted: false,
+ format: "html",
+ reason: "not-a-regular-file",
+ });
+ });
+
+ test("a file over the read cap is named source-too-large, as data", async () => {
+ const path = fixture("huge.csv", "a,b\n");
+ truncateSync(path, HTML_EXTRACT_MAX_SOURCE_BYTES + 1);
+ expect(await extractJson(path)).toEqual({
+ ok: true,
+ path,
+ extracted: false,
+ format: "csv",
+ reason: "source-too-large",
+ });
+ expect(await extractText(path)).toBe(
+ `extract: ${path} (csv)\n not extracted: source-too-large\n`,
+ );
+ });
+
+ test("a format with no extractor is named before the file is read", async () => {
+ const path = fixture("scan.pdf", "%PDF-1.7\n");
+ truncateSync(path, HTML_EXTRACT_MAX_SOURCE_BYTES + 1);
+ expect(await extractJson(path)).toEqual({
+ ok: true,
+ path,
+ extracted: false,
+ format: "pdf",
+ reason: "format-not-extractable",
+ });
+ });
+
+ test.each(["absent.html", "absent.pdf"])(
+ "a missing file is an error, not data (%s)",
+ async (name) => {
+ const res = await runCli(["brain", "extract", join(work, name)]);
+ expect(res.returncode).toBe(1);
+ expect(res.stderr).toContain(name);
+ },
+ );
+
+ test.skipIf(IS_WINDOWS).each([".html", ".md"])(
+ "a symbolic link is an error, not data (%s)",
+ async (ext) => {
+ const real = fixture(`real${ext}`, PAGE_HTML);
+ const link = join(work, `link${ext}`);
+ symlinkSync(real, link);
+ const res = await runCli(["brain", "extract", link]);
+ expect(res.returncode).toBe(1);
+ expect(res.stderr).toContain(`link${ext}`);
+ },
+ );
+
+ test("no file argument is a usage error", async () => {
+ const res = await runCli(["brain", "extract"]);
+ expect(res.returncode).toBe(2);
+ expect(res.stderr).toContain("usage: o2b brain extract <file> [--json]");
+ });
+});
diff --git a/tests/core/architecture/verdict-vocabulary-census.test.ts b/tests/core/architecture/verdict-vocabulary-census.test.ts
index b26ff5e5..e4f0b738 100644
--- a/tests/core/architecture/verdict-vocabulary-census.test.ts
+++ b/tests/core/architecture/verdict-vocabulary-census.test.ts
@@ -225,6 +225,22 @@ import {
SKIPPED_PAGE_REASON,
SKIPPED_PAGE_REASONS,
} from "../../../src/core/brain/ingest/extractable-gate.ts";
+import {
+ isSourceExtractSkipReason,
+ isSourceFormat,
+ SOURCE_EXTRACT_SKIP_REASON,
+ SOURCE_EXTRACT_SKIP_REASONS,
+ SOURCE_FORMAT,
+ SOURCE_FORMATS,
+} from "../../../src/core/brain/ingest/source-formats.ts";
+import {
+ isTableNoteSkipReason,
+ isTableTruncation,
+ TABLE_NOTE_SKIP_REASON,
+ TABLE_NOTE_SKIP_REASONS,
+ TABLE_TRUNCATION,
+ TABLE_TRUNCATIONS,
+} from "../../../src/core/brain/ingest/table-note.ts";
import {
EGRESS_REDACTION,
EGRESS_REDACTION_STATUSES,
@@ -773,6 +789,44 @@ const CENSUS: ReadonlyArray<VocabularyUnderCensus> = Object.freeze([
members: SKIPPED_PAGE_REASONS,
guard: isSkippedPageReason,
},
+ {
+ // The format a source file is, by extension. The token crosses the MCP
+ // wire (a planned file's `format`, a format skip's `detail`) and the CLI
+ // JSON verbatim, so a format added to the registry and forgotten in the
+ // list is one no reader can narrow.
+ name: "SOURCE_FORMAT",
+ values: SOURCE_FORMAT,
+ members: SOURCE_FORMATS,
+ guard: isSourceFormat,
+ },
+ {
+ // Why a source's content was not extracted. "Could not extract" is data
+ // on the ingest result and the extract verb's JSON, so the reason must
+ // survive the round trip through a string.
+ name: "SOURCE_EXTRACT_SKIP_REASON",
+ values: SOURCE_EXTRACT_SKIP_REASON,
+ members: SOURCE_EXTRACT_SKIP_REASONS,
+ guard: isSourceExtractSkipReason,
+ },
+ {
+ // Why a CSV or TSV source has no rendered table. The token crosses the
+ // MCP wire on the ingest result's `table` and the extract verb's JSON,
+ // so a reason added here and forgotten in the list is one no reader can
+ // narrow.
+ name: "TABLE_NOTE_SKIP_REASON",
+ values: TABLE_NOTE_SKIP_REASON,
+ members: TABLE_NOTE_SKIP_REASONS,
+ guard: isTableNoteSkipReason,
+ },
+ {
+ // Which cap cut a rendered table. Persisted as `table_truncated` on the
+ // summary page and read back across the wire, so every cap must survive
+ // the round trip through a string.
+ name: "TABLE_TRUNCATION",
+ values: TABLE_TRUNCATION,
+ members: TABLE_TRUNCATIONS,
+ guard: isTableTruncation,
+ },
{
// C1. What a given export path does about secrets on the way out.
// Registered because the value is the load-bearing field of a
@@ -1667,7 +1721,7 @@ const SCANNED = scanVocabularies(SOURCE_TREE);
* How many four-piece vocabularies `src/` currently holds. Measured, and
* kept as an equality rather than a floor - see the population test.
*/
-const VOCABULARY_POPULATION = 84;
+const VOCABULARY_POPULATION = 88;
const REGISTERED = new Map(CENSUS.map((entry) => [entry.name, entry] as const));
describe("verdict vocabulary census", () => {
diff --git a/tests/core/architecture/visibility-surface-census.test.ts b/tests/core/architecture/visibility-surface-census.test.ts
index a0893acb..c96b3f39 100644
--- a/tests/core/architecture/visibility-surface-census.test.ts
+++ b/tests/core/architecture/visibility-surface-census.test.ts
@@ -216,6 +216,15 @@ const NOTE_CONTENT_PRODUCERS: ReadonlyArray<ProducerRule> = Object.freeze([
specifierIncludes: "/brain/link-graph/unlinked-mentions.ts",
identifiers: ["findUnlinkedMentions"],
},
+ // The concept cluster hands back the backlink index's linkers and the
+ // unlinked mentions through a `src/core/` helper, so the analytics
+ // dispatcher that serves it was outside the population while its
+ // timeline, belief-evolution and concept-synthesis views named reserved
+ // records.
+ {
+ specifierIncludes: "/brain/link-graph/concept-cluster.ts",
+ identifiers: ["buildConceptCluster"],
+ },
{ specifierIncludes: "/brain/portability/sources.ts", identifiers: ["aggregateSources"] },
{ specifierIncludes: "/brain/query.ts", identifiers: ["queryByPreference", "queryByTopic"] },
{ specifierIncludes: "/brain/pref-audit.ts", identifiers: ["readPrefAudit"] },
@@ -470,8 +479,12 @@ function reasonProblems(entries: ReadonlyArray<VisibilitySurfaceEntry>): {
*
* 54 before the vocabulary gained the two compiled digests. The one new
* name is `brain_brief`, whose digest view renders preference records.
+ *
+ * 55 before the vocabulary gained `buildConceptCluster`. The one new name
+ * is `brain_analytics`, whose timeline, belief-evolution and
+ * concept-synthesis views render log events and backlinks.
*/
-const MCP_TOOL_POPULATION_SIZE = 55;
+const MCP_TOOL_POPULATION_SIZE = 56;
/** Measured: MCP resources + templates. */
const MCP_RESOURCE_POPULATION_SIZE = 8;
/** Measured: hand-enumerated CLI verb mirrors. */
@@ -533,6 +546,7 @@ describe("visibility surface census", () => {
"brain_agent_query",
"brain_backlinks",
"brain_bridges",
+ "brain_claims",
"brain_clusters",
"brain_context",
"brain_deep_synthesis",
diff --git a/tests/core/brain/architect-dependencies.test.ts b/tests/core/brain/architect-dependencies.test.ts
index 4a2548f1..93468e05 100644
--- a/tests/core/brain/architect-dependencies.test.ts
+++ b/tests/core/brain/architect-dependencies.test.ts
@@ -314,9 +314,41 @@ test("a module whose name a link cannot carry is named once and left out of link
.split("\n")
.filter((line) => line.includes("-->"));
expect(edges).toEqual([' mod3["web"] --> mod1["core"]']);
+ // The dropped edge is named under the diagram, as code spans, never as a link.
+ expect(regionBody(text, "module-dependencies").split("\n")).toContain(
+ 'Not drawn (a module name holds a character a link cannot carry): `"web"` -> `"a[b]"`',
+ );
const web = moduleNote(res, "web");
- expect(web).not.toContain("a[b]");
+ expect(web).not.toContain("modules/a[b]");
expect(web).toContain(`- [[Brain/projects/arch/${res.repoKey}/modules/core|core]]`);
+ expect(regionBody(web, "dependencies").split("\n")).toContain('Not linked: `"a[b]"`');
+});
+
+test("only unlinkable edges are named instead of a false no-edge sentence", () => {
+ put("packages/a[b]/package.json", JSON.stringify({ name: "@mono/ab" }));
+ put("packages/a[b]/index.ts", "// x\n");
+ put(
+ "packages/web/package.json",
+ JSON.stringify({ name: "@mono/web", dependencies: { "@mono/ab": "*" } }),
+ );
+ const res = generateArchDocs(vault, project);
+ expect(regionBody(overview(res), "module-dependencies")).toBe(
+ "Every declared module edge touches a module whose name a link cannot carry, " +
+ "so none is drawn:\n" +
+ '- `"web"` -> `"a[b]"`',
+ );
+ const web = regionBody(moduleNote(res, "web"), "dependencies").split("\n");
+ expect(web).toContain('Not linked: `"a[b]"`');
+ expect(web).not.toContain("Depends on: no other module");
+ expect(moduleNote(res, "web")).not.toContain("depends_on:");
+ // A module with no edge at all keeps the no-edge sentence.
+ expect(regionBody(moduleNote(res, "core"), "dependencies").split("\n")).toContain(
+ "Depends on: no other module",
+ );
+
+ const second = generateArchDocs(vault, project);
+ expect(second.updated).toBe(0);
+ expect(second.unchanged).toBe(2 + second.modulePaths.length);
});
// Windows refuses a control character in a directory name.
diff --git a/tests/core/brain/continuity/redaction.test.ts b/tests/core/brain/continuity/redaction.test.ts
new file mode 100644
index 00000000..1619fd32
--- /dev/null
+++ b/tests/core/brain/continuity/redaction.test.ts
@@ -0,0 +1,34 @@
+/**
+ * `safeContinuityPayload` flags a payload that held a private region and
+ * strips it, in time linear in the payload's size.
+ */
+
+import { describe, expect, test } from "bun:test";
+
+import { safeContinuityPayload } from "../../../../src/core/brain/continuity/redaction.ts";
+
+const PRIVATE_BODY = "between the tags";
+/** 720 KB of open-close pairs: the size the quadratic test took 21.8 s on. */
+const PRIVATE_RUN_BYTES = 720 * 1024;
+const PRIVATE_RUN_CEILING_MS = 1_000;
+
+describe("safeContinuityPayload", () => {
+ test("a private region is flagged and stripped", () => {
+ const result = safeContinuityPayload({ note: `a <private>${PRIVATE_BODY}</private> b` });
+ expect(result.private).toBe(true);
+ expect(JSON.stringify(result.payload)).not.toContain(PRIVATE_BODY);
+ });
+
+ test("a payload without one is not flagged", () => {
+ expect(safeContinuityPayload({ note: "plain text" }).private).toBe(false);
+ });
+
+ test("a long run of private tags is flagged in linear time", () => {
+ const unit = "<private>";
+ const run = unit.repeat(Math.ceil(PRIVATE_RUN_BYTES / unit.length));
+ const started = performance.now();
+ const result = safeContinuityPayload({ note: run });
+ expect(performance.now() - started).toBeLessThan(PRIVATE_RUN_CEILING_MS);
+ expect(result.private).toBe(true);
+ });
+});
diff --git a/tests/core/brain/doctor-issue-refs.test.ts b/tests/core/brain/doctor-issue-refs.test.ts
new file mode 100644
index 00000000..876abe95
--- /dev/null
+++ b/tests/core/brain/doctor-issue-refs.test.ts
@@ -0,0 +1,38 @@
+/**
+ * Every spelling a doctor finding names a record by.
+ *
+ * A finding is kept for a caller only when every artifact it names is
+ * readable, and a preference retired after the finding's reference was
+ * written lives on disk as `ret-<slug>`. Each place a finding can name the
+ * record - its target, a source, a message wikilink - must therefore
+ * answer to the retired spelling on its own, not only jointly.
+ */
+
+import { describe, expect, test } from "bun:test";
+
+import { doctorIssueRefs } from "../../../src/core/brain/diagnostics.ts";
+
+const VAULT = "vault-root";
+const LIVE_ID = "pref-bygone";
+const RETIRED_ID = "ret-bygone";
+const PLAIN_MESSAGE = "a finding";
+
+describe("doctorIssueRefs", () => {
+ test("a target spelled pref- also names its retired spelling", () => {
+ expect(doctorIssueRefs(VAULT, { target: LIVE_ID, message: PLAIN_MESSAGE })).toContain(
+ RETIRED_ID,
+ );
+ });
+
+ test("a message wikilink spelled pref- also names its retired spelling", () => {
+ expect(doctorIssueRefs(VAULT, { message: `broken link to [[${LIVE_ID}]]` })).toContain(
+ RETIRED_ID,
+ );
+ });
+
+ test("a source spelled pref- also names its retired spelling", () => {
+ expect(doctorIssueRefs(VAULT, { sources: [LIVE_ID], message: PLAIN_MESSAGE })).toContain(
+ RETIRED_ID,
+ );
+ });
+});
diff --git a/tests/core/brain/ingest/batch-plan.test.ts b/tests/core/brain/ingest/batch-plan.test.ts
index c91bcd58..71d0b027 100644
--- a/tests/core/brain/ingest/batch-plan.test.ts
+++ b/tests/core/brain/ingest/batch-plan.test.ts
@@ -15,7 +15,12 @@ import { tmpdir } from "node:os";
import { bootstrapBrain } from "../../../../src/core/brain/init.ts";
import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";
import { updateManifest } from "../../../../src/core/brain/ingest/content-manifest.ts";
-import { planBatches, type BatchPlan } from "../../../../src/core/brain/ingest/batch-plan.ts";
+import {
+ DEFAULT_INGESTIBLE_EXTENSIONS as PLANNER_DEFAULT_EXTENSIONS,
+ planBatches,
+ type BatchPlan,
+} from "../../../../src/core/brain/ingest/batch-plan.ts";
+import { DEFAULT_INGESTIBLE_EXTENSIONS } from "../../../../src/core/brain/ingest/source-formats.ts";
import { recordCompleted } from "../../../../src/core/brain/ingest/checkpoint.ts";
import { SKIPPED_PAGE_REASON } from "../../../../src/core/brain/ingest/extractable-gate.ts";
import { serializeBatchPlan } from "../../../../src/mcp/brain/ingest-tools.ts";
@@ -289,25 +294,25 @@ describe("planBatches — typed skip reasons + per-reason counts (P4)", () => {
describe("planBatches — unclassifiable files (P4)", () => {
test("counts files dropped for a non-ingestible extension, per extension", () => {
writeSized("Docs/a.md", 50);
- writeSized("Docs/pic.png", 50);
- writeSized("Docs/data.csv", 50);
+ writeSized("Docs/pic.bin", 50);
+ writeSized("Docs/data.dat", 50);
const plan = planBatches(vault, "Docs", CAPS);
expect(plan.unclassifiable.total).toBe(2);
- expect(plan.unclassifiable.byExtension).toEqual({ ".png": 1, ".csv": 1 });
+ expect(plan.unclassifiable.byExtension).toEqual({ ".bin": 1, ".dat": 1 });
// Counting must not alter the discovered set: the plan id (derived from
// the ingestible set) is the same with and without the foreign files.
- rmSync(join(vault, "Docs", "pic.png"));
- rmSync(join(vault, "Docs", "data.csv"));
+ rmSync(join(vault, "Docs", "pic.bin"));
+ rmSync(join(vault, "Docs", "data.dat"));
expect(planBatches(vault, "Docs", CAPS).planId).toBe(plan.planId);
});
test("hidden entries and ignore-rule matches are excluded by declaration, not counted", () => {
writeSized("Docs/a.md", 50);
- writeSized("Docs/.hidden.png", 50);
- writeSized("Docs/.hid/x.png", 50);
- writeSized("Docs/node_modules/lib/x.png", 50);
+ writeSized("Docs/.hidden.bin", 50);
+ writeSized("Docs/.hid/x.bin", 50);
+ writeSized("Docs/node_modules/lib/x.bin", 50);
writeFileSync(join(vault, "Docs", ".gitignore"), "node_modules/\n", "utf8");
const plan = planBatches(vault, "Docs", CAPS);
@@ -320,12 +325,113 @@ describe("planBatches — unclassifiable files (P4)", () => {
const clean = planBatches(vault, "Docs", CAPS);
expect("unclassifiable" in serializeBatchPlan(clean)).toBe(false);
- writeSized("Docs/pic.png", 50);
- writeSized("Docs/data.csv", 50);
+ writeSized("Docs/pic.bin", 50);
+ writeSized("Docs/data.dat", 50);
const withDrops = planBatches(vault, "Docs", CAPS);
expect(serializeBatchPlan(withDrops)["unclassifiable"]).toEqual({
total: 2,
- by_extension: { ".png": 1, ".csv": 1 },
+ by_extension: { ".bin": 1, ".dat": 1 },
});
});
});
+
+describe("planBatches — source formats", () => {
+ test("plans HTML and CSV with their format, names PDF and images as format skips, counts the rest", () => {
+ writeSized("Clips/notes.md", 15);
+ writeSized("Clips/page.html", 40);
+ writeSized("Clips/parts.csv", 22);
+ writeSized("Clips/report.pdf", 9);
+ writeSized("Clips/shot.png", 12);
+ writeSized("Clips/blob.bin", 3);
+
+ const plan = planBatches(vault, "Clips", CAPS);
+ expect(plan.batches.flatMap((b) => b.files)).toEqual([
+ { path: "Clips/notes.md", bytes: 15, status: "new" },
+ { path: "Clips/page.html", bytes: 40, status: "new", format: "html" },
+ { path: "Clips/parts.csv", bytes: 22, status: "new", format: "csv" },
+ ]);
+ expect(plan.skippedNonExtractable).toEqual([
+ { path: "Clips/report.pdf", reason: SKIPPED_PAGE_REASON.formatNotExtractable, detail: "pdf" },
+ { path: "Clips/shot.png", reason: SKIPPED_PAGE_REASON.formatNotExtractable, detail: "image" },
+ ]);
+ expect(plan.skipReasonCounts).toEqual({ [SKIPPED_PAGE_REASON.formatNotExtractable]: 2 });
+ expect(plan.unclassifiable).toEqual({ total: 1, byExtension: { ".bin": 1 } });
+ });
+
+ test("format skips precede schema skips, each sorted by path", () => {
+ page("Mixed/b.md", "memo");
+ page("Mixed/a.md", "paper");
+ writeSized("Mixed/z.pdf", 9);
+ writeSized("Mixed/c.docx", 9);
+ writeFileSync(
+ join(vault, "Brain", "_brain.yaml"),
+ "schema_version: 1\nschema:\n page_types:\n - paper\n - memo\n extractable:\n - paper\n",
+ "utf8",
+ );
+
+ const plan = planBatches(vault, "Mixed", CAPS);
+ expect(plan.skippedNonExtractable.map((s) => [s.path, s.reason, s.detail])).toEqual([
+ ["Mixed/c.docx", "format-not-extractable", "docx"],
+ ["Mixed/z.pdf", "format-not-extractable", "pdf"],
+ ["Mixed/b.md", "schema-type-not-extractable", "memo"],
+ ]);
+ expect(Object.keys(plan.skipReasonCounts)).toEqual([
+ "format-not-extractable",
+ "schema-type-not-extractable",
+ ]);
+ });
+
+ test("a caller's extensions override that lists .pdf plans it, with its format", () => {
+ writeSized("Clips/notes.md", 15);
+ writeSized("Clips/report.pdf", 9);
+
+ const plan = planBatches(vault, "Clips", { ...CAPS, extensions: [".md", ".pdf"] });
+ expect(plan.batches.flatMap((b) => b.files)).toEqual([
+ { path: "Clips/notes.md", bytes: 15, status: "new" },
+ { path: "Clips/report.pdf", bytes: 9, status: "new", format: "pdf" },
+ ]);
+ expect(plan.skippedNonExtractable).toEqual([]);
+ });
+
+ test("a Markdown-only tree serializes with no format key and no skip keys", () => {
+ writeSized("Docs/a.md", 10);
+ writeSized("Docs/b.txt", 20);
+
+ const wire = serializeBatchPlan(planBatches(vault, "Docs", CAPS));
+ expect(wire["batches"]).toEqual([
+ {
+ index: 0,
+ total_bytes: 30,
+ files: [
+ { path: "Docs/a.md", bytes: 10, status: "new" },
+ { path: "Docs/b.txt", bytes: 20, status: "new" },
+ ],
+ },
+ ]);
+ expect(Object.keys(wire)).toEqual([
+ "source_dir",
+ "max_batch_bytes",
+ "max_batch_files",
+ "total_files",
+ "total_bytes",
+ "skipped",
+ "plan_id",
+ "resumed_completed",
+ "batches",
+ ]);
+ });
+
+ test("a planned non-text file serializes format right after status", () => {
+ writeSized("Clips/page.html", 40);
+
+ const wire = serializeBatchPlan(planBatches(vault, "Clips", CAPS));
+ const files = (wire["batches"] as Array<{ files: Array<Record<string, unknown>> }>)[0]!.files;
+ expect(JSON.stringify(files[0])).toBe(
+ '{"path":"Clips/page.html","bytes":40,"status":"new","format":"html"}',
+ );
+ });
+
+ test("the planner re-exports the registry's default extensions", () => {
+ expect(PLANNER_DEFAULT_EXTENSIONS).toBe(DEFAULT_INGESTIBLE_EXTENSIONS);
+ });
+});
diff --git a/tests/core/brain/ingest/extract-html.test.ts b/tests/core/brain/ingest/extract-html.test.ts
new file mode 100644
index 00000000..81112fa4
--- /dev/null
+++ b/tests/core/brain/ingest/extract-html.test.ts
@@ -0,0 +1,512 @@
+/**
+ * The HTML extractor: a pure, linear scanner that turns the bytes of an
+ * HTML file into its title, its visible text and heading-derived parts.
+ * These tests pin the grammar (entities, skipped elements, private regions,
+ * whitespace, line breaks), the refusals and the bound on running time.
+ */
+
+import { describe, expect, test } from "bun:test";
+import { spawnSync } from "node:child_process";
+
+import { SOURCE_HASH_MAX_BYTES } from "../../../../src/core/brain/intake/source-trust.ts";
+import {
+ extractHtml,
+ HTML_EXTRACT_MAX_SOURCE_BYTES,
+ HTML_HEADING_MAX_CHARS,
+ HTML_PARTS_MAX,
+ type HtmlExtraction,
+ renderPartsSection,
+} from "../../../../src/core/brain/ingest/extract-html.ts";
+import { SOURCE_EXTRACT_SKIP_REASON } from "../../../../src/core/brain/ingest/source-formats.ts";
+import {
+ PRIVATE_REGION_PLACEHOLDER,
+ REDACTION_PLACEHOLDER,
+ SCAN_TRUNCATED_MARKER,
+} from "../../../../src/core/redactor.ts";
+import { extractTagValues, stripCode } from "../../../../src/core/tags.ts";
+import { extractWikilinks } from "../../../../src/core/vault.ts";
+import { fakeCredential } from "../../../helpers/fake-credentials.ts";
+
+const encoder = new TextEncoder();
+
+/** Small headings in the memory-bound probe: about 4 MiB of source. */
+const OMITTED_HEADINGS = 400_000;
+/**
+ * The memory the probe may grow by. Building every omitted part took about
+ * 850 MB on this input; the bounded scan takes about 40 MB.
+ */
+const SCAN_MEMORY_BOUND_BYTES = 256 * 1024 * 1024;
+/** The extractor as a module URL a probe in a fresh process imports (a URL, so a Windows path works). */
+const EXTRACT_HTML_MODULE = new URL(
+ "../../../../src/core/brain/ingest/extract-html.ts",
+ import.meta.url,
+).href;
+/**
+ * The time a source of huge headings may take. Redacting each heading
+ * unwindowed took 1.1 s for one 8 MiB heading and 2.3 s for a 4 MiB title
+ * and heading; the windowed pass leaves only the scan itself, about 40 ms
+ * for 8 MiB, so the ceiling keeps a margin of more than 20x.
+ */
+const HEADING_WINDOW_CEILING_MS = 1_000;
+/** The parts section of one part: heading, blank line, fence, the part line, fence. */
+const RENDERED_LINES_ONE_PART = 5;
+
+/** A URL-userinfo run of about `bytes`: the shape that made the unwindowed pass cost seconds. */
+function userinfoRun(bytes: number): string {
+ return "a://b:".repeat(Math.floor(bytes / 6));
+}
+
+/** The extraction of `html`, failing the test when the scanner refused it. */
+function extracted(html: string): HtmlExtraction {
+ const result = extractHtml(encoder.encode(html));
+ if (!result.extracted) throw new Error(`extraction refused: ${result.reason}`);
+ return result;
+}
+
+/** An extraction without its byte offsets, which differ between line-ending styles. */
+function withoutOffsets(x: HtmlExtraction): unknown {
+ return { text: x.text, parts: x.parts.map(({ sourceOffset: _offset, ...rest }) => rest) };
+}
+
+/** The visible text of `html`. */
+function textOf(html: string): string {
+ return extracted(html).text;
+}
+
+describe("extractHtml - text", () => {
+ test("reads the cap from the source hash ceiling", () => {
+ expect(HTML_EXTRACT_MAX_SOURCE_BYTES).toBe(SOURCE_HASH_MAX_BYTES);
+ });
+
+ test("block elements break lines, inline elements do not", () => {
+ expect(
+ textOf("<html><body><h1>Overview</h1><p>Fish <b>and</b> chips</p><div>Run it.</div></body>"),
+ ).toBe("Overview\nFish and chips\nRun it.");
+ });
+
+ test("decodes numeric entities and turns an invalid code point into U+FFFD", () => {
+ expect(textOf("<p>&#65;&#x42;&#X43;&#0;&#xD800;&#x110000;</p>")).toBe("ABC���");
+ });
+
+ test("decodes the six named entities and keeps any other named entity verbatim", () => {
+ expect(textOf("<p>&amp; &lt; &gt; &quot; &apos; a&nbsp;b &copy; &unknownthing;</p>")).toBe(
+ "& < > \" ' a b &copy; &unknownthing;",
+ );
+ });
+
+ test("a bare ampersand and a stray less-than are text", () => {
+ expect(textOf("<p>salt & pepper, 1 < 2, a <3 b, &#;</p>")).toBe(
+ "salt & pepper, 1 < 2, a <3 b, &#;",
+ );
+ });
+
+ test("raw-text and skipped elements contribute no text", () => {
+ const html = [
+ "<p>before</p>",
+ "<script>var x = '<p>not text</p>';</script>",
+ "<style>p { color: red }</style>",
+ "<noscript><p>enable scripts</p></noscript>",
+ "<template><p>template</p></template>",
+ "<svg><title>icon</title><text>svg text</text></svg>",
+ "<math><mi>x</mi></math>",
+ "<p>after</p>",
+ ].join("");
+ expect(textOf(html)).toBe("before\nafter");
+ });
+
+ test("a script whose body names another end tag stays skipped to its own end tag", () => {
+ expect(textOf("<script>document.write('</div></scr' + 'ipt>')</script ><p>shown</p>")).toBe(
+ "shown",
+ );
+ });
+
+ test("a private region becomes the placeholder", () => {
+ expect(textOf("<p>visible <private>hidden <h2>also hidden</h2> words</private> tail</p>")).toBe(
+ `visible ${PRIVATE_REGION_PLACEHOLDER} tail`,
+ );
+ });
+
+ test("a private region inside a heading stays out of the text and the part", () => {
+ const result = extracted("<h1>Plan <private>vault 4711</private> v2</h1><p>body</p>");
+ expect(result.text).toBe(`Plan ${PRIVATE_REGION_PLACEHOLDER} v2\nbody`);
+ expect(result.parts.map((p) => p.heading)).toEqual([`Plan ${PRIVATE_REGION_PLACEHOLDER} v2`]);
+ expect(renderPartsSection(result)).not.toContain("4711");
+ const unclosed = extracted("<h2>Plan <private>vault 4711</h2><p>after</p>");
+ expect(JSON.stringify(unclosed)).not.toContain("4711");
+ expect(JSON.stringify(unclosed)).not.toContain("after");
+ });
+
+ test("a private region inside a title or a textarea is not emitted", () => {
+ // Title and textarea content is read as raw text, so the private-region
+ // rule is applied to that text as well; the title reaches the CLI preview.
+ const titled = extracted("<title>Plan <private>vault 4711</private></title><p>x</p>");
+ expect(titled.title).toBe(`Plan ${PRIVATE_REGION_PLACEHOLDER}`);
+ const area = extracted("<textarea>a <private>vault 4711</private> b</textarea>");
+ expect(area.text).toBe(`a ${PRIVATE_REGION_PLACEHOLDER} b`);
+ // Stripped after decoding: an entity-encoded region hides too (fail-closed).
+ const encoded = extracted("<textarea>a &lt;private&gt;vault 4711&lt;/private&gt; b</textarea>");
+ expect(encoded.text).toBe(`a ${PRIVATE_REGION_PLACEHOLDER} b`);
+ });
+
+ test("a private region inside a textarea in a heading stays out of the parts", () => {
+ const result = extracted(
+ "<h1>Ops <textarea>key <private>vault 4711</private></textarea></h1><p>x</p>",
+ );
+ expect(result.parts.map((p) => p.heading)).toEqual([`Ops key ${PRIVATE_REGION_PLACEHOLDER}`]);
+ expect(renderPartsSection(result)).not.toContain("4711");
+ });
+
+ test("an unclosed private region hides everything after it", () => {
+ expect(textOf("<p>kept</p><private><p>hidden to the end</p>")).toBe(
+ `kept\n${PRIVATE_REGION_PLACEHOLDER}`,
+ );
+ });
+
+ test("void elements need no end tag and br breaks a line", () => {
+ expect(textOf("<p>one<br>two<br/>three<img src=x.png>four<hr>five</p>")).toBe(
+ "one\ntwo\nthreefour\nfive",
+ );
+ });
+
+ test("document-level text outside any element is kept", () => {
+ expect(textOf("loose text <b>bold</b> more")).toBe("loose text bold more");
+ });
+
+ test("whitespace collapses outside pre and survives inside it", () => {
+ expect(textOf("<p> a \t\n b </p><pre>\n x y\n\n z\n</pre><p>c</p>")).toBe(
+ "a b\n x y\n\n z\nc",
+ );
+ });
+
+ test("CRLF and lone CR give the same output as LF", () => {
+ const lf = "<p>a\nb</p><pre>one\ntwo\n</pre><h1>T</h1><p>end</p>";
+ const crlf = lf.replaceAll("\n", "\r\n");
+ const cr = lf.replaceAll("\n", "\r");
+ expect(withoutOffsets(extracted(crlf))).toEqual(withoutOffsets(extracted(lf)));
+ expect(withoutOffsets(extracted(cr))).toEqual(withoutOffsets(extracted(lf)));
+ expect(extracted(lf).text).toBe("a b\none\ntwo\nT\nend");
+ });
+
+ test("no attribute value is ever emitted", () => {
+ const credential = fakeCredential("ghp_", "attrValue0123456789abcdef");
+ const html =
+ `<a href="https://example.test/?k=${credential}" title='${credential}' ` +
+ `data-x=${credential}>link</a><img alt="${credential}">`;
+ const result = extracted(html);
+ expect(result.text).toBe("link");
+ expect(JSON.stringify(result)).not.toContain(credential);
+ });
+
+ test("a quoted attribute value may hold a greater-than sign", () => {
+ expect(textOf('<p title="a > b">body</p>')).toBe("body");
+ });
+
+ test("the title is captured and kept out of the text", () => {
+ const result = extracted(
+ "<html><head><title> Release\n notes &amp; more </title></head><body><p>x</p></body></html>",
+ );
+ expect(result.title).toBe("Release notes & more");
+ expect(result.text).toBe("x");
+ });
+
+ test("a title is folded onto one line", () => {
+ expect(extracted("<title>bell\u0007here</title>").title).toBe("bell here");
+ });
+
+ test("a document without a title answers null", () => {
+ expect(extracted("<p>x</p>").title).toBeNull();
+ });
+
+ test("comments, doctype and processing instructions are dropped", () => {
+ expect(textOf("<!DOCTYPE html><?xml version='1.0'?><!-- <p>no</p> --><p>yes</p>")).toBe("yes");
+ });
+
+ test.each(["<!-->", "<!--->"])("%s is an empty comment, not an open one", (comment) => {
+ const x = extracted(`<p>a</p>${comment}<h1>B</h1>`);
+ expect(x.text).toBe("a\nB");
+ expect(x.parts.filter((part) => part.level === 1).map((part) => part.heading)).toEqual(["B"]);
+ });
+
+ test("malformed and unclosed tags never throw and never leak markup", () => {
+ expect(textOf("<p>a</p><div class='open")).toBe("a");
+ expect(textOf("<p>a</p><!-- never closed <p>b</p>")).toBe("a");
+ expect(textOf("<p>a</p><b")).toBe("a");
+ expect(textOf("<p>a</p></>text")).toBe("a\ntext");
+ expect(textOf("<script>never closed <p>x</p>")).toBe("");
+ });
+
+ test("invalid UTF-8 is refused by name", () => {
+ const bytes = new Uint8Array([0x3c, 0x70, 0x3e, 0xff, 0xfe, 0x3c, 0x2f, 0x70, 0x3e]);
+ expect(extractHtml(bytes)).toEqual({
+ extracted: false,
+ reason: SOURCE_EXTRACT_SKIP_REASON.notUtf8,
+ });
+ });
+
+ test("a byte-order mark is not text", () => {
+ const bom = new Uint8Array([0xef, 0xbb, 0xbf, ...encoder.encode("<p>x</p>")]);
+ const result = extractHtml(bom);
+ expect(result.extracted && result.text).toBe("x");
+ });
+
+ test("runs in linear time on adversarial input", () => {
+ const size = 1_048_576;
+ const shapes = ["<a ", "<!--", "&#", "</scr", "<p title='", "&amp", "<h1>"];
+ for (const shape of shapes) {
+ const html = `<script>${shape.repeat(Math.ceil(size / shape.length))}`;
+ const plain = shape.repeat(Math.ceil(size / shape.length));
+ for (const input of [html, plain]) {
+ const started = performance.now();
+ extractHtml(encoder.encode(input));
+ expect(performance.now() - started).toBeLessThan(2_000);
+ }
+ }
+ });
+});
+
+/** The release-notes page of the CLI fixture (`cli-output/expected-output.md`). */
+const RELEASE_NOTES =
+ "<html><head><title>Release notes</title></head><body><h1>Overview</h1>" +
+ "<p>Fish &amp; chips</p><h2>Install</h2><p>Run it.</p></body></html>";
+
+describe("extractHtml - parts", () => {
+ test("pins the caps", () => {
+ expect(HTML_PARTS_MAX).toBe(256);
+ expect(HTML_HEADING_MAX_CHARS).toBe(200);
+ });
+
+ test("the fixture page gives two parts with levels, trails, spans and start-tag offsets", () => {
+ const result = extracted(RELEASE_NOTES);
+ expect(result.title).toBe("Release notes");
+ expect(result.text).toBe("Overview\nFish & chips\nInstall\nRun it.");
+ expect(result.parts).toEqual([
+ {
+ index: 0,
+ level: 1,
+ heading: "Overview",
+ trail: "Overview",
+ lineStart: 1,
+ lineEnd: 2,
+ sourceOffset: 53,
+ },
+ {
+ index: 1,
+ level: 2,
+ heading: "Install",
+ trail: "Overview > Install",
+ lineStart: 3,
+ lineEnd: 4,
+ sourceOffset: 93,
+ },
+ ]);
+ expect(result.partsOmitted).toBe(0);
+ });
+
+ test("a heading closes the sections at its level and below", () => {
+ const result = extracted(
+ "<h1>A</h1><h2>B</h2><h3>C</h3><p>c</p><h2>D</h2><h1>E</h1><h3>F</h3>",
+ );
+ expect(result.parts.map((p) => [p.level, p.trail, p.lineStart, p.lineEnd])).toEqual([
+ [1, "A", 1, 1],
+ [2, "A > B", 2, 2],
+ [3, "A > B > C", 3, 4],
+ [2, "A > D", 5, 5],
+ [1, "E", 6, 6],
+ [3, "E > F", 7, 7],
+ ]);
+ });
+
+ test("a source offset counts UTF-8 bytes, a byte-order mark included", () => {
+ const prefix = "<p>\u00e9\u4e2d\u{1F600}</p>"; // 2 + 3 + 4 bytes of text
+ const bytes = new Uint8Array([
+ 0xef,
+ 0xbb,
+ 0xbf,
+ ...encoder.encode(`${prefix}<h2 id="x">T</h2>`),
+ ]);
+ const result = extractHtml(bytes);
+ if (!result.extracted) throw new Error("refused");
+ expect(result.parts.at(-1)?.sourceOffset).toBe(3 + encoder.encode(prefix).length);
+ });
+
+ test("a preamble part exists only when text precedes the first heading", () => {
+ const withPreamble = extracted("<p>intro</p><p>more</p><h1>T</h1><p>body</p>");
+ expect(withPreamble.parts[0]).toEqual({
+ index: 0,
+ level: 0,
+ heading: "",
+ trail: "",
+ lineStart: 1,
+ lineEnd: 2,
+ sourceOffset: 0,
+ });
+ expect(withPreamble.parts[1]).toMatchObject({ index: 1, level: 1, lineStart: 3, lineEnd: 4 });
+ expect(extracted("<h1>T</h1><p>body</p>").parts[0]?.level).toBe(1);
+ expect(extracted("<p>no headings at all</p>").parts).toEqual([]);
+ });
+
+ test("an empty heading is no part and a heading inside a skipped region is none either", () => {
+ const result = extracted(
+ "<h1>A</h1><h2> </h2><private><h2>secret plan</h2></private><svg><h3>x</h3></svg><p>p</p>",
+ );
+ expect(result.parts.map((p) => p.heading)).toEqual(["A"]);
+ });
+
+ test("a heading is folded onto one line", () => {
+ expect(extracted("<h1>two<br>lines\there</h1>").parts[0]?.heading).toBe("two lines here");
+ });
+
+ test(`${HTML_PARTS_MAX + 1} headings give ${HTML_PARTS_MAX} parts and count the rest`, () => {
+ const html = Array.from({ length: HTML_PARTS_MAX + 1 }, (_, n) => `<h2>H${n}</h2>`).join("");
+ const result = extracted(html);
+ expect(result.parts.length).toBe(HTML_PARTS_MAX);
+ expect(result.partsOmitted).toBe(1);
+ expect(result.parts.at(-1)?.lineEnd).toBe(HTML_PARTS_MAX);
+ });
+
+ test("a credential in a heading or a title is redacted before the cap", () => {
+ const key = fakeCredential("sk-", "live-9f8e7d6c5b4a3210");
+ const password = fakeCredential("hunter", "2pass");
+ const result = extracted(
+ `<title>api_key=${key}</title><h1>api_key=${key}</h1>` +
+ `<h2>https://admin:${password}@db.example</h2>` +
+ `<h3>${"x".repeat(HTML_HEADING_MAX_CHARS - 25)} https://admin:${password}@db.example</h3><p>x</p>`,
+ );
+ const visible = JSON.stringify({ title: result.title, parts: result.parts });
+ for (const leaked of [key, password, password.slice(0, 5)])
+ expect(visible).not.toContain(leaked);
+ expect(result.title).toContain(REDACTION_PLACEHOLDER);
+ expect(result.parts[0]?.heading).toContain(REDACTION_PLACEHOLDER);
+ expect(renderPartsSection(result)).not.toContain(password);
+ });
+
+ describe("a huge heading or title is redacted in a bounded window", () => {
+ const MIB = 1 << 20;
+ test.each([
+ [
+ "one heading near the source cap",
+ `<h1>${userinfoRun(HTML_EXTRACT_MAX_SOURCE_BYTES - 64)}</h1>`,
+ ],
+ [
+ "a title and a heading",
+ `<title>${userinfoRun(4 * MIB - 64)}</title><h1>${userinfoRun(4 * MIB - 64)}</h1>`,
+ ],
+ ])("%s", (_name, html) => {
+ const started = performance.now();
+ const result = extracted(html);
+ expect(performance.now() - started).toBeLessThan(HEADING_WINDOW_CEILING_MS);
+ expect(Array.from(result.parts[0]?.heading ?? "").length).toBe(HTML_HEADING_MAX_CHARS);
+ });
+
+ test("a credential heading past the redactor's input cap keeps its part on one line", () => {
+ const result = extracted(`<h1>password=${"x".repeat(MIB + 16)}</h1><p>x</p>`);
+ const trail = result.parts[0]?.trail ?? "";
+ expect(trail).toContain(REDACTION_PLACEHOLDER);
+ expect(trail).not.toContain("\n");
+ expect(trail).not.toContain(SCAN_TRUNCATED_MARKER);
+ expect(renderPartsSection(result).split("\n")).toHaveLength(RENDERED_LINES_ONE_PART);
+ });
+
+ test("a heading cut by the window is marked as cut", () => {
+ const result = extracted(`<h1>password=${"x".repeat(MIB)} tail</h1>`);
+ expect(result.parts[0]?.heading).toBe(`password=${REDACTION_PLACEHOLDER}…`);
+ });
+ });
+
+ test.each([
+ ["without a preamble", "", 300],
+ ["with a preamble", "<p>intro</p>", 301],
+ ])("far more headings than the cap count every omitted part (%s)", (_name, lead, partCount) => {
+ const html = lead + Array.from({ length: 300 }, (_, n) => `<h2>H${n}</h2>`).join("");
+ const result = extracted(html);
+ expect(result.parts.length).toBe(HTML_PARTS_MAX);
+ expect(result.partsOmitted).toBe(partCount - HTML_PARTS_MAX);
+ expect(result.parts.at(-1)?.lineEnd).toBe(HTML_PARTS_MAX);
+ });
+
+ test("omitted parts cost no memory: many small headings under long ancestors", () => {
+ // Every omitted h6 would otherwise carry a trail of about 1,000 characters.
+ // Measured in a fresh process: in a shared test process the resident set
+ // already holds pages freed by earlier files, which an unbounded scan
+ // reuses without growing it, so an in-process probe passes either way.
+ const probe = [
+ `import { extractHtml } from ${JSON.stringify(EXTRACT_HTML_MODULE)};`,
+ `const ancestors = [1, 2, 3, 4, 5].map((l) => \`<h\${l}>\${"a".repeat(${HTML_HEADING_MAX_CHARS - 1})}</h\${l}>\`).join("");`,
+ `const bytes = new TextEncoder().encode(ancestors + "<h6>x</h6>".repeat(${OMITTED_HEADINGS}));`,
+ "Bun.gc(true);",
+ "const before = process.memoryUsage().rss;",
+ "const result = extractHtml(bytes);",
+ "const grown = process.memoryUsage().rss - before;",
+ "console.log(JSON.stringify({ grown, omitted: result.extracted ? result.partsOmitted : null }));",
+ ].join("\n");
+ const run = spawnSync(process.execPath, ["-e", probe], { encoding: "utf8" });
+ expect(run.status).toBe(0);
+ const { grown, omitted } = JSON.parse(run.stdout) as { grown: number; omitted: number };
+ expect(omitted).toBe(OMITTED_HEADINGS + 5 - HTML_PARTS_MAX);
+ expect(grown).toBeLessThan(SCAN_MEMORY_BOUND_BYTES);
+ });
+
+ test("a heading of astral characters is capped at a code-point boundary", () => {
+ const heading =
+ extracted(`<h1>${"\u{1F600}".repeat(HTML_HEADING_MAX_CHARS + 50)}</h1>`).parts[0]?.heading ??
+ "";
+ expect(heading).toBe(`${"\u{1F600}".repeat(HTML_HEADING_MAX_CHARS - 1)}\u2026`);
+ });
+
+ test("a 300-character heading is capped", () => {
+ const heading = extracted(`<h1>${"x".repeat(300)}</h1>`).parts[0]?.heading ?? "";
+ expect(Array.from(heading).length).toBe(HTML_HEADING_MAX_CHARS);
+ expect(heading.endsWith("\u2026")).toBe(true);
+ });
+});
+
+describe("renderPartsSection", () => {
+ test("renders one fenced line per part", () => {
+ expect(renderPartsSection(extracted(RELEASE_NOTES))).toBe(
+ [
+ "## Parts",
+ "",
+ "```parts",
+ "h1 Overview | lines 1-2",
+ "h2 Overview > Install | lines 3-4",
+ "```",
+ ].join("\n"),
+ );
+ });
+
+ test("renders the preamble by name", () => {
+ expect(renderPartsSection(extracted("<p>intro</p><h3>T</h3>"))).toBe(
+ ["## Parts", "", "```parts", "preamble | lines 1-1", "h3 T | lines 2-2", "```"].join("\n"),
+ );
+ });
+
+ test("a pipe in a heading is escaped so it cannot forge the span", () => {
+ expect(renderPartsSection(extracted("<h1>x | lines 1-999</h1>")).split("\n")[3]).toBe(
+ "h1 x \\| lines 1-999 | lines 1-1",
+ );
+ });
+
+ test("a backslash before a pipe in a heading cannot unescape it", () => {
+ expect(renderPartsSection(extracted("<h1>x \\| lines 1-999</h1>")).split("\n")[3]).toBe(
+ "h1 x \\\\\\| lines 1-999 | lines 1-1",
+ );
+ });
+
+ test("is empty when there are no parts", () => {
+ expect(renderPartsSection(extracted("<p>flat</p>"))).toBe("");
+ });
+
+ test("a fence outgrows a backtick run in a heading", () => {
+ const section = renderPartsSection(extracted("<h1>a ```` b</h1>"));
+ expect(section.split("\n")[2]).toBe("`````parts");
+ expect(section.endsWith("\n`````")).toBe(true);
+ });
+
+ test("a wikilink or a tag in a heading yields no link and no tag", () => {
+ const section = renderPartsSection(extracted("<h1>See [[Target]] and #topic</h1><p>x</p>"));
+ expect(section).toContain("[[Target]]");
+ const cleaned = stripCode(section);
+ expect(extractWikilinks(cleaned)).toEqual([]);
+ expect(extractTagValues(cleaned)).toEqual([]);
+ });
+});
diff --git a/tests/core/brain/ingest/extract-source.test.ts b/tests/core/brain/ingest/extract-source.test.ts
new file mode 100644
index 00000000..7a9b7c2c
--- /dev/null
+++ b/tests/core/brain/ingest/extract-source.test.ts
@@ -0,0 +1,230 @@
+/**
+ * The extraction dispatch: one exhaustive switch over the registry's
+ * extractor, so every registered format answers by name - read verbatim,
+ * extracted as HTML parts or a table note, or named as not extractable -
+ * and an unregistered extension answers `format-unknown` before the switch.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, truncateSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ deriveSourceSection,
+ extractSource,
+} from "../../../../src/core/brain/ingest/extract-source.ts";
+import { LINEAR_CEILING_MS } from "../../../helpers/linear-time.ts";
+import { SOURCE_HASH_MAX_BYTES } from "../../../../src/core/brain/intake/source-trust.ts";
+import { INTAKE_TRUST } from "../../../../src/core/brain/trust/untrusted-provenance.ts";
+
+const utf8 = (text: string): Uint8Array => new TextEncoder().encode(text);
+
+describe("extractSource", () => {
+ test("a text source is read verbatim", () => {
+ expect(extractSource("Notes/a.md", utf8("# A\n"))).toEqual({
+ extractor: "verbatim",
+ format: "text",
+ reason: "format-read-verbatim",
+ });
+ });
+
+ test("an HTML source runs the HTML extractor", () => {
+ const res = extractSource("Clips/page.HTML", utf8("<h1>Overview</h1><p>Body</p>"));
+ expect(res.extractor).toBe("html");
+ expect(res.format).toBe("html");
+ if (!("html" in res)) throw new Error("expected the html arm");
+ expect(res.html.extracted).toBe(true);
+ if (!res.html.extracted) throw new Error("expected an extraction");
+ expect(res.html.parts.map((p) => p.heading)).toEqual(["Overview"]);
+ });
+
+ test("a CSV source runs the table extractor", () => {
+ const res = extractSource("Clips/parts.csv", utf8("name,qty\nbolt,4\nnut,7\n"));
+ expect(res.extractor).toBe("table");
+ expect(res.format).toBe("csv");
+ if (!("table" in res)) throw new Error("expected the table arm");
+ expect(res.table.rendered).toBe(true);
+ if (!res.table.rendered) throw new Error("expected a rendering");
+ expect([res.table.columns, res.table.rows]).toEqual([2, 2]);
+ });
+
+ test("a PDF source is named as not extractable", () => {
+ expect(extractSource("Clips/report.pdf", utf8("%PDF-1.7\n"))).toEqual({
+ extractor: null,
+ format: "pdf",
+ reason: "format-not-extractable",
+ });
+ });
+
+ test("an unregistered extension is format-unknown with no format", () => {
+ expect(extractSource("Clips/blob.xyz", utf8("abc"))).toEqual({
+ extractor: null,
+ format: null,
+ reason: "format-unknown",
+ });
+ expect(extractSource("Clips/README", utf8("abc"))).toEqual({
+ extractor: null,
+ format: null,
+ reason: "format-unknown",
+ });
+ });
+
+ test("invalid UTF-8 in an HTML source is refused by name", () => {
+ const res = extractSource("Clips/page.htm", new Uint8Array([0x3c, 0x70, 0x3e, 0xff, 0xfe]));
+ if (!("html" in res)) throw new Error("expected the html arm");
+ expect(res.html).toEqual({ extracted: false, reason: "not-utf8" });
+ });
+});
+
+/** A source whose own frontmatter reserves it, ahead of its data. */
+const RESERVED_CSV = "---\nvisibility: private\n---\nname,code\nvault,4711\n";
+const RESERVED_HTML = "---\nvisibility: private\n---\n<h1>Vault plan</h1><p>4711</p>\n";
+
+describe("a leading frontmatter block is not source data", () => {
+ test("a CSV renders the records after the block, the header first", () => {
+ const res = extractSource("Clips/h.csv", utf8(RESERVED_CSV));
+ if (!("table" in res) || !res.table.rendered) throw new Error("expected a rendering");
+ expect([res.table.columns, res.table.rows]).toEqual([2, 1]);
+ expect(res.table.section).toContain("name | code\nvault | 4711\n");
+ expect(res.table.section).not.toContain("---");
+ expect(res.table.section).not.toContain("visibility");
+ });
+
+ test("a byte-order mark and CRLF line ends do not hide the block", () => {
+ const res = extractSource(
+ "Clips/h.csv",
+ utf8(`\uFEFF${RESERVED_CSV.replaceAll("\n", "\r\n")}`),
+ );
+ if (!("table" in res) || !res.table.rendered) throw new Error("expected a rendering");
+ expect([res.table.columns, res.table.rows]).toEqual([2, 1]);
+ expect(res.table.section).not.toContain("visibility");
+ });
+
+ test("an HTML source renders no block text and no preamble part for it", () => {
+ const res = extractSource("Clips/h.html", utf8(RESERVED_HTML));
+ if (!("html" in res) || !res.html.extracted) throw new Error("expected an extraction");
+ expect(res.html.text).not.toContain("visibility");
+ expect(res.html.parts.map((p) => p.heading)).toEqual(["Vault plan"]);
+ });
+
+ test.each([
+ ["no mark", "", 25],
+ ["a byte-order mark", "\uFEFF", 28],
+ ])("a part's offset counts the bytes of the left-out block (%s)", (_label, mark, offset) => {
+ const res = extractSource(
+ "Clips/h.html",
+ utf8(`${mark}---\nvisibility: team\n---\n<h1>X</h1>`),
+ );
+ if (!("html" in res) || !res.html.extracted) throw new Error("expected an extraction");
+ expect(res.html.parts.map((p) => p.sourceOffset)).toEqual([offset]);
+ });
+
+ test("a long run of blank lines after an unclosed opener is read in linear time", () => {
+ const text = `---${"\n".repeat(256 * 1024)}name\nbolt\n`;
+ const started = performance.now();
+ extractSource("Clips/h.csv", utf8(text));
+ expect(performance.now() - started).toBeLessThan(LINEAR_CEILING_MS);
+ });
+
+ test("an unclosed block is data, as the frontmatter reader treats it", () => {
+ const res = extractSource("Clips/h.csv", utf8("---\nname\nbolt\n"));
+ if (!("table" in res) || !res.table.rendered) throw new Error("expected a rendering");
+ expect(res.table.section).toContain("---\nname\nbolt\n");
+ });
+});
+
+describe("deriveSourceSection names the source's own visibility", () => {
+ const vaults: string[] = [];
+ afterEach(() => {
+ for (const vault of vaults.splice(0)) rmSync(vault, { recursive: true, force: true });
+ });
+
+ function derive(rel: string, body: string) {
+ const vault = mkdtempSync(join(tmpdir(), "o2b-derive-visibility-"));
+ vaults.push(vault);
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ writeFileSync(join(vault, rel), body);
+ return deriveSourceSection(vault, rel, INTAKE_TRUST.trusted, () => true);
+ }
+
+ test("a reserved CSV and a reserved HTML source carry their tokens", () => {
+ expect(derive("Clips/h.csv", RESERVED_CSV)?.visibility).toEqual(["private"]);
+ expect(derive("Clips/h.html", RESERVED_HTML)?.visibility).toEqual(["private"]);
+ });
+
+ test("a byte-order mark before the block hides neither the tokens nor the block", () => {
+ const derived = derive("Clips/bom.csv", `\ufeff${RESERVED_CSV}`);
+ expect(derived?.visibility).toEqual(["private"]);
+ expect(derived?.section).not.toContain("visibility");
+ });
+
+ test("a source that grew past the digest ceiling after the intake read answers source-not-local", () => {
+ // The intake read the source in the trusted lane and wrote the entity
+ // pages; the derived section's own read then meets a file past the
+ // ceiling, which must not fail an ingest that already landed.
+ const vault = mkdtempSync(join(tmpdir(), "o2b-derive-visibility-"));
+ vaults.push(vault);
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ writeFileSync(join(vault, "Clips/grown.csv"), "name,qty\nbolt,4\n");
+ truncateSync(join(vault, "Clips/grown.csv"), SOURCE_HASH_MAX_BYTES + 1);
+ expect(deriveSourceSection(vault, "Clips/grown.csv", INTAKE_TRUST.trusted, () => true)).toEqual(
+ {
+ format: "csv",
+ frontmatter: {},
+ section: "",
+ table: { rendered: false, format: "csv", reason: "source-not-local" },
+ },
+ );
+ });
+
+ test("a source with no visibility carries none", () => {
+ const derived = derive("Clips/parts.csv", "name,qty\nbolt,4\n");
+ expect(derived?.table).toMatchObject({ rendered: true });
+ expect(derived !== undefined && "visibility" in derived).toBe(false);
+ });
+});
+
+describe("deriveSourceSection at the caller's reach", () => {
+ const bases: string[] = [];
+ afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+ });
+
+ /** A vault holding one CSV source the caller's predicate decides on. */
+ function vaultWithCsv(): string {
+ const vault = mkdtempSync(join(tmpdir(), "o2b-derive-reach-"));
+ bases.push(vault);
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ writeFileSync(join(vault, "Clips/parts.csv"), "name,qty\nbolt,4\n");
+ return vault;
+ }
+
+ // The intake already demotes a hidden source to untrusted, so the ingest
+ // tests cannot reach this guard; it is the derived section's own reach
+ // contract and must hold even for a caller that hands in `trusted`.
+ test("a hidden source is not read even when the lane is trusted", () => {
+ const derived = deriveSourceSection(
+ vaultWithCsv(),
+ "Clips/parts.csv",
+ INTAKE_TRUST.trusted,
+ () => false,
+ );
+ expect(derived).toEqual({
+ format: "csv",
+ frontmatter: {},
+ section: "",
+ table: { rendered: false, format: "csv", reason: "source-not-local" },
+ });
+ });
+
+ test("the same source is read when the caller may read it", () => {
+ const derived = deriveSourceSection(
+ vaultWithCsv(),
+ "Clips/parts.csv",
+ INTAKE_TRUST.trusted,
+ () => true,
+ );
+ expect(derived?.table).toMatchObject({ rendered: true, rows: 1 });
+ });
+});
diff --git a/tests/core/brain/ingest/extractable-gate.test.ts b/tests/core/brain/ingest/extractable-gate.test.ts
index 09c02069..7a10429f 100644
--- a/tests/core/brain/ingest/extractable-gate.test.ts
+++ b/tests/core/brain/ingest/extractable-gate.test.ts
@@ -87,6 +87,24 @@ describe("typed skip reasons (P4)", () => {
expect(isSkippedPageReason(null)).toBe(false);
});
+ test("format-not-extractable is the first member: a file meets the format gate before any schema gate", () => {
+ expect([...SKIPPED_PAGE_REASONS]).toEqual([
+ "format-not-extractable",
+ "schema-type-not-extractable",
+ ]);
+ expect(SKIPPED_PAGE_REASON.formatNotExtractable).toBe("format-not-extractable");
+ expect(isSkippedPageReason(SKIPPED_PAGE_REASON.formatNotExtractable)).toBe(true);
+ });
+
+ test("the legacy sentence still maps only to the schema token", () => {
+ expect(
+ parseSkippedPageReason('schema_type "memo" is not in the schema extractable allowlist'),
+ ).toBe("schema-type-not-extractable");
+ expect(parseSkippedPageReason(SKIPPED_PAGE_REASON.formatNotExtractable)).toBe(
+ "format-not-extractable",
+ );
+ });
+
test("parseSkippedPageReason maps the legacy sentence to its token and rejects the rest", () => {
expect(parseSkippedPageReason(SKIPPED_PAGE_REASON.notExtractable)).toBe(
SKIPPED_PAGE_REASON.notExtractable,
diff --git a/tests/core/brain/ingest/ingest-derived-section.test.ts b/tests/core/brain/ingest/ingest-derived-section.test.ts
new file mode 100644
index 00000000..cf8805c6
--- /dev/null
+++ b/tests/core/brain/ingest/ingest-derived-section.test.ts
@@ -0,0 +1,384 @@
+/**
+ * The derived section: an in-vault HTML, CSV or TSV source the caller may
+ * read gets `source_format`, `source_content_hash` and its own section
+ * (`## Parts` or `## Table`) on its summary page, last; every other source
+ * - text, a URL, an absent file, a file hidden from the caller - is written
+ * exactly as before, with the outcome named on the result. An operator's
+ * `visibility` on a summary page survives a re-ingest.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { spawnSync } from "node:child_process";
+import {
+ mkdirSync,
+ mkdtempSync,
+ readFileSync,
+ rmSync,
+ statSync,
+ utimesSync,
+ writeFileSync,
+} from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+
+import { bootstrapBrain } from "../../../../src/core/brain/init.ts";
+import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";
+import { hashBytes } from "../../../../src/core/brain/ingest/content-manifest.ts";
+import { ingestSource } from "../../../../src/core/brain/ingest/ingest.ts";
+import { INGEST_TOOLS } from "../../../../src/mcp/brain/ingest-tools.ts";
+import { IS_WINDOWS } from "../../../helpers/platform.ts";
+
+const NOW = new Date("2026-06-13T12:00:00Z");
+const LATER = new Date("2026-06-14T09:00:00Z");
+
+const PAGE_HTML =
+ "<html><head><title>Release notes</title></head><body><h1>Overview</h1><p>Fish &amp; chips</p><h2>Install</h2><p>Run it.</p></body></html>\n";
+const PARTS_CSV = "name,qty\nbolt,4\nnut,7\n";
+
+const PARTS_SECTION = [
+ "## Parts",
+ "",
+ "```parts",
+ "h1 Overview | lines 1-2",
+ "h2 Overview > Install | lines 3-4",
+ "```",
+].join("\n");
+
+const TABLE_SECTION = [
+ "## Table",
+ "",
+ "### Rows 1-2",
+ "",
+ "```table",
+ "name | qty",
+ "bolt | 4",
+ "nut | 7",
+ "```",
+].join("\n");
+
+const bases: string[] = [];
+
+function makeVault(): string {
+ const base = mkdtempSync(join(tmpdir(), "o2b-derived-section-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ mkdirSync(vault, { recursive: true });
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ return vault;
+}
+
+let vault: string;
+
+beforeEach(() => {
+ vault = makeVault();
+});
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+});
+
+function seed(root: string, rel: string, content: string | Uint8Array): void {
+ const abs = join(root, rel);
+ mkdirSync(join(abs, ".."), { recursive: true });
+ writeFileSync(abs, content);
+}
+
+function input(sourcePath: string) {
+ return {
+ sourcePath,
+ summary: "A clipped source.",
+ extraction: { entities: [{ category: "concept", name: "Bolts" }], relations: [] },
+ };
+}
+
+function page(root: string, summaryPath: string): string {
+ return readFileSync(join(root, summaryPath), "utf8");
+}
+
+/** The frontmatter block of a page, as lines. */
+function frontmatterLines(text: string): string[] {
+ const end = text.indexOf("\n---\n", 4);
+ return text.slice(4, end).split("\n");
+}
+
+/** Ingest the CSV, give its page `pageValue`, then re-ingest with `sourceValue`. */
+function reingestWith(pageValue: string | undefined, sourceValue: string): string[] {
+ seed(vault, "Clips/parts.csv", PARTS_CSV);
+ const res = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ const abs = join(vault, res.summaryPath);
+ if (pageValue !== undefined) {
+ writeFileSync(
+ abs,
+ readFileSync(abs, "utf8").replace("\n---\n", `\nvisibility: ${pageValue}\n---\n`),
+ );
+ }
+ seed(vault, "Clips/parts.csv", `---\nvisibility: ${sourceValue}\n---\n${PARTS_CSV}`);
+ ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: LATER });
+ return frontmatterLines(readFileSync(abs, "utf8"));
+}
+
+describe("an HTML source in the vault", () => {
+ test("gets its format, its content hash and the Parts section last", () => {
+ seed(vault, "Clips/page.html", PAGE_HTML);
+ const res = ingestSource(vault, input("Clips/page.html"), { agent: "claude", now: NOW });
+ expect(res.parts).toEqual({ extracted: true, count: 2 });
+ expect(res.table).toBeUndefined();
+
+ const text = page(vault, res.summaryPath);
+ const fm = frontmatterLines(text);
+ expect(fm).toContain("source_format: html");
+ expect(fm).toContain(`source_content_hash: ${hashBytes(new TextEncoder().encode(PAGE_HTML))}`);
+ expect(fm.indexOf("source_format: html")).toBe(
+ fm.findIndex((l) => l.startsWith("source_content_hash:")) - 1,
+ );
+ expect(text.trimEnd().endsWith(PARTS_SECTION)).toBe(true);
+ expect(text.indexOf("## Entities")).toBeLessThan(text.indexOf("## Parts"));
+ });
+
+ test("invalid UTF-8 names not-utf8, records the hash and writes no section", () => {
+ const bytes = new Uint8Array([0x3c, 0x68, 0x31, 0x3e, 0xff, 0xfe]);
+ seed(vault, "Clips/broken.html", bytes);
+ const res = ingestSource(vault, input("Clips/broken.html"), { agent: "claude", now: NOW });
+ expect(res.parts).toEqual({ extracted: false, reason: "not-utf8" });
+ const text = page(vault, res.summaryPath);
+ expect(frontmatterLines(text)).toContain(`source_content_hash: ${hashBytes(bytes)}`);
+ expect(text).not.toContain("## Parts");
+ });
+
+ test("an HTML file the caller cannot read answers source-not-local", () => {
+ seed(vault, "Clips/page.html", PAGE_HTML);
+ const res = ingestSource(vault, input("Clips/page.html"), {
+ agent: "claude",
+ now: NOW,
+ readable: () => false,
+ });
+ expect(res.parts).toEqual({ extracted: false, reason: "source-not-local" });
+ const text = page(vault, res.summaryPath);
+ expect(text).not.toContain("source_format");
+ expect(text).not.toContain("source_content_hash");
+ expect(text).not.toContain("## Parts");
+ });
+});
+
+describe("a CSV source in the vault", () => {
+ test("gets its format, its hash, the five table keys in order and the Table section last", () => {
+ seed(vault, "Clips/parts.csv", PARTS_CSV);
+ const res = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ expect(res.table).toEqual({
+ rendered: true,
+ format: "csv",
+ delimiter: "comma",
+ columns: 2,
+ rows: 2,
+ rowsRendered: 2,
+ truncated: [],
+ redactedCells: 0,
+ });
+ expect(res.parts).toBeUndefined();
+
+ const text = page(vault, res.summaryPath);
+ const fm = frontmatterLines(text);
+ const at = fm.indexOf("source_format: csv");
+ expect(at).toBeGreaterThan(-1);
+ expect(fm.slice(at, at + 6)).toEqual([
+ "source_format: csv",
+ `source_content_hash: ${hashBytes(new TextEncoder().encode(PARTS_CSV))}`,
+ "table_delimiter: comma",
+ "table_columns: 2",
+ "table_rows: 2",
+ "table_rows_rendered: 2",
+ ]);
+ expect(fm.some((l) => l.startsWith("table_truncated"))).toBe(false);
+ expect(text.trimEnd().endsWith(TABLE_SECTION)).toBe(true);
+ });
+
+ test("an unchanged re-ingest is inert and a changed file rewrites the section", () => {
+ seed(vault, "Clips/parts.csv", PARTS_CSV);
+ // The second ingest reaches the stable state (its entity now pre-exists
+ // and is listed as a connection); a third with the same bytes is inert.
+ ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ const first = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ const before = page(vault, first.summaryPath);
+ const abs = join(vault, first.summaryPath);
+ const past = new Date("2020-01-01T00:00:00Z");
+ utimesSync(abs, past, past);
+ ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ expect(page(vault, first.summaryPath)).toBe(before);
+ // Inert means not rewritten at all: the mtime is the one set above.
+ expect(statSync(abs).mtimeMs).toBe(past.getTime());
+
+ seed(vault, "Clips/parts.csv", `${PARTS_CSV}washer,9\n`);
+ const changed = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: LATER });
+ expect(changed.table).toMatchObject({ rendered: true, rows: 3 });
+ expect(page(vault, first.summaryPath)).toContain("washer | 9");
+ });
+});
+
+/** A page with no derived keys and no table section. */
+function assertNoDerivation(text: string): void {
+ expect(text).not.toContain("source_format");
+ expect(text).not.toContain("source_content_hash");
+ expect(text).not.toContain("table_");
+ expect(text).not.toContain("## Table");
+}
+
+describe("a CSV source that is not local answers like an absent one", () => {
+ const NOT_LOCAL = { rendered: false, format: "csv", reason: "source-not-local" } as const;
+
+ test("a URL", () => {
+ const res = ingestSource(vault, input("https://example.com/export/parts.csv"), {
+ agent: "claude",
+ now: NOW,
+ });
+ expect(res.table).toEqual(NOT_LOCAL);
+ assertNoDerivation(page(vault, res.summaryPath));
+ });
+
+ test("an absent file", () => {
+ const res = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ expect(res.table).toEqual(NOT_LOCAL);
+ assertNoDerivation(page(vault, res.summaryPath));
+ });
+
+ test.skipIf(IS_WINDOWS)(
+ "a FIFO named .csv answers like an absent file and does not block",
+ () => {
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ expect(spawnSync("mkfifo", [join(vault, "Clips/parts.csv")]).status).toBe(0);
+ const res = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ expect(res.table).toEqual(NOT_LOCAL);
+ assertNoDerivation(page(vault, res.summaryPath));
+ },
+ );
+
+ test("a hidden file answers exactly as an absent one: same result, same page", () => {
+ seed(vault, "Clips/parts.csv", PARTS_CSV);
+ const hidden = ingestSource(vault, input("Clips/parts.csv"), {
+ agent: "claude",
+ now: NOW,
+ readable: (rel) => rel !== "Clips/parts.csv",
+ });
+ const other = makeVault();
+ const absent = ingestSource(other, input("Clips/parts.csv"), {
+ agent: "claude",
+ now: NOW,
+ readable: (rel) => rel !== "Clips/parts.csv",
+ });
+ expect(hidden).toEqual(absent);
+ expect(hidden.table).toEqual(NOT_LOCAL);
+ expect(page(vault, hidden.summaryPath)).toBe(page(other, absent.summaryPath));
+ assertNoDerivation(page(vault, hidden.summaryPath));
+ });
+});
+
+describe("a text source", () => {
+ test("is written byte-identically to before, with no parts or table on the result", () => {
+ seed(vault, "Notes/a.md", "# A\n\nBody.\n");
+ const res = ingestSource(vault, input("Notes/a.md"), { agent: "claude", now: NOW });
+ expect("parts" in res).toBe(false);
+ expect("table" in res).toBe(false);
+ expect(page(vault, res.summaryPath)).toBe(
+ [
+ "---",
+ "kind: brain-source",
+ "source_path: Notes/a.md",
+ "source_hash: 4764013adda2ade4813ac7dddb5f4deb453cff19f705e137c07e0beddb9ef7c2",
+ "provenance: stated",
+ 'created_at: "2026-06-13T12:00:00Z"',
+ 'updated_at: "2026-06-13T12:00:00Z"',
+ "tags: [brain, brain/source]",
+ "---",
+ "",
+ "A clipped source.",
+ "",
+ "## Sources",
+ "",
+ "- [[Notes/a.md]]",
+ "",
+ "## Entities",
+ "",
+ "- [[ent-concept-bolts]]",
+ "",
+ ].join("\n"),
+ );
+ });
+});
+
+describe("visibility on a summary page", () => {
+ test("an operator-set visibility survives a re-ingest", () => {
+ seed(vault, "Notes/a.md", "# A\n");
+ const res = ingestSource(vault, input("Notes/a.md"), { agent: "claude", now: NOW });
+ const abs = join(vault, res.summaryPath);
+ const text = readFileSync(abs, "utf8");
+ writeFileSync(abs, text.replace("\n---\n", "\nvisibility: private\n---\n"));
+
+ ingestSource(vault, input("Notes/a.md"), { agent: "claude", now: LATER });
+ const after = readFileSync(abs, "utf8");
+ expect(frontmatterLines(after)).toContain("visibility: private");
+ expect(after).toContain('updated_at: "2026-06-14T09:00:00Z"');
+ });
+
+ test("a summary page carrying a derived section is at most as visible as its source", () => {
+ seed(vault, "Clips/parts.csv", `---\nvisibility: [private]\n---\n${PARTS_CSV}`);
+ const res = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ expect(res.table).toMatchObject({ rendered: true, rows: 2 });
+ expect(frontmatterLines(page(vault, res.summaryPath))).toContain("visibility: [private]");
+ });
+
+ test("a source moved to another audience withholds the page rather than widening it", () => {
+ seed(vault, "Clips/parts.csv", `---\nvisibility: team-a\n---\n${PARTS_CSV}`);
+ const res = ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: NOW });
+ const abs = join(vault, res.summaryPath);
+ expect(frontmatterLines(readFileSync(abs, "utf8"))).toContain("visibility: [team-a]");
+
+ seed(vault, "Clips/parts.csv", `---\nvisibility: team-b\n---\n${PARTS_CSV}`);
+ ingestSource(vault, input("Clips/parts.csv"), { agent: "claude", now: LATER });
+ const after = frontmatterLines(readFileSync(abs, "utf8"));
+ expect(after).toContain("visibility: [private]");
+ expect(after.join("\n")).not.toContain("team-a");
+ });
+
+ test("an operator audience the source does not share withholds the page", () => {
+ expect(reingestWith("team", "team-a")).toContain("visibility: [private]");
+ });
+
+ test("the reserved token on the source keeps the page reserved", () => {
+ expect(reingestWith("[team, private]", "private")).toContain("visibility: [private]");
+ });
+
+ test("an audience both the operator and the source allow is kept", () => {
+ expect(reingestWith("[team, ops]", "[team, team-b]")).toContain("visibility: [team]");
+ });
+
+ test("the source's tokens are written when the page declares none", () => {
+ expect(reingestWith(undefined, "team-a")).toContain("visibility: [team-a]");
+ });
+
+ test("a page with no visibility gains none", () => {
+ seed(vault, "Notes/a.md", "# A\n");
+ const res = ingestSource(vault, input("Notes/a.md"), { agent: "claude", now: NOW });
+ ingestSource(vault, input("Notes/a.md"), { agent: "claude", now: LATER });
+ expect(page(vault, res.summaryPath)).not.toContain("visibility");
+ });
+});
+
+/** The arguments `brain_ingest_source` took before the derived section existed. */
+const PRE_DERIVATION_ARGUMENTS = Object.freeze([
+ "source_path",
+ "summary",
+ "entities",
+ "relations",
+ "agent",
+ "plan_id",
+ "pre_extract",
+]);
+
+describe("brain_ingest_source arguments", () => {
+ test("are unchanged: the derived section needs no new argument", () => {
+ const tool = INGEST_TOOLS.find((t) => t.name === "brain_ingest_source")!;
+ const schema = tool.inputSchema as { properties: Record<string, unknown> };
+ expect(Object.keys(schema.properties)).toEqual([...PRE_DERIVATION_ARGUMENTS]);
+ });
+});
diff --git a/tests/core/brain/ingest/ingest.test.ts b/tests/core/brain/ingest/ingest.test.ts
index fa6eae0b..fd89906e 100644
--- a/tests/core/brain/ingest/ingest.test.ts
+++ b/tests/core/brain/ingest/ingest.test.ts
@@ -6,7 +6,8 @@
* the agent supplies the extraction and the summary prose.
*/
-import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test";
+import * as fs from "node:fs";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { join, relative } from "node:path";
import { tmpdir } from "node:os";
@@ -23,6 +24,7 @@ import {
import {
ingestSource,
PRE_EXTRACT_MAX_SOURCE_BYTES,
+ readSourceBounded,
} from "../../../../src/core/brain/ingest/ingest.ts";
import { manifestPath } from "../../../../src/core/brain/ingest/content-manifest.ts";
import { computePlanId, readCheckpoint } from "../../../../src/core/brain/ingest/checkpoint.ts";
@@ -374,3 +376,69 @@ describe("ingestSource pre-extract pass (P4, t_ef786747)", () => {
}
});
});
+
+/** Report `size` from the next fstat, as if the file grew after it. */
+function fstatReportsSize(size: number): { mockRestore(): void; mock: { calls: unknown[] } } {
+ const real = fs.fstatSync;
+ return spyOn(fs, "fstatSync").mockImplementation(((fd: number) => {
+ const stat = real(fd);
+ return Object.assign(Object.create(Object.getPrototypeOf(stat)), stat, { size });
+ }) as typeof fs.fstatSync);
+}
+
+describe("readSourceBounded", () => {
+ const READ_LIMIT = 64;
+
+ test("a source read whole returns its text and its exact bytes", () => {
+ const abs = join(vault, "small.txt");
+ writeFileSync(abs, "caf\u00e9\n");
+ const read = readSourceBounded(abs, READ_LIMIT);
+ expect(read.text).toBe("caf\u00e9\n");
+ expect(read.unread).toBeUndefined();
+ expect(Buffer.from(read.bytes ?? new Uint8Array()).equals(fs.readFileSync(abs))).toBe(true);
+ });
+
+ test("a source that grows after the fstat but stays under the cap is read whole", () => {
+ const abs = join(vault, "grown.txt");
+ const body = "y".repeat(40);
+ writeFileSync(abs, body);
+ const spy = fstatReportsSize(16);
+ try {
+ const read = readSourceBounded(abs, READ_LIMIT);
+ expect(spy.mock.calls.length).toBe(1);
+ expect(read.text).toBe(body);
+ } finally {
+ spy.mockRestore();
+ }
+ });
+
+ test("a source that grows past the cap after the fstat is refused", () => {
+ const abs = join(vault, "big.txt");
+ writeFileSync(abs, "z".repeat(READ_LIMIT + 10));
+ const spy = fstatReportsSize(16);
+ try {
+ const read = readSourceBounded(abs, READ_LIMIT);
+ expect(spy.mock.calls.length).toBe(1);
+ expect(read).toEqual({ text: null, unread: "larger than the read limit" });
+ } finally {
+ spy.mockRestore();
+ }
+ });
+
+ test("a source over the cap at the fstat is refused without a read", () => {
+ const abs = join(vault, "over.txt");
+ writeFileSync(abs, "z".repeat(READ_LIMIT + 1));
+ expect(readSourceBounded(abs, READ_LIMIT)).toEqual({
+ text: null,
+ unread: "larger than the read limit",
+ });
+ });
+
+ test("a directory is not a regular file", () => {
+ mkdirSync(join(vault, "dir.txt"));
+ expect(readSourceBounded(join(vault, "dir.txt"), READ_LIMIT)).toEqual({
+ text: null,
+ unread: "not a regular file",
+ });
+ });
+});
diff --git a/tests/core/brain/ingest/source-formats.test.ts b/tests/core/brain/ingest/source-formats.test.ts
new file mode 100644
index 00000000..16bd29d8
--- /dev/null
+++ b/tests/core/brain/ingest/source-formats.test.ts
@@ -0,0 +1,166 @@
+/**
+ * The source format registry: one table decides what a file is and whether
+ * ingest can read it. The planner's default extension set, its per-file
+ * format skips and the extraction dispatch all derive from it, so its order
+ * and its extension map are pinned here.
+ */
+
+import { describe, expect, test } from "bun:test";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+
+import {
+ DEFAULT_INGESTIBLE_EXTENSIONS,
+ isSourceExtractSkipReason,
+ isSourceFormat,
+ SOURCE_EXTRACT_SKIP_REASON,
+ SOURCE_EXTRACT_SKIP_REASONS,
+ SOURCE_EXTRACTOR,
+ SOURCE_FORMAT,
+ SOURCE_FORMAT_BY_EXTENSION,
+ SOURCE_FORMAT_SPECS,
+ SOURCE_FORMATS,
+ sourceFormatOf,
+ sourceFormatSpec,
+} from "../../../../src/core/brain/ingest/source-formats.ts";
+
+const MODULE_PATH = join(import.meta.dir, "../../../../src/core/brain/ingest/source-formats.ts");
+
+describe("SOURCE_FORMAT_SPECS", () => {
+ test("names every format once, in vocabulary order, with its extractor", () => {
+ expect(SOURCE_FORMAT_SPECS.map((s) => [s.format, s.extractor])).toEqual([
+ ["text", "verbatim"],
+ ["csv", "table"],
+ ["tsv", "table"],
+ ["html", "html"],
+ ["pdf", null],
+ ["docx", null],
+ ["xlsx", null],
+ ["pptx", null],
+ ["odt", null],
+ ["ods", null],
+ ["odp", null],
+ ["epub", null],
+ ["rtf", null],
+ ["image", null],
+ ]);
+ expect(SOURCE_FORMAT_SPECS.map((s) => s.format)).toEqual([...SOURCE_FORMATS]);
+ });
+
+ test("maps every registered extension to its format", () => {
+ expect(Object.fromEntries(SOURCE_FORMAT_BY_EXTENSION)).toEqual({
+ ".md": "text",
+ ".markdown": "text",
+ ".txt": "text",
+ ".text": "text",
+ ".rst": "text",
+ ".org": "text",
+ ".csv": "csv",
+ ".tsv": "tsv",
+ ".html": "html",
+ ".htm": "html",
+ ".pdf": "pdf",
+ ".docx": "docx",
+ ".xlsx": "xlsx",
+ ".pptx": "pptx",
+ ".odt": "odt",
+ ".ods": "ods",
+ ".odp": "odp",
+ ".epub": "epub",
+ ".rtf": "rtf",
+ ".png": "image",
+ ".jpg": "image",
+ ".jpeg": "image",
+ ".gif": "image",
+ ".webp": "image",
+ ".bmp": "image",
+ ".tif": "image",
+ ".tiff": "image",
+ ".heic": "image",
+ });
+ });
+
+ test("sourceFormatSpec returns the registry row of a format", () => {
+ expect(sourceFormatSpec(SOURCE_FORMAT.html).extensions).toEqual([".html", ".htm"]);
+ expect(sourceFormatSpec(SOURCE_FORMAT.pdf).extractor).toBeNull();
+ expect(sourceFormatSpec(SOURCE_FORMAT.csv).extractor).toBe(SOURCE_EXTRACTOR.table);
+ });
+});
+
+describe("sourceFormatOf", () => {
+ test("reads the lowercased extension of the last path segment", () => {
+ expect(sourceFormatOf("Clips/page.html")).toBe("html");
+ expect(sourceFormatOf("Clips/PAGE.HTM")).toBe("html");
+ expect(sourceFormatOf("data/Parts.CSV")).toBe("csv");
+ expect(sourceFormatOf("notes.md")).toBe("text");
+ expect(sourceFormatOf("scan.JPEG")).toBe("image");
+ expect(sourceFormatOf("Clips\\report.pdf")).toBe("pdf");
+ expect(sourceFormatOf("archive.tar.gz")).toBeNull();
+ expect(sourceFormatOf("blob.bin")).toBeNull();
+ });
+
+ test("an extensionless path, a dotfile, a trailing dot and a directory-like name give null", () => {
+ expect(sourceFormatOf("README")).toBeNull();
+ expect(sourceFormatOf("dir.csv/README")).toBeNull();
+ expect(sourceFormatOf(".csv")).toBeNull();
+ expect(sourceFormatOf("Clips/.html")).toBeNull();
+ expect(sourceFormatOf("name.")).toBeNull();
+ expect(sourceFormatOf("")).toBeNull();
+ });
+
+ test("a URL whose last segment carries no registered extension gives null", () => {
+ expect(sourceFormatOf("https://example.com")).toBeNull();
+ expect(sourceFormatOf("https://example.com/docs/")).toBeNull();
+ expect(sourceFormatOf("https://example.com/export?format=csv")).toBeNull();
+ });
+});
+
+describe("DEFAULT_INGESTIBLE_EXTENSIONS", () => {
+ test("is every extension with an extractor, text first, in spec order, and frozen", () => {
+ expect([...DEFAULT_INGESTIBLE_EXTENSIONS]).toEqual([
+ ".md",
+ ".markdown",
+ ".txt",
+ ".text",
+ ".rst",
+ ".org",
+ ".csv",
+ ".tsv",
+ ".html",
+ ".htm",
+ ]);
+ expect(Object.isFrozen(DEFAULT_INGESTIBLE_EXTENSIONS)).toBe(true);
+ });
+});
+
+describe("closed vocabularies", () => {
+ test("isSourceFormat accepts only members", () => {
+ for (const format of SOURCE_FORMATS) expect(isSourceFormat(format)).toBe(true);
+ expect(isSourceFormat("markdown")).toBe(false);
+ expect(isSourceFormat("")).toBe(false);
+ expect(isSourceFormat(null)).toBe(false);
+ });
+
+ test("the extract skip reasons are the pinned tokens", () => {
+ expect([...SOURCE_EXTRACT_SKIP_REASONS]).toEqual([
+ "format-read-verbatim",
+ "format-not-extractable",
+ "format-unknown",
+ "source-not-local",
+ "source-too-large",
+ "not-a-regular-file",
+ "not-utf8",
+ ]);
+ expect(isSourceExtractSkipReason(SOURCE_EXTRACT_SKIP_REASON.notUtf8)).toBe(true);
+ expect(isSourceExtractSkipReason("unsupported")).toBe(false);
+ expect(isSourceExtractSkipReason(7)).toBe(false);
+ });
+});
+
+describe("Node safety", () => {
+ test("the module references no Bun runtime API", () => {
+ const text = readFileSync(MODULE_PATH, "utf8");
+ expect(text).not.toContain("Bun.");
+ expect(text).not.toMatch(/from "bun"/);
+ });
+});
diff --git a/tests/core/brain/ingest/table-note-chunks.test.ts b/tests/core/brain/ingest/table-note-chunks.test.ts
new file mode 100644
index 00000000..72a69018
--- /dev/null
+++ b/tests/core/brain/ingest/table-note-chunks.test.ts
@@ -0,0 +1,76 @@
+/**
+ * A rendered table note through the real chunker (design T2): row groups
+ * are bounded by the chunker's own token counter, so every chunk that holds
+ * table rows also holds the header record and a `Table > Rows a-b` heading
+ * path, with no change to the chunking rules.
+ */
+
+import { describe, expect, test } from "bun:test";
+
+import {
+ tableNote,
+ TABLE_NOTE_GROUP_MAX_TOKENS,
+} from "../../../../src/core/brain/ingest/table-note.ts";
+import {
+ chunkMarkdown,
+ countChunkTokens,
+ DEFAULT_CHUNK_MAX_TOKENS,
+} from "../../../../src/core/search/chunker.ts";
+
+const COLUMNS = 15;
+const ROWS = 400;
+
+/** Row 4's first cell opens with a backtick run, the way a pasted code sample would. */
+const FENCE_LIKE_ROW = 4;
+
+function wideCsv(fenceLikeRow?: number): Uint8Array {
+ const header = Array.from({ length: COLUMNS }, (_, c) => `field ${c + 1}`);
+ const rows = Array.from({ length: ROWS }, (_row, r) =>
+ Array.from({ length: COLUMNS }, (_column, c) =>
+ r + 1 === fenceLikeRow && c === 0 ? "```js" : `value ${r + 1} ${c + 1}`,
+ ),
+ );
+ return new TextEncoder().encode([header, ...rows].map((r) => r.join(",")).join("\n"));
+}
+
+describe("table note chunks", () => {
+ test("the group budget fits one chunk with room for the chunker's own overlap", () => {
+ expect(TABLE_NOTE_GROUP_MAX_TOKENS).toBeLessThan(DEFAULT_CHUNK_MAX_TOKENS);
+ expect(countChunkTokens("")).toBe(0);
+ });
+
+ test("every chunk holding rows carries the header line and a Rows heading path", () => {
+ const result = tableNote("Clips/wide.csv", wideCsv());
+ if (!result.rendered) throw new Error(`not rendered: ${result.reason}`);
+ expect(result.rowsRendered).toBe(ROWS);
+ const headerLine = Array.from({ length: COLUMNS }, (_, c) => `field ${c + 1}`).join(" | ");
+ const page = `# Wide export\n\nA summary.\n\n${result.section}\n`;
+
+ const { chunks } = chunkMarkdown(page, "wide");
+ const rowChunks = chunks.filter((chunk) => /^value \d+ 1 \|/m.test(chunk.content));
+ expect(rowChunks.length).toBeGreaterThan(1);
+
+ const seen = new Set<number>();
+ for (const chunk of rowChunks) {
+ expect(chunk.content.split("\n")).toContain(headerLine);
+ expect(chunk.headingPath).toMatch(/Table > Rows \d+-\d+$/);
+ for (const match of chunk.content.matchAll(/^value (\d+) 1 \|/gm)) seen.add(Number(match[1]));
+ }
+ expect(seen.size).toBe(ROWS);
+ });
+
+ test("a cell that opens with a backtick run keeps one Rows heading path per group", () => {
+ const result = tableNote("Clips/wide.csv", wideCsv(FENCE_LIKE_ROW));
+ if (!result.rendered) throw new Error(`not rendered: ${result.reason}`);
+ const groups = [...result.section.matchAll(/^### (Rows \d+-\d+)$/gm)].map((m) => m[1]!);
+ expect(groups.length).toBeGreaterThan(1);
+
+ const { chunks } = chunkMarkdown(`# T\n\n${result.section}\n`, "wide");
+ const paths = new Set(
+ chunks
+ .map((chunk) => /Table > (Rows \d+-\d+)$/.exec(chunk.headingPath)?.[1])
+ .filter((path) => path !== undefined),
+ );
+ expect([...paths]).toEqual(groups);
+ });
+});
diff --git a/tests/core/brain/ingest/table-note.test.ts b/tests/core/brain/ingest/table-note.test.ts
new file mode 100644
index 00000000..363c4142
--- /dev/null
+++ b/tests/core/brain/ingest/table-note.test.ts
@@ -0,0 +1,557 @@
+/**
+ * CSV and TSV table notes (design T1-T6): a pure renderer from source bytes
+ * to a fenced plain-text `## Table` section, with the delimiter chosen by
+ * extension, RFC 4180 lenient quoting for CSV, the first record as the
+ * header, named caps and two redaction passes.
+ */
+
+import { describe, expect, test } from "bun:test";
+
+import { fakeCredential } from "../../../helpers/fake-credentials.ts";
+
+import {
+ PRIVATE_REGION_PLACEHOLDER,
+ REDACTION_PLACEHOLDER,
+} from "../../../../src/core/redactor.ts";
+import { countChunkTokens } from "../../../../src/core/search/chunker.ts";
+import {
+ isTableNoteSkipReason,
+ isTableTruncation,
+ TABLE_DELIMITER,
+ TABLE_NOTE_GROUP_MAX_TOKENS,
+ TABLE_NOTE_MAX_BYTES,
+ TABLE_NOTE_MAX_CELL_CHARS,
+ TABLE_NOTE_MAX_COLUMNS,
+ TABLE_NOTE_MAX_ROWS,
+ TABLE_NOTE_ROWS_PER_GROUP,
+ TABLE_NOTE_SKIP_REASON,
+ TABLE_NOTE_SKIP_REASONS,
+ TABLE_TRUNCATION,
+ TABLE_TRUNCATIONS,
+ tableNote,
+ tableNoteFrontmatter,
+ type TableNoteRendered,
+ type TableNoteResult,
+ type TableNoteSkipped,
+} from "../../../../src/core/brain/ingest/table-note.ts";
+
+const encoder = new TextEncoder();
+
+/** A cell pass bounded by a window takes milliseconds; a whole-cell pass over megabytes takes seconds. */
+const CELL_PASS_CEILING_MS = 1_000;
+const MIB = 1 << 20;
+
+function bytesOf(text: string): Uint8Array {
+ return encoder.encode(text);
+}
+
+function rendered(result: TableNoteResult): TableNoteRendered {
+ if (!result.rendered) {
+ throw new Error(`expected a rendered table, got ${result.reason} ${result.detail ?? ""}`);
+ }
+ return result;
+}
+
+function skipped(result: TableNoteResult): TableNoteSkipped {
+ if (result.rendered) throw new Error("expected a skipped table");
+ return result;
+}
+
+/** The lines inside the first `table` fence of a section. */
+function fencedLines(section: string): string[] {
+ const lines = section.split("\n");
+ const open = lines.findIndex((line) => /^`{3,}table$/.test(line));
+ const fence = lines[open]!.slice(0, -"table".length);
+ const close = lines.indexOf(fence, open + 1);
+ return lines.slice(open + 1, close);
+}
+
+function csv(text: string, path = "Clips/data.csv"): TableNoteResult {
+ return tableNote(path, bytesOf(text));
+}
+
+describe("table-note vocabularies", () => {
+ test("skip reasons and truncation kinds are closed four-piece unions", () => {
+ expect(TABLE_NOTE_SKIP_REASONS).toEqual([
+ "source-not-local",
+ "not-utf8",
+ "contains-nul",
+ "malformed-quoting",
+ "empty",
+ ]);
+ expect(TABLE_NOTE_SKIP_REASONS.every(isTableNoteSkipReason)).toBe(true);
+ expect(isTableNoteSkipReason("format-unknown")).toBe(false);
+ expect(TABLE_TRUNCATIONS).toEqual(["rows", "columns", "cells", "bytes"]);
+ expect(TABLE_TRUNCATIONS.every(isTableTruncation)).toBe(true);
+ expect(isTableTruncation(42)).toBe(false);
+ expect(TABLE_NOTE_SKIP_REASON.malformedQuoting).toBe("malformed-quoting");
+ expect(TABLE_TRUNCATION.bytes).toBe("bytes");
+ expect(TABLE_DELIMITER).toEqual({ comma: "comma", semicolon: "semicolon", tab: "tab" });
+ });
+
+ test("a path that is not csv or tsv is a caller error naming the path", () => {
+ expect(() => tableNote("Clips/page.html", bytesOf("a,b"))).toThrow(TypeError);
+ expect(() => tableNote("Clips/page.html", bytesOf("a,b"))).toThrow("Clips/page.html");
+ expect(() => tableNote("Clips/noext", bytesOf("a,b"))).toThrow(TypeError);
+ });
+});
+
+describe("CSV parsing", () => {
+ test("the pinned shape: header first, cells joined by a spaced bar", () => {
+ const result = rendered(csv("name,qty\nbolt,4\nnut,7\n", "Clips/parts.csv"));
+ expect(result).toMatchObject({
+ format: "csv",
+ delimiter: "comma",
+ columns: 2,
+ rows: 2,
+ rowsRendered: 2,
+ truncated: [],
+ redactedCells: 0,
+ });
+ expect(result.section).toBe(
+ "## Table\n\n### Rows 1-2\n\n```table\nname | qty\nbolt | 4\nnut | 7\n```",
+ );
+ });
+
+ test("CRLF, LF and a lone CR all end a record; no trailing empty record", () => {
+ const expected = ["a | b", "1 | 2", "3 | 4"];
+ for (const text of ["a,b\r\n1,2\r\n3,4\r\n", "a,b\n1,2\n3,4", "a,b\r1,2\r3,4\r"]) {
+ const result = rendered(csv(text));
+ expect(fencedLines(result.section)).toEqual(expected);
+ expect(result.rows).toBe(2);
+ }
+ });
+
+ test("a UTF-8 BOM is stripped and fully empty lines are skipped", () => {
+ const result = rendered(csv("id,name\n\n1,x\r\n\r\n2,y\n\n"));
+ expect(fencedLines(result.section)).toEqual(["id | name", "1 | x", "2 | y"]);
+ expect(result.rows).toBe(2);
+ });
+
+ test("quoted fields hold delimiters and line breaks; a doubled quote is one quote", () => {
+ const result = rendered(csv('k,v\n"a,b","line1\nline2"\n"say ""hi""",""\n'));
+ expect(fencedLines(result.section)).toEqual(["k | v", "a,b | line1\\nline2", 'say "hi" | ']);
+ });
+
+ test("bare quotes inside an unquoted field and text after a closing quote are kept", () => {
+ const result = rendered(csv('k,v\n5" pipe,"ab"cd\n'));
+ expect(fencedLines(result.section)).toEqual(["k | v", '5" pipe | abcd']);
+ });
+
+ test("an unterminated quoted field is refused by name with its 1-based record number", () => {
+ const result = skipped(csv('h1,h2\n\nok,1\n"open,2\nmore,3\n'));
+ expect(result).toEqual({
+ rendered: false,
+ format: "csv",
+ reason: "malformed-quoting",
+ detail: "3",
+ });
+ });
+
+ test("ragged rows keep their own cell count; columns is the widest record", () => {
+ const result = rendered(csv("a,b\n1\n1,2,3\n"));
+ expect(fencedLines(result.section)).toEqual(["a | b", "1", "1 | 2 | 3"]);
+ expect(result.columns).toBe(3);
+ });
+
+ test("a semicolon export is read with semicolons; semicolons in comma cells are data", () => {
+ const semicolon = rendered(csv("name;price\nbolt;1,5\nnut;0,25\n"));
+ expect(semicolon.delimiter).toBe("semicolon");
+ expect(fencedLines(semicolon.section)).toEqual(["name | price", "bolt | 1,5", "nut | 0,25"]);
+
+ const comma = rendered(csv("a;b,c\n1;2,3\n"));
+ expect(comma.delimiter).toBe("comma");
+ expect(fencedLines(comma.section)).toEqual(["a;b | c", "1;2 | 3"]);
+
+ const single = rendered(csv("only\nx\n"));
+ expect(single.delimiter).toBe("comma");
+ });
+
+ test("a semicolon header with a comma inside a field is still read with semicolons", () => {
+ for (const text of ['name;"price, eur";qty\nAnn;1,5;2\n', "Name;Price, EUR;Qty\nAnn;1,5;2\n"]) {
+ const result = rendered(csv(text));
+ expect(result.delimiter).toBe("semicolon");
+ expect(result.columns).toBe(3);
+ expect(fencedLines(result.section)[1]).toBe("Ann | 1,5 | 2");
+ }
+ const quoted = rendered(csv('"a;b",c\n1,2\n'));
+ expect(quoted.delimiter).toBe("comma");
+ expect(fencedLines(quoted.section)).toEqual(["a;b | c", "1 | 2"]);
+ });
+
+ test("the first record is the header even when it looks like data", () => {
+ const result = rendered(csv("1,2\n3,4\n"));
+ expect(result.rows).toBe(1);
+ expect(fencedLines(result.section)).toEqual(["1 | 2", "3 | 4"]);
+ });
+
+ test("the extension decides case-insensitively", () => {
+ expect(rendered(tableNote("Clips/DATA.CSV", bytesOf("a,b\n1,2"))).format).toBe("csv");
+ });
+});
+
+describe("TSV parsing", () => {
+ test("TAB-delimited with quotes kept as literal text", () => {
+ const result = rendered(tableNote("Clips/data.tsv", bytesOf('k\tv\n"x"\t"a,b"\r\n')));
+ expect(result.format).toBe("tsv");
+ expect(result.delimiter).toBe("tab");
+ expect(fencedLines(result.section)).toEqual(["k | v", '"x" | "a,b"']);
+ });
+
+ test("an unbalanced quote in a TSV is data, not a refusal", () => {
+ const result = rendered(tableNote("Clips/data.tsv", bytesOf('k\tv\n"open\t1\n')));
+ expect(fencedLines(result.section)).toEqual(["k | v", '"open | 1']);
+ });
+});
+
+describe("refusals", () => {
+ test("a NUL byte is refused as contains-nul", () => {
+ expect(skipped(csv("a,b\n1,\u00002\n"))).toEqual({
+ rendered: false,
+ format: "csv",
+ reason: "contains-nul",
+ });
+ });
+
+ test("invalid UTF-8 is refused as not-utf8", () => {
+ const bytes = new Uint8Array([0x61, 0x2c, 0x62, 0x0a, 0xff, 0xfe, 0x2c, 0x31]);
+ expect(skipped(tableNote("Clips/data.tsv", bytes))).toEqual({
+ rendered: false,
+ format: "tsv",
+ reason: "not-utf8",
+ });
+ });
+
+ test("no records, or a header with no data records, is empty", () => {
+ for (const text of ["", "", "\n\r\n\n", "only,header\n"]) {
+ expect(skipped(csv(text)).reason).toBe("empty");
+ }
+ });
+});
+
+/** Every `### Rows a-b` group of a section with its fenced lines. */
+function groupsOf(
+ section: string,
+): Array<{ first: number; last: number; block: string; lines: string[] }> {
+ const parts = section.split(/\n\n(?=### Rows )/).slice(1);
+ return parts.map((part) => {
+ const match = /^### Rows (\d+)-(\d+)\n\n([\s\S]*?)(?:\n\nRendered \d+ of \d+ rows\.)?$/.exec(
+ part,
+ )!;
+ return {
+ first: Number(match[1]),
+ last: Number(match[2]),
+ block: `### Rows ${match[1]}-${match[2]}\n\n${match[3]}`,
+ lines: fencedLines(match[3]!),
+ };
+ });
+}
+
+function csvOf(header: ReadonlyArray<string>, rows: ReadonlyArray<ReadonlyArray<string>>): string {
+ return [header, ...rows].map((r) => r.join(",")).join("\n");
+}
+
+function numberedRows(count: number, columns: number, cell: (r: number, c: number) => string) {
+ return Array.from({ length: count }, (_row, r) =>
+ Array.from({ length: columns }, (_column, c) => cell(r + 1, c + 1)),
+ );
+}
+
+/** A 240-character cell with no whitespace: many bytes, one token. */
+function wideCell(r: number, c: number): string {
+ return `${r}-${c}-${"w".repeat(240)}`;
+}
+
+/** A five-word cell. */
+function wordsCell(r: number, c: number): string {
+ return `row ${r} cell ${c} text`;
+}
+
+describe("caps", () => {
+ test("more rows than the cap: the rest are counted, named and stated on a closing line", () => {
+ const header = ["id", "v"];
+ const result = rendered(
+ csv(
+ csvOf(
+ header,
+ numberedRows(TABLE_NOTE_MAX_ROWS + 1, 2, (r) => `${r}`),
+ ),
+ ),
+ );
+ expect(result.rows).toBe(TABLE_NOTE_MAX_ROWS + 1);
+ expect(result.rowsRendered).toBe(TABLE_NOTE_MAX_ROWS);
+ expect(result.truncated).toEqual(["rows"]);
+ expect(
+ result.section.endsWith(
+ `\n\nRendered ${TABLE_NOTE_MAX_ROWS} of ${TABLE_NOTE_MAX_ROWS + 1} rows.`,
+ ),
+ ).toBe(true);
+ expect(groupsOf(result.section).at(-1)!.last).toBe(TABLE_NOTE_MAX_ROWS);
+ });
+
+ test("a record wider than the cap is cut at the right", () => {
+ const width = TABLE_NOTE_MAX_COLUMNS + 6;
+ const header = Array.from({ length: width }, (_, c) => `c${c + 1}`);
+ const result = rendered(csv(csvOf(header, [header.map((_, c) => `${c + 1}`)])));
+ expect(result.columns).toBe(TABLE_NOTE_MAX_COLUMNS);
+ expect(result.truncated).toEqual(["columns"]);
+ const [head, row] = fencedLines(result.section);
+ expect(head!.split(" | ")).toEqual(header.slice(0, TABLE_NOTE_MAX_COLUMNS));
+ expect(row!.split(" | ").at(-1)).toBe(String(TABLE_NOTE_MAX_COLUMNS));
+ expect(result.section).not.toContain("Rendered ");
+ });
+
+ test("a 10,000-column record is cut to the column cap without dropping the row", () => {
+ const width = 10_000;
+ const header = Array.from({ length: width }, (_, c) => `c${c + 1}`);
+ const result = rendered(csv(csvOf(header, [header.map((_, c) => `${c + 1}`)])));
+ expect(result.columns).toBe(TABLE_NOTE_MAX_COLUMNS);
+ expect(result.rows).toBe(1);
+ expect(result.truncated).toEqual(["columns"]);
+ const [head, row] = fencedLines(result.section);
+ expect(head!.split(" | ")).toHaveLength(TABLE_NOTE_MAX_COLUMNS);
+ expect(row!.split(" | ")).toHaveLength(TABLE_NOTE_MAX_COLUMNS);
+ });
+
+ test("an unterminated quote that runs to the end of a large file is refused, not rendered", () => {
+ const result = skipped(csv(`h\nok\n"${"x,".repeat(50_000)}`));
+ expect(result).toEqual({
+ rendered: false,
+ format: "csv",
+ reason: "malformed-quoting",
+ detail: "3",
+ });
+ });
+
+ test("a long cell is cut at a code-point boundary and marked with an ellipsis", () => {
+ const long = "\u{1F600}".repeat(TABLE_NOTE_MAX_CELL_CHARS + 10);
+ const result = rendered(csv(`k,v\nshort,${long}\n`));
+ expect(result.truncated).toEqual(["cells"]);
+ const cell = fencedLines(result.section)[1]!.split(" | ")[1]!;
+ expect(Array.from(cell)).toHaveLength(TABLE_NOTE_MAX_CELL_CHARS);
+ expect(cell).toBe(`${"\u{1F600}".repeat(TABLE_NOTE_MAX_CELL_CHARS - 1)}\u2026`);
+ const exact = "x".repeat(TABLE_NOTE_MAX_CELL_CHARS);
+ expect(rendered(csv(`k\n${exact}\n`)).truncated).toEqual([]);
+ });
+
+ test("a megabyte of nested private tags in a cell is hidden in linear time", () => {
+ const nested = `${"<private>".repeat(MIB / 8)}</private>`;
+ const started = performance.now();
+ const result = rendered(csv(`k,v\nrow,${nested}\n`));
+ expect(performance.now() - started).toBeLessThan(CELL_PASS_CEILING_MS);
+ expect(fencedLines(result.section)[1]).toBe(`row | ${PRIVATE_REGION_PLACEHOLDER}`);
+ });
+
+ test("a megabyte-sized cell is redacted in a bounded window, not scanned whole", () => {
+ const userinfo = "a://b:".repeat((4 * MIB) / 6);
+ const started = performance.now();
+ const result = rendered(csv(`k,v\nrow,${userinfo}\n`));
+ expect(performance.now() - started).toBeLessThan(CELL_PASS_CEILING_MS);
+ expect(result.truncated).toEqual(["cells"]);
+ const value = fencedLines(result.section)[1]!.split(" | ")[1]!;
+ expect(Array.from(value)).toHaveLength(TABLE_NOTE_MAX_CELL_CHARS);
+ });
+
+ test("a cell past the scan window counts as cut even when its redacted window fits", () => {
+ const result = rendered(csv(`k,v\nrow,api_key=${"x".repeat(5_000)} tail\n`));
+ expect(fencedLines(result.section)[1]).toBe(`row | api_key=${REDACTION_PLACEHOLDER}`);
+ expect(result.truncated).toEqual(["cells"]);
+ });
+
+ test("groups that would pass the byte cap are dropped whole and named", () => {
+ const result = rendered(csv(csvOf(["a", "b", "c", "d"], numberedRows(900, 4, wideCell))));
+ expect(result.truncated).toEqual(["bytes"]);
+ expect(new TextEncoder().encode(result.section).length).toBeLessThanOrEqual(
+ TABLE_NOTE_MAX_BYTES,
+ );
+ expect(result.rowsRendered).toBeLessThan(900);
+ expect(result.rowsRendered).toBeGreaterThan(0);
+ expect(groupsOf(result.section).at(-1)!.last).toBe(result.rowsRendered);
+ expect(result.section.endsWith(`Rendered ${result.rowsRendered} of 900 rows.`)).toBe(true);
+ });
+
+ test("width and caps count only the rows the byte cap kept", () => {
+ const rows = numberedRows(TABLE_NOTE_MAX_ROWS - 1, 3, () => "w".repeat(250));
+ rows.push(Array.from({ length: 70 }, (_, c) => `x${c}`));
+ const result = rendered(csv(csvOf(["a", "b", "c"], rows)));
+ expect(result.rowsRendered).toBeLessThan(TABLE_NOTE_MAX_ROWS);
+ expect(result.columns).toBe(3);
+ expect(result.truncated).toEqual(["bytes"]);
+ });
+
+ test("several caps are named in the vocabulary order", () => {
+ const width = TABLE_NOTE_MAX_COLUMNS + 1;
+ const header = Array.from({ length: width }, (_, c) => `c${c}`);
+ const rows = numberedRows(TABLE_NOTE_MAX_ROWS + 5, width, (r, c) =>
+ r === 1 && c === 1 ? "y".repeat(400) : "1",
+ );
+ const result = rendered(csv(csvOf(header, rows)));
+ expect(result.truncated.slice(0, 3)).toEqual(["rows", "columns", "cells"]);
+ });
+});
+
+describe("row groups", () => {
+ test("a narrow table groups at most 50 rows, each group repeating the header", () => {
+ const result = rendered(
+ csv(
+ csvOf(
+ ["id", "v"],
+ numberedRows(120, 2, (r, c) => `${r}.${c}`),
+ ),
+ ),
+ );
+ const groups = groupsOf(result.section);
+ expect(groups.map((g) => [g.first, g.last])).toEqual([
+ [1, 50],
+ [51, 100],
+ [101, 120],
+ ]);
+ for (const group of groups) {
+ expect(group.lines[0]).toBe("id | v");
+ expect(group.lines).toHaveLength(group.last - group.first + 2);
+ }
+ expect(groups[1]!.lines[1]).toBe("51.1 | 51.2");
+ });
+
+ test("a wide table groups by the token budget as well, below the row cap", () => {
+ const header = Array.from({ length: 15 }, (_, c) => `column ${c}`);
+ const result = rendered(csv(csvOf(header, numberedRows(200, 15, wordsCell))));
+ const groups = groupsOf(result.section);
+ expect(groups.length).toBeGreaterThan(200 / TABLE_NOTE_ROWS_PER_GROUP);
+ let next = 1;
+ for (const group of groups) {
+ expect(group.first).toBe(next);
+ expect(group.last - group.first + 1).toBeLessThanOrEqual(TABLE_NOTE_ROWS_PER_GROUP);
+ expect(countChunkTokens(group.block)).toBeLessThanOrEqual(TABLE_NOTE_GROUP_MAX_TOKENS);
+ next = group.last + 1;
+ }
+ expect(next).toBe(201);
+ });
+
+ test("a single row over the token budget forms its own group", () => {
+ // Twenty cells of forty words each: more tokens than one group may hold.
+ const cell = Array.from({ length: 40 }, () => "ab").join(" ");
+ const big = Array.from({ length: 20 }, () => cell).join(",");
+ expect(countChunkTokens(big.replaceAll(",", " | "))).toBeGreaterThan(
+ TABLE_NOTE_GROUP_MAX_TOKENS,
+ );
+ const result = rendered(csv(`k,v\na,1\nb,${big}\nc,3\n`));
+ expect(groupsOf(result.section).map((g) => [g.first, g.last])).toEqual([
+ [1, 1],
+ [2, 2],
+ [3, 3],
+ ]);
+ });
+});
+
+describe("escapes and fences", () => {
+ test("bars, backslashes, tabs and line breaks are escaped inside a cell", () => {
+ const result = rendered(csv('k,v\n"a|b","c\\d\te\r\nf"\n'));
+ expect(fencedLines(result.section)[1]).toBe("a\\|b | c\\\\d\\te\\r\\nf");
+ });
+
+ test("every other C0 control, DEL and C1 control is escaped by code point", () => {
+ const result = rendered(
+ csv('k,v\nterm,"a\u001b]0;title\u0007b"\nmore,"\u007f\u0085\u009f\u0001"\n'),
+ );
+ expect(fencedLines(result.section).slice(1)).toEqual([
+ "term | a\\u{001B}]0;title\\u{0007}b",
+ "more | \\u{007F}\\u{0085}\\u{009F}\\u{0001}",
+ ]);
+ // oxlint-disable-next-line no-control-regex -- matching control characters is the point
+ expect(result.section).not.toMatch(/[\u0000-\u0008\u000b-\u001f\u007f-\u009f]/);
+ });
+
+ test("backticks in a cell are escaped, so a table body always takes a three-backtick fence", () => {
+ const result = rendered(csv("k,v\n```js,````\ny,a`b\n"));
+ expect(result.section).toContain("\n```table\n");
+ expect(result.section).not.toContain("````");
+ expect(fencedLines(result.section)).toEqual([
+ "k | v",
+ "\\`\\`\\`js | \\`\\`\\`\\`",
+ "y | a\\`b",
+ ]);
+ });
+
+ test("links and tags in cells stay inside the fence", () => {
+ const result = rendered(csv("k,v\n[[Secret Page]],#tag\n"));
+ expect(fencedLines(result.section)[1]).toBe("[[Secret Page]] | #tag");
+ });
+});
+
+describe("redaction", () => {
+ test("a key-like header redacts its column; other cells go through the value pass", () => {
+ const credential = fakeCredential("q7Lm", "Zp2x", "Vb9n", "Rt4k");
+ const userinfo = fakeCredential("https://deploy:", "hunter", "[email protected]/repo");
+ const text = `name,api_key,url\nalpha,${credential},${userinfo}\nbeta,,plain\n`;
+ const result = rendered(csv(text));
+ expect(result.redactedCells).toBe(2);
+ expect(result.section).not.toContain(credential);
+ expect(result.section).not.toContain("hunter2pass");
+ const [head, alpha, beta] = fencedLines(result.section);
+ expect(head).toBe("name | api_key | url");
+ expect(alpha).toBe(
+ `alpha | ${REDACTION_PLACEHOLDER} | https://${REDACTION_PLACEHOLDER}@example.com/repo`,
+ );
+ expect(beta).toBe("beta | | plain");
+ });
+
+ test("a keyword header over-redacts by design: the single key-name predicate decides", () => {
+ const result = rendered(csv("keyword,count\napple,3\npear,5\n"));
+ expect(result.redactedCells).toBe(2);
+ expect(fencedLines(result.section)).toEqual([
+ "keyword | count",
+ `${REDACTION_PLACEHOLDER} | 3`,
+ `${REDACTION_PLACEHOLDER} | 5`,
+ ]);
+ });
+
+ test("a private region spanning records hides the rows between its tags", () => {
+ const text = "name,note\n<private>,x\nbob,ssn 123-45-6789\n</private>,y\ncarol,z\n";
+ const result = rendered(csv(text));
+ expect(fencedLines(result.section)).toEqual([
+ "name | note",
+ `${PRIVATE_REGION_PLACEHOLDER} | y`,
+ "carol | z",
+ ]);
+ expect(result.rows).toBe(2);
+ expect(result.section).not.toContain("123-45-6789");
+ expect(result.section).not.toContain("private>");
+ const tsv = rendered(tableNote("Clips/data.tsv", bytesOf(text.replaceAll(",", "\t"))));
+ expect(tsv.section).not.toContain("123-45-6789");
+ });
+
+ test("order ids and hashes are table data, not tokens", () => {
+ const hash = "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08";
+ const result = rendered(csv(`order,digest\nA1B2C3D4E5F6G7H8,${hash}\n`));
+ expect(result.redactedCells).toBe(0);
+ expect(fencedLines(result.section)[1]).toBe(`A1B2C3D4E5F6G7H8 | ${hash}`);
+ });
+});
+
+describe("frontmatter", () => {
+ test("frontmatter keys in the pinned order; table_truncated only when something was cut", () => {
+ const plain = rendered(csv("name,qty\nbolt,4\nnut,7\n"));
+ expect(Object.entries(tableNoteFrontmatter(plain))).toEqual([
+ ["table_delimiter", "comma"],
+ ["table_columns", 2],
+ ["table_rows", 2],
+ ["table_rows_rendered", 2],
+ ]);
+ const cut = rendered(
+ csv(
+ csvOf(
+ ["id"],
+ numberedRows(TABLE_NOTE_MAX_ROWS + 1, 1, (r) => `${r}`),
+ ),
+ ),
+ );
+ expect(tableNoteFrontmatter(cut)).toEqual({
+ table_delimiter: "comma",
+ table_columns: 1,
+ table_rows: TABLE_NOTE_MAX_ROWS + 1,
+ table_rows_rendered: TABLE_NOTE_MAX_ROWS,
+ table_truncated: ["rows"],
+ });
+ });
+});
diff --git a/tests/core/brain/session-recall-private.test.ts b/tests/core/brain/session-recall-private.test.ts
new file mode 100644
index 00000000..fabba612
--- /dev/null
+++ b/tests/core/brain/session-recall-private.test.ts
@@ -0,0 +1,65 @@
+/**
+ * An imported session turn whose oversized text is externalized is
+ * flagged private exactly when the redactor finds a `<private>` region in
+ * it - the same detection the continuity sanitiser uses - and finding it
+ * takes linear time on a run of opening tags that never close.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { importSessionRecall } from "../../../src/core/brain/session-recall.ts";
+
+/** Long enough to be externalized under the default payload policy. */
+const BLOB = "QUJD".repeat(5_000);
+/** An opening tag with no `>`: the shape that made the old detection quadratic. */
+const OPEN_TAG_RUN = "<private ";
+/** 720 KB of unclosed opening tags. */
+const LONG_RUN_CHARS = 720 * 1024;
+/** The audit's bound for that input. */
+const LONG_RUN_BUDGET_MS = 1_000;
+
+let vault: string;
+
+beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "o2b-session-recall-private-"));
+});
+
+afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+});
+
+/** Import one turn and return whether its row is flagged private. */
+function importedPrivate(sessionId: string, text: string): boolean {
+ const result = importSessionRecall(vault, {
+ sessionId,
+ createdAt: "2026-09-20T10:00:00.000Z",
+ turns: [{ turnId: "t1", role: "user", timestamp: "2026-09-20T10:00:01.000Z", text }],
+ });
+ return result.rawTurns[0]!.private;
+}
+
+describe("an externalized session turn is flagged private by the redactor's region detection", () => {
+ test("a turn with no private region is not flagged", () => {
+ expect(importedPrivate("s-open", `ordinary ${BLOB}`)).toBe(false);
+ });
+
+ test("a closed private region flags the turn", () => {
+ expect(importedPrivate("s-closed", `<private>internal</private> ${BLOB}`)).toBe(true);
+ });
+
+ for (const prefix of ["tail <private", "keep <private note="]) {
+ test(`an unclosed tag (${JSON.stringify(prefix)}) flags the turn, as the redactor strips it`, () => {
+ expect(importedPrivate("s-unclosed", `${BLOB} ${prefix}`)).toBe(true);
+ });
+ }
+
+ test("720 KB of unclosed opening tags is imported in under a second", () => {
+ const text = `${BLOB} ${OPEN_TAG_RUN.repeat(Math.ceil(LONG_RUN_CHARS / OPEN_TAG_RUN.length))}`;
+ const started = performance.now();
+ importedPrivate("s-long", text);
+ expect(performance.now() - started).toBeLessThan(LONG_RUN_BUDGET_MS);
+ });
+});
diff --git a/tests/core/redactor.test.ts b/tests/core/redactor.test.ts
index dd81a36e..9536482a 100644
--- a/tests/core/redactor.test.ts
+++ b/tests/core/redactor.test.ts
@@ -5,6 +5,7 @@ import {
REDACTION_PLACEHOLDER,
SCAN_TRUNCATED_MARKER,
normaliseTextField,
+ privateRegionTexts,
redactRawOutput,
sanitiseTextField,
stripPrivateRegions,
@@ -30,11 +31,53 @@ describe("stripPrivateRegions", () => {
expect(stripPrivateRegions(input)).toBe(`keep ${PRIVATE_REGION_PLACEHOLDER}`);
});
+ test("an open tag whose attributes hold a bracket still opens a region", () => {
+ const input = 'a <private note="x<y">hidden</private> b';
+ expect(stripPrivateRegions(input)).not.toContain("hidden");
+ expect(privateRegionTexts(input).join("")).toContain("hidden");
+ });
+
+ test("an open tag cut off by the end of the text opens a region there", () => {
+ expect(stripPrivateRegions("keep <private note=")).toBe(`keep ${PRIVATE_REGION_PLACEHOLDER}`);
+ });
+
test("strips nested private regions atomically", () => {
const input = "before <private>a<private>b</private>c</private> after";
expect(stripPrivateRegions(input)).toBe(`before ${PRIVATE_REGION_PLACEHOLDER} after`);
});
+ describe("stays linear on adversarial tag runs", () => {
+ const MIB = 1 << 20;
+ const OPEN = "<private>";
+ const CLOSE = "</private>";
+ const runOf = (unit: string): string => unit.repeat(Math.ceil((2 * MIB) / unit.length));
+ const cases: ReadonlyArray<readonly [string, string, string]> = [
+ ["nested opens with one far close", runOf(OPEN) + CLOSE, PRIVATE_REGION_PLACEHOLDER],
+ [
+ "nested opens then as many closes",
+ runOf(OPEN) + CLOSE.repeat(Math.ceil((2 * MIB) / OPEN.length)),
+ PRIVATE_REGION_PLACEHOLDER,
+ ],
+ [
+ "open tags that never end in a bracket",
+ `x ${runOf("<private ")}`,
+ `x ${PRIVATE_REGION_PLACEHOLDER}`,
+ ],
+ ["a region of unended open tags", `${OPEN}${runOf("<private ")}`, PRIVATE_REGION_PLACEHOLDER],
+ ];
+ for (const [name, input, expected] of cases) {
+ test(name, () => {
+ const started = performance.now();
+ const out = stripPrivateRegions(input);
+ const elapsed = performance.now() - started;
+ expect(out).toBe(expected);
+ const kept = expected.replace(PRIVATE_REGION_PLACEHOLDER, "");
+ expect(privateRegionTexts(input).join("").length).toBe(input.length - kept.length);
+ expect(elapsed).toBeLessThan(LINEAR_CEILING_MS);
+ });
+ }
+ });
+
test("runs before assignment redaction in redactRawOutput", () => {
const input = "visible api_key=keep <private>api_key=secret</private>";
const out = redactRawOutput(input);
diff --git a/tests/core/search/index-root-coverage.test.ts b/tests/core/search/index-root-coverage.test.ts
new file mode 100644
index 00000000..46db616f
--- /dev/null
+++ b/tests/core/search/index-root-coverage.test.ts
@@ -0,0 +1,58 @@
+/**
+ * `indexRootCoverage` partitions the authorized note roots by whether the
+ * index holds a document under them. An optional `admit` predicate lets a
+ * caller count only the documents it may read, so a root reached solely
+ * through pages that caller cannot read answers exactly like an empty one.
+ */
+
+import { afterEach, beforeEach, describe, expect, test } from "bun:test";
+
+import { indexRootCoverage, indexVault } from "../../../src/core/search/indexer.ts";
+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+
+const ROOTS = Object.freeze(["Notes", "Journal", "Empty"]);
+const RESERVED_PAGE = "Journal/reserved.md";
+
+describe("indexRootCoverage", () => {
+ let config: ResolvedSearchConfig;
+ let cleanup: () => void;
+
+ beforeEach(async () => {
+ const temp = createTempVault("root-coverage");
+ cleanup = temp.cleanup;
+ writeMd(temp.vault, "Notes/open.md", "# Open\n\nlattice notes");
+ writeMd(
+ temp.vault,
+ RESERVED_PAGE,
+ "---\nvisibility: [private]\n---\n# Reserved\n\nlattice notes",
+ );
+ config = makeConfig({ vault: temp.vault, dbPath: temp.dbPath });
+ await indexVault(config);
+ });
+ afterEach(() => cleanup());
+
+ test("without an admit predicate every indexed document reaches its root", async () => {
+ const coverage = await indexRootCoverage(config, ROOTS);
+ expect(coverage.rootsWithDocuments).toEqual(["Notes", "Journal"]);
+ expect(coverage.rootsWithoutDocuments).toEqual(["Empty"]);
+ });
+
+ test("an admit predicate that rejects every path leaves every root without documents", async () => {
+ const coverage = await indexRootCoverage(config, ROOTS, () => false);
+ expect(coverage.rootsWithDocuments).toEqual([]);
+ expect(coverage.rootsWithoutDocuments).toEqual(["Notes", "Journal", "Empty"]);
+ });
+
+ test("admit sees each candidate path with the visibility the index measured for it", async () => {
+ const seen = new Map<string, ReadonlyArray<string>>();
+ const coverage = await indexRootCoverage(config, ROOTS, (path, indexedTags) => {
+ seen.set(path, indexedTags);
+ return indexedTags.length === 0;
+ });
+ expect(coverage.rootsWithDocuments).toEqual(["Notes"]);
+ expect(coverage.rootsWithoutDocuments).toEqual(["Journal", "Empty"]);
+ expect(seen.get(RESERVED_PAGE)).toEqual(["private"]);
+ expect(seen.get("Notes/open.md")).toEqual([]);
+ });
+});
diff --git a/tests/core/search/surfacing-gate.test.ts b/tests/core/search/surfacing-gate.test.ts
index 3b030ebc..8ba0d046 100644
--- a/tests/core/search/surfacing-gate.test.ts
+++ b/tests/core/search/surfacing-gate.test.ts
@@ -39,3 +39,24 @@ test("evaluateSurfacingGate honours an explicit recall request", () => {
reason: "explicit",
});
});
+
+test("evaluateSurfacingGate admits a multi-line prompt that starts with a command name", () => {
+ // The line-break test runs on the raw prompt: normalisation collapses
+ // whitespace, so a test on the normalised text could never see one.
+ expect(evaluateSurfacingGate({ prompt: "git status\nwhy does this fail" })).toEqual({
+ retrieve: true,
+ reason: "default_retrieve",
+ });
+ expect(evaluateSurfacingGate({ prompt: "git status\r\nwhy does this fail" }).reason).toBe(
+ "default_retrieve",
+ );
+ expect(evaluateSurfacingGate({ prompt: "git status" }).reason).toBe("shell_command");
+ expect(evaluateSurfacingGate({ prompt: "$ ls -la" }).reason).toBe("shell_command");
+ // A trailing line break alone does not make a one-line command multi-line.
+ expect(evaluateSurfacingGate({ prompt: "git status\n" }).reason).toBe("shell_command");
+ // Documented residue: a single-line prompt that starts with a command
+ // name is still skipped, even when the rest of it is prose.
+ expect(evaluateSurfacingGate({ prompt: "git push keeps failing after the rebase" }).reason).toBe(
+ "shell_command",
+ );
+});
diff --git a/tests/core/vault.test.ts b/tests/core/vault.test.ts
index d2658db3..5b0828a1 100644
--- a/tests/core/vault.test.ts
+++ b/tests/core/vault.test.ts
@@ -135,6 +135,33 @@ describe("parseFrontmatter", () => {
});
});
+describe("parseFrontmatterText after a byte-order mark", () => {
+ test("a byte-order mark before the opening fence still opens the block", () => {
+ expect(parseFrontmatterText("\ufeff---\nvisibility: private\n---\nbody")).toEqual([
+ { visibility: "private" },
+ "body",
+ ]);
+ });
+});
+
+describe("parseFrontmatterText on an unclosed opener", () => {
+ test("stays linear on a long run of blank lines after the opening fence", () => {
+ const text = `---${"\n".repeat(256 * 1024)}name: kept`;
+ const started = performance.now();
+ const [meta, body] = parseFrontmatterText(text);
+ expect(performance.now() - started).toBeLessThan(LINEAR_CEILING_MS);
+ expect(meta).toEqual({});
+ expect(body).toBe(text.trim());
+ });
+
+ test("blank lines after the opening fence still open the block", () => {
+ expect(parseFrontmatterText("---\n\nname: kept\n---\nbody")).toEqual([
+ { name: "kept" },
+ "body",
+ ]);
+ });
+});
+
describe("parseFrontmatterWithNotices", () => {
test("a dash-prefixed non-list line yields exactly one notice naming the line", () => {
// `-foo` is pinned above as "simply ignored as a non-key/value line".
diff --git a/tests/helpers/reach-log-fixture.ts b/tests/helpers/reach-log-fixture.ts
new file mode 100644
index 00000000..6371e738
--- /dev/null
+++ b/tests/helpers/reach-log-fixture.ts
@@ -0,0 +1,203 @@
+/**
+ * The shared vault pair of the log-reader reach A/B tests.
+ *
+ * Vault A holds a reserved confirmed preference with applied and violated
+ * evidence inside both the 24-hour and the 30-day windows, a lifecycle
+ * event outside a dream naming it, and a reserved preference a dream
+ * confirmed and then retired inside the window (only `ret-bygone` is on
+ * disk), with evidence logged before it was retired, so the event names
+ * it only as `pref-bygone`, inside both windows. Vault B never had either. Both carry the same public preference
+ * with its own evidence and a dream confirming it (shared in vault A with
+ * the reserved record), so a remote answer is never empty.
+ */
+
+import { readFileSync, writeFileSync } from "node:fs";
+import { join } from "node:path";
+
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { appendLogEvent } from "../../src/core/brain/log.ts";
+import { brainDirs } from "../../src/core/brain/paths.ts";
+import { writePreference } from "../../src/core/brain/preference.ts";
+import {
+ BRAIN_CONFIDENCE,
+ BRAIN_LOG_EVENT_KIND,
+ BRAIN_PREFERENCE_STATUS,
+} from "../../src/core/brain/types.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import type { TransportReach } from "../../src/core/graph/transport-reach.ts";
+import { REMOTE_DENY_VISIBILITY_TOKEN } from "../../src/core/graph/visibility.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+export const PRIVATE_SLUG = "withheld";
+export const PRIVATE_PATH = `Brain/preferences/pref-${PRIVATE_SLUG}.md`;
+/** Confirmed and then retired by a dream inside the window; only `ret-` is on disk. */
+export const RETIRED_SLUG = "bygone";
+export const SHARED_SLUG = "shared";
+const RESERVE_LINE = `visibility: [${REMOTE_DENY_VISIBILITY_TOKEN}]`;
+const HOUR_MS = 60 * 60 * 1000;
+export const DAY_MS = 24 * HOUR_MS;
+/** Inside the 24-hour window (and the daily brief's calendar day). */
+const RECENT_AGE_MS = HOUR_MS;
+/** Outside the 24-hour window, inside the 30-day one and the weekly window. */
+const OLDER_AGE_MS = 3 * DAY_MS;
+/** Any ISO-8601 instant or date: the two vaults are built moments apart. */
+const STAMP_RE = /\d{4}-\d{2}-\d{2}(?:T\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?Z?)?/g;
+/** The opaque per-vault handle `vault_path` carries: it differs between any two vaults. */
+const VAULT_HANDLE_RE = /vault:\/\/[0-9a-f]+/g;
+
+export interface ReachLogFixture {
+ readonly base: string;
+ readonly configPath: string;
+ readonly vault: string;
+ /** The calendar day of the recent events. */
+ readonly date: string;
+ /** The weekly window's end: the day after now, so the window holds every event. */
+ readonly weekEnd: string;
+}
+
+function confirmed(vault: string, slug: string): void {
+ writePreference(vault, {
+ slug,
+ topic: slug,
+ principle: `Rule ${slug}.`,
+ created_at: "2026-05-01T00:00:00Z",
+ unconfirmed_until: "2026-05-08T00:00:00Z",
+ status: BRAIN_PREFERENCE_STATUS.confirmed,
+ evidenced_by: [],
+ confirmed_at: "2026-05-02T00:00:00Z",
+ applied_count: 1,
+ violated_count: 0,
+ last_evidence_at: "2026-05-02T00:00:00Z",
+ confidence: BRAIN_CONFIDENCE.high,
+ confidence_value: 0.8,
+ });
+}
+
+function reserve(vault: string, rel: string): void {
+ const abs = join(vault, rel);
+ const text = readFileSync(abs, "utf8");
+ const close = text.indexOf("\n---\n", "---\n".length);
+ writeFileSync(abs, `${text.slice(0, close)}\n${RESERVE_LINE}${text.slice(close)}`);
+}
+
+function log(vault: string, timestamp: string, eventType: string, body: Record<string, unknown>) {
+ appendLogEvent(vault, { timestamp, eventType, body } as Parameters<typeof appendLogEvent>[1], {
+ deviceId: "",
+ });
+}
+
+function evidence(vault: string, slug: string, at: string, result: "applied" | "violated") {
+ log(vault, at, BRAIN_LOG_EVENT_KIND.applyEvidence, {
+ path: `Brain/preferences/pref-${slug}.md`,
+ preference: `[[pref-${slug}]]`,
+ artifact: `[[Notes/${slug}-${result}]]`,
+ result,
+ });
+}
+
+function dream(vault: string, at: string, confirmedIds: string[], retiredIds: string[]) {
+ log(vault, at, BRAIN_LOG_EVENT_KIND.dream, {
+ run_id: `run-${at}`,
+ ...(confirmedIds.length > 0 ? { confirmed: confirmedIds.map((id) => `[[${id}|rule]]`) } : {}),
+ ...(retiredIds.length > 0
+ ? { retired: retiredIds.map((id) => `[[${id}|rule]] (superseded-by-context)`) }
+ : {}),
+ });
+}
+
+/**
+ * Vault A (`withPrivate`) or vault B under `base`; `configLines` are
+ * appended to the config file.
+ */
+export function buildReachLogFixture(
+ base: string,
+ withPrivate: boolean,
+ configLines: ReadonlyArray<string> = [],
+): ReachLogFixture {
+ const vault = join(base, "vault");
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(
+ configPath,
+ [`vault: ${vault}`, "agent_name: claude", ...configLines, ""].join("\n"),
+ );
+ bootstrapBrain(vault, { configPath });
+ const now = Date.now();
+ const recent = new Date(now - RECENT_AGE_MS).toISOString();
+ const older = new Date(now - OLDER_AGE_MS).toISOString();
+ confirmed(vault, SHARED_SLUG);
+ evidence(vault, SHARED_SLUG, recent, "applied");
+ evidence(vault, SHARED_SLUG, older, "applied");
+ const sharedDream = [`pref-${SHARED_SLUG}`];
+ dream(vault, recent, withPrivate ? [...sharedDream, `pref-${PRIVATE_SLUG}`] : sharedDream, []);
+ if (withPrivate) {
+ confirmed(vault, PRIVATE_SLUG);
+ reserve(vault, PRIVATE_PATH);
+ for (const at of [recent, older]) {
+ evidence(vault, PRIVATE_SLUG, at, "applied");
+ evidence(vault, PRIVATE_SLUG, at, "violated");
+ }
+ dream(vault, older, [`pref-${PRIVATE_SLUG}`, `pref-${RETIRED_SLUG}`], []);
+ dream(vault, recent, [`pref-${RETIRED_SLUG}`], [`ret-${RETIRED_SLUG}`]);
+ for (const at of [recent, older]) {
+ // Written as apply-evidence writes it: a wikilink with the principle, no path.
+ log(vault, at, BRAIN_LOG_EVENT_KIND.applyEvidence, {
+ preference: `[[pref-${RETIRED_SLUG}|Rule ${RETIRED_SLUG}.]]`,
+ artifact: `[[Notes/${RETIRED_SLUG}-applied]]`,
+ result: "applied",
+ });
+ }
+ log(vault, recent, BRAIN_LOG_EVENT_KIND.forceConfirmed, {
+ path: PRIVATE_PATH,
+ preference: `[[pref-${PRIVATE_SLUG}]]`,
+ });
+ writeFileSync(
+ join(brainDirs(vault).retired, `ret-${RETIRED_SLUG}.md`),
+ [
+ "---",
+ "kind: brain-retired",
+ `id: ret-${RETIRED_SLUG}`,
+ RESERVE_LINE,
+ "created_at: 2026-05-01T00:00:00Z",
+ "tags: []",
+ `topic: ${RETIRED_SLUG}`,
+ "principle: retired principle",
+ "provenance: observed",
+ "retired_at: 2026-05-03T00:00:00Z",
+ "retired_reason: superseded-by-context",
+ "_status: retired",
+ "_evidenced_by: []",
+ "---",
+ "",
+ "Retired.",
+ "",
+ ].join("\n"),
+ );
+ }
+ return {
+ base,
+ configPath,
+ vault,
+ date: recent.slice(0, 10),
+ weekEnd: new Date(now + DAY_MS).toISOString().slice(0, 10),
+ };
+}
+
+/** A server at the reach given, or with no reach minted at all (a remote caller). */
+export function reachServer(f: ReachLogFixture, reach?: TransportReach): MCPServer {
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = f.configPath;
+ const config = { vault: f.vault, configPath: f.configPath };
+ return reach === undefined ? new MCPServer(config) : new MCPServer(config, { reach });
+}
+
+/** `value` as JSON with the fixture's paths, the vault handle and every instant replaced. */
+export function maskVolatile(f: ReachLogFixture, value: unknown): string {
+ let out = JSON.stringify(value);
+ for (const [path, label] of [
+ [f.vault, "<V>"],
+ [f.base, "<B>"],
+ ] as const) {
+ const once = JSON.stringify(path).slice(1, -1);
+ out = out.split(once).join(label).split(path).join(label);
+ }
+ return out.replace(VAULT_HANDLE_RE, "vault://<V>").replace(STAMP_RE, "<T>");
+}
diff --git a/tests/helpers/tool-probe-catalogue.ts b/tests/helpers/tool-probe-catalogue.ts
index 434100b4..8eacd309 100644
--- a/tests/helpers/tool-probe-catalogue.ts
+++ b/tests/helpers/tool-probe-catalogue.ts
@@ -25,6 +25,7 @@
import { runHygieneScan } from "../../src/core/brain/hygiene/scan.ts";
import { scanTriggers } from "../../src/core/brain/triggers/scan.ts";
import { transitionTrigger } from "../../src/core/brain/triggers/store.ts";
+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
export const OWNER_A = "agent-a";
export const OWNER_B = "agent-b";
@@ -141,6 +142,12 @@ export interface ProbeCall {
* the dispatch key cannot be read off a literal record.
*/
readonly label?: string;
+ /**
+ * The transport reach the probe calls with, for the one recipe whose
+ * owner content is only reachable at local reach. Absent means no reach
+ * minted (remote), the HTTP deployment case the matrix covers by default.
+ */
+ readonly reach?: TransportReach;
}
/** One tool, and every call recipe the probe drives it with. */
@@ -501,7 +508,13 @@ export const NON_CONTENT: ReadonlyArray<ProbeEntry> = [
name: "brain_clusters",
calls: [
{ args: { operation: "list" }, reason: REASON.aggregateOnly },
- { args: { operation: "run" }, reason: REASON.ownerFiltered },
+ {
+ args: { operation: "run" },
+ reason: REASON.ownerFiltered,
+ // The fixture's owner community forms through the evidence events on
+ // the Brain log page, which is a graph node only at local reach.
+ reach: TRANSPORT_REACH.local,
+ },
],
},
{ name: "brain_codegraph_report", calls: one({}, REASON.configuredCorpus) },
diff --git a/tests/mcp/agent-scope-matrix.test.ts b/tests/mcp/agent-scope-matrix.test.ts
index 45020825..e6ee68a8 100644
--- a/tests/mcp/agent-scope-matrix.test.ts
+++ b/tests/mcp/agent-scope-matrix.test.ts
@@ -88,6 +88,7 @@ import {
import { CHMOD_CANNOT_DENY } from "../helpers/platform.ts";
import { buildToolTable } from "../../src/mcp/tools.ts";
import type { ServerContext, ToolDefinition } from "../../src/mcp/tool-contract.ts";
+import type { TransportReach } from "../../src/core/graph/transport-reach.ts";
/**
* HOME is pinned for THIS file, and nothing pins it globally.
@@ -185,8 +186,10 @@ async function call(
name: string,
args: Record<string, unknown>,
agentName: string = OWNER_B,
+ reach?: TransportReach,
): Promise<string> {
- return JSON.stringify(await tool(name).handler({ ...ctx, agentName }, args));
+ const callCtx = reach === undefined ? { ...ctx, agentName } : { ...ctx, agentName, reach };
+ return JSON.stringify(await tool(name).handler(callCtx, args));
}
/** Vault-relative paths a listing surface returned. */
@@ -701,9 +704,13 @@ async function seedTwoOwnerFixture(): Promise<void> {
* repository has already shipped one of those (`schema_inspect view=lint`
* naming a host path in its parse error).
*/
-async function probeResponse(name: string, args: Record<string, unknown>): Promise<string> {
+async function probeResponse(
+ name: string,
+ args: Record<string, unknown>,
+ reach?: TransportReach,
+): Promise<string> {
try {
- return await call(name, args, OWNER_B);
+ return await call(name, args, OWNER_B, reach);
} catch (err) {
return `threw: ${err instanceof Error ? err.message : String(err)}`;
}
@@ -778,11 +785,11 @@ for (const entry of PROBE_ENTRIES) {
await seedTwoOwnerFixture();
setGate(GATE_MODE.fail);
- const scoped = await probeResponse(entry.name, probeArgs(recipe));
+ const scoped = await probeResponse(entry.name, probeArgs(recipe), recipe.reach);
expect(scoped, `${label} under fail\n${recipe.reason}`).not.toContain(CROSS_OWNER_MARKER);
setGate(GATE_MODE.off);
- const unscoped = await probeResponse(entry.name, probeArgs(recipe));
+ const unscoped = await probeResponse(entry.name, probeArgs(recipe), recipe.reach);
if (reaches) {
expect(unscoped, `${label} under off\n${recipe.reason}`).toContain(CROSS_OWNER_MARKER);
} else {
diff --git a/tests/mcp/brief-ids-reach.test.ts b/tests/mcp/brief-ids-reach.test.ts
new file mode 100644
index 00000000..188fb5f7
--- /dev/null
+++ b/tests/mcp/brief-ids-reach.test.ts
@@ -0,0 +1,115 @@
+/**
+ * The daily and weekly brief views answer at the caller's reach.
+ *
+ * Both views name preference ids from log events: status transitions
+ * from dream summaries, the weekly retired and contradiction lists, and
+ * the evidence artifacts behind `source_pointers`. Below local reach a
+ * row naming a record the caller cannot read is dropped, a source
+ * pointer is kept only when it is cited by evidence the caller may see,
+ * and no report snapshot is taken.
+ *
+ * The vault pair is tests/helpers/reach-log-fixture.ts. The counts (`events_by_kind`, `vault_delta`)
+ * are recomputed from the events the caller may see, so they are
+ * compared unmasked. A server with no reach minted is a remote caller.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { BRAIN_LOG_EVENT_KIND } from "../../src/core/brain/types.ts";
+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
+import {
+ buildReachLogFixture,
+ DAY_MS,
+ maskVolatile,
+ PRIVATE_SLUG,
+ reachServer as server,
+ type ReachLogFixture as Fixture,
+ RETIRED_SLUG,
+ SHARED_SLUG,
+} from "../helpers/reach-log-fixture.ts";
+
+const SNAPSHOTS_ON = "report_snapshots_enabled: true";
+
+const bases: string[] = [];
+const savedConfig = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+ if (savedConfig === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfig;
+});
+
+function fixture(withPrivate: boolean): Fixture {
+ const base = mkdtempSync(join(tmpdir(), "o2b-brief-ids-reach-"));
+ bases.push(base);
+ return buildReachLogFixture(base, withPrivate, [SNAPSHOTS_ON]);
+}
+
+type View = "daily" | "weekly";
+
+function dayBefore(date: string): string {
+ return new Date(Date.parse(`${date}T00:00:00Z`) - DAY_MS).toISOString().slice(0, 10);
+}
+
+async function brief(f: Fixture, view: View, reach?: TransportReach) {
+ const args = view === "daily" ? { view, date: f.date } : { view, week_end: f.weekEnd };
+ const result = await server(f, reach).callTool("brain_brief", args);
+ return result.structuredContent as Record<string, unknown>;
+}
+
+describe("brain_brief daily and weekly at the caller's reach", () => {
+ for (const view of ["daily", "weekly"] as const) {
+ test(`view=${view} answers remotely as if the reserved records never existed`, async () => {
+ const withheld = fixture(true);
+ const absent = fixture(false);
+ const remoteWithheld = await brief(withheld, view);
+ const remoteAbsent = await brief(absent, view);
+ expect(maskVolatile(withheld, remoteWithheld)).toBe(maskVolatile(absent, remoteAbsent));
+ // Not vacuous: the public preference, its evidence and its counts are named.
+ expect(remoteWithheld["events_by_kind"]).toMatchObject({
+ [BRAIN_LOG_EVENT_KIND.applyEvidence]: view === "daily" ? 1 : 2,
+ [BRAIN_LOG_EVENT_KIND.dream]: 1,
+ });
+ const text = JSON.stringify(remoteWithheld);
+ expect(text).toContain(`pref-${SHARED_SLUG}`);
+ expect(text).toContain(`Notes/${SHARED_SLUG}-applied`);
+ expect(text).not.toContain(PRIVATE_SLUG);
+ expect(text).not.toContain(RETIRED_SLUG);
+ expect(remoteWithheld["delta"]).toBeUndefined();
+ });
+
+ test(`view=${view} still names the reserved records locally and captures the delta`, async () => {
+ const f = fixture(true);
+ const local = await brief(f, view, TRANSPORT_REACH.local);
+ const text = JSON.stringify(local);
+ expect(text).toContain(`pref-${RETIRED_SLUG}`);
+ expect(text).toContain(`ret-${RETIRED_SLUG}`);
+ expect(text).toContain(`Notes/${PRIVATE_SLUG}-applied`);
+ expect(local["delta"]).toMatchObject({ prior_date: null });
+ });
+ }
+
+ test("view=weekly names the reserved preference's own transition and contradiction locally", async () => {
+ const local = await brief(fixture(true), "weekly", TRANSPORT_REACH.local);
+ const contradictions = local["contradictions"] as ReadonlyArray<Record<string, unknown>>;
+ expect(contradictions.map((row) => row["prefId"])).toContain(`pref-${PRIVATE_SLUG}`);
+ expect(JSON.stringify(local["status_transitions"])).toContain(`pref-${PRIVATE_SLUG}`);
+ });
+
+ test("a remote run takes no report snapshot, so a later local run is the first", async () => {
+ // The remote runs are dated a day earlier: a snapshot they took would
+ // be the prior the later local run diffs against.
+ const f = fixture(true);
+ await server(f).callTool("brain_brief", { view: "daily", date: dayBefore(f.date) });
+ await server(f).callTool("brain_brief", { view: "weekly", week_end: dayBefore(f.weekEnd) });
+ expect((await brief(f, "daily", TRANSPORT_REACH.local))["delta"]).toMatchObject({
+ prior_date: null,
+ });
+ expect((await brief(f, "weekly", TRANSPORT_REACH.local))["delta"]).toMatchObject({
+ prior_date: null,
+ });
+ });
+});
diff --git a/tests/mcp/derived-digest-search-reach.test.ts b/tests/mcp/derived-digest-search-reach.test.ts
index 145871cb..2810284e 100644
--- a/tests/mcp/derived-digest-search-reach.test.ts
+++ b/tests/mcp/derived-digest-search-reach.test.ts
@@ -14,6 +14,12 @@
* the digest page, the remote search finds neither the page nor the
* principle, so the remote assertion is not vacuous; an ordinary page is
* still found remotely, so the remote search is not simply empty.
+ *
+ * The corpus statement of a zero-result answer names the authorized note
+ * roots the index never reached. A second A/B pins that it answers at the
+ * caller's reach: a root holding only a reserved page and the same root
+ * empty give identical remote verdicts, while the local verdict still
+ * counts the root as reached.
*/
import { afterEach, describe, expect, test } from "bun:test";
@@ -158,3 +164,74 @@ describe("the compiled digest pages in any spelling the filesystem resolves to t
});
}
});
+
+/** The note root whose only page is reserved in one vault and absent in the other. */
+const RESERVED_ROOT = "Journal";
+const RESERVED_ROOT_PAGE = `${RESERVED_ROOT}/reserved.md`;
+const NOTES_ROOT = "Notes";
+const ROOTS_CONFIG = `schema_version: 1\nnotes:\n read_paths:\n - ${NOTES_ROOT}\n - ${RESERVED_ROOT}\n`;
+const ISO_INSTANT = /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z/g;
+const GATE_ARGS = Object.freeze({ prompt: ABSENT_QUERY, scores: [], match_quality: 0 });
+
+/** Two vaults alike but for one reserved page that is the only page under a note root. */
+async function rootFixture(withReservedPage: boolean): Promise<Fixture> {
+ const base = mkdtempSync(join(tmpdir(), "o2b-root-coverage-reach-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ writeFileSync(join(vault, "Brain", "_brain.yaml"), ROOTS_CONFIG);
+ mkdirSync(join(vault, NOTES_ROOT), { recursive: true });
+ writeFileSync(join(vault, OPEN_PATH), `# Open\n\nThis page mentions ${OPEN_MARKER}.\n`);
+ mkdirSync(join(vault, RESERVED_ROOT), { recursive: true });
+ if (withReservedPage) {
+ writeFileSync(
+ join(vault, RESERVED_ROOT_PAGE),
+ `---\n${RESERVE_LINE}\n---\n# Reserved\n\nA reserved journal entry.\n`,
+ );
+ }
+ await indexVault(resolveSearchConfig({ vault, configPath }), { force: true });
+ return { configPath, vault };
+}
+
+/** A server at the reach given, or with no reach minted at all (remote). */
+function serverAt(f: Fixture, reach?: TransportReach): MCPServer {
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = f.configPath;
+ const config = { vault: f.vault, configPath: f.configPath };
+ return reach === undefined ? new MCPServer(config) : new MCPServer(config, { reach });
+}
+
+/** The two zero-result answers, with the index build instant normalised. */
+async function zeroResultAnswers(f: Fixture, reach?: TransportReach): Promise<string> {
+ const server = serverAt(f, reach);
+ const searched = await server.callTool("brain_search", { query: ABSENT_QUERY });
+ const gated = await server.callTool("brain_recall_gate", { ...GATE_ARGS });
+ return JSON.stringify({ searched, gated }).replaceAll(ISO_INSTANT, "<instant>");
+}
+
+/** The recall gate's negative verdict for a local caller. */
+async function localGateNegative(f: Fixture): Promise<Record<string, unknown>> {
+ const answer = (await serverAt(f, TRANSPORT_REACH.local).callTool("brain_recall_gate", {
+ ...GATE_ARGS,
+ })) as { structuredContent: { negative: Record<string, unknown> } };
+ return answer.structuredContent.negative;
+}
+
+describe("the corpus statement's root coverage answers at the caller's reach", () => {
+ test("a root holding only a reserved page answers remotely like the same root empty", async () => {
+ const withPage = await zeroResultAnswers(await rootFixture(true));
+ const without = await zeroResultAnswers(await rootFixture(false));
+ expect(withPage).toBe(without);
+ // Not vacuous: the empty root is named as never reached.
+ expect(without).toContain("coverage-divergent");
+ expect(without).toContain(RESERVED_ROOT);
+ });
+
+ test("a local caller still counts the reserved page as reaching its root", async () => {
+ const withPage = await localGateNegative(await rootFixture(true));
+ expect(withPage["state"]).toBe("not_found");
+ const without = await localGateNegative(await rootFixture(false));
+ expect(without["unknown_reason"]).toBe("coverage-divergent");
+ });
+});
diff --git a/tests/mcp/ingest-derived-visibility-reach.test.ts b/tests/mcp/ingest-derived-visibility-reach.test.ts
new file mode 100644
index 00000000..0f7b8ccd
--- /dev/null
+++ b/tests/mcp/ingest-derived-visibility-reach.test.ts
@@ -0,0 +1,149 @@
+/**
+ * A summary page that carries a derived section is at most as visible as
+ * its source.
+ *
+ * The derived section copies source content (table rows, HTML headings)
+ * onto `Brain/sources/src-*.md`. A local caller may ingest a source whose
+ * own frontmatter reserves it; the summary page then inherits that
+ * visibility, so a later caller at remote reach finds neither the copied
+ * words through `brain_search` nor the page through the shared reach
+ * predicate every generic page reader asks, while a local caller may still
+ * read the page.
+ *
+ * The A/B control is the same source without its reserving block: the
+ * remote search then finds the derived words on the summary page, so the
+ * remote assertion is not vacuous. (A local `brain_search` is no control:
+ * its default scope leaves reserved pages out at every reach, and the
+ * reserved token cannot be requested as an argument.)
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
+import { indexVault, resolveSearchConfig } from "../../src/core/search/index.ts";
+import { isPathReadableAtReach } from "../../src/core/search/result-filters.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+/** A word only the source's data carries: a table row or an HTML heading. */
+const ROW_WORD = "zzderivedrowprobezz";
+const HEADING_WORD = "zzderivedheadingprobezz";
+const RESERVE_BLOCK = "---\nvisibility: private\n---\n";
+
+/** The byte-order mark a spreadsheet export may put before everything else. */
+const BYTE_ORDER_MARK = "\ufeff";
+
+interface Case {
+ readonly label: string;
+ readonly path: string;
+ /** What precedes the reserving block, kept in the open control too. */
+ readonly lead?: string;
+ /** The source's data, after the reserving block. */
+ readonly data: string;
+ readonly word: string;
+}
+
+const CASES: readonly Case[] = Object.freeze([
+ {
+ label: "a reserved CSV",
+ path: "Clips/reserved.csv",
+ data: `name,code\n${ROW_WORD},4711\n`,
+ word: ROW_WORD,
+ },
+ {
+ label: "a reserved CSV behind a byte-order mark",
+ path: "Clips/reserved-bom.csv",
+ lead: BYTE_ORDER_MARK,
+ data: `name,code\n${ROW_WORD},4711\n`,
+ word: ROW_WORD,
+ },
+ {
+ label: "a reserved HTML file",
+ path: "Clips/reserved.html",
+ data: `<h1>${HEADING_WORD}</h1><p>Body.</p>\n`,
+ word: HEADING_WORD,
+ },
+]);
+
+const bases: string[] = [];
+const savedConfig = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+ if (savedConfig === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfig;
+});
+
+interface Fixture {
+ readonly vault: string;
+ readonly configPath: string;
+ readonly summaryPath: string;
+}
+
+/** A server at the reach given, or with no reach minted at all (remote). */
+function serverAt(f: Pick<Fixture, "vault" | "configPath">, reach?: TransportReach): MCPServer {
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = f.configPath;
+ return new MCPServer(
+ { vault: f.vault, configPath: f.configPath },
+ reach !== undefined ? { reach } : undefined,
+ );
+}
+
+/** A vault where a local caller ingested the source, then indexed. */
+async function ingestedAtLocalReach(c: Case, reserved = true): Promise<Fixture> {
+ const base = mkdtempSync(join(tmpdir(), "o2b-derived-visibility-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ const lead = c.lead ?? "";
+ writeFileSync(join(vault, c.path), `${lead}${reserved ? RESERVE_BLOCK : ""}${c.data}`);
+ const result = (await serverAt({ vault, configPath }, TRANSPORT_REACH.local).callTool(
+ "brain_ingest_source",
+ {
+ source_path: c.path,
+ summary: "A clipped source.",
+ entities: [{ category: "concept", name: "Plan" }],
+ },
+ )) as Record<string, unknown>;
+ const structured = (result["structuredContent"] ?? result) as Record<string, unknown>;
+ await indexVault(resolveSearchConfig({ vault, configPath }), { force: true });
+ return { vault, configPath, summaryPath: String(structured["summary_path"]) };
+}
+
+/** A search with no reach minted, which fails closed to remote. */
+async function remoteSearch(f: Fixture, query: string): Promise<string> {
+ return JSON.stringify(await serverAt(f).callTool("brain_search", { query }));
+}
+
+describe("a summary page derived from a reserved source", () => {
+ for (const c of CASES) {
+ test(`${c.label}: a remote search finds the derived words only when the source is open`, async () => {
+ const open = await ingestedAtLocalReach(c, false);
+ const control = await remoteSearch(open, c.word);
+ expect(control).toContain(c.word);
+ expect(control).toContain(open.summaryPath);
+
+ const f = await ingestedAtLocalReach(c);
+ const remote = await remoteSearch(f, c.word);
+ expect(remote).not.toContain(c.word);
+ expect(remote).not.toContain(f.summaryPath);
+ });
+
+ test(`${c.label}: the summary page is withheld from a remote read`, async () => {
+ const f = await ingestedAtLocalReach(c);
+ expect(isPathReadableAtReach(f.vault, f.summaryPath, TRANSPORT_REACH.local, new Map())).toBe(
+ true,
+ );
+ expect(isPathReadableAtReach(f.vault, f.summaryPath, TRANSPORT_REACH.remote, new Map())).toBe(
+ false,
+ );
+ });
+ }
+});
diff --git a/tests/mcp/ingest-parts-reach.test.ts b/tests/mcp/ingest-parts-reach.test.ts
new file mode 100644
index 00000000..e4a0f36a
--- /dev/null
+++ b/tests/mcp/ingest-parts-reach.test.ts
@@ -0,0 +1,158 @@
+/**
+ * `brain_ingest_source` on an HTML vault file writes the heading-derived
+ * `## Parts` section onto the summary page and reports the part count on
+ * the wire, counts only.
+ *
+ * The parts reader reads source bytes, so it answers at the caller's reach:
+ * through the real MCPServer with no reach minted (which fails closed to
+ * remote), a vault holding an HTML-named source the caller may not read and
+ * a vault without that file give identical normalised results and pages -
+ * no parts, no digest, `source-not-local`.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join, relative } from "node:path";
+
+import { hashBytes } from "../../src/core/brain/ingest/content-manifest.ts";
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+const HTML_PATH = "Clips/page.html";
+const PAGE_HTML =
+ "<html><head><title>Release notes</title></head><body><h1>Overview</h1>" +
+ "<p>Fish &amp; chips</p><h2>Install</h2><p>Run it.</p></body></html>";
+const PARTS_SECTION =
+ "## Parts\n\n```parts\nh1 Overview | lines 1-2\nh2 Overview > Install | lines 3-4\n```";
+/** An HTML-named file holding a Markdown page its owner keeps private. */
+const PRIVATE_BODY = "---\nvisibility: private\n---\n<h1>Vault plan</h1><p>4711</p>\n";
+
+const bases: string[] = [];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+});
+
+interface Fixture {
+ readonly vault: string;
+ readonly server: MCPServer;
+}
+
+function fixture(
+ files: Readonly<Record<string, string>>,
+ reach?: typeof TRANSPORT_REACH.local,
+): Fixture {
+ const base = mkdtempSync(join(tmpdir(), "o2b-ingest-parts-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ for (const [rel, text] of Object.entries(files)) writeFileSync(join(vault, rel), text);
+ const server = new MCPServer({ vault, configPath }, reach !== undefined ? { reach } : undefined);
+ return { vault, server };
+}
+
+function structured(result: Record<string, unknown>): Record<string, unknown> {
+ const inner = result["structuredContent"];
+ return (inner !== undefined ? inner : result) as Record<string, unknown>;
+}
+
+async function ingest(f: Fixture, sourcePath: string): Promise<Record<string, unknown>> {
+ return structured(
+ await f.server.callTool("brain_ingest_source", {
+ source_path: sourcePath,
+ summary: "Release notes of the widget.",
+ entities: [{ category: "concept", name: "Widget" }],
+ }),
+ );
+}
+
+function pageOf(f: Fixture, result: Record<string, unknown>): string {
+ return readFileSync(join(f.vault, String(result["summary_path"])), "utf8");
+}
+
+const TIMESTAMP_RE = /^(created_at|updated_at): .*$/gm;
+
+/** Every Brain page outside the dated log, by forward-slash path, timestamps blanked. */
+function normalisedPages(vault: string): Record<string, string> {
+ const root = join(vault, "Brain");
+ const out: Record<string, string> = {};
+ for (const entry of readdirSync(root, { recursive: true, withFileTypes: true })) {
+ if (!entry.isFile() || !entry.name.endsWith(".md")) continue;
+ const rel = relative(root, join(entry.parentPath, entry.name)).replaceAll("\\", "/");
+ if (rel.startsWith("log/")) continue;
+ out[rel] = readFileSync(join(entry.parentPath, entry.name), "utf8").replace(
+ TIMESTAMP_RE,
+ "$1: T",
+ );
+ }
+ return Object.fromEntries(Object.entries(out).toSorted(([a], [b]) => a.localeCompare(b)));
+}
+
+describe("brain_ingest_source on an HTML source", () => {
+ test("reports the part count and ends the page with the Parts section", async () => {
+ const f = fixture({ [HTML_PATH]: PAGE_HTML });
+ const result = await ingest(f, HTML_PATH);
+ expect(result["parts"]).toEqual({ extracted: true, count: 2 });
+ const keys = Object.keys(result);
+ expect(keys.indexOf("parts")).toBe(keys.indexOf("capture_scope") + 1);
+ expect("table" in result).toBe(false);
+
+ const page = pageOf(f, result);
+ expect(page).toContain("\nsource_format: html\n");
+ expect(page).toContain(
+ `\nsource_content_hash: ${hashBytes(new TextEncoder().encode(PAGE_HTML))}\n`,
+ );
+ expect(page.trimEnd().endsWith(PARTS_SECTION)).toBe(true);
+ // Headings and line spans only: no body text of the source reaches the page.
+ expect(page).not.toContain("Fish");
+ });
+
+ test("an HTML file that is not UTF-8 is data: the page is written without parts", async () => {
+ const f = fixture({});
+ writeFileSync(join(f.vault, HTML_PATH), new Uint8Array([0x3c, 0x68, 0x31, 0x3e, 0xff]));
+ const result = await ingest(f, HTML_PATH);
+ expect(result["parts"]).toEqual({ extracted: false, reason: "not-utf8" });
+ expect(result["entities_created"]).toHaveLength(1);
+ expect(pageOf(f, result)).not.toContain("## Parts");
+ });
+
+ test("a Markdown source carries no parts key", async () => {
+ const f = fixture({ "Clips/notes.md": "# Notes\n" });
+ const result = await ingest(f, "Clips/notes.md");
+ expect("parts" in result).toBe(false);
+ expect(pageOf(f, result)).not.toContain("source_format");
+ });
+});
+
+describe("reach", () => {
+ test("a hidden HTML-named source and an absent one answer identically at remote reach", async () => {
+ const hidden = fixture({ [HTML_PATH]: PRIVATE_BODY });
+ const absent = fixture({});
+ const hiddenResult = await ingest(hidden, HTML_PATH);
+ const absentResult = await ingest(absent, HTML_PATH);
+
+ expect(JSON.stringify(hiddenResult)).toBe(JSON.stringify(absentResult));
+ expect(hiddenResult["parts"]).toEqual({ extracted: false, reason: "source-not-local" });
+ expect(normalisedPages(hidden.vault)).toEqual(normalisedPages(absent.vault));
+ const page = pageOf(hidden, hiddenResult);
+ expect(page).not.toContain("source_content_hash");
+ expect(page).not.toContain("source_format");
+ expect(page).not.toContain("## Parts");
+ expect(page).not.toContain("Vault plan");
+ });
+
+ test("at local reach the same file is read and its parts written", async () => {
+ const f = fixture({ [HTML_PATH]: PRIVATE_BODY }, TRANSPORT_REACH.local);
+ const result = await ingest(f, HTML_PATH);
+ expect(result["parts"]).toMatchObject({ extracted: true });
+ const page = pageOf(f, result);
+ expect(page).toContain("source_content_hash");
+ expect(page).toContain("h1 Vault plan | lines");
+ });
+});
diff --git a/tests/mcp/ingest-plan-format-reach.test.ts b/tests/mcp/ingest-plan-format-reach.test.ts
new file mode 100644
index 00000000..8c631ad1
--- /dev/null
+++ b/tests/mcp/ingest-plan-format-reach.test.ts
@@ -0,0 +1,107 @@
+/**
+ * `brain_ingest_batch_plan` names format skips the same way whatever the
+ * caller may read.
+ *
+ * Format routing runs before the reach predicate (visibility is a Markdown
+ * rule, and asking it about a binary would read the binary whole), so the
+ * format-skip list of a tree holding a page the caller cannot read must equal
+ * the list of the same tree without that page. Through the real MCPServer
+ * with no reach minted, which fails closed to remote.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+const PRIVATE_PATH = "Clips/private.md";
+const PRIVATE_BODY = "---\nvisibility: private\n---\n# Private plan\nThe code is 4711.\n";
+/**
+ * Hidden pages of the non-Markdown planned formats: HTML and CSV are now
+ * planned by default and pass the same reach predicate as Markdown, whose
+ * A/B is the first test below.
+ */
+const PRIVATE_PAGES: ReadonlyArray<readonly [string, string]> = Object.freeze([
+ ["Clips/private.csv", "---\nvisibility: private\n---\nname,code\nvault,4711\n"],
+ ["Clips/private.html", "---\nvisibility: private\n---\n<h1>Private plan</h1><p>4711</p>\n"],
+]);
+
+const bases: string[] = [];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+});
+
+interface Fixture {
+ readonly vault: string;
+ readonly server: MCPServer;
+}
+
+function fixture(
+ withPrivate: boolean,
+ reach?: typeof TRANSPORT_REACH.local,
+ privatePage: readonly [string, string] = [PRIVATE_PATH, PRIVATE_BODY],
+): Fixture {
+ const base = mkdtempSync(join(tmpdir(), "o2b-plan-format-reach-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ writeFileSync(join(vault, "Clips/open.md"), "# Open\n");
+ writeFileSync(join(vault, "Clips/page.html"), "<h1>Overview</h1>");
+ writeFileSync(join(vault, "Clips/report.pdf"), "%PDF-1.7\n");
+ writeFileSync(join(vault, "Clips/scan.png"), "png");
+ writeFileSync(join(vault, "Clips/blob.bin"), "abc");
+ if (withPrivate) writeFileSync(join(vault, privatePage[0]), privatePage[1]);
+ const server = new MCPServer({ vault, configPath }, reach !== undefined ? { reach } : undefined);
+ return { vault, server };
+}
+
+async function plan(f: Fixture): Promise<Record<string, unknown>> {
+ return (await f.server.callTool("brain_ingest_batch_plan", {
+ source_dir: "Clips",
+ })) as Record<string, unknown>;
+}
+
+function structured(result: Record<string, unknown>): Record<string, unknown> {
+ const inner = result["structuredContent"];
+ return (inner !== undefined ? inner : result) as Record<string, unknown>;
+}
+
+describe("format skips at remote reach", () => {
+ test("a tree with a hidden page and one without give identical plans", async () => {
+ const hidden = structured(await plan(fixture(true)));
+ const absent = structured(await plan(fixture(false)));
+ expect(JSON.stringify(hidden)).toBe(JSON.stringify(absent));
+ expect(hidden["skipped_non_extractable"]).toEqual([
+ { path: "Clips/report.pdf", reason: "format-not-extractable", detail: "pdf" },
+ { path: "Clips/scan.png", reason: "format-not-extractable", detail: "image" },
+ ]);
+ expect(hidden["unclassifiable"]).toEqual({ total: 1, by_extension: { ".bin": 1 } });
+ });
+
+ for (const page of PRIVATE_PAGES) {
+ test(`a hidden ${page[0]} leaves the remote plan identical to the tree without it`, async () => {
+ const hidden = structured(await plan(fixture(true, undefined, page)));
+ const absent = structured(await plan(fixture(false)));
+ expect(JSON.stringify(hidden)).toBe(JSON.stringify(absent));
+ const local = structured(await plan(fixture(true, TRANSPORT_REACH.local, page)));
+ expect(JSON.stringify(local["batches"])).toContain(page[0]);
+ });
+ }
+
+ test("at local reach the private page is planned and the format skips are the same", async () => {
+ const local = structured(await plan(fixture(true, TRANSPORT_REACH.local)));
+ const remote = structured(await plan(fixture(true)));
+ expect(local["skipped_non_extractable"]).toEqual(remote["skipped_non_extractable"]);
+ expect(JSON.stringify(local["batches"])).toContain(PRIVATE_PATH);
+ expect(JSON.stringify(remote["batches"])).not.toContain(PRIVATE_PATH);
+ });
+});
diff --git a/tests/mcp/ingest-reach-round3.test.ts b/tests/mcp/ingest-reach-round3.test.ts
new file mode 100644
index 00000000..08296add
--- /dev/null
+++ b/tests/mcp/ingest-reach-round3.test.ts
@@ -0,0 +1,117 @@
+/**
+ * An ingest of a source reserved by its own frontmatter records no digest
+ * of the reserved bytes on a page a remote reader may read.
+ *
+ * The entity pages an ingest creates are not reserved: they inherit no
+ * visibility from the source. Their `source_content_hash` was the sha256
+ * of the whole source file, so a remote caller could confirm a guessed
+ * content of a small reserved file by searching for its digest. Whoever
+ * ingests the reserved source (a local caller, or a remote one that
+ * cannot read it), every remotely readable page must carry the digest as
+ * often as in the vault that never ingested it: never.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { createHash } from "node:crypto";
+import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join, relative } from "node:path";
+
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
+import { isPathReadableAtReach } from "../../src/core/search/result-filters.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+const SOURCE_PATH = "Clips/reserved.csv";
+const RESERVE_LINE = "visibility: private\n";
+const PRIVATE_BODY = `---\n${RESERVE_LINE}---\nname,code\nzzdigestprobezz,4711\n`;
+const OPEN_BODY = PRIVATE_BODY.replace(RESERVE_LINE, "");
+const INGEST_ARGS = Object.freeze({
+ source_path: SOURCE_PATH,
+ summary: "A clipped source.",
+ entities: [{ category: "concept", name: "Plan" }],
+});
+const MARKDOWN_EXT = ".md";
+
+const bases: string[] = [];
+const savedConfig = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+ if (savedConfig === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfig;
+});
+
+interface Fixture {
+ readonly vault: string;
+ readonly configPath: string;
+}
+
+/** A vault holding the source with `body`, never ingested. */
+function vaultWithSource(body: string): Fixture {
+ const base = mkdtempSync(join(tmpdir(), "o2b-ingest-reach-r3-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ writeFileSync(join(vault, SOURCE_PATH), body);
+ return { vault, configPath };
+}
+
+/** Ingest the source through the server at `reach`, or with no reach minted (remote). */
+async function ingest(f: Fixture, reach?: TransportReach): Promise<void> {
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = f.configPath;
+ const server = new MCPServer(
+ { vault: f.vault, configPath: f.configPath },
+ reach !== undefined ? { reach } : undefined,
+ );
+ await server.callTool("brain_ingest_source", { ...INGEST_ARGS });
+}
+
+function markdownPages(dir: string): string[] {
+ return readdirSync(dir, { withFileTypes: true }).flatMap((entry) => {
+ const abs = join(dir, entry.name);
+ if (entry.isDirectory()) return markdownPages(abs);
+ return entry.name.endsWith(MARKDOWN_EXT) ? [abs] : [];
+ });
+}
+
+/** The pages a remote reader may read that carry `digest`, vault-relative and sorted. */
+function remotePagesCarrying(f: Fixture, digest: string): ReadonlyArray<string> {
+ const cache = new Map();
+ return markdownPages(f.vault)
+ .map((abs) => relative(f.vault, abs).replaceAll("\\", "/"))
+ .filter((rel) => isPathReadableAtReach(f.vault, rel, TRANSPORT_REACH.remote, cache))
+ .filter((rel) => readFileSync(join(f.vault, rel), "utf8").includes(digest))
+ .toSorted();
+}
+
+function sha256(text: string): string {
+ return createHash("sha256").update(text).digest("hex");
+}
+
+describe("an ingest of a reserved source records no digest a remote reader can read", () => {
+ for (const [label, reach] of [
+ ["a local caller", TRANSPORT_REACH.local],
+ ["a remote caller", undefined],
+ ] as const) {
+ test(`ingested by ${label}, as in the vault that never ingested it`, async () => {
+ const digest = sha256(PRIVATE_BODY);
+ const reserved = vaultWithSource(PRIVATE_BODY);
+ const never = vaultWithSource(PRIVATE_BODY);
+ await ingest(reserved, reach);
+ expect(remotePagesCarrying(reserved, digest)).toEqual(remotePagesCarrying(never, digest));
+ expect(remotePagesCarrying(reserved, digest)).toEqual([]);
+ });
+ }
+
+ test("an open source still has its digest recorded on the entity page (positive control)", async () => {
+ const open = vaultWithSource(OPEN_BODY);
+ await ingest(open, TRANSPORT_REACH.local);
+ const carrying = remotePagesCarrying(open, sha256(OPEN_BODY));
+ expect(carrying.some((rel) => rel.startsWith("Brain/entities/"))).toBe(true);
+ });
+});
diff --git a/tests/mcp/ingest-reserved-summary-reach.test.ts b/tests/mcp/ingest-reserved-summary-reach.test.ts
new file mode 100644
index 00000000..8aca31d0
--- /dev/null
+++ b/tests/mcp/ingest-reserved-summary-reach.test.ts
@@ -0,0 +1,111 @@
+/**
+ * A remote re-ingest of a summary page the caller may not read.
+ *
+ * A local caller ingests a source reserved by its own frontmatter, so the
+ * summary page inherits the reservation. The summary path is deterministic,
+ * so a later `brain_ingest_source` of the same source at remote reach (no
+ * reach minted) must answer exactly as for the same vault without that
+ * page, and must leave the page byte-identical: the answer is no oracle for
+ * the page's existence, and the call is no way to overwrite it.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+const SOURCE_PATH = "Clips/reserved.csv";
+const RESERVE_LINE = "visibility: private\n";
+const PRIVATE_BODY = `---\n${RESERVE_LINE}---\nname,code\nzzreingestprobezz,4711\n`;
+const INGEST_ARGS = Object.freeze({
+ source_path: SOURCE_PATH,
+ summary: "A clipped source.",
+ entities: [{ category: "concept", name: "Plan" }],
+});
+
+const bases: string[] = [];
+const savedConfig = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+ if (savedConfig === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfig;
+});
+
+interface Fixture {
+ readonly vault: string;
+ readonly configPath: string;
+}
+
+/** A server at the reach given, or with no reach minted at all (remote). */
+function serverAt(f: Fixture, reach?: TransportReach): MCPServer {
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = f.configPath;
+ return new MCPServer(
+ { vault: f.vault, configPath: f.configPath },
+ reach !== undefined ? { reach } : undefined,
+ );
+}
+
+async function ingest(f: Fixture, reach?: TransportReach): Promise<Record<string, unknown>> {
+ const result = (await serverAt(f, reach).callTool("brain_ingest_source", {
+ ...INGEST_ARGS,
+ })) as Record<string, unknown>;
+ return (result["structuredContent"] ?? result) as Record<string, unknown>;
+}
+
+/**
+ * A vault where a local caller ingested the reserved source; with
+ * `withSummary` false its summary page is then removed (the control).
+ */
+async function ingestedReservedSource(
+ withSummary: boolean,
+): Promise<Fixture & { readonly summaryPath: string }> {
+ const base = mkdtempSync(join(tmpdir(), "o2b-reserved-reingest-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ writeFileSync(join(vault, SOURCE_PATH), PRIVATE_BODY);
+ const f = { vault, configPath };
+ const summaryPath = String((await ingest(f, TRANSPORT_REACH.local))["summary_path"]);
+ if (!withSummary) rmSync(join(vault, summaryPath));
+ return { ...f, summaryPath };
+}
+
+describe("a remote re-ingest of a reserved summary page", () => {
+ test("answers as for the vault without the page and leaves the page byte-identical", async () => {
+ const reserved = await ingestedReservedSource(true);
+ const control = await ingestedReservedSource(false);
+ const summaryAbs = join(reserved.vault, reserved.summaryPath);
+ const before = readFileSync(summaryAbs);
+
+ const remote = await ingest(reserved);
+ expect(remote).toEqual(await ingest(control));
+ expect(remote["created"]).toBe(true);
+ expect(readFileSync(summaryAbs).equals(before)).toBe(true);
+ });
+
+ test("a reserved summary page is left as it is once its source is open", async () => {
+ const f = await ingestedReservedSource(true);
+ writeFileSync(join(f.vault, SOURCE_PATH), PRIVATE_BODY.replace(RESERVE_LINE, ""));
+ const summaryAbs = join(f.vault, f.summaryPath);
+ const before = readFileSync(summaryAbs);
+
+ expect((await ingest(f))["created"]).toBe(true);
+ expect(readFileSync(summaryAbs).equals(before)).toBe(true);
+ });
+
+ test("a local re-ingest still reads and rewrites the page (positive control)", async () => {
+ const reserved = await ingestedReservedSource(true);
+ const local = await ingest(reserved, TRANSPORT_REACH.local);
+ expect(local["created"]).toBe(false);
+ expect(local["summary_path"]).toBe(reserved.summaryPath);
+ });
+});
diff --git a/tests/mcp/ingest-source-cleanup-reach.test.ts b/tests/mcp/ingest-source-cleanup-reach.test.ts
new file mode 100644
index 00000000..6cb259ab
--- /dev/null
+++ b/tests/mcp/ingest-source-cleanup-reach.test.ts
@@ -0,0 +1,152 @@
+/**
+ * The source-cleanup tools answer at the caller's reach.
+ *
+ * A summary page derived from a reserved source inherits the source's
+ * visibility. `brain_search_by_source` and `brain_delete_by_source` find
+ * pages by the source they trace to, so at remote reach (no reach minted)
+ * they must answer exactly as for the same vault without that page: no
+ * entry, no count, no manifest entry for the reserved source, and a
+ * confirmed delete that leaves the reserved page byte-identical. A local
+ * caller is the positive control: it finds the page and may delete it.
+ *
+ * The entity page the same ingest writes is remote-readable and traces to
+ * the source in both vaults; it is written before the source's visibility
+ * is read, so it is not part of what this rule withholds.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { readManifest } from "../../src/core/brain/ingest/content-manifest.ts";
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+const SOURCE_PATH = "Clips/reserved.csv";
+const PRIVATE_BODY = "---\nvisibility: private\n---\nname,code\nzzcleanupprobezz,4711\n";
+
+const bases: string[] = [];
+const savedConfig = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+ if (savedConfig === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfig;
+});
+
+interface Fixture {
+ readonly vault: string;
+ readonly configPath: string;
+ /** The summary page a local ingest wrote, or null when none ran. */
+ readonly summaryPath: string | null;
+}
+
+/** A server at the reach given, or with no reach minted at all (remote). */
+function serverAt(f: Fixture, reach?: TransportReach): MCPServer {
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = f.configPath;
+ return new MCPServer(
+ { vault: f.vault, configPath: f.configPath },
+ reach !== undefined ? { reach } : undefined,
+ );
+}
+
+/** The structured payload of one tool call. */
+async function call(
+ f: Fixture,
+ tool: string,
+ args: Record<string, unknown>,
+ reach?: TransportReach,
+): Promise<Record<string, unknown>> {
+ const result = (await serverAt(f, reach).callTool(tool, args)) as Record<string, unknown>;
+ return (result["structuredContent"] ?? result) as Record<string, unknown>;
+}
+
+/**
+ * A vault where a local caller ingested the reserved source. With
+ * `withSummary` false the reserved summary page is then removed: the
+ * control, identical but for the page a remote caller may not read.
+ */
+async function ingestedReservedSource(withSummary: boolean): Promise<Fixture> {
+ const base = mkdtempSync(join(tmpdir(), "o2b-cleanup-reach-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ writeFileSync(join(vault, SOURCE_PATH), PRIVATE_BODY);
+ const bare: Fixture = { vault, configPath, summaryPath: null };
+ const ingest = await call(
+ bare,
+ "brain_ingest_source",
+ {
+ source_path: SOURCE_PATH,
+ summary: "A clipped source.",
+ entities: [{ category: "concept", name: "Plan" }],
+ },
+ TRANSPORT_REACH.local,
+ );
+ const summaryPath = String(ingest["summary_path"]);
+ if (withSummary) return { ...bare, summaryPath };
+ rmSync(join(vault, summaryPath));
+ return bare;
+}
+
+/** Fields minted per run (record and snapshot ids), named rather than compared. */
+const PER_RUN_FIELDS = Object.freeze(["audit_record_id", "snapshot_run_id"]);
+
+function normalised(result: Record<string, unknown>): Record<string, unknown> {
+ const out = { ...result };
+ for (const field of PER_RUN_FIELDS) if (out[field] !== null) out[field] = `<${field}>`;
+ return out;
+}
+
+describe("source-cleanup tools at the caller's reach", () => {
+ test("brain_search_by_source answers a remote caller as if the reserved page were absent", async () => {
+ const reserved = await ingestedReservedSource(true);
+ const control = await ingestedReservedSource(false);
+ const args = { source_file: SOURCE_PATH };
+
+ const local = await call(reserved, "brain_search_by_source", args, TRANSPORT_REACH.local);
+ expect(JSON.stringify(local)).toContain(String(reserved.summaryPath));
+
+ expect(await call(reserved, "brain_search_by_source", args)).toEqual(
+ await call(control, "brain_search_by_source", args),
+ );
+ });
+
+ test("a remote dry run of brain_delete_by_source plans nothing it may not read", async () => {
+ const reserved = await ingestedReservedSource(true);
+ const control = await ingestedReservedSource(false);
+ const args = { source_file: SOURCE_PATH };
+
+ const local = await call(reserved, "brain_delete_by_source", args, TRANSPORT_REACH.local);
+ expect(JSON.stringify(local)).toContain(String(reserved.summaryPath));
+ expect(local["manifest_entry"]).toBe(SOURCE_PATH);
+
+ const remote = await call(reserved, "brain_delete_by_source", args);
+ expect(remote["manifest_entry"]).toBeNull();
+ expect(remote).toEqual(await call(control, "brain_delete_by_source", args));
+ });
+
+ test("a remote confirmed delete leaves the reserved summary page byte-identical", async () => {
+ const reserved = await ingestedReservedSource(true);
+ const control = await ingestedReservedSource(false);
+ const args = { source_file: SOURCE_PATH, confirm: true };
+ const summaryAbs = join(reserved.vault, String(reserved.summaryPath));
+ const before = readFileSync(summaryAbs);
+
+ expect(normalised(await call(reserved, "brain_delete_by_source", args))).toEqual(
+ normalised(await call(control, "brain_delete_by_source", args)),
+ );
+ expect(readFileSync(summaryAbs).equals(before)).toBe(true);
+ expect(readManifest(reserved.vault).entries[SOURCE_PATH]).toBeDefined();
+
+ const local = await call(reserved, "brain_delete_by_source", args, TRANSPORT_REACH.local);
+ expect(local["deleted"]).toContain(reserved.summaryPath);
+ expect(existsSync(summaryAbs)).toBe(false);
+ });
+});
diff --git a/tests/mcp/ingest-tool-table.test.ts b/tests/mcp/ingest-tool-table.test.ts
new file mode 100644
index 00000000..c1fd8959
--- /dev/null
+++ b/tests/mcp/ingest-tool-table.test.ts
@@ -0,0 +1,216 @@
+/**
+ * `brain_ingest_source` on a CSV or TSV vault file renders its table note
+ * onto the summary page and reports it on the wire, counts and tokens only.
+ *
+ * The table renderer reads source bytes, so it answers at the caller's
+ * reach: through the real MCPServer with no reach minted (which fails
+ * closed to remote), a vault holding a `.csv`-named source the caller may
+ * not read and a vault without that file give identical normalised results
+ * and pages - no table, no digest, `source-not-local`.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join, relative } from "node:path";
+
+import { bootstrapBrain } from "../../src/core/brain/init.ts";
+import { hashBytes } from "../../src/core/brain/ingest/content-manifest.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+const CSV_PATH = "Clips/parts.csv";
+const TSV_PATH = "Clips/parts.tsv";
+const PARTS_CSV = "name,qty\nbolt,4\nnut,7\n";
+const PRIVATE_BODY = "---\nvisibility: private\n---\nname,code\nvault,4711\n";
+
+const bases: string[] = [];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+});
+
+interface Fixture {
+ readonly vault: string;
+ readonly server: MCPServer;
+}
+
+function fixture(
+ files: Readonly<Record<string, string>>,
+ reach?: typeof TRANSPORT_REACH.local,
+): Fixture {
+ const base = mkdtempSync(join(tmpdir(), "o2b-ingest-table-"));
+ bases.push(base);
+ const vault = join(base, "vault");
+ mkdirSync(join(vault, "Clips"), { recursive: true });
+ const configPath = join(base, "config.yaml");
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+ bootstrapBrain(vault, { configPath });
+ for (const [rel, text] of Object.entries(files)) writeFileSync(join(vault, rel), text);
+ const server = new MCPServer({ vault, configPath }, reach !== undefined ? { reach } : undefined);
+ return { vault, server };
+}
+
+function structured(result: Record<string, unknown>): Record<string, unknown> {
+ const inner = result["structuredContent"];
+ return (inner !== undefined ? inner : result) as Record<string, unknown>;
+}
+
+async function ingest(f: Fixture, sourcePath: string): Promise<Record<string, unknown>> {
+ return structured(
+ await f.server.callTool("brain_ingest_source", {
+ source_path: sourcePath,
+ summary: "Parts list.",
+ entities: [{ category: "concept", name: "Fasteners" }],
+ }),
+ );
+}
+
+function pageOf(f: Fixture, result: Record<string, unknown>): string {
+ return readFileSync(join(f.vault, String(result["summary_path"])), "utf8");
+}
+
+/** The frontmatter keys of a page, in order. */
+function frontmatterKeys(page: string): string[] {
+ const head = page.split("\n---\n")[0]!;
+ return head
+ .split("\n")
+ .slice(1)
+ .filter((line) => /^[a-z_]+:/.test(line))
+ .map((line) => line.slice(0, line.indexOf(":")));
+}
+
+const TIMESTAMP_RE = /^(created_at|updated_at): .*$/gm;
+
+/** Every Brain page outside the dated log, by forward-slash path, timestamps blanked. */
+function normalisedPages(vault: string): Record<string, string> {
+ const root = join(vault, "Brain");
+ const out: Record<string, string> = {};
+ for (const entry of readdirSync(root, { recursive: true, withFileTypes: true })) {
+ if (!entry.isFile() || !entry.name.endsWith(".md")) continue;
+ const rel = relative(root, join(entry.parentPath, entry.name)).replaceAll("\\", "/");
+ if (rel.startsWith("log/")) continue;
+ out[rel] = readFileSync(join(entry.parentPath, entry.name), "utf8").replace(
+ TIMESTAMP_RE,
+ "$1: T",
+ );
+ }
+ return Object.fromEntries(Object.entries(out).toSorted(([a], [b]) => a.localeCompare(b)));
+}
+
+describe("brain_ingest_source on a table source", () => {
+ test("the table wire field, the five page keys and the section last", async () => {
+ const f = fixture({ [CSV_PATH]: PARTS_CSV });
+ const result = await ingest(f, CSV_PATH);
+ expect(result["table"]).toEqual({
+ rendered: true,
+ format: "csv",
+ delimiter: "comma",
+ columns: 2,
+ rows: 2,
+ rows_rendered: 2,
+ redacted_cells: 0,
+ });
+ const keys = Object.keys(result);
+ expect(keys.indexOf("table")).toBe(keys.indexOf("capture_scope") + 1);
+
+ const page = pageOf(f, result);
+ const fm = frontmatterKeys(page);
+ const derived = [
+ "source_format",
+ "source_content_hash",
+ "table_delimiter",
+ "table_columns",
+ "table_rows",
+ "table_rows_rendered",
+ ];
+ expect(fm.filter((key) => derived.includes(key))).toEqual(derived);
+ expect(fm).not.toContain("table_truncated");
+ expect(page).toContain("\nsource_format: csv\n");
+ expect(page).toContain(
+ `\nsource_content_hash: ${hashBytes(new TextEncoder().encode(PARTS_CSV))}\n`,
+ );
+ expect(page).toContain("\ntable_delimiter: comma\n");
+ expect(page).toContain("\ntable_rows_rendered: 2\n");
+ expect(
+ page
+ .trimEnd()
+ .endsWith("## Table\n\n### Rows 1-2\n\n```table\nname | qty\nbolt | 4\nnut | 7\n```"),
+ ).toBe(true);
+ });
+
+ test("a TSV source and a truncated table name what they are", async () => {
+ const rows = Array.from({ length: 1_001 }, (_row, i) => `${i}\tx`).join("\n");
+ const f = fixture({ [TSV_PATH]: `id\tv\n${rows}\n` });
+ const result = await ingest(f, TSV_PATH);
+ expect(result["table"]).toMatchObject({
+ rendered: true,
+ format: "tsv",
+ delimiter: "tab",
+ rows: 1_001,
+ rows_rendered: 1_000,
+ truncated: ["rows"],
+ });
+ const page = pageOf(f, result);
+ expect(frontmatterKeys(page)).toContain("table_truncated");
+ expect(page.trimEnd().endsWith("Rendered 1000 of 1001 rows.")).toBe(true);
+ });
+
+ test("a refused table is data: the page and entities are still written", async () => {
+ const f = fixture({ [CSV_PATH]: 'name,qty\n"open,4\n' });
+ const result = await ingest(f, CSV_PATH);
+ expect(result["table"]).toEqual({
+ rendered: false,
+ format: "csv",
+ reason: "malformed-quoting",
+ detail: "2",
+ });
+ expect(result["entities_created"]).toHaveLength(1);
+ expect(pageOf(f, result)).not.toContain("## Table");
+ });
+
+ test("a text source carries no table key", async () => {
+ const f = fixture({ "Clips/notes.md": "# Notes\n" });
+ const result = await ingest(f, "Clips/notes.md");
+ expect("table" in result).toBe(false);
+ expect("parts" in result).toBe(false);
+ expect(pageOf(f, result)).not.toContain("source_format");
+ });
+});
+
+describe("reach", () => {
+ test("a hidden .csv-named source and an absent one answer identically at remote reach", async () => {
+ const hidden = fixture({ [CSV_PATH]: PRIVATE_BODY });
+ const absent = fixture({});
+ const hiddenResult = await ingest(hidden, CSV_PATH);
+ const absentResult = await ingest(absent, CSV_PATH);
+
+ expect(JSON.stringify(hiddenResult)).toBe(JSON.stringify(absentResult));
+ expect(hiddenResult["table"]).toEqual({
+ rendered: false,
+ format: "csv",
+ reason: "source-not-local",
+ });
+ expect(normalisedPages(hidden.vault)).toEqual(normalisedPages(absent.vault));
+ const page = pageOf(hidden, hiddenResult);
+ expect(page).not.toContain("source_content_hash");
+ expect(page).not.toContain("source_format");
+ expect(page).not.toContain("## Table");
+ expect(page).not.toContain("4711");
+ });
+
+ test("a URL source answers source-not-local without a digest", async () => {
+ const f = fixture({});
+ const result = await ingest(f, "https://example.com/parts.csv");
+ expect(result["table"]).toEqual({ rendered: false, format: "csv", reason: "source-not-local" });
+ expect(pageOf(f, result)).not.toContain("source_content_hash");
+ });
+
+ test("at local reach the same file is read and rendered", async () => {
+ const f = fixture({ [CSV_PATH]: PRIVATE_BODY }, TRANSPORT_REACH.local);
+ const result = await ingest(f, CSV_PATH);
+ expect(result["table"]).toMatchObject({ rendered: true, format: "csv" });
+ expect(pageOf(f, result)).toContain("source_content_hash");
+ });
+});
diff --git a/tests/mcp/log-readers-reach-round3.test.ts b/tests/mcp/log-readers-reach-round3.test.ts
new file mode 100644
index 00000000..7767a09e
--- /dev/null
+++ b/tests/mcp/log-readers-reach-round3.test.ts
@@ -0,0 +1,324 @@
+/**
+ * The remaining readers of Brain log content answer at the caller's reach.
+ *
+ * The rule is the one tests/mcp/log-readers-reach.test.ts pins for the
+ * analytics, brief and trace readers: below local reach an event naming
+ * a record the caller cannot read is dropped, a dream shared with such a
+ * record keeps its readable transitions, and a retired record is judged
+ * under its `pref-` spelling too. The rows here cover the raw log
+ * resource, `brain_query`'s log branches, the backlinks resource, the
+ * preference readers, the doctor and the MOC audit.
+ *
+ * The vault pair is tests/helpers/reach-log-fixture.ts. A server with no
+ * reach minted is a remote caller; each row also carries a local control
+ * proving the reserved record is there to hide.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { appendLogEvent } from "../../src/core/brain/log.ts";
+import { brainDirs } from "../../src/core/brain/paths.ts";
+import { BRAIN_LOG_EVENT_KIND } from "../../src/core/brain/types.ts";
+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
+import { REMOTE_DENY_VISIBILITY_TOKEN } from "../../src/core/graph/visibility.ts";
+import {
+ buildReachLogFixture,
+ maskVolatile,
+ PRIVATE_PATH,
+ PRIVATE_SLUG,
+ reachServer,
+ type ReachLogFixture as Fixture,
+ RETIRED_SLUG,
+ SHARED_SLUG,
+} from "../helpers/reach-log-fixture.ts";
+
+/** Relative ages ("1h ago") and durations, which tick between the two builds. */
+const AGE_RE = /\b\d+(?:ms|s|m|h|d)\b/g;
+/** A log heading's time of day, which ticks between the two builds. */
+const HEADING_TIME_RE = /\b\d{2}:\d{2}:\d{2}Z/g;
+/** A tool the product no longer ships, which the doctor reports when a page names it. */
+const REMOVED_TOOL = "brain_digest";
+/** The evidence artifact of the malformed-range row: a reserved note, by path. */
+const HIDDEN_NOTE = "Notes/zzhiddendoc";
+
+const bases: string[] = [];
+const savedConfig = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+ if (savedConfig === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfig;
+});
+
+function fixture(withPrivate: boolean): Fixture {
+ const base = mkdtempSync(join(tmpdir(), "o2b-log-readers-reach-r3-"));
+ bases.push(base);
+ return buildReachLogFixture(base, withPrivate);
+}
+
+/** Append one event to the fixture's calendar day, at a fixed time both vaults share. */
+function logOnDay(f: Fixture, eventType: string, body: Record<string, unknown>): void {
+ appendLogEvent(
+ f.vault,
+ { timestamp: `${f.date}T00:00:00Z`, eventType, body } as Parameters<typeof appendLogEvent>[1],
+ { deviceId: "" },
+ );
+}
+
+/**
+ * A dream confirming the public preference and, in vault A, the retired
+ * reserved record under its `pref-` spelling: only `ret-bygone` is on
+ * disk, so the transition names no page unless both spellings are asked.
+ */
+function retiredOnlyDream(withPrivate: boolean): (f: Fixture) => Promise<void> {
+ return async (f) => {
+ const confirmed = [`[[pref-${SHARED_SLUG}|rule]]`];
+ if (withPrivate) confirmed.push(`[[pref-${RETIRED_SLUG}|rule]]`);
+ logOnDay(f, BRAIN_LOG_EVENT_KIND.dream, { run_id: "run-retired-only", confirmed });
+ };
+}
+
+/**
+ * Vault A only: evidence on the public preference whose artifact is the
+ * retired reserved record under its `pref-` spelling.
+ */
+function retiredArtifactEvidence(withPrivate: boolean): (f: Fixture) => Promise<void> {
+ return async (f) => {
+ if (!withPrivate) return;
+ logOnDay(f, BRAIN_LOG_EVENT_KIND.applyEvidence, {
+ preference: `[[pref-${SHARED_SLUG}]]`,
+ artifact: `[[pref-${RETIRED_SLUG}|rule]]`,
+ result: "applied",
+ });
+ };
+}
+
+async function call(
+ f: Fixture,
+ tool: string,
+ args: Record<string, unknown>,
+ reach?: TransportReach,
+): Promise<unknown> {
+ try {
+ const result = (await reachServer(f, reach).callTool(tool, args)) as Record<string, unknown>;
+ return result["structuredContent"] ?? result;
+ } catch (err) {
+ // A refusal is an answer too: its message is compared like any other.
+ return (err as Error).message;
+ }
+}
+
+/** A resource read through the server's JSON-RPC surface: the result or the error frame. */
+async function read(f: Fixture, uri: string, reach?: TransportReach): Promise<unknown> {
+ const response = await reachServer(f, reach).handleRequest({
+ jsonrpc: "2.0",
+ id: 1,
+ method: "resources/read",
+ params: { uri },
+ });
+ return response === null ? null : (response.result ?? response.error);
+}
+
+function normalise(f: Fixture, value: unknown): string {
+ return maskVolatile(f, value).replace(AGE_RE, "<AGE>").replace(HEADING_TIME_RE, "<HMS>");
+}
+
+type Ask = (f: Fixture, reach?: TransportReach) => Promise<unknown>;
+type Prepare = (withPrivate: boolean) => (f: Fixture) => Promise<void>;
+
+/** The remote answers of both vaults, normalised, plus vault A's local answer. */
+async function abRow(
+ ask: Ask,
+ prepare: Prepare = () => async () => {},
+): Promise<{ withheld: string; absent: string; local: string }> {
+ const a = fixture(true);
+ const b = fixture(false);
+ await prepare(true)(a);
+ await prepare(false)(b);
+ return {
+ withheld: normalise(a, await ask(a)),
+ absent: normalise(b, await ask(b)),
+ local: normalise(a, await ask(a, TRANSPORT_REACH.local)),
+ };
+}
+
+const logResource: Ask = (f, reach) => read(f, `osb://log/${f.date}`, reach);
+const querySince: Ask = (f, reach) =>
+ call(f, "brain_query", { since: `${f.date}T00:00:00Z` }, reach);
+const queryTopic: Ask = (f, reach) => call(f, "brain_query", { topic: SHARED_SLUG }, reach);
+
+describe("a dream shared with a reserved preference is kept for its public transition", () => {
+ for (const [label, ask] of [
+ ["osb://log/{date}", logResource],
+ ["brain_query since", querySince],
+ ] as const) {
+ test(`${label} shows the shared dream remotely as the vault that never had the record`, async () => {
+ const row = await abRow(ask);
+ expect(row.withheld).toBe(row.absent);
+ expect(row.withheld).toContain(`pref-${SHARED_SLUG}`);
+ expect(row.withheld).not.toContain(PRIVATE_SLUG);
+ // Control: a local caller sees the dream naming the reserved record.
+ expect(row.local).toContain(`pref-${PRIVATE_SLUG}`);
+ });
+ }
+});
+
+describe("a retired reserved record is judged under its pref- spelling", () => {
+ for (const [label, ask, prepare] of [
+ ["osb://log/{date}", logResource, retiredOnlyDream],
+ ["brain_query since", querySince, retiredOnlyDream],
+ ["brain_query topic", queryTopic, retiredArtifactEvidence],
+ ] as const) {
+ test(`${label} names no retired reserved record remotely`, async () => {
+ const row = await abRow(ask, prepare);
+ expect(row.withheld).toBe(row.absent);
+ expect(row.withheld).not.toContain(RETIRED_SLUG);
+ expect(row.local).toContain(`pref-${RETIRED_SLUG}|rule`);
+ });
+ }
+
+ test("osb://backlinks/pref-bygone answers as an absent page", async () => {
+ const row = await abRow((f, reach) => read(f, `osb://backlinks/pref-${RETIRED_SLUG}`, reach));
+ expect(row.withheld).toBe(row.absent);
+ expect(row.local).toContain("log-dream");
+ });
+});
+
+// The fixture's retired page carries no `retired_by`, so the preference
+// parser refuses it and names the page in its error.
+describe("a reserved preference page that fails to parse answers as an absent one", () => {
+ for (const [label, ask] of [
+ [
+ "brain_query preference",
+ (f: Fixture, reach?: TransportReach) =>
+ call(f, "brain_query", { preference: `pref-${RETIRED_SLUG}` }, reach),
+ ],
+ [
+ "osb://preference/{id}",
+ (f: Fixture, reach?: TransportReach) =>
+ read(f, `osb://preference/pref-${RETIRED_SLUG}`, reach),
+ ],
+ ] as const) {
+ test(`${label} on the retired reserved record`, async () => {
+ const row = await abRow(ask);
+ expect(row.withheld).toBe(row.absent);
+ expect(row.withheld).not.toContain(`ret-${RETIRED_SLUG}`);
+ // Control: a local caller reaches the parser, which names the page.
+ expect(row.local).toContain(`ret-${RETIRED_SLUG}`);
+ });
+ }
+});
+
+/**
+ * Vault A's reserved preference names a removed tool and carries evidence
+ * whose artifact range is malformed, and the retired reserved record
+ * carries evidence logged under its `pref-` spelling; vault B carries none
+ * of them.
+ */
+function doctorFindings(withPrivate: boolean): (f: Fixture) => Promise<void> {
+ return async (f) => {
+ if (!withPrivate) return;
+ const abs = join(f.vault, PRIVATE_PATH);
+ writeFileSync(abs, `${readFileSync(abs, "utf8")}\nSee \`${REMOVED_TOOL}\`.\n`);
+ logOnDay(f, BRAIN_LOG_EVENT_KIND.applyEvidence, {
+ preference: `[[pref-${PRIVATE_SLUG}]]`,
+ artifact: `[[${HIDDEN_NOTE}:9-x]]`,
+ result: "applied",
+ });
+ // Evidence on the retired reserved record, logged under `pref-`.
+ logOnDay(f, BRAIN_LOG_EVENT_KIND.applyEvidence, {
+ preference: `[[pref-${RETIRED_SLUG}]]`,
+ artifact: `[[Notes/${RETIRED_SLUG}-applied]]`,
+ result: "applied",
+ });
+ };
+}
+
+/** The doctor's warnings and errors, sorted (their order follows the checks). */
+function findings(normalised: string): ReadonlyArray<string> {
+ const report = JSON.parse(normalised) as { warnings?: unknown[]; errors?: unknown[] };
+ return [...(report.warnings ?? []), ...(report.errors ?? [])]
+ .map((w) => JSON.stringify(w))
+ .toSorted();
+}
+
+describe("brain_doctor names no reserved record below local reach", () => {
+ test("every finding agrees with the vault that never had the record", async () => {
+ const row = await abRow((f, reach) => call(f, "brain_doctor", {}, reach), doctorFindings);
+ expect(findings(row.withheld)).toEqual(findings(row.absent));
+ expect(row.withheld).not.toContain(PRIVATE_SLUG);
+ expect(row.withheld).not.toContain(RETIRED_SLUG);
+ expect(row.withheld).not.toContain(HIDDEN_NOTE);
+ // Control: locally the orphan evidence, the malformed range and the
+ // removed-tool mention of the reserved preference are all reported.
+ const local = findings(row.local).join("\n");
+ for (const code of ["orphan-evidence", "malformed-evidence-range", "removed-tool-reference"]) {
+ expect(local).toContain(code);
+ }
+ expect(local).toContain(`${PRIVATE_SLUG}-applied`);
+ expect(local).toContain(`${RETIRED_SLUG}-applied`);
+ expect(local).toContain(HIDDEN_NOTE);
+ expect(local).toContain(`pref-${PRIVATE_SLUG}.md`);
+ });
+});
+
+describe("brain_moc_audit answers a reserved hub as an absent one", () => {
+ test(`id=pref-${PRIVATE_SLUG}`, async () => {
+ const row = await abRow((f, reach) =>
+ call(f, "brain_moc_audit", { id: `pref-${PRIVATE_SLUG}` }, reach),
+ );
+ expect(row.withheld).toBe(row.absent);
+ expect(row.withheld).toContain("hub note not found");
+ // Control: a local caller reaches the hub and is told it is no MOC.
+ expect(row.local).toContain("not a MOC");
+ });
+});
+
+/** Public preferences a public hub links to, enough of them for the hub to read as a MOC. */
+const HUB_MEMBERS = ["alpha", "beta", "gamma", "delta", "epsilon"].map((name) => `pref-${name}`);
+const HUB_ID = "pref-hub";
+/** A reserved member of the public hub, with no other backlink. */
+const VEILED_ID = "pref-veiled";
+/** A target the hub and the reserved member both link to, with no page. */
+const MISSING_ID = "pref-ghost";
+
+/** A minimal page in the preferences folder, the folder the audit resolves ids in. */
+function writePage(f: Fixture, id: string, body: string, frontmatter: string[] = []): void {
+ writeFileSync(
+ join(brainDirs(f.vault).preferences, `${id}.md`),
+ ["---", "kind: brain-preference", `id: ${id}`, ...frontmatter, "---", "", body, ""].join("\n"),
+ );
+}
+
+/** The `referenceCount` the audit reports for the missing target. */
+function missingReferences(normalised: string): number | undefined {
+ const report = JSON.parse(normalised) as {
+ candidate_missing: ReadonlyArray<{ id: string; referenceCount: number }>;
+ };
+ return report.candidate_missing.find((m) => m.id === MISSING_ID)?.referenceCount;
+}
+
+test("brain_moc_audit neither lists nor reads a reserved member of a public hub", async () => {
+ const a = fixture(true);
+ for (const id of HUB_MEMBERS) writePage(a, id, `Member ${id}.`);
+ writePage(a, VEILED_ID, `Reserved, linking [[${MISSING_ID}]].`, [
+ `visibility: [${REMOTE_DENY_VISIBILITY_TOKEN}]`,
+ ]);
+ const links = [...HUB_MEMBERS, VEILED_ID, MISSING_ID].map((id) => `- [[${id}]]`);
+ writePage(a, HUB_ID, links.join("\n"));
+ const remote = normalise(a, await call(a, "brain_moc_audit", { id: HUB_ID }));
+ const local = normalise(
+ a,
+ await call(a, "brain_moc_audit", { id: HUB_ID }, TRANSPORT_REACH.local),
+ );
+ expect(remote).toContain(HUB_MEMBERS[0]!);
+ expect(remote).not.toContain(VEILED_ID);
+ // The reserved member's own link is not counted: only the hub's is.
+ expect(missingReferences(remote)).toBe(1);
+ // Control: a local caller is shown the reserved member, and its link counts.
+ expect(local).toContain(VEILED_ID);
+ expect(missingReferences(local)).toBe(2);
+});
diff --git a/tests/mcp/log-readers-reach.test.ts b/tests/mcp/log-readers-reach.test.ts
new file mode 100644
index 00000000..febe2f6f
--- /dev/null
+++ b/tests/mcp/log-readers-reach.test.ts
@@ -0,0 +1,332 @@
+/**
+ * The readers of the Brain log answer at the caller's reach.
+ *
+ * The daily log names preference ids, retired ids and evidence artifacts
+ * in its event bodies. Below local reach the log page itself is not
+ * served by a generic page reader (search), and every reader that renders
+ * log events (analytics timeline, belief evolution, concept synthesis,
+ * the today dashboard, the digest view, the event trace, backlinks and the
+ * doctor) answers as if a record the caller cannot read had never been
+ * logged: an event naming one is dropped, and a dream is kept while one
+ * of its transitions is readable, showing only those.
+ *
+ * The vault pair is tests/helpers/reach-log-fixture.ts. A server with no
+ * reach minted is a remote caller; each row also carries a local control
+ * proving the reserved record is there to hide.
+ */
+
+import { afterEach, describe, expect, test } from "bun:test";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { BRAIN_LOG_EVENT_KIND } from "../../src/core/brain/types.ts";
+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";
+import { indexVault, resolveSearchConfig } from "../../src/core/search/index.ts";
+import {
+ buildReachLogFixture,
+ maskVolatile,
+ PRIVATE_SLUG,
+ reachServer,
+ type ReachLogFixture as Fixture,
+ RETIRED_SLUG,
+ SHARED_SLUG,
+} from "../helpers/reach-log-fixture.ts";
+
+/** Relative ages ("1h ago") and durations, which tick between the two builds. */
+const AGE_RE = /\b\d+(?:ms|s|m|h|d)\b/g;
+
+const bases: string[] = [];
+const savedConfig = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+
+afterEach(() => {
+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });
+ if (savedConfig === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfig;
+});
+
+function fixture(withPrivate: boolean): Fixture {
+ const base = mkdtempSync(join(tmpdir(), "o2b-log-readers-reach-"));
+ bases.push(base);
+ return buildReachLogFixture(base, withPrivate);
+}
+
+async function call(
+ f: Fixture,
+ tool: string,
+ args: Record<string, unknown>,
+ reach?: TransportReach,
+): Promise<unknown> {
+ const result = (await reachServer(f, reach).callTool(tool, args)) as Record<string, unknown>;
+ return result["structuredContent"] ?? result;
+}
+
+/** Vault paths, handles, instants and ages replaced. */
+function normalise(f: Fixture, value: unknown): string {
+ return maskVolatile(f, value).replace(AGE_RE, "<AGE>");
+}
+
+/** The remote answers of both vaults, normalised, plus vault A's local answer. */
+async function abRow(
+ tool: string,
+ args: (f: Fixture) => Record<string, unknown>,
+ prepare: (f: Fixture) => Promise<void> = async () => {},
+): Promise<{ withheld: string; absent: string; local: string }> {
+ const a = fixture(true);
+ const b = fixture(false);
+ await prepare(a);
+ await prepare(b);
+ return {
+ withheld: normalise(a, await call(a, tool, args(a))),
+ absent: normalise(b, await call(b, tool, args(b))),
+ local: normalise(a, await call(a, tool, args(a), TRANSPORT_REACH.local)),
+ };
+}
+
+async function indexed(f: Fixture): Promise<void> {
+ await indexVault(resolveSearchConfig({ vault: f.vault, configPath: f.configPath }), {
+ force: true,
+ });
+}
+
+/** The page paths a search answer names, sorted (scores depend on the corpus). */
+function searchPaths(normalised: string): ReadonlyArray<string> {
+ const results = (JSON.parse(normalised) as { results?: ReadonlyArray<{ path: string }> }).results;
+ return (results ?? []).map((r) => r.path).toSorted();
+}
+
+describe("brain_search does not serve the daily log below local reach", () => {
+ for (const word of [PRIVATE_SLUG, RETIRED_SLUG]) {
+ test(`query '${word}' answers remotely as if the record was never logged`, async () => {
+ const row = await abRow("brain_search", () => ({ query: word }), indexed);
+ expect(row.withheld).toBe(row.absent);
+ // Control: a local caller still finds the log page naming the record.
+ expect(row.local).toContain("Brain/log/");
+ expect(row.local).toContain(word);
+ });
+ }
+
+ test("query 'rule' names the same pages remotely and the log page only locally", async () => {
+ const row = await abRow("brain_search", () => ({ query: "rule" }), indexed);
+ expect(searchPaths(row.withheld)).toEqual(searchPaths(row.absent));
+ expect(row.withheld).not.toContain("Brain/log/");
+ expect(row.withheld).not.toContain(PRIVATE_SLUG);
+ expect(row.withheld).not.toContain(RETIRED_SLUG);
+ expect(searchPaths(row.local).some((p) => p.startsWith("Brain/log/"))).toBe(true);
+ });
+});
+
+/** Every row below: the remote answers agree, and the local control names the record. */
+async function expectAnswersAsAbsent(
+ tool: string,
+ args: (f: Fixture) => Record<string, unknown>,
+ localNames: ReadonlyArray<string>,
+ prepare?: (f: Fixture) => Promise<void>,
+): Promise<{ withheld: string; local: string }> {
+ const row = await abRow(tool, args, prepare);
+ expect(row.withheld).toBe(row.absent);
+ // A row whose own argument names the record echoes it as its target.
+ const named = JSON.stringify(args({ date: "" } as Fixture));
+ for (const slug of [PRIVATE_SLUG, RETIRED_SLUG]) {
+ if (!named.includes(slug)) expect(row.withheld).not.toContain(slug);
+ }
+ for (const name of localNames) expect(row.local).toContain(name);
+ return row;
+}
+
+describe("brain_analytics answers at the caller's reach", () => {
+ test("view=timeline with no filter", async () => {
+ const row = await expectAnswersAsAbsent("brain_analytics", () => ({ view: "timeline" }), [
+ `pref-${PRIVATE_SLUG}`,
+ `ret-${RETIRED_SLUG}`,
+ `Notes/${PRIVATE_SLUG}-violated`,
+ ]);
+ // Not vacuous: the public preference's evidence and the shared dream are listed.
+ expect(row.withheld).toContain(`Notes/${SHARED_SLUG}-applied`);
+ expect(row.withheld).toContain(BRAIN_LOG_EVENT_KIND.dream);
+ });
+
+ test("view=timeline since a date", async () => {
+ await expectAnswersAsAbsent("brain_analytics", (f) => ({ view: "timeline", since: f.date }), [
+ `pref-${PRIVATE_SLUG}`,
+ ]);
+ });
+
+ test("view=timeline for the reserved preference's own id", async () => {
+ await expectAnswersAsAbsent(
+ "brain_analytics",
+ () => ({ view: "timeline", pref_id: `pref-${PRIVATE_SLUG}` }),
+ [`Notes/${PRIVATE_SLUG}-applied`],
+ );
+ });
+
+ for (const args of [
+ { view: "belief_evolution", pref_id: `pref-${PRIVATE_SLUG}` },
+ { view: "belief_evolution", topic: PRIVATE_SLUG },
+ ]) {
+ test(`view=belief_evolution ${JSON.stringify(args)}`, async () => {
+ await expectAnswersAsAbsent("brain_analytics", () => args, [`Notes/${PRIVATE_SLUG}-`]);
+ });
+ }
+
+ test("view=belief_evolution for the retired record under its pref- spelling", async () => {
+ await expectAnswersAsAbsent(
+ "brain_analytics",
+ () => ({ view: "belief_evolution", pref_id: `pref-${RETIRED_SLUG}` }),
+ [`[[pref-${RETIRED_SLUG}|rule]]`],
+ );
+ });
+
+ test("view=belief_evolution for the public preference keeps its rows", async () => {
+ const row = await expectAnswersAsAbsent(
+ "brain_analytics",
+ () => ({ view: "belief_evolution", pref_id: `pref-${SHARED_SLUG}` }),
+ [`Notes/${SHARED_SLUG}-applied`],
+ );
+ expect(row.withheld).toContain(`Notes/${SHARED_SLUG}-applied`);
+ });
+
+ for (const id of [`pref-${PRIVATE_SLUG}`, `pref-${RETIRED_SLUG}`]) {
+ test(`view=concept_synthesis id=${id}`, async () => {
+ const row = await expectAnswersAsAbsent(
+ "brain_analytics",
+ () => ({ view: "concept_synthesis", id, include_unlinked: true }),
+ ["linkers", "log-"],
+ );
+ expect(JSON.parse(row.withheld)).toMatchObject({ linkers: [], unlinked_mentions: [] });
+ });
+ }
+});
+
+describe("brain_brief view=today answers at the caller's reach", () => {
+ test("the recent activity names no reserved record and counts only what it shows", async () => {
+ const row = await expectAnswersAsAbsent("brain_brief", () => ({ view: "today" }), [
+ `pref=pref-${PRIVATE_SLUG}`,
+ "confirmed=2",
+ ]);
+ expect(row.withheld).toContain(`artifact=Notes/${SHARED_SLUG}-applied`);
+ expect(row.withheld).toContain("confirmed=1");
+ });
+
+ test("a limit counts after the filter, not before", async () => {
+ await expectAnswersAsAbsent("brain_brief", () => ({ view: "today", limit: 2 }), [
+ `pref-${PRIVATE_SLUG}`,
+ ]);
+ });
+});
+
+describe("a dream shared with a reserved record is kept, showing only readable transitions", () => {
+ test("brain_brief view=digest counts the shared dream for its public transition", async () => {
+ const row = await expectAnswersAsAbsent(
+ "brain_brief",
+ () => ({ view: "digest", format: "json" }),
+ [`pref-${PRIVATE_SLUG}`],
+ );
+ // Not vacuous: the agent summary counts the public evidence and the shared dream.
+ const digest = JSON.parse((JSON.parse(row.withheld) as { content: string }).content) as {
+ agent_summary: ReadonlyArray<{ total_events: number }>;
+ };
+ expect(digest.agent_summary.map((a) => a.total_events)).toEqual([2]);
+ });
+
+ for (const args of [
+ {},
+ { kind: BRAIN_LOG_EVENT_KIND.dream },
+ { kind: BRAIN_LOG_EVENT_KIND.applyEvidence },
+ ]) {
+ test(`brain_event_trace ${JSON.stringify(args)}`, async () => {
+ const row = await expectAnswersAsAbsent(
+ "brain_event_trace",
+ (f) => ({ date: f.date, ...args }),
+ [
+ args.kind === BRAIN_LOG_EVENT_KIND.applyEvidence
+ ? `Notes/${PRIVATE_SLUG}-`
+ : PRIVATE_SLUG,
+ ],
+ );
+ expect(row.withheld).toContain(`pref-${SHARED_SLUG}`);
+ });
+ }
+});
+
+/** A readable preference page that contests `contested` by id. */
+function contester(f: Fixture, slug: string, contested: string): void {
+ writeFileSync(
+ join(f.vault, "Brain", "preferences", `pref-${slug}.md`),
+ [
+ "---",
+ "kind: brain-preference",
+ `id: pref-${slug}`,
+ `topic: ${slug}`,
+ `principle: Rule ${slug}.`,
+ `contradicts: [${contested}]`,
+ "---",
+ "",
+ "Contests a rule.",
+ "",
+ ].join("\n"),
+ );
+}
+
+describe("brain_claims answers at the caller's reach", () => {
+ for (const operation of ["current", "history"]) {
+ test(`operation=${operation} lists no reserved record and counts only what it lists`, async () => {
+ const row = await expectAnswersAsAbsent("brain_claims", () => ({ operation }), [
+ `pref-${PRIVATE_SLUG}`,
+ ]);
+ expect(row.withheld).toContain(`pref-${SHARED_SLUG}`);
+ });
+ }
+
+ test("a claim contesting a retired reserved record is dropped, as one contesting a reserved record is", async () => {
+ const f = fixture(true);
+ contester(f, "contests-retired", `pref-${RETIRED_SLUG}`);
+ contester(f, "contests-reserved", `pref-${PRIVATE_SLUG}`);
+ const remote = normalise(f, await call(f, "brain_claims", { operation: "history" }));
+ const local = normalise(
+ f,
+ await call(f, "brain_claims", { operation: "history" }, TRANSPORT_REACH.local),
+ );
+ expect(remote).toContain(`pref-${SHARED_SLUG}`);
+ expect(remote).not.toContain("contests-reserved");
+ expect(remote).not.toContain("contests-retired");
+ // Control: a local caller lists both contesting claims.
+ expect(local).toContain("contests-reserved");
+ expect(local).toContain("contests-retired");
+ });
+});
+
+/** The doctor's broken-backlinks findings in one normalised answer. */
+function brokenBacklinks(normalised: string): ReadonlyArray<{ code: string }> {
+ return (JSON.parse(normalised) as { warnings: ReadonlyArray<{ code: string }> }).warnings.filter(
+ (w) => w.code === "broken-backlinks",
+ );
+}
+
+describe("a retired record is judged under its pref- spelling too", () => {
+ test("brain_backlinks id=pref-bygone answers as an absent page", async () => {
+ await expectAnswersAsAbsent("brain_backlinks", () => ({ id: `pref-${RETIRED_SLUG}` }), [
+ "log-dream",
+ ]);
+ });
+
+ for (const args of [{ kind: BRAIN_LOG_EVENT_KIND.applyEvidence }, {}]) {
+ test(`brain_event_trace ${JSON.stringify(args)} drops evidence on the retired record`, async () => {
+ const row = await expectAnswersAsAbsent(
+ "brain_event_trace",
+ (f) => ({ date: f.date, ...args }),
+ [`Notes/${RETIRED_SLUG}-applied`],
+ );
+ expect(row.withheld).toContain(`Notes/${SHARED_SLUG}-applied`);
+ });
+ }
+
+ test("brain_doctor names no broken backlink to the retired record", async () => {
+ // Only the broken-backlinks findings are compared: the orphan-evidence
+ // findings name evidence artifacts by basename and are outside this row.
+ const row = await abRow("brain_doctor", () => ({}));
+ expect(brokenBacklinks(row.withheld)).toEqual(brokenBacklinks(row.absent));
+ expect(JSON.stringify(brokenBacklinks(row.withheld))).not.toContain(RETIRED_SLUG);
+ expect(JSON.stringify(brokenBacklinks(row.local))).toContain(`[[pref-${RETIRED_SLUG}]]`);
+ });
+});
diff --git a/tests/mcp/ner-tool.test.ts b/tests/mcp/ner-tool.test.ts
index 54c24dac..f86c04a1 100644
--- a/tests/mcp/ner-tool.test.ts
+++ b/tests/mcp/ner-tool.test.ts
@@ -205,11 +205,12 @@ describe("brain_intake_entities - a page withheld at the caller's reach", () =>
expect(page).not.toContain(SOURCE_CONTENT_HASH_FRONTMATTER_KEY);
});
- test("at local reach the same page is trusted and digested", async () => {
+ test("at local reach the same page is trusted, and no digest of it is recorded", async () => {
+ // The entity pages do not inherit the page's reservation, so its digest
+ // would be readable where the page is not.
const res = await intake(PRIVATE_PATH, { ...ctx, reach: TRANSPORT_REACH.local });
expect(res.trust).toBe(INTAKE_TRUST.trusted);
- expect(
- brainPageTexts(vault).some((page) => page.includes(SOURCE_CONTENT_HASH_FRONTMATTER_KEY)),
- ).toBe(true);
+ for (const page of brainPageTexts(vault))
+ expect(page).not.toContain(SOURCE_CONTENT_HASH_FRONTMATTER_KEY);
});
});
diff --git a/tests/mcp/recall-gate-turn-id.test.ts b/tests/mcp/recall-gate-turn-id.test.ts
new file mode 100644
index 00000000..904a38c8
--- /dev/null
+++ b/tests/mcp/recall-gate-turn-id.test.ts
@@ -0,0 +1,108 @@
+/**
+ * `brain_recall_gate` accepts the `turn_id` correlation argument its
+ * sibling `brain_context_pack` already declares, and records it on the
+ * `gate_telemetry` record when telemetry is on. The Hermes provider sends
+ * it on every prefetch; before the gate declared it, the closed input
+ * schema refused the whole call.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { CONTINUITY_TURN_ID_KEY } from "../../src/core/brain/continuity/types.ts";
+import { listGateTelemetry } from "../../src/core/brain/gate-telemetry.ts";
+import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";
+import { MCPServer } from "../../src/mcp/server.ts";
+
+/** The declared bound, shared with `session_id` and the context pack's `turn_id`. */
+const TURN_ID_MAX_CHARS = 512;
+const TURN_ID = "turn-7";
+const PROMPT = "what did we decide about the index?";
+
+let tmp: string;
+let vault: string;
+let configPath: string;
+const savedConfig = process.env["OPEN_SECOND_BRAIN_CONFIG"];
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-gate-turn-id-"));
+ vault = join(tmp, "vault");
+ mkdirSync(join(vault, "Brain"), { recursive: true });
+ configPath = join(tmp, "config.yaml");
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = configPath;
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+ if (savedConfig === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];
+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfig;
+});
+
+function server(telemetry: boolean): MCPServer {
+ writeFileSync(
+ configPath,
+ `vault: ${JSON.stringify(vault)}\n${telemetry ? 'recall_gate_telemetry: "true"\n' : ""}`,
+ );
+ return new MCPServer({ vault, configPath }, { reach: TRANSPORT_REACH.local });
+}
+
+test("the gate accepts turn_id and records it on the gate telemetry record", async () => {
+ const result = await server(true).callTool("brain_recall_gate", {
+ prompt: PROMPT,
+ session_id: "sess-1",
+ turn_id: TURN_ID,
+ });
+ expect(result.isError).toBe(false);
+ const records = listGateTelemetry(vault);
+ expect(records).toHaveLength(1);
+ expect(records[0]!.payload[CONTINUITY_TURN_ID_KEY]).toBe(TURN_ID);
+ expect(records[0]!.payload["session_id"]).toBe("sess-1");
+});
+
+test("a call without turn_id records no turn_id key", async () => {
+ await server(true).callTool("brain_recall_gate", { prompt: PROMPT });
+ const records = listGateTelemetry(vault);
+ expect(records).toHaveLength(1);
+ expect(CONTINUITY_TURN_ID_KEY in records[0]!.payload).toBe(false);
+});
+
+test("with telemetry off the gate accepts turn_id and writes nothing", async () => {
+ const result = await server(false).callTool("brain_recall_gate", {
+ prompt: PROMPT,
+ turn_id: TURN_ID,
+ });
+ expect(result.isError).toBe(false);
+ expect(listGateTelemetry(vault)).toHaveLength(0);
+});
+
+test("a turn_id over the declared bound is refused", async () => {
+ const call = server(true).callTool("brain_recall_gate", {
+ prompt: PROMPT,
+ turn_id: "t".repeat(TURN_ID_MAX_CHARS + 1),
+ });
+ // The bound's own message: a closed schema that does not declare turn_id
+ // also names the argument ("unknown argument 'turn_id'"), so a bare
+ // /turn_id/ passes even where the argument was never accepted.
+ await expect(call).rejects.toThrow(`argument 'turn_id' exceeds ${TURN_ID_MAX_CHARS} characters`);
+ expect(listGateTelemetry(vault)).toHaveLength(0);
+});
+
+test("a turn_id at the declared bound is accepted", async () => {
+ const result = await server(true).callTool("brain_recall_gate", {
+ prompt: PROMPT,
+ turn_id: "t".repeat(TURN_ID_MAX_CHARS),
+ });
+ expect(result.isError).toBe(false);
+});
+
+test("a session_id over the declared bound is refused even with telemetry off", async () => {
+ const call = server(false).callTool("brain_recall_gate", {
+ prompt: PROMPT,
+ session_id: "s".repeat(TURN_ID_MAX_CHARS + 1),
+ });
+ await expect(call).rejects.toThrow(
+ `argument 'session_id' exceeds ${TURN_ID_MAX_CHARS} characters`,
+ );
+});
diff --git a/tests/mcp/visibility-matrix.test.ts b/tests/mcp/visibility-matrix.test.ts
index 82ffa0fd..e41e6ec4 100644
--- a/tests/mcp/visibility-matrix.test.ts
+++ b/tests/mcp/visibility-matrix.test.ts
@@ -503,13 +503,7 @@ const COVERED_TOOLS: ReadonlySet<string> = new Set(
* queued and `brain_trigger operation=terminal` does not.
*/
const STILL_NAMED_AT_REMOTE: ReadonlySet<string> = new Set([
- "brain_analytics view=belief_evolution",
- "brain_analytics view=concept_synthesis",
- "brain_analytics view=timeline",
"brain_anticipatory_context",
- "brain_claims operation=at",
- "brain_claims operation=current",
- "brain_claims operation=history",
"brain_context_pack",
"brain_retention",
"brain_scaffold_stub action=list",
diff --git a/tests/python/test_static_schemas.py b/tests/python/test_static_schemas.py
index f6a6656c..aa8e5aaf 100644
--- a/tests/python/test_static_schemas.py
+++ b/tests/python/test_static_schemas.py
@@ -386,6 +386,10 @@ def schema_violations(schema: dict, args: dict) -> list[str]:
return violations
+# The turn id the lifecycle drive hands to one prefetch.
+_PREFETCH_TURN_ID = "turn-2"
+
+
class ProviderPayloadConformanceTests(unittest.TestCase):
"""Payloads the provider BUILDS must satisfy the schema the server publishes.
@@ -439,6 +443,9 @@ class ProviderPayloadConformanceTests(unittest.TestCase):
provider.initialize("sess-1", hermes_home=home)
provider.system_prompt_block()
provider.prefetch("what did we decide", session_id="sess-1")
+ # Hermes passes the turn id on prefetch; the provider forwards it to
+ # both recall calls, so the gate's schema must declare it too.
+ provider.prefetch("and after that", session_id="sess-1", turn_id=_PREFETCH_TURN_ID)
provider.sync_turn("u1", "a1", session_id="sess-1")
provider.sync_turn("u2", "a2", session_id="sess-1")
provider._drain_captures()
@@ -469,6 +476,16 @@ class ProviderPayloadConformanceTests(unittest.TestCase):
# path must fail here rather than read as a clean run.
self.assertEqual(exercised, self._PROVIDER_BUILT)
+ def test_the_turn_id_reaches_both_recall_calls_and_conforms(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ calls = self._drive_lifecycle(tmp)
+ schemas = {s["name"]: s["inputSchema"] for s in STATIC_TOOL_SCHEMAS}
+ for tool in ("brain_recall_gate", "brain_context_pack"):
+ carried = [args for name, args in calls if name == tool and "turn_id" in args]
+ with self.subTest(tool=tool):
+ self.assertEqual([a["turn_id"] for a in carried], [_PREFETCH_TURN_ID])
+ self.assertEqual(schema_violations(schemas[tool], carried[0]), [])
+
def test_the_flush_payload_carries_both_turn_bounds_as_strings(self):
with tempfile.TemporaryDirectory() as tmp:
calls = self._drive_lifecycle(tmp)
diff --git a/uv.lock b/uv.lock
index 5198ff47..dc73d279 100644
--- a/uv.lock
+++ b/uv.lock
@@ -4,5 +4,5 @@ requires-python = ">=3.11"
[[package]]
name = "open-second-brain"
-version = "1.68.0"
+version = "1.69.0"
source = { editable = "." }