Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: storage lifecycle for long-lived vaults (v1.85.0) (#257)
Serg Eroshe948a0403faed
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex 812d071e..a2d2a897 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.84.1",+ "version": "1.85.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 7013d139..e82d8376 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.84.1",+ "version": "1.85.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex ec95b4d5..5485983a 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.85.0] - 2026-10-15++Open Second Brain 1.85.0 gives a long-lived vault the means to watch its own storage and to keep its append-only ledgers in shape: a measured per-surface byte census with a days-to-capacity forecast that warns in graduated tiers, background producers that pause behind a fifth maintenance gate before the wall while interactive writes keep working until a hard, validated refusal at the critical tier, receipt ledgers that compact behind an explicit preview-bound apply, near-duplicate document copies that retire behind the hygiene review gate keeping the newest copy, bulk removals that report per-file failures instead of aborting, and large appended sources that yield their new text without reprocessing whole. No cap is configured by default: an uncapped vault reads unbounded, nothing pauses and nothing refuses.++### Added++- **A measured storage report: `o2b brain storage` and `brain_storage`.** One builder feeds both surfaces, and each invocation takes one census and appends one `storage_usage` sample, so the forecast series accumulates precisely because the report was asked for. The report answers three questions in one pass: what the vault's storage holds per surface (`brain_tree`, and the dot-directories `.snapshots/`, `.artifacts/`, `.archives/`, `.payloads/`, `.state/`, plus the derived store with its WAL/SHM sidecars - each measured or refused by name), where it is heading against the configured cap - a tier (`ok`, `warn` or `critical`, or `unbounded`, `unmeasurable`, `no_forecast`) with the percent-used and days-to-cap figures it rests on, the trend linear over the recorded samples from two up - and which sample this invocation recorded, appended or named when it could not be. The verb is a report and never fails on a finding: a critical tier still exits 0, because enforcement lives at the gates that consume the tier. Below local reach the census is not taken: the answer names the reason and carries no figures.+- **A storage policy block.** `storage:` joins `Brain/_brain.yaml` through the config template and the resolver: `cap_bytes` plus four thresholds (`warn_percent`, `critical_percent`, `warn_days`, `critical_days`, defaulting to 80 and 95 percent of the cap and 30 and 7 days to it at the measured trend), with the critical band required to sit past the warn band - a larger percent, a smaller day horizon. An absent `cap_bytes` means the vault is unbounded; setting a cap is what turns on the forecast tiers, the capacity gate and the storage-capacity doctor check.+- **The maintenance lane's fifth gate.** A configured cap arms the lane's capacity gate, evaluated last because it is the most expensive - a fresh byte census of the whole vault - and it decides from that fresh census at decision time, never the stored aggregate: a sample taken yesterday says nothing about a vault an agent wrote into this morning. Only a measured critical tier closes it - an unmeasurable store leaves the gate open with its own `capacity:unmeasurable` journal row, the same honesty the host-pressure gate practices - and a closed gate skips the pass with its own `skipped:capacity` verdict, the journal row carrying the tier and the reading behind it while the skipped run still appends its usage sample first, so the forecast series keeps accumulating exactly while producers pause. The lane's front doors carry the verdict and its remedy, and where the caller has no local reach the reading stays behind: `brain_maintenance status` renders the tier word and the unmeasurable reason without the three byte-and-percent figures. `--force` (MCP `force`) bypasses the four soft gates - window, busy, pressure and capacity - and every streak refusal; it never bypasses the lease.+- **Interactive writes keep working until the wall.** The lane's gate pauses background producers; it does not pause a session. `brain_ingest_source` refuses with `capacity_exhausted` once a fresh census measures the vault at the critical tier, and the check sits in the validate phase before the write path opens, so a refused ingest leaves the vault byte-identical. The refusal names the tier and the remedy at every reach; the whole-vault figures ride only where the census itself may be read.+- **Receipt compaction: `o2b brain receipts` and `brain_receipts`.** The continuity and metrics ledgers grow one record per event and never shrink; compaction gives them a drain. `scan` (the CLI's default mode) inventories both ledgers read-only, per shard: records, foldable, protected, unparseable, aggregates, bytes. `compact` folds expired records into per-kind, per-month aggregates inside the named shards - the mode and the `family:name` shard spelling are one shared vocabulary, so the verb and the tool cannot drift - and never runs without explicit shard ids from a scan. A compact previews unless `--apply` (MCP `apply`) is passed, takes a recovery-point snapshot before a confirmed run, rewrites each shard atomically, and honors a count guard: `--expect N` (MCP `expect`) asserts exactly N records will fold and aborts without writing on a mismatch, and `--strict` refuses a confirmed compact that carries no guard at all. Only records older than the age threshold (`--older-than-days` / `older_than_days`, default 30) and in no protected class fold; the protected classes never fold whatever their age - the `source_invalidation` audit trail and the two context-compaction kinds, whose vocabulary a storage operation must not collide with - so their lines stay verbatim, and one `receipt_compaction` audit record closes a confirmed run. Below local reach compact is refused by name; scan stays readable at any reach, and the `receipt-ledger-growth` doctor check reports ledger growth the same way.+- **Near-duplicate document retirement.** The opt-in `document-dedup` hygiene detector nominates near-duplicate copies of ingest summary pages - a lexical similarity of 0.8, or an embedding similarity of 0.97 when the semantic config can reach a provider - and selects the newest `created_at` as the canonical, a page without `created_at` being ineligible rather than guessed at by mtime. `apply` routes a selected retire finding through the retirement core: every inbound `[[stale-copy]]` wikilink inside the detector's scope retargets to the canonical spelling (a same-basename page elsewhere in `Brain/` keeps its links untouched, and the report names the skip), the copy is removed, its content-manifest entry is dropped - which is the resurrect guard, since with the entry gone the source can only return as a deliberate full re-ingest - and one `source_invalidation` audit record per copy names the canonical, the retired id, the dropped manifest key and the retained source-archive hash. The whole span sits behind the destructive-snapshot gate, and it fails soft per copy: one copy's failure is collected and named while the remaining copies still retire.+- **Incremental extraction of appended sources.** The content manifest's entries gain an optional extraction watermark (`extracted_bytes`, `extracted_at` - how many leading bytes the last extracting pass covered, strictly additive, the manifest schema stays at 3, and an entry without the fields reprocesses fully the next time it actually changes, so nothing forces an eager mass reprocess). When a re-ingest meets a watermarked record, the delta proof first shows the growth is a pure append - the SHA-256 of the leading watermark bytes must still equal the recorded digest, read through one non-blocking descriptor - and only then is the tail read through a UTF-8-safe from-offset read whose start snaps forward past a partial multi-byte sequence, bounded by a ceiling; the appended `{fromByte, toByte}` range rides the ingest result and the CLI extract preview, so the new text can be extracted without reprocessing the whole source. A rewritten source - even one that happens to grow - fails the prefix proof and answers no delta, and the full reprocess is the safe direction. The extraction contract stays at version 2.+- **Two doctor checks join the lifecycle.** `storage-capacity` reports fullness for a capped vault - tier, percent and trend where the census can be taken, and below local reach it answers in the uncertain stream that the reading was not taken, no figures - and skips its own gate when the vault already measures critical. `receipt-ledger-growth` reads both receipt ledgers and reports their foldable share before compaction becomes the remedy worth running.++### Fixed++- **A bulk source removal no longer aborts on the first blocked file.** A confirmed `deleteBySource` - the cleanup `o2b brain forget-source` and the `brain_delete_by_source` tool run - removed derived entries one at a time and threw away the whole run on the first removal it could not complete, leaving the vault half-cleaned with no record of what stayed. Each derived entry and included original now removes through a collecting wrapper: the remaining entries still land, the result carries a `failed` array of path and message records naming what stayed on disk, and the `source_invalidation` audit counts both sides honestly - removed N, failed M with the paths. The partial-progress counts persist even when the manifest write itself fails and the run rethrows.+- **Page-dedup merges and maintenance collections fail soft per item.** A page-dedup `--apply` was a bare loop where one refused merge or one blocked wikilink write aborted every cluster queued behind it; the loop now collects each refused merge as a key-and-message pair, still applies the later clusters, reports the failures beside the merged and wikilink counts, and the CLI exits 1 when the failure list is non-empty. The maintenance action collection behind the doctor, digest and health surfaces absorbs one scan's failure as a named collected error while the remaining signals still collect, so a thin report because a scan refused is distinguishable from a thin report because the vault is healthy.++### Changed++- **Docs:** `docs/cli-reference.md` corrects the maintenance-lane prose to the five-gate reality and the actual `--force` semantics, gains a storage-lifecycle section documenting `o2b brain storage` and `o2b brain receipts` with their flags and exit codes, and adds the opt-in `document-dedup` detector to the hygiene row; `docs/mcp.md` raises its advertised tool count to 117 and adds `brain_storage` and `brain_receipts` to the Tool Highlights table; `docs/how-it-works.md` states the lane's five gates; `docs/metrics.md` documents the `storage_usage` sample surface; the README's "What is new" names the release.+## [1.84.1] - 2026-10-15### Fixed@@ -8519,6 +8543,7 @@ Hermes / Claude Code / Codex / OpenClaw configurations do not change.- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.85.0]: https://github.com/itechmeat/open-second-brain/compare/v1.84.1...v1.85.0[1.84.1]: https://github.com/itechmeat/open-second-brain/compare/v1.84.0...v1.84.1[1.84.0]: https://github.com/itechmeat/open-second-brain/compare/v1.83.0...v1.84.0[1.83.0]: https://github.com/itechmeat/open-second-brain/compare/v1.82.0...v1.83.0diff --git a/README.md b/README.mdindex 806bef90..bc4d1243 100644--- a/README.md+++ b/README.md@@ -123,7 +123,7 @@ The full router with readiness criteria is [`install.md`](install.md); native Wi## What is new-1.83.0 wires four shipped seams, each an opt-in that is byte-identical when off. Search can cap how many rows one source import session holds in the reranker-facing candidate window (`search_pool_max_per_session` / `OPEN_SECOND_BRAIN_SEARCH_POOL_MAX_PER_SESSION`, default `0`): the cap sits after the relevance floor and before the MMR diversity rerank and widens the pool rather than truncating it, so a long imported session stops crowding out everything else, while a vault without session frontmatter never sees a difference. `skills_attach` can order its offers by observed invocation usage (`skills_attach_usage_ordering` / `OPEN_SECOND_BRAIN_SKILLS_ATTACH_USAGE_ORDERING`, default off): a bounded composite, `score * (1 + 0.5 * weight)`, lifts a daily skill over a slightly hotter one-off without ever crossing the discriminating-term floor; plugin-qualified invocation names join discovered skills through a deterministic longest-suffix rule, private records never count, and the returned scores stay lexical. `o2b brain bench memory --suite longmemeval` accepts `--sample-seed` plus `--sample-cap` to score a seeded, question-type-proportional subset of the dataset instead of the whole file - deterministic given the seed, immune to dataset file order, receipt-reported (`sample_population` / `sample_selected` / `sample_strata`), and folded into the run identity so a resumed run whose sampling options changed fails the identity guard by name. The opencode install learns OpenCode V2's native `mcp.servers` config shape and `.jsonc` config files: comment-bearing files are edited byte-preserving, the legacy top-level shape stays untouched, and a one-time drift after the host adds `mcp.servers` is documented with its fix. Every release is described in the [CHANGELOG](CHANGELOG.md).+1.85.0 gives a long-lived vault the means to watch its own storage and to keep its append-only ledgers in shape. `o2b brain storage` (and the `brain_storage` tool) measures a per-surface byte census and forecasts days to the configured `storage.cap_bytes` in graduated tiers; a measured critical tier pauses the maintenance lane behind its fifth gate and refuses interactive ingests with `capacity_exhausted` before any write, while an uncapped vault reads unbounded, nothing pauses and nothing refuses. `o2b brain receipts` (and `brain_receipts`) drains the append-only continuity and metrics ledgers by folding expired records into per-kind, per-month aggregates behind an explicit preview-bound apply, with a count guard and a recovery point around every confirmed run. An opt-in `document-dedup` hygiene detector nominates near-duplicate document copies for retirement behind the review gate, keeping the newest copy, retargeting the stale copies' inbound links and auditing each retirement; bulk source removals report per-file failures instead of aborting; and appended sources yield their new text through incremental extraction over a manifest watermark, no whole-source reprocess. Every release is described in the [CHANGELOG](CHANGELOG.md).## Documentationdiff --git a/docs/brainstorm/storage-lifecycle/cli-output/claude.md b/docs/brainstorm/storage-lifecycle/cli-output/claude.mdnew file mode 100644index 00000000..5e57b3d3--- /dev/null+++ b/docs/brainstorm/storage-lifecycle/cli-output/claude.md@@ -0,0 +1,38 @@+### Variant 1: Dedicated storage subsystem on top of the existing gates+- **Approach**: Add a new `src/core/brain/storage/` module that owns four things. (1) A footprint census keyed by the `STATE_SURFACES` rows, which charges each surface for the bytes it actually keeps on disk. (2) A rolling usage aggregate written to the metrics sink and used only for forecasting. (3) A closed `STORAGE_VERDICT` vocabulary (`ok | warn | fail | unmeasurable`) that applies the 80 percent / 30 day and 95 percent / 7 day thresholds through a new policy block. (4) A `storage_pressure` gate that the maintenance lane and the other background producers check before they run, so interactive writes are never gated. Reclaim (named `storage_reclaim` so it does not clash with the Hermes compaction names) and the near-duplicate retirement detector both reuse the hygiene scan/plan/apply review gate. Both run on one shared per-item fail-soft removal-run primitive, and page-dedup and `deleteBySource` are moved onto that same primitive. The incremental-extraction watermark ships alongside as an additive field in the content manifest.+- **Trade-offs**:+ - Pro: all four cards share one vocabulary, one verdict shape and one removal-run primitive, so card 4 is fixed once for every bulk-removal path, including the new ones.+ - Pro: every decision re-measures on disk through the census, and the aggregate only feeds the forecast, which meets the measured-or-refused constraint and the "advisory aggregate" constraint directly.+ - Pro: it reuses proven seams (the lane gate pattern, the hygiene review gate, `withDestructiveSnapshot`, the `planEmbeddingSpend` blocked/reason shape), so little new mechanism is invented.+ - Pro: the reclaim planner can have an explicit preserve list (clip-protected continuity keys, `source_invalidation`, `decision_change.v1`, the source archive) that is enforced by a type-level exclusion table, not by convention.+ - Con: it is the largest surface area: a new policy block, new doctor checks (counter drift, reclaim backlog), new diagnostic codes, new MCP/CLI verbs with parity tests, and census rows.+ - Con: a full on-disk census on every gate check costs I/O, so it needs a cheap stat-only path plus a cooperative checkpoint deadline.+- **Complexity**: large+- **Risk**: medium++### Variant 2: Extend the existing seams in place+- **Approach**: Grow `token-footprint.ts` into a combined token and byte capacity report with the warn/fail thresholds, and add the forecast as one more field computed from the metrics JSONL. Add a host-pressure-style "storage" input to the existing lane gates so the lane pauses near the wall. Card by card: compaction becomes a new `LANE_TASK` with a CLI preview/apply pair, duplicate retirement becomes one more hygiene detector with an applier-capability row, the watermark goes into the manifest, and the page-dedup and `deleteBySource` loops each get their own try/catch-and-collect fix.+- **Trade-offs**:+ - Pro: smallest diff and fastest to ship, because each card lands in the file that already owns that concern.+ - Pro: no new top-level module, and fewer new census rows and parity tests.+ - Con: backpressure only covers the maintenance lane. Other background producers (vector backfill, dream, ingest batches) would each need their own ad-hoc check, so the "pause all background producers" guarantee is spread across many sites.+ - Con: card 4 is fixed twice in two shapes, and the new reclaim and retirement appliers would need a third copy. The per-file failure semantics drift apart.+ - Con: overloading the token-footprint report mixes two capacity models, and the 95 percent fail state has no single owner that enforcement can call.+ - Con: preservation of the protected receipts lives in the logic of one lane task, so it is easier to break silently later.+- **Complexity**: medium+- **Risk**: medium++### Variant 3: Enforcement at write admission with charge-at-write accounting+- **Approach**: Route every in-vault write through the shared writer, which runs an admission check. Each write is tagged with a closed producer class (`interactive | background`) and charges a persisted SQLite usage counter with its kept bytes at write time. Background writes are refused above the warn line, and every write is refused above the fail line unless it is a reclaim. Reclaim seals and rolls receipt ledgers into segments, and segments outside the preserve list become eligible for retirement behind a plan/apply gate. Duplicate retirement and the watermark are handled as in the other variants, and per-file decomposition is built into the writer's batch API.+- **Trade-offs**:+ - Pro: the most precise and immediate backpressure. No producer can bypass it, because the write-site census already forces every write through a shared writer.+ - Pro: the forecast comes cheaply from a counter that is always current.+ - Con: the counter becomes a second source of truth that can drift (crashes, out-of-band edits in Obsidian, git checkouts). The constraint requires on-disk re-measurement at decision time anyway, so the counter's main benefit is undercut and the drift doctor check becomes load-bearing.+ - Con: putting enforcement on the hot write path risks blocking interactive writes through misclassification or a bug in admission. That is exactly the failure the task exists to remove.+ - Con: segment sealing changes the on-disk shape of the append-only continuity and receipt ledgers. That is a migration with readers everywhere, and it puts the `decision_change.v1` "never folded" contract at risk.+ - Con: it touches every write site at once, which makes it the largest blast radius in the codebase.+- **Complexity**: large+- **Risk**: high++### Recommended: Variant 1+**Rationale**: Variant 1 is the only one that gives one owner to all four requirements: the measured-at-decision enforcement, the advisory-only forecast, the "pause only background producers" gate and a single per-file fail-soft removal primitive. It does this by extending the lane, hygiene and destructive-gate patterns the kernel already trusts, not by touching the hot write path or migrating the append-only ledgers. Variant 2 is cheaper but leaves backpressure and failure decomposition scattered across sites, where they will drift. Variant 3 adds a drift-prone counter and puts at risk both the interactive-write guarantee and the receipt contracts that the constraints protect.diff --git a/docs/brainstorm/storage-lifecycle/cli-output/prompt.md b/docs/brainstorm/storage-lifecycle/cli-output/prompt.mdnew file mode 100644index 00000000..65c39906--- /dev/null+++ b/docs/brainstorm/storage-lifecycle/cli-output/prompt.md@@ -0,0 +1,75 @@+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.++# Task++Storage lifecycle wave for Open Second Brain: turn monotonic vault growth from a silent hard failure into a visible, plannable, reversible set of maintenance events. Four cards, one subsystem:++1. Storage capacity forecasting with graduated warnings, background write backpressure and on-demand receipt compaction (priority 4). A brain that hits its cumulative storage cap refuses every new write with no warning beforehand. Wanted: charge persisted records for what they actually keep, forecast days to exhaustion from a rolling usage aggregate, warn at 80 percent or 30 days and fail at 95 percent or 7 days, pause background producers before the wall so interactive writes keep working, and an explicit preview-bound compaction command plus doctor checks for compaction and counter drift.+2. Review-gated cleanup that retires older duplicate documents with their source links (priority 3). Detect near-identical stored documents, retire the stale copies plus their link records behind the existing review/apply gate, keeping the newest copy. Open decision: canonical selection must be merge-stable.+3. Incremental extraction of text appended since the last extraction run for large sources (priority 1 residue). Today a 1 MiB pre-extract read cap and a content manifest holding only {sha256, fingerprint} mean appended conversation yields nothing new; a per-source byte watermark in the manifest plus a capped tail read recovers it.+4. Per-file batch-failure decomposition in bulk removal runs so one bad file does not abort the run (priority 1 residue). The page-dedup apply loop and the deleteBySource per-entry loop today abort or rethrow mid-run.++# Project context++Open Second Brain: a local-first memory kernel for coding agents. TypeScript on Bun 1.4.0, no runtime dependencies; runs no model (the calling agent extracts and writes prose; the kernel stores, gates and audits). All durable state lives inside the operator's vault (`Brain/` markdown + `.open-second-brain/brain.sqlite` derived store). Kernel invariants: named typed errors with closed code unions registered through an exhaustive-members compile check; plan/apply gates on anything destructive (dry-run default, explicit ids/confirm, count guard, snapshot-before-destructive-write, recoverability verdict); measured-or-refused honesty (never a silent zero); server-side measurement, never caller claims; state-surface, write-site and destructive-site censuses that fail the build when a new durable file, write or deletion is unregistered; MCP tool and CLI verb parity tests.++Recent commits:+ef974e0d fix: comma-separated and legacy scopes no longer abort the dream pass, plus three test-isolation fixes (v1.84.1) (#254)+14a99584 feat: recall lane concurrency, overlapped_ms accounting, a caller-selectable packing policy and standing-instruction facets (v1.84.0) (#252)+e239d323 feat: shipped-seam wiring - a per-session recall pool cap, usage-ordered skill offers, seeded bench sampling and OpenCode V2 configs (v1.83.0) (#251)++Related files:+- src/core/state/surfaces.ts (STATE_SURFACES catalogue of every durable file)+- src/core/brain/token-footprint.ts (existing token-capacity report with warn threshold - the closest precedent)+- src/core/brain/maintenance/lane.ts (background lane: window/busy/host-pressure/lease gates, LANE_TASK vocabulary, verdicts journalled)+- src/core/brain/maintenance/journal.ts (MAINTENANCE_VERDICT closed vocabulary)+- src/core/brain/safeguard.ts (cooperative checkpoint deadlines per OPERATION)+- src/core/brain/maintenance/host-pressure.ts (measured | unmeasurable honesty pattern)+- src/core/search/embedding-spend.ts (planEmbeddingSpend gate verdict: blocked + reason)+- src/core/brain/metrics.ts (appendMetric JSONL sink, surface o2b.metrics.v1)+- src/core/brain/exact-state.ts (overwrite-only Brain/state/<aspect>.md lane)+- src/core/brain/policy/ (blocks/ resolvers over Brain/_brain.yaml; defaults.ts; additive template ratchet in src/core/brain/upgrade.ts mergeBrainYaml)+- src/core/brain/doctor.ts + src/core/brain/doctor/ (DoctorCheck registry; issue codes registered in diagnostics.ts DIAGNOSTIC_SIGNALS or excused in doctor-exits.ts)+- src/core/brain/continuity/ (append-only per-device JSONL ledger, 28 closed record kinds, clip-protected correlation keys, schema o2b.continuity.v1)+- src/core/brain/decisions/receipts.ts (decision_change.v1 append-only accountability - must never be folded)+- src/core/brain/provenance/source-archive.ts (content-addressed evidence archive, no-TTL by contract)+- src/core/brain/hygiene/ (detector registry in scan.ts, plan.ts review gate, apply.ts per-finding fail-soft, applier-capability.ts table)+- src/core/brain/page-dedup.ts (exact-duplicate merge: keep-oldest canonical, merged_into pointers, cycle-refusing setMergedInto, wikilink rewrite)+- src/core/brain/source-cleanup.ts (deleteBySource: dry-run default, confirm ladder, per-source manifest-entry drop, source_invalidation audit, withDestructiveSnapshot)+- src/core/brain/ingest/ingest.ts (ingestSource; PRE_EXTRACT_MAX_SOURCE_BYTES = 1_048_576; readSourceBounded)+- src/core/brain/ingest/content-manifest.ts (per-source {sha256, fingerprint}; parseEntryValue drops unknown field shapes toward reprocessing)+- src/core/brain/ingest/contract.ts (EXTRACT_CONTRACT_VERSION = 2; bump forces one reprocess of every recorded source)+- src/cli/aider.ts readTail (the existing appended-bytes read precedent)+- src/core/search/vector-backfill.ts (dry-run default backfill with spend gate - the pause/gate verdict shape)++Conventions:+- Closed vocabularies as frozen const objects with membership guards; errors carry closed code unions; the MCP boundary maps them by name via exhaustiveMembers (a core code missing from the registry fails tsc).+- New doctor issue codes must be registered in DIAGNOSTIC_SIGNALS or excused with a written reason in DOCTOR_EXIT_EXCLUSIONS (a census test enforces this).+- New durable files need a STATE_SURFACES row; new in-vault write sites need a shared writer or written exclusion; new deletion sites need withDestructiveSnapshot or a DESTRUCTIVE_SITES entry (census tests enforce all three).+- Policy keys live in per-block resolvers under src/core/brain/policy/blocks/ with bounded field checks that hard-error out-of-range values; new template keys ride the additive mergeBrainYaml ratchet on upgrade.+- English only in artifacts; no hardcoded natural-language phrases where a closed vocabulary belongs; repeated literals hoisted into named constants.++Constraints:+- Interactive operator writes must keep working under backpressure; only background producers pause.+- Compaction and duplicate retirement stay behind explicit preview-bound apply gates; dry-run default; count guards; snapshot before destructive writes; audit record per confirmed run.+- The rolling usage aggregate is advisory and forecast-only; enforcement decisions re-measure on disk at the moment of the decision.+- Receipt compaction must preserve: clip-protected continuity keys, source_invalidation records, decision_change.v1 append-only receipts, the source-archive no-TTL contract.+- No new runtime dependencies. No daemon process. Bun runs SQLite synchronously - no fake async cancellation; cooperative checkpoints only.+- Naming must not collide with Hermes context compaction vocabulary (pre_compact_extract, post_compact_audit).++# Required output format++Produce exactly 3 distinct architectural variants. For each variant:++### Variant N: <short name>+- **Approach**: 2-3 sentences describing the variant.+- **Trade-offs**: bullet list of pros and cons.+- **Complexity**: small | medium | large+- **Risk**: low | medium | high++After the three variants, add exactly one recommendation:++### Recommended: Variant N+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.++Output nothing outside of these sections.diff --git a/docs/brainstorm/storage-lifecycle/design.md b/docs/brainstorm/storage-lifecycle/design.mdnew file mode 100644index 00000000..910429f4--- /dev/null+++ b/docs/brainstorm/storage-lifecycle/design.md@@ -0,0 +1,89 @@+# Storage lifecycle - capacity forecast, background backpressure, gated reclamation and residue recovery++**Status:** draft+**Author:** orchestrator (via feature-release-playbook)+**Audience:** implementation++## Problem statement++Open Second Brain vaults grow monotonically with agent sessions and captures, and nothing today tells the operator how much runway is left or degrades gracefully when the store fills. The only quota handling is the embedding provider's, the only reclamation primitive is artifact-run TTL pruning, and the nearest capacity report counts tokens, not bytes. This wave turns that silent hard failure into a visible, plannable, reversible set of maintenance events: measure what the store keeps, forecast the runway, warn in graduated steps, pause background producers before the wall while interactive writes keep working, reclaim bytes behind explicit apply gates (receipt compaction, near-duplicate retirement), and recover more value from already-stored bytes (incremental appended-text extraction, per-file batch-failure decomposition so removal runs always complete).++## Scope++- A storage usage meter that charges every durable state surface for the bytes it actually keeps on disk, measured server-side, per-surface measured-or-refused.+- A rolling usage aggregate (one `storage_usage` metrics sample per run) and a linear forecast of days to an operator-configured cap, with graduated tiers `ok / warn / critical / unbounded / unmeasurable` (defaults: warn at 80 percent or 30 days, critical at 95 percent or 7 days).+- A `storage:` policy block in `Brain/_brain.yaml` (cap and threshold keys), riding the additive template ratchet; no cap configured means unbounded and nothing changes for existing vaults.+- Background write backpressure: a capacity gate on the maintenance lane (new `skipped:capacity` verdict, journalled with its reading) and a named `capacity_exhausted` refusal at the `brain_ingest_source` entry at the critical tier, zero writes. Interactive note writes, recall and search are never refused by the capacity layer.+- An explicit preview-bound receipt compaction command (`o2b brain receipts` scan/compact, MCP `brain_receipts`) that folds old records in continuity and metrics JSONL shards into per-month aggregates behind a dry-run default, explicit ids, count guard, snapshot gate and per-run audit record, preserving the protected record classes verbatim.+- A content-level near-duplicate document detector in the hygiene registry whose `retire` action, behind the existing review/apply gate, retires stale copies keep-newest, retargets inbound wikilinks to the canonical copy, drops the retired copies' content-manifest entries, and writes a `source_invalidation` audit record per retired copy.+- Incremental appended-text extraction: an additive per-source `extracted_bytes` watermark in the content manifest, a UTF-8-safe capped tail read, and appended-delta reporting through the existing ingest result and `o2b brain extract` preview.+- Per-file batch-failure decomposition in the bulk removal runs (`deleteBySource`, the page-dedup CLI apply loop, the maintenance collect loops) so one bad file is collected as a named failure and the run completes.+- Doctor checks: `storage-capacity` (tier warning and critical issues, unmeasurable reported as uncertain) and `receipt-ledger-growth` (advisory compaction backlog).++## Out of scope++- Raising the 1 MiB pre-extract cap or the 8 MiB source-trust digest ceiling; deferred post-write embedding and any embedding spend-cap change (the card's other halves, already served by `planEmbeddingSpend`).+- The soft-delete / auto-revive / purge lifecycle from t_d7071804; only its independently buildable residue (per-file decomposition) ships.+- Refusing interactive writes at any tier, and any hard failure of the vault itself; the capacity layer pauses producers and refuses only new-source ingest at critical.+- Compacting decision-change receipts (`decision_change.v1` append-only accountability), the source archive (no-TTL contract by design), snapshots, or `source_invalidation` / pre- and post-compact continuity records.+- Any daemon, watcher or cron introduction; background remains the operator's schedule invoking the maintenance lane.+- A shared generic "batch runner" abstraction; decomposition follows the house per-loop shapes with typed per-subject failure records.++## Chosen approach++Variant 1 from the brainstorm (dedicated storage subsystem on top of the existing gates), with one refinement: receipt compaction is its own gated command rather than a hygiene detector, because the hygiene finding schema targets vault pages while ledger folding targets JSONL shards under `Brain/log` and `Brain/metrics` with different reach and preservation rules; it reuses the brain_hygiene tool shape (closed mode enum, scan default, explicit ids, count guard, audit) without joining its detector registry. Everything else follows the consultant's variant: one `src/core/brain/storage/` module owns measurement, sampling, forecast and the gate decision; the maintenance lane gains a fourth soft gate; reclamation and retirement reuse the plan/apply, snapshot, count-guard and audit seams the kernel already trusts; near-duplicate retirement joins the hygiene detector registry; the watermark is additive manifest state; decomposition is applied per loop in place.++Enforcement re-measures on disk at the moment of every decision; the rolling aggregate is advisory and forecast-only. The tier evaluation is measured-or-refused: a surface that cannot be stat'd is reported in `unmeasured` with its named reason (the `HOST_PRESSURE` posture), never as zero bytes, and an unmeasurable store yields tier `unmeasurable` - the gate then stays open and the doctor check reports uncertain.++## Design decisions++- **No cap by default.** `cap_bytes` absent means tier `unbounded`: bytes are reported, no days-to-exhaustion is invented, no gate fires, no refusal happens. A forecast against an invented default cap would be exactly the misleading number this wave removes.+- **The critical tier refuses only new-source ingest.** At `critical`, `brain_ingest_source` refuses before any write with `StorageCapacityError` code `capacity_exhausted`, naming the remedy (run receipt compaction or duplicate retirement, or raise the cap). Operator note writes, recall, search and status keep working so the operator can always run the reclamation verbs; the gate at the ingest entry sits in `src/mcp/brain/ingest-tools.ts`, not in `ingestSource`, so the core ingest path stays capacity-blind and the file stays owned by the extraction lane.+- **Backpressure is a pre-run refusal, never a wait.** The lane's capacity gate decides before the lease-handled run starts, like window/busy/pressure; nothing blocks on a condition, no new lock exists, and there is no lock-order interaction with the writer lock or the lease. `--force` bypasses it like the other soft gates, never the lease. The usage sample is taken before the skip is returned, so the forecast series keeps accumulating exactly while producers are paused.+- **Gate ordering follows cheapest-first, least-surprising-first.** Window, busy, host pressure, then capacity last: the capacity gate is the most expensive (a byte walk) and its reading is journalled the way the pressure reading is - the decision to skip and the fact that the gate did not evaluate are different journal lines.+- **Canonical selection for retirement is keep-newest on `created_at`, and pages without `created_at` are ineligible.** The mtime fallback is unstable under merges (the merge itself moves the survivor's mtime; a second pass can reverse the pair - the documented GitHub #180 hazard), so a finding whose candidates lack `created_at` is emitted with evidence naming the ineligible members instead of a sortable group. Scope keys keep folding the composite scope; identical text in two scopes never collapses.+- **Retirement drops records with provenance, not silently.** Per retired copy: inbound wikilinks retargeted to the canonical (fence-aware, scope-aware `retargetWikilinks`), page removed, content-manifest entry dropped round-tripping the read contract (the t_586d5d8b rule; a dropped entry must not resurrect on re-ingest), and one exported `appendContinuitySourceInvalidation` record naming canonical id, retired id, dropped manifest key and the retained source-archive hash. The source archive itself is never deleted. Per-copy failures are collected; a copy whose retarget fails stays in place.+- **Receipt compaction preserves a closed protected set, and unknown kinds fail closed to keep.** Fold candidates are continuity and metrics JSONL records older than the invocation's `older_than_days`, folded into one aggregate line per (kind, month) carrying count, first/last timestamps and byte totals. Kept verbatim: `source_invalidation`, `pre_compact_extract`, `post_compact_audit`, any record whose payload carries a `CLIP_PROTECTED_PAYLOAD_KEYS` key, and any record whose kind the folder does not know (forward compatibility - an unknown kind is never folded). Decision receipts live in a separate ledger the command never opens. The audit record per confirmed run is a new additive continuity kind `receipt_compaction`; the naming stays disjoint from the Hermes context-compaction vocabulary on every surface.+- **Compaction rewrite is atomic per shard and fail-soft per shard.** New shard content is written to a temp sibling and renamed inside a `withDestructiveSnapshot` span; a shard that fails is collected with its reason and the run continues; the result reports honest matched/folded/kept/failed counts, and the doctor `receipt-ledger-growth` check reads the same scan output.+- **The watermark is additive manifest state with no contract bump.** `extracted_bytes` (and `extracted_at`) join `ManifestEntry`; `parseEntryValue` learns the extended shape. Absence of the field means "consumed from byte 0", so the first incremental pass over an old record reports the whole file as the delta - the manifest's own reads-degrade-toward-reprocessing rule, applied lazily instead of an eager `EXTRACT_CONTRACT_VERSION` bump that would reprocess every recorded source at once. Shrink falls back to a full re-read with the watermark reset; tail reads snap to the next UTF-8-safe boundary (newline-aligned) and ride the existing bounded-read discipline and caps. Source-trust classification is unchanged: the incremental lane changes what is read for extraction, not how a source is trusted.+- **Decomposition keeps the wire shapes the house already ships.** Each decomposed loop collects typed per-subject failure records (`{path, message}` for file removals, `{key, message}` for dedup clusters) and the CLI exit rules follow the import-session precedent (exit 1 when failures are collected, data reports still exit 0). `runDeletion` keeps its persist-partial-audit-then-throw posture for errors outside the per-entry loop; inside the loop a failing file no longer aborts the remaining entries. write-batch's all-or-nothing contract is untouched.+- **The storage report is its own surface, not a `computeBrainStatus` block.** A byte walk stats every file; folding it into the always-loaded status path would break that path's documented cost bound. `o2b brain storage` and `brain_storage` carry the measurement, forecast and tier; both answer `measured: false` with a named reason below local reach (the `link_integrity` model).+- **New named vocabulary everywhere.** `STORAGE_CAPACITY_TIER`, `MAINTENANCE_VERDICT.skippedCapacity`, `STORAGE_ERROR_CODES` (`capacity_unmeasurable`, `capacity_exhausted`) registered through `exhaustiveMembers`, hygiene action `retire`, detector id `document-dedup`, continuity kind `receipt_compaction`, doctor codes registered in `DIAGNOSTIC_SIGNALS` - every census gets its registration in the same task that adds the vocabulary.++## File changes++New files:++- `src/core/brain/storage/usage.ts`, `forecast.ts`, `gate.ts`, `capacity-error.ts` - measurement, sampling and forecast, the lane-gate decision, the leaf error module.+- `src/core/brain/policy/blocks/storage.ts` - the `storage:` block resolver.+- `src/core/brain/doctor/storage-capacity.ts`, `src/core/brain/doctor/receipt-ledger.ts` - the two new checks.+- `src/core/brain/receipts/types.ts`, `scan.ts`, `compact.ts` - the receipt compaction engine.+- `src/cli/brain/verbs/storage.ts`, `src/cli/brain/verbs/receipts.ts` - the two new CLI verbs.+- `src/mcp/brain/storage-tools.ts`, `src/mcp/brain/receipt-tools.ts` - the two new MCP tools.+- `src/core/brain/hygiene/detectors/document-dedup.ts` - the content-level near-duplicate detector.+- `src/core/brain/document-retire.ts` - the retirement applier core.+- `src/core/brain/ingest/tail-read.ts` - the UTF-8-safe capped tail read.+- Tests for every module above, plus census/parity updates where the census binds by list rather than by enumeration.++Modified files (owners are pinned per lane in plan.md):++- `src/core/brain/policy/load.ts`, `defaults.ts`, `validate.ts`, `src/core/brain/types.ts`, `src/core/brain/config-template.ts` - storage block wiring and template ratchet entry.+- `src/mcp/tool-error-codes.ts` - the new codes via `exhaustiveMembers`.+- `src/core/brain/doctor.ts`, `src/core/brain/diagnostics.ts` - registry rows and issue-code registrations (landed first by the substrate lane, appended by the compaction lane).+- `src/core/brain/maintenance/lane.ts`, `journal.ts`, `src/cli/brain/verbs/maintenance.ts`, `src/mcp/brain/admin-tools.ts` - the capacity gate, verdict and both surfaces' derivation.+- `src/mcp/brain/ingest-tools.ts` - the critical-tier admission refusal.+- `src/mcp/brain-tools.ts` - tool aggregation (substrate lane first, compaction lane appends).+- `src/core/brain/hygiene/types.ts`, `scan.ts`, `apply.ts`, `src/core/brain/applier-capability.ts` - detector registration, `retire` action, applier routing and capability row.+- `src/core/brain/continuity/types.ts` - the additive `receipt_compaction` kind.+- `src/core/brain/source-cleanup.ts`, `src/cli/brain/verbs/page-dedup.ts`, `src/core/brain/maintenance/collect.ts` - per-file decomposition.+- `src/core/brain/ingest/ingest.ts`, `content-manifest.ts`, `src/cli/brain/verbs/extract.ts` - watermark plumbing and appended-delta reporting.++## Risks and open questions++- **Backpressure must not starve the forecast.** The sample is taken before a `skipped:capacity` return; the explicit `brain storage` invocation samples too. Implementation must not place the sampler after the early return.+- **Byte-walk cost.** The walker is bounded by the lane's per-task safeguard deadline and the report command's own budget; the gate reuses the same measurement with a stat-only fast path per surface (directory sizes are not summable without walking, so the cost is honest I/O, bounded by the same deadline).+- **Reach honesty.** Every vault-wide byte figure is a whole-vault measurement; below local reach the report answers `measured: false` plus reason, and the lane gate is inherently local (the maintenance lane already refuses non-local runs).+- **Concurrent ledger appends during compaction.** Shard replacement races an appender; the rewrite takes the same per-file lock ladder the continuity store already uses (`withFileLock`), and the snapshot gate covers the destructive window.+- **Watermark correctness under editors that rewrite in place** (content changed, not appended): the manifest hash classifies the file `modified` and the full reprocess path applies; the watermark is stamped only when extraction covered the bytes it claims.+- **Census discipline.** Every new durable-write route goes through `atomicWriteFileSync` or a written write-site exclusion; every deletion sits inside `withDestructiveSnapshot`; ledger rewrites touch existing `STATE_SURFACES` rows, so no new row is expected - the state-surface census run in CI is the verifier.+- **Open question resolved during implementation:** whether `brain_hygiene`'s apply path needs any MCP-layer change for the `retire` action or whether the existing generic ids/count-guard routing covers it unchanged (expected: unchanged; verify against `hygiene-tools.ts` apply enforcement).diff --git a/docs/brainstorm/storage-lifecycle/plan.md b/docs/brainstorm/storage-lifecycle/plan.mdnew file mode 100644index 00000000..ff7075c9--- /dev/null+++ b/docs/brainstorm/storage-lifecycle/plan.md@@ -0,0 +1,160 @@+# Storage lifecycle - implementation plan++Six features across five parallel lanes. Each lane owns a disjoint file set; the only shared files are the pinned contracts listed below, and every contract names its owner and landing order. Tasks are TDD-ready: the acceptance test is written first and must fail with the expected error before implementation.++## Pinned cross-lane contracts++| # | File(s) | Exported symbols | Owner | Landing |+|---|---------|------------------|-------|---------|+| C1 | `src/core/brain/storage/usage.ts`, `forecast.ts`, `gate.ts`, `capacity-error.ts` | `measureStorageUsage`, `sampleStorageUsage`, `forecastStorageUsage`, `STORAGE_CAPACITY_TIER`, `StorageForecast`, `StorageUsage`, `evaluateCapacityGate`, `CapacityGateDecision`, `StorageCapacityError`, `STORAGE_ERROR_CODES` | Lane 1 | Lands first; Lane 2 and Lane 3 consume read-only |+| C2 | `src/core/brain/types.ts`, `src/core/brain/policy/blocks/storage.ts`, `policy/load.ts`, `policy/defaults.ts`, `policy/validate.ts`, `src/core/brain/config-template.ts` | `BrainStorageConfig`, `ResolvedBrainStorageConfig`, `BRAIN_STORAGE_DEFAULTS`, `resolveStorage` | Lane 1 | Lands first; no other lane edits any config file |+| C3 | `src/mcp/tool-error-codes.ts` | `STORAGE_ERROR_CODES` registered via `exhaustiveMembers` | Lane 1 | Lands first |+| C4 | `src/core/brain/doctor.ts`, `src/core/brain/diagnostics.ts` | `DOCTOR_CHECKS` row for `storageCapacityCheck` then `receiptLedgerCheck`; `DIAGNOSTIC_SIGNALS` codes | Lane 1 first, Lane 3 appends | Lane 3 depends on Lane 1 for these two files |+| C5 | `src/mcp/brain-tools.ts` | `STORAGE_TOOLS` aggregation then `RECEIPT_TOOLS` | Lane 1 first, Lane 3 appends | Lane 3 depends on Lane 1 for this file |+| C6 | `src/core/brain/source-cleanup.ts` | widened export `removeFile` (plus Lane 5's own loop decomposition) | Lane 5 | Lands before Lane 4 consumes `removeFile`; Lane 4 imports it, never edits the file |+| C7 | `src/core/brain/ingest/*`, `src/cli/brain/verbs/extract.ts` | watermark internals | Lane 5 | Lane 2 must not touch `ingest.ts`; the admission refusal lives in `src/mcp/brain/ingest-tools.ts` (Lane 2's file), so the two never meet |+| C8 | `src/core/brain/hygiene/types.ts`, `scan.ts`, `apply.ts`, `src/core/brain/applier-capability.ts` | `document-dedup` detector id, `retire` action, applier routing | Lane 4 | Exclusive to Lane 4 |+| C9 | `src/core/brain/maintenance/lane.ts`, `journal.ts`, `src/cli/brain/verbs/maintenance.ts`, `src/mcp/brain/admin-tools.ts` | capacity gate wiring, `MAINTENANCE_VERDICT.skippedCapacity` | Lane 2 | Exclusive to Lane 2 |+| C10 | `src/core/brain/continuity/types.ts` | additive `receipt_compaction` record kind | Lane 3 | Exclusive to Lane 3 |++No lane edits `src/core/brain/continuity/store.ts` (Lane 3 keeps the append path generic; Lane 4 imports the existing exported `appendContinuitySourceInvalidation`). No new `STATE_SURFACES` rows are expected: samples ride the metrics sink, the audit rides continuity, and ledger rewrites touch existing rows; the state-surface census run in CI verifies this per lane.++---++## Lane 1 - storage substrate (lands first)++Pure modules first, then registration, then surfaces.++### Task 1.1: usage meter+- **Files**: new `src/core/brain/storage/usage.ts`; new `tests/core/brain/storage-usage.test.ts`+- **Acceptance**: `measureStorageUsage(vault, opts)` on a sandbox vault returns per-tier byte totals (vault-content, derived) that match a hand-computed fixture (page files, `brain.sqlite` plus WAL/SHM sidecars, artifacts, snapshots, ledgers); a surface made unreadable (chmod fixture gated on `CHMOD_CANNOT_DENY`) appears in `unmeasured` with its named reason and never as zero bytes; the walker's per-dot-dir policy (artifacts, archives, snapshots all counted) is asserted; deterministic via injected `now` only.+- **Depends on**: none.++### Task 1.2: forecast and tiers+- **Files**: new `src/core/brain/storage/forecast.ts`; new `tests/core/brain/storage-forecast.test.ts`+- **Acceptance**: `forecastStorageUsage` over seeded `storage_usage` metric samples returns the linear slope, `days_to_cap` and tier; tier boundaries hit exactly at the default constants (warn 80 percent / 30 days, critical 95 percent / 7 days, module-level named constants); absent `cap_bytes` yields `unbounded`; an unreadable store yields `unmeasurable`; fewer than two samples yields the named no-forecast state; `sampleStorageUsage` appends one `storage_usage` metrics record per call, fail-soft, and is asserted idempotent per run id.+- **Depends on**: Task 1.1.++### Task 1.3: capacity error leaf and code registration+- **Files**: new `src/core/brain/storage/capacity-error.ts`; modified `src/mcp/tool-error-codes.ts` (C3); new `tests/mcp/storage-capacity-errors.test.ts`+- **Acceptance**: `StorageCapacityError` carries the closed `STORAGE_ERROR_CODES` union (`capacity_unmeasurable`, `capacity_exhausted`); registration compiles through `exhaustiveMembers`; the MCP envelope test asserts the codes map by name and never leak vault paths at remote reach.+- **Depends on**: none (parallel with 1.1/1.2 within the lane).++### Task 1.4: storage policy block+- **Files**: C2 set - new `src/core/brain/policy/blocks/storage.ts`, modified `src/core/brain/types.ts`, `policy/load.ts`, `policy/defaults.ts`, `policy/validate.ts`, `src/core/brain/config-template.ts`; new `tests/core/brain/storage-policy.test.ts`+- **Acceptance**: `resolveStorage` parses `cap_bytes`, `warn_percent`, `critical_percent`, `warn_days`, `critical_days` with bounded field checks that hard-error out-of-range values and a `critical_percent` that must exceed `warn_percent`; absent block resolves to the documented defaults with `cap_bytes` absent (unbounded); the template ratchet appends the new block to an existing `_brain.yaml` without rewriting existing values (mergeBrainYaml fixture).+- **Depends on**: none within the lane.++### Task 1.5: capacity gate decision+- **Files**: new `src/core/brain/storage/gate.ts`; extend `tests/core/brain/storage-forecast.test.ts` or a sibling `tests/core/brain/storage-gate.test.ts`+- **Acceptance**: `evaluateCapacityGate` returns `run` / `skip` with the tier and reading attached; unbounded and `unmeasurable` stores never skip; the decision is computed from a fresh `measureStorageUsage` call at decision time, never from the stored aggregate (asserted by mutating the aggregate between sample and decision).+- **Depends on**: Tasks 1.2 and 1.4.++### Task 1.6: storage-capacity doctor check+- **Files**: new `src/core/brain/doctor/storage-capacity.ts`; modified `src/core/brain/doctor.ts`, `src/core/brain/diagnostics.ts` (C4, first landing); new `tests/core/brain/doctor-storage-capacity.test.ts`+- **Acceptance**: `runDoctor` on a hand-built skeleton vault (doctor-drift pattern) emits a warning issue at the warn tier, an error issue at critical, and an `uncertain` entry for an unmeasurable store, reading `ctx.dbPath` fail-soft; both issue codes are registered in `DIAGNOSTIC_SIGNALS` and `tests/core/brain/doctor-exit-census.test.ts` passes unchanged in mechanism; reach filtering follows the doctor-counts-reach pattern.+- **Depends on**: Tasks 1.2 and 1.3.++### Task 1.7: report surfaces (CLI verb and MCP tool)+- **Files**: new `src/cli/brain/verbs/storage.ts`, `src/mcp/brain/storage-tools.ts`; modified `src/mcp/brain-tools.ts` (C5, first landing); new `tests/cli/brain/verbs/storage.test.ts`, `tests/mcp/brain-storage.test.ts`, `tests/mcp/storage-reach.test.ts`+- **Acceptance**: `o2b brain storage` and `brain_storage` return the measurement, forecast, tier and per-surface `unmeasured` reasons; below local reach the report answers `measured: false` with the named reason (link-integrity model); each invocation appends one usage sample; the brain-tools parity test derives both surfaces without a CLI_ONLY/MCP_ONLY entry.+- **Depends on**: Tasks 1.5 and 1.6.++---++## Lane 2 - background write backpressure++### Task 2.1: maintenance lane capacity gate+- **Files**: modified `src/core/brain/maintenance/lane.ts` (C9), `src/core/brain/maintenance/journal.ts` (C9); new `tests/core/brain/maintenance-capacity-gate.test.ts`+- **Acceptance**: `evaluateGates` gains the capacity check ordered after host pressure, active only when the vault's storage block configures `cap_bytes`; a critical tier returns the new `MAINTENANCE_VERDICT.skippedCapacity` with the reading carried alongside (the pressure-reading model); `skipped:capacity` journal rows parse and render; `--force` bypasses it, the lease is never bypassed; the usage sample is taken before the skip returns (asserted: a skipped run still appends a sample); unmeasurable leaves the gate open with the reason journalled.+- **Depends on**: Lane 1 Tasks 1.2, 1.4, 1.5 (C1, C2).++### Task 2.2: lane surfaces carry the verdict+- **Files**: modified `src/cli/brain/verbs/maintenance.ts` (C9), `src/mcp/brain/admin-tools.ts` (C9); extend `tests/core/brain/maintenance-capacity-gate.test.ts`; update `tests/mcp/maintenance-parity-census.test.ts` expectations+- **Acceptance**: CLI flags and MCP properties derive the new verdict and reading identically from source; the parity census passes in both directions with no new exclusion; a skipped-capacity run reports the tier and the remedy path in both surfaces.+- **Depends on**: Task 2.1.++### Task 2.3: ingest admission refusal+- **Files**: modified `src/mcp/brain/ingest-tools.ts` (C7 boundary - this file only); new `tests/mcp/ingest-capacity-refusal.test.ts`+- **Acceptance**: at the critical tier `brain_ingest_source` refuses before any write with the `capacity_exhausted` envelope naming the remedy (compaction, retirement, cap raise); zero bytes written to the vault (asserted with `brainPageTexts`); an unbounded or sub-critical vault ingests unchanged; the check is skipped entirely when no cap is configured (no measurement cost).+- **Depends on**: Lane 1 Tasks 1.3, 1.5 (C1, C3).++---++## Lane 3 - receipt compaction++### Task 3.1: ledger scan+- **Files**: new `src/core/brain/receipts/types.ts`, `src/core/brain/receipts/scan.ts`; new `tests/core/brain/receipts-scan.test.ts`+- **Acceptance**: `scanReceiptLedgers` inventories continuity and metrics shards with record counts, byte sizes, foldable counts above the age threshold and protected-record counts; the protected vocabulary (source_invalidation, pre/post-compact kinds, clip-protected-key carriers, unknown kinds) is asserted never counted foldable; the scan is read-only (byte-identical vault after run).+- **Depends on**: none.++### Task 3.2: fold engine+- **Files**: new `src/core/brain/receipts/compact.ts`; new `tests/core/brain/receipts-compact.test.ts`+- **Acceptance**: `compactReceiptLedgers` with explicit shard ids folds records older than `older_than_days` into one aggregate line per (kind, month) preserving count, first/last timestamps and byte totals; protected records survive verbatim in order; unknown kinds fail closed to keep; shrink is verified (output strictly smaller, protected content byte-present); the rewrite is atomic per shard inside `withDestructiveSnapshot` and per-shard fail-soft (a poisoned shard is collected, the run completes); re-run is idempotent (nothing left foldable, no write); the continuity record kind `receipt_compaction` (C10, additive to `src/core/brain/continuity/types.ts`) is appended exactly once per confirmed run and never on a dry run; `expect`/count-guard mismatch aborts before any write.+- **Depends on**: Task 3.1.++### Task 3.3: receipt-ledger-growth doctor check+- **Files**: new `src/core/brain/doctor/receipt-ledger.ts`; modified `src/core/brain/doctor.ts`, `src/core/brain/diagnostics.ts` (C4, appended after Lane 1); new `tests/core/brain/doctor-receipt-ledger.test.ts`+- **Acceptance**: a ledger above the growth threshold yields a warning issue naming the receipts command; a compacted ledger yields no issue; codes registered in `DIAGNOSTIC_SIGNALS`; doctor-exit census passes.+- **Depends on**: Task 3.1; Lane 1 Task 1.6 (C4 order).++### Task 3.4: receipt surfaces (CLI verb and MCP tool)+- **Files**: new `src/cli/brain/verbs/receipts.ts`, `src/mcp/brain/receipt-tools.ts`; modified `src/mcp/brain-tools.ts` (C5, appended after Lane 1); new `tests/cli/brain/verbs/receipts.test.ts`, `tests/mcp/brain-receipts.test.ts`+- **Acceptance**: `o2b brain receipts scan` is the default and writes nothing; `compact` requires explicit shard ids from a scan and enforces the count guard before mutating; the deny refusal (`refuseDocumentDeny`) fires on the mutation lane; the tool and verb derive from one source (parity test); output names the operation receipt compaction, disjoint from the context-compaction vocabulary.+- **Depends on**: Task 3.2, Task 3.3; Lane 1 Task 1.7 (C5 order).++---++## Lane 4 - near-duplicate document retirement++### Task 4.1: document-dedup detector+- **Files**: new `src/core/brain/hygiene/detectors/document-dedup.ts`; new `tests/core/brain/document-dedup-detector.test.ts`+- **Acceptance**: the detector groups ingest summary pages (kind `brain-source`) by content similarity at the lexical threshold (0.8) and, when a provider is configured, the embedding threshold (0.97), excluding merge-resolved pairs and keeping scope keys folded; canonical is the newest `created_at`; a group containing any member without `created_at` is emitted ineligible with the reason in evidence (mtime instability); finding ids are deterministic via `hygieneFindingId`.+- **Depends on**: none.++### Task 4.2: retire action vocabulary and routing+- **Files**: modified `src/core/brain/hygiene/types.ts`, `src/core/brain/hygiene/apply.ts`, `src/core/brain/applier-capability.ts` (C8); new `tests/core/brain/hygiene-retire-routing.test.ts`+- **Acceptance**: `retire` joins the closed action vocabulary and the applier-capability table; `buildHygienePlan` routes retire findings through the review path (excluded from unattended apply); `executeFinding` dispatches retire to the retirement core with the count guard semantics hygiene apply already enforces.+- **Depends on**: Task 4.1.++### Task 4.3: retirement applier core+- **Files**: new `src/core/brain/document-retire.ts`; new `tests/core/brain/document-retire.test.ts`+- **Acceptance**: per stale copy, the applier retargets inbound wikilinks to the canonical (retargetWikilinks report asserted), removes the page, drops the content-manifest entry round-tripping the read contract, and appends one `appendContinuitySourceInvalidation` record naming canonical id, retired id, dropped manifest key and the retained source-archive hash; a re-ingest of the retired copy's source does not resurrect a page (manifest-drop assertion); per-copy failure is collected (`{path, message}`), the failed copy stays on disk, and the run completes; the whole span sits inside `withDestructiveSnapshot` with a recoverability verdict; the source archive file is asserted still present.+- **Depends on**: Task 4.2; Lane 5 Task 5.1 (C6 - consumes the exported `removeFile`).++---++## Lane 5 - residues: decomposition and incremental extraction++### Task 5.1: deleteBySource per-file decomposition+- **Files**: modified `src/core/brain/source-cleanup.ts` (C6, lands first - widens the `removeFile` export for Lane 4); extend `tests/core/brain/source-cleanup.test.ts`+- **Acceptance**: a per-entry removal failure is collected as `{path, message}`, the remaining entries are still removed, the partial-progress audit and the final audit carry honest removed/failed counts, and the result shape gains the `failed` array; a run with zero failures is byte-identical in behavior to today; regression test asserts the pre-fix code fails this test (one injected failing file aborts the run today).+- **Depends on**: none.++### Task 5.2: page-dedup and collect loop decomposition+- **Files**: modified `src/cli/brain/verbs/page-dedup.ts`, `src/core/brain/maintenance/collect.ts`; new `tests/cli/brain/verbs/page-dedup-apply-fail-soft.test.ts`, extend the maintenance collect tests+- **Acceptance**: a MergeChainError on one cluster is collected (`{key, message}`), later clusters still apply, the report lists failures, and the CLI exits 1 when failures are non-empty; collect loops collect per-item errors in the module's existing report shape; write-batch all-or-nothing behavior is asserted unchanged.+- **Depends on**: none (parallel with 5.1 inside the lane).++### Task 5.3: tail read primitive+- **Files**: new `src/core/brain/ingest/tail-read.ts`; new `tests/core/brain/ingest/tail-read.test.ts`+- **Acceptance**: reading from a byte offset returns only appended text snapped forward to the first UTF-8-safe newline boundary; a file that shrank below the offset falls back to the full content with the shrink named; the bounded-read ceiling is enforced with the existing refusal vocabulary; multi-byte characters split across the boundary decode cleanly.+- **Depends on**: none.++### Task 5.4: manifest watermark+- **Files**: modified `src/core/brain/ingest/content-manifest.ts`; extend `tests/core/brain/ingest/content-manifest.test.ts`+- **Acceptance**: `parseEntryValue` accepts entries carrying additive `extracted_bytes` / `extracted_at` and preserves them through `updateManifest` round-trips; an entry without them degrades to byte 0; schema_version stays 3 (additive fields never bump); unknown field shapes still drop toward reprocessing exactly as today.+- **Depends on**: none.++### Task 5.5: incremental extraction plumbing+- **Files**: modified `src/core/brain/ingest/ingest.ts`, `src/cli/brain/verbs/extract.ts` (C7); new `tests/core/brain/ingest/incremental-extract.test.ts`+- **Acceptance**: an ingest of a source whose manifest record carries `extracted_bytes` reports the appended delta `{from_byte, to_byte}` on `IngestSourceResult` and stamps the new watermark only when the run covered those bytes; the CLI extract preview reports the appended slice metadata for a previously-extracted source; appending bytes to a fixture source twice yields two correct deltas with no overlap; a rewritten (not appended) source classifies modified and reprocesses fully, resetting the watermark; `EXTRACT_CONTRACT_VERSION` is asserted unchanged at 2 and no mass reprocess is triggered.+- **Depends on**: Tasks 5.3 and 5.4.++---++## Landing order and integration++1. Lane 1 lands whole (substrate first: Tasks 1.1-1.4 may interleave, 1.5-1.7 in order).+2. Lane 2 and Lane 3 start once Lane 1's dependency tasks land; Lane 5 starts immediately (independent); Lane 4 starts once Task 5.1 lands.+3. Integration order at the end of the wave: Lane 3's C4/C5 appends rebase onto Lane 1's landed registry rows; the full suite, both censuses (`doctor-exit-census`, `maintenance-parity-census`, `brain-tools-parity`, write-site and destructive-site) and the formatter run before each lane's commits per the playbook's commit discipline.diff --git a/docs/brainstorm/storage-lifecycle/variants.md b/docs/brainstorm/storage-lifecycle/variants.mdnew file mode 100644index 00000000..4eae8565--- /dev/null+++ b/docs/brainstorm/storage-lifecycle/variants.md@@ -0,0 +1,60 @@+# Storage lifecycle - variant audit trail++Consultant: Claude Code (`claude -p`), phase 0 of the feature-release-playbook. Prompt verbatim in `cli-output/prompt.md`, output verbatim in `cli-output/claude.md`, reproduced below without edits.++## Consultant output (verbatim)++### Variant 1: Dedicated storage subsystem on top of the existing gates+- **Approach**: Add a new `src/core/brain/storage/` module that owns four things. (1) A footprint census keyed by the `STATE_SURFACES` rows, which charges each surface for the bytes it actually keeps on disk. (2) A rolling usage aggregate written to the metrics sink and used only for forecasting. (3) A closed `STORAGE_VERDICT` vocabulary (`ok | warn | fail | unmeasurable`) that applies the 80 percent / 30 day and 95 percent / 7 day thresholds through a new policy block. (4) A `storage_pressure` gate that the maintenance lane and the other background producers check before they run, so interactive writes are never gated. Reclaim (named `storage_reclaim` so it does not clash with the Hermes compaction names) and the near-duplicate retirement detector both reuse the hygiene scan/plan/apply review gate. Both run on one shared per-item fail-soft removal-run primitive, and page-dedup and `deleteBySource` are moved onto that same primitive. The incremental-extraction watermark ships alongside as an additive field in the content manifest.+- **Trade-offs**:+ - Pro: all four cards share one vocabulary, one verdict shape and one removal-run primitive, so card 4 is fixed once for every bulk-removal path, including the new ones.+ - Pro: every decision re-measures on disk through the census, and the aggregate only feeds the forecast, which meets the measured-or-refused constraint and the "advisory aggregate" constraint directly.+ - Pro: it reuses proven seams (the lane gate pattern, the hygiene review gate, `withDestructiveSnapshot`, the `planEmbeddingSpend` blocked/reason shape), so little new mechanism is invented.+ - Pro: the reclaim planner can have an explicit preserve list (clip-protected continuity keys, `source_invalidation`, `decision_change.v1`, the source archive) that is enforced by a type-level exclusion table, not by convention.+ - Con: it is the largest surface area: a new policy block, new doctor checks (counter drift, reclaim backlog), new MCP/CLI verbs with parity tests, and census rows.+ - Con: a full on-disk census on every gate check costs I/O, so it needs a cheap stat-only path plus a cooperative checkpoint deadline.+- **Complexity**: large+- **Risk**: medium++### Variant 2: Extend the existing seams in place+- **Approach**: Grow `token-footprint.ts` into a combined token and byte capacity report with the warn/fail thresholds, and add the forecast as one more field computed from the metrics JSONL. Add a host-pressure-style "storage" input to the existing lane gates so the lane pauses near the wall. Card by card: compaction becomes a new `LANE_TASK` with a CLI preview/apply pair, duplicate retirement becomes one more hygiene detector with an applier-capability row, the watermark goes into the manifest, and the page-dedup and `deleteBySource` loops each get their own try/catch-and-collect fix.+- **Trade-offs**:+ - Pro: smallest diff and fastest to ship, because each card lands in the file that already owns that concern.+ - Pro: no new top-level module, and fewer new census rows and parity tests.+ - Con: backpressure only covers the maintenance lane. Other background producers (vector backfill, dream, ingest batches) would each need their own ad-hoc check, so the "pause all background producers" guarantee is spread across many sites.+ - Con: card 4 is fixed twice in two shapes, and the new reclaim and retirement appliers would need a third copy. The per-file failure semantics drift apart.+ - Con: overloading the token-footprint report mixes two capacity models, and the 95 percent fail state has no single owner that enforcement can call.+ - Con: preservation of the protected receipts lives in the logic of one lane task, so it is easier to break silently later.+- **Complexity**: medium+- **Risk**: medium++### Variant 3: Enforcement at write admission with charge-at-write accounting+- **Approach**: Route every in-vault write through the shared writer, which runs an admission check. Each write is tagged with a closed producer class (`interactive | background`) and charges a persisted SQLite usage counter with its kept bytes at write time. Background writes are refused above the warn line, and every write is refused above the fail line unless it is a reclaim. Reclaim seals and rolls receipt ledgers into segments, and segments outside the preserve list become eligible for retirement behind a plan/apply gate. Duplicate retirement and the watermark are handled as in the other variants, and per-file decomposition is built into the writer's batch API.+- **Trade-offs**:+ - Pro: the most precise and immediate backpressure. No producer can bypass it, because the write-site census already forces every write through a shared writer.+ - Pro: the forecast comes cheaply from a counter that is always current.+ - Con: the counter becomes a second source of truth that can drift (crashes, out-of-band edits in Obsidian, git checkouts). The constraint requires on-disk re-measurement at decision time anyway, so the counter's main benefit is undercut and the drift doctor check becomes load-bearing.+ - Con: putting enforcement on the hot write path risks blocking interactive writes through misclassification or a bug in admission. That is exactly the failure the task exists to remove.+ - Con: segment sealing changes the on-disk shape of the append-only continuity and receipt ledgers. That is a migration with readers everywhere, and it puts the `decision_change.v1` "never folded" contract at risk.+ - Con: it touches every write site at once, which makes it the largest blast radius in the codebase.+- **Complexity**: large+- **Risk**: high++### Recommended: Variant 1+**Rationale**: Variant 1 is the only one that gives one owner to all four requirements: the measured-at-decision enforcement, the advisory-only forecast, the "pause only background producers" gate and a single per-file fail-soft removal primitive. It does this by extending the lane, hygiene and destructive-gate patterns the kernel already trusts, not by touching the hot write path or migrating the append-only ledgers. Variant 2 is cheaper but leaves backpressure and failure decomposition scattered across sites, where they will drift. Variant 3 adds a drift-prone counter and puts at risk both the interactive-write guarantee and the receipt contracts that the constraints protect.++## Orchestrator decision++Variant 1 is adopted, with two refinements that do not change its architecture:++1. Receipt compaction is its own gated command (`o2b brain receipts` / `brain_receipts`, scan default, explicit ids, count guard, snapshot gate, per-run audit) rather than a hygiene detector. The hygiene finding schema targets vault pages with page-level actions; ledger folding targets JSONL shards under `Brain/log` and `Brain/metrics` with different subjects, reach rules and preservation classes. The command reuses the brain_hygiene tool shape without joining its detector registry. This also keeps the hygiene action vocabulary to the single addition the retirement feature needs (`retire`).+2. The shared per-item fail-soft removal-run primitive is declined in favor of decomposing each loop in place with the typed per-subject failure records the house already ships (`{path, message}`, `{key, message}`) and the standardized exit rule. One generic primitive would erase the typed subject names from the tool results and merge four loop shapes with different audit postures; the in-place decomposition keeps the wire shapes honest and matches the idiom of the indexer sync and import-session loops.++The consultant's Variant 1 verdict vocabulary (`ok | warn | fail`) is spelled `ok | warn | critical | unbounded | unmeasurable` in the design: `critical` over `fail` because the tier pauses producers and refuses new-source ingest but never fails the vault, and `unbounded` names the no-cap-configured state the consultant's list omits.++Premise checks recorded per card (verified against this tree, HEAD ef974e0d):++- t_2fdcb71f: verified in substance. Corrections: no `STORAGE_CAPACITY` error path exists today (the vocabulary is new); the card's `ingest.ts:337` anchor for the pre-extract cap is stale (now `ingest.ts:500`, already validator-corrected); the upstream warn/fail percentages are adopted as config-default constants that apply only when the operator configures a cap.+- t_2dd5bed9: verified. Exact-duplicate folding exists in search output; document-level keep-newest retirement with source-link dropping does not. The keep-newest decision must require `created_at` (the mtime fallback is merge-unstable, the recorded GitHub #180 hazard) - encoded as a design decision.+- t_a9cd4221: verified for the carved residue. No offset, size or mtime watermark exists anywhere in the ingest lane; `parseEntryValue` drops unknown field shapes; `readTail` (`src/cli/aider.ts:157-166`) is the appended-bytes precedent. The card's ceiling-raise and deferred-embedding halves are out of scope (spend gates already exist); no `EXTRACT_CONTRACT_VERSION` bump - the watermark degrades lazily toward reprocessing instead.+- t_d7071804: verified for the carved residue. The page-dedup CLI apply loop has no per-item error handling and `deleteBySource` rethrows after persisting partial progress; the soft-delete/revival lifecycle is explicitly out of scope and the card closes only for the delivered decomposition scope.diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex 9a5d55f8..87b69f98 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -533,7 +533,7 @@ o2b brain session-grep --query <text> [--session-id <id>] [--limit <n>] [o2b brain session-describe --session-id <id> [--topics] [--json]o2b brain session-expand <record-id> [--raw-limit <n>] [--cursor <offset>] [--json]o2b brain handoff <session-file> [--session-id <id>] [--format auto|claude|codex|hermes] [--json] - write Brain/handoffs/<date>-<scope>.md (since v0.37.0)-o2b brain hygiene scan | apply --ids <id,...> [--detectors conflicts,dedup,freshness,usefulness,slug-collisions,tags,capture-scope] [--dry-run] [--json] - hygiene findings pipeline; review findings never execute (since v1.3.0); the default sweep runs every detector except the opt-in ones (since v1.64.0: `slug-collisions` is default-on and reports same-stem groups such as topic.md beside topic-2.md as info findings, `tags` is opt-in and audits inline body tags only - frontmatter tags arrays are out of scope; since v1.67.0 `capture-scope` is default-on and warns, with proposed action review, when a retrievable page of active knowledge cites only sources that are currently url-only; since v1.81.0 the detector also names a cited source whose durable archive entry under Brain/.archives/ is missing or no longer matches the hash stamped on the page)+o2b brain hygiene scan | apply --ids <id,...> [--detectors conflicts,dedup,document-dedup,freshness,usefulness,slug-collisions,tags,capture-scope] [--dry-run] [--json] - hygiene findings pipeline; review findings never execute (since v1.3.0); the default sweep runs every detector except the opt-in ones (since v1.64.0: `slug-collisions` is default-on and reports same-stem groups such as topic.md beside topic-2.md as info findings, `tags` is opt-in and audits inline body tags only - frontmatter tags arrays are out of scope; since v1.67.0 `capture-scope` is default-on and warns, with proposed action review, when a retrievable page of active knowledge cites only sources that are currently url-only; since v1.81.0 the detector also names a cited source whose durable archive entry under Brain/.archives/ is missing or no longer matches the hash stamped on the page; since v1.85.0 `document-dedup` is opt-in - near-duplicate ingest summary copies at a lexical similarity of 0.8 or an embedding similarity of 0.97, keeping the newest `created_at` copy as the canonical, a page without `created_at` ineligible - because its remediation is the destructive-class `retire` action, which `apply` routes through the retirement core: inbound wikilinks retarget to the canonical within the detector's scope, the copy's manifest entry drops, one `source_invalidation` audit records each copy, and the whole span sits behind the destructive-snapshot gate, failing soft per copy)o2b brain hygiene scan: with the optional `dedup` decision-model use in enforce, dedup findings carry an advisory `decision_model` verdict and a confident `different` is listed last; `o2b brain doctor` annotates `entity-alias-candidate` warnings the same way; `apply` ignores verdictso2b brain refresh --stale [--dry-run] [--json] - targeted recompile of stale derived pages; orphans archive into Brain/.snapshots (since v1.3.0)o2b brain anticipate --session <id> [--refresh] [--signal <text>] [--json] - read or warm the anticipatory context cache for the session's lineage root (since v1.3.0)@@ -655,7 +655,7 @@ o2b brain maintenance run --cron-template [--interval N] [--format cron|The schema pack gains four additive ontology fields (`labels`, `link_constraints`, `attributes`, `frontmatter_tiers`) with audited mutations through `o2b brain schema apply`. Link constraints enforce at index materialization: a typed edge whose endpoint page types violate the declared pairs falls back to an untyped link, `o2b brain schema lint` lists each violation, and removing the constraint restores the edges on the next index run. Tier drift detection rides the same index pass - the snapshot keeps the expected value, so reindexes never absorb a hand-edit, and `brain_doctor` warns with the open count. Filter labelled recall with `o2b search <q> --property labels=<dim>/<value>`. Secrets protect against context leakage and vault sync exposure, not against root; every custody operation lands a no-values record in `Brain/log/secret-custody/`. A maintenance gate skip exits 0 so cron never alarms on a quiet hour.-The lane's two vault-side knobs live in `Brain/_brain.yaml` under `maintenance:`, because they answer what the cron line cannot. `host_pressure_percent` adds a fourth gate: skip when the host's one-minute run queue stands at or above that percentage of the CPUs this process may use. Unset by default, which leaves the gate off. Where the metric is degenerate - a platform whose load average is a constant, or a cgroup with a CPU bandwidth quota, where the run queue is the whole host's - the gate stays **open** and the journal carries a separate `pressure:unmeasurable` row naming the reason, so an unreadable host is never reported as a quiet one. `failure_streak_limit` (default 3) refuses a lane task that has failed that many times in a row in the run journal, naming the streak; a single journaled success clears it. The refusal is per task - the other three still run under the same lease - and there are two ways past it: `--retry <task>` (repeatable) attempts just that task with every gate the operator configured still in force, and `--force` runs the whole lane past every soft gate and every refusal. Both escapes exist on the MCP surface too - `brain_maintenance` takes `retry_tasks`, `busy_minutes`, `busy_threshold` and a `status` `limit`, so an agent reading the refusal can act on it without reaching for `force`, which switches off three gates it never meant to touch. A refused task is reported as `REFUSED`, never `FAILED`, and the run exits **7** rather than 1: nothing was attempted, so nothing failed, but a standing refusal is not the quiet hour that exits 0 either. An attempted failure still exits 1 and outranks a refusal in the same run. The `refused:streak` journal row carries the count it refused on, so the streak does not silently reset when its evidence rolls off the journal cap.+The lane's vault-side knobs live in `Brain/_brain.yaml` - the two original ones under `maintenance:`, the storage cap and its thresholds under `storage:` (since v1.85.0, see "Storage lifecycle" below) - because they answer what the cron line cannot. `host_pressure_percent` adds a gate over the window and busy checks: skip when the host's one-minute run queue stands at or above that percentage of the CPUs this process may use. Unset by default, which leaves the gate off. Where the metric is degenerate - a platform whose load average is a constant, or a cgroup with a CPU bandwidth quota, where the run queue is the whole host's - the gate stays **open** and the journal carries a separate `pressure:unmeasurable` row naming the reason, so an unreadable host is never reported as a quiet one. Since v1.85.0 a configured storage cap adds the lane's fifth gate, evaluated last because it is the most expensive: a fresh byte census of the whole vault, skipping the pass at a measured critical tier with its own `skipped:capacity` verdict, the journal row carrying the tier and the reading behind it while the skipped run still appends its usage sample first. A capacity skip is a gate skip like any other - it exits 0 - and where the caller has no local reach the `status` surface renders the tier word and the unmeasurable reason without the byte-and-percent figures. `failure_streak_limit` (default 3) refuses a lane task that has failed that many times in a row in the run journal, naming the streak; a single journaled success clears it. The refusal is per task - the other three still run under the same lease - and there are two ways past it: `--retry <task>` (repeatable) attempts just that task with every gate the operator configured still in force, and `--force` runs the whole lane past the four soft gates - window, busy, pressure and capacity - and every streak refusal, never past the lease. Both escapes exist on the MCP surface too - `brain_maintenance` takes `retry_tasks`, `busy_minutes`, `busy_threshold` and a `status` `limit`, so an agent reading the refusal can act on it without reaching for `force`, which widens its bypass to every soft gate and every streak refusal. A refused task is reported as `REFUSED`, never `FAILED`, and the run exits **7** rather than 1: nothing was attempted, so nothing failed, but a standing refusal is not the quiet hour that exits 0 either. An attempted failure still exits 1 and outranks a refusal in the same run. The `refused:streak` journal row carries the count it refused on, so the streak does not silently reset when its evidence rolls off the journal cap.Since v1.64.0 the lane's timeouts are honest and its embedding spend is named. A task killed at its safeguard deadline exits **6** (`probeIncomplete`), keyed on `timed_out` rows alone: the only proved fact is that the pass did not finish, which is not the same as a task failing; a proved failure (1) outranks a timeout, a timeout outranks a refusal (7), and the render says TIMED OUT with the safeguard detail. The reindex task stays keyword-only unless `maintenance_embeddings: true` (env `OPEN_SECOND_BRAIN_MAINTENANCE_EMBEDDINGS`, default `false`) opts it in; then it requests the embedding phase whenever the resolved semantic config can reach a provider, announces the pending-spend estimate before the pass, and journals the phase's own cost-gate result as the per-run receipt - model, tokens, estimated cost, and whether a bypass fired - on the task row, the journal line and the `maintenance_spend` metrics surface; the preview is an estimate by position, the receipt prices what the completed pass embedded, and a run killed mid-spend receipts nothing. `--force-cost` (MCP `force_cost`) bypasses a positive `embedding_cost_gate_usd` for this run, recorded on the receipt when it overrode a gate that would have refused. A safeguard timeout is journaled as `timed_out` and neither counts toward nor resets the failure streak that refuses a task.@@ -1358,6 +1358,68 @@ themselves: `--query`, `--since`, `--before`, `--limit` and`--snippet-chars` refuse under coverage, and `--char-budget` refuses undermatch.+### Storage lifecycle (since v1.85.0)++```text+o2b brain storage [--vault V] [--json] - the vault storage report: what the vault holds per surface (measured or refused by name), the forecast tier against the configured cap with the percent and days figures it rests on, and the storage_usage sample this invocation records; a report never fails on a finding, so a critical tier still exits 0+o2b brain receipts scan [--older-than-days N] [--json] | compact --shard <family:name> [--older-than-days N] [--apply] [--expect N] [--strict] [--json] - receipt compaction over the continuity and metrics ledgers; scan is the default and a pure read; compact folds expired records into per-kind, per-month aggregates in the named shards and previews without --apply; scan exits 1 when a shard error was collected, compact exits 1 when a fold failure was collected, and a usage error exits 2+```++An uncapped vault is unbounded: `o2b brain storage` reports the bytes and+the per-surface census, states `unbounded` plainly, invents no+days-to-exhaustion figure, and nothing pauses and nothing refuses.+Setting `storage.cap_bytes` in `Brain/_brain.yaml` turns on the tiers and+every consumer of them; the four thresholds (`warn_percent`,+`critical_percent`, `warn_days`, `critical_days`) default to 80 and 95+percent of the cap and 30 and 7 days to it at the measured trend, and the+critical band must sit past the warn band - a larger percent, a smaller+day horizon. Each invocation takes one census and appends one+`storage_usage` sample; from two or more samples a linear trend yields+the days-to-capacity figure, and a tier is decided by the worse of its+two bands. The census covers `brain_tree` and the dot-directories+(`.snapshots/`, `.artifacts/`, `.archives/`, `.payloads/`, `.state/`)+plus the derived store with its WAL/SHM sidecars; a surface that cannot+be measured is refused by name, never reported as zero. Below local+reach the census is not taken: the report answers `measured: false` with+the reason named and carries no figures. The CLI runs at local reach, so+it always sees the measured form.++The consumers are the maintenance lane's fifth gate (see "Write-time+integrity and governance" above): a fresh census at decision time, never+the stored aggregate, skipping the pass at a measured critical tier with+its own `skipped:capacity` verdict; `brain_ingest_source`'s admission+refusal, which answers `capacity_exhausted` in the validate phase before+the write path opens, so a refused ingest leaves the vault+byte-identical; and the `storage-capacity` doctor check, which reports+fullness for a capped vault and answers in the uncertain stream that the+reading was not taken when it cannot take one. A second doctor check+watches the ledgers rather than the vault: `receipt-ledger-growth` warns+per family once its foldable population crosses 1,000 records, naming+`o2b brain receipts scan` as the next command, and reports a ledger it+could not read in the uncertain stream - growth unknown, never flattened+to growth fine.++Receipt compaction gives the two append-only receipt ledgers - the+continuity ledger and the metrics ledger - a drain. `scan` inventories+every shard read-only (records, foldable, protected, unparseable,+aggregates, bytes) and names each shard it could not read; `compact`+folds the records older than `--older-than-days` (default 30) into+per-kind, per-month aggregate lines inside the shards the caller names+with `--shard family:name` ids from a scan - a compact with no shard ids+is a usage error, never a guess. It previews unless `--apply` is passed;+a confirmed run takes a recovery-point snapshot first and rewrites each+shard atomically. `--expect N` asserts exactly N records will fold and+aborts without writing on a mismatch; `--strict` refuses a confirmed+compact that carries no count guard. Records in the protected classes+never fold whatever their age - the `source_invalidation` audit trail+and the two context-compaction kinds - and one `receipt_compaction` audit+record closes a confirmed run. Compact is a local-reach surface (the+`brain_receipts` tool refuses it by name below local reach, while scan+stays readable at any reach); the CLI always runs at local reach. This+verb owns storage-ledger compaction only: prompt-context receipts stay+with `o2b brain context-receipts`, and the Hermes context-compaction+kinds are never folded under a storage operation's name.+### Source distillation```textdiff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex 0cd1ce9e..16116d6a 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -1619,7 +1619,7 @@ Until now almost nothing about a write into the Brain was declared: any frontmat**The keyfile can be wrapped under a passphrase.** `o2b brain secret unlock` verifies a passphrase (read from stdin or `--passphrase-from-env`, never argv) and - on a store whose keyfile is still raw - wraps the 32-byte key into a versioned scrypt envelope at the same path, then holds the unwrapped key in memory for that one process only; `lock` overwrites and drops it. There is no daemon: every other CLI invocation and the MCP server stay locked, and a key-bearing operation over a locked store refuses with a named error whose remedy is the unlock command (the refusal's prose carries no paths - the structured field does, for the surfaces allowed to name them). Every unlock and lock lands a no-values custody record. A credential reference, `$secret:NAME`, resolves store-first with environment fallback at the use sites that already name it (the exec allowlist, the embedding and rerank keys, the decision-model key, the Telegram capture token, the research keys); a locked store surfaces the named refusal rather than silently answering from the environment. `secret export` writes every entry as one passphrase-encrypted bundle to an operator-named file - the values re-encrypted under a key the passphrase derives, the names, env-var mappings and allowlists traveling in the clear past the shared egress redactor - and `secret import` decrypts and validates every entry (names, env-var mappings, allow patterns, all under the store's own rules) before writing anything. The honesty clause is the same for the keyfile envelope and for bundles: the passphrase is never persisted, logged or audited, and a lost one is unrecoverable - every stored value and every exported bundle stays unreadable forever, and nothing recovers them.-**Heavy work waits its turn.** The maintenance lane runs dream and reindex behind three gates: a tz-aware local-time window, a busy gate over recent recall telemetry, and an expiring SQLite lease in its own database so holding it never contends with the search index writer lock. Force bypasses the soft gates but never the lease, tasks run stale-first, and every attempt - including each gate refusal - lands in a bounded journal, because a lane you cannot audit is a lane you cannot trust.+**Heavy work waits its turn.** The maintenance lane runs dream and reindex behind five gates: a tz-aware local-time window, a busy gate over recent recall telemetry, an optional host-pressure gate, and - since v1.85.0, when a storage cap is configured - a capacity gate that skips a vault measuring at its critical tier before the failure-streak limit refuses a task that keeps failing; under them all sits an expiring SQLite lease in its own database so holding it never contends with the search index writer lock. Force bypasses the soft gates but never the lease, tasks run stale-first, and every attempt - including each gate refusal - lands in a bounded journal, because a lane you cannot audit is a lane you cannot trust.## Link and recall intelligence (since v0.45.0)diff --git a/docs/mcp.md b/docs/mcp.mdindex cf063240..b0615cf4 100644--- a/docs/mcp.md+++ b/docs/mcp.md@@ -351,7 +351,7 @@ unchanged.## Tool Highlights-The full server currently advertises 115 tools; the 18 deprecated predecessor+The full server currently advertises 117 tools; the 18 deprecated predecessornames were removed in 1.0.0 and now answer a precise INVALID_PARAMS tombstone(see "Consolidated views and deprecated aliases" below). The table highlightsthe operator-facing core,@@ -395,6 +395,8 @@ flags for a narrower per-process full server.| `brain_context_presets` | Show, suggest, or diff read-only context budget presets (`tight-context`, `long-context`) without writing config. | `operation` || `brain_pre_compact_extract` | Extract decision/commitment/outcome/rule/open-question records from bounded text into continuity storage. | `session_id`, `turn_start`, `turn_end`, `text` || `brain_hygiene` | Memory hygiene: `scan` findings (conflicts, dedup, freshness, usefulness; since v1.64.0 `slug-collisions` is default-on — same-stem allocation residue such as `topic.md` beside `topic-2.md` — and `tags` is opt-in, auditing inline body tags only, never frontmatter tags arrays; since v1.67.0 `capture-scope` is default-on and warns when active knowledge rests only on `url-only` sources; the default is the sweep of every registered detector except the opt-in ones), `apply` selected ids, `refresh` stale pages; since v1.81.0 `archive_get` resolves an ingest summary's `source_archive_hash` to its archived bytes under `Brain/.archives/` (local reach only, and the `capture-scope` scan also names a cited source whose archive entry is missing or mismatched). Resolver command comes from `_brain.yaml` only. When `hygiene_digest_enabled` is on in the machine config, the same default-sweep findings are also pushed at turn end as a one-line digest (Stop hook, once per change), so this pull surface doubles as a push channel. With the optional `dedup` decision-model use in enforce, `scan` dedup findings carry an advisory `decision_model` verdict; `apply` never reads it. | `mode` |+| `brain_storage` | Read-only storage report (since v1.85.0): a per-surface byte census measured or refused by name, the growth forecast against the configured `storage.cap_bytes` with its tier (`ok`/`warn`/`critical`, or `unbounded`/`unmeasurable`/`no_forecast`) and the percent and days figures it rests on, and the `storage_usage` sample this invocation records. Below local reach the census is not taken: the answer carries `measured: false` and the reason, no figures. A report, never a gate: a critical tier still answers successfully. | — |+| `brain_receipts` | Receipt compaction over the continuity and metrics ledgers (since v1.85.0): `scan` (the default) inventories both ledgers read-only per shard - records, foldable, protected, unparseable, aggregates, bytes; `compact` folds expired records into per-kind, per-month aggregates inside `family:name` shards named from a scan, preview-bound without `apply`, local reach only. `expect` asserts the fold count and aborts without writing on a mismatch; `strict` refuses a guardless confirmed compact. Never touches decision-change receipts, the source archive, or context compaction. | — (`shards` for `compact`) || `brain_anticipatory_context` | Turn-specific context bundle kept warm by lifecycle hooks, keyed by the session's lineage root; reports `cache_state` warm / stale / miss. | `session_id` || `brain_session_grep` | Search imported session recall raw turns and deterministic summary nodes; since v1.82.0 `operation: "coverage"` runs the coverage-first thread selection under a `char_budget` instead of a grep. | `query` || `brain_session_describe` | Describe raw-turn counts and summary depths for one imported session recall DAG; `topics: true` adds the adaptive topic-budget report the coverage operation fans out under (since v1.82.0). | `session_id` |diff --git a/docs/metrics.md b/docs/metrics.mdindex e17254d0..f3e6a670 100644--- a/docs/metrics.md+++ b/docs/metrics.md@@ -84,6 +84,7 @@ writes, and a metric record is a summary, not a report.| `self_heal_reindex` | `ensureVaultCurrent` (parent) and `o2b search reindex --self-heal <run-id>` (child) | parent row: `decision` (`spawned` / `skipped_writer_lock`), `run_id` (on `spawned`). Child row: `outcome` (`completed` / `failed`), `run_id`, `duration_ms`, `error` (on `failed`) || `index_freshen` (since v1.76.0) | `o2b search index --freshen <token>` (the background run freshen on read starts) | `outcome` (`completed` / `failed`), `duration_ms`, `changed` (on `completed`), `error` and `failures` (on `failed`). A row is written only when the run changed documents or failed || `maintenance_spend` (since v1.64.0) | `o2b brain maintenance run`, MCP `brain_maintenance` (`run`) | `task` (`reindex`), `model` (null when the config names none), `tokens`, `estimated_usd`, `forced` (`force-cost` overrode a positive gate), `lane` (always true: both front doors are the lane) |+| `storage_usage` (since v1.85.0) | `o2b brain storage`, MCP `brain_storage` (every report invocation), and a maintenance-lane run the capacity gate skips (sampled before the skip, from the gate's own census) | `run_id`, `total_bytes`, `bytes_by_tier` (per recoverability tier: `vault-content`, `derived`), `measured` (true when every surface answered), `unmeasured` (the surfaces refused, each with its reason). The series is the forecast's input: `o2b brain storage` reports its linear trend, and the run id is the idempotency key, so one invocation never writes a second row |Payload fields marked "(lane runs)" appear only on maintenance-laneemissions. All fields are additive-optional from a consumer's pointdiff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex a31df6ac..a2fdc345 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.84.1",+ "version": "1.85.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex 8dbad6a5..99fe488c 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.84.1",+ "version": "1.85.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 92120983..e6b2e6d5 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.84.1"+version: "1.85.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 2e2bb3f0..2588a657 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.84.1",+ "version": "1.85.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/codex/README.md b/plugins/codex/README.mdindex 53b88dda..ba52422d 100644--- a/plugins/codex/README.md+++ b/plugins/codex/README.md@@ -123,7 +123,7 @@ The full router with readiness criteria is [`install.md`](https://github.com/ite## What is new-1.83.0 wires four shipped seams, each an opt-in that is byte-identical when off. Search can cap how many rows one source import session holds in the reranker-facing candidate window (`search_pool_max_per_session` / `OPEN_SECOND_BRAIN_SEARCH_POOL_MAX_PER_SESSION`, default `0`): the cap sits after the relevance floor and before the MMR diversity rerank and widens the pool rather than truncating it, so a long imported session stops crowding out everything else, while a vault without session frontmatter never sees a difference. `skills_attach` can order its offers by observed invocation usage (`skills_attach_usage_ordering` / `OPEN_SECOND_BRAIN_SKILLS_ATTACH_USAGE_ORDERING`, default off): a bounded composite, `score * (1 + 0.5 * weight)`, lifts a daily skill over a slightly hotter one-off without ever crossing the discriminating-term floor; plugin-qualified invocation names join discovered skills through a deterministic longest-suffix rule, private records never count, and the returned scores stay lexical. `o2b brain bench memory --suite longmemeval` accepts `--sample-seed` plus `--sample-cap` to score a seeded, question-type-proportional subset of the dataset instead of the whole file - deterministic given the seed, immune to dataset file order, receipt-reported (`sample_population` / `sample_selected` / `sample_strata`), and folded into the run identity so a resumed run whose sampling options changed fails the identity guard by name. The opencode install learns OpenCode V2's native `mcp.servers` config shape and `.jsonc` config files: comment-bearing files are edited byte-preserving, the legacy top-level shape stays untouched, and a one-time drift after the host adds `mcp.servers` is documented with its fix. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).+1.85.0 gives a long-lived vault the means to watch its own storage and to keep its append-only ledgers in shape. `o2b brain storage` (and the `brain_storage` tool) measures a per-surface byte census and forecasts days to the configured `storage.cap_bytes` in graduated tiers; a measured critical tier pauses the maintenance lane behind its fifth gate and refuses interactive ingests with `capacity_exhausted` before any write, while an uncapped vault reads unbounded, nothing pauses and nothing refuses. `o2b brain receipts` (and `brain_receipts`) drains the append-only continuity and metrics ledgers by folding expired records into per-kind, per-month aggregates behind an explicit preview-bound apply, with a count guard and a recovery point around every confirmed run. An opt-in `document-dedup` hygiene detector nominates near-duplicate document copies for retirement behind the review gate, keeping the newest copy, retargeting the stale copies' inbound links and auditing each retirement; bulk source removals report per-file failures instead of aborting; and appended sources yield their new text through incremental extraction over a manifest watermark, no whole-source reprocess. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).## Documentationdiff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 92120983..e6b2e6d5 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.84.1"+version: "1.85.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex f86fbe71..a1ae18bc 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.84.1"+version = "1.85.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/brain.ts b/src/cli/brain.tsindex 4298227e..b36a4c63 100644--- a/src/cli/brain.ts+++ b/src/cli/brain.ts@@ -105,6 +105,8 @@ import {cmdBrainSgrep,cmdBrainContinuity,cmdBrainPayload,+ cmdBrainReceipts,+ cmdBrainStorage,cmdBrainSessionSummary,cmdBrainIdeaLineage,cmdBrainNoteHistory,@@ -401,6 +403,10 @@ export async function handleBrainSubcommand(argv: ReadonlyArray<string>): Promisreturn await cmdBrainContinuity(rest);case "payload":return await cmdBrainPayload(rest);+ case "receipts":+ return await cmdBrainReceipts(rest);+ case "storage":+ return await cmdBrainStorage(rest);case "session-summary":return await cmdBrainSessionSummary(rest);case "idea-lineage":diff --git a/src/cli/brain/verbs/extract.ts b/src/cli/brain/verbs/extract.tsindex 9cfb07b4..2f3097ea 100644--- a/src/cli/brain/verbs/extract.ts+++ b/src/cli/brain/verbs/extract.ts@@ -1,6 +1,6 @@/**- * `o2b brain extract <file> [--json]`: preview what ingest derives from one- * source file, for every registered format.+ * `o2b brain extract <file> [--json] [--vault <vault>]`: preview what ingest+ * derives from one source file, for every registered format.** An HTML file answers its title, its text and its heading-derived parts; a* CSV or TSV file answers its table note (the counts and the rendered@@ -10,9 +10,17 @@* the bounded reader refuses (not a regular file, larger than the read* limit). "Could not extract" is data: every such answer exits 0. Read-only* and deterministic; nothing is written.+ *+ * With a resolvable vault (`--vault`, else the configured one) and a source+ * the ingest manifest has already extracted, the preview also reports the+ * `appended` slice: the boundary-safe byte range the file grew by since its+ * recorded extraction watermark. It is additive data for an incremental+ * pass - a source with no watermark, a rewritten source, or no vault at all+ * answers without the section, never with a fake range.*/import { lstatSync } from "node:fs";+import { isAbsolute, relative, sep } from "node:path";import { oneLine } from "../../../core/brain/architect/manifests.ts";import {@@ -21,7 +29,12 @@ import {type HtmlPart,} from "../../../core/brain/ingest/extract-html.ts";import { extractSource, type SourceExtraction } from "../../../core/brain/ingest/extract-source.ts";-import { readSourceBounded, type SourceUnread } from "../../../core/brain/ingest/ingest.ts";+import {+ appendedExtractionDelta,+ readSourceBounded,+ type SourceUnread,+} from "../../../core/brain/ingest/ingest.ts";+import { canonicalNotePath } from "../../../core/path-safety.ts";import {SOURCE_EXTRACT_SKIP_REASON,SOURCE_EXTRACTOR,@@ -30,9 +43,9 @@ import {sourceFormatSpec,type SourceExtractSkipReason,} from "../../../core/brain/ingest/source-formats.ts";-import { fail, ok, okJson, parse, usageError } from "../helpers.ts";+import { brainVerbContext, CliError, fail, ok, okJson, parse, usageError } from "../helpers.ts";-const USAGE = "usage: o2b brain extract <file> [--json]";+const USAGE = "usage: o2b brain extract <file> [--json] [--vault <vault>]";/** The bytes handed to the dispatch for a format answered by name. */const NO_BYTES = new Uint8Array(0);/** How the text output names the format of an unregistered extension. */@@ -72,6 +85,50 @@ function extractFile(file: string): SourceExtraction {return extractSource(file, read.bytes);}+/** The appended slice a previously-extracted source grew by, in bytes. */+interface AppendedSlice {+ readonly fromByte: number;+ readonly toByte: number;+}++/**+ * The vault the appended section answers against, when one resolves. The+ * preview itself is vault-independent - it answers any file - so an+ * unresolvable vault is not an error here; it only means there is no ingest+ * manifest to consult and the section stays absent.+ */+function resolveOptionalVault(+ flags: Record<string, string | boolean | string[] | undefined>,+): string | null {+ try {+ return brainVerbContext(flags).vault;+ } catch (err) {+ if (err instanceof CliError) return null;+ throw err;+ }+}++/**+ * The appended slice `file` grew by since its recorded extraction watermark,+ * when the file sits inside `vault` and the manifest records one. The+ * pipeline's own delta pass answers this (`appendedExtractionDelta`), so the+ * preview reports the same fact the pipeline would: a source whose leading+ * watermark bytes no longer match the recorded digest is a rewrite, not an+ * append, and answers undefined exactly as the ingest's delta report does -+ * as do a file outside the vault, no watermark, a tail with no complete+ * line yet, and a tail past the pipeline's delta cap. The section is+ * additive data for an incremental pass, never a second verdict on the+ * extraction.+ */+function appendedSlice(vault: string | null, file: string): AppendedSlice | undefined {+ if (vault === null) return undefined;+ const rel = relative(vault, file);+ if (rel.length === 0 || isAbsolute(rel) || rel === ".." || rel.startsWith(`..${sep}`)) {+ return undefined;+ }+ return appendedExtractionDelta(vault, file, canonicalNotePath(rel.split(sep).join("/")));+}+/** The wire shape of one HTML part. */function partJson(part: HtmlPart): Record<string, unknown> {return {@@ -161,20 +218,35 @@ function extractionText(path: string, extraction: SourceExtraction): string[] {return [...lines, notExtractedLine(extraction.reason)];}+/** The text line naming the appended slice, for the non-JSON output. */+function appendedLine(appended: AppendedSlice): string {+ return ` appended: byte ${appended.fromByte}..${appended.toByte} since the recorded extraction watermark`;+}+export async function runBrainExtract(argv: string[]): Promise<number> {const { flags, positional } = parse(argv, {json: { type: "boolean" },+ vault: { type: "string" },});const file = positional[0];if (!file) return usageError(USAGE);try {const extraction = extractFile(file);+ // A corrupted manifest fails the command by name, like every ingest-lane+ // read of it; everything else about the appended section is additive.+ const appended = appendedSlice(resolveOptionalVault(flags), file);if (flags["json"]) {- okJson(extractionJson(file, extraction));+ const payload = extractionJson(file, extraction);+ if (appended !== undefined) {+ payload["appended"] = { from_byte: appended.fromByte, to_byte: appended.toByte };+ }+ okJson(payload);return 0;}- for (const line of extractionText(file, extraction)) ok(line);+ const lines = extractionText(file, extraction);+ if (appended !== undefined) lines.push(appendedLine(appended));+ for (const line of lines) ok(line);return 0;} catch (err) {return fail((err as Error).message ?? String(err));diff --git a/src/cli/brain/verbs/index.ts b/src/cli/brain/verbs/index.tsindex 860f55e6..5df60b9a 100644--- a/src/cli/brain/verbs/index.ts+++ b/src/cli/brain/verbs/index.ts@@ -86,6 +86,8 @@ export { cmdBrainLinks } from "./links.ts";export { cmdBrainProfile } from "./profile.ts";export { cmdBrainSgrep } from "./sgrep.ts";export { cmdBrainContinuity } from "./continuity.ts";+export { cmdBrainReceipts } from "./receipts.ts";+export { cmdBrainStorage } from "./storage.ts";export { cmdBrainPayload } from "./payload.ts";export { cmdBrainSessionSummary } from "./session-summary.ts";export { cmdBrainIdeaLineage } from "./idea-lineage.ts";diff --git a/src/cli/brain/verbs/maintenance.ts b/src/cli/brain/verbs/maintenance.tsindex 2f5978ee..ff0ab9dd 100644--- a/src/cli/brain/verbs/maintenance.ts+++ b/src/cli/brain/verbs/maintenance.ts@@ -2,7 +2,8 @@* `o2b brain maintenance <run|status>` (t_166d1226): the quiet-window* lane for heavy passes. `run` gates on the local-time window* (--window H-H, unset = always open), recent interactive query-rate,- * and the expiring SQLite lease, then executes dream, reindex,+ * host pressure, the vault's storage-capacity forecast, and the+ * expiring SQLite lease, then executes dream, reindex,* bridges, and clusters stale-first; --force bypasses the soft gates* and every streak refusal but never the lease, while --retry <task>* (repeatable) bypasses the streak refusal for that task alone and@@ -419,10 +420,17 @@ export async function cmdBrainMaintenance(argv: string[]): Promise<number> {okJson({verdict: result.verdict,tasks: result.tasks,+ // The capacity gate's report on a skipped:capacity run: the+ // same block the MCP tool attaches, from the same lane result,+ // so the two surfaces cannot disagree about what a skip says.+ ...(result.capacity !== undefined ? { capacity: result.capacity } : {}),...(spend !== undefined ? { spend } : {}),});} else {ok(`maintenance: ${result.verdict}`);+ if (result.capacity !== undefined) {+ ok(` capacity: ${result.capacity.tier} - ${result.capacity.remedy}`);+ }for (const t of result.tasks) ok(` ${renderTaskLine(t)}`);}// A stopped lane is reported as stopped, not as four failures. Thediff --git a/src/cli/brain/verbs/page-dedup.ts b/src/cli/brain/verbs/page-dedup.tsindex 0597c86e..ef52f473 100644--- a/src/cli/brain/verbs/page-dedup.ts+++ b/src/cli/brain/verbs/page-dedup.ts@@ -5,12 +5,90 @@* Default mode is read-only listing. `--apply` performs the merge* for every reported candidate: writes `merged_into:` on each* secondary and rewrites `[[<secondary>]]` references across the- * vault to point at the canonical page.+ * vault to point at the canonical page. A merge attempt that refuses+ * (a MergeChainError) or cannot write its links is collected and+ * reported per cluster; the remaining clusters still apply, and the+ * run exits 1 when any failure was collected.*/-import { findDuplicateCandidates, mergePage } from "../../../core/brain/page-dedup.ts";+import type { DedupCandidate } from "../../../core/brain/page-dedup.ts";+import {+ findDuplicateCandidates,+ mergePage,+ PatchWikilinksError,+} from "../../../core/brain/page-dedup.ts";import { brainVerbContext, fail, okJson, parse } from "../helpers.ts";+/**+ * One merge attempt that refused to land: the cluster's dedup key and the+ * named error. A chain refusal (MergeChainError) writes nothing, so the+ * secondary is left exactly as it was. A link-pass refusal can arrive+ * AFTER the `merged_into:` stamp has landed on the secondary - by then+ * the merge is half-applied, and {@link DedupApplyFailure.wikilinksUpdated}+ * carries how many files the rewrite repointed before the write that+ * refused.+ */+export interface DedupApplyFailure {+ readonly key: string;+ readonly message: string;+ /**+ * Files the link pass rewrote before the write that refused, present+ * only when the failure came after the rewrite had already run. Absent+ * means the secondary is untouched, not zero.+ */+ readonly wikilinksUpdated?: number;+}++/** What one apply pass over a set of candidates did. */+export interface DedupApplyReport {+ readonly merged: number;+ readonly wikilinksUpdated: number;+ /** Per-attempt failures, in cluster order; empty on a clean apply. */+ readonly failures: ReadonlyArray<DedupApplyFailure>;+}++/**+ * Merge every secondary into its canonical, collecting per-attempt failures+ * instead of throwing: one cluster's MergeChainError (or one blocked wikilink+ * write) leaves every later cluster still applied. The report carries the+ * honest counts alongside the failures, so the caller can exit non-zero+ * without re-deriving what landed.+ */+export function applyDedupCandidates(+ vault: string,+ candidates: ReadonlyArray<DedupCandidate>,+): DedupApplyReport {+ let mergedCount = 0;+ let wikilinksTouched = 0;+ const failures: DedupApplyFailure[] = [];+ for (const c of candidates) {+ for (const s of c.secondaries) {+ try {+ const res = mergePage(vault, s.id, c.canonical.id);+ mergedCount++;+ wikilinksTouched += res.wikilinksUpdated;+ } catch (err) {+ // A late failure (PatchWikilinksError) means the stamp landed and+ // part of the rewrite did too: keep the partial count on the+ // failure and fold it into the aggregate, which counts files the+ // vault actually took, not only the ones behind a clean merge.+ const partial = err instanceof PatchWikilinksError ? err.rewritten : undefined;+ if (partial !== undefined) wikilinksTouched += partial;+ failures.push({+ key: c.key,+ message: (err as Error)?.message ?? String(err),+ ...(partial === undefined ? {} : { wikilinksUpdated: partial }),+ });+ }+ }+ }+ return {+ merged: mergedCount,+ wikilinksUpdated: wikilinksTouched,+ failures: Object.freeze(failures),+ };+}+export async function cmdBrainPageDedup(argv: string[]): Promise<number> {const { flags } = parse(argv, {vault: { type: "string" },@@ -55,26 +133,31 @@ export async function cmdBrainPageDedup(argv: string[]): Promise<number> {return 0;}- let mergedCount = 0;- let wikilinksTouched = 0;- for (const c of report.candidates) {- for (const s of c.secondaries) {- const res = mergePage(vault, s.id, c.canonical.id);- mergedCount++;- wikilinksTouched += res.wikilinksUpdated;- }- }+ const applied = applyDedupCandidates(vault, report.candidates);+ // Collected failures are an honest non-zero exit, not a crash: every+ // merge that could land did land, and the failures name what did not.+ const exitCode = applied.failures.length > 0 ? 1 : 0;if (flags["json"]) {okJson({scanned: report.scanned,clusters: report.candidates.length,- merged: mergedCount,- wikilinks_updated: wikilinksTouched,+ merged: applied.merged,+ wikilinks_updated: applied.wikilinksUpdated,+ failures: applied.failures.map((f) => ({+ key: f.key,+ message: f.message,+ ...(f.wikilinksUpdated === undefined ? {} : { wikilinks_updated: f.wikilinksUpdated }),+ })),});- return 0;+ return exitCode;+ }+ process.stdout.write(`merged ${applied.merged} page(s) into canonicals.\n`);+ process.stdout.write(`wikilinks updated: ${applied.wikilinksUpdated}\n`);+ for (const f of applied.failures) {+ const partial =+ f.wikilinksUpdated === undefined ? "" : ` (wikilinks updated first: ${f.wikilinksUpdated})`;+ process.stdout.write(`failed cluster ${f.key}: ${f.message}${partial}\n`);}- process.stdout.write(`merged ${mergedCount} page(s) into canonicals.\n`);- process.stdout.write(`wikilinks updated: ${wikilinksTouched}\n`);- return 0;+ return exitCode;}diff --git a/src/cli/brain/verbs/receipts.ts b/src/cli/brain/verbs/receipts.tsnew file mode 100644index 00000000..d1b05fc8--- /dev/null+++ b/src/cli/brain/verbs/receipts.ts@@ -0,0 +1,181 @@+/**+ * `o2b brain receipts` - scan / compact over the receipt ledgers+ * (storage-lifecycle, Lane 3 task 3.4).+ *+ * o2b brain receipts [scan] [--older-than-days N] [--json]+ * o2b brain receipts compact --shard continuity:2026-08.jsonl[,family:name...]+ * [--older-than-days N] [--apply] [--expect N] [--strict] [--json]+ *+ * Scan is the default and is a pure read. Compact previews by default -+ * the rewrite happens only behind `--apply` - and never runs without+ * explicit shard ids from a scan. Modes and the shard-reference spelling+ * come from the shared receipts vocabulary, so this verb and the+ * brain_receipts tool cannot drift apart.+ */++import {+ compactReceiptLedgers,+ type ReceiptCompactResult,+} from "../../../core/brain/receipts/compact.ts";+import { scanReceiptLedgers } from "../../../core/brain/receipts/scan.ts";+import { compactView, scanView } from "../../../core/brain/receipts/views.ts";+import {+ isReceiptMode,+ RECEIPT_FOLD_DEFAULT_AGE_DAYS,+ RECEIPT_MODE,+ RECEIPT_MODES,+ RECEIPT_COMPACTION_OPERATION,+ parseReceiptShardRef,+ type ReceiptShardRef,+} from "../../../core/brain/receipts/types.ts";+import { brainVerbContext, fail, parse, resolveBrainAgent } from "../helpers.ts";++export const RECEIPTS_USAGE =+ `usage: o2b brain receipts <${RECEIPT_MODES.join("|")}> [options]\n` ++ ` o2b brain receipts scan [--older-than-days N] [--json]\n` ++ " o2b brain receipts compact --shard family:name [--older-than-days N] " ++ "[--apply] [--expect N] [--strict] [--json]";++/** Parse the repeatable `--shard family:name` flag into shard refs. */+function parseShards(raw: string | string[] | undefined): ReceiptShardRef[] {+ const values = raw === undefined ? [] : Array.isArray(raw) ? raw : [raw];+ const refs: ReceiptShardRef[] = [];+ for (const value of values.flatMap((entry) => entry.split(","))) {+ const trimmed = value.trim();+ if (trimmed === "") continue;+ refs.push(parseReceiptShardRef(trimmed));+ }+ return refs;+}++function renderScanJson(vault: string, olderThanDays: number): number {+ const report = scanReceiptLedgers(vault, { olderThanDays, now: new Date() });+ // The same view the brain_receipts tool answers with, so `--json` and+ // the wire cannot disagree on field spellings.+ process.stdout.write(`${JSON.stringify(scanView(report), null, 2)}\n`);+ return report.errors.length === 0 ? 0 : 1;+}++function renderScanText(vault: string, olderThanDays: number): number {+ const report = scanReceiptLedgers(vault, { olderThanDays, now: new Date() });+ process.stdout.write(+ `${RECEIPT_COMPACTION_OPERATION} scan @ ${report.generated_at} ` ++ `(older than ${report.olderThanDays} days, cutoff ${report.cutoff})\n`,+ );+ for (const shard of report.shards) {+ process.stdout.write(+ `- ${shard.family}:${shard.name} - ${shard.records} record(s), ${shard.foldable} foldable, ` ++ `${shard.protectedRecords} protected, ${shard.unparseable} unparseable, ` ++ `${shard.aggregates} aggregate(s), ${shard.bytes} bytes\n`,+ );+ }+ process.stdout.write(+ `total: ${report.totals.foldable} foldable of ${report.totals.records} record(s) across ` ++ `${report.totals.shards} shard(s)\n`,+ );+ for (const error of report.errors) {+ process.stdout.write(`! ${error.shard}: ${error.message}\n`);+ }+ return report.errors.length === 0 ? 0 : 1;+}++function renderCompact(result: ReceiptCompactResult, json: boolean): number {+ if (json) {+ process.stdout.write(`${JSON.stringify(compactView(result), null, 2)}\n`);+ return result.failures.length === 0 ? 0 : 1;+ }+ process.stdout.write(+ `${RECEIPT_COMPACTION_OPERATION}${result.dryRun ? " preview" : ""} - cutoff ` ++ `${result.cutoff} (older than ${result.olderThanDays} days)\n`,+ );+ for (const shard of result.shards) {+ process.stdout.write(+ `- ${shard.shard}: folded ${shard.folded}, kept ${shard.kept}, ` ++ `${shard.aggregates} aggregate(s), ${shard.bytesBefore} -> ${shard.bytesAfter} bytes\n`,+ );+ }+ for (const failure of result.failures) {+ process.stdout.write(`! ${failure.shard}: ${failure.message}\n`);+ }+ if (result.snapshotRunId !== undefined) {+ process.stdout.write(`recovery point: ${result.snapshotRunId}\n`);+ }+ if (result.dryRun) {+ process.stdout.write("preview only - pass --apply to fold these shards\n");+ }+ return result.failures.length === 0 ? 0 : 1;+}++export async function cmdBrainReceipts(argv: string[]): Promise<number> {+ // A bare invocation - or one that starts with a flag - scans: the+ // read-only mode is the default, and a mode nobody named is scanned+ // for rather than guessed as a mutation. A WORD that names no mode is+ // a usage error, never a silent scan of an intent nobody expressed.+ const first = argv[0];+ const named = isReceiptMode(first) ? first : undefined;+ if (first !== undefined && named === undefined && !first.startsWith("-")) {+ return fail(RECEIPTS_USAGE);+ }+ const mode = named ?? RECEIPT_MODE.scan;+ const { flags } = parse(named === undefined ? argv : argv.slice(1), {+ vault: { type: "string" },+ agent: { type: "string" },+ shard: { type: "string-array" },+ "older-than-days": { type: "string" },+ apply: { type: "boolean" },+ expect: { type: "string" },+ strict: { type: "boolean" },+ json: { type: "boolean" },+ });+ const { config, vault } = brainVerbContext(flags);++ let olderThanDays = RECEIPT_FOLD_DEFAULT_AGE_DAYS;+ const daysRaw = flags["older-than-days"];+ if (daysRaw !== undefined) {+ const days = Number(daysRaw);+ if (!Number.isInteger(days) || days <= 0) {+ return fail("--older-than-days must be a positive integer");+ }+ olderThanDays = days;+ }++ if (mode === RECEIPT_MODE.scan) {+ return flags["json"] === true+ ? renderScanJson(vault, olderThanDays)+ : renderScanText(vault, olderThanDays);+ }++ let shards: ReceiptShardRef[];+ try {+ shards = parseShards(flags["shard"] as string | string[] | undefined);+ } catch (exc) {+ return fail((exc as Error).message);+ }+ if (shards.length === 0) {+ return fail("compact requires --shard family:name ids from a scan");+ }+ const expectRaw = flags["expect"] as string | undefined;+ let expect: number | null = null;+ if (expectRaw !== undefined) {+ const n = Number(expectRaw);+ if (!Number.isInteger(n) || n < 0) return fail("--expect must be a non-negative integer");+ expect = n;+ }++ let result: ReceiptCompactResult;+ try {+ result = compactReceiptLedgers(vault, {+ shards,+ olderThanDays,+ now: new Date(),+ dryRun: flags["apply"] !== true,+ expect,+ strict: flags["strict"] === true,+ subject: { agent: resolveBrainAgent(flags, config), via: "config" },+ configPath: config,+ });+ } catch (exc) {+ return fail(`receipt compaction failed: ${(exc as Error).message ?? exc}`);+ }+ return renderCompact(result, flags["json"] === true);+}diff --git a/src/cli/brain/verbs/storage.ts b/src/cli/brain/verbs/storage.tsnew file mode 100644index 00000000..6925a90a--- /dev/null+++ b/src/cli/brain/verbs/storage.ts@@ -0,0 +1,113 @@+/**+ * `o2b brain storage` - the vault's storage report.+ *+ * The report answers three questions in one pass, from one builder shared+ * with the `brain_storage` MCP tool (`../../../mcp/brain/storage-tools.ts`,+ * which owns the rationale): what the vault's storage holds, per surface,+ * measured or refused by name; where it is heading against the configured+ * cap, as a tier with the numbers it rests on; and which usage sample this+ * invocation recorded. Each invocation takes one census and appends one+ * `storage_usage` sample - the forecast series accumulates precisely+ * because the report was asked for.+ *+ * An uncapped vault reports tier `unbounded` and says so plainly: bytes+ * are reported, no days-to-exhaustion is invented, nothing pauses and+ * nothing refuses. The verb is a report: it never fails on a finding -+ * a critical tier still exits 0, because the report succeeded and+ * enforcement lives at the gates that consume the tier.+ *+ * The CLI runs at local reach (a stdio child of its caller), so it always+ * gets the measured form; the refusal form of the report is rendered+ * anyway, named, rather than assumed impossible.+ */++import {+ STORAGE_FORECAST_TIER,+ type StorageForecast,+} from "../../../core/brain/storage/forecast.ts";+import { TRANSPORT_REACH } from "../../../core/graph/transport-reach.ts";+import {+ buildStorageReport,+ type StorageReport,+ type StorageReportRefusal,+} from "../../../mcp/brain/storage-tools.ts";+import { brainVerbContext, failWith, parse, writeJson } from "../helpers.ts";++/** Column widths of the per-surface listing; padded for scanability. */+const SURFACE_COLUMN = 16;+const TIER_COLUMN = 15;++export async function cmdBrainStorage(argv: string[]): Promise<number> {+ const { flags } = parse(argv, {+ vault: { type: "string" },+ json: { type: "boolean" },+ });+ const { vault } = brainVerbContext(flags);++ let report: StorageReport | StorageReportRefusal;+ try {+ report = buildStorageReport({ vault, reach: TRANSPORT_REACH.local });+ } catch (exc) {+ return failWith("build the storage report", exc);+ }++ if (flags["json"]) writeJson(report);+ else renderText(report);+ return 0;+}++/** The human-readable report; the `--json` payload is the builder's object as-is. */+function renderText(report: StorageReport | StorageReportRefusal): void {+ if (!report.measured) {+ renderRefusal(report);+ return;+ }+ const usage = report.usage;+ process.stdout.write(+ `usage: ${usage.totalBytes} bytes across ${usage.surfaces.length} surfaces` ++ ` (measured ${usage.measuredAt})\n`,+ );+ for (const surface of usage.surfaces) {+ process.stdout.write(+ ` ${surface.surface.padEnd(SURFACE_COLUMN)}${surface.tier.padEnd(TIER_COLUMN)}${surface.bytes}\n`,+ );+ }+ process.stdout.write(`forecast: ${renderForecast(report.forecast)}\n`);+ if (usage.unmeasured.length > 0) {+ const refused = usage.unmeasured.map((u) => `${u.surface} (${u.reason})`).join(", ");+ process.stdout.write(`unmeasured: ${refused}\n`);+ }+ if (report.sample.appended) {+ process.stdout.write(`sample: appended as ${report.sample.runId}\n`);+ } else {+ process.stdout.write(`sample: not appended - ${report.sample.appendError ?? "unknown"}\n`);+ }+}++/** The reach refusal, named the way the MCP caller sees it. */+function renderRefusal(report: StorageReportRefusal): void {+ process.stdout.write(`not measured: ${report.reason} - ${report.detail}\n`);+}++/** The forecast line: the tier first, then the numbers it rests on. */+function renderForecast(forecast: StorageForecast): string {+ if (forecast.tier === STORAGE_FORECAST_TIER.unbounded) {+ return "unbounded - no cap configured; nothing pauses and nothing refuses";+ }+ if (forecast.tier === STORAGE_FORECAST_TIER.unmeasurable) {+ const refused = forecast.unmeasured.map((u) => u.surface).join(", ");+ return `unmeasurable - the census was refused for ${refused}`;+ }+ if (forecast.tier === STORAGE_FORECAST_TIER.noForecast) {+ return "no_forecast - fewer than two usage samples are recorded, so no trend exists";+ }+ const parts: string[] = [];+ if (forecast.percentUsed !== null) {+ parts.push(`${forecast.percentUsed.toFixed(1)} percent of the ${forecast.capBytes}-byte cap`);+ }+ if (forecast.daysToCap !== null) {+ parts.push(`${forecast.daysToCap.toFixed(1)} days to the cap at the measured trend`);+ }+ if (parts.length === 0) return forecast.tier;+ return `${forecast.tier} - ${parts.join(", ")}`;+}diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.tsindex 74e53777..dda2e275 100644--- a/src/cli/command-manifest.ts+++ b/src/cli/command-manifest.ts@@ -420,6 +420,25 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({]),],),+ command(+ "receipts",+ "Receipt compaction surfaces: scan the continuity and metrics ledgers, fold expired records on compact",+ [+ flag("vault", "string"),+ flag("agent", "string"),+ flag("shard", "string-array"),+ flag("older-than-days", "string"),+ flag("apply", "boolean"),+ flag("expect", "string"),+ flag("strict", "boolean"),+ flag("json", "boolean"),+ ],+ ),+ command(+ "storage",+ "Render the vault storage report: usage per surface and the forecast tier against the cap",+ [flag("vault", "string"), flag("json", "boolean")],+ ),command("audit", "Render mutation audit trail"),command("generation-reports",diff --git a/src/core/brain/applier-capability.ts b/src/core/brain/applier-capability.tsindex 19cd1721..b5734519 100644--- a/src/core/brain/applier-capability.ts+++ b/src/core/brain/applier-capability.ts@@ -230,6 +230,12 @@ const ENTRIES: ReadonlyArray<RepairCapability> = [verdict: REPAIR_VERDICT.mechanical,fixer: "hygiene/apply.executeFinding -> archivePage",},+ {+ code: "retire",+ surface: REPAIR_SURFACE.hygieneApply,+ verdict: REPAIR_VERDICT.mechanical,+ fixer: "hygiene/apply.executeFinding -> retireDocumentCopies",+ },{code: "review",surface: REPAIR_SURFACE.hygieneApply,diff --git a/src/core/brain/config-template.ts b/src/core/brain/config-template.tsindex c3d3ca0d..fc2c17e7 100644--- a/src/core/brain/config-template.ts+++ b/src/core/brain/config-template.ts@@ -49,6 +49,7 @@ import {BRAIN_NOTES_DEFAULTS,BRAIN_SESSION_PAYLOAD_DEFAULTS,BRAIN_SESSIONS_DEFAULTS,+ BRAIN_STORAGE_DEFAULTS,BRAIN_TEMPORAL_DEFAULTS,DEFAULT_BRAIN_CONFIG,INJECT_BUDGET_CHARS_DEFAULT,@@ -600,6 +601,40 @@ export const BRAIN_CONFIG_TEMPLATE: ReadonlyArray<BrainTemplateBlock> = Object.fdef("payload_max_text_chars", BRAIN_SESSION_PAYLOAD_DEFAULTS.max_text_chars),],},+ {+ key: "storage",+ doc: [+ "Storage capacity policy (`o2b brain storage`). An absent cap_bytes",+ "means the vault is unbounded: nothing pauses and nothing refuses.",+ "Setting a cap turns on the forecast tiers and the storage-capacity",+ "doctor check; the four thresholds default to the constants the",+ "forecast ships with, and the critical band must sit past the warn",+ "band (a larger percent, a smaller day horizon).",+ ],+ emit: "commented-default",+ keys: [+ example("cap_bytes", 1073741824, [+ "Measured vault storage to treat as full, in bytes. The value",+ "below is an illustration, not a default - writing it would bound",+ "this vault. An explicit null also means unbounded.",+ ]),+ def("warn_percent", BRAIN_STORAGE_DEFAULTS.warn_percent, [+ "Percent of the cap where usage forecasts as warn.",+ ]),+ def("critical_percent", BRAIN_STORAGE_DEFAULTS.critical_percent, [+ "Percent of the cap where usage forecasts as critical. Must be",+ "greater than warn_percent.",+ ]),+ def("warn_days", BRAIN_STORAGE_DEFAULTS.warn_days, [+ "Days to the cap, at the measured trend, where usage forecasts as",+ "warn.",+ ]),+ def("critical_days", BRAIN_STORAGE_DEFAULTS.critical_days, [+ "Days to the cap, at the measured trend, where usage forecasts as",+ "critical. Must be smaller than warn_days.",+ ]),+ ],+ },{key: "temporal",doc: [diff --git a/src/core/brain/continuity/types.ts b/src/core/brain/continuity/types.tsindex 1f8fa206..d98d4b01 100644--- a/src/core/brain/continuity/types.ts+++ b/src/core/brain/continuity/types.ts@@ -96,7 +96,67 @@ export type ContinuityRecordKind =* externalization; turns carrying a private region are withheld whole.* Additive kind, no schema bump.*/- | "session_entity_mention";+ | "session_entity_mention"+ /**+ * One operator-applied receipt compaction run (storage-lifecycle+ * wave): the per-run audit of a confirmed fold of the receipt ledgers,+ * carrying the folded and kept counts, the shard outcomes and the+ * recovery point that preceded the rewrite. Written once per confirmed+ * run by the fold engine, never on a dry run.+ *+ * NAMING HAZARD, decided here: this is the STORAGE compaction audit.+ * It is deliberately disjoint from the Hermes context-compaction kinds+ * above (`pre_compact_extract`, `post_compact_audit`), which audit+ * context-window compaction; the two vocabularies share no surface+ * name, and a receipt_compaction record never describes a context+ * operation.+ */+ | "receipt_compaction";++/**+ * Every {@link ContinuityRecordKind} member as a runtime list, in union+ * declaration order. The union is compile-time only, but a reader that+ * must distinguish "a kind this build knows" from "a kind a newer build+ * minted" needs the members as values - the receipts scan classifies an+ * unknown kind fail-closed as never-foldable, and a runtime guard is how+ * it asks. Add a new union member and this list in the same edit; the+ * membership guard below is derived from the list, so the two cannot+ * disagree.+ */+export const CONTINUITY_RECORD_KINDS: ReadonlyArray<ContinuityRecordKind> = Object.freeze([+ "context_receipt",+ "recall_telemetry",+ "gate_telemetry",+ "pre_compact_extract",+ "post_compact_audit",+ "session_turn",+ "recent_turn",+ "session_summary_node",+ "session_summary_digest",+ "generation_report",+ "ingest_dedup",+ "mcp_route_latency",+ "token_impact",+ "token_impact_outcome",+ "context_pack_outcome",+ "context_pack_evidence",+ "host_memory_write",+ "recall_observed_use",+ "skill_invoked",+ "source_invalidation",+ "negative_recall",+ "decision_model_call",+ "decision_model_extract_commit",+ "session_entity_mention",+ "receipt_compaction",+]);++/** Narrow an unvalidated record's `kind` to a {@link ContinuityRecordKind}. */+export function isContinuityRecordKind(value: unknown): value is ContinuityRecordKind {+ return (+ typeof value === "string" && (CONTINUITY_RECORD_KINDS as ReadonlyArray<string>).includes(value)+ );+}export type ContinuityPayload = Readonly<Record<string, unknown>>;diff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.tsindex a0179f4a..3024e3e1 100644--- a/src/core/brain/diagnostics.ts+++ b/src/core/brain/diagnostics.ts@@ -626,6 +626,33 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new MapnextCommand: "o2b brain snapshot log",autoRepairable: false,},+ {+ // Storage capacity, for a vault whose operator configured a cap.+ // Both severities point at the report surface, which carries the+ // per-surface census and the trend; the repair - reclaiming space+ // or raising the cap - is an operator decision no verb performs.+ code: "storage-capacity-warn",+ issueClass: "storage usage past the warn band of the configured cap",+ nextCommand: "o2b brain storage",+ autoRepairable: false,+ },+ {+ code: "storage-capacity-critical",+ issueClass: "storage usage past the critical band: capacity-gated writes are refused",+ nextCommand: "o2b brain storage",+ autoRepairable: false,+ },+ {+ // Receipt-ledger growth (storage-lifecycle): the append-only+ // receipt ledgers crossed the foldable threshold. Unlike the+ // storage cap the repair is mechanical, so the row points at the+ // verb that previews the fold; the confirmed compact behind the+ // same command is the rewrite.+ code: "receipt-ledger-growth",+ issueClass: "receipt ledgers holding more foldable records than the growth threshold",+ nextCommand: "o2b brain receipts scan",+ autoRepairable: false,+ },{// An announced decommission of the configured embedding model,// inside the warning window. The exit is the curated catalog fordiff --git a/src/core/brain/doctor-exits.ts b/src/core/brain/doctor-exits.tsindex a08ddb6c..67fc3055 100644--- a/src/core/brain/doctor-exits.ts+++ b/src/core/brain/doctor-exits.ts@@ -318,6 +318,25 @@ const EXCLUSIONS: ReadonlyArray<DoctorExitExclusion> = ["or a path that belongs to another user entirely. Guessing one would act on a state this " +"pass never managed to observe",},+ {+ code: "receipt-ledger-unmeasured",+ reason:+ "the receipt ledgers are there and the scan over them was refused, so whether they grow " ++ "unbounded is unknown rather than fine. What repairs that depends on why - a permission " ++ "an operator has to decide to grant, or a regular file that replaced the ledger directory - " ++ "and the ledgers themselves must never be removed to make a scan succeed, since they are " ++ "the continuity history any fold would be rewriting",+ },+ {+ code: "storage-capacity-unmeasured",+ reason:+ "the vault is capped and the census over its surfaces was refused, so fullness is unknown " ++ "rather than fine. What repairs that depends on why - a permission an operator has to " ++ "decide to grant, a surface owned by another user after a sync that preserved ownership, " ++ "or storage that has started failing - and the surfaces themselves must never be removed " ++ "to make a census succeed, since the reading they hold is the fact the cap is judged " ++ "against",+ },{code: "preference-missing-field",reason:diff --git a/src/core/brain/doctor.ts b/src/core/brain/doctor.tsindex c6778784..4b28ce16 100644--- a/src/core/brain/doctor.ts+++ b/src/core/brain/doctor.ts@@ -84,6 +84,8 @@ import { rerankHealthCheck } from "./doctor/rerank-health-check.ts";import { payloadRegistryCheck } from "./doctor/payload-checks.ts";import { recallChannelCoverageCheck } from "./doctor/recall-channel-coverage.ts";import { recoveryPointLivenessCheck } from "./doctor/recovery-point-liveness.ts";+import { storageCapacityCheck } from "./doctor/storage-capacity.ts";+import { receiptLedgerCheck } from "./doctor/receipt-ledger.ts";import { removedToolReferenceCheck } from "./doctor/removed-tool-checks.ts";import { makeStaleDependencyCheck } from "./doctor/stale-dependency-check.ts";import { auditStaleDependencies } from "./stale-dependency.ts";@@ -218,6 +220,8 @@ const DOCTOR_CHECKS: ReadonlyArray<DoctorCheck> = Object.freeze([staleLockProbe,recallChannelCoverageCheck,recoveryPointLivenessCheck,+ storageCapacityCheck,+ receiptLedgerCheck,embeddingSunsetCheck,embeddingsHealthCheck,rerankHealthCheck,@@ -471,6 +475,7 @@ function resolveContext(dbPath: opts.dbPath,configPath: opts.configPath,...(opts.readable !== undefined ? { readable: opts.readable } : {}),+ ...(opts.reach !== undefined ? { reach: opts.reach } : {}),knownBasenames: collectAllBasenames(vault, {site: CONTEXT_SITE,consequence:diff --git a/src/core/brain/doctor/check.ts b/src/core/brain/doctor/check.tsindex 3872ebee..c01511b8 100644--- a/src/core/brain/doctor/check.ts+++ b/src/core/brain/doctor/check.ts@@ -9,6 +9,7 @@* the pass by being added to the registry array in `doctor.ts`.*/+import type { TransportReach } from "../../graph/transport-reach.ts";import type { BrainConfig, DoctorIssue } from "../types.ts";import type { LogRecord, PreferenceRecord, UnreadableLogDay } from "./records.ts";import type { DoctorUncertainEntry } from "./report.ts";@@ -48,6 +49,16 @@ export interface DoctorCheckContext {* no slot of a cap and moves no count. Absent means every page counts.*/readonly readable?: (rel: string) => boolean;+ /**+ * How far the caller had to reach, for the checks whose findings carry+ * whole-vault figures: those figures move when a page the caller may+ * not read appears or goes, so below local reach such a check answers+ * in the uncertain stream, by name and without the figures. Absent+ * reads as local - the shell caller, which already holds the+ * filesystem. A wire surface resolves its caller's reach and passes it+ * here; it never leaves this absent.+ */+ readonly reach?: TransportReach;/** Every valid wikilink target inside `Brain/`, keyed by basename. */readonly knownBasenames: ReadonlySet<string>;/**diff --git a/src/core/brain/doctor/receipt-ledger.ts b/src/core/brain/doctor/receipt-ledger.tsnew file mode 100644index 00000000..81566f08--- /dev/null+++ b/src/core/brain/doctor/receipt-ledger.ts@@ -0,0 +1,100 @@+/**+ * Receipt-ledger growth doctor check (storage-lifecycle, Lane 3 task 3.3).+ *+ * The receipt ledgers are append-only by design, so their growth is+ * normal and their UNBOUNDED growth is the condition an operator can+ * act on: the fold engine (`o2b brain receipts`) exists to fold expired+ * records into aggregates, and a ledger that crossed the growth+ * threshold is one that nothing has folded lately. The check reports+ * one warning per family whose foldable record count crosses the+ * threshold, naming the receipts scan as the next command, and stays+ * silent below it - a compacted or young ledger is healthy, not+ * "not looked at".+ *+ * What the scan could not read is reported, never flattened to zero:+ * an unreadable ledger directory and shards the sweep could not open+ * both reach the uncertain stream under+ * {@link RECEIPT_LEDGER_UNMEASURED_CODE}, because "growth unknown" and+ * "growth fine" are different answers and only one of them is honest.+ *+ * The scan is the same `scanReceiptLedgers` the receipts surfaces run -+ * the doctor never re-implements a census - so the warning counts+ * exactly what a confirmed compaction would fold, threshold included.+ */++import { continuityLogDir } from "../continuity/store.ts";+import { scanReceiptLedgers, receiptLedgerDir } from "../receipts/scan.ts";+import { RECEIPT_LEDGER_FAMILIES } from "../receipts/types.ts";+import type { DoctorIssue } from "../types.ts";+import type { DoctorCheck, DoctorCheckContext, DoctorFindings } from "./check.ts";+import { pushUncertain } from "./uncertain-stream.ts";++/** A receipt ledger's foldable population crossed the growth threshold. */+export const RECEIPT_LEDGER_GROWTH_CODE = "receipt-ledger-growth";++/** The receipt ledgers are there and the scan could not read them. */+export const RECEIPT_LEDGER_UNMEASURED_CODE = "receipt-ledger-unmeasured";++/**+ * Foldable records a family must accumulate before the check speaks.+ * High enough that an actively-used vault is not nagged over normal+ * append traffic, low enough that an unfolded ledger is reported while+ * the fold that would shrink it is still a single operation.+ */+export const RECEIPT_LEDGER_GROWTH_MIN_FOLDABLE = 1_000;++/**+ * The command the growth warning names and the diagnostics registry+ * resolves the code through: the scan is the preview, the confirmed+ * compact behind the same verb is the rewrite.+ */+export const RECEIPTS_NEXT_COMMAND = "o2b brain receipts scan";++export const receiptLedgerCheck: DoctorCheck = {+ failSoft: true,+ run(ctx: DoctorCheckContext, out: DoctorFindings): void {+ let report;+ try {+ report = scanReceiptLedgers(ctx.vault, { now: ctx.now });+ } catch {+ // The scan's own refusal, named: what the enumeration stumbled on+ // carries absolute host paths no doctor surface should repeat, and+ // the repair is the operator's to choose whatever the cause was.+ pushUncertain(out.uncertain, {+ code: RECEIPT_LEDGER_UNMEASURED_CODE,+ path: continuityLogDir(ctx.vault),+ message:+ "the receipt ledgers could not be scanned, so whether they are " ++ "growing unbounded is unknown",+ });+ return;+ }+ for (const error of report.errors) {+ pushUncertain(out.uncertain, {+ code: RECEIPT_LEDGER_UNMEASURED_CODE,+ // The shard's own vault-relative path: the stream merges on+ // (code, path), so two dark shards without it collapse into one+ // anonymous entry and every shard after the first goes unreported.+ path: error.path,+ message:+ `the receipt shard ${error.shard} is ${error.message}, so its ` ++ "foldable count is unknown",+ });+ }+ for (const family of RECEIPT_LEDGER_FAMILIES) {+ const shards = report.shards.filter((shard) => shard.family === family);+ const foldable = shards.reduce((total, shard) => total + shard.foldable, 0);+ if (foldable < RECEIPT_LEDGER_GROWTH_MIN_FOLDABLE) continue;+ const bytes = shards.reduce((total, shard) => total + shard.bytes, 0);+ out.issues.push({+ severity: "warning",+ code: RECEIPT_LEDGER_GROWTH_CODE,+ path: receiptLedgerDir(ctx.vault, family),+ message:+ `the ${family} receipt ledgers hold ${foldable} foldable records across ` ++ `${shards.length} shard(s) (${bytes} bytes on disk); folding expired records into ` ++ `aggregates would shrink them - see ${RECEIPTS_NEXT_COMMAND}`,+ } satisfies DoctorIssue);+ }+ },+};diff --git a/src/core/brain/doctor/report.ts b/src/core/brain/doctor/report.tsindex a8149446..cc2d9d4c 100644--- a/src/core/brain/doctor/report.ts+++ b/src/core/brain/doctor/report.ts@@ -6,6 +6,7 @@* shapes without the other importing it back.*/+import type { TransportReach } from "../../graph/transport-reach.ts";import type { SemanticHealthReport } from "../health/reconcile.ts";import type { DoctorIssue, InstructionFileCeilingWarning, TrustVerdict } from "../types.ts";@@ -44,6 +45,14 @@ export interface RunDoctorOptions {* (the operator's own shell).*/readonly readable?: (rel: string) => boolean;+ /**+ * How far the caller had to reach. Passed through to+ * {@link DoctorCheckContext.reach}, so a check whose findings carry+ * whole-vault figures answers below local reach in the uncertain+ * stream, by name and without the figures. Omitted reads as local+ * (the operator's own shell).+ */+ readonly reach?: TransportReach;/*** Optional precomputed dream summary (v0.10.16). When supplied,* the doctor runs the verification-delta helper and folds thediff --git a/src/core/brain/doctor/storage-capacity.ts b/src/core/brain/doctor/storage-capacity.tsnew file mode 100644index 00000000..26cb8c4b--- /dev/null+++ b/src/core/brain/doctor/storage-capacity.ts@@ -0,0 +1,140 @@+/**+ * The storage-capacity doctor check: how full a CAPPED vault is.+ *+ * The check is inert without a cap. An unbounded vault has no capacity+ * question to answer, and reporting "you are at 3 percent of no cap"+ * would be noise - the same reason the maintenance pressure gate never+ * fires unconfigured. The moment `storage.cap_bytes` is set, the check+ * measures the vault the way the capacity gate does and reports:+ *+ * - past the critical percent or horizon: an ERROR - a write the gate+ * refuses is a condition the operator has to see before the next+ * tool call explains it for them;+ * - past the warn band: a WARNING - the trend, while there is still+ * time to act on it;+ * - an unmeasurable store: an UNCERTAIN entry naming the refused+ * surfaces, never a quiet pass - a census that could not read a+ * surface cannot claim the vault has room.+ *+ * The reading is a whole-vault byte figure: it moves when a page the+ * caller may not read appears or goes, and no filtered form of it exists.+ * Below local reach (`ctx.reach`) it is therefore not reported - the same+ * local-reach rule `brain_storage` refuses its census under - and the+ * check lands the UNCERTAIN entry naming the reach instead, never a+ * quiet pass and never the figures. The check's own shell callers pass+ * no reach, which reads as local, so the operator's report is unchanged.+ *+ * The store is measured at `ctx.dbPath` when the pass carries one (the+ * store the operator's searches actually use) and through the default+ * resolution otherwise. The reading re-measures rather than trusting the+ * stored `storage_usage` aggregate, for the same reason the capacity+ * gate does: the aggregate is advisory, the census is the fact.+ */++import { resolveStorage } from "../policy.ts";+import { TRANSPORT_REACH } from "../../graph/transport-reach.ts";+import {+ STORAGE_FORECAST_TIER,+ forecastStorageUsage,+ type StorageForecastTier,+} from "../storage/forecast.ts";+import { measureStorageUsage } from "../storage/usage.ts";+import { brainDirs } from "../paths.ts";+import type { DoctorCheck, DoctorCheckContext, DoctorFindings } from "./check.ts";+import { pushUncertain } from "./uncertain-stream.ts";++/** The newest reading is past the warn band of the configured cap. */+export const STORAGE_CAPACITY_WARN_CODE = "storage-capacity-warn";++/** The newest reading is past the critical band: writes get refused. */+export const STORAGE_CAPACITY_CRITICAL_CODE = "storage-capacity-critical";++/** The census could not read every surface, so fullness is unknown. */+export const STORAGE_CAPACITY_UNMEASURED_CODE = "storage-capacity-unmeasured";++/** The command both issue codes point at: the report surface. */+const STORAGE_NEXT_COMMAND = "o2b brain storage";++/** Render a forecast tier's numbers into one operator-readable sentence. */+function describeReading(+ tier: StorageForecastTier,+ percent: number | null,+ days: number | null,+): string {+ const parts: string[] = [];+ if (percent !== null) parts.push(`${percent.toFixed(1)} percent of the configured cap`);+ if (days !== null) parts.push(`${days.toFixed(1)} days to the cap at the measured trend`);+ if (parts.length === 0) parts.push(`tier ${tier}`);+ return parts.join(", ");+}++export const storageCapacityCheck: DoctorCheck = {+ failSoft: true,+ run(ctx: DoctorCheckContext, out: DoctorFindings): void {+ if (ctx.config === undefined) return;+ const storage = resolveStorage(ctx.config);+ if (storage.cap_bytes === null) return;++ // The whole-vault reading is a local-reach fact (`brain_storage`'s+ // rule). Below local reach it is not taken for the caller at all, so+ // this answers in the uncertain stream under the same unmeasured+ // code - fullness unknown, stated by name, no figures - and the warn+ // and critical issues are never emitted, because either would carry+ // the census in its message.+ if (ctx.reach !== undefined && ctx.reach !== TRANSPORT_REACH.local) {+ pushUncertain(out.uncertain, {+ code: STORAGE_CAPACITY_UNMEASURED_CODE,+ path: brainDirs(ctx.vault).brain,+ message:+ "storage capacity is not evaluated at this reach, so whether the " ++ "vault is within its cap is unknown",+ });+ return;+ }++ const usage = measureStorageUsage(ctx.vault, {+ now: ctx.now,+ ...(ctx.dbPath !== undefined ? { storePath: ctx.dbPath } : {}),+ });+ if (!usage.complete) {+ pushUncertain(out.uncertain, {+ code: STORAGE_CAPACITY_UNMEASURED_CODE,+ path: brainDirs(ctx.vault).brain,+ message:+ `storage capacity could not be measured: the census was refused for ` ++ `${usage.unmeasured.map((u) => u.surface).join(", ")}, so whether the vault ` ++ `is within its ${storage.cap_bytes} byte cap is unknown`,+ });+ return;+ }++ const forecast = forecastStorageUsage(ctx.vault, {+ capBytes: storage.cap_bytes,+ latestUsage: usage,+ warnPercent: storage.warn_percent,+ criticalPercent: storage.critical_percent,+ warnDays: storage.warn_days,+ criticalDays: storage.critical_days,+ });+ if (forecast.tier === STORAGE_FORECAST_TIER.critical) {+ out.issues.push({+ severity: "error",+ code: STORAGE_CAPACITY_CRITICAL_CODE,+ message:+ `storage usage is ${describeReading(forecast.tier, forecast.percentUsed, forecast.daysToCap)} ` ++ `(${forecast.measuredBytes} of ${storage.cap_bytes} bytes); writes gated on capacity ` ++ `are refused past this line - reclaim space or raise the cap, then see ${STORAGE_NEXT_COMMAND}`,+ });+ return;+ }+ if (forecast.tier === STORAGE_FORECAST_TIER.warn) {+ out.issues.push({+ severity: "warning",+ code: STORAGE_CAPACITY_WARN_CODE,+ message:+ `storage usage is ${describeReading(forecast.tier, forecast.percentUsed, forecast.daysToCap)} ` ++ `(${forecast.measuredBytes} of ${storage.cap_bytes} bytes) - see ${STORAGE_NEXT_COMMAND}`,+ });+ }+ },+};diff --git a/src/core/brain/document-retire.ts b/src/core/brain/document-retire.tsnew file mode 100644index 00000000..22c9853f--- /dev/null+++ b/src/core/brain/document-retire.ts@@ -0,0 +1,387 @@+/**+ * Document retirement applier core (storage lifecycle, Lane 4; plan+ * task 4.3).+ *+ * The consumer of the `document-dedup` detector's findings: for each+ * near-duplicate copy a plan selects, retire the copy and keep the+ * canonical page -+ *+ * 1. retarget every inbound `[[stale-copy]]` wikilink to the+ * canonical spelling (the report is carried, never swallowed);+ * 2. remove the stale page (per-file removal via the shared+ * `removeFile`, so a failure is one copy's failure);+ * 3. drop the copy's content-manifest entry, round-tripping the+ * contract the manifest was read under - the same t_586d5d8b rule+ * deleteBySource applies: a removal never re-stamps the contract+ * the surviving entries were recorded under. The drop is the+ * resurrect guard: with the entry gone, the retired source+ * classifies `new` again, so no later ingest can claim the+ * retired page's records are current; the source can only return+ * as a deliberate full re-ingest;+ * 4. append one `source_invalidation` continuity record PER COPY+ * naming canonical id, retired id, dropped manifest key and the+ * retained source-archive hash - the audit the page-dedup merge+ * never needed, because a merge keeps both pages and this+ * operation deletes one.+ *+ * The whole span sits inside `withDestructiveSnapshot`. The snapshot+ * reason is `deleteBySource`'s: a retired summary page is a source's+ * derived material, which is exactly what that reason documents, and+ * the reason vocabulary lives in a file this lane does not own. The+ * source archive (`Brain/.archives/<hash>.txt`) is deliberately NOT+ * removed - it is evidence under a no-TTL contract, and the audit+ * names its digest so the provenance chain survives the page.+ *+ * Failure posture is per-copy fail-soft: one copy's failure is+ * collected as `{path, message}`, and the remaining copies still+ * retire. The message says WHERE the copy ended up: a failure BEFORE+ * the page's own removal (the existence, frontmatter or canonical+ * checks) leaves the copy on disk; a failure AFTER it - the manifest+ * drop, the continuity record - reports the page as REMOVED, because it+ * is, and names the step that lost it. Copy-level input mistakes (a+ * path equal to its own canonical) are caller bugs and refuse loudly+ * before any work.+ *+ * Dry runs never reach this module: the hygiene apply plan previews+ * retire findings without executing them, and the surfaces' count+ * guard runs on the selected findings before this applier is called.+ */++import { existsSync } from "node:fs";+import { join } from "node:path";++import { parseFrontmatter } from "../vault.ts";+import { ensureInsideVault } from "../path-safety.ts";+import { isoSecond } from "./time.ts";+import { readManifest, writeManifestAtomic, manifestPath } from "./ingest/content-manifest.ts";+import { appendContinuitySourceInvalidation } from "./continuity/store.ts";+import { removeFile } from "./source-cleanup.ts";+import { basenameCollides, retargetWikilinks } from "./page-dedup.ts";+import type { WikilinkRetargetReport } from "./page-dedup.ts";+import { BRAIN_SNAPSHOT_REASON } from "./types.ts";+import { withDestructiveSnapshot } from "./snapshot-gate.ts";+import { REVIEW_LANE } from "./write-gate.ts";+import {+ classifyRecoverability,+ type DestructiveBlastRadius,+ type RecoverabilityVerdict,+} from "./gates/recoverability.ts";+import { assertVaultIdentityForWrite } from "./vault-identity.ts";+import { refuseDocumentDeny, type WriteSubject } from "./write-disposition.ts";+import type { ResolveWriteDispositionOptions } from "./write-disposition.ts";+import { BRAIN_SOURCE_KIND } from "./ingest/ingest.ts";+import type { SourceCleanupFailure } from "./source-cleanup.ts";++/** The agent named on the audit record when the caller names none. */+const DEFAULT_AGENT = "document_retire";++/** Basename without the `.md` extension - the wikilink spelling of a page. */+function linkTarget(pagePath: string): string {+ const base = pagePath.split("/").at(-1) ?? pagePath;+ return base.replace(/\.md$/, "");+}++/**+ * Why a copy's inbound-link pass did not run: another page in the vault+ * carries the retired copy's basename, so the bare `[[basename]]`+ * spelling resolves to more than one page and a retarget keyed on the+ * spelling alone would repoint links that meant the OTHER page - edits+ * to pages outside the detector's near-duplicate scope, made on a guess.+ * The copy still retires; the skipped pass leaves a stale link, which a+ * reader can discover, where a mis-pointed link misleads silently.+ */+export const RETIRE_LINKS_SKIPPED_BASENAME_COLLISION =+ "another page carries the same basename; the bare-basename link spelling is ambiguous";++/** Frontmatter key carrying the digest of the archived source bytes. */+const SOURCE_ARCHIVE_HASH_KEY = "source_archive_hash";++/** Frontmatter key carrying the canonical source path a page derives from. */+const SOURCE_PATH_KEY = "source_path";++/** One near-duplicate copy to retire, with the page that survives it. */+export interface RetireDocumentCopyInput {+ /** Vault-relative path of the near-duplicate summary page to retire. */+ readonly path: string;+ /** Vault-relative path of the canonical page that survives. */+ readonly canonical: string;+}++export interface RetireDocumentsOptions {+ /** Named on the audit record. Defaults to {@link DEFAULT_AGENT}. */+ readonly agent?: string;+ /** Injected clock for the audit stamps. */+ readonly now?: Date;+ /**+ * Credential subject for the permissions-document deny consult, which+ * runs per copy before anything is read for deletion (the ingest+ * lane, matching deleteBySource). With no permissions document this+ * is a no-op.+ */+ readonly subject?: WriteSubject;+}++/** One copy that retired cleanly, with everything the run did to it. */+export interface RetiredDocumentCopy {+ readonly path: string;+ readonly canonical: string;+ /** The source path whose manifest entry was targeted, or `null`. */+ readonly manifestKey: string | null;+ /** True when the manifest was rewritten without the entry. */+ readonly manifestDropped: boolean;+ /**+ * The full retarget report for this copy's inbound wikilinks, `null`+ * when the pass was SKIPPED - {@link RetiredDocumentCopy.wikilinksSkipped}+ * names why. `null` is never a silent zero: a skipped pass left every+ * inbound link exactly as it was.+ */+ readonly wikilinks: WikilinkRetargetReport | null;+ /** Why the link pass did not run, `null` when it ran. */+ readonly wikilinksSkipped: string | null;+ /** Digest named in the audit; from the page's frontmatter, or `null`. */+ readonly sourceArchiveHash: string | null;+ /** The id of the `source_invalidation` continuity record this retirement appended. */+ readonly auditRecordId: string;+}++export interface RetireDocumentsResult {+ readonly retired: ReadonlyArray<RetiredDocumentCopy>;+ /**+ * Per-copy failures, collected. A failure BEFORE the page's own+ * removal leaves the copy on disk; a failure after it says so in its+ * message - the page is gone, and the message names the step that+ * lost it.+ */+ readonly failed: ReadonlyArray<SourceCleanupFailure>;+ readonly snapshotRunId: string | null;+ readonly snapshotPath: string | null;+ readonly recoverability: RecoverabilityVerdict;+}++/** What one copy's retirement read off the page before acting. */+interface CopyFacts {+ readonly sourcePath: string | null;+ readonly archiveHash: string | null;+ readonly kind: string | null;+}++function readCopyFacts(vault: string, pagePath: string): CopyFacts | null {+ let meta: Record<string, unknown>;+ try {+ [meta] = parseFrontmatter(join(vault, pagePath));+ } catch {+ return null;+ }+ const sourcePath = meta[SOURCE_PATH_KEY];+ const archiveHash = meta[SOURCE_ARCHIVE_HASH_KEY];+ return {+ kind: typeof meta["kind"] === "string" ? meta["kind"] : null,+ sourcePath: typeof sourcePath === "string" && sourcePath.length > 0 ? sourcePath : null,+ archiveHash: typeof archiveHash === "string" && archiveHash.length > 0 ? archiveHash : null,+ };+}++/**+ * Retire the given near-duplicate copies. Writes by contract: the+ * vault-identity write guard runs at the entry point, before any other+ * work, per the one rule every applier shares.+ */+export function retireDocumentCopies(+ vault: string,+ copies: ReadonlyArray<RetireDocumentCopyInput>,+ opts: RetireDocumentsOptions = {},+): RetireDocumentsResult {+ // Write guard first - see the module header. This applier always+ // writes when called; the dry-run boundary lives in the hygiene plan.+ assertVaultIdentityForWrite(vault);++ for (const copy of copies) {+ if (copy.path === copy.canonical) {+ throw new Error(+ `retireDocumentCopies: ${copy.path} is named as both the retired copy and the canonical`,+ );+ }+ }++ if (copies.length === 0) {+ return Object.freeze({+ retired: Object.freeze([]),+ failed: Object.freeze([]),+ snapshotRunId: null,+ snapshotPath: null,+ // Nothing to destroy, so the classifier says so - neither the+ // alarm of `unproven` nor the false comfort of `covered`.+ recoverability: classifyRecoverability({ recoveryPoint: false, blastRadius: {} }),+ });+ }++ // Deny-consulted evidence (write-side trust, Task 12), per copy and+ // BEFORE any read-for-deletion, one ledger row behind a deny - the+ // same ladder deleteBySource climbs. A deny throws here, so nothing+ // has been snapshotted or removed when it lands.+ const denyOpts: ResolveWriteDispositionOptions = opts.now !== undefined ? { now: opts.now } : {};+ for (const copy of copies) {+ refuseDocumentDeny(vault, REVIEW_LANE.ingest, opts.subject, {+ target: copy.path,+ ...denyOpts,+ });+ }++ const retired: RetiredDocumentCopy[] = [];+ const failed: SourceCleanupFailure[] = [];++ const runRetirement = (): void => {+ for (const copy of copies) {+ try {+ retired.push(retireOneCopy(vault, copy, opts));+ } catch (error) {+ failed.push({+ path: copy.path,+ message: error instanceof Error ? error.message : String(error),+ });+ }+ }+ };++ // Only snapshot when there is something to remove; a caller that+ // passes no work gets no archive churn.+ const blastRadius: DestructiveBlastRadius = { brainTopLevel: true, outsideBrainRoot: false };+ const gated = withDestructiveSnapshot(+ vault,+ BRAIN_SNAPSHOT_REASON.deleteBySource,+ runRetirement,+ {+ ...(opts.now !== undefined ? { now: opts.now } : {}),+ blastRadius,+ },+ );++ return Object.freeze({+ retired: Object.freeze(retired),+ failed: Object.freeze(failed),+ snapshotRunId: gated.snapshot.runId,+ snapshotPath: gated.snapshot.path,+ recoverability: gated.recoverability,+ });+}++/** Retire one copy. Throws to the caller's per-copy collector. */+function retireOneCopy(+ vault: string,+ copy: RetireDocumentCopyInput,+ opts: RetireDocumentsOptions,+): RetiredDocumentCopy {+ const abs = join(vault, copy.path);+ // Containment before existence: removeFile answers a vault-escaping path+ // with the same `false` a vanished page gets, and the message below would+ // then claim a copy vanished that was never inside the vault to begin+ // with. Named here for what it is, so the per-copy failure says the path+ // escapes the vault rather than inventing a disappearance.+ try {+ ensureInsideVault(abs, vault);+ } catch {+ throw new Error(`retired copy path escapes the vault: ${copy.path}`);+ }+ if (!existsSync(abs)) {+ throw new Error(`retired copy not found on disk: ${copy.path}`);+ }+ const facts = readCopyFacts(vault, copy.path);+ if (facts === null) {+ throw new Error(`retired copy frontmatter does not parse: ${copy.path}`);+ }+ if (facts.kind !== BRAIN_SOURCE_KIND) {+ throw new Error(`retired copy is not an ingest summary page (kind ${String(facts.kind)})`);+ }+ if (!existsSync(join(vault, copy.canonical))) {+ throw new Error(`canonical page not found on disk: ${copy.canonical}`);+ }++ // 1. Links first, so no reader between steps ever sees a stranded+ // `[[stale-copy]]`. Failure to write a file is collected in the+ // report, not thrown - the pass has already rewritten the files+ // before it (the retargetWikilinks argument, verbatim). The pass+ // keys on the bare basename, so when another page carries the same+ // basename the spelling is ambiguous: rewriting it would repoint+ // links that meant the other page. The copy then retires without its+ // link pass and the report names why - a stale link is discoverable,+ // a mis-pointed one misleads silently.+ const wikilinks = basenameCollides(vault, linkTarget(copy.path), copy.path)+ ? null+ : retargetWikilinks(vault, [{ from: linkTarget(copy.path), to: linkTarget(copy.canonical) }]);++ // 2. Remove the stale page. `false` here means the page vanished+ // between the existence check and the unlink - that is a failure to+ // report, not a success to claim.+ if (!removeFile(vault, copy.path)) {+ throw new Error(`retired copy vanished before removal: ${copy.path}`);+ }++ // Everything below runs AFTER the page is gone. A failure here cannot+ // leave the copy on disk, so it is rethrown AS that fact - the report+ // must never send an operator looking for a page that is not there.+ try {+ // 3. Drop the manifest entry for the copy's source, round-tripping+ // the read contract (t_586d5d8b): the surviving entries keep the+ // fingerprint they were recorded under, so the owed one-time+ // reprocess is never forfeited by a removal. The per-copy reason is+ // worded from what actually happened - a drop, a refusal, an entry+ // already absent, or nothing to drop at all - never from the key's+ // existence alone.+ let manifestDropped = false;+ let manifestClause: string;+ if (facts.sourcePath === null) {+ manifestClause = "no manifest key recorded";+ } else if (!existsSync(manifestPath(vault))) {+ manifestClause = `manifest absent, nothing to drop for ${facts.sourcePath}`;+ } else {+ const manifest = readManifest(vault);+ const entries = { ...manifest.entries };+ if (entries[facts.sourcePath] === undefined) {+ manifestClause = `manifest key ${facts.sourcePath} already absent`;+ } else {+ delete entries[facts.sourcePath];+ manifestDropped = writeManifestAtomic(vault, entries, manifest.contract);+ // A `false` from the atomic write after deleting a PRESENT key means+ // the serialized bytes were already identical - the key was already+ // gone on disk, i.e. a concurrent retirer got there first. The+ // reason names that success-shaped fact, not a failure to drop.+ manifestClause = manifestDropped+ ? `dropped manifest key ${facts.sourcePath}`+ : `manifest already without key ${facts.sourcePath}`;+ }+ }++ // 4. The provenance record: one per copy, naming what survives, what+ // went, what was dropped and what evidence is retained.+ const agent = opts.agent?.trim() ? opts.agent.trim() : DEFAULT_AGENT;+ const record = appendContinuitySourceInvalidation(vault, {+ createdAt: isoSecond(opts.now ?? new Date()),+ source: {+ id: facts.sourcePath ?? copy.path,+ ...(facts.sourcePath !== null ? { path: facts.sourcePath } : {}),+ kind: "source",+ },+ reason:+ `${agent}: retired near-duplicate copy ${linkTarget(copy.path)} into canonical ` ++ `${linkTarget(copy.canonical)}; ${manifestClause}; ` ++ `source archive ${facts.archiveHash ?? "none"} retained`,+ });++ return Object.freeze({+ path: copy.path,+ canonical: copy.canonical,+ manifestKey: facts.sourcePath,+ manifestDropped,+ wikilinks,+ wikilinksSkipped: wikilinks === null ? RETIRE_LINKS_SKIPPED_BASENAME_COLLISION : null,+ sourceArchiveHash: facts.archiveHash,+ auditRecordId: record.id,+ });+ } catch (error) {+ const detail = error instanceof Error ? error.message : String(error);+ throw new Error(+ `retirement removed ${copy.path}, then a post-removal step failed ` ++ `(the copy is not on disk): ${detail}`,+ );+ }+}diff --git a/src/core/brain/hygiene/apply.ts b/src/core/brain/hygiene/apply.tsindex 837886e6..7d467141 100644--- a/src/core/brain/hygiene/apply.ts+++ b/src/core/brain/hygiene/apply.ts@@ -11,7 +11,11 @@* - `recompile` -> the targeted-recompile executor, scoped to the* finding's page;* - `archive` / `forget` -> move the page into a dated- * `Brain/.snapshots/` directory - never delete.+ * `Brain/.snapshots/` directory - never delete;+ * - `retire` -> the document retirement core: a near-duplicate+ * copy is deleted, its manifest entry dropped and+ * its removal audited (destructive by design, so+ * the plan builder only selects it attended).** Dry-run previews the routed actions with zero writes. Per-finding* fail-soft; one audit record per non-dry run.@@ -21,11 +25,13 @@ import { join } from "node:path";import { appendAuditRecord } from "../../reliability/audit.ts";import { HYGIENE_AUDIT_DIR } from "../audit-dirs.ts";+import { retireDocumentCopies } from "../document-retire.ts";import { mergePreferences } from "../merge.ts";import { brainDirsForWrite } from "../paths.ts";import { archivePage, executeRecompile, planRecompile } from "../recompile.ts";import { appendClaimEvent } from "../truth/store.ts";import { assertVaultIdentityForWrite } from "../vault-identity.ts";+import type { WriteSubject } from "../write-disposition.ts";import type { HygienePlan } from "./plan.ts";import type { HygieneFinding } from "./types.ts";@@ -33,6 +39,15 @@ export interface HygieneApplyOptions {readonly dryRun?: boolean;readonly agent: string;readonly now: Date;+ /**+ * Credential subject for the destructive routes' deny consult. Only+ * the `retire` executor consults a permissions document today (through+ * the retirement core, the same ladder deleteBySource climbs); with no+ * subject the consult falls back to the ambient agent, so a surface+ * holding a request identity must pass it here or a rule naming that+ * identity is silently bypassed on this route.+ */+ readonly subject?: WriteSubject;}export interface HygieneAppliedAction {@@ -130,6 +145,45 @@ async function executeFinding(detail: `archived to ${target}`,});}+ case "retire": {+ // The evidence contract is the document-dedup detector's: the+ // canonical page lives in `evidence.canonical`, and every other+ // target is a stale copy to retire into it. Re-derived here so a+ // hand-built plan answers to the same shape a scan produced.+ const canonical = finding.evidence["canonical"];+ if (typeof canonical !== "string" || canonical.length === 0) {+ throw new Error("retire finding lacks a canonical page in its evidence");+ }+ const copies = finding.targets+ .filter((target) => target !== canonical)+ .map((path) => ({ path, canonical }));+ if (copies.length === 0) {+ throw new Error("retire finding names no retired copy besides the canonical");+ }+ const result = retireDocumentCopies(vault, copies, {+ agent: opts.agent,+ now: opts.now,+ ...(opts.subject !== undefined ? { subject: opts.subject } : {}),+ });+ // A per-copy failure BEFORE the removal leaves the copy on disk;+ // one after it reports the page as removed in its own message.+ // Surfacing the first failure through the finding's error keeps+ // the applied/errors split honest (the recompile executor's rule).+ if (result.failed.length > 0) {+ const first = result.failed[0]!;+ throw new Error(+ `retired ${result.retired.length}, failed ${result.failed.length}: ` ++ `${first.path}: ${first.message}`,+ );+ }+ return Object.freeze({+ finding_id: finding.id,+ action: "retire",+ detail: `retired ${result.retired.length} near-duplicate cop${+ result.retired.length === 1 ? "y" : "ies"+ } into ${canonical}`,+ });+ }default:throw new Error(`no executor for action '${finding.proposed_action}'`);}diff --git a/src/core/brain/hygiene/detectors/document-dedup.ts b/src/core/brain/hygiene/detectors/document-dedup.tsnew file mode 100644index 00000000..24990255--- /dev/null+++ b/src/core/brain/hygiene/detectors/document-dedup.ts@@ -0,0 +1,503 @@+/**+ * Document-dedup detector (storage lifecycle, Lane 4; plan task 4.1).+ *+ * Ingest summary pages (kind `brain-source`, `Brain/sources/`) written+ * from different sources can carry near-identical bodies - the same+ * material ingested twice, or one source copied over another. Each+ * redundant copy keeps its own source link, manifest entry and archive+ * stamp, so the vault pays for one fact several times. This detector+ * groups such pages and nominates the group for retirement: one page+ * survives as canonical, the others are retired through the hygiene+ * apply plan (action `retire`), which routes to the retirement core+ * (`document-retire.ts`).+ *+ * Two similarity layers, matching the preference dedup detector:+ *+ * - Lexical: token jaccard over the page body at+ * {@link DOCUMENT_DEDUP_LEXICAL_THRESHOLD}, bucketed by the folded+ * composite scope key so identical text in two scopes never groups+ * (per-scope keying, t_37c05a34).+ * - Embedding (`detectSemanticDocumentDedup`): cosine over a+ * configured provider's vectors at+ * {@link DOCUMENT_DEDUP_EMBEDDING_THRESHOLD}, crossing scope+ * buckets is deliberately NOT done here - documents are grouped+ * within scope only, because a scope boundary is an operator+ * decision a similarity score must not overrule. Any provider+ * problem falls back to the lexical layer, labeled as such.+ *+ * Canonical selection is keep-newest on `created_at`. A group holding+ * ANY member without `created_at` is ineligible: the page-dedup lane's+ * mtime fallback is merge-unstable (GitHub #180 - the merge itself+ * moves the mtime it falls back to), so such a group is reported for+ * review with the reason in evidence, never for retirement.+ *+ * Union-find components can be CHAINS (A~B, B~C, A!~C): one group, but+ * not one similarity. A member therefore retires only on its OWN+ * above-threshold pair against the chosen canonical; a member the+ * grouping reached only transitively is split into a review finding+ * with the reason in evidence, so keep-newest can never retire a page+ * no score tied to the survivor.+ *+ * Pages a merge already resolved leave the pool through the one shared+ * predicate every duplicate detector in the project asks+ * ({@link isMergeResolved}). The detector never mutates anything.+ */++import { existsSync, readdirSync } from "node:fs";+import { join } from "node:path";++import { parseFrontmatter } from "../../../vault.ts";+import { resolveSearchConfig } from "../../../search/index.ts";+import { makeProvider } from "../../../search/embeddings/provider.ts";+import { providerProducesVectors } from "../../../search/embeddings/contract.ts";+import type { EmbeddingProvider } from "../../../search/embeddings/contract.ts";+import { compositeScopeKey, scopeFromFrontmatter } from "../../../scope-key.ts";+import { BRAIN_SOURCES_REL } from "../../paths.ts";+import { BRAIN_SOURCE_KIND } from "../../ingest/ingest.ts";+import { NEAR_DUPLICATE_MIN_TOKENS, roundScore } from "../../near-duplicate.ts";+import { findSimilarPairs, tokenise } from "../../similarity.ts";+import { isMergeResolved, mergePointerLookup, readMergedInto } from "../../page-meta/page-id.ts";+import { DEDUP_EMBEDDING_THRESHOLD, DEDUP_LEXICAL_THRESHOLD, type DedupMethod } from "./dedup.ts";+import { hygieneFindingId } from "./id.ts";+import type { HygieneDetectorContext, HygieneFinding } from "../types.ts";++/** Inclusive token-jaccard lower bound for nominating a document pair. */+export const DOCUMENT_DEDUP_LEXICAL_THRESHOLD = DEDUP_LEXICAL_THRESHOLD;+/** Inclusive embedding-cosine lower bound for nominating a document pair. */+export const DOCUMENT_DEDUP_EMBEDDING_THRESHOLD = DEDUP_EMBEDDING_THRESHOLD;++/** Bound on how many summary pages enter pairwise comparison. */+export const DOCUMENT_DEDUP_CANDIDATE_CAP = 200;++/**+ * Why a group is reported for review instead of retirement. Stated once+ * so the finding evidence and the docs cannot drift apart.+ */+export const DOCUMENT_DEDUP_INELIGIBLE_REASON =+ "a group member has no created_at; the mtime fallback is merge-unstable (GitHub #180)";++/**+ * Why a grouped member is held out of a retirement finding: the+ * union-find component it sits in can be CHAINED (A~B, B~C, A!~C), so+ * membership alone proves nothing about the member the canonical was+ * picked to be. Retirement acts only on a member with its OWN+ * above-threshold pair against the chosen canonical; everyone else is+ * reported for review with this reason. Stated once so the finding+ * evidence and the docs cannot drift apart.+ */+export const DOCUMENT_DEDUP_TRANSITIVE_REASON =+ "a grouped member has no direct above-threshold similarity to the canonical; retirement never acts on transitive similarity";++export interface DocumentDedupOptions {+ readonly lexicalThreshold?: number;+}++export interface SemanticDocumentDedupOptions {+ /**+ * Embedding provider override. `undefined` resolves the vault's+ * configured provider; `null` (or an unusable provider) falls back+ * to the lexical layer.+ */+ readonly provider?: EmbeddingProvider | null;+ readonly threshold?: number;+ readonly lexicalThreshold?: number;+ /**+ * The caller's reach predicate, the same admission filter the lexical+ * layer applies: a page it refuses is absent - never embedded, never+ * scored, never named. Absent means read-everything.+ */+ readonly readable?: (rel: string) => boolean;+}++export interface SemanticDocumentDedupResult {+ readonly method: DedupMethod;+ readonly findings: ReadonlyArray<HygieneFinding>;+}++/** One ingest summary page the detector admitted. */+interface DocumentRecord {+ /** Vault-relative POSIX path - the id every source surface addresses. */+ readonly path: string;+ readonly body: string;+ /** ISO `created_at`, or `null` when absent, blank or unparseable. */+ readonly createdAtIso: string | null;+ /** The `merged_into:` pointer the page carries, or `null`. */+ readonly mergedInto: string | null;+ /** Folded composite scope key; empty when the page declares no scope. */+ readonly scopeKey: string;+}++/** One scored pair, carried into finding evidence. */+interface PairScore {+ readonly pair: readonly [string, string];+ readonly similarity: number;+}++/**+ * Every ingest summary page under `Brain/sources/`, in deterministic+ * order, after the three admission filters:+ *+ * - the caller's `readable` predicate (a refused page is absent - it+ * is not scored, not grouped and not named);+ * - the candidate cap, which bounds files READ and is applied before+ * the merge-resolved filter so the bound stays a bound;+ * - the merge-resolved predicate (GitHub #180), applied after the cap+ * the same way the preference dedup detector applies it.+ *+ * A page whose frontmatter does not parse is skipped, as in every other+ * detector; corruption is the doctor's job, not a scan abort.+ */+function collectDocumentRecords(+ vault: string,+ readable?: (rel: string) => boolean,+): DocumentRecord[] {+ const dir = join(vault, BRAIN_SOURCES_REL);+ if (!existsSync(dir)) return [];+ const out: DocumentRecord[] = [];+ for (const name of readdirSync(dir).toSorted()) {+ if (!name.endsWith(".md")) continue;+ const abs = join(dir, name);+ let meta: Record<string, unknown>;+ let body: string;+ try {+ [meta, body] = parseFrontmatter(abs);+ } catch {+ continue;+ }+ if (meta["kind"] !== BRAIN_SOURCE_KIND) continue;+ const path = `${BRAIN_SOURCES_REL}/${name}`;+ if (readable !== undefined && !readable(path)) continue;+ const rawCreated = meta["created_at"];+ const createdAtIso =+ typeof rawCreated === "string" && Number.isFinite(Date.parse(rawCreated)) ? rawCreated : null;+ out.push({+ path,+ body,+ createdAtIso,+ mergedInto: readMergedInto(meta),+ scopeKey: compositeScopeKey(scopeFromFrontmatter(meta)),+ });+ if (out.length >= DOCUMENT_DEDUP_CANDIDATE_CAP) break;+ }+ const pointerOf = mergePointerLookup(out.map((r) => [r.path, r.mergedInto] as const));+ return out.filter((r) => !isMergeResolved(r.path, pointerOf));+}++/**+ * Union-find over scored pairs: the groups retirement acts on. Pairs in+ * different scope buckets never share a component, because the bucket+ * walk below never produced such a pair.+ */+function connectedGroups(+ paths: ReadonlyArray<string>,+ pairs: ReadonlyArray<PairScore>,+): string[][] {+ const adjacency = new Map<string, string[]>();+ for (const path of paths) adjacency.set(path, []);+ for (const { pair } of pairs) {+ adjacency.get(pair[0])?.push(pair[1]);+ adjacency.get(pair[1])?.push(pair[0]);+ }+ const groups: string[][] = [];+ const seen = new Set<string>();+ for (const start of paths) {+ if (seen.has(start)) continue;+ const group: string[] = [];+ const stack = [start];+ while (stack.length > 0) {+ const current = stack.pop()!;+ if (seen.has(current)) continue;+ seen.add(current);+ group.push(current);+ for (const next of adjacency.get(current) ?? []) {+ if (!seen.has(next)) stack.push(next);+ }+ }+ if (group.length >= 2) groups.push(group.toSorted());+ }+ return groups;+}++/**+ * One finding per connected group. Canonical is the newest `created_at`+ * (ties broken by path ascending); any member without `created_at`+ * makes the whole group ineligible - the mtime fallback that ordering+ * would need is the one the merge itself moves (GitHub #180).+ */+function findingsFromGroups(+ records: ReadonlyArray<DocumentRecord>,+ pairs: ReadonlyArray<PairScore>,+ meta: { readonly method: DedupMethod; readonly threshold: number; readonly model?: string },+): HygieneFinding[] {+ const byPath = new Map(records.map((r) => [r.path, r]));+ const paths = records.map((r) => r.path);+ const findings: HygieneFinding[] = [];+ for (const group of connectedGroups(paths, pairs)) {+ const members = group.map((path) => byPath.get(path)!);+ const membersWithoutCreated = members.filter((m) => m.createdAtIso === null).map((m) => m.path);+ const groupPairs = pairs+ .filter((p) => group.includes(p.pair[0]) && group.includes(p.pair[1]))+ .toSorted((a, b) => (a.pair[0] < b.pair[0] ? -1 : a.pair[0] > b.pair[0] ? 1 : 0));+ const shared = {+ method: meta.method,+ threshold: meta.threshold,+ similarities: groupPairs,+ ...(meta.model !== undefined ? { model: meta.model } : {}),+ };+ if (membersWithoutCreated.length > 0) {+ const targets = members.map((m) => m.path);+ findings.push(+ Object.freeze({+ id: hygieneFindingId("document-dedup", targets),+ detector: "document-dedup" as const,+ severity: "warning" as const,+ title: `Near-duplicate documents ${targets.join(", ")} (not retirement-eligible)`,+ targets: Object.freeze(targets),+ proposed_action: "review" as const,+ evidence: Object.freeze({+ ...shared,+ ineligible: true,+ reason: DOCUMENT_DEDUP_INELIGIBLE_REASON,+ members_without_created_at: Object.freeze(membersWithoutCreated),+ }),+ }),+ );+ continue;+ }+ // Keep-newest: the newest created_at wins, ties break on the path+ // ascending. Deterministic and merge-stable - no field this decision+ // writes can move the field it sorted on.+ const canonical = members.toSorted((a, b) => {+ const diff = Date.parse(b.createdAtIso!) - Date.parse(a.createdAtIso!);+ if (diff !== 0) return diff;+ return a.path < b.path ? -1 : a.path > b.path ? 1 : 0;+ })[0]!;+ // The transitive-similarity gate. The component the union-find built+ // can be a chain (A~B, B~C, A!~C): keep-newest then picks a canonical+ // that one END of the chain is not similar to at all, and retiring+ // that end would delete a page no score ever tied to the survivor.+ // So a member retires only on its OWN recorded pair against the+ // canonical; the rest fall to review with the reason named.+ const directToCanonical = new Set<string>([canonical.path]);+ for (const pair of pairs) {+ const [x, y] = pair.pair;+ if (pair.similarity < meta.threshold) continue;+ if (x === canonical.path && byPath.has(y)) directToCanonical.add(y);+ if (y === canonical.path && byPath.has(x)) directToCanonical.add(x);+ }+ const heldForReview = members.filter((m) => !directToCanonical.has(m.path));+ if (heldForReview.length > 0) {+ const retiring = members.filter((m) => directToCanonical.has(m.path));+ const reviewTargets = [...heldForReview.map((m) => m.path), canonical.path].toSorted();+ findings.push(+ Object.freeze({+ id: hygieneFindingId(+ "document-dedup",+ members.map((m) => m.path),+ ),+ detector: "document-dedup" as const,+ severity: "warning" as const,+ title: `Near-duplicate documents ${reviewTargets.join(", ")} (not retirement-eligible)`,+ targets: Object.freeze(reviewTargets),+ proposed_action: "review" as const,+ evidence: Object.freeze({+ ...shared,+ canonical: canonical.path,+ ineligible: true,+ reason: DOCUMENT_DEDUP_TRANSITIVE_REASON,+ members_without_direct_pair: Object.freeze(heldForReview.map((m) => m.path).toSorted()),+ }),+ }),+ );+ // The similar subgroup still retires; its evidence narrows to the+ // pairs inside it, so the finding never cites a similarity the+ // canonical itself does not answer for.+ const retiringPaths = new Set(retiring.map((m) => m.path));+ const createdAt: Record<string, string> = {};+ for (const member of retiring) createdAt[member.path] = member.createdAtIso!;+ findings.push(+ Object.freeze({+ id: hygieneFindingId(+ "document-dedup",+ retiring.map((m) => m.path),+ ),+ detector: "document-dedup" as const,+ severity: "warning" as const,+ title: `Near-duplicate documents ${retiring.map((m) => m.path).join(", ")}`,+ targets: Object.freeze(retiring.map((m) => m.path)),+ proposed_action: "retire" as const,+ evidence: Object.freeze({+ ...shared,+ similarities: groupPairs.filter(+ (p) => retiringPaths.has(p.pair[0]) && retiringPaths.has(p.pair[1]),+ ),+ canonical: canonical.path,+ created_at: createdAt,+ }),+ }),+ );+ continue;+ }+ const createdAt: Record<string, string> = {};+ for (const member of members) createdAt[member.path] = member.createdAtIso!;+ findings.push(+ Object.freeze({+ id: hygieneFindingId(+ "document-dedup",+ members.map((m) => m.path),+ ),+ detector: "document-dedup" as const,+ severity: "warning" as const,+ title: `Near-duplicate documents ${members.map((m) => m.path).join(", ")}`,+ targets: Object.freeze(members.map((m) => m.path)),+ proposed_action: "retire" as const,+ evidence: Object.freeze({+ ...shared,+ canonical: canonical.path,+ created_at: createdAt,+ }),+ }),+ );+ }+ // Deterministic report order: by the group's first (lexicographically+ // smallest) target path, never by discovery order.+ return findings.toSorted((a, b) => (a.targets[0]! < b.targets[0]! ? -1 : 1));+}++/** Lexical pairing: token jaccard within scope buckets at the threshold. */+function lexicalPairs(records: ReadonlyArray<DocumentRecord>, threshold: number): PairScore[] {+ const entries = records.map((r) => ({+ id: r.path,+ bucketKey: r.scopeKey,+ tokens: tokenise(r.body),+ source: r,+ }));+ return findSimilarPairs(entries, { threshold }).map((pair) => ({+ pair: [pair.a.id, pair.b.id] as const,+ similarity: roundScore(pair.jaccard),+ }));+}++/**+ * Document-dedup detector, lexical layer. Deterministic, provider-free;+ * this is the form the hygiene scan registry runs.+ */+export function detectDocumentDedup(+ vault: string,+ ctx: HygieneDetectorContext,+ opts: DocumentDedupOptions = {},+): ReadonlyArray<HygieneFinding> {+ const threshold = opts.lexicalThreshold ?? DOCUMENT_DEDUP_LEXICAL_THRESHOLD;+ const records = collectDocumentRecords(vault, ctx.readable).filter(+ (r) => tokenise(r.body).size >= NEAR_DUPLICATE_MIN_TOKENS,+ );+ if (records.length < 2) return [];+ return findingsFromGroups(records, lexicalPairs(records, threshold), {+ method: "lexical",+ threshold,+ });+}++function resolveConfiguredProvider(vault: string): EmbeddingProvider | null {+ try {+ const provider = makeProvider(resolveSearchConfig({ vault }).semantic);+ return providerProducesVectors(provider) ? provider : null;+ } catch {+ return null;+ }+}++/**+ * Cosine of two EQUAL-dimension vectors - the caller guarantees the+ * dimension check above, because truncating a shorter vector into a+ * longer space scores a similarity over incomparable coordinates.+ */+function cosine(a: ReadonlyArray<number>, b: ReadonlyArray<number>): number {+ if (a.length !== b.length) return 0;+ let dot = 0;+ let normA = 0;+ let normB = 0;+ for (let i = 0; i < a.length; i++) {+ dot += a[i]! * b[i]!;+ normA += a[i]! * a[i]!;+ normB += b[i]! * b[i]!;+ }+ if (normA === 0 || normB === 0) return 0;+ return dot / (Math.sqrt(normA) * Math.sqrt(normB));+}++/**+ * Document-dedup detector, embedding layer: embed every eligible body+ * (bounded by the same cap as the lexical layer) and group pairs whose+ * cosine reaches the threshold, within scope buckets. Any provider+ * problem falls back to the deterministic lexical layer, labeled as+ * such - a report never passes lexical similarity off as semantic.+ */+export async function detectSemanticDocumentDedup(+ vault: string,+ opts: SemanticDocumentDedupOptions = {},+): Promise<SemanticDocumentDedupResult> {+ const provider = opts.provider === undefined ? resolveConfiguredProvider(vault) : opts.provider;+ const lexical = (): SemanticDocumentDedupResult =>+ Object.freeze({+ method: "lexical" as const,+ findings: detectDocumentDedup(+ vault,+ {+ now: new Date(0),+ ...(opts.readable !== undefined ? { readable: opts.readable } : {}),+ },+ opts.lexicalThreshold !== undefined ? { lexicalThreshold: opts.lexicalThreshold } : {},+ ),+ });+ // An INJECTED provider reaches here without passing through+ // `resolveConfiguredProvider`, so the contract predicate is asked again+ // rather than assumed - the same provenance-at-the-boundary rule the+ // preference dedup detector applies.+ if (provider === null || !providerProducesVectors(provider)) return lexical();++ const records = collectDocumentRecords(vault, opts.readable).filter(+ (r) => tokenise(r.body).size >= NEAR_DUPLICATE_MIN_TOKENS,+ );+ if (records.length < 2) return lexical();+ const threshold = opts.threshold ?? DOCUMENT_DEDUP_EMBEDDING_THRESHOLD;++ let vectors: number[][];+ try {+ vectors = await provider.embed(records.map((r) => r.body));+ } catch {+ return lexical();+ }+ if (vectors.length !== records.length) return lexical();+ // A ragged batch - vectors of differing dimensions - is the same+ // broken-provider shape as a count mismatch: the cosine below would+ // silently truncate to the shorter vector and fabricate a similarity+ // out of incomparable spaces, so the whole run answers lexical, labeled.+ const dimension = vectors[0]!.length;+ if (vectors.some((v) => v.length !== dimension)) return lexical();++ const pairs: PairScore[] = [];+ for (let i = 0; i < records.length; i++) {+ for (let j = i + 1; j < records.length; j++) {+ // Scope buckets fold here too: a similarity score must not+ // overrule the operator's scope boundary.+ if (records[i]!.scopeKey !== records[j]!.scopeKey) continue;+ const similarity = cosine(vectors[i]!, vectors[j]!);+ if (similarity < threshold) continue;+ pairs.push({+ pair: [records[i]!.path, records[j]!.path],+ similarity: roundScore(similarity),+ });+ }+ }+ return Object.freeze({+ method: "embedding" as const,+ findings: Object.freeze(+ findingsFromGroups(records, pairs, {+ method: "embedding",+ threshold,+ model: provider.model,+ }),+ ),+ });+}diff --git a/src/core/brain/hygiene/plan.ts b/src/core/brain/hygiene/plan.tsindex c07297e3..956644d2 100644--- a/src/core/brain/hygiene/plan.ts+++ b/src/core/brain/hygiene/plan.ts@@ -14,7 +14,7 @@*/import { REPAIR_VERDICT, requireRepairCapability } from "../applier-capability.ts";-import type { HygieneFinding, HygieneScanReport } from "./types.ts";+import type { HygieneFinding, HygieneProposedAction, HygieneScanReport } from "./types.ts";export interface HygienePlan {/** Findings selected for execution (never `review`). */@@ -30,12 +30,28 @@ export interface BuildHygienePlanOptions {readonly ids?: ReadonlyArray<string>;}+/**+ * Actions the unattended default selection never takes. A finding whose+ * action sits here is mechanically repairable - the capability table+ * says so - but the repair is destructive enough that it runs only when+ * a caller names its finding id, so nothing an uninvited sweep selects+ * can ever delete or drop a record. `retire` removes pages and drops+ * source records: it joins this set, and the applier-capability census+ * holds both halves of that claim (excluded unattended, selected+ * attended).+ */+export const UNATTENDED_EXCLUDED_ACTIONS: ReadonlySet<HygieneProposedAction> = new Set(["retire"]);+export function buildHygienePlan(report: HygieneScanReport,opts: BuildHygienePlanOptions = {},): HygienePlan {const byId = new Map(report.findings.map((finding) => [finding.id, finding]));const requested = opts.ids ?? report.findings.map((finding) => finding.id);+ // Explicit ids are the attended path: the caller read a scan and named+ // what should run. The omitted-ids default is unattended and holds the+ // destructive-class actions back even though they are mechanical.+ const attended = opts.ids !== undefined;const selected: HygieneFinding[] = [];const excludedReview: string[] = [];@@ -50,6 +66,10 @@ export function buildHygienePlan(excludedReview.push(id);continue;}+ if (!attended && UNATTENDED_EXCLUDED_ACTIONS.has(finding.proposed_action)) {+ excludedReview.push(id);+ continue;+ }selected.push(finding);}diff --git a/src/core/brain/hygiene/scan.ts b/src/core/brain/hygiene/scan.tsindex be6b1347..c36fa66c 100644--- a/src/core/brain/hygiene/scan.ts+++ b/src/core/brain/hygiene/scan.ts@@ -13,6 +13,7 @@import { detectCaptureScope } from "./detectors/capture-scope.ts";import { detectConflicts } from "./detectors/conflicts.ts";import { detectDedup } from "./detectors/dedup.ts";+import { detectDocumentDedup } from "./detectors/document-dedup.ts";import { detectFreshness } from "./detectors/freshness.ts";import { detectFrontmatterTags } from "./detectors/frontmatter-tags.ts";import { detectSlugCollisions } from "./detectors/slug-collisions.ts";@@ -31,6 +32,7 @@ import {const DETECTORS: Readonly<Record<HygieneDetectorId, HygieneDetector>> = Object.freeze({conflicts: (vault) => detectConflicts(vault),dedup: (vault, ctx) => detectDedup(vault, ctx),+ "document-dedup": (vault, ctx) => detectDocumentDedup(vault, ctx),freshness: (vault) => detectFreshness(vault),usefulness: (vault, ctx) => detectUsefulness(vault, ctx),"slug-collisions": (vault) => detectSlugCollisions(vault),diff --git a/src/core/brain/hygiene/types.ts b/src/core/brain/hygiene/types.tsindex 0a413200..91332e06 100644--- a/src/core/brain/hygiene/types.ts+++ b/src/core/brain/hygiene/types.ts@@ -12,6 +12,7 @@export const HYGIENE_DETECTOR_IDS = ["conflicts","dedup",+ "document-dedup","freshness","usefulness","slug-collisions",@@ -34,7 +35,10 @@ export type HygieneDetectorId = (typeof HYGIENE_DETECTOR_IDS)[number];* vaults that hold malformed entries), and further noisy detectors* register as opt-in by staying out of this list. `capture-scope` is* default-on: it fires only on retrievable knowledge whose every- * source is url-only.+ * source is url-only. `document-dedup` is opt-in: its remediation is+ * the destructive-class `retire` action, so a vault asks for it+ * explicitly rather than finding retirement candidates in every+ * uninvited sweep.** Members are compile-checked against the registered tuple; a registered* id missing here is simply default-off, never an error.@@ -64,6 +68,7 @@ export const HYGIENE_PROPOSED_ACTIONS = ["archive","recompile","forget",+ "retire","review",] as const;diff --git a/src/core/brain/ingest/content-manifest.ts b/src/core/brain/ingest/content-manifest.tsindex de397d40..ceb8b1c5 100644--- a/src/core/brain/ingest/content-manifest.ts+++ b/src/core/brain/ingest/content-manifest.ts@@ -95,6 +95,32 @@ export interface ManifestEntry {* contract, never skips.*/readonly fingerprint?: string;+ /**+ * The extraction watermark (t_a9cd4221): how many leading bytes of the+ * source the last extracting pass actually covered, when it recorded one.+ * ADDITIVE by design - schema_version stays 3, and a read of an entry+ * without the field answers "never extracted" ({@link extractedBytesOf}+ * degrades to byte 0), which reprocesses fully. The lazy direction: no+ * field shape change ever forces an eager mass reprocess. A malformed+ * shape on disk drops the FIELD, never the entry, for the same reason.+ */+ readonly extracted_bytes?: number;+ /**+ * When the pass that reached {@link extracted_bytes} finished, ISO-8601.+ * Absent together with the byte count on unwatermarked entries.+ */+ readonly extracted_at?: string;+}++/**+ * The entry's extraction watermark, degraded the lazy way: an entry recorded+ * before watermarks existed - or with a malformed one on disk - answers+ * byte 0, "never extracted", so the source reprocesses fully instead of+ * skipping. Never an eager mass reprocess: the absence costs one pass, once,+ * per source that actually grows again.+ */+export function extractedBytesOf(entry: ManifestEntry): number {+ return entry.extracted_bytes ?? 0;}/**@@ -271,12 +297,37 @@ function parseEntryValue(value: unknown): ManifestEntry | null {const record = value as Record<string, unknown>;if (typeof record["sha256"] !== "string") return null;const fingerprint = record["fingerprint"];+ // The watermark fields are accepted per-field and only when well-shaped:+ // a malformed one drops JUST the field, which answers byte 0 and+ // reprocesses lazily - dropping the whole entry would forfeit the+ // fingerprint and force the eager contract-changed reprocess instead.+ const extractedBytes = record["extracted_bytes"];+ const extractedAt = record["extracted_at"];return {sha256: record["sha256"],...(typeof fingerprint === "string" ? { fingerprint } : {}),+ ...(isByteOffset(extractedBytes) ? { extracted_bytes: extractedBytes } : {}),+ ...(typeof extractedAt === "string" && extractedAt.length > 0+ ? { extracted_at: extractedAt }+ : {}),};}+/** Is this value a byte offset a watermark may record? */+function isByteOffset(value: unknown): value is number {+ return typeof value === "number" && Number.isSafeInteger(value) && value >= 0;+}++/**+ * Does the entry carry recorded state the bare v1 digest shape cannot hold -+ * a per-entry fingerprint or an extraction watermark? Such an entry never+ * serializes down to v1: data a write could read back is data the write+ * must keep.+ */+function hasRecordedState(entry: ManifestEntry): boolean {+ return entry.fingerprint !== undefined || entry.extracted_bytes !== undefined;+}+/*** The changed-contract decision, derived from the entries against the LIVE* contract: `null` while any entry predates it (no fingerprint, or a@@ -317,10 +368,7 @@ function serializeManifest(const stamped: Record<string, ManifestEntry> = {};for (const key of sortedKeys) stamped[key] = { ...entries[key]!, fingerprint: live };body = { schema_version: SCHEMA_VERSION, contract: { fingerprint: live }, entries: stamped };- } else if (- contract === null &&- !Object.values(entries).some((entry) => entry.fingerprint !== undefined)- ) {+ } else if (contract === null && !Object.values(entries).some(hasRecordedState)) {// Legacy-shaped entries: the V1 shape - digests only, contract-less - so// a v1 read round-trips as v1 and the changed-contract marker survives.const digests: Record<string, string> = {};@@ -443,6 +491,12 @@ export function classifyPaths(* entirely when nothing changed. Returns `true` when the manifest was* rewritten.*+ * `records` pins a path's FULL entry: the caller computed the digest itself+ * and carries extra recorded state (the extraction watermark) the default+ * hash-and-stamp must not drop. A pinned path on disk records its entry+ * verbatim; a pinned path missing from disk is still dropped, like any+ * deleted source.+ ** The read, the merge and the write are ONE critical section, held under the* same sync lock every other Brain read-modify-write takes. Atomicity alone is* not enough: `atomicWriteFileSync` renames a complete file into place, so no@@ -451,7 +505,11 @@ export function classifyPaths(* The batch planner dispatches sources across parallel subagents, so that* overlap is the designed case rather than a rare one.*/-export function updateManifest(vault: string, paths: readonly string[]): boolean {+export function updateManifest(+ vault: string,+ paths: readonly string[],+ records: Readonly<Record<string, ManifestEntry>> = {},+): boolean {const handle = acquireLockSyncWithRetry(manifestPath(vault), undefined, {remedy: ingestLockRemedy,});@@ -461,11 +519,13 @@ export function updateManifest(vault: string, paths: readonly string[]): booleanfor (const path of paths) {const canonical = canonicalNotePath(path);const abs = join(vault, canonical);- if (existsSync(abs)) {- entries[canonical] = { sha256: hashPath(abs), fingerprint: live };- } else {+ if (!existsSync(abs)) {delete entries[canonical];+ continue;}+ const pinned = records[canonical];+ entries[canonical] =+ pinned !== undefined ? pinned : { sha256: hashPath(abs), fingerprint: live };}// Explicit contract: the live fingerprint rides along as the// current-contract MARKER only. The omitted form would stamp every entrydiff --git a/src/core/brain/ingest/ingest.ts b/src/core/brain/ingest/ingest.tsindex 959dd26b..4e39b3e9 100644--- a/src/core/brain/ingest/ingest.ts+++ b/src/core/brain/ingest/ingest.ts@@ -21,14 +21,14 @@import {closeSync,- constants as fsConstants,existsSync,- fstatSync,mkdirSync,openSync,readFileSync,readSync,+ statSync,} from "node:fs";+import { createHash } from "node:crypto";import { dirname, join, relative } from "node:path";import type { FrontmatterMap } from "../../types.ts";@@ -81,7 +81,15 @@ import {type Provenance,} from "../provenance/provenance.ts";import { PLAN_ID_LABEL, recordCompleted } from "./checkpoint.ts";-import { readManifest, updateManifest } from "./content-manifest.ts";+import { extractedBytesOf, hashPath, readManifest, updateManifest } from "./content-manifest.ts";+import { computeExtractionContractFingerprint } from "./contract.ts";+import { readTail } from "./tail-read.ts";+import {+ PRE_EXTRACT_MAX_SOURCE_BYTES,+ SOURCE_OPEN_FLAGS,+ SOURCE_UNREAD,+ readSourceBounded,+} from "./source-read.ts";import { deriveSourceSection, type PartsOutcome, type TableOutcome } from "./extract-source.ts";import {isCodeStructureSource,@@ -238,6 +246,26 @@ export interface IngestSourceResult {* the ordinary idempotent path once both endpoints are published.*/readonly deferredRelations?: ReadonlyArray<IntakeRelation>;+ /**+ * The appended delta this source grew by since its recorded extraction+ * watermark (t_a9cd4221): the boundary-safe byte range+ * {@link ./tail-read.ts} verified as a pure append - the recorded digest+ * still described the leading bytes. Present ONLY on that append shape: a+ * first ingest, a rewritten source, or a source with no watermark reports+ * no delta and reprocesses fully. The range is reporting, not state - the+ * stamp below always records the whole current source, because this+ * pipeline's contract is a whole-source extraction per run.+ *+ * The whole-source stamp is also what bounds a delta series: the tail+ * cut ends at the LAST COMPLETE newline, so an append with no complete+ * line yet reports an EMPTY delta (`toByte === fromByte`) while the+ * stamp still advances to the file's full size - the unterminated+ * fragment is extracted (extraction is whole-source) but announced by+ * no delta, and the next append's delta starts PAST it rather than+ * stitching it in. A fragment only enters a reported delta when the+ * append that carries it ends with a newline before this run reads.+ */+ readonly extractionDelta?: { readonly fromByte: number; readonly toByte: number };}/**@@ -446,6 +474,7 @@ export function ingestSource(// stays unrecorded until a publish arm actually writes; a re-ingest then// re-classifies the source as new and re-runs, which is the fail-safe// direction.+ let extractionDelta: IngestSourceResult["extractionDelta"];if (!withheld &&disposition.verdict !== "stage" &&@@ -453,7 +482,49 @@ export function ingestSource(resolvesInsideVault(vault, canonicalSource) &&existsSync(join(vault, canonicalSource))) {- updateManifest(vault, [canonicalSource]);+ const sourceAbs = join(vault, canonicalSource);+ // The source can still vanish (ENOENT) or change kind (a directory where+ // the classified file stood) between the existsSync above and these+ // reads - after the summary pages were written, where a throw would lose+ // this run's result. The old updateManifest path answered both shapes by+ // degrading (a vanished source dropped its entry), so both reads here+ // degrade the same fail-safe way: a source that cannot be read reports+ // no delta, and a stamp that cannot describe it is skipped - the source+ // simply reprocesses fully next time. The ENOENT window and the EISDIR+ // window are the same catch.+ try {+ extractionDelta = appendedExtractionDelta(vault, sourceAbs, canonicalSource);+ } catch {+ // No provable append can be reported for a source that cannot be read.+ }+ // Record the source's state under the live contract, WITH the extraction+ // watermark: this run's contract is a whole-source extraction, so the+ // stamp covers the full current file. That is what makes the NEXT+ // append verifiable - the recorded digest describes exactly the leading+ // extracted_bytes, so a later read either proves a pure append (delta+ // above) or classifies modified and reprocesses fully.+ let sourceSize: number | undefined;+ let sourceDigest: string | undefined;+ try {+ // One stat, taken BEFORE the digest: the stamp must describe one+ // snapshot of the source, so the byte count it claims is the file it+ // hashed, not a size re-read after the hashing walked a possibly+ // grown file.+ sourceSize = statSync(sourceAbs).size;+ sourceDigest = hashPath(sourceAbs);+ } catch {+ // Vanished or changed kind mid-flight: skip the stamp, reprocess fully.+ }+ if (sourceDigest !== undefined && sourceSize !== undefined) {+ updateManifest(vault, [canonicalSource], {+ [canonicalSource]: {+ sha256: sourceDigest,+ fingerprint: computeExtractionContractFingerprint(vault),+ extracted_bytes: sourceSize,+ extracted_at: isoSecond(opts.now),+ },+ });+ }// Record plan-scoped progress so an interrupted batch resumes at the item// boundary (t_ba1fa5f6). Only for real vault files - a URL/identity-only// source has no place in a folder plan's checkpoint. Best-effort: a@@ -486,6 +557,7 @@ export function ingestSource(...(intake.deferredRelations !== undefined? { deferredRelations: intake.deferredRelations }: {}),+ ...(extractionDelta !== undefined ? { extractionDelta } : {}),...(preExtract !== undefined ? { preExtract } : {}),...(derivation?.parts !== undefined ? { parts: derivation.parts } : {}),...(derivation?.table !== undefined ? { table: derivation.table } : {}),@@ -493,11 +565,17 @@ export function ingestSource(}/**- * The largest source the code-structure pass reads. A hand-written module- * is far smaller; a larger file is generated or vendored, and its seeds are- * not worth a parse whose cost grows with the file.+ * The shared bounded source-read substrate moved to {@link ./source-read.ts}:+ * the open discipline, the named refusal vocabulary, the whole-source bounded+ * read and the shared scale cap. The names re-export here so every existing+ * consumer keeps its import.*/-export const PRE_EXTRACT_MAX_SOURCE_BYTES = 1_048_576;+export {+ PRE_EXTRACT_MAX_SOURCE_BYTES,+ readSourceBounded,+ SOURCE_UNREAD,+ type SourceUnread,+} from "./source-read.ts";/** What the trusted-local branch's archive attempt concluded. */interface ArchiveCapture {@@ -569,57 +647,77 @@ function captureTrustedSourceArchive(}}-/** Why a source read yields no text, apart from a thrown error. */-const SOURCE_UNREAD = Object.freeze({- notAFile: "not a regular file",- tooLarge: "larger than the read limit",-} as const);+/**+ * The largest appended tail the extraction-delta pass reads ({@link ./tail-read.ts}):+ * a tail past this is a reprocess-scale change, not a delta. Starts from+ * {@link PRE_EXTRACT_MAX_SOURCE_BYTES} - the same scale rule the+ * whole-source pre-extract read applies. Exported so the CLI extract+ * preview draws its appended section under exactly this cap and never+ * reports a slice the pipeline itself would refuse.+ */+export const EXTRACTION_DELTA_MAX_BYTES = PRE_EXTRACT_MAX_SOURCE_BYTES;-/** Why {@link readSourceBounded} read no text. */-export type SourceUnread = (typeof SOURCE_UNREAD)[keyof typeof SOURCE_UNREAD];+/** Bytes hashed per positioned read while verifying an append's prefix. */+const PREFIX_HASH_CHUNK_BYTES = 65_536;/**- * How a source is opened: read-only and never blocking, so a FIFO met- * where a file was expected cannot stall the server. No `O_NOFOLLOW`: an- * in-vault symlink is admitted by the containment check in- * `runPreExtract` (`resolvesInsideVault`). The flag- * is POSIX; where the platform lacks it it is simply absent.+ * The appended delta `sourceAbs` grew by since its recorded extraction+ * watermark, when it grew by a PROVABLE pure append: the recorded digest+ * must still describe exactly the leading watermark bytes, or the source was+ * rewritten rather than appended and the honest answer is no delta - the+ * full reprocess the modified classification already triggers. The returned+ * range is boundary-safe ({@link ./tail-read.ts}): it ends at the last+ * complete newline, and it is empty when the append has none yet.+ *+ * Exported because the CLI extract preview reports the same fact (`o2b+ * brain extract`'s appended section): one helper, so the preview can never+ * announce a slice the pipeline's own delta pass would refuse, nor call an+ * append what the prefix proof calls a rewrite.*/-const SOURCE_OPEN_FLAGS = fsConstants.O_RDONLY | (fsConstants.O_NONBLOCK ?? 0);+export function appendedExtractionDelta(+ vault: string,+ sourceAbs: string,+ canonicalSource: string,+): { readonly fromByte: number; readonly toByte: number } | undefined {+ const entry = readManifest(vault).entries[canonicalSource];+ const watermark = entry === undefined ? 0 : extractedBytesOf(entry);+ if (watermark <= 0) return undefined;+ let size: number;+ try {+ size = statSync(sourceAbs).size;+ } catch {+ return undefined;+ }+ if (size <= watermark) return undefined;+ if (entry === undefined || !prefixHashMatches(sourceAbs, watermark, entry.sha256)) {+ return undefined;+ }+ const tail = readTail(sourceAbs, watermark, { maxBytes: EXTRACTION_DELTA_MAX_BYTES });+ if (tail.kind !== "appended") return undefined;+ return { fromByte: tail.fromByte, toByte: tail.toByte };+}/**- * Read a source through ONE descriptor, so the checks hold for the bytes- * read: the descriptor must be a regular file, and no more than `maxBytes`- * is read however large the file has grown since the `fstat`. The buffer is- * sized to the file plus one byte, so a small source costs a small buffer;- * a file that grew under the cap is read on into one buffer of the cap plus- * one byte. `bytes` are exactly the bytes read, for a caller that must judge- * their encoding itself.+ * Does the SHA-256 of the first `watermark` bytes of `sourceAbs` equal+ * `expected`? An unreadable or too-short prefix cannot PROVE an append, so+ * it answers false - the modified classification and its full reprocess are+ * the safe direction, never a skipped stale delta.*/-export function readSourceBounded(- absolute: string,- maxBytes: number,-):- | { readonly text: string; readonly bytes: Uint8Array; readonly unread?: undefined }- | { readonly text: null; readonly bytes?: undefined; readonly unread: SourceUnread } {- const fd = openSync(absolute, SOURCE_OPEN_FLAGS);+function prefixHashMatches(sourceAbs: string, watermark: number, expected: string): boolean {+ const fd = openSync(sourceAbs, SOURCE_OPEN_FLAGS);try {- const stat = fstatSync(fd);- if (!stat.isFile()) return { text: null, unread: SOURCE_UNREAD.notAFile };- if (stat.size > maxBytes) return { text: null, unread: SOURCE_UNREAD.tooLarge };- // One byte past the size, so growth since the fstat is seen.- let buffer = Buffer.allocUnsafe(stat.size + 1);- let filled = 0;- for (;;) {- // A full buffer is under the cap here: past it, the loop has returned.- if (filled === buffer.length) buffer = Buffer.concat([buffer], maxBytes + 1);- const count = readSync(fd, buffer, filled, buffer.length - filled, null);- if (count === 0) break;- filled += count;- if (filled > maxBytes) return { text: null, unread: SOURCE_UNREAD.tooLarge };+ const hash = createHash("sha256");+ const chunk = Buffer.allocUnsafe(Math.min(PREFIX_HASH_CHUNK_BYTES, watermark));+ let position = 0;+ let remaining = watermark;+ while (remaining > 0) {+ const count = readSync(fd, chunk, 0, Math.min(chunk.length, remaining), position);+ if (count === 0) return false;+ hash.update(count === chunk.length ? chunk : chunk.subarray(0, count));+ position += count;+ remaining -= count;}- const bytes = buffer.subarray(0, filled);- return { text: bytes.toString("utf8"), bytes };+ return hash.digest("hex") === expected;} finally {closeSync(fd);}diff --git a/src/core/brain/ingest/source-read.ts b/src/core/brain/ingest/source-read.tsnew file mode 100644index 00000000..761c83a3--- /dev/null+++ b/src/core/brain/ingest/source-read.ts@@ -0,0 +1,84 @@+/**+ * The shared bounded source-read substrate (Lane 5, t_a9cd4221).+ *+ * Everything the ingest lane's file reads share, hoisted into one leaf so+ * the whole-read and the from-offset tail read ({@link ./tail-read.ts})+ * cannot grow import cycles around the pipeline module:+ *+ * - the one open discipline: read-only, never blocking, so a FIFO met+ * where a file was expected cannot stall the server;+ * - the named refusal vocabulary every bounded read answers with - data+ * on the wire, never a thrown error;+ * - the whole-source bounded read itself;+ * - the shared scale cap: the largest source the code-structure pass+ * reads, which is also the scale at which an appended tail stops being+ * a delta and becomes a reprocess.+ *+ * Language-agnostic: bytes in, decoded UTF-8 text out; no natural-language+ * content is inspected.+ */++import { closeSync, constants as fsConstants, fstatSync, openSync, readSync } from "node:fs";++/**+ * The largest source the code-structure pass reads. A hand-written module+ * is far smaller; a larger file is generated or vendored, and its seeds are+ * not worth a parse whose cost grows with the file.+ */+export const PRE_EXTRACT_MAX_SOURCE_BYTES = 1_048_576;++/** Why a source read yields no text, apart from a thrown error. */+export const SOURCE_UNREAD = Object.freeze({+ notAFile: "not a regular file",+ tooLarge: "larger than the read limit",+} as const);++/** Why a bounded source read read no text. */+export type SourceUnread = (typeof SOURCE_UNREAD)[keyof typeof SOURCE_UNREAD];++/**+ * How a source is opened: read-only and never blocking, so a FIFO met+ * where a file was expected cannot stall the server. No `O_NOFOLLOW`: an+ * in-vault symlink is admitted by the containment check in+ * `runPreExtract` (`resolvesInsideVault`). The flag+ * is POSIX; where the platform lacks it it is simply absent.+ */+export const SOURCE_OPEN_FLAGS = fsConstants.O_RDONLY | (fsConstants.O_NONBLOCK ?? 0);++/**+ * Read a source through ONE descriptor, so the checks hold for the bytes+ * read: the descriptor must be a regular file, and no more than `maxBytes`+ * is read however large the file has grown since the `fstat`. The buffer is+ * sized to the file plus one byte, so a small source costs a small buffer;+ * a file that grew under the cap is read on into one buffer of the cap plus+ * one byte. `bytes` are exactly the bytes read, for a caller that must judge+ * their encoding itself.+ */+export function readSourceBounded(+ absolute: string,+ maxBytes: number,+):+ | { readonly text: string; readonly bytes: Uint8Array; readonly unread?: undefined }+ | { readonly text: null; readonly bytes?: undefined; readonly unread: SourceUnread } {+ const fd = openSync(absolute, SOURCE_OPEN_FLAGS);+ try {+ const stat = fstatSync(fd);+ if (!stat.isFile()) return { text: null, unread: SOURCE_UNREAD.notAFile };+ if (stat.size > maxBytes) return { text: null, unread: SOURCE_UNREAD.tooLarge };+ // One byte past the size, so growth since the fstat is seen.+ let buffer = Buffer.allocUnsafe(stat.size + 1);+ let filled = 0;+ for (;;) {+ // A full buffer is under the cap here: past it, the loop has returned.+ if (filled === buffer.length) buffer = Buffer.concat([buffer], maxBytes + 1);+ const count = readSync(fd, buffer, filled, buffer.length - filled, null);+ if (count === 0) break;+ filled += count;+ if (filled > maxBytes) return { text: null, unread: SOURCE_UNREAD.tooLarge };+ }+ const bytes = buffer.subarray(0, filled);+ return { text: bytes.toString("utf8"), bytes };+ } finally {+ closeSync(fd);+ }+}diff --git a/src/core/brain/ingest/tail-read.ts b/src/core/brain/ingest/tail-read.tsnew file mode 100644index 00000000..746f0816--- /dev/null+++ b/src/core/brain/ingest/tail-read.ts@@ -0,0 +1,173 @@+/**+ * The from-offset tail read for appended source text (t_a9cd4221).+ *+ * A growing source (a chat log, a session transcript) is re-extracted one+ * appended delta at a time instead of whole, keyed on the extraction+ * watermark the content manifest records ({@link ./content-manifest.ts}).+ * This module is the byte discipline that makes such a watermark honest:+ *+ * - Deltas are measured and sliced in BYTES, but the emitted text is cut+ * only at UTF-8-safe boundaries: the start snaps forward past any+ * partial multi-byte sequence a mid-codepoint offset can land on, and+ * the end is cut after the LAST complete newline, so an unfinished+ * trailing line stays uncovered for the next delta instead of being+ * covered by two deltas that each carry a fragment of it.+ * - `fromByte`/`toByte` are the exact byte range the returned text was+ * decoded from. A watermark stamped at `toByte` therefore never+ * overlaps the next delta and never claims bytes the text does not+ * carry. When no complete line exists in the appended range, the+ * delta covers nothing (`toByte === fromByte`) - and what happens to+ * the watermark then is the CALLER's decision, not this module's.+ * The production caller ({@link ./ingest.ts}) stamps the extraction+ * watermark at the file's FULL size on every trusted run, so an+ * empty delta still advances it past the unterminated fragment:+ * that fragment is announced by no delta, and the next append's+ * delta starts PAST it rather than stitching it in. Only a caller+ * that leaves the watermark at `fromByte` gets the+ * stitch-the-fragment-later behavior this boundary cut was shaped+ * for. The ingest lane accepts the trade because extraction itself+ * stays whole-source - the fragment is extracted, just never+ * announced as a delta.+ * - A file that SHRANK below the offset (rotated, truncated, rewritten)+ * falls back to the full current content with the shrink NAMED - the+ * full-reprocess direction, never a bogus negative slice.+ * - The read is bounded: past `maxBytes` it refuses with the shared+ * bounded-read refusal vocabulary ({@link SOURCE_UNREAD}), the same+ * names {@link ./ingest.ts} answers bounded reads with.+ *+ * One descriptor per read, positioned reads past the offset: a delta of a+ * large log costs the delta's bytes, not the file's.+ *+ * Language-agnostic: bytes in, decoded UTF-8 text out; no natural-language+ * content is inspected.+ */++import { closeSync, fstatSync, openSync, readSync } from "node:fs";++import {+ PRE_EXTRACT_MAX_SOURCE_BYTES,+ SOURCE_OPEN_FLAGS,+ SOURCE_UNREAD,+ type SourceUnread,+} from "./source-read.ts";++/** Bytes pulled per positioned read, so a delta never over-allocates. */+const READ_CHUNK_BYTES = 65_536;++/** UTF-8 continuation byte: 0b10xxxxxx - never a codepoint start. */+function isContinuationByte(byte: number): boolean {+ return (byte & 0b1100_0000) === 0b1000_0000;+}++/** What one tail read concluded. */+export type TailReadResult =+ | {+ /** Text was appended past `fromByte` (possibly only up to a line boundary). */+ readonly kind: "appended";+ /** The decoded delta text; empty when no complete line exists yet. */+ readonly text: string;+ /** First byte of the source the returned text was decoded from. */+ readonly fromByte: number;+ /** One past the last byte the text covers - the watermark position. */+ readonly toByte: number;+ }+ | {+ /** The file shrank below `fromByte`; the FULL content is the answer. */+ readonly kind: "shrank";+ readonly text: string;+ readonly fromByte: 0;+ readonly toByte: number;+ }+ | {+ /** The read refused: not a regular file, or past the ceiling. */+ readonly kind: "unread";+ readonly unread: SourceUnread;+ };++export interface TailReadOptions {+ /**+ * The most text one read may pull, in bytes. Absent: the same ceiling the+ * code-structure pre-extract pass applies - a tail past it is a+ * reprocess-scale change, not a delta.+ */+ readonly maxBytes?: number;+}++/**+ * Read the text appended to `absolute` since `fromByte`, boundary-safe and+ * bounded. A missing file throws (the caller that opened the ingest knows+ * the source existed moments ago; a vanished source is its own named error,+ * exactly as for {@link ./ingest.ts}'s bounded read).+ */+export function readTail(+ absolute: string,+ fromByte: number,+ opts: TailReadOptions = {},+): TailReadResult {+ const maxBytes = opts.maxBytes ?? PRE_EXTRACT_MAX_SOURCE_BYTES;+ const fd = openSync(absolute, SOURCE_OPEN_FLAGS);+ try {+ const stat = fstatSync(fd);+ if (!stat.isFile()) return { kind: "unread", unread: SOURCE_UNREAD.notAFile };+ if (stat.size < fromByte) {+ const full = readFrom(fd, 0, maxBytes);+ if (full.overCeiling) return { kind: "unread", unread: SOURCE_UNREAD.tooLarge };+ return {+ kind: "shrank",+ text: Buffer.from(full.bytes).toString("utf8"),+ fromByte: 0,+ toByte: full.bytes.length,+ };+ }+ const tail = readFrom(fd, fromByte, maxBytes);+ if (tail.overCeiling) return { kind: "unread", unread: SOURCE_UNREAD.tooLarge };+ return appendedDelta(tail.bytes, fromByte);+ } finally {+ closeSync(fd);+ }+}++/**+ * Cut the appended bytes into an honest delta: decode from the first+ * UTF-8-safe boundary, through the last complete newline, and report the+ * exact byte range the text covers.+ */+function appendedDelta(bytes: Uint8Array, fromByte: number): TailReadResult {+ let start = 0;+ while (start < bytes.length && isContinuationByte(bytes[start]!)) start++;+ // One past the last complete newline; `start` when the range holds no+ // complete line, in which case the delta covers nothing.+ let end = bytes.length;+ while (end > start && bytes[end - 1] !== 0x0a) end--;+ return {+ kind: "appended",+ text: Buffer.from(bytes.subarray(0, end).subarray(start)).toString("utf8"),+ fromByte: fromByte + start,+ toByte: fromByte + end,+ };+}++/**+ * Positioned reads from `start` through EOF, at most `maxBytes` bytes.+ * `overCeiling` is the named refusal: the caller answers with+ * {@link SOURCE_UNREAD.tooLarge} rather than truncating silently.+ */+function readFrom(+ fd: number,+ start: number,+ maxBytes: number,+): { readonly bytes: Uint8Array; readonly overCeiling: boolean } {+ const chunks: Buffer[] = [];+ let filled = 0;+ let position = start;+ for (;;) {+ const chunk = Buffer.allocUnsafe(Math.min(READ_CHUNK_BYTES, maxBytes + 1 - filled));+ const count = readSync(fd, chunk, 0, chunk.length, position);+ if (count === 0) break;+ chunks.push(count === chunk.length ? chunk : chunk.subarray(0, count));+ filled += count;+ position += count;+ if (filled > maxBytes) return { bytes: Buffer.concat(chunks), overCeiling: true };+ }+ return { bytes: Buffer.concat(chunks), overCeiling: false };+}diff --git a/src/core/brain/maintenance/collect.ts b/src/core/brain/maintenance/collect.tsindex 5f2e5d52..e734c7be 100644--- a/src/core/brain/maintenance/collect.ts+++ b/src/core/brain/maintenance/collect.ts@@ -7,6 +7,14 @@* `digest.ts` (read-only summary) and the `brain_doctor` MCP tool* can share the same collection logic without depending on each* other.+ *+ * The collection is fail-soft per signal, and the failures it absorbs+ * are NAMED: each failed scan is handed to the `onError` observer as a+ * {@link MaintenanceCollectError} while the remaining signals still+ * collect and the actions still return. A signal that fails+ * contributes no actions either way - what the observer buys is the+ * difference between a thin report because the vault is healthy and a+ * thin report because a scan refused.*/import { lintConsolidate } from "../lint-consolidate.ts";@@ -14,39 +22,74 @@ import { findDuplicateCandidates } from "../page-dedup.ts";import { computeTokenFootprint } from "../token-footprint.ts";import { scoreActions, type ActionItem } from "./action-scorer.ts";-export function collectMaintenanceActions(vault: string): ReadonlyArray<ActionItem> {+/** The maintenance scans this module collects, named for the failure report. */+export type MaintenanceCollectSignal = "dedup" | "lint" | "token-footprint";++/** One signal's collection failure: the named scan and the error it raised. */+export interface MaintenanceCollectError {+ readonly signal: MaintenanceCollectSignal;+ readonly message: string;+}++export interface CollectMaintenanceActionsOptions {+ /**+ * Observes each per-signal collection failure as it is absorbed. The scan+ * stays fail-soft either way; the doctor, digest and health surfaces call+ * without an observer and keep today's quiet contract, and a caller+ * diagnosing a thin report passes one to hear what refused.+ */+ readonly onError?: (error: MaintenanceCollectError) => void;+}++/**+ * Run one signal's scan, absorbing a failure as a named collected error+ * instead of a bare swallow. `undefined` means the signal contributes+ * nothing this run.+ */+function collectSignal<T>(+ signal: MaintenanceCollectSignal,+ onError: (error: MaintenanceCollectError) => void,+ run: () => T,+): T | undefined {+ try {+ return run();+ } catch (err) {+ onError({ signal, message: (err as Error)?.message ?? String(err) });+ return undefined;+ }+}++export function collectMaintenanceActions(+ vault: string,+ opts: CollectMaintenanceActionsOptions = {},+): ReadonlyArray<ActionItem> {+ const onError = opts.onError ?? (() => {});let dedupCandidates: Array<{ canonicalId: string; secondaryCount: number }> = [];let staleByLifecycle: Array<{ id: string; ageDays: number }> = [];let brokenLinks: Array<{ path: string; from: string }> = [];let tokenFootprint: { total: number; warnThreshold: number } | undefined;- try {- const dedup = findDuplicateCandidates(vault);+ const dedup = collectSignal("dedup", onError, () => findDuplicateCandidates(vault));+ if (dedup !== undefined) {dedupCandidates = dedup.candidates.map((c) => ({canonicalId: c.canonical.id,secondaryCount: c.secondaries.length,}));- } catch {- // Best-effort: a vault scan failure must not break the consumer.}- try {- const lint = lintConsolidate(vault, { apply: false });+ const lint = collectSignal("lint", onError, () => lintConsolidate(vault, { apply: false }));+ if (lint !== undefined) {staleByLifecycle = lint.demotions.map((d) => ({id: d.id,ageDays: d.ageDays,}));brokenLinks = lint.fixes.map((f) => ({ path: f.path, from: f.from }));- } catch {- // ignore}- try {- const footprint = computeTokenFootprint(vault);+ const footprint = collectSignal("token-footprint", onError, () => computeTokenFootprint(vault));+ if (footprint !== undefined) {tokenFootprint = {total: footprint.total,warnThreshold: footprint.warnThreshold,};- } catch {- // ignore}return scoreActions({diff --git a/src/core/brain/maintenance/journal.ts b/src/core/brain/maintenance/journal.tsindex d12db8d6..e38417d3 100644--- a/src/core/brain/maintenance/journal.ts+++ b/src/core/brain/maintenance/journal.ts@@ -24,6 +24,7 @@ import {} from "../ledger-shards.ts";import { DERIVED_STORE_DIR } from "../path-constants.ts";import { assertVaultIdentityForWrite } from "../vault-identity.ts";+import type { StorageForecastTier } from "../storage/forecast.ts";import type { HostPressureUnmeasurableReason } from "./host-pressure.ts";import { isCustomLaneTask } from "./custom-task-id.ts";import { renameWithRetry } from "../../fs-atomic.ts";@@ -39,11 +40,11 @@ export const MAINTENANCE_JOURNAL_CAP = 500;* that wrote them. The guard is the boundary for a row this build does* not understand.*- * Two of the members are not decisions about whether to run:- * `pressure:unmeasurable` is a NOTICE that a gate could not evaluate, and- * it is emitted on its own line beside the decision, never instead of it.- * `refused:streak` is a per-task refusal, so it appears among the task- * rows rather than in place of them.+ * Three of the members are not decisions about whether to run:+ * `pressure:unmeasurable` and `capacity:unmeasurable` are NOTICES that a+ * gate could not evaluate, and each is emitted on its own line beside the+ * decision, never instead of it. `refused:streak` is a per-task refusal,+ * so it appears among the task rows rather than in place of them.*/export const MAINTENANCE_VERDICT = Object.freeze({/** The gates were open; the row's task ran (or the lane ran). */@@ -52,11 +53,15 @@ export const MAINTENANCE_VERDICT = Object.freeze({skippedBusy: "skipped:busy",/** Host pressure was measured and stood at or above the configured percentage. */skippedPressure: "skipped:pressure",+ /** The storage-capacity gate measured the vault at the critical tier. */+ skippedCapacity: "skipped:capacity",skippedLease: "skipped:lease",/** This task has failed too many times in a row to retry unforced. */refusedStreak: "refused:streak",/** The host-pressure gate could not evaluate; it was left open. */pressureUnmeasurable: "pressure:unmeasurable",+ /** The storage-capacity gate could not evaluate; it was left open. */+ capacityUnmeasurable: "capacity:unmeasurable",});export const MAINTENANCE_VERDICTS: ReadonlyArray<string> = Object.freeze(@@ -91,6 +96,19 @@ export interface MaintenanceJournalEntry {readonly pressure_percent?: number;/** Why the gate could not read one, on a `pressure:unmeasurable` row. */readonly pressure_reason?: HostPressureUnmeasurableReason;+ /** Storage tier the capacity gate read, on a `skipped:capacity` row. */+ readonly capacity_tier?: StorageForecastTier;+ /**+ * The reading behind that tier, on a `skipped:capacity` row, riding the+ * way `pressure_percent` does: the percent of the configured cap, and+ * the measured bytes and cap it is a percent of. Written together,+ * only when the gate decision carried a reading.+ */+ readonly capacity_percent?: number;+ readonly capacity_measured_bytes?: number;+ readonly capacity_cap_bytes?: number;+ /** Why the capacity gate could not read a tier, on a `capacity:unmeasurable` row. */+ readonly capacity_reason?: string;/** Consecutive journaled failures behind a `refused:streak` row. */readonly streak?: number;/**@@ -146,6 +164,30 @@ export function recordedPriceSource(: PRICE_SOURCE_UNRECORDED;}+/**+ * The row a surface below local reach may render. A `skipped:capacity`+ * row carries the reading behind the skip - the whole-vault measured+ * bytes, the configured cap and the percent of one against the other -+ * and those are the same local-reach facts the storage report refuses to+ * take below local reach: totals that move when a page the caller may+ * not read appears or goes. The journal itself keeps them (the+ * operator's own record); this drops exactly the three reading fields+ * and keeps the tier word and the unmeasurable reason, which name a+ * verdict, never a size. A surface at local reach renders the row+ * unchanged.+ */+export function journalEntryWithoutCapacityReading(+ entry: MaintenanceJournalEntry,+): MaintenanceJournalEntry {+ const {+ capacity_percent: _percent,+ capacity_measured_bytes: _measured,+ capacity_cap_bytes: _cap,+ ...rest+ } = entry;+ return rest;+}+/*** The metrics surface a completed reindex pass's spend receipt is* recorded under. Both lane front doors (`o2b brain maintenance` anddiff --git a/src/core/brain/maintenance/lane.ts b/src/core/brain/maintenance/lane.tsindex b9ad4f10..9209eb53 100644--- a/src/core/brain/maintenance/lane.ts+++ b/src/core/brain/maintenance/lane.ts@@ -1,7 +1,7 @@/*** Quiet-window maintenance lane (write-time-integrity-governance,* t_166d1226; host pressure and the failure streak, t_992f0c33). Heavy- * passes (dream, reindex) run through four gates:+ * passes (dream, reindex) run through five gates:** 1. window - a configured local-time hour window (tz-aware,* midnight wrap supported); unconfigured = always open.@@ -14,7 +14,14 @@* stays open and journals WHY, on its own line, so a* platform that cannot measure pressure never looks* like a quiet one.- * 4. lease - the expiring SQLite lease; never bypassable, even+ * 4. capacity - the vault's own storage forecast, from+ * `storage.cap_bytes`; unconfigured (the default) =+ * always open and never measured. A measured critical+ * tier skips the whole lane; where the census is+ * degenerate the gate stays open and journals WHY, on+ * its own line, for the same reason the pressure gate+ * does.+ * 5. lease - the expiring SQLite lease; never bypassable, even* with --force, because two concurrent heavy passes on* one vault is the exact failure this lane exists to* prevent.@@ -33,8 +40,8 @@* to be a way to attempt the refused task again.** `force` is the wide one: it bypasses the soft gates (window, busy,- * pressure) and every streak refusal, for an operator who wants the pass- * NOW. Making it the only way out was the defect: an operator whose+ * pressure, capacity) and every streak refusal, for an operator who wants+ * the pass NOW. Making it the only way out was the defect: an operator whose* reindex is deterministically broken had to run the whole heavy lane* with their own window, busy and host-pressure protection switched off* in order to retry one task.@@ -54,7 +61,18 @@import { listRecallTelemetry } from "../recall-telemetry.ts";import { capOutput, OPERATION, SafeguardTimeoutError } from "../safeguard.ts";-import { loadMaintenanceConfigSafe } from "../policy/load.ts";+import { loadMaintenanceConfigSafe, loadStorageConfigSafe } from "../policy/load.ts";+import type { ResolvedBrainStorageConfig } from "../types.ts";+import {+ STORAGE_FORECAST_TIER,+ sampleStorageUsage,+ type StorageForecastTier,+} from "../storage/forecast.ts";+import {+ CAPACITY_GATE_DECISION,+ evaluateCapacityGate,+ type CapacityGateDecision,+} from "../storage/gate.ts";import { acquireLease, MAINTENANCE_LEASE_NAME, releaseLease } from "./lease.ts";import {appendJournal,@@ -113,6 +131,14 @@ export interface HostPressureGate {readonly percent: number;}+/**+ * The storage block the capacity gate decides from. The whole resolved+ * block rather than a narrow slice because the gate is the forecast's+ * tiers: the cap decides WHETHER the gate exists, and the four thresholds+ * decide where warn and critical sit.+ */+export type CapacityGate = ResolvedBrainStorageConfig;+export interface EvaluateGatesOptions {readonly now: Date;/** Absent = the window gate is always open (neutral default). */@@ -126,22 +152,38 @@ export interface EvaluateGatesOptions {readonly pressure?: HostPressureGate;/** Injectable measurement; defaults to reading this host. */readonly readPressure?: () => HostPressureReading;+ /**+ * Absent = the capacity gate is not configured: it never fires and+ * never measures. {@link runMaintenance} fills it from the vault's+ * `storage:` block when the caller does not name one, and only when+ * that block configures `cap_bytes` - an unbounded vault has no gate.+ */+ readonly capacity?: CapacityGate;+ /** Injectable gate decision; defaults to a fresh on-disk census. */+ readonly readCapacity?: (+ vault: string,+ capacity: CapacityGate,+ now: Date,+ ) => CapacityGateDecision;}/**- * What the soft gates decided, and what the pressure gate saw while- * deciding it.+ * What the soft gates decided, and what the pressure and capacity gates+ * saw while deciding it.*- * The reading is returned alongside the verdict rather than folded into+ * A reading is returned alongside the verdict rather than folded into* it because "the host was too loaded" and "this host cannot say how* loaded it is" are different facts that the caller must journal on* different lines: one is a decision not to run, the other is a gate- * that did not evaluate while the lane ran anyway.+ * that did not evaluate while the lane ran anyway. The capacity decision+ * rides the same way - the tier it read, or that it could not read one.*/export interface MaintenanceGateDecision {readonly verdict: MaintenanceVerdict;/** Absent unless the host-pressure gate was reached and configured. */readonly pressure?: HostPressureReading;+ /** Absent unless the capacity gate was reached and configured. */+ readonly capacity?: CapacityGateDecision;}/** Cap for persisted per-task error strings (journal + results). */@@ -230,8 +272,8 @@ export interface RunMaintenanceOptions extends EvaluateGatesOptions {readonly holder: string;readonly tasks: ReadonlyArray<MaintenanceTask>;/**- * Bypass the window, busy and pressure gates and every streak refusal -- * never the lease.+ * Bypass the window, busy, pressure and capacity gates and every streak+ * refusal - never the lease.*/readonly force?: boolean;/**@@ -261,6 +303,52 @@ export interface MaintenanceTaskResult {export interface RunMaintenanceResult {readonly verdict: MaintenanceVerdict;readonly tasks: ReadonlyArray<MaintenanceTaskResult>;+ /**+ * The capacity report, when the run was skipped for storage capacity:+ * the tier the gate measured, the reading behind it, and the remedy an+ * operator (or an agent relaying to one) can act on. Both front doors+ * attach it verbatim, so the two surfaces cannot disagree about what a+ * skip says.+ */+ readonly capacity?: MaintenanceCapacityReport;+}++/**+ * What an operator does about a lane the capacity gate skipped. The+ * reclamation verbs are the remedy, then a re-run; raising the cap is+ * the operator's own decision and names the file only they should edit.+ */+export const MAINTENANCE_CAPACITY_REMEDY =+ "free space by running receipt compaction or retiring near-duplicate copies, " ++ "or raise storage.cap_bytes in Brain/_brain.yaml, then re-run";++/** The capacity block both lane front doors attach to a skipped run. */+export interface MaintenanceCapacityReport {+ readonly tier: StorageForecastTier;+ readonly remedy: string;+ /**+ * The reading behind the tier, mirroring the pressure reading's ride+ * through the journal: the percent of the cap the gate measured, and+ * the measured bytes and configured cap it is a percent of. Null when+ * the gate decision carried no reading - a skip is always a measured+ * one in production, so null names a decision supplied from outside+ * the gate, never a tier the gate guessed at.+ */+ readonly capacity_percent: number | null;+ readonly measured_bytes: number | null;+ readonly cap_bytes: number | null;+}++/** The report for the gate decision that skipped a run. */+export function capacityReport(decision: CapacityGateDecision): MaintenanceCapacityReport {+ const reading = decision.reading;+ return {+ tier: decision.tier,+ remedy: MAINTENANCE_CAPACITY_REMEDY,+ capacity_percent: reading?.percentUsed ?? null,+ measured_bytes: reading?.measuredBytes ?? null,+ cap_bytes: reading?.capBytes ?? null,+ };}/** True when `now` falls inside the local-time window (wrap-aware). */@@ -285,9 +373,11 @@ export function dailyWindowContains(now: Date, window: DailyWindow): boolean {** The gates are ordered cheapest-first and, more importantly,* least-surprising-first: window and busy decide exactly as they did- * before this gate existed, and the host is only probed once both have- * let the pass through. A closed earlier gate therefore leaves no- * pressure reading, which is honest — nothing was measured.+ * before the pressure gate existed, the host is only probed once both+ * have let the pass through, and the capacity gate - the most expensive,+ * a byte walk over the whole vault - is probed last. A closed earlier+ * gate therefore leaves no pressure reading and no capacity decision,+ * which is honest - nothing was measured.*/export function evaluateGates(vault: string, opts: EvaluateGatesOptions): MaintenanceGateDecision {if (opts.window !== undefined && !dailyWindowContains(opts.now, opts.window)) {@@ -301,17 +391,61 @@ export function evaluateGates(vault: string, opts: EvaluateGatesOptions): Mainteconst recent = listRecallTelemetry(vault, { since });if (recent.length >= busy.threshold) return { verdict: MAINTENANCE_VERDICT.skippedBusy };- if (opts.pressure === undefined) return { verdict: MAINTENANCE_VERDICT.run };- const pressure = (opts.readPressure ?? measureHostPressure)();- // Unmeasurable leaves the gate OPEN. Closing it would stop this vault- // being maintained at all wherever the metric is degenerate, and- // treating the degenerate reading as a number would report a loaded- // host as an idle one - which is the failure the reading exists to- // make impossible. The caller journals the reason.- if (pressure.state === HOST_PRESSURE.measured && pressure.percent >= opts.pressure.percent) {- return { verdict: MAINTENANCE_VERDICT.skippedPressure, pressure };+ let pressure: HostPressureReading | undefined;+ if (opts.pressure !== undefined) {+ pressure = (opts.readPressure ?? measureHostPressure)();+ // Unmeasurable leaves the gate OPEN. Closing it would stop this vault+ // being maintained at all wherever the metric is degenerate, and+ // treating the degenerate reading as a number would report a loaded+ // host as an idle one - which is the failure the reading exists to+ // make impossible. The caller journals the reason.+ if (pressure.state === HOST_PRESSURE.measured && pressure.percent >= opts.pressure.percent) {+ return { verdict: MAINTENANCE_VERDICT.skippedPressure, pressure };+ }+ }++ const capacity = readCapacityDecision(vault, opts);+ if (capacity?.decision === CAPACITY_GATE_DECISION.skip) {+ return {+ verdict: MAINTENANCE_VERDICT.skippedCapacity,+ ...(pressure !== undefined ? { pressure } : {}),+ capacity,+ };}- return { verdict: MAINTENANCE_VERDICT.run, pressure };+ return {+ verdict: MAINTENANCE_VERDICT.run,+ ...(pressure !== undefined ? { pressure } : {}),+ ...(capacity !== undefined ? { capacity } : {}),+ };+}++/**+ * The capacity gate's decision, or `undefined` when the gate is not+ * configured: an unconfigured gate never measures, exactly like an+ * unconfigured pressure gate.+ */+function readCapacityDecision(+ vault: string,+ opts: EvaluateGatesOptions,+): CapacityGateDecision | undefined {+ if (opts.capacity === undefined) return undefined;+ return (opts.readCapacity ?? evaluateStorageCapacityGate)(vault, opts.capacity, opts.now);+}++/** The gate decision from a fresh on-disk census, never the stored aggregate. */+function evaluateStorageCapacityGate(+ vault: string,+ capacity: CapacityGate,+ now: Date,+): CapacityGateDecision {+ return evaluateCapacityGate(vault, {+ now,+ capBytes: capacity.cap_bytes,+ warnPercent: capacity.warn_percent,+ criticalPercent: capacity.critical_percent,+ warnDays: capacity.warn_days,+ criticalDays: capacity.critical_days,+ });}/**@@ -325,18 +459,23 @@ export async function runMaintenance(opts: RunMaintenanceOptions,): Promise<RunMaintenanceResult> {const nowIso = opts.now.toISOString();- // Vault-side policy: the pressure threshold the caller did not name,- // and the streak limit. An unreadable `_brain.yaml` raises here rather- // than resolving to defaults - both knobs decide whether heavy work- // starts, so a silent revert would run a pass the operator had gated.+ // Vault-side policy: the pressure threshold the caller did not name, the+ // storage block the capacity gate decides from, and the streak limit. An+ // unreadable `_brain.yaml` raises here rather than resolving to defaults+ // - every one of these knobs decides whether heavy work starts or a+ // write is refused, so a silent revert would run a pass the operator had+ // gated.const policy = loadMaintenanceConfigSafe(vault);+ const storage = loadStorageConfigSafe(vault);if (opts.force !== true) {const gate: EvaluateGatesOptions = {now: opts.now,...(opts.window !== undefined ? { window: opts.window } : {}),...(opts.busy !== undefined ? { busy: opts.busy } : {}),...(opts.readPressure !== undefined ? { readPressure: opts.readPressure } : {}),+ ...(opts.readCapacity !== undefined ? { readCapacity: opts.readCapacity } : {}),...resolvePressureGate(opts.pressure, policy.host_pressure_percent),+ ...resolveCapacityGate(opts.capacity, storage),};const decision = evaluateGates(vault, gate);// Two lines, never one: a gate that could not evaluate is its own@@ -351,7 +490,44 @@ export async function runMaintenance(pressure_reason: decision.pressure.reason,});}+ // The capacity gate's own notice line, for the same reason: an+ // unmeasurable store ran, and the journal has to say its gate was+ // open because nothing could be read - not because the vault is+ // healthy.+ if (+ decision.capacity !== undefined &&+ decision.capacity.tier === STORAGE_FORECAST_TIER.unmeasurable+ ) {+ appendJournal(vault, {+ ts: nowIso,+ holder: opts.holder,+ verdict: MAINTENANCE_VERDICT.capacityUnmeasurable,+ ...(decision.capacity.reason !== undefined+ ? { capacity_reason: decision.capacity.reason }+ : {}),+ });+ }if (decision.verdict !== MAINTENANCE_VERDICT.run) {+ // One report serves every skip surface below: the journal row and+ // both front doors read the same numbers, so they cannot disagree+ // about what a skip measured.+ const capacity =+ decision.verdict === MAINTENANCE_VERDICT.skippedCapacity && decision.capacity !== undefined+ ? capacityReport(decision.capacity)+ : undefined;+ // Sample BEFORE the skip is recorded or returned: the forecast+ // series must keep accumulating exactly while producers are paused,+ // so a skipped run still appends its sample. The gate's own census+ // is what is recorded - the decision already walked the vault, and+ // sampling by re-measuring would tax every skipped run with a+ // second full walk. Fail-soft by contract; a sample that cannot be+ // written never turns a skip into a crash.+ if (decision.verdict === MAINTENANCE_VERDICT.skippedCapacity) {+ sampleStorageUsage(vault, {+ now: opts.now,+ ...(decision.capacity?.census !== undefined ? { usage: decision.capacity.census } : {}),+ });+ }appendJournal(vault, {ts: nowIso,holder: opts.holder,@@ -359,8 +535,31 @@ export async function runMaintenance(...(decision.pressure?.state === HOST_PRESSURE.measured? { pressure_percent: decision.pressure.percent }: {}),+ ...(capacity !== undefined+ ? {+ capacity_tier: capacity.tier,+ // The reading rides the row the way the pressure percent+ // does - written together, only when the gate took one.+ // All three figures come from one reading, so the guard+ // names each field it writes: the journal entry carries no+ // null reading, only an absent one.+ ...(capacity.measured_bytes !== null &&+ capacity.capacity_percent !== null &&+ capacity.cap_bytes !== null+ ? {+ capacity_percent: capacity.capacity_percent,+ capacity_measured_bytes: capacity.measured_bytes,+ capacity_cap_bytes: capacity.cap_bytes,+ }+ : {}),+ }+ : {}),});- return { verdict: decision.verdict, tasks: [] };+ return {+ verdict: decision.verdict,+ tasks: [],+ ...(capacity !== undefined ? { capacity } : {}),+ };}}@@ -454,6 +653,22 @@ function resolvePressureGate(return {};}+/**+ * The capacity gate to evaluate: the caller's, else the vault's, and the+ * vault's only when its storage block configures `cap_bytes`. No cap is+ * the documented default and means unbounded: no gate, no byte walk, no+ * refusal - an operator who never asked for a capacity gate never gets+ * one, on any tier.+ */+function resolveCapacityGate(+ named: CapacityGate | undefined,+ configured: ResolvedBrainStorageConfig,+): { capacity?: CapacityGate } {+ if (named !== undefined) return { capacity: named };+ if (configured.cap_bytes !== null) return { capacity: configured };+ return {};+}+/** The task-result and journal rows of a streak refusal, or `undefined` to run. */function refuseOnStreak(vault: string,diff --git a/src/core/brain/page-dedup.ts b/src/core/brain/page-dedup.tsindex cf968380..128355c0 100644--- a/src/core/brain/page-dedup.ts+++ b/src/core/brain/page-dedup.ts@@ -572,6 +572,59 @@ export function retargetWikilinks(});}+/**+ * True when a page OTHER than `excludeRel` carries `basename` under the+ * retarget root: the bare-basename link spelling `[[basename]]` then+ * resolves to more than one page, and a retarget keyed on the spelling+ * alone would repoint links that meant the other page - silent+ * mis-linking of pages the caller never considered.+ *+ * The population is the retarget's own - same default root, same scope+ * rules - so a page whose links the pass would never touch cannot+ * manufacture a collision, and a page whose links it WOULD touch cannot+ * hide from one. A caller's `readable` predicate is deliberately not+ * applied: a withheld page still exists on disk, and existence is what+ * makes the spelling ambiguous.+ */+export function basenameCollides(+ vault: string,+ basename: string,+ excludeRel: string,+ opts: Pick<RetargetWikilinksOptions, "root"> = {},+): boolean {+ const root = opts.root === undefined ? BRAIN_ROOT_REL : opts.root;+ const abs = root === "" ? vault : join(vault, root);+ if (!existsSync(abs)) return false;+ const rules = resolveVaultScope(vault).rules;+ for (const file of markdownFilesUnder(vault, abs, rules)) {+ if (file.rel === excludeRel) continue;+ const name = file.rel.split("/").at(-1) ?? "";+ if (name.replace(/\.md$/, "") === basename) return true;+ }+ return false;+}++/**+ * A wikilink rewrite that gave up partway: `rewritten` files were+ * repointed before the write that refused.+ *+ * Carrying the count on the error is what lets the merge's fail-soft+ * caller report how far the pass got - the rewrite half landed, so a+ * bare message would leave the reader unable to tell how much of the+ * repoint the vault actually took.+ */+export class PatchWikilinksError extends Error {+ /** Files the pass rewrote before the one it could not. */+ readonly rewritten: number;+ constructor(rewritten: number, path: string, reason: string) {+ super(+ `patchWikilinks: rewrote ${rewritten} file(s), then could not write ` + `${path}: ${reason}`,+ );+ this.name = "PatchWikilinksError";+ this.rewritten = rewritten;+ }+}+/*** Rewrite every `[[<oldTarget>]]` reference inside `<vault>/Brain/`* to `[[<newTarget>]]`. Returns the number of files touched.@@ -582,10 +635,11 @@ export function retargetWikilinks(* basename spelling - and widening the merge's call site would have made* it answer questions it does not ask.*- * Raises on a write failure rather than reporting it. The merge's caller- * counts files and has nowhere to put the identity of one that did not- * take, so for THIS spelling a failure has to be an error; the note-file- * lifecycle, which does have somewhere to put it, reads `failed`.+ * Raises a {@link PatchWikilinksError} on a write failure rather than+ * reporting it. The merge's caller counts files and has nowhere to put+ * the identity of one that did not take, so for THIS spelling a failure+ * has to be an error; the note-file lifecycle, which does have somewhere+ * to put it, reads `failed`.*/export function patchWikilinks(vault: string, oldTarget: string, newTarget: string): number {if (oldTarget === newTarget) {@@ -598,10 +652,7 @@ export function patchWikilinks(vault: string, oldTarget: string, newTarget: striconst report = retargetWikilinks(vault, [{ from: oldTarget, to: newTarget }]);const first = report.failed[0];if (first !== undefined) {- throw new Error(- `patchWikilinks: rewrote ${report.rewritten.length} file(s), then could not write ` +- `${first.path}: ${first.reason}`,- );+ throw new PatchWikilinksError(report.rewritten.length, first.path, first.reason);}return report.rewritten.length;}diff --git a/src/core/brain/policy.ts b/src/core/brain/policy.tsindex 43a2656e..3949c663 100644--- a/src/core/brain/policy.ts+++ b/src/core/brain/policy.ts@@ -62,6 +62,7 @@ export {loadNotesConfigSafe,loadSnapshotDerivedStorePolicySafe,loadSnapshotRetentionSafe,+ loadStorageConfigSafe,loadTemporalConfigSafe,type BrainDerivedStorePolicy,type LoadBrainConfigResult,@@ -102,6 +103,20 @@ export {LESSONS_LIMIT_MAX,LESSONS_LIMIT_MIN,} from "./policy/blocks/lessons.ts";+export {+ BRAIN_STORAGE_DEFAULTS,+ STORAGE_CAP_BYTES_MIN,+ STORAGE_DAYS_MIN,+ STORAGE_PERCENT_MAX,+ STORAGE_PERCENT_MIN,+ resolveStorage,+} from "./policy/blocks/storage.ts";+export {+ STORAGE_CRITICAL_DAYS_DEFAULT,+ STORAGE_CRITICAL_PERCENT_DEFAULT,+ STORAGE_WARN_DAYS_DEFAULT,+ STORAGE_WARN_PERCENT_DEFAULT,+} from "./storage/forecast.ts";export {BRAIN_MAINTENANCE_DEFAULTS,MAINTENANCE_FAILURE_STREAK_LIMIT_DEFAULT,diff --git a/src/core/brain/policy/blocks/storage.ts b/src/core/brain/policy/blocks/storage.tsnew file mode 100644index 00000000..717c00d5--- /dev/null+++ b/src/core/brain/policy/blocks/storage.ts@@ -0,0 +1,234 @@+/**+ * The `storage:` block - the operator-configured capacity policy the+ * storage lifecycle reads: the byte cap, the two percent thresholds and+ * the two day horizons.+ *+ * One reason to change: what an operator may bound about their vault's+ * storage. Absent `cap_bytes` is the DEFAULT and means unbounded: no cap+ * by default means nothing pauses and nothing refuses, exactly as an+ * unconfigured maintenance window leaves the lane always open. A cap+ * only ever comes from an explicit operator decision recorded in this+ * file - the one file no tool call can rewrite.+ *+ * Out-of-range values are hard errors, following the policy stated in+ * `../field-checks.ts`: a knob that silently clamped would be+ * indistinguishable from the operator never having set it, and for a+ * gate that refuses writes the operator would believe a bound is in+ * force that is not - or believe it is not, while it is.+ *+ * The four thresholds default to the forecast's module constants+ * (`../../storage/forecast.ts`), imported rather than restated, so the+ * resolver and the forecast cannot disagree about what a default is.+ */++import type { BrainConfig, BrainStorageConfig, ResolvedBrainStorageConfig } from "../../types.ts";+import {+ STORAGE_CRITICAL_DAYS_DEFAULT,+ STORAGE_CRITICAL_PERCENT_DEFAULT,+ STORAGE_WARN_DAYS_DEFAULT,+ STORAGE_WARN_PERCENT_DEFAULT,+} from "../../storage/forecast.ts";+import { readBoundedInt } from "../field-checks.ts";+import { BrainConfigError } from "../errors.ts";+import { openBlock, warnUnknownKeys, type BlockParseContext } from "../key-index.ts";++const BLOCK = "storage";++/**+ * A cap must be a positive whole number of bytes. Zero would put every+ * vault past its cap the moment the block is written - not a cap but a+ * way to refuse everything, and the way to do that is a cap the operator+ * chose on purpose.+ */+export const STORAGE_CAP_BYTES_MIN = 1;++/** Percents are bounded to the open-at-zero, closed-at-100 range. */+export const STORAGE_PERCENT_MIN = 1;+export const STORAGE_PERCENT_MAX = 100;++/** Day horizons are positive whole days. */+export const STORAGE_DAYS_MIN = 1;++const KNOWN_KEYS = [+ "cap_bytes",+ "warn_percent",+ "critical_percent",+ "warn_days",+ "critical_days",+] as const;++/**+ * Defaults for an absent block. `cap_bytes` is `null` rather than a+ * number: the capacity gate has no neutral default, because any number+ * would start refusing writes on vaults whose operator never asked for a+ * gate - the same shape as the unconfigured maintenance pressure gate.+ */+export const BRAIN_STORAGE_DEFAULTS: ResolvedBrainStorageConfig = Object.freeze({+ cap_bytes: null,+ warn_percent: STORAGE_WARN_PERCENT_DEFAULT,+ critical_percent: STORAGE_CRITICAL_PERCENT_DEFAULT,+ warn_days: STORAGE_WARN_DAYS_DEFAULT,+ critical_days: STORAGE_CRITICAL_DAYS_DEFAULT,+}) as ResolvedBrainStorageConfig;++/** Merge a parsed `storage` block (or `undefined`) with the defaults. */+export function resolveStorage(cfg: BrainConfig): ResolvedBrainStorageConfig {+ const s = cfg.storage;+ if (s === undefined) return BRAIN_STORAGE_DEFAULTS;+ return {+ cap_bytes: s.cap_bytes ?? BRAIN_STORAGE_DEFAULTS.cap_bytes,+ warn_percent: s.warn_percent ?? BRAIN_STORAGE_DEFAULTS.warn_percent,+ critical_percent: s.critical_percent ?? BRAIN_STORAGE_DEFAULTS.critical_percent,+ warn_days: s.warn_days ?? BRAIN_STORAGE_DEFAULTS.warn_days,+ critical_days: s.critical_days ?? BRAIN_STORAGE_DEFAULTS.critical_days,+ };+}++/**+ * Shape:+ * storage:+ * cap_bytes: 1073741824 # positive integer bytes; absent = unbounded+ * warn_percent: 80 # integer in [1, 100]; must be < critical_percent+ * critical_percent: 95 # integer in [1, 100]; must be > warn_percent+ * warn_days: 30 # positive integer; must be > critical_days+ * critical_days: 7 # positive integer; must be < warn_days+ */+export function parseStorageBlock(ctx: BlockParseContext): BrainStorageConfig | undefined {+ const map = openBlock(ctx, BLOCK);+ if (map === undefined) return undefined;++ const capBytes = readCapBytes(map, ctx);+ const warnPercent = readBoundedInt(+ map,+ "warn_percent",+ STORAGE_PERCENT_MIN,+ STORAGE_PERCENT_MAX,+ "storage.warn_percent",+ ctx.source,+ );+ const criticalPercent = readBoundedInt(+ map,+ "critical_percent",+ STORAGE_PERCENT_MIN,+ STORAGE_PERCENT_MAX,+ "storage.critical_percent",+ ctx.source,+ );+ assertCriticalAboveWarn(warnPercent, criticalPercent, ctx);+ const warnDays = readBoundedInt(+ map,+ "warn_days",+ STORAGE_DAYS_MIN,+ Number.MAX_SAFE_INTEGER,+ "storage.warn_days",+ ctx.source,+ );+ const criticalDays = readBoundedInt(+ map,+ "critical_days",+ STORAGE_DAYS_MIN,+ Number.MAX_SAFE_INTEGER,+ "storage.critical_days",+ ctx.source,+ );+ assertCriticalNearerThanWarn(warnDays, criticalDays, ctx);++ warnUnknownKeys(ctx, map, KNOWN_KEYS, BLOCK);+ return {+ ...(capBytes !== undefined ? { cap_bytes: capBytes } : {}),+ ...(warnPercent !== undefined ? { warn_percent: warnPercent } : {}),+ ...(criticalPercent !== undefined ? { critical_percent: criticalPercent } : {}),+ ...(warnDays !== undefined ? { warn_days: warnDays } : {}),+ ...(criticalDays !== undefined ? { critical_days: criticalDays } : {}),+ };+}++/**+ * Read `cap_bytes`, also accepting an explicit `null` as "no cap" - the+ * template shows the key commented out, and an operator who uncomments+ * it with `null` is spelling the default intent, not a typo.+ */+function readCapBytes(+ map: Readonly<Record<string, unknown>>,+ ctx: BlockParseContext,+): number | undefined {+ if (!("cap_bytes" in map)) return undefined;+ const v = map["cap_bytes"];+ if (v === null) return undefined;+ if (typeof v !== "number" || !Number.isInteger(v) || v < STORAGE_CAP_BYTES_MIN) {+ throw new BrainConfigError(+ `must be a positive integer number of bytes; got ${JSON.stringify(v)}`,+ "storage.cap_bytes",+ ctx.source,+ );+ }+ return v;+}++/**+ * The critical percent band must begin ABOVE the warn band. Compared at+ * parse time, pair-wise over whichever of the two are present, each+ * against the default the other would resolve to - so `warn_percent: 95`+ * alone errors exactly as `critical_percent: 95` spelled beside an+ * 80-percent warn would. The error names the key(s) actually at fault:+ * the lone set key whose value broke the pair against the other's+ * default, or the pair when both were spelled and either could yield.+ */+function assertCriticalAboveWarn(+ warnPercent: number | undefined,+ criticalPercent: number | undefined,+ ctx: BlockParseContext,+): void {+ const warn = warnPercent ?? BRAIN_STORAGE_DEFAULTS.warn_percent;+ const critical = criticalPercent ?? BRAIN_STORAGE_DEFAULTS.critical_percent;+ if (criticalPercent !== undefined || warnPercent !== undefined) {+ if (critical <= warn) {+ const [field, message] =+ warnPercent !== undefined && criticalPercent !== undefined+ ? [+ "storage.warn_percent, storage.critical_percent",+ `the critical band must begin above the warn band: critical_percent (${critical}) must be greater than warn_percent (${warn})`,+ ]+ : warnPercent !== undefined+ ? [+ "storage.warn_percent",+ `must be smaller than critical_percent (${critical}); got ${warn}`,+ ]+ : [+ "storage.critical_percent",+ `must be greater than warn_percent (${warn}); got ${critical}`,+ ];+ throw new BrainConfigError(message, field, ctx.source);+ }+ }+}++/**+ * The critical day horizon must be NEARER than the warn horizon, same+ * rule, and the error names the key(s) actually at fault the same way.+ */+function assertCriticalNearerThanWarn(+ warnDays: number | undefined,+ criticalDays: number | undefined,+ ctx: BlockParseContext,+): void {+ const warn = warnDays ?? BRAIN_STORAGE_DEFAULTS.warn_days;+ const critical = criticalDays ?? BRAIN_STORAGE_DEFAULTS.critical_days;+ if (criticalDays !== undefined || warnDays !== undefined) {+ if (critical >= warn) {+ const [field, message] =+ warnDays !== undefined && criticalDays !== undefined+ ? [+ "storage.warn_days, storage.critical_days",+ `the critical horizon must be nearer than the warn horizon: critical_days (${critical}) must be smaller than warn_days (${warn})`,+ ]+ : warnDays !== undefined+ ? ["storage.warn_days", `must be greater than critical_days (${critical}); got ${warn}`]+ : [+ "storage.critical_days",+ `must be smaller than warn_days (${warn}); got ${critical}`,+ ];+ throw new BrainConfigError(message, field, ctx.source);+ }+ }+}diff --git a/src/core/brain/policy/load.ts b/src/core/brain/policy/load.tsindex 2fc39f94..66417f42 100644--- a/src/core/brain/policy/load.ts+++ b/src/core/brain/policy/load.ts@@ -26,6 +26,7 @@ import { brainConfigPath } from "../paths.ts";import { BrainConfigError, type BrainConfigLoadWarning } from "./errors.ts";import { validateBrainConfigDetailed } from "./validate.ts";import { DEFAULT_BRAIN_CONFIG } from "./defaults.ts";+import { BRAIN_STORAGE_DEFAULTS, resolveStorage } from "./blocks/storage.ts";import { BRAIN_MOST_APPLIED_DEFAULTS, resolveMostApplied } from "./blocks/active.ts";import { BRAIN_NOTES_DEFAULTS, resolveNotes } from "./blocks/notes.ts";import { BRAIN_TEMPORAL_DEFAULTS, resolveTemporal } from "./blocks/temporal.ts";@@ -261,6 +262,23 @@ export const loadMaintenanceConfigSafe = makeAbsentTolerantLoader(BRAIN_MAINTENANCE_DEFAULTS,);+/**+ * Load + resolve the `storage:` block, falling back to+ * `BRAIN_STORAGE_DEFAULTS` when the config file is absent - which leaves+ * the vault unbounded, exactly as an operator who has never configured a+ * cap has never asked for the gate.+ *+ * An unreadable config raises, and that is the point here as it is for+ * the maintenance loader: both outcomes decide whether writes are+ * refused, so answering with the defaults would run - or refuse - work+ * the operator's own file gates differently, on a vault that otherwise+ * looks healthy.+ */+export const loadStorageConfigSafe = makeAbsentTolerantLoader(+ resolveStorage,+ BRAIN_STORAGE_DEFAULTS,+);+/*** Load the RAW `install:` block, or `undefined` when the config file is* absent or declares no block.diff --git a/src/core/brain/policy/validate.ts b/src/core/brain/policy/validate.tsindex a3b09c23..85e60b7d 100644--- a/src/core/brain/policy/validate.ts+++ b/src/core/brain/policy/validate.ts@@ -48,6 +48,7 @@ import {} from "./blocks/continuity-hygiene.ts";import { parseFeedbackBlock } from "./blocks/feedback.ts";import { parseDecisionModelBlock } from "./blocks/decision-model.ts";+import { parseStorageBlock } from "./blocks/storage.ts";export interface ValidateResult {readonly config: BrainConfig;@@ -126,6 +127,7 @@ export function validateBrainConfigDetailed(const recall = parseRecallBlock(ctx);const feedback = parseFeedbackBlock(ctx);const decisionModel = parseDecisionModelBlock(ctx);+ const storage = parseStorageBlock(ctx);warnUnknownTopLevelKeys(ctx);@@ -159,6 +161,7 @@ export function validateBrainConfigDetailed(...(recallInject !== undefined ? { recall_inject: recallInject } : {}),...(feedback !== undefined ? { feedback } : {}),...(decisionModel !== undefined ? { decision_model: decisionModel } : {}),+ ...(storage !== undefined ? { storage } : {}),};return { config, warnings: ctx.warnings, knownKeys: ctx.keys };diff --git a/src/core/brain/receipts/compact.ts b/src/core/brain/receipts/compact.tsnew file mode 100644index 00000000..f8673e02--- /dev/null+++ b/src/core/brain/receipts/compact.ts@@ -0,0 +1,653 @@+/**+ * Receipt fold engine (storage-lifecycle, Lane 3 task 3.2).+ *+ * The confirmed half of receipt compaction: rewrite explicitly named+ * receipt-ledger shards, replacing the records older than the age+ * threshold with one aggregate line per (kind, month) and keeping+ * everything else - protected classes, young records, aggregates and+ * torn lines - byte-verbatim, in file order. Blank lines are the one+ * exception: the rewrite reads and writes non-blank lines, so a blank+ * line in the shard is dropped rather than preserved.+ *+ * The gates, in order, before anything is written:+ *+ * 1. Shard refs resolve through the family's name grammar - a name+ * that is not a shard is a caller error, never a traversal.+ * 2. A fresh per-shard recount against the disk (never the caller's+ * scan numbers) feeds the count guard: an `expect` mismatch aborts+ * before any write, snapshot included.+ * 3. A confirmed run that will write asserts vault identity and+ * consults the permissions document through `refuseDocumentDeny`+ * on the ingest mutation lane - the same ladder+ * `deleteBySource` runs.+ * 4. The rewrites sit inside one `withDestructiveSnapshot` span: the+ * recovery point is on disk before the first shard is touched, and+ * a snapshot failure aborts the run.+ *+ * Per shard the engine is fail-soft: a shard that cannot be read, has+ * drifted since the recount, or would not strictly shrink is collected+ * into `failures` and the run still completes. Writes are atomic per+ * shard - `atomicWriteFileSync` with `expectBefore` compare-and-swap -+ * under the shard's own lock. The real guarantee is narrower than "the+ * append paths hold this lock": it excludes the continuity appenders+ * only. A metrics shard is guarded by the compare-and-swap alone - the+ * metrics append path writes bare `appendFileSync` - so a concurrent+ * metrics append landing inside the rewrite window is lost without+ * error, not refused as drift. Those are advisory samples, and the+ * forecast series that reads them tolerates the gap.+ *+ * Idempotency: a re-run recounts a compacted shard, finds nothing+ * foldable, and writes nothing - no rewrite, no snapshot, no audit+ * record. The audit is the additive `receipt_compaction` continuity+ * kind (one record per confirmed run that folded, never on a dry run),+ * carrying the honest per-shard outcomes including failures. An audit+ * record that cannot be appended after the shards are durable is a+ * named outcome on the result (`auditError`), never a post-write throw:+ * the run's work is done by then, and discarding it would leave the+ * caller unable to tell what landed.+ *+ * Aggregate lines are stamped `o2b.receipt-fold.v1` and carry no record+ * fields: the continuity read model drops a row without+ * `id`/`kind`/`createdAt` and the metrics reader skips a row without a+ * string `run_at`, so an aggregate is inert bytes to every reader.+ */++import { readFileSync } from "node:fs";++import { resolveAgentName } from "../../config.ts";+import { atomicWriteFileSync, isFileDrift } from "../../fs-atomic.ts";+import { appendContinuityRecord } from "../continuity/store.ts";+import { assertExpectedCount } from "../count-guard.ts";+import type { RecoverabilityVerdict } from "../gates/recoverability.ts";+import { acquireLockSync } from "../sync-lockfile.ts";+import { BRAIN_SNAPSHOT_REASON } from "../types.ts";+import { withDestructiveSnapshot } from "../snapshot-gate.ts";+import { assertVaultIdentityForWrite } from "../vault-identity.ts";+import { refuseDocumentDeny, type WriteSubject } from "../write-disposition.ts";+import { REVIEW_LANE } from "../write-gate.ts";+import {+ canonicalSecond,+ classifyContinuityRecord,+ classifyMetricRecord,+ isFoldAggregateLine,+ RECEIPT_SHARD_UNREADABLE,+ resolveReceiptShardPath,+} from "./scan.ts";+import {+ RECEIPT_COMPACTION_OPERATION,+ RECEIPT_FOLD_DEFAULT_AGE_DAYS,+ RECEIPT_FOLD_SCHEMA,+ RECEIPT_LEDGER_FAMILY,+ receiptShardRefId,+ type ReceiptLedgerFamily,+ type ReceiptShardRef,+} from "./types.ts";++const MS_PER_DAY = 86_400_000;++/**+ * The closed sentence a confirmed fold carries when its audit record+ * could not be appended: the shards are folded and durable, the ledger+ * holds no `receipt_compaction` record for the run, and the result says+ * so instead of a throw that would discard everything the run did.+ */+export const RECEIPT_AUDIT_APPEND_FAILED =+ "the receipt_compaction audit record could not be appended; the shards above are folded as reported";++/** One confirmed shard rewrite, or what stopped it. */+export interface ReceiptFoldShardOutcome {+ /** The `family:name` spelling of the shard. */+ readonly shard: string;+ readonly folded: number;+ /** Records, aggregates and torn lines kept verbatim. */+ readonly kept: number;+ /** Aggregate lines the shard now carries. */+ readonly aggregates: number;+ readonly bytesBefore: number;+ readonly bytesAfter: number;+}++/** One fold group: what the aggregate line preserves as metadata. */+export interface ReceiptFoldGroup {+ readonly shard: string;+ /** Continuity kind, or the metrics surface. */+ readonly kind: string;+ /** UTC month the folded records belong to. */+ readonly month: string;+ readonly count: number;+ readonly first: string;+ readonly last: string;+ /**+ * Byte length of the folded record lines, newline separators excluded,+ * preserved inside the aggregate line.+ */+ readonly bytes: number;+}++/** One fold group together with where its aggregate line is written. */+interface GroupEntry {+ /** Index into the shard's lines of the group's first folded record. */+ readonly firstIndex: number;+ /** The fold group, replaced in place as later records join it. */+ group: ReceiptFoldGroup;+}++/** One shard the confirmed run could not rewrite, with the reason. */+export interface ReceiptFoldFailure {+ readonly shard: string;+ readonly message: string;+}++export interface ReceiptCompactTotals {+ readonly shards: number;+ readonly folded: number;+ readonly kept: number;+ readonly aggregates: number;+ readonly bytesBefore: number;+ readonly bytesAfter: number;+}++export interface ReceiptCompactResult {+ readonly operation: typeof RECEIPT_COMPACTION_OPERATION;+ readonly dryRun: boolean;+ readonly olderThanDays: number;+ readonly cutoff: string;+ readonly shards: ReadonlyArray<ReceiptFoldShardOutcome>;+ readonly groups: ReadonlyArray<ReceiptFoldGroup>;+ readonly failures: ReadonlyArray<ReceiptFoldFailure>;+ readonly totals: ReceiptCompactTotals;+ /** Present on a confirmed run that took its recovery point. */+ readonly snapshotRunId?: string;+ readonly recoverability?: RecoverabilityVerdict;+ /** Present on a confirmed run that appended its audit record. */+ readonly auditRecordId?: string;+ /**+ * Present on a confirmed run whose audit record could not be appended:+ * the shards above are folded and the recovery point exists, but the+ * ledger carries no `receipt_compaction` record for this run. A closed+ * sentence - the raw condition names absolute host paths no payload+ * carries; its detail goes to the operator's stderr only.+ */+ readonly auditError?: string;+}++export interface ReceiptCompactInput {+ /** Explicit shard selection, as a scan reported it. Required non-empty. */+ readonly shards: ReadonlyArray<ReceiptShardRef>;+ readonly olderThanDays?: number;+ readonly now?: Date;+ /** Preview with zero writes. Default true: preview-bound like a scan. */+ readonly dryRun?: boolean;+ /** Count guard: assert exactly this many records will fold. */+ readonly expect?: number | null;+ /** Count guard: refuse a guardless confirmed run. */+ readonly strict?: boolean;+ /** Credential subject for the permissions-document consult. */+ readonly subject?: WriteSubject;+ /** Config path the default subject's agent name resolves from. */+ readonly configPath?: string;+}++/** One shard's recount: the lines, their classification, and the fold groups. */+interface ShardPlan {+ readonly ref: ReceiptShardRef;+ readonly display: string;+ readonly path: string;+ /** Ledger base: the month for continuity, the surface for metrics. */+ readonly base: string;+ readonly lines: ReadonlyArray<string>;+ readonly cutoffMs: number;+ readonly foldable: number;+ readonly kept: number;+ /** Fold groups in first-occurrence order, each with its aggregate slot. */+ readonly groupEntries: ReadonlyArray<GroupEntry>;+ /** Exact text the recount read; the compare-and-swap anchor. */+ readonly text: string;+ readonly bytesBefore: number;+}++/** The line classification one pass over a shard's lines produces. */+interface LineVerdict {+ readonly foldable: boolean;+ /** Continuity kind, or the metrics surface; defined only when foldable. */+ readonly kind?: string;+ /** UTC month the record belongs to; defined only when foldable. */+ readonly month?: string;+ /** The record's own timestamp; defined only when foldable. */+ readonly timestamp?: string;+}++function verdictFor(+ family: ReceiptLedgerFamily,+ parsed: unknown,+ base: string,+ cutoffMs: number,+): LineVerdict {+ if (isFoldAggregateLine(parsed)) return { foldable: false };+ if (family === RECEIPT_LEDGER_FAMILY.continuity) {+ const record = parsed as {+ readonly kind?: unknown;+ readonly createdAt?: unknown;+ readonly payload?: unknown;+ };+ if (classifyContinuityRecord(record, cutoffMs) !== "foldable") return { foldable: false };+ // The month a continuity aggregate carries is the shard's own base.+ return {+ foldable: true,+ kind: record.kind as string,+ month: base,+ timestamp: record.createdAt as string,+ };+ }+ const record = parsed as {+ readonly surface?: unknown;+ readonly run_at?: unknown;+ };+ if (classifyMetricRecord(record, cutoffMs) !== "foldable") return { foldable: false };+ const timestamp = record.run_at as string;+ return {+ foldable: true,+ kind: record.surface as string,+ month: timestamp.slice(0, 7),+ timestamp,+ };+}++/**+ * The instant a foldable record's stamp stands for. Both classifiers+ * stamp a foldable verdict only under a grammar-validated UTC timestamp+ * (`isCanonicalUtcTimestamp`, `ISO_UTC_RE`), so the parse is finite by+ * construction; the instant, not the text, is what the aggregate's+ * first/last bounds order by.+ */+function instantOf(timestamp: string): number {+ return Date.parse(timestamp);+}++/**+ * Recount one named shard against the disk. A shard that cannot be read+ * yields a failure instead of a plan - named unreadable, with its+ * `family:name` identity and no filesystem detail, since the raw error+ * text would carry an absolute host path and a missing-versus-denied+ * oracle onto every surface the failure payload reaches. Every other+ * condition classifies into the plan.+ */+function planShard(+ vault: string,+ ref: ReceiptShardRef,+ cutoffMs: number,+): { readonly plan: ShardPlan } | { readonly failure: ReceiptFoldFailure } {+ const display = receiptShardRefId(ref);+ const resolved = resolveReceiptShardPath(vault, ref);+ let text: string;+ try {+ text = readFileSync(resolved.path, "utf8");+ } catch {+ return { failure: { shard: display, message: RECEIPT_SHARD_UNREADABLE } };+ }+ const lines = text.split("\n").filter((line) => line.trim() !== "");+ const byKey = new Map<string, GroupEntry>();+ const groupEntries: GroupEntry[] = [];+ let foldable = 0;+ let kept = 0;+ for (const [index, line] of lines.entries()) {+ let parsed: unknown;+ try {+ parsed = JSON.parse(line);+ } catch {+ kept += 1; // a torn line is nobody's to fold+ continue;+ }+ const verdict = verdictFor(ref.family, parsed, resolved.base, cutoffMs);+ if (!verdict.foldable) {+ kept += 1;+ continue;+ }+ foldable += 1;+ const key = `${verdict.kind}\u0000${verdict.month}`;+ const bytes = Buffer.byteLength(line, "utf8");+ const existing = byKey.get(key);+ if (existing === undefined) {+ const entry: GroupEntry = {+ firstIndex: index,+ group: {+ shard: display,+ kind: verdict.kind!,+ month: verdict.month!,+ count: 1,+ first: verdict.timestamp!,+ last: verdict.timestamp!,+ bytes,+ },+ };+ byKey.set(key, entry);+ groupEntries.push(entry);+ continue;+ }+ const prior = existing.group;+ const timestamp = verdict.timestamp!;+ existing.group = {+ ...prior,+ count: prior.count + 1,+ // The bounds compare by instant, never by stamp text: the canonical+ // grammars admit optional fractional seconds, and "." sorts before+ // "Z", so the LATER half-second stamp is the textually smaller one.+ // A folded record's own stamp is kept verbatim on whichever bound+ // its instant wins.+ first: instantOf(timestamp) < instantOf(prior.first) ? timestamp : prior.first,+ last: instantOf(timestamp) > instantOf(prior.last) ? timestamp : prior.last,+ bytes: prior.bytes + bytes,+ };+ }+ return {+ plan: {+ ref,+ display,+ path: resolved.path,+ base: resolved.base,+ lines,+ cutoffMs,+ foldable,+ kept,+ groupEntries: Object.freeze(groupEntries),+ text,+ bytesBefore: Buffer.byteLength(text, "utf8"),+ },+ };+}++/** The aggregate line one fold group folds into, with the fixed field order. */+function aggregateLine(family: ReceiptLedgerFamily, group: ReceiptFoldGroup): string {+ const fold: Record<string, unknown> =+ family === RECEIPT_LEDGER_FAMILY.continuity+ ? {+ family,+ kind: group.kind,+ month: group.month,+ count: group.count,+ first_created_at: group.first,+ last_created_at: group.last,+ bytes: group.bytes,+ }+ : {+ family,+ surface: group.kind,+ month: group.month,+ count: group.count,+ first_run_at: group.first,+ last_run_at: group.last,+ bytes: group.bytes,+ };+ return JSON.stringify({ schema: RECEIPT_FOLD_SCHEMA, fold });+}++/**+ * Rebuild one shard's content: kept lines verbatim in order, each fold+ * group's aggregate line at the position of the group's first folded+ * record. Deterministic for a given shard and cutoff.+ */+function rebuildContent(plan: ShardPlan): string {+ const byIndex = new Map<number, GroupEntry>(plan.groupEntries.map((e) => [e.firstIndex, e]));+ const emitted = new Set<string>();+ const out: string[] = [];+ for (const [index, line] of plan.lines.entries()) {+ const entry = byIndex.get(index);+ if (entry !== undefined) {+ const key = `${entry.group.kind}\u0000${entry.group.month}`;+ if (!emitted.has(key)) {+ emitted.add(key);+ out.push(aggregateLine(plan.ref.family, entry.group));+ }+ continue;+ }+ let parsed: unknown;+ try {+ parsed = JSON.parse(line);+ } catch {+ out.push(line); // torn lines stay+ continue;+ }+ const verdict = verdictFor(plan.ref.family, parsed, plan.base, plan.cutoffMs);+ if (verdict.foldable) continue; // represented by the aggregate above+ out.push(line);+ }+ return out.length === 0 ? "" : `${out.join("\n")}\n`;+}++/**+ * Fold the old records of the named receipt-ledger shards. See the+ * module header for the gate order and the failure contract.+ */+export function compactReceiptLedgers(+ vault: string,+ input: ReceiptCompactInput,+): ReceiptCompactResult {+ if (!Array.isArray(input.shards) || input.shards.length === 0) {+ throw new Error(+ "compactReceiptLedgers requires explicit shard ids from a scan; " ++ "a bare compaction never folds every shard blindly",+ );+ }+ const olderThanDays = input.olderThanDays ?? RECEIPT_FOLD_DEFAULT_AGE_DAYS;+ const nowMs = (input.now ?? new Date()).getTime();+ const cutoffMs = nowMs - olderThanDays * MS_PER_DAY;+ const cutoff = canonicalSecond(cutoffMs);+ const dryRun = input.dryRun !== false;++ // Gates 1+2: resolve every ref, recount against the disk, collect+ // per-shard plans and failures. Nothing has been written yet.+ const plans: ShardPlan[] = [];+ const failures: ReceiptFoldFailure[] = [];+ for (const ref of input.shards) {+ const result = planShard(vault, ref, cutoffMs);+ if ("failure" in result) failures.push(result.failure);+ else plans.push(result.plan);+ }+ const matched = plans.reduce((total, plan) => total + plan.foldable, 0);++ // Gate 3: the count guard, before any write including the snapshot.+ assertExpectedCount({+ matched,+ expect: input.expect ?? null,+ strict: input.strict === true,+ willMutate: !dryRun,+ matchList: plans.map((plan) => plan.display),+ });++ const base: Pick<ReceiptCompactResult, "operation" | "olderThanDays" | "cutoff"> = {+ operation: RECEIPT_COMPACTION_OPERATION,+ olderThanDays,+ cutoff,+ };++ // A dry run - and a confirmed run with nothing left foldable - writes+ // nothing: no rewrite, no snapshot, no audit record.+ if (dryRun || matched === 0) {+ const outcomes = plans.map(outcomeFromPlan);+ return Object.freeze({+ ...base,+ dryRun,+ shards: Object.freeze(outcomes),+ groups: Object.freeze(plans.flatMap((plan) => plan.groupEntries.map((e) => e.group))),+ failures: Object.freeze(failures),+ totals: totalsOf(outcomes, failures),+ });+ }++ // Gate 4: a confirmed run that will write. Identity, then the+ // permissions document, then the recovery point.+ assertVaultIdentityForWrite(vault);+ refuseDocumentDeny(+ vault,+ REVIEW_LANE.ingest,+ input.subject ?? { agent: resolveAgentName(input.configPath), via: "config" },+ );++ const gated = withDestructiveSnapshot(+ vault,+ BRAIN_SNAPSHOT_REASON.receiptCompaction,+ (snapshot) => ({ runId: snapshot.runId, ...rewriteShards(plans, failures) }),+ { now: input.now },+ );+ const outcomes = gated.result.outcomes;+ const foldedShards = new Set(outcomes.map((outcome) => outcome.shard));++ // One audit record per confirmed run that folded something, with the+ // honest outcomes - failures included - so the ledger says what the+ // run did even when a shard refused. A record that cannot be appended+ // is a NAMED outcome on the result, never a post-write throw: the+ // shards above are already folded and durable, and a throw here would+ // discard every outcome, failure and the recovery point with them.+ // The wire carries the closed sentence (the raw condition names+ // absolute host paths no payload repeats); the detail goes to the+ // operator's stderr only.+ let auditRecordId: string | undefined;+ let auditError: string | undefined;+ try {+ auditRecordId = appendContinuityRecord(vault, {+ kind: "receipt_compaction",+ createdAt: canonicalSecond(nowMs),+ payload: {+ operation: RECEIPT_COMPACTION_OPERATION,+ snapshot_run_id: gated.result.runId,+ older_than_days: olderThanDays,+ cutoff,+ shards: plans.map((plan) => plan.display),+ folded: sumOutcomes(outcomes, (o) => o.folded),+ kept: sumOutcomes(outcomes, (o) => o.kept),+ aggregates: sumOutcomes(outcomes, (o) => o.aggregates),+ bytes_before: sumOutcomes(outcomes, (o) => o.bytesBefore),+ bytes_after: sumOutcomes(outcomes, (o) => o.bytesAfter),+ failures: failures.map((failure) => ({ ...failure })),+ },+ }).id;+ } catch (err) {+ auditError = RECEIPT_AUDIT_APPEND_FAILED;+ process.stderr.write(+ `receipt compaction: ${RECEIPT_AUDIT_APPEND_FAILED} ` ++ `(${err instanceof Error ? err.message : String(err)})\n`,+ );+ }++ return Object.freeze({+ ...base,+ dryRun: false,+ shards: Object.freeze(outcomes),+ groups: Object.freeze(+ plans+ .filter((plan) => foldedShards.has(plan.display))+ .flatMap((plan) => plan.groupEntries.map((e) => e.group)),+ ),+ failures: Object.freeze(failures),+ totals: totalsOf(outcomes, failures),+ snapshotRunId: gated.result.runId,+ recoverability: gated.recoverability,+ ...(auditRecordId !== undefined ? { auditRecordId } : {}),+ ...(auditError !== undefined ? { auditError } : {}),+ });+}++/** Rewrite every planned shard, fail-soft per shard, under one snapshot. */+function rewriteShards(+ plans: ReadonlyArray<ShardPlan>,+ failures: ReceiptFoldFailure[],+): { readonly outcomes: ReadonlyArray<ReceiptFoldShardOutcome> } {+ const outcomes: ReceiptFoldShardOutcome[] = [];+ for (const plan of plans) {+ try {+ outcomes.push(rewriteShard(plan));+ } catch (err) {+ failures.push({+ shard: plan.display,+ message: err instanceof Error ? err.message : String(err),+ });+ }+ }+ return { outcomes: Object.freeze(outcomes) };+}++/**+ * Rewrite one shard: rebuild under the shard's lock, verify the fold+ * strictly shrinks the file, and swap atomically on the recount's bytes.+ * A drifted shard, a non-shrinking fold and an unwritable target are+ * throws the caller collects - never partial writes. Every throw names+ * the shard and the condition: the lock and compare-and-swap failures+ * are re-stated rather than forwarded, because the raw filesystem text+ * under them carries absolute host paths no surface should repeat.+ *+ * A shard the recount found nothing foldable in is answered without+ * touching the file: its rebuild is byte-identical at best - and one+ * byte longer when the file lacks a trailing newline - so the+ * strict-shrink guard would fail exactly the no-op the re-run contract+ * promises a mixed selection's already-compacted shards. Nothing-to-do+ * is an outcome, not a failure.+ */+function rewriteShard(plan: ShardPlan): ReceiptFoldShardOutcome {+ if (plan.foldable === 0) return outcomeFromPlan(plan);+ let handle: ReturnType<typeof acquireLockSync>;+ try {+ handle = acquireLockSync(plan.path);+ } catch {+ throw new Error(`${plan.display} could not be locked for the rewrite`);+ }+ try {+ const rebuilt = rebuildContent(plan);+ const bytesAfter = Buffer.byteLength(rebuilt, "utf8");+ if (bytesAfter >= plan.bytesBefore) {+ throw new Error(+ `folding ${plan.display} would not shrink it ` ++ `(${plan.bytesBefore} -> ${bytesAfter} bytes); refusing to rewrite without a size win`,+ );+ }+ try {+ atomicWriteFileSync(plan.path, rebuilt, { expectBefore: plan.text });+ } catch (err) {+ if (isFileDrift(err)) {+ throw new Error(`${plan.display} changed since it was recounted; re-scan and retry`);+ }+ throw new Error(`${plan.display} could not be rewritten`);+ }+ return Object.freeze({+ shard: plan.display,+ folded: plan.foldable,+ kept: plan.kept,+ aggregates: plan.groupEntries.length,+ bytesBefore: plan.bytesBefore,+ bytesAfter,+ });+ } finally {+ handle.release();+ }+}++function outcomeFromPlan(plan: ShardPlan): ReceiptFoldShardOutcome {+ return Object.freeze({+ shard: plan.display,+ folded: plan.foldable,+ kept: plan.kept,+ aggregates: plan.groupEntries.length,+ bytesBefore: plan.bytesBefore,+ bytesAfter: plan.bytesBefore, // nothing was written+ });+}++function totalsOf(+ outcomes: ReadonlyArray<ReceiptFoldShardOutcome>,+ failures: ReadonlyArray<ReceiptFoldFailure>,+): ReceiptCompactTotals {+ return Object.freeze({+ shards: outcomes.length + failures.length,+ folded: sumOutcomes(outcomes, (o) => o.folded),+ kept: sumOutcomes(outcomes, (o) => o.kept),+ aggregates: sumOutcomes(outcomes, (o) => o.aggregates),+ bytesBefore: sumOutcomes(outcomes, (o) => o.bytesBefore),+ bytesAfter: sumOutcomes(outcomes, (o) => o.bytesAfter),+ });+}++function sumOutcomes(+ outcomes: ReadonlyArray<ReceiptFoldShardOutcome>,+ pick: (outcome: ReceiptFoldShardOutcome) => number,+): number {+ return outcomes.reduce((total, outcome) => total + pick(outcome), 0);+}diff --git a/src/core/brain/receipts/scan.ts b/src/core/brain/receipts/scan.tsnew file mode 100644index 00000000..e4fe5298--- /dev/null+++ b/src/core/brain/receipts/scan.ts@@ -0,0 +1,368 @@+/**+ * Receipt-ledger scan (storage-lifecycle, Lane 3 task 3.1).+ *+ * The preview half of receipt compaction: inventory every shard of the+ * two append-only receipt ledgers - the continuity log and the metrics+ * sink - and count, per shard, the records old enough to fold, the+ * records in a protected class the fold must never touch, and the lines+ * that are neither. The scan is a PURE READ: it writes nothing, so a+ * scan can be handed to an operator (and to the count guard) as the+ * exact statement of what a confirmed fold would touch.+ *+ * Protected classes, never foldable whatever their age:+ *+ * - `source_invalidation` - the deleteBySource audit trail;+ * - `pre_compact_extract` / `post_compact_audit` - the Hermes+ * context-compaction receipts, which a storage fold must not+ * rewrite under a colliding name;+ * - any record whose payload carries one of+ * {@link CLIP_PROTECTED_PAYLOAD_KEYS} - folding would destroy the+ * session/agent correlation the clip rule guarantees survives;+ * - any continuity kind this build does not know - a newer build's+ * records are not this build's to fold (fail closed to keep);+ * - any record whose timestamp cannot be compared (fail closed).+ *+ * Ledger rewrites happen only in `compact.ts`; this module never binds a+ * write primitive.+ */++import { readFileSync, statSync } from "node:fs";+import { join, relative } from "node:path";++import { continuityLogDir, isCanonicalUtcTimestamp } from "../continuity/store.ts";+import { CLIP_PROTECTED_PAYLOAD_KEYS, isContinuityRecordKind } from "../continuity/types.ts";+import {+ JSONL_LEDGER_EXT,+ listShardedFiles,+ parseShardedName,+ type LedgerShardGrammar,+} from "../ledger-shards.ts";+import { metricsDir } from "../metrics.ts";+import { toPosix } from "../../path-safety.ts";+import {+ PROTECTED_CONTINUITY_KINDS,+ RECEIPT_FOLD_DEFAULT_AGE_DAYS,+ RECEIPT_FOLD_SCHEMA,+ RECEIPT_LEDGER_FAMILIES,+ RECEIPT_LEDGER_FAMILY,+ receiptShardRefId,+ type ReceiptLedgerFamily,+ type ReceiptScanError,+ type ReceiptScanReport,+ type ReceiptShardRef,+ type ReceiptShardScan,+} from "./types.ts";++/** File-name layout of the continuity ledger: one month per shard. */+const CONTINUITY_GRAMMAR: LedgerShardGrammar = Object.freeze({+ base: "\\d{4}-\\d{2}",+ extensions: Object.freeze([JSONL_LEDGER_EXT]),+});++/**+ * File-name layout of the metrics ledger: one surface per shard family.+ * The base shape is the documented metric-surface contract+ * (`[a-z][a-z0-9_]{0,63}`), spelled here because the metrics module+ * keeps its own grammars private.+ */+const METRICS_GRAMMAR: LedgerShardGrammar = Object.freeze({+ base: "[a-z][a-z0-9_]{0,63}",+ extensions: Object.freeze([JSONL_LEDGER_EXT]),+});++const MS_PER_DAY = 86_400_000;++/**+ * The message a shard read failure carries to every surface, whatever+ * `errno` the read failed with. The raw Node error text names absolute+ * host paths and distinguishes a missing shard from a denied one - a+ * map home and an absent/withheld oracle no caller should hold - so the+ * boundary reports the one class an operator can act on: the shard is+ * unreadable, and its identity travels beside the verdict, never the+ * filesystem's account of why.+ */+export const RECEIPT_SHARD_UNREADABLE = "unreadable";++/** ISO-8601 UTC timestamp shape, the one the metrics ledger writes. */+const ISO_UTC_RE = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,3})?Z$/;++/** How a ledger line classifies against the fold age and the protected classes. */+export type ReceiptLineClass = "foldable" | "protected" | "young";++/** True when `payload` carries any clip-protected correlation key. */+export function payloadCarriesProtectedKey(payload: unknown): boolean {+ if (payload === null || typeof payload !== "object" || Array.isArray(payload)) return false;+ return CLIP_PROTECTED_PAYLOAD_KEYS.some((key) => Object.hasOwn(payload, key));+}++/**+ * Classify one continuity record against the cutoff. Unknown kinds,+ * protected kinds, protected-key carriers and uncompared timestamps are+ * protected (fail closed); canonical timestamps at or after the cutoff+ * are young; everything else is foldable. Every field is optional: a+ * line missing one classifies through the same guards and fails closed.+ */+export function classifyContinuityRecord(+ record: { readonly kind?: unknown; readonly createdAt?: unknown; readonly payload?: unknown },+ cutoffMs: number,+): ReceiptLineClass {+ if (!isContinuityRecordKind(record.kind)) return "protected";+ if ((PROTECTED_CONTINUITY_KINDS as ReadonlyArray<string>).includes(record.kind)) {+ return "protected";+ }+ if (payloadCarriesProtectedKey(record.payload)) return "protected";+ if (typeof record.createdAt !== "string" || !isCanonicalUtcTimestamp(record.createdAt)) {+ return "protected";+ }+ return Date.parse(record.createdAt) >= cutoffMs ? "young" : "foldable";+}++/**+ * The record shape the metrics classification folds: the schema and+ * surface stamps as strings, a `run_at` string to derive the aggregate's+ * month from, and a payload that is a plain object. One predicate serves+ * both classifiers - the scan counts a parsed line outside this shape as+ * unparseable, and {@link classifyMetricRecord} fails closed to keep on+ * it - so the preview and the fold engine can never disagree about which+ * lines a confirmed fold would touch.+ */+export function metricRecordShaped(record: {+ readonly schema?: unknown;+ readonly surface?: unknown;+ readonly run_at?: unknown;+ readonly payload?: unknown;+}): record is {+ readonly schema: string;+ readonly surface: string;+ readonly run_at: string;+ readonly payload: Record<string, unknown>;+} {+ return (+ typeof record.schema === "string" &&+ typeof record.surface === "string" &&+ typeof record.run_at === "string" &&+ record.payload !== null &&+ typeof record.payload === "object" &&+ !Array.isArray(record.payload)+ );+}++/**+ * Classify one metrics record against the cutoff. A line outside the+ * {@link metricRecordShaped} record shape never folds - its aggregate+ * would carry no surface - and fails closed to keep, the same verdict the+ * scan reports it under. The same protected-key rule applies - a metrics+ * payload can correlate to a session too - and a `run_at` that is not an+ * ISO-8601 UTC stamp never folds: its month cannot be derived for the+ * aggregate, so it fails closed to keep.+ */+export function classifyMetricRecord(+ record: {+ readonly schema?: unknown;+ readonly surface?: unknown;+ readonly run_at?: unknown;+ readonly payload?: unknown;+ },+ cutoffMs: number,+): ReceiptLineClass {+ if (!metricRecordShaped(record)) return "protected";+ if (payloadCarriesProtectedKey(record.payload)) return "protected";+ if (!ISO_UTC_RE.test(record.run_at)) return "protected";+ return Date.parse(record.run_at) >= cutoffMs ? "young" : "foldable";+}++/** True when a parsed line is a fold aggregate stamped by this schema. */+export function isFoldAggregateLine(parsed: unknown): boolean {+ if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) return false;+ const record = parsed as Record<string, unknown>;+ return (+ record["schema"] === RECEIPT_FOLD_SCHEMA &&+ typeof record["fold"] === "object" &&+ record["fold"] !== null+ );+}++/** The ledger directory a family's shards live in. */+export function receiptLedgerDir(vault: string, family: ReceiptLedgerFamily): string {+ return family === RECEIPT_LEDGER_FAMILY.continuity ? continuityLogDir(vault) : metricsDir(vault);+}++/** The name grammar a family's shards follow. */+function familyGrammar(family: ReceiptLedgerFamily): LedgerShardGrammar {+ return family === RECEIPT_LEDGER_FAMILY.continuity ? CONTINUITY_GRAMMAR : METRICS_GRAMMAR;+}++/**+ * Resolve one shard ref to its file through the family's name grammar.+ * A name the grammar does not recognise - a typo, a path traversal, a+ * sync-conflict copy - is a named caller error before anything is read+ * or written.+ */+export function resolveReceiptShardPath(+ vault: string,+ ref: ReceiptShardRef,+): { readonly path: string; readonly base: string; readonly shardId: string } {+ const parsed = parseShardedName(ref.name, familyGrammar(ref.family));+ if (parsed === null) {+ throw new Error(`'${ref.name}' is not a ${ref.family} shard name`);+ }+ return {+ path: join(receiptLedgerDir(vault, ref.family), ref.name),+ base: parsed.base,+ shardId: parsed.shardId,+ };+}++/** Canonical UTC second-precision stamp, the shape every ledger compares. */+export function canonicalSecond(ms: number): string {+ return new Date(ms).toISOString().slice(0, 19) + "Z";+}++/** Parse one shard's non-blank lines into raw line texts. */+function readRawLines(text: string): ReadonlyArray<string> {+ return text.split("\n").filter((line) => line.trim() !== "");+}++/** One shard's counts against the cutoff. */+function scanShard(+ vault: string,+ family: ReceiptLedgerFamily,+ shard: {+ readonly name: string;+ readonly path: string;+ readonly base: string;+ readonly shardId: string;+ },+ cutoffMs: number,+): { row: ReceiptShardScan } | { error: ReceiptScanError } {+ const ref = receiptShardRefId({ family, name: shard.name });+ let text: string;+ let bytes: number;+ try {+ text = readShardText(shard.path);+ bytes = statSync(shard.path).size;+ } catch {+ return {+ error: {+ shard: ref,+ path: toPosix(relative(vault, shard.path)),+ message: RECEIPT_SHARD_UNREADABLE,+ },+ };+ }+ let records = 0;+ let foldable = 0;+ let protectedRecords = 0;+ let aggregates = 0;+ let unparseable = 0;+ for (const line of readRawLines(text)) {+ let parsed: unknown;+ try {+ parsed = JSON.parse(line);+ } catch {+ unparseable += 1;+ continue;+ }+ if (isFoldAggregateLine(parsed)) {+ aggregates += 1;+ continue;+ }+ if (family === RECEIPT_LEDGER_FAMILY.continuity) {+ const record = parsed as {+ readonly kind?: unknown;+ readonly createdAt?: unknown;+ readonly payload?: unknown;+ };+ if (typeof record.kind !== "string" || typeof record.createdAt !== "string") {+ unparseable += 1;+ continue;+ }+ records += 1;+ const classification = classifyContinuityRecord(record, cutoffMs);+ if (classification === "foldable") foldable += 1;+ else if (classification === "protected") protectedRecords += 1;+ continue;+ }+ const record = parsed as {+ readonly schema?: unknown;+ readonly surface?: unknown;+ readonly run_at?: unknown;+ readonly payload?: unknown;+ };+ if (!metricRecordShaped(record)) {+ unparseable += 1;+ continue;+ }+ records += 1;+ const classification = classifyMetricRecord(record, cutoffMs);+ if (classification === "foldable") foldable += 1;+ else if (classification === "protected") protectedRecords += 1;+ }+ return {+ row: Object.freeze({+ family,+ name: shard.name,+ path: toPosix(relative(vault, shard.path)),+ base: shard.base,+ shardId: shard.shardId,+ bytes,+ records,+ foldable,+ protectedRecords,+ aggregates,+ unparseable,+ }),+ };+}++function readShardText(path: string): string {+ return readFileSync(path, "utf8");+}++/**+ * Inventory every shard of both receipt ledgers. Read-only, fail-soft+ * per shard: a shard that cannot be read is an error row naming it, and+ * the rest of the inventory is still returned.+ */+export function scanReceiptLedgers(+ vault: string,+ opts: { readonly olderThanDays?: number; readonly now?: Date } = {},+): ReceiptScanReport {+ const olderThanDays = opts.olderThanDays ?? RECEIPT_FOLD_DEFAULT_AGE_DAYS;+ const nowMs = (opts.now ?? new Date()).getTime();+ const cutoffMs = nowMs - olderThanDays * MS_PER_DAY;+ const cutoff = canonicalSecond(cutoffMs);+ const shards: ReceiptShardScan[] = [];+ const errors: ReceiptScanError[] = [];+ for (const family of RECEIPT_LEDGER_FAMILIES) {+ for (const shard of listShardedFiles(receiptLedgerDir(vault, family), familyGrammar(family))) {+ const result = scanShard(vault, family, shard, cutoffMs);+ if ("error" in result) errors.push(result.error);+ else shards.push(result.row);+ }+ }+ const totals = Object.freeze({+ shards: shards.length,+ records: sum(shards, (s) => s.records),+ bytes: sum(shards, (s) => s.bytes),+ foldable: sum(shards, (s) => s.foldable),+ protectedRecords: sum(shards, (s) => s.protectedRecords),+ aggregates: sum(shards, (s) => s.aggregates),+ unparseable: sum(shards, (s) => s.unparseable),+ });+ return Object.freeze({+ generated_at: canonicalSecond(nowMs),+ olderThanDays,+ cutoff,+ shards: Object.freeze(shards),+ totals,+ errors: Object.freeze(errors),+ });+}++function sum(+ rows: ReadonlyArray<ReceiptShardScan>,+ pick: (row: ReceiptShardScan) => number,+): number {+ return rows.reduce((total, row) => total + pick(row), 0);+}diff --git a/src/core/brain/receipts/types.ts b/src/core/brain/receipts/types.tsnew file mode 100644index 00000000..07944097--- /dev/null+++ b/src/core/brain/receipts/types.ts@@ -0,0 +1,201 @@+/**+ * Receipt-ledger vocabulary (storage-lifecycle, Lane 3).+ *+ * "Receipt compaction" is the operator-applied fold of the two+ * append-only receipt ledgers - the continuity log+ * (`Brain/log/continuity/`) and the metrics sink (`Brain/metrics/`) -+ * into per-kind-month aggregate lines. Every identifier here belongs to+ * THAT vocabulary and is deliberately disjoint from the Hermes+ * context-compaction one (`pre_compact_extract`, `post_compact_audit`,+ * `brain_pre_compact_extract`): a tool name, a continuity kind or a+ * doctor code that could be read as either would misdescribe a+ * destructive storage operation as a context operation, so the two never+ * share a surface name.+ *+ * The ledgers compaction targets are the record ledgers only. The+ * decision-change receipts (`Brain/truth/decision-change*.jsonl`) are+ * append-only accountability by their own contract and are not+ * inventoried, folded or rewritten by anything in this lane; the source+ * archive (`Brain/.archives/`) keeps its no-TTL evidence contract for+ * the same reason.+ */++import type { ContinuityRecordKind } from "../continuity/types.ts";++/** The two receipt-ledger families the scan inventories and the fold rewrites. */+export const RECEIPT_LEDGER_FAMILY = Object.freeze({+ continuity: "continuity",+ metrics: "metrics",+} as const);++/** Closed union over {@link RECEIPT_LEDGER_FAMILY}. */+export type ReceiptLedgerFamily =+ (typeof RECEIPT_LEDGER_FAMILY)[keyof typeof RECEIPT_LEDGER_FAMILY];++/** Every family, in report order. */+export const RECEIPT_LEDGER_FAMILIES: ReadonlyArray<ReceiptLedgerFamily> = Object.freeze([+ RECEIPT_LEDGER_FAMILY.continuity,+ RECEIPT_LEDGER_FAMILY.metrics,+]);++/** Narrow an unvalidated family id from a caller or the wire. */+export function isReceiptLedgerFamily(value: unknown): value is ReceiptLedgerFamily {+ return (+ typeof value === "string" && (RECEIPT_LEDGER_FAMILIES as ReadonlyArray<string>).includes(value)+ );+}++/**+ * The operation label every receipt-compaction surface reports. One+ * spelling, shared by the CLI verb, the MCP tool and the continuity+ * audit payload, so the storage operation names itself identically+ * everywhere and never under a context-compaction name.+ */+export const RECEIPT_COMPACTION_OPERATION = "receipt compaction";++/**+ * The two surface modes, shared by the CLI verb and the MCP tool so the+ * tool's input schema and the verb's usage line cannot disagree. `scan`+ * is the default and writes nothing; `compact` previews unless the+ * caller applies it, and never runs without explicit shard ids.+ */+export const RECEIPT_MODE = Object.freeze({+ scan: "scan",+ compact: "compact",+} as const);++/** Closed union over {@link RECEIPT_MODE}. */+export type ReceiptMode = (typeof RECEIPT_MODE)[keyof typeof RECEIPT_MODE];++/** Every mode, in surface order. */+export const RECEIPT_MODES: ReadonlyArray<ReceiptMode> = Object.freeze([+ RECEIPT_MODE.scan,+ RECEIPT_MODE.compact,+]);++/** Narrow an unvalidated mode from a CLI flag or a tool argument. */+export function isReceiptMode(value: unknown): value is ReceiptMode {+ return typeof value === "string" && (RECEIPT_MODES as ReadonlyArray<string>).includes(value);+}++/**+ * Parse one `family:name` shard reference as the surfaces spell it. The+ * same parser the shard-name grammar enforces, so a flag that names no+ * real shard is a usage error on both surfaces, before anything runs.+ */+export function parseReceiptShardRef(value: string): ReceiptShardRef {+ const separator = value.indexOf(":");+ if (separator <= 0 || separator === value.length - 1) {+ throw new Error(`'${value}' is not a family:name shard reference`);+ }+ const family = value.slice(0, separator);+ const name = value.slice(separator + 1);+ if (!isReceiptLedgerFamily(family)) {+ throw new Error(+ `'${value}' names unknown ledger family '${family}'; expected one of: ` ++ RECEIPT_LEDGER_FAMILIES.join(", "),+ );+ }+ return Object.freeze({ family, name });+}++/**+ * The schema stamp on a fold aggregate line. A line carrying this stamp+ * is not a record: it is the per-(kind, month) summary the fold engine+ * replaced a run of expired records with. Both ledger readers tolerate+ * it by construction - the continuity read model drops a row without+ * `id`/`kind`/`createdAt` and the metrics reader skips a row without a+ * string `run_at` - so aggregates are inert bytes to every consumer.+ */+export const RECEIPT_FOLD_SCHEMA = "o2b.receipt-fold.v1";++/**+ * How old a record must be before it is foldable, when the caller names+ * no threshold. Thirty days: long enough that every live correlation+ * window has closed, short enough that a year of telemetry does not+ * accumulate first.+ */+export const RECEIPT_FOLD_DEFAULT_AGE_DAYS = 30;++/**+ * The continuity kinds the fold must NEVER touch, whatever their age.+ * `source_invalidation` is the deleteBySource audit trail; the two+ * Hermes context-compaction kinds are that feature's own receipts and+ * folding them under a storage operation's name would corrupt the one+ * vocabulary by colliding with the other.+ */+export const PROTECTED_CONTINUITY_KINDS: ReadonlyArray<ContinuityRecordKind> = Object.freeze([+ "source_invalidation",+ "pre_compact_extract",+ "post_compact_audit",+]);++/** One shard selected for compaction, as a scan reported it. */+export interface ReceiptShardRef {+ readonly family: ReceiptLedgerFamily;+ /** Shard file name within its ledger directory, as the scan spelled it. */+ readonly name: string;+}++/** The `family:name` spelling used in match lists, error rows and CLI flags. */+export function receiptShardRefId(ref: ReceiptShardRef): string {+ return `${ref.family}:${ref.name}`;+}++/** Per-shard inventory row of a receipt scan. */+export interface ReceiptShardScan {+ readonly family: ReceiptLedgerFamily;+ /** Shard file name within its ledger directory. */+ readonly name: string;+ /** Vault-relative POSIX path of the shard file. */+ readonly path: string;+ /** Ledger base: the UTC month for continuity, the surface for metrics. */+ readonly base: string;+ /** Empty string for the legacy un-sharded file. */+ readonly shardId: string;+ /** Shard file size in bytes. */+ readonly bytes: number;+ /** Parsed record lines: foldable + protected + young. */+ readonly records: number;+ /** Records older than the threshold and in no protected class. */+ readonly foldable: number;+ /** Records the fold must keep whatever their age (see the module header). */+ readonly protectedRecords: number;+ /** Aggregate lines already present from an earlier fold. */+ readonly aggregates: number;+ /** Lines that parsed as neither a record nor an aggregate. */+ readonly unparseable: number;+}++/** Roll-up of a scan over every shard it inventoried. */+export interface ReceiptScanTotals {+ readonly shards: number;+ readonly records: number;+ readonly bytes: number;+ readonly foldable: number;+ readonly protectedRecords: number;+ readonly aggregates: number;+ readonly unparseable: number;+}++/** One shard the scan could not read, collected instead of thrown. */+export interface ReceiptScanError {+ /** The `family:name` spelling of the shard. */+ readonly shard: string;+ /** Vault-relative POSIX path of the shard file. */+ readonly path: string;+ readonly message: string;+}++/** The scan result: the fold engine's preview and the doctor's input. */+export interface ReceiptScanReport {+ /** Canonical UTC instant the scan stamped, second precision. */+ readonly generated_at: string;+ /** The age threshold the scan applied. */+ readonly olderThanDays: number;+ /** Records at or after this instant are kept; before it, foldable. */+ readonly cutoff: string;+ readonly shards: ReadonlyArray<ReceiptShardScan>;+ readonly totals: ReceiptScanTotals;+ readonly errors: ReadonlyArray<ReceiptScanError>;+}diff --git a/src/core/brain/receipts/views.ts b/src/core/brain/receipts/views.tsnew file mode 100644index 00000000..acc09820--- /dev/null+++ b/src/core/brain/receipts/views.ts@@ -0,0 +1,91 @@+/**+ * Wire views over the receipt scan and fold results (storage-lifecycle,+ * Lane 3 task 3.4).+ *+ * ONE renderer per result, consumed by both surfaces - the+ * `o2b brain receipts` verb's `--json` output and the `brain_receipts`+ * tool's structured content - so the two cannot disagree on field+ * spellings (`dry_run`, `snapshot_run_id`, `protected_records`) or on+ * which fields exist. The core types stay camelCase for the engine; the+ * snake_case spellings live only here, at the boundary, exactly once.+ */++import type { ReceiptCompactResult } from "./compact.ts";+import { RECEIPT_COMPACTION_OPERATION } from "./types.ts";+import type { ReceiptScanReport } from "./types.ts";++/** The scan report as both surfaces render it. */+export function scanView(report: ReceiptScanReport): Record<string, unknown> {+ return {+ operation: RECEIPT_COMPACTION_OPERATION,+ generated_at: report.generated_at,+ older_than_days: report.olderThanDays,+ cutoff: report.cutoff,+ shards: report.shards.map((shard) => ({+ family: shard.family,+ name: shard.name,+ path: shard.path,+ base: shard.base,+ shard_id: shard.shardId,+ bytes: shard.bytes,+ records: shard.records,+ foldable: shard.foldable,+ protected_records: shard.protectedRecords,+ aggregates: shard.aggregates,+ unparseable: shard.unparseable,+ })),+ totals: {+ shards: report.totals.shards,+ records: report.totals.records,+ bytes: report.totals.bytes,+ foldable: report.totals.foldable,+ protected_records: report.totals.protectedRecords,+ aggregates: report.totals.aggregates,+ unparseable: report.totals.unparseable,+ },+ errors: report.errors.map(({ shard, path, message }) => ({ shard, path, message })),+ };+}++/** The fold result as both surfaces render it. */+export function compactView(result: ReceiptCompactResult): Record<string, unknown> {+ return {+ operation: result.operation,+ dry_run: result.dryRun,+ older_than_days: result.olderThanDays,+ cutoff: result.cutoff,+ shards: result.shards.map((shard) => ({+ shard: shard.shard,+ folded: shard.folded,+ kept: shard.kept,+ aggregates: shard.aggregates,+ bytes_before: shard.bytesBefore,+ bytes_after: shard.bytesAfter,+ })),+ groups: result.groups.map((group) => ({+ shard: group.shard,+ kind: group.kind,+ month: group.month,+ count: group.count,+ first: group.first,+ last: group.last,+ bytes: group.bytes,+ })),+ // Both rows are rendered field-by-field rather than passed through:+ // the wire holds the shard's identity and the sanitized verdict, and+ // a field the core later grows cannot ride along unreviewed.+ failures: result.failures.map(({ shard, message }) => ({ shard, message })),+ totals: {+ shards: result.totals.shards,+ folded: result.totals.folded,+ kept: result.totals.kept,+ aggregates: result.totals.aggregates,+ bytes_before: result.totals.bytesBefore,+ bytes_after: result.totals.bytesAfter,+ },+ ...(result.snapshotRunId !== undefined ? { snapshot_run_id: result.snapshotRunId } : {}),+ ...(result.recoverability !== undefined ? { recoverability: result.recoverability } : {}),+ ...(result.auditRecordId !== undefined ? { audit_record_id: result.auditRecordId } : {}),+ ...(result.auditError !== undefined ? { audit_error: result.auditError } : {}),+ };+}diff --git a/src/core/brain/source-cleanup.ts b/src/core/brain/source-cleanup.tsindex 1fc66343..08434a9c 100644--- a/src/core/brain/source-cleanup.ts+++ b/src/core/brain/source-cleanup.ts@@ -79,6 +79,9 @@ import { REVIEW_LANE } from "./write-gate.ts";/** How a page was found to trace back to the source. */export type SourceCleanupMatch = "source_path" | "session_ref" | "wikilink" | "evidenced_by";+/** The shared empty failure list a dry run (and every clean run) reports. */+const NO_FAILURES: ReadonlyArray<SourceCleanupFailure> = Object.freeze([]);+/** Coarse classification of the referencing page, by its home directory. */export type SourceCleanupKind =| "summary"@@ -132,6 +135,15 @@ export interface NoteDerivationEntry extends SourceCleanupEntry {readonly structuredProvenance: boolean;}+/**+ * One per-entry removal failure: the vault-relative path that stayed on disk+ * and the named fs error, so a retry knows exactly what is left.+ */+export interface SourceCleanupFailure {+ readonly path: string;+ readonly message: string;+}+export interface SourceCleanupPlan {/** Canonical form of the queried source. */readonly source: string;@@ -148,6 +160,14 @@ export interface SourceCleanupPlan {readonly manifestEntry: string | null;/** Paths actually removed this run (empty on a dry run). */readonly deleted: ReadonlyArray<string>;+ /**+ * Per-entry removal failures this run, with the named fs error (empty on a+ * dry run and on a clean run). One blocked file never aborts the cleanup:+ * every remaining entry is still attempted, the failures are reported here+ * with the failed paths left on disk, and the audit record counts both+ * sides honestly.+ */+ readonly failed: ReadonlyArray<SourceCleanupFailure>;readonly manifestEntryRemoved: boolean;/** Id of the audit continuity record, when one was written. */readonly auditRecordId: string | null;@@ -666,7 +686,14 @@ function findOriginals(return [canonical];}-function removeFile(vault: string, relPath: string): boolean {+/**+ * Remove one vault-relative file. `false` for an already-absent path (the+ * idempotent re-run), `true` when it was removed. Any other fs error throws:+ * callers that sweep many files collect per-file failures around this call+ * ({@link deleteBySource} does; the retire applier consumes it the same way),+ * never silently.+ */+export function removeFile(vault: string, relPath: string): boolean {let abs: string;try {abs = ensureInsideVault(join(vault, relPath), vault);@@ -749,6 +776,7 @@ export function deleteBySource(originals: Object.freeze(originals),manifestEntry: manifestKey,deleted: Object.freeze([]),+ failed: NO_FAILURES,manifestEntryRemoved: false,auditRecordId: null,snapshotRunId: null,@@ -764,21 +792,40 @@ export function deleteBySource(// Confirmed path — delete the derived entries first, then (opt-in) the// originals, then drop the manifest entry (an index artifact).const deleted: string[] = [];+ const failed: SourceCleanupFailure[] = [];let manifestEntryRemoved = false;+ /**+ * Attempt one removal, collecting a failure instead of throwing: one+ * blocked file must not abort the entries still queued behind it. The+ * collected failures ride the result and the audit, and the paths stay on+ * disk for a retry.+ */+ const removeCollecting = (relPath: string): void => {+ try {+ if (removeFile(vault, relPath)) deleted.push(relPath);+ } catch (err) {+ failed.push({ path: relPath, message: (err as Error)?.message ?? String(err) });+ }+ };+// Audit only a run that actually changed something; a no-op re-run writes// no record (idempotent). Factored so both the success path AND a partial// failure can persist the record of what was already removed.const writeInvalidationAudit = (): string | null => {if (deleted.length === 0 && !manifestEntryRemoved) return null;const agent = opts.agent?.trim() ? opts.agent.trim() : "delete_by_source";+ const reason =+ `${agent}: removed ${deleted.length} derived/original entr${deleted.length === 1 ? "y" : "ies"}` ++ `${manifestEntryRemoved ? " and the ingest manifest entry" : ""}` ++ `${includeOriginals ? " (originals included)" : ""}` ++ (failed.length > 0+ ? `; failed to remove ${failed.length}: ${failed.map((f) => f.path).join(", ")}`+ : "");const record = appendContinuitySourceInvalidation(vault, {createdAt: isoSecond(opts.now ?? new Date()),source: { id: canonical, path: canonical, kind: "source" },- reason:- `${agent}: removed ${deleted.length} derived/original entr${deleted.length === 1 ? "y" : "ies"}` +- `${manifestEntryRemoved ? " and the ingest manifest entry" : ""}` +- `${includeOriginals ? " (originals included)" : ""}`,+ reason,});return record.id;};@@ -790,13 +837,9 @@ export function deleteBySource(// gate so every confirmed cleanup has a pre-deletion recovery point.const runDeletion = (): void => {try {- for (const entry of derived) {- if (removeFile(vault, entry.path)) deleted.push(entry.path);- }+ for (const entry of derived) removeCollecting(entry.path);if (includeOriginals) {- for (const original of originals) {- if (removeFile(vault, original)) deleted.push(original);- }+ for (const original of originals) removeCollecting(original);}if (manifestKey !== null && existsSync(manifestPath(vault))) {@@ -815,10 +858,11 @@ export function deleteBySource(}}} catch (err) {- // A removeFile (or manifest write) that throws mid-cleanup would otherwise- // exit before the audit append, leaving already-deleted paths with no- // record for a retry to reconstruct. Persist what was removed so far, then- // rethrow; never let an audit-write error mask the original failure.+ // A manifest read/write that throws mid-cleanup would otherwise exit+ // before the audit append, leaving already-deleted paths with no record+ // for a retry to reconstruct. Persist what was removed (and what was+ // collected as failed) so far, then rethrow; never let an audit-write+ // error mask the original failure.try {writeInvalidationAudit();} catch {@@ -866,6 +910,7 @@ export function deleteBySource(originals: Object.freeze(originals),manifestEntry: manifestKey,deleted: Object.freeze(deleted),+ failed: Object.freeze(failed),manifestEntryRemoved,auditRecordId,snapshotRunId,diff --git a/src/core/brain/storage/capacity-error.ts b/src/core/brain/storage/capacity-error.tsnew file mode 100644index 00000000..f2c91d0f--- /dev/null+++ b/src/core/brain/storage/capacity-error.ts@@ -0,0 +1,45 @@+/**+ * The typed failure vocabulary of the storage lifecycle: the closed+ * code union and the {@link StorageCapacityError} carrier.+ *+ * A LEAF by construction - it imports nothing, exactly like+ * `SearchError` in `src/core/search/search-error.ts`, so every module+ * that raises or catches a capacity condition can depend on it without+ * dragging the storage modules (or the policy layer) into its import+ * graph.+ *+ * Two codes, and the difference is the whole vocabulary:+ *+ * - `capacity_unmeasurable` - the census could not be taken, so no+ * capacity question has an answer. Refusal by name is the honest+ * state: a gate that read an unmeasurable store as "plenty of room"+ * would be the silent no-op this project forbids.+ * - `capacity_exhausted` - the store measured past the critical+ * threshold, and the write that asked is refused before it lands.+ */++export const STORAGE_ERROR_CODES = ["capacity_unmeasurable", "capacity_exhausted"] as const;++export type StorageErrorCode = (typeof STORAGE_ERROR_CODES)[number];++export class StorageCapacityError extends Error {+ readonly code: StorageErrorCode;+ /**+ * The structured fields a refusal envelope carries beside the code,+ * when the thrower has facts to add (the tier reading behind an+ * admission refusal). Still a leaf: no imports, and the registered+ * classification reads the code alone, so these ride only where a+ * caller maps the leaf onto its envelope deliberately.+ */+ readonly details?: Readonly<Record<string, unknown>>;+ constructor(+ code: StorageErrorCode,+ message: string,+ details?: Readonly<Record<string, unknown>>,+ ) {+ super(message);+ this.name = "StorageCapacityError";+ this.code = code;+ this.details = details;+ }+}diff --git a/src/core/brain/storage/forecast.ts b/src/core/brain/storage/forecast.tsnew file mode 100644index 00000000..8211270b--- /dev/null+++ b/src/core/brain/storage/forecast.ts@@ -0,0 +1,378 @@+/**+ * The storage forecast and the usage sampler.+ *+ * The sampler is the write half: one `measureStorageUsage` census folded+ * into one `storage_usage` record in the metrics sink per run id. The+ * forecast is the read half: a linear trend over the stored samples plus+ * the freshest reading, projected against the configured cap.+ *+ * ## The two-truths rule+ *+ * The stored aggregate and a fresh census are different kinds of+ * evidence, and the forecast treats them differently on purpose. The+ * samples are advisory - a stale-counter lie is exactly what a+ * re-measured vault would produce if the forecast trusted the newest+ * recorded total - so samples alone never tier: fewer than two stored+ * samples is the named `no_forecast` state. A FRESH reading+ * (`latestUsage`, measured at decision time) is enforcement truth: hand+ * one in and the percent tiers compute from it even with an empty+ * history, which is what keeps a gate honest on a vault that filled up+ * before it ever sampled twice.+ *+ * ## Tier boundaries+ *+ * The boundaries are module-level constants and the comparison is+ * inclusive at each boundary: exactly 80 percent used, or exactly 30+ * days to the cap, is `warn`; exactly 95 percent, or exactly 7 days, is+ * `critical`. The defaults are consumed by the `storage:` policy block+ * (`../policy/blocks/storage.ts`), which overrides them with the same+ * names, so the template and the resolver cannot disagree about what a+ * default is.+ *+ * ## Refusals+ *+ * An absent cap is `unbounded`. A measurement that could not read every+ * surface is `unmeasurable` with the refused surfaces carried: a census+ * that missed a surface would understate usage by exactly those bytes,+ * so the forecast refuses instead of projecting from partial numbers -+ * the same measured-or-refused posture as the meter and `HOST_PRESSURE`.+ */++import { appendMetric, listMetrics, type MetricRecord } from "../metrics.ts";+import { measureStorageUsage, type StorageUnmeasuredSurface, type StorageUsage } from "./usage.ts";++/** The metric surface one usage sample rides. */+export const STORAGE_USAGE_METRIC_SURFACE = "storage_usage";++/**+ * The warn boundary, as percent of cap. Inclusive: exactly 80 warns.+ */+export const STORAGE_WARN_PERCENT_DEFAULT = 80;++/** The critical boundary, as percent of cap. Inclusive: exactly 95 is critical. */+export const STORAGE_CRITICAL_PERCENT_DEFAULT = 95;++/** The warn boundary in days to the cap, at the measured trend. Inclusive. */+export const STORAGE_WARN_DAYS_DEFAULT = 30;++/** The critical boundary in days to the cap, at the measured trend. Inclusive. */+export const STORAGE_CRITICAL_DAYS_DEFAULT = 7;++/** Stored samples needed before a linear trend exists. */+const MIN_TREND_SAMPLES = 2;++/** Milliseconds in one day; the unit the trend is reported in. */+const MS_PER_DAY = 86_400_000;++/** The closed forecast vocabulary. */+export const STORAGE_FORECAST_TIER = Object.freeze({+ ok: "ok",+ warn: "warn",+ critical: "critical",+ unbounded: "unbounded",+ unmeasurable: "unmeasurable",+ noForecast: "no_forecast",+} as const);++export type StorageForecastTier =+ (typeof STORAGE_FORECAST_TIER)[keyof typeof STORAGE_FORECAST_TIER];++/** The tiers that answer "how full", in increasing severity. */+export const STORAGE_MEASURED_TIERS: ReadonlyArray<StorageForecastTier> = Object.freeze([+ STORAGE_FORECAST_TIER.ok,+ STORAGE_FORECAST_TIER.warn,+ STORAGE_FORECAST_TIER.critical,+]);++/** Every tier, in declaration order - the measured half plus its refusals. */+export const STORAGE_FORECAST_TIERS: ReadonlyArray<StorageForecastTier> = Object.freeze([+ STORAGE_FORECAST_TIER.ok,+ STORAGE_FORECAST_TIER.warn,+ STORAGE_FORECAST_TIER.critical,+ STORAGE_FORECAST_TIER.unbounded,+ STORAGE_FORECAST_TIER.unmeasurable,+ STORAGE_FORECAST_TIER.noForecast,+]);++export function isStorageForecastTier(value: unknown): value is StorageForecastTier {+ return typeof value === "string" && Object.values(STORAGE_FORECAST_TIER).includes(value as never);+}++/** One forecast: a tier plus the numbers it rests on. */+export interface StorageForecast {+ readonly tier: StorageForecastTier;+ /**+ * The freshest trusted byte reading: `latestUsage` when one was handed+ * in, otherwise the newest sample. `null` when nothing was measured.+ */+ readonly measuredBytes: number | null;+ readonly capBytes: number | null;+ /** `measuredBytes / capBytes * 100`, or `null` with no cap or no reading. */+ readonly percentUsed: number | null;+ /** Bytes per day over the stored samples, or `null` with no trend. */+ readonly slopeBytesPerDay: number | null;+ /**+ * Days until the cap at the current trend, or `null` when there is no+ * trend or the trend does not reach the cap.+ */+ readonly daysToCap: number | null;+ /** The surfaces the freshest census could not read, when `unmeasurable`. */+ readonly unmeasured: ReadonlyArray<StorageUnmeasuredSurface>;+ /**+ * True when the stored history itself could not be read (the metrics+ * ledger is unreadable), so an absent trend means "could not look"+ * and not "nothing recorded". The tier still answers from a fresh+ * reading when there is one; without one the forecast stays+ * `no_forecast` rather than pretending the history was empty.+ */+ readonly samplesUnreadable: boolean;+}++export interface ForecastStorageUsageOptions {+ /** The configured cap, or `null`/absent for an unbounded vault. */+ readonly capBytes?: number | null;+ /**+ * A fresh census taken at decision time. Present, it is the reading+ * the percent tiers compute from; absent, the newest sample stands in+ * and fewer than two samples stays `no_forecast`.+ */+ readonly latestUsage?: StorageUsage;+ /** Threshold overrides; every one defaults to the module constant. */+ readonly warnPercent?: number;+ readonly criticalPercent?: number;+ readonly warnDays?: number;+ readonly criticalDays?: number;+}++/** One (time, bytes) point of the trend, in day units for exact boundaries. */+interface TrendPoint {+ readonly day: number;+ readonly bytes: number;+}++/** Parse the `storage_usage` samples into points; torn records drop out. */+function parseSamples(samples: ReadonlyArray<MetricRecord>): TrendPoint[] {+ const points: TrendPoint[] = [];+ for (const record of samples) {+ const t = Date.parse(record.run_at);+ const bytes = record.payload["total_bytes"];+ if (!Number.isFinite(t) || typeof bytes !== "number" || !Number.isFinite(bytes)) continue;+ points.push({ day: t / MS_PER_DAY, bytes });+ }+ return points.toSorted((a, b) => a.day - b.day);+}++/** Least-squares slope in bytes per day over the points. */+function linearSlope(points: ReadonlyArray<TrendPoint>): number | null {+ if (points.length < MIN_TREND_SAMPLES) return null;+ const n = points.length;+ const meanDay = points.reduce((s, p) => s + p.day, 0) / n;+ const meanBytes = points.reduce((s, p) => s + p.bytes, 0) / n;+ let cov = 0;+ let varDay = 0;+ for (const p of points) {+ const dx = p.day - meanDay;+ cov += dx * (p.bytes - meanBytes);+ varDay += dx * dx;+ }+ if (varDay === 0) return null;+ return cov / varDay;+}++/**+ * Forecast one vault's storage trajectory against its cap.+ *+ * Reads the vault's own `storage_usage` samples; hand `latestUsage` in+ * when the caller has just measured the vault and wants the decision on+ * fresh numbers (the capacity gate always does - see `./gate.ts`).+ */+export function forecastStorageUsage(+ vault: string,+ opts: ForecastStorageUsageOptions = {},+): StorageForecast {+ const capBytes = opts.capBytes ?? null;+ const warnPercent = opts.warnPercent ?? STORAGE_WARN_PERCENT_DEFAULT;+ const criticalPercent = opts.criticalPercent ?? STORAGE_CRITICAL_PERCENT_DEFAULT;+ const warnDays = opts.warnDays ?? STORAGE_WARN_DAYS_DEFAULT;+ const criticalDays = opts.criticalDays ?? STORAGE_CRITICAL_DAYS_DEFAULT;++ if (capBytes === null) {+ return {+ tier: STORAGE_FORECAST_TIER.unbounded,+ measuredBytes: opts.latestUsage?.totalBytes ?? null,+ capBytes: null,+ percentUsed: null,+ slopeBytesPerDay: null,+ daysToCap: null,+ unmeasured: [],+ samplesUnreadable: false,+ };+ }++ // The metrics module reads fail-soft for a missing ledger; a ledger+ // that exists but cannot be listed (a file in its place, a denial)+ // still throws out of that read, so the history read is guarded here+ // and the failure is NAMED on the result rather than flattened into+ // "no samples ever recorded".+ let samples: TrendPoint[];+ let samplesUnreadable = false;+ try {+ samples = parseSamples(listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE }));+ } catch {+ samples = [];+ samplesUnreadable = true;+ }+ const slope = linearSlope(samples);++ // The freshest trusted reading, and why it might be none.+ let measuredBytes: number | null;+ let unmeasured: ReadonlyArray<StorageUnmeasuredSurface>;+ if (opts.latestUsage !== undefined) {+ measuredBytes = opts.latestUsage.totalBytes;+ unmeasured = opts.latestUsage.unmeasured;+ } else {+ // The stored aggregate stands in only when it is all there is, and+ // a single stored point is never enough to tier on.+ if (samples.length < MIN_TREND_SAMPLES) {+ return {+ tier: STORAGE_FORECAST_TIER.noForecast,+ measuredBytes: null,+ capBytes,+ percentUsed: null,+ slopeBytesPerDay: null,+ daysToCap: null,+ unmeasured: [],+ samplesUnreadable,+ };+ }+ const newest = samples[samples.length - 1]!;+ measuredBytes = newest.bytes;+ unmeasured = [];+ }++ if (unmeasured.length > 0) {+ return {+ tier: STORAGE_FORECAST_TIER.unmeasurable,+ measuredBytes,+ capBytes,+ percentUsed: null,+ slopeBytesPerDay: slope,+ daysToCap: null,+ unmeasured,+ samplesUnreadable,+ };+ }++ const percentUsed = (measuredBytes / capBytes) * 100;+ const daysToCap = slope !== null && slope > 0 ? (capBytes - measuredBytes) / slope : null;++ let tier: StorageForecastTier;+ if (percentUsed >= criticalPercent || (daysToCap !== null && daysToCap <= criticalDays)) {+ tier = STORAGE_FORECAST_TIER.critical;+ } else if (percentUsed >= warnPercent || (daysToCap !== null && daysToCap <= warnDays)) {+ tier = STORAGE_FORECAST_TIER.warn;+ } else {+ tier = STORAGE_FORECAST_TIER.ok;+ }++ return {+ tier,+ measuredBytes,+ capBytes,+ percentUsed,+ slopeBytesPerDay: slope,+ daysToCap,+ unmeasured: [],+ samplesUnreadable,+ };+}++export interface SampleStorageUsageOptions {+ /**+ * The run id the sample is idempotent on: a retry that names the same+ * run id appends nothing. Defaults to a fresh unique id per call.+ */+ readonly runId?: string;+ /** Wall clock for the record's `run_at`; defaults to the current time. */+ readonly now?: Date;+ /**+ * A census already taken this invocation, recorded instead of taking a+ * second one. The capacity gate hands its fresh census here, so a+ * skipped lane run walks the vault once for the decision and no more;+ * absent, the sampler measures as it always did.+ */+ readonly usage?: StorageUsage;+}++export interface SampleStorageUsageResult {+ /** The census the sample records. */+ readonly usage: StorageUsage;+ readonly runId: string;+ /** False when the record was already present, or the append failed. */+ readonly appended: boolean;+ /** The named append failure, when `appended` is false for that reason. */+ readonly appendError?: string;+}++/** Mint a unique default run id; samples name no file, so a UUID suffices. */+function mintRunId(): string {+ return crypto.randomUUID();+}++/**+ * Take one usage census and append it to the `storage_usage` metrics+ * surface, one record per run id.+ *+ * Hand `usage` in when this invocation has already measured the vault and+ * the sample should record that census rather than a fresh one.+ *+ * Fail-soft by contract: a vault whose metrics directory cannot be+ * written still gets its census back, with the failure named - the+ * caller decides whether a report without its sample is fatal. Nothing+ * here throws.+ */+export function sampleStorageUsage(+ vault: string,+ opts: SampleStorageUsageOptions = {},+): SampleStorageUsageResult {+ const runId = opts.runId ?? mintRunId();+ const usage =+ opts.usage ?? measureStorageUsage(vault, opts.now !== undefined ? { now: opts.now } : {});+ let alreadyRecorded = false;+ try {+ alreadyRecorded = listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE }).some(+ (record) => record.payload["run_id"] === runId,+ );+ } catch {+ // The history read failed; the append below will fail too if the+ // ledger is truly unwritable, and it names that failure itself.+ alreadyRecorded = false;+ }+ if (alreadyRecorded) {+ return { usage, runId, appended: false };+ }+ try {+ appendMetric(vault, {+ surface: STORAGE_USAGE_METRIC_SURFACE,+ runAt: usage.measuredAt,+ payload: {+ run_id: runId,+ total_bytes: usage.totalBytes,+ bytes_by_tier: usage.bytesByTier,+ measured: usage.complete,+ unmeasured: usage.unmeasured,+ },+ });+ return { usage, runId, appended: true };+ } catch (err) {+ return {+ usage,+ runId,+ appended: false,+ appendError: err instanceof Error ? err.message : String(err),+ };+ }+}++/** Convenience re-export so consumers import the meter from one module. */+export { measureStorageUsage };diff --git a/src/core/brain/storage/gate.ts b/src/core/brain/storage/gate.tsnew file mode 100644index 00000000..d698aa04--- /dev/null+++ b/src/core/brain/storage/gate.ts@@ -0,0 +1,132 @@+/**+ * The capacity gate: one measured-or-refused decision about whether this+ * vault has room for more bytes.+ *+ * ## Fresh at decision time+ *+ * The decision RE-MEASURES the vault. The stored `storage_usage`+ * aggregate is advisory - a sample taken yesterday says nothing about a+ * vault an agent wrote into this morning - so a gate that decided from+ * it would enforce a stale-counter lie. The forecast supplies the trend;+ * the fresh census supplies the bytes the tiers compare against. A gate+ * that needed no fresh bytes is an unbounded one, and it does not+ * measure at all: no cap, no cost.+ *+ * ## Never skip on a refusal+ *+ * An unmeasurable store leaves the gate OPEN with the refused surfaces+ * named, exactly as the maintenance lane's host-pressure gate stays open+ * when the machine cannot be read: refusing to work forever on a vault+ * one directory cannot be listed in would be a worse failure than the+ * gate not closing. Only a MEASURED critical tier skips.+ */++import type { StorageForecastTier } from "./forecast.ts";+import { STORAGE_FORECAST_TIER, forecastStorageUsage } from "./forecast.ts";+import { measureStorageUsage, type StorageUsage } from "./usage.ts";++/** The closed decision vocabulary. */+export const CAPACITY_GATE_DECISION = Object.freeze({+ run: "run",+ skip: "skip",+} as const);++export type CapacityGateDecisionKind =+ (typeof CAPACITY_GATE_DECISION)[keyof typeof CAPACITY_GATE_DECISION];++/** The fresh measurement the decision rests on, when one was taken. */+export interface CapacityGateReading {+ readonly measuredBytes: number;+ readonly capBytes: number;+ readonly percentUsed: number | null;+ readonly daysToCap: number | null;+ readonly slopeBytesPerDay: number | null;+ /** ISO-8601 stamp of the fresh census, from its injected `now`. */+ readonly measuredAt: string;+}++/** One gate decision: whether to run, on which tier, on what reading. */+export interface CapacityGateDecision {+ readonly decision: CapacityGateDecisionKind;+ readonly tier: StorageForecastTier;+ /**+ * Why no number could be attached - the refused surface names, when+ * the tier is `unmeasurable`. Surface names only, never paths.+ */+ readonly reason?: string;+ /** The fresh census behind the decision; absent for unbounded and unmeasurable. */+ readonly reading?: CapacityGateReading;+ /**+ * The fresh census itself, when one was taken - present for+ * `unmeasurable` too, because a partial census is still a census.+ * The usage sampler records it instead of re-walking the vault (see+ * `sampleStorageUsage`), so a decision costs one walk and never two.+ */+ readonly census?: StorageUsage;+}++export interface EvaluateCapacityGateOptions {+ /** The configured cap in bytes, or `null`/absent for an unbounded vault. */+ readonly capBytes?: number | null;+ /** Threshold overrides; every one defaults to the forecast constant. */+ readonly warnPercent?: number;+ readonly criticalPercent?: number;+ readonly warnDays?: number;+ readonly criticalDays?: number;+ /** Wall clock for the fresh census's stamp; defaults to the current time. */+ readonly now?: Date;+}++/**+ * Decide whether `vault` has room, from a fresh census.+ *+ * Unbounded stores run without measuring. Everything else is measured+ * now; a census that could not read every surface runs with the refusal+ * named, and a measured critical tier is the only skip.+ */+export function evaluateCapacityGate(+ vault: string,+ opts: EvaluateCapacityGateOptions = {},+): CapacityGateDecision {+ const capBytes = opts.capBytes ?? null;+ if (capBytes === null) {+ return { decision: CAPACITY_GATE_DECISION.run, tier: STORAGE_FORECAST_TIER.unbounded };+ }++ const fresh = measureStorageUsage(vault, opts.now !== undefined ? { now: opts.now } : {});+ if (!fresh.complete) {+ return {+ decision: CAPACITY_GATE_DECISION.run,+ tier: STORAGE_FORECAST_TIER.unmeasurable,+ reason: `unmeasured surfaces: ${fresh.unmeasured.map((u) => u.surface).join(", ")}`,+ census: fresh,+ };+ }++ const forecast = forecastStorageUsage(vault, {+ capBytes,+ latestUsage: fresh,+ ...(opts.warnPercent !== undefined ? { warnPercent: opts.warnPercent } : {}),+ ...(opts.criticalPercent !== undefined ? { criticalPercent: opts.criticalPercent } : {}),+ ...(opts.warnDays !== undefined ? { warnDays: opts.warnDays } : {}),+ ...(opts.criticalDays !== undefined ? { criticalDays: opts.criticalDays } : {}),+ });++ const decision: CapacityGateDecision = {+ decision:+ forecast.tier === STORAGE_FORECAST_TIER.critical+ ? CAPACITY_GATE_DECISION.skip+ : CAPACITY_GATE_DECISION.run,+ tier: forecast.tier,+ reading: {+ measuredBytes: fresh.totalBytes,+ capBytes,+ percentUsed: forecast.percentUsed,+ daysToCap: forecast.daysToCap,+ slopeBytesPerDay: forecast.slopeBytesPerDay,+ measuredAt: fresh.measuredAt,+ },+ census: fresh,+ };+ return decision;+}diff --git a/src/core/brain/storage/usage.ts b/src/core/brain/storage/usage.tsnew file mode 100644index 00000000..2ce41532--- /dev/null+++ b/src/core/brain/storage/usage.ts@@ -0,0 +1,329 @@+/**+ * The storage usage meter: per-surface, measured-or-refused byte census+ * over a vault.+ *+ * Every byte this tool keeps inside a vault belongs to exactly one named+ * surface here, and every surface is reported one of two ways: a measured+ * byte total, or a named refusal in `unmeasured` - never a plausible+ * zero. A chmod-denied directory and an empty one produce different+ * answers, because "0 bytes" and "I could not look" are different facts,+ * and a capacity forecast built over the first would silently+ * understate usage by exactly the surface it could not read. The+ * measured-or-refused posture follows `HOST_PRESSURE`+ * (`../maintenance/host-pressure.ts`), applied to bytes instead of load.+ *+ * ## The per-dot-dir policy+ *+ * The index walker excludes dot-directories, and a usage meter that+ * reused that rule would miss most of what fills a vault. Each+ * dot-directory is therefore a NAMED surface with its own tier, decided+ * the way `STATE_SURFACES` decides it - recoverability, not location:+ *+ * - `.artifacts` - MCP tool-result scratch with a TTL; `derived`.+ * - `.snapshots` - recovery points; `vault-content`, because they are+ * the vault's own bytes copied and deleting them removes the only+ * rollback path.+ * - `.archives` - the content-addressed source evidence (no TTL);+ * `vault-content`.+ * - `.payloads` - externalized session content a restore must bring+ * back; `vault-content`.+ * - `.state` - internal machinery (the session-lineage ledger, the+ * write images, the anticipatory cache); `vault-content` as a whole,+ * because its regenerable minority (the anticipatory cache) does not+ * outweigh the records nothing can replay, and a capacity report+ * that overstated the deletable side would be the dangerous+ * direction to err in.+ *+ * Everything else under `Brain/` - pages, ledgers, the config file -+ * is the `brain_tree` surface, `vault-content`. The derived store+ * (`brain.sqlite` plus its WAL and SHM sidecars) is resolved through the+ * search layer's own resolver so the meter can never disagree with the+ * indexer about which file is the store; it is `derived`.+ *+ * The meter is read-only, never follows symlinks, and takes no+ * wall-clock reading: the timestamp it stamps comes from the injected+ * `now` alone.+ */++import { readdirSync, statSync } from "node:fs";+import { join } from "node:path";++import { BRAIN_ROOT_REL } from "../path-constants.ts";+import { resolveIndexPath } from "../../search/paths.ts";++/** Recoverability tiers, mirroring the state catalogue's vocabulary. */+export const STORAGE_TIER = Object.freeze({+ derived: "derived",+ vaultContent: "vault-content",+} as const);++export type StorageTier = (typeof STORAGE_TIER)[keyof typeof STORAGE_TIER];++/** The tiers, most-recoverable first, for report rendering. */+export const STORAGE_TIERS: ReadonlyArray<StorageTier> = Object.freeze([+ STORAGE_TIER.derived,+ STORAGE_TIER.vaultContent,+]);++/** The closed set of surfaces one measurement names. */+export const STORAGE_SURFACE = Object.freeze({+ /** Every `Brain/` byte outside the named dot-directories. */+ brainTree: "brain_tree",+ /** `Brain/.snapshots/` - recovery points. */+ snapshots: "snapshots",+ /** `Brain/.artifacts/` - TTL'd tool-result scratch. */+ artifacts: "artifacts",+ /** `Brain/.archives/` - content-addressed source evidence. */+ archives: "archives",+ /** `Brain/.payloads/` - externalized session content. */+ payloads: "payloads",+ /** `Brain/.state/` - internal machinery ledgers and write images. */+ internalState: "internal_state",+ /** The derived store and its WAL/SHM sidecars. */+ derivedStore: "derived_store",+} as const);++export type StorageSurface = (typeof STORAGE_SURFACE)[keyof typeof STORAGE_SURFACE];++/** Every surface, in canonical report order. */+export const STORAGE_SURFACES: ReadonlyArray<StorageSurface> = Object.freeze([+ STORAGE_SURFACE.brainTree,+ STORAGE_SURFACE.snapshots,+ STORAGE_SURFACE.artifacts,+ STORAGE_SURFACE.archives,+ STORAGE_SURFACE.payloads,+ STORAGE_SURFACE.internalState,+ STORAGE_SURFACE.derivedStore,+]);++/** Why a surface has no byte reading. */+export const STORAGE_UNMEASURED_REASON = Object.freeze({+ /** The surface exists and a read of it was denied. */+ surfaceUnreadable: "surface_unreadable",+} as const);++export type StorageUnmeasuredReason =+ (typeof STORAGE_UNMEASURED_REASON)[keyof typeof STORAGE_UNMEASURED_REASON];++export const STORAGE_UNMEASURED_REASONS: ReadonlyArray<StorageUnmeasuredReason> = Object.freeze(+ Object.values(STORAGE_UNMEASURED_REASON),+);++/** One surface's measured byte total. */+export interface StorageSurfaceReading {+ readonly surface: StorageSurface;+ readonly tier: StorageTier;+ readonly bytes: number;+}++/** One surface the meter could not read, named with its reason. */+export interface StorageUnmeasuredSurface {+ readonly surface: StorageSurface;+ readonly reason: StorageUnmeasuredReason;+}++/** One complete measured-or-refused census of a vault. */+export interface StorageUsage {+ /** ISO-8601 UTC stamp from the injected `now`. */+ readonly measuredAt: string;+ /** False when at least one surface was refused; see `unmeasured`. */+ readonly complete: boolean;+ /** The measured surfaces, in canonical order. Refused surfaces are absent. */+ readonly surfaces: ReadonlyArray<StorageSurfaceReading>;+ /** Byte totals per tier, over the measured surfaces only. */+ readonly bytesByTier: Readonly<Record<StorageTier, number>>;+ /** Sum of every measured byte. */+ readonly totalBytes: number;+ /** The refusals, in canonical surface order. */+ readonly unmeasured: ReadonlyArray<StorageUnmeasuredSurface>;+}++export interface MeasureStorageUsageOptions {+ /** Wall clock for the report stamp; defaults to the current time. */+ readonly now?: Date;+ /**+ * The derived store's file path, when the caller has already resolved+ * it (the doctor's context carries one). Defaults to the search+ * layer's own resolution for this vault.+ */+ readonly storePath?: string;+}++/** The tier each surface reports, named once beside the surface list. */+const SURFACE_TIERS: Readonly<Record<StorageSurface, StorageTier>> = Object.freeze({+ [STORAGE_SURFACE.brainTree]: STORAGE_TIER.vaultContent,+ [STORAGE_SURFACE.snapshots]: STORAGE_TIER.vaultContent,+ [STORAGE_SURFACE.artifacts]: STORAGE_TIER.derived,+ [STORAGE_SURFACE.archives]: STORAGE_TIER.vaultContent,+ [STORAGE_SURFACE.payloads]: STORAGE_TIER.vaultContent,+ [STORAGE_SURFACE.internalState]: STORAGE_TIER.vaultContent,+ [STORAGE_SURFACE.derivedStore]: STORAGE_TIER.derived,+});++/** The sqlite sidecar suffixes that belong to the store surface. */+const STORE_WAL_SUFFIX = "-wal";+const STORE_SHM_SUFFIX = "-shm";++/**+ * Whether an fs error names a missing path. Missing is a true answer+ * (the surface is measured-empty); every other error is a refusal.+ */+function isAbsent(err: unknown): boolean {+ const code = (err as NodeJS.ErrnoException).code;+ return code === "ENOENT";+}++/**+ * Byte-accumulate one regular file, or record the refusal. Symlinks are+ * skipped outright: `statSync` on one would follow it, and a usage meter+ * that followed links out of the vault would count bytes the vault does+ * not hold.+ */+function addFile(path: string, add: (bytes: number) => void, refuse: () => void): void {+ let info;+ try {+ info = statSync(path);+ } catch (err) {+ if (!isAbsent(err)) refuse();+ return;+ }+ if (!info.isFile()) return;+ add(info.size);+}++/**+ * Walk one directory tree, accumulating regular-file bytes. A directory+ * that cannot be enumerated refuses its surface once - and the bytes+ * read below the point of refusal are discarded with it: the caller+ * (`measureDirSurface`) records a refused surface in its refusal set+ * only, so no caller can read a partial total as the surface's size.+ */+function walkTree(+ root: string,+ skipTopLevel: ReadonlyArray<string>,+ add: (bytes: number) => void,+ refuse: () => void,+): void {+ let entries;+ try {+ entries = readdirSync(root, { withFileTypes: true });+ } catch (err) {+ if (!isAbsent(err)) refuse();+ return;+ }+ for (const entry of entries) {+ if (entry.isDirectory()) {+ if (skipTopLevel.includes(entry.name)) continue;+ walkTree(join(root, entry.name), [], add, refuse);+ continue;+ }+ if (!entry.isFile()) continue;+ addFile(join(root, entry.name), add, refuse);+ }+}++/** The `Brain/` dot-directories that are surfaces of their own. */+const DOT_DIR_SURFACES: Readonly<Record<string, StorageSurface>> = Object.freeze({+ [".snapshots"]: STORAGE_SURFACE.snapshots,+ [".artifacts"]: STORAGE_SURFACE.artifacts,+ [".archives"]: STORAGE_SURFACE.archives,+ [".payloads"]: STORAGE_SURFACE.payloads,+ [".state"]: STORAGE_SURFACE.internalState,+});++/** Measure one directory-backed surface. */+function measureDirSurface(+ surface: StorageSurface,+ root: string,+ skipTopLevel: ReadonlyArray<string>,+ readings: Map<StorageSurface, number>,+ refusals: Set<StorageSurface>,+): void {+ let bytes = 0;+ let refused = refusals.has(surface);+ const add = (n: number): void => {+ bytes += n;+ };+ const refuse = (): void => {+ refused = true;+ };+ walkTree(root, skipTopLevel, add, refuse);+ if (refused) refusals.add(surface);+ else readings.set(surface, bytes);+}++/** Measure the derived store surface: the store file plus its sidecars. */+function measureStoreSurface(+ storePath: string,+ readings: Map<StorageSurface, number>,+ refusals: Set<StorageSurface>,+): void {+ let bytes = 0;+ let refused = false;+ const add = (n: number): void => {+ bytes += n;+ };+ const refuse = (): void => {+ refused = true;+ };+ for (const suffix of ["", STORE_WAL_SUFFIX, STORE_SHM_SUFFIX]) {+ addFile(`${storePath}${suffix}`, add, refuse);+ }+ if (refused) refusals.add(STORAGE_SURFACE.derivedStore);+ else readings.set(STORAGE_SURFACE.derivedStore, bytes);+}++/**+ * Measure every storage surface of `vault`, or refuse by name.+ *+ * Absent surfaces read as measured zero. An unreadable one produces no+ * byte claim at all: it is left out of `surfaces` and named in+ * `unmeasured`, so no caller can mistake "could not look" for "empty".+ */+export function measureStorageUsage(+ vault: string,+ opts: MeasureStorageUsageOptions = {},+): StorageUsage {+ const readings = new Map<StorageSurface, number>();+ const refusals = new Set<StorageSurface>();++ const brain = join(vault, BRAIN_ROOT_REL);+ // The catch-all tree walks everything EXCEPT the dot-dirs that are+ // surfaces of their own; those five roots are measured separately.+ measureDirSurface(+ STORAGE_SURFACE.brainTree,+ brain,+ Object.keys(DOT_DIR_SURFACES),+ readings,+ refusals,+ );+ for (const [dir, surface] of Object.entries(DOT_DIR_SURFACES)) {+ measureDirSurface(surface, join(brain, dir), [], readings, refusals);+ }+ const storePath = opts.storePath ?? resolveIndexPath(vault, null);+ measureStoreSurface(storePath, readings, refusals);++ const surfaces = STORAGE_SURFACES.filter((s) => readings.has(s)).map((s) => ({+ surface: s,+ tier: SURFACE_TIERS[s],+ bytes: readings.get(s)!,+ }));+ const bytesByTier = Object.fromEntries(+ STORAGE_TIERS.map((tier) => [+ tier,+ surfaces.filter((s) => s.tier === tier).reduce((sum, s) => sum + s.bytes, 0),+ ]),+ ) as Record<StorageTier, number>;+ const unmeasured = STORAGE_SURFACES.filter((s) => refusals.has(s)).map((s) => ({+ surface: s,+ reason: STORAGE_UNMEASURED_REASON.surfaceUnreadable,+ }));+ return {+ measuredAt: (opts.now ?? new Date()).toISOString(),+ complete: unmeasured.length === 0,+ surfaces,+ bytesByTier,+ totalBytes: surfaces.reduce((sum, s) => sum + s.bytes, 0),+ unmeasured,+ };+}diff --git a/src/core/brain/types.ts b/src/core/brain/types.tsindex f8613028..9f68a2ce 100644--- a/src/core/brain/types.ts+++ b/src/core/brain/types.ts@@ -713,6 +713,14 @@ export const BRAIN_SNAPSHOT_REASON = Object.freeze({* payload files nothing in the vault references any more (t_35440e83).*/payloadGc: "payload-gc",+ /**+ * Pre-rewrite point taken before a confirmed receipt compaction folds+ * receipt-ledger shards in place (storage-lifecycle wave). Kebab-case+ * like its siblings because the value doubles as the archive's+ * filename prefix, and named for the receipts surface - not for the+ * Hermes context compaction, which rewrites no ledger.+ */+ receiptCompaction: "receipt-compaction",/** Deferred: a session boundary. No producer in this release. */sessionBoundary: "session-boundary",/** Deferred: a plan boundary. No producer in this release. */@@ -1617,6 +1625,36 @@ export interface ResolvedBrainMaintenanceConfig {readonly failure_streak_limit: number;}+/**+ * Optional `storage:` block (storage lifecycle). The operator's capacity+ * policy: the byte cap the vault refuses writes past, and the percent /+ * day thresholds the forecast warns and goes critical at. Every field+ * optional; absent meaning "the resolver's default decides". Absent+ * `cap_bytes` is unbounded - nothing pauses and nothing refuses.+ */+export interface BrainStorageConfig {+ /** Positive integer bytes; absent (or explicit `null`) = unbounded. */+ readonly cap_bytes?: number | null;+ /** Integer percent of the cap where the forecast warns. */+ readonly warn_percent?: number;+ /** Integer percent of the cap where the forecast goes critical. */+ readonly critical_percent?: number;+ /** Days to the cap at trend where the forecast warns. */+ readonly warn_days?: number;+ /** Days to the cap at trend where the forecast goes critical. */+ readonly critical_days?: number;+}++/** {@link BrainStorageConfig} with every field decided. */+export interface ResolvedBrainStorageConfig {+ /** `null` = no cap is configured; the vault is unbounded. */+ readonly cap_bytes: number | null;+ readonly warn_percent: number;+ readonly critical_percent: number;+ readonly warn_days: number;+ readonly critical_days: number;+}+/*** The `install:` block: the two settings that parameterise GENERATED* install and hook output.@@ -1971,6 +2009,14 @@ export interface BrainConfig {* the single implicit relevance slice.*/readonly recall_inject?: BrainRecallInjectConfig;+ /**+ * Optional `storage:` block (storage lifecycle). The operator's+ * capacity policy: an optional byte cap and the warn / critical+ * thresholds the forecast tiers at. Absent: callers fall back to+ * `BRAIN_STORAGE_DEFAULTS` via `resolveStorage`, with the cap absent -+ * unbounded, nothing pauses, nothing refuses.+ */+ readonly storage?: BrainStorageConfig;}/** Optional `hygiene:` block (continuity-hygiene-freshness suite). */diff --git a/src/mcp/brain-tools.ts b/src/mcp/brain-tools.tsindex cf1cd452..4caf336f 100644--- a/src/mcp/brain-tools.ts+++ b/src/mcp/brain-tools.ts@@ -43,6 +43,8 @@ import { LIFECYCLE_TOOLS } from "./brain/lifecycle-tools.ts";import { LIFECYCLE_FILE_TOOLS } from "./brain/lifecycle-file-tools.ts";import { DECISIONS_TOOLS } from "./brain/decisions-tools.ts";import { TENSION_TOOLS } from "./brain/tension-tools.ts";+import { STORAGE_TOOLS } from "./brain/storage-tools.ts";+import { RECEIPT_TOOLS } from "./brain/receipt-tools.ts";import type { ToolDefinition } from "./tool-contract.ts";export { vaultRelativeSafe } from "./brain/shared.ts";@@ -81,4 +83,9 @@ export const BRAIN_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([...LIFECYCLE_FILE_TOOLS,...DECISIONS_TOOLS,...TENSION_TOOLS,+ // Storage lifecycle: the measured-or-refused storage report, then the+ // receipt-compaction scan / compact surface over the continuity and+ // metrics receipt ledgers.+ ...STORAGE_TOOLS,+ ...RECEIPT_TOOLS,]);diff --git a/src/mcp/brain/admin-tools.ts b/src/mcp/brain/admin-tools.tsindex d7d80c2b..f243c833 100644--- a/src/mcp/brain/admin-tools.ts+++ b/src/mcp/brain/admin-tools.ts@@ -32,7 +32,11 @@ import type { ProgressSink } from "../../core/brain/progress.ts";import { readableAtContextReach, readableAtContextReachOrUndefined } from "./reach-readable.ts";import { requiredStringArg, toolSafeguard } from "./shared.ts";import { currentLease } from "../../core/brain/maintenance/lease.ts";-import { listJournal, MAINTENANCE_JOURNAL_CAP } from "../../core/brain/maintenance/journal.ts";+import {+ journalEntryWithoutCapacityReading,+ listJournal,+ MAINTENANCE_JOURNAL_CAP,+} from "../../core/brain/maintenance/journal.ts";import {CUSTOM_TASK_MAX,customTasksOffNotice,@@ -374,11 +378,18 @@ async function toolBrainMaintenance(// The journal depth is the caller's, as it is on the CLI. It was// hardcoded at ten here, so an agent reading a lane that had refused// a task days ago could not see far enough back to find the failures- // behind the streak - the one thing the journal is kept for.+ // behind the streak - the one thing the journal is kept for. Below+ // local reach the rows lose their capacity reading: the whole-vault+ // byte figures are the same local-reach facts `brain_storage` gates+ // (`journalEntryWithoutCapacityReading`), and status - unlike run -+ // answers at every reach.+ const local = contextReach(ctx) === TRANSPORT_REACH.local;const notice = customTasksOffNotice(resolveCustomTasks(ctx.configPath ?? undefined));return {lease: currentLease(ctx.vault, { now }),- journal: listJournal(ctx.vault, coerceInt(args, "limit", 10, 1, MAINTENANCE_JOURNAL_CAP)),+ journal: listJournal(ctx.vault, coerceInt(args, "limit", 10, 1, MAINTENANCE_JOURNAL_CAP)).map(+ (entry) => (local ? entry : journalEntryWithoutCapacityReading(entry)),+ ),...(notice !== null ? { notice } : {}),};}@@ -495,6 +506,10 @@ async function toolBrainMaintenance(return {verdict: result.verdict,tasks: result.tasks,+ // The capacity gate's report on a skipped:capacity run: the tier and+ // the remedy, attached verbatim from the lane so the two front doors+ // cannot disagree about what a skip says.+ ...(result.capacity !== undefined ? { capacity: result.capacity } : {}),...(spend !== undefined ? { spend } : {}),// Named, never dropped: a refused declaration journals nothing, and// the valid tasks still ran.@@ -580,7 +595,7 @@ export const ADMIN_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([{name: "brain_maintenance",description:- "Quiet-window, lease-guarded maintenance lane: run executes dream, reindex, bridges, clusters and declared custom:<name> tasks behind window, busy, pressure and streak gates; status renders lease and journal. Reindex embeds only with config maintenance_embeddings.",+ "Quiet-window, lease-guarded maintenance lane: run executes dream, reindex, bridges, clusters and declared custom:<name> tasks behind window, busy, pressure, capacity and streak gates; status renders lease and journal. Reindex embeds only with config maintenance_embeddings.",inputSchema: {type: "object",properties: {@@ -592,7 +607,7 @@ export const ADMIN_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([// A caller reaching for the widest escape has to be told what// it actually switches off.description:- "Bypass the window, busy and host-pressure gates and every streak refusal - never the lease (run). retry_tasks passes one task's streak with the gates kept.",+ "Bypass the window, busy, host-pressure and storage-capacity gates and every streak refusal; never the lease (run). retry_tasks passes one streak, gates kept.",},retry_tasks: {type: "array",diff --git a/src/mcp/brain/health-tools.ts b/src/mcp/brain/health-tools.tsindex fd8c0b1e..28a8fa2a 100644--- a/src/mcp/brain/health-tools.ts+++ b/src/mcp/brain/health-tools.ts@@ -123,6 +123,10 @@ async function toolBrainDoctor(// discovery would find....(ctx.configPath !== null ? { configPath: ctx.configPath } : {}),...(readable !== undefined ? { readable } : {}),+ // The checks whose findings carry whole-vault figures answer below+ // local reach in the uncertain stream, by name (`doctor/check.ts`).+ // Resolved fail-closed: a context that proved nothing is remote.+ reach: contextReach(ctx),});// Every issue names the artifact it is about - `path`, `target`, thediff --git a/src/mcp/brain/hygiene-tools.ts b/src/mcp/brain/hygiene-tools.tsindex 295c4d94..b18f9947 100644--- a/src/mcp/brain/hygiene-tools.ts+++ b/src/mcp/brain/hygiene-tools.ts@@ -61,12 +61,29 @@ import {type LinkRatchetUnmeasurableReason,} from "../../core/search/link-ratchet.ts";import { coerceBool, coerceStr } from "../coerce.ts";+import { redactErrorForCaller } from "../error-redaction.ts";import { INVALID_PARAMS, MCPError } from "../protocol.ts";import { MCP_PREVIEW_BUDGET } from "../preview-budget.ts";import type { ServerContext, ToolDefinition } from "../tool-contract.ts";import { enforceCountGuard, readCountGuardArgs, vaultRelativeSafe } from "./shared.ts";import { readableAtContextReach, readableAtContextReachOrUndefined } from "./reach-readable.ts";+/**+ * The prose of one error row in a scan, refresh or apply payload, stated+ * for the caller's reach. These rows ride the SUCCESS channel, which the+ * server's redactor never sees - it wraps thrown errors only - and a+ * detector or executor failure forwards Node's fs text, which names+ * absolute host paths, verbatim. Below local reach the same substitution+ * a throw gets at this reach is applied here, so one class of text+ * cannot be redacted on one channel and ride raw on the other; at local+ * reach the row is unchanged, as a throw's is.+ */+function errorRowForCaller<E extends { readonly message: string }>(ctx: ServerContext, row: E): E {+ const reach = contextReach(ctx);+ if (reach === TRANSPORT_REACH.local) return row;+ return { ...row, message: redactErrorForCaller(row.message, ctx.vault, reach) };+}+function coerceStringArray(args: Record<string, unknown>, key: string): string[] | undefined {const raw = args[key];if (raw === undefined) return undefined;@@ -312,7 +329,7 @@ async function toolBrainHygiene(rederived: result.rederived.map((page) => vaultRelativeSafe(ctx.vault, page)),archived: result.archived.map((page) => vaultRelativeSafe(ctx.vault, page)),manual: result.manual.map((page) => vaultRelativeSafe(ctx.vault, page)),- errors: result.errors,+ errors: result.errors.map((row) => errorRowForCaller(ctx, row)),};}@@ -379,7 +396,7 @@ async function toolBrainHygiene(annotated === null? findings.map((finding) => findingView(ctx.vault, finding)): annotated.map((a) => ({ ...findingView(ctx.vault, a.item), ...verdictFields(a) })),- errors: report.errors,+ errors: report.errors.map((row) => errorRowForCaller(ctx, row)),link_integrity: await linkIntegrityView(ctx),};}@@ -403,6 +420,11 @@ async function toolBrainHygiene(dryRun,agent: resolveAgentName(ctx.configPath ?? undefined),now,+ // The deny consult on the destructive retire route answers for the+ // credential-minted subject, never for the config agent - the same+ // threading every other destructive surface applies (write-side+ // trust, Task 7).+ ...(ctx.requestIdentity !== undefined ? { subject: ctx.requestIdentity } : {}),});return {mode,@@ -415,7 +437,7 @@ async function toolBrainHygiene(// Honest matched-vs-changed: findings selected vs. findings actually applied.matched: plan.selected.length,changed: result.applied.length,- errors: result.errors,+ errors: result.errors.map((row) => errorRowForCaller(ctx, row)),};}diff --git a/src/mcp/brain/ingest-tools.ts b/src/mcp/brain/ingest-tools.tsindex 81d2996a..ff98a1e0 100644--- a/src/mcp/brain/ingest-tools.ts+++ b/src/mcp/brain/ingest-tools.ts@@ -16,6 +16,18 @@ import { serializeImportCensus } from "../../core/brain/import-census.ts";import { planBatches, type BatchPlan } from "../../core/brain/ingest/batch-plan.ts";import { clearCheckpoint, PLAN_ID_LABEL } from "../../core/brain/ingest/checkpoint.ts";import { assertCheckpointId } from "../../core/brain/checkpoint-store.ts";+import { STORAGE_CAPACITY_CRITICAL_CODE } from "../../core/brain/doctor/storage-capacity.ts";+import { MAINTENANCE_CAPACITY_REMEDY } from "../../core/brain/maintenance/lane.ts";+import { loadStorageConfigSafe } from "../../core/brain/policy/load.ts";+import {+ StorageCapacityError,+ type StorageErrorCode,+} from "../../core/brain/storage/capacity-error.ts";+import {+ CAPACITY_GATE_DECISION,+ evaluateCapacityGate,+ type CapacityGateDecision,+} from "../../core/brain/storage/gate.ts";import { ingestSource } from "../../core/brain/ingest/ingest.ts";import type { PartsOutcome, TableOutcome } from "../../core/brain/ingest/extract-source.ts";import type { CodeEdgeSeed, PreExtractResult } from "../../core/brain/ingest/pre-extract.ts";@@ -37,6 +49,7 @@ import {} from "../../core/brain/source-cleanup.ts";import { resolveAgentName } from "../../core/config.ts";import { normalizeAgentScope } from "../../core/graph/agent-scope.ts";+import { TRANSPORT_REACH, type TransportReach } from "../../core/graph/transport-reach.ts";import { isPathOwnerVisible, type FrontmatterCache } from "../../core/search/result-filters.ts";import {coerceAgentScope,@@ -47,7 +60,8 @@ import {} from "../coerce.ts";import { MCP_PREVIEW_BUDGET } from "../preview-budget.ts";import { INVALID_PARAMS, MCPError } from "../protocol.ts";-import type { ServerContext, ToolDefinition } from "../tool-contract.ts";+import { codeForError } from "../tool-error-codes.ts";+import { contextReach, type ServerContext, type ToolDefinition } from "../tool-contract.ts";import { parseExtractionIntakeArgs } from "./intake-args.ts";import { readableAtContextReach } from "./reach-readable.ts";import {@@ -69,6 +83,98 @@ const BATCH_PLAN_TOOL = "brain_ingest_batch_plan";const DEFAULT_MAX_BATCH_BYTES = 1024 * 1024;const DEFAULT_MAX_BATCH_FILES = 25;+/** The wire code of the capacity admission refusal, from the leaf's closed union. */+const ADMISSION_REFUSAL_CODE: StorageErrorCode = "capacity_exhausted";++/**+ * Refuse the ingest when a fresh census measures the vault at its+ * critical storage tier - the interactive half of the storage lifecycle's+ * backpressure, beside the maintenance lane's gate.+ *+ * The check sits in the VALIDATE phase, before `ingestSource` opens the+ * write path: the commit phase is never entered, so there is nothing to+ * roll back, and a refusal leaves the vault byte-identical. Only a+ * MEASURED critical tier closes it, exactly as the lane's gate does - an+ * unmeasurable store stays open, because a census that could not be taken+ * is not a vault with room, but neither is it a vault without. No+ * configured cap is the documented default and means the check does not+ * exist: no gate, no byte walk, no cost.+ *+ * The gate itself is a write gate, not a secret, so it fires at every+ * reach - but the refusal throws the registered leaf carrying the+ * envelope's fields, and {@link admissionDetails} gates the whole-vault+ * figures by reach, for the same reason `brain_storage` refuses to take+ * its census below local reach.+ *+ * The remedy is the lane's own sentence, quoted rather than restated, so+ * the two walls an operator can hit - the paused lane and the refused+ * write - cannot disagree about what to do next.+ */+function refuseWhenVaultExhausted(vault: string, reach: TransportReach, now: Date): void {+ const storage = loadStorageConfigSafe(vault);+ if (storage.cap_bytes === null) return;+ const decision = evaluateCapacityGate(vault, {+ now,+ capBytes: storage.cap_bytes,+ warnPercent: storage.warn_percent,+ criticalPercent: storage.critical_percent,+ warnDays: storage.warn_days,+ criticalDays: storage.critical_days,+ });+ if (decision.decision !== CAPACITY_GATE_DECISION.skip) return;+ throw new StorageCapacityError(+ ADMISSION_REFUSAL_CODE,+ `${TOOL}: refused (${ADMISSION_REFUSAL_CODE}): the vault measured at the ` ++ `${decision.tier} storage tier. ${MAINTENANCE_CAPACITY_REMEDY}.`,+ admissionDetails(decision, reach),+ );+}++/**+ * The structured fields beside the admission refusal's code. The tier and+ * the remedy ride at every reach; the whole-vault figures do not - they+ * are the same local-reach facts the storage report refuses to take below+ * local reach (`buildStorageReport`), and handing them to a remote caller+ * would answer the refusal with a census and a growth oracle in one+ * payload. The `next_command` field stays: the local remedy is the one+ * thing a refused caller still needs in hand.+ */+function admissionDetails(+ decision: CapacityGateDecision,+ reach: TransportReach,+): Record<string, unknown> {+ const reading = decision.reading;+ return {+ tier: decision.tier,+ ...(reach === TRANSPORT_REACH.local+ ? {+ measured_bytes: reading?.measuredBytes ?? null,+ cap_bytes: reading?.capBytes ?? null,+ percent_used: reading?.percentUsed ?? null,+ }+ : {}),+ remedy: MAINTENANCE_CAPACITY_REMEDY,+ // The registered next step for a critical vault (the storage+ // report), resolved from the diagnostics registry - a refusal whose+ // value is the pointer never spells the command by hand.+ ...nextCommandField(STORAGE_CAPACITY_CRITICAL_CODE),+ };+}++/**+ * The MCP envelope for an admission refusal, from the thrown leaf: the+ * wire code is the code the registered `carried()` classification reads+ * off this leaf (`codeForError`), never a restated literal, and the data+ * fields are the leaf's own - so the envelope cannot drift from the+ * vocabulary the wire registry knows.+ */+function capacityRefusalEnvelope(err: StorageCapacityError): MCPError {+ return new MCPError(INVALID_PARAMS, err.message, {+ code: codeForError(err),+ ...err.details,+ });+}+async function toolBrainIngestSource(ctx: ServerContext,args: Record<string, unknown>,@@ -105,6 +211,17 @@ async function toolBrainIngestSource(claimedAgent ?? ctx.requestIdentity?.agent ?? resolveAgentName(ctx.configPath ?? undefined);return wrapToolErrors(TOOL, [IntakeValidationError], async () => {+ // The admission wall, in the validate phase: past this call the write+ // path opens, and the commit phase ahead of it is never rolled back.+ // The check throws the registered leaf; it is mapped onto its wire+ // envelope here, before the generic wrapper could flatten it into an+ // internal error.+ try {+ refuseWhenVaultExhausted(ctx.vault, contextReach(ctx), new Date());+ } catch (err) {+ if (err instanceof StorageCapacityError) throw capacityRefusalEnvelope(err);+ throw err;+ }let res: IngestSourceResult;try {res = ingestSource(diff --git a/src/mcp/brain/receipt-tools.ts b/src/mcp/brain/receipt-tools.tsnew file mode 100644index 00000000..6b7eda3e--- /dev/null+++ b/src/mcp/brain/receipt-tools.ts@@ -0,0 +1,222 @@+/**+ * Receipt surface: scan / compact over the receipt ledgers+ * (storage-lifecycle, Lane 3 task 3.4).+ *+ * One tool, two modes, derived from the shared receipts vocabulary the+ * `o2b brain receipts` verb uses, so the two surfaces cannot disagree on+ * the mode names or the `family:name` shard spelling; the responses are+ * the same views the verb's `--json` renders+ * (`src/core/brain/receipts/views.ts`), so neither can drift on field+ * spellings either:+ *+ * - `scan` (default): read-only inventory of both receipt ledgers -+ * per-shard record, foldable, protected and unparseable counts with+ * byte sizes. Writes nothing.+ * - `compact`: fold expired records into per-(kind, month) aggregates+ * in EXPLICITLY named shards. Preview-bound: without `apply` it is a+ * dry run; with `apply` the core runs the count guard, consults the+ * permissions document, takes the recovery point and rewrites each+ * shard atomically. The operation names itself receipt compaction -+ * deliberately disjoint from the Hermes context-compaction+ * vocabulary on every surface.+ */++import { resolveAgentName } from "../../core/config.ts";+import { compactReceiptLedgers } from "../../core/brain/receipts/compact.ts";+import { scanReceiptLedgers } from "../../core/brain/receipts/scan.ts";+import { compactView, scanView } from "../../core/brain/receipts/views.ts";+import {+ isReceiptMode,+ parseReceiptShardRef,+ RECEIPT_COMPACTION_OPERATION,+ RECEIPT_FOLD_DEFAULT_AGE_DAYS,+ RECEIPT_MODE,+ RECEIPT_MODES,+ type ReceiptShardRef,+} from "../../core/brain/receipts/types.ts";+import { CountGuardError } from "../../core/brain/count-guard.ts";+import { WriteRefusedError } from "../../core/brain/write-disposition.ts";+import { TRANSPORT_REACH } from "../../core/graph/transport-reach.ts";+import { coerceBool } from "../coerce.ts";+import { INVALID_PARAMS, MCPError } from "../protocol.ts";+import { MCP_PREVIEW_BUDGET } from "../preview-budget.ts";+import { COUNT_GUARD_WIRE_CODE } from "../tool-error-codes.ts";+import { contextReach, type ServerContext, type ToolDefinition } from "../tool-contract.ts";+import { readCountGuardArgs } from "./shared.ts";++/** Read and bound the age-threshold argument. */+function readOlderThanDays(args: Record<string, unknown>): number {+ const raw = args["older_than_days"];+ if (raw === undefined || raw === null) return RECEIPT_FOLD_DEFAULT_AGE_DAYS;+ const days = typeof raw === "number" ? raw : Number(raw);+ if (!Number.isInteger(days) || days <= 0) {+ throw new MCPError(INVALID_PARAMS, "'older_than_days' must be a positive integer");+ }+ return days;+}++/**+ * The reason a compact below local reach names, beside the reasons the+ * core itself refuses: the fold was not planned at all because of who+ * was asking, not because a shard refused.+ */+export const RECEIPT_COMPACT_REACH_REASON = "reach" as const;++/** Every reason a compact can name for answering without a plan. */+export type ReceiptCompactRefusalReason = typeof RECEIPT_COMPACT_REACH_REASON;++/** The response a caller below local reach receives: no plan, a named refusal. */+export interface ReceiptCompactRefusal {+ readonly operation: typeof RECEIPT_COMPACTION_OPERATION;+ readonly measured: false;+ readonly reason: ReceiptCompactRefusalReason;+ /** Why the compact is local-reach-only, stated for the caller. */+ readonly detail: string;+}++/** Parse the explicit `family:name` shard list compact requires. */+function readShards(args: Record<string, unknown>): ReceiptShardRef[] {+ const raw = args["shards"];+ if (!Array.isArray(raw) || raw.length === 0 || !raw.every((v) => typeof v === "string")) {+ throw new MCPError(+ INVALID_PARAMS,+ "compact requires explicit 'shards' (array of family:name strings) from a scan",+ );+ }+ return (raw as string[]).map((value) => {+ try {+ return parseReceiptShardRef(value);+ } catch (exc) {+ throw new MCPError(INVALID_PARAMS, (exc as Error).message);+ }+ });+}++async function toolBrainReceipts(+ ctx: ServerContext,+ args: Record<string, unknown>,+): Promise<Record<string, unknown>> {+ const mode = args["mode"] ?? RECEIPT_MODE.scan;+ if (!isReceiptMode(mode)) {+ throw new MCPError(INVALID_PARAMS, `'mode' must be one of: ${RECEIPT_MODES.join(", ")}`);+ }+ const olderThanDays = readOlderThanDays(args);++ if (mode === RECEIPT_MODE.scan) {+ return scanView(scanReceiptLedgers(ctx.vault, { olderThanDays, now: new Date() }));+ }++ // The destructive half, gated like the storage report gates its+ // whole-vault census (`./storage-tools.ts`): compact rewrites named+ // ledger shards in place and reports whole-ledger byte totals, and a+ // caller below local reach gets the named refusal instead of a plan -+ // no shard list, no figures, no write lane entered. The scan above+ // stays readable at any reach: it writes nothing.+ if (contextReach(ctx) !== TRANSPORT_REACH.local) {+ // Returned as the literal, `satisfies`-checked against the refusal+ // interface: the handler's wire type is Record<string, unknown>, and+ // a value held in the interface's name would not carry the index+ // signature that type asks for.+ return {+ operation: RECEIPT_COMPACTION_OPERATION,+ measured: false,+ reason: RECEIPT_COMPACT_REACH_REASON satisfies ReceiptCompactRefusalReason,+ detail:+ "receipt compaction folds records in named ledger shards and reports " ++ "whole-ledger byte totals; it answers at local reach only",+ } satisfies ReceiptCompactRefusal;+ }++ const shards = readShards(args);+ const apply = coerceBool(args, "apply") === true;+ const { expect, strict } = readCountGuardArgs(args);+ try {+ const result = compactReceiptLedgers(ctx.vault, {+ shards,+ olderThanDays,+ now: new Date(),+ dryRun: !apply,+ expect,+ strict,+ subject: ctx.requestIdentity ?? {+ agent: resolveAgentName(ctx.configPath ?? undefined),+ via: "config",+ },+ configPath: ctx.configPath ?? undefined,+ });+ return compactView(result);+ } catch (err) {+ // The caller's fault, by name: a permissions-document rule denied the+ // compact, or the count guard measured a different population than+ // the `expect` asserted - the mismatch numbers are the whole remedy,+ // so they travel beside the code.+ if (err instanceof WriteRefusedError) {+ throw new MCPError(INVALID_PARAMS, `brain_receipts: ${err.message}`, {+ code: err.code,+ rule: err.rule,+ agent: err.agent,+ next_command: err.nextCommand,+ });+ }+ if (err instanceof CountGuardError) {+ throw new MCPError(INVALID_PARAMS, `brain_receipts: ${err.message}`, {+ code: COUNT_GUARD_WIRE_CODE,+ matched: err.matched,+ expected: err.expected,+ });+ }+ throw err;+ }+}++export const RECEIPT_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([+ {+ name: "brain_receipts",+ previewBudget: MCP_PREVIEW_BUDGET,+ description:+ "Receipt compaction. `scan` (default) inventories both ledgers read-only; `compact` folds " ++ "expired records into per-kind, per-month aggregates in named shards, preview-bound without " ++ "`apply`, local reach only. Never touches decision-change receipts, the source archive, or " ++ "context compaction.",+ inputSchema: {+ type: "object",+ properties: {+ mode: {+ type: "string",+ enum: [...RECEIPT_MODES],+ description:+ "Surface mode: scan inventories both ledgers (continuity, metrics); compact folds. Default `scan`.",+ },+ older_than_days: {+ type: "integer",+ minimum: 1,+ description: `Age threshold for foldable records. Default ${RECEIPT_FOLD_DEFAULT_AGE_DAYS}.`,+ },+ shards: {+ type: "array",+ items: { type: "string" },+ description:+ "Shards to compact, as `family:name` ids from a scan (compact mode; required there).",+ },+ apply: {+ type: "boolean",+ description:+ "Perform the fold (compact mode). Without it compact is a dry run; a confirmed run " ++ "takes a recovery point first and rewrites each shard atomically.",+ },+ expect: {+ type: "integer",+ minimum: 0,+ description:+ "Count guard (compact mode): assert exactly N records will fold. On mismatch the run aborts without writing.",+ },+ strict: {+ type: "boolean",+ description: "Refuse a guardless confirmed compact (no `expect`). Default false.",+ },+ },+ additionalProperties: false,+ },+ handler: toolBrainReceipts,+ },+]);diff --git a/src/mcp/brain/storage-tools.ts b/src/mcp/brain/storage-tools.tsnew file mode 100644index 00000000..e4717d3d--- /dev/null+++ b/src/mcp/brain/storage-tools.ts@@ -0,0 +1,166 @@+/**+ * The storage report: one measured-or-refused answer about what this+ * vault's storage holds and where it is heading.+ *+ * One builder serves both front doors - `o2b brain storage` and this+ * `brain_storage` tool - so the two surfaces cannot disagree about the+ * census, the forecast or the tier: the report object IS the shared+ * source, and each surface only decides how to render it.+ *+ * ## What one invocation does+ *+ * The invocation takes ONE census and puts it to all three uses the+ * report has: the per-surface reading the caller sees, the fresh+ * `latestUsage` reading the forecast tiers on (the two-truths rule in+ * `../core/brain/storage/forecast.ts` - a stored aggregate is advisory,+ * a fresh census is the fact), and the one `storage_usage` sample the+ * invocation records. Each report therefore appends exactly one sample;+ * the series keeps accumulating precisely because the report was asked+ * for.+ *+ * ## Reach honesty+ *+ * The census is a whole-vault byte figure: it counts every page,+ * withheld pages included, so its totals move when a page the caller may+ * not read appears or goes, and no filtered form of it exists. Below+ * local reach the number is therefore not taken at all - the+ * `link_integrity` model (`./hygiene-tools.ts`) - and the report answers+ * `measured: false` with the named reason. No sample is recorded on a+ * refused call: the sample would require taking the very census the+ * refusal withholds.+ *+ * ## No cap means unbounded+ *+ * An absent `storage.cap_bytes` is the default and is reported as the+ * tier it is: `unbounded`. Bytes are reported, no days-to-exhaustion is+ * invented, nothing pauses and nothing refuses - a forecast against an+ * invented default cap would be exactly the misleading number the+ * storage lifecycle removes.+ */++import { loadBrainConfigDetailed, resolveStorage } from "../../core/brain/policy.ts";+import {+ forecastStorageUsage,+ sampleStorageUsage,+ type StorageForecast,+} from "../../core/brain/storage/forecast.ts";+import type { StorageUsage } from "../../core/brain/storage/usage.ts";+import { TRANSPORT_REACH, type TransportReach } from "../../core/graph/transport-reach.ts";+import type { ServerContext, ToolDefinition } from "../tool-contract.ts";+import { contextReach } from "../tool-contract.ts";+import { coerceFormat } from "../coerce.ts";+import { MCP_PREVIEW_BUDGET } from "../preview-budget.ts";++/**+ * The reason a report below local reach names, beside the reasons the+ * measurement itself returns (`surface_unreadable`): the census was not+ * taken at all because of who was asking, not because it failed.+ */+export const STORAGE_REACH_REASON = "reach" as const;++/** Every reason a report can name for answering without a measurement. */+export type StorageReportRefusalReason = typeof STORAGE_REACH_REASON;++/** The report a caller below local reach receives: no bytes, a named reason. */+export interface StorageReportRefusal {+ readonly measured: false;+ readonly reason: StorageReportRefusalReason;+ /** Why the census is local-reach-only, stated for the caller. */+ readonly detail: string;+}++/** The report a local-reach caller receives: census, forecast, sample. */+export interface StorageReport {+ readonly measured: true;+ /** The census the invocation took; refused surfaces are in `unmeasured`. */+ readonly usage: StorageUsage;+ /** The forecast over the configured cap, tiered on the fresh census. */+ readonly forecast: StorageForecast;+ /** The sample this invocation recorded, named when the append failed. */+ readonly sample: {+ readonly runId: string;+ readonly appended: boolean;+ readonly appendError?: string;+ };+}++/** Either a measured report or the named refusal to take one. */+export type StorageReportView = StorageReport | StorageReportRefusal;++export interface BuildStorageReportOptions {+ readonly vault: string;+ /** How far the caller had to reach; the census answers at local reach only. */+ readonly reach: TransportReach;+}++/**+ * Build one storage report, or the named reach refusal.+ *+ * A vault whose brain config cannot be read throws the named+ * `BrainConfigError`: a report about an uninitialized vault has no cap to+ * forecast against, and answering as an unbounded empty vault would be+ * the misleading default this subsystem refuses to invent.+ */+export function buildStorageReport(opts: BuildStorageReportOptions): StorageReportView {+ if (opts.reach !== TRANSPORT_REACH.local) {+ return {+ measured: false,+ reason: STORAGE_REACH_REASON satisfies StorageReportRefusalReason,+ detail: "storage usage is measured over the whole vault and is reported at local reach only",+ };+ }+ const storage = resolveStorage(loadBrainConfigDetailed(opts.vault).config);+ // One census, three uses: the reading, the forecast's fresh truth,+ // and the sample. Fail-soft by the sampler's contract - an append+ // failure is carried on the report, not swallowed.+ const sample = sampleStorageUsage(opts.vault);+ const forecast = forecastStorageUsage(opts.vault, {+ capBytes: storage.cap_bytes,+ latestUsage: sample.usage,+ warnPercent: storage.warn_percent,+ criticalPercent: storage.critical_percent,+ warnDays: storage.warn_days,+ criticalDays: storage.critical_days,+ });+ return {+ measured: true,+ usage: sample.usage,+ forecast,+ sample: {+ runId: sample.runId,+ appended: sample.appended,+ ...(sample.appendError !== undefined ? { appendError: sample.appendError } : {}),+ },+ };+}++async function toolBrainStorage(+ ctx: ServerContext,+ args: Record<string, unknown>,+): Promise<Record<string, unknown>> {+ const format = coerceFormat(args);+ const report = buildStorageReport({ vault: ctx.vault, reach: contextReach(ctx) });+ return { format, ...report };+}++export const STORAGE_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([+ {+ name: "brain_storage",+ description:+ "Storage report: per-surface byte census (measured or refused by name), the growth forecast against the cap with its tier, and the usage sample this invocation records. Below local reach the census is not taken: measured: false, reason named. Read-only over vault content; appends one usage sample.",+ inputSchema: {+ type: "object",+ properties: {+ format: {+ type: "string",+ enum: ["markdown", "json"],+ description:+ "Output format hint. Structured result is identical; caller decides rendering.",+ },+ },+ additionalProperties: false,+ },+ previewBudget: MCP_PREVIEW_BUDGET,+ handler: toolBrainStorage,+ },+]);diff --git a/src/mcp/tool-error-codes.ts b/src/mcp/tool-error-codes.tsindex c389c5e5..b8c33414 100644--- a/src/mcp/tool-error-codes.ts+++ b/src/mcp/tool-error-codes.ts@@ -58,6 +58,8 @@ import { BrainConfigError } from "../core/brain/policy/errors.ts";import { ResponseCheckError, SEMANTIC_VIOLATION_CODES } from "../core/brain/response-checks.ts";import { ResponseShapeError, SHAPE_VIOLATION_CODES } from "../core/brain/response-shape.ts";import { SafeguardAbortError, SafeguardTimeoutError } from "../core/brain/safeguard.ts";+import type { StorageErrorCode } from "../core/brain/storage/capacity-error.ts";+import { StorageCapacityError } from "../core/brain/storage/capacity-error.ts";import type { WriteBatchErrorCode } from "../core/brain/write-batch.ts";import { WriteBatchError } from "../core/brain/write-batch.ts";import { ConfigReadError } from "../core/config.ts";@@ -252,6 +254,25 @@ const HOST_MEMORY_WRITE_CODES = exhaustiveMembers<HostMemoryWriteErrorCode>()(["empty_content",]);+/**+ * The storage-capacity vocabulary. The leaf owns the canonical member+ * list as `STORAGE_ERROR_CODES`; the literals are spelled out here so+ * `exhaustiveMembers` fails `tsc` the moment the leaf grows a code this+ * table has not answered for - the same two-direction check every other+ * imported union gets.+ */+const STORAGE_CAPACITY_CODES = exhaustiveMembers<StorageErrorCode>()([+ "capacity_unmeasurable",+ "capacity_exhausted",+]);++/**+ * The leaf vocabulary, re-exported under the one name consumers of the+ * wire vocabulary already import from here. The canonical home stays in+ * the leaf; this only spares callers a second import path.+ */+export { STORAGE_ERROR_CODES } from "../core/brain/storage/capacity-error.ts";+/*** The token a count-guard refusal is reported under on the wire. The core* `CountGuardError` carries `COUNT_GUARD`; `brain_note_lifecycle` has@@ -291,6 +312,7 @@ export const TOOL_ERROR_CODES = Object.freeze([...PINNED_BATCH_CODES,...EXACT_STATE_CODES,...HOST_MEMORY_WRITE_CODES,+ ...STORAGE_CAPACITY_CODES,COUNT_GUARD_WIRE_CODE,...EXPIRATION_REFUSAL_CODES,] as const),@@ -365,6 +387,7 @@ const CLASSIFICATION: ReadonlyArray<ClassificationRule> = Object.freeze([carried(PinnedBatchError),carried(ExactStateError),carried(HostMemoryWriteError),+ carried(StorageCapacityError),carried(SearchError),carried(ResponseShapeError),carried(ResponseCheckError),diff --git a/tests/cli/brain-hygiene.test.ts b/tests/cli/brain-hygiene.test.tsindex f8627373..c6f381c2 100644--- a/tests/cli/brain-hygiene.test.ts+++ b/tests/cli/brain-hygiene.test.ts@@ -143,7 +143,7 @@ describe("o2b brain hygiene scan", () => {{ env: env() },);expect(out.returncode).not.toBe(0);- expect(out.stderr).toContain("conflicts, dedup, freshness, usefulness");+ expect(out.stderr).toContain("conflicts, dedup, document-dedup, freshness, usefulness");});});diff --git a/tests/cli/brain/verbs/page-dedup-apply-fail-soft.test.ts b/tests/cli/brain/verbs/page-dedup-apply-fail-soft.test.tsnew file mode 100644index 00000000..01a96840--- /dev/null+++ b/tests/cli/brain/verbs/page-dedup-apply-fail-soft.test.ts@@ -0,0 +1,211 @@+/**+ * The page-dedup apply loop is fail-soft per merge attempt (Lane 5, Task+ * 5.2): one cluster's MergeChainError - or any other per-attempt failure -+ * is collected as {key, message}, the later clusters still apply, the+ * report lists the failures, and the CLI exits 1 when failures are+ * non-empty. Before the decomposition one throw aborted every cluster+ * queued behind it.+ */++import {+ chmodSync,+ mkdirSync,+ mkdtempSync,+ readFileSync,+ rmSync,+ unlinkSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { afterEach, beforeEach, describe, expect, test } from "bun:test";++import { findDuplicateCandidates } from "../../../../src/core/brain/page-dedup.ts";+import {+ applyDedupCandidates,+ cmdBrainPageDedup,+} from "../../../../src/cli/brain/verbs/page-dedup.ts";+import { CHMOD_CANNOT_DENY } from "../../../helpers/platform.ts";++let tmp: string;+let vault: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-page-dedup-fail-soft-"));+ vault = join(tmp, "vault");+ mkdirSync(vault, { recursive: true });+ mkdirSync(join(vault, "Brain", "preferences"), { recursive: true });+ mkdirSync(join(vault, "Brain", "retired"), { recursive: true });+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++/**+ * One preference page in the dedup pool, with its own topic and principle.+ * `kind` selects the pool directory: `brain-preference` pages live in+ * Brain/preferences, `brain-retired` pages in Brain/retired, so two+ * clusters can sit in directories with independent write bits.+ */+function writePoolPage(+ poolDir: "preferences" | "retired",+ slug: string,+ topic: string,+ principle: string,+ body = "",+): string {+ const prefix = poolDir === "preferences" ? "pref" : "ret";+ const kind = poolDir === "preferences" ? "brain-preference" : "brain-retired";+ const path = join(vault, "Brain", poolDir, `${prefix}-${slug}.md`);+ writeFileSync(+ path,+ [+ "---",+ `kind: ${kind}`,+ `id: ${prefix}-${slug}`,+ "tags: [brain]",+ `topic: ${topic}`,+ "_status: confirmed",+ `principle: ${principle}`,+ "unconfirmed_until: 2026-01-15T00:00:00Z",+ "created_at: 2026-01-01T00:00:00Z",+ "---",+ "",+ body,+ "",+ ].join("\n"),+ "utf8",+ );+ return path;+}++describe("applyDedupCandidates", () => {+ test("a MergeChainError on one cluster is collected; later clusters still apply", () => {+ writePoolPage("preferences", "a", "topic-one", "the first rule", "see also [[pref-b]]");+ writePoolPage("preferences", "b", "topic-one", "the first rule");+ writePoolPage("preferences", "c", "topic-two", "the second rule", "see also [[pref-d]]");+ writePoolPage("preferences", "d", "topic-two", "the second rule");++ const report = findDuplicateCandidates(vault);+ expect(report.candidates.length).toBe(2);++ // Sabotage the FIRST cluster: its secondary vanishes between the scan+ // and the apply, which mergePage refuses with a MergeChainError+ // (MALFORMED, "secondary page not found").+ const first = report.candidates[0]!;+ const sabotaged = first.secondaries[0]!;+ unlinkSync(join(vault, "Brain", "preferences", `${sabotaged.id}.md`));++ const applied = applyDedupCandidates(vault, report.candidates);++ // The failure is named by cluster key and message.+ expect(applied.failures.length).toBe(1);+ expect(applied.failures[0]!.key).toBe(first.key);+ expect(applied.failures[0]!.message).toContain(`secondary page not found: ${sabotaged.id}`);++ // The later cluster still applied.+ expect(applied.merged).toBe(1);+ expect(applied.wikilinksUpdated).toBe(1);+ const second = report.candidates[1]!;+ expect(readFileSync(join(vault, "Brain", "preferences", "pref-d.md"), "utf8")).toContain(+ `merged_into: ${second.canonical.id}`,+ );+ });++ test("a clean apply reports zero failures", () => {+ writePoolPage("preferences", "a", "topic-one", "the first rule", "see also [[pref-b]]");+ writePoolPage("preferences", "b", "topic-one", "the first rule");++ const applied = applyDedupCandidates(vault, findDuplicateCandidates(vault).candidates);+ expect(applied.failures).toEqual([]);+ expect(applied.merged).toBe(1);+ });++ test("a link-pass failure reports the partial count and the stamped secondary", () => {+ if (CHMOD_CANNOT_DENY) return;+ // pref-a cites pref-b (rewritable), and a second citing page sits in+ // its own directory the test makes unwritable: the merge STAMP lands,+ // the link pass rewrites what it can, then fails.+ writePoolPage("preferences", "a", "topic-one", "the first rule", "see also [[pref-b]]");+ writePoolPage("preferences", "b", "topic-one", "the first rule");+ mkdirSync(join(vault, "Brain", "notes"), { recursive: true });+ writeFileSync(join(vault, "Brain", "notes", "note.md"), "see [[pref-b]] too\n", "utf8");++ const report = findDuplicateCandidates(vault);+ const notesDir = join(vault, "Brain", "notes");+ chmodSync(notesDir, 0o555);+ try {+ const applied = applyDedupCandidates(vault, report.candidates);++ expect(applied.failures).toHaveLength(1);+ expect(applied.failures[0]!.key).toBe(report.candidates[0]!.key);+ expect(applied.failures[0]!.message).toContain("could not write");+ // The honest count: the link pass rewrote the citing page it could+ // before it hit the one it could not.+ expect(applied.failures[0]!.wikilinksUpdated).toBe(1);+ // The merged half DID land before the failure - the failure report+ // must not leave the reader assuming the secondary is untouched.+ expect(readFileSync(join(vault, "Brain", "preferences", "pref-b.md"), "utf8")).toContain(+ "merged_into: pref-a",+ );+ } finally {+ chmodSync(notesDir, 0o755);+ }+ });+});++describe("cmdBrainPageDedup --apply with a failing cluster", () => {+ test("the run completes, lists the failure, and exits 1", async () => {+ if (CHMOD_CANNOT_DENY) return;+ writePoolPage("preferences", "a", "topic-one", "the first rule", "see also [[pref-b]]");+ writePoolPage("preferences", "b", "topic-one", "the first rule");+ writePoolPage("retired", "c", "topic-two", "the second rule", "see also [[ret-d]]");+ writePoolPage("retired", "d", "topic-two", "the second rule");++ // The writes are atomic (temp file + rename into place), so the denial+ // that blocks a merge is the DIRECTORY write bit: a read-only+ // Brain/preferences fails exactly the first cluster while Brain/retired+ // stays writable.+ const prefsDir = join(vault, "Brain", "preferences");+ chmodSync(prefsDir, 0o555);+ try {+ const exit = await captureVerb(["--vault", vault, "--apply", "--yes", "--json"]);+ expect(exit.code).toBe(1);+ const payload = JSON.parse(exit.stdout) as {+ merged: number;+ failures: Array<{ key: string; message: string }>;+ };+ expect(payload.failures.length).toBe(1);+ expect(payload.failures[0]!.message).toContain("EACCES");+ expect(payload.merged).toBe(1);+ // The later cluster still applied, in its writable directory.+ expect(readFileSync(join(vault, "Brain", "retired", "ret-d.md"), "utf8")).toContain(+ "merged_into: ret-c",+ );+ // The blocked cluster is untouched.+ expect(readFileSync(join(vault, "Brain", "preferences", "pref-b.md"), "utf8")).not.toContain(+ "merged_into",+ );+ } finally {+ chmodSync(prefsDir, 0o755);+ }+ });+});++/** Run the verb in-process, capturing stdout and the returned exit code. */+async function captureVerb(argv: string[]): Promise<{ code: number; stdout: string }> {+ const chunks: string[] = [];+ const originalWrite = process.stdout.write.bind(process.stdout);+ process.stdout.write = ((chunk: string | Uint8Array) => {+ chunks.push(typeof chunk === "string" ? chunk : Buffer.from(chunk).toString("utf8"));+ return true;+ }) as typeof process.stdout.write;+ try {+ const code = await cmdBrainPageDedup(argv);+ return { code, stdout: chunks.join("") };+ } finally {+ process.stdout.write = originalWrite;+ }+}diff --git a/tests/cli/brain/verbs/receipts.test.ts b/tests/cli/brain/verbs/receipts.test.tsnew file mode 100644index 00000000..0cebca0a--- /dev/null+++ b/tests/cli/brain/verbs/receipts.test.ts@@ -0,0 +1,208 @@+/**+ * `o2b brain receipts` at the CLI level (storage-lifecycle, Lane 3+ * task 3.4).+ *+ * Scan is the default and writes nothing. Compact previews unless+ * `--apply` is passed, never runs without explicit `--shard family:name`+ * ids from a scan, and honors the count guard before any write. The+ * parity half pins the surface vocabulary to the shared receipts modes:+ * the usage line and the brain_receipts input schema derive from the+ * same closed tuple, so the two surfaces cannot drift.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ continuityLogDir,+ listContinuityRecords,+} from "../../../../src/core/brain/continuity/store.ts";+import { metricsDir } from "../../../../src/core/brain/metrics.ts";+import { bootstrapBrain } from "../../../../src/core/brain/init.ts";+import { resetVaultIdentityPins } from "../../../../src/core/brain/vault-identity.ts";+import {+ RECEIPT_MODES,+ RECEIPT_COMPACTION_OPERATION,+} from "../../../../src/core/brain/receipts/types.ts";+import { RECEIPTS_USAGE } from "../../../../src/cli/brain/verbs/receipts.ts";+import { RECEIPT_TOOLS } from "../../../../src/mcp/brain/receipt-tools.ts";+import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";+import { runCli } from "../../../helpers/run-cli.ts";+import { digestVaultTree } from "../../../helpers/vault-digest.ts";++/** The ledger month the fixtures write; safely past any fold cutoff. */+const LEDGER_MONTH = "2026-08";++let vault: string;+let configHome: string;++function continuityLine(kind: string, createdAt: string): string {+ const id = `ctn_${createdAt.replace(/[^0-9]/g, "").slice(0, 14)}_${kind}`;+ return `${JSON.stringify({+ schema: "o2b.continuity.v1",+ id,+ kind,+ createdAt,+ sourceRefs: [],+ payload: {},+ private: false,+ redacted: false,+ })}\n`;+}++function seedLedger(): void {+ const lines: string[] = [];+ for (let i = 0; i < 5; i += 1) {+ lines.push(continuityLine("recall_telemetry", `${LEDGER_MONTH}-0${i + 1}T12:00:0${i}Z`));+ }+ mkdirSync(continuityLogDir(vault), { recursive: true });+ mkdirSync(metricsDir(vault), { recursive: true });+ writeFileSync(join(continuityLogDir(vault), `${LEDGER_MONTH}.jsonl`), lines.join(""), "utf8");+}++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-cli-receipts-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-cli-receipts-cfg-"));+ atomicWriteFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+});++afterEach(() => {+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++function env(): Record<string, string> {+ return { OPEN_SECOND_BRAIN_CONFIG: join(configHome, "config.yaml") };+}++describe("o2b brain receipts", () => {+ test("scan is the default mode and writes nothing", async () => {+ seedLedger();+ const before = digestVaultTree(vault);+ const bare = await runCli(["brain", "receipts", "--json"], { env: env() });+ expect(bare.returncode).toBe(0);+ const report = JSON.parse(bare.stdout) as {+ operation: string;+ totals: { foldable: number; records: number };+ shards: Array<{ family: string; name: string; foldable: number }>;+ };+ expect(report.operation).toBe(RECEIPT_COMPACTION_OPERATION);+ expect(report.totals.records).toBe(5);+ expect(report.totals.foldable).toBe(5);+ expect(report.shards[0]!.name).toBe("2026-08.jsonl");+ expect(digestVaultTree(vault)).toBe(before);+ // The named scan mode behaves identically.+ const named = await runCli(["brain", "receipts", "scan", "--json"], { env: env() });+ expect(named.returncode).toBe(0);+ expect(JSON.parse(named.stdout).totals.foldable).toBe(5);+ });++ test("compact previews by default and writes nothing", async () => {+ seedLedger();+ const before = digestVaultTree(vault);+ const out = await runCli(+ ["brain", "receipts", "compact", "--shard", `continuity:${LEDGER_MONTH}.jsonl`, "--json"],+ { env: env() },+ );+ expect(out.returncode).toBe(0);+ const result = JSON.parse(out.stdout) as {+ dry_run: boolean;+ totals: { folded: number };+ };+ expect(result.dry_run).toBe(true);+ expect(result.totals.folded).toBe(5);+ expect(digestVaultTree(vault)).toBe(before);+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);+ });++ test("compact --apply folds behind the count guard and appends one audit record", async () => {+ seedLedger();+ const out = await runCli(+ [+ "brain",+ "receipts",+ "compact",+ "--shard",+ `continuity:${LEDGER_MONTH}.jsonl`,+ "--apply",+ "--expect",+ "5",+ "--json",+ ],+ { env: env() },+ );+ expect(out.returncode).toBe(0);+ const result = JSON.parse(out.stdout) as {+ dry_run: boolean;+ totals: { folded: number };+ audit_record_id: string;+ snapshot_run_id: string;+ };+ expect(result.dry_run).toBe(false);+ expect(result.totals.folded).toBe(5);+ expect(result.snapshot_run_id).toBeDefined();+ const audits = listContinuityRecords(vault, { kind: "receipt_compaction" });+ expect(audits).toHaveLength(1);+ expect(audits[0]!.id).toBe(result.audit_record_id);+ });++ test("a count-guard mismatch aborts with the guard message and writes nothing", async () => {+ seedLedger();+ const before = digestVaultTree(vault);+ const out = await runCli(+ [+ "brain",+ "receipts",+ "compact",+ "--shard",+ `continuity:${LEDGER_MONTH}.jsonl`,+ "--apply",+ "--expect",+ "4",+ ],+ { env: env() },+ );+ expect(out.returncode).toBe(1);+ expect(out.stderr).toContain("--expect 4");+ expect(digestVaultTree(vault)).toBe(before);+ });++ test("compact without shard ids and a malformed shard ref are usage failures", async () => {+ const bare = await runCli(["brain", "receipts", "compact"], { env: env() });+ expect(bare.returncode).toBe(1);+ expect(bare.stderr).toContain("--shard");+ const malformed = await runCli(+ ["brain", "receipts", "compact", "--shard", "continuity:../../escape.jsonl"],+ { env: env() },+ );+ expect(malformed.returncode).toBe(1);+ expect(malformed.stderr).toContain("is not a continuity shard");+ const unknownFamily = await runCli(+ ["brain", "receipts", "compact", "--shard", "scratch:x.jsonl"],+ { env: env() },+ );+ expect(unknownFamily.returncode).toBe(1);+ expect(unknownFamily.stderr).toContain("unknown ledger family");+ });++ test("the surface vocabulary derives from one source on both surfaces", () => {+ // The verb's usage line and the tool's mode enum come from the same+ // closed tuple, so neither surface can rename a mode alone.+ for (const mode of RECEIPT_MODES) {+ expect(RECEIPTS_USAGE).toContain(mode);+ }+ const tool = RECEIPT_TOOLS[0]!;+ const schemaModes = (+ tool.inputSchema.properties as {+ mode: { enum: string[] };+ }+ ).mode.enum;+ expect(schemaModes).toEqual([...RECEIPT_MODES]);+ expect(tool.name).toBe("brain_receipts");+ });+});diff --git a/tests/cli/brain/verbs/storage.test.ts b/tests/cli/brain/verbs/storage.test.tsnew file mode 100644index 00000000..cfb61f24--- /dev/null+++ b/tests/cli/brain/verbs/storage.test.ts@@ -0,0 +1,281 @@+/**+ * `o2b brain storage` (storage-lifecycle, Lane 1 task 1.7).+ *+ * Claims pinned here:+ *+ * 1. The report carries the per-surface census, the forecast and the+ * usage sample this invocation records; an uncapped vault answers+ * tier `unbounded` plainly - bytes are reported, no days-to-cap is+ * invented, nothing refuses.+ * 2. Each invocation appends exactly one `storage_usage` sample.+ * 3. A cap positions the tier, and the stored trend travels into+ * `slope_bytes_per_day` and `days_to_cap` beside the fresh census.+ * 4. An unreadable surface is refused by name (`unmeasured` with its+ * reason), never reported as zero bytes, and the forecast answers+ * `unmeasurable` rather than projecting from a partial census.+ * 5. A vault with no brain config fails by name (exit 1), it does not+ * answer as an empty vault.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { cmdBrainStorage } from "../../../../src/cli/brain/verbs/storage.ts";+import { bootstrapBrain } from "../../../../src/core/brain/init.ts";+import { resetVaultIdentityPins } from "../../../../src/core/brain/vault-identity.ts";+import { brainConfigPath } from "../../../../src/core/brain/paths.ts";+import {+ metricsDir,+ listMetrics,+ appendMetric,+ type MetricRecord,+} from "../../../../src/core/brain/metrics.ts";+import {+ STORAGE_USAGE_METRIC_SURFACE,+ STORAGE_FORECAST_TIER,+} from "../../../../src/core/brain/storage/forecast.ts";+import { measureStorageUsage } from "../../../../src/core/brain/storage/usage.ts";+import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";+import { runCli } from "../../../helpers/run-cli.ts";+import { CHMOD_CANNOT_DENY } from "../../../helpers/platform.ts";++let vault: string;+let configHome: string;+let savedConfigEnv: string | undefined;++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-cli-storage-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-cli-storage-cfg-"));+ atomicWriteFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+ savedConfigEnv = process.env["OPEN_SECOND_BRAIN_CONFIG"];+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = join(configHome, "config.yaml");+});++afterEach(() => {+ if (savedConfigEnv === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfigEnv;+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++/** The report shape an invocation prints, as the tests read it. */+interface StorageReportJson {+ measured: boolean;+ usage: {+ measuredAt: string;+ complete: boolean;+ surfaces: ReadonlyArray<{ surface: string; tier: string; bytes: number }>;+ bytesByTier: Record<string, number>;+ totalBytes: number;+ unmeasured: ReadonlyArray<{ surface: string; reason: string }>;+ };+ forecast: {+ tier: string;+ measuredBytes: number | null;+ capBytes: number | null;+ percentUsed: number | null;+ slopeBytesPerDay: number | null;+ daysToCap: number | null;+ unmeasured: ReadonlyArray<{ surface: string; reason: string }>;+ samplesUnreadable: boolean;+ };+ sample: { runId: string; appended: boolean; appendError?: string };+}++function writePage(name: string, totalBytes: number): void {+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", name), "x".repeat(totalBytes), "utf8");+}++function configureCap(capBytes: number): void {+ writeFileSync(+ brainConfigPath(vault),+ `schema_version: 1\n\nstorage:\n cap_bytes: ${capBytes}\n`,+ "utf8",+ );+}++/** Seed one `storage_usage` sample at a fixed time and total. */+function seedSample(runAt: string, totalBytes: number, runId: string): void {+ mkdirSync(metricsDir(vault), { recursive: true });+ appendMetric(vault, {+ surface: STORAGE_USAGE_METRIC_SURFACE,+ runAt,+ payload: { run_id: runId, total_bytes: totalBytes, measured: true, unmeasured: [] },+ });+}++/** The usage samples recorded for this vault so far. */+function usageSamples(): MetricRecord[] {+ return listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE });+}++// ---- captured-output driving ----------------------------------------------++/** Everything the verb wrote to stdout and stderr during `drive`. */+interface Captured {+ readonly stdout: string;+ readonly stderr: string;+}++/**+ * Run the verb in-process with both output streams captured. The stream+ * methods are restored before this returns, even when the verb throws.+ */+async function drive(argv: ReadonlyArray<string>): Promise<{ captured: Captured; code: number }> {+ let out = "";+ let err = "";+ const originalOut = process.stdout.write;+ const originalErr = process.stderr.write;+ const append =+ (sink: (chunk: string) => void) =>+ (chunk: Uint8Array | string): boolean => {+ sink(typeof chunk === "string" ? chunk : Buffer.from(chunk).toString("utf8"));+ return true;+ };+ process.stdout.write = append((c) => (out += c)) as typeof process.stdout.write;+ process.stderr.write = append((c) => (err += c)) as typeof process.stderr.write;+ try {+ // The captured strings are read only after the verb has run, so the+ // object is built after the await - building it inline would snapshot+ // the buffers while they are still empty.+ const code = await cmdBrainStorage([...argv]);+ return { captured: { stdout: out, stderr: err }, code };+ } finally {+ process.stdout.write = originalOut;+ process.stderr.write = originalErr;+ }+}++async function runJsonReport(): Promise<{ code: number; report: StorageReportJson }> {+ const { captured, code } = await drive(["--json"]);+ return { code, report: JSON.parse(captured.stdout) as StorageReportJson };+}++describe("o2b brain storage", () => {+ test("an uncapped vault reports the census and the unbounded tier plainly", async () => {+ writePage("AA-page.md", 50_000);+ // No writes happen between this measure and the verb's own census,+ // so the totals must agree exactly.+ const expected = measureStorageUsage(vault);++ const { code, report } = await runJsonReport();++ expect(code).toBe(0);+ expect(report.measured).toBe(true);+ expect(report.usage.totalBytes).toBe(expected.totalBytes);+ expect(report.usage.surfaces.map((s) => s.surface)).toEqual(+ expected.surfaces.map((s) => s.surface),+ );+ expect(report.forecast.tier).toBe(STORAGE_FORECAST_TIER.unbounded);+ expect(report.forecast.capBytes).toBeNull();+ expect(report.forecast.percentUsed).toBeNull();+ expect(report.forecast.daysToCap).toBeNull();+ expect(report.sample.appended).toBe(true);+ });++ test("each invocation appends exactly one usage sample", async () => {+ expect(usageSamples()).toHaveLength(0);+ await cmdBrainStorage(["--json"]);+ const afterFirst = usageSamples();+ expect(afterFirst).toHaveLength(1);+ await cmdBrainStorage(["--json"]);+ const afterSecond = usageSamples();+ expect(afterSecond).toHaveLength(2);+ expect(afterSecond[0]!.payload["run_id"]).not.toBe(afterSecond[1]!.payload["run_id"]);+ });++ test("a cap positions the tier and the stored trend travels", async () => {+ writePage("AA-page.md", 50_000);+ // Measure only after every fixture write: the cap rewrite replaces+ // the bootstrap config template, which would otherwise shift the+ // census between the measure and the verb's own.+ configureCap(1_000_000);+ const total = measureStorageUsage(vault).totalBytes;+ // Cap at 2.2x the stable total: the fresh reading sits around 45+ // percent (never near the 80/95 percent bands), so the tier comes+ // from the horizon. The two stored samples rise a tenth of the total+ // per day and the verb's own sample continues that rise, so the cap+ // lands inside the warn horizon - past critical's 7 days, inside+ // warn's 30.+ const cap = Math.floor(total * 2.2);+ configureCap(cap);+ seedSample("2026-10-14T00:00:00.000Z", total - total / 10, "seed-older");+ seedSample("2026-10-15T00:00:00.000Z", total, "seed-newer");++ const { code, report } = await runJsonReport();++ expect(code).toBe(0);+ const forecast = report.forecast;+ expect(forecast.tier).toBe(STORAGE_FORECAST_TIER.warn);+ expect(forecast.capBytes).toBe(cap);+ expect(forecast.measuredBytes).not.toBeNull();+ // The percent and the days answer from the fresh census; the seeded+ // samples and the report's own sample are the only bytes that moved,+ // so the numeric identities are asserted relationally.+ expect(Math.abs((forecast.percentUsed! * cap) / 100 - forecast.measuredBytes!)).toBeLessThan(1);+ expect(+ Math.abs(forecast.measuredBytes! + forecast.daysToCap! * forecast.slopeBytesPerDay! - cap),+ ).toBeLessThan(1);+ // The slope is the seeded trend, not sample noise.+ expect(forecast.slopeBytesPerDay!).toBeGreaterThan(total / 30);+ expect(forecast.slopeBytesPerDay!).toBeLessThan(total / 10);+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "an unreadable surface is refused by name, never reported as zero",+ async () => {+ writePage("AA-page.md", 9_000);+ configureCap(10_000);+ mkdirSync(join(vault, "Brain", ".artifacts"), { recursive: true });+ writeFileSync(join(vault, "Brain", ".artifacts", "tool-result.json"), "{}", "utf8");+ chmodSync(join(vault, "Brain", ".artifacts"), 0o000);+ try {+ const { code, report } = await runJsonReport();+ expect(code).toBe(0);+ expect(report.usage.complete).toBe(false);+ expect(report.usage.unmeasured).toEqual([+ { surface: "artifacts", reason: "surface_unreadable" },+ ]);+ expect(report.usage.surfaces.map((s) => s.surface)).not.toContain("artifacts");+ expect(report.forecast.tier).toBe(STORAGE_FORECAST_TIER.unmeasurable);+ expect(report.forecast.unmeasured.map((u) => u.surface)).toEqual(["artifacts"]);+ } finally {+ chmodSync(join(vault, "Brain", ".artifacts"), 0o755);+ }+ },+ );++ test("a vault with no brain config fails by name", async () => {+ const bare = mkdtempSync(join(tmpdir(), "o2b-cli-storage-bare-"));+ try {+ const { captured, code } = await drive(["--json", "--vault", bare]);+ expect(code).toBe(1);+ expect(captured.stderr).toContain("config file does not exist");+ } finally {+ rmSync(bare, { recursive: true, force: true });+ }+ });++ test("the text report names the unbounded tier and the sample", async () => {+ const { captured, code } = await drive([]);+ expect(code).toBe(0);+ expect(captured.stdout).toContain("unbounded");+ expect(captured.stdout).toContain("sample: appended as");+ });++ test("the dispatcher routes `brain storage` to the verb", async () => {+ const r = await runCli(["brain", "storage", "--json"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: join(configHome, "config.yaml") },+ });+ expect(r.returncode).toBe(0);+ const report = JSON.parse(r.stdout) as StorageReportJson;+ expect(report.measured).toBe(true);+ expect(report.forecast.tier).toBe(STORAGE_FORECAST_TIER.unbounded);+ });+});diff --git a/tests/core/architecture/claim-attribution-census.test.ts b/tests/core/architecture/claim-attribution-census.test.tsindex f82c2d00..ee966541 100644--- a/tests/core/architecture/claim-attribution-census.test.ts+++ b/tests/core/architecture/claim-attribution-census.test.ts@@ -107,6 +107,10 @@ const PINNED_CONTINUITY_KINDS: ReadonlyArray<string> = Object.freeze(["decision_model_call","decision_model_extract_commit","session_entity_mention",+ // The storage-lifecycle wave's additive kind: the per-run audit of a+ // confirmed receipt compaction. Disjoint from the Hermes+ // context-compaction kinds above by name and by producer.+ "receipt_compaction",]);function truthModulePaths(): ReadonlyArray<string> {diff --git a/tests/core/architecture/source-archive-stamp-census.test.ts b/tests/core/architecture/source-archive-stamp-census.test.tsindex 41b33d89..5c0038c5 100644--- a/tests/core/architecture/source-archive-stamp-census.test.ts+++ b/tests/core/architecture/source-archive-stamp-census.test.ts@@ -60,6 +60,7 @@ const STAMP_KEY_LITERAL = "source_archive_hash";* the key in the archive_get schema.*/const STAMP_KEY_FILES: ReadonlyArray<string> = Object.freeze([+ posix.join("core", "brain", "document-retire.ts"),posix.join("core", "brain", "hygiene", "detectors", "capture-scope.ts"),posix.join("core", "brain", "ingest", "ingest.ts"),posix.join("core", "brain", "provenance", "source-archive.ts"),diff --git a/tests/core/architecture/verdict-vocabulary-census.test.ts b/tests/core/architecture/verdict-vocabulary-census.test.tsindex 6ec68fac..05861a99 100644--- a/tests/core/architecture/verdict-vocabulary-census.test.ts+++ b/tests/core/architecture/verdict-vocabulary-census.test.ts@@ -520,6 +520,19 @@ import {SURFACING_TIERS,isSurfacingTier,} from "../../../src/core/search/surfacing-key.ts";+import {+ isStorageForecastTier,+ STORAGE_FORECAST_TIER,+ STORAGE_FORECAST_TIERS,+} from "../../../src/core/brain/storage/forecast.ts";+import {+ isReceiptLedgerFamily,+ isReceiptMode,+ RECEIPT_LEDGER_FAMILY,+ RECEIPT_LEDGER_FAMILIES,+ RECEIPT_MODE,+ RECEIPT_MODES,+} from "../../../src/core/brain/receipts/types.ts";interface VocabularyUnderCensus {/** Identifies the vocabulary in a failure message. */@@ -1616,6 +1629,45 @@ const CENSUS: ReadonlyArray<VocabularyUnderCensus> = Object.freeze([members: SURFACING_TIERS,guard: isSurfacingTier,},+ {+ // The storage-lifecycle wave. How full a capped vault will be, and+ // what the gate does about it. The three measured tiers order by+ // severity, and the three refusals are separate members because they+ // answer different questions a reader branches on: `unbounded` is a+ // fact about the vault's configuration, `unmeasurable` a fact about+ // the census that missed a surface, and `no_forecast` the named+ // two-truths state of too few stored samples - collapsing any pair+ // would turn "the vault has no cap" into "the sampler is broken".+ // The values ride the storage gate's verdict and the doctor's+ // capacity check, both rendered surfaces an operator reads.+ name: "STORAGE_FORECAST_TIER",+ values: STORAGE_FORECAST_TIER,+ members: STORAGE_FORECAST_TIERS,+ guard: isStorageForecastTier,+ },+ {+ // The storage-lifecycle wave. Which receipt ledger one compaction+ // surface addresses. Two members because there are exactly two+ // families and the fold rewrites one shard at a time under its own+ // lock; a free string would let a caller name a ledger this build+ // does not scan and read the run back as a clean no-op.+ name: "RECEIPT_LEDGER_FAMILY",+ values: RECEIPT_LEDGER_FAMILY,+ members: RECEIPT_LEDGER_FAMILIES,+ guard: isReceiptLedgerFamily,+ },+ {+ // The storage-lifecycle wave. What one receipt-compaction invocation+ // was asked to do. `scan` inventories and writes nothing, `compact`+ // rewrites under a snapshot; the pair is the whole input schema the+ // CLI verb and the MCP tool share, and one guard over both the flag+ // and the wire argument is what keeps the two surfaces unable to+ // disagree about what the second one does.+ name: "RECEIPT_MODE",+ values: RECEIPT_MODE,+ members: RECEIPT_MODES,+ guard: isReceiptMode,+ },]);// ---------------------------------------------------------------------------@@ -1883,7 +1935,10 @@ const SCANNED = scanVocabularies(SOURCE_TREE);// 99: Lane C's REPLAY_EXPECTATION (replay corpus) and SURFACING_TIER// (graded surfacing) join the scanned population, the same append Lane// C's ARCHIVE_VERIFICATION made.-const VOCABULARY_POPULATION = 99;+// 102: the storage-lifecycle wave's three - STORAGE_FORECAST_TIER (the+// capacity gate's tiers), RECEIPT_LEDGER_FAMILY and RECEIPT_MODE (the+// receipt-compaction surfaces' shared input vocabulary).+const VOCABULARY_POPULATION = 102;const REGISTERED = new Map(CENSUS.map((entry) => [entry.name, entry] as const));describe("verdict vocabulary census", () => {diff --git a/tests/core/architecture/write-site-census.test.ts b/tests/core/architecture/write-site-census.test.tsindex 12248147..cc22892b 100644--- a/tests/core/architecture/write-site-census.test.ts+++ b/tests/core/architecture/write-site-census.test.ts@@ -1413,8 +1413,15 @@ const DIRECT_WRITE_ROWS = 82;* joins the population through its new `path-constants.ts` import (the* quote_check resolvers) carrying the config write it already made* through the same shared writer.+ *+ * 119 -> 120, from the storage-lifecycle wave and nothing else: the+ * receipt fold engine (`src/core/brain/receipts/compact.ts`) rewrites+ * each folded receipt-ledger shard through `atomicWriteFileSync` with+ * the `expectBefore` compare-and-swap, under the shard's own lock - a+ * machine JSONL rewrite that carries no note content, so it arrives in+ * the shared class needing no exclusion.*/-const SHARED_HELPER_ROWS = 119;+const SHARED_HELPER_ROWS = 120;// ----- Origin-channel coverage boundary (Unit C) ----------------------------@@ -1527,9 +1534,12 @@ const UNSTAMPED_DIRECT_ROWS = 81;* through the shared atomic writer (evidence bytes; no origin stamp on an* archive put), and the config writer joins the population through its new* path-constants import carrying the config write it already made (module- * state, not notes).+ * state, not notes). 116 -> 117: the receipt fold engine+ * (`src/core/brain/receipts/compact.ts`) rewrites its named shards through+ * the shared atomic writer (machine JSONL ledger rows, not notes; no+ * origin stamp on a fold rewrite).*/-const UNSTAMPED_SHARED_ROWS = 116;+const UNSTAMPED_SHARED_ROWS = 117;describe("in-vault write-site census", () => {test("every direct-fs write site carries a written exclusion", () => {diff --git a/tests/core/brain/applier-capability.test.ts b/tests/core/brain/applier-capability.test.tsindex f14fc83f..80315f4a 100644--- a/tests/core/brain/applier-capability.test.ts+++ b/tests/core/brain/applier-capability.test.ts@@ -35,7 +35,10 @@ import {} from "../../../src/core/brain/diagnostics.ts";import { LINT_CONSOLIDATE_KIND } from "../../../src/core/brain/lint-consolidate.ts";import { applyHygienePlan } from "../../../src/core/brain/hygiene/apply.ts";-import { buildHygienePlan } from "../../../src/core/brain/hygiene/plan.ts";+import {+ UNATTENDED_EXCLUDED_ACTIONS,+ buildHygienePlan,+} from "../../../src/core/brain/hygiene/plan.ts";import {HYGIENE_PROPOSED_ACTIONS,type HygieneFinding,@@ -137,18 +140,34 @@ describe("applier capability table - cannot disagree with the fixers", () => {test("the hygiene plan executes exactly the actions the table calls mechanical", () => {const mechanical: string[] = [];+ const heldForAttendance: string[] = [];const refused: string[] = [];for (const action of HYGIENE_PROPOSED_ACTIONS) {- const plan = buildHygienePlan(reportWithAction(action));- if (plan.selected.length === 1) mechanical.push(action);- else if (plan.excluded_review.length === 1) refused.push(action);- else throw new Error(`hygiene action '${action}' was neither selected nor excluded`);+ const unattended = buildHygienePlan(reportWithAction(action));+ if (unattended.selected.length === 1) {+ mechanical.push(action);+ continue;+ }+ if (unattended.excluded_review.length !== 1) {+ throw new Error(`hygiene action '${action}' was neither selected nor excluded`);+ }+ if (requireRepairCapability(action).verdict === REPAIR_VERDICT.refused) {+ refused.push(action);+ continue;+ }+ // A mechanical action the unattended default holds back is the+ // destructive class: it must still select when the caller names it+ // by id - that is the whole attended path.+ const attended = buildHygienePlan(reportWithAction(action), { ids: [`probe:${action}`] });+ expect(attended.selected.map((finding) => finding.id)).toEqual([`probe:${action}`]);+ heldForAttendance.push(action);}expect(mechanical.toSorted()).toEqual(- codesForSurface(REPAIR_SURFACE.hygieneApply).filter(- (code) => APPLIER_CAPABILITY.get(code)!.verdict === REPAIR_VERDICT.mechanical,- ),+ codesForSurface(REPAIR_SURFACE.hygieneApply)+ .filter((code) => APPLIER_CAPABILITY.get(code)!.verdict === REPAIR_VERDICT.mechanical)+ .filter((code) => !UNATTENDED_EXCLUDED_ACTIONS.has(code as HygieneProposedAction)),);+ expect(heldForAttendance).toEqual(["retire"]);expect(refused).toEqual(["review"]);});diff --git a/tests/core/brain/doctor-exit-census.test.ts b/tests/core/brain/doctor-exit-census.test.tsindex 699f5993..236ecf65 100644--- a/tests/core/brain/doctor-exit-census.test.ts+++ b/tests/core/brain/doctor-exit-census.test.ts@@ -200,12 +200,15 @@ const DOCTOR_REGISTERED_CODES: ReadonlyArray<string> = ["permissions-unreadable","principle-corrupted","recall-channel-silent",+ "receipt-ledger-growth","recovery-point-stale","rerank-endpoint-unconfigured","rerank-model-sunset-announced","schema-version-unknown","stale-claim","stale-dependency",+ "storage-capacity-critical",+ "storage-capacity-warn","sync-conflict-log","tier-drift","topic-key-collision",diff --git a/tests/core/brain/doctor-receipt-ledger.test.ts b/tests/core/brain/doctor-receipt-ledger.test.tsnew file mode 100644index 00000000..5cfc9818--- /dev/null+++ b/tests/core/brain/doctor-receipt-ledger.test.ts@@ -0,0 +1,192 @@+/**+ * Receipt-ledger growth doctor check (storage-lifecycle, Lane 3 task 3.3).+ *+ * A receipt ledger that keeps growing because nothing ever folded it is+ * a storage condition the operator can act on, and the act is the+ * receipts scan: the check reports a warning once a family's foldable+ * record count crosses the growth threshold, naming the command, and+ * stays silent when the ledger is compacted or young. A ledger the scan+ * cannot read is not a zero - it reaches the uncertain stream under its+ * own code, like every other measurement the doctor could not take.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { continuityLogDir } from "../../../src/core/brain/continuity/store.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";+import { metricsDir } from "../../../src/core/brain/metrics.ts";+import {+ RECEIPT_LEDGER_GROWTH_CODE,+ RECEIPT_LEDGER_GROWTH_MIN_FOLDABLE,+ RECEIPT_LEDGER_UNMEASURED_CODE,+} from "../../../src/core/brain/doctor/receipt-ledger.ts";+import { runDoctor } from "../../../src/core/brain/doctor.ts";+import { bootstrapBrain } from "../../../src/core/brain/init.ts";+import { compactReceiptLedgers } from "../../../src/core/brain/receipts/compact.ts";+import { RECEIPT_LEDGER_FAMILY } from "../../../src/core/brain/receipts/types.ts";+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";++const NOW = new Date("2026-09-25T00:00:00Z");++let vault: string;+let configHome: string;++function continuityLine(kind: string, createdAt: string): string {+ const id = `ctn_${createdAt.replace(/[^0-9]/g, "").slice(0, 14)}_${kind}`;+ return `${JSON.stringify({+ schema: "o2b.continuity.v1",+ id,+ kind,+ createdAt,+ sourceRefs: [],+ payload: {},+ private: false,+ redacted: false,+ })}\n`;+}++function receiptIssues(): Array<{ code: string; message: string; path?: string }> {+ const doctor = runDoctor(vault, { now: NOW });+ return [...doctor.errors, ...doctor.warnings].filter(+ (issue) => issue.code === RECEIPT_LEDGER_GROWTH_CODE,+ );+}++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-doctor-receipt-ledger-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-doctor-receipt-ledger-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath });+ mkdirSync(continuityLogDir(vault), { recursive: true });+ mkdirSync(metricsDir(vault), { recursive: true });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++describe("receiptLedgerCheck", () => {+ test("a ledger above the growth threshold yields one warning naming the receipts command", () => {+ const lines: string[] = [];+ for (let i = 0; i < RECEIPT_LEDGER_GROWTH_MIN_FOLDABLE + 100; i += 1) {+ const minute = (i % 60).toString().padStart(2, "0");+ lines.push(+ continuityLine(+ "recall_telemetry",+ `2026-08-01T12:${minute}:${(i % 60).toString().padStart(2, "0")}Z`,+ ),+ );+ }+ writeFileSync(join(continuityLogDir(vault), "2026-08.jsonl"), lines.join(""), "utf8");+ const issues = receiptIssues();+ expect(issues).toHaveLength(1);+ expect(issues[0]!.code).toBe(RECEIPT_LEDGER_GROWTH_CODE);+ expect(issues[0]!.message).toContain("o2b brain receipts scan");+ expect(issues[0]!.message).toContain(`${RECEIPT_LEDGER_GROWTH_MIN_FOLDABLE + 100}`);+ });++ test("a compacted ledger yields no issue", () => {+ const lines: string[] = [];+ for (let i = 0; i < RECEIPT_LEDGER_GROWTH_MIN_FOLDABLE + 100; i += 1) {+ const minute = (i % 60).toString().padStart(2, "0");+ lines.push(+ continuityLine(+ "recall_telemetry",+ `2026-08-01T12:${minute}:${(i % 60).toString().padStart(2, "0")}Z`,+ ),+ );+ }+ writeFileSync(join(continuityLogDir(vault), "2026-08.jsonl"), lines.join(""), "utf8");+ compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(receiptIssues()).toEqual([]);+ });++ test("a ledger below the threshold yields no issue", () => {+ writeFileSync(+ join(metricsDir(vault), "storage_usage.jsonl"),+ `${JSON.stringify({+ schema: "o2b.metrics.v1",+ surface: "storage_usage",+ run_at: "2026-08-01T00:00:00Z",+ payload: {},+ })}\n`,+ "utf8",+ );+ expect(receiptIssues()).toEqual([]);+ });++ test("a ledger the scan cannot read reaches the uncertain stream, never a zero", () => {+ // A regular file where the continuity ledger directory belongs: the+ // scan cannot enumerate it.+ rmSync(continuityLogDir(vault), { recursive: true, force: true });+ writeFileSync(continuityLogDir(vault), "not a directory\n", "utf8");+ const doctor = runDoctor(vault, { now: NOW });+ const uncertain = doctor.uncertain ?? [];+ const entry = uncertain.find((e) => e.code === RECEIPT_LEDGER_UNMEASURED_CODE);+ expect(entry).toBeDefined();+ // The scan's refusal is surfaced named, not as the raw filesystem+ // error: no errno verb, no absolute host path on the doctor payload.+ expect(entry!.message).toContain("could not be scanned");+ expect(entry!.message).not.toContain("ENOTDIR");+ expect(entry!.message).not.toContain(vault);+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "every unreadable shard is reported, each under its own path",+ () => {+ // The stream merges entries on (code, path): without the shard+ // identity on the entry, two unreadable shards collapsed into one+ // anonymous line and the operator could not tell which ledger to+ // look at, let alone how many shards were dark.+ const continuity = join(continuityLogDir(vault), "2026-08.jsonl");+ const metrics = join(metricsDir(vault), "storage_usage.jsonl");+ writeFileSync(continuity, continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"));+ writeFileSync(+ metrics,+ `${JSON.stringify({+ schema: "o2b.metrics.v1",+ surface: "storage_usage",+ run_at: "2026-08-01T00:00:00Z",+ payload: {},+ })}\n`,+ );+ chmodSync(continuity, 0o000);+ chmodSync(metrics, 0o000);+ try {+ const doctor = runDoctor(vault, { now: NOW });+ const unmeasured = (doctor.uncertain ?? []).filter(+ (entry) => entry.code === RECEIPT_LEDGER_UNMEASURED_CODE,+ );+ expect(unmeasured).toHaveLength(2);+ expect(unmeasured.map((entry) => entry.path).toSorted()).toEqual([+ "Brain/log/continuity/2026-08.jsonl",+ "Brain/metrics/storage_usage.jsonl",+ ]);+ for (const entry of unmeasured) {+ expect(entry.message).toContain("foldable count is unknown");+ }+ } finally {+ chmodSync(continuity, 0o644);+ chmodSync(metrics, 0o644);+ }+ },+ );++ test("both codes are classified: growth registered, unmeasured excused with a reason", async () => {+ const { DIAGNOSTIC_SIGNALS } = await import("../../../src/core/brain/diagnostics.ts");+ const { DOCTOR_EXIT_EXCLUSIONS } = await import("../../../src/core/brain/doctor-exits.ts");+ expect(DIAGNOSTIC_SIGNALS.get(RECEIPT_LEDGER_GROWTH_CODE)?.nextCommand).toBe(+ "o2b brain receipts scan",+ );+ expect(DOCTOR_EXIT_EXCLUSIONS.get(RECEIPT_LEDGER_UNMEASURED_CODE)).toBeDefined();+ });+});diff --git a/tests/core/brain/doctor-storage-capacity.test.ts b/tests/core/brain/doctor-storage-capacity.test.tsnew file mode 100644index 00000000..7f8a4ac8--- /dev/null+++ b/tests/core/brain/doctor-storage-capacity.test.ts@@ -0,0 +1,168 @@+/**+ * The storage-capacity doctor check.+ *+ * Claims pinned here:+ *+ * 1. With a cap configured, a vault measured inside the warn band+ * emits a warning issue and a vault past the critical band emits an+ * error issue, both under registered codes; without a cap the check+ * is inert - no cap, nothing to report.+ * 2. An unmeasurable store lands an `uncertain` entry naming the+ * refused surface, never a fake all-clear.+ * 3. The store is measured at `ctx.dbPath` when the pass carries one,+ * fail-soft: the doctor's resolved store path is the one that gets+ * measured, and an absent one falls back to the default resolution.+ * 4. Below local reach the reading is not reported at all: the check+ * answers in the uncertain stream, by name, and neither the warn+ * nor the critical issue - either would carry the census in its+ * message - is emitted. Absent reach reads as local, so the shell+ * caller's report is unchanged.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { runDoctor } from "../../../src/core/brain/doctor.ts";+import {+ STORAGE_CAPACITY_CRITICAL_CODE,+ STORAGE_CAPACITY_UNMEASURED_CODE,+ STORAGE_CAPACITY_WARN_CODE,+} from "../../../src/core/brain/doctor/storage-capacity.ts";+import { DIAGNOSTIC_SIGNALS } from "../../../src/core/brain/diagnostics.ts";+import { brainConfigPath } from "../../../src/core/brain/paths.ts";+import { TRANSPORT_REACH } from "../../../src/core/graph/transport-reach.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-doctor-storage-"));+ // Minimal Brain skeleton, the doctor-drift pattern.+ mkdirSync(join(vault, "Brain", "preferences"), { recursive: true });+ mkdirSync(join(vault, "Brain", "log"), { recursive: true });+ writeFileSync(brainConfigPath(vault), "schema_version: 1\n", "utf8");+ writeFileSync(join(vault, "Brain", "_BRAIN.md"), "---\ntitle: Brain\n---\n", "utf8");+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++function configureCap(capBytes: number): void {+ writeFileSync(+ brainConfigPath(vault),+ `schema_version: 1\n\nstorage:\n cap_bytes: ${capBytes}\n`,+ "utf8",+ );+}++function writePage(totalBytes: number): void {+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", "AA-page.md"), "x".repeat(totalBytes), "utf8");+}++describe("storageCapacityCheck", () => {+ test("an unbounded vault is silent - no cap, nothing to report", () => {+ writePage(50_000);+ const result = runDoctor(vault);+ expect(result.warnings.filter((i) => i.code === STORAGE_CAPACITY_WARN_CODE)).toEqual([]);+ expect(result.errors.filter((i) => i.code === STORAGE_CAPACITY_CRITICAL_CODE)).toEqual([]);+ });++ test("usage inside the warn band is silent", () => {+ configureCap(1_000_000);+ writePage(10_000);+ const result = runDoctor(vault);+ expect(result.warnings.filter((i) => i.code === STORAGE_CAPACITY_WARN_CODE)).toEqual([]);+ expect(result.errors.filter((i) => i.code === STORAGE_CAPACITY_CRITICAL_CODE)).toEqual([]);+ });++ test("usage past the warn band warns", () => {+ configureCap(10_000);+ writePage(9_000);+ const result = runDoctor(vault);+ const warns = result.warnings.filter((i) => i.code === STORAGE_CAPACITY_WARN_CODE);+ expect(warns).toHaveLength(1);+ expect(warns[0]!.message).toContain("storage");+ });++ test("usage past the critical band is an error", () => {+ configureCap(10_000);+ writePage(20_000);+ const result = runDoctor(vault);+ const errors = result.errors.filter((i) => i.code === STORAGE_CAPACITY_CRITICAL_CODE);+ expect(errors).toHaveLength(1);+ // The measured fact travels in the message.+ expect(errors[0]!.message).toMatch(/\d+ bytes/);+ });++ test("below local reach the reading is not reported - the reach is named instead", () => {+ configureCap(10_000);+ writePage(20_000);+ const result = runDoctor(vault, { reach: TRANSPORT_REACH.remote });+ // Neither tier issue is emitted: each would carry the whole-vault+ // census in its message prose.+ expect(result.errors.filter((i) => i.code === STORAGE_CAPACITY_CRITICAL_CODE)).toEqual([]);+ expect(result.warnings.filter((i) => i.code === STORAGE_CAPACITY_WARN_CODE)).toEqual([]);+ const unmeasured = (result.uncertain ?? []).filter(+ (e) => e.code === STORAGE_CAPACITY_UNMEASURED_CODE,+ );+ expect(unmeasured).toHaveLength(1);+ expect(unmeasured[0]!.message).toContain("reach");+ expect(unmeasured[0]!.message).not.toMatch(/\d/);+ });++ test("at local reach - named or absent - the reading is reported as before", () => {+ configureCap(10_000);+ writePage(20_000);+ for (const reach of [TRANSPORT_REACH.local, undefined]) {+ const result = runDoctor(vault, reach === undefined ? {} : { reach });+ expect(result.errors.some((i) => i.code === STORAGE_CAPACITY_CRITICAL_CODE)).toBe(true);+ }+ });++ test("both codes resolve to a registered next command", () => {+ for (const code of [STORAGE_CAPACITY_WARN_CODE, STORAGE_CAPACITY_CRITICAL_CODE]) {+ const signal = DIAGNOSTIC_SIGNALS.get(code);+ expect(signal).toBeDefined();+ expect(signal!.nextCommand.startsWith("o2b ")).toBe(true);+ expect(signal!.autoRepairable).toBe(false);+ }+ });++ test("the store is measured at ctx.dbPath when the pass carries one", () => {+ configureCap(10_000);+ // The pass names a 20 KB store outside the Brain tree; measured+ // usage crosses the critical band only if that path was used.+ const dbPath = join(vault, "elsewhere.sqlite");+ writeFileSync(dbPath, Buffer.alloc(20_000, 0x78), "utf8");+ const result = runDoctor(vault, { dbPath });+ expect(+ result.errors.some((i) => i.code === STORAGE_CAPACITY_CRITICAL_CODE),+ "the named store should be the one measured",+ ).toBe(true);+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "an unmeasurable store lands an uncertain entry, not an all-clear",+ () => {+ configureCap(10_000);+ writePage(9_000);+ mkdirSync(join(vault, "Brain", ".artifacts"), { recursive: true });+ chmodSync(join(vault, "Brain", ".artifacts"), 0o000);+ try {+ const result = runDoctor(vault);+ expect(result.warnings.filter((i) => i.code === STORAGE_CAPACITY_WARN_CODE)).toEqual([]);+ const unmeasured = (result.uncertain ?? []).filter(+ (e) => e.code === STORAGE_CAPACITY_UNMEASURED_CODE,+ );+ expect(unmeasured).toHaveLength(1);+ expect(unmeasured[0]!.message).toContain("artifacts");+ } finally {+ chmodSync(join(vault, "Brain", ".artifacts"), 0o755);+ }+ },+ );+});diff --git a/tests/core/brain/document-dedup-detector.test.ts b/tests/core/brain/document-dedup-detector.test.tsnew file mode 100644index 00000000..6722267d--- /dev/null+++ b/tests/core/brain/document-dedup-detector.test.ts@@ -0,0 +1,420 @@+/**+ * Document-dedup detector (storage lifecycle, Lane 4 task 4.1).+ *+ * Ingest summary pages (kind `brain-source`) that carry near-identical+ * bodies are retirement candidates: one canonical page should survive and+ * the stale copies should be retired with their records. The detector+ * only nominates groups; retirement happens through the hygiene apply+ * plan (action `retire`), which routes to the retirement core.+ *+ * Decisions pinned here:+ * - canonical is the NEWEST `created_at` (keep-newest);+ * - a group holding ANY member without `created_at` is ineligible -+ * the mtime fallback is merge-unstable (GitHub #180), so such a+ * group is reported for review, never for retirement;+ * - merge-resolved members leave the pool (the shared predicate);+ * - scope keys keep folding: identical text in two scopes never groups.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import type { EmbeddingProvider } from "../../../src/core/search/embeddings/contract.ts";+import { DEFAULT_SCAN_IDS, HYGIENE_DETECTOR_IDS } from "../../../src/core/brain/hygiene/types.ts";+import { applyHygienePlan } from "../../../src/core/brain/hygiene/apply.ts";+import { buildHygienePlan } from "../../../src/core/brain/hygiene/plan.ts";+import { runHygieneScan } from "../../../src/core/brain/hygiene/scan.ts";+import {+ DOCUMENT_DEDUP_LEXICAL_THRESHOLD,+ detectDocumentDedup,+ detectSemanticDocumentDedup,+} from "../../../src/core/brain/hygiene/detectors/document-dedup.ts";+import { hygieneFindingId } from "../../../src/core/brain/hygiene/detectors/id.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-document-dedup-"));+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++const NEAR_A =+ "Collect the latency metrics before optimizing the render loop and archive the traces weekly";+const NEAR_B =+ "Collect the latency metrics before optimizing the render loop and archive the traces daily";+const UNRELATED = "Use tabs inside makefiles and keep the tab width at two spaces";++interface SourcePageOptions {+ readonly body: string;+ readonly createdAt?: string;+ readonly sourcePath?: string;+ readonly mergedInto?: string;+ readonly kind?: string;+ readonly project?: string;+}++/** Write one ingest summary page; returns its vault-relative path. */+function writeSourcePage(slug: string, opts: SourcePageOptions): string {+ const lines = [+ "---",+ `kind: ${opts.kind ?? "brain-source"}`,+ `source_path: ${opts.sourcePath ?? `imports/${slug}.md`}`,+ `source_hash: ${"a".repeat(64)}`,+ ];+ if (opts.createdAt !== undefined) lines.push(`created_at: ${opts.createdAt}`);+ if (opts.mergedInto !== undefined) lines.push(`merged_into: ${opts.mergedInto}`);+ if (opts.project !== undefined) lines.push(`project: ${opts.project}`);+ lines.push("tags: [brain, brain/source]", "---", "", opts.body, "");+ const abs = join(vault, "Brain", "sources", `src-${slug}.md`);+ writeFileSync(abs, lines.join("\n"), "utf8");+ return `Brain/sources/src-${slug}.md`;+}++function detect(): ReturnType<typeof detectDocumentDedup> {+ return detectDocumentDedup(vault, { now: new Date(0) });+}++describe("detectDocumentDedup", () => {+ test("groups near-duplicate summary pages and keeps the newest created_at canonical", () => {+ const older = writeSourcePage("older", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ const newer = writeSourcePage("newer", { body: NEAR_B, createdAt: "2026-02-01T00:00:00Z" });+ writeSourcePage("unrelated", { body: UNRELATED, createdAt: "2026-03-01T00:00:00Z" });++ const findings = detect();+ expect(findings).toHaveLength(1);+ const finding = findings[0]!;+ expect(finding.detector).toBe("document-dedup");+ expect(finding.severity).toBe("warning");+ expect(finding.proposed_action).toBe("retire");+ expect(finding.targets).toEqual([older, newer].toSorted());+ expect(finding.id).toBe(hygieneFindingId("document-dedup", [older, newer]));+ expect(finding.evidence["method"]).toBe("lexical");+ expect(finding.evidence["threshold"]).toBe(DOCUMENT_DEDUP_LEXICAL_THRESHOLD);+ expect(finding.evidence["canonical"]).toBe(newer);+ });++ test("a created_at tie breaks on the id ascending", () => {+ const alpha = writeSourcePage("alpha", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ const beta = writeSourcePage("beta", { body: NEAR_B, createdAt: "2026-01-01T00:00:00Z" });+ const findings = detect();+ expect(findings).toHaveLength(1);+ expect(findings[0]!.evidence["canonical"]).toBe([alpha, beta].toSorted()[0]);+ });++ test("a group holding any member without created_at is ineligible for retirement", () => {+ const stamped = writeSourcePage("stamped", {+ body: NEAR_A,+ createdAt: "2026-01-01T00:00:00Z",+ });+ const unstamped = writeSourcePage("unstamped", { body: NEAR_B });++ const findings = detect();+ expect(findings).toHaveLength(1);+ const finding = findings[0]!;+ expect(finding.proposed_action).toBe("review");+ expect(finding.evidence["ineligible"]).toBe(true);+ expect(String(finding.evidence["reason"])).toContain("created_at");+ expect(finding.evidence["members_without_created_at"]).toEqual([unstamped]);+ expect(finding.evidence["canonical"]).toBeUndefined();+ expect(finding.targets).toEqual([stamped, unstamped].toSorted());+ });++ test("a member a merge already resolved leaves the pool", () => {+ writeSourcePage("live", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("resolved", {+ body: NEAR_B,+ createdAt: "2026-02-01T00:00:00Z",+ mergedInto: "pref-elsewhere",+ });+ expect(detect()).toHaveLength(0);+ });++ test("identical text in two scopes never groups", () => {+ writeSourcePage("alpha", {+ body: NEAR_A,+ createdAt: "2026-01-01T00:00:00Z",+ project: "alpha",+ });+ writeSourcePage("beta", {+ body: NEAR_B,+ createdAt: "2026-01-02T00:00:00Z",+ project: "beta",+ });+ expect(detect()).toHaveLength(0);+ });++ test("a pair below the lexical threshold is not nominated", () => {+ writeSourcePage("one", {+ body: "Collect the latency metrics before optimizing the render loop",+ createdAt: "2026-01-01T00:00:00Z",+ });+ writeSourcePage("two", {+ body: "Collect the garbage after every render pass and keep the loop quiet",+ createdAt: "2026-01-02T00:00:00Z",+ });+ expect(detect()).toHaveLength(0);+ });++ test("pages that are not ingest summaries are ignored", () => {+ writeSourcePage("pref", {+ body: NEAR_A,+ createdAt: "2026-01-01T00:00:00Z",+ kind: "brain-preference",+ });+ writeSourcePage("other", {+ body: NEAR_B,+ createdAt: "2026-01-02T00:00:00Z",+ kind: "brain-preference",+ });+ expect(detect()).toHaveLength(0);+ });++ test("pages too short to compare safely are skipped", () => {+ writeSourcePage("tiny-a", { body: "Use tabs", createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("tiny-b", { body: "Use tabs", createdAt: "2026-01-02T00:00:00Z" });+ expect(detect()).toHaveLength(0);+ });++ test("a member the caller cannot read is treated as absent", () => {+ writeSourcePage("visible", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ const hidden = writeSourcePage("hidden", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });+ const findings = detectDocumentDedup(vault, {+ now: new Date(0),+ readable: (rel) => rel !== hidden,+ });+ expect(findings).toHaveLength(0);+ });++ test("findings come out in a deterministic order across runs", () => {+ writeSourcePage("r-one", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("r-two", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });+ writeSourcePage("r-three", {+ body: "Watch the commit queue and rebase the patch series before review",+ createdAt: "2026-01-03T00:00:00Z",+ });+ writeSourcePage("r-four", {+ body: "Watch the commit queue and rebase the patch series before merge",+ createdAt: "2026-01-04T00:00:00Z",+ });+ const first = detect().map((f) => f.id);+ const second = detect().map((f) => f.id);+ expect(first).toEqual(second);+ expect(first).toHaveLength(2);+ });++ // Token sets of ten with a one-word slide: A~B and B~C clear the+ // lexical threshold while A~C (two slides apart) sits under it.+ const CHAIN_A = Array.from({ length: 10 }, (_, i) => `kw${String(i + 1).padStart(2, "0")}`).join(+ " ",+ );+ const CHAIN_B = Array.from({ length: 10 }, (_, i) => `kw${String(i + 2).padStart(2, "0")}`).join(+ " ",+ );+ const CHAIN_C = Array.from({ length: 10 }, (_, i) => `kw${String(i + 3).padStart(2, "0")}`).join(+ " ",+ );++ /**+ * The A-B-C chain: (A,B) and (B,C) pair above the threshold, (A,C)+ * does not, so the union-find still forms ONE component. Keep-newest+ * picks C - which is NOT directly similar to A - and the transitive+ * gate decides what may retire.+ */+ function seedChain(): { a: string; b: string; c: string } {+ const a = writeSourcePage("chain-a", { body: CHAIN_A, createdAt: "2026-01-01T00:00:00Z" });+ const b = writeSourcePage("chain-b", { body: CHAIN_B, createdAt: "2026-01-02T00:00:00Z" });+ const c = writeSourcePage("chain-c", { body: CHAIN_C, createdAt: "2026-01-03T00:00:00Z" });+ return { a, b, c };+ }++ test("a member with no direct pair to the canonical is never a retire target", () => {+ seedChain();++ const findings = detect();+ expect(findings).toHaveLength(2);+ const retire = findings.find((f) => f.proposed_action === "retire")!;+ const review = findings.find((f) => f.proposed_action === "review")!;++ // Only the copy DIRECTLY similar to the canonical C is retirable.+ const { b, c } = { b: "Brain/sources/src-chain-b.md", c: "Brain/sources/src-chain-c.md" };+ expect(retire.targets).toEqual([b, c].toSorted());+ expect(retire.evidence["canonical"]).toBe(c);+ // The transitive member lands in review with the reason in evidence.+ expect(review.targets).toEqual(["Brain/sources/src-chain-a.md", c].toSorted());+ expect(review.evidence["ineligible"]).toBe(true);+ expect(String(review.evidence["reason"])).toContain("transitive");+ expect(review.evidence["members_without_direct_pair"]).toEqual([+ "Brain/sources/src-chain-a.md",+ ]);+ });++ test("every member with a direct pair still retires through the pipeline", () => {+ const { a, b } = seedChain();+ const report = runHygieneScan(vault, { detectors: ["document-dedup"], now: new Date(0) });+ const retire = report.findings.find((f) => f.proposed_action === "retire")!;+ const plan = buildHygienePlan(report, { ids: [retire.id] });+ expect(plan.selected).toHaveLength(1);+ expect(plan.excluded_review).toEqual([]);++ // The finding is derelived by the applier from the same evidence+ // shape a scan produced: applying it must remove ONLY the copy+ // directly similar to the canonical (B) and leave the transitive+ // member (A) on disk.+ return applyHygienePlan(vault, plan, { agent: "tester", now: new Date(0) }).then((result) => {+ expect(result.errors).toEqual([]);+ expect(result.applied).toHaveLength(1);+ expect(result.applied[0]!.detail).toContain("retired 1 near-duplicate cop");+ expect(existsSync(join(vault, a))).toBe(true);+ expect(existsSync(join(vault, b))).toBe(false);+ });+ });+});++describe("document-dedup registration", () => {+ test("the detector is registered and opt-in", () => {+ expect(HYGIENE_DETECTOR_IDS).toContain("document-dedup");+ expect(DEFAULT_SCAN_IDS).not.toContain("document-dedup");+ });++ test("the default sweep does not run it; an explicit subset does", () => {+ writeSourcePage("scan-a", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("scan-b", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });++ const defaultSweep = runHygieneScan(vault, { now: new Date(0) });+ expect(defaultSweep.counts["document-dedup"]).toBeUndefined();++ const explicit = runHygieneScan(vault, {+ detectors: ["document-dedup"],+ now: new Date(0),+ });+ expect(explicit.errors).toHaveLength(0);+ expect(explicit.counts["document-dedup"]).toBe(1);+ expect(explicit.findings[0]!.proposed_action).toBe("retire");+ });+});++describe("detectSemanticDocumentDedup", () => {+ function fixedVectorProvider(vectors: Readonly<Record<string, number[]>>): EmbeddingProvider {+ return {+ name: "fixed",+ model: "fixed-test",+ dimension: 3,+ embed: (texts) => Promise.resolve(texts.map((text) => vectors[text] ?? [0, 0, 1])),+ ping: () => Promise.resolve({ ok: true as const, dimension: 3 }),+ };+ }++ test("groups by embedding cosine at the embedding threshold", async () => {+ const older = writeSourcePage("emb-old", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ const newer = writeSourcePage("emb-new", { body: NEAR_B, createdAt: "2026-02-01T00:00:00Z" });+ const provider = fixedVectorProvider({+ [NEAR_A]: [1, 0, 0],+ [NEAR_B]: [0.999, 0.0447, 0],+ [UNRELATED]: [0, 1, 0],+ });+ const result = await detectSemanticDocumentDedup(vault, { provider });+ expect(result.method).toBe("embedding");+ expect(result.findings).toHaveLength(1);+ const finding = result.findings[0]!;+ expect(finding.targets).toEqual([older, newer].toSorted());+ expect(finding.proposed_action).toBe("retire");+ expect(finding.evidence["method"]).toBe("embedding");+ expect(finding.evidence["model"]).toBe("fixed-test");+ expect(finding.evidence["canonical"]).toBe(newer);+ });++ test("falls back to the lexical layer, labeled, when the provider fails", async () => {+ writeSourcePage("lex-a", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("lex-b", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });+ const broken: EmbeddingProvider = {+ name: "broken",+ model: "broken",+ dimension: 3,+ embed: () => Promise.reject(new Error("provider down")),+ ping: () => Promise.resolve({ ok: false as const, reason: "down" }),+ };+ const result = await detectSemanticDocumentDedup(vault, { provider: broken });+ expect(result.method).toBe("lexical");+ expect(result.findings).toHaveLength(1);+ expect(result.findings[0]!.evidence["method"]).toBe("lexical");+ });++ test("no usable provider means the lexical layer", async () => {+ const result = await detectSemanticDocumentDedup(vault, { provider: null });+ expect(result.method).toBe("lexical");+ expect(result.findings).toHaveLength(0);+ });++ test("a page the caller cannot read is scored and named by no semantic pair", async () => {+ writeSourcePage("emb-vis-1", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("emb-vis-2", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });+ const hidden = writeSourcePage("emb-hidden", {+ body: NEAR_A,+ createdAt: "2026-01-03T00:00:00Z",+ });+ const provider = fixedVectorProvider({+ [NEAR_A]: [1, 0, 0],+ [NEAR_B]: [0.999, 0.0447, 0],+ });++ const result = await detectSemanticDocumentDedup(vault, {+ provider,+ readable: (rel) => rel !== hidden,+ });++ expect(result.method).toBe("embedding");+ expect(result.findings).toHaveLength(1);+ expect(JSON.stringify(result.findings)).not.toContain(hidden);+ });++ test("records across different scope keys never pair, semantically either", async () => {+ writeSourcePage("scope-alpha", {+ body: NEAR_A,+ createdAt: "2026-01-01T00:00:00Z",+ project: "alpha",+ });+ writeSourcePage("scope-beta", {+ body: NEAR_B,+ createdAt: "2026-01-02T00:00:00Z",+ project: "beta",+ });+ const provider = fixedVectorProvider({+ [NEAR_A]: [1, 0, 0],+ [NEAR_B]: [1, 0, 0],+ });++ const result = await detectSemanticDocumentDedup(vault, { provider });++ expect(result.method).toBe("embedding");+ expect(result.findings).toHaveLength(0);+ });++ test("a provider returning inconsistent vector dimensions falls back, labeled", async () => {+ writeSourcePage("rag-a", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("rag-b", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });+ // Dimension mismatch inside one embed call: the vectors-length check+ // models this for a count mismatch; a shape mismatch is the same+ // broken-provider answer.+ const ragged: EmbeddingProvider = {+ name: "ragged",+ model: "ragged",+ dimension: 3,+ embed: (texts) => Promise.resolve(texts.map((_, i) => (i === 0 ? [1, 0] : [1, 0, 0]))),+ ping: () => Promise.resolve({ ok: true as const, dimension: 3 }),+ };++ const result = await detectSemanticDocumentDedup(vault, { provider: ragged });++ expect(result.method).toBe("lexical");+ // The fallback still finds the lexical pair - it just never calls+ // the fabricated cosine score semantic.+ expect(result.findings).toHaveLength(1);+ expect(result.findings[0]!.evidence["method"]).toBe("lexical");+ });+});diff --git a/tests/core/brain/document-retire.test.ts b/tests/core/brain/document-retire.test.tsnew file mode 100644index 00000000..efc71a6d--- /dev/null+++ b/tests/core/brain/document-retire.test.ts@@ -0,0 +1,448 @@+/**+ * Document retirement applier core (storage lifecycle, Lane 4 task 4.3).+ *+ * Per stale copy the applier must: retarget inbound wikilinks to the+ * canonical, remove the page, drop the content-manifest entry+ * round-tripping the read contract, and append one `source_invalidation`+ * continuity record naming canonical id, retired id, dropped manifest+ * key and the retained source-archive hash. The whole span sits inside+ * the destructive-snapshot gate with a recoverability verdict; the+ * source archive is evidence and stays on disk. Per-copy failures are+ * collected ({path, message}), the failed copy stays on disk, and the+ * run completes.+ *+ * The manifest drop is the resurrect guard: with the entry gone, the+ * retired copy's source classifies `new` again, so the system never+ * claims a retired page's records are current and a later ingest can+ * only bring the source back as a deliberate full re-ingest.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import {+ chmodSync,+ existsSync,+ mkdirSync,+ mkdtempSync,+ readFileSync,+ rmSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { basename, join } from "node:path";++import {+ classifyPaths,+ readManifest,+ updateManifest,+} from "../../../src/core/brain/ingest/content-manifest.ts";+import { listContinuityRecords } from "../../../src/core/brain/continuity/store.ts";+import {+ RETIRE_LINKS_SKIPPED_BASENAME_COLLISION,+ retireDocumentCopies,+} from "../../../src/core/brain/document-retire.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";++let vault: string;++const NOW = new Date("2026-06-01T00:00:00Z");+const ARCHIVE_HASH = "b".repeat(64);++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-document-retire-"));+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ mkdirSync(join(vault, "imports"), { recursive: true });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++const CANONICAL_BODY =+ "Collect the latency metrics before optimizing the render loop and archive the traces weekly";+const DUPLICATE_BODY =+ "Collect the latency metrics before optimizing the render loop and archive the traces daily";++interface PageOptions {+ readonly body: string;+ readonly sourcePath: string;+ readonly createdAt: string;+ readonly archiveHash?: string;+}++/** Write one ingest summary page; returns its vault-relative path. */+function writeSourcePage(slug: string, opts: PageOptions): string {+ const lines = [+ "---",+ "kind: brain-source",+ `source_path: ${opts.sourcePath}`,+ `source_hash: ${"a".repeat(64)}`,+ ];+ if (opts.archiveHash !== undefined) {+ lines.push(`source_archive_hash: ${opts.archiveHash}`, "source_archive_bytes: 24");+ }+ lines.push("tags: [brain, brain/source]", `created_at: ${opts.createdAt}`, "---", "");+ lines.push(opts.body, "", "## Sources", "", `- [[${opts.sourcePath}]]`, "");+ const abs = join(vault, "Brain", "sources", `src-${slug}.md`);+ writeFileSync(abs, lines.join("\n"), "utf8");+ return `Brain/sources/src-${slug}.md`;+}++/**+ * Build the near-duplicate pair: canonical src-a and stale src-b, both+ * ingested (manifest entries recorded), plus a citing page whose+ * `[[src-b]]` link must be retargeted, and the durable source archive+ * for the stale copy's source.+ */+function seedPair(): { canonical: string; retired: string; citing: string } {+ writeFileSync(join(vault, "imports", "a.md"), "canonical source bytes\n", "utf8");+ writeFileSync(join(vault, "imports", "b.md"), "duplicate source bytes\n", "utf8");+ const canonical = writeSourcePage("a", {+ body: CANONICAL_BODY,+ sourcePath: "imports/a.md",+ createdAt: "2026-01-01T00:00:00Z",+ });+ const retired = writeSourcePage("b", {+ body: DUPLICATE_BODY,+ sourcePath: "imports/b.md",+ createdAt: "2026-02-01T00:00:00Z",+ archiveHash: ARCHIVE_HASH,+ });+ updateManifest(vault, ["imports/a.md", "imports/b.md"]);+ const citing = "Brain/note.md";+ writeFileSync(+ join(vault, citing),+ "# Note\n\nSee [[src-b]] for the duplicate and [[imports/a]] for the original.\n",+ "utf8",+ );+ mkdirSync(join(vault, "Brain", ".archives"), { recursive: true });+ writeFileSync(join(vault, "Brain", ".archives", `${ARCHIVE_HASH}.txt`), "redacted bytes", "utf8");+ return { canonical, retired, citing };+}++describe("retireDocumentCopies", () => {+ test("retires a stale copy end to end behind the snapshot gate", () => {+ const { canonical, retired, citing } = seedPair();+ const before = readFileSync(join(vault, citing), "utf8");++ const result = retireDocumentCopies(vault, [{ path: retired, canonical }], {+ agent: "tester",+ now: NOW,+ });++ expect(result.failed).toEqual([]);+ expect(result.retired).toHaveLength(1);+ const copy = result.retired[0]!;+ expect(copy.path).toBe(retired);+ expect(copy.canonical).toBe(canonical);+ expect(copy.manifestKey).toBe("imports/b.md");+ expect(copy.manifestDropped).toBe(true);+ expect(copy.sourceArchiveHash).toBe(ARCHIVE_HASH);+ expect(copy.auditRecordId).not.toBeNull();++ // The wikilink retarget report, asserted: the citing page matched,+ // was rewritten, and its [[src-b]] now names the canonical spelling.+ expect(copy.wikilinksSkipped).toBeNull();+ expect(copy.wikilinks!.matched).toEqual([citing]);+ expect(copy.wikilinks!.rewritten).toEqual([citing]);+ expect(copy.wikilinks!.failed).toEqual([]);+ expect(readFileSync(join(vault, citing), "utf8")).toContain("[[src-a]]");+ expect(readFileSync(join(vault, citing), "utf8")).not.toBe(before);++ // The stale page is gone; the canonical page survives.+ expect(existsSync(join(vault, retired))).toBe(false);+ expect(existsSync(join(vault, canonical))).toBe(true);++ // The whole span sat behind the snapshot gate with a verdict.+ expect(result.snapshotRunId).not.toBeNull();+ expect(result.snapshotPath).not.toBeNull();+ expect(existsSync(result.snapshotPath!)).toBe(true);+ expect(result.recoverability.state).toBe("covered");+ });++ test("drops the manifest entry round-tripping the read contract", () => {+ const { canonical, retired } = seedPair();+ const fingerprintBefore = readManifest(vault).entries["imports/a.md"]?.fingerprint;++ retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });++ const manifest = readManifest(vault);+ expect(manifest.entries["imports/b.md"]).toBeUndefined();+ // The surviving entry keeps the fingerprint it was recorded under -+ // a removal never re-stamps the contract the survivors answered.+ expect(manifest.entries["imports/a.md"]?.fingerprint).toBe(fingerprintBefore);+ });++ test("the manifest drop is the resurrect guard: the retired source classifies new", () => {+ const { canonical, retired } = seedPair();+ retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });++ const classification = classifyPaths(vault, ["imports/b.md"], readManifest(vault));+ expect(classification.new).toContain("imports/b.md");+ expect(classification.unchanged).not.toContain("imports/b.md");+ });++ test("appends one source_invalidation record per copy naming the provenance", () => {+ const { canonical, retired } = seedPair();+ const recordsBefore = listContinuityRecords(vault, { kind: "source_invalidation" }).length;++ retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });++ const records = listContinuityRecords(vault, { kind: "source_invalidation" });+ expect(records.length).toBe(recordsBefore + 1);+ const record = records.at(-1)!;+ expect(record.id).not.toBeNull();+ const reason = String((record.payload as Record<string, unknown>)["reason"]);+ expect(reason).toContain("src-a");+ expect(reason).toContain("src-b");+ expect(reason).toContain("imports/b.md");+ expect(reason).toContain(ARCHIVE_HASH);+ });++ test("the retained source archive is still on disk after the run", () => {+ const { canonical, retired } = seedPair();+ retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });+ expect(existsSync(join(vault, "Brain", ".archives", `${ARCHIVE_HASH}.txt`))).toBe(true);+ });++ test("a per-copy failure is collected, the copy stays on disk, and the run completes", () => {+ const { canonical, retired } = seedPair();+ const missing = "Brain/sources/src-gone.md";++ const result = retireDocumentCopies(+ vault,+ [+ { path: missing, canonical },+ { path: retired, canonical },+ ],+ { agent: "tester", now: NOW },+ );++ expect(result.retired.map((r) => r.path)).toEqual([retired]);+ expect(result.failed).toHaveLength(1);+ expect(result.failed[0]!.path).toBe(missing);+ expect(result.failed[0]!.message.length).toBeGreaterThan(0);+ // The run completed behind one snapshot span covering the copies.+ expect(result.snapshotRunId).not.toBeNull();+ });++ test("a basename collision skips the link pass and names why", () => {+ // An unrelated page in another namespace shares the retired copy's+ // basename, so the bare [[src-b]] spelling is ambiguous: rewriting it+ // would repoint links that meant the namesake. The copy still retires;+ // the pass is skipped by name and every link stays exactly as it was -+ // a stale link is discoverable, a mis-pointed one misleads silently.+ const { canonical, retired, citing } = seedPair();+ const namesake = "Brain/entities/src-b.md";+ mkdirSync(join(vault, "Brain", "entities"), { recursive: true });+ writeFileSync(join(vault, namesake), "# Src b\n\nThe page the link really means.\n", "utf8");++ const result = retireDocumentCopies(vault, [{ path: retired, canonical }], {+ agent: "tester",+ now: NOW,+ });++ expect(result.failed).toEqual([]);+ const copy = result.retired[0]!;+ expect(copy.path).toBe(retired);+ expect(copy.wikilinks).toBeNull();+ expect(copy.wikilinksSkipped).toBe(RETIRE_LINKS_SKIPPED_BASENAME_COLLISION);+ // The citing page keeps its spelling, the namesake page is untouched,+ // and the retired copy itself is still gone.+ expect(readFileSync(join(vault, citing), "utf8")).toContain("[[src-b]]");+ expect(existsSync(join(vault, namesake))).toBe(true);+ expect(existsSync(join(vault, retired))).toBe(false);+ });++ test("a copy whose canonical page is missing is refused per copy", () => {+ const { retired } = seedPair();+ const result = retireDocumentCopies(+ vault,+ [{ path: retired, canonical: "Brain/sources/src-missing.md" }],+ { agent: "tester", now: NOW },+ );+ expect(result.retired).toEqual([]);+ expect(result.failed).toHaveLength(1);+ expect(result.failed[0]!.path).toBe(retired);+ expect(existsSync(join(vault, retired))).toBe(true);+ });++ test("a page that is not an ingest summary is refused per copy", () => {+ const { canonical, retired } = seedPair();+ writeFileSync(+ join(vault, "Brain", "sources", "src-nota-000000000000.md"),+ "---\nkind: brain-preference\n---\n\nnot a summary\n",+ "utf8",+ );+ const result = retireDocumentCopies(+ vault,+ [{ path: "Brain/sources/src-nota-000000000000.md", canonical }],+ { agent: "tester", now: NOW },+ );+ expect(result.retired).toEqual([]);+ expect(result.failed).toHaveLength(1);+ expect(existsSync(join(vault, retired))).toBe(true);+ });++ test("retiring the canonical page itself is a caller bug and refuses loudly", () => {+ const { canonical } = seedPair();+ expect(() =>+ retireDocumentCopies(vault, [{ path: canonical, canonical }], { agent: "t", now: NOW }),+ ).toThrow(/canonical/);+ });++ test("no copies means no snapshot and nothing at risk", () => {+ const result = retireDocumentCopies(vault, [], { agent: "tester", now: NOW });+ expect(result.retired).toEqual([]);+ expect(result.failed).toEqual([]);+ expect(result.snapshotRunId).toBeNull();+ expect(result.recoverability.state).toBe("nothing_at_risk");+ });++ test("a failure AFTER the removal reports the copy as removed, not on disk", () => {+ if (CHMOD_CANNOT_DENY) return;+ const { canonical, retired } = seedPair();+ // The continuity append is the LAST step, after the page is gone:+ // a read-only continuity directory fails exactly that step.+ const continuityDir = join(vault, "Brain", "log", "continuity");+ mkdirSync(continuityDir, { recursive: true });+ chmodSync(continuityDir, 0o555);+ try {+ const result = retireDocumentCopies(vault, [{ path: retired, canonical }], {+ agent: "tester",+ now: NOW,+ });++ expect(result.failed).toHaveLength(1);+ expect(result.failed[0]!.path).toBe(retired);+ expect(result.failed[0]!.message).toContain("removed");+ expect(result.failed[0]!.message).toContain(retired);+ // The honest half of the old claim: the page is NOT on disk.+ expect(existsSync(join(vault, retired))).toBe(false);+ // The steps before the failure did land.+ expect(readManifest(vault).entries["imports/b.md"]).toBeUndefined();+ expect(listContinuityRecords(vault, { kind: "source_invalidation" })).toHaveLength(0);+ } finally {+ chmodSync(continuityDir, 0o755);+ }+ });++ test("the per-copy reason words the manifest outcome from what actually happened", () => {+ const { retired } = seedPair();+ retireDocumentCopies(vault, [{ path: retired, canonical: "Brain/sources/src-a.md" }], {+ agent: "tester",+ now: NOW,+ });+ const reason = String(+ (+ listContinuityRecords(vault, { kind: "source_invalidation" }).at(-1)!.payload as Record<+ string,+ unknown+ >+ )["reason"],+ );+ expect(reason).toContain("dropped manifest key imports/b.md");+ });++ test("a shared source_path retires twice: the second drop is named already-absent", () => {+ const { canonical } = seedPair();+ // A second copy declaring the SAME source_path as the first copy's.+ const second = writeSourcePage("b2", {+ body: DUPLICATE_BODY,+ sourcePath: "imports/b.md",+ createdAt: "2026-02-02T00:00:00Z",+ });+ const result = retireDocumentCopies(+ vault,+ [+ { path: second, canonical },+ { path: "Brain/sources/src-b.md", canonical },+ ],+ { agent: "tester", now: NOW },+ );+ expect(result.failed).toEqual([]);+ const reasons = listContinuityRecords(vault, { kind: "source_invalidation" }).map((r) =>+ String((r.payload as Record<string, unknown>)["reason"]),+ );+ expect(reasons.some((r) => r.includes("dropped manifest key imports/b.md"))).toBe(true);+ expect(reasons.some((r) => r.includes("manifest key imports/b.md already absent"))).toBe(true);+ });++ test("no manifest on disk is named, not claimed as a drop", () => {+ const { canonical, retired } = seedPair();+ rmSync(join(vault, ".open-second-brain", "ingest-manifest.json"));+ retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });+ const reason = String(+ (+ listContinuityRecords(vault, { kind: "source_invalidation" }).at(-1)!.payload as Record<+ string,+ unknown+ >+ )["reason"],+ );+ expect(reason).toContain("manifest");+ expect(reason).toContain("imports/b.md");+ expect(reason).not.toContain("dropped manifest key");+ });++ test("a copy with no recorded source_path names the absence, not a drop", () => {+ const { canonical } = seedPair();+ const keyless = writeSourcePage("keyless", {+ body: DUPLICATE_BODY,+ sourcePath: "",+ createdAt: "2026-02-03T00:00:00Z",+ });+ retireDocumentCopies(vault, [{ path: keyless, canonical }], { agent: "tester", now: NOW });+ const reason = String(+ (+ listContinuityRecords(vault, { kind: "source_invalidation" }).at(-1)!.payload as Record<+ string,+ unknown+ >+ )["reason"],+ );+ expect(reason).toContain("no manifest key recorded");+ expect(reason).not.toContain("dropped manifest key");+ });++ test("a copy path that escapes the vault is refused as a containment breach, not a vanish", () => {+ // A hand-built plan naming a path outside the vault: the file exists at+ // the escaped location and looks like a retireable summary, so the run+ // used to reach the removal and report the shared removeFile `false` as+ // "vanished before removal" - a disappearance that never happened. The+ // refusal is named for what it is, and the outside file is untouched.+ // The escaped sibling sits outside the vault temp dir the afterEach+ // removes, so it carries the run's unique suffix and cleans up itself.+ const { canonical } = seedPair();+ const escaped = join("..", `escaped-${basename(vault)}.md`);+ writeFileSync(+ join(vault, escaped),+ [+ "---",+ "kind: brain-source",+ "source_path: imports/b.md",+ `source_hash: ${"a".repeat(64)}`,+ "---",+ "",+ DUPLICATE_BODY,+ "",+ ].join("\n"),+ "utf8",+ );++ try {+ const result = retireDocumentCopies(vault, [{ path: escaped, canonical }], {+ agent: "tester",+ now: NOW,+ });++ expect(result.retired).toEqual([]);+ expect(result.failed).toHaveLength(1);+ expect(result.failed[0]!.path).toBe(escaped);+ expect(result.failed[0]!.message).toContain("escapes the vault");+ expect(result.failed[0]!.message).not.toContain("vanished");+ expect(existsSync(join(vault, escaped))).toBe(true);+ } finally {+ rmSync(join(vault, escaped));+ }+ });+});diff --git a/tests/core/brain/gates/recoverability.test.ts b/tests/core/brain/gates/recoverability.test.tsindex 538d5d8c..29a35269 100644--- a/tests/core/brain/gates/recoverability.test.ts+++ b/tests/core/brain/gates/recoverability.test.ts@@ -344,6 +344,10 @@ describe("the two pre-existing call sites return byte-identical payloads", () =>' "Brain/entities/concept/ent-concept-benchwidget.md",',' "Brain/sources/src-imports-benchmark-md-54d80ede894c.md"'," ],",+ // The per-entry failure list the batch decomposition added (empty on+ // a clean run) - a deliberate shape change, measured and typed here+ // for the same reason the recoverability block below is.+ ' "failed": [],',' "manifestEntryRemoved": true,',' "auditRecordId": "ctn_20260601000000_a68bcc9d65569947",',' "snapshotRunId": "delete-by-source-2026-06-01-000000",',diff --git a/tests/core/brain/hygiene-retire-routing.test.ts b/tests/core/brain/hygiene-retire-routing.test.tsnew file mode 100644index 00000000..68ec0eba--- /dev/null+++ b/tests/core/brain/hygiene-retire-routing.test.ts@@ -0,0 +1,296 @@+/**+ * Retire routing through the hygiene pipeline (storage lifecycle, Lane 4+ * task 4.2).+ *+ * The `retire` action joins the closed vocabulary and the applier+ * capability table as mechanical, but it is the destructive class: the+ * unattended plan default (no ids) holds it in the review path, and it+ * executes only when a caller names its finding id. `executeFinding`+ * then dispatches it to the retirement core; the count-guard semantics+ * the surfaces already enforce on `plan.selected` cover it unchanged.+ *+ * These tests drive the REAL pipeline end to end: detect on a fixture+ * vault, plan, apply - because the routing is only proved when the+ * retirement actually lands.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { readManifest, updateManifest } from "../../../src/core/brain/ingest/content-manifest.ts";+import { listContinuityRecords } from "../../../src/core/brain/continuity/store.ts";+import { applyHygienePlan } from "../../../src/core/brain/hygiene/apply.ts";+import { buildHygienePlan } from "../../../src/core/brain/hygiene/plan.ts";+import { runHygieneScan } from "../../../src/core/brain/hygiene/scan.ts";+import {+ HYGIENE_PROPOSED_ACTIONS,+ type HygieneFinding,+} from "../../../src/core/brain/hygiene/types.ts";+import { requireRepairCapability } from "../../../src/core/brain/applier-capability.ts";+import {+ WRITE_REFUSAL_CODES,+ type WriteSubject,+} from "../../../src/core/brain/write-disposition.ts";++let vault: string;+let savedAgentName: string | undefined;++const NOW = new Date("2026-06-01T00:00:00Z");+const CANONICAL_BODY =+ "Collect the latency metrics before optimizing the render loop and archive the traces weekly";+const DUPLICATE_BODY =+ "Collect the latency metrics before optimizing the render loop and archive the traces daily";++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-retire-routing-"));+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ mkdirSync(join(vault, "imports"), { recursive: true });+ savedAgentName = process.env["VAULT_AGENT_NAME"];+});++afterEach(() => {+ if (savedAgentName === undefined) delete process.env["VAULT_AGENT_NAME"];+ else process.env["VAULT_AGENT_NAME"] = savedAgentName;+ rmSync(vault, { recursive: true, force: true });+});++function writeSourcePage(slug: string, body: string, createdAt: string): string {+ const abs = join(vault, "Brain", "sources", `src-${slug}.md`);+ writeFileSync(+ abs,+ [+ "---",+ "kind: brain-source",+ `source_path: imports/${slug}.md`,+ `source_hash: ${"a".repeat(64)}`,+ "tags: [brain, brain/source]",+ `created_at: ${createdAt}`,+ "---",+ "",+ body,+ "",+ ].join("\n"),+ "utf8",+ );+ return `Brain/sources/src-${slug}.md`;+}++/**+ * Near-duplicate pair, both sources recorded in the manifest. The+ * detector keeps the NEWEST page as canonical, so src-b (created+ * 2026-02-01) survives and src-a (created 2026-01-01) is the stale+ * copy a retirement removes.+ */+function seedPair(): { canonical: string; retired: string } {+ writeFileSync(join(vault, "imports", "a.md"), "older source bytes\n", "utf8");+ writeFileSync(join(vault, "imports", "b.md"), "newer source bytes\n", "utf8");+ const retired = writeSourcePage("a", CANONICAL_BODY, "2026-01-01T00:00:00Z");+ const canonical = writeSourcePage("b", DUPLICATE_BODY, "2026-02-01T00:00:00Z");+ updateManifest(vault, ["imports/a.md", "imports/b.md"]);+ return { canonical, retired };+}++/** Scan with the document-dedup detector and return its retire finding. */+function scanRetireFinding(): HygieneFinding {+ const report = runHygieneScan(vault, { detectors: ["document-dedup"], now: NOW });+ const finding = report.findings.find((f) => f.proposed_action === "retire");+ if (finding === undefined) throw new Error("fixture produced no retire finding");+ return finding;+}++describe("retire joins the vocabulary and the capability table", () => {+ test("the vocabulary carries retire as a mechanical, attended-only action", () => {+ expect(HYGIENE_PROPOSED_ACTIONS).toContain("retire");+ const capability = requireRepairCapability("retire");+ expect(capability.verdict).toBe("mechanical");+ expect(+ capability.verdict === "mechanical" && capability.fixer.includes("retireDocumentCopies"),+ ).toBe(true);+ });++ test("an unattended plan holds retire findings in the review path", () => {+ seedPair();+ const finding = scanRetireFinding();+ const report = runHygieneScan(vault, { detectors: ["document-dedup"], now: NOW });++ const unattended = buildHygienePlan(report);+ expect(unattended.selected).toEqual([]);+ expect(unattended.excluded_review).toEqual([finding.id]);+ });++ test("an attended plan selects the retire finding by id", () => {+ seedPair();+ const finding = scanRetireFinding();+ const report = runHygieneScan(vault, { detectors: ["document-dedup"], now: NOW });++ const attended = buildHygienePlan(report, { ids: [finding.id] });+ expect(attended.selected.map((f) => f.id)).toEqual([finding.id]);+ expect(attended.excluded_review).toEqual([]);+ });+});++describe("applyHygienePlan dispatches retire to the retirement core", () => {+ test("an attended apply retires the stale copy for real", async () => {+ const { canonical, retired } = seedPair();+ const finding = scanRetireFinding();+ const plan = buildHygienePlan(+ runHygieneScan(vault, { detectors: ["document-dedup"], now: NOW }),+ {+ ids: [finding.id],+ },+ );++ const result = await applyHygienePlan(vault, plan, {+ agent: "tester",+ now: NOW,+ });++ expect(result.errors).toEqual([]);+ expect(result.applied).toHaveLength(1);+ expect(result.applied[0]!.action).toBe("retire");+ expect(result.applied[0]!.finding_id).toBe(finding.id);+ expect(result.applied[0]!.detail).toContain(canonical);++ expect(existsSync(join(vault, retired))).toBe(false);+ expect(existsSync(join(vault, canonical))).toBe(true);+ // The manifest entry dropped is the RETIRED copy's source; the+ // canonical copy's source stays recorded.+ expect(readManifest(vault).entries["imports/a.md"]).toBeUndefined();+ expect(readManifest(vault).entries["imports/b.md"]).toBeDefined();+ expect(+ listContinuityRecords(vault, { kind: "source_invalidation" }).some((r) => r.id !== null),+ ).toBe(true);+ });++ test("a dry run previews the retirement with zero writes", async () => {+ const { retired } = seedPair();+ const finding = scanRetireFinding();+ const plan = buildHygienePlan(+ runHygieneScan(vault, { detectors: ["document-dedup"], now: NOW }),+ {+ ids: [finding.id],+ },+ );++ const result = await applyHygienePlan(vault, plan, {+ dryRun: true,+ agent: "tester",+ now: NOW,+ });++ expect(result.dry_run).toBe(true);+ expect(result.planned.map((p) => p.action)).toEqual(["retire"]);+ expect(result.applied).toEqual([]);+ // Nothing moved: the page is still on disk and the manifest intact.+ expect(existsSync(join(vault, retired))).toBe(true);+ expect(readManifest(vault).entries["imports/a.md"]).toBeDefined();+ expect(listContinuityRecords(vault, { kind: "source_invalidation" })).toHaveLength(0);+ });++ test("a per-copy failure surfaces in the apply errors, fail-soft", async () => {+ const { canonical, retired } = seedPair();+ // The report - and the plan built from it - is captured BEFORE the+ // canonical vanishes, exactly as an operator's prior scan would be.+ const report = runHygieneScan(vault, { detectors: ["document-dedup"], now: NOW });+ const finding = report.findings.find((f) => f.proposed_action === "retire")!;+ // The retirement refuses the copy rather than strand every inbound+ // link to a page that is not there.+ rmSync(join(vault, canonical));+ const plan = buildHygienePlan(report, { ids: [finding.id] });++ const result = await applyHygienePlan(vault, plan, {+ agent: "tester",+ now: NOW,+ });++ expect(result.applied).toEqual([]);+ expect(result.errors).toHaveLength(1);+ expect(result.errors[0]!.finding_id).toBe(finding.id);+ expect(result.errors[0]!.message).toContain("canonical");+ // The failed copy stays on disk.+ expect(existsSync(join(vault, retired))).toBe(true);+ });+});++describe("the retire deny consult answers for the credential subject", () => {+ /** The subject a credential-minted request would carry (write-side trust). */+ const SUBJECT: WriteSubject = { agent: "codex", via: "token" };++ /**+ * A permissions document closing `codex` out of the ingest action the+ * retirement lane consults. The ambient agent is `claude`, so ONLY a+ * consult answering for the passed subject can produce this refusal -+ * the config fallback must never decide the retire route.+ */+ function seedDeniedPair(): { canonical: string; retired: string; finding: HygieneFinding } {+ process.env["VAULT_AGENT_NAME"] = "claude";+ const seeded = seedPair();+ writeFileSync(+ join(vault, "Brain", "_permissions.yaml"),+ "version: 1\ndefault_action: allow\nentries:\n - id: freeze\n agent: codex\n" ++ " action: ingest\n verdict: deny\n",+ "utf8",+ );+ return { ...seeded, finding: scanRetireFinding() };+ }++ function planFor(finding: HygieneFinding) {+ return buildHygienePlan(runHygieneScan(vault, { detectors: ["document-dedup"], now: NOW }), {+ ids: [finding.id],+ });+ }++ test("a document deny for the subject refuses the retire route before any write", async () => {+ const { canonical, retired, finding } = seedDeniedPair();++ const result = await applyHygienePlan(vault, planFor(finding), {+ agent: "tester",+ now: NOW,+ subject: SUBJECT,+ });++ expect(result.applied).toEqual([]);+ expect(result.errors).toHaveLength(1);+ expect(result.errors[0]!.finding_id).toBe(finding.id);+ // The refusal answers for the credential subject, the ingest action+ // the lane consults and the deciding rule - the same typed refusal+ // deleteBySource produces.+ expect(result.errors[0]!.message).toContain(WRITE_REFUSAL_CODES.documentDeny);+ expect(result.errors[0]!.message).toContain('"codex"');+ expect(result.errors[0]!.message).toContain("ingest");+ expect(result.errors[0]!.message).toContain("entry:freeze");+ // The deny consult ran before anything was snapshotted or removed:+ // the copy, its manifest entry and the canonical all survive.+ expect(existsSync(join(vault, retired))).toBe(true);+ expect(existsSync(join(vault, canonical))).toBe(true);+ expect(readManifest(vault).entries["imports/a.md"]).toBeDefined();+ expect(listContinuityRecords(vault, { kind: "source_invalidation" })).toHaveLength(0);+ const snapshots = join(vault, "Brain", ".snapshots");+ expect(existsSync(snapshots) ? readdirSync(snapshots) : []).toEqual([]);+ });++ test("a deny naming another agent leaves the retire applying under the same subject", async () => {+ process.env["VAULT_AGENT_NAME"] = "codex";+ const { canonical, retired } = seedPair();+ writeFileSync(+ join(vault, "Brain", "_permissions.yaml"),+ "version: 1\ndefault_action: allow\nentries:\n - id: freeze-gemini\n agent: gemini\n" ++ " action: ingest\n verdict: deny\n",+ "utf8",+ );+ const finding = scanRetireFinding();++ const result = await applyHygienePlan(vault, planFor(finding), {+ agent: "tester",+ now: NOW,+ subject: { agent: "claude", via: "token" },+ });++ expect(result.errors).toEqual([]);+ expect(result.applied).toHaveLength(1);+ expect(existsSync(join(vault, retired))).toBe(false);+ expect(existsSync(join(vault, canonical))).toBe(true);+ });+});diff --git a/tests/core/brain/ingest/content-manifest.test.ts b/tests/core/brain/ingest/content-manifest.test.tsindex 0272bd42..c6040f53 100644--- a/tests/core/brain/ingest/content-manifest.test.ts+++ b/tests/core/brain/ingest/content-manifest.test.ts@@ -17,6 +17,7 @@ import { ingestSource } from "../../../../src/core/brain/ingest/ingest.ts";import { computeExtractionContractFingerprint } from "../../../../src/core/brain/ingest/contract.ts";import {classifyPaths,+ extractedBytesOf,hashFile,hashTree,manifestPath,@@ -391,6 +392,119 @@ describe("extraction contract (t_586d5d8b)", () => {});});+describe("extraction watermark (t_a9cd4221)", () => {+ const WATERMARK = { extracted_bytes: 11, extracted_at: "2026-06-13T12:00:00Z" };++ test("additive watermark fields round-trip through reads and writes; schema stays 3", () => {+ writeSource("Inbox/log.md", "first line\n");+ updateManifest(vault, ["Inbox/log.md"]);+ const recorded = readManifest(vault).entries["Inbox/log.md"]!;++ // Hand-stamp the watermark the incremental extractor writes, through the+ // explicit-contract write form.+ writeManifestAtomic(+ vault,+ { "Inbox/log.md": { ...recorded, ...WATERMARK } },+ readManifest(vault).contract,+ );++ const reread = readManifest(vault).entries["Inbox/log.md"]!;+ expect(reread).toEqual({+ sha256: hashFile(join(vault, "Inbox", "log.md")),+ fingerprint: computeExtractionContractFingerprint(vault),+ ...WATERMARK,+ });++ // The additive fields never bump the schema version on disk.+ const onDisk = JSON.parse(readFileSync(manifestPath(vault), "utf8")) as {+ schema_version: number;+ };+ expect(onDisk.schema_version).toBe(3);+ });++ test("an untouched watermarked entry survives an updateManifest of another path", () => {+ writeSource("Inbox/log.md", "first line\n");+ updateManifest(vault, ["Inbox/log.md"]);+ const recorded = readManifest(vault).entries["Inbox/log.md"]!;+ writeManifestAtomic(+ vault,+ { "Inbox/log.md": { ...recorded, ...WATERMARK } },+ readManifest(vault).contract,+ );++ writeSource("Inbox/other.md", "a different source");+ updateManifest(vault, ["Inbox/other.md"]);++ const survivor = readManifest(vault).entries["Inbox/log.md"]!;+ expect(survivor.extracted_bytes).toBe(WATERMARK.extracted_bytes);+ expect(survivor.extracted_at).toBe(WATERMARK.extracted_at);+ });++ test("an entry without watermark fields degrades to byte 0", () => {+ writeSource("Inbox/a.md", "legacy source");+ updateManifest(vault, ["Inbox/a.md"]);+ const entry = readManifest(vault).entries["Inbox/a.md"]!;+ expect(entry.extracted_bytes).toBeUndefined();+ expect(extractedBytesOf(entry)).toBe(0);+ });++ test("a malformed watermark shape drops the fields, never the entry", () => {+ // Lazy degradation: the malformed watermark is absent, which answers+ // byte 0, which reprocesses fully - without forfeiting the entry's+ // fingerprint (no contract-changed cascade for one bad field).+ writeSource("Inbox/a.md", "source bytes");+ const digest = hashFile(join(vault, "Inbox", "a.md"));+ writeManifestBytes(+ JSON.stringify({+ schema_version: 3,+ contract: { fingerprint: computeExtractionContractFingerprint(vault) },+ entries: {+ "Inbox/a.md": {+ sha256: digest,+ fingerprint: computeExtractionContractFingerprint(vault),+ extracted_bytes: "many",+ extracted_at: 42,+ extracted_bytes_offset: -1,+ },+ },+ }),+ );+ const manifest = readManifest(vault);+ expect(manifest.entries["Inbox/a.md"]).toEqual({+ sha256: digest,+ fingerprint: computeExtractionContractFingerprint(vault),+ });+ expect(manifest.contract).not.toBeNull();+ });++ test("unknown entry shapes still drop toward reprocessing exactly as today", () => {+ writeSource("Inbox/b.md", "source b");+ writeManifestBytes(+ JSON.stringify({+ schema_version: 3,+ contract: { fingerprint: computeExtractionContractFingerprint(vault) },+ entries: { "Inbox/b.md": { other: true } },+ }),+ );+ const res = classifyPaths(vault, ["Inbox/b.md"], readManifest(vault));+ expect(res.new).toEqual(["Inbox/b.md"]);+ });++ test("updateManifest records a caller-supplied full entry verbatim", () => {+ writeSource("Inbox/log.md", "first line\n");+ const live = computeExtractionContractFingerprint(vault);+ const digest = hashFile(join(vault, "Inbox", "log.md"));+ updateManifest(vault, ["Inbox/log.md"], {+ "Inbox/log.md": { sha256: digest, fingerprint: live, ...WATERMARK },+ });+ expect(readManifest(vault).entries["Inbox/log.md"]).toEqual({+ sha256: digest,+ fingerprint: live,+ ...WATERMARK,+ });+ });+});+describe("ingestSource integration", () => {const extraction = {entities: [{ category: "concept", name: "Restaking" }],diff --git a/tests/core/brain/ingest/incremental-extract.test.ts b/tests/core/brain/ingest/incremental-extract.test.tsnew file mode 100644index 00000000..ff2e03f2--- /dev/null+++ b/tests/core/brain/ingest/incremental-extract.test.ts@@ -0,0 +1,269 @@+/**+ * Incremental extraction plumbing (Lane 5, Task 5.5, t_a9cd4221).+ *+ * An ingest whose manifest record carries the extraction watermark reports+ * the APPENDED delta on the result and stamps the watermark beside the+ * digest; the CLI extract preview reports the appended slice metadata for a+ * previously-extracted source. The pinned protocol:+ *+ * - a first ingest stamps the FULL current size as the watermark (the+ * ingest contract is a whole-source extraction) and reports no delta;+ * - an appended source reports exactly the bytes appended since the+ * watermark, boundary-safe ({@link ./tail-read.ts}), and two appends+ * yield two deltas that never overlap;+ * - a REWRITTEN source - even one that grew - classifies modified and+ * reprocesses fully, resetting the watermark, because the recorded+ * digest no longer describes the leading bytes;+ * - EXTRACT_CONTRACT_VERSION stays at 2: the watermark changes what is+ * REPORTED, not what extraction produces, so no contract bump and no+ * one-time mass reprocess is owed.+ */++import { mkdirSync, mkdtempSync, rmSync, appendFileSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { afterEach, beforeEach, describe, expect, test } from "bun:test";++import { runCli } from "../../../helpers/run-cli.ts";+import { EXTRACT_CONTRACT_VERSION } from "../../../../src/core/brain/ingest/contract.ts";+import {+ classifyPaths,+ hashFile,+ hashPath,+ readManifest,+} from "../../../../src/core/brain/ingest/content-manifest.ts";+import { planBatches } from "../../../../src/core/brain/ingest/batch-plan.ts";+import { IS_WINDOWS } from "../../../helpers/platform.ts";+import { ingestSource } from "../../../../src/core/brain/ingest/ingest.ts";+import { PRE_EXTRACT_MAX_SOURCE_BYTES } from "../../../../src/core/brain/ingest/source-read.ts";++let tmp: string;+let vault: string;++const NOW = new Date("2026-06-13T12:00:00Z");+const SOURCE = "imports/chat-log.md";++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-incremental-extract-"));+ vault = join(tmp, "vault");+ mkdirSync(join(vault, "imports"), { recursive: true });+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function writeSource(contents: string): void {+ writeFileSync(join(vault, SOURCE), contents, "utf8");+}++const INPUT = { sourcePath: SOURCE, summary: "A chat log summary.", extraction: { entities: [] } };+const OPTS = { agent: "tester", now: NOW };++describe("ingestSource watermark stamping", () => {+ test("a first ingest stamps the full size and reports no delta", () => {+ writeSource("one\ntwo\n");+ const res = ingestSource(vault, INPUT, OPTS);+ expect(res.extractionDelta).toBeUndefined();++ const entry = readManifest(vault).entries[SOURCE]!;+ expect(entry.extracted_bytes).toBe(8);+ expect(entry.extracted_at).toBe("2026-06-13T12:00:00Z");+ expect(entry.sha256).toBe(hashFile(join(vault, SOURCE)));+ });++ test("an appended source reports the appended delta and advances the watermark", () => {+ writeSource("one\ntwo\n");+ ingestSource(vault, INPUT, OPTS);++ appendFileSync(join(vault, SOURCE), "three\n", "utf8");+ const res = ingestSource(vault, INPUT, OPTS);++ expect(res.extractionDelta).toEqual({ fromByte: 8, toByte: 14 });+ const entry = readManifest(vault).entries[SOURCE]!;+ expect(entry.extracted_bytes).toBe(14);+ expect(entry.sha256).toBe(hashFile(join(vault, SOURCE)));+ });++ test("two appends yield two correct deltas with no overlap", () => {+ writeSource("one\ntwo\n");+ ingestSource(vault, INPUT, OPTS);+ appendFileSync(join(vault, SOURCE), "three\n", "utf8");+ const second = ingestSource(vault, INPUT, OPTS);+ appendFileSync(join(vault, SOURCE), "four\n", "utf8");+ const third = ingestSource(vault, INPUT, OPTS);++ expect(second.extractionDelta).toEqual({ fromByte: 8, toByte: 14 });+ expect(third.extractionDelta).toEqual({ fromByte: 14, toByte: 19 });+ // The watermarks advanced stepwise; no delta covers another's bytes.+ expect(readManifest(vault).entries[SOURCE]!.extracted_bytes).toBe(19);+ });++ test("a rewritten source classifies modified, reprocesses fully, resets the watermark", () => {+ writeSource("one\ntwo\n");+ ingestSource(vault, INPUT, OPTS);+ appendFileSync(join(vault, SOURCE), "three\n", "utf8");+ ingestSource(vault, INPUT, OPTS);+ expect(readManifest(vault).entries[SOURCE]!.extracted_bytes).toBe(14);++ // A rewrite that HAPPENS TO GROW: the leading bytes no longer match the+ // recorded digest, so this is modified, never an append.+ writeSource("an entirely different document\n");+ const res = ingestSource(vault, INPUT, OPTS);++ expect(res.extractionDelta).toBeUndefined();+ const entry = readManifest(vault).entries[SOURCE]!;+ expect(entry.extracted_bytes).toBe(31);+ expect(entry.sha256).toBe(hashFile(join(vault, SOURCE)));+ });++ test("an appended tail with no complete line reports an empty delta at the watermark", () => {+ writeSource("one\ntwo\n");+ ingestSource(vault, INPUT, OPTS);+ appendFileSync(join(vault, SOURCE), "half a line", "utf8");++ const res = ingestSource(vault, INPUT, OPTS);+ expect(res.extractionDelta).toEqual({ fromByte: 8, toByte: 8 });+ // The run still covered the whole current source, so the stamp is full.+ expect(readManifest(vault).entries[SOURCE]!.extracted_bytes).toBe(19);+ });+});++describe("the watermark never moves the extraction contract", () => {+ test("EXTRACT_CONTRACT_VERSION stays at 2 and a watermarked source reprocesses nothing", () => {+ expect(EXTRACT_CONTRACT_VERSION).toBe(2);++ writeSource("one\ntwo\n");+ ingestSource(vault, INPUT, OPTS);++ // Byte-identical and contract-current: the planner plans nothing for it.+ const plan = planBatches(vault, "imports", { maxBatchBytes: 10_000, maxBatchFiles: 100 });+ expect(plan.contractChanged).toBe(false);+ const planned = plan.batches.flatMap((b) => b.files).map((f) => f.path);+ expect(planned).toEqual([]);++ // And classification still answers unchanged - the additive fields never+ // touched the byte question.+ const res = classifyPaths(vault, [SOURCE], readManifest(vault));+ expect(res.unchanged).toEqual([SOURCE]);+ });+});++describe("the CLI extract preview reports the appended slice", () => {+ test("a previously-extracted source answers its appended byte range", async () => {+ writeSource("one\ntwo\n");+ ingestSource(vault, INPUT, OPTS);+ appendFileSync(join(vault, SOURCE), "three\n", "utf8");++ const res = await runCli(["brain", "extract", join(vault, SOURCE), "--json", "--vault", vault]);+ expect(res.returncode).toBe(0);+ const payload = JSON.parse(res.stdout) as {+ extracted: boolean;+ appended?: { from_byte: number; to_byte: number };+ };+ expect(payload.appended).toEqual({ from_byte: 8, to_byte: 14 });+ });++ test("a source with no recorded watermark answers without an appended section", async () => {+ writeSource("plain note\n");+ const res = await runCli(["brain", "extract", join(vault, SOURCE), "--json", "--vault", vault]);+ expect(res.returncode).toBe(0);+ const payload = JSON.parse(res.stdout) as { appended?: unknown };+ expect(payload.appended).toBeUndefined();+ });++ test("an append between the two caps answers without a section, like the pipeline", async () => {+ // The appended tail sits past the ingest delta cap (1 MiB) but under+ // the HTML whole-source cap (8 MiB): reprocess-scale for a delta,+ // ordinary for a whole read.+ writeSource("one\ntwo\n");+ ingestSource(vault, INPUT, OPTS);+ appendFileSync(+ join(vault, SOURCE),+ `${"x".repeat(PRE_EXTRACT_MAX_SOURCE_BYTES)}\nmore tail\n`,+ "utf8",+ );++ const res = await runCli(["brain", "extract", join(vault, SOURCE), "--json", "--vault", vault]);+ expect(res.returncode).toBe(0);+ const payload = JSON.parse(res.stdout) as { appended?: unknown };+ // The preview refuses exactly what the pipeline refuses: it must+ // never report a slice the delta pass would not accept.+ expect(payload.appended).toBeUndefined();+ const second = ingestSource(vault, INPUT, OPTS);+ expect(second.extractionDelta).toBeUndefined();+ });++ test("a rewritten source that grew answers without a section, like the pipeline", async () => {+ writeSource("one\ntwo\n");+ ingestSource(vault, INPUT, OPTS);+ // Bigger than the watermark, but the leading bytes no longer match the+ // recorded digest: the pipeline classifies this modified and reports no+ // delta, so the preview must not label it an append either.+ writeSource("ONE\nTWO\nTHREE\n");+ const res = await runCli(["brain", "extract", join(vault, SOURCE), "--json", "--vault", vault]);+ expect(res.returncode).toBe(0);+ const payload = JSON.parse(res.stdout) as { appended?: unknown };+ expect(payload.appended).toBeUndefined();+ const second = ingestSource(vault, INPUT, OPTS);+ expect(second.extractionDelta).toBeUndefined();+ });+});++describe("a source that changes kind mid-run degrades fail-safe", () => {+ // The injection unlinks the just-written source file and replaces it with a+ // directory mid-run; Windows can deny that unlink while the writer's handle+ // is still live (EBUSY/EPERM), so the swap never lands there.+ test.skipIf(IS_WINDOWS)(+ "a source swapped for a directory after classification reprocesses with no crash",+ () => {+ writeSource("one\ntwo\n");+ const first = ingestSource(vault, INPUT, OPTS);++ // The caller's reach predicate is asked about the summary page only+ // after the intake has already classified the source trusted (its stat+ // saw a regular file) and just before the manifest stamp, so swapping+ // the source for a directory inside the callback lands the change in+ // the window between the classification stat and the pre-stamp reads.+ // The stamp block then stats a directory whose size sits past the+ // recorded watermark, and the append-proof read used to die on EISDIR -+ // after the summary pages were written, losing the run's result. The+ // sibling ENOENT window (the source deleted inside the same span,+ // between the block's existsSync and the stat/hash reads) is answered+ // by the same fail-safe catch; it has no deterministic public-API hook.+ let swapped = false;+ const second = ingestSource(vault, INPUT, {+ ...OPTS,+ readable: (rel) => {+ if (rel === first.summaryPath && !swapped) {+ swapped = true;+ rmSync(join(vault, SOURCE));+ mkdirSync(join(vault, SOURCE));+ }+ return true;+ },+ });++ expect(swapped).toBe(true);+ // No crash: the run answers, and a source that cannot be read as a file+ // reports no append.+ expect(second.summaryPath).toBe(first.summaryPath);+ expect(second.extractionDelta).toBeUndefined();+ // The stamp that landed describes the directory as it now is, so the+ // next classification sees a changed source and reprocesses fully+ // rather than trusting a stale file digest.+ const entry = readManifest(vault).entries[SOURCE]!;+ expect(entry.sha256).toBe(hashPath(join(vault, SOURCE)));+ expect(entry.extracted_bytes).toBeGreaterThan(0);++ // The restored file is not an append against that stamp: the source+ // reprocesses fully, and the stamp lands on the real file again.+ rmSync(join(vault, SOURCE), { recursive: true });+ writeSource("one\ntwo\n");+ const third = ingestSource(vault, INPUT, OPTS);+ expect(third.extractionDelta).toBeUndefined();+ expect(readManifest(vault).entries[SOURCE]!.extracted_bytes).toBe(8);+ },+ );+});diff --git a/tests/core/brain/ingest/tail-read.test.ts b/tests/core/brain/ingest/tail-read.test.tsnew file mode 100644index 00000000..c3ecce54--- /dev/null+++ b/tests/core/brain/ingest/tail-read.test.ts@@ -0,0 +1,168 @@+/**+ * The from-offset tail read (Lane 5, Task 5.3) - the substrate for+ * incremental extraction of appended source text.+ *+ * The contract pinned here:+ * - a delta returns ONLY the text appended since `fromByte`, its start+ * snapped forward past any partial UTF-8 sequence (a mid-codepoint+ * offset decodes cleanly, never as U+FFFD) and its end cut after the+ * LAST complete newline, so an unfinished trailing line is left for+ * the next delta instead of being covered twice;+ * - `fromByte`/`toByte` are the exact byte range the returned text+ * covers, so a watermark stamped at `toByte` never overlaps the next+ * delta and never claims bytes the text does not carry;+ * - a file that shrank below the offset falls back to the FULL content+ * with the shrink named (the full-reprocess direction);+ * - the read ceiling is enforced with the existing bounded-read refusal+ * vocabulary (`larger than the read limit`), and a non-regular file+ * refuses the same way.+ */++import { mkdirSync, mkdtempSync, writeFileSync, appendFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { afterEach, beforeEach, describe, expect, test } from "bun:test";++import { readTail } from "../../../../src/core/brain/ingest/tail-read.ts";++let tmp: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-tail-read-"));+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function write(name: string, contents: string): string {+ const path = join(tmp, name);+ writeFileSync(path, contents, "utf8");+ return path;+}++describe("readTail - appended deltas", () => {+ test("returns the appended text from the offset, byte range exact", () => {+ const path = write("log.txt", "line1\nline2\n");+ const tail = readTail(path, 6);+ expect(tail).toEqual({+ kind: "appended",+ text: "line2\n",+ fromByte: 6,+ toByte: 12,+ });+ });++ test("from byte 0 the delta is the complete-line content", () => {+ const path = write("log.txt", "line1\nline2\n");+ expect(readTail(path, 0)).toEqual({+ kind: "appended",+ text: "line1\nline2\n",+ fromByte: 0,+ toByte: 12,+ });+ });++ test("an unfinished trailing line is left for the next delta", () => {+ const path = write("log.txt", "a\nb\ncde");+ expect(readTail(path, 0)).toEqual({+ kind: "appended",+ text: "a\nb\n",+ fromByte: 0,+ toByte: 4,+ });+ });++ test("an appended range with no complete line covers nothing", () => {+ const path = write("log.txt", "abc\nXYZ");+ expect(readTail(path, 4)).toEqual({+ kind: "appended",+ text: "",+ fromByte: 4,+ toByte: 4,+ });+ });++ test("an offset at end of file covers nothing", () => {+ const path = write("log.txt", "abc\n");+ expect(readTail(path, 4)).toEqual({+ kind: "appended",+ text: "",+ fromByte: 4,+ toByte: 4,+ });+ });++ test("a mid-codepoint offset snaps forward to a UTF-8-safe boundary", () => {+ // h(1) é(2) l l o \n - offset 2 lands inside the two-byte é.+ const path = write("log.txt", "héllo\n");+ const tail = readTail(path, 2);+ expect(tail.kind).toBe("appended");+ if (tail.kind !== "appended") return;+ expect(tail.text).toBe("llo\n");+ expect(tail.fromByte).toBe(3);+ expect(tail.toByte).toBe(7);+ expect(tail.text.includes("\uFFFD")).toBe(false);+ });++ test("multi-byte characters inside the delta decode cleanly", () => {+ const appended = "café — naïve 䏿–‡ \u{1F600}\nsecond line\n";+ const path = write("log.txt", "base\n");+ appendFileSync(path, appended, "utf8");+ const tail = readTail(path, 5);+ expect(tail).toEqual({+ kind: "appended",+ text: appended,+ fromByte: 5,+ toByte: 5 + Buffer.byteLength(appended, "utf8"),+ });+ });++ test("two appends yield two non-overlapping deltas whose text is exactly the appends", () => {+ const path = write("log.txt", "one\n");+ appendFileSync(path, "two\n", "utf8");+ const first = readTail(path, 4);+ expect(first).toEqual({ kind: "appended", text: "two\n", fromByte: 4, toByte: 8 });++ appendFileSync(path, "three\n", "utf8");+ const second = readTail(path, 8);+ expect(second).toEqual({ kind: "appended", text: "three\n", fromByte: 8, toByte: 14 });+ });+});++describe("readTail - shrink fallback", () => {+ test("a file that shrank below the offset falls back to the full content, named", () => {+ const path = write("log.txt", "rewritten small\n");+ const tail = readTail(path, 1024);+ expect(tail).toEqual({+ kind: "shrank",+ text: "rewritten small\n",+ fromByte: 0,+ toByte: 16,+ });+ });++ test("a shrunken file still refuses past the ceiling", () => {+ const path = write("log.txt", "x".repeat(2048));+ const tail = readTail(path, 4096, { maxBytes: 1024 });+ expect(tail).toEqual({ kind: "unread", unread: "larger than the read limit" });+ });+});++describe("readTail - bounded ceiling and refusals", () => {+ test("an appended range past the ceiling refuses with the bounded-read vocabulary", () => {+ const path = write("log.txt", "base\n");+ appendFileSync(path, "y".repeat(2048) + "\n", "utf8");+ expect(readTail(path, 5, { maxBytes: 1024 })).toEqual({+ kind: "unread",+ unread: "larger than the read limit",+ });+ });++ test("a directory refuses as not a regular file", () => {+ const dir = join(tmp, "dir");+ mkdirSync(dir, { recursive: true });+ expect(readTail(dir, 0)).toEqual({ kind: "unread", unread: "not a regular file" });+ });+});diff --git a/tests/core/brain/maintenance-capacity-gate.test.ts b/tests/core/brain/maintenance-capacity-gate.test.tsnew file mode 100644index 00000000..427f1bea--- /dev/null+++ b/tests/core/brain/maintenance-capacity-gate.test.ts@@ -0,0 +1,459 @@+/**+ * The maintenance lane's capacity gate (storage lifecycle, Lane 2): the+ * FIFTH gate, ordered after host pressure, active only when the vault's+ * `storage:` block configures `cap_bytes`.+ *+ * A critical tier skips the whole lane with its own verdict+ * (`skipped:capacity`), the reading rides the decision and the journal+ * the way the pressure reading does, an unmeasurable store leaves the+ * gate OPEN with the reason journalled (a vault that cannot be measured+ * must not look like one that paused), and `--force` bypasses it like+ * every other soft gate while the lease is never bypassed by anything.+ *+ * The usage sample is taken BEFORE the skip is returned: the forecast+ * series must keep accumulating exactly while producers are paused, so a+ * skipped run still appends its sample. A run the gate lets through+ * samples nothing here - the gate decision already measured, and a lane+ * pass that doubles every byte walk would tax every scheduled run for a+ * datum the storage report surfaces already collect.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { appendFileSync, existsSync, mkdirSync, mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { evaluateGates, runMaintenance } from "../../../src/core/brain/maintenance/lane.ts";+import { DERIVED_STORE_DIR } from "../../../src/core/brain/path-constants.ts";+import {+ appendJournal,+ listJournal,+ MAINTENANCE_VERDICT,+ type MaintenanceJournalEntry,+} from "../../../src/core/brain/maintenance/journal.ts";+import { acquireLease, currentLease } from "../../../src/core/brain/maintenance/lease.ts";+import { BRAIN_STORAGE_DEFAULTS } from "../../../src/core/brain/policy/blocks/storage.ts";+import {+ STORAGE_FORECAST_TIER,+ STORAGE_USAGE_METRIC_SURFACE,+} from "../../../src/core/brain/storage/forecast.ts";+import type { CapacityGateDecision } from "../../../src/core/brain/storage/gate.ts";+import type { ResolvedBrainStorageConfig } from "../../../src/core/brain/types.ts";+import { listMetrics } from "../../../src/core/brain/metrics.ts";+import { bootstrapBrain } from "../../../src/core/brain/init.ts";+import { brainConfigPath } from "../../../src/core/brain/paths.ts";+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";+import { emitRecallTelemetry, RECALL_CHANNEL } from "../../../src/core/brain/recall-telemetry.ts";+import { TRANSPORT_REACH } from "../../../src/core/graph/transport-reach.ts";+import { JSONRPC_VERSION, MCPServer, PROTOCOL_VERSION } from "../../../src/mcp/index.ts";+import { homeEnv } from "../../helpers/platform.ts";+import { runCli } from "../../helpers/run-cli.ts";++const NOW = new Date("2026-06-05T03:30:00Z");+/** The metrics surface Lane 1's `sampleStorageUsage` appends one record per call under. */+const STORAGE_USAGE_SURFACE = STORAGE_USAGE_METRIC_SURFACE;+/**+ * A cap every real vault clears many times over: whatever the bootstrapped+ * fixture holds, its usage is far past 95 percent of this, and a tier past+ * 100 percent of cap is still `critical` - the top measured tier.+ */+const CRITICAL_CAP_BYTES = 1000;+/** A cap no test fixture approaches, so the tier stays below the warn band. */+const ROOMY_CAP_BYTES = 1_000_000_000_000;++let vault: string;+let configHome: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-maint-cap-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-maint-cap-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);+ mkdirSync(join(vault, "Brain"), { recursive: true });+ bootstrapBrain(vault, { configPath });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++/** Configure the vault-side capacity gate, the way the pressure test does. */+function configureCap(capBytes: number): void {+ appendFileSync(brainConfigPath(vault), `\nstorage:\n cap_bytes: ${capBytes}\n`);+}++/** The resolved storage block a test names for the gate, defaults elsewhere. */+function capGate(capBytes: number): ResolvedBrainStorageConfig {+ return { ...BRAIN_STORAGE_DEFAULTS, cap_bytes: capBytes };+}++/**+ * A hand-built gate decision for the injected-reader tests. The lane must+ * treat whatever its reader returns as the decision of record, so these+ * fixtures are plain objects carrying only the fields the lane reads.+ */+function gateDecision(tier: string, skip: boolean, reason?: string): CapacityGateDecision {+ return {+ decision: skip ? "skip" : "run",+ tier,+ ...(reason !== undefined ? { reason } : {}),+ } as CapacityGateDecision;+}++function samples(): number {+ return listMetrics(vault, { surface: STORAGE_USAGE_SURFACE }).length;+}++/** The lease database the lane must not create before its gates decide. */+function leaseDb(vault: string): string {+ return join(vault, DERIVED_STORE_DIR, "maintenance.sqlite");+}++function journalRows(verdict: string): ReadonlyArray<MaintenanceJournalEntry> {+ return listJournal(vault).filter((e) => e.verdict === verdict);+}++describe("evaluateGates: the capacity gate", () => {+ test("an unconfigured gate never fires and never measures", () => {+ let measurements = 0;+ const decision = evaluateGates(vault, {+ now: NOW,+ readCapacity: () => {+ measurements += 1;+ return gateDecision(STORAGE_FORECAST_TIER.critical, true);+ },+ });+ expect(decision.verdict).toBe(MAINTENANCE_VERDICT.run);+ expect(decision.capacity).toBeUndefined();+ expect(measurements).toBe(0);+ });++ test("a critical tier skips with the reading carried alongside", () => {+ const critical = gateDecision(STORAGE_FORECAST_TIER.critical, true);+ const decision = evaluateGates(vault, {+ now: NOW,+ capacity: capGate(CRITICAL_CAP_BYTES),+ readCapacity: () => critical,+ });+ expect(decision.verdict).toBe(MAINTENANCE_VERDICT.skippedCapacity);+ expect(decision.capacity).toEqual(critical);+ });++ test("a sub-critical tier leaves the gate open with the reading attached", () => {+ const ok = gateDecision(STORAGE_FORECAST_TIER.ok, false);+ const decision = evaluateGates(vault, {+ now: NOW,+ capacity: capGate(ROOMY_CAP_BYTES),+ readCapacity: () => ok,+ });+ expect(decision.verdict).toBe(MAINTENANCE_VERDICT.run);+ expect(decision.capacity).toEqual(ok);+ });++ test("an unmeasurable store leaves the gate open", () => {+ const unmeasurable = gateDecision(STORAGE_FORECAST_TIER.unmeasurable, false, "test-reason");+ const decision = evaluateGates(vault, {+ now: NOW,+ capacity: capGate(CRITICAL_CAP_BYTES),+ readCapacity: () => unmeasurable,+ });+ // A threshold that would skip on any real reading stays open, because+ // nothing was measured - the same honesty the pressure gate practices.+ expect(decision.verdict).toBe(MAINTENANCE_VERDICT.run);+ expect(decision.capacity).toEqual(unmeasurable);+ });++ test("an earlier closed gate wins and the capacity gate is never measured", () => {+ let measurements = 0;+ const decision = evaluateGates(vault, {+ now: NOW,+ window: { startHour: 10, endHour: 12, tz: "UTC" },+ capacity: capGate(CRITICAL_CAP_BYTES),+ readCapacity: () => {+ measurements += 1;+ return gateDecision(STORAGE_FORECAST_TIER.critical, true);+ },+ });+ expect(decision.verdict).toBe(MAINTENANCE_VERDICT.skippedWindow);+ expect(decision.capacity).toBeUndefined();+ expect(measurements).toBe(0);+ });++ test("a busy vault is answered before the byte walk", () => {+ for (let i = 0; i < 5; i++) {+ emitRecallTelemetry(vault, {+ host: "claude-code",+ channel: RECALL_CHANNEL.cli,+ mode: "search",+ status: "ok",+ durationMs: 12,+ resultCount: 3,+ createdAt: new Date(NOW.getTime() - 60_000).toISOString(),+ });+ }+ let measurements = 0;+ const decision = evaluateGates(vault, {+ now: NOW,+ busy: { minutes: 10, threshold: 5 },+ capacity: capGate(CRITICAL_CAP_BYTES),+ readCapacity: () => {+ measurements += 1;+ return gateDecision(STORAGE_FORECAST_TIER.critical, true);+ },+ });+ expect(decision.verdict).toBe(MAINTENANCE_VERDICT.skippedBusy);+ expect(measurements).toBe(0);+ });+});++describe("the capacity gate through the lane", () => {+ test("a critical vault skips with its own verdict and journals the tier", async () => {+ configureCap(CRITICAL_CAP_BYTES);+ const result = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [+ {+ name: "dream",+ run: async () => {+ throw new Error("the capacity gate should have refused before this ran");+ },+ },+ ],+ });+ expect(result.verdict).toBe(MAINTENANCE_VERDICT.skippedCapacity);+ expect(result.tasks).toEqual([]);+ const row = journalRows(MAINTENANCE_VERDICT.skippedCapacity)[0];+ expect(row?.holder).toBe("worker-a");+ expect(row?.capacity_tier).toBe(STORAGE_FORECAST_TIER.critical);+ // The reading rides the row the way the pressure percent does: the+ // measured bytes, the configured cap, and the percent between them.+ expect(row?.capacity_cap_bytes).toBe(CRITICAL_CAP_BYTES);+ expect(row?.capacity_measured_bytes).toBeGreaterThan(0);+ expect(row?.capacity_percent).toBe((row!.capacity_measured_bytes! / CRITICAL_CAP_BYTES) * 100);+ });++ test("the sample is taken before the skip returns", async () => {+ configureCap(CRITICAL_CAP_BYTES);+ const before = samples();+ await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [{ name: "dream", run: async () => {} }],+ });+ // Exactly one: the skip's own sample, not one per task or per gate.+ expect(samples()).toBe(before + 1);+ });++ test("the skip's sample records the gate's census, not a second walk", async () => {+ configureCap(CRITICAL_CAP_BYTES);+ // A decision whose census names a byte total the real vault cannot+ // produce: if the lane sampled by re-measuring, the recorded sample+ // would carry the vault's real size instead - a second full walk+ // behind every skipped run.+ const census: CapacityGateDecision = {+ decision: "skip",+ tier: STORAGE_FORECAST_TIER.critical,+ reading: {+ measuredBytes: 777,+ capBytes: CRITICAL_CAP_BYTES,+ percentUsed: (777 / CRITICAL_CAP_BYTES) * 100,+ daysToCap: null,+ slopeBytesPerDay: null,+ measuredAt: NOW.toISOString(),+ },+ census: {+ measuredAt: NOW.toISOString(),+ complete: true,+ surfaces: [],+ bytesByTier: { derived: 0, "vault-content": 777 },+ totalBytes: 777,+ unmeasured: [],+ },+ };+ const before = samples();+ await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ readCapacity: () => census,+ tasks: [{ name: "dream", run: async () => {} }],+ });+ const recorded = listMetrics(vault, { surface: STORAGE_USAGE_SURFACE });+ expect(recorded).toHaveLength(before + 1);+ expect(recorded[0]!.payload["total_bytes"]).toBe(777);+ });++ test("a capacity-skipped run never touches the lease", async () => {+ configureCap(CRITICAL_CAP_BYTES);+ const result = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [{ name: "dream", run: async () => {} }],+ });+ expect(result.verdict).toBe(MAINTENANCE_VERDICT.skippedCapacity);+ // The fifth gate's order, pinned on disk: the capacity gate decides+ // BEFORE the lease is acquired, so a skipped run never even creates+ // the lease database. Lease acquisition moved ahead of the gate and+ // this file would come into existence behind every skip.+ expect(existsSync(leaseDb(vault))).toBe(false);+ expect(existsSync(`${leaseDb(vault)}-wal`)).toBe(false);+ // The verdict row is the gate's own, never a lease loss.+ expect(journalRows(MAINTENANCE_VERDICT.skippedLease)).toEqual([]);+ });++ test("a sub-critical vault runs and samples nothing", async () => {+ configureCap(ROOMY_CAP_BYTES);+ const before = samples();+ const result = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [{ name: "dream", run: async () => {} }],+ });+ expect(result.verdict).toBe(MAINTENANCE_VERDICT.run);+ expect(result.tasks[0]?.ok).toBe(true);+ expect(samples()).toBe(before);+ });++ test("a vault without a cap never measures", async () => {+ const before = samples();+ const result = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [{ name: "dream", run: async () => {} }],+ });+ expect(result.verdict).toBe(MAINTENANCE_VERDICT.run);+ expect(samples()).toBe(before);+ expect(journalRows(MAINTENANCE_VERDICT.skippedCapacity)).toEqual([]);+ });++ test("an unmeasurable store journals a notice on its own line and still runs", async () => {+ configureCap(CRITICAL_CAP_BYTES);+ const ran: string[] = [];+ const result = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ readCapacity: () =>+ gateDecision(STORAGE_FORECAST_TIER.unmeasurable, false, "test-unreadable"),+ tasks: [{ name: "dream", run: async () => void ran.push("dream") }],+ });+ expect(result.verdict).toBe(MAINTENANCE_VERDICT.run);+ expect(ran).toEqual(["dream"]);++ const journal = listJournal(vault);+ const notice = journal.find((e) => e.verdict === MAINTENANCE_VERDICT.capacityUnmeasurable);+ // Two different lines, never one: the notice that the gate could not+ // evaluate is not the row of the work that then went ahead.+ expect(notice?.capacity_reason).toBe("test-unreadable");+ expect(notice?.task).toBeUndefined();+ expect(journal.some((e) => e.verdict === MAINTENANCE_VERDICT.skippedCapacity)).toBe(false);+ expect(journal.some((e) => e.verdict === MAINTENANCE_VERDICT.run && e.task === "dream")).toBe(+ true,+ );+ });++ test("--force bypasses the capacity gate, and the lease is never bypassed", async () => {+ configureCap(CRITICAL_CAP_BYTES);+ const forced = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ force: true,+ tasks: [{ name: "dream", run: async () => {} }],+ });+ expect(forced.verdict).toBe(MAINTENANCE_VERDICT.run);+ expect(forced.tasks[0]?.ok).toBe(true);+ expect(journalRows(MAINTENANCE_VERDICT.skippedCapacity)).toEqual([]);++ acquireLease(vault, { holder: "other-worker", ttlMs: 600_000, now: NOW });+ const held = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ force: true,+ tasks: [{ name: "dream", run: async () => {} }],+ });+ expect(held.verdict).toBe(MAINTENANCE_VERDICT.skippedLease);+ expect(currentLease(vault, { now: NOW })?.holder).toBe("other-worker");+ });+});++/** The capacity block both lane front doors attach to a skipped run. */+interface CapacityReportJson {+ readonly tier: string;+ readonly remedy: string;+ readonly capacity_percent: number | null;+ readonly measured_bytes: number | null;+ readonly cap_bytes: number | null;+}++describe("lane surfaces carry the capacity verdict", () => {+ function seedSkipRow(): void {+ appendJournal(vault, {+ ts: NOW.toISOString(),+ holder: "worker-a",+ verdict: MAINTENANCE_VERDICT.skippedCapacity,+ capacity_tier: STORAGE_FORECAST_TIER.critical,+ });+ }++ test("status parses and renders a skipped:capacity row on both surfaces", async () => {+ seedSkipRow();+ const cli = await runCli(["brain", "maintenance", "status", "--vault", vault], {+ env: { OPEN_SECOND_BRAIN_CONFIG: join(configHome, "config.yaml"), ...homeEnv(configHome) },+ });+ expect(cli.returncode).toBe(0);+ expect(cli.stdout).toContain(MAINTENANCE_VERDICT.skippedCapacity);+ });++ test("a skipped-capacity run reports the tier and the remedy on the CLI", async () => {+ configureCap(CRITICAL_CAP_BYTES);+ const cli = await runCli(["brain", "maintenance", "run", "--vault", vault, "--json"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: join(configHome, "config.yaml"), ...homeEnv(configHome) },+ });+ expect(cli.returncode).toBe(0);+ const payload = JSON.parse(cli.stdout) as {+ verdict: string;+ capacity?: CapacityReportJson;+ };+ expect(payload.verdict).toBe(MAINTENANCE_VERDICT.skippedCapacity);+ expect(payload.capacity?.tier).toBe(STORAGE_FORECAST_TIER.critical);+ expect(payload.capacity?.remedy.length).toBeGreaterThan(0);+ // The gate's full reading rides the report, not the tier alone.+ expect(payload.capacity?.cap_bytes).toBe(CRITICAL_CAP_BYTES);+ expect(payload.capacity?.measured_bytes).toBeGreaterThan(0);+ expect(payload.capacity?.capacity_percent).toBe(+ (payload.capacity!.measured_bytes! / CRITICAL_CAP_BYTES) * 100,+ );+ });++ test("a skipped-capacity run reports the tier and the remedy over MCP", async () => {+ configureCap(CRITICAL_CAP_BYTES);+ const server = new MCPServer(+ { vault, configPath: join(configHome, "config.yaml") },+ { reach: TRANSPORT_REACH.local },+ );+ await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 1,+ method: "initialize",+ params: { protocolVersion: PROTOCOL_VERSION },+ });+ const res = (await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 2,+ method: "tools/call",+ params: { name: "brain_maintenance", arguments: { operation: "run" } },+ })) as { result?: { structuredContent?: Record<string, unknown> } };+ const payload = (res.result?.structuredContent ?? {}) as {+ verdict?: string;+ capacity?: CapacityReportJson;+ };+ expect(payload.verdict).toBe(MAINTENANCE_VERDICT.skippedCapacity);+ expect(payload.capacity?.tier).toBe(STORAGE_FORECAST_TIER.critical);+ expect(payload.capacity?.remedy.length).toBeGreaterThan(0);+ expect(payload.capacity?.cap_bytes).toBe(CRITICAL_CAP_BYTES);+ expect(payload.capacity?.measured_bytes).toBeGreaterThan(0);+ expect(payload.capacity?.capacity_percent).toBe(+ (payload.capacity!.measured_bytes! / CRITICAL_CAP_BYTES) * 100,+ );+ });+});diff --git a/tests/core/brain/maintenance-collect.test.ts b/tests/core/brain/maintenance-collect.test.tsnew file mode 100644index 00000000..9e852fa2--- /dev/null+++ b/tests/core/brain/maintenance-collect.test.ts@@ -0,0 +1,103 @@+/**+ * The maintenance signal collection is fail-soft per signal, and the+ * failures it absorbs are NAMED (Lane 5, Task 5.2): each failed scan is+ * reported to the `onError` observer as {signal, message} while the+ * remaining signals still collect and the actions still return. Before the+ * decomposition a failed signal was swallowed by a bare catch that said+ * nothing, so a thin maintenance report looked exactly like a healthy one.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ collectMaintenanceActions,+ type MaintenanceCollectError,+} from "../../../src/core/brain/maintenance/collect.ts";++let tmp: string;+let vault: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-maintenance-collect-"));+ vault = join(tmp, "vault");+ mkdirSync(join(vault, "Brain", "preferences"), { recursive: true });+ mkdirSync(join(vault, "Brain", "retired"), { recursive: true });+ mkdirSync(join(vault, "Brain", "inbox"), { recursive: true });+ mkdirSync(join(vault, "Brain", "log"), { recursive: true });+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++describe("collectMaintenanceActions", () => {+ test("a healthy vault collects actions and observes no errors", () => {+ writePref("a", "the rule", "see also [[pref-b]]");+ writePref("b", "the rule", "duplicate of the rule");++ const errors: MaintenanceCollectError[] = [];+ const actions = collectMaintenanceActions(vault, { onError: (e) => errors.push(e) });++ expect(errors).toEqual([]);+ expect(actions.some((a) => a.category === "dedup")).toBe(true);+ });++ test("a failed dedup scan is observed by name; the run still completes", () => {+ // The dedup scan walks Brain/preferences; a FILE at that path makes the+ // readdir refuse on every platform, no chmod needed.+ rmSync(join(vault, "Brain", "preferences"), { recursive: true });+ writeFileSync(join(vault, "Brain", "preferences"), "not a directory", "utf8");++ const errors: MaintenanceCollectError[] = [];+ const actions = collectMaintenanceActions(vault, { onError: (e) => errors.push(e) });++ const dedup = errors.find((e) => e.signal === "dedup");+ expect(dedup).toBeDefined();+ // Windows error text carries the path with the platform separator.+ expect(dedup!.message.replace(/\\/g, "/")).toContain("Brain/preferences");+ // The run completes and still answers with the actions array.+ expect(Array.isArray(actions)).toBe(true);+ });++ test("each failed signal is observed separately", () => {+ rmSync(join(vault, "Brain", "preferences"), { recursive: true });+ writeFileSync(join(vault, "Brain", "preferences"), "not a directory", "utf8");+ // The lint pass reads the inbox; a file at that path fails it too.+ rmSync(join(vault, "Brain", "inbox"), { recursive: true });+ writeFileSync(join(vault, "Brain", "inbox"), "not a directory", "utf8");++ const errors: MaintenanceCollectError[] = [];+ collectMaintenanceActions(vault, { onError: (e) => errors.push(e) });++ const signals = errors.map((e) => e.signal).toSorted();+ expect(signals).toContain("dedup");+ expect(signals).toContain("lint");+ // Every observed failure carries a message.+ for (const e of errors) expect(e.message.length).toBeGreaterThan(0);+ });+});++function writePref(slug: string, principle: string, body: string): void {+ writeFileSync(+ join(vault, "Brain", "preferences", `pref-${slug}.md`),+ [+ "---",+ "kind: brain-preference",+ `id: pref-${slug}`,+ "tags: [brain, brain/preference]",+ "topic: em-dashes",+ "_status: confirmed",+ `principle: ${principle}`,+ "unconfirmed_until: 2026-01-15T00:00:00Z",+ "created_at: 2026-01-01T00:00:00Z",+ "---",+ "",+ body,+ "",+ ].join("\n"),+ "utf8",+ );+}diff --git a/tests/core/brain/policy-safe-loaders.test.ts b/tests/core/brain/policy-safe-loaders.test.tsindex 9eebac1a..7fc5c0ce 100644--- a/tests/core/brain/policy-safe-loaders.test.ts+++ b/tests/core/brain/policy-safe-loaders.test.ts@@ -44,7 +44,9 @@ import {loadNotesConfigSafe,loadSnapshotDerivedStorePolicySafe,loadSnapshotRetentionSafe,+ loadStorageConfigSafe,loadTemporalConfigSafe,+ BRAIN_STORAGE_DEFAULTS,} from "../../../src/core/brain/policy.ts";/**@@ -117,6 +119,15 @@ const SAFE_LOADERS = [maxBytes: DEFAULT_BRAIN_CONFIG.snapshots.derived_store_max_bytes,} as unknown,},+ {+ // The resolved cap and thresholds are what the capacity gate and the+ // forecast read, so serving the defaults on a broken config would+ // measure a vault against a cap the operator never wrote - and skip,+ // or run, heavy work on the strength of a file nothing could parse.+ name: "loadStorageConfigSafe",+ load: loadStorageConfigSafe as (vault: string) => unknown,+ absentValue: BRAIN_STORAGE_DEFAULTS as unknown,+ },] as const;// ----- Reverse census: the table must account for every reader ---------------diff --git a/tests/core/brain/receipts-compact.test.ts b/tests/core/brain/receipts-compact.test.tsnew file mode 100644index 00000000..4afb2d34--- /dev/null+++ b/tests/core/brain/receipts-compact.test.ts@@ -0,0 +1,549 @@+/**+ * Receipt fold engine (storage-lifecycle, Lane 3 task 3.2).+ *+ * `compactReceiptLedgers` rewrites explicitly named receipt-ledger+ * shards: records older than the age threshold fold into one aggregate+ * line per (kind, month) preserving count, first/last timestamps and the+ * folded records' byte total; protected records survive verbatim, in+ * file order; unknown kinds fail closed to keep. The rewrite is atomic+ * per shard (compare-and-swap through `atomicWriteFileSync`) inside a+ * `withDestructiveSnapshot` span, and per-shard fail-soft: a poisoned+ * shard is collected and the run still completes with an honest audit.+ *+ * The audit is the additive `receipt_compaction` continuity kind: one+ * record per confirmed run, never on a dry run. A count-guard mismatch+ * aborts before any write, including the snapshot.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import {+ chmodSync,+ mkdirSync,+ mkdtempSync,+ readFileSync,+ readdirSync,+ rmSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ continuityLogDir,+ continuityLogPath,+ listContinuityRecords,+} from "../../../src/core/brain/continuity/store.ts";+import { metricsDir } from "../../../src/core/brain/metrics.ts";+import {+ compactReceiptLedgers,+ RECEIPT_AUDIT_APPEND_FAILED,+} from "../../../src/core/brain/receipts/compact.ts";+import { RECEIPT_SHARD_UNREADABLE } from "../../../src/core/brain/receipts/scan.ts";+import {+ RECEIPT_FOLD_SCHEMA,+ RECEIPT_LEDGER_FAMILY,+} from "../../../src/core/brain/receipts/types.ts";+import { CountGuardError } from "../../../src/core/brain/count-guard.ts";+import { listSnapshots } from "../../../src/core/brain/snapshot.ts";+import { bootstrapBrain } from "../../../src/core/brain/init.ts";+import { resetVaultIdentityPins } from "../../../src/core/brain/vault-identity.ts";+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";+import { WriteRefusedError } from "../../../src/core/brain/write-disposition.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";+import { digestVaultTree } from "../../helpers/vault-digest.ts";++const NOW = new Date("2026-09-25T00:00:00Z");++let vault: string;+let configHome: string;++function continuityLine(+ kind: string,+ createdAt: string,+ payload: Record<string, unknown> = {},+): string {+ const id = `ctn_${createdAt.replace(/[^0-9]/g, "").slice(0, 14)}_${kind}`;+ return `${JSON.stringify({+ schema: "o2b.continuity.v1",+ id,+ kind,+ createdAt,+ sourceRefs: [],+ payload,+ private: false,+ redacted: false,+ })}\n`;+}++function metricLine(surface: string, runAt: string, payload: Record<string, unknown> = {}): string {+ return `${JSON.stringify({ schema: "o2b.metrics.v1", surface, run_at: runAt, payload })}\n`;+}++function writeShard(dir: string, name: string, lines: ReadonlyArray<string> | string): void {+ const text = typeof lines === "string" ? lines : lines.join("");+ writeFileSync(join(dir, name), text, "utf8");+}++function readShard(dir: string, name: string): string {+ return readFileSync(join(dir, name), "utf8");+}++/** The continuity shard the happy-path tests fold: two old, one protected, one young. */+const MIXED_SHARD = [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z", { note: "first old" }),+ continuityLine("source_invalidation", "2026-08-11T12:00:00Z", { reason: "audit trail" }),+ continuityLine("recall_telemetry", "2026-08-20T12:00:00Z", { note: "second old" }),+ continuityLine("gate_telemetry", "2026-09-15T12:00:00Z", { fresh: true }),+].join("");++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-receipts-compact-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-receipts-compact-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: test-agent\n`);+ bootstrapBrain(vault, { configPath });+ mkdirSync(continuityLogDir(vault), { recursive: true });+ mkdirSync(metricsDir(vault), { recursive: true });+});++afterEach(() => {+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++describe("compactReceiptLedgers", () => {+ test("folds old records into one aggregate line per (kind, month), preserving the totals", () => {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ expect: 2,+ });+ expect(result.dryRun).toBe(false);+ expect(result.totals.folded).toBe(2);+ const after = readShard(continuityLogDir(vault), "2026-08.jsonl");+ const lines = after.split("\n").filter((line) => line.trim() !== "");+ // Two folded records became one aggregate line; protected and young kept.+ expect(lines).toHaveLength(3);+ const aggregate = JSON.parse(lines[0]!) as {+ schema: string;+ fold: Record<string, unknown>;+ };+ expect(aggregate.schema).toBe(RECEIPT_FOLD_SCHEMA);+ expect(aggregate.fold).toMatchObject({+ family: RECEIPT_LEDGER_FAMILY.continuity,+ kind: "recall_telemetry",+ month: "2026-08",+ count: 2,+ first_created_at: "2026-08-10T12:00:00Z",+ last_created_at: "2026-08-20T12:00:00Z",+ });+ // The folded records' byte total is preserved as metadata (record+ // lines, newline separators excluded).+ const foldedBytes =+ Buffer.byteLength(+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z", { note: "first old" }).trimEnd(),+ ) ++ Buffer.byteLength(+ continuityLine("recall_telemetry", "2026-08-20T12:00:00Z", {+ note: "second old",+ }).trimEnd(),+ );+ expect(aggregate.fold.bytes).toBe(foldedBytes);+ // Shrink verified: the rewrite is strictly smaller than the original.+ expect(Buffer.byteLength(after)).toBeLessThan(Buffer.byteLength(MIXED_SHARD));+ expect(result.totals.bytesBefore).toBe(Buffer.byteLength(MIXED_SHARD));+ expect(result.totals.bytesAfter).toBe(Buffer.byteLength(after));+ });++ test("protected records survive verbatim, in file order, and unknown kinds fail closed to keep", () => {+ const shard = [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),+ continuityLine("source_invalidation", "2026-08-11T12:00:00Z", { reason: "audit" }),+ continuityLine("kind_from_the_future", "2026-08-12T12:00:00Z"),+ continuityLine("gate_telemetry", "2026-08-13T12:00:00Z", { session_id: "sess-1" }),+ continuityLine("recall_telemetry", "2026-08-14T12:00:00Z"),+ ].join("");+ writeShard(continuityLogDir(vault), "2026-08.jsonl", shard);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.totals.folded).toBe(2);+ expect(result.totals.kept).toBe(3);+ const after = readShard(continuityLogDir(vault), "2026-08.jsonl");+ // Every kept line is byte-present, and their relative order holds.+ const keptLines = [+ continuityLine("source_invalidation", "2026-08-11T12:00:00Z", { reason: "audit" }),+ continuityLine("kind_from_the_future", "2026-08-12T12:00:00Z"),+ continuityLine("gate_telemetry", "2026-08-13T12:00:00Z", { session_id: "sess-1" }),+ ];+ let cursor = -1;+ for (const line of keptLines) {+ const at = after.indexOf(line.trim());+ expect(at).toBeGreaterThan(cursor);+ cursor = at;+ }+ });++ test("the aggregate's first and last compare by instant, not by stamp text", () => {+ // Both stamps pass the canonical grammar, which admits optional+ // fractional seconds - and "." sorts before "Z", so a text compare+ // puts the LATER half-second stamp first. Written later-instant-first+ // to make the text order and the instant order disagree.+ const shard = [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00.500Z", { note: "later half" }),+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z", { note: "earlier whole" }),+ ].join("");+ writeShard(continuityLogDir(vault), "2026-08.jsonl", shard);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ expect: 2,+ });+ expect(result.totals.folded).toBe(2);+ const aggregate = JSON.parse(+ readShard(continuityLogDir(vault), "2026-08.jsonl")+ .split("\n")+ .find((line) => line.trim() !== "")!,+ ) as { fold: Record<string, unknown> };+ expect(aggregate.fold.first_created_at).toBe("2026-08-10T12:00:00Z");+ expect(aggregate.fold.last_created_at).toBe("2026-08-10T12:00:00.500Z");+ });++ test("metrics shards fold per (surface, month)", () => {+ writeShard(metricsDir(vault), "storage_usage.jsonl", [+ metricLine("storage_usage", "2026-08-01T00:00:00Z", { bytes: 10 }),+ metricLine("storage_usage", "2026-08-05T00:00:00Z", { bytes: 11 }),+ metricLine("storage_usage", "2026-08-10T00:00:00Z", { bytes: 12 }),+ metricLine("storage_usage", "2026-08-15T00:00:00Z", { bytes: 13 }),+ metricLine("storage_usage", "2026-08-20T00:00:00Z", { bytes: 14 }),+ metricLine("storage_usage", "2026-07-05T00:00:00Z", { bytes: 5 }),+ metricLine("storage_usage", "2026-07-06T00:00:00Z", { bytes: 6 }),+ ]);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.metrics, name: "storage_usage.jsonl" }],+ now: NOW,+ dryRun: false,+ expect: 7,+ });+ expect(result.totals.folded).toBe(7);+ const lines = readShard(metricsDir(vault), "storage_usage.jsonl")+ .split("\n")+ .filter((line) => line.trim() !== "");+ expect(lines).toHaveLength(2);+ const folds = lines.map((line) => (JSON.parse(line) as { fold: Record<string, unknown> }).fold);+ expect(folds.map((f) => f.month).toSorted()).toEqual(["2026-07", "2026-08"]);+ const august = folds.find((f) => f.month === "2026-08")!;+ expect(august).toMatchObject({+ family: RECEIPT_LEDGER_FAMILY.metrics,+ surface: "storage_usage",+ count: 5,+ first_run_at: "2026-08-01T00:00:00Z",+ last_run_at: "2026-08-20T00:00:00Z",+ });+ });++ test("a metrics line outside the record shape is kept, never folded into a surface-less aggregate", () => {+ // The scan counts a parsed line without the full metrics record shape+ // (schema, surface and run_at strings, plain-object payload) as+ // unparseable. The fold engine must classify it the same way: gated+ // only on run_at, a line carrying nothing but an old stamp would fold+ // with an undefined surface, and JSON.stringify would drop that+ // surface from the aggregate - an aggregate no reader can attribute.+ const malformed = JSON.stringify({ run_at: "2026-07-01T00:00:00Z" });+ writeShard(metricsDir(vault), "storage_usage.jsonl", [+ metricLine("storage_usage", "2026-08-01T00:00:00Z", { bytes: 10 }),+ metricLine("storage_usage", "2026-08-05T00:00:00Z", { bytes: 11 }),+ metricLine("storage_usage", "2026-08-10T00:00:00Z", { bytes: 12 }),+ `${malformed}\n`,+ ]);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.metrics, name: "storage_usage.jsonl" }],+ now: NOW,+ dryRun: false,+ expect: 3,+ });+ expect(result.totals.folded).toBe(3);+ expect(result.totals.kept).toBe(1);+ expect(result.totals.aggregates).toBe(1);+ const after = readShard(metricsDir(vault), "storage_usage.jsonl");+ // The malformed line is still there, byte-verbatim.+ expect(after).toContain(malformed);+ for (const line of after.split("\n").filter((entry) => entry.trim() !== "")) {+ const fold = (JSON.parse(line) as { fold?: { surface?: unknown } }).fold;+ if (fold !== undefined) expect(typeof fold.surface).toBe("string");+ }+ });++ test("a confirmed run takes one recovery point and appends the audit record exactly once", () => {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.snapshotRunId).toBeDefined();+ expect(result.auditRecordId).toBeDefined();+ expect(result.recoverability).toBeDefined();+ const snapshots = listSnapshots(vault).snapshots;+ expect(snapshots.some((s) => s.run_id === result.snapshotRunId)).toBe(true);+ const audits = listContinuityRecords(vault, { kind: "receipt_compaction" });+ expect(audits).toHaveLength(1);+ expect(audits[0]!.id).toBe(result.auditRecordId!);+ });++ test("a dry run writes nothing: no rewrite, no snapshot, no audit record", () => {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ const before = digestVaultTree(vault);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: true,+ });+ expect(result.dryRun).toBe(true);+ expect(result.totals.folded).toBe(2);+ expect(digestVaultTree(vault)).toBe(before);+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);+ });++ test("a count-guard mismatch aborts before any write, snapshot included", () => {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ const before = digestVaultTree(vault);+ expect(() =>+ compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ expect: 3,+ }),+ ).toThrow(CountGuardError);+ expect(digestVaultTree(vault)).toBe(before);+ expect(readdirSync(join(vault, "Brain", ".snapshots"))).toEqual([]);+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);+ });++ test("a re-run over a compacted ledger folds nothing and writes nothing", () => {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ const after = digestVaultTree(vault);+ const audits = listContinuityRecords(vault, { kind: "receipt_compaction" }).length;+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.totals.folded).toBe(0);+ expect(digestVaultTree(vault)).toBe(after);+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(audits);+ });++ test("a shard whose fold would not shrink is collected and the rest of the run completes", () => {+ // One old record whose aggregate line would be LARGER than the+ // record itself: the engine refuses the rewrite rather than grow a+ // ledger, and the refusal is collected while the healthy shard+ // beside it still folds and the run still completes.+ writeShard(continuityLogDir(vault), "2026-07.jsonl", [+ continuityLine("recall_telemetry", "2026-07-10T12:00:00Z"),+ ]);+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ const result = compactReceiptLedgers(vault, {+ shards: [+ { family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-07.jsonl" },+ { family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" },+ ],+ now: NOW,+ dryRun: false,+ });+ expect(result.shards.map((s) => s.shard)).toEqual(["continuity:2026-08.jsonl"]);+ expect(result.failures).toHaveLength(1);+ expect(result.failures[0]!.shard).toBe("continuity:2026-07.jsonl");+ expect(result.failures[0]!.message).toContain("shrink");+ // The run still completed: the healthy shard folded, and the audit+ // names the refusal honestly.+ expect(result.totals.folded).toBe(2);+ const audits = listContinuityRecords(vault, { kind: "receipt_compaction" });+ expect(audits).toHaveLength(1);+ expect(JSON.stringify(audits[0]!.payload)).toContain("continuity:2026-07.jsonl");+ // The refused shard is untouched on disk.+ expect(readShard(continuityLogDir(vault), "2026-07.jsonl")).toContain("2026-07-10T12:00:00Z");+ });++ test("a lost audit record is a named outcome, never a throw that discards the run", () => {+ // The audit record lands in the continuity ledger under the run+ // clock's month, beside the shards being folded. A directory parked+ // at that exact path makes the append fail while the metrics shard+ // selected here folds untouched - the window where every shard is+ // already durable and a post-write throw would discard every+ // outcome, failure and recovery point with it.+ writeShard(+ metricsDir(vault),+ "storage_usage.jsonl",+ [+ metricLine("brain", "2026-06-01T00:00:00Z", { n: 1 }),+ metricLine("brain", "2026-06-02T00:00:00Z", { n: 2 }),+ metricLine("brain", "2026-06-03T00:00:00Z", { n: 3 }),+ metricLine("brain", "2026-06-04T00:00:00Z", { n: 4 }),+ ].join(""),+ );+ mkdirSync(continuityLogPath(vault, "2026-09"), { recursive: true });+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.metrics, name: "storage_usage.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.totals.folded).toBe(4);+ expect(result.auditRecordId).toBeUndefined();+ expect(result.auditError).toBe(RECEIPT_AUDIT_APPEND_FAILED);+ // The fold itself landed: the shard on disk holds the aggregate.+ expect(readShard(metricsDir(vault), "storage_usage.jsonl")).toContain(RECEIPT_FOLD_SCHEMA);+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);+ });++ test("a mixed selection reports a compacted shard as nothing-to-do, not a failure", () => {+ // The re-run contract: a compacted shard folds nothing and writes+ // nothing. A MIXED selection never reaches the run-level+ // nothing-foldable shortcut - the compacted shard arrives at the+ // rewriter with foldable 0, its rebuild is byte-identical, and the+ // strict-shrink guard then failed exactly the shard the contract+ // calls a no-op.+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ const compacted = readShard(continuityLogDir(vault), "2026-08.jsonl");+ writeShard(continuityLogDir(vault), "2026-07.jsonl", [+ continuityLine("recall_telemetry", "2026-07-10T12:00:00Z"),+ continuityLine("recall_telemetry", "2026-07-20T12:00:00Z"),+ ]);+ const result = compactReceiptLedgers(vault, {+ shards: [+ { family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-07.jsonl" },+ { family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" },+ ],+ now: NOW,+ dryRun: false,+ });+ expect(result.failures).toEqual([]);+ expect(result.shards.map((shard) => shard.shard).toSorted()).toEqual([+ "continuity:2026-07.jsonl",+ "continuity:2026-08.jsonl",+ ]);+ const untouched = result.shards.find((shard) => shard.shard === "continuity:2026-08.jsonl");+ expect(untouched).toBeDefined();+ expect(untouched!.folded).toBe(0);+ expect(untouched!.bytesAfter).toBe(untouched!.bytesBefore);+ // The compacted shard is byte-identical on disk.+ expect(readShard(continuityLogDir(vault), "2026-08.jsonl")).toBe(compacted);+ // The run still folded the shard that had foldable records.+ expect(result.totals.folded).toBe(2);+ expect(result.auditRecordId).toBeDefined();+ });++ test("a zero-foldable shard is never rewritten, so its bytes never grow", () => {+ // The rebuild joins the kept lines and appends one newline, so a+ // shard whose file lacks a trailing newline rebuilds one byte LONGER+ // than it stands. With nothing foldable there is nothing to write:+ // the file must come out byte-identical, not grown, not failed.+ writeShard(+ continuityLogDir(vault),+ "2026-08.jsonl",+ continuityLine("gate_telemetry", "2026-09-15T12:00:00Z").replace(/\n$/, ""),+ );+ writeShard(continuityLogDir(vault), "2026-07.jsonl", [+ continuityLine("recall_telemetry", "2026-07-10T12:00:00Z"),+ continuityLine("recall_telemetry", "2026-07-20T12:00:00Z"),+ ]);+ const before = readShard(continuityLogDir(vault), "2026-08.jsonl");+ const result = compactReceiptLedgers(vault, {+ shards: [+ { family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-07.jsonl" },+ { family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" },+ ],+ now: NOW,+ dryRun: false,+ });+ expect(result.failures).toEqual([]);+ expect(readShard(continuityLogDir(vault), "2026-08.jsonl")).toBe(before);+ expect(result.totals.folded).toBe(2);+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "an unreadable shard fails by name, never with the raw filesystem error",+ () => {+ // The failure rows ride the compact view to remote callers verbatim:+ // the raw Node error text would hand them an absolute host path and+ // an absent-versus-denied oracle. The boundary names the shard and+ // the one class the operator can act on: unreadable.+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ const shardPath = join(continuityLogDir(vault), "2026-08.jsonl");+ chmodSync(shardPath, 0o000);+ try {+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: true,+ });+ expect(result.failures).toHaveLength(1);+ expect(result.failures[0]!.shard).toBe("continuity:2026-08.jsonl");+ expect(result.failures[0]!.message).toBe(RECEIPT_SHARD_UNREADABLE);+ const rendered = JSON.stringify(result.failures);+ expect(rendered).not.toContain("ENOENT");+ expect(rendered).not.toContain("EACCES");+ expect(rendered).not.toContain(vault);+ } finally {+ chmodSync(shardPath, 0o644);+ }+ },+ );++ test("a shard ref that is not a valid shard name is refused before anything runs", () => {+ expect(() =>+ compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "../metrics/x.jsonl" }],+ now: NOW,+ dryRun: true,+ }),+ ).toThrow(/not a continuity shard/);+ });++ test("a permissions-document deny refuses the confirmed run before any write", () => {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ atomicWriteFileSync(+ join(vault, "Brain", "_permissions.yaml"),+ "version: 1\ndefault_action: allow\nentries:\n - id: freeze\n agent: test-agent\n" ++ " action: ingest\n verdict: deny\n",+ );+ const before = readShard(continuityLogDir(vault), "2026-08.jsonl");+ expect(() =>+ compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ subject: { agent: "test-agent", via: "config" },+ }),+ ).toThrow(WriteRefusedError);+ // The ledger shard is untouched. (The refusal itself writes its+ // decision-ledger row - that record is the point of the consult -+ // so the assertion is on the shard the run would have rewritten.)+ expect(readShard(continuityLogDir(vault), "2026-08.jsonl")).toBe(before);+ expect(readdirSync(join(vault, "Brain", ".snapshots"))).toEqual([]);+ });++ test("an empty shard selection is a named caller error, never a silent full scan", () => {+ expect(() => compactReceiptLedgers(vault, { shards: [], now: NOW, dryRun: true })).toThrow(+ /explicit shard ids/,+ );+ });+});diff --git a/tests/core/brain/receipts-scan.test.ts b/tests/core/brain/receipts-scan.test.tsnew file mode 100644index 00000000..246f23df--- /dev/null+++ b/tests/core/brain/receipts-scan.test.ts@@ -0,0 +1,351 @@+/**+ * Receipt ledger scan (storage-lifecycle, Lane 3 task 3.1).+ *+ * `scanReceiptLedgers` inventories the two append-only receipt ledgers -+ * the continuity log and the metrics sink - as the fold engine's preview:+ * per shard, how many records it holds, how many are old enough to fold,+ * and how many are in a protected class the fold must never touch. The+ * protected vocabulary is the audit posture named in the design:+ * `source_invalidation` (the deleteBySource audit trail), the two Hermes+ * context-compaction kinds (`pre_compact_extract`, `post_compact_audit`),+ * any record whose payload carries a clip-protected correlation key, and+ * any kind this build does not know (fail closed to keep). None of them+ * may ever be counted foldable.+ *+ * The scan is a pure read: a byte-identical vault after the run is part+ * of the acceptance, asserted with the whole-tree digest.+ */++import { describe, expect, test } from "bun:test";+import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { continuityLogDir } from "../../../src/core/brain/continuity/store.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";+import { metricsDir } from "../../../src/core/brain/metrics.ts";+import {+ RECEIPT_SHARD_UNREADABLE,+ scanReceiptLedgers,+} from "../../../src/core/brain/receipts/scan.ts";+import {+ RECEIPT_FOLD_DEFAULT_AGE_DAYS,+ RECEIPT_LEDGER_FAMILY,+ type ReceiptScanReport,+} from "../../../src/core/brain/receipts/types.ts";+import { digestVaultTree } from "../../helpers/vault-digest.ts";++const CONTINUITY_SCHEMA = "o2b.continuity.v1";+const METRICS_SCHEMA = "o2b.metrics.v1";++const NOW = new Date("2026-09-25T00:00:00Z");++function mintVault(): string {+ const root = mkdtempSync(join(tmpdir(), "o2b-receipts-scan-"));+ mkdirSync(continuityLogDir(root), { recursive: true });+ mkdirSync(metricsDir(root), { recursive: true });+ return root;+}++let vault: string;++function cleanup(): void {+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+}++/** One fixture continuity record line with a content-shaped fixture id. */+function continuityLine(+ kind: string,+ createdAt: string,+ payload: Record<string, unknown> = {},+): string {+ const id = `ctn_${createdAt.replace(/[^0-9]/g, "").slice(0, 14)}_${kind}`;+ return `${JSON.stringify({+ schema: CONTINUITY_SCHEMA,+ id,+ kind,+ createdAt,+ sourceRefs: [],+ payload,+ private: false,+ redacted: false,+ })}\n`;+}++function metricLine(surface: string, runAt: string, payload: Record<string, unknown> = {}): string {+ return `${JSON.stringify({+ schema: METRICS_SCHEMA,+ surface,+ run_at: runAt,+ payload,+ })}\n`;+}++function writeShard(dir: string, name: string, lines: ReadonlyArray<string>): void {+ writeFileSync(join(dir, name), lines.join(""), "utf8");+}++function shardOf(report: ReceiptScanReport, family: string, name: string) {+ const shard = report.shards.find((s) => s.family === family && s.name === name);+ expect(shard, `${family} shard ${name} is inventoried`).toBeDefined();+ return shard!;+}++describe("scanReceiptLedgers", () => {+ test("counts foldable records above the age threshold per shard", () => {+ vault = mintVault();+ try {+ const continuity = continuityLogDir(vault);+ writeShard(continuity, "2026-08.jsonl", [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),+ ]);+ writeShard(continuity, "2026-09.telescope.jsonl", [+ continuityLine("gate_telemetry", "2026-09-15T12:00:00Z"),+ ]);+ const report = scanReceiptLedgers(vault, { now: NOW });+ const august = shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-08.jsonl");+ expect(august.records).toBe(1);+ // 2026-08-10 is 46 days before the pinned now: past the 30-day fold age.+ expect(august.foldable).toBe(1);+ expect(august.protectedRecords).toBe(0);+ const september = shardOf(+ report,+ RECEIPT_LEDGER_FAMILY.continuity,+ "2026-09.telescope.jsonl",+ );+ // A record inside the window is neither foldable nor protected.+ expect(september.records).toBe(1);+ expect(september.foldable).toBe(0);+ expect(september.protectedRecords).toBe(0);+ expect(report.totals.foldable).toBe(1);+ } finally {+ cleanup();+ }+ });++ test("the protected vocabulary is never counted foldable", () => {+ vault = mintVault();+ try {+ const continuity = continuityLogDir(vault);+ // The deleteBySource audit trail.+ writeShard(continuity, "2026-07.jsonl", [+ continuityLine("source_invalidation", "2026-07-10T12:00:00Z", { reason: "test" }),+ ]);+ // The Hermes context-compaction kinds: a storage fold must stay+ // disjoint from that vocabulary on every surface, ledgers included.+ writeShard(continuity, "2026-06.jsonl", [+ continuityLine("pre_compact_extract", "2026-06-10T12:00:00Z"),+ continuityLine("post_compact_audit", "2026-06-11T12:00:00Z"),+ ]);+ // A clip-protected correlation key carrier: folding would destroy+ // the correlation the clip rule guarantees survives.+ writeShard(continuity, "2026-05.jsonl", [+ continuityLine("gate_telemetry", "2026-05-10T12:00:00Z", { session_id: "sess-1" }),+ ]);+ // An unknown kind fails closed to keep: a future build's records+ // are not this build's to fold.+ writeShard(continuity, "2026-04.jsonl", [+ continuityLine("kind_from_the_future", "2026-04-10T12:00:00Z"),+ ]);+ const report = scanReceiptLedgers(vault, { now: NOW });+ expect(shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-07.jsonl").foldable).toBe(0);+ expect(+ shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-07.jsonl").protectedRecords,+ ).toBe(1);+ expect(shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-06.jsonl").foldable).toBe(0);+ expect(+ shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-06.jsonl").protectedRecords,+ ).toBe(2);+ expect(shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-05.jsonl").foldable).toBe(0);+ expect(+ shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-05.jsonl").protectedRecords,+ ).toBe(1);+ expect(shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-04.jsonl").foldable).toBe(0);+ expect(+ shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-04.jsonl").protectedRecords,+ ).toBe(1);+ expect(report.totals.foldable).toBe(0);+ } finally {+ cleanup();+ }+ });++ test("inventories metrics shards with run_at-based ages and the same protected rule", () => {+ vault = mintVault();+ try {+ const metrics = metricsDir(vault);+ writeShard(metrics, "storage_usage.jsonl", [+ metricLine("storage_usage", "2026-08-01T00:00:00Z"),+ metricLine("storage_usage", "2026-08-15T00:00:00Z"),+ metricLine("storage_usage", "2026-09-20T00:00:00Z"),+ ]);+ writeShard(metrics, "recall_probe.telescope.jsonl", [+ metricLine("recall_probe", "2026-08-02T00:00:00Z", { session_id: "sess-1" }),+ ]);+ const report = scanReceiptLedgers(vault, { now: NOW });+ const usage = shardOf(report, RECEIPT_LEDGER_FAMILY.metrics, "storage_usage.jsonl");+ expect(usage.base).toBe("storage_usage");+ expect(usage.records).toBe(3);+ expect(usage.foldable).toBe(2);+ expect(usage.protectedRecords).toBe(0);+ const probe = shardOf(report, RECEIPT_LEDGER_FAMILY.metrics, "recall_probe.telescope.jsonl");+ expect(probe.foldable).toBe(0);+ expect(probe.protectedRecords).toBe(1);+ } finally {+ cleanup();+ }+ });++ test("a parsed line outside the metrics record shape is unparseable, never foldable", () => {+ // The same shape gate the fold engine's classifier carries: a line+ // the scan counts unparseable must never be foldable in a confirmed+ // run, or the preview and the rewrite would disagree.+ vault = mintVault();+ try {+ writeShard(metricsDir(vault), "storage_usage.jsonl", [+ metricLine("storage_usage", "2026-08-01T00:00:00Z"),+ `${JSON.stringify({ run_at: "2026-08-02T00:00:00Z" })}\n`,+ `${JSON.stringify({ schema: METRICS_SCHEMA, surface: "storage_usage" })}\n`,+ `${JSON.stringify({ schema: METRICS_SCHEMA, surface: "storage_usage", payload: [] })}\n`,+ ]);+ const report = scanReceiptLedgers(vault, { now: NOW });+ const shard = shardOf(report, RECEIPT_LEDGER_FAMILY.metrics, "storage_usage.jsonl");+ expect(shard.records).toBe(1);+ expect(shard.unparseable).toBe(3);+ expect(shard.foldable).toBe(1);+ expect(shard.protectedRecords).toBe(0);+ expect(report.totals.foldable).toBe(1);+ } finally {+ cleanup();+ }+ });++ test("byte sizes and vault-relative paths are reported per shard", () => {+ vault = mintVault();+ try {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),+ ]);+ const report = scanReceiptLedgers(vault, { now: NOW });+ const shard = shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-08.jsonl");+ expect(shard.bytes).toBeGreaterThan(0);+ expect(shard.path).toBe("Brain/log/continuity/2026-08.jsonl");+ expect(report.totals.bytes).toBe(shard.bytes);+ } finally {+ cleanup();+ }+ });++ test("a record exactly at the cutoff is kept; one second earlier folds", () => {+ vault = mintVault();+ try {+ const continuity = continuityLogDir(vault);+ writeShard(continuity, "2026-08.jsonl", [+ continuityLine("recall_telemetry", "2026-08-26T00:00:00Z"),+ ]);+ writeShard(continuity, "2026-07.jsonl", [+ continuityLine("recall_telemetry", "2026-07-25T23:59:59Z"),+ ]);+ const report = scanReceiptLedgers(vault, { now: NOW, olderThanDays: 30 });+ expect(shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-08.jsonl").foldable).toBe(0);+ expect(shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-07.jsonl").foldable).toBe(1);+ } finally {+ cleanup();+ }+ });++ test("torn lines are counted, never folded", () => {+ vault = mintVault();+ try {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),+ "{not json\n",+ ]);+ const report = scanReceiptLedgers(vault, { now: NOW });+ const shard = shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-08.jsonl");+ expect(shard.records).toBe(1);+ expect(shard.unparseable).toBe(1);+ expect(report.totals.unparseable).toBe(1);+ } finally {+ cleanup();+ }+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "a shard that cannot be read is reported unreadable by name, never with filesystem detail",+ () => {+ // The error rows ride the scan view to remote callers verbatim: the+ // raw Node error text would hand them an absolute host path and an+ // absent-versus-denied oracle. The boundary reports the shard+ // unreadable, with its vault-relative name, and nothing else.+ vault = mintVault();+ try {+ const shardPath = join(continuityLogDir(vault), "2026-08.jsonl");+ writeShard(continuityLogDir(vault), "2026-08.jsonl", [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),+ ]);+ chmodSync(shardPath, 0o000);+ try {+ const report = scanReceiptLedgers(vault, { now: NOW });+ expect(report.shards).toEqual([]);+ expect(report.errors).toHaveLength(1);+ const error = report.errors[0]!;+ expect(error.shard).toBe("continuity:2026-08.jsonl");+ expect(error.path).toBe("Brain/log/continuity/2026-08.jsonl");+ expect(error.message).toBe(RECEIPT_SHARD_UNREADABLE);+ const rendered = JSON.stringify(report.errors);+ expect(rendered).not.toContain("ENOENT");+ expect(rendered).not.toContain("EACCES");+ expect(rendered).not.toContain(vault);+ } finally {+ chmodSync(shardPath, 0o644);+ }+ } finally {+ cleanup();+ }+ },+ );++ test("the scan is read-only: the vault digest is byte-identical after the run", () => {+ vault = mintVault();+ try {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),+ ]);+ writeShard(metricsDir(vault), "storage_usage.jsonl", [+ metricLine("storage_usage", "2026-08-01T00:00:00Z"),+ ]);+ const before = digestVaultTree(vault);+ scanReceiptLedgers(vault, { now: NOW });+ expect(digestVaultTree(vault)).toBe(before);+ } finally {+ cleanup();+ }+ });++ test("a vault with no ledgers scans to an empty report without throwing", () => {+ vault = mintVault();+ try {+ rmSync(continuityLogDir(vault), { recursive: true, force: true });+ rmSync(metricsDir(vault), { recursive: true, force: true });+ const report = scanReceiptLedgers(vault, { now: NOW });+ expect(report.shards).toEqual([]);+ expect(report.totals.records).toBe(0);+ expect(report.totals.foldable).toBe(0);+ expect(report.errors).toEqual([]);+ } finally {+ cleanup();+ }+ });++ test("the age threshold defaults to the named constant and stamps the report", () => {+ vault = mintVault();+ try {+ const report = scanReceiptLedgers(vault, { now: NOW });+ expect(report.olderThanDays).toBe(RECEIPT_FOLD_DEFAULT_AGE_DAYS);+ expect(report.generated_at).toBe("2026-09-25T00:00:00Z");+ } finally {+ cleanup();+ }+ });+});diff --git a/tests/core/brain/session-identity-check.test.ts b/tests/core/brain/session-identity-check.test.tsindex fedf09b4..233aa50e 100644--- a/tests/core/brain/session-identity-check.test.ts+++ b/tests/core/brain/session-identity-check.test.ts@@ -355,6 +355,7 @@ const KIND_WRITERS: ReadonlyArray<{sessionScoped: false,writers: ["core/brain/continuity/store.ts",+ "core/brain/document-retire.ts","core/brain/source-cleanup.ts","core/brain/portability/knowledge-pack.ts",],@@ -371,6 +372,11 @@ const KIND_WRITERS: ReadonlyArray<{sessionScoped: false,writers: ["core/brain/link-graph/session-mining.ts"],},+ {+ kind: "receipt_compaction",+ sessionScoped: false,+ writers: ["core/brain/receipts/compact.ts"],+ },]);/**diff --git a/tests/core/brain/source-cleanup.test.ts b/tests/core/brain/source-cleanup.test.tsindex b225db8c..809ca623 100644--- a/tests/core/brain/source-cleanup.test.ts+++ b/tests/core/brain/source-cleanup.test.ts@@ -19,9 +19,11 @@*/import { afterEach, beforeEach, describe, expect, test } from "bun:test";-import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { chmodSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";-import { join } from "node:path";+import { dirname, join } from "node:path";++import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";import { ingestSource } from "../../../src/core/brain/ingest/ingest.ts";import { readManifest } from "../../../src/core/brain/ingest/content-manifest.ts";@@ -309,6 +311,63 @@ describe("deleteBySource - D1 snapshot gate", () => {});});+describe("deleteBySource — per-entry failure collection", () => {+ test("dry-run and clean confirmed runs report an empty failed array", () => {+ seedContaminatedVault();+ const dry = deleteBySource(vault, SOURCE, { now: NOW });+ expect(dry.failed).toEqual([]);++ const plan = deleteBySource(vault, SOURCE, { confirm: true, now: NOW });+ expect(plan.failed).toEqual([]);+ expect(plan.deleted.length).toBeGreaterThan(0);+ });++ test("a removal failure is collected; the remaining entries are still removed", () => {+ if (CHMOD_CANNOT_DENY) return;+ seedContaminatedVault();++ // Learn the derived entries first, then make one of them undeletable:+ // removing a file needs the write bit on its DIRECTORY, so a read-only+ // directory makes exactly that one removal fail.+ const preview = deleteBySource(vault, SOURCE, { now: NOW });+ const blockedPath = preview.derived.find((e) => e.kind === "entity")!.path;+ const blockedDir = join(vault, dirname(blockedPath));+ chmodSync(blockedDir, 0o555);+ try {+ const plan = deleteBySource(vault, SOURCE, { confirm: true, now: NOW });++ // The blocked entry is reported, named, and left on disk.+ expect(plan.failed.length).toBe(1);+ expect(plan.failed[0]!.path).toBe(blockedPath);+ expect(plan.failed[0]!.message).toContain("EACCES");+ expect(existsSync(join(vault, blockedPath))).toBe(true);+ expect(plan.deleted).not.toContain(blockedPath);++ // The remaining entries were still removed - one blocked file never+ // aborts the run.+ expect(plan.deleted.length).toBe(preview.derived.length - 1);+ for (const entry of preview.derived) {+ if (entry.path === blockedPath) continue;+ expect(existsSync(join(vault, entry.path))).toBe(false);+ expect(plan.deleted).toContain(entry.path);+ }+ expect(plan.manifestEntryRemoved).toBe(true);++ // The audit record is honest about both counts.+ expect(plan.auditRecordId).not.toBeNull();+ const audit = listContinuityRecords(vault, { kind: "source_invalidation" });+ expect(audit.length).toBe(1);+ const reason = String(audit[0]!.payload["reason"]);+ expect(reason).toContain(`removed ${plan.deleted.length}`);+ expect(reason).toContain("failed to remove 1");+ expect(reason).toContain(blockedPath);+ } finally {+ // Restore the write bit so afterEach can remove the sandbox.+ chmodSync(blockedDir, 0o755);+ }+ });+});+describe("deleteBySource — shared preference via managed _evidenced_by (CR #127.7)", () => {test("a wikilink-matched preference evidenced by a FOREIGN signal is reported, never deleted", () => {seedContaminatedVault();diff --git a/tests/core/brain/storage-forecast.test.ts b/tests/core/brain/storage-forecast.test.tsnew file mode 100644index 00000000..611894ca--- /dev/null+++ b/tests/core/brain/storage-forecast.test.ts@@ -0,0 +1,290 @@+/**+ * The storage forecast and the usage sampler (`forecastStorageUsage`,+ * `sampleStorageUsage`).+ *+ * Claims pinned here:+ *+ * 1. Over seeded `storage_usage` metric samples the forecast returns+ * the linear slope (bytes per day), the days left to the cap and a+ * tier, and the tier boundaries land EXACTLY on the default+ * constants: 80 percent / 30 days warns, 95 percent / 7 days is+ * critical.+ * 2. An absent `cap_bytes` is `unbounded`; an incomplete measurement is+ * `unmeasurable` - a census that could not read a surface refuses+ * the forecast by name instead of projecting from partial bytes.+ * 3. Fewer than two stored samples is the named `no_forecast` state:+ * the stored aggregate is advisory, never tier-eligible on its own.+ * A FRESH reading (`latestUsage`) is enforcement truth, so the+ * percent tiers compute from it even with an empty history.+ * 4. `sampleStorageUsage` appends exactly one `storage_usage` metrics+ * record per call, is idempotent per run id, and an append failure+ * is reported by name, never thrown.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { appendMetric, listMetrics, metricsDir } from "../../../src/core/brain/metrics.ts";+import {+ STORAGE_CRITICAL_DAYS_DEFAULT,+ STORAGE_CRITICAL_PERCENT_DEFAULT,+ STORAGE_USAGE_METRIC_SURFACE,+ STORAGE_WARN_DAYS_DEFAULT,+ STORAGE_WARN_PERCENT_DEFAULT,+ forecastStorageUsage,+ sampleStorageUsage,+} from "../../../src/core/brain/storage/forecast.ts";+import { measureStorageUsage } from "../../../src/core/brain/storage/usage.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-storage-forecast-"));+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++const CAP = 1000;+const NOW = new Date("2026-10-15T12:00:00Z");++/** Seed one `storage_usage` sample `daysAgo` days before the forecast instant. */+function seedSample(daysAgo: number, totalBytes: number): void {+ const t = new Date(Date.parse("2026-10-15T12:00:00Z") - daysAgo * 86_400_000);+ appendMetric(vault, {+ surface: STORAGE_USAGE_METRIC_SURFACE,+ runAt: t.toISOString().replace(".000Z", "Z"),+ payload: { run_id: `seed-${daysAgo}`, total_bytes: totalBytes, measured: true },+ });+}++describe("forecastStorageUsage", () => {+ test("returns the linear slope, days to cap and tier over seeded samples", () => {+ seedSample(2, 100);+ seedSample(1, 200);+ seedSample(0, 300);+ // Cap 5000 keeps the reading clear of every boundary: 6 percent+ // used, 47 days to the cap.+ const forecast = forecastStorageUsage(vault, { capBytes: 5000 });+ expect(forecast.tier).toBe("ok");+ expect(forecast.slopeBytesPerDay).toBe(100);+ expect(forecast.measuredBytes).toBe(300);+ expect(forecast.percentUsed).toBe(6);+ expect(forecast.daysToCap).toBe(47);+ expect(forecast.capBytes).toBe(5000);+ });++ test("exactly the warn percent warns and exactly the critical percent is critical", () => {+ expect(STORAGE_WARN_PERCENT_DEFAULT).toBe(80);+ expect(STORAGE_CRITICAL_PERCENT_DEFAULT).toBe(95);+ expect(STORAGE_WARN_DAYS_DEFAULT).toBe(30);+ expect(STORAGE_CRITICAL_DAYS_DEFAULT).toBe(7);++ // A gentle slope keeps the days boundary out of the decision: 20+ // days to the cap at 10 bytes per day.+ seedSample(1, 790);+ seedSample(0, 800);+ expect(forecastStorageUsage(vault, { capBytes: CAP }).tier).toBe("warn");++ seedSample(-1, 950);+ // 95 percent used, and 5 days to the cap - critical on both counts.+ expect(forecastStorageUsage(vault, { capBytes: CAP }).tier).toBe("critical");+ });++ test("the percent boundaries are inclusive with no trend to decide instead", () => {+ // The tier fixtures above sit inside the days bands too, so a+ // comparison flipped exclusive would still answer warn/critical+ // through the days path and the percent boundary would go unpinned.+ // A FLAT history has slope 0 and no daysToCap, so the percent+ // comparison alone decides: exactly 80 warns, exactly 95 is critical.+ seedSample(1, 800);+ seedSample(0, 800);+ const warn = forecastStorageUsage(vault, { capBytes: CAP });+ expect(warn.slopeBytesPerDay).toBe(0);+ expect(warn.daysToCap).toBe(null);+ expect(warn.percentUsed).toBe(80);+ expect(warn.tier).toBe("warn");++ rmSync(metricsDir(vault), { recursive: true, force: true });+ seedSample(1, 950);+ seedSample(0, 950);+ const critical = forecastStorageUsage(vault, { capBytes: CAP });+ expect(critical.daysToCap).toBe(null);+ expect(critical.percentUsed).toBe(95);+ expect(critical.tier).toBe("critical");+ });++ test("a history that cannot be listed answers samplesUnreadable, never no-samples", () => {+ // A metrics directory replaced by a plain file makes the ledger+ // listing throw; the forecast must NAME the unreadable history on+ // the result instead of flattening it into "no samples ever+ // recorded" - the two say opposite things about the vault's past.+ seedSample(1, 400);+ seedSample(0, 500);+ rmSync(metricsDir(vault), { recursive: true, force: true });+ writeFileSync(metricsDir(vault), "not a directory\n", "utf8");+ const forecast = forecastStorageUsage(vault, { capBytes: CAP });+ expect(forecast.samplesUnreadable).toBe(true);+ expect(forecast.tier).toBe("no_forecast");+ });++ test("a percent-only warn never outranks a critical days horizon", () => {+ // 280 of 350 bytes is exactly 80 percent (warn by percent), and a+ // slope of 10 bytes per day puts the cap exactly 7 days out+ // (critical by days). The critical comparison runs first, so the+ // days horizon decides the tier.+ seedSample(1, 270);+ seedSample(0, 280);+ const forecast = forecastStorageUsage(vault, { capBytes: 350 });+ expect(forecast.percentUsed).toBe(80);+ expect(forecast.daysToCap).toBe(7);+ expect(forecast.tier).toBe("critical");+ });++ test("exactly the warn days to cap warns", () => {+ // current 300, slope 10 per day: 30 days to 600, at the boundary.+ seedSample(1, 290);+ seedSample(0, 300);+ const warn = forecastStorageUsage(vault, { capBytes: 600 });+ expect(warn.slopeBytesPerDay).toBe(10);+ expect(warn.daysToCap).toBe(30);+ expect(warn.tier).toBe("warn");+ });++ test("exactly the critical days to cap is critical", () => {+ // current 300, slope 50 per day: 7 days to 650, at the boundary.+ seedSample(1, 250);+ seedSample(0, 300);+ const critical = forecastStorageUsage(vault, { capBytes: 650 });+ expect(critical.slopeBytesPerDay).toBe(50);+ expect(critical.daysToCap).toBe(7);+ expect(critical.tier).toBe("critical");+ });++ test("an absent cap is unbounded", () => {+ seedSample(1, 900);+ seedSample(0, 990);+ const forecast = forecastStorageUsage(vault, { capBytes: null });+ expect(forecast.tier).toBe("unbounded");+ expect(forecast.percentUsed).toBe(null);+ expect(forecast.daysToCap).toBe(null);+ });++ test("fewer than two stored samples is the named no-forecast state", () => {+ expect(forecastStorageUsage(vault, { capBytes: CAP }).tier).toBe("no_forecast");+ seedSample(0, 500);+ expect(forecastStorageUsage(vault, { capBytes: CAP }).tier).toBe("no_forecast");+ });++ test("a fresh reading is tier-eligible even with an empty history", () => {+ const usage = measureStorageUsage(vault);+ const forecast = forecastStorageUsage(vault, { capBytes: 100, latestUsage: usage });+ expect(forecast.measuredBytes).toBe(usage.totalBytes);+ expect(forecast.tier).toBe("ok");+ // One fresh reading is no trend.+ expect(forecast.slopeBytesPerDay).toBe(null);+ expect(forecast.daysToCap).toBe(null);+ });++ test("the freshest reading wins over the stored aggregate", () => {+ // Flat history at 90 percent: the stored aggregate alone warns.+ seedSample(1, 900);+ seedSample(0, 900);+ expect(forecastStorageUsage(vault, { capBytes: CAP }).tier).toBe("warn");+ // The fresh census of the same vault counts only the metric records+ // themselves - a fraction of the stored total - and answers ok.+ const usage = measureStorageUsage(vault);+ expect(usage.totalBytes).toBeLessThan(900);+ const forecast = forecastStorageUsage(vault, { capBytes: CAP, latestUsage: usage });+ expect(forecast.measuredBytes).toBe(usage.totalBytes);+ expect(forecast.tier).toBe("ok");+ });++ test("a shrinking vault never forecasts past the cap", () => {+ seedSample(1, 500);+ seedSample(0, 400);+ const forecast = forecastStorageUsage(vault, { capBytes: CAP });+ expect(forecast.slopeBytesPerDay).toBe(-100);+ expect(forecast.daysToCap).toBe(null);+ expect(forecast.tier).toBe("ok");+ });++ test.skipIf(CHMOD_CANNOT_DENY)("an incomplete measurement is unmeasurable", () => {+ seedSample(1, 100);+ seedSample(0, 200);+ mkdirSync(join(vault, "Brain", ".artifacts"), { recursive: true });+ chmodSync(join(vault, "Brain", ".artifacts"), 0o000);+ try {+ const usage = measureStorageUsage(vault);+ expect(usage.complete).toBe(false);+ const forecast = forecastStorageUsage(vault, { capBytes: CAP, latestUsage: usage });+ expect(forecast.tier).toBe("unmeasurable");+ expect(forecast.unmeasured).toHaveLength(1);+ } finally {+ chmodSync(join(vault, "Brain", ".artifacts"), 0o755);+ }+ });++ test("the forecast is deterministic for identical inputs", () => {+ seedSample(1, 100);+ seedSample(0, 200);+ const a = forecastStorageUsage(vault, { capBytes: CAP });+ const b = forecastStorageUsage(vault, { capBytes: CAP });+ expect(b).toEqual(a);+ });+});++describe("sampleStorageUsage", () => {+ test("appends exactly one storage_usage record per call", () => {+ const usage = measureStorageUsage(vault);+ const result = sampleStorageUsage(vault, { runId: "run-1", now: NOW });+ expect(result.appended).toBe(true);+ expect(result.runId).toBe("run-1");+ expect(result.usage.totalBytes).toBe(usage.totalBytes);+ const records = listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE });+ expect(records).toHaveLength(1);+ expect(records[0]!.surface).toBe(STORAGE_USAGE_METRIC_SURFACE);+ expect(records[0]!.payload["total_bytes"]).toBe(usage.totalBytes);+ expect(records[0]!.payload["measured"]).toBe(true);+ });++ test("is idempotent per run id and appends again for a new one", () => {+ sampleStorageUsage(vault, { runId: "run-1", now: NOW });+ const repeat = sampleStorageUsage(vault, { runId: "run-1", now: NOW });+ expect(repeat.appended).toBe(false);+ expect(listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE })).toHaveLength(1);+ const next = sampleStorageUsage(vault, { runId: "run-2", now: NOW });+ expect(next.appended).toBe(true);+ expect(listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE })).toHaveLength(2);+ });++ test("a pre-measured census is recorded instead of walking the vault again", () => {+ // The census a gate already took, before the vault grew: the sampler+ // must record THESE bytes, not take a second census that would+ // measure the growth.+ const usage = measureStorageUsage(vault, { now: NOW });+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", "AA-grown.md"), "x".repeat(50_000), "utf8");+ const result = sampleStorageUsage(vault, { runId: "run-1", now: NOW, usage });+ expect(result.usage).toBe(usage);+ expect(result.appended).toBe(true);+ const records = listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE });+ expect(records).toHaveLength(1);+ expect(records[0]!.payload["total_bytes"]).toBe(usage.totalBytes);+ });++ test("an append failure is reported by name, never thrown", () => {+ // Brain/metrics exists as a regular file, so creating the metrics+ // directory fails and the append cannot land.+ mkdirSync(join(vault, "Brain"), { recursive: true });+ writeFileSync(join(vault, "Brain", "metrics"), "not a directory", "utf8");+ const result = sampleStorageUsage(vault, { runId: "run-1", now: NOW });+ expect(result.appended).toBe(false);+ expect(result.appendError).toContain("EEXIST");+ expect(result.appendError).toContain("metrics");+ });+});diff --git a/tests/core/brain/storage-gate.test.ts b/tests/core/brain/storage-gate.test.tsnew file mode 100644index 00000000..6dd03159--- /dev/null+++ b/tests/core/brain/storage-gate.test.ts@@ -0,0 +1,159 @@+/**+ * The capacity gate decision (`evaluateCapacityGate`).+ *+ * Claims pinned here:+ *+ * 1. The decision is computed from a FRESH `measureStorageUsage` call+ * at decision time, never from the stored aggregate: mutating the+ * vault between `sampleStorageUsage` and the decision changes the+ * answer, in both directions.+ * 2. An unbounded store never skips and never measures; an+ * unmeasurable store never skips and names the refused surfaces.+ * 3. Only the critical tier skips; warn and ok run, with the tier and+ * reading attached either way.+ * 4. The gate reads its bounds through the resolved `storage:` config,+ * so an operator cap is the only thing that can close it.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { appendFileSync, chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../../src/core/brain/init.ts";+import { brainConfigPath } from "../../../src/core/brain/paths.ts";+import {+ STORAGE_FORECAST_TIER,+ sampleStorageUsage,+} from "../../../src/core/brain/storage/forecast.ts";+import { evaluateCapacityGate } from "../../../src/core/brain/storage/gate.ts";+import { measureStorageUsage } from "../../../src/core/brain/storage/usage.ts";+import { resolveStorage } from "../../../src/core/brain/policy.ts";+import { loadBrainConfig } from "../../../src/core/brain/policy.ts";+import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-storage-gate-"));+ bootstrapBrain(vault, {});+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++/** Append a storage block to the bootstrapped config. */+function configureCap(capBytes: number): void {+ appendFileSync(brainConfigPath(vault), `\nstorage:\n cap_bytes: ${capBytes}\n`);+}++function resolvedCap(): number | null {+ return resolveStorage(loadBrainConfig(vault)).cap_bytes;+}++function writePages(name: string, totalBytes: number): void {+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", name), "x".repeat(totalBytes), "utf8");+}++describe("evaluateCapacityGate", () => {+ test("an unbounded store runs without measuring", () => {+ const decision = evaluateCapacityGate(vault, { capBytes: resolvedCap() });+ expect(decision.decision).toBe("run");+ expect(decision.tier).toBe(STORAGE_FORECAST_TIER.unbounded);+ // No cap, no reading: the measurement was never taken.+ expect(decision.reading).toBeUndefined();+ });++ test("a bounded store far below the cap runs with its reading attached", () => {+ configureCap(10_000_000);+ writePages("AA-small.md", 100);+ const decision = evaluateCapacityGate(vault, { capBytes: resolvedCap() });+ expect(decision.decision).toBe("run");+ expect(decision.tier).toBe(STORAGE_FORECAST_TIER.ok);+ expect(decision.reading).toBeDefined();+ expect(decision.reading!.measuredBytes).toBe(measureStorageUsage(vault).totalBytes);+ expect(decision.reading!.percentUsed).toBeLessThan(1);+ });++ test("a store past the critical percent skips", () => {+ configureCap(1000);+ writePages("AA-big.md", 5000);+ const decision = evaluateCapacityGate(vault, { capBytes: resolvedCap() });+ expect(decision.decision).toBe("skip");+ expect(decision.tier).toBe(STORAGE_FORECAST_TIER.critical);+ expect(decision.reading!.percentUsed).toBeGreaterThan(95);+ });++ test("only the critical tier skips; the warn band runs", () => {+ // The bootstrapped skeleton lands inside the warn band of a 30 KB+ // cap: past 80 percent, short of 95.+ configureCap(30_000);+ const decision = evaluateCapacityGate(vault, { capBytes: resolvedCap() });+ expect(decision.tier).toBe(STORAGE_FORECAST_TIER.warn);+ expect(decision.decision).toBe("run");+ });++ test("the decision re-measures: the stored aggregate never decides", () => {+ // The bootstrapped skeleton sits near 30 KB, so 100 KB keeps the+ // base vault below every band while a grown page blows past it.+ const CAP = 100_000;+ configureCap(CAP);+ // Sample the small vault, then push it past the cap: the decision+ // must reflect the fresh bytes, not the sampled aggregate.+ sampleStorageUsage(vault, { runId: "before" });+ writePages("AA-grown.md", 150_000);+ const after = evaluateCapacityGate(vault, { capBytes: resolvedCap() });+ expect(after.decision).toBe("skip");+ expect(after.tier).toBe(STORAGE_FORECAST_TIER.critical);++ // And the other direction: a sampled-heavy vault that has been+ // emptied runs again.+ const heavy = mkdtempSync(join(tmpdir(), "o2b-storage-gate-heavy-"));+ try {+ bootstrapBrain(heavy, {});+ appendFileSync(brainConfigPath(heavy), `\nstorage:\n cap_bytes: ${CAP}\n`);+ mkdirSync(join(heavy, "Brain", "sources"), { recursive: true });+ writeFileSync(join(heavy, "Brain", "sources", "AA-big.md"), "x".repeat(150_000), "utf8");+ sampleStorageUsage(heavy, { runId: "heavy" });+ rmSync(join(heavy, "Brain", "sources", "AA-big.md"));+ const emptied = evaluateCapacityGate(heavy, { capBytes: CAP });+ expect(emptied.decision).toBe("run");+ } finally {+ rmSync(heavy, { recursive: true, force: true });+ }+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "an unmeasurable store never skips and names the refused surface",+ () => {+ configureCap(1000);+ mkdirSync(join(vault, "Brain", ".artifacts"), { recursive: true });+ chmodSync(join(vault, "Brain", ".artifacts"), 0o000);+ try {+ const decision = evaluateCapacityGate(vault, { capBytes: resolvedCap() });+ expect(decision.decision).toBe("run");+ expect(decision.tier).toBe(STORAGE_FORECAST_TIER.unmeasurable);+ expect(decision.reason).toContain("artifacts");+ // A refusal carries no percent: there is no number to attach.+ expect(decision.reading).toBeUndefined();+ } finally {+ chmodSync(join(vault, "Brain", ".artifacts"), 0o755);+ }+ },+ );++ test("threshold overrides move the skip line", () => {+ configureCap(1_000_000);+ writePages("AA-small.md", 100);+ const run = evaluateCapacityGate(vault, { capBytes: resolvedCap() });+ expect(run.decision).toBe("run");+ const skipped = evaluateCapacityGate(vault, {+ capBytes: resolvedCap(),+ criticalPercent: 1,+ });+ expect(skipped.decision).toBe("skip");+ expect(skipped.tier).toBe(STORAGE_FORECAST_TIER.critical);+ });+});diff --git a/tests/core/brain/storage-policy.test.ts b/tests/core/brain/storage-policy.test.tsnew file mode 100644index 00000000..c8b4a5d3--- /dev/null+++ b/tests/core/brain/storage-policy.test.ts@@ -0,0 +1,234 @@+/**+ * The `storage:` policy block (storage lifecycle, Lane 1).+ *+ * Claims pinned here:+ *+ * 1. `resolveStorage` parses the five keys and every present key is+ * bounded: out-of-range values HARD-error naming the field - a+ * silently clamped cap would be indistinguishable from no cap, and+ * for a gate that is the worst possible failure.+ * 2. `critical_percent` must exceed `warn_percent`, and `critical_days`+ * must be smaller than `warn_days` - the same coherence class: a+ * critical band that fires before the warn band inverts the tiers.+ * 3. An absent block resolves to the documented defaults with the cap+ * absent (unbounded), both through the validator and through+ * `loadStorageConfigSafe` on a vault with no `_brain.yaml`.+ * 4. The template ratchet: the block appears in the generated+ * template, `mergeBrainYaml` appends it to an existing `_brain.yaml`+ * without rewriting any existing value, and the merged file+ * validates and resolves.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { DEFAULT_BRAIN_CONFIG_YAML } from "../../../src/core/brain/config-template.ts";+import { parseBrainYaml } from "../../../src/core/brain/yaml-parse.ts";+import { mergeBrainYaml } from "../../../src/core/brain/upgrade.ts";+import { BRAIN_STORAGE_DEFAULTS, resolveStorage } from "../../../src/core/brain/policy.ts";+import {+ loadStorageConfigSafe,+ validateBrainConfigDetailed,+} from "../../../src/core/brain/policy.ts";+import { BrainConfigError } from "../../../src/core/brain/policy/errors.ts";+import { brainConfigPath } from "../../../src/core/brain/paths.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-storage-policy-"));+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++const SOURCE = "<test fixture>";++/** Validate a YAML snippet and resolve its storage block. */+function resolveFromYaml(yaml: string): ReturnType<typeof resolveStorage> {+ const { config } = validateBrainConfigDetailed(parseBrainYaml(yaml), SOURCE);+ return resolveStorage(config);+}++function writeConfig(yaml: string): void {+ mkdirSync(join(vault, "Brain"), { recursive: true });+ writeFileSync(brainConfigPath(vault), yaml, "utf8");+}++const FULL_BLOCK = `schema_version: 1++storage:+ cap_bytes: 1073741824+ warn_percent: 70+ critical_percent: 90+ warn_days: 21+ critical_days: 5+`;++describe("resolveStorage", () => {+ test("parses the five keys", () => {+ expect(resolveFromYaml(FULL_BLOCK)).toEqual({+ cap_bytes: 1073741824,+ warn_percent: 70,+ critical_percent: 90,+ warn_days: 21,+ critical_days: 5,+ });+ });++ test("an absent block resolves to the documented defaults, cap absent", () => {+ expect(BRAIN_STORAGE_DEFAULTS).toEqual({+ cap_bytes: null,+ warn_percent: 80,+ critical_percent: 95,+ warn_days: 30,+ critical_days: 7,+ });+ expect(resolveFromYaml("schema_version: 1")).toEqual(BRAIN_STORAGE_DEFAULTS);+ });++ test("an absent config file resolves to the defaults", () => {+ expect(loadStorageConfigSafe(vault)).toEqual(BRAIN_STORAGE_DEFAULTS);+ });++ test("a partial block keeps the defaults for absent keys", () => {+ expect(resolveFromYaml("schema_version: 1\n\nstorage:\n cap_bytes: 4096\n")).toEqual({+ ...BRAIN_STORAGE_DEFAULTS,+ cap_bytes: 4096,+ });+ });++ test.each([+ ["cap_bytes: 0", "storage.cap_bytes"],+ ["cap_bytes: -5", "storage.cap_bytes"],+ ["cap_bytes: 1.5", "storage.cap_bytes"],+ ['cap_bytes: "big"', "storage.cap_bytes"],+ ["warn_percent: 0", "storage.warn_percent"],+ ["warn_percent: 101", "storage.warn_percent"],+ ["warn_days: 0", "storage.warn_days"],+ ["critical_days: 0", "storage.critical_days"],+ ])("hard-errors the out-of-range value %s", (line, field) => {+ let errored = false;+ try {+ resolveFromYaml(`schema_version: 1\n\nstorage:\n ${line}\n`);+ } catch (err) {+ errored = true;+ expect(err).toBeInstanceOf(BrainConfigError);+ expect((err as BrainConfigError).field).toBe(field);+ }+ expect(errored).toBe(true);+ });++ test.each([[" warn_percent: 95"], [" warn_percent: 100"]])(+ "a warn band at or past the default critical band blames warn_percent: %s",+ (warn) => {+ let errored = false;+ try {+ resolveFromYaml(`schema_version: 1\n\nstorage:\n${warn}\n`);+ } catch (err) {+ errored = true;+ expect(err).toBeInstanceOf(BrainConfigError);+ // critical_percent is not in the file: the key the operator set is+ // the key at fault, not the default it conflicts with.+ expect((err as BrainConfigError).field).toBe("storage.warn_percent");+ }+ expect(errored).toBe(true);+ },+ );++ test.each([[" warn_days: 7"], [" warn_days: 1"]])(+ "a warn horizon at or nearer the default critical horizon blames warn_days: %s",+ (warn) => {+ let errored = false;+ try {+ resolveFromYaml(`schema_version: 1\n\nstorage:\n${warn}\n`);+ } catch (err) {+ errored = true;+ expect(err).toBeInstanceOf(BrainConfigError);+ expect((err as BrainConfigError).field).toBe("storage.warn_days");+ }+ expect(errored).toBe(true);+ },+ );++ test.each([+ [" warn_percent: 80", " critical_percent: 80"],+ [" warn_percent: 80", " critical_percent: 70"],+ ])("refuses a critical band that does not exceed the warn band", (warn, critical) => {+ let errored = false;+ try {+ resolveFromYaml(`schema_version: 1\n\nstorage:\n${warn}\n${critical}\n`);+ } catch (err) {+ errored = true;+ expect(err).toBeInstanceOf(BrainConfigError);+ // Both keys were spelled, so either could yield: the error names+ // the pair, never one side of a conflict the operator can resolve+ // from either direction.+ expect((err as BrainConfigError).field).toBe(+ "storage.warn_percent, storage.critical_percent",+ );+ }+ expect(errored).toBe(true);+ });++ test.each([+ [" warn_days: 7", " critical_days: 7"],+ [" warn_days: 7", " critical_days: 14"],+ ])("refuses a critical horizon not smaller than the warn horizon", (warn, critical) => {+ let errored = false;+ try {+ resolveFromYaml(`schema_version: 1\n\nstorage:\n${warn}\n${critical}\n`);+ } catch (err) {+ errored = true;+ expect(err).toBeInstanceOf(BrainConfigError);+ expect((err as BrainConfigError).field).toBe("storage.warn_days, storage.critical_days");+ }+ expect(errored).toBe(true);+ });++ test("an unreadable config file raises out of the safe loader", () => {+ writeConfig("schema_version: 1\n\nstorage:\n cap_bytes: 0\n");+ expect(() => loadStorageConfigSafe(vault)).toThrow(BrainConfigError);+ });+});++describe("the storage block in the template ratchet", () => {+ test("the generated template carries the block, commented at its defaults", () => {+ expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("storage:");+ expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("warn_percent: 80");+ expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("critical_percent: 95");+ expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("warn_days: 30");+ expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("critical_days: 7");+ });++ test("mergeBrainYaml preserves an existing storage block byte for byte", () => {+ // The template ships the block commented ("nothing new becomes+ // live"), so the merge has nothing to append for it; what the merge+ // must never do is rewrite an operator's own storage values. A live+ // file carrying an operator-set cap merges back byte-identical in+ // that block, and the merged file still validates and resolves.+ const live = [+ "schema_version: 1",+ "",+ "dream:",+ " candidate_threshold: 5",+ " unconfirmed_window_days: 7",+ " contradiction_window_days: 14",+ "",+ "storage:",+ " cap_bytes: 4096",+ "",+ ].join("\n");+ const merged = mergeBrainYaml(live, DEFAULT_BRAIN_CONFIG_YAML);+ expect(merged).toContain("candidate_threshold: 5");+ expect(merged).toContain("cap_bytes: 4096");+ expect(resolveFromYaml(merged)).toEqual({+ ...BRAIN_STORAGE_DEFAULTS,+ cap_bytes: 4096,+ });+ expect(merged).toContain("storage:");+ });+});diff --git a/tests/core/brain/storage-usage.test.ts b/tests/core/brain/storage-usage.test.tsnew file mode 100644index 00000000..a3c5eedc--- /dev/null+++ b/tests/core/brain/storage-usage.test.ts@@ -0,0 +1,206 @@+/**+ * The storage usage meter (`measureStorageUsage`).+ *+ * Claims pinned here:+ *+ * 1. Per-surface byte totals match a hand-computed fixture: the Brain+ * tree (pages, ledgers, config), the derived store plus its WAL and+ * SHM sidecars, and every dot-directory the meter names - artifacts,+ * archives, payloads and internal state are all counted, never+ * skipped by the dot-prefix rule that keeps them out of the index.+ * 2. Tiers split on recoverability, mirroring the state catalogue:+ * `vault-content` for bytes whose loss is permanent, `derived` for+ * bytes whose loss costs a recomputation.+ * 3. A surface that cannot be read is named in `unmeasured` with its+ * reason and never reported as zero bytes - a chmod-denied+ * directory is not an empty one.+ * 4. A surface that is genuinely absent reads as measured-empty: no+ * bytes is a true answer there, and it is not a refusal.+ * 5. The meter is deterministic through the injected `now` alone, and+ * never follows symlinks out of the vault.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { chmodSync, mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { resolveIndexPath } from "../../../src/core/search/paths.ts";+import {+ STORAGE_SURFACE,+ STORAGE_TIER,+ STORAGE_UNMEASURED_REASON,+ measureStorageUsage,+ type StorageUsage,+} from "../../../src/core/brain/storage/usage.ts";+import { CHMOD_CANNOT_DENY, IS_WINDOWS } from "../../helpers/platform.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-storage-usage-"));+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++const NOW = new Date("2026-10-15T12:00:00Z");++/** The exact byte length of a fixture file's content. */+function bytes(text: string): number {+ return Buffer.byteLength(text, "utf8");+}++const PAGE_TEXT = "# AA page\n\nsome body text for the meter to count\n";+const LEDGER_LINE = '{"schema":"o2b.metrics.v1","surface":"storage_usage"}\n';+const ARTIFACT_TEXT = '{"ok":true}\n';+const SNAPSHOT_TEXT = "tar-ish archive bytes\n";+const ARCHIVE_TEXT = "redacted source archive bytes\n";+const PAYLOAD_TEXT = "externalized session turn bytes\n";+const INTERNAL_TEXT = '{"lineage":"row"}\n';+const CONFIG_TEXT = "schema_version: 1\n";++/** The derived store fixture: one sqlite file plus its two sidecars. */+const STORE_MAIN = Buffer.alloc(4096, 0x61);+const STORE_WAL = Buffer.alloc(512, 0x62);+const STORE_SHM = Buffer.alloc(128, 0x63);++/** Hand-computed fixture totals, per surface. */+const FIXTURE_BYTES = {+ page: bytes(PAGE_TEXT),+ config: bytes(CONFIG_TEXT),+ ledger: bytes(LEDGER_LINE),+ artifact: bytes(ARTIFACT_TEXT),+ snapshot: bytes(SNAPSHOT_TEXT),+ archive: bytes(ARCHIVE_TEXT),+ payload: bytes(PAYLOAD_TEXT),+ internal: bytes(INTERNAL_TEXT),+ store: STORE_MAIN.length + STORE_WAL.length + STORE_SHM.length,+};++function writeText(rel: string, text: string): void {+ const full = join(vault, rel);+ mkdirSync(join(full, ".."), { recursive: true });+ writeFileSync(full, text, "utf8");+}++function writeStore(): void {+ const dir = join(vault, ".open-second-brain");+ mkdirSync(dir, { recursive: true });+ writeFileSync(resolveIndexPath(vault, null), STORE_MAIN);+ writeFileSync(`${resolveIndexPath(vault, null)}-wal`, STORE_WAL);+ writeFileSync(`${resolveIndexPath(vault, null)}-shm`, STORE_SHM);+}++function buildFixtureVault(): void {+ writeText("Brain/_brain.yaml", CONFIG_TEXT);+ writeText("Brain/sources/AA-page.md", PAGE_TEXT);+ writeText("Brain/log/continuity/2026-01.jsonl", LEDGER_LINE);+ writeText("Brain/.artifacts/run-1/AA-artifact.json", ARTIFACT_TEXT);+ writeText("Brain/.snapshots/AA-run.tar.zst", SNAPSHOT_TEXT);+ writeText("Brain/.archives/aa1b2c3d.txt", ARCHIVE_TEXT);+ writeText("Brain/.payloads/aabbccdd.txt", PAYLOAD_TEXT);+ writeText("Brain/.state/AA-lineage.jsonl", INTERNAL_TEXT);+ writeStore();+}++function surfaceBytes(usage: StorageUsage, name: string): number {+ const entry = usage.surfaces.find((s) => s.surface === name);+ expect(entry).toBeDefined();+ return entry!.bytes;+}++describe("measureStorageUsage", () => {+ test("per-surface totals match a hand-computed fixture", () => {+ buildFixtureVault();+ const usage = measureStorageUsage(vault, { now: NOW });++ expect(surfaceBytes(usage, STORAGE_SURFACE.brainTree)).toBe(+ FIXTURE_BYTES.page + FIXTURE_BYTES.config + FIXTURE_BYTES.ledger,+ );+ expect(surfaceBytes(usage, STORAGE_SURFACE.artifacts)).toBe(FIXTURE_BYTES.artifact);+ expect(surfaceBytes(usage, STORAGE_SURFACE.snapshots)).toBe(FIXTURE_BYTES.snapshot);+ expect(surfaceBytes(usage, STORAGE_SURFACE.archives)).toBe(FIXTURE_BYTES.archive);+ expect(surfaceBytes(usage, STORAGE_SURFACE.payloads)).toBe(FIXTURE_BYTES.payload);+ expect(surfaceBytes(usage, STORAGE_SURFACE.internalState)).toBe(FIXTURE_BYTES.internal);+ expect(surfaceBytes(usage, STORAGE_SURFACE.derivedStore)).toBe(FIXTURE_BYTES.store);+ });++ test("tier totals split on recoverability and fold into the total", () => {+ buildFixtureVault();+ const usage = measureStorageUsage(vault, { now: NOW });++ const derived = FIXTURE_BYTES.artifact + FIXTURE_BYTES.store;+ const vaultContent =+ FIXTURE_BYTES.page ++ FIXTURE_BYTES.config ++ FIXTURE_BYTES.ledger ++ FIXTURE_BYTES.snapshot ++ FIXTURE_BYTES.archive ++ FIXTURE_BYTES.payload ++ FIXTURE_BYTES.internal;++ expect(usage.bytesByTier[STORAGE_TIER.derived]).toBe(derived);+ expect(usage.bytesByTier[STORAGE_TIER.vaultContent]).toBe(vaultContent);+ expect(usage.totalBytes).toBe(derived + vaultContent);+ expect(usage.complete).toBe(true);+ expect(usage.unmeasured).toEqual([]);+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "an unreadable surface is named in unmeasured and never reported as zero bytes",+ () => {+ buildFixtureVault();+ chmodSync(join(vault, "Brain", ".artifacts"), 0o000);+ try {+ const usage = measureStorageUsage(vault, { now: NOW });+ const refused = usage.unmeasured.find((u) => u.surface === STORAGE_SURFACE.artifacts);+ expect(refused).toEqual({+ surface: STORAGE_SURFACE.artifacts,+ reason: STORAGE_UNMEASURED_REASON.surfaceUnreadable,+ });+ // No byte claim: the artifacts reading is absent from the measured+ // set entirely, not flattened to a lying zero.+ expect(usage.surfaces.find((s) => s.surface === STORAGE_SURFACE.artifacts)).toBeUndefined();+ expect(usage.complete).toBe(false);+ // The surfaces that could be read keep their measured bytes.+ expect(surfaceBytes(usage, STORAGE_SURFACE.brainTree)).toBe(+ FIXTURE_BYTES.page + FIXTURE_BYTES.config + FIXTURE_BYTES.ledger,+ );+ } finally {+ chmodSync(join(vault, "Brain", ".artifacts"), 0o755);+ }+ },+ );++ test("an absent surface reads as measured-empty, not as a refusal", () => {+ // No Brain tree, no store: a vault that has never been initialized.+ const usage = measureStorageUsage(vault, { now: NOW });+ expect(usage.totalBytes).toBe(0);+ expect(usage.unmeasured).toEqual([]);+ expect(usage.complete).toBe(true);+ expect(surfaceBytes(usage, STORAGE_SURFACE.brainTree)).toBe(0);+ expect(surfaceBytes(usage, STORAGE_SURFACE.derivedStore)).toBe(0);+ });++ test("the injected now alone decides the timestamp", () => {+ buildFixtureVault();+ const a = measureStorageUsage(vault, { now: NOW });+ const b = measureStorageUsage(vault, { now: NOW });+ expect(a.measuredAt).toBe("2026-10-15T12:00:00.000Z");+ expect(b).toEqual(a);+ });++ test("symlinks are never followed out of the vault", () => {+ if (IS_WINDOWS) return; // symlink creation needs privileges there+ buildFixtureVault();+ const outside = join(vault, "outside.md");+ writeFileSync(outside, "external bytes".repeat(100), "utf8");+ symlinkSync(outside, join(vault, "Brain", "AA-link.md"));+ const usage = measureStorageUsage(vault, { now: NOW });+ expect(surfaceBytes(usage, STORAGE_SURFACE.brainTree)).toBe(+ FIXTURE_BYTES.page + FIXTURE_BYTES.config + FIXTURE_BYTES.ledger,+ );+ });+});diff --git a/tests/core/brain/sync-lockfile.test.ts b/tests/core/brain/sync-lockfile.test.tsindex c3c7b7f9..26e681eb 100644--- a/tests/core/brain/sync-lockfile.test.ts+++ b/tests/core/brain/sync-lockfile.test.ts@@ -239,6 +239,7 @@ describe("every module that takes a lock is under a scanned root", () => {["core/brain/payload-registry.ts", "Brain"],["core/brain/preference-txn.ts", "Brain"],["core/brain/query-demand.ts", "Brain"],+ ["core/brain/receipts/compact.ts", "Brain"],["core/brain/sessions/discover.ts", ".open-second-brain"],["core/brain/skill-proposals.ts", "Brain"],]);diff --git a/tests/core/brain/vault-guard-census.test.ts b/tests/core/brain/vault-guard-census.test.tsindex afb28868..8b86a9e5 100644--- a/tests/core/brain/vault-guard-census.test.ts+++ b/tests/core/brain/vault-guard-census.test.ts@@ -66,6 +66,14 @@ const UNGUARDED_WITH_REASON: Readonly<Record<string, string>> = Object.freeze({"never writes. It names `openSync` only to open a project manifest " +"read-only (O_RDONLY | O_NOFOLLOW | O_NONBLOCK) through one descriptor so " +"the read is bounded; the file lives in the scanned project, outside the vault.",+ "src/core/brain/ingest/source-read.ts":+ "never writes. It names `openSync` only to open a source read-only " ++ "(O_RDONLY | O_NONBLOCK, the shared SOURCE_OPEN_FLAGS) through one " ++ "descriptor so the read is bounded, so there is no write to guard.",+ "src/core/brain/ingest/tail-read.ts":+ "never writes. Its one `openSync` reuses the read-only SOURCE_OPEN_FLAGS " ++ "for the from-offset delta read; the watermark it feeds is written by " ++ "the content manifest's guarded writer, not here.","src/core/brain/dedup-index-cache.ts":"machine-local derived cache OUTSIDE the vault (the user cache dir). It " +"writes nothing under Brain/, and a refusal would only cost the next " +diff --git a/tests/core/install/tool-ceiling.test.ts b/tests/core/install/tool-ceiling.test.tsindex db03893c..898df3f2 100644--- a/tests/core/install/tool-ceiling.test.ts+++ b/tests/core/install/tool-ceiling.test.ts@@ -345,7 +345,7 @@ describe("the capability report states the ceiling it runs under", () => {test("the advertised count is the one a host lists, not the registered one", () => {const evaluated = report("cursor");- expect(evaluated.available_tool_count).toBe(115);+ expect(evaluated.available_tool_count).toBe(117);expect(evaluated.advertised_tool_count).toBe(7);});});diff --git a/tests/helpers/tool-probe-catalogue.ts b/tests/helpers/tool-probe-catalogue.tsindex e32de062..3924804d 100644--- a/tests/helpers/tool-probe-catalogue.ts+++ b/tests/helpers/tool-probe-catalogue.ts@@ -296,8 +296,8 @@ export const REASONS_REACHING_OWNER_CONTENT: ReadonlySet<string> = new Set([REAS* `docs/mcp.md` and the release notes. Equalities, not floors - see the* test that reads them.*/-export const PROBE_ENTRY_COUNT = 101;-export const PROBE_RECIPE_COUNT = 235;+export const PROBE_ENTRY_COUNT = 103;+export const PROBE_RECIPE_COUNT = 238;export const PROBE_TWO_SIDED_COUNT = 32;/**@@ -923,6 +923,21 @@ export const NON_CONTENT: ReadonlyArray<ProbeEntry> = [{ args: { operation: "cost" }, reason: REASON.ownerlessLane },],},+ {+ name: "brain_receipts",+ calls: [+ // Default mode is the scan: counts, byte sizes and vault-relative+ // shard paths over the receipt ledgers, which carry no `owner:`+ // field to filter on - the same lane as the telemetry above.+ { args: {}, reason: REASON.ownerlessLane },+ // A dry-run compact over an absent shard: the fold plan and the+ // collected unreadable-shard failure, no write and no record body.+ {+ args: { mode: "compact", shards: ["continuity:receipt-probe-absent.jsonl"] },+ reason: REASON.ownerlessLane,+ },+ ],+ },{name: "brain_recurrence",calls: [@@ -996,6 +1011,13 @@ export const NON_CONTENT: ReadonlyArray<ProbeEntry> = [{ name: "brain_sources", calls: one({}, REASON.signalAuthorship) },{ name: "brain_stale_scan", calls: one({}, REASON.ownerFiltered) },{ name: "brain_status", calls: one({}, REASON.aggregateOnly) },+ {+ // The storage report is an aggregate like brain_status beside it:+ // per-surface byte counts and a forecast tier over the whole vault,+ // nothing an owner field could filter and nothing the probe wrote.+ name: "brain_storage",+ calls: one({}, REASON.aggregateOnly),+ },{ name: "brain_switch_vault", calls: one({ name: "probe-profile" }, REASON.writerEcho) },{name: "brain_tension",diff --git a/tests/mcp/brain-receipts.test.ts b/tests/mcp/brain-receipts.test.tsnew file mode 100644index 00000000..dc915813--- /dev/null+++ b/tests/mcp/brain-receipts.test.ts@@ -0,0 +1,242 @@+/**+ * `brain_receipts` at the MCP level (storage-lifecycle, Lane 3 task 3.4).+ *+ * Scan is the default and writes nothing. Compact requires explicit+ * `family:name` shards, previews without `apply`, and enforces the count+ * guard before mutating - a mismatched `expect` is an INVALID_PARAMS+ * envelope naming the count, and the vault is untouched. Compact is the+ * destructive half and answers at local reach only: below it the tool+ * refuses by name. The audit receipt_compaction record lands once per+ * confirmed run.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { continuityLogDir, listContinuityRecords } from "../../src/core/brain/continuity/store.ts";+import { metricsDir } from "../../src/core/brain/metrics.ts";+import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { resetVaultIdentityPins } from "../../src/core/brain/vault-identity.ts";+import { RECEIPT_SHARD_UNREADABLE } from "../../src/core/brain/receipts/scan.ts";+import {+ RECEIPT_COMPACTION_OPERATION,+ RECEIPT_MODES,+} from "../../src/core/brain/receipts/types.ts";+import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";+import { RECEIPT_COMPACT_REACH_REASON, RECEIPT_TOOLS } from "../../src/mcp/brain/receipt-tools.ts";+import { CHMOD_CANNOT_DENY } from "../helpers/platform.ts";+import { digestVaultTree } from "../helpers/vault-digest.ts";+import { JSONRPC_VERSION, MCPServer, PROTOCOL_VERSION } from "../../src/mcp/index.ts";++const LEDGER_MONTH = "2026-08";++/** The schema stamp every aggregate line the fold engine writes carries. */+const RECEIPT_LEDGER_MONTH_AGGREGATE_MARKER = "o2b.receipt-fold.v1";++let vault: string;++function continuityLine(kind: string, createdAt: string): string {+ const id = `ctn_${createdAt.replace(/[^0-9]/g, "").slice(0, 14)}_${kind}`;+ return `${JSON.stringify({+ schema: "o2b.continuity.v1",+ id,+ kind,+ createdAt,+ sourceRefs: [],+ payload: {},+ private: false,+ redacted: false,+ })}\n`;+}++function seedLedger(): void {+ const lines: string[] = [];+ for (let i = 0; i < 5; i += 1) {+ lines.push(continuityLine("recall_telemetry", `${LEDGER_MONTH}-0${i + 1}T12:00:0${i}Z`));+ }+ mkdirSync(continuityLogDir(vault), { recursive: true });+ mkdirSync(metricsDir(vault), { recursive: true });+ writeFileSync(join(continuityLogDir(vault), `${LEDGER_MONTH}.jsonl`), lines.join(""), "utf8");+}++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-mcp-receipts-"));+ bootstrapBrain(vault);+});++afterEach(() => {+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++async function init(server: MCPServer): Promise<void> {+ await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 1,+ method: "initialize",+ params: {+ protocolVersion: PROTOCOL_VERSION,+ capabilities: {},+ clientInfo: { name: "t", version: "0" },+ },+ });+ await server.handleRequest({ jsonrpc: JSONRPC_VERSION, method: "notifications/initialized" });+}++async function call(server: MCPServer, args: Record<string, unknown>): Promise<any> {+ return (await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 9,+ method: "tools/call",+ params: { name: "brain_receipts", arguments: args },+ })) as any;+}++describe("brain_receipts", () => {+ test("scan is the default mode and writes nothing", async () => {+ seedLedger();+ const server = new MCPServer({ vault });+ await init(server);+ const before = digestVaultTree(vault);+ const response = await call(server, {});+ const view = response.result.structuredContent;+ expect(view.operation).toBe(RECEIPT_COMPACTION_OPERATION);+ expect(view.totals.records).toBe(5);+ expect(view.totals.foldable).toBe(5);+ expect(view.shards[0]!.protected_records).toBe(0);+ expect(digestVaultTree(vault)).toBe(before);+ });++ test("an unknown mode is an INVALID_PARAMS envelope naming the closed modes", async () => {+ const server = new MCPServer({ vault });+ await init(server);+ const response = await call(server, { mode: "vacuum" });+ expect(response.error.code).toBe(-32602);+ expect(response.error.message).toContain("scan, compact");+ });++ test("compact without shard ids is an INVALID_PARAMS envelope", async () => {+ // Local reach: the reach refusal for the destructive mode is its own+ // test (`receipts-reach.test.ts`); this is the argument contract.+ const server = new MCPServer({ vault }, { reach: TRANSPORT_REACH.local });+ await init(server);+ const response = await call(server, { mode: "compact" });+ expect(response.error.code).toBe(-32602);+ expect(response.error.message).toContain("shards");+ });++ test("compact previews without apply and writes nothing", async () => {+ seedLedger();+ const server = new MCPServer({ vault }, { reach: TRANSPORT_REACH.local });+ await init(server);+ const before = digestVaultTree(vault);+ const response = await call(server, {+ mode: "compact",+ shards: [`continuity:${LEDGER_MONTH}.jsonl`],+ });+ const view = response.result.structuredContent;+ expect(view.dry_run).toBe(true);+ expect(view.totals.folded).toBe(5);+ expect(digestVaultTree(vault)).toBe(before);+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);+ });++ test("a mismatched expect aborts the confirmed compact without writing", async () => {+ seedLedger();+ const server = new MCPServer({ vault }, { reach: TRANSPORT_REACH.local });+ await init(server);+ const before = readFileSync(join(continuityLogDir(vault), `${LEDGER_MONTH}.jsonl`), "utf8");+ const response = await call(server, {+ mode: "compact",+ shards: [`continuity:${LEDGER_MONTH}.jsonl`],+ apply: true,+ expect: 4,+ });+ expect(response.error.code).toBe(-32602);+ expect(response.error.message).toContain("--expect 4");+ expect(readFileSync(join(continuityLogDir(vault), `${LEDGER_MONTH}.jsonl`), "utf8")).toBe(+ before,+ );+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);+ });++ test("a confirmed compact folds, appends one audit record, and reports the recovery point", async () => {+ seedLedger();+ const server = new MCPServer({ vault }, { reach: TRANSPORT_REACH.local });+ await init(server);+ const response = await call(server, {+ mode: "compact",+ shards: [`continuity:${LEDGER_MONTH}.jsonl`],+ apply: true,+ expect: 5,+ });+ const view = response.result.structuredContent;+ expect(view.dry_run).toBe(false);+ expect(view.totals.folded).toBe(5);+ expect(view.snapshot_run_id).toBeDefined();+ expect(view.audit_record_id).toBeDefined();+ expect(view.shards[0]!.bytes_after).toBeLessThan(view.shards[0]!.bytes_before);+ const audits = listContinuityRecords(vault, { kind: "receipt_compaction" });+ expect(audits).toHaveLength(1);+ expect(audits[0]!.id).toBe(view.audit_record_id);+ const after = readFileSync(join(continuityLogDir(vault), `${LEDGER_MONTH}.jsonl`), "utf8");+ expect(after).toContain(RECEIPT_LEDGER_MONTH_AGGREGATE_MARKER);+ });++ test("compact below local reach is refused by name, never planned", async () => {+ // A server whose transport minted no reach resolves to the+ // narrowest, so the destructive mode answers the named refusal: no+ // shard plan, no byte totals, and the vault untouched.+ seedLedger();+ const server = new MCPServer({ vault });+ await init(server);+ const before = digestVaultTree(vault);+ const response = await call(server, {+ mode: "compact",+ shards: [`continuity:${LEDGER_MONTH}.jsonl`],+ apply: true,+ });+ const view = response.result.structuredContent;+ expect(view.measured).toBe(false);+ expect(view.reason).toBe(RECEIPT_COMPACT_REACH_REASON);+ expect(view.shards).toBeUndefined();+ expect(view.totals).toBeUndefined();+ expect(digestVaultTree(vault)).toBe(before);+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);+ });++ test.skipIf(CHMOD_CANNOT_DENY)(+ "a shard the scan cannot read surfaces as unreadable, never as a filesystem error",+ async () => {+ // The scan view rides to remote callers verbatim, so the error rows+ // must carry the shard's name and the sanitized verdict only - the+ // raw Node text would hand over an absolute host path and a+ // missing-versus-denied oracle.+ seedLedger();+ const shard = join(continuityLogDir(vault), `${LEDGER_MONTH}.jsonl`);+ chmodSync(shard, 0o000);+ try {+ const server = new MCPServer({ vault });+ await init(server);+ const response = await call(server, {});+ const view = response.result.structuredContent;+ expect(view.errors).toHaveLength(1);+ expect(view.errors[0]!.shard).toBe(`continuity:${LEDGER_MONTH}.jsonl`);+ expect(view.errors[0]!.message).toBe(RECEIPT_SHARD_UNREADABLE);+ expect(JSON.stringify(view)).not.toContain(vault);+ } finally {+ chmodSync(shard, 0o644);+ }+ },+ );++ test("the surface vocabulary derives from the shared modes tuple", () => {+ const tool = RECEIPT_TOOLS[0]!;+ const schemaModes = (tool.inputSchema.properties as { mode: { enum: string[] } }).mode.enum;+ expect(schemaModes).toEqual([...RECEIPT_MODES]);+ expect(tool.description.toLowerCase()).toContain("receipt compaction");+ });+});diff --git a/tests/mcp/brain-storage.test.ts b/tests/mcp/brain-storage.test.tsnew file mode 100644index 00000000..0039ca92--- /dev/null+++ b/tests/mcp/brain-storage.test.ts@@ -0,0 +1,249 @@+/**+ * `brain_storage` (storage-lifecycle, Lane 1 task 1.7).+ *+ * Claims pinned here:+ *+ * 1. The tool and `o2b brain storage` derive from one builder: driven+ * on the same vault state, their reports are equal on every field+ * except the two per-invocation identity fields (`usage.measuredAt`+ * and `sample.runId`), which are masked to a fixed marker - never+ * value-compared, because their values ride the invoking clock and+ * can collide when two calls share one tick. No flag-or-property+ * exclusion list is needed because the payload IS the shared source.+ * 2. An uncapped vault answers tier `unbounded` - bytes reported, no+ * invented cap, nothing refused.+ * 3. Each invocation appends exactly one `storage_usage` sample.+ * 4. An unmeasurable census is carried (`unmeasured` with reasons),+ * never flattened into a zero or an invented projection.+ * 5. A vault with no brain config surfaces the named config error;+ * the tool does not answer as an empty vault.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { cmdBrainStorage } from "../../src/cli/brain/verbs/storage.ts";+import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { resetVaultIdentityPins } from "../../src/core/brain/vault-identity.ts";+import { brainConfigPath } from "../../src/core/brain/paths.ts";+import { metricsDir, listMetrics, type MetricRecord } from "../../src/core/brain/metrics.ts";+import { STORAGE_USAGE_METRIC_SURFACE } from "../../src/core/brain/storage/forecast.ts";+import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { STORAGE_TOOLS } from "../../src/mcp/brain/storage-tools.ts";+import type { ServerContext } from "../../src/mcp/tool-contract.ts";+import { CHMOD_CANNOT_DENY } from "../helpers/platform.ts";++const storage = STORAGE_TOOLS.find((tool) => tool.name === "brain_storage")!.handler;++let vault: string;+let configHome: string;+let savedConfigEnv: string | undefined;+let ctx: ServerContext;++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-mcp-storage-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-mcp-storage-cfg-"));+ atomicWriteFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+ savedConfigEnv = process.env["OPEN_SECOND_BRAIN_CONFIG"];+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = join(configHome, "config.yaml");+ ctx = {+ vault,+ configPath: join(configHome, "config.yaml"),+ repoRoot: null,+ reach: TRANSPORT_REACH.local,+ };+});++afterEach(() => {+ if (savedConfigEnv === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfigEnv;+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++interface ReportJson {+ measured: boolean;+ usage: Record<string, unknown>;+ forecast: Record<string, unknown>;+ sample: Record<string, unknown>;+}++function writePage(name: string, totalBytes: number): void {+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", name), "x".repeat(totalBytes), "utf8");+}++function configureCap(capBytes: number): void {+ writeFileSync(+ brainConfigPath(vault),+ `schema_version: 1\n\nstorage:\n cap_bytes: ${capBytes}\n`,+ "utf8",+ );+}++/** The `brain_storage` report for the CLI surface, driven in-process. */+async function cliReport(): Promise<ReportJson> {+ let out = "";+ const original = process.stdout.write;+ process.stdout.write = ((chunk: Uint8Array | string): boolean => {+ out += typeof chunk === "string" ? chunk : Buffer.from(chunk).toString("utf8");+ return true;+ }) as typeof process.stdout.write;+ try {+ const code = await cmdBrainStorage(["--json"]);+ expect(code).toBe(0);+ return JSON.parse(out) as ReportJson;+ } finally {+ process.stdout.write = original;+ }+}++/** The usage samples recorded for this vault so far. */+function usageSamples(): MetricRecord[] {+ return listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE });+}++/**+ * Every leaf path where `a` and `b` differ, as `dot.join`ed paths.+ * The parity claim is asserted over the full path set, so a field that+ * starts to drift between the two surfaces is named by the failure.+ */+function differingPaths(a: unknown, b: unknown, prefix = ""): string[] {+ if (typeof a === "object" && a !== null && typeof b === "object" && b !== null) {+ const ra = a as Record<string, unknown>;+ const rb = b as Record<string, unknown>;+ const keys = [...new Set([...Object.keys(ra), ...Object.keys(rb)])].toSorted();+ return keys.flatMap((key) =>+ differingPaths(ra[key], rb[key], `${prefix}${prefix ? "." : ""}${key}`),+ );+ }+ return Object.is(a, b) ? [] : [prefix];+}++/**+ * The leaf paths whose values are the invoking call's own identity: the+ * instant the census was taken and the run id the sample was recorded+ * under. A value comparison over them would make this claim depend on+ * the clock - two calls that straddle a tick differ, two that share one+ * collide - so the observed flake. Masking to a fixed marker, instead+ * of dropping the paths, keeps each field's PRESENCE part of the parity+ * claim while taking its value off the clock entirely.+ */+const INVOCATION_IDENTITY_PATHS: ReadonlyArray<string> = ["sample.runId", "usage.measuredAt"];+const MASKED_IDENTITY = "<invocation identity>";++/** A deep copy of `value` whose leaves at the invocation identity paths are masked. */+function maskPaths(value: unknown, prefix = ""): unknown {+ if (INVOCATION_IDENTITY_PATHS.includes(prefix)) return MASKED_IDENTITY;+ if (typeof value === "object" && value !== null) {+ const record = value as Record<string, unknown>;+ return Object.fromEntries(+ Object.entries(record).map(([key, child]) => [+ key,+ maskPaths(child, prefix === "" ? key : `${prefix}.${key}`),+ ]),+ );+ }+ return value;+}++describe("brain_storage", () => {+ test("the tool and the verb derive from one source", async () => {+ const fromVerb = await cliReport();+ // The verb's sample appended one metrics record; dropping the ledger+ // returns the vault to the state the verb's census measured, so the+ // tool's census sees the same bytes.+ rmSync(metricsDir(vault), { recursive: true, force: true });+ const fromTool = (await storage(ctx, {})) as { format: string } & ReportJson;+ // `format` is the transport's rendering hint, not report content.+ const { format, ...report } = fromTool;+ void format;+ // The two per-invocation identity fields are masked, not excluded+ // from the diff: masking keeps their presence in the claim while+ // taking their values off the clock, so two calls that share a tick+ // and two that straddle one prove the same thing.+ expect(differingPaths(maskPaths(fromVerb), maskPaths(report))).toEqual([]);+ });++ test("an uncapped vault answers unbounded, measured", async () => {+ writePage("AA-page.md", 20_000);+ const res = (await storage(ctx, {})) as ReportJson;+ expect(res.measured).toBe(true);+ expect(res.forecast["tier"]).toBe("unbounded");+ expect(res.forecast["capBytes"]).toBeNull();+ expect(res.forecast["percentUsed"]).toBeNull();+ expect(res.forecast["daysToCap"]).toBeNull();+ expect((res.usage as { complete: boolean }).complete).toBe(true);+ expect(res.sample["appended"]).toBe(true);+ });++ test("each invocation appends exactly one usage sample", async () => {+ expect(usageSamples()).toHaveLength(0);+ await storage(ctx, {});+ expect(usageSamples()).toHaveLength(1);+ await storage(ctx, {});+ expect(usageSamples()).toHaveLength(2);+ });++ test.skipIf(CHMOD_CANNOT_DENY)("an unmeasurable census is carried, not flattened", async () => {+ writePage("AA-page.md", 9_000);+ configureCap(10_000);+ mkdirSync(join(vault, "Brain", ".artifacts"), { recursive: true });+ chmodSync(join(vault, "Brain", ".artifacts"), 0o000);+ try {+ const res = (await storage(ctx, {})) as ReportJson;+ const usage = res.usage as {+ complete: boolean;+ unmeasured: ReadonlyArray<Record<string, string>>;+ surfaces: ReadonlyArray<{ surface: string }>;+ };+ expect(usage.complete).toBe(false);+ expect(usage.unmeasured).toEqual([{ surface: "artifacts", reason: "surface_unreadable" }]);+ expect(usage.surfaces.map((s) => s.surface)).not.toContain("artifacts");+ const forecast = res.forecast as {+ tier: string;+ unmeasured: ReadonlyArray<{ surface: string }>;+ };+ expect(forecast.tier).toBe("unmeasurable");+ expect(forecast.unmeasured.map((u) => u.surface)).toEqual(["artifacts"]);+ } finally {+ chmodSync(join(vault, "Brain", ".artifacts"), 0o755);+ }+ });++ test("a vault with no brain config surfaces the named config error", async () => {+ const bare = mkdtempSync(join(tmpdir(), "o2b-mcp-storage-bare-"));+ try {+ const bareCtx: ServerContext = {+ vault: bare,+ configPath: null,+ repoRoot: null,+ reach: TRANSPORT_REACH.local,+ };+ expect(async () => storage(bareCtx, {})).toThrow(/config file does not exist/);+ } finally {+ rmSync(bare, { recursive: true, force: true });+ }+ });++ test("the tool declares a bounded, described schema", () => {+ const def = STORAGE_TOOLS.find((tool) => tool.name === "brain_storage")!;+ expect(def.inputSchema["additionalProperties"]).toBe(false);+ expect(def.previewBudget).toBeDefined();+ });++ test("the description states the sample side effect beside the read-only claim", () => {+ const def = STORAGE_TOOLS.find((tool) => tool.name === "brain_storage")!;+ // A bare "Read-only." would contradict the one append every local+ // call makes: the description names both halves of the contract.+ expect(def.description).toContain("Read-only over vault content");+ expect(def.description).toContain("appends one usage sample");+ expect(def.description.length).toBeLessThanOrEqual(300);+ });+});diff --git a/tests/mcp/brain-tools-parity.test.ts b/tests/mcp/brain-tools-parity.test.tsindex 57a0158a..ee0efe07 100644--- a/tests/mcp/brain-tools-parity.test.ts+++ b/tests/mcp/brain-tools-parity.test.ts@@ -63,7 +63,15 @@* validated answer as speculative `source_type: auto_extract` inbox* signals), and `brain_design_note` (the one-shot sibling of the panel* lane: a read-only grounding pass over tension, decision and truth- * records, one envelope, and a committed note under `Brain/decisions/`).+ * records, one envelope, and a committed note under `Brain/decisions/`);+ * the storage-lifecycle release added `brain_receipts` (receipt+ * compaction: a read-only per-shard scan of the continuity and metrics+ * receipt ledgers, and the explicit-shard fold behind a count guard and+ * a recovery point - deliberately disjoint from the context-compaction+ * vocabulary) and `brain_storage` (the storage report: the per-surface+ * measured-or-refused byte census, the forecast and tier against the+ * configured cap, and the usage sample each invocation records; the+ * vault-wide measurement answers measured: false below local reach).*/import { describe, expect, test } from "bun:test";@@ -137,6 +145,7 @@ const FROZEN_BRAIN_TOOL_NAMES = ["brain_procedural_memory","brain_query","brain_recall_telemetry",+ "brain_receipts","brain_recurrence","brain_research_report","brain_retention",@@ -155,6 +164,7 @@ const FROZEN_BRAIN_TOOL_NAMES = ["brain_sources","brain_stale_scan","brain_status",+ "brain_storage","brain_switch_vault","brain_tension","brain_tiers",diff --git a/tests/mcp/hygiene-error-prose-reach.test.ts b/tests/mcp/hygiene-error-prose-reach.test.tsnew file mode 100644index 00000000..8ca98484--- /dev/null+++ b/tests/mcp/hygiene-error-prose-reach.test.ts@@ -0,0 +1,139 @@+/**+ * `brain_hygiene` states its error rows for the caller's reach.+ *+ * The scan, refresh and apply payloads carry error rows on the SUCCESS+ * channel, which the server's redactor never sees - it wraps thrown+ * errors only. A detector or executor failure forwards failure prose+ * that names absolute host paths (Node's fs text, a corrupted manifest's+ * path), so the surface states that prose for the caller's reach the+ * same way a throw is stated: substituted below local reach, verbatim at+ * local reach.+ *+ * Claims pinned here:+ *+ * 1. Below local reach an apply failure's prose carries the vault as a+ * placeholder, never the host path.+ * 2. At local reach the same failure's prose is unchanged - the+ * operator's own surface loses no detail.+ * 3. The failure itself is honest at both reaches: the row names the+ * finding, and the message says the retired copy is not on disk.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { manifestPath } from "../../src/core/brain/ingest/content-manifest.ts";+import { HYGIENE_TOOLS } from "../../src/mcp/brain/hygiene-tools.ts";+import type { ServerContext } from "../../src/mcp/tool-contract.ts";+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";++let tmp: string;+let vault: string;+let configPath: string;++const CANONICAL_BODY =+ "Collect the latency metrics before optimizing the render loop and archive the traces weekly";+const DUPLICATE_BODY =+ "Collect the latency metrics before optimizing the render loop and archive the traces daily";++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-hygiene-prose-reach-"));+ vault = join(tmp, "vault");+ configPath = join(tmp, "config.yaml");+ writeFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`, "utf8");+ bootstrapBrain(vault, { configPath });+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ mkdirSync(join(vault, "imports"), { recursive: true });+ writeFileSync(join(vault, "imports", "a.md"), "older source bytes\n", "utf8");+ writeFileSync(join(vault, "imports", "b.md"), "newer source bytes\n", "utf8");+ for (const [slug, body, createdAt] of [+ ["a", CANONICAL_BODY, "2026-01-01T00:00:00Z"],+ ["b", DUPLICATE_BODY, "2026-02-01T00:00:00Z"],+ ] as const) {+ writeFileSync(+ join(vault, "Brain", "sources", `src-${slug}.md`),+ [+ "---",+ "kind: brain-source",+ `source_path: imports/${slug}.md`,+ `source_hash: ${"a".repeat(64)}`,+ "tags: [brain, brain/source]",+ `created_at: ${createdAt}`,+ "---",+ "",+ body,+ "",+ ].join("\n"),+ "utf8",+ );+ }+ // The retirement's manifest step reads this file after the copy is off+ // disk; corrupted, its own error names the file by its host path - the+ // prose a caller's error row would otherwise carry verbatim.+ mkdirSync(join(vault, ".open-second-brain"), { recursive: true });+ writeFileSync(manifestPath(vault), "not json{", "utf8");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++function ctx(reach: TransportReach): ServerContext {+ return {+ vault,+ configPath,+ repoRoot: null,+ requestIdentity: { agent: "codex", via: "token" as const },+ reach,+ };+}++async function retireFindingId(): Promise<string> {+ const tool = HYGIENE_TOOLS.find((t) => t.name === "brain_hygiene")!;+ const scan = (await tool.handler(ctx(TRANSPORT_REACH.local), {+ mode: "scan",+ detectors: ["document-dedup"],+ })) as { findings: Array<{ id: string; proposed_action: string }> };+ const finding = scan.findings.find((f) => f.proposed_action === "retire");+ if (finding === undefined) throw new Error("fixture produced no retire finding");+ return finding.id;+}++async function applyRetire(+ reach: TransportReach,+ id: string,+): Promise<Array<{ finding_id: string; message: string }>> {+ const tool = HYGIENE_TOOLS.find((t) => t.name === "brain_hygiene")!;+ const result = (await tool.handler(ctx(reach), {+ mode: "apply",+ detectors: ["document-dedup"],+ ids: [id],+ })) as { errors: Array<{ finding_id: string; message: string }> };+ return result.errors;+}++describe("brain_hygiene error rows at the caller's reach", () => {+ test("below local reach an apply failure's prose carries the vault as a placeholder", async () => {+ const id = await retireFindingId();+ const errors = await applyRetire(TRANSPORT_REACH.remote, id);+ expect(errors).toHaveLength(1);+ expect(errors[0]!.finding_id).toBe(id);+ const message = errors[0]!.message;+ expect(message).toContain("corrupted JSON");+ expect(message).toContain("<vault>");+ expect(message).not.toContain(vault);+ // The failure stays honest about what landed: the copy is gone.+ expect(message).toContain("not on disk");+ });++ test("at local reach the same failure's prose is unchanged", async () => {+ const id = await retireFindingId();+ const errors = await applyRetire(TRANSPORT_REACH.local, id);+ expect(errors).toHaveLength(1);+ expect(errors[0]!.message).toContain(vault);+ expect(errors[0]!.message).not.toContain("<vault>");+ });+});diff --git a/tests/mcp/hygiene-link-integrity.test.ts b/tests/mcp/hygiene-link-integrity.test.tsindex 167ea9ca..e7439bae 100644--- a/tests/mcp/hygiene-link-integrity.test.ts+++ b/tests/mcp/hygiene-link-integrity.test.ts@@ -70,6 +70,7 @@ test("the detector tuple and its input-schema enum stay in lockstep", () => {expect(HYGIENE_DETECTOR_IDS).toEqual(["conflicts","dedup",+ "document-dedup","freshness","usefulness","slug-collisions",diff --git a/tests/mcp/hygiene-retire-deny.test.ts b/tests/mcp/hygiene-retire-deny.test.tsnew file mode 100644index 00000000..3de8016a--- /dev/null+++ b/tests/mcp/hygiene-retire-deny.test.ts@@ -0,0 +1,157 @@+/**+ * `brain_hygiene` apply answers the deny consult for the request's+ * credential identity (write-side trust).+ *+ * Retire is the destructive action the hygiene applier routes to+ * `retireDocumentCopies`, whose deny consult refuses on a+ * permissions-document rule naming the consult's subject. The sibling+ * destructive surfaces (`brain_delete_by_source`, `brain_receipts`+ * compaction) thread `ctx.requestIdentity` into that consult; the+ * hygiene apply must too - a rule naming the token's agent is enforced+ * here exactly as on the other destructive routes, instead of falling+ * back to the config agent and applying anyway.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { HYGIENE_TOOLS } from "../../src/mcp/brain/hygiene-tools.ts";++let tmp: string;+let vault: string;+let configPath: string;+const savedEnv: Record<string, string | undefined> = {};++const CANONICAL_BODY =+ "Collect the latency metrics before optimizing the render loop and archive the traces weekly";+const DUPLICATE_BODY =+ "Collect the latency metrics before optimizing the render loop and archive the traces daily";++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-hygiene-retire-deny-"));+ vault = join(tmp, "vault");+ configPath = join(tmp, "config.yaml");+ for (const key of ["VAULT_AGENT_NAME", "OPEN_SECOND_BRAIN_CONFIG"]) {+ savedEnv[key] = process.env[key];+ delete process.env[key];+ }+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = configPath;+ // The config declares `claude`: only a consult answering for the+ // request identity can ever refuse with `codex`.+ writeFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`, "utf8");+ bootstrapBrain(vault, { configPath });+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ mkdirSync(join(vault, "imports"), { recursive: true });+ writeFileSync(join(vault, "imports", "a.md"), "older source bytes\n", "utf8");+ writeFileSync(join(vault, "imports", "b.md"), "newer source bytes\n", "utf8");+ for (const [slug, body, createdAt] of [+ ["a", CANONICAL_BODY, "2026-01-01T00:00:00Z"],+ ["b", DUPLICATE_BODY, "2026-02-01T00:00:00Z"],+ ] as const) {+ writeFileSync(+ join(vault, "Brain", "sources", `src-${slug}.md`),+ [+ "---",+ "kind: brain-source",+ `source_path: imports/${slug}.md`,+ `source_hash: ${"a".repeat(64)}`,+ "tags: [brain, brain/source]",+ `created_at: ${createdAt}`,+ "---",+ "",+ body,+ "",+ ].join("\n"),+ "utf8",+ );+ }+});++afterEach(() => {+ for (const [key, value] of Object.entries(savedEnv)) {+ if (value === undefined) delete process.env[key];+ else process.env[key] = value;+ }+ rmSync(tmp, { recursive: true, force: true });+});++/** A hand-built per-request context whose credential minted `codex`. */+function identityCtx() {+ return {+ vault,+ configPath,+ repoRoot: null,+ requestIdentity: { agent: "codex", via: "token" as const },+ };+}++async function callApply(args: Record<string, unknown>): Promise<Record<string, unknown>> {+ const tool = HYGIENE_TOOLS.find((t) => t.name === "brain_hygiene")!;+ // The document-dedup detector is opt-in: every scan that must see the+ // retire finding names it, the apply's own re-scan included.+ return tool.handler(identityCtx() as any, {+ mode: "apply",+ detectors: ["document-dedup"],+ ...args,+ }) as Record<string, unknown>;+}++describe("brain_hygiene apply deny consult", () => {+ test("a document rule naming the request identity refuses the retire route", async () => {+ writeFileSync(+ join(vault, "Brain", "_permissions.yaml"),+ "version: 1\ndefault_action: allow\nentries:\n - id: freeze\n agent: codex\n" ++ " action: ingest\n verdict: deny\n",+ "utf8",+ );+ const scan = (await toolScan()) as {+ findings: Array<{ id: string; proposed_action: string }>;+ };+ const finding = scan.findings.find((f) => f.proposed_action === "retire");+ if (finding === undefined) throw new Error("fixture produced no retire finding");++ const result = await callApply({ ids: [finding.id] });++ expect(result.applied).toEqual([]);+ const errors = result.errors as Array<{ finding_id: string; message: string }>;+ expect(errors).toHaveLength(1);+ expect(errors[0]!.finding_id).toBe(finding.id);+ expect(errors[0]!.message).toContain("write-refused");+ expect(errors[0]!.message).toContain('"codex"');+ // Nothing was removed: the consult runs before the retirement.+ expect(existsSync(join(vault, "Brain", "sources", "src-a.md"))).toBe(true);+ expect(existsSync(join(vault, "Brain", "sources", "src-b.md"))).toBe(true);+ });++ test("a rule naming another agent leaves the apply retiring for the same identity", async () => {+ writeFileSync(+ join(vault, "Brain", "_permissions.yaml"),+ "version: 1\ndefault_action: allow\nentries:\n - id: freeze-gemini\n agent: gemini\n" ++ " action: ingest\n verdict: deny\n",+ "utf8",+ );+ const scan = (await toolScan()) as {+ findings: Array<{ id: string; proposed_action: string }>;+ };+ const finding = scan.findings.find((f) => f.proposed_action === "retire");+ if (finding === undefined) throw new Error("fixture produced no retire finding");++ const result = await callApply({ ids: [finding.id] });++ expect(result.errors).toEqual([]);+ expect(result.applied).toHaveLength(1);+ expect(existsSync(join(vault, "Brain", "sources", "src-a.md"))).toBe(false);+ });+});++/** A scan through the same identity, to read the retire finding id off. */+async function toolScan(): Promise<unknown> {+ const tool = HYGIENE_TOOLS.find((t) => t.name === "brain_hygiene")!;+ return tool.handler(identityCtx() as any, {+ mode: "scan",+ detectors: ["document-dedup"],+ });+}diff --git a/tests/mcp/ingest-capacity-refusal.test.ts b/tests/mcp/ingest-capacity-refusal.test.tsnew file mode 100644index 00000000..28860821--- /dev/null+++ b/tests/mcp/ingest-capacity-refusal.test.ts@@ -0,0 +1,259 @@+/**+ * `brain_ingest_source` capacity admission (storage lifecycle, Lane 2):+ * at the critical storage tier the ingest refuses BEFORE any write with+ * the `capacity_exhausted` envelope naming the remedy; an unbounded or+ * sub-critical vault ingests unchanged; and when no cap is configured+ * the check is skipped entirely - no measurement, no sample, no cost.+ *+ * The refusal sits at the tool's validate phase, ahead of `ingestSource`,+ * so the commit phase is never entered and never rolled back: the vault+ * is byte-identical after a refusal. Reads and the read-only batch plan+ * are never gated - reclamation must stay possible while producers are+ * refused.+ *+ * Reach honesty: the refusal itself is a write gate, so it fires at every+ * reach - but the whole-vault figures beside it are the same local-reach+ * facts the `brain_storage` report refuses to take below local reach.+ * A remote caller is told the wall exists and where the remedy lives,+ * never handed the census that describes the vault.+ */++import { afterEach, describe, expect, test } from "bun:test";+import {+ appendFileSync,+ mkdirSync,+ mkdtempSync,+ readdirSync,+ rmSync,+ statSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { brainConfigPath } from "../../src/core/brain/paths.ts";+import { STORAGE_ERROR_CODES } from "../../src/core/brain/storage/capacity-error.ts";+import { STORAGE_USAGE_METRIC_SURFACE } from "../../src/core/brain/storage/forecast.ts";+import { INVALID_PARAMS, MCPError } from "../../src/mcp/protocol.ts";+import { isToolErrorCode } from "../../src/mcp/tool-error-codes.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";+import { INGEST_TOOLS } from "../../src/mcp/brain/ingest-tools.ts";+import { JSONRPC_VERSION, MCPServer } from "../../src/mcp/index.ts";+import type { ServerContext } from "../../src/mcp/tool-contract.ts";+import { listMetrics } from "../../src/core/brain/metrics.ts";+import { brainPageTexts } from "../helpers/brain-pages.ts";+import { readRpcErrorCode } from "../helpers/tool-error-envelope.ts";++const SUMMARY = "Release notes of the capacity widget.";++/**+ * A cap every real vault clears many times over: whatever the bootstrapped+ * fixture holds is far past 95 percent of this, and usage past 100 percent+ * of cap is still `critical` - the top measured tier.+ */+const CRITICAL_CAP_BYTES = 1000;+/** A cap no test fixture approaches, so the tier stays below the warn band. */+const ROOMY_CAP_BYTES = 1_000_000_000_000;+/** The metrics surface a usage sample rides; the admission check appends none. */+const STORAGE_USAGE_SURFACE = STORAGE_USAGE_METRIC_SURFACE;++const bases: string[] = [];++afterEach(() => {+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });+});++interface Fixture {+ readonly vault: string;+ readonly configPath: string;+ readonly server: MCPServer;+}++function fixture(+ capBytes: number | undefined,+ reach: TransportReach = TRANSPORT_REACH.local,+): Fixture {+ const base = mkdtempSync(join(tmpdir(), "o2b-ingest-cap-"));+ bases.push(base);+ const vault = join(base, "vault");+ mkdirSync(join(vault, "notes"), { recursive: true });+ writeFileSync(join(vault, "notes", "source.md"), "# Widget source\n\nBody text.\n");+ const configPath = join(base, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);+ bootstrapBrain(vault, { configPath });+ if (capBytes !== undefined) {+ appendFileSync(brainConfigPath(vault), `\nstorage:\n cap_bytes: ${capBytes}\n`);+ }+ const server = new MCPServer({ vault, configPath }, { reach });+ return { vault, configPath, server };+}++/** The ingest arguments every call in this suite sends. */+function ingestArgs(): Record<string, unknown> {+ return {+ source_path: "notes/source.md",+ summary: SUMMARY,+ entities: [{ category: "concept", name: "Capacity widget" }],+ };+}++/** The three data keys that carry whole-vault figures to a caller. */+const FIGURE_KEYS = ["measured_bytes", "cap_bytes", "percent_used"] as const;++interface RpcResponse {+ readonly error?: { code: number; message: string; data?: Record<string, unknown> };+ readonly result?: {+ structuredContent?: Record<string, unknown>;+ content?: ReadonlyArray<{ text?: string }>;+ };+}++async function ingest(f: Fixture): Promise<RpcResponse> {+ const raw = await f.server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 7,+ method: "tools/call",+ params: {+ name: "brain_ingest_source",+ arguments: {+ source_path: "notes/source.md",+ summary: SUMMARY,+ entities: [{ category: "concept", name: "Capacity widget" }],+ },+ },+ });+ return raw as RpcResponse;+}++/** The tool's result object, from structured content or the text block. */+function payload(res: RpcResponse): Record<string, unknown> {+ const structured = res.result?.structuredContent;+ if (structured !== undefined) return structured;+ return JSON.parse(res.result?.content?.[0]?.text ?? "{}") as Record<string, unknown>;+}++/** Total bytes of every file under `root`, recursively - a write changes this. */+function vaultBytes(root: string): number {+ let total = 0;+ for (const entry of readdirSync(root, { recursive: true, withFileTypes: true })) {+ if (entry.isFile()) total += statSync(join(entry.parentPath, entry.name)).size;+ }+ return total;+}++describe("brain_ingest_source capacity admission", () => {+ test("at the critical tier the ingest refuses by name and writes nothing", async () => {+ const f = fixture(CRITICAL_CAP_BYTES);+ const before = vaultBytes(f.vault);+ const res = await ingest(f);+ const message = res.error?.message ?? "";+ expect(readRpcErrorCode(res)).toBe("capacity_exhausted");+ expect(STORAGE_ERROR_CODES).toContain("capacity_exhausted");+ // Registered on the wire registry (Lane 1's exhaustiveMembers row),+ // so the refusal answers with a token a client can branch on.+ expect(isToolErrorCode("capacity_exhausted")).toBe(true);+ expect(message).toContain("critical");+ expect(message).toContain("brain_ingest_source");+ // The remedy path is named, not inferred: reclaim by compaction or+ // retirement, or raise the cap.+ expect(message).toContain("cap_bytes");++ // Zero writes: the refusal happened in the validate phase, so no page,+ // ledger or metric moved anywhere in the vault.+ expect(vaultBytes(f.vault)).toBe(before);+ expect(brainPageTexts(f.vault).some((text) => text.includes(SUMMARY))).toBe(false);+ expect(listMetrics(f.vault, { surface: STORAGE_USAGE_SURFACE })).toEqual([]);+ });++ test("an unbounded vault ingests unchanged", async () => {+ const f = fixture(undefined);+ const res = await ingest(f);+ expect(res.error).toBeUndefined();+ expect(String(payload(res)["summary_path"] ?? "").length).toBeGreaterThan(0);+ expect(brainPageTexts(f.vault).some((text) => text.includes(SUMMARY))).toBe(true);+ });++ test("a sub-critical vault ingests unchanged", async () => {+ const f = fixture(ROOMY_CAP_BYTES);+ const res = await ingest(f);+ expect(res.error).toBeUndefined();+ expect(brainPageTexts(f.vault).some((text) => text.includes(SUMMARY))).toBe(true);+ });++ test("no cap configured means no measurement cost", async () => {+ const f = fixture(undefined);+ await ingest(f);+ // The unbounded vault ingested (the contract above), and the admission+ // check left no usage sample behind: with no cap in force the check+ // never measured.+ expect(listMetrics(f.vault, { surface: STORAGE_USAGE_SURFACE })).toEqual([]);+ });+});++describe("the admission refusal at the caller's reach", () => {+ test("a local refusal carries the measured reading beside the code", async () => {+ const f = fixture(CRITICAL_CAP_BYTES, TRANSPORT_REACH.local);+ const res = await ingest(f);+ expect(readRpcErrorCode(res)).toBe("capacity_exhausted");+ const data = res.error?.data ?? {};+ expect(data["tier"]).toBe("critical");+ expect(typeof data["measured_bytes"]).toBe("number");+ expect(data["cap_bytes"]).toBe(CRITICAL_CAP_BYTES);+ expect(typeof data["percent_used"]).toBe("number");+ });++ test("a remote refusal names the wall and carries no vault figures", async () => {+ const f = fixture(CRITICAL_CAP_BYTES, TRANSPORT_REACH.remote);+ const before = vaultBytes(f.vault);+ const res = await ingest(f);+ // The gate is a write gate, not a secret: the refusal itself fires at+ // every reach, by the same code.+ expect(readRpcErrorCode(res)).toBe("capacity_exhausted");+ const data = res.error?.data ?? {};+ expect(data["tier"]).toBe("critical");+ expect(data["remedy"]).toContain("cap_bytes");+ // The local remedy is still in hand: the registered next step for a+ // critical vault, not a figure a remote caller cannot act on.+ expect(data["next_command"]).toBe("o2b brain storage");+ // The census is a whole-vault figure (brain_storage's reach model):+ // below local reach none of it rides.+ for (const key of FIGURE_KEYS) {+ expect(Object.hasOwn(data, key)).toBe(false);+ }+ // Still a validate-phase refusal: the vault did not move.+ expect(vaultBytes(f.vault)).toBe(before);+ expect(listMetrics(f.vault, { surface: STORAGE_USAGE_SURFACE })).toEqual([]);+ });++ test("a context that established nothing is refused without figures", async () => {+ const f = fixture(CRITICAL_CAP_BYTES);+ // No `reach` field on a hand-built context: nobody established who was+ // asking, so the tool answers at the narrowest reach (fail closed).+ const ctx: ServerContext = {+ vault: f.vault,+ configPath: f.configPath,+ repoRoot: null,+ };+ const handler = INGEST_TOOLS.find((tool) => tool.name === "brain_ingest_source")!.handler;+ let refused: MCPError | undefined;+ try {+ await handler(ctx, ingestArgs());+ } catch (err) {+ refused = err instanceof MCPError ? err : undefined;+ }+ expect(refused).toBeDefined();+ expect(refused!.code).toBe(INVALID_PARAMS);+ expect(String(refused!.message)).toContain("capacity_exhausted");+ // The admission envelope is always built as a record (the refusal+ // mapper spreads the leaf's fields into one), so the keys below are+ // read off that record.+ const data = (refused!.data ?? {}) as Record<string, unknown>;+ expect(data["code"]).toBe("capacity_exhausted");+ for (const key of FIGURE_KEYS) {+ expect(Object.hasOwn(data, key)).toBe(false);+ }+ expect(listMetrics(f.vault, { surface: STORAGE_USAGE_SURFACE })).toEqual([]);+ });+});diff --git a/tests/mcp/maintenance-status-reach.test.ts b/tests/mcp/maintenance-status-reach.test.tsnew file mode 100644index 00000000..76d5dc94--- /dev/null+++ b/tests/mcp/maintenance-status-reach.test.ts@@ -0,0 +1,110 @@+/**+ * `brain_maintenance status` at the caller's reach (storage-lifecycle+ * self-review).+ *+ * A `skipped:capacity` journal row carries the reading behind the skip:+ * the whole-vault measured bytes, the configured cap and the percent of+ * one against the other. Those are the same local-reach facts the+ * storage report refuses to take below local reach and the ingest+ * admission refusal strips from its envelope - totals that move when a+ * page the caller may not read appears or goes, handed to a remote+ * caller as a growth oracle. The journal keeps them (it is the+ * operator's own record); the status answer renders them at local reach+ * only. The tier word and the unmeasurable reason stay at every reach:+ * they name a verdict, never a size.+ *+ * Claims pinned here:+ *+ * 1. Below local reach the status rows carry no capacity reading.+ * 2. At local reach the same rows carry it, so the operator's own+ * surface loses nothing.+ * 3. A hand-built context that established nothing answers at the+ * narrowest reach (fail closed): no reading.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { resetVaultIdentityPins } from "../../src/core/brain/vault-identity.ts";+import { MAINTENANCE_VERDICT, appendJournal } from "../../src/core/brain/maintenance/journal.ts";+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { ADMIN_TOOLS } from "../../src/mcp/brain/admin-tools.ts";+import type { ServerContext } from "../../src/mcp/tool-contract.ts";++const maintenance = ADMIN_TOOLS.find((tool) => tool.name === "brain_maintenance")!.handler;++let vault: string;+let configHome: string;+let ctx: ServerContext;++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-maint-status-reach-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-maint-status-reach-cfg-"));+ atomicWriteFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+ ctx = { vault, configPath: join(configHome, "config.yaml"), repoRoot: null };+ appendJournal(vault, {+ ts: "2026-10-15T12:00:00Z",+ holder: "worker-a",+ verdict: MAINTENANCE_VERDICT.skippedCapacity,+ capacity_tier: "critical",+ capacity_percent: 96.4,+ capacity_measured_bytes: 482_113,+ capacity_cap_bytes: 500_000,+ });+});++afterEach(() => {+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++interface StatusRow {+ readonly verdict: string;+ readonly capacity_tier?: string;+ readonly capacity_percent?: number;+ readonly capacity_measured_bytes?: number;+ readonly capacity_cap_bytes?: number;+}++async function statusRows(reach: TransportReach): Promise<ReadonlyArray<StatusRow>> {+ const res = (await maintenance({ ...ctx, reach }, { operation: "status" })) as Record<+ string,+ unknown+ >;+ return res["journal"] as ReadonlyArray<StatusRow>;+}++describe("brain_maintenance status at the caller's reach", () => {+ test("below local reach the rows carry the verdict, never the reading", async () => {+ const rows = await statusRows(TRANSPORT_REACH.remote);+ const row = rows.find((e) => e.verdict === MAINTENANCE_VERDICT.skippedCapacity)!;+ expect(row).toBeDefined();+ expect(row.capacity_tier).toBe("critical");+ expect(row.capacity_percent).toBeUndefined();+ expect(row.capacity_measured_bytes).toBeUndefined();+ expect(row.capacity_cap_bytes).toBeUndefined();+ });++ test("at local reach the rows still carry the reading", async () => {+ const rows = await statusRows(TRANSPORT_REACH.local);+ const row = rows.find((e) => e.verdict === MAINTENANCE_VERDICT.skippedCapacity)!;+ expect(row.capacity_measured_bytes).toBe(482_113);+ expect(row.capacity_cap_bytes).toBe(500_000);+ expect(row.capacity_percent).toBe(96.4);+ });++ test("a context that established nothing answers at the narrowest reach", async () => {+ const rows = await statusRows(TRANSPORT_REACH.remote);+ const rendered = JSON.stringify(rows);+ expect(rendered).not.toContain("capacity_measured_bytes");+ expect(rendered).not.toContain("capacity_cap_bytes");+ expect(rendered).not.toContain("capacity_percent");+ });+});diff --git a/tests/mcp/mcp.test.ts b/tests/mcp/mcp.test.tsindex 61d815b5..71384702 100644--- a/tests/mcp/mcp.test.ts+++ b/tests/mcp/mcp.test.ts@@ -252,6 +252,10 @@ describe("tool listing", () => {"brain_context_receipts","brain_event_trace","brain_recall_telemetry",+ // Receipt compaction over the receipt ledgers (storage-lifecycle).+ "brain_receipts",+ // The vault storage report (storage-lifecycle).+ "brain_storage",// Observed-use verdict feeding recall ranking (Retrieval & Ranking Quality)."brain_observed_use","brain_knowledge_gaps",diff --git a/tests/mcp/receipts-reach.test.ts b/tests/mcp/receipts-reach.test.tsnew file mode 100644index 00000000..f1179a7d--- /dev/null+++ b/tests/mcp/receipts-reach.test.ts@@ -0,0 +1,133 @@+/**+ * `brain_receipts` compact at the caller's reach (storage-lifecycle+ * wave review).+ *+ * Compact is the destructive half of the receipts surface: it rewrites+ * named ledger shards in place and reports whole-ledger byte totals -+ * the same figure class the storage report refuses to take below local+ * reach (`brain_storage`). Below local reach the fold is therefore not+ * planned at all: a named refusal, no shard plan, no byte totals, and+ * no write lane entered. Scan stays readable at any reach - a read-only+ * inventory withholds nothing a caller could not ask for another way.+ *+ * Claims pinned here:+ *+ * 1. A remote caller gets the named reach refusal, never a plan.+ * 2. The refusal carries no shard plan and no byte figures.+ * 3. A hand-built context that established nothing answers at the+ * narrowest reach (fail closed): refused, not run.+ * 4. At local reach the same tool previews the fold.+ * 5. Scan stays readable below local reach.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { continuityLogDir } from "../../src/core/brain/continuity/store.ts";+import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { RECEIPT_COMPACTION_OPERATION } from "../../src/core/brain/receipts/types.ts";+import { resetVaultIdentityPins } from "../../src/core/brain/vault-identity.ts";+import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";+import { RECEIPT_COMPACT_REACH_REASON, RECEIPT_TOOLS } from "../../src/mcp/brain/receipt-tools.ts";+import type { ServerContext } from "../../src/mcp/tool-contract.ts";++const receipts = RECEIPT_TOOLS.find((tool) => tool.name === "brain_receipts")!.handler;++const SHARD = "continuity:2026-08.jsonl";++let vault: string;+let configHome: string;+let ctx: ServerContext;++function continuityLine(kind: string, createdAt: string): string {+ const id = `ctn_${createdAt.replace(/[^0-9]/g, "").slice(0, 14)}_${kind}`;+ return `${JSON.stringify({+ schema: "o2b.continuity.v1",+ id,+ kind,+ createdAt,+ sourceRefs: [],+ payload: {},+ private: false,+ redacted: false,+ })}\n`;+}++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-receipts-reach-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-receipts-reach-cfg-"));+ writeFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+ mkdirSync(continuityLogDir(vault), { recursive: true });+ writeFileSync(+ join(continuityLogDir(vault), "2026-08.jsonl"),+ [continuityLine("recall_telemetry", "2026-08-10T12:00:00Z")].join(""),+ "utf8",+ );+ ctx = { vault, configPath: join(configHome, "config.yaml"), repoRoot: null };+});++afterEach(() => {+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++describe("brain_receipts compact at the caller's reach", () => {+ test("below local reach the fold is not planned and the refusal is named", async () => {+ const res = (await receipts(+ { ...ctx, reach: TRANSPORT_REACH.remote },+ { mode: "compact", shards: [SHARD] },+ )) as Record<string, unknown>;+ expect(res["operation"]).toBe(RECEIPT_COMPACTION_OPERATION);+ expect(res["measured"]).toBe(false);+ expect(res["reason"]).toBe(RECEIPT_COMPACT_REACH_REASON);+ expect(String(res["detail"])).toContain("local reach");+ });++ test("the refusal carries no shard plan and no byte figures", async () => {+ const res = (await receipts(+ { ...ctx, reach: TRANSPORT_REACH.remote },+ { mode: "compact", shards: [SHARD] },+ )) as Record<string, unknown>;+ const rendered = JSON.stringify(res);+ expect(res["shards"]).toBeUndefined();+ expect(res["totals"]).toBeUndefined();+ expect(res["groups"]).toBeUndefined();+ expect(rendered).not.toContain("bytes");+ expect(rendered).not.toContain(vault);+ });++ test("a context that established nothing answers at the narrowest reach", async () => {+ // No `reach` field: a hand-built context proved nothing about who is+ // asking, so the destructive mode must not treat it as local.+ const res = (await receipts(ctx, { mode: "compact", shards: [SHARD] })) as Record<+ string,+ unknown+ >;+ expect(res["measured"]).toBe(false);+ expect(res["reason"]).toBe(RECEIPT_COMPACT_REACH_REASON);+ });++ test("at local reach the same tool previews the fold", async () => {+ const res = (await receipts(+ { ...ctx, reach: TRANSPORT_REACH.local },+ { mode: "compact", shards: [SHARD] },+ )) as Record<string, unknown>;+ expect(res["dry_run"]).toBe(true);+ expect(res["totals"]).toBeDefined();+ });++ test("scan stays readable below local reach", async () => {+ const res = (await receipts({ ...ctx, reach: TRANSPORT_REACH.remote }, {})) as Record<+ string,+ unknown+ >;+ expect(res["operation"]).toBe(RECEIPT_COMPACTION_OPERATION);+ expect(res["totals"]).toBeDefined();+ expect((res["shards"] as ReadonlyArray<unknown>).length).toBe(1);+ });+});diff --git a/tests/mcp/storage-capacity-errors.test.ts b/tests/mcp/storage-capacity-errors.test.tsnew file mode 100644index 00000000..371a1c31--- /dev/null+++ b/tests/mcp/storage-capacity-errors.test.ts@@ -0,0 +1,68 @@+/**+ * The storage capacity error vocabulary at the MCP boundary.+ *+ * Claims pinned here:+ *+ * 1. `STORAGE_ERROR_CODES` is the closed pair (`capacity_unmeasurable`,+ * `capacity_exhausted`), registered in `TOOL_ERROR_CODES` through the+ * exhaustive-members check, so a code added to the leaf cannot fail+ * to reach the wire.+ * 2. `StorageCapacityError` maps BY NAME through `codeForError`: the+ * code the leaf chose is the code the wire carries, for both members.+ * 3. The error envelope carries only the schema tag and the code - no+ * message field exists for it - so no vault path can reach a remote+ * caller through the classified path.+ */++import { describe, expect, test } from "bun:test";++import {+ STORAGE_ERROR_CODES,+ StorageCapacityError,+} from "../../src/core/brain/storage/capacity-error.ts";+import {+ TOOL_ERROR_CODES,+ codeForError,+ isToolErrorCode,+ toolErrorMeta,+} from "../../src/mcp/tool-error-codes.ts";++const VAULT_PATH = "/tmp/o2b-some-vault/Brain/metrics";++describe("storage capacity error codes", () => {+ test("the leaf vocabulary is the closed pair and is registered", () => {+ expect(STORAGE_ERROR_CODES).toEqual(["capacity_unmeasurable", "capacity_exhausted"]);+ for (const code of STORAGE_ERROR_CODES) {+ expect(isToolErrorCode(code)).toBe(true);+ expect(TOOL_ERROR_CODES).toContain(code);+ }+ });++ test("the error maps by name through the classifier", () => {+ for (const code of STORAGE_ERROR_CODES) {+ const err = new StorageCapacityError(+ code,+ `storage capacity: the reading failed for ${VAULT_PATH}`,+ );+ expect(codeForError(err)).toBe(code);+ }+ });++ test("the classifier walks a cause chain to the typed error", () => {+ const typed = new StorageCapacityError(+ "capacity_exhausted",+ `storage capacity: vault ${VAULT_PATH} is past the critical threshold`,+ );+ const wrapped = new Error("capacity gate refused the write", { cause: typed });+ expect(codeForError(wrapped)).toBe("capacity_exhausted");+ });++ test("the wire envelope carries a code and schema only, never a message", () => {+ for (const code of STORAGE_ERROR_CODES) {+ const meta = toolErrorMeta(code);+ expect(Object.keys(meta).toSorted()).toEqual(["code", "schema"]);+ expect(meta.code).toBe(code);+ expect(JSON.stringify(meta)).not.toContain(VAULT_PATH);+ }+ });+});diff --git a/tests/mcp/storage-reach.test.ts b/tests/mcp/storage-reach.test.tsnew file mode 100644index 00000000..9f0dbce2--- /dev/null+++ b/tests/mcp/storage-reach.test.ts@@ -0,0 +1,160 @@+/**+ * `brain_storage` at the caller's reach (storage-lifecycle, Lane 1+ * task 1.7).+ *+ * The usage census is a whole-vault byte figure: it counts every page,+ * withheld pages included, so its totals move when a page the caller may+ * not read appears or goes, and no filtered form of it exists. Below+ * local reach the number is therefore not taken at all - the+ * `link_integrity` model (`brain_hygiene`): the report answers+ * `measured: false` with the named reason, and no usage sample is+ * recorded, because the sample would require taking the very census the+ * reach refusal withholds.+ *+ * Claims pinned here:+ *+ * 1. A remote caller gets the named reach refusal, never bytes.+ * 2. A remote call records no usage sample.+ * 3. A hand-built context that established nothing answers at the+ * narrowest reach (fail closed): refused, not measured.+ * 4. At local reach the same tool measures.+ * 5. The doctor's storage-capacity check answers by the same rule+ * through `brain_doctor`: below local reach the fullness is+ * uncertain by name and no tier issue - either would carry the+ * census in its message - is emitted; at local reach it measures.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { resetVaultIdentityPins } from "../../src/core/brain/vault-identity.ts";+import {+ STORAGE_CAPACITY_CRITICAL_CODE,+ STORAGE_CAPACITY_UNMEASURED_CODE,+ STORAGE_CAPACITY_WARN_CODE,+} from "../../src/core/brain/doctor/storage-capacity.ts";+import { listMetrics, type MetricRecord } from "../../src/core/brain/metrics.ts";+import { STORAGE_USAGE_METRIC_SURFACE } from "../../src/core/brain/storage/forecast.ts";+import { brainConfigPath } from "../../src/core/brain/paths.ts";+import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { HEALTH_TOOLS } from "../../src/mcp/brain/health-tools.ts";+import { STORAGE_REACH_REASON, STORAGE_TOOLS } from "../../src/mcp/brain/storage-tools.ts";+import type { ServerContext } from "../../src/mcp/tool-contract.ts";++const storage = STORAGE_TOOLS.find((tool) => tool.name === "brain_storage")!.handler;+const doctor = HEALTH_TOOLS.find((tool) => tool.name === "brain_doctor")!.handler;++let vault: string;+let configHome: string;+let ctx: ServerContext;++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-storage-reach-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-storage-reach-cfg-"));+ atomicWriteFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+ ctx = { vault, configPath: join(configHome, "config.yaml"), repoRoot: null };+});++afterEach(() => {+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++function usageSamples(): MetricRecord[] {+ return listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE });+}++describe("brain_storage at the caller's reach", () => {+ test("below local reach the census is not taken and the refusal is named", async () => {+ const res = (await storage({ ...ctx, reach: TRANSPORT_REACH.remote }, {})) as Record<+ string,+ unknown+ >;+ expect(res["measured"]).toBe(false);+ expect(res["reason"]).toBe(STORAGE_REACH_REASON);+ expect(String(res["detail"])).toContain("local reach");+ });++ test("the refusal carries no vault bytes and no vault path", async () => {+ const res = (await storage({ ...ctx, reach: TRANSPORT_REACH.remote }, {})) as Record<+ string,+ unknown+ >;+ const rendered = JSON.stringify(res);+ expect(rendered).not.toContain("total_bytes");+ expect(rendered).not.toContain(vault);+ expect(res["usage"]).toBeUndefined();+ expect(res["forecast"]).toBeUndefined();+ expect(res["sample"]).toBeUndefined();+ });++ test("a refused call records no usage sample", async () => {+ await storage({ ...ctx, reach: TRANSPORT_REACH.remote }, {});+ expect(usageSamples()).toHaveLength(0);+ });++ test("a context that established nothing answers at the narrowest reach", async () => {+ // No `reach` field: a hand-built context proved nothing about who is+ // asking, so the tool must not treat it as local.+ const res = (await storage(ctx, {})) as Record<string, unknown>;+ expect(res["measured"]).toBe(false);+ expect(res["reason"]).toBe(STORAGE_REACH_REASON);+ expect(usageSamples()).toHaveLength(0);+ });++ test("at local reach the same tool measures and samples", async () => {+ const res = (await storage({ ...ctx, reach: TRANSPORT_REACH.local }, {})) as Record<+ string,+ unknown+ >;+ expect(res["measured"]).toBe(true);+ expect(usageSamples()).toHaveLength(1);+ });+});++describe("the doctor's storage-capacity check at the caller's reach", () => {+ function configureCappedVaultPastCritical(): void {+ writeFileSync(+ brainConfigPath(vault),+ "schema_version: 1\n\nstorage:\n cap_bytes: 10000\n",+ "utf8",+ );+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", "AA-page.md"), "x".repeat(20_000), "utf8");+ }++ test("below local reach the fullness is uncertain by name, never measured", async () => {+ configureCappedVaultPastCritical();+ const res = (await doctor({ ...ctx, reach: TRANSPORT_REACH.remote }, {})) as Record<+ string,+ unknown+ >;+ const rendered = JSON.stringify(res);+ // Neither tier issue is emitted: each would carry the whole-vault+ // census in its message prose.+ expect(rendered).not.toContain(STORAGE_CAPACITY_WARN_CODE);+ expect(rendered).not.toContain(STORAGE_CAPACITY_CRITICAL_CODE);+ const unmeasured = ((res["uncertain"] as ReadonlyArray<Record<string, unknown>>) ?? []).filter(+ (entry) => entry["code"] === STORAGE_CAPACITY_UNMEASURED_CODE,+ );+ expect(unmeasured).toHaveLength(1);+ expect(String(unmeasured[0]!["message"])).toContain("reach");+ });++ test("at local reach the same tool reports the tier it measures", async () => {+ configureCappedVaultPastCritical();+ const res = (await doctor({ ...ctx, reach: TRANSPORT_REACH.local }, {})) as Record<+ string,+ unknown+ >;+ const rendered = JSON.stringify(res);+ expect(rendered).toContain(STORAGE_CAPACITY_CRITICAL_CODE);+ });+});diff --git a/tests/mcp/tool-error-codes.test.ts b/tests/mcp/tool-error-codes.test.tsindex 353920ab..2c802aca 100644--- a/tests/mcp/tool-error-codes.test.ts+++ b/tests/mcp/tool-error-codes.test.ts@@ -32,6 +32,7 @@ import { BrainConfigError } from "../../src/core/brain/policy/errors.ts";import { SHAPE_VIOLATION_CODES } from "../../src/core/brain/response-shape.ts";import { SEMANTIC_VIOLATION_CODES } from "../../src/core/brain/response-checks.ts";import { SafeguardAbortError, SafeguardTimeoutError } from "../../src/core/brain/safeguard.ts";+import { STORAGE_ERROR_CODES } from "../../src/core/brain/storage/capacity-error.ts";import { WriteBatchError } from "../../src/core/brain/write-batch.ts";import { ConfigReadError } from "../../src/core/config.ts";import { SEARCH_ERROR_CODES, SearchError } from "../../src/core/search/search-error.ts";@@ -178,6 +179,7 @@ describe("TOOL_ERROR_CODES", () => {const expected = new Set<string>([...NEW_TOKENS,...SEARCH_ERROR_CODES,+ ...STORAGE_ERROR_CODES,...Object.values(SHAPE_VIOLATION_CODES),...Object.values(SEMANTIC_VIOLATION_CODES),VAULT_FROZEN_REFUSAL,diff --git a/uv.lock b/uv.lockindex a92ff60a..3ee3f6aa 100644--- a/uv.lock+++ b/uv.lock@@ -4,5 +4,5 @@ requires-python = ">=3.11"[[package]]name = "open-second-brain"-version = "1.84.1"+version = "1.85.0"source = { editable = "." }