Local-first 🧠memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: session-mining economy - mining tombstones, a shadow trust-gate mode and a live-counts console (v1.86.0) (#258)
Serg Eroshee0d116560eed
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex a2d2a897..416302b7 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.85.0",+ "version": "1.86.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex e82d8376..3d8941ab 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.85.0",+ "version": "1.86.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5485983a..bb129a28 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,9 +5,33 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.86.0] - 2026-10-16++Open Second Brain 1.86.0 stops re-paying for sessions with nothing new to say and measures before it excludes: a session mine that ends with nothing written records a zero-yield tombstone that suppresses every later plan over the same transcript until the turn material changes, a distilled and undistilled census over the imported backlog feeds an ask-once distill envelope and an opt-in SessionStart nudge, the retrieval trust gate resolves an `off | shadow | enforce` mode where `shadow` measures every would-be refusal while excluding nothing, and the loopback explorer serves a live-counts console beside its graph. The release also repairs what the external review of v1.85.0 found in the shipped storage lifecycle: receipt compaction now folds only opt-in telemetry kinds, the capacity trend buckets its samples per day while the maintenance lane pauses a band before the wall, a retired document copy keeps its manifest marker, and the incremental extraction watermark stamps exactly the bytes the appended delta covered.++### Added++- **A zero-yield mining tombstone, with a one-call force re-mine.** `o2b brain extract-signals` and the `brain_extract_signals` tool now remember a barren mine: a commit that ends with nothing written and nothing staged records a tombstone for the session in a session mining ledger beside the import ledger (`<vault>/.open-second-brain/session-mining-ledger.json`, schema-versioned, read fail-closed on a corrupt file or an unknown schema version - a silent reset would unsuppress every barren session at once - written under the vault lock, and keyed on a SHA-256 of the exact turn material the plan embeds), and the next plan over that session at unchanged content returns the recorded breakdown instead of an envelope - `skipped { reason: mining_tombstone, mining_outcome { written, staged, deduped, durability_rejected } }` with `llm_step: null` and empty `turns_mined`, text mode naming the breakdown - so an unchanged transcript is never re-offered to the model. The invalidation is the content: a re-import that changes the session's turns changes the hash and re-opens the session with no clearing step. Because removing a downstream preference does not change the turns, `--force` (MCP `force`) bypasses the tombstone for exactly the one plan call it is given to - the remedy when the session should be re-proposed; a corrupt mining ledger refuses the plan by name even under force, and a commit that succeeds but cannot record its tombstone still exits 0, naming `mining_ledger_failure` on both surfaces.+- **A distilled and undistilled census, an ask-once distill envelope and an opt-in SessionStart nudge.** The census joins the import ledger to the mining record and classifies every imported session - `distilled` (mined with accepted yield on its current material), `known-nothing-new` (mined barren on it, the tombstone), or `undistilled` (never mined, or mined on material that has since changed) - joining on the portable session identity and reading vault-side only, so the honest count never opens or re-hashes a source log and stays affordable inside a hook's budget. `o2b brain import-session --propose-backlog` turns the undistilled rows into one ask-once envelope: the feedback-marker candidates the sessions' mined turns raise are deduplicated against the vault's shared dedup index before the ask - candidates the vault already holds are dropped and named, the rest collapse to one proposal per distinct hash across sessions with every contributing session listed, and a session whose every candidate was already known is reported rather than re-proposed. `backlog_nudge_enabled` (env `OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED`, default off) arms a SessionStart hook that reports the undistilled count as one `additionalContext` line, change-keyed rather than timed: injected when the count differs from the count injected last, an unchanged backlog never re-nagged, a drained backlog recorded without being announced; under its deadline the hook degrades to silence, never to a fabricated number, and flag off keeps the session preamble byte-identical.+- **A measure-only shadow mode on the retrieval trust gate.** The gate resolves a tri-state mode, `search_trust_gate_mode` (env `OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE`, `off | shadow | enforce`, default off): an explicit mode wins, with an invalid value refused naming the key actually in force, and absent the mode key the legacy `search_trust_gate_enabled` boolean maps `true` to `shadow` and `false` to `off`. `off` runs no gate at all and stays byte-identical; `shadow` runs the same deterministic controlled-vocabulary classification, excludes nothing, and records every would-be refusal with all of its structural reasons into the retrieval receipts per lane - the candidate pool and the relational arm never folded together - so an operator can read exactly what enforcement would drop before naming it; `enforce` is today's excluding behavior made explicit. The resolved mode rides the query-cache fingerprint so a mode change can never serve stale cached rows, the `--explain` envelope names it, and the human explain transcript now prints the refusal counts and rows beside the exclusions, so a shadow run no longer reads like a clean enforce run on the surface the operator defaults to.+- **An edge-level eligibility adjuster in the relational arm.** Each walked node and its arriving edge are classified from the same controlled-vocabulary frontmatter signals the page gate reads, plus the validity window at query time and the relation-polarity vocabulary: a `contradicts` arrival is the counted exclusion, `superseded_by` stays the annotated supersession backbone, and positive relations boost. Under `enforce` an ineligible arrival leaves the arm's contribution set; under `shadow` it is counted only. A seed is the caller's own entry point and is never refused for its own markers, the walk still expands through ineligible nodes - only the contribution set is gated at the arm boundary - and a refusal is a different act from the arm's identical-to-absent provenance convention: it is removed or kept-and-counted with named structural reason tokens on the arm lane of the same retrieval receipts, reach-filtered to pages the caller could read. The mode resolves from the same config the arm's other knobs answer to, so the arm and the page gate cannot disagree about the posture.+- **A live-counts console on the loopback explorer.** The explorer's live server gains two read-only routes beside the graph: `/counts.json` - the recall-telemetry summary, the most-recalled artifacts under a named cap, a written-per-day histogram over a trailing 30-day window bucketed from continuity record timestamps, and the latest report-snapshot date per surface, which reads as disabled while the feature is off rather than as empty data an operator could mistake for enabled-and-quiet - and `/status.json`, the operator snapshot. Both re-read the vault on every GET (the status route re-runs the dry-run dream each time, a bounded cost stated rather than cached away), a failed read answers a 500 naming the route and the error instead of an empty payload, and the page's right panel renders Counts and Vault status sections fetching both client-side, so a refresh is a fresh read; a static export states that the live routes are unreachable instead of firing fetches that can only fail.++### Fixed++- **Receipt compaction folds only opt-in telemetry kinds, never functional records.** The protection rule is inverted into a closed allowlist - `FOLDABLE_CONTINUITY_KINDS`, the pure telemetry kinds whose readers keep no per-record identity past the correlation window - with `KEPT_CONTINUITY_KINDS` naming every kind a reader reads back as data; folding is opt-in per kind, and a kind the build does not know fails closed to keep, the way a future build's records always did. v1.85.0's clip-key rule never protected `session_entity_mention` (its payload carries a `session_ref`), so a confirmed compact folded the records the session-recall seed, the hallways pass and the mining dedupe read back, the by-id `generation_report` lookups, the row-count-derived recent-turn sequence, and `receipt_compaction`'s own audit record - the next run folding the run before it. `recall_observed_use` and `negative_recall` sit on the kept list too, their readers named: one is the lifetime observed-reuse rate search ranking consumes, the other the answer-audit trail a corpus statement rests on. Beside the allowlist: repeated folds merge into the existing per-kind, per-month aggregate instead of stacking a second line; shard names are validated up front so a malformed `--shard` answers invalid params; scan offers and compact rewrites only this device's shard, a foreign shard refused by name on previews and confirmed runs alike - the fold's lock and compare-and-swap are local, so rewriting a peer's shard is exactly the divergent-copy conflict the per-device shard layout exists to prevent; and a text-mode `compact --apply` whose own audit record is lost in the rewrite reports the loss and exits 1.+- **The capacity trend buckets per day, and the maintenance lane pauses a band before the wall.** The forecast's slope is day-based: samples bucket to one point per UTC day (the day's last sample wins), a sample whose census was partial drops out of the fit entirely - an incomplete census understates the vault by exactly the surfaces it could not read, and the jump when they become readable again would otherwise fit as growth - and the fit needs three distinct days inside a trailing 30-day window before any `daysToCap` figure exists, so two calls a minute apart around one ordinary write no longer extrapolate the burst into a full vault, and a season-old slope (including a season-old reclamation) stops steering the line once the window slides past. The lane's capacity gate carries `storage.pause_percent` (template-documented, defaulting to the warn band, validated strictly below `critical_percent` - a pause at or past the refuse line could never pause anything): a measured percent at or past the line skips the pass with the `skipped:capacity` verdict while interactive ingest still works until the critical refusal. And the gate records its census as a `storage_usage` sample on every run where it evaluated, skip or proceed, so a vault where nobody runs the storage report still grows a trend the day-based tiers can fire on.+- **A retired document copy keeps its manifest marker, and a copy that cannot retire cleanly stays in place.** v1.85.0 dropped the retired copy's content-manifest entry as its resurrect guard, which re-classified the unchanged source as new and let the next batch plan quietly re-ingest it, undoing the retirement. The entry now stays, marked `retired_into` the canonical page (additive, the manifest schema stays at 3): an unchanged marked source answers a `retired` bucket and queues out onto `skipped_retired`, a real byte change re-admits it as modified, and a deliberate re-ingest re-records the entry without the marker. A copy whose inbound-link rewrite failed is refused before the removal - one locked citing note no longer leaves its `[[stale-copy]]` link pointing at a deleted page - and the apply detail carries the per-copy rewritten and failed link counts. A page elsewhere in `Brain/` carrying the same basename refuses the copy outright instead of retiring it, so no leftover `[[basename]]` is silently re-pointed to the namesake. The detector's bounds are named too: a candidate cap that bound the read set emits a review-only finding carrying the scanned and total counts instead of silently truncating, and the census that reads a retired-and-unchanged source reports it as found, never as a gap.+- **The extraction watermark stamps exactly the bytes the delta announced, and the ingest answer reports the appended range.** The v1.85.0 stamp always advanced `extracted_bytes` to the full file size while the tail cut ended at the last complete newline, so an unterminated trailing fragment fell out of every delta: the stamp claimed bytes no delta ever announced, and the next append's delta started past the fragment instead of stitching it in. On a proven append the stamp now lands at the delta's boundary-safe `toByte` and records the digest of exactly those leading bytes, so the prefix proof stays consistent with the watermark the delta series announced and the fragment rides the next delta; every other shape - first ingest, rewrite, shrink - stays a whole-source pass stamped at the full current file. `brain_ingest_source` now carries `extraction_delta` - the named kind beside the boundary-safe byte range - absent on every shape that is not a proven append; extraction itself stays whole-source, so the field is a report of what the append added, never a claim that only the tail was processed. Beside the boundary: a first ingest no longer walks its source for a delta that cannot exist, an append past the read ceiling is named instead of answered like a rewrite, and the watermark's stat-and-hash happens inside the manifest lock.+- **The censuses measure the vault they were pointed at, and every front door carries what its core collected.** The storage census counts the derived store's directory as a surface - the hook audit, the dead letters, the checkpoints, the session ledgers and the maintenance journal all accumulated outside the meter, and on a busy vault the forecast read ok while the disk filled - with the store files on their own surface so no byte lands in two, and the directory's non-regenerable contents (secret custody, lockfiles) counted as vault-content, the dangerous direction being to overstate the deletable side. It measures the store the configured location (`search_db_path` when set) points at rather than the default, so the lane gate, the ingest refusal and the doctor check agree about the same vault. `appendMetric` takes the shard's per-file lock the continuity appender holds, so a receipt fold's compare-and-swap rewrite can no longer move the shard out from under a concurrent append and silently drop the sample - including the `storage_usage` samples the gate and doctor read. `deleteBySource`'s per-file failed list now reaches both front doors - the `forget-source` verb's JSON payload and text report, and the MCP answer - with the confirmed run exiting 1 when a file stayed behind, matching the page-dedup verb's rule. The maintenance collectors' error observer is passed by all three production callers - digest, operator summary and doctor - so an absorbed scan failure renders as a named `collect_errors` row on every surface instead of a thin report that reads as healthy. And the live explorer routes refuse a non-loopback `Host` header by name.++### Changed++- **Docs:** `docs/cli-reference.md` states the five storage thresholds with the `pause_percent` band, the day-bucketed three-day-minimum trend and the whole-derived-store census, names the pause line in the maintenance-lane gate prose, states receipt compaction's opt-in-per-kind allowlist and this-device shard rule, documents the `--propose-backlog` mode with its conflicts, adds the `backlog_nudge_enabled` hook flag, and extends the `o2b brain explorer` entry with the console routes; `docs/how-it-works.md` states the pause line beside the five gates; the README's "What is new" names the release; the shipped 1.85.0 entry is reworded where this review proved the claim false at ship - the document-dedup bullet names the lexical layer as the wired one and the semantic layer as unreachable wiring, and the incremental-extraction bullet and opening paragraph report the appended byte range instead of claiming the new text is extracted without a full reprocess.+## [1.85.0] - 2026-10-15-Open Second Brain 1.85.0 gives a long-lived vault the means to watch its own storage and to keep its append-only ledgers in shape: a measured per-surface byte census with a days-to-capacity forecast that warns in graduated tiers, background producers that pause behind a fifth maintenance gate before the wall while interactive writes keep working until a hard, validated refusal at the critical tier, receipt ledgers that compact behind an explicit preview-bound apply, near-duplicate document copies that retire behind the hygiene review gate keeping the newest copy, bulk removals that report per-file failures instead of aborting, and large appended sources that yield their new text without reprocessing whole. No cap is configured by default: an uncapped vault reads unbounded, nothing pauses and nothing refuses.+Open Second Brain 1.85.0 gives a long-lived vault the means to watch its own storage and to keep its append-only ledgers in shape: a measured per-surface byte census with a days-to-capacity forecast that warns in graduated tiers, background producers that pause behind a fifth maintenance gate before the wall while interactive writes keep working until a hard, validated refusal at the critical tier, receipt ledgers that compact behind an explicit preview-bound apply, near-duplicate document copies that retire behind the hygiene review gate keeping the newest copy, bulk removals that report per-file failures instead of aborting, and large appended sources whose re-ingest proves a pure append and reports the exact appended byte range. No cap is configured by default: an uncapped vault reads unbounded, nothing pauses and nothing refuses.### Added@@ -16,8 +40,8 @@ Open Second Brain 1.85.0 gives a long-lived vault the means to watch its own sto- **The maintenance lane's fifth gate.** A configured cap arms the lane's capacity gate, evaluated last because it is the most expensive - a fresh byte census of the whole vault - and it decides from that fresh census at decision time, never the stored aggregate: a sample taken yesterday says nothing about a vault an agent wrote into this morning. Only a measured critical tier closes it - an unmeasurable store leaves the gate open with its own `capacity:unmeasurable` journal row, the same honesty the host-pressure gate practices - and a closed gate skips the pass with its own `skipped:capacity` verdict, the journal row carrying the tier and the reading behind it while the skipped run still appends its usage sample first, so the forecast series keeps accumulating exactly while producers pause. The lane's front doors carry the verdict and its remedy, and where the caller has no local reach the reading stays behind: `brain_maintenance status` renders the tier word and the unmeasurable reason without the three byte-and-percent figures. `--force` (MCP `force`) bypasses the four soft gates - window, busy, pressure and capacity - and every streak refusal; it never bypasses the lease.- **Interactive writes keep working until the wall.** The lane's gate pauses background producers; it does not pause a session. `brain_ingest_source` refuses with `capacity_exhausted` once a fresh census measures the vault at the critical tier, and the check sits in the validate phase before the write path opens, so a refused ingest leaves the vault byte-identical. The refusal names the tier and the remedy at every reach; the whole-vault figures ride only where the census itself may be read.- **Receipt compaction: `o2b brain receipts` and `brain_receipts`.** The continuity and metrics ledgers grow one record per event and never shrink; compaction gives them a drain. `scan` (the CLI's default mode) inventories both ledgers read-only, per shard: records, foldable, protected, unparseable, aggregates, bytes. `compact` folds expired records into per-kind, per-month aggregates inside the named shards - the mode and the `family:name` shard spelling are one shared vocabulary, so the verb and the tool cannot drift - and never runs without explicit shard ids from a scan. A compact previews unless `--apply` (MCP `apply`) is passed, takes a recovery-point snapshot before a confirmed run, rewrites each shard atomically, and honors a count guard: `--expect N` (MCP `expect`) asserts exactly N records will fold and aborts without writing on a mismatch, and `--strict` refuses a confirmed compact that carries no guard at all. Only records older than the age threshold (`--older-than-days` / `older_than_days`, default 30) and in no protected class fold; the protected classes never fold whatever their age - the `source_invalidation` audit trail and the two context-compaction kinds, whose vocabulary a storage operation must not collide with - so their lines stay verbatim, and one `receipt_compaction` audit record closes a confirmed run. Below local reach compact is refused by name; scan stays readable at any reach, and the `receipt-ledger-growth` doctor check reports ledger growth the same way.-- **Near-duplicate document retirement.** The opt-in `document-dedup` hygiene detector nominates near-duplicate copies of ingest summary pages - a lexical similarity of 0.8, or an embedding similarity of 0.97 when the semantic config can reach a provider - and selects the newest `created_at` as the canonical, a page without `created_at` being ineligible rather than guessed at by mtime. `apply` routes a selected retire finding through the retirement core: every inbound `[[stale-copy]]` wikilink inside the detector's scope retargets to the canonical spelling (a same-basename page elsewhere in `Brain/` keeps its links untouched, and the report names the skip), the copy is removed, its content-manifest entry is dropped - which is the resurrect guard, since with the entry gone the source can only return as a deliberate full re-ingest - and one `source_invalidation` audit record per copy names the canonical, the retired id, the dropped manifest key and the retained source-archive hash. The whole span sits behind the destructive-snapshot gate, and it fails soft per copy: one copy's failure is collected and named while the remaining copies still retire.-- **Incremental extraction of appended sources.** The content manifest's entries gain an optional extraction watermark (`extracted_bytes`, `extracted_at` - how many leading bytes the last extracting pass covered, strictly additive, the manifest schema stays at 3, and an entry without the fields reprocesses fully the next time it actually changes, so nothing forces an eager mass reprocess). When a re-ingest meets a watermarked record, the delta proof first shows the growth is a pure append - the SHA-256 of the leading watermark bytes must still equal the recorded digest, read through one non-blocking descriptor - and only then is the tail read through a UTF-8-safe from-offset read whose start snaps forward past a partial multi-byte sequence, bounded by a ceiling; the appended `{fromByte, toByte}` range rides the ingest result and the CLI extract preview, so the new text can be extracted without reprocessing the whole source. A rewritten source - even one that happens to grow - fails the prefix proof and answers no delta, and the full reprocess is the safe direction. The extraction contract stays at version 2.+- **Near-duplicate document retirement.** The opt-in `document-dedup` hygiene detector nominates near-duplicate copies of ingest summary pages - a lexical similarity of 0.8; the semantic embedding layer (the 0.97 threshold) ships exported and tested but reaches no scan caller yet, because the synchronous scan registry cannot call it, so no surface reaches it until that wiring lands - and selects the newest `created_at` as the canonical, a page without `created_at` being ineligible rather than guessed at by mtime. `apply` routes a selected retire finding through the retirement core: every inbound `[[stale-copy]]` wikilink inside the detector's scope retargets to the canonical spelling (a same-basename page elsewhere in `Brain/` keeps its links untouched, and the report names the skip), the copy is removed, its content-manifest entry is dropped - which is the resurrect guard, since with the entry gone the source can only return as a deliberate full re-ingest - and one `source_invalidation` audit record per copy names the canonical, the retired id, the dropped manifest key and the retained source-archive hash. The whole span sits behind the destructive-snapshot gate, and it fails soft per copy: one copy's failure is collected and named while the remaining copies still retire.+- **Incremental extraction of appended sources.** The content manifest's entries gain an optional extraction watermark (`extracted_bytes`, `extracted_at` - how many leading bytes the last extracting pass covered, strictly additive, the manifest schema stays at 3, and an entry without the fields reprocesses fully the next time it actually changes, so nothing forces an eager mass reprocess). When a re-ingest meets a watermarked record, the delta proof first shows the growth is a pure append - the SHA-256 of the leading watermark bytes must still equal the recorded digest, read through one non-blocking descriptor - and only then is the tail read through a UTF-8-safe from-offset read whose start snaps forward past a partial multi-byte sequence, bounded by a ceiling; the appended `{fromByte, toByte}` range rides the ingest result and the CLI extract preview as a report of what the append added - extraction itself stays whole-source, so the range is honest reporting, never a claim that only the tail was processed. A rewritten source - even one that happens to grow - fails the prefix proof and answers no delta, and the full reprocess is the safe direction. The extraction contract stays at version 2.- **Two doctor checks join the lifecycle.** `storage-capacity` reports fullness for a capped vault - tier, percent and trend where the census can be taken, and below local reach it answers in the uncertain stream that the reading was not taken, no figures - and skips its own gate when the vault already measures critical. `receipt-ledger-growth` reads both receipt ledgers and reports their foldable share before compaction becomes the remedy worth running.### Fixed@@ -8543,6 +8567,7 @@ Hermes / Claude Code / Codex / OpenClaw configurations do not change.- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.86.0]: https://github.com/itechmeat/open-second-brain/compare/v1.85.0...v1.86.0[1.85.0]: https://github.com/itechmeat/open-second-brain/compare/v1.84.1...v1.85.0[1.84.1]: https://github.com/itechmeat/open-second-brain/compare/v1.84.0...v1.84.1[1.84.0]: https://github.com/itechmeat/open-second-brain/compare/v1.83.0...v1.84.0diff --git a/README.md b/README.mdindex bc4d1243..3d09b6e3 100644--- a/README.md+++ b/README.md@@ -123,7 +123,7 @@ The full router with readiness criteria is [`install.md`](install.md); native Wi## What is new-1.85.0 gives a long-lived vault the means to watch its own storage and to keep its append-only ledgers in shape. `o2b brain storage` (and the `brain_storage` tool) measures a per-surface byte census and forecasts days to the configured `storage.cap_bytes` in graduated tiers; a measured critical tier pauses the maintenance lane behind its fifth gate and refuses interactive ingests with `capacity_exhausted` before any write, while an uncapped vault reads unbounded, nothing pauses and nothing refuses. `o2b brain receipts` (and `brain_receipts`) drains the append-only continuity and metrics ledgers by folding expired records into per-kind, per-month aggregates behind an explicit preview-bound apply, with a count guard and a recovery point around every confirmed run. An opt-in `document-dedup` hygiene detector nominates near-duplicate document copies for retirement behind the review gate, keeping the newest copy, retargeting the stale copies' inbound links and auditing each retirement; bulk source removals report per-file failures instead of aborting; and appended sources yield their new text through incremental extraction over a manifest watermark, no whole-source reprocess. Every release is described in the [CHANGELOG](CHANGELOG.md).+1.86.0 stops re-paying for sessions with nothing new to say and measures before it excludes. A session mine that ends with nothing written records a zero-yield tombstone in a session mining ledger, so the next plan over the same transcript returns the recorded breakdown instead of re-offering the turns to a model, with `--force` re-mining exactly the one call it is given to; a distilled and undistilled census joins the import and mining ledgers, `o2b brain import-session --propose-backlog` batches the undistilled backlog into one ask-once envelope deduplicated against what the vault already holds, and an opt-in `backlog_nudge_enabled` SessionStart hook reports the undistilled count once per change. The retrieval trust gate resolves `search_trust_gate_mode` (`off | shadow | enforce`): `shadow` records every would-be refusal with its structural reasons while excluding nothing, and the relational arm classifies each walked node and arriving edge under the same mode. The loopback explorer serves a live-counts console beside its graph (`/counts.json`, `/status.json`). The release also repairs what the external review of v1.85.0 found: receipt compaction folds only opt-in telemetry kinds, the capacity trend buckets its samples per day behind a three-day minimum while the maintenance lane pauses at `storage.pause_percent`, a retired document copy keeps its manifest marker so an unchanged source never quietly re-ingests, and the incremental extraction watermark stamps exactly the bytes the appended delta covered. Every release is described in the [CHANGELOG](CHANGELOG.md).## Documentationdiff --git a/docs/brainstorm/session-mining-economy/cli-output/claude.md b/docs/brainstorm/session-mining-economy/cli-output/claude.mdnew file mode 100644index 00000000..3b641e7d--- /dev/null+++ b/docs/brainstorm/session-mining-economy/cli-output/claude.md@@ -0,0 +1,26 @@+### Variant 1: Seam extension - one ledger, one mode, one server+- **Approach**: Every card becomes a strict extension of a mechanism Open Second Brain already ships. The session import ledger in `src/core/brain/sessions/discover.ts` gains a mining outcome per entry (a schema version bump): the same `hashFile` content hash that already keys the ledger keys the tombstone, so a changed session stops matching and re-mines for free, `recordSessionImports`' locked atomic write and vanished-path pruning carry over, the write happens only on the `commitExtractedSignals` path when zero items were written (keeping the deduped already-known and durability-rejected barren outcome classes apart), and `planExtractSignals` consults the entry before building the envelope, answering a typed suppressed verdict rather than an `ExtractSignalsError` so an adjudicated-barren session is not reported as a failure. The recall gate becomes a versioned `off | shadow | enforce` knob beside `retrievalTrustGateEnabled` in `ResolvedRecallConfig` (shadow the shipping default), realized as an edge-level adjuster inside `runRelationalArm` that reuses `trustGateAdjuster`'s rank-adjust verdict vocabulary and classifies each `RelationalPathStep` edge deterministically - grounding band from `truth/grounding.ts`, query-time validity via `parseValidityWindow` (the same grammar `closedSupersessionTip` already reads), relation polarity, asserted provenance from the claim ledger - counting refusals under fixed reason tokens like `RETRIEVAL_TRUST_EXCLUSION_REASON`, with ungrounded claims reported into a durable extraction-refusals record following the `UngroundedStatedClaim` precedent in `truth/stated-claims.ts`. The console is new routes on `buildLiveServer` (live counts from `discoverSessions` coverage and the mining ledger, recall views over `recall-telemetry.ts` and `report-snapshot.ts` deltas, an ops route over the brain status snapshot), keeping the loopback-only, read-only, per-GET re-read invariants, and the SessionStart nudge is a throttled count over the ledger beside the existing `hooks/hooks.json` injection with a last-offered stamp in the derived store.+- **Trade-offs**:+ - Pros: honors the reuse constraint literally (`trustGateAdjuster`, not a parallel gate); tombstone invalidation falls out of the content hash the ledger already computes; one state file, one lock, one serialization idiom per concern; byte-identical off paths are the existing gates' own contract, and the cache-slot already records whether the gate ran so a shadow receipt cannot be served as an enforced result.+ - Cons: a schema version bump on a shipped ledger is the wave's riskiest single edit, since `readSessionLedger` hard-refuses unknown schemas and every reader must move together; `planExtractSignals` gains a second class of non-error outcome the three existing refusal throws do not anticipate, so the suppressed verdict needs its own typed shape; mining state lives on a ledger whose discovery report is contractually read-only, so the write must stay on the commit path and nowhere else.+- **Complexity**: medium+- **Risk**: low++### Variant 2: Receipt-persisted economy+- **Approach**: Treat the wave's cross-run state as persisted receipts rather than a new ledger. The receipt vocabulary Open Second Brain already emits - `retrieval_decision_trace`, `memory_trust_assessment`, reconciliation reports, dead letters - gains durable per-run persistence in the derived store: a mining commit appends a receipt carrying its outcome counts, the shadow gate appends per-query refusal counts with reasons, and each distill proposal appends an asked-once record. `planExtractSignals` then consults accumulated receipts for the session's current content hash, the console becomes a viewer over the receipt store, and the SessionStart count and its throttle read receipt aggregates.+- **Trade-offs**:+ - Pros: one observation channel for everything the wave surfaces, so console, counts and refusals render exactly what the flows already emit; no schema change to any shipped ledger; ask-once batching needs no separate state because a proposal is itself a receipt; the shadow mode's measure-only discipline is the receipts' native shape.+ - Cons: receipts are per-response accounting by contract, so deriving tombstone eligibility from them conflates observation with decision state; retention or compaction becomes a correctness knob, where a pruned receipt silently re-admits a barren session; the plan gate must aggregate many receipts to answer one boolean, paying a scan per plan that a single keyed entry avoids; per-query shadow counts accumulate without bound until a policy exists.+- **Complexity**: medium+- **Risk**: medium++### Variant 3: Mining-economy journal+- **Approach**: One append-only journal for the whole session-mining economy: mining commits, gate classifications, distill proposals and throttle decisions each append typed events keyed on session content hash, and derived projections answer the wave's four questions - tombstone eligibility, undistilled counts, gate enforcement state, console views. Enforce mode reads the projection at query time, the explorer renders the same projections, and a compaction policy folds event history into snapshot rows.+- **Trade-offs**:+ - Pros: the strongest audit story, since every suppression and refusal is a replayable event carrying its reason; projections give the console and the SessionStart count one reader; future economy rules (per-session caps, spend accounting) become new projections over existing events rather than new stores.+ - Cons: introduces an event-sourcing substrate - append, compaction, projection rebuild - that Open Second Brain does not have, where every existing store is a locked read-modify-write snapshot; per-query gate events force a rotation policy that is itself an unresolved design question; the largest surface against the one-PR 40-60 file envelope and the hardest to hold to byte-identical off behavior.+- **Complexity**: large+- **Risk**: high++### Recommended: Variant 1+**Rationale**: The validator-settled scope already names these exact seams - reuse `trustGateAdjuster` rather than build a parallel gate, key the tombstone on the hash the session ledger already computes, and fold the console into `buildLiveServer` - so Variant 1 is the settled scope rendered as code, within the one-PR envelope and with the smallest deviation from shipped behavior. Variant 2 makes a barren session's eligibility depend on receipt retention policy, and Variant 3 buys an event-store substrate this wave neither needs nor could land safely.diff --git a/docs/brainstorm/session-mining-economy/cli-output/prompt.md b/docs/brainstorm/session-mining-economy/cli-output/prompt.mdnew file mode 100644index 00000000..2bcfef3c--- /dev/null+++ b/docs/brainstorm/session-mining-economy/cli-output/prompt.md@@ -0,0 +1,213 @@+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.++# Task++One Open Second Brain release wave, four related kanban cards (session-mining economy). The card bodies below are verbatim; each is followed by the validator/review comments that carry the settled scope decisions.++## Card t_05a0eaa4 (priority 4)+TITLE: [upstream:yantrikdb] Recall-time claim-chain eligibility gate (grounding status + shadow/enforce modes + extraction refusals)++BODY:+**Source**: https://github.com/yantrikos/yantrikdb/releases/tag/v0.22.0+**Repo**: yantrikos/yantrikdb (31★)+**Released**: v0.22.0 (2026-09-07T21:03:05Z)++## What+An enforcement layer on the recall lane: a versioned per-claim `grounding` status (0 never-validated / 1 cooperative / 2 extractor-bound-with-evidence-span), a durable `claim_chain_gate_mode` (off | shadow | enforce, shadow on every install) where under enforce every edge of a returned path must be eligible (grounded, valid as of query time, positive, asserted), plus an `extraction_refusals` table (ungrounded claims are reported, not silently stored). Value objects are admitted as claim objects where the relation admits values.++## Why useful for Open Second Brain+This is the quality/enforcement layer that sits on top of a multi-hop relational recall arm — it stops ungrounded or stale edges from participating in surfaced paths, and the shadow-first mode lets you measure what enforcement would refuse before turning it on. Open Second Brain has the recall traversal and context-item grounding as separate pieces but nothing that gates graph-claim edges at query time.++## Status in OSB+- **Verdict**: not_in_osb_useful+- **Codegraph hints**:+ - Relational recall arm exists but has NO per-edge eligibility gate: `src/core/search/pipeline/relational-arm.ts:35`, `src/core/search/relational-fanout.ts:70` (bounded depth-2 fan-out into RRF fusion).+ - Grounding exists only for CONTEXT ITEMS, not graph claims: `src/core/brain/provenance/epistemic.ts` (five statuses observed/derived/hypothesis/plan/unknown from frontmatter). No `grounding` column on claims/edges, no shadow/enforce recall mode, no `extraction_refusals` equivalent found.+ - Integrity discipline that a gate would extend: `src/core/search/store/state.ts:212` (schema-mismatch refusal), `src/cli/search/verbs/check.ts:69`.++## Notes+Related to but not the same as existing triage tasks: builds a recall-time gate on top of the multi-hop chain recall (t_c37af4e9) using claim grounding provenance (t_3e87656b) and admission/junk discipline (t_11d51443), and consumes the temporal validity windows (t_edba0c15) for the "valid as of query time" edge check. None of those four describes the shadow/enforce eligibility gate itself, so this is net-new capability worth its own card; the orchestrator may want to link it as a dependent of that cluster.++COMMENTS:+- [osb-triage-validator]: osb-triage-validator @ 2026-09-12:+- sanity: Correct two anchors - src/core/search/store/state.ts:212 and src/cli/search/verbs/check.ts:69 are a docblock line and an import line respectively, so neither shows the refusal the card cites. Add the gate the card misses: src/core/brain/trust/retrieval-gate.ts:1-20 and :97 (trustGateAdjuster), mounted at src/core/search/pipeline/post-rank.ts:128-131, configured by recall.retrievalTrustGateEnabled (src/core/search/types.ts:1318-1332). relational-arm.ts:34-36, relational-fanout.ts:70 and provenance/epistemic.ts verified.+- cluster: recall-lanes; depends on t_c37af4e9 (multi-hop chains, p2), t_3e87656b (claim grounding, p2) and t_edba0c15 (validity windows, p3) as the card says+- priority: set to 2: The enforcement PATTERN already exists at recall time and can be reused, so this is an extension rather than net-new capability - but it is genuinely last in a four-card chain, and building it before the grounding status and validity windows it consumes would leave a gate with nothing to read. Anchors verified against live source this run.+- premise: premise_partly_false - "Nothing gates edges at query time" is right about EDGES and wrong about the mechanism being absent. A recall-time eligibility gate already ships for pages: the retrieval trust gate contributes an exclude-with-reason verdict so quarantined material ranks zero and reaches no pack, classifying deterministically from controlled-vocabulary frontmatter only - never prose - over three signals each owned by an existing module (src/core/brain/trust/retrieval-gate.ts:1-20, :97), mounted as a rank adjuster on both the semantic and pure-lexical paths (src/core/search/pipeline/post-rank.ts:128-131). It is config-gated, excluded items are counted WITH REASONS into a retrieval_decision_trace receipt, every pack carries a trust-assessment receipt, and with the flag off ranking and outcome shape stay byte-identical (src/core/search/types.ts:1318-1331). That off/on-with-receipts shape is most of what the card calls shadow/enforce - what is missing is the middle mode that measures refusals WITHOUT excluding, plus per-edge (rather than per-page) eligibility. Rescope to: a third mode on the existing gate, an edge-level adjuster inside the relational arm, and the extraction-refusals record - and sequence it behind its three dependencies.+- [osb-triage-review]: osb-triage-review @ 2026-10-15: priority 2 -> 4. All three prerequisites are now done (t_3e87656b claim grounding, t_c37af4e9 multi-hop chains, t_edba0c15 validity windows), so the card is unblocked; the rescoped work is concrete: a measure-only shadow mode on the shipped retrieval trust gate (src/core/brain/trust/retrieval-gate.ts:97 trustGateAdjuster), an edge-level adjuster in the relational arm (src/core/search/pipeline/relational-arm.ts) and an extraction-refusals record. Recall quality is the active product focus (v1.82.0-v1.84.0), and this closes the last open card of that chain.++---++## Card t_93617709 (priority 4)+TITLE: [upstream:gbrain] Durable zero-yield ledger to stop re-mining sessions that never produce signals++BODY:+**Source**: https://github.com/garrytan/gbrain/releases/tag/v0.48.5.0+**Repo**: garrytan/gbrain (18439★)+**Released**: v0.48.5.0 (2026-09-08T03:35:57Z)++## What+Add a durable tombstone ledger that records which sessions were mined for signals and yielded zero accepted signals, so those sessions are excluded from future extraction planning until the session is edited/re-imported. Mirrors upstream's migration v146 zero-atom tombstone table, but scoped to OSB's session signal mining (`planExtractSignals`) rather than transcript atom extraction.++## Why useful for OSB+OSB currently rebuilds the same mining plan and re-sends the same user turns to the caller's LLM every time a session is re-scanned, even when a prior mining run produced nothing — pure repeated model spend for zero yield. A zero-yield ledger shrinks the recurring extraction work pool and keeps it shrunk, so re-scans skip sessions already proven barren until their content changes.++## Status in OSB+- **Verdict**: not_in_osb_useful+- **Codegraph hints**: `src/core/brain/extract-signals.ts:340` (`planExtractSignals` — read-only plan, no cross-run yield memory), `extract-signals.ts:347-380` (existing refusals cover empty/non-user sessions but NOT legitimate zero-signal outcomes), `commitExtractedSignals` (accepted-signal writer — natural point to also record a zero-yield tombstone when an accepted-signal count of 0 comes back). Ledger consumer would gate at `planExtractSignals`. Related but non-overlapping: `src/core/brain/dream.ts:274` (`applySalienceGate` — same "don't re-spend on nothing" discipline, but scoped to preferences, not session-mining pools); `src/core/brain/sessions/import.ts` (import-session — edit/re-import is the natural invalidation trigger); `src/mcp/brain/extract-tools.ts` (`brain_extract_signals` plan/commit lanes). Not found: any durable zero-yield/tombstone table for session mining — codegraph MCP unavailable this session; confirmed by direct read + 119 triage titles checked, none cover it (nearest t_0358af42, t_05bf583f both concern distillation counts/salience, not a re-mining ledger).++## Notes+- Invalidation semantics matter: a tombstone must be cleared (or made eligible again) when the session's user turns change — mirror upstream's "editing the file makes it eligible again" via the import/recall path so newly-added content is not permanently ignored.+- OSB runs no model itself (the LLM step is delegated to the caller), so the benefit lands as fewer proposed mining turns handed back to the caller — i.e., the ledger must suppress the plan, since OSB cannot observe token cost directly.+- Distinguish "zero accepted after mining" (tombstone) from the existing hard refusals at extract-signals.ts:347-380 (which already short-circuit before any LLM work) — the new state covers sessions that legitimately reach mining but yield nothing.++COMMENTS:+- [osb-triage-validator]: osb-triage-validator @ 2026-09-12:+- sanity: Correct the refusal range to src/core/brain/extract-signals.ts:345-376 (the card says :347-380); planExtractSignals at :340 and applySalienceGate at src/core/brain/dream.ts:274 verified. Add the commit-side shape the ledger would key on: CommitExtractedSignalsResult at :229-241 (written / staged / deduped / durabilityRejected / rejected).+- cluster: overlaps t_0358af42 (undistilled-session count, p2) - both want a per-session mining ledger; sequence together, no new link+- priority: set to 2: The gap is real and the seam is exact - suppress the plan, record on commit - but the invalidation rule is a genuine design decision and a tombstone that never clears would permanently blind the miner to a session the operator later adds to. Solid feature on an existing subsystem. Anchors verified against live source this run.+- premise: premise_confirmed - planExtractSignals is a pure read with no cross-run memory: it re-walks the session's turns and rebuilds the mining envelope every call (src/core/brain/extract-signals.ts:340-385), and its three refusals cover a non-capture boundary decision, zero imported turns, and zero user-authored turns (:345-376) - none of which is "mined and yielded nothing". Three notes for scope. The tombstone needs a richer trigger than "zero written": the commit result distinguishes written, staged under write approval, deduped, and durability-rejected (:229-241), and only the last two mean "this session had nothing worth keeping" - a session whose items all DEDUPED yielded nothing new but its content is already known, which is a different state from barren, and staging is not a yield failure at all. Second, the card is right that the benefit is fewer turns handed back, since the kernel never calls the model itself - so the ledger must gate the PLAN, not the commit. Third, the invalidation key should be the content hash the session-discovery ledger already uses rather than mtime (src/core/brain/sessions/discover.ts:14-24 explains why: rsync, Syncthing and git checkout all move the clock without changing a byte).+- [osb-triage-review]: osb-triage-review @ 2026-10-15: priority 2 -> 4. Real, still-absent spend saver on main (no zero-yield record anywhere in src/core/brain/extract-signals.ts), and the 2026-09-12 validator already answered the design questions: gate the plan not the commit, tombstone only on durability-rejected outcomes (CommitExtractedSignalsResult now at :244), invalidate on the content hash used by sessions/discover.ts. Anchors drifted (planExtractSignals moved from :340 to :443); pairs with t_0358af42.+Status: stale_anchors++---++## Card t_15ebcfc1 (priority 3)+TITLE: [upstream:supermemory] Local memory console with live counts and interactive memory graph++BODY:+**Source**: https://github.com/supermemoryai/supermemory/releases/tag/server-v0.0.7+**Repo**: supermemoryai/supermemory (25432★)+**Released**: server-v0.0.7 (2026-08-15T06:47:56Z)++## What+The self-hosted server's local console welcome page gains a Memory tab: live counts of documents and memories, recent items, container tags, and an interactive memory graph — a zero-setup local UI over the memory store for a self-hosted instance.++## Why useful for OSB+OSB's operator surface today is textual: `buildTodayDashboard` renders obligations, open loops, and activity as a markdown envelope, and brain state is otherwise only visible through CLI verbs. An interactive local view over the link graph — nodes for notes, live counts, recent captures — would give operators a fast visual sense of brain growth and structure without opening Obsidian or running queries, and would complement the existing read-only dashboard rather than replace it.++## Status in OSB+- **Verdict**: unverified+- **Codegraph hints**: src/core/brain/today-dashboard.ts:20 `buildTodayDashboard(vault, opts)` — read-only envelope over four sections (obligations, open loops, activity timeline, totals), frozen and text-rendered; no interactive/graphical surface exists. Link-graph data is available via src/core/brain/link-graph/ (format-wikilink.ts et al.) and search store links (src/core/search/store/links.ts). not found — no interactive UI or graph visualization surface anywhere in src/. Related but distinct: t_a160764a (fleet view over sibling agents) covers cross-agent counts, not the local memory graph.++## Notes+Verdict is `unverified`: the Claude analysis CLI was unavailable at capture time (OAuth session expired); body drafted from codegraph hints. Scope caution from OSB's own history: OSB deliberately treats visualization node limits as an external-partner concern (graphify viz), and thin passthroughs to absent binaries are forbidden — an OSB-side console should render from OSB's own link-graph store, not shell out.++**Also observed**:+- plur-ai/plur (154★) — v0.18.0 — sharpens the "live counts" half of this task with concrete surfaces: a written-per-day chart, a most-recalled list, per-engram recall counts with row-expand to the full engram; read-only by construction (browsing never fires lazy writes such as decay); loopback-only 127.0.0.1 bind by default with a warning on --host widening. OSB's shipped explorer (src/core/brain/explorer.ts buildLiveServer) already realizes the interactive-graph half over the rule graph; the recall-telemetry data (src/core/brain/recall-telemetry.ts, report-snapshot.ts Brain/metrics/ contract) exists but is not joined to any viewer — fold these views into the explorer (or a sibling surface) reusing its loopback + read-only invariants.++**Also observed**:+- Ar9av/obsidian-wiki (977★) — v2026.09.1 — read-only operations console at /ui (PR #210): service/runtime ops state for the human operator on the loopback web server; complements the memory-content views — an ops route on OSB's existing buildLiveServer (src/core/brain/explorer.ts:355) rendering the brain_status snapshot would cover it with no new infrastructure.++**Also observed**:+- yantrikos/yantrikdb (31★) — v0.23.0 — Memory Atlas exporter (serves export dir on 127.0.0.1; child-process isolation so no second SQLite handle opens the store; encrypted stores refused; mkstemp+os.replace artifact writes) plus read-only yantrikdb-tui terminal explorer (SQLite online-backup snapshot into a private temp dir, engine built on the copy, embedder identity verified by digest before search is enabled)++COMMENTS:+- [claude-dev-agent]: claude-dev-agent @ 2026-08-16: triage re-prioritisation pass (operator-directed, full-column review of all 62 triage tasks against v1.49.0 source)+- priority: kept at 0, meaning re-stated.+- PROCESSED and PARKED - the value 0 is unchanged but its meaning is not: it was 0 because the analysis CLI was down at capture, and it is now 0 because it has been read and judged. Same verdict as the other four UI items: a live console with an interactive memory graph needs a rendering surface that does not exist here, and the body's own caution already notes this project treats visualization as an external-partner concern. The non-UI half is already served: `buildTodayDashboard` renders counts and recent activity as text.+- rubric applied board-wide this pass: 4 = verified in-repo defect, acceptance criteria runnable today | 3 = internal, target identified, no open design question | 2 = internal, needs a design decision first | 1 = real but gated on an external product or decision, or a marginal increment | 0 = PARKED: processed and judged, verdict is the deliverable (NOT 'unprocessed').+- [claude-dev-agent]: claude-dev-agent @ 2026-08-16: CORRECTION to the re-prioritisation pass posted earlier today. The numbers move; the ranking argument on this card is unchanged and still applies.+- priority: 0 -> 1.+- What I got wrong: 0. On this board 0 means the priority has not been set yet, and 5 is an operator-only blocker pin above the working range - so the working range is 1..4 and every rank I assigned sat one step too low. Worse, my earlier comment redefined 0 as 'parked', which would have made 20 deliberately-judged cards read as untriaged and invited the next validator to re-triage them.+- The former top two tiers merge at 4 because the working range has four slots, not five: t_9b209fbe (the one verified in-repo defect with runnable acceptance criteria) now shares 4 with the five target-identified cards and stays first among them.+- corrected scale, in force board-wide: 4 = actionable now (internal, target identified, no open design question) | 3 = internal, needs a design decision first | 2 = real but gated on an external product or decision, or a marginal increment | 1 = PARKED (processed and judged; the verdict is the deliverable) | 0 = priority not yet set - no triage card should hold this after this pass | 5 = blocker, operator-set only, never self-assigned.+- [product-tech-lead]: Also surfaced upstream in plur-ai/plur v0.18.0 (https://github.com/plur-ai/plur/releases/tag/v0.18.0). The interactive-graph half of this task is already realized by src/core/brain/explorer.ts (buildLiveServer, rule-graph view); the plur release sharpens the live-counts half: written-per-day chart, most-recalled list, per-engram recall counts, strict read-only browse. The telemetry data exists (recall-telemetry.ts, report-snapshot.ts) but no viewer surfaces it — folding these views into the explorer would close this task's gap.+- [osb-triage-validator]: osb-triage-validator @ 2026-08-23T08:38Z:+- sanity: clean+- priority: set to 3: overriding the 2026-08-16 parked verdict (was 1) - its premise is now stale. The claim 'no rendering surface exists' was falsified by plur v0.18.0 evidence on 2026-08-21: src/core/brain/explorer.ts buildLiveServer (:354, loopback-only, read-only, re-reads vault per GET) is exactly the local console surface, verified live this run; it already serves a rule-graph view, and recall-telemetry.ts + report-snapshot.ts hold the counts data with no viewer. Work = fold live-counts views into the explorer. Internal, one bounded extension, no architectural unknowns.+- [product-tech-lead]: Also surfaced upstream in Ar9av/obsidian-wiki v2026.09.1 (https://github.com/Ar9av/obsidian-wiki/pull/210). They added a read-only operations console at /ui served by the wiki server — service/runtime state for the human operator, not just memory content. Takeaway: the console family should cover BOTH facets (memory counts/graph AND running-service ops snapshot); OSB already has the data via brain_status/second_brain_status MCP tools (src/mcp/registry-guard.ts:377) and the loopback server to host it (src/core/brain/explorer.ts:355 buildLiveServer) — an ops route there needs no new infrastructure.+- [product-tech-lead]: Also surfaced upstream in yantrikos/yantrikdb v0.23.0 (https://github.com/yantrikos/yantrikdb/releases/tag/v0.23.0): packaged Memory Atlas exporter (serves one store's export on 127.0.0.1, child-process isolation, encrypted stores refused, mkstemp+os.replace artifacts) plus a read-only terminal explorer (yantrikdb-tui) that snapshots the store via SQLite online-backup and never opens the source with the engine. Reinforces the same capability: an operator-facing read-only explorer over the brain store.++---++## Card t_0358af42 (priority 3)+TITLE: [upstream:iwe] Session-start count of undistilled sessions with cross-session-deduped, batched distill proposals++BODY:+**Source**: https://github.com/iwe-org/iwe/releases/tag/iwe-v0.21.0+**Repo**: iwe-org/iwe (1086★)+**Released**: iwe-v0.21.0 (2026-08-29T21:50:13Z)++## What+At session start, report how many past sessions remain undistilled and, at most once per configured interval, offer to read them. The distill flow asks about every candidate before writing (Keep all / Skip all / Let me pick), uses one candidate at a time for a few or a batched numbered multi-select for many, and reads the backlog with one reader per session so a fact several sessions raised is asked about only once.++## Why useful for OSB+OSB distills individual sources and mines single-session signals, but has no backlog-level "N sessions undistilled" nudge and no cross-session dedup, so a recurring fact gets re-proposed per session and the user is asked the same thing repeatedly.++## Status in OSB+- **Verdict**: not_in_osb_useful+- **Codegraph hints**: src/core/brain/distill/distill-source.ts (distills a single SOURCE into atomic claims, not session-backlog-aware); src/core/brain/extract-signals.ts (mines one session's turns into Brain/inbox/, no cross-session dedup); hooks/hooks.json (SessionStart injects budgeted active.md, no undistilled-session count)++## Notes+Two separable pieces: the session-start count/reminder gate (throttled to once per interval) and the cross-session-deduped batched proposal UI. The dedup step ("one reader per session, ask once") is the higher-value, harder part and depends on a per-session ledger of which sessions have been distilled.+++**Also observed**:+- plastic-labs/honcho (4753★) — PR #1115 — deriver backlog exported as metrics from an API endpoint: machine-readable backlog visibility complementing the session-start nudge surface.++COMMENTS:+- [product-tech-lead]: Also surfaced upstream in plastic-labs/honcho PR #1115 (https://github.com/plastic-labs/honcho/pull/1115). feat(api): export deriver backlog as metrics from an API endpoint — same OSB-side intent (visibility into the un-distilled/un-derived backlog) via a machine-readable metrics surface rather than a session-start nudge; if OSB grows an operational metrics endpoint, the undistilled count belongs there too.+- [osb-triage-validator]: osb-triage-validator @ 2026-09-12:+- sanity: The card has no path:line anchors and the hooks path is given as a bare file. Record: src/core/brain/sessions/discover.ts:1-25 (the outstanding-sessions sweep the card treats as absent), cross-session dedup at src/core/brain/dedup-hash.ts:55-70, SessionStart block at hooks/hooks.json:27. Strike "no cross-session dedup, so a recurring fact gets re-proposed per session".+- cluster: overlaps t_93617709 (durable zero-yield ledger) - both want a per-session mining ledger; sequence them together, no new link+- priority: set to 2: With the two refutations applied, the residue is a distillation-keyed view over an existing sweep plus a throttled SessionStart surface - bounded work on existing subsystems, but the proposal UI (ask-once batching) is still undesigned and the kernel cannot run the distill step itself. Anchors verified against live source this run.+- premise: premise_partly_false - Two claims are refuted. (1) Cross-session dedup already exists: buildDedupIndex scans the whole inbox and processed dirs by dedup_hash, vault-wide and not per session (src/core/brain/dedup-hash.ts:55-70), and commitExtractedSignals runs each item through it before the durability gate, naming what it rejected rather than dropping it silently (src/core/brain/extract-signals.ts:433-455). So a recurring fact is already suppressed at write time across sessions; what is missing is suppressing the ASK, i.e. dedup before the proposal is put to the operator. (2) The backlog census is not absent either: src/core/brain/sessions/discover.ts:1-25 already sweeps every session log on the machine and reports which are still outstanding, backed by a content-hash ledger explicitly built so a second run is cheap. It is keyed on IMPORT, not on distillation, and nothing surfaces it at session start (hooks/hooks.json:27 wires SessionStart to the active.md injection only). Rescope to: a distilled/undistilled key on that existing ledger, a throttled count on the SessionStart hook, and the ask-once batching layer above the dedup that already runs.+- [osb-triage-review]: osb-triage-review @ 2026-10-15: priority 2 -> 3. Internal and unchanged on main (no undistilled key or SessionStart count); the validator rescoped it to a distilled key on the sessions/discover.ts ledger, a throttled SessionStart count and ask-once batching above the existing dedup. Build it together with t_93617709, which shares the per-session mining record; the batching UI is the one open design point.++---++# Project context++Open Second Brain: a local-first memory/knowledge layer for AI agent runtimes, TypeScript on Bun, MCP server + CLI. Recent commits:+948a0403 feat: storage lifecycle for long-lived vaults (v1.85.0) (#257)+ef974e0d fix: comma-separated and legacy scopes no longer abort the dream pass, plus three test-isolation fixes (v1.84.1) (#254)+14a99584 feat: recall lane concurrency, overlapped_ms accounting, a caller-selectable packing policy and standing-instruction facets (v1.84.0) (#252)+e239d323 feat: shipped-seam wiring - a per-session recall pool cap, usage-ordered skill offers, seeded bench sampling and OpenCode V2 configs (v1.83.0) (#251)+44a8ccaf feat: recall quality - an iterative recall loop, coverage-first session recall and a measured surfacing key (v1.82.0) (#250)+259e4b70 fix(snapshot): decode readdir buffer entries as UTF-8 so exclusions match (v1.81.1) (#243)+28ea378b chore(deps): bump the github-actions group with 2 updates (#245)+745c4a95 chore(deps): bump the bun-minor-and-patch group with 4 updates (#246)+323fb9e4 feat: answer trust - verified quotes, claim provenance and durable source evidence (v1.81.0) (#249)+a00fa3f5 feat: recall quality - mined sessions, honest admission and a store that filters in SQL (v1.80.0) (#248)+3f029241 feat: write-side trust - identity, permission and review for every agent write (v1.79.0) (#247)+66f9d847 feat: credential custody under a passphrase and a verifiable trust surface (v1.78.0) (#244)+3afb94a5 feat: a truth layer you can correct and time-travel (v1.77.0) (#242)+e907c692 ci: balance windows test shards by measured duration (#241)+f3d7185e fix: keep generated cluster digests out of community detection (#240)+646a49df feat: freshen the search index on read, rewriting only changed chunks (v1.76.0) (#239)+78f924b9 feat: deliver context to subagent turns, hygiene digest at turn end and tiered context shrink (v1.75.0) (#238)+542d214b fix(opencode): support both plugin APIs (v1.74.1) (#190)+0cbb60b8 feat: near-duplicate defense across the fact lifecycle (v1.74.0) (#237)+23090696 chore(deps-dev): bump the bun-minor-and-patch group with 2 updates (#231)+Related files (from validator recon): src/core/brain/sessions/discover.ts, src/core/brain/extract-signals.ts, src/core/brain/dedup-hash.ts, src/core/brain/trust/retrieval-gate.ts, src/core/search/pipeline/post-rank.ts, src/core/search/pipeline/relational-arm.ts, src/core/search/types.ts, src/core/brain/explorer.ts, src/core/brain/recall-telemetry.ts, src/core/brain/report-snapshot.ts, hooks/hooks.json.++Conventions:+- Kernel stays LLM-free; extraction contracts name their LLM steps explicitly (needs-llm-step events).+- No silent fallbacks; errors surface by name. No stubs. No hardcoded natural-language phrases in any language.+- Strict TDD; one conventional commit per atomic unit; formatter and linter green before every commit.+- Test runs use a throwaway HOME (env HOME=$(mktemp -d) bun test <paths>).+- Config surfaces live in src/core/search/types.ts and friends; every new knob needs a default and a doctor story where relevant.++Constraints:+- The wave ships ONE PR, feature -> minor bump, about 40-60 files.+- The claim-chain eligibility gate must ship measure-only (shadow) by default.+- The zero-yield tombstone must key on session content hash so changed sessions re-mine.+- Reuse the shipped retrieval trust gate (trustGateAdjuster) rather than building a parallel mechanism.++# Required output format++Produce exactly 3 distinct architectural variants for the WHOLE four-card wave (one coherent design, not per-card). For each variant:++### Variant N: <short name>+- **Approach**: 2-3 sentences describing the variant.+- **Trade-offs**: bullet list of pros and cons.+- **Complexity**: small | medium | large+- **Risk**: low | medium | high++After the three variants, add exactly one recommendation:++### Recommended: Variant N+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.++Output nothing outside of these sections.diff --git a/docs/brainstorm/session-mining-economy/design.md b/docs/brainstorm/session-mining-economy/design.mdnew file mode 100644index 00000000..d4e91a8a--- /dev/null+++ b/docs/brainstorm/session-mining-economy/design.md@@ -0,0 +1,117 @@+# Session mining economy - one per-session mining record, a shadow claim-chain gate, a throttled backlog nudge and the live counts console++**Status:** draft+**Author:** feature-release-playbook (host wave)+**Audience:** implementation++## Problem statement++Open Second Brain spends model effort on session mining without any memory of what mining already concluded: `planExtractSignals` (src/core/brain/extract-signals.ts:443) is a pure read with no cross-run record, so a session that was mined once and yielded nothing is rebuilt and re-offered to the caller's model on every re-scan, and nothing tells the operator how much imported session backlog was never mined at all. On the recall side, a page-level eligibility gate ships (`trustGateAdjuster`, src/core/brain/trust/retrieval-gate.ts:97) but has no measure-only middle mode and no per-edge eligibility inside the relational arm, so enforcement cannot be observed before it is turned on. On the operator surface, the recall-telemetry and operator-snapshot data exist with no viewer, because the only local console is the loopback explorer and it serves the rule graph only. This wave ships the four rescoped cards together: a durable per-session mining record with a zero-yield tombstone, a throttled SessionStart undistilled count with ask-once batching, a shadow/enforce claim-chain eligibility gate reusing the shipped trust gate, and live-counts plus ops routes folded into the existing explorer.++## Scope++Card t_93617709 (durable zero-yield ledger), recon verdict premise_confirmed:++- A new shared per-session mining record module, src/core/brain/session-mining-ledger.ts, stored beside session-import-ledger.json in the derived store. Entry: schema version, portable sessionId, mined-content hash (SHA-256 over the mined turn material the plan embeds: turn id, text, timestamp), outcome breakdown (written, staged, deduped, durabilityRejected), recorded_at. Read fail-closed on corrupt or unknown schema; write under `acquireLockSyncWithRetry` with the vault-identity assertion, atomic, byte-identical no-op short-circuit, pruned of sessions whose turns no longer exist (the discover.ts idioms).+- Record-at-commit: `commitExtractedSignals` records the entry only on a successful commit with `written.length === 0 && staged === 0`, carrying the breakdown. Staging is never barren; a partial write (`ExtractSignalsWriteError`) never tombstones.+- Gate-at-plan: the gate runs inside `planExtractSignalsPrefiltered` before the decision-model prefilter, so a tombstoned session spends no decision-model requests either. The suppression is a typed, named outcome beside the prefilter's `skipped` outcome, carrying the recorded breakdown as evidence; both the CLI verb and the MCP tool render it symmetrically. An unchanged session stays suppressed; a re-import that changes turns changes the hash and re-opens eligibility with no clearing step.+- Operator escape hatch: an explicit force/re-mine flag on the CLI verb and a documented tool argument that bypass the gate for one plan call.+- A named, non-throwing failure field on `CommitExtractedSignalsResult` (and both surface outputs) when the commit succeeded but the tombstone write failed.++Card t_0358af42 (session-start undistilled count), recon verdict partly_false_rescoped:++- A distilled/undistilled census derived vault-side from the mining record joined to the import ledger on `sessionRefIdentity` (portable basename, src/core/brain/sessions/import.ts:315). No log re-hashing; sessions the plan refuses (boundary, zero imported turns, zero user turns) are excluded from the backlog because they can never be distilled.+- The census surfaced in the import-session verb's coverage output per runtime (`--status` family, src/cli/brain/verbs/import-session.ts).+- A default-off SessionStart hook (hooks/backlog-nudge.ts) copying the gap-agenda shape: flag via `resolveConfigFlag` with an env override, process ceiling, `additionalContext` count line, fail-open exit 0 with a byte-identical preamble when off, one hooks.json entry in the existing SessionStart block.+- Throttle state that is vault-global, overwrite-only and change-keyed (the hygiene-digest-state shape, hooks/lib/backlog-nudge-state.ts): the count is injected when the undistilled count has changed since the last injection, never on a timer that re-nags an unchanged backlog; under the 10 s hook budget it degrades to silence, never to a fabricated number.+- The ask-once batching layer as a pure core function (src/core/brain/distill/backlog-proposals.ts): takes the pending undistilled set plus a `loadDedupIndex` result, drops candidates whose `computeDedupHash` already exists in the vault (each drop named in a report), collapses the remainder to one proposal per distinct dedup hash across sessions, and emits a single batched envelope through a flag on the import-session verb for the calling agent to present.++Card t_05a0eaa4 (recall-time claim-chain eligibility gate), recon verdict partly_false_rescoped:++- A tri-state mode `off | shadow | enforce` on the shipped retrieval trust gate, resolved as `search_trust_gate_mode` beside `search_trust_gate_enabled` in src/core/search/index.ts with the `parseFusionMode` refusal discipline; default off; `enforce` is today's exclude behavior made explicit; shadow reuses `trustGateAdjuster`'s classification, excludes nothing, and records every would-be refusal with its structural reasons.+- An edge-level eligibility adjuster inside `runRelationalArm` (src/core/search/pipeline/relational-arm.ts): per-node and per-edge classification from the same controlled-vocabulary frontmatter signals the page gate reads (epistemic status, contamination and untrusted-source markers, `parseValidityWindow` at query time, relation polarity), dropped from the arm's contribution under enforce and counted under shadow, with named structural reason tokens.+- A named extraction-refusals record feeding the existing `retrieval_decision_trace` and `memory_trust_assessment` receipts (src/core/brain/trust/retrieval-receipts.ts): counts plus a reason histogram, reported separately for the pool and the arm, never folded into `excluded`. A record in the outcome, not a durable table.+- The resolved mode rides the query-cache key (src/core/search/pipeline/cache-slot.ts) and the explain envelope (src/core/search/explain-envelope.ts).++Card t_15ebcfc1 (local memory console), recon verdict partly_false_rescoped:++- A console collector module (src/core/brain/explorer-console.ts): `summarizeRecallTelemetry` passthrough, most-recalled top-N from `deriveUsageSignals` joined to artifact id/path under a named cap, a written-per-day histogram over a named 30-day window bucketed from continuity record createdAt, and the latest report-snapshot date per surface, fail-soft as "disabled" when snapshots are off. Frozen output, deterministic given `opts.now`.+- Two new GET routes on `buildLiveServer`: `/counts.json` (the collector payload) and `/status.json` (`buildOperatorSnapshot`). Loopback-only, re-read-per-GET, no write endpoints; `configPath` threaded in through an additive options object on `buildLiveServer`.+- Template sections for counts and ops in templates/brain-explorer.html, fetched client-side from the new routes; the `__GRAPH_JSON__` single-marker invariant is untouched.++## Out of scope++- Any model or network involvement in the mining ledger; any config knob for the ledger itself; mtime-based invalidation; reuse of the import ledger's absolute-path keys or files; clearing or rewriting the import ledger; an expiry or TTL (the content hash is the invalidation); removal of the three existing plan refusals; extensions to fact-extract, dream or other lanes.+- An interactive prompt loop in the kernel; any change to the write-time dedup gates (verified correct as shipped); a second machine-wide session sweep; a metrics endpoint for the undistilled count (the deferred honcho-PR residue; existing report surfaces first); undeclared-field mutation of session-import-ledger.json.+- A per-claim `grounding` status column or any schema migration; a new durable `extraction_refusals` table; durable cross-install mode persistence; an LLM in the gate; re-entry into relational-fanout.ts traversal internals; enforce-by-default anywhere; a per-call override of the gate mode (config is the only surface this wave).+- Non-loopback binds or `--host` widening; write endpoints, telemetry emission from the viewer, or `captureReportDelta` calls from a read surface; any caching layer, file watcher or Server-Sent Events; new visualization dependencies; a new on-disk schema served by the console; folding the today-dashboard envelope into the explorer; the v1.85.0 storage census and capacity tier on the ops route (deferred).++## Chosen approach++Variant 1, seam extension, agreed with the consultant. Every card becomes a strict extension of a mechanism Open Second Brain already ships, which is exactly what the validator-settled rescopes name: reuse `trustGateAdjuster` rather than build a parallel gate, key the tombstone on content the same way the session import ledger already does, and fold the console into `buildLiveServer` instead of inventing a surface. Variant 2 is declined because deriving tombstone eligibility from accumulated receipts makes a barren session's eligibility depend on receipt retention policy, and a pruned receipt would silently re-admit a barren session; Variant 3 is declined because an append-only event journal with compaction and projections is a substrate Open Second Brain does not have, against a wave whose every existing store is a locked read-modify-write snapshot.++One mechanism correction to the variant as written: the consultant placed the mining outcome in a schema version bump of session-import-ledger.json. The design instead puts the shared mining record in its own derived-store file, because the import ledger's reader rebuilds entries field-by-field and silently erases undeclared fields on rewrite (src/core/brain/sessions/discover.ts readEntry), and the import ledger's keys are absolute machine paths while the mining lane holds only the portable sessionId with no join back. A separate record keyed on the portable identity inherits the same locked, atomic, fail-closed, pruned idiom without touching a shipped ledger's schema, and serves both mining cards from one home.++## Design decisions++- One shared per-session mining record serves both mining cards. The 2026-09-12 validator pairing directed the zero-yield tombstone (t_93617709) and the distilled/undistilled key (t_0358af42) onto one ledger home; the record's outcome breakdown answers both. Distilled means mined with accepted yield on the current content (`written + staged > 0`); a session whose items all deduped or all durability-rejected is known-nothing-new (tombstoned), which is a different state from undistilled backlog; never-mined sessions and sessions whose content changed since the last mining are the undistilled count.+- The tombstone is keyed on a SHA-256 of the mined turn material itself (turn id, text, timestamp - exactly the envelope's inputs), not on the discover ledger's source-file byte hash. The import ledger hashes file bytes keyed by absolute machine paths; session_turn continuity records carry neither a source path nor a hash, and `sessionRefIdentity` shows the mining lane's key is the portable basename. Hashing the turn material yields the invalidation semantics for free: a re-import that changes turns changes the hash and re-opens eligibility with no explicit clearing step.+- The plan gate lives inside `planExtractSignalsPrefiltered`, not inside `planExtractSignals`. Both surfaces already route through the wrapper, so one choke point covers the CLI and MCP lanes, the core plan contract gains no second non-throwing outcome class, and the gate runs before the prefilter so a tombstoned session spends no decision-model requests. The outcome type lives beside the prefilter's `skipped` outcome; the suppression reason widens into a typed union of literals (the prefilter reasons plus the tombstone reason) so there is one "no envelope, and why" channel, and MCP and CLI outputs stay symmetric. Every suppression carries the recorded breakdown as evidence; an empty plan that reads as a clean run is the failure the existing refusals were shaped to avoid.+- Tombstone-write failure on an otherwise-successful commit surfaces as a named field on `CommitExtractedSignalsResult` and in both surface outputs, never a throw and never a silent drop. The signals are already on disk, so a throw would misreport a successful write as a failure, while swallowing would hide a ledger that is no longer trustworthy.+- Plan-to-commit race semantics are accepted and documented rather than engineered around. The commit recomputes the mined-content hash from the turns present at commit time; an import landing between plan and commit can record a tombstone over content the model never saw, which is wrong only for the never-mined tail and self-heals on the next edit. The window is one caller turn wide.+- The undistilled census is derived vault-side from the two ledgers joined on `sessionRefIdentity`, with no log re-hashing inside the hook path. Sessions the plan refuses are excluded because they can never be distilled; counting them would nag forever about unparsable material. The hook answers from this census; if the budget runs out it exits 0 silently.+- The SessionStart nudge is default-off and change-keyed. The card's fixed-interval wording is superseded: an interval that re-nags an unchanged backlog repeats the nag the card exists to kill, so the throttle is once per unchanged backlog, held in a vault-global, overwrite-only, size-one state file beside hygiene-digest.hash (the shipped precedent for vault-wide state whose lifetime is not per-session). The hook follows gap-agenda: flag via `resolveConfigFlag` with an env override, process ceiling, fail-open exit 0, byte-identical preamble when off.+- The ask-once batching layer is a pure core function; the kernel stays LLM-free and never runs an interactive loop. Dedup happens before the ask, not before the write: candidates whose `computeDedupHash` is already in the vault index are dropped from the ask and named in a report, the remainder is grouped under one proposal per distinct dedup hash across sessions, and the single batched envelope is emitted through the import-session verb for the calling agent to present.+- The claim-chain gate's default is off, and measure-only (shadow) is the default mode whenever the gate is enabled without an explicit mode. Resolution order: explicit `search_trust_gate_mode` wins (invalid values refused naming the key, `parseFusionMode` precedent); absent the mode key, the legacy `search_trust_gate_enabled` boolean maps true to shadow and false to off; the default with neither set is off. Upstream's shadow-on-every-install is not copied - every recall knob in Open Second Brain defaults off with an explicit enable, and shadow is non-destructive precisely so enablement can be the first step. A vault that wants today's excluding behavior names `enforce` explicitly; the explain envelope names the resolved mode so the change is visible. There is no per-call override this wave.+- Shadow mechanics add no verdict kind to the kernel. `applyRankAdjusters` keeps its single-pass `keep | multiply | exclude` contract; under shadow the mode-aware gate reuses `trustGateAdjuster`'s classification and returns keep for every item while the would-be refusals flow into the receipt build beside the existing flag-gated receipt path in post-rank. Only `off` is byte-identical: shadow adds receipt fields by design, so the query-cache key includes the resolved mode and the explain envelope reports it.+- Edge eligibility is decided at the arm boundary, not inside the fan-out. Each walked node is classified from the same controlled-vocabulary frontmatter signals as the page gate; an edge passes when both endpoints pass, its validity window covers query time (`parseValidityWindow`, the `closedSupersessionTip` grammar) and its polarity is not a counted exclusion (the existing relation-polarity vocabulary, not upstream's claim-status integers). Eligibility refusals are reportable - removed from the contribution set and counted with named structural reason tokens - which is a different act from the relational arm's identical-to-absent provenance convention, and the design keeps the two conventions distinct.+- Refusals surface as an extension of the two existing receipts, not a third receipt type and not a table. `excluded` keeps meaning actually dropped; shadow would-be counts and arm-level refusals are reported beside it, pool and arm reported separately so neither lane's numbers blur.+- The console payload is delivered by client-side fetch, keeping the template's single substituted marker. The graph stays server-side injected through `__GRAPH_JSON__` (the export mode's single-file property survives); the counts and ops sections fetch `/counts.json` and `/status.json` per render, which makes live refresh a re-fetch and a re-read. Two routes, of a piece with the explorer's existing two-route shape. The written-per-day window is a named 30-day constant, not a query parameter. Each `/status.json` GET re-runs the dry-run dream inside `buildOperatorSnapshot` - the most expensive read in the snapshot - and the design accepts that bounded, read-only cost rather than introducing a cache that would break the re-read-per-GET invariant and go stale silently.+- Premise corrections recorded from the recons (the design follows the recon wherever a recon corrects a card):+ - t_93617709 premise confirmed, with a keying correction: the discover ledger's content hash runs over source-log file bytes keyed by absolute machine paths, and there is no join back to the mining lane, so the tombstone hashes the mined turn material itself. Anchors drifted: `planExtractSignals` :340 to :443; the refusal block :347-380 to :449-479; `CommitExtractedSignalsResult` :229-241 to :244-269.+ - t_0358af42 premise partly false: vault-wide cross-session dedup already ships (buildDedupIndex over inbox, processed and archived, applied in `commitExtractedSignals` before the durability gate), and the machine-wide session census already ships (discoverSessions plus the content-hash import ledger, built for cheap reruns). The residue is the distillation keying, the SessionStart surface and the ask-once layer. The hook premise is stale in detail too: SessionStart wires three hooks (active-inject, opt-in gap-agenda, session-capture), none an undistilled count. Anchors drifted: buildDedupIndex :55-70 to :98; `commitExtractedSignals` :433-455 to :578; the SessionStart block :27 to :43-67.+ - t_05a0eaa4 premise partly false: a recall-time eligibility gate already ships for pages (`trustGateAdjuster` mounted in post-rank, config-gated, receipted, byte-identical when off), so the missing work is exactly the measure-only middle mode, per-edge eligibility on the relational arm and the refusals record. The card's own anchors were wrong and stay wrong: store/state.ts:212 is a docblock region (the real schema-mismatch refusal is store/lifecycle.ts:229 and :243) and cli/search/verbs/check.ts:69 is an import line (the verdict mechanism is recordVerdict at :289). Upstream's shadow-on-every-install default is rejected.+ - t_15ebcfc1 premise false: `buildLiveServer` (src/core/brain/explorer.ts:354, the card said :355) is exactly the local console surface - loopback-only, read-only, re-reads the vault per GET - and the interactive-graph half the card called missing already ships over the rule graph. The residue is the live-counts half plus the ops route. The card also conflated two stores: report-snapshot writes per-day JSON under Brain/reports, while the Brain/metrics JSONL contract is the separate metrics.ts module; this design reads the report-snapshot dates only.++## File changes++New files:++- src/core/brain/session-mining-ledger.ts - the shared per-session mining record (read, record, gate, prune, lock, atomic write, mined-turn hash helper).+- src/core/brain/sessions/mining-census.ts - the distilled/undistilled census over the mining record joined to the import ledger.+- src/core/brain/distill/backlog-proposals.ts - the ask-once batching pure function.+- hooks/backlog-nudge.ts - the default-off SessionStart count hook.+- hooks/lib/backlog-nudge-state.ts - the vault-global, overwrite-only, change-keyed throttle state.+- src/core/brain/explorer-console.ts - the console collector (telemetry summary, most-recalled top-N, per-day histogram, report-snapshot dates).+- tests/core/brain/session-mining-ledger.test.ts, tests/core/brain/sessions/mining-census.test.ts, tests/core/brain/distill/backlog-proposals.test.ts, tests/hooks/backlog-nudge.test.ts, tests/core/brain/explorer-console.test.ts.++Modified files:++- src/core/brain/extract-signals.ts - record-at-commit on the tombstone trigger; the ledger-failure field on `CommitExtractedSignalsResult`; the mined-turn hash call.+- src/core/brain/extract-signals-prefilter.ts - the plan gate ordered before the prefilter; the widened typed suppression outcome.+- src/mcp/brain/extract-tools.ts - suppressed-plan response fields, tool description, force argument.+- src/cli/brain/verbs/extract-signals.ts - suppressed-plan output, ledger-failure line, force flag.+- src/core/config.ts - the backlog nudge flag resolver beside `resolveGapLoopEnabled`.+- hooks/hooks.json - one SessionStart entry.+- src/cli/brain/verbs/import-session.ts - distilled/undistilled per runtime in the coverage output; the propose-backlog flag emitting the batched envelope.+- src/core/brain/trust/retrieval-gate.ts - the mode-aware gate reusing the shipped classification.+- src/core/brain/trust/retrieval-receipts.ts - the refusals section on both receipts.+- src/core/search/pipeline/post-rank.ts - mount the mode-aware gate; build receipts from the shadow outcome.+- src/core/search/pipeline/relational-arm.ts - the edge-level eligibility adjuster.+- src/core/search/types.ts, src/core/search/index.ts - the mode type and its resolution beside `search_trust_gate_enabled`.+- src/core/search/pipeline/cache-slot.ts, src/core/search/explain-envelope.ts - the mode in the cache key and the envelope.+- src/core/brain/explorer.ts - the two new GET routes; the additive options object carrying configPath.+- templates/brain-explorer.html - the counts and ops sections fetched client-side.+- src/cli/brain/verbs/explorer.ts - thread configPath; boot banner mentions the new routes.+- tests/core/config.test.ts, tests/core/brain/extract-signals.test.ts, tests/core/brain/extract-signals-prefilter.test.ts, tests/cli/brain-extract-signals.test.ts, tests/mcp/extract-signals-tool.test.ts, tests/cli/session-discovery.test.ts, tests/hooks/hooks-json-shape.test.ts, tests/core/brain/trust/retrieval-gate.test.ts, tests/core/brain/trust/retrieval-receipts.test.ts, tests/core/search/retrieval-trust-gate.test.ts, tests/core/search/relational-arm.test.ts, tests/core/search/config.test.ts, tests/core/brain/explorer.test.ts, tests/cli/brain-explorer.test.ts - extended per the plan.+- docs/cli-reference.md, docs/mcp.md, docs/how-it-works.md - the new surfaces documented on their existing entries.++## Risks and open questions++- SessionStart budget: the hook entry allows 10 s and discoverSessions' own docblock measures about 6 s cold over a large fixture, which is why the census is answered from the vault-side ledgers with no log hashing, and the hook degrades to silence rather than sweeping under deadline. The hook test asserts the fail-open path.+- The mining record must stay size-bounded: entries are tens of bytes, pruning removes sessions whose turns no longer exist, and the v1.85.0 doctor checks name derived-ledger growth regardless.+- Only `off` is byte-identical on the recall lane. Shadow changes the outcome shape by adding receipt fields; the cache key and the explain envelope must both carry the resolved mode or a cached off result could be served for a shadow query.+- The tombstone race window (plan and commit observing different turn sets) is accepted and documented; if it ever matters in practice, the recorded turn count is the additive remedy, and it is recorded in the entry breakdown from the start.+- The relational arm's identical-to-absent provenance convention and the reportable-refusal convention now coexist in one file; the arm tests must pin the distinction (a refusal is counted, a provenance omission is not reported).+- The explorer's per-GET dry-run dream cost is accepted and must be stated in the route's docs; if it ever needs relief, the remedy is an explicit operator action, not a silent cache.+- The config-template ratchet only fires for keys added to the resolver default tables; both new keys ride `resolveConfigFlag` and the `envOrConfig` path beside their existing siblings, so no template entry is expected - the ratchet test stays the guard.diff --git a/docs/brainstorm/session-mining-economy/plan.md b/docs/brainstorm/session-mining-economy/plan.mdnew file mode 100644index 00000000..8aaa524e--- /dev/null+++ b/docs/brainstorm/session-mining-economy/plan.md@@ -0,0 +1,83 @@+# Session mining economy - implementation plan++Four implementation lanes (A to D) plus one integration task (I). The two substrate tasks land first (Task 1, lane A: the shared mining record; Task 2, lane B: the config flag surface) so every lane can start without waiting on another lane. File ownership is disjoint across lanes with one noted exception: docs/cli-reference.md is edited by several lanes, each in its own section, landing in task order on the shared branch. Every task is test-driven: the named tests are written first and must fail with the expected error before implementation.++## Tasks++### Task 1: Session mining ledger module+- **Lane**: A+- **Files**: src/core/brain/session-mining-ledger.ts (new), tests/core/brain/session-mining-ledger.test.ts (new)+- **Acceptance**: With the failing tests written first: the ledger stores schema-versioned entries {sessionId, minedContentHash, outcome {written, staged, deduped, durabilityRejected}, recordedAt} in the derived store beside session-import-ledger.json; reading a corrupt or unknown-schema ledger refuses with a named error and never resets; writing takes the file lock with the vault-identity assertion, writes atomically, is a byte-identical no-op when content is unchanged, and prunes entries whose sessions have no turns; the mined-turn hash helper is a stable SHA-256 over (turn id, text, timestamp) and changes when any turn changes.+- **Depends on**: none++### Task 2: Backlog nudge config flag+- **Lane**: B+- **Files**: src/core/config.ts, tests/core/config.test.ts+- **Acceptance**: With the failing test written first in tests/core/config.test.ts: a `resolveBacklogNudgeEnabled(configPath?)` resolver beside `resolveGapLoopEnabled` rides `resolveConfigFlag` with env override `OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED` and config key `backlog_nudge_enabled`, defaulting false; the config-template ratchet test stays green (the key rides the flag path, not the resolver default tables).+- **Depends on**: none++### Task 3: Zero-yield tombstone gate at plan time+- **Lane**: A+- **Files**: src/core/brain/extract-signals-prefilter.ts, tests/core/brain/extract-signals-prefilter.test.ts, tests/core/brain/extract-signals.test.ts+- **Acceptance**: With the failing tests written first: inside `planExtractSignalsPrefiltered` and before the prefilter runs, a mining record whose hash matches the session's current turn material suppresses the plan with a typed outcome carrying the tombstone reason and the recorded breakdown as evidence; the reason union widens to typed literals without disturbing the prefilter's existing `skipped` reasons; unchanged content stays suppressed, content changed by re-import re-opens the session; the three existing plan refusals behave exactly as before.+- **Depends on**: Task 1++### Task 4: Record-at-commit with a named ledger failure field+- **Lane**: A+- **Files**: src/core/brain/extract-signals.ts, tests/core/brain/extract-signals.test.ts+- **Acceptance**: With the failing tests written first: a successful `commitExtractedSignals` with `written.length === 0 && staged === 0` records the ledger entry with the outcome breakdown; a commit with written or staged nonzero records nothing; an `ExtractSignalsWriteError` records nothing; the commit recomputes the mined-content hash from the turns present at commit time; when the commit succeeds but the ledger write fails, the result carries a named ledger-failure field and nothing throws.+- **Depends on**: Task 1++### Task 5: Tombstone wiring on both surfaces plus the re-mine escape hatch+- **Lane**: A+- **Files**: src/mcp/brain/extract-tools.ts, src/cli/brain/verbs/extract-signals.ts, tests/mcp/extract-signals-tool.test.ts, tests/cli/brain-extract-signals.test.ts, docs/cli-reference.md (extract-signals entry), docs/mcp.md (brain_extract_signals entry), docs/how-it-works.md (mining section)+- **Acceptance**: With the failing tests written first: a suppressed plan renders symmetrically on the MCP tool (response fields and tool description naming the tombstone reason and evidence) and on the CLI verb (output line and JSON fields); the ledger-failure field surfaces by name on both; the CLI force/re-mine flag and the documented MCP force argument bypass the gate for exactly one plan call and change no other behavior; docs state the invalidation semantics (content change re-opens; force for "I removed the downstream item and want it re-proposed").+- **Depends on**: Tasks 3, 4++### Task 6: Distilled and undistilled census+- **Lane**: B+- **Files**: src/core/brain/sessions/mining-census.ts (new), tests/core/brain/sessions/mining-census.test.ts (new), src/cli/brain/verbs/import-session.ts, tests/cli/session-discovery.test.ts+- **Acceptance**: With the failing tests written first: the census joins the mining record to the import ledger on `sessionRefIdentity` with no log re-hashing and classifies each imported session as distilled (yield on current content), known-nothing-new (tombstoned, content unchanged) or undistilled (never mined, or content changed since last mining); sessions the plan would refuse (boundary, zero imported turns, zero user turns) are excluded from the backlog; the import-session coverage output reports distilled and undistilled counts per runtime; a corrupt mining record refuses by name.+- **Depends on**: Tasks 1, 2++### Task 7: SessionStart backlog nudge hook+- **Lane**: B+- **Files**: hooks/backlog-nudge.ts (new), hooks/lib/backlog-nudge-state.ts (new), hooks/hooks.json, tests/hooks/backlog-nudge.test.ts (new), tests/hooks/hooks-json-shape.test.ts+- **Acceptance**: With the failing tests written first: the hook is a no-op exiting 0 with a byte-identical preamble when the flag is off; when on and the undistilled count differs from the last injected count, it injects one `additionalContext` line with the count; when the count is unchanged it stays silent; the throttle state is a vault-global, overwrite-only, size-one record beside the hygiene-digest state (path from path constants, never hard-coded); every path exits 0 and respects the process ceiling; the hooks.json SessionStart entry passes the shape test.+- **Depends on**: Tasks 2, 6++### Task 8: Ask-once batching layer+- **Lane**: B+- **Files**: src/core/brain/distill/backlog-proposals.ts (new), tests/core/brain/distill/backlog-proposals.test.ts (new), src/cli/brain/verbs/import-session.ts, tests/cli/session-discovery.test.ts+- **Acceptance**: With the failing tests written first: the pure function takes the pending undistilled set plus a `loadDedupIndex` result and returns one proposal per distinct `computeDedupHash` across sessions, each naming its contributing sessions, plus a report naming every candidate dropped because its dedup hash already exists in the vault; sessions raising only already-known material appear in the report and in no proposal; the import-session verb exposes the batched envelope as output for the calling agent to present, with no interactive loop in the kernel; the function handles an empty backlog as an explicit empty result.+- **Depends on**: Task 6++### Task 9: Measure-only shadow mode on the retrieval trust gate+- **Lane**: C+- **Files**: src/core/search/types.ts, src/core/search/index.ts, src/core/brain/trust/retrieval-gate.ts, src/core/brain/trust/retrieval-receipts.ts, src/core/search/pipeline/post-rank.ts, src/core/search/pipeline/cache-slot.ts, src/core/search/explain-envelope.ts, tests/core/search/config.test.ts, tests/core/brain/trust/retrieval-gate.test.ts, tests/core/brain/trust/retrieval-receipts.test.ts, tests/core/search/retrieval-trust-gate.test.ts+- **Acceptance**: With the failing tests written first: the resolved config carries `search_trust_gate_mode` (off | shadow | enforce), resolved as explicit mode, else legacy boolean true mapped to shadow, else off; an invalid value is refused naming the key after the `parseFusionMode` pattern; under off, ranking and outcome shape are byte-identical to today; under shadow, `trustGateAdjuster`'s classification runs, nothing is excluded, and every would-be refusal is recorded with its structural reasons into receipts whose `excluded` counts stay untouched; the query-cache key includes the resolved mode; the explain envelope names it; enforce reproduces today's excluding behavior exactly.+- **Depends on**: none++### Task 10: Edge-level eligibility adjuster in the relational arm+- **Lane**: C+- **Files**: src/core/search/pipeline/relational-arm.ts, tests/core/search/relational-arm.test.ts+- **Acceptance**: With the failing tests written first: each walked node and edge is classified from the same controlled-vocabulary frontmatter signals as the page gate plus `parseValidityWindow` at query time and the relation-polarity vocabulary; under enforce an ineligible edge is removed from the arm's contribution set, under shadow it is counted only; refusals carry named structural reason tokens and are reported per lane into the receipts, distinct from both `excluded` and the arm's identical-to-absent provenance convention (a refusal is counted, a provenance omission is not reported); traversal internals of relational-fanout.ts are unchanged.+- **Depends on**: Task 9++### Task 11: Console collector and explorer routes+- **Lane**: D+- **Files**: src/core/brain/explorer-console.ts (new), src/core/brain/explorer.ts, src/cli/brain/verbs/explorer.ts, tests/core/brain/explorer-console.test.ts (new), tests/core/brain/explorer.test.ts, tests/cli/brain-explorer.test.ts+- **Acceptance**: With the failing tests written first: the collector returns a frozen, deterministic payload (given `opts.now`) holding the telemetry summary, the most-recalled top-N under a named cap, the written-per-day histogram over a named 30-day window bucketed from continuity createdAt, and the latest report-snapshot date per surface reported as "disabled" when snapshots are off; `buildLiveServer` serves `/counts.json` from the collector and `/status.json` from `buildOperatorSnapshot`, both loopback-only, re-read per GET, with errors surfaced by name and no write endpoints; `configPath` reaches the server through an additive options object and the existing boot refusals are unchanged.+- **Depends on**: none++### Task 12: Console sections in the explorer template+- **Lane**: D+- **Files**: templates/brain-explorer.html, tests/cli/brain-explorer.test.ts+- **Acceptance**: With the failing tests written first: the template gains counts and ops sections that fetch `/counts.json` and `/status.json` client-side and render them in the template's existing vanilla-JS structure with no new dependencies; the `__GRAPH_JSON__` marker still appears exactly once and the single-marker validation still passes; the live spawn test fetches all four routes on loopback and asserts the shutdown behavior is unchanged.+- **Depends on**: Task 11++### Task 13: Wave integration verification+- **Lane**: I+- **Files**: none expected; cross-document consistency fixes in docs/cli-reference.md, docs/mcp.md, docs/how-it-works.md if review finds drift+- **Acceptance**: The full test suite passes; the cross-lane chain holds end to end: a barren commit records the entry (Task 4), the plan gate suppresses the next scan (Task 3), the census classifies the session known-nothing-new (Task 6) and the hook count excludes it (Task 7); a tombstoned session spends no decision-model requests on either surface; the recall lane with every new knob unset is byte-identical to the merge base; the console renders from the shipped stores with every knob off; the three docs files read as one coherent story across the four lanes; fmt and lint are green on the whole tree.+- **Depends on**: Tasks 5, 7, 8, 10, 12diff --git a/docs/brainstorm/session-mining-economy/variants.md b/docs/brainstorm/session-mining-economy/variants.mdnew file mode 100644index 00000000..1e6eb4fd--- /dev/null+++ b/docs/brainstorm/session-mining-economy/variants.md@@ -0,0 +1,34 @@+# Variants - session mining economy++Consultant output (docs/brainstorm/session-mining-economy/cli-output/claude.md) reproduced verbatim, followed by the orchestrator decision.++### Variant 1: Seam extension - one ledger, one mode, one server+- **Approach**: Every card becomes a strict extension of a mechanism Open Second Brain already ships. The session import ledger in `src/core/brain/sessions/discover.ts` gains a mining outcome per entry (a schema version bump): the same `hashFile` content hash that already keys the ledger keys the tombstone, so a changed session stops matching and re-mines for free, `recordSessionImports`' locked atomic write and vanished-path pruning carry over, the write happens only on the `commitExtractedSignals` path when zero items were written (keeping the deduped already-known and durability-rejected barren outcome classes apart), and `planExtractSignals` consults the entry before building the envelope, answering a typed suppressed verdict rather than an `ExtractSignalsError` so an adjudicated-barren session is not reported as a failure. The recall gate becomes a versioned `off | shadow | enforce` knob beside `retrievalTrustGateEnabled` in `ResolvedRecallConfig` (shadow the shipping default), realized as an edge-level adjuster inside `runRelationalArm` that reuses `trustGateAdjuster`'s rank-adjust verdict vocabulary and classifies each `RelationalPathStep` edge deterministically - grounding band from `truth/grounding.ts`, query-time validity via `parseValidityWindow` (the same grammar `closedSupersessionTip` already reads), relation polarity, asserted provenance from the claim ledger - counting refusals under fixed reason tokens like `RETRIEVAL_TRUST_EXCLUSION_REASON`, with ungrounded claims reported into a durable extraction-refusals record following the `UngroundedStatedClaim` precedent in `truth/stated-claims.ts`. The console is new routes on `buildLiveServer` (live counts from `discoverSessions` coverage and the mining ledger, recall views over `recall-telemetry.ts` and `report-snapshot.ts` deltas, an ops route over the brain status snapshot), keeping the loopback-only, read-only, per-GET re-read invariants, and the SessionStart nudge is a throttled count over the ledger beside the existing `hooks/hooks.json` injection with a last-offered stamp in the derived store.+- **Trade-offs**:+ - Pros: honors the reuse constraint literally (`trustGateAdjuster`, not a parallel gate); tombstone invalidation falls out of the content hash the ledger already computes; one state file, one lock, one serialization idiom per concern; byte-identical off paths are the existing gates' own contract, and the cache-slot already records whether the gate ran so a shadow receipt cannot be served as an enforced result.+ - Cons: a schema version bump on a shipped ledger is the wave's riskiest single edit, since `readSessionLedger` hard-refuses unknown schemas and every reader must move together; `planExtractSignals` gains a second class of non-error outcome the three existing refusal throws do not anticipate, so the suppressed verdict needs its own typed shape; mining state lives on a ledger whose discovery report is contractually read-only, so the write must stay on the commit path and nowhere else.+- **Complexity**: medium+- **Risk**: low++### Variant 2: Receipt-persisted economy+- **Approach**: Treat the wave's cross-run state as persisted receipts rather than a new ledger. The receipt vocabulary Open Second Brain already emits - `retrieval_decision_trace`, `memory_trust_assessment`, reconciliation reports, dead letters - gains durable per-run persistence in the derived store: a mining commit appends a receipt carrying its outcome counts, the shadow gate appends per-query refusal counts with reasons, and each distill proposal appends an asked-once record. `planExtractSignals` then consults accumulated receipts for the session's current content hash, the console becomes a viewer over the receipt store, and the SessionStart count and its throttle read receipt aggregates.+- **Trade-offs**:+ - Pros: one observation channel for everything the wave surfaces, so console, counts and refusals render exactly what the flows already emit; no schema change to any shipped ledger; ask-once batching needs no separate state because a proposal is itself a receipt; the shadow mode's measure-only discipline is the receipts' native shape.+ - Cons: receipts are per-response accounting by contract, so deriving tombstone eligibility from them conflates observation with decision state; retention or compaction becomes a correctness knob, where a pruned receipt silently re-admits a barren session; the plan gate must aggregate many receipts to answer one boolean, paying a scan per plan that a single keyed entry avoids; per-query shadow counts accumulate without bound until a policy exists.+- **Complexity**: medium+- **Risk**: medium++### Variant 3: Mining-economy journal+- **Approach**: One append-only journal for the whole session-mining economy: mining commits, gate classifications, distill proposals and throttle decisions each append typed events keyed on session content hash, and derived projections answer the wave's four questions - tombstone eligibility, undistilled counts, gate enforcement state, console views. Enforce mode reads the projection at query time, the explorer renders the same projections, and a compaction policy folds event history into snapshot rows.+- **Trade-offs**:+ - Pros: the strongest audit story, since every suppression and refusal is a replayable event carrying its reason; projections give the console and the SessionStart count one reader; future economy rules (per-session caps, spend accounting) become new projections over existing events rather than new stores.+ - Cons: introduces an event-sourcing substrate - append, compaction, projection rebuild - that Open Second Brain does not have, where every existing store is a locked read-modify-write snapshot; per-query gate events force a rotation policy that is itself an unresolved design question; the largest surface against the one-PR 40-60 file envelope and the hardest to hold to byte-identical off behavior.+- **Complexity**: large+- **Risk**: high++### Recommended: Variant 1+**Rationale**: The validator-settled scope already names these exact seams - reuse `trustGateAdjuster` rather than build a parallel gate, key the tombstone on the hash the session ledger already computes, and fold the console into `buildLiveServer` - so Variant 1 is the settled scope rendered as code, within the one-PR envelope and with the smallest deviation from shipped behavior. Variant 2 makes a barren session's eligibility depend on receipt retention policy, and Variant 3 buys an event-store substrate this wave neither needs nor could land safely.++## Orchestrator decision++Agreed: Variant 1. The recon-verified rescopes for all four cards already name Variant 1's exact seams (reuse `trustGateAdjuster`, content-hash keying, the console folded into `buildLiveServer`), so the recommendation renders the settled scope as code with the smallest deviation from shipped behavior; Variant 2 is declined because receipt retention must never decide a barren session's eligibility, and Variant 3 because the project has no event-store substrate and this wave cannot land one safely. One mechanism correction, adopted from the recon over the variant's text as written: the shared mining record is a new separate ledger keyed by the portable sessionId plus a SHA-256 over the mined turn material, not a schema version bump of session-import-ledger.json, because the import ledger's reader silently erases undeclared fields on rewrite, its keys are absolute machine paths with no join back to the mining lane, and the wave's riskiest single edit is an edit the design can simply not make. A second correction per the recon: upstream's shadow-as-shipping-default is not copied - the gate defaults off, and shadow is the default mode only when the gate is enabled without an explicit mode, keeping the repository's every-recall-knob-defaults-off discipline.diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex 87b69f98..2d64898f 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -439,7 +439,7 @@ o2b brain upgrade (CLI-only) Migrate release-owned files forward (_bo2b brain export Read-only dump of active preferences, or (since v1.50.0) a session-transcript dataset: --format json|llms-txt|transcripts-jsonl [--out <path>] [--force]; the transcript form takes --transcripts <file|dir> and reads no vault at all (see "The transcript corpus" below). Since v1.49.0 the preference bytes pass the shared egress redactor (stderr carries a notice when anything was removed), and a `pref-*.md` the parser cannot read is REFUSED, not skipped: exit 1 naming every unreadable file in one run rather than exit 0 over a shorter list. `o2b brain doctor` reports the same fileso2b brain bank-export (CLI-only) One-file backup bundle: preferences, the page graph, page contracts, the sources dashboard. Redacted on the way out; refuses with exit 1 on an unreadable `pref-*.md` (since v1.49.0)o2b brain bank-import (CLI-only) Restore a bank bundle (--mode skip|overwrite|merge) [--trusted-restore]. Preference rows restore UNTRUSTED by default: every row lands `unconfirmed`, unpinned, at low confidence, on a fresh trial window dated from the restore - including a row the bundle already marked unconfirmed - and the run names each row it reset; `--trusted-restore` keeps the carried status, confidence, pin and window verbatim, for a backup you vouch for. A malformed preference already in the DESTINATION does not abort the import: the rows restore and the run prints `topic-key check incomplete: <path>` for each rule the topic-collision scan could not read, because that list is then a partial answer (since v1.49.0)-o2b brain explorer (CLI-only) Force-directed HTML graph of Brain/preferences + retired; live HTTP on 127.0.0.1 or --export <path> single-file. Keyboard-accessible listbox + localStorage layout persistence. Double-click a node to open it in Obsidian (live mode). Since v1.49.0 BOTH modes refuse a Brain file they cannot parse - `--export` writes nothing and exits 1, and live mode does not start, because a browser showing a silently smaller graph is the same lie as a written file - and `--export` runs the shared redactor over the graph, so its output differs from a pre-v1.49.0 export on any vault holding a credential-shaped value+o2b brain explorer (CLI-only) Force-directed HTML graph of Brain/preferences + retired; live HTTP on 127.0.0.1 or --export <path> single-file. Keyboard-accessible listbox + localStorage layout persistence. Double-click a node to open it in Obsidian (live mode). Since v1.49.0 BOTH modes refuse a Brain file they cannot parse - `--export` writes nothing and exits 1, and live mode does not start, because a browser showing a silently smaller graph is the same lie as a written file - and `--export` runs the shared redactor over the graph, so its output differs from a pre-v1.49.0 export on any vault holding a credential-shaped value. Since v1.86.0 live mode serves two read-only console routes beside the graph, `/counts.json` (recall-telemetry summary, most-recalled artifacts, a written-per-day histogram over the trailing 30 days, per-surface report-snapshot dates that read as disabled while the feature is off) and `/status.json` (the operator snapshot), both re-reading the vault on every GET, and the page renders Counts and Vault status sections from them; a static export states that the live routes are unreachable instead of firing fetches that can only failo2b brain doctor Check Brain-specific invariants (status-vs-folder, broken wikilinks, ...). --remediate [--dry-run] plans a dependency-ordered repair and applies auto-safe content-hash re-stamps. Since v1.81.0 the report also prints a count-only archive section over Brain/.archives/ (entries, combined bytes, stamped pages whose entry is missing or mismatched, entries no page cites; silent on a vault that never archived a source) and runs the fail-soft `blank-session-identity` check, counting continuity records still carrying the legacy blank session id; both are informational and neither repairso2b brain health Semantic-health report (since v0.14.0): contradictory confirmed preferences, recurring concepts with no dedicated preference, stale claims, plus a clean | watch | investigate verdicto2b brain history Render a preference's edit-history timeline (since v0.14.0): one entry per content mutation (principle / scope / status before -> after)@@ -461,7 +461,7 @@ o2b brain mcp-landscape List MCP servers configured across the vault (sinco2b brain scan-inline Capture `@osb` markers from folders listed under `notes.read_paths` in _brain.yamlo2b brain today Read-only today dashboard: due and overdue obligations, open loops (`@osb loop <text>` markers, closed by `@osb loop close id=<id>`), merged recent activity, totals; --lookback-days (default 7), --limit (default 20), --json. Each section derives live; a failing section is reported and the rest still rendero2b brain apply-markers Turn `@osb set note=<target> field=<field> value=<value>` markers into schema-validated frontmatter writes. Report mode by default (writes nothing); --apply needs `guardrails.marker_writeback` in _brain.yaml, consumes applied markers so a re-run is idempotent, and leaves unresolvable or refused targets unconsumed; --path <subdir> (repeatable) or notes.read_paths-o2b brain import-session Replay signals from a registered agent session .jsonl (or directory); --recall also stores turns in the session recall DAG. Since v1.50.0 the path argument has a machine-wide sibling: --status reports per-runtime coverage, --discover lists what has never been imported, --discover --all imports it (see "Session logs this machine already has" below)+o2b brain import-session Replay signals from a registered agent session .jsonl (or directory); --recall also stores turns in the session recall DAG. Since v1.50.0 the path argument has a machine-wide sibling: --status reports per-runtime coverage, --discover lists what has never been imported, --discover --all imports it, and --propose-backlog batches the already-imported undistilled backlog into one ask-once distill envelope (see "Session logs this machine already has" below)o2b brain session-hook Internal hook bridge: read one lifecycle payload from stdin, capture prompt markers / brain_feedback, append lifecycle audit/log rowso2b brain context-receipts List/show opt-in prompt context receipt continuity records (since v0.29.0)o2b brain recall-telemetry List/summarise opt-in recall telemetry continuity records (since v0.29.0)@@ -655,7 +655,7 @@ o2b brain maintenance run --cron-template [--interval N] [--format cron|The schema pack gains four additive ontology fields (`labels`, `link_constraints`, `attributes`, `frontmatter_tiers`) with audited mutations through `o2b brain schema apply`. Link constraints enforce at index materialization: a typed edge whose endpoint page types violate the declared pairs falls back to an untyped link, `o2b brain schema lint` lists each violation, and removing the constraint restores the edges on the next index run. Tier drift detection rides the same index pass - the snapshot keeps the expected value, so reindexes never absorb a hand-edit, and `brain_doctor` warns with the open count. Filter labelled recall with `o2b search <q> --property labels=<dim>/<value>`. Secrets protect against context leakage and vault sync exposure, not against root; every custody operation lands a no-values record in `Brain/log/secret-custody/`. A maintenance gate skip exits 0 so cron never alarms on a quiet hour.-The lane's vault-side knobs live in `Brain/_brain.yaml` - the two original ones under `maintenance:`, the storage cap and its thresholds under `storage:` (since v1.85.0, see "Storage lifecycle" below) - because they answer what the cron line cannot. `host_pressure_percent` adds a gate over the window and busy checks: skip when the host's one-minute run queue stands at or above that percentage of the CPUs this process may use. Unset by default, which leaves the gate off. Where the metric is degenerate - a platform whose load average is a constant, or a cgroup with a CPU bandwidth quota, where the run queue is the whole host's - the gate stays **open** and the journal carries a separate `pressure:unmeasurable` row naming the reason, so an unreadable host is never reported as a quiet one. Since v1.85.0 a configured storage cap adds the lane's fifth gate, evaluated last because it is the most expensive: a fresh byte census of the whole vault, skipping the pass at a measured critical tier with its own `skipped:capacity` verdict, the journal row carrying the tier and the reading behind it while the skipped run still appends its usage sample first. A capacity skip is a gate skip like any other - it exits 0 - and where the caller has no local reach the `status` surface renders the tier word and the unmeasurable reason without the byte-and-percent figures. `failure_streak_limit` (default 3) refuses a lane task that has failed that many times in a row in the run journal, naming the streak; a single journaled success clears it. The refusal is per task - the other three still run under the same lease - and there are two ways past it: `--retry <task>` (repeatable) attempts just that task with every gate the operator configured still in force, and `--force` runs the whole lane past the four soft gates - window, busy, pressure and capacity - and every streak refusal, never past the lease. Both escapes exist on the MCP surface too - `brain_maintenance` takes `retry_tasks`, `busy_minutes`, `busy_threshold` and a `status` `limit`, so an agent reading the refusal can act on it without reaching for `force`, which widens its bypass to every soft gate and every streak refusal. A refused task is reported as `REFUSED`, never `FAILED`, and the run exits **7** rather than 1: nothing was attempted, so nothing failed, but a standing refusal is not the quiet hour that exits 0 either. An attempted failure still exits 1 and outranks a refusal in the same run. The `refused:streak` journal row carries the count it refused on, so the streak does not silently reset when its evidence rolls off the journal cap.+The lane's vault-side knobs live in `Brain/_brain.yaml` - the two original ones under `maintenance:`, the storage cap and its thresholds under `storage:` (since v1.85.0, see "Storage lifecycle" below) - because they answer what the cron line cannot. `host_pressure_percent` adds a gate over the window and busy checks: skip when the host's one-minute run queue stands at or above that percentage of the CPUs this process may use. Unset by default, which leaves the gate off. Where the metric is degenerate - a platform whose load average is a constant, or a cgroup with a CPU bandwidth quota, where the run queue is the whole host's - the gate stays **open** and the journal carries a separate `pressure:unmeasurable` row naming the reason, so an unreadable host is never reported as a quiet one. Since v1.85.0 a configured storage cap adds the lane's fifth gate, evaluated last because it is the most expensive: a fresh byte census of the whole vault, skipping the pass at a measured percent at or past `storage.pause_percent` (since v1.86.0; the line defaults to the warn band) or at the critical tier with its own `skipped:capacity` verdict, the journal row carrying the tier and the reading behind it, while every run the gate evaluated appends its usage sample - skip or proceed - so the day-based capacity trend keeps accumulating. A capacity skip is a gate skip like any other - it exits 0 - and where the caller has no local reach the `status` surface renders the tier word and the unmeasurable reason without the byte-and-percent figures. `failure_streak_limit` (default 3) refuses a lane task that has failed that many times in a row in the run journal, naming the streak; a single journaled success clears it. The refusal is per task - the other three still run under the same lease - and there are two ways past it: `--retry <task>` (repeatable) attempts just that task with every gate the operator configured still in force, and `--force` runs the whole lane past the four soft gates - window, busy, pressure and capacity - and every streak refusal, never past the lease. Both escapes exist on the MCP surface too - `brain_maintenance` takes `retry_tasks`, `busy_minutes`, `busy_threshold` and a `status` `limit`, so an agent reading the refusal can act on it without reaching for `force`, which widens its bypass to every soft gate and every streak refusal. A refused task is reported as `REFUSED`, never `FAILED`, and the run exits **7** rather than 1: nothing was attempted, so nothing failed, but a standing refusal is not the quiet hour that exits 0 either. An attempted failure still exits 1 and outranks a refusal in the same run. The `refused:streak` journal row carries the count it refused on, so the streak does not silently reset when its evidence rolls off the journal cap.Since v1.64.0 the lane's timeouts are honest and its embedding spend is named. A task killed at its safeguard deadline exits **6** (`probeIncomplete`), keyed on `timed_out` rows alone: the only proved fact is that the pass did not finish, which is not the same as a task failing; a proved failure (1) outranks a timeout, a timeout outranks a refusal (7), and the render says TIMED OUT with the safeguard detail. The reindex task stays keyword-only unless `maintenance_embeddings: true` (env `OPEN_SECOND_BRAIN_MAINTENANCE_EMBEDDINGS`, default `false`) opts it in; then it requests the embedding phase whenever the resolved semantic config can reach a provider, announces the pending-spend estimate before the pass, and journals the phase's own cost-gate result as the per-run receipt - model, tokens, estimated cost, and whether a bypass fired - on the task row, the journal line and the `maintenance_spend` metrics surface; the preview is an estimate by position, the receipt prices what the completed pass embedded, and a run killed mid-spend receipts nothing. `--force-cost` (MCP `force_cost`) bypasses a positive `embedding_cost_gate_usd` for this run, recorded on the receipt when it overrode a gate that would have refused. A safeguard timeout is journaled as `timed_out` and neither counts toward nor resets the failure streak that refuses a task.@@ -838,15 +838,21 @@ the audit line; because the search reads the same policy file, thedecision then ends in `error` with fault `retriever_failed` and nothingis injected until the file is fixed.-Search-side trust switches: `search_trust_gate_enabled` (env-`OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE`) zero-ranks quarantined material-(self-approval quarantine, untrusted-source provenance, entity-contamination) out of both semantic and lexical results and attaches the+Search-side trust switches: `search_trust_gate_mode` (env+`OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE`, `off | shadow | enforce`)+runs the retrieval trust gate over quarantined material (self-approval+quarantine, untrusted-source provenance, entity contamination) in both+semantic and lexical results: `enforce` zero-ranks it and attaches the`memory_trust_assessment` and `retrieval_decision_trace` receipts naming-every exclusion; `search_supersede_fade_enabled` (env-`OPEN_SECOND_BRAIN_SEARCH_SUPERSEDE_FADE`) makes an inbound `supersedes` /-`superseded_by` relation on a successor fade the unchanged superseded-note by a named multiplier. Both default off and leave ranking+every exclusion; `shadow` runs the same classification, excludes+nothing, and records every would-be refusal with its structural reasons+into the same receipts; `off` - the default - runs no gate. The legacy+`search_trust_gate_enabled` (env `OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE`)+stays as the bare enable: true maps to `shadow`, so excluding is named+explicitly with `enforce`; `search_supersede_fade_enabled` (env+`OPEN_SECOND_BRAIN_SEARCH_SUPERSEDE_FADE`) makes an inbound `supersedes`+/ `superseded_by` relation on a successor fade the unchanged superseded+note by a named multiplier. All default off and leave rankingbyte-identical when unset.### Knowledge intake and consolidation (since v1.36.0)@@ -860,8 +866,9 @@ o2b brain orphan-repair [--apply --confirm "apply orphan repair"] - detacho2b brain design-note <topic> [--payload <json> | --payload-file <path>] [--agent <name>] [--json] - the one-shot sibling of `o2b brain panel`. Without a payload it is read-only: it grounds the topic in the vault's tension records, decision records and truth projections and prints the single needs-llm-step envelope the calling agent answers, naming any store the vault holds nothing in (which is not the same as a store that matched nothing). With a payload it validates the written note and commits it as Brain/decisions/design-<date>-<topic>.md. The note must weigh named alternatives and mark EXACTLY ONE recommended: zero and two-plus are both refused, and the refusal states the count. A second note for the same topic on the same day is refused, never overwritteno2b brain skill-proposals page-candidates [--json] - read-only: gate the vault's user pages on the page-meta trio (core tier, non-stale lifecycle, high confidence) and an observed-reuse floor, skip any page an installed skill already covers, and return one needs-llm-step envelope per admitted page plus every skip with its reasono2b brain skill-proposals page-draft <page> (--payload <json> | --payload-file <path>) [--json] - validate a returned SKILL.md draft and stage it as a pending mature_page proposal INSIDE the vault; accept is what materializes the SKILL.md under the configured skills root, through the write-ahead journal-o2b brain extract-signals <session-ref> [--payload <json> | --payload-file <path>] [--agent <name>] [--json] - mine durable taste signals from an already-imported session's user turns. Without a payload it is read-only: it prints the turns it would mine and the single needs-llm-step envelope the calling agent answers. With a payload it validates the answer and writes the accepted items into Brain/inbox/ as speculative source_type: auto_extract signals, subject to the durability denylist and to Brain/pending/ staging when write approval is on. A payload over the per-session cap, an item below the confidence floor, or two items sharing a topic refuses the whole payload by name and writes nothing; a session with no imported turns is refused, never reported as empty+o2b brain extract-signals <session-ref> [--payload <json> | --payload-file <path>] [--force] [--agent <name>] [--json] - mine durable taste signals from an already-imported session's user turns. Without a payload it is read-only: it prints the turns it would mine and the single needs-llm-step envelope the calling agent answers. With a payload it validates the answer and writes the accepted items into Brain/inbox/ as speculative source_type: auto_extract signals, subject to the durability denylist and to Brain/pending/ staging when write approval is on. A payload over the per-session cap, an item below the confidence floor, or two items sharing a topic refuses the whole payload by name and writes nothing; a session with no imported turns is refused, never reported as emptyWith the optional decision-model turn pre-filter (use extract_prefilter, docs/decision-models/extract-prefilter.md) the --json plan may add turns_dropped, skipped { reason: decision_model_prefilter, turns_dropped } with llm_step: null (nothing to mine), and decision_model { degraded }; none appears with the use off. Payload items accept an optional string source_turn+ Since v1.86.0 a commit that ends with nothing written and nothing staged records a zero-yield tombstone for the session in the session mining ledger (<vault>/.open-second-brain/session-mining-ledger.json, beside the session import ledger, keyed on a SHA-256 of the mined turn material), and a plan over that session at unchanged content returns no envelope: --json carries skipped { reason: mining_tombstone, mining_outcome { written, staged, deduped, durability_rejected } } with llm_step: null and empty turns_mined, and text mode names the breakdown. --force bypasses the tombstone for exactly the one plan call it is given to - the remedy when a downstream preference was removed and the session should be re-proposed; a re-import that changes the session's turns re-opens it with no clearing step, and a corrupt mining ledger is refused by name rather than read as empty. A commit that succeeds but cannot record its tombstone still exits 0, naming the failure as mining_ledger_failure in --json and in textSince v1.74.0 every turns_mined entry carries timestamp, the turn's stored timestamp verbatim ("" when the source has none), and each prompt line reads [<turn-id> @ <timestamp>] <text>, or [<turn-id>] <text> for an undated turn - the run's clock is never substituted. The envelope instruction holds the items to five language-neutral rules: a time bound is written as an ISO 8601 date or interval resolved against the stating turn's timestamp (an end-only bound as an interval, because the dream pass reads a lone date as a start), conversational mechanics are skipped, one rule per item, a rule's conditions stay in its principle, and restatements are dropped. The dream pass turns the ISO bound into the preference's valid_from / valid_until through its existing temporal extraction; the extract lane writes neither field itself. Two items with the same topic refuse the payload under semantic_cross_item, the message naming both indices (items[i] and items[j] share topic "<topic>"), before any item is hashed or writteno2b brain deep-synthesis --json now also carries findings with causal_context, decomposed confidence (support, opposition, freshness, coverage), and the excluded_findings ledger with excluded_finding_count```@@ -922,6 +929,18 @@ once per change, folding warning- and action-severity findings intoper-detector counts and staying silent while the findings set isunchanged. Claude Code only: runtimes whose Stop block would force acontinuation turn stay silent.+`backlog_nudge_enabled` (env+`OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED`, default off) adds a+SessionStart line that reports the count of imported sessions never+distilled, read from a census that joins the session import ledger to+the per-session mining record without opening a source log. The nudge is+change-keyed, not timed: the line is injected when the count differs+from the count injected last (state: one overwrite-only file,+`.open-second-brain/backlog-nudge.count`), an unchanged backlog is never+re-nagged, and a drained backlog is recorded without being announced.+Under its deadline the hook degrades to silence, never to a fabricated+number; flag off keeps the session preamble byte-identical (since+v1.86.0).`o2b partner codegraph report` and `o2b doctor` aggregate codegraphstatus across every discovered code project, threading `project_path`per query when supported (feature-detected) and degrading with an@@ -1180,13 +1199,15 @@ roots `src/core/runtime/host-facts.ts` declares:o2b brain import-session --status Per-runtime found / imported / gap counts, and the roots each was looked for undero2b brain import-session --discover The same report, plus the absolute path of every log that has never been importedo2b brain import-session --discover --all Import that gap, one file at a time, through the same importSession the path form calls+o2b brain import-session --propose-backlog One ask-once envelope over the undistilled backlog (since v1.86.0): candidates whose dedup hash the vault already holds are dropped and named, the rest collapse to one proposal per distinct hash across sessions with every contributing session listed, and a session whose every candidate was already known is reported rather than re-proposedo2b brain import-session --status|--discover --progressNewline-delimited progress on stderr under the `sessions` operation, one `hash` stage```The modes never blend, and a conflicting combination is refused by name-rather than ranked: a path with `--discover` or `--status`, `--discover`-with `--status`, or `--all` without `--discover`. Either ranking would be a+rather than ranked: a path with `--discover`, `--status` or+`--propose-backlog`, two mode flags together, `--propose-backlog` with+`--all`, or `--all` without `--discover`. Either ranking would be aguess about which one the operator meant.`--discover --all` writes through the same `importSession` the path form@@ -1369,25 +1390,41 @@ An uncapped vault is unbounded: `o2b brain storage` reports the bytes andthe per-surface census, states `unbounded` plainly, invents nodays-to-exhaustion figure, and nothing pauses and nothing refuses.Setting `storage.cap_bytes` in `Brain/_brain.yaml` turns on the tiers and-every consumer of them; the four thresholds (`warn_percent`,-`critical_percent`, `warn_days`, `critical_days`) default to 80 and 95-percent of the cap and 30 and 7 days to it at the measured trend, and the-critical band must sit past the warn band - a larger percent, a smaller-day horizon. Each invocation takes one census and appends one-`storage_usage` sample; from two or more samples a linear trend yields-the days-to-capacity figure, and a tier is decided by the worse of its-two bands. The census covers `brain_tree` and the dot-directories-(`.snapshots/`, `.artifacts/`, `.archives/`, `.payloads/`, `.state/`)-plus the derived store with its WAL/SHM sidecars; a surface that cannot-be measured is refused by name, never reported as zero. Below local+every consumer of them; the five thresholds (`warn_percent`,+`pause_percent`, `critical_percent`, `warn_days`, `critical_days`)+default to 80 and 95 percent of the cap - the pause line riding the warn+band - and 30 and 7 days to it at the measured trend; the critical band+must sit past the warn band, and an explicit `pause_percent` must sit+below `critical_percent`, which is where interactive writes are refused+outright. Each invocation takes one census and appends one+`storage_usage` sample. Since v1.86.0 the trend is day-based: samples+bucket to one point per UTC day (the day's last sample wins), a sample+whose census was partial drops out of the fit entirely, and the fit needs+three distinct days inside a trailing 30-day window before any+days-to-capacity figure exists - two calls a minute apart around one+ordinary write no longer extrapolate the burst into a full vault - while+a tier is decided by the worse of its two bands, so a tier never rides a+trend that was never fitted. The census covers `brain_tree` and the+dot-directories (`.snapshots/`, `.artifacts/`, `.archives/`, `.payloads/`,+`.state/`) plus the derived store - since v1.86.0 both the store files+with their WAL/SHM sidecars and the directory's other contents, the+ledgers, dead letters and checkpoints the meter used to miss - and it+measures the store the configured location (`search_db_path` when set)+points at, so the report, the lane gate and the doctor check answer about+the same vault; a surface that cannot be measured is refused by name,+never reported as zero. Below localreach the census is not taken: the report answers `measured: false` withthe reason named and carries no figures. The CLI runs at local reach, soit always sees the measured form.The consumers are the maintenance lane's fifth gate (see "Write-timeintegrity and governance" above): a fresh census at decision time, never-the stored aggregate, skipping the pass at a measured critical tier with-its own `skipped:capacity` verdict; `brain_ingest_source`'s admission+the stored aggregate, skipping the pass at a measured percent at or past+`storage.pause_percent` (since v1.86.0) or at the critical tier with its+own `skipped:capacity` verdict, and recording its census as a+`storage_usage` sample on every run where the gate evaluated, skip or+proceed, so a vault where nobody runs the storage report still grows a+trend; `brain_ingest_source`'s admissionrefusal, which answers `capacity_exhausted` in the validate phase beforethe write path opens, so a refused ingest leaves the vaultbyte-identical; and the `storage-capacity` doctor check, which reports@@ -1401,8 +1438,12 @@ to growth fine.Receipt compaction gives the two append-only receipt ledgers - thecontinuity ledger and the metrics ledger - a drain. `scan` inventories-every shard read-only (records, foldable, protected, unparseable,-aggregates, bytes) and names each shard it could not read; `compact`+shards read-only (records, foldable, protected, unparseable, aggregates,+bytes) and names each shard it could not read; since v1.86.0 it offers+only this device's shard plus the legacy un-sharded file, and a foreign+shard named by hand to `compact` is refused by name on previews and+confirmed runs alike - another device's shard; compact it on the owning+device. `compact`folds the records older than `--older-than-days` (default 30) intoper-kind, per-month aggregate lines inside the shards the caller nameswith `--shard family:name` ids from a scan - a compact with no shard ids@@ -1410,9 +1451,14 @@ is a usage error, never a guess. It previews unless `--apply` is passed;a confirmed run takes a recovery-point snapshot first and rewrites eachshard atomically. `--expect N` asserts exactly N records will fold andaborts without writing on a mismatch; `--strict` refuses a confirmed-compact that carries no count guard. Records in the protected classes-never fold whatever their age - the `source_invalidation` audit trail-and the two context-compaction kinds - and one `receipt_compaction` audit+compact that carries no count guard. Since v1.86.0 folding is opt-in per+kind: a closed allowlist names the pure telemetry kinds whose readers+keep no per-record identity past the correlation window, and every other+continuity kind - the functional records, the audit trail, the kinds a+reader reads back as data - keeps verbatim whatever its age, an unknown+kind failing closed to keep the same way a future build's records always+do. Repeated folds merge into the existing per-kind, per-month aggregate+instead of stacking a second line, and one `receipt_compaction` auditrecord closes a confirmed run. Compact is a local-reach surface (the`brain_receipts` tool refuses it by name below local reach, while scanstays readable at any reach); the CLI always runs at local reach. Thisdiff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex 16116d6a..df7dea09 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -216,6 +216,8 @@ trail, and `move` retires the chain into `Brain/intentions/history/`.All four paths share a normalised payload hash so the same rule captured twice from different surfaces dedups automatically. Pinned preferences are exempt from automatic retire (`stale-no-evidence`, `expired-unconfirmed`, `rebutted`); only `o2b brain reject` can retire them.+Session mining has a memory of its own (since v1.86.0). When `o2b brain extract-signals` (or the `brain_extract_signals` tool) commits a mine that ends with nothing written and nothing staged - an empty answer, every item already deduped, every item refused by the durability gate, or any mix - it records a zero-yield tombstone for the session in a mining ledger beside the import ledger, keyed on a SHA-256 of the mined turn material (turn id, text, timestamp). The next plan over that session returns the recorded breakdown instead of an envelope, so an unchanged transcript is never re-offered to the model; a re-import that changes the turns changes the hash and re-opens the session with no clearing step. Because removing a downstream preference does not change a session's turns, the tombstone never lifts on its own: `--force` (MCP `force`) bypasses the gate for exactly one plan call when the session should be re-proposed. A commit that succeeds but cannot record its tombstone still succeeds and names the failure, so a ledger that stopped being written is never a silent fact.+`o2b brain watchdog` is the recovery surface for this layout. It checks theBrain config, required directories, and rebuildable search index, emitsexponential backoff metadata for schedulers, and writes an audit record. By@@ -1619,7 +1621,7 @@ Until now almost nothing about a write into the Brain was declared: any frontmat**The keyfile can be wrapped under a passphrase.** `o2b brain secret unlock` verifies a passphrase (read from stdin or `--passphrase-from-env`, never argv) and - on a store whose keyfile is still raw - wraps the 32-byte key into a versioned scrypt envelope at the same path, then holds the unwrapped key in memory for that one process only; `lock` overwrites and drops it. There is no daemon: every other CLI invocation and the MCP server stay locked, and a key-bearing operation over a locked store refuses with a named error whose remedy is the unlock command (the refusal's prose carries no paths - the structured field does, for the surfaces allowed to name them). Every unlock and lock lands a no-values custody record. A credential reference, `$secret:NAME`, resolves store-first with environment fallback at the use sites that already name it (the exec allowlist, the embedding and rerank keys, the decision-model key, the Telegram capture token, the research keys); a locked store surfaces the named refusal rather than silently answering from the environment. `secret export` writes every entry as one passphrase-encrypted bundle to an operator-named file - the values re-encrypted under a key the passphrase derives, the names, env-var mappings and allowlists traveling in the clear past the shared egress redactor - and `secret import` decrypts and validates every entry (names, env-var mappings, allow patterns, all under the store's own rules) before writing anything. The honesty clause is the same for the keyfile envelope and for bundles: the passphrase is never persisted, logged or audited, and a lost one is unrecoverable - every stored value and every exported bundle stays unreadable forever, and nothing recovers them.-**Heavy work waits its turn.** The maintenance lane runs dream and reindex behind five gates: a tz-aware local-time window, a busy gate over recent recall telemetry, an optional host-pressure gate, and - since v1.85.0, when a storage cap is configured - a capacity gate that skips a vault measuring at its critical tier before the failure-streak limit refuses a task that keeps failing; under them all sits an expiring SQLite lease in its own database so holding it never contends with the search index writer lock. Force bypasses the soft gates but never the lease, tasks run stale-first, and every attempt - including each gate refusal - lands in a bounded journal, because a lane you cannot audit is a lane you cannot trust.+**Heavy work waits its turn.** The maintenance lane runs dream and reindex behind five gates: a tz-aware local-time window, a busy gate over recent recall telemetry, an optional host-pressure gate, and - since v1.85.0, when a storage cap is configured - a capacity gate that skips a vault past its `storage.pause_percent` line (since v1.86.0; the line defaults to the warn band, and interactive ingest keeps working until the critical tier refuses it outright) or measuring at its critical tier, before the failure-streak limit refuses a task that keeps failing; under them all sits an expiring SQLite lease in its own database so holding it never contends with the search index writer lock. Force bypasses the soft gates but never the lease, tasks run stale-first, and every attempt - including each gate refusal - lands in a bounded journal, because a lane you cannot audit is a lane you cannot trust.## Link and recall intelligence (since v0.45.0)@@ -1706,8 +1708,9 @@ a `warning` finding with `proposed_action: review`. A local source thatwas deleted therefore surfaces as `url-only`, a `bounded-local` page countsas backed only while its excerpt still matches its digest, and quarantinedpages are skipped: they already carry the `untrusted_source` marker, which-names the same condition (and which the retrieval trust gate excludes when-`search_trust_gate_enabled` is on).+names the same condition (and which the retrieval trust gate excludes+under `search_trust_gate_mode: enforce` - the bare+`search_trust_gate_enabled` alone measures in `shadow`).## Non-Markdown sources (since v1.69.0)diff --git a/docs/mcp.md b/docs/mcp.mdindex b0615cf4..b6f97b20 100644--- a/docs/mcp.md+++ b/docs/mcp.md@@ -476,13 +476,17 @@ per-result `score_breakdown`: `retrieval_decision_trace` (evaluated /surfaced / excluded counts plus every excluded candidate as a compactreference with its structural reasons) and `memory_trust_assessment` (thesame exclusion set as a reason histogram). Both are by-products of the-retrieval trust gate; with the gate off the response carries-`retrieval_trace_unavailable` naming the switch that produces them-(`search_trust_gate_enabled`) instead of going quiet. Without `explain` no-key appears at all. In the same release `total` stopped echoing the row-count: it is the ranked candidate pool the `limit` sliced the returned rows-from, so `total > results.length` means the ranker had more candidates-than it handed you. Read it as candidates ranked, not as a corpus+retrieval trust gate, which runs in one of three modes+(`search_trust_gate_mode: off | shadow | enforce`; the bare+`search_trust_gate_enabled` alone maps to `shadow`, measuring without+excluding, and `enforce` is what excludes): with the gate off the+response carries `retrieval_trace_unavailable` naming the switches that+produce them (`search_trust_gate_enabled`, or+`search_trust_gate_mode: shadow`) instead of going quiet. Without+`explain` no key appears at all. In the same release `total` stopped+echoing the row count: it is the ranked candidate pool the `limit`+sliced the returned rows from, so `total > results.length` means the+ranker had more candidates than it handed you. Read it as candidates ranked, not as a corpusstatistic: the pool is capped at roughly three times the requested`limit`, so it moves with `limit` and saturates on a large vault, andlink traversal can add a document that matched no term at all.@@ -2169,6 +2173,21 @@ format characters), when it contains NUL, or when it exceeds the cap.`decision_model: { degraded }` when a request failed and every turn wassent. With the use `off` none of these fields appears. Commit items acceptan optional string `source_turn` naming the plan turn a rule came from.+- Since v1.86.0 `brain_extract_signals` remembers barren mines. A commit+ that ends with nothing written and nothing staged records a zero-yield+ tombstone in the session mining ledger beside the session import ledger,+ keyed on a SHA-256 of the mined turn material; the next plan over that+ session at unchanged content runs before the pre-filter and spends+ nothing, returning `skipped: { reason: "mining_tombstone", mining_outcome:+ { written, staged, deduped, durability_rejected } }` with `llm_step: null`+ and empty `turns_mined` - the recorded breakdown is the evidence, never a+ bare empty plan. A re-import that changes the session's turns changes the+ hash and re-opens it with no clearing step. The `force` tool argument+ bypasses the tombstone for exactly one plan call, for a caller that+ removed the downstream preference and wants the session re-proposed; a+ corrupt mining ledger refuses the plan by name rather than reading as+ empty. A commit that succeeds but cannot record its tombstone still+ succeeds and carries `mining_ledger_failure` by name.- Advisory decision-model fields (optional, see[decision-models/dedup-tension.md](decision-models/dedup-tension.md) and[decision-models/labels.md](decision-models/labels.md)). All of them arediff --git a/hooks/backlog-nudge.ts b/hooks/backlog-nudge.tsnew file mode 100644index 00000000..f137a857--- /dev/null+++ b/hooks/backlog-nudge.ts@@ -0,0 +1,109 @@+#!/usr/bin/env -S bun+/**+ * SessionStart hook: opt-in undistilled backlog nudge (session-mining-economy,+ * t_0358af42).+ *+ * When `backlog_nudge_enabled` is set (default OFF), reports how many+ * imported sessions have never been distilled, as ONE `additionalContext`+ * line, so the agent starts the session aware of the backlog still waiting.+ * The count comes from the vault-side mining census+ * (`src/core/brain/sessions/mining-census.ts`), which reads the import+ * ledger joined to the mining record and never hashes a source log - the+ * honest count is affordable inside this hook's budget.+ *+ * The nudge is change-keyed, not timed: the line is injected when the+ * undistilled count differs from the count injected last (state:+ * hooks/lib/backlog-nudge-state.ts, one overwrite-only file beside the+ * hygiene digest). An unchanged backlog is never re-nagged, and a drained+ * backlog is recorded without being announced - a line that says there is+ * nothing to do is the noise this hook exists to avoid.+ *+ * Under the deadline the hook degrades to silence, never to a fabricated+ * number: the census runs under the process ceiling's cooperative+ * safeguard, and a census that cannot verify its ledgers names the failure+ * on stderr and injects nothing.+ *+ * Flag off is an immediate no-op with zero output, keeping the session+ * preamble byte-identical. Fail-open: the hook arms the shared process+ * ceiling and exits 0 on every path.+ */++import { resolveBacklogNudgeEnabled, resolveVault } from "../src/core/config.ts";+import { isContextEventName } from "./lib/context-events.ts";+import {+ armProcessCeiling,+ ceilingSafeguard,+ resolveHookCeilingMs,+} from "./lib/process-ceiling.ts";+import { backlogNudgeCountMatches, writeBacklogNudgeCount } from "./lib/backlog-nudge-state.ts";+import { asHookPayload, readHookInput } from "./lib/stdin.ts";++async function main(): Promise<void> {+ // Fast opt-out FIRST: default OFF means an immediate no-op, no output.+ if (!resolveBacklogNudgeEnabled()) return;++ const ceilingMs = resolveHookCeilingMs();+ const disarm = armProcessCeiling({ ceilingMs });+ try {+ let payload;+ try {+ payload = asHookPayload(await readHookInput());+ } catch {+ return;+ }++ const hookEventName =+ typeof payload.hook_event_name === "string" && payload.hook_event_name.length > 0+ ? payload.hook_event_name+ : "SessionStart";+ // Default-closed: only an additionalContext-eligible event may emit.+ if (!isContextEventName(hookEventName)) return;++ const vault = resolveVault();+ if (vault === null) return;++ // Loaded only here, after every gate: the census and its ledger chain+ // are the one heavy import, and a flag-off session start pays nothing+ // for them (the hygiene-digest hook's lazy-load precedent).+ const { buildMiningCensus } = await import("../src/core/brain/sessions/mining-census.ts");+ let count: number;+ try {+ count = buildMiningCensus(vault, { safeguard: ceilingSafeguard(ceilingMs) }).undistilled;+ } catch (err) {+ // Never block, never guess: a census that cannot verify its ledgers+ // says why and injects nothing.+ process.stderr.write(+ `backlog-nudge: the backlog census failed: ${(err as Error).message ?? String(err)}\n`,+ );+ return;+ }++ // Changed-count gate. A missing or corrupt state file compares unequal,+ // so the next eligible session emits again (lose-not-duplicate).+ if (backlogNudgeCountMatches(vault, count)) return;++ // A drained backlog is recorded, not announced.+ if (count <= 0) {+ writeBacklogNudgeCount(vault, count);+ return;+ }++ const sessions = count === 1 ? "session" : "sessions";+ const line =+ `Open Second Brain session backlog: ${count} imported ${sessions} not yet distilled. ` ++ "Run 'o2b brain import-session --status' for the per-runtime breakdown.";+ process.stdout.write(+ JSON.stringify({ hookSpecificOutput: { hookEventName, additionalContext: line } }) + "\n",+ );++ // The state lands AFTER the emit: a crash in between loses the dedupe+ // and the next eligible session emits again, rather than never.+ writeBacklogNudgeCount(vault, count);+ } finally {+ disarm();+ }+}++main().catch(() => {+ // Never crash the runtime; the session start must proceed regardless.+});diff --git a/hooks/hooks.json b/hooks/hooks.jsonindex 992f1b08..7e30d8a3 100644--- a/hooks/hooks.json+++ b/hooks/hooks.json@@ -56,6 +56,12 @@"timeout": 10,"statusMessage": "OSB: injecting open recall-gap agenda (opt-in)"},+ {+ "type": "command",+ "command": "r=\"$CLAUDE_PLUGIN_ROOT\"; if [ -n \"$r\" ] && [ -x \"$r/scripts/o2b-hook\" ]; then exec \"$r/scripts/o2b-hook\" backlog-nudge; fi; command -v o2b-hook >/dev/null 2>&1 && exec o2b-hook backlog-nudge; exit 0",+ "timeout": 10,+ "statusMessage": "OSB: injecting undistilled backlog count (opt-in)"+ },{"type": "command","command": "r=\"$CLAUDE_PLUGIN_ROOT\"; if [ -n \"$r\" ] && [ -x \"$r/scripts/o2b-hook\" ]; then exec \"$r/scripts/o2b-hook\" session-capture; fi; command -v o2b-hook >/dev/null 2>&1 && exec o2b-hook session-capture; exit 0",diff --git a/hooks/lib/backlog-nudge-state.ts b/hooks/lib/backlog-nudge-state.tsnew file mode 100644index 00000000..0575e39a--- /dev/null+++ b/hooks/lib/backlog-nudge-state.ts@@ -0,0 +1,86 @@+/**+ * Backlog nudge state (session-mining-economy, t_0358af42).+ *+ * Once-per-change state for the SessionStart backlog nudge: one+ * overwrite-only file, `<vault>/.open-second-brain/backlog-nudge.count`,+ * holding the undistilled count the hook last injected. Vault-global and+ * un-synced derived state - beside `hygiene-digest.hash`, and for the same+ * reason: per-session scope is the wrong lifetime for "once per change",+ * because the backlog belongs to the vault and must re-arm in a new+ * session. Size one, self-overwriting, no cleanup path, so the file cannot+ * grow.+ *+ * Fail-soft on both sides, mirroring the hygiene-digest convention: a+ * missing, unreadable, non-regular or corrupt file reads as `null`, which+ * compares as "changed" so the next eligible session emits again rather+ * than staying silent forever, and a write reports `false` instead of+ * throwing so the hook's silent-exit-0 contract holds.+ *+ * The module is deliberately import-light (node:fs, the atomic writer, the+ * no-follow guard, the path constant), so the hook can statically import+ * it and still stay cheap when the feature flag is off. The heavy census+ * is the hook's own dynamic import, loaded after every gate.+ */++import { mkdirSync } from "node:fs";+import { dirname, join } from "node:path";++import { DERIVED_STORE_DIR } from "../../src/core/brain/path-constants.ts";+import {+ derivedDirIsSymlinked,+ readRegularFileNoFollow,+} from "../../src/core/derived-store-guard.ts";+import { atomicWriteText } from "../../src/core/fs-atomic.ts";++/** The single state file name, directly under {@link DERIVED_STORE_DIR}. */+export const BACKLOG_NUDGE_COUNT_FILENAME = "backlog-nudge.count";++/** A state file holds exactly one non-negative integer and nothing else. */+const COUNT_LINE_RE = /^\d+$/;++/** Absolute path of the state file for one vault. */+export function backlogNudgeStatePath(vault: string): string {+ return join(vault, DERIVED_STORE_DIR, BACKLOG_NUDGE_COUNT_FILENAME);+}++/**+ * The count last recorded for the vault, or `null` when nothing usable is+ * recorded: the file is absent, unreadable, not a regular file, or does+ * not hold a single non-negative integer. Never throws.+ */+export function readBacklogNudgeCount(vault: string): number | null {+ const read = readRegularFileNoFollow(backlogNudgeStatePath(vault));+ if (read.status !== "ok") return null;+ const line = read.text.trim();+ if (!COUNT_LINE_RE.test(line)) return null;+ const value = Number(line);+ return Number.isSafeInteger(value) ? value : null;+}++/**+ * True when `count` is exactly what the state file holds. A missing or+ * corrupt record compares unequal: the caller emits and rewrites, which is+ * the lose-not-duplicate direction.+ */+export function backlogNudgeCountMatches(vault: string, count: number): boolean {+ return readBacklogNudgeCount(vault) === count;+}++/**+ * Overwrite the state file with `count`, in place: an atomic rename over+ * the single file, no siblings left behind, no growth. Refuses (returning+ * `false`) when the derived store is itself a symbolic link - a vault+ * received from elsewhere could point it at any directory. Never throws.+ */+export function writeBacklogNudgeCount(vault: string, count: number): boolean {+ try {+ if (!Number.isSafeInteger(count) || count < 0) return false;+ if (derivedDirIsSymlinked(vault, DERIVED_STORE_DIR)) return false;+ const path = backlogNudgeStatePath(vault);+ mkdirSync(dirname(path), { recursive: true });+ atomicWriteText(path, `${count}\n`);+ return true;+ } catch {+ return false;+ }+}diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex a2fdc345..927d7fd3 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.85.0",+ "version": "1.86.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/openclaw/index.js b/openclaw/index.jsindex b2738582..d07c76b4 100644--- a/openclaw/index.js+++ b/openclaw/index.js@@ -2260,6 +2260,7 @@ __export(exports_config, {resolveAnswerQuoteCheckActive: () => resolveAnswerQuoteCheckActive,resolveAnswerQuoteCheckRecall: () => resolveAnswerQuoteCheckRecall,resolveAnswerQuoteCheckSubagent: () => resolveAnswerQuoteCheckSubagent,+ resolveBacklogNudgeEnabled: () => resolveBacklogNudgeEnabled,resolveBenchJudgeCmd: () => resolveBenchJudgeCmd,resolveContextPackOutcomeEnabled: () => resolveContextPackOutcomeEnabled,resolveDecisionRecallMaxPerSession: () => resolveDecisionRecallMaxPerSession,@@ -2874,6 +2875,9 @@ function resolvePartnerCodegraphDisabled(configPath) {function resolveGapLoopEnabled(configPath) {return resolveConfigFlag("OPEN_SECOND_BRAIN_GAP_LOOP_ENABLED", "gap_loop_enabled", configPath);}+function resolveBacklogNudgeEnabled(configPath) {+ return resolveConfigFlag("OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED", "backlog_nudge_enabled", configPath);+}function resolveGapLoopThreshold(configPath) {const env = process.env["OPEN_SECOND_BRAIN_GAP_LOOP_THRESHOLD"]?.trim();const raw = env || discoverConfig(configPath).data["gap_loop_threshold"]?.trim();diff --git a/package.json b/package.jsonindex 99fe488c..90ec3a48 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.85.0",+ "version": "1.86.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex e6b2e6d5..90f34cd1 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.85.0"+version: "1.86.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 2588a657..c53cf78e 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.85.0",+ "version": "1.86.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/codex/README.md b/plugins/codex/README.mdindex ba52422d..a902102c 100644--- a/plugins/codex/README.md+++ b/plugins/codex/README.md@@ -123,7 +123,7 @@ The full router with readiness criteria is [`install.md`](https://github.com/ite## What is new-1.85.0 gives a long-lived vault the means to watch its own storage and to keep its append-only ledgers in shape. `o2b brain storage` (and the `brain_storage` tool) measures a per-surface byte census and forecasts days to the configured `storage.cap_bytes` in graduated tiers; a measured critical tier pauses the maintenance lane behind its fifth gate and refuses interactive ingests with `capacity_exhausted` before any write, while an uncapped vault reads unbounded, nothing pauses and nothing refuses. `o2b brain receipts` (and `brain_receipts`) drains the append-only continuity and metrics ledgers by folding expired records into per-kind, per-month aggregates behind an explicit preview-bound apply, with a count guard and a recovery point around every confirmed run. An opt-in `document-dedup` hygiene detector nominates near-duplicate document copies for retirement behind the review gate, keeping the newest copy, retargeting the stale copies' inbound links and auditing each retirement; bulk source removals report per-file failures instead of aborting; and appended sources yield their new text through incremental extraction over a manifest watermark, no whole-source reprocess. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).+1.86.0 stops re-paying for sessions with nothing new to say and measures before it excludes. A session mine that ends with nothing written records a zero-yield tombstone in a session mining ledger, so the next plan over the same transcript returns the recorded breakdown instead of re-offering the turns to a model, with `--force` re-mining exactly the one call it is given to; a distilled and undistilled census joins the import and mining ledgers, `o2b brain import-session --propose-backlog` batches the undistilled backlog into one ask-once envelope deduplicated against what the vault already holds, and an opt-in `backlog_nudge_enabled` SessionStart hook reports the undistilled count once per change. The retrieval trust gate resolves `search_trust_gate_mode` (`off | shadow | enforce`): `shadow` records every would-be refusal with its structural reasons while excluding nothing, and the relational arm classifies each walked node and arriving edge under the same mode. The loopback explorer serves a live-counts console beside its graph (`/counts.json`, `/status.json`). The release also repairs what the external review of v1.85.0 found: receipt compaction folds only opt-in telemetry kinds, the capacity trend buckets its samples per day behind a three-day minimum while the maintenance lane pauses at `storage.pause_percent`, a retired document copy keeps its manifest marker so an unchanged source never quietly re-ingests, and the incremental extraction watermark stamps exactly the bytes the appended delta covered. Every release is described in the [CHANGELOG](https://github.com/itechmeat/open-second-brain/blob/main/CHANGELOG.md).## Documentationdiff --git a/plugins/codex/hooks/hooks.json b/plugins/codex/hooks/hooks.jsonindex d340e081..f487520d 100644--- a/plugins/codex/hooks/hooks.json+++ b/plugins/codex/hooks/hooks.json@@ -62,6 +62,13 @@"timeout": 10,"statusMessage": "OSB: injecting open recall-gap agenda (opt-in)"},+ {+ "type": "command",+ "command": "r=\"$CLAUDE_PLUGIN_ROOT\"; if [ -n \"$r\" ] && [ -x \"$r/scripts/o2b-hook\" ]; then exec \"$r/scripts/o2b-hook\" backlog-nudge; fi; command -v o2b-hook >/dev/null 2>&1 && exec o2b-hook backlog-nudge; exit 0",+ "commandWindows": "set NoDefaultCurrentDirectoryInExePath=1& where /q $PATH:o2b-hook && o2b-hook backlog-nudge & exit /b 0",+ "timeout": 10,+ "statusMessage": "OSB: injecting undistilled backlog count (opt-in)"+ },{"type": "command","command": "r=\"$CLAUDE_PLUGIN_ROOT\"; if [ -n \"$r\" ] && [ -x \"$r/scripts/o2b-hook\" ]; then exec \"$r/scripts/o2b-hook\" session-capture; fi; command -v o2b-hook >/dev/null 2>&1 && exec o2b-hook session-capture; exit 0",diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex e6b2e6d5..90f34cd1 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.85.0"+version: "1.86.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex a1ae18bc..e5b224b4 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.85.0"+version = "1.86.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/brain/verbs/explorer.ts b/src/cli/brain/verbs/explorer.tsindex b9d320e0..d9359e4c 100644--- a/src/cli/brain/verbs/explorer.ts+++ b/src/cli/brain/verbs/explorer.ts@@ -58,7 +58,7 @@ export async function cmdBrainExplorer(argv: string[]): Promise<number> {export: { type: "string" },force: { type: "boolean" },});- const { vault } = brainVerbContext(flags);+ const { config, vault } = brainVerbContext(flags);const exportPath = flags["export"] as string | undefined;const force = flags["force"] === true;@@ -96,7 +96,7 @@ export async function cmdBrainExplorer(argv: string[]): Promise<number> {let server: LiveServerHandle;try {- server = buildLiveServer(vault, port);+ server = buildLiveServer(vault, port, { configPath: config });} catch (err) {const msg = (err as Error).message ?? String(err);if (/EADDRINUSE|address already in use/i.test(msg))@@ -105,6 +105,7 @@ export async function cmdBrainExplorer(argv: string[]): Promise<number> {}ok(`Live explorer at ${server.url}`);+ info("Counts and operator status are served at /counts.json and /status.json.");info("Press Ctrl+C to stop.");await new Promise<void>((resolveStop) => {// `.catch(() => undefined).finally(...)` so the shutdown promisediff --git a/src/cli/brain/verbs/extract-signals.ts b/src/cli/brain/verbs/extract-signals.tsindex 848044da..b6ab79e4 100644--- a/src/cli/brain/verbs/extract-signals.ts+++ b/src/cli/brain/verbs/extract-signals.ts@@ -8,6 +8,20 @@* that answer and writes the accepted items as speculative inbox signals.* OSB runs no model on either path.*+ * A commit that answers a plan passes that plan's `surveyed_turns_hash`+ * back with `--surveyed-turns-hash`: it is what the zero-yield mining+ * tombstone keys on, so the record describes the material the verdict was+ * earned over rather than whatever the session holds by commit time. A+ * commit without it records nothing - without surveyed material there is+ * nothing honest to key a tombstone on.+ *+ * A session already mined at its current content that yielded nothing+ * accepted carries a tombstone in the session mining ledger; its plan phase+ * returns the recorded breakdown instead of an envelope (reason+ * `mining_tombstone`), and `--force` bypasses that gate for the one plan+ * call it is given to. A commit that succeeds but cannot record its+ * zero-yield tombstone still succeeds, and names the ledger failure.+ ** Exit codes: 0 on success, 1 on an operational failure or a refused* payload, 2 on usage errors.*/@@ -20,6 +34,7 @@ import {ExtractSignalsError,} from "../../../core/brain/extract-signals.ts";import {+ EXTRACT_MINING_TOMBSTONE_SKIP_REASON,planExtractSignalsPrefiltered,recordExtractPrefilterCommit,resolveExtractPrefilterConfig,@@ -36,6 +51,7 @@ import {import {brainVerbContext,fail,+ normalizeFlagString,ok,okJson,parse,@@ -45,7 +61,7 @@ import {const USAGE ="usage: o2b brain extract-signals <session-ref> [--payload <json> | --payload-file <path>] " +- "[--agent <name>] [--vault <path>] [--json]";+ "[--surveyed-turns-hash <sha256>] [--force] [--agent <name>] [--vault <path>] [--json]";/** Refusals the operator caused, reported without a stack. */const NAMED_REFUSALS = [ExtractSignalsError, ResponseShapeError, ResponseCheckError] as const;@@ -60,11 +76,19 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {agent: { type: "string" },payload: { type: "string" },"payload-file": { type: "string" },+ "surveyed-turns-hash": { type: "string" },+ force: { type: "boolean" },json: { type: "boolean" },});const sessionRef = positional[0];if (!sessionRef || sessionRef.trim() === "") return usageError(USAGE);const asJson = flags["json"] === true;+ const surveyedTurnsHashRaw = normalizeFlagString(flags["surveyed-turns-hash"]);+ if (flags["surveyed-turns-hash"] !== undefined && surveyedTurnsHashRaw === null) {+ return usageError(+ `${USAGE}\nerror: --surveyed-turns-hash must be a non-empty string when provided`,+ );+ }try {const { config, vault } = brainVerbContext(flags);@@ -81,6 +105,9 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {now: new Date(),decisionModel,tokenImpactEnabled: decisionModel !== null && resolveTokenImpactLedgerEnabled(config),+ // The re-mine escape: bypass the zero-yield tombstone for this one+ // plan call (no payload here, so this IS the plan call).+ force: flags["force"] === true,});if (asJson) {okJson({@@ -94,6 +121,10 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {text: t.text,timestamp: t.timestamp,})),+ // The hash of the material this plan offered; the commit that+ // answers it passes it back with --surveyed-turns-hash so the+ // mining tombstone keys on the surveyed material.+ surveyed_turns_hash: plan.surveyedTurnsHash,cap: plan.cap,confidence_floor: plan.confidenceFloor,llm_step: plan.llmStep,@@ -103,7 +134,19 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {? {skipped: {reason: plan.skipped.reason,- turns_dropped: plan.skipped.turnsDropped,+ ...(plan.skipped.turnsDropped !== undefined+ ? { turns_dropped: plan.skipped.turnsDropped }+ : {}),+ ...(plan.skipped.miningOutcome !== undefined+ ? {+ mining_outcome: {+ written: plan.skipped.miningOutcome.written,+ staged: plan.skipped.miningOutcome.staged,+ deduped: plan.skipped.miningOutcome.deduped,+ durability_rejected: plan.skipped.miningOutcome.durabilityRejected,+ },+ }+ : {}),},}: {}),@@ -115,6 +158,21 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {}ok(`session: ${plan.sessionId} (${plan.turnsScanned} imported turn(s))`);if (plan.skipped !== undefined) {+ if (plan.skipped.reason === EXTRACT_MINING_TOMBSTONE_SKIP_REASON) {+ const outcome = plan.skipped.miningOutcome!;+ // The recorded breakdown is the evidence, so every field is+ // rendered from the record, never asserted from a hardcoded+ // zero: a hand-restored or future-written entry must read the+ // same in text as the --json surface prints it.+ ok(+ `nothing to mine: this session was already mined at its current content and ` ++ `yielded nothing (written ${outcome.written}, staged ${outcome.staged}, ` ++ `deduped ${outcome.deduped}, ` ++ `durability-rejected ${outcome.durabilityRejected}); ` ++ `re-run with --force to re-mine it`,+ );+ return 0;+ }ok(`nothing to mine: the decision-model pre-filter dropped all ` +`${plan.skipped.turnsDropped} user turn(s)`,@@ -128,6 +186,7 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {ok(`limits: at most ${plan.cap} items, confidence >= ${plan.confidenceFloor}`);if (plan.llmStep !== null) {ok(`needs-llm-step: ${plan.llmStep.step} -> ${plan.llmStep.target_path}`);+ ok(`surveyed turns hash: ${plan.surveyedTurnsHash}`);}return 0;}@@ -157,6 +216,10 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {// Loaded with parallel I/O here, where the caller can await it; the// commit's own fallback is the synchronous walk.dedup: await loadDedupIndex(vault),+ // The answered plan's surveyed hash, relayed by the calling agent:+ // what the zero-yield tombstone keys on. Absent, the commit records+ // nothing - there is no material the verdict was earned over.+ ...(surveyedTurnsHashRaw !== null ? { surveyedTurnsHash: surveyedTurnsHashRaw } : {}),});recordExtractPrefilterCommit(vault, decisionModel, res, now);if (asJson) {@@ -168,6 +231,11 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {deduped: res.deduped,durability_rejected: res.durabilityRejected,rejected: res.rejected.map((r) => ({ topic: r.topic, reason: r.reason })),+ // Present only when the commit succeeded but the zero-yield+ // tombstone could not be recorded.+ ...(res.miningLedgerFailure !== undefined+ ? { mining_ledger_failure: res.miningLedgerFailure }+ : {}),// Present only when a leading <think> block was stripped (t_dac8bf7e)....(strip !== undefined ? { note: payloadStripNote(strip) } : {}),});@@ -178,6 +246,9 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {(res.staged > 0 ? ` (staged for approval)` : "") +`, deduped ${res.deduped}, durability-rejected ${res.durabilityRejected}`,);+ if (res.miningLedgerFailure !== undefined) {+ ok(`mining ledger not updated: ${res.miningLedgerFailure}`);+ }if (strip !== undefined) ok(payloadStripNote(strip));for (const r of res.rejected) ok(`rejected: ${r.topic} (${r.reason})`);return 0;diff --git a/src/cli/brain/verbs/forget-source.ts b/src/cli/brain/verbs/forget-source.tsindex 82002f76..b6ec4e23 100644--- a/src/cli/brain/verbs/forget-source.ts+++ b/src/cli/brain/verbs/forget-source.ts@@ -8,7 +8,8 @@* shared-derivation pages and aggregate surfaces (logs, reports, MOCs) are* reported, never deleted. `--include-originals` additionally removes the* original source file(s) outside `Brain/`. Every confirmed cleanup writes- * an auditable `source_invalidation` continuity record.+ * an auditable `source_invalidation` continuity record; a confirmed run+ * that could not remove an entry names it in both outputs and exits 1.*/import { deleteBySource } from "../../../core/brain/source-cleanup.ts";@@ -87,19 +88,30 @@ export async function cmdBrainForgetSource(argv: string[]): Promise<number> {mentions: plan.mentions.map((e) => ({ path: e.path, kind: e.kind, match: e.match })),originals: [...plan.originals],deleted: [...plan.deleted],+ // The per-entry removal failures ride the JSON (review M8): a+ // confirmed removal that could not delete a file must never read+ // as a clean run with a smaller count.+ failed: plan.failed.map((f) => ({ path: f.path, message: f.message })),manifest_entry_removed: plan.manifestEntryRemoved,audit_record_id: plan.auditRecordId,snapshot_run_id: plan.snapshotRunId,snapshot_path: plan.snapshotPath,recoverability: plan.recoverability,});- return 0;+ // Collected failures are an honest non-zero exit, not a crash: the+ // entries that could be removed were, and the JSON names what stayed.+ return plan.failed.length > 0 ? 1 : 0;}if (plan.confirmed) {ok(`forget-source: ${plan.source}`);ok(` deleted ${plan.deleted.length} file(s)`);for (const p of plan.deleted) ok(` - ${p}`);+ // The failed list reaches the text report too (review M8), with the+ // same exit rule the JSON path answers: failures are a non-zero exit.+ for (const f of plan.failed) {+ process.stderr.write(`error: failed to remove ${f.path}: ${f.message}\n`);+ }if (plan.manifestEntryRemoved) ok(" removed ingest manifest entry");if (plan.mentions.length > 0) {info(` ${plan.mentions.length} protected mention(s) NOT deleted (edit by hand):`);@@ -123,7 +135,7 @@ export async function cmdBrainForgetSource(argv: string[]): Promise<number> {);}}- return 0;+ return plan.failed.length > 0 ? 1 : 0;}// Dry-run report.diff --git a/src/cli/brain/verbs/import-session.ts b/src/cli/brain/verbs/import-session.tsindex a35c4f21..5a0f63ad 100644--- a/src/cli/brain/verbs/import-session.ts+++ b/src/cli/brain/verbs/import-session.ts@@ -23,6 +23,13 @@import { statSync } from "node:fs";import { resolveAgentName, resolveSessionCaptureRoles } from "../../../core/config.ts";+import { buildCaptureBoundary } from "../../../core/brain/capture-boundary.ts";+import { loadDedupIndex, type DedupIndexEntry } from "../../../core/brain/dedup-hash.ts";+import {+ buildBacklogProposals,+ type BacklogCandidate,+} from "../../../core/brain/distill/backlog-proposals.ts";+import { discoverMarkers, isFeedbackMarker } from "../../../core/brain/inline.ts";import {importSession,importSessionPath,@@ -37,6 +44,11 @@ import {type SessionDiscovery,type SessionImportRecord,} from "../../../core/brain/sessions/discover.ts";+import {+ buildMiningCensus,+ minedTurnMaterial,+ type SessionMiningCensus,+} from "../../../core/brain/sessions/mining-census.ts";import {SESSION_RESUME_DISCARD,type SessionResumeDiscard,@@ -85,14 +97,16 @@ export async function cmdBrainImportSession(argv: string[]): Promise<number> {discover: { type: "boolean" },status: { type: "boolean" },all: { type: "boolean" },+ "propose-backlog": { type: "boolean" },progress: { type: "boolean" },json: { type: "boolean" },});const discover = flags["discover"] === true;const status = flags["status"] === true;const importAll = flags["all"] === true;+ const proposeBacklog = flags["propose-backlog"] === true;const sessionPath = positional[0] ?? null;- assertOneMode(sessionPath, discover, status, importAll);+ assertOneMode(sessionPath, discover, status, importAll, proposeBacklog);const { config, vault } = brainVerbContext(flags);const explicitAgent = normalizeFlagString(flags["agent"]);if (flags["agent"] !== undefined && explicitAgent === null) {@@ -154,6 +168,7 @@ export async function cmdBrainImportSession(argv: string[]): Promise<number> {};if (sessionPath === null) {+ if (proposeBacklog) return proposeBacklogEnvelope({ vault, config, asJson });return await sweep({vault,config,@@ -252,6 +267,7 @@ function assertOneMode(discover: boolean,status: boolean,importAll: boolean,+ proposeBacklog: boolean,): void {if (discover && status) {throw new CliError(@@ -266,14 +282,23 @@ function assertOneMode("import of the gap it just listed. Pass --discover --all.",);}- if (sessionPath !== null && (discover || status)) {+ if (proposeBacklog && (discover || status || importAll)) {+ throw new CliError(+ `--propose-backlog and ${discover ? "--discover" : status ? "--status" : "--all"} cannot ` ++ "both be given: --propose-backlog batches the distill proposals the ALREADY IMPORTED " ++ "backlog raises, while the sweep flags report on or import what is on disk. Pass one.",+ );+ }+ if (sessionPath !== null && (discover || status || proposeBacklog)) {throw new CliError(- `${sessionPath} and ${discover ? "--discover" : "--status"} cannot both be given: a path ` ++ `${sessionPath} and ${+ discover ? "--discover" : status ? "--status" : "--propose-backlog"+ } cannot both be given: a path ` +"imports exactly that path, and the sweep works over the session roots this build " +"declares. Drop one.",);}- if (sessionPath === null && !discover && !status) {+ if (sessionPath === null && !discover && !status && !proposeBacklog) {throw new CliError("brain import-session requires a <path> argument, or --discover / --status to sweep the " +"session stores this machine declares.",@@ -318,7 +343,22 @@ async function sweep(req: SweepRequest): Promise<number> {}if (!req.importAll) {- emitCoverage(discovery, { asJson: req.asJson, listGap: req.discover });+ // The distillation census is vault-side (the two ledgers plus the+ // continuity turns the mining lane itself reads), so it rides the+ // coverage report without re-hashing a single source log. Its own+ // deadline is fresh: discovery's safeguard was spent on the sweep.+ let mining: SessionMiningCensus;+ try {+ mining = buildMiningCensus(req.vault, {+ safeguard: createSafeguard({+ operation: OPERATION.sessions,+ timeoutMs: resolveSafeguardTimeoutMs(OPERATION.sessions, req.config ?? undefined),+ }),+ });+ } catch (exc) {+ return fail(`session mining census failed: ${(exc as Error).message ?? exc}`);+ }+ emitCoverage(discovery, mining, { asJson: req.asJson, listGap: req.discover });return 0;}@@ -363,6 +403,114 @@ async function sweep(req: SweepRequest): Promise<number> {return failures.length > 0 ? 1 : 0;}+interface ProposeBacklogRequest {+ readonly vault: string;+ readonly config: string | null;+ readonly asJson: boolean;+}++/**+ * The ask-once half: `--propose-backlog`.+ *+ * Batches the distill proposals the ALREADY IMPORTED undistilled backlog+ * raises into one envelope for the calling agent to present: candidates+ * whose dedup hash already exists in the vault are dropped from the ask+ * (each named), the rest collapse to one proposal per distinct hash across+ * sessions. The kernel runs no interactive loop - it emits the batch, and+ * the presenting agent asks. An empty backlog is an explicit empty+ * envelope and exits 0; it is an answer, not a failure.+ *+ * The candidates are the feedback markers in the sessions' mined turn+ * material (the same material the mining plan embeds) - the one candidate+ * form the kernel can see without a model. The dedup index is the shared+ * vault-wide one, so proposals cross-deduplicate against every capture+ * path exactly as the write-time gate does.+ */+async function proposeBacklogEnvelope(req: ProposeBacklogRequest): Promise<number> {+ let census: SessionMiningCensus;+ try {+ census = buildMiningCensus(req.vault, {+ safeguard: createSafeguard({+ operation: OPERATION.sessions,+ timeoutMs: resolveSafeguardTimeoutMs(OPERATION.sessions, req.config ?? undefined),+ }),+ });+ } catch (exc) {+ return fail(`session mining census failed: ${(exc as Error).message ?? exc}`);+ }++ const boundary = buildCaptureBoundary(req.vault);+ const candidates: BacklogCandidate[] = [];+ for (const row of census.byRuntime) {+ for (const sessionId of row.undistilledSessionIds) {+ const material = minedTurnMaterial(req.vault, sessionId, boundary);+ // A census undistilled row is minable by construction; the null arm+ // is the boundary changing between the two reads, and contributes+ // no candidates rather than a guess.+ if (material === null) continue;+ for (const turn of material) {+ for (const marker of discoverMarkers(turn.text)) {+ if (!isFeedbackMarker(marker)) continue;+ candidates.push({+ sessionId,+ topic: marker.topic,+ signal: marker.signal,+ principle: marker.principle,+ ...(marker.scope !== undefined && marker.scope !== "" ? { scope: marker.scope } : {}),+ });+ }+ }+ }+ }++ let dedupIndex: Map<string, DedupIndexEntry>;+ try {+ dedupIndex = await loadDedupIndex(req.vault);+ } catch (exc) {+ return fail(`backlog proposal dedup index failed: ${(exc as Error).message ?? exc}`);+ }++ const envelope = buildBacklogProposals(candidates, dedupIndex);++ if (req.asJson) {+ okJson({+ undistilled: census.undistilled,+ proposals: envelope.proposals.map((p) => ({+ dedup_hash: p.dedupHash,+ topic: p.topic,+ signal: p.signal,+ principle: p.principle,+ ...(p.scope !== undefined ? { scope: p.scope } : {}),+ sessions: p.sessions,+ })),+ dropped: envelope.dropped.map((d) => ({+ session_id: d.sessionId,+ topic: d.topic,+ dedup_hash: d.dedupHash,+ })),+ report_only_sessions: envelope.reportOnlySessions,+ });+ return 0;+ }++ ok(`undistilled: ${census.undistilled}`);+ ok(`proposals: ${envelope.proposals.length}`);+ let number = 1;+ for (const proposal of envelope.proposals) {+ ok(`proposal ${number}: ${proposal.topic}`);+ ok(` signal: ${proposal.signal}`);+ ok(` principle: ${proposal.principle}`);+ if (proposal.scope !== undefined) ok(` scope: ${proposal.scope}`);+ ok(` sessions: ${proposal.sessions.join(", ")}`);+ number += 1;+ }+ ok(`dropped as already known: ${envelope.dropped.length}`);+ for (const drop of envelope.dropped) info(` ${drop.sessionId}: ${drop.topic}`);+ ok(`sessions with nothing new: ${envelope.reportOnlySessions.length}`);+ for (const sessionId of envelope.reportOnlySessions) info(` ${sessionId}`);+ return 0;+}+/** `path -> the runtime whose declared root contains it`, recorded in the ledger. */function recordUnderDeclaredRoots(vault: string, paths: ReadonlyArray<string>): void {const records: SessionImportRecord[] = [];@@ -421,9 +569,33 @@ function coverageJson(row: RuntimeCoverage): Record<string, unknown> {};}+/** One mining row, as the JSON surface serialises it. */+function miningRowJson(row: SessionMiningCensus["byRuntime"][number]): Record<string, unknown> {+ return {+ runtime: row.runtime,+ distilled: row.distilled,+ known_nothing_new: row.knownNothingNew,+ undistilled: row.undistilled,+ refused: row.refused,+ };+}++/** The distillation census, as the JSON surface serialises it. */+function miningJson(census: SessionMiningCensus): Record<string, unknown> {+ return {+ imported: census.imported,+ distilled: census.distilled,+ known_nothing_new: census.knownNothingNew,+ undistilled: census.undistilled,+ refused: census.refused,+ by_runtime: census.byRuntime.map(miningRowJson),+ };+}+/** Report what the sweep found, without importing anything. */function emitCoverage(discovery: SessionDiscovery,+ mining: SessionMiningCensus,opts: { asJson: boolean; listGap: boolean },): void {if (opts.asJson) {@@ -434,6 +606,7 @@ function emitCoverage(unparsable: discovery.unparsable,unreadable: discovery.unreadable,by_runtime: discovery.byRuntime.map(coverageJson),+ mining: miningJson(mining),...(opts.listGap? { would_import: discovery.byRuntime.flatMap((row) => [...row.gapPaths]) }: {}),@@ -444,12 +617,25 @@ function emitCoverage(ok(`imported: ${discovery.imported}`);ok(`gap: ${discovery.gap}`);if (discovery.unparsable > 0) ok(`unparsable: ${discovery.unparsable}`);+ if (mining.imported > 0) {+ ok(+ `mining: ${mining.distilled} distilled, ${mining.knownNothingNew} known-nothing-new, ` ++ `${mining.undistilled} undistilled`,+ );+ }for (const row of discovery.byRuntime) {ok(`runtime: ${row.runtime}`);ok(` found: ${row.found}`);ok(` imported: ${row.imported}`);ok(` gap: ${row.gap}`);if (row.unparsable > 0) ok(` unparsable: ${row.unparsable}`);+ const minedRow = mining.byRuntime.find((m) => m.runtime === row.runtime);+ if (minedRow !== undefined) {+ ok(` distilled: ${minedRow.distilled}`);+ ok(` known-nothing-new: ${minedRow.knownNothingNew}`);+ ok(` undistilled: ${minedRow.undistilled}`);+ if (minedRow.refused > 0) ok(` refused: ${minedRow.refused}`);+ }// Where it looked. "found: 0" without the root it looked under is an// answer an operator cannot act on or disbelieve.for (const root of row.roots) {diff --git a/src/cli/brain/verbs/page-dedup.ts b/src/cli/brain/verbs/page-dedup.tsindex ef52f473..b3e663bf 100644--- a/src/cli/brain/verbs/page-dedup.ts+++ b/src/cli/brain/verbs/page-dedup.ts@@ -20,16 +20,20 @@ import {import { brainVerbContext, fail, okJson, parse } from "../helpers.ts";/**- * One merge attempt that refused to land: the cluster's dedup key and the- * named error. A chain refusal (MergeChainError) writes nothing, so the- * secondary is left exactly as it was. A link-pass refusal can arrive- * AFTER the `merged_into:` stamp has landed on the secondary - by then- * the merge is half-applied, and {@link DedupApplyFailure.wikilinksUpdated}- * carries how many files the rewrite repointed before the write that- * refused.+ * One merge attempt that refused to land: the cluster's dedup key, the+ * SECONDARY that refused (review L8 - a cluster can hold several, and a+ * report that names only the cluster leaves the operator guessing which+ * page is still awaiting a decision), and the named error. A chain refusal+ * (MergeChainError) writes nothing, so the secondary is left exactly as it+ * was. A link-pass refusal can arrive AFTER the `merged_into:` stamp has+ * landed on the secondary - by then the merge is half-applied, and+ * {@link DedupApplyFailure.wikilinksUpdated} carries how many files the+ * rewrite repointed before the write that refused.*/export interface DedupApplyFailure {readonly key: string;+ /** The id of the secondary whose merge refused. */+ readonly secondary: string;readonly message: string;/*** Files the link pass rewrote before the write that refused, present@@ -76,6 +80,7 @@ export function applyDedupCandidates(if (partial !== undefined) wikilinksTouched += partial;failures.push({key: c.key,+ secondary: s.id,message: (err as Error)?.message ?? String(err),...(partial === undefined ? {} : { wikilinksUpdated: partial }),});@@ -146,6 +151,7 @@ export async function cmdBrainPageDedup(argv: string[]): Promise<number> {wikilinks_updated: applied.wikilinksUpdated,failures: applied.failures.map((f) => ({key: f.key,+ secondary: f.secondary,message: f.message,...(f.wikilinksUpdated === undefined ? {} : { wikilinks_updated: f.wikilinksUpdated }),})),@@ -157,7 +163,9 @@ export async function cmdBrainPageDedup(argv: string[]): Promise<number> {for (const f of applied.failures) {const partial =f.wikilinksUpdated === undefined ? "" : ` (wikilinks updated first: ${f.wikilinksUpdated})`;- process.stdout.write(`failed cluster ${f.key}: ${f.message}${partial}\n`);+ process.stdout.write(+ `failed cluster ${f.key}, secondary ${f.secondary}: ${f.message}${partial}\n`,+ );}return exitCode;}diff --git a/src/cli/brain/verbs/receipts.ts b/src/cli/brain/verbs/receipts.tsindex d1b05fc8..16de8e16 100644--- a/src/cli/brain/verbs/receipts.ts+++ b/src/cli/brain/verbs/receipts.ts@@ -80,9 +80,13 @@ function renderScanText(vault: string, olderThanDays: number): number {}function renderCompact(result: ReceiptCompactResult, json: boolean): number {+ // A lost audit record is a failure on every surface (L10): the shards+ // above are folded but the ledger carries no receipt_compaction record+ // for the run, and a silent exit 0 would read as a completed run.+ const failed = result.failures.length > 0 || result.auditError !== undefined;if (json) {process.stdout.write(`${JSON.stringify(compactView(result), null, 2)}\n`);- return result.failures.length === 0 ? 0 : 1;+ return failed ? 1 : 0;}process.stdout.write(`${RECEIPT_COMPACTION_OPERATION}${result.dryRun ? " preview" : ""} - cutoff ` +@@ -97,13 +101,23 @@ function renderCompact(result: ReceiptCompactResult, json: boolean): number {for (const failure of result.failures) {process.stdout.write(`! ${failure.shard}: ${failure.message}\n`);}+ if (result.auditError !== undefined) {+ process.stdout.write(`! ${result.auditError}\n`);+ }if (result.snapshotRunId !== undefined) {process.stdout.write(`recovery point: ${result.snapshotRunId}\n`);}+ if (result.snapshotBytes !== undefined && result.netDeltaBytes !== undefined) {+ process.stdout.write(+ `net: ${result.netDeltaBytes} bytes (ledger shrank ` ++ `${result.totals.bytesBefore - result.totals.bytesAfter}, ` ++ `recovery point added ${result.snapshotBytes})\n`,+ );+ }if (result.dryRun) {process.stdout.write("preview only - pass --apply to fold these shards\n");}- return result.failures.length === 0 ? 0 : 1;+ return failed ? 1 : 0;}export async function cmdBrainReceipts(argv: string[]): Promise<number> {diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.tsindex dda2e275..ed3fb588 100644--- a/src/cli/command-manifest.ts+++ b/src/cli/command-manifest.ts@@ -516,6 +516,7 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({flag("discover", "boolean"),flag("status", "boolean"),flag("all", "boolean"),+ flag("propose-backlog", "boolean"),flag("progress", "boolean"),flag("json", "boolean"),]),diff --git a/src/cli/search/outcome-render.ts b/src/cli/search/outcome-render.tsindex b0d81995..81194a71 100644--- a/src/cli/search/outcome-render.ts+++ b/src/cli/search/outcome-render.ts@@ -204,12 +204,36 @@ function noResultsLine(o: SearchOutcome): string {return `(${NO_RESULTS_LABEL}: ${why} - ${trail.empty.reason})`;}+/**+ * The transcript's refusal-count segment for the explain lead line, from+ * the trust assessment's per-lane aggregates: `refused 2 (arm 1, pool 1)`.+ * Present whenever refusal records ride the receipts - the shadow mode's+ * would-be refusals and, since the arm lane receipts its drops, an+ * enforce run's arm-lane drops alike - and absent otherwise, so a receipt+ * without refusal records renders exactly as it always did.+ */+function refusalSummary(assessment: SearchOutcome["memoryTrustAssessment"]): string {+ if (assessment?.refused === undefined) return "";+ const lanes = Object.entries(assessment.refusals_by_lane ?? {})+ .map(([lane, count]) => `${lane} ${count}`)+ .join(", ");+ return `${BULLET}refused ${assessment.refused}${lanes !== "" ? ` (${lanes})` : ""}`;+}+/*** The `--explain` block of the human transcript: the decision trace as* counts plus one line per excluded candidate, or the single line naming* why there is no trace to show. The trust assessment is the same* exclusion set folded into a reason histogram, so the transcript renders* the trace (the superset) and `--json` carries both receipts verbatim.+ *+ * When the gate is measuring - shadow, the mode the rollout recommends+ * first - nothing is excluded, so the enforce-shaped lead line alone+ * would read exactly like a clean run. The mode and the measured+ * refusals (per lane) are therefore named on the lead line, and every+ * refusal record prints as a row the way exclusions do; an enforce run+ * whose arm dropped an arrival renders the same refusal rows without the+ * mode stamp.*/function explainLines(o: SearchOutcome,@@ -228,12 +252,19 @@ function explainLines(return [`explain: ${why}`, ...overlap, ""];}const lines = [- `explain: evaluated ${trace.evaluated}${BULLET}surfaced ${trace.surfaced}${BULLET}excluded ${trace.excluded}`,+ `explain: evaluated ${trace.evaluated}${BULLET}surfaced ${trace.surfaced}${BULLET}excluded ${trace.excluded}` ++ (trace.mode !== undefined ? `${BULLET}mode ${trace.mode}` : "") ++ refusalSummary(assessment),...overlap,];for (const exclusion of trace.exclusions) {lines.push(` excluded: ${exclusion.path}${BULLET}${exclusion.reasons.join(", ")}`);}+ for (const refusal of trace.refusals ?? []) {+ lines.push(+ ` refused: ${refusal.path}${BULLET}${refusal.lane}${BULLET}${refusal.reasons.join(", ")}`,+ );+ }lines.push("");return lines;}diff --git a/src/cli/search/verbs/query.ts b/src/cli/search/verbs/query.tsindex 1fae2fbd..6e9e94f8 100644--- a/src/cli/search/verbs/query.ts+++ b/src/cli/search/verbs/query.ts@@ -194,7 +194,10 @@ export async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<numbeconst explainOptions = {explain: flagBoolean(flags, "explain"),crossVault: isGlobal,- trustGateEnabled: cfg.recall.retrievalTrustGateEnabled,+ // Engaged means the resolved mode is not off - the legacy boolean+ // alone cannot answer that (an explicit search_trust_gate_mode runs+ // the gate with the boolean left false).+ trustGateEnabled: cfg.recall.searchTrustGateMode !== undefined,};if (flagBoolean(flags, "json")) {diff --git a/src/core/brain/capture/inbox-drain.ts b/src/core/brain/capture/inbox-drain.tsindex bdab8466..c09bb82c 100644--- a/src/core/brain/capture/inbox-drain.ts+++ b/src/core/brain/capture/inbox-drain.ts@@ -19,6 +19,12 @@* staged captures and is a no-op (the processed marker is the idempotency* key). Unroutable items are reported with a reason and left in place.*+ * The source-reference route sits behind the same storage-capacity+ * admission wall `brain_ingest_source` keeps: a vault measured at its+ * critical tier refuses the ingest route in BOTH modes (a dry run+ * previews the refusal faithfully), leaves the capture staged, and+ * still drains its non-ingest routes.+ ** ## The hub hook (t_23bd347d)** An apply-mode idea route that succeeds stages its routed page's area-hub@@ -39,6 +45,9 @@ import { ensureInsideVault } from "../../path-safety.ts";import { formatFrontmatter, parseFrontmatter, slugify } from "../../vault.ts";import { addObligation, obligationExists, parseCadence, ObligationError } from "../obligations.ts";import { ingestSource } from "../ingest/ingest.ts";+import { CAPACITY_GATE_DECISION, evaluateIngestAdmission } from "../storage/gate.ts";+import type { CapacityGateDecision } from "../storage/gate.ts";+import { MAINTENANCE_CAPACITY_REMEDY } from "../maintenance/lane.ts";import { isoSecond } from "../time.ts";import {loadHubPagePool,@@ -126,6 +135,7 @@ function classify(vault: string,note: CaptureNote,opts: DrainOptions,+ admission: CapacityGateDecision | null,): RoutePlan | UnroutableCapture {const body = note.body.trim();const firstLine = body.split("\n", 1)[0]!.trim();@@ -134,7 +144,7 @@ function classify(return planObligation(vault, firstLine, opts);}if (URL_SHAPED_RE.test(body)) {- return planSource(vault, body, opts);+ return planSource(vault, body, opts, admission);}return planIdea(vault, body, opts);}@@ -174,7 +184,30 @@ function planObligation(};}-function planSource(vault: string, url: string, opts: DrainOptions): RoutePlan {+/**+ * The source-reference route, behind the same admission wall+ * `brain_ingest_source` keeps: only a MEASURED critical tier refuses, so+ * a capture an agent could not ingest by hand is not sneaked in by a+ * drain, and a capture the wall would admit is not refused by one. The+ * refusal is decided at PLAN time (one admission evaluation per drain,+ * never per capture - the census is a full vault walk) so a dry run is a+ * faithful preview of it, the same posture as the declared-vocabulary+ * gate below. The remedy is the lane's own sentence, quoted rather than+ * restated, so every surface that refuses on capacity says the same+ * thing.+ */+function planSource(+ vault: string,+ url: string,+ opts: DrainOptions,+ admission: CapacityGateDecision | null,+): RoutePlan | UnroutableCapture {+ if (admission?.decision === CAPACITY_GATE_DECISION.skip) {+ return new UnroutableCapture(+ `ingest refused (capacity_exhausted): the vault measured at the ${admission.tier} ` ++ `storage tier. ${MAINTENANCE_CAPACITY_REMEDY}.`,+ );+ }return {classification: "source-reference",action: "ingest-source",@@ -278,9 +311,19 @@ export function drainInbox(vault: string, opts: DrainOptions): DrainReport {let skippedCorrupt = 0;// The hub page pool is a full-vault parse: loaded once, on the first idea.let hubPool: HubPagePool | undefined;+ // One admission evaluation per drain, not per capture: the census is a+ // full vault walk, and the answer cannot change between captures of+ // the same run. A null decision means no wall stands this pass - no+ // configured cap, or a storage block that could not be validated,+ // which is named on the operator's stderr rather than wedging the+ // inbox shut.+ const admission = evaluateIngestAdmission(vault, { now: opts.now });+ if (admission.reason !== undefined) {+ process.stderr.write(`capture inbox drain: ${admission.reason}\n`);+ }for (const note of listStagedCaptures(vault)) {- const plan = classify(vault, note, opts);+ const plan = classify(vault, note, opts, admission.decision);if (plan instanceof UnroutableCapture) {unroutable += 1;items.push({diff --git a/src/core/brain/config-template.ts b/src/core/brain/config-template.tsindex fc2c17e7..007067d9 100644--- a/src/core/brain/config-template.ts+++ b/src/core/brain/config-template.ts@@ -607,9 +607,9 @@ export const BRAIN_CONFIG_TEMPLATE: ReadonlyArray<BrainTemplateBlock> = Object.f"Storage capacity policy (`o2b brain storage`). An absent cap_bytes","means the vault is unbounded: nothing pauses and nothing refuses.","Setting a cap turns on the forecast tiers and the storage-capacity",- "doctor check; the four thresholds default to the constants the",- "forecast ships with, and the critical band must sit past the warn",- "band (a larger percent, a smaller day horizon).",+ "doctor check; the thresholds default to the constants the forecast",+ "ships with, and the critical band must sit past the warn band and",+ "the pause line (a larger percent, a smaller day horizon).",],emit: "commented-default",keys: [@@ -625,6 +625,12 @@ export const BRAIN_CONFIG_TEMPLATE: ReadonlyArray<BrainTemplateBlock> = Object.f"Percent of the cap where usage forecasts as critical. Must be","greater than warn_percent.",]),+ def("pause_percent", BRAIN_STORAGE_DEFAULTS.pause_percent, [+ "Percent of the cap where background production pauses: the",+ "maintenance lane skips, while interactive ingest still works.",+ "Defaults to the warn band, and must stay below critical_percent,",+ "which is where writes are refused outright.",+ ]),def("warn_days", BRAIN_STORAGE_DEFAULTS.warn_days, ["Days to the cap, at the measured trend, where usage forecasts as","warn.",diff --git a/src/core/brain/digest.ts b/src/core/brain/digest.tsindex 21cbcf49..8e55459b 100644--- a/src/core/brain/digest.ts+++ b/src/core/brain/digest.ts@@ -45,7 +45,11 @@ import { computeAgentSummary, type AgentSummaryEntry } from "./digest-agent-summimport { findMergeCandidates } from "./merge-candidates.ts";import { computeMostApplied } from "./most-applied.ts";import { BRAIN_RETIRED_REL, brainDirs, vaultRelative } from "./paths.ts";-import { collectMaintenanceActions } from "./maintenance/collect.ts";+import {+ collectMaintenanceActions,+ type MaintenanceCollectError,+ type MaintenanceCollectSignal,+} from "./maintenance/collect.ts";import type { ActionItem } from "./maintenance/action-scorer.ts";import { loadActiveMostAppliedSafe } from "./policy.ts";import { normaliseWikilinkTarget, renderPrefLink, type LinkOutputFormat } from "./wikilink.ts";@@ -302,6 +306,18 @@ export interface DigestJson {* needs doing. Independent of the window.*/readonly actions: ReadonlyArray<ActionItem>;+ /**+ * The maintenance scans that refused while the digest was collected+ * (review M10), rendered as `{scan, message}` rows - the same shape the+ * operator-summary and doctor rows carry. A thin `actions` list must be+ * readable as healthy rather than indistinguishable from a scan that+ * failed. Present only when at least one scan refused, so a clean+ * payload stays byte-identical.+ */+ readonly collect_errors?: ReadonlyArray<{+ readonly scan: MaintenanceCollectSignal;+ readonly message: string;+ }>;/*** Aggregate vault trust verdict (v0.10.16). Absent when no* doctor input was threaded into the digest call; consumers that@@ -380,6 +396,14 @@ export function renderDigest(vault: string, opts: RenderDigestOptions = {}): Renmost_applied: data.most_applied,connection_health: data.connection_health,actions: data.actions,+ ...(data.collect_errors.length > 0+ ? {+ collect_errors: data.collect_errors.map((e) => ({+ scan: e.signal,+ message: e.message,+ })),+ }+ : {}),// Guarded: dreamSummary may arrive from an untyped JSON-RPC// boundary (or from a future caller that has not populated// every array). Optional-chain the inner length read.@@ -442,6 +466,8 @@ interface DigestData {readonly most_applied: DigestJsonMostApplied;readonly connection_health: DigestJsonConnectionHealth;readonly actions: ReadonlyArray<ActionItem>;+ /** Raw collected failures; rendered on the wire as `{scan, message}` rows. */+ readonly collect_errors: ReadonlyArray<MaintenanceCollectError>;}function collectDigestData(@@ -604,6 +630,10 @@ function collectDigestData(entries: mostAppliedEntries,};+ // The collector's absorbed scan failures (review M10): observed here so+ // the digest can say why its actions list came back thin.+ const collectErrors: MaintenanceCollectError[] = [];+return {new_unconfirmed,confirmed,@@ -616,7 +646,11 @@ function collectDigestData(agent_summary,most_applied,connection_health: computeConnectionHealth(vault, inbound, preferences, refs),- actions: refs.keep(collectMaintenanceActions(vault), (action) => [action.target]),+ actions: refs.keep(+ collectMaintenanceActions(vault, { onError: (e) => collectErrors.push(e) }),+ (action) => [action.target],+ ),+ collect_errors: collectErrors,};}diff --git a/src/core/brain/distill/backlog-proposals.ts b/src/core/brain/distill/backlog-proposals.tsnew file mode 100644index 00000000..11c90ac0--- /dev/null+++ b/src/core/brain/distill/backlog-proposals.ts@@ -0,0 +1,157 @@+/**+ * The ask-once batching layer over the undistilled backlog+ * (session-mining-economy, t_0358af42).+ *+ * Vault-wide cross-session dedup already ships at WRITE time+ * (`buildDedupIndex` plus the commit-time gate): a recurring fact is+ * written once, whatever number of sessions raised it. What shipped+ * nowhere was dedup of the ASK - the operator was still asked about the+ * same material once per session that raised it, because suppression+ * happened only after the proposal reached the write gate.+ *+ * `buildBacklogProposals` is the pure core that moves dedup before the+ * ask. It takes the pending distill candidates raised by undistilled+ * sessions plus a `loadDedupIndex` result, drops every candidate whose+ * `computeDedupHash` already exists in the vault (naming each drop), and+ * collapses the remainder to ONE proposal per distinct hash across+ * sessions, naming every contributing session. The output is one batched+ * envelope for the calling agent to present; the kernel stays LLM-free+ * and runs no interactive loop - it groups, it does not ask.+ *+ * A session whose every candidate was already known appears in+ * `reportOnlySessions` and in no proposal: the report says "known,+ * nothing new" about it, which is a different statement from proposing+ * its material again.+ */++import { computeDedupHash, type DedupIndexEntry } from "../dedup-hash.ts";++/** One pending distill candidate raised by an undistilled backlog session. */+export interface BacklogCandidate {+ /** The portable session identity that raised the candidate. */+ readonly sessionId: string;+ readonly topic: string;+ readonly signal: "positive" | "negative";+ readonly principle: string;+ readonly scope?: string;+}++/** One candidate suppressed from the ask because the vault already holds it. */+export interface BacklogProposalDrop {+ readonly sessionId: string;+ readonly topic: string;+ readonly dedupHash: string;+}++/** One ask-once proposal: everything the vault does not yet hold, per hash. */+export interface BacklogProposal {+ readonly dedupHash: string;+ /**+ * The representative form of the hash class: the candidate whose+ * (topic, signal, principle, scope) tuple sorts first within the class,+ * so the proposal text cannot depend on the caller's input order. The+ * dedup hash normalizes case and width, so one class can hold+ * candidates whose raw fields differ.+ */+ readonly topic: string;+ readonly signal: "positive" | "negative";+ readonly principle: string;+ readonly scope?: string;+ /** Every session that raised this hash, sorted, unique. */+ readonly sessions: ReadonlyArray<string>;+}++/** The batched envelope: what to ask, what was suppressed, who raised nothing new. */+export interface BacklogProposals {+ readonly proposals: ReadonlyArray<BacklogProposal>;+ readonly dropped: ReadonlyArray<BacklogProposalDrop>;+ /** Sessions whose every candidate was already known. Sorted. */+ readonly reportOnlySessions: ReadonlyArray<string>;+}++/**+ * Batch the pending candidates ask-once. Empty input is an explicit empty+ * envelope, not silence. Deterministic: proposals sort by dedup hash, each+ * proposal's representative fields are the content-stable pick within its+ * hash class, drops sort by session then topic, so two runs over the same+ * backlog agree byte for byte regardless of the input order.+ */+export function buildBacklogProposals(+ pending: ReadonlyArray<BacklogCandidate>,+ dedupIndex: ReadonlyMap<string, DedupIndexEntry>,+): BacklogProposals {+ const dropped: BacklogProposalDrop[] = [];+ const sessionsByHash = new Map<string, Set<string>>();+ const representativeByHash = new Map<string, BacklogCandidate>();+ const sessionsWithProposals = new Set<string>();++ for (const candidate of pending) {+ const hash = computeDedupHash({+ topic: candidate.topic,+ signal: candidate.signal,+ principle: candidate.principle,+ scope: candidate.scope,+ });+ if (dedupIndex.has(hash)) {+ dropped.push({ sessionId: candidate.sessionId, topic: candidate.topic, dedupHash: hash });+ continue;+ }+ let sessions = sessionsByHash.get(hash);+ if (sessions === undefined) {+ sessions = new Set();+ sessionsByHash.set(hash, sessions);+ }+ const current = representativeByHash.get(hash);+ if (current === undefined || precedes(candidate, current)) {+ representativeByHash.set(hash, candidate);+ }+ sessions.add(candidate.sessionId);+ sessionsWithProposals.add(candidate.sessionId);+ }++ const raised = new Set(pending.map((candidate) => candidate.sessionId));+ const reportOnlySessions = [...raised]+ .filter((sessionId) => !sessionsWithProposals.has(sessionId))+ .toSorted();++ const proposals: BacklogProposal[] = [...sessionsByHash.keys()].toSorted().map((hash) => {+ const representative = representativeByHash.get(hash)!;+ return {+ dedupHash: hash,+ topic: representative.topic,+ signal: representative.signal,+ principle: representative.principle,+ // The hash unifies a missing scope with an empty one, so the+ // representative's raw empty string is dropped too: a proposal that+ // carried it or not depending on which candidate won the class+ // would not be byte-for-byte deterministic.+ ...(representative.scope ? { scope: representative.scope } : {}),+ sessions: [...sessionsByHash.get(hash)!].toSorted(),+ };+ });+ dropped.sort((a, b) =>+ a.sessionId === b.sessionId+ ? a.topic === b.topic+ ? a.dedupHash.localeCompare(b.dedupHash)+ : a.topic.localeCompare(b.topic)+ : a.sessionId.localeCompare(b.sessionId),+ );++ return { proposals, dropped, reportOnlySessions };+}++/**+ * Whether `a`'s (topic, signal, principle, scope) tuple sorts before `b`'s,+ * field by field in code-unit order. The missing and the empty scope+ * compare equal, mirroring the dedup hash's own unification, so a class+ * whose members differ only there still picks one representative whatever+ * the input order.+ */+function precedes(a: BacklogCandidate, b: BacklogCandidate): boolean {+ if (a.topic !== b.topic) return a.topic < b.topic;+ if (a.signal !== b.signal) return a.signal < b.signal;+ if (a.principle !== b.principle) return a.principle < b.principle;+ const aScope = a.scope ?? "";+ const bScope = b.scope ?? "";+ return aScope < bScope;+}diff --git a/src/core/brain/doctor/storage-capacity.ts b/src/core/brain/doctor/storage-capacity.tsindex 26cb8c4b..22c3a1ba 100644--- a/src/core/brain/doctor/storage-capacity.ts+++ b/src/core/brain/doctor/storage-capacity.ts@@ -95,6 +95,7 @@ export const storageCapacityCheck: DoctorCheck = {const usage = measureStorageUsage(ctx.vault, {now: ctx.now,...(ctx.dbPath !== undefined ? { storePath: ctx.dbPath } : {}),+ ...(ctx.configPath !== undefined ? { configPath: ctx.configPath } : {}),});if (!usage.complete) {pushUncertain(out.uncertain, {diff --git a/src/core/brain/document-retire.ts b/src/core/brain/document-retire.tsindex 22c9853f..65ac70fe 100644--- a/src/core/brain/document-retire.ts+++ b/src/core/brain/document-retire.ts@@ -8,18 +8,24 @@** 1. retarget every inbound `[[stale-copy]]` wikilink to the* canonical spelling (the report is carried, never swallowed);+ * a write that could not land refuses the copy before the+ * removal, so no retired page strands the link it could not+ * repoint (review M7);* 2. remove the stale page (per-file removal via the shared* `removeFile`, so a failure is one copy's failure);- * 3. drop the copy's content-manifest entry, round-tripping the- * contract the manifest was read under - the same t_586d5d8b rule- * deleteBySource applies: a removal never re-stamps the contract- * the surviving entries were recorded under. The drop is the- * resurrect guard: with the entry gone, the retired source- * classifies `new` again, so no later ingest can claim the- * retired page's records are current; the source can only return- * as a deliberate full re-ingest;+ * 3. MARK the copy's source manifest entry `retired_into` the+ * canonical page, round-tripping the contract the manifest was+ * read under - the same t_586d5d8b rule deleteBySource applies: a+ * retirement never re-stamps the contract the surviving entries+ * were recorded under. The marker is the resurrect guard (review+ * H3): dropping the entry would classify the unchanged source+ * `new` again and the next batch plan would re-ingest it, quietly+ * undoing the retirement. With the marker, the unchanged source+ * classifies `retired` and no plan queues it; a real byte change+ * re-admits it as `modified`, and a deliberate re-ingest re-records+ * the entry without the marker;* 4. append one `source_invalidation` continuity record PER COPY- * naming canonical id, retired id, dropped manifest key and the+ * naming canonical id, retired id, marked manifest key and the* retained source-archive hash - the audit the page-dedup merge* never needed, because a merge keeps both pages and this* operation deletes one.@@ -36,18 +42,19 @@* collected as `{path, message}`, and the remaining copies still* retire. The message says WHERE the copy ended up: a failure BEFORE* the page's own removal (the existence, frontmatter or canonical- * checks) leaves the copy on disk; a failure AFTER it - the manifest- * drop, the continuity record - reports the page as REMOVED, because it- * is, and names the step that lost it. Copy-level input mistakes (a- * path equal to its own canonical) are caller bugs and refuse loudly- * before any work.+ * checks, the ambiguous-basename refusal, an inbound-link rewrite that+ * could not land) leaves the copy on disk; a failure AFTER it - the+ * manifest marker, the continuity record - reports the page as REMOVED,+ * because it is, and names the step that lost it. Copy-level input+ * mistakes (a path equal to its own canonical) are caller bugs and+ * refuse loudly before any work.** Dry runs never reach this module: the hygiene apply plan previews* retire findings without executing them, and the surfaces' count* guard runs on the selected findings before this applier is called.*/-import { existsSync } from "node:fs";+import { existsSync, statSync } from "node:fs";import { join } from "node:path";import { parseFrontmatter } from "../vault.ts";@@ -82,16 +89,19 @@ function linkTarget(pagePath: string): string {}/**- * Why a copy's inbound-link pass did not run: another page in the vault+ * Why a copy whose inbound-link pass cannot run unambiguously is REFUSED+ * (storage lifecycle, PR #257 review L5): another page in the vault* carries the retired copy's basename, so the bare `[[basename]]`* spelling resolves to more than one page and a retarget keyed on the* spelling alone would repoint links that meant the OTHER page - edits* to pages outside the detector's near-duplicate scope, made on a guess.- * The copy still retires; the skipped pass leaves a stale link, which a- * reader can discover, where a mis-pointed link misleads silently.+ * Removing the page anyway is worse than skipping the rewrite: every+ * leftover `[[basename]]` would then RESOLVE to the namesake, silently+ * re-pointed by a deletion instead of left visibly stale. So the copy is+ * refused and stays on disk; the finding comes back to the operator.*/-export const RETIRE_LINKS_SKIPPED_BASENAME_COLLISION =- "another page carries the same basename; the bare-basename link spelling is ambiguous";+export const RETIRE_BASENAME_COLLISION_REFUSAL =+ "another page carries the same basename, so the bare-basename link spelling is ambiguous";/** Frontmatter key carrying the digest of the archived source bytes. */const SOURCE_ARCHIVE_HASH_KEY = "source_archive_hash";@@ -127,23 +137,45 @@ export interface RetiredDocumentCopy {readonly canonical: string;/** The source path whose manifest entry was targeted, or `null`. */readonly manifestKey: string | null;- /** True when the manifest was rewritten without the entry. */- readonly manifestDropped: boolean;+ /** True when the manifest was rewritten to carry the `retired_into` marker. */+ readonly manifestRetired: boolean;/**- * The full retarget report for this copy's inbound wikilinks, `null`- * when the pass was SKIPPED - {@link RetiredDocumentCopy.wikilinksSkipped}- * names why. `null` is never a silent zero: a skipped pass left every- * inbound link exactly as it was.+ * The retired page's size in bytes, measured BEFORE its removal - the+ * freed half of the run's disk accounting (review M6).*/- readonly wikilinks: WikilinkRetargetReport | null;- /** Why the link pass did not run, `null` when it ran. */- readonly wikilinksSkipped: string | null;+ readonly pageBytes: number;+ /**+ * The full retarget report for this copy's inbound wikilinks. A copy+ * whose pass could not run unambiguously - the basename collision - is+ * refused before any write and lands in {@link RetireDocumentsResult.failed}+ * instead, so a report row always carries a pass that ran.+ */+ readonly wikilinks: WikilinkRetargetReport;/** Digest named in the audit; from the page's frontmatter, or `null`. */readonly sourceArchiveHash: string | null;/** The id of the `source_invalidation` continuity record this retirement appended. */readonly auditRecordId: string;}+/**+ * The run's disk accounting (review M6, retire half): the destructive span+ * archives `Brain/` into `Brain/.snapshots/` - a surface the storage census+ * counts - BEFORE it frees anything, so removal counts alone can read as a+ * shrink on a run that actually grew the vault. The result carries both+ * sides and the net between them.+ */+export interface RetireDiskAccounting {+ /**+ * Bytes of the snapshot archive the gate wrote for this run, or `null`+ * when no snapshot was taken (a run with no copies).+ */+ readonly snapshotBytes: number | null;+ /** Summed bytes of the pages the run removed, measured before removal. */+ readonly removedBytes: number;+ /** `snapshotBytes - removedBytes`; `null` beside a `null` snapshot size. */+ readonly netBytes: number | null;+}+export interface RetireDocumentsResult {readonly retired: ReadonlyArray<RetiredDocumentCopy>;/**@@ -153,6 +185,12 @@ export interface RetireDocumentsResult {* lost it.*/readonly failed: ReadonlyArray<SourceCleanupFailure>;+ /**+ * The run's disk accounting (review M6): both sides and the net+ * between them, as the module header promises. Per-copy bytes ride+ * {@link RetiredDocumentCopy.pageBytes}; this carries the run totals.+ */+ readonly disk: RetireDiskAccounting;readonly snapshotRunId: string | null;readonly snapshotPath: string | null;readonly recoverability: RecoverabilityVerdict;@@ -207,6 +245,7 @@ export function retireDocumentCopies(return Object.freeze({retired: Object.freeze([]),failed: Object.freeze([]),+ disk: Object.freeze({ snapshotBytes: null, removedBytes: 0, netBytes: null }),snapshotRunId: null,snapshotPath: null,// Nothing to destroy, so the classifier says so - neither the@@ -256,9 +295,21 @@ export function retireDocumentCopies(},);+ // The disk accounting (review M6, retire half): the archive sits in a+ // counted surface, so the result states what the snapshot cost beside+ // what the removals freed.+ const removedBytes = retired.reduce((sum, copy) => sum + copy.pageBytes, 0);+ const snapshotBytes = existsSync(gated.snapshot.path) ? statSync(gated.snapshot.path).size : null;+ const disk: RetireDiskAccounting = Object.freeze({+ snapshotBytes,+ removedBytes,+ netBytes: snapshotBytes === null ? null : snapshotBytes - removedBytes,+ });+return Object.freeze({retired: Object.freeze(retired),failed: Object.freeze(failed),+ disk,snapshotRunId: gated.snapshot.runId,snapshotPath: gated.snapshot.path,recoverability: gated.recoverability,@@ -285,6 +336,9 @@ function retireOneCopy(if (!existsSync(abs)) {throw new Error(`retired copy not found on disk: ${copy.path}`);}+ // Measured before anything acts: the freed half of the run's disk+ // accounting (review M6).+ const pageBytes = statSync(abs).size;const facts = readCopyFacts(vault, copy.path);if (facts === null) {throw new Error(`retired copy frontmatter does not parse: ${copy.path}`);@@ -297,17 +351,37 @@ function retireOneCopy(}// 1. Links first, so no reader between steps ever sees a stranded- // `[[stale-copy]]`. Failure to write a file is collected in the- // report, not thrown - the pass has already rewritten the files- // before it (the retargetWikilinks argument, verbatim). The pass- // keys on the bare basename, so when another page carries the same- // basename the spelling is ambiguous: rewriting it would repoint- // links that meant the other page. The copy then retires without its- // link pass and the report names why - a stale link is discoverable,- // a mis-pointed one misleads silently.- const wikilinks = basenameCollides(vault, linkTarget(copy.path), copy.path)- ? null- : retargetWikilinks(vault, [{ from: linkTarget(copy.path), to: linkTarget(copy.canonical) }]);+ // `[[stale-copy]]`. The pass keys on the bare basename, so when another+ // page carries the same basename the spelling is ambiguous: rewriting it+ // would repoint links that meant the other page, and removing the page+ // anyway would leave every leftover spelling resolving to the namesake+ // silently (review L5). The copy is refused BEFORE any write and stays+ // on disk - a refusal is one copy's collected failure.+ if (basenameCollides(vault, linkTarget(copy.path), copy.path)) {+ throw new Error(+ `retirement refused, the copy stays in place: ${RETIRE_BASENAME_COLLISION_REFUSAL}: ${copy.path}`,+ );+ }+ // Failure to write a file is collected in the report, not thrown - the+ // pass has already rewritten the files before it (the retargetWikilinks+ // argument, verbatim).+ const wikilinks = retargetWikilinks(vault, [+ { from: linkTarget(copy.path), to: linkTarget(copy.canonical) },+ ]);+ // A retarget write that failed refuses the copy (review M7): retiring+ // the page over an unreplicable link would strand exactly the inbound+ // reference the pass could not repoint - the design's "a copy whose+ // retarget fails stays in place". The refusal is BEFORE the removal, so+ // the copy stays on disk as one collected per-copy failure; the files+ // the pass DID rewrite now point at the canonical, which exists, so no+ // link is stranded by the partial rewrite either.+ if (wikilinks.failed.length > 0) {+ throw new Error(+ `inbound wikilink retarget left ${wikilinks.failed.length} file(s) unwritten; ` ++ `the copy stays in place: ` ++ wikilinks.failed.map((f) => `${f.path} (${f.reason})`).join("; "),+ );+ }// 2. Remove the stale page. `false` here means the page vanished// between the existence check and the unlink - that is a failure to@@ -320,39 +394,39 @@ function retireOneCopy(// leave the copy on disk, so it is rethrown AS that fact - the report// must never send an operator looking for a page that is not there.try {- // 3. Drop the manifest entry for the copy's source, round-tripping- // the read contract (t_586d5d8b): the surviving entries keep the- // fingerprint they were recorded under, so the owed one-time- // reprocess is never forfeited by a removal. The per-copy reason is- // worded from what actually happened - a drop, a refusal, an entry- // already absent, or nothing to drop at all - never from the key's- // existence alone.- let manifestDropped = false;+ // 3. Mark the manifest entry for the copy's source `retired_into` the+ // canonical page, round-tripping the read contract (t_586d5d8b): the+ // surviving entries keep the fingerprint they were recorded under, so+ // the owed one-time reprocess is never forfeited by a retirement. The+ // per-copy reason is worded from what actually happened - a mark, a+ // refusal, an entry already marked, an entry absent, or no key+ // recorded at all - never from the key's existence alone.+ let manifestRetired = false;let manifestClause: string;if (facts.sourcePath === null) {manifestClause = "no manifest key recorded";} else if (!existsSync(manifestPath(vault))) {- manifestClause = `manifest absent, nothing to drop for ${facts.sourcePath}`;+ manifestClause = `manifest absent, nothing to mark for ${facts.sourcePath}`;} else {const manifest = readManifest(vault);const entries = { ...manifest.entries };- if (entries[facts.sourcePath] === undefined) {- manifestClause = `manifest key ${facts.sourcePath} already absent`;+ const existing = entries[facts.sourcePath];+ if (existing === undefined) {+ manifestClause = `manifest key ${facts.sourcePath} absent, nothing to mark`;+ } else if (existing.retired_into === copy.canonical) {+ // The entry already carries THIS retirement's marker: the serialized+ // bytes below would be identical, and the clause names that+ // success-shaped fact - a concurrent retirer got there first.+ manifestClause = `manifest key ${facts.sourcePath} already marked retired`;} else {- delete entries[facts.sourcePath];- manifestDropped = writeManifestAtomic(vault, entries, manifest.contract);- // A `false` from the atomic write after deleting a PRESENT key means- // the serialized bytes were already identical - the key was already- // gone on disk, i.e. a concurrent retirer got there first. The- // reason names that success-shaped fact, not a failure to drop.- manifestClause = manifestDropped- ? `dropped manifest key ${facts.sourcePath}`- : `manifest already without key ${facts.sourcePath}`;+ entries[facts.sourcePath] = { ...existing, retired_into: copy.canonical };+ manifestRetired = writeManifestAtomic(vault, entries, manifest.contract);+ manifestClause = `marked manifest key ${facts.sourcePath} retired into ${copy.canonical}`;}}// 4. The provenance record: one per copy, naming what survives, what- // went, what was dropped and what evidence is retained.+ // went, what was marked and what evidence is retained.const agent = opts.agent?.trim() ? opts.agent.trim() : DEFAULT_AGENT;const record = appendContinuitySourceInvalidation(vault, {createdAt: isoSecond(opts.now ?? new Date()),@@ -371,9 +445,9 @@ function retireOneCopy(path: copy.path,canonical: copy.canonical,manifestKey: facts.sourcePath,- manifestDropped,+ manifestRetired,+ pageBytes,wikilinks,- wikilinksSkipped: wikilinks === null ? RETIRE_LINKS_SKIPPED_BASENAME_COLLISION : null,sourceArchiveHash: facts.archiveHash,auditRecordId: record.id,});@@ -382,6 +456,7 @@ function retireOneCopy(throw new Error(`retirement removed ${copy.path}, then a post-removal step failed ` +`(the copy is not on disk): ${detail}`,+ { cause: error },);}}diff --git a/src/core/brain/explorer-console.ts b/src/core/brain/explorer-console.tsnew file mode 100644index 00000000..ca2fdc62--- /dev/null+++ b/src/core/brain/explorer-console.ts@@ -0,0 +1,272 @@+/**+ * §14b Live-counts console collector (session-mining-economy, t_15ebcfc1).+ *+ * One read-only composition of the counts the loopback explorer's+ * console renders and no surface served before:+ *+ * - the recall-telemetry summary (`summarizeRecallTelemetry` verbatim);+ * - the most-recalled artifacts, aggregated by the shipped+ * `deriveUsageSignals` and labelled with the id/path identity the+ * telemetry records themselves carry, under a named cap;+ * - a written-per-day histogram over a named 30-day window, bucketed+ * from continuity record `createdAt` - every continuity append is a+ * write event, whatever its kind;+ * - the latest report-snapshot date per surface, fail-soft: with the+ * feature off the section reads as "disabled", never as empty data+ * an operator could mistake for "enabled, nothing written yet".+ *+ * Output is deterministic GIVEN the same `opts.now` and frozen like+ * `collectExplorerData`, so the live HTTP handler cannot mutate it.+ * The live route injects no clock: every GET stamps its own render+ * time into `generated_at` and the histogram window rolls at the UTC+ * day boundary, so two GETs agree on every recorded section but never+ * on the stamp. Byte-for-byte reproducibility across calls holds only+ * for callers that inject one `now`.+ *+ * Reads only. This module never appends a continuity record, never+ * captures a report delta, and never writes: a viewer must not fire the+ * lazy writes the stores it reads are allowed to fire.+ */++import { existsSync, readdirSync } from "node:fs";+import { join } from "node:path";++import { listContinuityRecords } from "./continuity/store.ts";+import { deriveUsageSignals, type SourceUsage } from "./continuity/usage-signal.ts";+import { brainDirs } from "./paths.ts";+import {+ listRecallTelemetry,+ summarizeRecallTelemetry,+ type RecallTelemetrySummary,+} from "./recall-telemetry.ts";+import { loadLatestReportSnapshot, reportSnapshotsEnabled } from "./report-snapshot.ts";++export const EXPLORER_CONSOLE_SCHEMA_VERSION = 1 as const;++/** Calendar days the written-per-day histogram covers, ending at `now`. */+export const EXPLORER_CONSOLE_WINDOW_DAYS = 30;++/**+ * Most-recalled entries reported. A named cap, not a query parameter:+ * query parameters add surface to a read-only console for a bound the+ * house can name in one constant.+ */+export const EXPLORER_CONSOLE_MOST_RECALLED_LIMIT = 10;++const DAY_MS = 86_400_000;++export interface ExplorerConsoleMostRecalled {+ /** Artifact id from the telemetry payload, or null when unnamed. */+ readonly id: string | null;+ /** Artifact path from the telemetry payload, or null when unnamed. */+ readonly path: string | null;+ readonly access_count: number;+ /** ISO timestamp of the most recent recorded recall. */+ readonly last_access_at: string;+}++export interface ExplorerConsoleDayBucket {+ /** UTC calendar date, `YYYY-MM-DD`. */+ readonly date: string;+ readonly count: number;+}++export interface ExplorerConsoleSnapshotSurface {+ readonly surface: string;+ /** Latest valid snapshot date, or null when the surface has none yet. */+ readonly latest_date: string | null;+}++/**+ * "disabled" while `report_snapshots_enabled` is off - a state statement,+ * distinct from "enabled with no snapshots yet".+ */+export type ExplorerConsoleReportSnapshots =+ | { readonly status: "disabled" }+ | {+ readonly status: "ok";+ readonly surfaces: ReadonlyArray<ExplorerConsoleSnapshotSurface>;+ };++export interface ExplorerConsolePayload {+ readonly schema_version: typeof EXPLORER_CONSOLE_SCHEMA_VERSION;+ readonly generated_at: string;+ readonly window_days: number;+ readonly window_since: string;+ readonly window_until: string;+ readonly telemetry: RecallTelemetrySummary;+ readonly most_recalled: ReadonlyArray<ExplorerConsoleMostRecalled>;+ readonly written_per_day: ReadonlyArray<ExplorerConsoleDayBucket>;+ readonly report_snapshots: ExplorerConsoleReportSnapshots;+}++export interface CollectExplorerConsoleOptions {+ /** Wall clock for `generated_at` and the histogram window. Defaults to `new Date()`. */+ readonly now?: Date;+ /** Config path for the report-snapshot enablement lookup. */+ readonly configPath?: string;+}++export function collectExplorerConsole(+ vault: string,+ opts: CollectExplorerConsoleOptions = {},+): ExplorerConsolePayload {+ const now = opts.now ?? new Date();+ const windowDays = windowDates(now);+ const since = `${windowDays[0]!}T00:00:00.000Z`;+ const until = `${windowDays[windowDays.length - 1]!}T23:59:59.999Z`;+ return Object.freeze({+ schema_version: EXPLORER_CONSOLE_SCHEMA_VERSION,+ generated_at: now.toISOString(),+ window_days: EXPLORER_CONSOLE_WINDOW_DAYS,+ window_since: since,+ window_until: until,+ telemetry: summarizeRecallTelemetry(vault),+ most_recalled: mostRecalled(vault),+ written_per_day: writtenPerDay(vault, since, until, windowDays),+ report_snapshots: reportSnapshots(vault, now, opts.configPath),+ });+}++/** The `window_days` UTC calendar dates ending at `now`'s date, oldest first. */+function windowDates(now: Date): ReadonlyArray<string> {+ const dates: string[] = [];+ for (let i = EXPLORER_CONSOLE_WINDOW_DAYS - 1; i >= 0; i--) {+ dates.push(new Date(now.getTime() - i * DAY_MS).toISOString().slice(0, 10));+ }+ return dates;+}++/**+ * Most-recalled artifacts over all recorded telemetry.+ *+ * `deriveUsageSignals` aggregates per artifact but registers ONE+ * aggregate object under BOTH identity spellings (id and path), so the+ * map entries are grouped back by aggregate identity - the same+ * object-identity join `usageForRecord` uses - and each group is+ * labelled from the roles the payload `top_artifacts` assign, which the+ * usage map alone cannot distinguish.+ */+function mostRecalled(vault: string): ReadonlyArray<ExplorerConsoleMostRecalled> {+ const signals = deriveUsageSignals(vault);+ const roles = identityRoles(vault);+ const groups = new Map<SourceUsage, Set<string>>();+ for (const [key, usage] of signals) {+ const keys = groups.get(usage) ?? new Set<string>();+ keys.add(key);+ groups.set(usage, keys);+ }+ const entries: ExplorerConsoleMostRecalled[] = [];+ for (const [usage, keys] of groups) {+ let id: string | null = null;+ let path: string | null = null;+ for (const key of keys) {+ if (id === null && roles.ids.has(key)) {+ id = key;+ } else if (path === null && roles.paths.has(key)) {+ path = key;+ } else if (id === null) {+ // A key the roles scan has not seen can only come from a record+ // appended between the two reads of the append-only log; keep it+ // visible as the primary identity rather than dropping the label.+ id = key;+ }+ }+ entries.push({+ id,+ path,+ access_count: usage.accessCount,+ last_access_at: new Date(usage.lastAccessAtMs).toISOString(),+ });+ }+ entries.sort((a, b) => {+ if (a.access_count !== b.access_count) return b.access_count - a.access_count;+ return displayKey(a).localeCompare(displayKey(b));+ });+ return Object.freeze(entries.slice(0, EXPLORER_CONSOLE_MOST_RECALLED_LIMIT));+}++function displayKey(entry: ExplorerConsoleMostRecalled): string {+ return entry.id ?? entry.path ?? "";+}++/** The id and path roles the telemetry payloads themselves assign. */+interface IdentityRoles {+ readonly ids: ReadonlySet<string>;+ readonly paths: ReadonlySet<string>;+}++function identityRoles(vault: string): IdentityRoles {+ const ids = new Set<string>();+ const paths = new Set<string>();+ for (const record of listRecallTelemetry(vault)) {+ const artifacts = record.payload["top_artifacts"];+ if (!Array.isArray(artifacts)) continue;+ for (const artifact of artifacts) {+ if (artifact === null || typeof artifact !== "object") continue;+ const entry = artifact as Record<string, unknown>;+ if (typeof entry["id"] === "string" && entry["id"].length > 0) ids.add(entry["id"]);+ if (typeof entry["path"] === "string" && entry["path"].length > 0) paths.add(entry["path"]);+ }+ }+ return { ids, paths };+}++/**+ * Continuity records bucketed per UTC calendar day across the window.+ * Every kind counts - the histogram is about write activity, and every+ * append is a write. Days with no writes are reported as zero rather+ * than left out, so the bucket list is a fixed 30-day axis.+ */+function writtenPerDay(+ vault: string,+ since: string,+ until: string,+ windowDays: ReadonlyArray<string>,+): ReadonlyArray<ExplorerConsoleDayBucket> {+ const counts = new Map<string, number>();+ for (const record of listContinuityRecords(vault, { since, until })) {+ const date = record.createdAt.slice(0, 10);+ counts.set(date, (counts.get(date) ?? 0) + 1);+ }+ return Object.freeze(+ windowDays.map((date) => Object.freeze({ date, count: counts.get(date) ?? 0 })),+ );+}++/**+ * The latest valid snapshot date per surface. While the feature is off+ * the section is the "disabled" state statement and nothing is read;+ * once on, each `Brain/reports/<surface>/` directory names a surface+ * and the fail-soft reader answers its latest date (a torn file is+ * skipped there, never reported).+ */+function reportSnapshots(+ vault: string,+ now: Date,+ configPath?: string,+): ExplorerConsoleReportSnapshots {+ if (!reportSnapshotsEnabled(configPath)) return Object.freeze({ status: "disabled" });+ const reportsDir = join(brainDirs(vault).brain, "reports");+ const surfaces = existsSync(reportsDir)+ ? readdirSync(reportsDir, { withFileTypes: true })+ .filter((entry) => entry.isDirectory())+ .map((entry) => entry.name)+ .toSorted()+ : [];+ // Strictly before the day after `now`, matching the delta callers:+ // a snapshot dated today is the latest one the next run can diff against.+ const beforeDate = new Date(now.getTime() + DAY_MS).toISOString().slice(0, 10);+ return Object.freeze({+ status: "ok",+ surfaces: Object.freeze(+ surfaces.map((surface): ExplorerConsoleSnapshotSurface => {+ const latest = loadLatestReportSnapshot(vault, surface, beforeDate);+ return Object.freeze({+ surface,+ latest_date: latest === null ? null : latest.date,+ });+ }),+ ),+ });+}diff --git a/src/core/brain/explorer.ts b/src/core/brain/explorer.tsindex 6eb25ea0..243f1dda 100644--- a/src/core/brain/explorer.ts+++ b/src/core/brain/explorer.ts@@ -36,6 +36,8 @@ import {type BrainPreference,type BrainRetired,} from "./types.ts";+import { collectExplorerConsole } from "./explorer-console.ts";+import { buildOperatorSnapshot } from "./operator-snapshot.ts";export const EXPLORER_SCHEMA_VERSION = 1 as const;@@ -335,6 +337,36 @@ export function __resetExplorerTemplateCacheForTests(): void {// ---- Live server -------------------------------------------------------+/**+ * The Host names this server answers to, compared case-insensitively and+ * with any port (or none). Everything else - an attacker-chosen rebinding+ * name above all - is refused before any route runs.+ */+const LOOPBACK_HOST_NAMES: ReadonlySet<string> = new Set(["127.0.0.1", "localhost", "[::1]"]);++/**+ * The Host header's name part, lowercased, with any port suffix removed:+ * `127.0.0.1:7777` → `127.0.0.1`, `[::1]:7777` → `[::1]`, `LOCALHOST` →+ * `localhost`. Answers null for a missing header (required on HTTP/1.1+ * and this server's contract regardless) or a value whose name part is+ * not one plain host token - several unbracketed colons, a malformed+ * bracket form - none of which a loopback client sends.+ */+function loopbackHostOf(header: string | null): string | null {+ if (header === null) return null;+ const value = header.trim().toLowerCase();+ if (value === "") return null;+ if (value.startsWith("[")) {+ const end = value.indexOf("]");+ if (end === -1) return null;+ const name = value.slice(0, end + 1);+ const rest = value.slice(end + 1);+ return rest === "" || rest.startsWith(":") ? name : null;+ }+ const colon = value.indexOf(":");+ return colon === -1 ? value : value.slice(0, colon);+}+export interface LiveServerHandle {readonly url: string;readonly port: number;@@ -342,20 +374,63 @@ export interface LiveServerHandle {}/**- * Boot a loopback-only HTTP server that serves the explorer template- * and a JSON endpoint. Each GET re-reads the vault — no caches, no- * watchers, no write endpoints. Binding to `127.0.0.1` is non-optional;- * the explorer is single-user observability and should not be- * reachable across the network.+ * Additive options for the live server. `configPath` reaches the+ * operator snapshot's search-index and profile lookups; a caller that+ * passes nothing behaves exactly as before.+ */+export interface LiveServerOptions {+ readonly configPath?: string;+}++/**+ * Boot a loopback-only HTTP server that serves the explorer template,+ * the graph JSON, the console counts (`/counts.json`, from+ * {@link collectExplorerConsole}) and the operator snapshot+ * (`/status.json`, from `buildOperatorSnapshot`). Each GET re-reads the+ * vault — no caches, no watchers, no write endpoints. That includes the+ * snapshot route: every GET re-runs the dry-run dream inside+ * `buildOperatorSnapshot`, the most expensive read it composes, and+ * that bounded read-only cost is accepted rather than papered over with+ * a cache that would go stale silently. Binding to `127.0.0.1` is+ * non-optional, and the bind alone is not a browser boundary - a+ * DNS-rebinding page can resolve an attacker-chosen hostname to+ * `127.0.0.1` and reach the routes same-origin - so every request whose+ * `Host` header is not a loopback name (127.0.0.1, localhost, [::1],+ * any port) is refused with a named 403 before any route runs.+ *+ * The JSON routes surface a failed read by name: a 500 whose body+ * carries the error message, never an empty payload that could read as+ * "no records".** Throws when the port is unavailable; the CLI catches `EADDRINUSE`* and turns it into a friendly exit message.*/-export function buildLiveServer(vault: string, port: number): LiveServerHandle {+export function buildLiveServer(+ vault: string,+ port: number,+ options: LiveServerOptions = {},+): LiveServerHandle {+ const configPath = options.configPath;+ const snapshotOptions = configPath === undefined ? {} : { configPath };const server = Bun.serve({hostname: "127.0.0.1",port,- fetch(req) {+ async fetch(req) {+ // Host allowlist (security review, finding 2): the loopback bind+ // does not stop a rebinding browser, whose request arrives with an+ // attacker-chosen Host. Loopback names only, any port; the Host+ // header is required.+ const hostName = loopbackHostOf(req.headers.get("host"));+ if (hostName === null || !LOOPBACK_HOST_NAMES.has(hostName)) {+ return jsonResponse(+ JSON.stringify({+ error:+ "explorer: refusing a request whose Host header is not a loopback name " ++ "(127.0.0.1, localhost, [::1], any port)",+ }),+ 403,+ );+ }const url = new URL(req.url);if (url.pathname === "/" || url.pathname === "/index.html") {const graph = collectExplorerData(vault);@@ -370,6 +445,22 @@ export function buildLiveServer(vault: string, port: number): LiveServerHandle {headers: { "content-type": "application/json; charset=utf-8" },});}+ if (url.pathname === "/counts.json") {+ try {+ const counts = collectExplorerConsole(vault, snapshotOptions);+ return jsonResponse(JSON.stringify(counts));+ } catch (err) {+ return errorResponse("/counts.json", err);+ }+ }+ if (url.pathname === "/status.json") {+ try {+ const snapshot = await buildOperatorSnapshot(vault, snapshotOptions);+ return jsonResponse(JSON.stringify(snapshot));+ } catch (err) {+ return errorResponse("/status.json", err);+ }+ }return new Response("not found", { status: 404 });},});@@ -393,6 +484,23 @@ export function buildLiveServer(vault: string, port: number): LiveServerHandle {// ---- Helpers ------------------------------------------------------------+function jsonResponse(body: string, status = 200): Response {+ return new Response(body, {+ status,+ headers: { "content-type": "application/json; charset=utf-8" },+ });+}++/**+ * A route's failure surfaces by name: the failing route and the error+ * message travel in the body, so a broken section can never read as a+ * healthy one that simply has no records.+ */+function errorResponse(route: string, err: unknown): Response {+ const message = err instanceof Error ? err.message : String(err);+ return jsonResponse(JSON.stringify({ error: `${route}: ${message}` }), 500);+}+function deriveEdges(index: BacklinkIndex, knownIds: ReadonlySet<string>): ExplorerEdge[] {const edges: ExplorerEdge[] = [];const seen = new Set<string>();diff --git a/src/core/brain/extract-signals-prefilter.ts b/src/core/brain/extract-signals-prefilter.tsindex ebbf7e9e..822840df 100644--- a/src/core/brain/extract-signals-prefilter.ts+++ b/src/core/brain/extract-signals-prefilter.ts@@ -25,6 +25,13 @@* reason in `decisionModel.degraded`. The decision only filters turns this* module's own code produced; it never writes to the vault.*+ * Before any of that, the plan passes the mining ledger's zero-yield gate+ * (session-mining-ledger.ts): a session whose current turn material was+ * already mined and yielded nothing accepted is suppressed outright, on+ * every configuration, before a single request is built - the tombstone's+ * saving is the caller's whole mining round trip, which is more than the+ * pre-filter could ever measure.+ ** With the use not `off`, the envelope's schema hints also name the* optional `source_turn` item field, so the commit can say which turn an* item came from and `report` can measure regret (accepted items whose@@ -72,6 +79,7 @@ import {type SignalExtractionPlan,} from "./extract-signals.ts";import type { NeedsLlmStep } from "./llm-step.ts";+import { readMiningTombstone, type SessionMiningOutcome } from "./session-mining-ledger.ts";import { listSessionRawTurns } from "./session-recall.ts";import { estimateTokens as estimateTextTokens } from "./text/tokenizer.ts";import { emitTokenImpact, TOKEN_COUNT_METHOD } from "./token-impact.ts";@@ -81,6 +89,14 @@ const USE = "extract_prefilter" as const;/** `skipped.reason` when every mined turn scored below the threshold. */export const EXTRACT_PREFILTER_SKIP_REASON = "decision_model_prefilter";+/**+ * `skipped.reason` when the session's current turn material is already+ * tombstoned in the mining ledger: it was mined before and the commit+ * yielded nothing accepted, so there is no envelope to answer and the+ * recorded breakdown rides along as evidence.+ */+export const EXTRACT_MINING_TOMBSTONE_SKIP_REASON = "mining_tombstone";+/** `token_impact.source` for this use's samples. */export const EXTRACT_PREFILTER_TOKEN_SOURCE = decisionTokenImpactSource("extract_prefilter");@@ -90,10 +106,21 @@ export interface PrefilteredSignalExtractionPlan extends Omit<SignalExtractionPlreadonly llmStep: NeedsLlmStep | null;/** Enforce only: ids of the turns left out of the envelope, in record order. */readonly turnsDropped?: ReadonlyArray<string>;- /** Enforce only: every mined turn was dropped, so there is nothing to mine. */+ /**+ * There is no envelope, and here is why - one channel for every+ * suppression. With the pre-filter reason, every mined turn was dropped+ * and `turnsDropped` counts them. With the tombstone reason, the+ * session's current material was already mined and yielded nothing+ * accepted; `miningOutcome` carries the recorded breakdown as evidence.+ */readonly skipped?: {- readonly reason: typeof EXTRACT_PREFILTER_SKIP_REASON;- readonly turnsDropped: number;+ readonly reason:+ | typeof EXTRACT_PREFILTER_SKIP_REASON+ | typeof EXTRACT_MINING_TOMBSTONE_SKIP_REASON;+ /** Present with the pre-filter reason: how many turns were dropped. */+ readonly turnsDropped?: number;+ /** Present with the tombstone reason: the recorded zero-yield breakdown. */+ readonly miningOutcome?: SessionMiningOutcome;};/** Present only when the use is not `off` and a request failed. */readonly decisionModel?: { readonly degraded: DecisionDegradeReason };@@ -107,6 +134,15 @@ export interface PrefilteredPlanOptions {readonly env?: Readonly<Record<string, string | undefined>>;/** The operator's `token_impact` ledger opt-in. */readonly tokenImpactEnabled?: boolean;+ /**+ * Bypass the mining tombstone gate for this ONE plan call - the re-mine+ * escape for "I removed the downstream item and want it re-proposed",+ * since removing a preference does not change a session's turns and so+ * never lifts its tombstone on its own. The ledger is still read and+ * still fails closed: force ignores a matching entry, never a corrupt+ * file, and it changes nothing else about the plan.+ */+ readonly force?: boolean;}/** Whether this configuration runs the pre-filter at all. */@@ -216,10 +252,48 @@ function questionsFor(count: number): Readonly<Record<string, DecisionQuestion>>// ----- the plan ------------------------------------------------------------------+/**+ * The tombstone gate, run on every plan this module returns and always+ * before any decision-model work, so a barren session costs no requests+ * here and none downstream either. A hit suppresses the whole plan: no+ * envelope, no turns, the recorded breakdown as evidence - shaped exactly+ * like the pre-filter's own skip, so there is one "no envelope, and why"+ * channel. A session the ledger cannot verify (corrupt file, unknown+ * schema) throws by the read's contract rather than passing as eligible -+ * `force` bypasses the HIT, never the read, so the refusal still names a+ * broken ledger to an operator who asked for a re-mine.+ *+ * The plan's `turnsMined` is both the gate's hash input and what a pass+ * leaves untouched; only a hit empties it. The token-impact ledger is not+ * consulted here: the avoided spend is the caller's mining inference, not+ * a decision-model one this module would otherwise have made.+ */+function miningTombstoneGate(+ vault: string,+ plan: SignalExtractionPlan,+ force: boolean,+): PrefilteredSignalExtractionPlan {+ const entry = readMiningTombstone(vault, plan.sessionId, plan.turnsMined);+ if (entry === null || force) return plan;+ return Object.freeze({+ ...plan,+ turnsMined: Object.freeze([]),+ llmStep: null,+ skipped: Object.freeze({+ reason: EXTRACT_MINING_TOMBSTONE_SKIP_REASON,+ miningOutcome: Object.freeze({ ...entry.outcome }),+ }),+ });+}+/*** The plan phase with the optional pre-filter. With the use `off` this is- * exactly {@link planExtractSignals}. Refusals ({@link ExtractSignalsError})- * are thrown as there, before any request.+ * exactly {@link planExtractSignals}, unless the session is tombstoned -+ * the mining ledger's zero-yield gate runs on every configuration, before+ * the pre-filter, because the spend it prevents is the caller's mining+ * round trip and not only this module's requests. Refusals+ * ({@link ExtractSignalsError}) are thrown as there, before the gate.+ * `opts.force` lifts the gate for this one call.*/export async function planExtractSignalsPrefiltered(vault: string,@@ -227,11 +301,30 @@ export async function planExtractSignalsPrefiltered(opts: PrefilteredPlanOptions,): Promise<PrefilteredSignalExtractionPlan> {const cfg = opts.decisionModel;- if (!extractPrefilterActive(cfg)) return planExtractSignals(vault, sessionId, { now: opts.now });+ const force = opts.force === true;+ if (!extractPrefilterActive(cfg)) {+ return miningTombstoneGate(+ vault,+ planExtractSignals(vault, sessionId, { now: opts.now }),+ force,+ );+ }const provider = opts.provider ?? makeDecisionProvider(cfg, opts.env ?? process.env, vault);- if (provider === null) return planExtractSignals(vault, sessionId, { now: opts.now });+ if (provider === null) {+ return miningTombstoneGate(+ vault,+ planExtractSignals(vault, sessionId, { now: opts.now }),+ force,+ );+ }- const full = planExtractSignals(vault, sessionId, { now: opts.now, sourceTurnHint: true });+ const planned = planExtractSignals(vault, sessionId, { now: opts.now, sourceTurnHint: true });+ const suppressed = miningTombstoneGate(vault, planned, force);+ if (suppressed.skipped !== undefined) return suppressed;+ // The gate passed the plan through unchanged, so `full` is the base+ // plan and its envelope is the non-null one the rest of this function+ // has always read.+ const full = planned;const mined = full.turnsMined;const correlationId = randomUUID();const chunks = planPrefilterChunks(mined, cfg!.maxStateTokens);diff --git a/src/core/brain/extract-signals.ts b/src/core/brain/extract-signals.tsindex 134bb8e1..484bfbcc 100644--- a/src/core/brain/extract-signals.ts+++ b/src/core/brain/extract-signals.ts@@ -30,6 +30,16 @@* forever - and the inbox trial lane plus the dream pass remains the only* route to a confirmed preference. Nothing here writes a preference.*+ * The commit is also the mining ledger's write side (session-mining-ledger.ts):+ * every successful commit that answers a plan records its outcome breakdown+ * over the turn material the PLAN surveyed - the plan carries the hash of+ * the turns it offered, and the surfaces pass it back with the answer. A+ * zero-yield record (nothing written, nothing staged) is the tombstone the+ * plan gate suppresses on; a yield record is the distilled state the mining+ * census reports. A commit that answers no plan of this module's attests+ * nothing and records nothing, and a partial write never records - a run+ * that FAILED did not end.+ ** Precision beats recall, as it does in `fact-extract.ts`: the cap and the* floor are refusals, not filters. A payload over either limit is rejected* whole and named, because an over-eager mine that silently kept its best@@ -39,7 +49,11 @@import { posix } from "node:path";import { buildReconciliationReport, type ReconciliationReport } from "../reconciliation-report.ts";-import { buildCaptureBoundary, type SessionCaptureDecision } from "./capture-boundary.ts";+import {+ buildCaptureBoundary,+ type CaptureBoundary,+ type SessionCaptureDecision,+} from "./capture-boundary.ts";import { tryRecordDeadLetter, type DeadLetterLane, type DeadLetterOutcome } from "./dead-letter.ts";import { buildDedupIndex, computeDedupHash, type DedupIndexEntry } from "./dedup-hash.ts";import { classifyDurability, resolveDurabilityDenylist } from "./gates/durability.ts";@@ -59,7 +73,8 @@ import {EXTRACTED_SIGNALS_SURFACE,SHAPE_ROOT_PATH,} from "./response-shape.ts";-import { listSessionRawTurns } from "./session-recall.ts";+import { listSessionRawTurns, type ExpandedRawTurn } from "./session-recall.ts";+import { minedTurnsHash, recordSessionMiningOutcome } from "./session-mining-ledger.ts";import { writeSignal } from "./signal.ts";import { isoDate, isoSecond } from "./time.ts";import { BRAIN_SIGNAL_SOURCE_TYPE, type BrainSignalSign } from "./types.ts";@@ -205,6 +220,14 @@ export interface SignalExtractionPlan {readonly turnsScanned: number;/** The user turns the prompt embeds, in record order. */readonly turnsMined: ReadonlyArray<MinedTurn>;+ /**+ * {@link minedTurnsHash} of {@link turnsMined}: the material this plan+ * put in front of the caller. The commit that answers this plan passes+ * it back as {@link CommitExtractedSignalsOptions.surveyedTurnsHash}, so+ * the mining record keys on the material the verdict was earned over,+ * never on whatever the session holds by commit time.+ */+ readonly surveyedTurnsHash: string;/** The limits the payload will be held to, echoed so the caller can read them. */readonly cap: number;readonly confidenceFloor: number;@@ -266,6 +289,15 @@ export interface CommitExtractedSignalsResult {* lost would misreport the one number an operator reads first.*/readonly reconciliation: ReconciliationReport;+ /**+ * Present only when the commit itself SUCCEEDED but recording the mining+ * outcome failed (session-mining-ledger.ts). The signals this result+ * reports are already on disk, so throwing would misreport a successful+ * write as a failed one - and swallowing the failure would hide a ledger+ * that is no longer trustworthy, so the error message rides here by+ * name, and on both surface outputs.+ */+ readonly miningLedgerFailure?: string;}export interface PlanExtractSignalsOptions {@@ -292,6 +324,16 @@ export interface CommitExtractedSignalsOptions {readonly writeApprovalEnabled?: boolean;/** Shared dedup index; built from the vault when omitted. */readonly dedup?: Map<string, DedupIndexEntry>;+ /**+ * The answered plan's {@link SignalExtractionPlan.surveyedTurnsHash}: the+ * hash of the turns the caller was actually shown. The mining record keys+ * on it, so an import landing between plan and commit cannot put a+ * tombstone on turns no model ever saw. Absent means this commit answers+ * no plan of this module's, and then there is no material the verdict was+ * earned over - so nothing is recorded, and the session stays eligible+ * (the safe direction, the same one an unreadable entry gets).+ */+ readonly surveyedTurnsHash?: string;}// ----- Semantic rules the descriptor language cannot express ---------------@@ -430,6 +472,30 @@ function readItems(payload: unknown): ReadonlyArray<unknown> | null {// ----- Phase one: what to mine ---------------------------------------------+/**+ * The user turns the prompt embeds, in record order: the import path's+ * role rule, the message-level capture boundary, and blank text dropped.+ * Shared by the plan (which offers them) and the mining census (whose+ * `minedTurnMaterial` must hash exactly what the plan would mine), so the+ * two can never disagree on what a run covered.+ */+export function selectMinedTurns(+ boundary: CaptureBoundary,+ turns: ReadonlyArray<ExpandedRawTurn>,+): MinedTurn[] {+ const mined: MinedTurn[] = [];+ for (const turn of turns) {+ if (turn.role !== MINED_TURN_ROLE) continue;+ // Message-level boundary, mirroring the import path: suppressed text+ // never reaches extraction, so it never reaches the prompt either.+ if (boundary.suppressMessage(turn.text)) continue;+ const text = turn.text.trim();+ if (text.length === 0) continue;+ mined.push(Object.freeze({ turnId: turn.turn_id, text, timestamp: turn.timestamp }));+ }+ return mined;+}+/*** Assemble the mining plan for one imported session. Read-only; writes* nothing and calls no model.@@ -462,16 +528,7 @@ export function planExtractSignals(`session ${trimmed} has no imported turns; import it first with 'o2b brain import-session --recall'`,);}- const mined: MinedTurn[] = [];- for (const turn of turns) {- if (turn.role !== MINED_TURN_ROLE) continue;- // Message-level boundary, mirroring the import path: suppressed text- // never reaches extraction, so it never reaches the prompt either.- if (boundary.suppressMessage(turn.text)) continue;- const text = turn.text.trim();- if (text.length === 0) continue;- mined.push(Object.freeze({ turnId: turn.turn_id, text, timestamp: turn.timestamp }));- }+ const mined = selectMinedTurns(boundary, turns);if (mined.length === 0) {throw new ExtractSignalsError(`session ${trimmed} has ${turns.length} imported turn(s) but none authored by the user; there is nothing to mine`,@@ -483,6 +540,7 @@ export function planExtractSignals(boundaryDecision,turnsScanned: turns.length,turnsMined: Object.freeze(mined),+ surveyedTurnsHash: minedTurnsHash(mined),cap: AUTO_EXTRACT_PER_SESSION_CAP,confidenceFloor: AUTO_EXTRACT_CONFIDENCE_FLOOR,llmStep: buildMiningStep(trimmed, mined, { sourceTurnHint: opts.sourceTurnHint === true }),@@ -585,7 +643,8 @@ export function commitExtractedSignals(if (trimmed.length === 0) {throw new ExtractSignalsError("extract-signals needs a session reference");}- const boundaryDecision = buildCaptureBoundary(vault).sessionDecision(trimmed);+ const boundary = buildCaptureBoundary(vault);+ const boundaryDecision = boundary.sessionDecision(trimmed);if (boundaryDecision !== "capture") {throw new ExtractSignalsError(`session ${trimmed} is classified '${boundaryDecision}' by the capture boundary; it may not produce signals`,@@ -695,10 +754,48 @@ export function commitExtractedSignals();}+ // Every commit that answers a plan is the ledger's write trigger: the+ // outcome breakdown is recorded over the hash the PLAN surveyed - the+ // material the verdict was earned over - whatever the yield was. The+ // zero-yield record is the tombstone the plan gate suppresses on; a+ // yield record is the distilled state the census reports, and it is NOT+ // a tombstone - the session stays minable. A thrown+ // ExtractSignalsWriteError never reaches here: a run that FAILED did not+ // end, so it records nothing. And a commit that answers no plan of this+ // module's attests nothing, so it records nothing: without surveyed+ // material there is nothing honest to key a record on, and failing open+ // to a re-mine is the safe direction.+ const staged = writeApprovalEnabled ? written.length : 0;+ let miningLedgerFailure: string | undefined;+ if (opts.surveyedTurnsHash !== undefined) {+ try {+ recordSessionMiningOutcome(+ vault,+ {+ sessionId: trimmed,+ minedContentHash: opts.surveyedTurnsHash,+ outcome: {+ written: written.length,+ staged,+ deduped,+ durabilityRejected: rejected.length,+ },+ },+ opts.now,+ );+ } catch (err) {+ // The signals already on disk are safe; throwing would misreport the+ // commit as failed, and dropping the error would leave a ledger that+ // silently stopped being written. Name it instead - the surfaces+ // carry it verbatim.+ miningLedgerFailure = err instanceof Error ? err.message : String(err);+ }+ }+return Object.freeze({sessionId: trimmed,written: Object.freeze(written),- staged: writeApprovalEnabled ? written.length : 0,+ staged,deduped,durabilityRejected: rejected.length,rejected: Object.freeze(rejected),@@ -707,6 +804,7 @@ export function commitExtractedSignals(found: written.length,missing: [],}),+ ...(miningLedgerFailure !== undefined ? { miningLedgerFailure } : {}),});}diff --git a/src/core/brain/hygiene/apply.ts b/src/core/brain/hygiene/apply.tsindex 7d467141..6dd827d0 100644--- a/src/core/brain/hygiene/apply.ts+++ b/src/core/brain/hygiene/apply.ts@@ -13,9 +13,11 @@* - `archive` / `forget` -> move the page into a dated* `Brain/.snapshots/` directory - never delete;* - `retire` -> the document retirement core: a near-duplicate- * copy is deleted, its manifest entry dropped and- * its removal audited (destructive by design, so- * the plan builder only selects it attended).+ * copy is deleted, its manifest entry marked+ * retired into the canonical, and its removal+ * audited (destructive by design, so the plan+ * builder only selects it attended); the applied+ * detail carries the per-copy link counts.** Dry-run previews the routed actions with zero writes. Per-finding* fail-soft; one audit record per non-dry run.@@ -176,12 +178,31 @@ async function executeFinding(`${first.path}: ${first.message}`,);}+ // The per-copy link accounting rides the detail (review M7): the+ // counts used to be dropped here, so an apply that rewrote ten+ // inbound links reported only "retired N" and one whose rewrite+ // was refused looked like every other.+ const links = result.retired.reduce(+ (acc, copy) => ({+ rewritten: acc.rewritten + copy.wikilinks.rewritten.length,+ failed: acc.failed + copy.wikilinks.failed.length,+ }),+ { rewritten: 0, failed: 0 },+ );+ // The disk accounting rides it too (review M6, retire half): the+ // snapshot archive lands in a counted surface before the removals+ // free anything, so the detail states the net, not only the shrink.+ const disk = result.disk;+ const net = disk.netBytes ?? 0;return Object.freeze({finding_id: finding.id,action: "retire",- detail: `retired ${result.retired.length} near-duplicate cop${- result.retired.length === 1 ? "y" : "ies"- } into ${canonical}`,+ detail:+ `retired ${result.retired.length} near-duplicate cop${+ result.retired.length === 1 ? "y" : "ies"+ } into ${canonical}; links: ${links.rewritten} rewritten, ${links.failed} failed; ` ++ `disk: snapshot ${disk.snapshotBytes ?? 0} bytes, removed ${disk.removedBytes} bytes, ` ++ `net ${net >= 0 ? "+" : ""}${net} bytes`,});}default:diff --git a/src/core/brain/hygiene/detectors/document-dedup.ts b/src/core/brain/hygiene/detectors/document-dedup.tsindex 24990255..463888fe 100644--- a/src/core/brain/hygiene/detectors/document-dedup.ts+++ b/src/core/brain/hygiene/detectors/document-dedup.ts@@ -16,7 +16,8 @@* - Lexical: token jaccard over the page body at* {@link DOCUMENT_DEDUP_LEXICAL_THRESHOLD}, bucketed by the folded* composite scope key so identical text in two scopes never groups- * (per-scope keying, t_37c05a34).+ * (per-scope keying, t_37c05a34). This is the layer the hygiene scan+ * registry runs.* - Embedding (`detectSemanticDocumentDedup`): cosine over a* configured provider's vectors at* {@link DOCUMENT_DEDUP_EMBEDDING_THRESHOLD}, crossing scope@@ -24,6 +25,13 @@* within scope only, because a scope boundary is an operator* decision a similarity score must not overrule. Any provider* problem falls back to the lexical layer, labeled as such.+ * WIRING STATUS (review L6): this layer is exported and tested but+ * NO scan path calls it yet - the scan registry is synchronous and+ * this layer is async, so reaching it needs the scan's signature to+ * go async across its callers first. Until that lands, the 0.97+ * embedding threshold is unreachable from every surface, and the+ * scan's similarity answers are lexical only; the CHANGELOG must not+ * claim the embedding layer for the wired detector.** Canonical selection is keep-newest on `created_at`. A group holding* ANY member without `created_at` is ineligible: the page-dedup lane's@@ -31,6 +39,10 @@* moves the mtime it falls back to), so such a group is reported for* review with the reason in evidence, never for retirement.*+ * The candidate cap bounds files READ; a cap that bound the read set+ * emits the {@link withCandidateCapNotice} coverage row, so a truncated+ * scan never answers clean (review M9).+ ** Union-find components can be CHAINS (A~B, B~C, A!~C): one group, but* not one similarity. A member therefore retires only on its OWN* above-threshold pair against the chosen canonical; a member the@@ -138,24 +150,50 @@ interface PairScore {* order, after the three admission filters:** - the caller's `readable` predicate (a refused page is absent - it- * is not scored, not grouped and not named);- * - the candidate cap, which bounds files READ and is applied before- * the merge-resolved filter so the bound stays a bound;- * - the merge-resolved predicate (GitHub #180), applied after the cap- * the same way the preference dedup detector applies it.+ * is not scored, not grouped, not named, and not read);+ * - the candidate cap, which bounds files READ: entries past the cap+ * are still COUNTED on {@link CollectedDocuments.listed}, never+ * opened;+ * - the merge-resolved predicate (GitHub #180), applied to the read+ * set the same way the preference dedup detector applies it.** A page whose frontmatter does not parse is skipped, as in every other- * detector; corruption is the doctor's job, not a scan abort.+ * detector; corruption is the doctor's job, not a scan abort. The count+ * of what the cap left unread is what lets both layers name their+ * coverage ({@link withCandidateCapNotice}) instead of answering clean+ * over a truncated scan (review M9).*/+interface CollectedDocuments {+ readonly records: DocumentRecord[];+ /**+ * The count the cap bounds: how many pages were collected BEFORE the+ * merge-resolved filter thinned them, i.e. the size of the read set.+ * The coverage notice gates on THIS count - gating on+ * {@link records}.length would let a merge-resolved page inside the+ * capped read set read as an under-cap scan and silence the notice+ * while pages past the cap were never opened (review M9).+ */+ readonly read: number;+ /** Every `.md` entry listed under the reach, read or not, sorted order. */+ readonly listed: number;+}+function collectDocumentRecords(vault: string,readable?: (rel: string) => boolean,-): DocumentRecord[] {+): CollectedDocuments {const dir = join(vault, BRAIN_SOURCES_REL);- if (!existsSync(dir)) return [];+ if (!existsSync(dir)) return { records: [], read: 0, listed: 0 };const out: DocumentRecord[] = [];+ let listed = 0;for (const name of readdirSync(dir).toSorted()) {if (!name.endsWith(".md")) continue;+ const path = `${BRAIN_SOURCES_REL}/${name}`;+ if (readable !== undefined && !readable(path)) continue;+ listed += 1;+ // The cap bounds files READ: past it the walk only counts, it never+ // opens the page. The count is what the coverage notice reports.+ if (out.length >= DOCUMENT_DEDUP_CANDIDATE_CAP) continue;const abs = join(dir, name);let meta: Record<string, unknown>;let body: string;@@ -165,8 +203,6 @@ function collectDocumentRecords(continue;}if (meta["kind"] !== BRAIN_SOURCE_KIND) continue;- const path = `${BRAIN_SOURCES_REL}/${name}`;- if (readable !== undefined && !readable(path)) continue;const rawCreated = meta["created_at"];const createdAtIso =typeof rawCreated === "string" && Number.isFinite(Date.parse(rawCreated)) ? rawCreated : null;@@ -177,10 +213,52 @@ function collectDocumentRecords(mergedInto: readMergedInto(meta),scopeKey: compositeScopeKey(scopeFromFrontmatter(meta)),});- if (out.length >= DOCUMENT_DEDUP_CANDIDATE_CAP) break;}const pointerOf = mergePointerLookup(out.map((r) => [r.path, r.mergedInto] as const));- return out.filter((r) => !isMergeResolved(r.path, pointerOf));+ return {+ records: out.filter((r) => !isMergeResolved(r.path, pointerOf)),+ read: out.length,+ listed,+ };+}++/**+ * Append the coverage notice a reached cap owes the caller (review M9):+ * only the first {@link DOCUMENT_DEDUP_CANDIDATE_CAP} summary pages in+ * path order were compared, so duplicates sorting after it were never+ * read, and without this row a truncated scan looks clean. The notice is+ * review-only - it never retires anything - and it fires only when the+ * cap actually bound the read set (`read` at the cap with pages left+ * listed), so a small vault with a few non-summary `.md` files never+ * sees it. `read` is the pre-filter collected count: the merge-resolved+ * pages a read set loses AFTERWARD cannot shrink what the cap bound.+ * Returned findings stay in the deterministic target order.+ */+function withCandidateCapNotice(+ listed: number,+ read: number,+ findings: HygieneFinding[],+): HygieneFinding[] {+ if (read < DOCUMENT_DEDUP_CANDIDATE_CAP || listed <= DOCUMENT_DEDUP_CANDIDATE_CAP) {+ return findings;+ }+ const notice = Object.freeze({+ id: hygieneFindingId("document-dedup", [BRAIN_SOURCES_REL]),+ detector: "document-dedup" as const,+ severity: "warning" as const,+ title:+ `Document-dedup compared only the first ${DOCUMENT_DEDUP_CANDIDATE_CAP} of ` ++ `${listed} listed pages; duplicates past the cap were never read`,+ targets: Object.freeze([BRAIN_SOURCES_REL]),+ proposed_action: "review" as const,+ evidence: Object.freeze({+ candidate_cap_reached: true,+ cap: DOCUMENT_DEDUP_CANDIDATE_CAP,+ scanned: read,+ total: listed,+ }),+ });+ return [...findings, notice].toSorted((a, b) => (a.targets[0]! < b.targets[0]! ? -1 : 1));}/**@@ -388,14 +466,17 @@ export function detectDocumentDedup(opts: DocumentDedupOptions = {},): ReadonlyArray<HygieneFinding> {const threshold = opts.lexicalThreshold ?? DOCUMENT_DEDUP_LEXICAL_THRESHOLD;- const records = collectDocumentRecords(vault, ctx.readable).filter(- (r) => tokenise(r.body).size >= NEAR_DUPLICATE_MIN_TOKENS,+ const { records, read, listed } = collectDocumentRecords(vault, ctx.readable);+ const compared = records.filter((r) => tokenise(r.body).size >= NEAR_DUPLICATE_MIN_TOKENS);+ if (compared.length < 2) return withCandidateCapNotice(listed, read, []);+ return withCandidateCapNotice(+ listed,+ read,+ findingsFromGroups(compared, lexicalPairs(compared, threshold), {+ method: "lexical",+ threshold,+ }),);- if (records.length < 2) return [];- return findingsFromGroups(records, lexicalPairs(records, threshold), {- method: "lexical",- threshold,- });}function resolveConfiguredProvider(vault: string): EmbeddingProvider | null {@@ -456,44 +537,50 @@ export async function detectSemanticDocumentDedup(// preference dedup detector applies.if (provider === null || !providerProducesVectors(provider)) return lexical();- const records = collectDocumentRecords(vault, opts.readable).filter(- (r) => tokenise(r.body).size >= NEAR_DUPLICATE_MIN_TOKENS,- );- if (records.length < 2) return lexical();+ const { records, read, listed } = collectDocumentRecords(vault, opts.readable);+ const compared = records.filter((r) => tokenise(r.body).size >= NEAR_DUPLICATE_MIN_TOKENS);+ // The same coverage notice the lexical layer owes (review M9), wrapped+ // around whichever result this run answers.+ const withCap = (result: SemanticDocumentDedupResult): SemanticDocumentDedupResult =>+ Object.freeze({+ ...result,+ findings: Object.freeze(withCandidateCapNotice(listed, read, [...result.findings])),+ });+ if (compared.length < 2) return withCap(lexical());const threshold = opts.threshold ?? DOCUMENT_DEDUP_EMBEDDING_THRESHOLD;let vectors: number[][];try {- vectors = await provider.embed(records.map((r) => r.body));+ vectors = await provider.embed(compared.map((r) => r.body));} catch {- return lexical();+ return withCap(lexical());}- if (vectors.length !== records.length) return lexical();+ if (vectors.length !== compared.length) return withCap(lexical());// A ragged batch - vectors of differing dimensions - is the same// broken-provider shape as a count mismatch: the cosine below would// silently truncate to the shorter vector and fabricate a similarity// out of incomparable spaces, so the whole run answers lexical, labeled.const dimension = vectors[0]!.length;- if (vectors.some((v) => v.length !== dimension)) return lexical();+ if (vectors.some((v) => v.length !== dimension)) return withCap(lexical());const pairs: PairScore[] = [];- for (let i = 0; i < records.length; i++) {- for (let j = i + 1; j < records.length; j++) {+ for (let i = 0; i < compared.length; i++) {+ for (let j = i + 1; j < compared.length; j++) {// Scope buckets fold here too: a similarity score must not// overrule the operator's scope boundary.- if (records[i]!.scopeKey !== records[j]!.scopeKey) continue;+ if (compared[i]!.scopeKey !== compared[j]!.scopeKey) continue;const similarity = cosine(vectors[i]!, vectors[j]!);if (similarity < threshold) continue;pairs.push({- pair: [records[i]!.path, records[j]!.path],+ pair: [compared[i]!.path, compared[j]!.path],similarity: roundScore(similarity),});}}- return Object.freeze({+ return withCap({method: "embedding" as const,findings: Object.freeze(- findingsFromGroups(records, pairs, {+ findingsFromGroups(compared, pairs, {method: "embedding",threshold,model: provider.model,diff --git a/src/core/brain/import-census.ts b/src/core/brain/import-census.tsindex 8254d814..c23bac44 100644--- a/src/core/brain/import-census.ts+++ b/src/core/brain/import-census.ts@@ -114,10 +114,16 @@ export function censusSessionSignals(/*** Ingest lane: read back the content-manifest entries of the sources an- * ingest claims to have completed. Found means `unchanged` - recorded, still- * on disk, and still hashing to the recorded digest. A path that is `new`- * (never recorded), `modified` (recorded hash no longer matches) or `missing`- * (gone from disk) is a claim the manifest cannot confirm.+ * ingest claims to have completed. Found means the manifest recorded the+ * path, it is still on disk, and it still hashes to the recorded digest -+ * the `unchanged` classification, and `retired` beside it: a retirement+ * marks the entry `retired_into` a canonical page and removes the DERIVED+ * copy, never the source, so a byte-identical source behind the marker is+ * a claim the vault still reproduces, not a loss (reconcilePlan re-reads+ * a plan's completed set, and a source retired between two runs of the+ * same plan is still on disk). A path that is `new` (never recorded),+ * `modified` (recorded hash no longer matches) or `missing` (gone from+ * disk) is a claim the manifest cannot confirm.*/export function censusIngestedPaths(vault: string,@@ -127,7 +133,8 @@ export function censusIngestedPaths(// Canonical on both sides: `classifyPaths` reports canonical keys, so a// claim spelled any other way would never match its own classification.const keys = [...new Set(claimedPaths.map((p) => canonicalNotePath(p)))];- const confirmed = new Set(classifyPaths(vault, keys, readManifest(vault)).unchanged);+ const classification = classifyPaths(vault, keys, readManifest(vault));+ const confirmed = new Set([...classification.unchanged, ...classification.retired]);return buildReadBackCensus(keys, (path) => confirmed.has(path));}diff --git a/src/core/brain/ingest/batch-plan.ts b/src/core/brain/ingest/batch-plan.tsindex e19c9475..713b856f 100644--- a/src/core/brain/ingest/batch-plan.ts+++ b/src/core/brain/ingest/batch-plan.ts@@ -200,6 +200,17 @@ export interface BatchPlan {readonly batches: readonly IngestBatch[];/** Canonical paths classified `unchanged` and therefore skipped, sorted. */readonly skipped: readonly string[];+ /**+ * Canonical paths whose manifest entry is marked `retired_into` a+ * canonical page (the resurrect guard, storage lifecycle review H3) and+ * whose bytes are unchanged, sorted. They are never queued - re-ingesting+ * one would quietly undo the retirement - while a real byte change+ * re-admits the source as `modified` and drops it from this list. Unlike+ * {@link BatchPlan.skipped} the skip survives a contract change: a+ * retirement is not a content change, so the marked entry keeps its skip+ * under the new contract too.+ */+ readonly skippedRetired: readonly string[];/*** Files skipped before extraction: first every file whose format the* source format registry names with no extractor (`format-not-extractable`,@@ -266,7 +277,9 @@ export interface BatchPlan {* Plan the ingest of a source directory into bounded parallel batches.** Discovers the ingestible files under `sourceDir` (a vault-relative path),- * skips those the content-hash manifest classifies `unchanged`, and packs the+ * skips those the content-hash manifest classifies `unchanged` (and those it+ * classifies `retired` - the H3 resurrect guard, reported on+ * {@link BatchPlan.skippedRetired} and never queued), and packs the* `new`/`modified` remainder greedily into batches that respect BOTH caps. A* single file larger than the byte cap cannot be split, so it forms its own* singleton batch (documented, honest - never silently dropped or truncated).@@ -416,6 +429,11 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOptif (contractChanged) {for (const p of classification.unchanged) status.set(p, PLANNED_FILE_STATUS.CONTRACT_CHANGED);}+ // The resurrect guard (H3): a retired entry never enters the status map -+ // not as new (its entry is recorded), not as contract-changed (a+ // retirement is not a content change, so the marker outlives a contract+ // move) - and therefore never lands in a batch.+ const skippedRetired = [...classification.retired].toSorted();const planned: PlannedFile[] = [...status.keys()].toSorted().map((path) => {const format = sourceFormatOf(path);return {@@ -434,8 +452,10 @@ export function planBatches(vault: string, sourceDir: string, opts: BatchPlanOptmaxBatchFiles: opts.maxBatchFiles,batches,// Nothing skips under a changed contract: every unchanged source is in the- // reprocessing list above, so the skip list is empty by construction.+ // reprocessing list above, so the skip list is empty by construction. The+ // retired skip is independent of that rule - see {@link BatchPlan.skippedRetired}.skipped: contractChanged ? [] : [...classification.unchanged].toSorted(),+ skippedRetired,skippedNonExtractable,skipReasonCounts: countSkipReasons(skippedNonExtractable),unclassifiable: aggregateUnclassifiable(unclassifiable),diff --git a/src/core/brain/ingest/content-manifest.ts b/src/core/brain/ingest/content-manifest.tsindex ceb8b1c5..992cae54 100644--- a/src/core/brain/ingest/content-manifest.ts+++ b/src/core/brain/ingest/content-manifest.ts@@ -110,17 +110,35 @@ export interface ManifestEntry {* Absent together with the byte count on unwatermarked entries.*/readonly extracted_at?: string;+ /**+ * The resurrect guard (storage lifecycle, PR #257 review H3): the vault+ * path of the canonical summary page this source's near-duplicate copy+ * was retired into. Set by the retirement core INSTEAD of dropping the+ * entry - a dropped entry would classify the unchanged source `new`+ * again and the next batch plan would re-ingest it, quietly undoing the+ * retirement. While the marker stands and the bytes are unchanged the+ * source classifies `retired` ({@link classifyPaths}) and no batch plan+ * queues it; a real content change re-admits it as `modified`, and a+ * deliberate re-ingest re-records the entry without the marker. ADDITIVE+ * by design - schema_version stays 3, and a malformed shape on disk+ * drops the FIELD, never the entry.+ */+ readonly retired_into?: string;}/*** The entry's extraction watermark, degraded the lazy way: an entry recorded- * before watermarks existed - or with a malformed one on disk - answers- * byte 0, "never extracted", so the source reprocesses fully instead of- * skipping. Never an eager mass reprocess: the absence costs one pass, once,- * per source that actually grows again.+ * before watermarks existed - one with a malformed shape on disk - or a+ * path with NO entry at all answers byte 0, "never extracted", so the+ * source reprocesses fully instead of skipping. Never an eager mass+ * reprocess: the absence costs one pass, once, per source that actually+ * grows again. The `undefined` acceptance is deliberate: every caller reads+ * the entry straight out of a manifest lookup (`entries[path]`), where a+ * source that was never ingested is undefined and "never extracted" is the+ * honest answer, not a crash.*/-export function extractedBytesOf(entry: ManifestEntry): number {- return entry.extracted_bytes ?? 0;+export function extractedBytesOf(entry: ManifestEntry | undefined): number {+ return entry?.extracted_bytes ?? 0;}/**@@ -166,6 +184,8 @@ export interface PathClassification {readonly modified: string[];/** On disk, present in the manifest, bytes identical. */readonly unchanged: string[];+ /** On disk, bytes identical, and the entry is marked retired into a canonical page. */+ readonly retired: string[];/** In the manifest (or requested) but no longer on disk. */readonly missing: string[];}@@ -301,8 +321,10 @@ function parseEntryValue(value: unknown): ManifestEntry | null {// a malformed one drops JUST the field, which answers byte 0 and// reprocesses lazily - dropping the whole entry would forfeit the// fingerprint and force the eager contract-changed reprocess instead.+ // The retired marker follows the same per-field rule (H3).const extractedBytes = record["extracted_bytes"];const extractedAt = record["extracted_at"];+ const retiredInto = record["retired_into"];return {sha256: record["sha256"],...(typeof fingerprint === "string" ? { fingerprint } : {}),@@ -310,6 +332,9 @@ function parseEntryValue(value: unknown): ManifestEntry | null {...(typeof extractedAt === "string" && extractedAt.length > 0? { extracted_at: extractedAt }: {}),+ ...(typeof retiredInto === "string" && retiredInto.length > 0+ ? { retired_into: retiredInto }+ : {}),};}@@ -320,12 +345,16 @@ function isByteOffset(value: unknown): value is number {/*** Does the entry carry recorded state the bare v1 digest shape cannot hold -- * a per-entry fingerprint or an extraction watermark? Such an entry never- * serializes down to v1: data a write could read back is data the write- * must keep.+ * a per-entry fingerprint, an extraction watermark, or the retired marker?+ * Such an entry never serializes down to v1: data a write could read back is+ * data the write must keep.*/function hasRecordedState(entry: ManifestEntry): boolean {- return entry.fingerprint !== undefined || entry.extracted_bytes !== undefined;+ return (+ entry.fingerprint !== undefined ||+ entry.extracted_bytes !== undefined ||+ entry.retired_into !== undefined+ );}/**@@ -445,7 +474,15 @@ export function writeManifestAtomic(* - not on disk → `missing`,* - on disk, not recorded → `new`,* - on disk, hash matches → `unchanged`,+ * - on disk, hash matches and the entry is marked retired → `retired`,* - on disk, hash differs → `modified`.+ * The `retired` bucket is the resurrect guard (storage lifecycle, review+ * H3): a near-duplicate retirement marks the entry `retired_into` a+ * canonical page instead of dropping it, and an unchanged source behind+ * that marker classifies `retired` - never `new` - so no batch plan+ * re-queues it. A hash DIFFERENCE still classifies `modified` whatever+ * the marker says: the marker describes the bytes it was recorded under,+ * and changed bytes are a real source again.* The buckets answer the BYTES question only - the extraction-contract* question is the manifest's derived `contract` field, kept deliberately* separate so a bytes-based read-back (the import census) is never affected@@ -458,7 +495,13 @@ export function classifyPaths(paths: readonly string[],manifest: ContentManifest,): PathClassification {- const result: PathClassification = { new: [], modified: [], unchanged: [], missing: [] };+ const result: PathClassification = {+ new: [],+ modified: [],+ unchanged: [],+ retired: [],+ missing: [],+ };for (const path of paths) {const canonical = canonicalNotePath(path);const abs = join(vault, canonical);@@ -470,15 +513,41 @@ export function classifyPaths(const live = hashPath(abs);if (recorded === undefined) {result.new.push(canonical);- } else if (recorded.sha256 === live) {- result.unchanged.push(canonical);- } else {+ } else if (recorded.sha256 !== live) {result.modified.push(canonical);+ } else if (recorded.retired_into !== undefined) {+ result.retired.push(canonical);+ } else {+ result.unchanged.push(canonical);}}return result;}+/** One path's watermark request: the stamp the caller wants recorded. */+export interface ManifestWatermarkRequest {+ /** ISO-8601 `extracted_at` stamp, from the caller's injected clock. */+ readonly at: string;+}++export interface UpdateManifestOptions {+ /**+ * Request an extraction watermark per path (review L1): updateManifest+ * stats and hashes the source ITSELF, INSIDE the manifest lock, and+ * records the full entry - the live digest, the live contract+ * fingerprint, the byte count of the file it hashed and the caller's+ * stamp. A request supersedes any pinned record for the same path: the+ * pinned form carries a digest computed OUTSIDE the lock, and two+ * concurrent runs over an appended source can then land the older+ * digest-and-watermark pair last - the race L1 names. With the request+ * there is no precomputed pair to go stale. A source that cannot be+ * stat'ed or hashed throws by name after the lock is released and+ * nothing is written; a caller that wants the vanished-source skip+ * guards the call the way the ingest pipeline already does.+ */+ readonly watermarks?: Readonly<Record<string, ManifestWatermarkRequest>>;+}+/*** Record post-ingest state for `paths`, merging into the existing manifest.* A path still on disk gets its current hash recorded TOGETHER with the live@@ -509,6 +578,7 @@ export function updateManifest(vault: string,paths: readonly string[],records: Readonly<Record<string, ManifestEntry>> = {},+ opts: UpdateManifestOptions = {},): boolean {const handle = acquireLockSyncWithRetry(manifestPath(vault), undefined, {remedy: ingestLockRemedy,@@ -523,6 +593,20 @@ export function updateManifest(delete entries[canonical];continue;}+ // The in-lock watermark request (L1): one stat, one hash, of the+ // file as it exists UNDER the lock - never a caller-precomputed+ // pair that a concurrent append can stale out.+ const watermark = opts.watermarks?.[canonical];+ if (watermark !== undefined) {+ const size = statSync(abs).size;+ entries[canonical] = {+ sha256: hashPath(abs),+ fingerprint: live,+ extracted_bytes: size,+ extracted_at: watermark.at,+ };+ continue;+ }const pinned = records[canonical];entries[canonical] =pinned !== undefined ? pinned : { sha256: hashPath(abs), fingerprint: live };diff --git a/src/core/brain/ingest/ingest.ts b/src/core/brain/ingest/ingest.tsindex 4e39b3e9..8ff00910 100644--- a/src/core/brain/ingest/ingest.ts+++ b/src/core/brain/ingest/ingest.ts@@ -22,13 +22,13 @@import {closeSync,existsSync,+ fstatSync,mkdirSync,openSync,readFileSync,readSync,- statSync,} from "node:fs";-import { createHash } from "node:crypto";+import { createHash, type Hash } from "node:crypto";import { dirname, join, relative } from "node:path";import type { FrontmatterMap } from "../../types.ts";@@ -81,9 +81,9 @@ import {type Provenance,} from "../provenance/provenance.ts";import { PLAN_ID_LABEL, recordCompleted } from "./checkpoint.ts";-import { extractedBytesOf, hashPath, readManifest, updateManifest } from "./content-manifest.ts";+import { extractedBytesOf, readManifest, updateManifest } from "./content-manifest.ts";import { computeExtractionContractFingerprint } from "./contract.ts";-import { readTail } from "./tail-read.ts";+import { cutAppendedDelta } from "./tail-read.ts";import {PRE_EXTRACT_MAX_SOURCE_BYTES,SOURCE_OPEN_FLAGS,@@ -252,20 +252,35 @@ export interface IngestSourceResult {* {@link ./tail-read.ts} verified as a pure append - the recorded digest* still described the leading bytes. Present ONLY on that append shape: a* first ingest, a rewritten source, or a source with no watermark reports- * no delta and reprocesses fully. The range is reporting, not state - the- * stamp below always records the whole current source, because this- * pipeline's contract is a whole-source extraction per run.+ * no delta and reprocesses fully.*- * The whole-source stamp is also what bounds a delta series: the tail- * cut ends at the LAST COMPLETE newline, so an append with no complete- * line yet reports an EMPTY delta (`toByte === fromByte`) while the- * stamp still advances to the file's full size - the unterminated- * fragment is extracted (extraction is whole-source) but announced by- * no delta, and the next append's delta starts PAST it rather than- * stitching it in. A fragment only enters a reported delta when the- * append that carries it ends with a newline before this run reads.+ * On the `appended` kind the stamp below advances the watermark to+ * exactly this range's `toByte` and records the digest of those leading+ * bytes (review M15), so the next append's prefix proof verifies against+ * the boundary the delta series announced. The range ends at the LAST+ * COMPLETE newline, so an append with no complete line yet reports an+ * EMPTY delta (`toByte === fromByte`) and the watermark stays at+ * `fromByte`: the unterminated fragment is extracted (extraction is+ * whole-source) but announced by no delta, and the NEXT append's delta+ * starts AT it, stitching the fragment in. A fragment only enters a+ * reported delta when the append that carries it ends with a newline+ * before this run reads.+ *+ * The `appended-over-cap` kind names a proven append whose tail passed+ * the delta cap: the range is the whole appended span, reported without+ * its text (review L3). The run stays a whole-source pass, so the stamp+ * still covered the full current file - the named outcome is the+ * difference from a rewrite, which reports no delta at all.+ *+ * Coverage (review L4): the watermark records what the CALLER+ * summarized, under the trusted lane's whole-source contract - a caller+ * the intake trusts has read the source it summarizes at its reach, so+ * every stamped byte was covered by this run's extraction. The optional+ * code-structure pre-extract pass is NOT that extraction; its skips+ * (unsupported format, source past the pre-extract cap) never gate the+ * stamp.*/- readonly extractionDelta?: { readonly fromByte: number; readonly toByte: number };+ readonly extractionDelta?: AppendedDeltaOutcome;}/**@@ -483,47 +498,94 @@ export function ingestSource(existsSync(join(vault, canonicalSource))) {const sourceAbs = join(vault, canonicalSource);- // The source can still vanish (ENOENT) or change kind (a directory where- // the classified file stood) between the existsSync above and these- // reads - after the summary pages were written, where a throw would lose- // this run's result. The old updateManifest path answered both shapes by- // degrading (a vanished source dropped its entry), so both reads here- // degrade the same fail-safe way: a source that cannot be read reports- // no delta, and a stamp that cannot describe it is skipped - the source- // simply reprocesses fully next time. The ENOENT window and the EISDIR- // window are the same catch.- try {- extractionDelta = appendedExtractionDelta(vault, sourceAbs, canonicalSource);- } catch {- // No provable append can be reported for a source that cannot be read.- }+ // The manifest read sits OUTSIDE the fs-race catch below: a corrupted or+ // unsupported manifest is its own named hard error (`readManifest`'s+ // contract), never a silently skipped stamp.+ const entry = readManifest(vault).entries[canonicalSource];+ const watermark = entry === undefined ? 0 : extractedBytesOf(entry);// Record the source's state under the live contract, WITH the extraction- // watermark: this run's contract is a whole-source extraction, so the- // stamp covers the full current file. That is what makes the NEXT- // append verifiable - the recorded digest describes exactly the leading- // extracted_bytes, so a later read either proves a pure append (delta- // above) or classifies modified and reprocesses fully.- let sourceSize: number | undefined;- let sourceDigest: string | undefined;+ // watermark. The append proof and the tail read run off ONE descriptor+ // and ONE pass over the bytes (review L2), so a file growing under the+ // reader cannot produce a proof and a delta that describe two versions.+ //+ // The stamp covers exactly the prefix the reported delta announces+ // (review M15): on a proven append the watermark advances to the delta's+ // boundary-safe `toByte` and the recorded digest is the digest of+ // exactly those leading bytes, so the next append's prefix proof+ // verifies against the watermark the delta series actually announced,+ // and an unterminated trailing fragment stays unstamped for the next+ // delta to stitch in. On every other shape - a first ingest, a+ // rewritten source, a shrink, an over-cap append - this run is a+ // whole-source pass and the stamp covers the full current file. Both+ // arms keep ONE invariant: the recorded digest always describes exactly+ // `extracted_bytes` leading bytes of the source.+ //+ // The source can still vanish (ENOENT) or change kind (a directory where+ // the classified file stood) between the existsSync above and this read -+ // after the summary pages were written, where a throw would lose this+ // run's result. Such a read degrades the fail-safe way: no delta is+ // reported and no stamp lands, so the source simply reprocesses fully+ // next time. The ENOENT window and the EISDIR window are the same catch.+ let manifestStamp: { readonly digest: string; readonly bytes: number } | undefined;+ let stampFullSize = false;try {- // One stat, taken BEFORE the digest: the stamp must describe one- // snapshot of the source, so the byte count it claims is the file it- // hashed, not a size re-read after the hashing walked a possibly- // grown file.- sourceSize = statSync(sourceAbs).size;- sourceDigest = hashPath(sourceAbs);+ const snapshot = readAppendSnapshot(+ sourceAbs,+ watermark,+ entry === undefined ? "" : entry.sha256,+ );+ const delta = appendedDeltaFromSnapshot(watermark, snapshot);+ if (delta !== undefined) extractionDelta = delta;+ if (delta?.kind === "appended") {+ // The decision above already extended the through-watermark hash+ // with exactly the announced bytes: its digest is the digest of the+ // leading `toByte` bytes, computed with no second read.+ manifestStamp = {+ digest: snapshot.hashThroughWatermark.digest("hex"),+ bytes: delta.toByte,+ };+ } else {+ // An over-cap append is a reprocess-scale change (review L3): the+ // run stays a whole-source pass, so its stamp covers the full+ // current file - only the delta TEXT was refused, and the named+ // outcome on the result says so.+ stampFullSize = true;+ }} catch {- // Vanished or changed kind mid-flight: skip the stamp, reprocess fully.+ // Vanished or changed kind mid-flight: no delta, no stamp - the source+ // reprocesses fully next time.}- if (sourceDigest !== undefined && sourceSize !== undefined) {+ if (manifestStamp !== undefined) {+ // The fragment stamp: a digest over exactly the announced prefix,+ // which only this run's snapshot holds, so it pins a precomputed+ // record under the lock. The full-size arms below carry no such+ // precomputed pair; this one keeps the L1 window bounded to the+ // fragment shape - two CONCURRENT ingests of one source, the later+ // snapshot taken before the earlier run's append landed - and the+ // recorded pair stays self-consistent either way.updateManifest(vault, [canonicalSource], {[canonicalSource]: {- sha256: sourceDigest,+ sha256: manifestStamp.digest,fingerprint: computeExtractionContractFingerprint(vault),- extracted_bytes: sourceSize,+ extracted_bytes: manifestStamp.bytes,extracted_at: isoSecond(opts.now),},});+ } else if (stampFullSize) {+ // The full-size stamp through the IN-LOCK watermark request (review+ // L1): updateManifest stats and hashes the source itself, under the+ // manifest lock, so no caller-precomputed digest-and-watermark pair+ // exists to go stale against a concurrent append. The recorded values+ // are the ones the whole-source arms always meant: the live full+ // digest at the live full size, under the live contract fingerprint.+ updateManifest(+ vault,+ [canonicalSource],+ {},+ {+ watermarks: { [canonicalSource]: { at: isoSecond(opts.now) } },+ },+ );}// Record plan-scoped progress so an interrupted batch resumes at the item// boundary (t_ba1fa5f6). Only for real vault files - a URL/identity-only@@ -647,6 +709,19 @@ function captureTrustedSourceArchive(}}+/**+ * The appended delta an ingest result reports. `appended` is the proven,+ * boundary-safe range whose text the delta pass read; `appended-over-cap`+ * is a PROVEN append whose tail passed the delta cap - the whole appended+ * range is reported without reading its text, so an append at reprocess+ * scale never answers the same shape a rewrite gets (review L3). Absent+ * for every shape that is not a proven append: a first ingest, a+ * rewritten source, a source with no watermark.+ */+export type AppendedDeltaOutcome =+ | { readonly kind: "appended"; readonly fromByte: number; readonly toByte: number }+ | { readonly kind: "appended-over-cap"; readonly fromByte: number; readonly toByte: number };+/*** The largest appended tail the extraction-delta pass reads ({@link ./tail-read.ts}):* a tail past this is a reprocess-scale change, not a delta. Starts from@@ -680,48 +755,145 @@ export function appendedExtractionDelta(canonicalSource: string,): { readonly fromByte: number; readonly toByte: number } | undefined {const entry = readManifest(vault).entries[canonicalSource];- const watermark = entry === undefined ? 0 : extractedBytesOf(entry);+ if (entry === undefined) return undefined;+ const watermark = extractedBytesOf(entry);if (watermark <= 0) return undefined;- let size: number;- try {- size = statSync(sourceAbs).size;- } catch {- return undefined;- }- if (size <= watermark) return undefined;- if (entry === undefined || !prefixHashMatches(sourceAbs, watermark, entry.sha256)) {- return undefined;- }- const tail = readTail(sourceAbs, watermark, { maxBytes: EXTRACTION_DELTA_MAX_BYTES });- if (tail.kind !== "appended") return undefined;- return { fromByte: tail.fromByte, toByte: tail.toByte };+ const snapshot = readAppendSnapshot(sourceAbs, watermark, entry.sha256);+ const delta = appendedDeltaFromSnapshot(watermark, snapshot);+ // The preview's contract is unchanged by the named over-cap outcome: a+ // tail past the cap answers without a section, exactly as before (the+ // section's whole value is the text-able slice).+ return delta !== undefined && delta.kind === "appended"+ ? { fromByte: delta.fromByte, toByte: delta.toByte }+ : undefined;+}++/** What ONE descriptor and one sequential pass over the source answered. */+interface AppendSnapshot {+ /** File size at the read; the prefix digest describes exactly this byteset's head. */+ readonly size: number;+ /**+ * True when the leading watermark bytes still hash to the recorded+ * digest: the append the tail reports is a PROVEN pure append, not a+ * rewrite that happens to have grown.+ */+ readonly prefixProven: boolean;+ /**+ * The bytes past the watermark. Within the delta cap the buffer is the+ * COMPLETE tail; past it only the fact is recorded (`overCap`), the bytes+ * never buffered.+ */+ readonly tail: { readonly overCap: true } | { readonly overCap: false; readonly bytes: Buffer };+ /**+ * Hash of exactly the leading watermark bytes. The caller extends it with+ * the announced delta's leading bytes to reach a digest at any boundary+ * the delta series names, computed with no second read (review L2).+ */+ readonly hashThroughWatermark: Hash;}/**- * Does the SHA-256 of the first `watermark` bytes of `sourceAbs` equal- * `expected`? An unreadable or too-short prefix cannot PROVE an append, so- * it answers false - the modified classification and its full reprocess are- * the safe direction, never a skipped stale delta.+ * Read `sourceAbs` once - one non-blocking descriptor, each byte read+ * exactly once - and answer everything the append proof and the delta cut+ * need: the prefix proof against `expectedDigest` and the appended tail.+ * The whole-source stamp is NOT answered here: the full-size arms record+ * it through updateManifest's in-lock watermark request (review L1), whose+ * hash is the only one that cannot go stale, so this pass never hashes+ * past the tail. `watermark <= 0` answers without reading a single byte -+ * no proof against nothing is possible and no tail can exist - so a first+ * ingest's only whole-source pass is the in-lock stamp hash itself, which+ * is the first watermark. A source that cannot be opened or is not a+ * regular file throws; the caller answers that fail-safe way.*/-function prefixHashMatches(sourceAbs: string, watermark: number, expected: string): boolean {+function readAppendSnapshot(+ sourceAbs: string,+ watermark: number,+ expectedDigest: string,+): AppendSnapshot {const fd = openSync(sourceAbs, SOURCE_OPEN_FLAGS);try {- const hash = createHash("sha256");- const chunk = Buffer.allocUnsafe(Math.min(PREFIX_HASH_CHUNK_BYTES, watermark));+ const stat = fstatSync(fd);+ if (!stat.isFile()) throw new Error(`not a regular file: ${sourceAbs}`);+ const size = stat.size;+ if (watermark <= 0) {+ // No watermark: the walk below would read every byte to EOF and+ // compute nothing - no prefix to hash, no tail to collect. The+ // full-size stamp this arm feeds IS the first watermark, recorded+ // through the in-lock request, so the guards above (the ENOENT and+ // not-a-file windows the caller's catch answers) are all this call+ // still owes.+ return {+ size,+ prefixProven: false,+ tail: { overCap: false, bytes: Buffer.alloc(0) },+ hashThroughWatermark: createHash("sha256"),+ };+ }+ const hashPrefix = createHash("sha256");+ const hashThroughWatermark = createHash("sha256");+ const tailParts: Buffer[] = [];+ let tailFilled = 0;+ const chunk = Buffer.allocUnsafe(PREFIX_HASH_CHUNK_BYTES);let position = 0;- let remaining = watermark;- while (remaining > 0) {- const count = readSync(fd, chunk, 0, Math.min(chunk.length, remaining), position);- if (count === 0) return false;- hash.update(count === chunk.length ? chunk : chunk.subarray(0, count));+ for (;;) {+ const count = readSync(fd, chunk, 0, chunk.length, position);+ if (count === 0) break;+ const buf = count === chunk.length ? chunk : chunk.subarray(0, count);+ const prefixEnd = Math.min(watermark - position, count);+ if (prefixEnd > 0) {+ const head = buf.subarray(0, prefixEnd);+ hashPrefix.update(head);+ hashThroughWatermark.update(head);+ }+ if (position + count > watermark && tailFilled <= EXTRACTION_DELTA_MAX_BYTES) {+ const part = buf.subarray(Math.max(0, watermark - position));+ tailParts.push(part);+ tailFilled += part.length;+ }position += count;- remaining -= count;}- return hash.digest("hex") === expected;+ return {+ size,+ prefixProven: hashPrefix.digest("hex") === expectedDigest,+ tail:+ tailFilled > EXTRACTION_DELTA_MAX_BYTES+ ? { overCap: true }+ : { overCap: false, bytes: Buffer.concat(tailParts) },+ hashThroughWatermark,+ };} finally {closeSync(fd);}}+/**+ * The appended delta a snapshot proves, boundary-safe: a pure append within+ * the delta cap cut at the last complete newline. Everything else - no+ * watermark, a rewrite, a shrink, a tail past the cap - answers no delta;+ * the modified classification and its full reprocess are the safe direction.+ *+ * On the proven shape this ALSO extends the snapshot's+ * `hashThroughWatermark` with exactly the announced bytes, so the caller's+ * next `digest()` is the digest of the leading `toByte` bytes - the stamp's+ * prefix, computed with no second read (review L2). Calling it is therefore+ * not idempotent on the snapshot; the CLI preview discards its snapshot and+ * never digests.+ */+function appendedDeltaFromSnapshot(+ watermark: number,+ snapshot: AppendSnapshot,+): AppendedDeltaOutcome | undefined {+ if (watermark <= 0 || !snapshot.prefixProven || snapshot.size <= watermark) return undefined;+ const tail = snapshot.tail;+ if (tail.overCap) {+ // The append is proven but its text is reprocess-scale: name it, and+ // report the whole appended range without reading it - only the range+ // is consumed (review L3).+ return { kind: "appended-over-cap", fromByte: watermark, toByte: snapshot.size };+ }+ const cut = cutAppendedDelta(tail.bytes, watermark);+ snapshot.hashThroughWatermark.update(tail.bytes.subarray(0, cut.toByte - watermark));+ return { kind: "appended", fromByte: cut.fromByte, toByte: cut.toByte };+}/*** Run the code-structure pre-extraction pass over a vault-file source. A sourcediff --git a/src/core/brain/ingest/tail-read.ts b/src/core/brain/ingest/tail-read.tsindex 746f0816..59ce712f 100644--- a/src/core/brain/ingest/tail-read.ts+++ b/src/core/brain/ingest/tail-read.ts@@ -19,15 +19,13 @@* delta covers nothing (`toByte === fromByte`) - and what happens to* the watermark then is the CALLER's decision, not this module's.* The production caller ({@link ./ingest.ts}) stamps the extraction- * watermark at the file's FULL size on every trusted run, so an- * empty delta still advances it past the unterminated fragment:- * that fragment is announced by no delta, and the next append's- * delta starts PAST it rather than stitching it in. Only a caller- * that leaves the watermark at `fromByte` gets the- * stitch-the-fragment-later behavior this boundary cut was shaped- * for. The ingest lane accepts the trade because extraction itself- * stays whole-source - the fragment is extracted, just never- * announced as a delta.+ * watermark at the announced delta's `toByte`, so an empty delta+ * leaves the watermark at `fromByte`: the unterminated trailing+ * fragment stays unstamped, and the next append's delta starts AT it+ * rather than past it - the fragment is stitched into the series the+ * first time an append ends with a newline. The ingest lane accepts+ * the trade because extraction itself stays whole-source - the+ * fragment is extracted, just never announced as a delta on its own.* - A file that SHRANK below the offset (rotated, truncated, rewritten)* falls back to the full current content with the shrink NAMED - the* full-reprocess direction, never a bogus negative slice.@@ -130,9 +128,20 @@ export function readTail(/*** Cut the appended bytes into an honest delta: decode from the first* UTF-8-safe boundary, through the last complete newline, and report the- * exact byte range the text covers.+ * exact byte range the text covers. Exported so the single-pass append+ * snapshot ({@link ./ingest.ts}) cuts the tail bytes it already read with+ * the same discipline {@link readTail} applies, never a second cut rule.*/-function appendedDelta(bytes: Uint8Array, fromByte: number): TailReadResult {+export interface AppendedCut {+ /** The decoded delta text; empty when no complete line exists yet. */+ readonly text: string;+ /** First byte of the source the returned text was decoded from. */+ readonly fromByte: number;+ /** One past the last byte the text covers - the watermark position. */+ readonly toByte: number;+}++export function cutAppendedDelta(bytes: Uint8Array, fromByte: number): AppendedCut {let start = 0;while (start < bytes.length && isContinuationByte(bytes[start]!)) start++;// One past the last complete newline; `start` when the range holds no@@ -140,13 +149,19 @@ function appendedDelta(bytes: Uint8Array, fromByte: number): TailReadResult {let end = bytes.length;while (end > start && bytes[end - 1] !== 0x0a) end--;return {- kind: "appended",text: Buffer.from(bytes.subarray(0, end).subarray(start)).toString("utf8"),fromByte: fromByte + start,toByte: fromByte + end,};}+/**+ * Shape {@link cutAppendedDelta} into a full tail-read answer.+ */+function appendedDelta(bytes: Uint8Array, fromByte: number): TailReadResult {+ return { kind: "appended", ...cutAppendedDelta(bytes, fromByte) };+}+/*** Positioned reads from `start` through EOF, at most `maxBytes` bytes.* `overCeiling` is the named refusal: the caller answers withdiff --git a/src/core/brain/maintenance/collect.ts b/src/core/brain/maintenance/collect.tsindex e734c7be..66a9b642 100644--- a/src/core/brain/maintenance/collect.ts+++ b/src/core/brain/maintenance/collect.ts@@ -34,9 +34,11 @@ export interface MaintenanceCollectError {export interface CollectMaintenanceActionsOptions {/*** Observes each per-signal collection failure as it is absorbed. The scan- * stays fail-soft either way; the doctor, digest and health surfaces call- * without an observer and keep today's quiet contract, and a caller- * diagnosing a thin report passes one to hear what refused.+ * stays fail-soft either way; the doctor, digest and operator-summary+ * surfaces pass a collector and render what refused as a named+ * `collect_errors` row (review M10), so a thin report because the vault+ * is healthy is no longer indistinguishable from a thin report because a+ * scan refused.*/readonly onError?: (error: MaintenanceCollectError) => void;}diff --git a/src/core/brain/maintenance/lane.ts b/src/core/brain/maintenance/lane.tsindex 9209eb53..82d55580 100644--- a/src/core/brain/maintenance/lane.ts+++ b/src/core/brain/maintenance/lane.ts@@ -16,11 +16,14 @@* like a quiet one.* 4. capacity - the vault's own storage forecast, from* `storage.cap_bytes`; unconfigured (the default) =- * always open and never measured. A measured critical- * tier skips the whole lane; where the census is- * degenerate the gate stays open and journals WHY, on- * its own line, for the same reason the pressure gate- * does.+ * always open and never measured. A measured tier at or+ * past the pause line (`storage.pause_percent`, default+ * the warn band) pauses the whole lane - background+ * production backs off one band before the wall, while+ * interactive ingest still works until the critical+ * tier refuses it. Where the census is degenerate the+ * gate stays open and journals WHY, on its own line,+ * for the same reason the pressure gate does.* 5. lease - the expiring SQLite lease; never bypassable, even* with --force, because two concurrent heavy passes on* one vault is the exact failure this lane exists to@@ -317,10 +320,19 @@ export interface RunMaintenanceResult {* What an operator does about a lane the capacity gate skipped. The* reclamation verbs are the remedy, then a re-run; raising the cap is* the operator's own decision and names the file only they should edit.+ *+ * The remedy also names pruning the recovery points, because both+ * reclamation verbs archive all of `Brain/` into `Brain/.snapshots/`+ * before they free anything - a compressed copy of the vault lands in a+ * COUNTED surface first - and a remedy that can grow the vault while it+ * shrinks the ledger is not one an operator can follow blind at 96+ * percent.*/export const MAINTENANCE_CAPACITY_REMEDY ="free space by running receipt compaction or retiring near-duplicate copies, " +- "or raise storage.cap_bytes in Brain/_brain.yaml, then re-run";+ "then prune the recovery points both write into Brain/.snapshots " ++ "(snapshots.retention_count bounds them), or raise storage.cap_bytes in " ++ "Brain/_brain.yaml, then re-run";/** The capacity block both lane front doors attach to a skipped run. */export interface MaintenanceCapacityReport {@@ -337,6 +349,13 @@ export interface MaintenanceCapacityReport {readonly capacity_percent: number | null;readonly measured_bytes: number | null;readonly cap_bytes: number | null;+ /**+ * The pause line the lane decided on (`storage.pause_percent`), when+ * the gate carried one. A tier of `ok` beside a skip names this line+ * as what closed the lane; a null names a decision with no pause line+ * in force, skipped on the critical tier alone.+ */+ readonly pause_percent: number | null;}/** The report for the gate decision that skipped a run. */@@ -348,6 +367,7 @@ export function capacityReport(decision: CapacityGateDecision): MaintenanceCapaccapacity_percent: reading?.percentUsed ?? null,measured_bytes: reading?.measuredBytes ?? null,cap_bytes: reading?.capBytes ?? null,+ pause_percent: decision.pausePercent ?? null,};}@@ -443,6 +463,9 @@ function evaluateStorageCapacityGate(capBytes: capacity.cap_bytes,warnPercent: capacity.warn_percent,criticalPercent: capacity.critical_percent,+ // The lane's own pause line: background production backs off one+ // band before the wall the ingest refusal keeps.+ pausePercent: capacity.pause_percent,warnDays: capacity.warn_days,criticalDays: capacity.critical_days,});@@ -507,6 +530,20 @@ export async function runMaintenance(: {}),});}+ // The gate evaluated, so its census is a sample - on a SKIP and on a+ // RUN alike. The M5 blind spot was sampling skips only: on a vault+ // where nobody runs the storage report, no sample was ever recorded,+ // no trend existed and the day-based tiers could never fire. The+ // decision already walked the vault, so recording its census costs+ // one walk and never two; day-bucketing in the forecast keeps the+ // extra rows cheap. Fail-soft by contract; a sample that cannot be+ // written never turns a run into a crash.+ if (decision.capacity?.census !== undefined) {+ sampleStorageUsage(vault, {+ now: opts.now,+ usage: decision.capacity.census,+ });+ }if (decision.verdict !== MAINTENANCE_VERDICT.run) {// One report serves every skip surface below: the journal row and// both front doors read the same numbers, so they cannot disagree@@ -515,19 +552,6 @@ export async function runMaintenance(decision.verdict === MAINTENANCE_VERDICT.skippedCapacity && decision.capacity !== undefined? capacityReport(decision.capacity): undefined;- // Sample BEFORE the skip is recorded or returned: the forecast- // series must keep accumulating exactly while producers are paused,- // so a skipped run still appends its sample. The gate's own census- // is what is recorded - the decision already walked the vault, and- // sampling by re-measuring would tax every skipped run with a- // second full walk. Fail-soft by contract; a sample that cannot be- // written never turns a skip into a crash.- if (decision.verdict === MAINTENANCE_VERDICT.skippedCapacity) {- sampleStorageUsage(vault, {- now: opts.now,- ...(decision.capacity?.census !== undefined ? { usage: decision.capacity.census } : {}),- });- }appendJournal(vault, {ts: nowIso,holder: opts.holder,diff --git a/src/core/brain/metrics.ts b/src/core/brain/metrics.tsindex dce09039..e63640c0 100644--- a/src/core/brain/metrics.ts+++ b/src/core/brain/metrics.ts@@ -11,10 +11,11 @@* {"schema":"o2b.metrics.v1","surface":"...","run_at":"...","payload":{...}}** Records are RUN-LEVEL (one per index run, discovery pass, benchmark- * or tuning run) - per-query events stay in recall telemetry. Writes- * use O_APPEND single lines (the maintenance-journal pattern) so- * concurrent writers interleave instead of racing a rewrite; reads- * are fail-soft (missing dir/file -> empty, torn lines skipped).+ * or tuning run) - per-query events stay in recall telemetry. Writes are+ * single-line O_APPEND appends under the shard's own per-file lock (the+ * same primitive the continuity appender holds), so concurrent appends+ * interleave and a rewrite cannot move the file out from under one;+ * reads are fail-soft (missing dir/file -> empty, torn lines skipped).** Evolution rule mirrors continuity records: additive optional fields* do NOT bump the version; renames, removals, or semantic changes@@ -31,6 +32,7 @@ import {shardedFileName,type LedgerShardGrammar,} from "./ledger-shards.ts";+import { acquireLockSync } from "./sync-lockfile.ts";import { assertVaultIdentityForWrite } from "./vault-identity.ts";/** On-disk schema version stamped on every metric record. */@@ -141,7 +143,19 @@ export function appendMetric(vault: string, input: AppendMetricInput): void {};const path = surfacePath(vault, input.surface);mkdirSync(dirname(path), { recursive: true });- appendFileSync(path, JSON.stringify(record) + "\n");+ // The same per-file lock the continuity appender holds: a bare O_APPEND+ // write interleaves with other APPENDS, but a shard REWRITE (the receipt+ // fold's compare-and-swap) moves the file out from under a concurrent+ // append and the sample is silently lost. Under the lock the rewrite+ // either waits for this append to land (and then reports drift against+ // its recount) or fails its own lock acquisition - either way the sample+ // is a named outcome, never a silent overwrite.+ const handle = acquireLockSync(path);+ try {+ appendFileSync(path, JSON.stringify(record) + "\n");+ } finally {+ handle.release();+ }}/**diff --git a/src/core/brain/policy/blocks/storage.ts b/src/core/brain/policy/blocks/storage.tsindex 717c00d5..b0168c02 100644--- a/src/core/brain/policy/blocks/storage.ts+++ b/src/core/brain/policy/blocks/storage.ts@@ -53,6 +53,7 @@ const KNOWN_KEYS = ["cap_bytes","warn_percent","critical_percent",+ "pause_percent","warn_days","critical_days",] as const;@@ -67,18 +68,27 @@ export const BRAIN_STORAGE_DEFAULTS: ResolvedBrainStorageConfig = Object.freeze(cap_bytes: null,warn_percent: STORAGE_WARN_PERCENT_DEFAULT,critical_percent: STORAGE_CRITICAL_PERCENT_DEFAULT,+ pause_percent: STORAGE_WARN_PERCENT_DEFAULT,warn_days: STORAGE_WARN_DAYS_DEFAULT,critical_days: STORAGE_CRITICAL_DAYS_DEFAULT,}) as ResolvedBrainStorageConfig;-/** Merge a parsed `storage` block (or `undefined`) with the defaults. */+/**+ * Merge a parsed `storage` block (or `undefined`) with the defaults.+ *+ * An absent `pause_percent` follows the block's warn band - the lane+ * pauses where the forecast warns unless the operator names the pause+ * line themselves.+ */export function resolveStorage(cfg: BrainConfig): ResolvedBrainStorageConfig {const s = cfg.storage;if (s === undefined) return BRAIN_STORAGE_DEFAULTS;+ const warnPercent = s.warn_percent ?? BRAIN_STORAGE_DEFAULTS.warn_percent;return {cap_bytes: s.cap_bytes ?? BRAIN_STORAGE_DEFAULTS.cap_bytes,- warn_percent: s.warn_percent ?? BRAIN_STORAGE_DEFAULTS.warn_percent,+ warn_percent: warnPercent,critical_percent: s.critical_percent ?? BRAIN_STORAGE_DEFAULTS.critical_percent,+ pause_percent: s.pause_percent ?? warnPercent,warn_days: s.warn_days ?? BRAIN_STORAGE_DEFAULTS.warn_days,critical_days: s.critical_days ?? BRAIN_STORAGE_DEFAULTS.critical_days,};@@ -90,6 +100,8 @@ export function resolveStorage(cfg: BrainConfig): ResolvedBrainStorageConfig {* cap_bytes: 1073741824 # positive integer bytes; absent = unbounded* warn_percent: 80 # integer in [1, 100]; must be < critical_percent* critical_percent: 95 # integer in [1, 100]; must be > warn_percent+ * pause_percent: 80 # integer in [1, 100]; must be < critical_percent+ * # and defaults to the warn band* warn_days: 30 # positive integer; must be > critical_days* critical_days: 7 # positive integer; must be < warn_days*/@@ -114,7 +126,16 @@ export function parseStorageBlock(ctx: BlockParseContext): BrainStorageConfig |"storage.critical_percent",ctx.source,);+ const pausePercent = readBoundedInt(+ map,+ "pause_percent",+ STORAGE_PERCENT_MIN,+ STORAGE_PERCENT_MAX,+ "storage.pause_percent",+ ctx.source,+ );assertCriticalAboveWarn(warnPercent, criticalPercent, ctx);+ assertPauseBelowCritical(pausePercent, criticalPercent, ctx);const warnDays = readBoundedInt(map,"warn_days",@@ -138,6 +159,7 @@ export function parseStorageBlock(ctx: BlockParseContext): BrainStorageConfig |...(capBytes !== undefined ? { cap_bytes: capBytes } : {}),...(warnPercent !== undefined ? { warn_percent: warnPercent } : {}),...(criticalPercent !== undefined ? { critical_percent: criticalPercent } : {}),+ ...(pausePercent !== undefined ? { pause_percent: pausePercent } : {}),...(warnDays !== undefined ? { warn_days: warnDays } : {}),...(criticalDays !== undefined ? { critical_days: criticalDays } : {}),};@@ -203,6 +225,32 @@ function assertCriticalAboveWarn(}}+/**+ * The pause line must sit strictly BELOW the critical band, whether+ * critical is spelled in the block or at its default: a pause at or past+ * the refuse line could never pause anything - the ingest wall fires+ * first at critical - so the knob would sit configured and dead. Only an+ * explicitly set `pause_percent` is checked: an absent one defaults to+ * the warn band, and warn-below-critical is already the existing pair+ * check's contract, so a default-inferred pause can never violate this+ * without that check firing first.+ */+function assertPauseBelowCritical(+ pausePercent: number | undefined,+ criticalPercent: number | undefined,+ ctx: BlockParseContext,+): void {+ if (pausePercent === undefined) return;+ const critical = criticalPercent ?? BRAIN_STORAGE_DEFAULTS.critical_percent;+ if (pausePercent < critical) return;+ throw new BrainConfigError(+ `must be smaller than critical_percent (${critical}); the pause line governs background ` ++ `work and interactive ingest is already refused at the critical band - got ${pausePercent}`,+ "storage.pause_percent",+ ctx.source,+ );+}+/*** The critical day horizon must be NEARER than the warn horizon, same* rule, and the error names the key(s) actually at fault the same way.diff --git a/src/core/brain/receipts/compact.ts b/src/core/brain/receipts/compact.tsindex f8673e02..f19150fa 100644--- a/src/core/brain/receipts/compact.ts+++ b/src/core/brain/receipts/compact.ts@@ -12,7 +12,10 @@* The gates, in order, before anything is written:** 1. Shard refs resolve through the family's name grammar - a name- * that is not a shard is a caller error, never a traversal.+ * that is not a shard is a caller error, never a traversal - and+ * name a shard THIS device owns: another device's shard is refused+ * by name, because the per-file lock and the compare-and-swap are+ * local only and a rewrite here would diverge the peer's copy.* 2. A fresh per-shard recount against the disk (never the caller's* scan numbers) feeds the count guard: an `expect` mismatch aborts* before any write, snapshot included.@@ -28,19 +31,25 @@* drifted since the recount, or would not strictly shrink is collected* into `failures` and the run still completes. Writes are atomic per* shard - `atomicWriteFileSync` with `expectBefore` compare-and-swap -- * under the shard's own lock. The real guarantee is narrower than "the- * append paths hold this lock": it excludes the continuity appenders- * only. A metrics shard is guarded by the compare-and-swap alone - the- * metrics append path writes bare `appendFileSync` - so a concurrent- * metrics append landing inside the rewrite window is lost without- * error, not refused as drift. Those are advisory samples, and the- * forecast series that reads them tolerates the gap.+ * under the shard's own lock. Both append paths hold that same lock (the+ * continuity appender always has, the metrics appender since its samples+ * stopped being losable), so an append landing inside a rewrite window+ * serializes behind it or fails loudly - the compare-and-swap then+ * reports drift against the recount instead of the rename swallowing the+ * appended line.** Idempotency: a re-run recounts a compacted shard, finds nothing* foldable, and writes nothing - no rewrite, no snapshot, no audit- * record. The audit is the additive `receipt_compaction` continuity+ * record. When a later run does fold new records into an+ * already-compacted shard (a slid cutoff, a re-import), it merges into+ * the aggregate line it finds - count and bytes summed, first and last+ * widened - so a shard never carries a second line per (kind, month).+ * The audit is the additive `receipt_compaction` continuity* kind (one record per confirmed run that folded, never on a dry run),- * carrying the honest per-shard outcomes including failures. An audit+ * carrying the honest per-shard outcomes including failures, and the+ * run's net cost - the ledger shrink beside the recovery point's own+ * bytes, both on the result and in the audit payload, since the archive+ * lands in a surface the storage census counts. An audit* record that cannot be appended after the shards are durable is a* named outcome on the result (`auditError`), never a post-write throw:* the run's work is done by then, and discarding it would leave the@@ -52,7 +61,7 @@* string `run_at`, so an aggregate is inert bytes to every reader.*/-import { readFileSync } from "node:fs";+import { readFileSync, statSync } from "node:fs";import { resolveAgentName } from "../../config.ts";import { atomicWriteFileSync, isFileDrift } from "../../fs-atomic.ts";@@ -70,6 +79,7 @@ import {classifyContinuityRecord,classifyMetricRecord,isFoldAggregateLine,+ RECEIPT_SHARD_FOREIGN_DEVICE,RECEIPT_SHARD_UNREADABLE,resolveReceiptShardPath,} from "./scan.ts";@@ -82,6 +92,7 @@ import {type ReceiptLedgerFamily,type ReceiptShardRef,} from "./types.ts";+import { resolveAppendShardId } from "../ledger-shards.ts";const MS_PER_DAY = 86_400_000;@@ -126,12 +137,30 @@ export interface ReceiptFoldGroup {/** One fold group together with where its aggregate line is written. */interface GroupEntry {- /** Index into the shard's lines of the group's first folded record. */- readonly firstIndex: number;+ /**+ * Line index the group's aggregate line is written at: the position of+ * the group's first folded record, or of the earliest prior aggregate+ * line the group merged, whichever comes first.+ */+ readonly emitIndex: number;/** The fold group, replaced in place as later records join it. */group: ReceiptFoldGroup;}+/**+ * The prior aggregate one existing fold line carries, when the line is+ * one this family's engine wrote and its numbers are intact enough to+ * merge into. Anything else stays verbatim.+ */+interface PriorFold {+ readonly kind: string;+ readonly month: string;+ readonly count: number;+ readonly bytes: number;+ readonly first: string;+ readonly last: string;+}+/** One shard the confirmed run could not rewrite, with the reason. */export interface ReceiptFoldFailure {readonly shard: string;@@ -161,6 +190,21 @@ export interface ReceiptCompactResult {readonly recoverability?: RecoverabilityVerdict;/** Present on a confirmed run that appended its audit record. */readonly auditRecordId?: string;+ /**+ * Present on a confirmed run that took its recovery point: the+ * archive's size in bytes, measured once durable. The recovery point+ * is a whole-`Brain/` archive written into a surface the storage+ * census counts as vault content BEFORE the ledger shrink lands, so+ * the shrink alone is not the run's cost (M6).+ */+ readonly snapshotBytes?: number;+ /**+ * Present with {@link snapshotBytes}: the ledger shrink minus the+ * recovery point's growth. Negative means the run left the vault+ * LARGER than it found it - the figure an operator at a capacity tier+ * needs before believing a fold moved the needle.+ */+ readonly netDeltaBytes?: number;/*** Present on a confirmed run whose audit record could not be appended:* the shards above are folded and the recovery point exists, but the@@ -201,6 +245,8 @@ interface ShardPlan {readonly kept: number;/** Fold groups in first-occurrence order, each with its aggregate slot. */readonly groupEntries: ReadonlyArray<GroupEntry>;+ /** Prior aggregate lines a merge consumed; the rebuild drops them. */+ readonly mergedPriorLines: ReadonlyArray<number>;/** Exact text the recount read; the compare-and-swap anchor. */readonly text: string;readonly bytesBefore: number;@@ -264,6 +310,47 @@ function instantOf(timestamp: string): number {return Date.parse(timestamp);}+/**+ * Read the prior aggregate one parsed line carries, when the line is a+ * fold line this family's engine wrote and its fields are intact enough+ * to merge into: matching family, string kind/surface and month, string+ * first/last bounds, and a positive safe-integer count and non-negative+ * byte total. Anything else - a foreign family's line, a corrupt one -+ * returns `null` and stays verbatim.+ */+function mergeableFold(parsed: unknown, family: ReceiptLedgerFamily): PriorFold | null {+ if (!isFoldAggregateLine(parsed)) return null;+ const fold = (parsed as { readonly fold: Record<string, unknown> }).fold;+ if (fold["family"] !== family) return null;+ const continuity = family === RECEIPT_LEDGER_FAMILY.continuity;+ const kind = fold[continuity ? "kind" : "surface"];+ const first = fold[continuity ? "first_created_at" : "first_run_at"];+ const last = fold[continuity ? "last_created_at" : "last_run_at"];+ const count = fold["count"];+ const bytes = fold["bytes"];+ if (typeof kind !== "string" || typeof fold["month"] !== "string") return null;+ if (typeof first !== "string" || typeof last !== "string") return null;+ if (typeof count !== "number" || !Number.isSafeInteger(count) || count < 1) return null;+ if (typeof bytes !== "number" || !Number.isSafeInteger(bytes) || bytes < 0) return null;+ return { kind, month: fold["month"] as string, count, bytes, first, last };+}++/**+ * Fold a prior aggregate's mass into a group this run is building: counts+ * and byte totals sum, and the first/last bounds widen by instant - the+ * same compare the fold itself uses, so a fractional-second stamp cannot+ * win a bound it does not own.+ */+function mergePrior(group: ReceiptFoldGroup, prior: PriorFold): ReceiptFoldGroup {+ return {+ ...group,+ count: group.count + prior.count,+ bytes: group.bytes + prior.bytes,+ first: instantOf(prior.first) < instantOf(group.first) ? prior.first : group.first,+ last: instantOf(prior.last) > instantOf(group.last) ? prior.last : group.last,+ };+}+/*** Recount one named shard against the disk. A shard that cannot be read* yields a failure instead of a plan - named unreadable, with its@@ -279,6 +366,13 @@ function planShard(): { readonly plan: ShardPlan } | { readonly failure: ReceiptFoldFailure } {const display = receiptShardRefId(ref);const resolved = resolveReceiptShardPath(vault, ref);+ // A shard is one device's file and the lock and compare-and-swap below+ // are local only: rewriting a peer's shard here would diverge the copy+ // it is appending to. Refused by name, on previews and confirmed runs+ // alike - a shard the owning device must fold.+ if (resolved.shardId !== "" && resolved.shardId !== resolveAppendShardId()) {+ return { failure: { shard: display, message: RECEIPT_SHARD_FOREIGN_DEVICE } };+ }let text: string;try {text = readFileSync(resolved.path, "utf8");@@ -288,6 +382,15 @@ function planShard(const lines = text.split("\n").filter((line) => line.trim() !== "");const byKey = new Map<string, GroupEntry>();const groupEntries: GroupEntry[] = [];+ // Prior aggregate lines a merge consumed: the rebuild drops them in+ // favour of the one merged aggregate line their group emits.+ const mergedPriorLines = new Set<number>();+ // Prior aggregates met before the group's first new record: one per+ // key, absorbed when the entry is created.+ const priorsByKey = new Map<+ string,+ Array<{ readonly index: number; readonly prior: PriorFold }>+ >();let foldable = 0;let kept = 0;for (const [index, line] of lines.entries()) {@@ -298,6 +401,27 @@ function planShard(kept += 1; // a torn line is nobody's to foldcontinue;}+ if (isFoldAggregateLine(parsed)) {+ // M13: an aggregate line this engine wrote joins its key's group -+ // repeated runs must never stack a second line per (kind, month).+ // A group with no new record yet is created by the first record+ // that reaches it; until then the line stays verbatim untouched.+ const prior = mergeableFold(parsed, ref.family);+ if (prior !== null) {+ const key = `${prior.kind}\u0000${prior.month}`;+ const entry = byKey.get(key);+ if (entry !== undefined) {+ entry.group = mergePrior(entry.group, prior);+ mergedPriorLines.add(index);+ continue;+ }+ const pendings = priorsByKey.get(key) ?? [];+ pendings.push({ index, prior });+ priorsByKey.set(key, pendings);+ }+ kept += 1;+ continue;+ }const verdict = verdictFor(ref.family, parsed, resolved.base, cutoffMs);if (!verdict.foldable) {kept += 1;@@ -308,18 +432,25 @@ function planShard(const bytes = Buffer.byteLength(line, "utf8");const existing = byKey.get(key);if (existing === undefined) {- const entry: GroupEntry = {- firstIndex: index,- group: {- shard: display,- kind: verdict.kind!,- month: verdict.month!,- count: 1,- first: verdict.timestamp!,- last: verdict.timestamp!,- bytes,- },+ let emitIndex = index;+ let group: ReceiptFoldGroup = {+ shard: display,+ kind: verdict.kind!,+ month: verdict.month!,+ count: 1,+ first: verdict.timestamp!,+ last: verdict.timestamp!,+ bytes,};+ const pendings = priorsByKey.get(key);+ if (pendings !== undefined) {+ for (const pending of pendings) {+ group = mergePrior(group, pending.prior);+ mergedPriorLines.add(pending.index);+ emitIndex = Math.min(emitIndex, pending.index);+ }+ }+ const entry: GroupEntry = { emitIndex, group };byKey.set(key, entry);groupEntries.push(entry);continue;@@ -350,6 +481,7 @@ function planShard(foldable,kept,groupEntries: Object.freeze(groupEntries),+ mergedPriorLines: Object.freeze([...mergedPriorLines]),text,bytesBefore: Buffer.byteLength(text, "utf8"),},@@ -383,23 +515,20 @@ function aggregateLine(family: ReceiptLedgerFamily, group: ReceiptFoldGroup): st/*** Rebuild one shard's content: kept lines verbatim in order, each fold- * group's aggregate line at the position of the group's first folded- * record. Deterministic for a given shard and cutoff.+ * group's - possibly merged - aggregate line at the group's emit index.+ * Deterministic for a given shard and cutoff.*/function rebuildContent(plan: ShardPlan): string {- const byIndex = new Map<number, GroupEntry>(plan.groupEntries.map((e) => [e.firstIndex, e]));- const emitted = new Set<string>();+ const emitAt = new Map<number, GroupEntry>(plan.groupEntries.map((e) => [e.emitIndex, e]));+ const merged = new Set(plan.mergedPriorLines);const out: string[] = [];for (const [index, line] of plan.lines.entries()) {- const entry = byIndex.get(index);+ const entry = emitAt.get(index);if (entry !== undefined) {- const key = `${entry.group.kind}\u0000${entry.group.month}`;- if (!emitted.has(key)) {- emitted.add(key);- out.push(aggregateLine(plan.ref.family, entry.group));- }+ out.push(aggregateLine(plan.ref.family, entry.group));continue;}+ if (merged.has(index)) continue; // replaced by the merged aggregate abovelet parsed: unknown;try {parsed = JSON.parse(line);@@ -491,6 +620,15 @@ export function compactReceiptLedgers();const outcomes = gated.result.outcomes;const foldedShards = new Set(outcomes.map((outcome) => outcome.shard));+ // The run's net cost (M6): the recovery point is a whole-`Brain/`+ // archive already on disk in a surface the storage census counts, so+ // the ledger shrink alone is not what the vault did. Measured off the+ // durable archive; `null` only if the stat itself fails, and then the+ // figures stay absent rather than reading as zero.+ const ledgerShrink =+ sumOutcomes(outcomes, (o) => o.bytesBefore) - sumOutcomes(outcomes, (o) => o.bytesAfter);+ const snapshotBytes = fileSizeBytes(gated.snapshot.path);+ const netDeltaBytes = snapshotBytes === null ? null : ledgerShrink - snapshotBytes;// One audit record per confirmed run that folded something, with the// honest outcomes - failures included - so the ledger says what the@@ -518,6 +656,9 @@ export function compactReceiptLedgers(aggregates: sumOutcomes(outcomes, (o) => o.aggregates),bytes_before: sumOutcomes(outcomes, (o) => o.bytesBefore),bytes_after: sumOutcomes(outcomes, (o) => o.bytesAfter),+ ...(snapshotBytes !== null+ ? { snapshot_bytes: snapshotBytes, net_delta_bytes: netDeltaBytes }+ : {}),failures: failures.map((failure) => ({ ...failure })),},}).id;@@ -542,11 +683,29 @@ export function compactReceiptLedgers(totals: totalsOf(outcomes, failures),snapshotRunId: gated.result.runId,recoverability: gated.recoverability,+ ...(snapshotBytes !== null+ ? { snapshotBytes, netDeltaBytes: ledgerShrink - snapshotBytes }+ : {}),...(auditRecordId !== undefined ? { auditRecordId } : {}),...(auditError !== undefined ? { auditError } : {}),});}+/**+ * Byte length of the recovery-point archive, measured off disk. `null`+ * only when the stat itself fails - the snapshot gate has already made+ * the archive durable or aborted the run - and the net figures then stay+ * absent rather than reading as zero.+ */+function fileSizeBytes(path: string): number | null {+ try {+ const stats = statSync(path);+ return stats.isFile() ? stats.size : null;+ } catch {+ return null;+ }+}+/** Rewrite every planned shard, fail-soft per shard, under one snapshot. */function rewriteShards(plans: ReadonlyArray<ShardPlan>,diff --git a/src/core/brain/receipts/scan.ts b/src/core/brain/receipts/scan.tsindex e4fe5298..b18da93b 100644--- a/src/core/brain/receipts/scan.ts+++ b/src/core/brain/receipts/scan.ts@@ -11,19 +11,27 @@** Protected classes, never foldable whatever their age:*- * - `source_invalidation` - the deleteBySource audit trail;- * - `pre_compact_extract` / `post_compact_audit` - the Hermes- * context-compaction receipts, which a storage fold must not- * rewrite under a colliding name;+ * - every continuity kind outside the closed+ * {@link FOLDABLE_CONTINUITY_KINDS} allowlist - folding is opt-in+ * per kind, and the functional kinds (the session read models, the+ * by-id lookups, the entity index, the audit trails) are read back+ * as data, not aggregated as telemetry. Unknown kinds fail closed to+ * keep the same way: a newer build's records are not this build's to+ * fold;* - any record whose payload carries one of* {@link CLIP_PROTECTED_PAYLOAD_KEYS} - folding would destroy the* session/agent correlation the clip rule guarantees survives;- * - any continuity kind this build does not know - a newer build's- * records are not this build's to fold (fail closed to keep);* - any record whose timestamp cannot be compared (fail closed).** Ledger rewrites happen only in `compact.ts`; this module never binds a* write primitive.+ *+ * The inventory is also per device: a shard is one device's file (the+ * legacy un-sharded name is the empty shard id), and only the shards the+ * LOCAL lock and compare-and-swap can guard are offered - this device's+ * shard plus the legacy file. A peer's shard never appears here, so no+ * surface can hand an operator a plan that would diverge another+ * device's copy; compact refuses one named by hand with the same verdict.*/import { readFileSync, statSync } from "node:fs";@@ -35,12 +43,13 @@ import {JSONL_LEDGER_EXT,listShardedFiles,parseShardedName,+ resolveAppendShardId,type LedgerShardGrammar,} from "../ledger-shards.ts";import { metricsDir } from "../metrics.ts";import { toPosix } from "../../path-safety.ts";import {- PROTECTED_CONTINUITY_KINDS,+ FOLDABLE_CONTINUITY_KINDS,RECEIPT_FOLD_DEFAULT_AGE_DAYS,RECEIPT_FOLD_SCHEMA,RECEIPT_LEDGER_FAMILIES,@@ -83,6 +92,17 @@ const MS_PER_DAY = 86_400_000;*/export const RECEIPT_SHARD_UNREADABLE = "unreadable";+/**+ * The closed sentence a foreign shard is refused with. Shards are named+ * per device so exactly one device ever writes a given file, while the+ * fold's lock and compare-and-swap are local only - a rewrite here would+ * diverge the copy the owning device is appending to, which is the+ * sync-conflict class the shard layout exists to prevent. The refusal+ * names the remedy: the fold belongs on the owning device.+ */+export const RECEIPT_SHARD_FOREIGN_DEVICE =+ "another device's shard; compact it on the owning device";+/** ISO-8601 UTC timestamp shape, the one the metrics ledger writes. */const ISO_UTC_RE = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{1,3})?Z$/;@@ -96,18 +116,20 @@ export function payloadCarriesProtectedKey(payload: unknown): boolean {}/**- * Classify one continuity record against the cutoff. Unknown kinds,- * protected kinds, protected-key carriers and uncompared timestamps are- * protected (fail closed); canonical timestamps at or after the cutoff- * are young; everything else is foldable. Every field is optional: a- * line missing one classifies through the same guards and fails closed.+ * Classify one continuity record against the cutoff. Folding is opt-in+ * per kind: an unknown kind, or any kind outside the closed+ * {@link FOLDABLE_CONTINUITY_KINDS} allowlist, a protected-key carrier+ * and an uncompared timestamp are all protected (fail closed); canonical+ * timestamps at or after the cutoff are young; a listed telemetry kind+ * older than that is foldable. Every field is optional: a line missing+ * one classifies through the same guards and fails closed.*/export function classifyContinuityRecord(record: { readonly kind?: unknown; readonly createdAt?: unknown; readonly payload?: unknown },cutoffMs: number,): ReceiptLineClass {if (!isContinuityRecordKind(record.kind)) return "protected";- if ((PROTECTED_CONTINUITY_KINDS as ReadonlyArray<string>).includes(record.kind)) {+ if (!(FOLDABLE_CONTINUITY_KINDS as ReadonlyArray<string>).includes(record.kind)) {return "protected";}if (payloadCarriesProtectedKey(record.payload)) return "protected";@@ -192,6 +214,19 @@ function familyGrammar(family: ReceiptLedgerFamily): LedgerShardGrammar {return family === RECEIPT_LEDGER_FAMILY.continuity ? CONTINUITY_GRAMMAR : METRICS_GRAMMAR;}+/**+ * The caller-error message for a shard ref whose name the family's name+ * grammar does not recognise - a typo, a path traversal, a sync-conflict+ * copy - or `null` when the name is a real shard of the family. The+ * boundaries validate through this BEFORE anything runs, so a malformed+ * shard id is the caller's invalid-params error, never an internal one.+ */+export function receiptShardNameError(ref: ReceiptShardRef): string | null {+ return parseShardedName(ref.name, familyGrammar(ref.family)) === null+ ? `'${ref.name}' is not a ${ref.family} shard name`+ : null;+}+/*** Resolve one shard ref to its file through the family's name grammar.* A name the grammar does not recognise - a typo, a path traversal, a@@ -202,10 +237,10 @@ export function resolveReceiptShardPath(vault: string,ref: ReceiptShardRef,): { readonly path: string; readonly base: string; readonly shardId: string } {- const parsed = parseShardedName(ref.name, familyGrammar(ref.family));- if (parsed === null) {- throw new Error(`'${ref.name}' is not a ${ref.family} shard name`);- }+ const nameError = receiptShardNameError(ref);+ if (nameError !== null) throw new Error(nameError);+ // The name error is null exactly when the grammar accepts the name.+ const parsed = parseShardedName(ref.name, familyGrammar(ref.family))!;return {path: join(receiptLedgerDir(vault, ref.family), ref.name),base: parsed.base,@@ -332,10 +367,16 @@ export function scanReceiptLedgers(const nowMs = (opts.now ?? new Date()).getTime();const cutoffMs = nowMs - olderThanDays * MS_PER_DAY;const cutoff = canonicalSecond(cutoffMs);+ const localShardId = resolveAppendShardId();const shards: ReceiptShardScan[] = [];const errors: ReceiptScanError[] = [];for (const family of RECEIPT_LEDGER_FAMILIES) {for (const shard of listShardedFiles(receiptLedgerDir(vault, family), familyGrammar(family))) {+ // A shard is one device's file: this device's shard and the legacy+ // un-sharded file are the only rewrites the local lock and+ // compare-and-swap can guard, so they are the only shards the+ // inventory offers. A peer's shard is its own device's to fold.+ if (shard.shardId !== "" && shard.shardId !== localShardId) continue;const result = scanShard(vault, family, shard, cutoffMs);if ("error" in result) errors.push(result.error);else shards.push(result.row);diff --git a/src/core/brain/receipts/types.ts b/src/core/brain/receipts/types.tsindex 07944097..13aa76a3 100644--- a/src/core/brain/receipts/types.ts+++ b/src/core/brain/receipts/types.ts@@ -118,16 +118,83 @@ export const RECEIPT_FOLD_SCHEMA = "o2b.receipt-fold.v1";export const RECEIPT_FOLD_DEFAULT_AGE_DAYS = 30;/**- * The continuity kinds the fold must NEVER touch, whatever their age.- * `source_invalidation` is the deleteBySource audit trail; the two- * Hermes context-compaction kinds are that feature's own receipts and- * folding them under a storage operation's name would corrupt the one- * vocabulary by colliding with the other.+ * The continuity kinds a fold may aggregate once they are past the age+ * threshold: the PURE TELEMETRY kinds, whose readers derive aggregate+ * statistics read-side (summarize/list/roll-up surfaces) and keep no+ * per-record identity past the correlation window. The list is CLOSED on+ * purpose - folding is opt-in per kind, and a kind that is not listed+ * here fails closed to keep, including every kind a future build adds+ * before it makes its fold decision. The census test+ * (`tests/core/brain/receipts-scan.test.ts`) fails when a new continuity+ * kind lands in the vocabulary without joining one of the two lists+ * below.*/-export const PROTECTED_CONTINUITY_KINDS: ReadonlyArray<ContinuityRecordKind> = Object.freeze([+export const FOLDABLE_CONTINUITY_KINDS: ReadonlyArray<ContinuityRecordKind> = Object.freeze([+ "recall_telemetry",+ "gate_telemetry",+ "ingest_dedup",+ "mcp_route_latency",+ "token_impact",+ "token_impact_outcome",+ "context_pack_outcome",+ "context_pack_evidence",+ "skill_invoked",+ "decision_model_call",+ "decision_model_extract_commit",+]);++/**+ * The continuity kinds the fold must NEVER touch, whatever their age -+ * every kind whose records are read back as data rather than aggregated+ * as telemetry, each named beside the reader that would lose it:+ *+ * - `source_invalidation` - the deleteBySource audit trail;+ * - `pre_compact_extract` / `post_compact_audit` - the Hermes+ * context-compaction receipts, which a storage fold must not rewrite+ * under a colliding name;+ * - `context_receipt` - looked up by id (`getContextReceipt`) and read+ * by the stale-dependency check;+ * - `session_turn` / `recent_turn` / `session_summary_node` /+ * `session_summary_digest` - the session read models; the+ * `recent_turn` sequence derives from the row count itself, so a+ * fold would restart it;+ * - `generation_report` - listed, filtered and joined as a read model;+ * - `host_memory_write` - the durable record of a host memory write;+ * - `session_entity_mention` - the cross-session entity index the+ * session-recall seed, the hallways pass and the mining dedupe seed+ * read back (its payload carries `session_ref`, so the clip-key rule+ * never protected it);+ * - `recall_observed_use` - read back as data, never aggregated: the+ * lifetime observed-reuse rate the search ranking consumes unfiltered+ * (`observedReuseRates`, folded per artifact by the candidate-signals+ * collector into every ranking that runs) and the dead-weight+ * report's per-source roll-up both read the rows; a fold past the+ * cutoff would replace old verdicts with an aggregate line every+ * reader drops, silently decaying a live ranking input;+ * - `negative_recall` - the answer-audit trail its own vocabulary entry+ * declares: the record exists so the corpus statement a "no" rested+ * on stays auditable after the answer has been given, which is+ * per-record identity past any correlation window; a fold would+ * destroy exactly that auditability;+ * - `receipt_compaction` - the compaction audit trail itself; a fold+ * that removed the record of the run before it would erase the only+ * account of what the fold has already destroyed.+ */+export const KEPT_CONTINUITY_KINDS: ReadonlyArray<ContinuityRecordKind> = Object.freeze(["source_invalidation","pre_compact_extract","post_compact_audit",+ "context_receipt",+ "session_turn",+ "recent_turn",+ "session_summary_node",+ "session_summary_digest",+ "generation_report",+ "host_memory_write",+ "session_entity_mention",+ "recall_observed_use",+ "negative_recall",+ "receipt_compaction",]);/** One shard selected for compaction, as a scan reported it. */diff --git a/src/core/brain/receipts/views.ts b/src/core/brain/receipts/views.tsindex acc09820..0b11d55c 100644--- a/src/core/brain/receipts/views.ts+++ b/src/core/brain/receipts/views.ts@@ -85,6 +85,11 @@ export function compactView(result: ReceiptCompactResult): Record<string, unknow},...(result.snapshotRunId !== undefined ? { snapshot_run_id: result.snapshotRunId } : {}),...(result.recoverability !== undefined ? { recoverability: result.recoverability } : {}),+ // The run's net cost (M6): the ledger shrink beside the recovery+ // point's own bytes - a whole-Brain archive in a counted surface -+ // so the caller can see growth the shrink did not pay for.+ ...(result.snapshotBytes !== undefined ? { snapshot_bytes: result.snapshotBytes } : {}),+ ...(result.netDeltaBytes !== undefined ? { net_delta_bytes: result.netDeltaBytes } : {}),...(result.auditRecordId !== undefined ? { audit_record_id: result.auditRecordId } : {}),...(result.auditError !== undefined ? { audit_error: result.auditError } : {}),};diff --git a/src/core/brain/session-mining-ledger.ts b/src/core/brain/session-mining-ledger.tsnew file mode 100644index 00000000..1e56b9df--- /dev/null+++ b/src/core/brain/session-mining-ledger.ts@@ -0,0 +1,345 @@+/**+ * The shared per-session mining record (session-mining-economy,+ * t_93617709).+ *+ * `planExtractSignals` is a pure read with no cross-run memory, so a+ * session that was mined once is rebuilt and re-offered to the calling+ * agent's model on every re-scan - repeated model spend, whether the mine+ * would yield anything or not. This ledger is the memory: every commit that+ * answers a plan records its outcome breakdown for the session, and the+ * plan gate consults the record before any decision-model work. A zero-+ * yield record (nothing written, nothing staged) is a tombstone that+ * suppresses the session by name; a yield record is the distilled state the+ * mining census reports, and suppresses nothing.+ *+ * ## The entry is keyed on content the mining lane actually holds+ *+ * The session import ledger (`sessions/discover.ts`) hashes the BYTES of a+ * source log and keys the hash on the log's absolute machine path. The+ * mining lane holds neither: `session_turn` continuity records carry no+ * source path and no hash, and the lane's session id is the portable+ * `sessionRefIdentity` basename, not a path. So the tombstone hashes the+ * mined turn material itself - exactly the (turn id, text, timestamp)+ * triples the plan embeds - under the portable id. The invalidation+ * semantics fall out for free: a re-import that changes a session's turns+ * changes the hash and re-opens the session with no clearing step, and a+ * session whose turns vanish leaves nothing for the gate to honor.+ *+ * ## The house ledger discipline, mirrored+ *+ * The shape is the import ledger's: schema-versioned, read fail-closed on+ * corrupt JSON or an unknown schema version (a silent reset would unsuppress+ * every barren session at once and send them all back to the model), written+ * under the same file lock with the vault-identity assertion, serialized+ * with sorted keys so a no-op rerun is byte-identical, and pruned of dead+ * entries at write time only, because a read that rewrote the state it read+ * would not be a read. The pruning here drops sessions with no imported+ * turns rather than files that no longer exist - the mining lane's+ * equivalent of a vanished path.+ *+ * The ledger is deterministic local state: no model, no network, no config+ * knob. Like the import ledger it is always on, because the spend it+ * prevents is the point of the feature.+ */++import { existsSync, readFileSync } from "node:fs";+import { join } from "node:path";++import { atomicWriteFileSync } from "../fs-atomic.ts";+import { listContinuityRecords } from "./continuity/store.ts";+import { hashBytes } from "./ingest/content-manifest.ts";+import { DERIVED_STORE_DIR } from "./path-constants.ts";+import { listSessionRawTurns, sessionIdOf } from "./session-recall.ts";+import { acquireLockSyncWithRetry, ingestLockRemedy } from "./sync-lockfile.ts";+import { isoSecond } from "./time.ts";+import { assertVaultIdentityForWrite } from "./vault-identity.ts";++/** Only ledger schema this build understands. Anything else is refused. */+export const SESSION_MINING_LEDGER_SCHEMA_VERSION = 1 as const;++/** Basename of the mining ledger inside the derived store. */+const SESSION_MINING_LEDGER_FILE = "session-mining-ledger.json";++/** Absolute path of the session mining ledger for a vault. */+export function sessionMiningLedgerPath(vault: string): string {+ return join(vault, DERIVED_STORE_DIR, SESSION_MINING_LEDGER_FILE);+}++/** How the mined session ended, in the commit result's own vocabulary. */+export interface SessionMiningOutcome {+ /** Signals written straight to the inbox. */+ readonly written: number;+ /** Of those, the ones staged into `Brain/pending/` for approval. */+ readonly staged: number;+ /** Items refused because their dedup hash was already in the vault. */+ readonly deduped: number;+ /** Items the durability gate refused. */+ readonly durabilityRejected: number;+}++/**+ * The turn material the plan embeds, as far as the hash is concerned:+ * the turn's id, its trimmed text and its stored timestamp verbatim.+ * Structurally the plan's `MinedTurn`, kept separate so this module does+ * not import the mining lane that records into it.+ */+export interface MinedTurnMaterial {+ readonly turnId: string;+ readonly text: string;+ readonly timestamp: string;+}++/** What was recorded about one mined session. */+export interface SessionMiningEntry {+ /**+ * The portable session id. Carried inside the entry as well as under its+ * key so a record is self-describing: a census or an operator reading the+ * file needs no key context to attribute it, and an entry misfiled under+ * another session's key is unreadable rather than misattributed.+ */+ readonly sessionId: string;+ /** {@link minedTurnsHash} of the turn material the mine covered. */+ readonly minedContentHash: string;+ /**+ * The outcome breakdown the last commit over this material recorded.+ * Zero written and zero staged makes the entry a tombstone (the plan+ * gate's evidence); a yield makes it the distilled state the census+ * reports - the same field, read through two contracts.+ */+ readonly outcome: SessionMiningOutcome;+ /** When the commit recorded it, to the second. */+ readonly recordedAt: string;+}++/** Portable session id → what was recorded about mining it. */+export interface SessionMiningLedger {+ readonly schema_version: typeof SESSION_MINING_LEDGER_SCHEMA_VERSION;+ readonly entries: Readonly<Record<string, SessionMiningEntry>>;+}++/**+ * Read the ledger. A missing file is an empty ledger - no session is+ * tombstoned, which is the correct answer for a vault that has mined+ * nothing. A CORRUPT or unknown-schema file is a hard error: a silent reset+ * would unsuppress every barren session at once and re-offer all of them to+ * the model on the next scan. The entries container is held to the same+ * contract: a container that is absent, null, an array or a primitive is+ * file-level corruption and refuses by name, while damage to a SINGLE entry+ * drops that entry only.+ */+export function readSessionMiningLedger(vault: string): SessionMiningLedger {+ const path = sessionMiningLedgerPath(vault);+ if (!existsSync(path)) {+ return { schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION, entries: {} };+ }+ let parsed: unknown;+ try {+ parsed = JSON.parse(readFileSync(path, "utf8"));+ } catch (e) {+ throw new Error(`session mining ledger is corrupted JSON: ${path}`, { cause: e });+ }+ if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) {+ throw new Error(`session mining ledger is not an object: ${path}`);+ }+ const obj = parsed as Record<string, unknown>;+ if (obj["schema_version"] !== SESSION_MINING_LEDGER_SCHEMA_VERSION) {+ throw new Error(+ `session mining ledger schema_version ${String(obj["schema_version"])} not supported ` ++ `(expected ${SESSION_MINING_LEDGER_SCHEMA_VERSION}): ${path}`,+ );+ }+ const entries: Record<string, SessionMiningEntry> = {};+ const raw = obj["entries"];+ // Container shape is file-level corruption, the same class as a corrupt+ // body or an unknown schema: a string, array, null or absent container+ // read as an empty ledger would silently unsuppress every barren session+ // at once, which is the outcome the hard errors exist to prevent. This is+ // different from per-ENTRY damage, which is dropped rather than guessed+ // at, because a dropped entry can only re-open one session for mining.+ if (raw === null || typeof raw !== "object" || Array.isArray(raw)) {+ throw new Error(`session mining ledger entries is not an object: ${path}`);+ }+ for (const [key, value] of Object.entries(raw as Record<string, unknown>)) {+ const entry = readEntry(key, value);+ if (entry !== null) entries[key] = entry;+ }+ return { schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION, entries };+}++/**+ * One entry, or `null` when it is not one this build can read. An+ * unreadable entry is dropped rather than guessed at: treated as absent it+ * can only re-open a session for mining (the safe direction), while+ * suppressing a session on a half-readable record would hide material+ * behind corrupted state. An entry filed under a DIFFERENT session's key is+ * unreadable too - attributing it to the key it sits under would suppress+ * one session on another session's evidence.+ */+function readEntry(key: string, value: unknown): SessionMiningEntry | null {+ if (value === null || typeof value !== "object" || Array.isArray(value)) return null;+ const row = value as Record<string, unknown>;+ const sessionId = row["sessionId"];+ const hash = row["minedContentHash"];+ const recordedAt = row["recordedAt"];+ if (typeof sessionId !== "string" || sessionId.length === 0 || sessionId !== key) return null;+ if (typeof hash !== "string" || hash.length === 0) return null;+ const outcome = readOutcome(row["outcome"]);+ if (outcome === null) return null;+ return {+ sessionId,+ minedContentHash: hash,+ outcome,+ recordedAt: typeof recordedAt === "string" ? recordedAt : "",+ };+}++function readOutcome(value: unknown): SessionMiningOutcome | null {+ if (value === null || typeof value !== "object" || Array.isArray(value)) return null;+ const row = value as Record<string, unknown>;+ const written = row["written"];+ const staged = row["staged"];+ const deduped = row["deduped"];+ const durabilityRejected = row["durabilityRejected"];+ if (+ typeof written !== "number" ||+ typeof staged !== "number" ||+ typeof deduped !== "number" ||+ typeof durabilityRejected !== "number"+ ) {+ return null;+ }+ return { written, staged, deduped, durabilityRejected };+}++/** Canonical ledger bytes: keys sorted, so a no-op rerun is byte-identical. */+function serializeLedger(entries: Record<string, SessionMiningEntry>): string {+ const sorted: Record<string, SessionMiningEntry> = {};+ for (const key of Object.keys(entries).toSorted()) sorted[key] = entries[key]!;+ return (+ JSON.stringify(+ { schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION, entries: sorted },+ null,+ 2,+ ) + "\n"+ );+}++/** One commit's mining outcome to record. */+export interface SessionMiningRecord {+ readonly sessionId: string;+ /** {@link minedTurnsHash} of the turn material the commit covered. */+ readonly minedContentHash: string;+ readonly outcome: SessionMiningOutcome;+}++/**+ * Record one session's mining outcome, keyed on its portable id. Returns+ * `true` when the ledger was rewritten, `false` on a byte-identical no-op.+ *+ * The read, the merge and the write are one critical section under the same+ * lock every other Brain read-modify-write takes: two mining commits for+ * different sessions can be running in two processes at once, and two+ * snapshots written back in sequence would each lose the other's entry.+ *+ * Every OTHER entry whose session no longer has turns is dropped in the+ * same pass, and so is the recorded one should its own session have none.+ * A session the plan refuses (zero imported turns) has nothing for the gate+ * to suppress, so its entry has no left reason to exist - and dropping only+ * never-mined-again sessions would have meant one pass that hears a+ * session's turn count and still leaves a dead key behind. The pruning+ * happens here and not in the read on purpose: the read stays a read. The+ * cost is one continuity-store pass per record building the set of sessions+ * that hold turns directly, plus a full turn listing only for an entry+ * whose session holds none of them - the rare root-ref-mined-over-segments+ * case, where the turns are reachable only through the lineage scope the+ * plan itself reads.+ */+export function recordSessionMiningOutcome(+ vault: string,+ record: SessionMiningRecord,+ now: Date = new Date(),+): boolean {+ assertVaultIdentityForWrite(vault);+ const path = sessionMiningLedgerPath(vault);+ const handle = acquireLockSyncWithRetry(path, undefined, { remedy: ingestLockRemedy });+ try {+ const entries: Record<string, SessionMiningEntry> = {+ ...readSessionMiningLedger(vault).entries,+ };+ entries[record.sessionId] = {+ sessionId: record.sessionId,+ minedContentHash: record.minedContentHash,+ outcome: { ...record.outcome },+ recordedAt: isoSecond(now),+ };+ // One store pass for the whole prune, mirroring the import ledger's+ // one-existsSync-per-entry idiom: a full turn listing per entry here+ // would be a whole-store read plus a lineage build per entry, growing+ // with the ledger's history instead of with this record.+ const sessionsWithTurns = new Set<string>();+ for (const turnRecord of listContinuityRecords(vault)) {+ if (turnRecord.kind !== "session_turn") continue;+ const sid = sessionIdOf(turnRecord.payload);+ if (sid !== undefined) sessionsWithTurns.add(sid);+ }+ for (const key of Object.keys(entries)) {+ if (sessionsWithTurns.has(key)) continue;+ // A session holding no direct turn record may still have turns,+ // reachable through the lineage scope (a root ref mined over+ // segment-recorded turns); the exact read decides those, and only+ // those.+ if (listSessionRawTurns(vault, key).length === 0) delete entries[key];+ }+ const next = serializeLedger(entries);+ if (existsSync(path) && readFileSync(path, "utf8") === next) return false;+ atomicWriteFileSync(path, next);+ return true;+ } finally {+ handle.release();+ }+}++/**+ * The mined-content hash: a SHA-256 over the turn material in list order,+ * each turn as its (turn id, text, timestamp) triple. JSON is the canonical+ * encoding, so the field boundaries are unambiguous no matter what the turn+ * text contains. A turn that changes in any field, or a list that changes+ * order, changes the hash - which is exactly the invalidation the gate+ * wants: the hash covers what the plan would embed, so a matching hash+ * means the recorded outcome was earned by the turns the caller is about to+ * be offered again.+ */+export function minedTurnsHash(turns: ReadonlyArray<MinedTurnMaterial>): string {+ const canonical = turns.map((turn) => ({+ turnId: turn.turnId,+ text: turn.text,+ timestamp: turn.timestamp,+ }));+ return hashBytes(new TextEncoder().encode(JSON.stringify(canonical)));+}++/**+ * The plan gate's answer: the recorded TOMBSTONE whose hash still matches+ * `turns`, or `null`. `null` means no entry (never mined), a changed turn+ * list (re-mined material, re-opened), a session id that has no record - or+ * an entry whose recorded outcome carried a yield: a yielding commit writes+ * the distilled state the census reports, not a suppression, so the session+ * stays minable and only the zero-yield record suppresses. A hit means the+ * session's CURRENT material was mined to nothing accepted; the caller+ * suppresses with {@link SessionMiningEntry.outcome} as evidence.+ *+ * A corrupt or unknown-schema ledger throws, by the read's contract: a gate+ * that could not verify the ledger must not wave the session through as if+ * the ledger were empty.+ */+export function readMiningTombstone(+ vault: string,+ sessionId: string,+ turns: ReadonlyArray<MinedTurnMaterial>,+): SessionMiningEntry | null {+ const hash = minedTurnsHash(turns);+ const entry = readSessionMiningLedger(vault).entries[sessionId.trim()];+ if (entry === undefined || entry.minedContentHash !== hash) return null;+ if (entry.outcome.written + entry.outcome.staged > 0) return null;+ return entry;+}diff --git a/src/core/brain/sessions/mining-census.ts b/src/core/brain/sessions/mining-census.tsnew file mode 100644index 00000000..f365fe8d--- /dev/null+++ b/src/core/brain/sessions/mining-census.ts@@ -0,0 +1,229 @@+/**+ * The distilled and undistilled census over imported sessions+ * (session-mining-economy, t_0358af42).+ *+ * The import ledger (`sessions/discover.ts`) knows what was imported; the+ * mining ledger (`session-mining-ledger.ts`) knows what was mined and with+ * what yield. This module joins the two and classifies every imported+ * session, so "how much of my imported backlog was never distilled" is one+ * read, not a re-scan of every transcript on the machine:+ *+ * - `distilled` - mined with accepted yield (written or staged > 0) on+ * the session's CURRENT turn material;+ * - `known-nothing-new` - mined barren on the current material (the+ * tombstone: every item deduped or durability-rejected). This is a+ * different state from backlog - the session is known, it just had+ * nothing new to say;+ * - `undistilled` - never mined, or mined on material that has since+ * changed (a re-import changed the turns, which changes the mined+ * content hash and re-opens the session with no clearing step);+ * - `refused` - sessions the plan would refuse (capture boundary, zero+ * imported turns, zero user turns). They can never be distilled, so+ * they are excluded from the backlog: counting them would nag forever+ * about unparsable material.+ *+ * ## The join is the portable identity, and the reads are vault-side+ *+ * The import ledger's keys are absolute machine paths; the mining ledger is+ * keyed by the portable `sessionRefIdentity` basename, which is what the+ * mining lane and every written signal's `session_ref` already carry. The+ * census joins on that basename, so a session whose log moved to another+ * absolute path (a rotated store, another machine's layout) still joins.+ * No source log is opened or re-hashed here: the mined-material hash runs+ * over the continuity turns the mining lane itself reads, which is what+ * makes the census answerable inside a SessionStart hook's budget. The+ * census reads only - neither ledger is rewritten, and a report that+ * quietly rewrote the state it reports on would not be a report.+ */++import { basename } from "node:path";++import { selectMinedTurns } from "../extract-signals.ts";+import { buildCaptureBoundary, type CaptureBoundary } from "../capture-boundary.ts";+import {+ minedTurnsHash,+ readSessionMiningLedger,+ type MinedTurnMaterial,+ type SessionMiningEntry,+} from "../session-mining-ledger.ts";+import type { Safeguard } from "../safeguard.ts";+import { listSessionRawTurns } from "../session-recall.ts";+import { readSessionLedger } from "./discover.ts";+import type { SessionRuntimeId } from "../../runtime/host-facts.ts";++/** Where an imported session stands with respect to distillation. */+export type SessionDistillationState = "distilled" | "known-nothing-new" | "undistilled";++/** One runtime's backlog: how much of what it imported is still undistilled. */+export interface RuntimeMiningCensus {+ readonly runtime: SessionRuntimeId;+ /** Every session the import ledger records under this runtime. */+ readonly imported: number;+ readonly distilled: number;+ readonly knownNothingNew: number;+ readonly undistilled: number;+ /**+ * Sessions the mining plan would refuse (boundary, zero imported turns,+ * zero user turns). Excluded from the backlog by construction;+ * `imported = distilled + knownNothingNew + undistilled + refused`.+ */+ readonly refused: number;+ /** The undistilled sessions' portable ids, sorted. */+ readonly undistilledSessionIds: ReadonlyArray<string>;+}++/** The vault-wide distilled and undistilled census. */+export interface SessionMiningCensus {+ readonly imported: number;+ readonly distilled: number;+ readonly knownNothingNew: number;+ readonly undistilled: number;+ readonly refused: number;+ /** One row per runtime the import ledger records, sorted by runtime id. */+ readonly byRuntime: ReadonlyArray<RuntimeMiningCensus>;+}++export interface BuildMiningCensusOptions {+ /**+ * Cooperative deadline, checked once per imported session. The census is+ * vault-side and never hashes a source log, but a vault with tens of+ * thousands of imported sessions still spends one turn listing per+ * session; a caller with a host deadline (the SessionStart hook) passes+ * its ceiling here so the read stops in time.+ */+ readonly safeguard?: Safeguard;+}++/**+ * Build the census. A corrupt or unknown-schema IMPORT ledger or MINING+ * record refuses by name (each read's own contract): a census that could+ * not verify one of its two sources must not report a clean backlog as if+ * the state were empty.+ */+export function buildMiningCensus(+ vault: string,+ opts: BuildMiningCensusOptions = {},+): SessionMiningCensus {+ const importedEntries = readSessionLedger(vault).entries;+ const miningEntries = readSessionMiningLedger(vault).entries;+ const boundary = buildCaptureBoundary(vault);++ // One identity can sit under several absolute paths (the same basename+ // imported from two machines' layouts); its state is a fact about the+ // identity, so it is classified once and applied to every path entry.+ const stateByIdentity = new Map<string, SessionDistillationState | "refused">();+ const stateOf = (identity: string): SessionDistillationState | "refused" => {+ const cached = stateByIdentity.get(identity);+ if (cached !== undefined) return cached;+ const state = classifySession(vault, boundary, miningEntries, identity);+ stateByIdentity.set(identity, state);+ return state;+ };++ interface Row {+ imported: number;+ refused: number;+ states: Record<SessionDistillationState, number>;+ ids: Set<string>;+ }+ const rows = new Map<SessionRuntimeId, Row>();+ const rowOf = (runtime: SessionRuntimeId): Row => {+ const existing = rows.get(runtime);+ if (existing !== undefined) return existing;+ const fresh: Row = {+ imported: 0,+ refused: 0,+ states: { distilled: 0, "known-nothing-new": 0, undistilled: 0 },+ ids: new Set(),+ };+ rows.set(runtime, fresh);+ return fresh;+ };++ for (const [path, entry] of Object.entries(importedEntries)) {+ opts.safeguard?.checkpoint();+ // The join identity is the portable sessionRefIdentity basename - what+ // the mining lane keys its tombstone and every signal's session_ref on+ // when the import pinned no explicit id. The import ledger's entry+ // does not carry the RESOLVED identity (an explicit --recall-session-id+ // import files its turns, tombstone and session_ref under the recall+ // id, not the basename), so such an import's mining state is invisible+ // to this join until the resolved id is recorded in the import ledger+ // itself; classifying the basename as refused here stays factually+ // right - the plan really would refuse it.+ const identity = basename(path);+ const state = stateOf(identity);+ const row = rowOf(entry.runtime);+ row.imported += 1;+ if (state === "refused") {+ row.refused += 1;+ continue;+ }+ row.states[state] += 1;+ if (state === "undistilled") row.ids.add(identity);+ }++ const byRuntime: RuntimeMiningCensus[] = [...rows.keys()].toSorted().map((runtime) => {+ const row = rows.get(runtime)!;+ return {+ runtime,+ imported: row.imported,+ distilled: row.states.distilled,+ knownNothingNew: row.states["known-nothing-new"],+ undistilled: row.states.undistilled,+ refused: row.refused,+ undistilledSessionIds: [...row.ids].toSorted(),+ };+ });+ const total = (pick: (row: RuntimeMiningCensus) => number): number =>+ byRuntime.reduce((sum, row) => sum + pick(row), 0);++ return {+ imported: total((row) => row.imported),+ distilled: total((row) => row.distilled),+ knownNothingNew: total((row) => row.knownNothingNew),+ undistilled: total((row) => row.undistilled),+ refused: total((row) => row.refused),+ byRuntime,+ };+}++/**+ * The mined turn material the plan would embed for one session, or `null`+ * when the session is one the plan refuses (boundary, zero imported turns,+ * zero user turns). The selection is the plan's own shared+ * {@link selectMinedTurns} - user role, unsuppressed text, trimmed and+ * non-empty, the turn's stored timestamp verbatim - so this census hashes+ * exactly what the plan would mine and cannot drift from it. `boundary` is+ * injectable so a caller walking many sessions builds it once.+ */+export function minedTurnMaterial(+ vault: string,+ sessionId: string,+ boundary: CaptureBoundary = buildCaptureBoundary(vault),+): ReadonlyArray<MinedTurnMaterial> | null {+ if (boundary.sessionDecision(sessionId) !== "capture") return null;+ const turns = listSessionRawTurns(vault, sessionId);+ if (turns.length === 0) return null;+ const mined = selectMinedTurns(boundary, turns);+ return mined.length === 0 ? null : mined;+}++/**+ * Classify one session identity: the plan's own refusal order (boundary+ * decision, zero imported turns, zero user turns), then the mining join.+ */+function classifySession(+ vault: string,+ boundary: ReturnType<typeof buildCaptureBoundary>,+ miningEntries: Readonly<Record<string, SessionMiningEntry>>,+ identity: string,+): SessionDistillationState | "refused" {+ const material = minedTurnMaterial(vault, identity, boundary);+ if (material === null) return "refused";+ const entry = miningEntries[identity];+ if (entry === undefined || entry.minedContentHash !== minedTurnsHash(material)) {+ return "undistilled";+ }+ return entry.outcome.written + entry.outcome.staged > 0 ? "distilled" : "known-nothing-new";+}diff --git a/src/core/brain/source-cleanup.ts b/src/core/brain/source-cleanup.tsindex 08434a9c..a35e78e2 100644--- a/src/core/brain/source-cleanup.ts+++ b/src/core/brain/source-cleanup.ts@@ -794,6 +794,10 @@ export function deleteBySource(const deleted: string[] = [];const failed: SourceCleanupFailure[] = [];let manifestEntryRemoved = false;+ // Set when the manifest entry deliberately STAYED because an original's+ // removal failed (review L7) - the audit names the keep, and the result's+ // `manifestEntryRemoved` stays false.+ let manifestEntryKeptForFailure = false;/*** Attempt one removal, collecting a failure instead of throwing: one@@ -813,11 +817,14 @@ export function deleteBySource(// no record (idempotent). Factored so both the success path AND a partial// failure can persist the record of what was already removed.const writeInvalidationAudit = (): string | null => {- if (deleted.length === 0 && !manifestEntryRemoved) return null;+ if (deleted.length === 0 && !manifestEntryRemoved && !manifestEntryKeptForFailure) {+ return null;+ }const agent = opts.agent?.trim() ? opts.agent.trim() : "delete_by_source";const reason =`${agent}: removed ${deleted.length} derived/original entr${deleted.length === 1 ? "y" : "ies"}` +`${manifestEntryRemoved ? " and the ingest manifest entry" : ""}` ++ `${manifestEntryKeptForFailure ? " (ingest manifest entry kept: an original's removal failed)" : ""}` +`${includeOriginals ? " (originals included)" : ""}` +(failed.length > 0? `; failed to remove ${failed.length}: ${failed.map((f) => f.path).join(", ")}`@@ -842,7 +849,15 @@ export function deleteBySource(for (const original of originals) removeCollecting(original);}- if (manifestKey !== null && existsSync(manifestPath(vault))) {+ // An original whose removal FAILED keeps its manifest entry (review+ // L7): the source file is still on disk, so dropping the entry would+ // classify it `new` at the next batch plan and the planner would+ // rebuild the very derived pages this run just removed. The audit+ // names the keep.+ const originalRemovalFailed =+ includeOriginals && originals.some((o) => failed.some((f) => f.path === o));+ manifestEntryKeptForFailure = originalRemovalFailed && manifestKey !== null;+ if (manifestKey !== null && existsSync(manifestPath(vault)) && !originalRemovalFailed) {const manifest = readManifest(vault);const entries = { ...manifest.entries };if (entries[manifestKey] !== undefined) {diff --git a/src/core/brain/storage/forecast.ts b/src/core/brain/storage/forecast.tsindex 8211270b..6f97a999 100644--- a/src/core/brain/storage/forecast.ts+++ b/src/core/brain/storage/forecast.ts@@ -19,6 +19,19 @@* history, which is what keeps a gate honest on a vault that filled up* before it ever sampled twice.*+ * ## The trend+ *+ * The slope is DAY-based, because the thing it forecasts is growth per+ * day. Samples are bucketed to one point per UTC day (the day's last+ * sample wins), the fit needs {@link MIN_TREND_DAYS} distinct days+ * before it exists at all, and it runs over the trailing+ * {@link TREND_WINDOW_DAYS} days only, so old history - including+ * reclamation - stops steering the line after the window slides past.+ * Until the minimum exists there is no `daysToCap` figure and the tier+ * answers from the percent alone: two calls of the storage report a+ * minute apart, with one ordinary write between them, must never+ * extrapolate a burst into a full vault (the H2 failure this replaces).+ ** ## Tier boundaries** The boundaries are module-level constants and the comparison is@@ -58,9 +71,25 @@ export const STORAGE_WARN_DAYS_DEFAULT = 30;/** The critical boundary in days to the cap, at the measured trend. Inclusive. */export const STORAGE_CRITICAL_DAYS_DEFAULT = 7;-/** Stored samples needed before a linear trend exists. */+/** Stored samples needed before the newest stored point may stand in for a census. */const MIN_TREND_SAMPLES = 2;+/**+ * Distinct UTC days the trend fits over. Below this there is no slope at+ * all: two closely spaced samples around one ordinary write would+ * otherwise extrapolate that burst to a huge daily growth and drive a+ * vault a third full to `critical`. Three days is the shortest span that+ * describes a rate rather than a single jump.+ */+export const MIN_TREND_DAYS = 3;++/**+ * The fit runs over this many trailing days of samples, never the whole+ * history: a vault that shrank a season ago must not carry that slope+ * forward once the window has slid past it.+ */+export const TREND_WINDOW_DAYS = 30;+/** Milliseconds in one day; the unit the trend is reported in. */const MS_PER_DAY = 86_400_000;@@ -109,11 +138,16 @@ export interface StorageForecast {readonly capBytes: number | null;/** `measuredBytes / capBytes * 100`, or `null` with no cap or no reading. */readonly percentUsed: number | null;- /** Bytes per day over the stored samples, or `null` with no trend. */+ /**+ * Bytes per day over the trailing window of day-bucketed samples, or+ * `null` with no trend: fewer than three distinct days fits no slope.+ */readonly slopeBytesPerDay: number | null;/*** Days until the cap at the current trend, or `null` when there is no- * trend or the trend does not reach the cap.+ * trend (under three distinct days) or the trend does not reach the+ * cap. Null is why the percent-only tiers exist: a tier can never ride+ * a trend that was never fitted.*/readonly daysToCap: number | null;/** The surfaces the freshest census could not read, when `unmeasurable`. */@@ -150,27 +184,59 @@ interface TrendPoint {readonly bytes: number;}-/** Parse the `storage_usage` samples into points; torn records drop out. */+/**+ * The UTC day index of a fractional-day timestamp (days since the epoch,+ * floored). Two samples share a bucket exactly when they share a UTC+ * calendar day.+ */+function utcDay(day: number): number {+ return Math.floor(day);+}++/**+ * Parse the `storage_usage` samples into trend points: torn records and+ * PARTIAL CENSUSES drop out (a sample is trend evidence only when its+ * payload asserts `measured: true` - an incomplete census understates the+ * vault by exactly the surfaces it could not read, and the jump when they+ * become readable again would otherwise fit as growth), the rest are+ * bucketed to one point per UTC day - the day's LAST sample, so a burst+ * of intra-day readings is noise the fit never sees - and the result is+ * ordered oldest first.+ */function parseSamples(samples: ReadonlyArray<MetricRecord>): TrendPoint[] {- const points: TrendPoint[] = [];+ const byDay = new Map<number, TrendPoint>();for (const record of samples) {+ if (record.payload["measured"] !== true) continue;const t = Date.parse(record.run_at);const bytes = record.payload["total_bytes"];if (!Number.isFinite(t) || typeof bytes !== "number" || !Number.isFinite(bytes)) continue;- points.push({ day: t / MS_PER_DAY, bytes });+ const point: TrendPoint = { day: t / MS_PER_DAY, bytes };+ const kept = byDay.get(utcDay(point.day));+ // Later in the day wins; a tie keeps the record read last, which is+ // the one appended later.+ if (kept === undefined || point.day >= kept.day) byDay.set(utcDay(point.day), point);}- return points.toSorted((a, b) => a.day - b.day);+ return [...byDay.values()].toSorted((a, b) => a.day - b.day);}-/** Least-squares slope in bytes per day over the points. */+/**+ * Least-squares slope in bytes per day over the day-bucketed points, or+ * `null` when no day-based trend exists: fewer than+ * {@link MIN_TREND_DAYS} distinct days inside the trailing+ * {@link TREND_WINDOW_DAYS}-day window fits nothing, and the caller+ * tiers on percent only.+ */function linearSlope(points: ReadonlyArray<TrendPoint>): number | null {- if (points.length < MIN_TREND_SAMPLES) return null;- const n = points.length;- const meanDay = points.reduce((s, p) => s + p.day, 0) / n;- const meanBytes = points.reduce((s, p) => s + p.bytes, 0) / n;+ if (points.length < MIN_TREND_DAYS) return null;+ const newestDay = points[points.length - 1]!.day;+ const window = points.filter((p) => p.day >= newestDay - TREND_WINDOW_DAYS);+ if (window.length < MIN_TREND_DAYS) return null;+ const n = window.length;+ const meanDay = window.reduce((s, p) => s + p.day, 0) / n;+ const meanBytes = window.reduce((s, p) => s + p.bytes, 0) / n;let cov = 0;let varDay = 0;- for (const p of points) {+ for (const p of window) {const dx = p.day - meanDay;cov += dx * (p.bytes - meanBytes);varDay += dx * dx;diff --git a/src/core/brain/storage/gate.ts b/src/core/brain/storage/gate.tsindex d698aa04..b1a279e7 100644--- a/src/core/brain/storage/gate.ts+++ b/src/core/brain/storage/gate.ts@@ -12,15 +12,23 @@* that needed no fresh bytes is an unbounded one, and it does not* measure at all: no cap, no cost.*- * ## Never skip on a refusal+ * ## Two skip lines, two callers*- * An unmeasurable store leaves the gate OPEN with the refused surfaces- * named, exactly as the maintenance lane's host-pressure gate stays open- * when the machine cannot be read: refusing to work forever on a vault- * one directory cannot be listed in would be a worse failure than the- * gate not closing. Only a MEASURED critical tier skips.+ * By default only a measured `critical` tier skips - the posture the+ * interactive ingest wall keeps, where `critical` is the refusal line.+ * A caller may also pass `pausePercent` (`storage.pause_percent`, the+ * maintenance lane's knob): any measured percent at or past that line+ * then skips too, one band before the wall, so background production+ * backs off while interactive ingest still works. An unmeasurable store+ * leaves the gate OPEN with the refused surfaces named, exactly as the+ * maintenance lane's host-pressure gate stays open when the machine+ * cannot be read: refusing to work forever on a vault one directory+ * cannot be listed in would be a worse failure than the gate not+ * closing. Only a MEASURED breach skips.*/+import { BrainConfigError } from "../policy/errors.ts";+import { loadStorageConfigSafe } from "../policy/load.ts";import type { StorageForecastTier } from "./forecast.ts";import { STORAGE_FORECAST_TIER, forecastStorageUsage } from "./forecast.ts";import { measureStorageUsage, type StorageUsage } from "./usage.ts";@@ -58,11 +66,18 @@ export interface CapacityGateDecision {readonly reading?: CapacityGateReading;/*** The fresh census itself, when one was taken - present for- * `unmeasurable` too, because a partial census is still a census.- * The usage sampler records it instead of re-walking the vault (see+ * `unmeasurable` too, because a partial census is still a census. The+ * usage sampler records it instead of re-walking the vault (see* `sampleStorageUsage`), so a decision costs one walk and never two.*/readonly census?: StorageUsage;+ /**+ * The percent-of-cap line the caller asked background work to pause+ * at, when one was passed. Rides every decision it shaped - `run` as+ * much as `skip` - so a report can show how far the vault stands from+ * the pause line, not only the tier it measured.+ */+ readonly pausePercent?: number;}export interface EvaluateCapacityGateOptions {@@ -71,6 +86,13 @@ export interface EvaluateCapacityGateOptions {/** Threshold overrides; every one defaults to the forecast constant. */readonly warnPercent?: number;readonly criticalPercent?: number;+ /**+ * The percent-of-cap line background work pauses at. Absent, only the+ * critical tier skips (the interactive ingest wall's posture); the+ * maintenance lane passes the configured `storage.pause_percent`, so+ * background production backs off one band before the wall.+ */+ readonly pausePercent?: number;readonly warnDays?: number;readonly criticalDays?: number;/** Wall clock for the fresh census's stamp; defaults to the current time. */@@ -112,9 +134,18 @@ export function evaluateCapacityGate(...(opts.criticalDays !== undefined ? { criticalDays: opts.criticalDays } : {}),});+ // A measured critical tier skips for every caller. A percent at or+ // past the caller's pause line skips too, but only for the caller that+ // asked for the band: the ingest wall (no pausePercent) keeps its+ // critical-only posture.+ const pauseBreached =+ opts.pausePercent !== undefined &&+ forecast.percentUsed !== null &&+ forecast.percentUsed >= opts.pausePercent;+const decision: CapacityGateDecision = {decision:- forecast.tier === STORAGE_FORECAST_TIER.critical+ forecast.tier === STORAGE_FORECAST_TIER.critical || pauseBreached? CAPACITY_GATE_DECISION.skip: CAPACITY_GATE_DECISION.run,tier: forecast.tier,@@ -127,6 +158,75 @@ export function evaluateCapacityGate(measuredAt: fresh.measuredAt,},census: fresh,+ ...(opts.pausePercent !== undefined ? { pausePercent: opts.pausePercent } : {}),};return decision;}++export interface EvaluateIngestAdmissionOptions {+ /** Wall clock for the fresh census; defaults to the current time. */+ readonly now?: Date;+}++/** One admission evaluation: a gate decision, or why there is none. */+export interface IngestAdmission {+ /**+ * The gate decision, when a configured cap was measured. `null` means+ * no wall stands this pass - an unbounded vault, or a storage block+ * that could not be read (see {@link reason}).+ */+ readonly decision: CapacityGateDecision | null;+ /**+ * Why no decision exists, when the wall was skipped rather than+ * passed: a `_brain.yaml` that fails validation names the failing+ * field here. Callers that have a warning channel write it; a wall+ * that pretended to stand behind an unreadable config would refuse+ * every capture forever on a typo in an unrelated block.+ */+ readonly reason?: string;+}++/**+ * The write-admission check behind `brain_ingest_source`, resolved once+ * for every background route that ingests - the capture inbox drain's+ * source-reference route included.+ *+ * An unbounded vault admits (`null`: no cap, no gate, no byte walk). A+ * configured cap is measured fresh, and only a MEASURED critical tier is+ * the refusal - exactly the interactive wall's posture, no pause line:+ * a capture an agent could not ingest by hand must not be sneaked in by+ * a drain, and a capture the wall would admit must not be refused by+ * one.+ *+ * A config that cannot be VALIDATED is not a vault without room: the+ * check fails open with the reason carried (see {@link IngestAdmission}),+ * because an ingest wall is a convenience-surface gate, not the commit+ * path - the core ingest reads no config, and a malformed unrelated+ * block must not wedge the capture inbox shut.+ */+export function evaluateIngestAdmission(+ vault: string,+ opts: EvaluateIngestAdmissionOptions = {},+): IngestAdmission {+ let storage;+ try {+ storage = loadStorageConfigSafe(vault);+ } catch (err) {+ if (!(err instanceof BrainConfigError)) throw err;+ return {+ decision: null,+ reason: `storage admission not evaluated: ${err.message}`,+ };+ }+ if (storage.cap_bytes === null) return { decision: null };+ return {+ decision: evaluateCapacityGate(vault, {+ now: opts.now ?? new Date(),+ capBytes: storage.cap_bytes,+ warnPercent: storage.warn_percent,+ criticalPercent: storage.critical_percent,+ warnDays: storage.warn_days,+ criticalDays: storage.critical_days,+ }),+ };+}diff --git a/src/core/brain/storage/usage.ts b/src/core/brain/storage/usage.tsindex 2ce41532..31eb2895 100644--- a/src/core/brain/storage/usage.ts+++ b/src/core/brain/storage/usage.ts@@ -38,7 +38,17 @@* is the `brain_tree` surface, `vault-content`. The derived store* (`brain.sqlite` plus its WAL and SHM sidecars) is resolved through the* search layer's own resolver so the meter can never disagree with the- * indexer about which file is the store; it is `derived`.+ * indexer about which file is the store; it is `derived`. The rest of+ * the store's directory (`.open-second-brain/` - the hook audit, the+ * watchdog fallback audit, the dead letters, the checkpoints, the+ * session ledgers, the maintenance journal and lease database) is the+ * `derived_store_dir` surface: it accumulates for months outside any+ * `Brain/` lane, and a census that skipped it would read a busy+ * hook-driven vault as flat while the disk filled. It reports as+ * `vault-content` as a whole - most of it is regenerable, but the+ * secret custody, the dead letters and the lockfiles are not, and a+ * capacity report that overstated the deletable side would be the+ * dangerous direction to err in.** The meter is read-only, never follows symlinks, and takes no* wall-clock reading: the timestamp it stamps comes from the injected@@ -48,8 +58,8 @@import { readdirSync, statSync } from "node:fs";import { join } from "node:path";-import { BRAIN_ROOT_REL } from "../path-constants.ts";-import { resolveIndexPath } from "../../search/paths.ts";+import { BRAIN_ROOT_REL, DERIVED_STORE_DIR } from "../path-constants.ts";+import { resolveConfiguredIndexPath } from "../../search/paths.ts";/** Recoverability tiers, mirroring the state catalogue's vocabulary. */export const STORAGE_TIER = Object.freeze({@@ -81,6 +91,12 @@ export const STORAGE_SURFACE = Object.freeze({internalState: "internal_state",/** The derived store and its WAL/SHM sidecars. */derivedStore: "derived_store",+ /**+ * `<vault>/.open-second-brain/` minus the store files - the+ * accumulating machinery state the store's directory carries beside+ * the store.+ */+ derivedStoreDir: "derived_store_dir",} as const);export type StorageSurface = (typeof STORAGE_SURFACE)[keyof typeof STORAGE_SURFACE];@@ -94,6 +110,7 @@ export const STORAGE_SURFACES: ReadonlyArray<StorageSurface> = Object.freeze([STORAGE_SURFACE.payloads,STORAGE_SURFACE.internalState,STORAGE_SURFACE.derivedStore,+ STORAGE_SURFACE.derivedStoreDir,]);/** Why a surface has no byte reading. */@@ -143,10 +160,22 @@ export interface MeasureStorageUsageOptions {readonly now?: Date;/*** The derived store's file path, when the caller has already resolved- * it (the doctor's context carries one). Defaults to the search- * layer's own resolution for this vault.+ * it (the doctor's context carries one). Absent, the store resolves+ * through the search layer's CONFIGURED resolution - the override+ * (`search_db_path` / `OPEN_SECOND_BRAIN_SEARCH_DB`) included - so the+ * census measures the store the indexer actually opens, never a stale+ * file the override left behind at the default location.*/readonly storePath?: string;+ /**+ * The machine-config file the caller resolved the search config from,+ * when it did so through an explicit `--config` path. Used for the+ * configured resolution above, so a `search_db_path` override in THAT+ * file moves the measured store the same way it moves the indexer;+ * absent, the resolver reads its own default file, which is the same+ * file an override-unaware caller would read.+ */+ readonly configPath?: string;}/** The tier each surface reports, named once beside the surface list. */@@ -158,6 +187,7 @@ const SURFACE_TIERS: Readonly<Record<StorageSurface, StorageTier>> = Object.free[STORAGE_SURFACE.payloads]: STORAGE_TIER.vaultContent,[STORAGE_SURFACE.internalState]: STORAGE_TIER.vaultContent,[STORAGE_SURFACE.derivedStore]: STORAGE_TIER.derived,+ [STORAGE_SURFACE.derivedStoreDir]: STORAGE_TIER.vaultContent,});/** The sqlite sidecar suffixes that belong to the store surface. */@@ -197,12 +227,16 @@ function addFile(path: string, add: (bytes: number) => void, refuse: () => void)* read below the point of refusal are discarded with it: the caller* (`measureDirSurface`) records a refused surface in its refusal set* only, so no caller can read a partial total as the surface's size.+ * `skipFiles` names absolute paths counted by a surface of their own+ * (the store files inside the store's directory), so no byte is counted+ * twice.*/function walkTree(root: string,skipTopLevel: ReadonlyArray<string>,add: (bytes: number) => void,refuse: () => void,+ skipFiles: ReadonlyArray<string> = [],): void {let entries;try {@@ -214,11 +248,13 @@ function walkTree(for (const entry of entries) {if (entry.isDirectory()) {if (skipTopLevel.includes(entry.name)) continue;- walkTree(join(root, entry.name), [], add, refuse);+ walkTree(join(root, entry.name), [], add, refuse, skipFiles);continue;}if (!entry.isFile()) continue;- addFile(join(root, entry.name), add, refuse);+ const filePath = join(root, entry.name);+ if (skipFiles.includes(filePath)) continue;+ addFile(filePath, add, refuse);}}@@ -238,6 +274,7 @@ function measureDirSurface(skipTopLevel: ReadonlyArray<string>,readings: Map<StorageSurface, number>,refusals: Set<StorageSurface>,+ skipFiles: ReadonlyArray<string> = [],): void {let bytes = 0;let refused = refusals.has(surface);@@ -247,7 +284,7 @@ function measureDirSurface(const refuse = (): void => {refused = true;};- walkTree(root, skipTopLevel, add, refuse);+ walkTree(root, skipTopLevel, add, refuse, skipFiles);if (refused) refusals.add(surface);else readings.set(surface, bytes);}@@ -300,7 +337,23 @@ export function measureStorageUsage(for (const [dir, surface] of Object.entries(DOT_DIR_SURFACES)) {measureDirSurface(surface, join(brain, dir), [], readings, refusals);}- const storePath = opts.storePath ?? resolveIndexPath(vault, null);+ const storePath = opts.storePath ?? resolveConfiguredIndexPath(vault, opts.configPath);+ // The store's directory is a counted surface of its own, minus the+ // store files - they are the derived_store surface, and the skip list+ // is what keeps one byte from landing in two surfaces.+ const storeFiles = [+ storePath,+ `${storePath}${STORE_WAL_SUFFIX}`,+ `${storePath}${STORE_SHM_SUFFIX}`,+ ];+ measureDirSurface(+ STORAGE_SURFACE.derivedStoreDir,+ join(vault, DERIVED_STORE_DIR),+ [],+ readings,+ refusals,+ storeFiles,+ );measureStoreSurface(storePath, readings, refusals);const surfaces = STORAGE_SURFACES.filter((s) => readings.has(s)).map((s) => ({diff --git a/src/core/brain/trust/operator-summary.ts b/src/core/brain/trust/operator-summary.tsindex 94ac9417..f210ae6d 100644--- a/src/core/brain/trust/operator-summary.ts+++ b/src/core/brain/trust/operator-summary.ts@@ -25,7 +25,10 @@ import { toPosix } from "../../path-safety.ts";import { runDoctor, type RunDoctorResult } from "../doctor.ts";import type { DreamRunSummary } from "../dream.ts";-import { collectMaintenanceActions } from "../maintenance/collect.ts";+import {+ collectMaintenanceActions,+ type MaintenanceCollectSignal,+} from "../maintenance/collect.ts";import type { ActionItem } from "../maintenance/action-scorer.ts";import type { DoctorIssueNaming } from "../diagnostics.ts";import { brainDirs } from "../paths.ts";@@ -52,6 +55,17 @@ export interface OperatorSummary {readonly verification_delta: VerificationDeltaResult;readonly top_actions: ReadonlyArray<ActionItem>;readonly instruction_file_warnings: ReadonlyArray<InstructionFileCeilingWarning>;+ /**+ * The maintenance scans that refused while the actions were collected+ * (review M10), rendered as `{scan, message}` rows - the same shape the+ * digest and doctor rows carry. A thin `top_actions` list must be+ * readable as healthy rather than indistinguishable from a scan that+ * failed. Present only when at least one scan refused.+ */+ readonly collect_errors?: ReadonlyArray<{+ readonly scan: MaintenanceCollectSignal;+ readonly message: string;+ }>;}export interface DigestSummary {@@ -135,10 +149,12 @@ export function buildOperatorSummary(maxLines: guardrails.instruction_file_max_lines,...(opts.readable !== undefined ? { readable: opts.readable } : {}),});+ const collectErrors: { scan: MaintenanceCollectSignal; message: string }[] = [];const topActions = safeTopActions(vault,opts.topActionsN ?? DEFAULT_TOP_ACTIONS_N,opts.keepAction,+ collectErrors,);const trust = computeTrustVerdict({@@ -161,6 +177,7 @@ export function buildOperatorSummary(verification_delta: verification,top_actions: topActions,instruction_file_warnings: instructionWarnings,+ ...(collectErrors.length > 0 ? { collect_errors: Object.freeze(collectErrors) } : {}),});}@@ -206,6 +223,15 @@ export function renderOperatorSummaryMarkdown(summary: OperatorSummary): string}}+ if (summary.collect_errors !== undefined && summary.collect_errors.length > 0) {+ out.push("");+ out.push("## Collection errors");+ out.push("");+ for (const e of summary.collect_errors) {+ out.push(`- ${e.scan}: ${e.message}`);+ }+ }+out.push("");return out.join("\n");}@@ -319,9 +345,12 @@ function safeTopActions(vault: string,topN: number,keepAction: BuildOperatorSummaryOptions["keepAction"],+ collectErrors: { scan: MaintenanceCollectSignal; message: string }[],): ReadonlyArray<ActionItem> {try {- const all = collectMaintenanceActions(vault);+ const all = collectMaintenanceActions(vault, {+ onError: (e) => collectErrors.push({ scan: e.signal, message: e.message }),+ });const kept = keepAction === undefined ? all : all.filter(keepAction);return Object.freeze(kept.slice(0, Math.max(0, topN)));} catch {diff --git a/src/core/brain/trust/retrieval-gate.ts b/src/core/brain/trust/retrieval-gate.tsindex db1bd277..16904791 100644--- a/src/core/brain/trust/retrieval-gate.ts+++ b/src/core/brain/trust/retrieval-gate.ts@@ -19,18 +19,36 @@* marker ({@link ENTITY_CONTAMINATION_FRONTMATTER_KEY}).** A quarantined candidate is never silently dropped: kernel 1 records it- * with these reasons into the retrieval_decision_trace receipt.+ * with these reasons into the retrieval_decision_trace receipt, and the+ * relational arm's edge-level classification - which decides before the+ * pool, where no pool exclusion could ever name its drops - records its+ * arrivals on the arm lane of the same receipts, reach-filtered to pages+ * the caller could read.+ *+ * The gate runs under a resolved tri-state mode+ * ({@link SEARCH_TRUST_GATE_MODE}, session-mining-economy t_05a0eaa4):+ * `enforce` excludes as described above, `shadow` runs the same+ * classification, excludes nothing and records every would-be refusal+ * with its structural reasons into the receipts, and `off` - the default+ * - does not run the gate at all, keeping ranking and the outcome shape+ * byte-identical. The legacy boolean enable flag maps `true` to `shadow`.*/import type { FrontmatterMap } from "../../types.ts";+import { envOrConfig, parseBool } from "../../validate.ts";+import { SearchError } from "../../search/search-error.ts";import {excludeVerdict,keepVerdict,type RankAdjuster,type RankAdjustVerdict,} from "../../search/rank-adjust.ts";-import type { BrainSearchResult } from "../../search/types.ts";+// The row type from the declared leaf, not the `types.ts` aggregate that+// re-exports it: the aggregate resolves this module's mode, so naming the+// aggregate back from here would close an import cycle.+import type { BrainSearchResult } from "../../search/search-result.ts";import { BRAIN_PREFERENCE_STATUS } from "../types.ts";+import { RETRIEVAL_TRUST_REFUSAL_LANE, type RetrievalTrustRefusal } from "./retrieval-receipts.ts";import { hasUntrustedSourceMarker } from "./untrusted-provenance.ts";import { hasEntityContaminationMarker } from "../truth/contamination.ts";@@ -47,6 +65,78 @@ export const RETRIEVAL_TRUST_EXCLUSION_REASON = Object.freeze({entityContamination: "entity_contamination",} as const);+/**+ * The resolved posture of the gate (session-mining-economy, t_05a0eaa4):+ * - `off` - the gate does not run; ranking and the outcome shape+ * stay byte-identical,+ * - `shadow` - the classification runs, nothing is excluded, and every+ * would-be refusal is recorded with its structural reasons+ * into the receipts,+ * - `enforce` - today's excluding behavior, made explicit.+ */+export const SEARCH_TRUST_GATE_MODE = Object.freeze({+ off: "off",+ shadow: "shadow",+ enforce: "enforce",+} as const);++export type SearchTrustGateMode =+ (typeof SEARCH_TRUST_GATE_MODE)[keyof typeof SEARCH_TRUST_GATE_MODE];++/** A mode under which the gate actually runs: shadow or enforce. */+export type ActiveSearchTrustGateMode = Exclude<SearchTrustGateMode, "off">;++/** True when `value` is one of the three mode tokens. */+export function isSearchTrustGateMode(value: string): value is SearchTrustGateMode {+ return (Object.values(SEARCH_TRUST_GATE_MODE) as string[]).includes(value);+}++/** Environment variable for the trust gate mode. */+export const SEARCH_TRUST_GATE_MODE_ENV = "OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE";+/** Machine-config key for the trust gate mode. */+export const SEARCH_TRUST_GATE_MODE_CONFIG = "search_trust_gate_mode";+/** Environment variable of the legacy boolean enable flag. */+export const SEARCH_TRUST_GATE_LEGACY_ENV = "OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE";+/** Machine-config key of the legacy boolean enable flag. */+export const SEARCH_TRUST_GATE_LEGACY_CONFIG = "search_trust_gate_enabled";++/**+ * Resolve the gate's mode: an explicit mode key wins (env over machine+ * config; an invalid value is a misconfiguration refused with a+ * `SearchError` naming the key actually in force - never silently+ * defaulted); absent the mode key, the legacy boolean maps `true` to+ * `shadow` and `false` to `off`; with neither set the default is `off`.+ * Shadow is non-destructive precisely so enablement can be the first+ * step: a vault that wants the excluding behavior names `enforce`.+ */+export function resolveSearchTrustGateMode(input: {+ readonly env?: NodeJS.ProcessEnv;+ readonly config?: Readonly<Record<string, string>>;+}): SearchTrustGateMode {+ const env = input.env ?? {};+ const config = input.config ?? {};+ const raw = envOrConfig(env, config, SEARCH_TRUST_GATE_MODE_ENV, SEARCH_TRUST_GATE_MODE_CONFIG);+ if (raw !== null) {+ if (isSearchTrustGateMode(raw)) return raw;+ const inForce = env[SEARCH_TRUST_GATE_MODE_ENV]+ ? SEARCH_TRUST_GATE_MODE_ENV+ : SEARCH_TRUST_GATE_MODE_CONFIG;+ throw new SearchError(+ "INVALID_INPUT",+ `${inForce} must be 'off', 'shadow' or 'enforce', got '${raw}'`,+ );+ }+ const legacy = envOrConfig(+ env,+ config,+ SEARCH_TRUST_GATE_LEGACY_ENV,+ SEARCH_TRUST_GATE_LEGACY_CONFIG,+ );+ return legacy !== null && parseBool(legacy, false, SEARCH_TRUST_GATE_LEGACY_CONFIG)+ ? SEARCH_TRUST_GATE_MODE.shadow+ : SEARCH_TRUST_GATE_MODE.off;+}+export interface RetrievalTrustVerdict {readonly quarantined: boolean;/** Structural reasons, sorted for a deterministic trace. */@@ -93,14 +183,57 @@ export function classifyRetrievalTrust(* its own. A quarantined candidate is excluded with the FIRST structural* reason; every reason still reaches the receipt via* {@link classifyRetrievalTrust} at trace-build time.+ *+ * Under `shadow` (session-mining-economy, t_05a0eaa4) the same+ * classification runs and every candidate is kept: the kernel's+ * single-pass verdict contract gains no kind, and the would-be refusals+ * are collected for the receipts by {@link collectTrustGateRefusals}+ * instead. `enforce` (the default) is the excluding behavior.*/-export function trustGateAdjuster(readFrontmatter: (path: string) => FrontmatterMap): RankAdjuster {+export function trustGateAdjuster(+ readFrontmatter: (path: string) => FrontmatterMap,+ mode: ActiveSearchTrustGateMode = "enforce",+): RankAdjuster {return {name: RETRIEVAL_TRUST_GATE_NAME,adjust(result: BrainSearchResult): RankAdjustVerdict {const verdict = classifyRetrievalTrust(readFrontmatter(result.path));if (!verdict.quarantined) return keepVerdict();+ if (mode === "shadow") return keepVerdict();return excludeVerdict(verdict.reasons[0]!);},};}++/**+ * Collect the pool lane's would-be refusals for the shadow mode: every+ * candidate the classification marks quarantined, as a compact reference+ * carrying ALL of its structural reasons (namespaced with the gate's+ * kernel name, matching how the kernel namespaces an exclusion), tagged+ * with the {@link RETRIEVAL_TRUST_REFUSAL_LANE.pool} lane. Nothing is+ * excluded here - the caller keeps ranking untouched and feeds the+ * returned record into the receipt build. Deterministic: pool order,+ * sorted reasons.+ */+export function collectTrustGateRefusals(+ ranked: ReadonlyArray<BrainSearchResult>,+ readFrontmatter: (path: string) => FrontmatterMap,+): RetrievalTrustRefusal[] {+ const refusals: RetrievalTrustRefusal[] = [];+ for (const candidate of ranked) {+ const verdict = classifyRetrievalTrust(readFrontmatter(candidate.path));+ if (!verdict.quarantined) continue;+ refusals.push(+ Object.freeze({+ document_id: candidate.documentId,+ chunk_id: candidate.chunkId,+ path: candidate.path,+ reasons: Object.freeze(+ verdict.reasons.map((reason) => `${RETRIEVAL_TRUST_GATE_NAME}:${reason}`),+ ),+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.pool,+ }),+ );+ }+ return refusals;+}diff --git a/src/core/brain/trust/retrieval-receipts.ts b/src/core/brain/trust/retrieval-receipts.tsindex 89d2c818..2c0bb617 100644--- a/src/core/brain/trust/retrieval-receipts.ts+++ b/src/core/brain/trust/retrieval-receipts.ts@@ -16,15 +16,84 @@** Both are pure builders over the kernel-1 outcome; snake_cased to match* the receipt payload convention.+ *+ * Shadow measurements (session-mining-economy, t_05a0eaa4): under the+ * gate's measure-only mode the receipts gain a refusal section beside the+ * exclusion counts - `excluded` keeps meaning actually dropped, and a+ * would-be refusal is counted with its structural reasons, reported per+ * lane (the rank-adjust pool and the relational arm separately) so+ * neither lane's numbers blur. Enforce receipts carry no refusal section+ * of their own for the pool lane - the pool's drops ARE the exclusions -+ * but the relational arm's drops happen BEFORE the pool, where no+ * exclusion can ever name them, so those ride the same refusal record:+ * the gate's never-silently-dropped posture holds on both lanes. Every+ * receipt an arm-less enforce run ever built stays byte-identical: the+ * enforce section is present only when the arm actually dropped+ * something - absent, never an empty placeholder.*/import type { RankAdjustExclusion } from "../../search/rank-adjust.ts";+/**+ * The lane a measured refusal came from: the kernel rank-adjust pool+ * (the page gate) or the relational arm's edge-level eligibility gate.+ * Lanes are reported separately and never folded together.+ */+export const RETRIEVAL_TRUST_REFUSAL_LANE = Object.freeze({+ pool: "pool",+ arm: "arm",+} as const);++export type RetrievalTrustRefusalLane =+ (typeof RETRIEVAL_TRUST_REFUSAL_LANE)[keyof typeof RETRIEVAL_TRUST_REFUSAL_LANE];++/**+ * One measured-only refusal as a compact reference (no body): a+ * candidate the eligibility classification marked ineligible that shadow+ * kept in the pool (or the arm kept in its contribution set).+ */+export interface RetrievalTrustRefusal {+ readonly document_id: number;+ readonly chunk_id: number;+ readonly path: string;+ /** Structural reason tokens, namespaced by their origin, sorted. */+ readonly reasons: ReadonlyArray<string>;+ readonly lane: RetrievalTrustRefusalLane;+}++/** The shadow-only section both receipt builders project. */+export interface RetrievalShadowRefusals {+ readonly refusals: ReadonlyArray<RetrievalTrustRefusal>;+}++/**+ * The enforce-mode section: the relational arm's drops. The arm removes an+ * ineligible node before the pool, so no pool exclusion can name it; these+ * refusal records are the drop's only receipt. Mutually exclusive with the+ * `shadow` section by construction - a run resolves exactly one mode.+ */+export interface RetrievalEnforcedRefusals {+ readonly refusals: ReadonlyArray<RetrievalTrustRefusal>;+}+export interface RetrievalReceiptInput {/** Count of candidates that survived the gate into the pack. */readonly surfaced: number;/** Candidates the gate excluded, with their namespaced reasons. */readonly excluded: ReadonlyArray<RankAdjustExclusion>;+ /**+ * Present only under the shadow mode: the would-be refusals measured+ * beside an untouched exclusion count. Absent - never an empty+ * placeholder - under enforce, whose receipts keep today's shape.+ */+ readonly shadow?: RetrievalShadowRefusals;+ /**+ * Present only under enforce, and only when the relational arm actually+ * dropped an arrival: the arm lane's refusal records, the drop's only+ * receipt. Absent otherwise, so every enforce receipt an arm-less run+ * ever built is byte-identical.+ */+ readonly enforce?: RetrievalEnforcedRefusals;}/** One excluded candidate as a compact reference (no body). */@@ -40,6 +109,18 @@ export interface RetrievalDecisionTrace {readonly surfaced: number;readonly excluded: number;readonly exclusions: ReadonlyArray<RetrievalDecisionTraceExclusion>;+ /** Present only under shadow: the mode the receipts were built under. */+ readonly mode?: "shadow";+ /**+ * Refusal records riding the trace: under shadow, every would-be+ * refusal across both lanes (pool then arm); under enforce, the+ * relational arm's drops - removed before the pool, where no exclusion+ * can name them. Present only when there are records - never an empty+ * placeholder.+ */+ readonly refused?: number;+ /** The refusal records themselves, same presence rule as `refused`. */+ readonly refusals?: ReadonlyArray<RetrievalTrustRefusal>;}export interface MemoryTrustAssessment {@@ -48,12 +129,81 @@ export interface MemoryTrustAssessment {readonly quarantined: number;/** Histogram of namespaced exclusion reasons across the excluded set. */readonly reason_counts: Readonly<Record<string, number>>;+ /** Present only under shadow: the mode the receipts were built under. */+ readonly mode?: "shadow";+ /**+ * Refusal aggregates for the records riding the receipts: under shadow,+ * every would-be refusal across both lanes; under enforce, the+ * relational arm's drops (the pool lane's drops are the exclusions+ * above, counted in `quarantined`). Present only when there are+ * records - never an empty placeholder.+ */+ readonly refused?: number;+ /**+ * Present with the refusal records: would-be refusal count per lane. A+ * lane with no refusals is present at 0 - a measured zero, not an+ * absence.+ */+ readonly refusals_by_lane?: Readonly<Record<string, number>>;+ /**+ * Present with the refusal records: the refusal reason histogram per+ * lane, keys sorted for a byte-stable serialization.+ */+ readonly refused_reason_counts?: Readonly<Record<string, Readonly<Record<string, number>>>>;+}++/**+ * Sort an object's keys into a new frozen record so a serialized+ * histogram is byte-stable for a given input.+ */+function sortedRecord<T>(input: Record<string, T>): Readonly<Record<string, T>> {+ const ordered: Record<string, T> = {};+ for (const key of Object.keys(input).toSorted()) ordered[key] = input[key]!;+ return Object.freeze(ordered);+}++/**+ * Build the per-lane refusal aggregates for the shadow section: the+ * total, the per-lane counts (every known lane present, 0 included) and+ * the per-lane reason histograms with sorted keys.+ */+function refusalAggregates(refusals: ReadonlyArray<RetrievalTrustRefusal>): {+ readonly refused: number;+ readonly refusals_by_lane: Readonly<Record<string, number>>;+ readonly refused_reason_counts: Readonly<Record<string, Readonly<Record<string, number>>>>;+} {+ const lanes: Record<string, number> = {};+ const histograms: Record<string, Record<string, number>> = {};+ for (const lane of Object.values(RETRIEVAL_TRUST_REFUSAL_LANE)) {+ lanes[lane] = 0;+ histograms[lane] = {};+ }+ for (const refusal of refusals) {+ lanes[refusal.lane] = (lanes[refusal.lane] ?? 0) + 1;+ const histogram = (histograms[refusal.lane] ??= {});+ for (const reason of refusal.reasons) {+ histogram[reason] = (histogram[reason] ?? 0) + 1;+ }+ }+ const orderedHistograms: Record<string, Readonly<Record<string, number>>> = {};+ for (const [lane, histogram] of Object.entries(histograms)) {+ orderedHistograms[lane] = sortedRecord(histogram);+ }+ return {+ refused: refusals.length,+ refusals_by_lane: sortedRecord(lanes),+ refused_reason_counts: sortedRecord(orderedHistograms),+ };}/*** Build the retrieval_decision_trace: every excluded candidate as a* compact reference (document / chunk id, path, reasons), plus the- * evaluated / surfaced / excluded counts.+ * evaluated / surfaced / excluded counts. Under shadow the would-be+ * refusals ride beside them with the mode named; under enforce the+ * relational arm's drops ride the same refusal record (no mode stamp -+ * an enforce receipt never names itself shadow), and a run where the arm+ * dropped nothing keeps the pre-mode shape exactly.*/export function buildRetrievalDecisionTrace(input: RetrievalReceiptInput): RetrievalDecisionTrace {const exclusions = input.excluded.map((e) =>@@ -69,12 +219,29 @@ export function buildRetrievalDecisionTrace(input: RetrievalReceiptInput): Retrisurfaced: input.surfaced,excluded: input.excluded.length,exclusions: Object.freeze(exclusions),+ ...(input.shadow === undefined+ ? {}+ : {+ mode: "shadow",+ refused: input.shadow.refusals.length,+ refusals: Object.freeze([...input.shadow.refusals]),+ }),+ ...(input.enforce === undefined+ ? {}+ : {+ refused: input.enforce.refusals.length,+ refusals: Object.freeze([...input.enforce.refusals]),+ }),});}/*** Build the memory_trust_assessment: the pack's trust posture as counts- * plus a deterministic histogram of the exclusion reasons.+ * plus a deterministic histogram of the exclusion reasons. Under shadow+ * the would-be refusals are aggregated per lane beside them - never+ * folded into `quarantined`, which keeps meaning actually dropped. Under+ * enforce the arm's drops aggregate the same way; a run where the arm+ * dropped nothing keeps the pre-mode shape exactly.*/export function buildMemoryTrustAssessment(input: RetrievalReceiptInput): MemoryTrustAssessment {const reasonCounts: Record<string, number> = {};@@ -83,13 +250,14 @@ export function buildMemoryTrustAssessment(input: RetrievalReceiptInput): MemoryreasonCounts[reason] = (reasonCounts[reason] ?? 0) + 1;}}- // Sort keys so the serialized histogram is byte-stable for a given input.- const ordered: Record<string, number> = {};- for (const key of Object.keys(reasonCounts).toSorted()) ordered[key] = reasonCounts[key]!;return Object.freeze({evaluated: input.surfaced + input.excluded.length,surfaced: input.surfaced,quarantined: input.excluded.length,- reason_counts: Object.freeze(ordered),+ reason_counts: sortedRecord(reasonCounts),+ ...(input.shadow === undefined+ ? {}+ : { mode: "shadow", ...refusalAggregates(input.shadow.refusals) }),+ ...(input.enforce === undefined ? {} : refusalAggregates(input.enforce.refusals)),});}diff --git a/src/core/brain/types.ts b/src/core/brain/types.tsindex 9f68a2ce..f0024a3a 100644--- a/src/core/brain/types.ts+++ b/src/core/brain/types.ts@@ -1639,6 +1639,13 @@ export interface BrainStorageConfig {readonly warn_percent?: number;/** Integer percent of the cap where the forecast goes critical. */readonly critical_percent?: number;+ /**+ * Integer percent of the cap where background production pauses (the+ * maintenance lane skips). Absent = the warn band: the lane pauses+ * where the forecast warns. Interactive ingest is never paused by+ * this - only a critical tier refuses a write.+ */+ readonly pause_percent?: number;/** Days to the cap at trend where the forecast warns. */readonly warn_days?: number;/** Days to the cap at trend where the forecast goes critical. */@@ -1651,6 +1658,13 @@ export interface ResolvedBrainStorageConfig {readonly cap_bytes: number | null;readonly warn_percent: number;readonly critical_percent: number;+ /**+ * The percent-of-cap line background work pauses at. Defaults to the+ * warn band, and moves with `warn_percent` when that block key is set+ * and `pause_percent` is not - the lane backs off where the forecast+ * warns unless the operator names the line themselves.+ */+ readonly pause_percent: number;readonly warn_days: number;readonly critical_days: number;}diff --git a/src/core/config.ts b/src/core/config.tsindex 810f1bd1..3441e209 100644--- a/src/core/config.ts+++ b/src/core/config.ts@@ -1711,6 +1711,21 @@ export function resolveGapLoopEnabled(configPath?: string): boolean {return resolveConfigFlag("OPEN_SECOND_BRAIN_GAP_LOOP_ENABLED", "gap_loop_enabled", configPath);}+/**+ * Session-start backlog nudge gate (session-mining-economy, t_0358af42).+ * Default OFF: the SessionStart hook that reports the undistilled-session+ * count stays dormant unless `backlog_nudge_enabled: "true"` (or the+ * matching env override). Flag off keeps the session preamble+ * byte-identical and writes no throttle state.+ */+export function resolveBacklogNudgeEnabled(configPath?: string): boolean {+ return resolveConfigFlag(+ "OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED",+ "backlog_nudge_enabled",+ configPath,+ );+}+/*** Optional gap-loop recurrence threshold override (A3). A valid positive* integer (env first, then config) wins; anything else returns undefined sodiff --git a/src/core/install/grok-asset.ts b/src/core/install/grok-asset.tsindex 429d5804..65203a53 100644--- a/src/core/install/grok-asset.ts+++ b/src/core/install/grok-asset.ts@@ -94,7 +94,19 @@ interface HookGroupSpec {const HOOK_SPEC: ReadonlyArray<{ event: string; groups: ReadonlyArray<HookGroupSpec> }> = [{event: "SessionStart",- groups: [{ hooks: ["active-inject", "gap-agenda", "session-capture"] }],+ groups: [+ {+ hooks: [+ "active-inject",+ "gap-agenda",+ // The opt-in undistilled-backlog nudge (session-mining-economy),+ // in the position the plugin's own SessionStart group lists it:+ // default-off like gap-agenda, fail-open, one context line.+ "backlog-nudge",+ "session-capture",+ ],+ },+ ],},{event: "UserPromptSubmit",diff --git a/src/core/search/explain-envelope.ts b/src/core/search/explain-envelope.tsindex d77d1e1d..09d17332 100644--- a/src/core/search/explain-envelope.ts+++ b/src/core/search/explain-envelope.ts@@ -33,14 +33,20 @@ export const TRUST_GATE_CONFIG_KEY = "search_trust_gate_enabled";/** Environment variable equivalent of {@link TRUST_GATE_CONFIG_KEY}. */export const TRUST_GATE_ENV_KEY = "OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE";+/** Config key that names the gate's mode (off | shadow | enforce). */+export const TRUST_GATE_MODE_CONFIG_KEY = "search_trust_gate_mode";+/*** Why an explain request carries no trace, and the exit that produces* one. The receipts are a by-product of the retrieval trust gate: with- * the gate off there is no evaluated / excluded set to report.+ * the gate off there is no evaluated / excluded set to report. The mode+ * key is named beside the legacy boolean: naming a mode (shadow first -+ * it measures without excluding) is the other enablement path.*/export const RETRIEVAL_TRACE_UNAVAILABLE =`no retrieval decision trace: the retrieval trust gate is off. ` +- `Enable it with ${TRUST_GATE_CONFIG_KEY}: true in the config, or ${TRUST_GATE_ENV_KEY}=1.`;+ `Enable it with ${TRUST_GATE_CONFIG_KEY}: true in the config, ${TRUST_GATE_ENV_KEY}=1, ` ++ `or name a mode with ${TRUST_GATE_MODE_CONFIG_KEY}: shadow.`;/*** The cross-vault union's own reason. A union outcome is merged from one@@ -64,7 +70,8 @@ export const RETRIEVAL_TRACE_NOT_MERGED =export const RETRIEVAL_TRACE_UNAVAILABLE_AND_NOT_MERGED =`no retrieval decision trace: the retrieval trust gate is off, and a cross-vault union ` +`would not merge the per-origin receipts even with it on. ` +- `Enable ${TRUST_GATE_CONFIG_KEY}: true (or ${TRUST_GATE_ENV_KEY}=1) and query a single vault.`;+ `Enable ${TRUST_GATE_CONFIG_KEY}: true (or ${TRUST_GATE_ENV_KEY}=1, or ` ++ `${TRUST_GATE_MODE_CONFIG_KEY}: shadow) and query a single vault.`;/** Response key carrying the retrieval decision trace. */export const RETRIEVAL_DECISION_TRACE_KEY = "retrieval_decision_trace";@@ -72,6 +79,14 @@ export const RETRIEVAL_DECISION_TRACE_KEY = "retrieval_decision_trace";/** Response key carrying the memory trust assessment. */export const MEMORY_TRUST_ASSESSMENT_KEY = "memory_trust_assessment";+/**+ * Response key carrying the gate's resolved mode. Present only when the+ * receipts name one - the shadow mode stamps its receipts, whose+ * serialization is how the envelope names the mode; enforce keeps+ * today's envelope exactly, and off has no receipts to name anything.+ */+export const RETRIEVAL_TRUST_GATE_MODE_KEY = "retrieval_trust_gate_mode";+/** Response key carrying the reason there is no trace to show. */export const RETRIEVAL_TRACE_UNAVAILABLE_KEY = "retrieval_trace_unavailable";@@ -92,10 +107,14 @@ export interface ExplainRequest {/** True when this outcome came from a cross-vault union. */readonly crossVault: boolean;/**- * The resolved `recall.retrievalTrustGateEnabled`. Neither this nor- * `crossVault` is derivable from a receipt-less outcome, and naming the- * wrong one of the two sends the operator to an exit that produces- * nothing - so the caller, which knows both, states both.+ * True when the retrieval trust gate is engaged in the caller's own+ * resolved config: its resolved mode is not off. The legacy boolean+ * alone cannot answer that - an explicit `search_trust_gate_mode` runs+ * the gate with the boolean left false - so the caller derives this+ * from the resolved mode's presence. Neither this nor `crossVault` is+ * derivable from a receipt-less outcome, and naming the wrong one of+ * the two sends the operator to an exit that produces nothing - so the+ * caller, which knows both, states both.*/readonly trustGateEnabled: boolean;}@@ -150,6 +169,9 @@ export function explainEnvelope(: {[RETRIEVAL_DECISION_TRACE_KEY]: trace,[MEMORY_TRUST_ASSESSMENT_KEY]: assessment,+ // The resolved mode, named only when the receipts carry one+ // (shadow stamps them; enforce and off name nothing).+ ...(trace.mode !== undefined ? { [RETRIEVAL_TRUST_GATE_MODE_KEY]: trace.mode } : {}),};if (timing === undefined) return receipts;return {diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex 27dbdf1e..c0e013d1 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -30,6 +30,10 @@ import {} from "./embeddings/credential-report.ts";import { loadRerankRegistry, expandRegisteredRerankProvider } from "./rerank/registry.ts";import { decisionModelModeFor, resolveDecisionModelConfig } from "../decision-model/config.ts";+import {+ resolveSearchTrustGateMode,+ SEARCH_TRUST_GATE_MODE,+} from "../brain/trust/retrieval-gate.ts";import { INPUT_WINDOW_TOKENS_KEY, resolveEmbeddingPrefixes } from "./embeddings/presets.ts";import {EMBEDDING_PRICE_MODEL_ENV,@@ -1314,6 +1318,12 @@ export function resolveSearchConfig(opts: {false,"search_trust_gate_enabled",);+ // The gate's mode (session-mining-economy, t_05a0eaa4): an explicit+ // `search_trust_gate_mode` wins (invalid values refused naming the key+ // in force, `parseFusionMode` discipline), else the legacy boolean+ // above maps true to shadow and false to off. The refusal for a bad+ // legacy boolean is the `parseBool` one above, naming that key.+ const searchTrustGateMode = resolveSearchTrustGateMode({ env, config });const supersedeFadeEnabled = parseBool(envOrConfig(env,@@ -1484,6 +1494,9 @@ export function resolveSearchConfig(opts: {relationalArmEnabled,metadataBoostGateEnabled,retrievalTrustGateEnabled,+ // Only the off value stays absent (the relationalRerankPin pattern),+ // so exact-shape assertions on the resolved config keep holding.+ ...(searchTrustGateMode !== SEARCH_TRUST_GATE_MODE.off ? { searchTrustGateMode } : {}),supersedeFadeEnabled,learnedWeightsEnabled,activationEnabled,diff --git a/src/core/search/pipeline/cache-slot.ts b/src/core/search/pipeline/cache-slot.tsindex 864534a1..1d0aeb58 100644--- a/src/core/search/pipeline/cache-slot.ts+++ b/src/core/search/pipeline/cache-slot.ts@@ -298,6 +298,11 @@ export function configFingerprint(config: ResolvedSearchConfig): string {relPol: r.relationPolarityEnabled,relArm: r.relationalArmEnabled,trustGate: r.retrievalTrustGateEnabled,+ // The gate's resolved mode decides whether the gate excludes, so a+ // shadow or enforce row must never be served under an off key.+ // Present only when a mode is resolved (off stays absent), so every+ // off key is unchanged from the pre-mode one.+ ...(r.searchTrustGateMode !== undefined ? { trustGateMode: r.searchTrustGateMode } : {}),supFade: r.supersedeFadeEnabled,lw: r.learnedWeightsEnabled,// Cross-encoder rerank re-orders the cached outcome, so a toggle ordiff --git a/src/core/search/pipeline/post-rank.ts b/src/core/search/pipeline/post-rank.tsindex 414ed0fe..97696858 100644--- a/src/core/search/pipeline/post-rank.ts+++ b/src/core/search/pipeline/post-rank.ts@@ -24,7 +24,8 @@ import {buildMemoryTrustAssessment,buildRetrievalDecisionTrace,} from "../../brain/trust/retrieval-receipts.ts";-import { trustGateAdjuster } from "../../brain/trust/retrieval-gate.ts";+import type { RetrievalTrustRefusal } from "../../brain/trust/retrieval-receipts.ts";+import { collectTrustGateRefusals, trustGateAdjuster } from "../../brain/trust/retrieval-gate.ts";import { applyRelationPolarityPhase } from "../graph-phases.ts";import { clamp01 } from "../../math.ts";import { isVisible, pageVisibility } from "../../graph/visibility.ts";@@ -88,6 +89,17 @@ export interface PostRankInput {readonly signal?: AbortSignal;/** The request's clock, for the rerank sunset decision. */readonly nowMs: number;+ /**+ * The relational arm's lane-tagged eligibility refusals (session-mining-+ * economy, t_05a0eaa4), threaded from the arm outcome so the receipts+ * carry the arm lane end to end. Present only when the arm ran and its+ * resolved gate mode was active. Under `shadow` they ride the shadow+ * section beside the pool's; under `enforce` they are the arm lane's+ * drops - removed before the pool, where no exclusion can name them -+ * and ride the receipts' enforce section, so the gate's+ * never-silently-dropped posture holds on both lanes.+ */+ readonly armRefusals?: ReadonlyArray<RetrievalTrustRefusal>;}/** Start `work` under a budget; serve `fallback()` when the budget wins. */@@ -595,11 +607,36 @@ export async function applyPostRankPhases(input: PostRankInput): Promise<PostRan// registered the pool is returned unchanged, so the default path is// byte-identical. Runs BEFORE the slice so a gate exclusion lets a// deeper survivor backfill the window rather than shrinking it.+ //+ // The trust gate's resolved mode (session-mining-economy, t_05a0eaa4)+ // decides how it runs: off registers nothing and builds no receipts+ // (byte-identical); enforce registers the excluding adjuster; shadow+ // registers the mode-aware keep-only adjuster and collects the+ // would-be refusals from the same pool the adjusters see, feeding the+ // receipt build below. The kernel's verdict contract is untouched.+ const trustGateMode = config.recall.searchTrustGateMode ?? "off";+ const readGateFrontmatter = (path: string): FrontmatterMap =>+ readCachedFrontmatter(frontmatterCache, config.vault, path);const rankAdjusters: RankAdjuster[] = [];- if (config.recall.retrievalTrustGateEnabled) {- rankAdjusters.push(- trustGateAdjuster((path) => readCachedFrontmatter(frontmatterCache, config.vault, path)),- );+ let shadowRefusals: ReadonlyArray<RetrievalTrustRefusal> = [];+ let enforcedArmRefusals: ReadonlyArray<RetrievalTrustRefusal> = [];+ if (trustGateMode !== "off") {+ rankAdjusters.push(trustGateAdjuster(readGateFrontmatter, trustGateMode));+ if (trustGateMode === "shadow") {+ // The pool lane first, then the arm lane appended after it: the+ // arm's own eligibility refusals arrive lane-tagged from the arm+ // outcome and are reported beside the pool's, never folded into it.+ shadowRefusals = [+ ...collectTrustGateRefusals(pinnedReranked, readGateFrontmatter),+ ...(input.armRefusals ?? []),+ ];+ } else {+ // Enforce: the pool's drops ARE the exclusions, but the arm's drops+ // happen before the pool, where no exclusion can ever name them -+ // they ride the receipts' enforce section instead, so a+ // quarantined candidate is never silently dropped on either lane.+ enforcedArmRefusals = input.armRefusals ?? [];+ }}if (config.recall.supersedeFadeEnabled) {// Relation-only supersede fade (t_c4a9cef8): fetch the pool's typed@@ -613,18 +650,29 @@ export async function applyPostRankPhases(input: PostRankInput): Promise<PostRan// Per-pack retrieval trust receipts (t_5f61130a): compact references// consistent with the context-receipt model. Built only when the gate// ran, so the outcome shape stays byte-identical on the default path.- const trustReceipts = config.recall.retrievalTrustGateEnabled- ? {- retrievalDecisionTrace: buildRetrievalDecisionTrace({- surfaced: adjusted.results.length,- excluded: adjusted.excluded,- }),- memoryTrustAssessment: buildMemoryTrustAssessment({- surfaced: adjusted.results.length,- excluded: adjusted.excluded,- }),- }- : null;+ // Under shadow the would-be refusals ride beside an untouched+ // `excluded`; under enforce the receipts keep the pre-mode shape.+ const trustReceipts =+ trustGateMode === "off"+ ? null+ : {+ retrievalDecisionTrace: buildRetrievalDecisionTrace({+ surfaced: adjusted.results.length,+ excluded: adjusted.excluded,+ ...(trustGateMode === "shadow" ? { shadow: { refusals: shadowRefusals } } : {}),+ ...(enforcedArmRefusals.length > 0+ ? { enforce: { refusals: enforcedArmRefusals } }+ : {}),+ }),+ memoryTrustAssessment: buildMemoryTrustAssessment({+ surfaced: adjusted.results.length,+ excluded: adjusted.excluded,+ ...(trustGateMode === "shadow" ? { shadow: { refusals: shadowRefusals } } : {}),+ ...(enforcedArmRefusals.length > 0+ ? { enforce: { refusals: enforcedArmRefusals } }+ : {}),+ }),+ };return {results: adjusted.results,diff --git a/src/core/search/pipeline/relational-arm.ts b/src/core/search/pipeline/relational-arm.tsindex 5938c8a0..2f15b849 100644--- a/src/core/search/pipeline/relational-arm.ts+++ b/src/core/search/pipeline/relational-arm.ts@@ -41,6 +41,16 @@ import {} from "../result-filters.ts";import { parseValidityWindow } from "../validity.ts";import { SearchError } from "../search-error.ts";+import {+ RETRIEVAL_TRUST_EXCLUSION_REASON,+ SEARCH_TRUST_GATE_MODE,+ classifyRetrievalTrust,+ resolveSearchTrustGateMode,+} from "../../brain/trust/retrieval-gate.ts";+import {+ RETRIEVAL_TRUST_REFUSAL_LANE,+ type RetrievalTrustRefusal,+} from "../../brain/trust/retrieval-receipts.ts";import type { BrainSearchResult } from "../search-result.ts";import type { Store } from "../store.ts";import type { ResolvedSearchConfig, SearchOptions } from "../types.ts";@@ -53,6 +63,44 @@ export const ENTITY_BRIDGES_ENV = "OPEN_SECOND_BRAIN_SEARCH_ENTITY_BRIDGES";/** Machine-config key for walking entity bridges. */export const ENTITY_BRIDGES_CONFIG = "search_entity_bridges_enabled";+/**+ * Namespace the arm's refusal reasons carry, mirroring how the kernel+ * namespaces an adjuster's exclusion: the origin of the verdict, never+ * prose. Combined with the per-lane split on the refusal record, an arm+ * refusal and a pool exclusion can never blur.+ */+export const RELATIONAL_ARM_REFUSAL_NAMESPACE = "relational_arm";++/**+ * The full structural reason vocabulary of the arm's edge-level+ * eligibility gate: the page gate's three controlled-vocabulary signals+ * reused verbatim ({@link RETRIEVAL_TRUST_EXCLUSION_REASON} - the same+ * controlled-vocabulary frontmatter signals the page gate reads), plus+ * the four tokens only an edge and its window can fail on. Fixed tokens,+ * not prose, so the same vault yields the same record.+ */+export const RELATIONAL_ARM_REFUSAL_REASON = Object.freeze({+ ...RETRIEVAL_TRUST_EXCLUSION_REASON,+ /** A DECLARED validity window value that cannot be parsed. */+ invalidValidityWindow: "invalid_validity_window",+ /** A declared window whose inclusive end is strictly past. */+ closedValidityWindow: "closed_validity_window",+ /** A declared window whose start is still ahead of query time. */+ pendingValidityWindow: "pending_validity_window",+ /** The arriving relation's polarity is a counted exclusion. */+ contradictsPolarity: "contradicts_polarity",+} as const);++/**+ * The relation polarities the edge gate counts as exclusions, from the+ * existing relation-polarity vocabulary: `contradicts` is the+ * relevant-but-not-endorsing edge the pool-level pass only warns about.+ * `superseded_by` is not here - it is the supersession backbone the arm+ * annotates (`supersededBy`), pointing stale-to-fresh - and the positive+ * relations boost.+ */+const REFUSAL_POLARITIES: ReadonlySet<string> = new Set(["contradicts"]);+/*** Whether the arm walks entity bridges: on by default (design decision 11),* overridable per environment or machine config. A present but invalid@@ -143,6 +191,42 @@ export interface RelationalReach {export interface RelationalArmOutcome {readonly rankedChunkIds: number[];readonly reachByChunk: Map<number, RelationalReach>;+ /**+ * The arm's lane of the retrieval trust receipts' refusal record+ * (session-mining-economy, t_05a0eaa4). Present only when the trust+ * gate's resolved mode is not off - the eligibility gate ran - and+ * lists every walked node whose arrival (its own frontmatter signals,+ * its validity window at query time, the arriving edge's polarity, or+ * the arrived edge's source endpoint) the classification marked+ * ineligible AND whose derivation the caller could read: a node, or an+ * edge source, withheld at the caller's reach and owner scope+ * contributes no record, so the receipts read the same whether the walk+ * crossed unreadable documents or not. This is a DIFFERENT act from the+ * reach gating's identical-to-absent provenance convention above: a+ * refusal is removed from (enforce) or kept in (shadow) the+ * contribution set AND counted here, while an omitted provenance step+ * is never named and never counted.+ */+ readonly refusals?: ReadonlyArray<RetrievalTrustRefusal>;+}++/**+ * Whether a caller at this reach and owner scope could read the page: the+ * exact verdict {@link gatePath} applies to every provenance step, so+ * both disclosure channels - the provenance beside a row and the trust+ * receipts' refusal records - are held to the same line.+ */+function isCallerReadable(+ vault: string,+ path: string,+ reach: TransportReach,+ agentScope: string | null,+ cache: FrontmatterCache,+): boolean {+ return (+ isPathReadableAtReach(vault, path, reach, cache) &&+ (agentScope === null || isPathOwnerVisible(vault, path, agentScope, cache))+ );}/**@@ -213,6 +297,12 @@ export function runRelationalArm(const config = machineConfigData(opts.configPath);const budgets = opts.budgets ?? resolveTraversalBudgets({ env: process.env, config });const bridges = opts.entityBridges ?? resolveEntityBridgesEnabled({ env: process.env, config });+ // The edge-level eligibility gate (session-mining-economy, t_05a0eaa4)+ // runs under the same resolved trust-gate mode the page gate answers+ // to - the same env and the caller's own config map, so the two gates+ // can never disagree about the posture. Off: no classification, no+ // refusal field, the outcome shape byte-identical.+ const trustGateMode = resolveSearchTrustGateMode({ env: process.env, config });const nodes = relationalFanout(fanoutStoreFor(store, bridges), seedDocIds, {maxDepth: RELATIONAL_MAX_DEPTH,edgeTypes: relQuery.edgeTypes,@@ -232,12 +322,64 @@ export function runRelationalArm(const titles = store.documentTitles();const nowMs = Date.now();const seenKeys = new Set<string>();+ const gateActive = trustGateMode !== SEARCH_TRUST_GATE_MODE.off;+ const refusals: RetrievalTrustRefusal[] = [];for (const node of nodes) {const rep = reps.get(node.documentId);if (rep === undefined) continue;const key = rrfKey({ origin: null, path: rep.path, chunkId: rep.chunkId });if (seenKeys.has(key)) continue;seenKeys.add(key);+ if (gateActive) {+ // The walked edge that carried the walk in is the path's last step;+ // its source endpoint is another walked node except at hop 1, where+ // the source is a seed the caller named itself.+ const lastStep = node.path.at(-1);+ const sourceStep = node.path.length >= 2 ? node.path.at(-2) : undefined;+ const sourceMeta = sourceStep === undefined ? undefined : titles.get(sourceStep.documentId);+ const sourcePath = sourceMeta?.path ?? null;+ const eligibility = classifyArrivalEligibility({+ vault,+ nodePath: rep.path,+ sourcePath,+ arrivingRelation: lastStep?.relation ?? null,+ cache: frontmatterCache,+ nowMs,+ });+ if (eligibility.node.length + eligibility.source.length > 0) {+ // A refusal is RECORDED only for what the caller could read at+ // its reach and owner scope - the identical-to-absent convention+ // the provenance gate below holds: a page withheld there+ // contributes nothing to the receipts, so the records a caller+ // sees read the same whether the walk crossed unreadable+ // documents or not. The enforcement verdict below stays the FULL+ // classification: the exclusion (enforce) or the kept+ // contribution (shadow) does not depend on what was recordable.+ const recordable = isCallerReadable(vault, rep.path, reach, agentScope, frontmatterCache)+ ? [+ ...eligibility.node,+ ...(sourcePath === null ||+ isCallerReadable(vault, sourcePath, reach, agentScope, frontmatterCache)+ ? eligibility.source+ : []),+ ]+ : [];+ if (recordable.length > 0) {+ refusals.push(+ Object.freeze({+ document_id: rep.documentId,+ chunk_id: rep.chunkId,+ path: rep.path,+ reasons: Object.freeze(recordable.toSorted()),+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.arm,+ }),+ );+ }+ // Enforce removes the ineligible edge's node from the+ // contribution set; shadow counts it and contributes as usual.+ if (trustGateMode === SEARCH_TRUST_GATE_MODE.enforce) continue;+ }+ }const gated = gatePath(vault, node.path, titles, reach, agentScope, frontmatterCache, nowMs);outcome.rankedChunkIds.push(rep.chunkId);outcome.reachByChunk.set(rep.chunkId, {@@ -246,7 +388,7 @@ export function runRelationalArm(...gated,});}- return outcome;+ return { ...outcome, ...(gateActive ? { refusals: Object.freeze(refusals) } : {}) };}/**@@ -322,6 +464,88 @@ function closedSupersessionTip(return (wikilink !== null ? wikilink[1]! : raw).trim();}+/**+ * The eligibility verdict for one walked node's arrival, split by which+ * page each reason was derived from. Four questions, all frontmatter-+ * structural, none prose:+ *+ * - the node's page and - for a node reached from another walked node -+ * the arrived edge's source page carry one of the page gate's three+ * controlled-vocabulary signals (a seed is the caller's own entry+ * point and is judged only as far as the walk reads it, never+ * refused for its own markers);+ * - the node's declared validity window covers query time+ * (`parseValidityWindow`, the `closedSupersessionTip` inclusive-end+ * grammar; a DECLARED value that cannot be parsed is a refusal of+ * its own, never a silent pass);+ * - the arriving relation's polarity is not a counted exclusion+ * (the existing relation-polarity vocabulary).+ *+ * The split exists for the receipts' disclosure line: the `node` reasons+ * describe the node's own page and arriving edge, the `source` reasons+ * describe ANOTHER page, and a record may carry a page's facts only when+ * that page is readable at the caller's reach and scope. The enforcement+ * verdict is the union and never differs between modes.+ *+ * An unreadable page answers no signals, the same tolerance the page+ * gate's shared frontmatter reader gives it.+ */+function classifyArrivalEligibility(input: {+ readonly vault: string;+ /** The arrived-at node's page path. */+ readonly nodePath: string;+ /** The arrived edge's source page path, when the walk records it. */+ readonly sourcePath: string | null;+ /** The relation that carried the walk into the node, when known. */+ readonly arrivingRelation: string | null;+ readonly cache: FrontmatterCache;+ readonly nowMs: number;+}): { readonly node: string[]; readonly source: string[] } {+ const node: string[] = [];+ const source: string[] = [];+ const pageGateReasons = (path: string, into: string[]): void => {+ const entry = readCachedFrontmatterEntry(input.cache, input.vault, path);+ if (entry.unreadable) return;+ for (const reason of classifyRetrievalTrust(entry.meta).reasons) {+ into.push(`${RELATIONAL_ARM_REFUSAL_NAMESPACE}:${reason}`);+ }+ };+ pageGateReasons(input.nodePath, node);+ if (input.sourcePath !== null) pageGateReasons(input.sourcePath, source);++ const entry = readCachedFrontmatterEntry(input.cache, input.vault, input.nodePath);+ if (!entry.unreadable) {+ // Null window: neither side declared - an open window, which covers+ // query time trivially.+ const window = parseValidityWindow(entry.meta);+ if (window !== null) {+ if (window.invalid) {+ node.push(+ `${RELATIONAL_ARM_REFUSAL_NAMESPACE}:${RELATIONAL_ARM_REFUSAL_REASON.invalidValidityWindow}`,+ );+ } else if (window.validFromMs !== null && window.validFromMs > input.nowMs) {+ node.push(+ `${RELATIONAL_ARM_REFUSAL_NAMESPACE}:${RELATIONAL_ARM_REFUSAL_REASON.pendingValidityWindow}`,+ );+ } else if (window.validUntilMs !== null && window.validUntilMs < input.nowMs) {+ node.push(+ `${RELATIONAL_ARM_REFUSAL_NAMESPACE}:${RELATIONAL_ARM_REFUSAL_REASON.closedValidityWindow}`,+ );+ }+ }+ }++ if (+ input.arrivingRelation !== null &&+ REFUSAL_POLARITIES.has(normalizeRelation(input.arrivingRelation))+ ) {+ node.push(+ `${RELATIONAL_ARM_REFUSAL_NAMESPACE}:${RELATIONAL_ARM_REFUSAL_REASON.contradictsPolarity}`,+ );+ }+ return { node, source };+}+/*** Gate one node's walked path at the caller's reach and owner scope:* unreadable steps and steps another agent owns (under a requested scope)@@ -350,13 +574,7 @@ function gatePath(for (const step of steps) {const meta = titles.get(step.documentId);const unreadable =- meta === undefined ||- !isPathReadableAtReach(vault, meta.path, reach, cache) ||- // The same owner-scope verdict the row-level filters apply to- // content rows: an owner-private page another agent owns is as- // withheld from provenance as an unreadable one. No requested- // scope means no ownership filtering - the opt-in default.- (agentScope !== null && !isPathOwnerVisible(vault, meta.path, agentScope, cache));+ meta === undefined || !isCallerReadable(vault, meta.path, reach, agentScope, cache);if (unreadable) continue;gated.push(step);if (supersededBy === undefined) {diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex f7a42279..cbd38ad9 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -1044,6 +1044,11 @@ export async function search(structured: shape.structured,frontmatterCache,nowMs,+ // The arm's eligibility refusals (present only under an active gate+ // mode) ride into the receipt build, so the shadow receipts report+ // the arm lane end to end; under enforce the receipts keep the+ // pre-mode shape and the field is not consulted.+ ...(relational.refusals !== undefined ? { armRefusals: relational.refusals } : {}),...(deadline !== null? {signal: deadline.signal,diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex 6cf2b41d..ddc9a59b 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -32,6 +32,7 @@ import type {MemoryTrustAssessment,RetrievalDecisionTrace,} from "../brain/trust/retrieval-receipts.ts";+import type { SearchTrustGateMode } from "../brain/trust/retrieval-gate.ts";import type { RetrievalTrail } from "./retrieval-trail.ts";import type { ProviderProbeState } from "./provider-probe.ts";@@ -1691,6 +1692,21 @@ export interface ResolvedRecallConfig {* outcome shape byte-identical.*/readonly retrievalTrustGateEnabled: boolean;+ /**+ * The retrieval trust gate's resolved mode (session-mining-economy,+ * t_05a0eaa4): `off` (the gate does not run - ranking and the outcome+ * shape stay byte-identical, so this value stays absent from the+ * resolved config like every other opt-in recall field), `shadow`+ * (the classification runs, nothing is excluded, every would-be+ * refusal is recorded with its structural reasons into the receipts)+ * or `enforce` (today's excluding behavior, made explicit). Resolved+ * as the explicit `search_trust_gate_mode` key (env+ * OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE), else the legacy+ * `search_trust_gate_enabled` boolean mapped true to shadow, else off.+ * The mode rides the query-cache key, and the explain envelope names+ * it under shadow.+ */+ readonly searchTrustGateMode?: SearchTrustGateMode;/*** Relation-only supersede fade (t_c4a9cef8), kernel 1's second* consumer. Off by default: when true, a candidate a surfaceddiff --git a/src/core/search/visibility-surface-registry.ts b/src/core/search/visibility-surface-registry.tsindex d3eeb858..58335339 100644--- a/src/core/search/visibility-surface-registry.ts+++ b/src/core/search/visibility-surface-registry.ts@@ -1458,4 +1458,16 @@ export const DIRECT_VAULT_READ_REGISTRY: ReadonlyArray<DirectVaultReadEntry> = O"own shell, so the reach question does not arise, and no page's reservation could " +"cover what it hands back.",},+ {+ file: "src/mcp/brain/ingest-tools.ts",+ category: D.discloses_nothing,+ reason:+ "stampedArchiveHashes readFileSync+parseFrontmatter's the summary pages of the " ++ "caller-named source's cleanup plan BEFORE a confirmed deletion erases the stamps " ++ "with the pages (the archive-orphan decision needs them). The page set comes from " ++ "the plan's own derived manifest - never from caller-supplied paths - and every " ++ "path in it is already named by the preview the same run serialized. What crosses " ++ "back is content hashes and orphan dispositions of that same set: no page title, " ++ "body or frontmatter value, so no page's reservation could cover the answer.",+ },]);diff --git a/src/core/state/surfaces.ts b/src/core/state/surfaces.tsindex 2dc003fd..88f4f301 100644--- a/src/core/state/surfaces.ts+++ b/src/core/state/surfaces.ts@@ -117,6 +117,7 @@ export const STATE_SURFACE_ID = Object.freeze({sessionImportCheckpoints: "session_import_checkpoints",writeDeadLetters: "write_dead_letters",sessionImportLedger: "session_import_ledger",+ sessionMiningLedger: "session_mining_ledger",installManifest: "install_manifest",protectManifest: "protect_manifest",bootstrapReceipt: "bootstrap_receipt",@@ -284,6 +285,7 @@ const INGEST_CHECKPOINT_DIR = "ingest-checkpoints";const SESSION_CHECKPOINT_DIR = "session-import-checkpoints";const DEAD_LETTER_DIR = "dead-letters";const SESSION_LEDGER_FILE = "session-import-ledger.json";+const SESSION_MINING_LEDGER_FILE = "session-mining-ledger.json";const INSTALL_MANIFEST_FILE = "install.lock.json";const PROTECT_MANIFEST_FILE = "protect.lock.json";const BOOTSTRAP_RECEIPT_FILE = "bootstrap.lock.json";@@ -501,6 +503,22 @@ export const STATE_SURFACES: ReadonlyArray<StateSurface> = Object.freeze(["imported log as a gap; re-importing them is safe but pays the full parse again.",sources: ["src/core/brain/sessions/discover.ts"],},+ {+ id: STATE_SURFACE_ID.sessionMiningLedger,+ label: "session mining ledger",+ tier: STATE_TIER.derived,+ derive: derivedStore(SESSION_MINING_LEDGER_FILE),+ override_env: null,+ override_config_key: null,+ carries_memory: false,+ reason:+ "One tombstone per agent session whose mining commit ended barren, keyed by the portable " ++ "session id and hashed on the mined turn material, so the plan gate suppresses an already-" ++ "mined zero-yield session instead of re-offering it to the model on every re-scan. " ++ "Deleting it reports no gap and loses nothing the vault holds; it only re-opens every " ++ "barren session, paying the full model spend again on the next scan.",+ sources: ["src/core/brain/session-mining-ledger.ts"],+ },{id: STATE_SURFACE_ID.installManifest,label: "install manifest",@@ -1143,6 +1161,13 @@ export const STATE_SURFACE_SWEEP_EXCLUSIONS: ReadonlyMap<string, string> = new M"so the weekly walk does not read derived state as vault content. A skipped directory is " +"not a surface this module owns",],+ [+ "src/core/brain/storage/usage.ts",+ "the capacity meter: it joins the store directory to MEASURE the store and the state beside " ++ "it (the derived_store and derived_store_dir surfaces of the storage census), never to " ++ "write one. It is the meter every row in this catalogue is charged by, and it creates " ++ "nothing",+ ],["src/cli/brain/verbs/bench.ts","its `.open-second-brain/bench-runs` default is resolved against the WORKING DIRECTORY, not " +diff --git a/src/mcp/brain/distill-tools.ts b/src/mcp/brain/distill-tools.tsindex bbc9bd4e..de7c7a2c 100644--- a/src/mcp/brain/distill-tools.ts+++ b/src/mcp/brain/distill-tools.ts@@ -152,8 +152,12 @@ export const DISTILL_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([// Note the asymmetry with `brain_intake_entities`, whose quarantine holds// by DEFAULT: it works through `status: quarantine` plus the entity page// status scope, which nothing gates. The same words mean less here.+ //+ // Session-mining-economy: the legacy boolean alone now mounts the gate+ // in the measure-only shadow mode, so exclusion requires the mode to+ // name `enforce` explicitly. The description states both keys.description:- "Distill one source into atomic claims; runs no model, checks quotes. Writes one idempotent page citing each claim as `[[source#^block]]`. A source outside this vault is marked `untrusted_source`, which ordinary reads still return unless `search_trust_gate_enabled` is on. `trust` names the lane.",+ "Distill one source into atomic claims; runs no model, checks quotes. Writes an idempotent page citing each claim as `[[source#^block]]`. `untrusted_source` pages surface unless the gate enforces (`search_trust_gate_mode: enforce`; `search_trust_gate_enabled` alone measures). `trust` names the lane.",inputSchema: {type: "object",properties: {diff --git a/src/mcp/brain/extract-tools.ts b/src/mcp/brain/extract-tools.tsindex 5d2a537d..5e20529f 100644--- a/src/mcp/brain/extract-tools.ts+++ b/src/mcp/brain/extract-tools.ts@@ -51,6 +51,10 @@ async function toolBrainExtractSignals(decisionModel,tokenImpactEnabled:decisionModel !== null && resolveTokenImpactLedgerEnabled(ctx.configPath ?? undefined),+ // The re-mine escape: bypass the zero-yield tombstone for this one+ // plan call, for "I removed the downstream item and want it+ // re-proposed".+ force: args["force"] === true,});return {phase: "plan",@@ -63,17 +67,34 @@ async function toolBrainExtractSignals(text: t.text,timestamp: t.timestamp,})),+ // The hash of the material this plan offered; the commit that+ // answers it passes it back as `surveyed_turns_hash` so the mining+ // tombstone keys on the surveyed material.+ surveyed_turns_hash: plan.surveyedTurnsHash,cap: plan.cap,confidence_floor: plan.confidenceFloor,llm_step: plan.llmStep,// Optional fields, present only while the decision-model turn- // pre-filter is on (never with the use `off`).+ // pre-filter is on (never with the use `off`) or when the session+ // carries a zero-yield mining tombstone....(plan.turnsDropped !== undefined ? { turns_dropped: plan.turnsDropped } : {}),...(plan.skipped !== undefined? {skipped: {reason: plan.skipped.reason,- turns_dropped: plan.skipped.turnsDropped,+ ...(plan.skipped.turnsDropped !== undefined+ ? { turns_dropped: plan.skipped.turnsDropped }+ : {}),+ ...(plan.skipped.miningOutcome !== undefined+ ? {+ mining_outcome: {+ written: plan.skipped.miningOutcome.written,+ staged: plan.skipped.miningOutcome.staged,+ deduped: plan.skipped.miningOutcome.deduped,+ durability_rejected: plan.skipped.miningOutcome.durabilityRejected,+ },+ }+ : {}),},}: {}),@@ -83,12 +104,22 @@ async function toolBrainExtractSignals(};}const now = new Date();+ const surveyedTurnsHash = coerceStr(args, "surveyed_turns_hash", false);const res = commitExtractedSignals(ctx.vault,session,{ items: args["items"] },// Loaded with parallel I/O here, where the tool can await it.- { agent, now, dedup: await loadDedupIndex(ctx.vault) },+ {+ agent,+ now,+ dedup: await loadDedupIndex(ctx.vault),+ // The answered plan's surveyed hash, echoed back by the caller:+ // what the zero-yield tombstone keys on. Absent, the commit+ // records nothing - there is no material the verdict was earned+ // over.+ ...(surveyedTurnsHash ? { surveyedTurnsHash } : {}),+ },);recordExtractPrefilterCommit(ctx.vault, decisionModel, res, now);return {@@ -106,6 +137,11 @@ async function toolBrainExtractSignals(deduped: res.deduped,durability_rejected: res.durabilityRejected,rejected: res.rejected.map((r) => ({ topic: r.topic, reason: r.reason })),+ // Present only when the commit succeeded but the zero-yield mining+ // tombstone could not be recorded - named, never thrown.+ ...(res.miningLedgerFailure !== undefined+ ? { mining_ledger_failure: res.miningLedgerFailure }+ : {}),};});}@@ -114,7 +150,7 @@ export const EXTRACT_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([{name: TOOL,description:- "Mine taste signals from an imported session's USER turns; runs no model. Without `items` returns the turns plus one needs-llm-step envelope. With `items` writes them to Brain/inbox/ as speculative `source_type: auto_extract` signals. Refuses over-cap, under-floor or repeated-topic payloads.",+ "Mine taste signals from an imported session's USER turns; runs no model. Omit `items` for the plan; with `items` pass the plan's `surveyed_turns_hash` back so a barren commit records its tombstone. Barren sessions skip as 'mining_tombstone'; `force` re-mines once. Refuses caps, floors, repeats.",inputSchema: {type: "object",properties: {@@ -122,6 +158,11 @@ export const EXTRACT_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([type: "string",description: "Recall session id of an already-imported session.",},+ surveyed_turns_hash: {+ type: "string",+ description:+ "Commit phase only: the `surveyed_turns_hash` from the plan response this payload answers. Omit it and a barren commit records nothing.",+ },items: {type: "array",description: "Mined signals; omit to get the plan and the envelope instead.",@@ -162,6 +203,11 @@ export const EXTRACT_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([type: "string",description: "Optional agent identity override; defaults to the server-resolved name.",},+ force: {+ type: "boolean",+ description:+ "Plan phase only: bypass the zero-yield mining tombstone for this one call and re-propose the session.",+ },},required: ["session"],additionalProperties: false,diff --git a/src/mcp/brain/health-tools.ts b/src/mcp/brain/health-tools.tsindex 28a8fa2a..83d8622b 100644--- a/src/mcp/brain/health-tools.ts+++ b/src/mcp/brain/health-tools.ts@@ -7,7 +7,10 @@*/import { resolveSearchConfig } from "../../core/search/index.ts";-import { collectMaintenanceActions } from "../../core/brain/maintenance/collect.ts";+import {+ collectMaintenanceActions,+ type MaintenanceCollectError,+} from "../../core/brain/maintenance/collect.ts";import { runDoctor } from "../../core/brain/doctor.ts";import { annotateEntityAliasIssues } from "../../core/brain/doctor/entity-alias-verdicts.ts";import { verdictFields } from "../../core/decision-model/pair-verdict.ts";@@ -182,6 +185,11 @@ async function toolBrainDoctor(// Resolved through the same table the CLI renderer reads.const noExit = noExitReasons([...errors, ...warnings, ...(uncertain ?? [])].map((i) => i.code));+ // The maintenance collector's absorbed scan failures (review M10):+ // observed here so a thin `suggested_actions` list is readable as+ // healthy rather than indistinguishable from a scan that refused.+ const collectErrors: MaintenanceCollectError[] = [];+return {format,ok,@@ -204,7 +212,10 @@ async function toolBrainDoctor(// because nothing here downgrades the `ok` flag - actions are// suggestions, not invariant violations.suggested_actions: view- .keep(collectMaintenanceActions(ctx.vault), (a) => [a.target])+ .keep(+ collectMaintenanceActions(ctx.vault, { onError: (e) => collectErrors.push(e) }),+ (a) => [a.target],+ ).map((a) => ({id: a.id,category: a.category,@@ -212,6 +223,11 @@ async function toolBrainDoctor(impact: a.impact,...(a.target !== undefined ? { target: a.target } : {}),})),+ // Rendered only when a scan refused, so a clean payload is+ // byte-identical to the pre-row shape (review M10).+ ...(collectErrors.length > 0+ ? { collect_errors: collectErrors.map((e) => ({ scan: e.signal, message: e.message })) }+ : {}),// v0.10.16: trust-layer fields. `trust_verdict` is always populated// by runDoctor; `verification_delta_summary` only when the caller// threads a dream summary through (not exposed via this tool'sdiff --git a/src/mcp/brain/ingest-tools.ts b/src/mcp/brain/ingest-tools.tsindex ff98a1e0..29713018 100644--- a/src/mcp/brain/ingest-tools.ts+++ b/src/mcp/brain/ingest-tools.ts@@ -9,7 +9,7 @@* Idempotent on the source path.*/-import { existsSync } from "node:fs";+import { existsSync, readFileSync } from "node:fs";import { join } from "node:path";import { serializeImportCensus } from "../../core/brain/import-census.ts";@@ -18,7 +18,17 @@ import { clearCheckpoint, PLAN_ID_LABEL } from "../../core/brain/ingest/checkpoiimport { assertCheckpointId } from "../../core/brain/checkpoint-store.ts";import { STORAGE_CAPACITY_CRITICAL_CODE } from "../../core/brain/doctor/storage-capacity.ts";import { MAINTENANCE_CAPACITY_REMEDY } from "../../core/brain/maintenance/lane.ts";-import { loadStorageConfigSafe } from "../../core/brain/policy/load.ts";+import { BRAIN_CONFIG_SUPPORTED_VERSIONS } from "../../core/brain/policy/defaults.ts";+import { BrainConfigError } from "../../core/brain/policy/errors.ts";+import { KeyIndexCollector, type BlockParseContext } from "../../core/brain/policy/key-index.ts";+import {+ BRAIN_STORAGE_DEFAULTS,+ parseStorageBlock,+ resolveStorage,+} from "../../core/brain/policy/blocks/storage.ts";+import type { BrainConfig, ResolvedBrainStorageConfig } from "../../core/brain/types.ts";+import { brainConfigPath } from "../../core/brain/paths.ts";+import { parseBrainYaml, type ParsedBlock } from "../../core/brain/yaml-parse.ts";import {StorageCapacityError,type StorageErrorCode,@@ -86,6 +96,24 @@ const DEFAULT_MAX_BATCH_FILES = 25;/** The wire code of the capacity admission refusal, from the leaf's closed union. */const ADMISSION_REFUSAL_CODE: StorageErrorCode = "capacity_exhausted";+/**+ * The registered wire code for an unreadable or invalid `_brain.yaml` - the+ * same code the error-code registry maps {@link BrainConfigError} to.+ */+const CONFIG_GATE_REFUSAL_CODE = "config_invalid";++/**+ * A broken `_brain.yaml` the admission gate cannot decide under. Its prose+ * is built from a relative config label, never the config's host path, so+ * the refusal can cross to a remote caller (see {@link loadGateStorageConfig}).+ */+class StorageGateConfigError extends Error {+ constructor(message: string) {+ super(message);+ this.name = "StorageGateConfigError";+ }+}+/*** Refuse the ingest when a fresh census measures the vault at its* critical storage tier - the interactive half of the storage lifecycle's@@ -111,7 +139,7 @@ const ADMISSION_REFUSAL_CODE: StorageErrorCode = "capacity_exhausted";* write - cannot disagree about what to do next.*/function refuseWhenVaultExhausted(vault: string, reach: TransportReach, now: Date): void {- const storage = loadStorageConfigSafe(vault);+ const storage = loadGateStorageConfig(vault);if (storage.cap_bytes === null) return;const decision = evaluateCapacityGate(vault, {now,@@ -161,6 +189,69 @@ function admissionDetails(};}+/**+ * The storage policy the admission gate decides under, read from ONLY the+ * `storage:` block of `_brain.yaml` (review M1).+ *+ * The gate's decision needs no other block, but the whole-file loader+ * validated all of them, so a typo in an unrelated block turned every+ * ingest into an internal error naming the config's host path - including+ * on vaults whose operator never configured a cap, where the documented+ * default is that nothing refuses. This reader answers the gate's actual+ * question: an absent config or block is the defaults (no cap, no gate),+ * and the rest of the file is never validated here.+ *+ * A file that cannot be read or parsed, an unsupported `schema_version`,+ * or a `storage:` block that does not validate is a policy this gate+ * cannot see through, and silently reading it as "no cap" would run+ * writes past an operator's bound. Those refuse BY NAME: the error+ * carries field-level prose built from a relative config label, and the+ * handler maps it onto the registered {@link CONFIG_GATE_REFUSAL_CODE}+ * envelope - never a path below local reach.+ */+function loadGateStorageConfig(vault: string): ResolvedBrainStorageConfig {+ const path = brainConfigPath(vault);+ if (!existsSync(path)) return BRAIN_STORAGE_DEFAULTS;+ let text: string;+ try {+ text = readFileSync(path, "utf8");+ } catch (err) {+ // Only the errno code is kept: a read-failure message embeds the+ // absolute path the call failed on.+ const code = (err as NodeJS.ErrnoException).code ?? "unreadable";+ throw new StorageGateConfigError(`_brain.yaml could not be read (${code})`);+ }+ let parsed: ParsedBlock;+ try {+ parsed = parseBrainYaml(text);+ } catch (err) {+ // The parser's messages name line numbers, never paths.+ throw new StorageGateConfigError(`_brain.yaml could not be parsed: ${(err as Error).message}`);+ }+ const version = parsed["schema_version"];+ if (typeof version !== "number" || !BRAIN_CONFIG_SUPPORTED_VERSIONS.includes(version)) {+ throw new StorageGateConfigError(+ `_brain.yaml: unsupported schema_version: ${JSON.stringify(version ?? null)}`,+ );+ }+ const ctx: BlockParseContext = {+ obj: parsed,+ // A relative label, never the host path: the field errors this context+ // raises become the refusal's prose.+ source: "_brain.yaml",+ warnings: [],+ keys: new KeyIndexCollector(),+ };+ try {+ // Only the storage block is parsed and resolved; `resolveStorage`+ // reads nothing else off the config.+ return resolveStorage({ storage: parseStorageBlock(ctx) } as BrainConfig);+ } catch (err) {+ if (err instanceof BrainConfigError) throw new StorageGateConfigError(err.message);+ throw err;+ }+}+/*** The MCP envelope for an admission refusal, from the thrown leaf: the* wire code is the code the registered `carried()` classification reads@@ -220,6 +311,14 @@ async function toolBrainIngestSource(refuseWhenVaultExhausted(ctx.vault, contextReach(ctx), new Date());} catch (err) {if (err instanceof StorageCapacityError) throw capacityRefusalEnvelope(err);+ // A broken config the gate cannot see through refuses by name, with+ // prose free of host paths (review M1) - never an internal error a+ // caller would retry into the same wall.+ if (err instanceof StorageGateConfigError) {+ throw new MCPError(INVALID_PARAMS, `${TOOL}: ${err.message}`, {+ code: CONFIG_GATE_REFUSAL_CODE,+ });+ }throw err;}let res: IngestSourceResult;@@ -298,6 +397,21 @@ async function toolBrainIngestSource(...nextCommandField(PENDING_STAGED_DIAGNOSTIC_CODE),}: {}),+ // The appended delta this run proved (t_a9cd4221, review M14): the+ // boundary-safe byte range beside its named kind, absent on every+ // shape that is not a proven append - a first ingest, a rewrite, a+ // source with no watermark. The range lets the caller see what this+ // summary is newly about; extraction itself stays whole-source, so+ // the field is reporting, never a claim that only the tail was read.+ ...(res.extractionDelta !== undefined+ ? {+ extraction_delta: {+ kind: res.extractionDelta.kind,+ from_byte: res.extractionDelta.fromByte,+ to_byte: res.extractionDelta.toByte,+ },+ }+ : {}),// Only for an HTML or CSV/TSV source, so a text source's payload is// byte-identical to before. Counts and tokens only, never content....(res.parts !== undefined ? { parts: serializeParts(res.parts) } : {}),@@ -457,6 +571,12 @@ function serializePlan(originals: [...plan.originals],manifest_entry: plan.manifestEntry,deleted: [...plan.deleted],+ // The per-entry removal failures reach the tool answer (review M8): a+ // confirmed removal that could not delete a file must never read as a+ // clean run with a smaller count. Always carried (empty on a clean or+ // dry run) so an automations can read the field unconditionally, like+ // `deleted` beside it.+ failed: plan.failed.map((f) => ({ path: f.path, message: f.message })),manifest_entry_removed: plan.manifestEntryRemoved,audit_record_id: plan.auditRecordId,// The archive-entry cleanup kind (answer trust, Task C4), reported as@@ -603,6 +723,11 @@ export function serializeBatchPlan(plan: BatchPlan): Record<string, unknown> {total_files: plan.totalFiles,total_bytes: plan.totalBytes,skipped: [...plan.skipped],+ // The H3 resurrect guard on the wire: sources whose manifest entry is+ // marked retired and whose bytes are unchanged. Only emitted when+ // non-empty, so a plan with no retired entry serializes byte-identically+ // to before.+ ...(plan.skippedRetired.length > 0 ? { skipped_retired: [...plan.skippedRetired] } : {}),// Only emitted when the manifest's recorded extraction contract differs// from the live one (t_586d5d8b), so a plan under an unchanged contract// serializes byte-identically to before. The count names how many filesdiff --git a/src/mcp/brain/receipt-tools.ts b/src/mcp/brain/receipt-tools.tsindex 6b7eda3e..a96fb6ef 100644--- a/src/mcp/brain/receipt-tools.ts+++ b/src/mcp/brain/receipt-tools.ts@@ -23,7 +23,7 @@import { resolveAgentName } from "../../core/config.ts";import { compactReceiptLedgers } from "../../core/brain/receipts/compact.ts";-import { scanReceiptLedgers } from "../../core/brain/receipts/scan.ts";+import { receiptShardNameError, scanReceiptLedgers } from "../../core/brain/receipts/scan.ts";import { compactView, scanView } from "../../core/brain/receipts/views.ts";import {isReceiptMode,@@ -85,7 +85,14 @@ function readShards(args: Record<string, unknown>): ReceiptShardRef[] {}return (raw as string[]).map((value) => {try {- return parseReceiptShardRef(value);+ const ref = parseReceiptShardRef(value);+ // Up front, beside the family check: a name the family's grammar+ // does not recognise is the caller's invalid argument - a plain+ // error escaping the core's per-shard handling would surface as an+ // internal error instead (L9).+ const nameError = receiptShardNameError(ref);+ if (nameError !== null) throw new Error(nameError);+ return ref;} catch (exc) {throw new MCPError(INVALID_PARAMS, (exc as Error).message);}@@ -106,12 +113,17 @@ async function toolBrainReceipts(return scanView(scanReceiptLedgers(ctx.vault, { olderThanDays, now: new Date() }));}- // The destructive half, gated like the storage report gates its- // whole-vault census (`./storage-tools.ts`): compact rewrites named- // ledger shards in place and reports whole-ledger byte totals, and a- // caller below local reach gets the named refusal instead of a plan -- // no shard list, no figures, no write lane entered. The scan above- // stays readable at any reach: it writes nothing.+ // The destructive half: compact rewrites named ledger shards in place,+ // and every guarantee the rewrite runs under - vault identity, the+ // permissions-document consult, the recovery point, the per-file locks -+ // is host-local. A caller below local reach gets the named refusal+ // instead of a plan: no shard list, no figures, no write lane entered.+ // The rationale names the write lane, not the byte figures (L11): the+ // scan above reports the same per-shard figures at any reach by design,+ // so figures were never the thing that stops at local reach - and the+ // growth check that reads the same scan runs on the host, where the+ // distinction has no meaning. The scan stays readable at any reach: it+ // writes nothing.if (contextReach(ctx) !== TRANSPORT_REACH.local) {// Returned as the literal, `satisfies`-checked against the refusal// interface: the handler's wire type is Record<string, unknown>, and@@ -122,8 +134,9 @@ async function toolBrainReceipts(measured: false,reason: RECEIPT_COMPACT_REACH_REASON satisfies ReceiptCompactRefusalReason,detail:- "receipt compaction folds records in named ledger shards and reports " +- "whole-ledger byte totals; it answers at local reach only",+ "receipt compaction rewrites named ledger shards in place; its write lane and " ++ "guards (vault identity, permissions document, recovery point) are host-local, " ++ "so it answers at local reach only",} satisfies ReceiptCompactRefusal;}diff --git a/src/mcp/search-tools.ts b/src/mcp/search-tools.tsindex 0a05e4f0..c0fd305e 100644--- a/src/mcp/search-tools.ts+++ b/src/mcp/search-tools.ts@@ -1323,7 +1323,10 @@ async function toolBrainSearch(...explainEnvelope(outcome, {explain,crossVault: globalSearch,- trustGateEnabled: config.recall.retrievalTrustGateEnabled,+ // Engaged means the resolved mode is not off - the legacy boolean+ // alone cannot answer that (an explicit search_trust_gate_mode runs+ // the gate with the boolean left false).+ trustGateEnabled: config.recall.searchTrustGateMode !== undefined,}),...(telemetryRecord ? { telemetry_id: telemetryRecord.id } : {}),// Iterative round-trip marker: the plan phase carries the deciderdiff --git a/templates/brain-explorer.html b/templates/brain-explorer.htmlindex aad9ef56..df62127f 100644--- a/templates/brain-explorer.html+++ b/templates/brain-explorer.html@@ -241,6 +241,21 @@font-style: italic;font-size: 12px;}+ /* Live counts console (session-mining-economy): the written-per-day+ histogram is a plain flex row of spans - no chart dependency. */+ .bars {+ display: flex;+ align-items: flex-end;+ gap: 2px;+ height: 40px;+ margin: 4px 0 2px 0;+ }+ .bars .bar {+ flex: 1;+ min-height: 1px;+ background: var(--accent);+ opacity: 0.75;+ }</style></head><body>@@ -323,6 +338,15 @@<div id="details-body"><p class="panel-empty">Select a node to inspect.</p></div>+ <hr />+ <h3>Counts</h3>+ <div id="counts-body">+ <p class="panel-empty">Loading counts...</p>+ </div>+ <h3>Vault status</h3>+ <div id="ops-body">+ <p class="panel-empty">Loading status...</p>+ </div></aside><script type="application/json" id="brain-data">@@ -1102,10 +1126,221 @@// some configurations.window.addEventListener("pagehide", saveLayout);+ // ---- Live counts and vault status (session-mining-economy) ----------+ // The counts and ops panels fetch /counts.json and /status.json+ // client-side, so a refresh is a fresh server-side read: the+ // server keeps no caches and neither does this page. A failed+ // fetch renders the route and the reason - never an empty panel,+ // which would read as a vault with no activity. A static export+ // cannot reach the loopback routes, so it states that instead of+ // firing fetches that can only fail.+ function fieldRow(k, vHtml) {+ return (+ '<div class="field"><span class="k">' ++ escHtml(k) ++ '</span><span class="v">' ++ vHtml ++ "</span></div>"+ );+ }++ function fetchJson(route) {+ return fetch(route).then((res) => {+ if (res.ok) return res.json();+ return res+ .json()+ .catch(() => ({}))+ .then((body) => {+ const detail = body && body.error ? ": " + body.error : "";+ throw new Error("HTTP " + res.status + detail);+ });+ });+ }++ function renderLoadError(elId, route, err) {+ const reason = err && err.message ? err.message : String(err);+ document.getElementById(elId).innerHTML =+ '<p class="panel-empty">' + escHtml(route + " unavailable: " + reason) + "</p>";+ }++ function renderConsoleOffline(elId) {+ document.getElementById(elId).innerHTML =+ '<p class="panel-empty">' ++ escHtml(+ "Live counts and vault status are served by the live explorer;" ++ " a static export shows the graph only.",+ ) ++ "</p>";+ }++ function renderCounts(counts) {+ const parts = [];+ const tel = counts.telemetry || {};+ parts.push(+ fieldRow(+ "recalls",+ escHtml(String(tel.total || 0)) ++ " total, " ++ escHtml(String(tel.empty_runs || 0)) ++ " empty",+ ),+ );+ const channels = Object.keys(tel.by_channel || {});+ if (channels.length > 0) {+ parts.push(+ fieldRow(+ "by channel",+ channels.map((c) => escHtml(c + " " + tel.by_channel[c])).join(", "),+ ),+ );+ }+ const recalled = counts.most_recalled || [];+ if (recalled.length > 0) {+ parts.push("<h3>Most recalled</h3>");+ parts.push(+ '<ul class="edge-list">' ++ recalled+ .map((a) => {+ const label = a.id || a.path || "(unknown artifact)";+ return (+ "<li>" ++ escHtml(label) ++ ' <span class="kind">[' ++ a.access_count ++ "x]</span></li>"+ );+ })+ .join("") ++ "</ul>",+ );+ }+ const days = counts.written_per_day || [];+ if (days.length > 0) {+ const max = days.reduce((m, d) => Math.max(m, d.count), 0);+ parts.push("<h3>Writes per day</h3>");+ parts.push(+ '<div class="bars">' ++ days+ .map((d) => {+ const h = max > 0 ? Math.max(1, Math.round((d.count / max) * 40)) : 1;+ return (+ '<span class="bar" title="' ++ escHtml(d.date + ": " + d.count) ++ '" style="height:' ++ h ++ 'px"></span>'+ );+ })+ .join("") ++ "</div>",+ );+ parts.push(+ '<div class="stats">' ++ escHtml(+ days[0].date ++ " to " ++ days[days.length - 1].date ++ ", peak " ++ max ++ "/day",+ ) ++ "</div>",+ );+ }+ const snaps = counts.report_snapshots || {};+ if (snaps.status === "ok") {+ const surfaces = snaps.surfaces || [];+ parts.push(+ fieldRow(+ "report snapshots",+ surfaces.length === 0+ ? "enabled, none yet"+ : surfaces+ .map((s) => escHtml(s.surface + ": " + (s.latest_date || "none")))+ .join(", "),+ ),+ );+ } else {+ parts.push(fieldRow("report snapshots", "disabled"));+ }+ document.getElementById("counts-body").innerHTML = parts.join("");+ }++ function renderOps(snap) {+ const parts = [];+ parts.push(+ '<p><span class="pill ' ++ (snap.healthy ? "confirmed" : "quarantine") ++ '">' ++ escHtml(snap.healthy ? "all clear" : snap.problems.length + " problem(s)") ++ "</span></p>",+ );+ parts.push(+ fieldRow(+ "counts",+ escHtml(+ "preferences " ++ snap.counts.preferences ++ ", retired " ++ snap.counts.retired ++ ", inbox " ++ snap.counts.inbox,+ ),+ ),+ );+ parts.push(fieldRow("stale / review", escHtml(snap.staleTotal + " / " + snap.reviewQueue)));+ parts.push(fieldRow("active profile", escHtml(snap.activeProfile || "none")));+ parts.push(+ fieldRow(+ "state files",+ escHtml(+ "config " ++ (snap.stateFiles.config ? "ok" : "MISSING") ++ ", search index " ++ (snap.stateFiles.searchIndex ? "present" : "absent"),+ ),+ ),+ );+ parts.push(fieldRow("health", escHtml(snap.healthVerdict)));+ if (snap.frozen) {+ parts.push(fieldRow("frozen", "yes - " + escHtml(snap.frozen.frozen_at)));+ }+ const problems = snap.problems || [];+ if (problems.length > 0) {+ parts.push('<ul class="edge-list">');+ for (const p of problems) {+ parts.push(+ "<li>" ++ escHtml("[" + p.code + "] " + p.detail) ++ '<br /><span class="kind">next: ' ++ escHtml(p.nextCommand) ++ "</span></li>",+ );+ }+ parts.push("</ul>");+ }+ document.getElementById("ops-body").innerHTML = parts.join("");+ }++ function loadConsole() {+ if (!VAULT_PATH) {+ renderConsoleOffline("counts-body");+ renderConsoleOffline("ops-body");+ return;+ }+ fetchJson("/counts.json")+ .then(renderCounts)+ .catch((err) => renderLoadError("counts-body", "/counts.json", err));+ fetchJson("/status.json")+ .then(renderOps)+ .catch((err) => renderLoadError("ops-body", "/status.json", err));+ }+// ---- Boot ------------------------------------------------------------resizeCanvas();placeInitial();applyFilters();+ loadConsole();// Persist the layout once after each settle. `unsavedMotion`// latches true whenever any visible node has non-trivial velocitydiff --git a/tests/cli/brain-explorer.test.ts b/tests/cli/brain-explorer.test.tsindex b1166392..095e5128 100644--- a/tests/cli/brain-explorer.test.ts+++ b/tests/cli/brain-explorer.test.ts@@ -9,7 +9,7 @@import { afterEach, beforeEach, describe, expect, test } from "bun:test";import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";-import { join } from "node:path";+import { dirname, join } from "node:path";import { writePreference } from "../../src/core/brain/preference.ts";import { REDACTION_PLACEHOLDER } from "../../src/core/redactor.ts";@@ -126,6 +126,12 @@ describe("o2b brain explorer --export", () => {const parsed = JSON.parse(match![1]!);expect(parsed.nodes.length).toBe(1);expect(parsed.nodes[0].id).toBe("pref-explorer-seed");+ // The console sections travel with the template; on a static export+ // they state that the live routes are unreachable instead of firing+ // fetches that can only fail against a file path.+ expect(body).toContain('id="counts-body"');+ expect(body).toContain('id="ops-body"');+ expect(body).toContain("static export");});test("the exported graph goes through the shared egress guard", async () => {@@ -216,7 +222,7 @@ describe("o2b brain explorer --export", () => {describe("o2b brain explorer (live)", () => {test(- "binds to 127.0.0.1, serves / and /data.json, shuts down on SIGINT",+ "binds to 127.0.0.1, serves all four routes, shuts down on SIGINT",async () => {await bootstrap();seedOnePref();@@ -265,6 +271,48 @@ describe("o2b brain explorer (live)", () => {expect(mainBody.includes("__GRAPH_JSON__")).toBe(false);const parsed = JSON.parse(dataBody);expect(parsed.nodes.length).toBe(1);+ // The counts and ops sections fetch the two console routes+ // client-side, in the page's own vanilla JS; the substituted+ // graph payload remains the page's single inline JSON block.+ expect(mainBody).toContain('id="counts-body"');+ expect(mainBody).toContain('id="ops-body"');+ expect(mainBody).toContain('fetchJson("/counts.json")');+ expect(mainBody).toContain('fetchJson("/status.json")');+ expect(mainBody.split('<script type="application/json" id="brain-data">').length).toBe(2);+ // The live-counts console routes (session-mining-economy): both+ // are loopback JSON re-read from the vault per GET.+ const countsRes = await fetch(`http://127.0.0.1:${port}/counts.json`);+ expect(countsRes.ok).toBe(true);+ const counts = JSON.parse(await countsRes.text()) as {+ schema_version: number;+ telemetry: { total: number };+ written_per_day: ReadonlyArray<unknown>;+ report_snapshots: unknown;+ };+ expect(counts.schema_version).toBe(1);+ expect(counts.telemetry.total).toBe(0);+ expect(counts.written_per_day.length).toBe(30);+ expect(counts.report_snapshots).toEqual({ status: "disabled" });+ const statusRes = await fetch(`http://127.0.0.1:${port}/status.json`);+ expect(statusRes.ok).toBe(true);+ const status = JSON.parse(await statusRes.text()) as {+ healthy: boolean;+ counts: { preferences: number };+ activeProfile: string | null;+ };+ expect(typeof status.healthy).toBe("boolean");+ expect(status.counts.preferences).toBe(1);+ // The verb threads its resolved config path into the server, so+ // the operator snapshot sees this run's profile registry.+ writeFileSync(+ join(dirname(config), "profiles.json"),+ JSON.stringify({ active: "test", profiles: { test: { vault } } }),+ );+ const profiled = await fetch(`http://127.0.0.1:${port}/status.json`);+ const profileStatus = JSON.parse(await profiled.text()) as {+ activeProfile: string | null;+ };+ expect(profileStatus.activeProfile).toBe("test");} finally {// Always tear the spawned CLI down so a failing assertion does// not leak the port into the next test on the same worker.diff --git a/tests/cli/brain-extract-signals.test.ts b/tests/cli/brain-extract-signals.test.tsindex 77adef6b..5be0bf1f 100644--- a/tests/cli/brain-extract-signals.test.ts+++ b/tests/cli/brain-extract-signals.test.ts@@ -9,14 +9,20 @@* `ok: false` - never a stack trace and never a zero exit.* 4. A missing session reference is a usage error (exit 2).* 5. The verb has its own `--help`, not a dump of the whole brain help.+ * 6. A commit that lands nothing tombstones the session, and the next+ * plan names it with the recorded breakdown; --force bypasses the+ * tombstone for exactly one plan call.+ * 7. A ledger write failure on a successful commit exits 0 and names the+ * failure, in --json and in text.*/import { afterEach, beforeEach, describe, expect, test } from "bun:test";-import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";-import { join } from "node:path";+import { dirname, join } from "node:path";import { AUTO_EXTRACT_PER_SESSION_CAP } from "../../src/core/brain/extract-signals.ts";+import { sessionMiningLedgerPath } from "../../src/core/brain/session-mining-ledger.ts";import { importSessionRecall } from "../../src/core/brain/session-recall.ts";import { runCli } from "../helpers/run-cli.ts";@@ -187,3 +193,205 @@ describe("think-block strip (t_dac8bf7e)", () => {expect("note" in payload).toBe(false);});});++// ----- The zero-yield mining tombstone on the surface ------------------------+//+// 6. A commit that lands nothing tombstones the session - keyed on the+// surveyed hash the plan printed, which the commit relays back - and+// the next plan says so by name with the recorded breakdown, in --json+// and in text, instead of re-offering the same turns. A commit that+// answers no plan records nothing, so the session stays eligible.+// 7. --force bypasses the tombstone for exactly one plan call: the forced+// plan carries the envelope, and the plan after it is suppressed again.+// 8. A ledger write failure on a successful commit exits 0 and names the+// failure in --json and in text - never a throw, never a silent drop.++function corruptMiningLedger(): void {+ mkdirSync(dirname(sessionMiningLedgerPath(vault)), { recursive: true });+ writeFileSync(sessionMiningLedgerPath(vault), "{ truncated");+}++/** The plan's surveyed hash, from the --json plan phase. */+async function surveyedHash(): Promise<string> {+ const r = await run([SESSION, "--json"]);+ expect(r.returncode).toBe(0);+ return (JSON.parse(r.stdout) as { surveyed_turns_hash: string }).surveyed_turns_hash;+}++describe("mining tombstone", () => {+ test("a barren commit tombstones the session and the next plan names it", async () => {+ const hash = await surveyedHash();+ const mine = await run([+ SESSION,+ "--payload",+ JSON.stringify({ items: [] }),+ "--surveyed-turns-hash",+ hash,+ "--json",+ ]);+ expect(mine.returncode).toBe(0);+ const r = await run([SESSION, "--json"]);+ expect(r.returncode).toBe(0);+ const payload = JSON.parse(r.stdout) as {+ ok: boolean;+ llm_step: unknown;+ turns_mined: unknown[];+ skipped?: {+ reason: string;+ mining_outcome?: {+ written: number;+ staged: number;+ deduped: number;+ durability_rejected: number;+ };+ };+ };+ expect(payload.ok).toBe(true);+ expect(payload.skipped?.reason).toBe("mining_tombstone");+ expect(payload.skipped?.mining_outcome).toEqual({+ written: 0,+ staged: 0,+ deduped: 0,+ durability_rejected: 0,+ });+ expect(payload.llm_step).toBeNull();+ expect(payload.turns_mined).toEqual([]);+ });++ test("a barren commit without the surveyed hash records nothing and fails open", async () => {+ // The relay is what makes the tombstone honest: without it the commit+ // has no material to key on, records nothing, and the session stays+ // eligible - never suppressed on unverified evidence.+ const mine = await run([SESSION, "--payload", JSON.stringify({ items: [] }), "--json"]);+ expect(mine.returncode).toBe(0);+ const r = await run([SESSION, "--json"]);+ const payload = JSON.parse(r.stdout) as { skipped?: unknown };+ expect(payload.skipped).toBeUndefined();+ });++ test("text mode names the tombstone and the force escape", async () => {+ const hash = await surveyedHash();+ await run([SESSION, "--payload", JSON.stringify({ items: [] }), "--surveyed-turns-hash", hash]);+ const r = await run([SESSION]);+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain("already mined");+ expect(r.stdout).toContain("--force");+ // The evidence rides the line too, not just the verdict.+ expect(r.stdout).toMatch(/deduped 0/);+ });++ test("the text line renders the recorded breakdown, field for field", async () => {+ // A first yield, then a re-mine of the same payload where the only+ // item dedups: the recorded outcome is written 0, staged 0, deduped 1,+ // and the suppressed plan's text line must read exactly that - the+ // same numbers the --json surface prints for the same record.+ const hash = await surveyedHash();+ await run([+ SESSION,+ "--payload",+ JSON.stringify({ items: [ITEM] }),+ "--surveyed-turns-hash",+ hash,+ "--json",+ ]);+ await run([+ SESSION,+ "--payload",+ JSON.stringify({ items: [ITEM] }),+ "--surveyed-turns-hash",+ hash,+ "--json",+ ]);+ const r = await run([SESSION]);+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain(+ "yielded nothing (written 0, staged 0, deduped 1, durability-rejected 0)",+ );+ });++ test("--force bypasses the tombstone for exactly one plan call", async () => {+ const hash = await surveyedHash();+ await run([+ SESSION,+ "--payload",+ JSON.stringify({ items: [] }),+ "--surveyed-turns-hash",+ hash,+ "--json",+ ]);+ const forced = await run([SESSION, "--force", "--json"]);+ expect(forced.returncode).toBe(0);+ const forcedPayload = JSON.parse(forced.stdout) as {+ llm_step: { status: string } | null;+ skipped?: unknown;+ };+ expect(forcedPayload.skipped).toBeUndefined();+ expect(forcedPayload.llm_step?.status).toBe("needs-llm-step");+ // The bypass is per call: the next unforced plan is suppressed again.+ const after = await run([SESSION, "--json"]);+ const afterPayload = JSON.parse(after.stdout) as { skipped?: { reason: string } };+ expect(afterPayload.skipped?.reason).toBe("mining_tombstone");+ });++ test("a yielding commit records its outcome and the session stays minable", async () => {+ // The distilled state rides the same record the tombstone does: the+ // yield is the census's evidence, not a suppression, so the next plan+ // still offers the session.+ const hash = await surveyedHash();+ const mine = await run([+ SESSION,+ "--payload",+ JSON.stringify({ items: [ITEM] }),+ "--surveyed-turns-hash",+ hash,+ "--json",+ ]);+ expect(mine.returncode).toBe(0);+ const ledger = JSON.parse(readFileSync(sessionMiningLedgerPath(vault), "utf8")) as {+ entries: Record<+ string,+ {+ outcome: { written: number; staged: number; deduped: number; durabilityRejected: number };+ }+ >;+ };+ expect(ledger.entries[SESSION]?.outcome).toEqual({+ written: 1,+ staged: 0,+ deduped: 0,+ durabilityRejected: 0,+ });+ const r = await run([SESSION, "--json"]);+ expect(r.returncode).toBe(0);+ expect((JSON.parse(r.stdout) as { skipped?: unknown }).skipped).toBeUndefined();+ });++ test("a ledger write failure on a successful commit exits 0 and names the failure", async () => {+ const hash = await surveyedHash();+ corruptMiningLedger();+ const mine = await run([+ SESSION,+ "--payload",+ JSON.stringify({ items: [] }),+ "--surveyed-turns-hash",+ hash,+ "--json",+ ]);+ expect(mine.returncode).toBe(0);+ const payload = JSON.parse(mine.stdout) as { ok: boolean; mining_ledger_failure?: string };+ expect(payload.ok).toBe(true);+ expect(payload.mining_ledger_failure).toContain("session mining ledger is corrupted JSON");++ // Text mode names it too, on the same exit code (this commit relays+ // the hash as well, so the record attempt happens here too).+ const text = await run([+ SESSION,+ "--payload",+ JSON.stringify({ items: [] }),+ "--surveyed-turns-hash",+ hash,+ ]);+ expect(text.returncode).toBe(0);+ expect(text.stdout).toContain("mining ledger");+ });+});diff --git a/tests/cli/brain/verbs/forget-source-failed.test.ts b/tests/cli/brain/verbs/forget-source-failed.test.tsnew file mode 100644index 00000000..8a99ea7a--- /dev/null+++ b/tests/cli/brain/verbs/forget-source-failed.test.ts@@ -0,0 +1,111 @@+/**+ * The `forget-source` front door names what did not delete (storage+ * lifecycle, PR #257 review M8). The core `deleteBySource` result has+ * carried `failed: [{path, message}]` since the per-entry failure+ * collection landed, but this verb left it out of both the JSON payload+ * and the text report and exited 0 - a confirmed removal that could not+ * delete three files looked like a clean run with a smaller count. The+ * confirmed run now carries `failed` in the JSON, prints the blocked+ * paths in text mode, and exits 1 when any failure was collected.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { chmodSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join } from "node:path";++import { cmdBrainForgetSource } from "../../../../src/cli/brain/verbs/forget-source.ts";+import { bootstrapBrain } from "../../../../src/core/brain/init.ts";+import { ingestSource } from "../../../../src/core/brain/ingest/ingest.ts";+import { resetVaultIdentityPins } from "../../../../src/core/brain/vault-identity.ts";+import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";+import { CHMOD_CANNOT_DENY } from "../../../helpers/platform.ts";++let vault: string;+let configHome: string;+let savedConfigEnv: string | undefined;++const SOURCE = "imports/benchmark.md";+const NOW = new Date("2026-06-01T00:00:00Z");++beforeEach(() => {+ resetVaultIdentityPins();+ vault = mkdtempSync(join(tmpdir(), "o2b-cli-forget-source-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-cli-forget-source-cfg-"));+ atomicWriteFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+ savedConfigEnv = process.env["OPEN_SECOND_BRAIN_CONFIG"];+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = join(configHome, "config.yaml");+ mkdirSync(join(vault, "imports"), { recursive: true });+ writeFileSync(join(vault, SOURCE), "benchmark rows\n", "utf8");+ ingestSource(+ vault,+ {+ sourcePath: SOURCE,+ summary: "Benchmark import summary.",+ extraction: { entities: [{ category: "concept", name: "BenchWidget" }] },+ },+ { agent: "tester", now: NOW },+ );+});++afterEach(() => {+ if (savedConfigEnv === undefined) delete process.env["OPEN_SECOND_BRAIN_CONFIG"];+ else process.env["OPEN_SECOND_BRAIN_CONFIG"] = savedConfigEnv;+ resetVaultIdentityPins();+ rmSync(vault, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+ rmSync(configHome, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });+});++/** Run the verb in-process, capturing stdout and the returned exit code. */+async function captureVerb(argv: string[]): Promise<{ code: number; stdout: string }> {+ const chunks: string[] = [];+ const originalWrite = process.stdout.write.bind(process.stdout);+ process.stdout.write = ((chunk: string | Uint8Array) => {+ chunks.push(typeof chunk === "string" ? chunk : Buffer.from(chunk).toString("utf8"));+ return true;+ }) as typeof process.stdout.write;+ try {+ const code = await cmdBrainForgetSource(argv);+ return { code, stdout: chunks.join("") };+ } finally {+ process.stdout.write = originalWrite;+ }+}++describe("forget-source carries the core failed list to the front door", () => {+ test("a confirmed run with a blocked removal exits 1 and names the path in JSON", async () => {+ if (CHMOD_CANNOT_DENY) return;+ // The summary page sits in Brain/sources; a read-only directory makes+ // exactly that removal fail while the run completes.+ const preview = await captureVerb([SOURCE, "--vault", vault, "--json"]);+ const previewPayload = JSON.parse(preview.stdout) as {+ derived: Array<{ path: string; kind: string }>;+ };+ const blockedPath = previewPayload.derived.find((e) => e.kind === "summary")!.path;+ const blockedDir = join(vault, dirname(blockedPath));+ chmodSync(blockedDir, 0o555);+ try {+ const run = await captureVerb([SOURCE, "--vault", vault, "--confirm", "--json"]);+ expect(run.code).toBe(1);+ const payload = JSON.parse(run.stdout) as {+ changed: number;+ failed: ReadonlyArray<{ path: string; message: string }>;+ };+ expect(payload.failed.length).toBe(1);+ expect(payload.failed[0]!.path).toBe(blockedPath);+ expect(payload.failed[0]!.message.length).toBeGreaterThan(0);+ expect(payload.changed).toBeGreaterThan(0);+ expect(existsSync(join(vault, blockedPath))).toBe(true);+ } finally {+ chmodSync(blockedDir, 0o755);+ }+ });++ test("a clean confirmed run exits 0 and names no failure", async () => {+ const run = await captureVerb([SOURCE, "--vault", vault, "--confirm", "--json"]);+ expect(run.code).toBe(0);+ const payload = JSON.parse(run.stdout) as { failed?: unknown };+ expect(payload.failed).toEqual([]);+ });+});diff --git a/tests/cli/brain/verbs/page-dedup-apply-fail-soft.test.ts b/tests/cli/brain/verbs/page-dedup-apply-fail-soft.test.tsindex 01a96840..94018bd9 100644--- a/tests/cli/brain/verbs/page-dedup-apply-fail-soft.test.ts+++ b/tests/cli/brain/verbs/page-dedup-apply-fail-soft.test.ts@@ -100,9 +100,12 @@ describe("applyDedupCandidates", () => {const applied = applyDedupCandidates(vault, report.candidates);- // The failure is named by cluster key and message.+ // The failure is named by cluster key, SECONDARY (review L8 - a cluster+ // can hold several secondaries, and the report must say which one+ // refused), and message.expect(applied.failures.length).toBe(1);expect(applied.failures[0]!.key).toBe(first.key);+ expect(applied.failures[0]!.secondary).toBe(sabotaged.id);expect(applied.failures[0]!.message).toContain(`secondary page not found: ${sabotaged.id}`);// The later cluster still applied.@@ -175,10 +178,12 @@ describe("cmdBrainPageDedup --apply with a failing cluster", () => {expect(exit.code).toBe(1);const payload = JSON.parse(exit.stdout) as {merged: number;- failures: Array<{ key: string; message: string }>;+ failures: Array<{ key: string; secondary?: string; message: string }>;};expect(payload.failures.length).toBe(1);expect(payload.failures[0]!.message).toContain("EACCES");+ // The wire names WHICH secondary refused (review L8).+ expect(payload.failures[0]!.secondary).toBe("pref-b");expect(payload.merged).toBe(1);// The later cluster still applied, in its writable directory.expect(readFileSync(join(vault, "Brain", "retired", "ret-d.md"), "utf8")).toContain(diff --git a/tests/cli/brain/verbs/receipts.test.ts b/tests/cli/brain/verbs/receipts.test.tsindex 0cebca0a..47035d87 100644--- a/tests/cli/brain/verbs/receipts.test.ts+++ b/tests/cli/brain/verbs/receipts.test.ts@@ -17,6 +17,7 @@ import { join } from "node:path";import {continuityLogDir,+ continuityLogPath,listContinuityRecords,} from "../../../../src/core/brain/continuity/store.ts";import { metricsDir } from "../../../../src/core/brain/metrics.ts";@@ -190,6 +191,40 @@ describe("o2b brain receipts", () => {expect(unknownFamily.stderr).toContain("unknown ledger family");});+ test("a confirmed run's text report names the net byte effect beside the recovery point", async () => {+ // M6 (confirm half): the recovery point is a whole-Brain archive in a+ // surface the storage census counts, so the report shows the ledger+ // shrink beside what the archive added - the net figure an operator+ // at a capacity tier decides on.+ seedLedger();+ const out = await runCli(+ ["brain", "receipts", "compact", "--shard", `continuity:${LEDGER_MONTH}.jsonl`, "--apply"],+ { env: env() },+ );+ expect(out.returncode).toBe(0);+ expect(out.stdout).toContain("recovery point:");+ expect(out.stdout).toMatch(/net: -?\d+ bytes \(ledger shrank \d+, recovery point added \d+\)/);+ });++ test("a confirmed run whose audit record cannot be appended fails loudly in text mode", async () => {+ // L10: the audit outcome rode only the JSON view, so a text-mode+ // `--apply` whose receipt_compaction record did not land exited 0+ // with no trace of the failure. A directory parked at the run month's+ // shard path makes the audit append fail after the shards are+ // durable - the run must print the named outcome and exit 1.+ seedLedger();+ const runMonth = new Date().toISOString().slice(0, 7);+ mkdirSync(continuityLogPath(vault, runMonth), { recursive: true });+ const out = await runCli(+ ["brain", "receipts", "compact", "--shard", `continuity:${LEDGER_MONTH}.jsonl`, "--apply"],+ { env: env() },+ );+ expect(out.returncode).toBe(1);+ expect(out.stdout).toContain("receipt_compaction audit record could not be appended");+ // The fold itself landed - the failure is only the audit trail's.+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);+ });+test("the surface vocabulary derives from one source on both surfaces", () => {// The verb's usage line and the tool's mode enum come from the same// closed tuple, so neither surface can rename a mode alone.diff --git a/tests/cli/brain/verbs/storage.test.ts b/tests/cli/brain/verbs/storage.test.tsindex cfb61f24..43b0eaa5 100644--- a/tests/cli/brain/verbs/storage.test.ts+++ b/tests/cli/brain/verbs/storage.test.ts@@ -197,16 +197,18 @@ describe("o2b brain storage", () => {// census between the measure and the verb's own.configureCap(1_000_000);const total = measureStorageUsage(vault).totalBytes;- // Cap at 2.2x the stable total: the fresh reading sits around 45- // percent (never near the 80/95 percent bands), so the tier comes- // from the horizon. The two stored samples rise a tenth of the total- // per day and the verb's own sample continues that rise, so the cap- // lands inside the warn horizon - past critical's 7 days, inside- // warn's 30.- const cap = Math.floor(total * 2.2);+ // Cap at twice the stable total: the fresh reading sits around half+ // the cap (never near the 80/95 percent bands), so the tier comes+ // from the horizon. A day-based trend needs three distinct days and+ // the verb's own fresh census supersedes the newest seeded day inside+ // the fit, which flattens the seeded rise to a fraction of the daily+ // step; twice the cap keeps the resulting horizon a few days inside+ // warn's 30 - and past critical's 7 - on every machine.+ const cap = Math.floor(total * 2.0);configureCap(cap);seedSample("2026-10-14T00:00:00.000Z", total - total / 10, "seed-older");seedSample("2026-10-15T00:00:00.000Z", total, "seed-newer");+ seedSample("2026-10-16T00:00:00.000Z", total + total / 20, "seed-newest");const { code, report } = await runJsonReport();diff --git a/tests/cli/distill-trust-lane.test.ts b/tests/cli/distill-trust-lane.test.tsindex 13fde169..b1db6204 100644--- a/tests/cli/distill-trust-lane.test.ts+++ b/tests/cli/distill-trust-lane.test.ts@@ -22,12 +22,14 @@* flag-gated, so its guarantee holds by default. A distillation page has* only the flag-gated retrieval trust gate, so the same words mean less.*- * These tests pin all three facts at once: the default resolves the gate- * off, an ordinary search DOES return the untrusted page with that- * default, and the same search stops returning it once the setting is on.- * The description is then required to name that setting - so if the- * default is ever flipped to true, the last test fails and the wording- * gets revisited rather than silently becoming understated.+ * These tests pin the facts at once: the default resolves the gate off,+ * an ordinary search DOES return the untrusted page with that default,+ * the legacy boolean alone now mounts the gate in shadow (the page still+ * returns, the refusal is counted), and the same search stops returning+ * it once the gate is named enforcing. The description is then required+ * to name that setting - so if the default is ever flipped, the last+ * test fails and the wording gets revisited rather than silently+ * becoming understated.*/import { afterEach, beforeEach, describe, expect, test } from "bun:test";@@ -112,9 +114,28 @@ describe("the quarantine is real, and it is opt-in", () => {expect(paths.some((p) => p.endsWith(path) || path.endsWith(p))).toBe(true);});- test("with the gate enabled, the same search stops returning it", async () => {+ test("with the legacy boolean alone, the search still returns it - the gate measures in shadow", async () => {+ // Session-mining-economy: the legacy boolean alone maps to the+ // measure-only mode, so the page surfaces and the refusal is counted.const path = await seedDistillation(`${TRUST_GATE_CONFIG_KEY}: true\n`);const paths = await queryPaths();+ expect(paths.some((p) => p.endsWith(path) || path.endsWith(p))).toBe(true);+ const explain = await runCli(["search", CLAIM_TERM, "--json", "--explain", "--limit", "20"], {+ env: env(),+ });+ expect(explain.returncode).toBe(0);+ const payload = JSON.parse(explain.stdout) as {+ retrieval_decision_trace?: { readonly mode?: string; readonly refused?: number };+ };+ expect(payload.retrieval_decision_trace?.mode).toBe("shadow");+ expect(payload.retrieval_decision_trace?.refused).toBeGreaterThanOrEqual(1);+ });++ test("with the gate enforcing, the same search stops returning it", async () => {+ const path = await seedDistillation(+ `${TRUST_GATE_CONFIG_KEY}: true\nsearch_trust_gate_mode: enforce\n`,+ );+ const paths = await queryPaths();expect(paths.some((p) => p.endsWith(path) || path.endsWith(p))).toBe(false);});});diff --git a/tests/cli/search-explain-trace.test.ts b/tests/cli/search-explain-trace.test.tsindex 019a44c5..2a3c5112 100644--- a/tests/cli/search-explain-trace.test.ts+++ b/tests/cli/search-explain-trace.test.ts@@ -21,6 +21,10 @@ import {RETRIEVAL_TRACE_UNAVAILABLE,type ExplainRequest,} from "../../src/core/search/explain-envelope.ts";+import {+ RETRIEVAL_TRUST_REFUSAL_LANE,+ type RetrievalTrustRefusal,+} from "../../src/core/brain/trust/retrieval-receipts.ts";import type { BrainSearchResult, SearchOutcome } from "../../src/core/search/types.ts";import { runCli } from "../helpers/run-cli.ts";@@ -167,6 +171,90 @@ describe("o2b search --explain: retrieval receipts", () => {const unavailable = renderOutcomeHuman(outcome(false), false, QUERY, EXPLAIN_ON);expect(unavailable).toContain(RETRIEVAL_TRACE_UNAVAILABLE);});++ test("human: a shadow run names the mode and its measured refusals per lane", () => {+ // Shadow's whole purpose is that the would-be refusals are visible to+ // the operator deciding whether to promote the mode; the human+ // transcript is that operator's default surface. With nothing+ // excluded, the enforce-shaped lead line alone would read exactly+ // like a clean run.+ const poolRow: RetrievalTrustRefusal = Object.freeze({+ document_id: 2,+ chunk_id: 2,+ path: "notes/quarantined.md",+ reasons: Object.freeze(["trust_gate:self_approval_quarantine"]),+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.pool,+ });+ const armRow: RetrievalTrustRefusal = Object.freeze({+ document_id: 6,+ chunk_id: 6,+ path: "notes/arm-reached.md",+ reasons: Object.freeze(["relational_arm:entity_contamination"]),+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.arm,+ });+ const shadowTrace = {+ ...TRACE,+ excluded: 0,+ exclusions: [],+ mode: "shadow" as const,+ refused: 2,+ refusals: [poolRow, armRow],+ };+ const shadowAssessment = {+ ...ASSESSMENT,+ quarantined: 0,+ mode: "shadow" as const,+ refused: 2,+ refusals_by_lane: { arm: 1, pool: 1 },+ refused_reason_counts: {+ arm: { "relational_arm:entity_contamination": 1 },+ pool: { "trust_gate:self_approval_quarantine": 1 },+ },+ };+ const o: SearchOutcome = Object.freeze({+ results: Object.freeze([RESULT]),+ warnings: Object.freeze([]),+ total: 2,+ idfWeightedCoverage: 1,+ retrievalDecisionTrace: shadowTrace,+ memoryTrustAssessment: shadowAssessment,+ });+ const rendered = renderOutcomeHuman(o, false, QUERY, EXPLAIN_ON);++ // The mode, and the refusal counts split per lane, on the lead line.+ expect(rendered).toContain("mode shadow");+ expect(rendered).toContain("refused 2 (arm 1, pool 1)");+ // Each refusal as a row, the way exclusions print.+ expect(rendered).toContain("refused: notes/quarantined.md");+ expect(rendered).toContain("refused: notes/arm-reached.md");+ expect(rendered).toContain("relational_arm:entity_contamination");++ // An enforce run with an arm drop renders the same refusal rows but+ // no mode stamp.+ const enforceTrace = {+ ...TRACE,+ refused: 1,+ refusals: [armRow],+ };+ const enforceAssessment = {+ ...ASSESSMENT,+ refused: 1,+ refusals_by_lane: { arm: 1, pool: 0 },+ refused_reason_counts: { arm: { "relational_arm:entity_contamination": 1 }, pool: {} },+ };+ const enforceOutcome: SearchOutcome = Object.freeze({+ results: Object.freeze([RESULT]),+ warnings: Object.freeze([]),+ total: 2,+ idfWeightedCoverage: 1,+ retrievalDecisionTrace: enforceTrace,+ memoryTrustAssessment: enforceAssessment,+ });+ const enforceRendered = renderOutcomeHuman(enforceOutcome, false, QUERY, EXPLAIN_ON);+ expect(enforceRendered).toContain("refused 1 (arm 1, pool 0)");+ expect(enforceRendered).toContain("refused: notes/arm-reached.md");+ expect(enforceRendered).not.toContain("mode");+ });});describe("o2b search --explain end to end", () => {@@ -188,7 +276,12 @@ describe("o2b search --explain end to end", () => {"---\ntitle: Bad\nstatus: quarantine\n---\n\n# Bad\n\nThe widget calibration is unsafe.\n",);const configPath = join(configHome, "config.yaml");- atomicWriteFileSync(configPath, `vault: ${vault}\nsearch_trust_gate_enabled: true\n`);+ // The gate must EXCLUDE here, and the legacy boolean alone now+ // measures (shadow): enforcement is named explicitly.+ atomicWriteFileSync(+ configPath,+ `vault: ${vault}\nsearch_trust_gate_enabled: true\nsearch_trust_gate_mode: enforce\n`,+ );const env = { OPEN_SECOND_BRAIN_CONFIG: configPath };const indexed = await runCli(["search", "index"], { env });@@ -224,4 +317,34 @@ describe("o2b search --explain end to end", () => {rmSync(configHome, { recursive: true, force: true });}});++ test("a cross-vault union under an explicit mode does not claim the gate is off", async () => {+ // The mode alone runs the gate - the legacy boolean stays unset - so+ // the union's unavailable reason must diagnose ONLY the union: the+ // gate-off half of the combined reason would be a false diagnosis+ // here, and its enablement advice a dead end.+ const vault = mkdtempSync(join(tmpdir(), "o2b-cli-trace-mode-vault-"));+ const configHome = mkdtempSync(join(tmpdir(), "o2b-cli-trace-mode-cfg-"));+ try {+ mkdirSync(join(vault, "notes"), { recursive: true });+ writeFileSync(+ join(vault, "notes", "clean.md"),+ "---\ntitle: Clean\n---\n\n# Clean\n\nThe widget calibration routine runs daily.\n",+ );+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nsearch_trust_gate_mode: enforce\n`);+ const env = { OPEN_SECOND_BRAIN_CONFIG: configPath };++ const indexed = await runCli(["search", "index"], { env });+ expect(indexed.returncode).toBe(0);++ const explained = await runCli(["search", QUERY, "--json", "--global", "--explain"], { env });+ expect(explained.returncode).toBe(0);+ const payload = JSON.parse(explained.stdout) as { retrieval_trace_unavailable?: string };+ expect(payload.retrieval_trace_unavailable).toBe(RETRIEVAL_TRACE_NOT_MERGED);+ } finally {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+ }+ });});diff --git a/tests/cli/session-discovery.test.ts b/tests/cli/session-discovery.test.tsindex d6634e08..b1d740c9 100644--- a/tests/cli/session-discovery.test.ts+++ b/tests/cli/session-discovery.test.ts@@ -54,15 +54,53 @@ interface CoverageRow {readonly roots: ReadonlyArray<{ readonly path: string; readonly present: boolean }>;}+interface MiningRowJson {+ readonly runtime: string;+ readonly distilled: number;+ readonly known_nothing_new: number;+ readonly undistilled: number;+ readonly refused: number;+}++interface MiningCensusJson {+ readonly imported: number;+ readonly distilled: number;+ readonly known_nothing_new: number;+ readonly undistilled: number;+ readonly refused: number;+ readonly by_runtime: ReadonlyArray<MiningRowJson>;+}+interface DiscoveryJson {readonly found: number;readonly imported: number;readonly gap: number;readonly by_runtime: ReadonlyArray<CoverageRow>;+ readonly mining?: MiningCensusJson;readonly would_import?: ReadonlyArray<string>;readonly files?: ReadonlyArray<{ readonly file: string }>;}+interface ProposalJson {+ readonly dedup_hash: string;+ readonly topic: string;+ readonly signal: string;+ readonly principle: string;+ readonly scope?: string;+ readonly sessions: ReadonlyArray<string>;+}++interface ProposeBacklogJson {+ readonly undistilled: number;+ readonly proposals: ReadonlyArray<ProposalJson>;+ readonly dropped: ReadonlyArray<{+ readonly session_id: string;+ readonly topic: string;+ readonly dedup_hash: string;+ }>;+ readonly report_only_sessions: ReadonlyArray<string>;+}+/*** A credential in the shape the redactor's `key: value` pass catches, so* the byte-identity claim below is about a fixture that actually carries@@ -141,6 +179,36 @@ function claudeLog(name: string, topic: string): string {return path;}+/**+ * A log whose marker carries no credential-shaped value. The parity+ * fixture above deliberately carries one, but a redacted value mangles+ * the quoted principle (the redactor rewrites the quoted value that+ * follows the credential key, so the principle no longer re-parses from+ * the imported turns and raises no backlog candidate), so the batching+ * tests below use clean markers.+ */+function claudeMarkerLog(name: string, topic: string): string {+ const dir = join(home, ".claude", "projects", "-srv-projects-example");+ mkdirSync(dir, { recursive: true });+ const path = join(dir, name);+ writeFileSync(+ path,+ JSON.stringify({+ parentUuid: null,+ sessionId: "s",+ entrypoint: "sdk-cli",+ type: "user",+ message: {+ role: "user",+ content: `@osb feedback positive topic=${topic} principle="Declare the roots exactly once."`,+ },+ uuid: `u-${topic}`,+ timestamp: "2026-08-16T09:00:00.000Z",+ }) + "\n",+ );+ return path;+}+/** Signal files in the inbox, ignoring the `processed/` subdirectory. */function inboxSignals(at: string): ReadonlyArray<string> {return readdirSync(brainDirs(at).inbox)@@ -312,6 +380,136 @@ describe("--status reports coverage without touching anything", () => {});});+describe("the mining census rides the coverage report", () => {+ test("a recall import with nothing mined yet is undistilled backlog", async () => {+ claudeLog("one.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--recall", "--json"])).returncode).toBe(0);++ const status = JSON.parse((await run(["--status", "--json"])).stdout) as DiscoveryJson;+ const mining = status.mining;+ expect(mining).toBeDefined();+ expect(mining?.imported).toBe(1);+ expect(mining?.undistilled).toBe(1);+ expect(mining?.distilled).toBe(0);+ expect(mining?.known_nothing_new).toBe(0);+ expect(mining?.refused).toBe(0);+ const claude = mining?.by_runtime.find((r) => r.runtime === "claude-code");+ expect(claude?.undistilled).toBe(1);+ });++ test("an import without recall never reaches the backlog, because the plan would refuse it", async () => {+ // Turns reach continuity only through --recall; without them the mining+ // plan refuses the session (zero imported turns), and the census+ // excludes what the plan refuses - it can never be distilled.+ claudeLog("one.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--json"])).returncode).toBe(0);++ const status = JSON.parse((await run(["--status", "--json"])).stdout) as DiscoveryJson;+ expect(status.mining?.imported).toBe(1);+ expect(status.mining?.refused).toBe(1);+ expect(status.mining?.undistilled).toBe(0);+ });++ test("the human rendering carries the mining lines per runtime", async () => {+ claudeLog("one.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--recall"])).returncode).toBe(0);++ const res = await run(["--status"]);+ expect(res.returncode).toBe(0);+ expect(res.stdout).toContain("mining: 0 distilled, 0 known-nothing-new, 1 undistilled");+ expect(res.stdout).toContain(" undistilled: 1");+ });+});++describe("--propose-backlog batches the distill proposals ask-once", () => {+ test("material the vault no longer holds is proposed, naming its session", async () => {+ claudeMarkerLog("one.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--recall", "--json"])).returncode).toBe(0);+ // The marker's signal is removed from the vault, so its dedup hash+ // leaves the index: the backlog raises material the vault holds+ // nowhere - exactly what the ask-once layer re-proposes.+ rmSync(join(brainDirs(vault).inbox, `sig-${todayUtc()}-alpha.md`));++ const res = await run(["--propose-backlog", "--json"]);+ expect(res.returncode).toBe(0);+ const body = JSON.parse(res.stdout) as ProposeBacklogJson;+ expect(body.undistilled).toBe(1);+ expect(body.proposals.length).toBe(1);+ expect(body.proposals[0]?.topic).toBe("alpha");+ expect(body.proposals[0]?.signal).toBe("positive");+ expect(body.proposals[0]?.sessions).toEqual(["one.jsonl"]);+ expect(body.dropped).toEqual([]);+ expect(body.report_only_sessions).toEqual([]);+ });++ test("material the vault already holds is dropped from the ask and named", async () => {+ // Two sessions raise the SAME marker; the first import writes it and+ // the second is deduped at write time - but without the batching layer+ // the ask would still have raised the same material twice, once per+ // session. Ask-once: zero proposals, two named drops, both sessions+ // reported as already known.+ claudeMarkerLog("one.jsonl", "alpha");+ claudeMarkerLog("two.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--recall", "--json"])).returncode).toBe(0);++ const res = await run(["--propose-backlog", "--json"]);+ expect(res.returncode).toBe(0);+ const body = JSON.parse(res.stdout) as ProposeBacklogJson;+ expect(body.proposals).toEqual([]);+ expect(body.dropped.length).toBe(2);+ expect(body.dropped.map((d) => d.session_id).toSorted()).toEqual(["one.jsonl", "two.jsonl"]);+ expect(body.dropped[0]?.dedup_hash).toBe(body.dropped[1]?.dedup_hash);+ expect(body.report_only_sessions).toEqual(["one.jsonl", "two.jsonl"]);+ });++ test("an empty backlog is an explicit empty envelope, not a failure", async () => {+ const res = await run(["--propose-backlog", "--json"]);+ expect(res.returncode).toBe(0);+ const body = JSON.parse(res.stdout) as ProposeBacklogJson;+ expect(body.undistilled).toBe(0);+ expect(body.proposals).toEqual([]);+ expect(body.dropped).toEqual([]);+ expect(body.report_only_sessions).toEqual([]);+ });++ test("the human rendering numbers the proposals and names the drops", async () => {+ claudeMarkerLog("one.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--recall", "--json"])).returncode).toBe(0);+ rmSync(join(brainDirs(vault).inbox, `sig-${todayUtc()}-alpha.md`));++ const res = await run(["--propose-backlog"]);+ expect(res.returncode).toBe(0);+ expect(res.stdout).toContain("undistilled: 1");+ expect(res.stdout).toContain("proposals: 1");+ expect(res.stdout).toContain("proposal 1: alpha");+ expect(res.stdout).toContain(" signal: positive");+ expect(res.stdout).toContain(" sessions: one.jsonl");+ expect(res.stdout).toContain("dropped as already known: 0");+ });++ test("mode conflicts are refused, naming both halves", async () => {+ const path = claudeMarkerLog("one.jsonl", "alpha");++ const withStatus = await run(["--propose-backlog", "--status"]);+ expect(withStatus.returncode).toBe(1);+ expect(withStatus.stderr).toContain("--propose-backlog");+ expect(withStatus.stderr).toContain("--status");++ const withDiscover = await run(["--propose-backlog", "--discover"]);+ expect(withDiscover.returncode).toBe(1);+ expect(withDiscover.stderr).toContain("--discover");++ const withAll = await run(["--propose-backlog", "--all"]);+ expect(withAll.returncode).toBe(1);+ expect(withAll.stderr).toContain("--all");++ const withPath = await run([path, "--propose-backlog"]);+ expect(withPath.returncode).toBe(1);+ expect(withPath.stderr).toContain(path);+ expect(withPath.stderr).toContain("--propose-backlog");+ }, 20000);+});+describe("--discover reports what would import and imports nothing", () => {test("it lists the gap and leaves the inbox empty", async () => {const path = claudeLog("one.jsonl", "alpha");@@ -584,6 +782,7 @@ describe("the surface is advertised where an operator looks", () => {"json","preserve-event-time","progress",+ "propose-backlog","recall","recall-session-id","recall-summary-group-size",diff --git a/tests/core/architecture/progress-census.test.ts b/tests/core/architecture/progress-census.test.tsindex 51738d0a..3d91f0f1 100644--- a/tests/core/architecture/progress-census.test.ts+++ b/tests/core/architecture/progress-census.test.ts@@ -130,6 +130,10 @@ const DECLARED_EXEMPTIONS: ReadonlyMap<string, string> = new Map<string, string>"EmbeddingPhaseOptions","A PHASE, not a run: called from indexInto and standalone by the vector backfill. Given a sink it would build a second counter, and a stream carrying two terminators cannot say which one ended the run - so it borrows the run owner's ProgressCounter instead, which is strictly more wiring than a sink, not less.",],+ [+ "BuildMiningCensusOptions",+ "A pure READ, not a run: the census opens no source log and writes nothing - it joins the two vault-side ledgers and classifies one imported session at a time, and the finished census IS the report. Both callers (the import-session verb and the backlog proposal) render one complete result and exit, so there is no stream to watch; the safeguard is what bounds the read inside a SessionStart hook's budget, and per-session events would report on a read whose every partial state is discarded.",+ ],]);/** A stage name, or a value on a frozen stage object, must look like this. */diff --git a/tests/core/architecture/state-surface-census.test.ts b/tests/core/architecture/state-surface-census.test.tsindex 774060ac..68b5e074 100644--- a/tests/core/architecture/state-surface-census.test.ts+++ b/tests/core/architecture/state-surface-census.test.ts@@ -79,12 +79,19 @@ const STORE_DIR_IDENTIFIER = "DERIVED_STORE_DIR";*/// 24: the bootstrap receipt writer joins the population with its row// (write-side-trust Task 14), the same join install.lock.json made.-const SWEPT_POPULATION_SIZE = 24;+// 24 -> 25: the session mining ledger joins the population with its row+// (session-mining-economy), the same join the session import ledger made.+// 25 -> 26: the storage usage meter joins, excused (storage lifecycle+// external review, M3): it joins the directory to MEASURE the store and the+// state beside it as census surfaces, and writes nothing.+const SWEPT_POPULATION_SIZE = 26;/** Declared surfaces today. Pinned for the same reason. */// 50: the source-archives row (answer-trust Task C1) joins the population,// the same join the decisions ledger and the bootstrap receipt made.-const DECLARED_SURFACE_COUNT = 50;+// 50 -> 51: the session mining ledger joins the population with its row+// (session-mining-economy), the same join the session import ledger made.+const DECLARED_SURFACE_COUNT = 51;/** An exclusion reason has to be an argument, not a label. */const MIN_REASON_LENGTH = 80;diff --git a/tests/core/architecture/visibility-surface-census.test.ts b/tests/core/architecture/visibility-surface-census.test.tsindex dc0c6465..94ffd66e 100644--- a/tests/core/architecture/visibility-surface-census.test.ts+++ b/tests/core/architecture/visibility-surface-census.test.ts@@ -972,7 +972,7 @@ const SURFACE_SOURCE_TREE = SURFACE_TREES.flatMap((root) => readTree(root));const DIRECT_VAULT_READERS = directVaultReadFiles(SURFACE_SOURCE_TREE);/** Measured: files in the surface trees that read a vault path directly. */-const DIRECT_VAULT_READ_POPULATION_SIZE = 5;+const DIRECT_VAULT_READ_POPULATION_SIZE = 6;/*** ## What this sweep cannot see, stated rather than implieddiff --git a/tests/core/architecture/write-site-census.test.ts b/tests/core/architecture/write-site-census.test.tsindex cc22892b..9f8f935d 100644--- a/tests/core/architecture/write-site-census.test.ts+++ b/tests/core/architecture/write-site-census.test.ts@@ -1420,8 +1420,14 @@ const DIRECT_WRITE_ROWS = 82;* the `expectBefore` compare-and-swap, under the shard's own lock - a* machine JSONL rewrite that carries no note content, so it arrives in* the shared class needing no exclusion.+ *+ * 120 -> 121, from the session-mining-economy wave and nothing else: the+ * session mining ledger (`src/core/brain/session-mining-ledger.ts`)+ * writes its tombstone JSON through `atomicWriteFileSync` under the+ * ledger file's own lock - derived-store machine state, not a note, so+ * it arrives in the shared class needing no exclusion.*/-const SHARED_HELPER_ROWS = 120;+const SHARED_HELPER_ROWS = 121;// ----- Origin-channel coverage boundary (Unit C) ----------------------------@@ -1537,9 +1543,12 @@ const UNSTAMPED_DIRECT_ROWS = 81;* state, not notes). 116 -> 117: the receipt fold engine* (`src/core/brain/receipts/compact.ts`) rewrites its named shards through* the shared atomic writer (machine JSONL ledger rows, not notes; no- * origin stamp on a fold rewrite).+ * origin stamp on a fold rewrite). 117 -> 118: the session mining ledger+ * (`src/core/brain/session-mining-ledger.ts`) writes its tombstone JSON+ * through the shared atomic writer (derived-store machine state, not+ * notes; no origin stamp on machine state).*/-const UNSTAMPED_SHARED_ROWS = 117;+const UNSTAMPED_SHARED_ROWS = 118;describe("in-vault write-site census", () => {test("every direct-fs write site carries a written exclusion", () => {diff --git a/tests/core/brain/capture/inbox-drain.test.ts b/tests/core/brain/capture/inbox-drain.test.tsindex ee428620..1f2b64a7 100644--- a/tests/core/brain/capture/inbox-drain.test.ts+++ b/tests/core/brain/capture/inbox-drain.test.ts@@ -165,6 +165,57 @@ test("a route that succeeds but whose archive fails is a distinct state; a re-ruexpect(noteBody.split(body).length - 1).toBe(1);});+// ── capacity admission: the drain's ingest route walks the same wall as brain_ingest_source ──++describe("the drain's ingest route under a capacity cap", () => {+ function configureCriticalCap(): void {+ // A cap everything clears: any capture makes the vault measure past+ // the critical tier, which is the only tier that refuses ingest.+ writeFileSync(+ join(vault, "Brain", "_brain.yaml"),+ "schema_version: 1\n\nstorage:\n cap_bytes: 10\n",+ "utf8",+ );+ }++ test("a critical vault refuses the source-reference route by name, in both modes", () => {+ configureCriticalCap();+ writeCaptureNote(vault, { body: "https://example.com/article", provenance: prov(1) });+ for (const apply of [false, true]) {+ const report = drainInbox(vault, { apply, agent: "tester", now: NOW });+ expect(report.items).toHaveLength(1);+ expect(report.items[0]!.action).toBe("skip");+ expect(report.items[0]!.classification).toBe("unroutable");+ expect(report.items[0]!.reason).toContain("capacity_exhausted");+ expect(report.routed).toBe(0);+ // Left staged: the route never ran, so a rerun after the operator+ // reclaims space converges.+ expect(listStagedCaptures(vault)).toHaveLength(1);+ expect(listArchivedCaptures(vault)).toHaveLength(0);+ }+ });++ test("the refusal never writes a summary page", () => {+ configureCriticalCap();+ writeCaptureNote(vault, { body: "https://example.com/article", provenance: prov(1) });+ drainInbox(vault, { apply: true, agent: "tester", now: NOW });+ expect(existsSync(join(vault, "Brain", "sources"))).toBe(false);+ });++ test("non-ingest routes still work while ingest is refused", () => {+ // The admission wall gates INGEST, exactly like brain_ingest_source:+ // an idea note writes no source bytes, so the capture inbox keeps+ // moving while the wall is closed.+ configureCriticalCap();+ writeCaptureNote(vault, { body: "https://example.com/article", provenance: prov(1) });+ writeCaptureNote(vault, { body: "a standalone atomic idea", provenance: prov(2) });+ const report = drainInbox(vault, { apply: true, agent: "tester", now: NOW });+ expect(report.items.map((i) => i.classification).toSorted()).toEqual(["idea", "unroutable"]);+ expect(report.routed).toBe(1);+ expect(existsSync(join(vault, CAPTURED_NOTES_DIR_REL))).toBe(true);+ });+});+test("an obligation marker defaults the cadence when none is given", () => {writeCaptureNote(vault, {body: `${CAPTURE_OBLIGATION_MARKER} tidy the desk`,diff --git a/tests/core/brain/continuity/reader-census.test.ts b/tests/core/brain/continuity/reader-census.test.tsindex 266b3560..b1be0ea1 100644--- a/tests/core/brain/continuity/reader-census.test.ts+++ b/tests/core/brain/continuity/reader-census.test.ts@@ -90,6 +90,10 @@ const DIRECT_STORE_READERS: Readonly<Record<string, string>> = Object.freeze({"src/core/brain/dedup-telemetry.ts":"reads `ingest_dedup` rollups. Carries a literal NUL byte, so text tools " +"that classify by content type do not see it at all.",+ "src/core/brain/explorer-console.ts":+ "reads EVERY kind inside the rendered window for the console's written-per-day " ++ "histogram - a write is a write whatever its kind, so only `createdAt` is read " ++ "and the count never depends on any record's content.","src/core/brain/gate-telemetry.ts": "reads `gate_telemetry` for recall-gate summaries.","src/core/brain/generation-reports.ts": "reads `generation_report` for list, summary and show.","src/core/brain/idea-lineage.ts":@@ -117,6 +121,9 @@ const DIRECT_STORE_READERS: Readonly<Record<string, string>> = Object.freeze({"src/core/brain/pre-compact-extract.ts":"reads `pre_compact_extract` for the append-time dedupe-key lookup.","src/core/brain/recall-telemetry.ts": "reads `recall_telemetry` for list and summary.",+ "src/core/brain/session-mining-ledger.ts":+ "reads the continuity store once per prune pass for the sessions that still " ++ "hold turns, so a tombstone whose session lost its turns is dropped.","src/core/brain/recent-turns.ts": "reads `recent_turn` for the sequence counter and the window.","src/core/brain/session-recall.ts":"reads the WHOLE store for `session_turn` / `session_summary_node`, plus a " +@@ -158,7 +165,10 @@ const READ_MODEL_READER_COUNT = 4;// 26 -> 27: the answer-trust wave registers the blank-session-identity// doctor check (`doctor/session-identity-check.ts`), which reads the whole// store once to count records per kind that name no session.-const DIRECT_STORE_READER_COUNT = 27;+// 27 -> 28: the session-mining-economy wave registers the explorer console+// collector (`explorer-console.ts`), which reads every kind inside the+// 30-day window for the written-per-day histogram.+const DIRECT_STORE_READER_COUNT = 29;/** A file the classifier must see, and that `grep` alone does not. */diff --git a/tests/core/brain/digest-actions.test.ts b/tests/core/brain/digest-actions.test.tsindex 98cd0802..7436c3f8 100644--- a/tests/core/brain/digest-actions.test.ts+++ b/tests/core/brain/digest-actions.test.ts@@ -5,13 +5,29 @@* when populated, and that an empty vault stays quiet.*/-import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { afterEach, beforeEach, describe, expect, mock, test } from "bun:test";import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";import { renderDigest } from "../../../src/core/brain/digest.ts";+// A forwarding mock over the dedup scan (the claim-provenance precedent):+// inert unless this file's failing-scan test turns it on, so the mocks+// never answer differently for any other test file sharing the process.+import * as pageDedupModule from "../../../src/core/brain/page-dedup.ts";++const realFindDuplicateCandidates = pageDedupModule.findDuplicateCandidates;+let failDedup = false;++mock.module("../../../src/core/brain/page-dedup.ts", () => ({+ ...pageDedupModule,+ findDuplicateCandidates: (vault: string) => {+ if (failDedup) throw new Error("the dedup scan refused");+ return realFindDuplicateCandidates(vault);+ },+}));+let vault: string;const DERIVED_KEYS = new Set([@@ -53,6 +69,37 @@ afterEach(() => {rmSync(vault, { recursive: true, force: true });});+describe("brain_digest collect_errors surface (review M10)", () => {+ test("a healthy vault renders no collect_errors row", () => {+ const res = renderDigest(vault, {+ format: "json",+ since: new Date("2026-05-24T00:00:00Z"),+ until: new Date("2026-05-25T00:00:00Z"),+ });+ const payload = JSON.parse(res.content) as { collect_errors?: unknown };+ expect(payload.collect_errors).toBeUndefined();+ });++ test("a failed scan is rendered as a named collect_errors row", () => {+ failDedup = true;+ try {+ const res = renderDigest(vault, {+ format: "json",+ since: new Date("2026-05-24T00:00:00Z"),+ until: new Date("2026-05-25T00:00:00Z"),+ });+ const payload = JSON.parse(res.content) as {+ collect_errors?: ReadonlyArray<{ scan: string; message: string }>;+ };+ expect(payload.collect_errors).toEqual([+ { scan: "dedup", message: "the dedup scan refused" },+ ]);+ } finally {+ failDedup = false;+ }+ });+});+describe("brain_digest actions surface", () => {test("empty vault renders no Actions section in markdown", () => {const res = renderDigest(vault, {diff --git a/tests/core/brain/distill/backlog-proposals.test.ts b/tests/core/brain/distill/backlog-proposals.test.tsnew file mode 100644index 00000000..6573d0af--- /dev/null+++ b/tests/core/brain/distill/backlog-proposals.test.ts@@ -0,0 +1,199 @@+/**+ * The ask-once batching layer over the undistilled backlog+ * (session-mining-economy, t_0358af42).+ *+ * Dedup happens before the ASK, not before the write: the write-time gates+ * in `extract-signals.ts` and the import path already suppress duplicate+ * WRITES, but nothing collapsed the ask - a recurring fact was re-proposed+ * to the operator once per session that raised it. `buildBacklogProposals`+ * is the pure core that fixes the ask: candidates whose dedup hash already+ * exists in the vault are dropped from the ask (each drop named), the+ * remainder collapses to one proposal per distinct hash across sessions,+ * and the whole batch is one envelope for the calling agent to present.+ * The kernel stays LLM-free and runs no interactive loop: it groups, it+ * does not ask.+ */++import { describe, expect, test } from "bun:test";++import { computeDedupHash, type DedupIndexEntry } from "../../../../src/core/brain/dedup-hash.ts";+import {+ buildBacklogProposals,+ type BacklogCandidate,+ type BacklogProposal,+} from "../../../../src/core/brain/distill/backlog-proposals.ts";++function candidate(overrides: Partial<BacklogCandidate> = {}): BacklogCandidate {+ return {+ sessionId: "alpha.jsonl",+ topic: "declare roots once",+ signal: "positive",+ principle: "Declare the roots exactly once.",+ ...overrides,+ };+}++function indexWith(...candidates: BacklogCandidate[]): Map<string, DedupIndexEntry> {+ const out = new Map<string, DedupIndexEntry>();+ for (const raised of candidates) {+ const hash = computeDedupHash({+ topic: raised.topic,+ signal: raised.signal,+ principle: raised.principle,+ scope: raised.scope,+ });+ out.set(hash, {+ id: `sig-${hash.slice(0, 8)}`,+ path: `Brain/inbox/sig-${hash.slice(0, 8)}.md`,+ });+ }+ return out;+}++function hashOf(raised: BacklogCandidate): string {+ return computeDedupHash({+ topic: raised.topic,+ signal: raised.signal,+ principle: raised.principle,+ scope: raised.scope,+ });+}++describe("buildBacklogProposals", () => {+ test("an empty backlog is an explicit empty result", () => {+ expect(buildBacklogProposals([], new Map())).toEqual({+ proposals: [],+ dropped: [],+ reportOnlySessions: [],+ });+ });++ test("unknown material becomes one proposal naming its session and hash", () => {+ const pending = [candidate()];+ const result = buildBacklogProposals(pending, new Map());+ expect(result.dropped).toEqual([]);+ expect(result.reportOnlySessions).toEqual([]);+ expect(result.proposals).toEqual([+ {+ dedupHash: hashOf(pending[0]!),+ topic: "declare roots once",+ signal: "positive",+ principle: "Declare the roots exactly once.",+ sessions: ["alpha.jsonl"],+ },+ ]);+ });++ test("the same material raised by several sessions collapses to one proposal", () => {+ const pending = [+ candidate(),+ candidate({ sessionId: "beta.jsonl" }),+ candidate({ sessionId: "gamma.jsonl" }),+ ];+ const result = buildBacklogProposals(pending, new Map());+ expect(result.proposals.length).toBe(1);+ expect(result.proposals[0]?.sessions).toEqual(["alpha.jsonl", "beta.jsonl", "gamma.jsonl"]);+ });++ test("material the vault already has is dropped from the ask, named", () => {+ const known = candidate();+ const pending = [known, candidate({ topic: "brand new rule", principle: "Write it down." })];+ const result = buildBacklogProposals(pending, indexWith(known));+ expect(result.proposals.map((p) => p.topic)).toEqual(["brand new rule"]);+ expect(result.dropped).toEqual([+ { sessionId: "alpha.jsonl", topic: "declare roots once", dedupHash: hashOf(known) },+ ]);+ // The session raised known AND new material, so it is not report-only.+ expect(result.reportOnlySessions).toEqual([]);+ });++ test("a session raising only already-known material appears in the report and in no proposal", () => {+ const knownA = candidate();+ const knownB = candidate({+ sessionId: "beta.jsonl",+ topic: "never paste tokens",+ principle: "Redact.",+ });+ const fresh = candidate({+ sessionId: "beta.jsonl",+ topic: "fresh rule",+ principle: "Brand new.",+ });+ // beta raises one known candidate and one fresh one; alpha raises only+ // the known one.+ const pending = [knownA, knownB, fresh];+ const result = buildBacklogProposals(pending, indexWith(knownA, knownB));+ expect(result.proposals.map((p) => p.sessions)).toEqual([["beta.jsonl"]]);+ expect(result.reportOnlySessions).toEqual(["alpha.jsonl"]);+ expect(result.dropped.length).toBe(2);+ });++ test("scope participates in the identity: same rule, different scope, two proposals", () => {+ const pending = [candidate({ scope: "project-a" }), candidate({ scope: "project-b" })];+ const result = buildBacklogProposals(pending, new Map());+ expect(result.proposals.length).toBe(2);+ expect(result.proposals.map((p) => p.scope).toSorted()).toEqual(["project-a", "project-b"]);+ });++ test("a candidate with no scope and one with an empty scope share a hash", () => {+ const pending = [candidate(), candidate({ sessionId: "beta.jsonl", scope: "" })];+ const result = buildBacklogProposals(pending, new Map());+ expect(result.proposals.length).toBe(1);+ expect(result.proposals[0]?.sessions).toEqual(["alpha.jsonl", "beta.jsonl"]);+ // The hash unifies the missing and the empty scope, so the proposal+ // must not carry one or not depending on the input order either.+ const reversed = buildBacklogProposals(pending.toReversed(), new Map());+ expect(result).toEqual(reversed);+ expect("scope" in result.proposals[0]!).toBe(false);+ });++ test("the result is deterministic: proposals by hash, drops by session then topic", () => {+ const known = candidate({ topic: "known rule", principle: "Known." });+ const alsoKnown = candidate({+ sessionId: "beta.jsonl",+ topic: "another known",+ principle: "Also known.",+ });+ const pending = [+ candidate({ topic: "second", principle: "Two." }),+ candidate({ sessionId: "beta.jsonl", topic: "first", principle: "One." }),+ alsoKnown,+ known,+ ];+ const first = buildBacklogProposals(pending, indexWith(known, alsoKnown));+ const second = buildBacklogProposals(pending.toReversed(), indexWith(known, alsoKnown));+ expect(first).toEqual(second);+ const hashes = first.proposals.map((p) => p.dedupHash);+ expect(hashes).toEqual([...hashes].toSorted());+ expect(first.proposals.map((p) => p.topic).toSorted()).toEqual(["first", "second"]);+ expect(first.dropped.map((d) => [d.sessionId, d.topic])).toEqual([+ ["alpha.jsonl", "known rule"],+ ["beta.jsonl", "another known"],+ ]);+ });++ test("one hash class picks its representative by content, not by input order", () => {+ // computeDedupHash normalizes case, so both spellings share a hash+ // while their raw fields differ; which one stands for the proposal+ // must not depend on which candidate the input listed first.+ const loud = candidate({+ sessionId: "beta.jsonl",+ topic: "Declare the Roots",+ principle: "Declare the roots exactly once.",+ });+ const quiet = candidate({+ sessionId: "gamma.jsonl",+ topic: "declare the roots",+ principle: "Declare the roots exactly once.",+ });+ const expectStable = (+ pending: ReadonlyArray<BacklogCandidate>,+ ): ReadonlyArray<BacklogProposal> => buildBacklogProposals(pending, new Map()).proposals;+ const first = expectStable([loud, quiet]);+ const reversed = expectStable([quiet, loud]);+ expect(first).toEqual(reversed);+ expect(first[0]?.topic).toBe("Declare the Roots");+ expect(first[0]?.principle).toBe("Declare the roots exactly once.");+ expect(first[0]?.sessions).toEqual(["beta.jsonl", "gamma.jsonl"]);+ });+});diff --git a/tests/core/brain/document-dedup-detector.test.ts b/tests/core/brain/document-dedup-detector.test.tsindex 6722267d..d5a35ecc 100644--- a/tests/core/brain/document-dedup-detector.test.ts+++ b/tests/core/brain/document-dedup-detector.test.ts@@ -181,6 +181,90 @@ describe("detectDocumentDedup", () => {expect(detect()).toHaveLength(0);});+ test("a reached candidate cap emits the coverage notice with scanned and total", () => {+ // The review M9 shape: a long-lived vault whose duplicates sort AFTER+ // the cap. 200 filler pages keep the first comparison slots (each with+ // a distinct token payload, so no filler pairs with another), and the+ // near-duplicate pair sorts last, so the cap never reads it. The scan+ // must say its coverage was bounded instead of answering clean.+ for (let i = 0; i < 200; i++) {+ const tag = `p${String(i).padStart(4, "0")}`;+ writeSourcePage(`filler-${String(i).padStart(4, "0")}`, {+ body: `${UNRELATED}\n${tag}a ${tag}b ${tag}c ${tag}d ${tag}e ${tag}f ${tag}g ${tag}h`,+ createdAt: "2026-01-01T00:00:00Z",+ });+ }+ writeSourcePage("zz-dup-a", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("zz-dup-b", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });++ const findings = detect();++ // The pair behind the cap was never read, so no retirement is claimed+ // for it - that is the bound working. The truncation itself is named.+ expect(findings.some((f) => f.proposed_action === "retire")).toBe(false);+ const notice = findings.find(+ (f) => (f.evidence as Record<string, unknown>)["candidate_cap_reached"] === true,+ );+ expect(notice).toBeDefined();+ expect(notice!.proposed_action).toBe("review");+ const evidence = notice!.evidence as Record<string, unknown>;+ expect(evidence["scanned"]).toBe(200);+ expect(evidence["total"]).toBe(202);+ expect(evidence["cap"]).toBe(200);+ // `listed` counts every readable .md under the reach - non-summary+ // kinds and unparseable frontmatter included - so the row's only+ // sentence about coverage must not call the skipped total+ // summary-page candidates.+ expect(notice!.title).toContain(`of 202 listed pages`);+ expect(notice!.title).not.toContain("summary-page candidates");+ });++ test("a merge-resolved page inside the capped read set does not hide the truncation", () => {+ // The M9 corner: the cap bounds what was COLLECTED, and the+ // merge-resolved filter thins the kept set afterwards. One+ // merged_into stamp among exactly the capped 200 leaves 199 records,+ // and a notice gated on the post-filter count reads 199 < cap and+ // goes quiet - while pages 201+ were never read. A truncated scan+ // must never answer clean, so the guard asks the pre-filter count.+ for (let i = 0; i < 200; i++) {+ const tag = `p${String(i).padStart(4, "0")}`;+ writeSourcePage(`filler-${String(i).padStart(4, "0")}`, {+ body: `${UNRELATED}\n${tag}a ${tag}b ${tag}c ${tag}d ${tag}e ${tag}f ${tag}g ${tag}h`,+ createdAt: "2026-01-01T00:00:00Z",+ // Exactly one page the read set collected was already resolved+ // by a page merge - ordinary in a vault that uses the merge lane.+ ...(i === 100 ? { mergedInto: "pref-elsewhere" } : {}),+ });+ }+ writeSourcePage("zz-dup-a", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("zz-dup-b", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });++ const findings = detect();++ // The pair behind the cap was never read, so no retirement is+ // claimed for it - and the truncation is named even though the+ // merge-resolved filter thinned the kept set below the cap.+ expect(findings.some((f) => f.proposed_action === "retire")).toBe(false);+ const notice = findings.find(+ (f) => (f.evidence as Record<string, unknown>)["candidate_cap_reached"] === true,+ );+ expect(notice).toBeDefined();+ const evidence = notice!.evidence as Record<string, unknown>;+ // `scanned` counts what the cap admitted - the pre-filter read set -+ // not the thinner kept set the merge filter left behind.+ expect(evidence["scanned"]).toBe(200);+ expect(evidence["total"]).toBe(202);+ });++ test("no coverage notice while the cap does not bind", () => {+ writeSourcePage("a", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });+ writeSourcePage("b", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });+ const findings = detect();+ expect(+ findings.some((f) => (f.evidence as Record<string, unknown>)["candidate_cap_reached"]),+ ).toBe(false);+ });+test("a member the caller cannot read is treated as absent", () => {writeSourcePage("visible", { body: NEAR_A, createdAt: "2026-01-01T00:00:00Z" });const hidden = writeSourcePage("hidden", { body: NEAR_B, createdAt: "2026-01-02T00:00:00Z" });diff --git a/tests/core/brain/document-retire.test.ts b/tests/core/brain/document-retire.test.tsindex efc71a6d..1c9ccafd 100644--- a/tests/core/brain/document-retire.test.ts+++ b/tests/core/brain/document-retire.test.ts@@ -11,10 +11,11 @@* collected ({path, message}), the failed copy stays on disk, and the* run completes.*- * The manifest drop is the resurrect guard: with the entry gone, the- * retired copy's source classifies `new` again, so the system never- * claims a retired page's records are current and a later ingest can- * only bring the source back as a deliberate full re-ingest.+ * The retired manifest marker is the resurrect guard: the entry stays,+ * marked `retired_into` the canonical page, so the unchanged source+ * classifies `retired` and no batch plan queues it again - while a real+ * content change re-admits it as `modified`, and a deliberate re-ingest+ * re-records the entry without the marker.*/import { afterEach, beforeEach, describe, expect, test } from "bun:test";@@ -25,11 +26,13 @@ import {mkdtempSync,readFileSync,rmSync,+ statSync,writeFileSync,} from "node:fs";import { tmpdir } from "node:os";import { basename, join } from "node:path";+import { planBatches } from "../../../src/core/brain/ingest/batch-plan.ts";import {classifyPaths,readManifest,@@ -37,7 +40,7 @@ import {} from "../../../src/core/brain/ingest/content-manifest.ts";import { listContinuityRecords } from "../../../src/core/brain/continuity/store.ts";import {- RETIRE_LINKS_SKIPPED_BASENAME_COLLISION,+ RETIRE_BASENAME_COLLISION_REFUSAL,retireDocumentCopies,} from "../../../src/core/brain/document-retire.ts";import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";@@ -135,13 +138,12 @@ describe("retireDocumentCopies", () => {expect(copy.path).toBe(retired);expect(copy.canonical).toBe(canonical);expect(copy.manifestKey).toBe("imports/b.md");- expect(copy.manifestDropped).toBe(true);+ expect(copy.manifestRetired).toBe(true);expect(copy.sourceArchiveHash).toBe(ARCHIVE_HASH);expect(copy.auditRecordId).not.toBeNull();// The wikilink retarget report, asserted: the citing page matched,// was rewritten, and its [[src-b]] now names the canonical spelling.- expect(copy.wikilinksSkipped).toBeNull();expect(copy.wikilinks!.matched).toEqual([citing]);expect(copy.wikilinks!.rewritten).toEqual([citing]);expect(copy.wikilinks!.failed).toEqual([]);@@ -159,26 +161,67 @@ describe("retireDocumentCopies", () => {expect(result.recoverability.state).toBe("covered");});- test("drops the manifest entry round-tripping the read contract", () => {+ test("marks the manifest entry retired, round-tripping the read contract", () => {const { canonical, retired } = seedPair();+ const before = readManifest(vault).entries["imports/b.md"];const fingerprintBefore = readManifest(vault).entries["imports/a.md"]?.fingerprint;retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });const manifest = readManifest(vault);- expect(manifest.entries["imports/b.md"]).toBeUndefined();+ // The entry STAYS, marked retired into the canonical page - the+ // resurrect guard. Dropping it would re-admit the unchanged source.+ expect(manifest.entries["imports/b.md"]?.retired_into).toBe(canonical);+ // The digest and fingerprint the entry was recorded under survive.+ expect(manifest.entries["imports/b.md"]?.sha256).toBe(before?.sha256);+ expect(manifest.entries["imports/b.md"]?.fingerprint).toBe(before?.fingerprint);// The surviving entry keeps the fingerprint it was recorded under -- // a removal never re-stamps the contract the survivors answered.+ // a retirement never re-stamps the contract the survivors answered.expect(manifest.entries["imports/a.md"]?.fingerprint).toBe(fingerprintBefore);});- test("the manifest drop is the resurrect guard: the retired source classifies new", () => {+ test("the retired marker is the resurrect guard: a batch plan never queues the retired source", () => {const { canonical, retired } = seedPair();retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });+ // The unchanged source classifies `retired`, never `new`.const classification = classifyPaths(vault, ["imports/b.md"], readManifest(vault));- expect(classification.new).toContain("imports/b.md");+ expect(classification.retired).toContain("imports/b.md");+ expect(classification.new).not.toContain("imports/b.md");expect(classification.unchanged).not.toContain("imports/b.md");++ // A batch plan over the folder does not queue the retired source.+ const plan = planBatches(vault, "imports", { maxBatchBytes: 1024 * 1024, maxBatchFiles: 100 });+ const queued = plan.batches.flatMap((b) => b.files.map((f) => f.path));+ expect(queued).not.toContain("imports/b.md");+ expect(plan.skippedRetired).toContain("imports/b.md");++ // A real content change re-admits it: the bytes moved, so the copy+ // classifies `modified` and the plan queues it again.+ writeFileSync(join(vault, "imports", "b.md"), "changed duplicate source bytes\n", "utf8");+ const after = classifyPaths(vault, ["imports/b.md"], readManifest(vault));+ expect(after.modified).toContain("imports/b.md");+ expect(after.retired).not.toContain("imports/b.md");+ const changedPlan = planBatches(vault, "imports", {+ maxBatchBytes: 1024 * 1024,+ maxBatchFiles: 100,+ });+ expect(changedPlan.batches.flatMap((b) => b.files.map((f) => f.path))).toContain(+ "imports/b.md",+ );+ });++ test("a deliberate re-ingest re-admits the retired source", () => {+ const { canonical, retired } = seedPair();+ retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });++ // A deliberate full re-ingest re-records the entry without the marker+ // (updateManifest is the post-ingest write), and the source classifies+ // `unchanged` again - the retirement claimed the records, not the file.+ updateManifest(vault, ["imports/b.md"]);+ const classification = classifyPaths(vault, ["imports/b.md"], readManifest(vault));+ expect(classification.unchanged).toContain("imports/b.md");+ expect(classification.retired).not.toContain("imports/b.md");});test("appends one source_invalidation record per copy naming the provenance", () => {@@ -204,6 +247,26 @@ describe("retireDocumentCopies", () => {expect(existsSync(join(vault, "Brain", ".archives", `${ARCHIVE_HASH}.txt`))).toBe(true);});+ test("the result reports the snapshot bytes and the net disk delta", () => {+ // Review M6 (retire half): the destructive span archives Brain/ into a+ // COUNTED surface (.snapshots) before it frees anything, so a result+ // carrying only removal counts hides that the vault can GROW. The+ // result carries the page bytes measured before the removal, the+ // snapshot archive's bytes, and the net between them.+ const { canonical, retired } = seedPair();+ const pageBytes = statSync(join(vault, retired)).size;++ const result = retireDocumentCopies(vault, [{ path: retired, canonical }], {+ agent: "tester",+ now: NOW,+ });++ expect(result.retired[0]!.pageBytes).toBe(pageBytes);+ expect(result.disk.removedBytes).toBe(pageBytes);+ expect(result.disk.snapshotBytes).toBe(statSync(result.snapshotPath!).size);+ expect(result.disk.netBytes).toBe((result.disk.snapshotBytes ?? 0) - pageBytes);+ });+test("a per-copy failure is collected, the copy stays on disk, and the run completes", () => {const { canonical, retired } = seedPair();const missing = "Brain/sources/src-gone.md";@@ -225,12 +288,14 @@ describe("retireDocumentCopies", () => {expect(result.snapshotRunId).not.toBeNull();});- test("a basename collision skips the link pass and names why", () => {+ test("a basename collision refuses the copy and keeps it on disk", () => {// An unrelated page in another namespace shares the retired copy's// basename, so the bare [[src-b]] spelling is ambiguous: rewriting it- // would repoint links that meant the namesake. The copy still retires;- // the pass is skipped by name and every link stays exactly as it was -- // a stale link is discoverable, a mis-pointed one misleads silently.+ // would repoint links that meant the namesake, and removing the page+ // anyway would re-point every leftover [[src-b]] at the namesake+ // SILENTLY - resolved by a deletion, never confirmed by an editor.+ // The copy is refused per copy and stays on disk instead (review L5);+ // the stale link stays exactly as visible as it was.const { canonical, retired, citing } = seedPair();const namesake = "Brain/entities/src-b.md";mkdirSync(join(vault, "Brain", "entities"), { recursive: true });@@ -241,16 +306,48 @@ describe("retireDocumentCopies", () => {now: NOW,});- expect(result.failed).toEqual([]);- const copy = result.retired[0]!;- expect(copy.path).toBe(retired);- expect(copy.wikilinks).toBeNull();- expect(copy.wikilinksSkipped).toBe(RETIRE_LINKS_SKIPPED_BASENAME_COLLISION);- // The citing page keeps its spelling, the namesake page is untouched,- // and the retired copy itself is still gone.- expect(readFileSync(join(vault, citing), "utf8")).toContain("[[src-b]]");+ expect(result.retired).toEqual([]);+ expect(result.failed).toHaveLength(1);+ expect(result.failed[0]!.path).toBe(retired);+ expect(result.failed[0]!.message).toContain(RETIRE_BASENAME_COLLISION_REFUSAL);+ // Nothing moved: the retired page, the namesake page, the citing+ // spelling, and the audit.+ expect(existsSync(join(vault, retired))).toBe(true);expect(existsSync(join(vault, namesake))).toBe(true);- expect(existsSync(join(vault, retired))).toBe(false);+ expect(readFileSync(join(vault, citing), "utf8")).toContain("[[src-b]]");+ expect(listContinuityRecords(vault, { kind: "source_invalidation" })).toHaveLength(0);+ });++ test("a failed inbound-link rewrite keeps the copy in place", () => {+ if (CHMOD_CANNOT_DENY) return;+ // One citing page sits in a directory this run cannot write: the+ // retarget report collects the write failure without throwing, and+ // retiring the copy over it would strand exactly the link the pass+ // could not repoint. The refusal comes BEFORE the removal, so the+ // copy stays on disk (review M7).+ const { canonical, retired } = seedPair();+ rmSync(join(vault, "Brain", "note.md"));+ mkdirSync(join(vault, "Brain", "links"), { recursive: true });+ writeFileSync(join(vault, "Brain", "links", "note.md"), "See [[src-b]] here.\n", "utf8");+ const linksDir = join(vault, "Brain", "links");+ chmodSync(linksDir, 0o555);+ try {+ const result = retireDocumentCopies(vault, [{ path: retired, canonical }], {+ agent: "tester",+ now: NOW,+ });++ expect(result.retired).toEqual([]);+ expect(result.failed).toHaveLength(1);+ expect(result.failed[0]!.path).toBe(retired);+ // The failure names the file whose rewrite did not land.+ expect(result.failed[0]!.message).toContain("links/note.md");+ // The copy is still on disk; nothing was claimed as retired.+ expect(existsSync(join(vault, retired))).toBe(true);+ expect(listContinuityRecords(vault, { kind: "source_invalidation" })).toHaveLength(0);+ } finally {+ chmodSync(linksDir, 0o755);+ }});test("a copy whose canonical page is missing is refused per copy", () => {@@ -319,7 +416,7 @@ describe("retireDocumentCopies", () => {// The honest half of the old claim: the page is NOT on disk.expect(existsSync(join(vault, retired))).toBe(false);// The steps before the failure did land.- expect(readManifest(vault).entries["imports/b.md"]).toBeUndefined();+ expect(readManifest(vault).entries["imports/b.md"]?.retired_into).toBe(canonical);expect(listContinuityRecords(vault, { kind: "source_invalidation" })).toHaveLength(0);} finally {chmodSync(continuityDir, 0o755);@@ -327,8 +424,8 @@ describe("retireDocumentCopies", () => {});test("the per-copy reason words the manifest outcome from what actually happened", () => {- const { retired } = seedPair();- retireDocumentCopies(vault, [{ path: retired, canonical: "Brain/sources/src-a.md" }], {+ const { canonical, retired } = seedPair();+ retireDocumentCopies(vault, [{ path: retired, canonical }], {agent: "tester",now: NOW,});@@ -340,10 +437,10 @@ describe("retireDocumentCopies", () => {>)["reason"],);- expect(reason).toContain("dropped manifest key imports/b.md");+ expect(reason).toContain(`marked manifest key imports/b.md retired into ${canonical}`);});- test("a shared source_path retires twice: the second drop is named already-absent", () => {+ test("a shared source_path retires twice: the second marker is named already-marked", () => {const { canonical } = seedPair();// A second copy declaring the SAME source_path as the first copy's.const second = writeSourcePage("b2", {@@ -363,11 +460,15 @@ describe("retireDocumentCopies", () => {const reasons = listContinuityRecords(vault, { kind: "source_invalidation" }).map((r) =>String((r.payload as Record<string, unknown>)["reason"]),);- expect(reasons.some((r) => r.includes("dropped manifest key imports/b.md"))).toBe(true);- expect(reasons.some((r) => r.includes("manifest key imports/b.md already absent"))).toBe(true);+ expect(reasons.some((r) => r.includes(`marked manifest key imports/b.md retired`))).toBe(true);+ // The second retirement marks the SAME key into the SAME canonical, so+ // the bytes are already identical and the clause names that fact.+ expect(+ reasons.some((r) => r.includes("manifest key imports/b.md already marked retired")),+ ).toBe(true);});- test("no manifest on disk is named, not claimed as a drop", () => {+ test("no manifest on disk is named, not claimed as a marker", () => {const { canonical, retired } = seedPair();rmSync(join(vault, ".open-second-brain", "ingest-manifest.json"));retireDocumentCopies(vault, [{ path: retired, canonical }], { agent: "tester", now: NOW });@@ -381,7 +482,7 @@ describe("retireDocumentCopies", () => {);expect(reason).toContain("manifest");expect(reason).toContain("imports/b.md");- expect(reason).not.toContain("dropped manifest key");+ expect(reason).not.toContain("marked manifest key");});test("a copy with no recorded source_path names the absence, not a drop", () => {@@ -401,7 +502,7 @@ describe("retireDocumentCopies", () => {)["reason"],);expect(reason).toContain("no manifest key recorded");- expect(reason).not.toContain("dropped manifest key");+ expect(reason).not.toContain("marked manifest key");});test("a copy path that escapes the vault is refused as a containment breach, not a vanish", () => {diff --git a/tests/core/brain/explorer-console.test.ts b/tests/core/brain/explorer-console.test.tsnew file mode 100644index 00000000..366e0ba1--- /dev/null+++ b/tests/core/brain/explorer-console.test.ts@@ -0,0 +1,222 @@+/**+ * Tests for the live-counts console collector (session-mining-economy,+ * t_15ebcfc1). We exercise:+ * - an empty vault: frozen, deterministic payload with telemetry+ * ABSENT (not zero), empty most-recalled, a fixed 30-day zero+ * histogram, and report snapshots reported as "disabled"+ * - the telemetry summary is the `summarizeRecallTelemetry` passthrough+ * - most-recalled ranks by recall count and joins the id/path identity+ * the records themselves carry (including an artifact referenced by+ * id only in one record and by id+path in another)+ * - most-recalled is capped at the named limit with a deterministic+ * tie-break+ * - the written-per-day histogram buckets continuity records inside+ * the named 30-day window and leaves out-of-window writes out+ * - report snapshots read per-surface latest dates when the feature is+ * enabled (a torn snapshot file is skipped), and read as "disabled"+ * when it is off+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ appendContinuityRecord,+ appendContinuitySourceInvalidation,+} from "../../../src/core/brain/continuity/store.ts";+import {+ collectExplorerConsole,+ EXPLORER_CONSOLE_MOST_RECALLED_LIMIT,+ EXPLORER_CONSOLE_WINDOW_DAYS,+} from "../../../src/core/brain/explorer-console.ts";+import { brainDirs } from "../../../src/core/brain/paths.ts";+import {+ emitRecallTelemetry,+ RECALL_CHANNEL,+ summarizeRecallTelemetry,+} from "../../../src/core/brain/recall-telemetry.ts";+import { writeReportSnapshot } from "../../../src/core/brain/report-snapshot.ts";++const NOW = new Date("2026-05-20T12:00:00.000Z");+const ENV_KEY = "OPEN_SECOND_BRAIN_REPORT_SNAPSHOTS";++let vault: string;+let previousEnv: string | undefined;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-explorer-console-"));+ previousEnv = process.env[ENV_KEY];+ delete process.env[ENV_KEY];+});++afterEach(() => {+ if (previousEnv === undefined) delete process.env[ENV_KEY];+ else process.env[ENV_KEY] = previousEnv;+ rmSync(vault, { recursive: true, force: true });+});++/** One recall-telemetry record with a fixed timestamp and artifacts. */+function emit(+ createdAt: string,+ artifacts: ReadonlyArray<{ readonly id: string; readonly path?: string }>,+ overrides: { readonly mode?: "search" | "query"; readonly channel?: "cli" | "mcp" } = {},+): void {+ emitRecallTelemetry(vault, {+ createdAt,+ host: "unit-test",+ channel: overrides.channel ?? RECALL_CHANNEL.cli,+ mode: overrides.mode ?? "search",+ status: "ok",+ durationMs: 5,+ resultCount: artifacts.length,+ topArtifacts: artifacts.map((artifact) => ({ ...artifact })),+ });+}++/**+ * A path-only artifact, exactly as the reader grammar accepts it+ * (`readArtifacts` narrows each entry to id-or-path): the emit input+ * type requires an id, so this record is written at the store level.+ */+function emitPathOnlyArtifact(createdAt: string, path: string): void {+ appendContinuityRecord(vault, {+ kind: "recall_telemetry",+ createdAt,+ sourceRefs: [{ id: path }],+ payload: { host: "unit-test", top_artifacts: [{ path }] },+ });+}++describe("collectExplorerConsole", () => {+ test("empty vault: frozen, deterministic, telemetry absent not zero, disabled snapshots", () => {+ const payload = collectExplorerConsole(vault, { now: NOW });+ const again = collectExplorerConsole(vault, { now: NOW });+ // Deterministic given opts.now, and frozen like collectExplorerData.+ expect(JSON.stringify(payload)).toBe(JSON.stringify(again));+ expect(Object.isFrozen(payload)).toBe(true);+ expect(Object.isFrozen(payload.most_recalled)).toBe(true);+ expect(Object.isFrozen(payload.written_per_day)).toBe(true);+ expect(payload.schema_version).toBe(1);+ expect(payload.generated_at).toBe("2026-05-20T12:00:00.000Z");+ expect(payload.telemetry.total).toBe(0);+ expect(payload.telemetry.by_channel).toEqual({});+ expect(payload.most_recalled).toEqual([]);+ expect(payload.report_snapshots).toEqual({ status: "disabled" });+ });++ test("telemetry summary is the summarizeRecallTelemetry passthrough", () => {+ emit("2026-05-20T15:00:00.000Z", [{ id: "pref-a" }, { id: "pref-b" }]);+ emit("2026-05-20T15:01:00.000Z", [], { mode: "query", channel: RECALL_CHANNEL.mcp });+ emitRecallTelemetry(vault, {+ createdAt: "2026-05-20T15:02:00.000Z",+ host: "unit-test",+ channel: RECALL_CHANNEL.cli,+ mode: "search",+ status: "empty",+ durationMs: 3,+ resultCount: 0,+ gaps: ["no_matching_context"],+ });+ const payload = collectExplorerConsole(vault, { now: NOW });+ expect(payload.telemetry).toEqual(summarizeRecallTelemetry(vault));+ expect(payload.telemetry.total).toBe(3);+ expect(payload.telemetry.by_channel).toEqual({ cli: 2, mcp: 1 });+ });++ test("most-recalled ranks by count and joins id and path identities", () => {+ const pathA = join(vault, "Brain", "preferences", "pref-a.md");+ // pref-a is referenced by id+path twice, so it tops the list; the+ // id-only/pref-b and id-only/pref-c singles tie below it. A fourth+ // artifact is referenced by PATH only.+ emit("2026-05-20T15:00:00.000Z", [{ id: "pref-a", path: pathA }, { id: "pref-b" }]);+ emit("2026-05-20T16:00:00.000Z", [{ id: "pref-a", path: pathA }]);+ emit("2026-05-20T16:30:00.000Z", [{ id: "pref-c" }]);+ emitPathOnlyArtifact("2026-05-20T17:00:00.000Z", join(vault, "Daily", "2026-05-20.md"));++ const { most_recalled } = collectExplorerConsole(vault, { now: NOW });+ expect(most_recalled[0]).toEqual({+ id: "pref-a",+ path: pathA,+ access_count: 2,+ last_access_at: "2026-05-20T16:00:00.000Z",+ });+ // The path-only artifact keeps its identity: id absent, path named.+ const pathOnly = most_recalled.find((entry) => entry.id === null);+ expect(pathOnly).toBeDefined();+ expect(pathOnly!.path).toBe(join(vault, "Daily", "2026-05-20.md"));+ expect(pathOnly!.access_count).toBe(1);+ expect(most_recalled.length).toBe(4);+ });++ test("most-recalled is capped at the named limit with a deterministic tie-break", () => {+ expect(EXPLORER_CONSOLE_MOST_RECALLED_LIMIT).toBe(10);+ const artifacts = Array.from({ length: 12 }, (_, i) => ({+ id: `pref-${String(i + 1).padStart(2, "0")}`,+ }));+ emit("2026-05-20T15:00:00.000Z", artifacts);+ const { most_recalled } = collectExplorerConsole(vault, { now: NOW });+ expect(most_recalled.length).toBe(EXPLORER_CONSOLE_MOST_RECALLED_LIMIT);+ // Equal counts break ties by ascending id, so the cap names the+ // same entries on every GET.+ expect(most_recalled.map((entry) => entry.id)).toEqual(+ artifacts.slice(0, EXPLORER_CONSOLE_MOST_RECALLED_LIMIT).map((artifact) => artifact.id),+ );+ });++ test("written-per-day histogram buckets the 30-day window from continuity createdAt", () => {+ expect(EXPLORER_CONSOLE_WINDOW_DAYS).toBe(30);+ emit("2026-05-20T09:00:00.000Z", [{ id: "pref-a" }]);+ emit("2026-05-05T09:00:00.000Z", [{ id: "pref-a" }]);+ // Outside the window: more than 30 days before NOW.+ emit("2026-04-01T09:00:00.000Z", [{ id: "pref-a" }]);+ // A non-telemetry continuity record counts too: every append is a+ // write event.+ appendContinuitySourceInvalidation(vault, {+ createdAt: "2026-05-20T10:00:00.000Z",+ source: { id: "pref-a" },+ reason: "unit-test",+ });++ const { written_per_day, window_since, window_until } = collectExplorerConsole(vault, {+ now: NOW,+ });+ expect(written_per_day.length).toBe(EXPLORER_CONSOLE_WINDOW_DAYS);+ expect(written_per_day[0]!.date).toBe("2026-04-21");+ expect(written_per_day[written_per_day.length - 1]!.date).toBe("2026-05-20");+ expect(window_since).toBe("2026-04-21T00:00:00.000Z");+ expect(window_until).toBe("2026-05-20T23:59:59.999Z");+ const byDate = new Map(written_per_day.map((bucket) => [bucket.date, bucket.count]));+ expect(byDate.get("2026-05-20")).toBe(2);+ expect(byDate.get("2026-05-05")).toBe(1);+ expect(byDate.get("2026-04-01")).toBeUndefined();+ // Zero days are reported as zero, so the axis is fixed.+ expect(byDate.get("2026-05-06")).toBe(0);+ });++ test("report snapshots read as disabled while the feature is off, even with files on disk", () => {+ mkdirSync(brainDirs(vault).brain, { recursive: true });+ writeReportSnapshot(vault, "daily", "2026-05-19", { kept: true });+ const { report_snapshots } = collectExplorerConsole(vault, { now: NOW });+ expect(report_snapshots).toEqual({ status: "disabled" });+ });++ test("report snapshots surface the latest valid date per surface when enabled", () => {+ process.env[ENV_KEY] = "true";+ mkdirSync(brainDirs(vault).brain, { recursive: true });+ writeReportSnapshot(vault, "daily", "2026-05-18", { day: 18 });+ writeReportSnapshot(vault, "daily", "2026-05-19", { day: 19 });+ writeReportSnapshot(vault, "weekly", "2026-05-12", { week: 20 });+ // A torn snapshot file must not become the latest date.+ writeFileSync(join(brainDirs(vault).brain, "reports", "daily", "2026-05-20.json"), "{torn");++ const { report_snapshots } = collectExplorerConsole(vault, { now: NOW });+ expect(report_snapshots.status).toBe("ok");+ if (report_snapshots.status !== "ok") return;+ expect(report_snapshots.surfaces).toEqual([+ { surface: "daily", latest_date: "2026-05-19" },+ { surface: "weekly", latest_date: "2026-05-12" },+ ]);+ });+});diff --git a/tests/core/brain/explorer.test.ts b/tests/core/brain/explorer.test.tsindex b9161d80..3c96f19c 100644--- a/tests/core/brain/explorer.test.ts+++ b/tests/core/brain/explorer.test.ts@@ -14,12 +14,13 @@*/import { afterEach, beforeEach, describe, expect, test } from "bun:test";-import { mkdirSync, mkdtempSync, rmSync } from "node:fs";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";import { basename, join } from "node:path";import { buildBacklinkIndex, backlinkCount } from "../../../src/core/brain/backlinks.ts";import {+ buildLiveServer,collectExplorerData,EXPLORER_SCHEMA_VERSION,renderExportedHtml,@@ -29,6 +30,7 @@ import {writePreference,type WritePreferenceInput,} from "../../../src/core/brain/preference.ts";+import { emitRecallTelemetry } from "../../../src/core/brain/recall-telemetry.ts";import {BRAIN_CONFIDENCE,BRAIN_PREFERENCE_STATUS,@@ -362,3 +364,178 @@ describe("renderExportedHtml", () => {expect(html).toContain("selectedId");});});++describe("buildLiveServer JSON routes", () => {+ const REPORT_SNAPSHOTS_ENV = "OPEN_SECOND_BRAIN_REPORT_SNAPSHOTS";+ let previousEnv: string | undefined;++ beforeEach(() => {+ previousEnv = process.env[REPORT_SNAPSHOTS_ENV];+ delete process.env[REPORT_SNAPSHOTS_ENV];+ });+ afterEach(() => {+ if (previousEnv === undefined) delete process.env[REPORT_SNAPSHOTS_ENV];+ else process.env[REPORT_SNAPSHOTS_ENV] = previousEnv;+ });++ test("serves /counts.json from the console collector, re-read per GET", async () => {+ emitRecallTelemetry(vault, {+ host: "unit-test",+ channel: "cli",+ mode: "search",+ status: "ok",+ durationMs: 1,+ resultCount: 1,+ topArtifacts: [{ id: "pref-seed" }],+ });+ const server = buildLiveServer(vault, 0);+ try {+ const first = await fetch(new URL("/counts.json", server.url));+ expect(first.ok).toBe(true);+ const body = JSON.parse(await first.text()) as {+ schema_version: number;+ telemetry: { total: number };+ written_per_day: ReadonlyArray<unknown>;+ report_snapshots: unknown;+ };+ expect(body.schema_version).toBe(1);+ expect(body.telemetry.total).toBe(1);+ expect(body.written_per_day.length).toBe(30);+ expect(body.report_snapshots).toEqual({ status: "disabled" });+ // A second GET re-reads: a record appended between requests is+ // visible without restarting the server.+ emitRecallTelemetry(vault, {+ host: "unit-test",+ channel: "cli",+ mode: "query",+ status: "empty",+ durationMs: 1,+ resultCount: 0,+ });+ const second = await fetch(new URL("/counts.json", server.url));+ const again = JSON.parse(await second.text()) as { telemetry: { total: number } };+ expect(again.telemetry.total).toBe(2);+ } finally {+ await server.close();+ }+ });++ test("serves /status.json from the operator snapshot with configPath threaded through", async () => {+ writePreference(vault, basePref("routed"));+ // The active profile is read from the profiles registry beside the+ // config path, so it is the observable for the additive options+ // object reaching buildOperatorSnapshot.+ const configPath = join(vault, "config.yaml");+ writeFileSync(+ join(vault, "profiles.json"),+ JSON.stringify({ active: "work", profiles: { work: { vault } } }),+ );+ const server = buildLiveServer(vault, 0, { configPath });+ try {+ const res = await fetch(new URL("/status.json", server.url));+ expect(res.ok).toBe(true);+ const body = JSON.parse(await res.text()) as {+ activeProfile: string | null;+ healthy: boolean;+ counts: { preferences: number };+ };+ expect(body.activeProfile).toBe("work");+ expect(typeof body.healthy).toBe("boolean");+ expect(body.counts.preferences).toBe(1);+ } finally {+ await server.close();+ }+ // Without the options object no config path reaches the snapshot,+ // so the profile lookup reads as none.+ const bare = buildLiveServer(vault, 0);+ try {+ const res = await fetch(new URL("/status.json", bare.url));+ const body = JSON.parse(await res.text()) as { activeProfile: string | null };+ expect(body.activeProfile).toBeNull();+ } finally {+ await bare.close();+ }+ });++ test("unknown routes still answer 404", async () => {+ const server = buildLiveServer(vault, 0);+ try {+ const res = await fetch(new URL("/nope.json", server.url));+ expect(res.status).toBe(404);+ } finally {+ await server.close();+ }+ });++ test("a broken counts section surfaces by name as a 500 JSON error", async () => {+ // With snapshots enabled the collector lists Brain/reports; a FILE+ // parked at that path makes the listing throw, and the route must+ // answer with the error message rather than an empty payload.+ process.env[REPORT_SNAPSHOTS_ENV] = "true";+ mkdirSync(join(vault, "Brain"), { recursive: true });+ writeFileSync(join(vault, "Brain", "reports"), "not a directory");+ const server = buildLiveServer(vault, 0);+ try {+ const res = await fetch(new URL("/counts.json", server.url));+ expect(res.status).toBe(500);+ const body = JSON.parse(await res.text()) as { error: string };+ expect(typeof body.error).toBe("string");+ expect(body.error).toContain("reports");+ } finally {+ await server.close();+ }+ });++ test("a non-loopback Host header is refused with a named 403 on every route", async () => {+ // The loopback BIND is not a browser boundary: a DNS-rebinding page+ // resolves an attacker hostname to 127.0.0.1 and the request arrives+ // carrying that hostname. The server itself must refuse it, on the+ // graph route and the new JSON routes alike.+ const server = buildLiveServer(vault, 0);+ try {+ const refusedOn = async (route: string): Promise<{ status: number; error: string }> => {+ const res = await fetch(new URL(route, server.url), {+ headers: { host: "attacker.example" },+ });+ const body = JSON.parse(await res.text()) as { error: string };+ return { status: res.status, error: body.error };+ };+ const forHtml = await refusedOn("/");+ expect(forHtml.status).toBe(403);+ expect(forHtml.error).toContain("loopback");+ const forGraph = await refusedOn("/data.json");+ expect(forGraph.status).toBe(403);+ expect(forGraph.error).toContain("loopback");+ const forCounts = await refusedOn("/counts.json");+ expect(forCounts.status).toBe(403);+ expect(forCounts.error).toContain("loopback");+ const forStatus = await refusedOn("/status.json");+ expect(forStatus.status).toBe(403);+ expect(forStatus.error).toContain("loopback");+ } finally {+ await server.close();+ }+ });++ test("a loopback Host name with any port is accepted", async () => {+ writePreference(vault, basePref("hosted"));+ const server = buildLiveServer(vault, 0);+ try {+ const servedWith = async (host: string): Promise<number> => {+ const res = await fetch(new URL("/counts.json", server.url), {+ headers: { host },+ });+ const body = JSON.parse(await res.text()) as { schema_version: number };+ return res.ok ? body.schema_version : -1;+ };+ // The default request already carries 127.0.0.1:<port>.+ const direct = await fetch(new URL("/counts.json", server.url));+ expect(direct.ok).toBe(true);+ // The other loopback spellings, port included, are the allowlist.+ expect(await servedWith(`localhost:${server.port}`)).toBe(1);+ expect(await servedWith(`[::1]:${server.port}`)).toBe(1);+ } finally {+ await server.close();+ }+ });+});diff --git a/tests/core/brain/extract-signals-prefilter.test.ts b/tests/core/brain/extract-signals-prefilter.test.tsindex bf2d52ac..b9f09ee4 100644--- a/tests/core/brain/extract-signals-prefilter.test.ts+++ b/tests/core/brain/extract-signals-prefilter.test.ts@@ -22,17 +22,26 @@import { afterEach, beforeEach, describe, expect, test } from "bun:test";import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";-import { join } from "node:path";+import { dirname, join } from "node:path";import { listContinuityRecords } from "../../../src/core/brain/continuity/store.ts";import { planExtractSignals } from "../../../src/core/brain/extract-signals.ts";import {+ EXTRACT_MINING_TOMBSTONE_SKIP_REASON,EXTRACT_PREFILTER_SKIP_REASON,EXTRACT_PREFILTER_TOKEN_SOURCE,planExtractSignalsPrefiltered,resolveExtractPrefilterConfig,} from "../../../src/core/brain/extract-signals-prefilter.ts";-import { importSessionRecall } from "../../../src/core/brain/session-recall.ts";+import {+ minedTurnsHash,+ recordSessionMiningOutcome,+ sessionMiningLedgerPath,+} from "../../../src/core/brain/session-mining-ledger.ts";+import {+ importSessionRecall,+ listSessionRawTurns,+} from "../../../src/core/brain/session-recall.ts";import type { SessionTurn } from "../../../src/core/brain/sessions/types.ts";import type { ResolvedDecisionModelConfig } from "../../../src/core/decision-model/config.ts";import type { DecisionDegradeReason } from "../../../src/core/decision-model/contract.ts";@@ -415,3 +424,163 @@ test("private regions never reach the provider", async () => {expect(JSON.stringify(provider.requests[0]!.state)).not.toContain("secret-lane");expect(JSON.stringify(provider.requests[0]!.state)).toContain("Prefer tabs.");});++// ----- The zero-yield mining tombstone gate (session-mining-economy) -------+//+// 8. A mining record whose hash matches the session's current turn+// material suppresses the plan with the tombstone reason and the+// recorded breakdown as evidence, before the pre-filter spends+// anything: no request, no record.+// 9. Changed turn material re-opens the session and the pre-filter runs+// as before; the pre-filter's own skip reason is untouched.++/** Records a zero-yield tombstone over the session's current turn material. */+function seedTombstone(+ sessionId: string,+ outcome = { written: 0, staged: 0, deduped: 2, durabilityRejected: 1 },+): void {+ const full = planExtractSignals(vault, sessionId, { now: NOW });+ recordSessionMiningOutcome(+ vault,+ { sessionId, minedContentHash: minedTurnsHash(full.turnsMined), outcome },+ NOW,+ );+}++describe("mining tombstone", () => {+ test("a matching tombstone suppresses the plan before any pre-filter request", async () => {+ seedTombstone(SESSION);+ const provider = new FakeDecisionProvider({ answer: () => 0.9 });+ const plan = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: config("enforce"),+ provider,+ });+ expect(plan.llmStep).toBeNull();+ expect(plan.skipped).toEqual({+ reason: EXTRACT_MINING_TOMBSTONE_SKIP_REASON,+ miningOutcome: { written: 0, staged: 0, deduped: 2, durabilityRejected: 1 },+ });+ expect(plan.turnsMined).toEqual([]);+ expect(plan.turnsDropped).toBeUndefined();+ expect(plan.decisionModel).toBeUndefined();+ // The gate ran before the pre-filter: no request, no record, no spend.+ expect(provider.requests).toHaveLength(0);+ expect(listDecisionModelCalls(vault)).toHaveLength(0);+ });++ test("the tombstone suppresses in shadow mode and with the use off alike", async () => {+ seedTombstone(SESSION);+ const suppresses = async (decisionModel: ResolvedDecisionModelConfig | null): Promise<void> => {+ const provider = new FakeDecisionProvider({ answer: () => 0.9 });+ const plan = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel,+ provider,+ });+ expect(plan.llmStep).toBeNull();+ expect(plan.skipped?.reason).toBe(EXTRACT_MINING_TOMBSTONE_SKIP_REASON);+ expect(provider.requests).toHaveLength(0);+ };+ await suppresses(config("shadow"));+ await suppresses(null);+ });++ test("changed turn material re-opens the session and the pre-filter runs", async () => {+ seedTombstone(SESSION);+ importSessionRecall(vault, {+ sessionId: SESSION,+ turns: [turn("t9", "user", "A turn imported after the barren mine.")],+ createdAt: NOW.toISOString(),+ });+ const provider = new FakeDecisionProvider({ answer: () => 0.9 });+ const plan = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: config("shadow"),+ provider,+ });+ expect(plan.skipped).toBeUndefined();+ expect(plan.llmStep).not.toBeNull();+ expect(provider.requests).toHaveLength(1);+ });++ test("the three plan refusals still throw, tombstone or not", async () => {+ // A session whose turns are all non-user can carry a tombstone (it has+ // turns, so the ledger keeps one), but the refusal fires before the+ // gate is ever consulted. The entry is seeded over the raw turns,+ // because the plan that would compute the hash refuses first.+ importSessionRecall(vault, {+ sessionId: "sess-quiet",+ turns: [turn("q1", "assistant", "Only I spoke here.")],+ createdAt: NOW.toISOString(),+ });+ const raw = listSessionRawTurns(vault, "sess-quiet");+ recordSessionMiningOutcome(+ vault,+ {+ sessionId: "sess-quiet",+ minedContentHash: minedTurnsHash(+ raw.map((t) => ({ turnId: t.turn_id, text: t.text.trim(), timestamp: t.timestamp })),+ ),+ outcome: { written: 0, staged: 0, deduped: 0, durabilityRejected: 0 },+ },+ NOW,+ );+ const provider = new FakeDecisionProvider({ answer: () => 0.9 });+ await expect(+ planExtractSignalsPrefiltered(vault, "sess-quiet", {+ now: NOW,+ decisionModel: config("enforce"),+ provider,+ }),+ ).rejects.toThrow(/user/);+ expect(provider.requests).toHaveLength(0);+ });++ test("a record with a yield does not suppress - the session stays minable", async () => {+ // The distilled state: the commit recorded a yield over the current+ // material. That is the census's distilled evidence, not a tombstone,+ // so the plan still runs (and a re-mine re-records over the same+ // hash).+ const full = planExtractSignals(vault, SESSION, { now: NOW });+ recordSessionMiningOutcome(+ vault,+ {+ sessionId: SESSION,+ minedContentHash: minedTurnsHash(full.turnsMined),+ outcome: { written: 3, staged: 0, deduped: 1, durabilityRejected: 0 },+ },+ NOW,+ );+ const provider = new FakeDecisionProvider({ answer: () => 0.9 });+ const plan = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: config("enforce"),+ provider,+ });+ expect(plan.skipped).toBeUndefined();+ expect(plan.llmStep).not.toBeNull();+ expect(provider.requests).toHaveLength(1);+ });++ test("force bypasses the hit, never the read: a corrupt ledger refuses a forced plan", async () => {+ // The escape hatch's safety story, from the option's own contract:+ // `force` ignores a matching entry, never a corrupt file. Reordering+ // the read behind the force check would turn the escape into a ledger+ // bypass; this pins the refusal on BOTH paths.+ mkdirSync(dirname(sessionMiningLedgerPath(vault)), { recursive: true });+ writeFileSync(sessionMiningLedgerPath(vault), "{ truncated");+ for (const force of [false, true]) {+ try {+ await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: null,+ force,+ });+ throw new Error(`expected the plan to refuse (force: ${String(force)})`);+ } catch (err) {+ expect((err as Error).message).toContain("session mining ledger is corrupted JSON");+ }+ }+ });+});diff --git a/tests/core/brain/extract-signals.test.ts b/tests/core/brain/extract-signals.test.tsindex c5898fec..b3c0ff18 100644--- a/tests/core/brain/extract-signals.test.ts+++ b/tests/core/brain/extract-signals.test.ts@@ -21,12 +21,27 @@* partial write it is: the error names the items already on disk and* the one that failed, rather than leaving the caller to discover* them.+ * 21. A session whose mining record matches its current turn material is+ * suppressed at plan time with a typed reason and the recorded+ * breakdown as evidence, and stays suppressed while unchanged.+ * 22. A re-import that changes the turns re-opens the session with no+ * clearing step.+ * 23. A corrupt mining ledger refuses the plan by name; it is never read+ * as an empty ledger.*/import { afterEach, beforeEach, expect, test } from "bun:test";-import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs";+import {+ existsSync,+ mkdirSync,+ mkdtempSync,+ readFileSync,+ readdirSync,+ rmSync,+ writeFileSync,+} from "node:fs";import { tmpdir } from "node:os";-import { join } from "node:path";+import { dirname, join } from "node:path";import {AUTO_EXTRACT_CONFIDENCE_FLOOR,@@ -39,6 +54,18 @@ import {} from "../../../src/core/brain/extract-signals.ts";import { listDeadLetters } from "../../../src/core/brain/dead-letter.ts";import { NEEDS_LLM_STEP } from "../../../src/core/brain/llm-step.ts";+import {+ EXTRACT_MINING_TOMBSTONE_SKIP_REASON,+ planExtractSignalsPrefiltered,+} from "../../../src/core/brain/extract-signals-prefilter.ts";+import {+ minedTurnsHash,+ readSessionMiningLedger,+ recordSessionMiningOutcome,+ SESSION_MINING_LEDGER_SCHEMA_VERSION,+ sessionMiningLedgerPath,+ type SessionMiningEntry,+} from "../../../src/core/brain/session-mining-ledger.ts";import {ResponseCheckError,SEMANTIC_VIOLATION_CODES,@@ -122,6 +149,9 @@ test("phase one carries exactly one spine envelope over the user turns", () => {expect(plan.sessionId).toBe(SESSION);expect(plan.turnsScanned).toBe(3);expect(plan.turnsMined.map((t) => t.turnId)).toEqual(["t1", "t3"]);+ // The surveyed hash names the material the plan put in front of the+ // caller; the commit passes it back so the mining record keys on it.+ expect(plan.surveyedTurnsHash).toBe(minedTurnsHash(plan.turnsMined));expect(plan.llmStep.status).toBe(NEEDS_LLM_STEP);expect(plan.llmStep.prompt).toContain("Always name the release theme");// The assistant turn is not mining material.@@ -522,3 +552,297 @@ test("distinct topics pass the duplicate-topic rule", () => {);expect(res.written.map((w) => w.topic)).toEqual(["release-notes-style", "module-names"]);});++// ----- The zero-yield mining tombstone at plan time (session-mining-economy)+//+// 21. A session whose mining record matches its current turn material is+// suppressed at plan time with a typed reason and the recorded+// breakdown as evidence - never an empty plan that reads as a clean+// run - and stays suppressed while the content is unchanged.+// 22. A re-import that changes the turns re-opens the session with no+// clearing step.+// 23. A corrupt mining ledger refuses the plan by name; it is never read+// as an empty ledger.++/** Records a zero-yield tombstone over the session's current mined material. */+function seedTombstone(): void {+ const full = planExtractSignals(vault, SESSION, { now: NOW });+ recordSessionMiningOutcome(+ vault,+ {+ sessionId: SESSION,+ minedContentHash: minedTurnsHash(full.turnsMined),+ outcome: { written: 0, staged: 0, deduped: 2, durabilityRejected: 0 },+ },+ NOW,+ );+}++test("a matching tombstone suppresses the plan with the recorded breakdown", async () => {+ seedTombstone();+ const plan = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: null,+ });+ expect(plan.llmStep).toBeNull();+ expect(plan.skipped).toEqual({+ reason: EXTRACT_MINING_TOMBSTONE_SKIP_REASON,+ miningOutcome: { written: 0, staged: 0, deduped: 2, durabilityRejected: 0 },+ });+ expect(plan.turnsMined).toEqual([]);+ // Unchanged content stays suppressed on every later scan.+ const again = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: null,+ });+ expect(again.skipped).toEqual(plan.skipped);+});++test("a re-import that changes the turns re-opens the session", async () => {+ seedTombstone();+ importTurns(SESSION, [turn("t9", "user", "A turn imported after the barren mine.")]);+ const plan = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: null,+ });+ expect(plan.skipped).toBeUndefined();+ expect(plan.llmStep).not.toBeNull();+ expect(plan.turnsMined.map((t) => t.turnId)).toEqual(["t1", "t3", "t9"]);+});++test("a corrupt mining ledger refuses the plan by name", async () => {+ seedTombstone();+ const path = sessionMiningLedgerPath(vault);+ writeFileSync(path, "{ truncated");+ try {+ await planExtractSignalsPrefiltered(vault, SESSION, { now: NOW, decisionModel: null });+ throw new Error("expected the plan to refuse");+ } catch (err) {+ expect((err as Error).message).toContain("session mining ledger is corrupted JSON");+ }+ // The ledger was never reset by the refused read.+ expect(readFileSync(path, "utf8")).toBe("{ truncated");+});++test("the session mining ledger schema version is pinned at 1", () => {+ // Its own test, not a stapled assertion inside the corrupt-ledger+ // error path: the value is load-bearing (the ledger's unknown-schema+ // refusal names it) but it is a constant of the read contract, not+ // part of any error-path contract.+ expect(SESSION_MINING_LEDGER_SCHEMA_VERSION).toBe(1);+});++// ----- Record-at-commit (session-mining-economy) ----------------------------+//+// 24. Every successful commit that answers a plan records its outcome+// breakdown over the surveyed hash: zero-yield as the tombstone's+// evidence, yield as the distilled state the census reports.+// 25. A commit that ends in a partial-write error records nothing - a+// failed run is not an outcome.+// 26. The tombstone keys on the material the model actually answered, so a+// turn imported between plan and commit is never suppressed unseen; a+// commit that answers no plan of this module's attests nothing and+// records nothing, which is the safe direction.+// 27. When the commit itself succeeds but the ledger write fails, the+// result names the failure and nothing throws; the field is absent+// whenever the ledger was not attempted or succeeded.++function ledgerEntry(): SessionMiningEntry | undefined {+ return readSessionMiningLedger(vault).entries[SESSION];+}++/** A commit that answers `planExtractSignals`, attesting the surveyed hash. */+function commitAnsweringPlan(+ items: ReadonlyArray<Record<string, unknown>>,+ opts: Partial<{ durabilityDenylist: ReadonlyArray<RegExp>; writeApprovalEnabled: boolean }> = {},+): { plan: ReturnType<typeof planExtractSignals>; res: ReturnType<typeof commitExtractedSignals> } {+ const plan = planExtractSignals(vault, SESSION, { now: NOW });+ const res = commitExtractedSignals(+ vault,+ SESSION,+ { items },+ {+ agent: "tester",+ now: NOW,+ surveyedTurnsHash: plan.surveyedTurnsHash,+ ...opts,+ },+ );+ return { plan, res };+}++test("an empty answer records the zero-yield tombstone over the planned material", () => {+ const { plan, res } = commitAnsweringPlan([]);+ expect(res.written).toEqual([]);+ const entry = ledgerEntry();+ expect(entry).toBeDefined();+ expect(entry!.minedContentHash).toBe(plan.surveyedTurnsHash);+ expect(entry!.minedContentHash).toBe(minedTurnsHash(plan.turnsMined));+ expect(entry!.outcome).toEqual({ written: 0, staged: 0, deduped: 0, durabilityRejected: 0 });+ expect(entry!.recordedAt).toBe("2026-08-22T10:00:00Z");+});++test("a commit whose items all dedupe records the zero-yield breakdown", () => {+ commitAnsweringPlan([item()]);+ const { res: second } = commitAnsweringPlan([item()]);+ expect(second.written).toEqual([]);+ expect(second.deduped).toBe(1);+ expect(ledgerEntry()!.outcome).toEqual({+ written: 0,+ staged: 0,+ deduped: 1,+ durabilityRejected: 0,+ });+});++test("a commit whose items all fail the durability gate records the refusals", () => {+ const { res } = commitAnsweringPlan(+ [item({ topic: "noise", principle: "temporary scratch note" })],+ {+ durabilityDenylist: [/scratch/],+ },+ );+ expect(res.written).toEqual([]);+ expect(res.durabilityRejected).toBe(1);+ expect(ledgerEntry()!.outcome).toEqual({+ written: 0,+ staged: 0,+ deduped: 0,+ durabilityRejected: 1,+ });+});++test("a yielding commit records its yield - the distilled state, not a tombstone", () => {+ const { plan, res } = commitAnsweringPlan([item()]);+ expect(res.written).toHaveLength(1);+ expect(pendingFiles().length).toBe(0);+ const entry = ledgerEntry();+ expect(entry).toBeDefined();+ expect(entry!.minedContentHash).toBe(plan.surveyedTurnsHash);+ expect(entry!.outcome).toEqual({ written: 1, staged: 0, deduped: 0, durabilityRejected: 0 });+});++test("a staged yield records itself the same way", () => {+ const { res } = commitAnsweringPlan([item()], { writeApprovalEnabled: true });+ expect(res.staged).toBe(1);+ expect(ledgerEntry()!.outcome).toEqual({+ written: 1,+ staged: 1,+ deduped: 0,+ durabilityRejected: 0,+ });+});++test("a partial-write failure records nothing - a failed run is not a barren one", () => {+ expect(() =>+ commitExtractedSignals(+ vault,+ SESSION,+ { items: [item({ topic: "module:names" })] },+ { agent: "tester", now: NOW },+ ),+ ).toThrow(ExtractSignalsWriteError);+ expect(inboxFiles()).toEqual([]);+ expect(readSessionMiningLedger(vault).entries).toEqual({});+});++test("the tombstone keys on the surveyed hash, not the turns present at commit time", () => {+ const plan = planExtractSignals(vault, SESSION, { now: NOW });+ importTurns(SESSION, [turn("t9", "user", "Imported after the plan was built.")]);+ commitExtractedSignals(+ vault,+ SESSION,+ { items: [] },+ {+ agent: "tester",+ now: NOW,+ surveyedTurnsHash: plan.surveyedTurnsHash,+ },+ );+ const entry = ledgerEntry()!;+ // The verdict was earned over the planned material, and that is the+ // material the record describes - never what the session holds by now.+ expect(entry.minedContentHash).toBe(minedTurnsHash(plan.turnsMined));+});++test("a turn imported between plan and commit is not suppressed unseen", async () => {+ const plan = planExtractSignals(vault, SESSION, { now: NOW });+ importTurns(SESSION, [turn("t9", "user", "Imported after the plan was built.")]);+ commitExtractedSignals(+ vault,+ SESSION,+ { items: [] },+ {+ agent: "tester",+ now: NOW,+ surveyedTurnsHash: plan.surveyedTurnsHash,+ },+ );+ // The tombstone covers the old material only, so the next plan still+ // offers the new turn to a model.+ const next = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: null,+ });+ expect(next.skipped).toBeUndefined();+ expect(next.turnsMined.map((t) => t.turnId)).toEqual(["t1", "t3", "t9"]);+});++test("a commit that answers no plan attests nothing and records nothing", () => {+ // Without a plan there is no material the verdict was earned over; a+ // tombstone keyed on whatever the session holds now could suppress turns+ // no model ever saw. Recording nothing keeps the session eligible.+ commitExtractedSignals(vault, SESSION, { items: [] }, { agent: "tester", now: NOW });+ expect(readSessionMiningLedger(vault).entries).toEqual({});+});++/** Plants a corrupt mining ledger so a read of it refuses by name. */+function seedCorruptLedger(): void {+ mkdirSync(dirname(sessionMiningLedgerPath(vault)), { recursive: true });+ writeFileSync(sessionMiningLedgerPath(vault), "{ truncated");+}++test("a ledger write failure on a successful commit is named, never thrown", () => {+ // The plan is built (and surveys the ledger) before the corruption, the+ // way a real round does: the commit then attests its hash and the record+ // attempt hits the corrupt file.+ const plan = planExtractSignals(vault, SESSION, { now: NOW });+ seedCorruptLedger();+ const res = commitExtractedSignals(+ vault,+ SESSION,+ { items: [] },+ {+ agent: "tester",+ now: NOW,+ surveyedTurnsHash: plan.surveyedTurnsHash,+ },+ );+ expect(res.written).toEqual([]);+ expect(res.miningLedgerFailure).toContain("session mining ledger is corrupted JSON");+ // The vault was never touched by the failed record.+ expect(inboxFiles()).toEqual([]);+});++test("a yield-bearing commit names a ledger failure too - it records like any other", () => {+ const plan = planExtractSignals(vault, SESSION, { now: NOW });+ seedCorruptLedger();+ const res = commitExtractedSignals(+ vault,+ SESSION,+ { items: [item()] },+ {+ agent: "tester",+ now: NOW,+ surveyedTurnsHash: plan.surveyedTurnsHash,+ },+ );+ expect(res.written).toHaveLength(1);+ expect(res.miningLedgerFailure).toContain("session mining ledger is corrupted JSON");+});++test("the ledger-failure field is absent when the record succeeded", () => {+ const { res } = commitAnsweringPlan([]);+ expect(ledgerEntry()).toBeDefined();+ expect("miningLedgerFailure" in res).toBe(false);+});diff --git a/tests/core/brain/hygiene-retire-routing.test.ts b/tests/core/brain/hygiene-retire-routing.test.tsindex 68ec0eba..26446444 100644--- a/tests/core/brain/hygiene-retire-routing.test.ts+++ b/tests/core/brain/hygiene-retire-routing.test.ts@@ -152,13 +152,23 @@ describe("applyHygienePlan dispatches retire to the retirement core", () => {expect(result.applied[0]!.action).toBe("retire");expect(result.applied[0]!.finding_id).toBe(finding.id);expect(result.applied[0]!.detail).toContain(canonical);+ // The per-copy link counts ride the detail (review M7): the executor+ // used to report only "retired N", dropping what the link pass did.+ expect(result.applied[0]!.detail).toMatch(/links: \d+ rewritten, \d+ failed/);+ // The disk accounting rides it too (review M6): snapshot bytes,+ // removed bytes, and the signed net between them.+ expect(result.applied[0]!.detail).toMatch(+ /disk: snapshot \d+ bytes, removed \d+ bytes, net [+-]\d+ bytes/,+ );expect(existsSync(join(vault, retired))).toBe(false);expect(existsSync(join(vault, canonical))).toBe(true);- // The manifest entry dropped is the RETIRED copy's source; the- // canonical copy's source stays recorded.- expect(readManifest(vault).entries["imports/a.md"]).toBeUndefined();+ // The manifest entry MARKED retired is the RETIRED copy's source; the+ // canonical copy's source stays recorded without the marker (H3: the+ // marker, not a drop, is what keeps the next batch plan honest).+ expect(readManifest(vault).entries["imports/a.md"]?.retired_into).toBe(canonical);expect(readManifest(vault).entries["imports/b.md"]).toBeDefined();+ expect(readManifest(vault).entries["imports/b.md"]?.retired_into).toBeUndefined();expect(listContinuityRecords(vault, { kind: "source_invalidation" }).some((r) => r.id !== null),).toBe(true);diff --git a/tests/core/brain/import-census.test.ts b/tests/core/brain/import-census.test.tsindex 72fbe971..cbba1d94 100644--- a/tests/core/brain/import-census.test.ts+++ b/tests/core/brain/import-census.test.ts@@ -30,7 +30,11 @@ import { DEFAULT_BRAIN_CONFIG_YAML } from "../../../src/core/brain/config-templaimport { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";import { brainDirs } from "../../../src/core/brain/paths.ts";import { computeDedupHash } from "../../../src/core/brain/dedup-hash.ts";-import { updateManifest } from "../../../src/core/brain/ingest/content-manifest.ts";+import {+ readManifest,+ updateManifest,+ writeManifestAtomic,+} from "../../../src/core/brain/ingest/content-manifest.ts";import { importSession } from "../../../src/core/brain/sessions/import.ts";import {buildReadBackCensus,@@ -205,6 +209,35 @@ describe("censusIngestedPaths - the ingest lane reads back its content hashes",expect(census.outcome).toBe(RECONCILIATION_OUTCOME.complete);});+ test("a retired source whose bytes are unchanged is found, not a lost write", () => {+ // The resurrect guard marks the entry retired_into a canonical page+ // and leaves the SOURCE on disk (only the derived copy is removed).+ // The read-back then runs again over the same plan's completed set+ // (reconcilePlan). Recorded, on disk, still hashing to the recorded+ // digest - that is the found definition, so the marker must not+ // turn a claim the vault can reproduce into a false lost-write+ // alarm.+ writeVaultFile("Docs/a.md", "alpha\n");+ updateManifest(vault, ["Docs/a.md"]);+ const manifest = readManifest(vault);+ writeManifestAtomic(+ vault,+ {+ ...manifest.entries,+ "Docs/a.md": {+ ...manifest.entries["Docs/a.md"]!,+ retired_into: "Brain/sources/src-canonical.md",+ },+ },+ manifest.contract,+ );++ const census = censusIngestedPaths(vault, ["Docs/a.md"]);+ expect(census.found).toBe(1);+ expect([...census.missing]).toEqual([]);+ expect(census.outcome).toBe(RECONCILIATION_OUTCOME.complete);+ });+test("a claimed path deleted after the write is named", () => {writeVaultFile("Docs/a.md", "alpha\n");writeVaultFile("Docs/b.md", "beta\n");diff --git a/tests/core/brain/ingest/batch-plan.test.ts b/tests/core/brain/ingest/batch-plan.test.tsindex 87b4e1f2..98ee86c8 100644--- a/tests/core/brain/ingest/batch-plan.test.ts+++ b/tests/core/brain/ingest/batch-plan.test.ts@@ -14,7 +14,11 @@ import { tmpdir } from "node:os";import { bootstrapBrain } from "../../../../src/core/brain/init.ts";import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";-import { hashFile, updateManifest } from "../../../../src/core/brain/ingest/content-manifest.ts";+import {+ hashFile,+ readManifest,+ updateManifest,+} from "../../../../src/core/brain/ingest/content-manifest.ts";import {DEFAULT_INGESTIBLE_EXTENSIONS as PLANNER_DEFAULT_EXTENSIONS,planBatches,@@ -54,6 +58,19 @@ function allPlannedPaths(plan: ReturnType<typeof planBatches>): string[] {const CAPS = { maxBatchBytes: 10_000, maxBatchFiles: 100 } as const;+/**+ * Mark one recorded manifest entry `retired_into` the given canonical page,+ * the way the retirement core does: the pinned record keeps the digest and+ * fingerprint it was recorded under and gains the marker.+ */+function markRetired(vaultDir: string, rel: string, canonicalPage: string): void {+ const recorded = readManifest(vaultDir).entries[rel];+ if (recorded === undefined) throw new Error(`fixture bug: ${rel} is not in the manifest`);+ updateManifest(vaultDir, [rel], {+ [rel]: { ...recorded, retired_into: canonicalPage },+ });+}+/** Write a page carrying an optional `schema_type` frontmatter field. */function page(rel: string, schemaType?: string): void {const abs = join(vault, rel);@@ -92,6 +109,60 @@ describe("planBatches — discovery + skip-unchanged", () => {expect(plan.totalFiles).toBe(2);});+ test("a retired source whose bytes are unchanged is never queued", () => {+ // The resurrect guard (H3): a near-duplicate retirement marks the+ // entry `retired_into` instead of dropping it, and the planner skips+ // the marked entry while its bytes stay unchanged.+ writeSized("Inbox/retired.md", 100);+ writeSized("Inbox/live.md", 100);+ updateManifest(vault, ["Inbox/retired.md", "Inbox/live.md"]);+ markRetired(vault, "Inbox/retired.md", "Brain/sources/src-live.md");++ const plan = planBatches(vault, "Inbox", CAPS);+ // Both recorded sources are queued-out: the plain-unchanged one through+ // the ordinary skip, the retired one through the resurrect guard.+ expect(allPlannedPaths(plan)).toEqual([]);+ expect(plan.skipped).toEqual(["Inbox/live.md"]);+ expect(plan.skippedRetired).toEqual(["Inbox/retired.md"]);++ // A content change re-admits the retired source as `modified`.+ writeSized("Inbox/retired.md", 120);+ const changed = planBatches(vault, "Inbox", CAPS);+ expect(allPlannedPaths(changed)).toContain("Inbox/retired.md");+ expect(changed.skippedRetired).toEqual([]);+ });++ test("a retired source stays queued-out even under a changed contract", () => {+ // A contract change reprocesses `unchanged` sources; a retirement is+ // not a content change, so the marked entry keeps its skip.+ setExtractable(["paper"]);+ writeSized("Inbox/retired.md", 100);+ page("Inbox/retired.md", "paper");+ updateManifest(vault, ["Inbox/retired.md"]);+ markRetired(vault, "Inbox/retired.md", "Brain/sources/src-elsewhere.md");+ // Widen the allowlist: the contract changes, the bytes do not.+ setExtractable(["paper", "memo"]);++ const plan = planBatches(vault, "Inbox", CAPS);+ expect(plan.contractChanged).toBe(true);+ expect(allPlannedPaths(plan)).toEqual([]);+ expect(plan.skippedRetired).toEqual(["Inbox/retired.md"]);+ });++ test("the wire carries skipped_retired only when non-empty", () => {+ writeSized("Docs/a.md", 50);+ updateManifest(vault, ["Docs/a.md"]);++ // No retired entry: byte-identical to the pre-H3 wire shape.+ const plain = serializeBatchPlan(planBatches(vault, "Docs", CAPS));+ expect(Object.keys(plain)).not.toContain("skipped_retired");++ markRetired(vault, "Docs/a.md", "Brain/sources/src-a.md");+ const wire = serializeBatchPlan(planBatches(vault, "Docs", CAPS));+ expect(wire["skipped_retired"]).toEqual(["Docs/a.md"]);+ expect(wire["total_files"]).toBe(0);+ });+test("only ingestible (text-bearing) files are discovered; binaries/media ignored", () => {writeSized("Docs/a.md", 50);writeSized("Docs/b.txt", 50);diff --git a/tests/core/brain/ingest/content-manifest.test.ts b/tests/core/brain/ingest/content-manifest.test.tsindex c6040f53..b4db168f 100644--- a/tests/core/brain/ingest/content-manifest.test.ts+++ b/tests/core/brain/ingest/content-manifest.test.ts@@ -116,6 +116,69 @@ describe("classifyPaths", () => {expect(res.unchanged).toEqual(["touched.md"]);expect(res.modified).toEqual([]);});++ test("an entry marked retired_into classifies `retired` while its bytes are unchanged", () => {+ // The resurrect guard (H3): a near-duplicate retirement marks the entry+ // instead of dropping it, so the unchanged source never reads as `new`.+ writeSource("dup.md", "duplicate bytes");+ updateManifest(vault, ["dup.md"]);+ const recorded = readManifest(vault).entries["dup.md"]!;+ writeManifestAtomic(vault, {+ "dup.md": { ...recorded, retired_into: "Brain/sources/src-canonical.md" },+ });++ const res = classifyPaths(vault, ["dup.md"], readManifest(vault));+ expect(res.retired).toEqual(["dup.md"]);+ expect(res.new).toEqual([]);+ expect(res.unchanged).toEqual([]);+ expect(res.modified).toEqual([]);+ });++ test("a retired entry whose bytes changed re-admits as `modified`", () => {+ writeSource("dup.md", "duplicate bytes");+ updateManifest(vault, ["dup.md"]);+ const recorded = readManifest(vault).entries["dup.md"]!;+ writeManifestAtomic(vault, {+ "dup.md": { ...recorded, retired_into: "Brain/sources/src-canonical.md" },+ });++ writeSource("dup.md", "the duplicate grew");+ const res = classifyPaths(vault, ["dup.md"], readManifest(vault));+ expect(res.modified).toEqual(["dup.md"]);+ expect(res.retired).toEqual([]);+ });++ test("a retired entry round-trips: it never serializes down to the v1 digest shape", () => {+ writeSource("dup.md", "duplicate bytes");+ updateManifest(vault, ["dup.md"]);+ const recorded = readManifest(vault).entries["dup.md"]!;+ writeManifestAtomic(vault, {+ "dup.md": { ...recorded, retired_into: "Brain/sources/src-canonical.md" },+ });++ // Data a write could read back is data the write must keep: the marker+ // survives a re-read and a re-write under the read-back contract.+ const reread = readManifest(vault).entries["dup.md"]!;+ expect(reread.retired_into).toBe("Brain/sources/src-canonical.md");+ writeManifestAtomic(vault, { "dup.md": reread });+ const onDisk = JSON.parse(readFileSync(manifestPath(vault), "utf8")) as {+ entries: Record<string, { retired_into?: string }>;+ };+ expect(onDisk.entries["dup.md"]?.retired_into).toBe("Brain/sources/src-canonical.md");+ });++ test("a malformed retired_into on disk drops the field, never the entry", () => {+ writeManifestBytes(+ JSON.stringify({+ schema_version: 3,+ contract: { fingerprint: computeExtractionContractFingerprint(vault) },+ entries: { "dup.md": { sha256: "a".repeat(64), retired_into: 42 } },+ }),+ );+ const entry = readManifest(vault).entries["dup.md"];+ expect(entry?.sha256).toBe("a".repeat(64));+ expect(entry?.retired_into).toBeUndefined();+ });});describe("manifest persistence", () => {@@ -503,6 +566,60 @@ describe("extraction watermark (t_a9cd4221)", () => {...WATERMARK,});});++ test("a requested watermark is stat'ed and hashed inside the manifest lock", () => {+ // The L1 shape: the pinned form carries a digest computed OUTSIDE the+ // lock, so two concurrent runs over an appended source can land the+ // older digest-and-watermark pair last. The request form computes+ // neither value until the lock is held: updateManifest reads the file+ // itself, so the recorded pair always describes the bytes the write+ // saw, not the bytes the caller once saw.+ writeSource("Inbox/grow.md", "append-only log, revision one\n");+ updateManifest(+ vault,+ ["Inbox/grow.md"],+ {},+ {+ watermarks: { "Inbox/grow.md": { at: "2026-06-13T12:00:00Z" } },+ },+ );+ const entry = readManifest(vault).entries["Inbox/grow.md"]!;+ expect(entry.sha256).toBe(hashFile(join(vault, "Inbox", "grow.md")));+ expect(entry.extracted_bytes).toBe(Buffer.byteLength("append-only log, revision one\n"));+ expect(entry.extracted_at).toBe("2026-06-13T12:00:00Z");+ expect(entry.fingerprint).toBe(computeExtractionContractFingerprint(vault));+ });++ test("a watermark request supersedes a stale caller-precomputed pair", () => {+ // The race, in one test: the caller hashes the OLD bytes (as ingest.ts+ // did before the lock), the source appends, and only then does the+ // manifest write take the lock. The recorded entry must describe the+ // CURRENT bytes, never the stale pair.+ writeSource("Inbox/grow.md", "revision one\n");+ const staleDigest = hashFile(join(vault, "Inbox", "grow.md"));+ const staleSize = Buffer.byteLength("revision one\n");+ writeSource("Inbox/grow.md", "revision one\nrevision two appended\n");++ updateManifest(+ vault,+ ["Inbox/grow.md"],+ {+ "Inbox/grow.md": {+ sha256: staleDigest,+ fingerprint: computeExtractionContractFingerprint(vault),+ extracted_bytes: staleSize,+ extracted_at: "2026-06-13T11:00:00Z",+ },+ },+ { watermarks: { "Inbox/grow.md": { at: "2026-06-13T12:00:00Z" } } },+ );++ const entry = readManifest(vault).entries["Inbox/grow.md"]!;+ expect(entry.sha256).toBe(hashFile(join(vault, "Inbox", "grow.md")));+ expect(entry.sha256).not.toBe(staleDigest);+ expect(entry.extracted_bytes).toBe(Buffer.byteLength("revision one\nrevision two appended\n"));+ expect(entry.extracted_at).toBe("2026-06-13T12:00:00Z");+ });});describe("ingestSource integration", () => {diff --git a/tests/core/brain/ingest/incremental-extract.test.ts b/tests/core/brain/ingest/incremental-extract.test.tsindex ff2e03f2..06cd38cb 100644--- a/tests/core/brain/ingest/incremental-extract.test.ts+++ b/tests/core/brain/ingest/incremental-extract.test.ts@@ -14,23 +14,64 @@* - a REWRITTEN source - even one that grew - classifies modified and* reprocesses fully, resetting the watermark, because the recorded* digest no longer describes the leading bytes;+ * - a PROVEN append whose tail passes the delta cap answers the named+ * `appended-over-cap` outcome carrying the whole appended range - never+ * silently the same answer as a rewrite (review L3); the run stays a+ * whole-source pass, so the stamp still covers the full current file;* - EXTRACT_CONTRACT_VERSION stays at 2: the watermark changes what is* REPORTED, not what extraction produces, so no contract bump and no* one-time mass reprocess is owed.*/-import { mkdirSync, mkdtempSync, rmSync, appendFileSync, writeFileSync } from "node:fs";+import * as fsNamespace from "node:fs";+import { mkdirSync, mkdtempSync, rmSync, appendFileSync, writeFileSync, statSync } from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";-import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { afterEach, beforeEach, describe, expect, mock, test } from "bun:test";++// Positioned-read observer for the delta pass. The ONLY positioned readSync+// reachable from ingestSource is the append-proof snapshot's byte walk; the+// trust and archive lanes read with a null position and the in-lock stamp+// hashes through readFileSync. Counting positioned bytes against the source's+// inode therefore measures exactly the delta pass, with no behavior change+// (every call is forwarded untouched). Same forwarding-mock idiom as+// digest-actions.test.ts: capture the originals BEFORE the mock.+const realReadSync = fsNamespace.readSync;+const realFstatSync = fsNamespace.fstatSync;+let positionedReadTarget: { readonly dev: number; readonly ino: number } | undefined;+let positionedReadBytes = 0;++mock.module("node:fs", () => ({+ ...fsNamespace,+ readSync(+ fd: number,+ buffer: Buffer,+ offset: number,+ length: number,+ position: number | null,+ ): number {+ const count = realReadSync(fd, buffer, offset, length, position);+ if (positionedReadTarget !== undefined && typeof position === "number" && count > 0) {+ try {+ const st = realFstatSync(fd);+ if (st.dev === positionedReadTarget.dev && st.ino === positionedReadTarget.ino) {+ positionedReadBytes += count;+ }+ } catch {+ // An unstattable fd is not the observed read; the count stays.+ }+ }+ return count;+ },+}));import { runCli } from "../../../helpers/run-cli.ts";import { EXTRACT_CONTRACT_VERSION } from "../../../../src/core/brain/ingest/contract.ts";import {classifyPaths,+ hashBytes,hashFile,- hashPath,readManifest,} from "../../../../src/core/brain/ingest/content-manifest.ts";import { planBatches } from "../../../../src/core/brain/ingest/batch-plan.ts";@@ -73,14 +114,54 @@ describe("ingestSource watermark stamping", () => {expect(entry.sha256).toBe(hashFile(join(vault, SOURCE)));});+ test("a first ingest issues no positioned read against the source", () => {+ // Several positioned-read chunks wide, so a wasted whole-source walk+ // would show up in the counter; a skipped one does not.+ writeSource("first-ingest line of a larger source\n".repeat(5000));+ const size = statSync(join(vault, SOURCE)).size;++ const sourceStat = statSync(join(vault, SOURCE));+ positionedReadTarget = { dev: sourceStat.dev, ino: sourceStat.ino };+ const before = positionedReadBytes;+ let res: ReturnType<typeof ingestSource>;+ try {+ res = ingestSource(vault, INPUT, OPTS);+ } finally {+ positionedReadTarget = undefined;+ }+ // With no watermark nothing can be proven and no tail can exist, so the+ // append-proof machinery has no byte of the source to read: the run's+ // only whole-source pass is the in-lock stamp hash (readFileSync,+ // invisible to this counter).+ expect(positionedReadBytes - before).toBe(0);+ // The observable answers the skip must preserve: no delta, and the+ // full-size in-lock stamp exactly as before.+ expect(res.extractionDelta).toBeUndefined();+ const entry = readManifest(vault).entries[SOURCE]!;+ expect(entry.extracted_bytes).toBe(size);+ expect(entry.sha256).toBe(hashFile(join(vault, SOURCE)));+ });+test("an appended source reports the appended delta and advances the watermark", () => {writeSource("one\ntwo\n");ingestSource(vault, INPUT, OPTS);appendFileSync(join(vault, SOURCE), "three\n", "utf8");- const res = ingestSource(vault, INPUT, OPTS);-- expect(res.extractionDelta).toEqual({ fromByte: 8, toByte: 14 });+ // Positive control for the positioned-read counter the first-ingest+ // test asserts against: the append path really does read the delta+ // through a positioned read, so a silently broken mock shows up here+ // instead of passing both tests vacuously.+ const sourceStat = statSync(join(vault, SOURCE));+ positionedReadTarget = { dev: sourceStat.dev, ino: sourceStat.ino };+ let res: ReturnType<typeof ingestSource>;+ try {+ res = ingestSource(vault, INPUT, OPTS);+ } finally {+ positionedReadTarget = undefined;+ }+ expect(positionedReadBytes).toBeGreaterThan(0);++ expect(res.extractionDelta).toEqual({ kind: "appended", fromByte: 8, toByte: 14 });const entry = readManifest(vault).entries[SOURCE]!;expect(entry.extracted_bytes).toBe(14);expect(entry.sha256).toBe(hashFile(join(vault, SOURCE)));@@ -94,8 +175,8 @@ describe("ingestSource watermark stamping", () => {appendFileSync(join(vault, SOURCE), "four\n", "utf8");const third = ingestSource(vault, INPUT, OPTS);- expect(second.extractionDelta).toEqual({ fromByte: 8, toByte: 14 });- expect(third.extractionDelta).toEqual({ fromByte: 14, toByte: 19 });+ expect(second.extractionDelta).toEqual({ kind: "appended", fromByte: 8, toByte: 14 });+ expect(third.extractionDelta).toEqual({ kind: "appended", fromByte: 14, toByte: 19 });// The watermarks advanced stepwise; no delta covers another's bytes.expect(readManifest(vault).entries[SOURCE]!.extracted_bytes).toBe(19);});@@ -118,15 +199,55 @@ describe("ingestSource watermark stamping", () => {expect(entry.sha256).toBe(hashFile(join(vault, SOURCE)));});- test("an appended tail with no complete line reports an empty delta at the watermark", () => {+ test("an appended tail with no complete line reports an empty delta and leaves the watermark", () => {writeSource("one\ntwo\n");ingestSource(vault, INPUT, OPTS);appendFileSync(join(vault, SOURCE), "half a line", "utf8");const res = ingestSource(vault, INPUT, OPTS);- expect(res.extractionDelta).toEqual({ fromByte: 8, toByte: 8 });- // The run still covered the whole current source, so the stamp is full.- expect(readManifest(vault).entries[SOURCE]!.extracted_bytes).toBe(19);+ expect(res.extractionDelta).toEqual({ kind: "appended", fromByte: 8, toByte: 8 });+ // The stamp covers exactly the announced prefix (review M15): the+ // unterminated fragment stays unstamped, and the next append's delta+ // starts AT it instead of past it.+ const entry = readManifest(vault).entries[SOURCE]!;+ expect(entry.extracted_bytes).toBe(8);+ expect(entry.sha256).toBe(hashBytes(new Uint8Array(Buffer.from("one\ntwo\n", "utf8"))));+ });++ test("a fragment stamp keeps the prefix proof consistent and the next delta stitches the fragment", () => {+ // The review's M15 probe: the delta cut ends at the last complete+ // newline, so the stamp must land there - at the bytes the delta+ // series actually announced - and never at the raw file size.+ writeSource("base\n");+ ingestSource(vault, INPUT, OPTS);+ expect(readManifest(vault).entries[SOURCE]!.extracted_bytes).toBe(5);++ appendFileSync(join(vault, SOURCE), "line1\nline2-part", "utf8");+ const second = ingestSource(vault, INPUT, OPTS);+ expect(second.extractionDelta).toEqual({ kind: "appended", fromByte: 5, toByte: 11 });++ // The stamp lands at the announced boundary (11), NOT at the file+ // size (21), and the digest describes exactly those leading bytes.+ const stamped = readManifest(vault).entries[SOURCE]!;+ expect(stamped.extracted_bytes).toBe(11);+ expect(stamped.sha256).toBe(hashBytes(new Uint8Array(Buffer.from("base\nline1\n", "utf8"))));++ // Accepted consequence (review M15): with the stamp behind EOF the+ // file classifies modified until the fragment is terminated - one+ // reclassification, never a stale skip.+ expect(classifyPaths(vault, [SOURCE], readManifest(vault)).modified).toEqual([SOURCE]);++ appendFileSync(join(vault, SOURCE), "-rest\nline3\n", "utf8");+ const third = ingestSource(vault, INPUT, OPTS);+ // The fragment rides the next delta instead of falling out of the+ // delta series for good.+ expect(third.extractionDelta).toEqual({ kind: "appended", fromByte: 11, toByte: 33 });+ const healed = readManifest(vault).entries[SOURCE]!;+ expect(healed.extracted_bytes).toBe(33);+ // The boundary is back at EOF, so the digest is a whole-file digest+ // again and the unchanged classification holds.+ expect(healed.sha256).toBe(hashFile(join(vault, SOURCE)));+ expect(classifyPaths(vault, [SOURCE], readManifest(vault)).unchanged).toEqual([SOURCE]);});});@@ -188,11 +309,24 @@ describe("the CLI extract preview reports the appended slice", () => {const res = await runCli(["brain", "extract", join(vault, SOURCE), "--json", "--vault", vault]);expect(res.returncode).toBe(0);const payload = JSON.parse(res.stdout) as { appended?: unknown };- // The preview refuses exactly what the pipeline refuses: it must- // never report a slice the delta pass would not accept.+ // The preview refuses exactly what the pipeline refuses to render: it+ // must never report a slice the delta pass would not accept.expect(payload.appended).toBeUndefined();const second = ingestSource(vault, INPUT, OPTS);- expect(second.extractionDelta).toBeUndefined();+ // The pipeline itself names the over-cap append instead of answering+ // the same shape a rewrite gets (review L3): the whole appended range,+ // no text.+ const size = statSync(join(vault, SOURCE)).size;+ expect(second.extractionDelta).toEqual({+ kind: "appended-over-cap",+ fromByte: 8,+ toByte: size,+ });+ // The run stayed a whole-source pass: the stamp covers the full+ // current file, digest included.+ const entry = readManifest(vault).entries[SOURCE]!;+ expect(entry.extracted_bytes).toBe(size);+ expect(entry.sha256).toBe(hashFile(join(vault, SOURCE)));});test("a rewritten source that grew answers without a section, like the pipeline", async () => {@@ -225,13 +359,12 @@ describe("a source that changes kind mid-run degrades fail-safe", () => {// after the intake has already classified the source trusted (its stat// saw a regular file) and just before the manifest stamp, so swapping// the source for a directory inside the callback lands the change in- // the window between the classification stat and the pre-stamp reads.- // The stamp block then stats a directory whose size sits past the- // recorded watermark, and the append-proof read used to die on EISDIR -- // after the summary pages were written, losing the run's result. The- // sibling ENOENT window (the source deleted inside the same span,- // between the block's existsSync and the stat/hash reads) is answered- // by the same fail-safe catch; it has no deterministic public-API hook.+ // the window between the classification stat and the single-pass+ // append snapshot. Opening a directory refuses (EISDIR), and the ONE+ // descriptor the pass reads answers for both the delta and the stamp:+ // no provable append is reported, and no stamp lands that the pass+ // could not also verify - the last good entry stands, and the source+ // reprocesses when its real bytes return.let swapped = false;const second = ingestSource(vault, INPUT, {...OPTS,@@ -246,19 +379,18 @@ describe("a source that changes kind mid-run degrades fail-safe", () => {});expect(swapped).toBe(true);- // No crash: the run answers, and a source that cannot be read as a file- // reports no append.+ // No crash: the run answers, and a source that cannot be read as a+ // file reports no append.expect(second.summaryPath).toBe(first.summaryPath);expect(second.extractionDelta).toBeUndefined();- // The stamp that landed describes the directory as it now is, so the- // next classification sees a changed source and reprocesses fully- // rather than trusting a stale file digest.+ // The manifest keeps describing the last read the pipeline could+ // actually take - the file as it stood at the first ingest.const entry = readManifest(vault).entries[SOURCE]!;- expect(entry.sha256).toBe(hashPath(join(vault, SOURCE)));- expect(entry.extracted_bytes).toBeGreaterThan(0);+ expect(entry.sha256).toBe(hashBytes(new Uint8Array(Buffer.from("one\ntwo\n", "utf8"))));+ expect(entry.extracted_bytes).toBe(8);- // The restored file is not an append against that stamp: the source- // reprocesses fully, and the stamp lands on the real file again.+ // The restored file is byte-identical to that standing stamp: it+ // classifies unchanged, and no delta is reported.rmSync(join(vault, SOURCE), { recursive: true });writeSource("one\ntwo\n");const third = ingestSource(vault, INPUT, OPTS);diff --git a/tests/core/brain/maintenance-capacity-gate.test.ts b/tests/core/brain/maintenance-capacity-gate.test.tsindex 427f1bea..b8505dcf 100644--- a/tests/core/brain/maintenance-capacity-gate.test.ts+++ b/tests/core/brain/maintenance-capacity-gate.test.ts@@ -19,7 +19,7 @@*/import { afterEach, beforeEach, describe, expect, test } from "bun:test";-import { appendFileSync, existsSync, mkdirSync, mkdtempSync, rmSync } from "node:fs";+import { appendFileSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";@@ -33,10 +33,12 @@ import {} from "../../../src/core/brain/maintenance/journal.ts";import { acquireLease, currentLease } from "../../../src/core/brain/maintenance/lease.ts";import { BRAIN_STORAGE_DEFAULTS } from "../../../src/core/brain/policy/blocks/storage.ts";+import { MAINTENANCE_CAPACITY_REMEDY } from "../../../src/core/brain/maintenance/lane.ts";import {STORAGE_FORECAST_TIER,STORAGE_USAGE_METRIC_SURFACE,} from "../../../src/core/brain/storage/forecast.ts";+import { evaluateIngestAdmission } from "../../../src/core/brain/storage/gate.ts";import type { CapacityGateDecision } from "../../../src/core/brain/storage/gate.ts";import type { ResolvedBrainStorageConfig } from "../../../src/core/brain/types.ts";import { listMetrics } from "../../../src/core/brain/metrics.ts";@@ -246,6 +248,37 @@ describe("the capacity gate through the lane", () => {expect(samples()).toBe(before + 1);});+ test("a gated run that proceeds records the gate's census as its sample too", async () => {+ // The M5 blind spot: the lane used to sample only on skips, so on a+ // vault where nobody runs the storage report the trend never existed+ // and the day-based tiers could never fire. Every run where the gate+ // evaluated - skip or proceed - now records its census, one walk,+ // never two.+ configureCap(ROOMY_CAP_BYTES);+ const before = samples();+ const result = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [{ name: "dream", run: async () => {} }],+ });+ expect(result.verdict).toBe(MAINTENANCE_VERDICT.run);+ expect(samples()).toBe(before + 1);+ const recorded = listMetrics(vault, { surface: STORAGE_USAGE_SURFACE });+ expect(recorded[recorded.length - 1]!.payload["measured"]).toBe(true);+ });++ test("a run where the capacity gate never evaluated samples nothing", async () => {+ // No cap configured: the gate never measured, so there is no census+ // to record and the lane must not invent one with a second walk.+ const before = samples();+ await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [{ name: "dream", run: async () => {} }],+ });+ expect(samples()).toBe(before);+ });+test("the skip's sample records the gate's census, not a second walk", async () => {configureCap(CRITICAL_CAP_BYTES);// A decision whose census names a byte total the real vault cannot@@ -302,7 +335,7 @@ describe("the capacity gate through the lane", () => {expect(journalRows(MAINTENANCE_VERDICT.skippedLease)).toEqual([]);});- test("a sub-critical vault runs and samples nothing", async () => {+ test("a sub-critical vault runs", async () => {configureCap(ROOMY_CAP_BYTES);const before = samples();const result = await runMaintenance(vault, {@@ -312,7 +345,68 @@ describe("the capacity gate through the lane", () => {});expect(result.verdict).toBe(MAINTENANCE_VERDICT.run);expect(result.tasks[0]?.ok).toBe(true);- expect(samples()).toBe(before);+ // The gate evaluated, so the census was recorded (the sample-every-+ // gated-run rule above) - that is the ONE sample a proceeding run+ // costs, and no skip row exists.+ expect(samples()).toBe(before + 1);+ expect(journalRows(MAINTENANCE_VERDICT.skippedCapacity)).toEqual([]);+ });++ test("the warn band pauses the lane, with the report and journal row to show it", async () => {+ // The M5 backpressure band: background work pauses one band BEFORE+ // the wall. The bootstrapped skeleton sits inside the warn band of a+ // 30 KB cap - a tier the lane used to run straight through, because+ // only critical skipped and critical was reserved for the ingest+ // refusal.+ configureCap(30_000);+ const ran: string[] = [];+ const result = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [{ name: "dream", run: async () => void ran.push("dream") }],+ });+ expect(result.verdict).toBe(MAINTENANCE_VERDICT.skippedCapacity);+ expect(ran).toEqual([]);+ expect(result.capacity?.tier).toBe(STORAGE_FORECAST_TIER.warn);+ expect(result.capacity?.capacity_percent).not.toBeNull();+ expect(result.capacity?.pause_percent).toBe(BRAIN_STORAGE_DEFAULTS.warn_percent);+ const row = journalRows(MAINTENANCE_VERDICT.skippedCapacity)[0];+ expect(row?.capacity_tier).toBe(STORAGE_FORECAST_TIER.warn);+ // The census a paused run recorded (measured: true) is what the+ // trend fits while producers are backed off.+ const recorded = listMetrics(vault, { surface: STORAGE_USAGE_SURFACE });+ expect(recorded).toHaveLength(1);+ expect(recorded[0]!.payload["measured"]).toBe(true);+ });++ test("a configured pause_percent below the warn band pauses earlier", async () => {+ // 20 percent of 100 KB measured: the tier is ok (far below the warn+ // band) and the operator's pause_percent: 15 pauses the lane anyway+ // - interactive ingest is untouched at this tier.+ configureCap(100_000);+ appendFileSync(brainConfigPath(vault), " pause_percent: 15\n");+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", "AA-page.md"), "x".repeat(20_000), "utf8");+ const result = await runMaintenance(vault, {+ now: NOW,+ holder: "worker-a",+ tasks: [{ name: "dream", run: async () => {} }],+ });+ expect(result.verdict).toBe(MAINTENANCE_VERDICT.skippedCapacity);+ expect(result.capacity?.tier).toBe(STORAGE_FORECAST_TIER.ok);+ expect(result.capacity?.pause_percent).toBe(15);+ expect(result.capacity?.capacity_percent).toBeGreaterThan(15);+ });++ test("an ingest refusal stays critical-only while the lane pauses at warn", async () => {+ // The two walls diverge on purpose: the SAME warn-band vault that+ // pauses the lane (the test above) does not refuse an interactive+ // write - evaluateIngestAdmission carries no pause line.+ configureCap(30_000);+ const admission = evaluateIngestAdmission(vault, { now: NOW });+ expect(admission.decision).not.toBeNull();+ expect(admission.decision!.tier).toBe(STORAGE_FORECAST_TIER.warn);+ expect(admission.decision!.decision).toBe("run");});test("a vault without a cap never measures", async () => {@@ -383,9 +477,20 @@ interface CapacityReportJson {readonly capacity_percent: number | null;readonly measured_bytes: number | null;readonly cap_bytes: number | null;+ readonly pause_percent: number | null;}describe("lane surfaces carry the capacity verdict", () => {+ test("the remedy names pruning the snapshot recovery points", () => {+ // M6: both reclamation verbs the remedy prescribes archive all of+ // Brain/ into Brain/.snapshots - a COUNTED surface - before they+ // free anything, so at 96 percent the "remedy" can first grow the+ // vault. The remedy names the retention knob, or an operator+ // following it blind reads pure shrink where there is a swap.+ expect(MAINTENANCE_CAPACITY_REMEDY).toContain("Brain/.snapshots");+ expect(MAINTENANCE_CAPACITY_REMEDY).toContain("snapshots.retention_count");+ });+function seedSkipRow(): void {appendJournal(vault, {ts: NOW.toISOString(),@@ -423,6 +528,8 @@ describe("lane surfaces carry the capacity verdict", () => {expect(payload.capacity?.capacity_percent).toBe((payload.capacity!.measured_bytes! / CRITICAL_CAP_BYTES) * 100,);+ // The pause line the lane decided on rides beside the reading.+ expect(payload.capacity?.pause_percent).toBe(BRAIN_STORAGE_DEFAULTS.warn_percent);});test("a skipped-capacity run reports the tier and the remedy over MCP", async () => {@@ -455,5 +562,6 @@ describe("lane surfaces carry the capacity verdict", () => {expect(payload.capacity?.capacity_percent).toBe((payload.capacity!.measured_bytes! / CRITICAL_CAP_BYTES) * 100,);+ expect(payload.capacity?.pause_percent).toBe(BRAIN_STORAGE_DEFAULTS.warn_percent);});});diff --git a/tests/core/brain/receipts-compact.test.ts b/tests/core/brain/receipts-compact.test.tsindex 4afb2d34..a6d5ab87 100644--- a/tests/core/brain/receipts-compact.test.ts+++ b/tests/core/brain/receipts-compact.test.ts@@ -33,13 +33,18 @@ import {continuityLogPath,listContinuityRecords,} from "../../../src/core/brain/continuity/store.ts";-import { metricsDir } from "../../../src/core/brain/metrics.ts";+import { metricsDir, appendMetric } from "../../../src/core/brain/metrics.ts";import {compactReceiptLedgers,RECEIPT_AUDIT_APPEND_FAILED,} from "../../../src/core/brain/receipts/compact.ts";-import { RECEIPT_SHARD_UNREADABLE } from "../../../src/core/brain/receipts/scan.ts";import {+ RECEIPT_SHARD_FOREIGN_DEVICE,+ RECEIPT_SHARD_UNREADABLE,+} from "../../../src/core/brain/receipts/scan.ts";+import { compactView } from "../../../src/core/brain/receipts/views.ts";+import {+ RECEIPT_COMPACTION_OPERATION,RECEIPT_FOLD_SCHEMA,RECEIPT_LEDGER_FAMILY,} from "../../../src/core/brain/receipts/types.ts";@@ -50,6 +55,7 @@ import { resetVaultIdentityPins } from "../../../src/core/brain/vault-identity.timport { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";import { WriteRefusedError } from "../../../src/core/brain/write-disposition.ts";import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";+import { withDeviceId } from "../../helpers/device-id.ts";import { digestVaultTree } from "../../helpers/vault-digest.ts";const NOW = new Date("2026-09-25T00:00:00Z");@@ -88,6 +94,11 @@ function readShard(dir: string, name: string): string {return readFileSync(join(dir, name), "utf8");}+/** Byte length of one record line, newline separators excluded. */+function recordLineBytes(line: string): number {+ return Buffer.byteLength(line.trimEnd(), "utf8");+}+/** The continuity shard the happy-path tests fold: two old, one protected, one young. */const MIXED_SHARD = [continuityLine("recall_telemetry", "2026-08-10T12:00:00Z", { note: "first old" }),@@ -347,6 +358,42 @@ describe("compactReceiptLedgers", () => {expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(audits);});+ test("the confirmed result reports the recovery point's bytes and the net delta", () => {+ // M6 (confirm half): the recovery point is a whole-Brain archive+ // written BEFORE the ledger shrink lands, into a surface the storage+ // census counts as vault content. A result carrying only the ledger+ // shrink reads as pure win even where the net effect is growth. The+ // confirmed result names both sides: the archive's bytes, and the+ // ledger shrink minus the recovery point's growth.+ writeShard(continuityLogDir(vault), "2026-08.jsonl", MIXED_SHARD);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.snapshotRunId).toBeDefined();+ expect(result.snapshotBytes).toBeGreaterThan(0);+ const shrink = result.totals.bytesBefore - result.totals.bytesAfter;+ expect(shrink).toBeGreaterThan(0);+ expect(result.netDeltaBytes).toBe(shrink - result.snapshotBytes!);+ // A dry run takes no recovery point, so it names neither.+ writeShard(continuityLogDir(vault), "2026-07.jsonl", [+ continuityLine("recall_telemetry", "2026-07-10T12:00:00Z"),+ ]);+ const preview = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-07.jsonl" }],+ now: NOW,+ dryRun: true,+ });+ expect(preview.snapshotBytes).toBeUndefined();+ expect(preview.netDeltaBytes).toBeUndefined();+ // Both figures ride the shared wire view.+ const view = compactView(result);+ expect(view["snapshot_bytes"]).toBe(result.snapshotBytes);+ expect(view["net_delta_bytes"]).toBe(result.netDeltaBytes);+ expect(compactView(preview)["snapshot_bytes"]).toBeUndefined();+ });+test("a shard whose fold would not shrink is collected and the rest of the run completes", () => {// One old record whose aggregate line would be LARGER than the// record itself: the engine refuses the rewrite rather than grow a@@ -409,6 +456,220 @@ describe("compactReceiptLedgers", () => {expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(0);});+ test("the review probe: entity mentions and the audit trail survive two confirmed runs", () => {+ // The H1 probe the external review ran: a 2026-08 shard holding 28+ // session_entity_mention records and 28 receipt_compaction records+ // ended 0 records / 4 aggregates after two confirmed runs - the+ // cross-session entity index and the fold's own audit trail folded+ // away, because folding used to be the default for every kind outside+ // a short protected list. Folding is now opt-in per kind and both+ // kinds are data, not telemetry: nothing here may fold, so two runs+ // leave all 56 records in place and write no aggregate at all.+ const lines: string[] = [];+ for (let i = 0; i < 28; i += 1) {+ const day = String((i % 25) + 1).padStart(2, "0");+ lines.push(+ continuityLine("session_entity_mention", `2026-08-${day}T12:00:00Z`, {+ session_ref: "imports/a.md",+ key: `sym:S${i}`,+ }),+ );+ lines.push(+ continuityLine("receipt_compaction", `2026-08-${day}T13:00:00Z`, {+ operation: RECEIPT_COMPACTION_OPERATION,+ }),+ );+ }+ writeShard(continuityLogDir(vault), "2026-08.jsonl", lines);+ for (let run = 0; run < 2; run += 1) {+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.totals.folded).toBe(0);+ expect(result.failures).toEqual([]);+ }+ // The read model still sees every record, and no aggregate line was+ // written. (Nothing folded, so the runs appended no audit either.)+ expect(listContinuityRecords(vault, { kind: "session_entity_mention" })).toHaveLength(28);+ expect(listContinuityRecords(vault, { kind: "receipt_compaction" })).toHaveLength(28);+ expect(readShard(continuityLogDir(vault), "2026-08.jsonl")).not.toContain(RECEIPT_FOLD_SCHEMA);+ });++ test("a later run merges into the existing (kind, month) aggregate instead of adding a second line", () => {+ // M13: the cutoff slides with each run and re-imports append new old+ // records into an already-compacted monthly shard, so a shard folded+ // twice used to gain a SECOND aggregate line per (kind, month) - a+ // growing stack the one-aggregate-per-key contract forbids, and whose+ // ~200-byte lines could tip a small fold into the strict-shrink+ // refusal. The fold now merges into the aggregate it finds: count and+ // bytes sum, first and last widen.+ writeShard(continuityLogDir(vault), "2026-08.jsonl", [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z", { note: "first" }),+ continuityLine("recall_telemetry", "2026-08-20T12:00:00Z", { note: "second" }),+ ]);+ const first = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(first.totals.aggregates).toBe(1);+ // A re-import appends another old record into the same monthly shard.+ const compacted = readShard(continuityLogDir(vault), "2026-08.jsonl");+ const reimported = continuityLine("recall_telemetry", "2026-08-15T12:00:00Z", {+ note: "re-imported",+ });+ writeShard(continuityLogDir(vault), "2026-08.jsonl", compacted + reimported);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.totals.folded).toBe(1);+ // The small fold merged instead of failing the strict-shrink guard.+ expect(result.failures).toEqual([]);+ const lines = readShard(continuityLogDir(vault), "2026-08.jsonl")+ .split("\n")+ .filter((line) => line.trim() !== "");+ expect(lines).toHaveLength(1);+ const fold = (JSON.parse(lines[0]!) as { fold: Record<string, unknown> }).fold;+ expect(fold.kind).toBe("recall_telemetry");+ expect(fold.month).toBe("2026-08");+ expect(fold.count).toBe(3);+ expect(fold.first_created_at).toBe("2026-08-10T12:00:00Z");+ expect(fold.last_created_at).toBe("2026-08-20T12:00:00Z");+ expect(fold.bytes).toBe(+ recordLineBytes(+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z", { note: "first" }),+ ) ++ recordLineBytes(+ continuityLine("recall_telemetry", "2026-08-20T12:00:00Z", { note: "second" }),+ ) ++ recordLineBytes(reimported),+ );+ });++ test("a metrics fold merges into the existing (surface, month) aggregate the same way", () => {+ writeShard(metricsDir(vault), "storage_usage.jsonl", [+ metricLine("storage_usage", "2026-08-01T00:00:00Z", { bytes: 10 }),+ metricLine("storage_usage", "2026-08-05T00:00:00Z", { bytes: 11 }),+ ]);+ compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.metrics, name: "storage_usage.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ const compacted = readShard(metricsDir(vault), "storage_usage.jsonl");+ writeShard(metricsDir(vault), "storage_usage.jsonl", [+ compacted,+ metricLine("storage_usage", "2026-08-03T00:00:00Z", { bytes: 12 }),+ ]);+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.metrics, name: "storage_usage.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.totals.folded).toBe(1);+ expect(result.failures).toEqual([]);+ const lines = readShard(metricsDir(vault), "storage_usage.jsonl")+ .split("\n")+ .filter((line) => line.trim() !== "");+ expect(lines).toHaveLength(1);+ const fold = (JSON.parse(lines[0]!) as { fold: Record<string, unknown> }).fold;+ expect(fold.surface).toBe("storage_usage");+ expect(fold.count).toBe(3);+ expect(fold.first_run_at).toBe("2026-08-01T00:00:00Z");+ expect(fold.last_run_at).toBe("2026-08-05T00:00:00Z");+ });++ test("appendMetric holds the shard's lock, so a rewrite window cannot swallow a sample", () => {+ // M12: the metrics append path wrote bare `appendFileSync` with no+ // lock, so a sample landing between the fold engine's compare-and-swap+ // re-read and its rename was overwritten without error - exactly the+ // loss the CAS was built to turn into reported drift. The append now+ // takes the same per-file lock the continuity appender takes, in both+ // directions: a contended append is a loud failure, never a silent+ // overwrite, and a compact that meets a held lock is a collected+ // failure, never a rewrite that wins the race.+ mkdirSync(metricsDir(vault), { recursive: true });+ const shardPath = join(metricsDir(vault), "storage_usage.jsonl");+ writeFileSync(+ shardPath,+ [+ metricLine("storage_usage", "2026-08-01T00:00:00Z", { bytes: 10 }),+ metricLine("storage_usage", "2026-08-02T00:00:00Z", { bytes: 11 }),+ ].join(""),+ "utf8",+ );+ writeFileSync(`${shardPath}.lock`, "held by another writer", "utf8");+ try {+ expect(() =>+ appendMetric(vault, {+ surface: "storage_usage",+ runAt: "2026-09-02T00:00:00Z",+ payload: { bytes: 12 },+ }),+ ).toThrow(/lock busy/);+ // Nothing landed while the lock was held.+ expect(readFileSync(shardPath, "utf8")).not.toContain("2026-09-02");+ // And the fold engine reports the contention instead of rewriting.+ const result = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.metrics, name: "storage_usage.jsonl" }],+ now: NOW,+ dryRun: false,+ });+ expect(result.shards).toEqual([]);+ expect(result.failures).toHaveLength(1);+ expect(result.failures[0]!.shard).toBe("metrics:storage_usage.jsonl");+ expect(result.failures[0]!.message).toContain("locked");+ } finally {+ rmSync(`${shardPath}.lock`, { force: true });+ }+ });++ test("another device's shard is refused by name; this device's shard beside it still folds", () => {+ // M11: shards are named per device so exactly one device writes a+ // given file, but the fold's lock and compare-and-swap are local+ // only - rewriting a peer's shard here is the divergent-copy conflict+ // the shard layout exists to prevent. A named foreign shard is a+ // refusal by name, and the local shard in the same selection folds.+ withDeviceId("", () => {+ writeShard(continuityLogDir(vault), "2026-08.telescope.jsonl", MIXED_SHARD);+ writeShard(continuityLogDir(vault), "2026-07.jsonl", [+ continuityLine("recall_telemetry", "2026-07-10T12:00:00Z", { note: "local a" }),+ continuityLine("recall_telemetry", "2026-07-20T12:00:00Z", { note: "local b" }),+ ]);+ const result = compactReceiptLedgers(vault, {+ shards: [+ { family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.telescope.jsonl" },+ { family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-07.jsonl" },+ ],+ now: NOW,+ dryRun: false,+ });+ expect(result.failures).toEqual([+ {+ shard: "continuity:2026-08.telescope.jsonl",+ message: RECEIPT_SHARD_FOREIGN_DEVICE,+ },+ ]);+ expect(result.totals.folded).toBe(2);+ // The foreign shard is untouched on disk.+ expect(readShard(continuityLogDir(vault), "2026-08.telescope.jsonl")).toBe(MIXED_SHARD);+ // A dry run refuses the same way, so a preview never promises a+ // peer's fold.+ const preview = compactReceiptLedgers(vault, {+ shards: [{ family: RECEIPT_LEDGER_FAMILY.continuity, name: "2026-08.telescope.jsonl" }],+ now: NOW,+ dryRun: true,+ });+ expect(preview.failures).toEqual([+ { shard: "continuity:2026-08.telescope.jsonl", message: RECEIPT_SHARD_FOREIGN_DEVICE },+ ]);+ });+ });+test("a mixed selection reports a compacted shard as nothing-to-do, not a failure", () => {// The re-run contract: a compacted shard folds nothing and writes// nothing. A MIXED selection never reaches the run-leveldiff --git a/tests/core/brain/receipts-scan.test.ts b/tests/core/brain/receipts-scan.test.tsindex 246f23df..316d61ea 100644--- a/tests/core/brain/receipts-scan.test.ts+++ b/tests/core/brain/receipts-scan.test.ts@@ -22,18 +22,24 @@ import { tmpdir } from "node:os";import { join } from "node:path";import { continuityLogDir } from "../../../src/core/brain/continuity/store.ts";+import type { ContinuityRecordKind } from "../../../src/core/brain/continuity/types.ts";+import { CONTINUITY_RECORD_KINDS } from "../../../src/core/brain/continuity/types.ts";import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";import { metricsDir } from "../../../src/core/brain/metrics.ts";import {+ classifyContinuityRecord,RECEIPT_SHARD_UNREADABLE,scanReceiptLedgers,} from "../../../src/core/brain/receipts/scan.ts";import {+ FOLDABLE_CONTINUITY_KINDS,+ KEPT_CONTINUITY_KINDS,RECEIPT_FOLD_DEFAULT_AGE_DAYS,RECEIPT_LEDGER_FAMILY,type ReceiptScanReport,} from "../../../src/core/brain/receipts/types.ts";import { digestVaultTree } from "../../helpers/vault-digest.ts";+import { withDeviceId } from "../../helpers/device-id.ts";const CONTINUITY_SCHEMA = "o2b.continuity.v1";const METRICS_SCHEMA = "o2b.metrics.v1";@@ -99,7 +105,7 @@ describe("scanReceiptLedgers", () => {writeShard(continuity, "2026-08.jsonl", [continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),]);- writeShard(continuity, "2026-09.telescope.jsonl", [+ writeShard(continuity, "2026-09.jsonl", [continuityLine("gate_telemetry", "2026-09-15T12:00:00Z"),]);const report = scanReceiptLedgers(vault, { now: NOW });@@ -108,11 +114,7 @@ describe("scanReceiptLedgers", () => {// 2026-08-10 is 46 days before the pinned now: past the 30-day fold age.expect(august.foldable).toBe(1);expect(august.protectedRecords).toBe(0);- const september = shardOf(- report,- RECEIPT_LEDGER_FAMILY.continuity,- "2026-09.telescope.jsonl",- );+ const september = shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-09.jsonl");// A record inside the window is neither foldable nor protected.expect(september.records).toBe(1);expect(september.foldable).toBe(0);@@ -170,6 +172,124 @@ describe("scanReceiptLedgers", () => {}});+ test("functional continuity kinds are never foldable, whatever their payload", () => {+ // Folding is opt-in per kind: the kinds readers read back as data,+ // not as telemetry, stay out of every aggregate even when their+ // payload carries none of the clip-protected correlation keys.+ // `session_entity_mention` is the case that forced the rule: its+ // payload carries `session_ref`, so the clip-key rule never saw it,+ // and folding it destroyed the cross-session entity index (the+ // session-recall seed, the hallways pass and the mining dedupe seed+ // read the rows back), the by-id `generation_report` lookups, the+ // row-count-derived `recent_turn` sequence and the compaction audit+ // trail of the run before this one.+ vault = mintVault();+ try {+ const functional: ReadonlyArray<ContinuityRecordKind> = [+ "session_entity_mention",+ "generation_report",+ "recent_turn",+ "receipt_compaction",+ "context_receipt",+ "session_turn",+ "session_summary_node",+ "session_summary_digest",+ "host_memory_write",+ "recall_observed_use",+ "negative_recall",+ ];+ const continuity = continuityLogDir(vault);+ writeShard(+ continuity,+ "2026-08.jsonl",+ functional.map((kind, index) =>+ continuityLine(kind, `2026-08-1${index}T12:00:00Z`, {+ session_ref: "imports/a.md",+ }),+ ),+ );+ const report = scanReceiptLedgers(vault, { now: NOW });+ const shard = shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-08.jsonl");+ expect(shard.records).toBe(functional.length);+ expect(shard.foldable).toBe(0);+ expect(shard.protectedRecords).toBe(functional.length);+ expect(report.totals.foldable).toBe(0);+ } finally {+ cleanup();+ }+ });++ test("the pure telemetry kinds remain foldable past the age threshold", () => {+ vault = mintVault();+ try {+ const telemetry: ReadonlyArray<ContinuityRecordKind> = FOLDABLE_CONTINUITY_KINDS;+ const continuity = continuityLogDir(vault);+ writeShard(+ continuity,+ "2026-08.jsonl",+ telemetry.map((kind, index) =>+ continuityLine(kind, `2026-08-01T12:00:${String(index % 60).padStart(2, "0")}Z`),+ ),+ );+ const report = scanReceiptLedgers(vault, { now: NOW });+ const shard = shardOf(report, RECEIPT_LEDGER_FAMILY.continuity, "2026-08.jsonl");+ expect(shard.records).toBe(telemetry.length);+ expect(shard.foldable).toBe(telemetry.length);+ expect(shard.protectedRecords).toBe(0);+ } finally {+ cleanup();+ }+ });++ test("the fold decision census covers every continuity kind in the vocabulary", () => {+ // The census: a kind added to `CONTINUITY_RECORD_KINDS` without a+ // fold decision - a row in the fold allowlist or a row in the+ // explicit keep list - fails here, so the allowlist can never drift+ // open behind the vocabulary. Exactly one decision per kind, and no+ // decision for a kind the vocabulary does not know.+ const kinds = new Set<string>(CONTINUITY_RECORD_KINDS);+ expect(FOLDABLE_CONTINUITY_KINDS.length).toBeGreaterThan(0);+ expect(KEPT_CONTINUITY_KINDS.length).toBeGreaterThan(0);+ for (const kind of FOLDABLE_CONTINUITY_KINDS) {+ expect(kinds.has(kind), `${kind} is foldable but unknown to the vocabulary`).toBe(true);+ }+ for (const kind of KEPT_CONTINUITY_KINDS) {+ expect(kinds.has(kind), `${kind} is kept but unknown to the vocabulary`).toBe(true);+ }+ const decisions = new Set<string>([...FOLDABLE_CONTINUITY_KINDS, ...KEPT_CONTINUITY_KINDS]);+ expect(decisions.size).toBe(FOLDABLE_CONTINUITY_KINDS.length + KEPT_CONTINUITY_KINDS.length);+ for (const kind of CONTINUITY_RECORD_KINDS) {+ expect(decisions.has(kind), `${kind} has no fold decision`).toBe(true);+ }+ // The classifier agrees with the lists it is built from: every+ // foldable kind past the cutoff folds, every kept kind does not.+ const cutoffMs = Date.parse("2026-09-01T00:00:00Z");+ for (const kind of FOLDABLE_CONTINUITY_KINDS) {+ expect(classifyContinuityRecord({ kind, createdAt: "2026-08-01T00:00:00Z" }, cutoffMs)).toBe(+ "foldable",+ );+ }+ for (const kind of KEPT_CONTINUITY_KINDS) {+ expect(classifyContinuityRecord({ kind, createdAt: "2026-08-01T00:00:00Z" }, cutoffMs)).toBe(+ "protected",+ );+ }+ // The ranking-consumed and audit-trail kinds are read back as data,+ // never aggregated as telemetry - the classification the allowlist's+ // own criterion demands, pinned against drifting back onto the+ // foldable side: `recall_observed_use` is the lifetime observed-reuse+ // rate the search ranking consumes unfiltered (`observedReuseRates`+ // via the candidate-signals collector, beside the dead-weight+ // report's read-side fold), and `negative_recall` is the+ // answer-audit trail its own vocabulary entry declares - the corpus+ // statement a "no" rested on stays auditable after the answer has+ // been given, which a fold past the cutoff would destroy.+ expect(KEPT_CONTINUITY_KINDS).toContain("recall_observed_use");+ expect(KEPT_CONTINUITY_KINDS).toContain("negative_recall");+ expect(FOLDABLE_CONTINUITY_KINDS).not.toContain("recall_observed_use");+ expect(FOLDABLE_CONTINUITY_KINDS).not.toContain("negative_recall");+ });+test("inventories metrics shards with run_at-based ages and the same protected rule", () => {vault = mintVault();try {@@ -179,7 +299,7 @@ describe("scanReceiptLedgers", () => {metricLine("storage_usage", "2026-08-15T00:00:00Z"),metricLine("storage_usage", "2026-09-20T00:00:00Z"),]);- writeShard(metrics, "recall_probe.telescope.jsonl", [+ writeShard(metrics, "recall_probe.jsonl", [metricLine("recall_probe", "2026-08-02T00:00:00Z", { session_id: "sess-1" }),]);const report = scanReceiptLedgers(vault, { now: NOW });@@ -188,7 +308,7 @@ describe("scanReceiptLedgers", () => {expect(usage.records).toBe(3);expect(usage.foldable).toBe(2);expect(usage.protectedRecords).toBe(0);- const probe = shardOf(report, RECEIPT_LEDGER_FAMILY.metrics, "recall_probe.telescope.jsonl");+ const probe = shardOf(report, RECEIPT_LEDGER_FAMILY.metrics, "recall_probe.jsonl");expect(probe.foldable).toBe(0);expect(probe.protectedRecords).toBe(1);} finally {@@ -323,6 +443,43 @@ describe("scanReceiptLedgers", () => {}});+ test("the scan offers only this device's shard and the legacy un-sharded file", () => {+ // M11: ledger shards are per device - only one device ever writes a+ // given file, and the lock and compare-and-swap a fold runs under are+ // local only. Offering another device's shard invites rewriting a+ // file a peer is appending to, which is exactly the sync conflict the+ // shard layout exists to prevent. The inventory therefore covers this+ // device's shard plus the legacy un-sharded file, and no others.+ withDeviceId("telescope", () => {+ vault = mintVault();+ try {+ writeShard(continuityLogDir(vault), "2026-08.jsonl", [+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),+ ]);+ writeShard(continuityLogDir(vault), "2026-07.telescope.jsonl", [+ continuityLine("gate_telemetry", "2026-07-10T12:00:00Z"),+ ]);+ writeShard(continuityLogDir(vault), "2026-06.other.jsonl", [+ continuityLine("gate_telemetry", "2026-06-10T12:00:00Z"),+ ]);+ writeShard(metricsDir(vault), "recall_probe.other.jsonl", [+ metricLine("recall_probe", "2026-06-11T12:00:00Z"),+ ]);+ const report = scanReceiptLedgers(vault, { now: NOW });+ // This device's shard and the legacy file, in family then name+ // order; the other device's continuity and metrics shards are+ // nobody's to offer here.+ expect(report.shards.map((shard) => `${shard.family}:${shard.name}`)).toEqual([+ "continuity:2026-07.telescope.jsonl",+ "continuity:2026-08.jsonl",+ ]);+ expect(report.totals.shards).toBe(2);+ } finally {+ cleanup();+ }+ });+ });+test("a vault with no ledgers scans to an empty report without throwing", () => {vault = mintVault();try {diff --git a/tests/core/brain/session-mining-ledger.test.ts b/tests/core/brain/session-mining-ledger.test.tsnew file mode 100644index 00000000..ac4621e4--- /dev/null+++ b/tests/core/brain/session-mining-ledger.test.ts@@ -0,0 +1,364 @@+/**+ * The shared per-session mining record (session-mining-economy,+ * t_93617709). Claims pinned here:+ *+ * 1. The ledger lives in the derived store beside the session import+ * ledger, keyed on the portable session id, and an entry carries the+ * mined-content hash, the outcome breakdown and the record time.+ * 2. A corrupt or unknown-schema ledger refuses with a named error and is+ * never reset by a read.+ * 3. The write takes the file lock, writes atomically, and is a+ * byte-identical no-op when nothing changed.+ * 4. A write prunes entries whose sessions no longer hold imported turns.+ * 5. The mined-turn hash is a stable SHA-256 over (turn id, text,+ * timestamp) and changes when any turn changes.+ * 6. The tombstone read answers the recorded entry only while the hash+ * still matches the session's current turn material.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ minedTurnsHash,+ readMiningTombstone,+ readSessionMiningLedger,+ recordSessionMiningOutcome,+ SESSION_MINING_LEDGER_SCHEMA_VERSION,+ sessionMiningLedgerPath,+ type SessionMiningEntry,+} from "../../../src/core/brain/session-mining-ledger.ts";+import { DERIVED_STORE_DIR } from "../../../src/core/brain/path-constants.ts";+import {+ importSessionRecall,+ listSessionRawTurns,+} from "../../../src/core/brain/session-recall.ts";+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";+import { acquireLockSync, LOCK_WAIT_BUDGET_ENV } from "../../../src/core/brain/sync-lockfile.ts";++const NOW = new Date("2026-10-15T08:09:08Z");+const SESSION = "sess-alpha";++let vault: string;++function turnMaterial(+ overrides: Partial<{ turnId: string; text: string; timestamp: string }> = {},+) {+ return {+ turnId: "t1",+ text: "Always name the release theme.",+ timestamp: "2026-10-15T07:00:00Z",+ ...overrides,+ };+}++const OUTCOME = { written: 0, staged: 0, deduped: 3, durabilityRejected: 1 };++function seedLedger(entries: Record<string, unknown>): void {+ mkdirSync(join(vault, DERIVED_STORE_DIR), { recursive: true });+ atomicWriteFileSync(+ sessionMiningLedgerPath(vault),+ `${JSON.stringify({ schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION, entries }, null, 2)}\n`,+ );+}++function seedTurns(sessionId: string): void {+ importSessionRecall(vault, {+ sessionId,+ turns: [+ {+ turnId: "t1",+ timestamp: "2026-10-15T07:00:00Z",+ role: "user",+ text: "Always name the release theme.",+ },+ ],+ createdAt: NOW.toISOString(),+ });+}++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-mining-ledger-"));+ mkdirSync(join(vault, "Brain"), { recursive: true });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++test("the ledger path is in the derived store beside the import ledger", () => {+ expect(sessionMiningLedgerPath(vault)).toBe(+ join(vault, DERIVED_STORE_DIR, "session-mining-ledger.json"),+ );+});++test("a missing ledger reads as empty", () => {+ expect(readSessionMiningLedger(vault)).toEqual({+ schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION,+ entries: {},+ });+});++test("an entry round-trips with the full breakdown and the record time", () => {+ seedTurns(SESSION);+ const wrote = recordSessionMiningOutcome(+ vault,+ { sessionId: SESSION, minedContentHash: "a".repeat(64), outcome: OUTCOME },+ NOW,+ );+ expect(wrote).toBe(true);+ const ledger = readSessionMiningLedger(vault);+ expect(ledger.schema_version).toBe(SESSION_MINING_LEDGER_SCHEMA_VERSION);+ const entry = ledger.entries[SESSION]!;+ expect(entry.sessionId).toBe(SESSION);+ expect(entry.minedContentHash).toBe("a".repeat(64));+ expect(entry.outcome).toEqual(OUTCOME);+ expect(entry.recordedAt).toBe("2026-10-15T08:09:08Z");+});++test("a corrupt ledger refuses by name and is never reset", () => {+ seedLedger({});+ const path = sessionMiningLedgerPath(vault);+ writeFileSync(path, "not json at all");+ expect(() => readSessionMiningLedger(vault)).toThrow(/session mining ledger is corrupted JSON/);+ expect(() => readSessionMiningLedger(vault)).toThrow(/session-mining-ledger\.json/);+ // The refused read rewrote nothing: the file is exactly as corrupt as before.+ expect(readFileSync(path, "utf8")).toBe("not json at all");+});++test("an unknown schema_version refuses by name", () => {+ seedLedger({});+ writeFileSync(+ sessionMiningLedgerPath(vault),+ `${JSON.stringify({ schema_version: 99, entries: {} }, null, 2)}\n`,+ );+ expect(() => readSessionMiningLedger(vault)).toThrow(/schema_version 99 not supported/);+ expect(() => readSessionMiningLedger(vault)).toThrow(/expected 1/);+});++test("a non-object ledger refuses by name", () => {+ seedLedger({});+ writeFileSync(sessionMiningLedgerPath(vault), "[]\n");+ expect(() => readSessionMiningLedger(vault)).toThrow(/not an object/);+});++test("a wrong-shaped entries container refuses by name, never reads as empty", () => {+ // Container-shape malformation is file-level corruption like a corrupt+ // JSON body or an unknown schema: reading it as an empty ledger would+ // silently unsuppress every barren session at once. Per-entry damage+ // stays different (dropped, never guessed) and is pinned above.+ const cases: ReadonlyArray<unknown> = [+ { schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION }, // entries absent+ { schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION, entries: null },+ { schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION, entries: [] },+ { schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION, entries: "nope" },+ { schema_version: SESSION_MINING_LEDGER_SCHEMA_VERSION, entries: 7 },+ ];+ for (const body of cases) {+ seedLedger({});+ writeFileSync(sessionMiningLedgerPath(vault), `${JSON.stringify(body)}\n`);+ expect(() => readSessionMiningLedger(vault)).toThrow(/entries is not an object/);+ expect(() => readSessionMiningLedger(vault)).toThrow(/session-mining-ledger\.json/);+ }+});++test("an entry this build cannot read is dropped, never guessed at", () => {+ seedTurns(SESSION);+ const good: SessionMiningEntry = {+ sessionId: SESSION,+ minedContentHash: "b".repeat(64),+ outcome: OUTCOME,+ recordedAt: "2026-10-15T08:09:08Z",+ };+ seedLedger({+ [SESSION]: good,+ "sess-broken-outcome": {+ sessionId: "sess-broken-outcome",+ minedContentHash: "c".repeat(64),+ outcome: { written: 0, deduped: 0 },+ recordedAt: "2026-10-15T08:09:08Z",+ },+ "sess-misfiled": { ...good, sessionId: "sess-elsewhere" },+ });+ const ledger = readSessionMiningLedger(vault);+ expect(Object.keys(ledger.entries)).toEqual([SESSION]);+});++test("recording is a byte-identical no-op when nothing changed", () => {+ seedTurns(SESSION);+ const record = { sessionId: SESSION, minedContentHash: "a".repeat(64), outcome: OUTCOME };+ recordSessionMiningOutcome(vault, record, NOW);+ const before = readFileSync(sessionMiningLedgerPath(vault), "utf8");+ const second = recordSessionMiningOutcome(vault, record, NOW);+ expect(second).toBe(false);+ expect(readFileSync(sessionMiningLedgerPath(vault), "utf8")).toBe(before);+});++test("recording rewrites when the breakdown or the content hash changes", () => {+ seedTurns(SESSION);+ const record = { sessionId: SESSION, minedContentHash: "a".repeat(64), outcome: OUTCOME };+ recordSessionMiningOutcome(vault, record, NOW);+ const rewrote = recordSessionMiningOutcome(+ vault,+ { ...record, outcome: { ...OUTCOME, deduped: 4 } },+ NOW,+ );+ expect(rewrote).toBe(true);+ expect(readSessionMiningLedger(vault).entries[SESSION]!.outcome.deduped).toBe(4);+});++test("a write prunes entries whose sessions no longer have turns", () => {+ seedTurns(SESSION);+ seedLedger({+ "sess-gone": {+ sessionId: "sess-gone",+ minedContentHash: "d".repeat(64),+ outcome: OUTCOME,+ recordedAt: "2026-10-15T08:09:08Z",+ },+ });+ recordSessionMiningOutcome(+ vault,+ { sessionId: SESSION, minedContentHash: "a".repeat(64), outcome: OUTCOME },+ NOW,+ );+ const entries = readSessionMiningLedger(vault).entries;+ expect(Object.keys(entries)).toEqual([SESSION]);+});++test("a recorded session with no turns leaves nothing behind", () => {+ // No turns are imported for this session: the entry is pruned in the+ // same pass that would write it, because a session the plan refuses has+ // nothing for the gate to suppress.+ const wrote = recordSessionMiningOutcome(+ vault,+ { sessionId: "sess-never-imported", minedContentHash: "a".repeat(64), outcome: OUTCOME },+ NOW,+ );+ expect(readSessionMiningLedger(vault).entries).toEqual({});+ expect(existsSync(sessionMiningLedgerPath(vault))).toBe(wrote);+});++test("a tombstone whose turns are visible only through lineage scope survives the prune", () => {+ // A root ref mined over segment-recorded turns: the turn records carry+ // the segment id, and the plan reaches them through the lineage scope.+ // The prune must consult that same scope, so a cheap session-id set+ // alone is not the whole answer - the entry is kept here.+ importSessionRecall(vault, {+ sessionId: "seg-b",+ turns: [+ {+ turnId: "b1",+ timestamp: "2026-10-15T07:00:00Z",+ role: "user",+ text: "Always name the release theme.",+ },+ ],+ createdAt: NOW.toISOString(),+ lineage: { rootId: "root-r", parentId: "seg-a", depth: 1, source: "payload" },+ });+ const scoped = listSessionRawTurns(vault, "root-r");+ expect(scoped.length).toBe(1);+ recordSessionMiningOutcome(+ vault,+ {+ sessionId: "root-r",+ minedContentHash: minedTurnsHash(+ scoped.map((t) => ({ turnId: t.turn_id, text: t.text.trim(), timestamp: t.timestamp })),+ ),+ outcome: OUTCOME,+ },+ NOW,+ );+ expect(readSessionMiningLedger(vault).entries["root-r"]).toBeDefined();+});++test("a contended lock refuses by name instead of writing unlocked", () => {+ seedTurns(SESSION);+ mkdirSync(join(vault, DERIVED_STORE_DIR), { recursive: true });+ const previous = process.env[LOCK_WAIT_BUDGET_ENV];+ process.env[LOCK_WAIT_BUDGET_ENV] = "50";+ const lock = acquireLockSync(sessionMiningLedgerPath(vault));+ try {+ expect(() =>+ recordSessionMiningOutcome(+ vault,+ { sessionId: SESSION, minedContentHash: "a".repeat(64), outcome: OUTCOME },+ NOW,+ ),+ ).toThrow(/lock busy/);+ } finally {+ lock.release();+ if (previous === undefined) delete process.env[LOCK_WAIT_BUDGET_ENV];+ else process.env[LOCK_WAIT_BUDGET_ENV] = previous;+ }+ // Once the lock is free the same record lands.+ expect(+ recordSessionMiningOutcome(+ vault,+ { sessionId: SESSION, minedContentHash: "a".repeat(64), outcome: OUTCOME },+ NOW,+ ),+ ).toBe(true);+});++test("the mined-turn hash is stable and order-sensitive", () => {+ const a = [turnMaterial(), turnMaterial({ turnId: "t2", text: "Never abbreviate modules." })];+ expect(minedTurnsHash(a)).toBe(+ minedTurnsHash([+ turnMaterial(),+ turnMaterial({ turnId: "t2", text: "Never abbreviate modules." }),+ ]),+ );+ expect(minedTurnsHash(a)).not.toBe(minedTurnsHash([a[1]!, a[0]!]));+});++test("the mined-turn hash changes when any turn changes", () => {+ const base = [turnMaterial()];+ expect(minedTurnsHash(base)).not.toBe(minedTurnsHash([turnMaterial({ turnId: "t9" })]));+ expect(minedTurnsHash(base)).not.toBe(minedTurnsHash([turnMaterial({ text: "Changed." })]));+ expect(minedTurnsHash(base)).not.toBe(+ minedTurnsHash([turnMaterial({ timestamp: "2026-10-15T07:01:00Z" })]),+ );+});++test("the tombstone read answers only while the hash still matches", () => {+ seedTurns(SESSION);+ const turns = [turnMaterial()];+ recordSessionMiningOutcome(+ vault,+ { sessionId: SESSION, minedContentHash: minedTurnsHash(turns), outcome: OUTCOME },+ NOW,+ );+ const hit = readMiningTombstone(vault, SESSION, turns);+ expect(hit).not.toBeNull();+ expect(hit!.outcome).toEqual(OUTCOME);+ // Unchanged content on a later scan stays suppressed.+ expect(readMiningTombstone(vault, SESSION, [turnMaterial()])).not.toBeNull();+ // A re-import that changed the turns re-opens the session.+ expect(readMiningTombstone(vault, SESSION, [turnMaterial({ text: "Edited turn." })])).toBeNull();+ expect(readMiningTombstone(vault, "sess-other", turns)).toBeNull();+});++test("a record with a yield is not a tombstone, but stays in the ledger", () => {+ // The distilled state: the same record shape, read through two+ // contracts. The gate must not suppress a session whose recorded mine+ // produced accepted yield; the census reads the yield from the same+ // entry.+ seedTurns(SESSION);+ const turns = [turnMaterial()];+ recordSessionMiningOutcome(+ vault,+ {+ sessionId: SESSION,+ minedContentHash: minedTurnsHash(turns),+ outcome: { written: 2, staged: 1, deduped: 0, durabilityRejected: 0 },+ },+ NOW,+ );+ expect(readMiningTombstone(vault, SESSION, turns)).toBeNull();+ const entry = readSessionMiningLedger(vault).entries[SESSION]!;+ expect(entry.outcome).toEqual({ written: 2, staged: 1, deduped: 0, durabilityRejected: 0 });+});diff --git a/tests/core/brain/sessions/mining-census.test.ts b/tests/core/brain/sessions/mining-census.test.tsnew file mode 100644index 00000000..b528d049--- /dev/null+++ b/tests/core/brain/sessions/mining-census.test.ts@@ -0,0 +1,359 @@+/**+ * The distilled and undistilled census (session-mining-economy,+ * t_0358af42).+ *+ * The import ledger knows what was imported; the mining ledger knows what+ * was mined and with what yield. The census joins the two on the portable+ * `sessionRefIdentity` basename - never on the import ledger's absolute+ * machine-local keys - and classifies every imported session:+ *+ * - `distilled`: mined with accepted yield on the CURRENT turn material;+ * - `known-nothing-new`: mined barren on the current material (a+ * tombstone), which is a different state from backlog;+ * - `undistilled`: never mined, or mined on material that has since+ * changed;+ * - refused: sessions the plan would refuse (boundary, zero imported+ * turns, zero user turns) can never be distilled and are excluded from+ * the backlog, because counting them would nag forever about+ * unparsable material.+ *+ * Nothing here re-hashes a source log: the join is vault-side, over the+ * two ledgers plus the continuity turns the mining lane itself reads.+ * Nothing here writes either.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../../../src/core/brain/init.ts";+import { brainConfigPath } from "../../../../src/core/brain/paths.ts";+import {+ commitExtractedSignals,+ planExtractSignals,+} from "../../../../src/core/brain/extract-signals.ts";+import {+ minedTurnsHash,+ sessionMiningLedgerPath,+} from "../../../../src/core/brain/session-mining-ledger.ts";+import { buildCaptureBoundary } from "../../../../src/core/brain/capture-boundary.ts";+import { listSessionRawTurns } from "../../../../src/core/brain/session-recall.ts";+import {+ recordSessionImports,+ sessionLedgerPath,+} from "../../../../src/core/brain/sessions/discover.ts";+import { buildMiningCensus } from "../../../../src/core/brain/sessions/mining-census.ts";+import { importSession } from "../../../../src/core/brain/sessions/import.ts";+import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";+import { SESSION_RUNTIME_ID } from "../../../../src/core/runtime/host-facts.ts";+import { DERIVED_STORE_DIR } from "../../../../src/core/brain/path-constants.ts";++let tmp: string;+let vault: string;+let sessionsDir: string;++const NOW = new Date("2026-09-18T10:00:00Z");++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-mining-census-"));+ vault = join(tmp, "vault");+ sessionsDir = mkdtempSync(join(tmpdir(), "o2b-mining-census-sessions-"));+ bootstrapBrain(vault);+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+ rmSync(sessionsDir, { recursive: true, force: true });+});++/** One Claude-format session file with the given turns. */+function writeSession(+ name: string,+ turns: ReadonlyArray<{ readonly role: "user" | "assistant"; readonly text: string }>,+): string {+ const lines = turns.map((turn, i) =>+ JSON.stringify({+ type: turn.role,+ parentUuid: null,+ entrypoint: "cli",+ sessionId: "s1",+ uuid: `u-${i}`,+ timestamp: "2026-09-18T09:00:00.000Z",+ message: { role: turn.role, content: [{ type: "text", text: turn.text }] },+ }),+ );+ const path = join(sessionsDir, name);+ writeFileSync(path, lines.join("\n") + "\n", "utf8");+ return path;+}++/** Import one log (with recall, so turns reach continuity) and record it. */+async function importOne(path: string): Promise<string> {+ await importSession(vault, path, { agent: "tester", now: NOW, recall: true });+ recordSessionImports(vault, [{ path, runtime: SESSION_RUNTIME_ID.claudeCode }]);+ return path;+}++/**+ * The mined turn material the plan would embed, derived exactly as+ * `planExtractSignals` derives it (user role, unsuppressed, trimmed,+ * non-empty) - the census must hash the same material the plan mines.+ */+function minedMaterial(sessionId: string): ReadonlyArray<{+ readonly turnId: string;+ readonly text: string;+ readonly timestamp: string;+}> {+ const boundary = buildCaptureBoundary(vault);+ return listSessionRawTurns(vault, sessionId)+ .filter((turn) => turn.role === "user")+ .filter((turn) => !boundary.suppressMessage(turn.text))+ .map((turn) => ({ turnId: turn.turn_id, text: turn.text.trim(), timestamp: turn.timestamp }))+ .filter((turn) => turn.text.length > 0);+}++/**+ * Mine one session through the REAL writer - a plan, then the commit that+ * answers it, attesting the plan's surveyed hash - with nothing accepted.+ * The zero-yield record the census reads as known-nothing-new is what that+ * commit actually writes.+ */+function mineBarren(sessionId: string): void {+ const plan = planExtractSignals(vault, sessionId, { now: NOW });+ commitExtractedSignals(+ vault,+ sessionId,+ { items: [] },+ { agent: "tester", now: NOW, surveyedTurnsHash: plan.surveyedTurnsHash },+ );+}++/**+ * Mine one session through the REAL writer to a written yield. The yield+ * record the census reads as distilled is what that commit actually+ * writes - no seeded hand-crafted ledger state.+ */+function mineYield(sessionId: string): void {+ const plan = planExtractSignals(vault, sessionId, { now: NOW });+ commitExtractedSignals(+ vault,+ sessionId,+ {+ items: [+ {+ topic: `rule-of-${sessionId.replace(/\.jsonl$/, "")}`,+ signal: "positive",+ principle: "Ship the parser before the next release.",+ confidence: 0.9,+ },+ ],+ },+ { agent: "tester", now: NOW, surveyedTurnsHash: plan.surveyedTurnsHash },+ );+}++describe("buildMiningCensus", () => {+ test("a vault that imported nothing reports zero everywhere", () => {+ const census = buildMiningCensus(vault);+ expect(census.imported).toBe(0);+ expect(census.distilled).toBe(0);+ expect(census.knownNothingNew).toBe(0);+ expect(census.undistilled).toBe(0);+ expect(census.refused).toBe(0);+ expect(census.byRuntime).toEqual([]);+ });++ test("an imported, never-mined session is undistilled", async () => {+ await importOne(writeSession("alpha.jsonl", [{ role: "user", text: "ship the parser" }]));+ const census = buildMiningCensus(vault);+ expect(census.imported).toBe(1);+ expect(census.undistilled).toBe(1);+ expect(census.distilled).toBe(0);+ expect(census.knownNothingNew).toBe(0);+ expect(census.refused).toBe(0);+ expect(census.byRuntime[0]?.runtime).toBe(SESSION_RUNTIME_ID.claudeCode);+ expect(census.byRuntime[0]?.undistilledSessionIds).toEqual(["alpha.jsonl"]);+ });++ test("a mined session with accepted yield on current content is distilled", async () => {+ await importOne(writeSession("alpha.jsonl", [{ role: "user", text: "ship the parser" }]));+ mineYield("alpha.jsonl");+ const census = buildMiningCensus(vault);+ expect(census.distilled).toBe(1);+ expect(census.undistilled).toBe(0);+ expect(census.knownNothingNew).toBe(0);+ });++ test("a mined session that yielded nothing accepted is known-nothing-new, not undistilled", async () => {+ await importOne(writeSession("alpha.jsonl", [{ role: "user", text: "ship the parser" }]));+ mineBarren("alpha.jsonl");+ const census = buildMiningCensus(vault);+ expect(census.knownNothingNew).toBe(1);+ expect(census.undistilled).toBe(0);+ expect(census.distilled).toBe(0);+ });++ test("a re-import that changes the turns re-opens the session", async () => {+ const path = writeSession("alpha.jsonl", [{ role: "user", text: "ship the parser" }]);+ await importOne(path);+ mineYield("alpha.jsonl");+ expect(buildMiningCensus(vault).distilled).toBe(1);++ // A second turn lands in the same log (an edit, or a resume), and the+ // session is re-imported: the mined material grows, the recorded hash+ // no longer matches, and the session is backlog again.+ writeFileSync(+ path,+ readFileSync(path, "utf8") ++ JSON.stringify({+ type: "user",+ parentUuid: null,+ entrypoint: "cli",+ sessionId: "s1",+ uuid: "u-9",+ timestamp: "2026-09-18T09:30:00.000Z",+ message: { role: "user", content: [{ type: "text", text: "also add tests" }] },+ }) ++ "\n",+ "utf8",+ );+ await importOne(path);+ expect(minedMaterial("alpha.jsonl").length).toBe(2);++ const census = buildMiningCensus(vault);+ expect(census.undistilled).toBe(1);+ expect(census.distilled).toBe(0);+ });++ test("a session the capture boundary refuses is excluded from the backlog", async () => {+ const config = brainConfigPath(vault);+ atomicWriteFileSync(+ config,+ readFileSync(config, "utf8") + '\nsessions:\n ignore_patterns:\n - "*blocked-*"\n',+ );+ const path = writeSession("blocked-one.jsonl", [{ role: "user", text: "ship the parser" }]);+ recordSessionImports(vault, [{ path, runtime: SESSION_RUNTIME_ID.claudeCode }]);+ const census = buildMiningCensus(vault);+ expect(census.imported).toBe(1);+ expect(census.refused).toBe(1);+ expect(census.undistilled).toBe(0);+ });++ test("a ledger entry whose session has no imported turns is excluded", async () => {+ const path = writeSession("never-imported.jsonl", [{ role: "user", text: "ship the parser" }]);+ recordSessionImports(vault, [{ path, runtime: SESSION_RUNTIME_ID.claudeCode }]);+ const census = buildMiningCensus(vault);+ expect(census.imported).toBe(1);+ expect(census.refused).toBe(1);+ expect(census.undistilled).toBe(0);+ });++ test("a session with imported turns but no user turns is excluded", async () => {+ await importOne(+ writeSession("assistant-only.jsonl", [{ role: "assistant", text: "build done" }]),+ );+ const census = buildMiningCensus(vault);+ expect(census.refused).toBe(1);+ expect(census.undistilled).toBe(0);+ });++ test("a message the boundary suppresses is not part of the mined material", async () => {+ const config = brainConfigPath(vault);+ atomicWriteFileSync(+ config,+ readFileSync(config, "utf8") ++ '\nsessions:\n ignore_message_patterns:\n - "noise-turn-present"\n',+ );+ await importOne(+ writeSession("alpha.jsonl", [+ { role: "user", text: "ship the parser" },+ { role: "user", text: "noise-turn-present" },+ ]),+ );+ // The real writer plans over the reduced material (the boundary+ // suppresses the noise turn before the plan embeds anything), attests+ // that hash, and the commit records it - so the census must hash the+ // same reduced material the plan mines for the session to be+ // distilled.+ mineYield("alpha.jsonl");+ const reduced = minedTurnsHash(+ minedMaterial("alpha.jsonl").filter((turn) => !turn.text.includes("noise-turn-present")),+ );+ expect(minedMaterial("alpha.jsonl").length).toBe(1);+ const ledger = JSON.parse(+ readFileSync(join(vault, DERIVED_STORE_DIR, "session-mining-ledger.json"), "utf8"),+ ) as { entries: Record<string, { minedContentHash: string }> };+ expect(ledger.entries["alpha.jsonl"]?.minedContentHash).toBe(reduced);+ expect(buildMiningCensus(vault).distilled).toBe(1);+ });++ test("the join keys on the portable basename, not the absolute path", async () => {+ const first = await importOne(+ writeSession("alpha.jsonl", [{ role: "user", text: "ship the parser" }]),+ );+ mineYield("alpha.jsonl");+ expect(buildMiningCensus(vault).distilled).toBe(1);++ // The log moves (a rotated store, another machine's layout): the import+ // ledger re-records it under the new absolute path, and the vanished+ // old path is pruned by the same write. The basename - the identity the+ // mining lane holds - still joins.+ const movedDir = join(tmp, "elsewhere");+ mkdirSync(movedDir, { recursive: true });+ const moved = join(movedDir, "alpha.jsonl");+ renameSync(first, moved);+ recordSessionImports(vault, [{ path: moved, runtime: SESSION_RUNTIME_ID.claudeCode }]);++ const census = buildMiningCensus(vault);+ expect(census.imported).toBe(1);+ expect(census.distilled).toBe(1);+ });++ test("counts are aggregated per runtime, by the ledger's own label", async () => {+ await importOne(writeSession("alpha.jsonl", [{ role: "user", text: "ship the parser" }]));+ const beta = writeSession("beta.jsonl", [{ role: "user", text: "write the docs" }]);+ await importOne(beta);+ mineYield("beta.jsonl");+ // The census trusts the runtime the import ledger recorded, the same+ // way `--status` reports per-runtime coverage; re-labelling the second+ // entry proves the rows split rather than merge.+ recordSessionImports(vault, [{ path: beta, runtime: SESSION_RUNTIME_ID.codex }]);++ const census = buildMiningCensus(vault);+ expect(census.imported).toBe(2);+ expect(census.undistilled).toBe(1);+ expect(census.distilled).toBe(1);+ const claude = census.byRuntime.find((r) => r.runtime === SESSION_RUNTIME_ID.claudeCode);+ const codex = census.byRuntime.find((r) => r.runtime === SESSION_RUNTIME_ID.codex);+ expect(claude?.undistilled).toBe(1);+ expect(claude?.undistilledSessionIds).toEqual(["alpha.jsonl"]);+ expect(codex?.distilled).toBe(1);+ expect(codex?.undistilled).toBe(0);+ });++ test("a corrupt mining record refuses by name", () => {+ const dir = join(vault, DERIVED_STORE_DIR);+ mkdirSync(dir, { recursive: true });+ writeFileSync(sessionMiningLedgerPath(vault), "{ this is not json", "utf8");+ expect(() => buildMiningCensus(vault)).toThrow(/session mining ledger is corrupted JSON/);+ });++ test("an unknown mining-record schema version refuses by name", () => {+ const dir = join(vault, DERIVED_STORE_DIR);+ mkdirSync(dir, { recursive: true });+ writeFileSync(+ sessionMiningLedgerPath(vault),+ JSON.stringify({ schema_version: 99, entries: {} }),+ "utf8",+ );+ expect(() => buildMiningCensus(vault)).toThrow(/schema_version 99 not supported/);+ });++ test("a corrupt import ledger refuses by name", () => {+ const dir = join(vault, DERIVED_STORE_DIR);+ mkdirSync(dir, { recursive: true });+ writeFileSync(sessionLedgerPath(vault), "{ this is not json", "utf8");+ expect(() => buildMiningCensus(vault)).toThrow(/session import ledger is corrupted JSON/);+ });+});diff --git a/tests/core/brain/source-cleanup.test.ts b/tests/core/brain/source-cleanup.test.tsindex 809ca623..7bd54550 100644--- a/tests/core/brain/source-cleanup.test.ts+++ b/tests/core/brain/source-cleanup.test.ts@@ -366,6 +366,42 @@ describe("deleteBySource — per-entry failure collection", () => {chmodSync(blockedDir, 0o755);}});++ test("an original whose removal failed keeps the manifest entry", () => {+ if (CHMOD_CANNOT_DENY) return;+ seedContaminatedVault();+ // The original sits OUTSIDE Brain/; a read-only imports directory makes+ // exactly that removal fail while every Brain-side entry still deletes.+ const originalsDir = join(vault, dirname(SOURCE));+ chmodSync(originalsDir, 0o555);+ try {+ const plan = deleteBySource(vault, SOURCE, {+ confirm: true,+ includeOriginals: true,+ now: NOW,+ });++ expect(plan.failed.some((f) => f.path === SOURCE)).toBe(true);+ expect(existsSync(join(vault, SOURCE))).toBe(true);+ // The manifest entry STAYS (review L7): the source file is still on+ // disk, so dropping the entry would classify it `new` and the next+ // batch plan would rebuild the derived pages this run just removed.+ expect(plan.manifestEntryRemoved).toBe(false);+ expect(readManifest(vault).entries[SOURCE]).toBeDefined();+ // The derived pages are still gone - the entry keeps them from being+ // rebuilt, it does not bring them back.+ const preview = deleteBySource(vault, SOURCE, { now: NOW });+ expect(preview.derived.every((e) => !existsSync(join(vault, e.path)))).toBe(true);+ // The audit says why the entry stayed.+ const audit = listContinuityRecords(vault, { kind: "source_invalidation" });+ expect(audit.length).toBe(1);+ const reason = String(audit[0]!.payload["reason"]);+ expect(reason).toContain("manifest entry kept");+ expect(reason).toContain(SOURCE);+ } finally {+ chmodSync(originalsDir, 0o755);+ }+ });});describe("deleteBySource — shared preference via managed _evidenced_by (CR #127.7)", () => {diff --git a/tests/core/brain/storage-forecast.test.ts b/tests/core/brain/storage-forecast.test.tsindex 611894ca..2d5304ac 100644--- a/tests/core/brain/storage-forecast.test.ts+++ b/tests/core/brain/storage-forecast.test.ts@@ -28,14 +28,17 @@ import { join } from "node:path";import { appendMetric, listMetrics, metricsDir } from "../../../src/core/brain/metrics.ts";import {+ MIN_TREND_DAYS,STORAGE_CRITICAL_DAYS_DEFAULT,STORAGE_CRITICAL_PERCENT_DEFAULT,STORAGE_USAGE_METRIC_SURFACE,STORAGE_WARN_DAYS_DEFAULT,STORAGE_WARN_PERCENT_DEFAULT,+ TREND_WINDOW_DAYS,forecastStorageUsage,sampleStorageUsage,} from "../../../src/core/brain/storage/forecast.ts";+import { evaluateCapacityGate } from "../../../src/core/brain/storage/gate.ts";import { measureStorageUsage } from "../../../src/core/brain/storage/usage.ts";import { CHMOD_CANNOT_DENY } from "../../helpers/platform.ts";@@ -62,6 +65,15 @@ function seedSample(daysAgo: number, totalBytes: number): void {});}+/** Seed one `storage_usage` sample at an exact instant, sub-day precision. */+function seedSampleAt(t: Date, totalBytes: number, runId: string, measured = true): void {+ appendMetric(vault, {+ surface: STORAGE_USAGE_METRIC_SURFACE,+ runAt: t.toISOString(),+ payload: { run_id: runId, total_bytes: totalBytes, measured },+ });+}+describe("forecastStorageUsage", () => {test("returns the linear slope, days to cap and tier over seeded samples", () => {seedSample(2, 100);@@ -84,27 +96,29 @@ describe("forecastStorageUsage", () => {expect(STORAGE_WARN_DAYS_DEFAULT).toBe(30);expect(STORAGE_CRITICAL_DAYS_DEFAULT).toBe(7);- // A gentle slope keeps the days boundary out of the decision: 20- // days to the cap at 10 bytes per day.+ // Two samples a day apart are under the three-day minimum: no slope,+ // so the percent comparison alone decides - exactly 80 warns.seedSample(1, 790);seedSample(0, 800);expect(forecastStorageUsage(vault, { capBytes: CAP }).tier).toBe("warn");seedSample(-1, 950);- // 95 percent used, and 5 days to the cap - critical on both counts.+ // 95 percent used; the three-day fit (790, 800, 950) gives 40 bytes+ // per day, so 1.25 days to the cap - critical on both counts.expect(forecastStorageUsage(vault, { capBytes: CAP }).tier).toBe("critical");});test("the percent boundaries are inclusive with no trend to decide instead", () => {- // The tier fixtures above sit inside the days bands too, so a+ // The tier fixtures below sit inside the days bands too, so a// comparison flipped exclusive would still answer warn/critical// through the days path and the percent boundary would go unpinned.- // A FLAT history has slope 0 and no daysToCap, so the percent- // comparison alone decides: exactly 80 warns, exactly 95 is critical.+ // Two distinct days fit no slope at all (the three-day minimum), so+ // the percent comparison alone decides: exactly 80 warns, exactly 95+ // is critical, and neither carries a daysToCap figure.seedSample(1, 800);seedSample(0, 800);const warn = forecastStorageUsage(vault, { capBytes: CAP });- expect(warn.slopeBytesPerDay).toBe(0);+ expect(warn.slopeBytesPerDay).toBe(null);expect(warn.daysToCap).toBe(null);expect(warn.percentUsed).toBe(80);expect(warn.tier).toBe("warn");@@ -113,6 +127,7 @@ describe("forecastStorageUsage", () => {seedSample(1, 950);seedSample(0, 950);const critical = forecastStorageUsage(vault, { capBytes: CAP });+ expect(critical.slopeBytesPerDay).toBe(null);expect(critical.daysToCap).toBe(null);expect(critical.percentUsed).toBe(95);expect(critical.tier).toBe("critical");@@ -134,9 +149,10 @@ describe("forecastStorageUsage", () => {test("a percent-only warn never outranks a critical days horizon", () => {// 280 of 350 bytes is exactly 80 percent (warn by percent), and a- // slope of 10 bytes per day puts the cap exactly 7 days out- // (critical by days). The critical comparison runs first, so the+ // three-day slope of 10 bytes per day puts the cap exactly 7 days+ // out (critical by days). The critical comparison runs first, so the// days horizon decides the tier.+ seedSample(2, 260);seedSample(1, 270);seedSample(0, 280);const forecast = forecastStorageUsage(vault, { capBytes: 350 });@@ -146,7 +162,9 @@ describe("forecastStorageUsage", () => {});test("exactly the warn days to cap warns", () => {- // current 300, slope 10 per day: 30 days to 600, at the boundary.+ // current 300, three-day slope 10 per day: 30 days to 600, at the+ // boundary.+ seedSample(2, 280);seedSample(1, 290);seedSample(0, 300);const warn = forecastStorageUsage(vault, { capBytes: 600 });@@ -156,7 +174,9 @@ describe("forecastStorageUsage", () => {});test("exactly the critical days to cap is critical", () => {- // current 300, slope 50 per day: 7 days to 650, at the boundary.+ // current 300, three-day slope 50 per day: 7 days to 650, at the+ // boundary.+ seedSample(2, 200);seedSample(1, 250);seedSample(0, 300);const critical = forecastStorageUsage(vault, { capBytes: 650 });@@ -205,6 +225,7 @@ describe("forecastStorageUsage", () => {});test("a shrinking vault never forecasts past the cap", () => {+ seedSample(2, 600);seedSample(1, 500);seedSample(0, 400);const forecast = forecastStorageUsage(vault, { capBytes: CAP });@@ -213,6 +234,104 @@ describe("forecastStorageUsage", () => {expect(forecast.tier).toBe("ok");});+ test("two samples a minute apart never produce a day-based tier", () => {+ // The H2 burst probe: a vault a third full, one ordinary 20 KB write+ // between two calls of the storage report a minute apart. The burst+ // extrapolates to tens of megabytes per day when fitted raw, which+ // read `critical` and refused ingest on a healthy vault. The trend+ // is day-based: both samples land on one UTC day, so no slope+ // exists, daysToCap stays null and the tier answers from the percent+ // alone - the fresh census the gate hands in measures 31 percent.+ expect(MIN_TREND_DAYS).toBe(3);+ const t0 = new Date(Date.parse("2026-10-15T12:00:00Z"));+ const base = 300 * 1024;+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", "AA-base.md"), "x".repeat(base), "utf8");+ seedSampleAt(t0, base, "burst-1");+ writeFileSync(join(vault, "Brain", "sources", "AA-grow.md"), "x".repeat(20 * 1024), "utf8");+ seedSampleAt(new Date(t0.getTime() + 60_000), base + 20 * 1024, "burst-2");+ // The gate's path: fresh census for the percent, day-based history+ // for the trend.+ const decision = evaluateCapacityGate(vault, {+ capBytes: 1024 * 1024,+ now: new Date(t0.getTime() + 90_000),+ });+ expect(decision.tier).toBe("ok");+ expect(decision.decision).toBe("run");+ expect(decision.reading!.percentUsed).toBeLessThan(35);+ expect(decision.reading!.slopeBytesPerDay).toBe(null);+ expect(decision.reading!.daysToCap).toBe(null);+ // The samples-only path holds to the two-truths rule: one bucketed+ // day is one observation, and samples alone never tier.+ const forecast = forecastStorageUsage(vault, { capBytes: 1024 * 1024 });+ expect(forecast.tier).toBe("no_forecast");+ });++ test("an intraday burst folds into the day's last sample once days exist", () => {+ // Three distinct days fit a slope, and the day bucket keeps only the+ // day's LAST sample: a burst of intra-day readings cannot steer the+ // fit, whatever their spacing.+ seedSampleAt(new Date("2026-10-13T09:00:00Z"), 100, "d1");+ seedSampleAt(new Date("2026-10-14T09:00:00Z"), 200, "d2");+ // A burst on the last day, hours BEFORE the settled reading: three+ // readings minutes apart that the bucket must fold away.+ seedSampleAt(new Date("2026-10-15T08:00:00Z"), 9_900, "d3a");+ seedSampleAt(new Date("2026-10-15T08:01:00Z"), 9_950, "d3b");+ seedSampleAt(new Date("2026-10-15T09:00:00Z"), 300, "d3-final");+ const forecast = forecastStorageUsage(vault, { capBytes: 10_000 });+ expect(forecast.slopeBytesPerDay).toBe(100);+ expect(forecast.measuredBytes).toBe(300);+ });++ test("the fit runs over the trailing window, not the whole history", () => {+ // A reclamation 40 days ago (the vault was emptied) must not drag the+ // slope: the fit sees only the last TREND_WINDOW_DAYS days, where the+ // vault grows at a steady 100 bytes per day.+ expect(TREND_WINDOW_DAYS).toBe(30);+ seedSample(40, 0);+ seedSample(2, 300);+ seedSample(1, 400);+ seedSample(0, 500);+ const forecast = forecastStorageUsage(vault, { capBytes: 100_000 });+ expect(forecast.slopeBytesPerDay).toBe(100);+ expect(forecast.tier).toBe("ok");+ });++ test("a partial census never enters the trend", () => {+ // A sample recorded with measured: false carries a total that+ // understates the vault by exactly the refused surfaces. Fed into the+ // fit, the jump when the surface becomes readable again reads as+ // growth; parseSamples skips it, so the three complete days fit the+ // line and the partial day is evidence for no number at all.+ seedSample(3, 100);+ seedSample(2, 200);+ seedSample(1, 300);+ appendMetric(vault, {+ surface: STORAGE_USAGE_METRIC_SURFACE,+ runAt: "2026-10-15T12:00:00Z",+ payload: { run_id: "partial", total_bytes: 999_999, measured: false },+ });+ const forecast = forecastStorageUsage(vault, { capBytes: 100_000 });+ expect(forecast.slopeBytesPerDay).toBe(100);+ expect(forecast.measuredBytes).toBe(300);+ });++ test("a sample without the measured flag is not trend evidence either", () => {+ // The flag is the census's own claim of completeness, and absence is+ // not a claim: an unflagged sample never completes the three-day+ // minimum the slope needs.+ seedSample(2, 100);+ seedSample(1, 200);+ appendMetric(vault, {+ surface: STORAGE_USAGE_METRIC_SURFACE,+ runAt: "2026-10-15T12:00:00Z",+ payload: { run_id: "unflagged", total_bytes: 42 },+ });+ const forecast = forecastStorageUsage(vault, { capBytes: 100_000 });+ expect(forecast.slopeBytesPerDay).toBe(null);+ expect(forecast.daysToCap).toBe(null);+ });+test.skipIf(CHMOD_CANNOT_DENY)("an incomplete measurement is unmeasurable", () => {seedSample(1, 100);seedSample(0, 200);diff --git a/tests/core/brain/storage-gate.test.ts b/tests/core/brain/storage-gate.test.tsindex 6dd03159..226ee9b5 100644--- a/tests/core/brain/storage-gate.test.ts+++ b/tests/core/brain/storage-gate.test.ts@@ -156,4 +156,43 @@ describe("evaluateCapacityGate", () => {expect(skipped.decision).toBe("skip");expect(skipped.tier).toBe(STORAGE_FORECAST_TIER.critical);});++ test("a pausePercent passed in closes the gate one band before the refuse line", () => {+ // The bootstrapped skeleton sits in the warn band of a 30 KB cap.+ // Without a pause line the gate RUNS there (the interactive wall's+ // posture, pinned above); the lane hands its pause_percent in, and+ // the same reading then skips - background work backs off before+ // the wall, interactive ingest still works.+ configureCap(30_000);+ const decision = evaluateCapacityGate(vault, {+ capBytes: resolvedCap(),+ pausePercent: 80,+ });+ expect(decision.tier).toBe(STORAGE_FORECAST_TIER.warn);+ expect(decision.decision).toBe("skip");+ // The line rides the decision, so a report can show how far the+ // vault was from the pause.+ expect(decision.pausePercent).toBe(80);+ });++ test("a pausePercent below the tier line pauses an ok vault", () => {+ configureCap(1_000_000);+ writePages("AA-small.md", 100);+ const decision = evaluateCapacityGate(vault, {+ capBytes: resolvedCap(),+ pausePercent: 1,+ });+ expect(decision.tier).toBe(STORAGE_FORECAST_TIER.ok);+ expect(decision.decision).toBe("skip");+ });++ test("a pausePercent above the measured percent leaves the gate open", () => {+ configureCap(10_000_000);+ const decision = evaluateCapacityGate(vault, {+ capBytes: resolvedCap(),+ pausePercent: 50,+ });+ expect(decision.decision).toBe("run");+ expect(decision.pausePercent).toBe(50);+ });});diff --git a/tests/core/brain/storage-policy.test.ts b/tests/core/brain/storage-policy.test.tsindex c8b4a5d3..9d4f1412 100644--- a/tests/core/brain/storage-policy.test.ts+++ b/tests/core/brain/storage-policy.test.ts@@ -3,16 +3,21 @@** Claims pinned here:*- * 1. `resolveStorage` parses the five keys and every present key is+ * 1. `resolveStorage` parses the six keys and every present key is* bounded: out-of-range values HARD-error naming the field - a* silently clamped cap would be indistinguishable from no cap, and* for a gate that is the worst possible failure.- * 2. `critical_percent` must exceed `warn_percent`, and `critical_days`- * must be smaller than `warn_days` - the same coherence class: a- * critical band that fires before the warn band inverts the tiers.+ * 2. `critical_percent` must exceed `warn_percent`, `critical_days`+ * must be smaller than `warn_days`, and `pause_percent` must sit+ * strictly below `critical_percent` - the same coherence class: a+ * critical band that fires before the warn band inverts the tiers,+ * and a pause line at or past the refuse line could never pause+ * before the refusal fired.* 3. An absent block resolves to the documented defaults with the cap* absent (unbounded), both through the validator and through- * `loadStorageConfigSafe` on a vault with no `_brain.yaml`.+ * `loadStorageConfigSafe` on a vault with no `_brain.yaml`. An+ * absent `pause_percent` follows the warn band - the lane pauses+ * where the forecast warns unless the operator moves the line.* 4. The template ratchet: the block appears in the generated* template, `mergeBrainYaml` appends it to an existing `_brain.yaml`* without rewriting any existing value, and the merged file@@ -64,16 +69,18 @@ storage:cap_bytes: 1073741824warn_percent: 70critical_percent: 90+ pause_percent: 50warn_days: 21critical_days: 5`;describe("resolveStorage", () => {- test("parses the five keys", () => {+ test("parses the six keys", () => {expect(resolveFromYaml(FULL_BLOCK)).toEqual({cap_bytes: 1073741824,warn_percent: 70,critical_percent: 90,+ pause_percent: 50,warn_days: 21,critical_days: 5,});@@ -84,6 +91,7 @@ describe("resolveStorage", () => {cap_bytes: null,warn_percent: 80,critical_percent: 95,+ pause_percent: 80,warn_days: 30,critical_days: 7,});@@ -101,6 +109,18 @@ describe("resolveStorage", () => {});});+ test("an absent pause_percent follows the warn band the block sets", () => {+ // The default pause line IS the warn band, so an operator who moves+ // warn moves the lane's pause with it unless they name pause_percent+ // themselves.+ expect(+ resolveFromYaml("schema_version: 1\n\nstorage:\n warn_percent: 65\n").pause_percent,+ ).toBe(65);+ expect(+ resolveFromYaml("schema_version: 1\n\nstorage:\n pause_percent: 40\n").pause_percent,+ ).toBe(40);+ });+test.each([["cap_bytes: 0", "storage.cap_bytes"],["cap_bytes: -5", "storage.cap_bytes"],@@ -108,6 +128,8 @@ describe("resolveStorage", () => {['cap_bytes: "big"', "storage.cap_bytes"],["warn_percent: 0", "storage.warn_percent"],["warn_percent: 101", "storage.warn_percent"],+ ["pause_percent: 0", "storage.pause_percent"],+ ["pause_percent: 101", "storage.pause_percent"],["warn_days: 0", "storage.warn_days"],["critical_days: 0", "storage.critical_days"],])("hard-errors the out-of-range value %s", (line, field) => {@@ -189,6 +211,35 @@ describe("resolveStorage", () => {expect(errored).toBe(true);});+ test.each([+ [" critical_percent: 90", " pause_percent: 95"],+ [" critical_percent: 90", " pause_percent: 90"],+ [" pause_percent: 99"],+ ])("refuses a pause line at or past the critical band", (...lines) => {+ // A pause line past the refuse line could never pause anything: the+ // ingest wall fires first at critical, so the knob would be dead+ // while looking configured. Same rule as the warn band: strictly+ // below critical, whether critical is spelled or the default.+ const block = `schema_version: 1\n\nstorage:\n${lines.join("\n")}\n`;+ let errored = false;+ try {+ resolveFromYaml(block);+ } catch (err) {+ errored = true;+ expect(err).toBeInstanceOf(BrainConfigError);+ expect((err as BrainConfigError).field).toBe("storage.pause_percent");+ }+ expect(errored).toBe(true);+ });++ test("a pause line below the critical band resolves", () => {+ expect(+ resolveFromYaml(+ "schema_version: 1\n\nstorage:\n critical_percent: 90\n pause_percent: 89\n warn_percent: 60\n",+ ).pause_percent,+ ).toBe(89);+ });+test("an unreadable config file raises out of the safe loader", () => {writeConfig("schema_version: 1\n\nstorage:\n cap_bytes: 0\n");expect(() => loadStorageConfigSafe(vault)).toThrow(BrainConfigError);@@ -200,6 +251,7 @@ describe("the storage block in the template ratchet", () => {expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("storage:");expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("warn_percent: 80");expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("critical_percent: 95");+ expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("pause_percent: 80");expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("warn_days: 30");expect(DEFAULT_BRAIN_CONFIG_YAML).toContain("critical_days: 7");});diff --git a/tests/core/brain/storage-usage.test.ts b/tests/core/brain/storage-usage.test.tsindex a3c5eedc..5de4426a 100644--- a/tests/core/brain/storage-usage.test.ts+++ b/tests/core/brain/storage-usage.test.ts@@ -23,9 +23,12 @@import { afterEach, beforeEach, describe, expect, test } from "bun:test";import { chmodSync, mkdirSync, mkdtempSync, rmSync, symlinkSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";-import { join } from "node:path";+import { basename, dirname, join, sep } from "node:path";+import { DERIVED_STORE_DIR, DERIVED_STORE_FILE } from "../../../src/core/brain/path-constants.ts";+import { brainConfigPath } from "../../../src/core/brain/paths.ts";import { resolveIndexPath } from "../../../src/core/search/paths.ts";+import { STATE_SURFACES } from "../../../src/core/state/surfaces.ts";import {STORAGE_SURFACE,STORAGE_TIER,@@ -33,6 +36,13 @@ import {measureStorageUsage,type StorageUsage,} from "../../../src/core/brain/storage/usage.ts";+import { evaluateCapacityGate } from "../../../src/core/brain/storage/gate.ts";+import {+ STORAGE_CAPACITY_CRITICAL_CODE,+ storageCapacityCheck,+} from "../../../src/core/brain/doctor/storage-capacity.ts";+import type { BrainConfig } from "../../../src/core/brain/types.ts";+import type { DoctorFindings } from "../../../src/core/brain/doctor/check.ts";import { CHMOD_CANNOT_DENY, IS_WINDOWS } from "../../helpers/platform.ts";let vault: string;@@ -204,3 +214,160 @@ describe("measureStorageUsage", () => {);});});++describe("the configured store location", () => {+ // The override the search layer honours: env first, then the machine+ // config's search_db_path. The meter resolves through the search+ // layer's own configured resolver, so the census can never disagree+ // with the indexer about which file is the store.+ const ENV_KEY = "OPEN_SECOND_BRAIN_SEARCH_DB";+ let savedEnv: string | undefined;+ let overrideHome: string;+ let overridePath: string;++ beforeEach(() => {+ savedEnv = process.env[ENV_KEY];+ delete process.env[ENV_KEY];+ overrideHome = mkdtempSync(join(tmpdir(), "o2b-store-override-"));+ overridePath = join(overrideHome, "moved.sqlite");+ });++ afterEach(() => {+ if (savedEnv === undefined) delete process.env[ENV_KEY];+ else process.env[ENV_KEY] = savedEnv;+ rmSync(overrideHome, { recursive: true, force: true });+ });++ test("the store is measured at the configured override, not the default location", () => {+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(join(vault, "Brain", "sources", "AA-page.md"), "x".repeat(1000), "utf8");+ // A stale database left behind at the default location: bytes the+ // search layer never reads. Pre-fix the meter counted THESE as the+ // store, and the real store - the one every search opens - not at+ // all.+ mkdirSync(join(vault, DERIVED_STORE_DIR), { recursive: true });+ writeFileSync(join(vault, DERIVED_STORE_DIR, DERIVED_STORE_FILE), "x".repeat(7000), "utf8");+ process.env[ENV_KEY] = overridePath;+ writeFileSync(overridePath, "x".repeat(3000), "utf8");+ const usage = measureStorageUsage(vault, { now: NOW });+ expect(surfaceBytes(usage, STORAGE_SURFACE.derivedStore)).toBe(3000);+ expect(usage.complete).toBe(true);+ });++ test("a search_db_path in an explicit config file is honoured through configPath", () => {+ // A caller that resolved the search config through an explicit+ // --config holds that path, and the store the indexer opens is the+ // one THAT file names. Resolving through the default config file+ // instead would measure whatever sits at the default location - on+ // this fixture, nothing - while the real store grows elsewhere.+ const configHome = mkdtempSync(join(tmpdir(), "o2b-store-cfgpath-"));+ try {+ const movedDb = join(configHome, "moved.sqlite");+ writeFileSync(movedDb, "x".repeat(2500), "utf8");+ const configPath = join(configHome, "config.yaml");+ writeFileSync(configPath, `vault: ${vault}\nsearch_db_path: ${movedDb}\n`, "utf8");++ const usage = measureStorageUsage(vault, { now: NOW, configPath });+ expect(surfaceBytes(usage, STORAGE_SURFACE.derivedStore)).toBe(2500);+ expect(usage.complete).toBe(true);+ } finally {+ rmSync(configHome, { recursive: true, force: true });+ }+ });++ test("under an override the lane gate and the doctor check agree", () => {+ // The doctor measures the store the operator's searches actually use+ // (ctx.dbPath); pre-fix the lane gate measured the default location+ // instead, so the doctor could report critical while the gate and+ // every ingest ran on a stale file's numbers.+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(brainConfigPath(vault), "schema_version: 1\n\nstorage:\n cap_bytes: 10000\n");+ writeFileSync(join(vault, "Brain", "sources", "AA-page.md"), "x".repeat(6000), "utf8");+ process.env[ENV_KEY] = overridePath;+ writeFileSync(overridePath, "x".repeat(3500), "utf8");++ const decision = evaluateCapacityGate(vault, { capBytes: 10_000, now: NOW });+ const out: DoctorFindings = { issues: [], uncertain: [] };+ storageCapacityCheck.run(+ {+ vault,+ now: NOW,+ config: { schema_version: 1, storage: { cap_bytes: 10_000 } } as BrainConfig,+ dbPath: overridePath,+ configPath: undefined,+ knownBasenames: new Set(),+ idIndex: new Map(),+ preferences: [],+ logs: [],+ },+ out,+ );+ const critical = out.issues.find((i) => i.code === STORAGE_CAPACITY_CRITICAL_CODE);+ expect(critical).toBeDefined();+ // The figures in the doctor's message are the figures the gate+ // decided on: one store, one census.+ const reported = /\((\d+) of \d+ bytes\)/u.exec(critical!.message);+ expect(reported).not.toBeNull();+ expect(Number(reported![1])).toBe(decision.reading!.measuredBytes);+ expect(decision.reading!.measuredBytes).toBeGreaterThan(9_000);+ });+});++describe("the census against the state catalogue", () => {+ test("every catalogued state surface lands in exactly one counted surface", () => {+ // The meter claims to charge every durable state surface for the+ // bytes it keeps on disk. STATE_SURFACES is the catalogue of those+ // locations: seed a file at every row's default location and require+ // the census to see every byte exactly once. A new writer under+ // `.open-second-brain/` (or anywhere else) with no counted surface -+ // or a path the walk excludes, or a byte counted by two surfaces -+ // breaks the delta.+ const PROBE_BYTES = 16;+ const paths = [...new Set(STATE_SURFACES.map((row) => row.derive(vault, null)))];+ // The meter's directory surfaces: a probe written AT one of these+ // roots would be a regular file standing where a walkable directory+ // belongs, and the walk would refuse the surface.+ const DIR_BASENAMES = [+ ".snapshots",+ ".artifacts",+ ".archives",+ ".payloads",+ ".state",+ DERIVED_STORE_DIR,+ ];+ const seeded = new Set<string>();+ for (const path of paths) {+ // A row whose path is a strict prefix of another row's path is a+ // DIRECTORY row: its probe goes inside it, or it would be a regular+ // file standing where another row's parent must be.+ const isParent = paths.some((other) => other !== path && other.startsWith(`${path}${sep}`));+ const target =+ isParent || DIR_BASENAMES.includes(basename(path)) ? join(path, "probe.bin") : path;+ if (seeded.has(target)) continue;+ seeded.add(target);+ mkdirSync(dirname(target), { recursive: true });+ writeFileSync(target, "x".repeat(PROBE_BYTES), "utf8");+ }+ const usage = measureStorageUsage(vault, { now: NOW });+ expect(usage.complete).toBe(true);+ expect(usage.totalBytes).toBe(seeded.size * PROBE_BYTES);+ });++ test("the store's directory is a counted surface beside the store itself", () => {+ // The accumulating state beside brain.sqlite: the hook audit, the+ // session ledger, the maintenance lease database - pre-fix none of+ // it was counted, and the forecast read a busy vault as flat.+ const storeDir = join(vault, DERIVED_STORE_DIR);+ mkdirSync(join(storeDir, "hook-audit"), { recursive: true });+ writeFileSync(join(storeDir, "hook-audit", "2026-W42.jsonl"), "x".repeat(700), "utf8");+ writeFileSync(join(storeDir, "sessions.jsonl"), "x".repeat(300), "utf8");+ writeFileSync(join(storeDir, "maintenance.sqlite"), "x".repeat(500), "utf8");+ // The store itself stays on its own surface, never double-counted.+ writeFileSync(join(storeDir, DERIVED_STORE_FILE), "x".repeat(1000), "utf8");+ writeFileSync(join(storeDir, `${DERIVED_STORE_FILE}-wal`), "x".repeat(200), "utf8");+ const usage = measureStorageUsage(vault, { now: NOW });+ expect(surfaceBytes(usage, STORAGE_SURFACE.derivedStoreDir)).toBe(1500);+ expect(surfaceBytes(usage, STORAGE_SURFACE.derivedStore)).toBe(1200);+ expect(usage.complete).toBe(true);+ });+});diff --git a/tests/core/brain/sync-lockfile.test.ts b/tests/core/brain/sync-lockfile.test.tsindex 26e681eb..bc5656b6 100644--- a/tests/core/brain/sync-lockfile.test.ts+++ b/tests/core/brain/sync-lockfile.test.ts@@ -219,10 +219,11 @@ describe("every module that takes a lock is under a scanned root", () => {/*** Declared lock sites, each with the vault-relative root* {@link lockScanRoots} reaches it through. Verified against the path- * builders: every entry resolves under `Brain/` except the three that+ * builders: every entry resolves under `Brain/` except the four that* live in the derived-store directory - the shared checkpoint substrate* (both the ingest and the session-import lanes lock through it), the- * ingest content manifest, and the session import ledger.+ * ingest content manifest, the session import ledger, and the session+ * mining ledger.*/const DECLARED: ReadonlyMap<string, string> = new Map([["core/brain/architect/generate.ts", "Brain"],@@ -234,12 +235,14 @@ describe("every module that takes a lock is under a scanned root", () => {["core/brain/idempotency-ledger.ts", "Brain"],["core/brain/ingest/content-manifest.ts", ".open-second-brain"],["core/brain/lineage/ledger.ts", "Brain"],+ ["core/brain/metrics.ts", "Brain"],["core/brain/link-graph/hallways.ts", "Brain"],["core/brain/link-graph/orphan-repair.ts", "Brain"],["core/brain/payload-registry.ts", "Brain"],["core/brain/preference-txn.ts", "Brain"],["core/brain/query-demand.ts", "Brain"],["core/brain/receipts/compact.ts", "Brain"],+ ["core/brain/session-mining-ledger.ts", ".open-second-brain"],["core/brain/sessions/discover.ts", ".open-second-brain"],["core/brain/skill-proposals.ts", "Brain"],]);diff --git a/tests/core/brain/trust/operator-summary.test.ts b/tests/core/brain/trust/operator-summary.test.tsindex 3dc7d9a3..5fc1617d 100644--- a/tests/core/brain/trust/operator-summary.test.ts+++ b/tests/core/brain/trust/operator-summary.test.ts@@ -91,6 +91,40 @@ describe("buildOperatorSummary - structural envelope", () => {});});+describe("buildOperatorSummary - collect_errors (review M10)", () => {+ test("a failed maintenance scan lands in the envelope as a named row", () => {+ // A FILE at Brain/preferences makes the dedup scan's readdir refuse;+ // every other reader on this surface degrades instead of throwing.+ rmSync(join(vault, "Brain", "preferences"), { recursive: true });+ writeFileSync(join(vault, "Brain", "preferences"), "not a directory", "utf8");++ const r = buildOperatorSummary(vault, {});+ // The break can refuse more than one scan on this surface (the lint+ // resolver shares the directory); the row is about WHAT refused, so+ // the assertion is on the shape and the named dedup entry.+ const errors = r.collect_errors ?? [];+ expect(errors.length).toBeGreaterThan(0);+ expect(errors.some((e) => e.scan === "dedup")).toBe(true);+ for (const e of errors) {+ expect(e.message.replace(/\\/g, "/")).toContain("preferences");+ }+ });++ test("a healthy vault carries no collect_errors row", () => {+ const r = buildOperatorSummary(vault, {});+ expect(r.collect_errors).toBeUndefined();+ });++ test("the markdown renderer names the collected errors when present", () => {+ rmSync(join(vault, "Brain", "preferences"), { recursive: true });+ writeFileSync(join(vault, "Brain", "preferences"), "not a directory", "utf8");++ const md = renderOperatorSummaryMarkdown(buildOperatorSummary(vault, {}));+ expect(md).toContain("## Collection errors");+ expect(md).toContain("dedup");+ });+});+describe("renderOperatorSummaryMarkdown", () => {test("markdown contains the three signal sections", () => {const r = buildOperatorSummary(vault, { dreamSummary: emptyDream() });diff --git a/tests/core/brain/trust/retrieval-gate.test.ts b/tests/core/brain/trust/retrieval-gate.test.tsindex 971314ff..109c2efb 100644--- a/tests/core/brain/trust/retrieval-gate.test.ts+++ b/tests/core/brain/trust/retrieval-gate.test.ts@@ -14,7 +14,9 @@ import { test, expect, describe } from "bun:test";import {RETRIEVAL_TRUST_EXCLUSION_REASON,+ RETRIEVAL_TRUST_GATE_NAME,classifyRetrievalTrust,+ collectTrustGateRefusals,trustGateAdjuster,} from "../../../../src/core/brain/trust/retrieval-gate.ts";import { UNTRUSTED_SOURCE_TAG } from "../../../../src/core/brain/untrusted-source.ts";@@ -67,10 +69,18 @@ describe("classifyRetrievalTrust", () => {});});-function result(path: string): BrainSearchResult {+/** Mixed-signal frontmatter map keyed by the path names the tests use. */+const frontmatter = (path: string): FrontmatterMap =>+ path === "bad.md"+ ? { status: "quarantine", [UNTRUSTED_SOURCE_TAG]: true }+ : path === "tainted.md"+ ? { [ENTITY_CONTAMINATION_FRONTMATTER_KEY]: true }+ : {};++function result(path: string, chunkId = 10): BrainSearchResult {return Object.freeze({documentId: 1,- chunkId: 10,+ chunkId,path,title: null,content: "body",@@ -101,4 +111,62 @@ describe("trustGateAdjuster", () => {const adjuster = trustGateAdjuster(() => ({ status: "confirmed" }));expect(adjuster.adjust(result("ok.md"))).toEqual({ kind: "keep" });});++ test("under shadow the classification runs but every candidate is kept", () => {+ const saw: string[] = [];+ const adjuster = trustGateAdjuster((path): FrontmatterMap => {+ saw.push(path);+ return path === "bad.md" ? { status: "quarantine" } : {};+ }, "shadow");+ expect(adjuster.adjust(result("bad.md"))).toEqual({ kind: "keep" });+ expect(adjuster.adjust(result("ok.md"))).toEqual({ kind: "keep" });+ // The classification itself still ran over the candidate.+ expect(saw).toEqual(["bad.md", "ok.md"]);+ });++ test("enforce named explicitly behaves exactly as the default", () => {+ const adjuster = trustGateAdjuster(+ (path): FrontmatterMap => (path === "bad.md" ? { status: "quarantine" } : {}),+ "enforce",+ );+ expect(adjuster.adjust(result("bad.md"))).toEqual({+ kind: "exclude",+ reason: RETRIEVAL_TRUST_EXCLUSION_REASON.selfApprovalQuarantine,+ });+ });+});++describe("collectTrustGateRefusals", () => {+ test("records every would-be refusal with all its structural reasons, pool lane", () => {+ const refusals = collectTrustGateRefusals(+ [result("bad.md", 10), result("ok.md", 11), result("tainted.md", 12)],+ frontmatter,+ );+ expect(refusals).toEqual([+ {+ document_id: 1,+ chunk_id: 10,+ path: "bad.md",+ reasons: [+ `${RETRIEVAL_TRUST_GATE_NAME}:${RETRIEVAL_TRUST_EXCLUSION_REASON.selfApprovalQuarantine}`,+ `${RETRIEVAL_TRUST_GATE_NAME}:${RETRIEVAL_TRUST_EXCLUSION_REASON.untrustedSourceProvenance}`,+ ],+ lane: "pool",+ },+ {+ document_id: 1,+ chunk_id: 12,+ path: "tainted.md",+ reasons: [+ `${RETRIEVAL_TRUST_GATE_NAME}:${RETRIEVAL_TRUST_EXCLUSION_REASON.entityContamination}`,+ ],+ lane: "pool",+ },+ ]);+ });++ test("a clean pool collects nothing", () => {+ expect(collectTrustGateRefusals([result("ok.md")], frontmatter)).toEqual([]);+ expect(collectTrustGateRefusals([], frontmatter)).toEqual([]);+ });});diff --git a/tests/core/brain/trust/retrieval-receipts.test.ts b/tests/core/brain/trust/retrieval-receipts.test.tsindex 426c34b7..45524fdd 100644--- a/tests/core/brain/trust/retrieval-receipts.test.ts+++ b/tests/core/brain/trust/retrieval-receipts.test.ts@@ -10,6 +10,8 @@ import { test, expect, describe } from "bun:test";import {buildMemoryTrustAssessment,buildRetrievalDecisionTrace,+ RETRIEVAL_TRUST_REFUSAL_LANE,+ type RetrievalTrustRefusal,} from "../../../../src/core/brain/trust/retrieval-receipts.ts";import type { RankAdjustExclusion } from "../../../../src/core/search/rank-adjust.ts";@@ -88,3 +90,131 @@ describe("buildMemoryTrustAssessment", () => {});});});++// ── Shadow measurements (session-mining-economy, t_05a0eaa4) ────────────────++const poolRefusal: RetrievalTrustRefusal = Object.freeze({+ document_id: 4,+ chunk_id: 40,+ path: "would-be.md",+ reasons: Object.freeze(["trust_gate:self_approval_quarantine"]),+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.pool,+});+const secondPoolRefusal: RetrievalTrustRefusal = Object.freeze({+ document_id: 5,+ chunk_id: 50,+ path: "would-be-2.md",+ reasons: Object.freeze(["trust_gate:entity_contamination"]),+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.pool,+});+const armRefusal: RetrievalTrustRefusal = Object.freeze({+ document_id: 6,+ chunk_id: 60,+ path: "arm-reached.md",+ reasons: Object.freeze(["relational_arm:entity_contamination"]),+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.arm,+});++describe("shadow refusals on the receipts", () => {+ test("the trace keeps excluded untouched and reports refusals beside it", () => {+ const trace = buildRetrievalDecisionTrace({+ surfaced: 5,+ excluded: [],+ shadow: { refusals: [poolRefusal, armRefusal] },+ });+ expect(trace).toEqual({+ evaluated: 5,+ surfaced: 5,+ excluded: 0,+ exclusions: [],+ mode: "shadow",+ refused: 2,+ refusals: [poolRefusal, armRefusal],+ });+ });++ test("without shadow input the trace keeps today's exact shape", () => {+ const trace = buildRetrievalDecisionTrace({ surfaced: 1, excluded: [] });+ expect(Object.keys(trace)).toEqual(["evaluated", "surfaced", "excluded", "exclusions"]);+ });++ test("the assessment reports refused counts per lane, never folded into quarantined", () => {+ const assessment = buildMemoryTrustAssessment({+ surfaced: 5,+ excluded: exclusions,+ shadow: { refusals: [poolRefusal, secondPoolRefusal, armRefusal] },+ });+ expect(assessment.mode).toBe("shadow");+ // `quarantined` keeps meaning actually dropped; the measured-only+ // refusals ride beside it, split per lane.+ expect(assessment.quarantined).toBe(2);+ expect(assessment.refused).toBe(3);+ expect(assessment.refusals_by_lane).toEqual({ arm: 1, pool: 2 });+ expect(assessment.refused_reason_counts).toEqual({+ arm: { "relational_arm:entity_contamination": 1 },+ pool: {+ "trust_gate:entity_contamination": 1,+ "trust_gate:self_approval_quarantine": 1,+ },+ });+ });++ test("without shadow input the assessment keeps today's exact shape", () => {+ const assessment = buildMemoryTrustAssessment({ surfaced: 4, excluded: [] });+ expect(Object.keys(assessment)).toEqual([+ "evaluated",+ "surfaced",+ "quarantined",+ "reason_counts",+ ]);+ });+});++// ── Enforced arm drops on the receipts (session-mining-economy) ─────────────++describe("enforced arm drops on the receipts", () => {+ test("the trace carries the arm's drops without a shadow mode stamp", () => {+ const trace = buildRetrievalDecisionTrace({+ surfaced: 5,+ excluded: exclusions,+ enforce: { refusals: [armRefusal] },+ });+ expect(trace.refused).toBe(1);+ expect(trace.refusals).toEqual([armRefusal]);+ // The drop is receipted, but an enforce receipt never names itself+ // shadow: the mode stamp stays shadow-only.+ expect("mode" in trace).toBe(false);+ });++ test("the assessment aggregates the arm's drops per lane beside quarantined", () => {+ const assessment = buildMemoryTrustAssessment({+ surfaced: 5,+ excluded: exclusions,+ enforce: { refusals: [armRefusal] },+ });+ // `quarantined` keeps meaning the pool's actual drops; the arm's+ // pre-pool drops ride beside them, split per lane.+ expect(assessment.quarantined).toBe(2);+ expect(assessment.refused).toBe(1);+ expect(assessment.refusals_by_lane).toEqual({ arm: 1, pool: 0 });+ expect(assessment.refused_reason_counts).toEqual({+ arm: { "relational_arm:entity_contamination": 1 },+ pool: {},+ });+ expect("mode" in assessment).toBe(false);+ });++ test("no enforce section keeps today's exact shape on both receipts", () => {+ const trace = buildRetrievalDecisionTrace({ surfaced: 1, excluded: [] });+ const assessment = buildMemoryTrustAssessment({ surfaced: 1, excluded: [] });+ expect("refusals" in trace).toBe(false);+ expect("refused" in trace).toBe(false);+ expect("refusals_by_lane" in assessment).toBe(false);+ expect(Object.keys(assessment)).toEqual([+ "evaluated",+ "surfaced",+ "quarantined",+ "reason_counts",+ ]);+ });+});diff --git a/tests/core/config.test.ts b/tests/core/config.test.tsindex 681e93bd..fa7ba027 100644--- a/tests/core/config.test.ts+++ b/tests/core/config.test.ts@@ -9,6 +9,7 @@ import {discoverConfig,parseSimpleYaml,resolveAgentName,+ resolveBacklogNudgeEnabled,resolveExposeHostPaths,resolveLinkOutputFormat,resolveDecisionRecallMaxPerSession,@@ -45,6 +46,7 @@ beforeEach(() => {"OPEN_SECOND_BRAIN_RECALL_ADEQUACY_WEAK","OPEN_SECOND_BRAIN_RECALL_ADEQUACY_MIN_RESULTS","OPEN_SECOND_BRAIN_EXPOSE_HOST_PATHS",+ "OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED","OPEN_SECOND_BRAIN_DECISION_RECALL_MAX_PER_SESSION","OPEN_SECOND_BRAIN_DECISION_RECALL_MIN_SPACING_TURNS",]) {@@ -517,6 +519,37 @@ describe("resolveExposeHostPaths", () => {});});+describe("resolveBacklogNudgeEnabled", () => {+ test("defaults to false when nothing configured", () => {+ expect(resolveBacklogNudgeEnabled(join(tmp, "missing.yaml"))).toBe(false);+ });++ test("reads true from config", () => {+ const cfg = join(tmp, "config.yaml");+ writeFileSync(cfg, 'backlog_nudge_enabled: "true"\n');+ expect(resolveBacklogNudgeEnabled(cfg)).toBe(true);+ });++ test("accepts 1 as truthy", () => {+ const cfg = join(tmp, "config.yaml");+ writeFileSync(cfg, 'backlog_nudge_enabled: "1"\n');+ expect(resolveBacklogNudgeEnabled(cfg)).toBe(true);+ });++ test("any other value stays false", () => {+ const cfg = join(tmp, "config.yaml");+ writeFileSync(cfg, 'backlog_nudge_enabled: "yes"\n');+ expect(resolveBacklogNudgeEnabled(cfg)).toBe(false);+ });++ test("env wins over config", () => {+ process.env["OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED"] = "true";+ const cfg = join(tmp, "config.yaml");+ writeFileSync(cfg, 'backlog_nudge_enabled: "false"\n');+ expect(resolveBacklogNudgeEnabled(cfg)).toBe(true);+ });+});+describe("vaultStoreReference", () => {test("has the vault:// prefix and 32 hex chars (128-bit keyed digest)", () => {const cfg = join(tmp, "config.yaml");diff --git a/tests/core/search/config.test.ts b/tests/core/search/config.test.tsindex a4ebdefd..57647de2 100644--- a/tests/core/search/config.test.ts+++ b/tests/core/search/config.test.ts@@ -44,6 +44,8 @@ const ENV_KEYS = ["OPEN_SECOND_BRAIN_SEARCH_CHAIN_STOP","OPEN_SECOND_BRAIN_SEARCH_CHAIN_STOP_SCORE","OPEN_SECOND_BRAIN_SEARCH_POOL_MAX_PER_SESSION",+ "OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE",+ "OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE",];beforeEach(() => {@@ -509,3 +511,96 @@ test("the bench pin accepts the pool-cap knob name in a recall pin", () => {/poolCapNonsense.*not a known knob/,);});++// ── Retrieval trust gate mode (session-mining-economy, t_05a0eaa4) ──────────++test("the trust gate mode defaults off and stays absent from the resolved recall config", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const recall = resolveSearchConfig({ vault: tmp, configPath }).recall;+ expect(recall.searchTrustGateMode).toBeUndefined();+ // Spread-omitted at off: exact-shape assertions on the resolved config+ // keep holding, exactly like the other opt-in recall fields.+ expect(Object.keys(recall)).not.toContain("searchTrustGateMode");+});++test("an explicit trust gate mode resolves from config, env winning", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_mode: "shadow"\n`);+ expect(resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode).toBe("shadow");++ writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_mode: "enforce"\n`);+ expect(resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode).toBe(+ "enforce",+ );++ process.env["OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE"] = "shadow";+ expect(resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode).toBe("shadow");+});++test("the legacy boolean maps true to shadow and false to off", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_enabled: true\n`);+ expect(resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode).toBe("shadow");++ writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_enabled: false\n`);+ expect(+ resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode,+ ).toBeUndefined();+});++test("an explicit mode wins over the legacy boolean, including explicit off", () => {+ writeFileSync(+ configPath,+ `vault: "${tmp}"\nsearch_trust_gate_enabled: true\nsearch_trust_gate_mode: "off"\n`,+ );+ expect(+ resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode,+ ).toBeUndefined();++ writeFileSync(+ configPath,+ `vault: "${tmp}"\nsearch_trust_gate_enabled: false\nsearch_trust_gate_mode: "enforce"\n`,+ );+ expect(resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode).toBe(+ "enforce",+ );+});++test("an invalid trust gate mode is refused naming the key in force", () => {+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_mode: "measure"\n`);+ let err: SearchError | null = null;+ try {+ resolveSearchConfig({ vault: tmp, configPath });+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("INVALID_INPUT");+ expect(err?.message).toContain("search_trust_gate_mode");+ expect(err?.message).toContain("measure");++ // Env in force: the refusal names the env var, never the config key+ // that lost the precedence race.+ process.env["OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE"] = "maybe";+ let envMessage = "";+ try {+ resolveSearchConfig({ vault: tmp, configPath });+ } catch (e) {+ envMessage = (e as Error).message;+ }+ expect(envMessage).toContain("OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE");+ expect(envMessage).not.toContain("search_trust_gate_mode must");+});++test("a trust gate mode change re-keys the config fingerprint", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const off = resolveSearchConfig({ vault: tmp, configPath });+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_mode: "shadow"\n`);+ const shadow = resolveSearchConfig({ vault: tmp, configPath });+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_mode: "enforce"\n`);+ const enforce = resolveSearchConfig({ vault: tmp, configPath });++ // The mode decides whether the gate excludes, so a change must+ // invalidate cached rows; the legacy boolean entry already re-keyed,+ // and the mode adds its own dimension on top.+ expect(configFingerprint(shadow)).not.toBe(configFingerprint(off));+ expect(configFingerprint(enforce)).not.toBe(configFingerprint(shadow));+ expect(configFingerprint(enforce)).not.toBe(configFingerprint(off));+});diff --git a/tests/core/search/hybrid-deadline.test.ts b/tests/core/search/hybrid-deadline.test.tsindex a701333f..03664e8e 100644--- a/tests/core/search/hybrid-deadline.test.ts+++ b/tests/core/search/hybrid-deadline.test.ts@@ -324,13 +324,16 @@ test("the new code rides the closed vocabulary and its sentence names the deadliexpect(sentence.toLowerCase()).toContain("deadline");});-/** Keyword-only config, trust gate on, remote rerank at `rerankBaseUrl`. */+/** Keyword-only config, trust gate enforcing, remote rerank at `rerankBaseUrl`. */function stalledRerankConfig(opts: { cacheEnabled?: boolean; rerankBaseUrl?: string } = {}) {return {...makeConfig({vault,dbPath,retrievalTrustGateEnabled: true,+ // The gate must EXCLUDE here, and the legacy boolean alone now+ // measures (shadow): enforcement is named explicitly.+ searchTrustGateMode: "enforce",cacheEnabled: opts.cacheEnabled ?? false,rerank: {enabled: true,diff --git a/tests/core/search/relational-arm.test.ts b/tests/core/search/relational-arm.test.tsindex c20ec994..8498fe64 100644--- a/tests/core/search/relational-arm.test.ts+++ b/tests/core/search/relational-arm.test.ts@@ -118,11 +118,10 @@ test("a non-relational query is byte-identical between arm on and off (rrf)", as// ─── entity bridges and arm runtime options (truth-correctable-time-aware) ───-import { Database } from "bun:sqlite";-import {ENTITY_BRIDGES_CONFIG,ENTITY_BRIDGES_ENV,+ RELATIONAL_ARM_REFUSAL_REASON,resolveEntityBridgesEnabled,runRelationalArm,} from "../../../src/core/search/pipeline/relational-arm.ts";@@ -471,3 +470,377 @@ test("the arm's knobs resolve from the caller's config path, not the default filawait store.close();}});++// ─── edge-level eligibility adjuster (session-mining-economy, t_05a0eaa4) ───+//+// Each walked node and its arriving edge are classified from the same+// controlled-vocabulary frontmatter signals the page gate reads, plus the+// validity window at query time and the relation-polarity vocabulary.+// Under enforce an ineligible edge is removed from the arm's contribution+// set; under shadow it is counted only. Refusals carry named structural+// reason tokens and are reported per lane into the receipts' refusal+// record - distinct from `excluded` and from the arm's identical-to-absent+// provenance convention: a refusal is counted, a provenance omission is+// not reported.++import { ENTITY_CONTAMINATION_FRONTMATTER_KEY } from "../../../src/core/brain/truth/contamination.ts";+import { RETRIEVAL_TRUST_REFUSAL_LANE } from "../../../src/core/brain/trust/retrieval-receipts.ts";+import type { RelationalArmOutcome } from "../../../src/core/search/pipeline/relational-arm.ts";++/** A config file naming the gate mode, at a path the arm can be handed. */+function modeConfigPath(mode: "shadow" | "enforce"): string {+ const path = join(vault, `arm-mode-${mode}.yaml`);+ writeFileSync(path, `search_trust_gate_mode: ${mode}\n`);+ return path;+}++function refusalPaths(outcome: RelationalArmOutcome): string[] {+ return (outcome.refusals ?? []).map((r) => r.path);+}++test("enforce removes an ineligible walked node from the arm's contribution set", () => {+ writeMd(vault, "seed.md", '---\nrelated: "[[dirty]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "dirty.md",+ `---\n${ENTITY_CONTAMINATION_FRONTMATTER_KEY}: true\n---\n\nbeta contaminated content.`,+ );+ const store = armFixtureStore({ docs: { seed: 1, dirty: 2 }, typed: [[1, "related", 2]] });+ const outcome = runRelationalArm(store, vault, "[[seed]] related", {+ configPath: modeConfigPath("enforce"),+ });+ // Removed from the contribution set: no chunk, no provenance.+ expect(outcome.rankedChunkIds).toEqual([]);+ expect(outcome.reachByChunk.size).toBe(0);+ // And counted, with the named structural reason, on the arm's lane.+ expect(outcome.refusals).toEqual([+ {+ document_id: 2,+ chunk_id: 2,+ path: "dirty.md",+ reasons: [`relational_arm:${RELATIONAL_ARM_REFUSAL_REASON.entityContamination}`],+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.arm,+ },+ ]);+});++test("shadow counts the same refusal but keeps the contribution", () => {+ writeMd(vault, "seed.md", '---\nrelated: "[[dirty]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "dirty.md",+ `---\n${ENTITY_CONTAMINATION_FRONTMATTER_KEY}: true\n---\n\nbeta contaminated content.`,+ );+ const store = armFixtureStore({ docs: { seed: 1, dirty: 2 }, typed: [[1, "related", 2]] });+ const outcome = runRelationalArm(store, vault, "[[seed]] related", {+ configPath: modeConfigPath("shadow"),+ });+ expect(outcome.rankedChunkIds).toEqual([2]);+ expect(outcome.reachByChunk.get(2)?.via).toEqual(["related"]);+ expect(outcome.refusals).toEqual([+ {+ document_id: 2,+ chunk_id: 2,+ path: "dirty.md",+ reasons: [`relational_arm:${RELATIONAL_ARM_REFUSAL_REASON.entityContamination}`],+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.arm,+ },+ ]);+});++test("mode off leaves the outcome shape byte-identical and reports nothing", () => {+ writeMd(vault, "seed.md", '---\nrelated: "[[dirty]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "dirty.md",+ `---\n${ENTITY_CONTAMINATION_FRONTMATTER_KEY}: true\n---\n\nbeta contaminated content.`,+ );+ const store = armFixtureStore({ docs: { seed: 1, dirty: 2 }, typed: [[1, "related", 2]] });+ const outcome = runRelationalArm(store, vault, "[[seed]] related", {});+ expect(outcome.rankedChunkIds).toEqual([2]);+ // No refusals key at all - not an empty placeholder - so the serialized+ // outcome is byte-identical to the pre-adjuster one.+ expect("refusals" in outcome).toBe(false);+ expect(JSON.stringify(outcome)).not.toContain("refusals");+});++test("a validity window not covering query time is refused with its own token", () => {+ writeMd(+ vault,+ "seed.md",+ '---\nrelated: "[[closed]]"\nextends: "[[pending]]"\n---\n\nalpha seed.',+ );+ writeMd(vault, "closed.md", '---\nvalid_until: "2000-01-01"\n---\n\nbeta closed-window content.');+ writeMd(+ vault,+ "pending.md",+ '---\nvalid_from: "2999-01-01"\n---\n\ngamma pending-window content.',+ );+ const store = armFixtureStore({+ docs: { seed: 1, closed: 2, pending: 3 },+ typed: [+ [1, "related", 2],+ [1, "extends", 3],+ ],+ });+ const outcome = runRelationalArm(store, vault, "[[seed]] related extends", {+ configPath: modeConfigPath("enforce"),+ });+ expect(outcome.rankedChunkIds).toEqual([]);+ expect(refusalPaths(outcome).toSorted()).toEqual(["closed.md", "pending.md"]);+ const byPath = new Map((outcome.refusals ?? []).map((r) => [r.path, r]));+ expect(byPath.get("closed.md")?.reasons).toEqual([+ `relational_arm:${RELATIONAL_ARM_REFUSAL_REASON.closedValidityWindow}`,+ ]);+ expect(byPath.get("pending.md")?.reasons).toEqual([+ `relational_arm:${RELATIONAL_ARM_REFUSAL_REASON.pendingValidityWindow}`,+ ]);+});++test("an unparseable declared window is refused where a current one passes", () => {+ const configPath = modeConfigPath("enforce");+ // A window whose inclusive end is still ahead covers query time: pass.+ writeMd(vault, "seed.md", '---\nrelated: "[[open]]"\n---\n\nalpha seed.');+ writeMd(vault, "open.md", '---\nvalid_until: "2999-01-01"\n---\n\nbeta still valid.');+ const openStore = armFixtureStore({ docs: { seed: 1, open: 2 }, typed: [[1, "related", 2]] });+ const current = runRelationalArm(openStore, vault, "[[seed]] related", { configPath });+ expect(current.rankedChunkIds).toEqual([2]);+ expect(current.refusals).toEqual([]);++ // A DECLARED value that cannot be parsed is a structural refusal, never+ // a silent pass: the window cannot be honored.+ writeMd(vault, "seed2.md", '---\nrelated: "[[garbled]]"\n---\n\nalpha seed two.');+ writeMd(vault, "garbled.md", "---\nvalid_until: not-a-date\n---\n\ngamma garbled window.");+ const garbledStore = armFixtureStore({+ docs: { seed2: 3, garbled: 4 },+ typed: [[3, "related", 4]],+ });+ const garbled = runRelationalArm(garbledStore, vault, "[[seed2]] related", { configPath });+ expect(refusalPaths(garbled)).toEqual(["garbled.md"]);+ expect(garbled.refusals?.[0]?.reasons).toEqual([+ `relational_arm:${RELATIONAL_ARM_REFUSAL_REASON.invalidValidityWindow}`,+ ]);+});++test("an edge whose polarity is a counted exclusion is refused", () => {+ writeMd(vault, "seed.md", '---\ncontradicts: "[[rival]]"\n---\n\nalpha seed.');+ writeMd(vault, "rival.md", "# Rival\n\nbeta contradictory content.");+ const store = armFixtureStore({ docs: { seed: 1, rival: 2 }, typed: [[1, "contradicts", 2]] });+ const outcome = runRelationalArm(store, vault, "[[seed]] contradicts", {+ configPath: modeConfigPath("enforce"),+ });+ expect(outcome.rankedChunkIds).toEqual([]);+ expect(outcome.refusals).toEqual([+ {+ document_id: 2,+ chunk_id: 2,+ path: "rival.md",+ reasons: [`relational_arm:${RELATIONAL_ARM_REFUSAL_REASON.contradictsPolarity}`],+ lane: RETRIEVAL_TRUST_REFUSAL_LANE.arm,+ },+ ]);+});++test("a chain through an ineligible node refuses the downstream arrival too", () => {+ writeMd(vault, "seed.md", '---\nrelated: "[[mid]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "mid.md",+ '---\nstatus: quarantine\nextends: "[[far]]"\n---\n\nbeta quarantined mid.',+ );+ writeMd(vault, "far.md", "# Far\n\ngamma downstream content.");+ const store = armFixtureStore({+ docs: { seed: 1, mid: 2, far: 3 },+ typed: [+ [1, "related", 2],+ [2, "extends", 3],+ ],+ });+ const outcome = runRelationalArm(store, vault, "[[seed]] related extends", {+ configPath: modeConfigPath("enforce"),+ });+ expect(outcome.rankedChunkIds).toEqual([]);+ expect(refusalPaths(outcome).toSorted()).toEqual(["far.md", "mid.md"]);+ // The far node's own frontmatter is clean: its refusal is the arrived+ // edge's source endpoint failing the same signals.+ const far = (outcome.refusals ?? []).find((r) => r.path === "far.md");+ expect(far?.reasons).toEqual(["relational_arm:self_approval_quarantine"]);+});++test("shadow counts a whole ineligible chain while the contributions stand", () => {+ writeMd(vault, "seed.md", '---\nrelated: "[[mid]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "mid.md",+ '---\nstatus: quarantine\nextends: "[[far]]"\n---\n\nbeta quarantined mid.',+ );+ writeMd(vault, "far.md", "# Far\n\ngamma downstream content.");+ const store = armFixtureStore({+ docs: { seed: 1, mid: 2, far: 3 },+ typed: [+ [1, "related", 2],+ [2, "extends", 3],+ ],+ });+ const outcome = runRelationalArm(store, vault, "[[seed]] related extends", {+ configPath: modeConfigPath("shadow"),+ });+ expect(outcome.rankedChunkIds.toSorted()).toEqual([2, 3]);+ expect(refusalPaths(outcome).toSorted()).toEqual(["far.md", "mid.md"]);+});++test("a provenance omission is not a refusal: reach gating stays identical-to-absent", () => {+ writeMd(vault, "seed.md", '---\nrelated: "[[mid]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "mid.md",+ '---\nvisibility: ["private"]\nextends: "[[far]]"\n---\n\nbeta private-but-clean mid.',+ );+ writeMd(vault, "far.md", "# Far\n\ngamma downstream content.");+ const store = armFixtureStore({+ docs: { seed: 1, mid: 2, far: 3 },+ typed: [+ [1, "related", 2],+ [2, "extends", 3],+ ],+ });+ const outcome = runRelationalArm(store, vault, "[[seed]] related extends", {+ configPath: modeConfigPath("enforce"),+ reach: TRANSPORT_REACH.remote,+ });+ // The walk still reaches both nodes - visibility gates the ordered+ // provenance path only - and the withheld step reports nothing.+ expect(outcome.rankedChunkIds.toSorted()).toEqual([2, 3]);+ expect(outcome.refusals).toEqual([]);+ const farReach = outcome.reachByChunk.get(3);+ expect(farReach).toBeDefined();+ // The far node's provenance path omits the unreadable-at-reach mid hop+ // without naming or counting it.+ expect(farReach!.path.some((step) => step.documentId === 2)).toBe(false);+});++test("a withheld quarantined arrival is receipted identically to its absence", () => {+ // The walked node is BOTH quarantined and reserved against remote+ // reads: the row-level filters withhold its row, so a refusal record+ // naming it would hand a caller the path, the ids and a structural+ // fact about a page it cannot read. The receipts must read exactly as+ // they would had the page never existed.+ writeMd(vault, "seed.md", '---\nrelated: "[[secret]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "secret.md",+ '---\nstatus: quarantine\nvisibility: ["private"]\n---\n\nbeta secret content.',+ );+ const store = armFixtureStore({ docs: { seed: 1, secret: 2 }, typed: [[1, "related", 2]] });+ const present = runRelationalArm(store, vault, "[[seed]] related", {+ configPath: modeConfigPath("shadow"),+ reach: TRANSPORT_REACH.remote,+ });+ expect(present.refusals).toEqual([]);+ expect(JSON.stringify(present)).not.toContain("secret.md");+ // The same walk over a store where the node does not exist refuses+ // nothing, and the refusal records are the same bytes.+ const absent = runRelationalArm(+ armFixtureStore({ docs: { seed: 1 }, typed: [] }),+ vault,+ "[[seed]] related",+ {+ configPath: modeConfigPath("shadow"),+ reach: TRANSPORT_REACH.remote,+ },+ );+ expect(JSON.stringify(present.refusals)).toBe(JSON.stringify(absent.refusals));+});++test("a withheld source page's signals ride no readable node's refusal", () => {+ // The far node's own frontmatter is clean; its refusal reasons come+ // entirely from the quarantined page the walk crossed - and that page+ // is reserved against the caller's reach. Naming far would disclose+ // mid's marker status, so mid contributes no reasons to any record.+ writeMd(vault, "seed.md", '---\nrelated: "[[mid]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "mid.md",+ '---\nstatus: quarantine\nvisibility: ["private"]\nextends: "[[far]]"\n---\n\nbeta hidden mid.',+ );+ writeMd(vault, "far.md", "# Far\n\ngamma downstream content.");+ const store = armFixtureStore({+ docs: { seed: 1, mid: 2, far: 3 },+ typed: [+ [1, "related", 2],+ [2, "extends", 3],+ ],+ });+ const shadow = runRelationalArm(store, vault, "[[seed]] related extends", {+ configPath: modeConfigPath("shadow"),+ reach: TRANSPORT_REACH.remote,+ });+ // Seeds are not walked nodes; both reached nodes still contribute under+ // shadow, and no refusal names anything about the withheld page.+ expect(shadow.rankedChunkIds.toSorted()).toEqual([2, 3]);+ expect(refusalPaths(shadow)).toEqual([]);+ // Enforce keeps the full excluding verdict - the drops stay, only the+ // records disappear.+ const enforced = runRelationalArm(store, vault, "[[seed]] related extends", {+ configPath: modeConfigPath("enforce"),+ reach: TRANSPORT_REACH.remote,+ });+ expect(enforced.rankedChunkIds).toEqual([]);+ expect(refusalPaths(enforced)).toEqual([]);+});++test("a malformed mode value fails only the relational query the gate governs", () => {+ writeMd(vault, "seed.md", '---\nrelated: "[[near]]"\n---\n\nalpha topic seed document.');+ writeMd(vault, "near.md", "# Near\n\nbeta content.");+ const store = armFixtureStore({ docs: { seed: 1, near: 2 }, typed: [[1, "related", 2]] });+ const configPath = join(vault, "arm-mode-bad.yaml");+ writeFileSync(configPath, "search_trust_gate_mode: measure\n");+ expect(() => runRelationalArm(store, vault, "alpha topic", { configPath })).not.toThrow();+ let message = "";+ try {+ runRelationalArm(store, vault, "[[seed]] related", { configPath });+ } catch (e) {+ message = (e as Error).message;+ }+ expect(message).toContain("search_trust_gate_mode");+});++test("the eligibility gate does not run when the arm does not walk", () => {+ // Non-relational query: no walk, no knob resolution, no refusals.+ writeMd(vault, "seed.md", "# Seed\n\nalpha topic.");+ const store = armFixtureStore({ docs: { seed: 1 }, typed: [] });+ const outcome = runRelationalArm(store, vault, "alpha topic", {+ configPath: modeConfigPath("enforce"),+ });+ expect(outcome.rankedChunkIds).toEqual([]);+ expect("refusals" in outcome).toBe(false);+});++test("the resolved trust-gate mode reaches the arm through the caller's config path", async () => {+ // The arm's only route to `dirty.md` is the walk: its body shares no+ // keyword terms with the query. Enforce withholds it from the pack;+ // shadow surfaces it with the arm's attribution, refusal counted.+ writeMd(vault, "seed.md", '---\nrelated: "[[dirty]]"\n---\n\nalpha topic seed document.');+ writeMd(+ vault,+ "dirty.md",+ `---\n${ENTITY_CONTAMINATION_FRONTMATTER_KEY}: true\n---\n\nbeta divergent content.`,+ );+ await indexVault(makeConfig({ vault, dbPath }));+ const configPath = join(vault, "arm-gate.yaml");+ const cfgFor = (mode: "shadow" | "enforce") => {+ writeFileSync(configPath, `search_trust_gate_mode: ${mode}\n`);+ return {+ ...makeConfig({ vault, dbPath, fusionMode: "rrf", relationalArmEnabled: true }),+ configPath,+ };+ };++ const enforced = await search(cfgFor("enforce"), { query: "alpha [[seed]] related" });+ expect(enforced.results.some((r) => r.path === "dirty.md")).toBe(false);++ const shadowed = await search(cfgFor("shadow"), { query: "alpha [[seed]] related" });+ const dirty = shadowed.results.find((r) => r.path === "dirty.md");+ expect(dirty).toBeDefined();+ expect(dirty!.reasons.some((x) => x.startsWith("relational:"))).toBe(true);+});diff --git a/tests/core/search/rerank.decision-model.search.test.ts b/tests/core/search/rerank.decision-model.search.test.tsindex 3ad16b46..6f4e7aa5 100644--- a/tests/core/search/rerank.decision-model.search.test.ts+++ b/tests/core/search/rerank.decision-model.search.test.ts@@ -410,6 +410,9 @@ describe("decision-model rerank under the hybrid deadline", () => {decision_model_timeout_ms: String(DECISION_TIMEOUT_MS),search_hybrid_deadline_ms: String(DEADLINE_MS),search_trust_gate_enabled: "true",+ // The gate must EXCLUDE here, and the legacy boolean alone now+ // measures (shadow): enforcement is named explicitly.+ search_trust_gate_mode: "enforce",});await indexVault(cfg);server.setReply(() => ({ hang: true }));diff --git a/tests/core/search/retrieval-trust-gate.test.ts b/tests/core/search/retrieval-trust-gate.test.tsindex b54545f3..c15a8df6 100644--- a/tests/core/search/retrieval-trust-gate.test.ts+++ b/tests/core/search/retrieval-trust-gate.test.ts@@ -1,21 +1,34 @@/**- * Retrieval trust gate over the real search path (t_5f61130a).+ * Retrieval trust gate over the real search path (t_5f61130a), with the+ * session-mining-economy measure-only mode (t_05a0eaa4).*- * With the gate on, quarantined material (classified structurally from- * frontmatter markers) is zero-ranked - excluded from the pack - while- * being counted with reasons in the retrieval_decision_trace receipt, and- * every pack also carries a memory_trust_assessment receipt. With the gate- * off the ranking output and the outcome shape are byte-identical, which- * the ablation test asserts directly (exact result / rank deltas).+ * Three resolved modes:+ * - off (default): byte-identical ranking and outcome shape.+ * - shadow: the classification runs, nothing is excluded, and every+ * would-be refusal is recorded with its structural reasons into the+ * receipts, whose `excluded` counts stay untouched.+ * - enforce: today's excluding behavior, made explicit.+ *+ * The legacy `search_trust_gate_enabled` boolean alone now maps to+ * shadow: a vault must name `enforce` to exclude.*/-import { test, expect, beforeEach, afterEach, describe } from "bun:test";+import { test, expect, beforeEach, afterEach, describe, spyOn } from "bun:test";+import { rmSync, writeFileSync } from "node:fs";+import { join } from "node:path";import { indexVault } from "../../../src/core/search/indexer.ts";import { search } from "../../../src/core/search/search.ts";-import type { BrainSearchResult } from "../../../src/core/search/types.ts";+import type { BrainSearchResult, ResolvedSearchConfig } from "../../../src/core/search/types.ts";+import { TRANSPORT_REACH } from "../../../src/core/graph/transport-reach.ts";import { UNTRUSTED_SOURCE_TAG } from "../../../src/core/brain/untrusted-source.ts";import { ENTITY_CONTAMINATION_FRONTMATTER_KEY } from "../../../src/core/brain/truth/contamination.ts";+import {+ EXPLAIN_OFF,+ explainEnvelope,+ RETRIEVAL_DECISION_TRACE_KEY,+ RETRIEVAL_TRUST_GATE_MODE_KEY,+} from "../../../src/core/search/explain-envelope.ts";import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";let vault: string;@@ -28,6 +41,14 @@ beforeEach(() => {afterEach(() => cleanup());+/**+ * Name the gate mode explicitly on an already-resolved config: the mode+ * is a recall knob of the resolved config, spread-omitted at off.+ */+function withMode(cfg: ResolvedSearchConfig, mode: "shadow" | "enforce"): ResolvedSearchConfig {+ return { ...cfg, recall: { ...cfg.recall, searchTrustGateMode: mode } };+}+function stableProjection(r: BrainSearchResult) {return {path: r.path,@@ -41,42 +62,115 @@ const CLEAN = "# Clean\n\nThe widget calibration routine runs every morning.";const QUARANTINED = "---\nstatus: quarantine\n---\n\n# Bad\n\nThe widget calibration is unsafe.";describe("retrieval trust gate on the search path", () => {- test("ablation: gate off vs on shows the exact result/rank delta", async () => {+ test("ablation: off vs shadow vs enforce shows the exact deltas", async () => {writeMd(vault, "clean.md", CLEAN);writeMd(vault, "quarantined.md", QUARANTINED);const off = makeConfig({ vault, dbPath });- const on = makeConfig({ vault, dbPath, retrievalTrustGateEnabled: true });+ const shadow = withMode(off, "shadow");+ const enforce = withMode(off, "enforce");await indexVault(off);- const gateOff = await search(off, { query: "widget calibration", limit: 5 });- const gateOn = await search(on, { query: "widget calibration", limit: 5 });+ // Freeze the clock across the three searches: recency decay is a+ // continuous function of wall-clock time, so two real Date.now()+ // reads even a millisecond apart would perturb the low-order digits+ // of the compared scores (recency clock drift aside) for reasons+ // that have nothing to do with the gate under test - the same spy+ // the relational-arm byte-identity test carries.+ const nowMs = Date.now();+ const dateNowSpy = spyOn(Date, "now").mockReturnValue(nowMs);+ let gateOff: Awaited<ReturnType<typeof search>>;+ let gateShadow: Awaited<ReturnType<typeof search>>;+ let gateEnforce: Awaited<ReturnType<typeof search>>;+ try {+ gateOff = await search(off, { query: "widget calibration", limit: 5 });+ gateShadow = await search(shadow, { query: "widget calibration", limit: 5 });+ gateEnforce = await search(enforce, { query: "widget calibration", limit: 5 });+ } finally {+ dateNowSpy.mockRestore();+ }- // Gate off: both notes surface; no receipts on the outcome.+ // Off: both notes surface; no receipts on the outcome.expect(gateOff.results.map((r) => r.path).toSorted()).toEqual(["clean.md", "quarantined.md"]);expect(gateOff.retrievalDecisionTrace).toBeUndefined();expect(gateOff.memoryTrustAssessment).toBeUndefined();- // Gate on: the quarantined note is the only rank delta - excluded.- expect(gateOn.results.map((r) => r.path)).toEqual(["clean.md"]);+ // Shadow: byte-identical ranking to off - the quarantined note still+ // surfaces - with the refusal measured in the receipts.+ expect(gateShadow.results.map(stableProjection)).toEqual(gateOff.results.map(stableProjection));++ // Enforce: the quarantined note is the only rank delta - excluded -+ // exactly the pre-mode excluding behavior.+ expect(gateEnforce.results.map((r) => r.path)).toEqual(["clean.md"]);const removed = gateOff.results.map((r) => r.path)- .filter((p) => !gateOn.results.some((r) => r.path === p));+ .filter((p) => !gateEnforce.results.some((r) => r.path === p));expect(removed).toEqual(["quarantined.md"]);- // The surviving note ranks byte-identically (recency clock drift aside).const cleanOff = gateOff.results.find((r) => r.path === "clean.md")!;- const cleanOn = gateOn.results.find((r) => r.path === "clean.md")!;- expect(stableProjection(cleanOn)).toEqual(stableProjection(cleanOff));+ const cleanEnforce = gateEnforce.results.find((r) => r.path === "clean.md")!;+ expect(stableProjection(cleanEnforce)).toEqual(stableProjection(cleanOff));+ });++ test("shadow measures the refusal without excluding: excluded stays untouched", async () => {+ writeMd(vault, "clean.md", CLEAN);+ writeMd(vault, "quarantined.md", QUARANTINED);+ const cfg = withMode(makeConfig({ vault, dbPath }), "shadow");++ await indexVault(cfg);+ const out = await search(cfg, { query: "widget calibration", limit: 5 });++ // Nothing was excluded: the quarantined note surfaces as usual.+ expect(out.results.map((r) => r.path).toSorted()).toEqual(["clean.md", "quarantined.md"]);+ const trace = out.retrievalDecisionTrace!;+ expect(trace.mode).toBe("shadow");+ expect(trace.excluded).toBe(0);+ expect(trace.exclusions).toEqual([]);+ expect(trace.surfaced).toBe(2);+ expect(trace.evaluated).toBe(2);+ expect(trace.refused).toBe(1);+ expect(trace.refusals).toHaveLength(1);+ expect(trace.refusals![0]!.path).toBe("quarantined.md");+ expect(trace.refusals![0]!.reasons).toEqual(["trust_gate:self_approval_quarantine"]);+ expect(trace.refusals![0]!.lane).toBe("pool");+ expect(typeof trace.refusals![0]!.document_id).toBe("number");+ expect(typeof trace.refusals![0]!.chunk_id).toBe("number");++ const assessment = out.memoryTrustAssessment!;+ expect(assessment.mode).toBe("shadow");+ expect(assessment.quarantined).toBe(0);+ expect(assessment.refused).toBe(1);+ // Both lanes are reported; a lane that measured nothing is a+ // measured zero, not an absence.+ expect(assessment.refusals_by_lane).toEqual({ arm: 0, pool: 1 });+ expect(assessment.refused_reason_counts).toEqual({+ pool: { "trust_gate:self_approval_quarantine": 1 },+ arm: {},+ });});- test("quarantined material reaches no pack but is counted with reasons", async () => {+ test("the legacy boolean alone now maps to shadow (measure, not exclude)", async () => {writeMd(vault, "clean.md", CLEAN);writeMd(vault, "quarantined.md", QUARANTINED);+ // No explicit mode: the legacy true maps to shadow.const cfg = makeConfig({ vault, dbPath, retrievalTrustGateEnabled: true });await indexVault(cfg);const out = await search(cfg, { query: "widget calibration", limit: 5 });- expect(out.results.some((r) => r.path === "quarantined.md")).toBe(false);+ expect(out.results.map((r) => r.path).toSorted()).toEqual(["clean.md", "quarantined.md"]);+ expect(out.retrievalDecisionTrace!.mode).toBe("shadow");+ expect(out.retrievalDecisionTrace!.refused).toBe(1);+ expect(out.retrievalDecisionTrace!.excluded).toBe(0);+ });++ test("enforce named explicitly keeps the excluding behavior and today's receipt shape", async () => {+ writeMd(vault, "clean.md", CLEAN);+ writeMd(vault, "quarantined.md", QUARANTINED);+ const cfg = withMode(makeConfig({ vault, dbPath, retrievalTrustGateEnabled: true }), "enforce");++ await indexVault(cfg);+ const out = await search(cfg, { query: "widget calibration", limit: 5 });++ expect(out.results.map((r) => r.path)).toEqual(["clean.md"]);const trace = out.retrievalDecisionTrace!;expect(trace.excluded).toBe(1);expect(trace.surfaced).toBe(1);@@ -84,14 +178,15 @@ describe("retrieval trust gate on the search path", () => {expect(trace.exclusions).toHaveLength(1);expect(trace.exclusions[0]!.path).toBe("quarantined.md");expect(trace.exclusions[0]!.reasons).toEqual(["trust_gate:self_approval_quarantine"]);-- const assessment = out.memoryTrustAssessment!;- expect(assessment.quarantined).toBe(1);- expect(assessment.surfaced).toBe(1);- expect(assessment.reason_counts).toEqual({ "trust_gate:self_approval_quarantine": 1 });+ // Enforce reproduces today's shape exactly: no mode key, no refusal+ // section.+ expect("mode" in trace).toBe(false);+ expect("refused" in trace).toBe(false);+ expect("refusals" in trace).toBe(false);+ expect("mode" in out.memoryTrustAssessment!).toBe(false);});- test("untrusted-source and contamination markers are also excluded", async () => {+ test("untrusted-source and contamination markers are refused under shadow with all reasons", async () => {writeMd(vault, "clean.md", CLEAN);writeMd(vault,@@ -103,34 +198,237 @@ describe("retrieval trust gate on the search path", () => {"tainted.md",`---\n${ENTITY_CONTAMINATION_FRONTMATTER_KEY}: true\n---\n\n# T\n\nwidget calibration tainted claim.`,);- const cfg = makeConfig({ vault, dbPath, retrievalTrustGateEnabled: true });+ const cfg = withMode(makeConfig({ vault, dbPath }), "shadow");await indexVault(cfg);const out = await search(cfg, { query: "widget calibration", limit: 5 });- expect(out.results.map((r) => r.path)).toEqual(["clean.md"]);+ // Nothing excluded, both would-be refusals measured per lane.+ expect(out.results.map((r) => r.path).toSorted()).toEqual([+ "clean.md",+ "tainted.md",+ "untrusted.md",+ ]);const assessment = out.memoryTrustAssessment!;- expect(assessment.quarantined).toBe(2);- expect(assessment.reason_counts).toEqual({- "trust_gate:entity_contamination": 1,- "trust_gate:untrusted_source_provenance": 1,+ expect(assessment.quarantined).toBe(0);+ expect(assessment.refused).toBe(2);+ expect(assessment.refused_reason_counts).toEqual({+ pool: {+ "trust_gate:entity_contamination": 1,+ "trust_gate:untrusted_source_provenance": 1,+ },+ arm: {},});+ const refusals = out.retrievalDecisionTrace!.refusals!;+ expect(refusals.map((r) => r.path).toSorted()).toEqual(["tainted.md", "untrusted.md"]);});- test("a clean vault ranks byte-identically with the gate on or off", async () => {+ test("a clean vault under shadow measures nothing and ranks byte-identically", async () => {writeMd(vault, "a.md", "# A\n\nnote about orchard pruning in spring");writeMd(vault, "b.md", "# B\n\nnote about orchard irrigation schedules");const off = makeConfig({ vault, dbPath });- const on = makeConfig({ vault, dbPath, retrievalTrustGateEnabled: true });+ const shadow = withMode(off, "shadow");await indexVault(off);const gateOff = await search(off, { query: "orchard", limit: 5 });- const gateOn = await search(on, { query: "orchard", limit: 5 });+ const gateShadow = await search(shadow, { query: "orchard", limit: 5 });- expect(gateOn.results.map(stableProjection)).toEqual(gateOff.results.map(stableProjection));- // Even with the gate engaged, a clean pack still carries the receipts,- // reporting zero exclusions.- const assessment = gateOn.memoryTrustAssessment!;- expect(assessment.quarantined).toBe(0);+ expect(gateShadow.results.map(stableProjection)).toEqual(gateOff.results.map(stableProjection));+ const assessment = gateShadow.memoryTrustAssessment!;+ expect(assessment.mode).toBe("shadow");+ expect(assessment.refused).toBe(0);+ expect(assessment.refusals_by_lane).toEqual({ arm: 0, pool: 0 });+ expect(assessment.refused_reason_counts).toEqual({ arm: {}, pool: {} });+ });++ test("the arm lane's refusals ride the receipts end to end under shadow", async () => {+ // The arm's only route to `neighbor.md` is the walk, and its refusal+ // reason is one the page gate never reads - a closed validity window -+ // so the measured arm lane cannot come from the pool's own pass.+ writeMd(vault, "seed.md", '---\nrelated: "[[neighbor]]"\n---\n\nalpha topic seed document.');+ writeMd(vault, "neighbor.md", '---\nvalid_until: "2020-01-01"\n---\n\nbeta divergent content.');+ await indexVault(makeConfig({ vault, dbPath }));+ const configPath = join(vault, "gate-mode.yaml");+ writeFileSync(configPath, "search_trust_gate_mode: shadow\n");+ const cfg = {+ ...makeConfig({+ vault,+ dbPath,+ fusionMode: "rrf",+ relationalArmEnabled: true,+ searchTrustGateMode: "shadow",+ }),+ configPath,+ };++ const out = await search(cfg, { query: "alpha [[seed]] related" });++ // Shadow keeps the arm's contribution: the walked node still surfaces,+ // carrying the arm's attribution.+ const neighbor = out.results.find((r) => r.path === "neighbor.md");+ expect(neighbor).toBeDefined();+ expect(neighbor!.reasons.some((x) => x.startsWith("relational:"))).toBe(true);++ const trace = out.retrievalDecisionTrace!;+ expect(trace.mode).toBe("shadow");+ expect(trace.excluded).toBe(0);+ expect(trace.refused).toBe(1);+ expect(trace.refusals).toEqual([+ {+ document_id: expect.any(Number),+ chunk_id: expect.any(Number),+ path: "neighbor.md",+ reasons: ["relational_arm:closed_validity_window"],+ lane: "arm",+ },+ ]);++ const assessment = out.memoryTrustAssessment!;+ expect(assessment.refused).toBe(1);+ expect(assessment.refusals_by_lane).toEqual({ arm: 1, pool: 0 });+ expect(assessment.refused_reason_counts).toEqual({+ arm: { "relational_arm:closed_validity_window": 1 },+ pool: {},+ });+ });++ test("a withheld quarantined page is receipted identically to its absence under shadow", async () => {+ // The walked page is quarantined AND reserved against the remote+ // caller's reach: no lane may name it in the receipts, so the refusal+ // section reads exactly as it would had the page never existed. A+ // second, readable walk target keeps the pool non-empty on both+ // sides, so the receipts exist to be compared.+ writeMd(+ vault,+ "seed.md",+ '---\nrelated: "[[hidden]]"\nextends: "[[visible]]"\n---\n\nalpha topic seed document.',+ );+ writeMd(+ vault,+ "hidden.md",+ '---\nstatus: quarantine\nvisibility: ["private"]\n---\n\nbeta hidden divergent content.',+ );+ writeMd(vault, "visible.md", "# Visible\n\ngamma readable neighbor content.");+ await indexVault(makeConfig({ vault, dbPath }));+ const configPath = join(vault, "gate-mode.yaml");+ writeFileSync(configPath, "search_trust_gate_mode: shadow\n");+ const cfg = {+ ...makeConfig({+ vault,+ dbPath,+ fusionMode: "rrf",+ relationalArmEnabled: true,+ searchTrustGateMode: "shadow",+ }),+ configPath,+ };++ const present = await search(cfg, {+ query: "alpha [[seed]] related extends",+ transportReach: TRANSPORT_REACH.remote,+ });+ const presentTrace = present.retrievalDecisionTrace!;+ expect(presentTrace.mode).toBe("shadow");+ expect(presentTrace.refused).toBe(0);+ expect(presentTrace.refusals).toEqual([]);++ // The same vault, the page gone: the refusal section is the same+ // bytes (the counts around it legitimately differ - the row vanished).+ rmSync(join(vault, "hidden.md"));+ await indexVault(cfg);+ const absent = await search(cfg, {+ query: "alpha [[seed]] related extends",+ transportReach: TRANSPORT_REACH.remote,+ });+ const absentTrace = absent.retrievalDecisionTrace!;+ expect(JSON.stringify({ refused: presentTrace.refused, refusals: presentTrace.refusals })).toBe(+ JSON.stringify({ refused: absentTrace.refused, refusals: absentTrace.refusals }),+ );+ });++ test("enforce records the arm lane's drops in the receipts (never silently dropped)", async () => {+ // The arm's only route to `dirty.md` is the walk: under enforce the+ // node is removed BEFORE the pool, so no pool exclusion can ever name+ // it. The gate's never-silently-dropped posture holds only if the+ // drop is receipted - on the arm lane, without a shadow mode stamp. A+ // clean second walk target keeps the pool non-empty, so the receipts+ // exist to be read.+ writeMd(+ vault,+ "seed.md",+ '---\nrelated: "[[dirty]]"\nextends: "[[clean2]]"\n---\n\nalpha topic seed document.',+ );+ writeMd(+ vault,+ "dirty.md",+ `---\n${ENTITY_CONTAMINATION_FRONTMATTER_KEY}: true\n---\n\nbeta divergent content.`,+ );+ writeMd(vault, "clean2.md", "# Clean two\n\ngamma readable neighbor content.");+ await indexVault(makeConfig({ vault, dbPath }));+ const configPath = join(vault, "gate-mode.yaml");+ writeFileSync(configPath, "search_trust_gate_mode: enforce\n");+ const cfg = {+ ...makeConfig({+ vault,+ dbPath,+ fusionMode: "rrf",+ relationalArmEnabled: true,+ searchTrustGateMode: "enforce",+ }),+ configPath,+ };++ const out = await search(cfg, { query: "alpha [[seed]] related extends" });++ // The excluding behavior is unchanged: the walked node never surfaces.+ expect(out.results.some((r) => r.path === "dirty.md")).toBe(false);++ const trace = out.retrievalDecisionTrace!;+ expect(trace.refused).toBe(1);+ expect(trace.refusals).toEqual([+ {+ document_id: expect.any(Number),+ chunk_id: expect.any(Number),+ path: "dirty.md",+ reasons: ["relational_arm:entity_contamination"],+ lane: "arm",+ },+ ]);+ // Still not a shadow receipt: no mode stamp on an enforce run.+ expect("mode" in trace).toBe(false);++ const assessment = out.memoryTrustAssessment!;+ expect(assessment.refused).toBe(1);+ expect(assessment.refusals_by_lane).toEqual({ arm: 1, pool: 0 });+ expect(assessment.refused_reason_counts).toEqual({+ arm: { "relational_arm:entity_contamination": 1 },+ pool: {},+ });+ expect("mode" in assessment).toBe(false);+ });++ test("the explain envelope names the resolved mode under shadow and stays silent under enforce", async () => {+ writeMd(vault, "clean.md", CLEAN);+ writeMd(vault, "quarantined.md", QUARANTINED);+ const off = makeConfig({ vault, dbPath });+ await indexVault(off);+ const shadowOutcome = await search(withMode(off, "shadow"), {+ query: "widget calibration",+ limit: 5,+ });+ const enforceOutcome = await search(withMode(off, "enforce"), {+ query: "widget calibration",+ limit: 5,+ });++ const request = { ...EXPLAIN_OFF, explain: true };+ const shadowEnvelope = explainEnvelope(shadowOutcome, request);+ expect(shadowEnvelope[RETRIEVAL_DECISION_TRACE_KEY]).toBeDefined();+ expect(shadowEnvelope[RETRIEVAL_TRUST_GATE_MODE_KEY]).toBe("shadow");++ // Enforce keeps today's envelope exactly: the receipts, no mode key.+ const enforceEnvelope = explainEnvelope(enforceOutcome, request);+ expect(enforceEnvelope[RETRIEVAL_DECISION_TRACE_KEY]).toBeDefined();+ expect(enforceEnvelope[RETRIEVAL_TRUST_GATE_MODE_KEY]).toBeUndefined();});});diff --git a/tests/core/state/surfaces.test.ts b/tests/core/state/surfaces.test.tsindex ad2b5f68..3ed1790b 100644--- a/tests/core/state/surfaces.test.ts+++ b/tests/core/state/surfaces.test.ts@@ -63,6 +63,7 @@ import { checkpointPath } from "../../../src/core/brain/ingest/checkpoint.ts";import { sessionCheckpointPath } from "../../../src/core/brain/sessions/checkpoint.ts";import { manifestPath as ingestManifestPath } from "../../../src/core/brain/ingest/content-manifest.ts";import { sessionLedgerPath } from "../../../src/core/brain/sessions/discover.ts";+import { sessionMiningLedgerPath } from "../../../src/core/brain/session-mining-ledger.ts";import { secretsDir } from "../../../src/core/brain/secrets/store.ts";import { manifestPath as installManifestPath } from "../../../src/core/install/manifest.ts";import { bootstrapReceiptPath } from "../../../src/cli/bootstrap/receipt.ts";@@ -266,6 +267,7 @@ const RESOLVER_BINDINGS: ReadonlyArray<readonly [StateSurfaceId, (vault: string)["session_import_checkpoints", (v) => dirname(sessionCheckpointPath(v, "0f1e2d3c"))],["write_dead_letters", (v) => deadLetterDir(v)],["session_import_ledger", (v) => sessionLedgerPath(v)],+ ["session_mining_ledger", (v) => sessionMiningLedgerPath(v)],["install_manifest", (v) => installManifestPath(v)],["bootstrap_receipt", (v) => bootstrapReceiptPath(v)],["hook_audit", (v) => hookAuditDir(v)],diff --git a/tests/fixtures/install/grok-hooks.golden.json b/tests/fixtures/install/grok-hooks.golden.jsonindex 5a87bd85..2ccf56c5 100644--- a/tests/fixtures/install/grok-hooks.golden.json+++ b/tests/fixtures/install/grok-hooks.golden.json@@ -19,6 +19,14 @@},"timeout": 10},+ {+ "type": "command",+ "command": "<BUN> run <REPO>/hooks/backlog-nudge.ts",+ "env": {+ "VAULT_AGENT_NAME": "grok-dev-agent"+ },+ "timeout": 10+ },{"type": "command","command": "<BUN> run <REPO>/hooks/session-capture.ts",diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.tsindex 30408274..a3e5dd17 100644--- a/tests/helpers/search-fixtures.ts+++ b/tests/helpers/search-fixtures.ts@@ -80,6 +80,14 @@ export function makeConfig(opts: {relationalArmEnabled?: boolean;/** Retrieval trust gate (kernel 1); defaults to false (opt-in). */retrievalTrustGateEnabled?: boolean;+ /**+ * The gate's resolved mode (session-mining-economy). Default: the same+ * mapping the resolver applies - the legacy boolean true alone resolves+ * to shadow, false and absence stay off (the field stays absent, like+ * every opt-in recall field at its off value). An explicit "off" omits+ * the field too, matching the spread-omitted resolved shape.+ */+ searchTrustGateMode?: "shadow" | "enforce" | "off";/** Relation-only supersede fade (kernel 1); defaults to false (opt-in). */supersedeFadeEnabled?: boolean;/** Learned recall weights from feedback; defaults to false (opt-in). */@@ -119,6 +127,18 @@ export function makeConfig(opts: {...opts.semantic,});const paths = opts.ignorePaths ?? DEFAULT_VAULT_IGNORE_PATHS;+ // The gate mode mirrors resolveSearchTrustGateMode's mapping: explicit+ // mode wins (off stays absent), else the legacy boolean maps true to+ // shadow - so the fixture can never build a resolved shape the+ // resolver would not produce.+ const gateMode: "shadow" | "enforce" | undefined =+ opts.searchTrustGateMode !== undefined+ ? opts.searchTrustGateMode === "off"+ ? undefined+ : opts.searchTrustGateMode+ : opts.retrievalTrustGateEnabled === true+ ? "shadow"+ : undefined;const scopeRules: VaultScopeRules = Object.freeze({ignore: Object.freeze(paths.map(classifyVaultPathRule)),include:@@ -154,6 +174,7 @@ export function makeConfig(opts: {relationPolarityEnabled: opts.relationPolarityEnabled ?? true,relationalArmEnabled: opts.relationalArmEnabled ?? false,retrievalTrustGateEnabled: opts.retrievalTrustGateEnabled ?? false,+ ...(gateMode !== undefined ? { searchTrustGateMode: gateMode } : {}),supersedeFadeEnabled: opts.supersedeFadeEnabled ?? false,learnedWeightsEnabled: opts.learnedWeightsEnabled ?? false,activationEnabled: opts.activationEnabled ?? true,diff --git a/tests/hooks/backlog-nudge.test.ts b/tests/hooks/backlog-nudge.test.tsnew file mode 100644index 00000000..6feb9772--- /dev/null+++ b/tests/hooks/backlog-nudge.test.ts@@ -0,0 +1,223 @@+/**+ * The SessionStart backlog nudge hook (session-mining-economy,+ * t_0358af42).+ *+ * The hook answers one question at session start - how much of the+ * imported session backlog has never been distilled - and only when the+ * answer has CHANGED since the last time it was injected. The census is+ * the vault-side mining census, so no source log is ever hashed inside+ * the hook's budget; under the deadline the hook degrades to silence,+ * never to a fabricated number.+ *+ * Flag off is an immediate no-op with zero output: the session preamble+ * stays byte-identical. Every path exits 0.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join, resolve } from "node:path";+import { fileURLToPath } from "node:url";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { importSession } from "../../src/core/brain/sessions/import.ts";+import { recordSessionImports, sessionLedgerPath } from "../../src/core/brain/sessions/discover.ts";+import { sessionMiningLedgerPath } from "../../src/core/brain/session-mining-ledger.ts";+import { DERIVED_STORE_DIR } from "../../src/core/brain/path-constants.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { SESSION_RUNTIME_ID } from "../../src/core/runtime/host-facts.ts";+import { homeEnv } from "../helpers/platform.ts";++const HOOKS_DIR = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..", "hooks");+const NOW = new Date("2026-09-18T10:00:00Z");++let vault: string;+let configHome: string;+let sessionsDir: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-nudge-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-nudge-cfg-"));+ sessionsDir = mkdtempSync(join(tmpdir(), "o2b-nudge-sessions-"));+ bootstrapBrain(vault);+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+ rmSync(sessionsDir, { recursive: true, force: true });+});++interface RunResult {+ readonly stdout: string;+ readonly stderr: string;+ readonly exit: number;+}++async function runHook(payload: unknown, env: Record<string, string> = {}): Promise<RunResult> {+ const inherited: Record<string, string> = {+ PATH: process.env["PATH"] ?? "",+ ...homeEnv(configHome),+ };+ const proc = Bun.spawn(["bun", "run", join(HOOKS_DIR, "backlog-nudge.ts")], {+ stdin: "pipe",+ stdout: "pipe",+ stderr: "pipe",+ env: { ...inherited, ...env },+ });+ proc.stdin.write(JSON.stringify(payload));+ await proc.stdin.end();+ const stdout = await new Response(proc.stdout).text();+ const stderr = await new Response(proc.stderr).text();+ const exit = await proc.exited;+ return { stdout, stderr, exit };+}++/** The flag-on environment, built per call: `vault` moves with every test. */+function flagOn(): Record<string, string> {+ return { OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED: "true", VAULT_DIR: vault };+}++const SESSION_START = { hook_event_name: "SessionStart" };++/** The state file: one overwrite-only count beside the hygiene digest. */+function statePath(): string {+ return join(vault, DERIVED_STORE_DIR, "backlog-nudge.count");+}++/** Session filenames stay unique across calls: the basename IS the identity. */+let seeded = 0;++/** Seed `n` imported, never-mined sessions, the way the import verb does. */+async function seedBacklog(n: number): Promise<void> {+ for (let i = 0; i < n; i++) {+ const path = join(sessionsDir, `backlog-${seeded}.jsonl`);+ seeded += 1;+ writeFileSync(+ path,+ JSON.stringify({+ type: "user",+ parentUuid: null,+ entrypoint: "cli",+ sessionId: `s-${seeded}`,+ uuid: `u-${seeded}`,+ timestamp: "2026-09-18T09:00:00.000Z",+ message: { role: "user", content: [{ type: "text", text: `backlog item ${seeded}` }] },+ }) + "\n",+ "utf8",+ );+ // Sequential on purpose: each importSession builds and consults its+ // own dedup index, and two concurrent imports would race the vault.+ // oxlint-disable-next-line no-await-in-loop+ await importSession(vault, path, { agent: "tester", now: NOW, recall: true });+ recordSessionImports(vault, [{ path, runtime: SESSION_RUNTIME_ID.claudeCode }]);+ }+}++interface InjectedPayload {+ readonly hookSpecificOutput?: {+ readonly hookEventName?: string;+ readonly additionalContext?: string;+ };+}++describe("backlog-nudge hook (SessionStart)", () => {+ test("flag off is an immediate no-op with zero output and no state", async () => {+ await seedBacklog(2);+ const r = await runHook(SESSION_START, { VAULT_DIR: vault });+ expect(r.exit).toBe(0);+ expect(r.stdout).toBe("");+ expect(existsSync(statePath())).toBe(false);+ });++ test("flag on with a changed count injects one line naming the count", async () => {+ await seedBacklog(1);+ const r = await runHook(SESSION_START, flagOn());+ expect(r.exit).toBe(0);+ const payload = JSON.parse(r.stdout) as InjectedPayload;+ expect(payload.hookSpecificOutput?.hookEventName).toBe("SessionStart");+ expect(payload.hookSpecificOutput?.additionalContext).toMatch(/1 imported session\b/);+ expect(payload.hookSpecificOutput?.additionalContext).not.toMatch(/2 imported sessions/);+ // The state lands only after the emit, overwrite-only, size one.+ expect(readFileSync(statePath(), "utf8")).toBe("1\n");+ });++ test("an unchanged count stays silent", async () => {+ await seedBacklog(1);+ expect((await runHook(SESSION_START, flagOn())).stdout).not.toBe("");+ const again = await runHook(SESSION_START, flagOn());+ expect(again.exit).toBe(0);+ expect(again.stdout).toBe("");+ });++ test("a grown backlog re-arms the nudge", async () => {+ await seedBacklog(1);+ await runHook(SESSION_START, flagOn());+ await seedBacklog(1);+ const r = await runHook(SESSION_START, flagOn());+ const payload = JSON.parse(r.stdout) as InjectedPayload;+ expect(payload.hookSpecificOutput?.additionalContext).toMatch(/2 imported sessions/);+ expect(readFileSync(statePath(), "utf8")).toBe("2\n");+ });++ test("a drained backlog is recorded, not announced, and re-arms when it returns", async () => {+ await seedBacklog(1);+ expect((await runHook(SESSION_START, flagOn())).stdout).not.toBe("");++ // The backlog empties (the sessions were mined, or the import records+ // went away). A line announcing zero work is noise; the state still+ // moves, so the NEXT change is measured against zero.+ rmSync(sessionLedgerPath(vault));+ const drained = await runHook(SESSION_START, flagOn());+ expect(drained.exit).toBe(0);+ expect(drained.stdout).toBe("");+ expect(readFileSync(statePath(), "utf8")).toBe("0\n");++ await seedBacklog(1);+ const returned = await runHook(SESSION_START, flagOn());+ const payload = JSON.parse(returned.stdout) as InjectedPayload;+ expect(payload.hookSpecificOutput?.additionalContext).toMatch(/1 imported session\b/);+ }, 20000);++ test("a count of zero with no recorded state is silent too", async () => {+ const r = await runHook(SESSION_START, flagOn());+ expect(r.exit).toBe(0);+ expect(r.stdout).toBe("");+ expect(readFileSync(statePath(), "utf8")).toBe("0\n");+ });++ test("an event that cannot carry additionalContext gets nothing", async () => {+ await seedBacklog(1);+ const r = await runHook({ hook_event_name: "PostToolUse" }, flagOn());+ expect(r.exit).toBe(0);+ expect(r.stdout).toBe("");+ expect(existsSync(statePath())).toBe(false);+ });++ test("no vault resolves to silence", async () => {+ const r = await runHook(SESSION_START, { OPEN_SECOND_BRAIN_BACKLOG_NUDGE_ENABLED: "true" });+ expect(r.exit).toBe(0);+ expect(r.stdout).toBe("");+ });++ test("a corrupt census source degrades to silence, naming itself on stderr", async () => {+ await seedBacklog(1);+ mkdirSync(dirname(sessionMiningLedgerPath(vault)), { recursive: true });+ writeFileSync(sessionMiningLedgerPath(vault), "{ this is not json", "utf8");+ const r = await runHook(SESSION_START, flagOn());+ expect(r.exit).toBe(0);+ expect(r.stdout).toBe("");+ expect(r.stderr).toContain("backlog-nudge");+ expect(r.stderr).toContain("session mining ledger is corrupted JSON");+ });++ test("a corrupt state file reads as changed, so the next eligible session emits", async () => {+ await seedBacklog(1);+ mkdirSync(dirname(statePath()), { recursive: true });+ atomicWriteFileSync(statePath(), "not a count\n");+ const r = await runHook(SESSION_START, flagOn());+ expect(r.exit).toBe(0);+ expect(JSON.parse(r.stdout as string)).toBeDefined();+ expect(readFileSync(statePath(), "utf8")).toBe("1\n");+ });+});diff --git a/tests/hooks/hooks-json-shape.test.ts b/tests/hooks/hooks-json-shape.test.tsindex 9521adf5..bbb5b13a 100644--- a/tests/hooks/hooks-json-shape.test.ts+++ b/tests/hooks/hooks-json-shape.test.ts@@ -231,3 +231,37 @@ describe("hooks.json subagent-inject entry", () => {expect(carriers[0]!.type).toBe("command");});});++describe("hooks.json SessionStart backlog-nudge entry", () => {+ const sessionStartGroups =+ (+ JSON.parse(readFileSync(HOOKS_JSON, "utf8")) as {+ hooks: Record<+ string,+ Array<{ matcher?: string; hooks: Array<HookEntry & { timeout?: number }> }>+ >;+ }+ ).hooks["SessionStart"] ?? [];++ test("the single SessionStart group carries exactly one backlog-nudge entry, after gap-agenda", () => {+ expect(sessionStartGroups.length).toBe(1);+ const hooks = sessionStartGroups[0]!.hooks;+ const agenda = hooks.findIndex((h) => h.command.includes("o2b-hook gap-agenda"));+ expect(agenda).toBeGreaterThanOrEqual(0);+ const nudgeIndices = hooks+ .map((h, i) => (h.command.includes("o2b-hook backlog-nudge") ? i : -1))+ .filter((i) => i >= 0);+ expect(nudgeIndices).toEqual([agenda + 1]);+ });++ test("the backlog-nudge entry declares the sibling timeout and the opt-in status line", () => {+ const hooks = sessionStartGroups[0]!.hooks;+ const entry = hooks.find((h) => h.command.includes("o2b-hook backlog-nudge"))!;+ expect(entry.type).toBe("command");+ expect(entry.timeout).toBe(10);+ expect(entry.statusMessage).toBe("OSB: injecting undistilled backlog count (opt-in)");+ // The dispatch target resolves to an existing hook file, so the+ // registered name can never silently no-op.+ expect(existsSync(join(REPO, "hooks", "backlog-nudge.ts"))).toBe(true);+ });+});diff --git a/tests/mcp/brain-search-explain-trace.test.ts b/tests/mcp/brain-search-explain-trace.test.tsindex 0d853f30..aaed70d6 100644--- a/tests/mcp/brain-search-explain-trace.test.ts+++ b/tests/mcp/brain-search-explain-trace.test.ts@@ -87,7 +87,12 @@ beforeEach(async () => {writeAged("notes/quarantined.md", QUARANTINED);const onPath = join(configHome, "gate-on.yaml");- atomicWriteFileSync(onPath, `vault: ${vault}\nsearch_trust_gate_enabled: true\n`);+ // The gate must EXCLUDE here, and the legacy boolean alone now+ // measures (shadow): enforcement is named explicitly.+ atomicWriteFileSync(+ onPath,+ `vault: ${vault}\nsearch_trust_gate_enabled: true\nsearch_trust_gate_mode: enforce\n`,+ );const offPath = join(configHome, "gate-off.yaml");atomicWriteFileSync(offPath, `vault: ${vault}\n`);gateOn = { vault, configPath: onPath };diff --git a/tests/mcp/delete-by-source-failed.test.ts b/tests/mcp/delete-by-source-failed.test.tsnew file mode 100644index 00000000..a4d5fbe7--- /dev/null+++ b/tests/mcp/delete-by-source-failed.test.ts@@ -0,0 +1,95 @@+/**+ * `brain_delete_by_source` carries the core `failed` list on the wire+ * (storage lifecycle, PR #257 review M8). The MCP serializer emitted+ * `deleted` and `changed` but never `failed`, so a confirmed removal+ * that could not delete a file answered as a clean run with a smaller+ * count. A blocked removal now names the path and the fs error in the+ * tool answer, the same list the CLI verb prints and exits 1 on.+ */++import { afterEach, describe, expect, test } from "bun:test";+import { chmodSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { ingestSource } from "../../src/core/brain/ingest/ingest.ts";+import { MCPServer } from "../../src/mcp/server.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { CHMOD_CANNOT_DENY } from "../helpers/platform.ts";++const SOURCE = "imports/benchmark.md";+const NOW = new Date("2026-06-01T00:00:00Z");++const bases: string[] = [];++afterEach(() => {+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });+});++interface Fixture {+ readonly vault: string;+ readonly server: MCPServer;+}++function fixture(): Fixture {+ const base = mkdtempSync(join(tmpdir(), "o2b-dbs-failed-"));+ bases.push(base);+ const vault = join(base, "vault");+ mkdirSync(join(vault, "imports"), { recursive: true });+ const configPath = join(base, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);+ bootstrapBrain(vault, { configPath });+ writeFileSync(join(vault, SOURCE), "benchmark rows\n", "utf8");+ ingestSource(+ vault,+ {+ sourcePath: SOURCE,+ summary: "Benchmark import summary.",+ extraction: { entities: [{ category: "concept", name: "BenchWidget" }] },+ },+ { agent: "tester", now: NOW },+ );+ return { vault, server: new MCPServer({ vault, configPath }) };+}++/** The tool's structured answer, unwrapped from the MCP envelope. */+async function call(f: Fixture, args: Record<string, unknown>): Promise<Record<string, unknown>> {+ const result = await f.server.callTool("brain_delete_by_source", args);+ return result["structuredContent"] as Record<string, unknown>;+}++describe("brain_delete_by_source names what did not delete", () => {+ test("a confirmed run with a blocked removal carries failed on the wire", async () => {+ if (CHMOD_CANNOT_DENY) return;+ const f = fixture();+ const preview = (await call(f, { source_file: SOURCE })) as {+ derived: Array<{ path: string; kind: string }>;+ };+ const blockedPath = preview.derived.find((e) => e.kind === "summary")!.path;+ const blockedDir = join(f.vault, dirname(blockedPath));+ chmodSync(blockedDir, 0o555);+ try {+ const answer = (await call(f, { source_file: SOURCE, confirm: true })) as {+ changed: number;+ failed: ReadonlyArray<{ path: string; message: string }>;+ };+ expect(answer.failed.length).toBe(1);+ expect(answer.failed[0]!.path).toBe(blockedPath);+ expect(answer.failed[0]!.message.length).toBeGreaterThan(0);+ // The rest of the run still landed, and the answer says both sides.+ expect(answer.changed).toBeGreaterThan(0);+ expect(existsSync(join(f.vault, blockedPath))).toBe(true);+ } finally {+ chmodSync(blockedDir, 0o755);+ }+ });++ test("a clean confirmed run carries no failure", async () => {+ const f = fixture();+ const answer = (await call(f, { source_file: SOURCE, confirm: true })) as {+ failed?: unknown;+ };+ expect(answer.failed).toEqual([]);+ });+});diff --git a/tests/mcp/doctor-collect-errors.test.ts b/tests/mcp/doctor-collect-errors.test.tsnew file mode 100644index 00000000..d5a14a8c--- /dev/null+++ b/tests/mcp/doctor-collect-errors.test.ts@@ -0,0 +1,83 @@+/*+ * `brain_doctor` renders the maintenance collector's absorbed scan+ * failures as a named `collect_errors` row (review M10). The collector+ * is fail-soft either way; before the observers were passed, one+ * throwing scan left the doctor, digest and health reports thin and+ * looking healthy - exactly the difference the row exists to name.+ */++import { afterEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { JSONRPC_VERSION, MCPServer } from "../../src/mcp/index.ts";++const bases: string[] = [];++afterEach(() => {+ for (const base of bases.splice(0)) rmSync(base, { recursive: true, force: true });+});++interface RpcResponse {+ readonly error?: { code: number; message: string; data?: Record<string, unknown> };+ readonly result?: {+ structuredContent?: Record<string, unknown>;+ content?: ReadonlyArray<{ text?: string }>;+ };+}++function fixture(breakPreferences: boolean): MCPServer {+ const base = mkdtempSync(join(tmpdir(), "o2b-doctor-collect-"));+ bases.push(base);+ const vault = join(base, "vault");+ mkdirSync(join(vault, "notes"), { recursive: true });+ writeFileSync(join(vault, "notes", "source.md"), "# Widget source\n\nBody text.\n");+ const configPath = join(base, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);+ bootstrapBrain(vault, { configPath });+ if (breakPreferences) {+ // A FILE at Brain/preferences makes the dedup scan's readdir refuse;+ // every doctor check degrades instead of throwing.+ rmSync(join(vault, "Brain", "preferences"), { recursive: true });+ writeFileSync(join(vault, "Brain", "preferences"), "not a directory", "utf8");+ }+ return new MCPServer({ vault, configPath }, {});+}++async function callDoctor(server: MCPServer): Promise<RpcResponse> {+ return (await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 7,+ method: "tools/call",+ params: { name: "brain_doctor", arguments: {} },+ })) as RpcResponse;+}++test("a refused scan is rendered as a named collect_errors row", async () => {+ const res = await callDoctor(fixture(true));+ expect(res.error).toBeUndefined();+ const s = res.result?.structuredContent ?? {};+ // The break can refuse more than one scan on this surface; the row is+ // about WHAT refused, so the assertion is on the shape and the named+ // dedup entry.+ const errors = (s["collect_errors"] ?? []) as ReadonlyArray<{+ scan: string;+ message: string;+ }>;+ expect(errors.length).toBeGreaterThan(0);+ expect(errors.some((e) => e.scan === "dedup")).toBe(true);+ for (const e of errors) {+ expect(e.message.replace(/\\/g, "/")).toContain("preferences");+ }+});++test("a healthy vault's payload carries no collect_errors row", async () => {+ const res = await callDoctor(fixture(false));+ expect(res.error).toBeUndefined();+ const s = res.result?.structuredContent ?? {};+ expect(s["collect_errors"]).toBeUndefined();+ expect(Array.isArray(s["suggested_actions"])).toBe(true);+});diff --git a/tests/mcp/extract-signals-tool.test.ts b/tests/mcp/extract-signals-tool.test.tsindex 45e031e8..c31f196c 100644--- a/tests/mcp/extract-signals-tool.test.ts+++ b/tests/mcp/extract-signals-tool.test.ts@@ -14,14 +14,19 @@* 6. Every `turns_mined` entry carries the turn's stored timestamp* verbatim, and the CLI `--json` plan over the same fixture carries* the identical projection.+ * 7. A session tombstoned by a barren commit gets no envelope: the plan+ * carries the tombstone reason and the recorded breakdown, `force`+ * bypasses it for one call, and a ledger failure rides the commit+ * response by name.*/import { afterEach, beforeEach, expect, test } from "bun:test";import { existsSync, mkdirSync, mkdtempSync, readdirSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";-import { join } from "node:path";+import { dirname, join } from "node:path";import { AUTO_EXTRACT_PER_SESSION_CAP } from "../../src/core/brain/extract-signals.ts";+import { sessionMiningLedgerPath } from "../../src/core/brain/session-mining-ledger.ts";import { importSessionRecall } from "../../src/core/brain/session-recall.ts";import { buildToolTable, findTool } from "../../src/mcp/tools.ts";import type { ServerContext } from "../../src/mcp/tool-contract.ts";@@ -142,3 +147,118 @@ test("turns_mined carries the stored turn timestamp, on the tool and the CLI aliconst payload = JSON.parse(cli.stdout) as { turns_mined: unknown };expect(payload.turns_mined).toEqual(expected);});++// ----- The zero-yield mining tombstone on the tool (session-mining-economy)+//+// 7. A session tombstoned by a barren commit gets no envelope: the plan+// carries the tombstone reason and the recorded breakdown, and no turn+// text rides out to the caller. The tombstone exists only when the+// commit relayed the plan's surveyed_turns_hash back.+// 8. The `force` argument bypasses the tombstone for exactly one plan+// call, and the tool declares it; so does the surveyed_turns_hash+// argument the commit relays.+// 9. A ledger write failure on a successful commit rides the commit+// response by name; the tool description documents both behaviors.++function corruptMiningLedger(): void {+ mkdirSync(dirname(sessionMiningLedgerPath(vault)), { recursive: true });+ writeFileSync(sessionMiningLedgerPath(vault), "{ truncated");+}++/** The plan response's surveyed hash, for the commit that answers it. */+async function surveyedHash(): Promise<string> {+ const plan = (await tool().handler(ctx, { session: SESSION })) as {+ surveyed_turns_hash: string;+ };+ return plan.surveyed_turns_hash;+}++test("a tombstoned session returns the recorded breakdown instead of an envelope", async () => {+ const hash = await surveyedHash();+ const mine = (await tool().handler(ctx, {+ session: SESSION,+ items: [],+ surveyed_turns_hash: hash,+ })) as {+ phase: string;+ written: unknown[];+ };+ expect(mine.phase).toBe("commit");+ expect(mine.written).toEqual([]);+ const plan = (await tool().handler(ctx, { session: SESSION })) as {+ llm_step: unknown;+ turns_mined: unknown[];+ skipped?: {+ reason: string;+ mining_outcome?: Record<string, number>;+ };+ };+ expect(plan.skipped?.reason).toBe("mining_tombstone");+ expect(plan.skipped?.mining_outcome).toEqual({+ written: 0,+ staged: 0,+ deduped: 0,+ durability_rejected: 0,+ });+ expect(plan.llm_step).toBeNull();+ expect(plan.turns_mined).toEqual([]);+});++test("a barren commit that never saw the plan records nothing and fails open", async () => {+ // Without the relayed hash the commit has no material the verdict was+ // earned over, so it records nothing rather than keying a tombstone on+ // whatever the session holds now.+ const mine = (await tool().handler(ctx, { session: SESSION, items: [] })) as {+ phase: string;+ written: unknown[];+ };+ expect(mine.phase).toBe("commit");+ expect(mine.written).toEqual([]);+ const plan = (await tool().handler(ctx, { session: SESSION })) as { skipped?: unknown };+ expect(plan.skipped).toBeUndefined();+});++test("force bypasses the tombstone for exactly one plan call and is declared", async () => {+ const hash = await surveyedHash();+ await tool().handler(ctx, { session: SESSION, items: [], surveyed_turns_hash: hash });+ const forced = (await tool().handler(ctx, { session: SESSION, force: true })) as {+ llm_step: { status: string } | null;+ skipped?: unknown;+ };+ expect(forced.skipped).toBeUndefined();+ expect(forced.llm_step?.status).toBe("needs-llm-step");+ const after = (await tool().handler(ctx, { session: SESSION })) as {+ skipped?: { reason: string };+ };+ expect(after.skipped?.reason).toBe("mining_tombstone");+ // The arguments are part of the published contract, not hidden escapes.+ const schema = tool().inputSchema as {+ properties: { force?: { type: string }; surveyed_turns_hash?: { type: string } };+ };+ expect(schema.properties.force?.type).toBe("boolean");+ expect(schema.properties.surveyed_turns_hash?.type).toBe("string");+});++test("the tool description documents the tombstone and the force escape", () => {+ const description = tool().description;+ expect(description).toContain("tombstone");+ expect(description).toContain("force");+ expect(description).toContain("surveyed_turns_hash");+});++test("a ledger write failure on a successful commit rides the commit response by name", async () => {+ const hash = await surveyedHash();+ corruptMiningLedger();+ const res = (await tool().handler(ctx, {+ session: SESSION,+ items: [],+ surveyed_turns_hash: hash,+ })) as {+ phase: string;+ written: unknown[];+ mining_ledger_failure?: string;+ };+ expect(res.phase).toBe("commit");+ expect(res.written).toEqual([]);+ expect(res.mining_ledger_failure).toContain("session mining ledger is corrupted JSON");+});diff --git a/tests/mcp/ingest-capacity-refusal.test.ts b/tests/mcp/ingest-capacity-refusal.test.tsindex 28860821..45133dd7 100644--- a/tests/mcp/ingest-capacity-refusal.test.ts+++ b/tests/mcp/ingest-capacity-refusal.test.ts@@ -74,6 +74,7 @@ interface Fixture {function fixture(capBytes: number | undefined,reach: TransportReach = TRANSPORT_REACH.local,+ configAppend = "",): Fixture {const base = mkdtempSync(join(tmpdir(), "o2b-ingest-cap-"));bases.push(base);@@ -86,6 +87,7 @@ function fixture(if (capBytes !== undefined) {appendFileSync(brainConfigPath(vault), `\nstorage:\n cap_bytes: ${capBytes}\n`);}+ if (configAppend.length > 0) appendFileSync(brainConfigPath(vault), configAppend);const server = new MCPServer({ vault, configPath }, { reach });return { vault, configPath, server };}@@ -192,6 +194,51 @@ describe("brain_ingest_source capacity admission", () => {});});+describe("the admission gate reads only the storage block (review M1)", () => {+ test("a typo in an unrelated block never breaks the ingest on a vault with no cap", async () => {+ // The review's probe: the maintenance block is invalid, and the+ // storage block the gate decides from is absent - no cap, the+ // documented default where nothing refuses. The whole-file validation+ // the old loader ran made this ingest an internal error naming a host+ // path; the block-scoped read answers the gate's actual question.+ const f = fixture(+ undefined,+ TRANSPORT_REACH.local,+ "\nmaintenance:\n host_pressure_percent: banana\n",+ );+ const res = await ingest(f);+ expect(res.error).toBeUndefined();+ expect(brainPageTexts(f.vault).some((text) => text.includes(SUMMARY))).toBe(true);+ });++ test("a broken storage block refuses by name, free of host paths", async () => {+ // The gate cannot know whether a cap is in force, so a broken policy+ // is never silently read as "no cap": the refusal names the field and+ // carries the registered code - and never the config's host path,+ // which a remote caller has no business learning.+ const f = fixture(undefined, TRANSPORT_REACH.local, "\nstorage:\n cap_bytes: banana\n");+ const res = await ingest(f);+ expect(readRpcErrorCode(res)).toBe("config_invalid");+ const message = res.error?.message ?? "";+ expect(message).toContain("brain_ingest_source");+ expect(message).toContain("storage.cap_bytes");+ expect(message).not.toContain(f.vault);+ });++ test("an unparseable config refuses by name, free of host paths", async () => {+ const f = fixture(+ undefined,+ TRANSPORT_REACH.local,+ "\nstorage:\n cap_bytes: 5\n warn_percent: 80\n",+ );+ const res = await ingest(f);+ expect(readRpcErrorCode(res)).toBe("config_invalid");+ const message = res.error?.message ?? "";+ expect(message).toContain("brain_ingest_source");+ expect(message).not.toContain(f.vault);+ });+});+describe("the admission refusal at the caller's reach", () => {test("a local refusal carries the measured reading beside the code", async () => {const f = fixture(CRITICAL_CAP_BYTES, TRANSPORT_REACH.local);diff --git a/tests/mcp/ingest-tool.test.ts b/tests/mcp/ingest-tool.test.tsindex 470f26b4..ac351aff 100644--- a/tests/mcp/ingest-tool.test.ts+++ b/tests/mcp/ingest-tool.test.ts@@ -5,7 +5,14 @@*/import { afterEach, beforeEach, describe, expect, test } from "bun:test";-import { existsSync, mkdtempSync, readFileSync, readdirSync, rmSync } from "node:fs";+import {+ appendFileSync,+ existsSync,+ mkdtempSync,+ readFileSync,+ readdirSync,+ rmSync,+} from "node:fs";import { join } from "node:path";import { tmpdir } from "node:os";@@ -304,6 +311,47 @@ describe("brain_ingest_source - capture_scope", () => {});});+/**+ * The appended-delta wire fields (review M14), following the capture_scope+ * pattern: `extraction_delta` is absent on every shape that is not a proven+ * append, and carries the named kind beside the snake_case byte range.+ */+describe("brain_ingest_source - extraction_delta", () => {+ interface DeltaResult {+ readonly extraction_delta?:+ | { readonly kind: string; readonly from_byte: number; readonly to_byte: number }+ | undefined;+ }+ const ingest = (sourcePath: string) =>+ handler(ctx, {+ source_path: sourcePath,+ summary: "Grew?",+ entities: [{ category: "concept", name: "Grew" }],+ }) as Promise<DeltaResult>;++ test("a first ingest answers no extraction_delta", async () => {+ seed("Log/growth.md", "one\n");+ const res = await ingest("Log/growth.md");+ expect(res.extraction_delta).toBeUndefined();+ });++ test("an appended source answers its boundary-safe range and kind", async () => {+ seed("Log/growth.md", "one\n");+ await ingest("Log/growth.md");+ appendFileSync(join(vault, "Log/growth.md"), "two\n", "utf8");+ const res = await ingest("Log/growth.md");+ expect(res.extraction_delta).toEqual({ kind: "appended", from_byte: 4, to_byte: 8 });+ });++ test("a rewritten source answers no extraction_delta", async () => {+ seed("Log/growth.md", "one\n");+ await ingest("Log/growth.md");+ writeFileSync(join(vault, "Log/growth.md"), "entirely different\n", "utf8");+ const res = await ingest("Log/growth.md");+ expect(res.extraction_delta).toBeUndefined();+ });+});+/*** The durable-archive result fields (answer trust, Task C4), following the* capture_scope pattern: `source_archive_hash` is always present, null whendiff --git a/tests/mcp/receipts-reach.test.ts b/tests/mcp/receipts-reach.test.tsindex f1179a7d..102ab8f1 100644--- a/tests/mcp/receipts-reach.test.ts+++ b/tests/mcp/receipts-reach.test.ts@@ -3,12 +3,23 @@* wave review).** Compact is the destructive half of the receipts surface: it rewrites- * named ledger shards in place and reports whole-ledger byte totals -- * the same figure class the storage report refuses to take below local- * reach (`brain_storage`). Below local reach the fold is therefore not- * planned at all: a named refusal, no shard plan, no byte totals, and+ * named ledger shards in place, under guarantees that are all host-local+ * - vault identity, the permissions-document consult, the recovery+ * point, the per-file locks. Below local reach the fold is therefore not+ * planned at all: a named refusal, no shard plan, no byte figures, and* no write lane entered. Scan stays readable at any reach - a read-only- * inventory withholds nothing a caller could not ask for another way.+ * inventory that withholds nothing a caller could not ask for another+ * way.+ *+ * Reach rationale, decided on review (L11): the refusal names the+ * host-local write lane, NOT the byte figures. It used to say it stopped+ * because compact "reports whole-ledger byte totals", while scan - and+ * the receipt-ledger-growth doctor check that reads the same scan, on+ * the host where reach has no meaning - return the same per-shard+ * figures at any reach. Figures were never the differentiator; the+ * destructive rewrite is. Dropping the figures below local reach was the+ * alternative and is worse: the scan's contract is to withhold nothing,+ * and the doctor has no reach to drop them at.** Claims pinned here:*@@ -30,6 +41,7 @@ import { bootstrapBrain } from "../../src/core/brain/init.ts";import { RECEIPT_COMPACTION_OPERATION } from "../../src/core/brain/receipts/types.ts";import { resetVaultIdentityPins } from "../../src/core/brain/vault-identity.ts";import { TRANSPORT_REACH } from "../../src/core/graph/transport-reach.ts";+import { INVALID_PARAMS, MCPError } from "../../src/mcp/protocol.ts";import { RECEIPT_COMPACT_REACH_REASON, RECEIPT_TOOLS } from "../../src/mcp/brain/receipt-tools.ts";import type { ServerContext } from "../../src/mcp/tool-contract.ts";@@ -112,6 +124,45 @@ describe("brain_receipts compact at the caller's reach", () => {expect(res["reason"]).toBe(RECEIPT_COMPACT_REACH_REASON);});+ test("a shard name outside the family grammar is INVALID_PARAMS, never an internal error", async () => {+ // L9: the `family:name` parser checks the family but not the name+ // grammar, and the grammar refusal used to surface as a plain Error+ // rethrown out of the handler - an internal error for what is the+ // caller's malformed argument. Validated up front, by name.+ for (const bad of ["continuity:../x", "continuity:.jsonl", "continuity:2026-08.jsonl.bak"]) {+ try {+ await receipts(+ { ...ctx, reach: TRANSPORT_REACH.local },+ { mode: "compact", shards: [bad] },+ );+ expect.unreachable(`'${bad}' must be refused`);+ } catch (err) {+ expect(err).toBeInstanceOf(MCPError);+ expect((err as MCPError).code).toBe(INVALID_PARAMS);+ expect((err as MCPError).message).toContain("is not a continuity shard name");+ }+ }+ });++ test("a grammar-valid foreign shard is a refusal row, not invalid params", async () => {+ // The boundary the grammar cannot express: a well-formed+ // `<base>.<deviceId>.jsonl` name is a real shard - another device's,+ // so the core answers a named failure row rather than a parameter+ // error.+ writeFileSync(+ join(continuityLogDir(vault), "2026-08.telescope.jsonl"),+ continuityLine("recall_telemetry", "2026-08-10T12:00:00Z"),+ "utf8",+ );+ const res = (await receipts(+ { ...ctx, reach: TRANSPORT_REACH.local },+ { mode: "compact", shards: ["continuity:2026-08.telescope.jsonl"] },+ )) as { failures: Array<{ shard: string; message: string }> };+ expect(res.failures).toHaveLength(1);+ expect(res.failures[0]!.shard).toBe("continuity:2026-08.telescope.jsonl");+ expect(res.failures[0]!.message).toContain("owning device");+ });+test("at local reach the same tool previews the fold", async () => {const res = (await receipts({ ...ctx, reach: TRANSPORT_REACH.local },diff --git a/uv.lock b/uv.lockindex 3ee3f6aa..805ebe60 100644--- a/uv.lock+++ b/uv.lock@@ -4,5 +4,5 @@ requires-python = ">=3.11"[[package]]name = "open-second-brain"-version = "1.85.0"+version = "1.86.0"source = { editable = "." }