Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
fix: v1.86.0 review repairs and embedding preservation across schema upgrades (v1.86.1) (#259)
Serg Eroshefd8a0c8dba55
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex 416302b7..df47d265 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.86.0",+ "version": "1.86.1","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 3d8941ab..f089a57f 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.86.0",+ "version": "1.86.1","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex bb129a28..fe444fdb 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,30 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.86.1] - 2026-10-16++Open Second Brain 1.86.1 repairs what the review of v1.86.0 found and closes two embedding-loss defects: a vault that had the retrieval trust gate on keeps excluding after the upgrade, a barren session mine records its tombstone in the documented flow, the undistilled census reads the session store once so the SessionStart nudge stays inside its budget, and a search index left on an older schema is upgraded in place with every stored embedding kept instead of being rebuilt keyword-only.++### Fixed++- **The legacy trust gate switch keeps excluding.** `search_trust_gate_enabled: true` (and its env var) maps to `enforce` again, as it did before 1.86.0; measure-only mode is reached only through an explicit `search_trust_gate_mode: shadow`. The explain envelope names the resolved mode under `enforce` as well as `shadow`, and a refused legacy value names the key or env var it actually came from.+- **The relational arm and the bench follow the gate mode in force.** A relational query that names `contradicts` edges returns them under the enforcing gate instead of an empty answer, the arm takes its mode from the caller's effective config, the bench pin accepts `searchTrustGateMode` and refuses the obsolete `retrievalTrustGateEnabled` by name, and an invalid mode in an override is refused.+- **A barren session mine records its zero-yield tombstone without a relayed hash.** A plan that hands out a mining envelope (and each mining step `--propose-backlog --json` hands out) records the hash of the turns it offered as a pending survey, and a barren commit without a relay writes its tombstone over that material while the session still holds it, so the documented flow (plan, then commit) tombstones; a commit whose turns or ledger write fail keeps the survey for its retry. A turn imported after the plan, two pending plans over different turns, or no plan since the last commit withholds the tombstone and is named in `mining_ledger_failure`, so turns no model saw are never suppressed; a yield is recorded either way. A relayed `surveyed_turns_hash` is validated (an empty one included), and one the session no longer matches writes no record and is named the same way.+- **The undistilled census reads the session store once.** The census and the mining ledger's prune share one session-turn index, so the SessionStart backlog nudge and `import-session --status` stay fast on vaults with thousands of imported sessions instead of growing with the square of the session count.+- **The distill backlog drains.** `import-session --propose-backlog` hands out extract-signals mining steps for the next five undistilled sessions (`mining_steps`, one per session identity, with `mining_steps_remaining`), and answering a step records the session as mined. `--status` and `--discover` keep reporting coverage when the mining census fails, naming the census as unavailable. A session imported under `--recall-session-id` is counted correctly, and a corrupt session mining ledger names its remedy.+- **An appended source whose last line has no newline is no longer re-queued by every batch plan**, and an appended tail longer than 64 KiB is hashed over its real bytes rather than a reused read buffer.+- **Every capacity census honours an explicit config file.** The maintenance capacity gate, the ingest admission wall, the capture inbox drain, the storage sampler and the storage report measure the store named by `--config`; the capture inbox drain refuses ingest on an invalid `storage:` block, as `brain_ingest_source` does, and ignores errors in other blocks.+- **Scan and removal failures no longer expose host paths** to callers below local reach in `brain_doctor`, the digest, the operator summary and `brain_delete_by_source`.+- **Receipt compaction keeps `skill_invoked` records**, so usage-ordered skill offers stay stable, and no longer counts a merged-away prior aggregate as kept.+- **The explorer console refuses cross-origin requests** before any route runs, so a web page can no longer make it run the dry-run dream through a blind request.+- **Smaller fixes:** the semantic document dedup names the candidate cap once when it falls back to the lexical layer, the retirement marker in the content manifest is written under the manifest lock, and metric appends wait briefly for a busy surface lock instead of dropping the sample.+- **A keyword-only write keeps the recorded embedding prefixes** of vectors it keeps, so the next semantic index run no longer clears compatible embeddings; a refused prefix-only change now names the prefix change instead of reporting a model change to the same model ([#255](https://github.com/itechmeat/open-second-brain/issues/255)).+- **An older-schema search index is upgraded in place.** On the first search and at session start, an index left on an older schema by a previous release is migrated in place under the writer lock, keeping every stored embedding and its recorded model and prefixes, instead of being rebuilt keyword-only; the background self-heal run does the same, says it migrated rather than that the index was current, and session start still catches up files changed since the last index run. A self-heal rebuild re-checks the index once it holds the writer lock and skips work another process already finished. When a full rebuild is unavoidable (newer schema, missing tables, a chunker change, a corrupt or integrity-condemned file), Open Second Brain reports how many stored embeddings it dropped, where the previous index is kept and how to restore them, and a later rebuild never replaces a `.bak` that holds more stored embeddings than the index it retires, and a crash during that swap restores the live index rather than the older `.bak` ([#256](https://github.com/itechmeat/open-second-brain/issues/256)).++### Changed++- The 1.86.0 entry's statement that the legacy boolean maps `true` to `shadow` no longer holds: it maps to `enforce`.+## [1.86.0] - 2026-10-16Open Second Brain 1.86.0 stops re-paying for sessions with nothing new to say and measures before it excludes: a session mine that ends with nothing written records a zero-yield tombstone that suppresses every later plan over the same transcript until the turn material changes, a distilled and undistilled census over the imported backlog feeds an ask-once distill envelope and an opt-in SessionStart nudge, the retrieval trust gate resolves an `off | shadow | enforce` mode where `shadow` measures every would-be refusal while excluding nothing, and the loopback explorer serves a live-counts console beside its graph. The release also repairs what the external review of v1.85.0 found in the shipped storage lifecycle: receipt compaction now folds only opt-in telemetry kinds, the capacity trend buckets its samples per day while the maintenance lane pauses a band before the wall, a retired document copy keeps its manifest marker, and the incremental extraction watermark stamps exactly the bytes the appended delta covered.@@ -8567,6 +8591,7 @@ Hermes / Claude Code / Codex / OpenClaw configurations do not change.- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.86.1]: https://github.com/itechmeat/open-second-brain/compare/v1.86.0...v1.86.1[1.86.0]: https://github.com/itechmeat/open-second-brain/compare/v1.85.0...v1.86.0[1.85.0]: https://github.com/itechmeat/open-second-brain/compare/v1.84.1...v1.85.0[1.84.1]: https://github.com/itechmeat/open-second-brain/compare/v1.84.0...v1.84.1diff --git a/docs/brainstorm/session-mining-economy/design.md b/docs/brainstorm/session-mining-economy/design.mdindex d4e91a8a..70079935 100644--- a/docs/brainstorm/session-mining-economy/design.md+++ b/docs/brainstorm/session-mining-economy/design.md@@ -62,7 +62,7 @@ One mechanism correction to the variant as written: the consultant placed the mi- The undistilled census is derived vault-side from the two ledgers joined on `sessionRefIdentity`, with no log re-hashing inside the hook path. Sessions the plan refuses are excluded because they can never be distilled; counting them would nag forever about unparsable material. The hook answers from this census; if the budget runs out it exits 0 silently.- The SessionStart nudge is default-off and change-keyed. The card's fixed-interval wording is superseded: an interval that re-nags an unchanged backlog repeats the nag the card exists to kill, so the throttle is once per unchanged backlog, held in a vault-global, overwrite-only, size-one state file beside hygiene-digest.hash (the shipped precedent for vault-wide state whose lifetime is not per-session). The hook follows gap-agenda: flag via `resolveConfigFlag` with an env override, process ceiling, fail-open exit 0, byte-identical preamble when off.- The ask-once batching layer is a pure core function; the kernel stays LLM-free and never runs an interactive loop. Dedup happens before the ask, not before the write: candidates whose `computeDedupHash` is already in the vault index are dropped from the ask and named in a report, the remainder is grouped under one proposal per distinct dedup hash across sessions, and the single batched envelope is emitted through the import-session verb for the calling agent to present.-- The claim-chain gate's default is off, and measure-only (shadow) is the default mode whenever the gate is enabled without an explicit mode. Resolution order: explicit `search_trust_gate_mode` wins (invalid values refused naming the key, `parseFusionMode` precedent); absent the mode key, the legacy `search_trust_gate_enabled` boolean maps true to shadow and false to off; the default with neither set is off. Upstream's shadow-on-every-install is not copied - every recall knob in Open Second Brain defaults off with an explicit enable, and shadow is non-destructive precisely so enablement can be the first step. A vault that wants today's excluding behavior names `enforce` explicitly; the explain envelope names the resolved mode so the change is visible. There is no per-call override this wave.+- The claim-chain gate's default is off, and measure-only (shadow) is the default mode whenever the gate is enabled without an explicit mode. Resolution order: explicit `search_trust_gate_mode` wins (invalid values refused naming the key, `parseFusionMode` precedent); absent the mode key, the legacy `search_trust_gate_enabled` boolean maps true to shadow and false to off; the default with neither set is off. Upstream's shadow-on-every-install is not copied - every recall knob in Open Second Brain defaults off with an explicit enable, and shadow is non-destructive precisely so enablement can be the first step. A vault that wants today's excluding behavior names `enforce` explicitly; the explain envelope names the resolved mode so the change is visible. There is no per-call override this wave. Amended after the v1.86.0 review: the legacy boolean maps true to `enforce`, so a vault that opted in before the mode existed keeps excluding across the upgrade, and `shadow` is reached only by naming it.- Shadow mechanics add no verdict kind to the kernel. `applyRankAdjusters` keeps its single-pass `keep | multiply | exclude` contract; under shadow the mode-aware gate reuses `trustGateAdjuster`'s classification and returns keep for every item while the would-be refusals flow into the receipt build beside the existing flag-gated receipt path in post-rank. Only `off` is byte-identical: shadow adds receipt fields by design, so the query-cache key includes the resolved mode and the explain envelope reports it.- Edge eligibility is decided at the arm boundary, not inside the fan-out. Each walked node is classified from the same controlled-vocabulary frontmatter signals as the page gate; an edge passes when both endpoints pass, its validity window covers query time (`parseValidityWindow`, the `closedSupersessionTip` grammar) and its polarity is not a counted exclusion (the existing relation-polarity vocabulary, not upstream's claim-status integers). Eligibility refusals are reportable - removed from the contribution set and counted with named structural reason tokens - which is a different act from the relational arm's identical-to-absent provenance convention, and the design keeps the two conventions distinct.- Refusals surface as an extension of the two existing receipts, not a third receipt type and not a table. `excluded` keeps meaning actually dropped; shadow would-be counts and arm-level refusals are reported beside it, pool and arm reported separately so neither lane's numbers blur.diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex 2d64898f..31894709 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -848,8 +848,9 @@ every exclusion; `shadow` runs the same classification, excludesnothing, and records every would-be refusal with its structural reasonsinto the same receipts; `off` - the default - runs no gate. The legacy`search_trust_gate_enabled` (env `OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE`)-stays as the bare enable: true maps to `shadow`, so excluding is named-explicitly with `enforce`; `search_supersede_fade_enabled` (env+stays as the bare enable: true maps to `enforce`, the excluding+behavior it always had, so measuring first is named explicitly with+`shadow`; `search_supersede_fade_enabled` (env`OPEN_SECOND_BRAIN_SEARCH_SUPERSEDE_FADE`) makes an inbound `supersedes`/ `superseded_by` relation on a successor fade the unchanged supersedednote by a named multiplier. All default off and leave ranking@@ -866,9 +867,9 @@ o2b brain orphan-repair [--apply --confirm "apply orphan repair"] - detacho2b brain design-note <topic> [--payload <json> | --payload-file <path>] [--agent <name>] [--json] - the one-shot sibling of `o2b brain panel`. Without a payload it is read-only: it grounds the topic in the vault's tension records, decision records and truth projections and prints the single needs-llm-step envelope the calling agent answers, naming any store the vault holds nothing in (which is not the same as a store that matched nothing). With a payload it validates the written note and commits it as Brain/decisions/design-<date>-<topic>.md. The note must weigh named alternatives and mark EXACTLY ONE recommended: zero and two-plus are both refused, and the refusal states the count. A second note for the same topic on the same day is refused, never overwritteno2b brain skill-proposals page-candidates [--json] - read-only: gate the vault's user pages on the page-meta trio (core tier, non-stale lifecycle, high confidence) and an observed-reuse floor, skip any page an installed skill already covers, and return one needs-llm-step envelope per admitted page plus every skip with its reasono2b brain skill-proposals page-draft <page> (--payload <json> | --payload-file <path>) [--json] - validate a returned SKILL.md draft and stage it as a pending mature_page proposal INSIDE the vault; accept is what materializes the SKILL.md under the configured skills root, through the write-ahead journal-o2b brain extract-signals <session-ref> [--payload <json> | --payload-file <path>] [--force] [--agent <name>] [--json] - mine durable taste signals from an already-imported session's user turns. Without a payload it is read-only: it prints the turns it would mine and the single needs-llm-step envelope the calling agent answers. With a payload it validates the answer and writes the accepted items into Brain/inbox/ as speculative source_type: auto_extract signals, subject to the durability denylist and to Brain/pending/ staging when write approval is on. A payload over the per-session cap, an item below the confidence floor, or two items sharing a topic refuses the whole payload by name and writes nothing; a session with no imported turns is refused, never reported as empty+o2b brain extract-signals <session-ref> [--payload <json> | --payload-file <path>] [--surveyed-turns-hash <sha256>] [--force] [--agent <name>] [--json] - mine durable taste signals from an already-imported session's user turns. Without a payload it is read-only: it prints the turns it would mine and the single needs-llm-step envelope the calling agent answers. With a payload it validates the answer and writes the accepted items into Brain/inbox/ as speculative source_type: auto_extract signals, subject to the durability denylist and to Brain/pending/ staging when write approval is on. A payload over the per-session cap, an item below the confidence floor, or two items sharing a topic refuses the whole payload by name and writes nothing; a session with no imported turns is refused, never reported as emptyWith the optional decision-model turn pre-filter (use extract_prefilter, docs/decision-models/extract-prefilter.md) the --json plan may add turns_dropped, skipped { reason: decision_model_prefilter, turns_dropped } with llm_step: null (nothing to mine), and decision_model { degraded }; none appears with the use off. Payload items accept an optional string source_turn- Since v1.86.0 a commit that ends with nothing written and nothing staged records a zero-yield tombstone for the session in the session mining ledger (<vault>/.open-second-brain/session-mining-ledger.json, beside the session import ledger, keyed on a SHA-256 of the mined turn material), and a plan over that session at unchanged content returns no envelope: --json carries skipped { reason: mining_tombstone, mining_outcome { written, staged, deduped, durability_rejected } } with llm_step: null and empty turns_mined, and text mode names the breakdown. --force bypasses the tombstone for exactly the one plan call it is given to - the remedy when a downstream preference was removed and the session should be re-proposed; a re-import that changes the session's turns re-opens it with no clearing step, and a corrupt mining ledger is refused by name rather than read as empty. A commit that succeeds but cannot record its tombstone still exits 0, naming the failure as mining_ledger_failure in --json and in text+ Since v1.86.0 a commit that ends with nothing written and nothing staged records a zero-yield tombstone for the session in the session mining ledger (<vault>/.open-second-brain/session-mining-ledger.json, beside the session import ledger, keyed on a SHA-256 of the mined turn material), and a plan over that session at unchanged content returns no envelope: --json carries skipped { reason: mining_tombstone, mining_outcome { written, staged, deduped, durability_rejected } } with llm_step: null and empty turns_mined, and text mode names the breakdown. --force bypasses the tombstone for exactly the one plan call it is given to - the remedy when a downstream preference was removed and the session should be re-proposed; a re-import that changes the session's turns re-opens it with no clearing step, and a corrupt mining ledger is refused by name rather than read as empty. The commit keys the record on the session's turns as they stand at commit time. A plan that hands out an envelope records the hash of the turns it offered as a pending survey (<vault>/.open-second-brain/session-mining-surveys.json), so plan, then a plain commit of its answer (an empty items list included) is all it takes; the barren commit writes its tombstone only while the session still holds exactly the surveyed turns, and a turn imported after the plan, two pending plans over different turns, or no plan since the last commit withholds the tombstone and names why in mining_ledger_failure (a yield is recorded either way). A plan whose survey could not be recorded carries survey_record_failure in --json and names it in text. --surveyed-turns-hash optionally passes the plan's surveyed_turns_hash back as a check - a value that is not a 64-character lowercase hex digest refuses the commit, and a value the session no longer matches writes the signals but no mining record and names why. A commit that succeeds but cannot record its tombstone still exits 0, naming the failure as mining_ledger_failure in --json and in textSince v1.74.0 every turns_mined entry carries timestamp, the turn's stored timestamp verbatim ("" when the source has none), and each prompt line reads [<turn-id> @ <timestamp>] <text>, or [<turn-id>] <text> for an undated turn - the run's clock is never substituted. The envelope instruction holds the items to five language-neutral rules: a time bound is written as an ISO 8601 date or interval resolved against the stating turn's timestamp (an end-only bound as an interval, because the dream pass reads a lone date as a start), conversational mechanics are skipped, one rule per item, a rule's conditions stay in its principle, and restatements are dropped. The dream pass turns the ISO bound into the preference's valid_from / valid_until through its existing temporal extraction; the extract lane writes neither field itself. Two items with the same topic refuse the payload under semantic_cross_item, the message naming both indices (items[i] and items[j] share topic "<topic>"), before any item is hashed or writteno2b brain deep-synthesis --json now also carries findings with causal_context, decomposed confidence (support, opposition, freshness, coverage), and the excluded_findings ledger with excluded_finding_count```@@ -1199,7 +1200,7 @@ roots `src/core/runtime/host-facts.ts` declares:o2b brain import-session --status Per-runtime found / imported / gap counts, and the roots each was looked for undero2b brain import-session --discover The same report, plus the absolute path of every log that has never been importedo2b brain import-session --discover --all Import that gap, one file at a time, through the same importSession the path form calls-o2b brain import-session --propose-backlog One ask-once envelope over the undistilled backlog (since v1.86.0): candidates whose dedup hash the vault already holds are dropped and named, the rest collapse to one proposal per distinct hash across sessions with every contributing session listed, and a session whose every candidate was already known is reported rather than re-proposed+o2b brain import-session --propose-backlog One ask-once envelope over the undistilled backlog (since v1.86.0): candidates whose dedup hash the vault already holds are dropped and named, the rest collapse to one proposal per distinct hash across sessions with every contributing session listed, and a session whose every candidate was already known is reported rather than re-proposed. The candidates are feedback markers, which import already writes, so they surface only material removed from the vault after import; what leaves a session undistilled is its free text, so --json also carries mining_steps - the extract-signals needs-llm-step for each of the first five undistilled sessions, with mining_steps_remaining counting the rest (one step per session identity, however many imported paths share it). Each step records its surveyed turns the way an extract-signals plan does (a step whose survey could not be recorded carries survey_record_failure). Answering a step with 'o2b brain extract-signals <session> --payload <json>' (an empty items list included) dedups against the vault and records the session as mined, so the backlog drains batch by batch; a session imported again since the step is not recorded as barreno2b brain import-session --status|--discover --progressNewline-delimited progress on stderr under the `sessions` operation, one `hash` stage```diff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex df7dea09..be39a8f6 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -1709,8 +1709,8 @@ was deleted therefore surfaces as `url-only`, a `bounded-local` page countsas backed only while its excerpt still matches its digest, and quarantinedpages are skipped: they already carry the `untrusted_source` marker, whichnames the same condition (and which the retrieval trust gate excludes-under `search_trust_gate_mode: enforce` - the bare-`search_trust_gate_enabled` alone measures in `shadow`).+under `search_trust_gate_mode: enforce` or the legacy+`search_trust_gate_enabled: true`; `shadow` only measures).## Non-Markdown sources (since v1.69.0)diff --git a/docs/mcp.md b/docs/mcp.mdindex b6f97b20..5bf5fc0b 100644--- a/docs/mcp.md+++ b/docs/mcp.md@@ -478,8 +478,8 @@ reference with its structural reasons) and `memory_trust_assessment` (thesame exclusion set as a reason histogram). Both are by-products of theretrieval trust gate, which runs in one of three modes(`search_trust_gate_mode: off | shadow | enforce`; the bare-`search_trust_gate_enabled` alone maps to `shadow`, measuring without-excluding, and `enforce` is what excludes): with the gate off the+`search_trust_gate_enabled` alone maps to `enforce`, which excludes, and+`shadow` measures without excluding): with the gate off theresponse carries `retrieval_trace_unavailable` naming the switches thatproduce them (`search_trust_gate_enabled`, or`search_trust_gate_mode: shadow`) instead of going quiet. Without@@ -2186,8 +2186,21 @@ format characters), when it contains NUL, or when it exceeds the cap.bypasses the tombstone for exactly one plan call, for a caller thatremoved the downstream preference and wants the session re-proposed; acorrupt mining ledger refuses the plan by name rather than reading as- empty. A commit that succeeds but cannot record its tombstone still- succeeds and carries `mining_ledger_failure` by name.+ empty. The commit keys its record on the session's turns as they stand+ at commit time. A plan that hands out an envelope records the hash of+ the turns it offered as a pending survey, so answering the envelope with+ `{session, items}` (an empty `items` included) is all it takes: the+ barren commit writes its tombstone only while the session still holds+ exactly the surveyed turns, and a turn imported after the plan, two+ pending plans over different turns, or no plan since the last commit+ withholds it and names why in `mining_ledger_failure` (a yield is+ recorded either way). A plan whose survey could not be recorded carries+ `survey_record_failure`. The optional `surveyed_turns_hash` argument+ passes the plan's hash back as a check - a value that is not a SHA-256+ hex digest (the empty string included) refuses the commit, and one the+ session no longer matches writes no mining record and names why. A+ commit that succeeds but cannot record its tombstone still succeeds and+ carries `mining_ledger_failure` by name.- Advisory decision-model fields (optional, see[decision-models/dedup-tension.md](decision-models/dedup-tension.md) and[decision-models/labels.md](decision-models/labels.md)). All of them arediff --git a/hooks/backlog-nudge.ts b/hooks/backlog-nudge.tsindex f137a857..5ac1b7da 100644--- a/hooks/backlog-nudge.ts+++ b/hooks/backlog-nudge.ts@@ -91,7 +91,8 @@ async function main(): Promise<void> {const sessions = count === 1 ? "session" : "sessions";const line =`Open Second Brain session backlog: ${count} imported ${sessions} not yet distilled. ` +- "Run 'o2b brain import-session --status' for the per-runtime breakdown.";+ "Run 'o2b brain import-session --status' for the per-runtime breakdown, or " ++ "'o2b brain import-session --propose-backlog --json' for the next sessions to mine.";process.stdout.write(JSON.stringify({ hookSpecificOutput: { hookEventName, additionalContext: line } }) + "\n",);diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex 927d7fd3..ff0818d8 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.86.0",+ "version": "1.86.1","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex 90ec3a48..aee8730c 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.86.0",+ "version": "1.86.1","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 90f34cd1..8032f9ca 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.86.0"+version: "1.86.1"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex c53cf78e..02030d69 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.86.0",+ "version": "1.86.1","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/codex/skills/brain-memory/SKILL.md b/plugins/codex/skills/brain-memory/SKILL.mdindex 8adfa8be..9f10aef1 100644--- a/plugins/codex/skills/brain-memory/SKILL.md+++ b/plugins/codex/skills/brain-memory/SKILL.md@@ -185,6 +185,11 @@ exact bytes with `brain_session_expand` `{payload: "<ref>"}` or`o2b brain payload get <ref>` rather than guessing what was elided.To mine taste signals from an imported session afterwards, call-`brain_extract_signals` `{session}` and answer its one envelope (optionally-citing each item's `source_turn`); a plan with `llm_step: null` and-`skipped` means there is nothing to mine, so commit nothing.+`brain_extract_signals` `{session}` and answer its one envelope by calling+it again with `{session, items}` (optionally citing each item's+`source_turn`). Commit even when you found nothing - `items: []` records+that the session had nothing new, so it is not offered to a model again;+if the commit names a `mining_ledger_failure` (the session changed since+the plan), plan it again rather than retrying the commit. A+plan with `llm_step: null` and `skipped` means there is nothing to mine, so+commit nothing.diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 90f34cd1..8032f9ca 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.86.0"+version: "1.86.1"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex e5b224b4..fea2ca56 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.86.0"+version = "1.86.1"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/skills/brain-memory/SKILL.md b/skills/brain-memory/SKILL.mdindex 8adfa8be..9f10aef1 100644--- a/skills/brain-memory/SKILL.md+++ b/skills/brain-memory/SKILL.md@@ -185,6 +185,11 @@ exact bytes with `brain_session_expand` `{payload: "<ref>"}` or`o2b brain payload get <ref>` rather than guessing what was elided.To mine taste signals from an imported session afterwards, call-`brain_extract_signals` `{session}` and answer its one envelope (optionally-citing each item's `source_turn`); a plan with `llm_step: null` and-`skipped` means there is nothing to mine, so commit nothing.+`brain_extract_signals` `{session}` and answer its one envelope by calling+it again with `{session, items}` (optionally citing each item's+`source_turn`). Commit even when you found nothing - `items: []` records+that the session had nothing new, so it is not offered to a model again;+if the commit names a `mining_ledger_failure` (the session changed since+the plan), plan it again rather than retrying the commit. A+plan with `llm_step: null` and `skipped` means there is nothing to mine, so+commit nothing.diff --git a/src/cli/brain/verbs/bench.ts b/src/cli/brain/verbs/bench.tsindex 526e266f..371e4024 100644--- a/src/cli/brain/verbs/bench.ts+++ b/src/cli/brain/verbs/bench.ts@@ -227,7 +227,7 @@ const BENCH_PIN_NESTED_KEYS: Readonly<Record<string, ReadonlyArray<string>>> = {"cacheTtlSeconds","relationPolarityEnabled","relationalArmEnabled",- "retrievalTrustGateEnabled",+ "searchTrustGateMode","supersedeFadeEnabled","learnedWeightsEnabled","activationEnabled",diff --git a/src/cli/brain/verbs/extract-signals.ts b/src/cli/brain/verbs/extract-signals.tsindex b6ab79e4..c764a238 100644--- a/src/cli/brain/verbs/extract-signals.ts+++ b/src/cli/brain/verbs/extract-signals.ts@@ -8,12 +8,15 @@* that answer and writes the accepted items as speculative inbox signals.* OSB runs no model on either path.*- * A commit that answers a plan passes that plan's `surveyed_turns_hash`- * back with `--surveyed-turns-hash`: it is what the zero-yield mining- * tombstone keys on, so the record describes the material the verdict was- * earned over rather than whatever the session holds by commit time. A- * commit without it records nothing - without surveyed material there is- * nothing honest to key a tombstone on.+ * Every commit records its outcome in the session mining ledger over the+ * session's current mined turns. The plan records the hash of the turns it+ * offered as a pending survey, so a barren commit leaves the zero-yield+ * tombstone without any relay - but only while the session still holds+ * the surveyed turns; turns imported after the plan withhold it, and the+ * commit says so. `--surveyed-turns-hash` optionally passes the answered+ * plan's `surveyed_turns_hash` back as a check: a malformed value refuses+ * the commit, and a value the session no longer matches writes the+ * signals but no mining record, and says so.** A session already mined at its current content that yielded nothing* accepted carries a tombstone in the session mining ledger; its plan phase@@ -121,9 +124,9 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {text: t.text,timestamp: t.timestamp,})),- // The hash of the material this plan offered; the commit that- // answers it passes it back with --surveyed-turns-hash so the- // mining tombstone keys on the surveyed material.+ // The hash of the material this plan offered; a commit may pass it+ // back with --surveyed-turns-hash to check the session did not+ // change in between.surveyed_turns_hash: plan.surveyedTurnsHash,cap: plan.cap,confidence_floor: plan.confidenceFloor,@@ -153,6 +156,11 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {...(plan.decisionModel !== undefined? { decision_model: { degraded: plan.decisionModel.degraded } }: {}),+ // Present only when the plan could not record its survey; a+ // barren commit then needs --surveyed-turns-hash to tombstone.+ ...(plan.surveyRecordFailure !== undefined+ ? { survey_record_failure: plan.surveyRecordFailure }+ : {}),});return 0;}@@ -187,6 +195,12 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {if (plan.llmStep !== null) {ok(`needs-llm-step: ${plan.llmStep.step} -> ${plan.llmStep.target_path}`);ok(`surveyed turns hash: ${plan.surveyedTurnsHash}`);+ if (plan.surveyRecordFailure !== undefined) {+ ok(+ `survey not recorded (${plan.surveyRecordFailure}); pass --surveyed-turns-hash ` ++ `with the commit so a barren answer still records its tombstone`,+ );+ }}return 0;}@@ -216,9 +230,9 @@ export async function cmdBrainExtractSignals(argv: string[]): Promise<number> {// Loaded with parallel I/O here, where the caller can await it; the// commit's own fallback is the synchronous walk.dedup: await loadDedupIndex(vault),- // The answered plan's surveyed hash, relayed by the calling agent:- // what the zero-yield tombstone keys on. Absent, the commit records- // nothing - there is no material the verdict was earned over.+ // The answered plan's surveyed hash, when relayed: a check that the+ // session did not change since the plan. Absent, the commit checks+ // the survey the plan recorded instead....(surveyedTurnsHashRaw !== null ? { surveyedTurnsHash: surveyedTurnsHashRaw } : {}),});recordExtractPrefilterCommit(vault, decisionModel, res, now);diff --git a/src/cli/brain/verbs/import-session.ts b/src/cli/brain/verbs/import-session.tsindex 5a0f63ad..cd400ccb 100644--- a/src/cli/brain/verbs/import-session.ts+++ b/src/cli/brain/verbs/import-session.ts@@ -49,6 +49,14 @@ import {minedTurnMaterial,type SessionMiningCensus,} from "../../../core/brain/sessions/mining-census.ts";+import { buildMiningStep } from "../../../core/brain/extract-signals.ts";+import { recordMiningSurvey } from "../../../core/brain/session-mining-surveys.ts";+import type { NeedsLlmStep } from "../../../core/brain/llm-step.ts";+import { minedTurnsHash } from "../../../core/brain/session-mining-ledger.ts";+import {+ buildSessionTurnIndex,+ type SessionTurnIndex,+} from "../../../core/brain/session-recall.ts";import {SESSION_RESUME_DISCARD,type SessionResumeDiscard,@@ -225,6 +233,7 @@ export async function cmdBrainImportSession(argv: string[]): Promise<number> {recordUnderDeclaredRoots(vault,result.files.map((f) => f.file),+ recallSessionId,);} catch (exc) {ledgerError = (exc as Error).message ?? String(exc);@@ -346,8 +355,11 @@ async function sweep(req: SweepRequest): Promise<number> {// The distillation census is vault-side (the two ledgers plus the// continuity turns the mining lane itself reads), so it rides the// coverage report without re-hashing a single source log. Its own- // deadline is fresh: discovery's safeguard was spent on the sweep.- let mining: SessionMiningCensus;+ // deadline is fresh: discovery's safeguard was spent on the sweep. A+ // census failure (a corrupt mining ledger, a deadline) is a fact about+ // the census, not about import coverage: the coverage the operator+ // asked for is still reported, with the census named unavailable.+ let mining: SessionMiningCensus | { readonly error: string };try {mining = buildMiningCensus(req.vault, {safeguard: createSafeguard({@@ -356,7 +368,7 @@ async function sweep(req: SweepRequest): Promise<number> {}),});} catch (exc) {- return fail(`session mining census failed: ${(exc as Error).message ?? exc}`);+ mining = { error: (exc as Error).message ?? String(exc) };}emitCoverage(discovery, mining, { asJson: req.asJson, listGap: req.discover });return 0;@@ -373,7 +385,8 @@ async function sweep(req: SweepRequest): Promise<number> {// the other's writes.// oxlint-disable-next-line no-await-in-loopfiles.push(await importSession(req.vault, entry.path, req.importOptions));- imported.push(entry);+ const pinned = req.importOptions.recallSessionId;+ imported.push(pinned !== undefined ? { ...entry, sessionId: pinned } : entry);} catch (exc) {// One unreadable log must not lose the rest of the sweep; the// failure is named and the run carries on.@@ -403,6 +416,13 @@ async function sweep(req: SweepRequest): Promise<number> {return failures.length > 0 ? 1 : 0;}+/**+ * How many undistilled sessions one `--propose-backlog` call hands out as+ * mining steps. Each step embeds the session's user turns, so the batch is+ * bounded; answered steps leave the backlog and the next call continues.+ */+const BACKLOG_MINING_BATCH = 5;+interface ProposeBacklogRequest {readonly vault: string;readonly config: string | null;@@ -420,20 +440,34 @@ interface ProposeBacklogRequest {* the presenting agent asks. An empty backlog is an explicit empty* envelope and exits 0; it is an answer, not a failure.*- * The candidates are the feedback markers in the sessions' mined turn- * material (the same material the mining plan embeds) - the one candidate- * form the kernel can see without a model. The dedup index is the shared+ * The marker candidates are the feedback markers in the sessions' mined+ * turn material (the same material the mining plan embeds) - the one+ * candidate form the kernel can see without a model. Import already writes+ * every marker it sees, so a marker proposal surfaces only material that+ * was removed from the vault after import. The dedup index is the shared* vault-wide one, so proposals cross-deduplicate against every capture* path exactly as the write-time gate does.+ *+ * What actually leaves a session undistilled is the free text a model+ * would mine, so the envelope also carries the mining steps for the first+ * {@link BACKLOG_MINING_BATCH} undistilled sessions: the same+ * needs-llm-step a `brain extract-signals` plan returns, per session. The+ * agent answers each with an `extract-signals` commit, which dedups+ * against the vault (so a fact a second session raises again is written+ * once) and records the session in the mining ledger - so every answered+ * step leaves the backlog, and the next call batches the next sessions.*/async function proposeBacklogEnvelope(req: ProposeBacklogRequest): Promise<number> {let census: SessionMiningCensus;+ let turnIndex: SessionTurnIndex;try {+ turnIndex = buildSessionTurnIndex(req.vault);census = buildMiningCensus(req.vault, {safeguard: createSafeguard({operation: OPERATION.sessions,timeoutMs: resolveSafeguardTimeoutMs(OPERATION.sessions, req.config ?? undefined),}),+ turns: turnIndex,});} catch (exc) {return fail(`session mining census failed: ${(exc as Error).message ?? exc}`);@@ -441,13 +475,36 @@ async function proposeBacklogEnvelope(req: ProposeBacklogRequest): Promise<numbeconst boundary = buildCaptureBoundary(req.vault);const candidates: BacklogCandidate[] = [];+ const miningSteps: Array<{+ readonly sessionId: string;+ readonly surveyedTurnsHash: string;+ readonly turnsMined: number;+ readonly llmStep: NeedsLlmStep;+ }> = [];+ // The census counts import-ledger PATHS, and one session identity can+ // sit under several paths or runtimes; the steps are per identity, so+ // the backlog walks each identity once and counts the minable ones+ // itself - the number the remaining-steps figure is measured against.+ const seen = new Set<string>();+ let minable = 0;for (const row of census.byRuntime) {for (const sessionId of row.undistilledSessionIds) {- const material = minedTurnMaterial(req.vault, sessionId, boundary);+ if (seen.has(sessionId)) continue;+ seen.add(sessionId);+ const material = minedTurnMaterial(req.vault, sessionId, boundary, turnIndex);// A census undistilled row is minable by construction; the null arm// is the boundary changing between the two reads, and contributes// no candidates rather than a guess.if (material === null) continue;+ minable += 1;+ if (miningSteps.length < BACKLOG_MINING_BATCH) {+ miningSteps.push({+ sessionId,+ surveyedTurnsHash: minedTurnsHash(material),+ turnsMined: material.length,+ llmStep: buildMiningStep(sessionId, material),+ });+ }for (const turn of material) {for (const marker of discoverMarkers(turn.text)) {if (!isFeedbackMarker(marker)) continue;@@ -471,8 +528,21 @@ async function proposeBacklogEnvelope(req: ProposeBacklogRequest): Promise<numbe}const envelope = buildBacklogProposals(candidates, dedupIndex);+ const remaining = Math.max(0, minable - miningSteps.length);if (req.asJson) {+ // The JSON form hands out the envelopes, so each step records its+ // survey the way an extract-signals plan does: a barren commit that+ // answers it without relaying the hash can then still tell whether the+ // session changed in between. A failed record is named on its step.+ const surveyFailures = new Map<string, string>();+ for (const step of miningSteps) {+ try {+ recordMiningSurvey(req.vault, step.sessionId, step.surveyedTurnsHash);+ } catch (exc) {+ surveyFailures.set(step.sessionId, exc instanceof Error ? exc.message : String(exc));+ }+ }okJson({undistilled: census.undistilled,proposals: envelope.proposals.map((p) => ({@@ -489,6 +559,16 @@ async function proposeBacklogEnvelope(req: ProposeBacklogRequest): Promise<numbededup_hash: d.dedupHash,})),report_only_sessions: envelope.reportOnlySessions,+ mining_steps: miningSteps.map((step) => ({+ session_id: step.sessionId,+ surveyed_turns_hash: step.surveyedTurnsHash,+ turns_mined: step.turnsMined,+ llm_step: step.llmStep,+ ...(surveyFailures.has(step.sessionId)+ ? { survey_record_failure: surveyFailures.get(step.sessionId) }+ : {}),+ })),+ mining_steps_remaining: remaining,});return 0;}@@ -508,15 +588,38 @@ async function proposeBacklogEnvelope(req: ProposeBacklogRequest): Promise<numbefor (const drop of envelope.dropped) info(` ${drop.sessionId}: ${drop.topic}`);ok(`sessions with nothing new: ${envelope.reportOnlySessions.length}`);for (const sessionId of envelope.reportOnlySessions) info(` ${sessionId}`);+ ok(`mining steps: ${miningSteps.length}`);+ for (const step of miningSteps) {+ ok(` mine: ${step.sessionId} (${step.turnsMined} user turn(s))`);+ }+ if (miningSteps.length > 0) {+ // This form hands out no envelope, so it records no survey: a barren+ // commit drains a session only over material a plan really offered.+ ok(+ " answer each with an extract-signals commit of its envelope's payload " ++ "(--json here hands out the envelopes); an empty answer to an envelope drains the session",+ );+ }+ if (remaining > 0) ok(`undistilled sessions after this batch: ${remaining}`);return 0;}/** `path -> the runtime whose declared root contains it`, recorded in the ledger. */-function recordUnderDeclaredRoots(vault: string, paths: ReadonlyArray<string>): void {+function recordUnderDeclaredRoots(+ vault: string,+ paths: ReadonlyArray<string>,+ recallSessionId: string | null,+): void {const records: SessionImportRecord[] = [];for (const path of paths) {const runtime = runtimeForPath(path);- if (runtime !== null) records.push({ path, runtime });+ if (runtime !== null) {+ records.push({+ path,+ runtime,+ ...(recallSessionId !== null ? { sessionId: recallSessionId } : {}),+ });+ }}if (records.length > 0) recordSessionImports(vault, records);}@@ -595,9 +698,11 @@ function miningJson(census: SessionMiningCensus): Record<string, unknown> {/** Report what the sweep found, without importing anything. */function emitCoverage(discovery: SessionDiscovery,- mining: SessionMiningCensus,+ censusOrError: SessionMiningCensus | { readonly error: string },opts: { asJson: boolean; listGap: boolean },): void {+ const miningError = "error" in censusOrError ? censusOrError.error : null;+ const mining = "error" in censusOrError ? null : censusOrError;if (opts.asJson) {okJson({found: discovery.found,@@ -606,7 +711,7 @@ function emitCoverage(unparsable: discovery.unparsable,unreadable: discovery.unreadable,by_runtime: discovery.byRuntime.map(coverageJson),- mining: miningJson(mining),+ ...(mining !== null ? { mining: miningJson(mining) } : { mining_error: miningError }),...(opts.listGap? { would_import: discovery.byRuntime.flatMap((row) => [...row.gapPaths]) }: {}),@@ -617,7 +722,8 @@ function emitCoverage(ok(`imported: ${discovery.imported}`);ok(`gap: ${discovery.gap}`);if (discovery.unparsable > 0) ok(`unparsable: ${discovery.unparsable}`);- if (mining.imported > 0) {+ if (miningError !== null) ok(`mining: unavailable (${miningError})`);+ if (mining !== null && mining.imported > 0) {ok(`mining: ${mining.distilled} distilled, ${mining.knownNothingNew} known-nothing-new, ` +`${mining.undistilled} undistilled`,@@ -629,7 +735,7 @@ function emitCoverage(ok(` imported: ${row.imported}`);ok(` gap: ${row.gap}`);if (row.unparsable > 0) ok(` unparsable: ${row.unparsable}`);- const minedRow = mining.byRuntime.find((m) => m.runtime === row.runtime);+ const minedRow = mining?.byRuntime.find((m) => m.runtime === row.runtime);if (minedRow !== undefined) {ok(` distilled: ${minedRow.distilled}`);ok(` known-nothing-new: ${minedRow.knownNothingNew}`);diff --git a/src/cli/brain/verbs/inbox-drain.ts b/src/cli/brain/verbs/inbox-drain.tsindex 99c2aedb..d0ca0689 100644--- a/src/cli/brain/verbs/inbox-drain.ts+++ b/src/cli/brain/verbs/inbox-drain.ts@@ -50,7 +50,12 @@ export async function cmdBrainInboxDrain(argv: string[]): Promise<number> {});const { config, vault } = brainVerbContext(flags);const agent = resolveBrainAgent(flags, config);- const report = drainInbox(vault, { apply: flags["apply"] === true, agent, now: new Date() });+ const report = drainInbox(vault, {+ apply: flags["apply"] === true,+ agent,+ now: new Date(),+ configPath: config,+ });const pending = flags["apply"] !== true && report.items.length > 0;diff --git a/src/cli/brain/verbs/maintenance.ts b/src/cli/brain/verbs/maintenance.tsindex ff0ab9dd..1d23a738 100644--- a/src/cli/brain/verbs/maintenance.ts+++ b/src/cli/brain/verbs/maintenance.ts@@ -410,6 +410,7 @@ export async function cmdBrainMaintenance(argv: string[]): Promise<number> {...(window !== undefined ? { window } : {}),busy: { minutes: busyMinutes, threshold: busyThreshold },...(retryTasks.length > 0 ? { retryTasks } : {}),+ ...(config ? { configPath: config } : {}),tasks: lane.tasks,});diff --git a/src/cli/search/verbs/indexing.ts b/src/cli/search/verbs/indexing.tsindex 123c3682..eb177873 100644--- a/src/cli/search/verbs/indexing.ts+++ b/src/cli/search/verbs/indexing.ts@@ -23,7 +23,12 @@ import {SearchError,serializeChunkWindowCensus,} from "../../../core/search/index.ts";-import type { ChunkWindowCensus, IndexStats } from "../../../core/search/index.ts";+import type {+ ChunkWindowCensus,+ IndexStats,+ IndexVaultOptions,+} from "../../../core/search/index.ts";+import { healIndex } from "../../../core/search/self-heal.ts";import {recordSelfHealOutcome,SELF_HEAL_REINDEX_OUTCOME,@@ -279,11 +284,14 @@ async function runFreshenChild(** `--self-heal <run-id>` marks the run as the automatic post-upgrade* rebuild that `ensureVaultCurrent` spawns detached, with every stream- * ignored because there is no terminal to write to. It changes nothing- * about the rebuild; it only makes the run's terminal outcome - success or- * the failure by name - land on the `self_heal_reindex` metrics surface- * under the run id the parent minted, which is the only place such a run- * can be read from afterwards and the only thing its two rows pair on.+ * ignored because there is no terminal to write to. Such a run heals+ * rather than rebuilds unconditionally: an index on an older schema is+ * migrated in place, keeping its stored embeddings, and an index another+ * process already brought current is left alone (`self-heal.ts`). The+ * run's terminal outcome - success or the failure by name - lands on the+ * `self_heal_reindex` metrics surface under the run id the parent minted,+ * which is the only place such a run can be read from afterwards and the+ * only thing its two rows pair on.*/export async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<number> {const { flags } = parseFlags(argv, {@@ -335,9 +343,10 @@ export async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<numconst cfg = resolveSelfHealConfig(flags, selfHeal, startedAt);const observation = observeIndexRun(flags, "reindex");const interrupt = onInterrupt(OPERATION.reindex);- let stats: IndexStats;+ let stats: IndexStats | null;+ let migrated = false;try {- stats = await reindexVault(cfg, {+ const reindexOpts: IndexVaultOptions = {safeguard: reindexSafeguard(flags),embeddings: flagBoolean(flags, "embeddings"),forceCost: flagBoolean(flags, "force-cost"),@@ -346,7 +355,16 @@ export async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<num? (e) => process.stderr.write(`${e.kind}\t${e.path}\n`): undefined,...(observation?.sink !== undefined ? { onProgress: observation.sink } : {}),- });+ };+ if (selfHeal === undefined) {+ stats = await reindexVault(cfg, reindexOpts);+ } else {+ // The automatic run heals rather than rebuilds: an older schema is+ // migrated in place so its stored embeddings survive (#256).+ const healed = await healIndex(cfg, reindexOpts);+ stats = healed.kind === "rebuilt" ? healed.stats : null;+ migrated = healed.kind === "migrated";+ }} catch (err) {// The only report a self-heal run's failure ever gets. Recorded before// the rethrow so the row exists whatever the caller's stderr is@@ -381,6 +399,15 @@ export async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<numDate.now() - startedAt,);}+ if (stats === null) {+ process.stderr.write(+ migrated+ ? `search index ${cfg.dbPath} was migrated in place to the current schema; ` ++ `its stored embeddings were kept and no rebuild was needed\n`+ : `search index ${cfg.dbPath} is current; no rebuild was needed\n`,+ );+ return 0;+ }reportIndexRun(stats, cfg, argv, flagBoolean(flags, "json"));return 0;}diff --git a/src/core/brain/capture/inbox-drain.ts b/src/core/brain/capture/inbox-drain.tsindex c09bb82c..cf768750 100644--- a/src/core/brain/capture/inbox-drain.ts+++ b/src/core/brain/capture/inbox-drain.ts@@ -46,7 +46,7 @@ import { formatFrontmatter, parseFrontmatter, slugify } from "../../vault.ts";import { addObligation, obligationExists, parseCadence, ObligationError } from "../obligations.ts";import { ingestSource } from "../ingest/ingest.ts";import { CAPACITY_GATE_DECISION, evaluateIngestAdmission } from "../storage/gate.ts";-import type { CapacityGateDecision } from "../storage/gate.ts";+import type { IngestAdmission } from "../storage/gate.ts";import { MAINTENANCE_CAPACITY_REMEDY } from "../maintenance/lane.ts";import { isoSecond } from "../time.ts";import {@@ -115,6 +115,8 @@ export interface DrainOptions {readonly apply: boolean;readonly agent: string;readonly now: Date;+ /** The explicit machine-config file the admission census resolves the store from. */+ readonly configPath?: string;}/** One item's route decided structurally, before any write. */@@ -135,7 +137,7 @@ function classify(vault: string,note: CaptureNote,opts: DrainOptions,- admission: CapacityGateDecision | null,+ admission: IngestAdmission,): RoutePlan | UnroutableCapture {const body = note.body.trim();const firstLine = body.split("\n", 1)[0]!.trim();@@ -200,11 +202,19 @@ function planSource(vault: string,url: string,opts: DrainOptions,- admission: CapacityGateDecision | null,+ admission: IngestAdmission,): RoutePlan | UnroutableCapture {- if (admission?.decision === CAPACITY_GATE_DECISION.skip) {+ // A storage block the wall cannot see through refuses, the same rule+ // brain_ingest_source applies: reading it as "no cap" would ingest past+ // the operator's bound. The capture stays staged for a rerun once the+ // block is fixed.+ if (admission.configRefusal !== undefined) {+ return new UnroutableCapture(`ingest refused (config_invalid): ${admission.configRefusal}`);+ }+ const decision = admission.decision;+ if (decision?.decision === CAPACITY_GATE_DECISION.skip) {return new UnroutableCapture(- `ingest refused (capacity_exhausted): the vault measured at the ${admission.tier} ` ++ `ingest refused (capacity_exhausted): the vault measured at the ${decision.tier} ` +`storage tier. ${MAINTENANCE_CAPACITY_REMEDY}.`,);}@@ -313,17 +323,16 @@ export function drainInbox(vault: string, opts: DrainOptions): DrainReport {let hubPool: HubPagePool | undefined;// One admission evaluation per drain, not per capture: the census is a// full vault walk, and the answer cannot change between captures of- // the same run. A null decision means no wall stands this pass - no- // configured cap, or a storage block that could not be validated,- // which is named on the operator's stderr rather than wedging the- // inbox shut.- const admission = evaluateIngestAdmission(vault, { now: opts.now });- if (admission.reason !== undefined) {- process.stderr.write(`capture inbox drain: ${admission.reason}\n`);- }+ // the same run. A null decision with no config refusal means no wall+ // stands this pass (no configured cap); a storage block the wall cannot+ // see through refuses the ingest route by name, like brain_ingest_source.+ const admission = evaluateIngestAdmission(vault, {+ now: opts.now,+ ...(opts.configPath !== undefined ? { configPath: opts.configPath } : {}),+ });for (const note of listStagedCaptures(vault)) {- const plan = classify(vault, note, opts, admission.decision);+ const plan = classify(vault, note, opts, admission);if (plan instanceof UnroutableCapture) {unroutable += 1;items.push({diff --git a/src/core/brain/digest.ts b/src/core/brain/digest.tsindex 8e55459b..1c699c7c 100644--- a/src/core/brain/digest.ts+++ b/src/core/brain/digest.ts@@ -46,6 +46,7 @@ import { findMergeCandidates } from "./merge-candidates.ts";import { computeMostApplied } from "./most-applied.ts";import { BRAIN_RETIRED_REL, brainDirs, vaultRelative } from "./paths.ts";import {+ closedCollectError,collectMaintenanceActions,type MaintenanceCollectError,type MaintenanceCollectSignal,@@ -362,13 +363,20 @@ export function renderDigest(vault: string, opts: RenderDigestOptions = {}): Ren);}- const data = collectDigestData(+ const collected = collectDigestData(vault,since,until,opts.agentScope,readerRefView(vault, opts.readable),);+ // A digest rendered for a reader below local reach (a `readable`+ // filter) names a failed scan without its raised message, which can+ // carry a host path.+ const data =+ opts.readable === undefined+ ? collected+ : { ...collected, collect_errors: collected.collect_errors.map(closedCollectError) };const empty = isEmpty(data);if (format === "json") {diff --git a/src/core/brain/distill/backlog-proposals.ts b/src/core/brain/distill/backlog-proposals.tsindex 11c90ac0..81c7e1c4 100644--- a/src/core/brain/distill/backlog-proposals.ts+++ b/src/core/brain/distill/backlog-proposals.ts@@ -129,17 +129,18 @@ export function buildBacklogProposals(sessions: [...sessionsByHash.get(hash)!].toSorted(),};});- dropped.sort((a, b) =>- a.sessionId === b.sessionId- ? a.topic === b.topic- ? a.dedupHash.localeCompare(b.dedupHash)- : a.topic.localeCompare(b.topic)- : a.sessionId.localeCompare(b.sessionId),- );+ dropped.sort(compareDrops);return { proposals, dropped, reportOnlySessions };}+/** Drops in (session, topic, dedup hash) order. */+function compareDrops(a: BacklogProposalDrop, b: BacklogProposalDrop): number {+ if (a.sessionId !== b.sessionId) return a.sessionId.localeCompare(b.sessionId);+ if (a.topic !== b.topic) return a.topic.localeCompare(b.topic);+ return a.dedupHash.localeCompare(b.dedupHash);+}+/*** Whether `a`'s (topic, signal, principle, scope) tuple sorts before `b`'s,* field by field in code-unit order. The missing and the empty scopediff --git a/src/core/brain/document-retire.ts b/src/core/brain/document-retire.tsindex 65ac70fe..7a09f2a5 100644--- a/src/core/brain/document-retire.ts+++ b/src/core/brain/document-retire.ts@@ -60,7 +60,7 @@ import { join } from "node:path";import { parseFrontmatter } from "../vault.ts";import { ensureInsideVault } from "../path-safety.ts";import { isoSecond } from "./time.ts";-import { readManifest, writeManifestAtomic, manifestPath } from "./ingest/content-manifest.ts";+import { markManifestRetired } from "./ingest/content-manifest.ts";import { appendContinuitySourceInvalidation } from "./continuity/store.ts";import { removeFile } from "./source-cleanup.ts";import { basenameCollides, retargetWikilinks } from "./page-dedup.ts";@@ -405,22 +405,21 @@ function retireOneCopy(let manifestClause: string;if (facts.sourcePath === null) {manifestClause = "no manifest key recorded";- } else if (!existsSync(manifestPath(vault))) {- manifestClause = `manifest absent, nothing to mark for ${facts.sourcePath}`;} else {- const manifest = readManifest(vault);- const entries = { ...manifest.entries };- const existing = entries[facts.sourcePath];- if (existing === undefined) {+ // One locked read-modify-write, so a concurrent ingest's manifest+ // write can neither be lost to this one nor drop the marker.+ const mark = markManifestRetired(vault, facts.sourcePath, copy.canonical);+ if (mark === "no-manifest") {+ manifestClause = `manifest absent, nothing to mark for ${facts.sourcePath}`;+ } else if (mark === "no-entry") {manifestClause = `manifest key ${facts.sourcePath} absent, nothing to mark`;- } else if (existing.retired_into === copy.canonical) {- // The entry already carries THIS retirement's marker: the serialized- // bytes below would be identical, and the clause names that- // success-shaped fact - a concurrent retirer got there first.+ } else if (mark === "already-marked") {+ // The entry already carries THIS retirement's marker: the clause+ // names that success-shaped fact - a concurrent retirer got there+ // first.manifestClause = `manifest key ${facts.sourcePath} already marked retired`;} else {- entries[facts.sourcePath] = { ...existing, retired_into: copy.canonical };- manifestRetired = writeManifestAtomic(vault, entries, manifest.contract);+ manifestRetired = true;manifestClause = `marked manifest key ${facts.sourcePath} retired into ${copy.canonical}`;}}diff --git a/src/core/brain/explorer.ts b/src/core/brain/explorer.tsindex 243f1dda..f4ba87a5 100644--- a/src/core/brain/explorer.ts+++ b/src/core/brain/explorer.ts@@ -431,6 +431,26 @@ export function buildLiveServer(403,);}+ // A loopback Host does not make a request the operator's own: any+ // web page can send a blind cross-origin GET to 127.0.0.1 and make+ // the server run the operator snapshot (a dry-run dream) even though+ // the page cannot read the answer. The browser names a cross-origin+ // sender in Sec-Fetch-Site and Origin; only the console's own page+ // (same-origin) and a typed URL (none) are served.+ const fetchSite = req.headers.get("sec-fetch-site");+ const origin = req.headers.get("origin");+ const ownOrigin = `http://${req.headers.get("host")}`;+ if (+ (fetchSite !== null && fetchSite !== "same-origin" && fetchSite !== "none") ||+ (origin !== null && origin !== ownOrigin)+ ) {+ return jsonResponse(+ JSON.stringify({+ error: "explorer: refusing a cross-origin request; open the console URL directly",+ }),+ 403,+ );+ }const url = new URL(req.url);if (url.pathname === "/" || url.pathname === "/index.html") {const graph = collectExplorerData(vault);diff --git a/src/core/brain/extract-signals-prefilter.ts b/src/core/brain/extract-signals-prefilter.tsindex 822840df..d5a47eaa 100644--- a/src/core/brain/extract-signals-prefilter.ts+++ b/src/core/brain/extract-signals-prefilter.ts@@ -80,6 +80,7 @@ import {} from "./extract-signals.ts";import type { NeedsLlmStep } from "./llm-step.ts";import { readMiningTombstone, type SessionMiningOutcome } from "./session-mining-ledger.ts";+import { closedSurveyFailure, recordMiningSurvey } from "./session-mining-surveys.ts";import { listSessionRawTurns } from "./session-recall.ts";import { estimateTokens as estimateTextTokens } from "./text/tokenizer.ts";import { emitTokenImpact, TOKEN_COUNT_METHOD } from "./token-impact.ts";@@ -124,6 +125,12 @@ export interface PrefilteredSignalExtractionPlan extends Omit<SignalExtractionPl};/** Present only when the use is not `off` and a request failed. */readonly decisionModel?: { readonly degraded: DecisionDegradeReason };+ /**+ * Present only when the plan handed out an envelope but could not record+ * its survey: a barren commit answering it then needs the relayed+ * `surveyed_turns_hash` to write its tombstone.+ */+ readonly surveyRecordFailure?: string;}export interface PrefilteredPlanOptions {@@ -143,6 +150,12 @@ export interface PrefilteredPlanOptions {* file, and it changes nothing else about the plan.*/readonly force?: boolean;+ /**+ * Name a survey-record failure by its store and errno code only, never+ * by the raised message, which carries an absolute host path. Set by+ * surfaces that answer a caller below local reach.+ */+ readonly closeHostErrors?: boolean;}/** Whether this configuration runs the pre-filter at all. */@@ -293,12 +306,39 @@ function miningTombstoneGate(* the pre-filter, because the spend it prevents is the caller's mining* round trip and not only this module's requests. Refusals* ({@link ExtractSignalsError}) are thrown as there, before the gate.- * `opts.force` lifts the gate for this one call.+ * `opts.force` lifts the gate for this one call. A plan that hands out an+ * envelope records its surveyed-turns hash as a pending survey - the one+ * write of the plan phase - so a commit that relays no hash can still+ * withhold a tombstone over turns the plan never offered.*/export async function planExtractSignalsPrefiltered(vault: string,sessionId: string,opts: PrefilteredPlanOptions,+): Promise<PrefilteredSignalExtractionPlan> {+ const plan = await planPrefilteredEnvelope(vault, sessionId, opts);+ // A plan that hands out an envelope records what it surveyed, so the+ // commit that answers it without relaying the hash can still tell+ // whether the session changed in between (session-mining-surveys.ts).+ // A failed record never fails the plan - the plan is still right - and+ // is named so the caller knows the commit will need the relay.+ if (plan.llmStep === null) return plan;+ try {+ recordMiningSurvey(vault, plan.sessionId, plan.surveyedTurnsHash, opts.now);+ return plan;+ } catch (err) {+ const raw = err instanceof Error ? err.message : String(err);+ return Object.freeze({+ ...plan,+ surveyRecordFailure: opts.closeHostErrors === true ? closedSurveyFailure(err) : raw,+ });+ }+}++async function planPrefilteredEnvelope(+ vault: string,+ sessionId: string,+ opts: PrefilteredPlanOptions,): Promise<PrefilteredSignalExtractionPlan> {const cfg = opts.decisionModel;const force = opts.force === true;diff --git a/src/core/brain/extract-signals.ts b/src/core/brain/extract-signals.tsindex 484bfbcc..ace41faf 100644--- a/src/core/brain/extract-signals.ts+++ b/src/core/brain/extract-signals.ts@@ -31,14 +31,14 @@* route to a confirmed preference. Nothing here writes a preference.** The commit is also the mining ledger's write side (session-mining-ledger.ts):- * every successful commit that answers a plan records its outcome breakdown- * over the turn material the PLAN surveyed - the plan carries the hash of- * the turns it offered, and the surfaces pass it back with the answer. A- * zero-yield record (nothing written, nothing staged) is the tombstone the- * plan gate suppresses on; a yield record is the distilled state the mining- * census reports. A commit that answers no plan of this module's attests- * nothing and records nothing, and a partial write never records - a run- * that FAILED did not end.+ * every successful commit records its outcome breakdown over the session's+ * current mined turns. A zero-yield record (nothing written, nothing+ * staged) is the tombstone the plan gate suppresses on, so it is written+ * only over material a plan put in front of a model: the hash the caller+ * relays, or else the one survey the plan surfaces recorded+ * (session-mining-surveys.ts). A yield record is the distilled state the+ * mining census reports. A partial write never records - a run that FAILED+ * did not end.** Precision beats recall, as it does in `fact-extract.ts`: the cap and the* floor are refusals, not filters. A payload over either limit is rejected@@ -75,6 +75,11 @@ import {} from "./response-shape.ts";import { listSessionRawTurns, type ExpandedRawTurn } from "./session-recall.ts";import { minedTurnsHash, recordSessionMiningOutcome } from "./session-mining-ledger.ts";+import {+ clearMiningSurvey,+ closedSurveyFailure,+ readMiningSurvey,+} from "./session-mining-surveys.ts";import { writeSignal } from "./signal.ts";import { isoDate, isoSecond } from "./time.ts";import { BRAIN_SIGNAL_SOURCE_TYPE, type BrainSignalSign } from "./types.ts";@@ -222,10 +227,11 @@ export interface SignalExtractionPlan {readonly turnsMined: ReadonlyArray<MinedTurn>;/*** {@link minedTurnsHash} of {@link turnsMined}: the material this plan- * put in front of the caller. The commit that answers this plan passes- * it back as {@link CommitExtractedSignalsOptions.surveyedTurnsHash}, so- * the mining record keys on the material the verdict was earned over,- * never on whatever the session holds by commit time.+ * put in front of the caller. The commit that answers this plan may pass+ * it back as {@link CommitExtractedSignalsOptions.surveyedTurnsHash} to+ * check the session did not change in between. Without the relay the+ * commit reads the survey the plan surfaces recorded for the session+ * instead.*/readonly surveyedTurnsHash: string;/** The limits the payload will be held to, echoed so the caller can read them. */@@ -326,14 +332,29 @@ export interface CommitExtractedSignalsOptions {readonly dedup?: Map<string, DedupIndexEntry>;/*** The answered plan's {@link SignalExtractionPlan.surveyedTurnsHash}: the- * hash of the turns the caller was actually shown. The mining record keys- * on it, so an import landing between plan and commit cannot put a- * tombstone on turns no model ever saw. Absent means this commit answers- * no plan of this module's, and then there is no material the verdict was- * earned over - so nothing is recorded, and the session stays eligible- * (the safe direction, the same one an unreadable entry gets).+ * hash of the turns the caller was actually shown. Optional: the commit+ * records over the hash of the session's CURRENT mined turns, and a+ * zero-yield record (the tombstone) additionally needs evidence that a+ * model saw exactly that material - this relay, or else the single+ * pending survey the plan surfaces recorded, so the documented flow+ * (plan, answer the envelope, commit the items) writes its tombstone+ * without relaying anything. When the hash IS relayed it is a check: it+ * must be a SHA-256 hex digest (anything else refuses the commit before+ * a write), and when it differs from the current material - an import+ * landed between plan and commit, or the hash came from another plan -+ * nothing is recorded and the commit names that as its mining ledger+ * failure. Without a relay, a survey that differs from the current+ * material, several pending surveys, or none withholds the tombstone and+ * is named the same way; a yield record is recorded regardless.*/readonly surveyedTurnsHash?: string;+ /**+ * Name a pending-survey store failure in `miningLedgerFailure` by its+ * store and errno code only, never by the raised message, which carries+ * the absolute host path. Set by surfaces that answer a caller below+ * local reach.+ */+ readonly closeHostErrors?: boolean;}// ----- Semantic rules the descriptor language cannot express ---------------@@ -623,6 +644,9 @@ function transcriptLabel(turn: MinedTurn): string {// ----- Phase two: what the caller mined ------------------------------------+/** A plan's surveyed-turns hash: a SHA-256 digest in lowercase hex. */+const SURVEYED_TURNS_HASH_RE = /^[0-9a-f]{64}$/;+/*** Validate the returned payload and write the accepted items.*@@ -650,6 +674,14 @@ export function commitExtractedSignals(`session ${trimmed} is classified '${boundaryDecision}' by the capture boundary; it may not produce signals`,);}+ if (+ opts.surveyedTurnsHash !== undefined &&+ !SURVEYED_TURNS_HASH_RE.test(opts.surveyedTurnsHash)+ ) {+ throw new ExtractSignalsError(+ `surveyed_turns_hash must be the 64-character lowercase hex digest a plan printed, got '${opts.surveyedTurnsHash}'`,+ );+ }// Structure first, then the cross-item rules. Nothing is written until// both gates pass, so a payload with one bad item lands nothing.assertResponseShape(EXTRACTED_SIGNALS_SURFACE, EXTRACTED_SIGNALS_SHAPE, payload);@@ -754,26 +786,66 @@ export function commitExtractedSignals();}- // Every commit that answers a plan is the ledger's write trigger: the- // outcome breakdown is recorded over the hash the PLAN surveyed - the- // material the verdict was earned over - whatever the yield was. The- // zero-yield record is the tombstone the plan gate suppresses on; a- // yield record is the distilled state the census reports, and it is NOT- // a tombstone - the session stays minable. A thrown- // ExtractSignalsWriteError never reaches here: a run that FAILED did not- // end, so it records nothing. And a commit that answers no plan of this- // module's attests nothing, so it records nothing: without surveyed- // material there is nothing honest to key a record on, and failing open- // to a re-mine is the safe direction.+ // Every commit is the ledger's write trigger: the outcome breakdown is+ // recorded over the hash of the session's current mined turns - the+ // material a plan issued now would offer, and the one the plan gate and+ // the census compare against. A yield record is the distilled state the+ // census reports, and it is NOT a tombstone - the session stays minable.+ // The zero-yield record IS the tombstone the plan gate suppresses on, so+ // it is written only over material a model was shown: the relayed plan+ // hash when there is one, else the one survey a plan recorded since the+ // last commit (session-mining-surveys.ts) - and only while the session+ // still holds that material. A thrown ExtractSignalsWriteError never+ // reaches here: a run that FAILED did not end, so it records nothing. A+ // session with no mined material records nothing either, and every+ // withheld record says why: failing open to a re-mine is the safe+ // direction.const staged = writeApprovalEnabled ? written.length : 0;+ const zeroYield = written.length === 0;let miningLedgerFailure: string | undefined;- if (opts.surveyedTurnsHash !== undefined) {+ let currentHash: string | null = null;+ const raw = (err: unknown): string => (err instanceof Error ? err.message : String(err));+ const surveyText = (err: unknown): string =>+ opts.closeHostErrors === true ? closedSurveyFailure(err) : raw(err);+ // Read, not consumed: the survey is cleared below only once this+ // commit's outcome is final, so a commit that fails transiently (the+ // turns unreadable, the ledger write refused) leaves it for the retry.+ let surveyed: ReadonlyArray<string> = [];+ let surveyFailure: string | undefined;+ try {+ surveyed = readMiningSurvey(vault, trimmed);+ } catch (err) {+ surveyFailure = surveyText(err);+ }+ let outcomeFinal = true;+ try {+ const mined = selectMinedTurns(boundary, listSessionRawTurns(vault, trimmed));+ currentHash = mined.length === 0 ? null : minedTurnsHash(mined);+ } catch (err) {+ miningLedgerFailure = raw(err);+ outcomeFinal = false;+ }+ if (currentHash !== null && opts.surveyedTurnsHash !== undefined) {+ if (opts.surveyedTurnsHash !== currentHash) {+ miningLedgerFailure =+ `the session's turns no longer match the surveyed_turns_hash this commit answers ` ++ `(${opts.surveyedTurnsHash}); no mining record was written, so the session stays minable`;+ currentHash = null;+ }+ } else if (currentHash !== null && zeroYield) {+ const withheld = withheldTombstoneReason(surveyed, surveyFailure, currentHash);+ if (withheld !== null) {+ miningLedgerFailure = withheld;+ currentHash = null;+ }+ }+ if (currentHash !== null) {try {recordSessionMiningOutcome(vault,{sessionId: trimmed,- minedContentHash: opts.surveyedTurnsHash,+ minedContentHash: currentHash,outcome: {written: written.length,staged,@@ -788,7 +860,17 @@ export function commitExtractedSignals(// commit as failed, and dropping the error would leave a ledger that// silently stopped being written. Name it instead - the surfaces// carry it verbatim.- miningLedgerFailure = err instanceof Error ? err.message : String(err);+ miningLedgerFailure = raw(err);+ outcomeFinal = false;+ }+ }+ if (outcomeFinal && surveyFailure === undefined) {+ try {+ clearMiningSurvey(vault, trimmed, surveyed);+ } catch (err) {+ // The outcome stands; a survey left behind can only attest material+ // a plan really offered, so it is named rather than thrown.+ miningLedgerFailure ??= `the answered plan survey could not be cleared (${surveyText(err)})`;}}@@ -808,6 +890,36 @@ export function commitExtractedSignals(});}+/**+ * Why a zero-yield commit that relayed no plan hash may not write its+ * tombstone, or `null` when it may: exactly one distinct material was+ * surveyed since the last commit and the session still holds it. Anything+ * else means the commit cannot attest that a model saw the turns it would+ * suppress.+ */+function withheldTombstoneReason(+ surveyed: ReadonlyArray<string>,+ surveyFailure: string | undefined,+ currentHash: string,+): string | null {+ const tail =+ "no zero-yield record was written, so the session stays minable; re-plan it, or relay the " ++ "plan's surveyed_turns_hash with the commit";+ if (surveyFailure !== undefined) {+ return `the plan survey of this session could not be read (${surveyFailure}); ${tail}`;+ }+ if (surveyed.length === 0) {+ return `no plan of this session was recorded since its last commit, so this commit cannot attest which turns a model saw; ${tail}`;+ }+ if (surveyed.length > 1) {+ return `${surveyed.length} plans over different turns of this session are pending, so this commit cannot tell which one it answers; ${tail}`;+ }+ if (surveyed[0] !== currentHash) {+ return `the session's turns changed since the plan this commit answers (${surveyed[0]}); ${tail}`;+ }+ return null;+}+/*** The key a missing item is NAMED by: its payload index and its topic.* The index carries the position and the topic carries the meaning,diff --git a/src/core/brain/hygiene/detectors/document-dedup.ts b/src/core/brain/hygiene/detectors/document-dedup.tsindex 463888fe..90a92ec4 100644--- a/src/core/brain/hygiene/detectors/document-dedup.ts+++ b/src/core/brain/hygiene/detectors/document-dedup.ts@@ -546,22 +546,22 @@ export async function detectSemanticDocumentDedup(...result,findings: Object.freeze(withCandidateCapNotice(listed, read, [...result.findings])),});- if (compared.length < 2) return withCap(lexical());+ if (compared.length < 2) return lexical();const threshold = opts.threshold ?? DOCUMENT_DEDUP_EMBEDDING_THRESHOLD;let vectors: number[][];try {vectors = await provider.embed(compared.map((r) => r.body));} catch {- return withCap(lexical());+ return lexical();}- if (vectors.length !== compared.length) return withCap(lexical());+ if (vectors.length !== compared.length) return lexical();// A ragged batch - vectors of differing dimensions - is the same// broken-provider shape as a count mismatch: the cosine below would// silently truncate to the shorter vector and fabricate a similarity// out of incomparable spaces, so the whole run answers lexical, labeled.const dimension = vectors[0]!.length;- if (vectors.some((v) => v.length !== dimension)) return withCap(lexical());+ if (vectors.some((v) => v.length !== dimension)) return lexical();const pairs: PairScore[] = [];for (let i = 0; i < compared.length; i++) {diff --git a/src/core/brain/ingest/content-manifest.ts b/src/core/brain/ingest/content-manifest.tsindex 992cae54..7e7f7fcd 100644--- a/src/core/brain/ingest/content-manifest.ts+++ b/src/core/brain/ingest/content-manifest.ts@@ -110,6 +110,17 @@ export interface ManifestEntry {* Absent together with the byte count on unwatermarked entries.*/readonly extracted_at?: string;+ /**+ * SHA-256 hex of exactly the leading {@link extracted_bytes} bytes, when+ * that prefix is shorter than the source - an append whose last line had+ * no newline yet, so the extracting pass stopped at the last complete+ * line. Read only by the append proof ({@link extractedDigestOf});+ * {@link sha256} stays the whole-file digest classification compares, so+ * such a source classifies `unchanged` until it changes again. Absent,+ * the prefix is the whole source and {@link sha256} describes it.+ * ADDITIVE: a malformed shape drops the field, never the entry.+ */+ readonly extracted_sha256?: string;/*** The resurrect guard (storage lifecycle, PR #257 review H3): the vault* path of the canonical summary page this source's near-duplicate copy@@ -141,6 +152,16 @@ export function extractedBytesOf(entry: ManifestEntry | undefined): number {return entry?.extracted_bytes ?? 0;}+/**+ * The digest of the entry's leading {@link extractedBytesOf} bytes - what+ * the append proof verifies a grown source against: the prefix digest when+ * the extracted prefix is shorter than the recorded source, else the+ * whole-file digest, which then describes the same bytes.+ */+export function extractedDigestOf(entry: ManifestEntry): string {+ return entry.extracted_sha256 ?? entry.sha256;+}+/*** The persisted manifest: a map from a canonical vault-relative source path* to the state recorded for it at last ingest, plus the contract the@@ -325,6 +346,7 @@ function parseEntryValue(value: unknown): ManifestEntry | null {const extractedBytes = record["extracted_bytes"];const extractedAt = record["extracted_at"];const retiredInto = record["retired_into"];+ const extractedSha = record["extracted_sha256"];return {sha256: record["sha256"],...(typeof fingerprint === "string" ? { fingerprint } : {}),@@ -335,6 +357,9 @@ function parseEntryValue(value: unknown): ManifestEntry | null {...(typeof retiredInto === "string" && retiredInto.length > 0? { retired_into: retiredInto }: {}),+ ...(typeof extractedSha === "string" && /^[0-9a-f]{64}$/.test(extractedSha)+ ? { extracted_sha256: extractedSha }+ : {}),};}@@ -353,6 +378,7 @@ function hasRecordedState(entry: ManifestEntry): boolean {return (entry.fingerprint !== undefined ||entry.extracted_bytes !== undefined ||+ entry.extracted_sha256 !== undefined ||entry.retired_into !== undefined);}@@ -621,6 +647,38 @@ export function updateManifest(}}+/** What {@link markManifestRetired} found and did. */+export type ManifestRetireMark = "no-manifest" | "no-entry" | "already-marked" | "marked";++/**+ * Stamp `retired_into` on one entry under the manifest lock - the same+ * critical section {@link updateManifest} takes - so a concurrent ingest+ * can neither lose its own write to this one nor overwrite the entry and+ * drop the marker. Every other field of the entry, and every other entry,+ * is kept as read inside the lock.+ */+export function markManifestRetired(+ vault: string,+ key: string,+ retiredInto: string,+): ManifestRetireMark {+ if (!existsSync(manifestPath(vault))) return "no-manifest";+ const handle = acquireLockSyncWithRetry(manifestPath(vault), undefined, {+ remedy: ingestLockRemedy,+ });+ try {+ const manifest = readManifest(vault);+ const existing = manifest.entries[key];+ if (existing === undefined) return "no-entry";+ if (existing.retired_into === retiredInto) return "already-marked";+ const entries = { ...manifest.entries, [key]: { ...existing, retired_into: retiredInto } };+ writeManifestAtomic(vault, entries, manifest.contract);+ return "marked";+ } finally {+ handle.release();+ }+}+/** Recursively collect regular-file relative paths under `dir` (POSIX slashes). */function collectFilesRel(dir: string, prefix: string, out: string[]): void {for (const entry of readdirSync(dir, { withFileTypes: true })) {diff --git a/src/core/brain/ingest/ingest.ts b/src/core/brain/ingest/ingest.tsindex 8ff00910..f3825ed1 100644--- a/src/core/brain/ingest/ingest.ts+++ b/src/core/brain/ingest/ingest.ts@@ -81,7 +81,12 @@ import {type Provenance,} from "../provenance/provenance.ts";import { PLAN_ID_LABEL, recordCompleted } from "./checkpoint.ts";-import { extractedBytesOf, readManifest, updateManifest } from "./content-manifest.ts";+import {+ extractedBytesOf,+ extractedDigestOf,+ readManifest,+ updateManifest,+} from "./content-manifest.ts";import { computeExtractionContractFingerprint } from "./contract.ts";import { cutAppendedDelta } from "./tail-read.ts";import {@@ -526,13 +531,15 @@ export function ingestSource(// run's result. Such a read degrades the fail-safe way: no delta is// reported and no stamp lands, so the source simply reprocesses fully// next time. The ENOENT window and the EISDIR window are the same catch.- let manifestStamp: { readonly digest: string; readonly bytes: number } | undefined;+ let manifestStamp:+ | { readonly digest: string; readonly wholeDigest: string; readonly bytes: number }+ | undefined;let stampFullSize = false;try {const snapshot = readAppendSnapshot(sourceAbs,watermark,- entry === undefined ? "" : entry.sha256,+ entry === undefined ? "" : extractedDigestOf(entry),);const delta = appendedDeltaFromSnapshot(watermark, snapshot);if (delta !== undefined) extractionDelta = delta;@@ -542,6 +549,7 @@ export function ingestSource(// leading `toByte` bytes, computed with no second read.manifestStamp = {digest: snapshot.hashThroughWatermark.digest("hex"),+ wholeDigest: snapshot.wholeDigest,bytes: delta.toByte,};} else {@@ -563,9 +571,15 @@ export function ingestSource(// fragment shape - two CONCURRENT ingests of one source, the later// snapshot taken before the earlier run's append landed - and the// recorded pair stays self-consistent either way.+ // The whole-file digest stays what classification compares, so an+ // unterminated fragment does not re-queue the source on every batch+ // plan; the prefix digest rides beside it for the next append proof.updateManifest(vault, [canonicalSource], {[canonicalSource]: {- sha256: manifestStamp.digest,+ sha256: manifestStamp.wholeDigest,+ ...(manifestStamp.digest !== manifestStamp.wholeDigest+ ? { extracted_sha256: manifestStamp.digest }+ : {}),fingerprint: computeExtractionContractFingerprint(vault),extracted_bytes: manifestStamp.bytes,extracted_at: isoSecond(opts.now),@@ -758,7 +772,7 @@ export function appendedExtractionDelta(if (entry === undefined) return undefined;const watermark = extractedBytesOf(entry);if (watermark <= 0) return undefined;- const snapshot = readAppendSnapshot(sourceAbs, watermark, entry.sha256);+ const snapshot = readAppendSnapshot(sourceAbs, watermark, extractedDigestOf(entry));const delta = appendedDeltaFromSnapshot(watermark, snapshot);// The preview's contract is unchanged by the named over-cap outcome: a// tail past the cap answers without a section, exactly as before (the@@ -790,6 +804,12 @@ interface AppendSnapshot {* the delta series names, computed with no second read (review L2).*/readonly hashThroughWatermark: Hash;+ /**+ * SHA-256 hex of every byte this one pass read: the whole source as the+ * snapshot saw it. Empty when `watermark <= 0` (no pass was made); only+ * the appended arm, which requires a watermark, reads it.+ */+ readonly wholeDigest: string;}/**@@ -827,8 +847,10 @@ function readAppendSnapshot(prefixProven: false,tail: { overCap: false, bytes: Buffer.alloc(0) },hashThroughWatermark: createHash("sha256"),+ wholeDigest: "",};}+ const hashWhole = createHash("sha256");const hashPrefix = createHash("sha256");const hashThroughWatermark = createHash("sha256");const tailParts: Buffer[] = [];@@ -839,6 +861,7 @@ function readAppendSnapshot(const count = readSync(fd, chunk, 0, chunk.length, position);if (count === 0) break;const buf = count === chunk.length ? chunk : chunk.subarray(0, count);+ hashWhole.update(buf);const prefixEnd = Math.min(watermark - position, count);if (prefixEnd > 0) {const head = buf.subarray(0, prefixEnd);@@ -846,7 +869,9 @@ function readAppendSnapshot(hashThroughWatermark.update(head);}if (position + count > watermark && tailFilled <= EXTRACTION_DELTA_MAX_BYTES) {- const part = buf.subarray(Math.max(0, watermark - position));+ // Copied: `chunk` is reused by the next read, so a view into it+ // would be overwritten whenever the tail spans more than one chunk.+ const part = Buffer.from(buf.subarray(Math.max(0, watermark - position)));tailParts.push(part);tailFilled += part.length;}@@ -860,6 +885,7 @@ function readAppendSnapshot(? { overCap: true }: { overCap: false, bytes: Buffer.concat(tailParts) },hashThroughWatermark,+ wholeDigest: hashWhole.digest("hex"),};} finally {closeSync(fd);diff --git a/src/core/brain/maintenance/collect.ts b/src/core/brain/maintenance/collect.tsindex 66a9b642..bd79c501 100644--- a/src/core/brain/maintenance/collect.ts+++ b/src/core/brain/maintenance/collect.ts@@ -29,6 +29,22 @@ export type MaintenanceCollectSignal = "dedup" | "lint" | "token-footprint";export interface MaintenanceCollectError {readonly signal: MaintenanceCollectSignal;readonly message: string;+ /** The errno code the scan failed with, when the error carried one. */+ readonly code?: string;+}++/**+ * The same failure with a closed message: the scan name plus the errno+ * code, never the raised message, which can carry a host path (an+ * `ENOENT: ... open '/home/...'`). For every surface that answers a+ * caller below local reach.+ */+export function closedCollectError(error: MaintenanceCollectError): MaintenanceCollectError {+ return {+ signal: error.signal,+ message: `${error.signal} scan failed (${error.code ?? "error"})`,+ ...(error.code !== undefined ? { code: error.code } : {}),+ };}export interface CollectMaintenanceActionsOptions {@@ -56,7 +72,12 @@ function collectSignal<T>(try {return run();} catch (err) {- onError({ signal, message: (err as Error)?.message ?? String(err) });+ const code = (err as NodeJS.ErrnoException | null)?.code;+ onError({+ signal,+ message: (err as Error)?.message ?? String(err),+ ...(typeof code === "string" ? { code } : {}),+ });return undefined;}}diff --git a/src/core/brain/maintenance/lane.ts b/src/core/brain/maintenance/lane.tsindex 82d55580..e7dd410c 100644--- a/src/core/brain/maintenance/lane.ts+++ b/src/core/brain/maintenance/lane.ts@@ -168,6 +168,11 @@ export interface EvaluateGatesOptions {capacity: CapacityGate,now: Date,) => CapacityGateDecision;+ /**+ * The explicit machine-config file the caller resolved from, so the+ * default census measures the store a `search_db_path` in it names.+ */+ readonly configPath?: string;}/**@@ -449,7 +454,8 @@ function readCapacityDecision(opts: EvaluateGatesOptions,): CapacityGateDecision | undefined {if (opts.capacity === undefined) return undefined;- return (opts.readCapacity ?? evaluateStorageCapacityGate)(vault, opts.capacity, opts.now);+ if (opts.readCapacity !== undefined) return opts.readCapacity(vault, opts.capacity, opts.now);+ return evaluateStorageCapacityGate(vault, opts.capacity, opts.now, opts.configPath);}/** The gate decision from a fresh on-disk census, never the stored aggregate. */@@ -457,9 +463,11 @@ function evaluateStorageCapacityGate(vault: string,capacity: CapacityGate,now: Date,+ configPath: string | undefined,): CapacityGateDecision {return evaluateCapacityGate(vault, {now,+ ...(configPath !== undefined ? { configPath } : {}),capBytes: capacity.cap_bytes,warnPercent: capacity.warn_percent,criticalPercent: capacity.critical_percent,@@ -497,6 +505,7 @@ export async function runMaintenance(...(opts.busy !== undefined ? { busy: opts.busy } : {}),...(opts.readPressure !== undefined ? { readPressure: opts.readPressure } : {}),...(opts.readCapacity !== undefined ? { readCapacity: opts.readCapacity } : {}),+ ...(opts.configPath !== undefined ? { configPath: opts.configPath } : {}),...resolvePressureGate(opts.pressure, policy.host_pressure_percent),...resolveCapacityGate(opts.capacity, storage),};diff --git a/src/core/brain/metrics.ts b/src/core/brain/metrics.tsindex e63640c0..d1b59b79 100644--- a/src/core/brain/metrics.ts+++ b/src/core/brain/metrics.ts@@ -32,7 +32,7 @@ import {shardedFileName,type LedgerShardGrammar,} from "./ledger-shards.ts";-import { acquireLockSync } from "./sync-lockfile.ts";+import { acquireLockSyncWithRetry } from "./sync-lockfile.ts";import { assertVaultIdentityForWrite } from "./vault-identity.ts";/** On-disk schema version stamped on every metric record. */@@ -122,6 +122,13 @@ function assertSurface(surface: string): void {}}+/**+ * How long a metric append waits for its surface file's lock: long enough+ * to ride out another append or a shard rewrite, short enough that a+ * telemetry write never stalls the operation it measures.+ */+const METRIC_APPEND_LOCK_WAIT_MS = 1_000;+/*** Append one run-level metric record to the surface's JSONL file.* Creates `Brain/metrics/` on first write.@@ -149,8 +156,10 @@ export function appendMetric(vault: string, input: AppendMetricInput): void {// append and the sample is silently lost. Under the lock the rewrite// either waits for this append to land (and then reports drift against// its recount) or fails its own lock acquisition - either way the sample- // is a named outcome, never a silent overwrite.- const handle = acquireLockSync(path);+ // is a named outcome, never a silent overwrite. The acquisition waits a+ // short budget: two appends to one surface are routine, and a lock that+ // refused at once lost the second sample to a swallowing caller.+ const handle = acquireLockSyncWithRetry(path, METRIC_APPEND_LOCK_WAIT_MS);try {appendFileSync(path, JSON.stringify(record) + "\n");} finally {diff --git a/src/core/brain/receipts/compact.ts b/src/core/brain/receipts/compact.tsindex f19150fa..36f939e4 100644--- a/src/core/brain/receipts/compact.ts+++ b/src/core/brain/receipts/compact.ts@@ -447,6 +447,9 @@ function planShard(for (const pending of pendings) {group = mergePrior(group, pending.prior);mergedPriorLines.add(pending.index);+ // Counted as kept when it was met; the rebuild now drops it in+ // favour of the merged line, so it is no longer kept.+ kept -= 1;emitIndex = Math.min(emitIndex, pending.index);}}diff --git a/src/core/brain/receipts/types.ts b/src/core/brain/receipts/types.tsindex 13aa76a3..3ce7381d 100644--- a/src/core/brain/receipts/types.ts+++ b/src/core/brain/receipts/types.ts@@ -138,7 +138,6 @@ export const FOLDABLE_CONTINUITY_KINDS: ReadonlyArray<ContinuityRecordKind> = Ob"token_impact_outcome","context_pack_outcome","context_pack_evidence",- "skill_invoked","decision_model_call","decision_model_extract_commit",]);@@ -178,7 +177,12 @@ export const FOLDABLE_CONTINUITY_KINDS: ReadonlyArray<ContinuityRecordKind> = Ob* destroy exactly that auditability;* - `receipt_compaction` - the compaction audit trail itself; a fold* that removed the record of the run before it would erase the only- * account of what the fold has already destroyed.+ * account of what the fold has already destroyed;+ * - `skill_invoked` - read back as data by `deriveSkillUsage`, whose+ * lifetime per-skill counts order the skill offers; a fold past the+ * cutoff would collapse a skill's usage to its last window and+ * reorder the offers (its payload carries `agent`, not `agent_id`, so+ * the clip-key rule never protected it either).*/export const KEPT_CONTINUITY_KINDS: ReadonlyArray<ContinuityRecordKind> = Object.freeze(["source_invalidation",@@ -195,6 +199,7 @@ export const KEPT_CONTINUITY_KINDS: ReadonlyArray<ContinuityRecordKind> = Object"recall_observed_use","negative_recall","receipt_compaction",+ "skill_invoked",]);/** One shard selected for compaction, as a scan reported it. */diff --git a/src/core/brain/session-mining-ledger.ts b/src/core/brain/session-mining-ledger.tsindex 1e56b9df..5b915030 100644--- a/src/core/brain/session-mining-ledger.ts+++ b/src/core/brain/session-mining-ledger.ts@@ -46,10 +46,9 @@ import { existsSync, readFileSync } from "node:fs";import { join } from "node:path";import { atomicWriteFileSync } from "../fs-atomic.ts";-import { listContinuityRecords } from "./continuity/store.ts";import { hashBytes } from "./ingest/content-manifest.ts";import { DERIVED_STORE_DIR } from "./path-constants.ts";-import { listSessionRawTurns, sessionIdOf } from "./session-recall.ts";+import { buildSessionTurnIndex } from "./session-recall.ts";import { acquireLockSyncWithRetry, ingestLockRemedy } from "./sync-lockfile.ts";import { isoSecond } from "./time.ts";import { assertVaultIdentityForWrite } from "./vault-identity.ts";@@ -117,6 +116,16 @@ export interface SessionMiningLedger {readonly entries: Readonly<Record<string, SessionMiningEntry>>;}+/**+ * The remedy every file-level refusal names: the refusal stops every plan,+ * the census and the nudge, so the operator must learn what removing the+ * file costs (every session re-opens for mining) and that nothing else is+ * lost.+ */+const CORRUPT_LEDGER_REMEDY =+ "; to recover, move the file aside or delete it - every imported session then becomes " ++ "eligible for mining again, and the next commits rebuild the ledger";+/*** Read the ledger. A missing file is an empty ledger - no session is* tombstoned, which is the correct answer for a vault that has mined@@ -136,16 +145,18 @@ export function readSessionMiningLedger(vault: string): SessionMiningLedger {try {parsed = JSON.parse(readFileSync(path, "utf8"));} catch (e) {- throw new Error(`session mining ledger is corrupted JSON: ${path}`, { cause: e });+ throw new Error(`session mining ledger is corrupted JSON: ${path}${CORRUPT_LEDGER_REMEDY}`, {+ cause: e,+ });}if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) {- throw new Error(`session mining ledger is not an object: ${path}`);+ throw new Error(`session mining ledger is not an object: ${path}${CORRUPT_LEDGER_REMEDY}`);}const obj = parsed as Record<string, unknown>;if (obj["schema_version"] !== SESSION_MINING_LEDGER_SCHEMA_VERSION) {throw new Error(`session mining ledger schema_version ${String(obj["schema_version"])} not supported ` +- `(expected ${SESSION_MINING_LEDGER_SCHEMA_VERSION}): ${path}`,+ `(expected ${SESSION_MINING_LEDGER_SCHEMA_VERSION}): ${path}${CORRUPT_LEDGER_REMEDY}`,);}const entries: Record<string, SessionMiningEntry> = {};@@ -157,7 +168,9 @@ export function readSessionMiningLedger(vault: string): SessionMiningLedger {// different from per-ENTRY damage, which is dropped rather than guessed// at, because a dropped entry can only re-open one session for mining.if (raw === null || typeof raw !== "object" || Array.isArray(raw)) {- throw new Error(`session mining ledger entries is not an object: ${path}`);+ throw new Error(+ `session mining ledger entries is not an object: ${path}${CORRUPT_LEDGER_REMEDY}`,+ );}for (const [key, value] of Object.entries(raw as Record<string, unknown>)) {const entry = readEntry(key, value);@@ -275,20 +288,12 @@ export function recordSessionMiningOutcome(// One store pass for the whole prune, mirroring the import ledger's// one-existsSync-per-entry idiom: a full turn listing per entry here// would be a whole-store read plus a lineage build per entry, growing- // with the ledger's history instead of with this record.- const sessionsWithTurns = new Set<string>();- for (const turnRecord of listContinuityRecords(vault)) {- if (turnRecord.kind !== "session_turn") continue;- const sid = sessionIdOf(turnRecord.payload);- if (sid !== undefined) sessionsWithTurns.add(sid);- }+ // with the ledger's history instead of with this record. The index+ // applies the lineage scope, so a root ref mined over segment-recorded+ // turns keeps its entry.+ const turns = buildSessionTurnIndex(vault);for (const key of Object.keys(entries)) {- if (sessionsWithTurns.has(key)) continue;- // A session holding no direct turn record may still have turns,- // reachable through the lineage scope (a root ref mined over- // segment-recorded turns); the exact read decides those, and only- // those.- if (listSessionRawTurns(vault, key).length === 0) delete entries[key];+ if (turns.turnsFor(key).length === 0) delete entries[key];}const next = serializeLedger(entries);if (existsSync(path) && readFileSync(path, "utf8") === next) return false;diff --git a/src/core/brain/session-mining-surveys.ts b/src/core/brain/session-mining-surveys.tsnew file mode 100644index 00000000..11955e9b--- /dev/null+++ b/src/core/brain/session-mining-surveys.ts@@ -0,0 +1,240 @@+/**+ * Pending mining surveys: what a plan put in front of a model, remembered+ * until the commit that answers it.+ *+ * The documented mining flow is plan, answer, commit - and the commit is+ * not required to relay the plan's `surveyed_turns_hash`. A commit that+ * relays nothing used to hash the session's turns as they stood at COMMIT+ * time, so an import landing between the plan and the commit made a+ * zero-yield commit tombstone turns no model had seen. This store closes+ * that gap without asking the caller for anything: every plan that hands+ * out a mining envelope records the hash of the material it offered, and+ * the commit that answers it clears the record. A zero-yield commit without a relay+ * writes its tombstone only when exactly one distinct material was+ * surveyed since the last commit and the session still holds it. The+ * commit reads the survey first and clears what it read only once its+ * outcome is final, so a commit that fails transiently (an unreadable+ * session, a ledger write that refused) leaves the survey for its retry.+ *+ * Kept apart from the mining ledger on purpose. The ledger fails closed+ * because a silent reset would unsuppress every barren session; a lost+ * survey can only withhold one tombstone (the session stays minable), so+ * the two have different failure economics and different owners - plans+ * write here, commits write there.+ *+ * A corrupt or unknown-schema file still refuses by name rather than+ * reading as empty: an empty read could make a session with two pending+ * surveys look like it had one, which is exactly the ambiguity the store+ * exists to detect.+ */++import { existsSync, readFileSync } from "node:fs";+import { join } from "node:path";++import { atomicWriteFileSync } from "../fs-atomic.ts";+import { DERIVED_STORE_DIR } from "./path-constants.ts";+import { acquireLockSyncWithRetry, ingestLockRemedy } from "./sync-lockfile.ts";+import { isoSecond } from "./time.ts";+import { assertVaultIdentityForWrite } from "./vault-identity.ts";++/** Only survey-store schema this build understands. */+export const SESSION_MINING_SURVEYS_SCHEMA_VERSION = 1 as const;++const SESSION_MINING_SURVEYS_FILE = "session-mining-surveys.json";++/**+ * Distinct hashes kept per session. Past two the answer is already+ * "ambiguous", so the cap only bounds the file; keeping the newest few+ * keeps the record honest about there being several.+ */+export const MAX_SURVEYED_HASHES_PER_SESSION = 8;++/** Sessions kept; the oldest surveys beyond this are dropped at write time. */+export const MAX_SURVEYED_SESSIONS = 512;++/** The remedy every file-level refusal names. */+const CORRUPT_SURVEYS_REMEDY =+ "; to recover, move the file aside or delete it - only pending plan surveys are lost, so " ++ "the next barren commit of an affected session records no tombstone until it is planned again";++/**+ * A store failure in a closed form: the store name plus the errno code,+ * never the raised message, which names the absolute host path. For every+ * surface that answers a caller below local reach (the same form as+ * `closedCollectError`).+ */+export function closedSurveyFailure(err: unknown): string {+ const code = (err as NodeJS.ErrnoException | null)?.code;+ return `session mining surveys store failed (${typeof code === "string" ? code : "error"})`;+}++/** Absolute path of the pending-survey store for a vault. */+export function sessionMiningSurveysPath(vault: string): string {+ return join(vault, DERIVED_STORE_DIR, SESSION_MINING_SURVEYS_FILE);+}++interface SurveyRow {+ readonly hashes: ReadonlyArray<string>;+ readonly surveyedAt: string;+}++/**+ * Session ids are caller data, so every map keyed by one is+ * null-prototype: an id such as `__proto__` is an own key like any other+ * instead of an assignment to the prototype.+ */+type SurveyMap = Record<string, SurveyRow>;++function emptySurveyMap(): SurveyMap {+ return Object.create(null) as SurveyMap;+}++function readSurveys(vault: string): SurveyMap {+ const path = sessionMiningSurveysPath(vault);+ if (!existsSync(path)) return emptySurveyMap();+ let parsed: unknown;+ try {+ parsed = JSON.parse(readFileSync(path, "utf8"));+ } catch (e) {+ throw new Error(+ `session mining surveys file is corrupted JSON: ${path}${CORRUPT_SURVEYS_REMEDY}`,+ {+ cause: e,+ },+ );+ }+ if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) {+ throw new Error(+ `session mining surveys file is not an object: ${path}${CORRUPT_SURVEYS_REMEDY}`,+ );+ }+ const obj = parsed as Record<string, unknown>;+ if (obj["schema_version"] !== SESSION_MINING_SURVEYS_SCHEMA_VERSION) {+ throw new Error(+ `session mining surveys schema_version ${String(obj["schema_version"])} not supported ` ++ `(expected ${SESSION_MINING_SURVEYS_SCHEMA_VERSION}): ${path}${CORRUPT_SURVEYS_REMEDY}`,+ );+ }+ const raw = obj["surveys"];+ if (raw === null || typeof raw !== "object" || Array.isArray(raw)) {+ throw new Error(+ `session mining surveys container is not an object: ${path}${CORRUPT_SURVEYS_REMEDY}`,+ );+ }+ const out = emptySurveyMap();+ for (const [key, value] of Object.entries(raw as Record<string, unknown>)) {+ if (value === null || typeof value !== "object" || Array.isArray(value)) continue;+ const row = value as Record<string, unknown>;+ const hashes = row["hashes"];+ if (!Array.isArray(hashes) || !hashes.every((h) => typeof h === "string" && h.length > 0)) {+ // A damaged row is kept as "several surveys" rather than dropped: a+ // dropped row would read as "no survey", and the commit would then+ // name the absence - but a row reduced to one hash could wrongly+ // attest a single survey. Two placeholder hashes force ambiguity.+ out[key] = { hashes: ["<unreadable>", "<unreadable-2>"], surveyedAt: "" };+ continue;+ }+ const surveyedAt = row["surveyedAt"];+ out[key] = {+ hashes: hashes as string[],+ surveyedAt: typeof surveyedAt === "string" ? surveyedAt : "",+ };+ }+ return out;+}++function writeSurveys(vault: string, surveys: SurveyMap): void {+ const path = sessionMiningSurveysPath(vault);+ const sorted = emptySurveyMap();+ for (const key of Object.keys(surveys).toSorted()) sorted[key] = surveys[key]!;+ const next =+ JSON.stringify(+ { schema_version: SESSION_MINING_SURVEYS_SCHEMA_VERSION, surveys: sorted },+ null,+ 2,+ ) + "\n";+ if (existsSync(path) && readFileSync(path, "utf8") === next) return;+ atomicWriteFileSync(path, next);+}++/**+ * Record that a plan offered `hash` for `sessionId`. A hash already pending+ * for the session is not added twice, so re-planning unchanged material+ * keeps the survey unambiguous.+ */+export function recordMiningSurvey(+ vault: string,+ sessionId: string,+ hash: string,+ now: Date = new Date(),+): void {+ assertVaultIdentityForWrite(vault);+ const path = sessionMiningSurveysPath(vault);+ const key = sessionId.trim();+ const handle = acquireLockSyncWithRetry(path, undefined, { remedy: ingestLockRemedy });+ try {+ const surveys = readSurveys(vault);+ const prior = surveys[key]?.hashes ?? [];+ const hashes = [...prior.filter((h) => h !== hash), hash].slice(+ -MAX_SURVEYED_HASHES_PER_SESSION,+ );+ surveys[key] = { hashes, surveyedAt: isoSecond(now) };+ const keys = Object.keys(surveys);+ if (keys.length > MAX_SURVEYED_SESSIONS) {+ // A damaged row reads with an empty surveyedAt and so goes first.+ // Evicting it is the safe direction: the session then reads as+ // having no survey, which withholds a tombstone rather than writing one.+ const oldestFirst = keys.toSorted(+ (a, b) =>+ surveys[a]!.surveyedAt.localeCompare(surveys[b]!.surveyedAt) || a.localeCompare(b),+ );+ for (const drop of oldestFirst.slice(0, keys.length - MAX_SURVEYED_SESSIONS)) {+ if (drop !== key) delete surveys[drop];+ }+ }+ writeSurveys(vault, surveys);+ } finally {+ handle.release();+ }+}++/**+ * The pending surveys of one session: the distinct hashes plans offered+ * since its last commit, oldest first, or `[]` when none was recorded.+ * Read-only - the commit clears them with {@link clearMiningSurvey} once+ * its outcome is final. Throws on a corrupt or unknown-schema store.+ */+export function readMiningSurvey(vault: string, sessionId: string): ReadonlyArray<string> {+ return readSurveys(vault)[sessionId.trim()]?.hashes ?? [];+}++/**+ * Remove `hashes` from one session's pending surveys - the ones a commit+ * read and answered. A hash a plan added after that read stays pending, so+ * a plan racing the commit is not lost. The row goes once it is empty.+ */+export function clearMiningSurvey(+ vault: string,+ sessionId: string,+ hashes: ReadonlyArray<string>,+): void {+ if (hashes.length === 0) return;+ const path = sessionMiningSurveysPath(vault);+ if (!existsSync(path)) return;+ assertVaultIdentityForWrite(vault);+ const key = sessionId.trim();+ const handle = acquireLockSyncWithRetry(path, undefined, { remedy: ingestLockRemedy });+ try {+ const surveys = readSurveys(vault);+ const row = surveys[key];+ if (row === undefined) return;+ const answered = new Set(hashes);+ const left = row.hashes.filter((h) => !answered.has(h));+ if (left.length === row.hashes.length) return;+ if (left.length === 0) delete surveys[key];+ else surveys[key] = { hashes: left, surveyedAt: row.surveyedAt };+ writeSurveys(vault, surveys);+ } finally {+ handle.release();+ }+}diff --git a/src/core/brain/session-recall.ts b/src/core/brain/session-recall.tsindex abf4ef4c..1f6dde70 100644--- a/src/core/brain/session-recall.ts+++ b/src/core/brain/session-recall.ts@@ -444,11 +444,64 @@ export function listSessionRawTurns(vault: string,sessionId: string,): ReadonlyArray<ExpandedRawTurn> {- return Object.freeze(- sessionRecallRecords(vault, sessionId)- .filter((record) => record.kind === "session_turn")- .map(expandedRawTurn),+ return buildSessionTurnIndex(vault).turnsFor(sessionId);+}++/** Per-session raw turn lists over one read of the continuity store. */+export interface SessionTurnIndex {+ /** Exactly what {@link listSessionRawTurns} answers for `sessionId`. */+ turnsFor(sessionId: string): ReadonlyArray<ExpandedRawTurn>;+}++/**+ * One pass over the continuity store and one lineage build, then any+ * number of per-session turn lists. A caller that walks many sessions+ * (the mining census, the mining ledger's prune) pays the store read once+ * instead of once per session, which is the difference between linear and+ * quadratic in the number of imported sessions. The lineage scope is the+ * one {@link listSessionRawTurns} applies - that function is this index+ * over a single session - so the census and the plan cannot drift in how+ * they scope a session.+ */+export function buildSessionTurnIndex(vault: string): SessionTurnIndex {+ const records = listContinuityRecords(vault).filter(+ (record) => record.kind === "session_turn" || record.kind === "session_summary_node",);+ const links = lineageLinks(vault, records);+ const rootMemo = new Map<string, string>();+ const rootOf = (sid: string): string => {+ let root = rootMemo.get(sid);+ if (root === undefined) {+ root = transitiveRootOf(sid, links);+ rootMemo.set(sid, root);+ }+ return root;+ };+ const segmentsByRoot = new Map<string, string[]>();+ for (const sid of links.keys()) {+ const root = rootOf(sid);+ const segments = segmentsByRoot.get(root);+ if (segments === undefined) segmentsByRoot.set(root, [sid]);+ else segments.push(sid);+ }+ const turnsBySession = new Map<string, ContinuityRecord[]>();+ for (const record of records) {+ if (record.kind !== "session_turn") continue;+ const sid = sessionIdOf(record.payload);+ if (sid === undefined) continue;+ const list = turnsBySession.get(sid);+ if (list === undefined) turnsBySession.set(sid, [record]);+ else list.push(record);+ }+ return {+ turnsFor(sessionId: string): ReadonlyArray<ExpandedRawTurn> {+ const rootId = rootOf(sessionId);+ const scope = new Set<string>([sessionId, rootId, ...(segmentsByRoot.get(rootId) ?? [])]);+ const scoped: ContinuityRecord[] = [];+ for (const sid of scope) scoped.push(...(turnsBySession.get(sid) ?? []));+ return Object.freeze(scoped.sort(compareRecords).map(expandedRawTurn));+ },+ };}export function expandSessionRecall(diff --git a/src/core/brain/sessions/discover.ts b/src/core/brain/sessions/discover.tsindex 2ae79499..06cbbd94 100644--- a/src/core/brain/sessions/discover.ts+++ b/src/core/brain/sessions/discover.ts@@ -97,6 +97,14 @@ export interface SessionLedgerEntry {readonly runtime: SessionRuntimeId;/** When this build imported it, to the second. */readonly imported_at: string;+ /**+ * The portable session id the import filed its turns under, present+ * only when the import pinned one explicitly (`--recall-session-id`);+ * absent, the id is the log's basename, as `sessionRefIdentity` derives+ * it. The mining census joins on this, so an explicitly-named import's+ * mining state is not misread as a refused session.+ */+ readonly session_id?: string;}/** Absolute log path β what was recorded about it. */@@ -163,6 +171,8 @@ export interface DiscoverSessionsOptions {export interface SessionImportRecord {readonly path: string;readonly runtime: SessionRuntimeId;+ /** The explicit recall session id the import used, when one was pinned. */+ readonly sessionId?: string;}/** The host this process is running on, when the caller named none. */@@ -221,11 +231,13 @@ function readEntry(value: unknown): SessionLedgerEntry | null {const hash = row["hash"];const runtime = row["runtime"];const importedAt = row["imported_at"];+ const sessionId = row["session_id"];if (typeof hash !== "string" || !isSessionRuntimeId(runtime)) return null;return {hash,runtime,imported_at: typeof importedAt === "string" ? importedAt : "",+ ...(typeof sessionId === "string" && sessionId !== "" ? { session_id: sessionId } : {}),};}@@ -280,6 +292,9 @@ export function recordSessionImports(hash: hashFile(record.path),runtime: record.runtime,imported_at: stamp,+ ...(record.sessionId !== undefined && record.sessionId.trim() !== ""+ ? { session_id: record.sessionId.trim() }+ : {}),};}for (const path of Object.keys(entries)) {diff --git a/src/core/brain/sessions/mining-census.ts b/src/core/brain/sessions/mining-census.tsindex f365fe8d..4f9eeba7 100644--- a/src/core/brain/sessions/mining-census.ts+++ b/src/core/brain/sessions/mining-census.ts@@ -30,8 +30,11 @@* census joins on that basename, so a session whose log moved to another* absolute path (a rotated store, another machine's layout) still joins.* No source log is opened or re-hashed here: the mined-material hash runs- * over the continuity turns the mining lane itself reads, which is what- * makes the census answerable inside a SessionStart hook's budget. The+ * over the continuity turns the mining lane itself reads, and the store is+ * read ONCE for the whole census ({@link buildSessionTurnIndex}) - a+ * per-session listing would re-read the whole store per session, which is+ * quadratic in the imported backlog. That is what makes the census+ * answerable inside a SessionStart hook's budget. The* census reads only - neither ledger is rewritten, and a report that* quietly rewrote the state it reports on would not be a report.*/@@ -47,7 +50,7 @@ import {type SessionMiningEntry,} from "../session-mining-ledger.ts";import type { Safeguard } from "../safeguard.ts";-import { listSessionRawTurns } from "../session-recall.ts";+import { buildSessionTurnIndex, type SessionTurnIndex } from "../session-recall.ts";import { readSessionLedger } from "./discover.ts";import type { SessionRuntimeId } from "../../runtime/host-facts.ts";@@ -86,12 +89,16 @@ export interface SessionMiningCensus {export interface BuildMiningCensusOptions {/*** Cooperative deadline, checked once per imported session. The census is- * vault-side and never hashes a source log, but a vault with tens of- * thousands of imported sessions still spends one turn listing per- * session; a caller with a host deadline (the SessionStart hook) passes- * its ceiling here so the read stops in time.+ * vault-side, never hashes a source log and reads the store once, but a+ * caller with a host deadline (the SessionStart hook) still passes its+ * ceiling here so the read stops in time on an outsized vault.*/readonly safeguard?: Safeguard;+ /**+ * The store read to classify against. Default: one+ * {@link buildSessionTurnIndex} pass for this census.+ */+ readonly turns?: SessionTurnIndex;}/**@@ -107,6 +114,7 @@ export function buildMiningCensus(const importedEntries = readSessionLedger(vault).entries;const miningEntries = readSessionMiningLedger(vault).entries;const boundary = buildCaptureBoundary(vault);+ const turns = opts.turns ?? buildSessionTurnIndex(vault);// One identity can sit under several absolute paths (the same basename// imported from two machines' layouts); its state is a fact about the@@ -115,7 +123,7 @@ export function buildMiningCensus(const stateOf = (identity: string): SessionDistillationState | "refused" => {const cached = stateByIdentity.get(identity);if (cached !== undefined) return cached;- const state = classifySession(vault, boundary, miningEntries, identity);+ const state = classifySession(vault, boundary, turns, miningEntries, identity);stateByIdentity.set(identity, state);return state;};@@ -142,16 +150,11 @@ export function buildMiningCensus(for (const [path, entry] of Object.entries(importedEntries)) {opts.safeguard?.checkpoint();- // The join identity is the portable sessionRefIdentity basename - what- // the mining lane keys its tombstone and every signal's session_ref on- // when the import pinned no explicit id. The import ledger's entry- // does not carry the RESOLVED identity (an explicit --recall-session-id- // import files its turns, tombstone and session_ref under the recall- // id, not the basename), so such an import's mining state is invisible- // to this join until the resolved id is recorded in the import ledger- // itself; classifying the basename as refused here stays factually- // right - the plan really would refuse it.- const identity = basename(path);+ // The join identity is the portable sessionRefIdentity - what the+ // mining lane keys its tombstone and every signal's session_ref on:+ // the explicit recall id the import recorded when it pinned one+ // (--recall-session-id), else the log's basename.+ const identity = entry.session_id ?? basename(path);const state = stateOf(identity);const row = rowOf(entry.runtime);row.imported += 1;@@ -194,16 +197,17 @@ export function buildMiningCensus(* zero user turns). The selection is the plan's own shared* {@link selectMinedTurns} - user role, unsuppressed text, trimmed and* non-empty, the turn's stored timestamp verbatim - so this census hashes- * exactly what the plan would mine and cannot drift from it. `boundary` is- * injectable so a caller walking many sessions builds it once.+ * exactly what the plan would mine and cannot drift from it. `boundary` and+ * `index` are injectable so a caller walking many sessions builds each once.*/export function minedTurnMaterial(vault: string,sessionId: string,boundary: CaptureBoundary = buildCaptureBoundary(vault),+ index?: SessionTurnIndex,): ReadonlyArray<MinedTurnMaterial> | null {if (boundary.sessionDecision(sessionId) !== "capture") return null;- const turns = listSessionRawTurns(vault, sessionId);+ const turns = (index ?? buildSessionTurnIndex(vault)).turnsFor(sessionId);if (turns.length === 0) return null;const mined = selectMinedTurns(boundary, turns);return mined.length === 0 ? null : mined;@@ -216,10 +220,11 @@ export function minedTurnMaterial(function classifySession(vault: string,boundary: ReturnType<typeof buildCaptureBoundary>,+ turns: SessionTurnIndex,miningEntries: Readonly<Record<string, SessionMiningEntry>>,identity: string,): SessionDistillationState | "refused" {- const material = minedTurnMaterial(vault, identity, boundary);+ const material = minedTurnMaterial(vault, identity, boundary, turns);if (material === null) return "refused";const entry = miningEntries[identity];if (entry === undefined || entry.minedContentHash !== minedTurnsHash(material)) {diff --git a/src/core/brain/source-cleanup.ts b/src/core/brain/source-cleanup.tsindex a35e78e2..1f89a125 100644--- a/src/core/brain/source-cleanup.ts+++ b/src/core/brain/source-cleanup.ts@@ -142,6 +142,8 @@ export interface NoteDerivationEntry extends SourceCleanupEntry {export interface SourceCleanupFailure {readonly path: string;readonly message: string;+ /** The errno code the removal failed with, when the error carried one. */+ readonly code?: string;}export interface SourceCleanupPlan {@@ -809,7 +811,12 @@ export function deleteBySource(try {if (removeFile(vault, relPath)) deleted.push(relPath);} catch (err) {- failed.push({ path: relPath, message: (err as Error)?.message ?? String(err) });+ const code = (err as NodeJS.ErrnoException | null)?.code;+ failed.push({+ path: relPath,+ message: (err as Error)?.message ?? String(err),+ ...(typeof code === "string" ? { code } : {}),+ });}};diff --git a/src/core/brain/storage/forecast.ts b/src/core/brain/storage/forecast.tsindex 6f97a999..b68e9b6b 100644--- a/src/core/brain/storage/forecast.ts+++ b/src/core/brain/storage/forecast.ts@@ -368,6 +368,8 @@ export interface SampleStorageUsageOptions {* absent, the sampler measures as it always did.*/readonly usage?: StorageUsage;+ /** The explicit machine-config file the census resolves the store from. */+ readonly configPath?: string;}export interface SampleStorageUsageResult {@@ -403,7 +405,11 @@ export function sampleStorageUsage(): SampleStorageUsageResult {const runId = opts.runId ?? mintRunId();const usage =- opts.usage ?? measureStorageUsage(vault, opts.now !== undefined ? { now: opts.now } : {});+ opts.usage ??+ measureStorageUsage(vault, {+ ...(opts.now !== undefined ? { now: opts.now } : {}),+ ...(opts.configPath !== undefined ? { configPath: opts.configPath } : {}),+ });let alreadyRecorded = false;try {alreadyRecorded = listMetrics(vault, { surface: STORAGE_USAGE_METRIC_SURFACE }).some(diff --git a/src/core/brain/storage/gate-config.ts b/src/core/brain/storage/gate-config.tsnew file mode 100644index 00000000..8aedc313--- /dev/null+++ b/src/core/brain/storage/gate-config.ts@@ -0,0 +1,98 @@+/**+ * The storage policy the ingest admission walls decide under, read from+ * ONLY the `storage:` block of `_brain.yaml`. Shared by the MCP ingest+ * tool and the capture inbox drain so the two walls cannot fail in+ * opposite directions on the same broken block.+ */++import { existsSync, readFileSync } from "node:fs";++import { brainConfigPath } from "../paths.ts";+import { BRAIN_CONFIG_SUPPORTED_VERSIONS } from "../policy/defaults.ts";+import { BrainConfigError } from "../policy/errors.ts";+import { KeyIndexCollector, type BlockParseContext } from "../policy/key-index.ts";+import {+ BRAIN_STORAGE_DEFAULTS,+ parseStorageBlock,+ resolveStorage,+} from "../policy/blocks/storage.ts";+import type { BrainConfig, ResolvedBrainStorageConfig } from "../types.ts";+import { parseBrainYaml, type ParsedBlock } from "../yaml-parse.ts";++/**+ * A broken `_brain.yaml` the admission gate cannot decide under. Its prose+ * is built from a relative config label, never the config's host path, so+ * the refusal can cross to a remote caller (see {@link loadGateStorageConfig}).+ */+export class StorageGateConfigError extends Error {+ constructor(message: string) {+ super(message);+ this.name = "StorageGateConfigError";+ }+}++/**+ * The storage policy the admission gate decides under, read from ONLY the+ * `storage:` block of `_brain.yaml` (review M1).+ *+ * The gate's decision needs no other block, but the whole-file loader+ * validated all of them, so a typo in an unrelated block turned every+ * ingest into an internal error naming the config's host path - including+ * on vaults whose operator never configured a cap, where the documented+ * default is that nothing refuses. This reader answers the gate's actual+ * question: an absent config or block is the defaults (no cap, no gate),+ * and the rest of the file is never validated here.+ *+ * A file that cannot be read or parsed, an unsupported `schema_version`,+ * or a `storage:` block that does not validate is a policy this gate+ * cannot see through, and silently reading it as "no cap" would run+ * writes past an operator's bound. Those refuse BY NAME: the error+ * carries field-level prose built from a relative config label, and the+ * MCP handler maps it onto the registered `config_invalid` envelope -+ * never a path below local reach. Both admission walls read through here+ * (`brain_ingest_source` and the capture inbox drain's+ * {@link evaluateIngestAdmission}), so they apply one rule: refuse on a+ * storage block they cannot see through, ignore every other block.+ */+export function loadGateStorageConfig(vault: string): ResolvedBrainStorageConfig {+ const path = brainConfigPath(vault);+ if (!existsSync(path)) return BRAIN_STORAGE_DEFAULTS;+ let text: string;+ try {+ text = readFileSync(path, "utf8");+ } catch (err) {+ // Only the errno code is kept: a read-failure message embeds the+ // absolute path the call failed on.+ const code = (err as NodeJS.ErrnoException).code ?? "unreadable";+ throw new StorageGateConfigError(`_brain.yaml could not be read (${code})`);+ }+ let parsed: ParsedBlock;+ try {+ parsed = parseBrainYaml(text);+ } catch (err) {+ // The parser's messages name line numbers, never paths.+ throw new StorageGateConfigError(`_brain.yaml could not be parsed: ${(err as Error).message}`);+ }+ const version = parsed["schema_version"];+ if (typeof version !== "number" || !BRAIN_CONFIG_SUPPORTED_VERSIONS.includes(version)) {+ throw new StorageGateConfigError(+ `_brain.yaml: unsupported schema_version: ${JSON.stringify(version ?? null)}`,+ );+ }+ const ctx: BlockParseContext = {+ obj: parsed,+ // A relative label, never the host path: the field errors this context+ // raises become the refusal's prose.+ source: "_brain.yaml",+ warnings: [],+ keys: new KeyIndexCollector(),+ };+ try {+ // Only the storage block is parsed and resolved; `resolveStorage`+ // reads nothing else off the config.+ return resolveStorage({ storage: parseStorageBlock(ctx) } as BrainConfig);+ } catch (err) {+ if (err instanceof BrainConfigError) throw new StorageGateConfigError(err.message);+ throw err;+ }+}diff --git a/src/core/brain/storage/gate.ts b/src/core/brain/storage/gate.tsindex b1a279e7..c2e211ca 100644--- a/src/core/brain/storage/gate.ts+++ b/src/core/brain/storage/gate.ts@@ -27,8 +27,7 @@* closing. Only a MEASURED breach skips.*/-import { BrainConfigError } from "../policy/errors.ts";-import { loadStorageConfigSafe } from "../policy/load.ts";+import { loadGateStorageConfig, StorageGateConfigError } from "./gate-config.ts";import type { StorageForecastTier } from "./forecast.ts";import { STORAGE_FORECAST_TIER, forecastStorageUsage } from "./forecast.ts";import { measureStorageUsage, type StorageUsage } from "./usage.ts";@@ -97,6 +96,12 @@ export interface EvaluateCapacityGateOptions {readonly criticalDays?: number;/** Wall clock for the fresh census's stamp; defaults to the current time. */readonly now?: Date;+ /**+ * The explicit machine-config file the caller resolved from, so a+ * `search_db_path` in it moves the measured store the same way it moves+ * the indexer (`measureStorageUsage`'s option). Absent, the default file.+ */+ readonly configPath?: string;}/**@@ -115,7 +120,10 @@ export function evaluateCapacityGate(return { decision: CAPACITY_GATE_DECISION.run, tier: STORAGE_FORECAST_TIER.unbounded };}- const fresh = measureStorageUsage(vault, opts.now !== undefined ? { now: opts.now } : {});+ const fresh = measureStorageUsage(vault, {+ ...(opts.now !== undefined ? { now: opts.now } : {}),+ ...(opts.configPath !== undefined ? { configPath: opts.configPath } : {}),+ });if (!fresh.complete) {return {decision: CAPACITY_GATE_DECISION.run,@@ -166,24 +174,26 @@ export function evaluateCapacityGate(export interface EvaluateIngestAdmissionOptions {/** Wall clock for the fresh census; defaults to the current time. */readonly now?: Date;+ /** The explicit machine-config file; see {@link EvaluateCapacityGateOptions.configPath}. */+ readonly configPath?: string;}-/** One admission evaluation: a gate decision, or why there is none. */+/** One admission evaluation: a gate decision, or the refusal that replaces one. */export interface IngestAdmission {/*** The gate decision, when a configured cap was measured. `null` means- * no wall stands this pass - an unbounded vault, or a storage block- * that could not be read (see {@link reason}).+ * no measurement was taken - an unbounded vault, or a storage block the+ * wall cannot see through (see {@link configRefusal}).*/readonly decision: CapacityGateDecision | null;/**- * Why no decision exists, when the wall was skipped rather than- * passed: a `_brain.yaml` that fails validation names the failing- * field here. Callers that have a warning channel write it; a wall- * that pretended to stand behind an unreadable config would refuse- * every capture forever on a typo in an unrelated block.+ * Present when the `storage:` block (or the file it sits in) cannot be+ * read, parsed or validated: the wall refuses ingest, naming the field,+ * exactly as `brain_ingest_source` refuses with `config_invalid`. Every+ * other block is never read here, so a typo elsewhere cannot wedge the+ * wall either way.*/- readonly reason?: string;+ readonly configRefusal?: string;}/**@@ -198,11 +208,12 @@ export interface IngestAdmission {* a drain, and a capture the wall would admit must not be refused by* one.*- * A config that cannot be VALIDATED is not a vault without room: the- * check fails open with the reason carried (see {@link IngestAdmission}),- * because an ingest wall is a convenience-surface gate, not the commit- * path - the core ingest reads no config, and a malformed unrelated- * block must not wedge the capture inbox shut.+ * The policy is read from the `storage:` block alone+ * ({@link loadGateStorageConfig}, the reader `brain_ingest_source` uses),+ * so the two walls apply one rule: a malformed unrelated block is never+ * read and cannot wedge the capture inbox shut, while a storage block the+ * wall cannot see through refuses ingest by name - reading it as "no cap"+ * would run writes past an operator's bound.*/export function evaluateIngestAdmission(vault: string,@@ -210,18 +221,16 @@ export function evaluateIngestAdmission(): IngestAdmission {let storage;try {- storage = loadStorageConfigSafe(vault);+ storage = loadGateStorageConfig(vault);} catch (err) {- if (!(err instanceof BrainConfigError)) throw err;- return {- decision: null,- reason: `storage admission not evaluated: ${err.message}`,- };+ if (!(err instanceof StorageGateConfigError)) throw err;+ return { decision: null, configRefusal: err.message };}if (storage.cap_bytes === null) return { decision: null };return {decision: evaluateCapacityGate(vault, {now: opts.now ?? new Date(),+ ...(opts.configPath !== undefined ? { configPath: opts.configPath } : {}),capBytes: storage.cap_bytes,warnPercent: storage.warn_percent,criticalPercent: storage.critical_percent,diff --git a/src/core/brain/trust/operator-summary.ts b/src/core/brain/trust/operator-summary.tsindex f210ae6d..6b0bce90 100644--- a/src/core/brain/trust/operator-summary.ts+++ b/src/core/brain/trust/operator-summary.ts@@ -26,6 +26,7 @@ import { toPosix } from "../../path-safety.ts";import { runDoctor, type RunDoctorResult } from "../doctor.ts";import type { DreamRunSummary } from "../dream.ts";import {+ closedCollectError,collectMaintenanceActions,type MaintenanceCollectSignal,} from "../maintenance/collect.ts";@@ -155,6 +156,7 @@ export function buildOperatorSummary(opts.topActionsN ?? DEFAULT_TOP_ACTIONS_N,opts.keepAction,collectErrors,+ opts.readable !== undefined,);const trust = computeTrustVerdict({@@ -346,10 +348,16 @@ function safeTopActions(topN: number,keepAction: BuildOperatorSummaryOptions["keepAction"],collectErrors: { scan: MaintenanceCollectSignal; message: string }[],+ restricted: boolean,): ReadonlyArray<ActionItem> {try {const all = collectMaintenanceActions(vault, {- onError: (e) => collectErrors.push({ scan: e.signal, message: e.message }),+ // Below local reach the raised message (which can carry a host+ // path) is replaced by the closed scan-and-code form.+ onError: (e) => {+ const shown = restricted ? closedCollectError(e) : e;+ collectErrors.push({ scan: shown.signal, message: shown.message });+ },});const kept = keepAction === undefined ? all : all.filter(keepAction);return Object.freeze(kept.slice(0, Math.max(0, topN)));diff --git a/src/core/brain/trust/retrieval-gate.ts b/src/core/brain/trust/retrieval-gate.tsindex 16904791..56cddf9d 100644--- a/src/core/brain/trust/retrieval-gate.ts+++ b/src/core/brain/trust/retrieval-gate.ts@@ -31,7 +31,9 @@* classification, excludes nothing and records every would-be refusal* with its structural reasons into the receipts, and `off` - the default* - does not run the gate at all, keeping ranking and the outcome shape- * byte-identical. The legacy boolean enable flag maps `true` to `shadow`.+ * byte-identical. The legacy boolean enable flag maps `true` to `enforce`,+ * the excluding behavior it always had, so an upgrade never quietly stops+ * excluding; `shadow` is reached only by naming it.*/import type { FrontmatterMap } from "../../types.ts";@@ -105,9 +107,10 @@ export const SEARCH_TRUST_GATE_LEGACY_CONFIG = "search_trust_gate_enabled";* config; an invalid value is a misconfiguration refused with a* `SearchError` naming the key actually in force - never silently* defaulted); absent the mode key, the legacy boolean maps `true` to- * `shadow` and `false` to `off`; with neither set the default is `off`.- * Shadow is non-destructive precisely so enablement can be the first- * step: a vault that wants the excluding behavior names `enforce`.+ * `enforce` - the excluding behavior it switched on before the mode+ * existed, so a vault that opted in keeps excluding across the upgrade -+ * and `false` to `off`; with neither set the default is `off`. Measuring+ * first is an explicit `search_trust_gate_mode: shadow`.*/export function resolveSearchTrustGateMode(input: {readonly env?: NodeJS.ProcessEnv;@@ -126,15 +129,32 @@ export function resolveSearchTrustGateMode(input: {`${inForce} must be 'off', 'shadow' or 'enforce', got '${raw}'`,);}+ return resolveSearchTrustGateLegacyEnabled({ env, config })+ ? SEARCH_TRUST_GATE_MODE.enforce+ : SEARCH_TRUST_GATE_MODE.off;+}++/**+ * The legacy boolean enable flag, env over machine config, default+ * false. An invalid value is refused naming the key actually in force,+ * so an operator who set the env var is not sent to a config key they+ * never wrote.+ */+export function resolveSearchTrustGateLegacyEnabled(input: {+ readonly env?: NodeJS.ProcessEnv;+ readonly config?: Readonly<Record<string, string>>;+}): boolean {+ const env = input.env ?? {};const legacy = envOrConfig(env,- config,+ input.config ?? {},SEARCH_TRUST_GATE_LEGACY_ENV,SEARCH_TRUST_GATE_LEGACY_CONFIG,);- return legacy !== null && parseBool(legacy, false, SEARCH_TRUST_GATE_LEGACY_CONFIG)- ? SEARCH_TRUST_GATE_MODE.shadow- : SEARCH_TRUST_GATE_MODE.off;+ const inForce = env[SEARCH_TRUST_GATE_LEGACY_ENV]+ ? SEARCH_TRUST_GATE_LEGACY_ENV+ : SEARCH_TRUST_GATE_LEGACY_CONFIG;+ return parseBool(legacy, false, inForce);}export interface RetrievalTrustVerdict {diff --git a/src/core/maintenance/ensure-current.ts b/src/core/maintenance/ensure-current.tsindex 34974c7a..0eaccff6 100644--- a/src/core/maintenance/ensure-current.ts+++ b/src/core/maintenance/ensure-current.ts@@ -23,13 +23,10 @@ import { defaultConfigPath } from "../config.ts";import { o2bCommand } from "./o2b-command.ts";import { brainConfigPath } from "../brain/paths.ts";import { resolveSearchConfig } from "../search/index.ts";-import { reindexVault } from "../search/indexer.ts";-import { CHUNKER_VERSION } from "../search/chunker.ts";-import { LATEST_SCHEMA_VERSION, readSchemaVersion } from "../search/schema.ts";-import { CHUNKER_VERSION_STATE_KEY, LAST_INDEXED_AT_STATE_KEY } from "../search/store/state.ts";+import { healIndex, inspectIndex } from "../search/self-heal.ts";+import { LAST_INDEXED_AT_STATE_KEY } from "../search/store/state.ts";import { maybeFreshenIndex, type FreshenDecision } from "../search/freshen.ts";import { isWriterLockHeld } from "../search/store/writer-lock.ts";-import type { ResolvedSearchConfig } from "../search/types.ts";import {mintSelfHealRunId,recordSelfHealSpawn,@@ -89,6 +86,11 @@ export interface EnsureCurrentResult {readonly brainUpgradeOutcome: SelfHealUpgradeOutcome | null;/** A search reindex was started (background) or completed (foreground). */readonly reindexTriggered: boolean;+ /**+ * In the foreground, the index was on an older schema and was migrated+ * in place, keeping its stored embeddings, so no reindex was needed.+ */+ readonly indexUpgraded: boolean;/*** In `background` mode, what was done about a reindex this run found* necessary. `null` when none was needed, or when the run was in the@@ -125,20 +127,6 @@ function message(e: unknown): string {return e instanceof Error ? (e.message ?? String(e)) : String(e);}-/**- * Whether the index holds chunks cut by older chunking rules: documents- * are present and the recorded CHUNKER_VERSION is not this binary's (or- * was never recorded). An unchanged file is never re-chunked by an- * incremental run, so only a rebuild replaces those chunks.- */-function chunksAreStale(db: Database): boolean {- const row = db- .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ? LIMIT 1")- .get(CHUNKER_VERSION_STATE_KEY);- if (row?.value === String(CHUNKER_VERSION)) return false;- return db.query<{ one: number }, []>("SELECT 1 AS one FROM documents LIMIT 1").get() !== null;-}-/** The index's `last_indexed_at`, or null when it cannot be read. */function readLastIndexedAt(dbPath: string): string | null {let db: Database;@@ -160,27 +148,6 @@ function readLastIndexedAt(dbPath: string): string | null {}}-/**- * Cheap, read-only check: is the search index absent, on a stale schema,- * or chunked by older rules?- */-function indexNeedsRebuild(config: ResolvedSearchConfig): boolean {- if (!existsSync(config.dbPath)) return true;- let db: Database;- try {- db = new Database(config.dbPath, { readonly: true });- } catch {- return true; // unreadable -> rebuild- }- try {- return readSchemaVersion(db) !== LATEST_SCHEMA_VERSION || chunksAreStale(db);- } catch {- return true; // corrupt / non-OSB file -> rebuild- } finally {- closeDatabase(db);- }-}-/*** Start the managed-file upgrade in a detached worker, unless a recorded* failure is cooling down or another worker has the vault.@@ -361,6 +328,7 @@ export async function ensureVaultCurrent(let brainUpgradeSpawn: SelfHealUpgradeSpawnDecision | null = null;let brainUpgradeOutcome: SelfHealUpgradeOutcome | null = null;let reindexTriggered = false;+ let indexUpgraded = false;let reindexSpawn: SelfHealSpawnDecision | null = null;let freshen: FreshenDecision | null = null;@@ -372,6 +340,7 @@ export async function ensureVaultCurrent(brainUpgradeSpawn: null,brainUpgradeOutcome: null,reindexTriggered: false,+ indexUpgraded: false,reindexSpawn: null,freshen: null,skipped: "not-initialized",@@ -398,28 +367,38 @@ export async function ensureVaultCurrent(errors.push(`brain-upgrade: ${message(e)}`);}- // 2. Search index - rebuild if absent, on a stale schema, or chunked by- // older rules.+ // 2. Search index - an older schema is migrated in place (the stored+ // embeddings survive); an absent, unreadable or newer index, or one+ // chunked by older rules, is rebuilt (#256). In the background the+ // detached self-heal child applies the same policy.try {const configPath = opts.configPath ?? defaultConfigPath();const config = resolveSearchConfig({ vault, configPath });- if (indexNeedsRebuild(config)) {+ // 3. A current index that has gone stale: catch it up in the+ // background so the session's first read is not the one that pays.+ const freshenIfStale = (): void => {+ freshen = maybeFreshenIndex(config, {+ lastIndexedAt: readLastIndexedAt(config.dbPath),+ ...(opts.freshenSpawn !== undefined ? { spawn: opts.freshenSpawn } : {}),+ });+ };+ if (inspectIndex(config) !== "current") {if (background) {reindexSpawn = startSelfHealReindex(vault, configPath, config.dbPath, errors);// A run that declined to spawn started nothing, and saying it did// would report a rebuild that is not this run's.reindexTriggered = reindexSpawn === SELF_HEAL_SPAWN.spawned;} else {- await reindexVault(config);- reindexTriggered = true;+ const healed = await healIndex(config);+ reindexTriggered = healed.kind === "rebuilt";+ indexUpgraded = healed.kind === "migrated";+ // An in-place upgrade (or a heal another process finished) touches+ // no document, so files changed since the last index run are still+ // stale - the full rebuild this replaced used to pick them up.+ if (!reindexTriggered) freshenIfStale();}} else {- // 3. A current index that has gone stale: catch it up in the- // background so the session's first read is not the one that pays.- freshen = maybeFreshenIndex(config, {- lastIndexedAt: readLastIndexedAt(config.dbPath),- ...(opts.freshenSpawn !== undefined ? { spawn: opts.freshenSpawn } : {}),- });+ freshenIfStale();}} catch (e) {errors.push(`search-reindex: ${message(e)}`);@@ -430,6 +409,7 @@ export async function ensureVaultCurrent(brainUpgradeSpawn,brainUpgradeOutcome,reindexTriggered,+ indexUpgraded,reindexSpawn,freshen,skipped: "",diff --git a/src/core/search/explain-envelope.ts b/src/core/search/explain-envelope.tsindex 09d17332..5bc32f16 100644--- a/src/core/search/explain-envelope.ts+++ b/src/core/search/explain-envelope.ts@@ -80,10 +80,11 @@ export const RETRIEVAL_DECISION_TRACE_KEY = "retrieval_decision_trace";export const MEMORY_TRUST_ASSESSMENT_KEY = "memory_trust_assessment";/**- * Response key carrying the gate's resolved mode. Present only when the- * receipts name one - the shadow mode stamps its receipts, whose- * serialization is how the envelope names the mode; enforce keeps- * today's envelope exactly, and off has no receipts to name anything.+ * Response key carrying the gate's resolved mode, present whenever the+ * gate ran. The shadow mode stamps its receipts; enforce keeps the+ * pre-mode receipt shape, so a stampless receipt names enforce - receipts+ * exist only when the gate ran in one of the two active modes. Off has no+ * receipts and so no mode key.*/export const RETRIEVAL_TRUST_GATE_MODE_KEY = "retrieval_trust_gate_mode";@@ -169,9 +170,9 @@ export function explainEnvelope(: {[RETRIEVAL_DECISION_TRACE_KEY]: trace,[MEMORY_TRUST_ASSESSMENT_KEY]: assessment,- // The resolved mode, named only when the receipts carry one- // (shadow stamps them; enforce and off name nothing).- ...(trace.mode !== undefined ? { [RETRIEVAL_TRUST_GATE_MODE_KEY]: trace.mode } : {}),+ // The resolved mode of the gate that built these receipts:+ // shadow stamps them, enforce leaves them stampless.+ [RETRIEVAL_TRUST_GATE_MODE_KEY]: trace.mode ?? "enforce",};if (timing === undefined) return receipts;return {diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex c0e013d1..216ba71e 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -31,6 +31,8 @@ import {import { loadRerankRegistry, expandRegisteredRerankProvider } from "./rerank/registry.ts";import { decisionModelModeFor, resolveDecisionModelConfig } from "../decision-model/config.ts";import {+ isSearchTrustGateMode,+ resolveSearchTrustGateLegacyEnabled,resolveSearchTrustGateMode,SEARCH_TRUST_GATE_MODE,} from "../brain/trust/retrieval-gate.ts";@@ -652,6 +654,18 @@ function validateResolvedConfig(config: ResolvedSearchConfig, extraBodySource: s}validateWeight(config.keywordWeight, "search_keyword_weight");validateWeight(config.semanticWeight, "search_semantic_weight");+ // A mode can arrive through an override (a bench pin) that skipped the+ // key resolution, so the merged value is checked here too.+ const gateMode: unknown = config.recall.searchTrustGateMode;+ if (+ gateMode !== undefined &&+ !(typeof gateMode === "string" && isSearchTrustGateMode(gateMode))+ ) {+ throw new SearchError(+ "INVALID_INPUT",+ `search_trust_gate_mode must be 'off', 'shadow' or 'enforce', got '${String(gateMode)}'`,+ );+ }if (config.keywordWeight + config.semanticWeight > 1.0 + 1e-9) {throw new SearchError("INVALID_INPUT",@@ -1313,16 +1327,13 @@ export function resolveSearchConfig(opts: {false,"search_metadata_boost_gate",);- const retrievalTrustGateEnabled = parseBool(- envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE", "search_trust_gate_enabled"),- false,- "search_trust_gate_enabled",- );+ const retrievalTrustGateEnabled = resolveSearchTrustGateLegacyEnabled({ env, config });// The gate's mode (session-mining-economy, t_05a0eaa4): an explicit// `search_trust_gate_mode` wins (invalid values refused naming the key// in force, `parseFusionMode` discipline), else the legacy boolean- // above maps true to shadow and false to off. The refusal for a bad- // legacy boolean is the `parseBool` one above, naming that key.+ // above maps true to enforce - the excluding behavior it always had -+ // and false to off. A bad legacy boolean is refused above, naming the+ // key in force.const searchTrustGateMode = resolveSearchTrustGateMode({ env, config });const supersedeFadeEnabled = parseBool(envOrConfig(diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsindex 00dc04cb..a7691022 100644--- a/src/core/search/indexer.ts+++ b/src/core/search/indexer.ts@@ -16,6 +16,7 @@import { accessSync, constants, existsSync, mkdirSync, readFileSync, statSync } from "node:fs";import { basename, dirname } from "node:path";+import { Database } from "bun:sqlite";import {resolveSemanticCapability,@@ -1285,6 +1286,24 @@ export async function reindexVault(config: ResolvedSearchConfig,opts?: IndexVaultOptions,): Promise<IndexStats> {+ const stats = await reindexVaultUnless(config, null, opts);+ // No predicate, so the rebuild always runs.+ return stats!;+}++/**+ * {@link reindexVault} that re-checks, once it HOLDS the writer lock,+ * whether the rebuild is still wanted: `stillCurrent` returning true+ * abandons it before anything is built or swapped, and the call returns+ * `null`. The self-heal paths pass their inspection here, so a second+ * process that queued behind a rebuild another process just finished does+ * not rebuild again over it (and over its `.bak`).+ */+export async function reindexVaultUnless(+ config: ResolvedSearchConfig,+ stillCurrent: (() => boolean) | null,+ opts?: IndexVaultOptions,+): Promise<IndexStats | null> {// The rebuild owns the run's counter from before the lock wait to after// the swap, and hands it to the staging build rather than letting that// build open one of its own. Two reasons, and they are the same reason@@ -1296,14 +1315,15 @@ export async function reindexVault(// `dbPath`), the command exited non-zero after a terminator that said// the run had finished.const progress = progressCounter(OPERATION.reindex, opts?.onProgress);- return await withProgressAsync(progress, () => reindexInto(config, progress, opts));+ return await withProgressAsync(progress, () => reindexInto(config, progress, stillCurrent, opts));}async function reindexInto(config: ResolvedSearchConfig,progress: ProgressCounter,+ stillCurrent: (() => boolean) | null,opts?: IndexVaultOptions,-): Promise<IndexStats> {+): Promise<IndexStats | null> {const newPath = config.dbPath + ".new";const bakPath = config.dbPath + ".bak";@@ -1327,6 +1347,7 @@ async function reindexInto(const release = await acquireWriterLock(config.dbPath);progress.advance(INDEX_STAGE.lock);try {+ if (stillCurrent !== null && stillCurrent()) return null;// Build into the temp file with an override config.const tempConfig: ResolvedSearchConfig = Object.freeze({...config,@@ -1384,15 +1405,27 @@ async function reindexInto(// restoreFromBakIfMissing in store.ts), so it cannot restore the stale// `.bak` over the freshly built index; a genuine crash leaves the lock// stale and the .bak restore on the next Store.open recovers.+ //+ // A `.bak` holding more stored embeddings than the live index is kept:+ // it is the only copy of vectors a keyword-only rebuild cannot+ // recompute, and replacing it with a vector-less live index (two+ // self-heal rebuilds in a row) would discard them while the rebuild+ // notice still points at `.bak`. The live index is parked beside it+ // for the swap and removed once the swap succeeds; a crash between the+ // renames leaves it at `.old`, which the next Store.open restores in+ // preference to the older `.bak` (restoreFromBakIfMissing).progress.start(INDEX_STAGE.swap, 1);- tryUnlink(bakPath);- tryRename(config.dbPath, bakPath); // no-op (ENOENT) on fresh reindex+ const keepBak = countStoredEmbeddings(bakPath) > countStoredEmbeddings(config.dbPath);+ const parkPath = keepBak ? config.dbPath + ".old" : bakPath;+ tryUnlink(parkPath);+ tryRename(config.dbPath, parkPath); // no-op (ENOENT) on fresh reindextry {renameWithRetry(newPath, config.dbPath); // must succeed β `newPath` was just built} catch (swapError) {- restoreLiveFromBak(bakPath, config.dbPath);+ restoreLiveFromBak(parkPath, config.dbPath);throw swapError;}+ if (keepBak) tryUnlink(parkPath);progress.advance(INDEX_STAGE.swap);return stats;} finally {@@ -1534,6 +1567,28 @@ async function withStagingStore(}}+/**+ * Stored vectors in the index at `dbPath`; 0 when it is absent or cannot+ * be read. A plain read-only open with no extension loaded: the count+ * reads the embeddings table's rows, not the vectors.+ */+export function countStoredEmbeddings(dbPath: string): number {+ if (!existsSync(dbPath)) return 0;+ let db: Database;+ try {+ db = new Database(dbPath, { readonly: true });+ } catch {+ return 0;+ }+ try {+ return db.query<{ n: number }, []>("SELECT COUNT(*) AS n FROM embeddings").get()?.n ?? 0;+ } catch {+ return 0;+ } finally {+ closeDatabase(db);+ }+}+/** `unlinkWithRetry` that tolerates ENOENT (file already absent). */function tryUnlink(p: string): void {try {@@ -1918,7 +1973,7 @@ export async function indexCheck(let vecVersion: string | null = null;try {// Use an in-memory DB so the check never touches the real index.- const { Database } = await import("bun:sqlite");+ // bun:sqlite is imported statically; the open below is the probe.const db = new Database(":memory:");sqliteOk = true;try {diff --git a/src/core/search/pipeline/relational-arm.ts b/src/core/search/pipeline/relational-arm.tsindex 2f15b849..d67383ae 100644--- a/src/core/search/pipeline/relational-arm.ts+++ b/src/core/search/pipeline/relational-arm.ts@@ -46,6 +46,7 @@ import {SEARCH_TRUST_GATE_MODE,classifyRetrievalTrust,resolveSearchTrustGateMode,+ type SearchTrustGateMode,} from "../../brain/trust/retrieval-gate.ts";import {RETRIEVAL_TRUST_REFUSAL_LANE,@@ -97,7 +98,9 @@ export const RELATIONAL_ARM_REFUSAL_REASON = Object.freeze({* relevant-but-not-endorsing edge the pool-level pass only warns about.* `superseded_by` is not here - it is the supersession backbone the arm* annotates (`supersededBy`), pointing stale-to-fresh - and the positive- * relations boost.+ * relations boost. A polarity is never counted against an edge whose+ * relation the query itself named: `[[X]] contradicts` asks for exactly+ * those arrivals, and refusing them would empty the answer it asked for.*/const REFUSAL_POLARITIES: ReadonlySet<string> = new Set(["contradicts"]);@@ -164,6 +167,14 @@ export interface RelationalArmOptions {* env-only, exactly as a caller that consulted no config file did.*/readonly configPath?: string | null;+ /**+ * The retrieval trust gate's resolved mode, threaded from the caller's+ * effective config so the edge gate and the page gate answer to one+ * posture even when a caller overrides the resolved config (a bench+ * pin). Default: {@link resolveSearchTrustGateMode} over env + machine+ * config.+ */+ readonly trustGateMode?: SearchTrustGateMode;}export interface RelationalReach {@@ -206,6 +217,12 @@ export interface RelationalArmOutcome {* refusal is removed from (enforce) or kept in (shadow) the* contribution set AND counted here, while an omitted provenance step* is never named and never counted.+ *+ * The one drop with no record: under enforce, a node whose only+ * reasons come from an arrived edge's source page the caller cannot+ * read is removed and not recorded, because any record would disclose+ * that an unreadable page sits upstream. Disclosure wins over the+ * never-silently-dropped posture there, and only there.*/readonly refusals?: ReadonlyArray<RetrievalTrustRefusal>;}@@ -298,11 +315,15 @@ export function runRelationalArm(const budgets = opts.budgets ?? resolveTraversalBudgets({ env: process.env, config });const bridges = opts.entityBridges ?? resolveEntityBridgesEnabled({ env: process.env, config });// The edge-level eligibility gate (session-mining-economy, t_05a0eaa4)- // runs under the same resolved trust-gate mode the page gate answers- // to - the same env and the caller's own config map, so the two gates- // can never disagree about the posture. Off: no classification, no- // refusal field, the outcome shape byte-identical.- const trustGateMode = resolveSearchTrustGateMode({ env: process.env, config });+ // runs under the trust-gate mode the page gate answers to - the+ // caller's effective config threads it in, so the two gates can never+ // disagree about the posture. Off: no classification, no refusal+ // field, the outcome shape byte-identical.+ const trustGateMode =+ opts.trustGateMode ?? resolveSearchTrustGateMode({ env: process.env, config });+ const namedRelations: ReadonlySet<string> = new Set(+ relQuery.edgeTypes.map((t) => normalizeRelation(t)),+ );const nodes = relationalFanout(fanoutStoreFor(store, bridges), seedDocIds, {maxDepth: RELATIONAL_MAX_DEPTH,edgeTypes: relQuery.edgeTypes,@@ -343,6 +364,7 @@ export function runRelationalArm(nodePath: rep.path,sourcePath,arrivingRelation: lastStep?.relation ?? null,+ namedRelations,cache: frontmatterCache,nowMs,});@@ -479,7 +501,8 @@ function closedSupersessionTip(* grammar; a DECLARED value that cannot be parsed is a refusal of* its own, never a silent pass);* - the arriving relation's polarity is not a counted exclusion- * (the existing relation-polarity vocabulary).+ * (the existing relation-polarity vocabulary), unless the query+ * named that relation itself.** The split exists for the receipts' disclosure line: the `node` reasons* describe the node's own page and arriving edge, the `source` reasons@@ -498,6 +521,8 @@ function classifyArrivalEligibility(input: {readonly sourcePath: string | null;/** The relation that carried the walk into the node, when known. */readonly arrivingRelation: string | null;+ /** The normalized edge types the query named; their polarity is asked for, never refused. */+ readonly namedRelations: ReadonlySet<string>;readonly cache: FrontmatterCache;readonly nowMs: number;}): { readonly node: string[]; readonly source: string[] } {@@ -535,9 +560,12 @@ function classifyArrivalEligibility(input: {}}+ const arriving =+ input.arrivingRelation === null ? null : normalizeRelation(input.arrivingRelation);if (- input.arrivingRelation !== null &&- REFUSAL_POLARITIES.has(normalizeRelation(input.arrivingRelation))+ arriving !== null &&+ REFUSAL_POLARITIES.has(arriving) &&+ !input.namedRelations.has(arriving)) {node.push(`${RELATIONAL_ARM_REFUSAL_NAMESPACE}:${RELATIONAL_ARM_REFUSAL_REASON.contradictsPolarity}`,diff --git a/src/core/search/pipeline/store-open.ts b/src/core/search/pipeline/store-open.tsindex 675d5c86..e276f022 100644--- a/src/core/search/pipeline/store-open.ts+++ b/src/core/search/pipeline/store-open.ts@@ -23,9 +23,10 @@ const SELF_HEALABLE_CODES: ReadonlySet<SearchErrorCode> = new Set<SearchErrorCod* index. After a plugin upgrade the on-disk index can be a different schema* version (`SCHEMA_MISMATCH`), not yet built (`INDEX_MISSING`), or corrupt* / truncated / non-OSB at the index path (`INDEX_UNREADABLE`); rather than- * forcing the user to run `o2b search reindex` / `o2b search index`, rebuild- * once and retry. `reindexVault` is imported lazily so the hot path never- * pulls in the indexer and there is no module cycle.+ * forcing the user to run `o2b search reindex` / `o2b search index`, heal+ * once and retry. An older schema is upgraded in place, so the stored+ * embeddings survive; only a state a migration cannot repair is rebuilt+ * (see `self-heal.ts`).** A rebuild that fails is not swallowed. It may legitimately be a* concurrent writer already rebuilding (`INDEX_LOCKED`), in which case the@@ -44,12 +45,48 @@ export async function openReadOrSelfHeal(config: ResolvedSearchConfig): Promise<openFailure = e;}+ // Imported lazily so the hot path never pulls in the indexer and there+ // is no module cycle.+ const heal = await import("../self-heal.ts");let healFailure: string | null = null;- try {- const { reindexVault } = await import("../indexer.ts");- await reindexVault(config);- } catch (e) {- healFailure = e instanceof Error ? e.message : String(e);+ let rebuild = true;++ // An older schema is migrated in place, keeping every stored vector and+ // its recorded identity (#256). A condemned or absent index, a newer+ // schema, and one the migration did not make readable still rebuild.+ if (openFailure.code === "SCHEMA_MISMATCH" && heal.inspectIndex(config) !== "rebuild") {+ let upgraded = true;+ try {+ // A concurrent healer may already have made it current; the+ // re-inspection skips a pointless write open.+ if (heal.inspectIndex(config) === "older-schema") await heal.upgradeIndexInPlace(config);+ } catch (e) {+ upgraded = false;+ if (e instanceof SearchError && e.code === "INDEX_LOCKED") {+ // Another writer holds the index; a rebuild would only queue behind+ // it to discard its work. Reported with the symptom if the final+ // retry fails.+ healFailure = e.message;+ rebuild = false;+ }+ // Any other failure (a migration rolled back) falls back to a rebuild.+ }+ if (upgraded) {+ try {+ return await Store.open(config, { mode: "read" });+ } catch (e) {+ if (!(e instanceof SearchError) || !SELF_HEALABLE_CODES.has(e.code)) throw e;+ openFailure = e;+ }+ }+ }++ if (rebuild) {+ try {+ await heal.rebuildIndexForSelfHeal(config);+ } catch (e) {+ healFailure = e instanceof Error ? e.message : String(e);+ }}try {diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex cbd38ad9..1380c577 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -42,6 +42,7 @@ import {runRelationalArm,} from "./pipeline/relational-arm.ts";import { resolveSearchRequest } from "./pipeline/request.ts";+import { SEARCH_TRUST_GATE_MODE } from "../brain/trust/retrieval-gate.ts";import {resolveIterativeRequest,runDeciderRound,@@ -673,6 +674,9 @@ export async function search(const relational = isRelationalArmActive(effectiveConfig, opts)? runRelationalArm(store, effectiveConfig.vault, query, {configPath: effectiveConfig.configPath,+ // One posture for the edge gate and the page gate: the mode the+ // effective config resolved, overrides included.+ trustGateMode: effectiveConfig.recall.searchTrustGateMode ?? SEARCH_TRUST_GATE_MODE.off,...(deadline !== null ? { isExpired: () => deadlineExpired(deadline) } : {}),...(opts.transportReach !== undefined ? { reach: opts.transportReach } : {}),// The caller's owner scope gates the provenance path nodes the@@ -1045,9 +1049,10 @@ export async function search(frontmatterCache,nowMs,// The arm's eligibility refusals (present only under an active gate- // mode) ride into the receipt build, so the shadow receipts report- // the arm lane end to end; under enforce the receipts keep the- // pre-mode shape and the field is not consulted.+ // mode) ride into the receipt build: under shadow they join the+ // pool's would-be refusals, and under enforce they fill the+ // receipts' enforce section, since the arm drops them before the+ // pool where no exclusion can name them....(relational.refusals !== undefined ? { armRefusals: relational.refusals } : {}),...(deadline !== null? {diff --git a/src/core/search/self-heal.ts b/src/core/search/self-heal.tsnew file mode 100644index 00000000..de80edf1--- /dev/null+++ b/src/core/search/self-heal.ts@@ -0,0 +1,181 @@+/**+ * Index self-heal policy shared by the read path, the session-start check+ * and the detached self-heal child: upgrade in place first, rebuild only+ * when an upgrade cannot help (#256).+ *+ * An index on an OLDER schema is migratable: the write open runs+ * `applyMigrations` in one transaction under the writer lock, and every+ * stored vector, the recorded embedding model and its prefixes survive+ * untouched. A full rebuild (`reindexVault`) starts from an empty index+ * and, without paid provider calls, cannot recompute the vectors - so it+ * is kept for the states a migration cannot repair: a missing or+ * unreadable file, a NEWER schema than this build knows, chunks cut by+ * older chunking rules, and an index whose tables are still missing after+ * the migration.+ *+ * Concurrency: the in-place upgrade takes the same writer lock every+ * writer takes, and `applyMigrations` re-reads the version under it, so a+ * second process that waited for the lock finds the index current and+ * migrates nothing. A process that gives up on the lock re-inspects the+ * index instead of assuming the upgrade failed.+ */++import { existsSync } from "node:fs";+import { Database } from "bun:sqlite";++import { closeDatabase } from "../sqlite-close.ts";+import { CHUNKER_VERSION } from "./chunker.ts";+import { countStoredEmbeddings, reindexVaultUnless, type IndexVaultOptions } from "./indexer.ts";+import { findSchemaGaps, LATEST_SCHEMA_VERSION, readSchemaVersion } from "./schema.ts";+import { consolidateWal, openWriteDatabase } from "./store/lifecycle.ts";+import { CHUNKER_VERSION_STATE_KEY, INTEGRITY_FAULT_STATE_KEY } from "./store/state.ts";+import { SearchError, type IndexStats, type ResolvedSearchConfig } from "./types.ts";++/**+ * What a cheap read-only look at the index file says:+ * - `current`: nothing to do.+ * - `older-schema`: a migratable older schema with current chunks.+ * - `rebuild`: absent, unreadable, newer than this build, chunked by+ * older rules, condemned by a recorded integrity fault, or stamped+ * current with tables or columns missing.+ */+export type IndexCondition = "current" | "older-schema" | "rebuild";++/**+ * Whether the index holds chunks cut by older chunking rules: documents+ * are present and the recorded CHUNKER_VERSION is not this binary's (or+ * was never recorded). An unchanged file is never re-chunked by an+ * incremental run, so only a rebuild replaces those chunks.+ */+function chunksAreStale(db: Database): boolean {+ const row = db+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ? LIMIT 1")+ .get(CHUNKER_VERSION_STATE_KEY);+ if (row?.value === String(CHUNKER_VERSION)) return false;+ return db.query<{ one: number }, []>("SELECT 1 AS one FROM documents LIMIT 1").get() !== null;+}++/** Whether a completed integrity check recorded a fault against the file. */+function hasRecordedIntegrityFault(db: Database): boolean {+ return (+ db+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ? LIMIT 1")+ .get(INTEGRITY_FAULT_STATE_KEY) !== null+ );+}++/** Classify the index at `config.dbPath` with read-only queries. */+export function inspectIndex(config: ResolvedSearchConfig): IndexCondition {+ if (!existsSync(config.dbPath)) return "rebuild";+ let db: Database;+ try {+ db = new Database(config.dbPath, { readonly: true });+ } catch {+ return "rebuild";+ }+ try {+ const version = readSchemaVersion(db);+ if (version > LATEST_SCHEMA_VERSION || chunksAreStale(db)) return "rebuild";+ // A file a completed integrity check condemned refuses every open, so+ // it is not current whatever its schema says; the rebuild is its exit.+ if (hasRecordedIntegrityFault(db)) return "rebuild";+ if (version < LATEST_SCHEMA_VERSION) return "older-schema";+ return findSchemaGaps(db).length > 0 ? "rebuild" : "current";+ } catch {+ return "rebuild"; // corrupt / non-Open Second Brain file+ } finally {+ closeDatabase(db);+ }+}++/**+ * Migrate the index in place: a write open (which runs `applyMigrations`+ * under the writer lock) and an orderly close. Nothing else runs - in+ * particular not the embedding-identity check - so the stored vectors and+ * their recorded model and prefixes are left exactly as they were,+ * whatever the caller's semantic configuration. Throws what the open+ * throws (`INDEX_LOCKED`, a failed migration rolled back).+ */+export async function upgradeIndexInPlace(config: ResolvedSearchConfig): Promise<void> {+ const opened = await openWriteDatabase(config, true);+ try {+ consolidateWal(opened.db);+ closeDatabase(opened.db);+ } finally {+ await opened.release();+ }+}++/**+ * The full rebuild a self-heal falls back to, which names the vectors it+ * could not carry over. A rebuild without `embeddings` computes none, so+ * every vector of the replaced index is gone from the live one; saying so+ * once, with where the old index went and how to restore the vectors, is+ * the difference between a visible cost and a silent downgrade to+ * keyword-only search.+ *+ * The rebuild re-inspects the index once it holds the writer lock and+ * returns `null` without building when the index is already current - a+ * concurrent healer finished first - so two self-heals never rebuild in a+ * row over each other's work.+ */+export async function rebuildIndexForSelfHeal(+ config: ResolvedSearchConfig,+ opts?: IndexVaultOptions,+): Promise<IndexStats | null> {+ const before = countStoredEmbeddings(config.dbPath);+ const stats = await reindexVaultUnless(config, () => inspectIndex(config) === "current", opts);+ if (stats === null) return null;+ if (before > 0) {+ const dropped = before - countStoredEmbeddings(config.dbPath);+ if (dropped > 0) {+ // eslint-disable-next-line no-console+ console.error(+ `search index rebuilt: ${dropped} stored embedding(s) were not carried over ` ++ `(the previous index is kept at ${config.dbPath}.bak until the next rebuild); ` ++ `restore them with: o2b search index --embeddings`,+ );+ }+ }+ return stats;+}++/** What {@link healIndex} did. */+export type HealOutcome =+ | { readonly kind: "current" }+ | { readonly kind: "migrated" }+ | { readonly kind: "rebuilt"; readonly stats: IndexStats };++/**+ * Bring the index current: migrate an older schema in place, and rebuild+ * only when the index still needs it afterwards. A failed upgrade is not+ * fatal - the re-inspection decides, so an upgrade another process+ * finished while this one waited for the lock counts as done. An upgrade+ * that could not take the lock and finds the index still old is rethrown+ * rather than rebuilt over: another writer holds the index, and a+ * rebuild would only queue behind it to discard what it just did.+ */+export async function healIndex(+ config: ResolvedSearchConfig,+ opts?: IndexVaultOptions,+): Promise<HealOutcome> {+ let condition = inspectIndex(config);+ if (condition === "current") return { kind: "current" };+ if (condition === "older-schema") {+ try {+ await upgradeIndexInPlace(config);+ } catch (e) {+ if (inspectIndex(config) === "current") return { kind: "migrated" };+ if (e instanceof SearchError && e.code === "INDEX_LOCKED") throw e;+ // eslint-disable-next-line no-console+ console.error(+ `in-place upgrade of ${config.dbPath} failed, rebuilding instead: ` ++ `${e instanceof Error ? e.message : String(e)}`,+ );+ }+ condition = inspectIndex(config);+ if (condition === "current") return { kind: "migrated" };+ }+ const stats = await rebuildIndexForSelfHeal(config, opts);+ return stats === null ? { kind: "current" } : { kind: "rebuilt", stats };+}diff --git a/src/core/search/store/lifecycle.ts b/src/core/search/store/lifecycle.tsindex 023e87f4..ae9ab3e9 100644--- a/src/core/search/store/lifecycle.ts+++ b/src/core/search/store/lifecycle.ts@@ -8,7 +8,7 @@*/import { Database } from "bun:sqlite";-import { existsSync, mkdirSync } from "node:fs";+import { existsSync, mkdirSync, unlinkSync } from "node:fs";import { dirname } from "node:path";import {@@ -170,10 +170,18 @@ function busySwitchWarning(dbPath: string, journalMode: JournalMode): string {* clobber; the caller's open path then re-evaluates existence and reports* honestly (INDEX_MISSING). The `.bak` restore of a genuine crash is* unaffected: with no live holder the lock is acquired immediately.+ *+ * A swap that keeps an embedding-richer `.bak` parks the live index at+ * `.old` instead (reindexVault). A crash between its renames leaves the+ * live index at `.old` and the OLDER index at `.bak`, so `.old` is+ * restored first - restoring `.bak` would roll back past the live index.+ * A `.old` beside a present live index is the leftover of a swap that+ * completed but crashed before removing it, and is deleted.*/export function restoreFromBakIfMissing(dbPath: string): void {const bak = dbPath + ".bak";- if (existsSync(dbPath) || !existsSync(bak)) return;+ const old = dbPath + ".old";+ if (!existsSync(old) && (existsSync(dbPath) || !existsSync(bak))) return;let release: (() => void) | null = null;try {release = acquireWriterLockSync(dbPath);@@ -185,10 +193,17 @@ export function restoreFromBakIfMissing(dbPath: string): void {try {// Re-check under the lock: the swap may have completed while we waited,// or another opener may have already restored.- if (existsSync(dbPath) || !existsSync(bak)) return;- renameWithRetry(bak, dbPath);+ if (existsSync(dbPath)) {+ if (existsSync(old)) unlinkSync(old);+ return;+ }+ let source: string | null = null;+ if (existsSync(old)) source = old;+ else if (existsSync(bak)) source = bak;+ if (source === null) return;+ renameWithRetry(source, dbPath);// eslint-disable-next-line no-console- console.error(`restored search index from ${bak} (previous reindex crash)`);+ console.error(`restored search index from ${source} (previous reindex crash)`);} catch {/* fall through β open path below will report INDEX_MISSING */} finally {diff --git a/src/core/search/store/vectors.ts b/src/core/search/store/vectors.tsindex 81dd7021..cdc4c298 100644--- a/src/core/search/store/vectors.ts+++ b/src/core/search/store/vectors.ts@@ -413,6 +413,15 @@ function rebuildRefusalBeforeClear(readonly previousDimension: number | null;readonly model: string | null;readonly dimension: number | null;+ /**+ * Set when the model and dimension are unchanged and only the prefix+ * pair differs, so the refusal names what actually changed instead of+ * a "model change from X to X" (#255).+ */+ readonly prefixChange?: {+ readonly from: EmbeddingPrefixPair;+ readonly to: EmbeddingPrefixPair;+ };},): EmbeddingRebuildRefusal | null {if (gate === undefined) return null;@@ -421,14 +430,23 @@ function rebuildRefusalBeforeClear(const capability = resolveSemanticCapability(gate.semantic);if (!isBlockedCapability(capability)) return null;const material = hasChunkRows(db) ? "" : "; no chunk material remains to rebuild from";+ const change =+ context.prefixChange === undefined+ ? `embedding model change from ${context.previousModel}/${context.previousDimension} to ` ++ `${context.model}/${context.dimension}`+ : `embedding prefix change from ` ++ `[${context.prefixChange.from.query}|${context.prefixChange.from.passage}] to ` ++ `[${context.prefixChange.to.query}|${context.prefixChange.to.passage}]`;+ const restore =+ context.prefixChange === undefined+ ? "restore the previous model or the provider credential"+ : "restore the previous prefixes or enable a provider that can re-embed";return {code: BLOCKED_TIER_ERROR_CODE[capability.tier],message:- `embedding model change from ${context.previousModel}/${context.previousDimension} to ` +- `${context.model}/${context.dimension} refused: ${stored} stored embedding(s) cannot be ` ++ `${change} refused: ${stored} stored embedding(s) cannot be ` +`rebuilt while the semantic capability is ${capability.tier}${material}; the stored ` +- `embeddings and the recorded model are kept; restore the previous model or the provider ` +- `credential, then reindex`,+ `embeddings and the recorded model are kept; ${restore}, then reindex`,};}@@ -495,7 +513,21 @@ export function ensureEmbeddingModel(previousDimension: prevDim,model,dimension,+ ...(!modelChanged && !dimChanged && prefixChanged+ ? {+ prefixChange: {+ from: { query: effectivePrevQueryPrefix, passage: effectivePrevPassagePrefix },+ to: prefixes,+ },+ }+ : {}),};+ // Vectors kept on a named-model -> null-model open (#255): this writer+ // has no model, so it never embedded anything and its prefix pair says+ // nothing about the stored vectors. Recording it would re-stamp them with+ // a foreign identity, and the next semantic writer would then see a+ // spurious prefix change and clear compatible vectors.+ const keptUnverified = modelRemoved && countEmbeddings(db) > 0;const refusal =modelChanged || dimChanged || (!modelRemoved && prefixChanged)? rebuildRefusalBeforeClear(db, gate, refusalContext)@@ -525,7 +557,7 @@ export function ensureEmbeddingModel(// just cleared, so it must not outlive them: leaving it would// claim an ABI for storage that no longer holds any.deleteState(db, EMBEDDING_VEC_VERSION_STATE_KEY);- } else if (modelRemoved && countEmbeddings(db) > 0) {+ } else if (keptUnverified) {// Named-model -> null-model: never a clear (see docblock), never silent.// This branch also deliberately precedes the prefix branch: with no// model configured there is nothing to verify a prefix clear against.@@ -554,7 +586,9 @@ export function ensureEmbeddingModel(if (vec.version !== null) {setState(db, EMBEDDING_VEC_VERSION_STATE_KEY, vec.version);}- if (prefixes !== undefined) {+ // Like the model and dimension: recorded only by a writer that owns the+ // vectors' identity, never by one that kept them unverified.+ if (prefixes !== undefined && !keptUnverified) {setState(db, EMBEDDING_PREFIX_QUERY_STATE_KEY, prefixes.query);setState(db, EMBEDDING_PREFIX_PASSAGE_STATE_KEY, prefixes.passage);}diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex ddc9a59b..cf280fbd 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -1702,7 +1702,7 @@ export interface ResolvedRecallConfig {* or `enforce` (today's excluding behavior, made explicit). Resolved* as the explicit `search_trust_gate_mode` key (env* OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE_MODE), else the legacy- * `search_trust_gate_enabled` boolean mapped true to shadow, else off.+ * `search_trust_gate_enabled` boolean mapped true to enforce, else off.* The mode rides the query-cache key, and the explain envelope names* it under shadow.*/diff --git a/src/core/search/visibility-surface-registry.ts b/src/core/search/visibility-surface-registry.tsindex 58335339..d3eeb858 100644--- a/src/core/search/visibility-surface-registry.ts+++ b/src/core/search/visibility-surface-registry.ts@@ -1458,16 +1458,4 @@ export const DIRECT_VAULT_READ_REGISTRY: ReadonlyArray<DirectVaultReadEntry> = O"own shell, so the reach question does not arise, and no page's reservation could " +"cover what it hands back.",},- {- file: "src/mcp/brain/ingest-tools.ts",- category: D.discloses_nothing,- reason:- "stampedArchiveHashes readFileSync+parseFrontmatter's the summary pages of the " +- "caller-named source's cleanup plan BEFORE a confirmed deletion erases the stamps " +- "with the pages (the archive-orphan decision needs them). The page set comes from " +- "the plan's own derived manifest - never from caller-supplied paths - and every " +- "path in it is already named by the preview the same run serialized. What crosses " +- "back is content hashes and orphan dispositions of that same set: no page title, " +- "body or frontmatter value, so no page's reservation could cover the answer.",- },]);diff --git a/src/core/state/surfaces.ts b/src/core/state/surfaces.tsindex 88f4f301..85b8674e 100644--- a/src/core/state/surfaces.ts+++ b/src/core/state/surfaces.ts@@ -118,6 +118,7 @@ export const STATE_SURFACE_ID = Object.freeze({writeDeadLetters: "write_dead_letters",sessionImportLedger: "session_import_ledger",sessionMiningLedger: "session_mining_ledger",+ sessionMiningSurveys: "session_mining_surveys",installManifest: "install_manifest",protectManifest: "protect_manifest",bootstrapReceipt: "bootstrap_receipt",@@ -286,6 +287,7 @@ const SESSION_CHECKPOINT_DIR = "session-import-checkpoints";const DEAD_LETTER_DIR = "dead-letters";const SESSION_LEDGER_FILE = "session-import-ledger.json";const SESSION_MINING_LEDGER_FILE = "session-mining-ledger.json";+const SESSION_MINING_SURVEYS_FILE = "session-mining-surveys.json";const INSTALL_MANIFEST_FILE = "install.lock.json";const PROTECT_MANIFEST_FILE = "protect.lock.json";const BOOTSTRAP_RECEIPT_FILE = "bootstrap.lock.json";@@ -519,6 +521,21 @@ export const STATE_SURFACES: ReadonlyArray<StateSurface> = Object.freeze(["barren session, paying the full model spend again on the next scan.",sources: ["src/core/brain/session-mining-ledger.ts"],},+ {+ id: STATE_SURFACE_ID.sessionMiningSurveys,+ label: "session mining surveys",+ tier: STATE_TIER.derived,+ derive: derivedStore(SESSION_MINING_SURVEYS_FILE),+ override_env: null,+ override_config_key: null,+ carries_memory: false,+ reason:+ "The surveyed-turns hash of every mining plan that handed out an envelope and has not been " ++ "answered yet, so a barren commit that relays no hash tombstones only the turns a model was " ++ "shown. Deleting it loses nothing the vault holds; the next barren commit of a pending " ++ "session records no tombstone until the session is planned again.",+ sources: ["src/core/brain/session-mining-surveys.ts"],+ },{id: STATE_SURFACE_ID.installManifest,label: "install manifest",diff --git a/src/mcp/brain/admin-tools.ts b/src/mcp/brain/admin-tools.tsindex f243c833..967805a7 100644--- a/src/mcp/brain/admin-tools.ts+++ b/src/mcp/brain/admin-tools.ts@@ -499,6 +499,9 @@ async function toolBrainMaintenance(busy,...(retryTasks.length > 0 ? { retryTasks } : {}),...(window !== undefined ? { window } : {}),+ // The server's own config file, so the capacity gate measures the+ // store a search_db_path in it names.+ ...(ctx.configPath ? { configPath: ctx.configPath } : {}),tasks: lane.tasks,});// The run-level spend block, the CLI JSON payload's shape.diff --git a/src/mcp/brain/distill-tools.ts b/src/mcp/brain/distill-tools.tsindex de7c7a2c..6e67a405 100644--- a/src/mcp/brain/distill-tools.ts+++ b/src/mcp/brain/distill-tools.ts@@ -153,11 +153,12 @@ export const DISTILL_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([// by DEFAULT: it works through `status: quarantine` plus the entity page// status scope, which nothing gates. The same words mean less here.//- // Session-mining-economy: the legacy boolean alone now mounts the gate- // in the measure-only shadow mode, so exclusion requires the mode to- // name `enforce` explicitly. The description states both keys.+ // Session-mining-economy: the legacy boolean still mounts the gate in+ // the excluding mode, and an explicit `search_trust_gate_mode: enforce`+ // does the same; `shadow` measures without excluding. The description+ // states both keys.description:- "Distill one source into atomic claims; runs no model, checks quotes. Writes an idempotent page citing each claim as `[[source#^block]]`. `untrusted_source` pages surface unless the gate enforces (`search_trust_gate_mode: enforce`; `search_trust_gate_enabled` alone measures). `trust` names the lane.",+ "Distill one source into atomic claims; runs no model, checks quotes. Writes an idempotent page citing each claim as `[[source#^block]]`. `untrusted_source` pages surface unless the gate enforces (`search_trust_gate_enabled: true` or `search_trust_gate_mode: enforce`). `trust` names the lane.",inputSchema: {type: "object",properties: {diff --git a/src/mcp/brain/extract-tools.ts b/src/mcp/brain/extract-tools.tsindex 5e20529f..dcbe6f45 100644--- a/src/mcp/brain/extract-tools.ts+++ b/src/mcp/brain/extract-tools.ts@@ -11,6 +11,7 @@import { commitExtractedSignals, ExtractSignalsError } from "../../core/brain/extract-signals.ts";import { loadDedupIndex } from "../../core/brain/dedup-hash.ts";+import { TRANSPORT_REACH } from "../../core/graph/transport-reach.ts";import {planExtractSignalsPrefiltered,recordExtractPrefilterCommit,@@ -20,8 +21,9 @@ import { ResponseCheckError } from "../../core/brain/response-checks.ts";import { ResponseShapeError } from "../../core/brain/response-shape.ts";import { resolveAgentName, resolveTokenImpactLedgerEnabled } from "../../core/config.ts";import { coerceStr } from "../coerce.ts";+import { INVALID_PARAMS, MCPError } from "../protocol.ts";import { MCP_PREVIEW_BUDGET } from "../preview-budget.ts";-import type { ServerContext, ToolDefinition } from "../tool-contract.ts";+import { contextReach, type ServerContext, type ToolDefinition } from "../tool-contract.ts";import { vaultRelativeSafe, wrapToolErrors } from "./shared.ts";const TOOL = "brain_extract_signals";@@ -45,6 +47,9 @@ async function toolBrainExtractSignals(// answer claiming the session states nothing, and it goes through the// validators like any other so the caller gets one contract.const decisionModel = resolveExtractPrefilterConfig(ctx.vault, ctx.configPath ?? undefined);+ // Below local reach a store failure is named without its raised+ // message, which carries the absolute host path.+ const closeHostErrors = contextReach(ctx) !== TRANSPORT_REACH.local;if (args["items"] === undefined) {const plan = await planExtractSignalsPrefiltered(ctx.vault, session, {now: new Date(),@@ -55,6 +60,7 @@ async function toolBrainExtractSignals(// plan call, for "I removed the downstream item and want it// re-proposed".force: args["force"] === true,+ closeHostErrors,});return {phase: "plan",@@ -67,9 +73,9 @@ async function toolBrainExtractSignals(text: t.text,timestamp: t.timestamp,})),- // The hash of the material this plan offered; the commit that- // answers it passes it back as `surveyed_turns_hash` so the mining- // tombstone keys on the surveyed material.+ // The hash of the material this plan offered, also recorded as a+ // pending survey; the commit that answers it may pass it back as+ // `surveyed_turns_hash` to check the session did not change.surveyed_turns_hash: plan.surveyedTurnsHash,cap: plan.cap,confidence_floor: plan.confidenceFloor,@@ -101,10 +107,21 @@ async function toolBrainExtractSignals(...(plan.decisionModel !== undefined? { decision_model: { degraded: plan.decisionModel.degraded } }: {}),+ // Present only when the plan could not record its survey; a barren+ // commit then needs `surveyed_turns_hash` to record its tombstone.+ ...(plan.surveyRecordFailure !== undefined+ ? { survey_record_failure: plan.surveyRecordFailure }+ : {}),};}const now = new Date();- const surveyedTurnsHash = coerceStr(args, "surveyed_turns_hash", false);+ // Read raw rather than through coerceStr, which maps "" to absent: an+ // empty relay is a malformed hash, and the core refuses it by name.+ const surveyedRaw = args["surveyed_turns_hash"];+ if (surveyedRaw !== undefined && surveyedRaw !== null && typeof surveyedRaw !== "string") {+ throw new MCPError(INVALID_PARAMS, "argument 'surveyed_turns_hash' must be a string");+ }+ const surveyedTurnsHash = typeof surveyedRaw === "string" ? surveyedRaw : undefined;const res = commitExtractedSignals(ctx.vault,session,@@ -114,11 +131,11 @@ async function toolBrainExtractSignals(agent,now,dedup: await loadDedupIndex(ctx.vault),- // The answered plan's surveyed hash, echoed back by the caller:- // what the zero-yield tombstone keys on. Absent, the commit- // records nothing - there is no material the verdict was earned- // over.- ...(surveyedTurnsHash ? { surveyedTurnsHash } : {}),+ // The answered plan's surveyed hash, when the caller echoes it: a+ // check that the session did not change since the plan. Absent,+ // the commit checks the survey the plan recorded instead.+ ...(surveyedTurnsHash !== undefined ? { surveyedTurnsHash } : {}),+ closeHostErrors,},);recordExtractPrefilterCommit(ctx.vault, decisionModel, res, now);@@ -150,7 +167,7 @@ export const EXTRACT_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([{name: TOOL,description:- "Mine taste signals from an imported session's USER turns; runs no model. Omit `items` for the plan; with `items` pass the plan's `surveyed_turns_hash` back so a barren commit records its tombstone. Barren sessions skip as 'mining_tombstone'; `force` re-mines once. Refuses caps, floors, repeats.",+ "Mine taste signals from an imported session's USER turns; runs no model. Omit `items` for the plan; with `items` commit, and a barren commit records a tombstone. Barren sessions skip as 'mining_tombstone'; `force` re-mines once. Refuses caps, floors, repeats.",inputSchema: {type: "object",properties: {@@ -161,7 +178,7 @@ export const EXTRACT_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([surveyed_turns_hash: {type: "string",description:- "Commit phase only: the `surveyed_turns_hash` from the plan response this payload answers. Omit it and a barren commit records nothing.",+ "Commit phase only, optional: the plan's `surveyed_turns_hash`. If the session changed since that plan, no mining record is written.",},items: {type: "array",diff --git a/src/mcp/brain/health-tools.ts b/src/mcp/brain/health-tools.tsindex 83d8622b..366cb991 100644--- a/src/mcp/brain/health-tools.ts+++ b/src/mcp/brain/health-tools.ts@@ -8,10 +8,12 @@import { resolveSearchConfig } from "../../core/search/index.ts";import {+ closedCollectError,collectMaintenanceActions,type MaintenanceCollectError,} from "../../core/brain/maintenance/collect.ts";import { runDoctor } from "../../core/brain/doctor.ts";+import { TRANSPORT_REACH } from "../../core/graph/transport-reach.ts";import { annotateEntityAliasIssues } from "../../core/brain/doctor/entity-alias-verdicts.ts";import { verdictFields } from "../../core/decision-model/pair-verdict.ts";import {@@ -226,7 +228,13 @@ async function toolBrainDoctor(// Rendered only when a scan refused, so a clean payload is// byte-identical to the pre-row shape (review M10)....(collectErrors.length > 0- ? { collect_errors: collectErrors.map((e) => ({ scan: e.signal, message: e.message })) }+ ? {+ collect_errors: collectErrors+ // Below local reach the raised message, which can carry a+ // host path, is replaced by the scan name and errno code.+ .map((e) => (contextReach(ctx) === TRANSPORT_REACH.local ? e : closedCollectError(e)))+ .map((e) => ({ scan: e.signal, message: e.message })),+ }: {}),// v0.10.16: trust-layer fields. `trust_verdict` is always populated// by runDoctor; `verification_delta_summary` only when the callerdiff --git a/src/mcp/brain/ingest-tools.ts b/src/mcp/brain/ingest-tools.tsindex 29713018..a16cab43 100644--- a/src/mcp/brain/ingest-tools.ts+++ b/src/mcp/brain/ingest-tools.ts@@ -9,7 +9,7 @@* Idempotent on the source path.*/-import { existsSync, readFileSync } from "node:fs";+import { existsSync } from "node:fs";import { join } from "node:path";import { serializeImportCensus } from "../../core/brain/import-census.ts";@@ -18,17 +18,6 @@ import { clearCheckpoint, PLAN_ID_LABEL } from "../../core/brain/ingest/checkpoiimport { assertCheckpointId } from "../../core/brain/checkpoint-store.ts";import { STORAGE_CAPACITY_CRITICAL_CODE } from "../../core/brain/doctor/storage-capacity.ts";import { MAINTENANCE_CAPACITY_REMEDY } from "../../core/brain/maintenance/lane.ts";-import { BRAIN_CONFIG_SUPPORTED_VERSIONS } from "../../core/brain/policy/defaults.ts";-import { BrainConfigError } from "../../core/brain/policy/errors.ts";-import { KeyIndexCollector, type BlockParseContext } from "../../core/brain/policy/key-index.ts";-import {- BRAIN_STORAGE_DEFAULTS,- parseStorageBlock,- resolveStorage,-} from "../../core/brain/policy/blocks/storage.ts";-import type { BrainConfig, ResolvedBrainStorageConfig } from "../../core/brain/types.ts";-import { brainConfigPath } from "../../core/brain/paths.ts";-import { parseBrainYaml, type ParsedBlock } from "../../core/brain/yaml-parse.ts";import {StorageCapacityError,type StorageErrorCode,@@ -38,6 +27,10 @@ import {evaluateCapacityGate,type CapacityGateDecision,} from "../../core/brain/storage/gate.ts";+import {+ loadGateStorageConfig,+ StorageGateConfigError,+} from "../../core/brain/storage/gate-config.ts";import { ingestSource } from "../../core/brain/ingest/ingest.ts";import type { PartsOutcome, TableOutcome } from "../../core/brain/ingest/extract-source.ts";import type { CodeEdgeSeed, PreExtractResult } from "../../core/brain/ingest/pre-extract.ts";@@ -102,18 +95,6 @@ const ADMISSION_REFUSAL_CODE: StorageErrorCode = "capacity_exhausted";*/const CONFIG_GATE_REFUSAL_CODE = "config_invalid";-/**- * A broken `_brain.yaml` the admission gate cannot decide under. Its prose- * is built from a relative config label, never the config's host path, so- * the refusal can cross to a remote caller (see {@link loadGateStorageConfig}).- */-class StorageGateConfigError extends Error {- constructor(message: string) {- super(message);- this.name = "StorageGateConfigError";- }-}-/*** Refuse the ingest when a fresh census measures the vault at its* critical storage tier - the interactive half of the storage lifecycle's@@ -138,11 +119,17 @@ class StorageGateConfigError extends Error {* the two walls an operator can hit - the paused lane and the refused* write - cannot disagree about what to do next.*/-function refuseWhenVaultExhausted(vault: string, reach: TransportReach, now: Date): void {+function refuseWhenVaultExhausted(+ vault: string,+ reach: TransportReach,+ now: Date,+ configPath: string | null,+): void {const storage = loadGateStorageConfig(vault);if (storage.cap_bytes === null) return;const decision = evaluateCapacityGate(vault, {now,+ ...(configPath ? { configPath } : {}),capBytes: storage.cap_bytes,warnPercent: storage.warn_percent,criticalPercent: storage.critical_percent,@@ -189,69 +176,6 @@ function admissionDetails(};}-/**- * The storage policy the admission gate decides under, read from ONLY the- * `storage:` block of `_brain.yaml` (review M1).- *- * The gate's decision needs no other block, but the whole-file loader- * validated all of them, so a typo in an unrelated block turned every- * ingest into an internal error naming the config's host path - including- * on vaults whose operator never configured a cap, where the documented- * default is that nothing refuses. This reader answers the gate's actual- * question: an absent config or block is the defaults (no cap, no gate),- * and the rest of the file is never validated here.- *- * A file that cannot be read or parsed, an unsupported `schema_version`,- * or a `storage:` block that does not validate is a policy this gate- * cannot see through, and silently reading it as "no cap" would run- * writes past an operator's bound. Those refuse BY NAME: the error- * carries field-level prose built from a relative config label, and the- * handler maps it onto the registered {@link CONFIG_GATE_REFUSAL_CODE}- * envelope - never a path below local reach.- */-function loadGateStorageConfig(vault: string): ResolvedBrainStorageConfig {- const path = brainConfigPath(vault);- if (!existsSync(path)) return BRAIN_STORAGE_DEFAULTS;- let text: string;- try {- text = readFileSync(path, "utf8");- } catch (err) {- // Only the errno code is kept: a read-failure message embeds the- // absolute path the call failed on.- const code = (err as NodeJS.ErrnoException).code ?? "unreadable";- throw new StorageGateConfigError(`_brain.yaml could not be read (${code})`);- }- let parsed: ParsedBlock;- try {- parsed = parseBrainYaml(text);- } catch (err) {- // The parser's messages name line numbers, never paths.- throw new StorageGateConfigError(`_brain.yaml could not be parsed: ${(err as Error).message}`);- }- const version = parsed["schema_version"];- if (typeof version !== "number" || !BRAIN_CONFIG_SUPPORTED_VERSIONS.includes(version)) {- throw new StorageGateConfigError(- `_brain.yaml: unsupported schema_version: ${JSON.stringify(version ?? null)}`,- );- }- const ctx: BlockParseContext = {- obj: parsed,- // A relative label, never the host path: the field errors this context- // raises become the refusal's prose.- source: "_brain.yaml",- warnings: [],- keys: new KeyIndexCollector(),- };- try {- // Only the storage block is parsed and resolved; `resolveStorage`- // reads nothing else off the config.- return resolveStorage({ storage: parseStorageBlock(ctx) } as BrainConfig);- } catch (err) {- if (err instanceof BrainConfigError) throw new StorageGateConfigError(err.message);- throw err;- }-}-/*** The MCP envelope for an admission refusal, from the thrown leaf: the* wire code is the code the registered `carried()` classification reads@@ -308,7 +232,7 @@ async function toolBrainIngestSource(// envelope here, before the generic wrapper could flatten it into an// internal error.try {- refuseWhenVaultExhausted(ctx.vault, contextReach(ctx), new Date());+ refuseWhenVaultExhausted(ctx.vault, contextReach(ctx), new Date(), ctx.configPath ?? null);} catch (err) {if (err instanceof StorageCapacityError) throw capacityRefusalEnvelope(err);// A broken config the gate cannot see through refuses by name, with@@ -556,6 +480,7 @@ function discardOrphanedArchiveEntry(vault: string, hash: string): boolean {function serializePlan(plan: SourceCleanupPlan,archiveEntries: ReadonlyArray<ArchiveCleanupEntry>,+ local: boolean,): Record<string, unknown> {return {source: plan.source,@@ -576,7 +501,12 @@ function serializePlan(// clean run with a smaller count. Always carried (empty on a clean or// dry run) so an automations can read the field unconditionally, like// `deleted` beside it.- failed: plan.failed.map((f) => ({ path: f.path, message: f.message })),+ // Below local reach the raised message, which can carry a host path,+ // is replaced by the errno code; the vault-relative path stays.+ failed: plan.failed.map((f) => ({+ path: f.path,+ message: local ? f.message : `removal failed (${f.code ?? "error"})`,+ })),manifest_entry_removed: plan.manifestEntryRemoved,audit_record_id: plan.auditRecordId,// The archive-entry cleanup kind (answer trust, Task C4), reported as@@ -684,14 +614,14 @@ async function toolBrainDeleteBySource(archiveEntries = orphanedArchiveEntries(ctx.vault, stamped, new Set(plan.deleted)).filter((entry) => discardOrphanedArchiveEntry(ctx.vault, entry.hash),);- return serializePlan(plan, archiveEntries);+ return serializePlan(plan, archiveEntries, contextReach(ctx) === TRANSPORT_REACH.local);}archiveEntries = orphanedArchiveEntries(ctx.vault,stamped,new Set(preview.derived.map((entry) => entry.path)),);- return serializePlan(preview, archiveEntries);+ return serializePlan(preview, archiveEntries, contextReach(ctx) === TRANSPORT_REACH.local);} catch (err) {// Write-side trust (Task 12): a permissions document rule denied the// confirmed cleanup. Named code, the rule that decided, and thediff --git a/src/mcp/brain/storage-tools.ts b/src/mcp/brain/storage-tools.tsindex e4717d3d..4445c6dd 100644--- a/src/mcp/brain/storage-tools.ts+++ b/src/mcp/brain/storage-tools.ts@@ -91,6 +91,8 @@ export interface BuildStorageReportOptions {readonly vault: string;/** How far the caller had to reach; the census answers at local reach only. */readonly reach: TransportReach;+ /** The explicit machine-config file the census resolves the store from. */+ readonly configPath?: string;}/**@@ -113,7 +115,10 @@ export function buildStorageReport(opts: BuildStorageReportOptions): StorageRepo// One census, three uses: the reading, the forecast's fresh truth,// and the sample. Fail-soft by the sampler's contract - an append// failure is carried on the report, not swallowed.- const sample = sampleStorageUsage(opts.vault);+ const sample = sampleStorageUsage(+ opts.vault,+ opts.configPath !== undefined ? { configPath: opts.configPath } : {},+ );const forecast = forecastStorageUsage(opts.vault, {capBytes: storage.cap_bytes,latestUsage: sample.usage,@@ -139,7 +144,11 @@ async function toolBrainStorage(args: Record<string, unknown>,): Promise<Record<string, unknown>> {const format = coerceFormat(args);- const report = buildStorageReport({ vault: ctx.vault, reach: contextReach(ctx) });+ const report = buildStorageReport({+ vault: ctx.vault,+ reach: contextReach(ctx),+ ...(ctx.configPath ? { configPath: ctx.configPath } : {}),+ });return { format, ...report };}diff --git a/tests/cli/brain-extract-signals.test.ts b/tests/cli/brain-extract-signals.test.tsindex 5be0bf1f..60cba903 100644--- a/tests/cli/brain-extract-signals.test.ts+++ b/tests/cli/brain-extract-signals.test.ts@@ -197,10 +197,12 @@ describe("think-block strip (t_dac8bf7e)", () => {// ----- The zero-yield mining tombstone on the surface ------------------------//// 6. A commit that lands nothing tombstones the session - keyed on the-// surveyed hash the plan printed, which the commit relays back - and-// the next plan says so by name with the recorded breakdown, in --json-// and in text, instead of re-offering the same turns. A commit that-// answers no plan records nothing, so the session stays eligible.+// session's current mined turns, with the plan's surveyed hash relayed+// back or with the plan's recorded survey standing in for it (a turn+// imported after the plan withholds it) - and the next plan says so by name with the+// recorded breakdown, in --json and in text, instead of re-offering the+// same turns. A relayed hash is a check: a malformed one refuses the+// commit before any write.// 7. --force bypasses the tombstone for exactly one plan call: the forced// plan carries the envelope, and the plan after it is suppressed again.// 8. A ledger write failure on a successful commit exits 0 and names the@@ -258,15 +260,58 @@ describe("mining tombstone", () => {expect(payload.turns_mined).toEqual([]);});- test("a barren commit without the surveyed hash records nothing and fails open", async () => {- // The relay is what makes the tombstone honest: without it the commit- // has no material to key on, records nothing, and the session stays- // eligible - never suppressed on unverified evidence.+ test("a barren commit without the surveyed hash still tombstones the session", async () => {+ // The documented synopsis: plan, then commit the payload alone. The+ // plan's recorded survey is all the tombstone may need.+ await surveyedHash();const mine = await run([SESSION, "--payload", JSON.stringify({ items: [] }), "--json"]);expect(mine.returncode).toBe(0);const r = await run([SESSION, "--json"]);- const payload = JSON.parse(r.stdout) as { skipped?: unknown };+ const payload = JSON.parse(r.stdout) as { skipped?: { reason: string } };+ expect(payload.skipped?.reason).toBe("mining_tombstone");+ });++ test("a turn imported after the plan keeps a relay-less barren commit from tombstoning", async () => {+ await surveyedHash();+ importSessionRecall(vault, {+ sessionId: SESSION,+ turns: [+ {+ turnId: "t2",+ timestamp: "2026-08-22T09:02:00Z",+ role: "user",+ text: "Keep release notes in English.",+ },+ ],+ createdAt: "2026-08-22T10:00:00Z",+ });+ const mine = await run([SESSION, "--payload", JSON.stringify({ items: [] }), "--json"]);+ expect(mine.returncode).toBe(0);+ expect(+ (JSON.parse(mine.stdout) as { mining_ledger_failure?: string }).mining_ledger_failure,+ ).toContain("changed since the plan");+ const r = await run([SESSION, "--json"]);+ const payload = JSON.parse(r.stdout) as {+ skipped?: unknown;+ turns_mined: Array<{ turn_id: string }>;+ };expect(payload.skipped).toBeUndefined();+ expect(payload.turns_mined.map((t) => t.turn_id)).toEqual(["t1", "t2"]);+ });++ test("a malformed surveyed hash refuses the commit before any write", async () => {+ const mine = await run([+ SESSION,+ "--payload",+ JSON.stringify({ items: [ITEM] }),+ "--surveyed-turns-hash",+ "not-a-hash",+ "--json",+ ]);+ expect(mine.returncode).toBe(1);+ expect(mine.stdout).toContain("surveyed_turns_hash must be");+ const r = await run([SESSION, "--json"]);+ expect((JSON.parse(r.stdout) as { skipped?: unknown }).skipped).toBeUndefined();});test("text mode names the tombstone and the force escape", async () => {diff --git a/tests/cli/distill-trust-lane.test.ts b/tests/cli/distill-trust-lane.test.tsindex b1db6204..69a30651 100644--- a/tests/cli/distill-trust-lane.test.ts+++ b/tests/cli/distill-trust-lane.test.ts@@ -24,9 +24,10 @@** These tests pin the facts at once: the default resolves the gate off,* an ordinary search DOES return the untrusted page with that default,- * the legacy boolean alone now mounts the gate in shadow (the page still- * returns, the refusal is counted), and the same search stops returning- * it once the gate is named enforcing. The description is then required+ * the legacy boolean alone keeps the excluding behavior it had before the+ * mode existed, the shadow mode measures without excluding (the page+ * still returns, the refusal is counted), and the same search stops+ * returning it once the gate is named enforcing. The description is then required* to name that setting - so if the default is ever flipped, the last* test fails and the wording gets revisited rather than silently* becoming understated.@@ -114,11 +115,17 @@ describe("the quarantine is real, and it is opt-in", () => {expect(paths.some((p) => p.endsWith(path) || path.endsWith(p))).toBe(true);});- test("with the legacy boolean alone, the search still returns it - the gate measures in shadow", async () => {- // Session-mining-economy: the legacy boolean alone maps to the- // measure-only mode, so the page surfaces and the refusal is counted.+ test("with the legacy boolean alone, the search stops returning it", async () => {+ // A vault that turned the gate on before the mode existed keeps+ // excluding across the upgrade.const path = await seedDistillation(`${TRUST_GATE_CONFIG_KEY}: true\n`);const paths = await queryPaths();+ expect(paths.some((p) => p.endsWith(path) || path.endsWith(p))).toBe(false);+ });++ test("with the gate in shadow, the search still returns it and counts the refusal", async () => {+ const path = await seedDistillation(`search_trust_gate_mode: shadow\n`);+ const paths = await queryPaths();expect(paths.some((p) => p.endsWith(path) || path.endsWith(p))).toBe(true);const explain = await runCli(["search", CLAIM_TERM, "--json", "--explain", "--limit", "20"], {env: env(),diff --git a/tests/cli/progress-emitter-census.test.ts b/tests/cli/progress-emitter-census.test.tsindex e3270e26..0ff7b097 100644--- a/tests/cli/progress-emitter-census.test.ts+++ b/tests/cli/progress-emitter-census.test.ts@@ -244,7 +244,8 @@ const EMITTERS: Readonly<Record<string, EmitterSite>> = Object.freeze({operation: OPERATION.reindex,stage: "walk",},- "core/search/indexer.ts#reindexVault": {+ // reindexVault delegates to reindexVaultUnless, which owns the counter.+ "core/search/indexer.ts#reindexVaultUnless": {entryPoint: "o2b search reindex --progress",operation: OPERATION.reindex,stage: "lock",diff --git a/tests/cli/search-explain-trace.test.ts b/tests/cli/search-explain-trace.test.tsindex 2a3c5112..976e8e1d 100644--- a/tests/cli/search-explain-trace.test.ts+++ b/tests/cli/search-explain-trace.test.ts@@ -113,10 +113,14 @@ describe("o2b search --explain: retrieval receipts", () => {expect(payload["retrieval_decision_trace"]).toEqual(TRACE);expect(payload["memory_trust_assessment"]).toEqual(ASSESSMENT);expect("retrieval_trace_unavailable" in payload).toBe(false);+ // Stampless receipts come from the enforcing gate, and the envelope+ // names that mode.+ expect(payload["retrieval_trust_gate_mode"]).toBe("enforce");// The rest of the payload is untouched: the explain keys are appended.const off = JSON.stringify(jsonForOutcome(outcome(true)));delete payload["retrieval_decision_trace"];delete payload["memory_trust_assessment"];+ delete payload["retrieval_trust_gate_mode"];expect(JSON.stringify(payload)).toBe(off);});@@ -276,8 +280,8 @@ describe("o2b search --explain end to end", () => {"---\ntitle: Bad\nstatus: quarantine\n---\n\n# Bad\n\nThe widget calibration is unsafe.\n",);const configPath = join(configHome, "config.yaml");- // The gate must EXCLUDE here, and the legacy boolean alone now- // measures (shadow): enforcement is named explicitly.+ // The gate must EXCLUDE here; the mode is named explicitly so the+ // test does not lean on the legacy boolean mapping.atomicWriteFileSync(configPath,`vault: ${vault}\nsearch_trust_gate_enabled: true\nsearch_trust_gate_mode: enforce\n`,diff --git a/tests/cli/session-discovery.test.ts b/tests/cli/session-discovery.test.tsindex b1d740c9..5adf99dd 100644--- a/tests/cli/session-discovery.test.ts+++ b/tests/cli/session-discovery.test.ts@@ -37,6 +37,8 @@ import { brainDirs } from "../../src/core/brain/paths.ts";import { DEFAULT_BRAIN_CONFIG_YAML } from "../../src/core/brain/config-template.ts";import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";import { sessionLedgerPath } from "../../src/core/brain/sessions/discover.ts";+import { sessionMiningLedgerPath } from "../../src/core/brain/session-mining-ledger.ts";+import { sessionMiningSurveysPath } from "../../src/core/brain/session-mining-surveys.ts";import { CHMOD_CANNOT_DENY } from "../helpers/platform.ts";import { runCli } from "../helpers/run-cli.ts";import { fakeCredential } from "../helpers/fake-credentials.ts";@@ -77,6 +79,7 @@ interface DiscoveryJson {readonly gap: number;readonly by_runtime: ReadonlyArray<CoverageRow>;readonly mining?: MiningCensusJson;+ readonly mining_error?: string;readonly would_import?: ReadonlyArray<string>;readonly files?: ReadonlyArray<{ readonly file: string }>;}@@ -99,6 +102,13 @@ interface ProposeBacklogJson {readonly dedup_hash: string;}>;readonly report_only_sessions: ReadonlyArray<string>;+ readonly mining_steps: ReadonlyArray<{+ readonly session_id: string;+ readonly surveyed_turns_hash: string;+ readonly turns_mined: number;+ readonly llm_step: { readonly status: string; readonly prompt: string };+ }>;+ readonly mining_steps_remaining: number;}/**@@ -410,6 +420,36 @@ describe("the mining census rides the coverage report", () => {expect(status.mining?.undistilled).toBe(0);});+ test("an explicit recall id is recorded, so the census does not read the session as refused", async () => {+ const path = claudeLog("one.jsonl", "alpha");+ expect((await run([path, "--recall", "--recall-session-id", "pinned-one"])).returncode).toBe(0);++ const status = JSON.parse((await run(["--status", "--json"])).stdout) as DiscoveryJson;+ expect(status.mining?.undistilled).toBe(1);+ expect(status.mining?.refused).toBe(0);+ });++ test("a census failure leaves the coverage report standing and names the census unavailable", async () => {+ // A corrupt mining ledger is refused by the census by design; that is+ // a fact about the census, not about what was imported.+ claudeLog("one.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--recall", "--json"])).returncode).toBe(0);+ mkdirSync(dirname(sessionMiningLedgerPath(vault)), { recursive: true });+ writeFileSync(sessionMiningLedgerPath(vault), "{ truncated");++ const json = await run(["--status", "--json"]);+ expect(json.returncode).toBe(0);+ const body = JSON.parse(json.stdout) as DiscoveryJson;+ expect(body.imported).toBe(1);+ expect(body.mining).toBeUndefined();+ expect(body.mining_error).toContain("session mining ledger");++ const text = await run(["--discover"]);+ expect(text.returncode).toBe(0);+ expect(text.stdout).toContain("found: 1");+ expect(text.stdout).toContain("mining: unavailable (");+ });+test("the human rendering carries the mining lines per runtime", async () => {claudeLog("one.jsonl", "alpha");expect((await run(["--discover", "--all", "--recall"])).returncode).toBe(0);@@ -462,6 +502,78 @@ describe("--propose-backlog batches the distill proposals ask-once", () => {expect(body.report_only_sessions).toEqual(["one.jsonl", "two.jsonl"]);});+ test("an undistilled session gets a mining step, and answering it drains the backlog", async () => {+ // The free text a model would mine is what leaves a session+ // undistilled, so the envelope hands out the mining step for it; an+ // extract-signals commit of the answer (empty here) records the+ // session as mined and the backlog drains.+ claudeLog("one.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--recall", "--json"])).returncode).toBe(0);++ const before = JSON.parse(+ (await run(["--propose-backlog", "--json"])).stdout,+ ) as ProposeBacklogJson;+ expect(before.undistilled).toBe(1);+ expect(before.mining_steps.map((s) => s.session_id)).toEqual(["one.jsonl"]);+ expect(before.mining_steps[0]?.llm_step.status).toBe("needs-llm-step");+ expect(before.mining_steps_remaining).toBe(0);++ const commit = await runCli(+ [+ "brain",+ "extract-signals",+ "one.jsonl",+ "--payload",+ JSON.stringify({ items: [] }),+ "--vault",+ vault,+ ],+ { env: { HOME: home } },+ );+ expect(commit.returncode).toBe(0);++ const after = JSON.parse(+ (await run(["--propose-backlog", "--json"])).stdout,+ ) as ProposeBacklogJson;+ expect(after.undistilled).toBe(0);+ expect(after.mining_steps).toEqual([]);+ });++ test("the text form lists the mining steps without a session-placeholder command", async () => {+ claudeLog("one.jsonl", "alpha");+ expect((await run(["--discover", "--all", "--recall", "--json"])).returncode).toBe(0);++ const res = await run(["--propose-backlog"]);+ expect(res.returncode).toBe(0);+ expect(res.stdout).toContain("mining steps: 1");+ expect(res.stdout).toContain("mine: one.jsonl");+ expect(res.stdout).toContain(+ "an extract-signals commit of its envelope's payload (--json here hands out the envelopes)",+ );+ expect(res.stdout).not.toContain("o2b brain extract-signals <session>");+ // No envelope was handed out, so no survey is recorded that a barren+ // commit could tombstone unseen turns against.+ expect(existsSync(sessionMiningSurveysPath(vault))).toBe(false);+ });++ test("one session under two paths is one mining step with none remaining", async () => {+ // The census counts import-ledger paths; the same basename imported+ // from two project directories is one session identity, so it gets+ // one step and leaves nothing behind it.+ claudeLog("one.jsonl", "alpha");+ const other = join(home, ".claude", "projects", "-srv-projects-other");+ mkdirSync(other, { recursive: true });+ writeFileSync(join(other, "one.jsonl"), claudeLine("alpha", "u-alpha"));+ expect((await run(["--discover", "--all", "--recall", "--json"])).returncode).toBe(0);++ const body = JSON.parse(+ (await run(["--propose-backlog", "--json"])).stdout,+ ) as ProposeBacklogJson;+ expect(body.undistilled).toBe(2);+ expect(body.mining_steps.map((s) => s.session_id)).toEqual(["one.jsonl"]);+ expect(body.mining_steps_remaining).toBe(0);+ });+test("an empty backlog is an explicit empty envelope, not a failure", async () => {const res = await run(["--propose-backlog", "--json"]);expect(res.returncode).toBe(0);diff --git a/tests/core/architecture/state-surface-census.test.ts b/tests/core/architecture/state-surface-census.test.tsindex 68b5e074..1d60b1fa 100644--- a/tests/core/architecture/state-surface-census.test.ts+++ b/tests/core/architecture/state-surface-census.test.ts@@ -84,14 +84,17 @@ const STORE_DIR_IDENTIFIER = "DERIVED_STORE_DIR";// 25 -> 26: the storage usage meter joins, excused (storage lifecycle// external review, M3): it joins the directory to MEASURE the store and the// state beside it as census surfaces, and writes nothing.-const SWEPT_POPULATION_SIZE = 26;+// 26 -> 27: the pending mining surveys join the population with their row+// (v1.86.1 review fixes), the same join the session mining ledger made.+const SWEPT_POPULATION_SIZE = 27;/** Declared surfaces today. Pinned for the same reason. */// 50: the source-archives row (answer-trust Task C1) joins the population,// the same join the decisions ledger and the bootstrap receipt made.// 50 -> 51: the session mining ledger joins the population with its row// (session-mining-economy), the same join the session import ledger made.-const DECLARED_SURFACE_COUNT = 51;+// 51 -> 52: the pending mining surveys row (v1.86.1 review fixes).+const DECLARED_SURFACE_COUNT = 52;/** An exclusion reason has to be an argument, not a label. */const MIN_REASON_LENGTH = 80;diff --git a/tests/core/architecture/visibility-surface-census.test.ts b/tests/core/architecture/visibility-surface-census.test.tsindex 94ffd66e..dc0c6465 100644--- a/tests/core/architecture/visibility-surface-census.test.ts+++ b/tests/core/architecture/visibility-surface-census.test.ts@@ -972,7 +972,7 @@ const SURFACE_SOURCE_TREE = SURFACE_TREES.flatMap((root) => readTree(root));const DIRECT_VAULT_READERS = directVaultReadFiles(SURFACE_SOURCE_TREE);/** Measured: files in the surface trees that read a vault path directly. */-const DIRECT_VAULT_READ_POPULATION_SIZE = 6;+const DIRECT_VAULT_READ_POPULATION_SIZE = 5;/*** ## What this sweep cannot see, stated rather than implieddiff --git a/tests/core/architecture/write-site-census.test.ts b/tests/core/architecture/write-site-census.test.tsindex 9f8f935d..e85722e2 100644--- a/tests/core/architecture/write-site-census.test.ts+++ b/tests/core/architecture/write-site-census.test.ts@@ -588,11 +588,12 @@ const DIRECT_WRITE_EXCLUSIONS: Readonly<Record<string, WriteExclusion>> = Object"re-confirmed, then the source unlinked.",},"src/core/search/store/lifecycle.ts": {- categories: [C.lifecycleMove],- calls: ["renameSync"],+ categories: [C.lifecycleMove, C.retentionDelete],+ calls: ["renameSync", "unlinkSync"],reason:- "restores the `.bak` sqlite index over a missing database after a crashed " +- "reindex, re-checked under the lock. A database file has no text-writer form.",+ "restores the parked `.old` (else the `.bak`) sqlite index over a missing database " ++ "after a crashed reindex, re-checked under the lock, and deletes a `.old` a completed " ++ "swap left beside the live index. A database file has no text-writer form.",},// --- Retention, reset and cleanup deletes ------------------------------@@ -1426,8 +1427,13 @@ const DIRECT_WRITE_ROWS = 82;* writes its tombstone JSON through `atomicWriteFileSync` under the* ledger file's own lock - derived-store machine state, not a note, so* it arrives in the shared class needing no exclusion.+ *+ * 121 -> 122, from the v1.86.1 review fixes and nothing else: the pending+ * mining surveys (`src/core/brain/session-mining-surveys.ts`) write their+ * JSON through `atomicWriteFileSync` under the file's own lock - derived-+ * store machine state, not a note, needing no exclusion.*/-const SHARED_HELPER_ROWS = 121;+const SHARED_HELPER_ROWS = 122;// ----- Origin-channel coverage boundary (Unit C) ----------------------------@@ -1546,9 +1552,11 @@ const UNSTAMPED_DIRECT_ROWS = 81;* origin stamp on a fold rewrite). 117 -> 118: the session mining ledger* (`src/core/brain/session-mining-ledger.ts`) writes its tombstone JSON* through the shared atomic writer (derived-store machine state, not- * notes; no origin stamp on machine state).+ * notes; no origin stamp on machine state). 118 -> 119: the pending mining+ * surveys (`src/core/brain/session-mining-surveys.ts`) write their JSON+ * the same way (derived-store machine state; no origin stamp).*/-const UNSTAMPED_SHARED_ROWS = 118;+const UNSTAMPED_SHARED_ROWS = 119;describe("in-vault write-site census", () => {test("every direct-fs write site carries a written exclusion", () => {diff --git a/tests/core/brain/capture/inbox-drain.test.ts b/tests/core/brain/capture/inbox-drain.test.tsindex 1f2b64a7..f63831e0 100644--- a/tests/core/brain/capture/inbox-drain.test.ts+++ b/tests/core/brain/capture/inbox-drain.test.ts@@ -195,6 +195,34 @@ describe("the drain's ingest route under a capacity cap", () => {}});+ test("an invalid storage block refuses the ingest route, as brain_ingest_source does", () => {+ // pause_percent at or above critical is refused by the storage block's+ // own validation; reading it as "no cap" would ingest past the bound.+ writeFileSync(+ join(vault, "Brain", "_brain.yaml"),+ "schema_version: 1\n\nstorage:\n cap_bytes: 10\n pause_percent: 99\n",+ "utf8",+ );+ writeCaptureNote(vault, { body: "https://example.com/article", provenance: prov(1) });+ const report = drainInbox(vault, { apply: true, agent: "tester", now: NOW });+ expect(report.items[0]!.classification).toBe("unroutable");+ expect(report.items[0]!.reason).toContain("config_invalid");+ expect(existsSync(join(vault, "Brain", "sources"))).toBe(false);+ expect(listStagedCaptures(vault)).toHaveLength(1);+ });++ test("an invalid unrelated block does not wedge the ingest route", () => {+ writeFileSync(+ join(vault, "Brain", "_brain.yaml"),+ "schema_version: 1\n\nmaintenance:\n host_pressure_percent: not-a-number\n",+ "utf8",+ );+ writeCaptureNote(vault, { body: "https://example.com/article", provenance: prov(1) });+ const report = drainInbox(vault, { apply: true, agent: "tester", now: NOW });+ expect(report.items[0]!.classification).toBe("source-reference");+ expect(report.routed).toBe(1);+ });+test("the refusal never writes a summary page", () => {configureCriticalCap();writeCaptureNote(vault, { body: "https://example.com/article", provenance: prov(1) });diff --git a/tests/core/brain/continuity/reader-census.test.ts b/tests/core/brain/continuity/reader-census.test.tsindex b1be0ea1..3dee522d 100644--- a/tests/core/brain/continuity/reader-census.test.ts+++ b/tests/core/brain/continuity/reader-census.test.ts@@ -121,9 +121,6 @@ const DIRECT_STORE_READERS: Readonly<Record<string, string>> = Object.freeze({"src/core/brain/pre-compact-extract.ts":"reads `pre_compact_extract` for the append-time dedupe-key lookup.","src/core/brain/recall-telemetry.ts": "reads `recall_telemetry` for list and summary.",- "src/core/brain/session-mining-ledger.ts":- "reads the continuity store once per prune pass for the sessions that still " +- "hold turns, so a tombstone whose session lost its turns is dropped.","src/core/brain/recent-turns.ts": "reads `recent_turn` for the sequence counter and the window.","src/core/brain/session-recall.ts":"reads the WHOLE store for `session_turn` / `session_summary_node`, plus a " +@@ -168,7 +165,7 @@ const READ_MODEL_READER_COUNT = 4;// 27 -> 28: the session-mining-economy wave registers the explorer console// collector (`explorer-console.ts`), which reads every kind inside the// 30-day window for the written-per-day histogram.-const DIRECT_STORE_READER_COUNT = 29;+const DIRECT_STORE_READER_COUNT = 28;/** A file the classifier must see, and that `grep` alone does not. */diff --git a/tests/core/brain/digest-actions.test.ts b/tests/core/brain/digest-actions.test.tsindex 7436c3f8..122e3a5b 100644--- a/tests/core/brain/digest-actions.test.ts+++ b/tests/core/brain/digest-actions.test.ts@@ -23,7 +23,9 @@ let failDedup = false;mock.module("../../../src/core/brain/page-dedup.ts", () => ({...pageDedupModule,findDuplicateCandidates: (vault: string) => {- if (failDedup) throw new Error("the dedup scan refused");+ // The raised message carries a host path, as a real refusal's does.+ if (failDedup)+ throw new Error(`the dedup scan refused: ${join(vault, "Brain", "preferences")}`);return realFindDuplicateCandidates(vault);},}));@@ -92,12 +94,36 @@ describe("brain_digest collect_errors surface (review M10)", () => {collect_errors?: ReadonlyArray<{ scan: string; message: string }>;};expect(payload.collect_errors).toEqual([- { scan: "dedup", message: "the dedup scan refused" },+ {+ scan: "dedup",+ message: `the dedup scan refused: ${join(vault, "Brain", "preferences")}`,+ },]);} finally {failDedup = false;}});++ test("below local reach (a readable filter) the row is the closed form, never the host path", () => {+ failDedup = true;+ try {+ const res = renderDigest(vault, {+ format: "json",+ since: new Date("2026-05-24T00:00:00Z"),+ until: new Date("2026-05-25T00:00:00Z"),+ readable: () => true,+ });+ const payload = JSON.parse(res.content) as {+ collect_errors?: ReadonlyArray<{ scan: string; message: string }>;+ };+ expect(payload.collect_errors).toEqual([+ { scan: "dedup", message: "dedup scan failed (error)" },+ ]);+ expect(res.content).not.toContain(tmpdir());+ } finally {+ failDedup = false;+ }+ });});describe("brain_digest actions surface", () => {diff --git a/tests/core/brain/document-dedup-detector.test.ts b/tests/core/brain/document-dedup-detector.test.tsindex d5a35ecc..ba342ddb 100644--- a/tests/core/brain/document-dedup-detector.test.ts+++ b/tests/core/brain/document-dedup-detector.test.ts@@ -429,6 +429,31 @@ describe("detectSemanticDocumentDedup", () => {expect(result.findings[0]!.evidence["method"]).toBe("lexical");});+ test("a lexical fallback over a capped scan names the cap once", async () => {+ // The lexical layer already carries its coverage notice; wrapping its+ // result again added a second row with the same id.+ for (let i = 0; i < 201; i++) {+ const tag = `p${String(i).padStart(4, "0")}`;+ writeSourcePage(`filler-${String(i).padStart(4, "0")}`, {+ body: `${UNRELATED}\n${tag}a ${tag}b ${tag}c ${tag}d ${tag}e ${tag}f ${tag}g ${tag}h`,+ createdAt: "2026-01-01T00:00:00Z",+ });+ }+ const broken: EmbeddingProvider = {+ name: "broken",+ model: "broken",+ dimension: 3,+ embed: () => Promise.reject(new Error("provider down")),+ ping: () => Promise.resolve({ ok: false as const, reason: "down" }),+ };+ const result = await detectSemanticDocumentDedup(vault, { provider: broken });+ expect(result.method).toBe("lexical");+ const notices = result.findings.filter(+ (f) => (f.evidence as Record<string, unknown>)["candidate_cap_reached"] === true,+ );+ expect(notices).toHaveLength(1);+ });+test("no usable provider means the lexical layer", async () => {const result = await detectSemanticDocumentDedup(vault, { provider: null });expect(result.method).toBe("lexical");diff --git a/tests/core/brain/explorer.test.ts b/tests/core/brain/explorer.test.tsindex 3c96f19c..950b5eaf 100644--- a/tests/core/brain/explorer.test.ts+++ b/tests/core/brain/explorer.test.ts@@ -517,6 +517,31 @@ describe("buildLiveServer JSON routes", () => {}});+ test("a cross-origin request is refused before any route runs", async () => {+ // A web page's blind no-cors GET carries a loopback Host but names its+ // own site in Sec-Fetch-Site and Origin; it must not make the server+ // build the operator snapshot.+ const server = buildLiveServer(vault, 0);+ try {+ const crossSite = await fetch(new URL("/status.json", server.url), {+ headers: { "sec-fetch-site": "cross-site" },+ });+ expect(crossSite.status).toBe(403);+ expect(((await crossSite.json()) as { error: string }).error).toContain("cross-origin");+ const foreignOrigin = await fetch(new URL("/counts.json", server.url), {+ headers: { origin: "https://attacker.example" },+ });+ expect(foreignOrigin.status).toBe(403);+ // The console's own page fetching its routes is same-origin.+ const own = await fetch(new URL("/counts.json", server.url), {+ headers: { "sec-fetch-site": "same-origin", origin: `http://127.0.0.1:${server.port}` },+ });+ expect(own.ok).toBe(true);+ } finally {+ await server.close();+ }+ });+test("a loopback Host name with any port is accepted", async () => {writePreference(vault, basePref("hosted"));const server = buildLiveServer(vault, 0);diff --git a/tests/core/brain/extract-signals.test.ts b/tests/core/brain/extract-signals.test.tsindex b3c0ff18..3e76804f 100644--- a/tests/core/brain/extract-signals.test.ts+++ b/tests/core/brain/extract-signals.test.ts@@ -66,6 +66,7 @@ import {sessionMiningLedgerPath,type SessionMiningEntry,} from "../../../src/core/brain/session-mining-ledger.ts";+import { sessionMiningSurveysPath } from "../../../src/core/brain/session-mining-surveys.ts";import {ResponseCheckError,SEMANTIC_VIOLATION_CODES,@@ -634,15 +635,18 @@ test("the session mining ledger schema version is pinned at 1", () => {// ----- Record-at-commit (session-mining-economy) ----------------------------//-// 24. Every successful commit that answers a plan records its outcome-// breakdown over the surveyed hash: zero-yield as the tombstone's+// 24. Every successful commit records its outcome breakdown over the+// session's current mined turns: zero-yield as the tombstone's// evidence, yield as the distilled state the census reports.// 25. A commit that ends in a partial-write error records nothing - a// failed run is not an outcome.-// 26. The tombstone keys on the material the model actually answered, so a-// turn imported between plan and commit is never suppressed unseen; a-// commit that answers no plan of this module's attests nothing and-// records nothing, which is the safe direction.+// 26. A turn imported between plan and commit is never suppressed unseen:+// a relayed plan hash that no longer matches records nothing and names+// why. Without a relay the plan's recorded survey stands in for it: the+// documented flow (plan, then a barren commit) tombstones while the+// session is unchanged, and a session that changed since the plan, two+// pending plans over different turns, or no plan at all withholds the+// tombstone by name. A yield records either way.// 27. When the commit itself succeeds but the ledger write fails, the// result names the failure and nothing throws; the field is absent// whenever the ledger was not attempted or succeeded.@@ -746,10 +750,10 @@ test("a partial-write failure records nothing - a failed run is not a barren oneexpect(readSessionMiningLedger(vault).entries).toEqual({});});-test("the tombstone keys on the surveyed hash, not the turns present at commit time", () => {+test("a relayed plan hash the session outgrew records nothing and names why", () => {const plan = planExtractSignals(vault, SESSION, { now: NOW });importTurns(SESSION, [turn("t9", "user", "Imported after the plan was built.")]);- commitExtractedSignals(+ const res = commitExtractedSignals(vault,SESSION,{ items: [] },@@ -759,10 +763,10 @@ test("the tombstone keys on the surveyed hash, not the turns present at commit tsurveyedTurnsHash: plan.surveyedTurnsHash,},);- const entry = ledgerEntry()!;- // The verdict was earned over the planned material, and that is the- // material the record describes - never what the session holds by now.- expect(entry.minedContentHash).toBe(minedTurnsHash(plan.turnsMined));+ // The verdict was earned over material the session no longer holds, so+ // no record may claim the current turns.+ expect(ledgerEntry()).toBeUndefined();+ expect(res.miningLedgerFailure).toContain(plan.surveyedTurnsHash);});test("a turn imported between plan and commit is not suppressed unseen", async () => {@@ -788,12 +792,124 @@ test("a turn imported between plan and commit is not suppressed unseen", async (expect(next.turnsMined.map((t) => t.turnId)).toEqual(["t1", "t3", "t9"]);});-test("a commit that answers no plan attests nothing and records nothing", () => {- // Without a plan there is no material the verdict was earned over; a- // tombstone keyed on whatever the session holds now could suppress turns- // no model ever saw. Recording nothing keeps the session eligible.- commitExtractedSignals(vault, SESSION, { items: [] }, { agent: "tester", now: NOW });- expect(readSessionMiningLedger(vault).entries).toEqual({});+/** The plan phase both surfaces run: it records the survey it hands out. */+function surfacePlan(): ReturnType<typeof planExtractSignalsPrefiltered> {+ return planExtractSignalsPrefiltered(vault, SESSION, { now: NOW, decisionModel: null });+}++/** A commit that relays no plan hash, as the documented flow does. */+function commitWithoutRelay(+ items: ReadonlyArray<Record<string, unknown>>,+): ReturnType<typeof commitExtractedSignals> {+ return commitExtractedSignals(vault, SESSION, { items }, { agent: "tester", now: NOW });+}++test("the documented flow - plan, then a barren commit without a relay - tombstones", async () => {+ const plan = await surfacePlan();+ const res = commitWithoutRelay([]);+ expect(res.miningLedgerFailure).toBeUndefined();+ expect(ledgerEntry()?.minedContentHash).toBe(plan.surveyedTurnsHash);+ expect((await surfacePlan()).skipped?.reason).toBe(EXTRACT_MINING_TOMBSTONE_SKIP_REASON);+});++test("re-planning unchanged turns keeps the survey unambiguous", async () => {+ await surfacePlan();+ await surfacePlan();+ expect(commitWithoutRelay([]).miningLedgerFailure).toBeUndefined();+ expect(ledgerEntry()).toBeDefined();+});++test("a turn imported after the plan withholds a relay-less tombstone and names why", async () => {+ const plan = await surfacePlan();+ importTurns(SESSION, [turn("t9", "user", "Imported after the plan was built.")]);+ const res = commitWithoutRelay([]);+ expect(ledgerEntry()).toBeUndefined();+ expect(res.miningLedgerFailure).toContain("changed since the plan");+ expect(res.miningLedgerFailure).toContain(plan.surveyedTurnsHash);+ // The new turn was never offered to a model, so the next plan offers it.+ const next = await surfacePlan();+ expect(next.skipped).toBeUndefined();+ expect(next.turnsMined.map((t) => t.turnId)).toEqual(["t1", "t3", "t9"]);+});++test("two pending plans over different turns withhold a relay-less tombstone", async () => {+ await surfacePlan();+ importTurns(SESSION, [turn("t9", "user", "Imported between the two plans.")]);+ await surfacePlan();+ // The commit cannot tell which of the two plans it answers.+ const res = commitWithoutRelay([]);+ expect(ledgerEntry()).toBeUndefined();+ expect(res.miningLedgerFailure).toContain("2 plans over different turns");+});++test("a barren commit with no recorded plan withholds the tombstone and names why", () => {+ const res = commitWithoutRelay([]);+ expect(ledgerEntry()).toBeUndefined();+ expect(res.miningLedgerFailure).toContain("no plan of this session was recorded");+});++test("a commit consumes the survey, so a second barren commit has none to attest", async () => {+ await surfacePlan();+ commitWithoutRelay([item()]);+ const res = commitWithoutRelay([]);+ expect(res.miningLedgerFailure).toContain("no plan of this session was recorded");+ // The first commit's yield record stands; the barren one did not replace it.+ expect(ledgerEntry()!.outcome.written).toBe(1);+});++test("a yield records without a relay or a survey - only the tombstone needs one", () => {+ const res = commitWithoutRelay([item()]);+ expect(res.written).toHaveLength(1);+ expect(res.miningLedgerFailure).toBeUndefined();+ expect(ledgerEntry()!.outcome.written).toBe(1);+});++test("a relayed hash still tombstones when no survey was recorded", () => {+ const { res } = commitAnsweringPlan([]);+ expect(res.miningLedgerFailure).toBeUndefined();+ expect(ledgerEntry()).toBeDefined();+});++test("a survey the plan cannot record is named on the plan, and the relay still tombstones", async () => {+ mkdirSync(dirname(sessionMiningSurveysPath(vault)), { recursive: true });+ writeFileSync(sessionMiningSurveysPath(vault), "{ truncated");+ const plan = await surfacePlan();+ expect(plan.llmStep).not.toBeNull();+ expect(plan.surveyRecordFailure).toContain("session mining surveys file is corrupted JSON");+ // Without the relay the commit cannot attest the material...+ expect(commitWithoutRelay([]).miningLedgerFailure).toContain("could not be read");+ expect(ledgerEntry()).toBeUndefined();+ // ...and with it the tombstone lands.+ const relayed = commitExtractedSignals(+ vault,+ SESSION,+ { items: [] },+ { agent: "tester", now: NOW, surveyedTurnsHash: plan.surveyedTurnsHash },+ );+ expect(relayed.miningLedgerFailure).toBeUndefined();+ expect(ledgerEntry()).toBeDefined();+});++test("a damaged survey row reads as ambiguous, never as one clean survey", async () => {+ // A dropped row plus the next plan would look like a single survey and+ // could attest material an earlier plan never offered.+ mkdirSync(dirname(sessionMiningSurveysPath(vault)), { recursive: true });+ writeFileSync(+ sessionMiningSurveysPath(vault),+ JSON.stringify({ schema_version: 1, surveys: { [SESSION]: { hashes: "not-a-list" } } }),+ );+ await surfacePlan();+ expect(commitWithoutRelay([]).miningLedgerFailure).toContain("plans over different turns");+ expect(ledgerEntry()).toBeUndefined();+});++test("a tombstone-suppressed plan hands out no envelope and records no survey", async () => {+ await surfacePlan();+ commitWithoutRelay([]);+ const skipped = await surfacePlan();+ expect(skipped.skipped?.reason).toBe(EXTRACT_MINING_TOMBSTONE_SKIP_REASON);+ expect(existsSync(sessionMiningSurveysPath(vault))).toBe(true);+ expect(JSON.parse(readFileSync(sessionMiningSurveysPath(vault), "utf8")).surveys).toEqual({});});/** Plants a corrupt mining ledger so a read of it refuses by name. */@@ -846,3 +962,37 @@ test("the ledger-failure field is absent when the record succeeded", () => {expect(ledgerEntry()).toBeDefined();expect("miningLedgerFailure" in res).toBe(false);});++test("a barren commit whose ledger record fails keeps the survey, so its retry tombstones", async () => {+ const plan = await surfacePlan();+ seedCorruptLedger();+ expect(commitWithoutRelay([]).miningLedgerFailure).toContain(+ "session mining ledger is corrupted JSON",+ );+ // The operator clears the damaged ledger and retries the same answer+ // without a relay: the survey the failed commit read must still attest it.+ rmSync(sessionMiningLedgerPath(vault));+ const retry = commitWithoutRelay([]);+ expect(retry.miningLedgerFailure).toBeUndefined();+ expect(ledgerEntry()?.minedContentHash).toBe(plan.surveyedTurnsHash);+});++test("below local reach a survey store failure names no host path", async () => {+ mkdirSync(dirname(sessionMiningSurveysPath(vault)), { recursive: true });+ writeFileSync(sessionMiningSurveysPath(vault), "{ truncated");+ const plan = await planExtractSignalsPrefiltered(vault, SESSION, {+ now: NOW,+ decisionModel: null,+ closeHostErrors: true,+ });+ expect(plan.surveyRecordFailure).toBe("session mining surveys store failed (error)");+ const res = commitExtractedSignals(+ vault,+ SESSION,+ { items: [] },+ { agent: "tester", now: NOW, closeHostErrors: true },+ );+ expect(res.miningLedgerFailure).toContain("session mining surveys store failed (error)");+ expect(res.miningLedgerFailure).not.toContain(vault);+ expect(res.miningLedgerFailure).not.toContain(tmpdir());+});diff --git a/tests/core/brain/ingest/content-manifest.test.ts b/tests/core/brain/ingest/content-manifest.test.tsindex b4db168f..054693ab 100644--- a/tests/core/brain/ingest/content-manifest.test.ts+++ b/tests/core/brain/ingest/content-manifest.test.ts@@ -21,12 +21,14 @@ import {hashFile,hashTree,manifestPath,+ markManifestRetired,readManifest,updateManifest,writeManifestAtomic,} from "../../../../src/core/brain/ingest/content-manifest.ts";import { planBatches } from "../../../../src/core/brain/ingest/batch-plan.ts";import { deleteBySource } from "../../../../src/core/brain/source-cleanup.ts";+import { acquireLockSync, LOCK_WAIT_BUDGET_ENV } from "../../../../src/core/brain/sync-lockfile.ts";let vault: string;let configHome: string;@@ -665,3 +667,33 @@ describe("ingestSource integration", () => {expect(readManifest(vault).entries).toEqual({});});});++describe("markManifestRetired", () => {+ test("marks the entry inside the manifest lock and keeps every other field", () => {+ writeSource("imports/a.md", "alpha\n");+ updateManifest(vault, ["imports/a.md"]);+ const before = readManifest(vault).entries["imports/a.md"]!;++ // While another writer holds the manifest lock, the mark waits for it+ // (here with no budget, so it refuses) instead of writing past it.+ const saved = process.env[LOCK_WAIT_BUDGET_ENV];+ process.env[LOCK_WAIT_BUDGET_ENV] = "0";+ const held = acquireLockSync(manifestPath(vault));+ try {+ expect(() => markManifestRetired(vault, "imports/a.md", "Brain/sources/canon.md")).toThrow();+ expect(readManifest(vault).entries["imports/a.md"]!.retired_into).toBeUndefined();+ } finally {+ held.release();+ if (saved === undefined) delete process.env[LOCK_WAIT_BUDGET_ENV];+ else process.env[LOCK_WAIT_BUDGET_ENV] = saved;+ }++ expect(markManifestRetired(vault, "imports/a.md", "Brain/sources/canon.md")).toBe("marked");+ const after = readManifest(vault).entries["imports/a.md"]!;+ expect(after).toEqual({ ...before, retired_into: "Brain/sources/canon.md" });+ expect(markManifestRetired(vault, "imports/a.md", "Brain/sources/canon.md")).toBe(+ "already-marked",+ );+ expect(markManifestRetired(vault, "imports/missing.md", "x.md")).toBe("no-entry");+ });+});diff --git a/tests/core/brain/ingest/incremental-extract.test.ts b/tests/core/brain/ingest/incremental-extract.test.tsindex 06cd38cb..6fb0e775 100644--- a/tests/core/brain/ingest/incremental-extract.test.ts+++ b/tests/core/brain/ingest/incremental-extract.test.ts@@ -211,7 +211,9 @@ describe("ingestSource watermark stamping", () => {// starts AT it instead of past it.const entry = readManifest(vault).entries[SOURCE]!;expect(entry.extracted_bytes).toBe(8);- expect(entry.sha256).toBe(hashBytes(new Uint8Array(Buffer.from("one\ntwo\n", "utf8"))));+ expect(entry.extracted_sha256).toBe(+ hashBytes(new Uint8Array(Buffer.from("one\ntwo\n", "utf8"))),+ );});test("a fragment stamp keeps the prefix proof consistent and the next delta stitches the fragment", () => {@@ -227,15 +229,20 @@ describe("ingestSource watermark stamping", () => {expect(second.extractionDelta).toEqual({ kind: "appended", fromByte: 5, toByte: 11 });// The stamp lands at the announced boundary (11), NOT at the file- // size (21), and the digest describes exactly those leading bytes.+ // size (21), and the prefix digest describes exactly those leading+ // bytes; the whole-file digest stays what classification compares.const stamped = readManifest(vault).entries[SOURCE]!;expect(stamped.extracted_bytes).toBe(11);- expect(stamped.sha256).toBe(hashBytes(new Uint8Array(Buffer.from("base\nline1\n", "utf8"))));+ expect(stamped.extracted_sha256).toBe(+ hashBytes(new Uint8Array(Buffer.from("base\nline1\n", "utf8"))),+ );+ expect(stamped.sha256).toBe(hashFile(join(vault, SOURCE)));- // Accepted consequence (review M15): with the stamp behind EOF the- // file classifies modified until the fragment is terminated - one- // reclassification, never a stale skip.- expect(classifyPaths(vault, [SOURCE], readManifest(vault)).modified).toEqual([SOURCE]);+ // The unterminated fragment does not re-queue the source: unchanged+ // bytes classify unchanged, so no batch plan pays for it again.+ expect(classifyPaths(vault, [SOURCE], readManifest(vault)).unchanged).toEqual([SOURCE]);+ const replan = planBatches(vault, "imports", { maxBatchBytes: 10_000, maxBatchFiles: 100 });+ expect(replan.batches.flatMap((b) => b.files).map((f) => f.path)).toEqual([]);appendFileSync(join(vault, SOURCE), "-rest\nline3\n", "utf8");const third = ingestSource(vault, INPUT, OPTS);@@ -244,13 +251,38 @@ describe("ingestSource watermark stamping", () => {expect(third.extractionDelta).toEqual({ kind: "appended", fromByte: 11, toByte: 33 });const healed = readManifest(vault).entries[SOURCE]!;expect(healed.extracted_bytes).toBe(33);- // The boundary is back at EOF, so the digest is a whole-file digest- // again and the unchanged classification holds.+ // The boundary is back at EOF, so the one whole-file digest describes+ // the extracted prefix too and no separate prefix digest is kept.expect(healed.sha256).toBe(hashFile(join(vault, SOURCE)));+ expect(healed.extracted_sha256).toBeUndefined();expect(classifyPaths(vault, [SOURCE], readManifest(vault)).unchanged).toEqual([SOURCE]);});});+describe("an appended tail longer than one read chunk", () => {+ test("the prefix digest covers the real appended bytes", () => {+ writeSource("base\n");+ ingestSource(vault, INPUT, OPTS);+ // Distinct lines well past one 64 KiB read, ending in a fragment, so+ // the stamp's prefix digest is computed over a multi-chunk tail.+ const lines = Array.from({ length: 6000 }, (_, i) => `line ${i} of the appended tail\n`).join(+ "",+ );+ appendFileSync(join(vault, SOURCE), `${lines}frag`, "utf8");++ const res = ingestSource(vault, INPUT, OPTS);+ const prefix = `base\n${lines}`;+ expect(res.extractionDelta).toEqual({+ kind: "appended",+ fromByte: 5,+ toByte: Buffer.byteLength(prefix, "utf8"),+ });+ expect(readManifest(vault).entries[SOURCE]!.extracted_sha256).toBe(+ hashBytes(new Uint8Array(Buffer.from(prefix, "utf8"))),+ );+ });+});+describe("the watermark never moves the extraction contract", () => {test("EXTRACT_CONTRACT_VERSION stays at 2 and a watermarked source reprocesses nothing", () => {expect(EXTRACT_CONTRACT_VERSION).toBe(2);diff --git a/tests/core/brain/metrics.test.ts b/tests/core/brain/metrics.test.tsindex cd0bb047..2d41e702 100644--- a/tests/core/brain/metrics.test.ts+++ b/tests/core/brain/metrics.test.ts@@ -8,7 +8,15 @@*/import { afterEach, beforeEach, describe, expect, test } from "bun:test";-import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync, mkdirSync } from "node:fs";+import {+ existsSync,+ mkdtempSync,+ readdirSync,+ readFileSync,+ rmSync,+ writeFileSync,+ mkdirSync,+} from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";@@ -17,6 +25,7 @@ import {listMetrics,METRICS_SCHEMA_VERSION,MetricSurfaceError,+ metricsDir,} from "../../../src/core/brain/metrics.ts";const NOW = "2026-06-05T10:00:00Z";@@ -32,6 +41,47 @@ afterEach(() => {});describe("appendMetric", () => {+ test("waits out a lock another writer briefly holds instead of losing the sample", async () => {+ appendMetric(vault, { surface: "recall_benchmark", runAt: NOW, payload: { n: 1 } });+ const file = join(metricsDir(vault), readdirSync(metricsDir(vault))[0]!);+ const marker = join(vault, "holder-ready");+ const go = join(vault, "parent-appending");+ const child = join(vault, "holder.ts");+ // The holder keeps the lock until the parent says it is about to+ // append, and only then lets go after a short hold - so the append+ // always starts against a held lock, however slowly either side runs.+ writeFileSync(+ child,+ [+ `import { acquireLockSync } from ${JSON.stringify(join(import.meta.dir, "../../../src/core/brain/sync-lockfile.ts"))};`,+ `const { existsSync, writeFileSync } = require("node:fs");`,+ "const [target, marker, go] = process.argv.slice(2);",+ "const h = acquireLockSync(target);",+ `writeFileSync(marker, "held");`,+ "while (!existsSync(go)) Bun.sleepSync(5);",+ "Bun.sleepSync(250);",+ "h.release();",+ ].join("\n"),+ "utf8",+ );+ const proc = Bun.spawn(["bun", child, file, marker, go], { stdout: "pipe", stderr: "pipe" });+ try {+ const until = Date.now() + 20_000;+ while (!existsSync(marker)) {+ if (Date.now() > until) throw new Error("the holder never took the lock");+ // oxlint-disable-next-line no-await-in-loop -- polling the child's progress+ await Bun.sleep(5);+ }+ writeFileSync(go, "now");+ appendMetric(vault, { surface: "recall_benchmark", runAt: NOW, payload: { n: 2 } });+ expect(listMetrics(vault, { surface: "recall_benchmark" })).toHaveLength(2);+ } finally {+ // Never leave the holder waiting on a marker an early failure skipped.+ if (!existsSync(go)) writeFileSync(go, "abort");+ await proc.exited;+ }+ });+test("writes one schema-stamped line per call into the surface file", () => {appendMetric(vault, { surface: "recall_benchmark", runAt: NOW, payload: { mrr: 0.91 } });appendMetric(vault, {diff --git a/tests/core/brain/receipts-compact.test.ts b/tests/core/brain/receipts-compact.test.tsindex a6d5ab87..ed58a1f1 100644--- a/tests/core/brain/receipts-compact.test.ts+++ b/tests/core/brain/receipts-compact.test.ts@@ -527,6 +527,8 @@ describe("compactReceiptLedgers", () => {dryRun: false,});expect(result.totals.folded).toBe(1);+ // The prior aggregate was merged away, not kept verbatim.+ expect(result.totals.kept).toBe(0);// The small fold merged instead of failing the strict-shrink guard.expect(result.failures).toEqual([]);const lines = readShard(continuityLogDir(vault), "2026-08.jsonl")diff --git a/tests/core/brain/receipts-scan.test.ts b/tests/core/brain/receipts-scan.test.tsindex 316d61ea..e6be217f 100644--- a/tests/core/brain/receipts-scan.test.ts+++ b/tests/core/brain/receipts-scan.test.ts@@ -286,6 +286,14 @@ describe("scanReceiptLedgers", () => {// been given, which a fold past the cutoff would destroy.expect(KEPT_CONTINUITY_KINDS).toContain("recall_observed_use");expect(KEPT_CONTINUITY_KINDS).toContain("negative_recall");+ // `skill_invoked` is the lifetime usage `deriveSkillUsage` ranks the+ // skill offers by; a fold would collapse it to the last window.+ expect(+ classifyContinuityRecord(+ { kind: "skill_invoked", createdAt: "2026-08-01T00:00:00Z" },+ cutoffMs,+ ),+ ).toBe("protected");expect(FOLDABLE_CONTINUITY_KINDS).not.toContain("recall_observed_use");expect(FOLDABLE_CONTINUITY_KINDS).not.toContain("negative_recall");});diff --git a/tests/core/brain/session-mining-ledger.test.ts b/tests/core/brain/session-mining-ledger.test.tsindex ac4621e4..4f5c7ecc 100644--- a/tests/core/brain/session-mining-ledger.test.ts+++ b/tests/core/brain/session-mining-ledger.test.ts@@ -124,6 +124,11 @@ test("a corrupt ledger refuses by name and is never reset", () => {writeFileSync(path, "not json at all");expect(() => readSessionMiningLedger(vault)).toThrow(/session mining ledger is corrupted JSON/);expect(() => readSessionMiningLedger(vault)).toThrow(/session-mining-ledger\.json/);+ // The refusal stops every plan and the census, so it names the remedy+ // and what the remedy costs.+ expect(() => readSessionMiningLedger(vault)).toThrow(+ /delete it - every imported session then becomes eligible for mining again/,+ );// The refused read rewrote nothing: the file is exactly as corrupt as before.expect(readFileSync(path, "utf8")).toBe("not json at all");});diff --git a/tests/core/brain/session-mining-surveys.test.ts b/tests/core/brain/session-mining-surveys.test.tsnew file mode 100644index 00000000..594aa9b7--- /dev/null+++ b/tests/core/brain/session-mining-surveys.test.ts@@ -0,0 +1,54 @@+/**+ * The pending mining-survey store (session-mining-surveys.ts), pinned at+ * its own seam where the commit flow cannot reach: a session id is caller+ * data, so an id that names an Object prototype key must store and read+ * back like any other, and a commit clears only the hashes it answered.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, readFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ clearMiningSurvey,+ readMiningSurvey,+ recordMiningSurvey,+ sessionMiningSurveysPath,+} from "../../../src/core/brain/session-mining-surveys.ts";++const NOW = new Date("2026-10-16T08:00:00Z");+const HASH_A = "a".repeat(64);+const HASH_B = "b".repeat(64);++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-mining-surveys-"));+ mkdirSync(join(vault, "Brain"), { recursive: true });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++test("a __proto__ session id is stored as an own key and reads back", () => {+ recordMiningSurvey(vault, "__proto__", HASH_A, NOW);+ expect(readMiningSurvey(vault, "__proto__")).toEqual([HASH_A]);+ const onDisk = JSON.parse(readFileSync(sessionMiningSurveysPath(vault), "utf8")) as {+ surveys: Record<string, unknown>;+ };+ expect(Object.keys(onDisk.surveys)).toEqual(["__proto__"]);+ // A prototype member is no survey of a session that was never planned.+ expect(readMiningSurvey(vault, "constructor")).toEqual([]);+});++test("clearing removes only the answered hashes, so a racing plan stays pending", () => {+ recordMiningSurvey(vault, "sess", HASH_A, NOW);+ const answered = readMiningSurvey(vault, "sess");+ recordMiningSurvey(vault, "sess", HASH_B, NOW);+ clearMiningSurvey(vault, "sess", answered);+ expect(readMiningSurvey(vault, "sess")).toEqual([HASH_B]);+ clearMiningSurvey(vault, "sess", [HASH_B]);+ expect(readMiningSurvey(vault, "sess")).toEqual([]);+});diff --git a/tests/core/brain/sessions/mining-census.test.ts b/tests/core/brain/sessions/mining-census.test.tsindex b528d049..ccb8f74a 100644--- a/tests/core/brain/sessions/mining-census.test.ts+++ b/tests/core/brain/sessions/mining-census.test.ts@@ -22,7 +22,7 @@* Nothing here writes either.*/-import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test";import { mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";import { tmpdir } from "node:os";import { join } from "node:path";@@ -48,6 +48,8 @@ import { importSession } from "../../../../src/core/brain/sessions/import.ts";import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";import { SESSION_RUNTIME_ID } from "../../../../src/core/runtime/host-facts.ts";import { DERIVED_STORE_DIR } from "../../../../src/core/brain/path-constants.ts";+import * as continuityStore from "../../../../src/core/brain/continuity/store.ts";+import type { AppendContinuityRecordInput } from "../../../../src/core/brain/continuity/types.ts";let tmp: string;let vault: string;@@ -310,6 +312,26 @@ describe("buildMiningCensus", () => {expect(census.distilled).toBe(1);});+ test("an import under an explicit recall id joins on that id, not the basename", async () => {+ // --recall-session-id files the turns and every mining record under+ // the pinned id; the census must follow it rather than read the log's+ // basename as a session the plan would refuse.+ const path = writeSession("pinned.jsonl", [{ role: "user", text: "ship the parser" }]);+ await importSession(vault, path, {+ agent: "tester",+ now: NOW,+ recall: true,+ recallSessionId: "pinned-session",+ });+ recordSessionImports(vault, [+ { path, runtime: SESSION_RUNTIME_ID.claudeCode, sessionId: "pinned-session" },+ ]);+ expect(buildMiningCensus(vault)).toMatchObject({ undistilled: 1, refused: 0 });++ mineYield("pinned-session");+ expect(buildMiningCensus(vault)).toMatchObject({ distilled: 1, undistilled: 0, refused: 0 });+ });+test("counts are aggregated per runtime, by the ledger's own label", async () => {await importOne(writeSession("alpha.jsonl", [{ role: "user", text: "ship the parser" }]));const beta = writeSession("beta.jsonl", [{ role: "user", text: "write the docs" }]);@@ -356,4 +378,62 @@ describe("buildMiningCensus", () => {writeFileSync(sessionLedgerPath(vault), "{ this is not json", "utf8");expect(() => buildMiningCensus(vault)).toThrow(/session import ledger is corrupted JSON/);});++ test("the census reads the continuity store a fixed number of times, whatever the backlog size", () => {+ // A census that re-read the store once per imported session costs+ // sessions x store - quadratic in the backlog - and spends the whole+ // SessionStart hook budget on a realistic sweep. Seed a few sessions+ // and a few hundred, and pin that the store reads do not grow.+ const seed = (count: number, offset: number): void => {+ const paths: string[] = [];+ const turns: AppendContinuityRecordInput[] = [];+ for (let i = offset; i < offset + count; i++) {+ const name = `scale-${i}.jsonl`;+ const path = join(sessionsDir, name);+ writeFileSync(path, "{}\n", "utf8");+ paths.push(path);+ for (let t = 0; t < 3; t++) {+ turns.push({+ kind: "session_turn",+ createdAt: "2026-09-18T09:00:00Z",+ payload: {+ session_id: name,+ turn_id: `t-${t}`,+ timestamp: "2026-09-18T09:00:00Z",+ role: t % 2 === 0 ? "user" : "assistant",+ text: `turn ${t} of session ${i}`,+ },+ });+ }+ }+ continuityStore.appendContinuityRecords(vault, turns);+ recordSessionImports(+ vault,+ paths.map((path) => ({ path, runtime: SESSION_RUNTIME_ID.claudeCode })),+ NOW,+ );+ };+ // The spied reader is the continuity store's full listing - the one+ // whole-store read the census is allowed to make, once per build.+ const storeReads = (): number => {+ const spy = spyOn(continuityStore, "listContinuityRecords");+ try {+ buildMiningCensus(vault);+ return spy.mock.calls.length;+ } finally {+ spy.mockRestore();+ }+ };++ seed(3, 0);+ const small = storeReads();+ // A census that stopped reading the store would pass the equality+ // below vacuously at zero.+ expect(small).toBeGreaterThan(0);+ expect(buildMiningCensus(vault).undistilled).toBe(3);+ seed(300, 3);+ const large = storeReads();+ expect(buildMiningCensus(vault).undistilled).toBe(303);+ expect(large).toBe(small);+ });});diff --git a/tests/core/brain/storage-usage.test.ts b/tests/core/brain/storage-usage.test.tsindex 5de4426a..2ce96bd7 100644--- a/tests/core/brain/storage-usage.test.ts+++ b/tests/core/brain/storage-usage.test.ts@@ -36,7 +36,12 @@ import {measureStorageUsage,type StorageUsage,} from "../../../src/core/brain/storage/usage.ts";-import { evaluateCapacityGate } from "../../../src/core/brain/storage/gate.ts";+import {+ evaluateCapacityGate,+ evaluateIngestAdmission,+} from "../../../src/core/brain/storage/gate.ts";+import { sampleStorageUsage } from "../../../src/core/brain/storage/forecast.ts";+import { runMaintenance } from "../../../src/core/brain/maintenance/lane.ts";import {STORAGE_CAPACITY_CRITICAL_CODE,storageCapacityCheck,@@ -275,6 +280,38 @@ describe("the configured store location", () => {}});+ test("an explicit config file moves the store for the lane gate, the ingest wall and the sampler", async () => {+ // A server started with --config holds that path; every census it+ // takes must measure the store THAT file names, or the doctor and the+ // gates disagree about the same vault.+ const configHome = mkdtempSync(join(tmpdir(), "o2b-store-cfgpath-gates-"));+ try {+ mkdirSync(join(vault, "Brain", "sources"), { recursive: true });+ writeFileSync(brainConfigPath(vault), "schema_version: 1\n\nstorage:\n cap_bytes: 10000\n");+ writeFileSync(join(vault, "Brain", "sources", "AA-page.md"), "x".repeat(6000), "utf8");+ const movedDb = join(configHome, "moved.sqlite");+ writeFileSync(movedDb, "x".repeat(3500), "utf8");+ const configPath = join(configHome, "config.yaml");+ writeFileSync(configPath, `vault: ${vault}\nsearch_db_path: ${movedDb}\n`, "utf8");++ const admission = evaluateIngestAdmission(vault, { now: NOW, configPath });+ expect(admission.decision?.decision).toBe("skip");++ const sample = sampleStorageUsage(vault, { now: NOW, configPath });+ expect(surfaceBytes(sample.usage, STORAGE_SURFACE.derivedStore)).toBe(3500);++ const lane = await runMaintenance(vault, {+ now: NOW,+ holder: "tester",+ tasks: [],+ configPath,+ });+ expect(lane.capacity?.tier).toBe("critical");+ } finally {+ rmSync(configHome, { recursive: true, force: true });+ }+ });+test("under an override the lane gate and the doctor check agree", () => {// The doctor measures the store the operator's searches actually use// (ctx.dbPath); pre-fix the lane gate measured the default locationdiff --git a/tests/core/brain/sync-lockfile.test.ts b/tests/core/brain/sync-lockfile.test.tsindex bc5656b6..e08454cb 100644--- a/tests/core/brain/sync-lockfile.test.ts+++ b/tests/core/brain/sync-lockfile.test.ts@@ -243,6 +243,7 @@ describe("every module that takes a lock is under a scanned root", () => {["core/brain/query-demand.ts", "Brain"],["core/brain/receipts/compact.ts", "Brain"],["core/brain/session-mining-ledger.ts", ".open-second-brain"],+ ["core/brain/session-mining-surveys.ts", ".open-second-brain"],["core/brain/sessions/discover.ts", ".open-second-brain"],["core/brain/skill-proposals.ts", "Brain"],]);diff --git a/tests/core/brain/trust/operator-summary.test.ts b/tests/core/brain/trust/operator-summary.test.tsindex 5fc1617d..53e37a74 100644--- a/tests/core/brain/trust/operator-summary.test.ts+++ b/tests/core/brain/trust/operator-summary.test.ts@@ -110,6 +110,19 @@ describe("buildOperatorSummary - collect_errors (review M10)", () => {}});+ test("below local reach (a readable filter) each row is the closed form, never the host path", () => {+ rmSync(join(vault, "Brain", "preferences"), { recursive: true });+ writeFileSync(join(vault, "Brain", "preferences"), "not a directory", "utf8");++ const r = buildOperatorSummary(vault, { readable: () => true });+ const errors = r.collect_errors ?? [];+ expect(errors.some((e) => e.scan === "dedup")).toBe(true);+ for (const e of errors) {+ expect(e.message).toMatch(new RegExp(`^${e.scan} scan failed \\(`));+ expect(e.message).not.toContain(tmpdir());+ }+ });+test("a healthy vault carries no collect_errors row", () => {const r = buildOperatorSummary(vault, {});expect(r.collect_errors).toBeUndefined();diff --git a/tests/core/maintenance/ensure-current.test.ts b/tests/core/maintenance/ensure-current.test.tsindex 9ff686f4..7ebdc479 100644--- a/tests/core/maintenance/ensure-current.test.ts+++ b/tests/core/maintenance/ensure-current.test.ts@@ -19,6 +19,12 @@ import { resolveSearchConfig } from "../../../src/core/search/index.ts";import { CHUNKER_VERSION } from "../../../src/core/search/chunker.ts";import { indexVault } from "../../../src/core/search/indexer.ts";import { LATEST_SCHEMA_VERSION, readSchemaVersion } from "../../../src/core/search/schema.ts";+import { Store } from "../../../src/core/search/store.ts";+import { FAKE_PROVIDER_KEY } from "../../helpers/fake-credentials.ts";+import { startFakeHttp } from "../../helpers/fake-http.ts";+import { rewindToPreV14Schema, stampSchemaVersion } from "../../helpers/schema-rewind.ts";+import { makeConfig } from "../../helpers/search-fixtures.ts";+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";let vault: string;let configHome: string;@@ -86,18 +92,70 @@ describe("ensureVaultCurrent", () => {expect(r.errors).toEqual([]);});- test("rebuilds a stale-schema index", async () => {+ test("rebuilds an index stamped by a newer schema", async () => {bootstrapBrain(vault, { configPath });await ensureVaultCurrent(vault, { background: false }); // build a current index- const db = new Database(dbPath());- db.run("UPDATE index_state SET value = '1' WHERE key = 'schema_version'");- db.close();+ stampSchemaVersion(dbPath(), LATEST_SCHEMA_VERSION + 1);const r = await ensureVaultCurrent(vault, { background: false });expect(r.reindexTriggered).toBe(true);expect(indexSchema()).toBe(LATEST_SCHEMA_VERSION);});+ test.skipIf(!sqliteVecLoadable())(+ "upgrades an older-schema semantic index in place, keeping its embeddings (#256)",+ async () => {+ bootstrapBrain(vault, { configPath });+ const server = await startFakeHttp();+ try {+ const semantic = makeConfig({+ vault,+ dbPath: dbPath(),+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: FAKE_PROVIDER_KEY,+ dimension: 4,+ queryPrefix: "query: ",+ passagePrefix: "passage: ",+ maxRetries: 1,+ },+ });+ atomicWriteFileSync(join(vault, "fox.md"), "# Fox\n\nThe quick brown fox.\n");+ await indexVault(semantic, { embeddings: true });+ const identity = async () => {+ const s = await Store.open(semantic, { mode: "read" });+ try {+ return [+ s.countEmbeddings(),+ s.getState("embedding_prefix_query"),+ s.getState("embedding_prefix_passage"),+ ];+ } finally {+ await s.close();+ }+ };+ const before = await identity();+ expect(before[0]).toBeGreaterThan(0);+ rewindToPreV14Schema(dbPath());++ // The session-start config is keyword-only: it must not touch the+ // embedding identity while it upgrades the index.+ const r = await ensureVaultCurrent(vault, { background: false });++ expect(r.errors).toEqual([]);+ expect(r.indexUpgraded).toBe(true);+ expect(r.reindexTriggered).toBe(false);+ expect(indexSchema()).toBe(LATEST_SCHEMA_VERSION);+ expect(await identity()).toEqual(before);+ } finally {+ await server.close();+ }+ },+ );+test("honors an explicit configPath for the search index", async () => {bootstrapBrain(vault, { configPath });// A second config pointing the index at a custom path.@@ -253,12 +311,27 @@ describe("ensureVaultCurrent: freshen on session start", () => {expect(calls).toHaveLength(0);});+ test.skipIf(!sqliteVecLoadable())(+ "an index upgraded in place that has gone stale is still caught up",+ async () => {+ // The in-place upgrade touches no document, so it must not stand in+ // for the incremental run the old full rebuild used to be.+ bootstrapBrain(vault, { configPath });+ await ensureVaultCurrent(vault, { background: false });+ ageIndex(300);+ rewindToPreV14Schema(dbPath());+ const r = await ensureVaultCurrent(vault, { background: false, freshenSpawn });+ expect(r.indexUpgraded).toBe(true);+ expect(r.reindexTriggered).toBe(false);+ expect(r.freshen).toBe("spawned");+ expect(calls).toHaveLength(1);+ },+ );+test("an index that needs a rebuild is left to self-heal, not freshened", async () => {bootstrapBrain(vault, { configPath });await ensureVaultCurrent(vault, { background: false });- const db = new Database(dbPath());- db.run("UPDATE index_state SET value = '1' WHERE key = 'schema_version'");- db.close();+ stampSchemaVersion(dbPath(), LATEST_SCHEMA_VERSION + 1);const r = await ensureVaultCurrent(vault, { background: false, freshenSpawn });expect(r.reindexTriggered).toBe(true);expect(r.freshen).toBeNull();diff --git a/tests/core/maintenance/self-heal-reindex.test.ts b/tests/core/maintenance/self-heal-reindex.test.tsindex b39f6e5f..79812018 100644--- a/tests/core/maintenance/self-heal-reindex.test.ts+++ b/tests/core/maintenance/self-heal-reindex.test.ts@@ -33,6 +33,13 @@ import { cmdSearchReindex } from "../../../src/cli/search/verbs/indexing.ts";import { resolveSearchConfig } from "../../../src/core/search/index.ts";import { acquireWriterLock } from "../../../src/core/search/store/writer-lock.ts";import { waitForSelfHealUpgradeWorker } from "../../helpers/self-heal-children.ts";+import { indexVault } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import { FAKE_PROVIDER_KEY } from "../../helpers/fake-credentials.ts";+import { startFakeHttp } from "../../helpers/fake-http.ts";+import { rewindToPreV14Schema } from "../../helpers/schema-rewind.ts";+import { makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";let vault: string;let configHome: string;@@ -194,6 +201,75 @@ describe("self-heal reindex: the spawn decision", () => {});describe("self-heal reindex: the child's terminal outcome", () => {+ test.skipIf(!sqliteVecLoadable())(+ "a child on an older-schema index upgrades it in place and keeps its embeddings (#256)",+ async () => {+ const server = await startFakeHttp();+ try {+ const semantic = makeConfig({+ vault,+ dbPath: dbPath(),+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: FAKE_PROVIDER_KEY,+ dimension: 4,+ maxRetries: 1,+ },+ });+ writeMd(vault, "fox.md", "# Fox\n\nThe quick brown fox.");+ await indexVault(semantic, { embeddings: true });+ const vectors = async () => {+ const s = await Store.open(semantic, { mode: "read" });+ try {+ return s.countEmbeddings();+ } finally {+ await s.close();+ }+ };+ const before = await vectors();+ expect(before).toBeGreaterThan(0);+ rewindToPreV14Schema(dbPath());+ const runId = mintSelfHealRunId();++ const stderr: string[] = [];+ const write = process.stderr.write.bind(process.stderr);+ process.stderr.write = ((chunk: string | Uint8Array) => {+ stderr.push(String(chunk));+ return true;+ }) as typeof process.stderr.write;+ let code: number;+ try {+ code = await cmdSearchReindex([+ "--self-heal",+ runId,+ "--vault",+ vault,+ "--config",+ configPath,+ ]);+ } finally {+ process.stderr.write = write;+ }++ expect(code).toBe(0);+ // A migration is reported as one, not as an index that was current.+ expect(stderr.join("")).toContain("was migrated in place");+ expect(stderr.join("")).not.toContain("is current; no rebuild was needed");+ expect(readSelfHealReindexRows(vault).find((row) => row.runId === runId)?.outcome).toBe(+ SELF_HEAL_REINDEX_OUTCOME.completed,+ );+ expect(await vectors()).toBe(before);+ expect(existsSync(`${dbPath()}.bak`)).toBe(false);+ } finally {+ await server.close();+ }+ },+ CONTENTION_BUDGET_MS,+ );+test("a child that loses the lock records the failure by name",async () => {diff --git a/tests/core/search/config.test.ts b/tests/core/search/config.test.tsindex 57647de2..04d0cc07 100644--- a/tests/core/search/config.test.ts+++ b/tests/core/search/config.test.ts@@ -512,6 +512,28 @@ test("the bench pin accepts the pool-cap knob name in a recall pin", () => {);});+test("the bench pin names the trust gate mode, and the dead legacy field is refused by name", () => {+ // A pin of the legacy field would measure with the gate off without+ // any sign, so it is refused; the mode is the pinnable knob.+ const pin = parseBenchPinnedKnobs('{"recall":{"searchTrustGateMode":"enforce"}}');+ expect(pin.recall?.searchTrustGateMode).toBe("enforce");+ expect(() => parseBenchPinnedKnobs('{"recall":{"retrievalTrustGateEnabled":true}}')).toThrow(+ /retrievalTrustGateEnabled.*not a known knob/,+ );+});++test("an override carrying an invalid trust gate mode is refused by the resolver", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ const base = resolveSearchConfig({ vault: tmp, configPath });+ expect(() =>+ resolveSearchConfig({+ vault: tmp,+ configPath,+ overrides: { recall: { ...base.recall, searchTrustGateMode: "measure" as "shadow" } },+ }),+ ).toThrow(/search_trust_gate_mode must be/);+});+// ββ Retrieval trust gate mode (session-mining-economy, t_05a0eaa4) ββββββββββtest("the trust gate mode defaults off and stays absent from the resolved recall config", () => {@@ -536,9 +558,11 @@ test("an explicit trust gate mode resolves from config, env winning", () => {expect(resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode).toBe("shadow");});-test("the legacy boolean maps true to shadow and false to off", () => {+test("the legacy boolean maps true to enforce and false to off", () => {writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_enabled: true\n`);- expect(resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode).toBe("shadow");+ expect(resolveSearchConfig({ vault: tmp, configPath }).recall.searchTrustGateMode).toBe(+ "enforce",+ );writeFileSync(configPath, `vault: "${tmp}"\nsearch_trust_gate_enabled: false\n`);expect(@@ -589,6 +613,14 @@ test("an invalid trust gate mode is refused naming the key in force", () => {expect(envMessage).not.toContain("search_trust_gate_mode must");});+test("an invalid legacy trust gate boolean from the env names the env var", () => {+ writeFileSync(configPath, `vault: "${tmp}"\n`);+ process.env["OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE"] = "yes";+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(+ "OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE must be",+ );+});+test("a trust gate mode change re-keys the config fingerprint", () => {writeFileSync(configPath, `vault: "${tmp}"\n`);const off = resolveSearchConfig({ vault: tmp, configPath });diff --git a/tests/core/search/embedding-model-gate.test.ts b/tests/core/search/embedding-model-gate.test.tsindex 3b0312f6..6214aae7 100644--- a/tests/core/search/embedding-model-gate.test.ts+++ b/tests/core/search/embedding-model-gate.test.ts@@ -258,10 +258,35 @@ test("a prefix change with a blocked capability tier refuses the clear", () => {ensureEmbeddingModel(db, NO_VEC, "m1", 4, { query: "q2", passage: "p2" }, DISABLED_GATE),);expect(outcome.refusal?.code).toBe("EMBEDDING_DISABLED");+ // The refusal names the prefix change, not a "model change from m1/4 to m1/4".+ expect(outcome.refusal?.message).toContain("[q1|p1] to [q2|p2]");+ expect(outcome.refusal?.message).not.toContain("model change");expect(embeddingCount()).toBe(1);expect(getState(db, EMBEDDING_PREFIX_QUERY_STATE_KEY)).toBe("q1");});+test("a keyword-only open that keeps the vectors leaves their recorded prefixes alone (#255)", () => {+ seedStoredEmbedding("m1", 4);+ setState(db, EMBEDDING_PREFIX_QUERY_STATE_KEY, "query: ");+ setState(db, EMBEDDING_PREFIX_PASSAGE_STATE_KEY, "passage: ");+ const embedded = { query: "query: ", passage: "passage: " };+ ensureEmbeddingModel(db, NO_VEC, "m1", 4, embedded, CONFIGURED_GATE);++ // A keyword-only writer: no model, empty prefixes.+ captureErrors(() =>+ ensureEmbeddingModel(db, NO_VEC, null, null, { query: "", passage: "" }, DISABLED_GATE),+ );+ expect(getState(db, EMBEDDING_PREFIX_QUERY_STATE_KEY)).toBe("query: ");+ expect(getState(db, EMBEDDING_PREFIX_PASSAGE_STATE_KEY)).toBe("passage: ");++ // The unchanged semantic config opens again: nothing changed, nothing cleared.+ const { errors } = captureErrors(() =>+ ensureEmbeddingModel(db, NO_VEC, "m1", 4, embedded, CONFIGURED_GATE),+ );+ expect(embeddingCount()).toBe(1);+ expect(errors.some((m) => m.includes("cleared"))).toBe(false);+});+test("a prefix change with a configured provider clears as today", () => {seedStoredEmbedding("m1", 4);setState(db, EMBEDDING_PREFIX_QUERY_STATE_KEY, "q1");diff --git a/tests/core/search/hybrid-deadline.test.ts b/tests/core/search/hybrid-deadline.test.tsindex 03664e8e..fd504ad1 100644--- a/tests/core/search/hybrid-deadline.test.ts+++ b/tests/core/search/hybrid-deadline.test.ts@@ -331,8 +331,8 @@ function stalledRerankConfig(opts: { cacheEnabled?: boolean; rerankBaseUrl?: strvault,dbPath,retrievalTrustGateEnabled: true,- // The gate must EXCLUDE here, and the legacy boolean alone now- // measures (shadow): enforcement is named explicitly.+ // The gate must EXCLUDE here; the mode is named explicitly so the+ // test does not lean on the legacy boolean mapping.searchTrustGateMode: "enforce",cacheEnabled: opts.cacheEnabled ?? false,rerank: {diff --git a/tests/core/search/reindex-swap-races.test.ts b/tests/core/search/reindex-swap-races.test.tsindex 62db4803..aadf77d8 100644--- a/tests/core/search/reindex-swap-races.test.ts+++ b/tests/core/search/reindex-swap-races.test.ts@@ -172,3 +172,41 @@ test("a failed final swap rename puts the previous index back live", async () =>expect(existsSync(`${dbPath}.bak`)).toBe(false);expect(readFileSync(dbPath).equals(liveBefore)).toBe(true);});++test("a crash mid-swap with the live index parked at .old restores .old, not the older .bak", async () => {+ // A swap that keeps an embedding-richer `.bak` parks the live index at+ // `.old`. A crash between the renames leaves the live index there and an+ // OLDER index at `.bak`; restoring `.bak` would roll back past it.+ writeMd(vault, "a.md", "# A\n\nalpha content");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);+ cpSync(dbPath, `${dbPath}.bak`);+ writeMd(vault, "b.md", "# B\n\nbeta content");+ await indexVault(cfg);+ cpSync(dbPath, `${dbPath}.old`);+ rmSync(dbPath);++ const store = await Store.open(cfg, { mode: "read" });+ let paths: string[];+ try {+ paths = [...store.listDocuments().keys()].toSorted();+ } finally {+ await store.close();+ }+ // The live index (both notes) came back, not the older `.bak` (one note).+ expect(paths).toEqual(["a.md", "b.md"]);+ expect(existsSync(`${dbPath}.old`)).toBe(false);+ expect(existsSync(`${dbPath}.bak`)).toBe(true);+});++test("a .old left beside a present live index by a crash after the swap is removed", async () => {+ writeMd(vault, "a.md", "# A\n\nalpha content");+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);+ cpSync(dbPath, `${dbPath}.old`);++ const store = await Store.open(cfg, { mode: "read" });+ await store.close();+ expect(existsSync(dbPath)).toBe(true);+ expect(existsSync(`${dbPath}.old`)).toBe(false);+});diff --git a/tests/core/search/relational-arm.test.ts b/tests/core/search/relational-arm.test.tsindex 8498fe64..0e5552d7 100644--- a/tests/core/search/relational-arm.test.ts+++ b/tests/core/search/relational-arm.test.ts@@ -622,23 +622,18 @@ test("an unparseable declared window is refused where a current one passes", ()]);});-test("an edge whose polarity is a counted exclusion is refused", () => {+test("a contradicts edge the query named is answered, not refused", () => {+ // `[[seed]] contradicts` asks for exactly the contradicting pages:+ // counting the polarity against the relation the query named would+ // empty the answer under enforce and inflate every shadow measurement.writeMd(vault, "seed.md", '---\ncontradicts: "[[rival]]"\n---\n\nalpha seed.');writeMd(vault, "rival.md", "# Rival\n\nbeta contradictory content.");const store = armFixtureStore({ docs: { seed: 1, rival: 2 }, typed: [[1, "contradicts", 2]] });const outcome = runRelationalArm(store, vault, "[[seed]] contradicts", {configPath: modeConfigPath("enforce"),});- expect(outcome.rankedChunkIds).toEqual([]);- expect(outcome.refusals).toEqual([- {- document_id: 2,- chunk_id: 2,- path: "rival.md",- reasons: [`relational_arm:${RELATIONAL_ARM_REFUSAL_REASON.contradictsPolarity}`],- lane: RETRIEVAL_TRUST_REFUSAL_LANE.arm,- },- ]);+ expect(outcome.rankedChunkIds).toEqual([2]);+ expect(outcome.refusals).toEqual([]);});test("a chain through an ineligible node refuses the downstream arrival too", () => {@@ -816,31 +811,33 @@ test("the eligibility gate does not run when the arm does not walk", () => {expect("refusals" in outcome).toBe(false);});-test("the resolved trust-gate mode reaches the arm through the caller's config path", async () => {- // The arm's only route to `dirty.md` is the walk: its body shares no- // keyword terms with the query. Enforce withholds it from the pack;- // shadow surfaces it with the arm's attribution, refusal counted.- writeMd(vault, "seed.md", '---\nrelated: "[[dirty]]"\n---\n\nalpha topic seed document.');- writeMd(- vault,- "dirty.md",- `---\n${ENTITY_CONTAMINATION_FRONTMATTER_KEY}: true\n---\n\nbeta divergent content.`,- );+test("the effective config's trust-gate mode reaches the arm, not a fresh machine-config read", async () => {+ // The arm's only route to `stale.md` is the walk: its body shares no+ // keyword terms with the query. Its closed validity window is a reason+ // only the edge gate reads, so the page gate cannot stand in for it.+ // The machine config says off; the effective config (as a bench pin+ // would override it) names the mode, and the arm must follow it.+ writeMd(vault, "seed.md", '---\nrelated: "[[stale]]"\n---\n\nalpha topic seed document.');+ writeMd(vault, "stale.md", "---\nvalid_until: 2000-01-01\n---\n\nbeta divergent content.");await indexVault(makeConfig({ vault, dbPath }));const configPath = join(vault, "arm-gate.yaml");- const cfgFor = (mode: "shadow" | "enforce") => {- writeFileSync(configPath, `search_trust_gate_mode: ${mode}\n`);- return {- ...makeConfig({ vault, dbPath, fusionMode: "rrf", relationalArmEnabled: true }),- configPath,- };- };+ writeFileSync(configPath, "search_trust_gate_mode: off\n");+ const cfgFor = (mode: "shadow" | "enforce") => ({+ ...makeConfig({+ vault,+ dbPath,+ fusionMode: "rrf",+ relationalArmEnabled: true,+ searchTrustGateMode: mode,+ }),+ configPath,+ });const enforced = await search(cfgFor("enforce"), { query: "alpha [[seed]] related" });- expect(enforced.results.some((r) => r.path === "dirty.md")).toBe(false);+ expect(enforced.results.some((r) => r.path === "stale.md")).toBe(false);const shadowed = await search(cfgFor("shadow"), { query: "alpha [[seed]] related" });- const dirty = shadowed.results.find((r) => r.path === "dirty.md");- expect(dirty).toBeDefined();- expect(dirty!.reasons.some((x) => x.startsWith("relational:"))).toBe(true);+ const stale = shadowed.results.find((r) => r.path === "stale.md");+ expect(stale).toBeDefined();+ expect(stale!.reasons.some((x) => x.startsWith("relational:"))).toBe(true);});diff --git a/tests/core/search/rerank.decision-model.search.test.ts b/tests/core/search/rerank.decision-model.search.test.tsindex 6f4e7aa5..42e50233 100644--- a/tests/core/search/rerank.decision-model.search.test.ts+++ b/tests/core/search/rerank.decision-model.search.test.ts@@ -410,8 +410,8 @@ describe("decision-model rerank under the hybrid deadline", () => {decision_model_timeout_ms: String(DECISION_TIMEOUT_MS),search_hybrid_deadline_ms: String(DEADLINE_MS),search_trust_gate_enabled: "true",- // The gate must EXCLUDE here, and the legacy boolean alone now- // measures (shadow): enforcement is named explicitly.+ // The gate must EXCLUDE here; the mode is named explicitly so the+ // test does not lean on the legacy boolean mapping.search_trust_gate_mode: "enforce",});await indexVault(cfg);diff --git a/tests/core/search/retrieval-trust-gate.test.ts b/tests/core/search/retrieval-trust-gate.test.tsindex c15a8df6..5daeb3d1 100644--- a/tests/core/search/retrieval-trust-gate.test.ts+++ b/tests/core/search/retrieval-trust-gate.test.ts@@ -9,8 +9,9 @@* receipts, whose `excluded` counts stay untouched.* - enforce: today's excluding behavior, made explicit.*- * The legacy `search_trust_gate_enabled` boolean alone now maps to- * shadow: a vault must name `enforce` to exclude.+ * The legacy `search_trust_gate_enabled` boolean alone maps to enforce,+ * the excluding behavior it had before the mode existed: a vault opts+ * into measuring by naming `shadow`.*/import { test, expect, beforeEach, afterEach, describe, spyOn } from "bun:test";@@ -147,19 +148,17 @@ describe("retrieval trust gate on the search path", () => {});});- test("the legacy boolean alone now maps to shadow (measure, not exclude)", async () => {+ test("the legacy boolean alone keeps excluding across the upgrade", async () => {writeMd(vault, "clean.md", CLEAN);writeMd(vault, "quarantined.md", QUARANTINED);- // No explicit mode: the legacy true maps to shadow.+ // No explicit mode, resolved the way a pre-mode config resolves.const cfg = makeConfig({ vault, dbPath, retrievalTrustGateEnabled: true });await indexVault(cfg);const out = await search(cfg, { query: "widget calibration", limit: 5 });- expect(out.results.map((r) => r.path).toSorted()).toEqual(["clean.md", "quarantined.md"]);- expect(out.retrievalDecisionTrace!.mode).toBe("shadow");- expect(out.retrievalDecisionTrace!.refused).toBe(1);- expect(out.retrievalDecisionTrace!.excluded).toBe(0);+ expect(out.results.map((r) => r.path)).toEqual(["clean.md"]);+ expect(out.retrievalDecisionTrace!.excluded).toBe(1);});test("enforce named explicitly keeps the excluding behavior and today's receipt shape", async () => {@@ -407,7 +406,7 @@ describe("retrieval trust gate on the search path", () => {expect("mode" in assessment).toBe(false);});- test("the explain envelope names the resolved mode under shadow and stays silent under enforce", async () => {+ test("the explain envelope names the resolved mode under both active modes", async () => {writeMd(vault, "clean.md", CLEAN);writeMd(vault, "quarantined.md", QUARANTINED);const off = makeConfig({ vault, dbPath });@@ -426,9 +425,10 @@ describe("retrieval trust gate on the search path", () => {expect(shadowEnvelope[RETRIEVAL_DECISION_TRACE_KEY]).toBeDefined();expect(shadowEnvelope[RETRIEVAL_TRUST_GATE_MODE_KEY]).toBe("shadow");- // Enforce keeps today's envelope exactly: the receipts, no mode key.+ // Enforce keeps the receipt shape but the envelope still names the+ // mode, so a vault's posture is visible whichever mode is in force.const enforceEnvelope = explainEnvelope(enforceOutcome, request);expect(enforceEnvelope[RETRIEVAL_DECISION_TRACE_KEY]).toBeDefined();- expect(enforceEnvelope[RETRIEVAL_TRUST_GATE_MODE_KEY]).toBeUndefined();+ expect(enforceEnvelope[RETRIEVAL_TRUST_GATE_MODE_KEY]).toBe("enforce");});});diff --git a/tests/core/search/self-heal-keeps-embeddings.test.ts b/tests/core/search/self-heal-keeps-embeddings.test.tsnew file mode 100644index 00000000..23a16b85--- /dev/null+++ b/tests/core/search/self-heal-keeps-embeddings.test.ts@@ -0,0 +1,215 @@+/**+ * The read path's self-heal keeps compatible stored embeddings (#256).+ *+ * An index left on an older schema by a previous release is upgraded in+ * place: its vectors, the recorded model and the recorded prefixes all+ * survive, so the first search after an upgrade is still hybrid and no+ * paid re-embedding is needed. Only the states a migration cannot repair+ * - a newer schema, missing tables - are rebuilt, and that rebuild says+ * which vectors it dropped and how to get them back.+ */+import { afterEach, beforeEach, expect, test } from "bun:test";+import { Database } from "bun:sqlite";+import { existsSync } from "node:fs";++import { countStoredEmbeddings, indexVault } from "../../../src/core/search/indexer.ts";+import { LATEST_SCHEMA_VERSION } from "../../../src/core/search/schema.ts";+import { search } from "../../../src/core/search/search.ts";+import { rebuildIndexForSelfHeal } from "../../../src/core/search/self-heal.ts";+import { Store } from "../../../src/core/search/store.ts";+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";+import { FAKE_PROVIDER_KEY } from "../../helpers/fake-credentials.ts";+import { startFakeHttp } from "../../helpers/fake-http.ts";+import { rewindToPreV14Schema, stampSchemaVersion } from "../../helpers/schema-rewind.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";++const VEC = sqliteVecLoadable();++let v: ReturnType<typeof createTempVault>;+let server: Awaited<ReturnType<typeof startFakeHttp>>;+let semantic: ResolvedSearchConfig;+let keywordOnly: ResolvedSearchConfig;++beforeEach(async () => {+ v = createTempVault("selfheal-emb");+ server = await startFakeHttp();+ writeMd(v.vault, "Notes/fox.md", "# Fox\n\nThe quick brown fox jumps over the lazy dog.");+ semantic = makeConfig({+ vault: v.vault,+ dbPath: v.dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: FAKE_PROVIDER_KEY,+ dimension: 4,+ queryPrefix: "query: ",+ passagePrefix: "passage: ",+ maxRetries: 1,+ },+ });+ keywordOnly = makeConfig({ vault: v.vault, dbPath: v.dbPath });+});++afterEach(async () => {+ await server.close();+ v.cleanup();+});++/** The stored-embedding facts a heal must not disturb. */+async function embeddingIdentity(): Promise<{+ schema: string | null;+ vectors: number;+ query: string | null;+ passage: string | null;+}> {+ const s = await Store.open(semantic, { mode: "read" });+ try {+ return {+ schema: s.getState("schema_version"),+ vectors: s.countEmbeddings(),+ query: s.getState("embedding_prefix_query"),+ passage: s.getState("embedding_prefix_passage"),+ };+ } finally {+ await s.close();+ }+}++async function captureStderr<T>(fn: () => Promise<T>): Promise<{ result: T; lines: string[] }> {+ const lines: string[] = [];+ const original = console.error;+ console.error = (msg: string) => lines.push(String(msg));+ try {+ return { result: await fn(), lines };+ } finally {+ console.error = original;+ }+}++test.skipIf(!VEC)(+ "an older-schema semantic index is upgraded in place and the first search is still hybrid",+ async () => {+ await indexVault(semantic, { embeddings: true });+ const before = await embeddingIdentity();+ expect(before.vectors).toBeGreaterThan(0);+ rewindToPreV14Schema(v.dbPath);++ const outcome = await search(semantic, { query: "fox", semantic: true, recordAccess: false });++ expect(outcome.results.some((r) => r.searchType === "hybrid")).toBe(true);+ expect(await embeddingIdentity()).toEqual({ ...before, schema: String(LATEST_SCHEMA_VERSION) });+ // Upgraded, not rebuilt: a rebuild leaves the replaced index behind.+ expect(existsSync(`${v.dbPath}.bak`)).toBe(false);+ },+);++test.skipIf(!VEC)(+ // Issued together from one process; whether the two heals overlap is up+ // to the scheduler, so the name claims only what the assertions show.+ "two searches issued together over an older-schema index both answer and keep the vectors",+ async () => {+ await indexVault(semantic, { embeddings: true });+ const before = await embeddingIdentity();+ rewindToPreV14Schema(v.dbPath);++ const outcomes = await Promise.all([+ search(semantic, { query: "fox", semantic: true, recordAccess: false }),+ search(keywordOnly, { query: "fox", recordAccess: false }),+ ]);++ for (const outcome of outcomes) expect(outcome.results.length).toBeGreaterThan(0);+ expect(await embeddingIdentity()).toEqual({ ...before, schema: String(LATEST_SCHEMA_VERSION) });+ expect(existsSync(`${v.dbPath}.bak`)).toBe(false);+ },+);++test.skipIf(!VEC)(+ "a keyword-only search that repairs an older-schema index leaves the embedding identity alone",+ async () => {+ await indexVault(semantic, { embeddings: true });+ const before = await embeddingIdentity();+ rewindToPreV14Schema(v.dbPath);++ const repaired = await search(keywordOnly, { query: "fox", recordAccess: false });+ expect(repaired.results.length).toBeGreaterThan(0);+ expect(await embeddingIdentity()).toEqual({ ...before, schema: String(LATEST_SCHEMA_VERSION) });++ // The semantic writer that comes next finds its vectors compatible.+ await indexVault(semantic, { embeddings: true });+ expect((await embeddingIdentity()).vectors).toBe(before.vectors);+ const outcome = await search(semantic, { query: "fox", semantic: true, recordAccess: false });+ expect(outcome.results.some((r) => r.searchType === "hybrid")).toBe(true);+ },+);++test.skipIf(!VEC)(+ "a newer-schema index is still rebuilt, and the rebuild names the vectors it dropped",+ async () => {+ await indexVault(semantic, { embeddings: true });+ const { vectors } = await embeddingIdentity();+ stampSchemaVersion(v.dbPath, LATEST_SCHEMA_VERSION + 1);++ const { result: outcome, lines } = await captureStderr(() =>+ search(keywordOnly, { query: "fox", recordAccess: false }),+ );++ expect(outcome.results.length).toBeGreaterThan(0);+ expect(existsSync(`${v.dbPath}.bak`)).toBe(true);+ const notice = lines.find((l) => l.includes("not carried over"));+ expect(notice).toContain(`${vectors} stored embedding(s)`);+ expect(notice).toContain("o2b search index --embeddings");+ },+);++test.skipIf(!VEC)(+ "a self-heal rebuild that finds the index already current builds nothing",+ async () => {+ // The second of two healers decided to rebuild before the first one+ // finished; by the time it holds the writer lock the index is current.+ await indexVault(semantic, { embeddings: true });+ const before = await embeddingIdentity();++ expect(await rebuildIndexForSelfHeal(keywordOnly)).toBeNull();++ expect(await embeddingIdentity()).toEqual(before);+ expect(existsSync(`${v.dbPath}.bak`)).toBe(false);+ },+);++test.skipIf(!VEC)(+ "a second keyword-only rebuild keeps the .bak that still holds the vectors",+ async () => {+ await indexVault(semantic, { embeddings: true });+ const { vectors } = await embeddingIdentity();+ expect(vectors).toBeGreaterThan(0);++ stampSchemaVersion(v.dbPath, LATEST_SCHEMA_VERSION + 1);+ await captureStderr(() => search(keywordOnly, { query: "fox", recordAccess: false }));+ expect(countStoredEmbeddings(`${v.dbPath}.bak`)).toBe(vectors);++ // A second rebuild over the vector-less live index must not replace+ // the only copy of the vectors the first rebuild's notice points at.+ stampSchemaVersion(v.dbPath, LATEST_SCHEMA_VERSION + 1);+ const { result } = await captureStderr(() =>+ search(keywordOnly, { query: "fox", recordAccess: false }),+ );+ expect(result.results.length).toBeGreaterThan(0);+ expect(countStoredEmbeddings(`${v.dbPath}.bak`)).toBe(vectors);+ expect(existsSync(`${v.dbPath}.old`)).toBe(false);+ },+);++test("an index stamped current but missing a table is still rebuilt", async () => {+ await indexVault(keywordOnly);+ const db = new Database(v.dbPath);+ db.exec("DROP TABLE document_attributes");+ db.close();++ const outcome = await search(keywordOnly, { query: "fox", recordAccess: false });++ expect(outcome.results.length).toBeGreaterThan(0);+ expect(existsSync(`${v.dbPath}.bak`)).toBe(true);+});diff --git a/tests/core/state/surfaces.test.ts b/tests/core/state/surfaces.test.tsindex 3ed1790b..9e8fc780 100644--- a/tests/core/state/surfaces.test.ts+++ b/tests/core/state/surfaces.test.ts@@ -64,6 +64,7 @@ import { sessionCheckpointPath } from "../../../src/core/brain/sessions/checkpoiimport { manifestPath as ingestManifestPath } from "../../../src/core/brain/ingest/content-manifest.ts";import { sessionLedgerPath } from "../../../src/core/brain/sessions/discover.ts";import { sessionMiningLedgerPath } from "../../../src/core/brain/session-mining-ledger.ts";+import { sessionMiningSurveysPath } from "../../../src/core/brain/session-mining-surveys.ts";import { secretsDir } from "../../../src/core/brain/secrets/store.ts";import { manifestPath as installManifestPath } from "../../../src/core/install/manifest.ts";import { bootstrapReceiptPath } from "../../../src/cli/bootstrap/receipt.ts";@@ -268,6 +269,7 @@ const RESOLVER_BINDINGS: ReadonlyArray<readonly [StateSurfaceId, (vault: string)["write_dead_letters", (v) => deadLetterDir(v)],["session_import_ledger", (v) => sessionLedgerPath(v)],["session_mining_ledger", (v) => sessionMiningLedgerPath(v)],+ ["session_mining_surveys", (v) => sessionMiningSurveysPath(v)],["install_manifest", (v) => installManifestPath(v)],["bootstrap_receipt", (v) => bootstrapReceiptPath(v)],["hook_audit", (v) => hookAuditDir(v)],diff --git a/tests/helpers/schema-rewind.ts b/tests/helpers/schema-rewind.tsnew file mode 100644index 00000000..cd8ef439--- /dev/null+++ b/tests/helpers/schema-rewind.ts@@ -0,0 +1,49 @@+/**+ * Turn a freshly built index into one an older binary left behind, so the+ * upgrade path can be exercised without an old checkout.+ */+import { Database } from "bun:sqlite";++import { sqliteVecLoadable } from "./sqlite-vec.ts";++/** The schema version before v14's artifacts existed. */+export const PRE_V14_SCHEMA_VERSION = 13;++/** Open `dbPath` read-write with sqlite-vec loaded when available. */+function openWithVec(dbPath: string): Database {+ const db = new Database(dbPath);+ if (sqliteVecLoadable()) {+ // DDL on a file holding a vec0 table needs the module registered.+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const vec = require("sqlite-vec") as { getLoadablePath(): string };+ db.loadExtension(vec.getLoadablePath());+ }+ return db;+}++/** Drop the v14 artifacts and stamp schema 13, as a v13 binary wrote it. */+export function rewindToPreV14Schema(dbPath: string): void {+ const db = openWithVec(dbPath);+ try {+ db.exec("DROP INDEX IF EXISTS idx_chunks_kind");+ db.exec("DROP INDEX IF EXISTS idx_documents_page_type");+ db.exec("DROP INDEX IF EXISTS idx_document_attributes_key_value");+ db.exec("DROP TABLE IF EXISTS document_attributes");+ db.exec("ALTER TABLE chunks DROP COLUMN chunk_kind");+ db.query("UPDATE index_state SET value = ? WHERE key = 'schema_version'").run(+ String(PRE_V14_SCHEMA_VERSION),+ );+ } finally {+ db.close();+ }+}++/** Stamp an arbitrary schema version without touching the tables. */+export function stampSchemaVersion(dbPath: string, version: number): void {+ const db = new Database(dbPath);+ try {+ db.query("UPDATE index_state SET value = ? WHERE key = 'schema_version'").run(String(version));+ } finally {+ db.close();+ }+}diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.tsindex a3e5dd17..cb9d8339 100644--- a/tests/helpers/search-fixtures.ts+++ b/tests/helpers/search-fixtures.ts@@ -46,6 +46,21 @@ export function writeSymlink(vault: string, relLink: string, absTarget: string):return abs;}+/**+ * The gate mode mirrors resolveSearchTrustGateMode's mapping: an explicit+ * mode wins (off stays absent), else the legacy boolean maps true to+ * enforce - so the fixture can never build a resolved shape the resolver+ * would not produce.+ */+function fixtureTrustGateMode(+ mode: "shadow" | "enforce" | "off" | undefined,+ legacyEnabled: boolean | undefined,+): "shadow" | "enforce" | undefined {+ if (mode === "off") return undefined;+ if (mode !== undefined) return mode;+ return legacyEnabled === true ? "enforce" : undefined;+}+export function makeConfig(opts: {vault: string;dbPath: string;@@ -83,7 +98,7 @@ export function makeConfig(opts: {/*** The gate's resolved mode (session-mining-economy). Default: the same* mapping the resolver applies - the legacy boolean true alone resolves- * to shadow, false and absence stay off (the field stays absent, like+ * to enforce, false and absence stay off (the field stays absent, like* every opt-in recall field at its off value). An explicit "off" omits* the field too, matching the spread-omitted resolved shape.*/@@ -127,18 +142,7 @@ export function makeConfig(opts: {...opts.semantic,});const paths = opts.ignorePaths ?? DEFAULT_VAULT_IGNORE_PATHS;- // The gate mode mirrors resolveSearchTrustGateMode's mapping: explicit- // mode wins (off stays absent), else the legacy boolean maps true to- // shadow - so the fixture can never build a resolved shape the- // resolver would not produce.- const gateMode: "shadow" | "enforce" | undefined =- opts.searchTrustGateMode !== undefined- ? opts.searchTrustGateMode === "off"- ? undefined- : opts.searchTrustGateMode- : opts.retrievalTrustGateEnabled === true- ? "shadow"- : undefined;+ const gateMode = fixtureTrustGateMode(opts.searchTrustGateMode, opts.retrievalTrustGateEnabled);const scopeRules: VaultScopeRules = Object.freeze({ignore: Object.freeze(paths.map(classifyVaultPathRule)),include:diff --git a/tests/hooks/backlog-nudge.test.ts b/tests/hooks/backlog-nudge.test.tsindex 6feb9772..5f83f9cf 100644--- a/tests/hooks/backlog-nudge.test.ts+++ b/tests/hooks/backlog-nudge.test.ts@@ -138,6 +138,11 @@ describe("backlog-nudge hook (SessionStart)", () => {expect(payload.hookSpecificOutput?.hookEventName).toBe("SessionStart");expect(payload.hookSpecificOutput?.additionalContext).toMatch(/1 imported session\b/);expect(payload.hookSpecificOutput?.additionalContext).not.toMatch(/2 imported sessions/);+ // The nudge points at the JSON form, the one that hands out the+ // mining envelopes.+ expect(payload.hookSpecificOutput?.additionalContext).toContain(+ "o2b brain import-session --propose-backlog --json",+ );// The state lands only after the emit, overwrite-only, size one.expect(readFileSync(statePath(), "utf8")).toBe("1\n");});diff --git a/tests/mcp/brain-search-explain-trace.test.ts b/tests/mcp/brain-search-explain-trace.test.tsindex aaed70d6..79690429 100644--- a/tests/mcp/brain-search-explain-trace.test.ts+++ b/tests/mcp/brain-search-explain-trace.test.ts@@ -87,8 +87,8 @@ beforeEach(async () => {writeAged("notes/quarantined.md", QUARANTINED);const onPath = join(configHome, "gate-on.yaml");- // The gate must EXCLUDE here, and the legacy boolean alone now- // measures (shadow): enforcement is named explicitly.+ // The gate must EXCLUDE here; the mode is named explicitly so the+ // test does not lean on the legacy boolean mapping.atomicWriteFileSync(onPath,`vault: ${vault}\nsearch_trust_gate_enabled: true\nsearch_trust_gate_mode: enforce\n`,@@ -157,6 +157,7 @@ describe("brain_search explain: retrieval receipts", () => {const on = (await call(gateOn, { explain: true })) as unknown as Record<string, unknown>;delete on["retrieval_decision_trace"];delete on["memory_trust_assessment"];+ delete on["retrieval_trust_gate_mode"];for (const r of on["results"] as Array<Record<string, unknown>>) delete r["score_breakdown"];expect(JSON.stringify(on)).toBe(JSON.stringify(first));diff --git a/tests/mcp/brain-storage.test.ts b/tests/mcp/brain-storage.test.tsindex 0039ca92..828c7f3b 100644--- a/tests/mcp/brain-storage.test.ts+++ b/tests/mcp/brain-storage.test.ts@@ -217,6 +217,20 @@ describe("brain_storage", () => {}});+ test("a server's own config moves the measured store with its search_db_path", async () => {+ // The process env still names the default config; only the server's+ // configPath points the store elsewhere, so a census that ignored it+ // would measure the absent default store as zero.+ const altDb = join(configHome, "alt-index.sqlite");+ writeFileSync(altDb, "x".repeat(12_345), "utf8");+ const altConfig = join(configHome, "alt.yaml");+ atomicWriteFileSync(altConfig, `vault: ${vault}\nsearch_db_path: ${altDb}\n`);+ const res = (await storage({ ...ctx, configPath: altConfig }, {})) as ReportJson;+ const surfaces = (res.usage as { surfaces: ReadonlyArray<{ surface: string; bytes: number }> })+ .surfaces;+ expect(surfaces.find((s) => s.surface === "derived_store")?.bytes).toBe(12_345);+ });+test("a vault with no brain config surfaces the named config error", async () => {const bare = mkdtempSync(join(tmpdir(), "o2b-mcp-storage-bare-"));try {diff --git a/tests/mcp/delete-by-source-failed.test.ts b/tests/mcp/delete-by-source-failed.test.tsindex a4d5fbe7..e7599ce6 100644--- a/tests/mcp/delete-by-source-failed.test.ts+++ b/tests/mcp/delete-by-source-failed.test.ts@@ -17,6 +17,7 @@ import { ingestSource } from "../../src/core/brain/ingest/ingest.ts";import { MCPServer } from "../../src/mcp/server.ts";import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";import { CHMOD_CANNOT_DENY } from "../helpers/platform.ts";+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";const SOURCE = "imports/benchmark.md";const NOW = new Date("2026-06-01T00:00:00Z");@@ -32,7 +33,7 @@ interface Fixture {readonly server: MCPServer;}-function fixture(): Fixture {+function fixture(reach?: TransportReach): Fixture {const base = mkdtempSync(join(tmpdir(), "o2b-dbs-failed-"));bases.push(base);const vault = join(base, "vault");@@ -50,7 +51,10 @@ function fixture(): Fixture {},{ agent: "tester", now: NOW },);- return { vault, server: new MCPServer({ vault, configPath }) };+ return {+ vault,+ server: new MCPServer({ vault, configPath }, reach !== undefined ? { reach } : {}),+ };}/** The tool's structured answer, unwrapped from the MCP envelope. */@@ -85,6 +89,26 @@ describe("brain_delete_by_source names what did not delete", () => {}});+ test("below local reach a failure names the errno code, never the host path", async () => {+ if (CHMOD_CANNOT_DENY) return;+ const f = fixture(TRANSPORT_REACH.remote);+ const preview = (await call(f, { source_file: SOURCE })) as {+ derived: Array<{ path: string; kind: string }>;+ };+ const blockedPath = preview.derived.find((e) => e.kind === "summary")!.path;+ const blockedDir = join(f.vault, dirname(blockedPath));+ chmodSync(blockedDir, 0o555);+ try {+ const answer = (await call(f, { source_file: SOURCE, confirm: true })) as {+ failed: ReadonlyArray<{ path: string; message: string }>;+ };+ expect(answer.failed[0]!.path).toBe(blockedPath);+ expect(answer.failed[0]!.message).toBe("removal failed (EACCES)");+ } finally {+ chmodSync(blockedDir, 0o755);+ }+ });+test("a clean confirmed run carries no failure", async () => {const f = fixture();const answer = (await call(f, { source_file: SOURCE, confirm: true })) as {diff --git a/tests/mcp/doctor-collect-errors.test.ts b/tests/mcp/doctor-collect-errors.test.tsindex d5a14a8c..f956abdf 100644--- a/tests/mcp/doctor-collect-errors.test.ts+++ b/tests/mcp/doctor-collect-errors.test.ts@@ -14,6 +14,7 @@ import { join } from "node:path";import { bootstrapBrain } from "../../src/core/brain/init.ts";import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";import { JSONRPC_VERSION, MCPServer } from "../../src/mcp/index.ts";+import { TRANSPORT_REACH, type TransportReach } from "../../src/core/graph/transport-reach.ts";const bases: string[] = [];@@ -29,7 +30,7 @@ interface RpcResponse {};}-function fixture(breakPreferences: boolean): MCPServer {+function fixture(breakPreferences: boolean, reach?: TransportReach): MCPServer {const base = mkdtempSync(join(tmpdir(), "o2b-doctor-collect-"));bases.push(base);const vault = join(base, "vault");@@ -44,7 +45,7 @@ function fixture(breakPreferences: boolean): MCPServer {rmSync(join(vault, "Brain", "preferences"), { recursive: true });writeFileSync(join(vault, "Brain", "preferences"), "not a directory", "utf8");}- return new MCPServer({ vault, configPath }, {});+ return new MCPServer({ vault, configPath }, reach !== undefined ? { reach } : {});}async function callDoctor(server: MCPServer): Promise<RpcResponse> {@@ -57,7 +58,8 @@ async function callDoctor(server: MCPServer): Promise<RpcResponse> {}test("a refused scan is rendered as a named collect_errors row", async () => {- const res = await callDoctor(fixture(true));+ // Local reach: the operator's own surface carries the raised message.+ const res = await callDoctor(fixture(true, TRANSPORT_REACH.local));expect(res.error).toBeUndefined();const s = res.result?.structuredContent ?? {};// The break can refuse more than one scan on this surface; the row is@@ -74,6 +76,20 @@ test("a refused scan is rendered as a named collect_errors row", async () => {}});+test("below local reach the row names the scan and errno, never the host path", async () => {+ const res = await callDoctor(fixture(true, TRANSPORT_REACH.remote));+ expect(res.error).toBeUndefined();+ const errors = (res.result?.structuredContent?.["collect_errors"] ?? []) as ReadonlyArray<{+ scan: string;+ message: string;+ }>;+ expect(errors.some((e) => e.scan === "dedup")).toBe(true);+ for (const e of errors) {+ expect(e.message).toMatch(new RegExp(`^${e.scan} scan failed \\(`));+ expect(e.message).not.toContain(tmpdir());+ }+});+test("a healthy vault's payload carries no collect_errors row", async () => {const res = await callDoctor(fixture(false));expect(res.error).toBeUndefined();diff --git a/tests/mcp/extract-signals-tool.test.ts b/tests/mcp/extract-signals-tool.test.tsindex c31f196c..5f951be4 100644--- a/tests/mcp/extract-signals-tool.test.ts+++ b/tests/mcp/extract-signals-tool.test.ts@@ -27,6 +27,7 @@ import { dirname, join } from "node:path";import { AUTO_EXTRACT_PER_SESSION_CAP } from "../../src/core/brain/extract-signals.ts";import { sessionMiningLedgerPath } from "../../src/core/brain/session-mining-ledger.ts";+import { sessionMiningSurveysPath } from "../../src/core/brain/session-mining-surveys.ts";import { importSessionRecall } from "../../src/core/brain/session-recall.ts";import { buildToolTable, findTool } from "../../src/mcp/tools.ts";import type { ServerContext } from "../../src/mcp/tool-contract.ts";@@ -152,8 +153,9 @@ test("turns_mined carries the stored turn timestamp, on the tool and the CLI ali//// 7. A session tombstoned by a barren commit gets no envelope: the plan// carries the tombstone reason and the recorded breakdown, and no turn-// text rides out to the caller. The tombstone exists only when the-// commit relayed the plan's surveyed_turns_hash back.+// text rides out to the caller. Answering the envelope is enough:+// the commit keys the record on the session's current turns, and a+// relayed surveyed_turns_hash is only a check.// 8. The `force` argument bypasses the tombstone for exactly one plan// call, and the tool declares it; so does the surveyed_turns_hash// argument the commit relays.@@ -204,16 +206,43 @@ test("a tombstoned session returns the recorded breakdown instead of an envelopeexpect(plan.turns_mined).toEqual([]);});-test("a barren commit that never saw the plan records nothing and fails open", async () => {- // Without the relayed hash the commit has no material the verdict was- // earned over, so it records nothing rather than keying a tombstone on- // whatever the session holds now.+test("the documented flow - plan, then commit the answer - tombstones without any relay", async () => {+ // The skill's flow: call with {session}, answer the envelope with+ // {session, items}. Nothing tells the caller to echo a hash, so the+ // tombstone must land without one.+ await tool().handler(ctx, { session: SESSION });const mine = (await tool().handler(ctx, { session: SESSION, items: [] })) as {phase: string;- written: unknown[];+ mining_ledger_failure?: string;};expect(mine.phase).toBe("commit");- expect(mine.written).toEqual([]);+ expect(mine.mining_ledger_failure).toBeUndefined();+ const plan = (await tool().handler(ctx, { session: SESSION })) as {+ skipped?: { reason: string };+ };+ expect(plan.skipped?.reason).toBe("mining_tombstone");+});++test("a relayed hash the session no longer matches records nothing and says so", async () => {+ // A hash from another plan (or a session that changed since) must not+ // overwrite the record with one that never matches.+ const foreign = "0".repeat(64);+ const mine = (await tool().handler(ctx, {+ session: SESSION,+ items: [],+ surveyed_turns_hash: foreign,+ })) as { mining_ledger_failure?: string };+ expect(mine.mining_ledger_failure).toContain(foreign);+ const plan = (await tool().handler(ctx, { session: SESSION })) as { skipped?: unknown };+ expect(plan.skipped).toBeUndefined();+});++test("an empty relayed hash is refused as malformed, not read as absent", async () => {+ await tool().handler(ctx, { session: SESSION });+ await expect(+ tool().handler(ctx, { session: SESSION, items: [], surveyed_turns_hash: "" }),+ ).rejects.toThrow("surveyed_turns_hash must be the 64-character lowercase hex digest");+ // The refusal came before any record, so the next plan still offers the turns.const plan = (await tool().handler(ctx, { session: SESSION })) as { skipped?: unknown };expect(plan.skipped).toBeUndefined();});@@ -243,7 +272,6 @@ test("the tool description documents the tombstone and the force escape", () =>const description = tool().description;expect(description).toContain("tombstone");expect(description).toContain("force");- expect(description).toContain("surveyed_turns_hash");});test("a ledger write failure on a successful commit rides the commit response by name", async () => {@@ -262,3 +290,18 @@ test("a ledger write failure on a successful commit rides the commit response byexpect(res.written).toEqual([]);expect(res.mining_ledger_failure).toContain("session mining ledger is corrupted JSON");});++test("a survey store failure reaches a caller below local reach without the host path", async () => {+ // This context minted no reach, so it answers fail-closed below local.+ mkdirSync(dirname(sessionMiningSurveysPath(vault)), { recursive: true });+ writeFileSync(sessionMiningSurveysPath(vault), "{ truncated");+ const plan = (await tool().handler(ctx, { session: SESSION })) as {+ survey_record_failure?: string;+ };+ expect(plan.survey_record_failure).toBe("session mining surveys store failed (error)");+ const mine = (await tool().handler(ctx, { session: SESSION, items: [] })) as {+ mining_ledger_failure?: string;+ };+ expect(mine.mining_ledger_failure).toContain("session mining surveys store failed (error)");+ expect(mine.mining_ledger_failure).not.toContain(tmp);+});diff --git a/uv.lock b/uv.lockindex 805ebe60..03c02a96 100644--- a/uv.lock+++ b/uv.lock@@ -4,5 +4,5 @@ requires-python = ">=3.11"[[package]]name = "open-second-brain"-version = "1.86.0"+version = "1.86.1"source = { editable = "." }