Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: what the index already knew (v1.44.0) (#157)
Sol Aitken3ee1963a16d7
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex 8e5a958c..8bea0c53 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.43.0",+ "version": "1.44.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex eb407794..9885d9f7 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.43.0",+ "version": "1.44.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","skills": "./skills","hooks": "./hooks/hooks.json",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 42a02c8b..7950fc78 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,47 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.44.0] - 2026-08-08++Thirteen units on the retrieval path and the store beneath it. A reconnaissance pass ran before any design and falsified the load-bearing premise of nine of the eleven tasks in scope, and what survived had a shape none of the task descriptions predicted: with two exceptions these were not missing capabilities but values the system already computed and discarded before anything could see them, and checks that were never run at all. A structure recording which candidates were evaluated, surfaced and excluded was built on every gated search and serialized by nothing. A content hash had been on every chunk since the first schema version and was never read back. A match-centred snippet function was written, tested, and mounted on the wrong surface. An expiry filter took an as-of date as a parameter and no caller ever passed one. A ranker accepted an injected clock and the stage above it did not inject.++Some of what was asked for is not here, and its absence is a decision rather than an omission. There is no runtime probe of an embedding model's input window, because no provider in this system exposes model metadata, no local model file exists, and the provider contract has no window field - the only implementations are a network round trip or a hardcoded table over an open set of endpoints. There is no second index-health counter surface, because `o2b search status` is already the eighth and the stall reasons the request described belong to a background drain loop this project does not run. There is no column type-drift check, because SQLite accepts a text value into an integer column and still reports the column as integer. There is no daily snapshot of the derived store, because `o2b brain snapshot` and `o2b brain rollback` already ship the checksum sidecar, the verification, the pre-restore diff, the confirmation and the retention that request called its differentiating pieces. And two of seven proposed recall-quality signals - acquisition risk and expected regret - are refused outright, with a census proving their names appear nowhere in the source, because there is no data behind either and an invented number is worse than an absent one.++Five reviewers were then given the finished branch and the claims made for it, with no knowledge of how the work was done, one told to treat every test as guilty until proven innocent. They found defects this wave had introduced, including one that turned a harmless refusal into silent data loss, one mechanism sealed by another unit in the same branch, and three claims in the branch's own comments and design document that were not true. Every one is fixed or named below.++### Added++- Exact-duplicate passages merge into one result naming the locations they were folded from, on the full and card surfaces alike. The identity is the pair of whole-file bytes and passage bytes, not the passage alone: frontmatter is stripped before chunking, so two records with the same prose but different dates, tiers or titles hash their chunks identically and are not duplicates.+- Every snippet surface anchors its window on the match instead of the head of the chunk, through one function with four callers. A passage that fits its cap is shown whole rather than slid past.+- The retrieval decision trace and the memory trust assessment - built by the trust gate and serialized by neither surface - are returned under the existing `explain` flag, absent rather than null when it is off, and naming the switch that produces them when the gate is not engaged.+- `o2b search check --integrity` runs the full structural scan on demand and records its verdict where the write-open gate reads it, so an index that only ever gets read can still be condemned.+- An oversize-chunk census compares the configured chunk size against the configured model's declared input window, which the curated presets now carry. The shipped default chunk size overflows the shipped default model by roughly two to three times, and nothing had noticed. The estimate is two-sided - a floor charging every code point a quarter token and a ceiling charging every non-ASCII code point a whole one - because a quarter-token-per-character proxy under-reports Han text roughly fourfold, and a census that stayed silent there would be claiming a pass it had not earned. Between the two bounds the verdict is `estimate-undecided`, which is reported rather than rounded to clean. The instruction prefix the provider prepends is measured into both, and which backends send one is an exhaustive switch, so a new backend fails to compile rather than silently inheriting a zero.+- A gated search records four computable recall-quality signals on the telemetry record it already writes, one record per query.+- `o2b brain query` gains `--at` and `--show-expired`, threading the as-of instant the expiry filter has accepted as a parameter all along.+- The paired holdout harness a release note has described since 1.36.0 now gates `o2b brain repair-lane --apply`, which is what makes that claim true. It had no production caller at all.++### Fixed++- A transient failure while opening the store for reading was classified as an unreadable index, which is self-healable, so a locked pragma triggered a silent rebuild that dropped every embedding. The busy timeout is now set before the statement that needs it, and an unrecognised failure is no longer grounds for a rebuild.+- A recorded integrity fault was honoured on read and ignored on write, and a completed scan refreshed the interval stamp, so a condemned index kept accepting writes for a day while every query refused it.+- The ranker read the wall clock itself because the stage above never passed the instant the request had already resolved. One instant now governs a whole search. The vector lane returned an arbitrary member of a tie band - and its two branches disagreed about which - because a `k` nearest selection happens before any SQL ordering; three candidate queries now end their ordering in a unique column.+- Reciprocal-rank fusion discarded the per-query intent profile, so a quoted phrase expressed a preference the opt-in fusion mode silently ignored, and the learned weights from operator feedback went with it.+- The entity identity kernel treated a typographic apostrophe and an ASCII one as different entities. The fold that joins them is derived from Unicode categories, not a character list, and the collision it creates between two records that already existed is refused with the command that resolves it rather than silently rewriting one label.+- Learned recall weights were validated for finiteness but not range, so a hand-edited file could inject a zero or negative lane multiplier - which under rank fusion deletes a lane or inverts its order.+- `show_expired` was silently ignored outside topic mode while its own schema text said topic mode only.+- The store verified its schema version integer and nothing else, so an index missing a table opened cleanly and failed later inside a query with a raw SQL error. The expected objects are derived by running the migrations, not by a second hand-maintained list.+- A corrupt-but-parseable index passed every check and served wrong results indefinitely; there was no integrity check anywhere in the codebase.+- The CLI truncation marker was keyed on the raw chunk length, so a chunk that fitted after whitespace collapse still claimed there was more.+- `total` echoed the number of rows returned, which is no information at all. It is the ranked pool the limit sliced from.+- The MCP tool descriptions still told every agent that `since` and `until` filter on when a document was modified. They have filtered on event time since 1.43.0.+- Entity extraction backtracked catastrophically on any unbroken run of letters carrying no digits - which is what an ordinary Han, Thai or Lao paragraph is. Measured at 11.7 seconds and zero matches for a single 2160-character chunk, growing cubically, so a Chinese vault of any size could not be indexed at all. The two mixed-script alternatives now consume their leading run atomically: 20 milliseconds on the same input, with identical output on every mixed-script sample.+- The explain surface named a cross-vault union as the reason there was no trace even when the trust gate was also off, sending the operator to a single-vault query that produces nothing either. When both hold, both are named.++### Changed++- The vendored Hermes tool schemas were re-vendored; three properties added in earlier releases had never reached them.+- `o2b search query`'s flags are declared in the command manifest, which generates the help and the completions. None of them had been.+## [1.43.0] - 2026-08-02Ten units on one seam: the boundary where content, authority and claims enter the vault. Open Second Brain could say a great deal about what it holds and almost nothing about where any of it came from, who was entitled to put it there, or what backs a claim made about it. An entity extracted from a scraped page became a first-class Brain entity on the same terms as one the operator typed. A write named any path it liked. A note was created with whatever frontmatter the caller assembled. A record was ranked by when its file was touched rather than by what it is about. An agent posted its own success and nothing recomputed it.@@ -6845,6 +6886,7 @@ plugin config (vault field)`, and exits with a clear- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.44.0]: https://github.com/itechmeat/open-second-brain/compare/v1.43.0...v1.44.0[1.43.0]: https://github.com/itechmeat/open-second-brain/compare/v1.42.0...v1.43.0[1.42.0]: https://github.com/itechmeat/open-second-brain/compare/v1.41.0...v1.42.0[1.41.0]: https://github.com/itechmeat/open-second-brain/compare/v1.40.0...v1.41.0diff --git a/README.md b/README.mdindex 8dba38e8..935b9b08 100644--- a/README.md+++ b/README.md@@ -8,16 +8,20 @@ Open Second Brain plugs into [Hermes Agent](https://github.com/NousResearch/herm## What is new-Open Second Brain 1.43.0 is about provenance at the boundary: what enters your vault, under whose authority, validated against what, and backed by what proof. Entities an agent extracts from an untrusted source - a scraped page, a fetched article - now land in a quarantine lane instead of becoming first-class Brain entities, and trust is derived from the shape of the source identity rather than from any word list. You can declare in `_brain.yaml` which path prefixes a caller-named write may touch, and a write outside them is refused with the command that resolves it; its authority is the config file, not the caller, because a caller can name itself anything. Note creation gains an idempotent skip whose result tells you which happened, validation before the write lands, and a template mode with a deliberately small grammar. A back-dated note - an imported log, a meeting record - now ranks by the date its body declares rather than by when the file was touched, with the source of that date recorded beside it. Schema mutations can be previewed before they touch the vault, which matters more than it sounds: `--dry-run` was previously parsed and ignored. And an outcome an agent posts about its own work now carries the kernel's own on-disk evidence beside the claim, with a mismatch recorded rather than resolved.+Open Second Brain 1.44.0 is about what the index already knew. Thirteen units on the retrieval path, and a reconnaissance pass that ran before any design falsified the load-bearing premise of nine of the eleven tasks in scope - so most of this release is not new capability but values the system already computed and then discarded before anything could see them. A record of which candidates a search evaluated, surfaced and excluded was built on every gated query and serialized by nothing. A content hash had been on every chunk since the first schema version and was never read back, so byte-identical copies of a file each took their own slot in your results. A match-centred snippet function existed, tested, mounted on the wrong surface, while every search preview truncated from the head of the chunk and hid the match you searched for. An expiry filter accepted an as-of date as a parameter that no caller ever passed. And the ranker read the wall clock itself, because the stage above it never handed over the instant the request had already resolved.-Two things this release deliberately did not build are worth knowing about, because their absence is a decision rather than an omission. There is no independent verifier, because agent identity here is an unauthenticated string and two records asserting two names chosen by one process are not two actors. And there is no pack fetched by URL, because a schema pack has no portable representation and no registry to install into. Five independent reviewers were given the finished branch and found thirty defects, nine of them introduced by this work and three of them mechanisms nothing could make fire; one of those three was removed rather than repaired. The [CHANGELOG](CHANGELOG.md) has the detail, including what was found and deliberately left open.+Two findings in this release are about damage rather than absence. A transient failure while opening the search index was being classified as corruption, and corruption triggers a silent rebuild - so a momentarily locked database could cost you every embedding in the vault, with the command reporting success. And entity extraction backtracked catastrophically on any unbroken run of letters without digits, which is what an ordinary Chinese, Thai or Lao paragraph is: eleven seconds for a single chunk, growing cubically, so a Chinese vault of any real size simply could not be indexed. Both are fixed.++Several things this release deliberately did not build are worth knowing about, because their absence is a decision. There is no runtime probe of an embedding model's input window, because nothing in this system can learn one without a network round trip or a hardcoded table over an open set of endpoints - so the curated presets declare their windows instead, and a model outside that table is reported as unmeasured rather than passed. There is no snapshot of the derived store, because `o2b brain snapshot` already ships every piece that request called differentiating. And two proposed recall-quality signals are refused outright, with a census proving their names appear nowhere in the source, because no data backs either and an invented number is worse than an absent one. Five independent reviewers were then given the finished branch with no knowledge of how it was built, one told to treat every test as guilty until proven innocent; what they found is in the [CHANGELOG](CHANGELOG.md), including three claims the branch made about itself that were not true.++Previous release, 1.43.0, was about provenance at the boundary: what enters your vault, under whose authority, validated against what, and backed by what proof. Entities an agent extracts from an untrusted source - a scraped page, a fetched article - now land in a quarantine lane instead of becoming first-class Brain entities, and trust is derived from the shape of the source identity rather than from any word list. You can declare in `_brain.yaml` which path prefixes a caller-named write may touch, and a write outside them is refused with the command that resolves it; its authority is the config file, not the caller, because a caller can name itself anything. Note creation gains an idempotent skip whose result tells you which happened, validation before the write lands, and a template mode with a deliberately small grammar. A back-dated note - an imported log, a meeting record - now ranks by the date its body declares rather than by when the file was touched, with the source of that date recorded beside it. Schema mutations can be previewed before they touch the vault, which matters more than it sounds: `--dry-run` was previously parsed and ignored. And an outcome an agent posts about its own work now carries the kernel's own on-disk evidence beside the claim, with a mismatch recorded rather than resolved.++Two things 1.43.0 deliberately did not build are worth knowing about, because their absence was a decision rather than an omission. There is no independent verifier, because agent identity here is an unauthenticated string and two records asserting two names chosen by one process are not two actors. And there is no pack fetched by URL, because a schema pack has no portable representation and no registry to install into. Five independent reviewers were given the finished branch and found thirty defects, nine of them introduced by this work and three of them mechanisms nothing could make fire; one of those three was removed rather than repaired. The [CHANGELOG](CHANGELOG.md) has the detail, including what was found and deliberately left open.Previous release, 1.42.0, was a structural wave with no new capability: the six modules that cost the most to change, both import cycles removed, and nine places where absence and inability-to-examine had been collapsed into one answer. Details live in the [CHANGELOG](CHANGELOG.md).Previous release, 1.41.0, was a signals-that-survive wave: eight units that carry a signal the system already computes the last few metres, to the decision it should have changed. Ingest now honours the repository's own `.gitignore`, nested ignore files, `.git/info/exclude` and submodule boundaries, through the same discovery module the hygiene scan uses - so a repo path stops pulling in the build artifacts it already declares as noise. A recall graded weak no longer evaporates when the call returns: repeated weak recall on the same question becomes a tracked, self-closing gap task. A query that names a period is ranked for that period instead of against it, detected from ISO and `since:`/`until:` tokens only so it behaves the same in every language. A scope-less capture says which routing signal it was missing and which scopes the vault actually has. Model-authored payloads are checked against a declared shape before they become knowledge, unconditionally - a validator that can be switched off is one the write path cannot rely on. Plain text can now seed a fact or a skill, not only a taste signal. And resumed work is recognised by a declared identity rather than by a fifteen-minute clock, so a branch or worktree switch stops breaking the chain. A ninth unit was built, reviewed, and reverted rather than shipped, because proving it inert was easier than pretending it was not - the [CHANGELOG](CHANGELOG.md) says why.-Previous release, 1.40.0, was a no-dead-ends wave: every issue `o2b brain doctor` reports carries the exact command that acts on it, terminal states name their next move through one rail a ratchet keeps singular, and where no command resolves a class the doctor gives the reason rather than falling silent. Details live in the [CHANGELOG](CHANGELOG.md).-## Why- **Lives in your Obsidian vault.** Open `Brain/preferences/pref-no-internal-abbrev.md` in Obsidian and you literally see what your agent learned about you - title, status, evidence count, confidence band, body text. Wikilinks, backlinks, graph view all work.diff --git a/docs/brainstorm/what-the-index-already-knew/cli-output/claude.md b/docs/brainstorm/what-the-index-already-knew/cli-output/claude.mdnew file mode 100644index 00000000..c908c743--- /dev/null+++ b/docs/brainstorm/what-the-index-already-knew/cli-output/claude.md@@ -0,0 +1,44 @@+### Variant 1: Invariant floor, then one explain envelope++- **Approach**: Land the correctness invariants first as unflagged fixes (inject `nowMs` from assembly, add unique tiebreakers to the three SQL orderings, fold the quote in `normalizeEntityName` with a named collision-resolution diagnostic, project `content_hash` into the hydrated row, thread `now` and the expired flag through the as-of surface, mount the existing match-window + char-span helpers on all three snippet surfaces, restore the intent profile inside the RRF relevance term). Then expose everything currently computed-and-discarded through a single opt-in `explain` envelope assembled once at the assembly stage — trace (evaluated/surfaced/excluded), true pre-truncation `total`, duplicate-merge records, shadow recall signals, snippet spans — serialized by one shared boundary projector that both the CLI and the MCP tool call. Store integrity (`PRAGMA integrity_check`) and schema table presence become one explicit `store check` path on the read-open seam that reports `not_run` rather than success when it cannot run; the oversize-chunk census ships as a file-level artifact from the existing token counts with the unprobeable true window declared as unknown.+- **Trade-offs**:+ - One flag, one absence contract, one place where "field is absent when off" is proved — the flag-on vs flag-off byte-identity comparison is written once, not eleven times.+ - The envelope is the natural home for the preview-budget escape hatch: a large trace becomes an artifact reference instead of a new tool or a description-length fight in the MCP registry.+ - Ordering is load-bearing and correct: a trace serialized over a wall-clock-reading ranker would faithfully document noise, so determinism must precede observability rather than ship beside it.+ - The envelope accretes unrelated concerns under one name; "explain" ends up meaning trace, dedup, shadow signals and spans at once, and later readers must be told which sub-object answers which question.+ - Sub-flags will eventually be wanted (shadow signals cost more to compute than the trace does), and retrofitting granularity into a shipped single flag is more work than starting granular.+ - The unflagged invariant fixes move real orderings, so the recall benchmark floors and the exact pass-count assertions must be re-pinned in the same wave, with the deltas attributed unit by unit rather than absorbed silently.+- **Complexity**: medium+- **Risk**: medium++### Variant 2: Eleven independent seam slices++- **Approach**: Treat each surviving unit as a self-contained vertical slice landed at the pipeline seam it belongs to — hydration gets `content_hash`, ranking gets injected time and tiebreakers, post-rank gets duplicate merge and shadow signals, fusion gets the intent profile, the registry gets the quote fold, the store gets presence and integrity checks, each snippet surface gets its own anchoring — with its own opt-in flag where a field is added, its own diagnostic code, and its own byte-identity proof. Shared discipline comes from a written convention checklist and the existing build ratchet, not from shared machinery. Each slice is independently revertable and independently reviewable.+- **Trade-offs**:+ - Maximum revert granularity: if the fusion weighting regresses the recall benchmark, that single slice backs out without disturbing the duplicate merge or the integrity gate.+ - Each slice's flag-off byte identity is trivially provable in isolation, which is exactly the shape the project's existing test convention wants.+ - No premature abstraction over units that are genuinely unrelated — the quote fold and the SQLite integrity check share nothing but a release wave.+ - Flag proliferation is real and permanent: five-plus new opt-in flags on one result type, a combinatorial on/off matrix nobody will fully test, and an MCP tool description that must name them all inside the registry's length limit.+ - Boundary plumbing gets rewritten per slice — each new optional field needs its own CLI serialization, its own MCP projection, its own preview-budget accounting — and the copies drift.+ - The trace and the `total` fix are the same defect seen twice (values the boundary refuses to carry); splitting them hides that the boundary projector, not the producers, is the thing that needs fixing.+ - Ordering discipline is only advisory, so nothing structurally prevents shipping the trace before determinism and pinning noise into the evaluation harnesses.+- **Complexity**: large+- **Risk**: high++### Variant 3: Hot path untouched, diagnostics as sidecar surfaces++- **Approach**: Keep the search result type essentially as it is — only the fixes that change the *answer* land inline (fusion weights, snippet anchoring, duplicate merge, quote fold, as-of threading, determinism) — and route every purely observational unit to a separate surface: a diagnostic verb that re-runs the query and emits the trace, true total and shadow signals; a store-check verb carrying schema presence and `PRAGMA integrity_check`; and a file-level census artifact for oversize chunks. Determinism is the enabling precondition rather than a peer unit: the sidecar's re-run is only trustworthy because the ranker no longer reads the wall clock.+- **Trade-offs**:+ - The hot result type gains no optional fields at all, so the preview budget, the description-length limit and the absence-assertion tests on the primary search surface are untouched.+ - Diagnostics can be as verbose as they need to be, since they are not competing with results for the same budget — the census in particular is a natural file artifact, not a response field.+ - The dependency between determinism and observability is structural rather than a matter of merge order, which is the strongest form of the guarantee.+ - Double execution: explaining a query means running it again, and any residual non-determinism (an unfixed source, a concurrent index write) makes the explanation describe a different run than the one the user saw — the failure mode is silent and confusing.+ - New surfaces cost registry budget in the MCP tool list and dispatcher entries in the CLI, and the constraint against a new default readiness probe means the store check must be explicitly invoked, weakening the "corrupt index serves wrong results indefinitely" fix to "corrupt index serves wrong results until someone asks".+ - Shadow signals computed in a second pass are shadow signals over a *replayed* context, which is weaker evidence than signals recorded in the run that actually served the user.+ - Some values genuinely belong inline — the true `total` is a property of the answer, not a diagnostic — so the split is not clean and the boundary ends up carrying one exception.+- **Complexity**: medium+- **Risk**: medium++### Recommended: Variant 1++**Rationale**: The wave's real shape is a boundary that refuses to carry values the core already computed, so the fix belongs at that boundary once rather than eleven times (Variant 2) or in a parallel surface that must re-derive them (Variant 3). A single envelope also matches the constraints most directly: one absence contract to prove flag-off byte identity, one place to hand a large trace to the existing artifact-reference escape hatch instead of spending MCP description or preview budget, and no new default probe. The invariant-floor-first ordering is the part that must not be traded away — serializing a trace or pinning shadow signals over a ranker that re-reads `Date.now()` would harden the non-determinism into the evaluation harnesses instead of removing it.diff --git a/docs/brainstorm/what-the-index-already-knew/cli-output/prompt.md b/docs/brainstorm/what-the-index-already-knew/cli-output/prompt.mdnew file mode 100644index 00000000..222a606c--- /dev/null+++ b/docs/brainstorm/what-the-index-already-knew/cli-output/prompt.md@@ -0,0 +1,84 @@+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.++# Task++Open Second Brain is a deterministic memory layer for AI agents over an Obsidian-compatible Markdown vault: a TypeScript/Bun CLI (`o2b`) plus an MCP server, with a derived SQLite index (`brain.sqlite`) holding documents, chunks, an FTS5 keyword index, a trigram index and embeddings.++Eleven kanban tasks were selected as one release wave, all on the retrieval path. A reconnaissance pass over the actual code falsified the load-bearing premise of nine of them. What remains after falsification is the real scope, and it has an unexpected shape: most of these are not missing capabilities but **computed values that are discarded before anyone can see them**, or **checks that were never run at all**. The wave must be designed around that shape rather than around the eleven original feature descriptions.++The surviving scope, unit by unit:++1. **Entity identity quote fold.** `normalizeEntityName` does `NFC + trim + collapse whitespace + lowercase`. No Unicode normal form maps U+2019 to U+0027, so `Taylor's` and `Taylor's` are two distinct canonical entities in the registry. Folding them structurally makes two previously-distinct records collide, and the registry refuses alias-claim collisions — that refusal must name its resolution rather than surfacing as an opaque upgrade failure. The function is documented in-code as a byte-stable identity kernel, so every currently-valid label must key identically to before.++2. **Weighted fusion under RRF.** A per-query intent profile (quoted phrase, wikilink density, query length) already scales the keyword/semantic/entity weights, and it is on by default. In the opt-in `rrf` fusion mode the relevance term discards that profile entirely, so a user who quotes a phrase gets no lexical preference and receives no signal that their intent was ignored.++3. **Ranking determinism.** The ranker resolves `nowMs` as `opts.nowMs ?? Date.now()`, and the assembly stage never passes it — so every live search re-reads the wall clock. The recency layer is non-zero by default, the first tie-break rung is an exact float comparison, and two identical calls can therefore return two different orderings. The test suite already documents this in comments and strips scores to work around it. Three candidate SQL statements also order by a score column with no unique tiebreaker.++4. **Duplicate passage merge.** A `content_hash` column has existed on chunks since the first schema version and is populated on every index run, but it is never projected into the hydrated row. Exact-duplicate passages are therefore demoted by the diversity reranker but never merged, and continue to occupy result slots.++5. **Match-anchored snippets.** Every snippet surface (600-char MCP content, 240-char cards, 140-char CLI) truncates from the head of the chunk, so a match in the middle of a long chunk is invisible in the preview. A match-centred window function already exists in this repository — mounted on the session-transcript grep surface, not on search — alongside a char-span-to-line-span helper that lives in the search module and has only that one caller.++6. **Retrieval decision trace.** A structure recording evaluated / surfaced / excluded candidates with per-exclusion reasons is built on every search and declared on the outcome type, and is serialized by neither the CLI nor the MCP surface. It is reachable only from a test. Separately, the reported `total` is assigned the returned row count, so a caller cannot distinguish "the vault has three matches" from "the vault has three hundred and you got ten".++7. **Shadow recall-quality signals.** Of seven proposed per-result signals, five are computable (three already computed and discarded) and two — acquisition risk and expected regret — have no backing data anywhere in the system and would have to be invented. The house pattern for a record-only lane that must never perturb production ordering already exists as a documented shadow-only advisor module.++8. **Oversize chunk census.** The default chunk size is 800 whitespace-delimited words; the recommended default embedding model has a 512-token input window. The shipped defaults overflow each other by roughly two to three times and nothing notices. A per-chunk token count is already stored. No provider in this system exposes model metadata and no local model file exists, so the true window cannot be probed without a network round trip or a hardcoded per-model table over an open set of endpoints.++9. **Schema presence check.** The read-open path verifies the schema version integer and nothing else. A database stamped with the current version but missing a table opens successfully and fails later inside a query with a raw SQL error.++10. **As-of expiry.** The expiration filter already takes `now` as a parameter with a default, and the topic-query options already thread it, but no surface ever passes it. The same surface is also missing the flag that reveals expired records, which exists on the MCP side only.++11. **Store integrity gate.** There is no SQLite integrity check anywhere in the codebase. A corrupt-but-parseable index passes the version check, opens cleanly, and serves wrong results indefinitely. The self-heal path triggers only on unreadable, mismatched or missing.++# Project context++Open Second Brain — TypeScript on the Bun runtime, roughly 900 modules under `src/`, with a deterministic kernel that makes no LLM calls. SQLite via `bun:sqlite`, FTS5 and a vector extension. Surfaces are a CLI and an MCP server over the same core.++Recent commits:++```+0ae4b097 feat: provenance at the boundary (v1.43.0)+7e6a5672 refactor: module boundaries and the fallbacks behind them (v1.42.0)+5ac866eb feat: signals that survive (v1.41.0)+0963ef0a feat: no dead ends - every diagnosis names its exit (v1.40.0)+f91a698b feat: context integrity gates (v1.39.0)+c31a2574 feat: semantic-health baseline watermark (v1.38.0)+b0c37977 feat: retrieval quality and context delivery (v1.37.0)+842d690f feat: knowledge intake and consolidation (v1.36.0)+95dc8577 feat: trusted recall and memory write surface (v1.35.0)+4b8100ca feat: source pipeline integrity and operator tooling (v1.34.0)+```++Related areas: the retrieval pipeline (request resolution, query shape, keyword lane, semantic lane, assembly, ranking, post-rank phases, final truncation), the SQLite store and its migration ladder, the entity registry over Markdown files, the MCP tool registry, and the CLI verb dispatcher.++Conventions:++- Every user-facing error resolves a registered diagnostic code that names the command which fixes it; a build ratchet enforces registration.+- A new field on a result type is opt-in and *absent* rather than null when the flag is off; tests assert key absence.+- Byte identity when a feature is off is proved by comparing a flag-on projection against a flag-off projection. There are no golden-blob snapshot tests.+- Census and ratchet tests must be demonstrably able to fail; a previous release shipped one that passed vacuously and it was treated as a defect.+- Search behaviour is pinned by a recall benchmark with hard floors and by exact pass-count assertions in several evaluation harnesses.++Constraints:++- **The SQLite schema version may not be bumped in this wave.** A bump forces every existing index into a mismatch, which triggers a silent self-heal reindex that drops every embedding unless the operator re-runs the embedding phase with an API key and spend. Every unit must therefore be a read projection, a new computation over existing columns, or a file-level artifact.+- No network calls in tests, and no natural-language word lists in any language for gating, classification, ranking or extraction — signals must be structural, corpus-frequency-derived, or score-derived.+- Fallbacks that silently do nothing are forbidden. Where a check cannot run, it must say that it did not run rather than reporting success.+- No new default readiness probe (an existing test pins the count), and the MCP registry enforces description length limits and a preview budget on every tool.++# Required output format++Produce exactly 3 distinct architectural variants. For each variant:++### Variant N: <short name>+- **Approach**: 2-3 sentences describing the variant.+- **Trade-offs**: bullet list of pros and cons.+- **Complexity**: small | medium | large+- **Risk**: low | medium | high++After the three variants, add exactly one recommendation:++### Recommended: Variant N+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.++Output nothing outside of these sections.diff --git a/docs/brainstorm/what-the-index-already-knew/design.md b/docs/brainstorm/what-the-index-already-knew/design.mdnew file mode 100644index 00000000..aebbca9b--- /dev/null+++ b/docs/brainstorm/what-the-index-already-knew/design.md@@ -0,0 +1,203 @@+# What the index already knew - eleven units on the retrieval path++**Status:** draft+**Author:** claude-dev-agent (via feature-release-playbook)+**Audience:** implementation++## Problem statement++Eleven kanban tasks were selected as one wave because they sit on one seam: the+path a query takes from text to answer, and the store that path reads. A+reconnaissance pass over the code preceded any design and falsified the+load-bearing premise of nine of them. What survived has a shape none of the task+descriptions predicted. These are almost never missing capabilities. They are+values the system already computes and then discards before anything can see+them, and checks that were never run at all.++A structure recording which candidates were evaluated, surfaced and excluded is+built on every single search and serialized by no surface. A content hash has+been on every chunk since the first schema version and is never read back. A+match-centred snippet function is written, tested, and mounted on the wrong+surface. An expiry filter takes an as-of date as a parameter and no caller ever+passes one. A ranker accepts an injected clock and the stage above it does not+inject. That is the wave.++## Scope++Nine invariant fixes and two observability units. Every one is a read+projection, a computation over columns that already exist, or a parameter that+is already declared - because the schema version cannot move in this wave.++**Invariant floor**++- Ranking determinism: the ranker stops reading the wall clock, and three+ candidate queries gain a unique tiebreaker.+- Fusion under `rrf` honours the per-query intent profile it currently discards.+- Exact-duplicate passages merge instead of occupying separate result slots.+- Every snippet surface anchors its window on the match instead of the head of+ the chunk.+- The entity identity kernel folds typographic quote variants, and the collision+ that fold creates names its own resolution.+- The as-of date and the show-expired flag reach the one surface where expiry is+ enforced.+- The store verifies that the tables it is about to query exist.+- The store verifies that it is not corrupt before it answers with it.+- An oversize-chunk census reports what the configured chunk size does against+ the configured model's declared window, and says so plainly when that window is+ unknown.++**Observability, after the floor and not before it**++- The retrieval decision trace and the true pre-truncation pool size are+ serialized under the existing explain flag.+- The recall-quality signals that are computable are recorded on the existing+ per-query telemetry record. The two that are not computable are refused.++## Out of scope++Named explicitly, because each was asked for and each is being declined on+evidence rather than deferred:++- **A runtime embedder-window probe.** No provider in this system exposes model+ metadata, no local model file exists, and the provider contract has no window+ field. The only implementations are a network round trip or a hardcoded table+ over an open set of endpoints.+- **A rechunk backfill verb.** `o2b search reindex` is already a full rechunk.+ An incremental one would need a record of the parameters the live index was+ built under, and that record is deleted before the swap.+- **A second index-health counter surface.** `indexStatus` already reports+ per-fault counts and named drift reasons through both the CLI and MCP, and it+ is the eighth such surface. The stall reasons the task asks for describe a+ background drain loop this project does not run.+- **Column type-drift detection.** SQLite accepts a text value into an integer+ column and still reports the column as integer. No table here is strict, there+ is no statement that can retype a column, and the drift that can really happen+ is invisible to the schema introspection the check would use.+- **`latest_only`.** Newest-wins is already the default on recall through three+ independent mechanisms, and the topic query already returns one current rule+ per topic. The documented flag is the one that turns it off.+- **`reference_date` on search.** It would either duplicate the shipped `until`+ filter or require a transaction-time column, and this wave bumps no schema.+- **A daily snapshot of the derived store and a restore verb.** `o2b brain+ snapshot` and `o2b brain rollback` already ship the checksum sidecar, the+ verification, the pre-restore diff that names what would change, the+ confirmation, the listing and the retention. What they do not cover is a+ derived artifact whose loss costs spend rather than information.+- **Possessive-suffix folding.** A suffix rule is a natural-language word list.+ The structural alternative already exists and is deliberately proposal-only.+- **Acquisition risk and expected regret as recall signals.** There is no cost+ model and no fetch ledger behind the first. The second needs a counterfactual+ outcome, and every outcome this system holds arrives after the turn from an+ explicit caller call. Both would be invented numbers.+- **A `best_span` byte range on the result row.** The row already carries a path+ and a line span, thirty-three test files read that shape, and the preview+ budget parks the payload regardless, so a wider row buys no inline tokens.++## Chosen approach++The consultant produced three variants and recommended the first: land the+correctness invariants unflagged, then expose the discarded values through one+opt-in envelope assembled at a single boundary, with the store checks on the+read-open seam and the census as a file-level artifact.++That recommendation is adopted, with one deviation.++**Deviation: the shadow recall signals do not join the explain envelope.** The+consultant flagged the risk itself - that an envelope named `explain` accretes+unrelated concerns - and here the two lanes have different consumers and+different lifetimes. The trace is returned to the caller of one query and dies+with the response. The signals are recorded to a persistent append-only log for+later analysis, behind a telemetry gate that already exists and already defaults+off. Merging them would compute per-result signals on every explain call that+does not want them, and would write to the continuity log on behalf of callers+who asked only to see a trace. The two gates stay separate: `explain` returns,+`telemetry` records.++**Ordering is a structural dependency, not a preference.** Determinism lands+first. Serializing a trace or recording signals over a ranker that re-reads the+clock would faithfully document noise and pin it into the evaluation harnesses.++## Design decisions++- **The clock is injected, not read.** The ranker already accepts `nowMs`. The+ fix is that assembly passes the request's already-resolved value instead of+ letting the ranker default to the wall clock. This is one line, and it is the+ only defect in this wave a user can observe as the same query returning two+ different orderings.+- **The determinism test must observe the failure it prevents.** A test that runs+ the same query twice with the cache off and compares row identity is the+ assertion the suite lacks; the existing near-neighbours either read from cache,+ or use fixtures with no score ties, or strip scores first precisely to work+ around this bug. The new test constructs a tie deliberately.+- **Weighted fusion, not weighted-by-exception.** Under `rrf` the intent profile+ is applied to the per-lane contributions rather than to a post-fusion score.+ That is a standard formulation, it preserves what reciprocal-rank fusion is+ for, and it makes a quoted phrase mean the same thing in both fusion modes.+- **The duplicate merge runs before diversity reranking**, where it is a linear+ pass that shrinks a quadratic loop, rather than after it, where it would waste+ the comparisons it makes redundant. Merged-away locations are reported on the+ surviving row, never silently dropped.+- **One snippet function, three callers.** The match-centred window and the+ char-span-to-line-span helper already exist. They are lifted into a shared+ module and the three head-truncating surfaces call it. When no query term+ occurs in the content the window is the head of the chunk, which is the same+ bytes as today. That is a defined behaviour with a test, not a silent+ fallback.+- **A fold that creates a collision must name the exit.** Folding quote variants+ in the identity kernel makes two previously-distinct entity records resolve to+ one key, and the registry refuses that. The refusal resolves a registered+ diagnostic naming the command that merges them. An upgrade that starts failing+ with an opaque error would be a worse outcome than the duplicate it fixes.+- **A check that cannot run says so.** The oversize census compares the+ configured chunk size against the configured model's declared window. For a+ model outside the curated preset table there is no window to compare against,+ and the census reports that the check did not run for that model. Treating an+ unknown window as a pass would be exactly the misleading do-nothing this+ project forbids.+- **Integrity is checked where the store is opened**, so that a+ corrupt-but-parseable index is classified as unreadable and takes the existing+ self-heal path, instead of answering questions with corrupt data indefinitely.+- **`total` stops being the row count.** It is currently assigned the length of+ the returned array, which makes it carry no information at all. The+ pre-truncation pool size is a property of the answer and is reported inline,+ not behind the explain flag.+- **Two signals are refused in writing.** Acquisition risk and expected regret+ are named in this document, in the changelog and on the board as not+ implementable on this architecture, rather than shipped as plausible numbers.++## File changes++Expected to be touched, by unit:++- Determinism: `src/core/search/pipeline/assemble.ts`, `src/core/search/store/keyword.ts`, `src/core/search/store/vectors.ts`, `src/core/search/store/trigram.ts`+- Fusion: `src/core/search/ranker.ts`, `src/core/search/fusion.ts`+- Duplicate merge: `src/core/search/store/chunks.ts`, `src/core/search/pipeline/assemble.ts`+- Snippets: new shared module under `src/core/search/`, `src/mcp/search-tools.ts`, `src/core/search/cards.ts`, `src/cli/search/outcome-render.ts`, `src/core/brain/session-recall.ts`+- Entity fold: `src/core/brain/entities/canonical.ts`, `src/core/brain/entities/registry.ts`, `src/core/brain/diagnostics.ts`+- As-of expiry: `src/cli/brain/verbs/query.ts`, `src/mcp/brain/query-tools.ts`, `src/mcp/search-tools.ts` (stale filter description)+- Schema presence: `src/core/search/store/lifecycle.ts`, `src/core/search/schema.ts`+- Integrity gate: `src/core/search/store/lifecycle.ts`, `src/core/search/pipeline/store-open.ts`+- Oversize census: `src/core/search/embeddings/presets.ts`, `src/core/search/indexer.ts`, `src/core/search/types.ts`, `src/cli/search/verbs/status.ts`+- Explain envelope: `src/core/search/pipeline/outcome.ts`, `src/cli/search/outcome-render.ts`, `src/mcp/search-tools.ts`+- Shadow signals: `src/core/brain/recall-telemetry.ts`, `src/mcp/search-tools.ts`++## Risks and open questions++- **The recall benchmark has roughly a tenth of margin** above its floors, and+ several evaluation harnesses assert exact pass counts rather than thresholds.+ Duplicate merge, fusion weighting and snippet anchoring can all move those+ numbers. Any movement is attributed to a named unit and explained, never+ absorbed silently.+- **The entity fold changes an identity kernel** that is documented in-code as+ byte-stable. The proof obligation is a corpus of currently-valid labels keying+ identically before and after, not a spot check.+- **The schema version is frozen for this wave.** A unit that concludes it needs+ a column stops and reports rather than bumping, because a bump forces every+ existing index through a self-heal reindex that drops embeddings unless the+ operator re-runs the embedding phase with spend.+- **An evaluation module has no production caller.** The graph-holdout harness is+ reachable only from tests. Whether it is wired to a surface or removed is a+ decision this wave takes explicitly rather than leaving it as it found it.+- **The continuity log has no retention policy.** Recording per-result signals+ keeps the existing per-query record shape rather than adding a record per+ result, because nothing prunes that file.diff --git a/docs/brainstorm/what-the-index-already-knew/plan.md b/docs/brainstorm/what-the-index-already-knew/plan.mdnew file mode 100644index 00000000..70c10dae--- /dev/null+++ b/docs/brainstorm/what-the-index-already-knew/plan.md@@ -0,0 +1,104 @@+# What the index already knew - implementation plan++Eleven units. The invariant floor lands first; the two observability units land+last and depend on it. Every unit is a read projection, a computation over+existing columns, or a parameter that already exists - the SQLite schema version+does not move in this wave.++## Tasks++### Task C: the ranker stops reading the wall clock+- **Files**: `src/core/search/pipeline/assemble.ts`, `src/core/search/store/keyword.ts`, `src/core/search/store/vectors.ts`, `src/core/search/store/trigram.ts`, `tests/core/search/`+- **Acceptance**: a test that runs `search()` twice with the query cache off,+ over a fixture built so that two rows tie on score, and asserts identical row+ identity and order. It must fail before the fix - paste the failure. The three+ candidate queries end their ordering in a unique column.+- **Depends on**: none. Everything else in the wave depends on this.++### Task B: `rrf` fusion honours the per-query intent profile+- **Files**: `src/core/search/ranker.ts`, `src/core/search/fusion.ts`, `tests/core/search/`+- **Acceptance**: a quoted-phrase query prefers the verbatim hit in `rrf` mode as+ it already does in `linear` mode. The default `linear` path is byte-identical -+ proved by comparing a flag-off projection, not by inspection.+- **Depends on**: C.++### Task D: exact-duplicate passages merge+- **Files**: `src/core/search/store/chunks.ts`, `src/core/search/pipeline/assemble.ts`, `tests/core/search/`+- **Acceptance**: two byte-identical passages in two files return one row naming+ both locations; two distinct passages from one file still return two rows; a+ near-duplicate is untouched and still demoted by the diversity reranker. The+ merge runs before diversity reranking. The returned window is not silently+ shorter than the requested limit.+- **Depends on**: C.++### Task E: snippets anchor on the match+- **Files**: new shared module under `src/core/search/`, `src/mcp/search-tools.ts`, `src/core/search/cards.ts`, `src/cli/search/outcome-render.ts`, `src/core/brain/session-recall.ts`, `tests/`+- **Acceptance**: a match late in a long chunk appears in the preview on all+ three surfaces. A chunk with no query-term occurrence returns the same bytes as+ today - asserted, not assumed. The existing session-grep behaviour is unchanged+ after the helper is lifted; one function has three callers, not three copies.+- **Depends on**: none.++### Task A: the entity identity kernel folds quote variants+- **Files**: `src/core/brain/entities/canonical.ts`, `src/core/brain/entities/registry.ts`, `src/core/brain/diagnostics.ts`, `tests/core/brain/`+- **Acceptance**: two entity records differing only in apostrophe form resolve to+ one identity key. The collision this creates raises a registered diagnostic+ naming the command that resolves it, never an opaque failure. A corpus of+ currently-valid labels keys identically before and after. A separate test locks+ the property that both apostrophe forms already retrieve the same rows through+ the real search lanes - it must be able to fail, so prove it against a+ deliberately broken tokenizer configuration or an injected fixture.+- **Depends on**: none.++### Task J: the as-of date reaches the surface that enforces expiry+- **Files**: `src/cli/brain/verbs/query.ts`, `src/mcp/brain/query-tools.ts`, `src/mcp/search-tools.ts`, `tests/`+- **Acceptance**: `o2b brain query` accepts an as-of instant and a show-expired+ flag, and a record that expired after that instant is returned. Omitting both+ produces byte-identical output to today. The `since`/`until` MCP descriptions+ stop claiming they filter on modification time.+- **Depends on**: none.++### Task I: the store verifies the tables it is about to query+- **Files**: `src/core/search/store/lifecycle.ts`, `src/core/search/schema.ts`, `tests/core/search/`+- **Acceptance**: a database stamped with the current version but missing a table+ or a column is refused at read-open through the existing mismatch diagnostic+ that names the reindex command, instead of failing later inside a query. The+ manifest is derived from the schema, not typed by hand twice.+- **Depends on**: none. Must not bump the schema version.++### Task K: a corrupt store is not answered from+- **Files**: `src/core/search/store/lifecycle.ts`, `src/core/search/pipeline/store-open.ts`, `tests/core/search/`+- **Acceptance**: a deliberately corrupted-but-parseable index is classified as+ unreadable and takes the existing self-heal path rather than serving results.+ The check's cost on a healthy store is measured and reported in the unit's+ report, not estimated.+- **Depends on**: I (both land on the same read-open seam).++### Task H: the oversize-chunk census+- **Files**: `src/core/search/embeddings/presets.ts`, `src/core/search/indexer.ts`, `src/core/search/types.ts`, `src/cli/search/verbs/status.ts`, `tests/`+- **Acceptance**: an index whose configured chunk size exceeds the configured+ model's declared window reports a counted, named warning with a registered+ exit. A model with no declared window reports that the check did not run for+ it - not a pass. The count comes from the token count already stored, with no+ schema change. A new statistics field is absent when the condition does not+ arise.+- **Depends on**: none.++### Task F: the trace that was built and thrown away+- **Files**: `src/core/search/pipeline/outcome.ts`, `src/cli/search/outcome-render.ts`, `src/mcp/search-tools.ts`, `tests/`+- **Acceptance**: the retrieval decision trace and the trust assessment are+ serialized on both surfaces under the existing explain flag, and absent - not+ null - when it is off. The reported total becomes the pre-truncation pool size+ and is reported inline; a query matching many documents no longer reports a+ total equal to the number of rows returned.+- **Depends on**: C. A trace over a wall-clock ranker documents noise.++### Task G: the recall signals that can be computed+- **Files**: `src/core/brain/recall-telemetry.ts`, `src/mcp/search-tools.ts`, `tests/`+- **Acceptance**: the computable signals are recorded on the existing per-query+ telemetry record behind its existing gate, one record per query and not one per+ result. Production ordering is proved unchanged with the gate on. The two+ uncomputable signals appear nowhere in the code. The unit also reports whether+ the graph-holdout module has any production caller and proposes wiring or+ removal - it does not remove anything without reporting first.+- **Depends on**: C, F.diff --git a/docs/brainstorm/what-the-index-already-knew/variants.md b/docs/brainstorm/what-the-index-already-knew/variants.mdnew file mode 100644index 00000000..78344baa--- /dev/null+++ b/docs/brainstorm/what-the-index-already-knew/variants.md@@ -0,0 +1,89 @@+# Variant audit trail++Consultant: Claude Code, single pass, exit 0. Fallback consultant not invoked -+the primary returned three parseable variants and a recommendation.++The prompt is at `cli-output/prompt.md` and the verbatim response at+`cli-output/claude.md`. The three variants are reproduced below in full,+followed by the orchestrator's decision.++### Variant 1: Invariant floor, then one explain envelope++- **Approach**: Land the correctness invariants first as unflagged fixes (inject `nowMs` from assembly, add unique tiebreakers to the three SQL orderings, fold the quote in `normalizeEntityName` with a named collision-resolution diagnostic, project `content_hash` into the hydrated row, thread `now` and the expired flag through the as-of surface, mount the existing match-window + char-span helpers on all three snippet surfaces, restore the intent profile inside the RRF relevance term). Then expose everything currently computed-and-discarded through a single opt-in `explain` envelope assembled once at the assembly stage — trace (evaluated/surfaced/excluded), true pre-truncation `total`, duplicate-merge records, shadow recall signals, snippet spans — serialized by one shared boundary projector that both the CLI and the MCP tool call. Store integrity (`PRAGMA integrity_check`) and schema table presence become one explicit `store check` path on the read-open seam that reports `not_run` rather than success when it cannot run; the oversize-chunk census ships as a file-level artifact from the existing token counts with the unprobeable true window declared as unknown.+- **Trade-offs**:+ - One flag, one absence contract, one place where "field is absent when off" is proved — the flag-on vs flag-off byte-identity comparison is written once, not eleven times.+ - The envelope is the natural home for the preview-budget escape hatch: a large trace becomes an artifact reference instead of a new tool or a description-length fight in the MCP registry.+ - Ordering is load-bearing and correct: a trace serialized over a wall-clock-reading ranker would faithfully document noise, so determinism must precede observability rather than ship beside it.+ - The envelope accretes unrelated concerns under one name; "explain" ends up meaning trace, dedup, shadow signals and spans at once, and later readers must be told which sub-object answers which question.+ - Sub-flags will eventually be wanted (shadow signals cost more to compute than the trace does), and retrofitting granularity into a shipped single flag is more work than starting granular.+ - The unflagged invariant fixes move real orderings, so the recall benchmark floors and the exact pass-count assertions must be re-pinned in the same wave, with the deltas attributed unit by unit rather than absorbed silently.+- **Complexity**: medium+- **Risk**: medium++### Variant 2: Eleven independent seam slices++- **Approach**: Treat each surviving unit as a self-contained vertical slice landed at the pipeline seam it belongs to — hydration gets `content_hash`, ranking gets injected time and tiebreakers, post-rank gets duplicate merge and shadow signals, fusion gets the intent profile, the registry gets the quote fold, the store gets presence and integrity checks, each snippet surface gets its own anchoring — with its own opt-in flag where a field is added, its own diagnostic code, and its own byte-identity proof. Shared discipline comes from a written convention checklist and the existing build ratchet, not from shared machinery. Each slice is independently revertable and independently reviewable.+- **Trade-offs**:+ - Maximum revert granularity: if the fusion weighting regresses the recall benchmark, that single slice backs out without disturbing the duplicate merge or the integrity gate.+ - Each slice's flag-off byte identity is trivially provable in isolation, which is exactly the shape the project's existing test convention wants.+ - No premature abstraction over units that are genuinely unrelated — the quote fold and the SQLite integrity check share nothing but a release wave.+ - Flag proliferation is real and permanent: five-plus new opt-in flags on one result type, a combinatorial on/off matrix nobody will fully test, and an MCP tool description that must name them all inside the registry's length limit.+ - Boundary plumbing gets rewritten per slice — each new optional field needs its own CLI serialization, its own MCP projection, its own preview-budget accounting — and the copies drift.+ - The trace and the `total` fix are the same defect seen twice (values the boundary refuses to carry); splitting them hides that the boundary projector, not the producers, is the thing that needs fixing.+ - Ordering discipline is only advisory, so nothing structurally prevents shipping the trace before determinism and pinning noise into the evaluation harnesses.+- **Complexity**: large+- **Risk**: high++### Variant 3: Hot path untouched, diagnostics as sidecar surfaces++- **Approach**: Keep the search result type essentially as it is — only the fixes that change the *answer* land inline (fusion weights, snippet anchoring, duplicate merge, quote fold, as-of threading, determinism) — and route every purely observational unit to a separate surface: a diagnostic verb that re-runs the query and emits the trace, true total and shadow signals; a store-check verb carrying schema presence and `PRAGMA integrity_check`; and a file-level census artifact for oversize chunks. Determinism is the enabling precondition rather than a peer unit: the sidecar's re-run is only trustworthy because the ranker no longer reads the wall clock.+- **Trade-offs**:+ - The hot result type gains no optional fields at all, so the preview budget, the description-length limit and the absence-assertion tests on the primary search surface are untouched.+ - Diagnostics can be as verbose as they need to be, since they are not competing with results for the same budget — the census in particular is a natural file artifact, not a response field.+ - The dependency between determinism and observability is structural rather than a matter of merge order, which is the strongest form of the guarantee.+ - Double execution: explaining a query means running it again, and any residual non-determinism (an unfixed source, a concurrent index write) makes the explanation describe a different run than the one the user saw — the failure mode is silent and confusing.+ - New surfaces cost registry budget in the MCP tool list and dispatcher entries in the CLI, and the constraint against a new default readiness probe means the store check must be explicitly invoked, weakening the "corrupt index serves wrong results indefinitely" fix to "corrupt index serves wrong results until someone asks".+ - Shadow signals computed in a second pass are shadow signals over a *replayed* context, which is weaker evidence than signals recorded in the run that actually served the user.+ - Some values genuinely belong inline — the true `total` is a property of the answer, not a diagnostic — so the split is not clean and the boundary ends up carrying one exception.+- **Complexity**: medium+- **Risk**: medium++### Recommended: Variant 1++**Rationale**: The wave's real shape is a boundary that refuses to carry values the core already computed, so the fix belongs at that boundary once rather than eleven times (Variant 2) or in a parallel surface that must re-derive them (Variant 3). A single envelope also matches the constraints most directly: one absence contract to prove flag-off byte identity, one place to hand a large trace to the existing artifact-reference escape hatch instead of spending MCP description or preview budget, and no new default probe. The invariant-floor-first ordering is the part that must not be traded away — serializing a trace or pinning shadow signals over a ranker that re-reads `Date.now()` would harden the non-determinism into the evaluation harnesses instead of removing it.++---++## Orchestrator decision++**Variant 1 is adopted, with one deviation.**++The recommendation is accepted on its central argument: the wave's real shape is+a boundary that refuses to carry values the core already computed, so the fix+belongs at that boundary once rather than eleven times. Variant 2 would have+produced five or more new opt-in flags on one result type with a combinatorial+matrix nobody would test, and would have hidden that the trace and the mis-set+total are the same defect seen twice. Variant 3 would have explained a query by+running it again, which makes the explanation describe a different execution than+the one the user saw - and its store check, being explicitly invoked, would have+weakened "a corrupt index serves wrong results indefinitely" to "until someone+asks".++**Deviation: the shadow recall signals stay out of the explain envelope.**++Variant 1 places the trace, the true total, the duplicate-merge records, the+snippet spans and the shadow signals in one envelope behind one flag. The+consultant named the risk in its own trade-off list, and for the signals it is+not a stylistic risk but a wrong coupling. The trace is returned to the caller of+one query and dies with the response. The signals are recorded to a persistent+append-only log for later analysis, behind a telemetry gate that already exists+and already defaults off. Merging them would compute per-result signals for every+caller who asked only to see a trace, and would write to a log file that has no+retention policy on behalf of callers who never asked to be logged. The gates+stay separate: `explain` returns, `telemetry` records.++**What is kept from the recommendation without change:** the invariant floor+lands before the observability units, and that ordering is a structural+dependency rather than a preference. Serializing a trace or recording signals+over a ranker that re-reads the wall clock would faithfully document noise and+pin it into the evaluation harnesses instead of removing it.diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex c1032c3c..4dcfaa7a 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -36,7 +36,7 @@ o2b brain digest Render a Markdown or JSON summary of recent Braino2b brain intent-review Read-only pre-dream review of active signal clusters; --now ISO; --json mirrors brain_intent_reviewo2b brain retention Recommendation-only lifecycle review over retired preferences and processed signals; --now ISO; --json mirrors brain_retentiono2b brain monthly Month-level Brain synthesis over timeline events, transitions, retirements, contradictions, and neglected areas; --month YYYY-MM; --json mirrors brain_brief view=monthly-o2b brain query Read helper: by preference, by topic, or by log timestamp+o2b brain query Read helper: by preference, by topic, or by log timestamp; --topic adds point-in-time recall over the expiration filter: --at <ISO|YYYY-MM-DD> evaluates expiration_date as of that instant (default now, unparseable is refused with exit 2, never coerced) and --show-expired keeps lapsed memories. Both apply to --topic only and are refused elsewhere; omitting both leaves the output byte-identicalo2b brain agent-query Read source-agent provenance; filters by --agent, --topic, --query, --kind, --limit; --json mirrors brain_agent_queryo2b brain agent-diff Compare source-agent coverage in browse/search/diff/map modes; --json mirrors brain_agent_diffo2b brain reject (CLI-only) Retire a preference; requires --reason "<text>". Subsequent signals on the same topic are suppressed.@@ -351,7 +351,7 @@ byte-identical when unset.o2b brain telegram-capture run | catchup - explicit inbound capture runner (fetch-based long-poll getUpdates); gated by telegram_bot_token (env TELEGRAM_BOT_TOKEN) and the telegram_chat_allowlist chat-id allowlist (env TELEGRAM_CHAT_ALLOWLIST, empty accepts nothing); each accepted message becomes one staged capture note under Brain/captures/ with provenance; rejected updates log one decision each; catchup replays captures since the last acknowledged one; a missing token is a typed startup erroro2b brain inbox-drain [--apply] - walk staged captures, classify each structurally (URL-shaped body -> source reference, explicit obligation marker -> task, otherwise atomic idea), route on apply (source ingest, note in captured/, obligation open), archive processed captures, and report every item with action and reason; dry-run default writes nothing; rerun after apply is a no-opo2b brain diarize <entity> [--json] - entity profile skeleton plus one needs-llm-step envelope; the stated-vs-evidenced section is computed deterministically (stated claims versus evidence frequency and recency), every line carrying an evidence identity; unknown entity is a typed error-o2b brain repair-lane [--apply --confirm "apply repair"] - propose link-graph edges ordered by identity strength (explicit references, session continuity, same-topic evidence; inferred opt-in) under a confidence threshold and a hard per-run write cap; dry-run default; reruns after apply converge to zero writes; the holdout harness reports graph lift separately from direct recall and fails on dangling or unhydrated targets+o2b brain repair-lane [--apply --confirm "apply repair"] - propose link-graph edges ordered by identity strength (explicit references, session continuity, same-topic evidence; inferred opt-in) under a confidence threshold and a hard per-run write cap; dry-run default; reruns after apply converge to zero writes. --apply plans first and then runs the paired graph-efficacy holdout harness over every edge that plan proposes, taking anchor and target from the edge itself: it reports graph lift (targets reachable only through the graph) apart from direct recall (targets already a 1-hop neighbour) and refuses the apply, writing nothing, when any target resolves to no durable-memory note or hydrates into no evidence. The refusal names the failing edges with their verdict and resolves its exit through the registered repair-holdout-unresolved diagnostic; --json carries the counts in an additive holdout object beside next_command. Dry-run does not evaluate the gate and its bytes are unchanged - it already names each unresolvable endpoint as a skip-missing-target decisiono2b brain deep-synthesis --json now also carries findings with causal_context, decomposed confidence (support, opposition, freshness, coverage), and the excluded_findings ledger with excluded_finding_count```@@ -581,11 +581,16 @@ o2b search "<query>" Hybrid full-text + semantic search across the vaul--query-doc '<lanes>' separates intent/lex/vec/hyde recall lanes--evidence-pack adds matched/missing term diagnostics, abstention text,IDF-weighted coverage, per-token union records, and a completeness verdict- --since/--until scope recall by document mtime (ISO date/datetime,+ --since/--until scope recall by event time - frontmatter validity,+ else the body-derived anchor, else mtime (ISO date/datetime,today / yesterday / last week / last month, or 24h / 7d / 2w shorthand)--include-superseded keeps superseded predecessors undemoted (history mode)--verbose adds per-result why_retrieved reasons+ --explain appends the retrieval decision trace and the memory trust+ assessment (what the trust gate evaluated, surfaced and excluded, with+ reasons); with the gate off it says so and names the switch that enables it--json for structured output (includes reasons[])+ total is the pre-truncation ranked pool, not the number of rows returnedCJK text is expanded for FTS recall without polluting returned contento2b search feedback Record explicit recall feedback for one result(--query Q --result <path> --verdict up|down; one JSON event file@@ -613,6 +618,20 @@ o2b search vector-backfill Run the vector phase ALONE for indexed chunks thatvector-backfill Brain log evento2b search status Index status; since v0.36.0 also reports the active embeddingsignature (<provider>:<model>:<dimension>) and a refresh-cost estimate+o2b search check Pre-flight diagnostics: vault, index directory, SQLite/FTS5, the+ vector extension, the embedding key, the provider, the vector ABI stamp+ --integrity additionally runs a full PRAGMA quick_check over the index+ file, on demand and regardless of the store's 24-hour interval gate.+ Opt-in because the scan is linear in the index size (~22 ms per MB, so+ about 30 s on a 1.3 GB index); the size and a time estimate are printed+ on stderr before it starts and the time spent is reported after.+ The verdict lands in the same integrity_checked_at / integrity_fault+ cells the write-open gate writes, so a condemned file is refused by+ every later read; a fault names o2b search reindex and exits 1.+ A store that has never been scanned reports previous_full_check: never -+ the absence of a fault is never reported as a pass.+ --json adds an integrity object; without the flag the output is+ unchanged and no index_state cell is touchedo2b search provider add NAME Register an OpenAI-compatible embedding endpoint (since v0.36.0)--base-url U --model M --env-key K (K is the env var NAME holding the key);persisted to Brain/search/embedding-providers.json, resolved after built-insdiff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex ab9b68ee..1163169b 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -776,10 +776,16 @@ Key behaviours, all driven from `Brain/_brain.yaml`-free `search_*` /unless `--force-cost`, and the local / unlisted-model price is 0.- **Fusion modes (v0.36.0).** `search_fusion_mode` is `linear` (default,the weighted sum below) or `rrf` - Reciprocal Rank Fusion, which scores- each candidate `Σ 1/(search_rrf_k + rank_in_lane)` across the keyword- and semantic lanes, min-max-normalised to `[0,1]`. RRF is weightless- and replaces only the relevance term; every boost below still applies,- and `linear` is bit-identical to pre-v0.36.0 ranking.+ each candidate `Σ w_lane/(search_rrf_k + rank_in_lane)` across the+ keyword and semantic lanes, min-max-normalised to `[0,1]`. RRF ignores+ the configured `search_keyword_weight` / `search_semantic_weight` -+ those calibrate two score magnitudes and this fusion reads none - but+ `w_lane` does carry the per-query intent profile (and the learned+ recall weights), so a quoted phrase favours the lexical lane in both+ fusion modes. A query that declares no intent weights both lanes 1,+ which is the classic weightless term. RRF replaces only the relevance+ term; every boost below still applies, and `linear` is bit-identical to+ pre-v0.36.0 ranking.- **Ranking.** `final_score = clamp01(keyword_weight·norm_BM25 +semantic_weight·cosine + link_boost + recency_boost + entity_boost)`(linear mode; in `rrf` mode the first two terms are replaced by the@@ -850,7 +856,10 @@ semantic_weight·cosine + link_boost + recency_boost + entity_boost)`vectors.The MCP `brain_search` tool returns at most 50 results with each-chunk's `content` truncated to 600 characters; diagnostic score+chunk's `content` capped at 600 characters - a window centred on the+first significant query term the chunk contains, marked with an ellipsis+on whichever side was cut, and the head of the chunk when it contains+none; diagnostic scorecomponents (`keywordScore`, `semanticScore`, `linkBoost`,`recencyBoost`) are intentionally absent from the MCP shape — theylive in the CLI's `--verbose` output only, to keep the agent contextdiff --git a/docs/mcp.md b/docs/mcp.mdindex 6604aacb..ede5c28a 100644--- a/docs/mcp.md+++ b/docs/mcp.md@@ -101,7 +101,9 @@ alias-to-replacement table lives in `docs/updating.md`.`vec:`, and `hyde:` lanes; `focus_query` / `focus_path_prefix` to steer asingle call; `since` / `until` time ranges (ISO date/datetime, `today`,`yesterday`, `last week`, `last month`, or `<n>h`/`<n>d`/`<n>w` shorthand,-filtered on document mtime); `include_superseded: true` to keep superseded+filtered on event time - frontmatter validity first, then the+body-derived anchor, with document mtime only as the last rung);+`include_superseded: true` to keep supersededpredecessors undemoted (history mode); and `evidence_pack: true` to returnsignificant/matched/missing terms, abstention text, terminal-state downrankreasons, per-result `why_retrieved`, IDF-weighted coverage with rare-term@@ -109,6 +111,21 @@ classification, per-token `union_records` for uncovered terms, and a`completeness` verdict whose `uncovered_but_present_in_corpus` list is thefalse-absence guard. It can also emit opt-in recall telemetry with`telemetry: true`.+Since v1.44.0 `explain: true` adds two top-level receipts alongside the+per-result `score_breakdown`: `retrieval_decision_trace` (evaluated /+surfaced / excluded counts plus every excluded candidate as a compact+reference with its structural reasons) and `memory_trust_assessment` (the+same exclusion set as a reason histogram). Both are by-products of the+retrieval trust gate; with the gate off the response carries+`retrieval_trace_unavailable` naming the switch that produces them+(`search_trust_gate_enabled`) instead of going quiet. Without `explain` no+key appears at all. In the same release `total` stopped echoing the row+count: it is the ranked candidate pool the `limit` sliced the returned rows+from, so `total > results.length` means the ranker had more candidates+than it handed you. Read it as candidates ranked, not as a corpus+statistic: the pool is capped at roughly three times the requested+`limit`, so it moves with `limit` and saturates on a large vault, and+link traversal can add a document that matched no term at all.A deterministic summary-search router (t_7b96f242) inspects each query forstructural summary signals - a source-targeted `source:<path>` token, or a`kind:<t>`/`type:<t>` token whose value is a declared artifact kind in thediff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex e5ab4458..e41c60c5 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.43.0",+ "version": "1.44.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex a737f7ab..e22eada8 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.43.0",+ "version": "1.44.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 4dc40c09..2d1f4fe5 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.43.0"+version: "1.44.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex f4bd1109..c8b3345b 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.43.0",+ "version": "1.44.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/_schemas.py b/plugins/hermes/_schemas.pyindex d2616007..57e181e2 100644--- a/plugins/hermes/_schemas.py+++ b/plugins/hermes/_schemas.py@@ -185,6 +185,12 @@ STATIC_TOOL_SCHEMAS: tuple[dict[str, Any], ...] = ('false (expired memories are ''silently dropped from the ''result).'},+ 'at': {'type': 'string',+ 'description': 'Topic mode only: evaluate '+ '`expiration_date` as of this instant, '+ 'so a memory that lapsed after it '+ 'comes back. ISO-8601 instant or '+ 'YYYY-MM-DD. Default now.'},'since': {'type': 'string','description': 'ISO-8601 timestamp; returns every ''Brain log event with timestamp >= '@@ -241,15 +247,17 @@ STATIC_TOOL_SCHEMAS: tuple[dict[str, Any], ...] = ('false.'},'since': {'type': 'string','maxLength': 64,- 'description': 'Time-aware recall: only documents '- 'modified at/after this point. ISO '- "date/datetime, 'today', 'yesterday', "- "'last week', 'last month', or "+ 'description': 'Hard filter on event time '+ '(validity, body anchor, mtime '+ 'last): at/after this point. ISO '+ 'date/datetime, today, yesterday, '+ 'last week, last month, ''<n>h/<n>d/<n>w.'},'until': {'type': 'string','maxLength': 64,- 'description': 'Time-aware recall: only documents '- 'modified at/before this point. Same '+ 'description': 'Hard filter on event time '+ '(validity, body anchor, mtime '+ 'last): at/before this point. Same '"forms as 'since'."},'limit': {'type': 'integer', 'minimum': 1, 'maximum': 50},'semantic': {'type': 'boolean'},@@ -262,10 +270,11 @@ STATIC_TOOL_SCHEMAS: tuple[dict[str, Any], ...] = ('persisted self-tuning. Absent ''leaves ranking unchanged.'},'explain': {'type': 'boolean',- 'description': 'Include a structured '- 'score_breakdown (per-layer numeric '- 'components) on each result. '- 'Default false.'},+ 'description': 'Add a per-result '+ 'score_breakdown plus the '+ 'retrieval_decision_trace and '+ 'memory_trust_assessment '+ 'receipts. Default false.'},'trust': {'type': 'boolean','description': 'Stamp each result with inline trust ''metadata (age_days, superseded, '@@ -319,6 +328,13 @@ STATIC_TOOL_SCHEMAS: tuple[dict[str, Any], ...] = ('keys is AND.','additionalProperties': {'type': 'array','items': {'type': 'string'}}},+ 'degree': {'type': 'array',+ 'description': 'Graph-degree predicates over '+ 'backlink/outlink counts, e.g. '+ "'backlinks=0' (orphans) or "+ "'outlinks>=5' (hubs); ANDed. "+ 'Absent = no filter.',+ 'items': {'type': 'string'}},'visibility': {'type': 'array','description': 'Optional content-visibility ''scope; untagged pages always '@@ -333,6 +349,20 @@ STATIC_TOOL_SCHEMAS: tuple[dict[str, Any], ...] = ('owner-tagged pages only their ''owner. Absent = no ownership ''filtering.'},+ 'session_scope': {'type': 'string',+ 'description': 'Optional session-scope '+ 'filter; pages with no '+ 'session always match, '+ 'session-tagged pages only '+ 'this session. Absent = no '+ 'session filtering.'},+ 'project_scope': {'type': 'string',+ 'description': 'Optional project-scope '+ 'filter; pages with no '+ 'project always match, '+ 'project-tagged pages only '+ 'this project. Absent = no '+ 'project filtering.'},'disclosure': {'type': 'string','enum': ['full', 'cards'],'description': "Result depth: 'full' "@@ -458,7 +488,14 @@ STATIC_TOOL_SCHEMAS: tuple[dict[str, Any], ...] = ('emitted telemetry.'},'turn_id': {'type': 'string','description': 'Optional turn id recorded on '- 'emitted telemetry.'}},+ 'emitted telemetry.'},+ 'agent_scope': {'type': 'string',+ 'description': 'Optional agent-ownership '+ 'scope; shared (ownerless) '+ 'memories always match, '+ 'owner-tagged memories only '+ 'their owner. Absent = no '+ 'ownership filtering.'}},'required': ['max_tokens'],'additionalProperties': False}},{'name': 'brain_pre_compact_extract',diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 4dc40c09..2d1f4fe5 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.43.0"+version: "1.44.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex 35ca90a1..db06fc7f 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.43.0"+version = "1.44.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/brain/help-text.ts b/src/cli/brain/help-text.tsindex d2618bf3..f094b9ed 100644--- a/src/cli/brain/help-text.ts+++ b/src/cli/brain/help-text.ts@@ -111,7 +111,7 @@ Brain verbs (observing memory):signal Fact signal lifecycle: retire <id> --reason <text>telegram-capture Inbound Telegram capture bot: run (long-poll) | catchupinbox-drain Classify and route staged captures (dry-run; --apply to route)- repair-lane Propose memory-graph edges (dry-run; --apply --confirm to write)+ repair-lane Propose memory-graph edges (dry-run; --apply --confirm to write, holdout-gated)session-grep Search imported session recall turns and summariessession-describe Describe an imported session recall DAGsession-expand Expand a session recall node to source turns@@ -248,8 +248,11 @@ export const VERB_HELP: Record<string, string> = {"transitions, retirements, contradictions, and neglected areas when\n" +"expected areas are supplied by future callers.\n",query:- "usage: o2b brain query --preference <id> | --topic <slug> | --since <ISO> [--vault <path>] [--json]\n" +- "Read-only lookup. One of --preference / --topic / --since is required.\n",+ "usage: o2b brain query --preference <id> | --topic <slug> | --since <ISO>\n" ++ " [--at <ISO|YYYY-MM-DD>] [--show-expired] [--vault <path>] [--json]\n" ++ "Read-only lookup. One of --preference / --topic / --since is required.\n" ++ "--at evaluates expiration_date as of that instant (default now) and\n" ++ "--show-expired keeps lapsed memories; both apply to --topic only.\n","agent-query":"usage: o2b brain agent-query [--agent <id>...] [--topic <slug>] [--query <text>]\n" +" [--kind signal|preference|log] [--limit <n>]\n" +@@ -832,7 +835,13 @@ export const VERB_HELP: Record<string, string> = {"writes nothing; --apply writes edges and requires the exact confirmation\n" +"phrase via --confirm. A confidence threshold and a hard per-run write cap\n" +"bound the writes; existing edges are skipped, so a rerun converges to zero\n" +- "writes. Inferred candidates are opt-in behind --include-inferred.\n",+ "writes. Inferred candidates are opt-in behind --include-inferred.\n" ++ "--apply is gated by the paired graph-efficacy holdout harness: every\n" ++ "proposed edge is checked as an (anchor, target) holdout, graph lift is\n" ++ "counted apart from direct recall, and the apply is refused without writing\n" ++ "anything when any target resolves to no durable-memory note or hydrates\n" ++ "into no evidence. Dry-run does not run the gate and is byte-identical; it\n" ++ "already names each unresolvable endpoint as a skip-missing-target decision.\n",agenda:"usage: o2b brain agenda --events <file|-> [args]\n" +"Deterministic agenda synthesis over caller-provided calendar events (JSON array or {events:[...]}).\n" +diff --git a/src/cli/brain/verbs/query.ts b/src/cli/brain/verbs/query.tsindex 4e19671b..0c17d731 100644--- a/src/cli/brain/verbs/query.ts+++ b/src/cli/brain/verbs/query.ts@@ -1,9 +1,30 @@+/**+ * `o2b brain query` - read-only lookup by preference, topic, or log+ * timestamp.+ *+ * Topic mode is the one surface in this system that ENFORCES per-memory+ * expiry (`filterExpired`, reached only through `queryByTopic`), so it is+ * also the only one where a point-in-time question is meaningful. Both+ * knobs the filter has always taken are exposed here:+ *+ * --at <ISO|YYYY-MM-DD> the instant `expiration_date` is compared+ * against, so a memory that lapsed after it is+ * recalled as it stood then;+ * --show-expired keep lapsed memories regardless of the clock.+ *+ * Neither reaches the preference or log-since modes, which run no expiry+ * filter, so supplying one there is refused rather than accepted and+ * ignored.+ */+import {queryByLogSince,queryByPreference,queryByTopic,BrainNotFoundError,+ type QueryByTopicOptions,} from "../../../core/brain/query.ts";+import { parseIsoUtc } from "../../../core/brain/health/iso-time.ts";import {brainVerbContext,fail,@@ -12,14 +33,24 @@ import {renderQueryLogText,renderQueryPreferenceText,renderQueryTopicText,+ usageError,} from "../helpers.ts";+/** The two point-in-time flags, which only the expiry filter reads. */+const AS_OF_FLAG = "--at";+const SHOW_EXPIRED_FLAG = "--show-expired";++/** Accepted `--at` forms, named in every refusal so the exit is actionable. */+const AS_OF_FORMS = "an ISO-8601 instant or YYYY-MM-DD date";+export async function cmdBrainQuery(argv: string[]): Promise<number> {const { flags } = parse(argv, {vault: { type: "string" },preference: { type: "string" },topic: { type: "string" },since: { type: "string" },+ at: { type: "string" },+ "show-expired": { type: "boolean" },json: { type: "boolean" },});const { vault } = brainVerbContext(flags);@@ -34,6 +65,35 @@ export async function cmdBrainQuery(argv: string[]): Promise<number> {`brain query: pick only one of --preference / --topic / --since (got ${modes.join(", ")})`,);+ // `--at` moves the clock the expiration filter compares against and+ // `--show-expired` turns that filter off; `queryByTopic` is the only+ // reader of either. Naming that here beats accepting a flag on+ // `--preference` / `--since` that would quietly change nothing.+ const atRaw = typeof flags["at"] === "string" ? flags["at"].trim() : null;+ const showExpired = flags["show-expired"] === true;+ if (flags["topic"] === undefined) {+ const misplaced = [+ ...(atRaw !== null ? [AS_OF_FLAG] : []),+ ...(showExpired ? [SHOW_EXPIRED_FLAG] : []),+ ];+ if (misplaced.length > 0) {+ return usageError(+ `brain query: ${misplaced.join(" / ")} only applies to --topic (the expiration filter runs there)`,+ );+ }+ }+ let asOf: Date | null = null;+ if (atRaw !== null) {+ // An unparseable instant is refused, never coerced to the wall+ // clock: answering "as of now" for a caller who asked "as of then"+ // answers a different question without saying so.+ const atMs = parseIsoUtc(atRaw);+ if (!Number.isFinite(atMs)) {+ return usageError(`brain query ${AS_OF_FLAG} must be ${AS_OF_FORMS}; got ${atRaw}`);+ }+ asOf = new Date(atMs);+ }+try {if (flags["preference"]) {const out = queryByPreference(vault, String(flags["preference"]));@@ -45,7 +105,11 @@ export async function cmdBrainQuery(argv: string[]): Promise<number> {return 0;}if (flags["topic"]) {- const out = queryByTopic(vault, String(flags["topic"]));+ const topicOptions: QueryByTopicOptions = {+ showExpired,+ ...(asOf !== null ? { now: asOf } : {}),+ };+ const out = queryByTopic(vault, String(flags["topic"]), topicOptions);if (flags["json"]) {process.stdout.write(JSON.stringify(out, null, 2) + "\n");} else {diff --git a/src/cli/brain/verbs/repair-lane.ts b/src/cli/brain/verbs/repair-lane.tsindex e8fbe4a7..f904e2de 100644--- a/src/cli/brain/verbs/repair-lane.ts+++ b/src/cli/brain/verbs/repair-lane.ts@@ -7,6 +7,30 @@* signals only (explicit references, session continuity, same-topic evidence);* inferred candidates are opt-in behind --include-inferred. Each decision is* reported with its identity strength, confidence, and action.+ *+ * ## The holdout gate (what-the-index-already-knew, final unit)+ *+ * The changelog has promised since v1.36.0 that "the paired holdout harness+ * measures graph lift separately from direct recall and fails the gate on any+ * dangling or unhydrated target". The harness (`link-graph/graph-holdout.ts`)+ * was written and tested and reached by nothing, so no operator could meet the+ * gate. This verb is its production caller.+ *+ * It rides on --apply rather than on a flag of its own, because the verb+ * already models the two modes the gate needs: dry-run is the preview and+ * stays byte-identical, --apply is the write path and is what the gate+ * refuses. An apply therefore PLANS first - one dry run, which writes nothing+ * - evaluates the harness over the edges that plan proposes, and only then+ * writes. Fail-closed on the harness's own terms: the refusal is+ * `HoldoutGateResult.passed`, not a second notion of pass derived here.+ *+ * The holdouts are every edge the lane proposes, not only the ones it would+ * write. A candidate the lane skips as `skip-missing-target` still came out of+ * the same structural evidence as the ones it keeps, and a graph whose+ * evidence base points at absent memory is exactly what the harness exists to+ * catch. Restricting the population to accepted writes would also make the+ * dangling half of the gate unreachable by construction, since the lane+ * already existence-checks every endpoint it writes.*/import {@@ -17,9 +41,32 @@ import {type RepairDecision,type RepairReport,} from "../../../core/brain/link-graph/repair-lane.ts";+import {+ evaluateGraphHoldouts,+ type GraphHoldout,+ type HoldoutGateResult,+} from "../../../core/brain/link-graph/graph-holdout.ts";+import { REPAIR_HOLDOUT_UNRESOLVED_CODE } from "../../../core/brain/diagnostics.ts";+import { NEXT_COMMAND_KEY, requireNextStep } from "../../../core/brain/next-step.ts";import { brainVerbContext, ok, okJson, parse } from "../helpers.ts";import { fail } from "../../output.ts";+/**+ * The gate's exit, resolved once at module scope: a registry that stopped+ * carrying this code fails at import rather than inside the refusal it was+ * meant to explain.+ */+const HOLDOUT_GATE_EXIT = requireNextStep(REPAIR_HOLDOUT_UNRESOLVED_CODE);++/** Upper bound on the individual failing edges a refusal names inline. */+const MAX_NAMED_HOLDOUT_FAILURES = 5;++/** Per-edge verdicts the refusal uses to say WHY a target did not qualify. */+const HOLDOUT_FAILURE_LABEL = Object.freeze({+ dangling: "dangling",+ unhydrated: "unhydrated",+} as const);+function decisionJson(decision: RepairDecision): Record<string, unknown> {return {source: decision.source,@@ -31,14 +78,87 @@ function decisionJson(decision: RepairDecision): Record<string, unknown> {};}-function reportJson(report: RepairReport): Record<string, unknown> {+/** Every edge the lane proposes, as an (anchor, target) holdout pair. */+function holdoutsFor(decisions: readonly RepairDecision[]): GraphHoldout[] {+ return decisions.map((decision) => ({ anchor: decision.source, target: decision.target }));+}++function holdoutJson(gate: HoldoutGateResult): Record<string, unknown> {+ return {+ passed: gate.passed,+ total: gate.total,+ resolved: gate.resolvedCount,+ dangling: gate.danglingCount,+ unhydrated: gate.unhydratedCount,+ direct_recall: gate.directRecall,+ graph_lift: gate.graphLift,+ };+}++/** The failing edges, each labelled with the verdict that failed it. */+function failingEdges(gate: HoldoutGateResult): string[] {+ return gate.resolutions+ .filter((resolution) => !resolution.hydrated)+ .map((resolution) => {+ const label = resolution.dangling+ ? HOLDOUT_FAILURE_LABEL.dangling+ : HOLDOUT_FAILURE_LABEL.unhydrated;+ return `${resolution.holdout.anchor} -> ${resolution.holdout.target} (${label})`;+ });+}++function holdoutRefusalMessage(gate: HoldoutGateResult): string {+ const failures = failingEdges(gate);+ const named = failures.slice(0, MAX_NAMED_HOLDOUT_FAILURES);+ const hidden = failures.length - named.length;+ const more = hidden > 0 ? ` and ${hidden} more` : "";+ return (+ `repair-lane holdout gate refused the apply: of ${gate.total} proposed edge(s), ` ++ `${gate.danglingCount} target(s) resolve to no durable memory and ` ++ `${gate.unhydratedCount} hydrate into no evidence ` ++ `(graph lift ${gate.graphLift}, direct recall ${gate.directRecall}). ` ++ `No edge was written. Unresolved: ${named.join("; ")}${more}. ` ++ `Run ${HOLDOUT_GATE_EXIT.nextCommand} to find the absent or empty notes behind them, ` ++ `then re-run the apply.`+ );+}++/**+ * The report payload. `gate` is null on the dry-run path, where the gate is+ * not evaluated at all and the payload must stay byte-identical.+ */+function reportJson(report: RepairReport, gate: HoldoutGateResult | null): Record<string, unknown> {return {mode: report.mode,written: report.written,decisions: report.decisions.map(decisionJson),+ ...(gate === null ? {} : { holdout: holdoutJson(gate) }),};}+function renderReport(+ report: RepairReport,+ gate: HoldoutGateResult | null,+ applied: boolean,+): void {+ ok(+ `repair-lane (${report.mode}): ${report.decisions.length} candidate(s), ${report.written} edge(s) written`,+ );+ for (const decision of report.decisions) {+ ok(+ ` [${decision.strength} ${decision.confidence.toFixed(2)}] ${decision.action}: ${decision.source} -> ${decision.target}`,+ );+ }+ if (gate !== null) {+ ok(+ ` holdout gate: ${gate.total} edge(s) verified, graph lift ${gate.graphLift}, direct recall ${gate.directRecall}`,+ );+ }+ if (!applied && report.written > 0) {+ ok(` re-run with --apply --confirm ${JSON.stringify(REPAIR_CONFIRM_PHRASE)} to write`);+ }+}+export async function cmdBrainRepairLane(argv: string[]): Promise<number> {const { flags } = parse(argv, {vault: { type: "string" },@@ -48,42 +168,52 @@ export async function cmdBrainRepairLane(argv: string[]): Promise<number> {json: { type: "boolean" },});const { vault } = brainVerbContext(flags);+ const asJson = flags["json"] === true;+ const apply = flags["apply"] === true;+ const laneOptions = { includeInferred: flags["include-inferred"] === true };- const candidates = collectRepairCandidates(vault);- let report: RepairReport;- try {- report = runRepairLane(vault, candidates, {- apply: flags["apply"] === true,- ...(typeof flags["confirm"] === "string" ? { confirm: flags["confirm"] } : {}),- includeInferred: flags["include-inferred"] === true,- });- } catch (error) {- if (error instanceof RepairConfirmationError) {- const message = `${error.message} (pass --confirm ${JSON.stringify(REPAIR_CONFIRM_PHRASE)})`;- if (flags["json"] === true) {- okJson({ ok: false, message });- return 1;- }- return fail(message);+ const refuse = (message: string, payload: Record<string, unknown> = {}): number => {+ if (asJson) {+ okJson({ ok: false, message, ...payload });+ return 1;}- throw error;+ return fail(message);+ };++ // The confirmation phrase is a precondition of the invocation, checked+ // before anything is collected or planned so that a refused apply leaves+ // stdout a pure error envelope under --json. `runRepairLane` re-checks it+ // below; this is the same typed error, not a second rule.+ if (apply && flags["confirm"] !== REPAIR_CONFIRM_PHRASE) {+ const message = `${new RepairConfirmationError().message} (pass --confirm ${JSON.stringify(REPAIR_CONFIRM_PHRASE)})`;+ return refuse(message);}- if (flags["json"] === true) {- okJson(reportJson(report));+ const candidates = collectRepairCandidates(vault);++ // Plan first, always as a dry run: the gate must see the proposed edges+ // before any of them reaches disk.+ const plan = runRepairLane(vault, candidates, { apply: false, ...laneOptions });+ if (!apply) {+ if (asJson) okJson(reportJson(plan, null));+ else renderReport(plan, null, false);return 0;}- ok(- `repair-lane (${report.mode}): ${report.decisions.length} candidate(s), ${report.written} edge(s) written`,- );- for (const decision of report.decisions) {- ok(- ` [${decision.strength} ${decision.confidence.toFixed(2)}] ${decision.action}: ${decision.source} -> ${decision.target}`,- );- }- if (!flags["apply"] && report.written > 0) {- ok(` re-run with --apply --confirm ${JSON.stringify(REPAIR_CONFIRM_PHRASE)} to write`);+ const gate = evaluateGraphHoldouts(vault, holdoutsFor(plan.decisions));+ if (!gate.passed) {+ return refuse(holdoutRefusalMessage(gate), {+ holdout: holdoutJson(gate),+ [NEXT_COMMAND_KEY]: HOLDOUT_GATE_EXIT.nextCommand,+ });}++ const report = runRepairLane(vault, candidates, {+ apply: true,+ confirm: REPAIR_CONFIRM_PHRASE,+ ...laneOptions,+ });+ if (asJson) okJson(reportJson(report, gate));+ else renderReport(report, gate, true);return 0;}diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.tsindex b50f385d..58d4f3e0 100644--- a/src/cli/command-manifest.ts+++ b/src/cli/command-manifest.ts@@ -130,7 +130,14 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({command("intent-review", "Review signal clusters before dream"),command("retention", "Review retired preference retention"),command("monthly", "Render month-level Brain synthesis"),- command("query", "Read preferences, topics, and logs"),+ command("query", "Read preferences, topics, and logs", [+ flag("vault", "string"),+ flag("preference", "string"),+ flag("topic", "string"),+ flag("since", "string"),+ flag("at", "string"),+ flag("show-expired", "boolean"),+ ]),command("agent-query", "Read source-agent provenance"),command("agent-diff", "Compare source-agent coverage"),command("reject", "Retire a preference"),@@ -388,7 +395,16 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({command("signal", "Fact signal lifecycle: retire a signal with a reason"),command("telegram-capture", "Inbound Telegram capture bot: long-poll run or catchup"),command("inbox-drain", "Classify and route staged captures (dry-run by default)"),- command("repair-lane", "Propose memory-graph edges (dry-run by default)"),+ command(+ "repair-lane",+ "Propose memory-graph edges (dry-run by default); --apply is holdout-gated",+ [+ flag("vault", "string"),+ flag("apply", "boolean"),+ flag("confirm", "string"),+ flag("include-inferred", "boolean"),+ ],+ ),command("session-grep", "Search imported session recall turns and summaries"),command("session-describe", "Describe an imported session recall DAG"),command("session-expand", "Expand a session recall node to its source turns"),@@ -430,12 +446,48 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({"Search the vault index",[],[- command("query", "Search the vault index"),+ // Declared in the order `cmdSearchQuery` parses them. There is no+ // per-verb `--help` for the search verbs, so `o2b help` is the only+ // place these are discoverable; a partial list here reads as+ // complete, so the whole `parseFlags` schema is modelled and+ // `tests/cli/search-query-flag-manifest.test.ts` holds it to that.+ command("query", "Search the vault index", [+ flag("vault", "string"),+ flag("config", "string"),+ flag("db", "string"),+ flag("limit", "string"),+ flag("semantic", "boolean"),+ flag("keyword-only", "boolean"),+ flag("path", "string"),+ flag("keyword-weight", "string"),+ flag("semantic-weight", "string"),+ flag("auto-refresh", "boolean"),+ flag("property", "string-array"),+ flag("degree", "string-array"),+ flag("visibility", "string-array"),+ flag("agent-scope", "string"),+ flag("query-doc", "string"),+ flag("expand", "boolean"),+ flag("disclosure", "string"),+ flag("profile", "string"),+ flag("evidence-pack", "boolean"),+ flag("include-superseded", "boolean"),+ flag("since", "string"),+ flag("until", "string"),+ flag("global", "boolean"),+ flag("no-record-access", "boolean"),+ flag("verbose", "boolean"),+ flag("explain", "boolean"),+ ]),command("index", "Incrementally update the search index"),command("reindex", "Rebuild the search index"),command("watch", "Watch the vault and incrementally sync the index on .md edits"),command("status", "Print search index status"),- command("check", "Run search pre-flight diagnostics"),+ command(+ "check",+ "Run search pre-flight diagnostics; --integrity adds a full structural scan of the index file",+ [flag("integrity", "boolean")],+ ),command("provider", "Manage embedding provider profiles"),command("expand", "Drill a search card into the fuller note and its raw chunk transcript"),command("focus", "Set, show, or clear the session recall focus bias"),@@ -508,6 +560,12 @@ export function nestedCommandNames(parent: string): string[] {return node?.commands?.map((item) => item.name) ?? [];}+/** One modelled subcommand, e.g. `nestedCommand("search", "query")`. */+export function nestedCommand(parent: string, child: string): CliCommandManifest | undefined {+ const node = CLI_COMMAND_MANIFEST.commands.find((item) => item.name === parent);+ return node?.commands?.find((item) => item.name === child);+}+export function allFlagNames(manifest: CliRootManifest = manifestForJson()): string[] {const names = new Set<string>();for (const flagSpec of manifest.flags) names.add(flagSpec.name);diff --git a/src/cli/search/outcome-render.ts b/src/cli/search/outcome-render.tsindex 0c347b3f..0a9a9ace 100644--- a/src/cli/search/outcome-render.ts+++ b/src/cli/search/outcome-render.ts@@ -6,6 +6,21 @@import { serializeEvidencePack, serializeSearchCard } from "../../core/search/index.ts";import type { SearchOutcome } from "../../core/search/index.ts";+import {+ EXPLAIN_OFF,+ explainEnvelope,+ retrievalTraceUnavailableReason,+ type ExplainRequest,+} from "../../core/search/explain-envelope.ts";+import { formatLinePointer } from "../../core/search/line-numbering.ts";+import type { DuplicatePassageLocation } from "../../core/search/search-result.ts";+import {+ ELLIPSIS,+ alignToCodePointStart,+ markWindow,+ matchOffset,+ windowStartWithin,+} from "../../core/search/snippet-window.ts";/** Field separator in the human transcript's header lines. */const BULLET = " • ";@@ -16,6 +31,67 @@ const NO_RESULTS = "(no results)";/** Longest snippet body shown per hit before it is elided. */const SNIPPET_MAX_CHARS = 140;+/**+ * Merged-away exact-duplicate locations for the `--json` payload, in the+ * snake_case grammar the rest of that payload uses.+ */+function jsonDuplicates(+ duplicates: ReadonlyArray<DuplicatePassageLocation>,+): ReadonlyArray<Record<string, unknown>> {+ return duplicates.map((d) => ({+ path: d.path,+ title: d.title,+ start_line: d.startLine,+ end_line: d.endLine,+ document_id: d.documentId,+ chunk_id: d.chunkId,+ }));+}++/**+ * The snippet body for one hit: whitespace-collapsed content, windowed on+ * the first significant query term it contains (task E). With no+ * occurrence the window is the head of the chunk — the same bytes this+ * line has always carried.+ *+ * The trailing marker is keyed on the COLLAPSED length, which is a+ * deliberate change from what this transcript used to do: keyed on the raw+ * length it claimed a cut whenever whitespace collapse alone had already+ * brought the chunk under the cap, promising more text than existed.+ *+ * A chunk that already fits under the cap is shown whole:+ * {@link windowStartWithin} opens it at the head rather than sliding the+ * window past text there was room to print. The start is then aligned to a+ * code-point boundary, because this surface slices UTF-16 units and an+ * anchored start can otherwise land inside a surrogate pair.+ */+function hitSnippet(content: string, query: string): string {+ const collapsed = content.trim().replace(/\s+/g, " ");+ const start = alignToCodePointStart(+ collapsed,+ windowStartWithin(matchOffset(collapsed, query), SNIPPET_MAX_CHARS, collapsed.length),+ );+ // The trailing marker asks about the text the window is cut from, which+ // is the COLLAPSED string - keying it on the raw length claimed a cut+ // whenever whitespace collapse alone brought the chunk under the cap.+ return markWindow(+ collapsed.slice(start, start + SNIPPET_MAX_CHARS),+ start,+ collapsed.length > start + SNIPPET_MAX_CHARS,+ ELLIPSIS,+ );+}++/**+ * The transcript's merged-away-duplicates line, or no line at all when+ * nothing was folded. One builder for both disclosure modes: the full row+ * holds location records and the card holds ready-made pointers, but the+ * line the reader sees must not differ between them.+ */+function duplicatesLine(pointers: ReadonlyArray<string>): ReadonlyArray<string> {+ return pointers.length > 0 ? [` duplicates: ${pointers.join(", ")}`] : [];+}+/** Cross-vault origin label, appended only when the hit carries one. */function originSuffix(origin: string | undefined): string {return origin !== undefined ? `${BULLET}${origin}` : "";@@ -29,7 +105,7 @@ function joinLines(lines: ReadonlyArray<string>): string {return lines.join("\n") + (lines.length > 0 ? "" : "\n");}-export function jsonForOutcome(o: SearchOutcome): unknown {+export function jsonForOutcome(o: SearchOutcome, options: ExplainRequest = EXPLAIN_OFF): unknown {return {results: o.results.map((r) => ({path: r.path,@@ -52,16 +128,62 @@ export function jsonForOutcome(o: SearchOutcome): unknown {document_id: r.documentId,chunk_id: r.chunkId,...(r.relations && r.relations.length > 0 ? { relations: r.relations } : {}),+ // Exact-duplicate merge (task D): the locations this row was folded+ // from. Absent, never null, when nothing was merged.+ ...(r.duplicates && r.duplicates.length > 0+ ? { duplicates: jsonDuplicates(r.duplicates) }+ : {}),})),warnings: o.warnings,total: o.total,...(o.cards ? { cards: o.cards.map(serializeSearchCard) } : {}),...(o.evidencePack ? { evidence_pack: serializeEvidencePack(o.evidencePack) } : {}),...(o.surface !== undefined ? { surface: o.surface } : {}),+ // Retrieval receipts (task F): the trace and the trust assessment the+ // gate already built, appended only when --explain asks. Absent, never+ // null, otherwise.+ ...explainEnvelope(o, options),};}-export function renderOutcomeHuman(o: SearchOutcome, verbose: boolean): string {+/**+ * The `--explain` block of the human transcript: the decision trace as+ * counts plus one line per excluded candidate, or the single line naming+ * why there is no trace to show. The trust assessment is the same+ * exclusion set folded into a reason histogram, so the transcript renders+ * the trace (the superset) and `--json` carries both receipts verbatim.+ */+function explainLines(o: SearchOutcome, request: ExplainRequest): ReadonlyArray<string> {+ const trace = o.retrievalDecisionTrace;+ const assessment = o.memoryTrustAssessment;+ // The trailing empty entry terminates the block with a newline, the way+ // a result entry does - `joinLines` adds no separator of its own.+ if (trace === undefined || assessment === undefined) {+ const why = retrievalTraceUnavailableReason(request.crossVault, request.trustGateEnabled);+ return [`explain: ${why}`, ""];+ }+ const lines = [+ `explain: evaluated ${trace.evaluated}${BULLET}surfaced ${trace.surfaced}${BULLET}excluded ${trace.excluded}`,+ ];+ for (const exclusion of trace.exclusions) {+ lines.push(` excluded: ${exclusion.path}${BULLET}${exclusion.reasons.join(", ")}`);+ }+ lines.push("");+ return lines;+}++/**+ * `query` is the query text the outcome answers. It anchors each hit's+ * snippet window on the match (task E) and is required, not optional: a+ * transcript that silently fell back to the head of the chunk is the+ * defect this parameter exists to prevent.+ */+export function renderOutcomeHuman(+ o: SearchOutcome,+ verbose: boolean,+ query: string,+ options: ExplainRequest = EXPLAIN_OFF,+): string {// Progressive disclosure layer 1: render compact cards. The agent// drills a hit with `o2b search expand --chunk <chunk_id>`.if (o.cards !== undefined) {@@ -72,9 +194,13 @@ export function renderOutcomeHuman(o: SearchOutcome, verbose: boolean): string {lines.push(` ${c.snippet}`);lines.push(` expand: o2b search expand --chunk ${c.chunkId}`);if (verbose && c.reasons.length > 0) lines.push(` why: ${c.reasons.join(", ")}`);+ // Exact-duplicate merge (task D): the same bytes live at these other+ // locations. The card carries them as pointers already.+ lines.push(...duplicatesLine(c.duplicatePointers ?? []));lines.push("");});for (const w of o.warnings) lines.push(`warning: ${w}`);+ if (options.explain) lines.push(...explainLines(o, options));return joinLines(lines);}const lines: string[] = [];@@ -96,8 +222,7 @@ export function renderOutcomeHuman(o: SearchOutcome, verbose: boolean): string {? `${BULLET}kw=${r.keywordScore.toFixed(2)} sem=${r.semanticScore.toFixed(2)} link=${r.linkBoost.toFixed(2)} rec=${r.recencyBoost.toFixed(2)}`: ""),);- const snippet = r.content.trim().replace(/\s+/g, " ").slice(0, SNIPPET_MAX_CHARS);- lines.push(` ${snippet}${r.content.length > SNIPPET_MAX_CHARS ? "…" : ""}`);+ lines.push(` ${hitSnippet(r.content, query)}`);if (verbose && r.reasons.length > 0) {lines.push(` why: ${r.reasons.join(", ")}`);}@@ -105,8 +230,16 @@ export function renderOutcomeHuman(o: SearchOutcome, verbose: boolean): string {const rel = r.relations.map((x) => `${x.relation} ${x.target}`).join(", ");lines.push(` relations: ${rel}`);}+ // Exact-duplicate merge (task D): the same bytes live at these other+ // locations. Rendered only when the merge actually folded something.+ lines.push(+ ...duplicatesLine(+ (r.duplicates ?? []).map((d) => formatLinePointer(d.path, d.startLine, d.endLine)),+ ),+ );lines.push("");});for (const w of o.warnings) lines.push(`warning: ${w}`);+ if (options.explain) lines.push(...explainLines(o, options));return joinLines(lines);}diff --git a/src/cli/search/verbs/check.ts b/src/cli/search/verbs/check.tsindex 3255311d..9141de96 100644--- a/src/cli/search/verbs/check.ts+++ b/src/cli/search/verbs/check.ts@@ -2,29 +2,311 @@* `o2b search check` — pre-flight diagnostics over everything search needs* (vault, index directory, SQLite/FTS5, the vector extension, the* embedding key, the provider) plus the ABI stamp of the stored vectors.+ *+ * `--integrity` adds the one probe the rest of this report cannot give:+ * a full `PRAGMA quick_check` over the index FILE, run on demand+ * (what-the-index-already-knew, unit K, reader-side half).+ *+ * The store's own integrity gate runs that scan at WRITE open, behind a+ * 24-hour interval. That closes the hole for anyone who indexes; it+ * closes nothing for a pure reader, who never opens the store for+ * writing, never triggers a scan, and therefore never gets the fault key+ * the read open refuses on. Since the index file lives inside the vault+ * and vaults are replicated between machines, that reader is the person+ * most likely to end up with a rotting index and no signal at all.+ *+ * Three properties are deliberate here:+ *+ * - It is OPT-IN. The scan is linear in the size of the index (~22 ms+ * per megabyte, so roughly half a minute on a 1.3 GB one) and+ * `search check` is advertised as a cheap pre-flight by onboarding,+ * by the semantic-search hint and by two registered diagnostic+ * exits. Absent the flag this verb emits exactly the bytes it always+ * did and touches no `index_state` cell.+ * - It REPORTS; it does not repair. The verdict is recorded in the same+ * two `index_state` cells the write-open gate writes - there is one+ * notion of this store's health, not two - and the exit is named for+ * the operator to run, never run for them.+ * - It never reports the ABSENCE of a fault as a pass. "Never checked"+ * and "checked and clean" are separate states of the world and stay+ * separate in both report shapes.+ *+ * One boundary note, stated rather than left to be discovered. `store.ts`+ * is the single SQL boundary for `core/search`, and it re-exports the+ * `index_state` keys the rest of the system reads - but not the two+ * integrity keys, and not `runIntegrityCheck`. This module therefore+ * reaches `store/lifecycle.ts`, `store/state.ts` and `store/sql.ts`+ * directly. Those are the SAME symbols the write-open gate uses, so the+ * two paths cannot disagree about what was scanned or where the verdict+ * lives; what is missing is a `Store` method to reach them through. The+ * clean home for the block below is `store.ts`, as one exported+ * `scanStoreIntegrity(config)`, and moving it there is a pure lift.*/+import { Database } from "bun:sqlite";+import { existsSync, statSync } from "node:fs";++import {+ SEARCH_INDEX_CORRUPT_CODE,+ SEARCH_INDEX_MISSING_CODE,+} from "../../../core/brain/diagnostics.ts";+import { nextCommandField } from "../../../core/brain/next-step.ts";import { formatStampMismatch } from "../../../core/integrity/stamp.ts";import { indexCheck, serializeStampMismatches } from "../../../core/search/index.ts";-import type { IndexCheckReport } from "../../../core/search/index.ts";-import { flagBoolean, parseFlags, resolveConfig, VAULT_FLAGS } from "../helpers.ts";+import type { IndexCheckReport, ResolvedSearchConfig } from "../../../core/search/index.ts";+import { acquireWriterLock } from "../../../core/search/store.ts";+import { runIntegrityCheck } from "../../../core/search/store/lifecycle.ts";+import { nowIso } from "../../../core/search/store/sql.ts";+import {+ deleteState,+ getState,+ INTEGRITY_CHECKED_AT_STATE_KEY,+ INTEGRITY_FAULT_STATE_KEY,+ setState,+} from "../../../core/search/store/state.ts";+import { emitNextStep } from "../../advisory-rail.ts";+import {+ flagBoolean,+ parseFlags,+ resolveConfig,+ searchAdvisoryStream,+ VAULT_FLAGS,+} from "../helpers.ts";-export async function cmdSearchCheck(argv: ReadonlyArray<string>): Promise<number> {- const { flags } = parseFlags(argv, {- ...VAULT_FLAGS,- json: { type: "boolean" },- });- const cfg = resolveConfig(flags);- const report = await indexCheck(cfg);- if (flagBoolean(flags, "json")) {- process.stdout.write(JSON.stringify(jsonForCheck(report)) + "\n");- } else {- process.stdout.write(renderCheckHuman(report));+// ─────────────────────────────────────────────────────────────────────────────+// The on-demand structural scan+// ─────────────────────────────────────────────────────────────────────────────++/**+ * What a completed scan concluded, or that it did not run. `notRun` is a+ * first-class arm rather than a fault: a check that could not run has not+ * found the file healthy and must never be rendered as if it had.+ */+const INTEGRITY_VERDICT = Object.freeze({+ ok: "ok",+ faulty: "faulty",+ notRun: "not-run",+} as const);++type IntegrityVerdictLabel = (typeof INTEGRITY_VERDICT)[keyof typeof INTEGRITY_VERDICT];++/**+ * Scan cost as measured on this project's own indexes: about 22 ms per+ * megabyte of index file (0.85 s on 38 MB, 30 s on 1.3 GB). Used only to+ * tell the operator what the command is about to spend, before it spends+ * it - the estimate never gates anything.+ */+const SCAN_MS_PER_MEGABYTE = 22;++const BYTES_PER_MEGABYTE = 1024 * 1024;+const MS_PER_SECOND = 1000;+/** Decimal places for the sizes and durations this verb prints. */+const REPORT_PRECISION = 1;++/**+ * Where the value column of this report's aligned `label:` lines starts.+ * Used to indent the continuation lines of a multi-line value.+ */+const VALUE_COLUMN = " ".repeat("vault_readable: ".length);++/**+ * The `integrity_checked_at` cell as found BEFORE this run. `at: null`+ * on the `read` arm is the load-bearing state - no full check has ever+ * completed against this file - and it is kept apart from `unreadable`+ * because a store nobody has verified and a store whose metadata is+ * itself damaged are different findings, and reporting the second as+ * "never" would be an invented fact.+ */+type PriorCheck =+ | { readonly kind: "read"; readonly at: string | null }+ | { readonly kind: "unreadable"; readonly detail: string };++/** The result of one on-demand integrity scan, in full. */+interface IntegrityReport {+ /** Whether a full `PRAGMA quick_check` actually completed this run. */+ readonly scanned: boolean;+ readonly verdict: IntegrityVerdictLabel;+ /** What this file said about its own last full check, or `null` when none ran. */+ readonly priorCheck: PriorCheck | null;+ /** The stamp this run recorded, or `null` when no scan ran. */+ readonly checkedAt: string | null;+ /** Wall time the scan itself took, or `null` when no scan ran. */+ readonly elapsedMs: number | null;+ /** Size of the file that was scanned, or `null` when there was none. */+ readonly indexBytes: number | null;+ /** The verdict text, only when the scan condemned the file. */+ readonly fault: string | null;+ /** Why the scan could not run, only on the `not-run` arm. */+ readonly reason: string | null;+ /** Whether the verdict reached `index_state`; false is reported, never hidden. */+ readonly recorded: boolean;+ /** Registered diagnostic code naming this state's exit, or `null` when clean. */+ readonly exitCode: string | null;+}++/** The `not-run` arm: a stated reason and the exit that ends it. */+function scanDidNotRun(reason: string, exitCode: string): IntegrityReport {+ return {+ scanned: false,+ verdict: INTEGRITY_VERDICT.notRun,+ priorCheck: null,+ checkedAt: null,+ elapsedMs: null,+ indexBytes: null,+ fault: null,+ reason,+ recorded: false,+ exitCode,+ };+}++/** `n` bytes as megabytes, for the size this verb quotes to the operator. */+function megabytes(bytes: number): string {+ return (bytes / BYTES_PER_MEGABYTE).toFixed(REPORT_PRECISION);+}++/**+ * `ms` in the unit that carries information at its magnitude. A small+ * vault's scan really does take a few milliseconds, and rounding that to+ * "0.0 s" would read as a scan that did not happen.+ */+function duration(ms: number): string {+ return ms < MS_PER_SECOND+ ? `${Math.round(ms)} ms`+ : `${(ms / MS_PER_SECOND).toFixed(REPORT_PRECISION)} s`;+}++/**+ * The line written BEFORE the scan starts. It goes to stderr in both+ * report shapes: an operator who typed the flag still cannot know that+ * their 1.3 GB index means a thirty-second wait, and stdout under+ * `--json` is a payload a caller parses.+ */+function announceScan(dbPath: string, bytes: number): void {+ const estimate = duration((bytes / BYTES_PER_MEGABYTE) * SCAN_MS_PER_MEGABYTE);+ process.stderr.write(+ `integrity scan: reading all ${megabytes(bytes)} MB of ${dbPath}; ` ++ `expect roughly ${estimate} (the scan is linear in the size of the index)\n`,+ );+}++/**+ * Record a completed scan's verdict in the SAME two cells the write-open+ * gate uses, so this verb adds no second notion of the store's health and+ * the read open starts refusing a condemned file immediately.+ *+ * Returns whether the record landed. A file damaged badly enough may+ * refuse the write; that is reported on stderr and carried in the report+ * rather than swallowed, because a fault nobody could record is a fault+ * the read path will not act on.+ */+function recordVerdict(db: Database, dbPath: string, stamp: string, fault: string | null): boolean {+ try {+ setState(db, INTEGRITY_CHECKED_AT_STATE_KEY, stamp);+ if (fault === null) deleteState(db, INTEGRITY_FAULT_STATE_KEY);+ else setState(db, INTEGRITY_FAULT_STATE_KEY, fault);+ return true;+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ process.stderr.write(`warning: could not record the integrity verdict in ${dbPath}: ${msg}\n`);+ return false;+ }+}++/** What this file records about its own last completed full check. */+function readPriorCheck(db: Database): PriorCheck {+ try {+ return { kind: "read", at: getState(db, INTEGRITY_CHECKED_AT_STATE_KEY) };+ } catch (e) {+ return { kind: "unreadable", detail: e instanceof Error ? e.message : String(e) };+ }+}++/**+ * Run the scan on demand, ignoring the interval gate the write path+ * honours: an operator who asked for a check has asked for THIS check,+ * and a gate that answered "one ran yesterday" would defeat the purpose+ * of the verb.+ *+ * The writer lock is held across the scan so no concurrent index run can+ * mutate the file underneath it, and so the verdict this run records+ * cannot race the one a write open records.+ */+async function scanIntegrity(config: ResolvedSearchConfig): Promise<IntegrityReport> {+ const dbPath = config.dbPath;+ if (!existsSync(dbPath)) {+ return scanDidNotRun(`no search index at ${dbPath}`, SEARCH_INDEX_MISSING_CODE);+ }+ const indexBytes = statSync(dbPath).size;+ announceScan(dbPath, indexBytes);++ const release = await acquireWriterLock(dbPath);+ try {+ let db: Database;+ try {+ db = new Database(dbPath);+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ return scanDidNotRun(`cannot open ${dbPath}: ${msg}`, SEARCH_INDEX_CORRUPT_CODE);+ }+ try {+ // Read the previous stamp through the SAME key the write-open gate+ // writes, so this verb consults the store's own record of its+ // health rather than a second one.+ const priorCheck = readPriorCheck(db);++ const startedAt = performance.now();+ const verdict = runIntegrityCheck(db);+ const elapsedMs = Math.round(performance.now() - startedAt);++ // `quick_check` passes any well-formed sqlite file, including one+ // that is not a search index at all. A PASS therefore needs the+ // metadata read above as its proof, or it would report a foreign+ // database as a healthy index - exactly the misleading pass this+ // unit exists to prevent. A FAILURE needs no such proof: a+ // malformed database at the index path is a fault either way, and+ // is reported as the fault it is.+ if (verdict.ok && priorCheck.kind === "unreadable") {+ return scanDidNotRun(+ `${dbPath} does not read as a search index: ${priorCheck.detail}`,+ SEARCH_INDEX_CORRUPT_CODE,+ );+ }++ const stamp = nowIso();+ const fault = verdict.ok ? null : verdict.fault;+ const recorded = recordVerdict(db, dbPath, stamp, fault);++ return {+ scanned: true,+ verdict: verdict.ok ? INTEGRITY_VERDICT.ok : INTEGRITY_VERDICT.faulty,+ priorCheck,+ checkedAt: stamp,+ elapsedMs,+ indexBytes,+ fault,+ reason: null,+ recorded,+ exitCode: verdict.ok ? null : SEARCH_INDEX_CORRUPT_CODE,+ };+ } finally {+ db.close();+ }+ } finally {+ await release();}- return report.fatal.length > 0 ? 1 : 0;}-function jsonForCheck(r: IndexCheckReport): unknown {+// ─────────────────────────────────────────────────────────────────────────────+// Rendering+// ─────────────────────────────────────────────────────────────────────────────++/** Render one pre-flight probe's verdict, in the wording the check has always used. */+function ok(passed: boolean): string {+ return passed ? "OK" : "MISSING";+}++function jsonForCheck(r: IndexCheckReport): Record<string, unknown> {return {vault_readable: r.vaultReadable,index_dir_writable: r.indexDirWritable,@@ -45,9 +327,28 @@ function jsonForCheck(r: IndexCheckReport): unknown {};}-/** Render one pre-flight probe's verdict, in the wording the check has always used. */-function ok(passed: boolean): string {- return passed ? "OK" : "MISSING";+/** The `integrity` sub-object, present only when the flag asked for one. */+function jsonForIntegrity(r: IntegrityReport): Record<string, unknown> {+ return {+ scanned: r.scanned,+ verdict: r.verdict,+ ...(r.scanned+ ? {+ // The key is present exactly when the cell was READ, so an+ // absent key is never mistaken for a store nobody checked.+ ...(r.priorCheck?.kind === "read"+ ? { previously_checked_at: r.priorCheck.at }+ : { previous_check_unreadable: r.priorCheck?.detail ?? null }),+ checked_at: r.checkedAt,+ elapsed_ms: r.elapsedMs,+ index_bytes: r.indexBytes,+ recorded: r.recorded,+ }+ : {}),+ ...(r.fault !== null ? { fault: r.fault } : {}),+ ...(r.reason !== null ? { reason: r.reason } : {}),+ ...(r.exitCode !== null ? nextCommandField(r.exitCode) : {}),+ };}function renderCheckHuman(r: IndexCheckReport): string {@@ -75,3 +376,76 @@ function renderCheckHuman(r: IndexCheckReport): string {}return lines.join("\n") + "\n";}++/** The three things this file can say about its own last full check. */+function describePriorCheck(prior: PriorCheck | null): string {+ if (prior === null || prior.kind === "unreadable") {+ return `unreadable (${prior?.detail ?? "no scan ran"})`;+ }+ return prior.at ?? "never";+}++/** The human integrity block, appended only when the flag asked for one. */+function renderIntegrityHuman(r: IntegrityReport): string {+ const lines: string[] = [];+ if (!r.scanned) {+ lines.push(`integrity: ${r.verdict} (${r.reason})`);+ return lines.join("\n") + "\n";+ }+ const cost =+ r.elapsedMs !== null && r.indexBytes !== null+ ? ` (scanned ${megabytes(r.indexBytes)} MB in ${duration(r.elapsedMs)})`+ : "";+ lines.push(`integrity: ${r.verdict}${cost}`);+ // "Never" is a state, not a blank: it is what makes a store nobody has+ // verified readable as such instead of as a healthy one. "Unreadable"+ // is a third state and is never folded into it.+ lines.push(`previous_full_check: ${describePriorCheck(r.priorCheck)}`);+ // `quick_check` reports one line per damaged tree, so the verdict is+ // routinely multi-line. Every line is kept - the trees it names are the+ // cause an operator reports - and the continuations are indented to the+ // value column so the block stays readable beside the aligned labels.+ if (r.fault !== null) {+ lines.push(`integrity_fault: ${r.fault.split("\n").join(`\n${VALUE_COLUMN}`)}`);+ }+ if (!r.recorded) {+ lines.push("integrity_recorded: NO (the verdict could not be written to the index)");+ }+ return lines.join("\n") + "\n";+}++// ─────────────────────────────────────────────────────────────────────────────+// Verb+// ─────────────────────────────────────────────────────────────────────────────++export async function cmdSearchCheck(argv: ReadonlyArray<string>): Promise<number> {+ const { flags } = parseFlags(argv, {+ ...VAULT_FLAGS,+ integrity: { type: "boolean" },+ json: { type: "boolean" },+ });+ const cfg = resolveConfig(flags);+ const jsonRequested = flagBoolean(flags, "json");+ const report = await indexCheck(cfg);+ // Absent the flag nothing below runs, nothing is written, and the two+ // report shapes are byte-identical to what they were before it existed.+ const integrity = flagBoolean(flags, "integrity") ? await scanIntegrity(cfg) : null;++ if (jsonRequested) {+ process.stdout.write(+ JSON.stringify({+ ...jsonForCheck(report),+ ...(integrity === null ? {} : { integrity: jsonForIntegrity(integrity) }),+ }) + "\n",+ );+ } else {+ process.stdout.write(renderCheckHuman(report));+ if (integrity !== null) process.stdout.write(renderIntegrityHuman(integrity));+ }+ if (integrity?.exitCode != null) {+ emitNextStep(integrity.exitCode, searchAdvisoryStream(argv, jsonRequested));+ }+ // A requested check that found a fault, or could not run at all, is not+ // a clean pre-flight - neither may exit 0.+ return report.fatal.length > 0 || (integrity !== null && integrity.exitCode !== null) ? 1 : 0;+}diff --git a/src/cli/search/verbs/indexing.ts b/src/cli/search/verbs/indexing.tsindex ce8c88c4..8e0bcf47 100644--- a/src/cli/search/verbs/indexing.ts+++ b/src/cli/search/verbs/indexing.ts@@ -8,10 +8,15 @@import { formatDegradationNotice } from "../../../core/integrity/degradation.ts";import { createSafeguard, resolveSafeguardTimeoutMs } from "../../../core/brain/safeguard.ts";-import { indexVault, reindexVault } from "../../../core/search/index.ts";-import type { IndexStats } from "../../../core/search/index.ts";+import {+ chunkWindowDiagnosticCode,+ indexVault,+ reindexVault,+ serializeChunkWindowCensus,+} from "../../../core/search/index.ts";+import type { ChunkWindowCensus, IndexStats } from "../../../core/search/index.ts";import { nextCommandField } from "../../../core/brain/next-step.ts";-import { emitNextStep } from "../../advisory-rail.ts";+import { emitNextSteps } from "../../advisory-rail.ts";import { CronTemplateError, renderCronTemplate } from "../../search-cron-template.ts";import {flagBoolean,@@ -105,9 +110,15 @@ function reportIndexRun(} else {process.stdout.write(renderStatsHuman(stats, cfg));}- if (complete) {- emitNextStep(exitCode, searchAdvisoryStream(argv, jsonRequested));- }+ // The census verdict is its own state with its own exit, and it is+ // reported whether or not the run completed - an incomplete run still+ // wrote the chunks it wrote. The batch form de-duplicates on the+ // resolved command, so a run with no census emits exactly the one line+ // it emitted before this feature.+ const codes: string[] = [];+ if (stats.chunkWindow !== undefined) codes.push(chunkWindowDiagnosticCode(stats.chunkWindow));+ if (complete) codes.push(exitCode);+ emitNextSteps(codes, searchAdvisoryStream(argv, jsonRequested));}export async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<number> {@@ -178,6 +189,47 @@ export async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<numreturn 0;}+/**+ * The census verdict on THIS surface: the shared wire shape plus the+ * registered exit for its state. The exit belongs here and not in the+ * shared serializer because it is a property of the surface, not of the+ * verdict - the status snapshot carries the same record without one.+ * A run can index every document cleanly and still have produced chunks+ * the model cannot read whole, so this exit is not the run's terminal+ * state either.+ */+function serializeChunkWindow(census: ChunkWindowCensus): Record<string, unknown> {+ return {+ ...serializeChunkWindowCensus(census),+ ...nextCommandField(chunkWindowDiagnosticCode(census)),+ };+}++/**+ * One human line per census verdict. It states the condition and no+ * command: the forward pointer is the advisory rail's, emitted for this+ * run's census code beside the run's terminal state.+ */+function describeChunkWindow(census: ChunkWindowCensus): string {+ if (census.verdict === "window-undeclared") {+ const model = census.model === null ? "(none configured)" : census.model;+ return `not checked - no input window is declared for ${model}`;+ }+ const against = `the ${census.windowTokens}-token window declared for ${census.model}`;+ if (census.verdict === "estimate-undecided") {+ return (+ `${census.chunksUndecided} of ${census.chunksMeasured} chunk(s) undecided against ` ++ `${against} - the character-count estimate does not bound non-Latin text`+ );+ }+ const undecided =+ census.chunksUndecided === undefined ? "" : `, ${census.chunksUndecided} undecided`;+ return (+ `${census.chunksOverWindow} of ${census.chunksMeasured} chunk(s) estimate above ` ++ `${against}${undecided}`+ );+}+function jsonForStats(stats: IndexStats, cfg: ResolvedSearchConfig): unknown {return {stats: {@@ -193,6 +245,14 @@ function jsonForStats(stats: IndexStats, cfg: ResolvedSearchConfig): unknown {...(stats.eventAnchorsPending > 0? { event_anchors_pending: stats.eventAnchorsPending }: {}),+ // Oversize-chunk census. Absent - not zero, not null - whenever it+ // has nothing to report, so an index whose chunks fit the model's+ // declared window emits exactly the payload it emitted before this+ // field existed. `window-undeclared` carries no count on purpose:+ // nothing was counted.+ ...(stats.chunkWindow === undefined+ ? {}+ : { chunk_window: serializeChunkWindow(stats.chunkWindow) }),},errors: stats.errors.map((e) => ({ path: e.path, message: e.message })),// Unit F. Conditional so a vault with no malformed frontmatter emits@@ -232,6 +292,12 @@ function renderStatsHuman(stats: IndexStats, cfg: ResolvedSearchConfig): string` event anchors: ${stats.eventAnchorsPending} document(s) indexed before anchors existed`,);}+ // Same placement rule as the event-anchor line above and for the same+ // reason: these chunks indexed fine, against a model whose declared+ // window they do not fit - or against one that declares no window.+ if (stats.chunkWindow !== undefined) {+ lines.push(` chunk window: ${describeChunkWindow(stats.chunkWindow)}`);+ }if (stats.errors.length > 0) {lines.push(` errors:`);for (const e of stats.errors) lines.push(` - ${e.path}: ${e.message}`);diff --git a/src/cli/search/verbs/query.ts b/src/cli/search/verbs/query.tsindex 49aac748..fb71f434 100644--- a/src/cli/search/verbs/query.ts+++ b/src/cli/search/verbs/query.ts@@ -60,6 +60,7 @@ export async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<numbe"no-record-access": { type: "boolean" },json: { type: "boolean" },verbose: { type: "boolean" },+ explain: { type: "boolean" },});const rawQueryDocument = flagString(flags, "query-doc");@@ -149,11 +150,23 @@ export async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<numbe? await searchAcrossVaults(resolveConfigPath(flags), cfg.vault, searchOpts, cfg): await search(cfg, searchOpts);+ // Retrieval receipts (what-the-index-already-knew, task F): --explain+ // appends the decision trace and the trust assessment the search+ // already built. Omitted, both projections are byte-identical to the+ // pre-flag ones.+ const explainOptions = {+ explain: flagBoolean(flags, "explain"),+ crossVault: isGlobal,+ trustGateEnabled: cfg.recall.retrievalTrustGateEnabled,+ };+if (flagBoolean(flags, "json")) {- process.stdout.write(JSON.stringify(jsonForOutcome(outcome)) + "\n");+ process.stdout.write(JSON.stringify(jsonForOutcome(outcome, explainOptions)) + "\n");return 0;}- process.stdout.write(renderOutcomeHuman(outcome, flagBoolean(flags, "verbose")));+ process.stdout.write(+ renderOutcomeHuman(outcome, flagBoolean(flags, "verbose"), query, explainOptions),+ );return 0;}diff --git a/src/cli/search/verbs/status.ts b/src/cli/search/verbs/status.tsindex 273994b9..6917c7e7 100644--- a/src/cli/search/verbs/status.ts+++ b/src/cli/search/verbs/status.ts@@ -67,6 +67,18 @@ function renderStatusHuman(s: IndexStatusSnapshot): string {lines.push(`embedding_key: ${s.embeddingKeyPresent ? "present" : "missing"}`);lines.push(`last_indexed_at: ${s.lastIndexedAt ?? "(never)"}`);lines.push(`last_full_index_at: ${s.lastFullIndexAt ?? "(never)"}`);+ // The oversize-chunk census when it could NOT run. Above the warnings+ // and not among them, deliberately: this line reports a check that did+ // not happen, and there is no command an operator runs about it. Its+ // absence here means the check ran - and if it found something, that+ // IS a warning below.+ if (s.chunkWindowUndeclared !== undefined) {+ const model = s.chunkWindowUndeclared.model ?? "(none configured)";+ lines.push(+ `chunk_window_check: not run - no input window is declared for ${model} ` ++ `(${s.chunkWindowUndeclared.chunksMeasured} chunk(s) uncompared)`,+ );+ }for (const w of s.warnings) lines.push(`warning: ${w}`);return lines.join("\n") + "\n";}diff --git a/src/core/brain/diagnostics.ts b/src/core/brain/diagnostics.tsindex 1fea48a2..fbfba67d 100644--- a/src/core/brain/diagnostics.ts+++ b/src/core/brain/diagnostics.ts@@ -54,6 +54,10 @@ import { parseFrontmatter, writeFrontmatterAtomic } from "../vault.ts";import { UnclassifiedRepairCodeError, requireMechanicalRepair } from "./applier-capability.ts";import { collectAllBasenames, runDoctor } from "./doctor.ts";+import {+ ENTITY_QUOTE_VARIANT_COLLISION_CODE,+ ENTITY_QUOTE_VARIANT_COLLISION_COMMAND,+} from "./entities/canonical.ts";import { scanDanglingWorkruns, WORKRUN_PHASE } from "./dream-workrun.ts";import { appendLogEvent } from "./log.ts";import { acquireLockSync } from "./sync-lockfile.ts";@@ -80,6 +84,25 @@ export interface DiagnosticSignal {readonly autoRepairable: boolean;}+/**+ * The two states of the search index FILE itself, named here so the+ * producer and the registry entry cannot drift into two spellings of one+ * code. They are separate classes because their exits differ: an absent+ * index is built incrementally, a malformed one is only ever replaced.+ */+export const SEARCH_INDEX_MISSING_CODE = "search-index-missing";+export const SEARCH_INDEX_CORRUPT_CODE = "search-index-corrupt";++/**+ * The state the memory-graph repair lane's holdout gate refuses in: at+ * least one edge the lane proposes has a target that resolves to no+ * durable-memory note, or resolves to one that hydrates into no evidence.+ * Named here beside the code it is registered under so the producer in+ * `cli/brain/verbs/repair-lane.ts` and the entry below cannot drift into+ * two spellings of one code.+ */+export const REPAIR_HOLDOUT_UNRESOLVED_CODE = "repair-holdout-unresolved";+/** Fixer codes (the two auto-repairable classes this release ships). */export const REPAIR_CODE = Object.freeze({walGap: "wal-gap",@@ -279,11 +302,26 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new MapautoRepairable: false,},{- code: "search-index-missing",+ code: SEARCH_INDEX_MISSING_CODE,issueClass: "search index not built",nextCommand: "o2b search index",autoRepairable: false,},+ {+ // The state a completed `PRAGMA quick_check` reports when the+ // index file is structurally malformed - pages damaged in place+ // by a truncated sync, a half-finished replication of the vault+ // directory, or failing storage. It is deliberately NOT+ // `search-index-missing`: that state's exit is `o2b search+ // index`, which is INCREMENTAL and would walk a vault whose+ // files have not changed, decide there is nothing to do, and+ // leave every damaged page exactly where it is. Only a rebuild+ // replaces the file, so only a rebuild ends this state.+ code: SEARCH_INDEX_CORRUPT_CODE,+ issueClass: "search index failed a structural integrity check",+ nextCommand: "o2b search reindex",+ autoRepairable: false,+ },// --- Semantic capability tiers (provenance-at-the-boundary, F1) ---// WHAT THE OPERATOR CONFIGURED, resolved by// `core/search/capability-tier.ts` and reported by the four@@ -355,6 +393,36 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new MapnextCommand: "o2b search event-anchor-backfill --apply",autoRepairable: false,},+ // --- Oversize-chunk census (what-the-index-already-knew, task H) ---+ // The configured chunk size and the configured model's declared+ // input window are two settings that overflow each other in+ // silence: a provider that truncates server-side returns a vector+ // of the right width for a prefix of the passage, and nothing+ // local can see that it happened. The census is what makes the+ // condition observable; these are its two exits.+ {+ code: "search-chunk-window-overflow",+ issueClass: "indexed chunks larger than the embedding model's input window",+ // The chunk boundaries are what must change, and only a rebuild+ // re-cuts them - `--embeddings` because vectors computed over the+ // old boundaries do not describe the new chunks. Lowering+ // `search_chunk_size` first is what makes the rebuild different+ // from the run that produced this state; no verb in this tool+ // writes a config key, so that half is named in the warning prose+ // rather than invented as a command here.+ nextCommand: "o2b search reindex --embeddings",+ autoRepairable: false,+ },+ {+ code: "search-chunk-window-undeclared",+ issueClass: "embedding model with no declared input window",+ // Not a fault and not a pass: the check could not run. The exit+ // is the curated catalog, because a declared window is exactly+ // what distinguishes a model in that table from one outside it,+ // and switching models is a config key no verb writes.+ nextCommand: "o2b search provider presets",+ autoRepairable: false,+ },{code: "git-history-absent",issueClass: "no ingested git history",@@ -391,6 +459,24 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new MapnextCommand: "o2b brain inbox-drain --apply",autoRepairable: false,},+ {+ // what-the-index-already-knew, final unit. The repair lane's+ // holdout gate refuses an apply whose proposed edges point at+ // memory that is absent or empty. The exit is the vault-wide+ // structural scan rather than the lane itself: the lane reports+ // WHICH edges fail (every decision names its action and its+ // endpoints), but what has to change is the referenced note - a+ // reference to a note that was deleted, or a note that exists+ // with nothing in it. The doctor is the surface that finds those+ // across the vault, and `--repair --apply` prunes the subset it+ // can safely repair. Deliberately NOT `broken-wikilink`: these+ // edges are not written yet, so its fixer would find nothing to+ // prune and the pointer would promise a repair that never runs.+ code: REPAIR_HOLDOUT_UNRESOLVED_CODE,+ issueClass: "proposed repair edges point at absent or empty memory",+ nextCommand: "o2b brain doctor",+ autoRepairable: false,+ },// The five states whose exits used to be retyped inside an `ok()`// string beside the rail rather than resolved through it.{@@ -453,6 +539,39 @@ export const DIAGNOSTIC_SIGNALS: ReadonlyMap<string, DiagnosticSignal> = new MapnextCommand: "o2b brain entity prune",autoRepairable: false,},+ {+ // what-the-index-already-knew, task A. The identity kernel now+ // folds typographic quote variants, so two records that differ+ // only in which apostrophe an editor inserted resolve to one+ // identity key. Records that coexisted before the fold therefore+ // collide after it, at a write seam the operator did nothing new+ // to reach - and a refusal met for the first time straight after+ // an upgrade is worse than the duplicate it fixes if it names no+ // way out.+ //+ // The exit is `entity archive`, not a merge verb: there is none,+ // and there should not be one here. Which of the two records+ // keeps the identity, its history and its edges is a reading of+ // both files - the same judgement `doctor-exits.ts` records for+ // `duplicate-entity` - so the command names the one mechanical+ // step that ends the collision and leaves the choice with the+ // operator.+ //+ // PRODUCER: `doctor/entity-checks.ts`, which splits the duplicate+ // report by cause and spells this code on the half the fold+ // created. That is what makes the registration live: the doctor's+ // surfaces resolve a finding's code through the rail, so the+ // command below is what an operator is shown. The registry's own+ // write-time and read-time refusals are thrown synchronously out+ // of core - too early for the rail, and unable to import it+ // without closing a module cycle - so they interpolate+ // `ENTITY_QUOTE_VARIANT_COLLISION_COMMAND`, the same constant this+ // entry is built from.+ code: ENTITY_QUOTE_VARIANT_COLLISION_CODE,+ issueClass: "entity labels differing only in typographic quote form",+ nextCommand: ENTITY_QUOTE_VARIANT_COLLISION_COMMAND,+ autoRepairable: false,+ },{// A subtree the doctor could not enter, reported into the// `uncertain` stream by every sweep that walks the store bydiff --git a/src/core/brain/doctor/entity-checks.ts b/src/core/brain/doctor/entity-checks.tsindex e31fd0cc..d81ab791 100644--- a/src/core/brain/doctor/entity-checks.ts+++ b/src/core/brain/doctor/entity-checks.ts@@ -9,14 +9,69 @@ import {entityLexicalAliasCandidates,resolveEntitySemanticDedupConfig,} from "../entities/semantic-dedup.ts";-import { buildEntityIndex } from "../entities/index-builder.ts";+import {+ anyDiffersOnlyByQuoteVariant,+ ENTITY_QUOTE_VARIANT_CAUSE,+ ENTITY_QUOTE_VARIANT_COLLISION_CODE,+ normalizeEntityName,+} from "../entities/canonical.ts";+import {+ buildEntityIndex,+ conflictClaimants,+ ENTITY_CONFLICT_KIND,+ type EntityIndex,+} from "../entities/index-builder.ts";import { findMalformedEntityLabels } from "../entities/label-hygiene.ts";+import type { BrainEntity, EntityConflict } from "../entities/types.ts";import type { DoctorCheck } from "./check.ts";/**- * `duplicate-entity` / `broken-entity-relation`: duplicates come- * straight from the index builder's conflict report; broken relations- * are edges whose target resolves to no entity id in the registry.+ * The label each claimant contributed to the contested key: its canonical+ * name for a name conflict, and the alias that matched for an alias one.+ *+ * `EntityConflict` carries `{kind, key, paths}` and not the raw labels,+ * which is the right shape for a finding about FILES - and it is also why+ * the cause has to be re-derived here. The alternative, widening the+ * shared record with the claimed forms, would put presentation data on a+ * type the index builder, the registry's write refusal and both doctor+ * surfaces all pass around, to serve one message. The paths are already+ * in the conflict and the index already holds the records, so this costs+ * a lookup instead.+ */+function claimedForms(index: EntityIndex, conflict: EntityConflict): string[] {+ const claimants: ReadonlyArray<BrainEntity> = conflictClaimants(index, conflict);+ if (conflict.kind === ENTITY_CONFLICT_KIND.duplicateName) {+ return claimants.map((c) => c.name);+ }+ return claimants.map(+ (c) => c.aliases.find((a) => normalizeEntityName(a) === conflict.key) ?? c.name,+ );+}++/** `ent-a (path-a), ent-b (path-b)` - the records, not just their files. */+function describeClaimants(index: EntityIndex, conflict: EntityConflict): string {+ return conflictClaimants(index, conflict)+ .map((c) => `${c.id} (${c.path})`)+ .join(", ");+}++/**+ * `duplicate-entity` / `entity-quote-variant-collision` /+ * `broken-entity-relation`: duplicates come straight from the index+ * builder's conflict report; broken relations are edges whose target+ * resolves to no entity id in the registry.+ *+ * The duplicate report is split by CAUSE, because the two halves have+ * different exits and `doctor-exits.ts` may only publish one answer per+ * code. A duplicate that arrived by hand edit or sync merge has no single+ * command - which of the two files keeps the identity is a reading of+ * both - and is written up that way. A duplicate the quote fold created+ * has one: the records were distinct identities under the previous+ * binary, nothing the operator did merged them, and archiving either one+ * ends the collision. That is the code registered in `DIAGNOSTIC_SIGNALS`,+ * and this is what produces it - every doctor surface resolves a finding's+ * code through the advisory rail, so the command reaches the operator from+ * the registry rather than from a sentence spelled here.*/export const entityRegistryCheck: DoctorCheck = {failSoft: true,@@ -25,11 +80,24 @@ export const entityRegistryCheck: DoctorCheck = {if (index.entities.length === 0 && index.conflicts.length === 0) return;for (const conflict of index.conflicts) {+ const subject = conflict.kind === ENTITY_CONFLICT_KIND.duplicateName ? "identity" : "alias";+ if (anyDiffersOnlyByQuoteVariant(claimedForms(index, conflict))) {+ issues.push({+ severity: "warning",+ code: ENTITY_QUOTE_VARIANT_COLLISION_CODE,+ message:+ `${subject} '${conflict.key}' is claimed by ${conflict.paths.length} entity ` ++ `records: ${describeClaimants(index, conflict)}. They were distinct identities ` ++ `before the identity kernel folded typographic quote variants - ` ++ `${ENTITY_QUOTE_VARIANT_CAUSE}. Archive the record you do not keep.`,+ });+ continue;+ }issues.push({severity: "warning",code: "duplicate-entity",message:- `${conflict.kind === "duplicate-name" ? "identity" : "alias"} '${conflict.key}' is ` ++ `${subject} '${conflict.key}' is ` +`claimed by ${conflict.paths.length} entity files: ${conflict.paths.join(", ")}. ` +"Merge them or archive the duplicates - lookups resolve to the first claimant only.",});diff --git a/src/core/brain/entities/canonical.ts b/src/core/brain/entities/canonical.tsindex 95c49409..efa4da61 100644--- a/src/core/brain/entities/canonical.ts+++ b/src/core/brain/entities/canonical.ts@@ -5,12 +5,154 @@* doctor lints, search alias expansion, and the fact-extraction router,* so every consumer compares like with like. Same normalization shape* as `extractEntities` in src/core/search/entities.ts: NFC, lowercase,- * collapsed whitespace.+ * collapsed whitespace - plus the quote-variant fold below.*/-/** NFC-normalise, trim, collapse whitespace runs, lowercase. */+/** A run of whitespace of any kind, collapsed to one space. */+const WHITESPACE_RUN_RE = /\s+/g;++/**+ * The one form every typographic quote variant folds to. U+0027 is the+ * ASCII apostrophe, so the fold never rewrites an ASCII byte: a label+ * that is already ASCII keys exactly as it did before the fold existed.+ */+export const QUOTE_VARIANT_FOLD_TARGET = "'";++/**+ * U+02BC MODIFIER LETTER APOSTROPHE. It is a LETTER by general category+ * (`\p{Lm}`), so it is outside the punctuation classes below and needs+ * its own entry - and it is an apostrophe by function, which is why an+ * editor or a paste can substitute it for U+0027 in a label.+ */+const MODIFIER_LETTER_APOSTROPHE = "ʼ";++/**+ * Every typographic quote variant, DERIVED from Unicode general+ * categories rather than enumerated by hand: initial quote punctuation+ * (`\p{Pi}`), final quote punctuation (`\p{Pf}`), and the modifier-letter+ * apostrophe. The engine resolves the membership, so the rule tracks the+ * Unicode tables it names instead of a list that ages.+ *+ * The exact footprint is asserted exhaustively over the whole code point+ * space in `tests/core/brain/entities/quote-variant-fold.test.ts`, so a+ * widened class cannot land unmeasured.+ */+const QUOTE_VARIANT_RE = new RegExp(`[\\p{Pi}\\p{Pf}${MODIFIER_LETTER_APOSTROPHE}]`, "gu");++/**+ * Fold every typographic quote variant onto {@link QUOTE_VARIANT_FOLD_TARGET}.+ *+ * No Unicode normal form does this: NFC, NFD, NFKC and NFKD all leave+ * U+2019 exactly where it is, so `Taylor's` and `Taylor’s` were two+ * canonical entities in one registry, differing only in which key a text+ * editor happened to insert. Identity is the one place that distinction+ * carries no information, so this is where it is discarded - the same+ * kind of deliberate loss as lowercasing and whitespace collapse.+ */+export function foldQuoteVariants(raw: string): string {+ return raw.replace(QUOTE_VARIANT_RE, QUOTE_VARIANT_FOLD_TARGET);+}++/**+ * The kernel's shape pass, WITHOUT the quote fold: NFC, trim, collapse+ * whitespace runs, lowercase. Separated from {@link normalizeEntityName}+ * so {@link differsOnlyByQuoteVariant} can name the difference the fold+ * makes rather than re-deriving it from a second copy of these steps.+ */+function normalizeEntityShape(raw: string): string {+ return raw.normalize("NFC").trim().replace(WHITESPACE_RUN_RE, " ").toLowerCase();+}++/** NFC-normalise, trim, collapse whitespace runs, lowercase, fold quote variants. */export function normalizeEntityName(raw: string): string {- return raw.normalize("NFC").trim().replace(/\s+/g, " ").toLowerCase();+ return foldQuoteVariants(normalizeEntityShape(raw));+}++/**+ * Registry code the quote-variant collision resolves its forward pointer+ * through, and the structural command that resolves it.+ *+ * Both live in this LEAF module rather than at the consumer, and the+ * `DIAGNOSTIC_SIGNALS` entry in `diagnostics.ts` is built from them, so+ * the string still has exactly one definition. The usual arrangement -+ * the consumer resolving `requireNextStep(code)` at module scope - is not+ * available to the REGISTRY: `next-step.ts` reads `diagnostics.ts`, which+ * reaches the entity registry through the doctor's entity checks and the+ * search query-expansion module, so a registry import of `next-step.ts`+ * closes an import cycle the architecture ratchet refuses+ * (`tests/core/architecture/import-cycles.test.ts`). This module imports+ * nothing, so both consumers can read it.+ *+ * The code's PRODUCER is `doctor/entity-checks.ts`, which spells it on the+ * finding for a collision the fold created; every doctor surface resolves+ * a finding's code through the rail (`nextCommandField` / `emitNextStep`),+ * which is how the other registered doctor codes reach an operator. The+ * registry's own refusals are thrown synchronously out of core and name+ * the command from {@link ENTITY_QUOTE_VARIANT_COLLISION_COMMAND} - the+ * same constant the registration is built from, so the two cannot drift.+ */+export const ENTITY_QUOTE_VARIANT_COLLISION_CODE = "entity-quote-variant-collision";+export const ENTITY_QUOTE_VARIANT_COLLISION_COMMAND = "o2b brain entity archive <name>";++/**+ * The one sentence naming WHY two labels that coexisted before the fold+ * cannot coexist after it. Written once here because three surfaces say+ * it - the registry's write refusal, the registry's ambiguity error, and+ * the doctor's finding - and an operator who meets it twice must read the+ * same words, or the two readings become two conditions.+ */+export const ENTITY_QUOTE_VARIANT_CAUSE =+ "the two differ only in typographic quote form and now resolve to one identity key";++/**+ * True when `a` and `b` are two spellings of one identity that ONLY the+ * quote fold unifies - they were distinct identities before it and are+ * one after it.+ *+ * This is the predicate a collision message asks so it can name the right+ * exit: a refusal an operator meets for the first time immediately after+ * an upgrade needs to say WHY two records that used to coexist no longer+ * can, and every other collision keeps the message it always had.+ */+export function differsOnlyByQuoteVariant(a: string, b: string): boolean {+ const shapeA = normalizeEntityShape(a);+ const shapeB = normalizeEntityShape(b);+ return shapeA !== shapeB && foldQuoteVariants(shapeA) === foldQuoteVariants(shapeB);+}++/**+ * True when ANY two of `labels` are one identity only the fold unifies.+ *+ * The n-ary form because a collision is a set of claimants, not a pair:+ * three files can claim one key, and a message that only ever compared+ * the first two would omit the cause on exactly the messiest vault.+ */+export function anyDiffersOnlyByQuoteVariant(labels: ReadonlyArray<string>): boolean {+ for (let i = 0; i < labels.length; i++) {+ for (let j = i + 1; j < labels.length; j++) {+ if (differsOnlyByQuoteVariant(labels[i]!, labels[j]!)) return true;+ }+ }+ return false;+}++/**+ * The tail a collision message carries when, and only when, the labels+ * were distinct identities before the quote fold and are one after it.+ * Empty for every other collision, so the messages an operator already+ * knows are unchanged and the new sentence means exactly what it says:+ * this refusal is new, and here is why.+ *+ * Lives beside the predicate rather than at the registry because the+ * doctor's finding needs the same words - see+ * {@link ENTITY_QUOTE_VARIANT_CAUSE}.+ */+export function quoteVariantCollisionTail(...labels: ReadonlyArray<string>): string {+ if (!anyDiffersOnlyByQuoteVariant(labels)) return "";+ return (+ ` - ${ENTITY_QUOTE_VARIANT_CAUSE}; ` ++ `run ${ENTITY_QUOTE_VARIANT_COLLISION_COMMAND} on the record you do not keep`+ );}/**@@ -39,9 +181,13 @@ export function entityIdentityKey(category: string, name: string): string {// ----- Label quality gate (A1 / t_657b365e) ---------------------------------//-// `normalizeEntityName` above stays byte-stable: it is the identity kernel-// and its output feeds `entityIdentityKey`, so every currently-valid clean-// label must key identically to before. The label QUALITY pass lives here as+// `normalizeEntityName` above stays byte-stable for every label that+// carries no typographic quote variant: it is the identity kernel and its+// output feeds `entityIdentityKey`, so such a label must key identically to+// before. The quote fold is the ONE deliberate exception, its footprint is+// exactly the Unicode classes `foldQuoteVariants` names, and the corpus+// proof lives in tests/core/brain/entities/quote-variant-fold.test.ts. The+// label QUALITY pass lives here as// a separate step applied BEFORE `normalizeEntityName` at every label-intake// boundary (entity creation, fact-extract anchoring, atomic-facts anchoring):// it strips Markdown decoration and surrounding punctuation, then rejectsdiff --git a/src/core/brain/entities/index-builder.ts b/src/core/brain/entities/index-builder.tsindex 171b2dca..4d64e533 100644--- a/src/core/brain/entities/index-builder.ts+++ b/src/core/brain/entities/index-builder.ts@@ -44,6 +44,54 @@ export interface EntityIndex {readonly conflicts: ReadonlyArray<EntityConflict>;}+/**+ * The two ways one key can be claimed twice, named once. The literals+ * were spelled at the two `claim` sites below, at the doctor's message,+ * and at every consumer that had to ask which kind it was holding.+ */+export const ENTITY_CONFLICT_KIND = Object.freeze({+ duplicateName: "duplicate-name",+ duplicateAlias: "duplicate-alias",+} as const satisfies Record<string, EntityConflictKind>);++/**+ * The conflict of `kind` claiming `key`, or `null` when the key is held+ * once (the ordinary case) or not at all.+ *+ * Published so the WRITE seam and the DOCTOR ask one question rather than+ * two: `upsertEntity` must refuse the identity keys the doctor reports as+ * contested, and a second re-derivation of "contested" here would be free+ * to disagree with this one.+ */+export function findEntityConflict(+ index: EntityIndex,+ kind: EntityConflictKind,+ key: string,+): EntityConflict | null {+ return index.conflicts.find((c) => c.kind === kind && c.key === key) ?? null;+}++/**+ * The records a conflict names, in the walk order it recorded them.+ *+ * `EntityConflict` carries paths and not records - deliberately, so the+ * doctor's finding stays a statement about files - and every consumer+ * that needs the ids or the raw labels behind those paths resolves them+ * here instead of widening the shared record type.+ */+export function conflictClaimants(+ index: EntityIndex,+ conflict: EntityConflict,+): ReadonlyArray<BrainEntity> {+ const byPath = new Map(index.entities.map((e) => [e.path, e]));+ const out: BrainEntity[] = [];+ for (const path of conflict.paths) {+ const entity = byPath.get(path);+ if (entity !== undefined) out.push(entity);+ }+ return out;+}+/** Parse one entity file; `null` when it is not a valid entity. */export function parseEntityFile(path: string): BrainEntity | null {const [meta, body] = parseFrontmatter(path);@@ -199,11 +247,11 @@ function buildEntityIndexUncached(files: string[]): EntityIndex {} catch {continue; // malformed category - doctor reports through conflicts elsewhere}- claim("duplicate-name", key, entity, byKey);+ claim(ENTITY_CONFLICT_KIND.duplicateName, key, entity, byKey);for (const alias of entity.aliases) {const normalized = normalizeEntityName(alias);if (!normalized) continue;- claim("duplicate-alias", normalized, entity, byAlias);+ claim(ENTITY_CONFLICT_KIND.duplicateAlias, normalized, entity, byAlias);}}diff --git a/src/core/brain/entities/registry.ts b/src/core/brain/entities/registry.tsindex e9260ed0..f0492cce 100644--- a/src/core/brain/entities/registry.ts+++ b/src/core/brain/entities/registry.ts@@ -31,10 +31,18 @@ import {assertValidEntityLabel,entityIdentityKey,normalizeEntityName,+ quoteVariantCollisionTail,validateEntityCategory,} from "./canonical.ts";import { resolveEntityLabelDenylist } from "./label-hygiene.ts";-import { buildEntityIndex, parseEntityFile, type EntityIndex } from "./index-builder.ts";+import {+ buildEntityIndex,+ conflictClaimants,+ ENTITY_CONFLICT_KIND,+ findEntityConflict,+ parseEntityFile,+ type EntityIndex,+} from "./index-builder.ts";import { ENTITY_STATUS_SCOPE, entityStatusInScope } from "./status-scope.ts";import {INTAKE_TRUST,@@ -129,12 +137,47 @@ export interface ArchiveEntityOptions {// ----- Lookup -----------------------------------------------------------------/** A category-less reference matched entities in several categories. */+/**+ * The label of `entity` that a normalized `query` actually matched: its+ * canonical name when that is what matched, otherwise the alias that did.+ *+ * A collision message must compare the forms that COLLIDED. Falling back+ * to the canonical name when an alias matched would compare the wrong+ * pair, and the quote-variant cause would then be named on a collision it+ * does not explain, or omitted from one it does.+ */+function matchedLabel(entity: BrainEntity, query: string): string {+ if (normalizeEntityName(entity.name) === query) return entity.name;+ return entity.aliases.find((a) => normalizeEntityName(a) === query) ?? entity.name;+}++/**+ * A reference matched more than one record.+ *+ * Two readings, and the message must not offer the wrong one. Across+ * categories the reference is under-specified and the exit is the+ * category - the message this error has always carried. WITHIN one+ * category it is not under-specified at all: two canonical records claim+ * one identity key, no category can separate them, and telling the+ * operator to pass one would name an exit that cannot work. The quote+ * fold is one way a vault arrives there, and when it is the cause the+ * message says so and names the command that ends it.+ */export class EntityAmbiguityError extends Error {- constructor(query: string, ids: ReadonlyArray<string>) {- super(- `entity reference '${query}' is ambiguous across categories: ${ids.join(", ")} - pass a category`,- );+ constructor(query: string, matches: ReadonlyArray<BrainEntity>) {+ const ids = matches.map((m) => m.id).join(", ");+ const categories = new Set(matches.map((m) => m.category));+ if (categories.size > 1) {+ super(`entity reference '${query}' is ambiguous across categories: ${ids} - pass a category`);+ } else {+ const normalized = normalizeEntityName(query);+ const tail = quoteVariantCollisionTail(...matches.map((m) => matchedLabel(m, normalized)));+ super(+ `entity reference '${query}' is claimed by ${matches.length} canonical records in one ` ++ `category (${[...categories][0]}): ${ids} - no category separates them, so one must be ` ++ `archived or merged${tail}`,+ );+ }this.name = "EntityAmbiguityError";}}@@ -160,12 +203,7 @@ function resolveActive(index: EntityIndex, ref: EntityRef): BrainEntity | null {(normalizeEntityName(e.name) === query ||e.aliases.some((a) => normalizeEntityName(a) === query)),);- if (matches.length > 1) {- throw new EntityAmbiguityError(- ref.query,- matches.map((m) => m.id),- );- }+ if (matches.length > 1) throw new EntityAmbiguityError(ref.query, matches);return matches[0] ?? null;}@@ -186,12 +224,7 @@ function resolveQuarantined(index: EntityIndex, ref: EntityRef): BrainEntity | n(normalizeEntityName(e.name) === query ||e.aliases.some((a) => normalizeEntityName(a) === query)),);- if (matches.length > 1) {- throw new EntityAmbiguityError(- ref.query,- matches.map((m) => m.id),- );- }+ if (matches.length > 1) throw new EntityAmbiguityError(ref.query, matches);return matches[0] ?? null;}@@ -319,14 +352,22 @@ function checkAliasClaims(seen.add(normalized);const aliasHolder = index.byAlias.get(normalized);if (aliasHolder && aliasHolder.id !== selfId) {+ // The holder's own spelling of the claimed form, so the collision+ // message can say whether the two labels differ by anything other+ // than quote shape. Falling back to its canonical name would compare+ // the wrong pair, so the alias that actually matched is found first.+ const heldForm =+ aliasHolder.aliases.find((a) => normalizeEntityName(a) === normalized) ?? aliasHolder.name;throw new Error(- `alias '${alias}' is already claimed by ${aliasHolder.id} (${aliasHolder.path})`,+ `alias '${alias}' is already claimed by ${aliasHolder.id} (${aliasHolder.path})` ++ quoteVariantCollisionTail(alias, heldForm),);}const nameHolder = index.byKey.get(`${category}:${normalized}`);if (nameHolder && nameHolder.id !== selfId) {throw new Error(- `alias '${alias}' collides with the canonical name of ${nameHolder.id} (${nameHolder.path})`,+ `alias '${alias}' collides with the canonical name of ${nameHolder.id} (${nameHolder.path})` ++ quoteVariantCollisionTail(alias, nameHolder.name),);}kept.push(alias);@@ -345,6 +386,42 @@ function allocateEntityId(index: EntityIndex, category: string, name: string): s}}+/**+ * Refuse a write onto an identity key more than one canonical record+ * already claims.+ *+ * The index keeps the walk's FIRST claimant in `byKey` so reads stay+ * deterministic, which is right for a read and wrong for a write: the+ * update arm would rewrite that record's `source_agent`, `updated_at` and+ * body, keep its label rather than the caller's, leave every other+ * claimant untouched, and report `created: false` - which reads as+ * "already there". A caller cannot tell that from a successful update,+ * and the record it did not mean to write was chosen by directory order.+ *+ * Refused for ANY contested key, not only the quote-variant one: the+ * silent wrong-record write does not depend on how the vault arrived at+ * two claimants. When the fold IS how it arrived, the message adds the+ * cause and the registered command, because that is the collision an+ * operator meets for the first time straight after an upgrade.+ *+ * Only the canonical lane is checked. `conflicts` is built at the+ * canonical status scope, so a quarantined namesake is not a claimant of+ * this key; the untrusted lane's own multiple-match case is the ambiguity+ * error in {@link resolveQuarantined}.+ */+function assertIdentityKeyUncontested(index: EntityIndex, key: string, category: string): void {+ const conflict = findEntityConflict(index, ENTITY_CONFLICT_KIND.duplicateName, key);+ if (conflict === null) return;+ const claimants = conflictClaimants(index, conflict);+ const listed = claimants.map((c) => `${c.id} (${c.path})`).join(", ");+ const tail = quoteVariantCollisionTail(...claimants.map((c) => c.name));+ throw new Error(+ `identity '${key}' (${category}) is claimed by ${conflict.paths.length} canonical entity ` ++ `records: ${listed} - a write would land on the first of them and leave the rest, so it ` ++ `is refused. Archive or merge the duplicates first${tail}.`,+ );+}+// ----- Operations ------------------------------------------------------------export function upsertEntity(vault: string, input: UpsertEntityInput): UpsertEntityResult {@@ -365,6 +442,7 @@ export function upsertEntity(vault: string, input: UpsertEntityInput): UpsertEnt// first, alias second, and never across the trusted/untrusted boundary.const untrusted = input.untrustedOrigin === true;const key = entityIdentityKey(category, name);+ if (!untrusted) assertIdentityKeyUncontested(index, key, category);const target = resolveWriteTarget(index, { category, query: name }, untrusted);if (target === null) {@@ -586,7 +664,8 @@ export function archiveEntity(if (holder) {throw new Error(`cannot restore ${target.id}: ${holder.id} already holds '${target.name}' ` +- `(${target.category}). Archive or rename ${holder.id} first, or merge the two.`,+ `(${target.category}). Archive or rename ${holder.id} first, or merge the two` ++ `${quoteVariantCollisionTail(target.name, holder.name)}.`,);}}diff --git a/src/core/brain/recall-inject.ts b/src/core/brain/recall-inject.tsindex febbd9a0..482666b7 100644--- a/src/core/brain/recall-inject.ts+++ b/src/core/brain/recall-inject.ts@@ -55,7 +55,7 @@ export interface RecallCandidate {readonly origin?: RecallSource["alias"];}-/** A retriever's ranked candidate set plus the corpus match total. */+/** A retriever's returned candidates plus the ranked pool they came from. */export interface RecallResultSet {readonly candidates: ReadonlyArray<RecallCandidate>;readonly total: number;diff --git a/src/core/brain/recall-telemetry.ts b/src/core/brain/recall-telemetry.tsindex 09b8813d..367db025 100644--- a/src/core/brain/recall-telemetry.ts+++ b/src/core/brain/recall-telemetry.ts@@ -1,5 +1,8 @@import { appendContinuityRecord, listContinuityRecords } from "./continuity/store.ts";import type { ContinuityRecord } from "./continuity/types.ts";+import { jaccard, tokenise } from "./similarity.ts";+import type { DefaultRelationType } from "../graph/relation-vocab.ts";+import type { BrainSearchResult } from "../search/search-result.ts";export type RecallTelemetryMode = "search" | "context_pack" | "pre_compress" | "query";export type RecallTelemetryStatus = "ok" | "empty" | "error" | "timeout";@@ -10,6 +13,243 @@ export interface RecallTelemetryArtifactInput {readonly score?: number;}+/**+ * Recall-quality signals (what-the-index-already-knew, task G).+ *+ * SHADOW-ONLY INVARIANT, the same one `retrieval-plan.ts` states: this+ * lane READS an already-ranked, already-frozen result window and returns+ * numbers. It exposes no handle that could mutate ranking, weight+ * policy, or the returned rows, and it is invoked only from inside the+ * gated telemetry thunk - with the gate off, `emitGatedTelemetry` never+ * calls the builder, so nothing here executes on the default path.+ *+ * Every signal is a value the retrieval pipeline ALREADY computed and+ * then discarded before any surface could see it:+ *+ * - alignment - the fused `score` and the per-lane `keywordScore` /+ * `semanticScore` the ranker produced;+ * - trust - the read-time {@link BrainSearchResult.trust} metadata,+ * attached only when the caller set `trust`, so `assessed` doubles+ * as the record of whether the caller asked at all;+ * - contradiction - the typed `contradicts` relation edges the+ * pipeline already surfaces on the row, counted with no dependency+ * on the trust opt-in;+ * - diversity - the same deterministic token-set Jaccard measure the+ * MMR reranker applies pair-by-pair, aggregated over the surfaced+ * window. It goes through the shared brain-layer tokeniser+ * ({@link ./similarity.ts}) rather than the reranker's private copy,+ * so this lane adds no second tokeniser; that tokeniser drops+ * single-character tokens, which makes the number comparable across+ * queries but not equal to a given MMR pair score. No embedding+ * round trip, no word list, no clock.+ *+ * ONE record per query. The continuity log has no retention policy, so+ * per-result detail is refused outright: everything here is a pool-level+ * count or mean, and the only quadratic walk is bounded by+ * {@link RECALL_SIGNALS_DIVERSITY_MAX_ROWS}.+ *+ * Epistemic provenance is deliberately NOT derived here. Unlike the four+ * above it is not a value the retrieval path already holds: no+ * frontmatter is carried on the result row and none is stored on the+ * index, so deriving it would add one file read and one frontmatter+ * parse per surfaced document to a live query. The record already names+ * every surfaced artifact by path, so an analysis pass over the log+ * derives the same status against the vault at zero recall-time cost.+ *+ * Two further signals asked for by the original task are refused rather+ * than invented; the reasons are recorded in+ * `docs/brainstorm/what-the-index-already-knew/design.md` under "Out of+ * scope", and a census test asserts their names appear nowhere in `src/`.+ */++/** Decimal places every derived signal is rounded to before recording. */+const SIGNAL_DECIMALS = 4;+/**+ * Upper bound on the rows the pairwise diversity walk compares. The walk+ * is quadratic, so an unbounded `limit` would make a gated query pay+ * `limit^2` token-set comparisons; the top rows are the ones an operator+ * reads, so the walk stops there and `diversity.compared` records the+ * bound that was actually applied.+ */+export const RECALL_SIGNALS_DIVERSITY_MAX_ROWS = 10;+/** The typed edge that marks a surfaced row as declaring a contradiction. */+const CONTRADICTS_RELATION: DefaultRelationType = "contradicts";++export interface RecallAlignmentSignal {+ /** Fused score of the top-ranked surfaced row. */+ readonly top: number;+ /** Mean fused score across the surfaced window. */+ readonly mean: number;+ /**+ * Fused-score gap between rank 1 and rank 2 as returned. Absent - not+ * zero - for a single-row window, where there is no second row to+ * measure against. May be negative when a rerank reordered the window.+ */+ readonly margin?: number;+ /** Summed keyword-lane contribution across the surfaced window. */+ readonly keyword_sum: number;+ /** Summed semantic-lane contribution across the surfaced window. */+ readonly semantic_sum: number;+}++export interface RecallTrustSignal {+ /**+ * Rows carrying read-time trust metadata. Zero means the caller did+ * not opt into `trust`, so the counts below carry no information -+ * they are not a clean bill of health.+ */+ readonly assessed: number;+ readonly superseded: number;+ readonly conflict: number;+ /** Oldest assessed row in whole days. Absent when `assessed` is 0. */+ readonly max_age_days?: number;+}++export interface RecallContradictionSignal {+ /** Surfaced rows declaring at least one `contradicts` edge. */+ readonly rows: number;+ /** Total `contradicts` edges across the surfaced window. */+ readonly edges: number;+}++export interface RecallDiversitySignal {+ /** Rows the pairwise walk actually compared (the applied bound). */+ readonly compared: number;+ /** Pairs compared: `compared * (compared - 1) / 2`. */+ readonly pairs: number;+ /** Mean pairwise token-set overlap. Absent when `pairs` is 0. */+ readonly mean_similarity?: number;+ /** Largest pairwise token-set overlap. Absent when `pairs` is 0. */+ readonly max_similarity?: number;+}++export interface RecallQualitySignals {+ /** Rows in the surfaced window the signals were derived from. */+ readonly rows: number;+ readonly alignment: RecallAlignmentSignal;+ readonly trust: RecallTrustSignal;+ readonly contradiction: RecallContradictionSignal;+ readonly diversity: RecallDiversitySignal;+}++/**+ * Recorded in place of the signals when the surface returned something+ * the derivation cannot read. A check that cannot run says so rather+ * than recording zeros that would read as a measured result.+ */+export interface RecallSignalsUnmeasured {+ readonly unmeasured: "disclosure_cards";+}++/**+ * Progressive-disclosure `cards` mode returns projected cards and an+ * EMPTY `results` array, so none of the per-row values these signals are+ * derived from exist on that path.+ */+export const RECALL_SIGNALS_UNMEASURED_CARDS: RecallSignalsUnmeasured = Object.freeze({+ unmeasured: "disclosure_cards",+});++function roundSignal(value: number): number {+ const factor = 10 ** SIGNAL_DECIMALS;+ return Math.round(value * factor) / factor;+}++/**+ * Derive the per-query recall signals from an already-ranked window.+ *+ * Returns `null` for an empty window: there is no pool to measure, and+ * the record's `result_count: 0` already states that. Pure - no I/O, no+ * clock - and non-mutating: the input array and its rows are read only.+ */+export function deriveRecallSignals(+ rows: ReadonlyArray<BrainSearchResult>,+): RecallQualitySignals | null {+ if (rows.length === 0) return null;++ let scoreSum = 0;+ let keywordSum = 0;+ let semanticSum = 0;+ let assessed = 0;+ let superseded = 0;+ let conflict = 0;+ let maxAgeDays = 0;+ let contradictionRows = 0;+ let contradictionEdges = 0;++ for (const row of rows) {+ scoreSum += row.score;+ keywordSum += row.keywordScore;+ semanticSum += row.semanticScore;+ if (row.trust !== undefined) {+ assessed += 1;+ if (row.trust.superseded) superseded += 1;+ if (row.trust.conflict) conflict += 1;+ if (row.trust.age_days > maxAgeDays) maxAgeDays = row.trust.age_days;+ }+ const edges = (row.relations ?? []).filter(+ (relation) => relation.relation === CONTRADICTS_RELATION,+ ).length;+ if (edges > 0) {+ contradictionRows += 1;+ contradictionEdges += edges;+ }+ }++ const top = rows[0]!;+ const second = rows[1];++ return Object.freeze({+ rows: rows.length,+ alignment: Object.freeze({+ top: roundSignal(top.score),+ mean: roundSignal(scoreSum / rows.length),+ ...(second !== undefined ? { margin: roundSignal(top.score - second.score) } : {}),+ keyword_sum: roundSignal(keywordSum),+ semantic_sum: roundSignal(semanticSum),+ }),+ trust: Object.freeze({+ assessed,+ superseded,+ conflict,+ ...(assessed > 0 ? { max_age_days: maxAgeDays } : {}),+ }),+ contradiction: Object.freeze({ rows: contradictionRows, edges: contradictionEdges }),+ diversity: derivePoolDiversity(rows),+ });+}++/**+ * Pool-level redundancy: the mean and maximum pairwise token-set Jaccard+ * over the top {@link RECALL_SIGNALS_DIVERSITY_MAX_ROWS} rows, using the+ * shared brain-layer tokeniser. A window with fewer than two rows has no+ * pair, so both means are absent rather than reported as a diversity of+ * zero, which would read as "maximally diverse".+ */+function derivePoolDiversity(rows: ReadonlyArray<BrainSearchResult>): RecallDiversitySignal {+ const compared = Math.min(rows.length, RECALL_SIGNALS_DIVERSITY_MAX_ROWS);+ if (compared < 2) return Object.freeze({ compared, pairs: 0 });++ const tokens = rows.slice(0, compared).map((row) => tokenise(row.content));+ let sum = 0;+ let max = 0;+ let pairs = 0;+ for (let i = 0; i < tokens.length; i++) {+ for (let j = i + 1; j < tokens.length; j++) {+ const similarity = jaccard(tokens[i]!, tokens[j]!);+ sum += similarity;+ if (similarity > max) max = similarity;+ pairs += 1;+ }+ }+ return Object.freeze({+ compared,+ pairs,+ mean_similarity: roundSignal(sum / pairs),+ max_similarity: roundSignal(max),+ });+}+export interface RecallTelemetryInput {readonly createdAt?: string;readonly host: string;@@ -22,6 +262,13 @@ export interface RecallTelemetryInput {readonly topArtifacts?: ReadonlyArray<RecallTelemetryArtifactInput>;readonly gaps?: ReadonlyArray<string>;readonly metadata?: Readonly<Record<string, unknown>>;+ /**+ * Recall-quality signals for this query (task G). Additive and+ * optional: a caller that supplies none produces a payload+ * byte-identical to the pre-signal one - the key is absent, never+ * null - so `CONTINUITY_SCHEMA_VERSION` does not move.+ */+ readonly signals?: RecallQualitySignals | RecallSignalsUnmeasured;}export interface RecallTelemetryOptions {@@ -76,6 +323,7 @@ export function emitRecallTelemetry(vault: string, input: RecallTelemetryInput):})),gaps,...(input.metadata ? { metadata: input.metadata } : {}),+ ...(input.signals ? { signals: input.signals } : {}),},});}diff --git a/src/core/brain/session-recall.ts b/src/core/brain/session-recall.tsindex ff610a10..cc388a7f 100644--- a/src/core/brain/session-recall.ts+++ b/src/core/brain/session-recall.ts@@ -1,6 +1,7 @@import { createHash } from "node:crypto";import { charSpanToLineSpan } from "../search/line-numbering.ts";+import { caseInsensitiveIndex, NO_MATCH_OFFSET, snippetAround } from "../search/snippet-window.ts";import {appendContinuityRecord,clipPayloadToBudget,@@ -499,24 +500,6 @@ function lineageScope(return scope;}-/**- * Case-insensitive search that returns an offset into the ORIGINAL `text`,- * so `snippet()` and `charSpanToLineSpan()` (both of which slice the- * original text) stay aligned. `text.toLowerCase()` is not always- * length-preserving (e.g. U+0130 lowercases to two code units), which would- * otherwise shift the offset taken from the lowercased copy. The fast path- * keeps normal text O(n) and bit-identical; the scan runs only when- * lowercasing changed the length.- */-function caseInsensitiveIndex(text: string, needleLower: string): number {- const lower = text.toLowerCase();- if (lower.length === text.length) return lower.indexOf(needleLower);- for (let i = 0; i < text.length; i++) {- if (text.slice(i).toLowerCase().startsWith(needleLower)) return i;- }- return -1;-}-function hitFor(record: ContinuityRecord,needle: string,@@ -525,14 +508,14 @@ function hitFor(const text = recordText(record);const haystack = text.toLowerCase();const index = caseInsensitiveIndex(text, needle);- if (index < 0) return null;+ if (index === NO_MATCH_OFFSET) return null;const score = (record.kind === "session_turn" ? 2 : 1) + occurrenceCount(haystack, needle);const span = charSpanToLineSpan(text, index, needle.length);const base = {id: record.id,kind: record.kind as "session_turn" | "session_summary_node",score,- snippet: snippet(text, index, snippetChars),+ snippet: snippetAround(text, index, snippetChars),line_start: span.lineStart,line_end: span.lineEnd,};@@ -626,12 +609,6 @@ function findByDedupeKey(vault: string, dedupeKey: string): ContinuityRecord | n);}-function snippet(text: string, index: number, maxChars: number): string {- const half = Math.floor(maxChars / 2);- const start = Math.max(0, index - half);- return text.slice(start, start + maxChars);-}-function oneLine(text: string, maxChars: number): string {return text.replace(/\s+/g, " ").trim().slice(0, maxChars);}diff --git a/src/core/search/cards.ts b/src/core/search/cards.tsindex 0c21ca36..8d4afb32 100644--- a/src/core/search/cards.ts+++ b/src/core/search/cards.ts@@ -7,6 +7,7 @@import { normalizeAgentScope } from "../graph/agent-scope.ts";import { formatLinePointer } from "./line-numbering.ts";import { isPathOwnerVisible } from "./result-filters.ts";+import { markWindow, matchOffset, toCodePointOffset, windowStartWithin } from "./snippet-window.ts";import { Store } from "./store.ts";import { SearchError } from "./types.ts";import type {@@ -19,16 +20,45 @@ import type {/** Max chars of a layer-1 card snippet — enough to judge a hit, cheap to carry. */const CARD_SNIPPET_CHARS = 240;+/**+ * Continuation marker for a card snippet that does not cover the whole+ * chunk. Three ASCII dots, not U+2026: the card grammar has shipped this+ * spelling since progressive disclosure landed, and the other preview+ * surfaces use the single-character form.+ */+const CARD_ELLIPSIS = "...";/** Default layer-3 raw-chunk page size for `expandHit`. */const DEFAULT_EXPAND_RAW_LIMIT = 10;+/**+ * The merged-away duplicate locations, as `path:Lstart-Lend` pointers.+ *+ * A card carries POINTERS rather than the full location records the+ * ranked row holds: the folded passage is byte-identical to the surviving+ * one by construction, so a caller never needs to expand a copy - it needs+ * to know where else the same bytes live, which is exactly what the card's+ * own `pointer` grammar says. That keeps the token-cheap layer cheap and+ * reuses the rendering the full transcript already uses.+ */+function duplicatePointers(result: BrainSearchResult): ReadonlyArray<string> {+ return Object.freeze(+ (result.duplicates ?? []).map((d) => formatLinePointer(d.path, d.startLine, d.endLine)),+ );+}+/*** Project a ranked result into a layer-1 card (progressive disclosure):* identity + score + the same `reasons`, a whitespace-collapsed snippet- * capped at {@link CARD_SNIPPET_CHARS}, and a `path:Lstart-Lend` pointer- * (D2 grammar) over the chunk's stored line span. No full content.+ * capped at {@link CARD_SNIPPET_CHARS} and anchored on the query match,+ * and a `path:Lstart-Lend` pointer (D2 grammar) over the chunk's stored+ * line span. No full content.+ *+ * `query` is the residual query text the row was retrieved for; it+ * decides where the snippet window opens. It is required rather than+ * optional because a card whose snippet silently reverted to the head of+ * the chunk is the defect this signature exists to prevent.*/-export function toSearchCard(result: BrainSearchResult): SearchCard {+export function toSearchCard(result: BrainSearchResult, query: string): SearchCard {return Object.freeze({chunkId: result.chunkId,documentId: result.documentId,@@ -36,22 +66,53 @@ export function toSearchCard(result: BrainSearchResult): SearchCard {title: result.title,score: result.score,reasons: result.reasons,- snippet: cardSnippet(result.content),+ snippet: cardSnippet(result.content, query),pointer: formatLinePointer(result.path, result.startLine, result.endLine),...(result.origin !== undefined ? { origin: result.origin } : {}),+ // Exact-duplicate merge (task D): the locations this row was folded+ // from, so cards mode reports them instead of silently dropping a+ // value the pipeline already computed. Absent, never null and never+ // empty, when nothing was merged.+ ...(result.duplicates !== undefined && result.duplicates.length > 0+ ? { duplicatePointers: duplicatePointers(result) }+ : {}),});}-export function cardSnippet(content: string): string {+/**+ * The card snippet: whitespace-collapsed content, windowed on the first+ * significant `query` term it contains, capped at+ * {@link CARD_SNIPPET_CHARS} code points.+ *+ * `query` defaults to the empty string, which has no significant terms+ * and is therefore the defined no-match case — the head of the chunk,+ * byte-identical to the pre-anchoring snippet. That default exists for+ * callers that hold content without a query at all; the ranked path+ * always passes one through {@link toSearchCard}.+ *+ * The cap bounds the BODY; each {@link CARD_ELLIPSIS} sits outside it, so+ * a window cut at both ends reaches `CARD_SNIPPET_CHARS + 2 *+ * CARD_ELLIPSIS.length` code points. Shrinking the body to hold a fixed+ * total would make an anchored card show LESS evidence than a+ * head-anchored one, which is the opposite of what anchoring is for; the+ * markers are continuation decoration, not content.+ */+export function cardSnippet(content: string, query = ""): string {const collapsed = content.replace(/\s+/g, " ").trim();- // Truncate on code points, not UTF-16 units: a raw `.slice` can cut an- // astral character (emoji, rare CJK) mid-surrogate-pair, shipping a lone- // surrogate that renders as U+FFFD. Spreading into an array iterates by- // code point, so the cap never splits a character.+ // Window and truncate on code points, not UTF-16 units: a raw `.slice`+ // can cut an astral character (emoji, rare CJK) mid-surrogate-pair,+ // shipping a lone surrogate that renders as U+FFFD. Spreading into an+ // array iterates by code point, so neither edge splits a character —+ // and an index into that array is always a code-point boundary, which+ // is why this surface needs no start alignment.const points = [...collapsed];- return points.length <= CARD_SNIPPET_CHARS- ? collapsed- : `${points.slice(0, CARD_SNIPPET_CHARS).join("")}...`;+ const start = windowStartWithin(+ toCodePointOffset(collapsed, matchOffset(collapsed, query)),+ CARD_SNIPPET_CHARS,+ points.length,+ );+ const end = start + CARD_SNIPPET_CHARS;+ return markWindow(points.slice(start, end).join(""), start, end < points.length, CARD_ELLIPSIS);}/**diff --git a/src/core/search/embeddings/presets.ts b/src/core/search/embeddings/presets.tsindex 391466a9..61a5a03c 100644--- a/src/core/search/embeddings/presets.ts+++ b/src/core/search/embeddings/presets.ts@@ -15,6 +15,8 @@* width, useful when setting `embedding_dimension` up front.*/+import type { EmbeddingProviderName } from "../types.ts";+/*** Instruction prefix an e5-family model expects before a search query* (memory-write-path-integrity B2). Trailing space is intentional: the model@@ -32,6 +34,23 @@ export interface EmbeddingModelPreset {readonly label: string;/** Native embedding dimension. */readonly dimension: number;+ /**+ * The model's DECLARED maximum input length, in the model's own+ * tokenizer's tokens, taken from its published specification (the+ * `max_seq_length` a sentence-transformers checkpoint ships in+ * `sentence_bert_config.json`, or the context length the model card+ * states). Input past it is truncated by the serving stack, and no+ * OpenAI-compatible endpoint reports that it happened - the vector+ * comes back the right width, silently describing a prefix of the+ * passage.+ *+ * Declared here rather than probed: no provider in this system exposes+ * model metadata and the {@link EmbeddingProvider} contract has no+ * window field, so the only alternatives are a network round trip or+ * nothing. A model outside this table therefore has NO declared window,+ * and that is reported as a check that did not run - never as a pass.+ */+ readonly inputWindowTokens: number;/** True when the model is trained for cross-lingual retrieval. */readonly multilingual: boolean;/** One-line guidance shown alongside the model. */@@ -58,6 +77,9 @@ export const EMBEDDING_MODEL_PRESETS: ReadonlyArray<EmbeddingModelPreset> = Objemodel: "intfloat/multilingual-e5-small",label: "multilingual-e5-small",dimension: 384,+ // sentence_bert_config.json: max_seq_length 512; the model card states+ // inputs longer than 512 tokens are truncated.+ inputWindowTokens: 512,multilingual: true,note: "Small, fast, strong multilingual default. Prefix inputs with 'query:'/'passage:'.",queryPrefix: E5_QUERY_PREFIX,@@ -67,6 +89,8 @@ export const EMBEDDING_MODEL_PRESETS: ReadonlyArray<EmbeddingModelPreset> = Objemodel: "BAAI/bge-m3",label: "bge-m3",dimension: 1024,+ // Model card: multi-granularity, input lengths up to 8192 tokens.+ inputWindowTokens: 8192,multilingual: true,note: "High-quality multilingual, 100+ languages. Larger vectors, higher cost.",},@@ -74,6 +98,10 @@ export const EMBEDDING_MODEL_PRESETS: ReadonlyArray<EmbeddingModelPreset> = Objemodel: "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",label: "paraphrase-multilingual-MiniLM-L12-v2",dimension: 384,+ // sentence_bert_config.json: max_seq_length 128. The narrowest window+ // in this table by a wide margin - the underlying encoder accepts+ // more, the shipped sentence-transformers configuration does not.+ inputWindowTokens: 128,multilingual: true,note: "Compact multilingual paraphrase model; good latency/quality balance.",},@@ -81,6 +109,8 @@ export const EMBEDDING_MODEL_PRESETS: ReadonlyArray<EmbeddingModelPreset> = Objemodel: "Alibaba-NLP/gte-multilingual-base",label: "gte-multilingual-base",dimension: 768,+ // Model card: long-context encoder, up to 8192 tokens.+ inputWindowTokens: 8192,multilingual: true,note: "Balanced multilingual retrieval model with long context.",},@@ -88,6 +118,9 @@ export const EMBEDDING_MODEL_PRESETS: ReadonlyArray<EmbeddingModelPreset> = Objemodel: "sentence-transformers/LaBSE",label: "LaBSE",dimension: 768,+ // sentence_bert_config.json: max_seq_length 256 (the sentence-transformers+ // port caps below the 512 the underlying BERT encoder allows).+ inputWindowTokens: 256,multilingual: true,note: "109-language sentence embeddings; strong cross-lingual alignment.",},@@ -95,6 +128,10 @@ export const EMBEDDING_MODEL_PRESETS: ReadonlyArray<EmbeddingModelPreset> = Objemodel: "BAAI/bge-small-zh-v1.5",label: "bge-small-zh-v1.5",dimension: 512,+ // sentence_bert_config.json: max_seq_length 512, matching the BERT+ // encoder's max_position_embeddings. Not the same number as the+ // embedding width above, which is 512 by coincidence.+ inputWindowTokens: 512,multilingual: false,note: "Chinese-optimized small model; pick when the vault is predominantly zh.",},@@ -108,6 +145,31 @@ export function findEmbeddingPreset(model: string): EmbeddingModelPreset | nullreturn EMBEDDING_MODEL_PRESETS.find((p) => p.model === model) ?? null;}+/**+ * The input window declared for `model`, or `null` when this table does+ * not declare one.+ *+ * `null` means UNKNOWN, and every caller must treat it as such. It is+ * deliberately NOT the shape {@link pricePerMillionTokens} uses, which+ * answers 0 for an unlisted model so the cost gate can never falsely+ * block: an unknown price is safe to treat as free because the+ * consequence of the fallback is that a gate declines to fire. An unknown+ * window has the opposite polarity - treating it as "fits" would report a+ * passing check for a condition nobody measured, which is the misleading+ * silence this census exists to remove.+ *+ * Structural lookup by exact model string only. There is no family+ * heuristic here on purpose: `e5` prefixes are a property of the+ * INSTRUCTION FORMAT and are safe to infer, while a window is a number+ * that differs between checkpoints of one family (512 for+ * multilingual-e5-small, 128 for a MiniLM sentence-transformers port),+ * so inferring one from a name would invent the value.+ */+export function declaredInputWindowTokens(model: string | null): number | null {+ if (model === null) return null;+ return findEmbeddingPreset(model)?.inputWindowTokens ?? null;+}+/*** Structural e5-family detection (memory-write-path-integrity B2). Matches the* `e5` token wherever it appears delimited by `/` or `-` in the model id@@ -120,6 +182,39 @@ export function isE5FamilyModel(model: string | null): boolean {return /(^|[/-])e5([/-]|$)/i.test(model);}+/**+ * The passage prefix the CONFIGURED BACKEND will actually prepend to an+ * indexed chunk before the provider counts a token.+ *+ * Not the same question as {@link resolveEmbeddingPrefixes}, which+ * answers what the configuration asks for. Only `openai-compat`+ * implements the instruction-prefix contract (`prefixFor` in+ * `openai-compat.ts`); ZeroEntropy's `embed` takes no kind and sends the+ * raw text, the local embedder hashes n-grams over it, and the null+ * provider sends nothing at all. A census that subtracted a configured+ * prefix from a backend that never sends one would report an overflow+ * nobody can hit.+ *+ * The switch is exhaustive over {@link EmbeddingProviderName} with no+ * default arm, so adding a backend fails to compile here rather than+ * silently inheriting "sends no prefix".+ */+export function passagePrefixSentByProvider(+ provider: EmbeddingProviderName,+ configuredPassagePrefix: string | undefined,+): string {+ switch (provider) {+ case "openai-compat":+ // Exactly `OpenAICompatProvider.prefixFor("passage")`: an absent+ // configured prefix is no prefix, an empty one is disabled.+ return configuredPassagePrefix ?? "";+ case "zeroentropy":+ case "local":+ case "disabled":+ return "";+ }+}+/** The prefix pair active for an embed run, after preset + config resolution. */export interface ResolvedEmbeddingPrefixes {readonly queryPrefix: string;diff --git a/src/core/search/embeddings/signature.ts b/src/core/search/embeddings/signature.tsindex 2b276c46..a8891458 100644--- a/src/core/search/embeddings/signature.ts+++ b/src/core/search/embeddings/signature.ts@@ -62,21 +62,160 @@ export function pricePerMillionTokens(model: string | null): number {return typeof rate === "number" && Number.isFinite(rate) && rate > 0 ? rate : 0;}+/**+ * Characters per estimated token. Lifted out of {@link estimateTokens}+ * because the ratio now has a second reader: the store-side oversize+ * census selects the same population with a length comparison instead of+ * materialising chunk content, and a second literal `4` in that predicate+ * would be a magic number the two sites could drift on.+ */+export const CHARS_PER_ESTIMATED_TOKEN = 4;+/*** Cheap token estimate: a chars/4 heuristic per text, rounded up, summed.* This is the same order-of-magnitude approximation embedding providers* use for quick quotes; it intentionally avoids a real tokenizer so the* estimate stays dependency-free and deterministic.+ *+ * A "character" here is a UTF-16 code unit (JavaScript `String.length`),+ * which matters to any reader that reproduces this arithmetic outside+ * JavaScript - see {@link charLengthOverTokenBudget}.*/export function estimateTokens(texts: ReadonlyArray<string>): number {let total = 0;for (const t of texts) {if (t.length === 0) continue;- total += Math.ceil(t.length / 4);+ total += Math.ceil(t.length / CHARS_PER_ESTIMATED_TOKEN);}return total;}+/**+ * The character length a single text must EXCEED for its+ * {@link estimateTokens} value to exceed `tokens`.+ *+ * `ceil(L / 4) > W` holds exactly when `L > 4W` for an integer window+ * `W`, so a caller that only needs the predicate - "does this text's+ * estimate exceed the budget" - can evaluate it against a length instead+ * of computing the estimate. That is what lets the oversize-chunk census+ * run as one SQL aggregate over `chunks.content` rather than reading+ * every chunk body into memory.+ */+export function charLengthOverTokenBudget(tokens: number): number {+ return tokens * CHARS_PER_ESTIMATED_TOKEN;+}++/**+ * The most tokens the census charges a single NON-ASCII code point.+ *+ * Characters-over-four is a Latin-script rule of thumb and nothing more.+ * A BERT-family tokenizer is close to character-level on Han text, so a+ * Chinese passage costs roughly one token per character - four times what+ * {@link estimateTokens} predicts. One token per non-ASCII code point is+ * the widest per-character cost this census models, and it is what makes+ * the ceiling below an upper bound for the scripts these presets target+ * rather than another Latin-only guess.+ */+export const MAX_TOKENS_PER_NON_ASCII_CODE_POINT = 1;++/**+ * The two quantities SQLite can measure over a TEXT column without+ * materialising it: `length()` (Unicode CODE POINTS) and+ * `octet_length()` (UTF-8 BYTES).+ *+ * They are the census's whole input, and between them they carry a+ * structural script signal: a code point outside ASCII costs at least one+ * extra UTF-8 byte, so `utf8Bytes - codePoints` is an exact upper bound on+ * how many of a text's code points are non-ASCII. That is a property of+ * the UTF-8 encoding, not a language list, and it holds for every script.+ */+export interface TextExtent {+ readonly codePoints: number;+ readonly utf8Bytes: number;+}++const UTF8 = new TextEncoder();++/** Measure a JavaScript string the way SQLite measures a TEXT value. */+export function textExtent(text: string): TextExtent {+ return Object.freeze({+ codePoints: [...text].length,+ utf8Bytes: UTF8.encode(text).length,+ });+}++/** Non-ASCII code points in a text, at most - see {@link TextExtent}. */+function maxNonAsciiCodePoints(extent: TextExtent): number {+ return Math.min(extent.codePoints, extent.utf8Bytes - extent.codePoints);+}++/**+ * Fewest tokens the text can plausibly cost: every code point at a+ * quarter token, which is {@link estimateTokens} generalised from UTF-16+ * code units to code points.+ *+ * Code points rather than code units because that is the unit SQLite+ * counts, and because it is the SMALLER of the two - a supplementary-plane+ * character is one code point and two code units - so the floor stays a+ * floor on both sides of the seam.+ */+export function tokenEstimateFloor(extent: TextExtent): number {+ return Math.ceil(extent.codePoints / CHARS_PER_ESTIMATED_TOKEN);+}++/**+ * Most tokens the text can plausibly cost: every non-ASCII code point at+ * {@link MAX_TOKENS_PER_NON_ASCII_CODE_POINT}, the rest at a quarter+ * token each.+ *+ * Equal to {@link tokenEstimateFloor} for pure ASCII, which is what keeps+ * an all-Latin index reporting exactly what it reported before this+ * two-sided bound existed.+ */+export function tokenEstimateCeiling(extent: TextExtent): number {+ const nonAscii = maxNonAsciiCodePoints(extent);+ const rest = extent.codePoints - nonAscii;+ return (+ nonAscii * MAX_TOKENS_PER_NON_ASCII_CODE_POINT + Math.ceil(rest / CHARS_PER_ESTIMATED_TOKEN)+ );+}++/**+ * Coefficient on the non-ASCII code-point count in the integer form of+ * `tokenEstimateCeiling(e) > W`.+ *+ * `n * M + ceil((C - n) / 4) > W` is equivalent, over integers, to+ * `C + (4M - 1) * n > 4W`. Exported because the census evaluates that+ * comparison in SQL, where the division cannot be rounded, and a second+ * literal there would be the drift this constant exists to prevent.+ */+export const NON_ASCII_CEILING_COEFFICIENT =+ CHARS_PER_ESTIMATED_TOKEN * MAX_TOKENS_PER_NON_ASCII_CODE_POINT - 1;++/**+ * The UTF-8 byte length a text must EXCEED before+ * {@link tokenEstimateCeiling} can possibly exceed `tokens` - a test on+ * the one measurement that is free.+ *+ * Write `k` for {@link NON_ASCII_CEILING_COEFFICIENT}, `C` for code+ * points and `B` for bytes. The ceiling's integer form is `C + k*n` with+ * `n = min(C, B - C)`, and over the admissible range of `C` for a given+ * `B` that expression peaks at `C = B/2`, where it equals `(1 + k)*B/2`.+ * So a text can only be over budget `4W` when `(1 + k)*B/2 > 4W`, which+ * is `B > 8W / (1 + k)`.+ *+ * Worth stating as its own predicate because SQLite answers it without+ * decoding anything: `octet_length()` reads a stored byte count, while+ * `length()` walks the whole UTF-8 string. On a model with a wide window+ * this skips the walk for the entire chunk table.+ */+export function utf8ByteFloorUnderTokenBudget(tokens: number): number {+ return Math.floor(+ (2 * charLengthOverTokenBudget(tokens)) /+ (CHARS_PER_ESTIMATED_TOKEN * MAX_TOKENS_PER_NON_ASCII_CODE_POINT),+ );+}+/** Estimated spend in USD for `tokens` against `model`'s rate. */export function estimateCostUsd(tokens: number, model: string | null): number {const rate = pricePerMillionTokens(model);diff --git a/src/core/search/entities.ts b/src/core/search/entities.tsindex b8a50802..f1c3c6e3 100644--- a/src/core/search/entities.ts+++ b/src/core/search/entities.ts@@ -29,7 +29,17 @@ const CAP_RUN_RE = /\p{Lu}[\p{L}\p{N}]*(?:[ \t]+\p{Lu}[\p{L}\p{N}]*)*/gu;// Single tokens that are notable regardless of position: CamelCase,// ALLCAPS, or letter+digit mixes.const CAMEL_RE = /\p{Ll}+\p{Lu}[\p{L}\p{N}]*/gu;-const ALLCAPS_OR_DIGIT_RE = /[\p{Lu}\p{N}]*\p{Lu}[\p{Lu}\p{N}]+|\p{L}+\p{N}+|\p{N}+\p{L}+/gu;+// The two mixed-script alternatives consume their leading run ATOMICALLY,+// through the `(?=(x+))\1` idiom JavaScript has instead of atomic groups.+// Written as a plain `\p{L}+\p{N}+`, a letter run carrying no digit makes+// the engine retry every shorter prefix at every start position, which is+// quadratic per run and cubic over a document - and an unbroken letter run+// with no digits is what an ordinary Han, Thai or Lao paragraph IS. Measured+// on 2160 Han characters: 11.7 s and zero matches before, 20 ms after, with+// identical output on every mixed-script sample. Indexing such a vault was+// effectively impossible.+const ALLCAPS_OR_DIGIT_RE =+ /[\p{Lu}\p{N}]*\p{Lu}[\p{Lu}\p{N}]+|(?=(\p{L}+))\1\p{N}+|(?=(\p{N}+))\2\p{L}+/gu;/** Lowercase + collapse internal whitespace. */function normalize(raw: string): string {diff --git a/src/core/search/explain-envelope.ts b/src/core/search/explain-envelope.tsnew file mode 100644index 00000000..6d0cce7b--- /dev/null+++ b/src/core/search/explain-envelope.ts@@ -0,0 +1,141 @@+/**+ * The explain envelope (what-the-index-already-knew, task F).+ *+ * Every gated search builds two compact-reference receipts - the+ * retrieval decision trace and the memory trust assessment - and until+ * this unit no surface serialized either: they were reachable only from a+ * test. This module is the single definition of how those receipts reach+ * a caller, so the CLI `--json` payload and the MCP response name the+ * same keys with the same bodies.+ *+ * Two rules the surfaces inherit by using it:+ *+ * - Off means absent. With the explain flag unset the envelope is+ * empty, so the response bytes are identical to the pre-change ones -+ * never a null placeholder.+ * - A check that could not run says so, and says which check. The+ * receipts exist only when the retrieval trust gate is engaged and a+ * cross-vault union never merges them, so asking for them without+ * them returns `retrieval_trace_unavailable` carrying whichever of+ * those two situations actually applies - never an unexplained+ * silence, and never the wrong explanation.+ *+ * The receipt bodies are already snake_cased by their builders+ * (src/core/brain/trust/retrieval-receipts.ts), so this is a key+ * projection and not a re-serialization.+ */++import type { SearchOutcome } from "./types.ts";++/** Config key that engages the retrieval trust gate. */+export const TRUST_GATE_CONFIG_KEY = "search_trust_gate_enabled";++/** Environment variable equivalent of {@link TRUST_GATE_CONFIG_KEY}. */+export const TRUST_GATE_ENV_KEY = "OPEN_SECOND_BRAIN_SEARCH_TRUST_GATE";++/**+ * Why an explain request carries no trace, and the exit that produces+ * one. The receipts are a by-product of the retrieval trust gate: with+ * the gate off there is no evaluated / excluded set to report.+ */+export const RETRIEVAL_TRACE_UNAVAILABLE =+ `no retrieval decision trace: the retrieval trust gate is off. ` ++ `Enable it with ${TRUST_GATE_CONFIG_KEY}: true in the config, or ${TRUST_GATE_ENV_KEY}=1.`;++/**+ * The cross-vault union's own reason. A union outcome is merged from one+ * search per origin and the receipts are per-origin: they reference+ * document and chunk ids that are only meaningful inside the store that+ * issued them, so merging them would produce ambiguous references. The+ * gate state is therefore NOT what is being reported here, and saying it+ * was would be a false diagnosis.+ */+export const RETRIEVAL_TRACE_NOT_MERGED =+ `no retrieval decision trace: a cross-vault union merges one search per origin and the ` ++ `per-origin receipts reference ids that are local to their own store. ` ++ `Run the query against a single vault to see its trace.`;++/**+ * Both situations at once. The gate is named first because it is the one+ * that has to change either way, and the union is named because changing+ * it alone still yields nothing - reporting only one of the two would+ * send the operator to an exit that does not produce a trace.+ */+export const RETRIEVAL_TRACE_UNAVAILABLE_AND_NOT_MERGED =+ `no retrieval decision trace: the retrieval trust gate is off, and a cross-vault union ` ++ `would not merge the per-origin receipts even with it on. ` ++ `Enable ${TRUST_GATE_CONFIG_KEY}: true (or ${TRUST_GATE_ENV_KEY}=1) and query a single vault.`;++/** Response key carrying the retrieval decision trace. */+export const RETRIEVAL_DECISION_TRACE_KEY = "retrieval_decision_trace";++/** Response key carrying the memory trust assessment. */+export const MEMORY_TRUST_ASSESSMENT_KEY = "memory_trust_assessment";++/** Response key carrying the reason there is no trace to show. */+export const RETRIEVAL_TRACE_UNAVAILABLE_KEY = "retrieval_trace_unavailable";++/**+ * What a surface asks the envelope for. `crossVault` is not derivable+ * from the outcome - a union and a single-vault search with the gate off+ * are both receipt-less - and the two cases have different exits, so the+ * caller that knows states it.+ */+export interface ExplainRequest {+ readonly explain: boolean;+ /** True when this outcome came from a cross-vault union. */+ readonly crossVault: boolean;+ /**+ * The resolved `recall.retrievalTrustGateEnabled`. Neither this nor+ * `crossVault` is derivable from a receipt-less outcome, and naming the+ * wrong one of the two sends the operator to an exit that produces+ * nothing - so the caller, which knows both, states both.+ */+ readonly trustGateEnabled: boolean;+}++/** The default for every surface: opt-in, single vault. */+export const EXPLAIN_OFF: ExplainRequest = Object.freeze({+ explain: false,+ crossVault: false,+ trustGateEnabled: false,+});++/**+ * Why this outcome carries no receipts, as the caller's own situation+ * explains it.+ */+export function retrievalTraceUnavailableReason(+ crossVault: boolean,+ trustGateEnabled: boolean,+): string {+ if (crossVault && !trustGateEnabled) return RETRIEVAL_TRACE_UNAVAILABLE_AND_NOT_MERGED;+ if (crossVault) return RETRIEVAL_TRACE_NOT_MERGED;+ return RETRIEVAL_TRACE_UNAVAILABLE;+}++/**+ * The explain keys for one outcome, ready to spread into a response+ * object. `explain: false` yields an empty object - the caller's payload+ * is then byte-identical to the pre-change one.+ */+export function explainEnvelope(+ outcome: SearchOutcome,+ request: ExplainRequest,+): Record<string, unknown> {+ if (!request.explain) return {};+ const trace = outcome.retrievalDecisionTrace;+ const assessment = outcome.memoryTrustAssessment;+ if (trace === undefined || assessment === undefined) {+ return {+ [RETRIEVAL_TRACE_UNAVAILABLE_KEY]: retrievalTraceUnavailableReason(+ request.crossVault,+ request.trustGateEnabled,+ ),+ };+ }+ return {+ [RETRIEVAL_DECISION_TRACE_KEY]: trace,+ [MEMORY_TRUST_ASSESSMENT_KEY]: assessment,+ };+}diff --git a/src/core/search/feedback.ts b/src/core/search/feedback.tsindex 3c40615a..1aafbc7b 100644--- a/src/core/search/feedback.ts+++ b/src/core/search/feedback.ts@@ -209,10 +209,19 @@ export function writeLearnedWeights(vault: string, weights: LearnedWeights): voiexport function readLearnedWeights(vault: string): LearnedWeights | null {try {const parsed = JSON.parse(readFileSync(learnedWeightsPath(vault), "utf8")) as LearnedWeights;- // All four multipliers must be finite numbers — a partially-corrupt- // file must not feed NaN into the weight composition.+ // All four multipliers must be finite numbers inside the bounds the+ // fold itself clamps to — a partially-corrupt file must not feed NaN+ // into the weight composition, and a hand-edited one must not inject a+ // value `computeLearnedWeights` could never have produced. Zero deletes+ // a lane and a negative value inverts its rank order under rank fusion,+ // so the bound is the guard, not merely finiteness.const muls = [parsed.keywordMul, parsed.semanticMul, parsed.entityMul, parsed.recencyMul];- if (!muls.every((m) => typeof m === "number" && Number.isFinite(m))) return null;+ const inBounds = (m: unknown): boolean =>+ typeof m === "number" &&+ Number.isFinite(m) &&+ m >= LEARNED_WEIGHT_MIN &&+ m <= LEARNED_WEIGHT_MAX;+ if (!muls.every(inBounds)) return null;if (typeof parsed.events !== "number") return null;return parsed;} catch {diff --git a/src/core/search/fusion.ts b/src/core/search/fusion.tsindex 1ca3e900..2bf3300d 100644--- a/src/core/search/fusion.ts+++ b/src/core/search/fusion.ts@@ -4,20 +4,74 @@* The default `linear` mode (a weighted sum of min-max-normalised BM25* and cosine, computed in `ranker.ts`) is unchanged. This module adds* `rrf` - Reciprocal Rank Fusion - which combines the sparse and dense- * lanes by RANK POSITION rather than by score magnitude:+ * lanes by RANK POSITION rather than by score magnitude.*- * rrf(chunk) = sum over lanes of 1 / (k + rank_in_lane)+ * RRF ignores the CONFIGURED lane weights (`search_keyword_weight` /+ * `search_semantic_weight`) on purpose: those calibrate two score+ * magnitudes against each other, and this fusion reads no magnitudes. It+ * does NOT ignore what the caller asked for. A per-query intent profile+ * (`query-plan.ts`, composed with the learned recall weights) states a+ * preference between the lanes themselves - a quoted phrase wants the+ * lexical lane - and that preference is applied to the per-lane+ * CONTRIBUTION, the standard weighted-RRF formulation:*- * RRF is weightless by design (it ignores the per-lane weights and the- * intent multipliers) and robust to lanes whose score scales differ. The- * raw RRF sums are min-max-normalised to [0, 1] so the fused relevance is+ * rrf(chunk) = sum over lanes of w_lane / (k + rank_in_lane)+ *+ * so it stays rank-based and scale-free. `w_lane = 1` on every lane+ * reproduces the classic term exactly, which is what a query that+ * declared no intent gets.+ *+ * The raw RRF sums are min-max-normalised to [0, 1] so the fused relevance is* on the same scale as the linear path and composes with the same* downstream boosts (link / recency / entity / tier / session focus).*/+import { SearchError } from "./search-error.ts";+/** Canonical RRF damping constant from the original Cormack et al. paper. */export const DEFAULT_RRF_K = 60;+/** The weight of a lane the caller expressed no preference about. */+export const NEUTRAL_LANE_WEIGHT = 1;++/**+ * The relational fan-out arm is always neutral: the intent profile+ * describes a lexical-versus-semantic preference, and a typed-edge+ * traversal is neither, so there is no multiplier that would honestly+ * apply to it.+ */+const RELATIONAL_LANE_WEIGHT = NEUTRAL_LANE_WEIGHT;++/**+ * Per-lane multipliers for {@link rrfFuse}. Both default to+ * {@link NEUTRAL_LANE_WEIGHT}; both must be finite and strictly positive.+ */+export interface RrfLaneWeights {+ readonly keyword: number;+ readonly semantic: number;+}++/** The command that clears the learned half of a lane weight. */+const LEARNED_WEIGHTS_RESET_COMMAND = "o2b search weights --reset";++/**+ * A lane weight of zero would delete the lane and a negative one would+ * invert its rank order - in both cases the fusion silently answers a+ * question nobody asked. The bounded intent profiles cannot produce+ * either; a hand-edited learned-weights file can, so the refusal names+ * that file's reset command as the exit.+ */+function assertLaneWeight(weight: number, lane: keyof RrfLaneWeights): void {+ if (Number.isFinite(weight) && weight > 0) return;+ throw new SearchError(+ "INVALID_INPUT",+ `rrf fusion: the ${lane} lane weight must be a finite positive number, got ${weight}. ` ++ "A non-positive weight inverts the lane's rank order instead of down-weighting it. " ++ "The weight is the query intent profile composed with the learned recall weights; " ++ `clear the learned half with: ${LEARNED_WEIGHTS_RESET_COMMAND}`,+ );+}+export type FusionMode = "linear" | "rrf";const FUSION_MODES: ReadonlySet<string> = new Set(["linear", "rrf"]);@@ -29,9 +83,13 @@ export function isFusionMode(value: string): value is FusionMode {/*** Fuse two lanes by reciprocal rank and min-max-normalise the result to* [0, 1]. Each input is the lane's chunk ids in best-first order; a chunk- * contributes `1 / (k + position)` for each lane it appears in. Returns a- * map from chunk id to its normalised fused relevance. An empty input on- * both lanes yields an empty map.+ * contributes `w_lane / (k + position)` for each lane it appears in.+ * Returns a map from chunk id to its normalised fused relevance. An empty+ * input on both lanes yields an empty map.+ *+ * `laneWeights` is the per-query intent preference. Omitting it (or+ * passing {@link NEUTRAL_LANE_WEIGHT} on both lanes) yields the classic+ * weightless term, bit for bit.*/export function rrfFuse(opts: {keywordRankedChunkIds: ReadonlyArray<number>;@@ -43,18 +101,26 @@ export function rrfFuse(opts: {*/relationalRankedChunkIds?: ReadonlyArray<number>;k: number;+ /** Per-lane intent multipliers; absent means neutral on both lanes. */+ laneWeights?: RrfLaneWeights;}): Map<number, number> {const k = Math.max(1, opts.k);+ const keywordWeight = opts.laneWeights?.keyword ?? NEUTRAL_LANE_WEIGHT;+ const semanticWeight = opts.laneWeights?.semantic ?? NEUTRAL_LANE_WEIGHT;+ assertLaneWeight(keywordWeight, "keyword");+ assertLaneWeight(semanticWeight, "semantic");const raw = new Map<number, number>();- const accumulate = (ids: ReadonlyArray<number>): void => {+ const accumulate = (ids: ReadonlyArray<number>, weight: number): void => {for (let i = 0; i < ids.length; i++) {const id = ids[i]!;- raw.set(id, (raw.get(id) ?? 0) + 1 / (k + (i + 1)));+ raw.set(id, (raw.get(id) ?? 0) + weight / (k + (i + 1)));}};- accumulate(opts.keywordRankedChunkIds);- accumulate(opts.semanticRankedChunkIds);- if (opts.relationalRankedChunkIds) accumulate(opts.relationalRankedChunkIds);+ accumulate(opts.keywordRankedChunkIds, keywordWeight);+ accumulate(opts.semanticRankedChunkIds, semanticWeight);+ if (opts.relationalRankedChunkIds) {+ accumulate(opts.relationalRankedChunkIds, RELATIONAL_LANE_WEIGHT);+ }if (raw.size === 0) return raw;diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex 039e8bda..86506814 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -40,6 +40,7 @@ type SearchConfigOverrides = Partial<export type {BrainSearchResult,+ ChunkWindowCensus,DisclosureMode,EvidencePack,ExpandedNote,@@ -61,7 +62,13 @@ export type {StructuredRecallQueryDocument,VaultIgnoreRule,} from "./types.ts";-export { EMBEDDING_QUOTA_MESSAGE, SearchError, SEARCH_ERROR_CODES } from "./types.ts";+export {+ chunkWindowDiagnosticCode,+ CHUNK_WINDOW_CODE,+ EMBEDDING_QUOTA_MESSAGE,+ SearchError,+ SEARCH_ERROR_CODES,+} from "./types.ts";export {parseStructuredRecallQueryDocument,structuredRecallQueryText,@@ -76,6 +83,7 @@ export {export { evaluateSurfacingGate, type SurfacingGateDecision } from "./surfacing-gate.ts";export { buildEvidencePack, serializeEvidencePack } from "./evidence-pack.ts";export {+ serializeChunkWindowCensus,serializeSearchCard,serializeIndexStatus,serializeStampMismatches,diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsindex cafd7483..db6210e0 100644--- a/src/core/search/indexer.ts+++ b/src/core/search/indexer.ts@@ -36,6 +36,7 @@ import {} from "./capability-tier.ts";import { chunkMarkdown } from "./chunker.ts";import { expandTextForCjkFts } from "./cjk-tokenizer.ts";+import { declaredInputWindowTokens, passagePrefixSentByProvider } from "./embeddings/presets.ts";import { makeProvider } from "./embeddings/provider.ts";import {embeddingSignature,@@ -71,11 +72,12 @@ import {LAST_INDEXED_AT_STATE_KEY,} from "./store.ts";import { LATEST_SCHEMA_VERSION } from "./schema.ts";-import { SearchError } from "./types.ts";+import { chunkWindowDiagnosticCode, SearchError } from "./types.ts";import { walkVault } from "./walker.ts";import { withTimeout } from "./with-timeout.ts";import type { ChunkInput, LinkInput } from "./store.ts";import type {+ ChunkWindowCensus,IndexCheckReport,IndexStats,IndexStatusSnapshot,@@ -131,6 +133,8 @@ interface MutableStats {eventAnchorsPending: number;backend: IndexStats["backend"];deferredReason: IndexStats["deferredReason"];+ /** Null until the census runs, and null whenever it has nothing to report. */+ chunkWindow: ChunkWindowCensus | null;}function newStats(): MutableStats {@@ -154,6 +158,8 @@ function newStats(): MutableStats {// Defaults to the deterministic offline backend.backend: "offline",deferredReason: null,+ // Taken at the end of the run, over the index the run left behind.+ chunkWindow: null,};}@@ -174,6 +180,10 @@ function freezeStats(s: MutableStats, durationMs: number): IndexStats {eventAnchorsPending: s.eventAnchorsPending,backend: s.backend,deferredReason: s.deferredReason,+ // Absent, not null, when the census has nothing to report: a run+ // with no oversize chunks and a declared window produces the exact+ // statistics object it produced before this field existed.+ ...(s.chunkWindow === null ? {} : { chunkWindow: s.chunkWindow }),durationMs,});}@@ -558,6 +568,15 @@ async function indexInto(// `since` / `until` filter now depends on the answer.stats.eventAnchorsPending = store.countUnexaminedEventAnchors();+ // Oversize-chunk census. Taken here, over the index this run just+ // wrote, because the chunk boundaries the run produced are exactly+ // what the configured model's declared window has to hold - and the+ // configured chunk size is expressed in whitespace WORDS while a+ // window is expressed in tokens, so the two settings can only be+ // compared through what they actually produced. The stored model+ // fills in a config that leaves `embedding_model` unset.+ stats.chunkWindow = censusChunkWindow(store, config, store.getState("embedding_model"));+const now = new Date().toISOString();store.setState(LAST_INDEXED_AT_STATE_KEY, now);// The SAME instant on a forced run: their equality is the@@ -598,6 +617,20 @@ async function indexInto(}}+/**+ * The model an embedding request would name right now. The local+ * provider has an implicit model string; every other provider takes the+ * configured one, falling back to what the index recorded when the+ * config leaves it unset.+ */+function activeEmbeddingModel(+ config: ResolvedSearchConfig,+ storedModel: string | null = null,+): string | null {+ if (config.semantic.provider === "local") return LOCAL_EMBEDDING_MODEL;+ return config.semantic.model ?? storedModel;+}+/*** Canonical signature of the ACTIVE embedding configuration for status* reporting. Null when semantic search is disabled. Stored model and@@ -610,11 +643,128 @@ function activeEmbeddingSignature(storedDim: number | null = null,): string | null {if (!config.semantic.enabled) return null;- const provider = config.semantic.provider;- const model =- provider === "local" ? LOCAL_EMBEDDING_MODEL : (config.semantic.model ?? storedModel);- const dimension = config.semantic.dimension ?? storedDim;- return embeddingSignature({ provider, model, dimension });+ return embeddingSignature({+ provider: config.semantic.provider,+ model: activeEmbeddingModel(config, storedModel),+ dimension: config.semantic.dimension ?? storedDim,+ });+}++/**+ * The config key whose value produced the chunk boundaries under census.+ * Named once so the warning and any future surface cannot spell it+ * differently from `resolveSearchConfig`.+ */+const CHUNK_SIZE_CONFIG_KEY = "search_chunk_size";++/**+ * Take the oversize-chunk census over an open index.+ *+ * Returns `null` when there is nothing to report, which covers three+ * genuinely different silences, all of them states in which no chunk can+ * overflow a window:+ *+ * - semantic search is off, so nothing is ever embedded;+ * - the provider is the offline local embedder, which hashes n-gram+ * features over the whole text - no encoder, no positional limit,+ * nothing to truncate. A provably absent window, not an unknown one;+ * - the census ran against a declared window and every chunk fits.+ *+ * The silences it does NOT produce are the two states the estimate+ * cannot back: a model with no declared window (`window-undeclared`),+ * and chunks the estimate cannot place on either side of a declared one+ * (`estimate-undecided`). A check that could not run and a check that+ * passed are not the same statement.+ */+function censusChunkWindow(+ store: Store,+ config: ResolvedSearchConfig,+ storedModel: string | null,+): ChunkWindowCensus | null {+ if (!config.semantic.enabled) return null;+ if (config.semantic.provider === "local") return null;+ const chunksMeasured = store.chunkCount();+ if (chunksMeasured === 0) return null;++ const model = activeEmbeddingModel(config, storedModel);+ const windowTokens = declaredInputWindowTokens(model);+ if (model === null || windowTokens === null) {+ return Object.freeze({ verdict: "window-undeclared" as const, model, chunksMeasured });+ }+ // The prefix the CONFIGURED backend prepends, not the one the config+ // asks for: only `openai-compat` implements the instruction-prefix+ // contract, and subtracting a prefix a backend never sends would+ // report an overflow nobody can hit.+ const tally = store.chunkWindowTally(+ windowTokens,+ passagePrefixSentByProvider(config.semantic.provider, config.semantic.passagePrefix),+ );+ if (tally.overWindow > 0) {+ return Object.freeze({+ verdict: "over-window" as const,+ model,+ windowTokens,+ chunksOverWindow: tally.overWindow,+ // Absent, never zero: an all-ASCII index has no undecided band and+ // carries the record it carried before this field existed.+ ...(tally.undecided > 0 ? { chunksUndecided: tally.undecided } : {}),+ chunksMeasured,+ });+ }+ if (tally.undecided > 0) {+ return Object.freeze({+ verdict: "estimate-undecided" as const,+ model,+ windowTokens,+ chunksUndecided: tally.undecided,+ chunksMeasured,+ });+ }+ return null;+}++/**+ * The operator-facing warning line for a census that MEASURED something+ * - chunks proved over the window, or chunks its estimate cannot place+ * inside one - ending in the command its REGISTERED diagnostic names.+ * The command is resolved from the registry rather than written here, so+ * this sentence cannot name an exit the registry does not hold; an+ * unregistered code throws at the point the sentence is built rather+ * than shipping a dead end.+ *+ * Typed to the two measured members, so the undeclared verdict cannot+ * reach the warning channel by accident - it is a snapshot field, and+ * the type is what enforces that rather than a branch nobody calls.+ *+ * The undecided line says in words that it is not a pass, because that+ * is the whole content of the state: the operator is being told the+ * check ran and could not clear these chunks, not that they are broken.+ */+async function formatChunkWindowMeasured(+ census: Extract<ChunkWindowCensus, { verdict: "over-window" | "estimate-undecided" }>,+): Promise<string> {+ const { requireNextStep } = await import("../brain/next-step.ts");+ const step = requireNextStep(chunkWindowDiagnosticCode(census));+ const remedy =+ `whatever does not fit is truncated by the provider, silently. ` ++ `Lower ${CHUNK_SIZE_CONFIG_KEY}, then run: ${step.nextCommand}`;+ if (census.verdict === "estimate-undecided") {+ return (+ `${census.chunksUndecided} of ${census.chunksMeasured} indexed chunk(s) cannot be ` ++ `placed inside the ${census.windowTokens}-token input window declared for ` ++ `${census.model}: the census estimates tokens from character counts, which does not ` ++ `bound a token window for non-Latin scripts, so this is not a pass - ${remedy}`+ );+ }+ const undecided =+ census.chunksUndecided === undefined+ ? ""+ : ` A further ${census.chunksUndecided} chunk(s) the estimate cannot place either way.`;+ return (+ `${census.chunksOverWindow} of ${census.chunksMeasured} indexed chunk(s) estimate above ` ++ `the ${census.windowTokens}-token input window declared for ${census.model}; ` ++ `${remedy}${undecided}`+ );}/**@@ -990,12 +1140,40 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexStwarnings.push(formatEmbeddingAbiDrift(abiDrift));}+ // Oversize-chunk census (what-the-index-already-knew, task H). The+ // two reportable states leave by different doors, following the+ // convention the ABI drift beside them already sets: the warning+ // channel is for what an operator can act on, the structured field+ // for what is simply true.+ //+ // - chunks past a DECLARED window is a measured fault with a+ // command behind it, so it is a warning. So is the undecided+ // band beside it: the census cannot rule the truncation out, the+ // remedy is the same lever, and it fires only for chunks that+ // actually carry non-ASCII text near the window - never on the+ // all-Latin index, which is what keeps it from becoming noise;+ // - a model with no declared window is a check that could not run.+ // It has no command - nobody can declare a window for someone+ // else's model - and it is the COMMON case, so a warning here+ // would repeat forever on the majority of configurations and+ // train operators to skip the channel. It rides the snapshot as+ // its own field instead, which still distinguishes it from a+ // pass.+ //+ // A census that cleared BOTH bounds is neither, so a status that had+ // no warning before this feature still has none.+ const chunkWindow = censusChunkWindow(store, config, model);+ const chunkWindowUndeclared = chunkWindow?.verdict === "window-undeclared" ? chunkWindow : null;+ if (+ chunkWindow !== null &&+ (chunkWindow.verdict === "over-window" || chunkWindow.verdict === "estimate-undecided")+ ) {+ warnings.push(await formatChunkWindowMeasured(chunkWindow));+ }+// Best-effort spend estimate to bring stale/missing embeddings current.// Only scan chunk content when the active model is actually priced.- const activeModel =- config.semantic.provider === "local"- ? LOCAL_EMBEDDING_MODEL- : (config.semantic.model ?? model);+ const activeModel = activeEmbeddingModel(config, model);let estimatedRefreshCostUsd = 0;if (config.semantic.enabled && pricePerMillionTokens(activeModel) > 0) {const pending = store.findChunksWithoutEmbeddings();@@ -1023,6 +1201,9 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexStlastIndexedAt: last,lastFullIndexAt: full,embeddingAbi: abiDrift,+ // Absent, not null, whenever the check could run - so a status+ // over a curated model serializes the bytes it always has.+ ...(chunkWindowUndeclared === null ? {} : { chunkWindowUndeclared }),warnings: Object.freeze(warnings),});} finally {diff --git a/src/core/search/pipeline/assemble.ts b/src/core/search/pipeline/assemble.tsindex 59be1008..bd0f2e6a 100644--- a/src/core/search/pipeline/assemble.ts+++ b/src/core/search/pipeline/assemble.ts@@ -20,6 +20,7 @@ import { semanticPoolSize } from "../semantic-phase.ts";import { applyTemporalBridge } from "../temporal-bridge.ts";import { applyPoolFilters, resolvePoolFilters, type FilterContext } from "./pool-filters.ts";import type { CandidateSignals } from "./candidate-signals.ts";+import type { DuplicatePassageLocation } from "../search-result.ts";import type { HydratedChunk, KeywordHit, SemanticHit, Store } from "../store.ts";import type { ResolvedTimeRange } from "../time-range.ts";import type { TemporalIntent } from "../temporal-intent.ts";@@ -36,6 +37,12 @@ import type {const POOL_LIMIT_MULTIPLIER = 3;const POOL_MIN_WIDTH = 30;+/**+ * Joins the two halves of a duplicate-identity key. Never occurs in a+ * sha256 hex digest, so the concatenation cannot be ambiguous.+ */+const DUPLICATE_KEY_SEPARATOR = ":";+/** Traversal is off at zero hops; MMR is off at a lambda of 1. */const NO_TRAVERSAL_HOPS = 0;const MMR_OFF_LAMBDA = 1;@@ -58,6 +65,15 @@ export interface AssemblyInput {readonly temporalIntent: TemporalIntent | null;readonly declaredEventTimeMs: (path: string) => number | null;readonly limit: number;+ /**+ * The single instant the request resolved, forwarded to the ranker so+ * the freshness layer scores against the same clock every other+ * time-resolving decision in this call used. Without it the ranker+ * falls back to its own `Date.now()` and a score - and therefore the+ * exact-float first rung of the tie-break ladder - carries a clock+ * reading taken at an arbitrary later point in the pipeline.+ */+ readonly nowMs: number;}interface Assembly {@@ -65,6 +81,91 @@ interface Assembly {readonly visible: ReadonlyArray<BrainSearchResult>;/** Whether the cap truncated the pool, as opposed to the pool running out. */readonly capHit: boolean;+ /** Rows the exact-duplicate merge folded into a higher-ranked row. */+ readonly mergedAway: number;+}++/** The location record a merged-away duplicate leaves on its survivor. */+function duplicateLocation(r: BrainSearchResult): DuplicatePassageLocation {+ return Object.freeze({+ documentId: r.documentId,+ chunkId: r.chunkId,+ path: r.path,+ title: r.title,+ startLine: r.startLine,+ endLine: r.endLine,+ });+}++/**+ * The duplicate-identity key of a hydrated candidate, or `null` when the+ * row cannot state one.+ *+ * Both halves are sha256 hex written on every index run since schema v1:+ * `chunks.content_hash` over the passage, `documents.content_hash` over+ * the whole source file. The document half is load-bearing, not belt and+ * braces - frontmatter is stripped before chunking, so two genuinely+ * distinct records that share a body (the same rule with a different+ * `freshness_trend`, the same prose with a different `authored_at`) hash+ * their chunks identically while the ranker scores them differently on+ * purpose. Requiring both means the merge fires only on a real copy of a+ * file, where the merged-away row provably carries nothing but its path.+ */+function duplicateKey(hydrated: HydratedChunk | undefined): string | null {+ if (hydrated?.contentHash === undefined || hydrated.documentHash === undefined) return null;+ return `${hydrated.documentHash}${DUPLICATE_KEY_SEPARATOR}${hydrated.contentHash}`;+}++/**+ * Fold byte-identical passages into their highest-ranked occurrence.+ *+ * Equal key means equal bytes at both the passage and the file level, so+ * the second occurrence adds no information to the answer while consuming+ * a slot in the caller's window; MMR only demoted it, and a demoted row is+ * still a returned row. Every folded location is recorded on the survivor,+ * so nothing is silently dropped.+ *+ * A near-duplicate hashes differently and is left entirely alone - the+ * diversity rerank remains the only thing that judges similarity.+ * A candidate whose hydrated row states no key (traversal expansions,+ * hydrated through the representative-chunk read) is passed through+ * untouched rather than treated as equal to its peers.+ *+ * `ranked` is score-sorted, so the survivor is always the best-ranked+ * occurrence and the relative order of the surviving rows is preserved.+ */+function mergeExactDuplicates(+ ranked: ReadonlyArray<BrainSearchResult>,+ hydrated: ReadonlyMap<number, HydratedChunk>,+): { readonly results: BrainSearchResult[]; readonly mergedAway: number } {+ const survivorAt = new Map<string, number>();+ const foldedInto = new Map<number, DuplicatePassageLocation[]>();+ const results: BrainSearchResult[] = [];+ for (const r of ranked) {+ const hash = duplicateKey(hydrated.get(r.chunkId));+ if (hash === null) {+ results.push(r);+ continue;+ }+ const at = survivorAt.get(hash);+ if (at === undefined) {+ survivorAt.set(hash, results.length);+ results.push(r);+ continue;+ }+ const folded = foldedInto.get(at);+ if (folded === undefined) foldedInto.set(at, [duplicateLocation(r)]);+ else folded.push(duplicateLocation(r));+ }+ let mergedAway = 0;+ for (const [at, locations] of foldedInto) {+ mergedAway += locations.length;+ results[at] = Object.freeze({+ ...results[at]!,+ duplicates: Object.freeze(locations),+ });+ }+ return { results, mergedAway };}export function assembleRankedResults(input: AssemblyInput): ReadonlyArray<BrainSearchResult> {@@ -126,6 +227,13 @@ export function assembleRankedResults(input: AssemblyInput): ReadonlyArray<Brain// phase can surface it. No-op (same reference) when the pool has none,// so a vault that never used the lane is byte-identical.ranked = applyExactStateBarrier(ranked).slice();+ // Exact-duplicate merge (what-the-index-already-knew, task D): fold+ // byte-identical passages into their best-ranked occurrence BEFORE+ // traversal and the diversity rerank, so a duplicate neither seeds+ // expansion twice nor pays for a quadratic Jaccard comparison that+ // the hash already settled. No-op when the pool holds no duplicates.+ const merge = mergeExactDuplicates(ranked, input.hydrated);+ ranked = merge.results;// Link-graph traversal (v0.13.0): walk outbound links from the top// hits and surface related documents not already matched, scored by// decay. No-op when maxHops == 0. Runs before MMR so expansions are@@ -164,22 +272,34 @@ export function assembleRankedResults(input: AssemblyInput): ReadonlyArray<Brainif (rerankActive) {ranked = rerankByRelevance(ranked).slice();}- return { ...applyPoolFilters(ranked, filters, filterContext), capHit };+ return {+ ...applyPoolFilters(ranked, filters, filterContext),+ capHit,+ mergedAway: merge.mergedAway,+ };};let assembled = assemble(rankLimit);- // Default-scope visibility (no explicit filter, so no overfetch above)- // can drop tagged pages and leave fewer than `limit` rows while more- // untagged matches sit deeper in the candidate pool. When that happens- // and the narrow cap was actually hit, re-assemble once at the wider- // cap from the same in-memory candidates - no extra DB fetch. Untagged- // vaults never drop rows, so this never fires and their results stay- // byte-identical.+ // Two phases can leave fewer than `limit` rows out of a pool that was+ // capped at `limit`, while more qualifying matches sit deeper in the+ // candidate list:+ //+ // - default-scope visibility (no explicit filter, so no overfetch+ // above) drops tagged pages; untagged vaults never drop a row, so+ // this never fires for them,+ // - the exact-duplicate merge folds byte-identical passages away.+ //+ // In either case, and only when the narrow cap was actually hit (as+ // opposed to the candidate list simply running out), re-assemble once+ // at the wider cap from the same in-memory candidates - no extra DB+ // fetch. A pool with no dropped and no merged rows never re-assembles,+ // so its results stay byte-identical.+ const visibilityDropped =+ !filters.canDropRows && assembled.visible.length < assembled.preVisibility;if (- !filters.canDropRows &&assembled.visible.length < limit &&- assembled.visible.length < assembled.preVisibility &&- assembled.capHit+ assembled.capHit &&+ (visibilityDropped || assembled.mergedAway > 0)) {const wideCap = Math.max(semanticPoolSize(limit),@@ -225,6 +345,11 @@ function rankCandidates(input: AssemblyInput, rankCap: number): BrainSearchResulsemanticWeight: opts.semanticWeight ?? config.semanticWeight,limit: rankCap,semanticEnabled: input.semanticEnabled,+ // The request's clock, not the ranker's. Ranking is called up to+ // twice per search (the visibility backfill re-assembles from the+ // same candidates), and both passes must judge freshness against+ // the same instant as the time filter and the temporal intent did.+ nowMs: input.nowMs,recency: {shape: config.recall.recencyShape,scale: config.recall.recencyScale,diff --git a/src/core/search/pipeline/outcome.ts b/src/core/search/pipeline/outcome.tsindex 141aa35a..823d1a34 100644--- a/src/core/search/pipeline/outcome.ts+++ b/src/core/search/pipeline/outcome.ts@@ -72,6 +72,15 @@ export interface OutcomeInput {readonly routedSurface: QuerySurface;readonly trustReceipts: TrustReceipts | null;readonly frontmatterCache: FrontmatterCache;+ /**+ * Size of the ranked pool the caller's `limit` sliced `results` out of+ * (what-the-index-already-knew, task F). Reported inline as `total`:+ * it is the only place the outcome says how many candidates were+ * ranked and then dropped, and it is a property of the answer rather+ * than a diagnostic, so no flag gates it. Always >= `results.length`,+ * because the results ARE its prefix.+ */+ readonly poolSize: number;}export function buildSearchOutcome(input: OutcomeInput): SearchOutcome {@@ -141,12 +150,14 @@ export function buildSearchOutcome(input: OutcomeInput): SearchOutcome {// only the surfaced depth differs - so the contract stays// deterministic and the default `full` path is byte-identical.if ((opts.disclosure ?? "full") === "cards") {- const cards = resultsOut.map(toSearchCard);+ // The residual query text anchors each card's snippet window on the+ // match instead of the head of the chunk (task E).+ const cards = resultsOut.map((result) => toSearchCard(result, query));return Object.freeze({results: NO_RESULTS,cards: Object.freeze(cards),warnings: Object.freeze(warnings),- total: cards.length,+ total: input.poolSize,...tail,});}@@ -154,7 +165,7 @@ export function buildSearchOutcome(input: OutcomeInput): SearchOutcome {return Object.freeze({results: Object.freeze(resultsOut),warnings: Object.freeze(warnings),- total: resultsOut.length,+ total: input.poolSize,...tail,});}diff --git a/src/core/search/pipeline/request.ts b/src/core/search/pipeline/request.tsindex a7ef88ed..1ed47214 100644--- a/src/core/search/pipeline/request.ts+++ b/src/core/search/pipeline/request.ts@@ -89,14 +89,14 @@ export function resolveSearchRequest(}const pathPrefix = assertSafePathPrefix(opts.pathPrefix);const policy = resolveSemanticPolicy(resolvedConfig, opts);+ // One clock for every time-resolving decision in this call, so the hard+ // filter, the session-focus expiry and the query-declared window below+ // cannot disagree by a few milliseconds of wall-clock drift.+ const nowMs = Date.now();const sessionFocus =opts.sessionFocus === undefined- ? readActiveSessionFocus(resolvedConfig, opts.focusSession, Date.now())+ ? readActiveSessionFocus(resolvedConfig, opts.focusSession, nowMs): opts.sessionFocus;- // One clock for every time-resolving decision in this call, so the hard- // filter and the query-declared window below cannot disagree by a few- // milliseconds of wall-clock drift.- const nowMs = Date.now();// Time-aware recall (recall-trust-suite): resolve since/until up front// so invalid input fails fast, before any store I/O.const timeRange =diff --git a/src/core/search/pipeline/store-open.ts b/src/core/search/pipeline/store-open.tsindex 812bdca9..675d5c86 100644--- a/src/core/search/pipeline/store-open.ts+++ b/src/core/search/pipeline/store-open.ts@@ -5,7 +5,18 @@import { Store } from "../store.ts";import { SearchError } from "../types.ts";-import type { ResolvedSearchConfig } from "../types.ts";+import type { ResolvedSearchConfig, SearchErrorCode } from "../types.ts";++/**+ * The open failures a rebuild can actually repair: a stale schema, an+ * absent index, and an index that is present but unreadable - which now+ * includes one condemned by the structural integrity gate.+ */+const SELF_HEALABLE_CODES: ReadonlySet<SearchErrorCode> = new Set<SearchErrorCode>([+ "INDEX_MISSING",+ "SCHEMA_MISMATCH",+ "INDEX_UNREADABLE",+]);/*** Open the index for reading, self-healing a stale, absent, or unreadable@@ -15,24 +26,42 @@ import type { ResolvedSearchConfig } from "../types.ts";* forcing the user to run `o2b search reindex` / `o2b search index`, rebuild* once and retry. `reindexVault` is imported lazily so the hot path never* pulls in the indexer and there is no module cycle.+ *+ * A rebuild that fails is not swallowed. It may legitimately be a+ * concurrent writer already rebuilding (`INDEX_LOCKED`), in which case the+ * retry below succeeds and there is nothing to report - so its failure is+ * held rather than thrown. But if the retry ALSO fails, the retry's error+ * describes the symptom while the held one describes why the repair did not+ * happen, and a caller shown only the symptom would be sent back to the+ * command that just failed. Both are reported.*/export async function openReadOrSelfHeal(config: ResolvedSearchConfig): Promise<Store> {+ let openFailure: SearchError;+ try {+ return await Store.open(config, { mode: "read" });+ } catch (e) {+ if (!(e instanceof SearchError) || !SELF_HEALABLE_CODES.has(e.code)) throw e;+ openFailure = e;+ }++ let healFailure: string | null = null;+ try {+ const { reindexVault } = await import("../indexer.ts");+ await reindexVault(config);+ } catch (e) {+ healFailure = e instanceof Error ? e.message : String(e);+ }+try {return await Store.open(config, { mode: "read" });} catch (e) {- if (- e instanceof SearchError &&- (e.code === "INDEX_MISSING" || e.code === "SCHEMA_MISMATCH" || e.code === "INDEX_UNREADABLE")- ) {- try {- const { reindexVault } = await import("../indexer.ts");- await reindexVault(config);- } catch {- // A concurrent writer may already be rebuilding (INDEX_LOCKED), or the- // rebuild failed - fall through and let the retry surface real state.- }- return await Store.open(config, { mode: "read" });- }- throw e;+ if (healFailure === null) throw e;+ const code: SearchErrorCode = e instanceof SearchError ? e.code : "INDEX_UNREADABLE";+ const symptom = e instanceof Error ? e.message : String(e);+ throw new SearchError(+ code,+ `${symptom} (the automatic rebuild of ${config.dbPath} after ` ++ `${openFailure.code} also failed: ${healFailure})`,+ );}}diff --git a/src/core/search/ranker.ts b/src/core/search/ranker.tsindex eeba966e..4cca5120 100644--- a/src/core/search/ranker.ts+++ b/src/core/search/ranker.ts@@ -384,6 +384,12 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSeasemanticRankedChunkIds: semanticRanked,relationalRankedChunkIds: inputs.relationalRankedChunkIds ?? [],k: opts.rrfK ?? DEFAULT_RRF_K,+ // The intent profile weights the per-lane CONTRIBUTIONS, not the+ // fused score: a post-fusion multiplier is a constant factor on+ // every candidate and could not express a lane preference at all.+ // A neutral profile passes 1 on both lanes, which is the classic+ // weightless term bit for bit.+ laneWeights: { keyword: kwMul, semantic: semMul },});}@@ -495,8 +501,10 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea// removed - `recencyAmplitude: 0` stays the only off switch.const recency = recencyBoost(c.mtime, nowMs, recencyOpts) * recMul * temporalDamping;// Relevance term: reciprocal-rank-fused when in rrf mode, otherwise- // the weighted sum of the normalised lanes. RRF is weightless, so the- // per-lane weights and intent multipliers do not apply to it.+ // the weighted sum of the normalised lanes. The fused value already+ // carries the intent multipliers (applied per lane, above); the+ // CONFIGURED lane weights stay out of it, because they calibrate two+ // score magnitudes and rank fusion reads no magnitudes.const rrf = rrfByChunk?.get(c.chunkId) ?? 0;const weighted =rrfByChunk !== nulldiff --git a/src/core/search/recall-hint.ts b/src/core/search/recall-hint.tsindex bb548964..b8411c9a 100644--- a/src/core/search/recall-hint.ts+++ b/src/core/search/recall-hint.ts@@ -48,12 +48,15 @@ export function deriveRecallHint(.join(", ");const top = results[0]!;- // `total` is the corpus match count and should be >= the returned slice;- // guard against a caller passing a smaller total so the string never- // reads "Recalled 10 of 3".+ // `total` is the ranked candidate pool the window was sliced from, NOT a+ // count of corpus matches: it is capped at roughly three times the+ // requested limit and link traversal can add a document that matched no+ // term. Calling it "matches" would state a corpus fact the number+ // cannot carry. It should be >= the returned slice; guard against a+ // caller passing a smaller total so the string never reads "10 of 3".const denom = Math.max(total, results.length);return (- `Recalled ${results.length} of ${denom} matches (${breakdown}). ` ++ `Recalled ${results.length} of ${denom} ranked candidates (${breakdown}). ` +`Top hit "${trimTitle(top.title)}" (${top.searchType}, score ${top.score.toFixed(2)}). ` +`See each result's reasons[] for why it surfaced.`);diff --git a/src/core/search/schema.ts b/src/core/search/schema.tsindex 7a999e3f..0b745f80 100644--- a/src/core/search/schema.ts+++ b/src/core/search/schema.ts@@ -7,7 +7,7 @@* Anchored in docs/plans/2026-05-16-brain-search-design.md §5.*/-import type { Database } from "bun:sqlite";+import { Database } from "bun:sqlite";import { SearchError } from "./types.ts";@@ -18,6 +18,12 @@ import { SearchError } from "./types.ts";*/export const LATEST_SCHEMA_VERSION = 11;+/**+ * The one command that rebuilds an index this binary cannot read. Every+ * refusal on the open path names it, and names it identically.+ */+export const REINDEX_COMMAND = "o2b search reindex";+/*** The wikilink-resolution basename of a stored document path: the final* `/`-segment with a trailing `.md` stripped. `notes/alpha.md` → `alpha`,@@ -542,6 +548,183 @@ export const MIGRATIONS: ReadonlyArray<Migration> = Object.freeze([},]);+// ─────────────────────────────────────────────────────────────────────────────+// The expected-object manifest+// ─────────────────────────────────────────────────────────────────────────────++/**+ * A table, or a column of a table, that the schema declares and an opened+ * database does not have. `column: null` means the whole table is absent.+ */+export interface SchemaGap {+ /** Table the gap is in. */+ readonly table: string;+ /** Missing column, or null when the table itself is missing. */+ readonly column: string | null;+}++/**+ * Table names present in a database. Internal `sqlite_*` tables are+ * excluded: they are SQLite's own bookkeeping (`sqlite_stat1` appears only+ * after `ANALYZE`), never something a query of ours reads.+ */+const TABLE_NAMES_SQL =+ "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%'";++/** Column names of one table. */+const TABLE_COLUMNS_SQL = "SELECT name FROM pragma_table_info(?)";++function tableNames(db: Database): Set<string> {+ return new Set(+ db+ .query<{ name: string }, []>(TABLE_NAMES_SQL)+ .all()+ .map((r) => r.name),+ );+}++function columnNames(db: Database, table: string): Set<string> {+ return new Set(+ db+ .query<{ name: string }, [string]>(TABLE_COLUMNS_SQL)+ .all(table)+ .map((r) => r.name),+ );+}++/**+ * Virtual tables the query path can run WITHOUT, and therefore must not be+ * refused for. The manifest requires what a query cannot survive the+ * absence of; a lane that already classifies its own table as missing and+ * degrades is not that.+ *+ * `chunk_trigram` is the opt-in substring broadener from v9. `trigram.ts`+ * detects its absence (`isMissingTrigramTable`) and returns no candidates+ * silently, which is a documented, tested behaviour. Demanding it here+ * would push a legitimately readable index into a reindex that drops every+ * embedding, to restore a table the query path never needed.+ *+ * `chunk_vec` needs no entry: it is created by `ensureVecTable` only when+ * sqlite-vec is loaded, so the reference database never has one to begin+ * with.+ */+const OPTIONAL_VIRTUAL_TABLES: ReadonlyArray<string> = Object.freeze(["chunk_trigram"]);++/** The shadow tables FTS5 creates alongside a virtual table it owns. */+const FTS5_SHADOW_SUFFIXES: ReadonlyArray<string> = Object.freeze([+ "_data",+ "_idx",+ "_docsize",+ "_content",+ "_config",+]);++function isOptionalTable(name: string): boolean {+ return OPTIONAL_VIRTUAL_TABLES.some(+ (base) => name === base || FTS5_SHADOW_SUFFIXES.some((suffix) => name === base + suffix),+ );+}++/**+ * Every required table and its columns. Used to build the reference+ * manifest from a database this module has just migrated itself, so every+ * table in it is one of ours and introspecting all of them is safe.+ *+ * It is NOT how a live index is inspected: `PRAGMA table_info` on a virtual+ * table whose module is not loaded raises (`no such module: vec0`), and a+ * live index legitimately carries a `chunk_vec` table that the read open+ * has not loaded sqlite-vec for yet. {@link findSchemaGaps} therefore asks+ * only about the tables the manifest actually names.+ */+function schemaCensus(db: Database): Map<string, Set<string>> {+ const census = new Map<string, Set<string>>();+ for (const table of tableNames(db)) {+ if (isOptionalTable(table)) continue;+ census.set(table, columnNames(db, table));+ }+ return census;+}++/** Memoised {@link expectedSchemaObjects}; the reference build is not free. */+let expectedObjects: ReadonlyMap<string, ReadonlySet<string>> | null = null;++/**+ * The tables and columns a current index must have, DERIVED by running the+ * frozen {@link MIGRATIONS} against a fresh in-memory database and asking+ * SQLite what that produced.+ *+ * Deriving it is the whole point. A second hand-maintained list of tables+ * and columns would drift from the migrations that create them, and a+ * manifest that has drifted is worse than no manifest: it either refuses a+ * healthy index or blesses a broken one. Running the migrations cannot+ * drift from the migrations.+ *+ * It includes the FTS5 shadow tables of `chunk_fts` (`chunk_fts_data` and+ * friends) because a keyword query reads straight through them, and+ * excludes {@link OPTIONAL_VIRTUAL_TABLES} because the query path is+ * written to survive their absence.+ *+ * The tokenizer clause does not appear: it changes how `chunk_fts` tokenizes,+ * never what it or its shadows are CALLED, so the reference is built at the+ * default and matches an index built under any configured tokenizer.+ *+ * Cost is one in-memory migration run (tens of milliseconds), paid lazily+ * once per process and memoised, then nothing.+ */+export function expectedSchemaObjects(): ReadonlyMap<string, ReadonlySet<string>> {+ if (expectedObjects !== null) return expectedObjects;+ let reference: Database;+ try {+ reference = new Database(":memory:");+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError(+ "INDEX_UNREADABLE",+ `cannot build the reference schema for the index presence check: ${msg}`,+ );+ }+ try {+ applyMigrations(reference);+ expectedObjects = schemaCensus(reference);+ return expectedObjects;+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError(+ "INDEX_UNREADABLE",+ `cannot build the reference schema for the index presence check: ${msg}`,+ );+ } finally {+ reference.close();+ }+}++/**+ * Tables and columns {@link expectedSchemaObjects} declares that `db` does+ * not have. Empty for a healthy current index. Extra tables are NOT a gap:+ * `chunk_vec`, `sqlite_stat1` and a future version's additions are all+ * legitimately present in a database this binary can still read.+ */+export function findSchemaGaps(db: Database): ReadonlyArray<SchemaGap> {+ const present = tableNames(db);+ const gaps: SchemaGap[] = [];+ for (const [table, columns] of expectedSchemaObjects()) {+ if (!present.has(table)) {+ gaps.push({ table, column: null });+ continue;+ }+ const actual = columnNames(db, table);+ for (const column of columns) {+ if (!actual.has(column)) gaps.push({ table, column });+ }+ }+ return gaps;+}++/** Render {@link SchemaGap}s as `table` / `table.column`, for an error message. */+export function formatSchemaGaps(gaps: ReadonlyArray<SchemaGap>): string {+ return gaps.map((g) => (g.column === null ? g.table : `${g.table}.${g.column}`)).join(", ");+}+/** Read the current schema version from `index_state`. Returns 0 if not yet recorded. */export function readSchemaVersion(db: Database): number {const row = db@@ -601,7 +784,7 @@ export function applyMigrations(db: Database, opts: ApplyMigrationsOptions = {})throw new SearchError("SCHEMA_MISMATCH",`index schema version ${current} is newer than this binary supports (${LATEST_SCHEMA_VERSION}). ` +- `Run: o2b search reindex`,+ `Run: ${REINDEX_COMMAND}`,);}diff --git a/src/core/search/search-result.ts b/src/core/search/search-result.tsindex f1b3ec32..b280ae43 100644--- a/src/core/search/search-result.ts+++ b/src/core/search/search-result.ts@@ -65,6 +65,23 @@ export interface TrustMetadata {readonly replacement: string | null;}+/**+ * One location a merged-away exact duplicate was found at+ * (what-the-index-already-knew, task D). A passage whose `content_hash`+ * equals a higher-ranked candidate's is folded into that candidate rather+ * than occupying a second slot in the caller's window; every location it+ * was folded from is recorded here so the caller can still see that the+ * same bytes live at more than one path.+ */+export interface DuplicatePassageLocation {+ readonly documentId: number;+ readonly chunkId: number;+ readonly path: string;+ readonly title: string | null;+ readonly startLine: number;+ readonly endLine: number;+}+export interface BrainSearchResult {readonly documentId: number;readonly chunkId: number;@@ -129,4 +146,12 @@ export interface BrainSearchResult {* absent, keeping the legacy result shape byte-identical.*/readonly origin?: string;+ /**+ * Locations of the exact duplicates folded into this row+ * (what-the-index-already-knew, task D). Present ONLY when at least one+ * byte-identical passage was merged away, so a corpus that holds no+ * duplicate passages produces rows byte-identical to pre-merge ones.+ * Never empty when present, and never contains this row's own location.+ */+ readonly duplicates?: ReadonlyArray<DuplicatePassageLocation>;}diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex 2e0dcf66..f2a4b5d4 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -235,6 +235,7 @@ export async function search(temporalIntent,declaredEventTimeMs: eventTime.declaredEventTimeMs,limit,+ nowMs,});const postRank = await applyPostRankPhases({@@ -248,6 +249,12 @@ export async function search(});for (const w of postRank.warnings) warnings.push(w);+ // The pool the window is cut from (task F). `postRank.results` is the+ // last stage that can still add, drop or re-order a candidate, so its+ // length is the honest answer to "how many did you rank before+ // handing me these" - reported inline as `total`.+ const poolSize = postRank.results.length;+const results = decorateFinalResults({store,results: postRank.results.slice(0, limit),@@ -273,6 +280,7 @@ export async function search(routedSurface,trustReceipts: postRank.trustReceipts,frontmatterCache,+ poolSize,}),);} finally {diff --git a/src/core/search/serialize.ts b/src/core/search/serialize.tsindex 62bd21a2..42d00b25 100644--- a/src/core/search/serialize.ts+++ b/src/core/search/serialize.ts@@ -5,7 +5,7 @@*/import type { StampMismatch } from "../integrity/stamp.ts";-import type { IndexStatusSnapshot, SearchCard } from "./types.ts";+import type { ChunkWindowCensus, IndexStatusSnapshot, SearchCard } from "./types.ts";/*** Wire shape for a stamp comparison. `recorded` is what the index@@ -36,6 +36,51 @@ export function serializeSearchCard(c: SearchCard): Record<string, unknown> {document_id: c.documentId,chunk_id: c.chunkId,...(c.origin !== undefined ? { origin: c.origin } : {}),+ // Exact-duplicate merge (task D): the `path:Lstart-Lend` pointers the+ // surviving card was folded from. Absent, never null, when nothing+ // was merged - the card carries the key only in that case.+ ...(c.duplicatePointers !== undefined ? { duplicate_pointers: c.duplicatePointers } : {}),+ };+}++/**+ * Wire shape of an oversize-chunk census verdict, shared by the index+ * run's report and the status snapshot so the two cannot describe one+ * verdict with different field names.+ *+ * No `next_command` here: whether a surface names an exit beside a+ * verdict is that surface's decision, and the index run spreads its own+ * on top. Baking one in would put a command on the status field the+ * ruling deliberately keeps out of the warning channel.+ */+export function serializeChunkWindowCensus(census: ChunkWindowCensus): Record<string, unknown> {+ if (census.verdict === "window-undeclared") {+ return {+ verdict: census.verdict,+ model: census.model,+ chunks_measured: census.chunksMeasured,+ };+ }+ if (census.verdict === "estimate-undecided") {+ // No `chunks_over_window` here on the same terms the undeclared+ // verdict carries no window: nothing was proved over it.+ return {+ verdict: census.verdict,+ model: census.model,+ window_tokens: census.windowTokens,+ chunks_undecided: census.chunksUndecided,+ chunks_measured: census.chunksMeasured,+ };+ }+ return {+ verdict: census.verdict,+ model: census.model,+ window_tokens: census.windowTokens,+ chunks_over_window: census.chunksOverWindow,+ // Absent, never zero - an index the estimate decided in full emits+ // the record it emitted before the undecided band existed.+ ...(census.chunksUndecided === undefined ? {} : { chunks_undecided: census.chunksUndecided }),+ chunks_measured: census.chunksMeasured,};}@@ -67,6 +112,15 @@ export function serializeIndexStatus(s: IndexStatusSnapshot): Record<string, unk...(s.embeddingAbi.length > 0? { embedding_abi: serializeStampMismatches(s.embeddingAbi) }: {}),+ // Emitted only when the oversize-chunk census could NOT run, on the+ // same terms and for the same reason: a status over a model with a+ // declared window is byte-identical to the pre-census output. The+ // two verdicts the census reaches by measuring - overflow, and the+ // band its estimate cannot decide - are not here; both are warnings,+ // because both name a command.+ ...(s.chunkWindowUndeclared === undefined+ ? {}+ : { chunk_window_undeclared: serializeChunkWindowCensus(s.chunkWindowUndeclared) }),warnings: s.warnings,};}diff --git a/src/core/search/snippet-window.ts b/src/core/search/snippet-window.tsnew file mode 100644index 00000000..b04a6ec2--- /dev/null+++ b/src/core/search/snippet-window.ts@@ -0,0 +1,225 @@+/**+ * Match-anchored snippet windows — the one place a preview decides WHERE+ * in a passage to start (what-the-index-already-knew, task E).+ *+ * No match offset exists anywhere in the retrieval path: FTS5's+ * `offsets()` / `snippet()` / `highlight()` are never called, a keyword+ * hit is `{chunkId, documentId, bm25}`, and a chunk carries LINE numbers,+ * not character offsets. Nothing is dropped upstream — the offset is+ * never produced, so the window is computed here, at assembly time, from+ * the result content and the query.+ *+ * Every preview surface used to truncate from the HEAD of the chunk,+ * which made a match in the middle of a long passage invisible in the+ * only text the caller reads. The match-centred window and the+ * case-insensitive offset scan already existed — written and tested —+ * mounted on the session-recall surface alone. They live here now, and+ * the three head-truncating surfaces (MCP `content`, layer-1 card+ * snippets, the CLI transcript) call them.+ *+ * ## What this module owns, and what it deliberately does not+ *+ * It owns ANCHORING: which offset a window centres on, and where that+ * window starts. It does NOT own each surface's cap or continuation+ * marker, because the three genuinely disagree — the card snippet counts+ * CODE POINTS and appends `...` beyond its cap, the MCP window counts+ * UTF-16 units and fits its marker INSIDE a fixed token budget, the CLI+ * transcript counts UTF-16 units and appends `…` outside its cap.+ * Forcing one policy on all three would change the bytes every one of+ * them emits today, for no gain.+ *+ * ## The no-match case is a defined behaviour, not a fallback+ *+ * When no significant query term occurs in the text, {@link matchOffset}+ * returns {@link NO_MATCH_OFFSET} and the window starts at+ * {@link HEAD_WINDOW_START} — the head of the chunk, the same bytes a+ * head truncation produced before anchoring existed. The sentinel is+ * named rather than encoded as offset 0 precisely so a caller can never+ * read the result as "a match was found at position 0".+ *+ * Pure module: no I/O, no clock, no corpus access.+ */++import { significantTerms } from "./coverage.ts";++/**+ * Returned by {@link matchOffset} when no significant query term occurs+ * in the text. Distinct from offset 0, which means a term matched at the+ * very start.+ */+export const NO_MATCH_OFFSET = -1;++/** Where an unanchored window starts: the head of the text. */+export const HEAD_WINDOW_START = 0;++/** Continuation marker for a window that does not cover the whole text. */+export const ELLIPSIS = "…";++/**+ * Case-insensitive search that returns an offset into the ORIGINAL+ * `text`, so a window slice and `charSpanToLineSpan` (both of which+ * slice the original text) stay aligned. `text.toLowerCase()` is not+ * always length-preserving (e.g. U+0130 lowercases to two code units),+ * which would otherwise shift an offset taken from the lowercased copy.+ * The fast path keeps normal text O(n) and bit-identical; the scan runs+ * only when lowercasing changed the length.+ */+export function caseInsensitiveIndex(text: string, needleLower: string): number {+ const lower = text.toLowerCase();+ if (lower.length === text.length) return lower.indexOf(needleLower);+ for (let i = 0; i < text.length; i++) {+ if (text.slice(i).toLowerCase().startsWith(needleLower)) return i;+ }+ return NO_MATCH_OFFSET;+}++/**+ * Earliest offset in `text` at which any SIGNIFICANT term of `query`+ * occurs, or {@link NO_MATCH_OFFSET} when none does.+ *+ * Term selection is structural and delegated whole to+ * {@link significantTerms}, the same splitter the coverage engine and the+ * evidence pack already anchor on: there is deliberately no stop-word+ * list, no language table and no second length rule invented here. A+ * corpus-common term that happens to match first costs at most a window+ * that starts where a common word does — the corpus-frequency answer to+ * "which term matters" lives in the IDF weighting downstream and needs+ * document frequencies no preview surface holds.+ *+ * Earliest-occurrence, not longest-term or densest-cluster: it is+ * deterministic, needs no corpus, and cannot reorder under a tie.+ */+export function matchOffset(text: string, query: string): number {+ let earliest = NO_MATCH_OFFSET;+ for (const term of significantTerms(query)) {+ const at = caseInsensitiveIndex(text, term.toLowerCase());+ if (at === NO_MATCH_OFFSET) continue;+ if (earliest === NO_MATCH_OFFSET || at < earliest) earliest = at;+ }+ return earliest;+}++/**+ * Start of a `maxChars`-wide window centred on `matchAt`, clamped to the+ * head of the text. `matchAt` at or below {@link NO_MATCH_OFFSET} is the+ * defined no-match case and yields {@link HEAD_WINDOW_START}.+ *+ * The tail is deliberately NOT clamped: a match near the end produces a+ * short window rather than one slid backwards to stay full width. That is+ * the behaviour session recall has always had, and sliding would move the+ * match away from the centre the caller is looking at.+ *+ * This is the raw centring rule, and it knows nothing about how long the+ * text is. A capped preview surface must call {@link windowStartWithin}+ * instead: this function alone will open a window past the head of a+ * passage that fitted inside the cap whole, discarding text the surface+ * had room to show.+ */+export function windowStart(matchAt: number, maxChars: number): number {+ if (matchAt <= NO_MATCH_OFFSET) return HEAD_WINDOW_START;+ return Math.max(HEAD_WINDOW_START, matchAt - Math.floor(maxChars / 2));+}++/**+ * Start of a `maxChars` window centred on `matchAt` in a text of+ * `textLength`: {@link windowStart}, plus the one bound that needs to know+ * how much text there is.+ *+ * A text that fits inside the cap whole opens at {@link HEAD_WINDOW_START}+ * and is therefore shown in full. Without this, a match past the+ * half-window slid the start forward and the surface truncated the HEAD of+ * a passage it had room to show entirely — and stamped a leading+ * continuation marker on the loss.+ *+ * `maxChars` and `textLength` must be measured in the SAME unit; which+ * unit that is stays the caller's decision, which is why this module+ * anchors windows and does not own caps. The card snippet counts code+ * points, the CLI transcript and the MCP window count UTF-16 units, and+ * each passes its own pair.+ *+ * Over-cap text is left to {@link windowStart} untouched, so a match near+ * the tail still produces a SHORT window rather than one slid backwards to+ * stay full width. Clamping the start to `textLength - maxChars` would+ * widen those windows, but it would also reverse the deliberate+ * short-window behaviour documented above and change every session-recall+ * snippet whose match sits near the end.+ */+export function windowStartWithin(matchAt: number, maxChars: number, textLength: number): number {+ if (textLength <= maxChars) return HEAD_WINDOW_START;+ return windowStart(matchAt, maxChars);+}++/** UTF-16 code units reserved for the high half of a surrogate pair. */+const HIGH_SURROGATE_MIN = 0xd800;+const HIGH_SURROGATE_MAX = 0xdbff;+/** UTF-16 code units reserved for the low half of a surrogate pair. */+const LOW_SURROGATE_MIN = 0xdc00;+const LOW_SURROGATE_MAX = 0xdfff;++/**+ * `utf16Offset`, moved back onto a code-point boundary when it lands on+ * the LOW half of a surrogate pair.+ *+ * A surface that slices UTF-16 units at an anchored start can open a+ * window inside an astral character (emoji, rare CJK), and the body then+ * begins with a lone low surrogate that becomes U+FFFD the moment the+ * payload is encoded as UTF-8. Backing up by one unit re-includes the high+ * half, so the character opens the window whole. The window's width is+ * measured from the returned start, so no cap is exceeded.+ *+ * An offset already on a boundary is returned unchanged, and so is a lone+ * low surrogate with no high half in front of it: the source text is+ * already ill-formed there, and inventing a pair would misrepresent it.+ * Callers that slice a code-point array (the card snippet) never need+ * this — every index into such an array is already a boundary.+ */+export function alignToCodePointStart(text: string, utf16Offset: number): number {+ if (utf16Offset <= HEAD_WINDOW_START) return utf16Offset;+ const unit = text.charCodeAt(utf16Offset);+ if (unit < LOW_SURROGATE_MIN || unit > LOW_SURROGATE_MAX) return utf16Offset;+ const previous = text.charCodeAt(utf16Offset - 1);+ if (previous < HIGH_SURROGATE_MIN || previous > HIGH_SURROGATE_MAX) return utf16Offset;+ return utf16Offset - 1;+}++/**+ * A `maxChars` window of `text` centred on the match at `index`. The+ * session-recall snippet: this is the function the three preview surfaces+ * were missing.+ *+ * Bounded by {@link windowStartWithin}, so a record shorter than+ * `maxChars` is returned whole instead of losing its head to a window+ * there was no room to move.+ */+export function snippetAround(text: string, index: number, maxChars: number): string {+ const start = windowStartWithin(index, maxChars, text.length);+ return text.slice(start, start + maxChars);+}++/**+ * Wrap a window body in continuation markers: `marker` is prefixed when+ * the window does not begin at the head of the source, and appended when+ * `hasMore` says it does not reach the end.+ *+ * A head-anchored window that covers the tail is returned unchanged,+ * which is exactly what keeps the no-match case byte-identical. Each+ * surface decides `hasMore` for itself, because each measures its cap in+ * its own units.+ */+export function markWindow(body: string, start: number, hasMore: boolean, marker: string): string {+ const head = start > HEAD_WINDOW_START ? marker : "";+ return `${head}${body}${hasMore ? marker : ""}`;+}++/**+ * Convert a UTF-16 offset into `text` to the count of CODE POINTS before+ * it, for a surface that caps its window in code points. Passes+ * {@link NO_MATCH_OFFSET} through unchanged so the sentinel survives the+ * conversion.+ */+export function toCodePointOffset(text: string, utf16Offset: number): number {+ if (utf16Offset <= NO_MATCH_OFFSET) return NO_MATCH_OFFSET;+ // `Array.from` iterates the string by code point, so a surrogate pair+ // counts once — which is the unit the caller's cap is measured in.+ return Array.from(text.slice(0, utf16Offset)).length;+}diff --git a/src/core/search/store.ts b/src/core/search/store.tsindex 34ea3036..5fbe3279 100644--- a/src/core/search/store.ts+++ b/src/core/search/store.ts@@ -780,6 +780,27 @@ export class Store {return census.counts(this.db, this.config.semantic.model, this.config.semantic.dimension);}+ /**+ * Indexed chunks split by what the oversize-chunk census can decide+ * about them against a model's declared input window: provably over,+ * and undecided by the estimate. One aggregate pass, no chunk bodies+ * read; see `store/counts.ts` for the unit and both bounds.+ *+ * `requestPrefix` is the instruction prefix the configured backend+ * prepends to every passage, because the provider counts it too.+ */+ chunkWindowTally(windowTokens: number, requestPrefix: string): census.ChunkWindowTally {+ return census.chunkWindowTally(this.db, windowTokens, requestPrefix);+ }++ /**+ * The provable half of {@link chunkWindowTally} for a backend that+ * prepends nothing.+ */+ chunksOverTokenWindow(windowTokens: number): number {+ return census.chunksOverTokenWindow(this.db, windowTokens);+ }+// ── direct accessors used by indexer/CLI/status ────────────────────────────/** Escape hatch for status queries that don't fit the typed API. */diff --git a/src/core/search/store/chunks.ts b/src/core/search/store/chunks.tsindex b7c02f84..2ac37ed8 100644--- a/src/core/search/store/chunks.ts+++ b/src/core/search/store/chunks.ts@@ -54,6 +54,26 @@ export interface HydratedChunk {* instant, keeping their ordering byte-identical.*/readonly authoredAt?: number | null;+ /**+ * sha256 of the chunk content, written on every index run since schema+ * v1 (`chunks.content_hash`). Projected by {@link hydrateChunks} so the+ * assembly stage can fold byte-identical passages into one result row.+ * ABSENT - not null - on reads that do not project it (the+ * representative-chunk read behind traversal expansion), keeping those+ * rows byte-identical to what they were before the projection existed.+ */+ readonly contentHash?: string;+ /**+ * sha256 of the WHOLE source file of this chunk's document, frontmatter+ * included (`documents.content_hash`, also schema v1). The chunk hash+ * alone cannot tell a copied file from two distinct records that happen+ * to share a body: frontmatter is stripped before chunking, so two notes+ * with the same prose but different `authored_at`, `freshness_trend` or+ * declared dates hash their chunks identically while ranking - correctly -+ * differently. Pairing the two hashes makes the duplicate merge fire only+ * on a genuine copy. Absent under the same rule as `contentHash`.+ */+ readonly documentHash?: string;}export function getChunksByDocument(db: Database, documentId: number): ChunkRow[] {@@ -211,12 +231,15 @@ export function hydrateChunks(end_line: number;mtime: number;authored_at: number | null;+ content_hash: string;+ document_hash: string;},number[]>("SELECT c.id AS chunk_id, c.document_id, d.path AS path, d.title AS title, " +" c.content AS content, c.start_line AS start_line, c.end_line AS end_line, d.mtime AS mtime, " +- " d.authored_at AS authored_at " ++ " d.authored_at AS authored_at, c.content_hash AS content_hash, " ++ " d.content_hash AS document_hash " +"FROM chunks c JOIN documents d ON d.id = c.document_id " +`WHERE c.id IN (${placeholders})`,)@@ -232,6 +255,8 @@ export function hydrateChunks(endLine: r.end_line,mtime: r.mtime,authoredAt: r.authored_at,+ contentHash: r.content_hash,+ documentHash: r.document_hash,});}return out;diff --git a/src/core/search/store/counts.ts b/src/core/search/store/counts.tsindex 9e6bf4b7..e7d17a4d 100644--- a/src/core/search/store/counts.ts+++ b/src/core/search/store/counts.ts@@ -6,9 +6,16 @@import { Database } from "bun:sqlite";+import { SearchError } from "../search-error.ts";import { countChunks } from "./chunks.ts";import { countDocuments } from "./documents.ts";import { countEmbeddings, staleEmbeddings } from "./vectors.ts";+import {+ charLengthOverTokenBudget,+ NON_ASCII_CEILING_COEFFICIENT,+ textExtent,+ utf8ByteFloorUnderTokenBudget,+} from "../embeddings/signature.ts";export interface StoreCounts {readonly documents: number;@@ -30,3 +37,135 @@ export function counts(staleEmbeddings: staleEmbeddings(db, configuredModel, configuredDimension),});}++/** Split of the chunk table by what the census can decide about it. */+export interface ChunkWindowTally {+ /** Chunks whose LOW estimate already exceeds the window. */+ readonly overWindow: number;+ /** Chunks the low estimate clears and the high estimate does not. */+ readonly undecided: number;+}++/** The prefix value that means "this backend prepends nothing". */+export const NO_REQUEST_PREFIX = "";++/**+ * One SQL pass, two aggregates: chunks the census can prove are over+ * `windowTokens`, and chunks it cannot place on either side of it.+ *+ * `requestPrefix` is the text the CONFIGURED BACKEND prepends to every+ * indexed chunk before the provider counts a token - `passage: ` for the+ * recommended e5 default, empty for a backend that implements no+ * instruction-prefix contract. It is measured into both thresholds+ * because the provider counts it: a census that ignored it would report+ * zero for chunks the request path puts over the window by exactly the+ * width of the prefix. It is a required argument for the same reason -+ * an optional one is the drift this parameter exists to close.+ *+ * The unit measured is a token ESTIMATE, never the `chunks.token_count`+ * column: that column is the chunker's whitespace WORD count, a+ * different unit from a model's token window, and comparing the two as+ * though they matched is the arithmetic this census exists to stop.+ *+ * ## Why two aggregates and not one+ *+ * Characters-over-four is a Latin-script heuristic. Two populations+ * break it in the same direction - the estimate reads LOW, so a single+ * count reports a clean census over chunks the provider truncates:+ *+ * - Han text, where a BERT-family tokenizer is roughly+ * character-level. `BAAI/bge-small-zh-v1.5` declares a 512-token+ * window, and a Chinese chunk that estimates at 512 costs about four+ * times that;+ * - supplementary-plane text, where SQLite's `length()` counts one per+ * character and `String.length` counts two, so the database reads+ * half the estimated size and a chunk estimated at twice the window+ * lands under a threshold expressed in code points.+ *+ * Neither is fixable by a tighter constant, and neither is observable in+ * SQL as a script test - bun:sqlite exposes no user-defined-function+ * hook, and materialising every chunk body costs more than the answer is+ * worth. What IS available in SQL is `octet_length()`, and UTF-8 makes+ * the byte count a structural script signal: every non-ASCII code point+ * costs at least one extra byte, so `octet_length - length` bounds the+ * non-ASCII code points exactly. Charging those at+ * `MAX_TOKENS_PER_NON_ASCII_CODE_POINT` gives the high estimate, and the+ * gap between the two is reported as undecided rather than passed.+ *+ * For pure ASCII the two estimates coincide - `octet_length` equals+ * `length`, the band is empty - so an all-Latin index is counted exactly+ * as it was before the high estimate existed.+ *+ * ## Cost+ *+ * Cheaper than the single aggregate it replaced, not dearer. `length()`+ * walks the UTF-8 body and dominates the query; `octet_length()` reads a+ * stored byte count. The `WHERE` clause therefore tests the free+ * measurement first ({@link utf8ByteFloorUnderTokenBudget}) and only the+ * survivors are walked - which on a wide-window model is no rows at all.+ * Measured on a 21 000-chunk / 137 MB fixture, warm: 307 ms before+ * against 316 ms after at a 512-token window, and 288 ms before against+ * 71 ms after at the 8192-token window `BAAI/bge-m3` declares.+ */+export function chunkWindowTally(+ db: Database,+ windowTokens: number,+ requestPrefix: string,+): ChunkWindowTally {+ const prefix = textExtent(requestPrefix);+ const row = db+ .query<+ { over_window: number; undecided: number },+ {+ $budget: number;+ $byteFloor: number;+ $coefficient: number;+ $prefixCodePoints: number;+ $prefixUtf8Bytes: number;+ }+ >(+ // MATERIALIZED, not a plain subquery: `code_points` is read three+ // times below, and without it SQLite re-evaluates `length()` -+ // a full walk of the chunk body - once per reference.+ `WITH extents AS MATERIALIZED (+ SELECT length(content) + $prefixCodePoints AS code_points,+ octet_length(content) + $prefixUtf8Bytes AS utf8_bytes+ FROM chunks+ WHERE octet_length(content) + $prefixUtf8Bytes > $byteFloor+ )+ SELECT+ COUNT(*) FILTER (WHERE code_points > $budget) AS over_window,+ COUNT(*) FILTER (+ WHERE code_points <= $budget+ AND code_points + $coefficient * min(code_points, utf8_bytes - code_points) > $budget+ ) AS undecided+ FROM extents`,+ )+ .get({+ $budget: charLengthOverTokenBudget(windowTokens),+ $byteFloor: utf8ByteFloorUnderTokenBudget(windowTokens),+ $coefficient: NON_ASCII_CEILING_COEFFICIENT,+ $prefixCodePoints: prefix.codePoints,+ $prefixUtf8Bytes: prefix.utf8Bytes,+ });+ if (row === null) {+ // Unreachable while the query is an unfiltered aggregate, and+ // therefore exactly the shape that must not degrade to a zero count:+ // a zero here would be a clean census nobody took.+ throw new SearchError(+ "INDEX_UNREADABLE",+ "chunk-window census: the aggregate over `chunks` returned no row, so no census was " ++ "taken. Run: o2b search check",+ );+ }+ return Object.freeze({ overWindow: row.over_window, undecided: row.undecided });+}++/**+ * The provable half of {@link chunkWindowTally} on its own, for a caller+ * that already knows the backend sends no prefix. Kept as a named+ * projection rather than a second query so the two cannot drift.+ */+export function chunksOverTokenWindow(db: Database, windowTokens: number): number {+ return chunkWindowTally(db, windowTokens, NO_REQUEST_PREFIX).overWindow;+}diff --git a/src/core/search/store/keyword.ts b/src/core/search/store/keyword.tsindex 05642578..d1bab386 100644--- a/src/core/search/store/keyword.ts+++ b/src/core/search/store/keyword.ts@@ -23,6 +23,17 @@ export interface KeywordHit {*/const CHUNK_FTS_BM25 = "bm25(chunk_fts, 1.0, 0.3)";+/**+ * Total ordering for the top-K cut. BM25 alone is not a total order -+ * two chunks with the same term frequencies and the same length score+ * identically - so `LIMIT` would otherwise take whichever tied rows the+ * scan happened to reach first, which is an artefact of the b-tree and+ * not a property of the query. `chunks.id` is `INTEGER PRIMARY KEY`, so+ * it is unique and already present; ending the sort on it makes the cut+ * reproducible without touching the schema.+ */+const CHUNK_ORDER = "ORDER BY bm25 ASC, chunk_id ASC";+export function ensureFts5(db: Database): void {// Probe FTS5 by attempting a benign expression. bun:sqlite ships with// FTS5 enabled in the embedded amalgamation; failing here means a@@ -102,7 +113,7 @@ export function keywordTopK("JOIN chunks c ON c.id = chunk_fts.rowid " +"JOIN documents d ON d.id = c.document_id " +"WHERE chunk_fts MATCH ? AND substr(d.path, 1, length(?)) = ? " +- "ORDER BY bm25 ASC LIMIT ?",+ `${CHUNK_ORDER} LIMIT ?`,).all(fts5Query, prefix, prefix, limit);return rows.map((r) => ({@@ -116,7 +127,7 @@ export function keywordTopK(.query<{ chunk_id: number; document_id: number; bm25: number }, [string, number]>(`SELECT c.id AS chunk_id, c.document_id AS document_id, ${CHUNK_FTS_BM25} AS bm25 ` +"FROM chunk_fts JOIN chunks c ON c.id = chunk_fts.rowid " +- "WHERE chunk_fts MATCH ? ORDER BY bm25 ASC LIMIT ?",+ `WHERE chunk_fts MATCH ? ${CHUNK_ORDER} LIMIT ?`,).all(fts5Query, limit);return rows.map((r) => ({diff --git a/src/core/search/store/lifecycle.ts b/src/core/search/store/lifecycle.tsindex cce7e044..ae9cc293 100644--- a/src/core/search/store/lifecycle.ts+++ b/src/core/search/store/lifecycle.ts@@ -11,10 +11,26 @@ import { Database } from "bun:sqlite";import { existsSync, mkdirSync, renameSync } from "node:fs";import { dirname } from "node:path";-import { applyMigrations, LATEST_SCHEMA_VERSION, readSchemaVersion } from "../schema.ts";+import {+ applyMigrations,+ findSchemaGaps,+ formatSchemaGaps,+ LATEST_SCHEMA_VERSION,+ readSchemaVersion,+ REINDEX_COMMAND,+ type SchemaGap,+} from "../schema.ts";import { SearchError } from "../types.ts";import type { ResolvedSearchConfig } from "../types.ts";import { ensureFts5 } from "./keyword.ts";+import { nowIso } from "./sql.ts";+import {+ deleteState,+ getState,+ INTEGRITY_CHECKED_AT_STATE_KEY,+ INTEGRITY_FAULT_STATE_KEY,+ setState,+} from "./state.ts";import { loadVecExtension } from "./vectors.ts";import { acquireWriterLock, acquireWriterLockSync } from "./writer-lock.ts";@@ -89,6 +105,19 @@ export function restoreFromBakIfMissing(dbPath: string): void {* index path can make readSchemaVersion throw raw SQLITE errors (e.g.* "no such table: index_state"). Surface those as a typed* INDEX_UNREADABLE so callers see a code, not a stray Error.+ *+ * The version integer alone is not the schema. A file stamped with the+ * current version whose `chunk_entities` table or `documents.authored_at`+ * column has gone missing passes the integer comparison and then fails+ * deep inside a query with a raw `no such table` / `no such column`, which+ * names no exit. The presence check closes that: it compares the file+ * against the manifest DERIVED from the migrations (see+ * {@link findSchemaGaps}) and refuses through the SAME `SCHEMA_MISMATCH`+ * that already names the reindex command, because a structurally+ * incomplete index is a schema mismatch by any useful definition.+ *+ * It is metadata-only - a table listing plus one `table_info` per expected+ * table - and costs about a third of a millisecond on the query hot path.*/function assertSchemaIsCurrent(db: Database, dbPath: string): void {let version: number;@@ -102,11 +131,195 @@ function assertSchemaIsCurrent(db: Database, dbPath: string): void {if (version !== LATEST_SCHEMA_VERSION) {throw new SearchError("SCHEMA_MISMATCH",- `index schema version ${version} != ${LATEST_SCHEMA_VERSION}. Run: o2b search reindex`,+ `index schema version ${version} != ${LATEST_SCHEMA_VERSION}. Run: ${REINDEX_COMMAND}`,+ );+ }+ let gaps: ReadonlyArray<SchemaGap>;+ try {+ gaps = findSchemaGaps(db);+ } catch (e) {+ if (e instanceof SearchError) throw e;+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("INDEX_UNREADABLE", `cannot read the schema of ${dbPath}: ${msg}`);+ }+ if (gaps.length > 0) {+ throw new SearchError(+ "SCHEMA_MISMATCH",+ `index at ${dbPath} is stamped schema version ${version} but is missing ` ++ `${formatSchemaGaps(gaps)}. Run: ${REINDEX_COMMAND}`,);}}+// ─────────────────────────────────────────────────────────────────────────────+// The structural integrity gate+// ─────────────────────────────────────────────────────────────────────────────++/** `PRAGMA quick_check`'s verdict text when the file is intact. */+const QUICK_CHECK_OK = "ok";++/**+ * Errors `PRAGMA quick_check` is asked to report before it stops. A+ * refusal needs a cause, not an inventory, and a bounded list keeps a+ * badly damaged file from producing a megabyte of verdict text.+ */+const QUICK_CHECK_MAX_ERRORS = 5;++/**+ * How long a completed full check stands for. The scan is linear in the+ * size of the index and I/O bound, so its rate is a property of the+ * machine, not of this code: between roughly 8 and 22 ms per megabyte on+ * the hosts it has been measured on, which is seconds to tens of seconds+ * for a gigabyte-scale index. That is why it can never run on the read+ * path (that is per query) and why even the write path runs it at most+ * once a day rather than on every open.+ *+ * This interval governs how often the SCAN runs. It says nothing about+ * whether an already-recorded fault is honoured - see+ * {@link runWriteOpenIntegrityGate}.+ */+const INTEGRITY_CHECK_INTERVAL_MS = 24 * 60 * 60 * 1000;++/** The outcome of a completed full structural scan. */+export type IntegrityVerdict =+ | { readonly ok: true }+ | { readonly ok: false; readonly fault: string };++/**+ * Run `PRAGMA quick_check` and classify the result.+ *+ * Three outcomes collapse to "faulty", and deliberately so: a verdict+ * other than `ok`, a scan that raises SQLITE_CORRUPT because the damage+ * stops it completing, and a scan that returns no verdict row at all.+ * None of them is evidence of a healthy file, so none of them may pass.+ */+export function runIntegrityCheck(db: Database): IntegrityVerdict {+ let rows: ReadonlyArray<{ quick_check: string }>;+ try {+ rows = db+ .query<{ quick_check: string }, []>(`PRAGMA quick_check(${QUICK_CHECK_MAX_ERRORS})`)+ .all();+ } catch (e) {+ return { ok: false, fault: e instanceof Error ? e.message : String(e) };+ }+ if (rows.length === 0) return { ok: false, fault: "quick_check returned no verdict" };+ const verdict = rows.map((r) => r.quick_check).join("; ");+ return verdict === QUICK_CHECK_OK ? { ok: true } : { ok: false, fault: verdict };+}++/** The message every refusal on a recorded fault carries. */+function integrityFaultMessage(dbPath: string, fault: string): string {+ return (+ `search index at ${dbPath} failed a structural integrity check: ${fault}. ` ++ `Run: ${REINDEX_COMMAND}`+ );+}++/**+ * The refusal every open owes a condemned file: consult the verdict a+ * previous FULL check recorded, and refuse while it stands.+ *+ * No open can afford to run the scan unconditionally - a read open happens+ * on every single query and the scan costs seconds - so what every open+ * can honestly do is refuse to touch a file that a completed check found+ * malformed. That costs one key lookup and is therefore affordable+ * everywhere, which is why it is applied everywhere rather than only on+ * the read path. The absence of a fault is NOT a pass and is never+ * reported as one: it means either that the last check passed or that no+ * check has run here yet, and `integrity_checked_at` is the only thing+ * that distinguishes those two.+ */+function assertNoRecordedIntegrityFault(db: Database, dbPath: string): void {+ const fault = getState(db, INTEGRITY_FAULT_STATE_KEY);+ if (fault === null) return;+ throw new SearchError("INDEX_UNREADABLE", integrityFaultMessage(dbPath, fault));+}++/**+ * Whether a fresh full scan is due: true unless a completed scan is+ * recorded within {@link INTEGRITY_CHECK_INTERVAL_MS}.+ *+ * A stamp that is absent, unparseable, or in the FUTURE (a clock that+ * moved backwards) all mean the same thing: nothing here can be relied on+ * to say when the last scan happened, so scan.+ */+function fullScanIsDue(db: Database): boolean {+ const stamp = getState(db, INTEGRITY_CHECKED_AT_STATE_KEY);+ const ageMs = stamp === null ? Number.NaN : Date.now() - Date.parse(stamp);+ return !(Number.isFinite(ageMs) && ageMs >= 0 && ageMs < INTEGRITY_CHECK_INTERVAL_MS);+}++/**+ * The write open's integrity gate: run the scan when one is due, and+ * otherwise honour the verdict the last one recorded - BOTH of its arms.+ *+ * That second clause is the whole point, and it used to be missing. The+ * recorded fault is a cached verdict and the interval is its lifetime; the+ * bug was applying that lifetime to the PASSING arm only. A completed scan+ * that condemns the file stamps `integrity_checked_at` exactly as a+ * passing one does, so an interval that short-circuited before anything+ * consulted `integrity_fault` skipped the gate entirely for the next 24+ * hours on precisely the files that had just been condemned - and+ * `o2b search check --integrity` produces that state on demand, so running+ * the diagnostic on a damaged index opened the window rather than closing+ * it. Within the interval the verdict now stands as a whole: a pass means+ * skip the scan, a fault means refuse, as it always has on the read path.+ *+ * A DUE scan still re-derives the verdict rather than trusting the cached+ * one, which is what lets a fault the operator has since repaired be+ * cleared instead of becoming permanent.+ *+ * The scan itself runs here and nowhere else. This is the one seam+ * where a multi-second scan is already noise - an index run reads, chunks+ * and often embeds the whole vault - and it is also the moment it matters+ * most, because everything after it WRITES into the file. The verdict is+ * recorded in `index_state`, which needs no migration, so the read path+ * can act on it for the price of one key lookup.+ *+ * The interval is the cost control for the scan alone: a burst of index+ * runs pays for one scan, not one per open. The price of that is stated+ * plainly - corruption arising inside the window is not noticed until the+ * window closes. Paying for a scan on every write open instead would put+ * tens of seconds in front of every `o2b brain tiers` invocation, and+ * paying it on the read path would put it in front of every query.+ *+ * The refusal leaves exactly one exit, and it is not blocked by this gate:+ * `reindexVault` builds into a `.new` staging file and swaps it in, so it+ * never opens the condemned file for writing. A condemned index therefore+ * stays repairable by `o2b search reindex` - and by the `openReadOrSelfHeal`+ * rebuild the query path performs for itself - which is what makes+ * refusing every other write open safe.+ *+ * Recording is best effort - a file damaged badly enough may refuse the+ * write - but the REFUSAL is not: the fault is thrown either way, naming+ * its cause and its exit.+ */+function runWriteOpenIntegrityGate(db: Database, dbPath: string): void {+ if (!fullScanIsDue(db)) {+ // No scan this open - so the last one's verdict is what stands, and a+ // condemning verdict must refuse here just as it does on a read open.+ assertNoRecordedIntegrityFault(db, dbPath);+ return;+ }++ const verdict = runIntegrityCheck(db);+ try {+ setState(db, INTEGRITY_CHECKED_AT_STATE_KEY, nowIso());+ if (verdict.ok) deleteState(db, INTEGRITY_FAULT_STATE_KEY);+ else setState(db, INTEGRITY_FAULT_STATE_KEY, verdict.fault);+ } catch (e) {+ if (verdict.ok) throw e;+ // The file is malformed AND will not accept the record of it. The+ // refusal below still names the cause; only the memo is lost.+ const msg = e instanceof Error ? e.message : String(e);+ // eslint-disable-next-line no-console+ console.error(`search store: could not record the integrity fault in ${dbPath}: ${msg}`);+ }+ if (!verdict.ok) {+ throw new SearchError("INDEX_UNREADABLE", integrityFaultMessage(dbPath, verdict.fault));+ }+}+/*** Open the index for reading: no lock, no migration. An index at an* older schema is refused rather than upgraded, because a reader holds@@ -130,10 +343,18 @@ export function openReadDatabase(config: ResolvedSearchConfig, loadVec: boolean)applyPragmas(db);ensureFts5(db);assertSchemaIsCurrent(db, config.dbPath);+ assertNoRecordedIntegrityFault(db, config.dbPath);return { db, vecVersion: loadVec ? loadVecExtension(db) : null };} catch (e) {db.close();- throw e;+ if (e instanceof SearchError) throw e;+ // `new Database()` above is lazy: a file that is not a SQLite database+ // at all only raises on the FIRST statement, which is `applyPragmas`.+ // Untyped, that error escaped the self-heal path this module's callers+ // rely on, so the very case the read path documents itself as healing -+ // a non-OSB file at the index path - was the one it did not heal.+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("INDEX_UNREADABLE", `cannot open ${config.dbPath}: ${msg}`);}}@@ -166,6 +387,7 @@ export async function openWriteDatabase(try {applyPragmas(db);applyMigrations(db, { ftsTokenize: config.ftsTokenize });+ runWriteOpenIntegrityGate(db, config.dbPath);ensureFts5(db);return { db, vecVersion: loadVec ? loadVecExtension(db) : null, release };} catch (e) {diff --git a/src/core/search/store/state.ts b/src/core/search/store/state.tsindex 7c9a5cd7..76eff4ea 100644--- a/src/core/search/store/state.ts+++ b/src/core/search/store/state.ts@@ -54,6 +54,24 @@ export const EMBEDDING_VEC_VERSION_STATE_KEY = "embedding_vec_version";export const LAST_INDEXED_AT_STATE_KEY = "last_indexed_at";export const LAST_FULL_INDEX_AT_STATE_KEY = "last_full_index_at";+/**+ * `index_state` keys behind the store-integrity gate (what-the-index-+ * already-knew, unit K).+ *+ * `integrity_checked_at` is the ISO instant a FULL `PRAGMA quick_check`+ * last COMPLETED on this file - written whether it passed or failed, and+ * never written by anything else. Absent therefore means "no full check+ * has ever run here", which is a different claim from "it passed", and+ * nothing reads it as a pass.+ *+ * `integrity_fault` carries the check's verdict text and exists ONLY while+ * the last completed check found the file malformed. A passing check+ * deletes it. Its presence is what a read open refuses on, because the+ * scan that produced it is far too expensive to repeat per query.+ */+export const INTEGRITY_CHECKED_AT_STATE_KEY = "integrity_checked_at";+export const INTEGRITY_FAULT_STATE_KEY = "integrity_fault";+const SELECT_STATE_SQL = "SELECT value FROM index_state WHERE key = ?";export function getState(db: Database, key: string): string | null {diff --git a/src/core/search/store/trigram.ts b/src/core/search/store/trigram.tsindex d746b379..793aa1d9 100644--- a/src/core/search/store/trigram.ts+++ b/src/core/search/store/trigram.ts@@ -20,6 +20,15 @@ export interface TrigramCandidateOutcome {const NO_WARNINGS: ReadonlyArray<string> = Object.freeze([]);+/**+ * Total ordering for the top-K cut, for the reason the keyword lane's+ * own `CHUNK_ORDER` gives: BM25 ties are common in a trigram index -+ * every chunk sharing the same trigram profile scores alike - and+ * `LIMIT` over a partial order picks by scan artefact. `chunks.id` is+ * `INTEGER PRIMARY KEY`, so no schema change is involved.+ */+const CHUNK_ORDER = "ORDER BY bm25 ASC, chunk_id ASC";+/*** The one condition this lane stays SILENT for: `chunk_trigram` itself is* absent, which is a store migrated to v9 but never rebuilt - the table@@ -96,7 +105,7 @@ export function trigramCandidates("JOIN chunks c ON c.id = chunk_trigram.rowid " +"JOIN documents d ON d.id = c.document_id " +"WHERE chunk_trigram MATCH ? AND substr(d.path, 1, length(?)) = ? " +- "ORDER BY bm25 ASC LIMIT ?",+ `${CHUNK_ORDER} LIMIT ?`,).all(trigramQuery, prefix, prefix, limit);return {@@ -108,7 +117,7 @@ export function trigramCandidates(.query<{ chunk_id: number; document_id: number; bm25: number }, [string, number]>("SELECT c.id AS chunk_id, c.document_id AS document_id, bm25(chunk_trigram) AS bm25 " +"FROM chunk_trigram JOIN chunks c ON c.id = chunk_trigram.rowid " +- "WHERE chunk_trigram MATCH ? ORDER BY bm25 ASC LIMIT ?",+ `WHERE chunk_trigram MATCH ? ${CHUNK_ORDER} LIMIT ?`,).all(trigramQuery, limit);return {diff --git a/src/core/search/store/vectors.ts b/src/core/search/store/vectors.tsindex 3fccabb0..a3c46f31 100644--- a/src/core/search/store/vectors.ts+++ b/src/core/search/store/vectors.ts@@ -360,6 +360,34 @@ export function ensureEmbeddingModel(});}+/**+ * Total ordering for the rows the KNN returns. Distance ties are not+ * exotic here - two chunks with identical text embed to the same vector+ * and therefore to the same distance - and the caller truncates, so a+ * partial order lets the scan decide which tied neighbour survives.+ * `chunk_id` is `chunks.id`, an `INTEGER PRIMARY KEY`, so the extra sort+ * key needs no schema change.+ */+const VEC_ORDER = "ORDER BY v.distance ASC, m.chunk_id ASC";++/**+ * How much wider than `limit` the `k` of the KNN is asked for.+ *+ * `ORDER BY` alone cannot make the cut deterministic: sqlite-vec selects+ * its `k` nearest rows FIRST, by its own internal order, and the sort+ * only ever reaches the rows that selection already kept. Asking for a+ * wider `k` and cutting here means a group of equidistant neighbours+ * straddling the boundary is resolved by the unique sort key rather than+ * by vec's traversal. The scan is a full one with a k-sized heap either+ * way, so the wider k costs heap, not passes.+ *+ * The prefixed branch has always widened by this factor for the+ * different reason that its path predicate drops rows after the KNN; the+ * two branches now share one constant instead of one of them carrying a+ * bare literal.+ */+const VEC_KNN_OVERFETCH = 4;+export function semanticTopK(db: Database,vecLoaded: boolean,@@ -373,6 +401,7 @@ export function semanticTopK();}const limit = Math.max(1, opts.limit | 0);+ const knn = limit * VEC_KNN_OVERFETCH;const prefix = opts.pathPrefix && opts.pathPrefix.length > 0 ? opts.pathPrefix : null;assertValidVector(queryVector, "semanticTopK");@@ -389,9 +418,9 @@ export function semanticTopK("JOIN chunks c ON c.id = m.chunk_id " +"JOIN documents d ON d.id = c.document_id " +"WHERE v.embedding MATCH ? AND k = ? AND substr(d.path, 1, length(?)) = ? " +- "ORDER BY v.distance ASC",+ VEC_ORDER,)- .all(buf, limit * 4, prefix, prefix);+ .all(buf, knn, prefix, prefix);return rows.slice(0, limit).map((r) => ({chunkId: r.chunk_id,documentId: r.document_id,@@ -406,10 +435,10 @@ export function semanticTopK("JOIN chunk_vec_map m ON m.vec_rowid = v.rowid " +"JOIN chunks c ON c.id = m.chunk_id " +"WHERE v.embedding MATCH ? AND k = ? " +- "ORDER BY v.distance ASC",+ VEC_ORDER,)- .all(buf, limit);- return rows.map((r) => ({+ .all(buf, knn);+ return rows.slice(0, limit).map((r) => ({chunkId: r.chunk_id,documentId: r.document_id,distance: r.distance,diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex 0486bcfc..9992e978 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -114,6 +114,112 @@ export interface QueryPlan {readonly temporalIntent?: TemporalIntent;}+/**+ * Result of the oversize-chunk census: what the configured chunk size+ * actually produced, measured against the configured embedding model's+ * DECLARED input window.+ *+ * The census owns a token ESTIMATE, not a tokenizer, so it answers with a+ * two-sided bound and three reportable states rather than a boolean:+ *+ * - `over-window` - the low estimate already exceeds the window, so+ * the request is over it whatever the real tokenizer does;+ * - `estimate-undecided` - the low estimate fits and the high estimate+ * does not. Characters-over-four is a Latin-script rule of thumb; on+ * Han text a BERT-family tokenizer is roughly character-level, and on+ * supplementary-plane text SQLite and JavaScript disagree about what+ * a "character" is. In that band the census cannot tell, and says so;+ * - `window-undeclared` - the model is outside the curated preset+ * table, so there is no window to compare against at all.+ *+ * "The census ran and every chunk fits under BOTH bounds" is the ABSENCE+ * of this value - a check with nothing to report has nothing to say, and+ * a field that is present-and-zero is a field every consumer has to+ * interpret. Absence is only ever produced when the bound the code can+ * actually back was cleared; the two states it cannot back are the two+ * verdicts above, which is what stops silence from meaning a pass the+ * estimate never earned.+ *+ * The census is not taken at all when there is no model whose window+ * could be exceeded - semantic search disabled, or the offline local+ * embedder, which hashes features over the whole text and has no input+ * window by construction.+ */+export type ChunkWindowCensus =+ | {+ /** Chunks were measured and some exceed the declared window. */+ readonly verdict: "over-window";+ /** The active embedding model the window was declared for. */+ readonly model: string;+ /** The model's declared window, in the model's own tokens. */+ readonly windowTokens: number;+ /** Chunks whose LOW estimate exceeds `windowTokens`. */+ readonly chunksOverWindow: number;+ /**+ * Chunks in the undecided band beside them. Absent, never zero,+ * when the estimate decided every remaining chunk - an all-ASCII+ * index has no band, and its record carries the bytes it carried+ * before this field existed.+ */+ readonly chunksUndecided?: number;+ /** Chunks the census looked at. */+ readonly chunksMeasured: number;+ }+ | {+ /**+ * Every chunk clears the low estimate, but some do not clear the+ * high one: the census measured them and cannot say.+ */+ readonly verdict: "estimate-undecided";+ /** The active embedding model the window was declared for. */+ readonly model: string;+ /** The model's declared window, in the model's own tokens. */+ readonly windowTokens: number;+ /** Chunks the estimate cannot place on either side of the window. */+ readonly chunksUndecided: number;+ /** Chunks the census looked at. */+ readonly chunksMeasured: number;+ }+ | {+ /** No window is declared for this model, so nothing was measured. */+ readonly verdict: "window-undeclared";+ /** The active model, or null when none is configured. */+ readonly model: string | null;+ /** Chunks the census WOULD have looked at. */+ readonly chunksMeasured: number;+ };++/**+ * The census member that reports a check which could not run. Narrowed+ * out of {@link ChunkWindowCensus} rather than redeclared, so the two+ * surfaces that carry it cannot drift on its fields.+ */+export type ChunkWindowUndeclared = Extract<ChunkWindowCensus, { verdict: "window-undeclared" }>;++/**+ * Registered diagnostic codes for the two census verdicts, resolved+ * through `core/brain/next-step.ts`. Declared beside the verdict they+ * belong to so the core surface that formats the warning and the CLI+ * surface that serializes the record cannot drift on which code names+ * which state.+ */+export const CHUNK_WINDOW_CODE = Object.freeze({+ "over-window": "search-chunk-window-overflow",+ // The same exit as measured overflow, and deliberately: the two states+ // differ in what the census KNOWS, not in what an operator does about+ // it. Chunks the estimate cannot place inside the window are fixed by+ // the one lever that fixes chunks known to be outside it - lower+ // `search_chunk_size` and rebuild - so pointing somewhere else would+ // invent a second remedy for one cause.+ "estimate-undecided": "search-chunk-window-overflow",+ "window-undeclared": "search-chunk-window-undeclared",+} as const);++/** The registered diagnostic code for a census verdict. */+export function chunkWindowDiagnosticCode(census: ChunkWindowCensus): string {+ return CHUNK_WINDOW_CODE[census.verdict];+}+export interface IndexStats {readonly added: number;readonly updated: number;@@ -207,6 +313,17 @@ export interface IndexStats {* backend ran (`backend === "semantic"`).*/readonly deferredReason: string | null;+ /**+ * Oversize-chunk census over the index this run left behind. Present+ * ONLY when it has something to report: chunks that exceed the+ * configured model's declared input window, chunks the estimate cannot+ * place inside it, or a model for which no window is declared and the+ * check therefore did not run. Absent when the census cleared BOTH of+ * its bounds, and absent when there is no model whose window could be+ * exceeded - so a run that has nothing to say about it emits the bytes+ * it emitted before this field existed.+ */+ readonly chunkWindow?: ChunkWindowCensus;readonly durationMs: number;}@@ -261,6 +378,26 @@ export interface IndexStatusSnapshot {* emitted only when there is drift.*/readonly embeddingAbi: ReadonlyArray<StampMismatch>;+ /**+ * The oversize-chunk census's UNMEASURABLE state: the configured+ * embedding model is outside the curated preset table, so it declares+ * no input window and no chunk could be compared against one. Present+ * only in that state, which makes this surface distinguish four things+ * a single boolean could not - measured clean under both bounds+ * (absent, and no warning), measured overflow and measured-undecidable+ * (absent here, one line each in {@link warnings}), and not measurable+ * at all (this field).+ *+ * Deliberately NOT a warning, unlike the overflow it sits beside. The+ * convention this snapshot already follows is that the structured+ * field says WHAT is true and the warning says what to run; there is+ * no command that declares a window for someone else's model, and a+ * line repeated on every status call for the majority of real+ * configurations is what teaches operators to stop reading warnings.+ * Reporting it as a field keeps the statement - nothing here claims+ * the check passed - without spending the warning channel on it.+ */+ readonly chunkWindowUndeclared?: ChunkWindowUndeclared;readonly warnings: ReadonlyArray<string>;}@@ -330,6 +467,22 @@ export interface SearchCard {readonly pointer: string;/** Cross-vault origin label, mirrored from the source result when set. */readonly origin?: string;+ /**+ * Locations of the exact duplicates folded into this row+ * (what-the-index-already-knew, task D), as `path:Lstart-Lend`+ * pointers in the same grammar as {@link SearchCard.pointer}.+ *+ * Pointers rather than the full {@link DuplicatePassageLocation}+ * records the ranked row carries: a folded passage is byte-identical to+ * the surviving one, so the caller never needs to expand a copy — it+ * needs to know where else the same bytes live, and a card exists to be+ * small.+ *+ * Present ONLY when at least one byte-identical passage was merged+ * away, so a corpus that holds no duplicate passages produces cards+ * byte-identical to pre-merge ones. Never empty when present.+ */+ readonly duplicatePointers?: ReadonlyArray<string>;}export interface ExpandHitInput {@@ -657,12 +810,30 @@ export interface SearchOptions {export interface SearchOutcome {readonly results: ReadonlyArray<BrainSearchResult>;readonly warnings: ReadonlyArray<string>;+ /**+ * Size of the ranked candidate pool the `limit` window was sliced out+ * of (what-the-index-already-knew, task F) - NOT the number of returned+ * rows, which is `results.length` (or `cards.length`). It answers the+ * one question the row count cannot: how many candidates were ranked+ * and then dropped. Always >= the returned row count, since those rows+ * are the pool's prefix; equal to it only when the pool fit inside the+ * limit. Zero when nothing matched.+ *+ * It is the pool retrieval actually ranked, not a corpus COUNT: the+ * lanes fetch a widened but bounded candidate set, so on a large vault+ * this is a lower bound on the number of matching chunks, never an+ * over-count.+ *+ * The pool is the post-rank set, so gate-excluded candidates are not+ * counted here - they are reported, with their reasons, on+ * {@link SearchOutcome.retrievalDecisionTrace}.+ */readonly total: number;/*** Layer-1 compact cards (progressive disclosure). Present only when the* caller set `disclosure: "cards"`; in that mode `results` is empty and- * `total` counts the cards. Absent on the default `full` path, keeping- * the legacy outcome shape byte-identical.+ * the cards carry the surfaced rows. Absent on the default `full` path,+ * keeping the legacy outcome shape byte-identical.*/readonly cards?: ReadonlyArray<SearchCard>;readonly evidencePack?: EvidencePack;@@ -724,9 +895,19 @@ export interface SearchOutcome {readonly surface?: QuerySurface;}+/**+ * The closed set of embedding backends `makeProvider` can build.+ *+ * Named rather than written inline because a second reader now has to+ * exhaust it: the oversize-chunk census asks which backend prepends an+ * instruction prefix to the text it sends, and that question has to fail+ * to COMPILE when a backend is added rather than silently answer "none".+ */+export type EmbeddingProviderName = "openai-compat" | "disabled" | "local" | "zeroentropy";+export interface ResolvedEmbeddingConfig {readonly enabled: boolean;- readonly provider: "openai-compat" | "disabled" | "local" | "zeroentropy";+ readonly provider: EmbeddingProviderName;readonly baseUrl: string | null;readonly model: string | null;readonly apiKey: string | null;diff --git a/src/mcp/brain/query-tools.ts b/src/mcp/brain/query-tools.tsindex 390d7829..c153b231 100644--- a/src/mcp/brain/query-tools.ts+++ b/src/mcp/brain/query-tools.ts@@ -17,6 +17,7 @@ import {queryByPreference,queryByTopic,} from "../../core/brain/query.ts";+import { parseIsoUtc } from "../../core/brain/health/iso-time.ts";import { diffAgentSources, type AgentSourceDiffMode } from "../../core/brain/agent-source/diff.ts";import { queryAgentSources } from "../../core/brain/agent-source/query.ts";import type { AgentSourceContributionKind } from "../../core/brain/agent-source/types.ts";@@ -44,6 +45,49 @@ import { loadGuardrailsConfigSafe } from "../../core/brain/policy.ts";import { normalizeAgentScope } from "../../core/graph/agent-scope.ts";import { isPreferenceVisible } from "../../core/brain/owner-scoped-facts.ts";+/** Accepted `at` forms, named in every refusal so the exit is actionable. */+const AS_OF_FORMS = "an ISO-8601 instant or YYYY-MM-DD date";++/**+ * The as-of instant the expiration filter is evaluated against.+ *+ * `queryByTopic` has taken a `now` option since C5 and this handler+ * dropped it, so an agent could ask to SEE lapsed memories+ * (`show_expired`) but never to ask what the brain held at an instant.+ * Only topic mode runs that filter, so an `at` supplied beside+ * `preference` / `since` is refused rather than quietly ignored, and an+ * unparseable one is refused rather than coerced to the wall clock -+ * which would answer a different question without saying so.+ */+const EXPIRY_ARG_SCOPE = "only applies to topic mode (the expiration filter runs there)";++/**+ * `show_expired` shares `at`'s scope: both steer the one expiration+ * filter, and that filter runs in topic mode alone. Its schema already+ * said so while the handler read it in topic mode and ignored it+ * everywhere else, so a caller asking to see lapsed memories beside+ * `preference` got the documented answer's opposite in silence. Two+ * arguments with one scope now refuse on the same terms.+ */+function assertExpiryArgScope(args: Record<string, unknown>, topicMode: boolean): void {+ if (topicMode) return;+ if (coerceBoolOptional(args, "show_expired") === undefined) return;+ throw new MCPError(INVALID_PARAMS, `brain_query: 'show_expired' ${EXPIRY_ARG_SCOPE}`);+}++function coerceAsOf(args: Record<string, unknown>, topicMode: boolean): Date | null {+ const raw = coerceStr(args, "at", false);+ if (raw === null) return null;+ if (!topicMode) {+ throw new MCPError(INVALID_PARAMS, `brain_query: 'at' ${EXPIRY_ARG_SCOPE}`);+ }+ const atMs = parseIsoUtc(raw);+ if (!Number.isFinite(atMs)) {+ throw new MCPError(INVALID_PARAMS, `brain_query: 'at' must be ${AS_OF_FORMS}; got ${raw}`);+ }+ return new Date(atMs);+}+async function toolBrainQuery(ctx: ServerContext,args: Record<string, unknown>,@@ -71,6 +115,9 @@ async function toolBrainQuery();}+ assertExpiryArgScope(args, topic !== null);+ const asOf = coerceAsOf(args, topic !== null);+// Recall telemetry (t_405b8053): per-call opt-in mirroring// brain_search. The payload carries the query KIND only - never the// supplied preference id / topic slug / timestamp value.@@ -137,9 +184,13 @@ async function toolBrainQuery(if (topic !== null) {// Expiration opt-in (C5): default drops memories past their- // expiration_date; show_expired re-includes them for audit.+ // expiration_date; show_expired re-includes them for audit, and+ // `at` moves the clock that filter compares against.const showExpired = coerceBoolOptional(args, "show_expired") ?? false;- const res = queryByTopic(ctx.vault, topic, { showExpired });+ const res = queryByTopic(ctx.vault, topic, {+ showExpired,+ ...(asOf !== null ? { now: asOf } : {}),+ });const resultCount = res.signals.length + res.all_log_events.length;emitQueryTelemetry(resultCount > 0 ? "ok" : "empty", resultCount);const topicPrefOwner = res.preference?.owner;@@ -492,6 +543,11 @@ export const QUERY_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([description:"Topic mode only: include memories past their `expiration_date`. Default false (expired memories are silently dropped from the result).",},+ at: {+ type: "string",+ description:+ "Topic mode only: evaluate `expiration_date` as of this instant, so a memory that lapsed after it comes back. ISO-8601 instant or YYYY-MM-DD. Default now.",+ },since: {type: "string",description: "ISO-8601 timestamp; returns every Brain log event with timestamp >= since.",diff --git a/src/mcp/search-tools.ts b/src/mcp/search-tools.tsindex 50f9eb47..865b9974 100644--- a/src/mcp/search-tools.ts+++ b/src/mcp/search-tools.ts@@ -46,11 +46,26 @@ import {coerceStringOptional,} from "./coerce.ts";import { MCP_PREVIEW_BUDGET } from "./preview-budget.ts";+import { explainEnvelope } from "../core/search/explain-envelope.ts";import { deriveRecallHint } from "../core/search/recall-hint.ts";+import {+ ELLIPSIS,+ HEAD_WINDOW_START,+ alignToCodePointStart,+ markWindow,+ matchOffset,+ windowStartWithin,+} from "../core/search/snippet-window.ts";import { projectScoreBreakdown } from "../core/search/enrich.ts";import { recordReinforce } from "../core/search/reinforce.ts";import { parseRecallBenchmarkDataset, runRecallBenchmark } from "../core/search/benchmark.ts";-import { emitRecallTelemetry } from "../core/brain/recall-telemetry.ts";+import {+ deriveRecallSignals,+ emitRecallTelemetry,+ RECALL_SIGNALS_UNMEASURED_CARDS,+ type RecallQualitySignals,+ type RecallSignalsUnmeasured,+} from "../core/brain/recall-telemetry.ts";import { emitGateTelemetry } from "../core/brain/gate-telemetry.ts";import { emitGatedTelemetry } from "../core/brain/continuity/emit.ts";import { recordQueryDemand, recordRecallAdequacyDemand } from "../core/brain/query-demand.ts";@@ -58,6 +73,8 @@ import { recordQueryDemand, recordRecallAdequacyDemand } from "../core/brain/queconst MCP_LIMIT_MAX = 50;const MCP_CONTENT_MAX = 600;const SEARCH_TIMEOUT_MS = 10_000;+/** Surfaced rows named as source refs on one recall-telemetry record. */+const TELEMETRY_TOP_ARTIFACTS_MAX = 10;const SEARCH_INPUT_SCHEMA: Record<string, unknown> = {type: "object",@@ -82,13 +99,13 @@ const SEARCH_INPUT_SCHEMA: Record<string, unknown> = {type: "string",maxLength: 64,description:- "Time-aware recall: only documents modified at/after this point. ISO date/datetime, 'today', 'yesterday', 'last week', 'last month', or <n>h/<n>d/<n>w.",+ "Hard filter on event time (validity, body anchor, mtime last): at/after this point. ISO date/datetime, today, yesterday, last week, last month, <n>h/<n>d/<n>w.",},until: {type: "string",maxLength: 64,description:- "Time-aware recall: only documents modified at/before this point. Same forms as 'since'.",+ "Hard filter on event time (validity, body anchor, mtime last): at/before this point. Same forms as 'since'.",},limit: { type: "integer", minimum: 1, maximum: MCP_LIMIT_MAX },semantic: { type: "boolean" },@@ -108,7 +125,7 @@ const SEARCH_INPUT_SCHEMA: Record<string, unknown> = {explain: {type: "boolean",description:- "Include a structured score_breakdown (per-layer numeric components) on each result. Default false.",+ "Add a per-result score_breakdown plus the retrieval_decision_trace and memory_trust_assessment receipts. Default false.",},trust: {type: "boolean",@@ -258,6 +275,27 @@ const SEARCH_OUTPUT_SCHEMA: NonNullable<ToolDefinition["outputSchema"]> = {conflict: { type: "boolean" },},},+ // Exact-duplicate merge: locations folded into this row.+ // Declared deliberately — this output schema does not set+ // `additionalProperties: false`, so an undeclared field would+ // pass validation silently and be invisible to a caller+ // reading the contract.+ duplicates: {+ type: "array",+ items: {+ type: "object",+ required: ["documentId", "chunkId", "path", "title", "startLine", "endLine"],+ properties: {+ documentId: { type: "integer" },+ chunkId: { type: "integer" },+ path: { type: "string" },+ // Nullable, same as the result title above.+ title: {},+ startLine: { type: "integer" },+ endLine: { type: "integer" },+ },+ },+ },},},},@@ -286,13 +324,63 @@ const SEARCH_OUTPUT_SCHEMA: NonNullable<ToolDefinition["outputSchema"]> = {document_id: { type: "integer" },chunk_id: { type: "integer" },origin: { type: "string" },+ // Exact-duplicate merge: the `path:Lstart-Lend` pointers this+ // card was folded from. Declared deliberately for the same+ // reason `duplicates` is on the full row above - this output+ // schema does not set `additionalProperties: false`, so an+ // undeclared field would validate silently and stay invisible+ // to a caller reading the contract.+ duplicate_pointers: { type: "array", items: { type: "string" } },},},},warnings: { type: "array", items: { type: "string" } },+ // The ranked pool the `limit` sliced these rows from (task F) - how+ // many candidates were ranked, not how many came back. The descriptor+ // grammar carries no prose, so the meaning is documented here, in+ // SearchOutcome.total, and in docs/mcp.md.total: { type: "integer" },recall_hint: { type: "string" },evidence_pack: { type: "object" },+ // Retrieval receipts under `explain` (what-the-index-already-knew,+ // task F). Declared deliberately: this output schema does not set+ // `additionalProperties: false`, so an undeclared key would pass+ // validation silently and stay invisible to a caller reading the+ // contract.+ retrieval_decision_trace: {+ type: "object",+ required: ["evaluated", "surfaced", "excluded", "exclusions"],+ properties: {+ evaluated: { type: "integer" },+ surfaced: { type: "integer" },+ excluded: { type: "integer" },+ exclusions: {+ type: "array",+ items: {+ type: "object",+ required: ["document_id", "chunk_id", "path", "reasons"],+ properties: {+ document_id: { type: "integer" },+ chunk_id: { type: "integer" },+ path: { type: "string" },+ reasons: { type: "array", items: { type: "string" } },+ },+ },+ },+ },+ },+ memory_trust_assessment: {+ type: "object",+ required: ["evaluated", "surfaced", "quarantined", "reason_counts"],+ properties: {+ evaluated: { type: "integer" },+ surfaced: { type: "integer" },+ quarantined: { type: "integer" },+ // Open-keyed histogram: one integer per namespaced exclusion reason.+ reason_counts: { type: "object" },+ },+ },+ retrieval_trace_unavailable: { type: "string" },telemetry_id: { type: "string" },},};@@ -442,9 +530,37 @@ function parseReinforceArgument(raw: unknown): string[] | undefined {return out;}-function truncateContent(c: string, max: number): string {- if (c.length <= max) return c;- return c.slice(0, max - 1) + "…";+/**+ * The `content` window for one MCP result: at most `max` characters+ * INCLUDING its continuation markers, centred on the first significant+ * query term the chunk contains (task E).+ *+ * With no occurrence the window is the head of the chunk and the bytes+ * are exactly what the pre-anchoring head truncation produced: content+ * at or under the cap passes through untouched, and content over it is+ * cut one character short to make room for the trailing marker.+ *+ * Content at or under the cap is a case of the general rule rather than an+ * early return of its own: {@link windowStartWithin} opens it at the head,+ * the budget is the whole cap, `hasMore` is false, and the slice is the+ * string itself. Stating it once means the "never truncate text the cap+ * had room for" invariant cannot drift between this surface and the two+ * that measure their caps in other units.+ *+ * The start is aligned to a code-point boundary because this surface+ * slices UTF-16 units: an anchored start can land on the low half of a+ * surrogate pair, and a body opening with a lone surrogate becomes U+FFFD+ * as soon as the JSON-RPC frame is encoded as UTF-8. The TRAILING cut is+ * deliberately left as it is - it has always cut on a raw offset, its+ * bytes are the pre-anchoring contract this wave must not move, and+ * unlike the head it is not a hazard anchoring introduced.+ */+function windowContent(c: string, query: string, max: number): string {+ const start = alignToCodePointStart(c, windowStartWithin(matchOffset(c, query), max, c.length));+ const budget = max - (start > HEAD_WINDOW_START ? ELLIPSIS.length : 0);+ const hasMore = start + budget < c.length;+ const body = c.slice(start, start + budget - (hasMore ? ELLIPSIS.length : 0));+ return markWindow(body, start, hasMore, ELLIPSIS);}export function searchErrorToMcp(e: SearchError): MCPError {@@ -648,8 +764,13 @@ async function toolBrainSearch(// `results`; both shapes carry documentId/chunkId/path/score, so the// telemetry count/status/top-artifacts stay honest either way.const surfaced = outcome.cards ?? outcome.results;- const telemetryRecord = emitGatedTelemetry(telemetry || undefined, () =>- emitRecallTelemetry(ctx.vault, {+ const telemetryRecord = emitGatedTelemetry(telemetry || undefined, () => {+ // Recall signals (what-the-index-already-knew, task G): derived+ // INSIDE the gated thunk, so with the gate off - the default - not a+ // single comparison runs. `search()` has already returned and its+ // outcome and rows are frozen, so this lane can only read them.+ const signals = searchRecallSignals(outcome);+ return emitRecallTelemetry(ctx.vault, {host: telemetryHost,...(telemetrySessionId !== undefined ? { sessionId: telemetrySessionId } : {}),...(telemetryTurnId !== undefined ? { turnId: telemetryTurnId } : {}),@@ -657,7 +778,7 @@ async function toolBrainSearch(status: surfaced.length > 0 ? "ok" : "empty",durationMs: Date.now() - startedAtMs,resultCount: surfaced.length,- topArtifacts: surfaced.slice(0, 10).map((result) => ({+ topArtifacts: surfaced.slice(0, TELEMETRY_TOP_ARTIFACTS_MAX).map((result) => ({id: `${result.documentId}:${result.chunkId}`,path: result.path,score: result.score,@@ -672,8 +793,9 @@ async function toolBrainSearch(warnings_count: outcome.warnings.length,...(pathPrefix !== undefined ? { path_prefix: pathPrefix } : {}),},- }),- );+ ...(signals !== null ? { signals } : {}),+ });+ });// Cross-query demand log (t_97091fff): persist the normalized query// terms, result count, and (when the evidence pack computed it) the// IDF-weighted coverage so recurring poorly-answered queries can be@@ -690,7 +812,7 @@ async function toolBrainSearch(results: outcome.results.map((r: BrainSearchResult) => ({path: r.path,title: r.title,- content: truncateContent(r.content, MCP_CONTENT_MAX),+ content: windowContent(r.content, query, MCP_CONTENT_MAX),score: r.score,startLine: r.startLine,endLine: r.endLine,@@ -704,6 +826,9 @@ async function toolBrainSearch(...(r.origin !== undefined ? { origin: r.origin } : {}),...(outcome.evidencePack ? { why_retrieved: r.reasons } : {}),...(r.relations && r.relations.length > 0 ? { relations: r.relations } : {}),+ // Exact-duplicate merge (task D): the locations this row was folded+ // from. Absent, never null, when nothing was merged.+ ...(r.duplicates && r.duplicates.length > 0 ? { duplicates: r.duplicates } : {}),})),...(outcome.cards ? { cards: outcome.cards.map(serializeSearchCard) } : {}),warnings: outcome.warnings,@@ -715,10 +840,36 @@ async function toolBrainSearch(// Absent on the generic path, so the default response stays// byte-identical....(outcome.surface !== undefined ? { surface: outcome.surface } : {}),+ // Retrieval receipts (what-the-index-already-knew, task F): the+ // decision trace and the trust assessment every gated search already+ // built and no surface serialized. Under `explain` only, absent -+ // never null - otherwise; when the gate is off, one line naming the+ // switch that produces them rather than an unexplained silence.+ ...explainEnvelope(outcome, {+ explain,+ crossVault: globalSearch,+ trustGateEnabled: config.recall.retrievalTrustGateEnabled,+ }),...(telemetryRecord ? { telemetry_id: telemetryRecord.id } : {}),};}+/**+ * The recall signals for one query, or `null` when the window is empty+ * and there is nothing to measure - `result_count: 0` already says that.+ *+ * Under `disclosure: "cards"` the pipeline returns projected cards and an+ * EMPTY `results` array, so none of the per-row values the signals are+ * derived from exist; that path records the named unmeasured marker+ * rather than an all-zero block that would read as a measurement.+ */+function searchRecallSignals(+ outcome: SearchOutcome,+): RecallQualitySignals | RecallSignalsUnmeasured | null {+ if (outcome.cards !== undefined) return RECALL_SIGNALS_UNMEASURED_CARDS;+ return deriveRecallSignals(outcome.results);+}+function searchTelemetryGaps(outcome: SearchOutcome): ReadonlyArray<string> {const gaps = new Set<string>();if (outcome.total === 0) gaps.add("no_matching_context");diff --git a/tests/cli/brain-query-as-of.test.ts b/tests/cli/brain-query-as-of.test.tsnew file mode 100644index 00000000..45a8cdae--- /dev/null+++ b/tests/cli/brain-query-as-of.test.ts@@ -0,0 +1,205 @@+/**+ * `o2b brain query` point-in-time recall (what-the-index-already-knew,+ * task J).+ *+ * The expiry filter has taken an as-of instant since C5 - `filterExpired`+ * defaults `now` to the wall clock and `queryByTopic` threads a `now`+ * option straight into it - and no surface ever passed one. The CLI+ * called `queryByTopic(vault, topic)` with no options object at all, so+ * "what did the brain hold as of last month" was unaskable from the one+ * surface that enforces expiry, and `--show-expired` (which MCP has had+ * since C5) did not exist on the CLI either.+ *+ * The byte-identity pins below are the EXACT stdout the pre-change binary+ * produced for the same fixture, captured before the flags existed. They+ * are literals rather than a re-serialised comparison so a change in key+ * order, indentation or trailing newline fails here.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { writeSignal, type WriteSignalInput } from "../../src/core/brain/signal.ts";+import { runCli } from "../helpers/run-cli.ts";++const TOPIC = "deploy";+/** The lapsed signal's expiration; every probe instant is chosen around it. */+const LAPSED_EXPIRY = "2026-07-15";+/** An instant BEFORE the expiry: the lapsed signal was still live then. */+const BEFORE_EXPIRY = "2026-06-01T00:00:00Z";+/** An instant AFTER the expiry, still deterministic against any wall clock. */+const AFTER_EXPIRY = "2026-08-01T00:00:00Z";++const LAPSED_PRINCIPLE = "Principle for lapsed";+const EVERGREEN_PRINCIPLE = "Principle for evergreen";++/** Exact pre-change stdout for `--topic deploy --json`, no new flags. */+const PINNED_JSON_STDOUT = `{+ "signals": [+ {+ "kind": "brain-signal",+ "id": "sig-2026-05-02-evergreen",+ "created_at": "2026-05-02T00:00:00Z",+ "tags": [+ "brain",+ "brain/signal",+ "brain/topic/deploy"+ ],+ "topic": "deploy",+ "signal": "positive",+ "agent": "tester",+ "principle": "Principle for evergreen"+ }+ ],+ "preference": null,+ "all_log_events": []+}+`;++/** Exact pre-change stdout for `--topic deploy` on the human surface. */+const PINNED_TEXT_STDOUT =+ "topic: deploy\n" ++ "preference: (none)\n" ++ "signals: 1\n" ++ " - sig-2026-05-02-evergreen (positive, 2026-05-02T00:00:00Z)\n" ++ "log_events: 0\n";++let tmp: string;+let vault: string;+let config: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-brain-query-as-of-"));+ vault = join(tmp, "vault");+ config = join(tmp, "config.yaml");+ writeFileSync(config, `vault: ${vault}\nagent_name: claude\n`);+ bootstrapBrain(vault, { configPath: config });+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function signal(slug: string, overrides: Partial<WriteSignalInput> = {}): WriteSignalInput {+ return {+ topic: TOPIC,+ signal: "positive",+ agent: "tester",+ principle: `Principle for ${slug}`,+ created_at: "2026-05-01T00:00:00Z",+ date: "2026-05-01",+ slug,+ ...overrides,+ };+}++/** One signal that has lapsed, one that never expires. */+function seed(): void {+ writeSignal(vault, signal("lapsed", { expiration_date: LAPSED_EXPIRY }));+ writeSignal(+ vault,+ signal("evergreen", { created_at: "2026-05-02T00:00:00Z", date: "2026-05-02" }),+ );+}++async function query(...args: string[]) {+ return runCli(["brain", "query", "--vault", vault, "--topic", TOPIC, ...args], {+ env: { OPEN_SECOND_BRAIN_CONFIG: config },+ });+}++function principles(stdout: string): string[] {+ const payload = JSON.parse(stdout) as { signals: ReadonlyArray<{ principle: string }> };+ return payload.signals.map((s) => s.principle).toSorted();+}++describe("brain query --at", () => {+ test("recalls what the brain held at an instant before the memory lapsed", async () => {+ seed();+ const r = await query("--json", "--at", BEFORE_EXPIRY);+ expect(r.stderr).toBe("");+ expect(r.returncode).toBe(0);+ expect(principles(r.stdout)).toEqual([EVERGREEN_PRINCIPLE, LAPSED_PRINCIPLE]);+ });++ test("an instant after the expiry drops the lapsed memory again", async () => {+ seed();+ const r = await query("--json", "--at", AFTER_EXPIRY);+ expect(r.returncode).toBe(0);+ expect(principles(r.stdout)).toEqual([EVERGREEN_PRINCIPLE]);+ });++ test("a date-only instant is accepted and read as UTC midnight", async () => {+ seed();+ const r = await query("--json", "--at", "2026-06-01");+ expect(r.returncode).toBe(0);+ expect(principles(r.stdout)).toEqual([EVERGREEN_PRINCIPLE, LAPSED_PRINCIPLE]);+ });++ test("an unparseable instant is refused by name, never coerced to now", async () => {+ seed();+ const r = await query("--json", "--at", "last month");+ expect(r.returncode).toBe(2);+ expect(r.stderr).toContain("--at");+ expect(r.stderr).toContain("ISO-8601 instant or YYYY-MM-DD date");+ expect(r.stderr).toContain("last month");+ // A refused query answers nothing; it must not fall through to today.+ expect(r.stdout).toBe("");+ });++ test("--at outside topic mode is refused rather than silently ignored", async () => {+ seed();+ const r = await runCli(+ ["brain", "query", "--vault", vault, "--since", BEFORE_EXPIRY, "--at", BEFORE_EXPIRY],+ { env: { OPEN_SECOND_BRAIN_CONFIG: config } },+ );+ expect(r.returncode).toBe(2);+ expect(r.stderr).toContain("--topic");+ });+});++describe("brain query --show-expired", () => {+ test("keeps a lapsed memory in the result", async () => {+ seed();+ const r = await query("--json", "--show-expired");+ expect(r.returncode).toBe(0);+ expect(principles(r.stdout)).toEqual([EVERGREEN_PRINCIPLE, LAPSED_PRINCIPLE]);+ });++ test("is refused outside topic mode", async () => {+ seed();+ const r = await runCli(+ ["brain", "query", "--vault", vault, "--since", BEFORE_EXPIRY, "--show-expired"],+ { env: { OPEN_SECOND_BRAIN_CONFIG: config } },+ );+ expect(r.returncode).toBe(2);+ expect(r.stderr).toContain("--topic");+ });+});++describe("byte identity when both flags are absent", () => {+ test("--json stdout is byte-for-byte the pre-change output", async () => {+ seed();+ const r = await query("--json");+ expect(r.returncode).toBe(0);+ expect(r.stdout).toBe(PINNED_JSON_STDOUT);+ });++ test("the human surface is byte-for-byte the pre-change output", async () => {+ seed();+ const r = await query();+ expect(r.returncode).toBe(0);+ expect(r.stdout).toBe(PINNED_TEXT_STDOUT);+ });++ test("the pins are not vacuous: either flag changes those exact bytes", async () => {+ seed();+ const withAt = await query("--json", "--at", BEFORE_EXPIRY);+ const withShowExpired = await query("--json", "--show-expired");+ expect(withAt.stdout).not.toBe(PINNED_JSON_STDOUT);+ expect(withShowExpired.stdout).not.toBe(PINNED_JSON_STDOUT);+ });+});diff --git a/tests/cli/brain-repair-lane-holdout-gate.test.ts b/tests/cli/brain-repair-lane-holdout-gate.test.tsnew file mode 100644index 00000000..1eca0fb4--- /dev/null+++ b/tests/cli/brain-repair-lane-holdout-gate.test.ts@@ -0,0 +1,183 @@+/**+ * The graph-efficacy holdout gate on `o2b brain repair-lane --apply`+ * (what-the-index-already-knew, final unit).+ *+ * `CHANGELOG.md` has shipped the claim since v1.36.0: "the paired holdout+ * harness measures graph lift separately from direct recall and fails the+ * gate on any dangling or unhydrated target". The harness existed; no+ * production caller reached it, so no operator could ever meet the gate.+ * These tests are what makes the shipped sentence true.+ *+ * The gate rides on the verb's existing write flag. Dry-run is the verb's+ * preview mode and stays byte-identical; `--apply` plans first, evaluates+ * the harness over the edges that plan proposes, and refuses to write when+ * the harness reports a target that resolves to nothing or hydrates to no+ * evidence.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { buildContinuityRecord } from "../../src/core/brain/continuity/store.ts";+import { DIAGNOSTIC_SIGNALS } from "../../src/core/brain/diagnostics.ts";+import { runCli } from "../helpers/run-cli.ts";++/** The code the gate's refusal resolves its exit through. */+const HOLDOUT_GATE_CODE = "repair-holdout-unresolved";++/** The month shard the fixture continuity record lands in. */+const FIXTURE_CONTINUITY_MONTH = "2026-06";+const FIXTURE_CONTINUITY_AT = `${FIXTURE_CONTINUITY_MONTH}-13T12:00:00Z`;++let tmp: string;+let vault: string;+let config: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-repair-holdout-cli-"));+ vault = join(tmp, "vault");+ mkdirSync(join(vault, "Brain"), { recursive: true });+ mkdirSync(join(vault, "Notes"), { recursive: true });+ config = join(tmp, "config.yaml");+ writeFileSync(config, `vault: "${vault}"\n`);+ writeNote("Notes/alpha.md", "Alpha", "This note discusses Beta at length.");+ writeNote("Notes/beta.md", "Beta", "standalone");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function writeNote(rel: string, title: string, body: string): void {+ writeFileSync(+ join(vault, rel),+ ["---", "kind: brain-note", `title: ${title}`, "---", "", body, ""].join("\n"),+ "utf8",+ );+}++/**+ * Record two notes as co-referenced in one session event. Written through+ * the store's own builder so the fixture cannot drift from the record shape+ * the reader parses.+ */+function writeContinuityPair(left: string, right: string): void {+ const record = buildContinuityRecord({+ kind: "recall_telemetry",+ createdAt: FIXTURE_CONTINUITY_AT,+ sourceRefs: [+ { id: "left", path: left },+ { id: "right", path: right },+ ],+ payload: { host: "test" },+ });+ const dir = join(vault, "Brain", "log", "continuity");+ mkdirSync(dir, { recursive: true });+ writeFileSync(join(dir, `${FIXTURE_CONTINUITY_MONTH}.jsonl`), `${JSON.stringify(record)}\n`);+}++const env = () => ({ OPEN_SECOND_BRAIN_CONFIG: config });++const applyArgs = ["brain", "repair-lane", "--apply", "--confirm", "apply repair", "--json"];++interface RefusalEnvelope {+ readonly ok: boolean;+ readonly message: string;+ readonly next_command?: string;+ readonly holdout?: Record<string, number | boolean>;+}++test("an apply whose proposed edges dangle is refused and writes nothing", async () => {+ // A session event co-references a live note and one that no longer+ // exists: the lane proposes that edge, and its target resolves to no+ // durable memory at all.+ writeContinuityPair("Notes/beta.md", "Notes/ghost.md");+ const before = readFileSync(join(vault, "Notes/alpha.md"), "utf8");++ const res = await runCli(applyArgs, { env: env() });++ expect(res.returncode).toBe(1);+ const envelope = JSON.parse(res.stdout) as RefusalEnvelope;+ expect(envelope.ok).toBe(false);+ expect(envelope.holdout?.["passed"]).toBe(false);+ expect(envelope.holdout?.["dangling"]).toBeGreaterThan(0);+ expect(envelope.message).toContain("Notes/ghost.md");+ // The refusal names its exit through the registered diagnostic.+ expect(envelope.next_command).toBe(DIAGNOSTIC_SIGNALS.get(HOLDOUT_GATE_CODE)!.nextCommand);+ // Nothing was written, including the edge that would have passed.+ expect(readFileSync(join(vault, "Notes/alpha.md"), "utf8")).toBe(before);+});++test("an apply whose proposed edge hydrates to no evidence is refused", async () => {+ // The target resolves to durable memory but carries no body, so the edge+ // cannot lift recall even though it is not dangling.+ writeNote("Notes/gamma.md", "Gamma", "");+ writeNote("Notes/delta.md", "Delta", "This note discusses Gamma at length.");+ const before = readFileSync(join(vault, "Notes/delta.md"), "utf8");++ const res = await runCli(applyArgs, { env: env() });++ expect(res.returncode).toBe(1);+ const envelope = JSON.parse(res.stdout) as RefusalEnvelope;+ expect(envelope.holdout?.["dangling"]).toBe(0);+ expect(envelope.holdout?.["unhydrated"]).toBeGreaterThan(0);+ expect(readFileSync(join(vault, "Notes/delta.md"), "utf8")).toBe(before);+});++test("a clean apply still writes the edge and reports the lift split", async () => {+ const res = await runCli(applyArgs, { env: env() });++ expect(res.returncode).toBe(0);+ const report = JSON.parse(res.stdout) as {+ written: number;+ holdout: Record<string, number | boolean>;+ };+ expect(report.written).toBeGreaterThan(0);+ expect(report.holdout["passed"]).toBe(true);+ expect(report.holdout["dangling"]).toBe(0);+ expect(report.holdout["unhydrated"]).toBe(0);+ // Direct recall and graph lift are counted apart, which is the whole+ // point of the paired harness.+ expect(Object.keys(report.holdout).toSorted()).toEqual([+ "dangling",+ "direct_recall",+ "graph_lift",+ "passed",+ "resolved",+ "total",+ "unhydrated",+ ]);+ expect(readFileSync(join(vault, "Notes/alpha.md"), "utf8")).toContain("beta");+});++test("without --apply the report bytes carry no gate at all", async () => {+ // Byte identity when the feature is off: the gate is evaluated on the+ // write branch only, so the preview a caller already parses is unchanged+ // even in a vault whose candidates would fail the gate.+ writeContinuityPair("Notes/beta.md", "Notes/ghost.md");++ const res = await runCli(["brain", "repair-lane", "--json"], { env: env() });++ expect(res.returncode).toBe(0);+ expect(res.stdout).not.toContain("holdout");+ expect(Object.keys(JSON.parse(res.stdout) as Record<string, unknown>)).toEqual([+ "ok",+ "mode",+ "written",+ "decisions",+ ]);+});++test("the human refusal names the failing edge and the command that resolves it", async () => {+ writeContinuityPair("Notes/beta.md", "Notes/ghost.md");++ const res = await runCli(["brain", "repair-lane", "--apply", "--confirm", "apply repair"], {+ env: env(),+ });++ expect(res.returncode).not.toBe(0);+ expect(res.stderr).toContain("Notes/ghost.md");+ expect(res.stderr).toContain(DIAGNOSTIC_SIGNALS.get(HOLDOUT_GATE_CODE)!.nextCommand);+});diff --git a/tests/cli/search-card-duplicates.test.ts b/tests/cli/search-card-duplicates.test.tsnew file mode 100644index 00000000..90bbad03--- /dev/null+++ b/tests/cli/search-card-duplicates.test.ts@@ -0,0 +1,96 @@+/**+ * `o2b search --cards` surfaces the merged-away duplicate locations+ * (what-the-index-already-knew, task D follow-up).+ *+ * The full-disclosure transcript and `--json` payload both name every+ * location a byte-identical passage was folded from. The cards projection+ * named none: the value was computed end-to-end and dropped at the card+ * boundary, so the token-cheap layer said "one hit here" about a passage+ * that lives at three paths.+ *+ * Absent, not null, when nothing was merged - in both projections.+ */++import { describe, expect, test } from "bun:test";++import { jsonForOutcome, renderOutcomeHuman } from "../../src/cli/search/outcome-render.ts";+import { toSearchCard } from "../../src/core/search/cards.ts";+import type { DuplicatePassageLocation } from "../../src/core/search/search-result.ts";+import type { BrainSearchResult, SearchOutcome } from "../../src/core/search/types.ts";++const QUERY = "quantum entanglement";++const FOLDED: ReadonlyArray<DuplicatePassageLocation> = Object.freeze([+ Object.freeze({+ documentId: 7,+ chunkId: 42,+ path: "archive/entanglement-copy.md",+ title: "Entanglement (copy)",+ startLine: 3,+ endLine: 9,+ }),+ Object.freeze({+ documentId: 8,+ chunkId: 43,+ path: "inbox/entanglement-again.md",+ title: null,+ startLine: 1,+ endLine: 1,+ }),+]);++const EXPECTED_POINTERS = ["archive/entanglement-copy.md:L3-L9", "inbox/entanglement-again.md:L1"];++function resultWith(duplicates?: ReadonlyArray<DuplicatePassageLocation>): BrainSearchResult {+ return Object.freeze({+ documentId: 1,+ chunkId: 1,+ path: "physics/entanglement.md",+ title: "Entanglement",+ content: "Quantum entanglement links two particles at a distance.",+ startLine: 1,+ endLine: 9,+ score: 1.5,+ keywordScore: 1.5,+ semanticScore: 0,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "keyword" as const,+ reasons: Object.freeze(["keyword: 1.50"]),+ ...(duplicates !== undefined ? { duplicates: Object.freeze(duplicates) } : {}),+ });+}++function cardsOutcome(duplicates?: ReadonlyArray<DuplicatePassageLocation>): SearchOutcome {+ return Object.freeze({+ results: Object.freeze([]),+ cards: Object.freeze([toSearchCard(resultWith(duplicates), QUERY)]),+ warnings: Object.freeze([]),+ total: 1,+ });+}++function jsonCards(outcome: SearchOutcome): Array<Record<string, unknown>> {+ return (jsonForOutcome(outcome) as { cards: Array<Record<string, unknown>> }).cards;+}++describe("search card duplicates on the CLI", () => {+ test("--json names every folded location on the surviving card", () => {+ expect(jsonCards(cardsOutcome(FOLDED))[0]!["duplicate_pointers"]).toEqual(EXPECTED_POINTERS);+ });++ test("--json omits the key entirely when nothing was merged", () => {+ const card = jsonCards(cardsOutcome())[0]!;+ expect("duplicate_pointers" in card).toBe(false);+ expect(JSON.stringify(card)).not.toContain("duplicate");+ });++ test("the cards transcript names the folded locations", () => {+ const rendered = renderOutcomeHuman(cardsOutcome(FOLDED), false, QUERY);+ expect(rendered.split("\n")).toContain(` duplicates: ${EXPECTED_POINTERS.join(", ")}`);+ });++ test("the cards transcript renders no duplicates line when nothing was merged", () => {+ expect(renderOutcomeHuman(cardsOutcome(), false, QUERY)).not.toContain("duplicates:");+ });+});diff --git a/tests/cli/search-check-integrity.test.ts b/tests/cli/search-check-integrity.test.tsnew file mode 100644index 00000000..bcdc4aee--- /dev/null+++ b/tests/cli/search-check-integrity.test.ts@@ -0,0 +1,416 @@+/**+ * `o2b search check --integrity` (what-the-index-already-knew, unit K,+ * reader-side half).+ *+ * The write-open integrity gate closes the corruption hole for anyone who+ * runs an index command. It closes nothing for a PURE reader: no write+ * open ever happens, no `PRAGMA quick_check` ever runs, and the fault key+ * the read open refuses on is therefore never written. Since the index+ * file lives inside the vault and vaults are replicated between machines,+ * that reader is a real person with a real rotting index.+ *+ * This flag is the operator-run scan that closes it. It is opt-in because+ * the scan is linear in the size of the index (~22 ms per megabyte, so+ * half a minute on a large one) and `search check` is advertised as a+ * cheap pre-flight in onboarding, in the semantic-search hint and in two+ * registered diagnostic exits.+ *+ * Three states must stay distinguishable, and the file asserts each:+ * NEVER CHECKED (no completed scan has ever run here), CLEAN (the last+ * completed scan passed) and FAULTY. The absence of a fault is not a+ * pass, and this verb must never report it as one.+ *+ * Nothing here reaches the network: the config switches semantic search+ * off, so no provider is constructed and no ping is sent.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { Database } from "bun:sqlite";+import { closeSync, mkdirSync, openSync, statSync, writeSync } from "node:fs";+import { dirname, join } from "node:path";++import { DIAGNOSTIC_SIGNALS } from "../../src/core/brain/diagnostics.ts";+import { indexVault } from "../../src/core/search/indexer.ts";+import { Store } from "../../src/core/search/store.ts";+import {+ INTEGRITY_CHECKED_AT_STATE_KEY,+ INTEGRITY_FAULT_STATE_KEY,+} from "../../src/core/search/store/state.ts";+import { createTempVault, makeConfig, writeMd } from "../helpers/search-fixtures.ts";+import { runCli } from "../helpers/run-cli.ts";++/** The registered exit a structural fault names. */+const INDEX_CORRUPT_CODE = "search-index-corrupt";++let vault: string;+let dbPath: string;+let configPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("search-check-integrity");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+ configPath = join(vault, "cli-config.yaml");+});++afterEach(() => {+ cleanup();+});++/** A config that keeps `indexCheck` entirely offline. */+async function writeConfig(): Promise<void> {+ await Bun.write(+ configPath,+ [`vault: "${vault}"`, "search_semantic_enabled: false", ""].join("\n"),+ );+}++/** A vault whose index has data pages past its middle. */+function seedVault(notes: number): void {+ for (let i = 0; i < notes; i++) {+ const body = `The quick brown fox jumps over the lazy dog number ${i}. `.repeat(40);+ writeMd(vault, `Notes/note-${i}.md`, `# Note ${i}\n\n${body}`);+ }+}++/**+ * Overwrite whole pages in the middle of the index file with garbage. The+ * header, the schema and the small metadata tables live on the first pages+ * and stay intact, so the file remains fully parseable - which is exactly+ * the state the store used to keep answering questions from.+ */+function corruptMiddlePages(path: string, pages: number): void {+ const size = statSync(path).size;+ const pageSize = readPageSize(path);+ corruptPagesAt(path, Math.floor(size / 2 / pageSize) * pageSize, pages);+}++/**+ * Overwrite the b-tree root page of one named table, located through+ * `sqlite_master` rather than guessed at. Page 1 carries the file header+ * and the schema, so the file still opens and still lists its tables -+ * only reads of THAT table raise. Deterministic, unlike relying on where+ * the middle of a particular fixture happens to fall.+ */+function corruptTableRoot(path: string, table: string): void {+ const db = new Database(path);+ let rootPage: number;+ try {+ rootPage =+ db+ .query<{ rootpage: number }, [string]>("SELECT rootpage FROM sqlite_master WHERE name = ?")+ .get(table)?.rootpage ?? 0;+ } finally {+ db.close();+ }+ expect(`${table} has a root page: ${rootPage > 1}`).toBe(`${table} has a root page: true`);+ const pageSize = readPageSize(path);+ corruptPagesAt(path, (rootPage - 1) * pageSize, 1);+}++function readPageSize(path: string): number {+ const db = new Database(path);+ try {+ return (db.query<{ page_size: number }, []>("PRAGMA page_size").get() as { page_size: number })+ .page_size;+ } finally {+ db.close();+ }+}++function corruptPagesAt(path: string, offset: number, pages: number): void {+ const pageSize = readPageSize(path);+ const fd = openSync(path, "r+");+ try {+ writeSync(fd, Buffer.alloc(pageSize * pages, 0xa5), 0, pageSize * pages, offset);+ } finally {+ closeSync(fd);+ }+}++/** Forget that any full scan ever ran here, without touching the fault key. */+function forgetCheckStamp(path: string): void {+ const db = new Database(path);+ db.run("DELETE FROM index_state WHERE key = ?", [INTEGRITY_CHECKED_AT_STATE_KEY]);+ db.close();+}++/** One `index_state` cell, read without opening a `Store`. */+function readState(path: string, key: string): string | null {+ const db = new Database(path, { readonly: true });+ try {+ return (+ db.query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?").get(key)+ ?.value ?? null+ );+ } finally {+ db.close();+ }+}++/** Run the verb, returning the parsed `--json` payload. */+async function checkJson(...extra: ReadonlyArray<string>): Promise<Record<string, unknown>> {+ const r = await runCli(+ [+ "search",+ "check",+ "--vault",+ vault,+ "--db",+ dbPath,+ "--config",+ configPath,+ "--json",+ ...extra,+ ],+ { env: { OPEN_SECOND_BRAIN_CONFIG: configPath } },+ );+ return { ...(JSON.parse(r.stdout) as Record<string, unknown>), _returncode: r.returncode };+}++/** The `integrity` sub-object of a `--integrity` payload. */+function integrityOf(payload: Record<string, unknown>): Record<string, unknown> {+ const block = payload["integrity"];+ expect(typeof block).toBe("object");+ return block as Record<string, unknown>;+}++test("a store that has never been checked reads as never checked, not as healthy", async () => {+ await writeConfig();+ seedVault(4);+ await indexVault(makeConfig({ vault, dbPath }));+ // The index run itself scanned a fresh, healthy file. Forget that, so+ // this is the pure reader's store: present, fine, never verified.+ forgetCheckStamp(dbPath);+ expect(readState(dbPath, INTEGRITY_CHECKED_AT_STATE_KEY)).toBeNull();++ const payload = await checkJson("--integrity");+ const integrity = integrityOf(payload);++ expect(integrity["scanned"]).toBe(true);+ // What the store knew BEFORE this run: nothing.+ expect(integrity["previously_checked_at"]).toBeNull();+ expect(integrity["verdict"]).toBe("ok");+ expect(payload["_returncode"]).toBe(0);++ // ...and now it knows, which is the whole point: the reader's store has+ // a completed scan recorded against it for the first time.+ expect(readState(dbPath, INTEGRITY_CHECKED_AT_STATE_KEY)).toBe(integrity["checked_at"] as string);+});++test("a clean store reports a pass, the time it spent, and records the completed scan", async () => {+ await writeConfig();+ seedVault(4);+ await indexVault(makeConfig({ vault, dbPath }));+ const before = readState(dbPath, INTEGRITY_CHECKED_AT_STATE_KEY);+ expect(before).toBeTruthy();++ const payload = await checkJson("--integrity");+ const integrity = integrityOf(payload);++ expect(integrity["scanned"]).toBe(true);+ expect(integrity["verdict"]).toBe("ok");+ expect(integrity["previously_checked_at"]).toBe(before);+ expect(integrity["fault"]).toBeUndefined();+ expect(integrity["next_command"]).toBeUndefined();+ expect(typeof integrity["elapsed_ms"]).toBe("number");+ expect(payload["_returncode"]).toBe(0);++ // The interval gate is bypassed on demand: a scan that ran inside the+ // 24-hour window still ran, and still re-stamped.+ expect(readState(dbPath, INTEGRITY_CHECKED_AT_STATE_KEY)).not.toBe(before);+ expect(readState(dbPath, INTEGRITY_FAULT_STATE_KEY)).toBeNull();+});++test("a corrupt-but-parseable store is condemned and names its registered exit", async () => {+ await writeConfig();+ seedVault(60);+ await indexVault(makeConfig({ vault, dbPath }));+ corruptMiddlePages(dbPath, 4);++ // The fixture's point: nothing about this file LOOKS wrong, and the+ // interval gate would skip the scan for another day.+ const probe = new Database(dbPath);+ expect(+ probe.query<{ n: number }, []>("SELECT count(*) AS n FROM chunks").get()?.n,+ ).toBeGreaterThan(0);+ probe.close();++ const payload = await checkJson("--integrity");+ const integrity = integrityOf(payload);++ expect(integrity["scanned"]).toBe(true);+ expect(integrity["verdict"]).toBe("faulty");+ expect(typeof integrity["fault"]).toBe("string");+ expect((integrity["fault"] as string).length).toBeGreaterThan(0);+ expect(integrity["next_command"]).toBe(DIAGNOSTIC_SIGNALS.get(INDEX_CORRUPT_CODE)!.nextCommand);+ expect(payload["_returncode"]).toBe(1);++ // The verdict lands in the SAME cell the write-open gate writes, so the+ // read path now refuses instead of answering from a malformed file -+ // which is the protection a pure reader never used to get.+ expect(readState(dbPath, INTEGRITY_FAULT_STATE_KEY)).toBeTruthy();+ await expect(Store.open(makeConfig({ vault, dbPath }), { mode: "read" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });+});++test("a store this verb condemned is refused by a WRITE open, not only by a read", async () => {+ await writeConfig();+ seedVault(60);+ await indexVault(makeConfig({ vault, dbPath }));+ corruptMiddlePages(dbPath, 4);++ const integrity = integrityOf(await checkJson("--integrity"));+ expect(integrity["verdict"]).toBe("faulty");+ expect(readState(dbPath, INTEGRITY_FAULT_STATE_KEY)).toBeTruthy();++ // A completed scan stamps `integrity_checked_at` whether it passed or+ // failed, so the write path's 24-hour interval now covers the whole next+ // day. The recorded fault must be honoured regardless of that interval:+ // running this diagnostic on a damaged index must not open a window in+ // which the indexer writes into the file the verb just condemned.+ expect(readState(dbPath, INTEGRITY_CHECKED_AT_STATE_KEY)).toBe(integrity["checked_at"] as string);+ await expect(Store.open(makeConfig({ vault, dbPath }), { mode: "write" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });+});++test("the registered exit exists and names a rebuild, not an incremental update", () => {+ const signal = DIAGNOSTIC_SIGNALS.get(INDEX_CORRUPT_CODE);+ expect(signal).toBeDefined();+ // `o2b search index` is incremental and would leave the damaged pages+ // exactly where they are; only a rebuild replaces the file.+ expect(signal!.nextCommand).toBe("o2b search reindex");+ expect(signal!.autoRepairable).toBe(false);+});++test("the human surface names the scan, its cost and its verdict", async () => {+ await writeConfig();+ seedVault(4);+ await indexVault(makeConfig({ vault, dbPath }));++ const r = await runCli(+ ["search", "check", "--vault", vault, "--db", dbPath, "--config", configPath, "--integrity"],+ { env: { OPEN_SECOND_BRAIN_CONFIG: configPath } },+ );+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain("integrity:");+ expect(r.stdout).toContain("ok");+ // The operator is told what the command is about to spend BEFORE it+ // spends it, on stderr so the machine stream stays a payload.+ expect(r.stderr).toContain("integrity scan");+});++test("without --integrity the verb spends nothing and emits the bytes it always did", async () => {+ await writeConfig();+ seedVault(4);+ await indexVault(makeConfig({ vault, dbPath }));+ forgetCheckStamp(dbPath);++ const plainBefore = await runCli(+ ["search", "check", "--vault", vault, "--db", dbPath, "--config", configPath, "--json"],+ { env: { OPEN_SECOND_BRAIN_CONFIG: configPath } },+ );+ expect(plainBefore.returncode).toBe(0);+ const payload = JSON.parse(plainBefore.stdout) as Record<string, unknown>;+ // Byte-identity when absent, asserted as the exact key sequence the+ // report carried before this flag existed - NOT a sorted set. Sorting+ // the keys discards the one property the bytes actually have: a+ // reordered emission would serialize different bytes and a set+ // comparison would call it identical.+ expect(Object.keys(payload)).toEqual([+ "vault_readable",+ "index_dir_writable",+ "sqlite_ok",+ "fts5_ok",+ "vec_extension",+ "embedding_key_resolved",+ "provider_reachable",+ "provider_reason",+ "warnings",+ "fatal",+ "recommendations",+ ]);+ // The flag-less run touched no `index_state` cell: a store nobody+ // checked stays a store nobody checked.+ expect(readState(dbPath, INTEGRITY_CHECKED_AT_STATE_KEY)).toBeNull();++ // ...and a scan in between does not leak into the flag-less bytes.+ await checkJson("--integrity");+ const plainAfter = await runCli(+ ["search", "check", "--vault", vault, "--db", dbPath, "--config", configPath, "--json"],+ { env: { OPEN_SECOND_BRAIN_CONFIG: configPath } },+ );+ expect(plainAfter.stdout).toBe(plainBefore.stdout);+});++test("a passing scan clears a stale fault the operator has already repaired", async () => {+ await writeConfig();+ seedVault(4);+ await indexVault(makeConfig({ vault, dbPath }));++ const stamper = new Database(dbPath);+ stamper.run("INSERT INTO index_state(key, value, updated_at) VALUES (?,?,?)", [+ INTEGRITY_FAULT_STATE_KEY,+ "stale fault from a previous generation",+ new Date().toISOString(),+ ]);+ stamper.close();++ const integrity = integrityOf(await checkJson("--integrity"));+ expect(integrity["verdict"]).toBe("ok");+ expect(readState(dbPath, INTEGRITY_FAULT_STATE_KEY)).toBeNull();+});++test("a store with no index file reports that the scan could not run", async () => {+ await writeConfig();+ writeMd(vault, "Notes/foo.md", "# Foo\n\nfox");++ const payload = await checkJson("--integrity");+ const integrity = integrityOf(payload);++ // Not a pass and not a fault: the check did not run, and says so.+ expect(integrity["scanned"]).toBe(false);+ expect(integrity["verdict"]).toBe("not-run");+ expect(integrity["next_command"]).toBe(+ DIAGNOSTIC_SIGNALS.get("search-index-missing")!.nextCommand,+ );+});++test("a well-formed sqlite file that is not a search index is never reported as healthy", async () => {+ await writeConfig();+ // `PRAGMA quick_check` passes this file happily: it is a perfectly+ // valid database. It is just not this tool's index, and a pass over it+ // would be the misleading verdict the whole unit exists to prevent.+ mkdirSync(dirname(dbPath), { recursive: true });+ const foreign = new Database(dbPath);+ foreign.exec("CREATE TABLE unrelated(id INTEGER PRIMARY KEY, note TEXT)");+ foreign.run("INSERT INTO unrelated(note) VALUES ('not an index')");+ foreign.close();++ const integrity = integrityOf(await checkJson("--integrity"));+ expect(integrity["scanned"]).toBe(false);+ expect(integrity["verdict"]).toBe("not-run");+ expect(integrity["next_command"]).toBe(DIAGNOSTIC_SIGNALS.get(INDEX_CORRUPT_CODE)!.nextCommand);+});++test("damage that reaches the metadata is still reported as a fault, not as an unknown", async () => {+ await writeConfig();+ // The damaged pages take `index_state` with them, so the verb cannot+ // read what this file said about its own last check. The scan still+ // completed and still condemned the file, and THAT is the finding -+ // "we could not tell" would send the operator nowhere.+ seedVault(4);+ await indexVault(makeConfig({ vault, dbPath }));+ corruptTableRoot(dbPath, "index_state");++ const integrity = integrityOf(await checkJson("--integrity"));+ expect(integrity["verdict"]).toBe("faulty");+ expect(integrity["next_command"]).toBe(DIAGNOSTIC_SIGNALS.get(INDEX_CORRUPT_CODE)!.nextCommand);+ // "Never checked" is not claimed over a cell that could not be read.+ expect(integrity["previously_checked_at"]).toBeUndefined();+ expect(typeof integrity["previous_check_unreadable"]).toBe("string");+});diff --git a/tests/cli/search-chunk-window-estimate.test.ts b/tests/cli/search-chunk-window-estimate.test.tsnew file mode 100644index 00000000..fe46e1ea--- /dev/null+++ b/tests/cli/search-chunk-window-estimate.test.ts@@ -0,0 +1,148 @@+/**+ * `o2b search index` / `o2b search status` over an index the census+ * cannot decide (what-the-index-already-knew, task H follow-up).+ *+ * The end-to-end half of `tests/core/search/chunk-window-estimate.ts`:+ * these runs go through real config resolution, so the instruction+ * prefix is the one the recommended default auto-resolves rather than one+ * a fixture wrote down, and the model window is the one the preset table+ * declares.+ *+ * The obligation discharged here is that a Chinese vault under the+ * Chinese preset stops being SILENT - it used to emit nothing on any+ * surface, and absence is documented to mean "every chunk fits" - while+ * an all-Latin vault emits the exact bytes it emitted before the+ * undecided band existed.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join } from "node:path";++import { requireNextStep } from "../../src/core/brain/next-step.ts";+import { declaredInputWindowTokens } from "../../src/core/search/embeddings/presets.ts";+import { runCli } from "../helpers/run-cli.ts";++let tmp: string;+let vault: string;+let configPath: string;++/** The Chinese preset, and the window its own model card declares. */+const ZH_MODEL = "BAAI/bge-small-zh-v1.5";+const ZH_WINDOW = declaredInputWindowTokens(ZH_MODEL)!;++const OVERFLOW_COMMAND = requireNextStep("search-chunk-window-overflow").nextCommand;++/**+ * The exact `stats` bytes a one-file, one-chunk run emitted before the+ * census had an undecided band. Written out rather than rebuilt from the+ * parsed object: a re-serialized comparison agrees with itself whatever+ * the writer emitted.+ */+const BASELINE_STATS_JSON =+ '"stats":{"added":1,"updated":0,"unchanged":0,"deleted":0,' ++ '"chunks_total":1,"embeddings_computed":0,"embeddings_retries":0}';++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-chunk-window-estimate-"));+ vault = join(tmp, "vault");+ mkdirSync(vault, { recursive: true });+ configPath = join(tmp, "config.yaml");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++/** Semantic on, no key use: nothing in these runs contacts a provider. */+function writeConfig(model: string): void {+ writeFileSync(+ configPath,+ [+ `vault: "${vault}"`,+ 'search_semantic_enabled: "true"',+ 'embedding_provider: "openai-compat"',+ 'embedding_base_url: "https://embeddings.invalid/v1"',+ `embedding_model: "${model}"`,+ 'embedding_api_key: "test-key"',+ ].join("\n") + "\n",+ );+}++function writeNote(rel: string, body: string): void {+ const abs = join(vault, rel);+ mkdirSync(dirname(abs), { recursive: true });+ writeFileSync(abs, body);+}++function env(): Record<string, string> {+ return { OPEN_SECOND_BRAIN_CONFIG: configPath };+}++/**+ * A Chinese body several times its model's window: a BERT-family Chinese+ * tokenizer is roughly character-level, so the character count IS about+ * the token count, while characters-over-four reads a quarter of it.+ */+const HAN_NOTE = `# 记录\n\n${"知识管理系统的检索质量取决于分块大小".repeat(35)}\n`;+const LATIN_NOTE = "# Short\n\nA note that fits any declared window.\n";++test("a Chinese vault under the Chinese preset is no longer silent", async () => {+ writeConfig(ZH_MODEL);+ writeNote("zh.md", HAN_NOTE);+ const r = await runCli(["search", "index", "--json"], { env: env() });+ expect(r.returncode).toBe(0);++ const payload = JSON.parse(r.stdout) as {+ stats: { chunk_window?: Record<string, unknown> };+ };+ const census = payload.stats.chunk_window;+ expect(census?.["verdict"]).toBe("estimate-undecided");+ expect(census!["model"]).toBe(ZH_MODEL);+ expect(census!["window_tokens"]).toBe(ZH_WINDOW);+ expect(census!["chunks_undecided"]).toBeGreaterThan(0);+ // Nothing was PROVED over the window, so no count claims it was.+ expect("chunks_over_window" in census!).toBe(false);+ expect(census!["next_command"]).toBe(OVERFLOW_COMMAND);+});++test("the undecided verdict states the reason in the human run", async () => {+ writeConfig(ZH_MODEL);+ writeNote("zh.md", HAN_NOTE);+ const r = await runCli(["search", "index"], { env: env() });+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain("chunk window:");+ expect(r.stdout).toContain("undecided against");+ expect(r.stdout).toContain(`${ZH_WINDOW}-token window`);+ expect(r.stdout).toContain(`next: ${OVERFLOW_COMMAND}\n`);+});++test("search status carries the undecided band as a warning that denies a pass", async () => {+ writeConfig(ZH_MODEL);+ writeNote("zh.md", HAN_NOTE);+ await runCli(["search", "index"], { env: env() });+ const r = await runCli(["search", "status", "--json"], { env: env() });+ expect(r.returncode).toBe(0);+ const status = JSON.parse(r.stdout) as { warnings: string[] };+ const warning = status.warnings.find((w) => w.includes(OVERFLOW_COMMAND));+ expect(warning ?? "(the census said nothing)").toContain("not a pass");+ expect(warning!).toContain("search_chunk_size");+ // The check RAN - it is not the undeclared-window state.+ expect("chunk_window_undeclared" in status).toBe(false);+});++test("an all-Latin vault emits the pre-band bytes on both surfaces", async () => {+ writeConfig(ZH_MODEL);+ writeNote("a.md", LATIN_NOTE);+ const indexed = await runCli(["search", "index", "--json"], { env: env() });+ expect(indexed.returncode).toBe(0);+ expect(indexed.stdout).toContain(BASELINE_STATS_JSON);+ expect(indexed.stdout).not.toContain("chunk_window");+ expect(indexed.stdout).not.toContain("undecided");++ const status = await runCli(["search", "status", "--json"], { env: env() });+ expect(status.returncode).toBe(0);+ expect(status.stdout).not.toContain("chunk_window");+ expect((JSON.parse(status.stdout) as { warnings: string[] }).warnings).toEqual([]);+});diff --git a/tests/cli/search-chunk-window.test.ts b/tests/cli/search-chunk-window.test.tsnew file mode 100644index 00000000..3870239a--- /dev/null+++ b/tests/cli/search-chunk-window.test.ts@@ -0,0 +1,247 @@+/**+ * `o2b search index` reports the oversize-chunk census, and reports+ * NOTHING when there is nothing to report (what-the-index-already-knew,+ * task H).+ *+ * The byte-identity obligation is discharged against the actual bytes of+ * the `stats` object the `--json` payload carries: a run whose chunks fit+ * the configured model's declared window emits the exact substring it+ * emitted before this field existed, character for character, and so does+ * a run with semantic search off. Both are compared to one literal, so a+ * field that leaked in as `null` or `0` would change those bytes and fail+ * here rather than in a consumer.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join } from "node:path";++import { requireNextStep } from "../../src/core/brain/next-step.ts";+import {+ declaredInputWindowTokens,+ RECOMMENDED_EMBEDDING_MODEL,+} from "../../src/core/search/embeddings/presets.ts";+import { charLengthOverTokenBudget } from "../../src/core/search/embeddings/signature.ts";+import { runCli } from "../helpers/run-cli.ts";++let tmp: string;+let vault: string;+let configPath: string;++/** The declared window of the shipped recommended default. */+const WINDOW = declaredInputWindowTokens(RECOMMENDED_EMBEDDING_MODEL)!;++/** A model deliberately outside the curated table. */+const UNCURATED_MODEL = "acme/custom-embed-9000";++/**+ * The exact `stats` bytes a one-file, one-chunk run emitted before the+ * census field existed. Written out rather than rebuilt from the parsed+ * object on purpose: a re-serialized comparison would agree with itself+ * no matter what the writer emitted.+ */+const BASELINE_STATS_JSON =+ '"stats":{"added":1,"updated":0,"unchanged":0,"deleted":0,' ++ '"chunks_total":1,"embeddings_computed":0,"embeddings_retries":0}';++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-chunk-window-"));+ vault = join(tmp, "vault");+ mkdirSync(vault, { recursive: true });+ configPath = join(tmp, "config.yaml");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function writeConfig(lines: ReadonlyArray<string>): void {+ writeFileSync(configPath, [`vault: "${vault}"`, ...lines].join("\n") + "\n");+}++/** Semantic on, no key use: nothing in these runs contacts a provider. */+function semanticConfig(model: string): ReadonlyArray<string> {+ return [+ 'search_semantic_enabled: "true"',+ 'embedding_provider: "openai-compat"',+ 'embedding_base_url: "https://embeddings.invalid/v1"',+ `embedding_model: "${model}"`,+ 'embedding_api_key: "test-key"',+ ];+}++function writeNote(rel: string, body: string): void {+ const abs = join(vault, rel);+ mkdirSync(dirname(abs), { recursive: true });+ writeFileSync(abs, body);+}++const SHORT_NOTE = "# Short\n\nA note that fits any declared window.\n";+const OVERSIZE_NOTE = `# Big\n\n${"overflow ".repeat(charLengthOverTokenBudget(WINDOW))}\n`;++async function index(json: boolean): Promise<{ stdout: string; returncode: number }> {+ const argv = json ? ["search", "index", "--json"] : ["search", "index"];+ const r = await runCli(argv, { env: { OPEN_SECOND_BRAIN_CONFIG: configPath } });+ return { stdout: r.stdout, returncode: r.returncode };+}++test("with semantic search off the stats bytes are the pre-feature bytes", async () => {+ writeConfig([]);+ writeNote("a.md", SHORT_NOTE);+ const r = await index(true);+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain(BASELINE_STATS_JSON);+ expect(r.stdout).not.toContain("chunk_window");+});++test("a clean census leaves those same bytes untouched", async () => {+ writeConfig(semanticConfig(RECOMMENDED_EMBEDDING_MODEL));+ writeNote("a.md", SHORT_NOTE);+ const r = await index(true);+ expect(r.returncode).toBe(0);+ // The census ran - the model is curated and its window is declared -+ // and found nothing, so it says nothing.+ expect(r.stdout).toContain(BASELINE_STATS_JSON);+ expect(r.stdout).not.toContain("chunk_window");+});++test("a clean census adds no line to the human report either", async () => {+ writeConfig(semanticConfig(RECOMMENDED_EMBEDDING_MODEL));+ writeNote("a.md", SHORT_NOTE);+ const r = await index(false);+ expect(r.returncode).toBe(0);+ expect(r.stdout).not.toContain("chunk window");+ expect(r.stdout).not.toContain(requireNextStep("search-chunk-window-overflow").nextCommand);+});++test("oversize chunks are counted, named, and carry their registered exit", async () => {+ writeConfig(semanticConfig(RECOMMENDED_EMBEDDING_MODEL));+ writeNote("big.md", OVERSIZE_NOTE);+ const r = await index(true);+ expect(r.returncode).toBe(0);+ const payload = JSON.parse(r.stdout) as {+ stats: {+ chunk_window?: {+ verdict: string;+ model: string;+ window_tokens: number;+ chunks_over_window: number;+ chunks_measured: number;+ next_command?: string;+ };+ };+ };+ const census = payload.stats.chunk_window;+ expect(census?.verdict).toBe("over-window");+ expect(census!.model).toBe(RECOMMENDED_EMBEDDING_MODEL);+ expect(census!.window_tokens).toBe(WINDOW);+ expect(census!.chunks_over_window).toBeGreaterThan(0);+ expect(census!.next_command).toBe(requireNextStep("search-chunk-window-overflow").nextCommand);+});++test("the human run states the count and the rail names the exit", async () => {+ writeConfig(semanticConfig(RECOMMENDED_EMBEDDING_MODEL));+ writeNote("big.md", OVERSIZE_NOTE);+ const r = await index(false);+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain("chunk window:");+ expect(r.stdout).toContain(`${WINDOW}-token window`);+ expect(r.stdout).toContain(+ `next: ${requireNextStep("search-chunk-window-overflow").nextCommand}\n`,+ );+});++test("an uncurated model reports a check that did not run, with no count", async () => {+ writeConfig(semanticConfig(UNCURATED_MODEL));+ writeNote("a.md", SHORT_NOTE);+ const r = await index(true);+ expect(r.returncode).toBe(0);+ const payload = JSON.parse(r.stdout) as {+ stats: { chunk_window?: Record<string, unknown> };+ };+ const census = payload.stats.chunk_window;+ expect(census?.["verdict"]).toBe("window-undeclared");+ expect(census!["model"]).toBe(UNCURATED_MODEL);+ // Nothing was counted, so no count is reported. A zero here would be a+ // measurement nobody took.+ expect("chunks_over_window" in census!).toBe(false);+ expect("window_tokens" in census!).toBe(false);+ expect(census!["next_command"]).toBe(+ requireNextStep("search-chunk-window-undeclared").nextCommand,+ );+});++test("the uncurated human report says the check did not run", async () => {+ writeConfig(semanticConfig(UNCURATED_MODEL));+ writeNote("a.md", SHORT_NOTE);+ const r = await index(false);+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain("not checked");+ expect(r.stdout).toContain(UNCURATED_MODEL);+ expect(r.stdout).toContain(+ `next: ${requireNextStep("search-chunk-window-undeclared").nextCommand}\n`,+ );+});++test("search status carries measured overflow as a warning", async () => {+ writeConfig(semanticConfig(RECOMMENDED_EMBEDDING_MODEL));+ writeNote("big.md", OVERSIZE_NOTE);+ await index(false);+ const r = await runCli(["search", "status", "--json"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(r.returncode).toBe(0);+ const status = JSON.parse(r.stdout) as { warnings: string[] };+ const command = requireNextStep("search-chunk-window-overflow").nextCommand;+ expect(status.warnings.some((w) => w.includes(command) && w.includes("search_chunk_size"))).toBe(+ true,+ );+ // A measured fault is not an unmeasurable one.+ expect("chunk_window_undeclared" in status).toBe(false);+});++test("search status distinguishes all three census states", async () => {+ // 1. Not measurable: a field, and NOT a warning.+ writeConfig(semanticConfig(UNCURATED_MODEL));+ writeNote("a.md", SHORT_NOTE);+ await index(false);+ const uncurated = await runCli(["search", "status", "--json"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(uncurated.returncode).toBe(0);+ const notMeasurable = JSON.parse(uncurated.stdout) as {+ warnings: string[];+ chunk_window_undeclared?: Record<string, unknown>;+ };+ expect(notMeasurable.chunk_window_undeclared).toEqual({+ verdict: "window-undeclared",+ model: UNCURATED_MODEL,+ chunks_measured: 1,+ });+ expect(notMeasurable.warnings).toEqual([]);++ // 2. Measured clean: neither. The same vault, the same chunks, a+ // model that declares a window - and the status says nothing at all.+ writeConfig(semanticConfig(RECOMMENDED_EMBEDDING_MODEL));+ await runCli(["search", "reindex"], { env: { OPEN_SECOND_BRAIN_CONFIG: configPath } });+ const clean = await runCli(["search", "status", "--json"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(clean.returncode).toBe(0);+ expect(clean.stdout).not.toContain("chunk_window");+ expect((JSON.parse(clean.stdout) as { warnings: string[] }).warnings).toEqual([]);+});++test("the human status names the check that did not run, outside the warnings", async () => {+ writeConfig(semanticConfig(UNCURATED_MODEL));+ writeNote("a.md", SHORT_NOTE);+ await index(false);+ const r = await runCli(["search", "status"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(r.returncode).toBe(0);+ expect(r.stdout).toContain(`chunk_window_check: not run`);+ expect(r.stdout).toContain(UNCURATED_MODEL);+ expect(r.stdout).not.toContain("warning:");+});diff --git a/tests/cli/search-duplicate-locations.test.ts b/tests/cli/search-duplicate-locations.test.tsnew file mode 100644index 00000000..142cc626--- /dev/null+++ b/tests/cli/search-duplicate-locations.test.ts@@ -0,0 +1,95 @@+/**+ * `o2b search` surfaces the merged-away duplicate locations+ * (what-the-index-already-knew, task D landed the merge, task E landed+ * the boundary).+ *+ * The exact-duplicate merge records every location a byte-identical+ * passage was folded from on `BrainSearchResult.duplicates`. Neither CLI+ * projection read it: the `--json` payload is an explicit field list and+ * the human transcript rendered nothing, so a value computed end-to-end+ * was dropped at the boundary.+ *+ * Absent, not null, when nothing was merged - in both projections.+ */++import { describe, expect, test } from "bun:test";++import { jsonForOutcome, renderOutcomeHuman } from "../../src/cli/search/outcome-render.ts";+import type { DuplicatePassageLocation } from "../../src/core/search/search-result.ts";+import type { BrainSearchResult, SearchOutcome } from "../../src/core/search/types.ts";++const QUERY = "quantum entanglement";++const FOLDED: DuplicatePassageLocation = Object.freeze({+ documentId: 7,+ chunkId: 42,+ path: "archive/entanglement-copy.md",+ title: "Entanglement (copy)",+ startLine: 3,+ endLine: 9,+});++function resultWith(duplicates?: ReadonlyArray<DuplicatePassageLocation>): BrainSearchResult {+ return Object.freeze({+ documentId: 1,+ chunkId: 1,+ path: "physics/entanglement.md",+ title: "Entanglement",+ content: "Quantum entanglement links two particles at a distance.",+ startLine: 1,+ endLine: 9,+ score: 1.5,+ keywordScore: 1.5,+ semanticScore: 0,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "keyword" as const,+ reasons: Object.freeze(["keyword: 1.50"]),+ ...(duplicates !== undefined ? { duplicates: Object.freeze(duplicates) } : {}),+ });+}++function outcomeWith(duplicates?: ReadonlyArray<DuplicatePassageLocation>): SearchOutcome {+ return Object.freeze({+ results: Object.freeze([resultWith(duplicates)]),+ warnings: Object.freeze([]),+ total: 1,+ });+}++function jsonRows(outcome: SearchOutcome): Array<Record<string, unknown>> {+ return (jsonForOutcome(outcome) as { results: Array<Record<string, unknown>> }).results;+}++describe("search duplicate locations on the CLI", () => {+ test("--json names every folded location on the surviving row", () => {+ const row = jsonRows(outcomeWith([FOLDED]))[0]!;+ expect(row["duplicates"]).toEqual([+ {+ path: FOLDED.path,+ title: FOLDED.title,+ start_line: FOLDED.startLine,+ end_line: FOLDED.endLine,+ document_id: FOLDED.documentId,+ chunk_id: FOLDED.chunkId,+ },+ ]);+ });++ test("--json omits the key entirely when nothing was merged", () => {+ const row = jsonRows(outcomeWith())[0]!;+ expect("duplicates" in row).toBe(false);+ });++ test("the human transcript names the folded locations as line pointers", () => {+ const rendered = renderOutcomeHuman(outcomeWith([FOLDED]), false, QUERY);+ expect(rendered.split("\n")).toContain(+ ` duplicates: ${FOLDED.path}:L${FOLDED.startLine}-L${FOLDED.endLine}`,+ );+ });++ test("the human transcript renders no duplicates line when nothing was merged", () => {+ const rendered = renderOutcomeHuman(outcomeWith(), false, QUERY);+ expect(rendered).not.toContain("duplicates:");+ });+});diff --git a/tests/cli/search-explain-trace.test.ts b/tests/cli/search-explain-trace.test.tsnew file mode 100644index 00000000..0031e7a9--- /dev/null+++ b/tests/cli/search-explain-trace.test.ts@@ -0,0 +1,224 @@+/**+ * `o2b search --explain` serializes the retrieval decision trace and the+ * memory trust assessment (what-the-index-already-knew, task F).+ *+ * Both receipts are built on every gated search and no CLI projection+ * read them. The flag-off bytes are proved identical against the SAME+ * outcome object rendered twice, so the comparison measures the flag and+ * nothing else - no second search, no clock drift, no tolerance.+ */++import { describe, expect, test } from "bun:test";+import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { jsonForOutcome, renderOutcomeHuman } from "../../src/cli/search/outcome-render.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import {+ RETRIEVAL_TRACE_NOT_MERGED,+ RETRIEVAL_TRACE_UNAVAILABLE_AND_NOT_MERGED,+ RETRIEVAL_TRACE_UNAVAILABLE,+ type ExplainRequest,+} from "../../src/core/search/explain-envelope.ts";+import type { BrainSearchResult, SearchOutcome } from "../../src/core/search/types.ts";+import { runCli } from "../helpers/run-cli.ts";++const QUERY = "widget calibration";++/** Single-vault explain: the flag on, no cross-vault union, gate engaged. */+const EXPLAIN_ON: ExplainRequest = Object.freeze({+ explain: true,+ crossVault: false,+ trustGateEnabled: true,+});++/** The same flag over a cross-vault union, whose receipts are not merged. */+const EXPLAIN_ON_GLOBAL: ExplainRequest = Object.freeze({+ explain: true,+ crossVault: true,+ trustGateEnabled: true,+});++const RESULT: BrainSearchResult = Object.freeze({+ documentId: 1,+ chunkId: 1,+ path: "notes/clean.md",+ title: "Clean",+ content: "The widget calibration routine runs daily.",+ startLine: 1,+ endLine: 9,+ score: 1.5,+ keywordScore: 1.5,+ semanticScore: 0,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "keyword" as const,+ reasons: Object.freeze(["keyword: 1.50"]),+});++const TRACE = Object.freeze({+ evaluated: 2,+ surfaced: 1,+ excluded: 1,+ exclusions: Object.freeze([+ Object.freeze({+ document_id: 2,+ chunk_id: 2,+ path: "notes/quarantined.md",+ reasons: Object.freeze(["trust_gate:self_approval_quarantine"]),+ }),+ ]),+});++const ASSESSMENT = Object.freeze({+ evaluated: 2,+ surfaced: 1,+ quarantined: 1,+ reason_counts: Object.freeze({ "trust_gate:self_approval_quarantine": 1 }),+});++/** With receipts (the trust gate ran) or without (the gate is off). */+function outcome(withReceipts: boolean): SearchOutcome {+ return Object.freeze({+ results: Object.freeze([RESULT]),+ warnings: Object.freeze([]),+ total: 2,+ ...(withReceipts ? { retrievalDecisionTrace: TRACE, memoryTrustAssessment: ASSESSMENT } : {}),+ });+}++describe("o2b search --explain: retrieval receipts", () => {+ test("json: no trace keys when explain is off, even with receipts on the outcome", () => {+ const withGate = JSON.stringify(jsonForOutcome(outcome(true)));+ const withoutGate = JSON.stringify(jsonForOutcome(outcome(false)));++ // Byte identity: the receipts are invisible until the flag asks.+ expect(withGate).toBe(withoutGate);+ expect(withGate).not.toContain("retrieval_decision_trace");+ expect(withGate).not.toContain("memory_trust_assessment");+ expect(withGate).not.toContain("retrieval_trace_unavailable");+ });++ test("json: explain serializes both receipts", () => {+ const payload = jsonForOutcome(outcome(true), EXPLAIN_ON) as Record<string, unknown>;++ expect(payload["retrieval_decision_trace"]).toEqual(TRACE);+ expect(payload["memory_trust_assessment"]).toEqual(ASSESSMENT);+ expect("retrieval_trace_unavailable" in payload).toBe(false);+ // The rest of the payload is untouched: the explain keys are appended.+ const off = JSON.stringify(jsonForOutcome(outcome(true)));+ delete payload["retrieval_decision_trace"];+ delete payload["memory_trust_assessment"];+ expect(JSON.stringify(payload)).toBe(off);+ });++ test("json: explain with no receipts names why instead of going quiet", () => {+ const payload = jsonForOutcome(outcome(false), EXPLAIN_ON) as Record<string, unknown>;++ expect(payload["retrieval_trace_unavailable"]).toBe(RETRIEVAL_TRACE_UNAVAILABLE);+ expect("retrieval_decision_trace" in payload).toBe(false);+ });++ test("json: a cross-vault union gives its own reason, not the gate's", () => {+ // A union merges one search per origin and carries no receipts, so+ // reporting the gate state here would be a false diagnosis.+ const payload = jsonForOutcome(outcome(false), EXPLAIN_ON_GLOBAL) as Record<string, unknown>;++ expect(payload["retrieval_trace_unavailable"]).toBe(RETRIEVAL_TRACE_NOT_MERGED);+ expect(payload["retrieval_trace_unavailable"]).not.toBe(RETRIEVAL_TRACE_UNAVAILABLE);+ });++ test("json: when the gate is off AND the search is global, both are named", () => {+ // Naming only the union would send the operator to a single-vault+ // query that still produces nothing, because the gate is off there+ // too. Naming only the gate would promise a trace a union cannot+ // merge. The exit has to carry both or it is a dead end either way.+ const both: ExplainRequest = Object.freeze({+ explain: true,+ crossVault: true,+ trustGateEnabled: false,+ });+ const payload = jsonForOutcome(outcome(false), both) as Record<string, unknown>;+ const why = payload["retrieval_trace_unavailable"];++ expect(why).toBe(RETRIEVAL_TRACE_UNAVAILABLE_AND_NOT_MERGED);+ expect(why).not.toBe(RETRIEVAL_TRACE_NOT_MERGED);+ expect(why).not.toBe(RETRIEVAL_TRACE_UNAVAILABLE);+ // Both exits are actually spelled, not merely alluded to.+ expect(String(why)).toContain("search_trust_gate_enabled");+ expect(String(why)).toContain("single vault");+ });++ test("human: the transcript is byte-identical until --explain asks", () => {+ const off = renderOutcomeHuman(outcome(true), false, QUERY);+ expect(off).toBe(renderOutcomeHuman(outcome(false), false, QUERY));++ const on = renderOutcomeHuman(outcome(true), false, QUERY, EXPLAIN_ON);+ expect(on.startsWith(off)).toBe(true);+ expect(on).toContain("evaluated 2");+ expect(on).toContain("excluded 1");+ expect(on).toContain("notes/quarantined.md");+ expect(on).toContain("trust_gate:self_approval_quarantine");++ const unavailable = renderOutcomeHuman(outcome(false), false, QUERY, EXPLAIN_ON);+ expect(unavailable).toContain(RETRIEVAL_TRACE_UNAVAILABLE);+ });+});++describe("o2b search --explain end to end", () => {+ test("the flag reaches the payload and total reports the pre-truncation pool", async () => {+ const vault = mkdtempSync(join(tmpdir(), "o2b-cli-trace-vault-"));+ const configHome = mkdtempSync(join(tmpdir(), "o2b-cli-trace-cfg-"));+ try {+ mkdirSync(join(vault, "notes"), { recursive: true });+ writeFileSync(+ join(vault, "notes", "clean.md"),+ "---\ntitle: Clean\n---\n\n# Clean\n\nThe widget calibration routine runs daily.\n",+ );+ writeFileSync(+ join(vault, "notes", "clean-two.md"),+ "---\ntitle: Clean two\n---\n\n# Clean two\n\nThe widget calibration schedule is weekly.\n",+ );+ writeFileSync(+ join(vault, "notes", "quarantined.md"),+ "---\ntitle: Bad\nstatus: quarantine\n---\n\n# Bad\n\nThe widget calibration is unsafe.\n",+ );+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nsearch_trust_gate_enabled: true\n`);+ const env = { OPEN_SECOND_BRAIN_CONFIG: configPath };++ const indexed = await runCli(["search", "index"], { env });+ expect(indexed.returncode).toBe(0);++ const plain = await runCli(["search", QUERY, "--json", "--limit", "1"], { env });+ expect(plain.returncode).toBe(0);+ const plainPayload = JSON.parse(plain.stdout) as Record<string, unknown>;+ expect("retrieval_decision_trace" in plainPayload).toBe(false);+ // One row returned out of a three-candidate pool, one of which the+ // gate excluded: total is the surviving pool, not the row count.+ expect((plainPayload["results"] as unknown[]).length).toBe(1);+ expect(plainPayload["total"]).toBe(2);++ const explained = await runCli(["search", QUERY, "--json", "--limit", "1", "--explain"], {+ env,+ });+ expect(explained.returncode).toBe(0);+ const payload = JSON.parse(explained.stdout) as {+ retrieval_decision_trace?: {+ evaluated: number;+ excluded: number;+ exclusions: Array<{ path: string }>;+ };+ memory_trust_assessment?: { quarantined: number };+ };+ expect(payload.retrieval_decision_trace!.evaluated).toBe(3);+ expect(payload.retrieval_decision_trace!.excluded).toBe(1);+ expect(payload.retrieval_decision_trace!.exclusions[0]!.path).toBe("notes/quarantined.md");+ expect(payload.memory_trust_assessment!.quarantined).toBe(1);+ } finally {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+ }+ });+});diff --git a/tests/cli/search-query-flag-manifest.test.ts b/tests/cli/search-query-flag-manifest.test.tsnew file mode 100644index 00000000..70f63ea5--- /dev/null+++ b/tests/cli/search-query-flag-manifest.test.ts@@ -0,0 +1,144 @@+/**+ * `o2b search query` advertises every flag it parses.+ *+ * There is no per-verb `--help` for the search verbs (`o2b search query+ * --help` exits 2 with "unknown flag: --help"), so `o2b help` and `o2b+ * help --json` are the ONLY discovery surface for this verb's flags. The+ * manifest declared none of them, which is worse than a partial list read+ * as complete: an agent parsing `help --json` concluded `search query`+ * takes nothing but `--json`.+ *+ * The direction of enumeration is the point, exactly as in+ * `manifest-completeness.test.ts`: this reads the `parseFlags` schema in+ * the VERB and asks the manifest to account for each key, never the+ * reverse. A manifest cannot vouch for its own completeness.+ */++import { describe, expect, test } from "bun:test";+import { readFileSync } from "node:fs";+import { join } from "node:path";++import { nestedCommand } from "../../src/cli/command-manifest.ts";+import { VAULT_FLAGS } from "../../src/cli/search/helpers.ts";++const QUERY_VERB_SOURCE = join(+ import.meta.dir,+ "..",+ "..",+ "src",+ "cli",+ "search",+ "verbs",+ "query.ts",+);++/**+ * The inherited flag the manifest states once in the help header instead+ * of repeating on every entry; parity is asserted over declared flags.+ */+const INHERITED_FLAG_NAME = "json";++/** `<name>: { type: "<type>"` — one entry of a `parseFlags` schema literal. */+const SCHEMA_ENTRY_RE = /(?:"([^"]+)"|([A-Za-z][\w-]*)):\s*\{\s*type:\s*"([^"]+)"/g;++/** The spread that pulls in the shared vault-addressing triple. */+const VAULT_SPREAD = "...VAULT_FLAGS";++/**+ * The `parseFlags(argv, { ... })` schema `query.ts` declares, as+ * `name -> type`, with `...VAULT_FLAGS` resolved from the real constant.+ */+function parsedFlagSchema(): ReadonlyMap<string, string> {+ const text = readFileSync(QUERY_VERB_SOURCE, "utf8");+ const start = text.indexOf("parseFlags(argv, {");+ expect(`query.ts calls parseFlags: ${start >= 0}`).toBe("query.ts calls parseFlags: true");+ const end = text.indexOf("});", start);+ expect(`the parseFlags call is terminated: ${end > start}`).toBe(+ "the parseFlags call is terminated: true",+ );+ const literal = text.slice(start, end);++ const out = new Map<string, string>();+ if (literal.includes(VAULT_SPREAD)) {+ for (const [name, spec] of Object.entries(VAULT_FLAGS)) {+ out.set(name, (spec as { type: string }).type);+ }+ }+ for (const match of literal.matchAll(SCHEMA_ENTRY_RE)) {+ out.set(match[1] ?? match[2]!, match[3]!);+ }+ return out;+}++/** The manifest's declared flags for `o2b search query`, as `name -> type`. */+function manifestFlagSchema(): ReadonlyMap<string, string> {+ const entry = nestedCommand("search", "query");+ expect(`the manifest models search query: ${entry !== undefined}`).toBe(+ "the manifest models search query: true",+ );+ return new Map((entry!.flags ?? []).map((f) => [f.name, f.type]));+}++/** Both sides minus the inherited flag the header already states. */+function comparable(schema: ReadonlyMap<string, string>): ReadonlyMap<string, string> {+ const out = new Map(schema);+ out.delete(INHERITED_FLAG_NAME);+ return out;+}++describe("o2b search query declares the flags it parses", () => {+ test("every parsed flag is modelled by the manifest, with the same type", () => {+ const parsed = comparable(parsedFlagSchema());+ const modelled = comparable(manifestFlagSchema());+ // Named, not counted: the failure message is the work to be done.+ const problems: string[] = [];+ for (const [name, type] of parsed) {+ const declared = modelled.get(name);+ if (declared === undefined) problems.push(`--${name} is parsed but not modelled`);+ else if (declared !== type) {+ problems.push(`--${name} is modelled as ${declared} but parsed as ${type}`);+ }+ }+ expect(problems.toSorted().join("\n")).toBe("");+ });++ test("no modelled flag outlives the parse that would accept it", () => {+ const parsed = comparable(parsedFlagSchema());+ const modelled = comparable(manifestFlagSchema());+ const stale = [...modelled.keys()].filter((name) => !parsed.has(name));+ expect(stale.toSorted().join("\n")).toBe("");+ });++ test("the flag the trust receipts arrived with is reachable from help", () => {+ // `--explain` is the branch's new flag and the reason this file exists.+ expect([...manifestFlagSchema().keys()]).toContain("explain");+ });++ test("the ratchet is not vacuous and can fail", () => {+ // Pin the measurement, not only its verdict: a regex that stopped+ // matching would report a clean sweep over an empty set.+ const parsed = comparable(parsedFlagSchema());+ expect(parsed.size).toBeGreaterThan(20);+ expect(parsed.get("explain")).toBe("boolean");+ expect(parsed.get("vault")).toBe("string");++ // Prove the comparison catches a violation: a verb schema carrying a+ // flag the manifest never heard of, and one whose type disagrees.+ const violating = new Map(parsed);+ violating.set("undeclared-flag", "boolean");+ violating.set("limit", "boolean");+ const modelled = comparable(manifestFlagSchema());+ const problems: string[] = [];+ for (const [name, type] of violating) {+ const declared = modelled.get(name);+ if (declared === undefined) problems.push(`--${name} is parsed but not modelled`);+ else if (declared !== type) {+ problems.push(`--${name} is modelled as ${declared} but parsed as ${type}`);+ }+ }+ expect(problems.toSorted()).toEqual([+ "--limit is modelled as string but parsed as boolean",+ "--undeclared-flag is parsed but not modelled",+ ]);+ });+});diff --git a/tests/cli/terminal-state-census.test.ts b/tests/cli/terminal-state-census.test.tsindex 73929c67..5df30f49 100644--- a/tests/cli/terminal-state-census.test.ts+++ b/tests/cli/terminal-state-census.test.ts@@ -186,14 +186,13 @@ const CLASSIFIED_BY_MODULE_WALK: Readonly<Record<string, StateNote>> = Object.fr"judgement over content that naming any one of would be a guess dressed as an " +"instruction.",},- "o2b search check": {- cls: STATE_CLASS.refusal,- reason:- "refuses through `resolveConfig` in `search/helpers.ts`, which throws a `CliError` " +- "naming `--vault` and `o2b init` when no vault resolves. Splitting `search.ts` by " +- "verb moved that refusal out of this verb's own file and changed nothing about which " +- "verb makes it.",- },+ // `o2b search check` sat here while its only refusal was the shared+ // `resolveConfig` one, a module hop away. It classifies in its own file+ // now: `--integrity` (what-the-index-already-knew, unit K) writes the+ // scan's cost notice to stderr and resolves a structural fault through+ // `emitNextStep`. The entry is deleted rather than reworded because an+ // inventory of states the walk RESCUES may not hold a state the walk no+ // longer has to rescue - that would be an exclusion covering nothing."o2b search weights": {cls: STATE_CLASS.refusal,reason:diff --git a/tests/core/brain/doctor-exit-census.test.ts b/tests/core/brain/doctor-exit-census.test.tsindex 1c1ea2c3..6c766a6f 100644--- a/tests/core/brain/doctor-exit-census.test.ts+++ b/tests/core/brain/doctor-exit-census.test.ts@@ -58,8 +58,8 @@*/import { describe, expect, test } from "bun:test";-import { readdirSync, readFileSync } from "node:fs";-import { join } from "node:path";+import { existsSync, readdirSync, readFileSync } from "node:fs";+import { dirname, join } from "node:path";import { DIAGNOSTIC_SIGNALS } from "../../../src/core/brain/diagnostics.ts";import {@@ -84,19 +84,50 @@ function checkModules(dir: string): string[] {return out;}+/**+ * An import statement that pulls a `*_CODE` identifier in from another+ * module, capturing that module's relative path so the scan can follow it.+ */+const IMPORTED_CODE_RE = /import \{[^}]*[A-Z0-9_]*CODE[A-Z0-9_]*[^}]*\} from "(\.[^"]+)";/gs;++/**+ * A check module may emit a code whose constant is declared elsewhere -+ * an import cycle can force the declaration into a leaf module the check+ * imports from. Those declaring modules are FOLLOWED from the check's own+ * import statements rather than listed here, for the same reason the+ * check directory is enumerated: a hand-maintained second list drifts,+ * and a code the census cannot read is a code it cannot account for.+ */+function codeConstantSources(paths: ReadonlyArray<string>): ReadonlyArray<string> {+ const followed = new Set<string>();+ for (const path of paths) {+ const source = readFileSync(path, "utf8");+ for (const match of source.matchAll(IMPORTED_CODE_RE)) {+ const resolved = join(dirname(path), match[1]!);+ if (existsSync(resolved)) followed.add(resolved);+ }+ }+ return [...followed].toSorted();+}+/** The registry plus every check module it runs, enumerated from disk. */-const DOCTOR_SOURCE_PATHS: ReadonlyArray<string> = [+const DOCTOR_CHECK_PATHS: ReadonlyArray<string> = [join(BRAIN_DIR, "doctor.ts"),...checkModules(CHECKS_DIR),];+const DOCTOR_SOURCE_PATHS: ReadonlyArray<string> = [+ ...DOCTOR_CHECK_PATHS,+ ...codeConstantSources(DOCTOR_CHECK_PATHS),+];+const DOCTOR_SOURCE = DOCTOR_SOURCE_PATHS.map((path) => readFileSync(path, "utf8")).join("\n");/** `code: "<literal>"` - the shape most of the doctor's codes take. */const LITERAL_CODE_RE = /\bcode: "([^"]+)"/g;/** `const <NAME>_CODE = "<literal>";` - a code hoisted to a constant. */-const CONSTANT_CODE_RE = /^const [A-Z0-9_]*CODE[A-Z0-9_]* = "([^"]+)";$/gm;+const CONSTANT_CODE_RE = /^(?:export )?const [A-Z0-9_]*CODE[A-Z0-9_]* = "([^"]+)";$/gm;/** Any `code:` property, for the shape guard below. */const CODE_SITE_RE = /^\s*code: (.+?),?$/gm;@@ -154,6 +185,7 @@ const DOCTOR_REGISTERED_CODES: ReadonlyArray<string> = ["dangling-workrun","duplicate-preferences","entity-label-malformed",+ "entity-quote-variant-collision","low-evidence-confirmed","orphan-evidence","principle-corrupted",@@ -272,7 +304,9 @@ describe("every code site is readable from the source", () => {const unresolved = codeSiteValues().filter((value) => CODE_IDENTIFIER_RE.test(value)).filter((value) => {- const declared = DOCTOR_SOURCE.match(new RegExp(`^const ${value} = "([^"]+)";$`, "m"))?.[1];+ const declared = DOCTOR_SOURCE.match(+ new RegExp(`^(?:export )?const ${value} = "([^"]+)";$`, "m"),+ )?.[1];return declared === undefined || !constants.has(declared);});expect([...new Set(unresolved)].toSorted().join("\n")).toBe("");diff --git a/tests/core/brain/entities/quote-variant-collision.test.ts b/tests/core/brain/entities/quote-variant-collision.test.tsnew file mode 100644index 00000000..7805a137--- /dev/null+++ b/tests/core/brain/entities/quote-variant-collision.test.ts@@ -0,0 +1,249 @@+/**+ * The collision the quote fold CREATES, at the seams a caller reaches+ * (what-the-index-already-knew, task A - review follow-up).+ *+ * `quote-variant-fold.test.ts` pins the kernel and the three refusals that+ * were already wired: alias-vs-alias, alias-vs-canonical-name, and the+ * archive release. None of those is the case the fold actually creates.+ * Folding typographic quote variants makes two previously-distinct+ * CANONICAL NAMES resolve to one identity key, and both records are on+ * disk already: the id allocator suffixes `-2` on a slug collision, so an+ * older binary wrote them side by side without ever seeing a conflict.+ *+ * That state reaches three seams, and this file pins what each must do:+ *+ * 1. `upsertEntity` - the write seam. Resolving a CONTESTED identity key+ * picks the walk's first claimant and rewrites it, reporting+ * `created: false`. The caller's label is discarded, the other record+ * is untouched, and nothing says so. It must refuse instead, naming+ * both records and the registered exit.+ * 2. The category-LESS read - `getEntity` with no category. It throws,+ * but says the reference is ambiguous "across categories" when both+ * records sit in ONE category, and names "pass a category" as the+ * exit - which resolves nothing here. It must name the real reason.+ * 3. The doctor - the pre-existing safety net. It must still report the+ * pair, and the finding must carry the code whose exit is registered+ * rather than the one whose exit is written up as absent.+ *+ * The category-SCOPED read deliberately keeps returning the first+ * claimant: `archiveEntity` resolves through it, and that archive IS the+ * registered exit, so a read that refused would put the remedy behind the+ * fault it reports. That decision is asserted here, not merely written+ * down.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../../../src/core/brain/init.ts";+import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";+import { DIAGNOSTIC_SIGNALS } from "../../../../src/core/brain/diagnostics.ts";+import { ENTITY_QUOTE_VARIANT_COLLISION_CODE } from "../../../../src/core/brain/entities/canonical.ts";+import { entityRegistryCheck } from "../../../../src/core/brain/doctor/entity-checks.ts";+import {+ archiveEntity,+ getEntity,+ upsertEntity,+} from "../../../../src/core/brain/entities/registry.ts";+import type { DoctorIssue } from "../../../../src/core/brain/types.ts";++const NOW = new Date("2026-06-02T12:00:00Z");+const AGENT = "claude-dev-agent";+const CATEGORY = "music";++/** U+2019 RIGHT SINGLE QUOTATION MARK - what an editor substitutes. */+const CURLY_LABEL = "Taylor’s Version";+/** U+0027 APOSTROPHE - the fold target, and what a keyboard types. */+const ASCII_LABEL = "Taylor's Version";++const CURLY_ID = "ent-music-taylor-s-version";+const ASCII_ID = "ent-music-taylor-s-version-2";++let vault: string;+let configHome: string;++/**+ * The two records an older binary left on disk. Written directly, because+ * the API cannot produce this state once the fold is in - which is exactly+ * why the state exists in the field and not in any suite.+ */+function seedPreFoldPair(): void {+ const dir = join(vault, "Brain", "entities", CATEGORY);+ mkdirSync(dir, { recursive: true });+ for (const [id, label] of [+ [CURLY_ID, CURLY_LABEL],+ [ASCII_ID, ASCII_LABEL],+ ] as const) {+ atomicWriteFileSync(+ join(dir, `${id}.md`),+ [+ "---",+ "kind: brain-entity",+ `entity_id: ${id}`,+ `category: ${CATEGORY}`,+ `name: ${label}`,+ "status: active",+ "created_at: 2026-06-01T00:00:00Z",+ "updated_at: 2026-06-01T00:00:00Z",+ "---",+ "",+ `# ${label}`,+ "",+ ].join("\n"),+ );+ }+}++/** The registered exit, read from the registry rather than retyped. */+function registeredExit(): string {+ const signal = DIAGNOSTIC_SIGNALS.get(ENTITY_QUOTE_VARIANT_COLLISION_CODE);+ expect(signal).toBeDefined();+ return signal!.nextCommand;+}++function runEntityDoctor(): DoctorIssue[] {+ const issues: DoctorIssue[] = [];+ entityRegistryCheck.run(+ { vault, configPath: join(configHome, "config.yaml") } as never,+ { issues, uncertain: [] } as never,+ );+ return issues;+}++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-quote-collision-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-quote-collision-cfg-"));+ atomicWriteFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+ seedPreFoldPair();+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++describe("the write seam refuses a contested identity key", () => {+ test("upsert into the folded pair refuses instead of rewriting one of them", () => {+ let message = "";+ let result: unknown = null;+ try {+ result = upsertEntity(vault, {+ category: CATEGORY,+ name: ASCII_LABEL,+ agent: AGENT,+ now: NOW,+ body: "# rewritten by the caller",+ });+ } catch (exc) {+ message = (exc as Error).message;+ }+ expect(result).toBeNull();+ expect(message).toContain(CURLY_ID);+ expect(message).toContain(ASCII_ID);+ expect(message).toContain(registeredExit());+ });++ test("the refusal names the quote-variant cause, not just the duplicate", () => {+ let message = "";+ try {+ upsertEntity(vault, { category: CATEGORY, name: ASCII_LABEL, agent: AGENT, now: NOW });+ } catch (exc) {+ message = (exc as Error).message;+ }+ expect(message).toContain("typographic quote form");+ });+});++describe("the category-less read names the real reason", () => {+ test("a same-category collision is not reported as ambiguity across categories", () => {+ let message = "";+ try {+ getEntity(vault, { query: ASCII_LABEL });+ } catch (exc) {+ message = (exc as Error).message;+ }+ expect(message).not.toContain("across categories");+ expect(message).not.toContain("pass a category");+ expect(message).toContain(CATEGORY);+ expect(message).toContain("typographic quote form");+ expect(message).toContain(registeredExit());+ });++ test("a genuine cross-category ambiguity keeps the message it always had", () => {+ upsertEntity(vault, { category: "people", name: "Ada", agent: AGENT, now: NOW });+ upsertEntity(vault, { category: "projects", name: "Ada", agent: AGENT, now: NOW });+ let message = "";+ try {+ getEntity(vault, { query: "Ada" });+ } catch (exc) {+ message = (exc as Error).message;+ }+ expect(message).toContain("ambiguous across categories");+ expect(message).toContain("pass a category");+ expect(message).not.toContain(registeredExit());+ });+});++describe("the registered exit stays reachable", () => {+ test("the category-scoped read still resolves, so archive can run", () => {+ const held = getEntity(vault, { category: CATEGORY, query: ASCII_LABEL });+ expect(held).not.toBeNull();+ expect(held!.id).toBe(CURLY_ID);+ const archived = archiveEntity(vault, { category: CATEGORY, query: ASCII_LABEL }, { now: NOW });+ expect(archived.id).toBe(CURLY_ID);+ // One archive ends the collision: the survivor is now reachable, and+ // the write seam is open again.+ const after = upsertEntity(vault, {+ category: CATEGORY,+ name: ASCII_LABEL,+ agent: AGENT,+ now: NOW,+ });+ expect(after.created).toBe(false);+ expect(after.entity.id).toBe(ASCII_ID);+ });+});++describe("the doctor reports the pair with the code that has an exit", () => {+ test("the finding carries the registered quote-variant code", () => {+ const issues = runEntityDoctor();+ const codes = issues.map((i) => i.code);+ expect(codes).toContain(ENTITY_QUOTE_VARIANT_COLLISION_CODE);+ const finding = issues.find((i) => i.code === ENTITY_QUOTE_VARIANT_COLLISION_CODE)!;+ expect(finding.message).toContain(CURLY_ID);+ expect(finding.message).toContain(ASCII_ID);+ expect(finding.message).toContain("typographic quote form");+ });++ test("an ordinary duplicate keeps the duplicate-entity code", () => {+ const dir = join(vault, "Brain", "entities", "people");+ mkdirSync(dir, { recursive: true });+ for (const id of ["ent-people-ada", "ent-people-ada-2"]) {+ atomicWriteFileSync(+ join(dir, `${id}.md`),+ [+ "---",+ "kind: brain-entity",+ `entity_id: ${id}`,+ "category: people",+ "name: Ada",+ "status: active",+ "created_at: 2026-06-01T00:00:00Z",+ "updated_at: 2026-06-01T00:00:00Z",+ "---",+ "",+ "# Ada",+ "",+ ].join("\n"),+ );+ }+ const issues = runEntityDoctor();+ const duplicate = issues.filter((i) => i.code === "duplicate-entity");+ expect(duplicate.length).toBe(1);+ expect(duplicate[0]!.message).toContain("people:ada");+ expect(duplicate[0]!.message).not.toContain("typographic quote form");+ });+});diff --git a/tests/core/brain/entities/quote-variant-fold.test.ts b/tests/core/brain/entities/quote-variant-fold.test.tsnew file mode 100644index 00000000..ec75ac4f--- /dev/null+++ b/tests/core/brain/entities/quote-variant-fold.test.ts@@ -0,0 +1,363 @@+/**+ * The identity kernel folds typographic quote variants (what-the-index-+ * already-knew, task A).+ *+ * `normalizeEntityName` used to do NFC + trim + collapse + lowercase and+ * nothing else, and no Unicode normal form maps U+2019 to U+0027 - so+ * `Taylor's` and `Taylor’s` were two canonical entities in one registry,+ * differing only in which key a text editor happened to insert.+ *+ * Three obligations are pinned here.+ *+ * 1. The fold merges the variants: two labels differing only in quote+ * form produce one identity key.+ * 2. The fold's FOOTPRINT is exactly the Unicode classes it claims. The+ * test below derives the population from `\p{Pi}`, `\p{Pf}` and the+ * modifier-letter apostrophe over the whole code point space and+ * asserts the kernel changes those code points and no others - so a+ * widened rule cannot slip in unmeasured.+ * 3. The kernel stays byte-stable for everything else. A corpus of+ * currently-valid labels is keyed by the PRE-FOLD kernel, reproduced+ * verbatim here, and by the shipped one, and the two must agree+ * character for character. That is the backward-compatibility+ * guarantee `canonical.ts` documents; a spot check is not a proof.+ *+ * The collision the fold creates is pinned in the registry block at the+ * bottom: two previously-distinct records now claim one key, and the+ * refusal must name the registered command that resolves the duplicate.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../../../src/core/brain/init.ts";+import { atomicWriteFileSync } from "../../../../src/core/fs-atomic.ts";+import { DIAGNOSTIC_SIGNALS } from "../../../../src/core/brain/diagnostics.ts";+import {+ differsOnlyByQuoteVariant,+ entityIdentityKey,+ foldQuoteVariants,+ normalizeEntityName,+ QUOTE_VARIANT_FOLD_TARGET,+ ENTITY_QUOTE_VARIANT_COLLISION_CODE,+} from "../../../../src/core/brain/entities/canonical.ts";+import { archiveEntity, upsertEntity } from "../../../../src/core/brain/entities/registry.ts";++/**+ * The identity kernel EXACTLY as it shipped before the fold: NFC, trim,+ * collapse whitespace runs, lowercase. Reproduced here rather than+ * imported so the byte-stability corpus is compared against a fixed+ * reference that a future edit to the kernel cannot move.+ */+function preFoldNormalize(raw: string): string {+ return raw.normalize("NFC").trim().replace(/\s+/g, " ").toLowerCase();+}++/** Upper bound of the Unicode code point space. */+const MAX_CODE_POINT = 0x10ffff;+/** The surrogate range, which is not a scalar value. */+const SURROGATE_FIRST = 0xd800;+const SURROGATE_LAST = 0xdfff;++/** `U+XXXX` for a code point, for readable failure output. */+function u(cp: number): string {+ return `U+${cp.toString(16).toUpperCase().padStart(4, "0")}`;+}++/**+ * The fold's declared population, derived from Unicode properties by the+ * engine rather than typed out: initial and final quotation punctuation,+ * plus the modifier-letter apostrophe (a LETTER by general category, so+ * outside `\p{Pi}`/`\p{Pf}`, and an apostrophe by function).+ */+const DECLARED_CLASS_RE = /[\p{Pi}\p{Pf}\u{02BC}]/u;++/** Every scalar value the declared class admits, in code point order. */+function declaredClassCodePoints(): number[] {+ const out: number[] = [];+ for (let cp = 0; cp <= MAX_CODE_POINT; cp++) {+ if (cp >= SURROGATE_FIRST && cp <= SURROGATE_LAST) continue;+ if (DECLARED_CLASS_RE.test(String.fromCodePoint(cp))) out.push(cp);+ }+ return out;+}++describe("foldQuoteVariants", () => {+ test("the two apostrophe forms produce one identity key", () => {+ expect(entityIdentityKey("people", "Taylor's Version")).toBe(+ entityIdentityKey("people", "Taylor’s Version"),+ );+ expect(normalizeEntityName("Taylor’s")).toBe("taylor's");+ });++ test("the ASCII apostrophe is the fold target and is itself untouched", () => {+ expect(QUOTE_VARIANT_FOLD_TARGET).toBe("'");+ expect(foldQuoteVariants("Taylor's")).toBe("Taylor's");+ });++ test("the footprint is exactly the declared Unicode classes, nothing wider", () => {+ const declared = declaredClassCodePoints();+ // The class is non-empty and bounded - a regex that stopped matching+ // would otherwise report a clean sweep over an empty set.+ expect(declared.length).toBeGreaterThan(0);++ const changed: number[] = [];+ for (let cp = 0; cp <= MAX_CODE_POINT; cp++) {+ if (cp >= SURROGATE_FIRST && cp <= SURROGATE_LAST) continue;+ const ch = String.fromCodePoint(cp);+ if (foldQuoteVariants(ch) !== ch) changed.push(cp);+ }+ expect(changed.map(u)).toEqual(declared.map(u));+ for (const cp of changed) {+ expect(`${u(cp)} -> ${foldQuoteVariants(String.fromCodePoint(cp))}`).toBe(+ `${u(cp)} -> ${QUOTE_VARIANT_FOLD_TARGET}`,+ );+ }+ });++ test("the modifier-letter apostrophe needs its own entry: it is a letter", () => {+ const modifierLetterApostrophe = "ʼ";+ expect(/\p{Lm}/u.test(modifierLetterApostrophe)).toBe(true);+ expect(/[\p{Pi}\p{Pf}]/u.test(modifierLetterApostrophe)).toBe(false);+ expect(foldQuoteVariants(modifierLetterApostrophe)).toBe(QUOTE_VARIANT_FOLD_TARGET);+ });++ test("no Unicode normal form would have done this - the premise of the fold", () => {+ const typographic = "Taylor’s";+ for (const form of ["NFC", "NFD", "NFKC", "NFKD"] as const) {+ expect(`${form}: ${typographic.normalize(form)}`).toBe(`${form}: ${typographic}`);+ }+ });+});++describe("kernel byte-stability", () => {+ /**+ * Currently-valid labels that carry no quote variant: the population+ * whose identity keys must not move. Latin with internal punctuation,+ * an ASCII apostrophe, a comma, CJK, Cyrillic, emoji, mixed scripts,+ * combining marks, digits and symbols.+ */+ const CORPUS: ReadonlyArray<string> = [+ "Ada",+ "Open Second Brain",+ "Node.js",+ "C++",+ "New York, NY",+ "e.g",+ "café",+ "café",+ "Ада",+ "Пётр Ильич",+ "北京",+ "日本語のエンティティ",+ "한국어",+ "🙂 emoji label",+ "Mixed 混合 Script Ада",+ "Taylor's Version",+ "6' 2\"",+ "AT&T",+ "R&D / Q3",+ " padded spaces ",+ "UPPER lower MiXeD",+ "version 2.0.1-rc1",+ "50% off",+ "@handle #tag",+ "a\tb\nc",+ "ʻokina Hawaiʻi",+ ];++ test("every corpus label keys identically to the pre-fold kernel", () => {+ for (const label of CORPUS) {+ expect(`${JSON.stringify(label)}: ${normalizeEntityName(label)}`).toBe(+ `${JSON.stringify(label)}: ${preFoldNormalize(label)}`,+ );+ expect(entityIdentityKey("people", label)).toBe(`people:${preFoldNormalize(label)}`);+ }+ });++ test("the corpus is discriminating: a quote variant does move", () => {+ const moved = "Taylor’s Version";+ expect(normalizeEntityName(moved)).not.toBe(preFoldNormalize(moved));+ });+});++describe("differsOnlyByQuoteVariant", () => {+ test("true when the folded forms agree and the unfolded ones do not", () => {+ expect(differsOnlyByQuoteVariant("Taylor's Version", "Taylor’s Version")).toBe(true);+ expect(differsOnlyByQuoteVariant("O’Brien", "O'Brien")).toBe(true);+ });++ test("false when the labels are already the same identity", () => {+ expect(differsOnlyByQuoteVariant("Ada", " ADA ")).toBe(false);+ expect(differsOnlyByQuoteVariant("Taylor’s", "Taylor’s")).toBe(false);+ });++ test("false when the labels differ by anything other than quote form", () => {+ expect(differsOnlyByQuoteVariant("Ada", "Ада")).toBe(false);+ expect(differsOnlyByQuoteVariant("Taylor's Version", "Taylor's Album")).toBe(false);+ });+});++describe("the collision the fold creates names its exit", () => {+ const NOW = new Date("2026-06-02T12:00:00Z");+ const LATER = new Date("2026-06-02T13:00:00Z");+ const AGENT = "claude-dev-agent";++ let vault: string;+ let configHome: string;++ beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-quote-fold-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-quote-fold-cfg-"));+ atomicWriteFileSync(join(configHome, "config.yaml"), `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath: join(configHome, "config.yaml") });+ });++ afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+ });++ /** The registered exit, read from the registry rather than retyped. */+ function registeredExit(): string {+ const signal = DIAGNOSTIC_SIGNALS.get(ENTITY_QUOTE_VARIANT_COLLISION_CODE);+ expect(signal).toBeDefined();+ return signal!.nextCommand;+ }++ test("the collision code is registered with a structural command", () => {+ const signal = DIAGNOSTIC_SIGNALS.get(ENTITY_QUOTE_VARIANT_COLLISION_CODE);+ expect(signal).toBeDefined();+ expect(signal!.nextCommand.startsWith("o2b ")).toBe(true);+ expect(signal!.autoRepairable).toBe(false);+ });++ test("an alias colliding on quote form alone refuses with the registered command", () => {+ upsertEntity(vault, {+ category: "music",+ name: "Taylor Swift",+ aliases: ["Taylor's Version"],+ agent: AGENT,+ now: NOW,+ });+ let message = "";+ try {+ upsertEntity(vault, {+ category: "music",+ name: "Red",+ aliases: ["Taylor’s Version"],+ agent: AGENT,+ now: LATER,+ });+ } catch (exc) {+ message = (exc as Error).message;+ }+ expect(message).toContain("Taylor’s Version");+ expect(message).toContain(registeredExit());+ });++ test("an alias colliding with a canonical NAME on quote form alone names it too", () => {+ upsertEntity(vault, {+ category: "music",+ name: "Taylor's Version",+ agent: AGENT,+ now: NOW,+ });+ let message = "";+ try {+ upsertEntity(vault, {+ category: "music",+ name: "Red",+ aliases: ["Taylor’s Version"],+ agent: AGENT,+ now: LATER,+ });+ } catch (exc) {+ message = (exc as Error).message;+ }+ expect(message).toContain("canonical name");+ expect(message).toContain(registeredExit());+ });++ test("an ordinary alias collision is unchanged - the exit is discriminating", () => {+ upsertEntity(vault, {+ category: "people",+ name: "Ada",+ aliases: ["Ада"],+ agent: AGENT,+ now: NOW,+ });+ let message = "";+ try {+ upsertEntity(vault, {+ category: "people",+ name: "Adam",+ aliases: ["Ада"],+ agent: AGENT,+ now: LATER,+ });+ } catch (exc) {+ message = (exc as Error).message;+ }+ expect(message).toContain("already claimed by");+ expect(message).not.toContain(registeredExit());+ });++ test("releasing an archived record onto a folded namesake names the exit", () => {+ // The upgrade shape: both files were on disk before the fold, so+ // neither write seam ever saw a collision. Written directly for that+ // reason - the API cannot produce this state once the fold is in.+ const dir = join(vault, "Brain", "entities", "music");+ mkdirSync(dir, { recursive: true });+ atomicWriteFileSync(+ join(dir, "ent-music-typographic.md"),+ [+ "---",+ "kind: brain-entity",+ "entity_id: ent-music-typographic",+ "category: music",+ "name: Taylor’s Version",+ "status: archived",+ "created_at: 2026-06-01T00:00:00Z",+ "updated_at: 2026-06-01T00:00:00Z",+ "archived_at: 2026-06-01T00:00:00Z",+ "---",+ "",+ "# Taylor’s Version",+ "",+ ].join("\n"),+ );+ atomicWriteFileSync(+ join(dir, "ent-music-ascii.md"),+ [+ "---",+ "kind: brain-entity",+ "entity_id: ent-music-ascii",+ "category: music",+ "name: Taylor's Version",+ "status: active",+ "created_at: 2026-06-01T00:00:00Z",+ "updated_at: 2026-06-01T00:00:00Z",+ "---",+ "",+ "# Taylor's Version",+ "",+ ].join("\n"),+ );++ let message = "";+ try {+ archiveEntity(+ vault,+ { category: "music", query: "Taylor’s Version" },+ { now: LATER, restore: true },+ );+ } catch (exc) {+ message = (exc as Error).message;+ }+ expect(message).toContain("ent-music-ascii");+ expect(message).toContain(registeredExit());+ });+});diff --git a/tests/core/brain/recall-signals.test.ts b/tests/core/brain/recall-signals.test.tsnew file mode 100644index 00000000..35e76ada--- /dev/null+++ b/tests/core/brain/recall-signals.test.ts@@ -0,0 +1,285 @@+/**+ * Recall-quality signals on the per-query telemetry record+ * (what-the-index-already-knew, task G).+ *+ * Four signals are derived from values the retrieval pipeline ALREADY+ * computed and then discarded: the fused and per-lane scores+ * (alignment), the opt-in trust metadata, the typed `contradicts` edges+ * surfaced on the row, and the deterministic token-set overlap of the+ * surfaced pool (diversity). Nothing here reads a file, a clock, or an+ * embedding.+ *+ * The two signals the wave refused - named in+ * `docs/brainstorm/what-the-index-already-knew/design.md` under "Out of+ * scope" - are asserted absent from `src/` by the census at the bottom,+ * which proves it can fail against an injected fixture.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { existsSync, readFileSync } from "node:fs";+import { mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ deriveRecallSignals,+ emitRecallTelemetry,+ RECALL_SIGNALS_DIVERSITY_MAX_ROWS,+ RECALL_SIGNALS_UNMEASURED_CARDS,+} from "../../../src/core/brain/recall-telemetry.ts";+import { clipPayloadToBudget } from "../../../src/core/brain/continuity/store.ts";+import type { BrainSearchResult } from "../../../src/core/search/search-result.ts";++const CREATED_AT = "2026-08-07T12:00:00.000Z";++function row(over: Partial<BrainSearchResult> & { readonly content: string }): BrainSearchResult {+ return {+ documentId: 1,+ chunkId: 1,+ path: "notes/a.md",+ title: "A",+ startLine: 1,+ endLine: 4,+ score: 0.5,+ keywordScore: 0.3,+ semanticScore: 0.2,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "hybrid",+ reasons: ["keyword: 0.30"],+ ...over,+ };+}++describe("deriveRecallSignals", () => {+ test("an empty pool has nothing to measure and returns null", () => {+ expect(deriveRecallSignals([])).toBeNull();+ });++ test("alignment reports the already-computed fused and per-lane scores", () => {+ const signals = deriveRecallSignals([+ row({ content: "alpha beta gamma", score: 0.8, keywordScore: 0.5, semanticScore: 0.3 }),+ row({ content: "delta epsilon zeta", score: 0.5, keywordScore: 0.2, semanticScore: 0.3 }),+ ])!;++ expect(signals.rows).toBe(2);+ expect(signals.alignment).toEqual({+ top: 0.8,+ mean: 0.65,+ margin: 0.3,+ keyword_sum: 0.7,+ semantic_sum: 0.6,+ });+ });++ test("a single-row pool omits the margin instead of reporting a fabricated zero", () => {+ const signals = deriveRecallSignals([row({ content: "alpha" })])!;+ expect("margin" in signals.alignment).toBe(false);+ });++ test("trust counts only the rows that actually carry trust metadata", () => {+ const untrusted = deriveRecallSignals([row({ content: "alpha" })])!;+ expect(untrusted.trust).toEqual({ assessed: 0, superseded: 0, conflict: 0 });++ const trusted = deriveRecallSignals([+ row({+ content: "alpha",+ trust: { age_days: 12, superseded: true, conflict: false, replacement: "notes/b.md" },+ }),+ row({+ content: "beta",+ trust: { age_days: 3, superseded: false, conflict: true, replacement: null },+ }),+ ])!;+ expect(trusted.trust).toEqual({+ assessed: 2,+ superseded: 1,+ conflict: 1,+ max_age_days: 12,+ });+ });++ test("contradiction counts typed edges without the trust opt-in", () => {+ const signals = deriveRecallSignals([+ row({+ content: "alpha",+ relations: [+ { relation: "contradicts", target: "notes/b.md" },+ { relation: "contradicts", target: "notes/c.md" },+ { relation: "related", target: "notes/d.md" },+ ],+ }),+ row({ content: "beta", relations: [{ relation: "related", target: "notes/e.md" }] }),+ ])!;++ expect(signals.contradiction).toEqual({ rows: 1, edges: 2 });+ });++ test("diversity is a pool-level token overlap, similarity absent when there is no pair", () => {+ const identical = deriveRecallSignals([+ row({ content: "alpha beta gamma" }),+ row({ content: "alpha beta gamma" }),+ ])!;+ expect(identical.diversity).toEqual({+ compared: 2,+ pairs: 1,+ mean_similarity: 1,+ max_similarity: 1,+ });++ const disjoint = deriveRecallSignals([+ row({ content: "alpha beta gamma" }),+ row({ content: "delta epsilon zeta" }),+ ])!;+ expect(disjoint.diversity.mean_similarity).toBe(0);+ expect(disjoint.diversity.max_similarity).toBe(0);++ const single = deriveRecallSignals([row({ content: "alpha beta" })])!;+ expect(single.diversity).toEqual({ compared: 1, pairs: 0 });+ });++ test("the pairwise walk is bounded, and the record names the bound it used", () => {+ const many = Array.from({ length: RECALL_SIGNALS_DIVERSITY_MAX_ROWS + 5 }, () =>+ row({ content: "alpha beta gamma" }),+ );+ const signals = deriveRecallSignals(many)!;++ expect(signals.rows).toBe(RECALL_SIGNALS_DIVERSITY_MAX_ROWS + 5);+ expect(signals.diversity.compared).toBe(RECALL_SIGNALS_DIVERSITY_MAX_ROWS);+ const k = RECALL_SIGNALS_DIVERSITY_MAX_ROWS;+ expect(signals.diversity.pairs).toBe((k * (k - 1)) / 2);+ });++ test("the derivation never mutates or reorders the pool it reads", () => {+ const rows = Object.freeze([+ row({ content: "alpha", score: 0.1 }),+ row({ content: "beta", score: 0.9 }),+ ]);+ const before = JSON.stringify(rows);+ deriveRecallSignals(rows);+ expect(JSON.stringify(rows)).toBe(before);+ });+});++describe("recall telemetry record", () => {+ let vault: string;++ beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-recall-signals-"));+ });++ afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ });++ const base = {+ createdAt: CREATED_AT,+ host: "unit",+ mode: "search" as const,+ status: "ok" as const,+ durationMs: 4,+ resultCount: 1,+ };++ test("byte identity: without signals the payload is unchanged, key absent and not null", () => {+ const record = emitRecallTelemetry(vault, base);++ expect(JSON.stringify(record.payload)).toBe(+ '{"host":"unit","mode":"search","status":"ok","duration_ms":4,"result_count":1,' ++ '"top_artifacts":[],"gaps":[]}',+ );+ expect("signals" in record.payload).toBe(false);+ });++ test("signals ride inside the single per-query record when supplied", () => {+ const signals = deriveRecallSignals([+ row({ content: "alpha beta" }),+ row({ content: "alpha gamma" }),+ ])!;+ const record = emitRecallTelemetry(vault, { ...base, signals });++ expect(record.payload["signals"]).toEqual(signals);+ // Every signal is a number, so the redactor has no string to rewrite:+ // the block survives the sanitiser byte-identically.+ expect(record.redacted).toBe(false);+ expect(record.private).toBe(false);+ });++ test("the cards surface records why it could not measure rather than going quiet", () => {+ const record = emitRecallTelemetry(vault, {+ ...base,+ signals: RECALL_SIGNALS_UNMEASURED_CARDS,+ });++ expect(record.payload["signals"]).toEqual({ unmeasured: "disclosure_cards" });+ expect(record.redacted).toBe(false);+ });++ test("under the shared clip budget the signals block drops whole, identity survives", () => {+ const signals = deriveRecallSignals([+ row({ content: "alpha beta" }),+ row({ content: "alpha gamma" }),+ ])!;+ const record = emitRecallTelemetry(vault, {+ ...base,+ sessionId: "sess-1",+ signals,+ });++ const full = JSON.stringify(record.payload).length;+ const clipped = clipPayloadToBudget(record.payload, full - 1);+ expect("signals" in clipped).toBe(false);+ expect(clipped["session_id"]).toBe("sess-1");+ });+});++/**+ * The two refused signals must not exist in the shipped source. The scan+ * is proved able to fail by running the same matcher over an injected+ * violating fixture.+ */+describe("refused recall signals census", () => {+ const REFUSED: ReadonlyArray<string> = [+ "acquisition_risk",+ "acquisitionRisk",+ "expected_regret",+ "expectedRegret",+ ];++ function violations(text: string): ReadonlyArray<string> {+ return REFUSED.filter((name) => text.includes(name));+ }++ /**+ * The population this census is only as good as. A glob that matched+ * nothing would report a clean sweep over an empty set, which is the+ * failure mode this whole wave is about, so the walk states its own+ * size before it states its verdict.+ */+ const MIN_SCANNED_SOURCES = 700;++ test("no shipped source file names a refused signal", () => {+ const root = join(import.meta.dir, "..", "..", "..", "src");+ const offenders: string[] = [];+ let scanned = 0;+ for (const file of new Bun.Glob("**/*.ts").scanSync({ cwd: root, absolute: true })) {+ scanned++;+ const found = violations(readFileSync(file, "utf8"));+ if (found.length > 0) offenders.push(`${file}: ${found.join(", ")}`);+ }+ // Non-vacuity first: a narrowed pattern or a moved tree makes the+ // sweep below meaningless, and it would pass.+ expect(`scanned ${scanned >= MIN_SCANNED_SOURCES}`).toBe("scanned true");+ // And the walk really reaches the module the signals live in, so a+ // count alone cannot stand in for coverage of the relevant tree.+ expect(+ offenders.length === 0 && existsSync(join(root, "core", "brain", "recall-telemetry.ts")),+ ).toBe(true);+ expect(offenders).toEqual([]);+ });++ test("the census can fail: an injected violating fixture is caught", () => {+ expect(violations("const acquisition_risk = 0.5;")).toEqual(["acquisition_risk"]);+ expect(violations("const expectedRegret = 0.5;")).toEqual(["expectedRegret"]);+ });+});diff --git a/tests/core/search/apostrophe-symmetry.test.ts b/tests/core/search/apostrophe-symmetry.test.tsnew file mode 100644index 00000000..05751e71--- /dev/null+++ b/tests/core/search/apostrophe-symmetry.test.ts@@ -0,0 +1,192 @@+/**+ * Apostrophe symmetry on the retrieval path (what-the-index-already-knew,+ * task A).+ *+ * The entity identity kernel now folds typographic quote variants. The+ * SEARCH side needed no such fold - it is already symmetric - but only by+ * accident of two independent choices that nothing pinned:+ *+ * - the word lane is built with `unicode61 remove_diacritics 2`, which+ * classifies U+0027 and U+2019 alike as separators, so both forms of+ * `Taylor's` produce the same tokens;+ * - the trigram lane is asymmetric at raw SQL level - `"taylor's"` and+ * `"taylor’s"` are different trigram sequences - and is saved by its+ * PLANNER, which splits the query on `[^\p{L}\p{N}]+` and so never+ * emits a term containing either apostrophe.+ *+ * Either choice could be changed by someone who did not know it was+ * load-bearing. This file is the lock: both query forms must retrieve the+ * same rows through the real lanes, and the same assertion run against a+ * deliberately non-symmetric tokenizer must FAIL, so the lock is proved+ * discriminating rather than vacuously true.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";++import { runFtsQuery } from "../../../src/core/search/fts.ts";+import { Store } from "../../../src/core/search/store.ts";+import { DEFAULT_FTS_TOKENIZE } from "../../../src/core/search/schema.ts";+import { planTrigramPrefilter } from "../../../src/core/search/trigram-prefilter.ts";+import { createTempVault, makeConfig } from "../../helpers/search-fixtures.ts";++/** The same phrase in both apostrophe forms. */+const ASCII_QUERY = "Taylor's Version";+const TYPOGRAPHIC_QUERY = "Taylor’s Version";++/** Where each form is stored, so neither query can win by exact-match luck. */+const ASCII_DOC = "Notes/ascii.md";+const TYPOGRAPHIC_DOC = "Notes/typographic.md";++const LIMIT = 20;++/**+ * A tokenizer that treats U+2019 as a token CHARACTER while U+0027 stays a+ * separator - the shape of a well-meant "keep smart quotes intact" config+ * change. `taylor’s` becomes one token and `taylor's` becomes two, so the+ * two query forms stop matching each other's documents.+ */+const ASYMMETRIC_FTS_TOKENIZE = `${DEFAULT_FTS_TOKENIZE} tokenchars ’`;++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("apostrophe-symmetry");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++/** A seeded store plus the document-id to path mapping its lanes report. */+interface Fixture {+ readonly store: Store;+ readonly paths: ReadonlyMap<number, string>;+}++/** A store holding one document per apostrophe form, under `ftsTokenize`. */+async function fixture(ftsTokenize: string): Promise<Fixture> {+ const store = await Store.open(+ { ...makeConfig({ vault, dbPath }), ftsTokenize },+ { mode: "write", loadVec: false },+ );+ const paths = new Map<number, string>();+ const seed = (path: string, content: string, hash: string) => {+ const doc = store.upsertDocument({+ path,+ title: path,+ contentHash: hash,+ mtime: 1700000000,+ size: content.length,+ });+ store.replaceChunks(doc, [+ {+ chunkIndex: 0,+ content,+ contentHash: `${hash}-c0`,+ startLine: 1,+ endLine: 1,+ tokenCount: 4,+ },+ ]);+ paths.set(doc, path);+ };+ seed(ASCII_DOC, `Re-recording ${ASCII_QUERY} in the studio`, "h-ascii");+ seed(TYPOGRAPHIC_DOC, `Re-recording ${TYPOGRAPHIC_QUERY} in the studio`, "h-typo");+ return { store, paths };+}++/** Hit document ids resolved to paths, sorted - the rows a lane returned. */+function hitPaths(fx: Fixture, hits: ReadonlyArray<{ readonly documentId: number }>): string[] {+ return hits+ .map((hit) => fx.paths.get(hit.documentId) ?? `unknown-document-${hit.documentId}`)+ .toSorted();+}++/** The document paths the WORD lane returns for `query`, sorted. */+function wordLanePaths(fx: Fixture, query: string): string[] {+ return hitPaths(fx, runFtsQuery(fx.store, query, { limit: LIMIT }));+}++/** The document paths the TRIGRAM lane returns for `query`, sorted. */+function trigramLanePaths(fx: Fixture, query: string): string[] {+ const plan = planTrigramPrefilter(query);+ if (plan.mode !== "match") return [];+ return hitPaths(fx, fx.store.trigramCandidates(plan.ftsQuery, { limit: LIMIT }).hits);+}++describe("both apostrophe forms retrieve the same rows", () => {+ test("the word lane is symmetric and reaches both documents", async () => {+ const fx = await fixture(DEFAULT_FTS_TOKENIZE);+ try {+ const ascii = wordLanePaths(fx, ASCII_QUERY);+ expect(ascii).toEqual([ASCII_DOC, TYPOGRAPHIC_DOC]);+ expect(wordLanePaths(fx, TYPOGRAPHIC_QUERY)).toEqual(ascii);+ } finally {+ await fx.store.close();+ }+ });++ test("the trigram lane is symmetric and reaches both documents", async () => {+ const fx = await fixture(DEFAULT_FTS_TOKENIZE);+ try {+ const ascii = trigramLanePaths(fx, ASCII_QUERY);+ expect(ascii).toEqual([ASCII_DOC, TYPOGRAPHIC_DOC]);+ expect(trigramLanePaths(fx, TYPOGRAPHIC_QUERY)).toEqual(ascii);+ } finally {+ await fx.store.close();+ }+ });++ test("the planner is why the trigram lane is symmetric", () => {+ const ascii = planTrigramPrefilter(ASCII_QUERY);+ const typographic = planTrigramPrefilter(TYPOGRAPHIC_QUERY);+ expect(ascii).toEqual(typographic);+ expect(ascii.mode).toBe("match");+ if (ascii.mode !== "match") throw new Error("unreachable: asserted above");+ for (const term of ascii.terms) {+ expect(`${term}: ${/['’]/.test(term)}`).toBe(`${term}: false`);+ }+ });+});++describe("the symmetry check can fail", () => {+ test("the trigram index itself is asymmetric - only the planner hides it", async () => {+ const fx = await fixture(DEFAULT_FTS_TOKENIZE);+ try {+ // Bypassing `planTrigramPrefilter` and querying the shadow with the+ // apostrophe-bearing terms directly: the trigram sequences differ, so+ // each form reaches only its own document. A planner change that let+ // an apostrophe into a term would land exactly here.+ const ascii = hitPaths(fx, fx.store.trigramCandidates('"taylor\'s"', { limit: LIMIT }).hits);+ const typographic = hitPaths(+ fx,+ fx.store.trigramCandidates('"taylor’s"', { limit: LIMIT }).hits,+ );+ expect(ascii).toEqual([ASCII_DOC]);+ expect(typographic).toEqual([TYPOGRAPHIC_DOC]);+ } finally {+ await fx.store.close();+ }+ });++ test("a non-symmetric tokenizer splits the two forms apart", async () => {+ const fx = await fixture(ASYMMETRIC_FTS_TOKENIZE);+ try {+ const ascii = wordLanePaths(fx, ASCII_QUERY);+ const typographic = wordLanePaths(fx, TYPOGRAPHIC_QUERY);+ // The assertion the symmetry test makes, inverted: under this+ // tokenizer each form reaches only its own document, so a test that+ // passed here would be passing on nothing.+ expect(ascii).toEqual([ASCII_DOC]);+ expect(typographic).toEqual([TYPOGRAPHIC_DOC]);+ expect(ascii).not.toEqual(typographic);+ } finally {+ await fx.store.close();+ }+ });+});diff --git a/tests/core/search/card-duplicate-locations.test.ts b/tests/core/search/card-duplicate-locations.test.tsnew file mode 100644index 00000000..577f07f3--- /dev/null+++ b/tests/core/search/card-duplicate-locations.test.ts@@ -0,0 +1,139 @@+/**+ * A layer-1 card reports the exact duplicates that were merged into it+ * (what-the-index-already-knew, task D follow-up).+ *+ * The design states: "Merged-away locations are reported on the surviving+ * row, never silently dropped." That held on the `full` path only. A+ * `SearchCard` carried no duplicate field at all, so `toSearchCard`+ * discarded a value the pipeline had already computed: three byte-identical+ * copies returned one card naming one path, with nothing saying two others+ * had been folded away. Cards mode is reachable from both the CLI and MCP.+ *+ * A card exists to be small, so it carries the merged-away locations as+ * `path:Lstart-Lend` POINTERS - the same grammar as its own `pointer`+ * field, and the same rendering the full transcript already uses - rather+ * than the full location records. The passage is byte-identical by+ * construction, so a caller never needs to expand a folded copy; it needs+ * to know where else the bytes live.+ *+ * House discipline for an optional field: ABSENT, not null and not empty,+ * when nothing was merged.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { serializeSearchCard } from "../../../src/core/search/serialize.ts";+import { toSearchCard } from "../../../src/core/search/cards.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import type { DuplicatePassageLocation } from "../../../src/core/search/search-result.ts";+import type { BrainSearchResult, ResolvedSearchConfig } from "../../../src/core/search/types.ts";++const QUERY = "quantum entanglement";++/** The passage duplicated verbatim across files below. */+const PASSAGE =+ "# Quantum entanglement\n\n" ++ "Quantum entanglement links two particles so that measuring one fixes " ++ "the other, regardless of the distance separating them. The correlation " ++ "is not a signal and carries no information on its own.\n";++const FOLDED: DuplicatePassageLocation = Object.freeze({+ documentId: 7,+ chunkId: 42,+ path: "archive/entanglement-copy.md",+ title: "Entanglement (copy)",+ startLine: 3,+ endLine: 9,+});++function resultWith(duplicates?: ReadonlyArray<DuplicatePassageLocation>): BrainSearchResult {+ return Object.freeze({+ documentId: 1,+ chunkId: 1,+ path: "physics/entanglement.md",+ title: "Entanglement",+ content: "Quantum entanglement links two particles at a distance.",+ startLine: 1,+ endLine: 9,+ score: 1.5,+ keywordScore: 1.5,+ semanticScore: 0,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "keyword" as const,+ reasons: Object.freeze(["keyword: 1.50"]),+ ...(duplicates !== undefined ? { duplicates: Object.freeze(duplicates) } : {}),+ });+}++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("card-duplicate-locations");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++test("the projection carries the folded locations as line pointers", () => {+ const card = toSearchCard(resultWith([FOLDED]), QUERY);+ expect(card.duplicatePointers).toEqual([+ `${FOLDED.path}:L${FOLDED.startLine}-L${FOLDED.endLine}`,+ ]);+});++test("a card with nothing merged carries no duplicate key at all", () => {+ const card = toSearchCard(resultWith(), QUERY);+ expect(Object.hasOwn(card, "duplicatePointers")).toBe(false);+ const wire = serializeSearchCard(card);+ expect(Object.hasOwn(wire, "duplicate_pointers")).toBe(false);+ // Absent from the bytes, not merely undefined on the object.+ expect(JSON.stringify(wire)).not.toContain("duplicate");+});++test("the wire projection names the folded locations under a snake_case key", () => {+ const wire = serializeSearchCard(toSearchCard(resultWith([FOLDED]), QUERY));+ expect(wire["duplicate_pointers"]).toEqual([+ `${FOLDED.path}:L${FOLDED.startLine}-L${FOLDED.endLine}`,+ ]);+});++test("three byte-identical copies return one card naming the two folded paths", async () => {+ writeMd(vault, "a/entanglement.md", PASSAGE);+ writeMd(vault, "b/entanglement.md", PASSAGE);+ writeMd(vault, "c/entanglement.md", PASSAGE);+ const cfg: ResolvedSearchConfig = makeConfig({ vault, dbPath });+ await indexVault(cfg);++ const outcome = await search(cfg, { query: QUERY, limit: 10, disclosure: "cards" });+ expect(outcome.cards).toBeDefined();+ expect(outcome.cards!.length).toBe(1);+ const card = outcome.cards![0]!;+ const pointers = card.duplicatePointers ?? [];+ expect(pointers.length).toBe(2);+ const named = [card.path, ...pointers.map((p) => p.slice(0, p.lastIndexOf(":")))].toSorted();+ expect(named).toEqual(["a/entanglement.md", "b/entanglement.md", "c/entanglement.md"]);+ for (const pointer of pointers) expect(pointer).toMatch(/:L\d+(-L\d+)?$/);+});++test("a corpus with no duplicates produces cards with no duplicate key", async () => {+ writeMd(vault, "one.md", "# Alpha\n\nQuantum alpha notes about the first topic entirely.\n");+ writeMd(vault, "two.md", "# Beta\n\nQuantum beta notes about the second topic entirely.\n");+ const cfg: ResolvedSearchConfig = makeConfig({ vault, dbPath });+ await indexVault(cfg);++ const outcome = await search(cfg, { query: "quantum notes", limit: 10, disclosure: "cards" });+ expect(outcome.cards!.length).toBe(2);+ for (const card of outcome.cards!) {+ expect(Object.hasOwn(card, "duplicatePointers")).toBe(false);+ expect(JSON.stringify(serializeSearchCard(card))).not.toContain("duplicate");+ }+});diff --git a/tests/core/search/chunk-window-census.test.ts b/tests/core/search/chunk-window-census.test.tsnew file mode 100644index 00000000..d3263d45--- /dev/null+++ b/tests/core/search/chunk-window-census.test.ts@@ -0,0 +1,400 @@+/**+ * The oversize-chunk census (what-the-index-already-knew, task H).+ *+ * Two shipped defaults overflow each other in silence: the chunker cuts+ * at `search_chunk_size` whitespace WORDS while the recommended model+ * declares an input window in TOKENS, several times smaller. Every+ * window number in this file is read from the preset table rather than+ * written down, so a corrected model card is a one-line edit there and+ * nothing here goes stale. A provider that truncates does so+ * server-side and returns a vector of the right width for a prefix of+ * the passage, so nothing local can observe it. The census is what+ * makes the condition observable, and it is arithmetic available+ * entirely offline.+ *+ * Three properties are pinned here, and each one can fail:+ *+ * - the predicate the store evaluates is EXACTLY the estimator the+ * embedding request path sizes batches with, proved by brute force+ * rather than by inspection;+ * - the ONE place the two disagree - SQLite counts code points where+ * JavaScript counts UTF-16 code units - is pinned as a bound and a+ * direction, not left to be discovered;+ * - a model with no declared window reports that the check did not+ * run. Never that it passed.+ *+ * The two verdicts leave `o2b search status` by different doors, and+ * that split is pinned here too. Measured overflow is a warning: it is a+ * fault with a command behind it. An undeclared window is a structured+ * field: there is no command that declares a window for someone else's+ * model, and it is the COMMON case, so a warning would repeat forever+ * and cost the channel its meaning. Both states remain distinguishable+ * from a pass, which is the property that matters.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";++import { indexStatus, indexVault } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import {+ charLengthOverTokenBudget,+ estimateTokens,+ LOCAL_EMBEDDING_MODEL,+} from "../../../src/core/search/embeddings/signature.ts";+import {+ declaredInputWindowTokens,+ EMBEDDING_MODEL_PRESETS,+ RECOMMENDED_EMBEDDING_MODEL,+} from "../../../src/core/search/embeddings/presets.ts";+import { requireNextStep } from "../../../src/core/brain/next-step.ts";+import { serializeIndexStatus } from "../../../src/core/search/serialize.ts";+import { chunkWindowDiagnosticCode } from "../../../src/core/search/types.ts";+import type {+ ResolvedEmbeddingConfig,+ ResolvedSearchConfig,+} from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("chunk-window");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++/** The curated window that governs the shipped recommended default. */+const RECOMMENDED_WINDOW = declaredInputWindowTokens(RECOMMENDED_EMBEDDING_MODEL)!;++/** A model string deliberately outside the curated table. */+const UNCURATED_MODEL = "acme/custom-embed-9000";++function cfg(+ semantic: Partial<ResolvedEmbeddingConfig> = {},+ overrides: Partial<ResolvedSearchConfig> = {},+): ResolvedSearchConfig {+ return Object.freeze({+ ...makeConfig({+ vault,+ dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ // Never reached: no test here computes embeddings, so no request+ // is ever made. The census is a read over already-indexed rows.+ baseUrl: "https://embeddings.invalid/v1",+ model: RECOMMENDED_EMBEDDING_MODEL,+ apiKey: "test-key",+ dimension: 384,+ ...semantic,+ },+ }),+ ...overrides,+ });+}++/** A note body that is one chunk and comfortably over `window` tokens. */+function oversizeBody(windowTokens: number): string {+ return "overflow ".repeat(charLengthOverTokenBudget(windowTokens));+}++// ── the predicate is the estimator ───────────────────────────────────────────++test("the census predicate equals the request path's estimator at every length", () => {+ for (const windowTokens of [1, 8, 128, RECOMMENDED_WINDOW]) {+ const threshold = charLengthOverTokenBudget(windowTokens);+ for (let len = 0; len <= threshold + 8; len++) {+ const text = "a".repeat(len);+ const byEstimator = estimateTokens([text]) > windowTokens;+ const byLength = len > threshold;+ expect(`w=${windowTokens} len=${len}: ${byLength}`).toBe(+ `w=${windowTokens} len=${len}: ${byEstimator}`,+ );+ }+ }+});++// ── the presets declare a window ─────────────────────────────────────────────++test("every curated preset declares a positive integer input window", () => {+ for (const p of EMBEDDING_MODEL_PRESETS) {+ expect(`${p.model}: ${Number.isInteger(p.inputWindowTokens)}`).toBe(`${p.model}: true`);+ expect(p.inputWindowTokens).toBeGreaterThan(0);+ }+});++test("an uncurated model declares no window, and null is not zero", () => {+ expect(declaredInputWindowTokens(UNCURATED_MODEL)).toBeNull();+ expect(declaredInputWindowTokens(null)).toBeNull();+ // The polarity that matters: unlike the price table, an absent entry+ // does not resolve to a permissive number.+ expect(declaredInputWindowTokens(RECOMMENDED_EMBEDDING_MODEL)).toBe(RECOMMENDED_WINDOW);+});++test("the shipped defaults are the ones that overflow each other", async () => {+ // The arithmetic the unit exists for, asserted rather than described:+ // a default-sized chunk of ordinary prose estimates well past the+ // recommended model's declared window.+ const config = cfg();+ const wordsPerChunk = config.chunkSize;+ const averageWord = "retrieval ".length;+ expect(estimateTokens(["x".repeat(wordsPerChunk * averageWord)])).toBeGreaterThan(+ RECOMMENDED_WINDOW * 2,+ );+});++// ── the store-side count ─────────────────────────────────────────────────────++test("the store counts exactly the chunks whose estimate exceeds the window", async () => {+ const window = 16;+ const threshold = charLengthOverTokenBudget(window);+ const store = await Store.open(cfg(), { mode: "write" });+ try {+ const docId = store.upsertDocument({+ path: "a.md",+ title: "A",+ contentHash: "hash-a",+ mtime: 1,+ size: 1,+ pageType: null,+ authoredAt: null,+ eventAnchor: null,+ });+ store.replaceChunks(docId, [+ {+ chunkIndex: 0,+ content: "b".repeat(threshold),+ contentHash: "h0",+ startLine: 1,+ endLine: 1,+ tokenCount: 1,+ },+ {+ chunkIndex: 1,+ content: "b".repeat(threshold + 1),+ contentHash: "h1",+ startLine: 2,+ endLine: 2,+ tokenCount: 1,+ },+ {+ chunkIndex: 2,+ content: "b".repeat(threshold * 3),+ contentHash: "h2",+ startLine: 3,+ endLine: 3,+ tokenCount: 1,+ },+ { chunkIndex: 3, content: "", contentHash: "h3", startLine: 4, endLine: 4, tokenCount: 0 },+ ]);+ // Exactly the two over the boundary - the one AT it is not over it,+ // which is the same boundary `estimateTokens` draws.+ expect(store.chunksOverTokenWindow(window)).toBe(2);+ expect(store.chunksOverTokenWindow(window * 4)).toBe(0);+ } finally {+ await store.close();+ }+});++test("the code-point / code-unit divergence is a lower bound, and only that", async () => {+ const window = 16;+ const threshold = charLengthOverTokenBudget(window);+ // A supplementary-plane character is ONE code point to SQLite's+ // length() and TWO code units to String.length. A chunk built entirely+ // from them sits at the threshold in the database and past it in the+ // estimator, so the census declines to count it.+ const astral = "\u{1F600}".repeat(threshold);+ expect(astral.length).toBe(threshold * 2);+ expect(estimateTokens([astral])).toBeGreaterThan(window);++ const store = await Store.open(cfg(), { mode: "write" });+ try {+ const docId = store.upsertDocument({+ path: "astral.md",+ title: null,+ contentHash: "hash-astral",+ mtime: 1,+ size: 1,+ pageType: null,+ authoredAt: null,+ eventAnchor: null,+ });+ store.replaceChunks(docId, [+ {+ chunkIndex: 0,+ content: astral,+ contentHash: "h0",+ startLine: 1,+ endLine: 1,+ tokenCount: 1,+ },+ ]);+ // The stated bound: never a false alarm, and short by at most one+ // code unit per supplementary character.+ expect(store.chunksOverTokenWindow(window)).toBe(0);+ // One more astral character past the code-point threshold and the+ // count fires, so the divergence is a bound and not a blind spot.+ store.replaceChunks(docId, [+ {+ chunkIndex: 0,+ content: "\u{1F600}".repeat(threshold + 1),+ contentHash: "h1",+ startLine: 1,+ endLine: 1,+ tokenCount: 1,+ },+ ]);+ expect(store.chunksOverTokenWindow(window)).toBe(1);+ } finally {+ await store.close();+ }+});++// ── the census verdicts ──────────────────────────────────────────────────────++test("an index whose chunks exceed the declared window reports a counted verdict", async () => {+ writeMd(vault, "big.md", `# Big\n\n${oversizeBody(RECOMMENDED_WINDOW)}\n`);+ const stats = await indexVault(cfg());+ expect(stats.chunkWindow?.verdict).toBe("over-window");+ const census = stats.chunkWindow!;+ if (census.verdict !== "over-window") throw new Error("verdict narrowed wrong");+ expect(census.model).toBe(RECOMMENDED_EMBEDDING_MODEL);+ expect(census.windowTokens).toBe(RECOMMENDED_WINDOW);+ expect(census.chunksOverWindow).toBeGreaterThan(0);+ expect(census.chunksOverWindow).toBeLessThanOrEqual(census.chunksMeasured);+});++test("an index whose chunks fit reports NOTHING - the field is absent, not zero", async () => {+ writeMd(vault, "small.md", "# Small\n\nA short note that fits any window.\n");+ const stats = await indexVault(cfg());+ expect("chunkWindow" in stats).toBe(false);+ expect(stats.chunkWindow).toBeUndefined();+});++test("a model with no declared window reports that the check did not run", async () => {+ writeMd(vault, "small.md", "# Small\n\nA short note that fits any window.\n");+ const stats = await indexVault(cfg({ model: UNCURATED_MODEL }));+ const census = stats.chunkWindow;+ expect(census?.verdict).toBe("window-undeclared");+ if (census?.verdict !== "window-undeclared") throw new Error("verdict narrowed wrong");+ expect(census.model).toBe(UNCURATED_MODEL);+ expect(census.chunksMeasured).toBeGreaterThan(0);+ // The failure mode this unit exists to prevent: the same vault under a+ // curated model reports nothing at all, so "silent" must not be the+ // uncurated model's answer too.+ expect("chunkWindow" in (await indexVault(cfg(), { force: true }))).toBe(false);+});++test("semantic search off and the offline embedder both take no census", async () => {+ writeMd(vault, "big.md", `# Big\n\n${oversizeBody(RECOMMENDED_WINDOW)}\n`);+ // Off: nothing is ever embedded, so no window can be exceeded.+ expect("chunkWindow" in (await indexVault(cfg({ enabled: false })))).toBe(false);+ // Local: feature hashing over the whole text - no encoder, no+ // positional limit. A provably absent window, not an unknown one.+ const local = await indexVault(cfg({ provider: "local", model: LOCAL_EMBEDDING_MODEL }), {+ force: true,+ });+ expect("chunkWindow" in local).toBe(false);+});++// ── the warning surface ──────────────────────────────────────────────────────++test("search status warns with the registered exit when chunks exceed the window", async () => {+ writeMd(vault, "big.md", `# Big\n\n${oversizeBody(RECOMMENDED_WINDOW)}\n`);+ await indexVault(cfg());+ const status = await indexStatus(cfg());+ const expectedCommand = requireNextStep("search-chunk-window-overflow").nextCommand;+ const warning = status.warnings.find((w) => w.includes(expectedCommand));+ expect(warning ?? "(no chunk-window warning)").toContain(RECOMMENDED_EMBEDDING_MODEL);+ expect(warning!).toContain(String(RECOMMENDED_WINDOW));+ // The other half of the remedy - the setting that produced the+ // boundaries - is named too, because a rebuild alone re-cuts the same+ // chunks.+ expect(warning!).toContain("search_chunk_size");+});++test("an uncurated model is a status FIELD, never a status warning", async () => {+ writeMd(vault, "small.md", "# Small\n\nA short note.\n");+ await indexVault(cfg({ model: UNCURATED_MODEL }));+ const status = await indexStatus(cfg({ model: UNCURATED_MODEL }));++ // The statement is made, so nothing here reads as a pass...+ expect(status.chunkWindowUndeclared?.verdict).toBe("window-undeclared");+ expect(status.chunkWindowUndeclared!.model).toBe(UNCURATED_MODEL);+ expect(status.chunkWindowUndeclared!.chunksMeasured).toBeGreaterThan(0);++ // ...and it is made WITHOUT spending the warning channel, which is+ // reserved for conditions that name a command. This is the common+ // case - most models are outside the curated table - so a warning+ // here would repeat forever and train operators to skip the channel.+ expect(status.warnings).toEqual([]);+});++test("a clean census sets no status field and no status warning", async () => {+ writeMd(vault, "small.md", "# Small\n\nA short note.\n");+ await indexVault(cfg());+ const status = await indexStatus(cfg());+ expect("chunkWindowUndeclared" in status).toBe(false);+ for (const code of ["search-chunk-window-overflow", "search-chunk-window-undeclared"]) {+ const command = requireNextStep(code).nextCommand;+ expect(`${code}: ${status.warnings.some((w) => w.includes(command))}`).toBe(`${code}: false`);+ }+});++test("measured overflow leaves the undeclared field absent", async () => {+ writeMd(vault, "big.md", `# Big\n\n${oversizeBody(RECOMMENDED_WINDOW)}\n`);+ await indexVault(cfg());+ const status = await indexStatus(cfg());+ // Three distinguishable states on one surface: this one is measured+ // and faulty, so it is a warning and NOT the unmeasurable field.+ expect("chunkWindowUndeclared" in status).toBe(false);+ expect(status.warnings.length).toBeGreaterThan(0);+});++test("the status wire shape carries the field only when the check could not run", async () => {+ writeMd(vault, "small.md", "# Small\n\nA short note.\n");++ await indexVault(cfg({ model: UNCURATED_MODEL }));+ const undeclared = serializeIndexStatus(await indexStatus(cfg({ model: UNCURATED_MODEL })));+ expect(undeclared["chunk_window_undeclared"]).toEqual({+ verdict: "window-undeclared",+ model: UNCURATED_MODEL,+ chunks_measured: 1,+ });++ await indexVault(cfg(), { force: true });+ const curated = serializeIndexStatus(await indexStatus(cfg()));+ expect("chunk_window_undeclared" in curated).toBe(false);+ // The bytes either side of where the key would have landed are the+ // pre-census bytes.+ expect(JSON.stringify(curated)).not.toContain("chunk_window");+});++// ── every verdict names a registered exit ────────────────────────────────────++test("both census verdicts resolve to a registered command", () => {+ const overflow = chunkWindowDiagnosticCode({+ verdict: "over-window",+ model: RECOMMENDED_EMBEDDING_MODEL,+ windowTokens: RECOMMENDED_WINDOW,+ chunksOverWindow: 1,+ chunksMeasured: 1,+ });+ const undeclared = chunkWindowDiagnosticCode({+ verdict: "window-undeclared",+ model: UNCURATED_MODEL,+ chunksMeasured: 1,+ });+ expect(requireNextStep(overflow).nextCommand.startsWith("o2b ")).toBe(true);+ expect(requireNextStep(undeclared).nextCommand.startsWith("o2b ")).toBe(true);+ // Different states, different exits: a rebuild does not answer "no+ // window is declared for this model".+ expect(requireNextStep(overflow).nextCommand).not.toBe(requireNextStep(undeclared).nextCommand);+});diff --git a/tests/core/search/chunk-window-estimate.test.ts b/tests/core/search/chunk-window-estimate.test.tsnew file mode 100644index 00000000..dc33b820--- /dev/null+++ b/tests/core/search/chunk-window-estimate.test.ts@@ -0,0 +1,361 @@+/**+ * What the oversize-chunk census can and cannot decide+ * (what-the-index-already-knew, task H follow-up).+ *+ * The census exists so a truncating provider stops being invisible. Three+ * ways it was still silent about chunks the provider truncates, each+ * reproduced here before it was fixed:+ *+ * - the request path PREFIXES every passage (`passage: ` for the+ * recommended e5 default) and the census did not, so a chunk sitting+ * inside the prefix's width of the window was over in the request and+ * zero in the count;+ * - characters-over-four is not a token estimate on Han text, where a+ * BERT-family tokenizer is roughly character-level. A vault under+ * `BAAI/bge-small-zh-v1.5` was several times over its window and the+ * census emitted nothing on any surface;+ * - SQLite counts code points where the estimator counts UTF-16 code+ * units, so fully-astral text reads HALF its estimated size in the+ * database - a chunk estimated at twice the window was missed.+ *+ * All three shared one shape: the census said nothing, and nothing was+ * defined to mean "every chunk fits". The fix is a two-sided bound - a+ * chunk over the low estimate is counted, a chunk over only the high+ * estimate is reported as undecided - so silence is earned rather than+ * assumed.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";++import { indexStatus } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import {+ estimateTokens,+ textExtent,+ tokenEstimateCeiling,+ tokenEstimateFloor,+ utf8ByteFloorUnderTokenBudget,+} from "../../../src/core/search/embeddings/signature.ts";+import {+ declaredInputWindowTokens,+ E5_PASSAGE_PREFIX,+ passagePrefixSentByProvider,+ RECOMMENDED_EMBEDDING_MODEL,+ resolveEmbeddingPrefixes,+} from "../../../src/core/search/embeddings/presets.ts";+import { requireNextStep } from "../../../src/core/brain/next-step.ts";+import type {+ EmbeddingProviderName,+ ResolvedEmbeddingConfig,+ ResolvedSearchConfig,+} from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("chunk-window-estimate");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++const RECOMMENDED_WINDOW = declaredInputWindowTokens(RECOMMENDED_EMBEDDING_MODEL)!;+const ZH_MODEL = "BAAI/bge-small-zh-v1.5";+const ZH_WINDOW = declaredInputWindowTokens(ZH_MODEL)!;++/** The prefix real config resolution hands the recommended default. */+const RESOLVED_PASSAGE_PREFIX = resolveEmbeddingPrefixes(+ RECOMMENDED_EMBEDDING_MODEL,+ null,+ null,+).passagePrefix;++function cfg(semantic: Partial<ResolvedEmbeddingConfig> = {}): ResolvedSearchConfig {+ return makeConfig({+ vault,+ dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: "https://embeddings.invalid/v1",+ model: RECOMMENDED_EMBEDDING_MODEL,+ apiKey: "test-key",+ dimension: 384,+ passagePrefix: RESOLVED_PASSAGE_PREFIX,+ ...semantic,+ },+ });+}++/** Write one chunk body straight into the index and close it. */+async function seedOneChunk(config: ResolvedSearchConfig, content: string): Promise<void> {+ const store = await Store.open(config, { mode: "write" });+ try {+ const docId = store.upsertDocument({+ path: "seed.md",+ title: "Seed",+ contentHash: "hash-seed",+ mtime: 1,+ size: content.length,+ pageType: null,+ authoredAt: null,+ eventAnchor: null,+ });+ store.replaceChunks(docId, [+ {+ chunkIndex: 0,+ content,+ contentHash: "chunk-seed",+ startLine: 1,+ endLine: 1,+ tokenCount: 1,+ },+ ]);+ } finally {+ await store.close();+ }+}++/** Every operator-visible statement the census makes on the status surface. */+async function censusSurface(config: ResolvedSearchConfig): Promise<{+ warnings: ReadonlyArray<string>;+ undeclared: boolean;+}> {+ const status = await indexStatus(config);+ return { warnings: status.warnings, undeclared: status.chunkWindowUndeclared !== undefined };+}++const OVERFLOW_COMMAND = requireNextStep("search-chunk-window-overflow").nextCommand;++// ── 1. the prefix the provider will actually send ────────────────────────────++test("a chunk pushed over the window BY the instruction prefix is counted", async () => {+ // The recommended multilingual default auto-resolves a 9-character+ // passage prefix, and the request path prepends it before the provider+ // ever counts a token.+ expect(RESOLVED_PASSAGE_PREFIX).toBe(E5_PASSAGE_PREFIX);++ const content = "a".repeat(RECOMMENDED_WINDOW * 4);+ // Without the prefix the estimate lands exactly ON the window; with it,+ // over. The request always carries it.+ expect(estimateTokens([content])).toBe(RECOMMENDED_WINDOW);+ expect(estimateTokens([RESOLVED_PASSAGE_PREFIX + content])).toBeGreaterThan(RECOMMENDED_WINDOW);++ await seedOneChunk(cfg(), content);+ const surface = await censusSurface(cfg());+ const warning = surface.warnings.find((w) => w.includes(OVERFLOW_COMMAND));+ expect(warning ?? "(the census said nothing)").toContain(String(RECOMMENDED_WINDOW));+});++test("a provider that sends no prefix has none subtracted", async () => {+ // ZeroEntropy embeds the raw text: its `embed` takes no kind and+ // prepends nothing, so the same chunk is exactly at the window and the+ // census must stay silent rather than borrow another provider's prefix.+ const zeroentropy = cfg({ provider: "zeroentropy" });+ await seedOneChunk(zeroentropy, "a".repeat(RECOMMENDED_WINDOW * 4));+ const surface = await censusSurface(zeroentropy);+ expect(surface.warnings).toEqual([]);+ expect(surface.undeclared).toBe(false);+});++// ── 2. Han text against a Han model ──────────────────────────────────────────++test("a Han-script vault over its declared window is not silent", async () => {+ // A BERT-family Chinese tokenizer is roughly character-level, so this+ // body is about 3.6x the model's 512-token window. Characters-over-four+ // estimates it at a quarter of that and reports a clean census.+ const han = "文".repeat(Math.round(ZH_WINDOW * 3.6));+ expect(estimateTokens([han])).toBeLessThan(ZH_WINDOW);++ const config = cfg({ model: ZH_MODEL, passagePrefix: "" });+ await seedOneChunk(config, han);+ const surface = await censusSurface(config);+ expect(`${surface.warnings.length} warning(s), undeclared=${surface.undeclared}`).not.toBe(+ "0 warning(s), undeclared=false",+ );+ expect(surface.warnings.join("\n")).toContain(OVERFLOW_COMMAND);+});++// ── 3. the astral bound ──────────────────────────────────────────────────────++test("a fully astral chunk estimated well past the window is not silent", async () => {+ // SQLite's length() counts code points; the estimator counts UTF-16+ // code units. For fully-astral text the database reads HALF the+ // estimated size, so a chunk estimated at nearly twice the window fell+ // under a threshold expressed in code points.+ const astral = "\u{1F600}".repeat(RECOMMENDED_WINDOW * 4 - 48);+ expect(estimateTokens([RESOLVED_PASSAGE_PREFIX + astral])).toBeGreaterThan(+ RECOMMENDED_WINDOW * 1.9,+ );++ await seedOneChunk(cfg(), astral);+ const surface = await censusSurface(cfg());+ expect(`${surface.warnings.length} warning(s), undeclared=${surface.undeclared}`).not.toBe(+ "0 warning(s), undeclared=false",+ );+});++// ── 4. the silence that IS earned ────────────────────────────────────────────++test("an all-ASCII chunk comfortably inside the window still says nothing", async () => {+ await seedOneChunk(cfg(), "a".repeat(RECOMMENDED_WINDOW));+ const surface = await censusSurface(cfg());+ expect(surface.warnings).toEqual([]);+ expect(surface.undeclared).toBe(false);+});++// ── the two bounds, as arithmetic ────────────────────────────────────────────++test("for pure ASCII the two estimates ARE the old one, so the band is empty", () => {+ for (let len = 0; len <= 64; len++) {+ const text = "a".repeat(len);+ const extent = textExtent(text);+ expect(`len=${len} floor=${tokenEstimateFloor(extent)}`).toBe(+ `len=${len} floor=${estimateTokens([text])}`,+ );+ expect(`len=${len} ceiling=${tokenEstimateCeiling(extent)}`).toBe(+ `len=${len} ceiling=${estimateTokens([text])}`,+ );+ }+});++test("the ceiling brackets what the old estimate under-reported on Han and astral text", () => {+ // Han: about one token per character under a BERT-family tokenizer.+ const han = "文".repeat(400);+ expect(estimateTokens([han])).toBe(100);+ expect(tokenEstimateCeiling(textExtent(han))).toBe(400);+ // Astral: SQLite reads half of what String.length reads, and the+ // ceiling covers both readings.+ const astral = "\u{1F600}".repeat(400);+ expect(tokenEstimateFloor(textExtent(astral))).toBe(100);+ expect(estimateTokens([astral])).toBe(200);+ expect(tokenEstimateCeiling(textExtent(astral))).toBe(400);+});++test("the floor never exceeds the ceiling, over every script mixture", () => {+ const alphabets = ["a", "é", "文", "\u{1F600}", " ", "Ж"];+ for (const first of alphabets) {+ for (const second of alphabets) {+ for (const repeats of [0, 1, 3, 17]) {+ const text = (first + second).repeat(repeats);+ const extent = textExtent(text);+ const floor = tokenEstimateFloor(extent);+ const ceiling = tokenEstimateCeiling(extent);+ expect(`${JSON.stringify(text)}: ${floor <= ceiling}`).toBe(+ `${JSON.stringify(text)}: true`,+ );+ }+ }+ }+});++test("the byte prefilter never skips a text either estimate would flag", () => {+ // The query tests the free measurement first and walks only the+ // survivors. That is only sound if a text under the byte floor is+ // provably under both estimates, so prove it by brute force over every+ // script mixture and every length either estimate can straddle.+ const units = ["a", "ab", "é", "文", "\u{1F600}", "Ж文", "a文", "\u{1F600}a"];+ for (const windowTokens of [1, 4, 16, 128, 512]) {+ const floor = utf8ByteFloorUnderTokenBudget(windowTokens);+ for (const unit of units) {+ for (let repeats = 0; repeats <= 40; repeats++) {+ const extent = textExtent(unit.repeat(repeats));+ if (extent.utf8Bytes > floor) continue;+ const flagged =+ tokenEstimateFloor(extent) > windowTokens || tokenEstimateCeiling(extent) > windowTokens;+ expect(`w=${windowTokens} ${JSON.stringify(unit)}x${repeats}: ${flagged}`).toBe(+ `w=${windowTokens} ${JSON.stringify(unit)}x${repeats}: false`,+ );+ }+ }+ }+});++// ── the store evaluates exactly those two predicates ─────────────────────────++test("the SQL tally reproduces the two predicates chunk for chunk", async () => {+ const windowTokens = 16;+ const prefix = RESOLVED_PASSAGE_PREFIX;+ const budget = windowTokens * 4;+ const bodies: string[] = [];+ // Straddle the low threshold in ASCII, then the high one in three+ // other scripts, then the empty chunk.+ for (const len of [budget - prefix.length - 1, budget - prefix.length, budget, budget + 1]) {+ bodies.push("a".repeat(len));+ }+ for (const unit of ["文", "\u{1F600}", "Ж", "é"]) {+ for (const count of [1, windowTokens - 1, windowTokens, windowTokens + 1, budget]) {+ bodies.push(unit.repeat(count));+ }+ }+ bodies.push("");++ const store = await Store.open(cfg(), { mode: "write" });+ try {+ const docId = store.upsertDocument({+ path: "mixed.md",+ title: null,+ contentHash: "hash-mixed",+ mtime: 1,+ size: 1,+ pageType: null,+ authoredAt: null,+ eventAnchor: null,+ });+ store.replaceChunks(+ docId,+ bodies.map((content, i) => ({+ chunkIndex: i,+ content,+ contentHash: `h${i}`,+ startLine: i + 1,+ endLine: i + 1,+ tokenCount: 1,+ })),+ );++ let overWindow = 0;+ let undecided = 0;+ for (const body of bodies) {+ const extent = textExtent(prefix + body);+ if (tokenEstimateFloor(extent) > windowTokens) overWindow++;+ else if (tokenEstimateCeiling(extent) > windowTokens) undecided++;+ }+ // The fixture must actually exercise both arms, or this test would+ // pass over a census that never counted anything.+ expect(`over=${overWindow > 0} undecided=${undecided > 0}`).toBe("over=true undecided=true");+ expect(store.chunkWindowTally(windowTokens, prefix)).toEqual({ overWindow, undecided });+ } finally {+ await store.close();+ }+});++// ── the prefix belongs to the backend, not to the config ─────────────────────++test("every embedding backend declares whether it sends a passage prefix", () => {+ const providers: ReadonlyArray<EmbeddingProviderName> = [+ "openai-compat",+ "zeroentropy",+ "local",+ "disabled",+ ];+ const sent = providers.map(+ (p) => `${p}=${JSON.stringify(passagePrefixSentByProvider(p, "P: "))}`,+ );+ // Only the OpenAI-compatible backend implements `prefixFor`; the rest+ // send the raw text, so a configured prefix must not be subtracted for+ // them. Adding a backend fails to compile in `presets.ts` rather than+ // defaulting into this list silently.+ expect(sent).toEqual(['openai-compat="P: "', 'zeroentropy=""', 'local=""', 'disabled=""']);+ // An absent configured prefix is no prefix; an empty one is disabled.+ // Both match `OpenAICompatProvider.prefixFor` exactly.+ expect(passagePrefixSentByProvider("openai-compat", undefined)).toBe("");+ expect(passagePrefixSentByProvider("openai-compat", "")).toBe("");+});diff --git a/tests/core/search/duplicate-passage-merge.test.ts b/tests/core/search/duplicate-passage-merge.test.tsnew file mode 100644index 00000000..739139a9--- /dev/null+++ b/tests/core/search/duplicate-passage-merge.test.ts@@ -0,0 +1,178 @@+/**+ * Exact-duplicate passage merge (what-the-index-already-knew, task D).+ *+ * `chunks.content_hash` has been written on every chunk since schema v1+ * and never read back. Two byte-identical passages therefore occupied two+ * result slots: MMR demoted the second one, but demotion still costs a+ * slot in the caller's window. The merge folds them into one row that+ * names every location the passage was found at, and it runs before the+ * diversity rerank so the quadratic Jaccard loop sees the shrunken pool.+ *+ * What must NOT change: a near-duplicate (different bytes, different+ * hash) is untouched and still handled by MMR; two distinct passages in+ * one file stay two rows; a corpus with no duplicates produces rows that+ * carry no `duplicates` key at all.+ */++import { test, expect, beforeEach, afterEach } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import type { BrainSearchResult, ResolvedSearchConfig } from "../../../src/core/search/types.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("duplicate-passage-merge");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++/** The passage duplicated verbatim across files in the tests below. */+const PASSAGE =+ "# Quantum entanglement\n\n" ++ "Quantum entanglement links two particles so that measuring one fixes " ++ "the other, regardless of the distance separating them. The correlation " ++ "is not a signal and carries no information on its own.\n";++async function indexed(overrides: Partial<Parameters<typeof makeConfig>[0]> = {}) {+ const cfg: ResolvedSearchConfig = makeConfig({ vault, dbPath, ...overrides });+ await indexVault(cfg);+ return cfg;+}++function locations(r: BrainSearchResult): ReadonlyArray<string> {+ return (r.duplicates ?? []).map((d) => d.path);+}++test("two byte-identical passages in two files return one row naming both locations", async () => {+ writeMd(vault, "physics/entanglement.md", PASSAGE);+ writeMd(vault, "archive/entanglement-copy.md", PASSAGE);+ const cfg = await indexed();++ const outcome = await search(cfg, { query: "quantum entanglement", limit: 10 });+ const hits = outcome.results.filter(+ (r) => r.path.endsWith("entanglement.md") || r.path.endsWith("entanglement-copy.md"),+ );++ expect(hits.length).toBe(1);+ const survivor = hits[0]!;+ const named = [survivor.path, ...locations(survivor)].toSorted();+ expect(named).toEqual(["archive/entanglement-copy.md", "physics/entanglement.md"]);+ expect(survivor.duplicates![0]!.startLine).toBeGreaterThan(0);+ expect(survivor.duplicates![0]!.chunkId).not.toBe(survivor.chunkId);+});++test("the same passage at three paths is reported as three locations on one row", async () => {+ writeMd(vault, "a/entanglement.md", PASSAGE);+ writeMd(vault, "b/entanglement.md", PASSAGE);+ writeMd(vault, "c/entanglement.md", PASSAGE);+ const cfg = await indexed();++ const outcome = await search(cfg, { query: "quantum entanglement", limit: 10 });+ expect(outcome.results.length).toBe(1);+ const survivor = outcome.results[0]!;+ expect([survivor.path, ...locations(survivor)].toSorted()).toEqual([+ "a/entanglement.md",+ "b/entanglement.md",+ "c/entanglement.md",+ ]);+});++/** A section long enough to become a chunk of its own at chunkSize 800. */+function section(label: string): string {+ return `## ${label}\n\n` + `Quantum ${label} is discussed here at length. `.repeat(20) + "\n";+}++test("two distinct passages from one file still return two rows", async () => {+ // chunkSize 800 / chunkMinSize 100: two sections well past the minimum+ // split into two chunks of the same document.+ writeMd(vault, "physics/notes.md", `${section("superposition")}\n${section("decoherence")}\n`);+ const cfg = await indexed();++ const outcome = await search(cfg, { query: "quantum", limit: 10 });+ const rows = outcome.results.filter((r) => r.path === "physics/notes.md");+ expect(rows.length).toBeGreaterThan(1);+ for (const r of rows) expect(r.duplicates).toBeUndefined();+});++test("a near-duplicate has a different hash, is not merged, and both rows survive", async () => {+ writeMd(vault, "physics/entanglement.md", PASSAGE);+ // One word differs: different bytes, different sha256, no merge.+ writeMd(vault, "archive/entanglement-near.md", PASSAGE.replace("particles", "photons"));+ const cfg = await indexed();++ const outcome = await search(cfg, { query: "quantum entanglement", limit: 10 });+ const paths = outcome.results.map((r) => r.path).toSorted();+ expect(paths).toEqual(["archive/entanglement-near.md", "physics/entanglement.md"]);+ for (const r of outcome.results) expect(r.duplicates).toBeUndefined();+});++test("two records sharing a body but not their frontmatter are two rows, not one", async () => {+ // Frontmatter is stripped before chunking, so these two chunks hash+ // identically - but they are distinct records, and the ranker scores+ // them differently on the strength of exactly the frontmatter the chunk+ // hash cannot see. The document hash is what keeps them apart.+ writeMd(vault, "left.md", `---\ntitle: Left\nfreshness_trend: stable\n---\n\n${PASSAGE}`);+ writeMd(vault, "right.md", `---\ntitle: Right\nfreshness_trend: weakening\n---\n\n${PASSAGE}`);+ const cfg = await indexed();++ const outcome = await search(cfg, { query: "quantum entanglement", limit: 10 });+ expect(outcome.results.map((r) => r.path).toSorted()).toEqual(["left.md", "right.md"]);+ for (const r of outcome.results) expect(r.duplicates).toBeUndefined();+});++test("a corpus with no duplicates carries no duplicates key on any row", async () => {+ writeMd(vault, "one.md", "# Alpha\n\nQuantum alpha notes about the first topic entirely.\n");+ writeMd(vault, "two.md", "# Beta\n\nQuantum beta notes about the second topic entirely.\n");+ const cfg = await indexed();++ const outcome = await search(cfg, { query: "quantum notes", limit: 10 });+ expect(outcome.results.length).toBe(2);+ for (const r of outcome.results) {+ // Absent, not present-and-empty: the key is missing from the row and+ // from the bytes any surface would serialize it to.+ expect(Object.hasOwn(r, "duplicates")).toBe(false);+ expect(JSON.stringify(r)).not.toContain("duplicates");+ }+});++/**+ * Window sizes the shortfall is checked at: at both, the two duplicates+ * are the top-ranked candidates and fill part of a pool capped at `limit`.+ */+const LIMITS_UNDER_TEST = [2, 3] as const;++/** A weaker match: the query terms occur once, buried in unrelated prose. */+function weakMatch(label: string): string {+ const filler = "Assorted laboratory bookkeeping and scheduling minutes for the reading group. ";+ return `# Reading group ${label}\n\n${filler.repeat(12)}One quantum entanglement item, ${label}.\n`;+}++test("a limit-N request over a pool holding duplicates still returns N rows", async () => {+ // MMR and traversal off: the rank pool collapses to exactly `limit`, so+ // the two top-ranked duplicates fill the whole window and merging them+ // shrinks the answer below `limit` unless the pool widens and re-runs.+ writeMd(vault, "dup-one.md", PASSAGE);+ writeMd(vault, "dup-two.md", PASSAGE);+ writeMd(vault, "other-a.md", weakMatch("alpha"));+ writeMd(vault, "other-b.md", weakMatch("beta"));+ const cfg = await indexed({ mmrLambda: 1, maxHops: 0 });++ const outcomes = await Promise.all(+ LIMITS_UNDER_TEST.map((limit) => search(cfg, { query: "quantum entanglement", limit })),+ );+ for (const [i, outcome] of outcomes.entries()) {+ const limit = LIMITS_UNDER_TEST[i]!;+ expect(outcome.results.length).toBe(limit);+ expect(new Set(outcome.results.map((r) => r.chunkId)).size).toBe(limit);+ }+});diff --git a/tests/core/search/entities.test.ts b/tests/core/search/entities.test.tsindex 9084f0f8..05ddcb0a 100644--- a/tests/core/search/entities.test.ts+++ b/tests/core/search/entities.test.ts@@ -54,3 +54,39 @@ test("dedupes and drops single-character noise", () => {test("returns a frozen array", () => {expect(Object.isFrozen(extractEntities("Open Second Brain"))).toBe(true);});++test("a long unbroken letter run does not make extraction backtrack catastrophically", () => {+ // A Han, Thai or Lao paragraph is one unbroken `\p{L}` run carrying no+ // digits, which is the worst input for a naive `\p{L}+\p{N}+`: the engine+ // retries every shorter prefix at every start position. Measured on this+ // exact input before the alternatives were made atomic: 11.7 seconds and+ // zero matches. A 500-note Chinese vault could not be indexed at all.+ //+ // The bound is deliberately loose - two orders of magnitude above the+ // post-fix cost - so this asserts "not catastrophic" rather than a+ // machine-specific number, and cannot flake on a slow runner.+ const CATASTROPHE_GUARD_MS = 1000;+ const han = "中".repeat(2160);++ const started = performance.now();+ const entities = extractEntities(han);+ const elapsed = performance.now() - started;++ // Non-vacuous: this input really does drive the alternative that used to+ // backtrack, and really does yield nothing, so the guard is timing the+ // failure path rather than an early exit.+ expect(entities).toEqual([]);+ expect(`${elapsed < CATASTROPHE_GUARD_MS} in ${Math.round(elapsed)}ms`).toBe(+ `true in ${Math.round(elapsed)}ms`,+ );+});++test("making the letter run atomic changed no extraction result", () => {+ // The guard above only says the regex is fast. This says it is the same+ // regex: every shape the mixed-script alternatives exist to catch still+ // comes back, across scripts.+ expect(extractEntities("H2O and GPT4 and v2 and NASA")).toEqual(["h2o", "gpt4", "nasa", "v2"]);+ expect(extractEntities("東京2020 and 2020東京")).toEqual(["東京2020", "2020東京"]);+ expect(extractEntities("тест2 and 2тест")).toEqual(["тест2", "2тест"]);+ expect(extractEntities("abc123def456")).toEqual(["abc123", "def456"]);+});diff --git a/tests/core/search/feedback.test.ts b/tests/core/search/feedback.test.tsindex 0f00f01d..be73dbd5 100644--- a/tests/core/search/feedback.test.ts+++ b/tests/core/search/feedback.test.ts@@ -26,6 +26,7 @@ import {LEARNED_WEIGHT_MIN,LEARNED_WEIGHT_MAX,type RecallFeedbackEvent,+ writeLearnedWeights,} from "../../../src/core/search/feedback.ts";import { indexVault } from "../../../src/core/search/indexer.ts";import { search } from "../../../src/core/search/search.ts";@@ -207,6 +208,28 @@ describe("derived weights file", () => {expect(loadFeedbackEvents(vault)).toHaveLength(1);});+ test("a hand-edited multiplier outside the fold's own bounds is refused", () => {+ // The fold clamps every multiplier into [LEARNED_WEIGHT_MIN,+ // LEARNED_WEIGHT_MAX], so a value outside that range is one+ // computeLearnedWeights could never have produced. Under rank fusion a+ // zero deletes a lane and a negative inverts its rank order, so the+ // reader refuses the file rather than composing with it.+ const sound = computeLearnedWeights([event({})]);+ writeLearnedWeights(vault, sound);+ expect(readLearnedWeights(vault)).not.toBeNull();++ for (const bad of [0, -1, LEARNED_WEIGHT_MAX + 0.1, LEARNED_WEIGHT_MIN - 0.1]) {+ writeLearnedWeights(vault, { ...sound, keywordMul: bad });+ expect(readLearnedWeights(vault)).toBeNull();+ }++ // The bound is inclusive at both ends - the fold can emit exactly these.+ for (const edge of [LEARNED_WEIGHT_MIN, LEARNED_WEIGHT_MAX]) {+ writeLearnedWeights(vault, { ...sound, keywordMul: edge });+ expect(readLearnedWeights(vault)).not.toBeNull();+ }+ });+test("fingerprint changes when weights change and is stable otherwise", () => {const before = learnedWeightsFingerprint(vault);expect(learnedWeightsFingerprint(vault)).toBe(before);diff --git a/tests/core/search/pool-total.test.ts b/tests/core/search/pool-total.test.tsnew file mode 100644index 00000000..22dd24b5--- /dev/null+++ b/tests/core/search/pool-total.test.ts@@ -0,0 +1,80 @@+/**+ * `SearchOutcome.total` is the pre-truncation ranked pool size+ * (what-the-index-already-knew, task F).+ *+ * Before this unit `total` was assigned the length of the returned array,+ * so it carried no information at all: a caller could not tell "the vault+ * holds three matches" from "the vault holds thirty and you were handed+ * three". The pool the final `limit` slice cuts from is a property of the+ * answer, not a diagnostic, so it is reported inline on every outcome.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++/** Matching notes, comfortably above the `limit` the queries below use. */+const MATCHING_NOTES = 12;+const LIMIT = 3;++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(async () => {+ ({ vault, dbPath, cleanup } = createTempVault("pool-total"));+ for (let i = 0; i < MATCHING_NOTES; i += 1) {+ writeMd(+ vault,+ `notes/note-${i}.md`,+ `---\ntitle: Note ${i}\n---\n\n# Note ${i}\n\nThe widget calibration routine number ${i}.\n`,+ );+ }+ await indexVault(makeConfig({ vault, dbPath, maxHops: 0 }), {});+});++afterEach(() => cleanup());++function config() {+ return makeConfig({ vault, dbPath, maxHops: 0 });+}++test("total is the ranked pool the limit slice cut from, not the row count", async () => {+ const out = await search(config(), { query: "widget calibration", limit: LIMIT });++ expect(out.results.length).toBe(LIMIT);+ expect(out.total).toBeGreaterThan(out.results.length);+ // The pool is the candidate set retrieval actually ranked, which is+ // widened past `limit` but still bounded by the lane fetch - so it is a+ // lower bound on the corpus matches, never an over-count.+ expect(out.total).toBeLessThanOrEqual(MATCHING_NOTES);+});++test("a limit wider than the pool reports the pool, so total never exceeds the corpus", async () => {+ const out = await search(config(), { query: "widget calibration", limit: MATCHING_NOTES + 5 });++ expect(out.results.length).toBe(MATCHING_NOTES);+ expect(out.total).toBe(MATCHING_NOTES);+});++test("cards disclosure reports the same pool as the full path", async () => {+ const full = await search(config(), { query: "widget calibration", limit: LIMIT });+ const cards = await search(config(), {+ query: "widget calibration",+ limit: LIMIT,+ disclosure: "cards",+ });++ expect(cards.cards!.length).toBe(LIMIT);+ expect(cards.total).toBe(full.total);+ expect(cards.total).toBeGreaterThan(cards.cards!.length);+});++test("a query with no candidates still reports zero", async () => {+ const out = await search(config(), { query: "zzzznonexistentterm", limit: LIMIT });++ expect(out.results).toEqual([]);+ expect(out.total).toBe(0);+});diff --git a/tests/core/search/rank-clock-determinism.test.ts b/tests/core/search/rank-clock-determinism.test.tsnew file mode 100644index 00000000..d0be42f1--- /dev/null+++ b/tests/core/search/rank-clock-determinism.test.ts@@ -0,0 +1,341 @@+/**+ * Ranking determinism: a search ranks against the instant its request+ * resolved, never against a wall clock the ranker reads for itself.+ *+ * The defect this locks down: `rankResults` defaulted `nowMs` to+ * `Date.now()` and the assembly stage never passed the request's+ * already-resolved instant, so the freshness layer folded a second,+ * later clock reading into every live score. Two rows that tie on every+ * intrinsic signal then order by whatever the clock happened to say at+ * the moment the ranker ran, and the tie-break ladder's first rung is an+ * exact float comparison.+ *+ * Three assertions, in increasing strength:+ * 1. the acceptance lock - two live searches over a fixture built to+ * tie return the same rows in the same order with the query cache+ * off;+ * 2. the census - during one `search()` call the wall clock is read+ * only where the request is resolved, and never from the ranker;+ * 3. the direct assembly test - `assembleRankedResults` is a pure+ * function of its input, so moving the process wall clock between+ * two invocations with the same input cannot change the ordering.+ *+ * A fourth covers the companion defect one layer down: the keyword,+ * trigram and vector candidate queries all truncate on a score column+ * that is not a total order, so each ends its sort on `chunks.id`.+ *+ * What that fourth test can and cannot witness, stated rather than+ * implied. It DOES witness the vector lane, where the fix is load-bearing+ * for a reason no ORDER BY can supply: the k-nearest selection happens+ * before SQL sees the rows, so the clause needs an over-fetch beside it+ * and removing either is caught here. For the keyword and trigram lanes+ * it witnesses only that a WRONG tiebreak is caught - removing the clause+ * entirely leaves this test green, because FTS5 scans a MATCH in rowid+ * order and SQLite's sorter is stable, so the natural order for a fixture+ * of this shape already equals `chunks.id ASC`. Those two clauses are+ * therefore defensive: they pin an order the engine currently gives for+ * free and is under no contract to keep. No fixture in this shape can+ * discriminate them, and claiming otherwise would be evidence this file+ * does not have.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { utimesSync } from "node:fs";+import { join } from "node:path";++import { indexVault, search } from "../../../src/core/search/index.ts";+import { assembleRankedResults } from "../../../src/core/search/pipeline/assemble.ts";+import { collectCandidateSignals } from "../../../src/core/search/pipeline/candidate-signals.ts";+import { Store } from "../../../src/core/search/store.ts";+import { planTrigramPrefilter } from "../../../src/core/search/trigram-prefilter.ts";+import type { KeywordHit } from "../../../src/core/search/store.ts";+import type { BrainSearchResult } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++/**+ * The only module that may read the wall clock on the `search()` path.+ * Every time-resolving decision in a call is derived from the single+ * instant resolved there; anything else re-reading the clock mid-pipeline+ * makes one call disagree with itself.+ */+const CLOCK_OWNER = "src/core/search/pipeline/request.ts";++const DAY_MS = 24 * 60 * 60 * 1000;+const SECONDS_PER_DAY = 24 * 60 * 60;++/** Two candidates a hair apart in BM25, and a third that sets the range. */+const BM25_LEAD = -5;+const BM25_RUNNER_UP = -4.99;+const BM25_FLOOR = -1;++/** Ages that put the recency layer above, then below, the BM25 gap. */+const STALE_AGE_DAYS = 60;+const CLOCK_JUMP_DAYS = 3650;++/** More tied candidates than the top-K asks for, so the cut has to choose. */+const TIED_DOCUMENT_COUNT = 8;+const TIED_TOP_K = 3;+const SEMANTIC_DIMENSION = 256;+const TIED_QUERY_TEXT = "migration rollback";+const TIED_FTS_QUERY = '"migration" AND "rollback"';++let tmp: ReturnType<typeof createTempVault>;++beforeEach(() => {+ tmp = createTempVault("rank-clock-determinism");+});++afterEach(() => {+ tmp.cleanup();+});++function identityProjection(results: ReadonlyArray<BrainSearchResult>) {+ return results.map((r) => `${r.path}#${r.chunkId}`);+}++// ---------------------------------------------------------------------------+// 1. Acceptance lock: two live searches over a genuine score tie.+// ---------------------------------------------------------------------------++test("two live searches over a scored tie return the same rows in the same order", async () => {+ // Byte-identical bodies in two files: identical term frequencies and+ // identical document lengths, so FTS5 scores them identically, and the+ // same indexing pass gives them the same second-resolution mtime. Every+ // ranked layer - keyword, recency, link, tag - is therefore equal, and+ // the composite score is an exact float tie.+ const body = "The rollback strategy reverts the migration in one step.\n";+ writeMd(tmp.vault, "left.md", `---\ntitle: Left\n---\n\n${body}`);+ writeMd(tmp.vault, "right.md", `---\ntitle: Right\n---\n\n${body}`);+ writeMd(tmp.vault, "other.md", `---\ntitle: Other\n---\n\nUnrelated prose about gardening.\n`);+ const config = makeConfig({ vault: tmp.vault, dbPath: tmp.dbPath, cacheEnabled: false });+ await indexVault(config, {});++ const first = await search(config, { query: "rollback strategy migration", limit: 10 });+ const second = await search(config, { query: "rollback strategy migration", limit: 10 });++ // The fixture really does tie - otherwise this test would be asserting+ // determinism over an ordering nothing could have disturbed.+ expect(first.results.length).toBeGreaterThanOrEqual(2);+ expect(first.results[0]!.score).toBe(first.results[1]!.score);++ expect(identityProjection(second.results)).toEqual(identityProjection(first.results));+});++// ---------------------------------------------------------------------------+// 2. Census: who reads the wall clock during one search.+// ---------------------------------------------------------------------------++test("one search() call reads the wall clock only where the request is resolved", async () => {+ writeMd(tmp.vault, "a.md", "---\ntitle: A\n---\n\nalpha beta gamma delta\n");+ writeMd(tmp.vault, "b.md", "---\ntitle: B\n---\n\nalpha beta gamma epsilon\n");+ const config = makeConfig({ vault: tmp.vault, dbPath: tmp.dbPath, cacheEnabled: false });+ await indexVault(config, {});++ const realNow = Date.now;+ const sites = new Set<string>();+ let capturing = false;+ const record = (): void => {+ if (!capturing) return;+ for (const frame of (new Error().stack ?? "").split("\n").slice(1)) {+ const match = /\((\/.*?\/(src\/core\/.*?\.ts)):\d+:\d+\)/.exec(frame);+ if (match) {+ sites.add(match[2]!);+ return;+ }+ }+ sites.add("unattributed");+ };+ Date.now = () => {+ record();+ return realNow();+ };+ try {+ capturing = true;+ await search(config, { query: "alpha beta gamma", limit: 5 });+ capturing = false;+ } finally {+ Date.now = realNow;+ }++ // Non-vacuous: the instrumentation did observe the clock being read.+ expect(sites.size).toBeGreaterThan(0);+ expect([...sites].toSorted()).toEqual([CLOCK_OWNER]);+});++// ---------------------------------------------------------------------------+// 3. Direct assembly test: same input, same ordering, whatever the clock.+// ---------------------------------------------------------------------------++test("assembleRankedResults orders on the injected instant, not the process clock", async () => {+ // `lead` wins on BM25 by a hair; `runnerUp` is fresh and `lead` is+ // stale, so at the pinned instant the recency layer more than covers+ // the BM25 gap and `runnerUp` ranks first. Far enough into the future+ // both freshness boosts collapse to zero and the BM25 gap decides -+ // the ordering inverts. That inversion is the observable the injected+ // clock has to suppress.+ writeMd(tmp.vault, "lead.md", "---\ntitle: Lead\n---\n\nalpha beta gamma alpha beta gamma\n");+ writeMd(tmp.vault, "runner-up.md", "---\ntitle: Runner Up\n---\n\nalpha beta gamma\n");+ writeMd(+ tmp.vault,+ "floor.md",+ "---\ntitle: Floor\n---\n\nalpha and a long tail of other words\n",+ );+ const nowMs = Date.now();+ const staleSeconds = (nowMs - STALE_AGE_DAYS * DAY_MS) / 1000;+ utimesSync(join(tmp.vault, "lead.md"), staleSeconds, staleSeconds);+ const config = makeConfig({+ vault: tmp.vault,+ dbPath: tmp.dbPath,+ mmrLambda: 1,+ maxHops: 0,+ cacheEnabled: false,+ });+ await indexVault(config, {});++ const store = await Store.open(config, { mode: "read" });+ try {+ const documents = store.listDocuments();+ const chunkIdFor = (path: string): number => {+ const doc = documents.get(path);+ if (doc === undefined) throw new Error(`fixture document missing from index: ${path}`);+ const chunks = store.chunksForDocument(doc.id);+ const first = chunks[0];+ if (first === undefined) throw new Error(`fixture document has no chunk: ${path}`);+ return first.id;+ };+ const leadChunk = chunkIdFor("lead.md");+ const runnerUpChunk = chunkIdFor("runner-up.md");+ const floorChunk = chunkIdFor("floor.md");+ const ids = [leadChunk, runnerUpChunk, floorChunk];+ const hydrated = store.hydrateChunks(ids);+ // The fixture's staleness has to survive indexing, or the two+ // candidates would share a freshness boost and nothing below could+ // move.+ const staleMtime = hydrated.get(leadChunk)!.mtime;+ const freshMtime = hydrated.get(runnerUpChunk)!.mtime;+ expect(freshMtime - staleMtime).toBeGreaterThanOrEqual((STALE_AGE_DAYS - 1) * SECONDS_PER_DAY);++ const documentIdFor = (chunkId: number): number => hydrated.get(chunkId)!.documentId;+ const keywordHits: ReadonlyArray<KeywordHit> = [+ { chunkId: leadChunk, documentId: documentIdFor(leadChunk), bm25: BM25_LEAD },+ { chunkId: runnerUpChunk, documentId: documentIdFor(runnerUpChunk), bm25: BM25_RUNNER_UP },+ { chunkId: floorChunk, documentId: documentIdFor(floorChunk), bm25: BM25_FLOOR },+ ];+ const frontmatterCache = new Map();+ const signals = collectCandidateSignals({+ store,+ config,+ ids,+ hydrated,+ query: "alpha beta gamma",+ frontmatterCache,+ temporalIntentActive: false,+ declaredEventTimeMs: () => null,+ });+ const input = {+ store,+ config,+ opts: { query: "alpha beta gamma", limit: ids.length },+ keywordHits,+ semanticHits: [],+ hydrated,+ signals,+ relationalRankedChunkIds: [],+ weightProfile: undefined,+ sessionFocus: null,+ semanticEnabled: false,+ structured: undefined,+ frontmatterCache,+ timeRange: null,+ temporalIntent: null,+ declaredEventTimeMs: () => null,+ limit: ids.length,+ nowMs,+ } as const;++ const realNow = Date.now;+ let atPinnedInstant: ReadonlyArray<BrainSearchResult>;+ let afterClockJump: ReadonlyArray<BrainSearchResult>;+ try {+ Date.now = () => nowMs;+ atPinnedInstant = assembleRankedResults(input);+ Date.now = () => nowMs + CLOCK_JUMP_DAYS * DAY_MS;+ afterClockJump = assembleRankedResults(input);+ } finally {+ Date.now = realNow;+ }++ // At the injected instant the fresh runner-up outranks the stale+ // BM25 leader: the fixture is genuinely clock-sensitive, so a ranker+ // that re-read the clock would have something to get wrong.+ expect(identityProjection(atPinnedInstant).slice(0, 2)).toEqual([+ `runner-up.md#${runnerUpChunk}`,+ `lead.md#${leadChunk}`,+ ]);+ expect(identityProjection(afterClockJump)).toEqual(identityProjection(atPinnedInstant));+ expect(afterClockJump.map((r) => r.score)).toEqual(atPinnedInstant.map((r) => r.score));+ } finally {+ await store.close();+ }+});++// ---------------------------------------------------------------------------+// 4. The three candidate queries end their ordering in a unique column.+// ---------------------------------------------------------------------------++test("the three candidate queries cut their top-K on a total order", async () => {+ // Byte-identical bodies across more documents than the top-K asks+ // for: every candidate scores the same BM25 in `chunk_fts` and in+ // `chunk_trigram`, and embeds to the same vector, so the score column+ // alone cannot decide which rows survive the `LIMIT`. Only the unique+ // final sort key can, and it is `chunks.id` ascending.+ const body = "Deterministic paragraph about migration rollback strategy planning.\n";+ for (let i = 0; i < TIED_DOCUMENT_COUNT; i++) {+ writeMd(tmp.vault, `tied-${i}.md`, `---\ntitle: Tied ${i}\n---\n\n${body}`);+ }+ const config = makeConfig({+ vault: tmp.vault,+ dbPath: tmp.dbPath,+ semantic: { enabled: true, provider: "local", dimension: SEMANTIC_DIMENSION },+ trigramPrefilterEnabled: true,+ trigramPrefilterMinChunks: 0,+ });+ await indexVault(config, { embeddings: true });++ const store = await Store.open(config, { mode: "read" });+ try {+ // The whole tied set first, so the expectation is derived from the+ // rows that actually tie rather than from every chunk in the vault+ // (each fixture note also yields a frontmatter chunk, which matches+ // nothing here).+ const wholeSet = store.keywordTopK(TIED_FTS_QUERY, { limit: TIED_DOCUMENT_COUNT * 2 });+ expect(wholeSet.length).toBe(TIED_DOCUMENT_COUNT);+ // Non-vacuous: the lane really did tie, so the cut below had nothing+ // but the unique key to go on.+ expect(new Set(wholeSet.map((h) => h.bm25)).size).toBe(1);+ const tiedIds = wholeSet.map((h) => h.chunkId).toSorted((a, b) => a - b);+ const expected = tiedIds.slice(0, TIED_TOP_K);++ const keyword = store.keywordTopK(TIED_FTS_QUERY, { limit: TIED_TOP_K });+ expect(keyword.map((h) => h.chunkId)).toEqual(expected);++ const trigramPlan = planTrigramPrefilter(TIED_QUERY_TEXT);+ if (trigramPlan.mode !== "match") throw new Error("expected a trigram match plan");+ const trigram = store.trigramCandidates(trigramPlan.ftsQuery, { limit: TIED_TOP_K });+ expect(trigram.warnings).toEqual([]);+ expect(new Set(trigram.hits.map((h) => h.bm25)).size).toBe(1);+ expect(trigram.hits.map((h) => h.chunkId)).toEqual(expected);++ // The semantic lane only ranks when sqlite-vec is present; without+ // it there is no KNN query to order and the assertion would be+ // meaningless rather than passing.+ expect(store.vecLoaded()).toBe(true);+ const vector = store.embeddingForChunk(tiedIds[0]!);+ if (vector === null) throw new Error("fixture chunk has no embedding");+ const semantic = store.semanticTopK(vector, { limit: TIED_TOP_K });+ expect(new Set(semantic.map((h) => h.distance)).size).toBe(1);+ expect(semantic.map((h) => h.chunkId)).toEqual(expected);+ } finally {+ await store.close();+ }+});diff --git a/tests/core/search/rrf-intent-weighting.test.ts b/tests/core/search/rrf-intent-weighting.test.tsnew file mode 100644index 00000000..17ac9e25--- /dev/null+++ b/tests/core/search/rrf-intent-weighting.test.ts@@ -0,0 +1,221 @@+/**+ * `rrf` fusion honours the per-query intent profile (Task B).+ *+ * Reciprocal-rank fusion used to drop the intent multipliers on the floor:+ * the relevance term was the raw fused rank sum, so a caller who quoted a+ * phrase got no lexical preference at all under `search_fusion_mode: rrf`,+ * while the same query in the default `linear` mode did. These tests pin+ * the three properties that fix has to hold at once:+ *+ * 1. the two fusion modes agree about a verbatim keyword hit when the+ * query plan classified the query as `exact`;+ * 2. a neutral profile reproduces classic, weightless RRF exactly - the+ * weighting is inert until a query actually declares an intent;+ * 3. the default `linear` path is untouched by the rrf knobs.+ */++import { test, expect } from "bun:test";++import { rankResults } from "../../../src/core/search/ranker.ts";+import { rrfFuse, DEFAULT_RRF_K } from "../../../src/core/search/fusion.ts";+import { buildQueryPlan, NEUTRAL_PROFILE } from "../../../src/core/search/query-plan.ts";+import { SearchError } from "../../../src/core/search/search-error.ts";+import type { KeywordHit, SemanticHit, HydratedChunk } from "../../../src/core/search/store.ts";+import type { BrainSearchResult } from "../../../src/core/search/types.ts";++const NOW = 1_750_000_000_000;+/** A year old: the recency layer contributes the same value to every row. */+const OLD = NOW / 1000 - 365 * 24 * 3600;++const KEYWORD_WEIGHT = 0.6;+const SEMANTIC_WEIGHT = 0.4;+const LIMIT = 10;++/** Chunk ids, named for the role each one plays in the fixture. */+const TOP_KEYWORD = 1;+/** The verbatim hit under test: strong in the keyword lane, absent from semantic. */+const VERBATIM = 2;+const WEAK_KEYWORD = 3;+/** The rival: sole occupant of the semantic lane, so rank 1 there. */+const SEMANTIC_ONLY = 4;++function hyd(chunkId: number): HydratedChunk {+ return Object.freeze({+ chunkId,+ documentId: chunkId * 10,+ path: `doc${chunkId}.md`,+ title: `Doc ${chunkId}`,+ content: `chunk ${chunkId}`,+ startLine: 1,+ endLine: 1,+ mtime: OLD,+ });+}++const KEYWORD: ReadonlyArray<KeywordHit> = [+ { chunkId: TOP_KEYWORD, documentId: TOP_KEYWORD * 10, bm25: -10 },+ { chunkId: VERBATIM, documentId: VERBATIM * 10, bm25: -9.5 },+ { chunkId: WEAK_KEYWORD, documentId: WEAK_KEYWORD * 10, bm25: -1 },+];+const SEMANTIC: ReadonlyArray<SemanticHit> = [+ { chunkId: SEMANTIC_ONLY, documentId: SEMANTIC_ONLY * 10, distance: 0.6 },+];+const HYDRATED: ReadonlyMap<number, HydratedChunk> = new Map(+ [TOP_KEYWORD, VERBATIM, WEAK_KEYWORD, SEMANTIC_ONLY].map((id) => [id, hyd(id)]),+);++function inputs() {+ return {+ keyword: KEYWORD,+ semantic: SEMANTIC,+ hydrated: HYDRATED,+ inboundLinkSources: new Map<number, ReadonlySet<number>>(),+ tagsByDoc: new Map<number, ReadonlySet<string>>(),+ };+}++/** Zero-based position of a chunk in a ranking. */+function positionOf(ranked: ReadonlyArray<BrainSearchResult>, chunkId: number): number {+ return ranked.findIndex((r) => r.chunkId === chunkId);+}++// ── 1. the two modes agree about a verbatim hit under an `exact` plan ───────++/** A quoted phrase: the classifier's own `exact` trigger, not a hand-built profile. */+const QUOTED_QUERY = '"canary rollout runbook"';++test("a quoted phrase classifies as exact, so the profile favours the keyword lane", () => {+ const plan = buildQueryPlan(QUOTED_QUERY);+ expect(plan.intent).toBe("exact");+ expect(plan.weightProfile.keywordMul).toBeGreaterThan(plan.weightProfile.semanticMul);+});++test("rrf mode prefers the verbatim keyword hit over the semantic-only rival, as linear does", () => {+ const plan = buildQueryPlan(QUOTED_QUERY);+ const base = {+ keywordWeight: KEYWORD_WEIGHT,+ semanticWeight: SEMANTIC_WEIGHT,+ limit: LIMIT,+ nowMs: NOW,+ weightProfile: plan.weightProfile,+ } as const;++ const linear = rankResults(inputs(), base);+ expect(positionOf(linear, VERBATIM)).toBeLessThan(positionOf(linear, SEMANTIC_ONLY));++ const fused = rankResults(inputs(), { ...base, fusionMode: "rrf", rrfK: DEFAULT_RRF_K });+ expect(positionOf(fused, VERBATIM)).toBeLessThan(positionOf(fused, SEMANTIC_ONLY));+});++test("without a declared intent, rrf keeps rewarding the other lane - the profile is the cause", () => {+ // The same fixture with a neutral profile: the semantic-only chunk is+ // rank 1 of its lane and outranks the verbatim hit, which is rank 2 of+ // its own. That is classic RRF and stays true - the previous test+ // changes ONLY because the query declared an intent.+ const fused = rankResults(inputs(), {+ keywordWeight: KEYWORD_WEIGHT,+ semanticWeight: SEMANTIC_WEIGHT,+ limit: LIMIT,+ nowMs: NOW,+ fusionMode: "rrf",+ rrfK: DEFAULT_RRF_K,+ });+ expect(positionOf(fused, SEMANTIC_ONLY)).toBeLessThan(positionOf(fused, VERBATIM));+});++// ── 2. a neutral profile reproduces classic RRF exactly ────────────────────++test("a neutral profile fuses byte-identically to weightless rrf", () => {+ const base = {+ keywordWeight: KEYWORD_WEIGHT,+ semanticWeight: SEMANTIC_WEIGHT,+ limit: LIMIT,+ nowMs: NOW,+ fusionMode: "rrf",+ rrfK: DEFAULT_RRF_K,+ } as const;+ const withoutProfile = JSON.stringify(rankResults(inputs(), base));+ const withNeutral = JSON.stringify(+ rankResults(inputs(), { ...base, weightProfile: NEUTRAL_PROFILE }),+ );+ expect(withNeutral).toBe(withoutProfile);+});++test("rrfFuse with unit lane weights equals rrfFuse without them", () => {+ const lanes = {+ keywordRankedChunkIds: [1, 2, 3],+ semanticRankedChunkIds: [3, 4],+ k: DEFAULT_RRF_K,+ };+ const weightless = [...rrfFuse(lanes).entries()];+ const unit = [...rrfFuse({ ...lanes, laneWeights: { keyword: 1, semantic: 1 } }).entries()];+ expect(unit).toEqual(weightless);+});++test("rrfFuse scales the per-lane contribution, not a post-fusion score", () => {+ // One chunk per lane at rank 1: weightless they tie, weighted they do+ // not, and the ratio of raw contributions is the ratio of the weights.+ // A post-fusion multiplier could not separate them at all.+ const fused = rrfFuse({+ keywordRankedChunkIds: [1],+ semanticRankedChunkIds: [2],+ k: DEFAULT_RRF_K,+ laneWeights: { keyword: 1.3, semantic: 0.7 },+ });+ expect(fused.get(1)!).toBeGreaterThan(fused.get(2)!);+ const tied = rrfFuse({+ keywordRankedChunkIds: [1],+ semanticRankedChunkIds: [2],+ k: DEFAULT_RRF_K,+ });+ expect(tied.get(1)!).toBe(tied.get(2)!);+});++test("a non-positive lane weight is refused, naming the cause and the exit", () => {+ const lanes = { keywordRankedChunkIds: [1], semanticRankedChunkIds: [2], k: DEFAULT_RRF_K };+ expect(() => rrfFuse({ ...lanes, laneWeights: { keyword: 0, semantic: 1 } })).toThrow(+ SearchError,+ );+ expect(() => rrfFuse({ ...lanes, laneWeights: { keyword: -1, semantic: 1 } })).toThrow(+ /o2b search weights --reset/,+ );+ expect(() => rrfFuse({ ...lanes, laneWeights: { keyword: 1, semantic: Number.NaN } })).toThrow(+ /semantic/,+ );+});++// ── 3. the linear path is inert to the rrf knobs ───────────────────────────++test("linear ranking is byte-identical across every intent profile and rrf knob", () => {+ // Flag-off (`fusionMode` omitted) against flag-on (`linear` named, and+ // an rrf damping constant supplied) for each intent the classifier can+ // produce. Full JSON, so scores, reasons and breakdowns are compared,+ // not just the order.+ const queries = [+ "", // neutral+ QUOTED_QUERY, // exact+ "[[Deploy Canary]]", // entity+ "how do we usually roll a release out to production safely", // broad+ ];+ const seen = new Set<string>();+ for (const query of queries) {+ const plan = buildQueryPlan(query);+ seen.add(plan.intent);+ const base = {+ keywordWeight: KEYWORD_WEIGHT,+ semanticWeight: SEMANTIC_WEIGHT,+ limit: LIMIT,+ nowMs: NOW,+ weightProfile: plan.weightProfile,+ } as const;+ const omitted = JSON.stringify(rankResults(inputs(), base));+ const explicit = JSON.stringify(rankResults(inputs(), { ...base, fusionMode: "linear" }));+ const withRrfK = JSON.stringify(+ rankResults(inputs(), { ...base, fusionMode: "linear", rrfK: 1 }),+ );+ expect(explicit).toBe(omitted);+ expect(withRrfK).toBe(omitted);+ }+ // The matrix really did exercise four distinct profiles.+ expect(seen).toEqual(new Set(["neutral", "exact", "entity", "broad"]));+});diff --git a/tests/core/search/snippet-match-anchor.test.ts b/tests/core/search/snippet-match-anchor.test.tsnew file mode 100644index 00000000..fff516b0--- /dev/null+++ b/tests/core/search/snippet-match-anchor.test.ts@@ -0,0 +1,208 @@+/**+ * Match-anchored snippet windows (what-the-index-already-knew, task E).+ *+ * Every preview surface truncated from the HEAD of the chunk, so a match+ * in the middle of a long passage was invisible in the preview the caller+ * actually reads: the card snippet stopped at 240 chars, the CLI+ * transcript at 140, the MCP `content` at 600. The window is now centred+ * on the first significant query term that occurs in the content.+ *+ * The no-match case is a DEFINED behaviour, not a fallback: when no+ * significant query term occurs, the window is the head of the chunk -+ * the same bytes the head truncation produced. Both halves are asserted+ * here, the second against literally-recomputed pre-change bytes rather+ * than against the function under test.+ */++import { afterEach, describe, expect, test } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { cardSnippet } from "../../../src/core/search/cards.ts";+import { renderOutcomeHuman } from "../../../src/cli/search/outcome-render.ts";+import {+ HEAD_WINDOW_START,+ NO_MATCH_OFFSET,+ markWindow,+ matchOffset,+ snippetAround,+ toCodePointOffset,+ windowStart,+} from "../../../src/core/search/snippet-window.ts";+import type { BrainSearchResult, SearchOutcome } from "../../../src/core/search/types.ts";++/** Mirrors `CARD_SNIPPET_CHARS` in `src/core/search/cards.ts`. */+const CARD_SNIPPET_CHARS = 240;+/** Mirrors `SNIPPET_MAX_CHARS` in `src/cli/search/outcome-render.ts`. */+const CLI_SNIPPET_MAX_CHARS = 140;++const FILLER = "This paragraph records background context for the passage. ";+/** The only sentence in the fixture carrying the anchor term. */+const MATCH_SENTENCE = "Quantum entanglement links two particles at a distance.";+/** Term present in the fixture; `significantTerms` keeps it (length >= 3). */+const ANCHOR_TERM = "entanglement";+/** Term absent from the fixture, so the window has nothing to anchor on. */+const ABSENT_TERM = "photosynthesis";++/** Filler long enough to push the match past every surface's cap. */+const LEAD = FILLER.repeat(12);+const CONTENT = `${LEAD}${MATCH_SENTENCE}\n${FILLER.repeat(12)}`;++function collapse(text: string): string {+ return text.replace(/\s+/g, " ").trim();+}++function resultWith(content: string): BrainSearchResult {+ return Object.freeze({+ documentId: 1,+ chunkId: 1,+ // Neither the path nor the title may carry the anchor term, or the+ // transcript assertion would pass on the header line alone.+ path: "physics/notes.md",+ title: "Notes",+ content,+ startLine: 1,+ endLine: 9,+ score: 1.5,+ keywordScore: 1.5,+ semanticScore: 0,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "keyword" as const,+ reasons: Object.freeze(["keyword: 1.50"]),+ });+}++function outcomeWith(content: string): SearchOutcome {+ return Object.freeze({+ results: Object.freeze([resultWith(content)]),+ warnings: Object.freeze([]),+ total: 1,+ });+}++describe("the anchoring kernel", () => {+ test("no significant term occurrence is the named sentinel, not offset 0", () => {+ expect(matchOffset(CONTENT, ABSENT_TERM)).toBe(NO_MATCH_OFFSET);+ expect(NO_MATCH_OFFSET).not.toBe(HEAD_WINDOW_START);+ // A term at the very start is offset 0 and must stay distinguishable+ // from "nothing matched".+ expect(matchOffset("this leads the chunk", "this")).toBe(HEAD_WINDOW_START);+ });++ test("the sentinel window is the head of the text", () => {+ expect(windowStart(NO_MATCH_OFFSET, CARD_SNIPPET_CHARS)).toBe(HEAD_WINDOW_START);+ expect(snippetAround("abcdefgh", NO_MATCH_OFFSET, 4)).toBe("abcd");+ });++ test("a window is centred on the match and clamped at the head", () => {+ expect(windowStart(1000, 200)).toBe(900);+ expect(windowStart(10, 200)).toBe(HEAD_WINDOW_START);+ });++ test("terms shorter than the significance floor never anchor a window", () => {+ // `significantTerms` keeps tokens of length >= 3, with no stop-word+ // list anywhere. A two-character query therefore has no terms at all.+ expect(matchOffset("an ox in the field", "ox")).toBe(NO_MATCH_OFFSET);+ });++ test("the earliest occurrence among several terms wins", () => {+ const text = "alpha beta gamma";+ expect(matchOffset(text, "gamma alpha")).toBe(text.indexOf("alpha"));+ });++ test("matching is case-insensitive and offsets index the original text", () => {+ const text = "The Entanglement of particles";+ expect(matchOffset(text, "ENTANGLEMENT")).toBe(text.indexOf("Entanglement"));+ });++ test("markers appear only on the sides the window does not reach", () => {+ expect(markWindow("body", HEAD_WINDOW_START, false, "…")).toBe("body");+ expect(markWindow("body", HEAD_WINDOW_START, true, "…")).toBe("body…");+ expect(markWindow("body", 5, true, "…")).toBe("…body…");+ });++ test("a code-point offset counts a surrogate pair once", () => {+ const text = "😀😀abc";+ expect(toCodePointOffset(text, text.indexOf("abc"))).toBe(2);+ expect(toCodePointOffset(text, NO_MATCH_OFFSET)).toBe(NO_MATCH_OFFSET);+ });+});++describe("match-anchored snippet windows", () => {+ test("the fixture really does place the match past every surface cap", () => {+ const at = CONTENT.indexOf(ANCHOR_TERM);+ expect(at).toBeGreaterThan(600);+ expect(CONTENT.includes(ABSENT_TERM)).toBe(false);+ });++ test("a card snippet shows a match that sits past the card cap", () => {+ const snippet = cardSnippet(CONTENT, ANCHOR_TERM);+ expect(snippet).toContain(ANCHOR_TERM);+ });++ test("the CLI transcript shows a match that sits past the transcript cap", () => {+ const rendered = renderOutcomeHuman(outcomeWith(CONTENT), false, ANCHOR_TERM);+ expect(rendered).toContain(ANCHOR_TERM);+ });++ test("no query-term occurrence: the card snippet is the head of the chunk", () => {+ const collapsed = collapse(CONTENT);+ const expected = `${[...collapsed].slice(0, CARD_SNIPPET_CHARS).join("")}...`;+ expect(cardSnippet(CONTENT, ABSENT_TERM)).toBe(expected);+ });++ test("no query-term occurrence: the CLI transcript is the head of the chunk", () => {+ const collapsed = CONTENT.trim().replace(/\s+/g, " ");+ const expected = ` ${collapsed.slice(0, CLI_SNIPPET_MAX_CHARS)}…`;+ const rendered = renderOutcomeHuman(outcomeWith(CONTENT), false, ABSENT_TERM);+ expect(rendered.split("\n")).toContain(expected);+ });+});++describe("the cards disclosure surface anchors through the real pipeline", () => {+ let cleanup: (() => void) | null = null;++ afterEach(() => {+ cleanup?.();+ cleanup = null;+ });++ test("a card snippet shows a match the head window would have cut", async () => {+ const v = createTempVault("snippet-match-anchor");+ cleanup = v.cleanup;+ // One paragraph under the 800-char chunk cap, so the whole fixture is+ // a single chunk and the match really does sit past the card cap.+ const body = `${FILLER.repeat(6)}${MATCH_SENTENCE}`;+ writeMd(v.vault, "physics/notes.md", `# Notes\n\n${body}\n`);+ const cfg = makeConfig({ vault: v.vault, dbPath: v.dbPath });+ await indexVault(cfg);++ const outcome = await search(cfg, {+ query: ANCHOR_TERM,+ limit: 5,+ disclosure: "cards",+ });+ expect(outcome.cards).toBeDefined();+ const card = outcome.cards!.find((c) => c.path === "physics/notes.md");+ expect(card).toBeDefined();+ expect(body.indexOf(ANCHOR_TERM)).toBeGreaterThan(CARD_SNIPPET_CHARS);+ expect(card!.snippet).toContain(ANCHOR_TERM);+ });+});++test("the transcript marker asks about the collapsed text, not the raw chunk", () => {+ // A chunk whose RAW length is over the cap but which collapses under it+ // is not cut, so it must carry no trailing marker. Keying the marker on+ // `content.length` claimed a cut that whitespace collapse had already+ // removed - the marker said "there is more" when there was not.+ const words = "alpha beta gamma delta ".repeat(4).trim(); // 91 chars collapsed+ const padded = words.replace(/ /g, "\n\n "); // same words, 400+ raw chars+ expect(padded.length).toBeGreaterThan(140);+ expect(padded.trim().replace(/\s+/g, " ").length).toBeLessThan(140);++ const rendered = renderOutcomeHuman(outcomeWith(padded), false, "alpha");+ expect(rendered).toContain("alpha beta gamma delta");+ expect(rendered).not.toContain("…");+});diff --git a/tests/core/search/snippet-window-bounds.test.ts b/tests/core/search/snippet-window-bounds.test.tsnew file mode 100644index 00000000..e439cdc9--- /dev/null+++ b/tests/core/search/snippet-window-bounds.test.ts@@ -0,0 +1,228 @@+/**+ * Window BOUNDS, as opposed to window anchoring+ * (what-the-index-already-knew, task E follow-up).+ *+ * Anchoring answered "which offset does the window centre on". Two+ * questions it left open are defects in their own right:+ *+ * 1. A window that opens at `matchAt - half` throws away the head of a+ * passage that ALREADY FITTED inside the cap, and stamps a leading+ * continuation marker on the loss. The MCP surface never had the bug+ * because it returns short content untouched before it anchors; the+ * card snippet and the CLI transcript did.+ * 2. A window that opens at a UTF-16 offset can open INSIDE a surrogate+ * pair, so the body begins with a lone low surrogate that renders as+ * U+FFFD once the payload is encoded as UTF-8. This hazard is new:+ * before anchoring, `start` was always 0.+ *+ * What must NOT change: the no-match window is still the head of the+ * chunk, byte for byte; and an over-cap passage whose match sits near the+ * tail still yields a SHORT window rather than one slid backwards to stay+ * full width, which is the behaviour session recall has always had and the+ * anchoring module states on purpose.+ */++import { describe, expect, test } from "bun:test";++import { cardSnippet } from "../../../src/core/search/cards.ts";+import { renderOutcomeHuman } from "../../../src/cli/search/outcome-render.ts";+import {+ HEAD_WINDOW_START,+ NO_MATCH_OFFSET,+ alignToCodePointStart,+ matchOffset,+ snippetAround,+ windowStart,+ windowStartWithin,+} from "../../../src/core/search/snippet-window.ts";+import type { BrainSearchResult, SearchOutcome } from "../../../src/core/search/types.ts";++/** Mirrors `CARD_SNIPPET_CHARS` in `src/core/search/cards.ts`. */+const CARD_SNIPPET_CHARS = 240;+/** Mirrors `CARD_ELLIPSIS` in `src/core/search/cards.ts`. */+const CARD_ELLIPSIS = "...";+/** Mirrors `SNIPPET_MAX_CHARS` in `src/cli/search/outcome-render.ts`. */+const CLI_SNIPPET_MAX_CHARS = 140;+/** Mirrors `ELLIPSIS` in `src/core/search/snippet-window.ts`. */+const CLI_ELLIPSIS = "…";+/** Indentation the CLI transcript puts before a snippet body. */+const CLI_SNIPPET_INDENT = " ";++const ANCHOR_TERM = "entanglement";+const ABSENT_TERM = "photosynthesis";+/** One filler word plus a space, so collapse is a no-op on the fixtures. */+const FILLER_WORD = "context ";++/**+ * A passage that fits inside the CARD cap whole, with the match past the+ * half-window: 192 filler chars then the 12-char term, 204 code points+ * against a cap of 240 and a half-window of 120.+ */+const CARD_UNDER_CAP = `${FILLER_WORD.repeat(24)}${ANCHOR_TERM}`;+/** The same shape against the CLI cap: 112 + 12 = 124 against a cap of 140. */+const CLI_UNDER_CAP = `${FILLER_WORD.repeat(14)}${ANCHOR_TERM}`;++const REPLACEMENT_CHAR = "�";++/** Round-trip through UTF-8, which is what any wire encoding does. */+function utf8RoundTrip(text: string): string {+ return new TextDecoder().decode(new TextEncoder().encode(text));+}++function resultWith(content: string): BrainSearchResult {+ return Object.freeze({+ documentId: 1,+ chunkId: 1,+ // Neither path nor title may carry the anchor term, or a transcript+ // assertion could pass on the header line alone.+ path: "physics/notes.md",+ title: "Notes",+ content,+ startLine: 1,+ endLine: 9,+ score: 1.5,+ keywordScore: 1.5,+ semanticScore: 0,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "keyword" as const,+ reasons: Object.freeze(["keyword: 1.50"]),+ });+}++function outcomeWith(content: string): SearchOutcome {+ return Object.freeze({+ results: Object.freeze([resultWith(content)]),+ warnings: Object.freeze([]),+ total: 1,+ });+}++/** The transcript line carrying the snippet body, indent stripped. */+function transcriptSnippet(content: string, query: string): string {+ const lines = renderOutcomeHuman(outcomeWith(content), false, query).split("\n");+ const line = lines.find((l) => l.startsWith(CLI_SNIPPET_INDENT) && !l.includes("line 1-9"));+ expect(line).toBeDefined();+ return line!.slice(CLI_SNIPPET_INDENT.length);+}++describe("a window never truncates text the cap had room for", () => {+ test("the fixtures really do put the match past the half-window", () => {+ expect([...CARD_UNDER_CAP].length).toBeLessThanOrEqual(CARD_SNIPPET_CHARS);+ expect(CARD_UNDER_CAP.indexOf(ANCHOR_TERM)).toBeGreaterThan(CARD_SNIPPET_CHARS / 2);+ expect(CLI_UNDER_CAP.length).toBeLessThanOrEqual(CLI_SNIPPET_MAX_CHARS);+ expect(CLI_UNDER_CAP.indexOf(ANCHOR_TERM)).toBeGreaterThan(CLI_SNIPPET_MAX_CHARS / 2);+ });++ test("the kernel opens at the head when the whole text fits the cap", () => {+ expect(windowStartWithin(200, CARD_SNIPPET_CHARS, 204)).toBe(HEAD_WINDOW_START);+ expect(windowStartWithin(120, CLI_SNIPPET_MAX_CHARS, 124)).toBe(HEAD_WINDOW_START);+ // Exactly at the cap is still "it fits".+ expect(windowStartWithin(200, CARD_SNIPPET_CHARS, CARD_SNIPPET_CHARS)).toBe(HEAD_WINDOW_START);+ });++ test("a card snippet that fits the cap keeps its head and carries no marker", () => {+ expect(cardSnippet(CARD_UNDER_CAP, ANCHOR_TERM)).toBe(CARD_UNDER_CAP);+ });++ test("a transcript snippet that fits the cap keeps its head and carries no marker", () => {+ expect(transcriptSnippet(CLI_UNDER_CAP, ANCHOR_TERM)).toBe(CLI_UNDER_CAP);+ });++ test("the same is true of the session-recall window the kernel also serves", () => {+ // 8 chars, cap 8: the whole text fits, so a match at offset 6 must not+ // slice the head off to centre a window there is no room to move.+ expect(snippetAround("abcdefgh", 6, 8)).toBe("abcdefgh");+ });+});++describe("an over-cap tail window stays short rather than sliding backwards", () => {+ test("the kernel does not clamp the start to len - maxChars", () => {+ // 402 code points, match at 390, cap 240. Sliding back would open at+ // 162 and return a full 240; the documented behaviour opens at 270 and+ // returns a short window that keeps the match near the centre.+ expect(windowStartWithin(390, CARD_SNIPPET_CHARS, 402)).toBe(390 - CARD_SNIPPET_CHARS / 2);+ expect(windowStartWithin(390, CARD_SNIPPET_CHARS, 402)).toBe(+ windowStart(390, CARD_SNIPPET_CHARS),+ );+ });++ test("the card snippet over the cap is the short tail window, leading marker only", () => {+ const content = `${"x".repeat(390)} ${ANCHOR_TERM}`;+ const points = [...content];+ const start = 391 - CARD_SNIPPET_CHARS / 2;+ const expected = `${CARD_ELLIPSIS}${points.slice(start, start + CARD_SNIPPET_CHARS).join("")}`;+ expect(cardSnippet(content, ANCHOR_TERM)).toBe(expected);+ expect([...cardSnippet(content, ANCHOR_TERM)].length).toBeLessThan(CARD_SNIPPET_CHARS);+ });+});++describe("the no-match window is still the head of the chunk, byte for byte", () => {+ /** Over-cap content so the pre-change head truncation actually cut. */+ const LONG = `${FILLER_WORD.repeat(200)}${ANCHOR_TERM}`;++ test("card: recomputed pre-change bytes", () => {+ const collapsed = LONG.replace(/\s+/g, " ").trim();+ const expected = `${[...collapsed].slice(0, CARD_SNIPPET_CHARS).join("")}${CARD_ELLIPSIS}`;+ expect(cardSnippet(LONG, ABSENT_TERM)).toBe(expected);+ });++ test("transcript: recomputed pre-change bytes", () => {+ const collapsed = LONG.trim().replace(/\s+/g, " ");+ const expected = `${collapsed.slice(0, CLI_SNIPPET_MAX_CHARS)}${CLI_ELLIPSIS}`;+ expect(transcriptSnippet(LONG, ABSENT_TERM)).toBe(expected);+ });++ test("under-cap content with no match passes through untouched on both surfaces", () => {+ expect(cardSnippet(CARD_UNDER_CAP, ABSENT_TERM)).toBe(CARD_UNDER_CAP);+ expect(transcriptSnippet(CLI_UNDER_CAP, ABSENT_TERM)).toBe(CLI_UNDER_CAP);+ });+});++describe("a card snippet declares its ceiling", () => {+ test("both markers fall outside the cap, so the body is never shortened", () => {+ const content = `${"z".repeat(400)} ${ANCHOR_TERM} ${"z".repeat(400)}`;+ const snippet = cardSnippet(content, ANCHOR_TERM);+ expect(snippet.startsWith(CARD_ELLIPSIS)).toBe(true);+ expect(snippet.endsWith(CARD_ELLIPSIS)).toBe(true);+ expect([...snippet].length).toBe(CARD_SNIPPET_CHARS + 2 * CARD_ELLIPSIS.length);+ });+});++describe("a window never opens inside a surrogate pair", () => {+ test("the kernel backs an offset off a low surrogate", () => {+ const text = "\u{1F600}\u{1F600}abc";+ // Offset 1 is the low half of the first emoji.+ expect(alignToCodePointStart(text, 1)).toBe(0);+ // Offsets that already sit on a boundary are returned unchanged.+ expect(alignToCodePointStart(text, 0)).toBe(0);+ expect(alignToCodePointStart(text, 2)).toBe(2);+ expect(alignToCodePointStart(text, 4)).toBe(4);+ });++ test("an already-lone low surrogate is left exactly where it is", () => {+ // Nothing to re-join: backing up here would invent a pair that the+ // source text does not contain.+ expect(alignToCodePointStart("\uDC00abc", 0)).toBe(0);+ expect(alignToCodePointStart("a\uDC00b", 1)).toBe(1);+ });++ test("the transcript window does not open on a lone low surrogate", () => {+ // 100 emoji (200 UTF-16 units), a space, then the term: the match sits+ // at 201, so the unaligned start is 131 - the low half of an emoji.+ const content = `${"\u{1F600}".repeat(100)} ${ANCHOR_TERM}${"b".repeat(100)}`;+ const unaligned = windowStart(matchOffset(content, ANCHOR_TERM), CLI_SNIPPET_MAX_CHARS);+ const code = content.charCodeAt(unaligned);+ expect(code).toBeGreaterThanOrEqual(0xdc00);+ expect(code).toBeLessThanOrEqual(0xdfff);++ const snippet = transcriptSnippet(content, ANCHOR_TERM);+ expect(utf8RoundTrip(snippet)).not.toContain(REPLACEMENT_CHAR);+ expect(snippet.startsWith(`${CLI_ELLIPSIS}\u{1F600}`)).toBe(true);+ });++ test("the sentinel window is unaffected by alignment", () => {+ expect(alignToCodePointStart("\u{1F600}abc", HEAD_WINDOW_START)).toBe(HEAD_WINDOW_START);+ expect(windowStartWithin(NO_MATCH_OFFSET, CARD_SNIPPET_CHARS, 10_000)).toBe(HEAD_WINDOW_START);+ });+});diff --git a/tests/core/search/store-integrity.test.ts b/tests/core/search/store-integrity.test.tsnew file mode 100644index 00000000..64622a20--- /dev/null+++ b/tests/core/search/store-integrity.test.ts@@ -0,0 +1,433 @@+/**+ * The two read-open gates that were never there: the tables and columns a+ * query is about to touch must EXIST (task I), and a store already known to+ * be structurally corrupt must not be answered from (task K).+ *+ * Both fixtures are built to be "parseable": the file opens, `index_state`+ * reads back the current schema version, and ordinary SELECTs succeed. That+ * is exactly the state in which the store used to keep serving answers.+ */+import { test, expect, beforeEach, afterEach } from "bun:test";+import { Database } from "bun:sqlite";+import {+ chmodSync,+ closeSync,+ mkdirSync,+ openSync,+ statSync,+ writeFileSync,+ writeSync,+} from "node:fs";+import { dirname } from "node:path";++import { indexVault, reindexVault } from "../../../src/core/search/indexer.ts";+import { openReadOrSelfHeal } from "../../../src/core/search/pipeline/store-open.ts";+import { search } from "../../../src/core/search/search.ts";+import {+ applyMigrations,+ expectedSchemaObjects,+ findSchemaGaps,+ LATEST_SCHEMA_VERSION,+ readSchemaVersion,+} from "../../../src/core/search/schema.ts";+import { Store } from "../../../src/core/search/store.ts";+import {+ INTEGRITY_CHECKED_AT_STATE_KEY,+ INTEGRITY_FAULT_STATE_KEY,+} from "../../../src/core/search/store/state.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("store-integrity");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++/** A vault big enough that the index file has data pages past its middle. */+function seedVault(notes: number): void {+ for (let i = 0; i < notes; i++) {+ const body = `The quick brown fox jumps over the lazy dog number ${i}. `.repeat(40);+ writeMd(vault, `Notes/note-${i}.md`, `# Note ${i}\n\n${body}`);+ }+}++/** Age the recorded check stamp past the gate's interval. */+function expireIntegrityStamp(path: string): void {+ const db = new Database(path);+ const long_ago = new Date(Date.now() - 25 * 60 * 60 * 1000).toISOString();+ db.run("UPDATE index_state SET value = ? WHERE key = ?", [+ long_ago,+ INTEGRITY_CHECKED_AT_STATE_KEY,+ ]);+ db.close();+}++/**+ * Overwrite whole pages in the middle of the index file with garbage. The+ * header, the schema and the small metadata tables live on the first pages+ * and stay intact, so the file remains fully parseable.+ */+function corruptMiddlePages(path: string, pages: number): void {+ const db = new Database(path);+ const pageSize = (+ db.query<{ page_size: number }, []>("PRAGMA page_size").get() as { page_size: number }+ ).page_size;+ db.close();+ const size = statSync(path).size;+ const offset = Math.floor(size / 2 / pageSize) * pageSize;+ const fd = openSync(path, "r+");+ try {+ writeSync(fd, Buffer.alloc(pageSize * pages, 0xa5), 0, pageSize * pages, offset);+ } finally {+ closeSync(fd);+ }+}++// ─── Task I: the tables and columns a query is about to touch ────────────────++test("the expected-object manifest is derived from the migrations, not hand-typed", () => {+ const fresh = new Database(":memory:");+ applyMigrations(fresh);++ // The derivation is the frozen MIGRATIONS themselves, so a freshly+ // migrated database has, by construction, no gaps.+ expect(findSchemaGaps(fresh)).toEqual([]);++ const expected = expectedSchemaObjects();+ // Non-vacuous: the manifest covers the real tables and their late-added+ // columns, including the FTS5 shadow tables a query reads through.+ expect(expected.get("documents")?.has("event_anchor_examined")).toBe(true);+ expect(expected.get("chunks")?.has("fts_content")).toBe(true);+ expect(expected.get("chunk_fts_data")?.has("block")).toBe(true);+ expect(expected.has("chunk_entities")).toBe(true);++ // The trigram lane is opt-in and degrades on its own missing table, so it+ // is deliberately NOT required - refusing an index for it would force a+ // reindex that drops every embedding to restore a table nothing needs.+ expect(expected.has("chunk_trigram")).toBe(false);+ expect(expected.has("chunk_trigram_data")).toBe(false);+ // ...and that exception must not spill onto a table that merely shares a+ // prefix with an optional one.+ expect(expected.has("chunk_vec_map")).toBe(true);++ // Demonstrably able to fail: inject a violating fixture.+ fresh.exec("DROP TABLE chunk_entities");+ fresh.exec("ALTER TABLE documents DROP COLUMN event_anchor_source");+ const gaps = findSchemaGaps(fresh);+ expect(gaps).toContainEqual({ table: "chunk_entities", column: null });+ expect(gaps).toContainEqual({ table: "documents", column: "event_anchor_source" });+ fresh.close();+});++test("a current-version index with a dropped TABLE is refused at read-open", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(4);+ await indexVault(config);++ const corruptor = new Database(dbPath);+ corruptor.exec("DROP TABLE chunk_entities");+ // Still stamped current — the version gate alone sees nothing wrong.+ expect(readSchemaVersion(corruptor)).toBe(LATEST_SCHEMA_VERSION);+ corruptor.close();++ await expect(Store.open(config, { mode: "read" })).rejects.toMatchObject({+ code: "SCHEMA_MISMATCH",+ });+ await expect(Store.open(config, { mode: "read" })).rejects.toThrow(/chunk_entities/);+ await expect(Store.open(config, { mode: "read" })).rejects.toThrow(/o2b search reindex/);+});++test("a current-version index with a dropped COLUMN is refused at read-open", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(4);+ await indexVault(config);++ const corruptor = new Database(dbPath);+ corruptor.exec("ALTER TABLE documents DROP COLUMN event_anchor_source");+ expect(readSchemaVersion(corruptor)).toBe(LATEST_SCHEMA_VERSION);+ corruptor.close();++ await expect(Store.open(config, { mode: "read" })).rejects.toMatchObject({+ code: "SCHEMA_MISMATCH",+ });+ await expect(Store.open(config, { mode: "read" })).rejects.toThrow(+ /documents\.event_anchor_source/,+ );+});++test("a healthy current index still opens for reading", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(4);+ await indexVault(config);+ const store = await Store.open(config, { mode: "read" });+ expect(store.schemaVersion()).toBe(LATEST_SCHEMA_VERSION);+ await store.close();+});++// ─── Task K: a corrupt store is not answered from ────────────────────────────++test("a page-corrupted index is structurally corrupt yet fully parseable", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(60);+ await indexVault(config);+ corruptMiddlePages(dbPath, 4);++ // The fixture's whole point: nothing about this file LOOKS wrong.+ const probe = new Database(dbPath);+ expect(readSchemaVersion(probe)).toBe(LATEST_SCHEMA_VERSION);+ expect(+ probe.query<{ n: number }, []>("SELECT count(*) AS n FROM chunks").get()?.n,+ ).toBeGreaterThan(0);+ // ...and yet the structural scan condemns it. Damage can either make the+ // scan report a malformed tree or stop it completing; both are faults.+ let verdict: string;+ try {+ verdict = probe+ .query<{ quick_check: string }, []>("PRAGMA quick_check(5)")+ .all()+ .map((r) => r.quick_check)+ .join("\n");+ } catch (e) {+ verdict = e instanceof Error ? e.message : String(e);+ }+ expect(verdict).not.toBe("ok");+ probe.close();+});++test("the write-open integrity gate detects the corruption, records it, and refuses", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(60);+ await indexVault(config);+ // The index run just scanned a healthy file, so age that stamp past the+ // interval: this is the next write open after the window closed.+ expireIntegrityStamp(dbPath);+ corruptMiddlePages(dbPath, 4);++ await expect(Store.open(config, { mode: "write" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });++ const probe = new Database(dbPath);+ expect(+ probe+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?")+ .get(INTEGRITY_FAULT_STATE_KEY)?.value,+ ).toBeTruthy();+ expect(+ probe+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?")+ .get(INTEGRITY_CHECKED_AT_STATE_KEY)?.value,+ ).toBeTruthy();+ probe.close();+});++test("a read-open refuses a store carrying a recorded integrity fault", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(4);+ await indexVault(config);++ const stamper = new Database(dbPath);+ stamper.run("INSERT INTO index_state(key, value, updated_at) VALUES (?,?,?)", [+ INTEGRITY_FAULT_STATE_KEY,+ "Tree 38 page 478: btreeInitPage() returns error code 11",+ new Date().toISOString(),+ ]);+ stamper.close();++ await expect(Store.open(config, { mode: "read" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });+ await expect(Store.open(config, { mode: "read" })).rejects.toThrow(/btreeInitPage/);+});++test("a search over a store with a recorded fault self-heals instead of answering", async () => {+ const config = makeConfig({ vault, dbPath });+ writeMd(vault, "Notes/foo.md", "# Foo\n\nThe quick brown fox jumps over the lazy dog.");+ await indexVault(config);++ const stamper = new Database(dbPath);+ stamper.run("INSERT INTO index_state(key, value, updated_at) VALUES (?,?,?)", [+ INTEGRITY_FAULT_STATE_KEY,+ "*** in database main *** Tree 38 page 478 malformed",+ new Date().toISOString(),+ ]);+ stamper.close();++ const outcome = await search(config, { query: "fox" });+ expect(outcome.results.length).toBeGreaterThan(0);++ // The rebuild produced a store with no fault recorded against it.+ const probe = new Database(dbPath);+ expect(+ probe+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?")+ .get(INTEGRITY_FAULT_STATE_KEY),+ ).toBeNull();+ probe.close();+});++test("a passing check clears a stale fault and stamps when it ran", async () => {+ const config = makeConfig({ vault, dbPath });+ writeMd(vault, "Notes/foo.md", "# Foo\n\nfox");+ await indexVault(config);++ const stamper = new Database(dbPath);+ stamper.run("INSERT INTO index_state(key, value, updated_at) VALUES (?,?,?)", [+ INTEGRITY_FAULT_STATE_KEY,+ "stale fault from a previous generation",+ new Date().toISOString(),+ ]);+ // Force the interval gate open again.+ stamper.run("DELETE FROM index_state WHERE key = ?", [INTEGRITY_CHECKED_AT_STATE_KEY]);+ stamper.close();++ const store = await Store.open(config, { mode: "write" });+ expect(store.getState(INTEGRITY_FAULT_STATE_KEY)).toBeNull();+ expect(store.getState(INTEGRITY_CHECKED_AT_STATE_KEY)).toBeTruthy();+ await store.close();+});++test("a self-heal that fails reports why, instead of only the symptom", async () => {+ // An unreadable index (not a sqlite file at all) in a directory the+ // rebuild cannot write its staging database into: the repair fails, and+ // the retry finds the same unreadable file it started with.+ writeMd(vault, "Notes/foo.md", "# Foo\n\nfox");+ const indexDir = dirname(dbPath);+ mkdirSync(indexDir, { recursive: true });+ writeFileSync(dbPath, "this is not a sqlite database");+ const config = makeConfig({ vault, dbPath });++ chmodSync(indexDir, 0o500);+ let failure: Error | null;+ try {+ failure = await openReadOrSelfHeal(config).then(+ (store) => {+ void store.close();+ return null;+ },+ (e: unknown) => e as Error,+ );+ } finally {+ chmodSync(indexDir, 0o700);+ }+ expect(failure).not.toBeNull();+ // Both halves: what is wrong now, and why the repair did not happen.+ expect(failure?.message).toMatch(/the automatic rebuild of/);+ expect(failure?.message).toMatch(/also failed/);+});++// ─── The recorded fault must be honoured wherever the store is opened ────────++/** One `index_state` cell, read without opening a `Store`. */+function readCell(path: string, key: string): string | null {+ const db = new Database(path);+ try {+ return (+ db.query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?").get(key)+ ?.value ?? null+ );+ } finally {+ db.close();+ }+}++/**+ * The state a COMPLETED condemning scan leaves behind, whatever produced+ * it: a fault, and a fresh `integrity_checked_at` stamping when that scan+ * finished. Both `runWriteOpenIntegrityGate` and `o2b search check+ * --integrity` write exactly this pair.+ */+function recordCondemningScan(path: string, fault: string): void {+ const db = new Database(path);+ const now = new Date().toISOString();+ for (const [key, value] of [+ [INTEGRITY_FAULT_STATE_KEY, fault],+ [INTEGRITY_CHECKED_AT_STATE_KEY, now],+ ] as ReadonlyArray<readonly [string, string]>) {+ db.run(+ "INSERT INTO index_state(key, value, updated_at) VALUES (?,?,?) " ++ "ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at",+ [key, value, now],+ );+ }+ db.close();+}++test("a write-open refuses a store carrying a recorded integrity fault", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(4);+ await indexVault(config);+ recordCondemningScan(dbPath, "Tree 38 page 478: btreeInitPage() returns error code 11");++ // The read open already refuses this file...+ await expect(Store.open(config, { mode: "read" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });+ // ...and the write open is the one that matters more, because everything+ // after it WRITES into the condemned file.+ await expect(Store.open(config, { mode: "write" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });+ await expect(Store.open(config, { mode: "write" })).rejects.toThrow(/btreeInitPage/);+ await expect(Store.open(config, { mode: "write" })).rejects.toThrow(/o2b search reindex/);+});++test("the write open that condemned the file refuses the NEXT write open too", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(60);+ await indexVault(config);+ expireIntegrityStamp(dbPath);+ corruptMiddlePages(dbPath, 4);++ // The scan runs, condemns the file, and records BOTH the fault and a+ // fresh `integrity_checked_at` naming when it finished.+ await expect(Store.open(config, { mode: "write" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });+ expect(readCell(dbPath, INTEGRITY_FAULT_STATE_KEY)).toBeTruthy();++ // The next write open falls INSIDE the interval that scan just re-stamped.+ // Skipping the expensive scan is correct; skipping the recorded fault is+ // not - the file is still condemned and still must not be written into.+ await expect(Store.open(config, { mode: "write" })).rejects.toMatchObject({+ code: "INDEX_UNREADABLE",+ });+});++test("a condemned index can still be rebuilt: reindexVault is not blocked by the fault", async () => {+ const config = makeConfig({ vault, dbPath });+ writeMd(vault, "Notes/foo.md", "# Foo\n\nThe quick brown fox jumps over the lazy dog.");+ await indexVault(config);+ recordCondemningScan(dbPath, "*** in database main *** Tree 38 page 478 malformed");++ // The one legitimate escape from the refusal must stay open, or the gate+ // would refuse the repair as well.+ await reindexVault(config);+ expect(readCell(dbPath, INTEGRITY_FAULT_STATE_KEY)).toBeNull();++ const store = await Store.open(config, { mode: "write" });+ await store.close();+});++test("the interval gate really skips the scan: corruption after a pass is not re-detected", async () => {+ const config = makeConfig({ vault, dbPath });+ seedVault(60);+ await indexVault(config); // stamps integrity_checked_at on a healthy store+ corruptMiddlePages(dbPath, 4);++ // Within the interval the multi-second scan does not run again, so this+ // open succeeds. That is the cost trade this design makes, stated as a+ // test rather than left implicit.+ const store = await Store.open(config, { mode: "write" });+ expect(store.getState(INTEGRITY_FAULT_STATE_KEY)).toBeNull();+ await store.close();+});diff --git a/tests/mcp/brain-query-as-of.test.ts b/tests/mcp/brain-query-as-of.test.tsnew file mode 100644index 00000000..d3c49eac--- /dev/null+++ b/tests/mcp/brain-query-as-of.test.ts@@ -0,0 +1,183 @@+/**+ * `brain_query` point-in-time recall + honest time-filter descriptions+ * (what-the-index-already-knew, task J).+ *+ * `queryByTopic` has accepted a `now` option since C5 and the MCP handler+ * passed `{ showExpired }` and dropped it, so an agent could ask to SEE+ * lapsed memories but never to ask what the brain held at an instant.+ *+ * The second half of this file is about a description, which is the only+ * thing an agent reads before choosing a tool argument: since v1.43.0+ * `since` / `until` filter on EVENT time (declared validity window, then+ * body-declared anchor, with storage mtime as the last rung), while their+ * schema descriptions still said "documents modified".+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { bootstrapBrain } from "../../src/core/brain/init.ts";+import { writeSignal, type WriteSignalInput } from "../../src/core/brain/signal.ts";+import { PROPERTY_DESCRIPTION_MAX } from "../../src/mcp/registry-guard.ts";+import { buildToolTable, findTool } from "../../src/mcp/tools.ts";+import type { ServerContext, ToolDefinition } from "../../src/mcp/tool-contract.ts";++const TOPIC = "deploy";+const LAPSED_EXPIRY = "2026-07-15";+const BEFORE_EXPIRY = "2026-06-01T00:00:00Z";+const AFTER_EXPIRY = "2026-08-01T00:00:00Z";+const LAPSED_PRINCIPLE = "Principle for lapsed";+const EVERGREEN_PRINCIPLE = "Principle for evergreen";++let tmp: string;+let vault: string;+let configPath: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-mcp-query-as-of-"));+ vault = join(tmp, "vault");+ configPath = join(tmp, "config.yaml");+ writeFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);+ bootstrapBrain(vault, { configPath });+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++function ctx(): ServerContext {+ return { vault, configPath, repoRoot: null };+}++function tool(name: string): ToolDefinition {+ return findTool(buildToolTable("full"), name);+}++function signal(slug: string, overrides: Partial<WriteSignalInput> = {}): WriteSignalInput {+ return {+ topic: TOPIC,+ signal: "positive",+ agent: "tester",+ principle: `Principle for ${slug}`,+ created_at: "2026-05-01T00:00:00Z",+ date: "2026-05-01",+ slug,+ ...overrides,+ };+}++function seed(): void {+ writeSignal(vault, signal("lapsed", { expiration_date: LAPSED_EXPIRY }));+ writeSignal(+ vault,+ signal("evergreen", { created_at: "2026-05-02T00:00:00Z", date: "2026-05-02" }),+ );+}++async function topicQuery(args: Record<string, unknown>): Promise<string[]> {+ const res = (await tool("brain_query").handler(ctx(), { topic: TOPIC, ...args })) as {+ signals: ReadonlyArray<{ principle: string }>;+ };+ return res.signals.map((s) => s.principle).toSorted();+}++describe("brain_query at", () => {+ test("recalls the memories live at the supplied instant", async () => {+ seed();+ expect(await topicQuery({ at: BEFORE_EXPIRY })).toEqual([+ EVERGREEN_PRINCIPLE,+ LAPSED_PRINCIPLE,+ ]);+ });++ test("an instant after the expiry drops the lapsed memory", async () => {+ seed();+ expect(await topicQuery({ at: AFTER_EXPIRY })).toEqual([EVERGREEN_PRINCIPLE]);+ });++ test("a date-only instant is accepted", async () => {+ seed();+ expect(await topicQuery({ at: "2026-06-01" })).toEqual([EVERGREEN_PRINCIPLE, LAPSED_PRINCIPLE]);+ });++ test("an unparseable instant is refused, never coerced to now", async () => {+ seed();+ await expect(topicQuery({ at: "last month" })).rejects.toThrow(+ /ISO-8601 instant or YYYY-MM-DD date/,+ );+ });++ test("'at' outside topic mode is refused rather than silently ignored", async () => {+ seed();+ await expect(+ tool("brain_query").handler(ctx(), { since: BEFORE_EXPIRY, at: BEFORE_EXPIRY }),+ ).rejects.toThrow(/topic/);+ });++ test("'show_expired' outside topic mode is refused on the same terms as 'at'", async () => {+ // Both arguments steer the one expiration filter, and that filter runs+ // in topic mode alone. Its schema already said so while the handler+ // read it in topic mode and ignored it elsewhere, so a caller asking+ // to see lapsed memories beside `since` got silence where it had been+ // promised the opposite. One scope, one refusal.+ seed();+ await expect(+ tool("brain_query").handler(ctx(), { since: BEFORE_EXPIRY, show_expired: true }),+ ).rejects.toThrow(/show_expired.*topic/);+ // Topic mode still accepts it.+ const ok = (await tool("brain_query").handler(ctx(), {+ topic: TOPIC,+ show_expired: true,+ })) as Record<string, unknown>;+ expect(ok["mode"]).toBe("topic");+ });++ test("absent 'at' leaves the default result untouched", async () => {+ seed();+ // The wall clock is past LAPSED_EXPIRY, so the default drops it.+ expect(await topicQuery({})).toEqual([EVERGREEN_PRINCIPLE]);+ });++ test("the input schema declares 'at' (additionalProperties is false)", () => {+ const schema = tool("brain_query").inputSchema as {+ properties: Record<string, { description?: string }>;+ additionalProperties?: boolean;+ };+ expect(schema.additionalProperties).toBe(false);+ expect(Object.keys(schema.properties)).toContain("at");+ });+});++describe("brain_search time-filter descriptions name event time", () => {+ function timeFilterDescription(key: "since" | "until"): string {+ const schema = tool("brain_search").inputSchema as {+ properties: Record<string, { description?: string }>;+ };+ return schema.properties[key]?.description ?? "";+ }++ test("neither filter claims to select on modification time any more", () => {+ for (const key of ["since", "until"] as const) {+ const description = timeFilterDescription(key);+ expect(`${key}: ${description.includes("modified")}`).toBe(`${key}: false`);+ }+ });++ test("both name the axis they actually filter on", () => {+ for (const key of ["since", "until"] as const) {+ const description = timeFilterDescription(key);+ expect(`${key}: ${description.includes("event time")}`).toBe(`${key}: true`);+ }+ });++ test("both stay inside the registry-guard property cap", () => {+ for (const key of ["since", "until"] as const) {+ const length = timeFilterDescription(key).length;+ expect(`${key} within cap: ${length <= PROPERTY_DESCRIPTION_MAX} (${length})`).toBe(+ `${key} within cap: true (${length})`,+ );+ }+ });+});diff --git a/tests/mcp/brain-search-card-duplicates.test.ts b/tests/mcp/brain-search-card-duplicates.test.tsnew file mode 100644index 00000000..880a7e3d--- /dev/null+++ b/tests/mcp/brain-search-card-duplicates.test.ts@@ -0,0 +1,109 @@+/**+ * `brain_search disclosure:'cards'` reports the merged-away duplicate+ * locations (what-the-index-already-knew, task D follow-up).+ *+ * The layer-1 card is the token-cheap surface an agent reads first, and it+ * named no folded location at all: two byte-identical copies produced one+ * card naming one path, so the caller could not learn that the same bytes+ * live somewhere else without paying for full disclosure. The card carries+ * `path:Lstart-Lend` pointers - the grammar it already speaks - not the+ * full location records.+ *+ * The output schema does not set `additionalProperties: false`, so the new+ * field is declared there deliberately; that declaration is asserted here+ * rather than assumed.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { SEARCH_TOOLS } from "../../src/mcp/search-tools.ts";+import { indexVault, resolveSearchConfig } from "../../src/core/search/index.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";++/** The passage duplicated verbatim across files below. */+const PASSAGE =+ "# Quantum entanglement\n\n" ++ "Quantum entanglement links two particles so that measuring one fixes " ++ "the other, regardless of the distance separating them. The correlation " ++ "is not a signal and carries no information on its own.\n";++const POINTER_KEY = "duplicate_pointers";++interface CardsOut {+ cards: Array<Record<string, unknown> & { path: string; duplicate_pointers?: string[] }>;+ total: number;+}++let vault: string;+let configHome: string;+let ctx: { vault: string; configPath: string };++const tool = () => SEARCH_TOOLS.find((t) => t.name === "brain_search")!;++function writeNote(rel: string, body: string): void {+ const abs = join(vault, rel);+ mkdirSync(join(abs, ".."), { recursive: true });+ writeFileSync(abs, body);+}++async function runCards(query: string): Promise<CardsOut> {+ await indexVault(resolveSearchConfig({ vault, configPath: ctx.configPath }), {});+ // eslint-disable-next-line @typescript-eslint/no-explicit-any+ return (await tool().handler(ctx as any, { query, disclosure: "cards" })) as CardsOut;+}++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-card-dup-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-card-dup-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);+ ctx = { vault, configPath };+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++describe("brain_search card duplicate pointers", () => {+ test("a merged card names the location the passage was folded from", async () => {+ writeNote("physics/entanglement.md", PASSAGE);+ writeNote("archive/entanglement-copy.md", PASSAGE);++ const out = await runCards("quantum entanglement");+ expect(out.cards.length).toBe(1);+ const card = out.cards[0]!;+ expect(card[POINTER_KEY]).toBeDefined();+ const pointers = card.duplicate_pointers!;+ expect(pointers.length).toBe(1);+ const named = [card.path, pointers[0]!.slice(0, pointers[0]!.lastIndexOf(":"))].toSorted();+ expect(named).toEqual(["archive/entanglement-copy.md", "physics/entanglement.md"]);+ expect(pointers[0]).toMatch(/:L\d+(-L\d+)?$/);+ });++ test("no duplicate key at all on a card when nothing was merged", async () => {+ writeNote("one.md", "# Alpha\n\nQuantum alpha notes about the first topic entirely.\n");+ writeNote("two.md", "# Beta\n\nQuantum beta notes about the second topic entirely.\n");++ const out = await runCards("quantum notes");+ expect(out.cards.length).toBe(2);+ for (const card of out.cards) {+ expect(POINTER_KEY in card).toBe(false);+ expect(JSON.stringify(card)).not.toContain("duplicate");+ }+ });++ test("the output schema declares the field on the card shape", () => {+ const schema = tool().outputSchema as {+ properties: {+ cards: { items: { properties: Record<string, { type?: string }> } };+ };+ };+ const declared = schema.properties.cards.items.properties[POINTER_KEY];+ expect(declared).toBeDefined();+ expect(declared!.type).toBe("array");+ });+});diff --git a/tests/mcp/brain-search-duplicates.test.ts b/tests/mcp/brain-search-duplicates.test.tsnew file mode 100644index 00000000..f4d5d7b7--- /dev/null+++ b/tests/mcp/brain-search-duplicates.test.ts@@ -0,0 +1,112 @@+/**+ * `brain_search` serializes the merged-away duplicate locations+ * (what-the-index-already-knew, task D landed the merge, task E landed+ * the boundary).+ *+ * The exact-duplicate merge folds a byte-identical passage into the+ * higher-ranked row and records every location it was folded from on+ * `BrainSearchResult.duplicates`. The MCP projection is an explicit field+ * whitelist, so that value was computed end-to-end and dropped at the+ * boundary - the caller saw one row and no way to learn the same bytes+ * live at three paths.+ *+ * House discipline for an optional field: ABSENT, not null, when nothing+ * was merged.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { SEARCH_TOOLS } from "../../src/mcp/search-tools.ts";+import { indexVault, resolveSearchConfig } from "../../src/core/search/index.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";++/** The passage duplicated verbatim across files below. */+const PASSAGE =+ "# Quantum entanglement\n\n" ++ "Quantum entanglement links two particles so that measuring one fixes " ++ "the other, regardless of the distance separating them. The correlation " ++ "is not a signal and carries no information on its own.\n";++interface DuplicateLocation {+ documentId: number;+ chunkId: number;+ path: string;+ title: string | null;+ startLine: number;+ endLine: number;+}++interface SearchOut {+ results: Array<Record<string, unknown> & { path: string; duplicates?: DuplicateLocation[] }>;+ total: number;+}++let vault: string;+let configHome: string;+let ctx: { vault: string; configPath: string };++const tool = () => SEARCH_TOOLS.find((t) => t.name === "brain_search")!;++function writeNote(rel: string, body: string): void {+ const abs = join(vault, rel);+ mkdirSync(join(abs, ".."), { recursive: true });+ writeFileSync(abs, body);+}++async function reindex(): Promise<void> {+ await indexVault(resolveSearchConfig({ vault, configPath: ctx.configPath }), {});+}++async function runSearch(query: string): Promise<SearchOut> {+ // eslint-disable-next-line @typescript-eslint/no-explicit-any+ return (await tool().handler(ctx as any, { query })) as SearchOut;+}++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-dup-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-dup-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);+ ctx = { vault, configPath };+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++describe("brain_search duplicate locations", () => {+ test("a merged row names every location the passage was folded from", async () => {+ writeNote("physics/entanglement.md", PASSAGE);+ writeNote("archive/entanglement-copy.md", PASSAGE);+ await reindex();++ const out = await runSearch("quantum entanglement");+ expect(out.results.length).toBe(1);+ const row = out.results[0]!;+ expect(row.duplicates).toBeDefined();+ expect(row.duplicates!.length).toBe(1);+ const named = [row.path, ...row.duplicates!.map((d) => d.path)].toSorted();+ expect(named).toEqual(["archive/entanglement-copy.md", "physics/entanglement.md"]);+ const folded = row.duplicates![0]!;+ expect(typeof folded.documentId).toBe("number");+ expect(typeof folded.chunkId).toBe("number");+ expect(folded.startLine).toBeGreaterThan(0);+ expect(folded.endLine).toBeGreaterThanOrEqual(folded.startLine);+ });++ test("no duplicates key at all when nothing was merged", async () => {+ writeNote("one.md", "# Alpha\n\nQuantum alpha notes about the first topic entirely.\n");+ writeNote("two.md", "# Beta\n\nQuantum beta notes about the second topic entirely.\n");+ await reindex();++ const out = await runSearch("quantum notes");+ expect(out.results.length).toBe(2);+ for (const row of out.results) {+ expect("duplicates" in row).toBe(false);+ }+ });+});diff --git a/tests/mcp/brain-search-explain-trace.test.ts b/tests/mcp/brain-search-explain-trace.test.tsnew file mode 100644index 00000000..0d853f30--- /dev/null+++ b/tests/mcp/brain-search-explain-trace.test.ts@@ -0,0 +1,159 @@+/**+ * `brain_search` serializes the retrieval decision trace and the memory+ * trust assessment under the existing `explain` flag+ * (what-the-index-already-knew, task F).+ *+ * Both receipts are built on every gated search and were, until this+ * unit, reachable only from a test: neither the MCP response nor the CLI+ * `--json` payload carried them. With the flag off the response shape is+ * byte-identical to the legacy one - no key, not a null.+ *+ * The fixture is deliberately clock-free: the notes are aged past the+ * recency epsilon and access recording is suppressed, so two identical+ * requests serialize to the same bytes and the flag-off / flag-on+ * comparison below is a real byte comparison rather than a tolerant one.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, mkdirSync, rmSync, utimesSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { indexVault, resolveSearchConfig } from "../../src/core/search/index.ts";+import { RETRIEVAL_TRACE_UNAVAILABLE } from "../../src/core/search/explain-envelope.ts";+import { SEARCH_TOOLS } from "../../src/mcp/search-tools.ts";++const QUERY = "widget calibration";+const CLEAN = "---\ntitle: Clean\n---\n\n# Clean\n\nThe widget calibration routine runs daily.\n";+const QUARANTINED =+ "---\ntitle: Bad\nstatus: quarantine\n---\n\n# Bad\n\nThe widget calibration is unsafe.\n";+/** Older than the recency epsilon (~202 days), so the boost is exactly 0. */+const AGED_SECONDS = Math.floor(Date.now() / 1000) - 900 * 24 * 60 * 60;++interface SearchResponse {+ readonly results: ReadonlyArray<Record<string, unknown>>;+ readonly total: number;+ readonly retrieval_decision_trace?: {+ readonly evaluated: number;+ readonly surfaced: number;+ readonly excluded: number;+ readonly exclusions: ReadonlyArray<{+ readonly document_id: number;+ readonly chunk_id: number;+ readonly path: string;+ readonly reasons: ReadonlyArray<string>;+ }>;+ };+ readonly memory_trust_assessment?: {+ readonly evaluated: number;+ readonly surfaced: number;+ readonly quarantined: number;+ readonly reason_counts: Record<string, number>;+ };+ readonly retrieval_trace_unavailable?: string;+}++let vault: string;+let configHome: string;+let gateOn: { vault: string; configPath: string };+let gateOff: { vault: string; configPath: string };++const tool = () => SEARCH_TOOLS.find((t) => t.name === "brain_search")!;++async function call(+ ctx: { vault: string; configPath: string },+ args: Record<string, unknown>,+): Promise<SearchResponse> {+ // eslint-disable-next-line @typescript-eslint/no-explicit-any+ return (await tool().handler(ctx as any, {+ query: QUERY,+ record_access: false,+ ...args,+ })) as unknown as SearchResponse;+}++function writeAged(relPath: string, body: string): void {+ const abs = join(vault, relPath);+ mkdirSync(join(abs, ".."), { recursive: true });+ writeFileSync(abs, body);+ utimesSync(abs, AGED_SECONDS, AGED_SECONDS);+}++beforeEach(async () => {+ vault = mkdtempSync(join(tmpdir(), "o2b-trace-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-trace-cfg-"));+ writeAged("notes/clean.md", CLEAN);+ writeAged("notes/quarantined.md", QUARANTINED);++ const onPath = join(configHome, "gate-on.yaml");+ atomicWriteFileSync(onPath, `vault: ${vault}\nsearch_trust_gate_enabled: true\n`);+ const offPath = join(configHome, "gate-off.yaml");+ atomicWriteFileSync(offPath, `vault: ${vault}\n`);+ gateOn = { vault, configPath: onPath };+ gateOff = { vault, configPath: offPath };++ await indexVault(resolveSearchConfig({ vault, configPath: offPath }), {});+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++describe("brain_search explain: retrieval receipts", () => {+ test("no trace and no assessment key when explain is absent", async () => {+ const out = await call(gateOn, {});++ expect(out.total).toBeGreaterThan(0);+ expect("retrieval_decision_trace" in out).toBe(false);+ expect("memory_trust_assessment" in out).toBe(false);+ expect("retrieval_trace_unavailable" in out).toBe(false);+ });++ test("explain serializes the trace and the assessment the gate already built", async () => {+ const out = await call(gateOn, { explain: true });++ const trace = out.retrieval_decision_trace!;+ expect(trace.evaluated).toBe(2);+ expect(trace.surfaced).toBe(1);+ expect(trace.excluded).toBe(1);+ expect(trace.exclusions).toHaveLength(1);+ expect(trace.exclusions[0]!.path).toBe("notes/quarantined.md");+ expect(trace.exclusions[0]!.reasons).toEqual(["trust_gate:self_approval_quarantine"]);+ expect(typeof trace.exclusions[0]!.document_id).toBe("number");+ expect(typeof trace.exclusions[0]!.chunk_id).toBe("number");++ const assessment = out.memory_trust_assessment!;+ expect(assessment.surfaced).toBe(1);+ expect(assessment.quarantined).toBe(1);+ expect(assessment.reason_counts).toEqual({ "trust_gate:self_approval_quarantine": 1 });++ // The excluded row is counted, never surfaced.+ expect(out.results.map((r) => r["path"])).toEqual(["notes/clean.md"]);+ });++ test("explain with the gate off names why there is no trace instead of going quiet", async () => {+ const out = await call(gateOff, { explain: true });++ expect("retrieval_decision_trace" in out).toBe(false);+ expect("memory_trust_assessment" in out).toBe(false);+ expect(out.retrieval_trace_unavailable).toBe(RETRIEVAL_TRACE_UNAVAILABLE);+ expect(out.retrieval_trace_unavailable).toContain("search_trust_gate_enabled");+ });++ test("byte identity: the flag-off payload is the flag-on payload minus the explain keys", async () => {+ const first = await call(gateOn, {});+ const second = await call(gateOn, {});+ // Precondition: this fixture is deterministic, so the comparison below+ // measures the flag and nothing else.+ expect(JSON.stringify(second)).toBe(JSON.stringify(first));++ const on = (await call(gateOn, { explain: true })) as unknown as Record<string, unknown>;+ delete on["retrieval_decision_trace"];+ delete on["memory_trust_assessment"];+ for (const r of on["results"] as Array<Record<string, unknown>>) delete r["score_breakdown"];++ expect(JSON.stringify(on)).toBe(JSON.stringify(first));+ });+});diff --git a/tests/mcp/brain-search-recall-signals.test.ts b/tests/mcp/brain-search-recall-signals.test.tsnew file mode 100644index 00000000..11b57b22--- /dev/null+++ b/tests/mcp/brain-search-recall-signals.test.ts@@ -0,0 +1,210 @@+/**+ * `brain_search` records the computable recall-quality signals on the+ * EXISTING per-query `recall_telemetry` record, behind the EXISTING+ * `telemetry` gate (what-the-index-already-knew, task G).+ *+ * Three properties are asserted here that the unit test cannot reach:+ *+ * - one record per query, never one per result - the continuity log has+ * no retention policy, so the record count is the thing that matters;+ * - production ordering is unchanged with the gate on, compared as+ * bytes rather than argued;+ * - the signals are derived from what the pipeline surfaced, so the+ * trust block is empty until the caller opts into `trust` while the+ * typed `contradicts` edges are counted either way.+ *+ * The fixture is deliberately clock-free: notes are aged well past the+ * recency epsilon and access recording is suppressed, so two identical+ * requests serialize identically and the gate-off / gate-on comparison+ * is a real byte comparison.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, utimesSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { indexVault, resolveSearchConfig } from "../../src/core/search/index.ts";+import { listRecallTelemetry } from "../../src/core/brain/recall-telemetry.ts";+import { SEARCH_TOOLS } from "../../src/mcp/search-tools.ts";++const QUERY = "widget calibration";+const TELEMETRY_HOST = "mcp-recall-signals";+/** Older than the recency epsilon (~202 days), so the boost is exactly 0. */+const AGED_SECONDS = Math.floor(Date.now() / 1000) - 900 * 24 * 60 * 60;++const CLAIM = [+ "---",+ "title: Claim",+ "contradicts: [[counter]]",+ "---",+ "",+ "The widget calibration routine runs daily against the reference jig.",+].join("\n");+const COUNTER = [+ "---",+ "title: Counter",+ "---",+ "",+ "The widget calibration routine is scheduled weekly, never daily.",+].join("\n");++interface SignalsBlock {+ readonly rows: number;+ readonly alignment: {+ readonly top: number;+ readonly mean: number;+ readonly margin?: number;+ readonly keyword_sum: number;+ readonly semantic_sum: number;+ };+ readonly trust: {+ readonly assessed: number;+ readonly superseded: number;+ readonly conflict: number;+ readonly max_age_days?: number;+ };+ readonly contradiction: { readonly rows: number; readonly edges: number };+ readonly diversity: {+ readonly compared: number;+ readonly pairs: number;+ readonly mean_similarity?: number;+ readonly max_similarity?: number;+ };+ readonly unmeasured?: string;+}++let vault: string;+let configHome: string;+let ctx: { vault: string; configPath: string };++const tool = () => SEARCH_TOOLS.find((t) => t.name === "brain_search")!;++async function call(args: Record<string, unknown> = {}): Promise<Record<string, unknown>> {+ // eslint-disable-next-line @typescript-eslint/no-explicit-any+ return (await tool().handler(ctx as any, {+ query: QUERY,+ record_access: false,+ ...args,+ })) as Record<string, unknown>;+}++function signalsOf(): SignalsBlock {+ const records = listRecallTelemetry(vault, { mode: "search", host: TELEMETRY_HOST });+ expect(records).toHaveLength(1);+ return records[0]!.payload["signals"] as SignalsBlock;+}++function writeAged(relPath: string, body: string): void {+ const abs = join(vault, relPath);+ mkdirSync(join(abs, ".."), { recursive: true });+ writeFileSync(abs, body);+ utimesSync(abs, AGED_SECONDS, AGED_SECONDS);+}++beforeEach(async () => {+ vault = mkdtempSync(join(tmpdir(), "o2b-signals-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-signals-cfg-"));+ writeAged("notes/claim.md", CLAIM);+ writeAged("notes/counter.md", COUNTER);+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);+ ctx = { vault, configPath };+ await indexVault(resolveSearchConfig({ vault, configPath }), {});+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++describe("brain_search recall signals", () => {+ test("the gate is off by default and writes no record at all", async () => {+ const out = await call();++ expect((out["results"] as unknown[]).length).toBeGreaterThan(1);+ expect(listRecallTelemetry(vault, { mode: "search" })).toHaveLength(0);+ expect("telemetry_id" in out).toBe(false);+ });++ test("one record per query, not one per result", async () => {+ const out = await call({ telemetry: true, telemetry_host: TELEMETRY_HOST });++ const rows = out["results"] as unknown[];+ expect(rows.length).toBeGreaterThan(1);+ const records = listRecallTelemetry(vault, { mode: "search", host: TELEMETRY_HOST });+ expect(records).toHaveLength(1);+ expect(records[0]!.payload["result_count"]).toBe(rows.length);+ });++ test("production ordering is byte-identical with the gate on", async () => {+ const off = await call();+ const on = await call({ telemetry: true, telemetry_host: TELEMETRY_HOST });++ expect(JSON.stringify(on["results"])).toBe(JSON.stringify(off["results"]));+ expect(on["total"]).toBe(off["total"]);+ // Everything except the record pointer the gate adds is unchanged.+ delete on["telemetry_id"];+ expect(JSON.stringify(on)).toBe(JSON.stringify(off));+ });++ test("alignment carries the scores the ranker already computed", async () => {+ const out = await call({ telemetry: true, telemetry_host: TELEMETRY_HOST });+ const rows = out["results"] as Array<Record<string, number>>;+ const signals = signalsOf();++ expect(signals.rows).toBe(rows.length);+ expect(signals.alignment.top).toBeCloseTo(rows[0]!["score"]!, 4);+ expect(signals.alignment.margin).toBeCloseTo(rows[0]!["score"]! - rows[1]!["score"]!, 4);+ expect(signals.alignment.keyword_sum).toBeGreaterThan(0);+ });++ test("contradiction counts typed edges with no trust opt-in", async () => {+ await call({ telemetry: true, telemetry_host: TELEMETRY_HOST });+ const signals = signalsOf();++ expect(signals.contradiction).toEqual({ rows: 1, edges: 1 });+ expect(signals.trust).toEqual({ assessed: 0, superseded: 0, conflict: 0 });+ });++ test("trust is assessed only when the caller asked for it", async () => {+ await call({ telemetry: true, telemetry_host: TELEMETRY_HOST, trust: true });+ const signals = signalsOf();++ expect(signals.trust.assessed).toBe(2);+ expect(signals.trust.conflict).toBe(1);+ expect(signals.trust.max_age_days).toBeGreaterThan(800);+ });++ test("diversity is a bounded pool-level overlap of the surfaced rows", async () => {+ await call({ telemetry: true, telemetry_host: TELEMETRY_HOST });+ const signals = signalsOf();++ expect(signals.diversity.compared).toBe(2);+ expect(signals.diversity.pairs).toBe(1);+ expect(signals.diversity.mean_similarity!).toBeGreaterThan(0);+ expect(signals.diversity.mean_similarity!).toBeLessThan(1);+ });++ test("the cards surface names why it could not measure instead of going quiet", async () => {+ await call({+ telemetry: true,+ telemetry_host: TELEMETRY_HOST,+ disclosure: "cards",+ });+ const signals = signalsOf();++ expect(signals.unmeasured).toBe("disclosure_cards");+ expect("alignment" in signals).toBe(false);+ });++ test("a single-row window omits the pairwise numbers it cannot compute", async () => {+ await call({ telemetry: true, telemetry_host: TELEMETRY_HOST, limit: 1 });+ const one = listRecallTelemetry(vault, { mode: "search", host: TELEMETRY_HOST })[0]!;+ const signals = one.payload["signals"] as SignalsBlock;+ expect(signals.rows).toBe(1);+ expect("margin" in signals.alignment).toBe(false);+ expect(signals.diversity).toEqual({ compared: 1, pairs: 0 });+ });+});diff --git a/tests/mcp/brain-search-snippet-anchor.test.ts b/tests/mcp/brain-search-snippet-anchor.test.tsnew file mode 100644index 00000000..c733d3a1--- /dev/null+++ b/tests/mcp/brain-search-snippet-anchor.test.ts@@ -0,0 +1,111 @@+/**+ * `brain_search` returns a match-anchored `content` window+ * (what-the-index-already-knew, task E).+ *+ * The MCP surface capped `content` at 600 chars taken from the HEAD of+ * the chunk, so a query term occurring later in a long passage never+ * reached the caller: the agent saw 600 characters of preamble and no+ * evidence of why the row ranked. The window is now centred on the first+ * significant query term the content contains.+ *+ * A chunk with no query-term occurrence is unchanged - the head window,+ * the same bytes - and that is asserted, not assumed.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { SEARCH_TOOLS } from "../../src/mcp/search-tools.ts";+import { indexVault, resolveSearchConfig, search } from "../../src/core/search/index.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";++/** Mirrors `MCP_CONTENT_MAX` in `src/mcp/search-tools.ts`. */+const MCP_CONTENT_MAX = 600;+/** Chunk cap for the fixture vault: one long passage stays one chunk. */+const FIXTURE_CHUNK_SIZE = 4000;++const FILLER = "This paragraph records background context for the passage. ";+const ANCHOR_TERM = "entanglement";+const MATCH_SENTENCE = `Quantum ${ANCHOR_TERM} links two particles at a distance.`;++/** Lead long enough that the match sits well past `MCP_CONTENT_MAX`. */+const LEAD = FILLER.repeat(24);+const BODY = `${LEAD}${MATCH_SENTENCE} ${FILLER.repeat(24)}`;++let vault: string;+let configHome: string;+let ctx: { vault: string; configPath: string };++const tool = () => SEARCH_TOOLS.find((t) => t.name === "brain_search")!;++interface SearchOut {+ results: Array<{ path: string; content: string }>;+ total: number;+}++async function runSearch(query: string): Promise<SearchOut> {+ // eslint-disable-next-line @typescript-eslint/no-explicit-any+ return (await tool().handler(ctx as any, { query })) as SearchOut;+}++/** The untruncated stored chunk the MCP row is a window onto. */+async function storedChunk(query: string): Promise<string> {+ const outcome = await search(resolveSearchConfig({ vault, configPath: ctx.configPath }), {+ query,+ limit: 10,+ });+ const row = outcome.results.find((r) => r.path === "notes/physics.md");+ expect(row).toBeDefined();+ return row!.content;+}++beforeEach(async () => {+ vault = mkdtempSync(join(tmpdir(), "o2b-anchor-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-anchor-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nsearch_chunk_size: ${FIXTURE_CHUNK_SIZE}\n`);+ mkdirSync(join(vault, "notes"), { recursive: true });+ writeFileSync(join(vault, "notes", "physics.md"), `---\ntitle: Physics\n---\n\n${BODY}\n`);+ ctx = { vault, configPath };+ await indexVault(resolveSearchConfig({ vault, configPath }), {});+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++describe("brain_search snippet anchoring", () => {+ test("the fixture stores one chunk with the match past the content cap", async () => {+ const chunk = await storedChunk(ANCHOR_TERM);+ expect(chunk.length).toBeGreaterThan(MCP_CONTENT_MAX);+ expect(chunk.indexOf(ANCHOR_TERM)).toBeGreaterThan(MCP_CONTENT_MAX);+ });++ test("a match past the content cap appears in the returned window", async () => {+ const out = await runSearch(ANCHOR_TERM);+ expect(out.total).toBeGreaterThan(0);+ const row = out.results.find((r) => r.path === "notes/physics.md");+ expect(row).toBeDefined();+ // The window is still capped, so this is not "the whole chunk came back".+ expect(row!.content.length).toBeLessThanOrEqual(MCP_CONTENT_MAX);+ expect(row!.content).toContain(ANCHOR_TERM);+ });++ test("a head-anchored window is the pre-change head truncation, byte for byte", async () => {+ // Both significant terms occur inside the first 600 chars of the+ // filler, so the anchored window starts at 0 - the defined head+ // window. The expected bytes are recomputed from the stored chunk+ // with the pre-change rule, not read back from the function.+ const query = "paragraph background";+ const chunk = await storedChunk(query);+ const expected =+ chunk.length <= MCP_CONTENT_MAX ? chunk : `${chunk.slice(0, MCP_CONTENT_MAX - 1)}…`;+ const out = await runSearch(query);+ const row = out.results.find((r) => r.path === "notes/physics.md");+ expect(row).toBeDefined();+ expect(row!.content).toBe(expected);+ });+});diff --git a/tests/mcp/brain-search-window-boundary.test.ts b/tests/mcp/brain-search-window-boundary.test.tsnew file mode 100644index 00000000..cd2cf967--- /dev/null+++ b/tests/mcp/brain-search-window-boundary.test.ts@@ -0,0 +1,129 @@+/**+ * `brain_search` never opens its `content` window inside a surrogate pair+ * (what-the-index-already-knew, task E follow-up).+ *+ * The MCP window slices UTF-16 units at a start that anchoring moved off+ * zero. When that start lands on the LOW half of a surrogate pair the+ * payload begins with a lone surrogate, which the JSON-RPC frame's UTF-8+ * encoding turns into U+FFFD - the caller reads a replacement character+ * where an emoji was. The hazard is new: before anchoring, the start was+ * always zero.+ *+ * The head window must stay byte-identical to the pre-anchoring head+ * truncation, and that is asserted against bytes recomputed from the+ * stored chunk rather than read back from the surface under test. That+ * expectation deliberately still carries the TRAILING cut's own+ * surrogate split: the tail has always cut on a raw UTF-16 offset, the+ * bytes it produces are the pre-change contract, and this wave does not+ * change them.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { SEARCH_TOOLS } from "../../src/mcp/search-tools.ts";+import { indexVault, resolveSearchConfig, search } from "../../src/core/search/index.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";++/** Mirrors `MCP_CONTENT_MAX` in `src/mcp/search-tools.ts`. */+const MCP_CONTENT_MAX = 600;+/** Mirrors `ELLIPSIS` in `src/core/search/snippet-window.ts`. */+const ELLIPSIS = "…";+/** Chunk cap for the fixture vault: the whole passage stays one chunk. */+const FIXTURE_CHUNK_SIZE = 4000;++/** Occurs at offset 0, so a query for it anchors the window at the head. */+const HEAD_TERM = "prelude";+const ANCHOR_TERM = "entanglement";+const EMOJI = "\u{1F600}";+const REPLACEMENT_CHAR = "�";+const NOTE_PATH = "notes/emoji.md";++/**+ * `prelude ` (8 units), 400 emoji (800 units), a space, then the anchor+ * term at offset 809. The unaligned window start is 809 - 300 = 509, the+ * low half of an emoji. The trailing run keeps the chunk over the cap.+ */+const BODY = `${HEAD_TERM} ${EMOJI.repeat(400)} ${ANCHOR_TERM}${" tail".repeat(40)}`;++let vault: string;+let configHome: string;+let ctx: { vault: string; configPath: string };++const tool = () => SEARCH_TOOLS.find((t) => t.name === "brain_search")!;++interface SearchOut {+ results: Array<{ path: string; content: string }>;+ total: number;+}++async function runSearch(query: string): Promise<SearchOut> {+ // eslint-disable-next-line @typescript-eslint/no-explicit-any+ return (await tool().handler(ctx as any, { query })) as SearchOut;+}++/** The untruncated stored chunk the MCP row is a window onto. */+async function storedChunk(query: string): Promise<string> {+ const outcome = await search(resolveSearchConfig({ vault, configPath: ctx.configPath }), {+ query,+ limit: 10,+ });+ const row = outcome.results.find((r) => r.path === NOTE_PATH);+ expect(row).toBeDefined();+ return row!.content;+}++async function contentFor(query: string): Promise<string> {+ const out = await runSearch(query);+ const row = out.results.find((r) => r.path === NOTE_PATH);+ expect(row).toBeDefined();+ return row!.content;+}++beforeEach(async () => {+ vault = mkdtempSync(join(tmpdir(), "o2b-window-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-window-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nsearch_chunk_size: ${FIXTURE_CHUNK_SIZE}\n`);+ mkdirSync(join(vault, "notes"), { recursive: true });+ writeFileSync(join(vault, NOTE_PATH), `---\ntitle: Emoji\n---\n\n${BODY}\n`);+ ctx = { vault, configPath };+ await indexVault(resolveSearchConfig({ vault, configPath }), {});+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++describe("brain_search content window boundaries", () => {+ test("the fixture really does anchor the window inside a surrogate pair", async () => {+ const chunk = await storedChunk(ANCHOR_TERM);+ expect(chunk.length).toBeGreaterThan(MCP_CONTENT_MAX);+ const unaligned = chunk.indexOf(ANCHOR_TERM) - Math.floor(MCP_CONTENT_MAX / 2);+ const unit = chunk.charCodeAt(unaligned);+ expect(unit).toBeGreaterThanOrEqual(0xdc00);+ expect(unit).toBeLessThanOrEqual(0xdfff);+ });++ test("the returned window survives a UTF-8 round trip with no replacement char", async () => {+ const content = await contentFor(ANCHOR_TERM);+ // The window opens on a whole emoji, so nothing at the HEAD degrades.+ expect(content.startsWith(`${ELLIPSIS}${EMOJI}`)).toBe(true);+ const roundTripped = new TextDecoder().decode(new TextEncoder().encode(content));+ expect(roundTripped).not.toContain(REPLACEMENT_CHAR);+ expect(content).toContain(ANCHOR_TERM);+ expect(content.length).toBeLessThanOrEqual(MCP_CONTENT_MAX);+ });++ test("a head-anchored window is the pre-change head truncation, byte for byte", async () => {+ const chunk = await storedChunk(HEAD_TERM);+ const expected =+ chunk.length <= MCP_CONTENT_MAX+ ? chunk+ : `${chunk.slice(0, MCP_CONTENT_MAX - ELLIPSIS.length)}${ELLIPSIS}`;+ expect(await contentFor(HEAD_TERM)).toBe(expected);+ });+});