Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: recall lane concurrency, overlapped_ms accounting, a caller-selectable packing policy and standing-instruction facets (v1.84.0) (#252)
Serg Eroshe14a99584c6b6
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex f21da8e0..8b20164f 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.83.0",+ "version": "1.84.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex a5268dbe..4e8fafe6 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.83.0",+ "version": "1.84.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex d98830d9..e2328ae4 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,19 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.84.0] - 2026-10-14++Open Second Brain 1.84.0 hides the recall lane's embed round trip behind the keyword work and accounts for the overlap: hybrid search now starts the semantic lane before the keyword lanes run and joins it where the serial await stood, `search --explain` reports the overlapped milliseconds, and the bench latency family carries the same figure per question and in aggregate. Every surface is byte-identical when the new option is absent, and no configuration changes.++### Added++- **The semantic lane starts before the keyword lanes.** Hybrid search's one caller wait - the query embed - is now put in flight before `runKeywordLane` runs: the lane starts through a join seam that attaches its rejection handler in the same tick, and the join stands where the serial await stood, in both deadline arms. The lane start stays after the query-cache probe, so cached answers still cost nothing; an already-expired composite deadline starts no embed at all (the never-start rule moved to the start site); and a keyword-step throw with the embed in flight can no longer strand an unhandled rejection - with a deadline the lane's late, non-abort failure is named by the `hybrid deadline: an abandoned lane failed after the answer:` sentence, without one it is defused. The keyword pool size reaches the lane as a supplier read at lane completion, so the `hybrid_degraded` signal keeps grading the same snapshot the serial pipeline captured; the post-join void and every later deadline checkpoint are untouched. Results, warning order and the query-cache keys are byte-identical: the deadline-degraded answer stays keyword-only, and the wall-clock budget is neither started earlier nor extended.+- **`overlapped_ms` on `search --explain`.** The overlap of the embed and keyword intervals - `max(0, min(embedEnd, keywordEnd) - max(embedStart, keywordStart))`, zero whenever the embed never ran - rides a per-call timing sink on the search options, never the outcome: the query cache persists whole outcomes, so an outcome field would freeze one call's timings into every cached row served afterwards. With `--explain`, the JSON payload carries an integer `overlapped_ms` beside `semantic_attempted`, and the human transcript prints the overlapped line in the explain block; without the flag the payload is byte-identical (off means absent). A cross-vault union captures no timing, because a union merges one search per origin and no honest single figure exists for it.+- **A caller-selectable packing policy over the pack's two candidate populations.** `brain_context_pack` gains `packing_policy` (`facts_first` or `retired_first`), strictly opt-in and byte-identical when omitted: inside one pack the preference-derived pages and the retired pages already share the single `max_tokens` budget, and the policy names which lane claims it first - `facts_first` claims for the distilled facts lane before the pages lane fills the exact remainder, `retired_first` hands the first claim to the retired pages lane - query-ordered in the `ranked` and `semantic` query modes, tier-then-recency ordered in the default substring mode, where the query filtered rather than ranked. Neither member is the omitted default: the omitted pack interleaves both populations by tier, while a named policy orders whole lanes - every preference before every retired page under `facts_first` - reserves the attention charge up front instead of drawing it last, and refuses a vault where one lane holds no candidate in reach, a call the omitted pack succeeds on. Claiming is strictly sequential with no minimum-page floor, the total never grows, the per-memory character cap stays lane-internal, and the synthesized attention block stays a reserved charge neither lane draws; lane overflow keeps the shipped `over-budget` skip reason. The pure `allocateRecallBudget` primitive lands beside the shared `applyCharBudget` (frozen vocabulary, narrowing guard, lane allocations plus any reserved charge always summing to the total), and the report's `packing` accounting - the policy name, the reserved attention charge, plus each lane's allocation, delivered spend, candidate count and item count - is wired through the one finalize funnel, so the char-budget path agrees with it; a zero budget with a policy named gets no accounting at all, because that call is refused before anything is returned. The refusals all fire by name and before any receipt or telemetry is written: a non-member value, a policy without a query (inert), a policy against a one-lane vault (INVALID_PARAMS naming the lane and the member, and both lanes when both are empty), a zero budget with a policy named, and a reserved attention charge that consumes the whole budget. The MCP argument mirrors `query_mode` argument for argument: the schema enum derives from the frozen tuple, the query pairing is declared `dependentRequired` style, and every refusal arrives on the wire as INVALID_PARAMS carrying the named message; the response gains `packing` only when the policy was named. Receipt and telemetry budget metadata and the prompt-prefix metric segment carry the policy as a closed-vocabulary member under the same only-when-named rule. No CLI flag (the `query_mode` MCP-first precedent) and no recall-inject or hook change - the hook runs a single lane, where a policy would be inert by the design's own refusal rule.+- **`overlapped_ms` in the bench latency family.** The bench passes the per-call timing sink for every retrieval question, so a run whose suite declares the embeddings capability and pins the semantic lane reports `overlapped_ms` per question and its mean in the aggregate `latency_ms` family - both present only when at least one question attempted the embed. Keyword-only runs (the default pin) add no key anywhere: the report JSON, the committed failure-modes baseline and the human output stay byte-identical, and the human `overlapped_ms: avg N` line prints only when the aggregate carries the figure.+- **Opt-in standing-instruction facets with first-mention provenance.** The `standing_query` attention action gains a `standingInstructionFacets` option: unset, every emitted string is the legacy `pref-<slug> (scope)` output byte for byte and sections carry no facet data; set, a surfaced preference whose provenance join resolves gains a first-mention annotation on its item string (` [first mention turn <session>#<turn>]`, or ` [first mention event ...]` for a capture-lifecycle origin, which is not guaranteed to be a turn id) plus a structured `StandingInstructionFacet` entry - the preference-corpus sibling of `ThreadFacet`, same contract shape (stable `pref:<id>` key, ordered origins, pure derivation), distinct name so the coverage lane's meaning of facet stays unambiguous. The join is a new pure read-time module (`src/core/brain/preference-provenance.ts`, no I/O, the signal lookup injected) over `evidenced_by` wikilinks and producer-aware `session_ref` parsing: import-session `basename#turn` resolves as a turn, capture-lifecycle `session:<id>#<event>` as an origin event labeled as what it is, and the bare `session:<id>` the auto-extract path persists resolves to nothing - the never-guess rule covers every other shape, so an unknown ref yields no provenance rather than an invented coordinate. The option changes annotations, never membership: ownership and retirement exclusions apply before annotation, so a facet cannot surface a preference the plain lane withholds; a preference whose join does not resolve keeps the plain string and is absent from the facets - an honest zero, never an implied provenance; and the whole lane performs zero writes, standing-rules opacity and the vault-identity guard unchanged.+- **A `standing_instruction` bench category gates the facet lane.** The new category joins `BENCH_CATEGORIES` and drives the shipped surfacing call with the facet option on over the fixture vault, so the lane ships measured: `evaluateQuestion` asserts the expected facet keys - `pref:<preference id>`, the preference id itself - and the expected first-mention annotation, whose kind word keeps an event-annotated origin from satisfying a turn expectation; a surfaced preference whose join does not resolve is a named failure, never an implied provenance. `BenchPinnedKnobs` gains the lane's identity marker: it changes no behavior - the category is armed by the fixture - but folds into the checkpoint hash, so a resumed run that would mix a facet-gate arm with a plain one fails the identity guard, while the default pin serializes byte-for-byte unchanged. Reports without the category keep their bytes, and the human `standing_instruction: <passed>/<total> (pass rate <n>)` line prints only when the quality buckets carry the category; the exit code stays all-pass-or-nonzero with the category in play.+## [1.83.0] - 2026-10-13Open Second Brain 1.83.0 wires four shipped seams, each opt-in and byte-identical when off: a per-source-session cap on the search candidate pool, usage-ordered skill offers on `skills_attach`, seeded question-type-proportional sampling for the LongMemEval bench suite, and the OpenCode V2 config shape for the opencode install. No new MCP tools, no new continuity record kinds, no new external dependencies.@@ -8497,6 +8510,7 @@ Hermes / Claude Code / Codex / OpenClaw configurations do not change.- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.84.0]: https://github.com/itechmeat/open-second-brain/compare/v1.83.0...v1.84.0[1.83.0]: https://github.com/itechmeat/open-second-brain/compare/v1.82.0...v1.83.0[1.82.0]: https://github.com/itechmeat/open-second-brain/compare/v1.81.1...v1.82.0[1.81.1]: https://github.com/itechmeat/open-second-brain/compare/v1.81.0...v1.81.1diff --git a/docs/brainstorm/recall-lane-latency/cli-output/prompt.md b/docs/brainstorm/recall-lane-latency/cli-output/prompt.mdnew file mode 100644index 00000000..d7a3a4a8--- /dev/null+++ b/docs/brainstorm/recall-lane-latency/cli-output/prompt.md@@ -0,0 +1,81 @@+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.++# Task++Epic "Recall Lane Latency and Caller Control" for Open Second Brain (v1.84.0 wave). Four kanban tasks ship as one release on one feature branch. Theme: make the recall lane faster and give callers honest, opt-in control and accounting over what it returns.++## Task 1 (t_862b4d29, p4): Run the semantic lane concurrently with the keyword lanes in hybrid search++Upstream gbrain v0.60.214.0 starts the query embedding and vector search at the same time as the keyword/lexical arms instead of after them, joining the vector side before fusion. On a large brain this removed roughly 130-150 ms of median search latency by hiding the query-embed round trip behind keyword execution, with byte-identical results and an `overlapped_ms` timing breakdown showing how much of each stage was hidden.++Open Second Brain's hybrid search runs its lanes strictly sequentially, so every semantic query pays the full query-embed latency after keyword retrieval finishes. Overlapping the lanes is a pure latency win for interactive recall (CLI and MCP search alike) with no change to ranking, and the same `overlapped_ms` accounting would keep `search --explain` timings honest.++Validator verdict: premise confirmed with minor line drift. The keyword lane is synchronous (`runKeywordLane`, src/core/search/pipeline/keyword-lane.ts:64) and completes at src/core/search/search.ts:413-431 before the semantic lane is awaited at 454-461 (`awaitWithinDeadline` at 457 under the composite deadline). The embed round trip is awaited inside the lane (semantic-phase.ts:185-189 embed, :232 semanticTopK). No start-early/join exists on the query path. Two risks: a dangling in-flight embed when the keyword lane throws (unhandled rejection / closed-store hazard), and `keywordHitCount` (semantic-lane.ts:32, read by `detectHybridDegrade` at 109-113) which would read zero at call time under start-early. `overlapped_ms` must ride a per-call sink, never `SearchOutcome`: the query cache persists whole outcomes (cache-slot.ts:472, finalize guard search.ts:402-410).++## Task 2 (t_767a6bd3, p3): Caller-selectable recall packing policy (query-first vs facts-first) for the limited reading budget++Upstream gbrain v0.58.0.0 adds an explicit, caller-selectable packing policy for recall's limited token reading budget: `query_first` spends the budget on matching source pages first and lets stored facts fill the remainder, versus the existing facts-first default. Exposed as a `budget_policy` option, with explicit packing accounting and identical validation across local and remote transports. Omitting the option preserves current behavior. Upstream deliberately keeps facts-first as default (a misrouted fact question regressed 1/1 to 0/1 under query-first).++Validator verdict: premise partly false. The budget primitive exists and the `queryMode` opt-in precedent ships (frozen tuple + guard + option at context-pack.ts:113-121/258, MCP coercion and refusals at pack-tools.ts:186-204, schema enum at 1056-1062, `dependentRequired` at 1141). But pages and facts are budgeted by two separate surfaces - `maxTokens` + `applyCharBudget` in src/core/brain/context-pack.ts (option at 245, loop 726-765, char budgets 715/815) versus the recall-inject hook's own caps (RECALL_INJECT_MAX_NOTES/MAX_CHARS/TIME_BUDGET_MS, recall-inject.ts:42-48, config specs in src/core/config.ts:1413-1447) - so there is no single budget for a packing order to reorder. Corrected scope: correlate the budgets first, then add the knob modeled on queryMode, opt-in per caller, default unchanged. Operator note for this wave: if the budget correlation grows during design, land concurrency, facets and the fixture first and keep the knob strictly opt-in or defer that one card. Both surfaces already denominate content with the same `estimateTokens` currency (shared from src/core/brain/text/tokenizer.ts), and src/core/brain/recall-budget.ts is the one module both trimming paths reuse.++## Task 3 (t_4b2ffde5, p3): Opt-in standing-instruction facets with first-mention provenance++Upstream yantrikdb v0.17.0 ships an opt-in "standing-instruction facets" lane: standing instructions are faceted for recall, each facet exposing the first mention turn, with a measured retrieval improvement (+0.875 on the standing-instruction category in the upstream harness, no default behavior change). Open Second Brain has a rich standing-preference system (Brain/preferences pref-* notes) but its runtime recall surface exposes bare `pref-<slug> (scope)` strings (src/core/brain/attention-flows.ts:196-221) with no first-mention provenance. Ship as opt-in and gate the lane on a bench run rather than shipping unmeasured.++Validator verdict: premise partly false. The facet gap is real, but provenance has a home: preferences carry `created_at`/`evidenced_by` (preference.ts:148/:151) and a dream-stamped `valid_from` (:248/:904); the turn id exists one hop away in `BrainSignal.session_ref` typed `<path>#<turn-id>` (types.ts:877), baked in by import-session (import.ts:445). The thread-recall lane has since shipped `ThreadFacet`/`ThreadFacetMention`/`ThreadFacetCluster` (src/core/brain/thread-topics.ts:38-59), so the work is a read-time join following that contract style, not a new stamp - and a distinct type name to avoid overloading "facet". Residual gap: extract-signals persists a bare `session:<id>` ref (extract-signals.ts:646), so auto-extracted preferences have no turn to join. Operator standing rules must never be rewritten; the lane is read-only. Capture lifecycle refs look like `session:<id>#<event>` (session-lifecycle.ts:858) - event ids, not guaranteed turn ids. The bench (`runMemoryBench`, src/core/bench/phases.ts) has a frozen category tuple (`BENCH_CATEGORIES`, types.ts:36-48), pinned knobs (`BenchPinnedKnobs`, types.ts:247-262, part of the run identity hash; `DEFAULT_BENCH_KNOB_PIN` at 260) and an all-pass-or-nonzero exit code.++## Task 4 (t_1b2b6932, p1): Shared deterministic OpenAI-compatible mock endpoint for local and CI test runs++Upstream honcho PR #1094 provides a single shared, deterministic OpenAI-API-compatible HTTP fixture (embeddings + rerank) so search/embedding/rerank flows can be integration-tested with no keys and no flakiness, collapsing duplicated per-test HTTP stubs.++Validator verdict: premise partly false - the consolidation already exists. tests/helpers/fake-http.ts is the shared deterministic fixture (44 references): Bun server with a per-test `setHandler` seam, a default handler serving only `/embeddings` with vectors derived deterministically from the input text (fake-http.ts:37-50), 404 otherwise, call counter, per-instance `/i{N}/v1` path segment added to defeat a port-reuse flake. Honest scope: add a `/rerank` route to the default handler (wrapped `{ results: [{ index, relevance_score }], model }` shape, one item per document, deterministic scoring derived from the request text) and move the cross-encoder tests onto it (tests/core/search/rerank.cross-encoder.test.ts stubs `globalThis.fetch` against a fake endpoint constant today; three pipeline files duplicate a `stubRerankFetch(winner)` stub). No standalone mock server, no chat-completions route (no chat flow exists in this tree). Watch the byte-pinned warning strings: the fixture JSON-encodes bodies, so raw-body 503 cases that pin `rerank_degraded:` messages should stay at the fetch seam.++# Project context++Open Second Brain: TypeScript on Bun 1.4.0, second brain for AI agents over Obsidian-compatible Markdown vaults plus SQLite (FTS5 + sqlite-vec) index. CLI (`o2b brain <verb>`, `o2b search`), MCP server, python Hermes plugin. Markdown files are authority; SQLite and state files are recomputable caches.++Recent commits:+e239d323 chore(release): v1.83.0+(recent waves shipped as single squashed feature PRs: recall lanes, decision model, thread recall, honest embedding spend, memory bench)++Related files:+- src/core/search/search.ts (composite deadline, awaitWithinDeadline, lane orchestration, finalize/cache guard)+- src/core/search/pipeline/keyword-lane.ts, pipeline/semantic-lane.ts, pipeline/post-rank.ts (raceRerank), pipeline/cache-slot.ts+- src/core/search/semantic-phase.ts (query embed, semanticTopK), src/core/search/types.ts (SearchOptions, freshenSpawn per-call hook precedent)+- src/core/search/explain-envelope.ts, src/cli/search/outcome-render.ts, src/cli/search/verbs/query.ts (--explain, off-means-absent house rule)+- src/core/brain/context-pack.ts (maxTokens packing loop, applyCharBudget, queryMode pattern), src/core/brain/recall-budget.ts (shared trim primitive), src/core/brain/recall-inject.ts (hook-side caps), src/core/config.ts (RECALL_INJECT_CAP_SPECS)+- src/mcp/brain/pack-tools.ts (query_mode coercion, refusals, schema enum, dependentRequired)+- src/core/brain/attention-flows.ts (standing_query emission), src/core/brain/thread-topics.ts (ThreadFacet contract), src/core/brain/preference.ts, src/core/brain/signal.ts, src/core/brain/sessions/import.ts, src/core/brain/extract-signals.ts, src/core/brain/session-lifecycle.ts+- src/core/bench/types.ts (BENCH_CATEGORIES, BenchPinnedKnobs, DEFAULT_BENCH_KNOB_PIN), src/core/bench/phases.ts (runMemoryBench), src/cli/brain/verbs/bench.ts+- tests/helpers/fake-http.ts (shared deterministic fixture)++Conventions:+- Deterministic, bounded, fail-closed core; no model calls on recall paths; provider-dependent work degrades cleanly under the composite deadline.+- Neutral defaults: an omitted new option is byte-identical to the previous release on every surface, pinned by tests (the queryMode absent-option pin, the bench baseline fixture, the search resultShape projections).+- Nothing silently falls back: refusals are named; degraded answers carry named warnings; a knob that cannot be honest on a surface is not shipped there.+- Every new capability ships as: core module + caller surface + tests (bun test) + docs; TDD with atomic conventional commits; oxlint/oxfmt baseline; TypeScript strict, no `as` cast crutches.+- Repeated values hoisted into named constants; no hardcoded natural-language phrases outside named constants; English everywhere.++Constraints:+- Do not change existing public APIs: new options are additive and optional; the frozen bench category tuple, knob pin, MCP tool list and pinned warning strings must stay stable or move deliberately.+- The recall-inject hook has a hard 2,500 ms retrieval budget; no change may add a second retrieval pass to it.+- The query cache persists whole outcomes, so nothing per-call (timings) may ride SearchOutcome.+- Operator note: if the packing-knob budget correlation grows during design, land concurrency, facets and the fixture first and keep the knob strictly opt-in or defer that one card.+- Four lanes must be able to land in parallel with disjoint files; shared seams (bench runner, explain schema, docs, CHANGELOG) need pinned hunk ownership.++# Required output format++Produce exactly 3 distinct architectural variants. For each variant:++### Variant N: <short name>+- **Approach**: 2-3 sentences describing the variant.+- **Trade-offs**: bullet list of pros and cons.+- **Complexity**: small | medium | large+- **Risk**: low | medium | high++After the three variants, add exactly one recommendation:++### Recommended: Variant N+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.++Output nothing outside of these sections.diff --git a/docs/brainstorm/recall-lane-latency/design.md b/docs/brainstorm/recall-lane-latency/design.mdnew file mode 100644index 00000000..7a1c799f--- /dev/null+++ b/docs/brainstorm/recall-lane-latency/design.md@@ -0,0 +1,77 @@+# Recall Lane Latency and Caller Control - overlapped hybrid lanes, opt-in packing policy, standing-instruction facets, rerank test fixture++**Status:** draft+**Author:** claude-dev-agent (via feature-release-playbook)+**Audience:** implementation++## Problem statement++The recall lane costs more and tells callers less than it should. Hybrid search runs its semantic lane strictly after the keyword lane completes (`runKeywordLane` at src/core/search/pipeline/keyword-lane.ts:64 is synchronous; the await stands at src/core/search/search.ts:454-461), so every semantic query pays the full query-embed round trip after keyword retrieval, and nothing reports how much of that could be hidden. Callers also cannot steer what the reading budget buys: pages and facts-side candidates are packed in one fixed order, and standing instructions surface as bare `pref-<slug> (scope)` strings (src/core/brain/attention-flows.ts:196-221) with no first-mention provenance even though the provenance already exists in frontmatter. Finally, the rerank contract is tested by stubbing `globalThis.fetch` against a fake endpoint because the shared fixture (tests/helpers/fake-http.ts) serves only `/embeddings`. Four kanban cards, one spine: make the recall lane faster, and give callers honest, opt-in control and accounting over what it returns.++## Premise verdicts (from recon, anchors re-verified at HEAD e239d323)++- **t_862b4d29 (semantic lane concurrency) - CONFIRMED.** The serial await is real: keyword lane completes at search.ts:413-431 (hits bound at 429, warnings merged 430-431) before the semantic lane is started and awaited at 454-461 (`await runSemanticLane` at 456, `awaitWithinDeadline` arm at 457-460). The embed round trip is awaited inside the lane (semantic-phase.ts:185-189 embed, :232 `semanticTopK`). No start-early/join exists on the query path. Minor card drift recorded by recon: `runKeywordLane` is at search.ts:413 (card said 430); the await-chain anchor 456 is exact. Two risks folded into the design: a dangling in-flight embed when the keyword lane throws (new failure mode with start-early; mitigation in Design decisions), and `keywordHitCount` being read at lane-start time (semantic-lane.ts:32, consumed by `detectHybridDegrade` at semantic-lane.ts:109-113) which would silently drop `hybrid_degraded` under start-early - hence the lazy supplier. `overlapped_ms` must ride a per-call sink, not `SearchOutcome`: the query cache persists the whole outcome (persistCachedOutcome call at src/core/search/pipeline/cache-slot.ts:472, guarded at search.ts:402-410), so an outcome field would freeze stale timings into cached rows.+- **t_767a6bd3 (packing policy) - PARTLY FALSE, corrected scope honored and narrowed once more.** The budget primitive and the `queryMode` opt-in precedent both exist (frozen tuple and guard at src/core/brain/context-pack.ts:113-121, option at :258, MCP coercion and refusals at src/mcp/brain/pack-tools.ts:186-204, schema enum at :1056-1062, `dependentRequired` at :1136-1141). But the premise of one budget to reorder is false: pages live under `ContextPackOptions.maxTokens` (context-pack.ts:245, loop at 726-765, `applyCharBudget` at 715 and 815) while the facts-side brief lives under the hook's own caps (`RECALL_INJECT_MAX_NOTES`/`MAX_CHARS`/`TIME_BUDGET_MS` at src/core/brain/recall-inject.ts:42-48). The two surfaces never run in one call, so a cross-surface packing order is not implementable without growing the hook past its 2,500 ms budget. Per the operator note the design scopes down: correlate the two candidate populations that DO share one budget - preference-derived candidates and page candidates inside `packContext`'s single `maxTokens` - and make the knob strictly opt-in, MCP-only, omitted meaning byte-identical. The recall-inject option, the config key and the hook threading (recon task 4) are deferred, not stubbed.+- **t_4b2ffde5 (standing-instruction facets) - PARTLY FALSE, rescope honored.** True: no standing-instruction facet mechanism and no first-mention surfacing exists. Overstated and corrected: provenance has a home - `created_at`/`evidenced_by` at src/core/brain/preference.ts:148/:151, dream-stamped `valid_from` at :248/:904, and the turn id one hop away in `BrainSignal.session_ref` typed as `<path>#<turn-id>` (src/core/brain/types.ts:877), baked in by import (src/core/brain/sessions/import.ts:445). The "zero facet symbols" claim is stale: `ThreadFacet`/`ThreadFacetMention`/`ThreadFacetCluster` shipped at src/core/brain/thread-topics.ts:38-59, so the new lane reuses that contract style with a distinct type name. Residual real gap: `commitExtractedSignals` persists a bare `session:<id>` ref (src/core/brain/extract-signals.ts:646), so auto-extracted preferences have no turn to join; the join reports no-provenance there honestly and the write-path fix is out of scope.+- **t_1b2b6932 (shared rerank fixture) - PARTLY FALSE, rescope honored.** The consolidation the card asked for already exists: tests/helpers/fake-http.ts is the shared deterministic fixture and its default handler (lines 37-50) serves only `/embeddings`, 404ing everything else. The honest scope is the validator's: add a `/rerank` route to the default handler and migrate the cross-encoder tests onto the fixture. The byte-pinned degraded-warning risk is handled by leaving raw-body 503 cases at the fetch seam (the fixture JSON-encodes bodies, which would change the pinned `rerank_degraded:` strings).++## Scope++- **Lane D - fixture substrate (lands first, test-only):** a deterministic `/rerank` branch in the fake-http default handler (wrapped response shape, one item per document, whitespace-overlap scoring `matches / (matches + 1)`, 400 on a body without an array `documents`, model echo) and migration of rerank.cross-encoder.test.ts plus the three pipeline rerank test files onto the fixture.+- **Lane A - lane concurrency:** start the semantic lane before `runKeywordLane`, join where the await stands today under both deadline arms, lazy `keywordHitCount` supplier read at lane completion, preserved warning merge order, rejection handling for a lane in flight when the keyword step throws, `overlapped_ms` measured as the intersection of the embed and keyword intervals and threaded through a per-call timing sink into `search --explain` and the bench latency family.+- **Lane B - packing policy:** a frozen `PACKING_POLICIES` vocabulary (`facts_first`, `query_first`) and a pure `allocateRecallBudget` helper in src/core/brain/recall-budget.ts; an optional `packingPolicy` on the context-pack options that reorders which candidate population claims the pack's existing single token budget; a `packing` accounting object on the report; the `brain_context_pack` MCP argument, schema enum and refusals mirroring `query_mode`; conditional receipt/telemetry/prompt-prefix metadata. Omitted everywhere means byte-identical.+- **Lane C - standing-instruction facets:** a pure read-time provenance join (module src/core/brain/preference-provenance.ts) over `evidenced_by` links and producer-aware `session_ref` parsing; a `StandingInstructionFacet` projection offered as an additive option on the attention-flow `standing_query` evaluation; a `standing_instruction` bench category with an additive knob pin; conditional bench verb surfacing; zero writes end to end.+- Docs and CHANGELOG hunks pinned per lane (see plan.md cross-lane contract).++## Out of scope++- **Recall-inject hook packing (deferred, not stubbed):** no `packingPolicy` on `RecallInjectOptions`, no `recall_budget_policy` config key, no hook change. The hook runs a single lane; a policy there is inert by the design's own refusal rule, and teaching it to fetch page bodies cannot fit the 2,500 ms budget. Its own card if wanted later.+- **CLI flag for the packing knob:** `o2b brain context-pack` stays as is; MCP-first parity follows the `query_mode` precedent (ranked/semantic shipped MCP-only; the CLI flag was a separate parity change).+- **MCP surfacing of `overlapped_ms`:** `searchAcrossVaults` merges per-origin outcomes and a per-call sink yields one value per origin; no honest single value exists for the merged payload. `search --explain` and the bench are the surfaces.+- **Extract-signals turn stamping:** persisting `source_turn` onto signal frontmatter so auto-extracted preferences become joinable is a write-path change beyond the corrected scope.+- **fake-http `rawBody`/stall extension and chat-completions route:** the three stream-level fetch stubs in rerank-failure.test.ts and the doctor tripwire stay at the fetch seam; no chat flow exists in this tree.+- **Standing-instruction facets over MCP or `ContextPackOptions`:** the facet ships as an option on the attention-flow evaluation and is proven by the bench gate, matching the upstream discipline of measuring before exposing. A pack-tool argument would also collide with lane B's file ownership.++## Chosen approach++Variant 2, "Substrate-First Opt-In Seams" (variants.md): the shared fixture grows its `/rerank` route first because every later lane's tests lean on it; then each recall improvement lands behind the seam it already owns, with byte-identical behavior whenever the new option is absent. Concurrency is start-early/join-late with the keyword lanes authoritative under the composite deadline; the packing policy is a pure allocation reorder inside the pack's single budget, opt-in per call; facets are a read-time join annotated only onto what recall already delivers, gated by a bench category so the lane ships measured; and `overlapped_ms` accounting rides a per-call timing sink so the query cache never sees it.++## Design decisions++- **Start-early, join-late, keyword lanes authoritative.** `search()` calls the semantic lane before `runKeywordLane`, holding the promise; the embed HTTP request leaves in that tick: the provider semaphore's synchronous fast path (`tryAcquire`) takes a free permit without yielding, so the dispatch precedes the keyword lanes; the pre-call await inside the provider would have deferred it past them, which the external review caught and the dispatch-order test now pins. The join happens where the await stands today (search.ts:454-461), keeping both arms: direct await when the deadline is null, the same `awaitWithinDeadline` semantics over the already-started promise otherwise (the helper gains an overload accepting a started promise). The post-join void (search.ts:469-473) and every later `deadlineExpired` checkpoint are untouched, so the keyword-only degrade contract stays exactly as shipped. The never-start rule (a deadline already expired at lane start does not start the embed) moves to the start site, mirroring awaitWithinDeadline's behavior at search.ts:225-229. Lane start stays after the cache probe so cached answers still cost nothing.+- **Lazy `keywordHitCount`.** `SemanticLaneInput.keywordHitCount` becomes `number | (() => number)`, read where `detectHybridDegrade` runs at lane completion (semantic-lane.ts:109-113). The union keeps the four numeric call sites in tests/core/search/semantic-phase-spend-gate.test.ts compiling. The value stays byte-identical: the lane cannot complete before the join, which is after `keywordHits` is assigned and before the second-pass reassignments - the same snapshot search.ts:451 evaluates today. The deadline's own closure (search.ts:329) is unchanged.+- **No dangling embed.** Attaching the rejection handler at start time is mandatory: with a deadline, a lane rejection arriving after abandonment routes to `noteAbandonedFailure` (search.ts:196-201, prefix constant at :108); without one, the handler joins the lane into the existing error path so there is no unhandled rejection and no `store.semanticTopK` against a closed store (the hazard already documented at semantic-phase.ts:224).+- **Byte-identical results.** Lane hit arrays are unchanged data, so fusion, MMR and the trail see identical inputs. Warning merge order is preserved: keyword warnings pushed synchronously right after `runKeywordLane` returns (today's 430-431), semantic warnings at the join (464-465). The regression harness is the `resultShape` projection (tests/core/search/hybrid-deadline.test.ts:110) plus new byte-shape assertions in tests/core/search/hybrid-overlap.test.ts.+- **`overlapped_ms` formula pinned:** `max(0, min(embedEndMs, keywordEndMs) - max(embedStartMs, keywordStartMs))`, zero whenever the lane did not attempt the embed. Interval intersection is symmetric and measures actual overlap; upstream's "embed hidden past keyword end" variant is derivable later from the same timing record if a consumer wants it.+- **Per-call timing sink, not an outcome field.** A `SearchOptions` hook in the `freshenSpawn` mold (src/core/search/types.ts:1131) receives one record (embed start/end, keyword start/end, overlapped, attempted) at the join. The query cache persists whole outcomes (cache-slot.ts:472), so nothing timing-shaped ever reaches `SearchOutcome` and cached rows cannot carry stale timings. Deadline-degraded answers are already never cached (search.ts:402-410).+- **Explain and bench projection.** The `--explain` envelope gains the timing only when the flag is on (house rule: off means absent, bytes identical; envelope at src/core/search/explain-envelope.ts:122, projection in src/cli/search/outcome-render.ts:114-208) and the human transcript prints the overlapped line only under `--explain`. The bench passes the sink per question (retrieve loop around src/core/bench/phases.ts:405-414) and the aggregate latency family (phases.ts:775-805) gains `overlapped_ms` only when at least one question attempted the semantic lane; keyword-only suites keep their report bytes and the committed baseline unchanged. The human line (src/cli/brain/verbs/bench.ts:431) gains a conditional `overlapped_ms` line under the same rule.+- **Packing: correlate first, reorder second, inside one budget.** `allocateRecallBudget(totalTokens, pagesLane, factsLane, policy)` is a pure helper beside `applyCharBudget` in recall-budget.ts - both lanes already denominate content with the same `estimateTokens` currency, and per-lane native caps (chars, note counts) stay lane-internal fit rules. Inside `packContext`, the facts lane is the preference-derived candidate population and the pages lane is the rest; `facts_first` (default member) reproduces the current outcome because tier gating already lets the distilled layer claim first; `query_first` lets the query-ordered page lane claim before the facts lane fills the remainder. Claiming is strictly sequential with no minimum-page floor, matching upstream's simple semantics; the empty-lane refusal covers the degenerate cases. The total never grows - the rule recall-inject already applies (recall-inject.ts:197). A policy against an empty lane is refused by name, never a silent no-op, mirroring pack-tools' "a mode with nothing to read is inert" refusal (pack-tools.ts:197). Naming follows the repo's option-naming style - `packingPolicy` / `packing_policy` - rather than upstream's `budget_policy`, because the option reorders packing under a fixed budget and never resizes the budget.+- **Packing knob strictly opt-in, omitted equals byte-identical.** With `packingPolicy` absent the partition never runs and the existing code path executes unchanged, pinned the way `context-pack-ranked-query.test.ts` pins the absent `queryMode`. The report's `packing` accounting object, the MCP response member, the receipt/telemetry metadata and the prompt-prefix segment appear only when the caller named a policy (`contextPackBudgetMetadata` at context-pack.ts:1011 feeds the receipt path; the prefix segment mirrors the `query_mode` conditional at context-pack.ts:998). The synced telemetry carries the policy name as a closed enum member, which satisfies the names-nothing-sensitive rule. `applyCharBudget`'s trimming and the attention block's reserved charge stay charged inside whichever lane draws them.+- **Facets: read-time join, thread-facet contract, honest gaps.** `StandingInstructionFacet` follows the `ThreadFacet` shape (stable `pref:<id>` key, ordered establishing turns, pure function) with a distinct name and a doc-comment cross-reference so the two facet meanings do not blur. The join parses `#` fragments producer-aware - `basename#turn` from import-session (import.ts:445), `session:<id>#<event>` from the capture lifecycle (src/core/brain/session-lifecycle.ts:858, labeled origin event, not origin turn), bare `session:<id>` from extract-signals (extract-signals.ts:646, no provenance) - qualifies the fragment with its session key (the `qualifiedTurnId` lesson), orders by the preference's own evidence, and returns the earliest as first mention. Unknown, absent or unrecognized shapes yield no provenance, never a guess. With the option unset, `pref-<slug> (scope)` output is byte-identical; with it set, only resolving preferences gain annotations, the retirement exclusion applies before annotation (a retired preference is renamed out of `preferences/`, so the directory walk never sees it) while the lane applies no owner filter - owner-held preferences surface identically with the option on and off - and the lane performs zero writes (standing-rules opacity and `assertStandingRulesNotTargeted` stay the boundary).+- **Facets measured before exposure.** The `standing_instruction` member joins `BENCH_CATEGORIES` (src/core/bench/types.ts:36-48); the retrieve phase drives the surfacing call with the facet option on for the new questions and `evaluateQuestion` asserts the preference id plus origin turn. `BenchPinnedKnobs` gains one additive field so lane-on runs get a distinct identity while `DEFAULT_BENCH_KNOB_PIN` (types.ts:260) and existing run ids stay untouched. The gate is two runs of the same suite (lane off asserts the default report is unchanged; lane on asserts the category passes); no comparison-report schema. The verb prints the category only when present, mirroring the conditional longmemeval block. Reports without the category serialize byte-identically.+- **Fixture: additive route, migration at the handler seam.** The `/rerank` branch lives inside `defaultHandler` (tests/helpers/fake-http.ts:37-50); no exported API changes, all existing consumers untouched. Scoring is whitespace-token overlap `matches / (matches + 1)` - deterministic, in [0, 1), monotone, no clock or randomness; a test forces an ordering by planting the query token in one document. Degenerate bodies get 400 so a misdirected test fails visibly. The cross-encoder suite migrates wholesale (capturing `setHandler` for request-shape assertions, `callCount()` for the empty-documents short-circuit); the three pipeline suites replace their duplicated `stubRerankFetch` with `doc.includes(winner)` handlers, preserving winner selection exactly. Raw-body 503 pins stay at the fetch seam, so the pinned `rerank_degraded:` strings do not change.++## File changes++New: `src/core/brain/preference-provenance.ts` (lane C join), `tests/core/search/hybrid-overlap.test.ts` (A), `tests/core/search/outcome-shape-census.test.ts` (A, the sanctioned outcome-key census), `tests/core/brain/preference-provenance.test.ts` and a standing-instruction facet/bench test file (C), a packing-policy test file for core plus one for the MCP surface (B).++Modified - lane A: `src/core/search/search.ts` (start-early/join, sink), `src/core/search/pipeline/semantic-lane.ts` (supplier type, timing record), `src/core/search/semantic-phase.ts` (embed timing), `src/core/search/types.ts` (sink option, timing record type), `src/core/search/index.ts` (SearchLaneTiming re-export), `src/core/search/pipeline/request.ts` (the readable `clock` port every lane-interval stamp reads through), `src/core/search/explain-envelope.ts`, `src/cli/search/outcome-render.ts`, `src/cli/search/verbs/query.ts`, `src/core/bench/phases.ts`, `src/cli/brain/verbs/bench.ts`, `src/core/bench/types.ts` (latency family only), `CHANGELOG.md` (creates the wave version section), `docs/updating.md`, plus tests/core/search/semantic-phase-spend-gate.test.ts and hybrid-deadline.test.ts extensions.++Modified - lane B: `src/core/brain/recall-budget.ts` (vocabulary, allocation), `src/core/brain/context-pack.ts` (option, partition, accounting), `src/core/brain/belief-semantic.ts` (`spentQueryDisclosure` extracted so the lane-empty refusal carries the spend sentence), `src/mcp/brain/pack-tools.ts` (argument, schema, response), docs/mcp.md, CHANGELOG.md, docs/updating.md.++Modified - lane C: `src/core/brain/attention-flows.ts` (additive option on the standing_query evaluation), `src/core/bench/types.ts` (category member, knob pin field), `src/core/bench/phases.ts` (category branches), `src/cli/brain/verbs/bench.ts` (conditional print; `src/core/bench/fixture.ts` needs no change - category validation already rides the tuple), `CHANGELOG.md` (appends facet bullets), `docs/updating.md`.++Modified - lane D: `tests/helpers/fake-http.ts` (default handler only), `tests/helpers/fake-http.test.ts`, `tests/core/search/rerank.cross-encoder.test.ts`, `tests/core/search/rerank.search.test.ts`, `tests/core/search/relational-rerank-pin.test.ts`, `tests/core/search/rerank-sunset.test.ts`.++Shared seams and hunk ownership are pinned in plan.md's cross-lane contract (bench files split between A and C; CHANGELOG section created by A, appended by B and C).++## Risks and open questions++- **Unhandled rejection window (lane A, new failure mode):** between lane start and join, a keyword-step throw can strand an in-flight embed. Mitigated by the attach-at-start rule above; the hybrid-overlap tests cover both deadline arms, and the existing stall and byte-shape tests in hybrid-deadline.test.ts must pass unchanged.+- **Windows timer clamping (lane A):** small overlaps can flatten to 0 on CI; timing assertions stay lower-bounded with margins, mirroring the existing deadline wall-clock tests.+- **Bench report shape (lanes A and C):** the aggregate JSON and the baseline fixture must stay byte-identical for keyword-only, lane-off runs. Both lanes' bench tasks pin this explicitly; the baseline (failure-mode rates) is verified unmoved.+- **Hook-side packing deferred (lane B):** resolved per the operator note rather than left open - the knob stays pack-only until a shared budget owner exists. Revisit only if a caller asks for facts-brief correlation inside the hook's budget.+- **Provenance coverage honesty (lane C):** most real preferences (auto-extracted, hand-authored) resolve no turn; the facet reports resolved counts honestly rather than implying provenance everywhere, and the bench fixtures state which population they annotate.+- **Fixture consumer breadth (lane D):** 42 test files consume `startFakeHttp`; the route is additive to the default handler only, but the full `bun test` run before landing the substrate is the guard against any consumer relying on `/rerank` 404ing (none found in recon).+- **Overlapped_ms on multi-vault MCP payloads:** intentionally unresolved; if a caller surfaces a need, the per-origin values can be exported per origin before any merged single-value decision is made. Owner: lane A follow-up card.+- **Query-first regression discipline:** upstream kept facts-first default because a misrouted fact question regressed under query-first; the same caution holds here - the default member preserves current behavior and the bench gate pattern (two runs) is the template any future default change must follow.diff --git a/docs/brainstorm/recall-lane-latency/plan.md b/docs/brainstorm/recall-lane-latency/plan.mdnew file mode 100644index 00000000..01f6cf40--- /dev/null+++ b/docs/brainstorm/recall-lane-latency/plan.md@@ -0,0 +1,168 @@+# Recall Lane Latency and Caller Control - implementation plan++Feature branch: `feat/recall-lane-latency`. TDD per task: write the failing test first, run it, watch it fail with the expected error, then implement. One atomic conventional commit per task, formatter + linter green before every commit.++Landing order: lane D first (pure test substrate), then lane A, then lane B, then lane C. Lanes B and C tasks C1-C2 are file-disjoint from A and may proceed in parallel on separate worktrees; the bench hunks are sequenced by the dependency edges below.++## Cross-lane contract (pinned)++Shared file seams. Parallel lanes never edit the same file; where two lanes must touch one file, the hunk boundaries and the owning lane are pinned here and the dependency edge sequences them.++| Shared file | Owner | Hunk |+|---|---|---|+| `tests/helpers/fake-http.ts`, `tests/helpers/fake-http.test.ts` | Lane D exclusively | `/rerank` default-handler branch and its self-test. No other lane touches the helper. |+| `src/core/search/**`, `src/cli/search/**`, search explain surfaces | Lane A exclusively | Start-early/join, supplier, timing sink, explain envelope and transcript. Lanes B and C never edit search files. |+| `src/core/brain/context-pack.ts`, `src/core/brain/recall-budget.ts`, `src/mcp/brain/pack-tools.ts`, `docs/mcp.md` | Lane B exclusively | Packing vocabulary, allocation, partition, accounting, MCP argument/schema/response, mcp.md section. Lanes A and C never edit these. |+| `src/core/brain/attention-flows.ts`, `src/core/brain/preference-provenance.ts` (new) | Lane C exclusively | Facet option on the standing_query evaluation, provenance join. |+| `src/core/bench/types.ts` | Lane A then Lane C (sequenced, disjoint regions) | A owns the aggregate latency-family field addition (report family near types.ts:378 region). C owns the `BENCH_CATEGORIES` member (types.ts:36-48) and the `BenchPinnedKnobs` field (types.ts:247-262). C3 depends on A6. |+| `src/core/bench/phases.ts` | Lane A then Lane C (sequenced) | A owns the per-question timing-sink pass-through in the retrieve loop (405-414 region) and the `overlapped_ms` aggregation in the report family (775-805 region). C owns the `standing_instruction` retrieve branch (alongside the session_handoff/proactive_recall branches, 399-501 region) and the `evaluateQuestion` branch (573-652 region). C3 depends on A6. |+| `src/cli/brain/verbs/bench.ts` | Lane A then Lane C (sequenced) | A owns the conditional `overlapped_ms` human line adjacent to line 431. C owns the conditional `standing_instruction` report block (mirroring the conditional longmemeval block). C4 depends on A6. |+| `CHANGELOG.md` | Lane A creates the version section; B and C append bullets | D adds none (test-only change, no entry per playbook). A lands first with the concurrency bullets; B appends the packing bullets; C appends the facet bullets. Sequential landing makes the hunks conflict-free. |+| `docs/updating.md` | A, B, C each add their own additive-options note | Distinct lines; landing order D, A, B, C keeps it conflict-free. |++Rule inherited by every lane: an omitted new option must be byte-identical on every surface it touches, and any new English user-facing string lives in a named constant.++## Lane D - fixture substrate (lands first, test-only)++### Task D1: `/rerank` default route in the fake-http fixture+- **Lane**: D+- **Files**: `tests/helpers/fake-http.test.ts` (failing tests first), `tests/helpers/fake-http.ts` (default handler body only; no exported API change)+- **Test first**: in the self-test, assert POST `{url}/rerank` with `{ model, query, documents }` returns 200 `{ results: [{ index, relevance_score }...], model }` with one item per document in input order; two identical requests produce byte-identical responses; a document containing a query token outscores one that does not (monotone in overlap); GET `/rerank` does not 200; a body without an array `documents` answers 400; the `/embeddings` branch behavior is unchanged (vector derivation, model echo, 404 elsewhere).+- **Then implement**: the branch inside `defaultHandler` per the design (whitespace-token overlap scoring `matches / (matches + 1)`, lowercase and punctuation-stripped, no clock or randomness; model echo `body.model ?? "fake-model"`; empty `documents` yields `{ results: [], model }`).+- **Done when**: the self-test is green and a sample of existing embedding-consumer suites (at minimum `tests/core/search/embeddings.test.ts` and `tests/core/search/rerank-failure.test.ts`) still pass, proving the additive route regressed nothing.+- **Depends on**: none++### Task D2: migrate `rerank.cross-encoder.test.ts` onto the fixture+- **Lane**: D+- **Files**: `tests/core/search/rerank.cross-encoder.test.ts`+- **Test first**: re-point one test at a started `FakeHttp` while keeping its assertions, watch the migration surface any contract mismatch, then migrate the rest: the endpoint constant becomes `server.url`; the canonical request-shape test rides the default route plus a capturing `setHandler` that records `req`; the out-of-order, bare-array, wrong-length and 503 variants become `setHandler` overrides returning the exact bodies they return today; the empty-documents test asserts `server.callCount() === 0`.+- **Done when**: the file contains no `globalThis.fetch` assignment, `let server: FakeHttp` wiring matches the house consumer pattern, and the suite is green.+- **Depends on**: D1++### Task D3: migrate the three pipeline rerank files onto the fixture+- **Lane**: D+- **Files**: `tests/core/search/rerank.search.test.ts`, `tests/core/search/relational-rerank-pin.test.ts`, `tests/core/search/rerank-sunset.test.ts`+- **Test first**: per file, start a server, pass `baseUrl: server.url` in the rerank config handed to `makeConfig`, and replace the duplicated `stubRerankFetch(winner)` with one local `setHandler` scoring `doc.includes(winner) ? 0.99 : 0.01` (preserving today's winner selection exactly, reading `req.body` from the already-parsed `FakeRequest`); the cache test's call-counting wrapper becomes `server.callCount()` deltas; the disabled-means-zero-HTTP tripwire becomes the same assertion on `callCount()`. The raw-body 503 pins stay at the fetch seam so the pinned `rerank_degraded:` warning strings do not change.+- **Done when**: rerank-related `globalThis.fetch` mutations remain only in `tests/core/search/rerank-failure.test.ts` (stream-level cases) and the doctor tripwire; full `bun test` over the migrated files is green.+- **Depends on**: D1++## Lane A - semantic lane concurrency (recon tasks 1-6)++### Task A1: lazy `keywordHitCount` supplier+- **Lane**: A+- **Files**: `src/core/search/pipeline/semantic-lane.ts`; `tests/core/search/semantic-phase-spend-gate.test.ts`+- **Test first**: a failing assertion where the lane is given a supplier whose backing value changes while the embed is in flight and the emitted `hybrid_degraded` warning reflects the completion-time count (today the type has no supplier, so a start-early caller would capture the call-time value); a second assertion pins that a plain number is still accepted (union `number | (() => number)`), keeping the four existing numeric call sites (266-292 region) compiling untouched.+- **Then implement**: widen `SemanticLaneInput.keywordHitCount`, resolve the supplier where `detectHybridDegrade` runs (semantic-lane.ts:109-113 region).+- **Done when**: new tests green; the four existing call sites unmodified; the suite green.+- **Depends on**: D1 (fixture available), none otherwise++### Task A2: extract the join seam+- **Lane**: A+- **Files**: `src/core/search/search.ts` (new helper/overload); `tests/core/search/hybrid-overlap.test.ts` (new)+- **Test first**: unit tests over the extracted seam using fake promises: (a) the keyword step runs to completion while the lane promise is still pending; (b) a keyword-step throw while the lane is in flight routes the lane's rejection to `noteAbandonedFailure` when a deadline exists and produces no unhandled rejection when none does; (c) the never-start rule - an already-expired deadline starts nothing and serves the fallback.+- **Then implement**: an `awaitWithinDeadline` overload accepting an already-started `Promise<T>` (or a `joinSemanticLane` helper carrying the same catch-attachment semantics, search.ts:220-250 region).+- **Done when**: the new tests are green; the existing rerank race and deadline tests pass unchanged.+- **Depends on**: none++### Task A3: start-early join in `search()`+- **Lane**: A+- **Files**: `src/core/search/search.ts`; `tests/core/search/hybrid-overlap.test.ts`+- **Test first**: using the fake-http pattern from hybrid-deadline.test.ts with a delayed embed handler: results byte-identical to the serial baseline via the `resultShape` projection (hybrid-deadline.test.ts:110 pattern), and the embed request observed arriving at the fake server before `search()` resolves; warning order pinned (keyword sentences before semantic sentences).+- **Then implement**: start the semantic lane before `runKeywordLane`, join at today's await site (search.ts:454-461 region), both deadline arms, signal still riding the lane input; attach the rejection handler at start time per the design; lane start stays after the cache probe.+- **Done when**: hybrid-overlap tests green AND the full existing `tests/core/search/hybrid-deadline.test.ts`, `hybrid-degrade.test.ts`, `tuning-semantic-lane.test.ts`, `iterative-recall.test.ts`, `relational-arm.test.ts` suites pass unchanged (byte-identical contract).+- **Depends on**: A1, A2++### Task A4: deadline-authoritative overlap and the abandoned-lane sentence+- **Lane**: A+- **Files**: `src/core/search/search.ts` (start-site never-start rule if not already in A2's helper); `tests/core/search/hybrid-deadline.test.ts` (extensions)+- **Test first**: with the lane started early and the clock at 300ms over a stalled embed, the answer stays keyword-only with `hybridDeadlineExceeded` plus `hybridDegraded` (the existing assertions must keep passing under the overlapped join); NEW: an embed rejecting with a non-abort error after the fire lands the `hybrid deadline: an abandoned lane failed after the answer:` sentence (prefix constant, search.ts:108) in warnings.+- **Done when**: all deadline tests green with the overlapped join active, including the wall-clock assertions (answer under 3s on a 300ms budget) and the never-cached contract.+- **Depends on**: A3++### Task A5: `overlapped_ms` through the timing sink and `search --explain`+- **Lane**: A+- **Files**: `src/core/search/types.ts` (per-call sink option in the `freshenSpawn` mold, timing record type), `src/core/search/semantic-phase.ts` (embed start/end capture), `src/core/search/pipeline/semantic-lane.ts` (record plumbed out), `src/core/search/search.ts` (interval capture around the keyword lane, sink invoked once at the join), `src/core/search/explain-envelope.ts`, `src/cli/search/outcome-render.ts`, `src/cli/search/verbs/query.ts`, `CHANGELOG.md` (creates the wave version section with the concurrency bullets - the section lanes B and C append to), `docs/updating.md` (own additive-options line); `tests/cli/search-explain-overlap.test.ts` (new, sibling of search-explain-trace.test.ts)+- **Test first**: with `--explain` and a delayed fake embed, the JSON payload carries an integer `overlapped_ms >= 0` and the human transcript prints the overlapped line; without `--explain` the payload bytes are unchanged (key absent, off-means-absent rule); a keyword-only run (embed never attempted) reports `overlapped_ms` of 0 with `attempted` false.+- **Done when**: new tests green; no timing field exists anywhere on `SearchOutcome`; the existing explain-trace suite passes unchanged.+- **Depends on**: A3++### Task A6: bench `overlapped_ms` plumbing+- **Lane**: A+- **Files**: `src/core/bench/types.ts` (aggregate latency-family field only), `src/core/bench/phases.ts` (sink per question in the retrieve loop, aggregation into the report family), `src/cli/brain/verbs/bench.ts` (conditional human line), `CHANGELOG.md` (appends the bench-accounting bullet to the section A5 created); `tests/core/bench/phases.test.ts` (extension)+- **Test first**: a bench run whose fixture exercises the semantic lane reports `overlapped_ms` in the latency family (per-question and aggregate); a keyword-only fixture reports it absent or zero without adding keys elsewhere; the human line prints only when the aggregate carries it; the failure-modes baseline fixture stays byte-identical.+- **Done when**: bench tests green, `tests/core/bench/failure-modes-baseline.test.ts` green and unmoved.+- **Depends on**: A5. Sequencing note: lane C's bench tasks build on this task's hunks (cross-lane contract).++## Lane B - packing policy (recon tasks 1-3 and 5; task 4 deferred per the operator note)++### Task B1: packing-policy vocabulary and pure allocation+- **Lane**: B+- **Files**: `src/core/brain/recall-budget.ts`; `tests/core/brain/recall-budget.packing.test.ts` (new)+- **Test first**: membership and narrowing cases mirroring the `isContextPackQueryMode` guard; allocation cases: `query_first` reordering, `facts_first` identity, empty page pool hands the budget to facts (and vice versa), facts exceeding the remainder, total conservation (the sum of lane allocations equals the total, reserved charges included), and per-lane native caps staying lane-internal.+- **Then implement**: frozen `PACKING_POLICIES = ["facts_first", "query_first"]` tuple, type, `isPackingPolicy` guard, and the pure `allocateRecallBudget` helper. Nothing else changes; no wiring.+- **Done when**: the new suite is green; no other file modified.+- **Depends on**: D1 not required; none++### Task B2: pack partition and report accounting+- **Lane**: B+- **Files**: `src/core/brain/context-pack.ts` (option field, partition activating only when the option is present, `packing` report object wired through the one finalize funnel); `tests/core/brain/context-pack-packing.test.ts` (new)+- **Test first**: absent `packingPolicy` is byte-identical to today (pin copied from the absent-`queryMode` pattern in context-pack-ranked-query.test.ts:86); with a policy, the report carries both lane spends and the policy name; skipped reasons unchanged; a policy against an empty lane is refused by name; the zero-budget early return and the char-budget path agree with the funnel.+- **Done when**: new tests green and the existing `context-pack.test.ts`, `context-pack-ranked-query.test.ts`, `context-pack-semantic.test.ts`, `pre-compress-pack.test.ts` suites pass untouched.+- **Depends on**: B1++### Task B3: MCP `brain_context_pack` argument, schema, response+- **Lane**: B+- **Files**: `src/mcp/brain/pack-tools.ts`; `tests/mcp/context-pack-tool.test.ts` (extension) or a sibling file for the packing cases+- **Test first**: INVALID_PARAMS for a non-member with the message naming the frozen members; the inert single-lane refusal; omitted argument produces a byte-identical response; the `packing` accounting object appears only when the policy was named; the schema enum derives from the tuple and the pairing is declared declaratively (`dependentRequired` style); `tests/mcp/brain-tools-parity.test.ts` stays green and unchanged (new argument, not a new tool).+- **Done when**: MCP tests green including the parity pin.+- **Depends on**: B2++### Task B4: explain surfaces and docs+- **Lane**: B+- **Files**: `src/core/brain/context-pack.ts` (receipt/telemetry metadata via `contextPackBudgetMetadata`, prompt-prefix segment gaining the policy only when named, mirroring the `query_mode` conditional at context-pack.ts:998); `docs/mcp.md` (section modeled on the query_mode one at 2054-2068); `docs/updating.md` (own line); `CHANGELOG.md` (appends packing bullets to the section lane A created); test coverage folded into `tests/core/brain/context-pack-packing.test.ts`+- **Test first**: prompt-prefix stability for unnamed callers; metadata presence only when the policy was named; the synced telemetry carries the policy as a closed enum member.+- **Done when**: tests green; docs complete; no CLI flag shipped (out of scope per design).+- **Depends on**: B3++## Lane C - standing-instruction facets (recon tasks 1-5)++### Task C1: pure origin-join core+- **Lane**: C+- **Files**: `src/core/brain/preference-provenance.ts` (new, no I/O); `tests/core/brain/preference-provenance.test.ts` (new)+- **Test first**: each producer format exercised from its live writer - import-session `basename#turn` (import.ts:445 shape), capture lifecycle `session:<id>#<event>` (session-lifecycle.ts:858 shape, resolved as origin event, labeled as such), extract-signals bare `session:<id>` (extract-signals.ts:646 shape, no provenance); malformed and missing refs; duplicate turns; deterministic ordering; the never-guess rule (unknown shape yields no provenance). Purity pin: the module imports no fs/vault APIs.+- **Then implement**: parse `<key>#<fragment>` session refs, resolve `evidenced_by` wikilinks to signal `session_ref` values via an injected lookup, return ordered `(sessionKey, fragment, kind)` hints with the earliest as first mention.+- **Done when**: the new suite is green and the purity pin holds.+- **Depends on**: D1 not required; none++### Task C2: opt-in facet projection on the standing_query evaluation+- **Lane**: C+- **Files**: `src/core/brain/attention-flows.ts` (additive option; new `StandingInstructionFacet` type in the `ThreadFacet` style with the doc-comment cross-reference); `tests/core/brain/standing-instruction-facets.test.ts` (new)+- **Test first**: with the option unset, the emitted `pref-<slug> (scope)` strings are byte-identical to today; with it set, only preferences whose join resolves gain origin annotations; owner-scoped and retired/tombstoned exclusions apply before annotation (a facet never makes a withheld preference visible); no write syscalls occur against a vault holding standing rules, scoped rules and owner-scoped preferences (fs write-guard).+- **Done when**: new tests green; `tests/core/brain/attention-standing-query.test.ts` and `attention-flows.test.ts` pass untouched.+- **Depends on**: C1++### Task C3: bench `standing_instruction` category and knob pin+- **Lane**: C+- **Files**: `src/core/bench/types.ts` (category member and additive `BenchPinnedKnobs` field only - region ownership per cross-lane contract), `src/core/bench/phases.ts` (retrieve branch driving the surfacing call with the facet option on, `evaluateQuestion` branch); `src/core/bench/fixture.ts` needs no change (category validation rides the existing tuple check); `tests/core/bench/standing-instruction.test.ts` (new)+- **Test first**: reports without the new category serialize byte-identically; lane-on questions assert the preference id plus origin turn; `DEFAULT_BENCH_KNOB_PIN` serializes byte-for-byte unchanged; a resumed run whose pin changed fails the identity guard.+- **Done when**: new tests green; `tests/core/bench/failure-modes-baseline.test.ts` green and unmoved; existing bench suites untouched.+- **Depends on**: C2, and A6 (bench file sequencing per cross-lane contract)++### Task C4: bench verb surfacing+- **Lane**: C+- **Files**: `src/cli/brain/verbs/bench.ts` (conditional category block only); `tests/cli/brain-bench.test.ts` (extension) or a sibling file+- **Test first**: a run whose report carries no `standing_instruction` category keeps stdout byte-identical; a lane-on run prints the new category and its rates; the exit code stays all-pass-or-nonzero with the new category in play.+- **Done when**: CLI tests green.+- **Depends on**: C3++### Task C5: invariants coverage+- **Lane**: C+- **Files**: `tests/core/brain/standing-instruction-facets.test.ts` (extension; fold into C2 if it stays small)+- **Test first**: one test asserting the whole lane performs zero writes against a vault that has standing rules, scoped rules and owner-scoped preferences, and that facet output never includes a preference the caller could not already see; resolved-count honesty (a population where nothing resolves reports zero resolved, not an implied provenance).+- **Done when**: green; the lane's read-only and visibility invariants are pinned by test.+- **Depends on**: C2++## Wave integration++- **Final docs pass**: `docs/updating.md` lines from all three lanes, `CHANGELOG.md` section completed by lane C (last to land), README untouched (no new verb or tool; all changes are options and accounting on existing surfaces).+- **Deliberately not planned**: recall-inject hook packing, the CLI packing flag, MCP `overlapped_ms`, extract-signals turn stamping, fake-http `rawBody`/stall mode - each recorded with its reason under design.md Out of scope.diff --git a/docs/brainstorm/recall-lane-latency/variants.md b/docs/brainstorm/recall-lane-latency/variants.mdnew file mode 100644index 00000000..f1b02698--- /dev/null+++ b/docs/brainstorm/recall-lane-latency/variants.md@@ -0,0 +1,48 @@+# Recall Lane Latency and Caller Control - brainstorm audit trail++Consultant: internal role-play per operator brief for this wave (no external CLI process spawned; the orchestrator generated the variants itself and records them verbatim below, which is the audit trail this project requires). Prompt that would have been sent: `cli-output/prompt.md`.++## Variants (consultant verbatim)++### Variant 1: Unified Recall Budget Owner+- **Approach**: Treat the wave as one budgeting subsystem change. Build a single recall-budget owner that both the context pack and the recall-inject hook consume, migrate both surfaces onto it in the same release, expose `packing_policy` on the pack tool and the hook config at once, and rebuild the test fixture as a new multi-endpoint fake (embeddings plus rerank plus chat) alongside the concurrency change.+- **Trade-offs**:+ - Pro: one budget model from day one; the packing policy is meaningful on every surface immediately.+ - Pro: the fixture grows once, in its own task, with all routes at the same time.+ - Con: the hook cannot fetch page bodies inside its 2,500 ms retrieval budget, so a unified owner either grows the hook's latency ceiling or silently narrows the policy on the hook path - exactly the silent fallback the standing brief forbids.+ - Con: four features now share one critical path; a regression in the budget owner blocks the whole wave, and the byte-identical guarantee must be argued across two surfaces instead of one.+ - Con: a chat-completions route has no consumer in this tree (no chat flow exists), so the fixture work would ship dead surface.+- **Complexity**: large+- **Risk**: high++### Variant 2: Substrate-First Opt-In Seams+- **Approach**: Land the shared test substrate first (add a `/rerank` route to the existing fake-http default handler and migrate the cross-encoder tests onto it), then ship each recall improvement behind its own existing seam: start-early/join-late lane overlap in `search()` with a lazy keyword-hit supplier and a per-call timing sink for `overlapped_ms` (never on `SearchOutcome`, which the query cache persists whole); a pure lane-allocation primitive in `recall-budget.ts` plus a strictly opt-in `packing_policy` on `brain_context_pack` only, where both candidate populations already meet under one token budget; a read-time provenance join that annotates standing-instruction facets on the attention-flow `standing_query` evaluation, gated by a new `standing_instruction` bench category. Every default path stays byte-identical; anything that cannot be pinned simple is deferred.+- **Trade-offs**:+ - Pro: each card lands on the seam it already owns, so lanes are file-disjoint and independently revertable.+ - Pro: the byte-identical contract is testable per seam with existing pin patterns (`resultShape` for search, the absent-option pins for the pack, the bench baseline for the fixture).+ - Pro: honors the validator corrections - correlate before reordering, join provenance instead of stamping it, extend the fixture instead of rebuilding it.+ - Con: the packing knob does not reach the recall-inject hook in this wave, so its upstream benefit (facts brief vs pages in one call) is only partially realized.+ - Con: `overlapped_ms` does not surface on merged multi-vault MCP payloads, only on `search --explain` and the bench.+- **Complexity**: medium+- **Risk**: low++### Variant 3: Concurrency-and-Facets Only+- **Approach**: Ship the fixture substrate, the lane overlap with its timing accounting, and the standing-instruction facets with the bench gate; drop the packing-policy card to a later wave entirely rather than risk a half-correlated budget.+- **Trade-offs**:+ - Pro: smallest blast radius; the wave is pure latency plus provenance with no budget semantics at all.+ - Pro: no risk of a knob that reads as available but is inert on the surfaces callers actually use.+ - Con: abandons a card whose corrected scope (correlate, then a strictly opt-in pack-only knob) is already small and fully expressible inside the pack's existing single budget.+ - Con: the recall lane gains an overlap metric but no caller control over what the budget buys, leaving the wave's "caller control" half undone.+- **Complexity**: small+- **Risk**: low++### Recommended: Variant 2+**Rationale**: It is the only variant that keeps all four cards' value while respecting the two hard project constraints: every omitted option must be byte-identical, and nothing may ship as an inert or silent fallback. Variant 1 fails the hook-latency reality the recon measured (a second retrieval pass cannot fit the hook's 2,500 ms budget) and couples four features into one critical path; Variant 3 leaves a small, well-scoped card undelivered for no technical reason. Variant 2 also matches the operator note exactly: concurrency, facets and the fixture land first, the packing knob stays strictly opt-in and scoped to the one surface where the two lanes already meet, with the hook side explicitly deferred rather than stubbed.++## Orchestrator decision++Variant 2 accepted. Project-context confirmation beyond the consultant's reasoning:++1. The operator note anticipated this exact decision point: "If the packing-knob budget correlation grows during design, land concurrency, facets and the fixture first and keep the knob strictly opt-in or defer that one card." The recon showed the correlation cannot span surfaces (the pack and the hook never run in one call), so the design scopes the knob to `brain_context_pack`, where preference-derived candidates and page candidates already share the single `maxTokens` budget, and defers the hook side instead of stubbing it.+2. The lane-assignment constraint reinforces Variant 2's seam choice: hosting the facet projection on the attention-flow evaluation (not a `ContextPackOptions` field) keeps lane B and lane C file-disjoint, which the four-lane plan requires.+3. Precedents pinned in the recon all point the same way: `queryMode` for opt-in vocabulary and MCP-first parity, `freshenSpawn` for a per-call sink on `SearchOptions`, `raceRerank`/`awaitWithinDeadline` for the join, `detectThreadFacets` for the pure-facet contract, and the bench knob pin for a measured gate.diff --git a/docs/mcp.md b/docs/mcp.mdindex 24e18360..cf063240 100644--- a/docs/mcp.md+++ b/docs/mcp.md@@ -2048,7 +2048,7 @@ format characters), when it contains NUL, or when it exceeds the cap.not the only record that it happened. Nothing in this tool removes a deadletter; `o2b state status` names the location and the records are listablefrom core. Single-artifact lanes are deliberately excluded: their one- failure IS the response. No new tool - the surface stays at 113.+ failure IS the response. No new tool - the surface is unchanged.- Since v1.53.0 `brain_context_pack` reads its `query` two ways. The default isthe substring filter it has always been - a case/Unicode-insensitive matchagainst `topic + principle`, every miss dropped with a `filter-miss` skip -@@ -2066,7 +2066,33 @@ format characters), when it contains NUL, or when it exceeds the cap.modes. The tool also stops discarding the report's `warnings`: injection-timetension warnings and the owner-scope observation now reach the caller as`brain_pre_compress_pack` has always passed them, absent when empty. No new- tool - the surface stays at 113.+ tool - the surface is unchanged.+- Since v1.84.0 `brain_context_pack` accepts `packing_policy`, a strictly+ opt-in claim order over the two candidate populations that already share+ the one `max_tokens` budget: the preference-derived pages (the facts+ lane, the distilled layer) and the retired pages (the pages lane).+ `facts_first` claims for the facts lane first; `retired_first` hands the+ first claim to the pages lane and lets the facts lane fill the exact+ remainder. Inside a lane the shipped comparator order holds: tier first,+ then recency, with query relevance joining the comparator only in the+ `ranked` and `semantic` modes - the pages lane is query-ordered only+ there, never in the default substring mode, where the query filtered+ rather than ranked. The total never grows, claiming is strictly sequential with+ no minimum-page floor, the per-memory character cap stays lane-internal,+ and the synthesized attention block stays a reserved charge neither lane+ draws. `packing_policy` requires `query` (the schema states the pairing+ as `dependentRequired` and the handler refuses a policy with nothing to+ order, exactly like a mode with nothing to read); a policy against a+ vault where one lane holds no candidate is refused by name with+ INVALID_PARAMS naming the lane and the member, never a silent no-op. The+ response gains a `packing` object - the policy name and each lane's+ allocation, delivered spend, candidate count and item count - only when+ the caller named a policy; an omitted `packing_policy` runs the unified+ comparator pack byte-identical, adds no member, and writes no receipts+ or telemetry keys. The receipt and telemetry budget metadata and the+ prompt-prefix metric segment carry the policy under the same rule, as a+ closed-vocabulary member of the frozen tuple. No new tool - the surface+ is unchanged.- Since v1.54.0 a page may RESERVE itself against remote reads, and thereservation is enforced rather than advisory. A vault page whose`visibility:` frontmatter carries the reserved token is not returned, not@@ -2130,7 +2156,7 @@ format characters), when it contains NUL, or when it exceeds the cap.and `write_id: null` arrives with an `audit_reason` naming what went wrong. Askipped create (`if_exists: "skip"`) authored no bytes, so it carries no`write_id` at all. `brain_writes` reads the record back. One new tool - the- surface moves from 113 to 114.+ surface grows by one.- With the optional decision-model turn pre-filter (`extract_prefilter`, see[`decision-models/extract-prefilter.md`](decision-models/extract-prefilter.md)),the `brain_extract_signals` plan may additively carry `turns_dropped` (turndiff --git a/docs/updating.md b/docs/updating.mdindex 1af0b6d0..c918802d 100644--- a/docs/updating.md+++ b/docs/updating.md@@ -13,48 +13,88 @@ change: the 18 hidden MCP alias tools left over from the token-dietconsolidation are removed. They were already absent from `tools/list`;since 1.0.0 calling one through `tools/call` answers an INVALID_PARAMStombstone naming the replacement instead of executing. Everything else+- CLI verbs, config keys, on-disk formats, the search schema ladder --is unchanged and now formally frozen under the stability policy-([`docs/stability.md`](stability.md)).+ is unchanged and now formally frozen under the stability policy+ ([`docs/stability.md`](stability.md)).Migrate by switching each removed name to its consolidated tool withthe matching `view` argument; every other parameter keeps its old nameand meaning:-| Removed tool | Replacement |-| --- | --- |-| `brain_digest` | `brain_brief` with `view: "digest"` |-| `brain_daily_brief` | `brain_brief` with `view: "daily"` |-| `brain_morning_brief` | `brain_brief` with `view: "morning"` |-| `brain_weekly_synthesis` | `brain_brief` with `view: "weekly"` |-| `brain_monthly_review` | `brain_brief` with `view: "monthly"` |-| `brain_operator_summary` | `brain_brief` with `view: "operator"` |-| `brain_timeline` | `brain_analytics` with `view: "timeline"` |-| `brain_attention_flows` | `brain_analytics` with `view: "attention_flows"` |-| `brain_belief_evolution` | `brain_analytics` with `view: "belief_evolution"` |+| Removed tool | Replacement |+| ------------------------- | -------------------------------------------------- |+| `brain_digest` | `brain_brief` with `view: "digest"` |+| `brain_daily_brief` | `brain_brief` with `view: "daily"` |+| `brain_morning_brief` | `brain_brief` with `view: "morning"` |+| `brain_weekly_synthesis` | `brain_brief` with `view: "weekly"` |+| `brain_monthly_review` | `brain_brief` with `view: "monthly"` |+| `brain_operator_summary` | `brain_brief` with `view: "operator"` |+| `brain_timeline` | `brain_analytics` with `view: "timeline"` |+| `brain_attention_flows` | `brain_analytics` with `view: "attention_flows"` |+| `brain_belief_evolution` | `brain_analytics` with `view: "belief_evolution"` || `brain_concept_synthesis` | `brain_analytics` with `view: "concept_synthesis"` |-| `get_active_schema_pack` | `schema_inspect` with `view: "active_pack"` |-| `list_schema_packs` | `schema_inspect` with `view: "packs"` |-| `schema_stats` | `schema_inspect` with `view: "stats"` |-| `schema_lint` | `schema_inspect` with `view: "lint"` |-| `schema_graph` | `schema_inspect` with `view: "graph"` |-| `schema_explain_type` | `schema_inspect` with `view: "explain_type"` |-| `schema_review_orphans` | `schema_inspect` with `view: "orphans"` |-| `reload_schema_pack` | `schema_inspect` with `view: "active_pack"` |+| `get_active_schema_pack` | `schema_inspect` with `view: "active_pack"` |+| `list_schema_packs` | `schema_inspect` with `view: "packs"` |+| `schema_stats` | `schema_inspect` with `view: "stats"` |+| `schema_lint` | `schema_inspect` with `view: "lint"` |+| `schema_graph` | `schema_inspect` with `view: "graph"` |+| `schema_explain_type` | `schema_inspect` with `view: "explain_type"` |+| `schema_review_orphans` | `schema_inspect` with `view: "orphans"` |+| `reload_schema_pack` | `schema_inspect` with `view: "active_pack"` |`o2b brain doctor` scans your vault-side surfaces (Brain notes, rootinstruction files such as `CLAUDE.md`/`AGENTS.md`, installed`.claude/skills/`) and warns with the exact replacement for any stalereference it finds (`removed-tool-reference`).+## Upgrading to 1.84.0++No step is required. Hybrid search now starts its semantic lane before the+keyword lanes run and joins it where the serial await stood, so the query+embed is hidden behind the keyword work; results, warnings and the+query-cache keys are byte-identical, and no configuration changes. The new+surfaces are additive options: `search --explain` reports `overlapped_ms`+(how much of the embed round trip the keyword lanes hid) only while the+flag is on, and the bench latency family reports the same figure only for+runs whose questions attempted the semantic lane. One absence is expected:+a warm query-cache hit serves the answer before the lanes start, so the+timing sink never fires and a cached answer's `--explain` output carries+neither `overlapped_ms` nor `semantic_attempted`; a fresh keyword-only run+still reports `overlapped_ms: 0` and the semantic attempt as false.++**`brain_context_pack` accepts `packing_policy`, strictly opt-in.** The new+MCP argument lets a caller name which candidate population claims the pack's+existing `max_tokens` budget first - the preference-derived pages+(`facts_first`) or the retired pages (`retired_first`) - and a policy against a+vault where one lane holds no candidate is refused by name. Omitted, it+changes no bytes of any response, receipt, telemetry record or prompt-prefix+metric; named, the response gains a `packing` accounting object and the+receipt and telemetry budget metadata gain a `packing_policy` member of the+frozen two-value vocabulary. There is no CLI flag and no config key - the+knob rides the MCP tool only, following the `query_mode` parity precedent.++**Standing-instruction facets are opt-in on the attention flow+evaluation, strictly read-only.** Callers that evaluate the+`standing_query` action keep the legacy `pref-<slug> (scope)` output+byte for byte unless they name `standingInstructionFacets`; named, a+surfaced preference whose `evidenced_by` evidence joins to a signal+`session_ref` gains a first-mention annotation, and preferences whose+join does not resolve stay unannotated rather than implying an origin.+No preference is admitted that the plain lane would not already deliver,+and the lane performs zero writes. The new `standing_instruction` bench+category exists to measure that join; runs without such questions+produce byte-identical reports and output.+## Upgrading to 1.83.0Nothing below needs a migration step. Every new surface in this release+- the per-source-session search pool cap, the usage-ordered skill-offers, the seeded LongMemEval sampling flags - ships default off and-is byte-identical when off, so an existing install behaves exactly as-before until an operator opts in. The one exception is written here-rather than left to be discovered at the terminal.+ offers, the seeded LongMemEval sampling flags - ships default off and+ is byte-identical when off, so an existing install behaves exactly as+ before until an operator opts in. The one exception is written here+ rather than left to be discovered at the terminal.**The first `o2b install --check --target opencode` after the hostwrites an `mcp.servers` object reports `drift`, and the first re-apply@@ -685,8 +725,8 @@ the LAN, a tailnet address) needs the per-endpoint opt-out in the operatorconfig or environment:```yaml-embedding_allow_insecure_http: true # OPEN_SECOND_BRAIN_EMBEDDING_ALLOW_INSECURE_HTTP-search_rerank_allow_insecure_http: true # OPEN_SECOND_BRAIN_SEARCH_RERANK_ALLOW_INSECURE_HTTP+embedding_allow_insecure_http: true # OPEN_SECOND_BRAIN_EMBEDDING_ALLOW_INSECURE_HTTP+search_rerank_allow_insecure_http: true # OPEN_SECOND_BRAIN_SEARCH_RERANK_ALLOW_INSECURE_HTTP```Both default to false, apply only to a base URL from the config or the@@ -875,12 +915,12 @@ values; a vault that contains none produces the same document.configured, the payload carried a single `mirror` string. It now carries`mirror` plus an optional `mirror_reason`, and the vocabulary changed:-| 1.48.0 | 1.49.0 |-| --- | --- |-| `mirror: "ok"` | unchanged |-| `mirror: "failed"` for an I/O failure | unchanged, plus `mirror_reason` |-| `mirror: "failed"` for a self-mirror | `mirror: "misconfigured"` plus `mirror_reason` |-| `mirror: "off"` | the key is absent entirely |+| 1.48.0 | 1.49.0 |+| ------------------------------------- | ---------------------------------------------- |+| `mirror: "ok"` | unchanged |+| `mirror: "failed"` for an I/O failure | unchanged, plus `mirror_reason` |+| `mirror: "failed"` for a self-mirror | `mirror: "misconfigured"` plus `mirror_reason` |+| `mirror: "off"` | the key is absent entirely |A self-mirror - `shared_namespace` resolving to the origin vault - isdecided before any I/O and is repaired by editing one config key, whilediff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex c0dff85e..0b50873f 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.83.0",+ "version": "1.84.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex 294c7e2e..4b07ab57 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.83.0",+ "version": "1.84.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 83ff955f..8901d446 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.83.0"+version: "1.84.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 8f0639b0..8d6a7a01 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.83.0",+ "version": "1.84.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/_schemas.py b/plugins/hermes/_schemas.pyindex ae3ca375..490efdc0 100644--- a/plugins/hermes/_schemas.py+++ b/plugins/hermes/_schemas.py@@ -529,133 +529,191 @@ STATIC_TOOL_SCHEMAS: tuple[dict[str, Any], ...] = ({'name': 'brain_feedback','unavailable (Cursor, Aider, raw Claude API). Read-only.','inputSchema': {'type': 'object', 'properties': {}, 'additionalProperties': False}},{'name': 'brain_context_pack',- 'description': 'Return the highest-tier, most recent vault slice that fits under `max_tokens`. '- 'Ordered core β supporting β peripheral, newest first; stops adding pages when '- 'the next page would exceed the budget. Read-only.',- 'inputSchema': {'type': 'object',- 'properties': {'max_tokens': {'type': 'integer',- 'minimum': 1,- 'description': 'Strict upper bound on the returned '- "slice's token count."},- 'query': {'type': 'string',- 'maxLength': 2000,- 'description': 'Optional query. Read as a '- 'case/Unicode-insensitive substring '- 'filter on topic + principle unless '- '`query_mode` says otherwise.'},- 'query_mode': {'type': 'string',- 'enum': ['substring', 'ranked', 'semantic'],- 'description': 'How `query` is read: `substring` '- '(default) drops misses; `ranked` '- 'orders by token overlap; '- '`semantic` orders by stored belief '- 'vectors, embedding the query '- 'once.'},- 'focus_session': {'type': 'string',- 'minLength': 1,- 'maxLength': 128,- 'description': 'Session id whose bound search '- 'focus boosts matching memories '- '(requires '- 'search_focus_context_pack).'},- 'max_chars_per_memory': {'type': 'integer',- 'minimum': 1,- 'description': 'Optional per-page '- 'character cap so one '- 'huge page cannot crowd '- 'out the rest; trimmed '- 'pages carry `trimmed: '- 'true`.'},- 'max_total_chars': {'type': 'integer',- 'minimum': 1,- 'description': 'Optional second ceiling (code '- 'points) on the cumulative '- 'size of the returned slice. '- 'Lowest-priority overflow is '- 'dropped with an '- '`over-char-budget` skip '- 'reason.'},- 'lanes': {'type': 'boolean',- 'description': 'When true, also return polarity-aware '- 'directives, constraints, and consider '- 'lanes. Legacy flat `items` remains '- 'present.'},- 'cache_stable': {'type': 'boolean',- 'description': 'When true, reorder the selected '- 'items by stable id and annotate '- 'their original rank.'},- 'dedup_repeated': {'type': 'boolean',- 'description': 'When true, replace repeated '- 'context bodies with reference '- 'hints to an earlier emitted '- 'item.'},- 'attention_flow_ids': {'type': 'array',- 'items': {'type': 'string'},- 'description': 'Optional declarative '- 'attention flow ids to '- 'inject as a synthetic '- 'context block.'},- 'receipt': {'type': 'boolean',- 'description': 'When true, emit an opt-in context '- 'receipt for this context-pack run.'},- 'receipt_host': {'type': 'string',- 'description': 'Optional host/runtime name for '- 'emitted receipts; defaults to '- '`mcp`.'},- 'recall_scores': {'type': 'array',- 'maxItems': 200,- 'items': {'type': 'number'},- 'description': 'Optional top-k recall scores; '- 'requires `match_quality`. '- 'Together they add an adequacy '- 'verdict: sufficient/proceed, '- 'weak/re_recall, '- 'insufficient/abstain.'},- 'match_quality': {'type': 'number',- 'minimum': 0,- 'maximum': 1,- 'description': 'Absolute match quality in '- "[0,1]: a search outcome's "- '`idf_weighted_coverage`. '- 'Required with `recall_scores`; '- 'the adequacy level reads this, '- 'never a score.'},- 'decision_answerable': {'type': 'number',- 'minimum': 0,- 'maximum': 1,- 'description': 'Advisory [0,1] '- 'probability from a '- "search's "- '`decision_model.answerable`. '- 'Needs `recall_scores` and '- '`match_quality`; never '- 'changes the level.'},- 'telemetry': {'type': 'boolean',- 'description': 'When true, emit an opt-in recall '- 'telemetry record for this '- 'context-pack run.'},- 'telemetry_host': {'type': 'string',- 'description': 'Optional host/runtime name for '- 'emitted telemetry; defaults to '- '`mcp`.'},- 'session_id': {'type': 'string',- 'description': 'Optional session id recorded on '- 'emitted telemetry.'},- 'turn_id': {'type': 'string',- 'description': 'Optional turn id recorded on emitted '- 'telemetry.'},- 'agent_scope': {'type': 'string',- 'description': 'Optional agent-ownership scope; '- 'shared (ownerless) memories '- 'always match, owner-tagged '- 'memories only their owner. Absent '- '= no ownership filtering.'}},- 'required': ['max_tokens'],- 'dependentRequired': {'recall_scores': ['match_quality'],- 'match_quality': ['recall_scores'],- 'decision_answerable': ['recall_scores', 'match_quality'],- 'query_mode': ['query']},- 'additionalProperties': False}},+ 'description': 'Return the highest-tier, most recent vault slice that fits under '+ '`max_tokens`. Ordered core β supporting β peripheral, newest first; '+ 'stops adding pages when the next page would exceed the budget. '+ 'Read-only.',+ 'inputSchema': {'type': 'object',+ 'properties': {'max_tokens': {'type': 'integer',+ 'minimum': 1,+ 'description': 'Strict upper bound on '+ "the returned slice's "+ 'token count.'},+ 'query': {'type': 'string',+ 'maxLength': 2000,+ 'description': 'Optional query. Read as a '+ 'case/Unicode-insensitive '+ 'substring filter on topic + '+ 'principle unless '+ '`query_mode` says '+ 'otherwise.'},+ 'query_mode': {'type': 'string',+ 'enum': ['substring',+ 'ranked',+ 'semantic'],+ 'description': 'How `query` is read: '+ '`substring` (default) '+ 'drops misses; `ranked` '+ 'orders by token '+ 'overlap; `semantic` '+ 'orders by stored belief '+ 'vectors, embedding the '+ 'query once.'},+ 'packing_policy': {'type': 'string',+ 'enum': ['facts_first',+ 'retired_first'],+ 'description': 'Budget claim order: '+ '`facts_first` or '+ '`retired_first`. '+ 'Neither is the '+ 'omitted default; a '+ 'vault where one '+ 'lane has no '+ 'candidate is '+ 'refused. Requires '+ '`query`.'},+ 'focus_session': {'type': 'string',+ 'minLength': 1,+ 'maxLength': 128,+ 'description': 'Session id whose '+ 'bound search focus '+ 'boosts matching '+ 'memories (requires '+ 'search_focus_context_pack).'},+ 'max_chars_per_memory': {'type': 'integer',+ 'minimum': 1,+ 'description': 'Optional '+ 'per-page '+ 'character cap '+ 'so one huge '+ 'page cannot '+ 'crowd out the '+ 'rest; trimmed '+ 'pages carry '+ '`trimmed: '+ 'true`.'},+ 'max_total_chars': {'type': 'integer',+ 'minimum': 1,+ 'description': 'Optional second '+ 'ceiling (code '+ 'points) on the '+ 'cumulative size of '+ 'the returned '+ 'slice. '+ 'Lowest-priority '+ 'overflow is '+ 'dropped with an '+ '`over-char-budget` '+ 'skip reason.'},+ 'lanes': {'type': 'boolean',+ 'description': 'When true, also return '+ 'polarity-aware directives, '+ 'constraints, and consider '+ 'lanes. Legacy flat `items` '+ 'remains present.'},+ 'cache_stable': {'type': 'boolean',+ 'description': 'When true, reorder '+ 'the selected items by '+ 'stable id and '+ 'annotate their '+ 'original rank.'},+ 'dedup_repeated': {'type': 'boolean',+ 'description': 'When true, replace '+ 'repeated context '+ 'bodies with '+ 'reference hints to '+ 'an earlier emitted '+ 'item.'},+ 'attention_flow_ids': {'type': 'array',+ 'items': {'type': 'string'},+ 'description': 'Optional '+ 'declarative '+ 'attention flow '+ 'ids to inject '+ 'as a synthetic '+ 'context block.'},+ 'receipt': {'type': 'boolean',+ 'description': 'When true, emit an opt-in '+ 'context receipt for this '+ 'context-pack run.'},+ 'receipt_host': {'type': 'string',+ 'description': 'Optional host/runtime '+ 'name for emitted '+ 'receipts; defaults to '+ '`mcp`.'},+ 'recall_scores': {'type': 'array',+ 'maxItems': 200,+ 'items': {'type': 'number'},+ 'description': 'Optional top-k '+ 'recall scores; '+ 'requires '+ '`match_quality`. '+ 'Together they add an '+ 'adequacy verdict: '+ 'sufficient/proceed, '+ 'weak/re_recall, '+ 'insufficient/abstain.'},+ 'match_quality': {'type': 'number',+ 'minimum': 0,+ 'maximum': 1,+ 'description': 'Absolute match '+ 'quality in [0,1]: a '+ "search outcome's "+ '`idf_weighted_coverage`. '+ 'Required with '+ '`recall_scores`; the '+ 'adequacy level reads '+ 'this, never a '+ 'score.'},+ 'decision_answerable': {'type': 'number',+ 'minimum': 0,+ 'maximum': 1,+ 'description': 'Advisory [0,1] '+ 'probability '+ 'from a '+ "search's "+ '`decision_model.answerable`. '+ 'Needs '+ '`recall_scores` '+ 'and '+ '`match_quality`; '+ 'never changes '+ 'the level.'},+ 'telemetry': {'type': 'boolean',+ 'description': 'When true, emit an '+ 'opt-in recall telemetry '+ 'record for this '+ 'context-pack run.'},+ 'telemetry_host': {'type': 'string',+ 'description': 'Optional '+ 'host/runtime name '+ 'for emitted '+ 'telemetry; defaults '+ 'to `mcp`.'},+ 'session_id': {'type': 'string',+ 'description': 'Optional session id '+ 'recorded on emitted '+ 'telemetry.'},+ 'turn_id': {'type': 'string',+ 'description': 'Optional turn id recorded '+ 'on emitted telemetry.'},+ 'agent_scope': {'type': 'string',+ 'description': 'Optional '+ 'agent-ownership scope; '+ 'shared (ownerless) '+ 'memories always match, '+ 'owner-tagged memories '+ 'only their owner. '+ 'Absent = no ownership '+ 'filtering.'}},+ 'required': ['max_tokens'],+ 'dependentRequired': {'recall_scores': ['match_quality'],+ 'match_quality': ['recall_scores'],+ 'decision_answerable': ['recall_scores',+ 'match_quality'],+ 'query_mode': ['query'],+ 'packing_policy': ['query']},+ 'additionalProperties': False}},{'name': 'brain_context_pack_outcome','description': 'Context-pack outcome loop. `post` records an outcome row for a carried sample id ''β first-pass/repair/retry counters plus three SEPARATE token signals (exact, 'diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 83ff955f..8901d446 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.83.0"+version: "1.84.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex 5984d1e9..6e2af161 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.83.0"+version = "1.84.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/brain/verbs/bench.ts b/src/cli/brain/verbs/bench.tsindex 38303756..526e266f 100644--- a/src/cli/brain/verbs/bench.ts+++ b/src/cli/brain/verbs/bench.ts@@ -20,6 +20,7 @@ import { loadReplayBenchFixture } from "../../../core/bench/replay.ts";import { runMemoryBench } from "../../../core/bench/phases.ts";import type {BenchPinnedKnobs,+ BenchReport,BenchSampleRequest,BenchSuiteLoad,BenchSuiteRegistration,@@ -35,9 +36,41 @@ const USAGE =const DEFAULT_RUNS_DIR = join(".open-second-brain", "bench-runs");const REPO_FIXTURE_DIR = join("tests", "fixtures", "bench");+/**+ * The overlapped-lane human line (recall-lane-latency), printed only when+ * the aggregate latency family carries the figure - a keyword-only run+ * keeps its stdout byte-identical.+ */+const OVERLAPPED_LINE_PREFIX = "overlapped_ms";++/**+ * The standing-instruction human line (recall-lane-latency), printed only+ * when the report's quality buckets carry the category - a run without+ * standing_instruction questions keeps its stdout byte-identical. The+ * headline passed/total/pass_rate excludes the category (buildReport), so+ * the category rates stay this named line of their own, never a fold into+ * the overall pass rate.+ */+const STANDING_INSTRUCTION_LINE_PREFIX = "standing_instruction";+/** A suite name must be safe to echo into errors, reports and run ids. */const SUITE_NAME_PATTERN = /^[a-z0-9][a-z0-9_-]*$/;+/**+ * The run's all-pass gate. The headline excludes the standing_instruction+ * category (its rate is the named line of its own, never a fold into the+ * overall pass rate), so the gate reads the category's own bucket beside+ * the headline: a failing facet question still exits nonzero, while a+ * passing one never shifts the headline rate a baseline compares.+ */+function runAllPass(report: BenchReport): boolean {+ const standing = report.quality.by_category["standing_instruction"];+ return (+ report.quality.passed === report.quality.total &&+ (standing === undefined || standing.passed === standing.total)+ );+}+/*** The path with no `--suite`: a native fixture loads directly. Not a* named suite - it is the byte-identical default the registry must@@ -128,6 +161,10 @@ const BENCH_PIN_KNOBS = ["scopeRules","semantic","rerank",+ // The standing-instruction facet lane's arm (recall-lane-latency):+ // listing it is what makes the lane-off arm reachable from the CLI+ // (`--pin '{"standingInstructionFacets":false}'`), not only the API.+ "standingInstructionFacets",] as const satisfies ReadonlyArray<keyof BenchPinnedKnobs>;/**@@ -422,13 +459,16 @@ export async function cmdBrainBench(argv: string[]): Promise<number> {if (flags["json"] === true) {process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);- return report.quality.passed === report.quality.total ? 0 : 1;+ return runAllPass(report) ? 0 : 1;}process.stdout.write(`bench ${report.fixture} run ${report.run_id}\n`);process.stdout.write(`quality: ${report.quality.passed}/${report.quality.total} (pass rate ${report.quality.pass_rate})\n`,);process.stdout.write(`latency_ms: avg ${report.latency_ms.avg} max ${report.latency_ms.max}\n`);+ if (report.latency_ms.overlapped_ms !== undefined) {+ process.stdout.write(`${OVERLAPPED_LINE_PREFIX}: avg ${report.latency_ms.overlapped_ms}\n`);+ }process.stdout.write(`context_cost: avg_chars ${report.context_cost.avg_chars} avg_injected_tokens ${report.context_cost.avg_injected_tokens}\n`,);@@ -452,13 +492,20 @@ export async function cmdBrainBench(argv: string[]): Promise<number> {"\n",);}+ if (report.quality.by_category["standing_instruction"] !== undefined) {+ const standing = report.quality.by_category["standing_instruction"];+ process.stdout.write(+ `${STANDING_INSTRUCTION_LINE_PREFIX}: ${standing.passed}/${standing.total} ` ++ `(pass rate ${round3(standing.passed / standing.total)})\n`,+ );+ }process.stdout.write(`judge: ${report.judge.status}\n`);for (const question of report.questions) {process.stdout.write(` ${question.pass ? "PASS" : "FAIL"} ${question.id} (${question.category})${question.failure ? ` - ${question.failure}` : ""}\n`,);}- return report.quality.passed === report.quality.total ? 0 : 1;+ return runAllPass(report) ? 0 : 1;}function resolveFixturePath(flag: string): string | null {@@ -467,3 +514,12 @@ function resolveFixturePath(flag: string): string | null {if (existsSync(repoLocal)) return repoLocal;return null;}++/**+ * Round to three decimals, the same figure the core bench reports carry+ * (`round3` in core/bench/phases.ts); the human standing_instruction line+ * prints the category rate at the same precision as the overall pass rate.+ */+function round3(value: number): number {+ return Math.round(value * 1000) / 1000;+}diff --git a/src/cli/search/outcome-render.ts b/src/cli/search/outcome-render.tsindex 21123ad7..b0d81995 100644--- a/src/cli/search/outcome-render.ts+++ b/src/cli/search/outcome-render.ts@@ -6,7 +6,7 @@import { decisionModelSearchEnvelope } from "../../core/decision-model/answerable.ts";import { serializeEvidencePack, serializeSearchCard } from "../../core/search/index.ts";-import type { SearchOutcome } from "../../core/search/index.ts";+import type { SearchLaneTiming, SearchOutcome } from "../../core/search/index.ts";import {EXPLAIN_OFF,explainEnvelope,@@ -37,6 +37,17 @@ const NO_RESULTS = `(${NO_RESULTS_LABEL})`;/** Longest snippet body shown per hit before it is elided. */const SNIPPET_MAX_CHARS = 140;+/**+ * The `--explain` overlapped-lane line (recall-lane-latency): how much of+ * the embed round trip the keyword lanes hid, printed only under the+ * flag, like the rest of the explain block.+ */+const OVERLAPPED_LINE_LABEL = "overlapped";++function overlappedLine(timing: SearchLaneTiming): string {+ return ` ${OVERLAPPED_LINE_LABEL}: ${timing.overlappedMs} ms`;+}+/*** Merged-away exact-duplicate locations for the `--json` payload, in the* snake_case grammar the rest of that payload uses.@@ -111,7 +122,11 @@ function joinLines(lines: ReadonlyArray<string>): string {return lines.join("\n") + (lines.length > 0 ? "" : "\n");}-export function jsonForOutcome(o: SearchOutcome, options: ExplainRequest = EXPLAIN_OFF): unknown {+export function jsonForOutcome(+ o: SearchOutcome,+ options: ExplainRequest = EXPLAIN_OFF,+ laneTiming?: SearchLaneTiming,+): unknown {return {results: o.results.map((r) => ({path: r.path,@@ -152,8 +167,9 @@ export function jsonForOutcome(o: SearchOutcome, options: ExplainRequest = EXPLA...retrievalTrailEnvelope(o),// Retrieval receipts (task F): the trace and the trust assessment the// gate already built, appended only when --explain asks. Absent, never- // null, otherwise.- ...explainEnvelope(o, options),+ // null, otherwise. The overlapped-lane timing rides the same flag-on+ // rule beside them.+ ...explainEnvelope(o, options, laneTiming),// Advisory decision-model answer (issue #213, Part 8); absent unless// the rerank request carried a valid `answerable` answer....decisionModelSearchEnvelope(o),@@ -195,17 +211,25 @@ function noResultsLine(o: SearchOutcome): string {* exclusion set folded into a reason histogram, so the transcript renders* the trace (the superset) and `--json` carries both receipts verbatim.*/-function explainLines(o: SearchOutcome, request: ExplainRequest): ReadonlyArray<string> {+function explainLines(+ o: SearchOutcome,+ request: ExplainRequest,+ laneTiming?: SearchLaneTiming,+): ReadonlyArray<string> {const trace = o.retrievalDecisionTrace;const assessment = o.memoryTrustAssessment;+ // The overlapped-lane line rides the explain block, after its lead+ // line, only when the caller captured a timing record.+ const overlap = laneTiming !== undefined ? [overlappedLine(laneTiming)] : [];// The trailing empty entry terminates the block with a newline, the way// a result entry does - `joinLines` adds no separator of its own.if (trace === undefined || assessment === undefined) {const why = retrievalTraceUnavailableReason(request.crossVault, request.trustGateEnabled);- return [`explain: ${why}`, ""];+ return [`explain: ${why}`, ...overlap, ""];}const lines = [`explain: evaluated ${trace.evaluated}${BULLET}surfaced ${trace.surfaced}${BULLET}excluded ${trace.excluded}`,+ ...overlap,];for (const exclusion of trace.exclusions) {lines.push(` excluded: ${exclusion.path}${BULLET}${exclusion.reasons.join(", ")}`);@@ -225,6 +249,7 @@ export function renderOutcomeHuman(verbose: boolean,query: string,options: ExplainRequest = EXPLAIN_OFF,+ laneTiming?: SearchLaneTiming,): string {// Progressive disclosure layer 1: render compact cards. The agent// drills a hit with `o2b search expand --chunk <chunk_id>`.@@ -242,7 +267,7 @@ export function renderOutcomeHuman(lines.push("");});for (const w of o.warnings) lines.push(`warning: ${w}`);- if (options.explain) lines.push(...explainLines(o, options));+ if (options.explain) lines.push(...explainLines(o, options, laneTiming));return joinLines(lines);}const lines: string[] = [];@@ -282,6 +307,6 @@ export function renderOutcomeHuman(lines.push("");});for (const w of o.warnings) lines.push(`warning: ${w}`);- if (options.explain) lines.push(...explainLines(o, options));+ if (options.explain) lines.push(...explainLines(o, options, laneTiming));return joinLines(lines);}diff --git a/src/cli/search/verbs/query.ts b/src/cli/search/verbs/query.tsindex 32de111e..1fae2fbd 100644--- a/src/cli/search/verbs/query.ts+++ b/src/cli/search/verbs/query.ts@@ -13,6 +13,7 @@ import {SEARCH_LIMIT_MAX,} from "../../../core/search/index.ts";import { searchAcrossVaults } from "../../../core/search/cross-vault.ts";+import type { SearchLaneTiming } from "../../../core/search/index.ts";import {parseDegreePredicate,type DegreePredicate,@@ -138,6 +139,13 @@ export async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<numbeconst until = flagString(flags, "until");const isGlobal = flagBoolean(flags, "global");+ // Overlapped-lane timing (recall-lane-latency): captured only when+ // --explain asks and only on the single-vault path - a cross-vault+ // union merges one search per origin and no honest single figure+ // exists for it. Omitted, the pipeline records nothing and every+ // projection stays byte-identical.+ const captureTiming = flagBoolean(flags, "explain") && !isGlobal;+ let laneTiming: SearchLaneTiming | undefined;const searchOpts = {query,limit: limitNum,@@ -169,6 +177,9 @@ export async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<numbe// surface opts in by default; --no-record-access suppresses it, and// cross-vault union never records (results span foreign vaults)....(!isGlobal && !flagBoolean(flags, "no-record-access") ? { recordAccess: true } : {}),+ ...(captureTiming+ ? { laneTiming: (timing: SearchLaneTiming) => void (laneTiming = timing) }+ : {}),};// Cross-vault union (t_72a22658): explicit per-call opt-in fans the// query out over profiles and read-only sources with origin labels.@@ -187,11 +198,13 @@ export async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<numbe};if (flagBoolean(flags, "json")) {- process.stdout.write(JSON.stringify(jsonForOutcome(outcome, explainOptions)) + "\n");+ process.stdout.write(+ JSON.stringify(jsonForOutcome(outcome, explainOptions, laneTiming)) + "\n",+ );return 0;}process.stdout.write(- renderOutcomeHuman(outcome, flagBoolean(flags, "verbose"), query, explainOptions),+ renderOutcomeHuman(outcome, flagBoolean(flags, "verbose"), query, explainOptions, laneTiming),);return 0;}diff --git a/src/core/bench/phases.ts b/src/core/bench/phases.tsindex 73c89a61..3034e6ba 100644--- a/src/core/bench/phases.ts+++ b/src/core/bench/phases.ts@@ -16,6 +16,7 @@import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";import { join } from "node:path";+import { evaluateAttentionFlow, listAttentionFlows } from "../brain/attention-flows.ts";import { packContext } from "../brain/context-pack.ts";import { TRANSPORT_REACH } from "../graph/transport-reach.ts";import { loadNormalizedContinuityRecords } from "../brain/continuity/read-model.ts";@@ -27,6 +28,7 @@ import {type RecallInjectDecision,} from "../brain/recall-inject.ts";import { resolveSearchConfig, search } from "../search/index.ts";+import type { SearchLaneTiming } from "../search/index.ts";import { evaluateSurfacingGate } from "../search/surfacing-gate.ts";import {BENCH_EMBEDDING_RECEIPT_FILENAME,@@ -77,6 +79,12 @@ import {* a smaller vault than the one under test.*/const BENCH_REACH = TRANSPORT_REACH.local;+/**+ * The standing-instruction facet category (recall-lane-latency). It is+ * the ONE category the headline quality family excludes: its rate stays+ * the named line of its own in `by_category`.+ */+const STANDING_INSTRUCTION_CATEGORY = "standing_instruction";export interface BenchRunOptions {readonly fixture: BenchFixture;@@ -134,6 +142,11 @@ export interface BenchRunOptions {interface RetrievedQuestion {readonly id: string;readonly latency_ms: number;+ /**+ * Overlapped lane milliseconds (recall-lane-latency), present only when+ * the semantic lane attempted the embed for this question.+ */+ readonly overlapped_ms?: number;/** Result paths in rank order (retrieval categories). */readonly paths?: ReadonlyArray<string>;/** Normalized turn texts (session_handoff). */@@ -152,6 +165,18 @@ interface RetrievedQuestion {readonly provenance_keys?: ReadonlyArray<string>;/** Whether a second identical extraction produced the same record ids. */readonly dedup_stable?: boolean;+ /**+ * The item strings the standing_query sections rendered, annotations+ * included (standing_instruction). The evaluator asserts the expected+ * earliest-cited coordinates against these.+ */+ readonly standing_items?: ReadonlyArray<string>;+ /**+ * Facet keys (`pref:<preference id>`) whose provenance join resolved+ * (standing_instruction). A preference the lane surfaced but could not+ * resolve is honestly absent - never an implied origin.+ */+ readonly facet_ids?: ReadonlyArray<string>;}/**@@ -310,7 +335,13 @@ export async function runMemoryBench(opts: BenchRunOptions): Promise<BenchReport// oxlint-disable-next-line no-await-in-loopfor (const question of opts.fixture.questions) {// oxlint-disable-next-line no-await-in-loop- const retrieved = await retrieveQuestion(vault, config, question, opts.recallFloor);+ const retrieved = await retrieveQuestion(+ vault,+ config,+ question,+ pinnedKnobs,+ opts.recallFloor,+ );writeFileSync(join(benchResultsDir(run.runDir), `${question.id}.json`),`${JSON.stringify(retrieved, null, 2)}\n`,@@ -400,18 +431,28 @@ async function retrieveQuestion(vault: string,config: ReturnType<typeof resolveSearchConfig>,question: BenchQuestion,+ pinnedKnobs: BenchPinnedKnobs,recallFloor?: number,): Promise<RetrievedQuestion> {const startedAt = Date.now();if (RETRIEVAL_CATEGORIES.has(question.category)) {+ // Overlapped lane accounting (recall-lane-latency): the per-call sink+ // rides this question's search and the figure is persisted only when+ // the semantic lane attempted the embed, so a keyword-only suite+ // keeps its report bytes.+ let laneTiming: SearchLaneTiming | undefined;const outcome = await search(config, {query: question.query ?? "",limit: question.top_k ?? 5,transportReach: BENCH_REACH,+ laneTiming: (timing) => {+ laneTiming = timing;+ },});return {id: question.id,latency_ms: Date.now() - startedAt,+ ...(laneTiming?.attempted === true ? { overlapped_ms: laneTiming.overlappedMs } : {}),paths: outcome.results.map((result) => result.path),};}@@ -477,6 +518,9 @@ async function retrieveQuestion(injected_tokens: injectedTokens(brief),};}+ if (question.category === STANDING_INSTRUCTION_CATEGORY) {+ return retrieveStandingInstruction(vault, question, startedAt, pinnedKnobs);+ }if (question.category === "write_fidelity") {return retrieveWriteFidelity(vault, question, startedAt);}@@ -555,6 +599,51 @@ function provenanceKeys(record: ContinuityRecord): ReadonlyArray<string> {return [...Object.keys(record), ...Object.keys(record.payload)];}+/**+ * Run the shipped standing_query surfacing lane over the fixture's+ * attention flows (standing_instruction, recall-lane-latency).+ *+ * The lane's arm follows the run's knob pin: absent or true, the facet+ * option is on - the category exists to measure the provenance join.+ * Pinned false, the plain lane runs and emits the legacy strings, so the+ * category's facet assertions fail by name: the off arm of the A/B the+ * design promises, which is what makes the pin change what a run+ * measures instead of only naming it. Every flow that declares the+ * `standing_query` action is evaluated and its surfaced items and+ * resolved facet keys persisted; a preference the lane surfaces but+ * whose join does not resolve contributes its plain string and no facet,+ * exactly as the option contract renders it.+ */+function retrieveStandingInstruction(+ vault: string,+ question: BenchQuestion,+ startedAt: number,+ pinnedKnobs: BenchPinnedKnobs,+): RetrievedQuestion {+ // Absent keeps the lane on, so existing runs and the default pin+ // measure the facet lane exactly as before the pin was wired.+ const facetsOn = pinnedKnobs.standingInstructionFacets !== false;+ const items: string[] = [];+ const facetIds: string[] = [];+ for (const flow of listAttentionFlows(vault)) {+ if (!flow.actions.includes("standing_query")) continue;+ const evaluation = facetsOn+ ? evaluateAttentionFlow(vault, flow.id, { standingInstructionFacets: true })+ : evaluateAttentionFlow(vault, flow.id, { standingInstructionFacets: false });+ for (const section of evaluation.sections) {+ if (section.action !== "standing_query") continue;+ items.push(...section.items);+ for (const facet of section.facets ?? []) facetIds.push(facet.key);+ }+ }+ return {+ id: question.id,+ latency_ms: Date.now() - startedAt,+ standing_items: items,+ facet_ids: facetIds,+ };+}+/*** Fixed authoring instant for write-fidelity extraction. Extraction* defaults `createdAt` to wall clock, which would make the record ids -@@ -578,6 +667,7 @@ function evaluateQuestion(id: question.id,category: question.category,latency_ms: retrieved.latency_ms,+ ...(retrieved.overlapped_ms !== undefined ? { overlapped_ms: retrieved.overlapped_ms } : {}),...(retrieved.context_chars !== undefined ? { context_chars: retrieved.context_chars } : {}),...(retrieved.injected_tokens !== undefined? { injected_tokens: retrieved.injected_tokens }@@ -638,6 +728,9 @@ function evaluateQuestion(}return Object.freeze({ ...base, pass: true });}+ if (question.category === STANDING_INSTRUCTION_CATEGORY) {+ return evaluateStandingInstruction(question, retrieved, base);+ }// budgetconst ids = retrieved.item_ids ?? [];const missingIds = (question.expected_ids ?? []).filter((id) => !ids.includes(id));@@ -724,6 +817,64 @@ function evaluateWriteFidelity(return Object.freeze({ ...base, pass: true });}+/**+ * Assert the standing-instruction facet lane delivered what the fixture+ * named (standing_instruction, recall-lane-latency).+ *+ * `expected_ids` are facet keys - `pref:<preference id>`, so the+ * preference id itself is what is asserted delivered. `expected_text`+ * is the full earliest-cited annotation that must CLOSE the first named+ * preference's item: the assertion finds the item for `expected_ids[0]`+ * and compares the annotation exactly, closing bracket included, so a+ * near-miss origin (`...#t1`) cannot satisfy an expectation about+ * another (`...#t10`) and an origin cannot pass by sitting on a+ * different preference's item. A fixture states the kind word ("earliest+ * cited turn ..."/"earliest cited session ...") so a session-annotated+ * origin cannot satisfy a turn expectation. A preference the lane+ * surfaced but could not resolve is a named failure, never an implied+ * provenance.+ */+function evaluateStandingInstruction(+ question: BenchQuestion,+ retrieved: RetrievedQuestion,+ base: ResultBase,+): BenchQuestionResult {+ const facetIds = retrieved.facet_ids ?? [];+ const missingFacets = (question.expected_ids ?? []).filter((id) => !facetIds.includes(id));+ if (missingFacets.length > 0) {+ return Object.freeze({+ ...base,+ pass: false,+ failure: `expected facet missing from the surfaced standing instructions: ${missingFacets.join(", ")}`,+ });+ }+ const expectedOrigin = question.expected_text;+ if (expectedOrigin !== undefined) {+ // Tie the annotation to the expected preference's own item: the+ // first expected facet key names the preference, and its item is the+ // `id (scope)` string the lane renders for it.+ const expectedKey = question.expected_ids?.[0];+ const expectedId =+ expectedKey !== undefined && expectedKey.startsWith("pref:")+ ? expectedKey.slice("pref:".length)+ : undefined;+ const item =+ expectedId === undefined+ ? undefined+ : (retrieved.standing_items ?? []).find((candidate) =>+ candidate.startsWith(`${expectedId} (`),+ );+ if (item === undefined || !item.endsWith(expectedOrigin)) {+ return Object.freeze({+ ...base,+ pass: false,+ failure: `expected origin annotation not found on the item for ${expectedKey ?? "(no facet named)"}: ${expectedOrigin}`,+ });+ }+ }+ return Object.freeze({ ...base, pass: true });+}+function evaluateSourceIsolation(question: BenchQuestion,retrieved: RetrievedQuestion,@@ -764,15 +915,29 @@ function buildReport(): BenchReport {const byCategory: Record<string, { passed: number; total: number }> = {};let passed = 0;+ let total = 0;for (const result of results) {const bucket = (byCategory[result.category] ??= { passed: 0, total: 0 });bucket.total += 1;if (result.pass) {bucket.passed += 1;- passed += 1;+ }+ // The standing_instruction category never folds into the headline+ // (recall-lane-latency): its rate stays the named line of its own in+ // `by_category`, so adding facet questions to a fixture cannot shift+ // the headline pass rate a baseline compares.+ if (result.category !== STANDING_INSTRUCTION_CATEGORY) {+ total += 1;+ if (result.pass) passed += 1;}}const latencies = results.map((result) => result.latency_ms);+ // The latency family's overlap figure (recall-lane-latency): averaged+ // over the questions that attempted the semantic lane, present only+ // when at least one did - a keyword-only report keeps its bytes.+ const overlappedMsValues = results+ .map((result) => result.overlapped_ms)+ .filter((value): value is number => value !== undefined);const avgChars = mean(results.map((result) => result.context_chars));// The suite's ONE token number, from the ONE estimator. The inline// `ceil(avg_chars / 4)` that stood here was a sixth formula in a repo@@ -796,8 +961,8 @@ function buildReport(created_at: run.checkpoint.created_at,quality: Object.freeze({passed,- total: results.length,- pass_rate: results.length > 0 ? round3(passed / results.length) : 0,+ total,+ pass_rate: total > 0 ? round3(passed / total) : 0,by_category: Object.freeze(Object.fromEntries(Object.entries(byCategory).toSorted(([a], [b]) => (a < b ? -1 : 1))),),@@ -808,6 +973,13 @@ function buildReport(? Math.round(latencies.reduce((sum, value) => sum + value, 0) / latencies.length): 0,max: latencies.length > 0 ? Math.max(...latencies) : 0,+ ...(overlappedMsValues.length > 0+ ? {+ overlapped_ms: Math.round(+ overlappedMsValues.reduce((sum, value) => sum + value, 0) / overlappedMsValues.length,+ ),+ }+ : {}),}),context_cost: Object.freeze({avg_chars: avgChars,diff --git a/src/core/bench/types.ts b/src/core/bench/types.tsindex 06c99f0e..35ec1bfc 100644--- a/src/core/bench/types.ts+++ b/src/core/bench/types.ts@@ -45,6 +45,11 @@ export const BENCH_CATEGORIES = ["proactive_recall","write_fidelity","source_isolation",+ // The standing-instruction facet suite (recall-lane-latency). Like the+ // failure-mode categories it drives a seam the search pipeline never+ // touches: the `standing_query` attention action, run with the opt-in+ // provenance facet on, so the join ships measured rather than assumed.+ "standing_instruction",] as const;export type BenchCategory = (typeof BENCH_CATEGORIES)[number];@@ -105,6 +110,14 @@ export interface BenchQuestion {/** session_handoff: which session's turns must be readable. */readonly session_id?: string;readonly expected_turns?: number;+ /**+ * standing_instruction: the full earliest-cited annotation that must+ * close the item of `expected_ids[0]`, compared exactly with the+ * closing bracket included - a near-miss origin cannot satisfy the+ * expectation as a substring. (session_handoff reads the same field+ * as an in-turns substring, so the exact-match rule is+ * category-specific.)+ */readonly expected_text?: string;/*** Official-prompt judge glue (longmemeval), absent for suites that@@ -250,6 +263,20 @@ export type BenchPinnedKnobs = Partial<readonly scopeRules?: VaultScopeRules;readonly semantic?: Partial<ResolvedEmbeddingConfig>;readonly rerank?: Partial<ResolvedRerankConfig>;+ /**+ * The standing-instruction facet lane's arm (recall-lane-latency), read+ * by the retrieve branch: absent or true, the `standing_instruction`+ * category evaluates with the provenance facet on; false runs the plain+ * lane, which emits the legacy strings so the category's facet+ * assertions fail by name - the off arm of the measured A/B, and the+ * reason the pin changes what a run measures instead of only naming it.+ * The value folds into the checkpoint hash either way, so a resumed run+ * that would mix a facet-gate arm with a plain one fails the guard+ * loudly instead of comparing them. Absent on every other run, so+ * `DEFAULT_BENCH_KNOB_PIN` and the existing run identities serialize+ * byte-identically.+ */+ readonly standingInstructionFacets?: boolean;};/**@@ -349,6 +376,13 @@ export interface BenchQuestionResult {/** One-line reason when pass is false. */readonly failure?: string;readonly latency_ms: number;+ /**+ * Overlapped lane milliseconds (recall-lane-latency): how much of this+ * question's embed round trip the keyword lanes hid, present only when+ * the semantic lane attempted the embed - a keyword-only run adds no+ * key, keeping its report bytes identical.+ */+ readonly overlapped_ms?: number;/** Characters of context this question injected, when it injected any. */readonly context_chars?: number;/** {@link estimateTokens} over the same string `context_chars` measured. */@@ -372,12 +406,32 @@ export interface BenchReport {readonly fixture_hash: string;readonly created_at: string;readonly quality: {+ /**+ * Headline pass count over every category EXCEPT+ * `standing_instruction` (recall-lane-latency): the facet lane's+ * rate stays the named line of its own in `by_category`, so adding+ * facet questions to a fixture never shifts the headline a baseline+ * compares.+ */readonly passed: number;+ /** Question count behind {@link BenchReport.quality.passed}, same exclusion. */readonly total: number;+ /** `passed / total` at report precision; `0` when nothing counts. */readonly pass_rate: number;+ /** Per-category passed/total, every category including the excluded one. */readonly by_category: Readonly<Record<string, { passed: number; total: number }>>;};- readonly latency_ms: { readonly avg: number; readonly max: number };+ readonly latency_ms: {+ readonly avg: number;+ readonly max: number;+ /**+ * Mean overlapped lane milliseconds (recall-lane-latency) over the+ * questions that attempted the semantic lane. Present only when at+ * least one did, so the default (keyword-only) report serializes+ * byte-identically without it.+ */+ readonly overlapped_ms?: number;+ };/*** Cost, kept apart from quality on purpose (the MemScore lesson stated* at the top of this file). `avg_injected_tokens` is the fourthdiff --git a/src/core/brain/attention-flows.ts b/src/core/brain/attention-flows.tsindex 08b271a9..3878bec0 100644--- a/src/core/brain/attention-flows.ts+++ b/src/core/brain/attention-flows.ts@@ -1,4 +1,4 @@-import { existsSync, mkdirSync, readdirSync, readFileSync } from "node:fs";+import { existsSync, mkdirSync, readdirSync } from "node:fs";import { join } from "node:path";import { parseFrontmatter, writeFrontmatterAtomic } from "../vault.ts";@@ -8,6 +8,12 @@ import { listProceduralMemory } from "./procedural-memory.ts";import { listPendingSkillProposals } from "./skill-proposals.ts";import { listRecurrenceEntries } from "./recurrence.ts";import { parsePreference } from "./preference.ts";+import { parseSignal } from "./signal.ts";+import {+ resolvePreferenceProvenance,+ type SessionRefOrigin,+ type SignalSessionRefLookup,+} from "./preference-provenance.ts";import { BRAIN_PREFERENCE_STATUS } from "./types.ts";export type AttentionFlowAction =@@ -33,6 +39,53 @@ export interface AttentionFlowRecipe {export interface AttentionFlowSection {readonly action: AttentionFlowAction;readonly items: ReadonlyArray<string>;+ /**+ * Structured standing-instruction facets, present ONLY while the+ * evaluation named `standingInstructionFacets` - omitted otherwise,+ * so an unset option leaves the section shape byte-identical to the+ * legacy one. A preference whose provenance join resolves gains an+ * entry; one whose join does not resolve stays absent, an honest+ * zero rather than an implied provenance.+ */+ readonly facets?: ReadonlyArray<StandingInstructionFacet>;+}++/**+ * One confirmed preference surfaced by the `standing_query` action,+ * with the origin coordinates its provenance join resolved.+ *+ * Contract-style cross-reference: this is the preference-corpus sibling+ * of `ThreadFacet` in `thread-topics.ts` - a stable key, ordered+ * origin coordinates, pure derivation - for a different corpus. A+ * thread facet selects the turns of one thread for the coverage pass;+ * this facet annotates a standing instruction with the origin its+ * evidence cites first. The distinct name is deliberate, so the coverage lane's+ * meaning of "facet" stays unambiguous.+ */+export interface StandingInstructionFacet {+ /** Stable facet key: `pref:<preference id>`. */+ readonly key: string;+ readonly preferenceId: string;+ readonly scope: string;+ /**+ * Resolved origins in the preference's `evidenced_by` order - never+ * re-sorted, and not a chronological claim; `[0]` is the earliest+ * cited origin, not the earliest signal in time.+ */+ readonly origins: ReadonlyArray<SessionRefOrigin>;+}++export interface AttentionFlowEvaluationOptions {+ /**+ * Opt-in standing-instruction provenance (recall lane). Omitted or+ * false, every emitted string is the legacy `id (scope)` output and+ * sections carry no `facets` field. True, a surfaced preference whose+ * provenance join resolves gains an earliest-cited annotation on its+ * item string plus a facet entry; the option changes annotations,+ * never membership - no preference is admitted that the plain lane+ * would not already deliver.+ */+ readonly standingInstructionFacets?: boolean;}export interface AttentionFlowEvaluation {@@ -102,7 +155,11 @@ export function listAttentionFlows(vault: string): ReadonlyArray<AttentionFlowRereturn Object.freeze(out);}-export function evaluateAttentionFlow(vault: string, flowId: string): AttentionFlowEvaluation {+export function evaluateAttentionFlow(+ vault: string,+ flowId: string,+ options: AttentionFlowEvaluationOptions = {},+): AttentionFlowEvaluation {const flow = listAttentionFlows(vault).find((item) => item.id === flowId);if (!flow) throw new Error(`attention flow not found: ${flowId}`);@@ -130,8 +187,21 @@ export function evaluateAttentionFlow(vault: string, flowId: string): AttentionFcontinue;}if (action === "standing_query") {- const items = listConfirmedPreferencesByScope(vault, flow.standingQueryScopes);- sections.push({ action, items: Object.freeze(items) });+ const facetsWanted = options.standingInstructionFacets === true;+ const surfaced = listConfirmedPreferencesByScope(+ vault,+ flow.standingQueryScopes,+ facetsWanted,+ );+ sections.push({+ action,+ items: Object.freeze(surfaced.items),+ // Key present only under the opt-in: an unset option leaves the+ // section byte-identical to the legacy shape.+ ...(options.standingInstructionFacets === true+ ? { facets: Object.freeze(surfaced.facets) }+ : {}),+ });}}@@ -188,20 +258,68 @@ function normalizeStandingQueryScopes(value: unknown): ReadonlyArray<string> {return Object.freeze([...new Set(out)]);}+/** Item annotation lead for an origin resolved to a real turn. */+const EARLIEST_CITED_TURN_PREFIX = " [earliest cited turn ";+/** Item annotation lead for an origin resolved to a capture-lifecycle coordinate: the fragment is the hook event's NAME, so the annotation claims the session, never a turn. */+const EARLIEST_CITED_SESSION_PREFIX = " [earliest cited session ";+/** Item annotation suffix. */+const EARLIEST_CITED_SUFFIX = "]";++/**+ * Read-only signal-id β `session_ref` lookup over the directories the+ * signal readers walk. The on-disk signal id always equals the file's+ * basename stem (see `writeSignal`), so a lookup is a direct probe per+ * directory; an absent, unreadable, or path-shaped id contributes no+ * provenance rather than an error.+ */+function signalSessionRefLookup(vault: string): SignalSessionRefLookup {+ const dirs = brainDirs(vault);+ const searchDirs = [dirs.inbox, dirs.processed, dirs.archived];+ return (signalId) => {+ // Filename-stem ids only: a lookup key is never a path.+ if (!/^[A-Za-z0-9][A-Za-z0-9._-]*$/.test(signalId)) return undefined;+ for (const dir of searchDirs) {+ const path = join(dir, `${signalId}.md`);+ if (!existsSync(path)) continue;+ try {+ return parseSignal(path).session_ref;+ } catch {+ // A signal that cannot be parsed proves nothing about where a+ // preference came from - no provenance, never a guess.+ return undefined;+ }+ }+ return undefined;+ };+}++/** The item string plus the facet for one surfaced preference, per the option state. */+interface SurfacedStandingInstruction {+ readonly items: string[];+ readonly facets: StandingInstructionFacet[];+}+/*** Confirmed preferences whose `scope` is one the flow declared. Structural* match on the frontmatter scope token; no natural-language matching. Returns- * `id (scope)` items, sorted for deterministic output.+ * `id (scope)` items, sorted for deterministic output. With the facet option+ * on, a preference whose provenance join resolves additionally gains an+ * earliest-cited annotation on its item string and a facet entry - annotations+ * only, never a membership change, so a facet cannot surface a preference the+ * plain lane would withhold.*/function listConfirmedPreferencesByScope(vault: string,scopes: ReadonlyArray<string>,-): ReadonlyArray<string> {- if (scopes.length === 0) return Object.freeze([]);+ facetsWanted: boolean,+): SurfacedStandingInstruction {+ const items: string[] = [];+ const facets: StandingInstructionFacet[] = [];+ if (scopes.length === 0) return { items, facets };const wanted = new Set(scopes);const dir = brainDirs(vault).preferences;- if (!existsSync(dir)) return Object.freeze([]);- const out: string[] = [];+ if (!existsSync(dir)) return { items, facets };+ const lookup = facetsWanted ? signalSessionRefLookup(vault) : null;for (const name of readdirSync(dir).toSorted()) {if (!name.endsWith(".md")) continue;try {@@ -211,13 +329,40 @@ function listConfirmedPreferencesByScope(pref.scope !== undefined &&wanted.has(pref.scope)) {- out.push(`${pref.id} (${pref.scope})`);+ const base = `${pref.id} (${pref.scope})`;+ // Off means the legacy bytes: no lookup runs, no annotation is+ // composed, the section gains no facet field.+ if (lookup === null) {+ items.push(base);+ continue;+ }+ const provenance = resolvePreferenceProvenance(+ { id: pref.id, evidenced_by: pref.evidenced_by },+ lookup,+ );+ if (provenance === null) {+ // Nothing resolved: the honest plain string, no implied origin.+ items.push(base);+ continue;+ }+ const earliest = provenance.origins[0]!;+ const prefix =+ earliest.kind === "turn" ? EARLIEST_CITED_TURN_PREFIX : EARLIEST_CITED_SESSION_PREFIX;+ items.push(+ `${base}${prefix}${earliest.sessionKey}#${earliest.fragment}${EARLIEST_CITED_SUFFIX}`,+ );+ facets.push({+ key: provenance.key,+ preferenceId: pref.id,+ scope: pref.scope,+ origins: provenance.origins,+ });}} catch {continue;}}- return Object.freeze(out);+ return { items, facets };}export function buildAttentionContextBlock(diff --git a/src/core/brain/belief-semantic.ts b/src/core/brain/belief-semantic.tsindex 701cf0e6..0c21706f 100644--- a/src/core/brain/belief-semantic.ts+++ b/src/core/brain/belief-semantic.ts@@ -148,11 +148,20 @@ export interface BeliefSemanticDeps {* refusal decided over the beliefs it keeps), so both read the same.*/export function discloseSpentQuery(e: SearchError, spent: BeliefSemanticQueryReport): SearchError {- return new SearchError(- e.code,- `${e.message} (the query embed was still spent: model ${spent.model}, ` +- `price source ${spent.priceSource}, ${spent.queryTokens} query token(s), ` +- `${formatEstimatedUsd(spent.estimatedUsd)})`,+ return new SearchError(e.code, `${e.message} ${spentQueryDisclosure(spent)}`);+}++/**+ * The one spend-disclosure sentence, shared by every refusal that lands+ * after the query embed was paid: the pack's lane-empty INVALID_PARAMS+ * reads the same wording the loader's SearchError refusals carry, so a+ * caller cannot learn the spend from one surface and not the other.+ */+export function spentQueryDisclosure(spent: BeliefSemanticQueryReport): string {+ return (+ `(the query embed was still spent: model ${spent.model}, ` ++ `price source ${spent.priceSource}, ${spent.queryTokens} query token(s), ` ++ `${formatEstimatedUsd(spent.estimatedUsd)})`);}diff --git a/src/core/brain/context-pack.ts b/src/core/brain/context-pack.tsindex ee7bf930..dfcccfec 100644--- a/src/core/brain/context-pack.ts+++ b/src/core/brain/context-pack.ts@@ -48,7 +48,14 @@ import { estimateTokens } from "./text/tokenizer.ts";import { densityScore } from "./context-density.ts";import { normalizeForDedup } from "./text/normalize.ts";import { jaccard, tokenise } from "./similarity.ts";-import { applyCharBudget, type CharBudgetDegradationMode } from "./recall-budget.ts";+import {+ allocateRecallBudget,+ applyCharBudget,+ type CharBudgetDegradationMode,+ type PackingPolicy,+ type RecallBudgetAllocation,+ type RecallBudgetLaneName,+} from "./recall-budget.ts";import { emitContextReceipt, type ContextReceiptOptions } from "./context-receipts.ts";import type { RecallAdequacyVerdict } from "./recall-adequacy.ts";import { emitGatedTelemetry } from "./continuity/emit.ts";@@ -141,6 +148,54 @@ export class ContextPackSemanticRelevanceMissingError extends Error {}}+/**+ * A named packing policy was asked for against a pack where it cannot+ * run: one of its two lanes has no candidate in reach, the budget is+ * zero and neither lane can ever draw, or the reserved attention charge+ * consumes the whole budget and neither lane can draw. Packing by the+ * named order would silently do nothing, so the call fails by name - the+ * policy member and every empty lane are in the message - and fails+ * before the receipt and telemetry are written, so a refused pack leaves+ * no record of a delivery.+ */+export class ContextPackPackingLaneEmptyError extends Error {+ private constructor(message: string) {+ super(message);+ this.name = "ContextPackPackingLaneEmptyError";+ }++ /** The partition's refusal: the named policy cannot see both lanes. */+ static emptyLanes(policy: PackingPolicy, emptyLanes: ReadonlyArray<RecallBudgetLaneName>) {+ const named =+ emptyLanes.length === 1+ ? `the ${emptyLanes[0]} lane is empty`+ : `the ${emptyLanes[0]} and ${emptyLanes[1]} lanes are empty`;+ return new ContextPackPackingLaneEmptyError(+ `context pack: packing_policy ${policy} needs both lanes; ${named}`,+ );+ }++ /** The short-circuit's refusal: a zero budget offers neither lane anything. */+ static zeroBudget(policy: PackingPolicy) {+ return new ContextPackPackingLaneEmptyError(+ `context pack: packing_policy ${policy} needs a positive token budget; ` ++ `neither lane can draw on zero tokens`,+ );+ }++ /**+ * The allocation's refusal: the reserved attention charge consumes the+ * whole budget, so both populated lanes draw on nothing.+ */+ static reservedCharge(policy: PackingPolicy, reservedTokens: number, maxTokens: number) {+ return new ContextPackPackingLaneEmptyError(+ `context pack: packing_policy ${policy} needs a lane budget beyond the reserved ` ++ `attention charge; the reserved charge of ${reservedTokens} tokens consumes the ` ++ `whole ${maxTokens}-token budget and neither lane can draw`,+ );+ }+}+export interface ContextPackItem extends ContextTransformAnnotations {readonly id: string;readonly path: string;@@ -195,6 +250,45 @@ export interface ContextPackSkipped {readonly reason: "over-budget" | "filter-miss" | "guard-blocked" | "over-char-budget";}+/**+ * What one packing lane held under the correlated budget. `spent` sums+ * the tokens of the items DELIVERED from the lane, so the two spends+ * and the reserved charge always add up to `tokensUsed` plus what the+ * char budget dropped.+ */+export interface ContextPackPackingLaneReport {+ /** The share of the shared total the lane held (first claim or remainder). */+ readonly allocation: number;+ /** Tokens actually delivered from this lane. */+ readonly spent: number;+ /** Candidates the lane brought into the budget, after the query filter. */+ readonly candidates: number;+ /** Items delivered from this lane. */+ readonly items: number;+}++/**+ * Accounting for the opt-in packing policy: which member was named, the+ * reserved charge the lane allocations were reduced by, and what each lane+ * held under the one shared budget. Present ONLY when the caller named+ * {@link ContextPackOptions.packingPolicy} - the same conditional shape as+ * `stamp`, `lanes` and `semantic`.+ */+export interface ContextPackPackingReport {+ readonly policy: PackingPolicy;+ /**+ * The reserved attention charge (the synthesized attention block) that+ * was paid before either lane claimed, so both lane allocations were+ * reduced by it. Present even when zero, so the accounting the caller+ * reads is whole: the two allocations plus this charge always sum to+ * the pack's `maxTokens`. A refused pack has no report at all - the+ * reserved-charge refusal throws before anything is returned.+ */+ readonly reservedTokens: number;+ readonly pages: ContextPackPackingLaneReport;+ readonly facts: ContextPackPackingLaneReport;+}+export interface ContextPackReport {readonly maxTokens: number;readonly tokensUsed: number;@@ -221,6 +315,11 @@ export interface ContextPackReport {readonly stamp?: ContextPackStamp;/** Present only in `semantic` mode; see {@link ContextPackSemanticReport}. */readonly semantic?: ContextPackSemanticReport;+ /**+ * Present only when the caller named a packing policy; see+ * {@link ContextPackPackingReport}.+ */+ readonly packing?: ContextPackPackingReport;}/**@@ -264,6 +363,24 @@ export interface ContextPackOptions {* read in no other.*/readonly semanticRelevance?: ReadonlyMap<string, number>;+ /**+ * Opt-in packing policy (recall-lane-latency wave): which lane claims+ * the pack's one existing token budget first when both candidate+ * populations are in reach - the preference-derived candidates (the+ * facts lane, the distilled layer) and the page candidates from the+ * retired directory (the pages lane). `facts_first` claims facts-side+ * first, `retired_first` hands the first claim to the retired pages+ * lane; the losing lane fills the exact remainder and the total never+ * grows. Omitted runs the unified comparator pack byte-identical to every+ * release before this one - no partition, no `packing` accounting.+ * With the option set, a lane with no candidate in reach, a zero+ * budget, or a reserved attention charge that consumes the whole+ * total - each of which offers neither lane anything to draw - is+ * refused by name ({@link ContextPackPackingLaneEmptyError}), never a+ * silent no-op, and the synthesized attention block stays a reserved+ * charge neither lane draws. See {@link PACKING_POLICIES}.+ */+ readonly packingPolicy?: PackingPolicy;/*** Per-memory character cap (v0.20.0): trim any single page's body to* this many code points before it consumes the token budget, so one@@ -387,6 +504,12 @@ interface Candidate {* the caller opts into `includeHistorical`.*/readonly supersededBy: string | null;+ /**+ * Which delivery directory the candidate came from. The opt-in+ * packing policy splits its two lanes on exactly this: preference+ * pages are the facts lane, retired pages are the pages lane.+ */+ readonly source: "preferences" | "retired";}function withOptionalLanes(@@ -410,6 +533,55 @@ function withOptionalLanes(};}+/**+ * Build one emitted pack item from a budgeted candidate. Shared by the+ * unified budget loop and the opt-in packing-policy loop so both+ * construct byte-identical items; the density recompute reads the+ * emitted (post-trim) body exactly as the unified loop always has.+ */+function toContextPackItem(+ c: Candidate,+ body: string,+ trimmed: boolean,+ tokens: number,+ densityRanking: boolean,+ densityScoreById: ReadonlyMap<string, number>,+): ContextPackItem {+ return {+ id: c.id,+ path: c.path,+ tier: c.tier,+ tokens,+ body,+ principle: c.principle,+ contextLane: c.contextLane,+ trimmed,+ epistemic: c.epistemic,+ evidenceRefs: c.evidenceRefs,+ ...(c.commitment !== null ? { commitment: c.commitment } : {}),+ // When the per-memory char budget trimmed this item, recompute density+ // from the emitted (post-trim) body/tokens so the surfaced value reflects+ // the actual content/token-cost the agent receives, not the pre-budget+ // candidate. Untrimmed items reuse the pre-computed score.+ ...(densityRanking+ ? {+ density: trimmed+ ? densityScore({ body, evidenceRefs: c.evidenceRefs, epistemic: c.epistemic }, tokens)+ : (densityScoreById.get(c.id) ?? 0),+ }+ : {}),+ ...(c.safety ? { safety: c.safety } : {}),+ };+}++/** A priced lane entry: the candidate as the lane would emit it, in tokens. */+interface PackedLaneEntry {+ readonly candidate: Candidate;+ readonly body: string;+ readonly trimmed: boolean;+ readonly tokens: number;+}+/*** Serve-with-correction coupling over the collected candidates* (truth-correctable-time-aware, contract item 3). A candidate whose@@ -461,6 +633,7 @@ function collectCandidates(// the "Why most readers keep the two-tuple form" section in// src/core/vault.ts. A pack returns injected content and carries no// per-candidate report, so it opens no channel of its own.+ const source: "preferences" | "retired" = dir === dirs.preferences ? "preferences" : "retired";const collected = collectPreferencePages(dir, { ownerScope });hiddenByOwnerScope += collected.hiddenByOwnerScope;for (const { name, path: full, meta, body } of collected.entries) {@@ -516,6 +689,7 @@ function collectCandidates(evidenceRefs: epistemic.evidenceRefs,commitment: readCommitmentTier(meta),supersededBy,+ source,...(contextSafetyReport(guarded) ? { safety: contextSafetyReport(guarded) } : {}),});}@@ -538,6 +712,14 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPacconst startedAtMs = Date.now();const semanticRelevance = semanticRelevanceFor(opts);if (!Number.isFinite(opts.maxTokens) || opts.maxTokens <= 0) {+ // The zero-budget short-circuit reads no vault: neither lane was ever+ // offered a budget to hold. A NAMED policy gets the same by-name+ // refusal the partition gives an empty lane - a zeroed accounting+ // object would answer a policy the caller asked for with a delivery+ // that never happened.+ if (opts.packingPolicy !== undefined) {+ throw ContextPackPackingLaneEmptyError.zeroBudget(opts.packingPolicy);+ }return finalizeContextPackReport(vault,opts,@@ -708,6 +890,249 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPacreturn a.id < b.id ? -1 : a.id > b.id ? 1 : 0;});+ // Opt-in packing policy (recall-lane-latency wave): correlate the two+ // candidate populations that share this one budget - preference pages+ // (the facts lane, the distilled layer) and retired pages (the pages+ // lane) - and let the caller name the claim order. The whole branch+ // runs ONLY when the policy is named; an omitted option never reaches+ // it, which is what keeps the unified pack byte-identical.+ if (opts.packingPolicy !== undefined) {+ const policy = opts.packingPolicy;+ const factsCandidates = candidates.filter((c) => c.source === "preferences");+ const pagesCandidates = candidates.filter((c) => c.source === "retired");+ const laneSkipped: ContextPackSkipped[] = [];+ // The substring reading still applies per lane, BEFORE the+ // allocation: a query-missed page never occupies its lane's budget.+ const applyLaneQuery = (list: ReadonlyArray<Candidate>): Candidate[] =>+ list.filter((c) => {+ if (query === null) return true;+ const haystack = normalizeForDedup(`${c.topic} ${c.principle}`);+ if (haystack.includes(query)) return true;+ laneSkipped.push({ id: c.id, tokens: c.tokens, reason: "filter-miss" });+ return false;+ });+ const factsFiltered = applyLaneQuery(factsCandidates);+ const pagesFiltered = applyLaneQuery(pagesCandidates);+ // A policy against an empty lane would silently do nothing; refuse+ // by name, before any receipt or telemetry exists to record a+ // delivery that never happened. Both empty lanes are named when both+ // are empty - the refusal states what it observed, not the first miss.+ const emptyLanes = [+ ...(factsFiltered.length === 0 ? (["facts"] as const) : []),+ ...(pagesFiltered.length === 0 ? (["pages"] as const) : []),+ ];+ if (emptyLanes.length > 0) {+ throw ContextPackPackingLaneEmptyError.emptyLanes(policy, emptyLanes);+ }+ // The synthesized attention block is a reserved charge against the+ // same total, prepared before the allocation so neither lane can+ // draw over it; the charge itself stays exactly where the unified+ // path draws it, after the lanes.+ let attention: {+ readonly body: string;+ readonly tokens: number;+ readonly safetyReport: ContextSafetyReport | undefined;+ } | null = null;+ if (opts.attentionFlowIds && opts.attentionFlowIds.length > 0) {+ const attentionBody = buildAttentionContextBlock(vault, opts.attentionFlowIds);+ if (attentionBody && attentionBody.trim()) {+ const guarded = guardBrainContextSnippet(attentionBody, {+ source: { id: "attention-flows", path: join(vault, "Brain", "attention", "flows") },+ ...(delimitUntrusted+ ? { delimitUntrusted: true, provenancePath: "Brain/attention/flows" }+ : {}),+ });+ const safeAttentionBody = guarded.safeText;+ if (!safeAttentionBody.trim()) {+ // The guard reduced the whole block to nothing - surface nothing+ // rather than an empty attention-flows item.+ laneSkipped.push({ id: "attention-flows", tokens: 0, reason: "filter-miss" });+ } else {+ const safetyReport = contextSafetyReport(guarded);+ attention = {+ body: safeAttentionBody,+ tokens: estimateTokens(safeAttentionBody),+ safetyReport,+ };+ }+ }+ }+ // A reserved charge that consumes the whole total leaves neither lane+ // anything to draw - the same silent delivery the zero-budget+ // short-circuit refuses, reached through the reserved charge instead+ // of a zero maxTokens. Both lanes hold candidates here (the+ // empty-lane refusal passed above), so packing by the named order+ // would answer with zeroed lane accounting; refuse by name instead,+ // before any receipt or telemetry exists to record a delivery that+ // never happened.+ if (attention !== null && attention.tokens >= opts.maxTokens) {+ throw ContextPackPackingLaneEmptyError.reservedCharge(+ policy,+ attention.tokens,+ opts.maxTokens,+ );+ }+ // Per-memory char cap stays lane-internal: trim inside the lane+ // first, then price the emitted text so the allocation charges what+ // the lane would actually deliver.+ const priceLane = (list: ReadonlyArray<Candidate>): PackedLaneEntry[] =>+ applyCharBudget(+ list.map((c) => ({ item: c, text: c.body })),+ {+ maxCharsPerEntry: opts.maxCharsPerMemory,+ ...(opts.degradation !== undefined ? { degradation: opts.degradation } : {}),+ },+ ).kept.map((k) => ({+ candidate: k.item,+ body: k.text,+ trimmed: k.trimmed,+ tokens: k.trimmed ? estimateTokens(k.text) : k.item.tokens,+ }));+ const factsEntries = priceLane(factsFiltered);+ const pagesEntries = priceLane(pagesFiltered);+ const allocation = allocateRecallBudget(+ opts.maxTokens,+ pagesEntries.map((entry) => ({ item: entry, tokens: entry.tokens })),+ factsEntries.map((entry) => ({ item: entry, tokens: entry.tokens })),+ policy,+ attention?.tokens ?? 0,+ );+ const claimOrder: ReadonlyArray<RecallBudgetLaneName> =+ policy === "retired_first" ? ["pages", "facts"] : ["facts", "pages"];+ // Lane overflow is an `over-budget` skip exactly like the unified+ // loop's; the policy adds no skip reason of its own.+ const tokensById = new Map<string, number>();+ for (const entry of [...factsEntries, ...pagesEntries]) {+ tokensById.set(entry.candidate.id, entry.tokens);+ }+ for (const lane of claimOrder) {+ for (const dropped of allocation[lane].dropped) {+ laneSkipped.push({+ id: dropped.candidate.id,+ tokens: tokensById.get(dropped.candidate.id) ?? 0,+ reason: "over-budget",+ });+ }+ }+ const items: ContextPackItem[] = [];+ for (const lane of claimOrder) {+ for (const kept of allocation[lane].kept) {+ items.push(+ toContextPackItem(+ kept.candidate,+ kept.body,+ kept.trimmed,+ kept.tokens,+ densityRanking,+ densityScoreById,+ ),+ );+ }+ }+ if (attention !== null) {+ const used = items.reduce((sum, item) => sum + item.tokens, 0);+ if (used + attention.tokens <= opts.maxTokens) {+ items.unshift({+ id: "attention-flows",+ path: join(vault, "Brain", "attention", "flows"),+ tier: PAGE_TIER.core,+ tokens: attention.tokens,+ body: attention.body,+ principle: "Declarative attention flow output",+ contextLane: "directives",+ trimmed: false,+ // Synthesized from the vault's attention-flow graph, not a+ // source-backed page, so it is `derived` with no evidence refs.+ epistemic: EPISTEMIC_STATUS.derived,+ evidenceRefs: [],+ ...(attention.safetyReport ? { safety: attention.safetyReport } : {}),+ });+ } else {+ laneSkipped.push({+ id: "attention-flows",+ tokens: attention.tokens,+ reason: "over-budget",+ });+ }+ }+ // Lane membership by id, so the delivered spends stay honest after+ // the char budget and the transforms have had their say.+ const laneById = new Map<string, RecallBudgetLaneName>();+ for (const c of factsCandidates) laneById.set(c.id, "facts");+ for (const c of pagesCandidates) laneById.set(c.id, "pages");+ const packingFor = (+ finalItems: ReadonlyArray<ContextPackItem>,+ ): { readonly packing: ContextPackPackingReport } => ({+ packing: Object.freeze({+ policy,+ reservedTokens: allocation.reservedTokens,+ pages: packingLaneReport("pages", allocation, laneById, finalItems, pagesFiltered.length),+ facts: packingLaneReport("facts", allocation, laneById, finalItems, factsFiltered.length),+ }),+ });+ // Total recall character cap: the same second ceiling the unified+ // path applies over the emitted items, dropping the lowest-priority+ // overflow - here, the tail of the lane order.+ if (opts.maxTotalChars && opts.maxTotalChars > 0) {+ const capped = applyCharBudget(+ items.map((i) => ({ item: i, text: i.body })),+ { maxTotalChars: opts.maxTotalChars },+ );+ if (capped.dropped.length > 0) {+ const keptItems = applyContextTransforms(+ capped.kept.map((k) => k.item),+ opts.transforms,+ );+ const droppedSet = new Set(capped.dropped);+ let recomputed = 0;+ for (const i of keptItems) recomputed += i.tokens;+ for (const d of items) {+ if (droppedSet.has(d)) {+ laneSkipped.push({+ id: d.id,+ tokens: d.tokens,+ reason: "over-char-budget",+ });+ }+ }+ return finalizeContextPackReport(+ vault,+ opts,+ {+ maxTokens: opts.maxTokens,+ tokensUsed: recomputed,+ items: Object.freeze(keptItems),+ skipped: Object.freeze(laneSkipped),+ ...withOptionalLanes(opts, keptItems),+ ...semanticReport,+ ...packingFor(keptItems),+ },+ startedAtMs,+ ownerScopeWarnings,+ withheldIds,+ );+ }+ }+ const finalItems = applyContextTransforms(items, opts.transforms);+ const finalTokensUsed = finalItems.reduce((sum, item) => sum + item.tokens, 0);+ return finalizeContextPackReport(+ vault,+ opts,+ {+ maxTokens: opts.maxTokens,+ tokensUsed: finalTokensUsed,+ items: Object.freeze(finalItems),+ skipped: Object.freeze(laneSkipped),+ ...withOptionalLanes(opts, finalItems),+ ...semanticReport,+ ...packingFor(finalItems),+ },+ startedAtMs,+ ownerScopeWarnings,+ withheldIds,+ );+ }+// Per-memory character cap (v0.20.0): trim oversized bodies in priority// order via the shared budget primitive before the token budget runs,// so one huge page cannot starve the rest. A trimmed body is re-tokenised@@ -736,31 +1161,7 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPacskipped.push({ id: c.id, tokens, reason: "over-budget" });continue;}- items.push({- id: c.id,- path: c.path,- tier: c.tier,- tokens,- body,- principle: c.principle,- contextLane: c.contextLane,- trimmed,- epistemic: c.epistemic,- evidenceRefs: c.evidenceRefs,- ...(c.commitment !== null ? { commitment: c.commitment } : {}),- // When the per-memory char budget trimmed this item, recompute density- // from the emitted (post-trim) body/tokens so the surfaced value reflects- // the actual content/token-cost the agent receives, not the pre-budget- // candidate. Untrimmed items reuse the pre-computed score.- ...(densityRanking- ? {- density: trimmed- ? densityScore({ body, evidenceRefs: c.evidenceRefs, epistemic: c.epistemic }, tokens)- : (densityScoreById.get(c.id) ?? 0),- }- : {}),- ...(c.safety ? { safety: c.safety } : {}),- });+ items.push(toContextPackItem(c, body, trimmed, tokens, densityRanking, densityScoreById));used += tokens;}@@ -845,7 +1246,7 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPac...semanticReport,},startedAtMs,- [],+ ownerScopeWarnings,withheldIds,);}@@ -870,6 +1271,29 @@ export function packContext(vault: string, opts: ContextPackOptions): ContextPac);}+/**+ * What one lane delivered: the allocation the correlated budget handed+ * it and the tokens actually present in the FINAL items, so the two+ * spends plus the reserved charge account for the whole budget no+ * matter what the char cap or the transforms dropped.+ */+function packingLaneReport(+ lane: RecallBudgetLaneName,+ allocation: RecallBudgetAllocation<PackedLaneEntry, PackedLaneEntry>,+ laneById: ReadonlyMap<string, RecallBudgetLaneName>,+ finalItems: ReadonlyArray<ContextPackItem>,+ candidates: number,+): ContextPackPackingLaneReport {+ let spent = 0;+ let items = 0;+ for (const item of finalItems) {+ if (laneById.get(item.id) !== lane) continue;+ spent += item.tokens;+ items += 1;+ }+ return Object.freeze({ allocation: allocation[lane].allocation, spent, candidates, items });+}+/*** The relevance map the `semantic` mode reads, or null in every other* mode. Throws {@link ContextPackSemanticRelevanceMissingError} when the@@ -996,6 +1420,14 @@ function finalizeContextPackReport(// query read two ways is two different requests, but a// caller that named none must keep the prefix it had....(opts.queryMode !== undefined ? { query_mode: opts.queryMode } : {}),+ // Added only when the caller named a policy, on the+ // same rule: the policy decides what the budget buys,+ // so the same query under two policies is two+ // different requests - and a caller that named none+ // keeps its cacheable prefix byte for byte.+ ...(opts.packingPolicy !== undefined+ ? { packing_policy: opts.packingPolicy }+ : {}),}),],}),@@ -1019,6 +1451,11 @@ function contextPackBudgetMetadata(? { max_chars_per_memory: opts.maxCharsPerMemory }: {}),...(opts.maxTotalChars !== undefined ? { max_total_chars: opts.maxTotalChars } : {}),+ // Named only when the caller named a policy: the receipt and the+ // synced telemetry record then carry a member of the frozen+ // PACKING_POLICIES vocabulary - which names nothing sensitive - and+ // an unnamed caller's records never grow the key.+ ...(opts.packingPolicy !== undefined ? { packing_policy: opts.packingPolicy } : {}),};}diff --git a/src/core/brain/preference-provenance.ts b/src/core/brain/preference-provenance.tsnew file mode 100644index 00000000..740f3e6a--- /dev/null+++ b/src/core/brain/preference-provenance.ts@@ -0,0 +1,194 @@+/**+ * Read-time provenance join for surfaced preferences (recall lane).+ *+ * A preference's `evidenced_by` frontmatter cites its origin signals as+ * wikilinks (`[[sig-...]]`), and a signal may carry a `session_ref` β+ * the coordinates of the conversation moment the fact came from. This+ * module joins the two at READ time: parse the ref, order the origins+ * by the preference's own evidence order, and hand the caller the+ * earliest cited one. The order is the preference's `evidenced_by`+ * order β never re-sorted, and never a chronological claim: merges+ * sort evidence lexically and windowed rewrites can drop older+ * signals from the list, so `origins[0]` is the earliest origin the+ * preference cites, not the earliest signal in time. Nothing is+ * written: no new frontmatter field, no preference rewrite, so a join+ * cannot corrupt the provenance it only reads.+ *+ * Three producers write `session_ref` today, each with its own shape,+ * and the parser is per-producer aware rather than shape-guessing:+ *+ * - import-session: `<portable basename>#<turn id>` (the ref is+ * composed in `src/core/brain/sessions/import.ts` from+ * `sessionRefIdentity`'s basename key; the fragment is a real+ * turn id) β resolves with kind `turn`;+ * - capture lifecycle: `session:<id>#<event>` (composed in+ * `src/core/brain/session-lifecycle.ts`; the fragment is a hook+ * event NAME β UserPromptSubmit, Stop, SessionEnd β not an event+ * id, so every prompt-sourced signal in a session shares the one+ * coordinate) β resolves with kind `session`: session-level+ * provenance, labeled as what it is;+ * - auto-extract: a bare session reference with no fragment+ * (`src/core/brain/extract-signals.ts` stores the caller's+ * session id verbatim) β resolves to NOTHING, because no turn is+ * available.+ *+ * The never-guess rule governs every shape outside those three: a ref+ * that is blank, fragment-less, double-hashed, or half-empty yields no+ * provenance rather than a best-effort reading. Callers report an+ * honestly-unresolved preference instead of an invented coordinate.+ *+ * Pure and deterministic, in the `detectThreadFacets` mold+ * (thread-topics.ts): no I/O, no clock, no randomness, no imports at+ * all β the signal lookup is injected, so the same input yields the+ * same origins in the same order across calls.+ */++/** How the fragment of a parsed `session_ref` should be read. */+export type SessionRefOriginKind = "turn" | "session";++/**+ * One origin coordinate parsed from a signal's `session_ref`.+ * `sessionKey` is the ref's key segment verbatim β the portable+ * session basename for import-session refs, the `session:<id>` form+ * for capture-lifecycle refs β so a consumer never sees a rewritten+ * identity.+ */+export interface SessionRefOrigin {+ readonly sessionKey: string;+ /** The ref's fragment: a turn id (`turn`) or a hook event name (`session`). */+ readonly fragment: string;+ readonly kind: SessionRefOriginKind;+}++/** The slice of a parsed preference the join needs. */+export interface PreferenceEvidenceSource {+ /** Preference id (`pref-<slug>`); becomes the stable `pref:<id>` key. */+ readonly id: string;+ /** Origin-signal wikilinks, in the preference's own evidence order. */+ readonly evidenced_by: ReadonlyArray<string>;+}++/**+ * Signal-id β `session_ref` lookup, injected so this module stays a+ * leaf. Returns `undefined` when the signal does not exist or carries+ * no `session_ref` β both mean "contributes no origin", never an error.+ */+export type SignalSessionRefLookup = (signalId: string) => string | undefined;++/**+ * The resolved provenance of one preference. `origins` follows the+ * preference's `evidenced_by` order (never re-sorted), with duplicate+ * coordinates folded into their first position, so `origins[0]` is the+ * earliest cited origin - an evidence-order figure, never a+ * chronological claim.+ */+export interface PreferenceProvenance {+ /** Stable facet key: `pref:<preference id>`. */+ readonly key: string;+ readonly preferenceId: string;+ readonly origins: ReadonlyArray<SessionRefOrigin>;+}++/** The capture-lifecycle key prefix; the one producer key that is not a basename. */+const CAPTURE_LIFECYCLE_SCHEME = "session:";+/** The stand-in id the lifecycle writer emits when the payload carries none. */+const CAPTURE_LIFECYCLE_UNKNOWN_ID = "unknown";+/**+ * The lifecycle writer's session-id grammar: the host's session id is a+ * UUID. The prefix alone is not evidence - an imported session file could+ * be named with the same prefix, and import keys are verbatim basenames -+ * so only the documented producer shapes mark a session-level fragment.+ */+const CAPTURE_LIFECYCLE_ID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;++/**+ * Parse one signal `session_ref` into its origin coordinates, or+ * `null` when the ref is absent, blank, fragment-less (the auto-extract+ * shape β no turn exists to cite), double-hashed (no producer emits+ * it), halves-empty, or `session:`-prefixed in a form neither producer+ * writes. The split is on the FIRST `#`, per the documented+ * `<key>#<fragment>` contract. A key in the capture-lifecycle writer's+ * exact forms - `session:<uuid>` or the `session:unknown` stand-in -+ * marks a hook-event fragment and resolves as `session` (session-level+ * provenance: the fragment names the hook, never a turn). Import keys+ * are verbatim basenames, so a basename that merely starts with the+ * scheme cannot be told from a lifecycle key by the ref alone: such a+ * ref yields no provenance rather than a guessed label - the+ * coordinates are never wrong, and no origin reads as no mention.+ */+export function parseSessionRef(ref: string): SessionRefOrigin | null {+ const trimmed = ref.trim();+ if (trimmed.length === 0) return null;+ const hash = trimmed.indexOf("#");+ // Bare reference: the auto-extract producer's shape, and no other+ // writer's. No fragment means no turn to cite β no provenance.+ if (hash < 0) return null;+ // A second `#` is a shape no producer emits; reading one would be a+ // guess about which segment is the fragment.+ if (trimmed.indexOf("#", hash + 1) >= 0) return null;+ const sessionKey = trimmed.slice(0, hash);+ const fragment = trimmed.slice(hash + 1);+ if (sessionKey.length === 0 || fragment.length === 0) return null;+ if (sessionKey.startsWith(CAPTURE_LIFECYCLE_SCHEME)) {+ const id = sessionKey.slice(CAPTURE_LIFECYCLE_SCHEME.length);+ if (id === CAPTURE_LIFECYCLE_UNKNOWN_ID || CAPTURE_LIFECYCLE_ID_RE.test(id)) {+ return Object.freeze({ sessionKey, fragment, kind: "session" });+ }+ // Neither lifecycle shape: a basename that collides with the scheme+ // (or an id no writer emits). The kind cannot be read without+ // guessing - no provenance, never a guessed label.+ return null;+ }+ return Object.freeze({ sessionKey, fragment, kind: "turn" });+}++/**+ * Evidence wikilinks are written as the exact `[[<signal id>]]` form;+ * anything else (a bare id, prose embedding a link, an aliased variant)+ * is not an evidence link this join may interpret, and yields no+ * target rather than a guessed one.+ */+const EXACT_EVIDENCE_WIKILINK_RE = /^\[\[([^[\]]+)\]\]$/;++function evidenceTarget(entry: string): string | null {+ // Inside a character class a lone `[` needs no escape, so the class+ // reads "one or more characters that are neither bracket kind".+ const match = EXACT_EVIDENCE_WIKILINK_RE.exec(entry.trim());+ if (match === null) return null;+ const target = match[1]?.trim() ?? "";+ return target.length > 0 ? target : null;+}++/**+ * Resolve one preference's origin provenance: walk `evidenced_by` in+ * order, resolve each exact evidence wikilink through `lookup`, parse+ * the signal's `session_ref`, and keep the origins that resolve β+ * duplicates folded into their first position. Returns `null` when+ * nothing resolves: an honestly-unresolved preference, never an empty+ * stand-in that could read as provenance.+ */+export function resolvePreferenceProvenance(+ preference: PreferenceEvidenceSource,+ lookup: SignalSessionRefLookup,+): PreferenceProvenance | null {+ const origins: SessionRefOrigin[] = [];+ const seen = new Set<string>();+ for (const entry of preference.evidenced_by) {+ const target = evidenceTarget(entry);+ if (target === null) continue;+ const ref = lookup(target);+ if (ref === undefined) continue;+ const origin = parseSessionRef(ref);+ if (origin === null) continue;+ const dedupKey = `${origin.kind}\u0000${origin.sessionKey}\u0000${origin.fragment}`;+ if (seen.has(dedupKey)) continue;+ seen.add(dedupKey);+ origins.push(origin);+ }+ if (origins.length === 0) return null;+ return Object.freeze({+ key: `pref:${preference.id}`,+ preferenceId: preference.id,+ origins: Object.freeze(origins),+ });+}diff --git a/src/core/brain/recall-budget.ts b/src/core/brain/recall-budget.tsindex 64eecaa4..d9d3a4a9 100644--- a/src/core/brain/recall-budget.ts+++ b/src/core/brain/recall-budget.ts@@ -174,3 +174,183 @@ export function applyCharBudget<T>(totalChars: used,});}++/**+ * How a caller wants the correlated recall budget claimed, one claim+ * order per member (recall-lane-latency wave).+ *+ * The premise of one reading budget over which a packing order reorders+ * is false ACROSS surfaces - pages and the facts-side brief are+ * budgeted by two separate surfaces with separate caps - so this+ * vocabulary names the claim order over the populations that DO share+ * one budget. Neither member is the omitted default: the omitted pack+ * interleaves both populations by tier, while a named policy orders whole+ * lanes, reserves the attention charge up front instead of drawing it+ * last, and - through the pack-level application of the policy - refuses+ * a vault where one lane holds no candidate in reach, a call the omitted+ * pack succeeds on. `facts_first` claims for the distilled facts lane+ * first and the retired pages fill the exact remainder. `retired_first`+ * hands the first claim to the retired pages lane, so a retired page+ * keeps its budget share even when facts-side demand would have+ * exhausted the remainder.+ */+export const PACKING_POLICIES = Object.freeze(["facts_first", "retired_first"] as const);++export type PackingPolicy = (typeof PACKING_POLICIES)[number];++/** Narrow an untrusted string to a {@link PACKING_POLICIES} member. */+export function isPackingPolicy(value: unknown): value is PackingPolicy {+ return typeof value === "string" && (PACKING_POLICIES as ReadonlyArray<string>).includes(value);+}++/** One candidate in a budget lane, priced in the shared token currency. */+export interface RecallBudgetEntry<T> {+ readonly item: T;+ /** Estimated token cost of the entry AS THE LANE WOULD EMIT IT. */+ readonly tokens: number;+}++export type RecallBudgetLaneName = "pages" | "facts";++export interface RecallBudgetLaneResult<T> {+ /** Which lane this result describes. */+ readonly name: RecallBudgetLaneName;+ /**+ * The share of the total this lane ends up holding: the first+ * claimer's allocation is exactly what it claimed; the second lane's+ * is the exact remainder handed to it (which the lane may under-use).+ * The two allocations plus the reserved charge always sum to the+ * total.+ */+ readonly allocation: number;+ /** Entries whose cost fits the lane's window, in input order. */+ readonly kept: ReadonlyArray<T>;+ /** Entries that did not fit, in input order. */+ readonly dropped: ReadonlyArray<T>;+ /** Sum of kept entries' token costs. */+ readonly spent: number;+}++export interface RecallBudgetAllocation<TPage, TFacts> {+ /** The policy that decided the claim order. */+ readonly policy: PackingPolicy;+ /** The shared total the lanes claimed against. */+ readonly totalTokens: number;+ /** Tokens charged before either lane claimed (see the argument). */+ readonly reservedTokens: number;+ readonly pages: RecallBudgetLaneResult<TPage>;+ readonly facts: RecallBudgetLaneResult<TFacts>;+}++/**+ * Walk one lane's claim over a budget window: entries are kept whole,+ * in input order, while the running spend stays inside the window; the+ * first entry that would overflow is dropped and the walk CONTINUES, so+ * a smaller later entry still fits (the same policy the context-pack+ * token loop applies to `over-budget` candidates). An entry with a+ * non-finite or negative cost is dropped rather than charged. The+ * primitive never splits or trims an entry: per-lane native caps+ * (chars, note counts) stay lane-internal fit rules, exactly as today.+ * The returned `spent` never exceeds the window.+ */+function claimLane<T>(+ name: RecallBudgetLaneName,+ entries: ReadonlyArray<RecallBudgetEntry<T>>,+ window: number,+): RecallBudgetLaneResult<T> {+ const kept: T[] = [];+ const dropped: T[] = [];+ let spent = 0;+ for (const { item, tokens } of entries) {+ if (!Number.isFinite(tokens) || tokens < 0) {+ dropped.push(item);+ continue;+ }+ if (spent + tokens > window) {+ dropped.push(item);+ continue;+ }+ spent += tokens;+ kept.push(item);+ }+ return Object.freeze({+ name,+ allocation: spent,+ kept: Object.freeze(kept),+ dropped: Object.freeze(dropped),+ spent,+ });+}++/** A lane result with its allocation restated to the window it holds. */+function withHeldWindow<T>(+ walk: RecallBudgetLaneResult<T>,+ heldWindow: number,+): RecallBudgetLaneResult<T> {+ return walk.allocation === heldWindow ? walk : Object.freeze({ ...walk, allocation: heldWindow });+}++/**+ * Correlate two recall populations over ONE shared token budget+ * (recall-lane-latency wave), beside the shared trimming primitive+ * above: both lanes already denominate content with the same+ * `estimateTokens` currency, so tokens are the correlation currency and+ * each lane's native caps stay its own business.+ *+ * Pure and deterministic. Claiming is strictly sequential with no+ * minimum-page floor: the policy names the first claimer, that lane+ * claims greedily against everything left after the reserved charge,+ * and the second lane fills the exact remainder - so conservation holds+ * by construction: `pages.allocation + facts.allocation + reservedTokens`+ * equals `totalTokens` whenever the total is usable. The total never+ * grows. A non-finite or non-positive total is a fail-closed empty+ * allocation: no lane claims anything, and the reserved charge is+ * dropped with it. A reserved charge over the total is clamped to it,+ * leaving both lanes nothing.+ *+ * The reserved charge (default 0) is a fixed cost paid before either+ * lane claims - the pack's synthesized attention block, for instance -+ * reported back on the allocation so every consumer accounts it inside+ * the same total instead of letting a lane draw over it.+ */+export function allocateRecallBudget<TPage, TFacts>(+ totalTokens: number,+ pagesLane: ReadonlyArray<RecallBudgetEntry<TPage>>,+ factsLane: ReadonlyArray<RecallBudgetEntry<TFacts>>,+ policy: PackingPolicy,+ reservedTokens = 0,+): RecallBudgetAllocation<TPage, TFacts> {+ const usable = Number.isFinite(totalTokens) && totalTokens > 0;+ const reserved = usable ? Math.min(Math.max(reservedTokens, 0), totalTokens) : 0;+ const claimable = usable ? totalTokens - reserved : 0;+ // The first claimer's allocation is exactly what it claimed, so the+ // second lane is handed the precise remainder and the books always+ // balance against the total. The second lane holds its whole window+ // even when it under-uses it.+ if (policy === "retired_first") {+ const pages = claimLane("pages", pagesLane, claimable);+ const facts = withHeldWindow(+ claimLane("facts", factsLane, claimable - pages.spent),+ claimable - pages.spent,+ );+ return Object.freeze({+ policy,+ totalTokens: usable ? totalTokens : 0,+ reservedTokens: reserved,+ pages,+ facts,+ });+ }+ const facts = claimLane("facts", factsLane, claimable);+ const pages = withHeldWindow(+ claimLane("pages", pagesLane, claimable - facts.spent),+ claimable - facts.spent,+ );+ return Object.freeze({+ policy,+ totalTokens: usable ? totalTokens : 0,+ reservedTokens: reserved,+ pages,+ facts,+ });+}diff --git a/src/core/search/embeddings/contract.ts b/src/core/search/embeddings/contract.tsindex 00c274b8..6e067bae 100644--- a/src/core/search/embeddings/contract.ts+++ b/src/core/search/embeddings/contract.ts@@ -19,6 +19,15 @@ export type EmbedKind = "query" | "passage";export interface EmbedCallOptions {/** The caller's cancellation: an aborted call stops its requests. */readonly signal?: AbortSignal;+ /**+ * The dispatch report (recall-lane-latency): invoked once per call,+ * for the first batch whose request is about to go on the wire - the+ * moment the caller's embed interval actually starts, not an earlier+ * pre-call instant. A provider that never dispatches (the offline+ * local embedder, or one built before the callback existed) never+ * calls it, and callers keep their own fallback stamp for that case.+ */+ readonly onDispatch?: () => void;}/**diff --git a/src/core/search/embeddings/http-util.ts b/src/core/search/embeddings/http-util.tsindex d08612a6..80d3c883 100644--- a/src/core/search/embeddings/http-util.ts+++ b/src/core/search/embeddings/http-util.ts@@ -339,6 +339,26 @@ export class Semaphore {await new Promise<void>((res) => this.waiters.push(res));return this.enter();}+ /**+ * Take a free permit WITHOUT ever yielding, or see there is none.+ *+ * `acquire()` is an `async` method, so `await acquire()` parks the+ * caller on a microtask even when a permit is free - and a microtask+ * cannot run until the caller's synchronous execution finishes. The+ * overlapped hybrid search (recall-lane-latency) starts the embed lane+ * in the caller's tick precisely so the HTTP request leaves before the+ * synchronous keyword lanes run; an await in the free-permit path+ * shipped the dispatch after them and the overlap never happened. The+ * fast path is `enter()` itself, which is synchronous: a free permit+ * reaches the caller in its own tick, `null` sends it to+ * {@link acquire} unchanged, and the permit contract (one release,+ * loud on a second) is identical either way.+ */+ tryAcquire(): SemaphorePermit | null {+ if (this.permits <= 0) return null;+ this.permits--;+ return this.enter();+ }}/**diff --git a/src/core/search/embeddings/openai-compat.ts b/src/core/search/embeddings/openai-compat.tsindex f204413c..b93614db 100644--- a/src/core/search/embeddings/openai-compat.ts+++ b/src/core/search/embeddings/openai-compat.ts@@ -305,11 +305,23 @@ export class OpenAICompatProvider implements EmbeddingProvider {// The caller's cancellation reaches every batch through the same// controller a failed sibling uses.const unlink = linkAbortSignal(opts?.signal, cancel);+ // The dispatch report fires once per call, for the first batch that+ // reaches the wire: the caller's embed interval starts when the+ // request actually leaves, not at an await inside a later batch task.+ let dispatched = false;const tasks = batches.map(async (batch) => {- const permit = await sem.acquire();+ // A free permit is taken synchronously, so the batch reaches `fetch`+ // in the caller's tick (recall-lane-latency): `await acquire()` alone+ // would park the dispatch on a microtask until the caller's+ // synchronous work - the keyword lanes - had finished.+ const permit = sem.tryAcquire() ?? (await sem.acquire());try {if (cancel.signal.aborted) return;+ if (dispatched === false && opts?.onDispatch !== undefined) {+ dispatched = true;+ opts.onDispatch();+ }const vectors = await this.embedBatchWithRetry(batch.map((b) => b.text),{diff --git a/src/core/search/embeddings/zeroentropy.ts b/src/core/search/embeddings/zeroentropy.tsindex 89ae43b7..bd320c3e 100644--- a/src/core/search/embeddings/zeroentropy.ts+++ b/src/core/search/embeddings/zeroentropy.ts@@ -125,11 +125,23 @@ export class ZeroEntropyProvider implements EmbeddingProvider {const out: number[][] = new Array(texts.length);const cancel = new AbortController();const unlink = linkAbortSignal(opts?.signal, cancel);+ // The dispatch report fires once per call, for the first batch that+ // reaches the wire: the caller's embed interval starts when the+ // request actually leaves, not at an await inside a later batch task.+ let dispatched = false;const tasks = batches.map(async (batch) => {- const permit = await sem.acquire();+ // A free permit is taken synchronously, so the batch reaches `fetch`+ // in the caller's tick (recall-lane-latency): `await acquire()` alone+ // would park the dispatch on a microtask until the caller's+ // synchronous work - the keyword lanes - had finished.+ const permit = sem.tryAcquire() ?? (await sem.acquire());try {if (cancel.signal.aborted) return;+ if (dispatched === false && opts?.onDispatch !== undefined) {+ dispatched = true;+ opts.onDispatch();+ }const vectors = await this.embedBatchWithRetry(batch.map((b) => b.text),{ maxAttempts: this.config.maxRetries, parentSignal: cancel.signal },diff --git a/src/core/search/explain-envelope.ts b/src/core/search/explain-envelope.tsindex 6d0cce7b..d77d1e1d 100644--- a/src/core/search/explain-envelope.ts+++ b/src/core/search/explain-envelope.ts@@ -25,7 +25,7 @@* projection and not a re-serialization.*/-import type { SearchOutcome } from "./types.ts";+import type { SearchLaneTiming, SearchOutcome } from "./types.ts";/** Config key that engages the retrieval trust gate. */export const TRUST_GATE_CONFIG_KEY = "search_trust_gate_enabled";@@ -75,6 +75,12 @@ export const MEMORY_TRUST_ASSESSMENT_KEY = "memory_trust_assessment";/** Response key carrying the reason there is no trace to show. */export const RETRIEVAL_TRACE_UNAVAILABLE_KEY = "retrieval_trace_unavailable";+/** Response key carrying the overlapped-lane milliseconds. */+export const OVERLAPPED_MS_KEY = "overlapped_ms";++/** Response key carrying whether the semantic lane attempted the embed. */+export const SEMANTIC_ATTEMPTED_KEY = "semantic_attempted";+/*** What a surface asks the envelope for. `crossVault` is not derivable* from the outcome - a union and a single-vault search with the gate off@@ -118,24 +124,37 @@ export function retrievalTraceUnavailableReason(* The explain keys for one outcome, ready to spread into a response* object. `explain: false` yields an empty object - the caller's payload* is then byte-identical to the pre-change one.+ *+ * `timing` (recall-lane-latency) is the per-call record the search's+ * timing sink handed the caller. It rides beside the receipts only under+ * an on flag - off means absent, never a null placeholder - and the+ * cross-vault union never captures one, so a merged payload carries no+ * timing rather than one origin's figure mislabeled as the union's.*/export function explainEnvelope(outcome: SearchOutcome,request: ExplainRequest,+ timing?: SearchLaneTiming,): Record<string, unknown> {if (!request.explain) return {};const trace = outcome.retrievalDecisionTrace;const assessment = outcome.memoryTrustAssessment;- if (trace === undefined || assessment === undefined) {- return {- [RETRIEVAL_TRACE_UNAVAILABLE_KEY]: retrievalTraceUnavailableReason(- request.crossVault,- request.trustGateEnabled,- ),- };- }+ const receipts =+ trace === undefined || assessment === undefined+ ? {+ [RETRIEVAL_TRACE_UNAVAILABLE_KEY]: retrievalTraceUnavailableReason(+ request.crossVault,+ request.trustGateEnabled,+ ),+ }+ : {+ [RETRIEVAL_DECISION_TRACE_KEY]: trace,+ [MEMORY_TRUST_ASSESSMENT_KEY]: assessment,+ };+ if (timing === undefined) return receipts;return {- [RETRIEVAL_DECISION_TRACE_KEY]: trace,- [MEMORY_TRUST_ASSESSMENT_KEY]: assessment,+ ...receipts,+ [OVERLAPPED_MS_KEY]: timing.overlappedMs,+ [SEMANTIC_ATTEMPTED_KEY]: timing.attempted,};}diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex c6b784ad..27dbdf1e 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -85,6 +85,7 @@ export type {ResolvedSearchConfig,SearchCard,SearchErrorCode,+ SearchLaneTiming,SearchOptions,SearchOutcome,SearchSessionFocus,diff --git a/src/core/search/pipeline/request.ts b/src/core/search/pipeline/request.tsindex 4b675141..da84c3c5 100644--- a/src/core/search/pipeline/request.ts+++ b/src/core/search/pipeline/request.ts@@ -66,6 +66,15 @@ export interface ResolvedSearchRequest {readonly sessionFocus: SearchSessionFocus | null;/** One clock for every time-resolving decision in this call. */readonly nowMs: number;+ /**+ * The request's clock, readable: the port every later wall-clock read+ * on the call - the lane-interval stamps, the composite deadline's+ * expiry checks - goes through, so the clock is read nowhere but this+ * module (the rank-clock census pins the site,+ * `tests/core/search/rank-clock-determinism.test.ts`). Each read is an+ * epoch-ms instant, the same register `nowMs` was resolved from.+ */+ readonly clock: () => number;readonly timeRange: ResolvedTimeRange | null;readonly temporalIntent: TemporalIntent | null;/** Whether local query expansion runs for this call. */@@ -138,6 +147,19 @@ export function resolveSearchRequest(// filter, the session-focus expiry and the query-declared window below// cannot disagree by a few milliseconds of wall-clock drift.const nowMs = Date.now();+ // The readable form of the same clock, handed to the phases that must+ // time a duration (the overlapped lane intervals, the composite+ // deadline): they read THROUGH this port, so the wall-clock read site+ // stays in this module and a downstream phase can never quietly bring+ // a second clock of its own. The port also keeps the call agreeing+ // with itself: a reading can never precede the instant the request was+ // resolved at, so no interval stamped through it starts before the+ // request that owns it, and a deadline's elapsed time is never+ // negative, however the host clock drifts between two reads.+ const clock = (): number => {+ const reading = Date.now();+ return reading > nowMs ? reading : nowMs;+ };const sessionFocus =opts.sessionFocus === undefined? readActiveSessionFocus(resolvedConfig, opts.focusSession, nowMs)@@ -207,6 +229,7 @@ export function resolveSearchRequest(policy,sessionFocus,nowMs,+ clock,timeRange,temporalIntent,expandActive,diff --git a/src/core/search/pipeline/semantic-lane.ts b/src/core/search/pipeline/semantic-lane.tsindex 1725f1fb..25e02bb0 100644--- a/src/core/search/pipeline/semantic-lane.ts+++ b/src/core/search/pipeline/semantic-lane.ts@@ -9,7 +9,7 @@ import { runSemanticPhase, semanticPoolSize } from "../semantic-phase.ts";import { contradictedAbiFields, formatEmbeddingAbiDrift } from "../store.ts";import { RETRIEVAL_DEGRADATION, noteDegradation } from "../retrieval-trail.ts";import type { RetrievalDegradationSink } from "../retrieval-trail.ts";-import type { SemanticPolicy } from "../semantic-phase.ts";+import type { SemanticEmbedTiming, SemanticPolicy } from "../semantic-phase.ts";import type { PropertyPushdown } from "../property-pushdown.ts";import type { Store } from "../store.ts";import type { TransportReach } from "../../graph/transport-reach.ts";@@ -28,10 +28,22 @@ export interface SemanticLaneInput {readonly kind?: "page" | "section";/** Promoted property pushdown (recall-quality, C4); absent leaves it unfiltered. */readonly pushdown?: PropertyPushdown;- /** Keyword pool size, the other half of the hybrid-degrade signal. */- readonly keywordHitCount: number;+ /**+ * Keyword pool size, the other half of the hybrid-degrade signal. A+ * supplier is read when the lane completes, not when it starts: the+ * overlapped caller starts this lane before the keyword lanes run and+ * can only hand over a closure over the pool they will have produced+ * (recall-lane-latency). A plain number is the snapshot form, for+ * callers that already hold the count.+ */+ readonly keywordHitCount: number | (() => number);/** The composite deadline's cancellation; absent when no deadline runs. */readonly signal?: AbortSignal;+ /**+ * The embed-interval record to stamp (recall-lane-latency), forwarded+ * to the phase and read by the join's caller. Absent records nothing.+ */+ readonly timing?: SemanticEmbedTiming;/*** The caller's reach, forwarded to the query-embed gateway. Absent* resolves to remote there, so an internal caller that must embed@@ -68,6 +80,7 @@ export async function runSemanticLane(input: SemanticLaneInput): Promise<Semantiexplicit: policy.explicit,...(input.signal !== undefined ? { signal: input.signal } : {}),...(input.transportReach !== undefined ? { transportReach: input.transportReach } : {}),+ ...(input.timing !== undefined ? { timing: input.timing } : {}),});attempted = semOutcome.attempted;hits = semOutcome.hits;@@ -105,11 +118,16 @@ export async function runSemanticLane(input: SemanticLaneInput): Promise<Semanti// structural warning when the caller wanted the semantic lane but it// did not run, so the query was served keyword-only. The granular// runSemanticPhase warnings above explain WHY; this is the single- // greppable flag a caller can test for.+ // greppable flag a caller can test for. The keyword count resolves+ // HERE, at completion: under the overlapped join the lane started+ // before the keyword lanes ran, and this is the first moment the+ // pool they produced exists to be counted.+ const keywordHitCount =+ typeof input.keywordHitCount === "function" ? input.keywordHitCount() : input.keywordHitCount;const degrade = detectHybridDegrade({wantSemantic: policy.wantSemantic,semanticAttempted: attempted,- keywordHitCount: input.keywordHitCount,+ keywordHitCount,});if (degrade !== null) {warnings.push(degrade);diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex 65ccb622..f7a42279 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -53,6 +53,7 @@ import { ResponseShapeError } from "../brain/response-shape.ts";import { resolveKindFilter } from "./property-pushdown.ts";import { noSecondPass, runSecondPassRecall } from "./pipeline/second-pass.ts";import { runSemanticLane, type SemanticLaneOutcome } from "./pipeline/semantic-lane.ts";+import type { SemanticEmbedTiming } from "./semantic-phase.ts";import { openReadOrSelfHeal } from "./pipeline/store-open.ts";import { resolveEffectiveWeights } from "./pipeline/weights.ts";import { detectHybridDegrade } from "./enrich.ts";@@ -125,12 +126,17 @@ const ABANDONED_LANE_FAILURE_PREFIX = "hybrid deadline: an abandoned lane failed* fires, the search completes keyword-only and names it with* `RETRIEVAL_DEGRADATION.hybridDeadlineExceeded` - never a stall, never a* silent partial.+ *+ * Exported for the overlap seam's tests, which build a real clock and pin+ * the abandoned-lane sentence byte for byte.*/-interface CompositeDeadline {+export interface CompositeDeadline {/** The configured budget, as resolved onto the request. */readonly budgetMs: number;/** Absolute fire time, on the request's clock so one clock rules the call. */readonly expiresAt: number;+ /** The request's clock, readable: every expiry check reads through it. */+ readonly clock: () => number;/*** Note the expiry once: the human warning sentence and the typed* degradation (with the elapsed/budget detail), plus the same@@ -154,13 +160,15 @@ interface CompositeDeadline {}function deadlineExpired(deadline: CompositeDeadline | null): boolean {- return deadline !== null && Date.now() >= deadline.expiresAt;+ return deadline !== null && deadline.clock() >= deadline.expiresAt;}-function createCompositeDeadline(input: {+export function createCompositeDeadline(input: {readonly budgetMs: number | null;/** The request's clock: the one `nowMs` every timed decision shares. */readonly startMs: number;+ /** The request's clock, readable: every check below reads through it. */+ readonly clock: () => number;readonly warnings: string[];readonly degraded: RetrievalDegradationSink;readonly wantSemantic: boolean;@@ -175,7 +183,7 @@ function createCompositeDeadline(input: {if (fired) return;fired = true;controller.abort();- const elapsedMs = Date.now() - input.startMs;+ const elapsedMs = input.clock() - input.startMs;input.warnings.push(`hybrid deadline ${budgetMs}ms exceeded after ${elapsedMs}ms; returning keyword-only results`,);@@ -202,6 +210,7 @@ function createCompositeDeadline(input: {return {budgetMs,expiresAt: input.startMs + budgetMs,+ clock: input.clock,fire,hasFired: () => fired,signal: controller.signal,@@ -209,6 +218,26 @@ function createCompositeDeadline(input: {};}+/**+ * The clock members the deadline-bounded lanes read. The full+ * {@link CompositeDeadline} satisfies it; the narrow shape keeps the+ * lane seams honest about what they touch and unit-testable without the+ * whole machinery.+ */+export interface LaneDeadline {+ /** Absolute fire time, on the request's clock. */+ readonly expiresAt: number;+ /**+ * Read the request's clock: the port the resolved request hands the+ * call, so a lane seam never reads the wall clock for itself.+ */+ readonly clock: () => number;+ /** Note the expiry (once) and abort the signal the lanes carry. */+ fire(): void;+ /** Name an abandoned lane's late, non-abort failure. */+ noteAbandonedFailure(error: unknown): void;+}+/*** Run one async lane under the deadline. A lane whose clock has already* run out is never started. A lane that loses the race is abandoned, not@@ -219,10 +248,11 @@ function createCompositeDeadline(input: {*/async function awaitWithinDeadline<T>(start: () => Promise<T>,- deadline: CompositeDeadline,+ deadline: LaneDeadline,fallback: () => T,+ hooks?: { readonly onRaceArmed?: () => void },): Promise<T> {- const remainingMs = deadline.expiresAt - Date.now();+ const remainingMs = deadline.expiresAt - deadline.clock();if (remainingMs <= 0) {deadline.fire();return fallback();@@ -234,6 +264,9 @@ async function awaitWithinDeadline<T>(void work.catch((error: unknown) => {if (abandoned) deadline.noteAbandonedFailure(error);});+ // From here the race owns the lane's rejection: a join seam that also+ // attached a handler at start time defuses it (never a double note).+ hooks?.onRaceArmed?.();let timer: ReturnType<typeof setTimeout> | undefined;const deadlineWon = new Promise<T>((resolve) => {timer = setTimeout(() => {@@ -254,6 +287,105 @@ function emptySemanticLane(): SemanticLaneOutcome {return { hits: [], attempted: false, warnings: [], degraded: [] };}+/**+ * The hidden window: the intersection of the embed interval and the+ * keyword interval, floored at 0. Extracted as a pure function because a+ * real overlapped run can legitimately measure zero - a small corpus lets+ * the synchronous FTS lane finish inside the wall clock's resolution - so+ * no integration run can pin the arithmetic itself. The unit tests over+ * this function are the exactness pin (swapped bounds, inverted order);+ * the integration suite only pins that the wiring carries a sane value.+ */+export function overlappedEmbedMs(+ embedStartMs: number,+ embedEndMs: number,+ keywordStartMs: number,+ keywordEndMs: number,+): number {+ return Math.max(0, Math.min(embedEndMs, keywordEndMs) - Math.max(embedStartMs, keywordStartMs));+}++/**+ * A lane started BEFORE the keyword lanes (recall-lane-latency): the+ * query embed is the one caller wait the keyword work can hide, so it is+ * put in flight here and joined where the serial await stood. The embed+ * request leaves in the caller's own tick; the keyword lanes then run to+ * completion synchronously while the round trip is outstanding.+ *+ * Two rules travel with the seam:+ *+ * - The never-start rule moves to the start site: a clock that has+ * already run out starts no embed at all. The FIRE waits for the+ * join: at the start site the keyword lanes have not run, and a fire+ * there would snapshot the empty keyword pool into the degrade+ * decision, losing the umbrella sentence the serial contract emits.+ * The join's never-start arm (inside {@link awaitWithinDeadline})+ * runs after the keyword lanes merged, so the same rule fires there+ * against the real pool - the snapshot point the serial pipeline had.+ * - The rejection handler attaches at START, not at join. Between the+ * two stand the synchronous keyword lanes, and a throw there unwinds+ * the call with the embed still in flight. With a deadline the+ * stranded lane's failure is the deadline's to name; without one the+ * keyword step's own error is what the caller sees, and the handler+ * only keeps the stranded rejection from crashing the process. Once+ * the race inside {@link awaitWithinDeadline} arms its own catch it+ * owns the rejection (rethrow before the fire, note after it) and+ * this handler defuses - but only then: a failure landing in the+ * start-to-join window, including after a join-time fire, stays the+ * deadline's to name.+ */+export interface SemanticLaneJoin<T> {+ /** Resolve the lane: its outcome, the fallback, or the lane's own error. */+ join(): Promise<T>;+}++export function startSemanticLane<T>(input: {+ /** Starts the lane; called in the caller's tick unless the clock ran out. */+ readonly start: () => Promise<T>;+ /** The composite clock; null runs the lane unbounded. */+ readonly deadline: LaneDeadline | null;+ /** The empty lane the deadline serves when it consumed the lane. */+ readonly fallback: () => T;+}): SemanticLaneJoin<T> {+ const deadline = input.deadline;+ if (deadline !== null && deadline.expiresAt - deadline.clock() <= 0) {+ // Never start the embed - but never fire here either: the keyword+ // lanes have not run, and the degrade decision must read the pool+ // they populated. Leave the fire to the join's never-start arm.+ return {+ join: () =>+ awaitWithinDeadline(+ () => {+ throw new Error(+ "the semantic lane never started: the hybrid deadline had already expired",+ );+ },+ deadline,+ input.fallback,+ ),+ };+ }+ const work = input.start();+ let raced = false;+ void work.catch((error: unknown) => {+ if (raced || deadline === null) return;+ deadline.noteAbandonedFailure(error);+ });+ return {+ join: async (): Promise<T> => {+ if (deadline === null) {+ raced = true;+ return await work;+ }+ return await awaitWithinDeadline(() => work, deadline, input.fallback, {+ onRaceArmed: () => {+ raced = true;+ },+ });+ },+ };+}+export async function search(config: ResolvedSearchConfig,opts: SearchOptions,@@ -270,6 +402,7 @@ export async function search(policy,sessionFocus,nowMs,+ clock,timeRange,temporalIntent,hybridDeadlineMs,@@ -323,6 +456,7 @@ export async function search(const deadline = createCompositeDeadline({budgetMs: hybridDeadlineMs,startMs: nowMs,+ clock,warnings,degraded,wantSemantic: policy.wantSemantic,@@ -409,7 +543,53 @@ export async function search(return outcome;};+ // Semantic candidates (may be skipped). The lane starts BEFORE the+ // keyword lanes (recall-lane-latency): its query embed is the one+ // caller wait the keyword work can hide, so it is put in flight here,+ // in this tick, and joined where the serial await stood below. The+ // keyword pool size rides the lane's lazy supplier - at start time the+ // keyword lanes have not run yet, and the lane reads the pool they+ // produced when it completes, the same snapshot the serial pipeline+ // captured at the join. Under the composite deadline the lane races+ // the clock from the join (the remaining budget is computed there, so+ // an early start neither starts the clock sooner nor extends it), the+ // deadline's signal rides in the lane input so a fire aborts the+ // in-flight embed, and a clock that already ran out starts nothing.+ let keywordHits: ReturnType<typeof runKeywordLane>["hits"] = [];+ // Per-call lane timing (recall-lane-latency): the record is owned+ // here, stamped by the phase around the embed, and folded into one+ // sink call at the join. Never an outcome field - the query cache+ // persists whole outcomes, so a field there would freeze this call's+ // timings into every cached row served afterwards.+ const laneTimingSink = opts.laneTiming;+ const embedTiming: SemanticEmbedTiming = { clock };+ const semanticLaneInput = {+ store,+ config: effectiveConfig,+ policy,+ query,+ semanticLaneQuery: shape.semanticLaneQuery,+ limit,+ pathPrefix,+ ...(kind !== undefined ? { kind } : {}),+ ...(pushdown !== null ? { pushdown } : {}),+ keywordHitCount: () => keywordHits.length,+ ...(opts.transportReach !== undefined ? { transportReach: opts.transportReach } : {}),+ ...(laneTimingSink !== undefined ? { timing: embedTiming } : {}),+ };+ const semanticLaneJoin = startSemanticLane({+ deadline,+ fallback: emptySemanticLane,+ start: () =>+ runSemanticLane(+ deadline === null ? semanticLaneInput : { ...semanticLaneInput, signal: deadline.signal },+ ),+ });+// Keyword candidates, with the plan the synonym step may have re-built.+ // The interval stamps read the request's clock port, never the wall+ // clock directly - the rank-clock census pins the read site.+ const keywordStartMs = clock();const keywordLane = runKeywordLane({store,recall: effectiveConfig.recall,@@ -426,7 +606,7 @@ export async function search(...(pushdown !== null ? { pushdown } : {}),matchMode: request.matchMode,});- let keywordHits = keywordLane.hits;+ keywordHits = keywordLane.hits;for (const w of keywordLane.warnings) warnings.push(w);for (const d of keywordLane.degraded) degraded.push(d);@@ -434,31 +614,37 @@ export async function search(effectiveConfig,keywordLane.plan,);+ // The hidden window the join waits through includes the weights+ // resolution, so the keyword interval ends here, not at the lane return.+ const keywordEndMs = clock();- // Semantic candidates (may be skipped). Under the composite deadline- // the lane races the clock and is abandoned past it, serving the empty- // lane outcome - the fire note explains the keyword-only answer.- const semanticLaneInput = {- store,- config: effectiveConfig,- policy,- query,- semanticLaneQuery: shape.semanticLaneQuery,- limit,- pathPrefix,- ...(kind !== undefined ? { kind } : {}),- ...(pushdown !== null ? { pushdown } : {}),- keywordHitCount: keywordHits.length,- ...(opts.transportReach !== undefined ? { transportReach: opts.transportReach } : {}),- };- const semanticLane =- deadline === null- ? await runSemanticLane(semanticLaneInput)- : await awaitWithinDeadline(- () => runSemanticLane({ ...semanticLaneInput, signal: deadline.signal }),- deadline,- emptySemanticLane,- );+ // The join, where the serial await stood: the keyword lanes ran to+ // completion above while the embed round trip was outstanding. Their+ // warnings merged first, and the lane's merge here keeps the same+ // keyword-before-semantic order the serial pipeline emitted.+ const semanticLane = await semanticLaneJoin.join();+ // One sink call at the join, on both deadline arms: the intervals and+ // their intersection, zero whenever no embed interval was stamped+ // (keyword-only, or a pre-flight refusal left no stamps). The start+ // is the dispatch report when the provider gave one, and the pre-call+ // fallback otherwise (a provider that never reports dispatches keeps+ // its whole-awaited-call interval). An embed that ran and was+ // discarded still reports its real interval.+ if (laneTimingSink !== undefined) {+ const embedStartMs = embedTiming.embedStartMs ?? embedTiming.preCallStartMs;+ const embedEndMs = embedTiming.embedEndMs;+ laneTimingSink({+ ...(embedStartMs !== undefined ? { embedStartMs } : {}),+ ...(embedEndMs !== undefined ? { embedEndMs } : {}),+ keywordStartMs,+ keywordEndMs,+ overlappedMs:+ embedStartMs !== undefined && embedEndMs !== undefined+ ? overlappedEmbedMs(embedStartMs, embedEndMs, keywordStartMs, keywordEndMs)+ : 0,+ attempted: semanticLane.attempted,+ });+ }let semanticHits = semanticLane.hits;let semanticAttempted = semanticLane.attempted;for (const w of semanticLane.warnings) warnings.push(w);diff --git a/src/core/search/semantic-phase.ts b/src/core/search/semantic-phase.tsindex 33fe154f..f0f7d11d 100644--- a/src/core/search/semantic-phase.ts+++ b/src/core/search/semantic-phase.ts@@ -60,6 +60,42 @@ export function semanticPoolSize(limit: number): number {return Math.max(limit * POOL_OVERFETCH, POOL_FLOOR);}+/**+ * The embed interval the phase stamps for the overlapped-lane timing+ * (recall-lane-latency). A plain mutable record the JOIN caller owns: the+ * lane forwards it, the phase writes it around the one caller wait it+ * has, and the caller reads it at the join. Absent stamps mean the embed+ * never ran - a pre-flight guard refused before any provider existed.+ */+export interface SemanticEmbedTiming {+ /**+ * The request's clock port: every stamp below reads through it, so the+ * wall clock is read nowhere but the request module (the rank-clock+ * census pins the site) and no stamp precedes the instant the request+ * was resolved at.+ */+ readonly clock: () => number;+ /**+ * The dispatch stamp: written by the provider's dispatch report, the+ * moment the request actually leaves. A provider that never reports a+ * dispatch leaves it unset, and the fold falls back to the pre-call+ * stamp.+ */+ embedStartMs?: number;+ /**+ * The resolution stamp: written when the phase's continuation around+ * the provider promise ran - the answer or the failure came back. It+ * is never a claim about any wire event past the call's own return.+ */+ embedEndMs?: number;+ /**+ * The pre-call fallback start: stamped just before the provider call+ * for providers that never report a dispatch, so their interval stays+ * the whole awaited call, as before the dispatch report existed.+ */+ preCallStartMs?: number;+}+interface SemanticPhaseOutcome {readonly attempted: boolean;readonly hits: ReturnType<Store["semanticTopK"]>;@@ -97,6 +133,11 @@ export async function runSemanticPhase(* everywhere else in the tree.*/transportReach?: TransportReach;+ /**+ * The embed-interval record to stamp (recall-lane-latency), written+ * around the embed await. Absent records nothing.+ */+ timing?: SemanticEmbedTiming;},): Promise<SemanticPhaseOutcome> {const warnings: string[] = [];@@ -180,15 +221,44 @@ export async function runSemanticPhase(}let queryVec: number[];+ // The embed start is stamped by the provider's dispatch report - the+ // moment the request actually leaves, which is the instant overlap is+ // measured from. The pre-call stamp below is only the fallback for+ // providers that never report a dispatch (the offline local embedder,+ // or one built before the callback existed): their interval stays the+ // whole awaited call, as before. With no sink there is no stamp and no+ // callback, and the call options are exactly what they were.+ const timing = opts.timing;+ if (timing !== undefined) timing.preCallStartMs = timing.clock();try {const provider = makeProvider(config.semantic);const vectors = await provider.embed([prepared.text],"query",- opts.signal !== undefined ? { signal: opts.signal } : undefined,+ timing === undefined && opts.signal === undefined+ ? undefined+ : {+ ...(opts.signal !== undefined ? { signal: opts.signal } : {}),+ ...(timing !== undefined+ ? {+ onDispatch: () => {+ timing.embedStartMs = timing.clock();+ },+ }+ : {}),+ },);+ if (timing !== undefined) timing.embedEndMs = timing.clock();queryVec = vectors[0] ?? [];} catch (e) {+ // The interval closes even on the failure arm: the overlap figure+ // stays coherent for a lane that ran and lost its embed.+ if (+ timing !== undefined &&+ (timing.embedStartMs !== undefined || timing.preCallStartMs !== undefined)+ ) {+ timing.embedEndMs = timing.clock();+ }if (opts.signal?.aborted === true) throw opts.signal.reason;if (opts.explicit) {// Defensive: provider methods are expected to throw SearchError,diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex ce5220b4..6cf2b41d 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -861,6 +861,66 @@ export interface EvidencePack {readonly completeness?: CompletenessReport;}+/**+ * Per-call lane timing (recall-lane-latency): when the embed round trip+ * ran relative to the keyword lanes, and how much of it they hid - the+ * intersection of the embed's dispatch-to-resolution interval with the+ * keyword interval, `max(0, min(embedEnd, keywordEnd) -+ * max(embedStart, keywordStart))`, zero whenever no embed interval was+ * stamped.+ *+ * This record rides a per-call sink and NEVER lands on+ * {@link SearchOutcome}: the query cache persists whole outcomes, so an+ * outcome field would freeze one call's timings into every cached row+ * served afterwards. Absent embed stamps mean the embed never ran.+ */+export interface SearchLaneTiming {+ /**+ * Absolute `Date.now()` stamp of the query embed's start - the moment+ * the provider put the request on the wire, as reported through the+ * provider's dispatch callback. A provider that does not report+ * dispatches falls back to the instant the phase called into it, so+ * its interval stays the whole awaited call. Absent when the embed+ * never ran.+ */+ readonly embedStartMs?: number;+ /**+ * Absolute `Date.now()` stamp of the query embed's RESOLUTION: when+ * the phase's continuation around the provider promise ran, i.e. the+ * answer (or the failure) came back - never a claim about any wire+ * event past that return. Absent when the embed never ran.+ */+ readonly embedEndMs?: number;+ /** Absolute `Date.now()` stamp taken when the keyword lanes started. */+ readonly keywordStartMs: number;+ /** Absolute `Date.now()` stamp taken when the keyword lanes and the weights resolution completed. */+ readonly keywordEndMs: number;+ /**+ * The overlapped milliseconds, an integer >= 0; 0 when no embed+ * interval was stamped. This key answers HOW MUCH of the keyword+ * interval the embed's dispatch-to-resolution hid, and it is reported+ * whenever both embed stamps exist - the interval was measured, even+ * if the lane's hits were discarded afterwards.+ */+ readonly overlappedMs: number;+ /**+ * Whether the semantic lane delivered a result to the join. An embed+ * that never ran or failed before the join reports false; an embed the+ * deadline's post-join void then discarded can still report true - the+ * sink is read at the join, and the void past it is the deadline's,+ * not this flag's.+ *+ * This key answers a different question than `overlappedMs` - whether+ * the lane's hits joined the answer, not whether embed time was+ * measured - so the two can legitimately disagree: a deadline fallback+ * that wins after the embed ran reports measured overlap alongside+ * `attempted: false`. Consumers pick per question; the bench reports+ * `overlapped_ms` only for questions that attempted, `--explain`+ * prints both as measured.+ */+ readonly attempted: boolean;+}+export interface SearchOptions {readonly query: string;readonly limit?: number;@@ -1129,6 +1189,15 @@ export interface SearchOptions {* it unset.*/readonly freshenSpawn?: (argv: string[]) => void;+ /**+ * Per-call lane-timing sink (recall-lane-latency), in the+ * {@link SearchOptions.freshenSpawn} mold: invoked once at the+ * semantic-lane join with the embed and keyword intervals and their+ * overlap. Nothing timing-shaped reaches the outcome or the query+ * cache - surfaces that want the figure project it themselves+ * (`search --explain`, the bench). Omitted records nothing.+ */+ readonly laneTiming?: (timing: SearchLaneTiming) => void;/*** Skip rerank kind `decision-model` for this call. Set by the hook* surfaces (recall inject), whose time budget is shorter than a decisiondiff --git a/src/mcp/brain/pack-tools.ts b/src/mcp/brain/pack-tools.tsindex c557b0a0..8b401033 100644--- a/src/mcp/brain/pack-tools.ts+++ b/src/mcp/brain/pack-tools.ts@@ -21,14 +21,21 @@ import { SearchError } from "../../core/search/search-error.ts";import {discloseSpentQuery,loadBeliefSemanticRelevance,+ spentQueryDisclosure,type BeliefSemanticRelevance,} from "../../core/brain/belief-semantic.ts";import { readActiveSessionFocus } from "../../core/search/session-focus.ts";import {CONTEXT_PACK_QUERY_MODES,+ ContextPackPackingLaneEmptyError,isContextPackQueryMode,packContext,} from "../../core/brain/context-pack.ts";+import {+ isPackingPolicy,+ PACKING_POLICIES,+ type PackingPolicy,+} from "../../core/brain/recall-budget.ts";import { assessRecallAdequacy } from "../../core/brain/recall-adequacy.ts";import { recordRecallAdequacyDemand } from "../../core/brain/query-demand.ts";import { emitGatedTelemetry } from "../../core/brain/continuity/emit.ts";@@ -205,6 +212,29 @@ async function toolBrainContextPack();}}+ // Opt-in packing policy (recall-lane-latency wave). Refused beside the+ // `dependentRequired` that declares it, exactly as query_mode is: a+ // policy with no query has no turn to pack for and would be the silent+ // no-op this project refuses. A blank or whitespace-only query is the+ // same refusal, not a pass: the empty string is dropped before the core+ // and the null query filter-passes every candidate, so the policy would+ // silently reorder an unordered pack - the exact evasion the raw read+ // above would otherwise allow.+ const packingPolicy = coerceStr(args, "packing_policy", false);+ if (packingPolicy !== null) {+ if (!isPackingPolicy(packingPolicy)) {+ throw new MCPError(+ INVALID_PARAMS,+ `brain_context_pack: packing_policy must be one of ${PACKING_POLICIES.join(", ")}`,+ );+ }+ if (query === undefined || query.trim() === "") {+ throw new MCPError(+ INVALID_PARAMS,+ "brain_context_pack: packing_policy requires query; a policy with nothing to order is inert",+ );+ }+ }if (query !== undefined && exceedsMcpQueryCap(query)) {throw new MCPError(INVALID_PARAMS, `brain_context_pack: ${MCP_QUERY_CAP_MESSAGE}`);}@@ -332,44 +362,63 @@ async function toolBrainContextPack();}): null;- const report = await withSearchErrorsOnWire(- async () =>- packContext(ctx.vault, {- ...(remote- ? { visible: (abs: string) => view.visible(vaultRelative(ctx.vault, abs)) }- : {}),- maxTokens,- ...(agentScope !== undefined ? { agentScope } : {}),- ...(densityRanking ? { densityRanking: true } : {}),- ...(sessionFocus !== null ? { sessionFocus } : {}),- ...(query ? { query } : {}),- ...(queryMode !== null ? { queryMode } : {}),- ...(includeLanes ? { includeLanes: true } : {}),- ...(receipt !== undefined ? { receipt } : {}),- ...(adequacy !== undefined ? { recallAdequacy: adequacy } : {}),- ...(answerableVerdict !== undefined ? { decisionAnswerable: answerableVerdict } : {}),- ...(cacheStable || dedupRepeated- ? {- transforms: {- ...(cacheStable ? { cacheStableOrdering: true } : {}),- ...(dedupRepeated ? { deduplicateRepeatedContext: true } : {}),- },- }- : {}),- ...(maxCharsPerMemory !== undefined ? { maxCharsPerMemory } : {}),- ...(maxTotalChars !== undefined ? { maxTotalChars } : {}),- ...(configuredDegradation(ctx.vault) !== undefined- ? { degradation: configuredDegradation(ctx.vault)! }- : {}),- ...(telemetry !== undefined ? { telemetry } : {}),- // The synthesized attention-flow block has no page of its own to ask- // the reach view about, so a remote caller does not get it (fail- // closed, as the per-item filter this replaced did).- ...(attentionFlowIds.length > 0 && !remote ? { attentionFlowIds } : {}),- ...(semantic !== null ? { semanticRelevance: semantic.relevanceByPath } : {}),- }),- discloseSpentPackQuery(semantic),- );+ let report: Awaited<ReturnType<typeof packContext>>;+ try {+ report = await withSearchErrorsOnWire(+ async () =>+ packContext(ctx.vault, {+ ...(remote+ ? { visible: (abs: string) => view.visible(vaultRelative(ctx.vault, abs)) }+ : {}),+ maxTokens,+ ...(agentScope !== undefined ? { agentScope } : {}),+ ...(densityRanking ? { densityRanking: true } : {}),+ ...(sessionFocus !== null ? { sessionFocus } : {}),+ ...(query ? { query } : {}),+ ...(queryMode !== null ? { queryMode } : {}),+ ...(packingPolicy !== null ? { packingPolicy: packingPolicy as PackingPolicy } : {}),+ ...(includeLanes ? { includeLanes: true } : {}),+ ...(receipt !== undefined ? { receipt } : {}),+ ...(adequacy !== undefined ? { recallAdequacy: adequacy } : {}),+ ...(answerableVerdict !== undefined ? { decisionAnswerable: answerableVerdict } : {}),+ ...(cacheStable || dedupRepeated+ ? {+ transforms: {+ ...(cacheStable ? { cacheStableOrdering: true } : {}),+ ...(dedupRepeated ? { deduplicateRepeatedContext: true } : {}),+ },+ }+ : {}),+ ...(maxCharsPerMemory !== undefined ? { maxCharsPerMemory } : {}),+ ...(maxTotalChars !== undefined ? { maxTotalChars } : {}),+ ...(configuredDegradation(ctx.vault) !== undefined+ ? { degradation: configuredDegradation(ctx.vault)! }+ : {}),+ ...(telemetry !== undefined ? { telemetry } : {}),+ // The synthesized attention-flow block has no page of its own to ask+ // the reach view about, so a remote caller does not get it (fail+ // closed, as the per-item filter this replaced did).+ ...(attentionFlowIds.length > 0 && !remote ? { attentionFlowIds } : {}),+ ...(semantic !== null ? { semanticRelevance: semantic.relevanceByPath } : {}),+ }),+ discloseSpentPackQuery(semantic),+ );+ } catch (e) {+ // A policy against a one-lane vault is a caller error the core+ // refuses by name; on the wire it is INVALID_PARAMS carrying that+ // same named message, never an internal error and never a no-op.+ if (e instanceof ContextPackPackingLaneEmptyError) {+ // The spend disclosure rides the refusal too: in semantic mode the+ // query embed was paid before the lanes were judged, and a refusal+ // that hides that spend under-discloses exactly when the caller+ // gets nothing back for it.+ throw new MCPError(+ INVALID_PARAMS,+ semantic !== null ? `${e.message} ${spentQueryDisclosure(semantic.report)}` : e.message,+ );+ }+ throw e;+ }const warnings = [...(report.warnings ?? []),...(semantic?.warnings ?? []),@@ -413,6 +462,9 @@ async function toolBrainContextPack(...(semantic !== null && report.semantic !== undefined? { semantic: semanticReportField(semantic, report.semantic) }: {}),+ // The packing accounting appears only when the caller named a+ // policy - the same conditional shape as `semantic` above.+ ...(report.packing ? { packing: report.packing } : {}),...(adequacy !== undefined? {adequacy: {@@ -1059,6 +1111,12 @@ export const PACK_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([description:"How `query` is read: `substring` (default) drops misses; `ranked` orders by token overlap; `semantic` orders by stored belief vectors, embedding the query once.",},+ packing_policy: {+ type: "string",+ enum: [...PACKING_POLICIES],+ description:+ "Budget claim order: `facts_first` or `retired_first`. Neither is the omitted default; a vault where one lane has no candidate is refused. Requires `query`.",+ },focus_session: {type: "string",minLength: 1,@@ -1134,11 +1192,14 @@ export const PACK_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([// discover the pairing instead of learning it from an INVALID_PARAMS// at call time. Built beside the enforcement; see// `recallAdequacyPairing`. `query_mode` joins it one-directionally:- // a mode needs a query, a query needs no mode.+ // a mode needs a query, a query needs no mode. `packing_policy`+ // joins the same way: a policy with nothing to order is inert, so+ // it is refused beside its `dependentRequired` entry.dependentRequired: {...recallAdequacyPairing(RECALL_SCORES_ARG_NAME),...decisionAnswerablePairing(RECALL_SCORES_ARG_NAME),query_mode: ["query"],+ packing_policy: ["query"],},additionalProperties: false,},diff --git a/tests/cli/brain-bench.test.ts b/tests/cli/brain-bench.test.tsindex 090cecc9..b14fd10c 100644--- a/tests/cli/brain-bench.test.ts+++ b/tests/cli/brain-bench.test.ts@@ -439,6 +439,12 @@ describe("bench knob pin and recall floor", () => {// A scalar knob's VALUE is not shape-checked here: an out-of-range// number is resolveSearchConfig's named error, not this parser's.expect(parseBenchPinnedKnobs('{"keywordWeight":99}')).toEqual({ keywordWeight: 99 });+ // The standing-instruction facet lane's arm is pinnable, so the+ // lane-off arm of the category is reachable from the CLI too+ // (recall-lane-latency).+ expect(parseBenchPinnedKnobs('{"standingInstructionFacets":false}')).toEqual({+ standingInstructionFacets: false,+ });expect(() => parseBenchPinnedKnobs('{"semantic":false}')).toThrow(/'semantic' must be a JSON object/,);@@ -742,3 +748,232 @@ describe("bench help text", () => {expect(flattened).toContain("[--sample-seed <value>] [--sample-cap <n>]");});});++/**+ * The standing_instruction human line (recall-lane-latency): printed only+ * when the report's quality buckets carry the category - a run without+ * standing_instruction questions keeps its stdout byte-identical, a+ * lane-on run prints the category beside its rates, and the exit code+ * stays all-pass-or-nonzero with the category in play.+ */+describe("bench standing_instruction human line", () => {+ /** The surfacing lane's fixture vault files, in the live writers' shapes. */+ const STANDING_FLOW_MD = [+ "---",+ "kind: brain-attention-flow",+ "id: open-loops",+ "title: Open loops",+ "actions:",+ " - standing_query",+ "standing_queries:",+ " - testing",+ "status: active",+ "---",+ "",+ "# Open loops",+ "",+ ].join("\n");+ const STANDING_PREF_MD = [+ "---",+ "kind: brain-preference",+ "id: pref-tabs",+ "created_at: 2026-06-01T00:00:00Z",+ "_confirmed_at: 2026-06-02T00:00:00Z",+ "unconfirmed_until: 2026-07-01T00:00:00Z",+ "tags:",+ " - bench",+ "topic: tabs",+ "_status: confirmed",+ "principle: Use tabs for indentation",+ "_evidenced_by:",+ ' - "[[sig-bench-turn]]"',+ "_applied_count: 0",+ "_violated_count: 0",+ "_last_evidence_at: null",+ "_confidence: low",+ "_confidence_value: null",+ "pinned: false",+ "scope: testing",+ "---",+ "",+ "Use tabs for indentation",+ "",+ ].join("\n");+ const STANDING_SIGNAL_MD = [+ "---",+ "kind: brain-signal",+ "id: sig-bench-turn",+ "created_at: 2026-06-01T00:00:00Z",+ "tags:",+ " - bench",+ "topic: tabs",+ "signal: positive",+ "agent: bench",+ "principle: use tabs for indentation",+ "scope: testing",+ "session_ref: bench-session.json#turn-2",+ "---",+ "",+ "recorded preference signal",+ "",+ ].join("\n");++ const STANDING_SUITE = "standing-stub";+ registerBenchSuite(STANDING_SUITE, {+ loader: () => ({+ fixture: parseBenchFixture({+ name: "standing-cli",+ notes: [+ { path: "Brain/attention/flows/open-loops.md", body: STANDING_FLOW_MD },+ { path: "Brain/preferences/pref-tabs.md", body: STANDING_PREF_MD },+ { path: "Brain/inbox/sig-bench-turn.md", body: STANDING_SIGNAL_MD },+ ],+ continuity: [],+ questions: [+ {+ id: "q-standing",+ category: "standing_instruction",+ expected_ids: ["pref:pref-tabs"],+ expected_text: "earliest cited turn bench-session.json#turn-2]",+ },+ ],+ }),+ }),+ capabilities: { embeddings: false },+ });++ /** The same fixture with the expectation unmet: the question fails. */+ const FAILING_SUITE = "standing-failing-stub";+ registerBenchSuite(FAILING_SUITE, {+ loader: () => ({+ fixture: parseBenchFixture({+ name: "standing-cli-failing",+ notes: [+ { path: "Brain/attention/flows/open-loops.md", body: STANDING_FLOW_MD },+ { path: "Brain/preferences/pref-tabs.md", body: STANDING_PREF_MD },+ { path: "Brain/inbox/sig-bench-turn.md", body: STANDING_SIGNAL_MD },+ ],+ continuity: [],+ questions: [+ {+ id: "q-standing",+ category: "standing_instruction",+ expected_ids: ["pref:pref-never-resolves"],+ },+ ],+ }),+ }),+ capabilities: { embeddings: false },+ });++ test("a run without the category keeps stdout byte-identical", async () => {+ const out = await runCli(+ ["brain", "bench", "memory", "--fixture", "core-recall", "--runs-dir", runsDir],+ {},+ );+ expect(out.returncode).toBe(0);+ expect(out.stdout).not.toContain("standing_instruction");+ }, 60_000);++ test("a lane-on run prints the new category and its rates", async () => {+ const out = await runCli(+ [+ "brain",+ "bench",+ "memory",+ "--fixture",+ "core-recall",+ "--suite",+ STANDING_SUITE,+ "--runs-dir",+ join(runsDir, "lane-on"),+ ],+ {},+ );+ expect(out.returncode).toBe(0);+ expect(out.stdout).toContain("standing_instruction: 1/1 (pass rate 1)");+ }, 60_000);++ test("the exit code stays all-pass-or-nonzero with the new category in play", async () => {+ const out = await runCli(+ [+ "brain",+ "bench",+ "memory",+ "--fixture",+ "core-recall",+ "--suite",+ FAILING_SUITE,+ "--runs-dir",+ join(runsDir, "failing"),+ ],+ {},+ );+ expect(out.returncode).not.toBe(0);+ // The failing question names the category and the missing facet.+ expect(out.stdout).toContain("FAIL q-standing (standing_instruction)");+ expect(out.stdout).toContain("pref:pref-never-resolves");+ }, 60_000);+});++/**+ * The overlapped_ms human line (recall-lane-latency): printed only when+ * the aggregate latency family carries the figure - a keyword-only run+ * (the default pin) keeps its stdout byte-identical, and a lane-on run+ * through an embeddings-capable suite prints it.+ */+describe("bench overlapped_ms human line", () => {+ const OVERLAP_SUITE = "overlap-stub";+ const OVERLAP_FIXTURE = parseBenchFixture({+ name: "overlap-stub",+ notes: [+ {+ path: "Brain/notes/rust.md",+ body: "# Rust\n\nThe systems team standardized on Rust for the new ingest service.\n",+ },+ ],+ questions: [+ {+ id: "q-rust",+ category: "single_hop",+ query: "systems language for the ingest service",+ expected_paths: ["Brain/notes/rust.md"],+ },+ ],+ });++ registerBenchSuite(OVERLAP_SUITE, {+ loader: () => ({ fixture: OVERLAP_FIXTURE }),+ capabilities: { embeddings: true },+ });++ test("the overlapped_ms line prints only when the aggregate carries it", async () => {+ // Keyword-only default run: no line, byte-identical stdout.+ const plain = await runCli(+ ["brain", "bench", "memory", "--fixture", "core-recall", "--runs-dir", runsDir],+ {},+ );+ expect(plain.returncode).toBe(0);+ expect(plain.stdout).not.toContain("overlapped_ms");++ // Lane-on run through the embeddings-capable suite: the line prints.+ const laneOn = await runCli(+ [+ "brain",+ "bench",+ "memory",+ "--fixture",+ "core-recall",+ "--suite",+ OVERLAP_SUITE,+ "--runs-dir",+ join(runsDir, "lane-on"),+ "--pin",+ JSON.stringify({ semantic: { enabled: true, provider: "local" } }),+ ],+ {},+ );+ expect(laneOn.returncode).toBe(0);+ expect(laneOn.stdout).toMatch(/overlapped_ms: avg \d+/);+ }, 60_000);+});diff --git a/tests/cli/search-explain-overlap.test.ts b/tests/cli/search-explain-overlap.test.tsnew file mode 100644index 00000000..23610516--- /dev/null+++ b/tests/cli/search-explain-overlap.test.ts@@ -0,0 +1,159 @@+/**+ * `o2b search --explain` reports the overlapped lane timing+ * (recall-lane-latency): how much of the semantic lane's embed round trip+ * the keyword lanes hid, measured as the intersection of the two+ * intervals. The timing rides a per-call sink - never SearchOutcome,+ * which the query cache persists whole - so the flag-off payload is+ * byte-identical (off means absent), and a keyword-only answer reports+ * zero with the lane not attempted.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { FAKE_PROVIDER_KEY } from "../helpers/fake-credentials.ts";+import { startFakeHttp, type FakeHttp } from "../helpers/fake-http.ts";+import { runCli } from "../helpers/run-cli.ts";++const QUERY = "quick brown fox";+const MODEL = "fake-model";++let vault: string;+let configHome: string;+let configPath: string;+let server: FakeHttp;++/** Embeddings-shaped body with the default handler's deterministic vectors. */+function embeddingsBody(req: { body: unknown }, delayMs: number) {+ const body = (req.body ?? {}) as { input?: string[]; model?: string };+ const inputs = Array.isArray(body.input) ? body.input : [];+ return {+ status: 200,+ body: {+ data: inputs.map((text, index) => {+ const tokens = text.split(/\s+/).filter(Boolean).length;+ return { object: "embedding", embedding: [tokens, index, text.length, 1], index };+ }),+ model: body.model ?? MODEL,+ },+ delayMs,+ };+}++beforeEach(async () => {+ server = await startFakeHttp();+ vault = mkdtempSync(join(tmpdir(), "o2b-explain-overlap-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-explain-overlap-cfg-"));+ mkdirSync(join(vault, "notes"), { recursive: true });+ writeFileSync(+ join(vault, "notes", "fox.md"),+ "---\ntitle: Fox\n---\n\n# Fox\n\nThe quick brown fox jumps over the lazy dog.\n",+ );+ writeFileSync(+ join(vault, "notes", "bar.md"),+ "---\ntitle: Bar\n---\n\n# Bar\n\nA different note about cats and turtles.\n",+ );+ // A real body of notes keeps the synchronous FTS lane comfortably above+ // the wall-clock resolution, so a positive-overlap pin cannot flake on a+ // same-millisecond keyword interval.+ for (let i = 0; i < 60; i++) {+ writeFileSync(+ join(vault, "notes", `filler-${i}.md`),+ `---\ntitle: Filler ${i}\n---\n\n# Filler ${i}\n\nNote ${i} about herons, sleds, compasses and other words the query never uses.\n`,+ );+ }+ configPath = join(configHome, "config.yaml");+ writeFileSync(+ configPath,+ [+ `vault: ${vault}`,+ "search_semantic_enabled: true",+ "embedding_provider: openai-compat",+ `embedding_base_url: ${server.url}`,+ `embedding_model: ${MODEL}`,+ `embedding_api_key: ${FAKE_PROVIDER_KEY}`,+ "embedding_dimension: 4",+ "embedding_cost_gate_usd: 0",+ "",+ ].join("\n"),+ );+}, 20_000);++afterEach(async () => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+ await server.close();+}, 20_000);++async function indexWithEmbeddings() {+ const indexed = await runCli(["search", "index", "--embeddings"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(indexed.returncode).toBe(0);+}++describe("o2b search --explain: overlapped lane timing", () => {+ test("the payload carries an integer overlapped_ms and the transcript prints the line", async () => {+ await indexWithEmbeddings();+ // A slow round trip gives the keyword lanes something to hide.+ server.setHandler((req) => embeddingsBody(req, 150));++ const explained = await runCli(["search", QUERY, "--json", "--limit", "5", "--explain"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(explained.returncode).toBe(0);+ const payload = JSON.parse(explained.stdout) as {+ overlapped_ms?: number;+ semantic_attempted?: boolean;+ };+ // The arithmetic itself is pinned exactly at the unit level+ // (overlappedEmbedMs, hybrid-overlap.test.ts): a real overlapped run on+ // a small corpus can legitimately measure zero - the synchronous FTS+ // lane finishes inside the clock resolution - so this integration pin+ // holds the wiring to a sane integer value and leaves the exactness to+ // the pure function.+ expect(payload.overlapped_ms).toBeGreaterThanOrEqual(0);+ expect(Number.isInteger(payload.overlapped_ms)).toBe(true);+ expect(payload.semantic_attempted).toBe(true);++ const human = await runCli(["search", QUERY, "--limit", "5", "--explain"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(human.returncode).toBe(0);+ expect(human.stdout).toContain("overlapped:");+ }, 30_000);++ test("without --explain the payload carries no timing keys", async () => {+ await indexWithEmbeddings();+ server.setHandler((req) => embeddingsBody(req, 150));++ const plain = await runCli(["search", QUERY, "--json", "--limit", "5"], {+ env: { OPEN_SECOND_BRAIN_CONFIG: configPath },+ });+ expect(plain.returncode).toBe(0);+ const payload = JSON.parse(plain.stdout) as Record<string, unknown>;+ // Off means absent: the timing never reaches the outcome, so the+ // flag-off payload is byte-identical to the pre-change one.+ expect("overlapped_ms" in payload).toBe(false);+ expect("semantic_attempted" in payload).toBe(false);+ }, 30_000);++ test("a keyword-only run reports zero overlap with the lane not attempted", async () => {+ await indexWithEmbeddings();+ server.setHandler((req) => embeddingsBody(req, 150));++ const explained = await runCli(+ ["search", QUERY, "--json", "--limit", "5", "--explain", "--keyword-only"],+ { env: { OPEN_SECOND_BRAIN_CONFIG: configPath } },+ );+ expect(explained.returncode).toBe(0);+ const payload = JSON.parse(explained.stdout) as {+ overlapped_ms?: number;+ semantic_attempted?: boolean;+ };+ expect(payload.overlapped_ms).toBe(0);+ expect(payload.semantic_attempted).toBe(false);+ }, 30_000);+});diff --git a/tests/core/bench/phases.test.ts b/tests/core/bench/phases.test.tsindex 69ffa4aa..c081a78c 100644--- a/tests/core/bench/phases.test.ts+++ b/tests/core/bench/phases.test.ts@@ -507,4 +507,32 @@ describe("the embeddings capability path", () => {writeFileSync(cachePath, `${JSON.stringify({ entries: {} }, null, 2)}\n`, "utf8");await expect(run()).rejects.toThrow(/schema 'undefined'/);}, 30_000);++ test("the latency family reports overlapped_ms when questions attempted the semantic lane", async () => {+ const report = await runMemoryBench({+ fixture: SEMANTIC_FIXTURE,+ runsDir,+ capabilities: CAPABILITIES,+ pinnedKnobs: SEMANTIC_PIN,+ });+ expect(report.quality.passed).toBe(report.quality.total);+ const semanticQuestions = report.questions.filter((q) => q.category === "single_hop");+ expect(semanticQuestions.length).toBeGreaterThan(0);+ for (const question of semanticQuestions) {+ expect(question.overlapped_ms).toBeGreaterThanOrEqual(0);+ expect(Number.isInteger(question.overlapped_ms)).toBe(true);+ }+ expect(report.latency_ms.overlapped_ms).toBeGreaterThanOrEqual(0);+ }, 30_000);++ test("a keyword-only run adds no overlapped_ms keys to the report", async () => {+ const report = await runMemoryBench({ fixture: FIXTURE, runsDir });+ // No question attempted the semantic lane, so no question carries the+ // figure - the latency block keeps exactly the keys it always had.+ expect(report.questions.every((q) => q.overlapped_ms === undefined)).toBe(true);+ expect(JSON.parse(JSON.stringify(report.latency_ms))).toEqual(+ expect.objectContaining({ avg: expect.any(Number), max: expect.any(Number) }),+ );+ expect(Object.keys(report.latency_ms).toSorted()).toEqual(["avg", "max"]);+ }, 30_000);});diff --git a/tests/core/bench/standing-instruction.test.ts b/tests/core/bench/standing-instruction.test.tsnew file mode 100644index 00000000..2bf94498--- /dev/null+++ b/tests/core/bench/standing-instruction.test.ts@@ -0,0 +1,403 @@+/**+ * The `standing_instruction` bench category (recall-lane-latency):+ * the standing-instruction facet lane measured end to end over a+ * disposable fixture vault.+ *+ * The category's retrieve branch drives the shipped `standing_query`+ * attention action with the facet option on - the provenance join, not+ * the search pipeline, is the seam under test - and `evaluateQuestion`+ * asserts the preference id plus its origin turn. Reports without the+ * category serialize byte-identically, the default knob pin is+ * untouched, and the facet-lane pin marker is part of the run identity:+ * a resumed run whose pin changed fails the guard instead of mixing+ * identities.+ *+ * The fixture vault files below hand-write the same facts the live+ * writers emit, in a YAML form `parsePreference`/`parseSignal` and the+ * attention-flow reader `listAttentionFlows` read to identical values -+ * the bytes are not the writers' own output (the writers quote+ * `#`-bearing scalars and emit inline arrays where the fixture writes+ * unquoted scalars and block lists). The bench materializes its vault+ * from raw fixture notes, so the parsers are the seam under+ * measurement; the producer formats themselves are pinned at the unit+ * layer (tests/core/brain/standing-instruction-facets.test.ts)+ * against the live writers.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, readFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { parseBenchFixture } from "../../../src/core/bench/fixture.ts";+import { runMemoryBench } from "../../../src/core/bench/phases.ts";+import {+ DEFAULT_BENCH_KNOB_PIN,+ type BenchFixture,+ type BenchPinnedKnobs,+} from "../../../src/core/bench/types.ts";++let runsDir: string;++beforeEach(() => {+ runsDir = mkdtempSync(join(tmpdir(), "o2b-bench-standing-"));+});++afterEach(() => {+ rmSync(runsDir, { recursive: true, force: true });+});++/**+ * The vault files the surfacing lane reads, in the shapes the live+ * writers emit. The import-session signal carries the `<basename>#<turn>`+ * ref (kind `turn`); the auto-extract signal carries the bare+ * `session:<id>` ref that resolves to nothing.+ */+const FLOW_MD = [+ "---",+ "kind: brain-attention-flow",+ "id: open-loops",+ "title: Open loops",+ "actions:",+ " - standing_query",+ "standing_queries:",+ " - testing",+ "status: active",+ "---",+ "",+ "# Open loops",+ "",+].join("\n");++function preferenceMd(slug: string, topic: string, principle: string, evidence: string): string {+ return [+ "---",+ "kind: brain-preference",+ `id: pref-${slug}`,+ "created_at: 2026-06-01T00:00:00Z",+ "_confirmed_at: 2026-06-02T00:00:00Z",+ "unconfirmed_until: 2026-07-01T00:00:00Z",+ "tags:",+ " - bench",+ `topic: ${topic}`,+ "_status: confirmed",+ `principle: ${principle}`,+ "_evidenced_by:",+ ` - "[[${evidence}]]"`,+ "_applied_count: 0",+ "_violated_count: 0",+ "_last_evidence_at: null",+ "_confidence: low",+ "_confidence_value: null",+ "pinned: false",+ "scope: testing",+ "---",+ "",+ principle,+ "",+ ].join("\n");+}++function signalMd(slug: string, topic: string, principle: string, sessionRef: string): string {+ return [+ "---",+ "kind: brain-signal",+ `id: ${slug}`,+ "created_at: 2026-06-01T00:00:00Z",+ "tags:",+ " - bench",+ `topic: ${topic}`,+ "signal: positive",+ "agent: bench",+ `principle: ${principle}`,+ "scope: testing",+ `session_ref: ${sessionRef}`,+ "---",+ "",+ "recorded preference signal",+ "",+ ].join("\n");+}++/** Built per test, so a fixture parse refusal fails one test, not the file. */+function standingFixture(): BenchFixture {+ return parseBenchFixture({+ name: "standing-instruction-bench",+ notes: STANDING_NOTES,+ continuity: [],+ questions: [+ {+ id: "q-standing",+ category: "standing_instruction",+ // The facet key (the preference id) that must be delivered ...+ expected_ids: ["pref:pref-tabs"],+ // ... and the full annotation that must close that preference's+ // item: compared exactly, closing bracket included, so a+ // near-miss origin (turn-2 vs turn-20) and a session-annotated+ // origin both fail by name.+ expected_text: "earliest cited turn bench-session.json#turn-2]",+ },+ {+ // The auto-extract shape resolves to no origin, so its facet is+ // honestly absent and the expectation fails by name.+ id: "q-standing-unresolved",+ category: "standing_instruction",+ expected_ids: ["pref:pref-commits"],+ },+ ],+ });+}++/** The standing category's fixture vault files, shared with the mixed fixture. */+const STANDING_NOTES: BenchFixture["notes"] = [+ {+ path: "Brain/attention/flows/open-loops.md",+ body: FLOW_MD,+ },+ {+ path: "Brain/preferences/pref-tabs.md",+ body: preferenceMd("tabs", "tabs", "Use tabs for indentation", "sig-bench-turn"),+ },+ {+ path: "Brain/preferences/pref-commits.md",+ body: preferenceMd("commits", "commits", "Prefer small commits", "sig-bench-bare"),+ },+ {+ path: "Brain/inbox/sig-bench-turn.md",+ body: signalMd(+ "sig-bench-turn",+ "tabs",+ "use tabs for indentation",+ "bench-session.json#turn-2",+ ),+ },+ {+ path: "Brain/inbox/sig-bench-bare.md",+ body: signalMd("sig-bench-bare", "commits", "prefer small commits", "session:conv-9"),+ },+];++/**+ * A retrieval question alongside the facet category: the control that+ * proves the headline counts the retrieval question alone.+ */+function mixedFixture(): BenchFixture {+ return parseBenchFixture({+ name: "standing-headline-mixed",+ notes: [+ {+ path: "Brain/notes/coffee.md",+ body: "# Coffee\n\nThe coffee preference is a flat white every morning.\n",+ },+ ...STANDING_NOTES,+ ],+ continuity: [],+ questions: [+ {+ id: "q-coffee",+ category: "single_hop",+ query: "coffee preference",+ top_k: 5,+ expected_paths: ["Brain/notes/coffee.md"],+ },+ {+ id: "q-standing",+ category: "standing_instruction",+ // A deliberately wrong origin: the question fails, and the+ // headline must not move because of it.+ expected_ids: ["pref:pref-tabs"],+ expected_text: "earliest cited turn nowhere.json#turn-9]",+ },+ {+ id: "q-standing-unresolved",+ category: "standing_instruction",+ expected_ids: ["pref:pref-commits"],+ },+ ],+ });+}++/** A keyword-only fixture: no standing_instruction questions anywhere. */+function keywordFixture(): BenchFixture {+ return parseBenchFixture({+ name: "standing-keyword-control",+ notes: [+ {+ path: "Brain/notes/coffee.md",+ body: "# Coffee\n\nThe coffee preference is a flat white every morning.\n",+ },+ ],+ continuity: [],+ questions: [+ {+ id: "q-coffee",+ category: "single_hop",+ query: "coffee preference",+ top_k: 5,+ expected_paths: ["Brain/notes/coffee.md"],+ },+ ],+ });+}++describe("bench standing_instruction category", () => {+ test("reports without the new category serialize byte-identically", async () => {+ const report = await runMemoryBench({ fixture: keywordFixture(), runsDir });+ const serialized = JSON.stringify(report, null, 2);+ // No standing_instruction key anywhere: the category is absent from+ // the quality buckets, the questions and every other family, so a+ // pre-category consumer reads the same bytes.+ expect(serialized).not.toContain("standing_instruction");+ expect(Object.keys(report)).toEqual([+ "schema",+ "run_id",+ "fixture",+ "fixture_hash",+ "created_at",+ "quality",+ "latency_ms",+ "context_cost",+ "failure_modes",+ "judge",+ "questions",+ ]);+ expect(Object.keys(report.quality.by_category)).toEqual(["single_hop"]);+ expect(report.quality.passed).toBe(report.quality.total);+ }, 30_000);++ test("the standing_instruction question asserts the preference id plus origin turn", async () => {+ const report = await runMemoryBench({ fixture: standingFixture(), runsDir });+ expect(report.quality.by_category["standing_instruction"]).toEqual({ passed: 1, total: 2 });+ const passing = report.questions.find((question) => question.id === "q-standing")!;+ expect(passing.category).toBe("standing_instruction");+ expect(passing.pass).toBe(true);+ // A standing result adds no metric fields: its shape is the shared+ // id/category/latency/pass core, nothing else.+ expect(Object.keys(passing).toSorted()).toEqual(["category", "id", "latency_ms", "pass"]);+ }, 30_000);++ test("an unresolved preference fails its question by name", async () => {+ const report = await runMemoryBench({ fixture: standingFixture(), runsDir });+ const unresolved = report.questions.find(+ (question) => question.id === "q-standing-unresolved",+ )!;+ expect(unresolved.pass).toBe(false);+ // The bare auto-extract ref resolves to no origin, so the facet is+ // absent - a named failure, never an implied provenance.+ expect(unresolved.failure).toContain("pref:pref-commits");+ }, 30_000);++ test("a near-miss origin never satisfies the expectation, closing bracket included", async () => {+ // The surfaced annotation ends ...#t10]; the expectation names+ // ...#t1 - a substring the old includes() check accepted as a false+ // positive. Tied to the expected preference's item and compared+ // exactly against the annotation with its closing bracket, it fails+ // by name instead.+ const nearMiss = parseBenchFixture({+ name: "standing-near-miss",+ notes: [+ ...STANDING_NOTES.filter((note) => note.path !== "Brain/inbox/sig-bench-turn.md"),+ {+ path: "Brain/inbox/sig-bench-turn.md",+ body: signalMd(+ "sig-bench-turn",+ "tabs",+ "use tabs for indentation",+ "bench-session.json#t10",+ ),+ },+ ],+ continuity: [],+ questions: [+ {+ id: "q-standing",+ category: "standing_instruction",+ expected_ids: ["pref:pref-tabs"],+ expected_text: "earliest cited turn bench-session.json#t1",+ },+ ],+ });+ const report = await runMemoryBench({ fixture: nearMiss, runsDir });+ const failing = report.questions.find((question) => question.id === "q-standing")!;+ expect(failing.pass).toBe(false);+ expect(failing.failure).toContain("earliest cited turn bench-session.json#t1");+ }, 30_000);++ test("the default knob pin serializes byte-for-byte unchanged", () => {+ expect(JSON.stringify(DEFAULT_BENCH_KNOB_PIN)).toBe('{"semantic":{"enabled":false}}');+ });++ test("standing_instruction questions never shift the headline pass rate", async () => {+ const report = await runMemoryBench({ fixture: mixedFixture(), runsDir });+ // The retrieval question alone is the headline: both facet questions+ // fail, yet passed/total/pass_rate count only q-coffee. Adding facet+ // questions to a fixture cannot move the rate a baseline compares.+ expect(report.quality.passed).toBe(1);+ expect(report.quality.total).toBe(1);+ expect(report.quality.pass_rate).toBe(1);+ // The category keeps its failures where they belong: its own named+ // bucket, which the human line prints.+ expect(report.quality.by_category["single_hop"]).toEqual({ passed: 1, total: 1 });+ expect(report.quality.by_category["standing_instruction"]).toEqual({ passed: 0, total: 2 });+ const failing = report.questions.find((question) => question.id === "q-standing")!;+ expect(failing.pass).toBe(false);+ }, 30_000);++ test("a pin of false runs the lane off and the facet expectations fail by name", async () => {+ const laneOffPin: BenchPinnedKnobs = {+ semantic: { enabled: false },+ standingInstructionFacets: false,+ };+ const report = await runMemoryBench({+ fixture: standingFixture(),+ runsDir,+ pinnedKnobs: laneOffPin,+ });+ // The plain lane emitted only the legacy strings, so both facet-key+ // expectations fail, naming the missing facet keys.+ expect(report.quality.by_category["standing_instruction"]).toEqual({ passed: 0, total: 2 });+ const failing = report.questions.find((question) => question.id === "q-standing")!;+ expect(failing.pass).toBe(false);+ expect(failing.failure).toContain("pref:pref-tabs");+ // The persisted raw record proves the off arm ran: plain legacy+ // strings, no annotations, and no facet ids at all.+ const raw = JSON.parse(+ readFileSync(join(runsDir, report.run_id, "results", "q-standing.json"), "utf8"),+ ) as { standing_items?: string[]; facet_ids?: string[] };+ expect(raw.standing_items).toEqual(["pref-commits (testing)", "pref-tabs (testing)"]);+ expect(raw.facet_ids).toEqual([]);+ }, 30_000);++ test("the facet-lane pin marker is part of the run identity", async () => {+ const laneOnPin: BenchPinnedKnobs = {+ semantic: { enabled: false },+ standingInstructionFacets: true,+ };+ const first = await runMemoryBench({+ fixture: standingFixture(),+ runsDir,+ pinnedKnobs: laneOnPin,+ });+ // The marker rides the checkpoint verbatim ...+ const checkpoint = JSON.parse(+ readFileSync(join(runsDir, first.run_id, "checkpoint.json"), "utf8"),+ ) as { pinned_knobs?: Record<string, unknown> };+ expect(checkpoint.pinned_knobs).toEqual({+ semantic: { enabled: false },+ standingInstructionFacets: true,+ });+ // ... so a resumed run under the plain pin fails the identity guard ...+ await expect(+ runMemoryBench({ fixture: standingFixture(), runsDir, resume: first.run_id }),+ ).rejects.toThrow(/run identity mismatch/);+ // ... and resuming under the same marker reuses the run.+ const resumed = await runMemoryBench({+ fixture: standingFixture(),+ runsDir,+ resume: first.run_id,+ pinnedKnobs: laneOnPin,+ });+ expect(resumed.run_id).toBe(first.run_id);+ expect(resumed.quality).toEqual(first.quality);+ }, 30_000);+});diff --git a/tests/core/brain/context-pack-packing.test.ts b/tests/core/brain/context-pack-packing.test.tsnew file mode 100644index 00000000..ee289e85--- /dev/null+++ b/tests/core/brain/context-pack-packing.test.ts@@ -0,0 +1,508 @@+/**+ * Opt-in packing policy on the context-pack lane (recall-lane-latency+ * wave).+ *+ * Inside one pack, the preference-derived candidates and the page+ * candidates (the retired directory) share the single `maxTokens`+ * budget. A caller that names a `packingPolicy` decides which lane+ * claims that budget first: `facts_first` (the distilled preference+ * layer) or `retired_first` (the pages lane, the retired preferences).+ * Every claim here:+ *+ * 1. An ABSENT `packingPolicy` is byte-identical to the unified+ * comparator pack - no partition runs, no `packing` key exists,+ * and a retired core page still outranks a newer supporting+ * preference across lanes.+ * 2. With a policy, the report carries both lane spends, the reserved+ * attention charge and the policy name, and the lane order decides+ * emission order; a refused pack has no report at all.+ * 3. Skip reasons keep the shipped vocabulary; lane overflow is an+ * `over-budget` skip, never a new reason.+ * 4. A policy against an empty lane is refused BY NAME before any+ * receipt or telemetry is written - never a silent no-op.+ * 5. The zero-budget early return and the char-budget path agree+ * with the one finalize funnel: the packing accounting is present+ * on every return path once the policy is named.+ * 6. The synthesized attention block stays a reserved charge against+ * the shared total, drawn by neither lane.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ ContextPackPackingLaneEmptyError,+ packContext,+ type ContextPackReport,+} from "../../../src/core/brain/context-pack.ts";+import { listAttentionFlows } from "../../../src/core/brain/attention-flows.ts";+import {+ getContextReceipt,+ listContextReceipts,+} from "../../../src/core/brain/context-receipts.ts";+import { listMetrics } from "../../../src/core/brain/metrics.ts";+import {+ canonicalSegment,+ deterministicPrefix,+ PROMPT_PREFIX_SURFACE,+} from "../../../src/core/brain/prompt-prefix.ts";+import { RECALL_CHANNEL, listRecallTelemetry } from "../../../src/core/brain/recall-telemetry.ts";+import { isPackingPolicy } from "../../../src/core/brain/recall-budget.ts";+import { brainConfigPath } from "../../../src/core/brain/paths.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-pack-packing-"));+ mkdirSync(join(vault, "Brain", "preferences"), { recursive: true });+ mkdirSync(join(vault, "Brain", "retired"), { recursive: true });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++function writePref(+ slug: string,+ fields: { tier: string; created_at: string; principle?: string; owner?: string },+): void {+ writeFileSync(+ join(vault, "Brain", "preferences", `pref-${slug}.md`),+ [+ "---",+ `id: pref-${slug}`,+ `topic: pref ${slug} topic`,+ `principle: ${fields.principle ?? `pref ${slug} principle`}`,+ `tier: ${fields.tier}`,+ `created_at: ${fields.created_at}`,+ ...(fields.owner !== undefined ? [`owner: ${fields.owner}`] : []),+ "---",+ "",+ `body of pref ${slug}`,+ ].join("\n"),+ );+}++function writeRetired(slug: string, fields: { tier: string; created_at: string }): void {+ writeFileSync(+ join(vault, "Brain", "retired", `ret-${slug}.md`),+ [+ "---",+ `id: ret-${slug}`,+ `topic: ret ${slug} topic`,+ "principle: retired page principle",+ `tier: ${fields.tier}`,+ `created_at: ${fields.created_at}`,+ "---",+ "",+ `body of ret ${slug}`,+ ].join("\n"),+ );+}++/** One preference lane pair and one retired page, over distinct tiers and dates. */+function writeMixedVault(): void {+ writePref("alpha", { tier: "core", created_at: "2026-03-01T00:00:00Z" });+ writePref("beta", { tier: "supporting", created_at: "2026-02-01T00:00:00Z" });+ writeRetired("old", { tier: "core", created_at: "2026-01-01T00:00:00Z" });+}++function tokensOf(report: ContextPackReport, id: string): number {+ const item = report.items.find((i) => i.id === id);+ if (item) return item.tokens;+ const skip = report.skipped.find((s) => s.id === id);+ if (skip) return skip.tokens;+ throw new Error(`fixture page not in report: ${id}`);+}++interface PackingShape {+ policy: string;+ reservedTokens: number;+ pages: { allocation: number; spent: number; candidates: number; items: number };+ facts: { allocation: number; spent: number; candidates: number; items: number };+}++function packingOf(report: ContextPackReport): PackingShape {+ return report.packing as unknown as PackingShape;+}++describe("context pack packing policy", () => {+ test("an absent packingPolicy keeps the unified comparator pack byte-identical", () => {+ writeMixedVault();+ const r = packContext(vault, { maxTokens: 10_000 });+ // Unified order: tier first, recency inside the tier - the retired+ // core page sits between the two preferences exactly as before any+ // partition existed, and no packing key exists at all.+ expect(r.items.map((i) => i.id)).toEqual(["pref-alpha", "ret-old", "pref-beta"]);+ expect(JSON.stringify(r)).not.toContain('"packing"');+ expect(Object.hasOwn(r, "packing")).toBe(false);+ });++ test("facts_first lets the facts lane claim first and the report carries both spends", () => {+ writeMixedVault();+ const unified = packContext(vault, { maxTokens: 10_000 });+ const r = packContext(vault, { maxTokens: 10_000, packingPolicy: "facts_first" });+ // Same population, different order: the preference lane emits as a+ // block before the retired page, which the unified order slotted by+ // tier alone.+ expect(r.items.map((i) => i.id)).toEqual(["pref-alpha", "pref-beta", "ret-old"]);+ expect(r.skipped).toEqual([]);+ const packing = packingOf(r);+ expect(packing.policy).toBe("facts_first");+ // The report names the reserved charge so the lane accounting is+ // whole: zero here, because no attention block was requested.+ expect(packing.reservedTokens).toBe(0);+ expect(packing.facts.spent).toBe(+ tokensOf(unified, "pref-alpha") + tokensOf(unified, "pref-beta"),+ );+ expect(packing.pages.spent).toBe(tokensOf(unified, "ret-old"));+ expect(packing.facts.items).toBe(2);+ expect(packing.pages.items).toBe(1);+ // Conservation, reserved charge (zero here) included.+ expect(packing.facts.allocation + packing.pages.allocation).toBe(10_000);+ expect(packing.facts.spent + packing.pages.spent).toBe(r.tokensUsed);+ });++ test("retired_first hands the first claim to the retired pages lane", () => {+ writeMixedVault();+ const unified = packContext(vault, { maxTokens: 10_000 });+ const retTokens = tokensOf(unified, "ret-old");+ const r = packContext(vault, { maxTokens: retTokens, packingPolicy: "retired_first" });+ expect(r.items.map((i) => i.id)).toEqual(["ret-old"]);+ expect(r.skipped.map((s) => s.reason)).toEqual(["over-budget", "over-budget"]);+ const packing = packingOf(r);+ expect(packing.policy).toBe("retired_first");+ expect(packing.pages.allocation).toBe(retTokens);+ expect(packing.pages.spent).toBe(retTokens);+ expect(packing.facts.allocation).toBe(0);+ expect(packing.facts.spent).toBe(0);+ });++ test("facts_first on a tight budget spends it on the facts lane", () => {+ writeMixedVault();+ const unified = packContext(vault, { maxTokens: 10_000 });+ const factsDemand = tokensOf(unified, "pref-alpha") + tokensOf(unified, "pref-beta");+ const r = packContext(vault, { maxTokens: factsDemand, packingPolicy: "facts_first" });+ expect(r.items.map((i) => i.id)).toEqual(["pref-alpha", "pref-beta"]);+ expect(r.skipped.map((s) => s.reason)).toEqual(["over-budget"]);+ const packing = packingOf(r);+ expect(packing.facts.allocation).toBe(factsDemand);+ expect(packing.pages.allocation).toBe(0);+ });++ test("skip reasons keep the shipped vocabulary under a policy", () => {+ // Both lanes hold a query match, so the pack runs; the two misses+ // filter-miss per lane and the losing lane's overflow is an+ // over-budget. No new reason exists.+ writeFileSync(+ join(vault, "Brain", "preferences", "pref-a.md"),+ [+ "---",+ "id: pref-a",+ "topic: needle in the facts",+ "principle: facts principle",+ "tier: core",+ "created_at: 2026-03-01T00:00:00Z",+ "---",+ "",+ "body",+ ].join("\n"),+ );+ writeFileSync(+ join(vault, "Brain", "preferences", "pref-b.md"),+ [+ "---",+ "id: pref-b",+ "topic: other facts",+ "principle: more facts principle",+ "tier: core",+ "created_at: 2026-03-02T00:00:00Z",+ "---",+ "",+ "body",+ ].join("\n"),+ );+ writeFileSync(+ join(vault, "Brain", "retired", "ret-a.md"),+ [+ "---",+ "id: ret-a",+ "topic: needle in the pages",+ "principle: pages principle",+ "tier: core",+ "created_at: 2026-01-01T00:00:00Z",+ "---",+ "",+ "body",+ ].join("\n"),+ );+ writeFileSync(+ join(vault, "Brain", "retired", "ret-b.md"),+ [+ "---",+ "id: ret-b",+ "topic: unrelated page",+ "principle: unrelated principle",+ "tier: core",+ "created_at: 2026-01-02T00:00:00Z",+ "---",+ "",+ "body",+ ].join("\n"),+ );+ const unified = packContext(vault, { maxTokens: 10_000, query: "needle" });+ const needleTokens = tokensOf(unified, "pref-a");+ const r = packContext(vault, {+ maxTokens: needleTokens,+ packingPolicy: "facts_first",+ query: "needle",+ });+ expect(r.items.map((i) => i.id)).toEqual(["pref-a"]);+ const reasons = new Set(r.skipped.map((s) => s.reason));+ expect(reasons.has("filter-miss")).toBe(true);+ expect(reasons.has("over-budget")).toBe(true);+ for (const reason of reasons) {+ expect(["filter-miss", "over-budget", "over-char-budget", "guard-blocked"]).toContain(reason);+ }+ });++ test("a policy against an empty lane is refused by name and writes no receipt", () => {+ writePref("alpha", { tier: "core", created_at: "2026-03-01T00:00:00Z" });+ let caught: unknown = null;+ try {+ packContext(vault, {+ maxTokens: 10_000,+ packingPolicy: "facts_first",+ receipt: { trigger: "context_pack", host: "packing-test" },+ });+ } catch (e) {+ caught = e;+ }+ expect(caught).toBeInstanceOf(ContextPackPackingLaneEmptyError);+ const err = caught as Error;+ expect(err.message).toContain("pages");+ expect(err.message).toContain("facts_first");+ // The refusal ran before the funnel: no delivery record exists.+ expect(listContextReceipts(vault)).toHaveLength(0);+ });++ test("the refusal names the empty lane whichever lane it is", () => {+ writeRetired("old", { tier: "core", created_at: "2026-01-01T00:00:00Z" });+ expect(() => packContext(vault, { maxTokens: 10_000, packingPolicy: "retired_first" })).toThrow(+ /facts lane is empty/,+ );+ });++ test("a named policy on a zero budget is refused by name, not answered with zeroed accounting", () => {+ writeMixedVault();+ // A zeroed `packing` object would answer a policy the caller named+ // with a delivery that never happened - the same silent no-op the+ // empty-lane refusal exists for.+ expect(() => packContext(vault, { maxTokens: 0, packingPolicy: "facts_first" })).toThrow(+ ContextPackPackingLaneEmptyError,+ );+ expect(() => packContext(vault, { maxTokens: 0, packingPolicy: "facts_first" })).toThrow(+ /facts_first needs a positive token budget/,+ );+ // The refusal runs before the funnel: no delivery record exists.+ expect(listContextReceipts(vault)).toHaveLength(0);+ // Without the policy the zero-budget short-circuit is unchanged.+ const r = packContext(vault, { maxTokens: 0 });+ expect(r.items).toEqual([]);+ expect(r.tokensUsed).toBe(0);+ expect("packing" in r).toBe(false);+ });++ test("a reserved charge that consumes the whole budget is refused by name, not answered with zeroed accounting", () => {+ writeMixedVault();+ listAttentionFlows(vault); // bootstraps the default flow fixtures+ const charged = packContext(vault, {+ maxTokens: 10_000,+ packingPolicy: "facts_first",+ attentionFlowIds: ["open-loops"],+ });+ const attentionTokens = charged.items.find((i) => i.id === "attention-flows")!.tokens;+ expect(attentionTokens).toBeGreaterThan(0);+ // A reserved charge equal to or over the total leaves neither lane+ // anything to draw - the same silent delivery the zero-budget refusal+ // exists for, reached through the reserved charge instead of a zero+ // maxTokens. Both lanes hold candidates, so the empty-lane refusal+ // does not fire either; the pack must fail by name.+ for (const total of [attentionTokens, attentionTokens - 1]) {+ let caught: unknown = null;+ try {+ packContext(vault, {+ maxTokens: total,+ packingPolicy: "facts_first",+ attentionFlowIds: ["open-loops"],+ receipt: { trigger: "context_pack", host: "packing-test" },+ });+ } catch (e) {+ caught = e;+ }+ expect(caught).toBeInstanceOf(ContextPackPackingLaneEmptyError);+ const err = caught as Error;+ // The refusal says what the caller must know: the policy, the+ // reserved charge and the total it consumed.+ expect(err.message).toContain("facts_first");+ expect(err.message).toContain(String(attentionTokens));+ expect(err.message).toContain(String(total));+ }+ // A refused pack has no accounting to read: the throw precedes the+ // funnel, so no `packing` report exists for either total - and no+ // receipt or telemetry record was written either.+ expect(listContextReceipts(vault)).toHaveLength(0);+ expect(listRecallTelemetry(vault, { mode: "context_pack" })).toHaveLength(0);+ });++ test("the refusal names both lanes when both are empty", () => {+ // An empty vault: neither population has a candidate in reach.+ expect(() => packContext(vault, { maxTokens: 10_000, packingPolicy: "facts_first" })).toThrow(+ /the facts and pages lanes are empty/,+ );+ });++ test("the owner-scope warning survives the policy char-budget overflow", () => {+ // Owner scope in warn mode with one other-agent memory hidden: the+ // warning rides extraWarnings, and the char-budget early return used+ // to pass an empty array there, dropping it exactly when the caller+ // also asked for the packing policy.+ writeFileSync(+ brainConfigPath(vault),+ "schema_version: 1\nintegrity:\n owner_scope_delivery: warn\n",+ );+ writePref("alpha", { tier: "core", created_at: "2026-03-01T00:00:00Z", owner: "agent-a" });+ writePref("beta", { tier: "supporting", created_at: "2026-02-01T00:00:00Z", owner: "agent-b" });+ writeRetired("old", { tier: "core", created_at: "2026-01-01T00:00:00Z" });+ const r = packContext(vault, {+ maxTokens: 10_000,+ packingPolicy: "facts_first",+ agentScope: "agent-a",+ maxTotalChars: 40,+ });+ expect(r.warnings?.some((w) => w.startsWith('owner scope "agent-a"'))).toBe(true);+ // The dropped tail is still the char-budget drop, by name.+ expect(r.skipped.some((s) => s.reason === "over-char-budget")).toBe(true);+ });++ test("the char-budget path agrees with the funnel", () => {+ writeMixedVault();+ const unified = packContext(vault, { maxTokens: 10_000 });+ const retTokens = tokensOf(unified, "ret-old");+ // Keep every body, then cap the total so exactly the lowest-priority+ // final item (the retired page, emitted last) drops.+ const keptChars = [...`${"body of pref alpha"}${"body of pref beta"}`].length;+ const r = packContext(vault, {+ maxTokens: 10_000,+ packingPolicy: "facts_first",+ maxTotalChars: keptChars,+ });+ expect(r.items.map((i) => i.id)).toEqual(["pref-alpha", "pref-beta"]);+ expect(r.skipped.map((s) => s.reason)).toEqual(["over-char-budget"]);+ const packing = packingOf(r);+ expect(packing.pages.items).toBe(0);+ expect(packing.pages.spent).toBe(0);+ expect(packing.facts.spent).toBe(r.tokensUsed);+ expect(packing.facts.allocation + packing.pages.allocation).toBe(10_000);+ // The dropped page's own token cost left with it.+ expect(r.tokensUsed).toBeLessThan(unified.tokensUsed);+ expect(retTokens).toBeGreaterThan(0);+ });++ test("the attention block stays a reserved charge neither lane draws", () => {+ writeMixedVault();+ listAttentionFlows(vault); // bootstraps the default flow fixtures+ const r = packContext(vault, {+ maxTokens: 10_000,+ packingPolicy: "facts_first",+ attentionFlowIds: ["open-loops"],+ });+ expect(r.items[0]!.id).toBe("attention-flows");+ const attentionTokens = r.items[0]!.tokens;+ const packing = packingOf(r);+ // The report carries the reserved charge itself (external review L3):+ // the caller can see what the two lane allocations were reduced by.+ expect(packing.reservedTokens).toBe(attentionTokens);+ // Conservation, reserved charge included: the two lane allocations+ // plus the attention block's charge fill the total exactly.+ expect(packing.facts.allocation + packing.pages.allocation + attentionTokens).toBe(10_000);+ expect(packing.facts.spent + packing.pages.spent + attentionTokens).toBe(r.tokensUsed);+ });++ test("receipt budget metadata names the policy only when one was named", () => {+ writeMixedVault();+ const unnamed = packContext(vault, {+ maxTokens: 10_000,+ receipt: { trigger: "context_pack", host: "packing-test" },+ });+ const unnamedRecord = getContextReceipt(vault, unnamed.receiptId!)!;+ const unnamedBudget = unnamedRecord.payload["budget"] as Record<string, unknown>;+ expect(unnamedBudget["max_tokens"]).toBe(10_000);+ expect(Object.hasOwn(unnamedBudget, "packing_policy")).toBe(false);+ const named = packContext(vault, {+ maxTokens: 10_000,+ packingPolicy: "facts_first",+ receipt: { trigger: "context_pack", host: "packing-test" },+ });+ const namedRecord = getContextReceipt(vault, named.receiptId!)!;+ const namedBudget = namedRecord.payload["budget"] as Record<string, unknown>;+ expect(namedBudget["packing_policy"]).toBe("facts_first");+ });++ test("the synced telemetry carries the policy as a closed enum member", () => {+ writeMixedVault();+ const named = packContext(vault, {+ maxTokens: 10_000,+ packingPolicy: "retired_first",+ telemetry: { host: "packing-test", channel: RECALL_CHANNEL.cli },+ });+ expect(named.telemetryId).toStartWith("ctn_");+ const records = listRecallTelemetry(vault, { mode: "context_pack" });+ expect(records).toHaveLength(1);+ const metadata = records[0]!.payload["metadata"] as Record<string, unknown>;+ // A member of the frozen vocabulary, never a free-form string.+ expect(metadata["packing_policy"]).toBe("retired_first");+ expect(isPackingPolicy(metadata["packing_policy"])).toBe(true);+ });++ test("the prompt-prefix segment gains the policy only when it was named", () => {+ writeMixedVault();+ const plain = packContext(vault, { maxTokens: 10_000, promptPrefix: true });+ const before = listMetrics(vault, { surface: PROMPT_PREFIX_SURFACE });+ expect(before).toHaveLength(1);+ // Exact pre-change bytes for an unnamed caller: the deterministic+ // segment carries no packing_policy key at all.+ const unnamedExpected = deterministicPrefix({+ kind: "context_pack",+ segments: [+ canonicalSegment({ lanes: "false", max_tokens: String(plain.maxTokens), query: "" }),+ ],+ });+ expect(before[0]!.payload["prefix_hash"]).toBe(unnamedExpected.hash);+ const named = packContext(vault, {+ maxTokens: 10_000,+ packingPolicy: "facts_first",+ promptPrefix: true,+ });+ const after = listMetrics(vault, { surface: PROMPT_PREFIX_SURFACE });+ expect(after).toHaveLength(2);+ // The named caller's prefix moves, deterministically, by the policy;+ // both records share one run_at second, so compare the hash pair.+ const namedExpected = deterministicPrefix({+ kind: "context_pack",+ segments: [+ canonicalSegment({+ lanes: "false",+ max_tokens: String(named.maxTokens),+ query: "",+ packing_policy: "facts_first",+ }),+ ],+ });+ const hashes = after.map((r) => r.payload["prefix_hash"]).toSorted();+ expect(hashes).toEqual([namedExpected.hash, unnamedExpected.hash].toSorted());+ expect(namedExpected.hash).not.toBe(unnamedExpected.hash);+ });+});diff --git a/tests/core/brain/preference-provenance.test.ts b/tests/core/brain/preference-provenance.test.tsnew file mode 100644index 00000000..afbd1c3c--- /dev/null+++ b/tests/core/brain/preference-provenance.test.ts@@ -0,0 +1,265 @@+/**+ * Pure origin-join core for standing-instruction provenance (recall lane).+ *+ * `preference-provenance.ts` is a read-time join: it parses a signal's+ * `session_ref` into origin coordinates and resolves a preference's+ * `evidenced_by` wikilinks to those coordinates through an injected+ * lookup. Three producers write `session_ref` today, each with its own+ * shape:+ *+ * - import-session: `<portable basename>#<turn id>` β the ref is+ * composed at src/core/brain/sessions/import.ts:445 from the key+ * `sessionRefIdentity` returns (the session file's basename, never+ * a machine-local path);+ * - capture lifecycle: `session:<id>#<event>` β composed by+ * `sessionReference` at src/core/brain/session-lifecycle.ts:858;+ * the fragment is a hook event NAME (UserPromptSubmit, Stop,+ * SessionEnd), not an event id, so it resolves as SESSION-level+ * provenance (kind `session`) and is labeled as such;+ * - auto-extract: a bare session reference with no fragment β+ * src/core/brain/extract-signals.ts:646 stores the caller's+ * session id verbatim, so no turn is available.+ *+ * The never-guess rule under every case below: a ref whose shape no+ * producer emits yields NO provenance, never a best-effort reading.+ */++import { describe, expect, test } from "bun:test";+import { readFileSync } from "node:fs";++import {+ parseSessionRef,+ resolvePreferenceProvenance,+} from "../../../src/core/brain/preference-provenance.ts";+// The live writer half of the import-session ref: the portable session+// key is exactly what `sessionRefIdentity` returns.+import { sessionRefIdentity } from "../../../src/core/brain/sessions/import.ts";++describe("parseSessionRef: producer shapes", () => {+ test("an import-session ref splits into portable basename and turn id", () => {+ // Live writer shape: `${sessionKey}#${input.turnId}` where the key+ // is `sessionRefIdentity`'s portable basename.+ const key = sessionRefIdentity("/host/only/path/sessions/claude-2026-06-01.jsonl");+ expect(key).toBe("claude-2026-06-01.jsonl");+ const origin = parseSessionRef(`${key}#turn-user-1`);+ expect(origin).toEqual({+ sessionKey: "claude-2026-06-01.jsonl",+ fragment: "turn-user-1",+ kind: "turn",+ });+ });++ test("a capture-lifecycle ref resolves as session-level provenance, labeled as such", () => {+ // Live writer shape: `session:${payload.sessionId}#${payload.event}`.+ // The fragment is the hook event's NAME, not an event id, so every+ // prompt-sourced signal in a session shares the one coordinate: the+ // kind says session, never a turn claim.+ const origin = parseSessionRef("session:00000000-0000-0000-0000-000000000001#UserPromptSubmit");+ expect(origin).toEqual({+ sessionKey: "session:00000000-0000-0000-0000-000000000001",+ fragment: "UserPromptSubmit",+ kind: "session",+ });+ });++ test("the degenerate unknown-session capture ref keeps its verbatim key", () => {+ // session-lifecycle stamps `unknown` when the host payload carries+ // no session id; the shape is still a lifecycle ref.+ const origin = parseSessionRef("session:unknown#SessionEnd");+ expect(origin).toEqual({+ sessionKey: "session:unknown",+ fragment: "SessionEnd",+ kind: "session",+ });+ });++ test("a session-prefixed key that is neither lifecycle form yields no provenance", () => {+ // Import keys are verbatim basenames: a session file named with the+ // lifecycle scheme collides with it, and the ref alone cannot tell+ // the two apart. The join refuses the label instead of guessing one -+ // no provenance, never a wrong kind.+ expect(parseSessionRef("session:my-session.jsonl#turn-user-1")).toBeNull();+ expect(parseSessionRef("session:conv-01#UserPromptSubmit")).toBeNull();+ });++ test("an auto-extract ref carries no fragment and yields no provenance", () => {+ // Live writer shape: the bare caller session id, no `#`.+ expect(parseSessionRef("session:conv-01")).toBeNull();+ });++ test("any ref without a fragment yields no provenance", () => {+ expect(parseSessionRef("claude-minimal.jsonl")).toBeNull();+ });+});++describe("parseSessionRef: malformed shapes are refused, never guessed", () => {+ test("missing and blank refs yield no provenance", () => {+ expect(parseSessionRef("")).toBeNull();+ expect(parseSessionRef(" ")).toBeNull();+ });++ test("an empty key or an empty fragment yields no provenance", () => {+ expect(parseSessionRef("#turn-1")).toBeNull();+ expect(parseSessionRef("claude-minimal.jsonl#")).toBeNull();+ expect(parseSessionRef("session:#UserPromptSubmit")).toBeNull();+ });++ test("a second hash β a shape no producer emits β yields no provenance", () => {+ expect(parseSessionRef("claude-minimal.jsonl#turn-1#extra")).toBeNull();+ expect(parseSessionRef("session:conv-01#UserPromptSubmit#x")).toBeNull();+ });+});++describe("resolvePreferenceProvenance", () => {+ const LOOKUP: Record<string, string | undefined> = {+ "sig-2026-05-16-mocking": "claude-minimal.jsonl#turn-user-1",+ "sig-2026-05-17-lifecycle": "session:00000000-0000-0000-0000-000000000002#UserPromptSubmit",+ "sig-2026-05-18-bare": "session:conv-02",+ "sig-2026-05-19-dup": "claude-minimal.jsonl#turn-user-1",+ };+ const lookup = (signalId: string): string | undefined => LOOKUP[signalId];++ test("resolves both producer kinds in evidence order, earliest cited first", () => {+ const provenance = resolvePreferenceProvenance(+ {+ id: "pref-db-mocking",+ evidenced_by: ["[[sig-2026-05-16-mocking]]", "[[sig-2026-05-17-lifecycle]]"],+ },+ lookup,+ );+ expect(provenance).not.toBeNull();+ expect(provenance?.key).toBe("pref:pref-db-mocking");+ expect(provenance?.preferenceId).toBe("pref-db-mocking");+ expect(provenance?.origins).toEqual([+ { sessionKey: "claude-minimal.jsonl", fragment: "turn-user-1", kind: "turn" },+ {+ sessionKey: "session:00000000-0000-0000-0000-000000000002",+ fragment: "UserPromptSubmit",+ kind: "session",+ },+ ]);+ // The earliest cited origin leads: the preference's evidence order,+ // never re-sorted, and never a chronological claim.+ expect(provenance?.origins[0]).toEqual({+ sessionKey: "claude-minimal.jsonl",+ fragment: "turn-user-1",+ kind: "turn",+ });+ });++ test("evidence whose signal is unknown is skipped without leaving a hole", () => {+ const provenance = resolvePreferenceProvenance(+ {+ id: "pref-partial",+ evidenced_by: ["[[sig-missing]]", "[[sig-2026-05-16-mocking]]"],+ },+ lookup,+ );+ expect(provenance?.origins).toEqual([+ { sessionKey: "claude-minimal.jsonl", fragment: "turn-user-1", kind: "turn" },+ ]);+ });++ test("an existing signal without a session_ref contributes no origin", () => {+ const provenance = resolvePreferenceProvenance(+ { id: "pref-noref", evidenced_by: ["[[sig-no-session-ref]]"] },+ () => undefined,+ );+ expect(provenance).toBeNull();+ });++ test("evidence entries that are not exact wikilinks are never guessed at", () => {+ const provenance = resolvePreferenceProvenance(+ {+ id: "pref-loose",+ evidenced_by: [+ "sig-2026-05-16-mocking", // bare id, no brackets+ "see [[sig-2026-05-16-mocking]] in context", // embedded in prose+ "", // empty entry+ "[[sig-2026-05-16-mocking|alias]]", // alias form: not what writers emit+ "[[sig-2026-05-16-mocking]]", // the one exact form+ ],+ },+ lookup,+ );+ expect(provenance?.origins).toEqual([+ { sessionKey: "claude-minimal.jsonl", fragment: "turn-user-1", kind: "turn" },+ ]);+ });++ test("duplicate origins dedupe to their first position", () => {+ const provenance = resolvePreferenceProvenance(+ {+ id: "pref-dup",+ evidenced_by: [+ "[[sig-2026-05-16-mocking]]",+ "[[sig-2026-05-17-lifecycle]]",+ "[[sig-2026-05-19-dup]]", // resolves to the same ref as the first+ ],+ },+ lookup,+ );+ expect(provenance?.origins).toEqual([+ { sessionKey: "claude-minimal.jsonl", fragment: "turn-user-1", kind: "turn" },+ {+ sessionKey: "session:00000000-0000-0000-0000-000000000002",+ fragment: "UserPromptSubmit",+ kind: "session",+ },+ ]);+ });++ test("ordering follows the preference's own evidence order, never a re-sort", () => {+ const provenance = resolvePreferenceProvenance(+ {+ id: "pref-order",+ evidenced_by: ["[[sig-late-z]]", "[[sig-early-a]]"],+ },+ (signalId) => (signalId === "sig-late-z" ? "zeta.jsonl#t9" : "alpha.jsonl#t1"),+ );+ expect(provenance?.origins).toEqual([+ { sessionKey: "zeta.jsonl", fragment: "t9", kind: "turn" },+ { sessionKey: "alpha.jsonl", fragment: "t1", kind: "turn" },+ ]);+ });++ test("a ref a producer cannot emit yields no provenance even through the lookup", () => {+ const provenance = resolvePreferenceProvenance(+ { id: "pref-weird", evidenced_by: ["[[sig-weird-ref]]"] },+ () => "not-even-wrong#k#v",+ );+ expect(provenance).toBeNull();+ });++ test("nothing resolves to null, not to an empty-hints value", () => {+ expect(+ resolvePreferenceProvenance({ id: "pref-none", evidenced_by: ["[[sig-missing]]"] }, lookup),+ ).toBeNull();+ expect(resolvePreferenceProvenance({ id: "pref-empty", evidenced_by: [] }, lookup)).toBeNull();+ });++ test("the join is deterministic across repeated calls", () => {+ const input = {+ id: "pref-det",+ evidenced_by: ["[[sig-2026-05-17-lifecycle]]", "[[sig-2026-05-16-mocking]]"],+ };+ const first = resolvePreferenceProvenance(input, lookup);+ const second = resolvePreferenceProvenance(input, lookup);+ expect(first).toEqual(second);+ expect(first?.origins[0]?.fragment).toBe("UserPromptSubmit");+ });+});++describe("purity pin", () => {+ test("the module is an import-free leaf: no fs, no path, no vault APIs", () => {+ const source = readFileSync(+ new URL("../../../src/core/brain/preference-provenance.ts", import.meta.url),+ "utf8",+ );+ expect(source).not.toMatch(/^[ \t]*import\b/m);+ expect(source).not.toMatch(/\brequire\s*\(/);+ expect(source).not.toContain("node:fs");+ expect(source).not.toContain("node:path");+ expect(source).not.toContain("vault");+ });+});diff --git a/tests/core/brain/recall-budget.packing.test.ts b/tests/core/brain/recall-budget.packing.test.tsnew file mode 100644index 00000000..1e3a8e94--- /dev/null+++ b/tests/core/brain/recall-budget.packing.test.ts@@ -0,0 +1,286 @@+/**+ * Packing-policy vocabulary and the pure lane-allocation primitive+ * (recall-lane-latency wave).+ *+ * The premise of one shared reading budget over which a packing order+ * reorders is false across surfaces - pages and the facts-side brief+ * are budgeted by two separate surfaces - so the wave first makes the+ * two candidate populations CORRELATABLE through one pure allocation+ * primitive beside `applyCharBudget`, then lets a caller opt into an+ * order over it. Claims pinned here:+ *+ * 1. `PACKING_POLICIES` is a frozen two-member tuple and+ * `isPackingPolicy` narrows exactly like the queryMode guard.+ * 2. `retired_first` lets the pages lane claim the shared budget+ * before the facts lane fills the remainder; `facts_first` claims+ * facts-side first (the delivery order the two surfaces already+ * observe today).+ * 3. A single populated lane is an identity pass: the whole claimable+ * budget goes to the lane that exists.+ * 4. A lane whose demand exceeds its remainder drops its overflow and+ * never drags the other lane's allocation negative.+ * 5. Total conservation: both lane allocations plus the reserved+ * charge equal the total, exactly.+ * 6. Per-lane native caps (chars, note counts) stay lane-internal:+ * the primitive allocates whole entries in tokens and never splits+ * or trims one.+ */++import { describe, expect, test } from "bun:test";++import {+ allocateRecallBudget,+ isPackingPolicy,+ PACKING_POLICIES,+ type RecallBudgetEntry,+} from "../../../src/core/brain/recall-budget.ts";++const page = (item: string, tokens: number): RecallBudgetEntry<string> => ({ item, tokens });++describe("PACKING_POLICIES vocabulary", () => {+ test("the tuple is frozen and ordered facts_first then retired_first", () => {+ expect([...PACKING_POLICIES]).toEqual(["facts_first", "retired_first"]);+ expect(Object.isFrozen(PACKING_POLICIES)).toBe(true);+ });++ test("isPackingPolicy accepts exactly the tuple members", () => {+ for (const policy of PACKING_POLICIES) {+ expect(isPackingPolicy(policy)).toBe(true);+ }+ });++ test("isPackingPolicy narrows like the queryMode guard: unknown, case and type refuse", () => {+ expect(isPackingPolicy("budget_policy")).toBe(false);+ expect(isPackingPolicy("FACTS_FIRST")).toBe(false);+ expect(isPackingPolicy("facts-first")).toBe(false);+ expect(isPackingPolicy("")).toBe(false);+ expect(isPackingPolicy(undefined)).toBe(false);+ expect(isPackingPolicy(null)).toBe(false);+ expect(isPackingPolicy(1)).toBe(false);+ expect(isPackingPolicy({})).toBe(false);+ expect(isPackingPolicy(["facts_first"])).toBe(false);+ });+});++describe("allocateRecallBudget", () => {+ test("retired_first lets the pages lane claim before the facts lane fills the remainder", () => {+ // Total 10: pages demand 7, facts demand 6. Pages claim first and fit+ // whole; facts keep only what the 3-token remainder holds.+ const r = allocateRecallBudget<string, string>(+ 10,+ [page("p1", 4), page("p2", 3)],+ [page("f1", 4), page("f2", 2)],+ "retired_first",+ );+ expect(r.pages.allocation).toBe(7);+ expect(r.pages.kept).toEqual(["p1", "p2"]);+ expect(r.pages.spent).toBe(7);+ expect(r.facts.allocation).toBe(3);+ expect(r.facts.kept).toEqual(["f2"]);+ expect(r.facts.dropped).toEqual(["f1"]);+ expect(r.facts.spent).toBe(2);+ expect(r.pages.allocation + r.facts.allocation).toBe(10);+ });++ test("facts_first claims facts-side first and pages take the remainder", () => {+ const r = allocateRecallBudget<string, string>(+ 10,+ [page("p1", 4), page("p2", 3)],+ [page("f1", 4), page("f2", 2)],+ "facts_first",+ );+ expect(r.facts.allocation).toBe(6);+ expect(r.facts.kept).toEqual(["f1", "f2"]);+ expect(r.facts.spent).toBe(6);+ expect(r.pages.allocation).toBe(4);+ expect(r.pages.kept).toEqual(["p1"]);+ expect(r.pages.dropped).toEqual(["p2"]);+ expect(r.pages.spent).toBe(4);+ expect(r.facts.allocation + r.pages.allocation).toBe(10);+ });++ test("the policy decides the winner over the same demand", () => {+ const pages = [page("p-big", 8)];+ const facts = [page("f-big", 8)];+ const retiredFirst = allocateRecallBudget(10, pages, facts, "retired_first");+ const factsFirst = allocateRecallBudget(10, pages, facts, "facts_first");+ expect(retiredFirst.pages.kept).toEqual(["p-big"]);+ expect(retiredFirst.facts.dropped).toEqual(["f-big"]);+ expect(factsFirst.facts.kept).toEqual(["f-big"]);+ expect(factsFirst.pages.dropped).toEqual(["p-big"]);+ });++ test("an empty pages lane hands the whole claimable budget to facts", () => {+ for (const policy of PACKING_POLICIES) {+ const r = allocateRecallBudget<string, string>(+ 10,+ [],+ [page("f1", 6), page("f2", 4)],+ policy,+ );+ expect(r.pages.allocation).toBe(0);+ expect(r.pages.kept).toEqual([]);+ expect(r.facts.allocation).toBe(10);+ expect(r.facts.kept).toEqual(["f1", "f2"]);+ expect(r.facts.spent).toBe(10);+ }+ });++ test("an empty facts lane hands the whole claimable budget to pages", () => {+ for (const policy of PACKING_POLICIES) {+ const r = allocateRecallBudget<string, string>(+ 10,+ [page("p1", 6), page("p2", 4)],+ [],+ policy,+ );+ expect(r.facts.allocation).toBe(0);+ expect(r.facts.kept).toEqual([]);+ expect(r.pages.allocation).toBe(10);+ expect(r.pages.kept).toEqual(["p1", "p2"]);+ }+ });++ test("facts_first with only facts populated is an identity pass", () => {+ const facts = [page("f1", 3), page("f2", 3)];+ const alone = allocateRecallBudget<string, string>(8, [], facts, "facts_first");+ expect(alone.facts.kept).toEqual(["f1", "f2"]);+ expect(alone.facts.spent).toBe(6);+ // The lane holds what it claimed; the untouched remainder is the+ // empty pages lane's window, and the books still balance.+ expect(alone.facts.allocation).toBe(6);+ expect(alone.pages.allocation).toBe(2);+ expect(alone.facts.allocation + alone.pages.allocation).toBe(8);+ });++ test("facts exceeding the remainder drop overflow and spend exactly the remainder", () => {+ // Pages claim 9 of 10; the facts lane fits exactly one more token.+ const r = allocateRecallBudget<string, string>(+ 10,+ [page("p1", 9)],+ [page("f1", 5), page("f2", 1)],+ "retired_first",+ );+ expect(r.pages.spent).toBe(9);+ expect(r.facts.allocation).toBe(1);+ expect(r.facts.dropped).toEqual(["f1"]);+ expect(r.facts.kept).toEqual(["f2"]);+ expect(r.facts.spent).toBe(1);+ });++ test("a reserved charge is paid before either lane claims and conservation holds", () => {+ const r = allocateRecallBudget<string, string>(+ 20,+ [page("p1", 5), page("p2", 9)],+ [page("f1", 4), page("f2", 4)],+ "retired_first",+ 6,+ );+ expect(r.reservedTokens).toBe(6);+ expect(r.pages.allocation).toBe(14);+ expect(r.pages.kept).toEqual(["p1", "p2"]);+ expect(r.facts.allocation).toBe(0);+ expect(r.facts.dropped).toEqual(["f1", "f2"]);+ // Total conservation, reserved charge included.+ expect(r.pages.allocation + r.facts.allocation + r.reservedTokens).toBe(20);+ expect(r.pages.spent + r.facts.spent + r.reservedTokens).toBeLessThanOrEqual(20);+ });++ test("a reserved charge at or over the total leaves both lanes nothing", () => {+ const r = allocateRecallBudget<string, string>(+ 5,+ [page("p1", 1)],+ [page("f1", 1)],+ "retired_first",+ 7,+ );+ expect(r.pages.allocation).toBe(0);+ expect(r.facts.allocation).toBe(0);+ expect(r.pages.kept).toEqual([]);+ expect(r.facts.kept).toEqual([]);+ });++ test("a lane entry that cannot fit whole is dropped, never split or trimmed", () => {+ // Native per-entry caps (chars, note counts) stay lane-internal: the+ // primitive sees whole-entry token costs and drops the overflow.+ const r = allocateRecallBudget<string, string>(+ 4,+ [page("huge", 100), page("small", 2)],+ [page("f1", 1)],+ "retired_first",+ );+ expect(r.pages.dropped).toEqual(["huge"]);+ expect(r.pages.kept).toEqual(["small"]);+ expect(r.pages.spent).toBe(2);+ });++ test("a smaller later entry still fits after a larger overflow is dropped", () => {+ const r = allocateRecallBudget<string, string>(+ 6,+ [page("a", 4), page("big", 5), page("c", 2)],+ [page("f1", 0)],+ "retired_first",+ );+ expect(r.pages.kept).toEqual(["a", "c"]);+ expect(r.pages.dropped).toEqual(["big"]);+ });++ test("an entry with a non-finite or negative cost is dropped, never charged", () => {+ const r = allocateRecallBudget<string, string>(+ 6,+ [page("nan", Number.NaN), page("neg", -3), page("ok", 2)],+ [],+ "retired_first",+ );+ expect(r.pages.dropped).toEqual(["nan", "neg"]);+ expect(r.pages.kept).toEqual(["ok"]);+ expect(r.pages.spent).toBe(2);+ });++ test("a zero-token entry always fits and spends nothing", () => {+ const r = allocateRecallBudget<string, string>(+ 2,+ [page("p1", 2), page("free", 0)],+ [],+ "retired_first",+ );+ expect(r.pages.kept).toEqual(["p1", "free"]);+ expect(r.pages.spent).toBe(2);+ });++ test("a non-finite or non-positive total is a fail-closed empty allocation", () => {+ for (const total of [0, -1, Number.NaN, Number.POSITIVE_INFINITY]) {+ const r = allocateRecallBudget<string, string>(+ total,+ [page("p1", 1)],+ [page("f1", 1)],+ "retired_first",+ 1,+ );+ expect(r.pages.allocation).toBe(0);+ expect(r.facts.allocation).toBe(0);+ expect(r.pages.kept).toEqual([]);+ expect(r.facts.kept).toEqual([]);+ expect(r.reservedTokens).toBe(0);+ }+ });++ test("the allocation is deterministic and frozen for identical inputs", () => {+ const pages = [page("p1", 3), page("p2", 3)];+ const facts = [page("f1", 2)];+ const a = allocateRecallBudget(8, pages, facts, "retired_first", 1);+ const b = allocateRecallBudget(8, pages, facts, "retired_first", 1);+ expect(JSON.stringify(a)).toBe(JSON.stringify(b));+ expect(Object.isFrozen(a)).toBe(true);+ expect(Object.isFrozen(a.pages)).toBe(true);+ expect(Object.isFrozen(a.facts)).toBe(true);+ expect(Object.isFrozen(a.pages.kept)).toBe(true);+ expect(Object.isFrozen(a.facts.dropped)).toBe(true);+ });++ test("the policy rides the result by name", () => {+ const r = allocateRecallBudget(4, [], [], "facts_first");+ expect(r.policy).toBe("facts_first");+ expect(r.totalTokens).toBe(4);+ });+});diff --git a/tests/core/brain/standing-instruction-facets.test.ts b/tests/core/brain/standing-instruction-facets.test.tsnew file mode 100644index 00000000..c926e4af--- /dev/null+++ b/tests/core/brain/standing-instruction-facets.test.ts@@ -0,0 +1,365 @@+/**+ * Opt-in standing-instruction facets on the `standing_query` attention+ * action (recall lane).+ *+ * With the facet option unset, every emitted string is the legacy+ * `pref-<slug> (scope)` output, byte for byte, and sections carry no+ * facet data at all. With it set, a surfaced preference whose+ * provenance join resolves gains an earliest-cited annotation on its item+ * string plus a structured facet entry; a preference whose join does+ * not resolve keeps the plain string and is absent from `facets` β an+ * honestly-unresolved preference, never an implied provenance.+ *+ * The facet option changes annotations, never membership: the set of+ * preferences a section names is identical with the option on and off,+ * so a facet can never make a withheld preference visible. The+ * import-session fixture below is written by the LIVE import writer+ * (`importSession` over the checked-in claude fixture), so the joined+ * `session_ref` shape is the one production emits, not a hand-written+ * stand-in.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, mkdirSync, readdirSync, rmSync, writeFileSync } from "node:fs";+import { join, resolve } from "node:path";+import { tmpdir } from "node:os";++import { bootstrapBrain } from "../../../src/core/brain/init.ts";+import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";+import {+ attentionFlowsDir,+ brainDirs,+ brainScopedRulePath,+ brainStandingRulesPath,+} from "../../../src/core/brain/paths.ts";+import { writeFrontmatterAtomic } from "../../../src/core/vault.ts";+import { moveToRetired, writePreference } from "../../../src/core/brain/preference.ts";+import { parseSignal, writeSignal } from "../../../src/core/brain/signal.ts";+import { importSession } from "../../../src/core/brain/sessions/import.ts";+import { BRAIN_RETIRED_REASON } from "../../../src/core/brain/types.ts";+import {+ buildAttentionContextBlock,+ evaluateAttentionFlow,+ renderAttentionFlow,+} from "../../../src/core/brain/attention-flows.ts";+import { changedPaths, digestVaultFiles, digestVaultTree } from "../../helpers/vault-digest.ts";++const CLAUDE = resolve("tests/fixtures/sessions/claude-minimal.jsonl");+const NOW = new Date("2026-06-01T00:00:00Z");++let vault: string;+let configHome: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-facet-vault-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-facet-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);+ bootstrapBrain(vault, { configPath });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++function writeConfirmed(slug: string, scope: string, evidenced_by: string[], owner?: string): void {+ writePreference(vault, {+ slug,+ topic: slug,+ principle: `rule ${slug}`,+ created_at: "2026-06-01T00:00:00Z",+ unconfirmed_until: "2026-07-01T00:00:00Z",+ confirmed_at: "2026-06-02T00:00:00Z",+ status: "confirmed",+ evidenced_by,+ scope,+ ...(owner !== undefined ? { owner } : {}),+ });+}++function writeStandingFlow(scopes: string[]): void {+ writeFrontmatterAtomic(+ join(attentionFlowsDir(vault), "loops.md"),+ {+ kind: "brain-attention-flow",+ id: "loops",+ title: "Open loops",+ actions: ["standing_query"],+ standing_queries: scopes,+ status: "active",+ },+ "# Open loops",+ { overwrite: true },+ );+}++/**+ * Run the LIVE session-import writer over the checked-in claude fixture+ * and return the written signal's id and session_ref. The marker turn+ * (`turn-user-1`) emits the feedback signal whose ref is composed as+ * `${sessionKey}#${turnId}` with the portable basename key.+ */+async function importFixtureSignal(): Promise<{ id: string; session_ref: string }> {+ const res = await importSession(vault, CLAUDE, { agent: "test", now: NOW });+ expect(res.signals_created).toBeGreaterThan(0);+ const inbox = brainDirs(vault).inbox;+ const found: { id: string; session_ref: string }[] = [];+ for (const name of readdirSync(inbox).toSorted()) {+ if (!name.endsWith(".md")) continue;+ const signal = parseSignal(join(inbox, name));+ if (signal.session_ref !== undefined && signal.session_ref.endsWith("#turn-user-1")) {+ found.push({ id: signal.id, session_ref: signal.session_ref });+ }+ }+ expect(found.length).toBe(1);+ return found[0]!;+}++/** A capture-lifecycle-shaped signal: `session:<id>#<event>` (lifecycle.ts:858). */+function writeLifecycleSignal(): string {+ return writeSignal(vault, {+ topic: "lifecycle-rule",+ signal: "positive",+ agent: "test",+ principle: "greet before asking",+ created_at: "2026-06-01T00:00:00Z",+ date: "2026-06-01",+ slug: "lifecycle-rule",+ scope: "testing",+ session_ref: "session:00000000-0000-0000-0000-000000000003#UserPromptSubmit",+ source: ["[[session:00000000-0000-0000-0000-000000000003#UserPromptSubmit]]"],+ }).id;+}++/** An auto-extract-shaped signal: a bare session id, no fragment (extract-signals.ts:646). */+function writeBareSignal(): string {+ return writeSignal(vault, {+ topic: "bare-rule",+ signal: "positive",+ agent: "test",+ principle: "prefer small commits",+ created_at: "2026-06-01T00:00:00Z",+ date: "2026-06-01",+ slug: "bare-rule",+ scope: "testing",+ session_ref: "session:conv-02",+ source: ["[[session:conv-02]]"],+ }).id;+}++function standingSection(evaluation: ReturnType<typeof evaluateAttentionFlow>) {+ const section = evaluation.sections.find((s) => s.action === "standing_query");+ expect(section).toBeDefined();+ return section!;+}++/** Strip an earliest-cited annotation, leaving the legacy `id (scope)` string. */+function stripAnnotation(item: string): string {+ return item.split(" [earliest cited ")[0]!;+}++describe("standing-instruction facets: opt-in gate", () => {+ test("with the option unset the items are the legacy strings and carry no facet data", async () => {+ const ref = await importFixtureSignal();+ expect(ref.session_ref).toBe("claude-minimal.jsonl#turn-user-1");+ writeConfirmed("imported", "testing", [`[[${ref.id}]]`]);+ writeStandingFlow(["testing"]);++ const section = standingSection(evaluateAttentionFlow(vault, "loops"));+ expect(section.items).toEqual(["pref-imported (testing)"]);+ expect(Object.hasOwn(section, "facets")).toBe(false);+ });++ test("the option changes annotations, never membership", async () => {+ const ref = await importFixtureSignal();+ writeConfirmed("secret", "testing", [`[[${ref.id}]]`], "alice");+ writeStandingFlow(["testing"]);++ const off = standingSection(evaluateAttentionFlow(vault, "loops"));+ const on = standingSection(+ evaluateAttentionFlow(vault, "loops", { standingInstructionFacets: true }),+ );+ // Same preference ids on both sides: the facet annotates what the+ // lane already delivers and admits nothing new.+ expect(on.items.map(stripAnnotation)).toEqual([...off.items]);+ expect(off.items).toEqual(["pref-secret (testing)"]);+ expect(on.items).toEqual([+ "pref-secret (testing) [earliest cited turn claude-minimal.jsonl#turn-user-1]",+ ]);+ });+});++describe("standing-instruction facets: provenance annotation", () => {+ test("only preferences whose join resolves gain annotations and facet entries", async () => {+ const ref = await importFixtureSignal();+ const lifecycleId = writeLifecycleSignal();+ const bareId = writeBareSignal();+ writeConfirmed("imported", "testing", [`[[${ref.id}]]`]);+ writeConfirmed("lifecycle", "testing", [`[[${lifecycleId}]]`]);+ writeConfirmed("bare", "testing", [`[[${bareId}]]`]);+ writeConfirmed("noevidence", "testing", []);+ writeStandingFlow(["testing"]);++ const section = standingSection(+ evaluateAttentionFlow(vault, "loops", { standingInstructionFacets: true }),+ );+ // Items stay in filename order; the bare auto-extract ref resolves+ // to no origin, so that preference keeps the plain string.+ expect(section.items).toEqual([+ "pref-bare (testing)",+ "pref-imported (testing) [earliest cited turn claude-minimal.jsonl#turn-user-1]",+ "pref-lifecycle (testing) [earliest cited session session:00000000-0000-0000-0000-000000000003#UserPromptSubmit]",+ "pref-noevidence (testing)",+ ]);+ expect(section.facets).toEqual([+ {+ key: "pref:pref-imported",+ preferenceId: "pref-imported",+ scope: "testing",+ origins: [{ sessionKey: "claude-minimal.jsonl", fragment: "turn-user-1", kind: "turn" }],+ },+ {+ key: "pref:pref-lifecycle",+ preferenceId: "pref-lifecycle",+ scope: "testing",+ origins: [+ {+ sessionKey: "session:00000000-0000-0000-0000-000000000003",+ fragment: "UserPromptSubmit",+ kind: "session",+ },+ ],+ },+ ]);+ });++ test("a retired preference is never annotated back into view", async () => {+ const ref = await importFixtureSignal();+ writeConfirmed("doomed", "testing", [`[[${ref.id}]]`]);+ writeStandingFlow(["testing"]);+ moveToRetired(+ vault,+ join(brainDirs(vault).preferences, "pref-doomed.md"),+ BRAIN_RETIRED_REASON.rebutted,+ {+ now: NOW,+ retired_by: "test",+ },+ );++ const section = standingSection(+ evaluateAttentionFlow(vault, "loops", { standingInstructionFacets: true }),+ );+ expect(section.items).toEqual([]);+ expect(section.facets).toEqual([]);+ });++ test("a population where nothing resolves reports zero resolved facets", async () => {+ const bareId = writeBareSignal();+ writeConfirmed("bare", "testing", [`[[${bareId}]]`]);+ writeConfirmed("quiet", "testing", ["[[sig-2026-05-01-missing]]"]);+ writeConfirmed("empty", "testing", []);+ writeStandingFlow(["testing"]);++ const section = standingSection(+ evaluateAttentionFlow(vault, "loops", { standingInstructionFacets: true }),+ );+ expect(section.items).toEqual([+ "pref-bare (testing)",+ "pref-empty (testing)",+ "pref-quiet (testing)",+ ]);+ expect(section.facets).toEqual([]);+ });+});++describe("standing-instruction facets: lane invariants", () => {+ /** Preference ids named by a section's items, annotations and scopes stripped. */+ function itemIds(section: ReturnType<typeof standingSection>): string[] {+ return section.items.map((item) => stripAnnotation(item).split(" (")[0]!);+ }++ test("the whole lane performs zero writes on a fully populated vault", async () => {+ const ref = await importFixtureSignal();+ const lifecycleId = writeLifecycleSignal();+ // The operator-authored layers the lane must never touch, plus the+ // preference populations it reads.+ writeFileSync(brainStandingRulesPath(vault), "# Standing rules\n\nnever force push\n", "utf8");+ mkdirSync(join(brainScopedRulePath(vault, "project", "release"), ".."), { recursive: true });+ writeFileSync(+ brainScopedRulePath(vault, "project", "release"),+ "# Scoped rule\n\ntag before merge\n",+ "utf8",+ );+ writeConfirmed("imported", "testing", [`[[${ref.id}]]`]);+ writeConfirmed("lifecycle", "testing", [`[[${lifecycleId}]]`]);+ writeConfirmed("secret", "testing", ["[[sig-2026-05-01-missing]]"], "alice");+ writeConfirmed("quiet", "testing", []);+ writeStandingFlow(["testing"]);++ // Warm-up: the one-time lazy default-flow bootstrap is pre-existing+ // behavior outside this lane; settle it BEFORE the guarded window.+ evaluateAttentionFlow(vault, "loops");++ const beforeFiles = digestVaultFiles(vault);+ const before = digestVaultTree(vault);+ evaluateAttentionFlow(vault, "loops");+ evaluateAttentionFlow(vault, "loops", { standingInstructionFacets: true });+ renderAttentionFlow(vault, "loops");+ buildAttentionContextBlock(vault, ["loops"]);+ // The per-file digests cover every byte of every file and every+ // path name, so a created, deleted, or edited file anywhere under+ // the vault fails this assertion - including the standing rules,+ // the scoped rule, and any preference file.+ expect(changedPaths(beforeFiles, digestVaultFiles(vault))).toEqual([]);+ expect(digestVaultTree(vault)).toBe(before);+ });++ test("facet output never includes a preference the caller could not already see", async () => {+ const ref = await importFixtureSignal();+ const bareId = writeBareSignal();+ writeConfirmed("imported", "testing", [`[[${ref.id}]]`]);+ writeConfirmed("secret", "testing", [`[[${ref.id}]]`], "alice");+ writeConfirmed("bare", "testing", [`[[${bareId}]]`]);+ writeConfirmed("doomed", "testing", [`[[${ref.id}]]`]);+ moveToRetired(+ vault,+ join(brainDirs(vault).preferences, "pref-doomed.md"),+ BRAIN_RETIRED_REASON.rebutted,+ { now: NOW, retired_by: "test" },+ );+ writeStandingFlow(["testing"]);++ const off = itemIds(standingSection(evaluateAttentionFlow(vault, "loops")));+ const on = standingSection(+ evaluateAttentionFlow(vault, "loops", { standingInstructionFacets: true }),+ );+ // The retired preference is in neither; every facet names a+ // preference the unset lane already delivered.+ const visible = new Set(off);+ expect(off).toEqual(["pref-bare", "pref-imported", "pref-secret"]);+ for (const facet of on.facets!) {+ expect(visible.has(facet.preferenceId)).toBe(true);+ }+ expect(on.facets!.map((f) => f.preferenceId)).toEqual(["pref-imported", "pref-secret"]);+ });++ test("the resolved count is honest in a mixed population", async () => {+ const ref = await importFixtureSignal();+ const bareId = writeBareSignal();+ writeConfirmed("imported", "testing", [`[[${ref.id}]]`]);+ writeConfirmed("bare", "testing", [`[[${bareId}]]`]);+ writeConfirmed("quiet", "testing", []);+ writeStandingFlow(["testing"]);++ const section = standingSection(+ evaluateAttentionFlow(vault, "loops", { standingInstructionFacets: true }),+ );+ // Exactly the preferences whose join resolved, and not one more:+ // three surfaced items, one facet, the rest reported unresolved.+ expect(itemIds(section)).toEqual(["pref-bare", "pref-imported", "pref-quiet"]);+ expect(section.facets!.length).toBe(1);+ expect(section.facets!.map((f) => f.preferenceId)).toEqual(["pref-imported"]);+ });+});diff --git a/tests/core/search/hybrid-deadline.test.ts b/tests/core/search/hybrid-deadline.test.tsindex 7b4ecadb..a701333f 100644--- a/tests/core/search/hybrid-deadline.test.ts+++ b/tests/core/search/hybrid-deadline.test.ts@@ -198,6 +198,37 @@ test("a stalled semantic lane past the budget yields keyword-only results and naexpect(out.warnings.some((w) => w.includes("hybrid deadline"))).toBe(true);});+test("under the overlapped join the clock stays authoritative and its own abort stays unnamed", async () => {+ // The lane now starts before the keyword lanes and is joined where the+ // serial await stood, so the stall races the SAME clock from an earlier+ // point: the fire still voids whatever the lane squeezed in, the answer+ // is keyword-only inside the budget, and the abandoned lane's rejection+ // - which is the deadline's own abort, propagated by the phase - is+ // recognised by name and never becomes a second sentence beside the+ // fire note.+ if (!sqliteVecLoadable()) return;+ await seedCorpus();+ const cfg = configWithDeadline(300);+ await indexVault(cfg, { embeddings: true });++ server.setHandler(stallHandler());+ const started = Date.now();+ const out = await search(cfg, { query: "fox", limit: 5, semantic: true });+ expect(Date.now() - started).toBeLessThan(3_000);+ expect(out.results.map((r) => r.searchType)).toEqual(["keyword"]);+ const codes = out.retrievalTrail?.degraded.map((d) => d.code) ?? [];+ expect(codes).toContain(RETRIEVAL_DEGRADATION.hybridDeadlineExceeded);+ expect(codes).toContain(RETRIEVAL_DEGRADATION.hybridDegraded);++ // Let the abandoned lane's rejection settle before judging the+ // warnings, so a late sentence cannot land after the assertion.+ await Bun.sleep(50);+ const abandonedSentences = out.warnings.filter((w) =>+ w.startsWith("hybrid deadline: an abandoned lane failed after the answer:"),+ );+ expect(abandonedSentences).toEqual([]);+});+test("the deadline degrade is keyword-only byte-shaped like the provider-unavailable degrade", async () => {if (!sqliteVecLoadable()) return;await seedCorpus();diff --git a/tests/core/search/hybrid-overlap.test.ts b/tests/core/search/hybrid-overlap.test.tsnew file mode 100644index 00000000..b8bb2687--- /dev/null+++ b/tests/core/search/hybrid-overlap.test.ts@@ -0,0 +1,535 @@+/**+ * The overlapped-lane join seam (recall-lane-latency): the semantic lane+ * starts before the keyword lanes run and is joined where the serial await+ * stood. The unit half runs over fake promises, so the catch-attachment+ * semantics are pinned without an HTTP round trip: the lane starts at+ * seam-creation time (the keyword step's window), a keyword-step throw+ * never strands an in-flight lane as an unhandled rejection, and a clock+ * that already ran out starts nothing. The integration half runs search()+ * against the fake embed endpoint and pins the overlapped contract: the+ * same rows as the serial pipeline produced, the embed in flight across+ * the join, keyword sentences before semantic sentences - and the one+ * property the lane exists for, that the embed request actually DISPATCHES+ * before the keyword lanes finish (a semaphore acquire that yields a+ * microtask on the free-permit path would ship the dispatch after them,+ * and the ordering test below is the pin that catches it).+ */++import { test, expect, beforeEach, afterEach } from "bun:test";+import { Database } from "bun:sqlite";++import { indexVault } from "../../../src/core/search/indexer.ts";+import {+ createCompositeDeadline,+ overlappedEmbedMs,+ search,+ startSemanticLane,+} from "../../../src/core/search/search.ts";+import type { SearchLaneTiming } from "../../../src/core/search/types.ts";+import { FAKE_PROVIDER_KEY } from "../../helpers/fake-credentials.ts";+import type { RetrievalDegradationSink } from "../../../src/core/search/retrieval-trail.ts";+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";++/**+ * A real composite deadline with a controlled clock: `expiresAt` is what+ * the seam reads, so the budget is anchored backwards from it.+ */+function realDeadline(expiresAt: number): {+ readonly deadline: NonNullable<ReturnType<typeof createCompositeDeadline>>;+ readonly warnings: string[];+} {+ const warnings: string[] = [];+ const budgetMs = 60_000;+ const deadline = createCompositeDeadline({+ budgetMs,+ startMs: expiresAt - budgetMs,+ clock: () => Date.now(),+ warnings,+ degraded: [],+ wantSemantic: true,+ keywordHitCount: () => 1,+ });+ if (deadline === null) throw new Error("a named budget must build a deadline");+ return { deadline, warnings };+}++test("the lane starts when the seam is created, so the keyword step runs while it is pending", async () => {+ const gate = Promise.withResolvers<string>();+ let settled = false;+ const lane = startSemanticLane({+ start: () =>+ gate.promise.then((value) => {+ settled = true;+ return value;+ }),+ deadline: null,+ fallback: () => {+ throw new Error("the fallback must never run for a joined lane");+ },+ });+ // The caller's synchronous keyword step runs HERE, between start and+ // join, while the lane promise is still in flight.+ expect(settled).toBe(false);+ gate.resolve("lane outcome");+ expect(await lane.join()).toBe("lane outcome");+});++test("a keyword-step throw with a deadline routes the lane's rejection to the abandoned-failure note", async () => {+ const gate = Promise.withResolvers<string>();+ const { deadline, warnings } = realDeadline(Date.now() + 60_000);+ const lane = startSemanticLane({+ start: () => gate.promise,+ deadline,+ fallback: () => "fallback",+ });+ // The keyword step throws and the call unwinds without ever joining;+ // the in-flight lane rejects afterwards.+ const keywordStep = (): never => {+ throw new Error("keyword lane exploded");+ };+ expect(() => keywordStep()).toThrow("keyword lane exploded");+ // The joiner exists even though this caller never reaches its join.+ expect(typeof lane.join).toBe("function");+ gate.reject(new Error("embed exploded"));+ await Bun.sleep(10);+ expect(+ warnings.some((w) =>+ w.startsWith(+ "hybrid deadline: an abandoned lane failed after the answer: Error: embed exploded",+ ),+ ),+ ).toBe(true);+});++test("a keyword-step throw with no deadline leaves no unhandled rejection", async () => {+ const unhandled: unknown[] = [];+ const onUnhandled = (reason: unknown): void => {+ unhandled.push(reason);+ };+ process.on("unhandledRejection", onUnhandled);+ try {+ const gate = Promise.withResolvers<string>();+ const lane = startSemanticLane({+ start: () => gate.promise,+ deadline: null,+ fallback: () => "fallback",+ });+ const keywordStep = (): never => {+ throw new Error("keyword lane exploded");+ };+ expect(() => keywordStep()).toThrow("keyword lane exploded");+ // The joiner exists even though this caller never reaches its join.+ expect(typeof lane.join).toBe("function");+ gate.reject(new Error("embed exploded"));+ await Bun.sleep(20);+ // Nobody joined; the seam owns the stranded lane and its rejection.+ expect(unhandled).toEqual([]);+ } finally {+ process.off("unhandledRejection", onUnhandled);+ }+});++test("an already-expired deadline starts nothing and fires at the join, after the keyword lanes", async () => {+ let started = false;+ const { deadline, warnings } = realDeadline(Date.now() - 1);+ const lane = startSemanticLane({+ start: () => {+ started = true;+ return Promise.resolve("lane");+ },+ deadline,+ fallback: () => "fallback",+ });+ expect(started).toBe(false);+ // The fire waits for the join: at the start site the keyword lanes have+ // not run, so no degrade decision exists yet.+ expect(warnings).toEqual([]);+ expect(await lane.join()).toBe("fallback");+ expect(warnings.some((w) => w.includes("hybrid deadline"))).toBe(true);+});++test("a join-time fire reads the keyword pool the lanes populated, not the empty initializer", async () => {+ const pool: number[] = [];+ const warnings: string[] = [];+ const degraded: RetrievalDegradationSink = [];+ const budgetMs = 60_000;+ const deadline = createCompositeDeadline({+ budgetMs,+ startMs: Date.now() - budgetMs - 1,+ clock: () => Date.now(),+ warnings,+ degraded,+ wantSemantic: true,+ keywordHitCount: () => pool.length,+ });+ if (deadline === null) throw new Error("a named budget must build a deadline");+ const lane = startSemanticLane({+ start: () => Promise.resolve("lane"),+ deadline,+ fallback: () => "fallback",+ });+ // The synchronous keyword lanes run here and populate the pool, exactly+ // as the serial pipeline's snapshot point assumed.+ pool.push(2);+ expect(await lane.join()).toBe("fallback");+ expect(warnings.some((w) => w.includes("hybrid deadline"))).toBe(true);+ // The umbrella fires: the degrade decision saw the real pool, not the+ // empty initializer a start-site fire would have read.+ expect(warnings).toContain("hybrid_degraded: semantic lane unavailable, served keyword-only");+});++test("a lane that fails after a join-time fire is named, not defused", async () => {+ const gate = Promise.withResolvers<string>();+ const warnings: string[] = [];+ const degraded: RetrievalDegradationSink = [];+ let now = Date.now();+ const deadline = createCompositeDeadline({+ budgetMs: 60_000,+ startMs: now - 30_000,+ clock: () => now,+ warnings,+ degraded,+ wantSemantic: true,+ keywordHitCount: () => 1,+ });+ if (deadline === null) throw new Error("a named budget must build a deadline");+ const lane = startSemanticLane({+ start: () => gate.promise,+ deadline,+ fallback: () => "fallback",+ });+ // The clock expires while the keyword lanes run, before the join: the+ // join's never-start arm fires the deadline and serves the fallback.+ now += 31_000;+ expect(await lane.join()).toBe("fallback");+ expect(deadline.hasFired()).toBe(true);+ // The lane then rejects with a non-abort failure. The race never armed,+ // so the at-start handler is still the deadline's ear for this lane.+ gate.reject(new Error("embed exploded after the join-time fire"));+ await Bun.sleep(20);+ expect(+ warnings.some((w) =>+ w.startsWith(+ "hybrid deadline: an abandoned lane failed after the answer: Error: embed exploded after the join-time fire",+ ),+ ),+ ).toBe(true);+});++test("a joined lane that loses the race serves the fallback and names a late failure", async () => {+ const gate = Promise.withResolvers<string>();+ const { deadline, warnings } = realDeadline(Date.now() + 25);+ const lane = startSemanticLane({+ start: () => gate.promise,+ deadline,+ fallback: () => "fallback",+ });+ expect(await lane.join()).toBe("fallback");+ expect(warnings.some((w) => w.includes("hybrid deadline"))).toBe(true);+ gate.reject(new Error("late embed failure"));+ await Bun.sleep(20);+ expect(+ warnings.some((w) =>+ w.startsWith(+ "hybrid deadline: an abandoned lane failed after the answer: Error: late embed failure",+ ),+ ),+ ).toBe(true);+});++test("the overlap arithmetic is the exact intersection of the two intervals", () => {+ // The embed starts with the seam and the keyword lane runs wholly+ // underneath it: the overlap is the keyword interval, byte for byte.+ expect(overlappedEmbedMs(1_000, 1_500, 1_010, 1_090)).toBe(80);+ // The keyword lane outlasts the embed: the embed tail is what hid.+ expect(overlappedEmbedMs(1_000, 1_050, 1_020, 1_200)).toBe(30);+ // Disjoint intervals never invent overlap, in either order.+ expect(overlappedEmbedMs(1_000, 1_050, 1_100, 1_200)).toBe(0);+ expect(overlappedEmbedMs(1_100, 1_200, 1_000, 1_050)).toBe(0);+ // Identical stamps (a sub-millisecond lane on a coarse clock) are zero,+ // never negative.+ expect(overlappedEmbedMs(1_000, 1_000, 1_000, 1_000)).toBe(0);+ // A stamp pair that crosses (end before start) floors at zero.+ expect(overlappedEmbedMs(1_050, 1_000, 1_000, 1_200)).toBe(0);+});++test("a lane that rejects before the fire rethrows through the join", async () => {+ const gate = Promise.withResolvers<string>();+ const { deadline } = realDeadline(Date.now() + 60_000);+ const lane = startSemanticLane({+ start: () => gate.promise,+ deadline,+ fallback: () => "fallback",+ });+ gate.reject(new Error("embed refused"));+ const note = await lane.join().then(+ () => "resolved",+ (error: unknown) => (error as Error).message,+ );+ expect(note).toBe("embed refused");+});++// ββ search() over a real index βββββββββββββββββββββββββββββββββββββββββββββββ++let vault: string;+let dbPath: string;+let cleanup: () => void;+let server: FakeHttp;++beforeEach(async () => {+ const v = createTempVault("hybrid-overlap");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+ server = await startFakeHttp();+}, 20_000);++afterEach(async () => {+ cleanup();+ await server.close();+}, 20_000);++function semanticConfig(extra: Record<string, unknown> = {}) {+ return makeConfig({+ vault,+ dbPath,+ semantic: {+ enabled: true,+ provider: "openai-compat",+ baseUrl: server.url,+ model: "fake-model",+ apiKey: FAKE_PROVIDER_KEY,+ dimension: 4,+ timeoutMs: 5_000,+ concurrency: 2,+ batchSize: 8,+ costGateUsd: 0,+ maxRetries: 1,+ ...extra,+ },+ });+}++async function seedCorpus() {+ writeMd(vault, "Notes/fox.md", "# Fox\n\nThe quick brown fox jumps over the lazy dog.");+ writeMd(vault, "Other/bar.md", "# Bar\n\nA different note about cats and turtles.");+}++/** Embeddings-shaped body with the default handler's deterministic vectors. */+function embeddingsBody(req: { body: unknown }) {+ const body = (req.body ?? {}) as { input?: string[]; model?: string };+ const inputs = Array.isArray(body.input) ? body.input : [];+ return {+ status: 200,+ body: {+ data: inputs.map((text, index) => {+ const tokens = text.split(/\s+/).filter(Boolean).length;+ return { object: "embedding", embedding: [tokens, index, text.length, 1], index };+ }),+ model: body.model ?? "fake-model",+ },+ };+}++/** The stable projection the deadline suite pins: rows, order, typing. */+function resultShape(out: Awaited<ReturnType<typeof search>>): ReadonlyArray<unknown> {+ return out.results.map((r) => [r.path, r.searchType, r.score > 0, r.reasons]);+}++test("a delayed embed joined under both deadline arms keeps one result shape", async () => {+ if (!sqliteVecLoadable()) return;+ await seedCorpus();+ const cfg = semanticConfig();+ await indexVault(cfg, { embeddings: true });+ // A slow round trip makes the join actually wait; both deadline arms+ // must serve the same rows the serial pipeline produced.+ server.setHandler((req) => ({ ...embeddingsBody(req), delayMs: 150 }));++ const withClock = await search(cfg, { query: "fox", limit: 5, semantic: true });+ const clockOff = await search(+ { ...cfg, hybridDeadlineMs: 0 },+ {+ query: "fox",+ limit: 5,+ semantic: true,+ },+ );+ expect(resultShape(withClock)).toEqual(resultShape(clockOff));+ // The join delivered the lane's hits into the fused ranking.+ expect(withClock.results.some((r) => r.searchType === "hybrid")).toBe(true);+});++test("the embed request is in flight across the join while the answer is pending", async () => {+ if (!sqliteVecLoadable()) return;+ await seedCorpus();+ const cfg = semanticConfig();+ await indexVault(cfg, { embeddings: true });+ const arrived = Promise.withResolvers<void>();+ server.setHandler((req) => {+ if (req.path.endsWith("/embeddings")) arrived.resolve();+ return { ...embeddingsBody(req), delayMs: 200 };+ });++ let resolved = false;+ const started = Date.now();+ const pending = search(cfg, { query: "fox", limit: 5, semantic: true }).then((out) => {+ resolved = true;+ return out;+ });+ await arrived.promise;+ // The round trip was put in flight and has not answered: the answer+ // cannot be served ahead of the lane it joined.+ expect(resolved).toBe(false);+ const out = await pending;+ // Generous bounds, robust to timer clamping: the answer waited out the+ // embed delay it joined rather than cutting the lane short.+ expect(Date.now() - started).toBeGreaterThanOrEqual(180);+ expect(out.results.some((r) => r.searchType === "hybrid")).toBe(true);+});++test("the embed request dispatches before the keyword lanes finish", async () => {+ if (!sqliteVecLoadable()) return;+ await seedCorpus();+ const cfg = semanticConfig();+ await indexVault(cfg, { embeddings: true });++ // The ordering is read as EVENT order at two synchronous seams, not as+ // wall-clock arithmetic: a millisecond-resolution clock stamps a+ // dispatch and a keyword lane that share a tick with the same reading,+ // so a timestamp comparison flakes on exactly the runs it exists to+ // catch. The fetch seam records the embed dispatch; the keyword lane's+ // bm25 read records the keyword lanes' execution - the one+ // `bm25(chunk_fts` query in the store, so the query-shape step's+ // document-frequency reads (a COUNT, no bm25) never log a false marker.+ const events: string[] = [];+ let dispatchMs: number | undefined;+ const realFetch = globalThis.fetch;+ globalThis.fetch = ((...args: Parameters<typeof fetch>) => {+ const input = args[0];+ const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url;+ if (url.endsWith("/embeddings") && !events.includes("dispatch")) {+ events.push("dispatch");+ dispatchMs = Date.now();+ }+ return realFetch(...args);+ }) as typeof fetch;+ const realQuery = Database.prototype.query;+ Database.prototype.query = function (this: Database, sql: string): unknown {+ if (sql.includes("bm25(chunk_fts")) events.push("keyword-fts");+ return (realQuery as unknown as (this: Database, sql: string) => unknown).call(this, sql);+ } as typeof Database.prototype.query;++ // The fake server shares this process's event loop, so its handler can+ // only observe the request once the synchronous keyword work yields -+ // arrival at the handler can never precede the keyword stamps. The+ // dispatch at the fetch seam is the ordering observable; the arrival+ // log pins that the observed dispatch reached the endpoint.+ const arrivals: number[] = [];+ server.setHandler((req) => {+ if (req.path.endsWith("/embeddings")) arrivals.push(Date.now());+ return { ...embeddingsBody(req), delayMs: 150 };+ });++ let timing: SearchLaneTiming | undefined;+ let out: Awaited<ReturnType<typeof search>>;+ try {+ out = await search(cfg, {+ query: "fox",+ limit: 5,+ semantic: true,+ laneTiming: (t) => void (timing = t),+ });+ } finally {+ Database.prototype.query = realQuery;+ globalThis.fetch = realFetch;+ }++ // Both seams were hit, and the answer joined the ranked lanes.+ expect(events).toContain("dispatch");+ expect(events).toContain("keyword-fts");+ expect(out.results.some((r) => r.searchType === "hybrid")).toBe(true);+ // The load-bearing ordering: the request left inside the lane-start+ // tick, before the keyword lanes' own read ran - the keyword lanes+ // executed while the round trip was outstanding, which is the overlap+ // the lane exists to buy. A dispatch parked on a microtask by a+ // yielding semaphore acquire lands after that read and fails here.+ expect(events.indexOf("dispatch")).toBeLessThan(events.indexOf("keyword-fts"));+ // The same ordering in the sink's stamps: the dispatch precedes the+ // keyword interval's opening and, with it, the keyword lane's end stamp.+ if (+ timing === undefined ||+ dispatchMs === undefined ||+ timing.embedStartMs === undefined ||+ timing.embedEndMs === undefined+ ) {+ throw new Error("the lane timing sink or the dispatch/resolution stamps never arrived");+ }+ expect(dispatchMs).toBeLessThanOrEqual(timing.keywordStartMs);+ expect(dispatchMs).toBeLessThanOrEqual(timing.keywordEndMs);+ // The sink's start stamp is the dispatch report: the phase stamped the+ // moment the provider handed the request toward the wire, not an+ // earlier pre-call instant, and the resolution stamp sits behind the+ // delayed answer, past the keyword lane's end.+ expect(timing.embedStartMs).toBeLessThanOrEqual(dispatchMs);+ expect(timing.embedEndMs).toBeGreaterThan(timing.keywordEndMs);+ // The delayed endpoint kept the lane's resolution behind the keyword+ // lane, so the dispatch really did precede the answer.+ expect(arrivals.length).toBeGreaterThan(0);+ expect(arrivals[0]).toBeLessThan(timing.embedEndMs ?? Number.MAX_SAFE_INTEGER);+});++test("a provider that never reports a dispatch keeps the pre-call interval as the fallback", async () => {+ if (!sqliteVecLoadable()) return;+ await seedCorpus();+ // The offline local embedder predates the dispatch report and finishes+ // synchronously: the phase's pre-call stamp is the only start it can+ // have, and the honest overlap for it is zero - its embed is done+ // before the keyword lanes open, so none of that interval was hidden.+ const cfg = semanticConfig({ provider: "local" });+ await indexVault(cfg, { embeddings: true });+ let timing: SearchLaneTiming | undefined;+ const out = await search(cfg, {+ query: "fox",+ limit: 5,+ semantic: true,+ laneTiming: (t) => void (timing = t),+ });+ if (+ timing === undefined ||+ timing.embedStartMs === undefined ||+ timing.embedEndMs === undefined+ ) {+ throw new Error("the lane timing sink or its fallback interval never arrived");+ }+ expect(timing.embedEndMs).toBeGreaterThanOrEqual(timing.embedStartMs);+ expect(timing.overlappedMs).toBe(0);+ expect(out.results.some((r) => r.searchType === "hybrid")).toBe(true);+});++test("keyword sentences precede semantic sentences in the merged warnings", async () => {+ if (!sqliteVecLoadable()) return;+ await seedCorpus();+ const cfg = semanticConfig({ inputWindowTokens: 8 });+ await indexVault(cfg, { embeddings: true });+ // Keyword sentence: a desynced FTS table rebuilt at query time.+ const db = new Database(dbPath);+ try {+ db.run("DELETE FROM chunk_fts");+ } finally {+ db.close();+ }+ // Semantic sentence: the query cut to the model's input window; the+ // embed is delayed so the lane completes after the keyword lane.+ server.setHandler((req) => ({ ...embeddingsBody(req), delayMs: 150 }));++ const out = await search(cfg, { query: "abcdefghij".repeat(20), limit: 5, semantic: true });++ const keywordSentence = out.warnings.findIndex((w) => w.includes("rebuilt FTS"));+ const semanticSentence = out.warnings.findIndex((w) =>+ w.includes("embedding_input_window_tokens"),+ );+ expect(keywordSentence).toBeGreaterThanOrEqual(0);+ expect(semanticSentence).toBeGreaterThan(keywordSentence);+});diff --git a/tests/core/search/outcome-shape-census.test.ts b/tests/core/search/outcome-shape-census.test.tsnew file mode 100644index 00000000..d2bcb313--- /dev/null+++ b/tests/core/search/outcome-shape-census.test.ts@@ -0,0 +1,89 @@+/**+ * Census over the search outcome's serialized shape (recall-lane-latency).+ *+ * The wave's timing channel is a per-call sink option, never a field on+ * {@link SearchOutcome}: the query cache persists whole outcomes, so one+ * call's timings frozen into a cached row would serve every later caller+ * a stale measurement. The sink keeps that honest only while the outcome+ * type stays timing-free, and nothing else pins it - the explain tests+ * pin the flag-off WIRE payload, which a new field would not touch.+ *+ * Two pins, one run:+ *+ * - The top-level key set of a plain keyword-only outcome is exactly+ * the sanctioned list. A new top-level field - timing or otherwise -+ * breaks this census and forces a conscious decision, which is the+ * point: a field added "just for telemetry" is exactly the one that+ * slides into the cache.+ * - No key anywhere in the serialized outcome names the timing+ * vocabulary the wave introduced (the sink's record names, their+ * wire spellings, and the generic timing words). The walk is+ * recursive, so a nested record cannot carry what the top level+ * refuses.+ */++import { afterAll, describe, expect, test } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++const SANCTIONED_TOP_LEVEL_KEYS = ["idfWeightedCoverage", "results", "total", "warnings"];++const TIMING_VOCABULARY = [+ "overlappedMs",+ "overlapped_ms",+ "embedStartMs",+ "embedEndMs",+ "keywordStartMs",+ "keywordEndMs",+ "laneTiming",+ "semanticAttempted",+];++function collectKeys(value: unknown, into: Set<string>): void {+ if (Array.isArray(value)) {+ for (const item of value) collectKeys(item, into);+ return;+ }+ if (value !== null && typeof value === "object") {+ for (const [key, nested] of Object.entries(value as Record<string, unknown>)) {+ into.add(key);+ collectKeys(nested, into);+ }+ }+}++describe("search outcome shape census", () => {+ const v = createTempVault("outcome-shape-census");+ let outcome: Awaited<ReturnType<typeof search>> | undefined;++ test("setup: a plain keyword-only run", async () => {+ writeMd(+ v.vault,+ "notes/fox.md",+ "---\ntitle: Fox\n---\n\n# Fox\n\nThe quick brown fox jumps over the lazy dog.\n",+ );+ const config = makeConfig({ vault: v.vault, dbPath: v.dbPath });+ await indexVault(config);+ outcome = await search(config, { query: "fox", limit: 5, keywordOnly: true });+ expect(outcome.results.length).toBeGreaterThan(0);+ }, 30_000);++ test("the outcome's top-level key set is exactly the sanctioned list", () => {+ if (outcome === undefined) throw new Error("the setup run produced no outcome");+ expect(Object.keys(outcome).sort()).toEqual(SANCTIONED_TOP_LEVEL_KEYS);+ });++ test("no key anywhere in the serialized outcome names the timing vocabulary", () => {+ if (outcome === undefined) throw new Error("the setup run produced no outcome");+ const keys = new Set<string>();+ collectKeys(outcome, keys);+ const offenders = TIMING_VOCABULARY.filter((name) => keys.has(name));+ expect(offenders).toEqual([]);+ });++ afterAll(() => {+ v.cleanup();+ });+});diff --git a/tests/core/search/relational-rerank-pin.test.ts b/tests/core/search/relational-rerank-pin.test.tsindex efeeafb2..e0f72229 100644--- a/tests/core/search/relational-rerank-pin.test.ts+++ b/tests/core/search/relational-rerank-pin.test.ts@@ -23,8 +23,15 @@ import { applyRelationalRerankPin } from "../../../src/core/search/pipeline/relaimport { resolveSearchConfig } from "../../../src/core/search/index.ts";import { indexVault } from "../../../src/core/search/indexer.ts";import { search } from "../../../src/core/search/search.ts";+import { FAKE_PROVIDER_KEY } from "../../helpers/fake-credentials.ts";import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";import type { BrainSearchResult } from "../../../src/core/search/search-result.ts";+import {+ startFakeHttp,+ type FakeHttp,+ type FakeRequest,+ type FakeResponseSpec,+} from "../../helpers/fake-http.ts";// βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ// Pure protect rule@@ -143,19 +150,32 @@ test("search_relational_rerank_pin: true resolves onto the rerank config", () =>let vault: string;let dbPath: string;let cleanup: () => void;+let server: FakeHttp;-const realFetch = globalThis.fetch;+/** The score of the document containing `winner`. */+const WINNER_SCORE = 0.99;+/** The score of every document that does not contain `winner`. */+const REST_SCORE = 0.01;-/** Stub the rerank endpoint: passages matching `winner` score high, the rest ~0. */-function stubRerankFetch(winner: string): void {- globalThis.fetch = (async (_url: string, init: RequestInit) => {- const body = JSON.parse(String(init.body)) as { documents: string[] };- const results = body.documents.map((doc, index) => ({- index,- relevance_score: doc.includes(winner) ? 0.99 : 0.01,- }));- return new Response(JSON.stringify({ results }), { status: 200 });- }) as unknown as typeof fetch;+/** Handler scoring the document that contains `winner` highest. */+function winnerHandler(winner: string): (req: FakeRequest) => FakeResponseSpec {+ return (req) => {+ const documents = (req.body as { documents: string[] }).documents;+ return {+ status: 200,+ body: {+ results: documents.map((doc, index) => ({+ index,+ relevance_score: doc.includes(winner) ? WINNER_SCORE : REST_SCORE,+ })),+ },+ };+ };+}++/** A 503 endpoint answer: rerank degrades to the heuristic order. */+function unavailable(): FakeResponseSpec {+ return { status: 503, body: { error: "boom" } };}/**@@ -166,6 +186,7 @@ function stubRerankFetch(winner: string): void {* fresh, so the heuristic order puts it first.*/async function build(): Promise<void> {+ server = await startFakeHttp();const v = createTempVault("relational-rerank-pin");vault = v.vault;dbPath = v.dbPath;@@ -189,32 +210,34 @@ async function build(): Promise<void> {await indexVault(makeConfig({ vault, dbPath }));}-const RERANK_ENABLED = {- enabled: true,- baseUrl: "https://api.example.com/v1",- model: "rerank-1",- apiKey: "secret",- topK: 10,-} as const;+/** The rerank endpoint config, pointed at the per-test fixture. */+function rerankEnabled() {+ return {+ enabled: true,+ baseUrl: server.url,+ model: "rerank-1",+ apiKey: FAKE_PROVIDER_KEY,+ topK: 10,+ } as const;+}test("pin on: the cross-encoder cannot sink the relational hit below its heuristic position", async () => {- await build();try {+ await build();const base = {fusionMode: "rrf" as const,relationalArmEnabled: true,- rerank: RERANK_ENABLED,+ rerank: rerankEnabled(),};// The heuristic order (the endpoint fails -> rerank degrades to it):// the freshly-written relational hit leads.- globalThis.fetch = (async () =>- new Response("boom", { status: 503 })) as unknown as typeof fetch;+ server.setHandler(unavailable);const heuristic = await search(makeConfig({ vault, dbPath, ...base }), {query: "[[hub]] related",});expect(heuristic.results[0]!.path).toBe("neighbor.md");// The cross-encoder prefers the far node and buries the relational hit.- stubRerankFetch("gamma");+ server.setHandler(winnerHandler("gamma"));const buried = await search(makeConfig({ vault, dbPath, ...base }), {query: "[[hub]] related",});@@ -225,7 +248,7 @@ test("pin on: the cross-encoder cannot sink the relational hit below its heuristvault,dbPath,...base,- rerank: { ...RERANK_ENABLED, relationalRerankPin: true },+ rerank: { ...rerankEnabled(), relationalRerankPin: true },}),{ query: "[[hub]] related" },);@@ -234,20 +257,20 @@ test("pin on: the cross-encoder cannot sink the relational hit below its heurist"relational_pin: floored at pre-rerank position 0",);} finally {- globalThis.fetch = realFetch;+ await server.close();cleanup();}});test("pin on: a relational hit the cross-encoder genuinely prefers may rise", async () => {- await build();try {+ await build();const base = {fusionMode: "rrf" as const,relationalArmEnabled: true,- rerank: { ...RERANK_ENABLED, relationalRerankPin: true },+ rerank: { ...rerankEnabled(), relationalRerankPin: true },};- stubRerankFetch("beta");+ server.setHandler(winnerHandler("beta"));const pinned = await search(makeConfig({ vault, dbPath, ...base }), {query: "[[hub]] related",});@@ -255,7 +278,7 @@ test("pin on: a relational hit the cross-encoder genuinely prefers may rise", asexpect(pinned.results[0]!.reasons.some((r) => r.startsWith("relational_pin:"))).toBe(false);expect(pinned.results[0]!.reasons.some((r) => r.startsWith("cross_encoder:"))).toBe(true);} finally {- globalThis.fetch = realFetch;+ await server.close();cleanup();}});@@ -265,14 +288,14 @@ const project = (o: Awaited<ReturnType<typeof search>>) =>o.results.map((r) => ({ path: r.path, score: r.score, reasons: r.reasons }));test("pin off keeps rerank behavior byte-identical", async () => {- await build();try {+ await build();const base = {fusionMode: "rrf" as const,relationalArmEnabled: true,- rerank: RERANK_ENABLED,+ rerank: rerankEnabled(),};- stubRerankFetch("gamma");+ server.setHandler(winnerHandler("gamma"));// One frozen clock across both calls: recency decay is a continuous// function of wall-clock time, and two real Date.now() reads would// perturb the low-order digits of `score` for reasons that have@@ -290,7 +313,7 @@ test("pin off keeps rerank behavior byte-identical", async () => {vault,dbPath,...base,- rerank: { ...RERANK_ENABLED, relationalRerankPin: false },+ rerank: { ...rerankEnabled(), relationalRerankPin: false },}),{ query: "[[hub]] related" },);@@ -299,7 +322,7 @@ test("pin off keeps rerank behavior byte-identical", async () => {}expect(project(off)).toEqual(project(absent));} finally {- globalThis.fetch = realFetch;+ await server.close();cleanup();}});diff --git a/tests/core/search/rerank-sunset.test.ts b/tests/core/search/rerank-sunset.test.tsindex 568605a2..5d1e98b8 100644--- a/tests/core/search/rerank-sunset.test.ts+++ b/tests/core/search/rerank-sunset.test.ts@@ -202,23 +202,18 @@ describe("through search()", () => {let vault: string;let dbPath: string;let cleanup: () => void;- const realFetch = globalThis.fetch;- let calls = 0;+ let fake: FakeHttp;- beforeEach(() => {+ beforeEach(async () => {({ vault, dbPath, cleanup } = createTempVault("rerank-sunset"));writeMd(vault, "strong.md", "# Strong\n\nfox fox fox fox the quick brown fox jumps high.");writeMd(vault, "weak.md", "# Weak\n\nA note mostly about cats, with one fox mention here.");- calls = 0;- globalThis.fetch = (async (_url: string, init: RequestInit) => {- calls += 1;- const body = JSON.parse(String(init.body)) as { documents: string[] };- const results = body.documents.map((_d, index) => ({ index, relevance_score: index }));- return new Response(JSON.stringify({ results }), { status: 200 });- }) as unknown as typeof fetch;+ // No per-test handler: the healthy call rides the fixture's default+ // /rerank route, and the skip verdicts must never reach HTTP at all.+ fake = await startFakeHttp();});- afterEach(() => {- globalThis.fetch = realFetch;+ afterEach(async () => {+ await fake.close();cleanup();});@@ -227,7 +222,7 @@ describe("through search()", () => {vault,dbPath,cacheEnabled: true,- rerank: { enabled: true, baseUrl: "https://api.example.com/v1", model, apiKey: "k" },+ rerank: { enabled: true, baseUrl: fake.url, model, apiKey: "k" },});}@@ -236,7 +231,7 @@ describe("through search()", () => {await indexVault(config);const base = await search(makeConfig({ vault, dbPath }), { query: "fox", limit: 10 });const first = await search(config, { query: "fox", limit: 10 });- expect(calls).toBe(0);+ expect(fake.callCount()).toBe(0);expect(first.results.map((r) => r.path)).toEqual(base.results.map((r) => r.path));expect(first.retrievalTrail?.degraded).toContainEqual({code: RETRIEVAL_DEGRADATION.rerankModelSunset,@@ -245,7 +240,7 @@ describe("through search()", () => {// the answer is written to the query cache.expect(queryCacheRows(dbPath)).toBe(1);const second = await search(config, { query: "fox", limit: 10 });- expect(calls).toBe(0);+ expect(fake.callCount()).toBe(0);expect(second.retrievalTrail).toEqual(first.retrievalTrail);});@@ -253,7 +248,7 @@ describe("through search()", () => {const config = cfg(NEGATIVE_SHIPPED_MODEL);await indexVault(config);const out = await search(config, { query: "fox", limit: 10 });- expect(calls).toBe(1);+ expect(fake.callCount()).toBe(1);expect(out.retrievalTrail).toBeUndefined();});});diff --git a/tests/core/search/rerank.cross-encoder.test.ts b/tests/core/search/rerank.cross-encoder.test.tsindex 15ea174b..d02c4d78 100644--- a/tests/core/search/rerank.cross-encoder.test.ts+++ b/tests/core/search/rerank.cross-encoder.test.ts@@ -2,107 +2,107 @@* OpenAI-compatible cross-encoder rerank provider HTTP shape* (retrieval-precision-quality-loop, card A / t_110867f5).*- * Exercises the `/rerank` request/response contract with a stubbed- * `fetch`, covering both the wrapped `{ results: [...] }` and the bare- * array response shapes, index realignment, and provider-shaped errors.+ * Exercises the `/rerank` request/response contract against the shared+ * fake HTTP fixture, covering both the wrapped `{ results: [...] }` and+ * the bare array response shapes, index realignment, and provider-shaped+ * errors. Error shapes beyond the fixture's reach (raw-text bodies,+ * stream-level faults) live in rerank-failure.test.ts.*/-import { afterEach, describe, expect, test } from "bun:test";+import { afterEach, beforeEach, describe, expect, test } from "bun:test";import { CrossEncoderRerankProvider } from "../../../src/core/search/rerank/cross-encoder.ts";import { SearchError } from "../../../src/core/search/types.ts";+import { FAKE_PROVIDER_KEY } from "../../helpers/fake-credentials.ts";+import { startFakeHttp, type FakeHttp, type FakeRequest } from "../../helpers/fake-http.ts";-const ENDPOINT = Object.freeze({- baseUrl: "https://api.example.com/v1",- model: "rerank-1",- apiKey: "secret",-});+const MODEL = "rerank-1";-const realFetch = globalThis.fetch;-afterEach(() => {- globalThis.fetch = realFetch;-});+describe("CrossEncoderRerankProvider", () => {+ let server: FakeHttp;-function stubFetch(handler: (url: string, init: RequestInit) => Response): void {- globalThis.fetch = (async (url: string, init: RequestInit) =>- handler(String(url), init)) as unknown as typeof fetch;-}+ /** The endpoint under test, pointed at the per-test fixture. */+ function endpoint(): { baseUrl: string; model: string; apiKey: string } {+ return { baseUrl: server.url, model: MODEL, apiKey: FAKE_PROVIDER_KEY };+ }++ beforeEach(async () => {+ server = await startFakeHttp();+ });++ afterEach(async () => {+ await server.close();+ });-describe("CrossEncoderRerankProvider", () => {test("posts to {base}/rerank with model, query, documents and bearer auth", async () => {- let seenUrl = "";- let seenBody: Record<string, unknown> = {};- let seenAuth = "";- stubFetch((url, init) => {- seenUrl = url;- seenAuth = (init.headers as Record<string, string>)["authorization"] ?? "";- seenBody = JSON.parse(String(init.body));- return new Response(- JSON.stringify({+ let seen: FakeRequest | undefined;+ server.setHandler((req) => {+ seen = req;+ return {+ status: 200,+ body: {results: [{ index: 0, relevance_score: 0.2 },{ index: 1, relevance_score: 0.9 },],- }),- { status: 200 },- );+ },+ };});- const provider = new CrossEncoderRerankProvider(ENDPOINT);+ const provider = new CrossEncoderRerankProvider(endpoint());const scores = await provider.rerank("hello", ["doc a", "doc b"]);- expect(seenUrl).toBe("https://api.example.com/v1/rerank");- expect(seenAuth).toBe("Bearer secret");- expect(seenBody).toEqual({ model: "rerank-1", query: "hello", documents: ["doc a", "doc b"] });+ const req = seen!;+ expect(req.method).toBe("POST");+ expect(req.path).toBe(`${new URL(server.url).pathname}/rerank`);+ expect(req.headers["authorization"]).toBe(`Bearer ${FAKE_PROVIDER_KEY}`);+ expect(req.body).toEqual({ model: MODEL, query: "hello", documents: ["doc a", "doc b"] });expect(scores).toEqual([0.2, 0.9]);});test("realigns out-of-order results back to input index order", async () => {- stubFetch(- () =>- new Response(- JSON.stringify({- results: [- { index: 2, relevance_score: 0.9 },- { index: 0, relevance_score: 0.1 },- { index: 1, relevance_score: 0.5 },- ],- }),- { status: 200 },- ),- );- const provider = new CrossEncoderRerankProvider(ENDPOINT);+ server.setHandler(() => ({+ status: 200,+ body: {+ results: [+ { index: 2, relevance_score: 0.9 },+ { index: 0, relevance_score: 0.1 },+ { index: 1, relevance_score: 0.5 },+ ],+ },+ }));+ const provider = new CrossEncoderRerankProvider(endpoint());const scores = await provider.rerank("q", ["a", "b", "c"]);expect(scores).toEqual([0.1, 0.5, 0.9]);});+ test("rides the default /rerank route without a per-test handler", async () => {+ // No setHandler: the fixture's default overlap scores flow through the+ // provider's wrapped-shape parser and come back realigned to one+ // score per input document, whatever order the route answered in.+ const provider = new CrossEncoderRerankProvider(endpoint());+ expect(await provider.rerank("fox", ["a fox here", "nothing relevant"])).toEqual([0.5, 0]);+ });+test("accepts the bare array response shape with `score`", async () => {- stubFetch(- () =>- new Response(- JSON.stringify([- { index: 0, score: 0.7 },- { index: 1, score: 0.3 },- ]),- { status: 200 },- ),- );- const provider = new CrossEncoderRerankProvider(ENDPOINT);+ server.setHandler(() => ({+ status: 200,+ body: [+ { index: 0, score: 0.7 },+ { index: 1, score: 0.3 },+ ],+ }));+ const provider = new CrossEncoderRerankProvider(endpoint());expect(await provider.rerank("q", ["a", "b"])).toEqual([0.7, 0.3]);});test("empty documents short-circuits without a request", async () => {- let called = false;- stubFetch(() => {- called = true;- return new Response("[]", { status: 200 });- });- const provider = new CrossEncoderRerankProvider(ENDPOINT);+ const provider = new CrossEncoderRerankProvider(endpoint());expect(await provider.rerank("q", [])).toEqual([]);- expect(called).toBe(false);+ expect(server.callCount()).toBe(0);});test("non-2xx surfaces a RERANK_PROVIDER_HTTP error", async () => {- stubFetch(() => new Response("upstream boom", { status: 503 }));- const provider = new CrossEncoderRerankProvider(ENDPOINT);+ server.setHandler(() => ({ status: 503, body: { error: "upstream boom" } }));+ const provider = new CrossEncoderRerankProvider(endpoint());try {await provider.rerank("q", ["a"]);throw new Error("expected throw");@@ -114,13 +114,11 @@ describe("CrossEncoderRerankProvider", () => {});test("a wrong-length response is rejected", async () => {- stubFetch(- () =>- new Response(JSON.stringify({ results: [{ index: 0, relevance_score: 0.5 }] }), {- status: 200,- }),- );- const provider = new CrossEncoderRerankProvider(ENDPOINT);+ server.setHandler(() => ({+ status: 200,+ body: { results: [{ index: 0, relevance_score: 0.5 }] },+ }));+ const provider = new CrossEncoderRerankProvider(endpoint());let thrown: unknown;try {await provider.rerank("q", ["a", "b"]);diff --git a/tests/core/search/rerank.search.test.ts b/tests/core/search/rerank.search.test.tsindex 5ea9a920..46d0facc 100644--- a/tests/core/search/rerank.search.test.ts+++ b/tests/core/search/rerank.search.test.ts@@ -3,9 +3,12 @@* (retrieval-precision-quality-loop, card A / t_110867f5).** End-to-end: config-enabled rerank widens the candidate pool, calls the- * OpenAI-compatible `/rerank` endpoint (stubbed `fetch`), and re-orders- * the final window. Disabled stays byte-identical; enabled-but-unconfigured- * fails closed; an endpoint error degrades gracefully with a warning.+ * OpenAI-compatible `/rerank` endpoint on the shared fake HTTP fixture,+ * and re-orders the final window. Disabled stays byte-identical;+ * enabled-but-unconfigured fails closed; an endpoint error degrades+ * gracefully with a warning. The one warning pin that names a raw+ * (non-JSON) 503 body stays at the fetch seam, which the fixture cannot+ * express.*/import { test, expect, beforeEach, afterEach, spyOn } from "bun:test";@@ -18,65 +21,87 @@ import {} from "../../../src/core/search/retrieval-trail.ts";import { search } from "../../../src/core/search/search.ts";import { SearchError } from "../../../src/core/search/types.ts";+import {+ startFakeHttp,+ type FakeHttp,+ type FakeRequest,+ type FakeResponseSpec,+} from "../../helpers/fake-http.ts";+import { FAKE_PROVIDER_KEY } from "../../helpers/fake-credentials.ts";import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";let vault: string;let dbPath: string;let cleanup: () => void;+let server: FakeHttp;const realFetch = globalThis.fetch;-beforeEach(() => {+beforeEach(async () => {({ vault, dbPath, cleanup } = createTempVault("rerank-search"));writeMd(vault, "strong.md", "# Strong\n\nfox fox fox fox the quick brown fox jumps high.");writeMd(vault, "weak.md", "# Weak\n\nA note mostly about cats, with one fox mention here.");writeMd(vault, "mid.md", "# Mid\n\nThe fox and the hound; a fox story about foxes.");+ server = await startFakeHttp();});-afterEach(() => {+afterEach(async () => {globalThis.fetch = realFetch;+ await server.close();cleanup();});-const ENABLED = {- enabled: true,- baseUrl: "https://api.example.com/v1",- model: "rerank-1",- apiKey: "secret",- topK: 10,-} as const;--/** Score the document containing `winner` highest, everything else low. */-function stubRerankFetch(winner: string): void {- globalThis.fetch = (async (_url: string, init: RequestInit) => {- const body = JSON.parse(String(init.body)) as { documents: string[] };- const results = body.documents.map((doc, index) => ({- index,- relevance_score: doc.includes(winner) ? 0.99 : 0.01,- }));- return new Response(JSON.stringify({ results }), { status: 200 });- }) as unknown as typeof fetch;+/** The score of the document containing `winner`. */+const WINNER_SCORE = 0.99;+/** The score of every document that does not contain `winner`. */+const REST_SCORE = 0.01;++/** Handler scoring the document that contains `winner` highest. */+function winnerHandler(winner: string): (req: FakeRequest) => FakeResponseSpec {+ return (req) => {+ const documents = (req.body as { documents: string[] }).documents;+ return {+ status: 200,+ body: {+ results: documents.map((doc, index) => ({+ index,+ relevance_score: doc.includes(winner) ? WINNER_SCORE : REST_SCORE,+ })),+ },+ };+ };+}++/** A 503 endpoint answer: rerank degrades to the heuristic order. */+function unavailable(): FakeResponseSpec {+ return { status: 503, body: { error: "upstream boom" } };+}++/** The rerank endpoint config, pointed at the per-test fixture. */+function rerankEnabled() {+ return {+ enabled: true,+ baseUrl: server.url,+ model: "rerank-1",+ apiKey: FAKE_PROVIDER_KEY,+ topK: 10,+ } as const;}test("disabled (default) is byte-identical to the pre-feature baseline", async () => {const cfg = makeConfig({ vault, dbPath });await indexVault(cfg);- let called = false;- globalThis.fetch = (async () => {- called = true;- return new Response("[]", { status: 200 });- }) as unknown as typeof fetch;const base = await search(cfg, { query: "fox", limit: 10 });- expect(called).toBe(false); // zero HTTP cost when disabled+ expect(server.callCount()).toBe(0); // zero HTTP cost when disabledexpect(base.results.length).toBeGreaterThanOrEqual(3);// The heuristic ranker puts the dense "strong" hit first.expect(base.results[0]!.path).toBe("strong.md");});test("enabled: the cross-encoder's top-scored doc lands first", async () => {- const cfg = makeConfig({ vault, dbPath, rerank: ENABLED });+ const cfg = makeConfig({ vault, dbPath, rerank: rerankEnabled() });await indexVault(cfg);// Promote the "cats" note (weak.md) that the heuristic ranker placed low.- stubRerankFetch("cats");+ server.setHandler(winnerHandler("cats"));const out = await search(cfg, { query: "fox", limit: 10 });expect(out.results[0]!.path).toBe("weak.md");expect(out.results[0]!.reasons.some((r) => r.startsWith("cross_encoder: "))).toBe(true);@@ -89,10 +114,9 @@ test("enabled + unconfigured endpoint fails closed", async () => {});test("enabled + endpoint error degrades to heuristic order with a warning", async () => {- const cfg = makeConfig({ vault, dbPath, rerank: ENABLED });+ const cfg = makeConfig({ vault, dbPath, rerank: rerankEnabled() });await indexVault(cfg);- globalThis.fetch = (async () =>- new Response("upstream boom", { status: 503 })) as unknown as typeof fetch;+ server.setHandler(unavailable);const base = await search(makeConfig({ vault, dbPath }), { query: "fox", limit: 10 });const out = await search(cfg, { query: "fox", limit: 10 });expect(out.results.map((r) => r.path)).toEqual(base.results.map((r) => r.path));@@ -101,21 +125,15 @@ test("enabled + endpoint error degrades to heuristic order with a warning", asyn// βββ the rerank liveness code on the retrieval trail βββββββββββββββββββββββββ-/** A 503 endpoint that counts the requests it answered. */-function failingRerankFetch(): { calls: () => number } {- let calls = 0;- globalThis.fetch = (async () => {- calls += 1;- return new Response("upstream boom", { status: 503 });- }) as unknown as typeof fetch;- return { calls: () => calls };-}-test("a failed rerank names rerank-provider-unavailable with its category", async () => {- const cfg = makeConfig({ vault, dbPath, rerank: ENABLED });+ const cfg = makeConfig({ vault, dbPath, rerank: rerankEnabled() });await indexVault(cfg);const base = await search(makeConfig({ vault, dbPath }), { query: "fox", limit: 10 });- failingRerankFetch();+ // The raw (non-JSON) 503 body is the one rerank case the fixture cannot+ // express: the fixture JSON-encodes every body, and this test pins the+ // resulting warning byte-identical to v1.65.0. It stays at the fetch seam.+ globalThis.fetch = (async () =>+ new Response("upstream boom", { status: 503 })) as unknown as typeof fetch;const out = await search(cfg, { query: "fox", limit: 10 });expect(out.results.map((r) => r.path)).toEqual(base.results.map((r) => r.path));// The human signal is byte-identical to v1.65.0.@@ -127,36 +145,30 @@ test("a failed rerank names rerank-provider-unavailable with its category", asyn});test("a failed rerank is served but never cached, so the next query retries", async () => {- const cfg = makeConfig({ vault, dbPath, rerank: ENABLED, cacheEnabled: true });+ const cfg = makeConfig({ vault, dbPath, rerank: rerankEnabled(), cacheEnabled: true });await indexVault(cfg);- const endpoint = failingRerankFetch();+ server.setHandler(unavailable);await search(cfg, { query: "fox", limit: 10 });const second = await search(cfg, { query: "fox", limit: 10 });- expect(endpoint.calls()).toBe(2);+ expect(server.callCount()).toBe(2);expect(second.retrievalTrail?.degraded.map((d) => d.code)).toContain(RETRIEVAL_DEGRADATION.rerankProviderUnavailable,);});test("a healthy rerank is cached as before", async () => {- const cfg = makeConfig({ vault, dbPath, rerank: ENABLED, cacheEnabled: true });+ const cfg = makeConfig({ vault, dbPath, rerank: rerankEnabled(), cacheEnabled: true });await indexVault(cfg);- let calls = 0;- stubRerankFetch("cats");- const scoring = globalThis.fetch;- globalThis.fetch = (async (...args: Parameters<typeof fetch>) => {- calls += 1;- return scoring(...args);- }) as unknown as typeof fetch;+ server.setHandler(winnerHandler("cats"));await search(cfg, { query: "fox", limit: 10 });await search(cfg, { query: "fox", limit: 10 });- expect(calls).toBe(1);+ expect(server.callCount()).toBe(1);});test("a healthy rerank and a disabled rerank carry no trail", async () => {- const enabled = makeConfig({ vault, dbPath, rerank: ENABLED });+ const enabled = makeConfig({ vault, dbPath, rerank: rerankEnabled() });await indexVault(enabled);- stubRerankFetch("cats");+ server.setHandler(winnerHandler("cats"));expect((await search(enabled, { query: "fox", limit: 10 })).retrievalTrail).toBeUndefined();const disabled = makeConfig({ vault, dbPath });expect((await search(disabled, { query: "fox", limit: 10 })).retrievalTrail).toBeUndefined();diff --git a/tests/core/search/semantic-phase-spend-gate.test.ts b/tests/core/search/semantic-phase-spend-gate.test.tsindex b2a76015..b7e49edf 100644--- a/tests/core/search/semantic-phase-spend-gate.test.ts+++ b/tests/core/search/semantic-phase-spend-gate.test.ts@@ -295,6 +295,35 @@ test("an empty-fit lane with no keyword hit still reads as a stopped lane", asynexpect(semanticLaneMissing(out.degraded.map((d) => d.code))).toBe(true);});+test("the lane reads the keyword count at completion, not at start", async () => {+ // The overlapped caller (recall-lane-latency) starts this lane before+ // the keyword lanes run, so the count it holds at start time is zero by+ // construction. The lane must therefore read the count when the embed+ // settles - by then the keyword lanes have produced their hits - and a+ // count captured at start would silently drop the hybrid_degraded+ // signal the contract owes the caller.+ let keywordHits = 0;+ server.setHandler(() => {+ // The keyword lanes complete while this embed is in flight.+ keywordHits = 3;+ return { status: 500, body: { error: { message: "embed down" } } };+ });+ const out = await runSemanticLane({+ ...laneInput(gatedConfig(), "q", "local"),+ keywordHitCount: () => keywordHits,+ });+ expect(out.attempted).toBe(false);+ expect(codes(out.degraded)).toContain(RETRIEVAL_DEGRADATION.hybridDegraded);+ expect(out.warnings.some((w) => w.startsWith("hybrid_degraded:"))).toBe(true);+});++test("the keyword count still accepts a plain number and grades the signal with it", async () => {+ const hit = await runSemanticLane({ ...laneInput(gatedConfig(), "q"), keywordHitCount: 2 });+ expect(codes(hit.degraded)).toContain(RETRIEVAL_DEGRADATION.hybridDegraded);+ const none = await runSemanticLane({ ...laneInput(gatedConfig(), "q"), keywordHitCount: 0 });+ expect(codes(none.degraded)).not.toContain(RETRIEVAL_DEGRADATION.hybridDegraded);+});+// ββ search() over a real index βββββββββββββββββββββββββββββββββββββββββββββββconst NOTE = "# Fox\n\nThe quick brown fox jumps over the lazy dog.";diff --git a/tests/helpers/fake-http.test.ts b/tests/helpers/fake-http.test.tsindex 0565f842..4737a548 100644--- a/tests/helpers/fake-http.test.ts+++ b/tests/helpers/fake-http.test.ts@@ -1,7 +1,10 @@/*** The fake HTTP server's own contract: a handler that throws or rejects* answers 500 with its message, so a test fails on the real error rather- * than timing out on a request nobody answers.+ * than timing out on a request nobody answers. The default handler's+ * `/rerank` route contract is pinned here too: deterministic scores, one+ * per document, answered in score order like the real providers, with+ * malformed bodies failing visibly.*/import { afterEach, expect, test } from "bun:test";@@ -15,6 +18,20 @@ afterEach(async () => {server = null;});+function postJson(base: string, path: string, body: unknown): Promise<Response> {+ return fetch(`${base}${path}`, {+ method: "POST",+ headers: { "content-type": "application/json" },+ body: JSON.stringify(body),+ });+}++const RERANK_REQUEST = Object.freeze({+ model: "rerank-1",+ query: "fox runs",+ documents: Object.freeze(["a fox here", "nothing relevant", "fox runs far"]),+});+test("a rejecting handler answers 500 with the error message", async () => {server = await startFakeHttp();server.setHandler(async () => {@@ -34,3 +51,64 @@ test("a synchronously throwing handler answers the same way", async () => {expect(res.status).toBe(500);expect(await res.json()).toEqual({ error: "sync failure" });});++test("the default handler answers POST /rerank with one deterministic score per document", async () => {+ server = await startFakeHttp();+ const base = server.url;+ const res = await postJson(base, "/rerank", RERANK_REQUEST);+ expect(res.status).toBe(200);+ const body = (await res.json()) as {+ model: string;+ results: { index: number; relevance_score: number }[];+ };+ expect(body.model).toBe("rerank-1");+ // One item per document, in SCORE order: the default route sorts by+ // relevance_score descending, stable on index, the way real providers+ // answer, and the rerank client realigns the items through their own+ // `index`. "fox runs far" shares two query tokens (2/3), "a fox here"+ // one (1/2), "nothing relevant" none (0).+ expect(body.results.map((r) => r.index)).toEqual([2, 0, 1]);+ expect(body.results.map((r) => r.relevance_score)).toEqual([2 / 3, 0.5, 0]);+ // The same request answers byte-identically: no randomness, no clock.+ const replay = await postJson(base, "/rerank", RERANK_REQUEST);+ expect(replay.status).toBe(200);+ expect(await replay.text()).toBe(JSON.stringify(body));+});++test("malformed /rerank bodies fail visibly instead of scoring", async () => {+ server = await startFakeHttp();+ const base = server.url;+ // No documents key.+ expect((await postJson(base, "/rerank", { model: "m", query: "q" })).status).toBe(400);+ // documents is not an array.+ expect((await postJson(base, "/rerank", { query: "q", documents: "a fox" })).status).toBe(400);+ // A document entry that is not a string.+ expect((await postJson(base, "/rerank", { query: "q", documents: ["a", 3] })).status).toBe(400);+ // No query key.+ expect((await postJson(base, "/rerank", { model: "m", documents: ["a"] })).status).toBe(400);+ // An empty document list is well-formed and scores nothing (the client+ // short-circuits before sending; this is defensive).+ const empty = await postJson(base, "/rerank", { model: "m", query: "q", documents: [] });+ expect(empty.status).toBe(200);+ expect(await empty.json()).toEqual({ results: [], model: "m" });+});++test("GET /rerank is not routed by the default handler", async () => {+ server = await startFakeHttp();+ const res = await fetch(`${server.url}/rerank`);+ expect(res.status).toBe(404);+ expect(await res.json()).toEqual({ error: "not_found" });+});++test("the /embeddings default route is unchanged by the /rerank route", async () => {+ server = await startFakeHttp();+ const res = await postJson(server.url, "/embeddings", { input: ["a b c"], model: "emb-1" });+ expect(res.status).toBe(200);+ const body = (await res.json()) as {+ model: string;+ data: { index: number; embedding: number[] }[];+ };+ expect(body.model).toBe("emb-1");+ expect(body.data[0]?.index).toBe(0);+ expect(body.data[0]?.embedding).toEqual([3, 0, 5, 1]);+});diff --git a/tests/helpers/fake-http.ts b/tests/helpers/fake-http.tsindex 244d6f20..0dfa1e43 100644--- a/tests/helpers/fake-http.ts+++ b/tests/helpers/fake-http.ts@@ -34,8 +34,74 @@ export interface FakeHttp {callCount: () => number;}+/** Path tail the default handler serves embeddings on. */+const EMBEDDINGS_ROUTE = "/embeddings";+/** Path tail the default handler serves rerank on. */+const RERANK_ROUTE = "/rerank";+/** Model name echoed when a request carries none. */+const DEFAULT_FAKE_MODEL = "fake-model";+/**+ * Smoothing that keeps the overlap score in [0, 1) and monotone in the+ * number of shared tokens: matches / (matches + SMOOTHING).+ */+const RERANK_OVERLAP_SMOOTHING = 1;++/** Lowercased word tokens with punctuation stripped, deduplicated. */+function tokensOf(text: string): Set<string> {+ const out = new Set<string>();+ for (const word of text.toLowerCase().split(/\s+/)) {+ const token = word.replace(/\p{P}+/gu, "");+ if (token !== "") out.add(token);+ }+ return out;+}++/**+ * Deterministic query-document score from token overlap, in [0, 1) and+ * monotone in the number of tokens the two share. No randomness and no+ * clock, so identical requests replay byte-identically.+ */+function overlapScore(query: string, document: string): number {+ const queryTokens = tokensOf(query);+ let matches = 0;+ for (const token of tokensOf(document)) {+ if (queryTokens.has(token)) matches += 1;+ }+ return matches / (matches + RERANK_OVERLAP_SMOOTHING);+}++interface RerankRequestBody {+ model?: string;+ query?: unknown;+ documents?: unknown;+}++function rerankRoute(req: FakeRequest): FakeResponseSpec {+ const body = (req.body ?? {}) as RerankRequestBody;+ if (typeof body.query !== "string" || !Array.isArray(body.documents)) {+ return { status: 400, body: { error: "rerank request: query and documents are required" } };+ }+ if (!body.documents.every((d) => typeof d === "string")) {+ return { status: 400, body: { error: "rerank request: documents must be strings" } };+ }+ const query = body.query;+ const scored = (body.documents as string[]).map((document, index) => ({+ index,+ relevance_score: overlapScore(query, document),+ }));+ // Real providers (Cohere, Jina, TEI) answer in score order, not input+ // order, so the default route does too: sorted by relevance_score+ // descending, stable on index (Array#sort is stable). The rerank+ // client realigns the items back through each item's own `index`, and+ // going through the default route keeps that realignment exercised by+ // every suite that rides the shared fixture, not only by the suites+ // that install an explicit out-of-order handler.+ const results = scored.sort((a, b) => b.relevance_score - a.relevance_score);+ return { status: 200, body: { results, model: body.model ?? DEFAULT_FAKE_MODEL } };+}+function defaultHandler(req: FakeRequest): FakeResponseSpec {- if (req.path.endsWith("/embeddings") && req.method === "POST") {+ if (req.path.endsWith(EMBEDDINGS_ROUTE) && req.method === "POST") {const body = (req.body ?? {}) as { input?: string[]; model?: string };const inputs = Array.isArray(body.input) ? body.input : [];const data = inputs.map((text, index) => {@@ -44,7 +110,10 @@ function defaultHandler(req: FakeRequest): FakeResponseSpec {const v = [tokens, index, text.length, 1];return { object: "embedding", embedding: v, index };});- return { status: 200, body: { data, model: body.model ?? "fake-model" } };+ return { status: 200, body: { data, model: body.model ?? DEFAULT_FAKE_MODEL } };+ }+ if (req.method === "POST" && req.path.endsWith(RERANK_ROUTE)) {+ return rerankRoute(req);}return { status: 404, body: { error: "not_found" } };}diff --git a/tests/mcp/context-pack-packing-tool.test.ts b/tests/mcp/context-pack-packing-tool.test.tsnew file mode 100644index 00000000..3e121ea1--- /dev/null+++ b/tests/mcp/context-pack-packing-tool.test.ts@@ -0,0 +1,256 @@+/**+ * MCP surface of the opt-in packing policy on `brain_context_pack`+ * (recall-lane-latency wave).+ *+ * The knob rides the existing tool - a new argument, never a new tool -+ * and mirrors the `query_mode` discipline argument for argument:+ *+ * 1. The schema enum DERIVES from the frozen core tuple, ends its+ * description, and declares the query pairing `dependentRequired`+ * style so a schema-driven client discovers it without a refusal.+ * 2. A non-member is refused with INVALID_PARAMS naming the frozen+ * members.+ * 3. A policy without a query - absent, blank or whitespace-only - is+ * refused: a policy with nothing to order is inert, exactly like a+ * mode with nothing to read.+ * 4. The omitted argument is byte-identical: no `packing` member+ * exists on the response and the unified order is untouched.+ * 5. With a policy, the response carries the `packing` accounting -+ * policy name, the reserved attention charge and both lane spends -+ * and only then.+ * 6. A policy against a one-lane vault is refused by name through the+ * wire as INVALID_PARAMS, never a silent no-op.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { JSONRPC_VERSION, MCPServer, PROTOCOL_VERSION } from "../../src/mcp/index.ts";+import { buildToolTable } from "../../src/mcp/tools.ts";+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";+import { PACKING_POLICIES } from "../../src/core/brain/recall-budget.ts";++let tmp: string;+let vault: string;+let configHome: string;+let configPath: string;+const savedEnv: Record<string, string | undefined> = {};++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-mcp-pack-packing-"));+ vault = join(tmp, "vault");+ mkdirSync(join(vault, "Brain", "preferences"), { recursive: true });+ mkdirSync(join(vault, "Brain", "retired"), { recursive: true });+ configHome = mkdtempSync(join(tmpdir(), "o2b-mcp-pack-packing-cfg-"));+ configPath = join(configHome, "config.yaml");+ for (const k of ["VAULT_AGENT_NAME", "VAULT_TIMEZONE", "VAULT_DIR", "OPEN_SECOND_BRAIN_CONFIG"]) {+ savedEnv[k] = process.env[k];+ delete process.env[k];+ }+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = configPath;+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+ for (const [k, v] of Object.entries(savedEnv)) {+ if (v === undefined) delete process.env[k];+ else process.env[k] = v;+ }+});++async function initialize(server: MCPServer): Promise<void> {+ await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 1,+ method: "initialize",+ params: {+ protocolVersion: PROTOCOL_VERSION,+ capabilities: {},+ clientInfo: { name: "pack-packing-test", version: "0" },+ },+ });+ await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ method: "notifications/initialized",+ });+}++async function callPack(+ server: MCPServer,+ args: Record<string, unknown>,+): Promise<Record<string, unknown>> {+ const r = (await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 9,+ method: "tools/call",+ params: { name: "brain_context_pack", arguments: args },+ })) as { result: { content: ReadonlyArray<{ type: string; text: string }> } };+ return JSON.parse(r.result.content[0]!.text) as Record<string, unknown>;+}++async function callPackRaw(+ server: MCPServer,+ args: Record<string, unknown>,+): Promise<{ error?: { code: number; message: string } }> {+ return (await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 11,+ method: "tools/call",+ params: { name: "brain_context_pack", arguments: args },+ })) as { error?: { code: number; message: string } };+}++/** One preference page and one retired page whose topics share the fixture query. */+function writeBothLanes(): void {+ writeFileSync(+ join(vault, "Brain", "preferences", "pref-fix.md"),+ "---\nid: pref-fix\ntopic: packing fixture\nprinciple: the distilled preference layer\ntier: core\ncreated_at: 2026-03-01T00:00:00Z\n---\n",+ );+ writeFileSync(+ join(vault, "Brain", "retired", "ret-fix.md"),+ "---\nid: ret-fix\ntopic: packing fixture\nprinciple: a retired page\ntier: core\ncreated_at: 2026-01-01T00:00:00Z\n---\n",+ );+}++const FIXTURE_QUERY = "packing fixture";++describe("brain_context_pack packing_policy declaration", () => {+ test("the schema enum derives from the frozen tuple and states the pairing", () => {+ const tool = buildToolTable("full").find((t) => t.name === "brain_context_pack")!;+ const schema = tool.inputSchema as {+ properties: Record<string, { type?: string; enum?: string[]; description?: string }>;+ dependentRequired?: Record<string, ReadonlyArray<string>>;+ additionalProperties?: boolean;+ };+ const declared = schema.properties["packing_policy"]!;+ expect(declared.type).toBe("string");+ expect(declared.enum).toEqual([...PACKING_POLICIES]);+ expect(declared.description).toContain("facts_first");+ expect(declared.description).toContain("retired_first");+ // The description does not sell `facts_first` as the omitted default:+ // it states the up-front reserved charge and the one-lane refusal the+ // omitted call does not make (external review M4).+ expect(declared.description).toContain("Neither is the omitted default");+ expect(declared.description).toContain("one lane has no candidate");+ expect(declared.description!.endsWith(".")).toBe(true);+ expect(schema.additionalProperties).toBe(false);+ expect(schema.dependentRequired!["packing_policy"]).toEqual(["query"]);+ });+});++describe("brain_context_pack packing_policy refusals", () => {+ test("a non-member is refused naming the frozen members", async () => {+ const server = new MCPServer({ vault, configPath });+ await initialize(server);+ const r = await callPackRaw(server, {+ max_tokens: 10_000,+ query: FIXTURE_QUERY,+ packing_policy: "budget_policy",+ });+ expect(r.error?.code).toBe(-32602);+ expect(r.error!.message).toContain("packing_policy");+ for (const member of PACKING_POLICIES) {+ expect(r.error!.message).toContain(member);+ }+ });++ test("a policy without a query is refused as inert", async () => {+ const server = new MCPServer({ vault, configPath });+ await initialize(server);+ const r = await callPackRaw(server, { max_tokens: 10_000, packing_policy: "facts_first" });+ expect(r.error?.code).toBe(-32602);+ expect(r.error!.message).toContain("packing_policy");+ expect(r.error!.message).toContain("query");+ expect(r.error!.message).toContain("inert");+ });++ test("a policy with a blank or whitespace-only query is refused as inert exactly like an absent one", async () => {+ // An empty string survives the raw read, is dropped before the core+ // as falsy, and the null query filter-passes every candidate - so the+ // policy would silently reorder an unordered pack. Whitespace survives+ // the drop and substring-passes every haystack. Both are the same+ // nothing-to-order call the absent-query refusal names.+ writeBothLanes();+ const server = new MCPServer({ vault, configPath });+ await initialize(server);+ for (const query of ["", " ", " "]) {+ const r = await callPackRaw(server, {+ max_tokens: 10_000,+ query,+ packing_policy: "retired_first",+ });+ expect(r.error?.code).toBe(-32602);+ expect(r.error!.message).toContain("packing_policy");+ expect(r.error!.message).toContain("query");+ expect(r.error!.message).toContain("inert");+ }+ });++ test("a policy against a one-lane vault is refused by name as INVALID_PARAMS", async () => {+ // Facts lane only: the retired directory holds nothing, so the+ // pages lane is empty and the policy would silently do nothing.+ writeFileSync(+ join(vault, "Brain", "preferences", "pref-fix.md"),+ "---\nid: pref-fix\ntopic: packing fixture\nprinciple: the distilled layer\ntier: core\n---\n",+ );+ const server = new MCPServer({ vault, configPath });+ await initialize(server);+ const r = await callPackRaw(server, {+ max_tokens: 10_000,+ query: FIXTURE_QUERY,+ packing_policy: "facts_first",+ });+ expect(r.error?.code).toBe(-32602);+ expect(r.error!.message).toContain("pages lane is empty");+ expect(r.error!.message).toContain("facts_first");+ });+});++describe("brain_context_pack packing_policy response", () => {+ test("the omitted argument keeps the response byte-identical: no packing member", async () => {+ writeBothLanes();+ const server = new MCPServer({ vault, configPath });+ await initialize(server);+ const out = await callPack(server, { max_tokens: 10_000, query: FIXTURE_QUERY });+ expect(Object.hasOwn(out, "packing")).toBe(false);+ expect(JSON.stringify(out)).not.toContain('"packing"');+ // The unified comparator order, untouched by the absent option.+ const items = out["items"] as Array<{ id: string }>;+ expect(items.map((i) => i.id)).toEqual(["pref-fix", "ret-fix"]);+ });++ test("a named policy adds the packing accounting with both lane spends", async () => {+ writeBothLanes();+ const server = new MCPServer({ vault, configPath });+ await initialize(server);+ const out = await callPack(server, {+ max_tokens: 10_000,+ query: FIXTURE_QUERY,+ packing_policy: "facts_first",+ });+ const packing = out["packing"] as {+ policy: string;+ reservedTokens: number;+ pages: { allocation: number; spent: number; candidates: number; items: number };+ facts: { allocation: number; spent: number; candidates: number; items: number };+ };+ expect(packing.policy).toBe("facts_first");+ // The reserved charge rides the wire accounting too; zero here, since+ // the fixture requests no attention block.+ expect(packing.reservedTokens).toBe(0);+ expect(packing.facts.items).toBe(1);+ expect(packing.pages.items).toBe(1);+ expect(packing.facts.spent + packing.pages.spent).toBe(out["tokens_used"] as number);+ expect(packing.facts.allocation + packing.pages.allocation + packing.reservedTokens).toBe(+ 10_000,+ );+ // Facts first: the preference page leads despite the retired page's+ // unified-order slot.+ const items = out["items"] as Array<{ id: string }>;+ expect(items.map((i) => i.id)).toEqual(["pref-fix", "ret-fix"]);+ });+});diff --git a/tests/mcp/context-pack-tool.test.ts b/tests/mcp/context-pack-tool.test.tsindex bc6cc91c..6ee01123 100644--- a/tests/mcp/context-pack-tool.test.ts+++ b/tests/mcp/context-pack-tool.test.ts@@ -722,6 +722,29 @@ describe("brain_context_pack tool β semantic query mode", () => {},);+ test.skipIf(!VEC_LOADABLE)(+ "a lane-empty refusal in semantic mode discloses the spent query embed",+ async () => {+ writeLocalSemanticConfig();+ writeBelief("pref-deploy", "never ship on friday");+ await indexBeliefs(true);+ const server = new MCPServer({ vault, configPath });+ await initialize(server);+ // The retired directory holds no page, so the partition refuses+ // whichever order was named - after the query embed was paid.+ const r = await callPackRpc(server, {+ ...SEMANTIC_ARGS,+ packing_policy: "facts_first",+ });+ expect(readRpcErrorCode(r)).toBe("invalid_params");+ const message = r.error!.message;+ expect(message).toContain("packing_policy facts_first needs both lanes");+ // The one ungated spend is named in the refusal, not dropped from it.+ expect(message).toContain("the query embed was still spent");+ expect(message).toContain(LOCAL_EMBEDDING_MODEL);+ },+ );+test("a disabled tier surfaces EMBEDDING_DISABLED on the wire", async () => {writeBelief("pref-brevity", "keep answers short and brief");const server = new MCPServer({ vault, configPath });diff --git a/uv.lock b/uv.lockindex 363b05e5..996e199e 100644--- a/uv.lock+++ b/uv.lock@@ -4,5 +4,5 @@ requires-python = ">=3.11"[[package]]name = "open-second-brain"-version = "1.83.0"+version = "1.84.0"source = { editable = "." }