Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: retrieval & ranking quality (v1.28.0) (#133)
Sol Aitkenac26a6754d2f
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex a3d57f8b..1b12a869 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.27.1",+ "version": "1.28.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 4e0ed035..62f1241f 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.27.1",+ "version": "1.28.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","skills": "./skills","hooks": "./hooks/hooks.json",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 27d517cc..b907c032 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,9 +5,26 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).-## [1.27.1] - 2026-07-10+## [1.28.0] - 2026-07-10++A retrieval and ranking quality release: embedding-layer robustness, faster and smarter candidate retrieval, a fully offline reranker with a per-store eval gate, ranking signals learned from real outcomes, and a named benchmark to score the whole. Every new surface is additive and off by default where it changes an existing path; the release adds no new dependency and no ML runtime, and the kernel still calls no LLM.++### Added++- **Curated embedding-model presets and a recommended default.** A static, shippable catalog of known-good multilingual-first embedding models (with native dimensions and guidance) is surfaced at provider registration via `o2b search provider presets`; `provider add` defaults `--model` to the recommended model when omitted. Advisory only - a custom `--model` stays first-class and verbatim. No server, no network.+- **Native ZeroEntropy embedding provider.** `zeroentropy` is now a first-class embedding provider calling the native `POST {base}/models/embed` API directly over `fetch` (no SDK, no dependency), with Bearer auth, input-order-preserving vectors, and Matryoshka `dimensions` when configured. It reuses the shared retry/backoff and `ping` contract; the resolved-provider union stays closed.+- **Trigram candidate prefilter for large-vault search.** A native SQLite FTS5 `trigram` shadow index (schema v9, populated in-place by a reindex-safe migration, no dependency) backs an opt-in prefilter that merges substring / partial-token candidates the word tokenizer misses into the keyword pool - a strict superset of substring matches, so it never drops a result. It engages only past a corpus-size floor and skips short, CJK, and low-selectivity queries. Off by default (byte-identical); knobs fold into the query-cache fingerprint.+- **Graph-index query pre-pass with multi-hop BFS.** A new read planner (`o2b search plan`) ranks seed notes by title match and wikilink degree over index metadata, connects them through multi-hop BFS on the link graph, and returns a `should_read` shortlist - surfacing notes reachable only via several hops that the 1-hop link boost misses. An `index_only` mode answers from title / degree / link metadata with zero note bodies hydrated. Pure-stdlib.+- **Bundled offline reranker and per-store reranker eval gate.** A deterministic, dependency-free `local` reranker (coverage / proximity / density blend) behind the existing rerank interface makes reranking work fully offline with no model runtime or network, selected via `search_rerank_kind=local`. A new evaluation gate runs the recall benchmark rerank-off vs rerank-on and recommends enabling only when it lifts ranking without regressing hit@k.+- **Session-end observed-use verdict feeding recall ranking.** A new `recall_observed_use` continuity record captures a USED / IGNORED / CONTRADICTED verdict per injected memory - classified deterministically from transcript token overlap plus a structural stance flip (no LLM) or supplied by the host via the new `brain_observed_use` MCP tool (mirroring `brain_apply_evidence`). The folded observed-reuse rate becomes a preferred, capped recall-ranking signal so demonstrably reused memories outrank merely-predicted-important ones; contradictions demote. Inspectable via `brain_recall_telemetry observed_reuse`. Byte-identical when no verdicts exist.+- **LoCoMo benchmark suite.** `o2b brain bench memory --suite locomo` converts a LoCoMo-shaped dataset into the existing staged harness's fixture and runs it as a parallel named suite (`locomo-*`), so runs are comparable to the published long-conversation-memory literature. OSB's six categories stay canonical; the run is deterministic and network-free by default, with the LLM judge opt-in. Ships a small committed sample dataset.++### Changed++- **Multi-key fallback for the embeddings API key.** A provider profile's `envKey` may now be an ordered probe list (a single string still works and stays byte-identical); resolution takes the first present key and the `OpenAICompatProvider` fails over to the next on an HTTP 401/403, pinning the first that authenticates. `o2b search provider add --env-key A,B` registers a probe list. Rerank stays single-key. Removes the single point of failure in the only networked path without touching the LLM-free kernel.+- **Outcome-validated procedural recall.** Procedural entries and their usage sidecar gain additive success / failure counters (outcome-free vaults stay byte-identical), a host-supplied `recordProceduralOutcome` (deterministic, no LLM), and success-rate ranking that surfaces proven procedures and sinks failing ones, with the usage count as the neutral prior for unproven ones. Exposed via `brain_procedural_memory mark_outcome` and `list ranked:true`.+-A robustness fix for the Hermes memory provider's executable resolution. No kernel behavior changes.### Fixed@@ -6412,6 +6429,7 @@ plugin config (vault field)`, and exits with a clear- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.28.0]: https://github.com/itechmeat/open-second-brain/compare/v1.27.1...v1.28.0[1.27.1]: https://github.com/itechmeat/open-second-brain/compare/v1.27.0...v1.27.1[1.27.0]: https://github.com/itechmeat/open-second-brain/compare/v1.26.1...v1.27.0[1.24.0]: https://github.com/itechmeat/open-second-brain/compare/v1.23.1...v1.24.0diff --git a/docs/brainstorm/retrieval-ranking-quality/design.md b/docs/brainstorm/retrieval-ranking-quality/design.mdnew file mode 100644index 00000000..d27efe61--- /dev/null+++ b/docs/brainstorm/retrieval-ranking-quality/design.md@@ -0,0 +1,240 @@+# Retrieval & Ranking Quality - Design++**Status:** draft+**Author:** feature-release-playbook+**Audience:** implementation++## Problem statement++Open Second Brain retrieves and ranks well for small vaults but has four+structural gaps that limit retrieval quality and, critically, our ability to+*measure* it:++1. **Embedding-layer fragility.** The opt-in semantic layer resolves exactly one+ API key and hard-fails (`EMBEDDING_KEY_MISSING`) if it is absent or invalid;+ users get no guidance on which model to register; and only OpenAI-compatible+ endpoints are supported, so specialized providers require an extra HTTP hop.+2. **Candidate retrieval does not scale or plan.** Every query hydrates and+ scores the FTS union vector union pool with no cheap narrowing stage for large+ vaults, and there is no read-planning pre-pass that decides *which* notes are+ worth opening or answers from index metadata alone. Link reasoning is 1-hop.+3. **Reranking requires the network.** The only reranker+ (`CrossEncoderRerankProvider`) needs a configured remote endpoint, so recall+ quality degrades to zero-rerank for offline/local-only vaults, and nothing+ verifies that reranking actually helps a given store before enabling it.+4. **Ranking has no outcome signal.** Recall ranking is predicted-importance and+ recency driven; procedural memory ranks by raw usage count. Neither knows+ whether an injected memory was actually *used* or whether a procedure actually+ *worked*, so memories that are repeatedly injected but never used keep+ floating up, and a runbook that reliably fails cannot sink.++And spanning all of it: OSB's staged bench harness is MemoryBench-inspired with+its own categories, not aligned to any named, published benchmark, so its numbers+are not comparable to the memory-agent literature.++## Scope++Nine leaf tasks, implemented on one branch (`feat/retrieval-ranking-quality`):++- `t_2f99725a` - multi-key fallback for the embeddings API key.+- `t_33f30fdc` - curated embedding-model presets + recommended default.+- `t_8d49f059` - native ZeroEntropy embedding provider.+- `t_4a672b84` - trigram candidate prefilter for large-vault search.+- `t_59ae326f` - graph-index BFS query pre-pass + should-read shortlist + index-only.+- `t_9f95ebb6` - bundled offline reranker + per-store reranker eval gate.+- `t_65588d8b` - session-end observed-use verdict feeding recall ranking.+- `t_703f7b18` - outcome-validated procedural recall ranked by success rate.+- `t_8dabe2b0` - LoCoMo benchmark suite over the existing bench harness.++## Out of scope++- Any core-path rewrite / unified retrieval planner (Variant 2). Each feature is+ an isolated, gated stage.+- Rebuilding the bench harness. LoCoMo is an additive loader over the unchanged+ staged pipeline.+- Pulling the children of in-scope parents: `t_267f3b4c` (reranker fit-check),+ `t_7d5a3589` (tombstone/usage slices) stay out.+- `HYATLAS_HOME`-style runtime-root consolidation - OSB is state-in-vault by+ design.+- On-chain / Solana anchoring - never in scope for this project.+- Making an LLM judge the canonical bench score. The judge stays advisory and+ fail-open; default runs are deterministic and network-free.++## Hard invariant: zero new dependencies, no heavy runtime++**This release adds NO new npm dependencies and NO ML runtime.** The core today+has exactly one runtime dependency (`proper-lockfile`) and no model runtime+(no `onnxruntime`, `transformers.js`/`@xenova`, `torch`, `tokenizers`). Every+feature here is implementable in pure TypeScript over the standard library plus+the existing optional native deps.++The one task whose upstream framing implies a heavy dependency is the offline+reranker (`t_9f95ebb6`, upstream: "ms-marco-MiniLM-L6 cross-encoder"). A real+neural cross-encoder would require a model runtime and a bundled model file+(tens to hundreds of MB, native binaries). We do NOT take that path. Instead,+mirroring the existing local embedder (`local-provider.ts`, a pure+feature-hashing character-trigram embedder with no model and no dependency), the+bundled offline reranker is a **deterministic pure-TypeScript reranker**: a+cross-encoder-shaped scorer built from lexical/structural query-document features+(term overlap, phrase proximity, coverage, field/position signals) behind the+existing `RerankProvider` interface. This is exactly the "no external dependency+in the core" ethos the task itself cites, and the per-store eval gate exists+precisely to prove the reranker helps a given store before it is enabled.++ZeroEntropy is a native HTTP provider (uses `fetch`, no SDK). Trigram prefilter,+BFS pre-pass, LoCoMo loader, multi-key, presets, and the outcome-signal features+are all pure TypeScript. If any implementation appears to need a heavy+dependency, the feature is redesigned in the OSB idiom, not the dependency added.++## Chosen approach++**Variant 1 - additive, opt-in, byte-identical-when-off** (see `variants.md`).+Every feature is a new stage or module that defaults OFF and, when disabled,+leaves existing behavior byte-identical (the repo-wide "disabled -> identical+reference" discipline). Implementation follows the sequence hint so later work+builds on earlier: embeddings -> candidate retrieval -> reranking -> outcome+signals -> benchmark last (so LoCoMo scores the improved-but-optional stack).++## Design decisions++- **Multi-key as an ordered probe list, backward-compatible.**+ `ProviderProfile.envKey` accepts either the existing single string or an+ ordered list; `expandRegisteredProvider` resolves the first env var that is+ present and non-empty. The explicit+ `OPEN_SECOND_BRAIN_EMBEDDING_KEY`/`embedding_api_key` path still wins first.+ Probing is lazy (resolve the first present key; validation happens on first+ real embed call via the existing `ping`/retry path) to avoid a startup auth+ storm. Single-key configs stay byte-identical. Scope stays strictly in the+ embeddings layer; the LLM-free kernel is untouched.++- **Presets as a static, advisory catalog consulted at registration.** A+ shippable `EMBEDDING_MODEL_PRESETS` list (curated multilingual-first models ++ a recommended default) lives beside the existing pricing/identity table in+ `embeddings/`. The CLI `provider add` flow gains `--model` preset resolution+ and a `--list-presets` affordance; custom `--model` entry remains first-class.+ No server, no network, no new subsystem.++- **ZeroEntropy as a native closed-union provider.** Add `"zeroentropy"` to the+ `EmbeddingProvider` factory, the resolved-config provider union, and+ `parseProvider`/`BUILTIN_PROVIDERS`, with a native provider class calling the+ ZeroEntropy embed API over `fetch` (no SDK). It reuses the retry/backoff and+ `ping` contract of the interface. Reserve the name in+ `RESERVED_PROVIDER_NAMES`.++- **Trigram prefilter as a strict-superset narrowing stage.** A new in-memory /+ fts5-trigram candidate producer runs ahead of full scoring for large vaults,+ contributing to (never subtracting from) the candidate pool, with automatic+ fallback to the full scan for short, CJK, or low-selectivity queries. Acceptance+ proves set-containment: the prefiltered candidate set is a superset of what the+ exhaustive scorer would surface on fixtures. Gated behind a default-off+ `search_trigram_prefilter*` config key; folded into the query-cache fingerprint.++- **Graph-index BFS pre-pass reuses existing graph primitives.** A new pre-pass+ ranks candidate notes by title/tag/summary match + wikilink degree + tier,+ connects top candidates via multi-hop BFS over the link graph+ (`link-graph/graph-index.ts` snapshot + a BFS modeled on+ `search/traversal.ts:expandByTraversal`), and returns a `should_read` shortlist.+ An `index_only` mode answers from index metadata (title/tag/summary/degree) with+ zero note bodies read. Exposed as an opt-in mode; the default query path is+ unchanged.++- **Offline reranker + per-store eval gate.** New `LocalRerankProvider`+ (pure-TS, deterministic) selected via a new `kind`/`provider` field on+ `ResolvedRerankConfig` so `makeRerankProvider` can branch to it with no network.+ The per-store eval gate is a small new module that runs the existing+ `runRecallBenchmark` twice (rerank off vs on) and compares metric families+ against thresholds to decide whether reranking helps that store - reusing the+ benchmark runner rather than extending gate-telemetry.++- **Observed-use verdict as a new continuity kind + host-supplied MCP tool.**+ No-LLM kernel: the verdict (USED / IGNORED / CONTRADICTED) is either computed+ deterministically (match injected-memory anchors/content against later+ transcript spans, reusing `deriveNoteStance` for CONTRADICTED) or accepted from+ the host via a new MCP write tool mirroring `brain_apply_evidence`. Persisted as+ a new `recall_observed_use` continuity kind correlated by `session_id`/`turn_id`+ to the existing `recall_telemetry` rows. The aggregated observed-reuse rate+ becomes the preferred recall-ranking signal (a capped ranker input,+ `reuseRateByDoc`), with predicted importance as the fallback prior. The blend is+ inspectable via `brain_recall_telemetry`. CONTRADICTED routes into the existing+ contradiction/declared-thesis path.++- **Procedural success-rate ranking as additive counters.** Add+ `successCount`/`failureCount` (derived `successRate`) to+ `ProceduralMemoryEntry` and the `usage.jsonl` sidecar (additive; old lines+ parse). A `recordProceduralOutcome(vault, id, success|failure)` (host-supplied+ enum, deterministic) mirrors `markProceduralMemoryUsed`; a+ `brain_procedural_memory` `mark_outcome` operation mirrors `mark_used`. Recall+ ranks by `successRate` primary with usage/recency as the fallback prior. No LLM.++- **LoCoMo as an additive named suite.** A LoCoMo loader converts the LoCoMo+ dataset (conversation sessions + QA) into the existing `BenchFixture` shape+ (dialogue turns -> `session_turn` continuity and/or synthesized notes; QA ->+ `BenchQuestion` in existing categories with `expected_text`/`expected_paths`),+ fed to the unchanged `runMemoryBench`, emitting `o2b.bench.v1` named+ `locomo-*`. OSB's six categories stay canonical. The dataset ships as a small,+ committed, deterministic fixture (network-free); the optional judge stage+ remains opt-in for richer answer grading.++## File changes (by task, incremental)++- **t_2f99725a:** `src/core/search/embeddings/registry.ts` (widen `envKey` to+ ordered list + `isProfile`/`validateProfile`), `.../provider-resolve.ts`+ (probe-list resolution), `src/core/search/index.ts` (config plumbing);+ tests under `tests/core/search/embeddings.registry.test.ts`,+ `embeddings.provider-resolve.test.ts`.+- **t_33f30fdc:** new `src/core/search/embeddings/presets.ts`,+ `src/cli/search.ts` (registration flow + `--list-presets`); tests+ `tests/core/search/embeddings.presets.test.ts`,+ `tests/cli/search-provider-cli.test.ts`.+- **t_8d49f059:** new `src/core/search/embeddings/zeroentropy.ts`,+ `.../provider.ts` (factory branch), `src/core/search/types.ts` (union),+ `src/core/search/index.ts` (`parseProvider`/`BUILTIN_PROVIDERS`),+ `.../registry.ts` (reserved name); tests `tests/core/search/embeddings.zeroentropy.test.ts`.+- **t_4a672b84:** new `src/core/search/trigram-prefilter.ts`,+ `src/core/search/search.ts` (candidate-source wiring),+ `.../index.ts` (config), `.../query-cache.ts`/fingerprint; tests+ `tests/core/search/trigram-prefilter.test.ts`, superset assertion in+ `tests/core/search/search.*.test.ts`.+- **t_59ae326f:** new `src/core/search/graph-prepass.ts`,+ `src/core/search/search.ts` (opt-in mode), reuse+ `src/core/brain/link-graph/graph-index.ts`; tests+ `tests/core/search/graph-prepass.test.ts`.+- **t_9f95ebb6:** new `src/core/search/rerank/local.ts`,+ `.../rerank/provider.ts` (factory branch), `.../rerank/index.ts` + `types.ts`+ (config kind), new `src/core/search/rerank-eval-gate.ts`; tests+ `tests/core/search/rerank.local.test.ts`, `rerank-eval-gate.test.ts`.+- **t_65588d8b:** new `src/core/brain/observed-use.ts`,+ `src/core/brain/continuity/types.ts` (new kind), `src/mcp/brain/feedback-tools.ts`+ (new write tool), `src/core/search/ranker.ts` (`reuseRateByDoc` input) ++ `search.ts` (populate), `src/core/brain/recall-telemetry.ts` (blend surface);+ tests `tests/core/brain.observed-use.test.ts`, `tests/mcp/observed-use-tool.test.ts`.+- **t_703f7b18:** `src/core/brain/procedural-memory.ts` (counters + outcome fn),+ `src/mcp/brain/procedure-tools.ts` (`mark_outcome`); tests+ `tests/core/brain.procedural-outcome.test.ts`,+ `tests/mcp/procedural-learning-tools.test.ts`.+- **t_8dabe2b0:** new `src/core/bench/locomo.ts` (loader),+ `src/cli/brain/verbs/bench.ts` (`--suite locomo`), fixture under+ `tests/fixtures/bench/locomo-*.json`; tests `tests/core/bench/locomo.test.ts`.++## Risks++- **Query-cache staleness.** Any ranking-affecting stage (trigram, BFS,+ observed-reuse) must extend `configFingerprint`/`buildCacheKey` or it will serve+ stale cached rows within TTL. Mitigation: fold every new knob into the+ fingerprint; test cache invalidation.+- **Strict-superset correctness for the trigram prefilter.** A prefilter that+ drops a result the exhaustive scorer would keep is a silent recall regression.+ Mitigation: acceptance test asserts set-containment on fixtures incl.+ short/CJK/low-selectivity fallback; fall back to full scan on any doubt.+- **Byte-identical drift.** A new default-on behavior would break existing+ snapshot/determinism tests. Mitigation: all features default OFF; assert+ disabled == identical reference.+- **No-LLM kernel violation.** Observed-use and procedural outcome must never+ infer via an LLM in the kernel. Mitigation: deterministic-structural or+ host-supplied enum only, mirroring `brain_apply_evidence`.+- **LoCoMo dataset licensing/size.** Ship a small, self-contained, deterministic+ fixture rather than the full external dataset; document how to point the loader+ at a full local copy. No network by default.+- **Heavy-dependency temptation** (offline reranker). Mitigation: pure-TS+ deterministic reranker; the eval gate justifies enabling it per store. Hard+ invariant above.diff --git a/docs/brainstorm/retrieval-ranking-quality/plan.md b/docs/brainstorm/retrieval-ranking-quality/plan.mdnew file mode 100644index 00000000..0442d81c--- /dev/null+++ b/docs/brainstorm/retrieval-ranking-quality/plan.md@@ -0,0 +1,202 @@+# Retrieval & Ranking Quality - Implementation Plan++**Status:** draft+**Author:** feature-release-playbook+**Audience:** implementation++Per-task implementation plan. Implement in the order below (the release sequence+hint): embeddings robustness -> candidate retrieval -> reranking -> outcome+signals -> benchmark last. Every task is TDD (failing test first), gated+default-OFF where it changes behavior, and adds zero npm dependencies.++Legend: **Files** = code + test touch-points; **Acceptance** = the concrete test+that proves the task; **Depends on** = ordering constraint within this branch.++---++## 1. t_2f99725a - Multi-key fallback for the embeddings API key++- **Files:**+ - `src/core/search/embeddings/registry.ts` - `ProviderProfile.envKey` accepts+ `string | readonly string[]`; `isProfile`/`validateProfile` accept both;+ `expandRegisteredProvider` resolves the first present, non-empty env var and+ records which key won.+ - `src/core/search/embeddings/provider-resolve.ts` - `resolveOpenAiCompatEndpoint`+ gains an ordered-probe path (first non-empty wins, else throw the existing+ `EMBEDDING_KEY_MISSING`).+ - `src/core/search/index.ts` - config plumbing preserves single-key precedence.+ - Tests: `tests/core/search/embeddings.registry.test.ts`,+ `tests/core/search/embeddings.provider-resolve.test.ts`.+- **Acceptance:** given a profile whose `envKey` is `["A","B"]` with `A` unset and+ `B` set, `expandRegisteredProvider` resolves `B`'s value; with a single-string+ `envKey` behavior is byte-identical to today; with none set it throws+ `EMBEDDING_KEY_MISSING`.+- **Depends on:** none (first task).++## 2. t_33f30fdc - Curated embedding-model presets + recommended default++- **Files:**+ - new `src/core/search/embeddings/presets.ts` - `EMBEDDING_MODEL_PRESETS`+ (curated multilingual-first entries: id, label, dimension, note) ++ `RECOMMENDED_EMBEDDING_MODEL`.+ - `src/cli/search.ts` - `provider add` resolves `--model` against presets,+ defaults to the recommended model when omitted, and supports+ `--list-presets`; custom `--model` still accepted verbatim.+ - Tests: `tests/core/search/embeddings.presets.test.ts`,+ `tests/cli/search-provider-cli.test.ts`.+- **Acceptance:** `o2b search provider add p --base-url U --env-key K` with no+ `--model` persists the recommended default; `--list-presets` prints the catalog;+ a custom `--model foo/bar` persists verbatim; presets carry stable dimensions.+- **Depends on:** none (independent; touches registration UX only).++## 3. t_8d49f059 - Native ZeroEntropy embedding provider++- **Files:**+ - new `src/core/search/embeddings/zeroentropy.ts` - `ZeroEntropyProvider`+ implements `EmbeddingProvider` (`embed`, `ping`, `consumeRetryCount`) over the+ ZeroEntropy embed API via `fetch`; retry/backoff mirrors openai-compat.+ - `src/core/search/embeddings/provider.ts` - `makeProvider` branch for+ `"zeroentropy"`.+ - `src/core/search/types.ts` - add `"zeroentropy"` to the resolved provider+ union.+ - `src/core/search/index.ts` - `parseProvider` + `BUILTIN_PROVIDERS`.+ - `src/core/search/embeddings/registry.ts` - reserve name.+ - Tests: `tests/core/search/embeddings.zeroentropy.test.ts` (fake-http helper).+- **Acceptance:** with `provider: "zeroentropy"` configured, `embed` returns+ correctly-shaped vectors from a fake ZeroEntropy endpoint, `ping` reports the+ dimension, auth/absent-key errors map to the existing `SearchError` codes; the+ provider union stays closed (invalid provider still throws `INVALID_INPUT`).+- **Depends on:** none (parallel to 1-2, shares embeddings surface; sequence after+ 1-2 to avoid registry merge churn).++## 4. t_4a672b84 - Trigram candidate prefilter for large-vault search++- **Files:**+ - new `src/core/search/trigram-prefilter.ts` - deterministic trigram candidate+ producer (extract query trigrams; select candidate chunkIds by trigram+ overlap); short/CJK/low-selectivity detection triggers full-scan fallback.+ - `src/core/search/search.ts` - wire as an additional candidate source into the+ union (never removes candidates); default-off.+ - `src/core/search/index.ts` - `search_trigram_prefilter*` config keys+ (enabled, min corpus size, selectivity threshold), default off.+ - `src/core/search/search.ts` `configFingerprint` / `query-cache.ts` - fold the+ knobs into the cache key.+ - Tests: `tests/core/search/trigram-prefilter.test.ts`, superset assertion in a+ search-level test.+- **Acceptance:** on a fixture vault, the prefiltered candidate set is a strict+ superset of the exhaustive scorer's candidate set for normal queries; short+ (< 3 chars), CJK, and low-selectivity queries fall back to full scan (identical+ results); enabling/disabling changes the cache key.+- **Depends on:** none code-wise; sequence after embeddings so retrieval work is+ contiguous.++## 5. t_59ae326f - Graph-index BFS query pre-pass + should-read + index-only++- **Files:**+ - new `src/core/search/graph-prepass.ts` - rank candidate notes by+ title/tag/summary match + wikilink degree + tier; multi-hop BFS over the link+ graph (reuse `src/core/brain/link-graph/graph-index.ts` snapshot; BFS modeled+ on `src/core/search/traversal.ts:expandByTraversal`); emit `should_read`+ shortlist; `index_only` mode answers from index metadata with zero bodies.+ - `src/core/search/search.ts` - opt-in pre-pass mode; default path unchanged.+ - Tests: `tests/core/search/graph-prepass.test.ts`.+- **Acceptance:** on a linked fixture vault, the pre-pass returns a `should_read`+ shortlist ordered by the combined signal; a 2-hop question surfaces a note+ reachable only via multi-hop BFS that the 1-hop boost misses; `index_only`+ returns an answer with zero note bodies hydrated (assert no body reads).+- **Depends on:** 4 (retrieval-stage cohesion; independent modules).++## 6. t_9f95ebb6 - Bundled offline reranker + per-store reranker eval gate++- **Files:**+ - new `src/core/search/rerank/local.ts` - `LocalRerankProvider` implements+ `RerankProvider` with a deterministic pure-TS cross-encoder-shaped scorer+ (term overlap, phrase proximity, coverage, position signals). No network, no+ dependency.+ - `src/core/search/rerank/provider.ts` - `makeRerankProvider` branch on a new+ `kind`/`provider` field.+ - `src/core/search/rerank/index.ts` + `src/core/search/types.ts` - add the+ provider-kind field to `ResolvedRerankConfig`; select local vs openai-compat.+ - new `src/core/search/rerank-eval-gate.ts` - run `runRecallBenchmark` with+ rerank off vs on and compare metric families against thresholds -> enable/keep+ decision per store.+ - Tests: `tests/core/search/rerank.local.test.ts`,+ `tests/core/search/rerank-eval-gate.test.ts`.+- **Acceptance:** `LocalRerankProvider.rerank(query, docs)` returns order-aligned+ scores, higher for a doc that better matches the query, fully offline and+ deterministic; wired via config `kind: "local"` it reorders results in an+ end-to-end search with no network; the eval gate returns `improves: true` when+ rerank raises hit@k/MRR on a fixture and `improves: false` when it does not.+- **Depends on:** 4-5 (reranking sits after candidate retrieval).++## 7. t_65588d8b - Session-end observed-use verdict feeding recall ranking++- **Files:**+ - new `src/core/brain/observed-use.ts` - deterministic verdict computation+ (match injected-memory anchors/content against later transcript spans; reuse+ `deriveNoteStance` for CONTRADICTED) AND acceptance of a host-supplied verdict;+ fold observed records into a per-artifact reuse rate.+ - `src/core/brain/continuity/types.ts` - add `recall_observed_use` kind.+ - `src/mcp/brain/feedback-tools.ts` - new write tool mirroring+ `brain_apply_evidence` (enum-validated verdict, role gate, typed errors).+ - `src/core/search/ranker.ts` - new `reuseRateByDoc?` input (capped boost /+ preferred signal with importance as fallback prior).+ - `src/core/search/search.ts` - populate `reuseRateByDoc` from folded records.+ - `src/core/brain/recall-telemetry.ts` - surface the blend so it is inspectable.+ - Tests: `tests/core/brain.observed-use.test.ts`,+ `tests/mcp/observed-use-tool.test.ts`, ranker input test.+- **Acceptance:** given injected memories and a transcript, USED/IGNORED are+ classified deterministically and a CONTRADICTED case routes via+ `deriveNoteStance`; the host tool records a verdict per injected memory+ (enum-validated, idempotent); a memory with a high observed-reuse rate ranks+ above an equally-relevant memory with none; kernel calls no LLM. Outcome-free+ vaults stay byte-identical.+- **Depends on:** none code-wise; sequence after retrieval so the ranker input is+ added once the retrieval stages are stable.++## 8. t_703f7b18 - Outcome-validated procedural recall ranked by success rate++- **Files:**+ - `src/core/brain/procedural-memory.ts` - add `successCount`/`failureCount`+ (derived `successRate`) to `ProceduralMemoryEntry` and the `usage.jsonl`+ sidecar (additive; old lines parse); `recordProceduralOutcome(vault, id,+ success|failure, {now})` mirrors `markProceduralMemoryUsed`; recall ranks by+ `successRate` primary, usage/recency fallback prior.+ - `src/mcp/brain/procedure-tools.ts` - `mark_outcome` operation mirroring+ `mark_used`.+ - Tests: `tests/core/brain.procedural-outcome.test.ts`,+ `tests/mcp/procedural-learning-tools.test.ts`.+- **Acceptance:** recording success/failure updates counters (order-insensitive+ fold); a procedure with a high success rate outranks a more-used but+ lower-success-rate procedure; entries without outcomes rank by the existing+ usage prior (byte-identical for outcome-free vaults); ranking is deterministic,+ no LLM.+- **Depends on:** 7 (both are outcome-signal features; independent modules).++## 9. t_8dabe2b0 - LoCoMo benchmark suite over the existing bench harness++- **Files:**+ - new `src/core/bench/locomo.ts` - loader converting the LoCoMo dataset+ (sessions + QA) into `BenchFixture`(s) named `locomo-*` (dialogue turns ->+ `session_turn` continuity and/or synthesized notes; QA -> `BenchQuestion`+ with existing categories + `expected_text`/`expected_paths`).+ - `src/cli/brain/verbs/bench.ts` - `--suite locomo` resolution branch.+ - new fixture `tests/fixtures/bench/locomo-sample.json` - small, deterministic,+ self-contained LoCoMo-shaped sample.+ - Tests: `tests/core/bench/locomo.test.ts`.+- **Acceptance:** the loader emits a valid `BenchFixture` (passes+ `parseBenchFixture`); `runMemoryBench` over the LoCoMo fixture produces an+ `o2b.bench.v1` report named `locomo-*` deterministically and network-free;+ OSB's six categories remain canonical (unchanged core-recall run); the optional+ judge stage stays opt-in and defaults to `skipped`.+- **Depends on:** 1-8 (benchmark last so it can score the improved stack).++---++## Cross-cutting acceptance (Phase 7 QA)++- `bun run validate` green, zero warnings.+- `bun run check:paths:strict` clean.+- Every "disabled" feature proven byte-identical to pre-branch behavior.+- Query-cache invalidation proven for every ranking-affecting knob.+- No new entry in `package.json` `dependencies`/`optionalDependencies`.diff --git a/docs/brainstorm/retrieval-ranking-quality/variants.md b/docs/brainstorm/retrieval-ranking-quality/variants.mdnew file mode 100644index 00000000..0527754a--- /dev/null+++ b/docs/brainstorm/retrieval-ranking-quality/variants.md@@ -0,0 +1,114 @@+# Retrieval & Ranking Quality - Architectural Variants++**Status:** draft+**Author:** feature-release-playbook+**Audience:** implementation++This release bundles nine leaf tasks under one theme: make Open Second Brain's+retrieval demonstrably better AND measurable. The tasks span four layers -+embedding-layer robustness, candidate retrieval, reranking, and outcome-learned+ranking signals - plus a named benchmark to score the whole. The question these+variants answer is not "which features" (scope is locked) but **how to structure,+sequence, and integrate them** so nine features land as one coherent release+without destabilizing the core query path.++Three cross-cutting architectures were considered.++---++## Variant 1 - Additive, opt-in, byte-identical-when-off++**Approach.** Each feature ships as a new, independently gated stage or module+that defaults OFF and preserves byte-identical behavior when disabled, matching+the repo's existing "disabled -> identical reference" discipline (rerank+`rerank/index.ts:104-105`, ranker neutral multipliers, default-off telemetry+gates). Concretely:++- **Embedding robustness** widens `ProviderProfile.envKey` into an ordered probe+ list (backward-compatible single string still accepted), adds a static preset+ catalog consulted only at CLI registration time, and adds a native ZeroEntropy+ provider as a new closed-union branch alongside `openai-compat`.+- **Candidate retrieval** adds a trigram prefilter as a *strict superset* stage+ ahead of full scoring (falls back to full scan for short/CJK/low-selectivity+ queries) and a graph-index BFS pre-pass producing a `should_read` shortlist+ with an `index_only` short-circuit - both new, both gated.+- **Reranking** adds a bundled offline cross-encoder behind the existing+ `RerankProvider` interface (new provider kind, no network) plus a per-store+ eval gate built on the existing recall-benchmark runner.+- **Outcome signals** add a session-end observed-use verdict (new continuity+ kind + MCP write tool mirroring `brain_apply_evidence`) folded into an+ additive capped ranker boost, and additive success/failure counters on+ procedural entries with success-rate ranking.+- **Benchmark** adds a LoCoMo loader emitting the existing `BenchFixture` shape+ into the unchanged staged harness, named as a parallel suite.++**Trade-offs.** Maximum safety and testability: every feature is isolated,+individually revertable, and provable against the pre-existing exhaustive path.+The cost is less cross-feature cohesion - the trigram prefilter, BFS pre-pass,+and reranker remain three separate stages rather than one planner, so a future+"unified retrieval planner" refactor is deferred (but not blocked).++**Complexity:** medium. **Risk:** low.++---++## Variant 2 - Unified retrieval-planner rework++**Approach.** Introduce a single new "retrieval planning" layer that subsumes the+trigram prefilter, the graph-index BFS pre-pass, and reranker selection into one+coordinated candidate-planning stage inserted into `search()`. The planner owns+candidate narrowing (trigram), candidate expansion (BFS), and candidate+re-ordering (rerank) as phases of one pipeline object, with the embedding and+outcome-signal features feeding it as inputs.++**Trade-offs.** Highest cohesion and the cleanest long-term architecture: one+place decides *which* candidates survive and *how* they are ordered. But it+rewrites the core `search()` candidate-union region (`search.ts:414-492,+834-988`), risking the repo-wide byte-identical-when-off invariant and the query+cache fingerprint contract (`search.ts:110-137`). It also couples nine+independent tasks into one large change, making TDD-per-task and per-feature+revert far harder, and inflates review surface. The benchmark and embedding+tasks gain nothing from the coupling.++**Complexity:** large. **Risk:** high.++---++## Variant 3 - Signal-fold-first (outcome-centric)++**Approach.** Build a shared "signal fold" substrate first - a generic+one-file-per-signal append + pure-fold aggregate (the pattern already used by+`activation/store.ts`, `feedback.ts`, `reinforce.ts`) - then express the+observed-use verdict, the procedural success rate, and the per-store reranker+eval gate as three consumers of that substrate. Retrieval features layer on top+consuming the learned signals.++**Trade-offs.** Elegant for the three learning features and would reduce+duplication among them. But four of the nine tasks (multi-key fallback, preset+catalog, ZeroEntropy, LoCoMo) have nothing to do with signal folds, so the+architecture only covers part of the release and forces an artificial "phase 0+substrate" that delays the independently-shippable embedding work. The existing+fold helpers are already reusable directly, so a new generic substrate is+speculative abstraction (YAGNI) ahead of a second real consumer.++**Complexity:** medium-large. **Risk:** medium.++---++## Recommended: Variant 1 - Additive, opt-in, byte-identical-when-off++Variant 1 is the only architecture that fits all nine tasks equally, honors the+repo's load-bearing invariants (byte-identical-when-disabled, default-off gates,+pure disposable folds, closed provider union, no-LLM kernel), and keeps each task+independently test-driven and revertable. It matches every upstream task's own+additive framing and the plan's sequence hint (embeddings -> retrieval ->+reranking -> outcome signals -> benchmark last), and it lets the LoCoMo suite+measure the improved-but-optional stack without any feature being on by default.++Variant 2's unified planner is the right *eventual* shape but is out of scope+here: coupling nine tasks into a core-path rewrite trades this release's low risk+for cohesion the tasks do not yet need. Variant 3's substrate is premature+abstraction over three consumers when the existing fold helpers already serve.++Variant 1 does not preclude a later planner refactor - each stage lands behind a+clean seam that a future unification can absorb.diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex 32373d9f..9fe3b92d 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.27.1",+ "version": "1.28.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex b369cc85..14794c5e 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.27.1",+ "version": "1.28.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 8bb3024d..b7bfe4ba 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.27.1"+version: "1.28.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 937cb15d..5faa252c 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.27.1",+ "version": "1.28.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 8bb3024d..b7bfe4ba 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.27.1"+version: "1.28.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex 3640923c..f79cb525 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.27.1"+version = "1.28.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/brain/verbs/bench.ts b/src/cli/brain/verbs/bench.tsindex bceb35c3..2d678e53 100644--- a/src/cli/brain/verbs/bench.ts+++ b/src/cli/brain/verbs/bench.ts@@ -14,12 +14,13 @@ import { existsSync } from "node:fs";import { join, resolve } from "node:path";import { loadBenchFixture } from "../../../core/bench/fixture.ts";+import { loadLocomoFixture } from "../../../core/bench/locomo.ts";import { runMemoryBench } from "../../../core/bench/phases.ts";import { defaultConfigPath, resolveBenchJudgeCmd } from "../../../core/config.ts";import { fail, parse } from "../helpers.ts";const USAGE =- "usage: o2b brain bench memory --fixture <name|path> [--resume <run-id>] [--runs-dir <dir>] [--json]";+ "usage: o2b brain bench memory --fixture <name|path> [--suite locomo] [--resume <run-id>] [--runs-dir <dir>] [--json]";const DEFAULT_RUNS_DIR = join(".open-second-brain", "bench-runs");const REPO_FIXTURE_DIR = join("tests", "fixtures", "bench");@@ -28,12 +29,17 @@ export async function cmdBrainBench(argv: string[]): Promise<number> {if (argv[0] !== "memory") return fail(USAGE);const { flags } = parse(argv.slice(1), {fixture: { type: "string" },+ suite: { type: "string" },resume: { type: "string" },"runs-dir": { type: "string" },json: { type: "boolean" },});const fixtureFlag = typeof flags["fixture"] === "string" ? flags["fixture"].trim() : "";if (fixtureFlag === "") return fail(USAGE);+ const suite = typeof flags["suite"] === "string" ? flags["suite"].trim() : "";+ if (suite !== "" && suite !== "locomo") {+ return fail(`brain bench memory: unknown --suite '${suite}' (supported: locomo)`);+ }const fixturePath = resolveFixturePath(fixtureFlag);if (fixturePath === null) {return fail(@@ -41,7 +47,10 @@ export async function cmdBrainBench(argv: string[]): Promise<number> {);}- const fixture = loadBenchFixture(fixturePath);+ // The LoCoMo suite loads a LoCoMo-shaped dataset and converts it to a+ // BenchFixture; the default path loads a native fixture directly.+ const fixture =+ suite === "locomo" ? loadLocomoFixture(fixturePath) : loadBenchFixture(fixturePath);const runsDir = resolve(typeof flags["runs-dir"] === "string" && flags["runs-dir"].trim() !== ""? flags["runs-dir"].trim()diff --git a/src/cli/search.ts b/src/cli/search.tsindex 0cf219dd..2b96b43f 100644--- a/src/cli/search.ts+++ b/src/cli/search.ts@@ -49,6 +49,7 @@ import {addRerankProviderProfile,removeRerankProviderProfile,getRerankProviderProfile,+ planRead,serializeEvidencePack,serializeSearchCard,serializeIndexStatus,@@ -64,6 +65,11 @@ import type {SearchSessionFocus,SearchOutcome,} from "../core/search/index.ts";+import {+ EMBEDDING_MODEL_PRESETS,+ RECOMMENDED_EMBEDDING_MODEL,+ type EmbeddingModelPreset,+} from "../core/search/embeddings/presets.ts";import { searchAcrossVaults } from "../core/search/cross-vault.ts";import { IndexWatchPlanner } from "../core/search/index-watch.ts";import { IndexWatchRunner } from "../core/search/watch-runner.ts";@@ -85,6 +91,7 @@ const KNOWN_VERBS = new Set(["weights","provider","rerank-provider",+ "plan","watch",]);@@ -124,6 +131,8 @@ export async function handleSearchSubcommand(argv: ReadonlyArray<string>): Promireturn await cmdSearchProvider(rest);case "rerank-provider":return await cmdSearchRerankProvider(rest);+ case "plan":+ return await cmdSearchPlan(rest);default:process.stderr.write(`error: unknown search verb: ${verb}\n`);return 2;@@ -351,7 +360,25 @@ interface CliRegistryProfile {readonly name: string;readonly baseUrl: string;readonly defaultModel: string;- readonly envKey: string;+ readonly envKey: string | ReadonlyArray<string>;+}++/** Render an env-key that may be a single name or an ordered probe list. */+function formatEnvKey(envKey: string | ReadonlyArray<string>): string {+ return typeof envKey === "string" ? envKey : envKey.join(",");+}++/**+ * Parse a `--env-key` flag into a single name or an ordered probe list.+ * Accepts a comma-separated list so multi-key failover is CLI-registrable;+ * a single name stays a plain string (byte-identical single-key profile).+ */+function parseEnvKeyFlag(raw: string): string | string[] {+ const parts = raw+ .split(",")+ .map((k) => k.trim())+ .filter((k) => k !== "");+ return parts.length <= 1 ? (parts[0] ?? "") : parts;}/**@@ -367,6 +394,18 @@ interface ProviderRegistryOps<T extends CliRegistryProfile> {readonly get: (vault: string, name: string) => T | null;readonly add: (vault: string, profile: CliRegistryProfile) => ReadonlyArray<T>;readonly remove: (vault: string, name: string) => { removed: boolean };+ /**+ * When true, `--env-key` accepts a comma-separated probe list for+ * multi-key failover (embedding provider only). Rerank stays single-key.+ */+ readonly multiKey?: boolean;+ /**+ * Curated model catalog surfaced via the `presets` action and used as the+ * `--model` default when omitted (embedding provider only). Advisory.+ */+ readonly presets?: ReadonlyArray<EmbeddingModelPreset>;+ /** Recommended default model string when `--model` is omitted. */+ readonly recommendedModel?: string;}/**@@ -380,9 +419,12 @@ async function runProviderRegistryCommand<T extends CliRegistryProfile>(): Promise<number> {const { verb, kind } = ops;const action = argv[0];- if (!action || !["add", "list", "show", "remove"].includes(action)) {+ const allowed = ops.presets+ ? ["add", "list", "show", "remove", "presets"]+ : ["add", "list", "show", "remove"];+ if (!action || !allowed.includes(action)) {throw new CliError(- `usage: o2b search ${verb} <add NAME --base-url U --model M --env-key K | list | show NAME | remove NAME> [--json]`,+ `usage: o2b search ${verb} <add NAME --base-url U [--model M] --env-key K | list | show NAME | remove NAME${ops.presets ? " | presets" : ""}> [--json]`,);}const { flags, positional } = parseFlags(argv.slice(1), {@@ -394,8 +436,23 @@ async function runProviderRegistryCommand<T extends CliRegistryProfile>("env-key": { type: "string" },json: { type: "boolean" },});- const cfg = resolveConfig(flags);const json = flags["json"] === true;++ // `presets` is a static catalog listing; it needs no vault/config.+ if (action === "presets" && ops.presets) {+ if (json) {+ process.stdout.write(JSON.stringify(ops.presets) + "\n");+ return 0;+ }+ process.stdout.write(`curated embedding models (recommended: ${ops.recommendedModel}):\n`);+ for (const p of ops.presets) {+ const tag = p.multilingual ? "multilingual" : "monolingual";+ process.stdout.write(` ${p.model}\n dim=${p.dimension} ${tag} - ${p.note}\n`);+ }+ return 0;+ }++ const cfg = resolveConfig(flags);const name = typeof positional[0] === "string" ? positional[0] : undefined;if (action === "list") {@@ -409,7 +466,9 @@ async function runProviderRegistryCommand<T extends CliRegistryProfile>(return 0;}for (const p of registry) {- process.stdout.write(`${p.name} ${p.baseUrl} model=${p.defaultModel} env=${p.envKey}\n`);+ process.stdout.write(+ `${p.name} ${p.baseUrl} model=${p.defaultModel} env=${formatEnvKey(p.envKey)}\n`,+ );}return 0;}@@ -424,7 +483,7 @@ async function runProviderRegistryCommand<T extends CliRegistryProfile>(process.stdout.write(json? JSON.stringify(profile) + "\n"- : `${profile.name}\n base-url: ${profile.baseUrl}\n model: ${profile.defaultModel}\n env-key: ${profile.envKey}\n`,+ : `${profile.name}\n base-url: ${profile.baseUrl}\n model: ${profile.defaultModel}\n env-key: ${formatEnvKey(profile.envKey)}\n`,);return 0;}@@ -444,20 +503,27 @@ async function runProviderRegistryCommand<T extends CliRegistryProfile>(if (!name)throw new CliError(`usage: o2b search ${verb} add NAME --base-url U --model M --env-key K`);const baseUrl = typeof flags["base-url"] === "string" ? (flags["base-url"] as string) : undefined;- const model = typeof flags["model"] === "string" ? (flags["model"] as string) : undefined;+ const flagModel = typeof flags["model"] === "string" ? (flags["model"] as string) : undefined;const envKey = typeof flags["env-key"] === "string" ? (flags["env-key"] as string) : undefined;+ // `--model` may be omitted when the registry has a recommended default+ // (embedding provider); custom models remain first-class and verbatim.+ const model = flagModel ?? ops.recommendedModel;+ const modelHint = ops.recommendedModel ? "[--model M]" : "--model M";if (!baseUrl || !model || !envKey) {throw new CliError(- `${verb} add requires --base-url, --model, and --env-key (the env var NAME holding the API key)`,+ `${verb} add requires --base-url, ${modelHint}, and --env-key (the env var NAME holding the API key)`,);}- const registry = ops.add(cfg.vault, { name, baseUrl, defaultModel: model, envKey });+ const envKeyValue = ops.multiKey ? parseEnvKeyFlag(envKey) : envKey;+ const registry = ops.add(cfg.vault, { name, baseUrl, defaultModel: model, envKey: envKeyValue });const added = registry.find((p) => p.name === name)!;if (json) {process.stdout.write(JSON.stringify(added) + "\n");} else {+ const envHint = Array.isArray(envKeyValue) ? envKeyValue.join(" or ") : envKeyValue;+ const modelNote = flagModel ? "" : ` (defaulted --model to recommended '${model}')`;process.stdout.write(- `added ${kind} '${name}' (set ${envKey} in the environment to supply its key)\n`,+ `added ${kind} '${name}'${modelNote} (set ${envHint} in the environment to supply its key)\n`,);}return 0;@@ -471,6 +537,9 @@ async function cmdSearchProvider(argv: ReadonlyArray<string>): Promise<number> {get: getProviderProfile,add: addProviderProfile,remove: removeProviderProfile,+ multiKey: true,+ presets: EMBEDDING_MODEL_PRESETS,+ recommendedModel: RECOMMENDED_EMBEDDING_MODEL,});}@@ -482,11 +551,60 @@ async function cmdSearchRerankProvider(argv: ReadonlyArray<string>): Promise<numkind: "rerank provider",load: loadRerankRegistry,get: getRerankProviderProfile,- add: addRerankProviderProfile,+ // Rerank stays single-key; coerce any probe-list shape back to a string.+ add: (vault, profile) =>+ addRerankProviderProfile(vault, { ...profile, envKey: formatEnvKey(profile.envKey) }),remove: removeRerankProviderProfile,});}+// βββ plan (graph-index query pre-pass) βββββββββββββββββββββββββββββββββββββββ++async function cmdSearchPlan(argv: ReadonlyArray<string>): Promise<number> {+ const { flags, positional } = parseFlags(argv, {+ vault: { type: "string" },+ config: { type: "string" },+ db: { type: "string" },+ hops: { type: "string", default: "2" },+ limit: { type: "string", default: "10" },+ "index-only": { type: "boolean" },+ json: { type: "boolean" },+ });+ const query = positional.join(" ").trim();+ if (query === "")+ throw new CliError('usage: o2b search plan "<query>" [--index-only] [--hops N]');+ const cfg = resolveConfig(flags);+ const maxHops = Number(flags["hops"] ?? "2");+ const shortlistLimit = Number(flags["limit"] ?? "10");+ if (!Number.isInteger(maxHops) || maxHops < 0) {+ throw new CliError(`--hops must be a non-negative integer, got '${flags["hops"]}'`);+ }+ if (!Number.isInteger(shortlistLimit) || shortlistLimit < 1) {+ throw new CliError(`--limit must be a positive integer, got '${flags["limit"]}'`);+ }+ const plan = await planRead(cfg, query, {+ indexOnly: flags["index-only"] === true,+ maxHops,+ shortlistLimit,+ });+ if (flags["json"] === true) {+ process.stdout.write(JSON.stringify(plan) + "\n");+ return 0;+ }+ process.stdout.write(`${plan.mode} (notes read: ${plan.notesRead})\n`);+ if (plan.shortlist.length === 0) {+ process.stdout.write(" no candidates\n");+ return 0;+ }+ for (const e of plan.shortlist) {+ const title = e.title ?? "(untitled)";+ process.stdout.write(+ ` ${e.path} "${title}" hops=${e.hops} degree=${e.degree} [${e.reasons.join(",")}]\n`,+ );+ }+ return 0;+}+// βββ query ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββasync function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<number> {diff --git a/src/core/bench/locomo.ts b/src/core/bench/locomo.tsnew file mode 100644index 00000000..b59eb49f--- /dev/null+++ b/src/core/bench/locomo.ts@@ -0,0 +1,199 @@+/**+ * LoCoMo benchmark suite adapter (Retrieval & Ranking Quality, t_8dabe2b0).+ *+ * LoCoMo (Long Conversational Memory) is a widely-cited long-conversation+ * memory benchmark. OSB already owns a staged bench harness+ * (ingest -> index -> retrieve -> evaluate -> report) with its own+ * MemoryBench-inspired categories; this module adds LoCoMo as an ADDITIVE,+ * parallel NAMED suite by converting a LoCoMo-shaped dataset into the+ * existing {@link BenchFixture} the unchanged `runMemoryBench` consumes.+ * OSB's six categories stay canonical; LoCoMo questions are expressed in+ * terms of them (single_hop / temporal / multi_evidence).+ *+ * Deterministic and network-free: the loader is pure over its input, the+ * run scores by deterministic substring / path containment, and the+ * optional LLM judge stays opt-in (`bench_judge_cmd`) - never the default.+ * No harness-mechanic change and no new dependency.+ */++import { readFileSync } from "node:fs";++import { parseBenchFixture } from "./fixture.ts";+import type { BenchCategory, BenchFixture, BenchQuestion } from "./types.ts";++export interface LocomoTurn {+ readonly speaker: string;+ readonly text: string;+ /** ISO-ish timestamp; normalised to canonical UTC for continuity. */+ readonly timestamp: string;+}++export interface LocomoSession {+ readonly session_id: string;+ readonly turns: ReadonlyArray<LocomoTurn>;+}++export interface LocomoQa {+ readonly id: string;+ readonly question: string;+ readonly answer?: string;+ /** LoCoMo category hint (e.g. "temporal", "multi-hop"); mapped below. */+ readonly category?: string;+ /** Session ids that hold the evidence for the answer. Non-empty. */+ readonly evidence_sessions: ReadonlyArray<string>;+}++export interface LocomoDataset {+ readonly name?: string;+ readonly sessions: ReadonlyArray<LocomoSession>;+ readonly qa: ReadonlyArray<LocomoQa>;+}++class LocomoParseError extends Error {}++function asString(value: unknown, field: string): string {+ if (typeof value !== "string" || value.trim() === "") {+ throw new LocomoParseError(`LoCoMo dataset: ${field} must be a non-empty string`);+ }+ return value;+}++/**+ * A non-empty, parseable timestamp string. Validated here so a bad value+ * fails with a field-identifying `LocomoParseError` at parse time rather+ * than a bare `RangeError` when `locomoToBenchFixture` normalises it.+ */+function asValidTimestamp(value: unknown, field: string): string {+ const s = asString(value, field);+ if (Number.isNaN(new Date(s).getTime())) {+ throw new LocomoParseError(`LoCoMo dataset: ${field} must be a valid date, got '${s}'`);+ }+ return s;+}++/** Parse and validate a raw LoCoMo dataset. Fail-fast with a named error. */+export function parseLocomoDataset(raw: unknown): LocomoDataset {+ if (raw === null || typeof raw !== "object") {+ throw new LocomoParseError("LoCoMo dataset must be an object");+ }+ const r = raw as Record<string, unknown>;+ if (!Array.isArray(r["sessions"]) || r["sessions"].length === 0) {+ throw new LocomoParseError("LoCoMo dataset needs a non-empty `sessions` array");+ }+ if (!Array.isArray(r["qa"]) || r["qa"].length === 0) {+ throw new LocomoParseError("LoCoMo dataset needs a non-empty `qa` array");+ }+ const sessions: LocomoSession[] = r["sessions"].map((s, i) => {+ if (s === null || typeof s !== "object") {+ throw new LocomoParseError(`sessions[${i}] must be an object`);+ }+ const sd = s as Record<string, unknown>;+ const sessionId = asString(sd["session_id"], `sessions[${i}].session_id`);+ if (!Array.isArray(sd["turns"]) || sd["turns"].length === 0) {+ throw new LocomoParseError(`sessions[${i}].turns must be a non-empty array`);+ }+ const turns: LocomoTurn[] = sd["turns"].map((t, j) => {+ if (t === null || typeof t !== "object") {+ throw new LocomoParseError(`sessions[${i}].turns[${j}] must be an object`);+ }+ const td = t as Record<string, unknown>;+ return {+ speaker: asString(td["speaker"], `sessions[${i}].turns[${j}].speaker`),+ text: asString(td["text"], `sessions[${i}].turns[${j}].text`),+ timestamp: asValidTimestamp(td["timestamp"], `sessions[${i}].turns[${j}].timestamp`),+ };+ });+ return { session_id: sessionId, turns };+ });+ const knownSessions = new Set(sessions.map((s) => s.session_id));+ const qa: LocomoQa[] = r["qa"].map((q, i) => {+ if (q === null || typeof q !== "object") {+ throw new LocomoParseError(`qa[${i}] must be an object`);+ }+ const qd = q as Record<string, unknown>;+ const evidence = qd["evidence_sessions"];+ if (!Array.isArray(evidence) || evidence.length === 0) {+ throw new LocomoParseError(`qa[${i}].evidence_sessions must be a non-empty array`);+ }+ const evidenceSessions = evidence.map((e, k) => {+ const sid = asString(e, `qa[${i}].evidence_sessions[${k}]`);+ if (!knownSessions.has(sid)) {+ throw new LocomoParseError(`qa[${i}] references unknown session '${sid}'`);+ }+ return sid;+ });+ return {+ id: asString(qd["id"], `qa[${i}].id`),+ question: asString(qd["question"], `qa[${i}].question`),+ ...(typeof qd["answer"] === "string" ? { answer: qd["answer"] } : {}),+ ...(typeof qd["category"] === "string" ? { category: qd["category"] } : {}),+ evidence_sessions: evidenceSessions,+ };+ });+ return {+ ...(typeof r["name"] === "string" ? { name: r["name"] } : {}),+ sessions,+ qa,+ };+}++/** Vault-relative note path for a LoCoMo session. */+function sessionNotePath(sessionId: string): string {+ const safe = sessionId.replace(/[^A-Za-z0-9_-]/g, "-");+ return `sessions/${safe}.md`;+}++/** Map a LoCoMo QA to one of OSB's canonical retrieval categories. */+function mapCategory(qa: LocomoQa): BenchCategory {+ if (qa.evidence_sessions.length > 1) return "multi_evidence";+ const hint = (qa.category ?? "").toLowerCase();+ if (hint.includes("temporal") || hint.includes("time")) return "temporal";+ return "single_hop";+}++/**+ * Convert a parsed LoCoMo dataset into an OSB {@link BenchFixture}, named+ * `locomo-<name>`. Sessions become evidence notes (and session_turn+ * continuity records); QA pairs become retrieval questions whose+ * `expected_paths` point at their evidence session notes and whose+ * `expected_text` is the declared answer (deterministic containment).+ */+export function locomoToBenchFixture(dataset: LocomoDataset): BenchFixture {+ const notes = dataset.sessions.map((s) => ({+ path: sessionNotePath(s.session_id),+ body:+ `# Session ${s.session_id}\n\n` + s.turns.map((t) => `${t.speaker}: ${t.text}`).join("\n"),+ }));+ const continuity = dataset.sessions.flatMap((s) =>+ s.turns.map((t) => ({+ kind: "session_turn" as const,+ created_at: new Date(t.timestamp).toISOString(),+ payload: { session_id: s.session_id, speaker: t.speaker, text: t.text },+ })),+ );+ const questions: BenchQuestion[] = dataset.qa.map((qa) => ({+ id: qa.id,+ category: mapCategory(qa),+ query: qa.question,+ top_k: 5,+ expected_paths: qa.evidence_sessions.map(sessionNotePath),+ ...(qa.answer ? { expected_text: qa.answer } : {}),+ }));+ const name = `locomo-${dataset.name ?? "suite"}`;+ // Validate through the canonical parser so a converted fixture is+ // guaranteed to be exactly what runMemoryBench accepts.+ return parseBenchFixture({ name, notes, continuity, questions });+}++/** Load a LoCoMo dataset file and convert it to a BenchFixture. */+export function loadLocomoFixture(path: string): BenchFixture {+ let raw: unknown;+ try {+ raw = JSON.parse(readFileSync(path, "utf8"));+ } catch (e) {+ throw new LocomoParseError(+ `LoCoMo dataset not readable JSON at ${path}: ${e instanceof Error ? e.message : String(e)}`,+ );+ }+ return locomoToBenchFixture(parseLocomoDataset(raw));+}diff --git a/src/core/brain/continuity/types.ts b/src/core/brain/continuity/types.tsindex ff671455..5da7ae4d 100644--- a/src/core/brain/continuity/types.ts+++ b/src/core/brain/continuity/types.ts@@ -25,6 +25,7 @@ export type ContinuityRecordKind =| "token_impact_outcome"| "context_pack_outcome"| "host_memory_write"+ | "recall_observed_use"| "source_invalidation";export type ContinuityPayload = Readonly<Record<string, unknown>>;diff --git a/src/core/brain/observed-use.ts b/src/core/brain/observed-use.tsnew file mode 100644index 00000000..4c66c59b--- /dev/null+++ b/src/core/brain/observed-use.ts@@ -0,0 +1,179 @@+/**+ * Session-end observed-use verdict per injected memory (Retrieval &+ * Ranking Quality, t_65588d8b).+ *+ * At session end each memory that was injected this session is classified+ * as USED / IGNORED / CONTRADICTED, persisted as a `recall_observed_use`+ * continuity record, and folded into an observed-reuse rate per artifact+ * that recall ranking prefers over predicted importance.+ *+ * NO-LLM KERNEL INVARIANT. The kernel never calls a model to read the+ * transcript. A verdict is either:+ * - computed DETERMINISTICALLY here ({@link classifyObservedUse}) by+ * matching an injected memory's distinctive tokens against later+ * transcript turns, with CONTRADICTED derived from a structural stance+ * flip via the shared `deriveNoteStance` primitive (no vocabulary), or+ * - supplied ALREADY-STRUCTURED by the host via the MCP write tool,+ * mirroring `brain_apply_evidence`.+ * The kernel only stores and aggregates.+ */++import { appendContinuityRecord, listContinuityRecords } from "./continuity/store.ts";+import type { ContinuityRecord } from "./continuity/types.ts";+import { DEFAULT_NEGATION_MARKERS, deriveNoteStance } from "./health/contradiction.ts";++export const OBSERVED_USE_VERDICTS = ["USED", "IGNORED", "CONTRADICTED"] as const;+export type ObservedUseVerdict = (typeof OBSERVED_USE_VERDICTS)[number];++export function isObservedUseVerdict(value: unknown): value is ObservedUseVerdict {+ return value === "USED" || value === "IGNORED" || value === "CONTRADICTED";+}++/** One injected memory's observed-use verdict. */+export interface ObservedUseEntry {+ /** Stable memory id (e.g. "docId:chunkId" or a note id). */+ readonly id: string;+ /** Vault-relative path, when known (the ranking join key prefers it). */+ readonly path?: string;+ readonly verdict: ObservedUseVerdict;+}++export interface ObservedUseInput {+ readonly createdAt?: string;+ readonly host: string;+ readonly sessionId?: string;+ readonly turnId?: string;+ readonly entries: ReadonlyArray<ObservedUseEntry>;+}++/** An injected memory to classify against the transcript. */+export interface InjectedMemory {+ readonly id: string;+ readonly path?: string;+ readonly content: string;+}++const TOKEN_RE = /[\p{L}\p{N}]{2,}/gu;+const MIN_SHARED_TOKENS = 2;++function tokenSet(text: string): Set<string> {+ const out = new Set<string>();+ for (const m of text.toLowerCase().matchAll(TOKEN_RE)) out.add(m[0]);+ return out;+}++function sharedCount(a: ReadonlySet<string>, b: ReadonlySet<string>): number {+ let n = 0;+ const [small, large] = a.size <= b.size ? [a, b] : [b, a];+ for (const t of small) if (large.has(t)) n++;+ return n;+}++/**+ * Deterministically classify each injected memory against the session+ * transcript turns. A memory is USED when a later turn echoes at least+ * {@link MIN_SHARED_TOKENS} of its distinctive tokens; CONTRADICTED when+ * such an overlapping turn takes the opposite structural stance (a+ * `deriveNoteStance` sign flip); IGNORED otherwise. No model, no+ * per-language vocabulary beyond the shared structural negation markers.+ */+export function classifyObservedUse(+ injected: ReadonlyArray<InjectedMemory>,+ transcriptTurns: ReadonlyArray<string>,+): ObservedUseEntry[] {+ const turns = transcriptTurns.map((t) => tokenSet(t));+ return injected.map((mem) => {+ const memTokens = tokenSet(mem.content);+ const memStance = deriveNoteStance(memTokens, DEFAULT_NEGATION_MARKERS);+ let verdict: ObservedUseVerdict = "IGNORED";+ for (const turn of turns) {+ if (sharedCount(memTokens, turn) < MIN_SHARED_TOKENS) continue;+ const turnStance = deriveNoteStance(turn, DEFAULT_NEGATION_MARKERS);+ if (turnStance !== memStance) {+ verdict = "CONTRADICTED";+ break; // a contradiction outranks a plain reuse.+ }+ verdict = "USED";+ }+ return {+ id: mem.id,+ ...(mem.path ? { path: mem.path } : {}),+ verdict,+ };+ });+}++/**+ * Persist a session's observed-use verdicts as one continuity record.+ * `sourceRefs` point at the injected artifacts so a fold can join by id or+ * path. Returns the appended record.+ */+export function emitObservedUse(vault: string, input: ObservedUseInput): ContinuityRecord {+ const createdAt = input.createdAt ?? new Date().toISOString();+ const entries = [...input.entries];+ return appendContinuityRecord(vault, {+ kind: "recall_observed_use",+ createdAt,+ sourceRefs: entries.map((e) => ({ id: e.id, ...(e.path ? { path: e.path } : {}) })),+ payload: {+ host: input.host,+ ...(input.sessionId ? { session_id: input.sessionId } : {}),+ ...(input.turnId ? { turn_id: input.turnId } : {}),+ entries: entries.map((e) => ({+ id: e.id,+ ...(e.path ? { path: e.path } : {}),+ verdict: e.verdict,+ })),+ },+ });+}++/** Aggregated observed-use for one artifact. */+export interface ObservedReuse {+ readonly used: number;+ readonly ignored: number;+ readonly contradicted: number;+ readonly total: number;+ /**+ * Ranking signal in [0, 1]: `(used - contradicted) / total`, clamped.+ * Contradictions demote; a purely-ignored memory scores 0.+ */+ readonly score: number;+}++/**+ * Fold every `recall_observed_use` record into a per-artifact reuse map,+ * keyed by `path` when present else `id`. Order-insensitive.+ */+export function observedReuseRates(vault: string): Map<string, ObservedReuse> {+ const acc = new Map<string, { used: number; ignored: number; contradicted: number }>();+ for (const record of listContinuityRecords(vault, { kind: "recall_observed_use" })) {+ const entries = record.payload["entries"];+ if (!Array.isArray(entries)) continue;+ for (const raw of entries) {+ if (raw === null || typeof raw !== "object") continue;+ const e = raw as Record<string, unknown>;+ const verdict = e["verdict"];+ if (!isObservedUseVerdict(verdict)) continue;+ const key =+ typeof e["path"] === "string" && e["path"] !== ""+ ? (e["path"] as string)+ : typeof e["id"] === "string"+ ? (e["id"] as string)+ : null;+ if (key === null) continue;+ const cur = acc.get(key) ?? { used: 0, ignored: 0, contradicted: 0 };+ if (verdict === "USED") cur.used++;+ else if (verdict === "IGNORED") cur.ignored++;+ else cur.contradicted++;+ acc.set(key, cur);+ }+ }+ const out = new Map<string, ObservedReuse>();+ for (const [key, c] of acc) {+ const total = c.used + c.ignored + c.contradicted;+ const score = total > 0 ? Math.max(0, Math.min(1, (c.used - c.contradicted) / total)) : 0;+ out.set(key, { used: c.used, ignored: c.ignored, contradicted: c.contradicted, total, score });+ }+ return out;+}diff --git a/src/core/brain/procedural-memory.ts b/src/core/brain/procedural-memory.tsindex 2e6ebd1a..d0fb4fc3 100644--- a/src/core/brain/procedural-memory.ts+++ b/src/core/brain/procedural-memory.ts@@ -23,6 +23,25 @@ export interface ProceduralMemoryEntry {readonly version: string | null;readonly lastUsedAt: string | null;readonly usedCount: number;+ /**+ * Outcome-validated recall (t_703f7b18). Times this procedure was applied+ * and the host reported the downstream result. `successRate` ranks recall+ * (see {@link rankProceduralMemory}); usage count is the fallback prior+ * for procedures with no recorded outcomes. Additive: absent in an+ * outcome-free vault (defaults 0), so pre-outcome indexes read identically.+ */+ readonly successCount: number;+ readonly failureCount: number;+}++/** Host-reported outcome of applying a procedure. */+export type ProceduralOutcome = "success" | "failure";++interface UsageRecord {+ readonly usedCount: number;+ readonly lastUsedAt: string | null;+ readonly successCount: number;+ readonly failureCount: number;}export interface ProceduralReconcileOptions {@@ -49,6 +68,8 @@ export function reconcileProceduralMemory(...entry,lastUsedAt: used?.lastUsedAt ?? old?.lastUsedAt ?? null,usedCount: used?.usedCount ?? old?.usedCount ?? 0,+ successCount: used?.successCount ?? old?.successCount ?? 0,+ failureCount: used?.failureCount ?? old?.failureCount ?? 0,} satisfies ProceduralMemoryEntry;});@@ -86,12 +107,54 @@ export function listProceduralMemory(vault: string): ReadonlyArray<ProceduralMementries?: Array<ProceduralMemoryEntry>;};if (!Array.isArray(parsed.entries)) return Object.freeze([]);- return Object.freeze(parsed.entries);+ // Normalize additive outcome fields so a pre-outcome index (written+ // before t_703f7b18) reads with defaults instead of `undefined`.+ return Object.freeze(+ parsed.entries.map((entry) =>+ Object.freeze({+ ...entry,+ successCount: typeof entry.successCount === "number" ? entry.successCount : 0,+ failureCount: typeof entry.failureCount === "number" ? entry.failureCount : 0,+ }),+ ),+ );} catch {return Object.freeze([]);}}+/**+ * Validated success rate in [0, 1], or null when the procedure has no+ * recorded outcomes yet. `successes / (successes + failures)`.+ */+export function proceduralSuccessRate(entry: ProceduralMemoryEntry): number | null {+ const total = entry.successCount + entry.failureCount;+ return total > 0 ? entry.successCount / total : null;+}++/**+ * Rank procedures by validated success rate, not raw usage (t_703f7b18).+ * Proven-successful procedures rise, proven-failing ones sink, and+ * procedures with no recorded outcomes sit in a neutral band ordered by+ * the usage prior (usedCount then recency). Deterministic; no LLM.+ */+export function rankProceduralMemory(+ entries: ReadonlyArray<ProceduralMemoryEntry>,+): ReadonlyArray<ProceduralMemoryEntry> {+ const NEUTRAL = 0.5;+ const key = (e: ProceduralMemoryEntry): number => proceduralSuccessRate(e) ?? NEUTRAL;+ return [...entries].toSorted((a, b) => {+ const ra = key(a);+ const rb = key(b);+ if (ra !== rb) return rb - ra;+ if (a.usedCount !== b.usedCount) return b.usedCount - a.usedCount;+ const la = a.lastUsedAt ?? "";+ const lb = b.lastUsedAt ?? "";+ if (la !== lb) return la < lb ? 1 : -1;+ return a.id.localeCompare(b.id);+ });+}+export function markProceduralMemoryUsed(vault: string,id: string,@@ -117,6 +180,52 @@ export function markProceduralMemoryUsed(usage.set(id, {usedCount: (prev?.usedCount ?? 0) + 1,lastUsedAt: now.toISOString(),+ successCount: prev?.successCount ?? 0,+ failureCount: prev?.failureCount ?? 0,+ });+ writeUsageMap(vault, usage);+ const graph = rebuildProceduralGraph(vault);+ rebuildProceduralHints(vault, { graph });++ return next.find((entry) => entry.id === id) ?? null;+}++/**+ * Record the host-reported OUTCOME of applying a procedure (t_703f7b18).+ * Increments successCount or failureCount on the entry and the usage+ * sidecar; does NOT touch usedCount or lastUsedAt (that is+ * {@link markProceduralMemoryUsed}). The kernel never infers the outcome -+ * it is a structured enum supplied by the host, mirroring+ * `brain_apply_evidence`. Returns the updated entry, or null for an+ * unknown id. The fold is order-insensitive.+ */+export function recordProceduralOutcome(+ vault: string,+ id: string,+ outcome: ProceduralOutcome,+): ProceduralMemoryEntry | null {+ const entries = listProceduralMemory(vault);+ const target = entries.find((entry) => entry.id === id);+ if (!target) return null;++ const next = entries.map((entry) =>+ entry.id === id+ ? {+ ...entry,+ successCount: entry.successCount + (outcome === "success" ? 1 : 0),+ failureCount: entry.failureCount + (outcome === "failure" ? 1 : 0),+ }+ : entry,+ );+ writeIndex(vault, next);++ const usage = readUsageMap(vault);+ const prev = usage.get(id);+ usage.set(id, {+ usedCount: prev?.usedCount ?? target.usedCount,+ lastUsedAt: prev?.lastUsedAt ?? target.lastUsedAt,+ successCount: (prev?.successCount ?? 0) + (outcome === "success" ? 1 : 0),+ failureCount: (prev?.failureCount ?? 0) + (outcome === "failure" ? 1 : 0),});writeUsageMap(vault, usage);const graph = rebuildProceduralGraph(vault);@@ -152,6 +261,8 @@ function collectEntries(vault: string, roots: ReadonlyArray<string>): Proceduralversion: asStringOrNull(fm["version"]),lastUsedAt: null,usedCount: 0,+ successCount: 0,+ failureCount: 0,});}}@@ -241,13 +352,11 @@ function writeIndex(vault: string, entries: ReadonlyArray<ProceduralMemoryEntry>atomicWriteFileSync(path, `${payload}\n`);}-function readUsageMap(- vault: string,-): Map<string, { usedCount: number; lastUsedAt: string | null }> {+function readUsageMap(vault: string): Map<string, UsageRecord> {const path = proceduralMemoryUsagePath(vault);if (!existsSync(path)) return new Map();- const out = new Map<string, { usedCount: number; lastUsedAt: string | null }>();+ const out = new Map<string, UsageRecord>();for (const line of readFileSync(path, "utf8").split("\n")) {if (!line.trim()) continue;try {@@ -257,6 +366,8 @@ function readUsageMap(out.set(id, {usedCount: typeof parsed["usedCount"] === "number" ? parsed["usedCount"] : 0,lastUsedAt: typeof parsed["lastUsedAt"] === "string" ? parsed["lastUsedAt"] : null,+ successCount: typeof parsed["successCount"] === "number" ? parsed["successCount"] : 0,+ failureCount: typeof parsed["failureCount"] === "number" ? parsed["failureCount"] : 0,});} catch {continue;@@ -265,23 +376,23 @@ function readUsageMap(return out;}-function writeUsageMap(- vault: string,- usage: ReadonlyMap<string, { usedCount: number; lastUsedAt: string | null }>,-): void {+function writeUsageMap(vault: string, usage: ReadonlyMap<string, UsageRecord>): void {const path = proceduralMemoryUsagePath(vault);mkdirSync(ensureInsideVault(dirname(path), vault), { recursive: true });const lines: string[] = [];for (const [id, value] of [...usage.entries()].toSorted((left, right) =>left[0].localeCompare(right[0]),)) {- lines.push(- JSON.stringify({- id,- usedCount: value.usedCount,- lastUsedAt: value.lastUsedAt,- }),- );+ // Omit zero outcome counters so an outcome-free vault's usage file stays+ // byte-identical to the pre-t_703f7b18 format.+ const record: Record<string, unknown> = {+ id,+ usedCount: value.usedCount,+ lastUsedAt: value.lastUsedAt,+ };+ if (value.successCount > 0) record["successCount"] = value.successCount;+ if (value.failureCount > 0) record["failureCount"] = value.failureCount;+ lines.push(JSON.stringify(record));}atomicWriteFileSync(path, `${lines.join("\n")}${lines.length > 0 ? "\n" : ""}`);}diff --git a/src/core/search/embeddings/http-util.ts b/src/core/search/embeddings/http-util.tsnew file mode 100644index 00000000..7f8d64f9--- /dev/null+++ b/src/core/search/embeddings/http-util.ts@@ -0,0 +1,63 @@+/**+ * Shared, dependency-free helpers for HTTP embedding providers+ * (openai-compat, zeroentropy). Pure utilities only - no network, no+ * config knowledge - so each provider owns its own request/response+ * mapping while reusing identical batching, concurrency, backoff, and+ * unit-normalisation semantics.+ */++/** Statuses that warrant a transient retry with backoff. */+export const RETRYABLE_STATUSES: ReadonlySet<number> = new Set([429, 500, 502, 503, 504]);++/** Auth statuses that trigger failover to the next probe key. */+export const AUTH_STATUSES: ReadonlySet<number> = new Set([401, 403]);++export function sleep(ms: number): Promise<void> {+ return new Promise((res) => setTimeout(res, ms));+}++/** Apply Β±25% jitter to a backoff base (never negative). */+export function jittered(base: number): number {+ const jitter = base * 0.25 * (Math.random() * 2 - 1);+ return Math.max(0, base + jitter);+}++/** A minimal counting semaphore for bounded batch concurrency. */+export class Semaphore {+ private permits: number;+ private readonly waiters: Array<() => void> = [];+ constructor(n: number) {+ this.permits = Math.max(1, n | 0);+ }+ async acquire(): Promise<void> {+ if (this.permits > 0) {+ this.permits--;+ return;+ }+ await new Promise<void>((res) => this.waiters.push(res));+ this.permits--;+ }+ release(): void {+ this.permits++;+ const next = this.waiters.shift();+ if (next) next();+ }+}++/** Unit-normalise a vector in place so cosine similarity equals 1 - L2Β²/2. */+export function unitNormaliseInPlace(v: number[]): number[] {+ let s = 0;+ for (const x of v) s += x * x;+ const norm = Math.sqrt(s);+ if (norm === 0) return v;+ for (let i = 0; i < v.length; i++) v[i] = (v[i] ?? 0) / norm;+ return v;+}++/** Split an array into fixed-size chunks (step >= 1). */+export function chunkArray<T>(arr: ReadonlyArray<T>, size: number): T[][] {+ const step = Math.max(1, size | 0);+ const out: T[][] = [];+ for (let i = 0; i < arr.length; i += step) out.push(arr.slice(i, i + step) as T[]);+ return out;+}diff --git a/src/core/search/embeddings/openai-compat.ts b/src/core/search/embeddings/openai-compat.tsindex 2a07c010..1cb84e3e 100644--- a/src/core/search/embeddings/openai-compat.ts+++ b/src/core/search/embeddings/openai-compat.ts@@ -20,8 +20,15 @@import { SearchError } from "../types.ts";import type { ResolvedEmbeddingConfig } from "../types.ts";import type { EmbeddingProvider } from "./provider.ts";--const RETRYABLE_STATUSES = new Set([429, 500, 502, 503, 504]);+import {+ AUTH_STATUSES,+ RETRYABLE_STATUSES,+ Semaphore,+ chunkArray,+ jittered,+ sleep,+ unitNormaliseInPlace,+} from "./http-util.ts";interface OpenAiEmbeddingResponse {readonly data: ReadonlyArray<{@@ -31,52 +38,6 @@ interface OpenAiEmbeddingResponse {readonly model?: string;}-function sleep(ms: number): Promise<void> {- return new Promise((res) => setTimeout(res, ms));-}--function jittered(base: number): number {- const jitter = base * 0.25 * (Math.random() * 2 - 1); // Β±25%- return Math.max(0, base + jitter);-}--class Semaphore {- private permits: number;- private readonly waiters: Array<() => void> = [];- constructor(n: number) {- this.permits = Math.max(1, n | 0);- }- async acquire(): Promise<void> {- if (this.permits > 0) {- this.permits--;- return;- }- await new Promise<void>((res) => this.waiters.push(res));- this.permits--;- }- release(): void {- this.permits++;- const next = this.waiters.shift();- if (next) next();- }-}--function unitNormaliseInPlace(v: number[]): number[] {- let s = 0;- for (const x of v) s += x * x;- const norm = Math.sqrt(s);- if (norm === 0) return v;- for (let i = 0; i < v.length; i++) v[i] = (v[i] ?? 0) / norm;- return v;-}--function chunkArray<T>(arr: ReadonlyArray<T>, size: number): T[][] {- const step = Math.max(1, size | 0);- const out: T[][] = [];- for (let i = 0; i < arr.length; i += step) out.push(arr.slice(i, i + step) as T[]);- return out;-}-interface ResolvedHttp {readonly url: string;readonly apiKey: string;@@ -102,6 +63,16 @@ function resolveHttp(config: ResolvedEmbeddingConfig): ResolvedHttp {return { url: `${base}/embeddings`, apiKey: config.apiKey };}+/**+ * Ordered API-key failover list. Uses `config.apiKeys` when present and+ * non-empty, otherwise the single resolved key. `resolveHttp` has already+ * guaranteed a non-empty first key, so the list is never empty.+ */+function resolveKeys(config: ResolvedEmbeddingConfig, fallbackKey: string): string[] {+ const list = (config.apiKeys ?? []).map((k) => k.trim()).filter((k) => k !== "");+ return list.length > 0 ? list : [fallbackKey];+}+export interface OpenAICompatProviderOptions {/** Override default `[1000, 2000]` ms backoffs (used by tests). */readonly backoffMs?: ReadonlyArray<number>;@@ -114,6 +85,9 @@ export class OpenAICompatProvider implements EmbeddingProvider {private readonly config: ResolvedEmbeddingConfig;private readonly http: ResolvedHttp;private readonly backoffMs: ReadonlyArray<number>;+ /** Ordered probe keys; `activeKeyIndex` pins the first that authenticates. */+ private readonly keys: ReadonlyArray<string>;+ private activeKeyIndex = 0;constructor(config: ResolvedEmbeddingConfig, opts?: OpenAICompatProviderOptions) {this.config = config;@@ -121,6 +95,11 @@ export class OpenAICompatProvider implements EmbeddingProvider {this.model = config.model!;this._dimension = config.dimension;this.backoffMs = opts?.backoffMs ?? [1000, 2000];+ this.keys = resolveKeys(config, this.http.apiKey);+ }++ private get activeKey(): string {+ return this.keys[this.activeKeyIndex] ?? this.http.apiKey;}get dimension(): number | null {@@ -191,9 +170,50 @@ export class OpenAICompatProvider implements EmbeddingProvider {}}+ /**+ * Embed one batch with transient-retry AND multi-key failover: run the+ * active key through the full transient-retry budget; on an auth error+ * (HTTP 401/403) advance to the next probe key and retry, pinning the+ * first key that authenticates. Bounded by the number of keys, so a+ * fully-invalid probe list terminates and surfaces the auth error. With+ * a single key the outer loop runs once - byte-identical to before.+ */private async embedBatchWithRetry(texts: string[],opts?: { maxAttempts?: number; parentSignal?: AbortSignal },+ ): Promise<number[][]> {+ let lastError: SearchError | null = null;+ for (let keyAttempt = 0; keyAttempt < this.keys.length; keyAttempt++) {+ const usedKeyIndex = this.activeKeyIndex;+ try {+ return await this.embedBatchOnKey(texts, opts);+ } catch (e) {+ const err =+ e instanceof SearchError ? e : new SearchError("EMBEDDING_PROVIDER_HTTP", String(e));+ lastError = err;+ if (!this.isAuthError(err)) throw err;+ if (usedKeyIndex === this.activeKeyIndex && this.activeKeyIndex < this.keys.length - 1) {+ // We are the first to see this key fail; advance to the next.+ this.activeKeyIndex++;+ } else if (this.activeKeyIndex <= usedKeyIndex) {+ // No further key to fail over to; surface the auth error.+ throw err;+ }+ // Otherwise a concurrent batch already advanced the key; retry with it.+ }+ }+ throw lastError ?? new SearchError("EMBEDDING_PROVIDER_HTTP", "key failover exhausted");+ }++ private isAuthError(e: SearchError): boolean {+ if (e.code !== "EMBEDDING_PROVIDER_HTTP") return false;+ const m = e.message.match(/HTTP (\d+)/);+ return m ? AUTH_STATUSES.has(Number(m[1])) : false;+ }++ private async embedBatchOnKey(+ texts: string[],+ opts?: { maxAttempts?: number; parentSignal?: AbortSignal },): Promise<number[][]> {const maxAttempts = opts?.maxAttempts ?? 3;let lastError: SearchError | null = null;@@ -241,7 +261,7 @@ export class OpenAICompatProvider implements EmbeddingProvider {method: "POST",headers: {"content-type": "application/json",- authorization: `Bearer ${this.http.apiKey}`,+ authorization: `Bearer ${this.activeKey}`,},body: JSON.stringify({model: this.model,diff --git a/src/core/search/embeddings/presets.ts b/src/core/search/embeddings/presets.tsnew file mode 100644index 00000000..6c9a5196--- /dev/null+++ b/src/core/search/embeddings/presets.ts@@ -0,0 +1,89 @@+/**+ * Curated embedding-model presets (Retrieval & Ranking Quality).+ *+ * A static, shippable catalog of known-good embedding models surfaced when+ * a user registers an OpenAI-compatible provider, plus a recommended+ * multilingual default. Advisory only: the free-form custom `--model`+ * entry stays first-class, and OSB targets arbitrary OpenAI-compatible+ * endpoints, so a preset is guidance (model string + dimension + a note),+ * never a constraint. No server, no network - the list is consulted+ * entirely at registration time.+ *+ * Presets are multilingual-first because OSB is language-agnostic: a+ * multilingual default avoids the dimension/quality mistakes that later+ * force a full re-embed. The `dimension` is the model's native embedding+ * width, useful when setting `embedding_dimension` up front.+ */++/** One curated embedding model the registration flow can recommend. */+export interface EmbeddingModelPreset {+ /** Model string sent to the endpoint (`embedding_model` / profile defaultModel). */+ readonly model: string;+ /** Short human label for CLI listings. */+ readonly label: string;+ /** Native embedding dimension. */+ readonly dimension: number;+ /** True when the model is trained for cross-lingual retrieval. */+ readonly multilingual: boolean;+ /** One-line guidance shown alongside the model. */+ readonly note: string;+}++/**+ * Curated catalog. Ordered best-general-default first. These are the+ * widely-deployed open multilingual embedding models; a provider exposing+ * them under a different string can still be registered with a custom+ * `--model`.+ */+export const EMBEDDING_MODEL_PRESETS: ReadonlyArray<EmbeddingModelPreset> = Object.freeze([+ {+ model: "intfloat/multilingual-e5-small",+ label: "multilingual-e5-small",+ dimension: 384,+ multilingual: true,+ note: "Small, fast, strong multilingual default. Prefix inputs with 'query:'/'passage:'.",+ },+ {+ model: "BAAI/bge-m3",+ label: "bge-m3",+ dimension: 1024,+ multilingual: true,+ note: "High-quality multilingual, 100+ languages. Larger vectors, higher cost.",+ },+ {+ model: "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",+ label: "paraphrase-multilingual-MiniLM-L12-v2",+ dimension: 384,+ multilingual: true,+ note: "Compact multilingual paraphrase model; good latency/quality balance.",+ },+ {+ model: "Alibaba-NLP/gte-multilingual-base",+ label: "gte-multilingual-base",+ dimension: 768,+ multilingual: true,+ note: "Balanced multilingual retrieval model with long context.",+ },+ {+ model: "sentence-transformers/LaBSE",+ label: "LaBSE",+ dimension: 768,+ multilingual: true,+ note: "109-language sentence embeddings; strong cross-lingual alignment.",+ },+ {+ model: "BAAI/bge-small-zh-v1.5",+ label: "bge-small-zh-v1.5",+ dimension: 512,+ multilingual: false,+ note: "Chinese-optimized small model; pick when the vault is predominantly zh.",+ },+]);++/** The recommended general-purpose default model string. */+export const RECOMMENDED_EMBEDDING_MODEL: string = EMBEDDING_MODEL_PRESETS[0]!.model;++/** Look up a preset by exact model string (null when not curated). */+export function findEmbeddingPreset(model: string): EmbeddingModelPreset | null {+ return EMBEDDING_MODEL_PRESETS.find((p) => p.model === model) ?? null;+}diff --git a/src/core/search/embeddings/provider.ts b/src/core/search/embeddings/provider.tsindex 14db8971..6923535a 100644--- a/src/core/search/embeddings/provider.ts+++ b/src/core/search/embeddings/provider.ts@@ -44,6 +44,12 @@ export function makeProvider(config: ResolvedEmbeddingConfig): EmbeddingProviderrequire("./openai-compat.ts") as typeof import("./openai-compat.ts");return new OpenAICompatProvider(config);}+ if (config.provider === "zeroentropy") {+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const { ZeroEntropyProvider } =+ require("./zeroentropy.ts") as typeof import("./zeroentropy.ts");+ return new ZeroEntropyProvider(config);+ }throw new SearchError("INVALID_INPUT",`unknown embedding_provider '${String((config as { provider?: string }).provider)}'`,diff --git a/src/core/search/embeddings/registry.ts b/src/core/search/embeddings/registry.tsindex 38023925..e3e7811f 100644--- a/src/core/search/embeddings/registry.ts+++ b/src/core/search/embeddings/registry.ts@@ -24,8 +24,19 @@ export interface ProviderProfile {readonly name: string;readonly baseUrl: string;readonly defaultModel: string;- /** Environment variable NAME the API key is read from (never the key). */- readonly envKey: string;+ /**+ * Environment variable NAME(s) the API key is read from (never the key).+ * A single string keeps the original single-key behaviour; an ordered+ * list is a probe list resolved first-present-wins, with the remaining+ * present keys kept as request-time failover candidates.+ */+ readonly envKey: string | ReadonlyArray<string>;+}++/** Normalise `envKey` (string | list) to an ordered list of env-var names. */+function envKeyList(envKey: ProviderProfile["envKey"]): string[] {+ const raw = typeof envKey === "string" ? [envKey] : [...envKey];+ return raw.map((k) => k.trim()).filter((k) => k !== "");}/** Built-in provider names a profile may not reuse. */@@ -33,6 +44,7 @@ export const RESERVED_PROVIDER_NAMES: ReadonlyArray<string> = Object.freeze(["openai-compat","disabled","local",+ "zeroentropy",]);const NAME_RE = /^[a-z0-9][a-z0-9-]*$/;@@ -41,6 +53,11 @@ export function providerRegistryPath(vault: string): string {return join(vault, "Brain", "search", "embedding-providers.json");}+function isEnvKey(value: unknown): value is string | ReadonlyArray<string> {+ if (typeof value === "string") return true;+ return Array.isArray(value) && value.every((k) => typeof k === "string");+}+function isProfile(value: unknown): value is ProviderProfile {if (value === null || typeof value !== "object") return false;const v = value as Record<string, unknown>;@@ -48,7 +65,7 @@ function isProfile(value: unknown): value is ProviderProfile {typeof v["name"] === "string" &&typeof v["baseUrl"] === "string" &&typeof v["defaultModel"] === "string" &&- typeof v["envKey"] === "string"+ isEnvKey(v["envKey"]));}@@ -98,8 +115,11 @@ function validateProfile(profile: ProviderProfile): void {if (profile.defaultModel.trim() === "") {throw new SearchError("INVALID_INPUT", "provider default-model must not be empty");}- if (profile.envKey.trim() === "") {- throw new SearchError("INVALID_INPUT", "provider env-key must not be empty");+ if (envKeyList(profile.envKey).length === 0) {+ throw new SearchError(+ "INVALID_INPUT",+ "provider env-key must not be empty (and a probe list must hold at least one non-empty name)",+ );}}@@ -109,11 +129,14 @@ function validateProfile(profile: ProviderProfile): void {*/export function addProviderProfile(vault: string, profile: ProviderProfile): ProviderProfile[] {validateProfile(profile);+ const keys = envKeyList(profile.envKey);const normalised: ProviderProfile = Object.freeze({name: profile.name.trim(),baseUrl: profile.baseUrl.trim(),defaultModel: profile.defaultModel.trim(),- envKey: profile.envKey.trim(),+ // Preserve the single-string shape for single-key profiles so existing+ // registry files stay byte-identical; store a list only when given one.+ envKey: typeof profile.envKey === "string" ? keys[0]! : Object.freeze(keys),});const without = loadProviderRegistry(vault).filter((p) => p.name !== normalised.name);const next = [...without, normalised];@@ -142,13 +165,18 @@ export interface ExpandedProvider {readonly provider: "openai-compat";readonly baseUrl: string;readonly model: string;+ /** First present probe key (backward-compatible single value); null if none set. */readonly apiKey: string | null;+ /** All present probe keys in profile order; request-time failover candidates. */+ readonly apiKeys: ReadonlyArray<string>;}/**- * Expand a registered provider name into `openai-compat` config, reading- * the API key from `env[profile.envKey]`. Returns null when the name is- * not registered (the caller then falls back to built-in resolution).+ * Expand a registered provider name into `openai-compat` config, resolving+ * the API key(s) from the profile's `envKey` probe list (first-present+ * wins; the rest are kept as failover candidates). Returns null when the+ * name is not registered (the caller then falls back to built-in+ * resolution).*/export function expandRegisteredProvider(name: string,@@ -157,11 +185,14 @@ export function expandRegisteredProvider(): ExpandedProvider | null {const profile = registry.find((p) => p.name === name);if (!profile) return null;- const apiKey = env[profile.envKey];+ const present = envKeyList(profile.envKey)+ .map((varName) => env[varName])+ .filter((v): v is string => v !== undefined && v !== "");return Object.freeze({provider: "openai-compat" as const,baseUrl: profile.baseUrl,model: profile.defaultModel,- apiKey: apiKey !== undefined && apiKey !== "" ? apiKey : null,+ apiKey: present[0] ?? null,+ apiKeys: Object.freeze(present),});}diff --git a/src/core/search/embeddings/zeroentropy.ts b/src/core/search/embeddings/zeroentropy.tsnew file mode 100644index 00000000..85133747--- /dev/null+++ b/src/core/search/embeddings/zeroentropy.ts@@ -0,0 +1,279 @@+/**+ * Native ZeroEntropy embedding provider (Retrieval & Ranking Quality).+ *+ * Calls the ZeroEntropy native embed API (`POST {base}/models/embed`)+ * directly over `fetch` - no SDK, no new dependency - instead of routing+ * through the OpenAI-compatible wrapper. Request/response shape per the+ * ZeroEntropy API reference:+ *+ * request: { model, input_type, input: string[], encoding_format, dimensions? }+ * response: { results: [{ embedding: number[] }, ...], usage? }+ *+ * `results` preserves input order (there is no per-item index field), so+ * vectors map positionally. Network discipline mirrors the OpenAI-compat+ * provider: bounded concurrency, per-request timeout, retry on 429/5xx and+ * network/timeout with jittered backoff, vectors unit-normalised so cosine+ * equals 1 - L2Β²/2.+ *+ * The interface is symmetric (one `embed`), so `input_type` is fixed to+ * "document" - consistent with how OSB treats query and passage embeddings+ * uniformly across every provider.+ */++import { SearchError } from "../types.ts";+import type { ResolvedEmbeddingConfig } from "../types.ts";+import type { EmbeddingProvider } from "./provider.ts";+import {+ RETRYABLE_STATUSES,+ Semaphore,+ chunkArray,+ jittered,+ sleep,+ unitNormaliseInPlace,+} from "./http-util.ts";++const ZEROENTROPY_INPUT_TYPE = "document";++interface ZeroEntropyEmbeddingResponse {+ readonly results: ReadonlyArray<{ readonly embedding: ReadonlyArray<number> }>;+}++interface ResolvedHttp {+ readonly url: string;+ readonly apiKey: string;+}++function resolveHttp(config: ResolvedEmbeddingConfig): ResolvedHttp {+ if (!config.baseUrl) {+ throw new SearchError(+ "INVALID_INPUT",+ "embedding_base_url is required when semantic is enabled",+ );+ }+ if (!config.model) {+ throw new SearchError("INVALID_INPUT", "embedding_model is required when semantic is enabled");+ }+ if (!config.apiKey) {+ throw new SearchError(+ "EMBEDDING_KEY_MISSING",+ "embedding_api_key is required when semantic is enabled",+ );+ }+ const base = config.baseUrl.replace(/\/+$/, "");+ return { url: `${base}/models/embed`, apiKey: config.apiKey };+}++export interface ZeroEntropyProviderOptions {+ /** Override default `[1000, 2000]` ms backoffs (used by tests). */+ readonly backoffMs?: ReadonlyArray<number>;+}++export class ZeroEntropyProvider implements EmbeddingProvider {+ readonly name = "zeroentropy";+ readonly model: string;+ private _dimension: number | null;+ private readonly config: ResolvedEmbeddingConfig;+ private readonly http: ResolvedHttp;+ private readonly backoffMs: ReadonlyArray<number>;+ private retriesSeen = 0;++ constructor(config: ResolvedEmbeddingConfig, opts?: ZeroEntropyProviderOptions) {+ this.config = config;+ this.http = resolveHttp(config);+ this.model = config.model!;+ this._dimension = config.dimension;+ this.backoffMs = opts?.backoffMs ?? [1000, 2000];+ }++ get dimension(): number | null {+ return this._dimension;+ }++ consumeRetryCount(): number {+ const n = this.retriesSeen;+ this.retriesSeen = 0;+ return n;+ }++ async embed(texts: ReadonlyArray<string>): Promise<number[][]> {+ if (texts.length === 0) return [];+ const batches = chunkArray(+ texts.map((t, i) => ({ text: t, originalIndex: i })),+ this.config.batchSize,+ );+ const sem = new Semaphore(this.config.concurrency);+ const out: number[][] = new Array(texts.length);+ const cancel = new AbortController();++ const tasks = batches.map(async (batch) => {+ await sem.acquire();+ try {+ if (cancel.signal.aborted) return;+ const vectors = await this.embedBatchWithRetry(+ batch.map((b) => b.text),+ { parentSignal: cancel.signal },+ );+ for (let i = 0; i < vectors.length; i++) {+ out[batch[i]!.originalIndex] = vectors[i]!;+ }+ } finally {+ sem.release();+ }+ });++ try {+ try {+ await Promise.all(tasks);+ } catch (firstError) {+ cancel.abort();+ await Promise.allSettled(tasks);+ if (firstError instanceof SearchError) throw firstError;+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", String(firstError));+ }+ } finally {+ cancel.abort();+ }+ return out;+ }++ async ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }> {+ try {+ const vectors = await this.embedBatchWithRetry(["check"], { maxAttempts: 1 });+ const v = vectors[0];+ if (!v) return { ok: false, reason: "empty response" };+ return { ok: true, dimension: v.length };+ } catch (e) {+ const reason = e instanceof Error ? e.message : String(e);+ return { ok: false, reason };+ }+ }++ private async embedBatchWithRetry(+ texts: string[],+ opts?: { maxAttempts?: number; parentSignal?: AbortSignal },+ ): Promise<number[][]> {+ const maxAttempts = opts?.maxAttempts ?? 3;+ let lastError: SearchError | null = null;+ for (let attempt = 1; attempt <= maxAttempts; attempt++) {+ if (opts?.parentSignal?.aborted) {+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", "embed cancelled");+ }+ try {+ return await this.embedBatchOnce(texts, opts?.parentSignal);+ } catch (e) {+ const err = this.classifyError(e);+ if (!err.retriable || attempt >= maxAttempts) throw err.error;+ lastError = err.error;+ this.retriesSeen++;+ const wait = jittered(+ this.backoffMs[attempt - 1] ?? this.backoffMs[this.backoffMs.length - 1] ?? 4000,+ );+ await sleep(wait);+ }+ }+ throw lastError ?? new SearchError("EMBEDDING_PROVIDER_HTTP", "retry loop exhausted");+ }++ private async embedBatchOnce(texts: string[], parentSignal?: AbortSignal): Promise<number[][]> {+ const controller = new AbortController();+ const timer = setTimeout(() => controller.abort(), this.config.timeoutMs);+ const onParentAbort = () => controller.abort();+ if (parentSignal) {+ if (parentSignal.aborted) controller.abort();+ else parentSignal.addEventListener("abort", onParentAbort, { once: true });+ }+ let response: Response;+ try {+ response = await fetch(this.http.url, {+ method: "POST",+ headers: {+ "content-type": "application/json",+ authorization: `Bearer ${this.http.apiKey}`,+ },+ body: JSON.stringify({+ model: this.model,+ input_type: ZEROENTROPY_INPUT_TYPE,+ input: texts,+ encoding_format: "float",+ ...(this._dimension !== null ? { dimensions: this._dimension } : {}),+ }),+ signal: controller.signal,+ });+ } catch (e) {+ const cause = e instanceof Error ? e : new Error(String(e));+ if (cause.name === "AbortError") {+ if (parentSignal?.aborted) {+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", "embed cancelled");+ }+ throw new SearchError(+ "EMBEDDING_PROVIDER_TIMEOUT",+ `embedding request timed out after ${this.config.timeoutMs}ms`,+ );+ }+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `network error: ${cause.message}`);+ } finally {+ clearTimeout(timer);+ if (parentSignal) parentSignal.removeEventListener("abort", onParentAbort);+ }++ if (!response.ok) {+ const body = await response.text().catch(() => "");+ const head = body.slice(0, 300);+ throw new SearchError(+ "EMBEDDING_PROVIDER_HTTP",+ `embedding HTTP ${response.status}: ${head || response.statusText}`,+ );+ }++ let json: ZeroEntropyEmbeddingResponse;+ try {+ json = (await response.json()) as ZeroEntropyEmbeddingResponse;+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `embedding response not JSON: ${msg}`);+ }++ if (!Array.isArray(json.results) || json.results.length !== texts.length) {+ throw new SearchError(+ "EMBEDDING_PROVIDER_HTTP",+ `embedding response shape: expected ${texts.length} vectors, got ${json.results?.length ?? "none"}`,+ );+ }++ const ordered: number[][] = new Array(texts.length);+ for (let i = 0; i < json.results.length; i++) {+ const item = json.results[i]!;+ if (!Array.isArray(item.embedding)) {+ throw new SearchError(+ "EMBEDDING_PROVIDER_HTTP",+ `embedding response: results[${i}].embedding is not an array`,+ );+ }+ const arr = (item.embedding as ReadonlyArray<number>).slice();+ if (this._dimension === null) {+ this._dimension = arr.length;+ } else if (this._dimension !== arr.length) {+ throw new SearchError(+ "EMBEDDING_DIMENSION_MISMATCH",+ `embedding dimension changed mid-batch: expected ${this._dimension}, got ${arr.length}`,+ );+ }+ ordered[i] = unitNormaliseInPlace(arr);+ }+ return ordered;+ }++ private classifyError(e: unknown): { retriable: boolean; error: SearchError } {+ if (e instanceof SearchError) {+ if (e.message.includes("embed cancelled")) return { retriable: false, error: e };+ if (e.code === "EMBEDDING_PROVIDER_TIMEOUT") return { retriable: true, error: e };+ if (e.code === "EMBEDDING_PROVIDER_HTTP") {+ const m = e.message.match(/HTTP (\d+)/);+ if (m) return { retriable: RETRYABLE_STATUSES.has(Number(m[1])), error: e };+ return { retriable: true, error: e };+ }+ return { retriable: false, error: e };+ }+ return { retriable: false, error: new SearchError("EMBEDDING_PROVIDER_HTTP", String(e)) };+ }+}diff --git a/src/core/search/enrich.ts b/src/core/search/enrich.tsindex bd420c1b..d6f18b38 100644--- a/src/core/search/enrich.ts+++ b/src/core/search/enrich.ts@@ -42,6 +42,7 @@ export function projectScoreBreakdown(result: BrainSearchResult): ScoreBreakdownentity: 0,activation: 0,coAccess: 0,+ reuse: 0,link: result.linkBoost,recency: result.recencyBoost,tier: 1,diff --git a/src/core/search/graph-prepass.ts b/src/core/search/graph-prepass.tsnew file mode 100644index 00000000..b8863bce--- /dev/null+++ b/src/core/search/graph-prepass.ts@@ -0,0 +1,231 @@+/**+ * Graph-index query pre-pass (Retrieval & Ranking Quality, t_59ae326f).+ *+ * A read-planning stage that decides WHICH notes are worth opening, and+ * can answer from index metadata alone. Given a query it:+ *+ * 1. Ranks candidate notes by title / summary term match, wikilink+ * degree, and (optional) tier weight - the "seeds".+ * 2. Connects the top seeds through MULTI-HOP BFS over the resolved+ * wikilink graph (the existing `getGraphSnapshot` undirected+ * adjacency), so a note reachable only via several link hops is+ * surfaced - the capability the 1-hop link boost in the ranker lacks.+ * 3. Returns a `should_read` shortlist instead of hydrating chunks+ * blindly.+ *+ * In `index_only` mode it ranks and answers from the index (path, title,+ * degree, link paths) with ZERO note bodies read - only `documentTitles`+ * (a `documents`-table read) is consulted. This complements `recall_gate`+ * (decides IF to recall) and `context_pack` (budget-bounds WHAT is+ * injected) with a WHICH-to-read planner. Pure-stdlib, no new dependency.+ */++import { Store } from "./store.ts";+import { getGraphSnapshot } from "../brain/link-graph/graph-index.ts";+import type { ResolvedSearchConfig } from "./types.ts";++/** One entry in the should-read shortlist. */+export interface ShouldReadEntry {+ readonly documentId: number;+ readonly path: string;+ readonly title: string | null;+ readonly degree: number;+ /** Minimum link-hops from any seed (0 = the note is itself a seed). */+ readonly hops: number;+ readonly score: number;+ /** Preview of the note head; null in index_only mode (no body read). */+ readonly summary: string | null;+ /** Explainable contributions, e.g. ["title", "degree", "bfs:2"]. */+ readonly reasons: ReadonlyArray<string>;+}++export interface GraphPrepassResult {+ readonly mode: "should_read" | "index_only";+ readonly shortlist: ReadonlyArray<ShouldReadEntry>;+ /** Count of note bodies hydrated (always 0 in index_only mode). */+ readonly notesRead: number;+}++export interface GraphPrepassOptions {+ /** Answer from index metadata with zero note bodies read. */+ readonly indexOnly?: boolean;+ /** Max seeds carried into BFS (default 10). */+ readonly seedLimit?: number;+ /** BFS hop depth over the wikilink graph (default 2). */+ readonly maxHops?: number;+ /** Per-hop score multiplier in (0, 1] (default 0.5). */+ readonly hopDecay?: number;+ /** Max shortlist entries returned (default 10). */+ readonly shortlistLimit?: number;+ /** Weight of the normalized wikilink degree signal (default 0.15). */+ readonly degreeWeight?: number;+ /** Summary-preview length in characters for should_read mode (default 200). */+ readonly summaryChars?: number;+}++const WORD_RE = /[\p{L}\p{N}]{2,}/gu;++function queryTerms(query: string): string[] {+ const out = new Set<string>();+ for (const m of query.toLowerCase().matchAll(WORD_RE)) out.add(m[0]);+ return [...out];+}++/** Fraction of query terms present in `text` (case-insensitive substring). */+function termMatchFraction(text: string | null, terms: ReadonlyArray<string>): number {+ if (!text || terms.length === 0) return 0;+ const hay = text.toLowerCase();+ let hits = 0;+ for (const t of terms) if (hay.includes(t)) hits++;+ return hits / terms.length;+}++/**+ * Plan a should-read shortlist for a query. Deterministic: ties break by+ * ascending path, so identical index + query yields identical output.+ */+export function planReadShortlist(+ store: Store,+ query: string,+ opts: GraphPrepassOptions = {},+): GraphPrepassResult {+ const indexOnly = opts.indexOnly === true;+ const seedLimit = Math.max(1, Math.floor(opts.seedLimit ?? 10));+ const maxHops = Math.max(0, Math.floor(opts.maxHops ?? 2));+ const hopDecay = opts.hopDecay ?? 0.5;+ const shortlistLimit = Math.max(1, Math.floor(opts.shortlistLimit ?? 10));+ const degreeWeight = opts.degreeWeight ?? 0.15;+ const summaryChars = Math.max(0, Math.floor(opts.summaryChars ?? 200));++ const snapshot = getGraphSnapshot(store);+ const titles = store.documentTitles();+ const terms = queryTerms(query);++ const maxDegree = Math.max(1, ...[...snapshot.degree.values()], 1);++ // ββ seed ranking: title match + degree (+ summary match when bodies allowed).+ const seedScore = new Map<number, number>();+ const seedReasons = new Map<number, string[]>();+ // A note becomes a seed only when the query matches its title - degree+ // merely BOOSTS a matched seed, it never seeds an unrelated note. This+ // keeps the pre-pass query-driven (no match -> no seeds -> empty plan)+ // and index-light (title is a `documents`-column read, no body scan).+ for (const [id, meta] of titles) {+ const titleFrac = termMatchFraction(meta.title, terms);+ if (titleFrac <= 0) continue;+ const degree = snapshot.degree.get(id) ?? 0;+ const degreeNorm = (degree / maxDegree) * degreeWeight;+ const reasons = ["title"];+ if (degreeNorm > 0) reasons.push("degree");+ seedScore.set(id, titleFrac + degreeNorm);+ seedReasons.set(id, reasons);+ }++ // In should_read mode, enrich seeds with a summary-match signal from the+ // note head (one representative chunk per candidate seed).+ const summaries = new Map<number, string>();+ if (!indexOnly && seedScore.size > 0) {+ const seedIds = [...seedScore.keys()];+ const reps = store.representativeChunks(seedIds);+ for (const [id, chunk] of reps) {+ const head = chunk.content.slice(0, summaryChars);+ summaries.set(id, head);+ const frac = termMatchFraction(chunk.content, terms);+ if (frac > 0) {+ seedScore.set(id, (seedScore.get(id) ?? 0) + frac * 0.5);+ const r = seedReasons.get(id) ?? [];+ if (!r.includes("summary")) r.push("summary");+ seedReasons.set(id, r);+ }+ }+ }++ const seeds = [...seedScore.entries()]+ .toSorted((a, b) =>+ a[1] !== b[1]+ ? b[1] - a[1]+ : (titles.get(a[0])?.path ?? "") < (titles.get(b[0])?.path ?? "")+ ? -1+ : 1,+ )+ .slice(0, seedLimit);++ // ββ multi-hop BFS from the seeds over the undirected wikilink graph.+ // Each node keeps the best (score, hops) it is reached with.+ const best = new Map<number, { score: number; hops: number }>();+ for (const [seedId, sScore] of seeds) {+ best.set(seedId, { score: sScore, hops: 0 });+ }+ for (const [seedId, sScore] of seeds) {+ let frontier: number[] = [seedId];+ const seen = new Set<number>([seedId]);+ for (let hop = 1; hop <= maxHops; hop++) {+ const next: number[] = [];+ const decayed = sScore * hopDecay ** hop;+ for (const node of frontier) {+ for (const nb of snapshot.adjacency.get(node) ?? []) {+ if (seen.has(nb)) continue;+ seen.add(nb);+ next.push(nb);+ const prev = best.get(nb);+ if (!prev || decayed > prev.score) best.set(nb, { score: decayed, hops: hop });+ else if (decayed > 0 && hop < prev.hops) best.set(nb, { score: prev.score, hops: hop });+ }+ }+ frontier = next;+ if (frontier.length === 0) break;+ }+ }++ const seedIdSet = new Set(seeds.map(([id]) => id));+ const ranked = [...best.entries()]+ .map(([id, v]) => ({ id, ...v }))+ .toSorted((a, b) =>+ a.score !== b.score+ ? b.score - a.score+ : (titles.get(a.id)?.path ?? "") < (titles.get(b.id)?.path ?? "")+ ? -1+ : 1,+ )+ .slice(0, shortlistLimit);++ const shortlist: ShouldReadEntry[] = ranked.map((r) => {+ const meta = titles.get(r.id);+ const reasons = [...(seedReasons.get(r.id) ?? [])];+ if (!seedIdSet.has(r.id) || r.hops > 0) reasons.push(`bfs:${r.hops}`);+ return Object.freeze({+ documentId: r.id,+ path: meta?.path ?? String(r.id),+ title: meta?.title ?? null,+ degree: snapshot.degree.get(r.id) ?? 0,+ hops: r.hops,+ score: r.score,+ summary: indexOnly ? null : (summaries.get(r.id) ?? null),+ reasons: Object.freeze(reasons),+ });+ });++ return Object.freeze({+ mode: indexOnly ? "index_only" : "should_read",+ shortlist: Object.freeze(shortlist),+ notesRead: indexOnly ? 0 : summaries.size,+ });+}++/**+ * Config-level wrapper: open a read store, plan the shortlist, close.+ * The CLI/MCP surface calls this; unit tests call {@link planReadShortlist}+ * directly against an open store.+ */+export async function planRead(+ config: ResolvedSearchConfig,+ query: string,+ opts: GraphPrepassOptions = {},+): Promise<GraphPrepassResult> {+ const store = await Store.open(config, { mode: "read", loadVec: false });+ try {+ return planReadShortlist(store, query, opts);+ } finally {+ await store.close();+ }+}diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex 912ec4d6..af3cfa24 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -95,6 +95,12 @@ export {type ApplyCrossEncoderRerankOptions,type RerankTelemetryEvent,} from "./rerank/index.ts";+export { LocalRerankProvider, scoreLocalRerank } from "./rerank/local.ts";+export {+ runRerankEvalGate,+ type RerankEvalGateOptions,+ type RerankEvalGateResult,+} from "./rerank-eval-gate.ts";export { resolveIndexPath } from "./paths.ts";export {@@ -106,6 +112,8 @@ export {type IndexProgressEvent,} from "./indexer.ts";export { search, expandHit, SEARCH_LIMIT_MIN, SEARCH_LIMIT_MAX } from "./search.ts";+export { planReadShortlist, planRead } from "./graph-prepass.ts";+export type { GraphPrepassOptions, GraphPrepassResult, ShouldReadEntry } from "./graph-prepass.ts";export {captureRecallFeedback,computeLearnedWeights,@@ -145,6 +153,9 @@ const DEFAULTS = {shutdownGraceSeconds: 5,resumeReindex: false,chainStopScore: 0.8,+ trigramPrefilterEnabled: false,+ trigramPrefilterMinChunks: 5000,+ trigramPrefilterMaxSelectivity: 0.5,rerankTopK: 20,rerankMinScore: 0,};@@ -286,14 +297,21 @@ function validateResolvedConfig(config: ResolvedSearchConfig): void {function parseProvider(raw: string | null): ResolvedEmbeddingConfig["provider"] {if (raw === null) return DEFAULTS.provider;- if (raw === "openai-compat" || raw === "disabled" || raw === "local") return raw;+ if (raw === "openai-compat" || raw === "disabled" || raw === "local" || raw === "zeroentropy") {+ return raw;+ }throw new SearchError("INVALID_INPUT",- `embedding_provider must be 'openai-compat', 'local', 'disabled', or a registered provider name, got '${raw}'`,+ `embedding_provider must be 'openai-compat', 'zeroentropy', 'local', 'disabled', or a registered provider name, got '${raw}'`,);}-const BUILTIN_PROVIDERS: ReadonlySet<string> = new Set(["openai-compat", "disabled", "local"]);+const BUILTIN_PROVIDERS: ReadonlySet<string> = new Set([+ "openai-compat",+ "disabled",+ "local",+ "zeroentropy",+]);/*** Resolve a non-built-in `embedding_provider` name against the registry.@@ -413,6 +431,11 @@ export function resolveSearchConfig(opts: {const baseUrl = explicitBaseUrl ?? registryExpansion?.baseUrl ?? null;const model = explicitModel ?? registryExpansion?.model ?? null;const apiKey = explicitApiKey ?? registryExpansion?.apiKey ?? null;+ // Multi-key failover list: an explicit key is single-valued; otherwise+ // use the registry profile's ordered probe list. Empty when no key.+ const apiKeys: ReadonlyArray<string> = explicitApiKey+ ? [explicitApiKey]+ : (registryExpansion?.apiKeys ?? (apiKey ? [apiKey] : []));const dimRaw = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_DIM", "embedding_dimension");const dimension =dimRaw === null ? null : parseInteger(dimRaw, 0, "embedding_dimension", { min: 1 });@@ -446,6 +469,7 @@ export function resolveSearchConfig(opts: {baseUrl,model,apiKey,+ apiKeys: Object.freeze(apiKeys),dimension,timeoutMs,concurrency,@@ -506,8 +530,22 @@ export function resolveSearchConfig(opts: {DEFAULTS.rerankMinScore,"search_rerank_min_score",);+ const rerankKindRaw = envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_KIND",+ "search_rerank_kind",+ );+ if (rerankKindRaw !== null && rerankKindRaw !== "openai-compat" && rerankKindRaw !== "local") {+ throw new SearchError(+ "INVALID_INPUT",+ `search_rerank_kind must be 'openai-compat' or 'local', got '${rerankKindRaw}'`,+ );+ }+ const rerankKind = rerankKindRaw === "local" ? "local" : "openai-compat";const rerank: ResolvedRerankConfig = Object.freeze({enabled: rerankEnabled,+ kind: rerankKind,baseUrl: rerankBaseUrl,model: rerankModel,envKey: rerankEnvKey,@@ -651,6 +689,37 @@ export function resolveSearchConfig(opts: {DEFAULTS.chainStopScore,"search_chain_stop_score",);+ const trigramPrefilterEnabled = parseBool(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_TRIGRAM_PREFILTER",+ "search_trigram_prefilter_enabled",+ ),+ DEFAULTS.trigramPrefilterEnabled,+ "search_trigram_prefilter_enabled",+ );+ const trigramPrefilterMinChunks = parseInteger(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_TRIGRAM_MIN_CHUNKS",+ "search_trigram_prefilter_min_chunks",+ ),+ DEFAULTS.trigramPrefilterMinChunks,+ "search_trigram_prefilter_min_chunks",+ { min: 0 },+ );+ const trigramPrefilterMaxSelectivity = parseFloat01(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_TRIGRAM_MAX_SELECTIVITY",+ "search_trigram_prefilter_max_selectivity",+ ),+ DEFAULTS.trigramPrefilterMaxSelectivity,+ "search_trigram_prefilter_max_selectivity",+ );const shutdownGraceSeconds = parseInteger(envOrConfig(env,@@ -688,6 +757,9 @@ export function resolveSearchConfig(opts: {selfTuningEnabled,chainStopEnabled,chainStopScore,+ trigramPrefilterEnabled,+ trigramPrefilterMinChunks,+ trigramPrefilterMaxSelectivity,});const base: ResolvedSearchConfig = Object.freeze({diff --git a/src/core/search/ranker.ts b/src/core/search/ranker.tsindex a20ed07f..d2cd4e5b 100644--- a/src/core/search/ranker.ts+++ b/src/core/search/ranker.ts@@ -64,6 +64,14 @@ export interface RankerInputs {* entries (and the absent map) stay neutral.*/readonly trendByDoc?: ReadonlyMap<number, string>;+ /**+ * Optional observed-reuse score per chunk in [0, 1] (t_65588d8b): the+ * folded USED-vs-CONTRADICTED rate of the chunk's document, the preferred+ * outcome signal over predicted importance. Missing entries (and the+ * absent map) contribute zero boost, so a vault with no observed-use+ * verdicts ranks bit-identically.+ */+ readonly reuseRateByChunk?: ReadonlyMap<number, number>;}/** Freshness-trend multipliers on the relevance portion. */@@ -168,6 +176,7 @@ function buildReasons(parts: {entityBoost?: number;activationBoost?: number;coAccessBoost?: number;+ reuseBoost?: number;trend?: string;trendMul?: number;sessionFocus?: number;@@ -186,6 +195,9 @@ function buildReasons(parts: {if (parts.coAccessBoost && parts.coAccessBoost > 0) {reasons.push(`co_access: ${fmt(parts.coAccessBoost)}`);}+ if (parts.reuseBoost && parts.reuseBoost > 0) {+ reasons.push(`observed_reuse: ${fmt(parts.reuseBoost)}`);+ }if (parts.trend !== undefined && parts.trendMul !== undefined && parts.trendMul !== 1) {reasons.push(`freshness_trend: ${parts.trend} x${fmt(parts.trendMul)}`);}@@ -214,6 +226,7 @@ function buildBreakdown(parts: {entityBoost?: number;activationBoost?: number;coAccessBoost?: number;+ reuseBoost?: number;trendMul?: number;sessionFocus?: number;rrf?: number;@@ -225,6 +238,7 @@ function buildBreakdown(parts: {entity: parts.entityBoost ?? 0,activation: parts.activationBoost ?? 0,coAccess: parts.coAccessBoost ?? 0,+ reuse: parts.reuseBoost ?? 0,link: parts.linkBoost,recency: parts.recency,tier: parts.tierMul,@@ -396,6 +410,14 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea}}const coAccessBoost = Math.min(0.03, coAccessRaw);+ // Observed-reuse boost (t_65588d8b): the folded USED-vs-CONTRADICTED+ // rate of the chunk's document. Capped at 0.06 - larger than the+ // activation / co-access caps so a memory the agent demonstrably reused+ // outranks one merely predicted-important, yet still a bounded re-ranker+ // that never floats an irrelevant chunk. Zero (byte-identical) when no+ // observed-use verdicts exist.+ const reuseRate = clamp01(inputs.reuseRateByChunk?.get(c.chunkId) ?? 0);+ const reuseBoost = Math.min(0.06, reuseRate * 0.06);const sessionFocus = scoreSessionFocusTarget(hyd, opts.sessionFocus, nowMs);const score = clamp01(weighted * tierMul * trendMul +@@ -404,6 +426,7 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSeaentityBoost +activationBoost +coAccessBoost ++ reuseBoost +sessionFocus,);@@ -423,6 +446,7 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSearecencyBoost: recency,searchType: c.searchType,reasons: buildReasons({+ reuseBoost,keywordScore: c.keywordScore,semanticScore: semanticEnabled ? c.semanticScore : 0,linkBoost,@@ -436,6 +460,7 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSearrf: rrfByChunk !== null ? rrf : 0,}),breakdown: buildBreakdown({+ reuseBoost,keywordScore: c.keywordScore,semanticScore: semanticEnabled ? c.semanticScore : 0,linkBoost,diff --git a/src/core/search/rerank-eval-gate.ts b/src/core/search/rerank-eval-gate.tsnew file mode 100644index 00000000..e67dabf5--- /dev/null+++ b/src/core/search/rerank-eval-gate.ts@@ -0,0 +1,86 @@+/**+ * Per-store reranker evaluation gate (Retrieval & Ranking Quality,+ * t_9f95ebb6).+ *+ * Before a store commits to reranking, this gate MEASURES whether the+ * reranker actually improves recall/ranking quality on that store's own+ * labelled queries. It runs the existing recall-benchmark twice - rerank+ * OFF then ON - over the same dataset and compares the metric families+ * (hit@k, MRR). Reranking is recommended only when it lifts ranking without+ * regressing hit@k, so a store where the reranker does not help never+ * silently enables one that hurts.+ *+ * Reuses `runRecallBenchmark` for the metrics rather than a bespoke scorer,+ * and is deterministic and offline when the store and reranker are (the+ * bundled "local" reranker always is).+ */++import { runRecallBenchmark } from "./benchmark.ts";+import type { RecallBenchmarkDataset, RecallBenchmarkReport } from "./benchmark.ts";+import type { ResolvedRerankConfig, ResolvedSearchConfig } from "./types.ts";++export interface RerankEvalGateOptions {+ /** Rank depth for the benchmark (defaults to the benchmark default). */+ readonly k?: number;+ /** Reranker kind to evaluate (defaults to "local", the offline reranker). */+ readonly kind?: ResolvedRerankConfig["kind"];+ /** Minimum MRR lift required to recommend enabling. Default 0.01. */+ readonly minMrrDelta?: number;+ /** Minimum hit@k lift that alone justifies enabling. Default 0.01. */+ readonly minHitDelta?: number;+}++export interface RerankEvalGateResult {+ /** Whether reranking is recommended for this store. */+ readonly improves: boolean;+ readonly recommendation: "enable" | "keep-disabled";+ readonly baseline: RecallBenchmarkReport;+ readonly reranked: RecallBenchmarkReport;+ readonly deltas: { readonly hitAtK: number; readonly mrr: number };+}++function withRerank(+ config: ResolvedSearchConfig,+ enabled: boolean,+ kind: ResolvedRerankConfig["kind"],+): ResolvedSearchConfig {+ return Object.freeze({+ ...config,+ rerank: Object.freeze({ ...config.rerank, enabled, kind }),+ });+}++/**+ * Evaluate whether reranking helps this store. Runs the benchmark with+ * rerank off and on and compares. Recommends enabling only when MRR lifts+ * by at least `minMrrDelta` (or hit@k by `minHitDelta`) AND hit@k does not+ * regress - a strictly safe promotion rule.+ */+export async function runRerankEvalGate(+ config: ResolvedSearchConfig,+ dataset: RecallBenchmarkDataset,+ opts: RerankEvalGateOptions = {},+): Promise<RerankEvalGateResult> {+ const kind = opts.kind ?? "local";+ const minMrrDelta = opts.minMrrDelta ?? 0.01;+ const minHitDelta = opts.minHitDelta ?? 0.01;+ const k = opts.k;++ const baseline = await runRecallBenchmark(withRerank(config, false, kind), dataset, { k });+ const reranked = await runRecallBenchmark(withRerank(config, true, kind), dataset, { k });++ const hitDelta = reranked.hitAtK - baseline.hitAtK;+ const mrrDelta = reranked.mrr - baseline.mrr;++ // Never recommend a reranker that drops hit@k; among non-regressing+ // candidates, require a real MRR or hit@k lift.+ const improves = hitDelta >= 0 && (mrrDelta >= minMrrDelta || hitDelta >= minHitDelta);++ return Object.freeze({+ improves,+ recommendation: improves ? "enable" : "keep-disabled",+ baseline,+ reranked,+ deltas: { hitAtK: hitDelta, mrr: mrrDelta },+ });+}diff --git a/src/core/search/rerank/index.ts b/src/core/search/rerank/index.tsindex 6bffcd3e..fc12b2c7 100644--- a/src/core/search/rerank/index.ts+++ b/src/core/search/rerank/index.ts@@ -104,23 +104,35 @@ export async function applyCrossEncoderRerank(// Invariant 1: disabled -> zero-cost no-op, byte-identical input.if (!config.enabled) return results;- // Invariant 2: fail closed when enabled-but-unconfigured. Resolution- // throws a typed SearchError; we deliberately do NOT catch it - a- // misconfiguration is an operator error, not a hot-path degrade.- const endpoint = resolveOpenAiCompatEndpoint(- {- enabled: true,- baseUrl: config.baseUrl,- model: config.model,- envKey: config.envKey,- apiKey: config.apiKey,- env: opts.env,- },- "search_rerank",- );- // resolveOpenAiCompatEndpoint only returns null when `enabled` is false,- // which we ruled out above; the narrowing keeps TypeScript honest.- if (endpoint === null) return results;+ // Resolve the provider. The bundled offline reranker ("local") needs no+ // endpoint - it is deterministic and network-free. The remote+ // "openai-compat" path resolves fail-closed (Invariant 2): a misconfigured+ // endpoint throws a typed SearchError (an operator error, not a hot-path+ // degrade). An injected provider (tests) bypasses both.+ let provider = opts.provider;+ if (!provider) {+ if (config.kind === "local") {+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const { LocalRerankProvider } = require("./local.ts") as typeof import("./local.ts");+ provider = new LocalRerankProvider();+ } else {+ const endpoint = resolveOpenAiCompatEndpoint(+ {+ enabled: true,+ baseUrl: config.baseUrl,+ model: config.model,+ envKey: config.envKey,+ apiKey: config.apiKey,+ env: opts.env,+ },+ "search_rerank",+ );+ // resolveOpenAiCompatEndpoint only returns null when `enabled` is+ // false, which we ruled out above; the narrowing keeps TS honest.+ if (endpoint === null) return results;+ provider = makeRerankProvider(endpoint, { timeoutMs: opts.timeoutMs });+ }+ }if (results.length === 0) return results;@@ -129,8 +141,6 @@ export async function applyCrossEncoderRerank(const tail = results.slice(topK);const documents = head.map((r) => r.content);- const provider = opts.provider ?? makeRerankProvider(endpoint, { timeoutMs: opts.timeoutMs });-let scores: number[];try {scores = await provider.rerank(query, documents);diff --git a/src/core/search/rerank/local.ts b/src/core/search/rerank/local.tsnew file mode 100644index 00000000..fd273f8e--- /dev/null+++ b/src/core/search/rerank/local.ts@@ -0,0 +1,83 @@+/**+ * Bundled offline reranker (Retrieval & Ranking Quality, t_9f95ebb6).+ *+ * A deterministic, dependency-free cross-encoder-SHAPED reranker: it jointly+ * scores a query against each candidate document and returns one relevance+ * score per document, aligned to input order - the same contract the remote+ * cross-encoder provider implements, but computed locally with no model+ * runtime and no network.+ *+ * Following the OSB idiom (the local embedder is pure feature-hashing, not a+ * bundled neural model), the score is a blend of lexical/structural+ * query-document features rather than a learned transformer:+ * - coverage - fraction of distinct query terms present in the document,+ * - proximity - how tightly the matched query terms cluster,+ * - density - bounded query-term frequency within the document.+ * Coverage dominates, so a document that answers more of the query ranks+ * above one that merely repeats a single term. Deterministic: identical+ * (query, documents) always yields identical scores.+ */++import type { RerankProvider } from "./provider.ts";++const TOKEN_RE = /[\p{L}\p{N}]{2,}/gu;++function tokenize(text: string): string[] {+ const out: string[] = [];+ for (const m of text.toLowerCase().matchAll(TOKEN_RE)) out.push(m[0]);+ return out;+}++function clamp01(x: number): number {+ return x < 0 ? 0 : x > 1 ? 1 : x;+}++/** Blend weights (sum to 1). Coverage-dominant by design. */+const W_COVERAGE = 0.6;+const W_PROXIMITY = 0.25;+const W_DENSITY = 0.15;++export function scoreLocalRerank(query: string, document: string): number {+ const queryTerms = [...new Set(tokenize(query))];+ if (queryTerms.length === 0) return 0;++ const docTokens = tokenize(document);+ if (docTokens.length === 0) return 0;++ const counts = new Map<string, number>();+ const firstPos = new Map<string, number>();+ for (let i = 0; i < docTokens.length; i++) {+ const t = docTokens[i]!;+ counts.set(t, (counts.get(t) ?? 0) + 1);+ if (!firstPos.has(t)) firstPos.set(t, i);+ }++ const present = queryTerms.filter((t) => counts.has(t));+ if (present.length === 0) return 0;++ const coverage = present.length / queryTerms.length;++ let densityAcc = 0;+ for (const t of queryTerms) densityAcc += Math.min(counts.get(t) ?? 0, 3);+ const density = densityAcc / (3 * queryTerms.length);++ let proximity = coverage;+ if (present.length > 1) {+ const positions = present.map((t) => firstPos.get(t)!);+ const span = Math.max(...positions) - Math.min(...positions);+ // present terms in the smallest span -> proximity 1; spread out -> lower.+ proximity = clamp01(present.length / (span + 1));+ }++ return clamp01(W_COVERAGE * coverage + W_PROXIMITY * proximity + W_DENSITY * density);+}++/** The bundled offline reranker. Stateless, deterministic, network-free. */+export class LocalRerankProvider implements RerankProvider {+ readonly name = "local";+ readonly model = "local-lexical-v1";++ rerank(query: string, documents: ReadonlyArray<string>): Promise<number[]> {+ return Promise.resolve(documents.map((doc) => scoreLocalRerank(query, doc)));+ }+}diff --git a/src/core/search/schema.ts b/src/core/search/schema.tsindex b55c268c..29deaad5 100644--- a/src/core/search/schema.ts+++ b/src/core/search/schema.ts@@ -16,7 +16,7 @@ import { SearchError } from "./types.ts";* this raises `SCHEMA_MISMATCH` on open β the operator must reindex* with a newer binary.*/-export const LATEST_SCHEMA_VERSION = 8;+export const LATEST_SCHEMA_VERSION = 9;/*** The wikilink-resolution basename of a stored document path: the final@@ -367,6 +367,48 @@ export const MIGRATIONS: ReadonlyArray<Migration> = Object.freeze([for (const row of rows) update.run(documentBasename(row.path), row.id);},},+ {+ // v9 (Retrieval & Ranking Quality) - opt-in trigram candidate index.+ // A second FTS5 shadow over the same `fts_content` column, tokenized+ // with the built-in `trigram` tokenizer, so a query can gather+ // substring / partial-token candidates that the `unicode61` word+ // tokenizer misses. Read only when `search_trigram_prefilter_enabled`+ // is on; otherwise it is inert extra index data and result ordering is+ // byte-identical. Additive and reindex-safe: `rebuild` populates it+ // from existing chunks in-place, and dedicated triggers keep it in+ // sync. No new dependency - the trigram tokenizer ships with SQLite.+ version: 9,+ up(db) {+ db.exec(`+ DROP TRIGGER IF EXISTS chunks_tri_ai;+ DROP TRIGGER IF EXISTS chunks_tri_ad;+ DROP TRIGGER IF EXISTS chunks_tri_au;+ DROP TABLE IF EXISTS chunk_trigram;++ CREATE VIRTUAL TABLE chunk_trigram USING fts5(+ fts_content,+ content='chunks',+ content_rowid='id',+ tokenize='trigram'+ );++ CREATE TRIGGER chunks_tri_ai AFTER INSERT ON chunks BEGIN+ INSERT INTO chunk_trigram(rowid, fts_content) VALUES (new.id, new.fts_content);+ END;+ CREATE TRIGGER chunks_tri_ad AFTER DELETE ON chunks BEGIN+ INSERT INTO chunk_trigram(chunk_trigram, rowid, fts_content)+ VALUES('delete', old.id, old.fts_content);+ END;+ CREATE TRIGGER chunks_tri_au AFTER UPDATE ON chunks BEGIN+ INSERT INTO chunk_trigram(chunk_trigram, rowid, fts_content)+ VALUES('delete', old.id, old.fts_content);+ INSERT INTO chunk_trigram(rowid, fts_content) VALUES (new.id, new.fts_content);+ END;++ INSERT INTO chunk_trigram(chunk_trigram) VALUES('rebuild');+ `);+ },+ },]);/** Read the current schema version from `index_state`. Returns 0 if not yet recorded. */diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex 6a7e4d90..2d02ebc7 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -17,6 +17,7 @@ import type { FrontmatterMap } from "../types.ts";import { isVisible, normalizeVisibilityScope, pageVisibility } from "../graph/visibility.ts";import { isOwnerVisible, normalizeAgentScope, pageOwner } from "../graph/agent-scope.ts";import { makeProvider } from "./embeddings/provider.ts";+import { isLowSelectivity, planTrigramPrefilter } from "./trigram-prefilter.ts";import {composeWeightProfiles,fnv1aHex,@@ -26,6 +27,7 @@ import {readLearnedWeights,} from "./feedback.ts";import { parseFreshnessTrend } from "../brain/temporal/freshness-trend.ts";+import { observedReuseRates } from "../brain/observed-use.ts";import { effectiveActivation, halfLifeDays, resolveActivationKind } from "./activation/decay.ts";import {ACCESS_EVENT_PATHS_CAP,@@ -133,6 +135,11 @@ function configFingerprint(config: ResolvedSearchConfig): string {rrkModel: config.rerank.model,rrkTopK: config.rerank.topK,rrkMin: config.rerank.minScore,+ // Trigram prefilter augments the candidate pool, so a toggle or+ // selectivity change must invalidate cached rows.+ tri: r.trigramPrefilterEnabled,+ triMin: r.trigramPrefilterMinChunks,+ triSel: r.trigramPrefilterMaxSelectivity,});}@@ -443,6 +450,38 @@ export async function search(for (const w of kwOutcome.warnings) warnings.push(w);}}+ // Trigram candidate source (t_4a672b84): opt-in. On large vaults, merge+ // substring / partial-token matches the word tokenizer missed into the+ // keyword candidate pool. A strict superset of substring matches, so it+ // only ADDS candidates (deduped by chunkId; existing keyword hits keep+ // their bm25). Skipped for short/CJK/low-selectivity queries and when+ // disabled - leaving kwHits byte-identical.+ if (+ config.recall.trigramPrefilterEnabled &&+ store.chunkCount() >= config.recall.trigramPrefilterMinChunks+ ) {+ const trigramPlan = planTrigramPrefilter(query);+ if (trigramPlan.mode === "match") {+ const corpus = store.chunkCount();+ const cand = store.trigramCandidates(trigramPlan.ftsQuery, {+ limit: limit * config.recall.poolMultiplier,+ pathPrefix,+ });+ if (+ cand.length > 0 &&+ !isLowSelectivity(cand.length, corpus, config.recall.trigramPrefilterMaxSelectivity)+ ) {+ const seen = new Set(kwHits.map((h) => h.chunkId));+ for (const h of cand) {+ if (!seen.has(h.chunkId)) {+ kwHits.push(h);+ seen.add(h.chunkId);+ }+ }+ }+ }+ }+const intentProfile = config.recall.intentEnabled ? plan.weightProfile : undefined;// Learned recall weights (recall-trust-suite): opt-in multipliers// derived from explicit feedback compose with the intent profile.@@ -776,6 +815,25 @@ export async function search(if (byDoc.size > 0) trendByDoc = byDoc;}+ // Observed-reuse boost (t_65588d8b): fold the session-end USED/IGNORED/+ // CONTRADICTED verdicts into a per-document reuse score and map it onto+ // each candidate chunk. Keyed by path (else id) to match how verdicts+ // are recorded. Empty (byte-identical) when no verdicts exist.+ let reuseRateByChunk: ReadonlyMap<number, number> | undefined;+ {+ const reuse = observedReuseRates(config.vault);+ if (reuse.size > 0) {+ const byChunk = new Map<number, number>();+ for (const chunkId of idsList) {+ const h = hydrated.get(chunkId);+ if (h === undefined) continue;+ const entry = reuse.get(h.path) ?? reuse.get(`${h.documentId}:${chunkId}`);+ if (entry !== undefined && entry.score > 0) byChunk.set(chunkId, entry.score);+ }+ if (byChunk.size > 0) reuseRateByChunk = byChunk;+ }+ }+// When a property filter is active, overfetch the ranked// candidates so the post-filter result set still has a chance// of producing `limit` matching rows. Without this, the@@ -849,6 +907,7 @@ export async function search(...(activationByChunk !== undefined ? { activationByChunk } : {}),...(coAccessByChunk !== undefined ? { coAccessByChunk } : {}),...(trendByDoc !== undefined ? { trendByDoc } : {}),+ ...(reuseRateByChunk !== undefined ? { reuseRateByChunk } : {}),},{keywordWeight: opts.keywordWeight ?? config.keywordWeight,diff --git a/src/core/search/store.ts b/src/core/search/store.tsindex 649d1af8..6a0d1964 100644--- a/src/core/search/store.ts+++ b/src/core/search/store.ts@@ -1586,6 +1586,57 @@ export class Store {}));}+ /**+ * Trigram candidate lookup over the `chunk_trigram` FTS5 shadow (v9).+ * Returns bm25-ordered keyword hits whose content matches the trigram+ * query - a strict superset of exact substring matches for the query's+ * terms, used as an opt-in candidate source that broadens large-vault+ * keyword recall (substring / partial-token matches the word tokenizer+ * misses). Fails soft to an empty list if the trigram table is absent+ * (a migrated-but-not-reindexed index always has it via the v9 rebuild).+ */+ trigramCandidates(+ trigramQuery: string,+ opts: { readonly limit: number; readonly pathPrefix?: string | null },+ ): KeywordHit[] {+ const limit = Math.max(1, opts.limit | 0);+ const prefix = opts.pathPrefix && opts.pathPrefix.length > 0 ? opts.pathPrefix : null;+ try {+ if (prefix) {+ const rows = this.db+ .query<+ { chunk_id: number; document_id: number; bm25: number },+ [string, string, string, number]+ >(+ "SELECT c.id AS chunk_id, c.document_id AS document_id, bm25(chunk_trigram) AS bm25 " ++ "FROM chunk_trigram " ++ "JOIN chunks c ON c.id = chunk_trigram.rowid " ++ "JOIN documents d ON d.id = c.document_id " ++ "WHERE chunk_trigram MATCH ? AND substr(d.path, 1, length(?)) = ? " ++ "ORDER BY bm25 ASC LIMIT ?",+ )+ .all(trigramQuery, prefix, prefix, limit);+ return rows.map((r) => ({ chunkId: r.chunk_id, documentId: r.document_id, bm25: r.bm25 }));+ }+ const rows = this.db+ .query<{ chunk_id: number; document_id: number; bm25: number }, [string, number]>(+ "SELECT c.id AS chunk_id, c.document_id AS document_id, bm25(chunk_trigram) AS bm25 " ++ "FROM chunk_trigram JOIN chunks c ON c.id = chunk_trigram.rowid " ++ "WHERE chunk_trigram MATCH ? ORDER BY bm25 ASC LIMIT ?",+ )+ .all(trigramQuery, limit);+ return rows.map((r) => ({ chunkId: r.chunk_id, documentId: r.document_id, bm25: r.bm25 }));+ } catch {+ return [];+ }+ }++ /** Total indexed chunk count - used to judge trigram-prefilter selectivity. */+ chunkCount(): number {+ const row = this.db.query<{ n: number }, []>("SELECT COUNT(*) AS n FROM chunks").get();+ return row?.n ?? 0;+ }+semanticTopK(queryVector: ReadonlyArray<number> | Float32Array,opts: { readonly limit: number; readonly pathPrefix?: string | null },@@ -1744,6 +1795,23 @@ export class Store {* The traversal layer surfaces this when a linked document is not* already a relevance hit.*/+ /**+ * Document id -> { path, title } for every indexed document, reading+ * ONLY the `documents` table (never chunk bodies). The graph query+ * pre-pass uses this for its index-only short-circuit so it can rank and+ * answer from index metadata with zero note bodies hydrated.+ */+ documentTitles(): Map<number, { readonly path: string; readonly title: string | null }> {+ const out = new Map<number, { path: string; title: string | null }>();+ const rows = this.db+ .query<{ id: number; path: string; title: string | null }, []>(+ "SELECT id, path, title FROM documents",+ )+ .all();+ for (const r of rows) out.set(r.id, { path: r.path, title: r.title });+ return out;+ }+representativeChunks(documentIds: ReadonlyArray<number>): Map<number, HydratedChunk> {const out = new Map<number, HydratedChunk>();if (documentIds.length === 0) return out;diff --git a/src/core/search/trigram-prefilter.ts b/src/core/search/trigram-prefilter.tsnew file mode 100644index 00000000..33948c27--- /dev/null+++ b/src/core/search/trigram-prefilter.ts@@ -0,0 +1,85 @@+/**+ * Trigram candidate prefilter (Retrieval & Ranking Quality, t_4a672b84).+ *+ * A deterministic planner over the `chunk_trigram` FTS5 shadow (schema v9).+ * Given a query it decides whether to gather trigram candidates and, if so,+ * builds the FTS5 MATCH string. The trigram source is a STRICT SUPERSET of+ * exact substring matches for the query's terms, so unioning it into the+ * candidate pool can only add candidates the word-tokenized keyword lane+ * missed (substring / partial-token matches) - it never drops a result.+ *+ * It falls back to the normal path (no trigram source) for:+ * - short queries (no term of at least 3 characters - the trigram+ * tokenizer's minimum token width),+ * - CJK queries (already covered by the v5 CJK fts_content+ * expansion; trigram shingling over ideographs is+ * redundant and low-precision here),+ * - low-selectivity (the term set matches too large a fraction of the+ * corpus to be worth widening - judged post-query).+ *+ * Pure and dependency-free: the SQLite `trigram` tokenizer does the+ * indexing; this module only plans queries and judges selectivity.+ */++/** Minimum term length the trigram tokenizer can match. */+export const TRIGRAM_MIN_TERM_LEN = 3;++const CJK_RE = /[γ-γΏγ-γγ -γΏγ-δΆΏδΈ-ιΏΏο€-﫿ヲ-οΎκ°-ν―]/;++/** True when the text contains any CJK (Han/Kana/Hangul) codepoint. */+export function containsCjk(text: string): boolean {+ return CJK_RE.test(text);+}++/**+ * Split a query into candidate trigram terms: alphanumeric runs of at+ * least {@link TRIGRAM_MIN_TERM_LEN} characters, lower-cased. Terms shorter+ * than the trigram width cannot be matched and are dropped.+ */+export function extractTrigramTerms(query: string): string[] {+ const out: string[] = [];+ for (const raw of query.toLowerCase().split(/[^\p{L}\p{N}]+/u)) {+ if (raw.length >= TRIGRAM_MIN_TERM_LEN) out.push(raw);+ }+ return out;+}++/** Escape a term for use as an FTS5 quoted string (double any `"`). */+function escapeFts5(term: string): string {+ return `"${term.replace(/"/g, '""')}"`;+}++/** Outcome of planning a trigram prefilter for a query. */+export type TrigramPlan =+ | { readonly mode: "skip"; readonly reason: "short" | "cjk" }+ | { readonly mode: "match"; readonly ftsQuery: string; readonly terms: ReadonlyArray<string> };++/**+ * Plan the trigram prefilter for a query. Returns a `skip` decision+ * (with the reason) when the query does not qualify, or a `match` with the+ * conjunctive FTS5 query over the qualifying terms. Selectivity is judged+ * separately (post-query) via {@link isLowSelectivity}.+ */+export function planTrigramPrefilter(query: string): TrigramPlan {+ if (containsCjk(query)) return { mode: "skip", reason: "cjk" };+ const terms = extractTrigramTerms(query);+ if (terms.length === 0) return { mode: "skip", reason: "short" };+ // Conjunctive AND: a chunk must contain every term's trigrams. This is a+ // superset of "contains every term as a substring".+ const ftsQuery = terms.map(escapeFts5).join(" AND ");+ return { mode: "match", ftsQuery, terms };+}++/**+ * A trigram candidate set is "low-selectivity" (not worth widening the+ * pool with) when it covers more than `maxSelectivity` of the corpus.+ * Returns false for an empty corpus.+ */+export function isLowSelectivity(+ candidateCount: number,+ corpusChunkCount: number,+ maxSelectivity: number,+): boolean {+ if (corpusChunkCount <= 0) return false;+ return candidateCount / corpusChunkCount > maxSelectivity;+}diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex 4fa65815..d29870ab 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -56,6 +56,8 @@ export interface ScoreBreakdown {readonly entity: number;readonly activation: number;readonly coAccess: number;+ /** Observed-reuse boost (t_65588d8b); 0 when no verdicts apply. */+ readonly reuse: number;readonly link: number;readonly recency: number;readonly tier: number;@@ -570,10 +572,17 @@ export interface SearchOutcome {export interface ResolvedEmbeddingConfig {readonly enabled: boolean;- readonly provider: "openai-compat" | "disabled" | "local";+ readonly provider: "openai-compat" | "disabled" | "local" | "zeroentropy";readonly baseUrl: string | null;readonly model: string | null;readonly apiKey: string | null;+ /**+ * Ordered API-key failover list (multi-key fallback). When present and+ * non-empty, the provider starts on the first key and, on an auth error+ * (HTTP 401/403), fails over to the next, pinning the first that works.+ * Absent/empty means single-key behaviour over `apiKey` (byte-identical).+ */+ readonly apiKeys?: ReadonlyArray<string>;readonly dimension: number | null;readonly timeoutMs: number;readonly concurrency: number;@@ -599,6 +608,13 @@ export interface ResolvedEmbeddingConfig {*/export interface ResolvedRerankConfig {readonly enabled: boolean;+ /**+ * Reranker backend (Retrieval & Ranking Quality, t_9f95ebb6).+ * "openai-compat" (default) resolves a remote `/rerank` endpoint;+ * "local" uses the bundled offline deterministic reranker, which needs+ * no base_url / model / key and never touches the network.+ */+ readonly kind: "openai-compat" | "local";/** OpenAI-compatible base URL (trailing slashes stripped) or null. */readonly baseUrl: string | null;readonly model: string | null;@@ -732,6 +748,22 @@ export interface ResolvedRecallConfig {readonly chainStopEnabled: boolean;/** Normalized-score threshold in [0, 1] that triggers the chain-stop. */readonly chainStopScore: number;+ /**+ * Trigram candidate prefilter (Retrieval & Ranking Quality, t_4a672b84).+ * Off by default: when true and a query qualifies (a term of at least 3+ * chars, non-CJK), the trigram FTS5 shadow contributes an additional+ * candidate source that broadens large-vault keyword recall with+ * substring / partial-token matches (a strict superset of substring+ * matches - it never drops a result). Disabled -> byte-identical.+ */+ readonly trigramPrefilterEnabled: boolean;+ /** Minimum corpus chunk count before the trigram prefilter engages. */+ readonly trigramPrefilterMinChunks: number;+ /**+ * Skip the trigram source when its candidate set exceeds this fraction+ * of the corpus (low selectivity - not worth widening the pool). [0, 1].+ */+ readonly trigramPrefilterMaxSelectivity: number;}export interface ResolvedSearchConfig {diff --git a/src/mcp/brain/feedback-tools.ts b/src/mcp/brain/feedback-tools.tsindex 40936071..6b7942b8 100644--- a/src/mcp/brain/feedback-tools.ts+++ b/src/mcp/brain/feedback-tools.ts@@ -41,6 +41,11 @@ import { appendBrainNote } from "../../core/brain/note.ts";import { mirrorSignal, resolveSharedNamespace } from "../../core/brain/shared-namespace.ts";import { INTERNAL_ERROR, INVALID_PARAMS, MCPError } from "../protocol.ts";import type { ServerContext, ToolDefinition } from "../tools.ts";+import {+ emitObservedUse,+ isObservedUseVerdict,+ type ObservedUseEntry,+} from "../../core/brain/observed-use.ts";import { coerceStr, coerceBool, coerceIsoDate } from "../coerce.ts";import { vaultRelativeSafe } from "./shared.ts";@@ -432,6 +437,56 @@ async function toolBrainApplyEvidence(}}+// ----- brain_observed_use (t_65588d8b) -------------------------------------++/**+ * Record session-end observed-use verdicts (USED / IGNORED / CONTRADICTED)+ * per injected memory, mirroring `brain_apply_evidence`: the host supplies+ * already-structured verdicts (no LLM in the kernel), which are folded into+ * the observed-reuse ranking signal. The kernel only stores and aggregates.+ */+async function toolBrainObservedUse(+ ctx: ServerContext,+ args: Record<string, unknown>,+): Promise<Record<string, unknown>> {+ const rawEntries = args["entries"];+ if (!Array.isArray(rawEntries) || rawEntries.length === 0) {+ throw new MCPError(INVALID_PARAMS, "argument 'entries' must be a non-empty array");+ }+ const entries: ObservedUseEntry[] = rawEntries.map((raw, i) => {+ if (raw === null || typeof raw !== "object") {+ throw new MCPError(INVALID_PARAMS, `entries[${i}] must be an object`);+ }+ const e = raw as Record<string, unknown>;+ const id = typeof e["id"] === "string" ? e["id"].trim() : "";+ if (id === "") throw new MCPError(INVALID_PARAMS, `entries[${i}].id is required`);+ const verdict = e["verdict"];+ if (!isObservedUseVerdict(verdict)) {+ throw new MCPError(+ INVALID_PARAMS,+ `entries[${i}].verdict must be 'USED', 'IGNORED', or 'CONTRADICTED'`,+ );+ }+ const path = typeof e["path"] === "string" && e["path"] !== "" ? e["path"] : undefined;+ return { id, verdict, ...(path ? { path } : {}) };+ });++ const host = coerceStr(args, "host", false) ?? "mcp";+ const sessionId = coerceStr(args, "session_id", false) ?? undefined;+ const turnId = coerceStr(args, "turn_id", false) ?? undefined;+ const record = emitObservedUse(ctx.vault, {+ host,+ ...(sessionId ? { sessionId } : {}),+ ...(turnId ? { turnId } : {}),+ entries,+ });+ return {+ recorded: entries.length,+ record_id: record.id,+ created_at: record.createdAt,+ };+}+// ----- brain_note (Β§32B, v0.10.8) ------------------------------------------/**@@ -638,4 +693,38 @@ export const FEEDBACK_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([},handler: toolBrainNote,},+ {+ name: "brain_observed_use",+ description:+ "Record session-end observed-use verdicts (USED/IGNORED/CONTRADICTED) per injected memory. The host supplies structured verdicts; the kernel stores and aggregates them into the observed-reuse recall-ranking signal (no LLM).",+ inputSchema: {+ type: "object",+ properties: {+ entries: {+ type: "array",+ description: "One verdict per injected memory.",+ items: {+ type: "object",+ properties: {+ id: { type: "string", description: "Injected memory id (e.g. docId:chunkId)." },+ path: { type: "string", description: "Vault-relative path when known." },+ verdict: {+ type: "string",+ enum: ["USED", "IGNORED", "CONTRADICTED"],+ description: "Observed use of this memory in the session.",+ },+ },+ required: ["id", "verdict"],+ additionalProperties: false,+ },+ },+ host: { type: "string", description: "Host label; defaults to 'mcp'." },+ session_id: { type: "string", description: "Correlation id for the session." },+ turn_id: { type: "string", description: "Correlation id for the prompt-submit turn." },+ },+ required: ["entries"],+ additionalProperties: false,+ },+ handler: toolBrainObservedUse,+ },]);diff --git a/src/mcp/brain/procedure-tools.ts b/src/mcp/brain/procedure-tools.tsindex c5817a39..d05444c7 100644--- a/src/mcp/brain/procedure-tools.ts+++ b/src/mcp/brain/procedure-tools.ts@@ -16,6 +16,8 @@ import {import {listProceduralMemory,markProceduralMemoryUsed,+ recordProceduralOutcome,+ rankProceduralMemory,reconcileProceduralMemory,} from "../../core/brain/procedural-memory.ts";import { readProceduralGraph, rebuildProceduralGraph } from "../../core/brain/procedural-graph.ts";@@ -93,7 +95,9 @@ async function toolBrainProceduralMemory(}if (operation === "list") {const entries = listProceduralMemory(ctx.vault);- return { total: entries.length, entries };+ // Opt-in success-rate ranking (t_703f7b18); default order unchanged.+ const ordered = args["ranked"] === true ? rankProceduralMemory(entries) : entries;+ return { total: ordered.length, entries: ordered };}if (operation === "mark_used") {const id = requiredStringArg("brain_procedural_memory", args, "id");@@ -103,9 +107,24 @@ async function toolBrainProceduralMemory(}return { ...updated };}+ if (operation === "mark_outcome") {+ const id = requiredStringArg("brain_procedural_memory", args, "id");+ const outcome = requiredStringArg("brain_procedural_memory", args, "outcome");+ if (outcome !== "success" && outcome !== "failure") {+ throw new MCPError(+ INVALID_PARAMS,+ "brain_procedural_memory: outcome must be 'success' or 'failure'",+ );+ }+ const updated = recordProceduralOutcome(ctx.vault, id, outcome);+ if (!updated) {+ throw new MCPError(INVALID_PARAMS, `brain_procedural_memory: unknown entry id: ${id}`);+ }+ return { ...updated };+ }throw new MCPError(INVALID_PARAMS,- "brain_procedural_memory: operation must be one of reconcile|list|mark_used",+ "brain_procedural_memory: operation must be one of reconcile|list|mark_used|mark_outcome",);}@@ -225,13 +244,13 @@ export const PROCEDURE_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([{name: "brain_procedural_memory",description:- "Reconcile/list procedural memory index and update usage sidecar (reconcile, list, mark_used).",+ "Reconcile/list procedural memory and update the usage + outcome sidecar (reconcile, list, mark_used, mark_outcome). list accepts ranked:true to order by validated success rate.",inputSchema: {type: "object",properties: {operation: {type: "string",- enum: ["reconcile", "list", "mark_used"],+ enum: ["reconcile", "list", "mark_used", "mark_outcome"],description: "Tool operation.",},roots: {@@ -241,7 +260,16 @@ export const PROCEDURE_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([},id: {type: "string",- description: "Procedural entry id for mark_used.",+ description: "Procedural entry id for mark_used / mark_outcome.",+ },+ outcome: {+ type: "string",+ enum: ["success", "failure"],+ description: "Host-reported result for mark_outcome.",+ },+ ranked: {+ type: "boolean",+ description: "list only: order entries by validated success rate.",},},required: ["operation"],diff --git a/src/mcp/brain/recall-tools.ts b/src/mcp/brain/recall-tools.tsindex 8d3d939f..a79e8964 100644--- a/src/mcp/brain/recall-tools.ts+++ b/src/mcp/brain/recall-tools.ts@@ -12,6 +12,7 @@ import { parseRecallBenchmarkDataset, runRecallBenchmark } from "../../core/searimport { loadTunedParameters, resetTuning, tuneRecall } from "../../core/search/tuning.ts";import { listGateTelemetry, summarizeGateTelemetry } from "../../core/brain/gate-telemetry.ts";import { computeMemoryCostMeter } from "../../core/brain/memory-cost-meter.ts";+import { observedReuseRates } from "../../core/brain/observed-use.ts";import {isRecallTelemetryMode,isRecallTelemetryStatus,@@ -211,6 +212,15 @@ async function toolBrainRecallTelemetry(});return { ...summary };}+ // Observed-use blend (t_65588d8b): the folded per-artifact+ // USED/IGNORED/CONTRADICTED reuse signal that recall ranking prefers over+ // predicted importance. Exposed here so the blend is inspectable.+ if (operation === "observed_reuse") {+ const artifacts = [...observedReuseRates(ctx.vault).entries()]+ .map(([key, r]) => ({ key, ...r }))+ .toSorted((a, b) => (a.score !== b.score ? b.score - a.score : a.key < b.key ? -1 : 1));+ return { vault_path: ctx.vault, total: artifacts.length, artifacts };+ }// Write-vs-read cost meter (memory cost meter): folds write volume// against the read telemetry above. Period-based; mode/status/host/limit// do not apply to the write side, so reject them to mirror the CLI's@@ -245,7 +255,7 @@ async function toolBrainRecallTelemetry(}throw new MCPError(INVALID_PARAMS,- "brain_recall_telemetry: operation must be list, summary, gate_list, gate_summary, or cost",+ "brain_recall_telemetry: operation must be list, summary, gate_list, gate_summary, observed_reuse, or cost",);}@@ -843,9 +853,9 @@ export const RECALL_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([properties: {operation: {type: "string",- enum: ["list", "summary", "gate_list", "gate_summary", "cost"],+ enum: ["list", "summary", "gate_list", "gate_summary", "observed_reuse", "cost"],description:- "list/summary for recall telemetry; gate_list/gate_summary for recall-gate decisions; cost for the write-vs-read cost meter.",+ "list/summary for recall telemetry; gate_list/gate_summary for gate decisions; observed_reuse for the observed-use ranking signal; cost for the cost meter.",},mode: {type: "string",diff --git a/src/mcp/registry-guard.ts b/src/mcp/registry-guard.tsindex 9c13988a..8e6ccade 100644--- a/src/mcp/registry-guard.ts+++ b/src/mcp/registry-guard.ts@@ -83,6 +83,7 @@ export const PREVIEW_BUDGET_EXEMPT: Readonly<Record<string, string>> = Object.frbrain_feedback: "write; returns a small fixed-shape ack",brain_apply_evidence: "write; returns a small fixed-shape ack",brain_note: "write; returns a small fixed-shape ack",+ brain_observed_use: "write; returns a small fixed-shape ack (records/aggregates count)",brain_create_note: "write; returns the created note path and a created flag",brain_pinned_context: "pinned.md is operator-curated and small by practice",brain_recall_feedback: "write; returns one event receipt plus bounded weights",diff --git a/tests/cli/search-provider-cli.test.ts b/tests/cli/search-provider-cli.test.tsindex 33e2c30a..908ba1e7 100644--- a/tests/cli/search-provider-cli.test.ts+++ b/tests/cli/search-provider-cli.test.ts@@ -105,3 +105,66 @@ test("provider show of an unknown name exits non-zero", async () => {const show = await runCli(["search", "provider", "show", "ghost"], { env: env() });expect(show.returncode).not.toBe(0);});++test("provider presets lists the curated catalog without a vault", async () => {+ const presets = await runCli(["search", "provider", "presets", "--json"], { env: {} });+ expect(presets.returncode).toBe(0);+ const arr = JSON.parse(presets.stdout);+ expect(Array.isArray(arr)).toBe(true);+ expect(arr.length).toBeGreaterThan(0);+ expect(arr[0]).toHaveProperty("model");+ expect(arr[0]).toHaveProperty("dimension");+});++test("provider add without --model defaults to the recommended preset", async () => {+ const add = await runCli(+ ["search", "provider", "add", "defaulted", "--base-url", "https://x/v1", "--env-key", "K"],+ { env: env() },+ );+ expect(add.returncode).toBe(0);+ const show = await runCli(["search", "provider", "show", "defaulted", "--json"], { env: env() });+ const profile = JSON.parse(show.stdout);+ expect(profile.defaultModel).toBe("intfloat/multilingual-e5-small");+});++test("provider add with a custom --model persists it verbatim", async () => {+ await runCli(+ [+ "search",+ "provider",+ "add",+ "custom",+ "--base-url",+ "https://x/v1",+ "--model",+ "acme/custom-embed-9000",+ "--env-key",+ "K",+ ],+ { env: env() },+ );+ const show = await runCli(["search", "provider", "show", "custom", "--json"], { env: env() });+ expect(JSON.parse(show.stdout).defaultModel).toBe("acme/custom-embed-9000");+});++test("provider add accepts a comma-separated --env-key probe list", async () => {+ const add = await runCli(+ [+ "search",+ "provider",+ "add",+ "multi",+ "--base-url",+ "https://x/v1",+ "--model",+ "m",+ "--env-key",+ "PRIMARY_KEY,SECONDARY_KEY",+ ],+ { env: env() },+ );+ expect(add.returncode).toBe(0);+ const list = await runCli(["search", "provider", "list", "--json"], { env: env() });+ const arr = JSON.parse(list.stdout);+ expect(arr[0].envKey).toEqual(["PRIMARY_KEY", "SECONDARY_KEY"]);+});diff --git a/tests/core/bench/locomo.test.ts b/tests/core/bench/locomo.test.tsnew file mode 100644index 00000000..83e30344--- /dev/null+++ b/tests/core/bench/locomo.test.ts@@ -0,0 +1,84 @@+import { test, expect, afterEach } from "bun:test";+import { mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ loadLocomoFixture,+ locomoToBenchFixture,+ parseLocomoDataset,+} from "../../../src/core/bench/locomo.ts";+import { parseBenchFixture } from "../../../src/core/bench/fixture.ts";+import { runMemoryBench } from "../../../src/core/bench/phases.ts";+import { BENCH_REPORT_SCHEMA } from "../../../src/core/bench/types.ts";++const SAMPLE = join("tests", "fixtures", "bench", "locomo-sample.json");++const dirs: string[] = [];+afterEach(() => {+ for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true });+});++test("the loader converts a LoCoMo dataset into a valid BenchFixture", () => {+ const fixture = loadLocomoFixture(SAMPLE);+ // Round-trips through the canonical parser (so the harness accepts it).+ expect(() => parseBenchFixture(fixture)).not.toThrow();+ expect(fixture.name).toBe("locomo-sample");+ expect(fixture.notes.map((n) => n.path).toSorted()).toEqual(["sessions/s1.md", "sessions/s2.md"]);+ // Continuity holds one session_turn per turn (4 total).+ expect(fixture.continuity.length).toBe(4);+ for (const c of fixture.continuity) expect(c.kind).toBe("session_turn");+});++test("QA categories map onto OSB's canonical categories", () => {+ const fixture = loadLocomoFixture(SAMPLE);+ const byId = new Map(fixture.questions.map((q) => [q.id, q]));+ expect(byId.get("locomo-q1")!.category).toBe("single_hop");+ expect(byId.get("locomo-q3")!.category).toBe("temporal");+ expect(byId.get("locomo-q1")!.expected_paths).toEqual(["sessions/s1.md"]);+ expect(byId.get("locomo-q1")!.expected_text).toBe("Friday");+});++test("running the LoCoMo suite produces an o2b.bench.v1 report deterministically", async () => {+ const fixture = loadLocomoFixture(SAMPLE);+ const runsDir = mkdtempSync(join(tmpdir(), "o2b-locomo-runs-"));+ dirs.push(runsDir);+ const report = await runMemoryBench({ fixture, runsDir });+ expect(report.schema).toBe(BENCH_REPORT_SCHEMA);+ expect(report.fixture).toBe("locomo-sample");+ // Deterministic offline run: the judge stays skipped by default.+ expect(report.judge.status).toBe("skipped");+ // Every query's evidence session is keyword-distinctive, so all pass.+ expect(report.quality.passed).toBe(report.quality.total);+ expect(report.quality.total).toBe(3);+}, 30_000);++test("parseLocomoDataset rejects an unparseable turn timestamp", () => {+ expect(() =>+ parseLocomoDataset({+ sessions: [+ {+ session_id: "s1",+ turns: [{ speaker: "A", text: "hi there", timestamp: "not-a-date" }],+ },+ ],+ qa: [{ id: "q", question: "what", evidence_sessions: ["s1"] }],+ }),+ ).toThrow(/valid date/);+});++test("parseLocomoDataset rejects a QA with no evidence sessions", () => {+ expect(() =>+ locomoToBenchFixture(+ parseLocomoDataset({+ sessions: [+ {+ session_id: "s1",+ turns: [{ speaker: "A", text: "hi there", timestamp: "2026-01-01T00:00:00Z" }],+ },+ ],+ qa: [{ id: "q", question: "what", evidence_sessions: [] }],+ }),+ ),+ ).toThrow(/evidence_sessions/);+});diff --git a/tests/core/brain.observed-use.test.ts b/tests/core/brain.observed-use.test.tsnew file mode 100644index 00000000..9993e0ea--- /dev/null+++ b/tests/core/brain.observed-use.test.ts@@ -0,0 +1,84 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdirSync, mkdtempSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ classifyObservedUse,+ emitObservedUse,+ observedReuseRates,+} from "../../src/core/brain/observed-use.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-observed-use-"));+ mkdirSync(join(vault, "Brain"), { recursive: true });+});+afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++test("classifyObservedUse marks a memory USED when a later turn echoes it", () => {+ const verdicts = classifyObservedUse(+ [{ id: "1", path: "ml.md", content: "gradient descent optimizes the loss surface" }],+ ["Earlier we applied gradient descent to optimize the model."],+ );+ expect(verdicts[0]!.verdict).toBe("USED");+});++test("classifyObservedUse marks an unreferenced memory IGNORED", () => {+ const verdicts = classifyObservedUse(+ [{ id: "1", content: "green turtles migrate across oceans" }],+ ["The build pipeline failed on the lint step."],+ );+ expect(verdicts[0]!.verdict).toBe("IGNORED");+});++test("classifyObservedUse marks a stance flip CONTRADICTED", () => {+ const verdicts = classifyObservedUse(+ [{ id: "1", content: "the production deploy is safe" }],+ ["Actually the production deploy is not safe right now."],+ );+ expect(verdicts[0]!.verdict).toBe("CONTRADICTED");+});++test("observedReuseRates folds verdicts into a per-artifact score", () => {+ emitObservedUse(vault, {+ host: "t",+ entries: [+ { id: "a", path: "a.md", verdict: "USED" },+ { id: "b", path: "b.md", verdict: "IGNORED" },+ ],+ });+ emitObservedUse(vault, {+ host: "t",+ entries: [+ { id: "a", path: "a.md", verdict: "USED" },+ { id: "c", path: "c.md", verdict: "CONTRADICTED" },+ ],+ });+ const rates = observedReuseRates(vault);+ expect(rates.get("a.md")!.used).toBe(2);+ expect(rates.get("a.md")!.score).toBe(1); // 2 used / 2 total+ expect(rates.get("b.md")!.score).toBe(0); // ignored only+ expect(rates.get("c.md")!.score).toBe(0); // contradicted demotes to 0+});++test("a contradicted-heavy artifact scores below a used one", () => {+ emitObservedUse(vault, {+ host: "t",+ entries: [+ { id: "x", path: "x.md", verdict: "USED" },+ { id: "x", path: "x.md", verdict: "CONTRADICTED" },+ ],+ });+ const r = observedReuseRates(vault).get("x.md")!;+ expect(r.used).toBe(1);+ expect(r.contradicted).toBe(1);+ expect(r.score).toBe(0); // (1 - 1) / 2+});++test("an empty vault yields an empty reuse map (byte-identical ranking)", () => {+ expect(observedReuseRates(vault).size).toBe(0);+});diff --git a/tests/core/brain.procedural-outcome.test.ts b/tests/core/brain.procedural-outcome.test.tsnew file mode 100644index 00000000..6fd367f3--- /dev/null+++ b/tests/core/brain.procedural-outcome.test.ts@@ -0,0 +1,110 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ listProceduralMemory,+ markProceduralMemoryUsed,+ proceduralSuccessRate,+ rankProceduralMemory,+ recordProceduralOutcome,+ reconcileProceduralMemory,+} from "../../src/core/brain/procedural-memory.ts";+import { proceduralMemoryUsagePath } from "../../src/core/brain/paths.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-proc-outcome-"));+ mkdirSync(join(vault, "Brain", "procedures"), { recursive: true });+ writeFileSync(join(vault, "Brain", "procedures", "alpha.md"), "# Alpha\n\nA reliable runbook.\n");+ writeFileSync(join(vault, "Brain", "procedures", "beta.md"), "# Beta\n\nA flaky runbook.\n");+ reconcileProceduralMemory(vault, { roots: [join(vault, "Brain", "procedures")] });+});+afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++function idFor(title: string): string {+ const e = listProceduralMemory(vault).find((x) => x.title === title);+ if (!e) throw new Error(`no entry ${title}`);+ return e.id;+}++test("fresh entries carry zero outcome counters and a null success rate", () => {+ const entries = listProceduralMemory(vault);+ expect(entries.length).toBe(2);+ for (const e of entries) {+ expect(e.successCount).toBe(0);+ expect(e.failureCount).toBe(0);+ expect(proceduralSuccessRate(e)).toBeNull();+ }+});++test("recordProceduralOutcome accumulates success/failure counters", () => {+ const alpha = idFor("Alpha");+ recordProceduralOutcome(vault, alpha, "success");+ recordProceduralOutcome(vault, alpha, "success");+ recordProceduralOutcome(vault, alpha, "failure");+ const e = listProceduralMemory(vault).find((x) => x.id === alpha)!;+ expect(e.successCount).toBe(2);+ expect(e.failureCount).toBe(1);+ expect(proceduralSuccessRate(e)).toBeCloseTo(2 / 3, 6);+});++test("an unknown id returns null", () => {+ expect(recordProceduralOutcome(vault, "pmem-nope", "success")).toBeNull();+});++test("ranking surfaces the proven procedure above the failing one", () => {+ const alpha = idFor("Alpha");+ const beta = idFor("Beta");+ recordProceduralOutcome(vault, alpha, "success");+ recordProceduralOutcome(vault, alpha, "success");+ recordProceduralOutcome(vault, beta, "failure");+ recordProceduralOutcome(vault, beta, "failure");+ const ranked = rankProceduralMemory(listProceduralMemory(vault));+ expect(ranked[0]!.id).toBe(alpha); // rate 1.0+ expect(ranked[1]!.id).toBe(beta); // rate 0.0+});++test("an unproven procedure ranks between a proven-good and a proven-bad one", () => {+ const alpha = idFor("Alpha");+ const beta = idFor("Beta");+ recordProceduralOutcome(vault, alpha, "success"); // rate 1.0+ recordProceduralOutcome(vault, beta, "failure"); // rate 0.0+ // A third, unproven procedure.+ writeFileSync(join(vault, "Brain", "procedures", "gamma.md"), "# Gamma\n\nUntested.\n");+ reconcileProceduralMemory(vault, { roots: [join(vault, "Brain", "procedures")] });+ const gamma = idFor("Gamma");+ const ranked = rankProceduralMemory(listProceduralMemory(vault)).map((e) => e.id);+ expect(ranked).toEqual([alpha, gamma, beta]);+});++test("outcome recording is order-insensitive", () => {+ const alpha = idFor("Alpha");+ for (const o of ["success", "failure", "success"] as const)+ recordProceduralOutcome(vault, alpha, o);+ const first = listProceduralMemory(vault).find((x) => x.id === alpha)!;++ // Fresh vault, reversed order.+ rmSync(vault, { recursive: true, force: true });+ mkdirSync(join(vault, "Brain", "procedures"), { recursive: true });+ writeFileSync(join(vault, "Brain", "procedures", "alpha.md"), "# Alpha\n\nA reliable runbook.\n");+ reconcileProceduralMemory(vault, { roots: [join(vault, "Brain", "procedures")] });+ for (const o of ["success", "success", "failure"] as const)+ recordProceduralOutcome(vault, alpha, o);+ const second = listProceduralMemory(vault).find((x) => x.id === alpha)!;++ expect(second.successCount).toBe(first.successCount);+ expect(second.failureCount).toBe(first.failureCount);+});++test("outcome-free usage sidecar stays byte-identical (no outcome keys written)", () => {+ const alpha = idFor("Alpha");+ markProceduralMemoryUsed(vault, alpha);+ const usage = readFileSync(proceduralMemoryUsagePath(vault), "utf8");+ expect(usage).not.toContain("successCount");+ expect(usage).not.toContain("failureCount");+});diff --git a/tests/core/search/embeddings.presets.test.ts b/tests/core/search/embeddings.presets.test.tsnew file mode 100644index 00000000..3b26967c--- /dev/null+++ b/tests/core/search/embeddings.presets.test.ts@@ -0,0 +1,34 @@+import { test, expect } from "bun:test";++import {+ EMBEDDING_MODEL_PRESETS,+ RECOMMENDED_EMBEDDING_MODEL,+ findEmbeddingPreset,+} from "../../../src/core/search/embeddings/presets.ts";++test("the catalog is non-empty and multilingual-first", () => {+ expect(EMBEDDING_MODEL_PRESETS.length).toBeGreaterThan(0);+ expect(EMBEDDING_MODEL_PRESETS[0]!.multilingual).toBe(true);+});++test("every preset carries a stable positive dimension and a note", () => {+ for (const p of EMBEDDING_MODEL_PRESETS) {+ expect(p.model.trim()).not.toBe("");+ expect(p.label.trim()).not.toBe("");+ expect(p.dimension).toBeGreaterThan(0);+ expect(p.note.trim()).not.toBe("");+ }+});++test("model strings are unique", () => {+ const models = EMBEDDING_MODEL_PRESETS.map((p) => p.model);+ expect(new Set(models).size).toBe(models.length);+});++test("the recommended default is one of the presets", () => {+ expect(findEmbeddingPreset(RECOMMENDED_EMBEDDING_MODEL)).not.toBeNull();+});++test("findEmbeddingPreset returns null for a custom model", () => {+ expect(findEmbeddingPreset("acme/custom-embed-9000")).toBeNull();+});diff --git a/tests/core/search/embeddings.registry.test.ts b/tests/core/search/embeddings.registry.test.tsindex b777cb5e..87650285 100644--- a/tests/core/search/embeddings.registry.test.ts+++ b/tests/core/search/embeddings.registry.test.ts@@ -87,6 +87,7 @@ test("expandRegisteredProvider resolves to openai-compat with the env key", () =baseUrl: nim.baseUrl,model: nim.defaultModel,apiKey: "secret-123",+ apiKeys: ["secret-123"],});});@@ -94,6 +95,32 @@ test("expandRegisteredProvider yields a null apiKey when the env var is unset",const registry = addProviderProfile(vault, nim);const expanded = expandRegisteredProvider("nvidia-nim", registry, {});expect(expanded?.apiKey).toBeNull();+ expect(expanded?.apiKeys).toEqual([]);+});++test("a profile envKey may be an ordered probe list; the first present key wins", () => {+ const registry = addProviderProfile(vault, {+ ...nim,+ envKey: ["NIM_PRIMARY", "NIM_SECONDARY", "NIM_TERTIARY"],+ });+ // primary unset, secondary + tertiary set -> secondary wins, tertiary is the fallback.+ const expanded = expandRegisteredProvider("nvidia-nim", registry, {+ NIM_SECONDARY: "key-b",+ NIM_TERTIARY: "key-c",+ });+ expect(expanded?.apiKey).toBe("key-b");+ expect(expanded?.apiKeys).toEqual(["key-b", "key-c"]);+});++test("probe-list profiles round-trip through the registry file", () => {+ addProviderProfile(vault, { ...nim, envKey: ["A_KEY", "B_KEY"] });+ const loaded = getProviderProfile(vault, "nvidia-nim");+ expect(loaded?.envKey).toEqual(["A_KEY", "B_KEY"]);+});++test("an empty probe list is rejected", () => {+ expect(() => addProviderProfile(vault, { ...nim, envKey: [] })).toThrow(/env/i);+ expect(() => addProviderProfile(vault, { ...nim, envKey: [" "] })).toThrow(/env/i);});test("expandRegisteredProvider returns null for an unknown name", () => {diff --git a/tests/core/search/embeddings.test.ts b/tests/core/search/embeddings.test.tsindex fdfb4b15..2027c991 100644--- a/tests/core/search/embeddings.test.ts+++ b/tests/core/search/embeddings.test.ts@@ -155,6 +155,61 @@ test("embed() throws EMBEDDING_PROVIDER_HTTP after exhausting retries on 5xx", aexpect(server.callCount()).toBe(3);});+test("embed() fails over to the next key on 401 and pins the working key", async () => {+ const seenKeys: string[] = [];+ server.setHandler((req) => {+ const auth = req.headers["authorization"] ?? "";+ const key = auth.replace(/^Bearer\s+/i, "");+ seenKeys.push(key);+ if (key === "stale-key") return { status: 401, body: { error: "invalid_api_key" } };+ const body = req.body as { input: string[]; model: string };+ const data = body.input.map((_, idx) => ({+ object: "embedding",+ embedding: [1, 0, 0, 0],+ index: idx,+ }));+ return { status: 200, body: { data, model: body.model } };+ });++ const p = new OpenAICompatProvider(+ cfg({ apiKeys: ["stale-key", "good-key"], dimension: 4, batchSize: 1, concurrency: 1 }),+ { backoffMs: [5, 5] },+ );+ // First call fails over stale -> good; second call must reuse the pinned good key.+ const out1 = await p.embed(["a"]);+ const out2 = await p.embed(["b"]);+ expect(out1.length).toBe(1);+ expect(out2.length).toBe(1);+ expect(seenKeys[0]).toBe("stale-key");+ expect(seenKeys.slice(1)).toEqual(["good-key", "good-key"]);+});++test("embed() surfaces the auth error when every probe key fails", async () => {+ server.setHandler(() => ({ status: 401, body: { error: "invalid_api_key" } }));+ const p = new OpenAICompatProvider(cfg({ apiKeys: ["a", "b"] }), { backoffMs: [5, 5] });+ let err: SearchError | null = null;+ try {+ await p.embed(["x"]);+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_PROVIDER_HTTP");+ expect(err?.message).toContain("401");+});++test("embed() with a single key fails fast on 401 (byte-identical single-key path)", async () => {+ server.setHandler(() => ({ status: 401, body: { error: "invalid_api_key" } }));+ const p = new OpenAICompatProvider(cfg(), { backoffMs: [5, 5] });+ let err: SearchError | null = null;+ try {+ await p.embed(["x"]);+ } catch (e) {+ err = e as SearchError;+ }+ expect(err?.code).toBe("EMBEDDING_PROVIDER_HTTP");+ expect(server.callCount()).toBe(1);+});+test("embed() honours per-request timeout via AbortController", async () => {server.setHandler(() => ({ status: 200, body: { data: [] }, delayMs: 200 }));const p = new OpenAICompatProvider(cfg({ timeoutMs: 30 }), { backoffMs: [5, 5] });diff --git a/tests/core/search/embeddings.zeroentropy.test.ts b/tests/core/search/embeddings.zeroentropy.test.tsnew file mode 100644index 00000000..6ac7dc30--- /dev/null+++ b/tests/core/search/embeddings.zeroentropy.test.ts@@ -0,0 +1,129 @@+import { test, expect, beforeEach, afterEach } from "bun:test";++import { ZeroEntropyProvider } from "../../../src/core/search/embeddings/zeroentropy.ts";+import { makeProvider } from "../../../src/core/search/embeddings/provider.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import type { ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";++let server: FakeHttp;++function cfg(overrides: Partial<ResolvedEmbeddingConfig> = {}): ResolvedEmbeddingConfig {+ return Object.freeze({+ enabled: true,+ provider: "zeroentropy",+ baseUrl: server.url,+ model: "zembed-1",+ apiKey: "ze-test-key",+ dimension: null,+ timeoutMs: 5_000,+ concurrency: 2,+ batchSize: 32,+ costGateUsd: 0,+ ...overrides,+ });+}++/** ZeroEntropy-shaped handler: POST /models/embed -> {results:[{embedding}]}. */+function zeHandler(dim = 4) {+ return (req: {+ method: string;+ path: string;+ body: unknown;+ }): { status: number; body: unknown } => {+ if (!req.path.endsWith("/models/embed") || req.method !== "POST") {+ return { status: 404, body: { error: "not_found" } };+ }+ const body = req.body as { input: string[]; dimensions?: number };+ const width = body.dimensions ?? dim;+ const results = body.input.map((text, index) => ({+ // Deterministic vector: leading component from length, then index.+ embedding: [text.length, index, ...new Array(Math.max(0, width - 2)).fill(1)],+ }));+ return { status: 200, body: { results, usage: { total_bytes: 1, total_tokens: 1 } } };+ };+}++beforeEach(async () => {+ server = await startFakeHttp();+});++afterEach(async () => {+ await server.close();+});++test("embed() returns unit-normalised vectors in input order", async () => {+ server.setHandler(zeHandler(4));+ const p = new ZeroEntropyProvider(cfg({ dimension: 4 }));+ const out = await p.embed(["aaaa", "bb", "c"]);+ expect(out.length).toBe(3);+ for (const v of out) {+ const norm = Math.sqrt(v.reduce((s, x) => s + x * x, 0));+ expect(norm).toBeCloseTo(1.0, 6);+ }+ // First input longest -> largest leading component before normalisation.+ expect(out[0]![0]).toBeGreaterThan(out[2]![0]!);+});++test("embed() sends Bearer auth and the native request body", async () => {+ let seenAuth = "";+ let seenBody: Record<string, unknown> = {};+ server.setHandler((req) => {+ seenAuth = req.headers["authorization"] ?? "";+ seenBody = req.body as Record<string, unknown>;+ const body = req.body as { input: string[] };+ return {+ status: 200,+ body: { results: body.input.map(() => ({ embedding: [1, 0, 0, 0] })) },+ };+ });+ const p = new ZeroEntropyProvider(cfg({ dimension: 4 }));+ await p.embed(["x"]);+ expect(seenAuth).toBe("Bearer ze-test-key");+ expect(seenBody["model"]).toBe("zembed-1");+ expect(seenBody["input_type"]).toBe("document");+ expect(seenBody["input"]).toEqual(["x"]);+});++test("embed() forwards a configured dimension to the API", async () => {+ let seenDim: unknown;+ server.setHandler((req) => {+ seenDim = (req.body as { dimensions?: number }).dimensions;+ return zeHandler(640)(req);+ });+ const p = new ZeroEntropyProvider(cfg({ dimension: 640 }));+ const out = await p.embed(["x"]);+ expect(seenDim).toBe(640);+ expect(out[0]!.length).toBe(640);+});++test("ping() returns ok with dimension", async () => {+ server.setHandler(zeHandler(4));+ const p = new ZeroEntropyProvider(cfg({ dimension: 4 }));+ const res = await p.ping();+ if (!res.ok) throw new Error("expected ok");+ expect(res.dimension).toBe(4);+});++test("embed() retries on 503 then succeeds", async () => {+ let calls = 0;+ server.setHandler((req) => {+ calls++;+ if (calls === 1) return { status: 503, body: { error: "busy" } };+ return zeHandler(4)(req);+ });+ const p = new ZeroEntropyProvider(cfg({ dimension: 4 }), { backoffMs: [10, 20] });+ const out = await p.embed(["once"]);+ expect(out.length).toBe(1);+ expect(calls).toBe(2);+});++test("makeProvider returns a ZeroEntropyProvider for provider 'zeroentropy'", () => {+ const p = makeProvider(cfg({ dimension: 4 }));+ expect(p).toBeInstanceOf(ZeroEntropyProvider);+ expect(p.name).toBe("zeroentropy");+});++test("makeProvider throws EMBEDDING_KEY_MISSING when key absent", () => {+ expect(() => makeProvider(cfg({ apiKey: null }))).toThrow(SearchError);+});diff --git a/tests/core/search/enrich.test.ts b/tests/core/search/enrich.test.tsindex 53de2f37..30a9d615 100644--- a/tests/core/search/enrich.test.ts+++ b/tests/core/search/enrich.test.ts@@ -43,6 +43,7 @@ test("projects the ranker breakdown verbatim when present", () => {entity: 0.02,activation: 0,coAccess: 0,+ reuse: 0,link: 0.01,recency: 0.03,tier: 1.2,diff --git a/tests/core/search/graph-prepass.test.ts b/tests/core/search/graph-prepass.test.tsnew file mode 100644index 00000000..dc652581--- /dev/null+++ b/tests/core/search/graph-prepass.test.ts@@ -0,0 +1,96 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { Store } from "../../../src/core/search/store.ts";+import { planReadShortlist } from "../../../src/core/search/graph-prepass.ts";+import { makeConfig } from "../../helpers/search-fixtures.ts";+import type { ResolvedSearchConfig } from "../../../src/core/search/types.ts";++let vault: string;+let config: ResolvedSearchConfig;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-graph-prepass-"));+ mkdirSync(join(vault, "Brain"), { recursive: true });+ config = makeConfig({ vault, dbPath: join(vault, "index.sqlite") });+});+afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++// A link chain hub -> mid -> leaf, so `leaf` is reachable from `hub` only+// via two hops. Plus an unrelated note.+function writeChain(): void {+ writeFileSync(join(vault, "hub.md"), "# Alpha Topic\n\nThe entry point. See [[mid]].\n");+ writeFileSync(join(vault, "mid.md"), "# Middle Note\n\nBridges onward to [[leaf]].\n");+ writeFileSync(join(vault, "leaf.md"), "# Leaf Detail\n\nThe deep conclusion.\n");+ writeFileSync(join(vault, "other.md"), "# Turtles\n\nUnrelated content about oceans.\n");+}++test("should_read ranks the title match first and reaches a 2-hop note via BFS", async () => {+ writeChain();+ await indexVault(config);+ const store = await Store.open(config, { mode: "read" });+ try {+ const plan = planReadShortlist(store, "Alpha Topic", { maxHops: 2, hopDecay: 0.5 });+ expect(plan.mode).toBe("should_read");+ const paths = plan.shortlist.map((e) => e.path);+ // The title match is the top seed.+ expect(paths[0]).toBe("hub.md");+ // `leaf.md` is reachable only through hub -> mid -> leaf (2 hops).+ const leaf = plan.shortlist.find((e) => e.path === "leaf.md");+ expect(leaf).toBeDefined();+ expect(leaf!.hops).toBe(2);+ expect(leaf!.reasons).toContain("bfs:2");+ // The unrelated note is not surfaced by this query.+ expect(paths).not.toContain("other.md");+ } finally {+ await store.close();+ }+});++test("index_only answers from metadata with zero note bodies read", async () => {+ writeChain();+ await indexVault(config);+ const store = await Store.open(config, { mode: "read" });+ try {+ const plan = planReadShortlist(store, "Alpha Topic", { indexOnly: true, maxHops: 2 });+ expect(plan.mode).toBe("index_only");+ expect(plan.notesRead).toBe(0);+ expect(plan.shortlist.length).toBeGreaterThan(0);+ // No summaries hydrated in index-only mode.+ for (const e of plan.shortlist) expect(e.summary).toBeNull();+ expect(plan.shortlist[0]!.path).toBe("hub.md");+ } finally {+ await store.close();+ }+});++test("should_read hydrates a bounded number of summaries (bodies read > 0)", async () => {+ writeChain();+ await indexVault(config);+ const store = await Store.open(config, { mode: "read" });+ try {+ const plan = planReadShortlist(store, "Alpha Topic", { maxHops: 1 });+ expect(plan.notesRead).toBeGreaterThan(0);+ const hub = plan.shortlist.find((e) => e.path === "hub.md");+ expect(hub!.summary).toContain("entry point");+ } finally {+ await store.close();+ }+});++test("a query matching nothing yields an empty shortlist", async () => {+ writeChain();+ await indexVault(config);+ const store = await Store.open(config, { mode: "read" });+ try {+ const plan = planReadShortlist(store, "zzzznomatch", { maxHops: 2 });+ expect(plan.shortlist.length).toBe(0);+ } finally {+ await store.close();+ }+});diff --git a/tests/core/search/ranker.test.ts b/tests/core/search/ranker.test.tsindex e57890cd..1d747635 100644--- a/tests/core/search/ranker.test.ts+++ b/tests/core/search/ranker.test.ts@@ -51,6 +51,40 @@ test("union of keyword + semantic produces hybrid scoring", () => {expect(map.get(3)?.searchType).toBe("semantic");});+test("observed-reuse boost lifts an equally-relevant chunk (t_65588d8b)", () => {+ const kw: KeywordHit[] = [+ { chunkId: 1, documentId: 10, bm25: -3 },+ { chunkId: 2, documentId: 11, bm25: -3 },+ ];+ const hydrated = new Map<number, HydratedChunk>([+ [1, hyd(1, 10, OLD_MTIME)],+ [2, hyd(2, 11, OLD_MTIME)],+ ]);+ const base = {+ keyword: kw,+ semantic: [] as SemanticHit[],+ hydrated,+ inboundLinkSources: new Map<number, ReadonlySet<number>>(),+ tagsByDoc: new Map<number, ReadonlySet<string>>(),+ };+ const opts = { keywordWeight: 0.6, semanticWeight: 0.4, limit: 10, nowMs: NOW };++ // Without a reuse signal, the two chunks are tied.+ const neutral = rankResults(base, opts);+ expect(neutral[0]!.score).toBeCloseTo(neutral[1]!.score, 6);++ // With chunk 2 carrying a high observed-reuse score, it must lead.+ const boosted = rankResults({ ...base, reuseRateByChunk: new Map([[2, 1]]) }, opts);+ expect(boosted[0]!.chunkId).toBe(2);+ const c1 = boosted.find((r) => r.chunkId === 1)!;+ const c2 = boosted.find((r) => r.chunkId === 2)!;+ expect(c2.score).toBeGreaterThan(c1.score);+ // Explainability: the boost surfaces in reasons and the breakdown.+ expect(c2.reasons.some((r) => r.startsWith("observed_reuse:"))).toBe(true);+ expect(c2.breakdown!.reuse).toBeGreaterThan(0);+ expect(c1.breakdown!.reuse).toBe(0);+});+test("score is clamped to [0,1] even with boosts", () => {const ranked = rankResults({diff --git a/tests/core/search/rerank-eval-gate.test.ts b/tests/core/search/rerank-eval-gate.test.tsnew file mode 100644index 00000000..859b41eb--- /dev/null+++ b/tests/core/search/rerank-eval-gate.test.ts@@ -0,0 +1,46 @@+import { test, expect, afterEach } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { runRerankEvalGate } from "../../../src/core/search/rerank-eval-gate.ts";+import type { RecallBenchmarkDataset } from "../../../src/core/search/benchmark.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++const cleanups: Array<() => void> = [];+afterEach(() => {+ for (const c of cleanups.splice(0)) c();+});++async function indexed() {+ const v = createTempVault("rerank-gate");+ cleanups.push(v.cleanup);+ writeMd(v.vault, "ml.md", "# Machine learning\n\nGradient descent optimizes the loss surface.");+ writeMd(v.vault, "garden.md", "# Gardening\n\nTomatoes and compost in the spring season.");+ const cfg = makeConfig({ vault: v.vault, dbPath: v.dbPath });+ await indexVault(cfg, {});+ return cfg;+}++const dataset: RecallBenchmarkDataset = {+ queries: [{ id: "q1", query: "gradient descent", expected: ["ml.md"] }],+};++test("the eval gate runs both benchmarks and returns comparable metric families", async () => {+ const cfg = await indexed();+ const gate = await runRerankEvalGate(cfg, dataset, { kind: "local" });+ expect(typeof gate.improves).toBe("boolean");+ expect(["enable", "keep-disabled"]).toContain(gate.recommendation);+ expect(gate.baseline.hitAtK).toBeGreaterThanOrEqual(0);+ expect(gate.reranked.hitAtK).toBeGreaterThanOrEqual(0);+ expect(gate.deltas).toHaveProperty("mrr");+ expect(gate.deltas).toHaveProperty("hitAtK");+});++test("no lift (answer already top-ranked) recommends keep-disabled", async () => {+ const cfg = await indexed();+ const gate = await runRerankEvalGate(cfg, dataset, { kind: "local" });+ // The single expected doc is already the only strong match, so reranking+ // cannot lift ranking further: no improvement, no regression.+ expect(gate.deltas.hitAtK).toBe(0);+ expect(gate.improves).toBe(false);+ expect(gate.recommendation).toBe("keep-disabled");+});diff --git a/tests/core/search/rerank.config.test.ts b/tests/core/search/rerank.config.test.tsindex 8b3dcd61..4eefe0e7 100644--- a/tests/core/search/rerank.config.test.ts+++ b/tests/core/search/rerank.config.test.ts@@ -72,6 +72,7 @@ test("explicit config keys resolve, with the api key read from env-key", () => {const cfg = resolveSearchConfig({ vault: tmp, configPath });expect(cfg.rerank).toEqual({enabled: true,+ kind: "openai-compat",baseUrl: "https://api.example.com/v1/",model: "rerank-1",envKey: "MY_RERANK_KEY",diff --git a/tests/core/search/rerank.local.test.ts b/tests/core/search/rerank.local.test.tsnew file mode 100644index 00000000..9ef792c3--- /dev/null+++ b/tests/core/search/rerank.local.test.ts@@ -0,0 +1,79 @@+import { test, expect } from "bun:test";++import { LocalRerankProvider, scoreLocalRerank } from "../../../src/core/search/rerank/local.ts";+import { applyCrossEncoderRerank } from "../../../src/core/search/rerank/index.ts";+import type { BrainSearchResult, ResolvedRerankConfig } from "../../../src/core/search/types.ts";++const LOCAL: ResolvedRerankConfig = Object.freeze({+ enabled: true,+ kind: "local",+ baseUrl: null,+ model: null,+ envKey: null,+ apiKey: null,+ topK: 20,+ minScore: 0,+});++function result(id: number, content: string): BrainSearchResult {+ return Object.freeze({+ documentId: id,+ chunkId: id,+ path: `${id}.md`,+ title: null,+ content,+ startLine: 1,+ endLine: 1,+ score: 1 / id,+ keywordScore: 0,+ semanticScore: 0,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "keyword" as const,+ reasons: Object.freeze([]),+ });+}++test("scoreLocalRerank rewards more query-term coverage", () => {+ const both = scoreLocalRerank("gradient descent", "gradient descent optimizes the loss");+ const one = scoreLocalRerank("gradient descent", "gradient boosting is different");+ const none = scoreLocalRerank("gradient descent", "turtles migrate across oceans");+ expect(both).toBeGreaterThan(one);+ expect(one).toBeGreaterThan(none);+ expect(none).toBe(0);+});++test("scoreLocalRerank is deterministic and bounded to [0,1]", () => {+ const a = scoreLocalRerank("neural network", "a neural network learns");+ const b = scoreLocalRerank("neural network", "a neural network learns");+ expect(a).toBe(b);+ expect(a).toBeGreaterThanOrEqual(0);+ expect(a).toBeLessThanOrEqual(1);+});++test("LocalRerankProvider returns one score per document, in input order", async () => {+ const p = new LocalRerankProvider();+ const scores = await p.rerank("alpha beta", ["nothing here", "alpha beta gamma", "alpha only"]);+ expect(scores.length).toBe(3);+ expect(scores[1]).toBeGreaterThan(scores[2]!);+ expect(scores[2]).toBeGreaterThan(scores[0]!);+});++test("applyCrossEncoderRerank with kind 'local' reorders offline (no endpoint)", async () => {+ const results = [+ result(1, "turtles and oceans, unrelated"),+ result(2, "machine learning with gradient descent"),+ result(3, "gradient partially mentioned"),+ ];+ const reordered = await applyCrossEncoderRerank(results, "gradient descent", LOCAL);+ // The best lexical match for "gradient descent" is doc 2; it must lead.+ expect(reordered[0]!.path).toBe("2.md");+ expect(reordered.length).toBe(3);+ expect(reordered[0]!.reasons.some((r) => r.startsWith("cross_encoder:"))).toBe(true);+});++test("disabled rerank returns the identical reference (byte-identical)", async () => {+ const results = [result(1, "a"), result(2, "b")];+ const out = await applyCrossEncoderRerank(results, "q", { ...LOCAL, enabled: false });+ expect(out).toBe(results);+});diff --git a/tests/core/search/rerank.test.ts b/tests/core/search/rerank.test.tsindex 69183867..ffa8c5b4 100644--- a/tests/core/search/rerank.test.ts+++ b/tests/core/search/rerank.test.ts@@ -42,6 +42,7 @@ function result(id: number, content: string): BrainSearchResult {const BASE: ResolvedRerankConfig = Object.freeze({enabled: false,+ kind: "openai-compat",baseUrl: null,model: null,envKey: null,diff --git a/tests/core/search/store-aliases.test.ts b/tests/core/search/store-aliases.test.tsindex 30f31791..42411845 100644--- a/tests/core/search/store-aliases.test.ts+++ b/tests/core/search/store-aliases.test.ts@@ -53,8 +53,8 @@ function targetDocId(store: Store, sourceId: number): number | null {return row?.target_document_id ?? null;}-test("schema version is 8", () => {- expect(LATEST_SCHEMA_VERSION).toBe(8);+test("schema version is 9", () => {+ expect(LATEST_SCHEMA_VERSION).toBe(9);});test("normalizeAlias lower-cases, trims, and NFC-normalises", () => {diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.tsindex 0f60908a..79d73c84 100644--- a/tests/core/search/store.test.ts+++ b/tests/core/search/store.test.ts@@ -60,9 +60,13 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCoselfTuningEnabled: false,chainStopEnabled: false,chainStopScore: 0.8,+ trigramPrefilterEnabled: false,+ trigramPrefilterMinChunks: 5000,+ trigramPrefilterMaxSelectivity: 0.5,}),rerank: Object.freeze({enabled: false,+ kind: "openai-compat",baseUrl: null,model: null,envKey: null,diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.tsindex 59788621..720e01e9 100644--- a/tests/core/search/store.vec.test.ts+++ b/tests/core/search/store.vec.test.ts@@ -63,9 +63,13 @@ function semanticConfig(selfTuningEnabled: false,chainStopEnabled: false,chainStopScore: 0.8,+ trigramPrefilterEnabled: false,+ trigramPrefilterMinChunks: 5000,+ trigramPrefilterMaxSelectivity: 0.5,}),rerank: Object.freeze({enabled: false,+ kind: "openai-compat",baseUrl: null,model: null,envKey: null,diff --git a/tests/core/search/trigram-prefilter.test.ts b/tests/core/search/trigram-prefilter.test.tsnew file mode 100644index 00000000..e07ed2bd--- /dev/null+++ b/tests/core/search/trigram-prefilter.test.ts@@ -0,0 +1,102 @@+import { test, expect, afterEach } from "bun:test";++import {+ containsCjk,+ extractTrigramTerms,+ isLowSelectivity,+ planTrigramPrefilter,+} from "../../../src/core/search/trigram-prefilter.ts";+import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/index.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++// βββ pure planner ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++test("planTrigramPrefilter skips a short query (no term >= 3 chars)", () => {+ expect(planTrigramPrefilter("a b cd").mode).toBe("skip");+ const plan = planTrigramPrefilter("ab");+ expect(plan).toEqual({ mode: "skip", reason: "short" });+});++test("planTrigramPrefilter skips a CJK query", () => {+ expect(planTrigramPrefilter("ζ₯ζ¬θͺ").mode).toBe("skip");+ expect(planTrigramPrefilter("mix ζ₯ζ¬").mode).toBe("skip");+ expect(containsCjk("hello")).toBe(false);+});++test("planTrigramPrefilter builds a conjunctive FTS query for qualifying terms", () => {+ const plan = planTrigramPrefilter("Hello World");+ expect(plan.mode).toBe("match");+ if (plan.mode !== "match") throw new Error("expected match");+ expect(plan.terms).toEqual(["hello", "world"]);+ expect(plan.ftsQuery).toBe('"hello" AND "world"');+});++test("extractTrigramTerms drops sub-trigram tokens", () => {+ expect(extractTrigramTerms("go to the sea")).toEqual(["the", "sea"]);+});++test("isLowSelectivity compares candidate fraction to the threshold", () => {+ expect(isLowSelectivity(60, 100, 0.5)).toBe(true);+ expect(isLowSelectivity(40, 100, 0.5)).toBe(false);+ expect(isLowSelectivity(5, 0, 0.5)).toBe(false);+});++// βββ integration: strict-superset + fallback βββββββββββββββββββββββββββββββββ++const cleanups: Array<() => void> = [];+afterEach(() => {+ for (const c of cleanups.splice(0)) c();+});++async function fixture() {+ const v = createTempVault("trigram");+ cleanups.push(v.cleanup);+ // "reindexing" contains the substring "index" but the word tokenizer+ // stores it as one token, so a keyword MATCH on "index" misses it.+ writeMd(v.vault, "a.md", "# Ops\n\nReindexing the vault can be slow on large corpora.");+ writeMd(v.vault, "b.md", "# Turtles\n\nGreen turtles migrate across oceans.");+ return v;+}++test("enabled trigram prefilter surfaces a substring match the keyword lane misses", async () => {+ const v = await fixture();+ const off = makeConfig({ vault: v.vault, dbPath: v.dbPath });+ await indexVault(off, {});++ const disabled = await search(off, { query: "index", limit: 10 });+ const enabledCfg = makeConfig({+ vault: v.vault,+ dbPath: v.dbPath,+ trigramPrefilterEnabled: true,+ trigramPrefilterMinChunks: 0,+ });+ const enabled = await search(enabledCfg, { query: "index", limit: 10 });++ const disabledPaths = new Set(disabled.results.map((r) => r.path));+ const enabledPaths = new Set(enabled.results.map((r) => r.path));++ // Strict superset: every disabled result is still present when enabled.+ for (const p of disabledPaths) expect(enabledPaths.has(p)).toBe(true);+ // And the substring-only document now surfaces (it did not before).+ expect(disabledPaths.has("a.md")).toBe(false);+ expect(enabledPaths.has("a.md")).toBe(true);+});++test("short and CJK queries fall back (byte-identical to disabled)", async () => {+ const v = await fixture();+ const enabledCfg = makeConfig({+ vault: v.vault,+ dbPath: v.dbPath,+ trigramPrefilterEnabled: true,+ trigramPrefilterMinChunks: 0,+ });+ await indexVault(enabledCfg, {});+ const offCfg = makeConfig({ vault: v.vault, dbPath: v.dbPath });++ for (const q of ["ab", "ζ΅·"]) {+ const on = await search(enabledCfg, { query: q, limit: 10 });+ const off = await search(offCfg, { query: q, limit: 10 });+ expect(on.results.map((r) => r.path)).toEqual(off.results.map((r) => r.path));+ }+});diff --git a/tests/fixtures/bench/locomo-sample.json b/tests/fixtures/bench/locomo-sample.jsonnew file mode 100644index 00000000..6daae177--- /dev/null+++ b/tests/fixtures/bench/locomo-sample.json@@ -0,0 +1,58 @@+{+ "name": "sample",+ "sessions": [+ {+ "session_id": "s1",+ "turns": [+ {+ "speaker": "Alice",+ "text": "The project deadline milestone slipped and is now set for Friday.",+ "timestamp": "2026-01-05T10:00:00Z"+ },+ {+ "speaker": "Bob",+ "text": "Understood, I will move the milestone review to Friday morning.",+ "timestamp": "2026-01-05T10:01:00Z"+ }+ ]+ },+ {+ "session_id": "s2",+ "turns": [+ {+ "speaker": "Alice",+ "text": "The budget for an additional headcount was approved by finance.",+ "timestamp": "2026-01-06T09:00:00Z"+ },+ {+ "speaker": "Bob",+ "text": "Great, I will brief the recruiter about the approved headcount budget.",+ "timestamp": "2026-01-06T09:02:00Z"+ }+ ]+ }+ ],+ "qa": [+ {+ "id": "locomo-q1",+ "question": "deadline milestone",+ "answer": "Friday",+ "category": "single-hop",+ "evidence_sessions": ["s1"]+ },+ {+ "id": "locomo-q2",+ "question": "budget headcount recruiter",+ "answer": "approved",+ "category": "single-hop",+ "evidence_sessions": ["s2"]+ },+ {+ "id": "locomo-q3",+ "question": "milestone review Friday",+ "answer": "Friday",+ "category": "temporal-reasoning",+ "evidence_sessions": ["s1"]+ }+ ]+}diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.tsindex 4ac03744..b771c26c 100644--- a/tests/helpers/search-fixtures.ts+++ b/tests/helpers/search-fixtures.ts@@ -81,6 +81,10 @@ export function makeConfig(opts: {rrfK?: number;/** Optional cross-encoder rerank overrides; defaults to disabled. */rerank?: Partial<ResolvedRerankConfig>;+ /** Trigram candidate prefilter; defaults to false (opt-in). */+ trigramPrefilterEnabled?: boolean;+ /** Minimum corpus chunks before the trigram prefilter engages; defaults to 5000. */+ trigramPrefilterMinChunks?: number;}): ResolvedSearchConfig {const baseSemantic: ResolvedEmbeddingConfig = Object.freeze({enabled: false,@@ -135,9 +139,13 @@ export function makeConfig(opts: {selfTuningEnabled: opts.selfTuningEnabled ?? false,chainStopEnabled: false,chainStopScore: 0.8,+ trigramPrefilterEnabled: opts.trigramPrefilterEnabled ?? false,+ trigramPrefilterMinChunks: opts.trigramPrefilterMinChunks ?? 5000,+ trigramPrefilterMaxSelectivity: 0.5,}),rerank: Object.freeze({enabled: false,+ kind: "openai-compat",baseUrl: null,model: null,envKey: null,diff --git a/tests/mcp/brain-tools-parity.test.ts b/tests/mcp/brain-tools-parity.test.tsindex 94e4ab1b..099b2224 100644--- a/tests/mcp/brain-tools-parity.test.ts+++ b/tests/mcp/brain-tools-parity.test.ts@@ -91,6 +91,7 @@ const FROZEN_BRAIN_TOOL_NAMES = ["brain_note","brain_note_history","brain_obligation",+ "brain_observed_use","brain_pinned_context","brain_pre_compact_extract","brain_pre_compress_pack",diff --git a/tests/mcp/mcp.test.ts b/tests/mcp/mcp.test.tsindex bf4c76d6..f7762dcf 100644--- a/tests/mcp/mcp.test.ts+++ b/tests/mcp/mcp.test.ts@@ -222,6 +222,8 @@ describe("tool listing", () => {"brain_context_receipts","brain_event_trace","brain_recall_telemetry",+ // Observed-use verdict feeding recall ranking (Retrieval & Ranking Quality).+ "brain_observed_use","brain_knowledge_gaps","brain_context_presets","brain_pre_compact_extract",@@ -591,7 +593,9 @@ describe("stdio loop", () => {// memory-signal-provenance-lifecycle A3) = 96.// + brain_distill_source (source distillation into atomic claims,// ingestion-import-robustness t_2e2e959f) = 97.- expect(list.result.tools.length).toBe(97);+ // + brain_observed_use (session-end observed-use verdict feeding recall+ // ranking, retrieval-ranking-quality t_65588d8b) = 98.+ expect(list.result.tools.length).toBe(98);});test("returns parse error for invalid JSON", async () => {diff --git a/tests/mcp/procedural-learning-tools.test.ts b/tests/mcp/procedural-learning-tools.test.tsindex 2f5925b3..94b540ec 100644--- a/tests/mcp/procedural-learning-tools.test.ts+++ b/tests/mcp/procedural-learning-tools.test.ts@@ -122,6 +122,20 @@ describe("procedural learning MCP tools", () => {});expect(marked.usedCount).toBeGreaterThanOrEqual(1);+ // Outcome-validated recall (t_703f7b18): record a success and a+ // failure, then confirm the ranked list orders by success rate.+ const outcome = await callTool(server, "brain_procedural_memory", {+ operation: "mark_outcome",+ id: entries[0]!["id"],+ outcome: "success",+ });+ expect(outcome.successCount).toBe(1);+ const ranked = await callTool(server, "brain_procedural_memory", {+ operation: "list",+ ranked: true,+ });+ expect((ranked.entries as unknown[]).length).toBe(entries.length);+const graphRebuild = await callTool(server, "brain_procedural_graph", {operation: "rebuild",});diff --git a/tests/mcp/recall-telemetry-tool.test.ts b/tests/mcp/recall-telemetry-tool.test.tsindex ee9cbcc2..af6cfda2 100644--- a/tests/mcp/recall-telemetry-tool.test.ts+++ b/tests/mcp/recall-telemetry-tool.test.ts@@ -280,3 +280,41 @@ describe("brain_recall_telemetry tool", () => {}});});++describe("brain_observed_use tool (t_65588d8b)", () => {+ test("records verdicts and surfaces them via observed_reuse", async () => {+ const server = new MCPServer({ vault, configPath });+ await initialize(server);++ const rec = await callTool(server, "brain_observed_use", {+ entries: [+ { id: "1", path: "a.md", verdict: "USED" },+ { id: "2", path: "b.md", verdict: "CONTRADICTED" },+ ],+ });+ expect(rec.recorded).toBe(2);++ const reuse = await callTelemetry(server, { operation: "observed_reuse" });+ expect(reuse.total).toBe(2);+ const artifacts = reuse.artifacts as Array<Record<string, unknown>>;+ const a = artifacts.find((x) => x.key === "a.md");+ expect(a?.score).toBe(1);+ const b = artifacts.find((x) => x.key === "b.md");+ expect(b?.score).toBe(0);+ });++ test("rejects an invalid verdict", async () => {+ const server = new MCPServer({ vault, configPath });+ await initialize(server);+ const response = (await server.handleRequest({+ jsonrpc: JSONRPC_VERSION,+ id: 9,+ method: "tools/call",+ params: {+ name: "brain_observed_use",+ arguments: { entries: [{ id: "1", verdict: "MAYBE" }] },+ },+ })) as { error?: { code: number } };+ expect(response.error?.code).toBe(INVALID_PARAMS);+ });+});diff --git a/tests/mcp/removed-tools.test.ts b/tests/mcp/removed-tools.test.tsindex 01829ea5..7f71b04c 100644--- a/tests/mcp/removed-tools.test.ts+++ b/tests/mcp/removed-tools.test.ts@@ -159,5 +159,7 @@ test("the shadow surface is gone: no hidden tools, removed names unlisted", asyn// memory-signal-provenance-lifecycle A3) = 96.// + brain_distill_source (source distillation into atomic claims,// ingestion-import-robustness t_2e2e959f) = 97.- expect(list.result.tools.length).toBe(97);+ // + brain_observed_use (observed-use verdict feeding recall ranking,+ // retrieval-ranking-quality t_65588d8b) = 98.+ expect(list.result.tools.length).toBe(98);});