Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat(brain): semantic entity dedup and cross-encoder rerank (1.23.0) (#120)
Sol Aitkenb9bbcb1622e2
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex e9c7b95a..9cc29cdd 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.22.0",+ "version": "1.23.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 504d17ef..48a021b5 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.22.0",+ "version": "1.23.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","skills": "./skills","hooks": "./hooks/hooks.json",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 23a7423c..ce808161 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,39 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [1.23.0] - 2026-07-05++Two optional precision layers that complete the retrieval-precision loop+started in 1.22.0. Each is off by default with byte-identical output when+unconfigured, and the kernel still calls no LLM.++### Added++- **Proposal-only semantic entity dedup.** With `entity_semantic_dedup_enabled`+ set, `brain_doctor` (and the `doctor` lint surface) surfaces lexical+ entity-name variants ("Google LLC" vs "Google Inc") as PROPOSAL-ONLY+ alias-merge candidates through a deterministic jaccard layer+ (`entity_semantic_dedup_lexical_threshold`, default 0.8). An embedding-cosine+ reader (`entity_semantic_dedup_threshold`, default 0.92) reuses the configured+ embedding provider and is exposed as a library reader for apply plans. The+ deterministic `entityIdentityKey` is never rewritten and no entity is ever+ auto-merged; a human or apply plan owns the actual merge, preserving audit+ friendliness. Optional `identity_type` tagging derives from frontmatter and+ structural signals only (language-agnostic by construction). Registry and+ search behavior is byte-identical until opted in.+- **Optional cross-encoder rerank stage.** A learned final reader step+ re-scores the top-K fused search candidates jointly against the query,+ appended after every heuristic rerank. Off by default+ (`search_rerank_enabled`); when on, it resolves an OpenAI-compatible+ `/rerank` endpoint (`search_rerank_base_url` / `search_rerank_model` /+ `search_rerank_env_key`, or a profile registered via+ `o2b search rerank-provider add`) and re-orders the+ `search_rerank_top_k` candidates by the returned relevance, promoting a+ genuinely-relevant hit the heuristic ranker placed deep. Zero HTTP cost and+ byte-identical ordering when disabled; a request-time endpoint error+ degrades to the heuristic ordering (a `rerank_degraded:` warning, never a+ throw). Most valuable on the `thorough` profile.+## [1.22.0] - 2026-07-05A retrieval-precision and quality-loop pass. Four new surfaces close the@@ -6172,6 +6205,7 @@ plugin config (vault field)`, and exits with a clear- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[1.23.0]: https://github.com/itechmeat/open-second-brain/compare/v1.22.0...v1.23.0[1.22.0]: https://github.com/itechmeat/open-second-brain/compare/v1.21.0...v1.22.0[1.21.0]: https://github.com/itechmeat/open-second-brain/compare/v1.20.0...v1.21.0[1.20.0]: https://github.com/itechmeat/open-second-brain/compare/v1.19.1...v1.20.0diff --git a/README.md b/README.mdindex 662d3074..0f94adaa 100644--- a/README.md+++ b/README.md@@ -8,7 +8,7 @@ Open Second Brain plugs into [Hermes Agent](https://github.com/NousResearch/herm## What is new-Open Second Brain 1.22.0 closes the retrieval quality loop on four surfaces. Packed and recalled items now carry a structural epistemic status (observed, derived, hypothesis, plan, or unknown) plus evidence refs, derived from existing graph metadata so a consuming model can tell a source-backed fact from a conjecture (always on, fields absent when unknown). A thin recall adequacy verdict classifies each recall as sufficient, weak, or insufficient with a recommended action (proceed, re-recall, abstain) and an optional escalate flag, surfaced in recall gate output and context receipts when the caller passes scores (thresholds configurable). A persisted cross-query demand log records each recall and an aggregation reader ranks recurring queries the vault answers poorly, exposed as the brain_knowledge_gaps tool and o2b brain knowledge-gaps (the log is written only when recall gate telemetry is opt-in, queries normalized before append). A unified lessons digest folds positive knowledge and dead-ends into one signed, recency-scored, corroboration-tiered corpus under Brain/lessons.md, regenerated every dream pass and loaded on session start. Every new field is optional and absent by default, so callers that do not opt in see byte-identical output, and the kernel still calls no LLM.+Open Second Brain 1.23.0 adds two optional precision layers that complete the retrieval-precision loop started in 1.22.0, each off by default with byte-identical output when unconfigured. A semantic entity dedup pass surfaces lexical entity-name variants as proposal-only alias-merge candidates when entity_semantic_dedup_enabled is set, via a deterministic jaccard layer with an embedding-cosine reader that reuses the configured embedding provider, and never rewrites the deterministic identity key. An optional cross-encoder rerank stage re-scores the top-K fused search candidates jointly against the query through an OpenAI-compatible endpoint (search_rerank_enabled, off by default), most valuable on the thorough profile, with zero HTTP cost and unchanged ordering when disabled and a graceful degrade to heuristic ordering on endpoint error. The kernel still calls no LLM.## Whydiff --git a/docs/brainstorm/semantic-retrieval-precision/cli-output/claude.md b/docs/brainstorm/semantic-retrieval-precision/cli-output/claude.mdnew file mode 100644index 00000000..d3f4d505--- /dev/null+++ b/docs/brainstorm/semantic-retrieval-precision/cli-output/claude.md@@ -0,0 +1,29 @@+## Variant 1+Approach: The parent lands a single generic `learned/` precision layer under `src/core/search` β one module exposing a provider-resolution + graceful-no-op guard and one abstract "learned stage" contract that both features implement. Entity semantic dedup becomes the first consumer (emitting alias-merge candidates) and the cross-encoder becomes the second (emitting a rescored top-K order), so both cards route through one shared abstraction and one guarded call-site convention.+Trade-offs:+- Maximal DRY on the opt-in/no-op guard and provider validation; one place to audit invariant #2/#3 compliance.+- Couples two stages that live in different subtrees (`brain/entities`/`hygiene` vs `search`) and have genuinely different contracts (proposal-only candidate pairs vs reader-tail rescoring), forcing a lowest-common-denominator interface that fits neither cleanly.+- The child cannot land until the abstract contract is finalized, so the shared module itself becomes the collision seam rather than eliminating it; a mid-stream contract change ripples into both cards.+Complexity: large+Risk: medium++## Variant 2+Approach: Each card independently mirrors an already-proven pattern and touches a different subtree, sharing only one small new seam. The parent adds an entity semantic-dedup detector modeled directly on `hygiene/detectors/dedup.ts` (embedding-cosine over registry entities, `method`-labeled lexical fallback, nominates alias-merge candidates into doctor lints / `registry.ts` alias resolution β never rewrites the deterministic key), and while doing so extracts a thin provider-resolution/no-op helper. The child adds an isolated cross-encoder stage in the `search.ts` reader tail after `rerankByRelevance`, mirroring `openai-compat.ts` fail-closed config validation and importing that same helper.+Trade-offs:+- Lowest collision surface: parent edits live in `brain/entities` + `brain/hygiene`, child edits in `search`; the only shared new artifact is the small provider helper the parent lands and the child extends β a clean already-merged seam.+- Each stage keeps a contract shaped to its own job (candidate pairs vs rescored order), honoring KISS and avoiding a premature super-abstraction.+- Two guarded call sites instead of one, so invariant #2 byte-identical verification must be checked in two places rather than centrally.+Complexity: medium+Risk: low++## Variant 3+Approach: Extend the existing named-provider registry (`embeddings/registry.ts`, `Brain/search/embedding-providers.json`) with a `role` field so one enriched registry resolves both the embedding used by the dedup pass and the cross-encoder rerank endpoint. The parent adds role-aware resolution plus the entity semantic pass; the child registers the cross-encoder as another role and consumes it in the reader tail.+Trade-offs:+- Single unified config surface for every learned-model endpoint; operators manage one registry file.+- Mutates a shared file-first schema that is already persisted and sorted-on-write, risking backward-compat and Syncthing-peer determinism (invariants #5/#7) for records that currently assume embedding-only semantics.+- The registry becomes a hot collision point both cards must edit, inverting the "child extends an already-merged seam" goal; schema migration and fail-soft load paths must be re-validated for existing vaults.+Complexity: large+Risk: high++## Recommended: Variant 2+Variant 2 best fits OSB's stated conventions. The two cards naturally live in different subtrees, so mirroring the two proven patterns already in the repo β the `hygiene/detectors/dedup.ts` cosine-with-labeled-lexical-fallback for the parent and the `openai-compat.ts` fail-closed validation for the child β reuses established discipline without inventing a new abstraction, satisfying "reuse, do not duplicate" while respecting KISS. It directly realizes the intended collision structure from the scope: the parent establishes a small provider-resolution/no-op helper as its seam, the child extends that already-merged helper, and their substantive edits never overlap, so the one-PR-many-cards-one-at-a-time cadence (invariant #6) holds without ordering gymnastics. It also keeps each stage's contract honest β the parent stays proposal-only feeding candidates into doctor/registry alias resolution (invariant #8) while the deterministic `entityIdentityKey` is untouched, and the child stays a graceful no-op / fail-open final reader stage (invariants #2/#3) β whereas Variant 1's shared contract would blur those two very different shapes and Variant 3 would put both cards in conflict over a persisted, determinism-sensitive registry schema. The only cost, two guarded call sites instead of one, is the cheapest of the three risks to verify and is exactly the "single guarded call site whose off-branch is unchanged" property invariant #2 calls easiest to check β applied twice.diff --git a/docs/brainstorm/semantic-retrieval-precision/cli-output/prompt.md b/docs/brainstorm/semantic-retrieval-precision/cli-output/prompt.mdnew file mode 100644index 00000000..4a72247c--- /dev/null+++ b/docs/brainstorm/semantic-retrieval-precision/cli-output/prompt.md@@ -0,0 +1,226 @@+# Architecture consultation - semantic-retrieval-precision++You are a senior software architect advising on the design of a focused 2-card feature scope+for an existing TypeScript project. Produce **exactly 3 distinct architectural variants** and+then **exactly one recommendation**. Variants and recommendation only - no code, nothing+outside those sections.++## Output format (follow EXACTLY)++```+## Variant 1+Approach: <2-3 sentences>+Trade-offs:+- <bullet>+Complexity: small | medium | large+Risk: low | medium | high++## Variant 2+Approach: <2-3 sentences>+Trade-offs:+- <bullet>+Complexity: small | medium | large+Risk: low | medium | high++## Variant 3+Approach: <2-3 sentences>+Trade-offs:+- <bullet>+Complexity: small | medium | large+Risk: low | medium | high++## Recommended: Variant N+<rationale - why this variant over the others, in the context of the project conventions below>+```++Do not include code, pseudo-code, or file listings. No preamble, no closing remarks.++## Project++- **Name**: Open Second Brain (package `open-second-brain`, v1.22.0).+- **Language/runtime**: TypeScript (strict), runs on Bun. Obsidian-native second-brain memory+ layer for AI agents. Plain Markdown vault under `Brain/`. Ships a deterministic CLI (`o2b`),+ an MCP server, and native plugins (Hermes/Claude Code/Codex/OpenClaw/opencode/Grok).+- **Invariants you MUST honor** (load-bearing conventions; violating any breaks the release):+ 1. **The kernel calls no LLM.** Every decision in search/recall/ranking/entity logic is+ deterministic. Optional learned-model surfaces (cross-encoder rerank, an embedding provider+ for candidate scoring) are external service calls, not in-kernel models - they must be+ opt-in and a graceful no-op / fail-open when unconfigured.+ 2. **Default-off switches are bit-identical to today.** Any ranking/score/output change must+ produce byte-identical results when the new path is off. The easiest property to verify is+ a single guarded call site whose off-branch is unchanged, and optional fields absent by+ default rather than present-with-empty.+ 3. **Telemetry is fail-open** via `emitGatedTelemetry` - never throws into the hot path.+ 4. **Language-agnostic.** No hardcoded natural-language word lists (stopwords, keywords,+ negations, company-suffix stripping) in any language. Use structural signals, frontmatter+ fields, corpus document frequency / IDF, or explicit agent/LLM extraction.+ 5. **Syncthing-peer determinism.** Anything written to the vault is deterministic across+ machines (stable sort keys, stable slugs, no wall-clock-only tiebreaks).+ 6. **One release = one PR = many cards, driven ONE AT A TIME on a shared branch.** Each card+ builds on commits the previous cards already landed. Non-conflict is achieved by ordering+ the real collision pairs so the later card extends an already-merged seam.+ 7. **Markdown-first / file-first persistence.** New durable state is plain `.md` / `.json`+ under `Brain/`, never a hidden black box; a registered provider profile stores env-key+ NAMES only, never secret values.+ 8. **Proposal-only writes that touch identity.** Anything that could collapse two distinct+ records (entity alias-merge, fact conflict resolution) must SURFACE candidates and never+ silently rewrite; the deterministic core owns the actual merge.+- **Engineering rules**: SOLID / KISS / DRY; no misleading fallbacks (a lexical heuristic must+ never be passed off as semantic - it must be labeled with its `method`); no hardcoding;+ English-only strings in code while logic stays abstract/multi-language.++## Scope: this release ships exactly TWO cards together on one shared branch+`feat/semantic-retrieval-precision`, driven one at a time. This scope completes the previously+half-shipped retrieval-precision-quality-loop: Card A (cross-encoder rerank) was deferred from+v1.22.0 because its dependency (semantic entity dedup) was out of scope then. Both remaining p4+cards are dependency-closed now and share the same upstream provenance and direction: learned-model+retrieval precision that PRESERVES the deterministic core by being opt-in / proposal-only.++### Card 1 (parent) - t_47fd9523 - Semantic (embedding cosine) entity dedup++OSB's entity dedup is purely deterministic string-normalization: `normalizeEntityName` (NFC,+lowercase, collapse whitespace) + `entityIdentityKey` (`<category>:<name>`) in+`src/core/brain/entities/canonical.ts`. Add an OPTIONAL semantic dedup pass (embedding cosine+similarity) that surfaces lexical variants the deterministic key cannot catch (e.g. "Google LLC"+vs "Google Inc", "OpenAI" vs "Open AI") as alias-merge CANDIDATES, plus optional identity_type+tagging and evolution-chain tracking.++- **Why useful**: OSB ingests entities from multiple sources/agents that name the same+ real-world entity differently; the deterministic key silently treats these as distinct records,+ fragmenting the knowledge graph. A cosine-similarity candidate pass (surfaced to doctor lints /+ registry alias resolution, NOT auto-merged) would catch these near-duplicates while preserving+ OSB's deterministic, LLM-free, audit-friendly core (semantic step is opt-in and only proposes,+ never silently rewrites).+- **Constraint**: keep semantic merge opt-in and proposal-only to preserve deterministic-core+ guarantees; reuse the EXISTING embedding provider+ (`src/core/search/embeddings/local-provider.ts`, `openai-compat.ts`, the `provider.ts`+ abstraction and `registry.ts`) rather than a new dependency.+- **Existing seams that already do adjacent work**: `src/core/brain/hygiene/detectors/dedup.ts`+ ALREADY runs an embedding-cosine layer over PREFERENCES (threshold 0.97) with a clearly-labeled+ `method: "lexical"` jaccard fallback, and only nominates pairs (merging happens through the+ hygiene apply plan). That detector is the proven pattern to mirror for ENTITIES. The canonical+ kernel (`canonical.ts`) is consumed by the registry duplicate-refusal/alias resolution+ (`registry.ts`), doctor lints, search alias expansion (`entity-alias.ts`), and the+ fact-extraction router.++### Card 2 (child) - t_110867f5 - Optional pluggable cross-encoder rerank stage++OSB has hybrid search (rank-fusion of semantic + keyword in `src/core/search/fusion.ts`, scored+in `ranker.ts`) and several HEURISTIC rerank stages (relevance rerank `enrich.ts:130`+`rerankByRelevance`, entity-match boost, MMR diversity `mmr.ts`, usage/activation decay) but no+learned cross-encoder re-scoring pass. Add an OPTIONAL, opt-in cross-encoder rerank stage+(OpenAI-compatible / pluggable endpoint) that re-scores the top-K fused candidates as a final+reader step.++- **Why useful**: a learned cross-encoder re-ranks query/document pairs jointly, improving+ precision-at-top beyond what heuristic scoring achieves - most valuable for the "thorough"+ search profile where recall quality matters most. Making it opt-in and endpoint-pluggable keeps+ OSB's deterministic core intact (default behavior unchanged when no rerank endpoint is+ configured).+- **Constraint**: mirror the EXISTING embedding-provider config pattern (key + endpoint + env-key+ registry, graceful no-op when unset) so the stage is zero-cost when disabled. On endpoint error+ it degrades gracefully to the heuristic-ordered input and logs via `emitGatedTelemetry`+ (fail-open), never throwing into the hot path.+- **Where it lands**: the reader tail in `src/core/search/search.ts` (~lines 838-846), appended+ AFTER the existing heuristic reranks (MMR `mmrRerank`, then `rerankByRelevance`), as an isolated+ final stage. Config resolves through `resolveSearchConfig` in `src/core/search/index.ts`.++### Dependency between the two cards+t_110867f5 is the CHILD of t_47fd9523 (the parent's semantic-dedup pass establishes the+"learned-model retrieval precision that stays opt-in/proposal-only" seam and the provider-reuse+pattern; the child's cross-encoder re-uses the same provider-abstraction discipline). Drive order+is PARENT then CHILD: t_47fd9523 -> t_110867f5.++## Task bodies (verbatim)++### t_47fd9523 (parent)+[upstream:HyAtlas-Memory] Add semantic (embedding cosine) entity dedup to complement+deterministic identity keys.+- Source: HyAtlas-Memory v2.0.0. OSB's entity dedup is purely deterministic string-normalization.+ Add an optional semantic dedup pass (embedding cosine similarity) surfacing lexical variants as+ alias-merge candidates, plus optional identity_type tagging and evolution-chain tracking.+- Verdict in OSB: present_weaker. Codegraph hints: `src/core/brain/entities/canonical.ts`+ (normalizeEntityName, entityIdentityKey - deterministic-only; consumed by registry+ duplicate-refusal/alias resolution, doctor lints, search alias expansion, fact-extraction+ router). No cosine/embedding-similarity dedup path - not found; OSB dedup is+ string-normalization only.+- Notes: keep semantic merge opt-in and proposal-only to preserve deterministic-core guarantees;+ reuse the existing embedding provider rather than a new dependency.++### t_110867f5 (child)+[upstream:HyAtlas-Memory] Optional pluggable cross-encoder rerank stage for the search reader.+- Source: HyAtlas-Memory v2.0.0. OSB has hybrid search + heuristic reranks but no learned+ cross-encoder re-scoring. Add an optional, opt-in cross-encoder rerank stage+ (OpenAI-compatible / pluggable endpoint) that re-scores the top-K fused candidates as a final+ reader step.+- Verdict in OSB: present_weaker. Codegraph hints: hybrid/fusion at `src/core/search/fusion.ts`,+ `src/core/search/ranker.ts`, `src/core/search/types.ts`; heuristic reranks at+ `src/core/search/enrich.ts` (rerankByRelevance), `entities.ts`, `mmr.ts`, `activation/decay.ts`.+ No learned/cross-encoder rerank model - not found; all rerank is heuristic.+- Notes: mirror the existing embedding-provider config pattern (key + endpoint, graceful no-op+ when unset) so the stage is zero-cost when disabled.++## Recent git log (for cadence and seam conventions)++```+b8d709e fix: keep full MCP status output and normalize codegraph paths (#116)+a98bed1 feat(brain): retrieval precision and quality loop (v1.22.0) (#118)+4281605 feat(brain): integrity & safety hardening suite (1.21.0) (#115)+33b4fba feat: recall precision, coverage, and provenance hardening (v1.18.0) (#110)+35b824e feat: Recall & Working-Memory Quality Suite (v1.10.0) (#99)+```++Each suite lands as ONE PR with many cards driven one at a time on a shared branch; the prior+`retrieval-precision-quality-loop` suite (v1.22.0, #118) shipped 5 cards and EXPLICITLY deferred+the cross-encoder card (t_110867f5) as Card A because its parent (semantic entity dedup,+t_47fd9523) was out of scope. This scope completes that half-shipped loop.++## Related files / proven patterns to reuse (not duplicate)++- `src/core/brain/entities/canonical.ts` - the ONE place entity identity is computed+ (`normalizeEntityName`, `entityIdentityKey`); consumed by registry, doctor, search alias,+ fact-extraction. The semantic-dedup pass must NOT alter the deterministic key - it proposes+ candidates that feed INTO alias resolution.+- `src/core/brain/entities/registry.ts` - canonical entity registry; `upsertEntity` resolves+ through names AND aliases before creating, refuses duplicates at the write seam; doctor lints+ catch hand-edited/sync duplicates. Candidate-merge proposals surface here.+- `src/core/brain/hygiene/detectors/dedup.ts` - the PROVEN pattern: embedding-cosine layer over+ preferences (threshold 0.97) + clearly-labeled `method: "lexical"` jaccard fallback; nominates+ pairs only, merging via the hygiene apply plan. Card 1 mirrors this for ENTITIES.+- `src/core/search/embeddings/provider.ts` - the `EmbeddingProvider` abstraction (`makeProvider`).+- `src/core/search/embeddings/local-provider.ts` - dependency-free deterministic hashing-trick+ embedder; the privacy-first no-cloud path.+- `src/core/search/embeddings/openai-compat.ts` - OpenAI-compatible `/v1/embeddings` provider with+ retry/timeout/semaphore; the recommended provider for semantic depth. Its config-validation+ (fail-closed when enabled-but-missing base_url/model/key) is the pattern for the cross-encoder.+- `src/core/search/embeddings/registry.ts` - named-provider registry persisted to+ `Brain/search/embedding-providers.json` (env-key NAMES only, fail-soft load, sorted on write).+- `src/core/search/fusion.ts` - rank-fusion (linear + RRF); the cross-encoder runs AFTER fusion.+- `src/core/search/ranker.ts` - pure scoring function; heuristic reranks compose after it.+- `src/core/search/enrich.ts` - `rerankByRelevance` (deterministic, stable); the cross-encoder+ appends after it in the reader tail.+- `src/core/search/search.ts` - the reader tail (~838-846): MMR, then relevance rerank, then+ property/visibility/agent-scope filters. The cross-encoder is a new isolated final stage here.+- `src/core/search/index.ts` - `resolveSearchConfig` (config + env + validation); the single place+ new `search_rerank_*` keys resolve.++## Constraints & conventions to weigh in the variants++- Both cards must be default-off with byte-identical off-branches (invariant #2). The parent's+ semantic-dedup pass must be PROPOSAL-ONLY (invariant #8): it surfaces alias-merge candidates to+ doctor lints / registry alias resolution and never silently rewrites identity. The child's+ cross-encoder is a graceful no-op when unconfigured and fail-open on endpoint error.+- Reuse, do not duplicate: the embedding-provider abstraction and the hygiene-detector dedup+ pattern are proven. A new embedding dependency or a parallel provider registry that diverges+ from `registry.ts` would violate DRY and the established file-first registry discipline.+- The two cards share a collision surface: BOTH touch the "learned model as an opt-in,+ proposal/no-op layer over the deterministic core" pattern and BOTH reuse the embedding-provider+ abstraction. The variants must address how to order/structure them so the child extends the+ parent's already-merged seam rather than colliding (e.g. parent establishes an entity-candidate+ module + provider-reuse helper; child adds a search-rerank module that reuses the same+ provider-validation discipline).+- Language-agnostic (invariant #4): the semantic-dedup cosine uses the embedder, NOT a+ company-suffix / legal-form word list. identity_type tagging must derive from frontmatter /+ structural signals, not NL keywords.++Now produce the three variants and the single recommendation.diff --git a/docs/brainstorm/semantic-retrieval-precision/design.md b/docs/brainstorm/semantic-retrieval-precision/design.mdnew file mode 100644index 00000000..fb1b71a6--- /dev/null+++ b/docs/brainstorm/semantic-retrieval-precision/design.md@@ -0,0 +1,161 @@+# Design - semantic-retrieval-precision++Branch: `feat/semantic-retrieval-precision`+Slug: `semantic-retrieval-precision`+Cards: parent `t_47fd9523` (semantic entity dedup), child `t_110867f5` (cross-encoder rerank)++## Problem++The v1.22.0 `retrieval-precision-quality-loop` suite shipped five cards but explicitly deferred+Card A (cross-encoder rerank, `t_110867f5`) because its parent β semantic entity dedup+(`t_47fd9523`) β was out of scope. This release completes that half-shipped loop by landing the+two remaining p4 cards from the same upstream (HyAtlas-Memory), one direction: learned-model+retrieval precision that PRESERVES OSB's deterministic core by being opt-in / proposal-only.++Two concrete gaps remain:++1. **Entity identity is purely lexical.** OSB's entity dedup is deterministic+ string-normalization only: `normalizeEntityName` (NFC, lowercase, collapse whitespace) ++ `entityIdentityKey` (`<category>:<name>`) in `src/core/brain/entities/canonical.ts`. OSB+ ingests entities from multiple sources and agents that name the same real-world entity+ differently ("Google LLC" vs "Google Inc", "OpenAI" vs "Open AI"); the deterministic key+ silently treats these as distinct records, fragmenting the knowledge graph. No cosine /+ embedding-similarity dedup path exists, so near-duplicate entities are never surfaced.+2. **No learned re-scoring pass in search.** OSB has hybrid rank-fusion (`fusion.ts` scored in+ `ranker.ts`) and several heuristic rerank stages (relevance `enrich.ts`, entity boost, MMR+ `mmr.ts`, usage/activation decay) but no learned cross-encoder that jointly re-scores+ query/document pairs. Precision-at-top is therefore bounded by what lexical + cosine+ heuristics can express β most keenly on the "thorough" search profile where recall quality+ matters most.++## Scope++Both cards ship in this release, driven one at a time on the shared branch in dependency order so+the child extends an already-merged seam rather than colliding:++- **parent** `t_47fd9523` - optional, opt-in semantic (embedding cosine) entity dedup that+ surfaces lexical variants as alias-merge CANDIDATES (into doctor lints / registry alias+ resolution), never silently rewriting the deterministic identity key; optional `identity_type`+ tagging and evolution-chain tracking. Mirrors the proven+ `src/core/brain/hygiene/detectors/dedup.ts` pattern (cosine layer + clearly-labeled lexical+ fallback, nominates pairs only) for ENTITIES.+- **child** `t_110867f5` - optional, opt-in cross-encoder rerank stage (OpenAI-compatible /+ pluggable endpoint) that re-scores the top-K fused candidates as a final reader step, appended+ after the existing heuristic reranks in `search.ts`. Mirrors the `openai-compat.ts`+ fail-closed-validation + graceful-no-op pattern.++**Drive order:** parent β child (`t_47fd9523` β `t_110867f5`). The parent establishes a thin+provider-resolution/no-op helper that the child reuses; their substantive edits live in different+subtrees (`brain/entities` + `brain/hygiene` vs `search`) and never overlap.++## Out of scope++- A learned model living inside the kernel. The semantic-dedup cosine uses the existing embedding+ provider; the cross-encoder is an opt-in external service call. Both are graceful no-ops when+ unconfigured and the kernel still calls no LLM.+- Any change to default ranking/score/output when a feature is off. Each card is a single guarded+ call site whose off-branch is byte-identical to today.+- Auto-merging entities. The semantic-dedup pass is PROPOSAL-ONLY: it surfaces alias-merge+ candidates; the deterministic core (registry `upsertEntity` alias resolution + doctor lints)+ owns the actual merge, preserving audit-friendliness.+- Company-suffix / legal-form stripping or any natural-language word list for entity similarity.+ The cosine uses the embedder; `identity_type` derives from frontmatter / structural signals+ only (language-agnostic, invariant #4).+- A new embedding dependency or a parallel provider registry that diverges from+ `embeddings/registry.ts`. The existing `EmbeddingProvider` abstraction is reused.+- A web dashboard. New surfaces ship as CLI / MCP readers or plain Markdown / JSON under `Brain/`.+- The other (already-shipped) cards of the `retrieval-precision-quality-loop` suite.++## Chosen approach++**Variant 2 - each card independently mirrors an already-proven pattern and touches a different+subtree, sharing only one small new seam (a thin provider-resolution/no-op helper the parent lands+and the child extends).**++Both cards default-off with byte-identical off-branches (a single guarded call site each β the+easiest property to verify per invariant #2). The parent's semantic-dedup pass is proposal-only+(invariant #8): it feeds alias-merge candidates into doctor lints / registry alias resolution and+never touches the deterministic `entityIdentityKey`. The child's cross-encoder is a graceful+no-op when unconfigured and fail-open on endpoint error. This is the only variant whose collision+structure is resolved by the cheapest sufficient mechanism β the parent lands a small helper as its+seam and the child extends the already-merged helper, while their substantive edits never overlap β+so the one-PR-many-cards-one-at-a-time cadence holds without a premature super-abstraction+(Variant 1) or a shared persisted registry schema that both cards must edit (Variant 3).++- **parent β helper**: the parent extracts a thin provider-resolution/no-op guard (mirroring+ `openai-compat.ts` validation discipline) while landing the entity semantic-dedup detector.+- **child β helper**: the child's cross-encoder imports the parent's helper for its+ provider-resolution, then adds its isolated final reader stage.++Telemetry on the child routes through `emitGatedTelemetry` and stays fail-open. The kernel calls+no LLM.++## Design decisions++### parent (t_47fd9523) - semantic entity dedup (proposal-only candidate surfacing)++A new entity semantic-dedup pass modeled directly on the proven+`src/core/brain/hygiene/detectors/dedup.ts`: an embedding-cosine layer over the canonical entity+registry (configurable threshold, default high to surface only near-duplicates) with a+clearly-labeled `method: "lexical"` fallback (the shared `findMergeCandidates` jaccard detector+over entity names) when vectors are unavailable. It NOMINATES alias-merge candidate pairs only β+never rewrites the deterministic `entityIdentityKey` in `canonical.ts`. Candidates surface into the+existing doctor-lint / registry alias-resolution seam (`registry.ts` `upsertEntity` already resolves+through names AND aliases), so a human-or-apply plan decides the merge, preserving audit-friendliness.++Optional `identity_type` tagging (e.g. person / org / concept / product) is derived from+frontmatter / structural signals, NOT a natural-language word list β language-agnostic by+construction (invariant #4). Optional evolution-chain tracking records the alias-merge history so+a later operator can see how a name evolved. Reuses the EXISTING embedding provider+(`local-provider.ts` / `openai-compat.ts` via `makeProvider`) rather than a new dependency. While+landing this, the parent extracts a thin provider-resolution/no-op helper (mirroring+`openai-compat.ts`'s fail-closed-validation + graceful-no-op discipline) that the child reuses.++Config surface resolves through the existing config path with default-off keys+(`entity_semantic_dedup_*`); the detector is off by default, so registry/search behavior is+byte-identical until opted in.++### child (t_110867f5) - cross-encoder rerank (search-side, no vault writes)++A `rerank/` module under `src/core/search` with an OpenAI-compatible cross-encoder provider+(base_url + model + env-resolved api key, re-scoring top-K query/doc pairs), mirroring the+`openai-compat.ts` fail-closed-validation + retry/timeout discipline and reusing the parent's+provider-resolution helper. The stage re-scores the top-K fused candidates as a final reader step+appended after the existing heuristic reranks in `search.ts` (~after `rerankByRelevance`). Config+surface (`search_rerank_*` keys: enabled, model, base_url, env_key, top_k, min_score) resolves+through `resolveSearchConfig` (config + env + validation), and is most valuable on the "thorough"+search profile. When disabled (the default), the stage returns its input unchanged β bit-identical+ordering, zero HTTP cost. On endpoint error it degrades gracefully to the heuristic-ordered input+and logs via `emitGatedTelemetry` (fail-open), never throwing into the hot path.++## File changes (per card, indicative)++- **parent**: new `src/core/brain/entities/semantic-dedup.ts` (cosine + labeled lexical fallback,+ nominates alias-merge candidate pairs); optional `identity_type` derivation + evolution-chain+ tracking helpers; a thin `src/core/search/embeddings/provider-resolve.ts` (or inline helper)+ provider-resolution/no-op guard; default-off config keys threaded through the existing config+ path; candidate surfacing wired into doctor lints / `registry.ts` alias resolution behind a+ feature flag; tests under the module's sibling `.test.ts`.+- **child**: new `src/core/search/rerank/cross-encoder.ts` (the rerank provider: OpenAI-compatible+ endpoint, base_url + model + env-resolved api key, re-scores top-K query/doc pairs); config keys+ `search_rerank_*` via `resolveSearchConfig`; one guarded call site in the reader tail of+ `search.ts` (after `rerankByRelevance`); tests under the module's sibling `.test.ts`.++## Risks++- **Per-card bit-identity (two guarded call sites).** Mitigated by each card having exactly one+ guarded call site whose off-branch returns its input unchanged / emits no candidates; acceptance+ tests assert off-branch output equality against the pre-feature baseline.+- **Semantic-dedup false-positive candidates.** Mitigated by a high default cosine threshold+ (surfacing only near-duplicates) AND proposal-only semantics β a candidate never silently merges;+ the deterministic core + a human/apply plan own the actual merge. The lexical fallback is always+ labeled with its `method` so a report never passes lexical similarity off as semantic.+- **Cross-encoder latency/availability leaking into the hot path.** Mitigated by default-off, a+ strict timeout, and fail-open degradation to heuristic-ordered input on endpoint error; zero HTTP+ cost when unconfigured.+- **Provider-reuse drift between the two cards.** Mitigated by the parent landing the+ provider-resolution helper as its seam and the child extending the already-merged helper (single+ validation-discipline source), rather than each card re-implementing provider resolution.+- **Syncthing-peer determinism (evolution-chain / candidate files).** Mitigated by stable sort+ keys, stable slugs, and deterministic candidate emission (no wall-clock-only tiebreaks) in any+ vault write.diff --git a/docs/brainstorm/semantic-retrieval-precision/plan.md b/docs/brainstorm/semantic-retrieval-precision/plan.mdnew file mode 100644index 00000000..0c6b2211--- /dev/null+++ b/docs/brainstorm/semantic-retrieval-precision/plan.md@@ -0,0 +1,104 @@+# Plan - semantic-retrieval-precision++Branch: `feat/semantic-retrieval-precision`+Slug: `semantic-retrieval-precision`+Combined design: `docs/brainstorm/semantic-retrieval-precision/design.md`+Drive order: **parent β child** (`t_47fd9523` β `t_110867f5`) in dependency order; the parent+lands a thin provider-resolution/no-op helper that the child reuses. Cards are driven ONE AT A+TIME on the shared branch. Each worker builds on the commits the previously-driven in-scope+cards already landed and must not duplicate or conflict with sibling tasks.++Global conventions for every card:++- TDD: write the failing test first, then implement until it passes. Run `bun run validate`+ (typecheck + lint + test) before claiming the card done.+- Default-off switch per card (one config key), off-branch byte-identical to today.+- Telemetry via `emitGatedTelemetry`, fail-open.+- Kernel calls no LLM. Both the semantic-dedup cosine and the cross-encoder reuse the existing+ embedding-provider abstraction / an external endpoint, and are graceful no-ops when unconfigured.+- No TypeScript cast crutches; build correct shapes with conditional spreads / narrowing.+- English-only strings in code; logic is language-agnostic (no hardcoded NL word lists).+- Proposal-only for identity-affecting writes (parent): surfaces candidates, never silently+ rewrites the deterministic `entityIdentityKey`.++In-scope cards shipping together in this one release:++| id | card | one-line title |+|---|---|---|+| `t_47fd9523` | parent | Semantic (embedding cosine) entity dedup surfacing lexical variants as alias-merge candidates |+| `t_110867f5` | child | Optional pluggable cross-encoder rerank stage for the search reader |++---++## parent (t_47fd9523) - Semantic (embedding cosine) entity dedup++### Files+- `src/core/brain/entities/semantic-dedup.ts` - the entity semantic-dedup pass modeled on+ `src/core/brain/hygiene/detectors/dedup.ts`: an embedding-cosine layer over the canonical entity+ registry (configurable threshold, default high to surface only near-duplicates) plus a+ clearly-labeled `method: "lexical"` fallback (the shared `findMergeCandidates` jaccard detector+ over entity names) when vectors are unavailable. NOMINATES alias-merge candidate pairs only β+ never rewrites `entityIdentityKey` in `canonical.ts`.+- optional helpers for `identity_type` tagging (derived from frontmatter / structural signals,+ not NL keywords) and evolution-chain tracking of alias-merge history.+- a thin provider-resolution/no-op helper (e.g.+ `src/core/search/embeddings/provider-resolve.ts`) mirroring `openai-compat.ts`+ fail-closed-validation + graceful-no-op discipline β THIS IS THE SEAM THE CHILD REUSES.+- default-off config keys (`entity_semantic_dedup_*`: enabled, threshold, lexical_threshold) via+ the existing config path.+- candidate surfacing wired into doctor lints / `registry.ts` alias resolution behind the feature+ flag.+- tests under the module's sibling `.test.ts` (follow repo convention).++### Acceptance (a passing test)+- Disabled (default): the canonical registry, doctor lints, and search behavior are byte-identical+ to the pre-feature baseline; no candidate is emitted.+- Enabled + embedding provider available: two entities with the same deterministic key as nothing+ but lexical variants ("Google LLC" vs "Google Inc") are surfaced as an alias-merge CANDIDATE+ pair with `method: "embedding"`; the deterministic `entityIdentityKey` is NOT rewritten (assert+ the key is unchanged after the pass).+- Enabled + no vectors available: the lexical fallback surfaces a candidate pair with+ `method: "lexical"` β clearly labeled, never passed off as semantic.+- Proposal-only: a candidate never auto-merges; it only feeds into the existing doctor-lint /+ registry alias-resolution seam (assert no entity file is rewritten by the pass).+- `identity_type` (when present) derives from frontmatter/structural signals, not NL keywords;+ absent by default.+- Cosine ranking / candidate emission is deterministic across two runs on the same registry+ (Syncthing-peer safe; stable sort keys, no wall-clock-only tiebreaks).++### Depends on+- None. Lands first (establishes the provider-resolution/no-op helper seam + the entity+ candidate-surfacing pattern). Must not touch the deterministic `canonical.ts` key computation,+ the search reader tail, or any `src/core/search/rerank/*` path.++---++## child (t_110867f5) - Optional pluggable cross-encoder rerank stage++### Files+- `src/core/search/rerank/cross-encoder.ts` - the rerank provider (OpenAI-compatible endpoint:+ base_url + model + env-resolved api key, re-scores top-K query/doc pairs). Mirrors+ `openai-compat.ts` fail-closed-validation + retry/timeout discipline; REUSES the parent's+ provider-resolution helper (do not re-implement provider resolution).+- config keys threaded through `resolveSearchConfig` in `src/core/search/index.ts`:+ `search_rerank_enabled` (default false), `search_rerank_model`, `search_rerank_base_url`,+ `search_rerank_env_key`, `search_rerank_top_k`, `search_rerank_min_score`, with env parity ++ validation.+- one guarded call site in the reader tail of `src/core/search/search.ts` (appended AFTER the+ existing heuristic reranks β after `mmrRerank` and `rerankByRelevance`, ~lines 838-846).+- tests under the module's sibling `.test.ts`.++### Acceptance (a passing test)+- Disabled (default): the reader tail output is byte-identical to the pre-feature baseline for the+ same query/results (snapshot/diff equality asserted in a test).+- Enabled + endpoint unconfigured: validation fails closed with a clear error (mirrors+ `openai-compat.ts` when semantic is enabled without base_url/model).+- Enabled + endpoint returns scores: top-K are re-ordered by the cross-encoder scores; a doc the+ heuristic ranker placed 3rd but the cross-encoder scored highest lands 1st.+- Enabled + endpoint errors: stage degrades to the heuristic-ordered input (no throw into the hot+ path) and emits one fail-open telemetry record via `emitGatedTelemetry`.++### Depends on+- parent `t_47fd9523` landed (so the provider-resolution/no-op helper seam exists and the branch+ is stable at the search reader tail). Must not touch `brain/entities`, `canonical.ts`,+ `brain/hygiene`, or any vault persistence β pure search-side, no vault writes.diff --git a/docs/brainstorm/semantic-retrieval-precision/variants.md b/docs/brainstorm/semantic-retrieval-precision/variants.mdnew file mode 100644index 00000000..098bf36e--- /dev/null+++ b/docs/brainstorm/semantic-retrieval-precision/variants.md@@ -0,0 +1,104 @@+# Architecture variants - semantic-retrieval-precision++Branch: `feat/semantic-retrieval-precision`. Consultant output is preserved verbatim in+`cli-output/claude.md`; this file restates the three variants verbatim and records the final+decision and rationale.++## Variant 1+Approach: The parent lands a single generic `learned/` precision layer under `src/core/search` β+one module exposing a provider-resolution + graceful-no-op guard and one abstract "learned stage"+contract that both features implement. Entity semantic dedup becomes the first consumer (emitting+alias-merge candidates) and the cross-encoder becomes the second (emitting a rescored top-K+order), so both cards route through one shared abstraction and one guarded call-site convention.+Trade-offs:+- Maximal DRY on the opt-in/no-op guard and provider validation; one place to audit invariant+ #2/#3 compliance.+- Couples two stages that live in different subtrees (`brain/entities`/`hygiene` vs `search`) and+ have genuinely different contracts (proposal-only candidate pairs vs reader-tail rescoring),+ forcing a lowest-common-denominator interface that fits neither cleanly.+- The child cannot land until the abstract contract is finalized, so the shared module itself+ becomes the collision seam rather than eliminating it; a mid-stream contract change ripples into+ both cards.+Complexity: large+Risk: medium++## Variant 2+Approach: Each card independently mirrors an already-proven pattern and touches a different+subtree, sharing only one small new seam. The parent adds an entity semantic-dedup detector+modeled directly on `hygiene/detectors/dedup.ts` (embedding-cosine over registry entities,+`method`-labeled lexical fallback, nominates alias-merge candidates into doctor lints /+`registry.ts` alias resolution β never rewrites the deterministic key), and while doing so+extracts a thin provider-resolution/no-op helper. The child adds an isolated cross-encoder stage+in the `search.ts` reader tail after `rerankByRelevance`, mirroring `openai-compat.ts`+fail-closed config validation and importing that same helper.+Trade-offs:+- Lowest collision surface: parent edits live in `brain/entities` + `brain/hygiene`, child edits+ in `search`; the only shared new artifact is the small provider helper the parent lands and the+ child extends β a clean already-merged seam.+- Each stage keeps a contract shaped to its own job (candidate pairs vs rescored order), honoring+ KISS and avoiding a premature super-abstraction.+- Two guarded call sites instead of one, so invariant #2 byte-identical verification must be+ checked in two places rather than centrally.+Complexity: medium+Risk: low++## Variant 3+Approach: Extend the existing named-provider registry (`embeddings/registry.ts`,+`Brain/search/embedding-providers.json`) with a `role` field so one enriched registry resolves+both the embedding used by the dedup pass and the cross-encoder rerank endpoint. The parent adds+role-aware resolution plus the entity semantic pass; the child registers the cross-encoder as+another role and consumes it in the reader tail.+Trade-offs:+- Single unified config surface for every learned-model endpoint; operators manage one registry+ file.+- Mutates a shared file-first schema that is already persisted and sorted-on-write, risking+ backward-compat and Syncthing-peer determinism (invariants #5/#7) for records that currently+ assume embedding-only semantics.+- The registry becomes a hot collision point both cards must edit, inverting the "child extends an+ already-merged seam" goal; schema migration and fail-soft load paths must be re-validated for+ existing vaults.+Complexity: large+Risk: high++## Recommended: Variant 2+Variant 2 best fits OSB's stated conventions. The two cards naturally live in different subtrees,+so mirroring the two proven patterns already in the repo β the `hygiene/detectors/dedup.ts`+cosine-with-labeled-lexical-fallback for the parent and the `openai-compat.ts` fail-closed+validation for the child β reuses established discipline without inventing a new abstraction,+satisfying "reuse, do not duplicate" while respecting KISS. It directly realizes the intended+collision structure from the scope: the parent establishes a small provider-resolution/no-op+helper as its seam, the child extends that already-merged helper, and their substantive edits+never overlap, so the one-PR-many-cards-one-at-a-time cadence (invariant #6) holds without+ordering gymnastics. It also keeps each stage's contract honest β the parent stays proposal-only+feeding candidates into doctor/registry alias resolution (invariant #8) while the deterministic+`entityIdentityKey` is untouched, and the child stays a graceful no-op / fail-open final reader+stage (invariants #2/#3) β whereas Variant 1's shared contract would blur those two very+different shapes and Variant 3 would put both cards in conflict over a persisted,+determinism-sensitive registry schema. The only cost, two guarded call sites instead of one, is+the cheapest of the three risks to verify and is exactly the "single guarded call site whose+off-branch is unchanged" property invariant #2 calls easiest to check β applied twice.++## Final decision: Variant 2 (accepted)++Agreed with the consultant's recommendation. Variant 2 is the only variant whose collision+structure is resolved by the cheapest sufficient mechanism β the parent lands a small+provider-resolution/no-op helper as its seam and the child extends the already-merged helper,+while their substantive edits live in different subtrees (`brain/entities` + `brain/hygiene` vs+`search`) and never overlap β so the one-PR-many-cards-one-at-a-time cadence (invariant #6)+holds without a premature super-abstraction. It preserves every load-bearing convention: an+LLM-free kernel (both the cosine and the cross-encoder reuse the existing embedding-provider+abstraction / an external endpoint and are graceful no-ops when unconfigured), default-off+byte-identical off-branches (invariant #2, two guarded call sites β the easiest property to+verify, applied twice), fail-open telemetry (invariant #3), language-agnostic logic (invariant+#4, the cosine uses the embedder not an NL word list), Syncthing-peer determinism (invariant #5),+and proposal-only identity writes (invariant #8, the parent surfaces alias-merge candidates and+never rewrites the deterministic `entityIdentityKey`).++Variant 1's shared abstract `learned/` contract couples two stages with genuinely different+contracts (proposal-only candidate pairs vs reader-tail rescoring) behind a+lowest-common-denominator interface and makes the shared module itself the collision seam.+Variant 3 mutates a persisted, determinism-sensitive registry schema that both cards must edit,+inverting the "child extends an already-merged seam" goal and risking backward-compat for+existing vaults. Neither is warranted for this two-card scope.++Drive order: **parent β child** (`t_47fd9523` β `t_110867f5`).diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex d5e0d5e0..3fff8524 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -241,6 +241,7 @@ Wikilinks to frontmatter `aliases:` resolve at index materialization (schema v7)```texto2b brain dream [run] [--dry-run] | stage | validate <run-id> | apply <run-id> | discard <run-id> | list - staged lifecycle over a persisted proposal bundle; validate/apply exit 1 on drifto2b brain doctor gains the removed-tool-reference warning: vault notes, root instruction files, and installed skills naming a tool removed in 1.0.0 are flagged with the replacement+o2b brain doctor opt-in `entity-alias-candidate` lint (off by default): with `entity_semantic_dedup_enabled: true` surfaces lexical entity-name variants ("Google LLC" vs "Google Inc") as PROPOSAL-ONLY alias-merge candidates via a deterministic jaccard layer (`entity_semantic_dedup_lexical_threshold`, default 0.8); never auto-merges or rewrites the identity key. The embedding-cosine layer (`entity_semantic_dedup_threshold`, default 0.92, reuses the configured embedding provider) is exposed as a library reader for apply planso2b brain daily | weekly | monthly | morning-brief | timelinegain additive timezone + local_time JSON fields when `timezone:` is configured; storage stays canonical UTCo2b brain digest | daily | weeklydiff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex 8661c415..7d062151 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -804,6 +804,18 @@ semantic_weightΒ·cosine + link_boost + recency_boost + entity_boost)`indexes each chunk's heading breadcrumb in a dedicated FTS column(weighted below content) so a mid-document chunk keeps its topicalanchor. Entity and heading layers populate on the next reindex.+- **Optional cross-encoder rerank (opt-in).** A learned final reader step+ that re-scores the top-K fused candidates jointly against the query,+ appended after every heuristic rerank. Off by default+ (`search_rerank_enabled`); when on, it resolves an OpenAI-compatible+ `/rerank` endpoint (`search_rerank_base_url` / `search_rerank_model` /+ `search_rerank_env_key`, or a profile registered via+ `o2b search rerank-provider add`) and re-orders the `search_rerank_top_k`+ candidates by the returned relevance, promoting a genuinely-relevant hit+ the heuristic ranker placed deep. Zero HTTP cost and byte-identical+ ordering when disabled; a request-time endpoint error degrades to the+ heuristic ordering (a `rerank_degraded:` warning, never a throw). Most+ valuable on the `thorough` profile.- **Query plan + recall economy (v0.20.0).** A pure structural passclassifies each query's intent (neutral / exact / entity / broad) fromits shape - quoted phrases, FTS wildcards, wikilinks, entity-tokendiff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex eb80dddc..460c26a9 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "1.22.0",+ "version": "1.23.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex a7f13e59..cf5bca88 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.22.0",+ "version": "1.23.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex 1f5c1458..2715bf9a 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.22.0"+version: "1.23.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex 1237650e..ceeff3f5 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "1.22.0",+ "version": "1.23.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex 1f5c1458..2715bf9a 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "1.22.0"+version: "1.23.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex 2c5a61ab..c2b430be 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "1.22.0"+version = "1.23.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/search.ts b/src/cli/search.tsindex 01e7f23c..0b1ec926 100644--- a/src/cli/search.ts+++ b/src/cli/search.ts@@ -45,6 +45,10 @@ import {addProviderProfile,removeProviderProfile,getProviderProfile,+ loadRerankRegistry,+ addRerankProviderProfile,+ removeRerankProviderProfile,+ getRerankProviderProfile,} from "../core/search/index.ts";import type {IndexCheckReport,@@ -75,6 +79,7 @@ const KNOWN_VERBS = new Set(["feedback","weights","provider",+ "rerank-provider","watch",]);@@ -112,6 +117,8 @@ export async function handleSearchSubcommand(argv: ReadonlyArray<string>): Promireturn await cmdSearchWeights(rest);case "provider":return await cmdSearchProvider(rest);+ case "rerank-provider":+ return await cmdSearchRerankProvider(rest);default:process.stderr.write(`error: unknown search verb: ${verb}\n`);return 2;@@ -421,6 +428,102 @@ async function cmdSearchProvider(argv: ReadonlyArray<string>): Promise<number> {return 0;}+// βββ rerank provider registry (retrieval-precision-quality-loop, card A) βββββ++async function cmdSearchRerankProvider(argv: ReadonlyArray<string>): Promise<number> {+ const action = argv[0];+ if (!action || !["add", "list", "show", "remove"].includes(action)) {+ throw new CliError(+ "usage: o2b search rerank-provider <add NAME --base-url U --model M --env-key K | list | show NAME | remove NAME> [--json]",+ );+ }+ const { flags, positional } = parseFlags(argv.slice(1), {+ vault: { type: "string" },+ config: { type: "string" },+ db: { type: "string" },+ "base-url": { type: "string" },+ model: { type: "string" },+ "env-key": { type: "string" },+ json: { type: "boolean" },+ });+ const cfg = resolveConfig(flags);+ const json = flags["json"] === true;+ const name = typeof positional[0] === "string" ? positional[0] : undefined;++ if (action === "list") {+ const registry = loadRerankRegistry(cfg.vault);+ if (json) {+ process.stdout.write(JSON.stringify(registry) + "\n");+ return 0;+ }+ if (registry.length === 0) {+ process.stdout.write("no registered rerank providers\n");+ return 0;+ }+ for (const p of registry) {+ process.stdout.write(`${p.name} ${p.baseUrl} model=${p.defaultModel} env=${p.envKey}\n`);+ }+ return 0;+ }++ if (action === "show") {+ if (!name) throw new CliError("usage: o2b search rerank-provider show NAME");+ const profile = getRerankProviderProfile(cfg.vault, name);+ if (!profile) {+ process.stderr.write(`error: no registered rerank provider named '${name}'\n`);+ return 1;+ }+ process.stdout.write(+ json+ ? JSON.stringify(profile) + "\n"+ : `${profile.name}\n base-url: ${profile.baseUrl}\n model: ${profile.defaultModel}\n env-key: ${profile.envKey}\n`,+ );+ return 0;+ }++ if (action === "remove") {+ if (!name) throw new CliError("usage: o2b search rerank-provider remove NAME");+ const { removed } = removeRerankProviderProfile(cfg.vault, name);+ if (json) {+ process.stdout.write(JSON.stringify({ removed, name }) + "\n");+ } else {+ process.stdout.write(+ removed ? `removed rerank provider '${name}'\n` : `no such rerank provider '${name}'\n`,+ );+ }+ return removed ? 0 : 1;+ }++ // add+ if (!name)+ throw new CliError(+ "usage: o2b search rerank-provider add NAME --base-url U --model M --env-key K",+ );+ const baseUrl = typeof flags["base-url"] === "string" ? (flags["base-url"] as string) : undefined;+ const model = typeof flags["model"] === "string" ? (flags["model"] as string) : undefined;+ const envKey = typeof flags["env-key"] === "string" ? (flags["env-key"] as string) : undefined;+ if (!baseUrl || !model || !envKey) {+ throw new CliError(+ "rerank-provider add requires --base-url, --model, and --env-key (the env var NAME holding the API key)",+ );+ }+ const registry = addRerankProviderProfile(cfg.vault, {+ name,+ baseUrl,+ defaultModel: model,+ envKey,+ });+ const added = registry.find((p) => p.name === name)!;+ if (json) {+ process.stdout.write(JSON.stringify(added) + "\n");+ } else {+ process.stdout.write(+ `added rerank provider '${name}' (set ${envKey} in the environment to supply its key)\n`,+ );+ }+ return 0;+}+// βββ query ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββasync function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<number> {diff --git a/src/core/brain/doctor.ts b/src/core/brain/doctor.tsindex e83fa184..6469af6d 100644--- a/src/core/brain/doctor.ts+++ b/src/core/brain/doctor.ts@@ -44,6 +44,10 @@ import { extractWikilinks, listVaultBasenames, parseFrontmatter } from "../vaultimport { resolveVaultScope } from "../vault-scope/index.ts";import { buildBacklinkIndex } from "./backlinks.ts";import { buildEntityIndex } from "./entities/index-builder.ts";+import {+ entityLexicalAliasCandidates,+ resolveEntitySemanticDedupConfig,+} from "./entities/semantic-dedup.ts";import { buildCaptureBoundary } from "./capture-boundary.ts";import { verifyContentHash } from "./content-hash.ts";import { readTierDriftCount } from "./frontmatter-tiers.ts";@@ -1272,6 +1276,25 @@ function checkEntities(vault: string, issues: DoctorIssue[]): void {});}}++ // semantic-retrieval-precision (t_47fd9523): opt-in, proposal-only+ // alias-merge candidates. Off by default β this block is a no-op and+ // the report is byte-identical to the baseline. The doctor is+ // synchronous and embeds nothing, so it uses the deterministic lexical+ // (jaccard-over-names) layer; the embedding layer lives on the CLI/MCP+ // reader. Candidates are NOMINATIONS β never an auto-merge.+ const dedupCfg = resolveEntitySemanticDedupConfig();+ if (dedupCfg.enabled) {+ for (const c of entityLexicalAliasCandidates(vault, { threshold: dedupCfg.lexicalThreshold })) {+ issues.push({+ severity: "warning",+ code: "entity-alias-candidate",+ message:+ `possible alias-merge: '${c.name_a}' (${c.a}) ~ '${c.name_b}' (${c.b}) ` ++ `[${c.method} ${c.similarity}]. Review and add an alias to merge β never auto-merged.`,+ });+ }+ }}function collectAllBasenames(vault: string): ReadonlySet<string> {diff --git a/src/core/brain/entities/semantic-dedup.ts b/src/core/brain/entities/semantic-dedup.tsnew file mode 100644index 00000000..a801731e--- /dev/null+++ b/src/core/brain/entities/semantic-dedup.ts@@ -0,0 +1,351 @@+/**+ * Semantic (embedding cosine) entity dedup+ * (semantic-retrieval-precision, parent t_47fd9523).+ *+ * OSB's entity identity is purely deterministic string-normalization:+ * `normalizeEntityName` + `entityIdentityKey` in `canonical.ts`. That key+ * silently treats lexical variants of the same real-world entity as+ * distinct records ("Google LLC" vs "Google Inc", "OpenAI" vs "Open AI"),+ * fragmenting the graph. This pass complements the key with an OPT-IN,+ * PROPOSAL-ONLY layer that surfaces those variants as alias-merge+ * CANDIDATES. Modeled on the proven hygiene dedup detector+ * (`hygiene/detectors/dedup.ts`): an embedding-cosine layer over the+ * canonical registry, with a clearly-labeled `method: "lexical"` jaccard+ * fallback when vectors are unavailable.+ *+ * It NOMINATES pairs only β it never rewrites `entityIdentityKey`, never+ * touches an entity file. Candidates feed the doctor-lint / registry+ * alias-resolution seam, where a human (or an apply plan) owns the merge,+ * preserving OSB's deterministic, audit-friendly core.+ */++import { discoverConfig } from "../../config.ts";+import { parseFrontmatter } from "../../vault.ts";+import { envOrConfig, parseBool, parseFloat01 } from "../../validate.ts";+import { resolveConfiguredEmbeddingProvider } from "../../search/embeddings/provider-resolve.ts";+import type { EmbeddingProvider } from "../../search/embeddings/provider.ts";+import { jaccard, tokenise } from "../similarity.ts";+import { buildEntityIndex } from "./index-builder.ts";+import { normalizeEntityName } from "./canonical.ts";+import { BRAIN_ENTITY_STATUS, type BrainEntity } from "./types.ts";++/**+ * Cosine threshold for the embedding layer. Default deliberately high so+ * only near-duplicates surface (few false positives; the deterministic+ * core plus a human own the merge).+ */+export const ENTITY_DEDUP_EMBEDDING_THRESHOLD = 0.92;+/** Jaccard threshold for the lexical fallback. */+export const ENTITY_DEDUP_LEXICAL_THRESHOLD = 0.8;+/** Bound on how many active entities enter pairwise comparison. */+const ENTITY_DEDUP_CANDIDATE_CAP = 500;++export type EntityDedupMethod = "embedding" | "lexical";++/** A nominated alias-merge candidate pair β never auto-applied. */+export interface EntityAliasCandidate {+ /** Deterministic pair id: `<entity-id-a>::<entity-id-b>` (ids sorted). */+ readonly id: string;+ readonly method: EntityDedupMethod;+ readonly category: string;+ /** Lexicographically smaller entity id. */+ readonly a: string;+ readonly b: string;+ readonly name_a: string;+ readonly name_b: string;+ /** Cosine (embedding) or jaccard (lexical) similarity, rounded to 4dp. */+ readonly similarity: number;+ /** Structural identity type when present in frontmatter; absent otherwise. */+ readonly identity_type_a?: string;+ readonly identity_type_b?: string;+}++export interface EntitySemanticDedupResult {+ readonly method: EntityDedupMethod;+ readonly candidates: ReadonlyArray<EntityAliasCandidate>;+}++export interface EntitySemanticDedupOptions {+ /**+ * Embedding provider override. `undefined` resolves the vault's+ * configured provider; `null` (or an unusable provider) falls back to+ * the lexical layer.+ */+ readonly provider?: EmbeddingProvider | null;+ readonly threshold?: number;+ readonly lexicalThreshold?: number;+ readonly configPath?: string;+}++export interface EntityLexicalOptions {+ readonly threshold?: number;+}++/** Resolved default-off config surface for the pass. */+export interface EntitySemanticDedupConfig {+ readonly enabled: boolean;+ readonly threshold: number;+ readonly lexicalThreshold: number;+}++/**+ * Resolve the `entity_semantic_dedup_*` config family (env wins over the+ * config file). Off by default: with no config the pass never runs, so+ * registry / doctor / search behavior is byte-identical to the baseline.+ */+export function resolveEntitySemanticDedupConfig(configPath?: string): EntitySemanticDedupConfig {+ const env = process.env;+ const config = discoverConfig(configPath).data;+ const enabled = parseBool(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_ENABLED",+ "entity_semantic_dedup_enabled",+ ),+ false,+ "entity_semantic_dedup_enabled",+ );+ const threshold = parseFloat01(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_THRESHOLD",+ "entity_semantic_dedup_threshold",+ ),+ ENTITY_DEDUP_EMBEDDING_THRESHOLD,+ "entity_semantic_dedup_threshold",+ );+ const lexicalThreshold = parseFloat01(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_LEXICAL_THRESHOLD",+ "entity_semantic_dedup_lexical_threshold",+ ),+ ENTITY_DEDUP_LEXICAL_THRESHOLD,+ "entity_semantic_dedup_lexical_threshold",+ );+ return Object.freeze({ enabled, threshold, lexicalThreshold });+}++/**+ * Structural identity type: the operator-set `identity_type` frontmatter+ * key (e.g. `org` / `person` / `product`), validated as a slug. Derived+ * from frontmatter / structural signals, NEVER from parsing the name for+ * natural-language keywords β language-agnostic by construction. Absent+ * (undefined) unless the key is present and well-formed.+ */+export function deriveIdentityType(entity: BrainEntity): string | undefined {+ let raw: unknown;+ try {+ raw = parseFrontmatter(entity.path)[0]["identity_type"];+ } catch {+ return undefined;+ }+ if (typeof raw !== "string") return undefined;+ const t = raw.normalize("NFC").trim().toLowerCase();+ return /^[a-z0-9][a-z0-9-]*$/.test(t) ? t : undefined;+}++/**+ * Evolution chain: the canonical name followed by its aliases in file+ * order (deduplicated by normalized form). This is the structural record+ * of how a name evolved as variants merged into the canonical entity β+ * read-only, deterministic (the file is the single source of truth).+ */+export function entityEvolutionChain(entity: BrainEntity): ReadonlyArray<string> {+ const seen = new Set<string>();+ const chain: string[] = [];+ for (const name of [entity.name, ...entity.aliases]) {+ const norm = normalizeEntityName(name);+ if (!norm || seen.has(norm)) continue;+ seen.add(norm);+ chain.push(name);+ }+ return Object.freeze(chain);+}++/** Active entities, id-sorted, capped β the comparison universe. */+function activeEntities(vault: string): BrainEntity[] {+ const entities = buildEntityIndex(vault).entities.filter(+ (e) => e.status === BRAIN_ENTITY_STATUS.active,+ );+ return entities.slice(0, ENTITY_DEDUP_CANDIDATE_CAP);+}++/** True when a and b already reference each other by canonical name / alias. */+function alreadyLinked(a: BrainEntity, b: BrainEntity): boolean {+ const aNames = new Set([a.name, ...a.aliases].map(normalizeEntityName));+ const bNames = new Set([b.name, ...b.aliases].map(normalizeEntityName));+ for (const n of bNames) if (aNames.has(n)) return true;+ return false;+}++interface Pairing {+ readonly a: BrainEntity;+ readonly b: BrainEntity;+}++/**+ * Enumerate comparable pairs: same category, distinct entities, not+ * already linked by an alias. Cross-category pairs are excluded β aliases+ * resolve within a category, so a cross-category "match" is noise.+ */+function comparablePairs(entities: ReadonlyArray<BrainEntity>): Pairing[] {+ const byCategory = new Map<string, BrainEntity[]>();+ for (const e of entities) {+ const bucket = byCategory.get(e.category) ?? [];+ bucket.push(e);+ byCategory.set(e.category, bucket);+ }+ const pairs: Pairing[] = [];+ for (const bucket of byCategory.values()) {+ for (let i = 0; i < bucket.length; i++) {+ for (let j = i + 1; j < bucket.length; j++) {+ const a = bucket[i]!;+ const b = bucket[j]!;+ if (alreadyLinked(a, b)) continue;+ pairs.push({ a, b });+ }+ }+ }+ return pairs;+}++function cosine(a: ReadonlyArray<number>, b: ReadonlyArray<number>): number {+ let dot = 0;+ let normA = 0;+ let normB = 0;+ for (let i = 0; i < Math.min(a.length, b.length); i++) {+ dot += a[i]! * b[i]!;+ normA += a[i]! * a[i]!;+ normB += b[i]! * b[i]!;+ }+ if (normA === 0 || normB === 0) return 0;+ return dot / (Math.sqrt(normA) * Math.sqrt(normB));+}++const identityTypeCache = new WeakMap<BrainEntity, string | undefined>();+function cachedIdentityType(entity: BrainEntity): string | undefined {+ if (identityTypeCache.has(entity)) return identityTypeCache.get(entity);+ const t = deriveIdentityType(entity);+ identityTypeCache.set(entity, t);+ return t;+}++function makeCandidate(+ method: EntityDedupMethod,+ a: BrainEntity,+ b: BrainEntity,+ similarity: number,+): EntityAliasCandidate {+ // Sort the pair by id so `id`, `a`, `b` are stable regardless of walk order.+ const [lo, hi] = a.id.localeCompare(b.id) <= 0 ? [a, b] : [b, a];+ const typeLo = cachedIdentityType(lo);+ const typeHi = cachedIdentityType(hi);+ return Object.freeze({+ id: `${lo.id}::${hi.id}`,+ method,+ category: lo.category,+ a: lo.id,+ b: hi.id,+ name_a: lo.name,+ name_b: hi.name,+ similarity: Number(similarity.toFixed(4)),+ ...(typeLo !== undefined ? { identity_type_a: typeLo } : {}),+ ...(typeHi !== undefined ? { identity_type_b: typeHi } : {}),+ });+}++/** Stable ordering: category, then the two ids. */+function sortCandidates(candidates: EntityAliasCandidate[]): EntityAliasCandidate[] {+ return candidates.toSorted((x, y) => {+ const byCategory = x.category.localeCompare(y.category);+ if (byCategory !== 0) return byCategory;+ const byA = x.a.localeCompare(y.a);+ if (byA !== 0) return byA;+ return x.b.localeCompare(y.b);+ });+}++/**+ * Deterministic lexical fallback: token jaccard over entity names within+ * a category. Synchronous and pure-read β this is the layer the doctor+ * lint uses (the doctor is sync and never embeds).+ */+export function entityLexicalAliasCandidates(+ vault: string,+ opts: EntityLexicalOptions = {},+): ReadonlyArray<EntityAliasCandidate> {+ const threshold = opts.threshold ?? ENTITY_DEDUP_LEXICAL_THRESHOLD;+ const pairs = comparablePairs(activeEntities(vault));+ const out: EntityAliasCandidate[] = [];+ for (const { a, b } of pairs) {+ const similarity = jaccard(tokenise(a.name), tokenise(b.name));+ if (similarity < threshold) continue;+ out.push(makeCandidate("lexical", a, b, similarity));+ }+ return Object.freeze(sortCandidates(out));+}++/**+ * Surface alias-merge candidates. Prefers the embedding-cosine layer when+ * a usable provider is available; any provider problem falls back to the+ * deterministic lexical layer, labeled `method: "lexical"` so a report+ * never passes lexical similarity off as semantic. Never writes.+ */+export async function detectEntityAliasCandidates(+ vault: string,+ opts: EntitySemanticDedupOptions = {},+): Promise<EntitySemanticDedupResult> {+ const provider =+ opts.provider === undefined+ ? resolveConfiguredEmbeddingProvider(+ vault,+ opts.configPath ? { configPath: opts.configPath } : {},+ )+ : opts.provider;++ const lexical = (): EntitySemanticDedupResult =>+ Object.freeze({+ method: "lexical" as const,+ candidates: entityLexicalAliasCandidates(+ vault,+ opts.lexicalThreshold !== undefined ? { threshold: opts.lexicalThreshold } : {},+ ),+ });++ if (provider === null || provider.name === "null") return lexical();++ const entities = activeEntities(vault);+ const pairs = comparablePairs(entities);+ if (pairs.length === 0) return Object.freeze({ method: "embedding" as const, candidates: [] });++ const threshold = opts.threshold ?? ENTITY_DEDUP_EMBEDDING_THRESHOLD;++ let vectors: number[][];+ try {+ vectors = await provider.embed(entities.map((e) => e.name));+ } catch {+ return lexical();+ }+ if (vectors.length !== entities.length) return lexical();++ const vectorById = new Map<string, number[]>();+ for (let i = 0; i < entities.length; i++) vectorById.set(entities[i]!.id, vectors[i]!);++ const out: EntityAliasCandidate[] = [];+ for (const { a, b } of pairs) {+ const va = vectorById.get(a.id);+ const vb = vectorById.get(b.id);+ if (!va || !vb) continue;+ const similarity = cosine(va, vb);+ if (similarity < threshold) continue;+ out.push(makeCandidate("embedding", a, b, similarity));+ }+ return Object.freeze({+ method: "embedding" as const,+ candidates: Object.freeze(sortCandidates(out)),+ });+}diff --git a/src/core/search/embeddings/provider-resolve.ts b/src/core/search/embeddings/provider-resolve.tsnew file mode 100644index 00000000..ff73a868--- /dev/null+++ b/src/core/search/embeddings/provider-resolve.ts@@ -0,0 +1,116 @@+/**+ * Provider-resolution / no-op seam (semantic-retrieval-precision, parent+ * t_47fd9523).+ *+ * Two shared, dependency-light helpers modeled on `openai-compat.ts`'s+ * fail-closed-validation + graceful-no-op discipline:+ *+ * - `resolveConfiguredEmbeddingProvider` resolves the vault's+ * configured embedding provider, returning `null` (never throwing+ * into the hot path) when semantic search is disabled or resolution+ * fails. The entity semantic-dedup pass uses this for its cosine+ * layer.+ * - `resolveOpenAiCompatEndpoint` validates an OpenAI-compatible+ * endpoint config (base_url + model + env-resolved api key) once,+ * fail-closed when a feature is enabled but its endpoint is+ * incomplete, and a graceful no-op (null) when the feature is off.+ * This is THE SEAM the child cross-encoder rerank stage reuses so+ * provider resolution is not re-implemented per feature.+ */++import { discoverConfig } from "../../config.ts";+import { SearchError } from "../types.ts";+import { makeProvider, type EmbeddingProvider } from "./provider.ts";+import { resolveSearchConfig } from "../index.ts";++/**+ * Resolve the vault's configured embedding provider. Returns `null` when+ * semantic search is disabled (the null provider) or when resolution+ * throws β callers then fall back to a deterministic path rather than+ * failing. Mirrors the inline guard in the hygiene dedup detector.+ */+export function resolveConfiguredEmbeddingProvider(+ vault: string,+ opts: { readonly configPath?: string } = {},+): EmbeddingProvider | null {+ try {+ const configPath = opts.configPath ?? discoverConfig().path;+ const provider = makeProvider(resolveSearchConfig({ vault, configPath }).semantic);+ return provider.name === "null" ? null : provider;+ } catch {+ return null;+ }+}++/** A fully-resolved OpenAI-compatible endpoint (base_url + model + key). */+export interface OpenAiCompatEndpoint {+ /** Base URL with trailing slashes stripped. */+ readonly baseUrl: string;+ readonly model: string;+ readonly apiKey: string;+}++/** Inputs to {@link resolveOpenAiCompatEndpoint}. */+export interface ResolveEndpointInput {+ /** Feature flag; when false the resolver is a graceful no-op. */+ readonly enabled: boolean;+ readonly baseUrl: string | null;+ readonly model: string | null;+ /** Env var NAME the api key is read from (looked up in `env`). */+ readonly envKey?: string | null;+ /** Directly-provided api key; wins over the `envKey` lookup. */+ readonly apiKey?: string | null;+ /** Environment map; defaults to `process.env`. */+ readonly env?: Readonly<Record<string, string | undefined>>;+}++function nonEmpty(value: string | null | undefined): string | null {+ if (value === null || value === undefined) return null;+ const trimmed = value.trim();+ return trimmed === "" ? null : trimmed;+}++/**+ * Resolve an OpenAI-compatible endpoint config, fail-closed. Returns+ * `null` when `enabled` is false (graceful no-op β zero cost, no+ * validation). When enabled, requires base_url, model, and a non-empty+ * api key (from `apiKey`, else `env[envKey]`), throwing a `SearchError`+ * with a `label`-prefixed message otherwise β the same discipline+ * `openai-compat.ts` applies when semantic is enabled without an+ * endpoint. `label` names the config family, e.g. `search_rerank`.+ */+export function resolveOpenAiCompatEndpoint(+ input: ResolveEndpointInput,+ label: string,+): OpenAiCompatEndpoint | null {+ if (!input.enabled) return null;++ const baseUrl = nonEmpty(input.baseUrl);+ if (baseUrl === null) {+ throw new SearchError(+ "INVALID_INPUT",+ `${label}_base_url is required when ${label} is enabled`,+ );+ }+ const model = nonEmpty(input.model);+ if (model === null) {+ throw new SearchError("INVALID_INPUT", `${label}_model is required when ${label} is enabled`);+ }++ const env = input.env ?? process.env;+ const direct = nonEmpty(input.apiKey);+ const fromEnv = input.envKey ? nonEmpty(env[input.envKey]) : null;+ const apiKey = direct ?? fromEnv;+ if (apiKey === null) {+ throw new SearchError(+ "EMBEDDING_KEY_MISSING",+ `${label} api key is required when ${label} is enabled`,+ );+ }++ return Object.freeze({+ baseUrl: baseUrl.replace(/\/+$/, ""),+ model,+ apiKey,+ });+}diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex 240da29a..28605c1c 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -18,17 +18,22 @@ import {expandRegisteredProvider,type ExpandedProvider,} from "./embeddings/registry.ts";+import { loadRerankRegistry, expandRegisteredRerankProvider } from "./rerank/registry.ts";import { SearchError } from "./types.ts";import type {ResolvedEmbeddingConfig,ResolvedRecallConfig,+ ResolvedRerankConfig,ResolvedSearchConfig,VaultIgnoreRule,} from "./types.ts";-type SearchConfigOverrides = Partial<Omit<ResolvedSearchConfig, "ignoreRules" | "semantic">> & {+type SearchConfigOverrides = Partial<+ Omit<ResolvedSearchConfig, "ignoreRules" | "semantic" | "rerank">+> & {readonly ignoreRules?: ReadonlyArray<VaultIgnoreRule>;readonly semantic?: Partial<ResolvedEmbeddingConfig>;+ readonly rerank?: Partial<ResolvedRerankConfig>;};export type {@@ -43,6 +48,7 @@ export type {IndexStatusSnapshot,ResolvedEmbeddingConfig,ResolvedRecallConfig,+ ResolvedRerankConfig,ResolvedSearchConfig,SearchCard,SearchErrorCode,@@ -75,6 +81,19 @@ export {RESERVED_PROVIDER_NAMES,type ProviderProfile,} from "./embeddings/registry.ts";+export {+ loadRerankRegistry,+ addRerankProviderProfile,+ removeRerankProviderProfile,+ getRerankProviderProfile,+ rerankRegistryPath,+ type RerankProviderProfile,+} from "./rerank/registry.ts";+export {+ applyCrossEncoderRerank,+ type ApplyCrossEncoderRerankOptions,+ type RerankTelemetryEvent,+} from "./rerank/index.ts";export { resolveIndexPath } from "./paths.ts";export {@@ -125,6 +144,8 @@ const DEFAULTS = {shutdownGraceSeconds: 5,resumeReindex: false,chainStopScore: 0.8,+ rerankTopK: 20,+ rerankMinScore: 0,};function parseFusionMode(raw: string | null): "linear" | "rrf" {@@ -187,6 +208,20 @@ function parseNonNegativeFloat(raw: string | null, fallback: number, fieldName:return n;}+/**+ * Parse a finite float over the whole real line (e.g. a rerank floor,+ * which can be negative for backends that emit logits rather than [0, 1]+ * relevance).+ */+function parseFiniteFloat(raw: string | null, fallback: number, fieldName: string): number {+ if (raw === null) return fallback;+ const n = Number(raw);+ if (!Number.isFinite(n)) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be a finite number, got '${raw}'`);+ }+ return n;+}+function validateIntegerRange(n: number, fieldName: string, range?: IntegerRange): void {if (range?.min !== undefined && n < range.min) {throw new SearchError("INVALID_INPUT", `${fieldName} must be >= ${range.min}, got ${n}`);@@ -411,6 +446,69 @@ export function resolveSearchConfig(opts: {costGateUsd,});+ // Cross-encoder rerank (retrieval-precision-quality-loop, card A). Off+ // by default; when off the reader tail is byte-identical. A registered+ // provider name (`search_rerank_provider`) supplies base_url / model /+ // env-key defaults; explicit `search_rerank_*` config/env always wins.+ const rerankEnabled = parseBool(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RERANK_ENABLED", "search_rerank_enabled"),+ false,+ "search_rerank_enabled",+ );+ const rerankProviderName = envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_PROVIDER",+ "search_rerank_provider",+ );+ const rerankProfile =+ rerankProviderName !== null+ ? expandRegisteredRerankProvider(rerankProviderName, loadRerankRegistry(opts.vault))+ : null;+ const rerankBaseUrl =+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_BASE_URL",+ "search_rerank_base_url",+ ) ??+ rerankProfile?.baseUrl ??+ null;+ const rerankModel =+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RERANK_MODEL", "search_rerank_model") ??+ rerankProfile?.model ??+ null;+ const rerankEnvKey =+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RERANK_ENV_KEY", "search_rerank_env_key") ??+ rerankProfile?.envKey ??+ null;+ const rerankApiKey = rerankEnvKey !== null ? (env[rerankEnvKey] ?? null) : null;+ const rerankTopK = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RERANK_TOP_K", "search_rerank_top_k"),+ DEFAULTS.rerankTopK,+ "search_rerank_top_k",+ { min: 1 },+ );+ const rerankMinScore = parseFiniteFloat(+ envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_MIN_SCORE",+ "search_rerank_min_score",+ ),+ DEFAULTS.rerankMinScore,+ "search_rerank_min_score",+ );+ const rerank: ResolvedRerankConfig = Object.freeze({+ enabled: rerankEnabled,+ baseUrl: rerankBaseUrl,+ model: rerankModel,+ envKey: rerankEnvKey,+ apiKey: rerankApiKey !== null && rerankApiKey !== "" ? rerankApiKey : null,+ topK: rerankTopK,+ minScore: rerankMinScore,+ });+const mmrLambda = parseFloat01(envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_MMR_LAMBDA", "search_mmr_lambda"),DEFAULTS.mmrLambda,@@ -598,6 +696,7 @@ export function resolveSearchConfig(opts: {rrfK,semantic,recall,+ rerank,shutdownGraceMs: shutdownGraceSeconds * 1000,resumeReindex,});@@ -610,6 +709,7 @@ export function resolveSearchConfig(opts: {...base,...opts.overrides,semantic: Object.freeze({ ...base.semantic, ...opts.overrides.semantic }),+ rerank: Object.freeze({ ...base.rerank, ...opts.overrides.rerank }),ignoreRules: opts.overrides.ignoreRules? Object.freeze([...opts.overrides.ignoreRules]): base.ignoreRules,diff --git a/src/core/search/rerank/cross-encoder.ts b/src/core/search/rerank/cross-encoder.tsnew file mode 100644index 00000000..7ab0d65c--- /dev/null+++ b/src/core/search/rerank/cross-encoder.ts@@ -0,0 +1,163 @@+/**+ * OpenAI-compatible cross-encoder rerank provider+ * (retrieval-precision-quality-loop, card A / t_110867f5).+ *+ * Calls a `{base_url}/rerank` endpoint β the shape Cohere, Jina,+ * text-embeddings-inference, and vLLM's rerank route share β with+ * `{ model, query, documents }` and reads one relevance score per+ * document back. Both the wrapped `{ results: [{ index, relevance_score }] }`+ * shape and the bare `[{ index, score }]` array shape are accepted, so a+ * single provider works across compatible backends.+ *+ * Network rules mirror `embeddings/openai-compat.ts` at a smaller scale:+ * one request (the top-K candidate set is small, no batching), a+ * per-request timeout, and provider-shaped `SearchError`s. Retries are+ * intentionally omitted: this is an opt-in final reader step that+ * degrades gracefully to the heuristic ordering on ANY failure (see+ * `applyCrossEncoderRerank`), so a slow retry loop would only add latency+ * to the hot path for a result the caller already has a good answer for.+ */++import { SearchError } from "../types.ts";+import type { OpenAiCompatEndpoint } from "../embeddings/provider-resolve.ts";+import type { RerankProvider } from "./provider.ts";++/** Default per-request timeout when the caller does not override it. */+export const DEFAULT_RERANK_TIMEOUT_MS = 5000;++interface RerankResultItem {+ readonly index: number;+ readonly relevance_score?: number;+ readonly score?: number;+}++interface WrappedRerankResponse {+ readonly results: ReadonlyArray<RerankResultItem>;+}++function extractItems(json: unknown): ReadonlyArray<RerankResultItem> {+ if (Array.isArray(json)) return json as ReadonlyArray<RerankResultItem>;+ if (+ json !== null &&+ typeof json === "object" &&+ Array.isArray((json as WrappedRerankResponse).results)+ ) {+ return (json as WrappedRerankResponse).results;+ }+ throw new SearchError(+ "RERANK_PROVIDER_HTTP",+ "rerank response shape: expected an array or a { results: [...] } object",+ );+}++function scoreOf(item: RerankResultItem): number {+ const raw = item.relevance_score ?? item.score;+ if (typeof raw !== "number" || !Number.isFinite(raw)) {+ throw new SearchError(+ "RERANK_PROVIDER_HTTP",+ `rerank response: item at index ${item.index} has no finite relevance score`,+ );+ }+ return raw;+}++export class CrossEncoderRerankProvider implements RerankProvider {+ readonly name = "openai-compat-rerank";+ readonly model: string;+ private readonly endpoint: OpenAiCompatEndpoint;+ private readonly url: string;+ private readonly timeoutMs: number;++ constructor(endpoint: OpenAiCompatEndpoint, opts?: { readonly timeoutMs?: number }) {+ this.endpoint = endpoint;+ this.model = endpoint.model;+ this.url = `${endpoint.baseUrl}/rerank`;+ this.timeoutMs = opts?.timeoutMs ?? DEFAULT_RERANK_TIMEOUT_MS;+ }++ async rerank(query: string, documents: ReadonlyArray<string>): Promise<number[]> {+ if (documents.length === 0) return [];++ const controller = new AbortController();+ const timer = setTimeout(() => controller.abort(), this.timeoutMs);+ let response: Response;+ try {+ response = await fetch(this.url, {+ method: "POST",+ headers: {+ "content-type": "application/json",+ authorization: `Bearer ${this.endpoint.apiKey}`,+ },+ body: JSON.stringify({+ model: this.model,+ query,+ documents: [...documents],+ }),+ signal: controller.signal,+ });+ } catch (e) {+ const cause = e instanceof Error ? e : new Error(String(e));+ if (cause.name === "AbortError") {+ throw new SearchError(+ "RERANK_PROVIDER_HTTP",+ `rerank request timed out after ${this.timeoutMs}ms`,+ );+ }+ throw new SearchError("RERANK_PROVIDER_HTTP", `network error: ${cause.message}`);+ } finally {+ clearTimeout(timer);+ }++ if (!response.ok) {+ const body = await response.text().catch(() => "");+ const head = body.slice(0, 300);+ throw new SearchError(+ "RERANK_PROVIDER_HTTP",+ `rerank HTTP ${response.status}: ${head || response.statusText}`,+ );+ }++ let json: unknown;+ try {+ json = await response.json();+ } catch (e) {+ const msg = e instanceof Error ? e.message : String(e);+ throw new SearchError("RERANK_PROVIDER_HTTP", `rerank response not JSON: ${msg}`);+ }++ const items = extractItems(json);+ if (items.length !== documents.length) {+ throw new SearchError(+ "RERANK_PROVIDER_HTTP",+ `rerank response shape: expected ${documents.length} scores, got ${items.length}`,+ );+ }++ // Map back to input order. Each item carries its own `index`, so a+ // backend that returns them sorted by score is realigned here.+ const scores: number[] = Array.from({ length: documents.length }, () => Number.NaN);+ const seen: boolean[] = Array.from({ length: documents.length }, () => false);+ for (const item of items) {+ if (+ typeof item.index !== "number" ||+ item.index < 0 ||+ item.index >= documents.length ||+ !Number.isInteger(item.index)+ ) {+ throw new SearchError(+ "RERANK_PROVIDER_HTTP",+ `rerank response: out-of-range index ${item.index}`,+ );+ }+ if (seen[item.index]) {+ throw new SearchError(+ "RERANK_PROVIDER_HTTP",+ `rerank response: duplicate index ${item.index}`,+ );+ }+ seen[item.index] = true;+ scores[item.index] = scoreOf(item);+ }+ return scores;+ }+}diff --git a/src/core/search/rerank/index.ts b/src/core/search/rerank/index.tsnew file mode 100644index 00000000..6bffcd3e--- /dev/null+++ b/src/core/search/rerank/index.ts@@ -0,0 +1,154 @@+/**+ * Cross-encoder rerank stage - the final reader step+ * (retrieval-precision-quality-loop, card A / t_110867f5).+ *+ * {@link applyCrossEncoderRerank} re-scores the top-K fused candidates+ * jointly against the query using a learned cross-encoder, then re-orders+ * that block by the returned relevance scores. It is appended AFTER the+ * heuristic reranks (relevance rerank, entity boost, MMR, usage decay) in+ * the reader tail, so it is the last word on ordering.+ *+ * Three invariants, mirroring the embedding-provider discipline:+ * 1. Disabled (the default) -> the input is returned unchanged, with no+ * HTTP call, no allocation of a new array, and no telemetry. The+ * reader tail is byte-identical to the pre-feature baseline.+ * 2. Enabled but unconfigured -> fail closed. Endpoint resolution+ * (`resolveOpenAiCompatEndpoint`) throws a typed `SearchError`, the+ * same discipline `openai-compat.ts` applies when semantic is+ * enabled without a base_url/model/key. This is a configuration+ * error the operator must fix, not a hot-path degrade.+ * 3. Enabled and configured but the endpoint errors at request time ->+ * degrade gracefully to the heuristic-ordered input (never throw+ * into the hot path) and emit one fail-open telemetry event.+ */++import { resolveOpenAiCompatEndpoint } from "../embeddings/provider-resolve.ts";+import type { BrainSearchResult, ResolvedRerankConfig } from "../types.ts";+import { makeRerankProvider, type RerankProvider } from "./provider.ts";++/** Fixed-precision so the reason string is stable for a given score. */+function fmtScore(x: number): string {+ return x.toFixed(4);+}++export interface RerankTelemetryEvent {+ readonly status: "applied" | "error";+ /** Present on `error`: the provider-shaped failure message. */+ readonly reason?: string;+ /** Number of top candidates handed to the cross-encoder. */+ readonly candidateCount: number;+}++export interface ApplyCrossEncoderRerankOptions {+ /** Inject a provider (tests / alternate backends). Defaults to the HTTP one. */+ readonly provider?: RerankProvider;+ /** Environment map for env-key resolution; defaults to `process.env`. */+ readonly env?: Readonly<Record<string, string | undefined>>;+ /** Per-request timeout override for the default provider. */+ readonly timeoutMs?: number;+ /**+ * Fail-open telemetry sink. Invoked at most once per call - on success+ * (`applied`) or on a request-time degrade (`error`). Never invoked on+ * the disabled path. The caller wraps this so a throwing sink can never+ * fail the search.+ */+ readonly onTelemetry?: (event: RerankTelemetryEvent) => void;+}++interface Scored {+ readonly result: BrainSearchResult;+ readonly index: number;+ readonly score: number;+}++/**+ * Re-order the top-K block by cross-encoder score, honouring the+ * `minScore` floor. Deterministic given the scores: qualifying candidates+ * (score >= floor) sort by score descending, ties broken by original+ * index; below-floor candidates keep their original relative order and+ * sit below the qualifying ones. Each reranked hit gains a+ * `cross_encoder:` reason so explainability shows the stage fired.+ */+function reorderTopK(+ head: ReadonlyArray<BrainSearchResult>,+ scores: ReadonlyArray<number>,+ minScore: number,+): BrainSearchResult[] {+ const scored: Scored[] = head.map((result, index) => ({+ result,+ index,+ score: scores[index] ?? Number.NEGATIVE_INFINITY,+ }));+ const qualifying = scored.filter((s) => s.score >= minScore);+ const belowFloor = scored.filter((s) => s.score < minScore);+ qualifying.sort((a, b) => b.score - a.score || a.index - b.index);+ return [...qualifying, ...belowFloor].map((s) =>+ Object.freeze({+ ...s.result,+ reasons: Object.freeze([...s.result.reasons, `cross_encoder: ${fmtScore(s.score)}`]),+ }),+ );+}++/**+ * Apply the optional cross-encoder rerank stage. Returns the input+ * unchanged when disabled or on a request-time endpoint error; throws a+ * typed `SearchError` when enabled but the endpoint is unconfigured.+ */+export async function applyCrossEncoderRerank(+ results: ReadonlyArray<BrainSearchResult>,+ query: string,+ config: ResolvedRerankConfig,+ opts: ApplyCrossEncoderRerankOptions = {},+): Promise<ReadonlyArray<BrainSearchResult>> {+ // Invariant 1: disabled -> zero-cost no-op, byte-identical input.+ if (!config.enabled) return results;++ // Invariant 2: fail closed when enabled-but-unconfigured. Resolution+ // throws a typed SearchError; we deliberately do NOT catch it - a+ // misconfiguration is an operator error, not a hot-path degrade.+ const endpoint = resolveOpenAiCompatEndpoint(+ {+ enabled: true,+ baseUrl: config.baseUrl,+ model: config.model,+ envKey: config.envKey,+ apiKey: config.apiKey,+ env: opts.env,+ },+ "search_rerank",+ );+ // resolveOpenAiCompatEndpoint only returns null when `enabled` is false,+ // which we ruled out above; the narrowing keeps TypeScript honest.+ if (endpoint === null) return results;++ if (results.length === 0) return results;++ const topK = Math.min(Math.max(1, config.topK), results.length);+ const head = results.slice(0, topK);+ const tail = results.slice(topK);+ const documents = head.map((r) => r.content);++ const provider = opts.provider ?? makeRerankProvider(endpoint, { timeoutMs: opts.timeoutMs });++ let scores: number[];+ try {+ scores = await provider.rerank(query, documents);+ if (scores.length !== documents.length) {+ throw new Error(`expected ${documents.length} scores, got ${scores.length}`);+ }+ } catch (e) {+ // Invariant 3: request-time degrade. Return the heuristic ordering+ // untouched and emit one fail-open telemetry event.+ opts.onTelemetry?.({+ status: "error",+ reason: e instanceof Error ? e.message : String(e),+ candidateCount: topK,+ });+ return results;+ }++ const reordered = reorderTopK(head, scores, config.minScore);+ opts.onTelemetry?.({ status: "applied", candidateCount: topK });+ return [...reordered, ...tail];+}diff --git a/src/core/search/rerank/provider.ts b/src/core/search/rerank/provider.tsnew file mode 100644index 00000000..bda229ee--- /dev/null+++ b/src/core/search/rerank/provider.ts@@ -0,0 +1,48 @@+/**+ * Cross-encoder rerank provider abstraction (retrieval-precision-quality-loop,+ * card A / t_110867f5).+ *+ * A rerank provider jointly re-scores a query against a set of candidate+ * documents and returns one relevance score per document, aligned to the+ * input order. Unlike the embedding provider (which vectorises text for+ * the index), a rerank provider is a READ-TIME final reader step: it takes+ * the already-fused, heuristically-ranked top-K candidates and produces a+ * learned re-scoring the deterministic core cannot.+ *+ * The interface is intentionally minimal so a test can inject a+ * deterministic stub without any HTTP, and so alternate learned rerankers+ * (a local model, a different endpoint shape) can implement it later.+ *+ * Mirrors `embeddings/provider.ts`'s discipline: the module graph stays+ * small (the OpenAI-compatible implementation is imported lazily by the+ * factory) so a vault that never enables rerank pays nothing.+ */++import type { OpenAiCompatEndpoint } from "../embeddings/provider-resolve.ts";++export interface RerankProvider {+ readonly name: string;+ readonly model: string;+ /**+ * Score each document's relevance to the query. Returns an array of the+ * same length as `documents`, aligned by index. Higher is more relevant.+ * Throws on any provider/transport failure β the caller+ * ({@link applyCrossEncoderRerank}) is responsible for the fail-open.+ */+ rerank(query: string, documents: ReadonlyArray<string>): Promise<number[]>;+}++/**+ * Build the default OpenAI-compatible cross-encoder rerank provider for a+ * resolved endpoint. Lazily imported so the cross-encoder HTTP module is+ * only loaded when rerank is actually enabled and configured.+ */+export function makeRerankProvider(+ endpoint: OpenAiCompatEndpoint,+ opts?: { readonly timeoutMs?: number },+): RerankProvider {+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const { CrossEncoderRerankProvider } =+ require("./cross-encoder.ts") as typeof import("./cross-encoder.ts");+ return new CrossEncoderRerankProvider(endpoint, opts);+}diff --git a/src/core/search/rerank/registry.ts b/src/core/search/rerank/registry.tsnew file mode 100644index 00000000..2bf6d338--- /dev/null+++ b/src/core/search/rerank/registry.ts@@ -0,0 +1,161 @@+/**+ * Rerank-provider registry (retrieval-precision-quality-loop, card A /+ * t_110867f5).+ *+ * Named cross-encoder provider profiles persisted to+ * `Brain/search/rerank-providers.json`, added and removed at runtime+ * through the CLI so users register an OpenAI-compatible rerank endpoint+ * without editing config. A registered name resolves to `base_url ++ * model + env-key` at config-resolution time (see+ * {@link expandRegisteredRerankProvider}); an explicit `search_rerank_*`+ * config/env value always wins over the profile's field.+ *+ * Only env-key NAMES are stored, never secret values - the file is safe+ * to sync. Loading is fail-soft: an absent or malformed registry yields+ * an empty list and never throws into the hot path.+ *+ * Mirrors `embeddings/registry.ts` deliberately: the two registries are+ * the same shape so the CLI verbs, validation, and on-disk determinism+ * stay consistent between the embedding and rerank surfaces.+ */++import { mkdirSync, readFileSync, writeFileSync } from "node:fs";+import { dirname, join } from "node:path";++import { SearchError } from "../types.ts";++/** A registered OpenAI-compatible rerank provider profile. */+export interface RerankProviderProfile {+ readonly name: string;+ readonly baseUrl: string;+ readonly defaultModel: string;+ /** Environment variable NAME the API key is read from (never the key). */+ readonly envKey: string;+}++const NAME_RE = /^[a-z0-9][a-z0-9-]*$/;++export function rerankRegistryPath(vault: string): string {+ return join(vault, "Brain", "search", "rerank-providers.json");+}++function isProfile(value: unknown): value is RerankProviderProfile {+ if (value === null || typeof value !== "object") return false;+ const v = value as Record<string, unknown>;+ return (+ typeof v["name"] === "string" &&+ typeof v["baseUrl"] === "string" &&+ typeof v["defaultModel"] === "string" &&+ typeof v["envKey"] === "string"+ );+}++/** Load the registry, fail-soft: absent or malformed file -> empty list. */+export function loadRerankRegistry(vault: string): RerankProviderProfile[] {+ let text: string;+ try {+ text = readFileSync(rerankRegistryPath(vault), "utf8");+ } catch {+ return [];+ }+ let parsed: unknown;+ try {+ parsed = JSON.parse(text);+ } catch {+ return [];+ }+ if (!Array.isArray(parsed)) return [];+ return parsed.filter(isProfile).map((p) => Object.freeze({ ...p }));+}++function writeRerankRegistry(vault: string, registry: ReadonlyArray<RerankProviderProfile>): void {+ const path = rerankRegistryPath(vault);+ mkdirSync(dirname(path), { recursive: true });+ // Sort by name so the on-disk file is deterministic across edits.+ const sorted = [...registry].toSorted((a, b) => a.name.localeCompare(b.name));+ writeFileSync(path, JSON.stringify(sorted, null, 2) + "\n");+}++function validateProfile(profile: RerankProviderProfile): void {+ const name = profile.name.trim();+ if (!NAME_RE.test(name)) {+ throw new SearchError(+ "INVALID_INPUT",+ `rerank provider name must be a lowercase slug ([a-z0-9-], starting alphanumeric), got '${profile.name}'`,+ );+ }+ if (profile.baseUrl.trim() === "") {+ throw new SearchError("INVALID_INPUT", "rerank provider base-url must not be empty");+ }+ if (profile.defaultModel.trim() === "") {+ throw new SearchError("INVALID_INPUT", "rerank provider default-model must not be empty");+ }+ if (profile.envKey.trim() === "") {+ throw new SearchError("INVALID_INPUT", "rerank provider env-key must not be empty");+ }+}++/**+ * Add (or upsert by name) a rerank provider profile and persist. Returns+ * the full registry after the change.+ */+export function addRerankProviderProfile(+ vault: string,+ profile: RerankProviderProfile,+): RerankProviderProfile[] {+ validateProfile(profile);+ const normalised: RerankProviderProfile = Object.freeze({+ name: profile.name.trim(),+ baseUrl: profile.baseUrl.trim(),+ defaultModel: profile.defaultModel.trim(),+ envKey: profile.envKey.trim(),+ });+ const without = loadRerankRegistry(vault).filter((p) => p.name !== normalised.name);+ const next = [...without, normalised];+ writeRerankRegistry(vault, next);+ return loadRerankRegistry(vault);+}++/** Remove a profile by name; reports whether one was present. */+export function removeRerankProviderProfile(+ vault: string,+ name: string,+): { removed: boolean; registry: RerankProviderProfile[] } {+ const current = loadRerankRegistry(vault);+ const next = current.filter((p) => p.name !== name);+ const removed = next.length !== current.length;+ if (removed) writeRerankRegistry(vault, next);+ return { removed, registry: loadRerankRegistry(vault) };+}++export function getRerankProviderProfile(+ vault: string,+ name: string,+): RerankProviderProfile | null {+ return loadRerankRegistry(vault).find((p) => p.name === name) ?? null;+}++/** Fields a registered name expands into at config-resolution time. */+export interface ExpandedRerankProvider {+ readonly baseUrl: string;+ readonly model: string;+ readonly envKey: string;+}++/**+ * Expand a registered rerank provider name into its endpoint fields.+ * Returns null when the name is not registered (the caller then relies on+ * the explicit `search_rerank_*` config keys alone).+ */+export function expandRegisteredRerankProvider(+ name: string,+ registry: ReadonlyArray<RerankProviderProfile>,+): ExpandedRerankProvider | null {+ const profile = registry.find((p) => p.name === name);+ if (!profile) return null;+ return Object.freeze({+ baseUrl: profile.baseUrl,+ model: profile.defaultModel,+ envKey: profile.envKey,+ });+}diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex 6bcb1205..10b812df 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -80,6 +80,8 @@ import type { StructuredRecallQueryDocument } from "./structured-query.ts";import { expandQuery } from "./query-expansion.ts";import { applyTunedParameters, loadTunedParameters } from "./tuning.ts";import { resolveRecallProfile } from "./profiles.ts";+import { applyCrossEncoderRerank } from "./rerank/index.ts";+import { emitGatedTelemetry } from "../brain/continuity/emit.ts";interface SemanticPolicy {/** caller asked for semantic on or off (true), or accepted the default (false). */@@ -122,6 +124,14 @@ function configFingerprint(config: ResolvedSearchConfig): string {synMax: r.synonymMaxTerms,relPol: r.relationPolarityEnabled,lw: r.learnedWeightsEnabled,+ // Cross-encoder rerank re-orders the cached outcome, so a toggle or+ // knob change must invalidate cached rows (the endpoint identity is+ // omitted - a key rotation on the same model does not change ordering+ // semantics, and secrets never belong in a cache key).+ rrk: config.rerank.enabled,+ rrkModel: config.rerank.model,+ rrkTopK: config.rerank.topK,+ rrkMin: config.rerank.minScore,});}@@ -750,12 +760,20 @@ export async function search(// off by default keep results byte-identical.const scoreFloor = opts.threshold ?? 0;const rerankActive = opts.rerank === true;+ // Cross-encoder rerank (retrieval-precision-quality-loop, card A): the+ // learned final reader step re-scores the top-K fused candidates. When+ // it is on, the candidate pool must be at least `top_k` wide so a+ // genuinely-relevant hit the heuristic ranker placed deep can be pulled+ // into the final window. Off by default keeps the pool byte-identical.+ const crossEncoderActive = config.rerank.enabled;const maxHops = opts.maxHops ?? config.recall.maxHops;const traversalActive = maxHops > 0;const baseRankLimit =hasFrontmatterFilter || hasStructuredExclusions ? Math.max(limit * 5, 50) : limit;const rankLimit =- mmrActive || traversalActive ? Math.max(baseRankLimit, limit * 3, 30) : baseRankLimit;+ mmrActive || traversalActive || crossEncoderActive+ ? Math.max(baseRankLimit, limit * 3, 30, crossEncoderActive ? config.rerank.topK : 0)+ : baseRankLimit;// Rank β traverse β diversify β property filter β visibility scope,// for a given candidate cap. Returns the pre-visibility count, the@@ -895,7 +913,20 @@ export async function search(opts.reinforce !== undefined? applyReinforceBoost(polarized, loadReinforceStrengths(config.vault)): polarized;- const sliced = reinforced.slice(0, limit);+ // Cross-encoder rerank (retrieval-precision-quality-loop, card A): the+ // final reader step, appended AFTER every heuristic rerank. Disabled+ // (default) returns the pool unchanged (byte-identical); enabled but+ // unconfigured throws a typed config error; enabled + a request-time+ // endpoint error degrades to the heuristic ordering and records one+ // fail-open telemetry warning. Runs over the widened pool so a deep+ // candidate can be promoted into the final `limit` window below.+ const reranked = await applyCrossEncoderRerank(reinforced, query, config.rerank, {+ onTelemetry: (event) =>+ emitGatedTelemetry(event.status === "error", () => {+ warnings.push(`rerank_degraded: ${event.reason ?? "endpoint error"}`);+ }),+ });+ const sliced = reranked.slice(0, limit);// Explainability: when learned weights affected this ranking, every// surfaced result says so (acceptance: "search explanations show// when learned weights affected a result").diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex cd07f20c..4fa65815 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -22,6 +22,7 @@ export const SEARCH_ERROR_CODES = ["EMBEDDING_PROVIDER_TIMEOUT","EMBEDDING_DIMENSION_MISMATCH","EMBEDDING_COST_GATE",+ "RERANK_PROVIDER_HTTP","INDEX_LOCKED","INVALID_INPUT",] as const;@@ -585,6 +586,38 @@ export interface ResolvedEmbeddingConfig {readonly costGateUsd: number;}+/**+ * Optional cross-encoder rerank stage (retrieval-precision-quality-loop,+ * card A). A learned final reader step that re-scores the top-K fused+ * candidates jointly against the query, appended after the heuristic+ * reranks. Off by default (`enabled: false`), in which case the stage is+ * a zero-cost no-op and result ordering is byte-identical to the+ * pre-feature baseline. When enabled, the endpoint (OpenAI-compatible+ * `/rerank`) is resolved fail-closed through+ * `resolveOpenAiCompatEndpoint`; on any endpoint error the stage degrades+ * to the heuristic ordering and never throws into the hot path.+ */+export interface ResolvedRerankConfig {+ readonly enabled: boolean;+ /** OpenAI-compatible base URL (trailing slashes stripped) or null. */+ readonly baseUrl: string | null;+ readonly model: string | null;+ /** Env var NAME the API key is read from (never the key itself). */+ readonly envKey: string | null;+ /** API key resolved from `envKey` at config-resolution time, else null. */+ readonly apiKey: string | null;+ /** How many top fused candidates to re-score. Must be >= 1. */+ readonly topK: number;+ /**+ * Relevance floor in the cross-encoder's score space. A reranked+ * candidate scoring below this is not promoted above candidates the+ * heuristic ranker placed higher; it sinks below the qualifying+ * candidates but is never dropped (result count is preserved). Default+ * 0 promotes every candidate a non-negative-scoring endpoint returns.+ */+ readonly minScore: number;+}+/*** Recall-quality tunables (v0.13.0). Each layer is bounded and* deterministic; the defaults enable the layer while leaving a clear@@ -738,6 +771,12 @@ export interface ResolvedSearchConfig {readonly rrfK: number;readonly semantic: ResolvedEmbeddingConfig;readonly recall: ResolvedRecallConfig;+ /**+ * Optional cross-encoder rerank stage (retrieval-precision-quality-loop,+ * card A). Off by default; when off the reader tail is byte-identical to+ * the pre-feature baseline.+ */+ readonly rerank: ResolvedRerankConfig;/*** Grace window (ms) the `o2b search watch` shutdown waits for an* in-flight index pass to settle at a cooperative boundary beforediff --git a/tests/core/brain/doctor.entities.test.ts b/tests/core/brain/doctor.entities.test.tsindex 09ef40da..e15e938c 100644--- a/tests/core/brain/doctor.entities.test.ts+++ b/tests/core/brain/doctor.entities.test.ts@@ -61,6 +61,37 @@ describe("doctor entity lints", () => {expect(dup[0]!.message).toContain("people:ada");});+ test("alias-merge candidate lint is off by default (byte-identical report)", () => {+ upsertEntity(vault, { category: "org", name: "Google LLC", agent: "a", now: NOW });+ upsertEntity(vault, { category: "org", name: "Google Inc", agent: "a", now: NOW });+ const out = runDoctor(vault);+ expect(+ [...out.warnings, ...out.errors].filter((i) => i.code === "entity-alias-candidate"),+ ).toEqual([]);+ });++ test("enabled: lexical variants surface as an entity-alias-candidate warning", () => {+ upsertEntity(vault, { category: "org", name: "Google LLC", agent: "a", now: NOW });+ upsertEntity(vault, { category: "org", name: "Google Inc", agent: "a", now: NOW });+ const prevEnabled = process.env["OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_ENABLED"];+ const prevThreshold = process.env["OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_LEXICAL_THRESHOLD"];+ process.env["OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_ENABLED"] = "1";+ process.env["OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_LEXICAL_THRESHOLD"] = "0.3";+ try {+ const cand = runDoctor(vault).warnings.filter((i) => i.code === "entity-alias-candidate");+ expect(cand).toHaveLength(1);+ expect(cand[0]!.message).toContain("[lexical");+ expect(cand[0]!.message).toContain("never auto-merged");+ } finally {+ if (prevEnabled === undefined)+ delete process.env["OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_ENABLED"];+ else process.env["OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_ENABLED"] = prevEnabled;+ if (prevThreshold === undefined)+ delete process.env["OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_LEXICAL_THRESHOLD"];+ else process.env["OPEN_SECOND_BRAIN_ENTITY_SEMANTIC_DEDUP_LEXICAL_THRESHOLD"] = prevThreshold;+ }+ });+test("a relation pointing at a missing entity surfaces as broken-entity-relation", () => {upsertEntity(vault, { category: "people", name: "Ada", agent: "a", now: NOW });upsertEntity(vault, { category: "projects", name: "Open Second Brain", agent: "a", now: NOW });diff --git a/tests/core/brain/entities.semantic-dedup.test.ts b/tests/core/brain/entities.semantic-dedup.test.tsnew file mode 100644index 00000000..4e293f95--- /dev/null+++ b/tests/core/brain/entities.semantic-dedup.test.ts@@ -0,0 +1,226 @@+/**+ * Semantic (embedding cosine) entity dedup+ * (semantic-retrieval-precision, parent t_47fd9523).+ *+ * A proposal-only pass that surfaces lexical variants of the same+ * real-world entity ("Google LLC" vs "Google Inc") as alias-merge+ * CANDIDATES, complementing the deterministic `entityIdentityKey`. It+ * NEVER rewrites the identity key or any entity file.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, readFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { atomicWriteFileSync } from "../../../src/core/fs-atomic.ts";+import { bootstrapBrain } from "../../../src/core/brain/init.ts";+import { upsertEntity } from "../../../src/core/brain/entities/registry.ts";+import { buildEntityIndex } from "../../../src/core/brain/entities/index-builder.ts";+import { entityIdentityKey } from "../../../src/core/brain/entities/canonical.ts";+import {+ detectEntityAliasCandidates,+ entityLexicalAliasCandidates,+ entityEvolutionChain,+ deriveIdentityType,+ ENTITY_DEDUP_EMBEDDING_THRESHOLD,+} from "../../../src/core/brain/entities/semantic-dedup.ts";+import type { EmbeddingProvider } from "../../../src/core/search/embeddings/provider.ts";++let vault: string;+let configHome: string;++const NOW = new Date("2026-06-02T12:00:00Z");++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-entity-dedup-"));+ configHome = mkdtempSync(join(tmpdir(), "o2b-entity-dedup-cfg-"));+ const configPath = join(configHome, "config.yaml");+ atomicWriteFileSync(configPath, `vault: ${vault}\n`);+ bootstrapBrain(vault, { configPath });+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+ rmSync(configHome, { recursive: true, force: true });+});++function seed(category: string, name: string, aliases?: string[]): void {+ upsertEntity(vault, {+ category,+ name,+ ...(aliases ? { aliases } : {}),+ agent: "test-agent",+ now: NOW,+ });+}++/** Provider that maps specific names to fixed unit vectors. */+function fixedVectorProvider(vectors: Readonly<Record<string, number[]>>): EmbeddingProvider {+ return {+ name: "fixed",+ model: "fixed-test",+ dimension: 3,+ embed: (texts) => Promise.resolve(texts.map((t) => vectors[t] ?? [0, 0, 1])),+ ping: () => Promise.resolve({ ok: true as const, dimension: 3 }),+ };+}++describe("detectEntityAliasCandidates - embedding layer", () => {+ test("surfaces lexical variants as an embedding candidate; never rewrites the key", async () => {+ seed("org", "Google LLC");+ seed("org", "Google Inc");++ const before = buildEntityIndex(vault);+ const keyLlc = entityIdentityKey("org", "Google LLC");+ const keyInc = entityIdentityKey("org", "Google Inc");+ const raw = before.entities.map((e) => readFileSync(e.path, "utf8"));++ const provider = fixedVectorProvider({+ "Google LLC": [1, 0, 0],+ "Google Inc": [0.999, 0.0447, 0],+ });+ const result = await detectEntityAliasCandidates(vault, { provider });++ expect(result.method).toBe("embedding");+ expect(result.candidates).toHaveLength(1);+ const c = result.candidates[0]!;+ expect(c.method).toBe("embedding");+ expect(c.category).toBe("org");+ expect([c.name_a, c.name_b].toSorted()).toEqual(["Google Inc", "Google LLC"]);+ expect(c.similarity).toBeGreaterThanOrEqual(ENTITY_DEDUP_EMBEDDING_THRESHOLD);++ // Proposal-only: the deterministic keys and every entity file are unchanged.+ expect(entityIdentityKey("org", "Google LLC")).toBe(keyLlc);+ expect(entityIdentityKey("org", "Google Inc")).toBe(keyInc);+ const after = buildEntityIndex(vault);+ expect(after.entities.map((e) => readFileSync(e.path, "utf8"))).toEqual(raw);+ });++ test("distinct entities below the threshold are not surfaced", async () => {+ seed("org", "Google LLC");+ seed("org", "Amazon Inc");+ const provider = fixedVectorProvider({+ "Google LLC": [1, 0, 0],+ "Amazon Inc": [0, 1, 0],+ });+ const result = await detectEntityAliasCandidates(vault, { provider });+ expect(result.method).toBe("embedding");+ expect(result.candidates).toHaveLength(0);+ });++ test("does not pair entities in different categories", async () => {+ seed("org", "Mercury");+ seed("people", "Mercury Jones");+ const provider = fixedVectorProvider({+ Mercury: [1, 0, 0],+ "Mercury Jones": [1, 0, 0],+ });+ const result = await detectEntityAliasCandidates(vault, { provider });+ expect(result.candidates).toHaveLength(0);+ });++ test("does not re-nominate a pair already linked by an alias", async () => {+ seed("org", "Google LLC", ["Google Inc"]);+ seed("org", "Google Inc"); // resolves to the same entity via alias β one file only+ const provider = fixedVectorProvider({ "Google LLC": [1, 0, 0] });+ const result = await detectEntityAliasCandidates(vault, { provider });+ expect(result.candidates).toHaveLength(0);+ });++ test("candidate emission is deterministic across two runs", async () => {+ seed("org", "Google LLC");+ seed("org", "Google Inc");+ seed("org", "Google Incorporated");+ const provider = fixedVectorProvider({+ "Google LLC": [1, 0, 0],+ "Google Inc": [0.999, 0.0447, 0],+ "Google Incorporated": [0.998, 0.0632, 0],+ });+ const a = await detectEntityAliasCandidates(vault, { provider });+ const b = await detectEntityAliasCandidates(vault, { provider });+ expect(a).toEqual(b);+ // Stable order by (category, a, b) β field-wise, not joined-string.+ const keys = a.candidates.map((c) => [c.category, c.a, c.b]);+ const sorted = keys.toSorted(+ (x, y) =>+ x[0]!.localeCompare(y[0]!) || x[1]!.localeCompare(y[1]!) || x[2]!.localeCompare(y[2]!),+ );+ expect(keys).toEqual(sorted);+ });+});++describe("detectEntityAliasCandidates - lexical fallback", () => {+ test("no provider β lexical fallback, clearly labeled method:lexical", async () => {+ seed("org", "Google LLC");+ seed("org", "Google Inc");+ const result = await detectEntityAliasCandidates(vault, {+ provider: null,+ lexicalThreshold: 0.3,+ });+ expect(result.method).toBe("lexical");+ expect(result.candidates.length).toBeGreaterThanOrEqual(1);+ expect(result.candidates.every((c) => c.method === "lexical")).toBe(true);+ });++ test("provider that throws falls back to lexical, never throws", async () => {+ seed("org", "Google LLC");+ seed("org", "Google Inc");+ const boom: EmbeddingProvider = {+ name: "boom",+ model: "boom",+ dimension: 3,+ embed: () => Promise.reject(new Error("provider down")),+ ping: () => Promise.resolve({ ok: false as const, reason: "down" }),+ };+ const result = await detectEntityAliasCandidates(vault, {+ provider: boom,+ lexicalThreshold: 0.3,+ });+ expect(result.method).toBe("lexical");+ });++ test("unconfigured vault (provider undefined) resolves to null β lexical, no throw", async () => {+ seed("org", "Google LLC");+ seed("org", "Google Inc");+ // No embedding provider configured for this vault: real resolution+ // returns null and the pass degrades to the deterministic layer.+ const result = await detectEntityAliasCandidates(vault, { lexicalThreshold: 0.3 });+ expect(result.method).toBe("lexical");+ });++ test("entityLexicalAliasCandidates is synchronous and pure-read", () => {+ seed("org", "Google LLC");+ seed("org", "Google Inc");+ const before = buildEntityIndex(vault).entities.map((e) => readFileSync(e.path, "utf8"));+ const candidates = entityLexicalAliasCandidates(vault, { threshold: 0.3 });+ expect(candidates.length).toBeGreaterThanOrEqual(1);+ const after = buildEntityIndex(vault).entities.map((e) => readFileSync(e.path, "utf8"));+ expect(after).toEqual(before);+ });+});++describe("identity_type + evolution-chain helpers", () => {+ test("identity_type is absent by default", () => {+ seed("org", "Google LLC");+ const entity = buildEntityIndex(vault).entities[0]!;+ expect(deriveIdentityType(entity)).toBeUndefined();+ });++ test("identity_type derives from a frontmatter/structural signal", () => {+ seed("org", "Google LLC");+ const entity = buildEntityIndex(vault).entities[0]!;+ // Append the structural identity_type frontmatter key by hand.+ const raw = readFileSync(entity.path, "utf8");+ const withType = raw.replace(/^name: .*$/m, (m) => `${m}\nidentity_type: org`);+ atomicWriteFileSync(entity.path, withType);+ const reloaded = buildEntityIndex(vault).entities[0]!;+ expect(deriveIdentityType(reloaded)).toBe("org");+ });++ test("evolution chain is the canonical name followed by its aliases", () => {+ seed("org", "Google LLC", ["Google Inc", "Google"]);+ const entity = buildEntityIndex(vault).entities[0]!;+ expect(entityEvolutionChain(entity)).toEqual(["Google LLC", "Google Inc", "Google"]);+ });+});diff --git a/tests/core/search/embeddings.provider-resolve.test.ts b/tests/core/search/embeddings.provider-resolve.test.tsnew file mode 100644index 00000000..e493ecff--- /dev/null+++ b/tests/core/search/embeddings.provider-resolve.test.ts@@ -0,0 +1,84 @@+/**+ * Provider-resolution / no-op seam (semantic-retrieval-precision, parent+ * t_47fd9523). The parent lands this helper as the seam the child+ * cross-encoder reuses, so its fail-closed-validation + graceful-no-op+ * discipline is tested here directly.+ */++import { describe, expect, test } from "bun:test";++import { SearchError } from "../../../src/core/search/types.ts";+import { resolveOpenAiCompatEndpoint } from "../../../src/core/search/embeddings/provider-resolve.ts";++describe("resolveOpenAiCompatEndpoint", () => {+ test("disabled β graceful no-op (null), no validation", () => {+ expect(+ resolveOpenAiCompatEndpoint({ enabled: false, baseUrl: null, model: null }, "search_rerank"),+ ).toBeNull();+ });++ test("enabled + missing base_url fails closed with a clear error", () => {+ expect(() =>+ resolveOpenAiCompatEndpoint(+ { enabled: true, baseUrl: null, model: "m", apiKey: "k" },+ "search_rerank",+ ),+ ).toThrow(/search_rerank_base_url is required/);+ });++ test("enabled + missing model fails closed", () => {+ expect(() =>+ resolveOpenAiCompatEndpoint(+ { enabled: true, baseUrl: "https://x/v1", model: null, apiKey: "k" },+ "search_rerank",+ ),+ ).toThrow(/search_rerank_model is required/);+ });++ test("enabled + missing api key fails closed with EMBEDDING_KEY_MISSING", () => {+ try {+ resolveOpenAiCompatEndpoint(+ { enabled: true, baseUrl: "https://x/v1", model: "m", apiKey: null },+ "search_rerank",+ );+ throw new Error("expected throw");+ } catch (e) {+ expect(e).toBeInstanceOf(SearchError);+ expect((e as SearchError).code).toBe("EMBEDDING_KEY_MISSING");+ }+ });++ test("resolves the api key from the named env variable", () => {+ const resolved = resolveOpenAiCompatEndpoint(+ {+ enabled: true,+ baseUrl: "https://x/v1/",+ model: "rerank-1",+ envKey: "MY_RERANK_KEY",+ env: { MY_RERANK_KEY: "secret-value" },+ },+ "search_rerank",+ );+ expect(resolved).not.toBeNull();+ expect(resolved).toEqual({+ baseUrl: "https://x/v1",+ model: "rerank-1",+ apiKey: "secret-value",+ });+ });++ test("a direct api key wins over the env lookup", () => {+ const resolved = resolveOpenAiCompatEndpoint(+ {+ enabled: true,+ baseUrl: "https://x/v1",+ model: "m",+ apiKey: "direct",+ envKey: "MY_RERANK_KEY",+ env: { MY_RERANK_KEY: "from-env" },+ },+ "search_rerank",+ );+ expect(resolved?.apiKey).toBe("direct");+ });+});diff --git a/tests/core/search/rerank.config.test.ts b/tests/core/search/rerank.config.test.tsnew file mode 100644index 00000000..8b3dcd61--- /dev/null+++ b/tests/core/search/rerank.config.test.ts@@ -0,0 +1,159 @@+/**+ * Cross-encoder rerank config resolution (retrieval-precision-quality-loop,+ * card A / t_110867f5). The `search_rerank_*` keys resolve through+ * `resolveSearchConfig` (config file + env parity), a registered provider+ * name supplies endpoint defaults, and explicit keys win.+ */++import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, writeFileSync, rmSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { resolveSearchConfig } from "../../../src/core/search/index.ts";+import { addRerankProviderProfile } from "../../../src/core/search/rerank/registry.ts";+import { SearchError } from "../../../src/core/search/types.ts";++let tmp: string;+let configPath: string;++const ENV_KEYS = [+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_ENABLED",+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_PROVIDER",+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_BASE_URL",+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_MODEL",+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_ENV_KEY",+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_TOP_K",+ "OPEN_SECOND_BRAIN_SEARCH_RERANK_MIN_SCORE",+ "MY_RERANK_KEY",+];+let origEnv: Record<string, string | undefined>;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "osb-rerank-cfg-"));+ configPath = join(tmp, "config.yaml");+ origEnv = {};+ for (const k of ENV_KEYS) {+ origEnv[k] = process.env[k];+ delete process.env[k];+ }+});+afterEach(() => {+ for (const k of ENV_KEYS) {+ if (origEnv[k] === undefined) delete process.env[k];+ else process.env[k] = origEnv[k];+ }+ rmSync(tmp, { recursive: true, force: true });+});++test("default: rerank disabled, byte-identical zero-cost stage", () => {+ writeFileSync(configPath, "vault: " + tmp + "\n");+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.rerank.enabled).toBe(false);+ expect(cfg.rerank.baseUrl).toBeNull();+ expect(cfg.rerank.topK).toBe(20);+ expect(cfg.rerank.minScore).toBe(0);+});++test("explicit config keys resolve, with the api key read from env-key", () => {+ process.env["MY_RERANK_KEY"] = "sk-rerank";+ writeFileSync(+ configPath,+ [+ "vault: " + tmp,+ "search_rerank_enabled: true",+ "search_rerank_base_url: https://api.example.com/v1/",+ "search_rerank_model: rerank-1",+ "search_rerank_env_key: MY_RERANK_KEY",+ "search_rerank_top_k: 8",+ "search_rerank_min_score: 0.25",+ ].join("\n") + "\n",+ );+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.rerank).toEqual({+ enabled: true,+ baseUrl: "https://api.example.com/v1/",+ model: "rerank-1",+ envKey: "MY_RERANK_KEY",+ apiKey: "sk-rerank",+ topK: 8,+ minScore: 0.25,+ });+});++test("a registered provider name supplies endpoint defaults", () => {+ addRerankProviderProfile(tmp, {+ name: "jina",+ baseUrl: "https://api.jina.ai/v1",+ defaultModel: "jina-reranker-v2",+ envKey: "MY_RERANK_KEY",+ });+ process.env["MY_RERANK_KEY"] = "jina-secret";+ writeFileSync(+ configPath,+ ["vault: " + tmp, "search_rerank_enabled: true", "search_rerank_provider: jina"].join("\n") ++ "\n",+ );+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.rerank.baseUrl).toBe("https://api.jina.ai/v1");+ expect(cfg.rerank.model).toBe("jina-reranker-v2");+ expect(cfg.rerank.envKey).toBe("MY_RERANK_KEY");+ expect(cfg.rerank.apiKey).toBe("jina-secret");+});++test("explicit keys win over a registered provider profile", () => {+ addRerankProviderProfile(tmp, {+ name: "jina",+ baseUrl: "https://api.jina.ai/v1",+ defaultModel: "jina-reranker-v2",+ envKey: "MY_RERANK_KEY",+ });+ writeFileSync(+ configPath,+ [+ "vault: " + tmp,+ "search_rerank_enabled: true",+ "search_rerank_provider: jina",+ "search_rerank_model: override-model",+ ].join("\n") + "\n",+ );+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.rerank.model).toBe("override-model");+ expect(cfg.rerank.baseUrl).toBe("https://api.jina.ai/v1"); // still from profile+});++test("env var overrides the config file", () => {+ writeFileSync(configPath, ["vault: " + tmp, "search_rerank_top_k: 5"].join("\n") + "\n");+ process.env["OPEN_SECOND_BRAIN_SEARCH_RERANK_TOP_K"] = "12";+ const cfg = resolveSearchConfig({ vault: tmp, configPath });+ expect(cfg.rerank.topK).toBe(12);+});++test("a non-integer top_k fails closed", () => {+ writeFileSync(configPath, ["vault: " + tmp, "search_rerank_top_k: notanumber"].join("\n") + "\n");+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(SearchError);+});++test("overrides.rerank accepts a partial config and merges it over the base", () => {+ // Resolves the base with rerank enabled so every field is populated, then+ // overrides only topK. The merged config keeps base.baseUrl/model but takes+ // the partial topK. This also guards the type: a partial rerank override+ // (e.g. { topK: 5 }) must type-check like the semantic override does.+ process.env["MY_RERANK_KEY"] = "sk-rerank";+ writeFileSync(+ configPath,+ [+ "vault: " + tmp,+ "search_rerank_enabled: true",+ "search_rerank_base_url: https://api.example.com/v1/",+ "search_rerank_model: rerank-1",+ "search_rerank_env_key: MY_RERANK_KEY",+ "search_rerank_top_k: 20",+ "search_rerank_min_score: 0",+ ].join("\n") + "\n",+ );+ const cfg = resolveSearchConfig({ vault: tmp, configPath, overrides: { rerank: { topK: 5 } } });+ expect(cfg.rerank.topK).toBe(5);+ expect(cfg.rerank.baseUrl).toBe("https://api.example.com/v1/");+ expect(cfg.rerank.model).toBe("rerank-1");+});diff --git a/tests/core/search/rerank.cross-encoder.test.ts b/tests/core/search/rerank.cross-encoder.test.tsnew file mode 100644index 00000000..15ea174b--- /dev/null+++ b/tests/core/search/rerank.cross-encoder.test.ts@@ -0,0 +1,132 @@+/**+ * OpenAI-compatible cross-encoder rerank provider HTTP shape+ * (retrieval-precision-quality-loop, card A / t_110867f5).+ *+ * Exercises the `/rerank` request/response contract with a stubbed+ * `fetch`, covering both the wrapped `{ results: [...] }` and the bare+ * array response shapes, index realignment, and provider-shaped errors.+ */++import { afterEach, describe, expect, test } from "bun:test";++import { CrossEncoderRerankProvider } from "../../../src/core/search/rerank/cross-encoder.ts";+import { SearchError } from "../../../src/core/search/types.ts";++const ENDPOINT = Object.freeze({+ baseUrl: "https://api.example.com/v1",+ model: "rerank-1",+ apiKey: "secret",+});++const realFetch = globalThis.fetch;+afterEach(() => {+ globalThis.fetch = realFetch;+});++function stubFetch(handler: (url: string, init: RequestInit) => Response): void {+ globalThis.fetch = (async (url: string, init: RequestInit) =>+ handler(String(url), init)) as unknown as typeof fetch;+}++describe("CrossEncoderRerankProvider", () => {+ test("posts to {base}/rerank with model, query, documents and bearer auth", async () => {+ let seenUrl = "";+ let seenBody: Record<string, unknown> = {};+ let seenAuth = "";+ stubFetch((url, init) => {+ seenUrl = url;+ seenAuth = (init.headers as Record<string, string>)["authorization"] ?? "";+ seenBody = JSON.parse(String(init.body));+ return new Response(+ JSON.stringify({+ results: [+ { index: 0, relevance_score: 0.2 },+ { index: 1, relevance_score: 0.9 },+ ],+ }),+ { status: 200 },+ );+ });+ const provider = new CrossEncoderRerankProvider(ENDPOINT);+ const scores = await provider.rerank("hello", ["doc a", "doc b"]);+ expect(seenUrl).toBe("https://api.example.com/v1/rerank");+ expect(seenAuth).toBe("Bearer secret");+ expect(seenBody).toEqual({ model: "rerank-1", query: "hello", documents: ["doc a", "doc b"] });+ expect(scores).toEqual([0.2, 0.9]);+ });++ test("realigns out-of-order results back to input index order", async () => {+ stubFetch(+ () =>+ new Response(+ JSON.stringify({+ results: [+ { index: 2, relevance_score: 0.9 },+ { index: 0, relevance_score: 0.1 },+ { index: 1, relevance_score: 0.5 },+ ],+ }),+ { status: 200 },+ ),+ );+ const provider = new CrossEncoderRerankProvider(ENDPOINT);+ const scores = await provider.rerank("q", ["a", "b", "c"]);+ expect(scores).toEqual([0.1, 0.5, 0.9]);+ });++ test("accepts the bare array response shape with `score`", async () => {+ stubFetch(+ () =>+ new Response(+ JSON.stringify([+ { index: 0, score: 0.7 },+ { index: 1, score: 0.3 },+ ]),+ { status: 200 },+ ),+ );+ const provider = new CrossEncoderRerankProvider(ENDPOINT);+ expect(await provider.rerank("q", ["a", "b"])).toEqual([0.7, 0.3]);+ });++ test("empty documents short-circuits without a request", async () => {+ let called = false;+ stubFetch(() => {+ called = true;+ return new Response("[]", { status: 200 });+ });+ const provider = new CrossEncoderRerankProvider(ENDPOINT);+ expect(await provider.rerank("q", [])).toEqual([]);+ expect(called).toBe(false);+ });++ test("non-2xx surfaces a RERANK_PROVIDER_HTTP error", async () => {+ stubFetch(() => new Response("upstream boom", { status: 503 }));+ const provider = new CrossEncoderRerankProvider(ENDPOINT);+ try {+ await provider.rerank("q", ["a"]);+ throw new Error("expected throw");+ } catch (e) {+ expect(e).toBeInstanceOf(SearchError);+ expect((e as SearchError).code).toBe("RERANK_PROVIDER_HTTP");+ expect((e as SearchError).message).toMatch(/503/);+ }+ });++ test("a wrong-length response is rejected", async () => {+ stubFetch(+ () =>+ new Response(JSON.stringify({ results: [{ index: 0, relevance_score: 0.5 }] }), {+ status: 200,+ }),+ );+ const provider = new CrossEncoderRerankProvider(ENDPOINT);+ let thrown: unknown;+ try {+ await provider.rerank("q", ["a", "b"]);+ } catch (e) {+ thrown = e;+ }+ expect(thrown).toBeInstanceOf(SearchError);+ });+});diff --git a/tests/core/search/rerank.registry.test.ts b/tests/core/search/rerank.registry.test.tsnew file mode 100644index 00000000..7b317d39--- /dev/null+++ b/tests/core/search/rerank.registry.test.ts@@ -0,0 +1,134 @@+/**+ * Rerank-provider registry (retrieval-precision-quality-loop, card A /+ * t_110867f5). Mirrors the embedding registry: env-key NAMES only,+ * fail-soft load, deterministic on-disk order.+ */++import { afterEach, beforeEach, describe, expect, test } from "bun:test";+import { mkdtempSync, rmSync, writeFileSync, mkdirSync, readFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ addRerankProviderProfile,+ expandRegisteredRerankProvider,+ getRerankProviderProfile,+ loadRerankRegistry,+ removeRerankProviderProfile,+ rerankRegistryPath,+} from "../../../src/core/search/rerank/registry.ts";+import { SearchError } from "../../../src/core/search/types.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "rerank-registry-"));+});+afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++describe("rerank registry", () => {+ test("absent registry loads fail-soft to an empty list", () => {+ expect(loadRerankRegistry(vault)).toEqual([]);+ });++ test("add persists a profile and round-trips through load", () => {+ addRerankProviderProfile(vault, {+ name: "jina",+ baseUrl: "https://api.jina.ai/v1",+ defaultModel: "jina-reranker-v2",+ envKey: "JINA_API_KEY",+ });+ const loaded = loadRerankRegistry(vault);+ expect(loaded).toHaveLength(1);+ expect(loaded[0]).toEqual({+ name: "jina",+ baseUrl: "https://api.jina.ai/v1",+ defaultModel: "jina-reranker-v2",+ envKey: "JINA_API_KEY",+ });+ });++ test("stored file contains the env-key NAME, never a secret value", () => {+ addRerankProviderProfile(vault, {+ name: "cohere",+ baseUrl: "https://api.cohere.ai/v1",+ defaultModel: "rerank-english-v3.0",+ envKey: "COHERE_API_KEY",+ });+ const raw = readFileSync(rerankRegistryPath(vault), "utf8");+ expect(raw).toContain("COHERE_API_KEY");+ expect(raw).not.toContain("secret");+ });++ test("add upserts by name and keeps the file name-sorted", () => {+ addRerankProviderProfile(vault, {+ name: "zeta",+ baseUrl: "https://z/v1",+ defaultModel: "m",+ envKey: "Z_KEY",+ });+ addRerankProviderProfile(vault, {+ name: "alpha",+ baseUrl: "https://a/v1",+ defaultModel: "m",+ envKey: "A_KEY",+ });+ addRerankProviderProfile(vault, {+ name: "alpha",+ baseUrl: "https://a2/v1",+ defaultModel: "m2",+ envKey: "A_KEY2",+ });+ const loaded = loadRerankRegistry(vault);+ expect(loaded.map((p) => p.name)).toEqual(["alpha", "zeta"]);+ expect(loaded.find((p) => p.name === "alpha")?.baseUrl).toBe("https://a2/v1");+ });++ test("remove reports presence and deletes the profile", () => {+ addRerankProviderProfile(vault, {+ name: "jina",+ baseUrl: "https://api.jina.ai/v1",+ defaultModel: "jina-reranker-v2",+ envKey: "JINA_API_KEY",+ });+ expect(removeRerankProviderProfile(vault, "nope").removed).toBe(false);+ expect(removeRerankProviderProfile(vault, "jina").removed).toBe(true);+ expect(getRerankProviderProfile(vault, "jina")).toBeNull();+ });++ test("invalid name is rejected with a typed error", () => {+ expect(() =>+ addRerankProviderProfile(vault, {+ name: "Bad Name",+ baseUrl: "https://x/v1",+ defaultModel: "m",+ envKey: "K",+ }),+ ).toThrow(SearchError);+ });++ test("a malformed registry file loads fail-soft to empty", () => {+ mkdirSync(join(vault, "Brain", "search"), { recursive: true });+ writeFileSync(rerankRegistryPath(vault), "{ not json");+ expect(loadRerankRegistry(vault)).toEqual([]);+ });++ test("expandRegisteredRerankProvider resolves endpoint fields by name", () => {+ const registry = [+ {+ name: "jina",+ baseUrl: "https://api.jina.ai/v1",+ defaultModel: "jina-reranker-v2",+ envKey: "JINA_API_KEY",+ },+ ];+ expect(expandRegisteredRerankProvider("jina", registry)).toEqual({+ baseUrl: "https://api.jina.ai/v1",+ model: "jina-reranker-v2",+ envKey: "JINA_API_KEY",+ });+ expect(expandRegisteredRerankProvider("missing", registry)).toBeNull();+ });+});diff --git a/tests/core/search/rerank.search.test.ts b/tests/core/search/rerank.search.test.tsnew file mode 100644index 00000000..fb158f5c--- /dev/null+++ b/tests/core/search/rerank.search.test.ts@@ -0,0 +1,95 @@+/**+ * Cross-encoder rerank wired through `search()`+ * (retrieval-precision-quality-loop, card A / t_110867f5).+ *+ * End-to-end: config-enabled rerank widens the candidate pool, calls the+ * OpenAI-compatible `/rerank` endpoint (stubbed `fetch`), and re-orders+ * the final window. Disabled stays byte-identical; enabled-but-unconfigured+ * fails closed; an endpoint error degrades gracefully with a warning.+ */++import { test, expect, beforeEach, afterEach } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++const realFetch = globalThis.fetch;++beforeEach(() => {+ ({ vault, dbPath, cleanup } = createTempVault("rerank-search"));+ writeMd(vault, "strong.md", "# Strong\n\nfox fox fox fox the quick brown fox jumps high.");+ writeMd(vault, "weak.md", "# Weak\n\nA note mostly about cats, with one fox mention here.");+ writeMd(vault, "mid.md", "# Mid\n\nThe fox and the hound; a fox story about foxes.");+});+afterEach(() => {+ globalThis.fetch = realFetch;+ cleanup();+});++const ENABLED = {+ enabled: true,+ baseUrl: "https://api.example.com/v1",+ model: "rerank-1",+ apiKey: "secret",+ topK: 10,+} as const;++/** Score the document containing `winner` highest, everything else low. */+function stubRerankFetch(winner: string): void {+ globalThis.fetch = (async (_url: string, init: RequestInit) => {+ const body = JSON.parse(String(init.body)) as { documents: string[] };+ const results = body.documents.map((doc, index) => ({+ index,+ relevance_score: doc.includes(winner) ? 0.99 : 0.01,+ }));+ return new Response(JSON.stringify({ results }), { status: 200 });+ }) as unknown as typeof fetch;+}++test("disabled (default) is byte-identical to the pre-feature baseline", async () => {+ const cfg = makeConfig({ vault, dbPath });+ await indexVault(cfg);+ let called = false;+ globalThis.fetch = (async () => {+ called = true;+ return new Response("[]", { status: 200 });+ }) as unknown as typeof fetch;+ const base = await search(cfg, { query: "fox", limit: 10 });+ expect(called).toBe(false); // zero HTTP cost when disabled+ expect(base.results.length).toBeGreaterThanOrEqual(3);+ // The heuristic ranker puts the dense "strong" hit first.+ expect(base.results[0]!.path).toBe("strong.md");+});++test("enabled: the cross-encoder's top-scored doc lands first", async () => {+ const cfg = makeConfig({ vault, dbPath, rerank: ENABLED });+ await indexVault(cfg);+ // Promote the "cats" note (weak.md) that the heuristic ranker placed low.+ stubRerankFetch("cats");+ const out = await search(cfg, { query: "fox", limit: 10 });+ expect(out.results[0]!.path).toBe("weak.md");+ expect(out.results[0]!.reasons.some((r) => r.startsWith("cross_encoder: "))).toBe(true);+});++test("enabled + unconfigured endpoint fails closed", async () => {+ const cfg = makeConfig({ vault, dbPath, rerank: { enabled: true } });+ await indexVault(cfg);+ await expect(search(cfg, { query: "fox", limit: 10 })).rejects.toBeInstanceOf(SearchError);+});++test("enabled + endpoint error degrades to heuristic order with a warning", async () => {+ const cfg = makeConfig({ vault, dbPath, rerank: ENABLED });+ await indexVault(cfg);+ globalThis.fetch = (async () =>+ new Response("upstream boom", { status: 503 })) as unknown as typeof fetch;+ const base = await search(makeConfig({ vault, dbPath }), { query: "fox", limit: 10 });+ const out = await search(cfg, { query: "fox", limit: 10 });+ expect(out.results.map((r) => r.path)).toEqual(base.results.map((r) => r.path));+ expect(out.warnings.some((w) => w.startsWith("rerank_degraded:"))).toBe(true);+});diff --git a/tests/core/search/rerank.test.ts b/tests/core/search/rerank.test.tsnew file mode 100644index 00000000..69183867--- /dev/null+++ b/tests/core/search/rerank.test.ts@@ -0,0 +1,188 @@+/**+ * Optional pluggable cross-encoder rerank stage+ * (retrieval-precision-quality-loop, card A / t_110867f5).+ *+ * Covers the four acceptance cases from the plan:+ * 1. Disabled (default): input returned unchanged (byte-identical).+ * 2. Enabled + unconfigured: fail closed with a clear typed error.+ * 3. Enabled + endpoint returns scores: top-K re-ordered; a doc the+ * heuristic ranker placed 3rd but scored highest lands 1st.+ * 4. Enabled + endpoint errors: degrades to the heuristic input (no+ * throw) and emits exactly one fail-open telemetry event.+ */++import { describe, expect, test } from "bun:test";++import {+ applyCrossEncoderRerank,+ type RerankTelemetryEvent,+} from "../../../src/core/search/rerank/index.ts";+import type { RerankProvider } from "../../../src/core/search/rerank/provider.ts";+import { SearchError } from "../../../src/core/search/types.ts";+import type { BrainSearchResult, ResolvedRerankConfig } from "../../../src/core/search/types.ts";++function result(id: number, content: string): BrainSearchResult {+ return Object.freeze({+ documentId: id,+ chunkId: id,+ path: `note-${id}.md`,+ title: `Note ${id}`,+ content,+ startLine: 1,+ endLine: 2,+ score: 1 - id * 0.1,+ keywordScore: 0.5,+ semanticScore: 0.5,+ linkBoost: 0,+ recencyBoost: 0,+ searchType: "hybrid" as const,+ reasons: Object.freeze([`fts5_bm25: 0.500`]),+ });+}++const BASE: ResolvedRerankConfig = Object.freeze({+ enabled: false,+ baseUrl: null,+ model: null,+ envKey: null,+ apiKey: null,+ topK: 20,+ minScore: 0,+});++/** A deterministic stub returning caller-supplied scores by document index. */+function stubProvider(scoresByContent: (content: string, i: number) => number): RerankProvider {+ return {+ name: "stub",+ model: "stub-1",+ async rerank(_query, documents) {+ return documents.map((d, i) => scoresByContent(d, i));+ },+ };+}++describe("applyCrossEncoderRerank", () => {+ test("disabled (default) returns the exact input reference unchanged", async () => {+ const input = Object.freeze([result(1, "a"), result(2, "b"), result(3, "c")]);+ const out = await applyCrossEncoderRerank(input, "q", BASE);+ // Same reference: no allocation, byte-identical ordering.+ expect(out).toBe(input);+ });++ test("enabled but unconfigured fails closed with a clear typed error", async () => {+ const cfg = Object.freeze({ ...BASE, enabled: true });+ let thrown: unknown;+ try {+ await applyCrossEncoderRerank([result(1, "a")], "q", cfg, { env: {} });+ } catch (e) {+ thrown = e;+ }+ expect(thrown).toBeInstanceOf(SearchError);+ expect((thrown as SearchError).message).toMatch(/search_rerank_base_url is required/);+ });++ test("enabled + scores: the highest-scored doc lands first", async () => {+ const cfg = Object.freeze({+ ...BASE,+ enabled: true,+ baseUrl: "https://x/v1",+ model: "rerank-1",+ apiKey: "k",+ });+ const input = [result(1, "first"), result(2, "second"), result(3, "third")];+ // The heuristic order is [1, 2, 3]; the cross-encoder scores doc #3+ // (originally 3rd) highest, so it must land 1st.+ const provider = stubProvider((content) =>+ content === "third" ? 0.99 : content === "first" ? 0.5 : 0.1,+ );+ const out = await applyCrossEncoderRerank(input, "q", cfg, { provider });+ expect(out.map((r) => r.documentId)).toEqual([3, 1, 2]);+ // Every reranked hit gains the explainability reason.+ expect(out[0]!.reasons.some((r) => r.startsWith("cross_encoder: "))).toBe(true);+ });++ test("enabled + endpoint error: degrades to input and emits one telemetry event", async () => {+ const cfg = Object.freeze({+ ...BASE,+ enabled: true,+ baseUrl: "https://x/v1",+ model: "rerank-1",+ apiKey: "k",+ });+ const input = [result(1, "a"), result(2, "b")];+ const provider: RerankProvider = {+ name: "boom",+ model: "boom-1",+ async rerank() {+ throw new SearchError("RERANK_PROVIDER_HTTP", "rerank HTTP 503: unavailable");+ },+ };+ const events: RerankTelemetryEvent[] = [];+ const out = await applyCrossEncoderRerank(input, "q", cfg, {+ provider,+ onTelemetry: (e) => events.push(e),+ });+ expect(out.map((r) => r.documentId)).toEqual([1, 2]); // unchanged order+ expect(events).toHaveLength(1);+ expect(events[0]!.status).toBe("error");+ expect(events[0]!.reason).toMatch(/503/);+ });++ test("minScore floor: a below-floor doc sinks below qualifying ones but is kept", async () => {+ const cfg = Object.freeze({+ ...BASE,+ enabled: true,+ baseUrl: "https://x/v1",+ model: "rerank-1",+ apiKey: "k",+ minScore: 0.3,+ });+ const input = [result(1, "a"), result(2, "b"), result(3, "c")];+ // doc #1 below floor, #2 and #3 above; #3 highest.+ const provider = stubProvider((content) =>+ content === "a" ? 0.1 : content === "b" ? 0.4 : 0.9,+ );+ const out = await applyCrossEncoderRerank(input, "q", cfg, { provider });+ // qualifying sorted desc: [3, 2], then below-floor in original order: [1].+ expect(out.map((r) => r.documentId)).toEqual([3, 2, 1]);+ });++ test("only the top-K are re-scored; the tail keeps its order and text", async () => {+ const cfg = Object.freeze({+ ...BASE,+ enabled: true,+ baseUrl: "https://x/v1",+ model: "rerank-1",+ apiKey: "k",+ topK: 2,+ });+ const input = [result(1, "a"), result(2, "b"), result(3, "c"), result(4, "d")];+ // Reverse the top-2 by score; docs 3,4 are the untouched tail.+ const seenDocs: string[][] = [];+ const provider: RerankProvider = {+ name: "stub",+ model: "stub-1",+ async rerank(_q, documents) {+ seenDocs.push([...documents]);+ return documents.map((_d, i) => (i === 0 ? 0.1 : 0.9));+ },+ };+ const out = await applyCrossEncoderRerank(input, "q", cfg, { provider });+ expect(seenDocs[0]).toEqual(["a", "b"]); // only top-K handed to the model+ expect(out.map((r) => r.documentId)).toEqual([2, 1, 3, 4]);+ // Tail entries are untouched (no cross_encoder reason).+ expect(out[2]!.reasons.some((r) => r.startsWith("cross_encoder: "))).toBe(false);+ });++ test("empty input is a no-op even when enabled+configured", async () => {+ const cfg = Object.freeze({+ ...BASE,+ enabled: true,+ baseUrl: "https://x/v1",+ model: "rerank-1",+ apiKey: "k",+ });+ const out = await applyCrossEncoderRerank([], "q", cfg, { provider: stubProvider(() => 1) });+ expect(out).toEqual([]);+ });+});diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.tsindex 2697f317..0f60908a 100644--- a/tests/core/search/store.test.ts+++ b/tests/core/search/store.test.ts@@ -61,6 +61,15 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCochainStopEnabled: false,chainStopScore: 0.8,}),+ rerank: Object.freeze({+ enabled: false,+ baseUrl: null,+ model: null,+ envKey: null,+ apiKey: null,+ topK: 20,+ minScore: 0,+ }),shutdownGraceMs: 5_000,resumeReindex: false,...overrides,diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.tsindex 489ff638..59788621 100644--- a/tests/core/search/store.vec.test.ts+++ b/tests/core/search/store.vec.test.ts@@ -64,6 +64,15 @@ function semanticConfig(chainStopEnabled: false,chainStopScore: 0.8,}),+ rerank: Object.freeze({+ enabled: false,+ baseUrl: null,+ model: null,+ envKey: null,+ apiKey: null,+ topK: 20,+ minScore: 0,+ }),shutdownGraceMs: 5_000,resumeReindex: false,...overrides,diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.tsindex e6691fc5..4ac03744 100644--- a/tests/helpers/search-fixtures.ts+++ b/tests/helpers/search-fixtures.ts@@ -11,7 +11,11 @@ import {DEFAULT_VAULT_IGNORE_PATHS,type VaultIgnoreRule,} from "../../src/core/vault-scope/defaults.ts";-import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "../../src/core/search/types.ts";+import type {+ ResolvedSearchConfig,+ ResolvedEmbeddingConfig,+ ResolvedRerankConfig,+} from "../../src/core/search/types.ts";export function createTempVault(prefix: string): {vault: string;@@ -75,6 +79,8 @@ export function makeConfig(opts: {selfTuningEnabled?: boolean;/** RRF damping constant; defaults to 60. */rrfK?: number;+ /** Optional cross-encoder rerank overrides; defaults to disabled. */+ rerank?: Partial<ResolvedRerankConfig>;}): ResolvedSearchConfig {const baseSemantic: ResolvedEmbeddingConfig = Object.freeze({enabled: false,@@ -130,6 +136,16 @@ export function makeConfig(opts: {chainStopEnabled: false,chainStopScore: 0.8,}),+ rerank: Object.freeze({+ enabled: false,+ baseUrl: null,+ model: null,+ envKey: null,+ apiKey: null,+ topK: 20,+ minScore: 0,+ ...opts.rerank,+ }),shutdownGraceMs: 5_000,resumeReindex: false,});