Local-first π§ memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly π΄ dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.
TypeScript
feat: Embedding Provider Suite - local embedder, provider registry, cost gate, RRF fusion (#67)
Sol Aitkeneda202df3561
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.jsonindex 1c89c52b..a3e2bcc9 100644--- a/.claude-plugin/plugin.json+++ b/.claude-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.35.0",+ "version": "0.36.0","description": "Plugin-first second brain package for AI agents and humans.","author": {"name": "Open Second Brain contributors"diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.jsonindex 4acf6587..8f155531 100644--- a/.codex-plugin/plugin.json+++ b/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.35.0",+ "version": "0.36.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","skills": "./skills","hooks": "./hooks/hooks.json",diff --git a/CHANGELOG.md b/CHANGELOG.mdindex eccf4dfe..c35382cb 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,55 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [0.36.0] - 2026-06-03++Embedding Provider Suite: four changes to the semantic layer that make+recall cheaper, more portable, and provider-flexible - an offline+embedder that needs no cloud, a runtime registry for embedding+endpoints, spend visibility with a cost gate, and an alternative+rank-fusion mode - all behind one shared signature kernel and all+off-by-default where they could change behaviour.++### Added++- **Offline local embedder.** A new `local` embedding provider produces+ deterministic vectors with no cloud call, no API key, and no model+ download: token unigrams and character trigrams are feature-hashed+ (FNV-1a, signed buckets) into a configurable fixed dimension (default+ 256) and unit-normalised, so the existing cosine ranker math is+ unchanged. Set `embedding_provider: local` (and+ `search_semantic_enabled: true`) for a privacy-first, no-cloud recall+ path; both indexing and query embedding run without a key. It is a+ lexical baseline - `openai-compat` remains the recommended provider for+ semantic depth.+- **CLI provider registry.** Register OpenAI-compatible embedding+ endpoints at runtime with `o2b search provider add | list | show |+ remove`, persisted to `Brain/search/embedding-providers.json` (base+ URL, default model, and the NAME of the environment variable holding+ the key - never the key itself, so the file is safe to sync). A+ registered name resolves to `openai-compat` config during config+ resolution, AFTER the built-ins, so it never shadows an explicitly+ configured key; explicit config or environment always wins over a+ profile's fields. The resolved provider union stays closed.+- **Embedding cost gate and signature reporting.** A best-effort+ per-model pricing table plus a chars/4 token estimate gate large+ embedding runs: when `embedding_cost_gate_usd` is positive (default 0 =+ disabled) and the estimated spend exceeds it, the run is refused with+ the estimate, and `o2b search index --force-cost` overrides. The local+ provider and any unlisted model price at 0 and never block.+ `o2b search status` now reports the active+ `<provider>:<model>:<dimension>` embedding signature and a best-effort+ refresh-cost estimate.+- **Reciprocal Rank Fusion mode.** `search_fusion_mode: rrf` (default+ `linear`) fuses the sparse (BM25) and dense (cosine) lanes by rank+ position - `1 / (search_rrf_k + rank)`, `search_rrf_k` default 60 -+ rather than by weighted score magnitude, rewarding cross-lane presence+ and staying robust to differing score scales. The fused relevance is+ min-max-normalised so the link, recency, entity, tier, and+ session-focus boosts compose unchanged, and results carry an `rrf:`+ recall reason. The default `linear` mode keeps ranking bit-identical;+ storage stays SQLite + sqlite-vec + FTS5 (no cloud vector database).+## [0.35.0] - 2026-06-02Memory Integrity Suite: four changes that make the memory itself@@ -4064,6 +4113,7 @@ plugin config (vault field)`, and exits with a clear- Sandbox vault and plugin manifest fixtures for tests.- GitHub release workflow for tag-based and manually dispatched releases.+[0.36.0]: https://github.com/itechmeat/open-second-brain/compare/v0.35.0...v0.36.0[0.35.0]: https://github.com/itechmeat/open-second-brain/compare/v0.34.0...v0.35.0[0.34.0]: https://github.com/itechmeat/open-second-brain/compare/v0.33.0...v0.34.0[0.33.0]: https://github.com/itechmeat/open-second-brain/compare/v0.32.1...v0.33.0diff --git a/README.md b/README.mdindex 006792b4..188091d9 100644--- a/README.md+++ b/README.md@@ -114,7 +114,7 @@ The capabilities you actually feel day to day:These are the headline capabilities. The full surface also includes: importing Claude Code memory directories, daily logging-discipline cron with a complexity-to-thinking warning, cross-project pointers for shared vaults, transient pinned session context through `brain_pinned_context` / `Brain/pinned.md`, source-agent query and comparison (`brain_agent_query`, `brain_agent_diff`, `o2b brain agent-query`, `o2b brain agent-diff`), the codegraph partner skill, vault hygiene lints, per-MOC coverage audit, concept synthesis, an operator dashboard, configurable Markdown-link presentation output, MCP output-contract checks for core agent-facing tools, private-region stripping with `<private>...</private>`, the v0.12.0 Brain Integrity Suite (content-hash drift detection, durable workrun checkpoints, destructive-from-confirmed retire gate, and the `brain_review_candidates` MCP tool that previews what the next dream pass would do without writing anything), the v0.13.0 Hybrid Search and Recall Quality suite (explainable recall, MMR diversity, link-graph traversal, entity-boosted retrieval, header-anchored chunking), the v0.14.0 Semantic Brain Health and Self-Maintenance suite (cross-preference contradiction detection, concept-gap and stale-claim surfacing, per-preference edit-history, a clean/watch/investigate reconciliation verdict via `brain_health`, and dependency-ordered `doctor --remediate`), the v0.17.0 Brain Lifecycle Review Suite (`intent-review`, `retention`, `monthly`, lifecycle schemas, and discipline complexity ratio), and the v0.18.0 MCP context economy (a per-tool preview budget that returns a bounded inline preview and parks the full payload in a vault-local artifact the agent fetches on demand with `brain_artifact_get`, plus a `recall_hint` one-liner on search results), and the v0.19.0 typed graph semantics (typed frontmatter relationships - `related` / `extends` / `contradicts` / `superseded_by` - that surface inline on search results, a `visibility:` scoping axis for `brain_search`, and a `brain_mcp_landscape` view of the Model Context Protocol servers configured across the vault), and the v0.20.0 Recall and Ranking Quality suite (a configurable Weibull recency-decay curve, structural query-intent classification that re-weights ranking by query shape, language-agnostic synonym expansion via local co-occurrence, per-memory and total recall character budgets for `brain_context_pack`, a persistent query cache that self-invalidates on a corpus-generation change, and a `brain_pre_compress_pack` tool that returns a budgeted top-preferences addendum for a host runtime to inject before context compression), and the v0.21.0 Brain Lifecycle Suite (a per-preference mutation audit trail keyed by preference id - create / promote / update / retire / merge with revision + content-hash before/after, via `o2b brain audit` and `brain_audit`; a multi-phase `dream` pipeline that names its close / reconcile / synthesize / heal / log seams and reports a structured per-phase summary without rewriting the proven internals; deterministic reconcile-phase domain classification that buckets contradictions into claims / entity / decisions / source-freshness, auto-resolves only the decisive source-freshness case, and surfaces the rest as open questions instead of forcing a merge; a budgeted `o2b brain morning-brief` / `brain_morning_brief` session-start summary of top preferences, recent open questions, and recent notes; language-agnostic ISO-8601 temporal extraction that fills preference validity windows from signal text; and opt-in heal-phase vault enrichment that links exact title mentions across user pages), and the v0.22.0 Vault portability + session economy suite (a deterministic lossless session codec - `expand(compress(x)) === x` for all input, opt-in on the signal store and exposed as `o2b brain codec`; an `o2b brain sources` / `brain_sources` read-only dashboard of signals by agent and source type; vault-map `{{role}}` tokens resolved via an optional `Brain/_vault-map.yaml` and wired into scan-inline read paths and graph import; named multi-vault profiles with `o2b vault profile` + a `brain_switch_vault` tool, activated by a pointer in config; and `o2b brain graph-export` / `graph-import` for moving a vault's knowledge graph between layouts with skip/overwrite/merge conflict modes). Browse [`docs/cli-reference.md`](docs/cli-reference.md) for every verb and [`docs/how-it-works.md`](docs/how-it-works.md) for the mental model.-The v0.26.0 surface adds audited schema mutation/admin tools, real-time lifecycle capture hooks for prompt markers and `brain_feedback` tool calls, CJK-aware FTS recall, and `brain_watchdog` / `o2b brain watchdog` recovery probes. The v0.27.0 surface adds recall-control and trust tools: structured search query documents, session focus, MCP `brain_recall_gate`, polarity-aware context lanes, and opt-in evidence packs for CLI/MCP search. The v0.28.0 surface adds a deterministic prompt-injection guard for automatically surfaced Brain context, agent-blind `$secret:NAME` references, and preview foundations for source-scoped forget, privacy-scanned knowledge packs, and oversized payload externalization. The v0.29.0 surface adds opt-in context receipts and recall telemetry, read-only context budget presets, pre-compaction extraction, and a continuity-backed session recall DAG. The v0.33.0 surface adds the Recall Trust Suite: relation-aware recall polarity for typed memory edges, an explicit retrieval feedback loop with bounded learned weights (`o2b search feedback` / `weights`, MCP `brain_recall_feedback`), time-aware recall (`--since` / `--until` with natural-language ranges), verified multi-record recall (per-token union records, IDF-weighted coverage, rare-term abstention), and a search-completeness verdict with a false-absence guard. The v0.34.0 surface adds the Token Diet: a budgeted SessionStart injection of `Brain/active.md` (default 8,000 chars, `active.inject_budget_chars`) with the post-compaction path moved to the supported SessionStart `compact` matcher, a once-per-session post-write reminder cadence, frontmatter escape-amplification fixed at the parser with a one-shot `brain upgrade --apply` repair for already-corrupted preference files, three consolidated read tools (`brain_brief`, `brain_analytics`, `schema_inspect`) with 18 hidden deprecated aliases, registry description caps with a contract guard, preview budgets by default, and a `scripts/measure-token-surface.ts` report. The v0.35.0 surface adds the Memory Integrity Suite: a canonical entity registry under `Brain/entities/` (one Markdown entity per `(category, name)` with alias resolution, archive/restore, typed relations, `o2b brain entity` CLI verbs, a read-only `brain_entity` MCP tool, doctor duplicate/relation lints, and alias-aware search boost with `entity_canonical` explanations), per-device Brain log shards (`Brain/log/<date>.<deviceId>.jsonl` + `.md` keyed by a device-local id, merged reads through `readLogDay`/`listLogDates`, no more Syncthing write conflicts, legacy days untouched), capture boundaries (a `sessions:` block with ignored/stateless session globs and message suppression regexes, applied before any extraction at both live-hook and import seams, counted but never stored raw; keep suppression regexes simple - they run on every message, and a catastrophic-backtracking pattern slows capture), and deterministic regex fact extraction (seven precision-first families over user turns, `source_type: extracted` signals with family-scoped dedup and canonical entity anchors).+The v0.26.0 surface adds audited schema mutation/admin tools, real-time lifecycle capture hooks for prompt markers and `brain_feedback` tool calls, CJK-aware FTS recall, and `brain_watchdog` / `o2b brain watchdog` recovery probes. The v0.27.0 surface adds recall-control and trust tools: structured search query documents, session focus, MCP `brain_recall_gate`, polarity-aware context lanes, and opt-in evidence packs for CLI/MCP search. The v0.28.0 surface adds a deterministic prompt-injection guard for automatically surfaced Brain context, agent-blind `$secret:NAME` references, and preview foundations for source-scoped forget, privacy-scanned knowledge packs, and oversized payload externalization. The v0.29.0 surface adds opt-in context receipts and recall telemetry, read-only context budget presets, pre-compaction extraction, and a continuity-backed session recall DAG. The v0.33.0 surface adds the Recall Trust Suite: relation-aware recall polarity for typed memory edges, an explicit retrieval feedback loop with bounded learned weights (`o2b search feedback` / `weights`, MCP `brain_recall_feedback`), time-aware recall (`--since` / `--until` with natural-language ranges), verified multi-record recall (per-token union records, IDF-weighted coverage, rare-term abstention), and a search-completeness verdict with a false-absence guard. The v0.34.0 surface adds the Token Diet: a budgeted SessionStart injection of `Brain/active.md` (default 8,000 chars, `active.inject_budget_chars`) with the post-compaction path moved to the supported SessionStart `compact` matcher, a once-per-session post-write reminder cadence, frontmatter escape-amplification fixed at the parser with a one-shot `brain upgrade --apply` repair for already-corrupted preference files, three consolidated read tools (`brain_brief`, `brain_analytics`, `schema_inspect`) with 18 hidden deprecated aliases, registry description caps with a contract guard, preview budgets by default, and a `scripts/measure-token-surface.ts` report. The v0.35.0 surface adds the Memory Integrity Suite: a canonical entity registry under `Brain/entities/` (one Markdown entity per `(category, name)` with alias resolution, archive/restore, typed relations, `o2b brain entity` CLI verbs, a read-only `brain_entity` MCP tool, doctor duplicate/relation lints, and alias-aware search boost with `entity_canonical` explanations), per-device Brain log shards (`Brain/log/<date>.<deviceId>.jsonl` + `.md` keyed by a device-local id, merged reads through `readLogDay`/`listLogDates`, no more Syncthing write conflicts, legacy days untouched), capture boundaries (a `sessions:` block with ignored/stateless session globs and message suppression regexes, applied before any extraction at both live-hook and import seams, counted but never stored raw; keep suppression regexes simple - they run on every message, and a catastrophic-backtracking pattern slows capture), and deterministic regex fact extraction (seven precision-first families over user turns, `source_type: extracted` signals with family-scoped dedup and canonical entity anchors). The v0.36.0 surface adds the Embedding Provider Suite: an offline `local` embedding provider (deterministic feature hashing, no cloud, no key, no model download - set `embedding_provider: local` for a privacy-first recall path), a CLI provider registry (`o2b search provider add | list | show | remove`, persisted to `Brain/search/embedding-providers.json` storing only environment-variable names, resolved after the built-ins so a configured key is never shadowed), an embedding cost gate (`embedding_cost_gate_usd`, default 0 = off; refuses an over-budget run unless `o2b search index --force-cost`) with the active `<provider>:<model>:<dimension>` signature and a refresh-cost estimate in `o2b search status`, and an opt-in Reciprocal Rank Fusion mode (`search_fusion_mode: rrf`, `search_rrf_k` default 60) that fuses the keyword and semantic lanes by rank position instead of weighted magnitude while the default `linear` mode stays bit-identical and storage remains SQLite + sqlite-vec + FTS5.## CLIdiff --git a/docs/brainstorm/embedding-provider-suite/cli-output/claude.md b/docs/brainstorm/embedding-provider-suite/cli-output/claude.mdnew file mode 100644index 00000000..0b4f3d34--- /dev/null+++ b/docs/brainstorm/embedding-provider-suite/cli-output/claude.md@@ -0,0 +1,35 @@+### Variant 1: Inline per-subsystem extension+- **Approach**: Each unit lands in the module that already owns its concern, following the existing closed-union/factory conventions verbatim. RRF becomes a `fusionMode` branch inside `ranker.ts`; the `local` provider extends the `makeProvider` switch + a new `local-provider.ts`; the registry is a read/merge helper in `config.ts` wired into `resolveSearchConfig` and new CLI verbs; cost/signature logic extends `indexer.ts` (gate, estimate) and `store.ts` (signature stamp). A single tiny `signature()` helper is the only shared new primitive.+- **Trade-offs**:+ - Smallest diff; each change reviewable against its sibling code and existing tests.+ - Lowest risk to the bit-identical-when-off and closed-union rules β they're untouched except where extended.+ - Fastest path to shipping the p4/p3/p2 units independently.+ - Cost/signature/pricing knowledge gets sprinkled across indexer + store + config rather than living in one place; mild DRY pressure as Unit 4's signature and Unit 3's registry both touch provider identity.+ - Registryβfactory coupling is implicit (two call sites must agree on precedence ordering).+- **Complexity**: small+- **Risk**: low++### Variant 2: Shared signature/provenance kernel + registry module+- **Approach**: Introduce two focused new domain modules β `embeddings/signature.ts` (canonicalize `{provider, model, dimension}` β stable signature, hold the per-model pricing table, compute stale-detection and spend estimates) and `embeddings/registry.ts` (load/persist provider definitions, merge after built-ins) β that Units 2/3/4 all depend on. RRF lands as a small `fusion` strategy split out of `ranker.ts`. The factory, CLI, indexer, and store all consume the kernel rather than re-deriving provider identity.+- **Trade-offs**:+ - Directly satisfies the stated "one shared canonicalization/signature kernel rather than per-call-site duplication" constraint.+ - Cost model, signature, and stale-detection become independently unit-testable in isolation from the store/indexer.+ - `corpusGeneration()` and per-chunk stamping converge on one signature definition, reducing drift risk in Unit 4.+ - More upfront structure than a pure inline approach; introduces module boundaries that must be designed before any unit ships, slightly coupling the four deliverables.+ - Medium blast radius: indexer/store/config all gain a new dependency edge, so the off-path fail-soft behaviour must be re-verified at each seam.+- **Complexity**: medium+- **Risk**: medium++### Variant 3: Descriptor/strategy plugin refactor+- **Approach**: Generalize the whole subsystem around uniform abstractions β providers become "descriptors" (built-in or registered) carrying capabilities (dimension, pricing, local/remote, signature), and fusion becomes a pluggable `FusionStrategy` interface (`linear` | `rrf`). The closed string union is replaced by a descriptor registry; all four units become instances of "register a descriptor / register a strategy."+- **Trade-offs**:+ - Most extensible: future providers and fusion modes drop in without touching core dispatch.+ - Cleanest conceptual model β registry, cost, signature, and fusion all hang off one extensibility spine.+ - Replaces the closed union that `resolveSearchConfig`/`parseProvider` depend on, risking the convention that provider selection is a validated closed set, and broadening the surface that must stay bit-identical-when-off.+ - Largest diff and the most existing tests (explainable `reasons`, parser validation, corpus generation) to re-baseline; over-built relative to four concrete units.+ - Highest chance of public-API churn on `EmbeddingProvider`/`ResolvedEmbeddingConfig` consumers, triggering the migration constraint.+- **Complexity**: large+- **Risk**: high++### Recommended: Variant 2+**Rationale**: The kernel approach is the only one that directly honours the explicit "one shared canonicalization/signature kernel" constraint while keeping the closed union, fail-soft, and bit-identical-when-off rules intact β Units 2, 3, and 4 genuinely share provider-identity and signature logic, so centralizing it prevents the drift that Variant 1 invites and avoids the over-reach of Variant 3's full plugin rewrite. It scales to four units with isolated, testable seams without forcing public-API breakage or re-baselining the ranker's explainability contract.diff --git a/docs/brainstorm/embedding-provider-suite/cli-output/prompt.md b/docs/brainstorm/embedding-provider-suite/cli-output/prompt.mdnew file mode 100644index 00000000..7c93c2a0--- /dev/null+++ b/docs/brainstorm/embedding-provider-suite/cli-output/prompt.md@@ -0,0 +1,73 @@+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.++# Task++This is an epic bundling four related kanban tasks that all extend the embedding/semantic-search subsystem of Open Second Brain (a local-first, Obsidian-Markdown agent memory store). The four atomic units, in priority order:++## Unit 1 (p4) β Hybrid dense+sparse RRF ranking+Add Reciprocal Rank Fusion (RRF) as an alternative to the current weighted-linear-sum fusion of BM25 (sparse keyword) and cosine (dense vector) scores. RRF combines the two rankings by rank position (score = sum over lanes of 1/(k + rank)) rather than by min-max-normalised magnitudes. VENDOR-AGNOSTIC ONLY: the upstream inspiration (Tencent TCVDB) is a cloud vector database; we are NOT adding a cloud vector DB. We keep SQLite + sqlite-vec + FTS5 as the storage layer and only add the rank-fusion mode.++## Unit 2 (p3) β Offline local embedder provider+The provider abstraction currently has only `openai-compat` (remote HTTP) and `disabled` (null). Add a `local` provider that produces embeddings with no cloud call, no API key, and no surprise bills β aligned with the product's local-first/privacy promise. The default install must stay lightweight; the local model/runtime can be a lazy/optional dependency. Needs predictable dimension, a model signature, and clear diagnostics when the runtime is unavailable.++## Unit 3 (p2) β CLI provider registry+Today an embedding provider is configured via env vars or by editing the YAML config. Add CLI verbs (`provider add/list/show/remove`) that persist provider definitions (base-url, default-model, env-key for the API key) to a registry file, so users add/remove providers at runtime without editing config. Registered providers are auto-detected AFTER built-ins so they never shadow an explicitly configured key.++## Unit 4 (p2) β Embedding cost model + stale-detection via signatures+Add model-aware embedding pricing (price per million tokens), an embedding "signature" (model + dimension, and possibly provider) stamped per chunk, stale-detection that flags chunks whose signature no longer matches the active config, and a cost gate that blocks a large (re-)embedding run when estimated spend exceeds a configurable threshold. The store already clears all embeddings on a model/dimension change and stamps a per-chunk `embedding_hash`; this unit adds cost visibility, a finer-grained signature, and the spend gate.++# Project context++Project: Open Second Brain. Language: TypeScript. Runtime: Bun. Tests: `bun test`. Lint: oxlint (baseline must not grow). Format: oxfmt. Typecheck: tsc.++Recent commits:+dee5ab7 feat: Memory Integrity Suite - canonical entities, conflict-free log shards, capture boundaries, fact extraction (#66)+5066e71 feat: Token Diet - budgeted injection, reminder cadence, consolidated MCP surface (#65)+c494fd8 feat(search): Recall Trust Suite - relation polarity, learned weights, time-aware and verified recall (#64)+0952dfc feat: become a native Hermes memory provider (#62)+bd49cd5 feat: recall and ranking quality - Weibull recency, query intent, synonym expansion, recall budgets, query cache, pre-compress pack (#47)++Related files (already studied):+- src/core/search/embeddings/provider.ts β `EmbeddingProvider` interface (name, model, dimension, embed(), ping(), consumeRetryCount?()); `makeProvider(config)` factory dispatching on `config.provider` with lazy require().+- src/core/search/embeddings/openai-compat.ts β remote provider: semaphore-bounded batches, retry/backoff, unit-normalised vectors, abort on first failure.+- src/core/search/embeddings/null-provider.ts β disabled provider.+- src/core/search/types.ts β `ResolvedEmbeddingConfig { enabled, provider: "openai-compat"|"disabled", baseUrl, model, apiKey, dimension, timeoutMs, concurrency, batchSize }`; `ResolvedSearchConfig` with keywordWeight/semanticWeight.+- src/core/search/index.ts β `resolveSearchConfig()`: parses embedding_* keys from YAML + env (`OPEN_SECOND_BRAIN_EMBEDDING_*`); `parseProvider()` only accepts openai-compat|disabled; DEFAULTS table (keywordWeight 0.6, semanticWeight 0.4).+- src/core/search/indexer.ts β `populateEmbeddings()`: drives provider.embed() in super-batches, stamps per-chunk sha256(embedding) as `embedding_hash`, calls `store.ensureEmbeddingModel(model, dim)`, `store.vecUpsert()`; `indexStatus()` reports embeddings/staleEmbeddings/embeddingModel/embeddingDimension; `indexCheck()` pings provider.+- src/core/search/store.ts β `vecUpsert(chunkId, vec, model, dim, embeddingHash)`; `ensureEmbeddingModel()` clears embeddings on model/dim change; `corpusGeneration()` = embeddingModel + dimension + schemaVersion + indexRevision (gates query cache); `findChunksWithoutEmbeddings()`; `staleEmbeddings()`; per-chunk `embeddings(chunk_id, model, dimension, embedding_hash, created_at, updated_at)` table.+- src/core/search/ranker.ts β `rankResults()`: current fusion is weighted linear sum `keywordWeight*kwMul*kwNorm + semanticWeight*semMul*semNorm` where kwNorm is min-max-normalised inverted BM25 and semNorm is cosine-from-L2-distance; explainable `reasons` array (`fts5_bm25`, `semantic_cos`, ...).+- src/cli/search.ts β search CLI verbs (reindex, check, focus, feedback, ...).+- src/core/config.ts β `setConfigValue(key, value, path)`, `discoverConfig()`, device-id pattern (env override -> read -> generate under directory lock); YAML-subset parser does NOT unescape backslashes in quoted scalars.++Conventions:+- Provider selection is a closed string union resolved in `resolveSearchConfig`; adding a provider means extending the union, the parser, and the `makeProvider` factory.+- The store is the single source of truth for the embedding model/dimension; `corpusGeneration()` already fingerprints them and self-invalidates the query cache.+- Every config knob has a YAML key + an `OPEN_SECOND_BRAIN_*` env override, with a default in the DEFAULTS table and range validation.+- Ranking layers are bounded, deterministic, and have an explicit off switch that keeps untouched vaults bit-identical; explainable `reasons` strings are part of the contract and are asserted in tests.+- Lazy `require()` keeps the module graph small for users who never enable semantic search.+- Fail-soft: a bad/absent registry or runtime degrades to current behaviour, never throws into the hot path.++Constraints:+- Do NOT add a cloud/hosted vector database. Storage stays SQLite + sqlite-vec + FTS5.+- Do NOT add a heavy mandatory dependency to the default install; any local-embedder model/runtime must be optional/lazy.+- Do NOT break the existing `openai-compat`/`disabled` providers or change their on-disk index format incompatibly.+- Keep ranking changes opt-in and bit-identical-when-off (an existing project rule).+- No new public-API breakage of `EmbeddingProvider` / `ResolvedEmbeddingConfig` consumers without a migration.+- SOLID / KISS / DRY. One shared canonicalization/signature kernel rather than per-call-site duplication.++# Required output format++Produce exactly 3 distinct architectural variants. For each variant:++### Variant N: <short name>+- **Approach**: 2-3 sentences describing the variant.+- **Trade-offs**: bullet list of pros and cons.+- **Complexity**: small | medium | large+- **Risk**: low | medium | high++After the three variants, add exactly one recommendation:++### Recommended: Variant N+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.++Output nothing outside of these sections.diff --git a/docs/brainstorm/embedding-provider-suite/design.md b/docs/brainstorm/embedding-provider-suite/design.mdnew file mode 100644index 00000000..bbe9c72b--- /dev/null+++ b/docs/brainstorm/embedding-provider-suite/design.md@@ -0,0 +1,73 @@+# Embedding Provider Suite - shared signature kernel, local embedder, provider registry, RRF fusion++**Status:** draft+**Author:** claude-dev-agent (via feature-release-playbook)+**Audience:** implementation++## Problem statement++Open Second Brain's semantic layer has one real embedding provider (`openai-compat`, remote HTTP) and one null provider. There is no offline/local-first embedder, no runtime way to register providers without editing config, no cost visibility or spend gate before a large embedding run, and exactly one rank-fusion mode (weighted linear sum of min-max-normalised BM25 and cosine). Four triage tasks (t_9d24c9fd p4, t_d1fbfeee p3, t_f42f1b69 p2, t_4796dff3 p2) all extend this one subsystem and share provider-identity logic.++## Scope++- **Unit 1 - Hybrid RRF fusion (t_9d24c9fd):** add Reciprocal Rank Fusion as an alternative to linear fusion of the sparse (BM25) and dense (cosine) lanes. Opt-in via `search_fusion_mode`; default `linear` keeps ranking bit-identical. No cloud vector DB - storage stays SQLite + sqlite-vec + FTS5.+- **Unit 2 - Local embedder (t_d1fbfeee):** add a `local` provider that produces deterministic embeddings with no cloud call, no API key, and no model download - a character n-gram feature-hashing embedder. Predictable dimension, stable model signature, clear diagnostics.+- **Unit 3 - Provider registry (t_f42f1b69):** CLI verbs (`o2b search provider add/list/show/remove`) that persist named provider profiles (base-url, default-model, env-key) to a vault-local registry file. A registered name resolves to an `openai-compat` config at config-resolution time; built-ins are never shadowed.+- **Unit 4 - Cost model + signatures (t_4796dff3):** one signature kernel that canonicalises `{provider, model, dimension}`, a model-aware price-per-million-tokens table, a token/cost estimator, signature-based stale reporting in status/check, and a cost gate that blocks an embedding run whose estimated spend exceeds a configurable threshold.++## Out of scope++- Cloud/hosted vector databases (Qdrant/Milvus/Weaviate/TCVDB) - explicitly deferred.+- A heavy bundled transformer model. The local embedder is dependency-free; opting into a larger local model is future work.+- Per-token exact billing reconciliation against a provider invoice - we estimate, we do not reconcile.+- Replacing the closed provider union with a descriptor/plugin spine (consultant Variant 3) - rejected as over-built.++## Chosen approach++Consultant **Variant 2 - shared signature/provenance kernel + registry module**. Two focused new domain modules carry the logic Units 2/3/4 share, and a small fusion strategy module carries Unit 1:++- `embeddings/signature.ts` - canonicalise `{provider, model, dimension}` to a stable signature string; hold the per-model pricing table; estimate tokens and cost; decide staleness by comparing an active signature with a stored one. The single source of provider-identity truth, consumed by the factory, indexer, store, and CLI rather than re-derived per call site.+- `embeddings/registry.ts` - load/persist provider profiles from `Brain/search/embedding-providers.json`; expand a registered name into `openai-compat` config fields during `resolveSearchConfig`, after built-ins so a configured built-in key is never shadowed. Fail-soft: an absent or malformed registry resolves to no extra providers.+- `embeddings/local-provider.ts` - `LocalProvider implements EmbeddingProvider`: deterministic character n-gram feature hashing into a fixed-dimension unit-normalised vector. No network, no key, no download.+- `search/fusion.ts` - pure `linearFusion` and `rrfFusion(lanes, k)` helpers split out of `ranker.ts`; the ranker calls the selected strategy and keeps every existing boost (link/tag/recency/entity/tier/session-focus) unchanged.++## Design decisions++- **Closed union stays closed.** The only new resolved provider member is `local`. Registry entries are config-time indirection that resolve to `provider: "openai-compat"`, so `ResolvedEmbeddingConfig.provider` remains `"openai-compat" | "disabled" | "local"` and `parseProvider` keeps validating a closed set. Registry expansion happens before `parseProvider` so a registered name never reaches the validator.+- **Local embedder = feature hashing, not a model.** A character n-gram (n=3, plus token unigrams) hashing trick gives an offline, deterministic, zero-dependency baseline that works the instant the package installs - matching the upstream "tiny default embedder, no model download" intent. Vectors are unit-normalised so the existing cosine-from-L2 ranker math is unchanged. Quality limits are documented; the design leaves room for a future opt-in transformer.+- **Signature is the one fingerprint.** `embeddingSignature(provider, model, dimension)` is reused by `corpusGeneration()` (cache invalidation), per-run stale reporting, and the cost estimate, so the four units cannot drift on what "the same embedding configuration" means. The store keeps its existing clear-on-model/dimension-change safety; the signature adds finer-grained reporting and gating on top, it does not replace the clear.+- **Cost gate is off by default.** `embedding_cost_gate_usd` defaults to `0` (disabled). When positive and a run's estimated spend exceeds it, the indexer throws `EMBEDDING_COST_GATE` carrying the estimate; an explicit override (`--force-cost` / config) proceeds. The local provider prices at `0`, so it is never gated.+- **RRF is off by default.** `search_fusion_mode` defaults to `linear`; with it the ranker output is byte-for-byte the current behaviour. `rrf` mode fuses lane ranks with `1 / (rrf_k + rank)` (`search_rrf_k` default 60, the canonical RRF constant) and adds an `rrf:` reason; boosts still compose on top.+- **Every knob follows the house pattern:** a YAML key, an `OPEN_SECOND_BRAIN_*` env override, a `DEFAULTS` entry, and range validation.+- **Fail-soft everywhere:** a bad registry, an unavailable local runtime, or an unknown pricing entry degrades to current behaviour or a clear diagnostic, never an exception in the hot path.++## File changes++New source:+- `src/core/search/embeddings/signature.ts`+- `src/core/search/embeddings/registry.ts`+- `src/core/search/embeddings/local-provider.ts`+- `src/core/search/fusion.ts`++Modified source:+- `src/core/search/embeddings/provider.ts` - `makeProvider` `local` branch.+- `src/core/search/types.ts` - provider union `+ "local"`; `fusionMode`, `rrfK`, `costGateUsd` on resolved config; signature + estimated-refresh-cost fields on the status snapshot.+- `src/core/search/index.ts` - `parseProvider` local; registry expansion; fusion/rrf/cost-gate config parsing; `DEFAULTS`.+- `src/core/search/indexer.ts` - cost gate + estimate in `populateEmbeddings`; signature + refresh-cost in `indexStatus`.+- `src/core/search/store.ts` - signature helpers / signature-based stale count; `corpusGeneration` reuses the kernel.+- `src/core/search/ranker.ts` - delegate relevance fusion to `fusion.ts`; rrf path; `rrf` reason.+- `src/cli/search.ts` - `provider` subverbs; cost-gate flag on reindex.+- `src/cli/command-manifest.ts`, `src/cli/help-text.ts`, `src/cli/completions.ts` - register and document the new verbs.++New tests (one focused suite per unit + integration):+- `tests/embeddings.local-provider.test.ts`, `tests/embeddings.signature.test.ts`, `tests/embeddings.registry.test.ts`, `tests/search.fusion-rrf.test.ts`, `tests/indexer.cost-gate.test.ts`, `tests/search-provider.cli.test.ts`, `tests/search-config.embedding-suite.test.ts`, plus an e2e search slice exercising rrf + local provider.++Docs:+- `README.md`, `CHANGELOG.md`, `docs/cli-reference.md`, `docs/how-it-works.md`.++## Risks and open questions++- **Local embedder quality.** Feature hashing is lexical, not semantic; near-synonyms it cannot match. Mitigation: document it as a no-cloud baseline, keep `openai-compat` the recommended path for semantic depth, and leave the dimension configurable.+- **RRF re-baselining.** RRF changes scores when enabled, so e2e ranking fixtures must assert `linear` (default) stays bit-identical and add separate `rrf` fixtures. Mitigation: gate behind the off-by-default switch and test both modes.+- **Pricing table staleness.** Hardcoded per-model prices drift as providers change rates. Mitigation: treat unknown models as price `0` (never falsely gate), document the table as best-effort, and allow a per-model override via config.+- **Registry file and Syncthing.** The registry stores env-key *names*, never secret values, so it is safe to sync across devices. Confirmed no secret material lands on disk.diff --git a/docs/brainstorm/embedding-provider-suite/plan.md b/docs/brainstorm/embedding-provider-suite/plan.mdnew file mode 100644index 00000000..9b7eaee5--- /dev/null+++ b/docs/brainstorm/embedding-provider-suite/plan.md@@ -0,0 +1,40 @@+# Embedding Provider Suite - implementation plan++Implementation order follows the dependency graph: the shared kernel first (Unit 4's signature/pricing is depended on by Units 2 and 3), then the local provider, then the registry, then RRF (independent). Each task is one atomic conventional commit on `feat/embedding-provider-suite`, fmt+lint green before commit.++## Tasks++### Task 1: Signature + cost kernel (Unit 4 foundation)+- **Files**: `src/core/search/embeddings/signature.ts` (new); `tests/embeddings.signature.test.ts` (new).+- **What**: `embeddingSignature(provider, model, dimension)` -> canonical `provider:model:dim` string; `EMBEDDING_PRICING` table (per-model USD/Mtok, unknown -> 0); `estimateTokens(texts)` (chars/4 heuristic); `estimateCostUsd(tokens, model)`; `isStaleSignature(active, stored)`.+- **Acceptance**: unit tests cover canonicalisation (case/whitespace/null model), known + unknown pricing, token + cost estimate, stale comparison. No I/O.+- **Depends on**: none.++### Task 2: Local embedder provider (Unit 2)+- **Files**: `src/core/search/embeddings/local-provider.ts` (new); `src/core/search/embeddings/provider.ts` (makeProvider `local` branch); `src/core/search/types.ts` (provider union `+ "local"`); `src/core/search/index.ts` (`parseProvider` accepts `local`, default model/dimension for local); `tests/embeddings.local-provider.test.ts` (new).+- **What**: `LocalProvider` - deterministic character-trigram + token-unigram feature hashing into a configurable fixed dimension (default 256), unit-normalised; `name="local"`, `model="hashing-ngram-v1"`, `ping()` always ok with the configured dimension; no network, no key.+- **Acceptance**: identical text -> identical vector; different text -> different vector; vectors unit-normalised; dimension honoured; `makeProvider({provider:"local"})` returns a `LocalProvider`; `parseProvider("local")` ok.+- **Depends on**: Task 1 (pricing entry for the local model = 0).++### Task 3: Provider registry + CLI (Unit 3)+- **Files**: `src/core/search/embeddings/registry.ts` (new); `src/core/search/index.ts` (registry expansion before `parseProvider`); `src/cli/search.ts` (`provider add/list/show/remove`); `src/cli/command-manifest.ts`, `src/cli/help-text.ts`, `src/cli/completions.ts`; `tests/embeddings.registry.test.ts` (new); `tests/search-provider.cli.test.ts` (new).+- **What**: load/persist `Brain/search/embedding-providers.json` (`{name, baseUrl, defaultModel, envKey}[]`); `expandRegisteredProvider(name, registry, env)` -> openai-compat fields; CLI verbs to add/list/show/remove with validation (reserved names `openai-compat`/`disabled`/`local` refused). Fail-soft on absent/malformed file.+- **Acceptance**: round-trip add->list->show->remove; registered name resolves to openai-compat config with apiKey from `env[envKey]`; built-in names cannot be registered; malformed JSON -> empty registry + warning, never throw.+- **Depends on**: Task 1 (signature reused in show output).++### Task 4: Cost gate + signature reporting in indexer/store (Unit 4)+- **Files**: `src/core/search/indexer.ts` (estimate + gate in `populateEmbeddings`; signature + estimatedRefreshCostUsd in `indexStatus`); `src/core/search/store.ts` (signature-based stale helper; `corpusGeneration` reuses kernel); `src/core/search/types.ts` (`costGateUsd` on resolved config; signature + cost fields on status snapshot); `src/core/search/index.ts` (`embedding_cost_gate_usd` parse + default 0); `src/cli/search.ts` (`--force-cost` flag on reindex); `tests/indexer.cost-gate.test.ts` (new).+- **What**: before embedding a batch set, estimate tokens+cost; if `costGateUsd > 0` and estimate exceeds it and not overridden, throw `EMBEDDING_COST_GATE` with the estimate; surface active signature + refresh cost in status. Local provider (price 0) never gated.+- **Acceptance**: gate blocks an over-threshold run and the message carries the estimate; `--force-cost` proceeds; gate `0` (default) never blocks; status reports the signature; `corpusGeneration` unchanged output for unchanged config.+- **Depends on**: Tasks 1, 2.++### Task 5: RRF fusion mode (Unit 1)+- **Files**: `src/core/search/fusion.ts` (new); `src/core/search/ranker.ts` (delegate relevance fusion; rrf path; `rrf` reason); `src/core/search/types.ts` (`fusionMode`, `rrfK` on resolved config); `src/core/search/index.ts` (`search_fusion_mode` + `search_rrf_k` parse, defaults `linear`/60); `tests/search.fusion-rrf.test.ts` (new).+- **What**: `linearFusion(kw, sem, kwWeight, semWeight)` (current math) and `rrfFusion(laneRanks, k)`; ranker selects by `fusionMode`; `linear` byte-identical to today; `rrf` fuses lane ranks, adds `rrf:` reason, boosts still compose.+- **Acceptance**: `linear` mode reproduces current ranking fixtures exactly; `rrf` mode orders by reciprocal-rank sum on a constructed two-lane fixture; invalid mode rejected by config parse.+- **Depends on**: none (independent; sequenced last to isolate ranking re-baseline).++### Task 6: Integration + docs+- **Files**: e2e search slice (rrf + local provider) in an existing or new e2e test; `README.md`; `CHANGELOG.md` (v0.36.0); `docs/cli-reference.md`; `docs/how-it-works.md`.+- **Acceptance**: full `bun test` green; new capabilities documented capability-first; CHANGELOG single v0.36.0 entry bundling all four units.+- **Depends on**: Tasks 1-5.diff --git a/docs/brainstorm/embedding-provider-suite/variants.md b/docs/brainstorm/embedding-provider-suite/variants.mdnew file mode 100644index 00000000..baa360f7--- /dev/null+++ b/docs/brainstorm/embedding-provider-suite/variants.md@@ -0,0 +1,32 @@+# Embedding Provider Suite - brainstorm variants (audit trail)++Primary consultant: `claude -p` (Claude Code). Exit 0, three parseable variants, no fallback needed. Full transcript: `cli-output/claude.md`. Prompt: `cli-output/prompt.md`.++## Variant 1: Inline per-subsystem extension++- **Approach**: Each unit lands in the module that already owns its concern, following existing closed-union/factory conventions verbatim. RRF becomes a `fusionMode` branch inside `ranker.ts`; the `local` provider extends the `makeProvider` switch + a new `local-provider.ts`; the registry is a read/merge helper in `config.ts` wired into `resolveSearchConfig` and new CLI verbs; cost/signature logic extends `indexer.ts` and `store.ts`. A single tiny `signature()` helper is the only shared new primitive.+- **Trade-offs**: Smallest diff, lowest risk to the bit-identical/closed-union rules, fastest independent shipping; BUT cost/signature/pricing knowledge sprinkled across indexer + store + config (DRY pressure), and registry-to-factory precedence coupling is implicit across two call sites.+- **Complexity**: small+- **Risk**: low++## Variant 2: Shared signature/provenance kernel + registry module++- **Approach**: Two focused new domain modules - `embeddings/signature.ts` (canonicalise `{provider, model, dimension}` -> signature, hold pricing table, compute staleness + spend estimates) and `embeddings/registry.ts` (load/persist provider definitions, merge after built-ins) - that Units 2/3/4 all depend on. RRF lands as a small `fusion` strategy split out of `ranker.ts`. Factory, CLI, indexer, store all consume the kernel rather than re-deriving provider identity.+- **Trade-offs**: Directly satisfies the "one shared canonicalization/signature kernel" constraint; cost/signature/stale independently unit-testable; `corpusGeneration()` and per-chunk stamping converge on one signature, reducing drift. BUT more upfront structure, and a medium blast radius (indexer/store/config gain a dependency edge whose fail-soft behaviour must be re-verified at each seam).+- **Complexity**: medium+- **Risk**: medium++## Variant 3: Descriptor/strategy plugin refactor++- **Approach**: Generalise the subsystem around uniform abstractions - providers become capability-carrying "descriptors" (built-in or registered), fusion becomes a pluggable `FusionStrategy` interface. The closed string union is replaced by a descriptor registry; all four units become "register a descriptor / register a strategy."+- **Trade-offs**: Most extensible and cleanest conceptual model. BUT replaces the closed union that `resolveSearchConfig`/`parseProvider` depend on, broadens the bit-identical-when-off surface, has the largest diff and most tests to re-baseline, and the highest chance of public-API churn triggering the migration constraint - over-built relative to four concrete units.+- **Complexity**: large+- **Risk**: high++## Consultant recommendation: Variant 2++> The kernel approach is the only one that directly honours the explicit "one shared canonicalization/signature kernel" constraint while keeping the closed union, fail-soft, and bit-identical-when-off rules intact - Units 2, 3, and 4 genuinely share provider-identity and signature logic, so centralizing it prevents the drift that Variant 1 invites and avoids the over-reach of Variant 3's full plugin rewrite. It scales to four units with isolated, testable seams without forcing public-API breakage or re-baselining the ranker's explainability contract.++## Orchestrator decision: accept Variant 2++Accepted without override. The decisive factor is the project's own DRY constraint: Unit 4's per-chunk signature and the existing `corpusGeneration()` fingerprint must agree on what "the same embedding configuration" means, and Unit 3's registry resolves provider identity that Unit 4 then signs - centralising this in `signature.ts` is the design that prevents three call sites from drifting. Variant 1 would ship slightly faster but re-introduces exactly that drift risk across indexer/store/config; Variant 3's descriptor rewrite would break the closed-union convention and the explainable-`reasons` test contract for no benefit at four-unit scale. Variant 2 keeps the closed union closed (registry entries resolve to `openai-compat`; only `local` is a new member), keeps RRF and the cost gate off by default for bit-identical behaviour, and isolates each unit behind a testable seam.diff --git a/docs/cli-reference.md b/docs/cli-reference.mdindex 4c426e72..cb191dd5 100644--- a/docs/cli-reference.md+++ b/docs/cli-reference.md@@ -181,8 +181,29 @@ o2b search focus status Show the active focus; --json emits { active, focuo2b search focus clear Clear the persisted focus file next to the search indexo2b search reindex Rebuild the SQLite + FTS5 index from scratch(required after upgrading to v0.13.0 recall schema or v0.26.0 CJK FTS content)+ --force-cost bypasses the embedding cost gate for this run (since v0.36.0)+o2b search index Incrementally update the index; --embeddings computes vectors+ --force-cost bypasses the embedding cost gate (since v0.36.0)+o2b search status Index status; since v0.36.0 also reports the active embedding+ signature (<provider>:<model>:<dimension>) and a refresh-cost estimate+o2b search provider add NAME Register an OpenAI-compatible embedding endpoint (since v0.36.0)+ --base-url U --model M --env-key K (K is the env var NAME holding the key);+ persisted to Brain/search/embedding-providers.json, resolved after built-ins+o2b search provider list List registered provider profiles (--json for the array)+o2b search provider show NAME Show one registered profile (--json)+o2b search provider remove Remove a registered profile by NAME```+Embedding providers (since v0.36.0): `embedding_provider` accepts the+built-in `openai-compat`, the offline `local` feature-hashing embedder+(no cloud, no key, no model download; `embedding_dimension` default 256),+`disabled`, or any name registered via `o2b search provider add`.+`embedding_cost_gate_usd` (default 0 = off) refuses an embedding run whose+estimated spend exceeds it unless `--force-cost`. `search_fusion_mode`+(default `linear`) may be set to `rrf` to fuse the keyword and semantic+lanes by reciprocal rank (`search_rrf_k`, default 60); `linear` keeps+ranking bit-identical.+Typed relations participate in ranking (relation polarity): a page whosefrontmatter declares `superseded_by:` is demoted when it matches and itssuccessor is boosted or pulled in, `contradicts:` surfaces warning-stylediff --git a/docs/how-it-works.md b/docs/how-it-works.mdindex d3d247fb..169f1e84 100644--- a/docs/how-it-works.md+++ b/docs/how-it-works.md@@ -728,8 +728,25 @@ Key behaviours, all driven from `Brain/_brain.yaml`-free `search_*` /mode for concurrent reads, `proper-lockfile` on the index path forwriter exclusivity (three attempts, 1 s backoff, then`INDEX_LOCKED`).+- **Embedding providers (v0.36.0).** `embedding_provider` selects+ `openai-compat` (remote `/v1/embeddings`), `local` (an offline+ feature-hashing embedder: token unigrams + character trigrams hashed+ into a fixed unit-normalised vector, no cloud / key / model download),+ `disabled`, or a name registered through `o2b search provider add`+ (which resolves to `openai-compat` config after the built-ins). One+ `signature.ts` kernel canonicalises `<provider>:<model>:<dimension>`+ and prices it; `embedding_cost_gate_usd` refuses an over-budget run+ unless `--force-cost`, and the local / unlisted-model price is 0.+- **Fusion modes (v0.36.0).** `search_fusion_mode` is `linear` (default,+ the weighted sum below) or `rrf` - Reciprocal Rank Fusion, which scores+ each candidate `Ξ£ 1/(search_rrf_k + rank_in_lane)` across the keyword+ and semantic lanes, min-max-normalised to `[0,1]`. RRF is weightless+ and replaces only the relevance term; every boost below still applies,+ and `linear` is bit-identical to pre-v0.36.0 ranking.- **Ranking.** `final_score = clamp01(keyword_weightΒ·norm_BM25 +-semantic_weightΒ·cosine + link_boost + recency_boost + entity_boost)`.+semantic_weightΒ·cosine + link_boost + recency_boost + entity_boost)`+ (linear mode; in `rrf` mode the first two terms are replaced by the+ fused reciprocal-rank relevance).BM25 is min-max-normalised within the candidate set; cosine is`1 - L2Β² / 2` on unit-normalised vectors; link boost rewardscandidates that other candidates reference via `[[wikilink]]` /diff --git a/openclaw.plugin.json b/openclaw.plugin.jsonindex 195cee2c..1dca35c4 100644--- a/openclaw.plugin.json+++ b/openclaw.plugin.json@@ -2,7 +2,7 @@"id": "open-second-brain","name": "Open Second Brain","description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",- "version": "0.35.0",+ "version": "0.36.0","activation": { "onStartup": true },"skills": ["./skills"],"contracts": {diff --git a/package.json b/package.jsonindex 41d5461d..a1f967ce 100644--- a/package.json+++ b/package.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.35.0",+ "version": "0.36.0","private": false,"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.","keywords": [diff --git a/plugin.yaml b/plugin.yamlindex c1fdcd75..963c149c 100644--- a/plugin.yaml+++ b/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "0.35.0"+version: "0.36.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.jsonindex a510a7cd..c17f4a6f 100644--- a/plugins/codex/.codex-plugin/plugin.json+++ b/plugins/codex/.codex-plugin/plugin.json@@ -1,6 +1,6 @@{"name": "open-second-brain",- "version": "0.35.0",+ "version": "0.36.0","description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.","author": {"name": "Open Second Brain contributors",diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yamlindex c1fdcd75..963c149c 100644--- a/plugins/hermes/plugin.yaml+++ b/plugins/hermes/plugin.yaml@@ -1,5 +1,5 @@name: open-second-brain-version: "0.35.0"+version: "0.36.0"description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."author: "Open Second Brain contributors"memory_provider: truediff --git a/pyproject.toml b/pyproject.tomlindex d101a99e..a45e5e4a 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"# CLI entry points (those moved to `package.json` `bin`).[project]name = "open-second-brain"-version = "0.35.0"+version = "0.36.0"description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."readme = "README.md"requires-python = ">=3.11"diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.tsindex 952d26df..71bb1104 100644--- a/src/cli/command-manifest.ts+++ b/src/cli/command-manifest.ts@@ -164,6 +164,7 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({command("reindex", "Rebuild the search index"),command("status", "Print search index status"),command("check", "Run search pre-flight diagnostics"),+ command("provider", "Manage embedding provider profiles"),],),command(diff --git a/src/cli/search.ts b/src/cli/search.tsindex 98fe4f65..22de5e7a 100644--- a/src/cli/search.ts+++ b/src/cli/search.ts@@ -35,6 +35,10 @@ import {LEARNED_WEIGHT_MIN,LEARNED_WEIGHT_MAX,type LearnedWeights,+ loadProviderRegistry,+ addProviderProfile,+ removeProviderProfile,+ getProviderProfile,} from "../core/search/index.ts";import type {IndexCheckReport,@@ -57,6 +61,7 @@ const KNOWN_VERBS = new Set(["focus","feedback","weights",+ "provider",]);export async function handleSearchSubcommand(argv: ReadonlyArray<string>): Promise<number> {@@ -87,6 +92,8 @@ export async function handleSearchSubcommand(argv: ReadonlyArray<string>): Promireturn await cmdSearchFeedback(rest);case "weights":return await cmdSearchWeights(rest);+ case "provider":+ return await cmdSearchProvider(rest);default:process.stderr.write(`error: unknown search verb: ${verb}\n`);return 2;@@ -293,6 +300,95 @@ function formatLearnedWeights(w: LearnedWeights): string {);}+// βββ provider registry (Embedding Provider Suite) ββββββββββββββββββββββββββββ++async function cmdSearchProvider(argv: ReadonlyArray<string>): Promise<number> {+ const action = argv[0];+ if (!action || !["add", "list", "show", "remove"].includes(action)) {+ throw new CliError(+ "usage: o2b search provider <add NAME --base-url U --model M --env-key K | list | show NAME | remove NAME> [--json]",+ );+ }+ const { flags, positional } = parseFlags(argv.slice(1), {+ vault: { type: "string" },+ config: { type: "string" },+ db: { type: "string" },+ "base-url": { type: "string" },+ model: { type: "string" },+ "env-key": { type: "string" },+ json: { type: "boolean" },+ });+ const cfg = resolveConfig(flags);+ const json = flags["json"] === true;+ const name = typeof positional[0] === "string" ? positional[0] : undefined;++ if (action === "list") {+ const registry = loadProviderRegistry(cfg.vault);+ if (json) {+ process.stdout.write(JSON.stringify(registry) + "\n");+ return 0;+ }+ if (registry.length === 0) {+ process.stdout.write("no registered providers\n");+ return 0;+ }+ for (const p of registry) {+ process.stdout.write(`${p.name} ${p.baseUrl} model=${p.defaultModel} env=${p.envKey}\n`);+ }+ return 0;+ }++ if (action === "show") {+ if (!name) throw new CliError("usage: o2b search provider show NAME");+ const profile = getProviderProfile(cfg.vault, name);+ if (!profile) {+ process.stderr.write(`error: no registered provider named '${name}'\n`);+ return 1;+ }+ process.stdout.write(+ json+ ? JSON.stringify(profile) + "\n"+ : `${profile.name}\n base-url: ${profile.baseUrl}\n model: ${profile.defaultModel}\n env-key: ${profile.envKey}\n`,+ );+ return 0;+ }++ if (action === "remove") {+ if (!name) throw new CliError("usage: o2b search provider remove NAME");+ const { removed } = removeProviderProfile(cfg.vault, name);+ if (json) {+ process.stdout.write(JSON.stringify({ removed, name }) + "\n");+ } else {+ process.stdout.write(+ removed ? `removed provider '${name}'\n` : `no such provider '${name}'\n`,+ );+ }+ return removed ? 0 : 1;+ }++ // add+ if (!name)+ throw new CliError("usage: o2b search provider add NAME --base-url U --model M --env-key K");+ const baseUrl = typeof flags["base-url"] === "string" ? (flags["base-url"] as string) : undefined;+ const model = typeof flags["model"] === "string" ? (flags["model"] as string) : undefined;+ const envKey = typeof flags["env-key"] === "string" ? (flags["env-key"] as string) : undefined;+ if (!baseUrl || !model || !envKey) {+ throw new CliError(+ "provider add requires --base-url, --model, and --env-key (the env var NAME holding the API key)",+ );+ }+ const registry = addProviderProfile(cfg.vault, { name, baseUrl, defaultModel: model, envKey });+ const added = registry.find((p) => p.name === name)!;+ if (json) {+ process.stdout.write(JSON.stringify(added) + "\n");+ } else {+ process.stdout.write(+ `added provider '${name}' (set ${envKey} in the environment to supply its key)\n`,+ );+ }+ return 0;+}+// βββ query ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββasync function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<number> {@@ -528,6 +624,7 @@ async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<number> {db: { type: "string" },embeddings: { type: "boolean" },force: { type: "boolean" },+ "force-cost": { type: "boolean" },concurrency: { type: "string" },verbose: { type: "boolean" },json: { type: "boolean" },@@ -538,6 +635,7 @@ async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<number> {const stats = await indexVault(cfg, {embeddings: flags["embeddings"] === true,force: flags["force"] === true,+ forceCost: flags["force-cost"] === true,onFile: (e) => {events.push(e);if (flags["verbose"]) {@@ -601,6 +699,7 @@ async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<number> {config: { type: "string" },db: { type: "string" },embeddings: { type: "boolean" },+ "force-cost": { type: "boolean" },concurrency: { type: "string" },json: { type: "boolean" },verbose: { type: "boolean" },@@ -624,6 +723,7 @@ async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<number> {const cfg = resolveConfig(flags);const stats = await reindexVault(cfg, {embeddings: flags["embeddings"] === true,+ forceCost: flags["force-cost"] === true,onFile: flags["verbose"] ? (e) => process.stderr.write(`${e.kind}\t${e.path}\n`) : undefined,});if (flags["json"]) {@@ -664,6 +764,8 @@ function jsonForStatus(s: IndexStatusSnapshot): unknown {stale_embeddings: s.staleEmbeddings,embedding_model: s.embeddingModel,embedding_dimension: s.embeddingDimension,+ embedding_signature: s.embeddingSignature,+ estimated_refresh_cost_usd: s.estimatedRefreshCostUsd,vec_extension: s.vecExtension,semantic_enabled: s.semanticEnabled,embedding_key_present: s.embeddingKeyPresent,@@ -685,6 +787,10 @@ function renderStatusHuman(s: IndexStatusSnapshot): string {lines.push(`embeddings: ${s.embeddings} (stale: ${s.staleEmbeddings})`);lines.push(`embedding_model: ${s.embeddingModel ?? "(none)"}`);lines.push(`embedding_dimension: ${s.embeddingDimension ?? "(none)"}`);+ lines.push(`embedding_signature: ${s.embeddingSignature ?? "(disabled)"}`);+ if (s.estimatedRefreshCostUsd > 0) {+ lines.push(`refresh_cost_est: $${s.estimatedRefreshCostUsd.toFixed(4)}`);+ }lines.push(`vec_extension: ${s.vecExtension}`);lines.push(`semantic_enabled: ${s.semanticEnabled}`);lines.push(`embedding_key: ${s.embeddingKeyPresent ? "present" : "missing"}`);diff --git a/src/core/search/embeddings/local-provider.ts b/src/core/search/embeddings/local-provider.tsnew file mode 100644index 00000000..b12ad4b3--- /dev/null+++ b/src/core/search/embeddings/local-provider.ts@@ -0,0 +1,83 @@+/**+ * Offline local embedder (Embedding Provider Suite).+ *+ * A dependency-free, deterministic embedder that works the instant the+ * package installs - no cloud call, no API key, no model download. It+ * uses the feature-hashing ("hashing trick") technique: token unigrams+ * and character trigrams are hashed into a fixed-dimension vector with a+ * signed bucket, then unit-normalised so the existing cosine-from-L2+ * ranker math is unchanged.+ *+ * This is a lexical baseline, not a transformer: it matches shared+ * tokens and subword shapes, not paraphrase or semantics. It exists so a+ * privacy-first install has a no-cloud recall path; `openai-compat`+ * remains the recommended provider for semantic depth. A future opt-in+ * local transformer can register as a distinct model without touching+ * this one.+ */++import type { EmbeddingProvider } from "./provider.ts";+import { LOCAL_EMBEDDING_MODEL } from "./signature.ts";++/** Default vector width when `embedding_dimension` is not configured. */+export const LOCAL_DEFAULT_DIMENSION = 256;++/** FNV-1a 32-bit hash, seeded so the two derived hashes diverge. */+function fnv1a(text: string, seed: number): number {+ let h = (0x811c9dc5 ^ seed) >>> 0;+ for (let i = 0; i < text.length; i++) {+ h ^= text.charCodeAt(i);+ // h *= 16777619, kept in 32-bit via the shift-add decomposition.+ h = (h + ((h << 1) + (h << 4) + (h << 7) + (h << 8) + (h << 24))) >>> 0;+ }+ return h >>> 0;+}++/** Tokens (unigrams) plus padded character trigrams, the hashed features. */+function* features(text: string): Generator<string> {+ const tokens = text+ .normalize("NFC")+ .toLowerCase()+ .split(/[^\p{L}\p{N}]+/u);+ for (const tok of tokens) {+ if (tok.length === 0) continue;+ yield `w:${tok}`;+ const padded = `^${tok}$`;+ for (let i = 0; i + 3 <= padded.length; i++) {+ yield `t:${padded.slice(i, i + 3)}`;+ }+ }+}++function embedOne(text: string, dimension: number): number[] {+ const vec: number[] = Array.from({ length: dimension }, () => 0);+ for (const feature of features(text)) {+ const bucket = fnv1a(feature, 0) % dimension;+ const sign = (fnv1a(feature, 0x9e3779b9) & 1) === 0 ? 1 : -1;+ vec[bucket]! += sign;+ }+ let sumSquares = 0;+ for (const x of vec) sumSquares += x * x;+ const len = Math.sqrt(sumSquares);+ if (len === 0) return vec; // empty / featureless text -> zero vector+ for (let i = 0; i < dimension; i++) vec[i]! /= len;+ return vec;+}++export class LocalProvider implements EmbeddingProvider {+ readonly name = "local";+ readonly model = LOCAL_EMBEDDING_MODEL;+ readonly dimension: number;++ constructor(dimension: number = LOCAL_DEFAULT_DIMENSION) {+ this.dimension = Math.max(1, Math.floor(dimension));+ }++ async embed(texts: ReadonlyArray<string>): Promise<number[][]> {+ return texts.map((t) => embedOne(t, this.dimension));+ }++ async ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }> {+ return { ok: true, dimension: this.dimension };+ }+}diff --git a/src/core/search/embeddings/provider.ts b/src/core/search/embeddings/provider.tsindex 10e10810..14db8971 100644--- a/src/core/search/embeddings/provider.ts+++ b/src/core/search/embeddings/provider.ts@@ -32,6 +32,12 @@ export function makeProvider(config: ResolvedEmbeddingConfig): EmbeddingProviderconst { NullProvider } = require("./null-provider.ts") as typeof import("./null-provider.ts");return new NullProvider();}+ if (config.provider === "local") {+ // eslint-disable-next-line @typescript-eslint/no-require-imports+ const { LocalProvider, LOCAL_DEFAULT_DIMENSION } =+ require("./local-provider.ts") as typeof import("./local-provider.ts");+ return new LocalProvider(config.dimension ?? LOCAL_DEFAULT_DIMENSION);+ }if (config.provider === "openai-compat") {// eslint-disable-next-line @typescript-eslint/no-require-importsconst { OpenAICompatProvider } =diff --git a/src/core/search/embeddings/registry.ts b/src/core/search/embeddings/registry.tsnew file mode 100644index 00000000..38023925--- /dev/null+++ b/src/core/search/embeddings/registry.ts@@ -0,0 +1,167 @@+/**+ * Provider registry (Embedding Provider Suite).+ *+ * Named provider profiles persisted to+ * `Brain/search/embedding-providers.json`, added and removed at runtime+ * through the CLI so users register an OpenAI-compatible endpoint+ * without editing config. A registered name resolves to `openai-compat`+ * config at config-resolution time (see `expandRegisteredProvider`),+ * AFTER the built-ins, so a configured built-in key is never shadowed+ * and the resolved provider union stays closed.+ *+ * Only env-key NAMES are stored, never secret values - the file is safe+ * to sync. Loading is fail-soft: an absent or malformed registry yields+ * an empty list and never throws into the hot path.+ */++import { mkdirSync, readFileSync, writeFileSync } from "node:fs";+import { dirname, join } from "node:path";++import { SearchError } from "../types.ts";++/** A registered OpenAI-compatible provider profile. */+export interface ProviderProfile {+ readonly name: string;+ readonly baseUrl: string;+ readonly defaultModel: string;+ /** Environment variable NAME the API key is read from (never the key). */+ readonly envKey: string;+}++/** Built-in provider names a profile may not reuse. */+export const RESERVED_PROVIDER_NAMES: ReadonlyArray<string> = Object.freeze([+ "openai-compat",+ "disabled",+ "local",+]);++const NAME_RE = /^[a-z0-9][a-z0-9-]*$/;++export function providerRegistryPath(vault: string): string {+ return join(vault, "Brain", "search", "embedding-providers.json");+}++function isProfile(value: unknown): value is ProviderProfile {+ if (value === null || typeof value !== "object") return false;+ const v = value as Record<string, unknown>;+ return (+ typeof v["name"] === "string" &&+ typeof v["baseUrl"] === "string" &&+ typeof v["defaultModel"] === "string" &&+ typeof v["envKey"] === "string"+ );+}++/** Load the registry, fail-soft: absent or malformed file -> empty list. */+export function loadProviderRegistry(vault: string): ProviderProfile[] {+ let text: string;+ try {+ text = readFileSync(providerRegistryPath(vault), "utf8");+ } catch {+ return [];+ }+ let parsed: unknown;+ try {+ parsed = JSON.parse(text);+ } catch {+ return [];+ }+ if (!Array.isArray(parsed)) return [];+ return parsed.filter(isProfile).map((p) => Object.freeze({ ...p }));+}++function writeProviderRegistry(vault: string, registry: ReadonlyArray<ProviderProfile>): void {+ const path = providerRegistryPath(vault);+ mkdirSync(dirname(path), { recursive: true });+ // Sort by name so the on-disk file is deterministic across edits.+ const sorted = [...registry].toSorted((a, b) => a.name.localeCompare(b.name));+ writeFileSync(path, JSON.stringify(sorted, null, 2) + "\n");+}++function validateProfile(profile: ProviderProfile): void {+ const name = profile.name.trim();+ if (!NAME_RE.test(name)) {+ throw new SearchError(+ "INVALID_INPUT",+ `provider name must be a lowercase slug ([a-z0-9-], starting alphanumeric), got '${profile.name}'`,+ );+ }+ if (RESERVED_PROVIDER_NAMES.includes(name)) {+ throw new SearchError(+ "INVALID_INPUT",+ `provider name '${name}' is reserved for a built-in provider`,+ );+ }+ if (profile.baseUrl.trim() === "") {+ throw new SearchError("INVALID_INPUT", "provider base-url must not be empty");+ }+ if (profile.defaultModel.trim() === "") {+ throw new SearchError("INVALID_INPUT", "provider default-model must not be empty");+ }+ if (profile.envKey.trim() === "") {+ throw new SearchError("INVALID_INPUT", "provider env-key must not be empty");+ }+}++/**+ * Add (or upsert by name) a provider profile and persist. Returns the+ * full registry after the change.+ */+export function addProviderProfile(vault: string, profile: ProviderProfile): ProviderProfile[] {+ validateProfile(profile);+ const normalised: ProviderProfile = Object.freeze({+ name: profile.name.trim(),+ baseUrl: profile.baseUrl.trim(),+ defaultModel: profile.defaultModel.trim(),+ envKey: profile.envKey.trim(),+ });+ const without = loadProviderRegistry(vault).filter((p) => p.name !== normalised.name);+ const next = [...without, normalised];+ writeProviderRegistry(vault, next);+ return loadProviderRegistry(vault);+}++/** Remove a profile by name; reports whether one was present. */+export function removeProviderProfile(+ vault: string,+ name: string,+): { removed: boolean; registry: ProviderProfile[] } {+ const current = loadProviderRegistry(vault);+ const next = current.filter((p) => p.name !== name);+ const removed = next.length !== current.length;+ if (removed) writeProviderRegistry(vault, next);+ return { removed, registry: loadProviderRegistry(vault) };+}++export function getProviderProfile(vault: string, name: string): ProviderProfile | null {+ return loadProviderRegistry(vault).find((p) => p.name === name) ?? null;+}++/** Fields a registered name expands into at config-resolution time. */+export interface ExpandedProvider {+ readonly provider: "openai-compat";+ readonly baseUrl: string;+ readonly model: string;+ readonly apiKey: string | null;+}++/**+ * Expand a registered provider name into `openai-compat` config, reading+ * the API key from `env[profile.envKey]`. Returns null when the name is+ * not registered (the caller then falls back to built-in resolution).+ */+export function expandRegisteredProvider(+ name: string,+ registry: ReadonlyArray<ProviderProfile>,+ env: Readonly<Record<string, string | undefined>>,+): ExpandedProvider | null {+ const profile = registry.find((p) => p.name === name);+ if (!profile) return null;+ const apiKey = env[profile.envKey];+ return Object.freeze({+ provider: "openai-compat" as const,+ baseUrl: profile.baseUrl,+ model: profile.defaultModel,+ apiKey: apiKey !== undefined && apiKey !== "" ? apiKey : null,+ });+}diff --git a/src/core/search/embeddings/signature.ts b/src/core/search/embeddings/signature.tsnew file mode 100644index 00000000..2b276c46--- /dev/null+++ b/src/core/search/embeddings/signature.ts@@ -0,0 +1,115 @@+/**+ * Embedding signature + cost kernel (Embedding Provider Suite).+ *+ * The single source of provider-identity truth shared by the local+ * embedder, the provider registry, the cost gate, and the store's+ * corpus-generation fingerprint. Centralising it here keeps those call+ * sites from drifting on what "the same embedding configuration" means.+ *+ * Pure and I/O-free: canonicalise an identity to a stable signature+ * string, look up a best-effort per-model price, estimate tokens and+ * spend, and compare two signatures for staleness.+ */++/** Identity triple that determines whether two embeddings are comparable. */+export interface EmbeddingIdentity {+ readonly provider: string;+ readonly model: string | null;+ readonly dimension: number | null;+}++/** Model name produced by the offline local embedder (priced at 0). */+export const LOCAL_EMBEDDING_MODEL = "hashing-ngram-v1";++/** Sentinel for a null model/dimension so signatures stay parseable. */+const NULL_FIELD = "?";++function canonicalToken(raw: string): string {+ return raw.normalize("NFC").trim().toLowerCase();+}++/**+ * Canonical signature `<provider>:<model>:<dimension>`. Provider and+ * model are NFC-normalised, trimmed, and lowercased; a null model or+ * dimension renders as the stable `?` sentinel. Two configurations that+ * produce the same signature yield comparable vectors.+ */+export function embeddingSignature(id: EmbeddingIdentity): string {+ const provider = canonicalToken(id.provider);+ const model = id.model === null ? NULL_FIELD : canonicalToken(id.model);+ const dimension = id.dimension === null ? NULL_FIELD : String(id.dimension);+ return `${provider}:${model}:${dimension}`;+}++/**+ * Best-effort embedding price in USD per million input tokens, keyed by+ * model name. Rates drift as providers change pricing, so the table is+ * deliberately small and any model NOT listed is treated as free+ * (price 0) - the cost gate must never falsely block on a model whose+ * rate we do not know. The local embedder is listed explicitly at 0.+ */+export const EMBEDDING_PRICING: Readonly<Record<string, number>> = Object.freeze({+ [LOCAL_EMBEDDING_MODEL]: 0,+ "text-embedding-3-small": 0.02,+ "text-embedding-3-large": 0.13,+ "text-embedding-ada-002": 0.1,+});++/** Per-million-token price for a model; 0 for the local/unknown/null case. */+export function pricePerMillionTokens(model: string | null): number {+ if (model === null) return 0;+ const rate = EMBEDDING_PRICING[canonicalToken(model)];+ return typeof rate === "number" && Number.isFinite(rate) && rate > 0 ? rate : 0;+}++/**+ * Cheap token estimate: a chars/4 heuristic per text, rounded up, summed.+ * This is the same order-of-magnitude approximation embedding providers+ * use for quick quotes; it intentionally avoids a real tokenizer so the+ * estimate stays dependency-free and deterministic.+ */+export function estimateTokens(texts: ReadonlyArray<string>): number {+ let total = 0;+ for (const t of texts) {+ if (t.length === 0) continue;+ total += Math.ceil(t.length / 4);+ }+ return total;+}++/** Estimated spend in USD for `tokens` against `model`'s rate. */+export function estimateCostUsd(tokens: number, model: string | null): number {+ const rate = pricePerMillionTokens(model);+ if (rate === 0) return 0;+ return (tokens / 1_000_000) * rate;+}++/** True when the active signature differs from the stored one. */+export function isStaleSignature(active: string, stored: string): boolean {+ return active !== stored;+}++/** Outcome of a cost-gate evaluation for an embedding run. */+export interface CostGateResult {+ readonly tokens: number;+ readonly estimatedUsd: number;+ readonly blocked: boolean;+}++/**+ * Evaluate whether an embedding run should be blocked on estimated spend.+ * A run is blocked only when the gate is positive, the run is not forced,+ * and the estimate strictly exceeds the gate. A zero gate (the default)+ * and free models (local/unknown -> estimate 0) never block.+ */+export function evaluateCostGate(opts: {+ texts: ReadonlyArray<string>;+ model: string | null;+ gateUsd: number;+ forced?: boolean;+}): CostGateResult {+ const tokens = estimateTokens(opts.texts);+ const estimatedUsd = estimateCostUsd(tokens, opts.model);+ const blocked = opts.gateUsd > 0 && opts.forced !== true && estimatedUsd > opts.gateUsd;+ return { tokens, estimatedUsd, blocked };+}diff --git a/src/core/search/fusion.ts b/src/core/search/fusion.tsnew file mode 100644index 00000000..004c2a43--- /dev/null+++ b/src/core/search/fusion.ts@@ -0,0 +1,66 @@+/**+ * Rank-fusion strategies for hybrid recall (Embedding Provider Suite).+ *+ * The default `linear` mode (a weighted sum of min-max-normalised BM25+ * and cosine, computed in `ranker.ts`) is unchanged. This module adds+ * `rrf` - Reciprocal Rank Fusion - which combines the sparse and dense+ * lanes by RANK POSITION rather than by score magnitude:+ *+ * rrf(chunk) = sum over lanes of 1 / (k + rank_in_lane)+ *+ * RRF is weightless by design (it ignores the per-lane weights and the+ * intent multipliers) and robust to lanes whose score scales differ. The+ * raw RRF sums are min-max-normalised to [0, 1] so the fused relevance is+ * on the same scale as the linear path and composes with the same+ * downstream boosts (link / recency / entity / tier / session focus).+ */++/** Canonical RRF damping constant from the original Cormack et al. paper. */+export const DEFAULT_RRF_K = 60;++export type FusionMode = "linear" | "rrf";++const FUSION_MODES: ReadonlySet<string> = new Set(["linear", "rrf"]);++export function isFusionMode(value: string): value is FusionMode {+ return FUSION_MODES.has(value);+}++/**+ * Fuse two lanes by reciprocal rank and min-max-normalise the result to+ * [0, 1]. Each input is the lane's chunk ids in best-first order; a chunk+ * contributes `1 / (k + position)` for each lane it appears in. Returns a+ * map from chunk id to its normalised fused relevance. An empty input on+ * both lanes yields an empty map.+ */+export function rrfFuse(opts: {+ keywordRankedChunkIds: ReadonlyArray<number>;+ semanticRankedChunkIds: ReadonlyArray<number>;+ k: number;+}): Map<number, number> {+ const k = Math.max(1, opts.k);+ const raw = new Map<number, number>();+ const accumulate = (ids: ReadonlyArray<number>): void => {+ for (let i = 0; i < ids.length; i++) {+ const id = ids[i]!;+ raw.set(id, (raw.get(id) ?? 0) + 1 / (k + (i + 1)));+ }+ };+ accumulate(opts.keywordRankedChunkIds);+ accumulate(opts.semanticRankedChunkIds);++ if (raw.size === 0) return raw;++ const scores = [...raw.values()];+ const min = Math.min(...scores);+ const max = Math.max(...scores);+ const out = new Map<number, number>();+ if (max === min) {+ for (const id of raw.keys()) out.set(id, 1);+ return out;+ }+ for (const [id, score] of raw) {+ out.set(id, (score - min) / (max - min));+ }+ return out;+}diff --git a/src/core/search/index.ts b/src/core/search/index.tsindex 2e931e72..8182a639 100644--- a/src/core/search/index.ts+++ b/src/core/search/index.ts@@ -12,6 +12,12 @@ import {} from "../validate.ts";import { resolveVaultScope } from "../vault-scope/index.ts";import { resolveIndexPath } from "./paths.ts";+import { isFusionMode, DEFAULT_RRF_K } from "./fusion.ts";+import {+ loadProviderRegistry,+ expandRegisteredProvider,+ type ExpandedProvider,+} from "./embeddings/registry.ts";import { SearchError } from "./types.ts";import type {ResolvedEmbeddingConfig,@@ -54,6 +60,15 @@ export {} from "./session-focus.ts";export { evaluateSurfacingGate, type SurfacingGateDecision } from "./surfacing-gate.ts";export { buildEvidencePack, type EvidencePack } from "./evidence-pack.ts";+export {+ loadProviderRegistry,+ addProviderProfile,+ removeProviderProfile,+ getProviderProfile,+ providerRegistryPath,+ RESERVED_PROVIDER_NAMES,+ type ProviderProfile,+} from "./embeddings/registry.ts";export { resolveIndexPath } from "./paths.ts";export {@@ -88,6 +103,7 @@ const DEFAULTS = {timeoutMs: 10_000,concurrency: 4,batchSize: 32,+ costGateUsd: 0,mmrLambda: 0.7,maxHops: 1,hopDecay: 0.5,@@ -97,8 +113,19 @@ const DEFAULTS = {recencyAmplitude: 0.05,synonymMaxTerms: 3,cacheTtlSeconds: 300,+ fusionMode: "linear" as const,+ rrfK: DEFAULT_RRF_K,};+function parseFusionMode(raw: string | null): "linear" | "rrf" {+ if (raw === null) return DEFAULTS.fusionMode;+ if (isFusionMode(raw)) return raw;+ throw new SearchError(+ "INVALID_INPUT",+ `search_fusion_mode must be 'linear' or 'rrf', got '${raw}'`,+ );+}+type IntegerRange = { readonly min?: number; readonly max?: number };function parseInteger(@@ -140,6 +167,16 @@ function parsePositiveFloat(raw: string | null, fallback: number, fieldName: strreturn n;}+/** Parse a non-negative finite float (e.g. a cost gate; 0 disables). */+function parseNonNegativeFloat(raw: string | null, fallback: number, fieldName: string): number {+ if (raw === null) return fallback;+ const n = Number(raw);+ if (!Number.isFinite(n) || n < 0) {+ throw new SearchError("INVALID_INPUT", `${fieldName} must be a number >= 0, got '${raw}'`);+ }+ return n;+}+function validateIntegerRange(n: number, fieldName: string, range?: IntegerRange): void {if (range?.min !== undefined && n < range.min) {throw new SearchError("INVALID_INPUT", `${fieldName} must be >= ${range.min}, got ${n}`);@@ -190,13 +227,35 @@ function validateResolvedConfig(config: ResolvedSearchConfig): void {function parseProvider(raw: string | null): ResolvedEmbeddingConfig["provider"] {if (raw === null) return DEFAULTS.provider;- if (raw === "openai-compat" || raw === "disabled") return raw;+ if (raw === "openai-compat" || raw === "disabled" || raw === "local") return raw;throw new SearchError("INVALID_INPUT",- `embedding_provider must be 'openai-compat' or 'disabled', got '${raw}'`,+ `embedding_provider must be 'openai-compat', 'local', 'disabled', or a registered provider name, got '${raw}'`,);}+const BUILTIN_PROVIDERS: ReadonlySet<string> = new Set(["openai-compat", "disabled", "local"]);++/**+ * Resolve a non-built-in `embedding_provider` name against the registry.+ * Returns null when the name is null, a built-in, or not registered - the+ * caller then defers to `parseProvider` (which validates built-ins and+ * raises a clear error for an unknown name). Fail-soft: a bad registry+ * yields null, never an exception.+ */+function resolveRegistryProvider(+ rawProvider: string | null,+ vault: string,+ env: NodeJS.ProcessEnv,+): ExpandedProvider | null {+ if (rawProvider === null || BUILTIN_PROVIDERS.has(rawProvider)) return null;+ try {+ return expandRegisteredProvider(rawProvider, loadProviderRegistry(vault), env);+ } catch {+ return null;+ }+}+export function resolveSearchConfig(opts: {vault: string;configPath?: string;@@ -234,6 +293,15 @@ export function resolveSearchConfig(opts: {DEFAULTS.semanticWeight,"search_semantic_weight",);+ const fusionMode = parseFusionMode(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_FUSION_MODE", "search_fusion_mode"),+ );+ const rrfK = parseInteger(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RRF_K", "search_rrf_k"),+ DEFAULTS.rrfK,+ "search_rrf_k",+ { min: 1 },+ );// v0.10.9: single source of truth lives in Brain/_brain.yaml under// `vault.ignore_paths`. The legacy `search_ignore_paths` config key@@ -246,17 +314,40 @@ export function resolveSearchConfig(opts: {false,"search_semantic_enabled",);- const provider = parseProvider(- envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER", "embedding_provider"),+ const rawProvider = envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER",+ "embedding_provider",);- const baseUrl = envOrConfig(+ // A provider name that is not a built-in is looked up in the registry+ // AFTER the built-ins, so it never shadows an explicitly configured+ // built-in. A registered name resolves to openai-compat fields; an+ // unknown name falls through to parseProvider's clear validation error.+ const registryExpansion = resolveRegistryProvider(rawProvider, opts.vault, env);+ const provider = registryExpansion ? "openai-compat" : parseProvider(rawProvider);+ const explicitBaseUrl = envOrConfig(env,config,"OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL","embedding_base_url",);- const model = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_MODEL", "embedding_model");- const apiKey = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_KEY", "embedding_api_key");+ const explicitModel = envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_EMBEDDING_MODEL",+ "embedding_model",+ );+ const explicitApiKey = envOrConfig(+ env,+ config,+ "OPEN_SECOND_BRAIN_EMBEDDING_KEY",+ "embedding_api_key",+ );+ // Explicit config/env always wins over the registry profile's fields.+ const baseUrl = explicitBaseUrl ?? registryExpansion?.baseUrl ?? null;+ const model = explicitModel ?? registryExpansion?.model ?? null;+ const apiKey = explicitApiKey ?? registryExpansion?.apiKey ?? null;const dimRaw = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_DIM", "embedding_dimension");const dimension =dimRaw === null ? null : parseInteger(dimRaw, 0, "embedding_dimension", { min: 1 });@@ -278,6 +369,11 @@ export function resolveSearchConfig(opts: {"embedding_batch_size",{ min: 1 },);+ const costGateUsd = parseNonNegativeFloat(+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_COST_GATE", "embedding_cost_gate_usd"),+ DEFAULTS.costGateUsd,+ "embedding_cost_gate_usd",+ );const semantic: ResolvedEmbeddingConfig = Object.freeze({enabled: semanticEnabled,@@ -289,6 +385,7 @@ export function resolveSearchConfig(opts: {timeoutMs,concurrency,batchSize,+ costGateUsd,});const mmrLambda = parseFloat01(@@ -415,6 +512,8 @@ export function resolveSearchConfig(opts: {chunkOverlap,keywordWeight,semanticWeight,+ fusionMode,+ rrfK,semantic,recall,});diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.tsindex 3ff8a583..cc65c22b 100644--- a/src/core/search/indexer.ts+++ b/src/core/search/indexer.ts@@ -28,6 +28,14 @@ import { basename, dirname } from "node:path";import { chunkMarkdown } from "./chunker.ts";import { expandTextForCjkFts } from "./cjk-tokenizer.ts";import { makeProvider } from "./embeddings/provider.ts";+import {+ embeddingSignature,+ estimateCostUsd,+ estimateTokens,+ evaluateCostGate,+ pricePerMillionTokens,+ LOCAL_EMBEDDING_MODEL,+} from "./embeddings/signature.ts";import { extractLinks } from "./links.ts";import { extractFrontmatterRelations } from "../graph/frontmatter-relations.ts";import { parseFrontmatterText } from "../vault.ts";@@ -56,6 +64,8 @@ export interface IndexVaultOptions {readonly embeddings?: boolean;/** When true, every file is reindexed even if hash + mtime match. */readonly force?: boolean;+ /** When true, bypass the embedding cost gate for this run. */+ readonly forceCost?: boolean;readonly onFile?: (event: IndexProgressEvent) => void;}@@ -264,7 +274,7 @@ async function indexInto(store.resolveLinkTargets();if (opts?.embeddings) {- await populateEmbeddings(store, config, stats);+ await populateEmbeddings(store, config, stats, opts?.forceCost === true);}const now = new Date().toISOString();@@ -285,10 +295,30 @@ async function indexInto(}}+/**+ * Canonical signature of the ACTIVE embedding configuration for status+ * reporting. Null when semantic search is disabled. Stored model and+ * dimension fill in fields the config leaves unset (e.g. an+ * auto-detected dimension or the local provider's implicit model).+ */+function activeEmbeddingSignature(+ config: ResolvedSearchConfig,+ storedModel: string | null = null,+ storedDim: number | null = null,+): string | null {+ if (!config.semantic.enabled) return null;+ const provider = config.semantic.provider;+ const model =+ provider === "local" ? LOCAL_EMBEDDING_MODEL : (config.semantic.model ?? storedModel);+ const dimension = config.semantic.dimension ?? storedDim;+ return embeddingSignature({ provider, model, dimension });+}+async function populateEmbeddings(store: Store,config: ResolvedSearchConfig,stats: MutableStats,+ forceCost: boolean,): Promise<void> {if (!config.semantic.enabled) {throw new SearchError(@@ -296,7 +326,8 @@ async function populateEmbeddings("set search_semantic_enabled=true and embedding_* keys to compute embeddings",);}- if (!config.semantic.apiKey) {+ // The offline local provider needs no key; every remote provider does.+ if (config.semantic.provider !== "local" && !config.semantic.apiKey) {throw new SearchError("EMBEDDING_KEY_MISSING","embedding_api_key is required when computing embeddings",@@ -314,6 +345,24 @@ async function populateEmbeddings(const provider = makeProvider(config.semantic);const model = config.semantic.model ?? provider.model;++ // Cost gate: estimate the spend for the whole pending set up front and+ // refuse the run when it exceeds the configured ceiling, unless forced.+ // The local provider (price 0) and unknown-price models never block.+ const gate = evaluateCostGate({+ texts: pending.map((p) => p.content),+ model,+ gateUsd: config.semantic.costGateUsd,+ forced: forceCost,+ });+ if (gate.blocked) {+ throw new SearchError(+ "EMBEDDING_COST_GATE",+ `estimated embedding cost $${gate.estimatedUsd.toFixed(4)} for ${pending.length} chunk(s) ` ++ `exceeds embedding_cost_gate_usd $${config.semantic.costGateUsd.toFixed(4)}. ` ++ `Re-run with --force-cost to proceed or raise the gate.`,+ );+ }const batchSize = Math.max(1, config.semantic.batchSize);// Hand the provider a super-batch sized to fully saturate its// internal `embedding_concurrency` semaphore. Without this multiplier@@ -423,6 +472,8 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexStstaleEmbeddings: 0,embeddingModel: null,embeddingDimension: null,+ embeddingSignature: activeEmbeddingSignature(config),+ estimatedRefreshCostUsd: 0,vecExtension: "unknown" as const,semanticEnabled: config.semantic.enabled,embeddingKeyPresent: !!config.semantic.apiKey,@@ -445,10 +496,29 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexStif (config.semantic.enabled && !store.vecLoaded()) {warnings.push("sqlite-vec unavailable; semantic search disabled this session");}- if (config.semantic.enabled && !config.semantic.apiKey) {+ if (+ config.semantic.enabled &&+ config.semantic.provider !== "local" &&+ !config.semantic.apiKey+ ) {warnings.push("embedding_api_key not configured; semantic search disabled");}+ // Best-effort spend estimate to bring stale/missing embeddings current.+ // Only scan chunk content when the active model is actually priced.+ const activeModel =+ config.semantic.provider === "local"+ ? LOCAL_EMBEDDING_MODEL+ : (config.semantic.model ?? model);+ let estimatedRefreshCostUsd = 0;+ if (config.semantic.enabled && pricePerMillionTokens(activeModel) > 0) {+ const pending = store.findChunksWithoutEmbeddings();+ estimatedRefreshCostUsd = estimateCostUsd(+ estimateTokens(pending.map((p) => p.content)),+ activeModel,+ );+ }+return Object.freeze({indexPath: config.dbPath,exists: true,@@ -459,6 +529,8 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexStstaleEmbeddings: counts.staleEmbeddings,embeddingModel: model,embeddingDimension: dim,+ embeddingSignature: activeEmbeddingSignature(config, model, dim),+ estimatedRefreshCostUsd,vecExtension: store.vecLoaded() ? ("loaded" as const) : ("unavailable" as const),semanticEnabled: config.semantic.enabled,embeddingKeyPresent: !!config.semantic.apiKey,diff --git a/src/core/search/ranker.ts b/src/core/search/ranker.tsindex b1238013..209e52b8 100644--- a/src/core/search/ranker.ts+++ b/src/core/search/ranker.ts@@ -12,6 +12,7 @@import { PAGE_TIER_DEFAULT, tierWeight, type PageTier } from "../brain/page-meta/tier.ts";import { weibullDecay, DEFAULT_RECENCY, type WeibullRecencyOptions } from "./recency.ts";import { scoreSessionFocusTarget, type SearchSessionFocus } from "./session-focus.ts";+import { rrfFuse, DEFAULT_RRF_K, type FusionMode } from "./fusion.ts";import type { KeywordHit, SemanticHit, HydratedChunk } from "./store.ts";import type { BrainSearchResult, WeightProfile } from "./types.ts";@@ -60,6 +61,15 @@ export interface RankerOptions {*/readonly weightProfile?: WeightProfile;readonly sessionFocus?: SearchSessionFocus | null;+ /**+ * Rank-fusion mode (Embedding Provider Suite). `linear` (default) is+ * the weighted sum of normalised BM25 and cosine; `rrf` fuses the two+ * lanes by reciprocal rank. Absent or `linear` keeps ranking+ * bit-identical to pre-suite behaviour.+ */+ readonly fusionMode?: FusionMode;+ /** RRF damping constant; defaults to {@link DEFAULT_RRF_K}. */+ readonly rrfK?: number;}const DAY_MS = 24 * 60 * 60 * 1000;@@ -131,10 +141,12 @@ function buildReasons(parts: {tierMul: number;entityBoost?: number;sessionFocus?: number;+ rrf?: number;}): ReadonlyArray<string> {const reasons: string[] = [];if (parts.keywordScore > 0) reasons.push(`fts5_bm25: ${fmt(parts.keywordScore)}`);if (parts.semanticScore > 0) reasons.push(`semantic_cos: ${fmt(parts.semanticScore)}`);+ if (parts.rrf && parts.rrf > 0) reasons.push(`rrf: ${fmt(parts.rrf)}`);if (parts.entityBoost && parts.entityBoost > 0) {reasons.push(`entity_match: ${fmt(parts.entityBoost)}`);}@@ -167,6 +179,25 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea}}+ // Reciprocal Rank Fusion (Embedding Provider Suite): fuse the lanes by+ // rank position instead of weighted magnitude. Off by default; when on,+ // it replaces the linear relevance term while every boost still applies.+ const fusionMode: FusionMode = opts.fusionMode ?? "linear";+ let rrfByChunk: Map<number, number> | null = null;+ if (fusionMode === "rrf") {+ const keywordRanked = inputs.keyword+ .toSorted((a, b) => a.bm25 - b.bm25) // lower BM25 = better+ .map((h) => h.chunkId);+ const semanticRanked = semanticEnabled+ ? inputs.semantic.toSorted((a, b) => a.distance - b.distance).map((h) => h.chunkId)+ : [];+ rrfByChunk = rrfFuse({+ keywordRankedChunkIds: keywordRanked,+ semanticRankedChunkIds: semanticRanked,+ k: opts.rrfK ?? DEFAULT_RRF_K,+ });+ }+const candidates = new Map<number, Candidate>();for (const h of inputs.keyword) {candidates.set(h.chunkId, {@@ -248,9 +279,15 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSeaconst tag = tagBoostFor(c);const linkBoost = Math.min(0.05, link + tag);const recency = recencyBoost(c.mtime, nowMs, recencyOpts) * recMul;+ // Relevance term: reciprocal-rank-fused when in rrf mode, otherwise+ // the weighted sum of the normalised lanes. RRF is weightless, so the+ // per-lane weights and intent multipliers do not apply to it.+ const rrf = rrfByChunk?.get(c.chunkId) ?? 0;const weighted =- opts.keywordWeight * kwMul * c.keywordScore +- (semanticEnabled ? opts.semanticWeight * semMul : 0) * c.semanticScore;+ rrfByChunk !== null+ ? rrf+ : opts.keywordWeight * kwMul * c.keywordScore ++ (semanticEnabled ? opts.semanticWeight * semMul : 0) * c.semanticScore;// Tier multiplier applied to the relevance portion only so the// tag / link / recency boosts stay tier-neutral. Default// `supporting` β 1.0 keeps untagged vaults bit-identical.@@ -287,6 +324,7 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSeatierMul,entityBoost,sessionFocus,+ rrf: rrfByChunk !== null ? rrf : 0,}),}),);diff --git a/src/core/search/search.ts b/src/core/search/search.tsindex d7200f3e..97bd19e5 100644--- a/src/core/search/search.ts+++ b/src/core/search/search.ts@@ -451,6 +451,8 @@ export async function search(},...(weightProfile !== undefined ? { weightProfile } : {}),...(sessionFocus !== undefined ? { sessionFocus } : {}),+ fusionMode: config.fusionMode,+ rrfK: config.rrfK,},);const capHit = ranked.length >= rankCap;@@ -820,7 +822,8 @@ async function runSemanticPhase(warnings.push("semantic not enabled in config; using keyword-only");return { attempted: false, hits: [], warnings };}- if (!config.semantic.apiKey) {+ // The offline local provider needs no key; every remote provider does.+ if (config.semantic.provider !== "local" && !config.semantic.apiKey) {if (opts.explicit) {throw new SearchError("EMBEDDING_KEY_MISSING", "embedding key not configured");}diff --git a/src/core/search/types.ts b/src/core/search/types.tsindex c40d64f0..19697665 100644--- a/src/core/search/types.ts+++ b/src/core/search/types.ts@@ -21,6 +21,7 @@ export const SEARCH_ERROR_CODES = ["EMBEDDING_PROVIDER_HTTP","EMBEDDING_PROVIDER_TIMEOUT","EMBEDDING_DIMENSION_MISMATCH",+ "EMBEDDING_COST_GATE","INDEX_LOCKED","INVALID_INPUT",] as const;@@ -136,6 +137,19 @@ export interface IndexStatusSnapshot {readonly staleEmbeddings: number;readonly embeddingModel: string | null;readonly embeddingDimension: number | null;+ /**+ * Canonical `<provider>:<model>:<dimension>` fingerprint of the ACTIVE+ * embedding configuration (Embedding Provider Suite). Null when+ * semantic search is disabled. Compare with the stored model/dimension+ * to reason about staleness after a config change.+ */+ readonly embeddingSignature: string | null;+ /**+ * Best-effort USD estimate to (re-)embed the chunks that currently+ * lack a current embedding, at the active model's rate. 0 for the+ * local/unknown-price case.+ */+ readonly estimatedRefreshCostUsd: number;readonly vecExtension: VecExtensionState;readonly semanticEnabled: boolean;readonly embeddingKeyPresent: boolean;@@ -232,7 +246,7 @@ export interface SearchOutcome {export interface ResolvedEmbeddingConfig {readonly enabled: boolean;- readonly provider: "openai-compat" | "disabled";+ readonly provider: "openai-compat" | "disabled" | "local";readonly baseUrl: string | null;readonly model: string | null;readonly apiKey: string | null;@@ -240,6 +254,12 @@ export interface ResolvedEmbeddingConfig {readonly timeoutMs: number;readonly concurrency: number;readonly batchSize: number;+ /**+ * Spend ceiling in USD for a single embedding run (Embedding Provider+ * Suite). 0 (default) disables the gate. When positive, an embedding+ * run whose estimated cost exceeds this is refused unless forced.+ */+ readonly costGateUsd: number;}/**@@ -328,6 +348,15 @@ export interface ResolvedSearchConfig {readonly chunkOverlap: number;readonly keywordWeight: number;readonly semanticWeight: number;+ /**+ * Rank-fusion mode (Embedding Provider Suite). `linear` (default) is+ * the weighted sum of normalised BM25 and cosine; `rrf` fuses the two+ * lanes by reciprocal rank. `linear` keeps ranking bit-identical to+ * pre-suite behaviour.+ */+ readonly fusionMode: "linear" | "rrf";+ /** Reciprocal Rank Fusion damping constant (only used when rrf). */+ readonly rrfK: number;readonly semantic: ResolvedEmbeddingConfig;readonly recall: ResolvedRecallConfig;}diff --git a/tests/cli/search-provider-cli.test.ts b/tests/cli/search-provider-cli.test.tsnew file mode 100644index 00000000..33e2c30a--- /dev/null+++ b/tests/cli/search-provider-cli.test.ts@@ -0,0 +1,107 @@+/**+ * CLI tests for `o2b search provider <add|list|show|remove>`.+ *+ * Forks the real `bun src/cli/main.ts` binary via `runCli`, asserting+ * exit codes, JSON shape, and the persisted registry file.+ */++import { afterEach, beforeEach, expect, test } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { runCli } from "../helpers/run-cli.ts";++let tmp: string;+let vault: string;+let config: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-provider-cli-"));+ vault = join(tmp, "vault");+ mkdirSync(vault, { recursive: true });+ config = join(tmp, "config.yaml");+ writeFileSync(config, `vault: "${vault}"\n`);+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++const env = () => ({ OPEN_SECOND_BRAIN_CONFIG: config });+const addArgs = [+ "search",+ "provider",+ "add",+ "nvidia-nim",+ "--base-url",+ "https://integrate.api.nvidia.com/v1",+ "--model",+ "nvidia/nv-embed-v1",+ "--env-key",+ "NIM_API_KEY",+];++test("provider add then list shows the registered profile", async () => {+ const add = await runCli(addArgs, { env: env() });+ expect(add.returncode).toBe(0);+ expect(existsSync(join(vault, "Brain", "search", "embedding-providers.json"))).toBe(true);++ const list = await runCli(["search", "provider", "list"], { env: env() });+ expect(list.returncode).toBe(0);+ expect(list.stdout).toContain("nvidia-nim");+});++test("provider list --json returns the profile array", async () => {+ await runCli(addArgs, { env: env() });+ const list = await runCli(["search", "provider", "list", "--json"], { env: env() });+ expect(list.returncode).toBe(0);+ const arr = JSON.parse(list.stdout);+ expect(Array.isArray(arr)).toBe(true);+ expect(arr[0]).toMatchObject({+ name: "nvidia-nim",+ baseUrl: "https://integrate.api.nvidia.com/v1",+ defaultModel: "nvidia/nv-embed-v1",+ envKey: "NIM_API_KEY",+ });+});++test("provider show prints a single profile", async () => {+ await runCli(addArgs, { env: env() });+ const show = await runCli(["search", "provider", "show", "nvidia-nim", "--json"], { env: env() });+ expect(show.returncode).toBe(0);+ expect(JSON.parse(show.stdout)).toMatchObject({ name: "nvidia-nim" });+});++test("provider remove deletes the profile", async () => {+ await runCli(addArgs, { env: env() });+ const rm = await runCli(["search", "provider", "remove", "nvidia-nim"], { env: env() });+ expect(rm.returncode).toBe(0);+ const list = await runCli(["search", "provider", "list", "--json"], { env: env() });+ expect(JSON.parse(list.stdout)).toEqual([]);+});++test("provider add with a reserved name fails with exit 2", async () => {+ const add = await runCli(+ [+ "search",+ "provider",+ "add",+ "local",+ "--base-url",+ "https://x/v1",+ "--model",+ "m",+ "--env-key",+ "K",+ ],+ { env: env() },+ );+ expect(add.returncode).toBe(2);+ expect(add.stderr).toContain("reserved");+});++test("provider show of an unknown name exits non-zero", async () => {+ const show = await runCli(["search", "provider", "show", "ghost"], { env: env() });+ expect(show.returncode).not.toBe(0);+});diff --git a/tests/core/search/embeddings.cost-gate.test.ts b/tests/core/search/embeddings.cost-gate.test.tsnew file mode 100644index 00000000..610946d6--- /dev/null+++ b/tests/core/search/embeddings.cost-gate.test.ts@@ -0,0 +1,76 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ evaluateCostGate,+ LOCAL_EMBEDDING_MODEL,+} from "../../../src/core/search/embeddings/signature.ts";+import { resolveSearchConfig } from "../../../src/core/search/index.ts";++// ββ pure cost-gate kernel ββββββββββββββββββββββββββββββββββββββββββββββββββββ++test("evaluateCostGate estimates spend for a priced model", () => {+ // 4,000,000 chars -> ~1,000,000 tokens at chars/4.+ const texts = ["x".repeat(4_000_000)];+ const r = evaluateCostGate({ texts, model: "text-embedding-3-small", gateUsd: 0 });+ expect(r.tokens).toBe(1_000_000);+ expect(r.estimatedUsd).toBeGreaterThan(0);+});++test("evaluateCostGate blocks when the estimate exceeds a positive gate", () => {+ const texts = ["x".repeat(4_000_000)]; // ~1M tokens+ const r = evaluateCostGate({ texts, model: "text-embedding-3-small", gateUsd: 0.001 });+ expect(r.blocked).toBe(true);+});++test("evaluateCostGate does not block when forced", () => {+ const texts = ["x".repeat(4_000_000)];+ const r = evaluateCostGate({+ texts,+ model: "text-embedding-3-small",+ gateUsd: 0.001,+ forced: true,+ });+ expect(r.blocked).toBe(false);+});++test("a zero gate (default) never blocks", () => {+ const texts = ["x".repeat(40_000_000)];+ const r = evaluateCostGate({ texts, model: "text-embedding-3-small", gateUsd: 0 });+ expect(r.blocked).toBe(false);+});++test("the local model never blocks regardless of gate or volume", () => {+ const texts = ["x".repeat(40_000_000)];+ const r = evaluateCostGate({ texts, model: LOCAL_EMBEDDING_MODEL, gateUsd: 0.0001 });+ expect(r.estimatedUsd).toBe(0);+ expect(r.blocked).toBe(false);+});++// ββ config parsing βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++let tmp: string;+let config: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-costgate-"));+ config = join(tmp, "config.yaml");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++test("embedding_cost_gate_usd defaults to 0 (disabled)", () => {+ writeFileSync(config, `vault: "${tmp}"\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath: config });+ expect(cfg.semantic.costGateUsd).toBe(0);+});++test("embedding_cost_gate_usd is parsed from config", () => {+ writeFileSync(config, `vault: "${tmp}"\nembedding_cost_gate_usd: "2.5"\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath: config });+ expect(cfg.semantic.costGateUsd).toBe(2.5);+});diff --git a/tests/core/search/embeddings.local-provider.test.ts b/tests/core/search/embeddings.local-provider.test.tsnew file mode 100644index 00000000..05a9c472--- /dev/null+++ b/tests/core/search/embeddings.local-provider.test.ts@@ -0,0 +1,102 @@+import { test, expect } from "bun:test";++import {+ LocalProvider,+ LOCAL_DEFAULT_DIMENSION,+} from "../../../src/core/search/embeddings/local-provider.ts";+import { makeProvider } from "../../../src/core/search/embeddings/provider.ts";+import { LOCAL_EMBEDDING_MODEL } from "../../../src/core/search/embeddings/signature.ts";+import type { ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";++function norm(v: ReadonlyArray<number>): number {+ return Math.sqrt(v.reduce((s, x) => s + x * x, 0));+}++test("identifies as the local provider with the hashing model", () => {+ const p = new LocalProvider(256);+ expect(p.name).toBe("local");+ expect(p.model).toBe(LOCAL_EMBEDDING_MODEL);+ expect(p.dimension).toBe(256);+});++test("default dimension is applied when none is configured", () => {+ const p = new LocalProvider();+ expect(p.dimension).toBe(LOCAL_DEFAULT_DIMENSION);+});++test("produces one unit-normalised vector per input, in order", async () => {+ const p = new LocalProvider(128);+ const out = await p.embed(["alpha beta", "gamma delta", "epsilon"]);+ expect(out.length).toBe(3);+ for (const v of out) {+ expect(v.length).toBe(128);+ expect(norm(v)).toBeCloseTo(1.0, 6);+ }+});++test("is deterministic: identical text yields an identical vector", async () => {+ const p = new LocalProvider(64);+ const [a] = await p.embed(["the quick brown fox"]);+ const [b] = await p.embed(["the quick brown fox"]);+ expect(a).toEqual(b!);+});++test("distinct text yields distinct vectors", async () => {+ const p = new LocalProvider(64);+ const [a, b] = await p.embed(["machine learning", "garden vegetables"]);+ expect(a).not.toEqual(b!);+});++test("empty text yields a zero vector of the right length", async () => {+ const p = new LocalProvider(32);+ const [v] = await p.embed([""]);+ expect(v!.length).toBe(32);+ expect(norm(v!)).toBe(0);+});++test("embed() with no inputs returns an empty array", async () => {+ const p = new LocalProvider();+ expect(await p.embed([])).toEqual([]);+});++test("ping() always succeeds offline with the configured dimension", async () => {+ const p = new LocalProvider(200);+ const res = await p.ping();+ expect(res).toEqual({ ok: true, dimension: 200 });+});++test("makeProvider returns a LocalProvider for provider 'local'", () => {+ const config: ResolvedEmbeddingConfig = Object.freeze({+ enabled: true,+ provider: "local",+ baseUrl: null,+ model: null,+ apiKey: null,+ dimension: 384,+ timeoutMs: 10_000,+ concurrency: 1,+ batchSize: 1,+ costGateUsd: 0,+ });+ const p = makeProvider(config);+ expect(p).toBeInstanceOf(LocalProvider);+ expect(p.dimension).toBe(384);+ expect(p.model).toBe(LOCAL_EMBEDDING_MODEL);+});++test("makeProvider local needs no api key", () => {+ const config: ResolvedEmbeddingConfig = Object.freeze({+ enabled: true,+ provider: "local",+ baseUrl: null,+ model: null,+ apiKey: null,+ dimension: null,+ timeoutMs: 10_000,+ concurrency: 1,+ batchSize: 1,+ costGateUsd: 0,+ });+ expect(() => makeProvider(config)).not.toThrow();+ expect(makeProvider(config).dimension).toBe(LOCAL_DEFAULT_DIMENSION);+});diff --git a/tests/core/search/embeddings.registry.test.ts b/tests/core/search/embeddings.registry.test.tsnew file mode 100644index 00000000..b777cb5e--- /dev/null+++ b/tests/core/search/embeddings.registry.test.ts@@ -0,0 +1,107 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import {+ loadProviderRegistry,+ addProviderProfile,+ removeProviderProfile,+ getProviderProfile,+ expandRegisteredProvider,+ providerRegistryPath,+ RESERVED_PROVIDER_NAMES,+} from "../../../src/core/search/embeddings/registry.ts";+import { SearchError } from "../../../src/core/search/types.ts";++let vault: string;++beforeEach(() => {+ vault = mkdtempSync(join(tmpdir(), "o2b-registry-"));+});++afterEach(() => {+ rmSync(vault, { recursive: true, force: true });+});++const nim = {+ name: "nvidia-nim",+ baseUrl: "https://integrate.api.nvidia.com/v1",+ defaultModel: "nvidia/nv-embed-v1",+ envKey: "NIM_API_KEY",+};++test("empty vault has an empty registry", () => {+ expect(loadProviderRegistry(vault)).toEqual([]);+});++test("add -> list -> get -> remove round-trips a profile", () => {+ const after = addProviderProfile(vault, nim);+ expect(after).toHaveLength(1);+ expect(loadProviderRegistry(vault)).toHaveLength(1);+ expect(getProviderProfile(vault, "nvidia-nim")).toMatchObject(nim);++ const { removed, registry } = removeProviderProfile(vault, "nvidia-nim");+ expect(removed).toBe(true);+ expect(registry).toEqual([]);+ expect(getProviderProfile(vault, "nvidia-nim")).toBeNull();+});++test("registry persists to Brain/search/embedding-providers.json", () => {+ addProviderProfile(vault, nim);+ expect(providerRegistryPath(vault)).toBe(+ join(vault, "Brain", "search", "embedding-providers.json"),+ );+});++test("add upserts an existing name rather than duplicating", () => {+ addProviderProfile(vault, nim);+ const after = addProviderProfile(vault, { ...nim, defaultModel: "nvidia/nv-embed-v2" });+ expect(after).toHaveLength(1);+ expect(getProviderProfile(vault, "nvidia-nim")?.defaultModel).toBe("nvidia/nv-embed-v2");+});++test("reserved built-in names cannot be registered", () => {+ for (const reserved of RESERVED_PROVIDER_NAMES) {+ expect(() => addProviderProfile(vault, { ...nim, name: reserved })).toThrow(SearchError);+ }+});++test("invalid profile fields are rejected", () => {+ expect(() => addProviderProfile(vault, { ...nim, name: "Bad Name" })).toThrow(/name/i);+ expect(() => addProviderProfile(vault, { ...nim, baseUrl: "" })).toThrow(/base/i);+ expect(() => addProviderProfile(vault, { ...nim, defaultModel: "" })).toThrow(/model/i);+ expect(() => addProviderProfile(vault, { ...nim, envKey: "" })).toThrow(/env/i);+});++test("removing an absent profile reports removed:false", () => {+ const { removed } = removeProviderProfile(vault, "ghost");+ expect(removed).toBe(false);+});++test("expandRegisteredProvider resolves to openai-compat with the env key", () => {+ const registry = addProviderProfile(vault, nim);+ const expanded = expandRegisteredProvider("nvidia-nim", registry, { NIM_API_KEY: "secret-123" });+ expect(expanded).toEqual({+ provider: "openai-compat",+ baseUrl: nim.baseUrl,+ model: nim.defaultModel,+ apiKey: "secret-123",+ });+});++test("expandRegisteredProvider yields a null apiKey when the env var is unset", () => {+ const registry = addProviderProfile(vault, nim);+ const expanded = expandRegisteredProvider("nvidia-nim", registry, {});+ expect(expanded?.apiKey).toBeNull();+});++test("expandRegisteredProvider returns null for an unknown name", () => {+ expect(expandRegisteredProvider("missing", [], {})).toBeNull();+});++test("a malformed registry file degrades to empty, never throws", () => {+ mkdirSync(join(vault, "Brain", "search"), { recursive: true });+ writeFileSync(providerRegistryPath(vault), "{ not json");+ expect(loadProviderRegistry(vault)).toEqual([]);+});diff --git a/tests/core/search/embeddings.signature.test.ts b/tests/core/search/embeddings.signature.test.tsnew file mode 100644index 00000000..e46c8638--- /dev/null+++ b/tests/core/search/embeddings.signature.test.ts@@ -0,0 +1,84 @@+import { test, expect } from "bun:test";++import {+ embeddingSignature,+ pricePerMillionTokens,+ estimateTokens,+ estimateCostUsd,+ isStaleSignature,+ EMBEDDING_PRICING,+ LOCAL_EMBEDDING_MODEL,+} from "../../../src/core/search/embeddings/signature.ts";++test("embeddingSignature canonicalises provider/model/dimension", () => {+ expect(+ embeddingSignature({+ provider: "openai-compat",+ model: "text-embedding-3-small",+ dimension: 1536,+ }),+ ).toBe("openai-compat:text-embedding-3-small:1536");+});++test("embeddingSignature lowercases and trims provider + model", () => {+ expect(+ embeddingSignature({+ provider: " OpenAI-Compat ",+ model: " Text-Embedding-3-Small ",+ dimension: 256,+ }),+ ).toBe("openai-compat:text-embedding-3-small:256");+});++test("embeddingSignature renders null model/dimension as a stable sentinel", () => {+ expect(embeddingSignature({ provider: "local", model: null, dimension: null })).toBe("local:?:?");+});++test("two identities with the same fields share a signature; different dim differs", () => {+ const a = embeddingSignature({ provider: "local", model: "hashing-ngram-v1", dimension: 256 });+ const b = embeddingSignature({ provider: "local", model: "hashing-ngram-v1", dimension: 256 });+ const c = embeddingSignature({ provider: "local", model: "hashing-ngram-v1", dimension: 512 });+ expect(a).toBe(b);+ expect(a).not.toBe(c);+});++test("pricePerMillionTokens returns a positive rate for a known remote model", () => {+ expect(pricePerMillionTokens("text-embedding-3-small")).toBeGreaterThan(0);+});++test("pricePerMillionTokens is 0 for the local model and for unknown models", () => {+ expect(pricePerMillionTokens(LOCAL_EMBEDDING_MODEL)).toBe(0);+ expect(pricePerMillionTokens("some-model-nobody-priced")).toBe(0);+ expect(pricePerMillionTokens(null)).toBe(0);+});++test("local model is present in the pricing table at 0", () => {+ expect(EMBEDDING_PRICING[LOCAL_EMBEDDING_MODEL]).toBe(0);+});++test("estimateTokens sums a chars/4 heuristic and rounds up", () => {+ // "abcd" -> 1 token; "abcdefgh" -> 2 tokens.+ expect(estimateTokens(["abcd"])).toBe(1);+ expect(estimateTokens(["abcdefgh"])).toBe(2);+ expect(estimateTokens(["abcde"])).toBe(2); // ceil(5/4)+ expect(estimateTokens([])).toBe(0);+ expect(estimateTokens(["", ""])).toBe(0);+});++test("estimateCostUsd scales tokens by the per-million rate", () => {+ // 1,000,000 tokens * rate == rate USD.+ const rate = pricePerMillionTokens("text-embedding-3-small");+ expect(estimateCostUsd(1_000_000, "text-embedding-3-small")).toBeCloseTo(rate, 9);+ expect(estimateCostUsd(500_000, "text-embedding-3-small")).toBeCloseTo(rate / 2, 9);+});++test("estimateCostUsd is 0 for local/unknown models regardless of token count", () => {+ expect(estimateCostUsd(10_000_000, LOCAL_EMBEDDING_MODEL)).toBe(0);+ expect(estimateCostUsd(10_000_000, "unknown")).toBe(0);+});++test("isStaleSignature is true only when active and stored differ", () => {+ expect(isStaleSignature("local:m:256", "local:m:256")).toBe(false);+ expect(isStaleSignature("local:m:256", "local:m:512")).toBe(true);+ expect(isStaleSignature("openai-compat:a:1536", "local:m:256")).toBe(true);+});diff --git a/tests/core/search/embeddings.test.ts b/tests/core/search/embeddings.test.tsindex d18770db..fdfb4b15 100644--- a/tests/core/search/embeddings.test.ts+++ b/tests/core/search/embeddings.test.ts@@ -20,6 +20,7 @@ function cfg(overrides: Partial<ResolvedEmbeddingConfig> = {}): ResolvedEmbeddintimeoutMs: 5_000,concurrency: 2,batchSize: 32,+ costGateUsd: 0,...overrides,});}@@ -214,6 +215,7 @@ test("makeProvider returns NullProvider when semantic is disabled", () => {timeoutMs: 10_000,concurrency: 1,batchSize: 1,+ costGateUsd: 0,});expect(p).toBeInstanceOf(NullProvider);});@@ -230,6 +232,7 @@ test("makeProvider throws when key is missing under openai-compat", () => {timeoutMs: 10_000,concurrency: 1,batchSize: 1,+ costGateUsd: 0,}),).toThrow(/embedding_api_key/);});diff --git a/tests/core/search/fusion.test.ts b/tests/core/search/fusion.test.tsnew file mode 100644index 00000000..cde7f819--- /dev/null+++ b/tests/core/search/fusion.test.ts@@ -0,0 +1,65 @@+import { test, expect } from "bun:test";++import { rrfFuse, isFusionMode, DEFAULT_RRF_K } from "../../../src/core/search/fusion.ts";++test("DEFAULT_RRF_K is the canonical 60", () => {+ expect(DEFAULT_RRF_K).toBe(60);+});++test("isFusionMode accepts the two modes and rejects others", () => {+ expect(isFusionMode("linear")).toBe(true);+ expect(isFusionMode("rrf")).toBe(true);+ expect(isFusionMode("weighted")).toBe(false);+ expect(isFusionMode("")).toBe(false);+});++test("empty lanes produce an empty fusion map", () => {+ expect(rrfFuse({ keywordRankedChunkIds: [], semanticRankedChunkIds: [], k: 60 }).size).toBe(0);+});++test("a chunk ranked top in both lanes scores highest", () => {+ // chunk 1: rank 1 in both lanes. chunk 2: rank 2 keyword, absent semantic.+ // chunk 3: absent keyword, rank 1 semantic... order them.+ const fused = rrfFuse({+ keywordRankedChunkIds: [1, 2],+ semanticRankedChunkIds: [1, 3],+ k: 60,+ });+ expect(fused.get(1)!).toBeGreaterThan(fused.get(2)!);+ expect(fused.get(1)!).toBeGreaterThan(fused.get(3)!);+});++test("normalised scores span [0,1] with the best at 1 and worst at 0", () => {+ const fused = rrfFuse({+ keywordRankedChunkIds: [1, 2, 3],+ semanticRankedChunkIds: [1, 2, 3],+ k: 60,+ });+ const values = [...fused.values()];+ expect(Math.max(...values)).toBeCloseTo(1, 9);+ expect(Math.min(...values)).toBeCloseTo(0, 9);+});++test("reciprocal-rank ordering matches rank sums", () => {+ // keyword: A(1) B(2) C(3); semantic: C(1) B(2) A(3).+ // RRF sums: A = 1/61 + 1/63; B = 1/62 + 1/62; C = 1/63 + 1/61.+ // A and C tie; B sits between by raw sum (2/62 vs 1/61+1/63).+ const fused = rrfFuse({+ keywordRankedChunkIds: [10, 20, 30],+ semanticRankedChunkIds: [30, 20, 10],+ k: 60,+ });+ expect(fused.get(10)!).toBeCloseTo(fused.get(30)!, 9);+ // A/C raw = 1/61 + 1/63 β 0.032277; B raw = 2/62 β 0.032258 -> B is lowest.+ expect(fused.get(20)!).toBeLessThan(fused.get(10)!);+});++test("a single populated lane still ranks by position", () => {+ const fused = rrfFuse({+ keywordRankedChunkIds: [5, 6, 7],+ semanticRankedChunkIds: [],+ k: 60,+ });+ expect(fused.get(5)!).toBeGreaterThan(fused.get(6)!);+ expect(fused.get(6)!).toBeGreaterThan(fused.get(7)!);+});diff --git a/tests/core/search/indexer.embeddings.test.ts b/tests/core/search/indexer.embeddings.test.tsindex b9a9db11..dc3e4820 100644--- a/tests/core/search/indexer.embeddings.test.ts+++ b/tests/core/search/indexer.embeddings.test.ts@@ -39,6 +39,7 @@ function semanticConfig(model = "fake-model", dim = 4) {timeoutMs: 5_000,concurrency: 2,batchSize: 8,+ costGateUsd: 0,},});}@@ -102,6 +103,7 @@ test("--embeddings without a key throws EMBEDDING_KEY_MISSING", async () => {timeoutMs: 5_000,concurrency: 1,batchSize: 8,+ costGateUsd: 0,},});let err: SearchError | null = null;diff --git a/tests/core/search/local-embedder-rrf.integration.test.ts b/tests/core/search/local-embedder-rrf.integration.test.tsnew file mode 100644index 00000000..cdca2f7a--- /dev/null+++ b/tests/core/search/local-embedder-rrf.integration.test.ts@@ -0,0 +1,82 @@+/**+ * Integration slice for the Embedding Provider Suite: index a vault with+ * the offline LOCAL embedder (no network, no key) and search it under+ * both fusion modes. Exercises the whole pipeline - config resolution,+ * indexer cost gate (free for local), vec storage, semantic recall, and+ * RRF fusion - end to end.+ */++import { test, expect, beforeEach, afterEach } from "bun:test";++import { indexVault } from "../../../src/core/search/indexer.ts";+import { indexStatus } from "../../../src/core/search/indexer.ts";+import { search } from "../../../src/core/search/search.ts";+import { LOCAL_EMBEDDING_MODEL } from "../../../src/core/search/embeddings/signature.ts";+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";++let vault: string;+let dbPath: string;+let cleanup: () => void;++beforeEach(() => {+ const v = createTempVault("local-rrf");+ vault = v.vault;+ dbPath = v.dbPath;+ cleanup = v.cleanup;+});++afterEach(() => {+ cleanup();+});++function localConfig(fusionMode: "linear" | "rrf" = "linear") {+ return makeConfig({+ vault,+ dbPath,+ fusionMode,+ semantic: {+ enabled: true,+ provider: "local",+ baseUrl: null,+ model: null,+ apiKey: null,+ dimension: 128,+ timeoutMs: 5_000,+ concurrency: 2,+ batchSize: 8,+ costGateUsd: 0,+ },+ });+}++test("local embedder indexes with no network and embeds every chunk", async () => {+ if (!sqliteVecLoadable()) return;+ writeMd(vault, "a.md", "# Machine learning\n\nNeural networks and gradient descent.");+ writeMd(vault, "b.md", "# Gardening\n\nTomatoes and compost in spring.");+ const cfg = localConfig();++ const stats = await indexVault(cfg, { embeddings: true });+ expect(stats.embeddingsComputed).toBeGreaterThanOrEqual(2);++ const status = await indexStatus(cfg);+ expect(status.embeddingModel).toBe(LOCAL_EMBEDDING_MODEL);+ expect(status.embeddingDimension).toBe(128);+ expect(status.embeddingSignature).toBe(`local:${LOCAL_EMBEDDING_MODEL}:128`);+ // The local provider is free, so there is never a refresh cost.+ expect(status.estimatedRefreshCostUsd).toBe(0);+});++test("local embedder produces semantic hits and rrf surfaces its reason", async () => {+ if (!sqliteVecLoadable()) return;+ writeMd(vault, "ml.md", "# Machine learning\n\nNeural networks learn from data.");+ writeMd(vault, "garden.md", "# Gardening\n\nTomatoes need water and sun.");+ const cfg = localConfig("rrf");+ await indexVault(cfg, { embeddings: true });++ const outcome = await search(cfg, { query: "neural networks", limit: 5, semantic: true });+ expect(outcome.results.length).toBeGreaterThan(0);+ // At least one result fused through RRF carries the rrf reason.+ const anyRrf = outcome.results.some((r) => r.reasons.some((x) => x.startsWith("rrf:")));+ expect(anyRrf).toBe(true);+});diff --git a/tests/core/search/ranker-rrf.test.ts b/tests/core/search/ranker-rrf.test.tsnew file mode 100644index 00000000..706cf57c--- /dev/null+++ b/tests/core/search/ranker-rrf.test.ts@@ -0,0 +1,132 @@+import { test, expect, beforeEach, afterEach } from "bun:test";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";++import { rankResults } from "../../../src/core/search/ranker.ts";+import { resolveSearchConfig, SearchError } from "../../../src/core/search/index.ts";+import type { KeywordHit, SemanticHit, HydratedChunk } from "../../../src/core/search/store.ts";++const NOW = 1_750_000_000_000;+const OLD = NOW / 1000 - 365 * 24 * 3600; // no recency boost++function hyd(chunkId: number, docId: number): HydratedChunk {+ return Object.freeze({+ chunkId,+ documentId: docId,+ path: `doc${docId}.md`,+ title: `Doc ${docId}`,+ content: `chunk ${chunkId}`,+ startLine: 1,+ endLine: 1,+ mtime: OLD,+ });+}++// chunk 1: top of the keyword lane only. chunk 2: present in BOTH lanes+// (keyword rank 2, semantic rank 1). chunk 3: semantic lane only.+const KW: KeywordHit[] = [+ { chunkId: 1, documentId: 10, bm25: -10 }, // best keyword, single-lane+ { chunkId: 2, documentId: 11, bm25: -5 },+];+const SEM: SemanticHit[] = [+ { chunkId: 2, documentId: 11, distance: 0.1 }, // best semantic+ { chunkId: 3, documentId: 12, distance: 0.2 },+];+const HYDRATED = new Map<number, HydratedChunk>([+ [1, hyd(1, 10)],+ [2, hyd(2, 11)],+ [3, hyd(3, 12)],+]);++function inputs() {+ return {+ keyword: KW,+ semantic: SEM,+ hydrated: HYDRATED,+ inboundLinkSources: new Map(),+ tagsByDoc: new Map(),+ };+}++test("rrf mode promotes the chunk present in BOTH lanes above a single-lane top hit", () => {+ // chunk 2 (keyword rank 2 + semantic rank 1) fuses above chunk 1 (top+ // of the keyword lane but absent from the semantic lane), because RRF+ // rewards cross-lane presence: 1/62 + 1/61 > 1/61.+ const ranked = rankResults(inputs(), {+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ limit: 10,+ nowMs: NOW,+ fusionMode: "rrf",+ rrfK: 60,+ });+ expect(ranked[0]!.chunkId).toBe(2);+ expect(ranked[0]!.reasons.some((r) => r.startsWith("rrf:"))).toBe(true);+ // The single-lane chunk 3 (semantic only, rank 2) fuses lowest.+ expect(ranked[ranked.length - 1]!.chunkId).toBe(3);+});++test("linear mode (default) does NOT add an rrf reason", () => {+ const ranked = rankResults(inputs(), {+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ limit: 10,+ nowMs: NOW,+ });+ for (const r of ranked) {+ expect(r.reasons.some((x) => x.startsWith("rrf:"))).toBe(false);+ }+});++test("linear mode ranking is unchanged when fusionMode is omitted vs explicit linear", () => {+ const base = inputs();+ const omitted = rankResults(base, {+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ limit: 10,+ nowMs: NOW,+ });+ const explicit = rankResults(base, {+ keywordWeight: 0.6,+ semanticWeight: 0.4,+ limit: 10,+ nowMs: NOW,+ fusionMode: "linear",+ });+ expect(omitted.map((r) => r.chunkId)).toEqual(explicit.map((r) => r.chunkId));+ expect(omitted.map((r) => r.score)).toEqual(explicit.map((r) => r.score));+});++// ββ config parsing βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ++let tmp: string;+let config: string;++beforeEach(() => {+ tmp = mkdtempSync(join(tmpdir(), "o2b-rrf-cfg-"));+ config = join(tmp, "config.yaml");+});++afterEach(() => {+ rmSync(tmp, { recursive: true, force: true });+});++test("search_fusion_mode defaults to linear with rrf_k 60", () => {+ writeFileSync(config, `vault: "${tmp}"\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath: config });+ expect(cfg.fusionMode).toBe("linear");+ expect(cfg.rrfK).toBe(60);+});++test("search_fusion_mode parses rrf", () => {+ writeFileSync(config, `vault: "${tmp}"\nsearch_fusion_mode: "rrf"\nsearch_rrf_k: "40"\n`);+ const cfg = resolveSearchConfig({ vault: tmp, configPath: config });+ expect(cfg.fusionMode).toBe("rrf");+ expect(cfg.rrfK).toBe(40);+});++test("an invalid search_fusion_mode is rejected", () => {+ writeFileSync(config, `vault: "${tmp}"\nsearch_fusion_mode: "weighted"\n`);+ expect(() => resolveSearchConfig({ vault: tmp, configPath: config })).toThrow(SearchError);+});diff --git a/tests/core/search/search.test.ts b/tests/core/search/search.test.tsindex 8b1148a5..ff56b4d3 100644--- a/tests/core/search/search.test.ts+++ b/tests/core/search/search.test.ts@@ -41,6 +41,7 @@ function semanticConfig() {timeoutMs: 5_000,concurrency: 2,batchSize: 8,+ costGateUsd: 0,},});}@@ -339,6 +340,7 @@ test("explicit --semantic without vec extension throws (configured-down case)",timeoutMs: 5_000,concurrency: 1,batchSize: 8,+ costGateUsd: 0,},});await indexVault(cfg);diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.tsindex 9d8a22ca..5cd98d32 100644--- a/tests/core/search/store.test.ts+++ b/tests/core/search/store.test.ts@@ -25,6 +25,7 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCotimeoutMs: 10_000,concurrency: 4,batchSize: 32,+ costGateUsd: 0,});return Object.freeze({vault: tmp,@@ -34,6 +35,8 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCochunkOverlap: 100,keywordWeight: 0.6,semanticWeight: 0.4,+ fusionMode: "linear" as const,+ rrfK: 60,semantic,recall: Object.freeze({mmrLambda: 0.7,diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.tsindex 1ca2825b..57c3de78 100644--- a/tests/core/search/store.vec.test.ts+++ b/tests/core/search/store.vec.test.ts@@ -28,6 +28,7 @@ function semanticConfig(timeoutMs: 10_000,concurrency: 4,batchSize: 32,+ costGateUsd: 0,});return Object.freeze({vault: tmp,@@ -37,6 +38,8 @@ function semanticConfig(chunkOverlap: 100,keywordWeight: 0.6,semanticWeight: 0.4,+ fusionMode: "linear" as const,+ rrfK: 60,semantic,recall: Object.freeze({mmrLambda: 0.7,diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.tsindex 07624870..2a12f094 100644--- a/tests/helpers/search-fixtures.ts+++ b/tests/helpers/search-fixtures.ts@@ -65,6 +65,10 @@ export function makeConfig(opts: {relationPolarityEnabled?: boolean;/** Learned recall weights from feedback; defaults to false (opt-in). */learnedWeightsEnabled?: boolean;+ /** Rank-fusion mode; defaults to "linear". */+ fusionMode?: "linear" | "rrf";+ /** RRF damping constant; defaults to 60. */+ rrfK?: number;}): ResolvedSearchConfig {const baseSemantic: ResolvedEmbeddingConfig = Object.freeze({enabled: false,@@ -76,6 +80,7 @@ export function makeConfig(opts: {timeoutMs: 10_000,concurrency: 4,batchSize: 32,+ costGateUsd: 0,});const semantic: ResolvedEmbeddingConfig = Object.freeze({...baseSemantic,@@ -93,6 +98,8 @@ export function makeConfig(opts: {chunkOverlap: 100,keywordWeight: 0.6,semanticWeight: 0.4,+ fusionMode: opts.fusionMode ?? "linear",+ rrfK: opts.rrfK ?? 60,semantic,recall: Object.freeze({mmrLambda: opts.mmrLambda ?? 0.7,