Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

feat: Embedding Provider Suite - local embedder, provider registry, cost gate, RRF fusion (#67)

Sol Aitkeneda202df3561
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index 1c89c52b..a3e2bcc9 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "0.35.0",
+ "version": "0.36.0",
"description": "Plugin-first second brain package for AI agents and humans.",
"author": {
"name": "Open Second Brain contributors"
diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json
index 4acf6587..8f155531 100644
--- a/.codex-plugin/plugin.json
+++ b/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "0.35.0",
+ "version": "0.36.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"skills": "./skills",
"hooks": "./hooks/hooks.json",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index eccf4dfe..c35382cb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,55 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [0.36.0] - 2026-06-03
+
+Embedding Provider Suite: four changes to the semantic layer that make
+recall cheaper, more portable, and provider-flexible - an offline
+embedder that needs no cloud, a runtime registry for embedding
+endpoints, spend visibility with a cost gate, and an alternative
+rank-fusion mode - all behind one shared signature kernel and all
+off-by-default where they could change behaviour.
+
+### Added
+
+- **Offline local embedder.** A new `local` embedding provider produces
+ deterministic vectors with no cloud call, no API key, and no model
+ download: token unigrams and character trigrams are feature-hashed
+ (FNV-1a, signed buckets) into a configurable fixed dimension (default
+ 256) and unit-normalised, so the existing cosine ranker math is
+ unchanged. Set `embedding_provider: local` (and
+ `search_semantic_enabled: true`) for a privacy-first, no-cloud recall
+ path; both indexing and query embedding run without a key. It is a
+ lexical baseline - `openai-compat` remains the recommended provider for
+ semantic depth.
+- **CLI provider registry.** Register OpenAI-compatible embedding
+ endpoints at runtime with `o2b search provider add | list | show |
+ remove`, persisted to `Brain/search/embedding-providers.json` (base
+ URL, default model, and the NAME of the environment variable holding
+ the key - never the key itself, so the file is safe to sync). A
+ registered name resolves to `openai-compat` config during config
+ resolution, AFTER the built-ins, so it never shadows an explicitly
+ configured key; explicit config or environment always wins over a
+ profile's fields. The resolved provider union stays closed.
+- **Embedding cost gate and signature reporting.** A best-effort
+ per-model pricing table plus a chars/4 token estimate gate large
+ embedding runs: when `embedding_cost_gate_usd` is positive (default 0 =
+ disabled) and the estimated spend exceeds it, the run is refused with
+ the estimate, and `o2b search index --force-cost` overrides. The local
+ provider and any unlisted model price at 0 and never block.
+ `o2b search status` now reports the active
+ `<provider>:<model>:<dimension>` embedding signature and a best-effort
+ refresh-cost estimate.
+- **Reciprocal Rank Fusion mode.** `search_fusion_mode: rrf` (default
+ `linear`) fuses the sparse (BM25) and dense (cosine) lanes by rank
+ position - `1 / (search_rrf_k + rank)`, `search_rrf_k` default 60 -
+ rather than by weighted score magnitude, rewarding cross-lane presence
+ and staying robust to differing score scales. The fused relevance is
+ min-max-normalised so the link, recency, entity, tier, and
+ session-focus boosts compose unchanged, and results carry an `rrf:`
+ recall reason. The default `linear` mode keeps ranking bit-identical;
+ storage stays SQLite + sqlite-vec + FTS5 (no cloud vector database).
+
## [0.35.0] - 2026-06-02
Memory Integrity Suite: four changes that make the memory itself
@@ -4064,6 +4113,7 @@ plugin config (vault field)`, and exits with a clear
- Sandbox vault and plugin manifest fixtures for tests.
- GitHub release workflow for tag-based and manually dispatched releases.
+[0.36.0]: https://github.com/itechmeat/open-second-brain/compare/v0.35.0...v0.36.0
[0.35.0]: https://github.com/itechmeat/open-second-brain/compare/v0.34.0...v0.35.0
[0.34.0]: https://github.com/itechmeat/open-second-brain/compare/v0.33.0...v0.34.0
[0.33.0]: https://github.com/itechmeat/open-second-brain/compare/v0.32.1...v0.33.0
diff --git a/README.md b/README.md
index 006792b4..188091d9 100644
--- a/README.md
+++ b/README.md
@@ -114,7 +114,7 @@ The capabilities you actually feel day to day:
These are the headline capabilities. The full surface also includes: importing Claude Code memory directories, daily logging-discipline cron with a complexity-to-thinking warning, cross-project pointers for shared vaults, transient pinned session context through `brain_pinned_context` / `Brain/pinned.md`, source-agent query and comparison (`brain_agent_query`, `brain_agent_diff`, `o2b brain agent-query`, `o2b brain agent-diff`), the codegraph partner skill, vault hygiene lints, per-MOC coverage audit, concept synthesis, an operator dashboard, configurable Markdown-link presentation output, MCP output-contract checks for core agent-facing tools, private-region stripping with `<private>...</private>`, the v0.12.0 Brain Integrity Suite (content-hash drift detection, durable workrun checkpoints, destructive-from-confirmed retire gate, and the `brain_review_candidates` MCP tool that previews what the next dream pass would do without writing anything), the v0.13.0 Hybrid Search and Recall Quality suite (explainable recall, MMR diversity, link-graph traversal, entity-boosted retrieval, header-anchored chunking), the v0.14.0 Semantic Brain Health and Self-Maintenance suite (cross-preference contradiction detection, concept-gap and stale-claim surfacing, per-preference edit-history, a clean/watch/investigate reconciliation verdict via `brain_health`, and dependency-ordered `doctor --remediate`), the v0.17.0 Brain Lifecycle Review Suite (`intent-review`, `retention`, `monthly`, lifecycle schemas, and discipline complexity ratio), and the v0.18.0 MCP context economy (a per-tool preview budget that returns a bounded inline preview and parks the full payload in a vault-local artifact the agent fetches on demand with `brain_artifact_get`, plus a `recall_hint` one-liner on search results), and the v0.19.0 typed graph semantics (typed frontmatter relationships - `related` / `extends` / `contradicts` / `superseded_by` - that surface inline on search results, a `visibility:` scoping axis for `brain_search`, and a `brain_mcp_landscape` view of the Model Context Protocol servers configured across the vault), and the v0.20.0 Recall and Ranking Quality suite (a configurable Weibull recency-decay curve, structural query-intent classification that re-weights ranking by query shape, language-agnostic synonym expansion via local co-occurrence, per-memory and total recall character budgets for `brain_context_pack`, a persistent query cache that self-invalidates on a corpus-generation change, and a `brain_pre_compress_pack` tool that returns a budgeted top-preferences addendum for a host runtime to inject before context compression), and the v0.21.0 Brain Lifecycle Suite (a per-preference mutation audit trail keyed by preference id - create / promote / update / retire / merge with revision + content-hash before/after, via `o2b brain audit` and `brain_audit`; a multi-phase `dream` pipeline that names its close / reconcile / synthesize / heal / log seams and reports a structured per-phase summary without rewriting the proven internals; deterministic reconcile-phase domain classification that buckets contradictions into claims / entity / decisions / source-freshness, auto-resolves only the decisive source-freshness case, and surfaces the rest as open questions instead of forcing a merge; a budgeted `o2b brain morning-brief` / `brain_morning_brief` session-start summary of top preferences, recent open questions, and recent notes; language-agnostic ISO-8601 temporal extraction that fills preference validity windows from signal text; and opt-in heal-phase vault enrichment that links exact title mentions across user pages), and the v0.22.0 Vault portability + session economy suite (a deterministic lossless session codec - `expand(compress(x)) === x` for all input, opt-in on the signal store and exposed as `o2b brain codec`; an `o2b brain sources` / `brain_sources` read-only dashboard of signals by agent and source type; vault-map `{{role}}` tokens resolved via an optional `Brain/_vault-map.yaml` and wired into scan-inline read paths and graph import; named multi-vault profiles with `o2b vault profile` + a `brain_switch_vault` tool, activated by a pointer in config; and `o2b brain graph-export` / `graph-import` for moving a vault's knowledge graph between layouts with skip/overwrite/merge conflict modes). Browse [`docs/cli-reference.md`](docs/cli-reference.md) for every verb and [`docs/how-it-works.md`](docs/how-it-works.md) for the mental model.
-The v0.26.0 surface adds audited schema mutation/admin tools, real-time lifecycle capture hooks for prompt markers and `brain_feedback` tool calls, CJK-aware FTS recall, and `brain_watchdog` / `o2b brain watchdog` recovery probes. The v0.27.0 surface adds recall-control and trust tools: structured search query documents, session focus, MCP `brain_recall_gate`, polarity-aware context lanes, and opt-in evidence packs for CLI/MCP search. The v0.28.0 surface adds a deterministic prompt-injection guard for automatically surfaced Brain context, agent-blind `$secret:NAME` references, and preview foundations for source-scoped forget, privacy-scanned knowledge packs, and oversized payload externalization. The v0.29.0 surface adds opt-in context receipts and recall telemetry, read-only context budget presets, pre-compaction extraction, and a continuity-backed session recall DAG. The v0.33.0 surface adds the Recall Trust Suite: relation-aware recall polarity for typed memory edges, an explicit retrieval feedback loop with bounded learned weights (`o2b search feedback` / `weights`, MCP `brain_recall_feedback`), time-aware recall (`--since` / `--until` with natural-language ranges), verified multi-record recall (per-token union records, IDF-weighted coverage, rare-term abstention), and a search-completeness verdict with a false-absence guard. The v0.34.0 surface adds the Token Diet: a budgeted SessionStart injection of `Brain/active.md` (default 8,000 chars, `active.inject_budget_chars`) with the post-compaction path moved to the supported SessionStart `compact` matcher, a once-per-session post-write reminder cadence, frontmatter escape-amplification fixed at the parser with a one-shot `brain upgrade --apply` repair for already-corrupted preference files, three consolidated read tools (`brain_brief`, `brain_analytics`, `schema_inspect`) with 18 hidden deprecated aliases, registry description caps with a contract guard, preview budgets by default, and a `scripts/measure-token-surface.ts` report. The v0.35.0 surface adds the Memory Integrity Suite: a canonical entity registry under `Brain/entities/` (one Markdown entity per `(category, name)` with alias resolution, archive/restore, typed relations, `o2b brain entity` CLI verbs, a read-only `brain_entity` MCP tool, doctor duplicate/relation lints, and alias-aware search boost with `entity_canonical` explanations), per-device Brain log shards (`Brain/log/<date>.<deviceId>.jsonl` + `.md` keyed by a device-local id, merged reads through `readLogDay`/`listLogDates`, no more Syncthing write conflicts, legacy days untouched), capture boundaries (a `sessions:` block with ignored/stateless session globs and message suppression regexes, applied before any extraction at both live-hook and import seams, counted but never stored raw; keep suppression regexes simple - they run on every message, and a catastrophic-backtracking pattern slows capture), and deterministic regex fact extraction (seven precision-first families over user turns, `source_type: extracted` signals with family-scoped dedup and canonical entity anchors).
+The v0.26.0 surface adds audited schema mutation/admin tools, real-time lifecycle capture hooks for prompt markers and `brain_feedback` tool calls, CJK-aware FTS recall, and `brain_watchdog` / `o2b brain watchdog` recovery probes. The v0.27.0 surface adds recall-control and trust tools: structured search query documents, session focus, MCP `brain_recall_gate`, polarity-aware context lanes, and opt-in evidence packs for CLI/MCP search. The v0.28.0 surface adds a deterministic prompt-injection guard for automatically surfaced Brain context, agent-blind `$secret:NAME` references, and preview foundations for source-scoped forget, privacy-scanned knowledge packs, and oversized payload externalization. The v0.29.0 surface adds opt-in context receipts and recall telemetry, read-only context budget presets, pre-compaction extraction, and a continuity-backed session recall DAG. The v0.33.0 surface adds the Recall Trust Suite: relation-aware recall polarity for typed memory edges, an explicit retrieval feedback loop with bounded learned weights (`o2b search feedback` / `weights`, MCP `brain_recall_feedback`), time-aware recall (`--since` / `--until` with natural-language ranges), verified multi-record recall (per-token union records, IDF-weighted coverage, rare-term abstention), and a search-completeness verdict with a false-absence guard. The v0.34.0 surface adds the Token Diet: a budgeted SessionStart injection of `Brain/active.md` (default 8,000 chars, `active.inject_budget_chars`) with the post-compaction path moved to the supported SessionStart `compact` matcher, a once-per-session post-write reminder cadence, frontmatter escape-amplification fixed at the parser with a one-shot `brain upgrade --apply` repair for already-corrupted preference files, three consolidated read tools (`brain_brief`, `brain_analytics`, `schema_inspect`) with 18 hidden deprecated aliases, registry description caps with a contract guard, preview budgets by default, and a `scripts/measure-token-surface.ts` report. The v0.35.0 surface adds the Memory Integrity Suite: a canonical entity registry under `Brain/entities/` (one Markdown entity per `(category, name)` with alias resolution, archive/restore, typed relations, `o2b brain entity` CLI verbs, a read-only `brain_entity` MCP tool, doctor duplicate/relation lints, and alias-aware search boost with `entity_canonical` explanations), per-device Brain log shards (`Brain/log/<date>.<deviceId>.jsonl` + `.md` keyed by a device-local id, merged reads through `readLogDay`/`listLogDates`, no more Syncthing write conflicts, legacy days untouched), capture boundaries (a `sessions:` block with ignored/stateless session globs and message suppression regexes, applied before any extraction at both live-hook and import seams, counted but never stored raw; keep suppression regexes simple - they run on every message, and a catastrophic-backtracking pattern slows capture), and deterministic regex fact extraction (seven precision-first families over user turns, `source_type: extracted` signals with family-scoped dedup and canonical entity anchors). The v0.36.0 surface adds the Embedding Provider Suite: an offline `local` embedding provider (deterministic feature hashing, no cloud, no key, no model download - set `embedding_provider: local` for a privacy-first recall path), a CLI provider registry (`o2b search provider add | list | show | remove`, persisted to `Brain/search/embedding-providers.json` storing only environment-variable names, resolved after the built-ins so a configured key is never shadowed), an embedding cost gate (`embedding_cost_gate_usd`, default 0 = off; refuses an over-budget run unless `o2b search index --force-cost`) with the active `<provider>:<model>:<dimension>` signature and a refresh-cost estimate in `o2b search status`, and an opt-in Reciprocal Rank Fusion mode (`search_fusion_mode: rrf`, `search_rrf_k` default 60) that fuses the keyword and semantic lanes by rank position instead of weighted magnitude while the default `linear` mode stays bit-identical and storage remains SQLite + sqlite-vec + FTS5.
## CLI
diff --git a/docs/brainstorm/embedding-provider-suite/cli-output/claude.md b/docs/brainstorm/embedding-provider-suite/cli-output/claude.md
new file mode 100644
index 00000000..0b4f3d34
--- /dev/null
+++ b/docs/brainstorm/embedding-provider-suite/cli-output/claude.md
@@ -0,0 +1,35 @@
+### Variant 1: Inline per-subsystem extension
+- **Approach**: Each unit lands in the module that already owns its concern, following the existing closed-union/factory conventions verbatim. RRF becomes a `fusionMode` branch inside `ranker.ts`; the `local` provider extends the `makeProvider` switch + a new `local-provider.ts`; the registry is a read/merge helper in `config.ts` wired into `resolveSearchConfig` and new CLI verbs; cost/signature logic extends `indexer.ts` (gate, estimate) and `store.ts` (signature stamp). A single tiny `signature()` helper is the only shared new primitive.
+- **Trade-offs**:
+ - Smallest diff; each change reviewable against its sibling code and existing tests.
+ - Lowest risk to the bit-identical-when-off and closed-union rules β€” they're untouched except where extended.
+ - Fastest path to shipping the p4/p3/p2 units independently.
+ - Cost/signature/pricing knowledge gets sprinkled across indexer + store + config rather than living in one place; mild DRY pressure as Unit 4's signature and Unit 3's registry both touch provider identity.
+ - Registry↔factory coupling is implicit (two call sites must agree on precedence ordering).
+- **Complexity**: small
+- **Risk**: low
+
+### Variant 2: Shared signature/provenance kernel + registry module
+- **Approach**: Introduce two focused new domain modules β€” `embeddings/signature.ts` (canonicalize `{provider, model, dimension}` β†’ stable signature, hold the per-model pricing table, compute stale-detection and spend estimates) and `embeddings/registry.ts` (load/persist provider definitions, merge after built-ins) β€” that Units 2/3/4 all depend on. RRF lands as a small `fusion` strategy split out of `ranker.ts`. The factory, CLI, indexer, and store all consume the kernel rather than re-deriving provider identity.
+- **Trade-offs**:
+ - Directly satisfies the stated "one shared canonicalization/signature kernel rather than per-call-site duplication" constraint.
+ - Cost model, signature, and stale-detection become independently unit-testable in isolation from the store/indexer.
+ - `corpusGeneration()` and per-chunk stamping converge on one signature definition, reducing drift risk in Unit 4.
+ - More upfront structure than a pure inline approach; introduces module boundaries that must be designed before any unit ships, slightly coupling the four deliverables.
+ - Medium blast radius: indexer/store/config all gain a new dependency edge, so the off-path fail-soft behaviour must be re-verified at each seam.
+- **Complexity**: medium
+- **Risk**: medium
+
+### Variant 3: Descriptor/strategy plugin refactor
+- **Approach**: Generalize the whole subsystem around uniform abstractions β€” providers become "descriptors" (built-in or registered) carrying capabilities (dimension, pricing, local/remote, signature), and fusion becomes a pluggable `FusionStrategy` interface (`linear` | `rrf`). The closed string union is replaced by a descriptor registry; all four units become instances of "register a descriptor / register a strategy."
+- **Trade-offs**:
+ - Most extensible: future providers and fusion modes drop in without touching core dispatch.
+ - Cleanest conceptual model β€” registry, cost, signature, and fusion all hang off one extensibility spine.
+ - Replaces the closed union that `resolveSearchConfig`/`parseProvider` depend on, risking the convention that provider selection is a validated closed set, and broadening the surface that must stay bit-identical-when-off.
+ - Largest diff and the most existing tests (explainable `reasons`, parser validation, corpus generation) to re-baseline; over-built relative to four concrete units.
+ - Highest chance of public-API churn on `EmbeddingProvider`/`ResolvedEmbeddingConfig` consumers, triggering the migration constraint.
+- **Complexity**: large
+- **Risk**: high
+
+### Recommended: Variant 2
+**Rationale**: The kernel approach is the only one that directly honours the explicit "one shared canonicalization/signature kernel" constraint while keeping the closed union, fail-soft, and bit-identical-when-off rules intact β€” Units 2, 3, and 4 genuinely share provider-identity and signature logic, so centralizing it prevents the drift that Variant 1 invites and avoids the over-reach of Variant 3's full plugin rewrite. It scales to four units with isolated, testable seams without forcing public-API breakage or re-baselining the ranker's explainability contract.
diff --git a/docs/brainstorm/embedding-provider-suite/cli-output/prompt.md b/docs/brainstorm/embedding-provider-suite/cli-output/prompt.md
new file mode 100644
index 00000000..7c93c2a0
--- /dev/null
+++ b/docs/brainstorm/embedding-provider-suite/cli-output/prompt.md
@@ -0,0 +1,73 @@
+You are brainstorming architectural variants for the following task. Do not write code. Do not write a final design. Only produce variants and a recommendation.
+
+# Task
+
+This is an epic bundling four related kanban tasks that all extend the embedding/semantic-search subsystem of Open Second Brain (a local-first, Obsidian-Markdown agent memory store). The four atomic units, in priority order:
+
+## Unit 1 (p4) β€” Hybrid dense+sparse RRF ranking
+Add Reciprocal Rank Fusion (RRF) as an alternative to the current weighted-linear-sum fusion of BM25 (sparse keyword) and cosine (dense vector) scores. RRF combines the two rankings by rank position (score = sum over lanes of 1/(k + rank)) rather than by min-max-normalised magnitudes. VENDOR-AGNOSTIC ONLY: the upstream inspiration (Tencent TCVDB) is a cloud vector database; we are NOT adding a cloud vector DB. We keep SQLite + sqlite-vec + FTS5 as the storage layer and only add the rank-fusion mode.
+
+## Unit 2 (p3) β€” Offline local embedder provider
+The provider abstraction currently has only `openai-compat` (remote HTTP) and `disabled` (null). Add a `local` provider that produces embeddings with no cloud call, no API key, and no surprise bills β€” aligned with the product's local-first/privacy promise. The default install must stay lightweight; the local model/runtime can be a lazy/optional dependency. Needs predictable dimension, a model signature, and clear diagnostics when the runtime is unavailable.
+
+## Unit 3 (p2) β€” CLI provider registry
+Today an embedding provider is configured via env vars or by editing the YAML config. Add CLI verbs (`provider add/list/show/remove`) that persist provider definitions (base-url, default-model, env-key for the API key) to a registry file, so users add/remove providers at runtime without editing config. Registered providers are auto-detected AFTER built-ins so they never shadow an explicitly configured key.
+
+## Unit 4 (p2) β€” Embedding cost model + stale-detection via signatures
+Add model-aware embedding pricing (price per million tokens), an embedding "signature" (model + dimension, and possibly provider) stamped per chunk, stale-detection that flags chunks whose signature no longer matches the active config, and a cost gate that blocks a large (re-)embedding run when estimated spend exceeds a configurable threshold. The store already clears all embeddings on a model/dimension change and stamps a per-chunk `embedding_hash`; this unit adds cost visibility, a finer-grained signature, and the spend gate.
+
+# Project context
+
+Project: Open Second Brain. Language: TypeScript. Runtime: Bun. Tests: `bun test`. Lint: oxlint (baseline must not grow). Format: oxfmt. Typecheck: tsc.
+
+Recent commits:
+dee5ab7 feat: Memory Integrity Suite - canonical entities, conflict-free log shards, capture boundaries, fact extraction (#66)
+5066e71 feat: Token Diet - budgeted injection, reminder cadence, consolidated MCP surface (#65)
+c494fd8 feat(search): Recall Trust Suite - relation polarity, learned weights, time-aware and verified recall (#64)
+0952dfc feat: become a native Hermes memory provider (#62)
+bd49cd5 feat: recall and ranking quality - Weibull recency, query intent, synonym expansion, recall budgets, query cache, pre-compress pack (#47)
+
+Related files (already studied):
+- src/core/search/embeddings/provider.ts β€” `EmbeddingProvider` interface (name, model, dimension, embed(), ping(), consumeRetryCount?()); `makeProvider(config)` factory dispatching on `config.provider` with lazy require().
+- src/core/search/embeddings/openai-compat.ts β€” remote provider: semaphore-bounded batches, retry/backoff, unit-normalised vectors, abort on first failure.
+- src/core/search/embeddings/null-provider.ts β€” disabled provider.
+- src/core/search/types.ts β€” `ResolvedEmbeddingConfig { enabled, provider: "openai-compat"|"disabled", baseUrl, model, apiKey, dimension, timeoutMs, concurrency, batchSize }`; `ResolvedSearchConfig` with keywordWeight/semanticWeight.
+- src/core/search/index.ts β€” `resolveSearchConfig()`: parses embedding_* keys from YAML + env (`OPEN_SECOND_BRAIN_EMBEDDING_*`); `parseProvider()` only accepts openai-compat|disabled; DEFAULTS table (keywordWeight 0.6, semanticWeight 0.4).
+- src/core/search/indexer.ts β€” `populateEmbeddings()`: drives provider.embed() in super-batches, stamps per-chunk sha256(embedding) as `embedding_hash`, calls `store.ensureEmbeddingModel(model, dim)`, `store.vecUpsert()`; `indexStatus()` reports embeddings/staleEmbeddings/embeddingModel/embeddingDimension; `indexCheck()` pings provider.
+- src/core/search/store.ts β€” `vecUpsert(chunkId, vec, model, dim, embeddingHash)`; `ensureEmbeddingModel()` clears embeddings on model/dim change; `corpusGeneration()` = embeddingModel + dimension + schemaVersion + indexRevision (gates query cache); `findChunksWithoutEmbeddings()`; `staleEmbeddings()`; per-chunk `embeddings(chunk_id, model, dimension, embedding_hash, created_at, updated_at)` table.
+- src/core/search/ranker.ts β€” `rankResults()`: current fusion is weighted linear sum `keywordWeight*kwMul*kwNorm + semanticWeight*semMul*semNorm` where kwNorm is min-max-normalised inverted BM25 and semNorm is cosine-from-L2-distance; explainable `reasons` array (`fts5_bm25`, `semantic_cos`, ...).
+- src/cli/search.ts β€” search CLI verbs (reindex, check, focus, feedback, ...).
+- src/core/config.ts β€” `setConfigValue(key, value, path)`, `discoverConfig()`, device-id pattern (env override -> read -> generate under directory lock); YAML-subset parser does NOT unescape backslashes in quoted scalars.
+
+Conventions:
+- Provider selection is a closed string union resolved in `resolveSearchConfig`; adding a provider means extending the union, the parser, and the `makeProvider` factory.
+- The store is the single source of truth for the embedding model/dimension; `corpusGeneration()` already fingerprints them and self-invalidates the query cache.
+- Every config knob has a YAML key + an `OPEN_SECOND_BRAIN_*` env override, with a default in the DEFAULTS table and range validation.
+- Ranking layers are bounded, deterministic, and have an explicit off switch that keeps untouched vaults bit-identical; explainable `reasons` strings are part of the contract and are asserted in tests.
+- Lazy `require()` keeps the module graph small for users who never enable semantic search.
+- Fail-soft: a bad/absent registry or runtime degrades to current behaviour, never throws into the hot path.
+
+Constraints:
+- Do NOT add a cloud/hosted vector database. Storage stays SQLite + sqlite-vec + FTS5.
+- Do NOT add a heavy mandatory dependency to the default install; any local-embedder model/runtime must be optional/lazy.
+- Do NOT break the existing `openai-compat`/`disabled` providers or change their on-disk index format incompatibly.
+- Keep ranking changes opt-in and bit-identical-when-off (an existing project rule).
+- No new public-API breakage of `EmbeddingProvider` / `ResolvedEmbeddingConfig` consumers without a migration.
+- SOLID / KISS / DRY. One shared canonicalization/signature kernel rather than per-call-site duplication.
+
+# Required output format
+
+Produce exactly 3 distinct architectural variants. For each variant:
+
+### Variant N: <short name>
+- **Approach**: 2-3 sentences describing the variant.
+- **Trade-offs**: bullet list of pros and cons.
+- **Complexity**: small | medium | large
+- **Risk**: low | medium | high
+
+After the three variants, add exactly one recommendation:
+
+### Recommended: Variant N
+**Rationale**: 2-3 sentences explaining why this variant over the others, considering the project context and constraints above.
+
+Output nothing outside of these sections.
diff --git a/docs/brainstorm/embedding-provider-suite/design.md b/docs/brainstorm/embedding-provider-suite/design.md
new file mode 100644
index 00000000..bbe9c72b
--- /dev/null
+++ b/docs/brainstorm/embedding-provider-suite/design.md
@@ -0,0 +1,73 @@
+# Embedding Provider Suite - shared signature kernel, local embedder, provider registry, RRF fusion
+
+**Status:** draft
+**Author:** claude-dev-agent (via feature-release-playbook)
+**Audience:** implementation
+
+## Problem statement
+
+Open Second Brain's semantic layer has one real embedding provider (`openai-compat`, remote HTTP) and one null provider. There is no offline/local-first embedder, no runtime way to register providers without editing config, no cost visibility or spend gate before a large embedding run, and exactly one rank-fusion mode (weighted linear sum of min-max-normalised BM25 and cosine). Four triage tasks (t_9d24c9fd p4, t_d1fbfeee p3, t_f42f1b69 p2, t_4796dff3 p2) all extend this one subsystem and share provider-identity logic.
+
+## Scope
+
+- **Unit 1 - Hybrid RRF fusion (t_9d24c9fd):** add Reciprocal Rank Fusion as an alternative to linear fusion of the sparse (BM25) and dense (cosine) lanes. Opt-in via `search_fusion_mode`; default `linear` keeps ranking bit-identical. No cloud vector DB - storage stays SQLite + sqlite-vec + FTS5.
+- **Unit 2 - Local embedder (t_d1fbfeee):** add a `local` provider that produces deterministic embeddings with no cloud call, no API key, and no model download - a character n-gram feature-hashing embedder. Predictable dimension, stable model signature, clear diagnostics.
+- **Unit 3 - Provider registry (t_f42f1b69):** CLI verbs (`o2b search provider add/list/show/remove`) that persist named provider profiles (base-url, default-model, env-key) to a vault-local registry file. A registered name resolves to an `openai-compat` config at config-resolution time; built-ins are never shadowed.
+- **Unit 4 - Cost model + signatures (t_4796dff3):** one signature kernel that canonicalises `{provider, model, dimension}`, a model-aware price-per-million-tokens table, a token/cost estimator, signature-based stale reporting in status/check, and a cost gate that blocks an embedding run whose estimated spend exceeds a configurable threshold.
+
+## Out of scope
+
+- Cloud/hosted vector databases (Qdrant/Milvus/Weaviate/TCVDB) - explicitly deferred.
+- A heavy bundled transformer model. The local embedder is dependency-free; opting into a larger local model is future work.
+- Per-token exact billing reconciliation against a provider invoice - we estimate, we do not reconcile.
+- Replacing the closed provider union with a descriptor/plugin spine (consultant Variant 3) - rejected as over-built.
+
+## Chosen approach
+
+Consultant **Variant 2 - shared signature/provenance kernel + registry module**. Two focused new domain modules carry the logic Units 2/3/4 share, and a small fusion strategy module carries Unit 1:
+
+- `embeddings/signature.ts` - canonicalise `{provider, model, dimension}` to a stable signature string; hold the per-model pricing table; estimate tokens and cost; decide staleness by comparing an active signature with a stored one. The single source of provider-identity truth, consumed by the factory, indexer, store, and CLI rather than re-derived per call site.
+- `embeddings/registry.ts` - load/persist provider profiles from `Brain/search/embedding-providers.json`; expand a registered name into `openai-compat` config fields during `resolveSearchConfig`, after built-ins so a configured built-in key is never shadowed. Fail-soft: an absent or malformed registry resolves to no extra providers.
+- `embeddings/local-provider.ts` - `LocalProvider implements EmbeddingProvider`: deterministic character n-gram feature hashing into a fixed-dimension unit-normalised vector. No network, no key, no download.
+- `search/fusion.ts` - pure `linearFusion` and `rrfFusion(lanes, k)` helpers split out of `ranker.ts`; the ranker calls the selected strategy and keeps every existing boost (link/tag/recency/entity/tier/session-focus) unchanged.
+
+## Design decisions
+
+- **Closed union stays closed.** The only new resolved provider member is `local`. Registry entries are config-time indirection that resolve to `provider: "openai-compat"`, so `ResolvedEmbeddingConfig.provider` remains `"openai-compat" | "disabled" | "local"` and `parseProvider` keeps validating a closed set. Registry expansion happens before `parseProvider` so a registered name never reaches the validator.
+- **Local embedder = feature hashing, not a model.** A character n-gram (n=3, plus token unigrams) hashing trick gives an offline, deterministic, zero-dependency baseline that works the instant the package installs - matching the upstream "tiny default embedder, no model download" intent. Vectors are unit-normalised so the existing cosine-from-L2 ranker math is unchanged. Quality limits are documented; the design leaves room for a future opt-in transformer.
+- **Signature is the one fingerprint.** `embeddingSignature(provider, model, dimension)` is reused by `corpusGeneration()` (cache invalidation), per-run stale reporting, and the cost estimate, so the four units cannot drift on what "the same embedding configuration" means. The store keeps its existing clear-on-model/dimension-change safety; the signature adds finer-grained reporting and gating on top, it does not replace the clear.
+- **Cost gate is off by default.** `embedding_cost_gate_usd` defaults to `0` (disabled). When positive and a run's estimated spend exceeds it, the indexer throws `EMBEDDING_COST_GATE` carrying the estimate; an explicit override (`--force-cost` / config) proceeds. The local provider prices at `0`, so it is never gated.
+- **RRF is off by default.** `search_fusion_mode` defaults to `linear`; with it the ranker output is byte-for-byte the current behaviour. `rrf` mode fuses lane ranks with `1 / (rrf_k + rank)` (`search_rrf_k` default 60, the canonical RRF constant) and adds an `rrf:` reason; boosts still compose on top.
+- **Every knob follows the house pattern:** a YAML key, an `OPEN_SECOND_BRAIN_*` env override, a `DEFAULTS` entry, and range validation.
+- **Fail-soft everywhere:** a bad registry, an unavailable local runtime, or an unknown pricing entry degrades to current behaviour or a clear diagnostic, never an exception in the hot path.
+
+## File changes
+
+New source:
+- `src/core/search/embeddings/signature.ts`
+- `src/core/search/embeddings/registry.ts`
+- `src/core/search/embeddings/local-provider.ts`
+- `src/core/search/fusion.ts`
+
+Modified source:
+- `src/core/search/embeddings/provider.ts` - `makeProvider` `local` branch.
+- `src/core/search/types.ts` - provider union `+ "local"`; `fusionMode`, `rrfK`, `costGateUsd` on resolved config; signature + estimated-refresh-cost fields on the status snapshot.
+- `src/core/search/index.ts` - `parseProvider` local; registry expansion; fusion/rrf/cost-gate config parsing; `DEFAULTS`.
+- `src/core/search/indexer.ts` - cost gate + estimate in `populateEmbeddings`; signature + refresh-cost in `indexStatus`.
+- `src/core/search/store.ts` - signature helpers / signature-based stale count; `corpusGeneration` reuses the kernel.
+- `src/core/search/ranker.ts` - delegate relevance fusion to `fusion.ts`; rrf path; `rrf` reason.
+- `src/cli/search.ts` - `provider` subverbs; cost-gate flag on reindex.
+- `src/cli/command-manifest.ts`, `src/cli/help-text.ts`, `src/cli/completions.ts` - register and document the new verbs.
+
+New tests (one focused suite per unit + integration):
+- `tests/embeddings.local-provider.test.ts`, `tests/embeddings.signature.test.ts`, `tests/embeddings.registry.test.ts`, `tests/search.fusion-rrf.test.ts`, `tests/indexer.cost-gate.test.ts`, `tests/search-provider.cli.test.ts`, `tests/search-config.embedding-suite.test.ts`, plus an e2e search slice exercising rrf + local provider.
+
+Docs:
+- `README.md`, `CHANGELOG.md`, `docs/cli-reference.md`, `docs/how-it-works.md`.
+
+## Risks and open questions
+
+- **Local embedder quality.** Feature hashing is lexical, not semantic; near-synonyms it cannot match. Mitigation: document it as a no-cloud baseline, keep `openai-compat` the recommended path for semantic depth, and leave the dimension configurable.
+- **RRF re-baselining.** RRF changes scores when enabled, so e2e ranking fixtures must assert `linear` (default) stays bit-identical and add separate `rrf` fixtures. Mitigation: gate behind the off-by-default switch and test both modes.
+- **Pricing table staleness.** Hardcoded per-model prices drift as providers change rates. Mitigation: treat unknown models as price `0` (never falsely gate), document the table as best-effort, and allow a per-model override via config.
+- **Registry file and Syncthing.** The registry stores env-key *names*, never secret values, so it is safe to sync across devices. Confirmed no secret material lands on disk.
diff --git a/docs/brainstorm/embedding-provider-suite/plan.md b/docs/brainstorm/embedding-provider-suite/plan.md
new file mode 100644
index 00000000..9b7eaee5
--- /dev/null
+++ b/docs/brainstorm/embedding-provider-suite/plan.md
@@ -0,0 +1,40 @@
+# Embedding Provider Suite - implementation plan
+
+Implementation order follows the dependency graph: the shared kernel first (Unit 4's signature/pricing is depended on by Units 2 and 3), then the local provider, then the registry, then RRF (independent). Each task is one atomic conventional commit on `feat/embedding-provider-suite`, fmt+lint green before commit.
+
+## Tasks
+
+### Task 1: Signature + cost kernel (Unit 4 foundation)
+- **Files**: `src/core/search/embeddings/signature.ts` (new); `tests/embeddings.signature.test.ts` (new).
+- **What**: `embeddingSignature(provider, model, dimension)` -> canonical `provider:model:dim` string; `EMBEDDING_PRICING` table (per-model USD/Mtok, unknown -> 0); `estimateTokens(texts)` (chars/4 heuristic); `estimateCostUsd(tokens, model)`; `isStaleSignature(active, stored)`.
+- **Acceptance**: unit tests cover canonicalisation (case/whitespace/null model), known + unknown pricing, token + cost estimate, stale comparison. No I/O.
+- **Depends on**: none.
+
+### Task 2: Local embedder provider (Unit 2)
+- **Files**: `src/core/search/embeddings/local-provider.ts` (new); `src/core/search/embeddings/provider.ts` (makeProvider `local` branch); `src/core/search/types.ts` (provider union `+ "local"`); `src/core/search/index.ts` (`parseProvider` accepts `local`, default model/dimension for local); `tests/embeddings.local-provider.test.ts` (new).
+- **What**: `LocalProvider` - deterministic character-trigram + token-unigram feature hashing into a configurable fixed dimension (default 256), unit-normalised; `name="local"`, `model="hashing-ngram-v1"`, `ping()` always ok with the configured dimension; no network, no key.
+- **Acceptance**: identical text -> identical vector; different text -> different vector; vectors unit-normalised; dimension honoured; `makeProvider({provider:"local"})` returns a `LocalProvider`; `parseProvider("local")` ok.
+- **Depends on**: Task 1 (pricing entry for the local model = 0).
+
+### Task 3: Provider registry + CLI (Unit 3)
+- **Files**: `src/core/search/embeddings/registry.ts` (new); `src/core/search/index.ts` (registry expansion before `parseProvider`); `src/cli/search.ts` (`provider add/list/show/remove`); `src/cli/command-manifest.ts`, `src/cli/help-text.ts`, `src/cli/completions.ts`; `tests/embeddings.registry.test.ts` (new); `tests/search-provider.cli.test.ts` (new).
+- **What**: load/persist `Brain/search/embedding-providers.json` (`{name, baseUrl, defaultModel, envKey}[]`); `expandRegisteredProvider(name, registry, env)` -> openai-compat fields; CLI verbs to add/list/show/remove with validation (reserved names `openai-compat`/`disabled`/`local` refused). Fail-soft on absent/malformed file.
+- **Acceptance**: round-trip add->list->show->remove; registered name resolves to openai-compat config with apiKey from `env[envKey]`; built-in names cannot be registered; malformed JSON -> empty registry + warning, never throw.
+- **Depends on**: Task 1 (signature reused in show output).
+
+### Task 4: Cost gate + signature reporting in indexer/store (Unit 4)
+- **Files**: `src/core/search/indexer.ts` (estimate + gate in `populateEmbeddings`; signature + estimatedRefreshCostUsd in `indexStatus`); `src/core/search/store.ts` (signature-based stale helper; `corpusGeneration` reuses kernel); `src/core/search/types.ts` (`costGateUsd` on resolved config; signature + cost fields on status snapshot); `src/core/search/index.ts` (`embedding_cost_gate_usd` parse + default 0); `src/cli/search.ts` (`--force-cost` flag on reindex); `tests/indexer.cost-gate.test.ts` (new).
+- **What**: before embedding a batch set, estimate tokens+cost; if `costGateUsd > 0` and estimate exceeds it and not overridden, throw `EMBEDDING_COST_GATE` with the estimate; surface active signature + refresh cost in status. Local provider (price 0) never gated.
+- **Acceptance**: gate blocks an over-threshold run and the message carries the estimate; `--force-cost` proceeds; gate `0` (default) never blocks; status reports the signature; `corpusGeneration` unchanged output for unchanged config.
+- **Depends on**: Tasks 1, 2.
+
+### Task 5: RRF fusion mode (Unit 1)
+- **Files**: `src/core/search/fusion.ts` (new); `src/core/search/ranker.ts` (delegate relevance fusion; rrf path; `rrf` reason); `src/core/search/types.ts` (`fusionMode`, `rrfK` on resolved config); `src/core/search/index.ts` (`search_fusion_mode` + `search_rrf_k` parse, defaults `linear`/60); `tests/search.fusion-rrf.test.ts` (new).
+- **What**: `linearFusion(kw, sem, kwWeight, semWeight)` (current math) and `rrfFusion(laneRanks, k)`; ranker selects by `fusionMode`; `linear` byte-identical to today; `rrf` fuses lane ranks, adds `rrf:` reason, boosts still compose.
+- **Acceptance**: `linear` mode reproduces current ranking fixtures exactly; `rrf` mode orders by reciprocal-rank sum on a constructed two-lane fixture; invalid mode rejected by config parse.
+- **Depends on**: none (independent; sequenced last to isolate ranking re-baseline).
+
+### Task 6: Integration + docs
+- **Files**: e2e search slice (rrf + local provider) in an existing or new e2e test; `README.md`; `CHANGELOG.md` (v0.36.0); `docs/cli-reference.md`; `docs/how-it-works.md`.
+- **Acceptance**: full `bun test` green; new capabilities documented capability-first; CHANGELOG single v0.36.0 entry bundling all four units.
+- **Depends on**: Tasks 1-5.
diff --git a/docs/brainstorm/embedding-provider-suite/variants.md b/docs/brainstorm/embedding-provider-suite/variants.md
new file mode 100644
index 00000000..baa360f7
--- /dev/null
+++ b/docs/brainstorm/embedding-provider-suite/variants.md
@@ -0,0 +1,32 @@
+# Embedding Provider Suite - brainstorm variants (audit trail)
+
+Primary consultant: `claude -p` (Claude Code). Exit 0, three parseable variants, no fallback needed. Full transcript: `cli-output/claude.md`. Prompt: `cli-output/prompt.md`.
+
+## Variant 1: Inline per-subsystem extension
+
+- **Approach**: Each unit lands in the module that already owns its concern, following existing closed-union/factory conventions verbatim. RRF becomes a `fusionMode` branch inside `ranker.ts`; the `local` provider extends the `makeProvider` switch + a new `local-provider.ts`; the registry is a read/merge helper in `config.ts` wired into `resolveSearchConfig` and new CLI verbs; cost/signature logic extends `indexer.ts` and `store.ts`. A single tiny `signature()` helper is the only shared new primitive.
+- **Trade-offs**: Smallest diff, lowest risk to the bit-identical/closed-union rules, fastest independent shipping; BUT cost/signature/pricing knowledge sprinkled across indexer + store + config (DRY pressure), and registry-to-factory precedence coupling is implicit across two call sites.
+- **Complexity**: small
+- **Risk**: low
+
+## Variant 2: Shared signature/provenance kernel + registry module
+
+- **Approach**: Two focused new domain modules - `embeddings/signature.ts` (canonicalise `{provider, model, dimension}` -> signature, hold pricing table, compute staleness + spend estimates) and `embeddings/registry.ts` (load/persist provider definitions, merge after built-ins) - that Units 2/3/4 all depend on. RRF lands as a small `fusion` strategy split out of `ranker.ts`. Factory, CLI, indexer, store all consume the kernel rather than re-deriving provider identity.
+- **Trade-offs**: Directly satisfies the "one shared canonicalization/signature kernel" constraint; cost/signature/stale independently unit-testable; `corpusGeneration()` and per-chunk stamping converge on one signature, reducing drift. BUT more upfront structure, and a medium blast radius (indexer/store/config gain a dependency edge whose fail-soft behaviour must be re-verified at each seam).
+- **Complexity**: medium
+- **Risk**: medium
+
+## Variant 3: Descriptor/strategy plugin refactor
+
+- **Approach**: Generalise the subsystem around uniform abstractions - providers become capability-carrying "descriptors" (built-in or registered), fusion becomes a pluggable `FusionStrategy` interface. The closed string union is replaced by a descriptor registry; all four units become "register a descriptor / register a strategy."
+- **Trade-offs**: Most extensible and cleanest conceptual model. BUT replaces the closed union that `resolveSearchConfig`/`parseProvider` depend on, broadens the bit-identical-when-off surface, has the largest diff and most tests to re-baseline, and the highest chance of public-API churn triggering the migration constraint - over-built relative to four concrete units.
+- **Complexity**: large
+- **Risk**: high
+
+## Consultant recommendation: Variant 2
+
+> The kernel approach is the only one that directly honours the explicit "one shared canonicalization/signature kernel" constraint while keeping the closed union, fail-soft, and bit-identical-when-off rules intact - Units 2, 3, and 4 genuinely share provider-identity and signature logic, so centralizing it prevents the drift that Variant 1 invites and avoids the over-reach of Variant 3's full plugin rewrite. It scales to four units with isolated, testable seams without forcing public-API breakage or re-baselining the ranker's explainability contract.
+
+## Orchestrator decision: accept Variant 2
+
+Accepted without override. The decisive factor is the project's own DRY constraint: Unit 4's per-chunk signature and the existing `corpusGeneration()` fingerprint must agree on what "the same embedding configuration" means, and Unit 3's registry resolves provider identity that Unit 4 then signs - centralising this in `signature.ts` is the design that prevents three call sites from drifting. Variant 1 would ship slightly faster but re-introduces exactly that drift risk across indexer/store/config; Variant 3's descriptor rewrite would break the closed-union convention and the explainable-`reasons` test contract for no benefit at four-unit scale. Variant 2 keeps the closed union closed (registry entries resolve to `openai-compat`; only `local` is a new member), keeps RRF and the cost gate off by default for bit-identical behaviour, and isolates each unit behind a testable seam.
diff --git a/docs/cli-reference.md b/docs/cli-reference.md
index 4c426e72..cb191dd5 100644
--- a/docs/cli-reference.md
+++ b/docs/cli-reference.md
@@ -181,8 +181,29 @@ o2b search focus status Show the active focus; --json emits { active, focu
o2b search focus clear Clear the persisted focus file next to the search index
o2b search reindex Rebuild the SQLite + FTS5 index from scratch
(required after upgrading to v0.13.0 recall schema or v0.26.0 CJK FTS content)
+ --force-cost bypasses the embedding cost gate for this run (since v0.36.0)
+o2b search index Incrementally update the index; --embeddings computes vectors
+ --force-cost bypasses the embedding cost gate (since v0.36.0)
+o2b search status Index status; since v0.36.0 also reports the active embedding
+ signature (<provider>:<model>:<dimension>) and a refresh-cost estimate
+o2b search provider add NAME Register an OpenAI-compatible embedding endpoint (since v0.36.0)
+ --base-url U --model M --env-key K (K is the env var NAME holding the key);
+ persisted to Brain/search/embedding-providers.json, resolved after built-ins
+o2b search provider list List registered provider profiles (--json for the array)
+o2b search provider show NAME Show one registered profile (--json)
+o2b search provider remove Remove a registered profile by NAME
```
+Embedding providers (since v0.36.0): `embedding_provider` accepts the
+built-in `openai-compat`, the offline `local` feature-hashing embedder
+(no cloud, no key, no model download; `embedding_dimension` default 256),
+`disabled`, or any name registered via `o2b search provider add`.
+`embedding_cost_gate_usd` (default 0 = off) refuses an embedding run whose
+estimated spend exceeds it unless `--force-cost`. `search_fusion_mode`
+(default `linear`) may be set to `rrf` to fuse the keyword and semantic
+lanes by reciprocal rank (`search_rrf_k`, default 60); `linear` keeps
+ranking bit-identical.
+
Typed relations participate in ranking (relation polarity): a page whose
frontmatter declares `superseded_by:` is demoted when it matches and its
successor is boosted or pulled in, `contradicts:` surfaces warning-style
diff --git a/docs/how-it-works.md b/docs/how-it-works.md
index d3d247fb..169f1e84 100644
--- a/docs/how-it-works.md
+++ b/docs/how-it-works.md
@@ -728,8 +728,25 @@ Key behaviours, all driven from `Brain/_brain.yaml`-free `search_*` /
mode for concurrent reads, `proper-lockfile` on the index path for
writer exclusivity (three attempts, 1 s backoff, then
`INDEX_LOCKED`).
+- **Embedding providers (v0.36.0).** `embedding_provider` selects
+ `openai-compat` (remote `/v1/embeddings`), `local` (an offline
+ feature-hashing embedder: token unigrams + character trigrams hashed
+ into a fixed unit-normalised vector, no cloud / key / model download),
+ `disabled`, or a name registered through `o2b search provider add`
+ (which resolves to `openai-compat` config after the built-ins). One
+ `signature.ts` kernel canonicalises `<provider>:<model>:<dimension>`
+ and prices it; `embedding_cost_gate_usd` refuses an over-budget run
+ unless `--force-cost`, and the local / unlisted-model price is 0.
+- **Fusion modes (v0.36.0).** `search_fusion_mode` is `linear` (default,
+ the weighted sum below) or `rrf` - Reciprocal Rank Fusion, which scores
+ each candidate `Ξ£ 1/(search_rrf_k + rank_in_lane)` across the keyword
+ and semantic lanes, min-max-normalised to `[0,1]`. RRF is weightless
+ and replaces only the relevance term; every boost below still applies,
+ and `linear` is bit-identical to pre-v0.36.0 ranking.
- **Ranking.** `final_score = clamp01(keyword_weightΒ·norm_BM25 +
-semantic_weightΒ·cosine + link_boost + recency_boost + entity_boost)`.
+semantic_weightΒ·cosine + link_boost + recency_boost + entity_boost)`
+ (linear mode; in `rrf` mode the first two terms are replaced by the
+ fused reciprocal-rank relevance).
BM25 is min-max-normalised within the candidate set; cosine is
`1 - L2Β² / 2` on unit-normalised vectors; link boost rewards
candidates that other candidates reference via `[[wikilink]]` /
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index 195cee2c..1dca35c4 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -2,7 +2,7 @@
"id": "open-second-brain",
"name": "Open Second Brain",
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",
- "version": "0.35.0",
+ "version": "0.36.0",
"activation": { "onStartup": true },
"skills": ["./skills"],
"contracts": {
diff --git a/package.json b/package.json
index 41d5461d..a1f967ce 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "0.35.0",
+ "version": "0.36.0",
"private": false,
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",
"keywords": [
diff --git a/plugin.yaml b/plugin.yaml
index c1fdcd75..963c149c 100644
--- a/plugin.yaml
+++ b/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "0.35.0"
+version: "0.36.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.json
index a510a7cd..c17f4a6f 100644
--- a/plugins/codex/.codex-plugin/plugin.json
+++ b/plugins/codex/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "0.35.0",
+ "version": "0.36.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yaml
index c1fdcd75..963c149c 100644
--- a/plugins/hermes/plugin.yaml
+++ b/plugins/hermes/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "0.35.0"
+version: "0.36.0"
description: "Open Second Brain - native Hermes memory provider backed by an Obsidian-compatible Markdown vault."
author: "Open Second Brain contributors"
memory_provider: true
diff --git a/pyproject.toml b/pyproject.toml
index d101a99e..a45e5e4a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
# CLI entry points (those moved to `package.json` `bin`).
[project]
name = "open-second-brain"
-version = "0.35.0"
+version = "0.36.0"
description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."
readme = "README.md"
requires-python = ">=3.11"
diff --git a/src/cli/command-manifest.ts b/src/cli/command-manifest.ts
index 952d26df..71bb1104 100644
--- a/src/cli/command-manifest.ts
+++ b/src/cli/command-manifest.ts
@@ -164,6 +164,7 @@ export const CLI_COMMAND_MANIFEST: CliRootManifest = Object.freeze({
command("reindex", "Rebuild the search index"),
command("status", "Print search index status"),
command("check", "Run search pre-flight diagnostics"),
+ command("provider", "Manage embedding provider profiles"),
],
),
command(
diff --git a/src/cli/search.ts b/src/cli/search.ts
index 98fe4f65..22de5e7a 100644
--- a/src/cli/search.ts
+++ b/src/cli/search.ts
@@ -35,6 +35,10 @@ import {
LEARNED_WEIGHT_MIN,
LEARNED_WEIGHT_MAX,
type LearnedWeights,
+ loadProviderRegistry,
+ addProviderProfile,
+ removeProviderProfile,
+ getProviderProfile,
} from "../core/search/index.ts";
import type {
IndexCheckReport,
@@ -57,6 +61,7 @@ const KNOWN_VERBS = new Set([
"focus",
"feedback",
"weights",
+ "provider",
]);
export async function handleSearchSubcommand(argv: ReadonlyArray<string>): Promise<number> {
@@ -87,6 +92,8 @@ export async function handleSearchSubcommand(argv: ReadonlyArray<string>): Promi
return await cmdSearchFeedback(rest);
case "weights":
return await cmdSearchWeights(rest);
+ case "provider":
+ return await cmdSearchProvider(rest);
default:
process.stderr.write(`error: unknown search verb: ${verb}\n`);
return 2;
@@ -293,6 +300,95 @@ function formatLearnedWeights(w: LearnedWeights): string {
);
}
+// ─── provider registry (Embedding Provider Suite) ────────────────────────────
+
+async function cmdSearchProvider(argv: ReadonlyArray<string>): Promise<number> {
+ const action = argv[0];
+ if (!action || !["add", "list", "show", "remove"].includes(action)) {
+ throw new CliError(
+ "usage: o2b search provider <add NAME --base-url U --model M --env-key K | list | show NAME | remove NAME> [--json]",
+ );
+ }
+ const { flags, positional } = parseFlags(argv.slice(1), {
+ vault: { type: "string" },
+ config: { type: "string" },
+ db: { type: "string" },
+ "base-url": { type: "string" },
+ model: { type: "string" },
+ "env-key": { type: "string" },
+ json: { type: "boolean" },
+ });
+ const cfg = resolveConfig(flags);
+ const json = flags["json"] === true;
+ const name = typeof positional[0] === "string" ? positional[0] : undefined;
+
+ if (action === "list") {
+ const registry = loadProviderRegistry(cfg.vault);
+ if (json) {
+ process.stdout.write(JSON.stringify(registry) + "\n");
+ return 0;
+ }
+ if (registry.length === 0) {
+ process.stdout.write("no registered providers\n");
+ return 0;
+ }
+ for (const p of registry) {
+ process.stdout.write(`${p.name} ${p.baseUrl} model=${p.defaultModel} env=${p.envKey}\n`);
+ }
+ return 0;
+ }
+
+ if (action === "show") {
+ if (!name) throw new CliError("usage: o2b search provider show NAME");
+ const profile = getProviderProfile(cfg.vault, name);
+ if (!profile) {
+ process.stderr.write(`error: no registered provider named '${name}'\n`);
+ return 1;
+ }
+ process.stdout.write(
+ json
+ ? JSON.stringify(profile) + "\n"
+ : `${profile.name}\n base-url: ${profile.baseUrl}\n model: ${profile.defaultModel}\n env-key: ${profile.envKey}\n`,
+ );
+ return 0;
+ }
+
+ if (action === "remove") {
+ if (!name) throw new CliError("usage: o2b search provider remove NAME");
+ const { removed } = removeProviderProfile(cfg.vault, name);
+ if (json) {
+ process.stdout.write(JSON.stringify({ removed, name }) + "\n");
+ } else {
+ process.stdout.write(
+ removed ? `removed provider '${name}'\n` : `no such provider '${name}'\n`,
+ );
+ }
+ return removed ? 0 : 1;
+ }
+
+ // add
+ if (!name)
+ throw new CliError("usage: o2b search provider add NAME --base-url U --model M --env-key K");
+ const baseUrl = typeof flags["base-url"] === "string" ? (flags["base-url"] as string) : undefined;
+ const model = typeof flags["model"] === "string" ? (flags["model"] as string) : undefined;
+ const envKey = typeof flags["env-key"] === "string" ? (flags["env-key"] as string) : undefined;
+ if (!baseUrl || !model || !envKey) {
+ throw new CliError(
+ "provider add requires --base-url, --model, and --env-key (the env var NAME holding the API key)",
+ );
+ }
+ const registry = addProviderProfile(cfg.vault, { name, baseUrl, defaultModel: model, envKey });
+ const added = registry.find((p) => p.name === name)!;
+ if (json) {
+ process.stdout.write(JSON.stringify(added) + "\n");
+ } else {
+ process.stdout.write(
+ `added provider '${name}' (set ${envKey} in the environment to supply its key)\n`,
+ );
+ }
+ return 0;
+}
+
// ─── query ────────────────────────────────────────────────────────────────────
async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<number> {
@@ -528,6 +624,7 @@ async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<number> {
db: { type: "string" },
embeddings: { type: "boolean" },
force: { type: "boolean" },
+ "force-cost": { type: "boolean" },
concurrency: { type: "string" },
verbose: { type: "boolean" },
json: { type: "boolean" },
@@ -538,6 +635,7 @@ async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<number> {
const stats = await indexVault(cfg, {
embeddings: flags["embeddings"] === true,
force: flags["force"] === true,
+ forceCost: flags["force-cost"] === true,
onFile: (e) => {
events.push(e);
if (flags["verbose"]) {
@@ -601,6 +699,7 @@ async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<number> {
config: { type: "string" },
db: { type: "string" },
embeddings: { type: "boolean" },
+ "force-cost": { type: "boolean" },
concurrency: { type: "string" },
json: { type: "boolean" },
verbose: { type: "boolean" },
@@ -624,6 +723,7 @@ async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<number> {
const cfg = resolveConfig(flags);
const stats = await reindexVault(cfg, {
embeddings: flags["embeddings"] === true,
+ forceCost: flags["force-cost"] === true,
onFile: flags["verbose"] ? (e) => process.stderr.write(`${e.kind}\t${e.path}\n`) : undefined,
});
if (flags["json"]) {
@@ -664,6 +764,8 @@ function jsonForStatus(s: IndexStatusSnapshot): unknown {
stale_embeddings: s.staleEmbeddings,
embedding_model: s.embeddingModel,
embedding_dimension: s.embeddingDimension,
+ embedding_signature: s.embeddingSignature,
+ estimated_refresh_cost_usd: s.estimatedRefreshCostUsd,
vec_extension: s.vecExtension,
semantic_enabled: s.semanticEnabled,
embedding_key_present: s.embeddingKeyPresent,
@@ -685,6 +787,10 @@ function renderStatusHuman(s: IndexStatusSnapshot): string {
lines.push(`embeddings: ${s.embeddings} (stale: ${s.staleEmbeddings})`);
lines.push(`embedding_model: ${s.embeddingModel ?? "(none)"}`);
lines.push(`embedding_dimension: ${s.embeddingDimension ?? "(none)"}`);
+ lines.push(`embedding_signature: ${s.embeddingSignature ?? "(disabled)"}`);
+ if (s.estimatedRefreshCostUsd > 0) {
+ lines.push(`refresh_cost_est: $${s.estimatedRefreshCostUsd.toFixed(4)}`);
+ }
lines.push(`vec_extension: ${s.vecExtension}`);
lines.push(`semantic_enabled: ${s.semanticEnabled}`);
lines.push(`embedding_key: ${s.embeddingKeyPresent ? "present" : "missing"}`);
diff --git a/src/core/search/embeddings/local-provider.ts b/src/core/search/embeddings/local-provider.ts
new file mode 100644
index 00000000..b12ad4b3
--- /dev/null
+++ b/src/core/search/embeddings/local-provider.ts
@@ -0,0 +1,83 @@
+/**
+ * Offline local embedder (Embedding Provider Suite).
+ *
+ * A dependency-free, deterministic embedder that works the instant the
+ * package installs - no cloud call, no API key, no model download. It
+ * uses the feature-hashing ("hashing trick") technique: token unigrams
+ * and character trigrams are hashed into a fixed-dimension vector with a
+ * signed bucket, then unit-normalised so the existing cosine-from-L2
+ * ranker math is unchanged.
+ *
+ * This is a lexical baseline, not a transformer: it matches shared
+ * tokens and subword shapes, not paraphrase or semantics. It exists so a
+ * privacy-first install has a no-cloud recall path; `openai-compat`
+ * remains the recommended provider for semantic depth. A future opt-in
+ * local transformer can register as a distinct model without touching
+ * this one.
+ */
+
+import type { EmbeddingProvider } from "./provider.ts";
+import { LOCAL_EMBEDDING_MODEL } from "./signature.ts";
+
+/** Default vector width when `embedding_dimension` is not configured. */
+export const LOCAL_DEFAULT_DIMENSION = 256;
+
+/** FNV-1a 32-bit hash, seeded so the two derived hashes diverge. */
+function fnv1a(text: string, seed: number): number {
+ let h = (0x811c9dc5 ^ seed) >>> 0;
+ for (let i = 0; i < text.length; i++) {
+ h ^= text.charCodeAt(i);
+ // h *= 16777619, kept in 32-bit via the shift-add decomposition.
+ h = (h + ((h << 1) + (h << 4) + (h << 7) + (h << 8) + (h << 24))) >>> 0;
+ }
+ return h >>> 0;
+}
+
+/** Tokens (unigrams) plus padded character trigrams, the hashed features. */
+function* features(text: string): Generator<string> {
+ const tokens = text
+ .normalize("NFC")
+ .toLowerCase()
+ .split(/[^\p{L}\p{N}]+/u);
+ for (const tok of tokens) {
+ if (tok.length === 0) continue;
+ yield `w:${tok}`;
+ const padded = `^${tok}$`;
+ for (let i = 0; i + 3 <= padded.length; i++) {
+ yield `t:${padded.slice(i, i + 3)}`;
+ }
+ }
+}
+
+function embedOne(text: string, dimension: number): number[] {
+ const vec: number[] = Array.from({ length: dimension }, () => 0);
+ for (const feature of features(text)) {
+ const bucket = fnv1a(feature, 0) % dimension;
+ const sign = (fnv1a(feature, 0x9e3779b9) & 1) === 0 ? 1 : -1;
+ vec[bucket]! += sign;
+ }
+ let sumSquares = 0;
+ for (const x of vec) sumSquares += x * x;
+ const len = Math.sqrt(sumSquares);
+ if (len === 0) return vec; // empty / featureless text -> zero vector
+ for (let i = 0; i < dimension; i++) vec[i]! /= len;
+ return vec;
+}
+
+export class LocalProvider implements EmbeddingProvider {
+ readonly name = "local";
+ readonly model = LOCAL_EMBEDDING_MODEL;
+ readonly dimension: number;
+
+ constructor(dimension: number = LOCAL_DEFAULT_DIMENSION) {
+ this.dimension = Math.max(1, Math.floor(dimension));
+ }
+
+ async embed(texts: ReadonlyArray<string>): Promise<number[][]> {
+ return texts.map((t) => embedOne(t, this.dimension));
+ }
+
+ async ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }> {
+ return { ok: true, dimension: this.dimension };
+ }
+}
diff --git a/src/core/search/embeddings/provider.ts b/src/core/search/embeddings/provider.ts
index 10e10810..14db8971 100644
--- a/src/core/search/embeddings/provider.ts
+++ b/src/core/search/embeddings/provider.ts
@@ -32,6 +32,12 @@ export function makeProvider(config: ResolvedEmbeddingConfig): EmbeddingProvider
const { NullProvider } = require("./null-provider.ts") as typeof import("./null-provider.ts");
return new NullProvider();
}
+ if (config.provider === "local") {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports
+ const { LocalProvider, LOCAL_DEFAULT_DIMENSION } =
+ require("./local-provider.ts") as typeof import("./local-provider.ts");
+ return new LocalProvider(config.dimension ?? LOCAL_DEFAULT_DIMENSION);
+ }
if (config.provider === "openai-compat") {
// eslint-disable-next-line @typescript-eslint/no-require-imports
const { OpenAICompatProvider } =
diff --git a/src/core/search/embeddings/registry.ts b/src/core/search/embeddings/registry.ts
new file mode 100644
index 00000000..38023925
--- /dev/null
+++ b/src/core/search/embeddings/registry.ts
@@ -0,0 +1,167 @@
+/**
+ * Provider registry (Embedding Provider Suite).
+ *
+ * Named provider profiles persisted to
+ * `Brain/search/embedding-providers.json`, added and removed at runtime
+ * through the CLI so users register an OpenAI-compatible endpoint
+ * without editing config. A registered name resolves to `openai-compat`
+ * config at config-resolution time (see `expandRegisteredProvider`),
+ * AFTER the built-ins, so a configured built-in key is never shadowed
+ * and the resolved provider union stays closed.
+ *
+ * Only env-key NAMES are stored, never secret values - the file is safe
+ * to sync. Loading is fail-soft: an absent or malformed registry yields
+ * an empty list and never throws into the hot path.
+ */
+
+import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+
+import { SearchError } from "../types.ts";
+
+/** A registered OpenAI-compatible provider profile. */
+export interface ProviderProfile {
+ readonly name: string;
+ readonly baseUrl: string;
+ readonly defaultModel: string;
+ /** Environment variable NAME the API key is read from (never the key). */
+ readonly envKey: string;
+}
+
+/** Built-in provider names a profile may not reuse. */
+export const RESERVED_PROVIDER_NAMES: ReadonlyArray<string> = Object.freeze([
+ "openai-compat",
+ "disabled",
+ "local",
+]);
+
+const NAME_RE = /^[a-z0-9][a-z0-9-]*$/;
+
+export function providerRegistryPath(vault: string): string {
+ return join(vault, "Brain", "search", "embedding-providers.json");
+}
+
+function isProfile(value: unknown): value is ProviderProfile {
+ if (value === null || typeof value !== "object") return false;
+ const v = value as Record<string, unknown>;
+ return (
+ typeof v["name"] === "string" &&
+ typeof v["baseUrl"] === "string" &&
+ typeof v["defaultModel"] === "string" &&
+ typeof v["envKey"] === "string"
+ );
+}
+
+/** Load the registry, fail-soft: absent or malformed file -> empty list. */
+export function loadProviderRegistry(vault: string): ProviderProfile[] {
+ let text: string;
+ try {
+ text = readFileSync(providerRegistryPath(vault), "utf8");
+ } catch {
+ return [];
+ }
+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(text);
+ } catch {
+ return [];
+ }
+ if (!Array.isArray(parsed)) return [];
+ return parsed.filter(isProfile).map((p) => Object.freeze({ ...p }));
+}
+
+function writeProviderRegistry(vault: string, registry: ReadonlyArray<ProviderProfile>): void {
+ const path = providerRegistryPath(vault);
+ mkdirSync(dirname(path), { recursive: true });
+ // Sort by name so the on-disk file is deterministic across edits.
+ const sorted = [...registry].toSorted((a, b) => a.name.localeCompare(b.name));
+ writeFileSync(path, JSON.stringify(sorted, null, 2) + "\n");
+}
+
+function validateProfile(profile: ProviderProfile): void {
+ const name = profile.name.trim();
+ if (!NAME_RE.test(name)) {
+ throw new SearchError(
+ "INVALID_INPUT",
+ `provider name must be a lowercase slug ([a-z0-9-], starting alphanumeric), got '${profile.name}'`,
+ );
+ }
+ if (RESERVED_PROVIDER_NAMES.includes(name)) {
+ throw new SearchError(
+ "INVALID_INPUT",
+ `provider name '${name}' is reserved for a built-in provider`,
+ );
+ }
+ if (profile.baseUrl.trim() === "") {
+ throw new SearchError("INVALID_INPUT", "provider base-url must not be empty");
+ }
+ if (profile.defaultModel.trim() === "") {
+ throw new SearchError("INVALID_INPUT", "provider default-model must not be empty");
+ }
+ if (profile.envKey.trim() === "") {
+ throw new SearchError("INVALID_INPUT", "provider env-key must not be empty");
+ }
+}
+
+/**
+ * Add (or upsert by name) a provider profile and persist. Returns the
+ * full registry after the change.
+ */
+export function addProviderProfile(vault: string, profile: ProviderProfile): ProviderProfile[] {
+ validateProfile(profile);
+ const normalised: ProviderProfile = Object.freeze({
+ name: profile.name.trim(),
+ baseUrl: profile.baseUrl.trim(),
+ defaultModel: profile.defaultModel.trim(),
+ envKey: profile.envKey.trim(),
+ });
+ const without = loadProviderRegistry(vault).filter((p) => p.name !== normalised.name);
+ const next = [...without, normalised];
+ writeProviderRegistry(vault, next);
+ return loadProviderRegistry(vault);
+}
+
+/** Remove a profile by name; reports whether one was present. */
+export function removeProviderProfile(
+ vault: string,
+ name: string,
+): { removed: boolean; registry: ProviderProfile[] } {
+ const current = loadProviderRegistry(vault);
+ const next = current.filter((p) => p.name !== name);
+ const removed = next.length !== current.length;
+ if (removed) writeProviderRegistry(vault, next);
+ return { removed, registry: loadProviderRegistry(vault) };
+}
+
+export function getProviderProfile(vault: string, name: string): ProviderProfile | null {
+ return loadProviderRegistry(vault).find((p) => p.name === name) ?? null;
+}
+
+/** Fields a registered name expands into at config-resolution time. */
+export interface ExpandedProvider {
+ readonly provider: "openai-compat";
+ readonly baseUrl: string;
+ readonly model: string;
+ readonly apiKey: string | null;
+}
+
+/**
+ * Expand a registered provider name into `openai-compat` config, reading
+ * the API key from `env[profile.envKey]`. Returns null when the name is
+ * not registered (the caller then falls back to built-in resolution).
+ */
+export function expandRegisteredProvider(
+ name: string,
+ registry: ReadonlyArray<ProviderProfile>,
+ env: Readonly<Record<string, string | undefined>>,
+): ExpandedProvider | null {
+ const profile = registry.find((p) => p.name === name);
+ if (!profile) return null;
+ const apiKey = env[profile.envKey];
+ return Object.freeze({
+ provider: "openai-compat" as const,
+ baseUrl: profile.baseUrl,
+ model: profile.defaultModel,
+ apiKey: apiKey !== undefined && apiKey !== "" ? apiKey : null,
+ });
+}
diff --git a/src/core/search/embeddings/signature.ts b/src/core/search/embeddings/signature.ts
new file mode 100644
index 00000000..2b276c46
--- /dev/null
+++ b/src/core/search/embeddings/signature.ts
@@ -0,0 +1,115 @@
+/**
+ * Embedding signature + cost kernel (Embedding Provider Suite).
+ *
+ * The single source of provider-identity truth shared by the local
+ * embedder, the provider registry, the cost gate, and the store's
+ * corpus-generation fingerprint. Centralising it here keeps those call
+ * sites from drifting on what "the same embedding configuration" means.
+ *
+ * Pure and I/O-free: canonicalise an identity to a stable signature
+ * string, look up a best-effort per-model price, estimate tokens and
+ * spend, and compare two signatures for staleness.
+ */
+
+/** Identity triple that determines whether two embeddings are comparable. */
+export interface EmbeddingIdentity {
+ readonly provider: string;
+ readonly model: string | null;
+ readonly dimension: number | null;
+}
+
+/** Model name produced by the offline local embedder (priced at 0). */
+export const LOCAL_EMBEDDING_MODEL = "hashing-ngram-v1";
+
+/** Sentinel for a null model/dimension so signatures stay parseable. */
+const NULL_FIELD = "?";
+
+function canonicalToken(raw: string): string {
+ return raw.normalize("NFC").trim().toLowerCase();
+}
+
+/**
+ * Canonical signature `<provider>:<model>:<dimension>`. Provider and
+ * model are NFC-normalised, trimmed, and lowercased; a null model or
+ * dimension renders as the stable `?` sentinel. Two configurations that
+ * produce the same signature yield comparable vectors.
+ */
+export function embeddingSignature(id: EmbeddingIdentity): string {
+ const provider = canonicalToken(id.provider);
+ const model = id.model === null ? NULL_FIELD : canonicalToken(id.model);
+ const dimension = id.dimension === null ? NULL_FIELD : String(id.dimension);
+ return `${provider}:${model}:${dimension}`;
+}
+
+/**
+ * Best-effort embedding price in USD per million input tokens, keyed by
+ * model name. Rates drift as providers change pricing, so the table is
+ * deliberately small and any model NOT listed is treated as free
+ * (price 0) - the cost gate must never falsely block on a model whose
+ * rate we do not know. The local embedder is listed explicitly at 0.
+ */
+export const EMBEDDING_PRICING: Readonly<Record<string, number>> = Object.freeze({
+ [LOCAL_EMBEDDING_MODEL]: 0,
+ "text-embedding-3-small": 0.02,
+ "text-embedding-3-large": 0.13,
+ "text-embedding-ada-002": 0.1,
+});
+
+/** Per-million-token price for a model; 0 for the local/unknown/null case. */
+export function pricePerMillionTokens(model: string | null): number {
+ if (model === null) return 0;
+ const rate = EMBEDDING_PRICING[canonicalToken(model)];
+ return typeof rate === "number" && Number.isFinite(rate) && rate > 0 ? rate : 0;
+}
+
+/**
+ * Cheap token estimate: a chars/4 heuristic per text, rounded up, summed.
+ * This is the same order-of-magnitude approximation embedding providers
+ * use for quick quotes; it intentionally avoids a real tokenizer so the
+ * estimate stays dependency-free and deterministic.
+ */
+export function estimateTokens(texts: ReadonlyArray<string>): number {
+ let total = 0;
+ for (const t of texts) {
+ if (t.length === 0) continue;
+ total += Math.ceil(t.length / 4);
+ }
+ return total;
+}
+
+/** Estimated spend in USD for `tokens` against `model`'s rate. */
+export function estimateCostUsd(tokens: number, model: string | null): number {
+ const rate = pricePerMillionTokens(model);
+ if (rate === 0) return 0;
+ return (tokens / 1_000_000) * rate;
+}
+
+/** True when the active signature differs from the stored one. */
+export function isStaleSignature(active: string, stored: string): boolean {
+ return active !== stored;
+}
+
+/** Outcome of a cost-gate evaluation for an embedding run. */
+export interface CostGateResult {
+ readonly tokens: number;
+ readonly estimatedUsd: number;
+ readonly blocked: boolean;
+}
+
+/**
+ * Evaluate whether an embedding run should be blocked on estimated spend.
+ * A run is blocked only when the gate is positive, the run is not forced,
+ * and the estimate strictly exceeds the gate. A zero gate (the default)
+ * and free models (local/unknown -> estimate 0) never block.
+ */
+export function evaluateCostGate(opts: {
+ texts: ReadonlyArray<string>;
+ model: string | null;
+ gateUsd: number;
+ forced?: boolean;
+}): CostGateResult {
+ const tokens = estimateTokens(opts.texts);
+ const estimatedUsd = estimateCostUsd(tokens, opts.model);
+ const blocked = opts.gateUsd > 0 && opts.forced !== true && estimatedUsd > opts.gateUsd;
+ return { tokens, estimatedUsd, blocked };
+}
diff --git a/src/core/search/fusion.ts b/src/core/search/fusion.ts
new file mode 100644
index 00000000..004c2a43
--- /dev/null
+++ b/src/core/search/fusion.ts
@@ -0,0 +1,66 @@
+/**
+ * Rank-fusion strategies for hybrid recall (Embedding Provider Suite).
+ *
+ * The default `linear` mode (a weighted sum of min-max-normalised BM25
+ * and cosine, computed in `ranker.ts`) is unchanged. This module adds
+ * `rrf` - Reciprocal Rank Fusion - which combines the sparse and dense
+ * lanes by RANK POSITION rather than by score magnitude:
+ *
+ * rrf(chunk) = sum over lanes of 1 / (k + rank_in_lane)
+ *
+ * RRF is weightless by design (it ignores the per-lane weights and the
+ * intent multipliers) and robust to lanes whose score scales differ. The
+ * raw RRF sums are min-max-normalised to [0, 1] so the fused relevance is
+ * on the same scale as the linear path and composes with the same
+ * downstream boosts (link / recency / entity / tier / session focus).
+ */
+
+/** Canonical RRF damping constant from the original Cormack et al. paper. */
+export const DEFAULT_RRF_K = 60;
+
+export type FusionMode = "linear" | "rrf";
+
+const FUSION_MODES: ReadonlySet<string> = new Set(["linear", "rrf"]);
+
+export function isFusionMode(value: string): value is FusionMode {
+ return FUSION_MODES.has(value);
+}
+
+/**
+ * Fuse two lanes by reciprocal rank and min-max-normalise the result to
+ * [0, 1]. Each input is the lane's chunk ids in best-first order; a chunk
+ * contributes `1 / (k + position)` for each lane it appears in. Returns a
+ * map from chunk id to its normalised fused relevance. An empty input on
+ * both lanes yields an empty map.
+ */
+export function rrfFuse(opts: {
+ keywordRankedChunkIds: ReadonlyArray<number>;
+ semanticRankedChunkIds: ReadonlyArray<number>;
+ k: number;
+}): Map<number, number> {
+ const k = Math.max(1, opts.k);
+ const raw = new Map<number, number>();
+ const accumulate = (ids: ReadonlyArray<number>): void => {
+ for (let i = 0; i < ids.length; i++) {
+ const id = ids[i]!;
+ raw.set(id, (raw.get(id) ?? 0) + 1 / (k + (i + 1)));
+ }
+ };
+ accumulate(opts.keywordRankedChunkIds);
+ accumulate(opts.semanticRankedChunkIds);
+
+ if (raw.size === 0) return raw;
+
+ const scores = [...raw.values()];
+ const min = Math.min(...scores);
+ const max = Math.max(...scores);
+ const out = new Map<number, number>();
+ if (max === min) {
+ for (const id of raw.keys()) out.set(id, 1);
+ return out;
+ }
+ for (const [id, score] of raw) {
+ out.set(id, (score - min) / (max - min));
+ }
+ return out;
+}
diff --git a/src/core/search/index.ts b/src/core/search/index.ts
index 2e931e72..8182a639 100644
--- a/src/core/search/index.ts
+++ b/src/core/search/index.ts
@@ -12,6 +12,12 @@ import {
} from "../validate.ts";
import { resolveVaultScope } from "../vault-scope/index.ts";
import { resolveIndexPath } from "./paths.ts";
+import { isFusionMode, DEFAULT_RRF_K } from "./fusion.ts";
+import {
+ loadProviderRegistry,
+ expandRegisteredProvider,
+ type ExpandedProvider,
+} from "./embeddings/registry.ts";
import { SearchError } from "./types.ts";
import type {
ResolvedEmbeddingConfig,
@@ -54,6 +60,15 @@ export {
} from "./session-focus.ts";
export { evaluateSurfacingGate, type SurfacingGateDecision } from "./surfacing-gate.ts";
export { buildEvidencePack, type EvidencePack } from "./evidence-pack.ts";
+export {
+ loadProviderRegistry,
+ addProviderProfile,
+ removeProviderProfile,
+ getProviderProfile,
+ providerRegistryPath,
+ RESERVED_PROVIDER_NAMES,
+ type ProviderProfile,
+} from "./embeddings/registry.ts";
export { resolveIndexPath } from "./paths.ts";
export {
@@ -88,6 +103,7 @@ const DEFAULTS = {
timeoutMs: 10_000,
concurrency: 4,
batchSize: 32,
+ costGateUsd: 0,
mmrLambda: 0.7,
maxHops: 1,
hopDecay: 0.5,
@@ -97,8 +113,19 @@ const DEFAULTS = {
recencyAmplitude: 0.05,
synonymMaxTerms: 3,
cacheTtlSeconds: 300,
+ fusionMode: "linear" as const,
+ rrfK: DEFAULT_RRF_K,
};
+function parseFusionMode(raw: string | null): "linear" | "rrf" {
+ if (raw === null) return DEFAULTS.fusionMode;
+ if (isFusionMode(raw)) return raw;
+ throw new SearchError(
+ "INVALID_INPUT",
+ `search_fusion_mode must be 'linear' or 'rrf', got '${raw}'`,
+ );
+}
+
type IntegerRange = { readonly min?: number; readonly max?: number };
function parseInteger(
@@ -140,6 +167,16 @@ function parsePositiveFloat(raw: string | null, fallback: number, fieldName: str
return n;
}
+/** Parse a non-negative finite float (e.g. a cost gate; 0 disables). */
+function parseNonNegativeFloat(raw: string | null, fallback: number, fieldName: string): number {
+ if (raw === null) return fallback;
+ const n = Number(raw);
+ if (!Number.isFinite(n) || n < 0) {
+ throw new SearchError("INVALID_INPUT", `${fieldName} must be a number >= 0, got '${raw}'`);
+ }
+ return n;
+}
+
function validateIntegerRange(n: number, fieldName: string, range?: IntegerRange): void {
if (range?.min !== undefined && n < range.min) {
throw new SearchError("INVALID_INPUT", `${fieldName} must be >= ${range.min}, got ${n}`);
@@ -190,13 +227,35 @@ function validateResolvedConfig(config: ResolvedSearchConfig): void {
function parseProvider(raw: string | null): ResolvedEmbeddingConfig["provider"] {
if (raw === null) return DEFAULTS.provider;
- if (raw === "openai-compat" || raw === "disabled") return raw;
+ if (raw === "openai-compat" || raw === "disabled" || raw === "local") return raw;
throw new SearchError(
"INVALID_INPUT",
- `embedding_provider must be 'openai-compat' or 'disabled', got '${raw}'`,
+ `embedding_provider must be 'openai-compat', 'local', 'disabled', or a registered provider name, got '${raw}'`,
);
}
+const BUILTIN_PROVIDERS: ReadonlySet<string> = new Set(["openai-compat", "disabled", "local"]);
+
+/**
+ * Resolve a non-built-in `embedding_provider` name against the registry.
+ * Returns null when the name is null, a built-in, or not registered - the
+ * caller then defers to `parseProvider` (which validates built-ins and
+ * raises a clear error for an unknown name). Fail-soft: a bad registry
+ * yields null, never an exception.
+ */
+function resolveRegistryProvider(
+ rawProvider: string | null,
+ vault: string,
+ env: NodeJS.ProcessEnv,
+): ExpandedProvider | null {
+ if (rawProvider === null || BUILTIN_PROVIDERS.has(rawProvider)) return null;
+ try {
+ return expandRegisteredProvider(rawProvider, loadProviderRegistry(vault), env);
+ } catch {
+ return null;
+ }
+}
+
export function resolveSearchConfig(opts: {
vault: string;
configPath?: string;
@@ -234,6 +293,15 @@ export function resolveSearchConfig(opts: {
DEFAULTS.semanticWeight,
"search_semantic_weight",
);
+ const fusionMode = parseFusionMode(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_FUSION_MODE", "search_fusion_mode"),
+ );
+ const rrfK = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_RRF_K", "search_rrf_k"),
+ DEFAULTS.rrfK,
+ "search_rrf_k",
+ { min: 1 },
+ );
// v0.10.9: single source of truth lives in Brain/_brain.yaml under
// `vault.ignore_paths`. The legacy `search_ignore_paths` config key
@@ -246,17 +314,40 @@ export function resolveSearchConfig(opts: {
false,
"search_semantic_enabled",
);
- const provider = parseProvider(
- envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER", "embedding_provider"),
+ const rawProvider = envOrConfig(
+ env,
+ config,
+ "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER",
+ "embedding_provider",
);
- const baseUrl = envOrConfig(
+ // A provider name that is not a built-in is looked up in the registry
+ // AFTER the built-ins, so it never shadows an explicitly configured
+ // built-in. A registered name resolves to openai-compat fields; an
+ // unknown name falls through to parseProvider's clear validation error.
+ const registryExpansion = resolveRegistryProvider(rawProvider, opts.vault, env);
+ const provider = registryExpansion ? "openai-compat" : parseProvider(rawProvider);
+ const explicitBaseUrl = envOrConfig(
env,
config,
"OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL",
"embedding_base_url",
);
- const model = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_MODEL", "embedding_model");
- const apiKey = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_KEY", "embedding_api_key");
+ const explicitModel = envOrConfig(
+ env,
+ config,
+ "OPEN_SECOND_BRAIN_EMBEDDING_MODEL",
+ "embedding_model",
+ );
+ const explicitApiKey = envOrConfig(
+ env,
+ config,
+ "OPEN_SECOND_BRAIN_EMBEDDING_KEY",
+ "embedding_api_key",
+ );
+ // Explicit config/env always wins over the registry profile's fields.
+ const baseUrl = explicitBaseUrl ?? registryExpansion?.baseUrl ?? null;
+ const model = explicitModel ?? registryExpansion?.model ?? null;
+ const apiKey = explicitApiKey ?? registryExpansion?.apiKey ?? null;
const dimRaw = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_DIM", "embedding_dimension");
const dimension =
dimRaw === null ? null : parseInteger(dimRaw, 0, "embedding_dimension", { min: 1 });
@@ -278,6 +369,11 @@ export function resolveSearchConfig(opts: {
"embedding_batch_size",
{ min: 1 },
);
+ const costGateUsd = parseNonNegativeFloat(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_COST_GATE", "embedding_cost_gate_usd"),
+ DEFAULTS.costGateUsd,
+ "embedding_cost_gate_usd",
+ );
const semantic: ResolvedEmbeddingConfig = Object.freeze({
enabled: semanticEnabled,
@@ -289,6 +385,7 @@ export function resolveSearchConfig(opts: {
timeoutMs,
concurrency,
batchSize,
+ costGateUsd,
});
const mmrLambda = parseFloat01(
@@ -415,6 +512,8 @@ export function resolveSearchConfig(opts: {
chunkOverlap,
keywordWeight,
semanticWeight,
+ fusionMode,
+ rrfK,
semantic,
recall,
});
diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.ts
index 3ff8a583..cc65c22b 100644
--- a/src/core/search/indexer.ts
+++ b/src/core/search/indexer.ts
@@ -28,6 +28,14 @@ import { basename, dirname } from "node:path";
import { chunkMarkdown } from "./chunker.ts";
import { expandTextForCjkFts } from "./cjk-tokenizer.ts";
import { makeProvider } from "./embeddings/provider.ts";
+import {
+ embeddingSignature,
+ estimateCostUsd,
+ estimateTokens,
+ evaluateCostGate,
+ pricePerMillionTokens,
+ LOCAL_EMBEDDING_MODEL,
+} from "./embeddings/signature.ts";
import { extractLinks } from "./links.ts";
import { extractFrontmatterRelations } from "../graph/frontmatter-relations.ts";
import { parseFrontmatterText } from "../vault.ts";
@@ -56,6 +64,8 @@ export interface IndexVaultOptions {
readonly embeddings?: boolean;
/** When true, every file is reindexed even if hash + mtime match. */
readonly force?: boolean;
+ /** When true, bypass the embedding cost gate for this run. */
+ readonly forceCost?: boolean;
readonly onFile?: (event: IndexProgressEvent) => void;
}
@@ -264,7 +274,7 @@ async function indexInto(
store.resolveLinkTargets();
if (opts?.embeddings) {
- await populateEmbeddings(store, config, stats);
+ await populateEmbeddings(store, config, stats, opts?.forceCost === true);
}
const now = new Date().toISOString();
@@ -285,10 +295,30 @@ async function indexInto(
}
}
+/**
+ * Canonical signature of the ACTIVE embedding configuration for status
+ * reporting. Null when semantic search is disabled. Stored model and
+ * dimension fill in fields the config leaves unset (e.g. an
+ * auto-detected dimension or the local provider's implicit model).
+ */
+function activeEmbeddingSignature(
+ config: ResolvedSearchConfig,
+ storedModel: string | null = null,
+ storedDim: number | null = null,
+): string | null {
+ if (!config.semantic.enabled) return null;
+ const provider = config.semantic.provider;
+ const model =
+ provider === "local" ? LOCAL_EMBEDDING_MODEL : (config.semantic.model ?? storedModel);
+ const dimension = config.semantic.dimension ?? storedDim;
+ return embeddingSignature({ provider, model, dimension });
+}
+
async function populateEmbeddings(
store: Store,
config: ResolvedSearchConfig,
stats: MutableStats,
+ forceCost: boolean,
): Promise<void> {
if (!config.semantic.enabled) {
throw new SearchError(
@@ -296,7 +326,8 @@ async function populateEmbeddings(
"set search_semantic_enabled=true and embedding_* keys to compute embeddings",
);
}
- if (!config.semantic.apiKey) {
+ // The offline local provider needs no key; every remote provider does.
+ if (config.semantic.provider !== "local" && !config.semantic.apiKey) {
throw new SearchError(
"EMBEDDING_KEY_MISSING",
"embedding_api_key is required when computing embeddings",
@@ -314,6 +345,24 @@ async function populateEmbeddings(
const provider = makeProvider(config.semantic);
const model = config.semantic.model ?? provider.model;
+
+ // Cost gate: estimate the spend for the whole pending set up front and
+ // refuse the run when it exceeds the configured ceiling, unless forced.
+ // The local provider (price 0) and unknown-price models never block.
+ const gate = evaluateCostGate({
+ texts: pending.map((p) => p.content),
+ model,
+ gateUsd: config.semantic.costGateUsd,
+ forced: forceCost,
+ });
+ if (gate.blocked) {
+ throw new SearchError(
+ "EMBEDDING_COST_GATE",
+ `estimated embedding cost $${gate.estimatedUsd.toFixed(4)} for ${pending.length} chunk(s) ` +
+ `exceeds embedding_cost_gate_usd $${config.semantic.costGateUsd.toFixed(4)}. ` +
+ `Re-run with --force-cost to proceed or raise the gate.`,
+ );
+ }
const batchSize = Math.max(1, config.semantic.batchSize);
// Hand the provider a super-batch sized to fully saturate its
// internal `embedding_concurrency` semaphore. Without this multiplier
@@ -423,6 +472,8 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexSt
staleEmbeddings: 0,
embeddingModel: null,
embeddingDimension: null,
+ embeddingSignature: activeEmbeddingSignature(config),
+ estimatedRefreshCostUsd: 0,
vecExtension: "unknown" as const,
semanticEnabled: config.semantic.enabled,
embeddingKeyPresent: !!config.semantic.apiKey,
@@ -445,10 +496,29 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexSt
if (config.semantic.enabled && !store.vecLoaded()) {
warnings.push("sqlite-vec unavailable; semantic search disabled this session");
}
- if (config.semantic.enabled && !config.semantic.apiKey) {
+ if (
+ config.semantic.enabled &&
+ config.semantic.provider !== "local" &&
+ !config.semantic.apiKey
+ ) {
warnings.push("embedding_api_key not configured; semantic search disabled");
}
+ // Best-effort spend estimate to bring stale/missing embeddings current.
+ // Only scan chunk content when the active model is actually priced.
+ const activeModel =
+ config.semantic.provider === "local"
+ ? LOCAL_EMBEDDING_MODEL
+ : (config.semantic.model ?? model);
+ let estimatedRefreshCostUsd = 0;
+ if (config.semantic.enabled && pricePerMillionTokens(activeModel) > 0) {
+ const pending = store.findChunksWithoutEmbeddings();
+ estimatedRefreshCostUsd = estimateCostUsd(
+ estimateTokens(pending.map((p) => p.content)),
+ activeModel,
+ );
+ }
+
return Object.freeze({
indexPath: config.dbPath,
exists: true,
@@ -459,6 +529,8 @@ export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexSt
staleEmbeddings: counts.staleEmbeddings,
embeddingModel: model,
embeddingDimension: dim,
+ embeddingSignature: activeEmbeddingSignature(config, model, dim),
+ estimatedRefreshCostUsd,
vecExtension: store.vecLoaded() ? ("loaded" as const) : ("unavailable" as const),
semanticEnabled: config.semantic.enabled,
embeddingKeyPresent: !!config.semantic.apiKey,
diff --git a/src/core/search/ranker.ts b/src/core/search/ranker.ts
index b1238013..209e52b8 100644
--- a/src/core/search/ranker.ts
+++ b/src/core/search/ranker.ts
@@ -12,6 +12,7 @@
import { PAGE_TIER_DEFAULT, tierWeight, type PageTier } from "../brain/page-meta/tier.ts";
import { weibullDecay, DEFAULT_RECENCY, type WeibullRecencyOptions } from "./recency.ts";
import { scoreSessionFocusTarget, type SearchSessionFocus } from "./session-focus.ts";
+import { rrfFuse, DEFAULT_RRF_K, type FusionMode } from "./fusion.ts";
import type { KeywordHit, SemanticHit, HydratedChunk } from "./store.ts";
import type { BrainSearchResult, WeightProfile } from "./types.ts";
@@ -60,6 +61,15 @@ export interface RankerOptions {
*/
readonly weightProfile?: WeightProfile;
readonly sessionFocus?: SearchSessionFocus | null;
+ /**
+ * Rank-fusion mode (Embedding Provider Suite). `linear` (default) is
+ * the weighted sum of normalised BM25 and cosine; `rrf` fuses the two
+ * lanes by reciprocal rank. Absent or `linear` keeps ranking
+ * bit-identical to pre-suite behaviour.
+ */
+ readonly fusionMode?: FusionMode;
+ /** RRF damping constant; defaults to {@link DEFAULT_RRF_K}. */
+ readonly rrfK?: number;
}
const DAY_MS = 24 * 60 * 60 * 1000;
@@ -131,10 +141,12 @@ function buildReasons(parts: {
tierMul: number;
entityBoost?: number;
sessionFocus?: number;
+ rrf?: number;
}): ReadonlyArray<string> {
const reasons: string[] = [];
if (parts.keywordScore > 0) reasons.push(`fts5_bm25: ${fmt(parts.keywordScore)}`);
if (parts.semanticScore > 0) reasons.push(`semantic_cos: ${fmt(parts.semanticScore)}`);
+ if (parts.rrf && parts.rrf > 0) reasons.push(`rrf: ${fmt(parts.rrf)}`);
if (parts.entityBoost && parts.entityBoost > 0) {
reasons.push(`entity_match: ${fmt(parts.entityBoost)}`);
}
@@ -167,6 +179,25 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea
}
}
+ // Reciprocal Rank Fusion (Embedding Provider Suite): fuse the lanes by
+ // rank position instead of weighted magnitude. Off by default; when on,
+ // it replaces the linear relevance term while every boost still applies.
+ const fusionMode: FusionMode = opts.fusionMode ?? "linear";
+ let rrfByChunk: Map<number, number> | null = null;
+ if (fusionMode === "rrf") {
+ const keywordRanked = inputs.keyword
+ .toSorted((a, b) => a.bm25 - b.bm25) // lower BM25 = better
+ .map((h) => h.chunkId);
+ const semanticRanked = semanticEnabled
+ ? inputs.semantic.toSorted((a, b) => a.distance - b.distance).map((h) => h.chunkId)
+ : [];
+ rrfByChunk = rrfFuse({
+ keywordRankedChunkIds: keywordRanked,
+ semanticRankedChunkIds: semanticRanked,
+ k: opts.rrfK ?? DEFAULT_RRF_K,
+ });
+ }
+
const candidates = new Map<number, Candidate>();
for (const h of inputs.keyword) {
candidates.set(h.chunkId, {
@@ -248,9 +279,15 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea
const tag = tagBoostFor(c);
const linkBoost = Math.min(0.05, link + tag);
const recency = recencyBoost(c.mtime, nowMs, recencyOpts) * recMul;
+ // Relevance term: reciprocal-rank-fused when in rrf mode, otherwise
+ // the weighted sum of the normalised lanes. RRF is weightless, so the
+ // per-lane weights and intent multipliers do not apply to it.
+ const rrf = rrfByChunk?.get(c.chunkId) ?? 0;
const weighted =
- opts.keywordWeight * kwMul * c.keywordScore +
- (semanticEnabled ? opts.semanticWeight * semMul : 0) * c.semanticScore;
+ rrfByChunk !== null
+ ? rrf
+ : opts.keywordWeight * kwMul * c.keywordScore +
+ (semanticEnabled ? opts.semanticWeight * semMul : 0) * c.semanticScore;
// Tier multiplier applied to the relevance portion only so the
// tag / link / recency boosts stay tier-neutral. Default
// `supporting` β†’ 1.0 keeps untagged vaults bit-identical.
@@ -287,6 +324,7 @@ export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSea
tierMul,
entityBoost,
sessionFocus,
+ rrf: rrfByChunk !== null ? rrf : 0,
}),
}),
);
diff --git a/src/core/search/search.ts b/src/core/search/search.ts
index d7200f3e..97bd19e5 100644
--- a/src/core/search/search.ts
+++ b/src/core/search/search.ts
@@ -451,6 +451,8 @@ export async function search(
},
...(weightProfile !== undefined ? { weightProfile } : {}),
...(sessionFocus !== undefined ? { sessionFocus } : {}),
+ fusionMode: config.fusionMode,
+ rrfK: config.rrfK,
},
);
const capHit = ranked.length >= rankCap;
@@ -820,7 +822,8 @@ async function runSemanticPhase(
warnings.push("semantic not enabled in config; using keyword-only");
return { attempted: false, hits: [], warnings };
}
- if (!config.semantic.apiKey) {
+ // The offline local provider needs no key; every remote provider does.
+ if (config.semantic.provider !== "local" && !config.semantic.apiKey) {
if (opts.explicit) {
throw new SearchError("EMBEDDING_KEY_MISSING", "embedding key not configured");
}
diff --git a/src/core/search/types.ts b/src/core/search/types.ts
index c40d64f0..19697665 100644
--- a/src/core/search/types.ts
+++ b/src/core/search/types.ts
@@ -21,6 +21,7 @@ export const SEARCH_ERROR_CODES = [
"EMBEDDING_PROVIDER_HTTP",
"EMBEDDING_PROVIDER_TIMEOUT",
"EMBEDDING_DIMENSION_MISMATCH",
+ "EMBEDDING_COST_GATE",
"INDEX_LOCKED",
"INVALID_INPUT",
] as const;
@@ -136,6 +137,19 @@ export interface IndexStatusSnapshot {
readonly staleEmbeddings: number;
readonly embeddingModel: string | null;
readonly embeddingDimension: number | null;
+ /**
+ * Canonical `<provider>:<model>:<dimension>` fingerprint of the ACTIVE
+ * embedding configuration (Embedding Provider Suite). Null when
+ * semantic search is disabled. Compare with the stored model/dimension
+ * to reason about staleness after a config change.
+ */
+ readonly embeddingSignature: string | null;
+ /**
+ * Best-effort USD estimate to (re-)embed the chunks that currently
+ * lack a current embedding, at the active model's rate. 0 for the
+ * local/unknown-price case.
+ */
+ readonly estimatedRefreshCostUsd: number;
readonly vecExtension: VecExtensionState;
readonly semanticEnabled: boolean;
readonly embeddingKeyPresent: boolean;
@@ -232,7 +246,7 @@ export interface SearchOutcome {
export interface ResolvedEmbeddingConfig {
readonly enabled: boolean;
- readonly provider: "openai-compat" | "disabled";
+ readonly provider: "openai-compat" | "disabled" | "local";
readonly baseUrl: string | null;
readonly model: string | null;
readonly apiKey: string | null;
@@ -240,6 +254,12 @@ export interface ResolvedEmbeddingConfig {
readonly timeoutMs: number;
readonly concurrency: number;
readonly batchSize: number;
+ /**
+ * Spend ceiling in USD for a single embedding run (Embedding Provider
+ * Suite). 0 (default) disables the gate. When positive, an embedding
+ * run whose estimated cost exceeds this is refused unless forced.
+ */
+ readonly costGateUsd: number;
}
/**
@@ -328,6 +348,15 @@ export interface ResolvedSearchConfig {
readonly chunkOverlap: number;
readonly keywordWeight: number;
readonly semanticWeight: number;
+ /**
+ * Rank-fusion mode (Embedding Provider Suite). `linear` (default) is
+ * the weighted sum of normalised BM25 and cosine; `rrf` fuses the two
+ * lanes by reciprocal rank. `linear` keeps ranking bit-identical to
+ * pre-suite behaviour.
+ */
+ readonly fusionMode: "linear" | "rrf";
+ /** Reciprocal Rank Fusion damping constant (only used when rrf). */
+ readonly rrfK: number;
readonly semantic: ResolvedEmbeddingConfig;
readonly recall: ResolvedRecallConfig;
}
diff --git a/tests/cli/search-provider-cli.test.ts b/tests/cli/search-provider-cli.test.ts
new file mode 100644
index 00000000..33e2c30a
--- /dev/null
+++ b/tests/cli/search-provider-cli.test.ts
@@ -0,0 +1,107 @@
+/**
+ * CLI tests for `o2b search provider <add|list|show|remove>`.
+ *
+ * Forks the real `bun src/cli/main.ts` binary via `runCli`, asserting
+ * exit codes, JSON shape, and the persisted registry file.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { runCli } from "../helpers/run-cli.ts";
+
+let tmp: string;
+let vault: string;
+let config: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-provider-cli-"));
+ vault = join(tmp, "vault");
+ mkdirSync(vault, { recursive: true });
+ config = join(tmp, "config.yaml");
+ writeFileSync(config, `vault: "${vault}"\n`);
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+const env = () => ({ OPEN_SECOND_BRAIN_CONFIG: config });
+const addArgs = [
+ "search",
+ "provider",
+ "add",
+ "nvidia-nim",
+ "--base-url",
+ "https://integrate.api.nvidia.com/v1",
+ "--model",
+ "nvidia/nv-embed-v1",
+ "--env-key",
+ "NIM_API_KEY",
+];
+
+test("provider add then list shows the registered profile", async () => {
+ const add = await runCli(addArgs, { env: env() });
+ expect(add.returncode).toBe(0);
+ expect(existsSync(join(vault, "Brain", "search", "embedding-providers.json"))).toBe(true);
+
+ const list = await runCli(["search", "provider", "list"], { env: env() });
+ expect(list.returncode).toBe(0);
+ expect(list.stdout).toContain("nvidia-nim");
+});
+
+test("provider list --json returns the profile array", async () => {
+ await runCli(addArgs, { env: env() });
+ const list = await runCli(["search", "provider", "list", "--json"], { env: env() });
+ expect(list.returncode).toBe(0);
+ const arr = JSON.parse(list.stdout);
+ expect(Array.isArray(arr)).toBe(true);
+ expect(arr[0]).toMatchObject({
+ name: "nvidia-nim",
+ baseUrl: "https://integrate.api.nvidia.com/v1",
+ defaultModel: "nvidia/nv-embed-v1",
+ envKey: "NIM_API_KEY",
+ });
+});
+
+test("provider show prints a single profile", async () => {
+ await runCli(addArgs, { env: env() });
+ const show = await runCli(["search", "provider", "show", "nvidia-nim", "--json"], { env: env() });
+ expect(show.returncode).toBe(0);
+ expect(JSON.parse(show.stdout)).toMatchObject({ name: "nvidia-nim" });
+});
+
+test("provider remove deletes the profile", async () => {
+ await runCli(addArgs, { env: env() });
+ const rm = await runCli(["search", "provider", "remove", "nvidia-nim"], { env: env() });
+ expect(rm.returncode).toBe(0);
+ const list = await runCli(["search", "provider", "list", "--json"], { env: env() });
+ expect(JSON.parse(list.stdout)).toEqual([]);
+});
+
+test("provider add with a reserved name fails with exit 2", async () => {
+ const add = await runCli(
+ [
+ "search",
+ "provider",
+ "add",
+ "local",
+ "--base-url",
+ "https://x/v1",
+ "--model",
+ "m",
+ "--env-key",
+ "K",
+ ],
+ { env: env() },
+ );
+ expect(add.returncode).toBe(2);
+ expect(add.stderr).toContain("reserved");
+});
+
+test("provider show of an unknown name exits non-zero", async () => {
+ const show = await runCli(["search", "provider", "show", "ghost"], { env: env() });
+ expect(show.returncode).not.toBe(0);
+});
diff --git a/tests/core/search/embeddings.cost-gate.test.ts b/tests/core/search/embeddings.cost-gate.test.ts
new file mode 100644
index 00000000..610946d6
--- /dev/null
+++ b/tests/core/search/embeddings.cost-gate.test.ts
@@ -0,0 +1,76 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ evaluateCostGate,
+ LOCAL_EMBEDDING_MODEL,
+} from "../../../src/core/search/embeddings/signature.ts";
+import { resolveSearchConfig } from "../../../src/core/search/index.ts";
+
+// ── pure cost-gate kernel ────────────────────────────────────────────────────
+
+test("evaluateCostGate estimates spend for a priced model", () => {
+ // 4,000,000 chars -> ~1,000,000 tokens at chars/4.
+ const texts = ["x".repeat(4_000_000)];
+ const r = evaluateCostGate({ texts, model: "text-embedding-3-small", gateUsd: 0 });
+ expect(r.tokens).toBe(1_000_000);
+ expect(r.estimatedUsd).toBeGreaterThan(0);
+});
+
+test("evaluateCostGate blocks when the estimate exceeds a positive gate", () => {
+ const texts = ["x".repeat(4_000_000)]; // ~1M tokens
+ const r = evaluateCostGate({ texts, model: "text-embedding-3-small", gateUsd: 0.001 });
+ expect(r.blocked).toBe(true);
+});
+
+test("evaluateCostGate does not block when forced", () => {
+ const texts = ["x".repeat(4_000_000)];
+ const r = evaluateCostGate({
+ texts,
+ model: "text-embedding-3-small",
+ gateUsd: 0.001,
+ forced: true,
+ });
+ expect(r.blocked).toBe(false);
+});
+
+test("a zero gate (default) never blocks", () => {
+ const texts = ["x".repeat(40_000_000)];
+ const r = evaluateCostGate({ texts, model: "text-embedding-3-small", gateUsd: 0 });
+ expect(r.blocked).toBe(false);
+});
+
+test("the local model never blocks regardless of gate or volume", () => {
+ const texts = ["x".repeat(40_000_000)];
+ const r = evaluateCostGate({ texts, model: LOCAL_EMBEDDING_MODEL, gateUsd: 0.0001 });
+ expect(r.estimatedUsd).toBe(0);
+ expect(r.blocked).toBe(false);
+});
+
+// ── config parsing ───────────────────────────────────────────────────────────
+
+let tmp: string;
+let config: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-costgate-"));
+ config = join(tmp, "config.yaml");
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+test("embedding_cost_gate_usd defaults to 0 (disabled)", () => {
+ writeFileSync(config, `vault: "${tmp}"\n`);
+ const cfg = resolveSearchConfig({ vault: tmp, configPath: config });
+ expect(cfg.semantic.costGateUsd).toBe(0);
+});
+
+test("embedding_cost_gate_usd is parsed from config", () => {
+ writeFileSync(config, `vault: "${tmp}"\nembedding_cost_gate_usd: "2.5"\n`);
+ const cfg = resolveSearchConfig({ vault: tmp, configPath: config });
+ expect(cfg.semantic.costGateUsd).toBe(2.5);
+});
diff --git a/tests/core/search/embeddings.local-provider.test.ts b/tests/core/search/embeddings.local-provider.test.ts
new file mode 100644
index 00000000..05a9c472
--- /dev/null
+++ b/tests/core/search/embeddings.local-provider.test.ts
@@ -0,0 +1,102 @@
+import { test, expect } from "bun:test";
+
+import {
+ LocalProvider,
+ LOCAL_DEFAULT_DIMENSION,
+} from "../../../src/core/search/embeddings/local-provider.ts";
+import { makeProvider } from "../../../src/core/search/embeddings/provider.ts";
+import { LOCAL_EMBEDDING_MODEL } from "../../../src/core/search/embeddings/signature.ts";
+import type { ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";
+
+function norm(v: ReadonlyArray<number>): number {
+ return Math.sqrt(v.reduce((s, x) => s + x * x, 0));
+}
+
+test("identifies as the local provider with the hashing model", () => {
+ const p = new LocalProvider(256);
+ expect(p.name).toBe("local");
+ expect(p.model).toBe(LOCAL_EMBEDDING_MODEL);
+ expect(p.dimension).toBe(256);
+});
+
+test("default dimension is applied when none is configured", () => {
+ const p = new LocalProvider();
+ expect(p.dimension).toBe(LOCAL_DEFAULT_DIMENSION);
+});
+
+test("produces one unit-normalised vector per input, in order", async () => {
+ const p = new LocalProvider(128);
+ const out = await p.embed(["alpha beta", "gamma delta", "epsilon"]);
+ expect(out.length).toBe(3);
+ for (const v of out) {
+ expect(v.length).toBe(128);
+ expect(norm(v)).toBeCloseTo(1.0, 6);
+ }
+});
+
+test("is deterministic: identical text yields an identical vector", async () => {
+ const p = new LocalProvider(64);
+ const [a] = await p.embed(["the quick brown fox"]);
+ const [b] = await p.embed(["the quick brown fox"]);
+ expect(a).toEqual(b!);
+});
+
+test("distinct text yields distinct vectors", async () => {
+ const p = new LocalProvider(64);
+ const [a, b] = await p.embed(["machine learning", "garden vegetables"]);
+ expect(a).not.toEqual(b!);
+});
+
+test("empty text yields a zero vector of the right length", async () => {
+ const p = new LocalProvider(32);
+ const [v] = await p.embed([""]);
+ expect(v!.length).toBe(32);
+ expect(norm(v!)).toBe(0);
+});
+
+test("embed() with no inputs returns an empty array", async () => {
+ const p = new LocalProvider();
+ expect(await p.embed([])).toEqual([]);
+});
+
+test("ping() always succeeds offline with the configured dimension", async () => {
+ const p = new LocalProvider(200);
+ const res = await p.ping();
+ expect(res).toEqual({ ok: true, dimension: 200 });
+});
+
+test("makeProvider returns a LocalProvider for provider 'local'", () => {
+ const config: ResolvedEmbeddingConfig = Object.freeze({
+ enabled: true,
+ provider: "local",
+ baseUrl: null,
+ model: null,
+ apiKey: null,
+ dimension: 384,
+ timeoutMs: 10_000,
+ concurrency: 1,
+ batchSize: 1,
+ costGateUsd: 0,
+ });
+ const p = makeProvider(config);
+ expect(p).toBeInstanceOf(LocalProvider);
+ expect(p.dimension).toBe(384);
+ expect(p.model).toBe(LOCAL_EMBEDDING_MODEL);
+});
+
+test("makeProvider local needs no api key", () => {
+ const config: ResolvedEmbeddingConfig = Object.freeze({
+ enabled: true,
+ provider: "local",
+ baseUrl: null,
+ model: null,
+ apiKey: null,
+ dimension: null,
+ timeoutMs: 10_000,
+ concurrency: 1,
+ batchSize: 1,
+ costGateUsd: 0,
+ });
+ expect(() => makeProvider(config)).not.toThrow();
+ expect(makeProvider(config).dimension).toBe(LOCAL_DEFAULT_DIMENSION);
+});
diff --git a/tests/core/search/embeddings.registry.test.ts b/tests/core/search/embeddings.registry.test.ts
new file mode 100644
index 00000000..b777cb5e
--- /dev/null
+++ b/tests/core/search/embeddings.registry.test.ts
@@ -0,0 +1,107 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { mkdtempSync, rmSync, mkdirSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ loadProviderRegistry,
+ addProviderProfile,
+ removeProviderProfile,
+ getProviderProfile,
+ expandRegisteredProvider,
+ providerRegistryPath,
+ RESERVED_PROVIDER_NAMES,
+} from "../../../src/core/search/embeddings/registry.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+
+let vault: string;
+
+beforeEach(() => {
+ vault = mkdtempSync(join(tmpdir(), "o2b-registry-"));
+});
+
+afterEach(() => {
+ rmSync(vault, { recursive: true, force: true });
+});
+
+const nim = {
+ name: "nvidia-nim",
+ baseUrl: "https://integrate.api.nvidia.com/v1",
+ defaultModel: "nvidia/nv-embed-v1",
+ envKey: "NIM_API_KEY",
+};
+
+test("empty vault has an empty registry", () => {
+ expect(loadProviderRegistry(vault)).toEqual([]);
+});
+
+test("add -> list -> get -> remove round-trips a profile", () => {
+ const after = addProviderProfile(vault, nim);
+ expect(after).toHaveLength(1);
+ expect(loadProviderRegistry(vault)).toHaveLength(1);
+ expect(getProviderProfile(vault, "nvidia-nim")).toMatchObject(nim);
+
+ const { removed, registry } = removeProviderProfile(vault, "nvidia-nim");
+ expect(removed).toBe(true);
+ expect(registry).toEqual([]);
+ expect(getProviderProfile(vault, "nvidia-nim")).toBeNull();
+});
+
+test("registry persists to Brain/search/embedding-providers.json", () => {
+ addProviderProfile(vault, nim);
+ expect(providerRegistryPath(vault)).toBe(
+ join(vault, "Brain", "search", "embedding-providers.json"),
+ );
+});
+
+test("add upserts an existing name rather than duplicating", () => {
+ addProviderProfile(vault, nim);
+ const after = addProviderProfile(vault, { ...nim, defaultModel: "nvidia/nv-embed-v2" });
+ expect(after).toHaveLength(1);
+ expect(getProviderProfile(vault, "nvidia-nim")?.defaultModel).toBe("nvidia/nv-embed-v2");
+});
+
+test("reserved built-in names cannot be registered", () => {
+ for (const reserved of RESERVED_PROVIDER_NAMES) {
+ expect(() => addProviderProfile(vault, { ...nim, name: reserved })).toThrow(SearchError);
+ }
+});
+
+test("invalid profile fields are rejected", () => {
+ expect(() => addProviderProfile(vault, { ...nim, name: "Bad Name" })).toThrow(/name/i);
+ expect(() => addProviderProfile(vault, { ...nim, baseUrl: "" })).toThrow(/base/i);
+ expect(() => addProviderProfile(vault, { ...nim, defaultModel: "" })).toThrow(/model/i);
+ expect(() => addProviderProfile(vault, { ...nim, envKey: "" })).toThrow(/env/i);
+});
+
+test("removing an absent profile reports removed:false", () => {
+ const { removed } = removeProviderProfile(vault, "ghost");
+ expect(removed).toBe(false);
+});
+
+test("expandRegisteredProvider resolves to openai-compat with the env key", () => {
+ const registry = addProviderProfile(vault, nim);
+ const expanded = expandRegisteredProvider("nvidia-nim", registry, { NIM_API_KEY: "secret-123" });
+ expect(expanded).toEqual({
+ provider: "openai-compat",
+ baseUrl: nim.baseUrl,
+ model: nim.defaultModel,
+ apiKey: "secret-123",
+ });
+});
+
+test("expandRegisteredProvider yields a null apiKey when the env var is unset", () => {
+ const registry = addProviderProfile(vault, nim);
+ const expanded = expandRegisteredProvider("nvidia-nim", registry, {});
+ expect(expanded?.apiKey).toBeNull();
+});
+
+test("expandRegisteredProvider returns null for an unknown name", () => {
+ expect(expandRegisteredProvider("missing", [], {})).toBeNull();
+});
+
+test("a malformed registry file degrades to empty, never throws", () => {
+ mkdirSync(join(vault, "Brain", "search"), { recursive: true });
+ writeFileSync(providerRegistryPath(vault), "{ not json");
+ expect(loadProviderRegistry(vault)).toEqual([]);
+});
diff --git a/tests/core/search/embeddings.signature.test.ts b/tests/core/search/embeddings.signature.test.ts
new file mode 100644
index 00000000..e46c8638
--- /dev/null
+++ b/tests/core/search/embeddings.signature.test.ts
@@ -0,0 +1,84 @@
+import { test, expect } from "bun:test";
+
+import {
+ embeddingSignature,
+ pricePerMillionTokens,
+ estimateTokens,
+ estimateCostUsd,
+ isStaleSignature,
+ EMBEDDING_PRICING,
+ LOCAL_EMBEDDING_MODEL,
+} from "../../../src/core/search/embeddings/signature.ts";
+
+test("embeddingSignature canonicalises provider/model/dimension", () => {
+ expect(
+ embeddingSignature({
+ provider: "openai-compat",
+ model: "text-embedding-3-small",
+ dimension: 1536,
+ }),
+ ).toBe("openai-compat:text-embedding-3-small:1536");
+});
+
+test("embeddingSignature lowercases and trims provider + model", () => {
+ expect(
+ embeddingSignature({
+ provider: " OpenAI-Compat ",
+ model: " Text-Embedding-3-Small ",
+ dimension: 256,
+ }),
+ ).toBe("openai-compat:text-embedding-3-small:256");
+});
+
+test("embeddingSignature renders null model/dimension as a stable sentinel", () => {
+ expect(embeddingSignature({ provider: "local", model: null, dimension: null })).toBe("local:?:?");
+});
+
+test("two identities with the same fields share a signature; different dim differs", () => {
+ const a = embeddingSignature({ provider: "local", model: "hashing-ngram-v1", dimension: 256 });
+ const b = embeddingSignature({ provider: "local", model: "hashing-ngram-v1", dimension: 256 });
+ const c = embeddingSignature({ provider: "local", model: "hashing-ngram-v1", dimension: 512 });
+ expect(a).toBe(b);
+ expect(a).not.toBe(c);
+});
+
+test("pricePerMillionTokens returns a positive rate for a known remote model", () => {
+ expect(pricePerMillionTokens("text-embedding-3-small")).toBeGreaterThan(0);
+});
+
+test("pricePerMillionTokens is 0 for the local model and for unknown models", () => {
+ expect(pricePerMillionTokens(LOCAL_EMBEDDING_MODEL)).toBe(0);
+ expect(pricePerMillionTokens("some-model-nobody-priced")).toBe(0);
+ expect(pricePerMillionTokens(null)).toBe(0);
+});
+
+test("local model is present in the pricing table at 0", () => {
+ expect(EMBEDDING_PRICING[LOCAL_EMBEDDING_MODEL]).toBe(0);
+});
+
+test("estimateTokens sums a chars/4 heuristic and rounds up", () => {
+ // "abcd" -> 1 token; "abcdefgh" -> 2 tokens.
+ expect(estimateTokens(["abcd"])).toBe(1);
+ expect(estimateTokens(["abcdefgh"])).toBe(2);
+ expect(estimateTokens(["abcde"])).toBe(2); // ceil(5/4)
+ expect(estimateTokens([])).toBe(0);
+ expect(estimateTokens(["", ""])).toBe(0);
+});
+
+test("estimateCostUsd scales tokens by the per-million rate", () => {
+ // 1,000,000 tokens * rate == rate USD.
+ const rate = pricePerMillionTokens("text-embedding-3-small");
+ expect(estimateCostUsd(1_000_000, "text-embedding-3-small")).toBeCloseTo(rate, 9);
+ expect(estimateCostUsd(500_000, "text-embedding-3-small")).toBeCloseTo(rate / 2, 9);
+});
+
+test("estimateCostUsd is 0 for local/unknown models regardless of token count", () => {
+ expect(estimateCostUsd(10_000_000, LOCAL_EMBEDDING_MODEL)).toBe(0);
+ expect(estimateCostUsd(10_000_000, "unknown")).toBe(0);
+});
+
+test("isStaleSignature is true only when active and stored differ", () => {
+ expect(isStaleSignature("local:m:256", "local:m:256")).toBe(false);
+ expect(isStaleSignature("local:m:256", "local:m:512")).toBe(true);
+ expect(isStaleSignature("openai-compat:a:1536", "local:m:256")).toBe(true);
+});
diff --git a/tests/core/search/embeddings.test.ts b/tests/core/search/embeddings.test.ts
index d18770db..fdfb4b15 100644
--- a/tests/core/search/embeddings.test.ts
+++ b/tests/core/search/embeddings.test.ts
@@ -20,6 +20,7 @@ function cfg(overrides: Partial<ResolvedEmbeddingConfig> = {}): ResolvedEmbeddin
timeoutMs: 5_000,
concurrency: 2,
batchSize: 32,
+ costGateUsd: 0,
...overrides,
});
}
@@ -214,6 +215,7 @@ test("makeProvider returns NullProvider when semantic is disabled", () => {
timeoutMs: 10_000,
concurrency: 1,
batchSize: 1,
+ costGateUsd: 0,
});
expect(p).toBeInstanceOf(NullProvider);
});
@@ -230,6 +232,7 @@ test("makeProvider throws when key is missing under openai-compat", () => {
timeoutMs: 10_000,
concurrency: 1,
batchSize: 1,
+ costGateUsd: 0,
}),
).toThrow(/embedding_api_key/);
});
diff --git a/tests/core/search/fusion.test.ts b/tests/core/search/fusion.test.ts
new file mode 100644
index 00000000..cde7f819
--- /dev/null
+++ b/tests/core/search/fusion.test.ts
@@ -0,0 +1,65 @@
+import { test, expect } from "bun:test";
+
+import { rrfFuse, isFusionMode, DEFAULT_RRF_K } from "../../../src/core/search/fusion.ts";
+
+test("DEFAULT_RRF_K is the canonical 60", () => {
+ expect(DEFAULT_RRF_K).toBe(60);
+});
+
+test("isFusionMode accepts the two modes and rejects others", () => {
+ expect(isFusionMode("linear")).toBe(true);
+ expect(isFusionMode("rrf")).toBe(true);
+ expect(isFusionMode("weighted")).toBe(false);
+ expect(isFusionMode("")).toBe(false);
+});
+
+test("empty lanes produce an empty fusion map", () => {
+ expect(rrfFuse({ keywordRankedChunkIds: [], semanticRankedChunkIds: [], k: 60 }).size).toBe(0);
+});
+
+test("a chunk ranked top in both lanes scores highest", () => {
+ // chunk 1: rank 1 in both lanes. chunk 2: rank 2 keyword, absent semantic.
+ // chunk 3: absent keyword, rank 1 semantic... order them.
+ const fused = rrfFuse({
+ keywordRankedChunkIds: [1, 2],
+ semanticRankedChunkIds: [1, 3],
+ k: 60,
+ });
+ expect(fused.get(1)!).toBeGreaterThan(fused.get(2)!);
+ expect(fused.get(1)!).toBeGreaterThan(fused.get(3)!);
+});
+
+test("normalised scores span [0,1] with the best at 1 and worst at 0", () => {
+ const fused = rrfFuse({
+ keywordRankedChunkIds: [1, 2, 3],
+ semanticRankedChunkIds: [1, 2, 3],
+ k: 60,
+ });
+ const values = [...fused.values()];
+ expect(Math.max(...values)).toBeCloseTo(1, 9);
+ expect(Math.min(...values)).toBeCloseTo(0, 9);
+});
+
+test("reciprocal-rank ordering matches rank sums", () => {
+ // keyword: A(1) B(2) C(3); semantic: C(1) B(2) A(3).
+ // RRF sums: A = 1/61 + 1/63; B = 1/62 + 1/62; C = 1/63 + 1/61.
+ // A and C tie; B sits between by raw sum (2/62 vs 1/61+1/63).
+ const fused = rrfFuse({
+ keywordRankedChunkIds: [10, 20, 30],
+ semanticRankedChunkIds: [30, 20, 10],
+ k: 60,
+ });
+ expect(fused.get(10)!).toBeCloseTo(fused.get(30)!, 9);
+ // A/C raw = 1/61 + 1/63 β‰ˆ 0.032277; B raw = 2/62 β‰ˆ 0.032258 -> B is lowest.
+ expect(fused.get(20)!).toBeLessThan(fused.get(10)!);
+});
+
+test("a single populated lane still ranks by position", () => {
+ const fused = rrfFuse({
+ keywordRankedChunkIds: [5, 6, 7],
+ semanticRankedChunkIds: [],
+ k: 60,
+ });
+ expect(fused.get(5)!).toBeGreaterThan(fused.get(6)!);
+ expect(fused.get(6)!).toBeGreaterThan(fused.get(7)!);
+});
diff --git a/tests/core/search/indexer.embeddings.test.ts b/tests/core/search/indexer.embeddings.test.ts
index b9a9db11..dc3e4820 100644
--- a/tests/core/search/indexer.embeddings.test.ts
+++ b/tests/core/search/indexer.embeddings.test.ts
@@ -39,6 +39,7 @@ function semanticConfig(model = "fake-model", dim = 4) {
timeoutMs: 5_000,
concurrency: 2,
batchSize: 8,
+ costGateUsd: 0,
},
});
}
@@ -102,6 +103,7 @@ test("--embeddings without a key throws EMBEDDING_KEY_MISSING", async () => {
timeoutMs: 5_000,
concurrency: 1,
batchSize: 8,
+ costGateUsd: 0,
},
});
let err: SearchError | null = null;
diff --git a/tests/core/search/local-embedder-rrf.integration.test.ts b/tests/core/search/local-embedder-rrf.integration.test.ts
new file mode 100644
index 00000000..cdca2f7a
--- /dev/null
+++ b/tests/core/search/local-embedder-rrf.integration.test.ts
@@ -0,0 +1,82 @@
+/**
+ * Integration slice for the Embedding Provider Suite: index a vault with
+ * the offline LOCAL embedder (no network, no key) and search it under
+ * both fusion modes. Exercises the whole pipeline - config resolution,
+ * indexer cost gate (free for local), vec storage, semantic recall, and
+ * RRF fusion - end to end.
+ */
+
+import { test, expect, beforeEach, afterEach } from "bun:test";
+
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { indexStatus } from "../../../src/core/search/indexer.ts";
+import { search } from "../../../src/core/search/search.ts";
+import { LOCAL_EMBEDDING_MODEL } from "../../../src/core/search/embeddings/signature.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+import { sqliteVecLoadable } from "../../helpers/sqlite-vec.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ const v = createTempVault("local-rrf");
+ vault = v.vault;
+ dbPath = v.dbPath;
+ cleanup = v.cleanup;
+});
+
+afterEach(() => {
+ cleanup();
+});
+
+function localConfig(fusionMode: "linear" | "rrf" = "linear") {
+ return makeConfig({
+ vault,
+ dbPath,
+ fusionMode,
+ semantic: {
+ enabled: true,
+ provider: "local",
+ baseUrl: null,
+ model: null,
+ apiKey: null,
+ dimension: 128,
+ timeoutMs: 5_000,
+ concurrency: 2,
+ batchSize: 8,
+ costGateUsd: 0,
+ },
+ });
+}
+
+test("local embedder indexes with no network and embeds every chunk", async () => {
+ if (!sqliteVecLoadable()) return;
+ writeMd(vault, "a.md", "# Machine learning\n\nNeural networks and gradient descent.");
+ writeMd(vault, "b.md", "# Gardening\n\nTomatoes and compost in spring.");
+ const cfg = localConfig();
+
+ const stats = await indexVault(cfg, { embeddings: true });
+ expect(stats.embeddingsComputed).toBeGreaterThanOrEqual(2);
+
+ const status = await indexStatus(cfg);
+ expect(status.embeddingModel).toBe(LOCAL_EMBEDDING_MODEL);
+ expect(status.embeddingDimension).toBe(128);
+ expect(status.embeddingSignature).toBe(`local:${LOCAL_EMBEDDING_MODEL}:128`);
+ // The local provider is free, so there is never a refresh cost.
+ expect(status.estimatedRefreshCostUsd).toBe(0);
+});
+
+test("local embedder produces semantic hits and rrf surfaces its reason", async () => {
+ if (!sqliteVecLoadable()) return;
+ writeMd(vault, "ml.md", "# Machine learning\n\nNeural networks learn from data.");
+ writeMd(vault, "garden.md", "# Gardening\n\nTomatoes need water and sun.");
+ const cfg = localConfig("rrf");
+ await indexVault(cfg, { embeddings: true });
+
+ const outcome = await search(cfg, { query: "neural networks", limit: 5, semantic: true });
+ expect(outcome.results.length).toBeGreaterThan(0);
+ // At least one result fused through RRF carries the rrf reason.
+ const anyRrf = outcome.results.some((r) => r.reasons.some((x) => x.startsWith("rrf:")));
+ expect(anyRrf).toBe(true);
+});
diff --git a/tests/core/search/ranker-rrf.test.ts b/tests/core/search/ranker-rrf.test.ts
new file mode 100644
index 00000000..706cf57c
--- /dev/null
+++ b/tests/core/search/ranker-rrf.test.ts
@@ -0,0 +1,132 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { rankResults } from "../../../src/core/search/ranker.ts";
+import { resolveSearchConfig, SearchError } from "../../../src/core/search/index.ts";
+import type { KeywordHit, SemanticHit, HydratedChunk } from "../../../src/core/search/store.ts";
+
+const NOW = 1_750_000_000_000;
+const OLD = NOW / 1000 - 365 * 24 * 3600; // no recency boost
+
+function hyd(chunkId: number, docId: number): HydratedChunk {
+ return Object.freeze({
+ chunkId,
+ documentId: docId,
+ path: `doc${docId}.md`,
+ title: `Doc ${docId}`,
+ content: `chunk ${chunkId}`,
+ startLine: 1,
+ endLine: 1,
+ mtime: OLD,
+ });
+}
+
+// chunk 1: top of the keyword lane only. chunk 2: present in BOTH lanes
+// (keyword rank 2, semantic rank 1). chunk 3: semantic lane only.
+const KW: KeywordHit[] = [
+ { chunkId: 1, documentId: 10, bm25: -10 }, // best keyword, single-lane
+ { chunkId: 2, documentId: 11, bm25: -5 },
+];
+const SEM: SemanticHit[] = [
+ { chunkId: 2, documentId: 11, distance: 0.1 }, // best semantic
+ { chunkId: 3, documentId: 12, distance: 0.2 },
+];
+const HYDRATED = new Map<number, HydratedChunk>([
+ [1, hyd(1, 10)],
+ [2, hyd(2, 11)],
+ [3, hyd(3, 12)],
+]);
+
+function inputs() {
+ return {
+ keyword: KW,
+ semantic: SEM,
+ hydrated: HYDRATED,
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ };
+}
+
+test("rrf mode promotes the chunk present in BOTH lanes above a single-lane top hit", () => {
+ // chunk 2 (keyword rank 2 + semantic rank 1) fuses above chunk 1 (top
+ // of the keyword lane but absent from the semantic lane), because RRF
+ // rewards cross-lane presence: 1/62 + 1/61 > 1/61.
+ const ranked = rankResults(inputs(), {
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ limit: 10,
+ nowMs: NOW,
+ fusionMode: "rrf",
+ rrfK: 60,
+ });
+ expect(ranked[0]!.chunkId).toBe(2);
+ expect(ranked[0]!.reasons.some((r) => r.startsWith("rrf:"))).toBe(true);
+ // The single-lane chunk 3 (semantic only, rank 2) fuses lowest.
+ expect(ranked[ranked.length - 1]!.chunkId).toBe(3);
+});
+
+test("linear mode (default) does NOT add an rrf reason", () => {
+ const ranked = rankResults(inputs(), {
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ limit: 10,
+ nowMs: NOW,
+ });
+ for (const r of ranked) {
+ expect(r.reasons.some((x) => x.startsWith("rrf:"))).toBe(false);
+ }
+});
+
+test("linear mode ranking is unchanged when fusionMode is omitted vs explicit linear", () => {
+ const base = inputs();
+ const omitted = rankResults(base, {
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ limit: 10,
+ nowMs: NOW,
+ });
+ const explicit = rankResults(base, {
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ limit: 10,
+ nowMs: NOW,
+ fusionMode: "linear",
+ });
+ expect(omitted.map((r) => r.chunkId)).toEqual(explicit.map((r) => r.chunkId));
+ expect(omitted.map((r) => r.score)).toEqual(explicit.map((r) => r.score));
+});
+
+// ── config parsing ───────────────────────────────────────────────────────────
+
+let tmp: string;
+let config: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-rrf-cfg-"));
+ config = join(tmp, "config.yaml");
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+test("search_fusion_mode defaults to linear with rrf_k 60", () => {
+ writeFileSync(config, `vault: "${tmp}"\n`);
+ const cfg = resolveSearchConfig({ vault: tmp, configPath: config });
+ expect(cfg.fusionMode).toBe("linear");
+ expect(cfg.rrfK).toBe(60);
+});
+
+test("search_fusion_mode parses rrf", () => {
+ writeFileSync(config, `vault: "${tmp}"\nsearch_fusion_mode: "rrf"\nsearch_rrf_k: "40"\n`);
+ const cfg = resolveSearchConfig({ vault: tmp, configPath: config });
+ expect(cfg.fusionMode).toBe("rrf");
+ expect(cfg.rrfK).toBe(40);
+});
+
+test("an invalid search_fusion_mode is rejected", () => {
+ writeFileSync(config, `vault: "${tmp}"\nsearch_fusion_mode: "weighted"\n`);
+ expect(() => resolveSearchConfig({ vault: tmp, configPath: config })).toThrow(SearchError);
+});
diff --git a/tests/core/search/search.test.ts b/tests/core/search/search.test.ts
index 8b1148a5..ff56b4d3 100644
--- a/tests/core/search/search.test.ts
+++ b/tests/core/search/search.test.ts
@@ -41,6 +41,7 @@ function semanticConfig() {
timeoutMs: 5_000,
concurrency: 2,
batchSize: 8,
+ costGateUsd: 0,
},
});
}
@@ -339,6 +340,7 @@ test("explicit --semantic without vec extension throws (configured-down case)",
timeoutMs: 5_000,
concurrency: 1,
batchSize: 8,
+ costGateUsd: 0,
},
});
await indexVault(cfg);
diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.ts
index 9d8a22ca..5cd98d32 100644
--- a/tests/core/search/store.test.ts
+++ b/tests/core/search/store.test.ts
@@ -25,6 +25,7 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCo
timeoutMs: 10_000,
concurrency: 4,
batchSize: 32,
+ costGateUsd: 0,
});
return Object.freeze({
vault: tmp,
@@ -34,6 +35,8 @@ function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchCo
chunkOverlap: 100,
keywordWeight: 0.6,
semanticWeight: 0.4,
+ fusionMode: "linear" as const,
+ rrfK: 60,
semantic,
recall: Object.freeze({
mmrLambda: 0.7,
diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.ts
index 1ca2825b..57c3de78 100644
--- a/tests/core/search/store.vec.test.ts
+++ b/tests/core/search/store.vec.test.ts
@@ -28,6 +28,7 @@ function semanticConfig(
timeoutMs: 10_000,
concurrency: 4,
batchSize: 32,
+ costGateUsd: 0,
});
return Object.freeze({
vault: tmp,
@@ -37,6 +38,8 @@ function semanticConfig(
chunkOverlap: 100,
keywordWeight: 0.6,
semanticWeight: 0.4,
+ fusionMode: "linear" as const,
+ rrfK: 60,
semantic,
recall: Object.freeze({
mmrLambda: 0.7,
diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.ts
index 07624870..2a12f094 100644
--- a/tests/helpers/search-fixtures.ts
+++ b/tests/helpers/search-fixtures.ts
@@ -65,6 +65,10 @@ export function makeConfig(opts: {
relationPolarityEnabled?: boolean;
/** Learned recall weights from feedback; defaults to false (opt-in). */
learnedWeightsEnabled?: boolean;
+ /** Rank-fusion mode; defaults to "linear". */
+ fusionMode?: "linear" | "rrf";
+ /** RRF damping constant; defaults to 60. */
+ rrfK?: number;
}): ResolvedSearchConfig {
const baseSemantic: ResolvedEmbeddingConfig = Object.freeze({
enabled: false,
@@ -76,6 +80,7 @@ export function makeConfig(opts: {
timeoutMs: 10_000,
concurrency: 4,
batchSize: 32,
+ costGateUsd: 0,
});
const semantic: ResolvedEmbeddingConfig = Object.freeze({
...baseSemantic,
@@ -93,6 +98,8 @@ export function makeConfig(opts: {
chunkOverlap: 100,
keywordWeight: 0.6,
semanticWeight: 0.4,
+ fusionMode: opts.fusionMode ?? "linear",
+ rrfK: opts.rrfK ?? 60,
semantic,
recall: Object.freeze({
mmrLambda: opts.mmrLambda ?? 0.7,