Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript

v0.10.0: Brain search β€” SQLite + FTS5 + optional semantic over the vault

Sol Aitken2ad0af506253
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index e8660784..247dba69 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "0.9.1",
+ "version": "0.10.0",
"description": "Plugin-first second brain package for AI agents and humans.",
"author": {
"name": "Open Second Brain contributors"
diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json
index 319bbf03..1f5f3ccc 100644
--- a/.codex-plugin/plugin.json
+++ b/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "0.9.1",
+ "version": "0.10.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"skills": "./skills",
"hooks": "./hooks/hooks.json",
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0d8caadc..52b0ffc6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,76 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+## [0.10.0] - 2026-05-16
+
+Full-text search over the vault as a deterministic, filesystem-first
+layer. Index lives at `<vault>/.open-second-brain/brain.sqlite`
+(SQLite + FTS5, schema versioned). Optional semantic layer via
+`sqlite-vec` plus any OpenAI-compatible `/v1/embeddings` provider
+(OpenRouter, OpenAI, Together, Google's OpenAI-compat endpoint,
+local Ollama, Hermes proxy). Closes design plan
+`docs/plans/2026-05-16-brain-search-design.md`.
+
+### Added
+
+- **Core module `src/core/search/`** with isolated walker, chunker,
+ store (the only SQL boundary), FTS query, links, ranker, indexer,
+ search, and embedding providers (`null-provider`, `openai-compat`).
+ Public surface: `resolveSearchConfig`, `indexVault`, `reindexVault`,
+ `indexStatus`, `indexCheck`, `search`, plus typed `SearchError`
+ codes.
+- **CLI** namespace `o2b search` with verbs `query` (default),
+ `index`, `reindex`, `status`, `check`. Human and `--json` output;
+ `--auto-refresh` for read-time incremental indexing.
+- **MCP tool** `brain_search` (read-only, agent-facing). Diagnostic
+ score components (`keywordScore`, `semanticScore`, `linkBoost`,
+ `recencyBoost`) are intentionally absent from the MCP shape; they
+ live in CLI `--verbose` only. Content per chunk is truncated to
+ 600 characters. Index-management verbs are NOT exposed over MCP
+ (operator business, never agent business β€” design Β§3 principle 5).
+- **`second_brain_status`** gains a `search.*` block: index path,
+ schema version, document/chunk/embedding counts, embedding model
+ and dimension, sqlite-vec status, key presence (redacted),
+ `last_indexed_at`, `last_full_index_at`. Reports
+ `{ exists: false, hint }` when the index has not been built yet.
+- **Ranking** combines min-max-normalised BM25, cosine similarity
+ on unit-normalised vectors, in-result wikilink boost (capped at
+ 0.03), shared-tag boost (capped at 0.02), and recency steps
+ (≀7d β†’ 0.05, ≀30d β†’ 0.025, ≀90d β†’ 0.01). Tie-break on equal
+ final score: keyword desc, mtime desc, chunk id asc.
+- **Atomic reindex** via `brain.sqlite.new` + same-directory rename
+ swap with `brain.sqlite.bak` retention. Auto-restore from `.bak`
+ on open if the main file is missing.
+- **Embedding-model fingerprint** stored in `index_state`. Changing
+ `embedding_model` or `embedding_dimension` drops `embeddings`,
+ `chunk_vec`, and `chunk_vec_map` on next open, logs one line, and
+ preserves `chunks` + `chunk_fts`. The next
+ `o2b search index --embeddings` repopulates vectors.
+- **Concurrency guard** via `proper-lockfile` on the index path:
+ three attempts, 1s backoff, then `INDEX_LOCKED`. Readers do not
+ take the lock; WAL handles concurrent reads safely.
+- **Semantic-unavailable policy** (design Β§7): implicit semantic
+ (config default) warns and falls back to keyword-only; explicit
+ `--semantic` / `semantic: true` throws a typed `SearchError` so
+ the failure cannot hide. Data-state cases (no embeddings yet)
+ warn and skip even when explicit β€” running
+ `o2b search index --embeddings` is the right answer there, not a
+ panic.
+
+### Changed
+
+- **`sqlite-vec`** added to `optionalDependencies`. The runtime
+ detects whether the loadable extension is present on disk and
+ records availability in `index_state.vec_extension_available`;
+ no failure if the platform package is missing.
+
+### Notes
+
+- v0.10.0 ships schema version 1 only. Future migrations follow the
+ same `MIGRATIONS[]` pattern in `src/core/search/schema.ts`.
+- `o2b index` (the Markdown index generator at `AI Wiki/index.md`)
+ is unchanged. The new system is `o2b search index`.
+
## [0.9.1] - 2026-05-15
Active-preferences digest, MCP Resources, and a visibility expansion
@@ -1081,6 +1151,11 @@ Hermes / Claude Code / Codex / OpenClaw configurations do not change.
- Sandbox vault and plugin manifest fixtures for tests.
- GitHub release workflow for tag-based and manually dispatched releases.
+[0.10.0]: https://github.com/itechmeat/open-second-brain/compare/v0.9.1...v0.10.0
+[0.9.1]: https://github.com/itechmeat/open-second-brain/compare/v0.9.0...v0.9.1
+[0.9.0]: https://github.com/itechmeat/open-second-brain/compare/v0.8.1...v0.9.0
+[0.8.1]: https://github.com/itechmeat/open-second-brain/compare/v0.8.0...v0.8.1
+[0.8.0]: https://github.com/itechmeat/open-second-brain/compare/v0.7.0...v0.8.0
[0.7.0]: https://github.com/itechmeat/open-second-brain/compare/v0.6.2...v0.7.0
[0.6.2]: https://github.com/itechmeat/open-second-brain/compare/v0.6.1...v0.6.2
[0.6.1]: https://github.com/itechmeat/open-second-brain/compare/v0.6.0...v0.6.1
diff --git a/README.md b/README.md
index f07e00e1..232f3016 100644
--- a/README.md
+++ b/README.md
@@ -13,6 +13,36 @@ and **not** an LLM-driven knowledge store β€” the `dream` consolidation
pass is pure deterministic counters. The plugin never writes hidden
state outside the configured vault and config directory.
+## How it works
+
+Three loops cooperate over plain Markdown:
+
+- **Capture.** Agents call `brain_feedback` to drop taste signals into
+ `Brain/inbox/`.
+- **Accretion.** A deterministic `dream` pass clusters repeat signals
+ into rules β€” counters and atomic file moves, no LLM.
+- **Application.** Agents log whether each rule was `applied` /
+ `violated` / `outdated`. `Brain/active.md` is auto-regenerated and
+ injected at session start; `brain_search` exposes full-text search
+ across the whole vault.
+
+```mermaid
+flowchart LR
+ Agent[Agent / human]
+ Agent -- brain_feedback --> Inbox[(Brain/inbox/)]
+ Inbox -. dream .-> Pref[(Brain/preferences/)]
+ Agent -- brain_apply_evidence --> Log[(Brain/log/)]
+ Log -. dream .-> Pref
+ Pref -. regenerate .-> Active[Brain/active.md]
+ Active -- SessionStart hook --> Agent
+ Vault[Vault Markdown] -- o2b search index --> FTS[(SQLite + FTS5<br/>brain_search)]
+ FTS --> Agent
+```
+
+Mechanics, dream pipeline, state diagram, snapshot model, hygiene
+lints, MCP resources, the v0.10.0 search layer, and the full set of
+safety invariants are in [`docs/how-it-works.md`](docs/how-it-works.md).
+
## Supported runtimes
| Runtime | Integration | Notes |
diff --git a/bun.lock b/bun.lock
index 0a7d3740..4f7c5812 100644
--- a/bun.lock
+++ b/bun.lock
@@ -13,6 +13,9 @@
"@types/proper-lockfile": "^4.1.4",
"typescript": "^5.5.0",
},
+ "optionalDependencies": {
+ "sqlite-vec": "^0.1.9",
+ },
},
},
"packages": {
@@ -34,6 +37,18 @@
"signal-exit": ["[email protected]", "", {}, "sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ=="],
+ "sqlite-vec": ["[email protected]", "", { "optionalDependencies": { "sqlite-vec-darwin-arm64": "0.1.9", "sqlite-vec-darwin-x64": "0.1.9", "sqlite-vec-linux-arm64": "0.1.9", "sqlite-vec-linux-x64": "0.1.9", "sqlite-vec-windows-x64": "0.1.9" } }, "sha512-L7XJWRIBNvR9O5+vh1FQ+IGkh/3D2AzVksW5gdtk28m78Hy8skFD0pqReKH1Yp0/BUKRGcffgKvyO/EON5JXpA=="],
+
+ "sqlite-vec-darwin-arm64": ["[email protected]", "", { "os": "darwin", "cpu": "arm64" }, "sha512-jSsZpE42OfBkGL/ItyJTVCUwl6o6Ka3U5rc4j+UBDIQzC1ulSSKMEhQLthsOnF/MdAf1MuAkYhkdKmmcjaIZQg=="],
+
+ "sqlite-vec-darwin-x64": ["[email protected]", "", { "os": "darwin", "cpu": "x64" }, "sha512-KDlVyqQT7pnOhU1ymB9gs7dMbSoVmKHitT+k1/xkjarcX8bBqPxWrGlK/R+C5WmWkfvWwyq5FfXfiBYCBs6PlA=="],
+
+ "sqlite-vec-linux-arm64": ["[email protected]", "", { "os": "linux", "cpu": "arm64" }, "sha512-5wXVJ9c9kR4CHm/wVqXb/R+XUHTdpZ4nWbPHlS+gc9qQFVHs92Km4bPnCKX4rtcPMzvNis+SIzMJR1SCEwpuUw=="],
+
+ "sqlite-vec-linux-x64": ["[email protected]", "", { "os": "linux", "cpu": "x64" }, "sha512-w3tCH8xK2finW8fQJ/m8uqKodXUZ9KAuAar2UIhz4BHILfpE0WM/MTGCRfa7RjYbrYim5Luk3guvMOGI7T7JQA=="],
+
+ "sqlite-vec-windows-x64": ["[email protected]", "", { "os": "win32", "cpu": "x64" }, "sha512-y3gEIyy/17bq2QFPQOWLE68TYWcRZkBQVA2XLrTPHNTOp55xJi/BBBmOm40tVMDMjtP+Elpk6UBUXdaq+46b0Q=="],
+
"typescript": ["[email protected]", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
"undici-types": ["[email protected]", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
diff --git a/docs/how-it-works.md b/docs/how-it-works.md
index 2b923b36..e6092411 100644
--- a/docs/how-it-works.md
+++ b/docs/how-it-works.md
@@ -23,25 +23,29 @@ no surprise, no hallucinated memory.
## Vault layout
The vault holds three top-level agent-facing directories. Brain owns
-its own.
+its own, plus a sibling derived-index directory.
```text
<vault>/
β”œβ”€β”€ Brain/ # observing memory (agent-writable)
β”‚ β”œβ”€β”€ _brain.yaml # schema, thresholds, retention
β”‚ β”œβ”€β”€ _BRAIN.md # operating manual for agents
+β”‚ β”œβ”€β”€ active.md # derived: confirmed + quarantine + recently retired
β”‚ β”œβ”€β”€ inbox/ # raw taste signals
β”‚ β”‚ β”œβ”€β”€ sig-<date>-<slug>.md
-β”‚ β”‚ └── processed/ # signals already folded into rules
+β”‚ β”‚ └── processed/ # signals already folded into rules
β”‚ β”œβ”€β”€ preferences/ # active rules
-β”‚ β”‚ └── pref-<slug>.md # status: unconfirmed | confirmed
+β”‚ β”‚ └── pref-<slug>.md # status: unconfirmed | confirmed | quarantine
β”‚ β”œβ”€β”€ retired/ # archived rules
-β”‚ β”‚ └── ret-<slug>.md # retired_reason: stale-no-evidence | expired-unconfirmed | rebutted | user-rejected
+β”‚ β”‚ └── ret-<slug>.md # retired_reason: stale-no-evidence | expired-unconfirmed | rebutted | user-rejected | quarantine-violated | superseded-by-context
β”‚ β”œβ”€β”€ log/ # daily event log
β”‚ β”‚ └── YYYY-MM-DD.md # append-only, typed events
β”‚ └── .snapshots/ # pre-dream snapshots
β”‚ └── dream-<run-id>.tar.zst
β”‚
+β”œβ”€β”€ .open-second-brain/ # derived search index (rebuildable)
+β”‚ └── brain.sqlite # SQLite + FTS5 + optional sqlite-vec
+β”‚
β”œβ”€β”€ AI Wiki/ # curated knowledge surface
β”‚ β”œβ”€β”€ identity/ # user.md, agents.md
β”‚ β”œβ”€β”€ index.md / hot.md
@@ -52,6 +56,11 @@ its own.
└── YYYY.MM.DD.md
```
+`Brain/active.md` and `.open-second-brain/brain.sqlite` are **derived** β€”
+both can be deleted and rebuilt at any time (`o2b brain dream` and
+`o2b search reindex` respectively). They are excluded from the agent's
+write contract.
+
```mermaid
flowchart LR
Agent[Agent / MCP client]
@@ -75,7 +84,7 @@ there through the `payment_*` tools).
## A preference's lifecycle
-A preference moves between four states from first signal to retirement:
+A preference moves between five states from first signal to retirement:
```mermaid
stateDiagram-v2
@@ -83,28 +92,49 @@ stateDiagram-v2
Inbox --> Unconfirmed: dream (β‰₯candidate_threshold<br/>same-sign signals on topic)
Unconfirmed --> Confirmed: brain_apply_evidence applied<br/>(first such event)
Unconfirmed --> Retired_Expired: dream (now > unconfirmed_until,<br/>no applied evidence)
+ Confirmed --> Quarantine: dream (violated_count β‰₯ applied_count<br/>AND applied_count > low_max_applied)
+ Quarantine --> Confirmed: dream (applied_count > violated_count)
+ Quarantine --> Retired_Quarantine: dream (one further violated)
Confirmed --> Retired_Stale: dream (last_evidence_at<br/>older than stale_evidence_days)
Confirmed --> Retired_Rebutted: dream (β‰₯candidate_threshold<br/>opposite-sign signals collected)
+ Confirmed --> Retired_Superseded: dream (one apply-evidence with<br/>result: outdated)
+ Quarantine --> Retired_Superseded: dream (one apply-evidence with<br/>result: outdated)
Confirmed --> Retired_UserRejected: o2b brain reject
+ Quarantine --> Retired_UserRejected: o2b brain reject
Unconfirmed --> Retired_UserRejected: o2b brain reject
Retired_Expired --> [*]
Retired_Stale --> [*]
Retired_Rebutted --> [*]
+ Retired_Quarantine --> [*]
+ Retired_Superseded --> [*]
Retired_UserRejected --> [*]
note right of Confirmed
Pinned preferences (pinned: true)
- skip all three automatic retire
- transitions. Only o2b brain reject
- can retire them.
+ skip the three "natural decay"
+ retire transitions (stale,
+ expired, rebutted) and the
+ quarantine downgrade β€” dream
+ emits retain-pinned instead.
+ A context-driven outdated still
+ retires a pinned pref; only
+ o2b brain reject and outdated
+ evidence can move it.
end note
```
-`Inbox` is not really a state of the preference β€” it's the staging
+`Inbox` is not really a state of the preference β€” it is the staging
area for the signals that will eventually create one. The first real
state is `Unconfirmed`: the rule exists but has not yet been applied
in real work.
+`Quarantine` (added in v0.9.1) is a probation state for confirmed rules
+whose recent evidence has turned dominantly negative without yet
+crossing the rebuttal threshold. A quarantined rule is still active β€”
+the agent reads it β€” but one further `violated` retires it with
+`quarantine-violated`, and recovery happens automatically when `applied`
+events overtake the violated count.
+
## End-to-end signal β†’ rule flow
A typical sequence from a user remark to a confirmed preference:
@@ -168,9 +198,9 @@ A single dream invocation is a deterministic pipeline:
flowchart TD
Start([o2b brain dream]) --> Load[Load _brain.yaml]
Load --> Scan[Scan inbox/, preferences/, retired/, log/]
- Scan --> Plan[Plan transitions:<br/>new unconfirmed, promotions,<br/>retires, signal moves]
+ Scan --> Plan[Plan transitions:<br/>new unconfirmed, promotions,<br/>quarantine in/out, retires, signal moves]
Plan --> Changed{Any changes planned?}
- Changed -- no --> NoOp([return changed: false<br/>no snapshot, no log])
+ Changed -- no --> NoOp([regenerate active.md<br/>return changed: false])
Changed -- yes --> Snap["createSnapshot to<br/>.snapshots/dream-RUNID.tar.zst"]
Snap --> SnapErr{Snapshot OK?}
SnapErr -- fail --> Abort([throw; no state changed])
@@ -184,23 +214,34 @@ flowchart TD
A3 -- yes --> A4[Promote to confirmed,<br/>stamp confirmed_at]
A3 -- no --> A5
A4 --> A5[Recompute confidence]
- A5 --> A6{Pinned?}
- A6 -- yes --> A7[Skip auto-retire,<br/>log retain-pinned]
- A6 -- no --> A8{Retire conditions?}
- A8 -- stale --> A9[Move to retired/<br/>reason: stale-no-evidence]
- A8 -- expired --> A10[Move to retired/<br/>reason: expired-unconfirmed]
- A8 -- rebutted --> A11[Move to retired/<br/>reason: rebutted]
- A8 -- none --> A12[Leave in preferences/]
- A9 --> A13
- A10 --> A13
- A11 --> A13
- A12 --> A13
- A7 --> A13
- A13[Move consumed signals to processed/]
+ A5 --> A6{outdated evidence?}
+ A6 -- yes --> A6r[Move to retired/<br/>reason: superseded-by-context]
+ A6 -- no --> A7{Pinned?}
+ A7 -- yes --> A7l[Skip auto-retire + quarantine,<br/>log retain-pinned]
+ A7 -- no --> A8{Quarantine conditions?}
+ A8 -- enter --> A8q[status: quarantine]
+ A8 -- exit --> A8c[status: confirmed]
+ A8 -- already-quarantine + violated --> A8r[Move to retired/<br/>reason: quarantine-violated]
+ A8 -- none --> A9{Retire conditions?}
+ A9 -- stale --> A10[Move to retired/<br/>reason: stale-no-evidence]
+ A9 -- expired --> A11[Move to retired/<br/>reason: expired-unconfirmed]
+ A9 -- rebutted --> A12[Move to retired/<br/>reason: rebutted]
+ A9 -- none --> A13[Leave in preferences/]
+ A6r --> A14
+ A7l --> A14
+ A8q --> A14
+ A8c --> A14
+ A8r --> A14
+ A10 --> A14
+ A11 --> A14
+ A12 --> A14
+ A13 --> A14
+ A14[Move consumed signals to processed/]
end
Apply --> LogEvent["Append dream event to log/today.md"]
- LogEvent --> Prune[pruneSnapshots to retention_count]
+ LogEvent --> ActMd[regenerate Brain/active.md]
+ ActMd --> Prune[pruneSnapshots to retention_count]
Prune --> Done([return changed: true with summary])
```
@@ -244,6 +285,42 @@ Defaults from `_brain.yaml`:
All four are tunable per vault in `Brain/_brain.yaml`.
+## Active preferences injection
+
+A confirmed rule is useless if the agent does not see it during work.
+v0.9.1 closes that gap with three cooperating surfaces, all derived
+from the same source of truth (`Brain/preferences/`):
+
+```mermaid
+flowchart LR
+ Dream[dream pass] -- regenerate --> Active[Brain/active.md]
+ Pin[o2b brain pin / unpin] -- regenerate --> Active
+ Active -- SessionStart hook --> ClaudeStart["Claude Code / Codex<br/>session start"]
+ Active -- PostCompact hook --> ClaudeCompact["Claude Code / Codex<br/>after /compact"]
+ Active -- "osb://preferences/active" --> MCPHost["MCP host (Hermes, others)"]
+```
+
+- **`Brain/active.md`** is a derived Markdown digest: confirmed
+ preferences (id, scope, confidence, principle), quarantined
+ preferences with their applied / violated counters, and the three
+ most recently retired entries. The writer is idempotent β€” if the
+ rendered body matches the file on disk, no I/O happens.
+- **SessionStart hook** (`startup | resume | clear`) and
+ **PostCompact hook** (`manual | auto`) inject the body as
+ `additionalContext` so the agent sees current rules at the start of
+ every session and again after `/compact` (otherwise the
+ hook-injected context is dropped from long sessions). Both fail
+ closed β€” any error path exits 0 with no output so the runtime
+ proceeds unaffected.
+- **MCP Resources** expose the same content for hosts that prefer
+ pull access (`osb://preferences/active` and friends in the table
+ above). The MCP `initialize` reply advertises the `resources`
+ capability so clients know to enumerate them.
+
+`active.md` is **not** edited by hand β€” `o2b brain dream` and pin /
+unpin re-derive it from `preferences/` and `retired/`. Deleting the
+file is harmless: the next dream regenerates it.
+
## CLI / MCP surface
The same operations are reachable through two channels. Read columns
@@ -252,20 +329,41 @@ are mirrored in MCP; destructive operations are CLI-only by design.
| Operation | CLI verb | MCP tool | Side effect |
|--------------------------|----------------------------|-----------------------|------------------------|
| Bootstrap layer | `o2b brain init` | β€” | creates `Brain/` skeleton |
-| Record taste signal | `o2b brain feedback` | `brain_feedback` | writes signal + log event |
-| Consolidation pass | `o2b brain dream` | `brain_dream` | mutates preferences/retired, atomic snapshot |
-| Record application | `o2b brain apply-evidence` | `brain_apply_evidence`| appends log event |
-| Render summary | `o2b brain digest` | `brain_digest` | read-only |
+| Record taste signal | `o2b brain feedback` | `brain_feedback` | writes signal + log event (sanitised) |
+| Consolidation pass | `o2b brain dream` | `brain_dream` | mutates preferences/retired, atomic snapshot, regenerates `Brain/active.md` |
+| Record application | `o2b brain apply-evidence` | `brain_apply_evidence`| appends log event; `result ∈ {applied, violated, outdated}`; artifact supports `[[file:120-145]]` ranges |
+| Render summary | `o2b brain digest` | `brain_digest` | read-only; includes `top_applied` and `top_referenced` sections |
| Inspect state | `o2b brain query` | `brain_query` | read-only |
-| Validate invariants | `o2b brain doctor` | `brain_doctor` | read-only |
+| Computed backlinks | `o2b brain backlinks <id>` | `brain_backlinks` | read-only; inverted reference map across `preferences/`, `retired/`, `log/` |
+| Validate invariants | `o2b brain doctor` | `brain_doctor` | read-only; six lint rules |
+| Full-text search | `o2b search "<query>"` | `brain_search` | read-only; FTS5 + optional semantic |
+| Manage search index | `o2b search index \| reindex \| status \| check` | β€” (CLI-only) | builds / inspects `<vault>/.open-second-brain/brain.sqlite` |
+| Operational snapshot | `o2b status` | `second_brain_status` | read-only; `brain.*` + `search.*` blocks |
| Retire manually | `o2b brain reject` | β€” (CLI-only) | moves pref β†’ retired/ |
-| Toggle pin | `o2b brain pin / unpin` | β€” (CLI-only) | flips `pinned` field |
+| Toggle pin | `o2b brain pin / unpin` | β€” (CLI-only) | flips `pinned` field; regenerates `Brain/active.md` |
| Restore snapshot | `o2b brain rollback` | β€” (CLI-only) | overwrites Brain/ from snapshot |
Operations that change the **protected set** (`pin`, `unpin`,
-`reject`, `rollback`) are kept off the MCP surface so an autonomous
-agent cannot quietly alter what is shielded from automatic retire or
-roll back state.
+`reject`, `rollback`) and the **index lifecycle** (`search index`,
+`reindex`, `check`) are kept off the MCP surface β€” protected-set
+moves so an autonomous agent cannot quietly alter what is shielded
+from automatic retire, and index management because it is operator
+business, never agent business.
+
+In addition to the tools above, MCP hosts can pull structured Brain
+content as **resources** (added in v0.9.1):
+
+| Resource URI | Body |
+|-----------------------------------|---------------------------------------------------|
+| `osb://preferences/active` | `Brain/active.md` (auto-regenerated on first read if missing) |
+| `osb://digest/latest` | most recent `brain_digest` rendering |
+| `osb://status` | Markdown form of the `second_brain_status.brain` block |
+| `osb://preference/{id}` | one `pref-` / `ret-` file |
+| `osb://topic/{slug}` | every signal + active/retired pref for a topic |
+| `osb://log/{date}` | one day's `Brain/log/YYYY-MM-DD.md` |
+| `osb://backlinks/{id}` | inverted reference map for `<id>` |
+
+Resources are pure read; mutating verbs stay on `tools/call`.
## Snapshots and rollback
@@ -289,6 +387,45 @@ A snapshot captures every file under `Brain/` **except** `.snapshots/`
itself β€” otherwise rollback would erase any snapshots taken after
this one. Retention defaults to ten newest archives.
+## Hygiene: sanitisation and lints
+
+Two cheap, deterministic layers keep the data clean.
+
+**Input sanitisation** (v0.9.1). Every Brain writer routes text fields
+through `src/core/redactor.ts` (promoted from Pay Memory) plus a
+`normaliseTextField` / `sanitiseTextField` pair:
+
+- C0 control characters are stripped (except `\t` and `\n`); `U+2028`
+ and `U+2029` collapse to `\n`; text is NFC-normalised.
+- Length caps per field: `principle` ≀ 512 (single-line), `scope` ≀
+ 128, `raw` ≀ 4096, `source[]` items ≀ 512, `artifact` ≀ 512
+ (single-line), `note` ≀ 4096.
+- Secret-shaped substrings (`api_key=…`, `token: …`, `bearer …`,
+ `authorization: …`, `password=…`, `private_key=…`, etc.) are masked
+ with `***REDACTED***`. The `raw` field is the only verbatim-quote
+ surface and is intentionally **not** redacted β€” it carries the
+ original user phrasing.
+- Inputs that sanitise down to empty fall into the existing "missing
+ field" error branch, so the writer never persists a placeholder
+ signal.
+
+**Doctor lints** (`o2b brain doctor`, `brain_doctor` MCP). All six are
+pure functions over the on-disk Brain state β€” no LLM, no network:
+
+| Code | Triggers when |
+|---|---|
+| `broken-backlinks` | a `[[pref-…]]` / `[[ret-…]]` / `[[sig-…]]` reference exists in `preferences/`, `retired/`, `inbox/`, or `log/`, but the target file does not |
+| `duplicate-preferences` | pairwise Jaccard β‰₯ 0.7 on principle tokens within the same `(topic, scope)` bucket |
+| `low-evidence-confirmed` | `status: confirmed` with `applied_count ≀ low_max_applied` AND `confirmed_at` older than `unconfirmed_window_days` |
+| `pinned-without-recent-evidence` | `pinned: true` with no evidence or evidence older than `stale_evidence_days` |
+| `malformed-evidence-range` | `apply-evidence` `artifact` uses `[[file:…]]` range syntax but fails validation (`:abc-def`, `:120-100`, bare `:`) |
+| `orphan-evidence` | `apply-evidence` `artifact` wikilink does not resolve to any file in the vault |
+
+With `--strict`, warnings demote `ok` to `false` so CI can gate on
+hygiene. The lints are read-only β€” `brain_doctor --fix` is explicitly
+out of scope because auto-modifying state runs against the
+"explicit-driven" invariant.
+
```mermaid
flowchart LR
BrainRoot[Brain/]
@@ -335,6 +472,89 @@ graph LR
- **OpenClaw** runs tools natively in the plugin's Node.js process
(no subprocess, by security-scanner requirement).
+## Full-text search
+
+v0.10.0 adds a deterministic search layer over the entire vault β€” not
+just `Brain/`. The index lives at `<vault>/.open-second-brain/brain.sqlite`
+(SQLite + FTS5) and is fully rebuildable from the Markdown files.
+Semantic search is optional and pluggable: `sqlite-vec` + any
+OpenAI-compatible `/v1/embeddings` endpoint when configured.
+
+```mermaid
+flowchart LR
+ subgraph Build [o2b search index]
+ direction LR
+ Walker[walker<br/>vault-relative paths] --> Chunker[chunker<br/>structural split + token packer]
+ Chunker --> Store[(SQLite store<br/>documents / chunks / links / embeddings)]
+ Chunker --> Links[extract wikilinks /<br/>markdown links / tags]
+ Links --> Store
+ Store --> FTS[(chunk_fts<br/>FTS5 BM25)]
+ Store -.optional.-> Vec[(chunk_vec<br/>sqlite-vec)]
+ Texts[chunk content] -. on --embeddings .-> Provider[OpenAI-compat<br/>embeddings provider]
+ Provider --> Vec
+ end
+
+ subgraph Query [search]
+ direction LR
+ Q["o2b search '<q>'<br/>brain_search"] --> KW[FTS5 keyword<br/>limit Γ— 3 candidates]
+ Q -. semantic on .-> SemQ[provider.embed query]
+ SemQ --> SemS[vec_search<br/>limit Γ— 5 candidates]
+ KW --> Ranker[ranker<br/>min-max BM25 + cosine<br/>+ link boost + recency boost]
+ SemS --> Ranker
+ Ranker --> Results[ranked results]
+ end
+```
+
+Key behaviours, all driven from `Brain/_brain.yaml`-free `search_*` /
+`embedding_*` config keys:
+
+- **Chunker.** Two passes: a structural split (headings, fenced code,
+ lists, tables, frontmatter), then a token-budget packer with
+ overlap. YAML frontmatter becomes synthetic `chunk_index: 0` so
+ `tags:` and other front-matter fields are searchable through FTS5
+ alongside the body.
+- **Store boundary.** `src/core/search/store.ts` is the single SQL
+ home; every other module talks to it through a typed surface. WAL
+ mode for concurrent reads, `proper-lockfile` on the index path for
+ writer exclusivity (three attempts, 1 s backoff, then
+ `INDEX_LOCKED`).
+- **Ranking.** `final_score = clamp01(keyword_weightΒ·norm_BM25 +
+ semantic_weightΒ·cosine + link_boost + recency_boost)`. BM25 is
+ min-max-normalised within the candidate set; cosine is
+ `1 - L2Β² / 2` on unit-normalised vectors; link boost rewards
+ candidates that other candidates reference via `[[wikilink]]` /
+ markdown link (capped at 0.03) or share a tag with (capped at
+ 0.02); recency is a step function on `mtime`.
+- **Semantic policy.** Implicit semantic (config default) warns and
+ falls back to keyword-only when sqlite-vec is unavailable, the key
+ is missing, the provider is down, or no embeddings exist yet.
+ Explicit `--semantic` / `semantic: true` on infrastructure failure
+ throws a typed `SearchError` so a misconfigured run cannot hide.
+ The data-state case (zero embeddings) always warns and skips β€”
+ running `o2b search index --embeddings` is the right answer there.
+- **Atomic reindex.** `o2b search reindex` writes to
+ `brain.sqlite.new`, renames to `brain.sqlite`, and keeps the
+ previous file as `brain.sqlite.bak`. If the main file is missing on
+ open and a `.bak` exists, it is auto-restored with a stderr notice.
+- **Embedding model fingerprint.** Model + dimension are recorded in
+ the index. Changing either drops the `embeddings` and `chunk_vec`
+ tables on next open, logs one line, and preserves `chunks` and
+ `chunk_fts`; the next `o2b search index --embeddings` repopulates
+ vectors.
+
+The MCP `brain_search` tool returns at most 50 results with each
+chunk's `content` truncated to 600 characters; diagnostic score
+components (`keywordScore`, `semanticScore`, `linkBoost`,
+`recencyBoost`) are intentionally absent from the MCP shape β€” they
+live in the CLI's `--verbose` output only, to keep the agent context
+small. Index-management verbs (`index`, `reindex`, `check`) are
+CLI-only.
+
+Full design and migration notes:
+[`docs/plans/2026-05-16-brain-search-design.md`](plans/2026-05-16-brain-search-design.md)
+and the matching implementation plan
+[`docs/plans/2026-05-16-brain-search-impl.md`](plans/2026-05-16-brain-search-impl.md).
+
## Safety properties
These are invariants of the system, not configuration to enable.
@@ -433,9 +653,15 @@ re-renders it from the current template.
- **`docs/architecture.md`** β€” layered system architecture beyond
Brain (vault model, runtime adapters, configuration model).
-- **`docs/plans/2026-05-15-brain-observing-memory.md`** β€” the design
- document, including the full file-format specs and test strategy.
+- **`docs/plans/2026-05-15-brain-observing-memory.md`** β€” the Brain
+ design document, including the full file-format specs and test
+ strategy.
- **`docs/plans/2026-05-15-brain-roadmap.md`** β€” trigger-based roadmap
of future capabilities (`BRAIN-FUT-NNN` entries).
+- **`docs/plans/2026-05-16-brain-search-design.md`** and
+ **`docs/plans/2026-05-16-brain-search-impl.md`** β€” design and
+ implementation plan for the v0.10.0 full-text search layer.
+- **`docs/mcp.md`** β€” protocol, schemas, lifecycle, and resource
+ surface of the MCP server.
- **`Brain/_BRAIN.md`** (in any initialised vault) β€” the operating
manual for agents working with that specific vault.
diff --git a/docs/plans/2026-05-16-brain-search-design.md b/docs/plans/2026-05-16-brain-search-design.md
new file mode 100644
index 00000000..f8564cbe
--- /dev/null
+++ b/docs/plans/2026-05-16-brain-search-design.md
@@ -0,0 +1,973 @@
+# Brain Search β€” Full-text Index over the Vault
+
+Status: design
+Target: Open Second Brain v0.10.0
+Authors: Sergey Eroshenkov (product), Claude (drafting)
+
+## 1. Overview
+
+Open Second Brain stores every durable artifact as Markdown in an
+Obsidian-compatible vault. The Brain layer adds preferences, signals, and a
+daily log; Pay Memory adds an auditable trail of paid actions; AI Wiki and
+Daily areas accumulate notes that may be either agent- or user-authored. Until
+now, the only way to find anything inside the vault has been to read the file
+the caller already names: `brain_query` resolves a known slug, `vault_health`
+walks paths, the Read tool requires an absolute path. There is no lookup by
+content.
+
+This document specifies **Brain Search** β€” a deterministic, filesystem-first
+search layer over the entire vault. Keyword retrieval is the contract; a
+semantic layer is optional and pluggable. The implementation must hold to the
+same invariants as the rest of the project: no daemon, no external service is
+required to run search, Markdown files remain the source of truth, the index
+is a fully rebuildable derived artifact.
+
+This plan covers Tier-S item Β§2 of `Projects/OpenSecondBrain/Features/_summary`
+and supersedes the more sketch-like `Projects/OpenSecondBrain/Features/search`
+note. Items Β§9 (inline `@osb` markers), Β§12 (merge suggestions), Β§14 (HTML
+explorer) are explicitly **not** included.
+
+## 2. Scope of v0.10.0
+
+In scope:
+
+- New module `src/core/search/` with isolated responsibilities (walker,
+ chunker, store, FTS, links, ranker, embedding providers).
+- New CLI namespace `o2b search` with verbs `query` (default) / `index` /
+ `reindex` / `status` / `check`.
+- New MCP tool `brain_search` (read-only).
+- Extension of the existing `second_brain_status` MCP tool with a
+ `search.*` block.
+- SQLite index at `<vault>/.open-second-brain/brain.sqlite` (overridable),
+ WAL mode, schema versioned through `index_state.schema_version`.
+- Optional semantic layer through `sqlite-vec` and any OpenAI-compatible
+ `/v1/embeddings` endpoint (OpenRouter, OpenAI, Google's OpenAI-compat
+ endpoint, Together, local Ollama, Hermes proxy when authenticated).
+- Vault-wide indexing with a configurable ignore list (defaults: `.git`,
+ `node_modules`, `.open-second-brain`, `.obsidian/cache`, `.trash`,
+ `.stversions`).
+- Incremental reindexing on file modification and removal.
+
+Out of scope:
+
+- Inline `@osb` marker parsing.
+- Merge-suggestion detection in `brain_dream` or `brain_digest`.
+- HTML/graph explorer.
+- Long-running file watcher / daemon.
+- LLM-based answer synthesis.
+- Multi-vault aggregation.
+- A non-OpenAI-shape embedding provider (e.g. Google's native
+ `:embedContent` endpoint). Users route Google through OpenRouter or
+ Google's OpenAI-compatible endpoint.
+
+## 3. Architectural Principles
+
+1. **Filesystem-first.** Markdown is the source of truth. The SQLite index
+ is recoverable: drop the file, run `o2b search reindex`, recover.
+2. **Single SQL boundary.** Only `src/core/search/store.ts` imports
+ `bun:sqlite`. Every other module talks to the store through a typed
+ surface. This keeps backend substitution feasible.
+3. **No daemon, no watcher.** Index is updated by explicit commands. An
+ opt-in `--auto-refresh` flag on `o2b search query` runs incremental
+ indexing once before the read.
+4. **Optional semantic, never silent.** Semantic search is off unless the
+ user enables it. When enabled but unavailable (extension missing,
+ provider unreachable, no compatible embeddings), the system emits an
+ explicit `warnings` entry rather than degrading silently.
+5. **Read/write asymmetry on MCP.** The MCP surface only exposes
+ `brain_search` and a status enrichment. Index management is operator
+ business, never agent business.
+6. **No prompt-shaped fallbacks.** A misconfigured provider or a missing
+ key produces a typed `SearchError`, not a heuristic re-route.
+
+## 4. Module Layout
+
+```
+src/core/search/
+ types.ts // BrainSearchResult, IndexStats, SearchError, IndexCheckReport
+ paths.ts // resolveIndexPath(vault, override?)
+ store.ts // bun:sqlite wrapper; only file holding SQL
+ schema.ts // DDL + MIGRATIONS[]
+ walker.ts // vault traversal with ignore-paths
+ chunker.ts // Markdown β†’ chunks { content, start_line, end_line }
+ indexer.ts // indexVault({...}) coordinating walker/chunker/store/embeddings
+ fts.ts // FTS5 query construction and BM25 retrieval
+ links.ts // wikilink / markdown-link / tag extraction
+ ranker.ts // pure: merge keyword + semantic + boosts
+ embeddings/
+ provider.ts // EmbeddingProvider interface, makeProvider(config)
+ openai-compat.ts // OpenAICompatProvider
+ null-provider.ts // NullProvider (raises EmbeddingDisabledError)
+ index.ts // public re-exports for CLI and MCP
+
+src/cli/search.ts // verb dispatcher: query | index | reindex | status | check
+src/mcp/search-tools.ts // brain_search tool, second_brain_status enrichment
+```
+
+Dependency direction is strictly downward:
+
+```
+cli/mcp β†’ indexer/search β†’ fts/links/ranker/embeddings β†’ store β†’ schema
+```
+
+There are no cycles. `embeddings/*` does not import `store` β€” embeddings are
+computed outside and passed in to `store.vecUpsert(...)`.
+
+### Invariants
+
+- `store.ts` is the only module importing `bun:sqlite`.
+- `NullProvider.embed()` throws `SearchError("EMBEDDING_DISABLED", ...)` β€”
+ not a no-op β€” so a caller that reaches it has a configuration bug.
+- Search reads chunk content from SQLite, not from disk. The disk is only
+ read by the indexer when refreshing.
+- The existing `o2b index` (Markdown index generator at
+ `AI Wiki/index.md`) is not modified.
+
+## 5. SQLite Schema
+
+Location: `<vault>/.open-second-brain/brain.sqlite`. Overridable through
+`search_db_path` in config and `--db` CLI flag. Pragmas applied on open:
+`journal_mode=WAL`, `foreign_keys=ON`, `synchronous=NORMAL`.
+
+```sql
+CREATE TABLE documents (
+ id INTEGER PRIMARY KEY,
+ path TEXT NOT NULL UNIQUE,
+ title TEXT,
+ content_hash TEXT NOT NULL,
+ mtime INTEGER NOT NULL,
+ size INTEGER NOT NULL,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ indexed_at TEXT NOT NULL
+);
+CREATE INDEX idx_documents_path ON documents(path);
+CREATE INDEX idx_documents_mtime ON documents(mtime);
+
+CREATE TABLE chunks (
+ id INTEGER PRIMARY KEY,
+ document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
+ chunk_index INTEGER NOT NULL,
+ content TEXT NOT NULL,
+ content_hash TEXT NOT NULL,
+ start_line INTEGER NOT NULL,
+ end_line INTEGER NOT NULL,
+ token_count INTEGER NOT NULL,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ UNIQUE(document_id, chunk_index)
+);
+CREATE INDEX idx_chunks_document ON chunks(document_id);
+
+CREATE VIRTUAL TABLE chunk_fts USING fts5(
+ content,
+ content='chunks',
+ content_rowid='id',
+ tokenize='unicode61 remove_diacritics 2'
+);
+-- Triggers chunks_ai / chunks_ad / chunks_au keep chunk_fts in sync.
+-- FTS holds only `content` (the single source column on `chunks`). Title
+-- and path are not separate FTS columns; they reach the result set via
+-- JOIN to `documents` at query time, and title text remains searchable
+-- because Β§6's synthetic frontmatter chunk embeds it as part of the
+-- chunk body.
+
+CREATE TABLE embeddings (
+ chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,
+ model TEXT NOT NULL,
+ dimension INTEGER NOT NULL,
+ embedding_hash TEXT NOT NULL,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL
+);
+CREATE INDEX idx_embeddings_model ON embeddings(model);
+
+-- Only when sqlite-vec is loaded:
+CREATE VIRTUAL TABLE chunk_vec USING vec0(embedding float[<DIM>]);
+CREATE TABLE chunk_vec_map (
+ chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,
+ vec_rowid INTEGER NOT NULL UNIQUE
+);
+
+CREATE TABLE links (
+ id INTEGER PRIMARY KEY,
+ source_document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
+ source_chunk_id INTEGER REFERENCES chunks(id) ON DELETE CASCADE,
+ target_path TEXT,
+ target_document_id INTEGER REFERENCES documents(id) ON DELETE SET NULL,
+ link_text TEXT,
+ link_type TEXT NOT NULL CHECK(link_type IN ('wikilink','markdown_link','tag')),
+ created_at TEXT NOT NULL
+);
+CREATE INDEX idx_links_source ON links(source_document_id);
+CREATE INDEX idx_links_target_doc ON links(target_document_id);
+CREATE INDEX idx_links_target_path ON links(target_path);
+
+CREATE TABLE index_state (
+ key TEXT PRIMARY KEY,
+ value TEXT NOT NULL,
+ updated_at TEXT NOT NULL
+);
+-- Known keys: schema_version, embedding_model, embedding_dimension,
+-- last_full_index_at, vec_extension_available.
+```
+
+### Decisions
+
+- `documents.path` is vault-relative with POSIX separators. The walker
+ normalises `\` to `/` on upsert. This keeps the schema portable across
+ Syncthing peers.
+- `content_hash` exists on two levels (document and chunk). File hash is
+ the fast-path skip for incremental reindex; chunk hash is the skip key
+ for recomputing embeddings.
+- FTS5 uses `content='chunks'` (external content) so the chunk text is
+ not duplicated. Standard `_ai/_ad/_au` triggers keep the FTS table in
+ sync.
+- Tokeniser is `unicode61 remove_diacritics 2`. This covers Latin and
+ Cyrillic; stemming is not enabled because it would introduce a
+ dependency and break determinism across machines.
+- `sqlite-vec` is loaded via `Database.loadExtension(...)`. If it
+ succeeds, the virtual table `chunk_vec` and its map are created.
+ Dimension is fixed at first index and recorded in `index_state`.
+ Changing the embedding model or dimension drops `embeddings`,
+ `chunk_vec`, and `chunk_vec_map`; `chunks` and `chunk_fts` are
+ preserved.
+- **`chunk_vec` is a virtual `vec0` table; SQLite foreign-key cascade
+ does not reach it.** Only `chunk_vec_map` cascades from `chunks`.
+ Every `store` mutation that deletes chunks therefore takes the
+ explicit two-step shape: first
+ `DELETE FROM chunk_vec WHERE rowid IN (SELECT vec_rowid FROM chunk_vec_map WHERE chunk_id IN (...))`,
+ then delete from `chunks` (which cascades to `chunk_vec_map` and via
+ triggers removes the FTS rows). This contract is enforced by
+ `store.deleteChunks(ids)` / `store.replaceChunks(documentId, ...)` /
+ `store.deleteDocument(path)`; no caller talks to `chunk_vec`
+ directly. `tests/core/search/store.vec.test.ts` verifies that
+ deleting a document leaves zero rows in `chunk_vec` and
+ `chunk_vec_map`.
+- WAL mode lets readers (`o2b search query`) run while a writer
+ (`o2b search index`) holds an exclusive lock. The writer lock is
+ enforced through `proper-lockfile` on `brain.sqlite.lock`.
+
+## 6. Chunking
+
+Two passes.
+
+**Pass 1 β€” structural split.** A single linewise traversal of the file
+produces blocks. The block boundaries are:
+
+- ATX heading (`^#{1,6}\s`) β€” starts a new block.
+- Blank line separating non-empty content β€” paragraph boundary.
+- Fenced code (`` ``` `` or `~~~`) β€” atomic block. An unclosed fence runs
+ to EOF.
+- YAML frontmatter (`---` … `---` at start of file) β€” atomic block.
+- Bulleted or numbered list of one nesting level β€” atomic until first
+ non-list line.
+- Markdown table (line with `|` followed by a `|---` separator) β€” atomic
+ until blank line.
+
+**Pass 2 β€” pack blocks into chunks.** The packer accumulates blocks while
+`tokens(chunk) + tokens(next_block) ≀ max_tokens`. Atomic blocks larger
+than `max_tokens` become single chunks unmodified. A heading-only block
+starts a new chunk only if the current chunk has a non-heading and
+`tokens(chunk) β‰₯ min_tokens`; otherwise the heading attaches to the
+current chunk. Each chunk begins with an overlap of the last
+`overlap_tokens` lines from the previous chunk; `start_line`/`end_line`
+are recorded from the first non-overlap line so line ranges remain
+truthful.
+
+Parameters for v0.10.0:
+
+| Parameter | Value | Rationale |
+| ----------------- | ----: | ------------------------------------ |
+| `max_tokens` | 800 | mid of 500–1000 recommendation |
+| `min_tokens` | 100 | smaller chunks return uninformative hits |
+| `overlap_tokens` | 100 | ~12.5% β€” preserves context cheaply |
+
+Tokens are approximated by whitespace-separated word count
+(`text.split(/\s+/).filter(Boolean).length`). Deterministic, dependency-free.
+
+### Title resolution
+
+First non-empty wins: `title:` in YAML frontmatter β†’ first H1 in body β†’
+filename with `-`/`_` replaced by spaces.
+
+### Frontmatter as synthetic chunk
+
+If YAML frontmatter exists and is non-empty, it becomes `chunk_index=0`
+with the raw frontmatter text as content. This makes `tags:` and other
+frontmatter fields searchable through FTS5. Files without frontmatter
+start at `chunk_index=0` from their first structural chunk. Malformed
+frontmatter (unterminated block, broken syntax) does not abort the
+file: it is omitted from chunking, the file is otherwise indexed
+normally, and a warning is recorded in `IndexStats.errors` for the
+operator to act on.
+
+### Edge cases
+
+- Empty file / whitespace-only: `documents` row is written for tracking,
+ zero `chunks`.
+- Non-UTF8 bytes: read fails fatally, file is logged as an error in
+ `IndexStats.errors`, no `documents` row is written.
+- Symlinks: followed only when the target is inside the vault (verified
+ via `realpath`). External symlinks are skipped with a warning.
+
+## 7. Ranking
+
+Final score:
+
+```
+final_score = clamp01(
+ keyword_weight * keyword_score
+ + semantic_weight * semantic_score
+ + link_boost
+ + recency_boost
+)
+```
+
+Defaults: `keyword_weight = 0.6`, `semantic_weight = 0.4`. When semantic
+is disabled the weights collapse to `1.0 / 0.0`. CLI/MCP/config may
+override; the ranker validates that the sum is `≀ 1.0`.
+
+**keyword_score** is min-max normalised BM25 within the query's result
+set: `keyword_score = (bm25_raw - min) / (max - min)` over the top-K
+keyword candidates; if `max == min`, every candidate gets `1.0`. This
+removes the dependency on absolute BM25 magnitudes (which depend on
+corpus size).
+
+**semantic_score** is cosine similarity, computed via L2 on
+unit-normalised vectors. Vectors are normalised to unit length at write
+time, so `cosine_similarity = 1 - (l2_distanceΒ² / 2)`, mapped to
+`[0, 1]` with `max(0, similarity)`.
+
+**link_boost** ∈ `[0, 0.05]`. Per chunk: `+0.02` if another result in
+the same query references it via `[[wikilink]]` (cap `0.03`); `+0.01` if
+it shares a tag with another result (cap `0.02`).
+
+**recency_boost** ∈ `[0, 0.05]`. Step function on `mtime`:
+≀ 7d β†’ `0.05`, ≀ 30d β†’ `0.025`, ≀ 90d β†’ `0.01`, older β†’ `0`.
+
+### Candidate sets
+
+Keyword candidates: top `K_kw = limit * 3`. Semantic candidates: top
+`K_sem = max(limit * 5, 50)`. The two sets are unioned by `chunk_id`;
+absent scores are zero on the other side. Tie-break order on equal
+`final_score`: `keyword_score desc`, `mtime desc`, `chunk_id asc`.
+
+### `searchType`
+
+Returned as diagnostic only:
+
+- `"hybrid"` if the chunk came from both sets.
+- `"keyword"` if only FTS5.
+- `"semantic"` if only sqlite-vec.
+
+### Semantic-unavailable behaviour
+
+A semantic request is **explicit** when the caller actively asks for
+semantic search: CLI `--semantic`, MCP `semantic: true`. Anything else
+(`--keyword-only`, MCP `semantic: false`, the implicit default that
+honours `search_semantic_enabled` from config) is **implicit**.
+
+The rule:
+
+- **Implicit + anything missing β†’ warn + keyword-only fallback, exit 0.**
+ The caller did not pin the search to semantic; degrading is safe.
+- **Explicit + infrastructure missing β†’ fail hard.** sqlite-vec did not
+ load, or the embedding provider returned an error / timed out: the
+ ranker cannot honour the request, so it surfaces a typed
+ `SearchError`. CLI exit 1, MCP `INTERNAL_ERROR`.
+- **Explicit + data state recoverable β†’ warn + keyword-only fallback,
+ exit 0.** The index simply has no embeddings yet for the current
+ model. No infrastructure is broken; the user needs
+ `o2b search index --embeddings`, not a panic.
+
+Concretely:
+
+| Trigger | Implicit | Explicit |
+|---|---|---|
+| `search_semantic_enabled=false` | keyword-only, no warning | n/a (config gate doesn't apply when explicit) |
+| sqlite-vec failed to load | warn `"sqlite-vec unavailable, semantic disabled this session"`, exit 0 | `VEC_EXTENSION_UNAVAILABLE`, exit 1 |
+| Embedding provider unreachable / timeout | warn `"embedding provider unavailable: <reason>"`, exit 0 | `EMBEDDING_PROVIDER_HTTP` / `_TIMEOUT`, exit 1 |
+| Embedding key missing | warn `"embedding key not configured; semantic disabled"`, exit 0 | `EMBEDDING_KEY_MISSING`, exit 1 |
+| Index has no compatible embeddings | warn `"no compatible embeddings; run: o2b search index --embeddings"`, exit 0 | same warning, exit 0 (data state, not infra) |
+
+## 8. CLI
+
+All verbs sit under `o2b search`. The default verb is `query`
+(positional argument is the query string).
+
+### `o2b search <query>`
+
+Flags: `--vault`, `--db`, `--limit` (1..100, default 10), `--semantic`,
+`--keyword-only`, `--path` (prefix filter), `--keyword-weight`,
+`--semantic-weight`, `--auto-refresh`, `--json`, `--verbose`.
+
+Human format:
+
+```
+[1] Brain/preferences/pref-russian-replies.md β€’ 0.81
+ line 12-38 β€’ hybrid β€’ mtime 2026-05-14
+ ΠŸΡ€Π°Π²ΠΈΠ»Π° общСния: всСгда ΠΎΡ‚Π²Π΅Ρ‡Π°Ρ‚ΡŒ ΠΏΠΎ-русски, ΠΊΡ€ΠΎΠΌΠ΅ ΠΊΠΎΠ³Π΄Π° Ρ€Π΅Ρ‡ΡŒ о…
+```
+
+JSON format: object
+`{ results: BrainSearchResult[]; warnings: string[]; total: number }`.
+`results` is the array of ranked hits, `warnings` is empty when nothing
+to report, `total` is `min(matched, limit)`.
+
+Exit codes: `0` always for success including zero results, `1` for runtime
+errors, `2` for bad flags.
+
+### `o2b search index`
+
+Incremental indexing. Flags: `--vault`, `--db`, `--embeddings`,
+`--force` (recompute everything, do not delete file), `--concurrency`
+(1..16, default 4), `--verbose`, `--json`.
+
+The writer holds an exclusive lock through `proper-lockfile`. A
+concurrent run fails fast with `INDEX_LOCKED`.
+
+Output (human):
+
+```
+indexing vault: /root/vault
+ added: 23 files, 187 chunks
+ updated: 4 files, 31 chunks
+ unchanged: 412 files
+ deleted: 2 files (chunks purged)
+ embeddings: 218/218 OK (4 retries)
+done in 12.4s
+```
+
+JSON: `{stats: {...}, errors: [...], duration_ms}`.
+
+### `o2b search reindex`
+
+Full rebuild. Writes to `brain.sqlite.new`, then `rename(.new, .)`
+atomically, then the old file is preserved as `.bak` until the next
+successful reindex. On startup, an orphaned `.bak` is restored if the
+main file is missing or unreadable.
+
+### `o2b search status`
+
+Read-only summary. Does not take the writer lock. Human and `--json`
+forms. Reports index path, schema version, document/chunk/embedding
+counts, embedding model and dimension, sqlite-vec status, embedding key
+presence (not value), `last_indexed_at`, `last_full_index_at`,
+`stale_embeddings` count, and warnings. With no index: prints
+`index: not initialised. Run: o2b search index` and exits 0.
+
+### `o2b search check`
+
+Pre-flight diagnostic. Verifies vault readable, index directory writable,
+`bun:sqlite` open and FTS5 supported, sqlite-vec loadable (if semantic
+enabled), embedding key resolvable (config or env), and β€” if key is
+present β€” performs a single `embed(["check"])` probe with a 5-second
+timeout. Exits 0 unless a hard precondition fails (1/2/3); semantic
+issues are reported as warnings, exit 0.
+
+### Parity table
+
+| Verb | CLI | MCP |
+| -------- | ------------------------------------ | ------------------------- |
+| query | `o2b search "..."` | `brain_search` |
+| index | `o2b search index [--embeddings]` | β€” |
+| reindex | `o2b search reindex` | β€” |
+| status | `o2b search status` | `second_brain_status.search` |
+| check | `o2b search check` | β€” |
+
+Mutating and HTTP-spending verbs are operator-only.
+
+## 9. MCP
+
+### `brain_search`
+
+Input schema:
+
+```json
+{
+ "type": "object",
+ "properties": {
+ "query": { "type": "string", "minLength": 1, "maxLength": 2000 },
+ "limit": { "type": "integer", "minimum": 1, "maximum": 50, "default": 10 },
+ "semantic": { "type": "boolean", "default": null },
+ "keyword_only": { "type": "boolean", "default": false },
+ "path_prefix": { "type": "string", "maxLength": 256 }
+ },
+ "required": ["query"],
+ "additionalProperties": false
+}
+```
+
+`semantic: null` uses the config default; `true` or `false` overrides for
+the call. `path_prefix` is passed through `ensureInsideVault` to reject
+escape attempts.
+
+Output:
+
+```json
+{
+ "results": [
+ {
+ "path": "Brain/preferences/pref-russian-replies.md",
+ "title": "Russian replies",
+ "content": "...",
+ "score": 0.81,
+ "startLine": 12,
+ "endLine": 38,
+ "searchType": "hybrid"
+ }
+ ],
+ "warnings": [],
+ "total": 1
+}
+```
+
+`content` is truncated to 600 characters per chunk with `…` suffix on
+overflow. The diagnostic score components (`keywordScore`,
+`semanticScore`, `linkBoost`, `recencyBoost`) are intentionally absent β€”
+they are noise for the agent and live in CLI `--verbose` only.
+
+`limit` is capped at 50 (not 100 as for CLI) to keep the agent context
+small.
+
+### Error mapping
+
+| Situation | MCP code | Message |
+| ------------------------------------------ | ----------------- | -------------------------------------------------------------------- |
+| missing/empty `query` | `INVALID_PARAMS` | `missing required argument: query` |
+| `path_prefix` escapes vault | `INVALID_PARAMS` | `path_prefix escapes vault` |
+| index file missing | `INTERNAL_ERROR` | `search index not initialised. Run: o2b search index` |
+| index file unreadable | `INTERNAL_ERROR` | `search index unreadable: <reason>` |
+| explicit `semantic=true` without vec | `INTERNAL_ERROR` | `semantic search unavailable: sqlite-vec extension not loaded` |
+| explicit `semantic=true`, key missing | `INTERNAL_ERROR` | `embedding key not configured` |
+| explicit `semantic=true`, provider down | `INTERNAL_ERROR` | `embedding provider unavailable: <reason>` |
+| operation > 10s | `INTERNAL_ERROR` | `search timeout after 10000ms` |
+
+Implicit `semantic: null` never produces an error from any of the
+infrastructure rows above β€” those become entries in the `warnings`
+array per Β§7's table.
+
+### `second_brain_status` enrichment
+
+The existing tool gains a `search` block when `search_enabled !== false`:
+
+```json
+{
+ "search": {
+ "index_path": "/root/vault/.open-second-brain/brain.sqlite",
+ "exists": true,
+ "schema_version": 1,
+ "documents": 441,
+ "chunks": 1827,
+ "embeddings": 1827,
+ "stale_embeddings": 0,
+ "embedding_model": "google/gemini-embedding-2-preview",
+ "embedding_dimension": 768,
+ "vec_extension": "loaded",
+ "semantic_enabled": true,
+ "embedding_key_present": true,
+ "last_indexed_at": "2026-05-16T14:22:17Z",
+ "last_full_index_at": "2026-05-10T09:01:03Z"
+ }
+}
+```
+
+`embedding_key_present` is a boolean; the key itself is redacted by the
+existing `redactMapping` helper. When the index does not exist, the
+block is `{ "exists": false, "hint": "run: o2b search index" }`.
+
+## 10. Configuration
+
+The existing config parser handles flat `key: value` YAML only. All new
+keys use prefixes `search_*` and `embedding_*`:
+
+```yaml
+# Existing
+vault: "/root/vault"
+timezone: "Europe/Belgrade"
+agent_name: "@claude-vps-agent"
+
+# New (all optional)
+search_enabled: "true"
+search_db_path: "" # blank β†’ default <vault>/.open-second-brain/brain.sqlite
+search_chunk_size: "800"
+search_chunk_overlap: "100"
+search_keyword_weight: "0.6"
+search_semantic_weight: "0.4"
+search_ignore_paths: ".git,node_modules,.open-second-brain,.obsidian/cache,.trash,.stversions"
+
+search_semantic_enabled: "false"
+embedding_provider: "openai-compat"
+embedding_base_url: "https://openrouter.ai/api/v1"
+embedding_model: "google/gemini-embedding-2-preview"
+embedding_api_key: "" # blank β†’ check env; if env empty, semantic disabled
+embedding_dimension: "" # blank β†’ autodetect from first response
+embedding_timeout_ms: "10000"
+embedding_concurrency: "4"
+embedding_batch_size: "32"
+```
+
+ENV resolution order is `env β†’ config β†’ default` per key:
+
+| Config key | ENV | Default |
+| ----------------------- | -------------------------------------------- | ------------------------------------------------ |
+| `search_db_path` | `OPEN_SECOND_BRAIN_SEARCH_DB` | `<vault>/.open-second-brain/brain.sqlite` |
+| `search_semantic_enabled` | `OPEN_SECOND_BRAIN_SEARCH_SEMANTIC` | `false` |
+| `embedding_provider` | `OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER` | `openai-compat` |
+| `embedding_base_url` | `OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL` | β€” (required if semantic on) |
+| `embedding_model` | `OPEN_SECOND_BRAIN_EMBEDDING_MODEL` | β€” (required if semantic on) |
+| `embedding_api_key` | `OPEN_SECOND_BRAIN_EMBEDDING_KEY` | β€” (required if semantic on) |
+| `embedding_dimension` | `OPEN_SECOND_BRAIN_EMBEDDING_DIM` | autodetect |
+
+`search_ignore_paths` is comma-separated, exact directory-name match
+(not glob). Numeric and boolean fields are parsed with explicit
+validation; invalid values cause `INVALID_INPUT` (CLI exit 2) rather
+than a silent fallback.
+
+`embedding_api_key` falls under the existing `SECRET_KEY_PARTS = ["key",
+"token", ...]` heuristic and is redacted to `[REDACTED]` by
+`redactMapping`. The exact name is also added to the redactor's list as
+belt-and-braces.
+
+### `o2b init` instruction block
+
+When `search_semantic_enabled` is `true` (in config or env) but no key
+is resolvable, and when the user runs `o2b init`, the bootstrap output
+appends:
+
+```
+Search is enabled (keyword-only).
+
+To enable semantic search, set the following (config file path is
+printed below; alternatively use the environment variables):
+
+ search_semantic_enabled: "true"
+ embedding_base_url: "https://openrouter.ai/api/v1"
+ embedding_model: "google/gemini-embedding-2-preview"
+ embedding_api_key: "<your key>"
+
+Config file: ~/.config/open-second-brain/config.yaml
+Or via env: OPEN_SECOND_BRAIN_SEARCH_SEMANTIC=true
+ OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL=...
+ OPEN_SECOND_BRAIN_EMBEDDING_MODEL=...
+ OPEN_SECOND_BRAIN_EMBEDDING_KEY=...
+
+Then:
+ o2b search check
+ o2b search index --embeddings
+```
+
+The block prints once, only during `o2b init`. No nagging on every CLI
+invocation. `o2b search check` is the dedicated diagnostic command.
+
+## 11. Embedding Provider
+
+```ts
+export interface EmbeddingProvider {
+ readonly name: string;
+ readonly model: string;
+ readonly dimension: number | null;
+ embed(texts: readonly string[]): Promise<number[][]>;
+ ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }>;
+}
+
+export function makeProvider(config: ResolvedEmbeddingConfig): EmbeddingProvider;
+```
+
+Two concrete implementations:
+
+- `OpenAICompatProvider` for `embedding_provider: openai-compat`. Posts
+ `{ model, input: texts, encoding_format: "float" }` to
+ `${base_url}/embeddings` with `Authorization: Bearer <api_key>`.
+ Batches `texts` into groups of `embedding_batch_size`. Concurrency is
+ limited by a semaphore of width `embedding_concurrency`. Vectors are
+ unit-normalised before being returned (so cosine works through L2 on
+ `chunk_vec`). Retries on `429/500/502/503/504` and network errors
+ with exponential backoff `1s/2s/4s + jitter`, three attempts total;
+ other 4xx fail fast.
+- `NullProvider` for `embedding_provider: disabled` and the implicit
+ case when semantic is off. Its `embed()` throws
+ `SearchError("EMBEDDING_DISABLED", ...)`; this exists to surface
+ configuration bugs rather than silently no-op.
+
+`makeProvider` throws `SearchError("INVALID_INPUT", ...)` synchronously
+for an unknown `embedding_provider` value β€” eagerly at construction
+time, not lazily on the first `embed()` call.
+
+The single OpenAI-compatible class is intentional. OpenRouter, OpenAI,
+Together, Mistral, Google's OpenAI-compat endpoint, local Ollama
+(`http://localhost:11434/v1`), and a future authenticated Hermes proxy
+all conform to this shape. Adding a Google-native `:embedContent`
+adapter is deferred until a concrete user need surfaces; users can
+already reach Google models through OpenRouter.
+
+## 12. Public API
+
+All public functions take a fully-resolved `ResolvedSearchConfig`.
+Resolution happens once at the CLI/MCP boundary through
+`resolveSearchConfig({...})`; the search module itself never reads
+env or `config.yaml`. This keeps CLI / MCP / tests on a single
+resolution path and isolates the search module from process-wide
+state.
+
+```ts
+// types.ts
+export interface ResolvedSearchConfig {
+ readonly vault: string;
+ readonly dbPath: string;
+ readonly ignorePaths: ReadonlyArray<string>;
+ readonly chunkSize: number;
+ readonly chunkOverlap: number;
+ readonly keywordWeight: number;
+ readonly semanticWeight: number;
+ readonly semantic: {
+ readonly enabled: boolean;
+ readonly provider: "openai-compat" | "disabled";
+ readonly baseUrl: string | null;
+ readonly model: string | null;
+ readonly apiKey: string | null; // never logged; redactMapping handles display
+ readonly dimension: number | null; // null = autodetect at first call
+ readonly timeoutMs: number;
+ readonly concurrency: number;
+ readonly batchSize: number;
+ };
+}
+
+// index.ts
+export function resolveSearchConfig(opts: {
+ vault?: string;
+ configPath?: string;
+ overrides?: Partial<ResolvedSearchConfig>; // CLI flag overrides go here
+}): ResolvedSearchConfig;
+
+export async function indexVault(
+ config: ResolvedSearchConfig,
+ opts?: {
+ embeddings?: boolean;
+ force?: boolean;
+ onFile?: (event: {
+ path: string;
+ kind: "added" | "updated" | "unchanged" | "deleted" | "error";
+ message?: string;
+ }) => void;
+ },
+): Promise<IndexStats>;
+
+export async function reindexVault(
+ config: ResolvedSearchConfig,
+ opts?: { embeddings?: boolean },
+): Promise<IndexStats>;
+
+export async function search(
+ config: ResolvedSearchConfig,
+ opts: SearchOptions,
+): Promise<SearchOutcome>;
+
+export async function indexStatus(
+ config: ResolvedSearchConfig,
+): Promise<IndexStatusSnapshot>;
+
+export async function indexCheck(
+ config: ResolvedSearchConfig,
+): Promise<IndexCheckReport>;
+```
+
+`resolveSearchConfig` implements the env β†’ config β†’ default chain
+documented in Β§10 and validates numeric/boolean shapes; an invalid
+field raises `SearchError("INVALID_INPUT", ...)`. `overrides` is
+how `o2b search query --keyword-weight 0.8` reaches the ranker
+without redundant flag-parsing inside the search module.
+
+Result types β€” `BrainSearchResult`, `IndexStats`, `IndexStatusSnapshot`,
+`IndexCheckReport`, `SearchOutcome` β€” are exported from `types.ts` and
+returned `Object.freeze`-d. The `onFile` callback is the sole progress
+channel; CLI consumes it for `--verbose` output, tests intercept it for
+assertions.
+
+## 13. Migrations and Recovery
+
+Schema version is tracked in `index_state.schema_version`. Migrations
+are an array of `{ version, up(db) }` functions applied sequentially in
+a transaction. v0.10.0 ships `version 1` only (the initial schema).
+Future minor versions add migrations; major schema changes that cannot
+be migrated raise `SCHEMA_MISMATCH` with a `o2b search reindex` hint.
+
+Embedding-model or dimension changes do not go through the migration
+list. On open, if `index_state.embedding_model` or `embedding_dimension`
+mismatches the resolved config, the store drops `embeddings`,
+`chunk_vec`, `chunk_vec_map`, logs one line `embedding model changed
+from X to Y, embeddings cleared`, and updates `index_state`. `chunks`
+and `chunk_fts` are preserved. The next `o2b search index --embeddings`
+repopulates vectors.
+
+`o2b search reindex` writes to `brain.sqlite.new`, atomically renames to
+`brain.sqlite`, and keeps the previous file as `brain.sqlite.bak` until
+the next successful reindex. If `brain.sqlite` is missing or unreadable
+at open time and a `.bak` exists, the `.bak` is restored automatically
+with a stderr notice.
+
+## 14. Errors
+
+```ts
+export class SearchError extends Error {
+ readonly code: SearchErrorCode;
+}
+
+export type SearchErrorCode =
+ | "INDEX_MISSING"
+ | "INDEX_UNREADABLE"
+ | "SCHEMA_MISMATCH"
+ | "VEC_EXTENSION_UNAVAILABLE"
+ | "EMBEDDING_DISABLED"
+ | "EMBEDDING_KEY_MISSING"
+ | "EMBEDDING_PROVIDER_HTTP"
+ | "EMBEDDING_PROVIDER_TIMEOUT"
+ | "EMBEDDING_DIMENSION_MISMATCH"
+ | "INDEX_LOCKED"
+ | "INVALID_INPUT";
+```
+
+CLI mapping: `INVALID_INPUT β†’ exit 2`, everything else β†’ exit 1.
+
+MCP mapping: `INVALID_INPUT β†’ INVALID_PARAMS`, everything else β†’
+`INTERNAL_ERROR`.
+
+Non-`SearchError` exceptions (sqlite-vec DDL crashes, malformed
+extensions) bubble up wrapped as `INDEX_UNREADABLE` with the original
+message preserved.
+
+## 15. Concurrency and Locking
+
+- Writers (`index`, `reindex`) hold an exclusive `proper-lockfile` on
+ `brain.sqlite.lock`. Retry policy: three attempts, 1s backoff. After
+ exhaustion: `INDEX_LOCKED`.
+- Readers do not take the lock; WAL is sufficient.
+- A single search invocation uses one connection inside a `BEGIN`/
+ `COMMIT` pair so keyword and semantic queries see a consistent
+ snapshot.
+
+## 16. Idempotency
+
+- `o2b search index` repeated with no changes: `unchanged: N`,
+ `added/updated/deleted = 0`, `embeddings_computed = 0`.
+- `o2b search reindex` repeated: both runs succeed; the second `.bak`
+ replaces the first.
+- `documents.path` is unique; double-indexing a file replaces its row
+ rather than duplicating.
+
+## 17. Testing
+
+Test framework: `bun test` (already the project standard). Helpers:
+
+```
+tests/helpers/
+ search-fixtures.ts // build temporary vaults with fixed content
+ mock-embedding.ts // MockEmbeddingProvider β€” deterministic sha256-derived vectors
+ fake-http.ts // local OpenAI-compatible server for provider tests
+```
+
+Test files:
+
+| File | Coverage |
+| --------------------------------------------- | ------------------------------------------------------------------------------------- |
+| `tests/core/search/walker.test.ts` | ignore list, symlinks, hidden dirs, non-md files |
+| `tests/core/search/chunker.test.ts` | empty file, single line, code fence > max, frontmatter + H1, heading without body, lists, tables, UTF-8 Cyrillic |
+| `tests/core/search/store.test.ts` | open/migrate, upsert/replace/delete, FTS triggers, schema-mismatch error, INDEX_LOCKED |
+| `tests/core/search/store.vec.test.ts` | when sqlite-vec loads: round-trip; when not: graceful skip |
+| `tests/core/search/fts.test.ts` | BM25 ordering, path-prefix filter, Cyrillic, escape of query special chars |
+| `tests/core/search/links.test.ts` | wikilink, markdown link, tag; resolve `target_document_id` when target is indexed |
+| `tests/core/search/ranker.test.ts` | union, weights, clamp, link/recency boost, tie-break, keyword-only fallback |
+| `tests/core/search/indexer.test.ts` | E2E: index β†’ incremental skip β†’ modify β†’ delete β†’ reindex |
+| `tests/core/search/indexer.embeddings.test.ts`| `onFile` events, embeddings computed for new/updated, not for unchanged, model-change clears embeddings |
+| `tests/core/search/embeddings.test.ts` | `OpenAICompatProvider` against fake-http: batching, sort by `index`, retry, timeout, dim-mismatch, 4xx fail-fast |
+| `tests/core/search/search.test.ts` | keyword / semantic / hybrid / keyword-only / path-filter; warning when vec unavailable |
+| `tests/cli/search.test.ts` | every verb; flag parsing; exit codes 0/1/2; JSON output |
+| `tests/mcp/search.test.ts` | `brain_search` happy path; `INVALID_PARAMS` cases; `INTERNAL_ERROR` cases; warning case |
+
+### Performance bench (informative)
+
+`tests/core/search/bench.test.ts` is opt-in (`bun test --filter="bench"`).
+Baselines, captured in this document on first run:
+
+- Indexing 1000 synthetic Markdown files: under 30s without embeddings;
+ under 5 minutes with embeddings (network-bound).
+- Keyword search across five representative queries: average under
+ 100 ms.
+- Hybrid search with a mocked vector store: average under 200 ms.
+
+CI does not run the bench.
+
+## 18. Acceptance Criteria
+
+A reviewer marks the PR ready to merge when all 20 items below are
+verified, either by an automated test or by a documented manual check:
+
+1. `o2b search index` creates `<vault>/.open-second-brain/brain.sqlite`
+ with all tables and `schema_version=1`.
+2. A second `o2b search index` with no file changes reports
+ `unchanged: N`, `added/updated/deleted = 0`,
+ `embeddings_computed = 0`.
+3. Modifying a file's content and `mtime` produces an `updated` entry,
+ deletes the old chunks, and indexes the new chunks.
+4. Removing a file from disk produces a `deleted` entry on the next
+ `index`; queries no longer return its content.
+5. `o2b search "<Cyrillic>"` returns hits against a Cyrillic-content
+ vault.
+6. `o2b search reindex` rebuilds atomically; if killed mid-run, the next
+ open restores from `.bak`.
+7. `o2b search status` without an index prints `not initialised`; with
+ an index prints accurate counts.
+8. `o2b search check` without `embedding_api_key` prints MISSING and the
+ key-setup instructions (exit 0); with an unreachable provider it
+ prints the provider error (exit 0).
+9. `o2b search index --embeddings` without a resolvable key fails with
+ `EMBEDDING_KEY_MISSING` (exit 1).
+10. With a working provider (`fake-http` in the test), `o2b search
+ index --embeddings` writes vectors into `chunk_vec` and populates
+ `embeddings.embedding_hash` correctly.
+11. `o2b search "..." --semantic` with sqlite-vec loaded but empty
+ `embeddings` returns a keyword-only result plus the
+ `no compatible embeddings` warning, exit 0 (data-state case,
+ treated identically for implicit and explicit per Β§7).
+12. Implicit semantic (no `--semantic`, semantic enabled in config)
+ with sqlite-vec unloaded β†’ keyword-only result + warning, exit 0.
+ Explicit `o2b search "..." --semantic` with sqlite-vec unloaded
+ β†’ exit 1, error code `VEC_EXTENSION_UNAVAILABLE`. Explicit
+ `--semantic` with provider unreachable β†’ exit 1, error code
+ `EMBEDDING_PROVIDER_HTTP` or `_TIMEOUT`.
+13. Changing `embedding_model` in config drops `embeddings`,
+ `chunk_vec`, `chunk_vec_map`; logs one line; preserves FTS5 and
+ `chunks`; next `index --embeddings` repopulates.
+14. MCP `brain_search` returns results with `content ≀ 600` chars,
+ without diagnostic score components, with `limit ≀ 50`.
+15. Two concurrent `o2b search index` processes: the first succeeds,
+ the second exits with `INDEX_LOCKED`.
+16. The existing `o2b index` (Markdown index generator) continues to
+ work unchanged; its tests remain green.
+17. MCP `second_brain_status` exposes a `search.*` block with accurate
+ counters.
+18. Performance benchmark (informative, not a merge gate): keyword
+ search across 1000 chunks below 100 ms; hybrid below 500 ms
+ excluding the query-embedding round trip.
+19. All new TypeScript files pass `tsc --noEmit` with the project's
+ strict configuration.
+20. `bun test` is green end to end; existing Brain, Pay Memory, and CLI
+ suites do not regress.
+
+## 19. Versioning and CHANGELOG
+
+Release: `0.10.0` (minor bump β€” new feature, backwards compatible). One
+CHANGELOG entry under `## 0.10.0`:
+
+```
+### Added
+- Brain search: SQLite + FTS5 index over the entire vault with optional
+ semantic layer via sqlite-vec and any OpenAI-compatible embeddings
+ provider. CLI verbs `o2b search query|index|reindex|status|check`,
+ MCP tool `brain_search`. See docs/plans/2026-05-16-brain-search-design.md.
+```
+
+No `[Unreleased]` section (project convention). The version is updated
+in `package.json` and `pyproject.toml` as part of the merge through
+`bun run sync-version`.
diff --git a/docs/plans/2026-05-16-brain-search-impl.md b/docs/plans/2026-05-16-brain-search-impl.md
new file mode 100644
index 00000000..2ae64325
--- /dev/null
+++ b/docs/plans/2026-05-16-brain-search-impl.md
@@ -0,0 +1,656 @@
+# Brain Search Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Ship a deterministic full-text search over the vault with optional semantic layer, exposed via CLI (`o2b search`) and MCP (`brain_search`).
+
+**Architecture:** New isolated module `src/core/search/` owns walker, chunker, SQLite store (single SQL boundary), FTS query, links, ranker, embedding providers. CLI and MCP consume a resolved config plus the public API. Keyword search is always available; semantic search loads `sqlite-vec` and any OpenAI-compatible `/v1/embeddings` provider when configured.
+
+**Tech Stack:** TypeScript on Bun, `bun:sqlite` (FTS5 built-in), `sqlite-vec` extension (optional), `proper-lockfile`, OpenAI-compatible HTTP embeddings.
+
+**Source of truth for behaviour:** [`docs/plans/2026-05-16-brain-search-design.md`](./2026-05-16-brain-search-design.md). Every task here implements a slice of that spec β€” if a conflict appears, the spec wins and the plan must be updated.
+
+---
+
+## Plan-wide conventions
+
+These apply to every task; do not re-state them per step.
+
+- **Imports.** Production code uses `node:`-prefixed Node builtins (`node:fs`, `node:path`) and `bun:sqlite`. Tests use `import { test, expect, describe, beforeEach, afterEach } from "bun:test"`. Path: always `.ts` extensions in imports (project convention, `allowImportingTsExtensions: true`).
+- **Result shape.** Every public-API return value is `Object.freeze`-d at the call site that produces it. This mirrors `src/core/brain/query.ts`.
+- **Errors.** All user-facing errors from this module are instances of `SearchError` (defined in Task 1). Non-`SearchError` exceptions bubble up as-is β€” do not wrap them.
+- **No git from this plan.** The user does git work themselves. Each task ends with **Pause for review (no commit)** instead of `git add`/`git commit`. The reviewer either runs `bun test` themselves to confirm green, or relies on the in-task `bun test ...` output.
+- **No bait fallbacks.** "Silent semantic downgrade" is allowed only on the implicit code path (config-default semantic). On the explicit code path (CLI `--semantic`, MCP `semantic: true`) the implementation MUST throw `SearchError` and let CLI/MCP translate to the right exit code.
+- **Atomic writes** for any file touched outside SQLite go through `src/core/fs-atomic.ts:atomicWriteFileSync`. SQLite files are mutated through prepared statements inside `BEGIN`/`COMMIT`; the temp-file rename pattern is reserved for `o2b search reindex` (Task 24).
+- **Verification.** Every task ends with `bun test tests/path/to/file.test.ts` and an expected pass count. If the suite has > 10 tests, expect `(pass N)` exactly. CI green is `bun test` across the repo β€” confirmed at the end of each Phase.
+
+## File map
+
+Create:
+
+```
+src/core/search/types.ts β€” SearchError, BrainSearchResult, IndexStats, IndexStatusSnapshot, IndexCheckReport, SearchOutcome, ResolvedSearchConfig, SearchOptions
+src/core/search/paths.ts β€” resolveIndexPath(config) β†’ string
+src/core/search/schema.ts β€” DDL, MIGRATIONS[], applyV1
+src/core/search/store.ts β€” Store class wrapping bun:sqlite (only SQL home)
+src/core/search/walker.ts β€” walkVault(config) async-iterates *.md paths
+src/core/search/chunker.ts β€” chunkMarkdown(text) β†’ BlockChunk[]
+src/core/search/links.ts β€” extractLinks(content) β†’ ExtractedLink[]
+src/core/search/fts.ts β€” runFtsQuery(store, query, opts) β†’ FtsHit[]
+src/core/search/ranker.ts β€” mergeResults(kw, sem, opts) β†’ BrainSearchResult[]
+src/core/search/indexer.ts β€” indexVault(config, opts), reindexVault, etc.
+src/core/search/embeddings/provider.ts β€” EmbeddingProvider interface + makeProvider()
+src/core/search/embeddings/null-provider.ts β€” NullProvider
+src/core/search/embeddings/openai-compat.ts β€” OpenAICompatProvider
+src/core/search/index.ts β€” Public re-export surface for CLI + MCP
+src/cli/search.ts β€” `o2b search` verb dispatcher
+src/mcp/search-tools.ts β€” brain_search MCP tool + search status enrichment
+tests/helpers/search-fixtures.ts β€” createTempVault(name, files), writeMd helper
+tests/helpers/mock-embedding.ts β€” MockEmbeddingProvider (deterministic sha256 vectors)
+tests/helpers/fake-http.ts β€” startFakeOpenAi() returns {url, close, setResponse}
+tests/core/search/types.test.ts
+tests/core/search/paths.test.ts
+tests/core/search/config.test.ts
+tests/core/search/schema.test.ts
+tests/core/search/store.test.ts
+tests/core/search/store.vec.test.ts
+tests/core/search/walker.test.ts
+tests/core/search/chunker.test.ts
+tests/core/search/links.test.ts
+tests/core/search/fts.test.ts
+tests/core/search/ranker.test.ts
+tests/core/search/indexer.test.ts
+tests/core/search/indexer.embeddings.test.ts
+tests/core/search/embeddings.test.ts
+tests/core/search/search.test.ts
+tests/cli/search.test.ts
+tests/mcp/search.test.ts
+```
+
+Modify:
+
+```
+src/core/config.ts β€” extend SECRET_KEY_PARTS list to include 'embedding_api_key' explicitly (belt-and-braces); add config keys to discovery types if needed.
+src/core/init.ts β€” print semantic-instructions block after vault bootstrap (Task 52).
+src/cli/main.ts β€” register 'search' verb in dispatcher (Task 30).
+src/mcp/tools.ts β€” register brain_search + extend second_brain_status payload (Tasks 32, 33).
+package.json β€” add `sqlite-vec` to optionalDependencies; bump version 0.9.1 β†’ 0.10.0 (Task 53).
+pyproject.toml β€” bump version 0.9.1 β†’ 0.10.0 via `bun run sync-version` (Task 53).
+CHANGELOG.md β€” `## 0.10.0` entry (Task 53).
+```
+
+---
+
+## Phase 1 β€” Foundation
+
+Three small modules that everything else depends on. No SQLite yet.
+
+### Task 1: Types and SearchError
+
+**Files:**
+- Create: `src/core/search/types.ts`
+- Test: `tests/core/search/types.test.ts`
+
+- [ ] **Step 1: Write the failing test for SearchError shape**
+
+`tests/core/search/types.test.ts`:
+
+```ts
+import { test, expect } from "bun:test";
+import { SearchError } from "../../../src/core/search/types.ts";
+
+test("SearchError carries a typed code", () => {
+ const err = new SearchError("INDEX_MISSING", "no index at /tmp/x");
+ expect(err).toBeInstanceOf(Error);
+ expect(err.name).toBe("SearchError");
+ expect(err.code).toBe("INDEX_MISSING");
+ expect(err.message).toBe("no index at /tmp/x");
+});
+
+test("SearchError preserves stack trace", () => {
+ const err = new SearchError("INVALID_INPUT", "bad");
+ expect(err.stack).toContain("SearchError");
+});
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `bun test tests/core/search/types.test.ts`
+Expected: FAIL with "Cannot find module" or similar.
+
+- [ ] **Step 3: Implement types.ts**
+
+`src/core/search/types.ts`:
+
+```ts
+/**
+ * Public types for `src/core/search/*`. Plain data β€” no behaviour, no I/O.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§12, Β§14.
+ */
+
+export const SEARCH_ERROR_CODES = [
+ "INDEX_MISSING",
+ "INDEX_UNREADABLE",
+ "SCHEMA_MISMATCH",
+ "VEC_EXTENSION_UNAVAILABLE",
+ "EMBEDDING_DISABLED",
+ "EMBEDDING_KEY_MISSING",
+ "EMBEDDING_PROVIDER_HTTP",
+ "EMBEDDING_PROVIDER_TIMEOUT",
+ "EMBEDDING_DIMENSION_MISMATCH",
+ "INDEX_LOCKED",
+ "INVALID_INPUT",
+] as const;
+export type SearchErrorCode = (typeof SEARCH_ERROR_CODES)[number];
+
+export class SearchError extends Error {
+ readonly code: SearchErrorCode;
+ constructor(code: SearchErrorCode, message: string) {
+ super(message);
+ this.name = "SearchError";
+ this.code = code;
+ }
+}
+
+export interface BrainSearchResult {
+ readonly documentId: number;
+ readonly chunkId: number;
+ readonly path: string;
+ readonly title: string | null;
+ readonly content: string;
+ readonly startLine: number;
+ readonly endLine: number;
+ readonly score: number;
+ readonly keywordScore: number;
+ readonly semanticScore: number;
+ readonly linkBoost: number;
+ readonly recencyBoost: number;
+ readonly searchType: "keyword" | "semantic" | "hybrid";
+}
+
+export interface IndexStats {
+ readonly added: number;
+ readonly updated: number;
+ readonly unchanged: number;
+ readonly deleted: number;
+ readonly chunksTotal: number;
+ readonly embeddingsComputed: number;
+ readonly embeddingsRetries: number;
+ readonly errors: ReadonlyArray<{ readonly path: string; readonly message: string }>;
+ readonly durationMs: number;
+}
+
+export interface IndexStatusSnapshot {
+ readonly indexPath: string;
+ readonly exists: boolean;
+ readonly schemaVersion: number | null;
+ readonly documents: number;
+ readonly chunks: number;
+ readonly embeddings: number;
+ readonly staleEmbeddings: number;
+ readonly embeddingModel: string | null;
+ readonly embeddingDimension: number | null;
+ readonly vecExtension: "loaded" | "unavailable" | "unknown";
+ readonly semanticEnabled: boolean;
+ readonly embeddingKeyPresent: boolean;
+ readonly lastIndexedAt: string | null;
+ readonly lastFullIndexAt: string | null;
+ readonly warnings: ReadonlyArray<string>;
+}
+
+export interface IndexCheckReport {
+ readonly vaultReadable: boolean;
+ readonly indexDirWritable: boolean;
+ readonly sqliteOk: boolean;
+ readonly fts5Ok: boolean;
+ readonly vecExtension: "loaded" | "unavailable" | "not-attempted";
+ readonly embeddingKeyResolved: boolean;
+ readonly providerReachable: boolean | null; // null = not probed
+ readonly providerReason: string | null;
+ readonly warnings: ReadonlyArray<string>;
+ readonly fatal: ReadonlyArray<string>;
+}
+
+export interface SearchOptions {
+ readonly query: string;
+ readonly limit?: number;
+ readonly semantic?: boolean | null;
+ readonly keywordOnly?: boolean;
+ readonly pathPrefix?: string;
+ readonly keywordWeight?: number;
+ readonly semanticWeight?: number;
+}
+
+export interface SearchOutcome {
+ readonly results: ReadonlyArray<BrainSearchResult>;
+ readonly warnings: ReadonlyArray<string>;
+ readonly total: number;
+}
+
+export interface ResolvedEmbeddingConfig {
+ readonly enabled: boolean;
+ readonly provider: "openai-compat" | "disabled";
+ readonly baseUrl: string | null;
+ readonly model: string | null;
+ readonly apiKey: string | null;
+ readonly dimension: number | null;
+ readonly timeoutMs: number;
+ readonly concurrency: number;
+ readonly batchSize: number;
+}
+
+export interface ResolvedSearchConfig {
+ readonly vault: string;
+ readonly dbPath: string;
+ readonly ignorePaths: ReadonlyArray<string>;
+ readonly chunkSize: number;
+ readonly chunkOverlap: number;
+ readonly keywordWeight: number;
+ readonly semanticWeight: number;
+ readonly semantic: ResolvedEmbeddingConfig;
+}
+```
+
+- [ ] **Step 4: Run test to verify pass**
+
+Run: `bun test tests/core/search/types.test.ts`
+Expected: `2 pass, 0 fail`.
+
+- [ ] **Step 5: Typecheck**
+
+Run: `bun run typecheck`
+Expected: no errors. (Other modules may still error if they were partially implemented in earlier work β€” at this point only types.ts is new.)
+
+- [ ] **Step 6: Pause for review (no commit)**
+
+### Task 2: paths.ts β€” resolveIndexPath
+
+**Files:**
+- Create: `src/core/search/paths.ts`
+- Test: `tests/core/search/paths.test.ts`
+
+- [ ] **Step 1: Write the failing test**
+
+`tests/core/search/paths.test.ts`:
+
+```ts
+import { test, expect } from "bun:test";
+import { resolveIndexPath } from "../../../src/core/search/paths.ts";
+
+test("default points under <vault>/.open-second-brain", () => {
+ expect(resolveIndexPath("/v", null)).toBe("/v/.open-second-brain/brain.sqlite");
+});
+
+test("explicit override wins", () => {
+ expect(resolveIndexPath("/v", "/tmp/custom.sqlite")).toBe("/tmp/custom.sqlite");
+});
+
+test("blank override falls back to default", () => {
+ expect(resolveIndexPath("/v", "")).toBe("/v/.open-second-brain/brain.sqlite");
+});
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `bun test tests/core/search/paths.test.ts`
+Expected: FAIL with "Cannot find module".
+
+- [ ] **Step 3: Implement paths.ts**
+
+`src/core/search/paths.ts`:
+
+```ts
+/**
+ * Resolve the on-disk location of the search index file.
+ *
+ * Default: `<vault>/.open-second-brain/brain.sqlite`. Overridable
+ * through CLI `--db` or config `search_db_path`.
+ */
+
+import { join } from "node:path";
+
+export function resolveIndexPath(vault: string, override: string | null): string {
+ if (override && override.trim() !== "") return override;
+ return join(vault, ".open-second-brain", "brain.sqlite");
+}
+```
+
+- [ ] **Step 4: Run test to verify pass**
+
+Run: `bun test tests/core/search/paths.test.ts`
+Expected: `3 pass, 0 fail`.
+
+- [ ] **Step 5: Pause for review (no commit)**
+
+### Task 3: resolveSearchConfig
+
+**Files:**
+- Create: `src/core/search/index.ts` (just the `resolveSearchConfig` export; will grow later)
+- Test: `tests/core/search/config.test.ts`
+
+- [ ] **Step 1: Write the failing test**
+
+`tests/core/search/config.test.ts`:
+
+```ts
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { resolveSearchConfig } from "../../../src/core/search/index.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+
+let tmp: string;
+let configPath: string;
+let origEnv: Record<string, string | undefined>;
+
+const ENV_KEYS = [
+ "OPEN_SECOND_BRAIN_SEARCH_DB",
+ "OPEN_SECOND_BRAIN_SEARCH_SEMANTIC",
+ "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER",
+ "OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL",
+ "OPEN_SECOND_BRAIN_EMBEDDING_MODEL",
+ "OPEN_SECOND_BRAIN_EMBEDDING_KEY",
+ "OPEN_SECOND_BRAIN_EMBEDDING_DIM",
+];
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "osb-cfg-"));
+ configPath = join(tmp, "config.yaml");
+ origEnv = {};
+ for (const k of ENV_KEYS) {
+ origEnv[k] = process.env[k];
+ delete process.env[k];
+ }
+});
+
+afterEach(() => {
+ for (const k of ENV_KEYS) {
+ if (origEnv[k] === undefined) delete process.env[k];
+ else process.env[k] = origEnv[k];
+ }
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+test("defaults are returned when config and env are empty", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\n`);
+ const cfg = resolveSearchConfig({ vault: tmp, configPath });
+ expect(cfg.vault).toBe(tmp);
+ expect(cfg.dbPath).toBe(join(tmp, ".open-second-brain", "brain.sqlite"));
+ expect(cfg.chunkSize).toBe(800);
+ expect(cfg.chunkOverlap).toBe(100);
+ expect(cfg.keywordWeight).toBe(0.6);
+ expect(cfg.semanticWeight).toBe(0.4);
+ expect(cfg.semantic.enabled).toBe(false);
+ expect(cfg.semantic.provider).toBe("openai-compat");
+ expect(cfg.semantic.apiKey).toBeNull();
+ expect(cfg.ignorePaths).toContain(".git");
+ expect(cfg.ignorePaths).toContain(".open-second-brain");
+});
+
+test("env overrides config which overrides defaults", () => {
+ writeFileSync(
+ configPath,
+ [
+ `vault: "${tmp}"`,
+ `search_chunk_size: "500"`,
+ `search_semantic_enabled: "true"`,
+ `embedding_base_url: "https://config.example/v1"`,
+ `embedding_model: "config-model"`,
+ `embedding_api_key: "cfg-key"`,
+ ].join("\n") + "\n",
+ );
+ process.env["OPEN_SECOND_BRAIN_EMBEDDING_MODEL"] = "env-model";
+ const cfg = resolveSearchConfig({ vault: tmp, configPath });
+ expect(cfg.chunkSize).toBe(500);
+ expect(cfg.semantic.enabled).toBe(true);
+ expect(cfg.semantic.baseUrl).toBe("https://config.example/v1");
+ expect(cfg.semantic.model).toBe("env-model"); // env wins
+ expect(cfg.semantic.apiKey).toBe("cfg-key");
+});
+
+test("overrides win over both env and config", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_size: "500"\n`);
+ process.env["OPEN_SECOND_BRAIN_SEARCH_SEMANTIC"] = "false";
+ const cfg = resolveSearchConfig({
+ vault: tmp,
+ configPath,
+ overrides: { keywordWeight: 0.9, semanticWeight: 0.1 },
+ });
+ expect(cfg.keywordWeight).toBe(0.9);
+ expect(cfg.semanticWeight).toBe(0.1);
+});
+
+test("invalid numeric value throws INVALID_INPUT", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_size: "not-a-number"\n`);
+ let err: SearchError | null = null;
+ try {
+ resolveSearchConfig({ vault: tmp, configPath });
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err).not.toBeNull();
+ expect(err?.code).toBe("INVALID_INPUT");
+ expect(err?.message).toContain("search_chunk_size");
+});
+
+test("weights summing above 1 is rejected", () => {
+ writeFileSync(
+ configPath,
+ `vault: "${tmp}"\nsearch_keyword_weight: "0.7"\nsearch_semantic_weight: "0.5"\n`,
+ );
+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/sum.*1/);
+});
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `bun test tests/core/search/config.test.ts`
+Expected: FAIL with "Cannot find module" (resolveSearchConfig not exported yet).
+
+- [ ] **Step 3: Implement resolveSearchConfig**
+
+`src/core/search/index.ts`:
+
+```ts
+/**
+ * Public surface for `src/core/search/*`. Currently exports the config
+ * resolver only; indexer/search/status/check are added in later phases.
+ */
+
+import { discoverConfig } from "../config.ts";
+import { resolveIndexPath } from "./paths.ts";
+import { SearchError } from "./types.ts";
+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "./types.ts";
+
+const DEFAULT_IGNORE_PATHS = [
+ ".git",
+ "node_modules",
+ ".open-second-brain",
+ ".obsidian/cache",
+ ".trash",
+ ".stversions",
+];
+
+const DEFAULTS = {
+ chunkSize: 800,
+ chunkOverlap: 100,
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ provider: "openai-compat" as const,
+ timeoutMs: 10_000,
+ concurrency: 4,
+ batchSize: 32,
+};
+
+function envOrConfig(
+ env: NodeJS.ProcessEnv,
+ config: Record<string, string>,
+ envKey: string,
+ configKey: string,
+): string | null {
+ const e = env[envKey];
+ if (e !== undefined && e !== "") return e;
+ const c = config[configKey];
+ if (c !== undefined && c !== "") return c;
+ return null;
+}
+
+function parseInteger(raw: string | null, fallback: number, fieldName: string): number {
+ if (raw === null) return fallback;
+ const n = Number(raw);
+ if (!Number.isFinite(n) || !Number.isInteger(n)) {
+ throw new SearchError("INVALID_INPUT", `${fieldName} must be an integer, got '${raw}'`);
+ }
+ return n;
+}
+
+function parseFloat01(raw: string | null, fallback: number, fieldName: string): number {
+ if (raw === null) return fallback;
+ const n = Number(raw);
+ if (!Number.isFinite(n) || n < 0 || n > 1) {
+ throw new SearchError("INVALID_INPUT", `${fieldName} must be a number in [0, 1], got '${raw}'`);
+ }
+ return n;
+}
+
+function parseBool(raw: string | null, fallback: boolean, fieldName: string): boolean {
+ if (raw === null) return fallback;
+ if (raw === "true" || raw === "1") return true;
+ if (raw === "false" || raw === "0") return false;
+ throw new SearchError("INVALID_INPUT", `${fieldName} must be 'true' or 'false', got '${raw}'`);
+}
+
+function parseProvider(raw: string | null): ResolvedEmbeddingConfig["provider"] {
+ if (raw === null) return DEFAULTS.provider;
+ if (raw === "openai-compat" || raw === "disabled") return raw;
+ throw new SearchError("INVALID_INPUT", `embedding_provider must be 'openai-compat' or 'disabled', got '${raw}'`);
+}
+
+function parseIgnorePaths(raw: string | null): string[] {
+ if (raw === null) return DEFAULT_IGNORE_PATHS;
+ return raw.split(",").map((s) => s.trim()).filter((s) => s.length > 0);
+}
+
+export function resolveSearchConfig(opts: {
+ vault: string;
+ configPath?: string;
+ overrides?: Partial<ResolvedSearchConfig>;
+}): ResolvedSearchConfig {
+ const env = process.env;
+ const config = opts.configPath
+ ? discoverConfig(opts.configPath).data
+ : {} as Record<string, string>;
+
+ const dbPath = resolveIndexPath(
+ opts.vault,
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_DB", "search_db_path"),
+ );
+
+ const chunkSize = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CHUNK_SIZE", "search_chunk_size"),
+ DEFAULTS.chunkSize,
+ "search_chunk_size",
+ );
+ const chunkOverlap = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CHUNK_OVERLAP", "search_chunk_overlap"),
+ DEFAULTS.chunkOverlap,
+ "search_chunk_overlap",
+ );
+ const keywordWeight = parseFloat01(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_KW_WEIGHT", "search_keyword_weight"),
+ DEFAULTS.keywordWeight,
+ "search_keyword_weight",
+ );
+ const semanticWeight = parseFloat01(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SEM_WEIGHT", "search_semantic_weight"),
+ DEFAULTS.semanticWeight,
+ "search_semantic_weight",
+ );
+ if (keywordWeight + semanticWeight > 1.0 + 1e-9) {
+ throw new SearchError(
+ "INVALID_INPUT",
+ `keyword_weight + semantic_weight must sum to <= 1, got ${keywordWeight} + ${semanticWeight}`,
+ );
+ }
+
+ const ignorePaths = parseIgnorePaths(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_IGNORE", "search_ignore_paths"),
+ );
+
+ const semanticEnabled = parseBool(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SEMANTIC", "search_semantic_enabled"),
+ false,
+ "search_semantic_enabled",
+ );
+ const provider = parseProvider(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER", "embedding_provider"),
+ );
+ const baseUrl = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL", "embedding_base_url");
+ const model = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_MODEL", "embedding_model");
+ const apiKey = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_KEY", "embedding_api_key");
+ const dimRaw = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_DIM", "embedding_dimension");
+ const dimension = dimRaw === null ? null : parseInteger(dimRaw, 0, "embedding_dimension");
+ const timeoutMs = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_TIMEOUT", "embedding_timeout_ms"),
+ DEFAULTS.timeoutMs,
+ "embedding_timeout_ms",
+ );
+ const concurrency = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_CONCURRENCY", "embedding_concurrency"),
+ DEFAULTS.concurrency,
+ "embedding_concurrency",
+ );
+ const batchSize = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_BATCH", "embedding_batch_size"),
+ DEFAULTS.batchSize,
+ "embedding_batch_size",
+ );
+
+ const semantic: ResolvedEmbeddingConfig = Object.freeze({
+ enabled: semanticEnabled,
+ provider,
+ baseUrl,
+ model,
+ apiKey,
+ dimension,
+ timeoutMs,
+ concurrency,
+ batchSize,
+ });
+
+ const base: ResolvedSearchConfig = Object.freeze({
+ vault: opts.vault,
+ dbPath,
+ ignorePaths: Object.freeze([...ignorePaths]),
+ chunkSize,
+ chunkOverlap,
+ keywordWeight,
+ semanticWeight,
+ semantic,
+ });
+
+ if (!opts.overrides) return base;
+ return Object.freeze({
+ ...base,
+ ...opts.overrides,
+ semantic: Object.freeze({ ...base.semantic, ...(opts.overrides.semantic ?? {}) }),
+ ignorePaths: opts.overrides.ignorePaths
+ ? Object.freeze([...opts.overrides.ignorePaths])
+ : base.ignorePaths,
+ });
+}
+```
+
+- [ ] **Step 4: Run test to verify pass**
+
+Run: `bun test tests/core/search/config.test.ts`
+Expected: `5 pass, 0 fail`.
+
+- [ ] **Step 5: Run wider typecheck**
+
+Run: `bun run typecheck`
+Expected: no errors.
+
+- [ ] **Step 6: Pause for review (no commit)**
diff --git a/openclaw.plugin.json b/openclaw.plugin.json
index 36b1793b..0b2cddc5 100644
--- a/openclaw.plugin.json
+++ b/openclaw.plugin.json
@@ -2,7 +2,7 @@
"id": "open-second-brain",
"name": "Open Second Brain",
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults.",
- "version": "0.9.1",
+ "version": "0.10.0",
"activation": { "onStartup": true },
"skills": ["./skills"],
"contracts": {
diff --git a/package.json b/package.json
index 352c01dc..763ad572 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "0.9.1",
+ "version": "0.10.0",
"type": "module",
"private": false,
"description": "Second brain for AI agents using Obsidian-compatible Markdown vaults. Works with Hermes, Claude Code, Codex, and OpenClaw.",
@@ -47,5 +47,8 @@
},
"openclaw": {
"extensions": ["./openclaw/index.js"]
+ },
+ "optionalDependencies": {
+ "sqlite-vec": "^0.1.9"
}
}
diff --git a/plugin.yaml b/plugin.yaml
index 3d11dd35..a76abe97 100644
--- a/plugin.yaml
+++ b/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "0.9.1"
+version: "0.10.0"
description: "Hermes plugin entrypoint for OpenSecondBrain vault health checks and the optional MCP tool server."
author: "OpenSecondBrain contributors"
kind: standalone
diff --git a/plugins/codex/.codex-plugin/plugin.json b/plugins/codex/.codex-plugin/plugin.json
index ca4bc433..20153cf1 100644
--- a/plugins/codex/.codex-plugin/plugin.json
+++ b/plugins/codex/.codex-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "open-second-brain",
- "version": "0.9.1",
+ "version": "0.10.0",
"description": "Plugin-first second brain package for Codex, Hermes, Claude Code, OpenClaw, and other agent runtimes.",
"author": {
"name": "Open Second Brain contributors",
diff --git a/plugins/hermes/plugin.yaml b/plugins/hermes/plugin.yaml
index 62c134b1..b6e76575 100644
--- a/plugins/hermes/plugin.yaml
+++ b/plugins/hermes/plugin.yaml
@@ -1,5 +1,5 @@
name: open-second-brain
-version: "0.9.1"
+version: "0.10.0"
description: "Hermes adapter for OpenSecondBrain vault health checks and optional MCP tool server."
author: "OpenSecondBrain contributors"
kind: standalone
diff --git a/pyproject.toml b/pyproject.toml
index 32469c6e..3d183f0b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
# CLI entry points (those moved to `package.json` `bin`).
[project]
name = "open-second-brain"
-version = "0.9.1"
+version = "0.10.0"
description = "Hermes Python shim for Open Second Brain. Most of the project (CLI, MCP server, OpenClaw plugin) is TypeScript on Bun; see package.json."
readme = "README.md"
requires-python = ">=3.11"
diff --git a/src/cli/main.ts b/src/cli/main.ts
index 386ce326..6c38e19c 100644
--- a/src/cli/main.ts
+++ b/src/cli/main.ts
@@ -44,6 +44,7 @@ import type { ReceiptPolicyStatus } from "../core/pay-memory/types.ts";
import { listVaultPages, writeFrontmatter } from "../core/vault.ts";
import { CliError, parseFlags } from "./argparse.ts";
import { handleBrainSubcommand } from "./brain.ts";
+import { handleSearchSubcommand } from "./search.ts";
import {
installCli,
renderInstallResult,
@@ -223,9 +224,67 @@ async function cmdInit(argv: string[]): Promise<number> {
process.stdout.write(`timezone registered: ${timezone}\n`);
process.stdout.write(`timezone persisted to: ${configPath}\n`);
}
+ writeSearchInitBlock(configPath);
return 0;
}
+/**
+ * Print the post-init search-onboarding block (design Β§10).
+ *
+ * Always advertises `o2b search index`. When the user has already
+ * flipped `search_semantic_enabled` to true but no embedding key is
+ * resolvable, the detailed configuration template is appended. The
+ * block prints once, only during `o2b init` β€” no nagging on other
+ * CLI invocations (the dedicated diagnostic is `o2b search check`).
+ */
+function writeSearchInitBlock(configPath: string): void {
+ process.stdout.write("\nSearch:\n");
+ process.stdout.write(" next: o2b search index # build the vault search index\n");
+
+ const data = discoverConfig(configPath).data;
+ // Accept "true" / "True" / "TRUE" / "1" from both the config file
+ // and the env override β€” matches the parseBool helper inside
+ // resolveSearchConfig so the init banner does not lie about state.
+ const truthy = (v: unknown): boolean => {
+ if (typeof v !== "string") return false;
+ const s = v.trim().toLowerCase();
+ return s === "true" || s === "1";
+ };
+ const enabled =
+ truthy(data["search_semantic_enabled"]) ||
+ truthy(process.env["OPEN_SECOND_BRAIN_SEARCH_SEMANTIC"]);
+ const keyPresent = !!(
+ (data["embedding_api_key"] && data["embedding_api_key"] !== "") ||
+ process.env["OPEN_SECOND_BRAIN_EMBEDDING_KEY"]
+ );
+ if (!enabled || keyPresent) return;
+
+ process.stdout.write(
+ [
+ "",
+ "Semantic search is enabled but no embedding key is configured.",
+ "",
+ "Either set in the config file (printed above), or via env vars:",
+ "",
+ ` search_semantic_enabled: "true"`,
+ ` embedding_base_url: "https://openrouter.ai/api/v1"`,
+ ` embedding_model: "google/gemini-embedding-2-preview"`,
+ ` embedding_api_key: "<your key>"`,
+ "",
+ "Env equivalents:",
+ " OPEN_SECOND_BRAIN_SEARCH_SEMANTIC=true",
+ " OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL=...",
+ " OPEN_SECOND_BRAIN_EMBEDDING_MODEL=...",
+ " OPEN_SECOND_BRAIN_EMBEDDING_KEY=...",
+ "",
+ "Then:",
+ " o2b search check",
+ " o2b search index --embeddings",
+ "",
+ ].join("\n"),
+ );
+}
+
async function cmdDoctor(argv: string[]): Promise<number> {
const { flags } = parseFlags(argv, {
vault: { type: "string" },
@@ -1193,6 +1252,13 @@ Brain (observing memory):
brain unpin Clear the pinned flag
brain rollback Restore Brain/ from a snapshot (--list / <run_id>)
brain doctor Validate Brain invariants (--strict promotes warnings)
+
+Search:
+ search "<query>" Search the vault index (default verb is 'query')
+ search index Incrementally update the index from the vault
+ search reindex Rebuild the index atomically (.new -> rename -> .bak)
+ search status Print index summary (counts, model, vec extension)
+ search check Pre-flight diagnostics (SQLite, FTS5, vec, provider)
`;
function sortedReplacer(_key: string, value: unknown): unknown {
@@ -1274,6 +1340,8 @@ export async function main(argv: ReadonlyArray<string>): Promise<number> {
return await cmdPaymentDigest(rest);
case "brain":
return await handleBrainSubcommand(rest);
+ case "search":
+ return await handleSearchSubcommand(rest);
default:
process.stderr.write(`error: unknown command: ${command}\n`);
process.stderr.write(HELP);
diff --git a/src/cli/search.ts b/src/cli/search.ts
new file mode 100644
index 00000000..0f5b3ffa
--- /dev/null
+++ b/src/cli/search.ts
@@ -0,0 +1,422 @@
+/**
+ * `o2b search` subcommand dispatcher.
+ *
+ * Routes the five Brain Search verbs (design doc Β§8) to thin wrappers
+ * over `src/core/search/*`. The core modules own all I/O; this file
+ * only parses flags, resolves the vault, shapes exit codes, and renders
+ * either human-readable or JSON output.
+ *
+ * o2b search "<query>" β†’ cmdSearchQuery (default verb)
+ * o2b search index β†’ cmdSearchIndex
+ * o2b search reindex β†’ cmdSearchReindex
+ * o2b search status β†’ cmdSearchStatus
+ * o2b search check β†’ cmdSearchCheck
+ */
+
+import { defaultConfigPath, resolveVault } from "../core/config.ts";
+import {
+ indexCheck,
+ indexStatus,
+ indexVault,
+ reindexVault,
+ resolveSearchConfig,
+ search,
+ SearchError,
+} from "../core/search/index.ts";
+import type {
+ IndexCheckReport,
+ IndexProgressEvent,
+ IndexStats,
+ IndexStatusSnapshot,
+ ResolvedSearchConfig,
+ SearchOutcome,
+} from "../core/search/index.ts";
+import { CliError, parseFlags } from "./argparse.ts";
+
+const KNOWN_VERBS = new Set(["query", "index", "reindex", "status", "check"]);
+
+export async function handleSearchSubcommand(argv: ReadonlyArray<string>): Promise<number> {
+ // First positional is verb iff it matches a known verb. Otherwise the
+ // default verb is `query` and the positional is the query string.
+ let verb = "query";
+ let rest = argv;
+ if (argv.length > 0 && KNOWN_VERBS.has(argv[0]!)) {
+ verb = argv[0]!;
+ rest = argv.slice(1);
+ }
+
+ try {
+ switch (verb) {
+ case "query":
+ return await cmdSearchQuery(rest);
+ case "index":
+ return await cmdSearchIndex(rest);
+ case "reindex":
+ return await cmdSearchReindex(rest);
+ case "status":
+ return await cmdSearchStatus(rest);
+ case "check":
+ return await cmdSearchCheck(rest);
+ default:
+ process.stderr.write(`error: unknown search verb: ${verb}\n`);
+ return 2;
+ }
+ } catch (e) {
+ if (e instanceof CliError) {
+ process.stderr.write(`error: ${e.message}\n`);
+ return 2;
+ }
+ if (e instanceof SearchError) {
+ process.stderr.write(`error: ${e.message} [${e.code}]\n`);
+ return e.code === "INVALID_INPUT" ? 2 : 1;
+ }
+ process.stderr.write(`error: ${e instanceof Error ? e.message : String(e)}\n`);
+ return 1;
+ }
+}
+
+function resolveConfig(
+ flags: Record<string, string | boolean | string[] | undefined>,
+): ResolvedSearchConfig {
+ const flagVault = typeof flags["vault"] === "string" ? (flags["vault"] as string) : undefined;
+ const configPath =
+ typeof flags["config"] === "string" ? (flags["config"] as string) : defaultConfigPath();
+ const vault = flagVault ?? resolveVault(configPath) ?? null;
+ if (!vault) {
+ throw new CliError(
+ "no vault configured. Pass --vault <path> or run `o2b init --vault <path> ...` first.",
+ );
+ }
+ const dbFlag = typeof flags["db"] === "string" ? (flags["db"] as string) : undefined;
+ const kwFlag = typeof flags["keyword-weight"] === "string" ? Number(flags["keyword-weight"]) : undefined;
+ const semFlag = typeof flags["semantic-weight"] === "string" ? Number(flags["semantic-weight"]) : undefined;
+ const concurrencyFlag =
+ typeof flags["concurrency"] === "string" ? Number(flags["concurrency"]) : undefined;
+ const overrides = {
+ ...(dbFlag !== undefined ? { dbPath: dbFlag } : {}),
+ ...(kwFlag !== undefined ? { keywordWeight: kwFlag } : {}),
+ ...(semFlag !== undefined ? { semanticWeight: semFlag } : {}),
+ ...(concurrencyFlag !== undefined ? { semantic: { concurrency: concurrencyFlag } } : {}),
+ };
+ return resolveSearchConfig({ vault, configPath, overrides });
+}
+
+// ─── query ────────────────────────────────────────────────────────────────────
+
+async function cmdSearchQuery(argv: ReadonlyArray<string>): Promise<number> {
+ const { flags, positional } = parseFlags(argv, {
+ vault: { type: "string" },
+ config: { type: "string" },
+ db: { type: "string" },
+ limit: { type: "string", default: "10" },
+ semantic: { type: "boolean" },
+ "keyword-only": { type: "boolean" },
+ path: { type: "string" },
+ "keyword-weight": { type: "string" },
+ "semantic-weight": { type: "string" },
+ "auto-refresh": { type: "boolean" },
+ json: { type: "boolean" },
+ verbose: { type: "boolean" },
+ });
+
+ if (positional.length === 0) {
+ throw new CliError("query string is required");
+ }
+ if (flags["semantic"] === true && flags["keyword-only"] === true) {
+ throw new CliError("--semantic and --keyword-only are mutually exclusive");
+ }
+ const query = positional.join(" ");
+ const limitNum = Number(flags["limit"] ?? "10");
+ if (!Number.isInteger(limitNum) || limitNum < 1 || limitNum > 100) {
+ throw new CliError("--limit must be an integer in 1..100");
+ }
+
+ const cfg = resolveConfig(flags);
+
+ if (flags["auto-refresh"]) {
+ try {
+ await indexVault(cfg);
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : String(e);
+ process.stderr.write(`auto-refresh failed: ${msg}\n`);
+ }
+ }
+
+ // Pass `undefined` when no explicit flag is set so `search()` falls back
+ // to the config default. Passing `null` works by accident today but blurs
+ // the implicit/explicit policy boundary in Β§7 of the search design.
+ const semanticOverride: boolean | undefined =
+ flags["semantic"] === true ? true : flags["keyword-only"] === true ? false : undefined;
+
+ const outcome = await search(cfg, {
+ query,
+ limit: limitNum,
+ semantic: semanticOverride,
+ keywordOnly: flags["keyword-only"] === true,
+ pathPrefix: typeof flags["path"] === "string" ? (flags["path"] as string) : undefined,
+ });
+
+ if (flags["json"]) {
+ process.stdout.write(JSON.stringify(jsonForOutcome(outcome)) + "\n");
+ return 0;
+ }
+ process.stdout.write(renderOutcomeHuman(outcome, flags["verbose"] === true));
+ return 0;
+}
+
+function jsonForOutcome(o: SearchOutcome): unknown {
+ return {
+ results: o.results.map((r) => ({
+ path: r.path,
+ title: r.title,
+ content: r.content,
+ score: r.score,
+ keyword_score: r.keywordScore,
+ semantic_score: r.semanticScore,
+ link_boost: r.linkBoost,
+ recency_boost: r.recencyBoost,
+ start_line: r.startLine,
+ end_line: r.endLine,
+ search_type: r.searchType,
+ document_id: r.documentId,
+ chunk_id: r.chunkId,
+ })),
+ warnings: o.warnings,
+ total: o.total,
+ };
+}
+
+function renderOutcomeHuman(o: SearchOutcome, verbose: boolean): string {
+ const lines: string[] = [];
+ if (o.results.length === 0) {
+ lines.push("(no results)");
+ }
+ o.results.forEach((r, i) => {
+ const score = r.score.toFixed(2);
+ lines.push(`[${i + 1}] ${r.path} β€’ ${score}`);
+ lines.push(
+ ` line ${r.startLine}-${r.endLine} β€’ ${r.searchType}` +
+ (verbose
+ ? ` β€’ kw=${r.keywordScore.toFixed(2)} sem=${r.semanticScore.toFixed(2)} link=${r.linkBoost.toFixed(2)} rec=${r.recencyBoost.toFixed(2)}`
+ : ""),
+ );
+ const snippet = r.content.trim().replace(/\s+/g, " ").slice(0, 140);
+ lines.push(` ${snippet}${r.content.length > 140 ? "…" : ""}`);
+ lines.push("");
+ });
+ for (const w of o.warnings) lines.push(`warning: ${w}`);
+ return lines.join("\n") + (lines.length > 0 ? "" : "\n");
+}
+
+// ─── index ────────────────────────────────────────────────────────────────────
+
+async function cmdSearchIndex(argv: ReadonlyArray<string>): Promise<number> {
+ const { flags } = parseFlags(argv, {
+ vault: { type: "string" },
+ config: { type: "string" },
+ db: { type: "string" },
+ embeddings: { type: "boolean" },
+ force: { type: "boolean" },
+ concurrency: { type: "string" },
+ verbose: { type: "boolean" },
+ json: { type: "boolean" },
+ });
+ const cfg = resolveConfig(flags);
+
+ const events: IndexProgressEvent[] = [];
+ const stats = await indexVault(cfg, {
+ embeddings: flags["embeddings"] === true,
+ force: flags["force"] === true,
+ onFile: (e) => {
+ events.push(e);
+ if (flags["verbose"]) {
+ const msg = e.message ? ` ${e.message}` : "";
+ process.stderr.write(`${e.kind}\t${e.path}${msg}\n`);
+ }
+ },
+ });
+
+ if (flags["json"]) {
+ process.stdout.write(JSON.stringify(jsonForStats(stats, cfg)) + "\n");
+ return 0;
+ }
+ process.stdout.write(renderStatsHuman(stats, cfg));
+ return 0;
+}
+
+function jsonForStats(stats: IndexStats, cfg: ResolvedSearchConfig): unknown {
+ return {
+ stats: {
+ added: stats.added,
+ updated: stats.updated,
+ unchanged: stats.unchanged,
+ deleted: stats.deleted,
+ chunks_total: stats.chunksTotal,
+ embeddings_computed: stats.embeddingsComputed,
+ embeddings_retries: stats.embeddingsRetries,
+ },
+ errors: stats.errors.map((e) => ({ path: e.path, message: e.message })),
+ duration_ms: stats.durationMs,
+ vault: cfg.vault,
+ db_path: cfg.dbPath,
+ };
+}
+
+function renderStatsHuman(stats: IndexStats, cfg: ResolvedSearchConfig): string {
+ const lines: string[] = [];
+ lines.push(`indexing vault: ${cfg.vault}`);
+ lines.push(` added: ${stats.added} files, ${stats.chunksTotal} chunks`);
+ lines.push(` updated: ${stats.updated} files`);
+ lines.push(` unchanged: ${stats.unchanged} files`);
+ lines.push(` deleted: ${stats.deleted} files`);
+ if (stats.embeddingsComputed > 0 || stats.embeddingsRetries > 0) {
+ lines.push(
+ ` embeddings: ${stats.embeddingsComputed} computed (${stats.embeddingsRetries} retries)`,
+ );
+ }
+ if (stats.errors.length > 0) {
+ lines.push(` errors:`);
+ for (const e of stats.errors) lines.push(` - ${e.path}: ${e.message}`);
+ }
+ lines.push(`done in ${(stats.durationMs / 1000).toFixed(1)}s`);
+ return lines.join("\n") + "\n";
+}
+
+// ─── reindex ──────────────────────────────────────────────────────────────────
+
+async function cmdSearchReindex(argv: ReadonlyArray<string>): Promise<number> {
+ const { flags } = parseFlags(argv, {
+ vault: { type: "string" },
+ config: { type: "string" },
+ db: { type: "string" },
+ embeddings: { type: "boolean" },
+ concurrency: { type: "string" },
+ json: { type: "boolean" },
+ verbose: { type: "boolean" },
+ });
+ const cfg = resolveConfig(flags);
+ const stats = await reindexVault(cfg, {
+ embeddings: flags["embeddings"] === true,
+ onFile: flags["verbose"]
+ ? (e) => process.stderr.write(`${e.kind}\t${e.path}\n`)
+ : undefined,
+ });
+ if (flags["json"]) {
+ process.stdout.write(JSON.stringify(jsonForStats(stats, cfg)) + "\n");
+ return 0;
+ }
+ process.stdout.write(renderStatsHuman(stats, cfg));
+ return 0;
+}
+
+// ─── status ───────────────────────────────────────────────────────────────────
+
+async function cmdSearchStatus(argv: ReadonlyArray<string>): Promise<number> {
+ const { flags } = parseFlags(argv, {
+ vault: { type: "string" },
+ config: { type: "string" },
+ db: { type: "string" },
+ json: { type: "boolean" },
+ });
+ const cfg = resolveConfig(flags);
+ const status = await indexStatus(cfg);
+ if (flags["json"]) {
+ process.stdout.write(JSON.stringify(jsonForStatus(status)) + "\n");
+ return 0;
+ }
+ process.stdout.write(renderStatusHuman(status));
+ return 0;
+}
+
+function jsonForStatus(s: IndexStatusSnapshot): unknown {
+ return {
+ index_path: s.indexPath,
+ exists: s.exists,
+ schema_version: s.schemaVersion,
+ documents: s.documents,
+ chunks: s.chunks,
+ embeddings: s.embeddings,
+ stale_embeddings: s.staleEmbeddings,
+ embedding_model: s.embeddingModel,
+ embedding_dimension: s.embeddingDimension,
+ vec_extension: s.vecExtension,
+ semantic_enabled: s.semanticEnabled,
+ embedding_key_present: s.embeddingKeyPresent,
+ last_indexed_at: s.lastIndexedAt,
+ last_full_index_at: s.lastFullIndexAt,
+ warnings: s.warnings,
+ };
+}
+
+function renderStatusHuman(s: IndexStatusSnapshot): string {
+ if (!s.exists) {
+ return `index: not initialised. Run: o2b search index\n path: ${s.indexPath}\n`;
+ }
+ const lines: string[] = [];
+ lines.push(`index: ${s.indexPath}`);
+ lines.push(`schema_version: ${s.schemaVersion}`);
+ lines.push(`documents: ${s.documents}`);
+ lines.push(`chunks: ${s.chunks}`);
+ lines.push(`embeddings: ${s.embeddings} (stale: ${s.staleEmbeddings})`);
+ lines.push(`embedding_model: ${s.embeddingModel ?? "(none)"}`);
+ lines.push(`embedding_dimension: ${s.embeddingDimension ?? "(none)"}`);
+ lines.push(`vec_extension: ${s.vecExtension}`);
+ lines.push(`semantic_enabled: ${s.semanticEnabled}`);
+ lines.push(`embedding_key: ${s.embeddingKeyPresent ? "present" : "missing"}`);
+ lines.push(`last_indexed_at: ${s.lastIndexedAt ?? "(never)"}`);
+ lines.push(`last_full_index_at: ${s.lastFullIndexAt ?? "(never)"}`);
+ for (const w of s.warnings) lines.push(`warning: ${w}`);
+ return lines.join("\n") + "\n";
+}
+
+// ─── check ────────────────────────────────────────────────────────────────────
+
+async function cmdSearchCheck(argv: ReadonlyArray<string>): Promise<number> {
+ const { flags } = parseFlags(argv, {
+ vault: { type: "string" },
+ config: { type: "string" },
+ db: { type: "string" },
+ json: { type: "boolean" },
+ });
+ const cfg = resolveConfig(flags);
+ const report = await indexCheck(cfg);
+ if (flags["json"]) {
+ process.stdout.write(JSON.stringify(jsonForCheck(report)) + "\n");
+ } else {
+ process.stdout.write(renderCheckHuman(report));
+ }
+ return report.fatal.length > 0 ? 1 : 0;
+}
+
+function jsonForCheck(r: IndexCheckReport): unknown {
+ return {
+ vault_readable: r.vaultReadable,
+ index_dir_writable: r.indexDirWritable,
+ sqlite_ok: r.sqliteOk,
+ fts5_ok: r.fts5Ok,
+ vec_extension: r.vecExtension,
+ embedding_key_resolved: r.embeddingKeyResolved,
+ provider_reachable: r.providerReachable,
+ provider_reason: r.providerReason,
+ warnings: r.warnings,
+ fatal: r.fatal,
+ };
+}
+
+function renderCheckHuman(r: IndexCheckReport): string {
+ const lines: string[] = [];
+ const ok = (b: boolean) => (b ? "OK" : "MISSING");
+ lines.push(`vault_readable: ${ok(r.vaultReadable)}`);
+ lines.push(`index_dir_writable: ${ok(r.indexDirWritable)}`);
+ lines.push(`sqlite_ok: ${ok(r.sqliteOk)}`);
+ lines.push(`fts5_ok: ${ok(r.fts5Ok)}`);
+ lines.push(`vec_extension: ${r.vecExtension}`);
+ lines.push(`embedding_key: ${ok(r.embeddingKeyResolved)}`);
+ if (r.providerReachable !== null) {
+ lines.push(`provider_reachable: ${r.providerReachable ? "OK" : "FAIL"}`);
+ if (r.providerReason) lines.push(`provider_reason: ${r.providerReason}`);
+ }
+ for (const w of r.warnings) lines.push(`warning: ${w}`);
+ for (const f of r.fatal) lines.push(`fatal: ${f}`);
+ return lines.join("\n") + "\n";
+}
diff --git a/src/core/search/chunker.ts b/src/core/search/chunker.ts
new file mode 100644
index 00000000..8e20dc1c
--- /dev/null
+++ b/src/core/search/chunker.ts
@@ -0,0 +1,432 @@
+/**
+ * Markdown chunker β€” two-pass structural split + token-budget packer.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§6.
+ *
+ * Token approximation: whitespace word count. Deterministic across
+ * Bun/Node, dependency-free, machine-independent β€” so the same vault
+ * hashes the same chunks on every Syncthing peer.
+ */
+
+const DEFAULT_MAX_TOKENS = 800;
+const DEFAULT_MIN_TOKENS = 100;
+const DEFAULT_OVERLAP_TOKENS = 100;
+
+export interface ChunkOptions {
+ readonly maxTokens?: number;
+ readonly minTokens?: number;
+ readonly overlapTokens?: number;
+}
+
+export interface MarkdownChunk {
+ readonly chunkIndex: number;
+ readonly content: string;
+ readonly startLine: number;
+ readonly endLine: number;
+ readonly tokenCount: number;
+}
+
+export interface ChunkResult {
+ readonly title: string | null;
+ readonly chunks: ReadonlyArray<MarkdownChunk>;
+ readonly warnings: ReadonlyArray<string>;
+}
+
+interface Line {
+ readonly num: number;
+ readonly text: string;
+}
+
+type BlockKind = "frontmatter" | "code" | "heading" | "list" | "table" | "paragraph";
+
+interface Block {
+ readonly kind: BlockKind;
+ readonly lines: ReadonlyArray<Line>;
+ readonly tokenCount: number;
+}
+
+function countTokens(text: string): number {
+ if (text.length === 0) return 0;
+ let count = 0;
+ let inWord = false;
+ for (let i = 0; i < text.length; i++) {
+ const c = text.charCodeAt(i);
+ const isSpace = c === 0x20 || c === 0x09 || c === 0x0a || c === 0x0d;
+ if (!isSpace) {
+ if (!inWord) {
+ count++;
+ inWord = true;
+ }
+ } else {
+ inWord = false;
+ }
+ }
+ return count;
+}
+
+function tokensOfLines(lines: ReadonlyArray<Line>): number {
+ let total = 0;
+ for (const l of lines) total += countTokens(l.text);
+ return total;
+}
+
+function makeBlock(kind: BlockKind, lines: Line[]): Block {
+ return { kind, lines, tokenCount: tokensOfLines(lines) };
+}
+
+function isBlank(s: string): boolean {
+ return s.trim() === "";
+}
+
+const FENCE_RE = /^(?:```|~~~)/;
+const HEADING_RE = /^#{1,6}\s/;
+const LIST_ITEM_RE = /^\s*(?:[-*+]\s|\d+\.\s)/;
+const TABLE_SEP_RE = /^\s*\|?\s*:?-{3,}:?\s*(?:\|\s*:?-{3,}:?\s*)+\|?\s*$/;
+
+/** Split the raw text into a 1-indexed line array. */
+function splitLines(text: string): Line[] {
+ const out: Line[] = [];
+ let lineNum = 1;
+ let start = 0;
+ for (let i = 0; i < text.length; i++) {
+ const c = text.charCodeAt(i);
+ if (c === 0x0a /* \n */) {
+ const seg = text.slice(start, i);
+ const cleaned = seg.endsWith("\r") ? seg.slice(0, -1) : seg;
+ out.push({ num: lineNum++, text: cleaned });
+ start = i + 1;
+ }
+ }
+ if (start <= text.length) {
+ const seg = text.slice(start);
+ if (seg !== "" || out.length === 0) {
+ const cleaned = seg.endsWith("\r") ? seg.slice(0, -1) : seg;
+ out.push({ num: lineNum, text: cleaned });
+ }
+ }
+ return out;
+}
+
+function extractFrontmatter(
+ lines: ReadonlyArray<Line>,
+): { block: Block | null; remaining: ReadonlyArray<Line>; warnings: string[] } {
+ if (lines.length === 0 || lines[0]!.text.trim() !== "---") {
+ return { block: null, remaining: lines, warnings: [] };
+ }
+ for (let i = 1; i < lines.length; i++) {
+ if (lines[i]!.text.trim() === "---") {
+ const fmLines = lines.slice(0, i + 1);
+ return {
+ block: makeBlock("frontmatter", [...fmLines]),
+ remaining: lines.slice(i + 1),
+ warnings: [],
+ };
+ }
+ }
+ // Unterminated frontmatter: omit FM block, keep file otherwise indexable.
+ return {
+ block: null,
+ remaining: lines.slice(1),
+ warnings: ["malformed frontmatter (no closing '---')"],
+ };
+}
+
+function readCodeBlock(lines: ReadonlyArray<Line>, start: number): { end: number } {
+ // `lines[start]` is the opening fence.
+ const fenceText = lines[start]!.text.trim();
+ const fenceMark = fenceText.startsWith("```") ? "```" : "~~~";
+ for (let i = start + 1; i < lines.length; i++) {
+ if (lines[i]!.text.trim().startsWith(fenceMark)) return { end: i };
+ }
+ return { end: lines.length - 1 }; // unclosed: run to EOF
+}
+
+function readList(lines: ReadonlyArray<Line>, start: number): { end: number } {
+ let i = start + 1;
+ while (i < lines.length) {
+ const t = lines[i]!.text;
+ if (isBlank(t)) return { end: i - 1 };
+ if (HEADING_RE.test(t)) return { end: i - 1 };
+ if (FENCE_RE.test(t)) return { end: i - 1 };
+ // Continuation of list: list item or indented continuation.
+ if (LIST_ITEM_RE.test(t) || /^\s{2,}/.test(t)) {
+ i++;
+ continue;
+ }
+ return { end: i - 1 };
+ }
+ return { end: lines.length - 1 };
+}
+
+function isTableStart(lines: ReadonlyArray<Line>, start: number): boolean {
+ const a = lines[start];
+ const b = lines[start + 1];
+ if (!a || !b) return false;
+ if (!a.text.includes("|")) return false;
+ return TABLE_SEP_RE.test(b.text);
+}
+
+function readTable(lines: ReadonlyArray<Line>, start: number): { end: number } {
+ // Header (start), separator (start+1), then rows until blank or non-pipe line.
+ let i = start + 2;
+ while (i < lines.length) {
+ const t = lines[i]!.text;
+ if (isBlank(t)) return { end: i - 1 };
+ if (!t.includes("|")) return { end: i - 1 };
+ i++;
+ }
+ return { end: lines.length - 1 };
+}
+
+function readParagraph(lines: ReadonlyArray<Line>, start: number): { end: number } {
+ let i = start + 1;
+ while (i < lines.length) {
+ const t = lines[i]!.text;
+ if (isBlank(t)) return { end: i - 1 };
+ if (HEADING_RE.test(t)) return { end: i - 1 };
+ if (FENCE_RE.test(t)) return { end: i - 1 };
+ if (LIST_ITEM_RE.test(t)) return { end: i - 1 };
+ if (isTableStart(lines, i)) return { end: i - 1 };
+ i++;
+ }
+ return { end: lines.length - 1 };
+}
+
+function splitIntoBlocks(lines: ReadonlyArray<Line>): { blocks: Block[]; warnings: string[] } {
+ const blocks: Block[] = [];
+ const warnings: string[] = [];
+ const { block: fm, remaining, warnings: fmWarn } = extractFrontmatter(lines);
+ warnings.push(...fmWarn);
+ if (fm) blocks.push(fm);
+
+ let i = 0;
+ while (i < remaining.length) {
+ const text = remaining[i]!.text;
+ if (isBlank(text)) {
+ i++;
+ continue;
+ }
+ if (FENCE_RE.test(text)) {
+ const { end } = readCodeBlock(remaining, i);
+ blocks.push(makeBlock("code", [...remaining.slice(i, end + 1)]));
+ i = end + 1;
+ continue;
+ }
+ if (HEADING_RE.test(text)) {
+ blocks.push(makeBlock("heading", [remaining[i]!]));
+ i++;
+ continue;
+ }
+ if (LIST_ITEM_RE.test(text)) {
+ const { end } = readList(remaining, i);
+ blocks.push(makeBlock("list", [...remaining.slice(i, end + 1)]));
+ i = end + 1;
+ continue;
+ }
+ if (isTableStart(remaining, i)) {
+ const { end } = readTable(remaining, i);
+ blocks.push(makeBlock("table", [...remaining.slice(i, end + 1)]));
+ i = end + 1;
+ continue;
+ }
+ const { end } = readParagraph(remaining, i);
+ blocks.push(makeBlock("paragraph", [...remaining.slice(i, end + 1)]));
+ i = end + 1;
+ }
+
+ return { blocks, warnings };
+}
+
+interface DraftChunk {
+ blocks: Block[];
+ startLine: number;
+ endLine: number;
+ tokens: number;
+}
+
+function newDraft(): DraftChunk {
+ return { blocks: [], startLine: 0, endLine: 0, tokens: 0 };
+}
+
+function takeOverlap(prev: DraftChunk, overlapTokens: number): Line[] {
+ if (overlapTokens <= 0 || prev.blocks.length === 0) return [];
+ // Flatten lines of previous chunk's body in order, then take tail until token budget.
+ const allLines: Line[] = [];
+ for (const b of prev.blocks) for (const l of b.lines) allLines.push(l);
+ let acc = 0;
+ const tail: Line[] = [];
+ for (let i = allLines.length - 1; i >= 0; i--) {
+ const t = countTokens(allLines[i]!.text);
+ if (acc + t > overlapTokens && tail.length > 0) break;
+ tail.unshift(allLines[i]!);
+ acc += t;
+ }
+ return tail;
+}
+
+function emitChunk(
+ index: number,
+ overlap: ReadonlyArray<Line>,
+ draft: DraftChunk,
+): MarkdownChunk {
+ const bodyLines: Line[] = [];
+ for (const b of draft.blocks) for (const l of b.lines) bodyLines.push(l);
+ const allText = [
+ ...overlap.map((l) => l.text),
+ ...bodyLines.map((l) => l.text),
+ ].join("\n");
+ const tokenCount = countTokens(allText);
+ return Object.freeze({
+ chunkIndex: index,
+ content: allText,
+ startLine: bodyLines[0]?.num ?? overlap[0]?.num ?? 1,
+ endLine: bodyLines[bodyLines.length - 1]?.num ?? overlap[overlap.length - 1]?.num ?? 1,
+ tokenCount,
+ });
+}
+
+function packBlocks(
+ blocks: ReadonlyArray<Block>,
+ opts: { maxTokens: number; minTokens: number; overlapTokens: number },
+): MarkdownChunk[] {
+ const out: MarkdownChunk[] = [];
+ let draft = newDraft();
+ let prevForOverlap: DraftChunk | null = null;
+ let pendingOverlap: Line[] = [];
+
+ const flush = () => {
+ if (draft.blocks.length === 0) return;
+ const chunk = emitChunk(out.length, pendingOverlap, draft);
+ out.push(chunk);
+ prevForOverlap = draft;
+ pendingOverlap = takeOverlap(prevForOverlap, opts.overlapTokens);
+ draft = newDraft();
+ };
+
+ for (const block of blocks) {
+ if (block.kind === "frontmatter") {
+ // Flush any in-progress (shouldn't happen β€” frontmatter is first), then
+ // emit the frontmatter as its own chunk *without* overlap since nothing
+ // precedes it.
+ flush();
+ const fmContent = block.lines.map((l) => l.text).join("\n");
+ out.push(
+ Object.freeze({
+ chunkIndex: out.length,
+ content: fmContent,
+ startLine: block.lines[0]!.num,
+ endLine: block.lines[block.lines.length - 1]!.num,
+ tokenCount: countTokens(fmContent),
+ }),
+ );
+ // The next chunk should NOT include frontmatter text as overlap.
+ prevForOverlap = null;
+ pendingOverlap = [];
+ continue;
+ }
+
+ // Atomic large block β€” emit as its own chunk.
+ if (block.tokenCount > opts.maxTokens) {
+ flush();
+ const standalone: DraftChunk = {
+ blocks: [block],
+ startLine: block.lines[0]!.num,
+ endLine: block.lines[block.lines.length - 1]!.num,
+ tokens: block.tokenCount,
+ };
+ const chunk = emitChunk(out.length, pendingOverlap, standalone);
+ out.push(chunk);
+ prevForOverlap = standalone;
+ pendingOverlap = takeOverlap(prevForOverlap, opts.overlapTokens);
+ continue;
+ }
+
+ if (block.kind === "heading") {
+ const hasNonHeading = draft.blocks.some((b) => b.kind !== "heading");
+ if (hasNonHeading && draft.tokens >= opts.minTokens) {
+ flush();
+ }
+ draft.blocks.push(block);
+ draft.tokens += block.tokenCount;
+ continue;
+ }
+
+ if (draft.tokens + block.tokenCount > opts.maxTokens && draft.blocks.length > 0) {
+ flush();
+ }
+ draft.blocks.push(block);
+ draft.tokens += block.tokenCount;
+ }
+
+ flush();
+ return out;
+}
+
+function stripQuotes(s: string): string {
+ const t = s.trim();
+ if (t.length >= 2 && ((t.startsWith('"') && t.endsWith('"')) || (t.startsWith("'") && t.endsWith("'")))) {
+ return t.slice(1, -1);
+ }
+ return t;
+}
+
+function resolveTitle(
+ frontmatterBlock: Block | null,
+ blocks: ReadonlyArray<Block>,
+ filenameBase: string | null,
+): string | null {
+ if (frontmatterBlock) {
+ for (const l of frontmatterBlock.lines) {
+ const m = l.text.match(/^title:\s*(.*)$/);
+ if (m) {
+ const v = stripQuotes(m[1] ?? "");
+ if (v) return v;
+ }
+ }
+ }
+ for (const b of blocks) {
+ if (b.kind === "heading") {
+ const text = b.lines[0]!.text.replace(/^#{1,6}\s+/, "").trim();
+ if (text) return text;
+ }
+ if (b.kind !== "frontmatter") break;
+ }
+ if (filenameBase) return filenameBase.replace(/[-_]+/g, " ").trim() || filenameBase;
+ return null;
+}
+
+/**
+ * Two-pass markdown chunker. Pure function: same input β†’ same output.
+ *
+ * Returns `chunks: []` for empty or whitespace-only input; the indexer
+ * still records a `documents` row so the file is tracked.
+ */
+export function chunkMarkdown(
+ text: string,
+ filenameBase: string | null,
+ opts?: ChunkOptions,
+): ChunkResult {
+ const maxTokens = opts?.maxTokens ?? DEFAULT_MAX_TOKENS;
+ const minTokens = opts?.minTokens ?? DEFAULT_MIN_TOKENS;
+ const overlapTokens = opts?.overlapTokens ?? DEFAULT_OVERLAP_TOKENS;
+
+ const allLines = splitLines(text);
+ if (allLines.every((l) => isBlank(l.text))) {
+ return Object.freeze({
+ title: filenameBase ? filenameBase.replace(/[-_]+/g, " ") : null,
+ chunks: Object.freeze([]),
+ warnings: Object.freeze([]),
+ });
+ }
+
+ const { blocks, warnings } = splitIntoBlocks(allLines);
+ const frontmatter = blocks.find((b) => b.kind === "frontmatter") ?? null;
+ const title = resolveTitle(frontmatter, blocks, filenameBase);
+ const chunks = packBlocks(blocks, { maxTokens, minTokens, overlapTokens });
+
+ return Object.freeze({
+ title,
+ chunks: Object.freeze(chunks),
+ warnings: Object.freeze(warnings),
+ });
+}
diff --git a/src/core/search/embeddings/null-provider.ts b/src/core/search/embeddings/null-provider.ts
new file mode 100644
index 00000000..50cdab22
--- /dev/null
+++ b/src/core/search/embeddings/null-provider.ts
@@ -0,0 +1,30 @@
+/**
+ * Sentinel embedding provider that surfaces configuration bugs.
+ *
+ * `embed()` throwing is intentional β€” a caller that reaches this path
+ * has set semantic on the call but the runtime has no provider wired
+ * up. Better to fail loud than to silently no-op and persist no
+ * vectors.
+ */
+
+import { SearchError } from "../types.ts";
+import type { EmbeddingProvider } from "./provider.ts";
+
+export class NullProvider implements EmbeddingProvider {
+ readonly name = "null";
+ readonly model = "";
+ readonly dimension = null;
+
+ embed(_texts: ReadonlyArray<string>): Promise<number[][]> {
+ return Promise.reject(
+ new SearchError(
+ "EMBEDDING_DISABLED",
+ "embeddings disabled: no provider configured. Set embedding_provider, embedding_base_url, embedding_model, embedding_api_key.",
+ ),
+ );
+ }
+
+ async ping(): Promise<{ ok: false; reason: string }> {
+ return { ok: false, reason: "embeddings disabled" };
+ }
+}
diff --git a/src/core/search/embeddings/openai-compat.ts b/src/core/search/embeddings/openai-compat.ts
new file mode 100644
index 00000000..406bdb2a
--- /dev/null
+++ b/src/core/search/embeddings/openai-compat.ts
@@ -0,0 +1,343 @@
+/**
+ * OpenAI-compatible `/v1/embeddings` provider.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§11.
+ *
+ * Network rules:
+ * - One concurrent batch per semaphore slot (`embedding_concurrency`).
+ * - Each batch contains up to `embedding_batch_size` texts.
+ * - Per-request timeout: `embedding_timeout_ms`.
+ * - Retry on `429`, `5xx`, and network/timeout errors with exponential
+ * backoff `1s/2s` (+ Β±25% jitter), three attempts total. Other 4xx
+ * fail fast.
+ * - Vectors are unit-normalised so cosine similarity equals
+ * `1 - L2Β² / 2` (see ranker).
+ *
+ * Provider-shaped, never-prompt-shaped errors. The caller decides
+ * whether to surface (`--semantic`) or warn (implicit).
+ */
+
+import { SearchError } from "../types.ts";
+import type { ResolvedEmbeddingConfig } from "../types.ts";
+import type { EmbeddingProvider } from "./provider.ts";
+
+const RETRYABLE_STATUSES = new Set([429, 500, 502, 503, 504]);
+
+interface OpenAiEmbeddingResponse {
+ readonly data: ReadonlyArray<{ readonly embedding: ReadonlyArray<number>; readonly index: number }>;
+ readonly model?: string;
+}
+
+function sleep(ms: number): Promise<void> {
+ return new Promise((res) => setTimeout(res, ms));
+}
+
+function jittered(base: number): number {
+ const jitter = base * 0.25 * (Math.random() * 2 - 1); // Β±25%
+ return Math.max(0, base + jitter);
+}
+
+class Semaphore {
+ private permits: number;
+ private readonly waiters: Array<() => void> = [];
+ constructor(n: number) {
+ this.permits = Math.max(1, n | 0);
+ }
+ async acquire(): Promise<void> {
+ if (this.permits > 0) {
+ this.permits--;
+ return;
+ }
+ await new Promise<void>((res) => this.waiters.push(res));
+ this.permits--;
+ }
+ release(): void {
+ this.permits++;
+ const next = this.waiters.shift();
+ if (next) next();
+ }
+}
+
+function unitNormaliseInPlace(v: number[]): number[] {
+ let s = 0;
+ for (const x of v) s += x * x;
+ const norm = Math.sqrt(s);
+ if (norm === 0) return v;
+ for (let i = 0; i < v.length; i++) v[i] = (v[i] ?? 0) / norm;
+ return v;
+}
+
+function chunkArray<T>(arr: ReadonlyArray<T>, size: number): T[][] {
+ const step = Math.max(1, size | 0);
+ const out: T[][] = [];
+ for (let i = 0; i < arr.length; i += step) out.push(arr.slice(i, i + step) as T[]);
+ return out;
+}
+
+interface ResolvedHttp {
+ readonly url: string;
+ readonly apiKey: string;
+}
+
+function resolveHttp(config: ResolvedEmbeddingConfig): ResolvedHttp {
+ if (!config.baseUrl) {
+ throw new SearchError("INVALID_INPUT", "embedding_base_url is required when semantic is enabled");
+ }
+ if (!config.model) {
+ throw new SearchError("INVALID_INPUT", "embedding_model is required when semantic is enabled");
+ }
+ if (!config.apiKey) {
+ throw new SearchError(
+ "EMBEDDING_KEY_MISSING",
+ "embedding_api_key is required when semantic is enabled",
+ );
+ }
+ const base = config.baseUrl.replace(/\/+$/, "");
+ return { url: `${base}/embeddings`, apiKey: config.apiKey };
+}
+
+export interface OpenAICompatProviderOptions {
+ /** Override default `[1000, 2000]` ms backoffs (used by tests). */
+ readonly backoffMs?: ReadonlyArray<number>;
+}
+
+export class OpenAICompatProvider implements EmbeddingProvider {
+ readonly name = "openai-compat";
+ readonly model: string;
+ private _dimension: number | null;
+ private readonly config: ResolvedEmbeddingConfig;
+ private readonly http: ResolvedHttp;
+ private readonly backoffMs: ReadonlyArray<number>;
+
+ constructor(config: ResolvedEmbeddingConfig, opts?: OpenAICompatProviderOptions) {
+ this.config = config;
+ this.http = resolveHttp(config);
+ this.model = config.model!;
+ this._dimension = config.dimension;
+ this.backoffMs = opts?.backoffMs ?? [1000, 2000];
+ }
+
+ get dimension(): number | null {
+ return this._dimension;
+ }
+
+ async embed(texts: ReadonlyArray<string>): Promise<number[][]> {
+ if (texts.length === 0) return [];
+ const batches = chunkArray(
+ texts.map((t, i) => ({ text: t, originalIndex: i })),
+ this.config.batchSize,
+ );
+ const sem = new Semaphore(this.config.concurrency);
+ const out: number[][] = new Array(texts.length);
+
+ // Shared abort controller cancels in-flight siblings and skips
+ // queued ones the moment any batch fails. Without this, a 4xx on
+ // batch #1 still bills the remaining N-1 batches that were already
+ // scheduled by Promise.all.
+ const cancel = new AbortController();
+
+ const tasks = batches.map(async (batch) => {
+ await sem.acquire();
+ try {
+ if (cancel.signal.aborted) return;
+ const vectors = await this.embedBatchWithRetry(batch.map((b) => b.text), {
+ parentSignal: cancel.signal,
+ });
+ for (let i = 0; i < vectors.length; i++) {
+ out[batch[i]!.originalIndex] = vectors[i]!;
+ }
+ } finally {
+ sem.release();
+ }
+ });
+
+ try {
+ // Promise.all rejects on first rejection. Abort siblings so they
+ // don't keep hitting the provider; then await the cancelled tasks
+ // via allSettled to avoid unhandled-rejection warnings.
+ try {
+ await Promise.all(tasks);
+ } catch (firstError) {
+ cancel.abort();
+ await Promise.allSettled(tasks);
+ if (firstError instanceof SearchError) throw firstError;
+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", String(firstError));
+ }
+ } finally {
+ // No-op if already settled; keeps semaphore + listeners cleaned up.
+ cancel.abort();
+ }
+ return out;
+ }
+
+ async ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }> {
+ try {
+ const vectors = await this.embedBatchWithRetry(["check"], { maxAttempts: 1 });
+ const v = vectors[0];
+ if (!v) return { ok: false, reason: "empty response" };
+ return { ok: true, dimension: v.length };
+ } catch (e) {
+ const reason = e instanceof Error ? e.message : String(e);
+ return { ok: false, reason };
+ }
+ }
+
+ private async embedBatchWithRetry(
+ texts: string[],
+ opts?: { maxAttempts?: number; parentSignal?: AbortSignal },
+ ): Promise<number[][]> {
+ const maxAttempts = opts?.maxAttempts ?? 3;
+ let lastError: SearchError | null = null;
+ for (let attempt = 1; attempt <= maxAttempts; attempt++) {
+ if (opts?.parentSignal?.aborted) {
+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", "embed cancelled");
+ }
+ try {
+ return await this.embedBatchOnce(texts, opts?.parentSignal);
+ } catch (e) {
+ const err = this.classifyError(e);
+ if (!err.retriable || attempt >= maxAttempts) throw err.error;
+ lastError = err.error;
+ this.retriesSeen++;
+ const wait = jittered(this.backoffMs[attempt - 1] ?? (this.backoffMs[this.backoffMs.length - 1] ?? 4000));
+ await sleep(wait);
+ }
+ }
+ throw lastError ?? new SearchError("EMBEDDING_PROVIDER_HTTP", "retry loop exhausted");
+ }
+
+ /** Retries since construction; `consumeRetryCount()` resets the tally. */
+ retriesSeen = 0;
+
+ /** Read-and-reset retry counter β€” indexer uses this to populate IndexStats. */
+ consumeRetryCount(): number {
+ const n = this.retriesSeen;
+ this.retriesSeen = 0;
+ return n;
+ }
+
+ private async embedBatchOnce(
+ texts: string[],
+ parentSignal?: AbortSignal,
+ ): Promise<number[][]> {
+ const controller = new AbortController();
+ const timer = setTimeout(() => controller.abort(), this.config.timeoutMs);
+ const onParentAbort = () => controller.abort();
+ if (parentSignal) {
+ if (parentSignal.aborted) controller.abort();
+ else parentSignal.addEventListener("abort", onParentAbort, { once: true });
+ }
+ let response: Response;
+ try {
+ response = await fetch(this.http.url, {
+ method: "POST",
+ headers: {
+ "content-type": "application/json",
+ authorization: `Bearer ${this.http.apiKey}`,
+ },
+ body: JSON.stringify({
+ model: this.model,
+ input: texts,
+ encoding_format: "float",
+ }),
+ signal: controller.signal,
+ });
+ } catch (e) {
+ const cause = e instanceof Error ? e : new Error(String(e));
+ if (cause.name === "AbortError") {
+ if (parentSignal?.aborted) {
+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", "embed cancelled");
+ }
+ throw new SearchError(
+ "EMBEDDING_PROVIDER_TIMEOUT",
+ `embedding request timed out after ${this.config.timeoutMs}ms`,
+ );
+ }
+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `network error: ${cause.message}`);
+ } finally {
+ clearTimeout(timer);
+ if (parentSignal) parentSignal.removeEventListener("abort", onParentAbort);
+ }
+
+ if (!response.ok) {
+ const body = await response.text().catch(() => "");
+ const head = body.slice(0, 300);
+ throw new SearchError(
+ "EMBEDDING_PROVIDER_HTTP",
+ `embedding HTTP ${response.status}: ${head || response.statusText}`,
+ );
+ }
+
+ let json: OpenAiEmbeddingResponse;
+ try {
+ json = (await response.json()) as OpenAiEmbeddingResponse;
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : String(e);
+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `embedding response not JSON: ${msg}`);
+ }
+
+ if (!Array.isArray(json.data) || json.data.length !== texts.length) {
+ throw new SearchError(
+ "EMBEDDING_PROVIDER_HTTP",
+ `embedding response shape: expected ${texts.length} vectors, got ${json.data?.length ?? "none"}`,
+ );
+ }
+
+ const ordered: number[][] = new Array(texts.length);
+ for (const item of json.data) {
+ if (typeof item.index !== "number" || item.index < 0 || item.index >= texts.length) {
+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `embedding response: out-of-range index ${item.index}`);
+ }
+ if (!Array.isArray(item.embedding)) {
+ throw new SearchError(
+ "EMBEDDING_PROVIDER_HTTP",
+ `embedding response: data[${item.index}].embedding is not an array`,
+ );
+ }
+ const arr = (item.embedding as ReadonlyArray<number>).slice();
+ if (this._dimension === null) {
+ this._dimension = arr.length;
+ } else if (this._dimension !== arr.length) {
+ throw new SearchError(
+ "EMBEDDING_DIMENSION_MISMATCH",
+ `embedding dimension changed mid-batch: expected ${this._dimension}, got ${arr.length}`,
+ );
+ }
+ ordered[item.index] = unitNormaliseInPlace(arr);
+ }
+ for (let i = 0; i < ordered.length; i++) {
+ if (!Array.isArray(ordered[i])) {
+ throw new SearchError("EMBEDDING_PROVIDER_HTTP", `embedding response: missing vector for index ${i}`);
+ }
+ }
+ return ordered;
+ }
+
+ private classifyError(e: unknown): { retriable: boolean; error: SearchError } {
+ if (e instanceof SearchError) {
+ // Parent-cancelled batches surface as a synthetic
+ // EMBEDDING_PROVIDER_HTTP "embed cancelled". Retrying them would
+ // race against the abort and re-spend budget on a batch the
+ // outer Promise already gave up on.
+ if (e.message.includes("embed cancelled")) {
+ return { retriable: false, error: e };
+ }
+ if (e.code === "EMBEDDING_PROVIDER_TIMEOUT") return { retriable: true, error: e };
+ if (e.code === "EMBEDDING_PROVIDER_HTTP") {
+ // Parse status from message if present.
+ const m = e.message.match(/HTTP (\d+)/);
+ if (m) {
+ const status = Number(m[1]);
+ return { retriable: RETRYABLE_STATUSES.has(status), error: e };
+ }
+ // Network errors fall here β€” retriable.
+ return { retriable: true, error: e };
+ }
+ return { retriable: false, error: e };
+ }
+ return {
+ retriable: false,
+ error: new SearchError("EMBEDDING_PROVIDER_HTTP", String(e)),
+ };
+ }
+}
diff --git a/src/core/search/embeddings/provider.ts b/src/core/search/embeddings/provider.ts
new file mode 100644
index 00000000..10f519d9
--- /dev/null
+++ b/src/core/search/embeddings/provider.ts
@@ -0,0 +1,44 @@
+/**
+ * Embedding-provider abstraction. The store does not call into HTTP;
+ * the indexer drives provider.embed() and passes raw vectors to
+ * store.vecUpsert().
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§11.
+ */
+
+import { SearchError } from "../types.ts";
+import type { ResolvedEmbeddingConfig } from "../types.ts";
+
+export interface EmbeddingProvider {
+ readonly name: string;
+ readonly model: string;
+ readonly dimension: number | null;
+ embed(texts: ReadonlyArray<string>): Promise<number[][]>;
+ ping(): Promise<{ ok: true; dimension: number } | { ok: false; reason: string }>;
+ /**
+ * Optional read-and-reset of provider-internal retry tally. The
+ * indexer consumes this after each `embed()` to populate
+ * `IndexStats.embeddingsRetries`. Providers that never retry
+ * (NullProvider, MockEmbeddingProvider) leave this undefined.
+ */
+ consumeRetryCount?(): number;
+}
+
+export function makeProvider(config: ResolvedEmbeddingConfig): EmbeddingProvider {
+ // Lazy imports to keep the module graph small for users who never
+ // enable semantic search.
+ if (config.provider === "disabled" || !config.enabled) {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports
+ const { NullProvider } = require("./null-provider.ts") as typeof import("./null-provider.ts");
+ return new NullProvider();
+ }
+ if (config.provider === "openai-compat") {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports
+ const { OpenAICompatProvider } = require("./openai-compat.ts") as typeof import("./openai-compat.ts");
+ return new OpenAICompatProvider(config);
+ }
+ throw new SearchError(
+ "INVALID_INPUT",
+ `unknown embedding_provider '${String((config as { provider?: string }).provider)}'`,
+ );
+}
diff --git a/src/core/search/fts.ts b/src/core/search/fts.ts
new file mode 100644
index 00000000..09e47439
--- /dev/null
+++ b/src/core/search/fts.ts
@@ -0,0 +1,37 @@
+/**
+ * FTS5 query construction and keyword retrieval.
+ *
+ * The user-typed query is treated as a bag of phrase tokens joined by
+ * implicit AND. Each token is wrapped in double quotes so FTS5's
+ * operator words (`AND`, `OR`, `NOT`, `NEAR`) and metacharacters
+ * (`*`, `(`, `)`, `:`, `^`) lose their special meaning. Internal `"`
+ * is escaped as `""` per the FTS5 grammar.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§7.
+ */
+
+import type { KeywordHit, Store } from "./store.ts";
+
+export function buildFtsMatch(rawQuery: string): string {
+ const tokens = rawQuery
+ .split(/\s+/)
+ .map((t) => t.trim())
+ .filter((t) => t.length > 0);
+ if (tokens.length === 0) return "";
+ return tokens.map((t) => `"${t.replace(/"/g, '""')}"`).join(" ");
+}
+
+export interface RunFtsOptions {
+ readonly limit: number;
+ readonly pathPrefix?: string | null;
+}
+
+export function runFtsQuery(
+ store: Store,
+ rawQuery: string,
+ opts: RunFtsOptions,
+): KeywordHit[] {
+ const match = buildFtsMatch(rawQuery);
+ if (match === "") return [];
+ return store.keywordTopK(match, opts);
+}
diff --git a/src/core/search/index.ts b/src/core/search/index.ts
new file mode 100644
index 00000000..2b33a487
--- /dev/null
+++ b/src/core/search/index.ts
@@ -0,0 +1,293 @@
+/**
+ * Public surface for `src/core/search/*`: config resolution plus the
+ * index/search/status/check entry points used by CLI and MCP layers.
+ */
+
+import { discoverConfig } from "../config.ts";
+import { resolveIndexPath } from "./paths.ts";
+import { SearchError } from "./types.ts";
+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "./types.ts";
+
+type SearchConfigOverrides = Partial<Omit<ResolvedSearchConfig, "ignorePaths" | "semantic">> & {
+ readonly ignorePaths?: ReadonlyArray<string>;
+ readonly semantic?: Partial<ResolvedEmbeddingConfig>;
+};
+
+export type {
+ BrainSearchResult,
+ IndexCheckReport,
+ IndexStats,
+ IndexStatusSnapshot,
+ ResolvedEmbeddingConfig,
+ ResolvedSearchConfig,
+ SearchErrorCode,
+ SearchOptions,
+ SearchOutcome,
+} from "./types.ts";
+export { SearchError, SEARCH_ERROR_CODES } from "./types.ts";
+
+export { resolveIndexPath } from "./paths.ts";
+export {
+ indexVault,
+ reindexVault,
+ indexStatus,
+ indexCheck,
+ type IndexVaultOptions,
+ type IndexProgressEvent,
+} from "./indexer.ts";
+export { search } from "./search.ts";
+
+const DEFAULT_IGNORE_PATHS = [
+ ".git",
+ "node_modules",
+ ".open-second-brain",
+ ".obsidian/cache",
+ ".trash",
+ ".stversions",
+];
+
+const DEFAULTS = {
+ chunkSize: 800,
+ chunkOverlap: 100,
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ provider: "openai-compat" as const,
+ timeoutMs: 10_000,
+ concurrency: 4,
+ batchSize: 32,
+};
+
+function envOrConfig(
+ env: NodeJS.ProcessEnv,
+ config: Readonly<Record<string, string>>,
+ envKey: string,
+ configKey: string,
+): string | null {
+ const e = env[envKey];
+ if (e !== undefined && e !== "") return e;
+ const c = config[configKey];
+ if (c !== undefined && c !== "") return c;
+ return null;
+}
+
+function parseInteger(
+ raw: string | null,
+ fallback: number,
+ fieldName: string,
+ opts?: { readonly min?: number; readonly max?: number },
+): number {
+ if (raw === null) return fallback;
+ const n = Number(raw);
+ if (!Number.isFinite(n) || !Number.isInteger(n)) {
+ throw new SearchError("INVALID_INPUT", `${fieldName} must be an integer, got '${raw}'`);
+ }
+ validateIntegerRange(n, fieldName, opts);
+ return n;
+}
+
+function validateIntegerRange(
+ n: number,
+ fieldName: string,
+ opts?: { readonly min?: number; readonly max?: number },
+): void {
+ if (opts?.min !== undefined && n < opts.min) {
+ throw new SearchError("INVALID_INPUT", `${fieldName} must be >= ${opts.min}, got ${n}`);
+ }
+ if (opts?.max !== undefined && n > opts.max) {
+ throw new SearchError("INVALID_INPUT", `${fieldName} must be <= ${opts.max}, got ${n}`);
+ }
+}
+
+function parseFloat01(raw: string | null, fallback: number, fieldName: string): number {
+ if (raw === null) return fallback;
+ const n = Number(raw);
+ if (!Number.isFinite(n) || n < 0 || n > 1) {
+ throw new SearchError(
+ "INVALID_INPUT",
+ `${fieldName} must be a number in [0, 1], got '${raw}'`,
+ );
+ }
+ return n;
+}
+
+function validateWeight(n: number, fieldName: string): void {
+ if (!Number.isFinite(n) || n < 0 || n > 1) {
+ throw new SearchError("INVALID_INPUT", `${fieldName} must be a number in [0, 1], got '${n}'`);
+ }
+}
+
+function validateResolvedConfig(config: ResolvedSearchConfig): void {
+ validateIntegerRange(config.chunkSize, "search_chunk_size", { min: 1 });
+ validateIntegerRange(config.chunkOverlap, "search_chunk_overlap", { min: 0 });
+ if (config.chunkOverlap >= config.chunkSize) {
+ throw new SearchError(
+ "INVALID_INPUT",
+ `search_chunk_overlap must be smaller than search_chunk_size, got ${config.chunkOverlap} >= ${config.chunkSize}`,
+ );
+ }
+ validateWeight(config.keywordWeight, "search_keyword_weight");
+ validateWeight(config.semanticWeight, "search_semantic_weight");
+ if (config.keywordWeight + config.semanticWeight > 1.0 + 1e-9) {
+ throw new SearchError(
+ "INVALID_INPUT",
+ `keyword_weight + semantic_weight must sum to <= 1, got ${config.keywordWeight} + ${config.semanticWeight}`,
+ );
+ }
+ if (config.semantic.dimension !== null) {
+ validateIntegerRange(config.semantic.dimension, "embedding_dimension", { min: 1 });
+ }
+ validateIntegerRange(config.semantic.timeoutMs, "embedding_timeout_ms", { min: 1 });
+ validateIntegerRange(config.semantic.concurrency, "embedding_concurrency", { min: 1 });
+ validateIntegerRange(config.semantic.batchSize, "embedding_batch_size", { min: 1 });
+}
+
+function parseBool(raw: string | null, fallback: boolean, fieldName: string): boolean {
+ if (raw === null) return fallback;
+ if (raw === "true" || raw === "1") return true;
+ if (raw === "false" || raw === "0") return false;
+ throw new SearchError(
+ "INVALID_INPUT",
+ `${fieldName} must be 'true' or 'false', got '${raw}'`,
+ );
+}
+
+function parseProvider(raw: string | null): ResolvedEmbeddingConfig["provider"] {
+ if (raw === null) return DEFAULTS.provider;
+ if (raw === "openai-compat" || raw === "disabled") return raw;
+ throw new SearchError(
+ "INVALID_INPUT",
+ `embedding_provider must be 'openai-compat' or 'disabled', got '${raw}'`,
+ );
+}
+
+function parseIgnorePaths(raw: string | null): string[] {
+ if (raw === null) return [...DEFAULT_IGNORE_PATHS];
+ return raw
+ .split(",")
+ .map((s) => s.trim())
+ .filter((s) => s.length > 0);
+}
+
+export function resolveSearchConfig(opts: {
+ vault: string;
+ configPath?: string;
+ overrides?: SearchConfigOverrides;
+}): ResolvedSearchConfig {
+ const env = process.env;
+ const config: Readonly<Record<string, string>> = opts.configPath
+ ? discoverConfig(opts.configPath).data
+ : {};
+
+ const dbPath = resolveIndexPath(
+ opts.vault,
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_DB", "search_db_path"),
+ );
+
+ const chunkSize = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CHUNK_SIZE", "search_chunk_size"),
+ DEFAULTS.chunkSize,
+ "search_chunk_size",
+ { min: 1 },
+ );
+ const chunkOverlap = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_CHUNK_OVERLAP", "search_chunk_overlap"),
+ DEFAULTS.chunkOverlap,
+ "search_chunk_overlap",
+ { min: 0 },
+ );
+ const keywordWeight = parseFloat01(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_KW_WEIGHT", "search_keyword_weight"),
+ DEFAULTS.keywordWeight,
+ "search_keyword_weight",
+ );
+ const semanticWeight = parseFloat01(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SEM_WEIGHT", "search_semantic_weight"),
+ DEFAULTS.semanticWeight,
+ "search_semantic_weight",
+ );
+ if (keywordWeight + semanticWeight > 1.0 + 1e-9) {
+ throw new SearchError(
+ "INVALID_INPUT",
+ `keyword_weight + semantic_weight must sum to <= 1, got ${keywordWeight} + ${semanticWeight}`,
+ );
+ }
+
+ const ignorePaths = parseIgnorePaths(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_IGNORE", "search_ignore_paths"),
+ );
+
+ const semanticEnabled = parseBool(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_SEARCH_SEMANTIC", "search_semantic_enabled"),
+ false,
+ "search_semantic_enabled",
+ );
+ const provider = parseProvider(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER", "embedding_provider"),
+ );
+ const baseUrl = envOrConfig(
+ env,
+ config,
+ "OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL",
+ "embedding_base_url",
+ );
+ const model = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_MODEL", "embedding_model");
+ const apiKey = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_KEY", "embedding_api_key");
+ const dimRaw = envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_DIM", "embedding_dimension");
+ const dimension =
+ dimRaw === null ? null : parseInteger(dimRaw, 0, "embedding_dimension", { min: 1 });
+ const timeoutMs = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_TIMEOUT", "embedding_timeout_ms"),
+ DEFAULTS.timeoutMs,
+ "embedding_timeout_ms",
+ { min: 1 },
+ );
+ const concurrency = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_CONCURRENCY", "embedding_concurrency"),
+ DEFAULTS.concurrency,
+ "embedding_concurrency",
+ { min: 1 },
+ );
+ const batchSize = parseInteger(
+ envOrConfig(env, config, "OPEN_SECOND_BRAIN_EMBEDDING_BATCH", "embedding_batch_size"),
+ DEFAULTS.batchSize,
+ "embedding_batch_size",
+ { min: 1 },
+ );
+
+ const semantic: ResolvedEmbeddingConfig = Object.freeze({
+ enabled: semanticEnabled,
+ provider,
+ baseUrl,
+ model,
+ apiKey,
+ dimension,
+ timeoutMs,
+ concurrency,
+ batchSize,
+ });
+
+ const base: ResolvedSearchConfig = Object.freeze({
+ vault: opts.vault,
+ dbPath,
+ ignorePaths: Object.freeze([...ignorePaths]),
+ chunkSize,
+ chunkOverlap,
+ keywordWeight,
+ semanticWeight,
+ semantic,
+ });
+
+ validateResolvedConfig(base);
+
+ if (!opts.overrides) return base;
+ const merged = Object.freeze({
+ ...base,
+ ...opts.overrides,
+ semantic: Object.freeze({ ...base.semantic, ...(opts.overrides.semantic ?? {}) }),
+ ignorePaths: opts.overrides.ignorePaths
+ ? Object.freeze([...opts.overrides.ignorePaths])
+ : base.ignorePaths,
+ });
+ validateResolvedConfig(merged);
+ return merged;
+}
diff --git a/src/core/search/indexer.ts b/src/core/search/indexer.ts
new file mode 100644
index 00000000..59b43354
--- /dev/null
+++ b/src/core/search/indexer.ts
@@ -0,0 +1,525 @@
+/**
+ * Index lifecycle orchestrator.
+ *
+ * `indexVault` is the incremental path: walk β†’ diff against stored
+ * documents β†’ upsert/replace/delete. `reindexVault` is the atomic
+ * rebuild: write to `brain.sqlite.new`, then a same-file rename swap
+ * with `.bak` retention.
+ *
+ * `indexStatus` and `indexCheck` are the read-side diagnostics that
+ * power `o2b search status|check` and the MCP status enrichment.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§6, Β§8,
+ * Β§13, Β§15.
+ */
+
+import { createHash } from "node:crypto";
+import {
+ accessSync,
+ constants,
+ mkdirSync,
+ readFileSync,
+ renameSync,
+ statSync,
+ unlinkSync,
+} from "node:fs";
+import { basename, dirname } from "node:path";
+
+import { chunkMarkdown } from "./chunker.ts";
+import { makeProvider } from "./embeddings/provider.ts";
+import { extractLinks } from "./links.ts";
+import { Store } from "./store.ts";
+import { SearchError } from "./types.ts";
+import { walkVault } from "./walker.ts";
+import { withTimeout } from "./with-timeout.ts";
+import type { ChunkInput, LinkInput } from "./store.ts";
+import type {
+ IndexCheckReport,
+ IndexStats,
+ IndexStatusSnapshot,
+ ResolvedSearchConfig,
+} from "./types.ts";
+
+// ─────────────────────────────────────────────────────────────────────────────
+
+export interface IndexProgressEvent {
+ readonly path: string;
+ readonly kind: "added" | "updated" | "unchanged" | "deleted" | "error";
+ readonly message?: string;
+}
+
+export interface IndexVaultOptions {
+ readonly embeddings?: boolean;
+ /** When true, every file is reindexed even if hash + mtime match. */
+ readonly force?: boolean;
+ readonly onFile?: (event: IndexProgressEvent) => void;
+}
+
+interface MutableStats {
+ added: number;
+ updated: number;
+ unchanged: number;
+ deleted: number;
+ chunksTotal: number;
+ embeddingsComputed: number;
+ embeddingsRetries: number;
+ errors: Array<{ readonly path: string; readonly message: string }>;
+}
+
+function newStats(): MutableStats {
+ return {
+ added: 0,
+ updated: 0,
+ unchanged: 0,
+ deleted: 0,
+ chunksTotal: 0,
+ embeddingsComputed: 0,
+ embeddingsRetries: 0,
+ errors: [],
+ };
+}
+
+function freezeStats(s: MutableStats, durationMs: number): IndexStats {
+ return Object.freeze({
+ added: s.added,
+ updated: s.updated,
+ unchanged: s.unchanged,
+ deleted: s.deleted,
+ chunksTotal: s.chunksTotal,
+ embeddingsComputed: s.embeddingsComputed,
+ embeddingsRetries: s.embeddingsRetries,
+ errors: Object.freeze([...s.errors]),
+ durationMs,
+ });
+}
+
+function sha256(text: string): string {
+ return createHash("sha256").update(text).digest("hex");
+}
+
+const UTF8_FATAL = new TextDecoder("utf-8", { fatal: true });
+
+function readUtf8(absPath: string): string {
+ const buf = readFileSync(absPath);
+ try {
+ return UTF8_FATAL.decode(buf);
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : String(e);
+ throw new SearchError("INVALID_INPUT", `file is not valid UTF-8: ${absPath} (${msg})`);
+ }
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// indexVault β€” incremental
+// ─────────────────────────────────────────────────────────────────────────────
+
+export async function indexVault(
+ config: ResolvedSearchConfig,
+ opts?: IndexVaultOptions,
+): Promise<IndexStats> {
+ return indexInto(config, opts);
+}
+
+async function indexInto(
+ config: ResolvedSearchConfig,
+ opts?: IndexVaultOptions,
+ storeOverride?: Store,
+): Promise<IndexStats> {
+ const t0 = Date.now();
+ const ownsStore = !storeOverride;
+ const store = storeOverride ?? (await Store.open(config, { mode: "write" }));
+ const stats = newStats();
+
+ try {
+ const existing = store.listDocuments();
+ const seen = new Set<string>();
+
+ for (const file of walkVault(config)) {
+ // Mark seen FIRST. If anything downstream throws (read fault,
+ // chunker bug, transient FS error), the file must not look
+ // "missing" to the deletion sweep below β€” that would wipe a
+ // present file from the index just because a single read failed.
+ seen.add(file.relPath);
+ try {
+ const content = readUtf8(file.absPath);
+ const contentHash = sha256(content);
+ const mtimeSec = Math.floor(file.stat.mtimeMs / 1000);
+ const prev = existing.get(file.relPath);
+
+ if (!opts?.force && prev && prev.contentHash === contentHash && prev.mtime === mtimeSec) {
+ stats.unchanged++;
+ opts?.onFile?.({ path: file.relPath, kind: "unchanged" });
+ continue;
+ }
+
+ const filenameBase = basename(file.relPath, ".md");
+ const chunkResult = chunkMarkdown(content, filenameBase, {
+ maxTokens: config.chunkSize,
+ overlapTokens: config.chunkOverlap,
+ });
+ for (const w of chunkResult.warnings) {
+ stats.errors.push({ path: file.relPath, message: w });
+ }
+
+ const docId = store.upsertDocument({
+ path: file.relPath,
+ title: chunkResult.title,
+ contentHash,
+ mtime: mtimeSec,
+ size: file.stat.size,
+ });
+
+ const chunkInputs: ChunkInput[] = chunkResult.chunks.map((c) => ({
+ chunkIndex: c.chunkIndex,
+ content: c.content,
+ contentHash: sha256(c.content),
+ startLine: c.startLine,
+ endLine: c.endLine,
+ tokenCount: c.tokenCount,
+ }));
+ const chunkIds = store.replaceChunks(docId, chunkInputs);
+
+ const links: LinkInput[] = [];
+ for (let i = 0; i < chunkResult.chunks.length; i++) {
+ const cid = chunkIds[i]!;
+ const extracted = extractLinks(chunkResult.chunks[i]!.content);
+ for (const l of extracted) {
+ links.push({
+ sourceChunkId: cid,
+ targetPath: l.targetPath,
+ linkText: l.linkText,
+ linkType: l.linkType,
+ });
+ }
+ }
+ store.replaceLinks(docId, links);
+
+ stats.chunksTotal += chunkInputs.length;
+ if (!prev) {
+ stats.added++;
+ opts?.onFile?.({ path: file.relPath, kind: "added" });
+ } else {
+ stats.updated++;
+ opts?.onFile?.({ path: file.relPath, kind: "updated" });
+ }
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : String(e);
+ stats.errors.push({ path: file.relPath, message: msg });
+ opts?.onFile?.({ path: file.relPath, kind: "error", message: msg });
+ }
+ }
+
+ for (const [path] of existing) {
+ if (!seen.has(path)) {
+ store.deleteDocument(path);
+ stats.deleted++;
+ opts?.onFile?.({ path, kind: "deleted" });
+ }
+ }
+
+ store.resolveLinkTargets();
+
+ if (opts?.embeddings) {
+ await populateEmbeddings(store, config, stats);
+ }
+
+ const now = new Date().toISOString();
+ store.setState("last_indexed_at", now);
+ if (opts?.force) store.setState("last_full_index_at", now);
+
+ return freezeStats(stats, Date.now() - t0);
+ } finally {
+ if (ownsStore) await store.close();
+ }
+}
+
+async function populateEmbeddings(
+ store: Store,
+ config: ResolvedSearchConfig,
+ stats: MutableStats,
+): Promise<void> {
+ if (!config.semantic.enabled) {
+ throw new SearchError(
+ "EMBEDDING_DISABLED",
+ "set search_semantic_enabled=true and embedding_* keys to compute embeddings",
+ );
+ }
+ if (!store.vecLoaded()) {
+ throw new SearchError(
+ "VEC_EXTENSION_UNAVAILABLE",
+ "sqlite-vec did not load; cannot store embeddings",
+ );
+ }
+ if (!config.semantic.apiKey) {
+ throw new SearchError(
+ "EMBEDDING_KEY_MISSING",
+ "embedding_api_key is required when computing embeddings",
+ );
+ }
+
+ const pending = store.findChunksWithoutEmbeddings();
+ if (pending.length === 0) return;
+
+ const provider = makeProvider(config.semantic);
+ const model = config.semantic.model ?? provider.model;
+ const batchSize = Math.max(1, config.semantic.batchSize);
+ // Hand the provider a super-batch sized to fully saturate its
+ // internal `embedding_concurrency` semaphore. Without this multiplier
+ // the indexer's outer loop would serialise provider.embed() calls and
+ // the configured concurrency would never kick in.
+ const superBatch = batchSize * Math.max(1, config.semantic.concurrency);
+
+ for (let i = 0; i < pending.length; i += superBatch) {
+ const batch = pending.slice(i, i + superBatch);
+ const texts = batch.map((p) => p.content);
+ const vectors = await provider.embed(texts);
+ stats.embeddingsRetries += provider.consumeRetryCount?.() ?? 0;
+ if (vectors.length !== batch.length) {
+ throw new SearchError(
+ "EMBEDDING_PROVIDER_HTTP",
+ `provider returned ${vectors.length} vectors for ${batch.length} inputs`,
+ );
+ }
+ // Lock in the auto-detected dimension on the very first batch.
+ const dim = provider.dimension ?? vectors[0]?.length ?? 0;
+ if (dim <= 0) {
+ throw new SearchError("EMBEDDING_DIMENSION_MISMATCH", "provider returned vectors of zero length");
+ }
+ store.ensureEmbeddingModel(model, dim);
+ for (let j = 0; j < batch.length; j++) {
+ const chunkId = batch[j]!.chunkId;
+ const vec = vectors[j]!;
+ const embHash = sha256(vec.map((x) => x.toFixed(8)).join(","));
+ store.vecUpsert(chunkId, vec, model, dim, embHash);
+ stats.embeddingsComputed++;
+ }
+ }
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// reindexVault β€” atomic full rebuild
+// ─────────────────────────────────────────────────────────────────────────────
+
+export async function reindexVault(
+ config: ResolvedSearchConfig,
+ opts?: IndexVaultOptions,
+): Promise<IndexStats> {
+ const newPath = config.dbPath + ".new";
+ const bakPath = config.dbPath + ".bak";
+
+ mkdirSync(dirname(config.dbPath), { recursive: true });
+ tryUnlink(newPath);
+
+ // Build into the temp file with an override config.
+ const tempConfig: ResolvedSearchConfig = Object.freeze({
+ ...config,
+ dbPath: newPath,
+ });
+ const stats = await indexVault(tempConfig, { ...opts, force: true });
+
+ // Same-directory rename swap. The two renames are each atomic on
+ // POSIX; the gap between them is the only crash window, which the
+ // .bak restore on Store.open handles.
+ tryUnlink(bakPath);
+ tryRename(config.dbPath, bakPath); // no-op (ENOENT) on fresh reindex
+ renameSync(newPath, config.dbPath); // must succeed β€” `newPath` was just built
+ return stats;
+}
+
+/** `unlinkSync` that tolerates ENOENT (file already absent). */
+function tryUnlink(p: string): void {
+ try {
+ unlinkSync(p);
+ } catch (e) {
+ if (!isEnoent(e)) throw e;
+ }
+}
+
+/** `renameSync` that tolerates ENOENT on the source. */
+function tryRename(from: string, to: string): void {
+ try {
+ renameSync(from, to);
+ } catch (e) {
+ if (!isEnoent(e)) throw e;
+ }
+}
+
+function isEnoent(e: unknown): boolean {
+ return typeof e === "object" && e !== null && (e as { code?: string }).code === "ENOENT";
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// indexStatus
+// ─────────────────────────────────────────────────────────────────────────────
+
+export async function indexStatus(config: ResolvedSearchConfig): Promise<IndexStatusSnapshot> {
+ let store: Store;
+ try {
+ store = await Store.open(config, { mode: "read" });
+ } catch (e) {
+ if (e instanceof SearchError && e.code === "INDEX_MISSING") {
+ return Object.freeze({
+ indexPath: config.dbPath,
+ exists: false,
+ schemaVersion: null,
+ documents: 0,
+ chunks: 0,
+ embeddings: 0,
+ staleEmbeddings: 0,
+ embeddingModel: null,
+ embeddingDimension: null,
+ vecExtension: "unknown" as const,
+ semanticEnabled: config.semantic.enabled,
+ embeddingKeyPresent: !!config.semantic.apiKey,
+ lastIndexedAt: null,
+ lastFullIndexAt: null,
+ warnings: Object.freeze([]),
+ });
+ }
+ throw e;
+ }
+ try {
+ const counts = store.counts();
+ const model = store.getState("embedding_model");
+ const dimRaw = store.getState("embedding_dimension");
+ const dim = dimRaw ? Number(dimRaw) : null;
+ const last = store.getState("last_indexed_at");
+ const full = store.getState("last_full_index_at");
+
+ const warnings: string[] = [];
+ if (config.semantic.enabled && !store.vecLoaded()) {
+ warnings.push("sqlite-vec unavailable; semantic search disabled this session");
+ }
+ if (config.semantic.enabled && !config.semantic.apiKey) {
+ warnings.push("embedding_api_key not configured; semantic search disabled");
+ }
+
+ return Object.freeze({
+ indexPath: config.dbPath,
+ exists: true,
+ schemaVersion: store.schemaVersion(),
+ documents: counts.documents,
+ chunks: counts.chunks,
+ embeddings: counts.embeddings,
+ staleEmbeddings: counts.staleEmbeddings,
+ embeddingModel: model,
+ embeddingDimension: dim,
+ vecExtension: store.vecLoaded() ? ("loaded" as const) : ("unavailable" as const),
+ semanticEnabled: config.semantic.enabled,
+ embeddingKeyPresent: !!config.semantic.apiKey,
+ lastIndexedAt: last,
+ lastFullIndexAt: full,
+ warnings: Object.freeze(warnings),
+ });
+ } finally {
+ await store.close();
+ }
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// indexCheck
+// ─────────────────────────────────────────────────────────────────────────────
+
+function isDirectoryWritable(dir: string): boolean {
+ try {
+ // `recursive: true` is idempotent β€” no separate existsSync check.
+ mkdirSync(dir, { recursive: true });
+ accessSync(dir, constants.W_OK);
+ return true;
+ } catch {
+ return false;
+ }
+}
+
+export async function indexCheck(config: ResolvedSearchConfig): Promise<IndexCheckReport> {
+ const warnings: string[] = [];
+ const fatal: string[] = [];
+
+ let vaultReadable = false;
+ try {
+ if (statSync(config.vault).isDirectory()) {
+ vaultReadable = true;
+ } else {
+ fatal.push(`vault path exists but is not a directory: ${config.vault}`);
+ }
+ } catch {
+ fatal.push(`vault not readable: ${config.vault}`);
+ }
+
+ const dir = dirname(config.dbPath);
+ const indexDirWritable = isDirectoryWritable(dir);
+ if (!indexDirWritable) fatal.push(`index directory not writable: ${dir}`);
+
+ let sqliteOk = false;
+ let fts5Ok = false;
+ let vecExtension: "loaded" | "unavailable" | "not-attempted" = "not-attempted";
+ try {
+ // Use an in-memory DB so the check never touches the real index.
+ const { Database } = await import("bun:sqlite");
+ const db = new Database(":memory:");
+ sqliteOk = true;
+ try {
+ db.exec(
+ "CREATE VIRTUAL TABLE probe USING fts5(content, tokenize='unicode61 remove_diacritics 2')",
+ );
+ fts5Ok = true;
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : String(e);
+ fatal.push(`FTS5 not available: ${msg}`);
+ }
+ if (config.semantic.enabled) {
+ try {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports
+ const vec = require("sqlite-vec") as { getLoadablePath(): string };
+ db.loadExtension(vec.getLoadablePath());
+ db.query("SELECT vec_version()").get();
+ vecExtension = "loaded";
+ } catch (e) {
+ vecExtension = "unavailable";
+ warnings.push(`sqlite-vec unavailable: ${e instanceof Error ? e.message : String(e)}`);
+ }
+ }
+ db.close();
+ } catch (e) {
+ fatal.push(`bun:sqlite open failed: ${e instanceof Error ? e.message : String(e)}`);
+ }
+
+ const embeddingKeyResolved = !!(config.semantic.enabled && config.semantic.apiKey);
+ if (config.semantic.enabled && !config.semantic.apiKey) {
+ warnings.push("embedding_api_key not configured");
+ }
+
+ let providerReachable: boolean | null = null;
+ let providerReason: string | null = null;
+ if (config.semantic.enabled && embeddingKeyResolved) {
+ try {
+ const provider = makeProvider(config.semantic);
+ const probe = await withTimeout(provider.ping(), 5_000);
+ if (probe.ok) {
+ providerReachable = true;
+ } else {
+ providerReachable = false;
+ providerReason = probe.reason;
+ warnings.push(`embedding provider check failed: ${probe.reason}`);
+ }
+ } catch (e) {
+ providerReachable = false;
+ providerReason = e instanceof Error ? e.message : String(e);
+ warnings.push(`embedding provider check failed: ${providerReason}`);
+ }
+ }
+
+ return Object.freeze({
+ vaultReadable,
+ indexDirWritable,
+ sqliteOk,
+ fts5Ok,
+ vecExtension,
+ embeddingKeyResolved,
+ providerReachable,
+ providerReason,
+ warnings: Object.freeze(warnings),
+ fatal: Object.freeze(fatal),
+ });
+}
+
diff --git a/src/core/search/links.ts b/src/core/search/links.ts
new file mode 100644
index 00000000..eaea43a3
--- /dev/null
+++ b/src/core/search/links.ts
@@ -0,0 +1,100 @@
+/**
+ * Extract wikilinks, markdown links, and tags from a chunk of Markdown.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§5 (links
+ * table). Returns the link rows that `store.replaceLinks` expects, but
+ * leaves the source-chunk binding to the indexer (it knows which
+ * chunk produced the content).
+ *
+ * Code fences and inline-code spans are stripped before extraction so
+ * a code sample mentioning `[[foo]]` or `#hash` does not become a real
+ * link. This is best-effort: nested fenced fences are rare and we err
+ * on the side of stripping too much rather than capturing junk links.
+ */
+
+export type LinkType = "wikilink" | "markdown_link" | "tag";
+
+export interface ExtractedLink {
+ readonly targetPath: string | null;
+ readonly linkText: string | null;
+ readonly linkType: LinkType;
+}
+
+const CODE_FENCE_RE = /(^|\n)(```|~~~)[^\n]*\n[\s\S]*?(?:\n(?:```|~~~)[^\n]*|$)/g;
+const INLINE_CODE_RE = /`[^`\n]*`/g;
+const WIKILINK_RE = /\[\[([^\]\n|]+?)(?:\|([^\]\n]+))?\]\]/g;
+// Negative lookbehind so `![alt](url)` image embeds are NOT captured as
+// markdown_link. CodeRabbit caught this regression on PR #15.
+const MD_LINK_RE = /(?<!!)\[([^\]\n]*)\]\(([^)\n\s]+)(?:\s+"[^"\n]*")?\)/g;
+// Obsidian-style tag: #word where word starts with a letter/_ and may contain
+// letters, digits, dashes, underscores, and '/' for hierarchy.
+const TAG_RE = /(^|[^\w\/])#([A-Za-z_][\w\-/]*)/g;
+
+function stripCode(text: string): string {
+ let out = text.replace(CODE_FENCE_RE, "\n");
+ out = out.replace(INLINE_CODE_RE, " ");
+ return out;
+}
+
+function isUrl(target: string): boolean {
+ return /^[a-z][a-z0-9+.\-]*:/i.test(target) || target.startsWith("//");
+}
+
+function isMailto(target: string): boolean {
+ return target.toLowerCase().startsWith("mailto:");
+}
+
+function dedupe(links: ExtractedLink[]): ExtractedLink[] {
+ const seen = new Set<string>();
+ const out: ExtractedLink[] = [];
+ for (const l of links) {
+ const key = `${l.linkType}|${l.targetPath ?? ""}|${l.linkText ?? ""}`;
+ if (seen.has(key)) continue;
+ seen.add(key);
+ out.push(l);
+ }
+ return out;
+}
+
+export function extractLinks(content: string): ExtractedLink[] {
+ const cleaned = stripCode(content);
+ const out: ExtractedLink[] = [];
+
+ for (const m of cleaned.matchAll(WIKILINK_RE)) {
+ const target = (m[1] ?? "").trim();
+ const alt = m[2] ? m[2].trim() : null;
+ if (target === "") continue;
+ out.push({
+ targetPath: target,
+ linkText: alt,
+ linkType: "wikilink",
+ });
+ }
+
+ for (const m of cleaned.matchAll(MD_LINK_RE)) {
+ const text = (m[1] ?? "").trim();
+ const target = (m[2] ?? "").trim();
+ if (target === "" || isUrl(target) || isMailto(target)) continue;
+ // Strip the optional `#anchor` fragment for path matching.
+ const hashIdx = target.indexOf("#");
+ const path = hashIdx >= 0 ? target.slice(0, hashIdx) : target;
+ if (path === "") continue;
+ out.push({
+ targetPath: path,
+ linkText: text || null,
+ linkType: "markdown_link",
+ });
+ }
+
+ for (const m of cleaned.matchAll(TAG_RE)) {
+ const tag = (m[2] ?? "").trim();
+ if (tag === "") continue;
+ out.push({
+ targetPath: null,
+ linkText: tag,
+ linkType: "tag",
+ });
+ }
+
+ return dedupe(out);
+}
diff --git a/src/core/search/paths.ts b/src/core/search/paths.ts
new file mode 100644
index 00000000..eb658ac0
--- /dev/null
+++ b/src/core/search/paths.ts
@@ -0,0 +1,16 @@
+/**
+ * Resolve the on-disk location of the search index file.
+ *
+ * Default: `<vault>/.open-second-brain/brain.sqlite`. Overridable
+ * through CLI `--db` or config `search_db_path`.
+ */
+
+import { join } from "node:path";
+
+export function resolveIndexPath(vault: string, override: string | null): string {
+ if (override !== null) {
+ const trimmed = override.trim();
+ if (trimmed !== "") return trimmed;
+ }
+ return join(vault, ".open-second-brain", "brain.sqlite");
+}
diff --git a/src/core/search/ranker.ts b/src/core/search/ranker.ts
new file mode 100644
index 00000000..2e1bf6d8
--- /dev/null
+++ b/src/core/search/ranker.ts
@@ -0,0 +1,216 @@
+/**
+ * Pure ranking function. Combines normalised BM25, cosine semantic
+ * similarity, link-graph boost, and recency boost into the final score.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§7.
+ *
+ * The ranker imports no I/O modules. Callers (search.ts) gather the
+ * inputs from the store and pass them in. This makes it trivially
+ * testable and substitutable.
+ */
+
+import type { KeywordHit, SemanticHit, HydratedChunk } from "./store.ts";
+import type { BrainSearchResult } from "./types.ts";
+
+export interface RankerInputs {
+ readonly keyword: ReadonlyArray<KeywordHit>;
+ readonly semantic: ReadonlyArray<SemanticHit>;
+ readonly hydrated: ReadonlyMap<number, HydratedChunk>;
+ /** For each chunkId: set of OTHER document ids linking to its document. */
+ readonly inboundLinkSources: ReadonlyMap<number, ReadonlySet<number>>;
+ /** For each chunkId: the tag set of its document. */
+ readonly tagsByDoc: ReadonlyMap<number, ReadonlySet<string>>;
+}
+
+export interface RankerOptions {
+ readonly keywordWeight: number;
+ readonly semanticWeight: number;
+ readonly limit: number;
+ /** Unix-ms reference time for recency. Defaults to Date.now(). */
+ readonly nowMs?: number;
+ /** When false, semantic_score is ignored regardless of inputs. */
+ readonly semanticEnabled?: boolean;
+}
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+
+function clamp01(x: number): number {
+ if (!Number.isFinite(x)) return 0;
+ if (x < 0) return 0;
+ if (x > 1) return 1;
+ return x;
+}
+
+/** Min-max normalise BM25 within the candidate set. Lower BM25 is better. */
+function normalizeBm25(hits: ReadonlyArray<KeywordHit>): Map<number, number> {
+ const out = new Map<number, number>();
+ if (hits.length === 0) return out;
+ // FTS5 bm25() returns smaller-is-better values (often negative). We invert
+ // to "larger is better" by negating, then min-max.
+ const scores = hits.map((h) => -h.bm25);
+ const min = Math.min(...scores);
+ const max = Math.max(...scores);
+ if (max === min) {
+ for (const h of hits) out.set(h.chunkId, 1);
+ return out;
+ }
+ hits.forEach((h, i) => {
+ out.set(h.chunkId, (scores[i]! - min) / (max - min));
+ });
+ return out;
+}
+
+/** Map L2-on-unit-vectors distance β†’ cosine similarity in [0, 1]. */
+function semanticFromDistance(distance: number): number {
+ const sim = 1 - (distance * distance) / 2;
+ return clamp01(sim);
+}
+
+function recencyBoost(mtime: number, nowMs: number): number {
+ const ageMs = Math.max(0, nowMs - mtime * 1000);
+ const ageDays = ageMs / DAY_MS;
+ if (ageDays <= 7) return 0.05;
+ if (ageDays <= 30) return 0.025;
+ if (ageDays <= 90) return 0.01;
+ return 0;
+}
+
+interface Candidate {
+ chunkId: number;
+ documentId: number;
+ keywordScore: number;
+ semanticScore: number;
+ searchType: "keyword" | "semantic" | "hybrid";
+ mtime: number;
+}
+
+export function rankResults(inputs: RankerInputs, opts: RankerOptions): BrainSearchResult[] {
+ const nowMs = opts.nowMs ?? Date.now();
+ const semanticEnabled = opts.semanticEnabled !== false;
+
+ const kwNorm = normalizeBm25(inputs.keyword);
+
+ const semNorm = new Map<number, number>();
+ if (semanticEnabled) {
+ for (const h of inputs.semantic) {
+ semNorm.set(h.chunkId, semanticFromDistance(h.distance));
+ }
+ }
+
+ const candidates = new Map<number, Candidate>();
+ for (const h of inputs.keyword) {
+ candidates.set(h.chunkId, {
+ chunkId: h.chunkId,
+ documentId: h.documentId,
+ keywordScore: kwNorm.get(h.chunkId) ?? 0,
+ semanticScore: 0,
+ searchType: "keyword",
+ mtime: inputs.hydrated.get(h.chunkId)?.mtime ?? 0,
+ });
+ }
+ if (semanticEnabled) {
+ for (const h of inputs.semantic) {
+ const existing = candidates.get(h.chunkId);
+ if (existing) {
+ existing.semanticScore = semNorm.get(h.chunkId) ?? 0;
+ existing.searchType = "hybrid";
+ } else {
+ candidates.set(h.chunkId, {
+ chunkId: h.chunkId,
+ documentId: h.documentId,
+ keywordScore: 0,
+ semanticScore: semNorm.get(h.chunkId) ?? 0,
+ searchType: "semantic",
+ mtime: inputs.hydrated.get(h.chunkId)?.mtime ?? 0,
+ });
+ }
+ }
+ }
+
+ // Cross-result tables for boosts.
+ const candidateChunks = Array.from(candidates.values());
+ const candidateDocIds = new Set(candidateChunks.map((c) => c.documentId));
+
+ // Build a per-document tag map so the tag boost counts distinct docs,
+ // not chunks. Without this dedup a doc with K candidate chunks would
+ // inflate every other candidate's tag count K-fold.
+ const tagsByDocId = new Map<number, ReadonlySet<string>>();
+ for (const c of candidateChunks) {
+ if (tagsByDocId.has(c.documentId)) continue;
+ const t = inputs.tagsByDoc.get(c.chunkId);
+ if (t && t.size > 0) tagsByDocId.set(c.documentId, t);
+ }
+
+ function linkBoostFor(c: Candidate): number {
+ const sources = inputs.inboundLinkSources.get(c.chunkId);
+ if (!sources || sources.size === 0) return 0;
+ let count = 0;
+ for (const s of sources) {
+ if (s === c.documentId) continue;
+ if (candidateDocIds.has(s)) count++;
+ }
+ const raw = count * 0.02;
+ return Math.min(0.03, raw);
+ }
+
+ function tagBoostFor(c: Candidate): number {
+ const mine = tagsByDocId.get(c.documentId);
+ if (!mine || mine.size === 0) return 0;
+ let count = 0;
+ for (const [otherDocId, theirs] of tagsByDocId) {
+ if (otherDocId === c.documentId) continue;
+ for (const tag of mine) {
+ if (theirs.has(tag)) {
+ count++;
+ break;
+ }
+ }
+ }
+ const raw = count * 0.01;
+ return Math.min(0.02, raw);
+ }
+
+ const ranked: BrainSearchResult[] = [];
+ for (const c of candidateChunks) {
+ const hyd = inputs.hydrated.get(c.chunkId);
+ if (!hyd) continue;
+ const link = linkBoostFor(c);
+ const tag = tagBoostFor(c);
+ const linkBoost = Math.min(0.05, link + tag);
+ const recency = recencyBoost(c.mtime, nowMs);
+ const weighted =
+ opts.keywordWeight * c.keywordScore +
+ (semanticEnabled ? opts.semanticWeight : 0) * c.semanticScore;
+ const score = clamp01(weighted + linkBoost + recency);
+
+ ranked.push(
+ Object.freeze({
+ documentId: c.documentId,
+ chunkId: c.chunkId,
+ path: hyd.path,
+ title: hyd.title,
+ content: hyd.content,
+ startLine: hyd.startLine,
+ endLine: hyd.endLine,
+ score,
+ keywordScore: c.keywordScore,
+ semanticScore: c.semanticScore,
+ linkBoost,
+ recencyBoost: recency,
+ searchType: c.searchType,
+ }),
+ );
+ }
+
+ // Tie-break per design Β§7: final_score desc, keywordScore desc, mtime desc, chunkId asc.
+ ranked.sort((a, b) => {
+ if (b.score !== a.score) return b.score - a.score;
+ if (b.keywordScore !== a.keywordScore) return b.keywordScore - a.keywordScore;
+ const am = inputs.hydrated.get(a.chunkId)?.mtime ?? 0;
+ const bm = inputs.hydrated.get(b.chunkId)?.mtime ?? 0;
+ if (bm !== am) return bm - am;
+ return a.chunkId - b.chunkId;
+ });
+
+ return ranked.slice(0, Math.max(1, opts.limit));
+}
diff --git a/src/core/search/schema.ts b/src/core/search/schema.ts
new file mode 100644
index 00000000..2a17e7bc
--- /dev/null
+++ b/src/core/search/schema.ts
@@ -0,0 +1,212 @@
+/**
+ * DDL and migrations for the search index. The SQLite database itself
+ * is managed by `store.ts`; this module is data + pure functions over
+ * a connection so future minor versions can append migrations without
+ * touching the store class.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§5.
+ */
+
+import type { Database } from "bun:sqlite";
+
+import { SearchError } from "./types.ts";
+
+/**
+ * Latest schema version this code understands. A DB with a value above
+ * this raises `SCHEMA_MISMATCH` on open β€” the operator must reindex
+ * with a newer binary.
+ */
+export const LATEST_SCHEMA_VERSION = 1;
+
+const DDL_V1 = `
+CREATE TABLE IF NOT EXISTS documents (
+ id INTEGER PRIMARY KEY,
+ path TEXT NOT NULL UNIQUE,
+ title TEXT,
+ content_hash TEXT NOT NULL,
+ mtime INTEGER NOT NULL,
+ size INTEGER NOT NULL,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ indexed_at TEXT NOT NULL
+);
+CREATE INDEX IF NOT EXISTS idx_documents_path ON documents(path);
+CREATE INDEX IF NOT EXISTS idx_documents_mtime ON documents(mtime);
+
+CREATE TABLE IF NOT EXISTS chunks (
+ id INTEGER PRIMARY KEY,
+ document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
+ chunk_index INTEGER NOT NULL,
+ content TEXT NOT NULL,
+ content_hash TEXT NOT NULL,
+ start_line INTEGER NOT NULL,
+ end_line INTEGER NOT NULL,
+ token_count INTEGER NOT NULL,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL,
+ UNIQUE(document_id, chunk_index)
+);
+CREATE INDEX IF NOT EXISTS idx_chunks_document ON chunks(document_id);
+
+CREATE VIRTUAL TABLE IF NOT EXISTS chunk_fts USING fts5(
+ content,
+ content='chunks',
+ content_rowid='id',
+ tokenize='unicode61 remove_diacritics 2'
+);
+
+CREATE TRIGGER IF NOT EXISTS chunks_ai AFTER INSERT ON chunks BEGIN
+ INSERT INTO chunk_fts(rowid, content) VALUES (new.id, new.content);
+END;
+CREATE TRIGGER IF NOT EXISTS chunks_ad AFTER DELETE ON chunks BEGIN
+ INSERT INTO chunk_fts(chunk_fts, rowid, content) VALUES('delete', old.id, old.content);
+END;
+CREATE TRIGGER IF NOT EXISTS chunks_au AFTER UPDATE ON chunks BEGIN
+ INSERT INTO chunk_fts(chunk_fts, rowid, content) VALUES('delete', old.id, old.content);
+ INSERT INTO chunk_fts(rowid, content) VALUES (new.id, new.content);
+END;
+
+CREATE TABLE IF NOT EXISTS embeddings (
+ chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,
+ model TEXT NOT NULL,
+ dimension INTEGER NOT NULL,
+ embedding_hash TEXT NOT NULL,
+ created_at TEXT NOT NULL,
+ updated_at TEXT NOT NULL
+);
+CREATE INDEX IF NOT EXISTS idx_embeddings_model ON embeddings(model);
+
+CREATE TABLE IF NOT EXISTS chunk_vec_map (
+ chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,
+ vec_rowid INTEGER NOT NULL UNIQUE
+);
+
+CREATE TABLE IF NOT EXISTS links (
+ id INTEGER PRIMARY KEY,
+ source_document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
+ source_chunk_id INTEGER REFERENCES chunks(id) ON DELETE CASCADE,
+ target_path TEXT,
+ target_document_id INTEGER REFERENCES documents(id) ON DELETE SET NULL,
+ link_text TEXT,
+ link_type TEXT NOT NULL CHECK(link_type IN ('wikilink','markdown_link','tag')),
+ created_at TEXT NOT NULL
+);
+CREATE INDEX IF NOT EXISTS idx_links_source ON links(source_document_id);
+CREATE INDEX IF NOT EXISTS idx_links_target_doc ON links(target_document_id);
+CREATE INDEX IF NOT EXISTS idx_links_target_path ON links(target_path);
+
+CREATE TABLE IF NOT EXISTS index_state (
+ key TEXT PRIMARY KEY,
+ value TEXT NOT NULL,
+ updated_at TEXT NOT NULL
+);
+`;
+
+interface Migration {
+ readonly version: number;
+ readonly up: (db: Database) => void;
+}
+
+export const MIGRATIONS: ReadonlyArray<Migration> = Object.freeze([
+ {
+ version: 1,
+ up(db) {
+ db.exec(DDL_V1);
+ },
+ },
+]);
+
+/** Read the current schema version from `index_state`. Returns 0 if not yet recorded. */
+export function readSchemaVersion(db: Database): number {
+ const row = db
+ .query<{ value: string }, []>(
+ "SELECT value FROM index_state WHERE key = 'schema_version' LIMIT 1",
+ )
+ .get();
+ if (!row) return 0;
+ const n = Number(row.value);
+ if (!Number.isFinite(n) || !Number.isInteger(n) || n < 0) {
+ throw new SearchError(
+ "INDEX_UNREADABLE",
+ `index_state.schema_version is not an integer: '${row.value}'`,
+ );
+ }
+ return n;
+}
+
+function setSchemaVersion(db: Database, version: number): void {
+ const now = new Date().toISOString();
+ db.run(
+ "INSERT INTO index_state(key, value, updated_at) VALUES('schema_version', ?, ?) " +
+ "ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at",
+ [String(version), now],
+ );
+}
+
+/**
+ * Apply every pending migration in a single transaction. On a fresh
+ * database (no `index_state`), v1 creates the entire schema. On an
+ * existing one we run only those with `version > current`.
+ *
+ * Returns the version after migration.
+ */
+export function applyMigrations(db: Database): number {
+ const hasIndexState = db
+ .query<{ name: string }, []>(
+ "SELECT name FROM sqlite_master WHERE type='table' AND name='index_state'",
+ )
+ .get();
+
+ const current = hasIndexState ? readSchemaVersion(db) : 0;
+
+ if (current > LATEST_SCHEMA_VERSION) {
+ throw new SearchError(
+ "SCHEMA_MISMATCH",
+ `index schema version ${current} is newer than this binary supports (${LATEST_SCHEMA_VERSION}). ` +
+ `Run: o2b search reindex`,
+ );
+ }
+
+ if (current === LATEST_SCHEMA_VERSION) return current;
+
+ db.exec("BEGIN");
+ try {
+ for (const migration of MIGRATIONS) {
+ if (migration.version <= current) continue;
+ migration.up(db);
+ setSchemaVersion(db, migration.version);
+ }
+ db.exec("COMMIT");
+ } catch (e) {
+ db.exec("ROLLBACK");
+ throw e;
+ }
+ return LATEST_SCHEMA_VERSION;
+}
+
+/**
+ * Create the `chunk_vec` virtual table at the given dimension. Caller
+ * must have already loaded sqlite-vec; this only issues the DDL.
+ *
+ * No-op if the table already exists.
+ */
+export function ensureVecTable(db: Database, dimension: number): void {
+ if (!Number.isInteger(dimension) || dimension <= 0) {
+ throw new SearchError(
+ "INVALID_INPUT",
+ `vec dimension must be a positive integer, got ${dimension}`,
+ );
+ }
+ const exists = db
+ .query<{ name: string }, []>(
+ "SELECT name FROM sqlite_master WHERE type='table' AND name='chunk_vec'",
+ )
+ .get();
+ if (exists) return;
+ db.exec(`CREATE VIRTUAL TABLE chunk_vec USING vec0(embedding float[${dimension}])`);
+}
+
+/** Drop the `chunk_vec` virtual table if it exists. */
+export function dropVecTable(db: Database): void {
+ db.exec("DROP TABLE IF EXISTS chunk_vec");
+}
diff --git a/src/core/search/search.ts b/src/core/search/search.ts
new file mode 100644
index 00000000..9c797104
--- /dev/null
+++ b/src/core/search/search.ts
@@ -0,0 +1,203 @@
+/**
+ * Public search query: orchestrates FTS5, semantic vector search, link
+ * + recency boosts, and the keyword-only fallback policy.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§7, Β§9.
+ *
+ * The function is async because semantic search requires an HTTP call
+ * (the provider embeds the query). Keyword-only paths stay sync inside
+ * the store but the public surface stays uniform.
+ */
+
+import { makeProvider } from "./embeddings/provider.ts";
+import { runFtsQuery } from "./fts.ts";
+import { rankResults } from "./ranker.ts";
+import { Store } from "./store.ts";
+import { SearchError } from "./types.ts";
+import type {
+ BrainSearchResult,
+ ResolvedSearchConfig,
+ SearchOptions,
+ SearchOutcome,
+} from "./types.ts";
+
+interface SemanticPolicy {
+ /** caller asked for semantic on or off (true), or accepted the default (false). */
+ readonly explicit: boolean;
+ /** does the caller want semantic at all? */
+ readonly wantSemantic: boolean;
+}
+
+function resolveSemanticPolicy(
+ config: ResolvedSearchConfig,
+ opts: SearchOptions,
+): SemanticPolicy {
+ if (opts.keywordOnly === true) {
+ return { explicit: true, wantSemantic: false };
+ }
+ if (opts.semantic === true) return { explicit: true, wantSemantic: true };
+ if (opts.semantic === false) return { explicit: true, wantSemantic: false };
+ return { explicit: false, wantSemantic: config.semantic.enabled };
+}
+
+function assertSafePathPrefix(prefix: string | undefined): string | undefined {
+ if (!prefix) return undefined;
+ if (prefix.includes("..") || prefix.startsWith("/")) {
+ throw new SearchError("INVALID_INPUT", "path_prefix escapes vault");
+ }
+ return prefix;
+}
+
+export async function search(
+ config: ResolvedSearchConfig,
+ opts: SearchOptions,
+): Promise<SearchOutcome> {
+ const query = (opts.query ?? "").trim();
+ if (!query) {
+ throw new SearchError("INVALID_INPUT", "missing required argument: query");
+ }
+ const limit = Math.max(1, Math.min(100, opts.limit ?? 10));
+ const pathPrefix = assertSafePathPrefix(opts.pathPrefix);
+ const policy = resolveSemanticPolicy(config, opts);
+ const warnings: string[] = [];
+
+ const store = await Store.open(config, { mode: "read" });
+ try {
+ // Keyword candidates.
+ const kwHits = runFtsQuery(store, query, {
+ limit: limit * 3,
+ pathPrefix,
+ });
+
+ // Semantic candidates (may be skipped).
+ let semHits: ReturnType<Store["semanticTopK"]> = [];
+ let semanticAttempted = false;
+ if (policy.wantSemantic) {
+ const semOutcome = await runSemanticPhase(store, config, query, {
+ limit: Math.max(limit * 5, 50),
+ pathPrefix,
+ explicit: policy.explicit,
+ });
+ semanticAttempted = semOutcome.attempted;
+ semHits = semOutcome.hits;
+ for (const w of semOutcome.warnings) warnings.push(w);
+ }
+
+ // Hydrate.
+ const allChunkIds = new Set<number>();
+ for (const h of kwHits) allChunkIds.add(h.chunkId);
+ for (const h of semHits) allChunkIds.add(h.chunkId);
+ const idsList = Array.from(allChunkIds);
+ if (idsList.length === 0) {
+ return Object.freeze({
+ results: Object.freeze([] as ReadonlyArray<BrainSearchResult>),
+ warnings: Object.freeze(warnings),
+ total: 0,
+ });
+ }
+
+ const hydrated = store.hydrateChunks(idsList);
+ const inboundLinkSources = store.inboundLinkSources(idsList);
+ const tagsByDoc = store.tagsByChunkDocument(idsList);
+
+ const ranked = rankResults(
+ {
+ keyword: kwHits,
+ semantic: semHits,
+ hydrated,
+ inboundLinkSources,
+ tagsByDoc,
+ },
+ {
+ keywordWeight: opts.keywordWeight ?? config.keywordWeight,
+ semanticWeight: opts.semanticWeight ?? config.semanticWeight,
+ limit,
+ semanticEnabled: policy.wantSemantic && semanticAttempted,
+ },
+ );
+
+ return Object.freeze({
+ results: Object.freeze(ranked),
+ warnings: Object.freeze(warnings),
+ total: ranked.length,
+ });
+ } finally {
+ await store.close();
+ }
+}
+
+interface SemanticPhaseOutcome {
+ readonly attempted: boolean;
+ readonly hits: ReturnType<Store["semanticTopK"]>;
+ readonly warnings: string[];
+}
+
+async function runSemanticPhase(
+ store: Store,
+ config: ResolvedSearchConfig,
+ query: string,
+ opts: { limit: number; pathPrefix: string | undefined; explicit: boolean },
+): Promise<SemanticPhaseOutcome> {
+ const warnings: string[] = [];
+
+ if (!store.vecLoaded()) {
+ if (opts.explicit) {
+ throw new SearchError(
+ "VEC_EXTENSION_UNAVAILABLE",
+ "semantic search unavailable: sqlite-vec extension not loaded",
+ );
+ }
+ warnings.push("sqlite-vec unavailable, semantic disabled this session");
+ return { attempted: false, hits: [], warnings };
+ }
+ if (!config.semantic.enabled) {
+ // Defensive: should be handled at policy layer, but in case caller
+ // forced wantSemantic without enabling, treat as implicit warning.
+ warnings.push("semantic not enabled in config; using keyword-only");
+ return { attempted: false, hits: [], warnings };
+ }
+ if (!config.semantic.apiKey) {
+ if (opts.explicit) {
+ throw new SearchError("EMBEDDING_KEY_MISSING", "embedding key not configured");
+ }
+ warnings.push("embedding key not configured; semantic disabled");
+ return { attempted: false, hits: [], warnings };
+ }
+
+ // Data-state check: are there embeddings to search against at all?
+ const counts = store.counts();
+ if (counts.embeddings === 0) {
+ warnings.push("no compatible embeddings; run: o2b search index --embeddings");
+ return { attempted: false, hits: [], warnings };
+ }
+
+ let queryVec: number[];
+ try {
+ const provider = makeProvider(config.semantic);
+ const vectors = await provider.embed([query]);
+ queryVec = vectors[0] ?? [];
+ } catch (e) {
+ if (opts.explicit) {
+ // Defensive: provider methods are expected to throw SearchError,
+ // but wrap anything else (e.g. an unexpected runtime failure)
+ // so callers always see a typed code rather than a bare Error.
+ if (e instanceof SearchError) throw e;
+ const msg = e instanceof Error ? e.message : String(e);
+ throw new SearchError(
+ "EMBEDDING_PROVIDER_HTTP",
+ `embedding provider failure: ${msg}`,
+ );
+ }
+ const msg = e instanceof Error ? e.message : String(e);
+ warnings.push(`embedding provider unavailable: ${msg}`);
+ return { attempted: false, hits: [], warnings };
+ }
+
+ if (queryVec.length === 0) {
+ warnings.push("embedding provider returned an empty vector; semantic skipped");
+ return { attempted: false, hits: [], warnings };
+ }
+
+ const hits = store.semanticTopK(queryVec, { limit: opts.limit, pathPrefix: opts.pathPrefix });
+ return { attempted: true, hits, warnings };
+}
diff --git a/src/core/search/store.ts b/src/core/search/store.ts
new file mode 100644
index 00000000..f7847df6
--- /dev/null
+++ b/src/core/search/store.ts
@@ -0,0 +1,960 @@
+/**
+ * Single SQL boundary for `src/core/search/*`. Every read or write of
+ * the index passes through this module. Other modules use the typed
+ * surface defined here so that:
+ *
+ * - the SQLite backend can be swapped without touching callers;
+ * - the explicit two-step deletion of `chunk_vec` rows (which the
+ * SQLite FK cascade does NOT reach, see design Β§5) is centralised
+ * in one place;
+ * - the embedding-model fingerprint check runs in one place at open
+ * time.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§5, Β§15.
+ */
+
+import { Database } from "bun:sqlite";
+import { existsSync, mkdirSync, renameSync } from "node:fs";
+import { dirname } from "node:path";
+import lockfile from "proper-lockfile";
+
+import { SearchError } from "./types.ts";
+import type { ResolvedSearchConfig } from "./types.ts";
+import {
+ applyMigrations,
+ dropVecTable,
+ ensureVecTable,
+ LATEST_SCHEMA_VERSION,
+ readSchemaVersion,
+} from "./schema.ts";
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Types
+// ─────────────────────────────────────────────────────────────────────────────
+
+export interface StoreOpenOptions {
+ /** "read" never locks; "write" acquires an exclusive proper-lockfile. */
+ readonly mode: "read" | "write";
+ /** When false, the vec extension is not auto-loaded (used by tests). */
+ readonly loadVec?: boolean;
+}
+
+export interface DocumentInput {
+ readonly path: string; // vault-relative POSIX
+ readonly title: string | null;
+ readonly contentHash: string;
+ readonly mtime: number; // unix seconds
+ readonly size: number;
+}
+
+export interface DocumentSummary {
+ readonly id: number;
+ readonly contentHash: string;
+ readonly mtime: number;
+}
+
+export interface ChunkInput {
+ readonly chunkIndex: number;
+ readonly content: string;
+ readonly contentHash: string;
+ readonly startLine: number;
+ readonly endLine: number;
+ readonly tokenCount: number;
+}
+
+export interface ChunkRow {
+ readonly id: number;
+ readonly documentId: number;
+ readonly chunkIndex: number;
+ readonly content: string;
+ readonly contentHash: string;
+ readonly startLine: number;
+ readonly endLine: number;
+ readonly tokenCount: number;
+}
+
+export interface LinkInput {
+ readonly sourceChunkId: number | null;
+ readonly targetPath: string | null;
+ readonly linkText: string | null;
+ readonly linkType: "wikilink" | "markdown_link" | "tag";
+}
+
+export interface KeywordHit {
+ readonly chunkId: number;
+ readonly documentId: number;
+ /** Lower is better (FTS5 returns negative bm25). */
+ readonly bm25: number;
+}
+
+export interface SemanticHit {
+ readonly chunkId: number;
+ readonly documentId: number;
+ /** L2 distance on unit-normalised vectors. */
+ readonly distance: number;
+}
+
+export interface HydratedChunk {
+ readonly chunkId: number;
+ readonly documentId: number;
+ readonly path: string;
+ readonly title: string | null;
+ readonly content: string;
+ readonly startLine: number;
+ readonly endLine: number;
+ readonly mtime: number;
+}
+
+export interface StoreCounts {
+ readonly documents: number;
+ readonly chunks: number;
+ readonly embeddings: number;
+ /** Embeddings whose `model`/`dimension` no longer match the current config. */
+ readonly staleEmbeddings: number;
+}
+
+export interface ModelChangeOutcome {
+ readonly wasChanged: boolean;
+ readonly previousModel: string | null;
+ readonly previousDimension: number | null;
+ readonly currentModel: string | null;
+ readonly currentDimension: number | null;
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Helpers
+// ─────────────────────────────────────────────────────────────────────────────
+
+function applyPragmas(db: Database): void {
+ db.exec("PRAGMA journal_mode = WAL");
+ db.exec("PRAGMA foreign_keys = ON");
+ db.exec("PRAGMA synchronous = NORMAL");
+}
+
+function ensureFts5(db: Database): void {
+ // Probe FTS5 by attempting a benign expression. bun:sqlite ships with
+ // FTS5 enabled in the embedded amalgamation; failing here means a
+ // custom build without FTS5 and the index is unusable.
+ try {
+ db.query("SELECT fts5_source('chunk_fts')").get();
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : String(e);
+ // Older SQLite returns "no such function" or "no such table" depending
+ // on whether FTS5 is even compiled in. We treat any failure here as
+ // FTS5-unavailable rather than guessing the build flag.
+ if (!/fts5|chunk_fts/i.test(msg)) {
+ throw new SearchError("INDEX_UNREADABLE", `FTS5 probe failed: ${msg}`);
+ }
+ }
+}
+
+function tryLoadVecExtension(db: Database): boolean {
+ try {
+ // sqlite-vec is an optional dependency. Wrap the import + load so
+ // a missing platform package degrades to "extension unavailable"
+ // instead of crashing the process.
+ // eslint-disable-next-line @typescript-eslint/no-require-imports
+ const vec = require("sqlite-vec") as { getLoadablePath(): string };
+ db.loadExtension(vec.getLoadablePath());
+ // Confirm by calling vec_version() β€” guards against partial loads.
+ db.query("SELECT vec_version()").get();
+ return true;
+ } catch {
+ return false;
+ }
+}
+
+function nowIso(): string {
+ return new Date().toISOString();
+}
+
+function vecToBuffer(values: ReadonlyArray<number> | Float32Array): Buffer {
+ const arr = values instanceof Float32Array ? values : Float32Array.from(values);
+ return Buffer.from(arr.buffer, arr.byteOffset, arr.byteLength);
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Store
+// ─────────────────────────────────────────────────────────────────────────────
+
+export class Store {
+ private db: Database;
+ private readonly config: ResolvedSearchConfig;
+ private readonly _vecLoaded: boolean;
+ private readonly release: (() => Promise<void>) | null;
+ private closed = false;
+
+ private constructor(
+ db: Database,
+ config: ResolvedSearchConfig,
+ vecLoaded: boolean,
+ release: (() => Promise<void>) | null,
+ ) {
+ this.db = db;
+ this.config = config;
+ this._vecLoaded = vecLoaded;
+ this.release = release;
+ }
+
+ static async open(config: ResolvedSearchConfig, opts: StoreOpenOptions): Promise<Store> {
+ const loadVec = opts.loadVec !== false;
+
+ // Crash recovery: if the main index is missing but `.bak` is present
+ // from a failed `reindex` rename window, restore it. Stderr notice
+ // (not a thrown error) so existing tooling keeps working.
+ if (!existsSync(config.dbPath)) {
+ const bak = config.dbPath + ".bak";
+ if (existsSync(bak)) {
+ try {
+ renameSync(bak, config.dbPath);
+ // eslint-disable-next-line no-console
+ console.error(`restored search index from ${bak} (previous reindex crash)`);
+ } catch {
+ /* fall through β€” open path below will report INDEX_MISSING */
+ }
+ }
+ }
+
+ if (opts.mode === "read") {
+ if (!existsSync(config.dbPath)) {
+ throw new SearchError(
+ "INDEX_MISSING",
+ `search index not initialised at ${config.dbPath}. Run: o2b search index`,
+ );
+ }
+ let db: Database;
+ try {
+ db = new Database(config.dbPath);
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : String(e);
+ throw new SearchError("INDEX_UNREADABLE", `cannot open ${config.dbPath}: ${msg}`);
+ }
+ try {
+ applyPragmas(db);
+ ensureFts5(db);
+ let version: number;
+ try {
+ version = readSchemaVersion(db);
+ } catch (e) {
+ // A corrupt or non-OSB sqlite file at the index path can make
+ // readSchemaVersion throw raw SQLITE errors (e.g. "no such
+ // table: index_state"). Surface those as a typed
+ // INDEX_UNREADABLE so callers see a code, not a stray Error.
+ if (e instanceof SearchError) throw e;
+ const msg = e instanceof Error ? e.message : String(e);
+ throw new SearchError(
+ "INDEX_UNREADABLE",
+ `cannot read schema_version from ${config.dbPath}: ${msg}`,
+ );
+ }
+ if (version !== LATEST_SCHEMA_VERSION) {
+ throw new SearchError(
+ "SCHEMA_MISMATCH",
+ `index schema version ${version} != ${LATEST_SCHEMA_VERSION}. Run: o2b search reindex`,
+ );
+ }
+ const vecLoaded = loadVec && tryLoadVecExtension(db);
+ return new Store(db, config, vecLoaded, null);
+ } catch (e) {
+ db.close();
+ throw e;
+ }
+ }
+
+ // mode === "write"
+ mkdirSync(dirname(config.dbPath), { recursive: true });
+ if (!existsSync(config.dbPath)) {
+ const seed = new Database(config.dbPath);
+ seed.close();
+ }
+
+ let release: () => Promise<void>;
+ try {
+ release = await lockfile.lock(config.dbPath, {
+ retries: { retries: 3, factor: 1, minTimeout: 1000, maxTimeout: 1000 },
+ stale: 60_000,
+ realpath: false,
+ });
+ } catch (e) {
+ const msg = e instanceof Error ? e.message : String(e);
+ throw new SearchError(
+ "INDEX_LOCKED",
+ `another writer holds the search index lock: ${msg}`,
+ );
+ }
+
+ let db: Database;
+ try {
+ db = new Database(config.dbPath);
+ } catch (e) {
+ await release();
+ const msg = e instanceof Error ? e.message : String(e);
+ throw new SearchError("INDEX_UNREADABLE", `cannot open ${config.dbPath}: ${msg}`);
+ }
+
+ try {
+ applyPragmas(db);
+ applyMigrations(db);
+ ensureFts5(db);
+ const vecLoaded = loadVec && tryLoadVecExtension(db);
+ const store = new Store(db, config, vecLoaded, release);
+ store.ensureEmbeddingModel(config.semantic.model, config.semantic.dimension);
+ return store;
+ } catch (e) {
+ try {
+ db.close();
+ } catch {
+ /* ignore close errors */
+ }
+ await release();
+ throw e;
+ }
+ }
+
+ vecLoaded(): boolean {
+ return this._vecLoaded;
+ }
+
+ schemaVersion(): number {
+ return readSchemaVersion(this.db);
+ }
+
+ async close(): Promise<void> {
+ if (this.closed) return;
+ this.closed = true;
+ try {
+ // Writer mode: consolidate WAL into the main file and switch back to
+ // DELETE journal mode so the `-wal`/`-shm` siblings are removed. This
+ // matters for `reindexVault`: after the temp-file rename swap, any
+ // orphan `*-wal` next to the new main would trigger
+ // SQLITE_IOERR_SHORT_READ on the next open.
+ if (this.release) {
+ try {
+ this.db.exec("PRAGMA wal_checkpoint(TRUNCATE)");
+ this.db.exec("PRAGMA journal_mode = DELETE");
+ } catch (e) {
+ // Don't fail the close, but make the failure visible β€” an
+ // unconsolidated WAL is the exact thing that triggers
+ // SQLITE_IOERR_SHORT_READ after a `reindexVault` rename swap.
+ const msg = e instanceof Error ? e.message : String(e);
+ // eslint-disable-next-line no-console
+ console.error(`search store: WAL consolidation failed on close: ${msg}`);
+ }
+ }
+ this.db.close();
+ } finally {
+ if (this.release) await this.release();
+ }
+ }
+
+ // ── index_state KV ─────────────────────────────────────────────────────────
+
+ getState(key: string): string | null {
+ const row = this.db
+ .query<{ value: string }, [string]>("SELECT value FROM index_state WHERE key = ?")
+ .get(key);
+ return row?.value ?? null;
+ }
+
+ setState(key: string, value: string): void {
+ this.db.run(
+ "INSERT INTO index_state(key, value, updated_at) VALUES (?, ?, ?) " +
+ "ON CONFLICT(key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at",
+ [key, value, nowIso()],
+ );
+ }
+
+ deleteState(key: string): void {
+ this.db.run("DELETE FROM index_state WHERE key = ?", [key]);
+ }
+
+ // ── documents ──────────────────────────────────────────────────────────────
+
+ listDocuments(): Map<string, DocumentSummary> {
+ const rows = this.db
+ .query<{ id: number; path: string; content_hash: string; mtime: number }, []>(
+ "SELECT id, path, content_hash, mtime FROM documents",
+ )
+ .all();
+ const map = new Map<string, DocumentSummary>();
+ for (const r of rows) {
+ map.set(r.path, { id: r.id, contentHash: r.content_hash, mtime: r.mtime });
+ }
+ return map;
+ }
+
+ getDocumentIdByPath(path: string): number | null {
+ const row = this.db
+ .query<{ id: number }, [string]>("SELECT id FROM documents WHERE path = ?")
+ .get(path);
+ return row?.id ?? null;
+ }
+
+ upsertDocument(doc: DocumentInput): number {
+ const now = nowIso();
+ // SQLite RETURNING on INSERT...ON CONFLICT works in 3.35+; bun:sqlite ships modern SQLite.
+ const row = this.db
+ .query<{ id: number }, [string, string | null, string, number, number, string, string, string]>(
+ "INSERT INTO documents(path, title, content_hash, mtime, size, created_at, updated_at, indexed_at) " +
+ "VALUES (?, ?, ?, ?, ?, ?, ?, ?) " +
+ "ON CONFLICT(path) DO UPDATE SET " +
+ " title = excluded.title, " +
+ " content_hash = excluded.content_hash, " +
+ " mtime = excluded.mtime, " +
+ " size = excluded.size, " +
+ " updated_at = excluded.updated_at, " +
+ " indexed_at = excluded.indexed_at " +
+ "RETURNING id",
+ )
+ .get(doc.path, doc.title, doc.contentHash, doc.mtime, doc.size, now, now, now);
+ if (!row) {
+ throw new SearchError("INDEX_UNREADABLE", `upsertDocument returned no id for '${doc.path}'`);
+ }
+ return row.id;
+ }
+
+ /**
+ * Delete a document and everything that hangs off it. The vec rows
+ * are removed first because the FK cascade does not reach the
+ * `chunk_vec` virtual table.
+ */
+ deleteDocument(path: string): void {
+ const id = this.getDocumentIdByPath(path);
+ if (id === null) return;
+ this.purgeVecRowsForDocument(id);
+ this.db.run("DELETE FROM documents WHERE id = ?", [id]);
+ }
+
+ // ── chunks ─────────────────────────────────────────────────────────────────
+
+ getChunksByDocument(documentId: number): ChunkRow[] {
+ const rows = this.db
+ .query<
+ {
+ id: number;
+ document_id: number;
+ chunk_index: number;
+ content: string;
+ content_hash: string;
+ start_line: number;
+ end_line: number;
+ token_count: number;
+ },
+ [number]
+ >(
+ "SELECT id, document_id, chunk_index, content, content_hash, start_line, end_line, token_count " +
+ "FROM chunks WHERE document_id = ? ORDER BY chunk_index",
+ )
+ .all(documentId);
+ return rows.map((r) => ({
+ id: r.id,
+ documentId: r.document_id,
+ chunkIndex: r.chunk_index,
+ content: r.content,
+ contentHash: r.content_hash,
+ startLine: r.start_line,
+ endLine: r.end_line,
+ tokenCount: r.token_count,
+ }));
+ }
+
+ /**
+ * Atomically replace every chunk for a document. Old vec rows are
+ * removed first; FTS5 stays in sync via the chunks_ai/ad/au triggers.
+ * Returns the new chunk ids in `chunkIndex` order.
+ */
+ replaceChunks(documentId: number, chunks: ReadonlyArray<ChunkInput>): number[] {
+ const ids: number[] = [];
+ this.db.exec("BEGIN");
+ try {
+ this.purgeVecRowsForDocument(documentId);
+ this.db.run("DELETE FROM chunks WHERE document_id = ?", [documentId]);
+ const insert = this.db.prepare<
+ { id: number },
+ [number, number, string, string, number, number, number, string, string]
+ >(
+ "INSERT INTO chunks(document_id, chunk_index, content, content_hash, start_line, end_line, token_count, created_at, updated_at) " +
+ "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING id",
+ );
+ const now = nowIso();
+ for (const c of chunks) {
+ const row = insert.get(
+ documentId,
+ c.chunkIndex,
+ c.content,
+ c.contentHash,
+ c.startLine,
+ c.endLine,
+ c.tokenCount,
+ now,
+ now,
+ );
+ if (!row) throw new SearchError("INDEX_UNREADABLE", "chunk insert returned no id");
+ ids.push(row.id);
+ }
+ this.db.exec("COMMIT");
+ } catch (e) {
+ this.db.exec("ROLLBACK");
+ throw e;
+ }
+ return ids;
+ }
+
+ /**
+ * Delete a set of chunks by id. Vec rows removed first.
+ */
+ deleteChunks(chunkIds: ReadonlyArray<number>): void {
+ if (chunkIds.length === 0) return;
+ this.db.exec("BEGIN");
+ try {
+ this.purgeVecRowsByChunkIds(chunkIds);
+ const placeholders = chunkIds.map(() => "?").join(",");
+ this.db.run(`DELETE FROM chunks WHERE id IN (${placeholders})`, chunkIds as number[]);
+ this.db.exec("COMMIT");
+ } catch (e) {
+ this.db.exec("ROLLBACK");
+ throw e;
+ }
+ }
+
+ private purgeVecRowsForDocument(documentId: number): void {
+ if (!this._vecLoaded) return;
+ const vecRows = this.db
+ .query<{ vec_rowid: number }, [number]>(
+ "SELECT vec_rowid FROM chunk_vec_map WHERE chunk_id IN (SELECT id FROM chunks WHERE document_id = ?)",
+ )
+ .all(documentId);
+ if (vecRows.length === 0) return;
+ this.purgeVecRowidsRaw(vecRows.map((r) => r.vec_rowid));
+ }
+
+ private purgeVecRowsByChunkIds(chunkIds: ReadonlyArray<number>): void {
+ if (!this._vecLoaded || chunkIds.length === 0) return;
+ const placeholders = chunkIds.map(() => "?").join(",");
+ const vecRows = this.db
+ .query<{ vec_rowid: number }, number[]>(
+ `SELECT vec_rowid FROM chunk_vec_map WHERE chunk_id IN (${placeholders})`,
+ )
+ .all(...(chunkIds as number[]));
+ if (vecRows.length === 0) return;
+ this.purgeVecRowidsRaw(vecRows.map((r) => r.vec_rowid));
+ }
+
+ private purgeVecRowidsRaw(vecRowids: number[]): void {
+ if (!this._vecLoaded || vecRowids.length === 0) return;
+ const placeholders = vecRowids.map(() => "?").join(",");
+ this.db.run(`DELETE FROM chunk_vec WHERE rowid IN (${placeholders})`, vecRowids);
+ }
+
+ // ── embeddings ─────────────────────────────────────────────────────────────
+
+ /**
+ * Insert or replace a single embedding. The vec table receives the
+ * raw float32 bytes; the metadata row in `embeddings` tracks model /
+ * dimension / hash for stale detection.
+ *
+ * Throws VEC_EXTENSION_UNAVAILABLE if sqlite-vec didn't load. The
+ * caller decides whether to surface this (explicit semantic) or warn
+ * and skip (implicit semantic).
+ */
+ vecUpsert(
+ chunkId: number,
+ vector: ReadonlyArray<number> | Float32Array,
+ model: string,
+ dimension: number,
+ embeddingHash: string,
+ ): void {
+ if (!this._vecLoaded) {
+ throw new SearchError(
+ "VEC_EXTENSION_UNAVAILABLE",
+ "sqlite-vec extension not loaded; cannot store embeddings",
+ );
+ }
+ const len = vector instanceof Float32Array ? vector.length : vector.length;
+ if (len !== dimension) {
+ throw new SearchError(
+ "EMBEDDING_DIMENSION_MISMATCH",
+ `vector dimension ${len} != configured dimension ${dimension}`,
+ );
+ }
+ this.db.exec("BEGIN");
+ try {
+ const existing = this.db
+ .query<{ vec_rowid: number }, [number]>(
+ "SELECT vec_rowid FROM chunk_vec_map WHERE chunk_id = ?",
+ )
+ .get(chunkId);
+ const buf = vecToBuffer(vector);
+ let vecRowid: number;
+ if (existing) {
+ this.db.run("UPDATE chunk_vec SET embedding = ? WHERE rowid = ?", [buf, existing.vec_rowid]);
+ vecRowid = existing.vec_rowid;
+ } else {
+ this.db.run("INSERT INTO chunk_vec(embedding) VALUES (?)", [buf]);
+ const row = this.db
+ .query<{ id: number }, []>("SELECT last_insert_rowid() AS id")
+ .get();
+ if (!row) throw new SearchError("INDEX_UNREADABLE", "chunk_vec insert returned no rowid");
+ vecRowid = row.id;
+ this.db.run(
+ "INSERT INTO chunk_vec_map(chunk_id, vec_rowid) VALUES (?, ?)",
+ [chunkId, vecRowid],
+ );
+ }
+ const now = nowIso();
+ this.db.run(
+ "INSERT INTO embeddings(chunk_id, model, dimension, embedding_hash, created_at, updated_at) " +
+ "VALUES (?, ?, ?, ?, ?, ?) " +
+ "ON CONFLICT(chunk_id) DO UPDATE SET " +
+ " model = excluded.model, dimension = excluded.dimension, " +
+ " embedding_hash = excluded.embedding_hash, updated_at = excluded.updated_at",
+ [chunkId, model, dimension, embeddingHash, now, now],
+ );
+ this.db.exec("COMMIT");
+ } catch (e) {
+ this.db.exec("ROLLBACK");
+ throw e;
+ }
+ }
+
+ getEmbeddingHash(chunkId: number): string | null {
+ const row = this.db
+ .query<{ embedding_hash: string }, [number]>(
+ "SELECT embedding_hash FROM embeddings WHERE chunk_id = ?",
+ )
+ .get(chunkId);
+ return row?.embedding_hash ?? null;
+ }
+
+ /**
+ * Drop all embeddings + vec storage. Used when the configured model
+ * or dimension changes. `chunks` and `chunk_fts` are preserved.
+ */
+ clearEmbeddings(): void {
+ this.db.exec("BEGIN");
+ try {
+ this.db.run("DELETE FROM embeddings");
+ this.db.run("DELETE FROM chunk_vec_map");
+ if (this._vecLoaded) dropVecTable(this.db);
+ this.db.exec("COMMIT");
+ } catch (e) {
+ this.db.exec("ROLLBACK");
+ throw e;
+ }
+ }
+
+ /**
+ * Compare the configured embedding model/dimension with what was
+ * recorded in `index_state` on the last index run. If they differ
+ * and both old + new are non-null, drop embeddings + vec table and
+ * log one line per design Β§13. First-time set just records state.
+ */
+ ensureEmbeddingModel(model: string | null, dimension: number | null): ModelChangeOutcome {
+ const prevModel = this.getState("embedding_model");
+ const prevDimRaw = this.getState("embedding_dimension");
+ const prevDim = prevDimRaw === null ? null : Number(prevDimRaw);
+
+ const modelChanged = prevModel !== null && model !== null && prevModel !== model;
+ const dimChanged =
+ prevDim !== null && dimension !== null && Number.isFinite(prevDim) && prevDim !== dimension;
+
+ if (modelChanged || dimChanged) {
+ this.clearEmbeddings();
+ // eslint-disable-next-line no-console
+ console.error(
+ `embedding model changed from ${prevModel}/${prevDim} to ${model}/${dimension}, embeddings cleared`,
+ );
+ this.deleteState("embedding_model");
+ this.deleteState("embedding_dimension");
+ }
+
+ if (model !== null) this.setState("embedding_model", model);
+ if (dimension !== null) this.setState("embedding_dimension", String(dimension));
+
+ // (Re)create vec table when we know the dimension and vec is loaded.
+ if (this._vecLoaded && dimension !== null) {
+ ensureVecTable(this.db, dimension);
+ }
+
+ return Object.freeze({
+ wasChanged: modelChanged || dimChanged,
+ previousModel: prevModel,
+ previousDimension: prevDim,
+ currentModel: model,
+ currentDimension: dimension,
+ });
+ }
+
+ // ── links ──────────────────────────────────────────────────────────────────
+
+ replaceLinks(sourceDocumentId: number, links: ReadonlyArray<LinkInput>): void {
+ this.db.exec("BEGIN");
+ try {
+ this.db.run("DELETE FROM links WHERE source_document_id = ?", [sourceDocumentId]);
+ if (links.length > 0) {
+ const insert = this.db.prepare<
+ undefined,
+ [number, number | null, string | null, string | null, string, string]
+ >(
+ "INSERT INTO links(source_document_id, source_chunk_id, target_path, link_text, link_type, created_at) " +
+ "VALUES (?, ?, ?, ?, ?, ?)",
+ );
+ const now = nowIso();
+ for (const l of links) {
+ insert.run(sourceDocumentId, l.sourceChunkId, l.targetPath, l.linkText, l.linkType, now);
+ }
+ }
+ this.db.exec("COMMIT");
+ } catch (e) {
+ this.db.exec("ROLLBACK");
+ throw e;
+ }
+ }
+
+ resolveLinkTargets(): void {
+ this.db.run(
+ "UPDATE links SET target_document_id = (SELECT id FROM documents WHERE documents.path = links.target_path) " +
+ "WHERE target_path IS NOT NULL",
+ );
+ }
+
+ // ── search ─────────────────────────────────────────────────────────────────
+
+ /**
+ * Top-K BM25 keyword hits. `fts5Query` is an already-escaped FTS5
+ * MATCH expression; building it is fts.ts's job.
+ */
+ keywordTopK(
+ fts5Query: string,
+ opts: { readonly limit: number; readonly pathPrefix?: string | null },
+ ): KeywordHit[] {
+ const limit = Math.max(1, opts.limit | 0);
+ const prefix = opts.pathPrefix && opts.pathPrefix.length > 0 ? opts.pathPrefix : null;
+
+ if (prefix) {
+ const rows = this.db
+ .query<
+ { chunk_id: number; document_id: number; bm25: number },
+ [string, string, string, number]
+ >(
+ "SELECT c.id AS chunk_id, c.document_id AS document_id, bm25(chunk_fts) AS bm25 " +
+ "FROM chunk_fts " +
+ "JOIN chunks c ON c.id = chunk_fts.rowid " +
+ "JOIN documents d ON d.id = c.document_id " +
+ "WHERE chunk_fts MATCH ? AND substr(d.path, 1, length(?)) = ? " +
+ "ORDER BY bm25 ASC LIMIT ?",
+ )
+ .all(fts5Query, prefix, prefix, limit);
+ return rows.map((r) => ({ chunkId: r.chunk_id, documentId: r.document_id, bm25: r.bm25 }));
+ }
+
+ const rows = this.db
+ .query<{ chunk_id: number; document_id: number; bm25: number }, [string, number]>(
+ "SELECT c.id AS chunk_id, c.document_id AS document_id, bm25(chunk_fts) AS bm25 " +
+ "FROM chunk_fts JOIN chunks c ON c.id = chunk_fts.rowid " +
+ "WHERE chunk_fts MATCH ? ORDER BY bm25 ASC LIMIT ?",
+ )
+ .all(fts5Query, limit);
+ return rows.map((r) => ({ chunkId: r.chunk_id, documentId: r.document_id, bm25: r.bm25 }));
+ }
+
+ semanticTopK(
+ queryVector: ReadonlyArray<number> | Float32Array,
+ opts: { readonly limit: number; readonly pathPrefix?: string | null },
+ ): SemanticHit[] {
+ if (!this._vecLoaded) {
+ throw new SearchError(
+ "VEC_EXTENSION_UNAVAILABLE",
+ "sqlite-vec extension not loaded; semantic search unavailable",
+ );
+ }
+ const limit = Math.max(1, opts.limit | 0);
+ const prefix = opts.pathPrefix && opts.pathPrefix.length > 0 ? opts.pathPrefix : null;
+
+ const buf = vecToBuffer(queryVector);
+ if (prefix) {
+ const rows = this.db
+ .query<
+ { chunk_id: number; document_id: number; distance: number },
+ [Buffer, number, string, string]
+ >(
+ "SELECT m.chunk_id AS chunk_id, c.document_id AS document_id, v.distance AS distance " +
+ "FROM chunk_vec v " +
+ "JOIN chunk_vec_map m ON m.vec_rowid = v.rowid " +
+ "JOIN chunks c ON c.id = m.chunk_id " +
+ "JOIN documents d ON d.id = c.document_id " +
+ "WHERE v.embedding MATCH ? AND k = ? AND substr(d.path, 1, length(?)) = ? " +
+ "ORDER BY v.distance ASC",
+ )
+ .all(buf, limit * 4, prefix, prefix);
+ return rows.slice(0, limit).map((r) => ({
+ chunkId: r.chunk_id,
+ documentId: r.document_id,
+ distance: r.distance,
+ }));
+ }
+
+ const rows = this.db
+ .query<{ chunk_id: number; document_id: number; distance: number }, [Buffer, number]>(
+ "SELECT m.chunk_id AS chunk_id, c.document_id AS document_id, v.distance AS distance " +
+ "FROM chunk_vec v " +
+ "JOIN chunk_vec_map m ON m.vec_rowid = v.rowid " +
+ "JOIN chunks c ON c.id = m.chunk_id " +
+ "WHERE v.embedding MATCH ? AND k = ? " +
+ "ORDER BY v.distance ASC",
+ )
+ .all(buf, limit);
+ return rows.map((r) => ({ chunkId: r.chunk_id, documentId: r.document_id, distance: r.distance }));
+ }
+
+ hydrateChunks(chunkIds: ReadonlyArray<number>): Map<number, HydratedChunk> {
+ const out = new Map<number, HydratedChunk>();
+ if (chunkIds.length === 0) return out;
+ const placeholders = chunkIds.map(() => "?").join(",");
+ const rows = this.db
+ .query<
+ {
+ chunk_id: number;
+ document_id: number;
+ path: string;
+ title: string | null;
+ content: string;
+ start_line: number;
+ end_line: number;
+ mtime: number;
+ },
+ number[]
+ >(
+ "SELECT c.id AS chunk_id, c.document_id, d.path AS path, d.title AS title, " +
+ " c.content AS content, c.start_line AS start_line, c.end_line AS end_line, d.mtime AS mtime " +
+ "FROM chunks c JOIN documents d ON d.id = c.document_id " +
+ `WHERE c.id IN (${placeholders})`,
+ )
+ .all(...(chunkIds as number[]));
+ for (const r of rows) {
+ out.set(r.chunk_id, {
+ chunkId: r.chunk_id,
+ documentId: r.document_id,
+ path: r.path,
+ title: r.title,
+ content: r.content,
+ startLine: r.start_line,
+ endLine: r.end_line,
+ mtime: r.mtime,
+ });
+ }
+ return out;
+ }
+
+ /**
+ * For each chunk id, return the document ids that link TO that
+ * chunk's document via wikilink or markdown_link. Pure data; the
+ * ranker decides how to convert this into a boost.
+ */
+ inboundLinkSources(candidateChunkIds: ReadonlyArray<number>): Map<number, Set<number>> {
+ const out = new Map<number, Set<number>>();
+ if (candidateChunkIds.length === 0) return out;
+ const placeholders = candidateChunkIds.map(() => "?").join(",");
+ const rows = this.db
+ .query<
+ { chunk_id: number; source_document_id: number },
+ number[]
+ >(
+ "SELECT c.id AS chunk_id, l.source_document_id " +
+ `FROM chunks c JOIN links l ON l.target_document_id = c.document_id ` +
+ `WHERE c.id IN (${placeholders}) AND l.link_type IN ('wikilink','markdown_link') ` +
+ ` AND l.source_document_id != c.document_id`,
+ )
+ .all(...(candidateChunkIds as number[]));
+ for (const r of rows) {
+ let set = out.get(r.chunk_id);
+ if (!set) {
+ set = new Set();
+ out.set(r.chunk_id, set);
+ }
+ set.add(r.source_document_id);
+ }
+ return out;
+ }
+
+ /**
+ * For each chunk id, the set of tag link_text values associated with
+ * its document. Two chunks "share a tag" iff their tag sets intersect.
+ */
+ tagsByChunkDocument(candidateChunkIds: ReadonlyArray<number>): Map<number, Set<string>> {
+ const out = new Map<number, Set<string>>();
+ if (candidateChunkIds.length === 0) return out;
+ const placeholders = candidateChunkIds.map(() => "?").join(",");
+ const rows = this.db
+ .query<
+ { chunk_id: number; tag: string },
+ number[]
+ >(
+ "SELECT c.id AS chunk_id, l.link_text AS tag " +
+ `FROM chunks c JOIN links l ON l.source_document_id = c.document_id ` +
+ `WHERE c.id IN (${placeholders}) AND l.link_type = 'tag' AND l.link_text IS NOT NULL`,
+ )
+ .all(...(candidateChunkIds as number[]));
+ for (const r of rows) {
+ let set = out.get(r.chunk_id);
+ if (!set) {
+ set = new Set();
+ out.set(r.chunk_id, set);
+ }
+ set.add(r.tag);
+ }
+ return out;
+ }
+
+ // ── counts ─────────────────────────────────────────────────────────────────
+
+ counts(): StoreCounts {
+ const docs = this.db.query<{ c: number }, []>("SELECT count(*) AS c FROM documents").get();
+ const chunks = this.db.query<{ c: number }, []>("SELECT count(*) AS c FROM chunks").get();
+ const emb = this.db.query<{ c: number }, []>("SELECT count(*) AS c FROM embeddings").get();
+ const stale = this.staleEmbeddings();
+ return Object.freeze({
+ documents: docs?.c ?? 0,
+ chunks: chunks?.c ?? 0,
+ embeddings: emb?.c ?? 0,
+ staleEmbeddings: stale,
+ });
+ }
+
+ private staleEmbeddings(): number {
+ const model = this.config.semantic.model;
+ const dimension = this.config.semantic.dimension;
+ if (!model || !dimension) {
+ // No baseline to compare against: 0 stale by convention.
+ return 0;
+ }
+ const row = this.db
+ .query<{ c: number }, [string, number]>(
+ "SELECT count(*) AS c FROM embeddings WHERE model != ? OR dimension != ?",
+ )
+ .get(model, dimension);
+ return row?.c ?? 0;
+ }
+
+ // ── direct accessors used by indexer/CLI/status ────────────────────────────
+
+ /**
+ * Chunks that have no row in `embeddings`. Used by the indexer to
+ * populate vectors after a fresh index or after the model-change drop.
+ */
+ findChunksWithoutEmbeddings(): Array<{ chunkId: number; content: string }> {
+ const rows = this.db
+ .query<{ id: number; content: string }, []>(
+ "SELECT c.id AS id, c.content AS content FROM chunks c " +
+ "LEFT JOIN embeddings e ON e.chunk_id = c.id " +
+ "WHERE e.chunk_id IS NULL ORDER BY c.id",
+ )
+ .all();
+ return rows.map((r) => ({ chunkId: r.id, content: r.content }));
+ }
+
+ /** Escape hatch for status queries that don't fit the typed API. */
+ rawQuery<T>(sql: string, params: ReadonlyArray<string | number | null> = []): T[] {
+ return this.db.query<T, (string | number | null)[]>(sql).all(...(params as (string | number | null)[]));
+ }
+}
diff --git a/src/core/search/types.ts b/src/core/search/types.ts
new file mode 100644
index 00000000..44da1b67
--- /dev/null
+++ b/src/core/search/types.ts
@@ -0,0 +1,135 @@
+/**
+ * Public types for `src/core/search/*`. Plain data β€” no behaviour, no I/O.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§12, Β§14.
+ */
+
+export const SEARCH_ERROR_CODES = [
+ "INDEX_MISSING",
+ "INDEX_UNREADABLE",
+ "SCHEMA_MISMATCH",
+ "VEC_EXTENSION_UNAVAILABLE",
+ "EMBEDDING_DISABLED",
+ "EMBEDDING_KEY_MISSING",
+ "EMBEDDING_PROVIDER_HTTP",
+ "EMBEDDING_PROVIDER_TIMEOUT",
+ "EMBEDDING_DIMENSION_MISMATCH",
+ "INDEX_LOCKED",
+ "INVALID_INPUT",
+] as const;
+export type SearchErrorCode = (typeof SEARCH_ERROR_CODES)[number];
+
+export class SearchError extends Error {
+ readonly code: SearchErrorCode;
+ constructor(code: SearchErrorCode, message: string) {
+ super(message);
+ this.name = "SearchError";
+ this.code = code;
+ }
+}
+
+export interface BrainSearchResult {
+ readonly documentId: number;
+ readonly chunkId: number;
+ readonly path: string;
+ readonly title: string | null;
+ readonly content: string;
+ readonly startLine: number;
+ readonly endLine: number;
+ readonly score: number;
+ readonly keywordScore: number;
+ readonly semanticScore: number;
+ readonly linkBoost: number;
+ readonly recencyBoost: number;
+ readonly searchType: "keyword" | "semantic" | "hybrid";
+}
+
+export interface IndexStats {
+ readonly added: number;
+ readonly updated: number;
+ readonly unchanged: number;
+ readonly deleted: number;
+ readonly chunksTotal: number;
+ readonly embeddingsComputed: number;
+ readonly embeddingsRetries: number;
+ readonly errors: ReadonlyArray<{ readonly path: string; readonly message: string }>;
+ readonly durationMs: number;
+}
+
+/**
+ * State of the optional sqlite-vec extension. `not-attempted` covers
+ * the diagnostic path where we never tried to load (e.g. `check` on a
+ * vault with semantic disabled); `unknown` covers status snapshots
+ * taken before any open has happened (index file missing).
+ */
+export type VecExtensionState = "loaded" | "unavailable" | "unknown" | "not-attempted";
+
+export interface IndexStatusSnapshot {
+ readonly indexPath: string;
+ readonly exists: boolean;
+ readonly schemaVersion: number | null;
+ readonly documents: number;
+ readonly chunks: number;
+ readonly embeddings: number;
+ readonly staleEmbeddings: number;
+ readonly embeddingModel: string | null;
+ readonly embeddingDimension: number | null;
+ readonly vecExtension: VecExtensionState;
+ readonly semanticEnabled: boolean;
+ readonly embeddingKeyPresent: boolean;
+ readonly lastIndexedAt: string | null;
+ readonly lastFullIndexAt: string | null;
+ readonly warnings: ReadonlyArray<string>;
+}
+
+export interface IndexCheckReport {
+ readonly vaultReadable: boolean;
+ readonly indexDirWritable: boolean;
+ readonly sqliteOk: boolean;
+ readonly fts5Ok: boolean;
+ readonly vecExtension: VecExtensionState;
+ readonly embeddingKeyResolved: boolean;
+ readonly providerReachable: boolean | null;
+ readonly providerReason: string | null;
+ readonly warnings: ReadonlyArray<string>;
+ readonly fatal: ReadonlyArray<string>;
+}
+
+export interface SearchOptions {
+ readonly query: string;
+ readonly limit?: number;
+ readonly semantic?: boolean | null;
+ readonly keywordOnly?: boolean;
+ readonly pathPrefix?: string;
+ readonly keywordWeight?: number;
+ readonly semanticWeight?: number;
+}
+
+export interface SearchOutcome {
+ readonly results: ReadonlyArray<BrainSearchResult>;
+ readonly warnings: ReadonlyArray<string>;
+ readonly total: number;
+}
+
+export interface ResolvedEmbeddingConfig {
+ readonly enabled: boolean;
+ readonly provider: "openai-compat" | "disabled";
+ readonly baseUrl: string | null;
+ readonly model: string | null;
+ readonly apiKey: string | null;
+ readonly dimension: number | null;
+ readonly timeoutMs: number;
+ readonly concurrency: number;
+ readonly batchSize: number;
+}
+
+export interface ResolvedSearchConfig {
+ readonly vault: string;
+ readonly dbPath: string;
+ readonly ignorePaths: ReadonlyArray<string>;
+ readonly chunkSize: number;
+ readonly chunkOverlap: number;
+ readonly keywordWeight: number;
+ readonly semanticWeight: number;
+ readonly semantic: ResolvedEmbeddingConfig;
+}
diff --git a/src/core/search/walker.ts b/src/core/search/walker.ts
new file mode 100644
index 00000000..a579ffa2
--- /dev/null
+++ b/src/core/search/walker.ts
@@ -0,0 +1,131 @@
+/**
+ * Walk the vault, yielding `.md` files that should be indexed.
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§6 edge cases
+ * (symlinks, ignore list).
+ */
+
+import { readdirSync, statSync, realpathSync, type Dirent, type Stats } from "node:fs";
+import { join, relative, sep } from "node:path";
+
+import type { ResolvedSearchConfig } from "./types.ts";
+
+export interface WalkedFile {
+ /** Absolute path on disk. */
+ readonly absPath: string;
+ /** Vault-relative POSIX path. */
+ readonly relPath: string;
+ readonly stat: Stats;
+}
+
+function toPosix(p: string): string {
+ return sep === "/" ? p : p.split(sep).join("/");
+}
+
+/**
+ * Parse the ignore list into two sets:
+ *
+ * - bare names (no `/`) match a directory whose name equals the entry
+ * anywhere in the tree;
+ * - relative paths match the vault-relative directory exactly.
+ */
+function parseIgnore(ignorePaths: ReadonlyArray<string>): {
+ names: Set<string>;
+ relPaths: Set<string>;
+} {
+ const names = new Set<string>();
+ const relPaths = new Set<string>();
+ for (const raw of ignorePaths) {
+ const e = raw.trim();
+ if (e === "") continue;
+ if (e.includes("/")) relPaths.add(e);
+ else names.add(e);
+ }
+ return { names, relPaths };
+}
+
+function isInsideVault(absTarget: string, vaultReal: string): boolean {
+ const targetReal = (() => {
+ try {
+ return realpathSync(absTarget);
+ } catch {
+ return null;
+ }
+ })();
+ if (targetReal === null) return false;
+ return targetReal === vaultReal || targetReal.startsWith(vaultReal + sep);
+}
+
+/**
+ * Synchronous generator yielding every `.md` file under `config.vault`
+ * (respecting `config.ignorePaths`). The caller drives the iteration so
+ * the indexer can pipeline reads + writes without buffering the whole
+ * tree.
+ */
+export function* walkVault(config: ResolvedSearchConfig): Generator<WalkedFile> {
+ const vaultReal = (() => {
+ try {
+ return realpathSync(config.vault);
+ } catch {
+ return config.vault;
+ }
+ })();
+ const { names, relPaths } = parseIgnore(config.ignorePaths);
+ // Track real (canonical) paths of visited directories so a symlink
+ // pointing back at an ancestor (or sibling) cannot send the walker
+ // into an infinite loop. `isInsideVault` covers escape outside the
+ // vault but is not acyclic on its own.
+ const seenDirs = new Set<string>([vaultReal]);
+
+ function* walk(dir: string): Generator<WalkedFile> {
+ let entries: Dirent[];
+ try {
+ entries = readdirSync(dir, { withFileTypes: true, encoding: "utf8" }) as Dirent[];
+ } catch {
+ return;
+ }
+ // Sort by name so two identical vaults produce the same traversal
+ // order across filesystems and platforms β€” important for the
+ // deterministic-indexing contract and for stable Syncthing peers.
+ entries.sort((a, b) => (a.name < b.name ? -1 : a.name > b.name ? 1 : 0));
+ for (const entry of entries) {
+ const absPath = join(dir, entry.name);
+ const relPathRaw = relative(vaultReal, absPath);
+ if (relPathRaw === "" || relPathRaw.startsWith("..")) continue;
+ const relPath = toPosix(relPathRaw);
+
+ const isLinkHint = entry.isSymbolicLink();
+
+ let stat: Stats;
+ try {
+ stat = statSync(absPath);
+ } catch {
+ continue;
+ }
+
+ if (stat.isDirectory()) {
+ if (names.has(entry.name)) continue;
+ if (relPaths.has(relPath)) continue;
+ let dirReal: string;
+ try {
+ dirReal = realpathSync(absPath);
+ } catch {
+ continue;
+ }
+ if (dirReal !== vaultReal && !dirReal.startsWith(vaultReal + sep)) continue;
+ if (seenDirs.has(dirReal)) continue;
+ seenDirs.add(dirReal);
+ yield* walk(absPath);
+ continue;
+ }
+
+ if (!stat.isFile()) continue;
+ if (!entry.name.toLowerCase().endsWith(".md")) continue;
+ if (isLinkHint && !isInsideVault(absPath, vaultReal)) continue;
+
+ yield { absPath, relPath, stat };
+ }
+ }
+
+ yield* walk(vaultReal);
+}
diff --git a/src/core/search/with-timeout.ts b/src/core/search/with-timeout.ts
new file mode 100644
index 00000000..0dafde1b
--- /dev/null
+++ b/src/core/search/with-timeout.ts
@@ -0,0 +1,35 @@
+/**
+ * Race `p` against a fixed timeout. The factory builds the rejection
+ * value so callers can throw the typed error class they need
+ * (`SearchError`, `MCPError`, a plain `Error`, …) without each call
+ * site reimplementing this utility.
+ */
+
+export function withTimeout<T>(
+ p: Promise<T>,
+ ms: number,
+ rejectionFactory: (ms: number) => unknown = (n) => new Error(`timeout after ${n}ms`),
+): Promise<T> {
+ return new Promise<T>((resolve, reject) => {
+ const timer = setTimeout(() => {
+ // If a buggy factory throws while building the timeout error, the
+ // promise would otherwise hang forever β€” settle with whatever the
+ // factory threw instead.
+ try {
+ reject(rejectionFactory(ms));
+ } catch (e) {
+ reject(e);
+ }
+ }, ms);
+ p.then(
+ (v) => {
+ clearTimeout(timer);
+ resolve(v);
+ },
+ (e) => {
+ clearTimeout(timer);
+ reject(e);
+ },
+ );
+ });
+}
diff --git a/src/mcp/search-tools.ts b/src/mcp/search-tools.ts
new file mode 100644
index 00000000..3040336b
--- /dev/null
+++ b/src/mcp/search-tools.ts
@@ -0,0 +1,214 @@
+/**
+ * MCP tool registry slice for Brain Search.
+ *
+ * Exposes `brain_search` (read-only, agent-facing) and the
+ * `search.*` enrichment used by `second_brain_status`. Index management
+ * verbs (`index`, `reindex`, `check`) are intentionally NOT exposed
+ * over MCP β€” they are operator business, never agent business
+ * (design doc Β§3, principle 5).
+ *
+ * Anchored in docs/plans/2026-05-16-brain-search-design.md Β§9.
+ */
+
+import {
+ indexStatus,
+ resolveSearchConfig,
+ search,
+ SearchError,
+} from "../core/search/index.ts";
+import type {
+ BrainSearchResult,
+ SearchOutcome,
+} from "../core/search/index.ts";
+import { withTimeout } from "../core/search/with-timeout.ts";
+import { INTERNAL_ERROR, INVALID_PARAMS, MCPError } from "./protocol.ts";
+import type { ServerContext, ToolDefinition } from "./tools.ts";
+
+const MCP_LIMIT_MAX = 50;
+const MCP_CONTENT_MAX = 600;
+const SEARCH_TIMEOUT_MS = 10_000;
+
+const SEARCH_INPUT_SCHEMA: Record<string, unknown> = {
+ type: "object",
+ properties: {
+ query: { type: "string", minLength: 1, maxLength: 2000 },
+ limit: { type: "integer", minimum: 1, maximum: MCP_LIMIT_MAX },
+ semantic: { type: "boolean" },
+ keyword_only: { type: "boolean" },
+ path_prefix: { type: "string", maxLength: 256 },
+ },
+ required: ["query"],
+ additionalProperties: false,
+};
+
+function coerceBool(args: Record<string, unknown>, key: string): boolean | undefined {
+ if (!(key in args)) return undefined;
+ const v = args[key];
+ if (v === undefined || v === null) return undefined;
+ if (typeof v !== "boolean") {
+ throw new MCPError(INVALID_PARAMS, `argument '${key}' must be a boolean`);
+ }
+ return v;
+}
+
+function coerceStringOptional(
+ args: Record<string, unknown>,
+ key: string,
+ maxLen: number,
+): string | undefined {
+ if (!(key in args)) return undefined;
+ const v = args[key];
+ if (v === undefined || v === null) return undefined;
+ if (typeof v !== "string") {
+ throw new MCPError(INVALID_PARAMS, `argument '${key}' must be a string`);
+ }
+ if (v.length > maxLen) {
+ throw new MCPError(INVALID_PARAMS, `argument '${key}' exceeds ${maxLen} characters`);
+ }
+ return v;
+}
+
+function searchTimeoutError(ms: number): MCPError {
+ return new MCPError(INTERNAL_ERROR, `search timeout after ${ms}ms`);
+}
+
+function truncateContent(c: string, max: number): string {
+ if (c.length <= max) return c;
+ return c.slice(0, max - 1) + "…";
+}
+
+function searchErrorToMcp(e: SearchError): MCPError {
+ if (e.code === "INVALID_INPUT") return new MCPError(INVALID_PARAMS, e.message);
+ if (e.code === "INDEX_MISSING") {
+ return new MCPError(INTERNAL_ERROR, "search index not initialised. Run: o2b search index");
+ }
+ if (e.code === "INDEX_UNREADABLE") {
+ return new MCPError(INTERNAL_ERROR, `search index unreadable: ${e.message}`);
+ }
+ if (e.code === "VEC_EXTENSION_UNAVAILABLE") {
+ return new MCPError(
+ INTERNAL_ERROR,
+ "semantic search unavailable: sqlite-vec extension not loaded",
+ );
+ }
+ if (e.code === "EMBEDDING_KEY_MISSING") {
+ return new MCPError(INTERNAL_ERROR, "embedding key not configured");
+ }
+ if (e.code === "EMBEDDING_PROVIDER_HTTP" || e.code === "EMBEDDING_PROVIDER_TIMEOUT") {
+ return new MCPError(INTERNAL_ERROR, `embedding provider unavailable: ${e.message}`);
+ }
+ return new MCPError(INTERNAL_ERROR, `${e.message} [${e.code}]`);
+}
+
+async function toolBrainSearch(
+ ctx: ServerContext,
+ args: Record<string, unknown>,
+): Promise<Record<string, unknown>> {
+ const query = args["query"];
+ if (typeof query !== "string" || query.trim() === "") {
+ throw new MCPError(INVALID_PARAMS, "missing required argument: query");
+ }
+ if (query.length > 2000) {
+ throw new MCPError(INVALID_PARAMS, "argument 'query' exceeds 2000 characters");
+ }
+
+ let limit = 10;
+ if ("limit" in args && args["limit"] !== undefined && args["limit"] !== null) {
+ const raw = args["limit"];
+ if (typeof raw !== "number" || !Number.isInteger(raw)) {
+ throw new MCPError(INVALID_PARAMS, "argument 'limit' must be an integer");
+ }
+ if (raw < 1 || raw > MCP_LIMIT_MAX) {
+ throw new MCPError(INVALID_PARAMS, `argument 'limit' must be between 1 and ${MCP_LIMIT_MAX}`);
+ }
+ limit = raw;
+ }
+
+ const semantic = coerceBool(args, "semantic");
+ const keywordOnly = coerceBool(args, "keyword_only") ?? false;
+ const pathPrefix = coerceStringOptional(args, "path_prefix", 256);
+
+ const config = resolveSearchConfig({
+ vault: ctx.vault,
+ configPath: ctx.configPath ?? undefined,
+ });
+
+ let outcome: SearchOutcome;
+ try {
+ outcome = await withTimeout(
+ search(config, {
+ query,
+ limit,
+ semantic: semantic ?? null,
+ keywordOnly,
+ pathPrefix,
+ }),
+ SEARCH_TIMEOUT_MS,
+ searchTimeoutError,
+ );
+ } catch (e) {
+ if (e instanceof SearchError) throw searchErrorToMcp(e);
+ if (e instanceof MCPError) throw e;
+ throw new MCPError(INTERNAL_ERROR, e instanceof Error ? e.message : String(e));
+ }
+
+ return {
+ results: outcome.results.map((r: BrainSearchResult) => ({
+ path: r.path,
+ title: r.title,
+ content: truncateContent(r.content, MCP_CONTENT_MAX),
+ score: r.score,
+ startLine: r.startLine,
+ endLine: r.endLine,
+ searchType: r.searchType,
+ })),
+ warnings: outcome.warnings,
+ total: outcome.total,
+ };
+}
+
+export const SEARCH_TOOLS: ReadonlyArray<ToolDefinition> = Object.freeze([
+ {
+ name: "brain_search",
+ description:
+ "Full-text search across the vault. Optional semantic layer when configured. Read-only.",
+ inputSchema: SEARCH_INPUT_SCHEMA,
+ handler: toolBrainSearch,
+ },
+]);
+
+/**
+ * `search.*` block for `second_brain_status`. Mirrors design Β§9
+ * exactly. Never throws β€” returns `{ exists: false, hint }` if the
+ * index does not exist; surfaces errors as `error: "<message>"`.
+ */
+export async function buildSearchStatusBlock(ctx: ServerContext): Promise<Record<string, unknown>> {
+ try {
+ const config = resolveSearchConfig({
+ vault: ctx.vault,
+ configPath: ctx.configPath ?? undefined,
+ });
+ const snap = await indexStatus(config);
+ if (!snap.exists) {
+ return { exists: false, hint: "run: o2b search index" };
+ }
+ return {
+ index_path: snap.indexPath,
+ exists: true,
+ schema_version: snap.schemaVersion,
+ documents: snap.documents,
+ chunks: snap.chunks,
+ embeddings: snap.embeddings,
+ stale_embeddings: snap.staleEmbeddings,
+ embedding_model: snap.embeddingModel,
+ embedding_dimension: snap.embeddingDimension,
+ vec_extension: snap.vecExtension,
+ semantic_enabled: snap.semanticEnabled,
+ embedding_key_present: snap.embeddingKeyPresent,
+ last_indexed_at: snap.lastIndexedAt,
+ last_full_index_at: snap.lastFullIndexAt,
+ };
+ } catch (e) {
+ return { exists: false, error: e instanceof Error ? e.message : String(e) };
+ }
+}
diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts
index 66e7c8c4..c1058044 100644
--- a/src/mcp/tools.ts
+++ b/src/mcp/tools.ts
@@ -29,6 +29,7 @@ import { computeBrainStatus } from "../core/brain/status.ts";
import { doctor } from "../core/doctor.ts";
import { appendEvent, validateEventTime } from "../core/event-log.ts";
import { BRAIN_TOOLS } from "./brain-tools.ts";
+import { SEARCH_TOOLS, buildSearchStatusBlock } from "./search-tools.ts";
import {
checkPolicy,
consumePendingRequest,
@@ -170,6 +171,8 @@ async function toolStatus(ctx: ServerContext): Promise<Record<string, unknown>>
// Safe to call on a vault that has no Brain layer yet β€” returns
// `present: false` with zero counts.
const brain = vaultExists ? computeBrainStatus(ctx.vault) : null;
+ const searchDisabled = discovery.data["search_enabled"] === "false";
+ const search = vaultExists && !searchDisabled ? await buildSearchStatusBlock(ctx) : null;
return {
config_path: String(discovery.path),
config_exists: discovery.exists,
@@ -178,6 +181,7 @@ async function toolStatus(ctx: ServerContext): Promise<Record<string, unknown>>
vault_path: ctx.vault,
vault_exists: vaultExists,
...(brain ? { brain } : {}),
+ ...(search ? { search } : {}),
};
}
@@ -636,6 +640,7 @@ export function buildToolTable(): ToolDefinition[] {
// use (`o2b append-event`); only the entries in the exported tool
// array are removed β€” see Β§11.1 of the v0.9.0 design doc.
...BRAIN_TOOLS,
+ ...SEARCH_TOOLS,
{
name: "vault_health",
description: "Run vault, config, and plugin manifest health checks.",
diff --git a/tests/cli/search.test.ts b/tests/cli/search.test.ts
new file mode 100644
index 00000000..4ef10583
--- /dev/null
+++ b/tests/cli/search.test.ts
@@ -0,0 +1,169 @@
+/**
+ * CLI tests for `o2b search *`.
+ *
+ * Each test forks the actual `bun src/cli/main.ts` binary via `runCli`,
+ * the same harness `tests/cli/brain.test.ts` uses. We assert exit codes,
+ * canonical text patterns, and the JSON shape produced by `--json`.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import {
+ existsSync,
+ mkdirSync,
+ mkdtempSync,
+ rmSync,
+ writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { runCli } from "../helpers/run-cli.ts";
+
+let tmp: string;
+let vault: string;
+let config: string;
+
+function writeVaultFile(rel: string, content: string): void {
+ const abs = join(vault, rel);
+ mkdirSync(join(abs, ".."), { recursive: true });
+ writeFileSync(abs, content);
+}
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-search-cli-"));
+ vault = join(tmp, "vault");
+ mkdirSync(vault, { recursive: true });
+ config = join(tmp, "config.yaml");
+ writeFileSync(config, `vault: "${vault}"\n`);
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+test("search index creates the index file under <vault>/.open-second-brain/", async () => {
+ writeVaultFile("a.md", "# A\n\nhello world");
+ const out = await runCli(["search", "index"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(0);
+ expect(existsSync(join(vault, ".open-second-brain", "brain.sqlite"))).toBe(true);
+});
+
+test("search status without an index reports 'not initialised' and exits 0", async () => {
+ const out = await runCli(["search", "status"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(0);
+ expect(out.stdout).toContain("not initialised");
+});
+
+test("search status --json after an index returns documents count", async () => {
+ writeVaultFile("a.md", "# A\n\nbody");
+ writeVaultFile("b.md", "# B\n\nbody");
+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });
+ const out = await runCli(["search", "status", "--json"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(0);
+ const obj = JSON.parse(out.stdout);
+ expect(obj.exists).toBe(true);
+ expect(obj.documents).toBe(2);
+ expect(obj.schema_version).toBe(1);
+});
+
+test("search query returns a human-readable hit for indexed content", async () => {
+ writeVaultFile("notes/foo.md", "# Foo\n\nthe quick brown fox jumps over the lazy dog");
+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });
+ const out = await runCli(["search", "fox"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(0);
+ expect(out.stdout).toContain("notes/foo.md");
+});
+
+test("search query --json returns structured results", async () => {
+ writeVaultFile("notes/foo.md", "# Foo\n\nfox content");
+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });
+ const out = await runCli(["search", "fox", "--json", "--limit", "5"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(0);
+ const obj = JSON.parse(out.stdout);
+ expect(Array.isArray(obj.results)).toBe(true);
+ expect(obj.results.length).toBeGreaterThan(0);
+ expect(obj.results[0].path).toBe("notes/foo.md");
+});
+
+test("search query on missing index fails with exit 1", async () => {
+ const out = await runCli(["search", "nothing-here"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(1);
+ expect(out.stderr).toContain("INDEX_MISSING");
+});
+
+test("search check reports vault_readable and sqlite_ok on a fresh vault", async () => {
+ const out = await runCli(["search", "check", "--json"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(0);
+ const obj = JSON.parse(out.stdout);
+ expect(obj.vault_readable).toBe(true);
+ expect(obj.sqlite_ok).toBe(true);
+ expect(obj.fts5_ok).toBe(true);
+});
+
+test("search reindex rebuilds the index atomically", async () => {
+ writeVaultFile("a.md", "# A");
+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });
+ writeVaultFile("b.md", "# B");
+ const out = await runCli(["search", "reindex"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(0);
+ expect(existsSync(join(vault, ".open-second-brain", "brain.sqlite"))).toBe(true);
+ expect(existsSync(join(vault, ".open-second-brain", "brain.sqlite.bak"))).toBe(true);
+});
+
+test("unknown flag exits with code 2", async () => {
+ const out = await runCli(["search", "index", "--bogus"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(2);
+});
+
+test("invalid numeric search flags exit with code 2 before touching the index", async () => {
+ let out = await runCli(["search", "fox", "--keyword-weight", "nan"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(2);
+ expect(out.stderr).toContain("search_keyword_weight");
+
+ out = await runCli(["search", "index", "--concurrency", "0"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(2);
+ expect(out.stderr).toContain("embedding_concurrency");
+});
+
+test("path-prefix escaping returns exit 2 with INVALID_INPUT", async () => {
+ writeVaultFile("a.md", "# A");
+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });
+ const out = await runCli(["search", "A", "--path", "../etc/"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(2);
+ expect(out.stderr).toContain("INVALID_INPUT");
+});
+
+test("the default verb is `query` when first positional is unknown", async () => {
+ writeVaultFile("a.md", "# A\n\nalpha word");
+ await runCli(["search", "index"], { env: { OPEN_SECOND_BRAIN_CONFIG: config } });
+ // No explicit verb, just a query token:
+ const out = await runCli(["search", "alpha"], {
+ env: { OPEN_SECOND_BRAIN_CONFIG: config },
+ });
+ expect(out.returncode).toBe(0);
+ expect(out.stdout).toContain("a.md");
+});
diff --git a/tests/core/search/chunker.test.ts b/tests/core/search/chunker.test.ts
new file mode 100644
index 00000000..1231a373
--- /dev/null
+++ b/tests/core/search/chunker.test.ts
@@ -0,0 +1,172 @@
+import { test, expect } from "bun:test";
+import { chunkMarkdown } from "../../../src/core/search/chunker.ts";
+
+test("empty file yields no chunks but a title from filename", () => {
+ const r = chunkMarkdown("", "my_note");
+ expect(r.chunks.length).toBe(0);
+ expect(r.title).toBe("my note");
+});
+
+test("whitespace-only file yields no chunks", () => {
+ const r = chunkMarkdown("\n\n \n", "blank-file");
+ expect(r.chunks.length).toBe(0);
+});
+
+test("single short paragraph becomes one chunk", () => {
+ const r = chunkMarkdown("Hello world.", "doc");
+ expect(r.chunks.length).toBe(1);
+ expect(r.chunks[0]?.content).toContain("Hello world");
+ expect(r.chunks[0]?.startLine).toBe(1);
+ expect(r.chunks[0]?.endLine).toBe(1);
+ expect(r.chunks[0]?.tokenCount).toBe(2);
+});
+
+test("frontmatter becomes synthetic chunk 0 with raw text", () => {
+ const text = `---
+title: My Title
+tags:
+ - foo
+ - bar
+---
+
+# Body H1
+
+Some body paragraph.`;
+ const r = chunkMarkdown(text, "ignored");
+ expect(r.title).toBe("My Title");
+ expect(r.chunks.length).toBeGreaterThanOrEqual(2);
+ expect(r.chunks[0]?.content).toContain("title: My Title");
+ expect(r.chunks[0]?.content).toContain("tags:");
+});
+
+test("malformed frontmatter (no closing ---) is dropped with a warning", () => {
+ const text = `---
+title: Unterminated
+
+# Body
+content here`;
+ const r = chunkMarkdown(text, "ignored");
+ expect(r.warnings.length).toBe(1);
+ expect(r.warnings[0]).toContain("frontmatter");
+ // Body still chunked.
+ expect(r.chunks.length).toBeGreaterThanOrEqual(1);
+});
+
+test("title falls back to first H1 when no frontmatter title", () => {
+ const text = `# The Heading
+
+Body.`;
+ const r = chunkMarkdown(text, "fallback");
+ expect(r.title).toBe("The Heading");
+});
+
+test("title falls back to filename when no frontmatter and no H1", () => {
+ const r = chunkMarkdown("Just a paragraph.", "my-note_file");
+ expect(r.title).toBe("my note file");
+});
+
+test("code fence is atomic and not split", () => {
+ const long = Array.from({ length: 50 }, (_, i) => `line ${i}`).join("\n");
+ const text = "Some intro.\n\n```py\n" + long + "\n```\n\nMore prose.";
+ const r = chunkMarkdown(text, "doc");
+ // Find a chunk containing the opening fence; it should also contain the closing fence.
+ const fenceChunks = r.chunks.filter((c) => c.content.includes("```py"));
+ expect(fenceChunks.length).toBe(1);
+ expect(fenceChunks[0]?.content).toContain("line 0");
+ expect(fenceChunks[0]?.content).toContain("line 49");
+});
+
+test("oversized code fence becomes its own chunk and is not truncated", () => {
+ const huge = Array.from({ length: 2000 }, (_, i) => `word${i}`).join(" ");
+ const text = "Intro paragraph.\n\n```\n" + huge + "\n```\n\nOutro.";
+ const r = chunkMarkdown(text, "doc", { maxTokens: 100, minTokens: 50, overlapTokens: 0 });
+ const fenceChunk = r.chunks.find((c) => c.content.includes("word1999"));
+ expect(fenceChunk).toBeDefined();
+ expect(fenceChunk?.content).toContain("word0");
+ expect(fenceChunk?.tokenCount).toBeGreaterThan(100);
+});
+
+test("heading attaches to current chunk when below min_tokens", () => {
+ const text = `# Tiny
+
+word.
+
+# Next Heading
+
+# Even Next`;
+ const r = chunkMarkdown(text, "doc", { maxTokens: 800, minTokens: 100, overlapTokens: 0 });
+ // All very small; expect a single chunk.
+ expect(r.chunks.length).toBe(1);
+ expect(r.chunks[0]?.content).toContain("# Tiny");
+ expect(r.chunks[0]?.content).toContain("# Next Heading");
+});
+
+test("heading starts a new chunk when current >= min_tokens", () => {
+ // Generate a paragraph with >=100 tokens, then a new heading.
+ const body = Array.from({ length: 150 }, (_, i) => `w${i}`).join(" ");
+ const text = `# A\n\n${body}\n\n# B\n\nmore.`;
+ const r = chunkMarkdown(text, "doc", { maxTokens: 800, minTokens: 100, overlapTokens: 0 });
+ // Find chunk starting with "# B"
+ expect(r.chunks.length).toBeGreaterThanOrEqual(2);
+ const second = r.chunks[1]!;
+ expect(second.content).toContain("# B");
+ expect(second.content).toContain("more");
+});
+
+test("list block stays atomic until first non-list line", () => {
+ const text = `- one
+- two
+- three
+
+Next paragraph.`;
+ const r = chunkMarkdown(text, "doc", { maxTokens: 100, minTokens: 50, overlapTokens: 0 });
+ const listChunk = r.chunks.find((c) => c.content.includes("- one"));
+ expect(listChunk?.content).toContain("- three");
+});
+
+test("table is captured atomically", () => {
+ const text = `| a | b |
+|---|---|
+| 1 | 2 |
+| 3 | 4 |
+
+After table.`;
+ const r = chunkMarkdown(text, "doc", { maxTokens: 100, minTokens: 50, overlapTokens: 0 });
+ const tableChunk = r.chunks.find((c) => c.content.includes("| a | b |"));
+ expect(tableChunk?.content).toContain("| 3 | 4 |");
+});
+
+test("UTF-8 Cyrillic content roundtrips and tokenises sensibly", () => {
+ const text = `# ΠŸΡ€ΠΈΠ²Π΅Ρ‚
+
+Π­Ρ‚ΠΎ тСстовая строка с ΠΊΠΈΡ€ΠΈΠ»Π»ΠΈΡ†Π΅ΠΉ. ΠŸΡ€ΠΈΠ²Π΅Ρ‚, ΠΌΠΈΡ€!`;
+ const r = chunkMarkdown(text, "ru-doc");
+ expect(r.chunks.length).toBe(1);
+ expect(r.chunks[0]?.content).toContain("ΠŸΡ€ΠΈΠ²Π΅Ρ‚");
+ expect(r.chunks[0]?.content).toContain("ΠΊΠΈΡ€ΠΈΠ»Π»ΠΈΡ†Π΅ΠΉ");
+ expect(r.chunks[0]?.tokenCount).toBeGreaterThan(5);
+});
+
+test("overlap prepends previous chunk tail to next", () => {
+ // Force two chunks via small maxTokens; verify second begins with overlap.
+ const para1 = Array.from({ length: 50 }, (_, i) => `a${i}`).join(" ");
+ const para2 = Array.from({ length: 50 }, (_, i) => `b${i}`).join(" ");
+ const text = `${para1}\n\n${para2}`;
+ const r = chunkMarkdown(text, "doc", { maxTokens: 50, minTokens: 10, overlapTokens: 10 });
+ expect(r.chunks.length).toBeGreaterThanOrEqual(2);
+ const second = r.chunks[1]!;
+ // The second chunk should contain something from a-range (tail) AND the b-range start.
+ expect(second.content).toContain("b0");
+ expect(/a4\d/.test(second.content)).toBe(true);
+});
+
+test("line numbers in chunks point to original (non-overlap) lines", () => {
+ const text = `line A
+line B
+line C
+
+line D
+line E`;
+ const r = chunkMarkdown(text, "doc", { maxTokens: 4, minTokens: 1, overlapTokens: 0 });
+ expect(r.chunks[0]?.startLine).toBe(1);
+});
diff --git a/tests/core/search/config.test.ts b/tests/core/search/config.test.ts
new file mode 100644
index 00000000..afe25b66
--- /dev/null
+++ b/tests/core/search/config.test.ts
@@ -0,0 +1,158 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { resolveSearchConfig } from "../../../src/core/search/index.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+
+let tmp: string;
+let configPath: string;
+let origEnv: Record<string, string | undefined>;
+
+const ENV_KEYS = [
+ "OPEN_SECOND_BRAIN_SEARCH_DB",
+ "OPEN_SECOND_BRAIN_SEARCH_SEMANTIC",
+ "OPEN_SECOND_BRAIN_SEARCH_CHUNK_SIZE",
+ "OPEN_SECOND_BRAIN_SEARCH_CHUNK_OVERLAP",
+ "OPEN_SECOND_BRAIN_SEARCH_KW_WEIGHT",
+ "OPEN_SECOND_BRAIN_SEARCH_SEM_WEIGHT",
+ "OPEN_SECOND_BRAIN_SEARCH_IGNORE",
+ "OPEN_SECOND_BRAIN_EMBEDDING_PROVIDER",
+ "OPEN_SECOND_BRAIN_EMBEDDING_BASE_URL",
+ "OPEN_SECOND_BRAIN_EMBEDDING_MODEL",
+ "OPEN_SECOND_BRAIN_EMBEDDING_KEY",
+ "OPEN_SECOND_BRAIN_EMBEDDING_DIM",
+ "OPEN_SECOND_BRAIN_EMBEDDING_TIMEOUT",
+ "OPEN_SECOND_BRAIN_EMBEDDING_CONCURRENCY",
+ "OPEN_SECOND_BRAIN_EMBEDDING_BATCH",
+];
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "osb-cfg-"));
+ configPath = join(tmp, "config.yaml");
+ origEnv = {};
+ for (const k of ENV_KEYS) {
+ origEnv[k] = process.env[k];
+ delete process.env[k];
+ }
+});
+
+afterEach(() => {
+ for (const k of ENV_KEYS) {
+ if (origEnv[k] === undefined) delete process.env[k];
+ else process.env[k] = origEnv[k];
+ }
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+test("defaults are returned when config and env are empty", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\n`);
+ const cfg = resolveSearchConfig({ vault: tmp, configPath });
+ expect(cfg.vault).toBe(tmp);
+ expect(cfg.dbPath).toBe(join(tmp, ".open-second-brain", "brain.sqlite"));
+ expect(cfg.chunkSize).toBe(800);
+ expect(cfg.chunkOverlap).toBe(100);
+ expect(cfg.keywordWeight).toBe(0.6);
+ expect(cfg.semanticWeight).toBe(0.4);
+ expect(cfg.semantic.enabled).toBe(false);
+ expect(cfg.semantic.provider).toBe("openai-compat");
+ expect(cfg.semantic.apiKey).toBeNull();
+ expect(cfg.ignorePaths).toContain(".git");
+ expect(cfg.ignorePaths).toContain(".open-second-brain");
+});
+
+test("env overrides config which overrides defaults", () => {
+ writeFileSync(
+ configPath,
+ [
+ `vault: "${tmp}"`,
+ `search_chunk_size: "500"`,
+ `search_semantic_enabled: "true"`,
+ `embedding_base_url: "https://config.example/v1"`,
+ `embedding_model: "config-model"`,
+ `embedding_api_key: "cfg-key"`,
+ ].join("\n") + "\n",
+ );
+ process.env["OPEN_SECOND_BRAIN_EMBEDDING_MODEL"] = "env-model";
+ const cfg = resolveSearchConfig({ vault: tmp, configPath });
+ expect(cfg.chunkSize).toBe(500);
+ expect(cfg.semantic.enabled).toBe(true);
+ expect(cfg.semantic.baseUrl).toBe("https://config.example/v1");
+ expect(cfg.semantic.model).toBe("env-model");
+ expect(cfg.semantic.apiKey).toBe("cfg-key");
+});
+
+test("overrides win over both env and config", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_size: "500"\n`);
+ process.env["OPEN_SECOND_BRAIN_SEARCH_SEMANTIC"] = "false";
+ const cfg = resolveSearchConfig({
+ vault: tmp,
+ configPath,
+ overrides: { keywordWeight: 0.9, semanticWeight: 0.1 },
+ });
+ expect(cfg.keywordWeight).toBe(0.9);
+ expect(cfg.semanticWeight).toBe(0.1);
+});
+
+test("invalid numeric value throws INVALID_INPUT", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_size: "not-a-number"\n`);
+ let err: SearchError | null = null;
+ try {
+ resolveSearchConfig({ vault: tmp, configPath });
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err).not.toBeNull();
+ expect(err?.code).toBe("INVALID_INPUT");
+ expect(err?.message).toContain("search_chunk_size");
+});
+
+test("weights summing above 1 is rejected", () => {
+ writeFileSync(
+ configPath,
+ `vault: "${tmp}"\nsearch_keyword_weight: "0.7"\nsearch_semantic_weight: "0.5"\n`,
+ );
+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/sum.*1/);
+});
+
+test("non-positive integer knobs are rejected before provider construction", () => {
+ for (const key of [
+ "search_chunk_size",
+ "embedding_dimension",
+ "embedding_timeout_ms",
+ "embedding_concurrency",
+ "embedding_batch_size",
+ ]) {
+ writeFileSync(configPath, `vault: "${tmp}"\n${key}: "0"\n`);
+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(key);
+ }
+});
+
+test("chunk overlap must be non-negative and smaller than chunk size", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\nsearch_chunk_overlap: "-1"\n`);
+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/search_chunk_overlap/);
+
+ writeFileSync(
+ configPath,
+ `vault: "${tmp}"\nsearch_chunk_size: "100"\nsearch_chunk_overlap: "100"\n`,
+ );
+ expect(() => resolveSearchConfig({ vault: tmp, configPath })).toThrow(/smaller than/);
+});
+
+test("overrides are validated after merging", () => {
+ writeFileSync(configPath, `vault: "${tmp}"\n`);
+ expect(() =>
+ resolveSearchConfig({
+ vault: tmp,
+ configPath,
+ overrides: { keywordWeight: Number.NaN },
+ }),
+ ).toThrow(/search_keyword_weight/);
+ expect(() =>
+ resolveSearchConfig({
+ vault: tmp,
+ configPath,
+ overrides: { semantic: { batchSize: 0 } },
+ }),
+ ).toThrow(/embedding_batch_size/);
+});
diff --git a/tests/core/search/embeddings.test.ts b/tests/core/search/embeddings.test.ts
new file mode 100644
index 00000000..a8f9ac1e
--- /dev/null
+++ b/tests/core/search/embeddings.test.ts
@@ -0,0 +1,246 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+
+import { OpenAICompatProvider } from "../../../src/core/search/embeddings/openai-compat.ts";
+import { NullProvider } from "../../../src/core/search/embeddings/null-provider.ts";
+import { makeProvider } from "../../../src/core/search/embeddings/provider.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+import type { ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";
+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";
+
+let server: FakeHttp;
+
+function cfg(overrides: Partial<ResolvedEmbeddingConfig> = {}): ResolvedEmbeddingConfig {
+ return Object.freeze({
+ enabled: true,
+ provider: "openai-compat",
+ baseUrl: server.url,
+ model: "fake-model",
+ apiKey: "test-key",
+ dimension: null,
+ timeoutMs: 5_000,
+ concurrency: 2,
+ batchSize: 32,
+ ...overrides,
+ });
+}
+
+beforeEach(async () => {
+ server = await startFakeHttp();
+});
+
+afterEach(async () => {
+ await server.close();
+});
+
+test("embed() returns unit-normalised vectors in input order", async () => {
+ const p = new OpenAICompatProvider(cfg());
+ const out = await p.embed(["a", "b", "c"]);
+ expect(out.length).toBe(3);
+ // The default fake handler echoes [tokens, index, len, 1]. Index differs.
+ // After unit-normalisation, ||v|| == 1.
+ for (const v of out) {
+ const norm = Math.sqrt(v.reduce((s, x) => s + x * x, 0));
+ expect(norm).toBeCloseTo(1.0, 6);
+ }
+});
+
+test("embed() preserves input order across batches", async () => {
+ const p = new OpenAICompatProvider(cfg({ batchSize: 2 }));
+ const inputs = ["one", "two", "three", "four", "five"];
+ const out = await p.embed(inputs);
+ expect(out.length).toBe(5);
+ // Server returns >2 calls because batchSize=2 β†’ 3 batches.
+ expect(server.callCount()).toBe(3);
+});
+
+test("embed() sorts response.data by `index` even if server returns shuffled", async () => {
+ server.setHandler((req) => {
+ const body = (req.body as { input: string[]; model: string });
+ const indices = body.input.map((_, i) => i);
+ // Shuffle: reverse.
+ const shuffled = [...indices].reverse();
+ const data = shuffled.map((origIdx) => {
+ const text = body.input[origIdx]!;
+ const v = [text.length, origIdx, 1, 1];
+ return { object: "embedding", embedding: v, index: origIdx };
+ });
+ return { status: 200, body: { data, model: body.model } };
+ });
+
+ const p = new OpenAICompatProvider(cfg({ dimension: 4 }), { backoffMs: [5, 5] });
+ const out = await p.embed(["aaaa", "bb"]);
+ // First input "aaaa" has length 4 β€” unit-normalised first component dominates.
+ // Second input "bb" has length 2.
+ expect(out[0]!.length).toBe(4);
+ expect(out[1]!.length).toBe(4);
+ // Sanity: the first input's first dimension > second input's first dimension.
+ // After normalisation, monotonic relationship preserved because all other
+ // components are identical between the two outputs (index, 1, 1).
+ // (We don't need exact values; just confirm we received the right vector
+ // per the original `index`.)
+ expect(out[0]).not.toEqual(out[1]);
+});
+
+test("embed() rejects duplicate response indexes that leave a missing vector", async () => {
+ server.setHandler((req) => {
+ const body = req.body as { input: string[]; model: string };
+ return {
+ status: 200,
+ body: {
+ data: [
+ { object: "embedding", embedding: [1, 0, 0, 0], index: 0 },
+ { object: "embedding", embedding: [0, 1, 0, 0], index: 0 },
+ ],
+ model: body.model,
+ },
+ };
+ });
+
+ const p = new OpenAICompatProvider(cfg({ dimension: 4 }), { backoffMs: [5, 5] });
+ let err: SearchError | null = null;
+ try {
+ await p.embed(["a", "b"]);
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("EMBEDDING_PROVIDER_HTTP");
+ expect(err?.message).toContain("missing vector");
+});
+
+test("embed() retries on 503 then succeeds", async () => {
+ let calls = 0;
+ server.setHandler((req) => {
+ calls++;
+ if (calls === 1) return { status: 503, body: { error: "busy" } };
+ return {
+ status: 200,
+ body: {
+ data: [{ object: "embedding", embedding: [0.5, 0.5, 0.5, 0.5], index: 0 }],
+ model: (req.body as { model: string }).model,
+ },
+ };
+ });
+
+ const p = new OpenAICompatProvider(cfg(), { backoffMs: [10, 20] });
+ const out = await p.embed(["once"]);
+ expect(out.length).toBe(1);
+ expect(calls).toBe(2);
+});
+
+test("embed() fails fast on 400 (non-retriable 4xx)", async () => {
+ server.setHandler(() => ({ status: 400, body: { error: "bad request" } }));
+ const p = new OpenAICompatProvider(cfg(), { backoffMs: [10, 20] });
+ let err: SearchError | null = null;
+ try {
+ await p.embed(["x"]);
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("EMBEDDING_PROVIDER_HTTP");
+ expect(err?.message).toContain("400");
+ expect(server.callCount()).toBe(1);
+});
+
+test("embed() throws EMBEDDING_PROVIDER_HTTP after exhausting retries on 5xx", async () => {
+ server.setHandler(() => ({ status: 503, body: { error: "busy" } }));
+ const p = new OpenAICompatProvider(cfg(), { backoffMs: [10, 20] });
+ let err: SearchError | null = null;
+ try {
+ await p.embed(["x"]);
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("EMBEDDING_PROVIDER_HTTP");
+ expect(server.callCount()).toBe(3);
+});
+
+test("embed() honours per-request timeout via AbortController", async () => {
+ server.setHandler(() => ({ status: 200, body: { data: [] }, delayMs: 200 }));
+ const p = new OpenAICompatProvider(cfg({ timeoutMs: 30 }), { backoffMs: [5, 5] });
+ let err: SearchError | null = null;
+ try {
+ await p.embed(["x"]);
+ } catch (e) {
+ err = e as SearchError;
+ }
+ // Either the timeout or the shape-mismatch is acceptable as the surfaced
+ // error, but it should be a SearchError from this module.
+ expect(err?.code).toBe("EMBEDDING_PROVIDER_TIMEOUT");
+}, 15_000);
+
+test("embed() rejects EMBEDDING_DIMENSION_MISMATCH across batch", async () => {
+ let i = 0;
+ server.setHandler((req) => {
+ const body = req.body as { input: string[]; model: string };
+ const data = body.input.map((_, idx) => ({
+ object: "embedding",
+ embedding: i++ === 0 ? [1, 0, 0, 0] : [1, 0, 0],
+ index: idx,
+ }));
+ return { status: 200, body: { data, model: body.model } };
+ });
+ const p = new OpenAICompatProvider(cfg({ batchSize: 1, concurrency: 1 }));
+ let err: SearchError | null = null;
+ try {
+ await p.embed(["a", "b"]);
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("EMBEDDING_DIMENSION_MISMATCH");
+});
+
+test("ping() returns ok with dimension on success", async () => {
+ const p = new OpenAICompatProvider(cfg());
+ const res = await p.ping();
+ if (!res.ok) throw new Error("expected ok");
+ expect(res.dimension).toBeGreaterThan(0);
+});
+
+test("ping() returns ok:false when server is unreachable", async () => {
+ await server.close();
+ const p = new OpenAICompatProvider(cfg(), { backoffMs: [10] });
+ const res = await p.ping();
+ expect(res.ok).toBe(false);
+});
+
+test("makeProvider returns NullProvider when semantic is disabled", () => {
+ const p = makeProvider({
+ enabled: false,
+ provider: "disabled",
+ baseUrl: null,
+ model: null,
+ apiKey: null,
+ dimension: null,
+ timeoutMs: 10_000,
+ concurrency: 1,
+ batchSize: 1,
+ });
+ expect(p).toBeInstanceOf(NullProvider);
+});
+
+test("makeProvider throws when key is missing under openai-compat", () => {
+ expect(() =>
+ makeProvider({
+ enabled: true,
+ provider: "openai-compat",
+ baseUrl: "https://x/v1",
+ model: "m",
+ apiKey: null,
+ dimension: null,
+ timeoutMs: 10_000,
+ concurrency: 1,
+ batchSize: 1,
+ }),
+ ).toThrow(/embedding_api_key/);
+});
+
+test("NullProvider.embed() throws EMBEDDING_DISABLED", async () => {
+ const p = new NullProvider();
+ let err: SearchError | null = null;
+ try {
+ await p.embed(["x"]);
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("EMBEDDING_DISABLED");
+});
diff --git a/tests/core/search/fts.test.ts b/tests/core/search/fts.test.ts
new file mode 100644
index 00000000..f84f6127
--- /dev/null
+++ b/tests/core/search/fts.test.ts
@@ -0,0 +1,133 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+
+import { buildFtsMatch, runFtsQuery } from "../../../src/core/search/fts.ts";
+import { Store } from "../../../src/core/search/store.ts";
+import { createTempVault, makeConfig } from "../../helpers/search-fixtures.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ const v = createTempVault("fts");
+ vault = v.vault;
+ dbPath = v.dbPath;
+ cleanup = v.cleanup;
+});
+
+afterEach(() => {
+ cleanup();
+});
+
+async function fixture() {
+ const store = await Store.open(makeConfig({ vault, dbPath }), { mode: "write", loadVec: false });
+ const d1 = store.upsertDocument({
+ path: "Notes/alpha.md",
+ title: "Alpha",
+ contentHash: "h1",
+ mtime: 1700000000,
+ size: 10,
+ });
+ store.replaceChunks(d1, [
+ {
+ chunkIndex: 0,
+ content: "the quick brown fox jumps over the lazy dog",
+ contentHash: "c1",
+ startLine: 1,
+ endLine: 1,
+ tokenCount: 9,
+ },
+ {
+ chunkIndex: 1,
+ content: "Π±Ρ€ΠΎΠΆΡƒ ΠΏΠΎ Ρ‚ΠΈΡ…ΠΈΠΌ ΡƒΠ»ΠΈΡ†Π°ΠΌ ΠΏΠ΅Ρ‚Π΅Ρ€Π±ΡƒΡ€Π³Π°. white nights.",
+ contentHash: "c2",
+ startLine: 2,
+ endLine: 2,
+ tokenCount: 7,
+ },
+ ]);
+ const d2 = store.upsertDocument({
+ path: "Other/beta.md",
+ title: "Beta",
+ contentHash: "h2",
+ mtime: 1700001000,
+ size: 10,
+ });
+ store.replaceChunks(d2, [
+ {
+ chunkIndex: 0,
+ content: "fox in beta document repeats: fox fox fox",
+ contentHash: "c3",
+ startLine: 1,
+ endLine: 1,
+ tokenCount: 7,
+ },
+ ]);
+ return { store, d1, d2 };
+}
+
+test("buildFtsMatch quotes tokens for safety", () => {
+ expect(buildFtsMatch("hello")).toBe('"hello"');
+ expect(buildFtsMatch("two words")).toBe('"two" "words"');
+});
+
+test("buildFtsMatch escapes embedded double quotes", () => {
+ expect(buildFtsMatch('he said "hi"')).toBe('"he" "said" """hi"""');
+});
+
+test("buildFtsMatch ignores empty input", () => {
+ expect(buildFtsMatch("")).toBe("");
+ expect(buildFtsMatch(" ")).toBe("");
+});
+
+test("buildFtsMatch defangs FTS5 operator words and metacharacters", () => {
+ // Without quoting, "AND" would be a boolean operator and "*" a prefix glob.
+ expect(buildFtsMatch("AND foo*")).toBe('"AND" "foo*"');
+});
+
+test("runFtsQuery returns BM25-ordered hits across documents", async () => {
+ const { store, d2 } = await fixture();
+ const hits = runFtsQuery(store, "fox", { limit: 10 });
+ expect(hits.length).toBeGreaterThanOrEqual(2);
+ // Beta document has the term repeated four times β†’ should rank above alpha.
+ expect(hits[0]?.documentId).toBe(d2);
+ await store.close();
+});
+
+test("runFtsQuery respects path_prefix filter", async () => {
+ const { store, d1 } = await fixture();
+ const hits = runFtsQuery(store, "fox", { limit: 10, pathPrefix: "Notes/" });
+ expect(hits.length).toBe(1);
+ expect(hits[0]?.documentId).toBe(d1);
+ await store.close();
+});
+
+test("runFtsQuery finds Cyrillic content", async () => {
+ const { store, d1 } = await fixture();
+ const hits = runFtsQuery(store, "Ρ‚ΠΈΡ…ΠΈΠΌ", { limit: 10 });
+ expect(hits.length).toBe(1);
+ expect(hits[0]?.documentId).toBe(d1);
+ await store.close();
+});
+
+test("runFtsQuery returns empty array on empty query", async () => {
+ const { store } = await fixture();
+ expect(runFtsQuery(store, " ", { limit: 5 })).toEqual([]);
+ await store.close();
+});
+
+test("runFtsQuery survives FTS5-operator-like queries safely", async () => {
+ const { store } = await fixture();
+ // Without quoting, the bare token "AND" would be a syntax error (left-hand
+ // operand missing). Quoted as a phrase it's just a literal word match.
+ // No row contains "AND" literally so the result is empty, but the query
+ // must not throw and must not return spurious matches.
+ expect(() => runFtsQuery(store, "AND", { limit: 5 })).not.toThrow();
+ expect(runFtsQuery(store, "AND", { limit: 5 })).toEqual([]);
+ // A query mixing operator-like tokens with a real token returns matches
+ // when all tokens are present in the document.
+ const hits = runFtsQuery(store, "fox", { limit: 5 });
+ expect(hits.length).toBeGreaterThan(0);
+ await store.close();
+});
+
diff --git a/tests/core/search/indexer.embeddings.test.ts b/tests/core/search/indexer.embeddings.test.ts
new file mode 100644
index 00000000..d1f7b0df
--- /dev/null
+++ b/tests/core/search/indexer.embeddings.test.ts
@@ -0,0 +1,138 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { Store } from "../../../src/core/search/store.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+let server: FakeHttp;
+
+beforeEach(async () => {
+ const v = createTempVault("indexer-emb");
+ vault = v.vault;
+ dbPath = v.dbPath;
+ cleanup = v.cleanup;
+ server = await startFakeHttp();
+});
+
+afterEach(async () => {
+ cleanup();
+ await server.close();
+});
+
+function semanticConfig(model = "fake-model", dim = 4) {
+ return makeConfig({
+ vault,
+ dbPath,
+ semantic: {
+ enabled: true,
+ provider: "openai-compat",
+ baseUrl: server.url,
+ model,
+ apiKey: "test-key",
+ dimension: dim,
+ timeoutMs: 5_000,
+ concurrency: 2,
+ batchSize: 8,
+ },
+ });
+}
+
+test("index --embeddings populates chunk_vec for new chunks", async () => {
+ writeMd(vault, "a.md", "# A\n\nFirst note about something.");
+ writeMd(vault, "b.md", "# B\n\nSecond note discussing things.");
+ const cfg = semanticConfig();
+
+ const stats = await indexVault(cfg, { embeddings: true });
+ expect(stats.embeddingsComputed).toBeGreaterThanOrEqual(2);
+
+ const store = await Store.open(cfg, { mode: "read" });
+ const counts = store.counts();
+ expect(counts.embeddings).toBe(counts.chunks);
+ await store.close();
+});
+
+test("unchanged files do NOT trigger re-embedding on the next run", async () => {
+ writeMd(vault, "a.md", "# A\n\nHello.");
+ const cfg = semanticConfig();
+ await indexVault(cfg, { embeddings: true });
+ const before = server.callCount();
+
+ const second = await indexVault(cfg, { embeddings: true });
+ expect(second.embeddingsComputed).toBe(0);
+ expect(server.callCount()).toBe(before);
+});
+
+test("changing the embedding model drops embeddings and the next index repopulates", async () => {
+ writeMd(vault, "a.md", "# A\n\nHello.");
+ const first = await indexVault(semanticConfig("modelA", 4), { embeddings: true });
+ expect(first.embeddingsComputed).toBeGreaterThan(0);
+
+ const before = server.callCount();
+ const second = await indexVault(semanticConfig("modelB", 4), { embeddings: true });
+ // All chunks re-embedded under the new model.
+ expect(second.embeddingsComputed).toBeGreaterThan(0);
+ expect(server.callCount()).toBeGreaterThan(before);
+});
+
+test("--embeddings without a key throws EMBEDDING_KEY_MISSING", async () => {
+ writeMd(vault, "a.md", "# A");
+ const cfg = makeConfig({
+ vault,
+ dbPath,
+ semantic: {
+ enabled: true,
+ provider: "openai-compat",
+ baseUrl: server.url,
+ model: "fake-model",
+ apiKey: null,
+ dimension: 4,
+ timeoutMs: 5_000,
+ concurrency: 1,
+ batchSize: 8,
+ },
+ });
+ let err: SearchError | null = null;
+ try {
+ await indexVault(cfg, { embeddings: true });
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("EMBEDDING_KEY_MISSING");
+});
+
+test("--embeddings with semantic disabled throws EMBEDDING_DISABLED", async () => {
+ writeMd(vault, "a.md", "# A");
+ const cfg = makeConfig({ vault, dbPath }); // semantic.enabled defaults to false
+ let err: SearchError | null = null;
+ try {
+ await indexVault(cfg, { embeddings: true });
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("EMBEDDING_DISABLED");
+});
+
+test("onFile callback fires for every classification", async () => {
+ writeMd(vault, "stay.md", "# A");
+ writeMd(vault, "go.md", "# B");
+ const cfg = makeConfig({ vault, dbPath });
+ await indexVault(cfg);
+
+ // Change one, delete another, add a third.
+ writeMd(vault, "stay.md", "# A\n\nupdated body.");
+ await Bun.$`rm ${vault}/go.md`.quiet();
+ writeMd(vault, "new.md", "# C");
+
+ const events: Array<{ kind: string; path: string }> = [];
+ await indexVault(cfg, { onFile: (e) => events.push({ kind: e.kind, path: e.path }) });
+
+ const kinds = new Set(events.map((e) => e.kind));
+ expect(kinds.has("added")).toBe(true);
+ expect(kinds.has("updated")).toBe(true);
+ expect(kinds.has("deleted")).toBe(true);
+});
diff --git a/tests/core/search/indexer.test.ts b/tests/core/search/indexer.test.ts
new file mode 100644
index 00000000..58c5e981
--- /dev/null
+++ b/tests/core/search/indexer.test.ts
@@ -0,0 +1,165 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { existsSync, rmSync, utimesSync, writeFileSync } from "node:fs";
+import { join } from "node:path";
+
+import { indexVault, reindexVault, indexStatus, indexCheck } from "../../../src/core/search/indexer.ts";
+import { Store } from "../../../src/core/search/store.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ const v = createTempVault("indexer");
+ vault = v.vault;
+ dbPath = v.dbPath;
+ cleanup = v.cleanup;
+});
+
+afterEach(() => {
+ cleanup();
+});
+
+test("first index adds every file; second run reports unchanged", async () => {
+ writeMd(vault, "a.md", "# A\n\nFirst note.");
+ writeMd(vault, "sub/b.md", "# B\n\nSecond note.");
+ const cfg = makeConfig({ vault, dbPath });
+
+ const first = await indexVault(cfg);
+ expect(first.added).toBe(2);
+ expect(first.updated).toBe(0);
+ expect(first.unchanged).toBe(0);
+ expect(first.deleted).toBe(0);
+ expect(first.chunksTotal).toBeGreaterThanOrEqual(2);
+
+ const second = await indexVault(cfg);
+ expect(second.added).toBe(0);
+ expect(second.updated).toBe(0);
+ expect(second.unchanged).toBe(2);
+ expect(second.embeddingsComputed).toBe(0);
+});
+
+test("modifying a file produces an `updated` event and replaces chunks", async () => {
+ const abs = writeMd(vault, "x.md", "# Old\n\noriginal content.");
+ const cfg = makeConfig({ vault, dbPath });
+ await indexVault(cfg);
+
+ // Mutate content and mtime.
+ writeFileSync(abs, "# New\n\nnew content that should be found.");
+ const t = Date.now() / 1000 + 60;
+ utimesSync(abs, t, t);
+
+ const events: Array<{ path: string; kind: string }> = [];
+ const stats = await indexVault(cfg, { onFile: (e) => events.push({ path: e.path, kind: e.kind }) });
+ expect(stats.updated).toBe(1);
+ expect(events.find((e) => e.path === "x.md")?.kind).toBe("updated");
+
+ // The stale text is gone from FTS.
+ const store = await Store.open(cfg, { mode: "read", loadVec: false });
+ expect(store.keywordTopK('"original"', { limit: 5 }).length).toBe(0);
+ expect(store.keywordTopK('"new"', { limit: 5 }).length).toBe(1);
+ await store.close();
+});
+
+test("removing a file from disk produces a `deleted` event next run", async () => {
+ writeMd(vault, "stays.md", "# A");
+ const removed = writeMd(vault, "removed.md", "# B");
+ const cfg = makeConfig({ vault, dbPath });
+ await indexVault(cfg);
+ rmSync(removed);
+
+ const events: string[] = [];
+ const stats = await indexVault(cfg, { onFile: (e) => events.push(`${e.kind}:${e.path}`) });
+ expect(stats.deleted).toBe(1);
+ expect(events).toContain("deleted:removed.md");
+
+ const store = await Store.open(cfg, { mode: "read", loadVec: false });
+ expect(store.listDocuments().size).toBe(1);
+ await store.close();
+});
+
+test("indexStatus reports not initialised when index file is absent", async () => {
+ const cfg = makeConfig({ vault, dbPath });
+ const status = await indexStatus(cfg);
+ expect(status.exists).toBe(false);
+ expect(status.documents).toBe(0);
+});
+
+test("indexStatus reports accurate counts after an index run", async () => {
+ writeMd(vault, "a.md", "# A\n\nbody");
+ writeMd(vault, "b.md", "# B\n\nbody");
+ const cfg = makeConfig({ vault, dbPath });
+ await indexVault(cfg);
+
+ const status = await indexStatus(cfg);
+ expect(status.exists).toBe(true);
+ expect(status.schemaVersion).toBe(1);
+ expect(status.documents).toBe(2);
+ expect(status.chunks).toBeGreaterThanOrEqual(2);
+ expect(status.lastIndexedAt).toBeTruthy();
+});
+
+test("indexCheck reports vault readable, sqlite ok, fts ok", async () => {
+ const cfg = makeConfig({ vault, dbPath });
+ const report = await indexCheck(cfg);
+ expect(report.vaultReadable).toBe(true);
+ expect(report.indexDirWritable).toBe(true);
+ expect(report.sqliteOk).toBe(true);
+ expect(report.fts5Ok).toBe(true);
+ expect(report.fatal.length).toBe(0);
+});
+
+test("reindexVault rebuilds atomically and preserves .bak", async () => {
+ writeMd(vault, "a.md", "# A");
+ const cfg = makeConfig({ vault, dbPath });
+ await indexVault(cfg);
+
+ expect(existsSync(dbPath)).toBe(true);
+ expect(existsSync(dbPath + ".bak")).toBe(false);
+
+ writeMd(vault, "b.md", "# B");
+ const stats = await reindexVault(cfg);
+ expect(stats.added).toBeGreaterThanOrEqual(2);
+ expect(existsSync(dbPath)).toBe(true);
+ expect(existsSync(dbPath + ".bak")).toBe(true);
+ expect(existsSync(dbPath + ".new")).toBe(false);
+
+ // Stats reflect the rebuilt index (2 docs from scratch).
+ const status = await indexStatus(cfg);
+ expect(status.documents).toBe(2);
+});
+
+test("Store.open auto-restores from .bak when main file is missing", async () => {
+ writeMd(vault, "a.md", "# A");
+ const cfg = makeConfig({ vault, dbPath });
+ await indexVault(cfg);
+ await reindexVault(cfg);
+
+ // Simulate crash mid-reindex by removing the main file (we still have .bak).
+ rmSync(dbPath);
+ expect(existsSync(dbPath + ".bak")).toBe(true);
+
+ const status = await indexStatus(cfg);
+ expect(status.exists).toBe(true);
+});
+
+test("UTF-8 only: invalid bytes record an error, do not crash the run", async () => {
+ // Latin-1 high-byte content is not valid UTF-8 sequencing here.
+ const bad = join(vault, "broken.md");
+ writeFileSync(bad, Buffer.from([0xc3, 0x28])); // 0xc3 0x28 is invalid UTF-8 sequence
+ writeMd(vault, "good.md", "valid content");
+ const cfg = makeConfig({ vault, dbPath });
+
+ const stats = await indexVault(cfg);
+ expect(stats.added).toBe(1); // only good.md
+ expect(stats.errors.some((e) => e.path === "broken.md")).toBe(true);
+});
+
+test("indexVault tolerates empty vault", async () => {
+ const cfg = makeConfig({ vault, dbPath });
+ const stats = await indexVault(cfg);
+ expect(stats.added).toBe(0);
+ expect(stats.chunksTotal).toBe(0);
+});
+
diff --git a/tests/core/search/links.test.ts b/tests/core/search/links.test.ts
new file mode 100644
index 00000000..9252af07
--- /dev/null
+++ b/tests/core/search/links.test.ts
@@ -0,0 +1,82 @@
+import { test, expect } from "bun:test";
+import { extractLinks } from "../../../src/core/search/links.ts";
+
+test("extracts wikilinks with and without alt text", () => {
+ const links = extractLinks("See [[foo-note]] and [[bar|the bar]].");
+ const wikilinks = links.filter((l) => l.linkType === "wikilink");
+ expect(wikilinks.length).toBe(2);
+ expect(wikilinks[0]).toEqual({ linkType: "wikilink", targetPath: "foo-note", linkText: null });
+ expect(wikilinks[1]).toEqual({ linkType: "wikilink", targetPath: "bar", linkText: "the bar" });
+});
+
+test("extracts relative markdown links and skips external URLs", () => {
+ const text = "See [the spec](./design.md) and [Google](https://google.com).";
+ const links = extractLinks(text);
+ const md = links.filter((l) => l.linkType === "markdown_link");
+ expect(md.length).toBe(1);
+ expect(md[0]?.targetPath).toBe("./design.md");
+ expect(md[0]?.linkText).toBe("the spec");
+});
+
+test("image embeds (`![alt](url)`) are not captured as markdown links", () => {
+ const text = "![cover](assets/cover.png) and a real [Spec](./design.md)";
+ const links = extractLinks(text);
+ const md = links.filter((l) => l.linkType === "markdown_link");
+ expect(md.length).toBe(1);
+ expect(md[0]?.targetPath).toBe("./design.md");
+ expect(md.find((l) => l.targetPath === "assets/cover.png")).toBeUndefined();
+});
+
+test("strips anchor fragments from markdown link targets", () => {
+ const links = extractLinks("[Anchor](notes/x.md#section)");
+ const md = links.filter((l) => l.linkType === "markdown_link");
+ expect(md[0]?.targetPath).toBe("notes/x.md");
+});
+
+test("extracts Obsidian-style tags including hierarchy", () => {
+ const links = extractLinks("Tagged #foo and #bar/baz here. Not a tag: word#nope.");
+ const tags = links.filter((l) => l.linkType === "tag").map((l) => l.linkText);
+ expect(tags).toContain("foo");
+ expect(tags).toContain("bar/baz");
+ expect(tags).not.toContain("nope");
+});
+
+test("ignores wikilinks and tags inside fenced code blocks", () => {
+ const text = `Outside [[real-link]].
+
+\`\`\`
+this is code with [[fake-link]] and #not-a-tag
+\`\`\`
+
+After fence.`;
+ const links = extractLinks(text);
+ const wikilinks = links.filter((l) => l.linkType === "wikilink").map((l) => l.targetPath);
+ expect(wikilinks).toContain("real-link");
+ expect(wikilinks).not.toContain("fake-link");
+ const tags = links.filter((l) => l.linkType === "tag").map((l) => l.linkText);
+ expect(tags).not.toContain("not-a-tag");
+});
+
+test("ignores wikilinks inside inline code spans", () => {
+ const links = extractLinks("Inline `[[code-link]]` and real [[outside]].");
+ const wikilinks = links.filter((l) => l.linkType === "wikilink").map((l) => l.targetPath);
+ expect(wikilinks).toEqual(["outside"]);
+});
+
+test("deduplicates identical links from the same chunk", () => {
+ const links = extractLinks("[[same]] [[same]] [[same|with alt]] #tag #tag");
+ const wikilinks = links.filter((l) => l.linkType === "wikilink");
+ // Two distinct: (same, null) and (same, "with alt")
+ expect(wikilinks.length).toBe(2);
+ const tags = links.filter((l) => l.linkType === "tag");
+ expect(tags.length).toBe(1);
+});
+
+test("empty content returns empty array", () => {
+ expect(extractLinks("")).toEqual([]);
+});
+
+test("skips mailto:", () => {
+ const links = extractLinks("[email me](mailto:[email protected])");
+ expect(links.length).toBe(0);
+});
diff --git a/tests/core/search/paths.test.ts b/tests/core/search/paths.test.ts
new file mode 100644
index 00000000..491612b1
--- /dev/null
+++ b/tests/core/search/paths.test.ts
@@ -0,0 +1,14 @@
+import { test, expect } from "bun:test";
+import { resolveIndexPath } from "../../../src/core/search/paths.ts";
+
+test("default points under <vault>/.open-second-brain", () => {
+ expect(resolveIndexPath("/v", null)).toBe("/v/.open-second-brain/brain.sqlite");
+});
+
+test("explicit override wins", () => {
+ expect(resolveIndexPath("/v", "/tmp/custom.sqlite")).toBe("/tmp/custom.sqlite");
+});
+
+test("blank override falls back to default", () => {
+ expect(resolveIndexPath("/v", "")).toBe("/v/.open-second-brain/brain.sqlite");
+});
diff --git a/tests/core/search/ranker.test.ts b/tests/core/search/ranker.test.ts
new file mode 100644
index 00000000..7fd79a30
--- /dev/null
+++ b/tests/core/search/ranker.test.ts
@@ -0,0 +1,238 @@
+import { test, expect } from "bun:test";
+import { rankResults } from "../../../src/core/search/ranker.ts";
+import type { KeywordHit, SemanticHit, HydratedChunk } from "../../../src/core/search/store.ts";
+
+function hyd(chunkId: number, docId: number, mtime: number): HydratedChunk {
+ return Object.freeze({
+ chunkId,
+ documentId: docId,
+ path: `doc${docId}.md`,
+ title: `Doc ${docId}`,
+ content: `chunk ${chunkId}`,
+ startLine: 1,
+ endLine: 1,
+ mtime,
+ });
+}
+
+const NOW = 1_750_000_000_000; // ms
+const RECENT_MTIME = NOW / 1000 - 3 * 24 * 3600; // 3 days ago in unix seconds β†’ 0.05 recency
+const OLD_MTIME = NOW / 1000 - 365 * 24 * 3600; // >90d β†’ 0 recency
+
+test("union of keyword + semantic produces hybrid scoring", () => {
+ const kw: KeywordHit[] = [
+ { chunkId: 1, documentId: 10, bm25: -5 },
+ { chunkId: 2, documentId: 11, bm25: -2 },
+ ];
+ const sem: SemanticHit[] = [
+ { chunkId: 2, documentId: 11, distance: 0.5 },
+ { chunkId: 3, documentId: 12, distance: 0.1 },
+ ];
+ const hydrated = new Map<number, HydratedChunk>([
+ [1, hyd(1, 10, OLD_MTIME)],
+ [2, hyd(2, 11, OLD_MTIME)],
+ [3, hyd(3, 12, OLD_MTIME)],
+ ]);
+ const ranked = rankResults(
+ {
+ keyword: kw,
+ semantic: sem,
+ hydrated,
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ },
+ { keywordWeight: 0.6, semanticWeight: 0.4, limit: 10, nowMs: NOW },
+ );
+ const map = new Map(ranked.map((r) => [r.chunkId, r]));
+ expect(map.get(1)?.searchType).toBe("keyword");
+ expect(map.get(2)?.searchType).toBe("hybrid");
+ expect(map.get(3)?.searchType).toBe("semantic");
+});
+
+test("score is clamped to [0,1] even with boosts", () => {
+ const ranked = rankResults(
+ {
+ keyword: [{ chunkId: 1, documentId: 10, bm25: -10 }],
+ semantic: [{ chunkId: 1, documentId: 10, distance: 0 }],
+ hydrated: new Map([[1, hyd(1, 10, RECENT_MTIME)]]),
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ },
+ { keywordWeight: 0.6, semanticWeight: 0.4, limit: 10, nowMs: NOW },
+ );
+ expect(ranked[0]?.score).toBeGreaterThan(0);
+ expect(ranked[0]?.score).toBeLessThanOrEqual(1);
+});
+
+test("keyword-only fallback ignores semantic when semanticEnabled=false", () => {
+ const ranked = rankResults(
+ {
+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],
+ semantic: [{ chunkId: 2, documentId: 11, distance: 0.0 }],
+ hydrated: new Map([
+ [1, hyd(1, 10, OLD_MTIME)],
+ [2, hyd(2, 11, OLD_MTIME)],
+ ]),
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ },
+ {
+ keywordWeight: 1.0,
+ semanticWeight: 0,
+ limit: 10,
+ nowMs: NOW,
+ semanticEnabled: false,
+ },
+ );
+ expect(ranked.length).toBe(1);
+ expect(ranked[0]?.chunkId).toBe(1);
+ expect(ranked[0]?.searchType).toBe("keyword");
+});
+
+test("min-max normalisation: best BM25 hit gets 1.0, worst 0.0", () => {
+ const ranked = rankResults(
+ {
+ keyword: [
+ { chunkId: 1, documentId: 10, bm25: -10 },
+ { chunkId: 2, documentId: 11, bm25: -1 },
+ ],
+ semantic: [],
+ hydrated: new Map([
+ [1, hyd(1, 10, OLD_MTIME)],
+ [2, hyd(2, 11, OLD_MTIME)],
+ ]),
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ },
+ { keywordWeight: 1.0, semanticWeight: 0, limit: 10, nowMs: NOW },
+ );
+ const c1 = ranked.find((r) => r.chunkId === 1)!;
+ const c2 = ranked.find((r) => r.chunkId === 2)!;
+ expect(c1.keywordScore).toBe(1);
+ expect(c2.keywordScore).toBe(0);
+});
+
+test("single-result keyword gets keywordScore=1 (max==min branch)", () => {
+ const ranked = rankResults(
+ {
+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],
+ semantic: [],
+ hydrated: new Map([[1, hyd(1, 10, OLD_MTIME)]]),
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ },
+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },
+ );
+ expect(ranked[0]?.keywordScore).toBe(1);
+});
+
+test("link_boost adds +0.02 when another candidate doc points to this chunk's doc", () => {
+ const inbound = new Map<number, ReadonlySet<number>>([
+ // chunk 1 (doc 10) is pointed to by doc 11
+ [1, new Set([11])],
+ ]);
+ const ranked = rankResults(
+ {
+ keyword: [
+ { chunkId: 1, documentId: 10, bm25: -5 },
+ { chunkId: 2, documentId: 11, bm25: -5 },
+ ],
+ semantic: [],
+ hydrated: new Map([
+ [1, hyd(1, 10, OLD_MTIME)],
+ [2, hyd(2, 11, OLD_MTIME)],
+ ]),
+ inboundLinkSources: inbound,
+ tagsByDoc: new Map(),
+ },
+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },
+ );
+ const c1 = ranked.find((r) => r.chunkId === 1)!;
+ expect(c1.linkBoost).toBeCloseTo(0.02, 6);
+});
+
+test("tag_boost adds +0.01 when candidates share a tag", () => {
+ const tags = new Map<number, ReadonlySet<string>>([
+ [1, new Set(["shared"])],
+ [2, new Set(["shared", "extra"])],
+ ]);
+ const ranked = rankResults(
+ {
+ keyword: [
+ { chunkId: 1, documentId: 10, bm25: -5 },
+ { chunkId: 2, documentId: 11, bm25: -5 },
+ ],
+ semantic: [],
+ hydrated: new Map([
+ [1, hyd(1, 10, OLD_MTIME)],
+ [2, hyd(2, 11, OLD_MTIME)],
+ ]),
+ inboundLinkSources: new Map(),
+ tagsByDoc: tags,
+ },
+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },
+ );
+ expect(ranked[0]?.linkBoost).toBeCloseTo(0.01, 6);
+});
+
+test("recency_boost is 0.05 for ≀7d, 0.025 for ≀30d, 0.01 for ≀90d, 0 older", () => {
+ const make = (mt: number) =>
+ rankResults(
+ {
+ keyword: [{ chunkId: 1, documentId: 10, bm25: -5 }],
+ semantic: [],
+ hydrated: new Map([[1, hyd(1, 10, mt)]]),
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ },
+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },
+ )[0]!.recencyBoost;
+
+ const baseSec = NOW / 1000;
+ expect(make(baseSec - 3 * 86400)).toBe(0.05);
+ expect(make(baseSec - 20 * 86400)).toBe(0.025);
+ expect(make(baseSec - 60 * 86400)).toBe(0.01);
+ expect(make(baseSec - 200 * 86400)).toBe(0);
+});
+
+test("tie-break: equal final_score β†’ higher keywordScore wins; then mtime; then chunkId", () => {
+ // Two chunks with identical score after combination: same kw=1, no semantic, no boosts.
+ const ranked = rankResults(
+ {
+ keyword: [
+ { chunkId: 2, documentId: 11, bm25: -5 },
+ { chunkId: 1, documentId: 10, bm25: -5 },
+ ],
+ semantic: [],
+ hydrated: new Map([
+ [1, hyd(1, 10, OLD_MTIME)],
+ [2, hyd(2, 11, OLD_MTIME)],
+ ]),
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ },
+ { keywordWeight: 1, semanticWeight: 0, limit: 10, nowMs: NOW },
+ );
+ // Same score, same kw, same mtime, lower chunkId first β†’ [1, 2]
+ expect(ranked.map((r) => r.chunkId)).toEqual([1, 2]);
+});
+
+test("limit truncates after ranking", () => {
+ const kw: KeywordHit[] = [];
+ const hydrated = new Map<number, HydratedChunk>();
+ for (let i = 0; i < 30; i++) {
+ kw.push({ chunkId: i + 1, documentId: 100 + i, bm25: -(i + 1) });
+ hydrated.set(i + 1, hyd(i + 1, 100 + i, OLD_MTIME));
+ }
+ const ranked = rankResults(
+ {
+ keyword: kw,
+ semantic: [],
+ hydrated,
+ inboundLinkSources: new Map(),
+ tagsByDoc: new Map(),
+ },
+ { keywordWeight: 1, semanticWeight: 0, limit: 5, nowMs: NOW },
+ );
+ expect(ranked.length).toBe(5);
+});
diff --git a/tests/core/search/schema.test.ts b/tests/core/search/schema.test.ts
new file mode 100644
index 00000000..38976114
--- /dev/null
+++ b/tests/core/search/schema.test.ts
@@ -0,0 +1,145 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { Database } from "bun:sqlite";
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import {
+ applyMigrations,
+ LATEST_SCHEMA_VERSION,
+ readSchemaVersion,
+ ensureVecTable,
+ dropVecTable,
+} from "../../../src/core/search/schema.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+
+let tmp: string;
+let dbPath: string;
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "osb-schema-"));
+ dbPath = join(tmp, "test.sqlite");
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+function tableNames(db: Database): Set<string> {
+ const rows = db
+ .query<{ name: string }, []>(
+ "SELECT name FROM sqlite_master WHERE type IN ('table','virtual') OR type='table' OR type='trigger' OR type='view'",
+ )
+ .all();
+ return new Set(rows.map((r) => r.name));
+}
+
+function objectNames(db: Database, type: string): Set<string> {
+ const rows = db
+ .query<{ name: string }, [string]>(
+ "SELECT name FROM sqlite_master WHERE type = ? AND name NOT LIKE 'sqlite_%'",
+ )
+ .all(type);
+ return new Set(rows.map((r) => r.name));
+}
+
+test("applyMigrations on a fresh db creates the v1 schema", () => {
+ const db = new Database(dbPath);
+ const v = applyMigrations(db);
+ expect(v).toBe(LATEST_SCHEMA_VERSION);
+
+ const tables = objectNames(db, "table");
+ expect(tables.has("documents")).toBe(true);
+ expect(tables.has("chunks")).toBe(true);
+ expect(tables.has("embeddings")).toBe(true);
+ expect(tables.has("chunk_vec_map")).toBe(true);
+ expect(tables.has("links")).toBe(true);
+ expect(tables.has("index_state")).toBe(true);
+
+ const triggers = objectNames(db, "trigger");
+ expect(triggers.has("chunks_ai")).toBe(true);
+ expect(triggers.has("chunks_ad")).toBe(true);
+ expect(triggers.has("chunks_au")).toBe(true);
+
+ // FTS5 virtual table is registered as type 'table'.
+ expect(tables.has("chunk_fts")).toBe(true);
+
+ expect(readSchemaVersion(db)).toBe(1);
+ db.close();
+});
+
+test("applyMigrations is idempotent", () => {
+ const db = new Database(dbPath);
+ applyMigrations(db);
+ const first = tableNames(db).size;
+ applyMigrations(db);
+ applyMigrations(db);
+ expect(tableNames(db).size).toBe(first);
+ expect(readSchemaVersion(db)).toBe(1);
+ db.close();
+});
+
+test("readSchemaVersion returns 0 when index_state is missing", () => {
+ const db = new Database(dbPath);
+ expect(() => readSchemaVersion(db)).toThrow(/no such table/);
+ db.close();
+});
+
+test("applyMigrations throws SCHEMA_MISMATCH if db is newer than binary", () => {
+ const db = new Database(dbPath);
+ applyMigrations(db);
+ // Simulate a future version.
+ db.run(
+ "UPDATE index_state SET value = ? WHERE key = 'schema_version'",
+ [String(LATEST_SCHEMA_VERSION + 7)],
+ );
+
+ let err: SearchError | null = null;
+ try {
+ applyMigrations(db);
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err).not.toBeNull();
+ expect(err?.code).toBe("SCHEMA_MISMATCH");
+ expect(err?.message).toContain("reindex");
+ db.close();
+});
+
+test("ensureVecTable + dropVecTable round-trip", () => {
+ // sqlite-vec is optional; load it for this test. If not available we
+ // skip β€” vec-specific behaviour is exercised in store.vec.test.ts.
+ const db = new Database(dbPath);
+ applyMigrations(db);
+
+ try {
+ const vec = require("sqlite-vec");
+ db.loadExtension(vec.getLoadablePath());
+ } catch {
+ db.close();
+ return;
+ }
+
+ ensureVecTable(db, 4);
+ ensureVecTable(db, 4); // idempotent
+
+ const tables = objectNames(db, "table");
+ expect(tables.has("chunk_vec")).toBe(true);
+
+ // Insert a vector.
+ db.run("INSERT INTO chunk_vec(rowid, embedding) VALUES (1, ?)", [JSON.stringify([0.1, 0.2, 0.3, 0.4])]);
+ const row = db.query<{ c: number }, []>("SELECT count(*) AS c FROM chunk_vec").get();
+ expect(row?.c).toBe(1);
+
+ dropVecTable(db);
+ expect(objectNames(db, "table").has("chunk_vec")).toBe(false);
+ db.close();
+});
+
+test("ensureVecTable rejects non-positive dimension", () => {
+ const db = new Database(dbPath);
+ applyMigrations(db);
+ expect(() => ensureVecTable(db, 0)).toThrow(/positive integer/);
+ expect(() => ensureVecTable(db, -1)).toThrow(/positive integer/);
+ db.close();
+});
diff --git a/tests/core/search/search.test.ts b/tests/core/search/search.test.ts
new file mode 100644
index 00000000..93dbfa75
--- /dev/null
+++ b/tests/core/search/search.test.ts
@@ -0,0 +1,185 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+
+import { indexVault } from "../../../src/core/search/indexer.ts";
+import { search } from "../../../src/core/search/search.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+import { createTempVault, makeConfig, writeMd } from "../../helpers/search-fixtures.ts";
+import { startFakeHttp, type FakeHttp } from "../../helpers/fake-http.ts";
+
+let vault: string;
+let dbPath: string;
+let cleanup: () => void;
+let server: FakeHttp;
+
+beforeEach(async () => {
+ const v = createTempVault("search");
+ vault = v.vault;
+ dbPath = v.dbPath;
+ cleanup = v.cleanup;
+ server = await startFakeHttp();
+});
+
+afterEach(async () => {
+ cleanup();
+ await server.close();
+});
+
+function semanticConfig() {
+ return makeConfig({
+ vault,
+ dbPath,
+ semantic: {
+ enabled: true,
+ provider: "openai-compat",
+ baseUrl: server.url,
+ model: "fake-model",
+ apiKey: "test-key",
+ dimension: 4,
+ timeoutMs: 5_000,
+ concurrency: 2,
+ batchSize: 8,
+ },
+ });
+}
+
+async function seedKeyword() {
+ writeMd(
+ vault,
+ "Notes/foo.md",
+ "# Foo\n\nThe quick brown fox jumps over the lazy dog. Foxes are interesting.",
+ );
+ writeMd(vault, "Other/bar.md", "# Bar\n\nA different note about cats and turtles.");
+ writeMd(vault, "Notes/ru.md", "# Ru\n\nΠšΠΈΡ€ΠΈΠ»Π»ΠΈΡ‡Π΅ΡΠΊΠΈΠΉ тСкст ΠΏΡ€ΠΎ Π±Ρ€ΠΎΠΆΠ΅Π½ΠΈΠ΅ тСстового Π·Π΅Ρ€Π½Π°.");
+ return makeConfig({ vault, dbPath });
+}
+
+test("search returns keyword-only results when semantic is disabled", async () => {
+ const cfg = await seedKeyword();
+ await indexVault(cfg);
+
+ const out = await search(cfg, { query: "fox", limit: 5 });
+ expect(out.results.length).toBe(1);
+ expect(out.results[0]?.path).toBe("Notes/foo.md");
+ expect(out.results[0]?.searchType).toBe("keyword");
+ expect(out.warnings).toEqual([]);
+});
+
+test("search returns empty (no error) when no results match", async () => {
+ const cfg = await seedKeyword();
+ await indexVault(cfg);
+ const out = await search(cfg, { query: "nonexistentterm", limit: 5 });
+ expect(out.results).toEqual([]);
+ expect(out.total).toBe(0);
+});
+
+test("search supports Cyrillic query against Cyrillic content", async () => {
+ const cfg = await seedKeyword();
+ await indexVault(cfg);
+ const out = await search(cfg, { query: "Π±Ρ€ΠΎΠΆΠ΅Π½ΠΈΠ΅", limit: 5 });
+ expect(out.results.length).toBe(1);
+ expect(out.results[0]?.path).toBe("Notes/ru.md");
+});
+
+test("path_prefix filter scopes results", async () => {
+ const cfg = await seedKeyword();
+ await indexVault(cfg);
+ const all = await search(cfg, { query: "quick", limit: 5 });
+ expect(all.results.length).toBe(1);
+ const scoped = await search(cfg, { query: "quick", limit: 5, pathPrefix: "Other/" });
+ expect(scoped.results.length).toBe(0);
+});
+
+test("path_prefix escaping is rejected with INVALID_INPUT", async () => {
+ const cfg = await seedKeyword();
+ await indexVault(cfg);
+ let err: SearchError | null = null;
+ try {
+ await search(cfg, { query: "x", pathPrefix: "../etc/" });
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("INVALID_INPUT");
+});
+
+test("empty query throws INVALID_INPUT", async () => {
+ const cfg = await seedKeyword();
+ await indexVault(cfg);
+ let err: SearchError | null = null;
+ try {
+ await search(cfg, { query: " " });
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("INVALID_INPUT");
+});
+
+test("missing index throws INDEX_MISSING", async () => {
+ const cfg = await seedKeyword();
+ // skip indexVault
+ let err: SearchError | null = null;
+ try {
+ await search(cfg, { query: "fox" });
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("INDEX_MISSING");
+});
+
+test("hybrid search combines keyword and semantic when both contribute", async () => {
+ writeMd(vault, "A/foo.md", "alpha beta gamma");
+ writeMd(vault, "A/bar.md", "delta epsilon zeta");
+ const cfg = semanticConfig();
+ await indexVault(cfg, { embeddings: true });
+
+ const out = await search(cfg, { query: "alpha", limit: 5, semantic: true });
+ expect(out.results.length).toBeGreaterThan(0);
+ // The keyword hit should be in the top result; searchType may be hybrid.
+ const first = out.results[0]!;
+ expect(["keyword", "hybrid", "semantic"]).toContain(first.searchType);
+});
+
+test("explicit --semantic without vec extension throws (configured-down case)", async () => {
+ await seedKeyword();
+ const cfg = makeConfig({
+ vault,
+ dbPath,
+ semantic: {
+ enabled: true,
+ provider: "openai-compat",
+ baseUrl: server.url,
+ model: "fake-model",
+ apiKey: "k",
+ dimension: 4,
+ timeoutMs: 5_000,
+ concurrency: 1,
+ batchSize: 8,
+ },
+ });
+ await indexVault(cfg);
+ // Now disable vec by closing the index and reopening through a config
+ // that uses a fresh dbPath without vec. We can't unload the extension β€”
+ // so emulate by stubbing config.semantic.enabled=true but no embeddings
+ // recorded. The store loads vec OK but counts.embeddings===0, which is
+ // the data-state warning case (warn + skip even when explicit).
+ const out = await search(cfg, { query: "fox", limit: 5, semantic: true });
+ expect(out.warnings.some((w) => w.includes("no compatible embeddings"))).toBe(true);
+});
+
+test("implicit semantic + no embeddings β†’ keyword-only + warning, no throw", async () => {
+ await seedKeyword();
+ const cfg = semanticConfig();
+ await indexVault(cfg); // no --embeddings
+ const out = await search(cfg, { query: "fox", limit: 5 });
+ expect(out.warnings.some((w) => w.includes("no compatible embeddings"))).toBe(true);
+ expect(out.results.length).toBeGreaterThan(0);
+});
+
+test("limit truncates results", async () => {
+ for (let i = 0; i < 20; i++) {
+ writeMd(vault, `notes/n${i}.md`, `# Note ${i}\n\nrepeated word repeated word repeated word`);
+ }
+ const cfg = makeConfig({ vault, dbPath });
+ await indexVault(cfg);
+ const out = await search(cfg, { query: "repeated", limit: 5 });
+ expect(out.results.length).toBe(5);
+});
diff --git a/tests/core/search/store.test.ts b/tests/core/search/store.test.ts
new file mode 100644
index 00000000..cb730cbb
--- /dev/null
+++ b/tests/core/search/store.test.ts
@@ -0,0 +1,287 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { mkdtempSync, rmSync, existsSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { Store } from "../../../src/core/search/store.ts";
+import { SearchError } from "../../../src/core/search/types.ts";
+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";
+
+let tmp: string;
+let dbPath: string;
+
+function makeConfig(overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchConfig {
+ const semantic: ResolvedEmbeddingConfig = Object.freeze({
+ enabled: false,
+ provider: "openai-compat",
+ baseUrl: null,
+ model: null,
+ apiKey: null,
+ dimension: null,
+ timeoutMs: 10_000,
+ concurrency: 4,
+ batchSize: 32,
+ });
+ return Object.freeze({
+ vault: tmp,
+ dbPath,
+ ignorePaths: Object.freeze([".git"]),
+ chunkSize: 800,
+ chunkOverlap: 100,
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ semantic,
+ ...(overrides ?? {}),
+ });
+}
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "osb-store-"));
+ dbPath = join(tmp, "brain.sqlite");
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+test("open in write mode creates file and applies migrations", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ expect(existsSync(dbPath)).toBe(true);
+ expect(store.schemaVersion()).toBe(1);
+ await store.close();
+});
+
+test("open in read mode throws INDEX_MISSING when file doesn't exist", async () => {
+ let err: SearchError | null = null;
+ try {
+ await Store.open(makeConfig(), { mode: "read", loadVec: false });
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("INDEX_MISSING");
+});
+
+test("open in read mode succeeds after writer creates the file", async () => {
+ const w = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ await w.close();
+ const r = await Store.open(makeConfig(), { mode: "read", loadVec: false });
+ expect(r.schemaVersion()).toBe(1);
+ await r.close();
+});
+
+test("upsertDocument is unique on path; second call updates same id", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ const id1 = store.upsertDocument({
+ path: "Daily/2026-05-16.md",
+ title: "2026-05-16",
+ contentHash: "h1",
+ mtime: 1700000000,
+ size: 12,
+ });
+ const id2 = store.upsertDocument({
+ path: "Daily/2026-05-16.md",
+ title: "renamed",
+ contentHash: "h2",
+ mtime: 1700000099,
+ size: 22,
+ });
+ expect(id2).toBe(id1);
+
+ const docs = store.listDocuments();
+ expect(docs.size).toBe(1);
+ expect(docs.get("Daily/2026-05-16.md")?.contentHash).toBe("h2");
+ await store.close();
+});
+
+test("replaceChunks populates FTS via triggers", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ const docId = store.upsertDocument({
+ path: "notes/hello.md",
+ title: "Hello",
+ contentHash: "h",
+ mtime: 1700000000,
+ size: 20,
+ });
+ const ids = store.replaceChunks(docId, [
+ {
+ chunkIndex: 0,
+ content: "hello world from open second brain",
+ contentHash: "c0",
+ startLine: 1,
+ endLine: 1,
+ tokenCount: 6,
+ },
+ {
+ chunkIndex: 1,
+ content: "second chunk talks about cats and dogs",
+ contentHash: "c1",
+ startLine: 2,
+ endLine: 2,
+ tokenCount: 8,
+ },
+ ]);
+ expect(ids.length).toBe(2);
+
+ const hits = store.keywordTopK("brain", { limit: 10 });
+ expect(hits.length).toBe(1);
+ expect(hits[0]?.chunkId).toBe(ids[0]!);
+
+ const noHits = store.keywordTopK("nonexistentterm", { limit: 10 });
+ expect(noHits.length).toBe(0);
+ await store.close();
+});
+
+test("replaceChunks twice replaces all old chunks", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ const docId = store.upsertDocument({
+ path: "x.md",
+ title: null,
+ contentHash: "h",
+ mtime: 0,
+ size: 1,
+ });
+ store.replaceChunks(docId, [
+ { chunkIndex: 0, content: "alpha bravo", contentHash: "c0", startLine: 1, endLine: 1, tokenCount: 2 },
+ ]);
+ store.replaceChunks(docId, [
+ { chunkIndex: 0, content: "charlie delta", contentHash: "c1", startLine: 1, endLine: 1, tokenCount: 2 },
+ ]);
+ expect(store.keywordTopK("alpha", { limit: 5 }).length).toBe(0);
+ expect(store.keywordTopK("charlie", { limit: 5 }).length).toBe(1);
+ expect(store.getChunksByDocument(docId).length).toBe(1);
+ await store.close();
+});
+
+test("deleteDocument cascades to chunks and FTS", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ const docId = store.upsertDocument({
+ path: "doomed.md",
+ title: null,
+ contentHash: "h",
+ mtime: 0,
+ size: 1,
+ });
+ store.replaceChunks(docId, [
+ { chunkIndex: 0, content: "doomed content", contentHash: "c", startLine: 1, endLine: 1, tokenCount: 2 },
+ ]);
+ expect(store.keywordTopK("doomed", { limit: 5 }).length).toBe(1);
+
+ store.deleteDocument("doomed.md");
+ expect(store.listDocuments().size).toBe(0);
+ expect(store.keywordTopK("doomed", { limit: 5 }).length).toBe(0);
+ await store.close();
+});
+
+test("path-prefix filter constrains keywordTopK", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ const d1 = store.upsertDocument({ path: "A/foo.md", title: null, contentHash: "h", mtime: 0, size: 1 });
+ const d2 = store.upsertDocument({ path: "B/bar.md", title: null, contentHash: "h", mtime: 0, size: 1 });
+ store.replaceChunks(d1, [
+ { chunkIndex: 0, content: "alpha alpha alpha", contentHash: "c", startLine: 1, endLine: 1, tokenCount: 3 },
+ ]);
+ store.replaceChunks(d2, [
+ { chunkIndex: 0, content: "alpha bravo", contentHash: "c", startLine: 1, endLine: 1, tokenCount: 2 },
+ ]);
+ const hitsA = store.keywordTopK("alpha", { limit: 10, pathPrefix: "A/" });
+ expect(hitsA.length).toBe(1);
+ expect(hitsA[0]?.documentId).toBe(d1);
+ await store.close();
+});
+
+test("path-prefix filter treats SQL wildcard characters literally", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ const literal = store.upsertDocument({ path: "A_/foo.md", title: null, contentHash: "h1", mtime: 0, size: 1 });
+ const wildcardMatch = store.upsertDocument({ path: "AB/bar.md", title: null, contentHash: "h2", mtime: 0, size: 1 });
+ store.replaceChunks(literal, [
+ { chunkIndex: 0, content: "alpha literal", contentHash: "c1", startLine: 1, endLine: 1, tokenCount: 2 },
+ ]);
+ store.replaceChunks(wildcardMatch, [
+ { chunkIndex: 0, content: "alpha wildcard", contentHash: "c2", startLine: 1, endLine: 1, tokenCount: 2 },
+ ]);
+ const hits = store.keywordTopK("alpha", { limit: 10, pathPrefix: "A_/" });
+ expect(hits.map((h) => h.documentId)).toEqual([literal]);
+ await store.close();
+});
+
+test("counts() reflects documents/chunks", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ const d = store.upsertDocument({ path: "a.md", title: null, contentHash: "h", mtime: 0, size: 1 });
+ store.replaceChunks(d, [
+ { chunkIndex: 0, content: "one", contentHash: "c0", startLine: 1, endLine: 1, tokenCount: 1 },
+ { chunkIndex: 1, content: "two", contentHash: "c1", startLine: 2, endLine: 2, tokenCount: 1 },
+ ]);
+ const c = store.counts();
+ expect(c.documents).toBe(1);
+ expect(c.chunks).toBe(2);
+ expect(c.embeddings).toBe(0);
+ await store.close();
+});
+
+test("getState / setState / deleteState round-trip", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ expect(store.getState("unset")).toBeNull();
+ store.setState("foo", "bar");
+ expect(store.getState("foo")).toBe("bar");
+ store.setState("foo", "bar2");
+ expect(store.getState("foo")).toBe("bar2");
+ store.deleteState("foo");
+ expect(store.getState("foo")).toBeNull();
+ await store.close();
+});
+
+test("ensureEmbeddingModel sets state on first call and clears on change", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+
+ const first = store.ensureEmbeddingModel("modelA", 384);
+ expect(first.wasChanged).toBe(false);
+ expect(store.getState("embedding_model")).toBe("modelA");
+ expect(store.getState("embedding_dimension")).toBe("384");
+
+ // Insert a fake embedding row so we can witness clearing.
+ // Bypass typed API because there's no chunk; SQLite FK would block, so use unrelated insert.
+ // Instead, mimic the change against an existing chunk-less embeddings row:
+ // (skip β€” we verify only state cleanup; full embeddings cascade lives in store.vec.test.ts)
+
+ const second = store.ensureEmbeddingModel("modelB", 768);
+ expect(second.wasChanged).toBe(true);
+ expect(store.getState("embedding_model")).toBe("modelB");
+ expect(store.getState("embedding_dimension")).toBe("768");
+ await store.close();
+});
+
+test("vecUpsert without sqlite-vec throws VEC_EXTENSION_UNAVAILABLE", async () => {
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ expect(() =>
+ store.vecUpsert(1, [0.1, 0.2, 0.3], "m", 3, "h"),
+ ).toThrow(/sqlite-vec/);
+ await store.close();
+});
+
+test("schema-mismatch is surfaced on open", async () => {
+ // Prepare a db at schema 2 (above LATEST=1)
+ const store = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ store.setState("schema_version", "99");
+ await store.close();
+
+ let err: SearchError | null = null;
+ try {
+ const s = await Store.open(makeConfig(), { mode: "read", loadVec: false });
+ await s.close();
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("SCHEMA_MISMATCH");
+});
+
+test("two writers contend β†’ second fails with INDEX_LOCKED", async () => {
+ const first = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ let err: SearchError | null = null;
+ try {
+ const second = await Store.open(makeConfig(), { mode: "write", loadVec: false });
+ await second.close();
+ } catch (e) {
+ err = e as SearchError;
+ }
+ expect(err?.code).toBe("INDEX_LOCKED");
+ await first.close();
+}, 30_000);
diff --git a/tests/core/search/store.vec.test.ts b/tests/core/search/store.vec.test.ts
new file mode 100644
index 00000000..39acf85e
--- /dev/null
+++ b/tests/core/search/store.vec.test.ts
@@ -0,0 +1,162 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { Store } from "../../../src/core/search/store.ts";
+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "../../../src/core/search/types.ts";
+
+let tmp: string;
+let dbPath: string;
+
+function semanticConfig(model: string, dim: number, overrides?: Partial<ResolvedSearchConfig>): ResolvedSearchConfig {
+ const semantic: ResolvedEmbeddingConfig = Object.freeze({
+ enabled: true,
+ provider: "openai-compat",
+ baseUrl: "https://x/v1",
+ model,
+ apiKey: "k",
+ dimension: dim,
+ timeoutMs: 10_000,
+ concurrency: 4,
+ batchSize: 32,
+ });
+ return Object.freeze({
+ vault: tmp,
+ dbPath,
+ ignorePaths: Object.freeze([".git"]),
+ chunkSize: 800,
+ chunkOverlap: 100,
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ semantic,
+ ...(overrides ?? {}),
+ });
+}
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "osb-vec-"));
+ dbPath = join(tmp, "brain.sqlite");
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+});
+
+function vecAvailable(): boolean {
+ try {
+ require("sqlite-vec");
+ return true;
+ } catch {
+ return false;
+ }
+}
+
+function unit(values: number[]): number[] {
+ const norm = Math.hypot(...values);
+ if (norm === 0) return values;
+ return values.map((v) => v / norm);
+}
+
+test("vec round-trip when sqlite-vec is loaded", async () => {
+ if (!vecAvailable()) return;
+ const store = await Store.open(semanticConfig("m1", 4), { mode: "write" });
+ expect(store.vecLoaded()).toBe(true);
+
+ const docId = store.upsertDocument({
+ path: "vec/a.md",
+ title: null,
+ contentHash: "h",
+ mtime: 0,
+ size: 1,
+ });
+ const [c0, c1] = store.replaceChunks(docId, [
+ { chunkIndex: 0, content: "alpha", contentHash: "h0", startLine: 1, endLine: 1, tokenCount: 1 },
+ { chunkIndex: 1, content: "bravo", contentHash: "h1", startLine: 2, endLine: 2, tokenCount: 1 },
+ ]);
+
+ store.vecUpsert(c0!, unit([1, 0, 0, 0]), "m1", 4, "eh0");
+ store.vecUpsert(c1!, unit([0, 1, 0, 0]), "m1", 4, "eh1");
+
+ // Query close to first vector β†’ first chunk is nearest.
+ const hits = store.semanticTopK(unit([0.9, 0.1, 0, 0]), { limit: 5 });
+ expect(hits.length).toBe(2);
+ expect(hits[0]?.chunkId).toBe(c0!);
+ expect(hits[1]?.chunkId).toBe(c1!);
+ expect(hits[0]!.distance).toBeLessThan(hits[1]!.distance);
+ await store.close();
+});
+
+test("deleting a document leaves zero rows in chunk_vec/chunk_vec_map", async () => {
+ if (!vecAvailable()) return;
+ const store = await Store.open(semanticConfig("m1", 4), { mode: "write" });
+ const docId = store.upsertDocument({
+ path: "vec/b.md",
+ title: null,
+ contentHash: "h",
+ mtime: 0,
+ size: 1,
+ });
+ const [c0, c1] = store.replaceChunks(docId, [
+ { chunkIndex: 0, content: "x", contentHash: "h0", startLine: 1, endLine: 1, tokenCount: 1 },
+ { chunkIndex: 1, content: "y", contentHash: "h1", startLine: 2, endLine: 2, tokenCount: 1 },
+ ]);
+ store.vecUpsert(c0!, unit([1, 0, 0, 0]), "m1", 4, "eh0");
+ store.vecUpsert(c1!, unit([0, 1, 0, 0]), "m1", 4, "eh1");
+
+ const beforeVec = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM chunk_vec");
+ const beforeMap = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM chunk_vec_map");
+ expect(beforeVec[0]?.c).toBe(2);
+ expect(beforeMap[0]?.c).toBe(2);
+
+ store.deleteDocument("vec/b.md");
+
+ const afterVec = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM chunk_vec");
+ const afterMap = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM chunk_vec_map");
+ expect(afterVec[0]?.c).toBe(0);
+ expect(afterMap[0]?.c).toBe(0);
+ await store.close();
+});
+
+test("ensureEmbeddingModel drops chunk_vec when dimension changes", async () => {
+ if (!vecAvailable()) return;
+ const store = await Store.open(semanticConfig("m1", 4), { mode: "write" });
+
+ // Touch the vec table to confirm it exists.
+ let tables = store.rawQuery<{ name: string }>(
+ "SELECT name FROM sqlite_master WHERE type='table' AND name='chunk_vec'",
+ );
+ expect(tables.length).toBe(1);
+
+ // Change to a different model+dim.
+ const change = store.ensureEmbeddingModel("m2", 8);
+ expect(change.wasChanged).toBe(true);
+ expect(change.previousModel).toBe("m1");
+ expect(change.currentModel).toBe("m2");
+
+ // Vec table recreated at new dim, embeddings empty.
+ tables = store.rawQuery<{ name: string }>(
+ "SELECT name FROM sqlite_master WHERE type='table' AND name='chunk_vec'",
+ );
+ expect(tables.length).toBe(1);
+ const emb = store.rawQuery<{ c: number }>("SELECT count(*) AS c FROM embeddings");
+ expect(emb[0]?.c).toBe(0);
+ await store.close();
+});
+
+test("dimension mismatch is rejected on vecUpsert", async () => {
+ if (!vecAvailable()) return;
+ const store = await Store.open(semanticConfig("m1", 4), { mode: "write" });
+ const docId = store.upsertDocument({
+ path: "z.md",
+ title: null,
+ contentHash: "h",
+ mtime: 0,
+ size: 1,
+ });
+ const [c0] = store.replaceChunks(docId, [
+ { chunkIndex: 0, content: "x", contentHash: "h", startLine: 1, endLine: 1, tokenCount: 1 },
+ ]);
+ expect(() => store.vecUpsert(c0!, [0.1, 0.2, 0.3], "m1", 4, "eh")).toThrow(/dimension/);
+ await store.close();
+});
diff --git a/tests/core/search/types.test.ts b/tests/core/search/types.test.ts
new file mode 100644
index 00000000..57c62d1a
--- /dev/null
+++ b/tests/core/search/types.test.ts
@@ -0,0 +1,15 @@
+import { test, expect } from "bun:test";
+import { SearchError } from "../../../src/core/search/types.ts";
+
+test("SearchError carries a typed code", () => {
+ const err = new SearchError("INDEX_MISSING", "no index at /tmp/x");
+ expect(err).toBeInstanceOf(Error);
+ expect(err.name).toBe("SearchError");
+ expect(err.code).toBe("INDEX_MISSING");
+ expect(err.message).toBe("no index at /tmp/x");
+});
+
+test("SearchError preserves stack trace", () => {
+ const err = new SearchError("INVALID_INPUT", "bad");
+ expect(err.stack).toContain("SearchError");
+});
diff --git a/tests/core/search/walker.test.ts b/tests/core/search/walker.test.ts
new file mode 100644
index 00000000..2814565f
--- /dev/null
+++ b/tests/core/search/walker.test.ts
@@ -0,0 +1,91 @@
+import { test, expect, beforeEach, afterEach } from "bun:test";
+import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { walkVault } from "../../../src/core/search/walker.ts";
+import { createTempVault, writeMd, writeSymlink, makeConfig } from "../../helpers/search-fixtures.ts";
+
+let vault: string;
+let cleanup: () => void;
+
+beforeEach(() => {
+ const v = createTempVault("walker");
+ vault = v.vault;
+ cleanup = v.cleanup;
+});
+
+afterEach(() => {
+ cleanup();
+});
+
+function collect(cfg: ReturnType<typeof makeConfig>): string[] {
+ const out: string[] = [];
+ for (const f of walkVault(cfg)) out.push(f.relPath);
+ return out.sort();
+}
+
+test("returns *.md files relative to vault, POSIX paths", () => {
+ writeMd(vault, "a.md", "# a");
+ writeMd(vault, "b/c.md", "# c");
+ writeMd(vault, "b/d/e.md", "# e");
+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });
+ expect(collect(cfg)).toEqual(["a.md", "b/c.md", "b/d/e.md"]);
+});
+
+test("skips non-md files", () => {
+ writeMd(vault, "keep.md", "x");
+ writeMd(vault, "drop.txt", "x");
+ writeMd(vault, "Notes/README", "x");
+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });
+ expect(collect(cfg)).toEqual(["keep.md"]);
+});
+
+test("skips ignored bare directory names anywhere in tree", () => {
+ writeMd(vault, "keep.md", "x");
+ writeMd(vault, ".git/dropped.md", "x");
+ writeMd(vault, "node_modules/lib/index.md", "x");
+ writeMd(vault, "deep/.git/also-dropped.md", "x");
+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });
+ expect(collect(cfg)).toEqual(["keep.md"]);
+});
+
+test("skips ignored relative-path directories", () => {
+ writeMd(vault, "keep.md", "x");
+ writeMd(vault, ".obsidian/cache/cached.md", "x");
+ writeMd(vault, ".obsidian/other.md", "x"); // kept (cache is the specific subdir)
+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });
+ expect(collect(cfg)).toEqual([".obsidian/other.md", "keep.md"]);
+});
+
+test("ignores symlinks pointing outside vault", () => {
+ const outside = mkdtempSync(join(tmpdir(), "osb-walker-out-"));
+ writeFileSync(join(outside, "leak.md"), "secret");
+ mkdirSync(join(vault, "inside"), { recursive: true });
+
+ // Symlink file β†’ outside file
+ writeSymlink(vault, "evil.md", join(outside, "leak.md"));
+ // Symlink dir β†’ outside dir
+ writeSymlink(vault, "linked-dir", outside);
+
+ writeMd(vault, "real.md", "x");
+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });
+ const found = collect(cfg);
+ expect(found).toContain("real.md");
+ expect(found).not.toContain("evil.md");
+ expect(found.find((p) => p.startsWith("linked-dir/"))).toBeUndefined();
+ rmSync(outside, { recursive: true, force: true });
+});
+
+test("hidden non-ignored dirs are still walked", () => {
+ writeMd(vault, ".hiddendir/note.md", "x");
+ writeMd(vault, "normal.md", "x");
+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite"), ignorePaths: [".git"] });
+ expect(collect(cfg)).toEqual([".hiddendir/note.md", "normal.md"]);
+});
+
+test("empty directories produce zero files", () => {
+ mkdirSync(join(vault, "empty"), { recursive: true });
+ const cfg = makeConfig({ vault, dbPath: join(vault, "x.sqlite") });
+ expect(collect(cfg)).toEqual([]);
+});
diff --git a/tests/helpers/fake-http.ts b/tests/helpers/fake-http.ts
new file mode 100644
index 00000000..4c0069dc
--- /dev/null
+++ b/tests/helpers/fake-http.ts
@@ -0,0 +1,94 @@
+/**
+ * Tiny Bun-native HTTP server stub for embedding-provider tests.
+ *
+ * Each test can install a handler that decides what to return for any
+ * incoming request. The default handler returns OpenAI-shaped vectors
+ * deterministically derived from the input texts (no randomness β€” so
+ * `toBeCloseTo` checks stay stable).
+ */
+
+
+export interface FakeRequest {
+ readonly method: string;
+ readonly path: string;
+ readonly headers: Record<string, string>;
+ readonly body: unknown;
+}
+
+export interface FakeResponseSpec {
+ status?: number;
+ headers?: Record<string, string>;
+ body?: unknown;
+ delayMs?: number;
+}
+
+type Handler = (req: FakeRequest, callIndex: number) => FakeResponseSpec | Promise<FakeResponseSpec>;
+
+export interface FakeHttp {
+ url: string;
+ close: () => Promise<void>;
+ setHandler: (h: Handler) => void;
+ /** Number of requests received since creation. */
+ callCount: () => number;
+}
+
+function defaultHandler(req: FakeRequest): FakeResponseSpec {
+ if (req.path.endsWith("/embeddings") && req.method === "POST") {
+ const body = (req.body ?? {}) as { input?: string[]; model?: string };
+ const inputs = Array.isArray(body.input) ? body.input : [];
+ const data = inputs.map((text, index) => {
+ // Deterministic 4-dim vector from token count + position.
+ const tokens = text.split(/\s+/).filter(Boolean).length;
+ const v = [tokens, index, text.length, 1];
+ return { object: "embedding", embedding: v, index };
+ });
+ return { status: 200, body: { data, model: body.model ?? "fake-model" } };
+ }
+ return { status: 404, body: { error: "not_found" } };
+}
+
+export async function startFakeHttp(): Promise<FakeHttp> {
+ let handler: Handler = defaultHandler;
+ let count = 0;
+ const server = Bun.serve({
+ port: 0,
+ fetch: async (req) => {
+ const url = new URL(req.url);
+ let body: unknown = null;
+ const ctype = req.headers.get("content-type") ?? "";
+ if (ctype.startsWith("application/json")) {
+ try {
+ body = await req.json();
+ } catch {
+ body = null;
+ }
+ }
+ const headers: Record<string, string> = {};
+ req.headers.forEach((v, k) => {
+ headers[k] = v;
+ });
+ const idx = count++;
+ const resp = await handler({ method: req.method, path: url.pathname, headers, body }, idx);
+ if (resp.delayMs && resp.delayMs > 0) {
+ await new Promise<void>((r) => setTimeout(r, resp.delayMs));
+ }
+ return new Response(
+ resp.body === undefined ? "" : JSON.stringify(resp.body),
+ {
+ status: resp.status ?? 200,
+ headers: { "content-type": "application/json", ...(resp.headers ?? {}) },
+ },
+ );
+ },
+ });
+
+ const url = `http://127.0.0.1:${server.port}/v1`;
+ return {
+ url,
+ close: () => server.stop(true) as unknown as Promise<void>,
+ setHandler: (h: Handler) => {
+ handler = h;
+ },
+ callCount: () => count,
+ };
+}
diff --git a/tests/helpers/mock-embedding.ts b/tests/helpers/mock-embedding.ts
new file mode 100644
index 00000000..4cb1eb95
--- /dev/null
+++ b/tests/helpers/mock-embedding.ts
@@ -0,0 +1,54 @@
+/**
+ * Deterministic in-process embedding provider for indexer tests.
+ *
+ * Vectors are derived from sha256 of the input text: every 4 bytes of
+ * the digest become one signed float in [-1, 1], padded/truncated to
+ * the requested dimension, then unit-normalised. Two equal inputs
+ * always produce the same vector β€” so assertions on chunk identity
+ * stay stable across runs.
+ */
+
+import { createHash } from "node:crypto";
+
+import type { EmbeddingProvider } from "../../src/core/search/embeddings/provider.ts";
+
+export class MockEmbeddingProvider implements EmbeddingProvider {
+ readonly name = "mock";
+ readonly model: string;
+ readonly dimension: number;
+ callCount = 0;
+
+ constructor(opts?: { model?: string; dimension?: number }) {
+ this.model = opts?.model ?? "mock-embed";
+ this.dimension = opts?.dimension ?? 8;
+ }
+
+ embed(texts: ReadonlyArray<string>): Promise<number[][]> {
+ this.callCount++;
+ return Promise.resolve(texts.map((t) => this.vectorFor(t)));
+ }
+
+ async ping(): Promise<{ ok: true; dimension: number }> {
+ return { ok: true, dimension: this.dimension };
+ }
+
+ private vectorFor(text: string): number[] {
+ const v = new Array<number>(this.dimension).fill(0);
+ let acc = createHash("sha256").update(text).digest();
+ let bytes: number[] = Array.from(acc);
+ while (bytes.length < this.dimension * 2) {
+ acc = createHash("sha256").update(acc).digest();
+ bytes = bytes.concat(Array.from(acc));
+ }
+ for (let i = 0; i < this.dimension; i++) {
+ const b1 = bytes[i * 2] ?? 0;
+ const b2 = bytes[i * 2 + 1] ?? 0;
+ v[i] = (b1 - 128) / 128 + ((b2 - 128) / 128) * 0.5;
+ }
+ let s = 0;
+ for (const x of v) s += x * x;
+ const norm = Math.sqrt(s);
+ if (norm === 0) return v;
+ return v.map((x) => x / norm);
+ }
+}
diff --git a/tests/helpers/search-fixtures.ts b/tests/helpers/search-fixtures.ts
new file mode 100644
index 00000000..c364846e
--- /dev/null
+++ b/tests/helpers/search-fixtures.ts
@@ -0,0 +1,76 @@
+/**
+ * Helpers for building short-lived vault directories for search tests.
+ */
+
+import { mkdtempSync, mkdirSync, writeFileSync, symlinkSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join } from "node:path";
+
+import type { ResolvedSearchConfig, ResolvedEmbeddingConfig } from "../../src/core/search/types.ts";
+
+export function createTempVault(prefix: string): {
+ vault: string;
+ dbPath: string;
+ cleanup: () => void;
+} {
+ const vault = mkdtempSync(join(tmpdir(), `osb-${prefix}-`));
+ const dbPath = join(vault, ".open-second-brain", "brain.sqlite");
+ return {
+ vault,
+ dbPath,
+ cleanup: () => rmSync(vault, { recursive: true, force: true }),
+ };
+}
+
+export function writeMd(vault: string, relPath: string, content: string): string {
+ const abs = join(vault, relPath);
+ mkdirSync(dirname(abs), { recursive: true });
+ writeFileSync(abs, content);
+ return abs;
+}
+
+export function writeSymlink(vault: string, relLink: string, absTarget: string): string {
+ const abs = join(vault, relLink);
+ mkdirSync(dirname(abs), { recursive: true });
+ symlinkSync(absTarget, abs);
+ return abs;
+}
+
+export function makeConfig(opts: {
+ vault: string;
+ dbPath: string;
+ ignorePaths?: ReadonlyArray<string>;
+ semantic?: Partial<ResolvedEmbeddingConfig>;
+}): ResolvedSearchConfig {
+ const baseSemantic: ResolvedEmbeddingConfig = Object.freeze({
+ enabled: false,
+ provider: "openai-compat",
+ baseUrl: null,
+ model: null,
+ apiKey: null,
+ dimension: null,
+ timeoutMs: 10_000,
+ concurrency: 4,
+ batchSize: 32,
+ });
+ const semantic: ResolvedEmbeddingConfig = Object.freeze({ ...baseSemantic, ...(opts.semantic ?? {}) });
+ return Object.freeze({
+ vault: opts.vault,
+ dbPath: opts.dbPath,
+ ignorePaths: Object.freeze([
+ ...(opts.ignorePaths ?? [
+ ".git",
+ "node_modules",
+ ".open-second-brain",
+ ".obsidian/cache",
+ ".trash",
+ ".stversions",
+ ]),
+ ]),
+ chunkSize: 800,
+ chunkOverlap: 100,
+ keywordWeight: 0.6,
+ semanticWeight: 0.4,
+ semantic,
+ });
+}
diff --git a/tests/mcp/mcp.test.ts b/tests/mcp/mcp.test.ts
index 41260491..f7a38431 100644
--- a/tests/mcp/mcp.test.ts
+++ b/tests/mcp/mcp.test.ts
@@ -165,6 +165,8 @@ describe("tool listing", () => {
"payment_request_approval",
"payment_request_status",
"payment_request_consume",
+ // Search (added in v0.10.0).
+ "brain_search",
]),
);
// Explicit grep: legacy writable tools are no longer advertised.
@@ -285,8 +287,8 @@ describe("stdio loop", () => {
const list = JSON.parse(lines[1]!);
expect(init.id).toBe(1);
expect(list.id).toBe(2);
- // v0.9.0: 3 core (status/query/health) + 6 Brain + 8 Pay Memory = 17.
- expect(list.result.tools.length).toBe(18);
+ // v0.10.0: 3 core (status/query/health) + 7 Brain + 8 Pay Memory + 1 Search = 19.
+ expect(list.result.tools.length).toBe(19);
});
test("returns parse error for invalid JSON", async () => {
diff --git a/tests/mcp/search.test.ts b/tests/mcp/search.test.ts
new file mode 100644
index 00000000..1c4a5eb8
--- /dev/null
+++ b/tests/mcp/search.test.ts
@@ -0,0 +1,187 @@
+/**
+ * MCP integration tests for `brain_search` and the `search.*`
+ * enrichment on `second_brain_status`.
+ *
+ * Like the Brain MCP suite, these go through `MCPServer.handleRequest`
+ * end-to-end β€” so registration, schema validation, error mapping, and
+ * the actual search query are all exercised together.
+ */
+
+import { afterEach, beforeEach, expect, test } from "bun:test";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { JSONRPC_VERSION, MCPServer, PROTOCOL_VERSION } from "../../src/mcp/index.ts";
+import { indexVault } from "../../src/core/search/indexer.ts";
+import { resolveSearchConfig } from "../../src/core/search/index.ts";
+import { atomicWriteFileSync } from "../../src/core/fs-atomic.ts";
+
+let tmp: string;
+let vault: string;
+let configPath: string;
+const savedEnv: Record<string, string | undefined> = {};
+
+beforeEach(() => {
+ tmp = mkdtempSync(join(tmpdir(), "o2b-mcp-search-"));
+ vault = join(tmp, "vault");
+ mkdirSync(vault, { recursive: true });
+ configPath = join(tmp, "config.yaml");
+ for (const k of ["VAULT_AGENT_NAME", "VAULT_TIMEZONE", "VAULT_DIR", "OPEN_SECOND_BRAIN_CONFIG"]) {
+ savedEnv[k] = process.env[k];
+ delete process.env[k];
+ }
+ process.env["OPEN_SECOND_BRAIN_CONFIG"] = configPath;
+ atomicWriteFileSync(configPath, `vault: ${vault}\nagent_name: claude\n`);
+});
+
+afterEach(() => {
+ rmSync(tmp, { recursive: true, force: true });
+ for (const [k, v] of Object.entries(savedEnv)) {
+ if (v === undefined) delete process.env[k];
+ else process.env[k] = v;
+ }
+});
+
+async function initialize(server: MCPServer): Promise<void> {
+ await server.handleRequest({
+ jsonrpc: JSONRPC_VERSION,
+ id: 1,
+ method: "initialize",
+ params: {
+ protocolVersion: PROTOCOL_VERSION,
+ capabilities: {},
+ clientInfo: { name: "search-test", version: "0" },
+ },
+ });
+ await server.handleRequest({
+ jsonrpc: JSONRPC_VERSION,
+ method: "notifications/initialized",
+ });
+}
+
+function call(server: MCPServer, name: string, args: Record<string, unknown> = {}): Promise<unknown> {
+ return server.handleRequest({
+ jsonrpc: JSONRPC_VERSION,
+ id: 99,
+ method: "tools/call",
+ params: { name, arguments: args },
+ }) as Promise<unknown>;
+}
+
+function makeServer(): MCPServer {
+ return new MCPServer({ vault, configPath });
+}
+
+function writeMd(rel: string, content: string): void {
+ const abs = join(vault, rel);
+ mkdirSync(join(abs, ".."), { recursive: true });
+ writeFileSync(abs, content);
+}
+
+function extractToolResult(resp: any): Record<string, unknown> {
+ // MCP tool/call wraps the handler return value in { content: [{ type:"text", text: "<json>" }] }.
+ const content = resp?.result?.content?.[0];
+ if (content?.type === "text" && typeof content.text === "string") {
+ return JSON.parse(content.text);
+ }
+ // Some servers also expose `structuredContent` directly; accept both.
+ if (resp?.result?.structuredContent) return resp.result.structuredContent;
+ return resp?.result ?? {};
+}
+
+test("brain_search is advertised by tools/list", async () => {
+ const server = makeServer();
+ await initialize(server);
+ const r = (await server.handleRequest({
+ jsonrpc: JSONRPC_VERSION,
+ id: 10,
+ method: "tools/list",
+ })) as any;
+ const tools = r.result.tools as Array<{ name: string }>;
+ expect(tools.find((t) => t.name === "brain_search")).toBeDefined();
+});
+
+test("brain_search happy path returns paths and respects 600-char content cap", async () => {
+ writeMd("notes/foo.md", "# Foo\n\n" + "fox ".repeat(400));
+ const cfg = resolveSearchConfig({ vault, configPath });
+ await indexVault(cfg);
+
+ const server = makeServer();
+ await initialize(server);
+ const resp = await call(server, "brain_search", { query: "fox", limit: 5 });
+ const body = extractToolResult(resp);
+ const results = body["results"] as Array<{ content: string; path: string; searchType: string; score: number }>;
+ expect(Array.isArray(results)).toBe(true);
+ expect(results.length).toBeGreaterThan(0);
+ expect(results[0]?.path).toBe("notes/foo.md");
+ expect(results[0]?.content.length).toBeLessThanOrEqual(600);
+ // The MCP shape does NOT expose diagnostic score components.
+ expect((results[0] as Record<string, unknown>)["keywordScore"]).toBeUndefined();
+});
+
+test("brain_search rejects missing query with INVALID_PARAMS", async () => {
+ const server = makeServer();
+ await initialize(server);
+ const resp = (await call(server, "brain_search", {})) as any;
+ expect(resp?.error).toBeDefined();
+ expect(resp.error.code).toBe(-32602); // INVALID_PARAMS
+});
+
+test("brain_search rejects path_prefix that escapes vault", async () => {
+ writeMd("a.md", "# A");
+ const cfg = resolveSearchConfig({ vault, configPath });
+ await indexVault(cfg);
+
+ const server = makeServer();
+ await initialize(server);
+ const resp = (await call(server, "brain_search", { query: "A", path_prefix: "../etc/" })) as any;
+ expect(resp?.error?.code).toBe(-32602);
+ expect(resp.error.message).toContain("path_prefix");
+});
+
+test("brain_search on missing index surfaces INTERNAL_ERROR with hint", async () => {
+ const server = makeServer();
+ await initialize(server);
+ const resp = (await call(server, "brain_search", { query: "fox" })) as any;
+ expect(resp?.error?.code).toBe(-32603); // INTERNAL_ERROR
+ expect(resp.error.message).toMatch(/not initialised|index/i);
+});
+
+test("brain_search rejects limit > 50", async () => {
+ writeMd("a.md", "# A");
+ const cfg = resolveSearchConfig({ vault, configPath });
+ await indexVault(cfg);
+
+ const server = makeServer();
+ await initialize(server);
+ const resp = (await call(server, "brain_search", { query: "A", limit: 100 })) as any;
+ expect(resp?.error?.code).toBe(-32602);
+});
+
+test("second_brain_status includes search block after indexing", async () => {
+ writeMd("a.md", "# A\n\nbody");
+ const cfg = resolveSearchConfig({ vault, configPath });
+ await indexVault(cfg);
+
+ const server = makeServer();
+ await initialize(server);
+ const resp = await call(server, "second_brain_status", {});
+ const body = extractToolResult(resp);
+ const search = body["search"] as Record<string, unknown> | undefined;
+ expect(search).toBeDefined();
+ expect(search!["exists"]).toBe(true);
+ expect(search!["documents"]).toBe(1);
+ expect(search!["schema_version"]).toBe(1);
+});
+
+test("second_brain_status reports search.exists=false when index missing", async () => {
+ const server = makeServer();
+ await initialize(server);
+ const resp = await call(server, "second_brain_status", {});
+ const body = extractToolResult(resp);
+ const search = body["search"] as Record<string, unknown> | undefined;
+ expect(search).toBeDefined();
+ expect(search!["exists"]).toBe(false);
+ expect(search!["hint"]).toMatch(/o2b search index/);
+});